Mechanism and Emergence of Stacked Attention Heads in Multi-Layer Transformers
Authors: Tiberiu Mușat
Venue: ICLR 2025 Poster (2025)
Keywords: mechanistic interpretability
large language models
transformers
emergent abilities
curriculum learning
reasoning
Primary area: interpretability and explainable AI

Abstract: In this paper, I introduce the retrieval problem, a simple yet common reasoning task that can be solved only by transformers with a minimum number of layers, which grows logarithmically with the input size. I empirically show that large language models can solve the task under different prompting formulations without any fine-tuning. To understand how transformers solve the retrieval problem, I train several transformers on a minimal formulation. Successful learning occurs only under the presence of an implicit curriculum. I uncover the learned mechanisms by studying the attention maps in the trained transformers. I also study the training process, uncovering that attention heads always emerge in a specific sequence guided by the implicit curriculum.

Published as a conference paper at ICLR 2025
MECHANISM AND EMERGENCE OF STACKED ATTEN- TION HEADS IN MULTI-LAYER TRANSFORMERS
Tiberiu Muşat
ETH Zürich, Switzerland Giotto.ai, Switzerland
ABSTRACT
In this paper, I introduce the retrieval problem, simple yet common reasoning task that can be solved only by transformers with a minimum number of layers, which grows logarithmically with the input size. empirically show that large lan- guage models can solve the task under different prompting formulations without any fine-tuning. To understand how transformers solve the retrieval problem, train several transformers on a minimal formulation. Successful learning occurs only under the presence of implicit curriculum. uncover the learned mech- anisms by studying the attention maps in the trained transformers. I also study the training process, uncovering that attention heads always emerge in specific sequence guided by the implicit curriculum.
Retrieval Question
Conditional Retrieval Question
A 7
O ??
Alice lives in Wonderland. Harry's mother is Alice. [...]
A
C = B
Wizards live with their mothers. [...] Harry is a wizard. [...]
D = C
[...]
What is the value of D?
Where does Harry live?
Large Language
Large Language
The value of D is 7.
Harry lives in Wonderland.
[..]] irrelevant information to ensure full
[-..] inrelevant information to ensure full reasoning e.g. Mary pirate. Aslan lives in Narnia, etc.
retrieval, e.g. H 2, G = H, etc.
Figure Illustrative examples of retrieval and conditional retrieval questions.
INTRODUCTION
How do neural networks solve the tasks that they are trained on? there a clear algorithm hiding behind the millions of unintelligible weights and biases? These are the questions that the field of mechanistic interpretability tries answer. If successful, this line of research could lead better understanding of neural networks and the development of Al systems with increased safety, reliability, and efficiency (Doshi- Velez & Kim 2017, et 2020 Elhage al. 2021).
Transformers (Vaswani 2017) have become the dominant architecture in natural language process- ing, achieving of results on wide range of tasks (Brown) 2020 Achiam et al. 2023).
Recent interpretability research has successfully uncovered the mechanisms learned by single-layer
(Nanda et al. 2023 Quirke al. 2023) and two-layer (Olsson et al., 2022) transformers. However, understanding the mechanisms learned by deeper transformers remains an open problem. Auto- matic circuit analysis of large language models provides valuable insights about isolated circuits that span several layers, but such circuits remain not fully understood (Wang al. 2022] Conmy et al. 2023). Therefore, understanding the mechanisms of multi-layer transformers is a crucial step towards understanding state-of-the-art language models.

---

I also propose a more general variant of the retrieval problem, which term the conditional retrieval problem, where each retrieval step could depend on multiple previously retrieved values, not just the last one. For example, given the input sequence ... ayz...ay. aax. .a, predicting the token would constitute conditional retrieval problem. The retrieval steps in the retrieval problem are Published as a conference paper at ICLR 2025
2 MY CONTRIBUTION
In this paper, try to answer the following questions:
QI. Are there tasks that can be solved only by transformers with a specific depth?
Q2 Are large language models able to solve such tasks without specific fine-tuning?
Q3. What is the mechanism that transformers use to solve the task?
Q4. How does this mechanism emerge during training?
I answer Q1 positively by introducing the retrieval problem, as well close variant that I term the conditional retrieval problem (Section 3). answer Q2 positively in Section 4 by empirically showing that large language models can solve both problems without any specific fine-tuning under multiple prompting formulations. In Section 5 provide formal proof that the retrieval problem requires minimum number of transformer layers that is logarithmic in the imput size.
This suggests that large language models have learned complex mechanism formed by multiple stacked attention heads. To elucidate this mechanism (Q3), train several transformers on minimal formulation of the retrieval problem (Sections and 7 In Section X by studying the attention maps in the trained transformers, uncover multiple possible mechanisms that I term retrieval heads. Regarding the training process (Q4), I find that retrieval heads emerge only under the presence of an implicit curriculum and always specific sequence (Section 9
The retrieval problem also has important implications for emergent abilities in large language mod- els (Wei et al. 2022). which I discuss in Section I0
THE RETRIEVAL PROBLEM
3.1 DEFINITION
The retrieval problem is directly inspired by the induction problem introduced by Olsson et al. (2022). Given a sequence of tokens ... ab. .a, the induction problem requires the model predict the token b. directly extend this formulation by increasing the number of induction steps to D. Given input sequence ... ID-1
.t 22 102 I. the retrieval problem consists in predicting the token æp. By setting D = 1, we recover exactly the original induction problem. Throughout this paper, I also refer the tokens in the retrieval chain using capital letters (i.e., A for Zp. B for æ1, C for Z2, and so on).
1 D..........
perfectly linear, while the conditional retrieval, they form a directed acyclic graph.
3.2 MOTIVATION
The retrieval problem is implicitly present as a subproblem in many common language tasks such as working with relations between persons, tracking the evolution of concept, solving mathematical and reasoning problems, programming, and many more. Consider the following real-world example from the Wikipedia article on Ilamas:
"Llamas are social animals and live with others herd. [...] A cria (from Spanish for 'baby') is the name for a baby Ilama. alpaca, vicuña, or guanaco. Crias are typically born with all the females of the herd gathering around."
An autoregressive language model trying to predict the second occurrence of the word herd (rather than flock, group, or pack) would need to first retrieve the fact that crias are llamas, and then it to retrieve the fact that llamas live in herds. This process is essentially a retrieval problem with = 2.
From point of view, the retrieval problem is essentially about working with relations between tities, which is fundamental for language and reasoning. This makes it an ideal testbed for studying the inner workings of large language models.
2

---

Published as a conference paper at ICLR 2025
4 LARGE LANGUAGE MODELS
To better illustrate the task and to enable benchmarking of large language models, propose specific formulations of the retrieval problems: 3 retrieval formulations and 2 conditional retrieval formulations.
F1. Equations formulation: ".=
c=?"
F2. Lives-with formulation: "Alice lives in Switzerland. Bob lives with Alice. Charlie lives with Bob. David lives with Charlie. Where does David live?"
F3. Kingdoms formulation:
"Alice lives in Novaria. Novarians believe in harmonianism. Har- monianists eat lamb. Lamb contains Zephyrium. Zephyrium causes Chronogy. Who has Chronogy?"
F4. Functions formulation (conditional retrieval): "f(2)= 3. g == a. g(a) ?
F5. Relativey formulation (conditional retrieval): "Jane lives in Switzerland. Alex's mother is Jane. Engineers live with their mothers. Alex is an engineer. Where does Alex live?"
To ensure that the retrieval problem is not trivially solvable by just finding the noun that fits the question, I interleave multiple retrieval chains in the same question. This ensures that the model performs the entire reasoning chain. To facilitate benchmarking, also ask the model to output the answer directly without any additional words and repeat sampling until an acceptable answer generated. In Appendix A provide examples of the full prompts, correct answers, and acceptable answers for each formulation.
I test the large language models on 500 randomly generated questions for each formulation. The results are presented Figure 2 For the equations formulation, also measure the accuracy for different difficulty levels D (number of equations) and I find that large language models solve almost perfectly for D < Great performance also achieved on the lives-with and kingdoms formulations with D 5, well as the conditional retrieval formulations fianctions and relatives.
Accuracy state-of of-thh--rtt language models retrieval lequations
Accuracy of GPT different retrieval and conditional retrieval tasks
100
GPT-4o
"Lives with" fommuation (D=5)
Claude 3.5 Sonnet
Gemini 1.5 Pro
80
"Kingdoms" formulation (Dm5)
Accuracy
70
60
"Functions" fomulation
30
Raatiionn SURKSIN
"Ralatives
20
10
12
40
80
100
Number retrival staps (D)
Accunacy 196
Figure 2: Accuracy of large language models the retrieval and conditional retrieval problems. Dashed lines indicate the accuracy of random guessing. Full prompts and benchmarking details are provided in Appendix
THEORETICAL ANALYSIS OF INFORMATION FLOW
In this section, theoretically establish that solving the retrieval problem requires a minimum num- ber of transformer layers that grows logarithmically with the number of retrieval steps D. model the information flow between different positions during self-attention under the following simplifying assumptions:
Assumption 1. During self-attention, a position can only attend to another position if they already share piece of information.

---

Published as a conference paper at ICLR 2025
This assumption is motivated by the fact that position only attend to another position if their key and query vectors align. Constructing aligned key-query pairs is only possible if the two positions already share some information. More precisely, the shared information must be located in the row spaces of the query and key matrices of the attending and attended positions, respectively.
Assumption 2. When a position attends another position, it retrieves all the information contained in the attended position.
This assumption simplifies the analysis by ignoring network capacity limitations. Solving this set- ting will give us a lower bound on the number of layers required to solve the retrieval problem in the case of a limited network capacity.
X4
X5
X3
X4
X2
X3
X1
Xg
X5
Figure 3: Positions that contain shared information before any trans- former layers in the case of D = 5. Top edges denote shared token embeddings. Bottom edges denote shared positional encodings.
Based on this simplified model of the attention mechanism, we can prove the following result:
Theorem 1. The last position in the sequence cannot retrieve the embedding vector &D of the target token with transformer layers if < log;(2D).
This implies that at least log, (2D) transformer layers are required to solve retrieval problem with D retrieval steps. Note that this result is lower bound that does not take into account the limitations of network capacity, causal masking, or training dynamics. In practice, we can expect the number of required layers to be even higher.
I defer the complete proof Appendix B The intuition behind this result comes from the fact that the last position in the sequence cannot retrieve the target token æp without retrieving everything in-between (i.e., 31, Ø2, @ D--1). However, it is possible to show that the number of retrieved tokens can grow most by factor of 3 after each attention layer, hence the logarithmic growth of the number of required layers.
MINIMAL PROBLEM FORMULATION
In order better study the mechanism by which transformers solve the retrieval problem, I introduce
minimal formulation of the retrieval problem with N retrieval chains, D retrieval steps per chain, and K embedding dimensions. use N and K = throughout. Each chain contains + unique symbols forming D pairs and one query. For every input sequence, each of the N(D 1) unique symbols is assigned K K-ddimnnsional vector whose components are sampled i.i.d from standard normal distribution.
I create the input sequences by perfectly interleaving the pairs of symbols forming each retrieval chain, followed by the N query symbols. I randomly shuffle the query vectors. I also shuffle the input pairs from different chains within the same retrieval step. Finally, concatenate each token embedding with K K-dimensional rotary positional encoding (Su et al. (023). Each input sequence will contain N(2D + 1) vectors of dimension 2K. The output sequences consist of N vectors, one for each query token.
7 IMPLICIT CURRICULUM & NUMBER OF LAYERS
I consider two possible formulations: an implicit curriculum (IC) formulation and non-IC formu- lation. In the IC formulation, the target vectors have DN dimensions and contain all the tokens forming each retrieval chain concatenated (except the query token Æg). In the non-IC formulation, the target vectors are --dimennioona and contain only the last token of each retrieval chain, namely D-
4

---

Published as a conference paper at ICLR 2025
My initial experiments suggest that the implicit curriculum (IC) plays crucial role in the successful learning of the retrieval problem. To better quantify this effect, conduct two comprehensive sets of experiments, one for each formulation (IC and non-IC). For each formulation, train 64 transformers with to 8 layers (8 transformers for each number of layers). plot the final validation loss averaged across all runs with the same number of layers in Figure (left).
To better understand the connection between the number of layers and the difficulty of the retrieval problem, I also plot the partial validation loss for each position in the retrieval chains in the IC formulation (Figure 4 right). use D = 5 for the IC formulation to better illustrate this connection, but only D for the non-IC formulation to illustrate the importance of the implicit curriculum.
Final Validation by Layers
Partial
Layers
1.2
K. D=5
1.0
0.0
X,101
0005
0.6
0.6
0.4
0.2
0.0
0.0
Figure 4: Final validation loss by number of layers, averaged across multiple runs. Left: IC vs. non-IC formulations. Right: Partial validation loss for each position the retrieval chains (IC only).
7.1 TRAINING DETAILS
For each formulation and number of layers, I train 8 transformers following the recipe Radford et al. (2019). Each transformer has 8 attention heads per layer and residual streams of size 128. train for 10k steps using the Adam optimizer (Kingma 2014) with a learning rate of -3, decoupled weight decay of 0.1 (Loshchilov) 2017). a batch size of 512, 220 randomly generated training amples, layer normalization (Ba et al. 2016), no dropout, and mean squared error loss. We measure the final validation loss by averaging the validation loss over the last 100 training steps.
7.2 RESULTS
First, observe that the IC formulation is essential for successful learning. In the non-IC formulation, the transformers fail to learn the retrieval problem even for 3. regardless of the number of layers. I confirm that for 100% of the non-IC runs, the final validation loss is above 0.7.
Second, empirically confirm the connection between the number of layers and the difficulty of the retrieval problem. For the IC formulation, the later positions in the retrieval chains (corresponding to greater D) are more difficult to learn and require more layers.
Third, I find our first hint regarding the emergence of retrieval heads. During training with IC, the partial losses for earlier positions in the retrieval chains always decrease faster than the partial losses for later positions. confirm that in 100% of the IC runs. the partial loss goes below 0.5 for œ; first, then for Ø2, and so on. will further investigate this phenomenon in section
THE CIRCUITS LEARNED
To understand the mechanism learned by transformers to solve the retrieval problem, train three transformers (denoted as A, B, and C) with 12 layers and only one attention head per layer on the retrieval problem with D 3. then manually reverse-engineer the circuits learned by the transformers by studying their attention maps. The uncovered circuits are depicted Figure I describe my reverse-engineering process in detail in Appendix
5

---

Published as a conference paper at ICLR 2025
Transformer A
Transformer
Transformer
Lewer
Laper
Lowr
12
11
12
11
10
11
7
10
Figure 5: Reverse-engineered circuits from three 12-layer transformers trained the retrieval problem with D 3 and IC.
8.1 TRAINING DETAILS
I follow similar training recipe as in the previous section. Each transformer has 12 layers, one attention head per layer, and residual streams of size 128. train each transformer for 24k steps, a batch size of 128, and 262k randomly generated training examples (IC).
8.2 RESULTS
I find that transformers A, B, and C achieve validation mean squared error of less than 0.01. By studying the attention maps in the trained transformers, observe that most attention heads do not perform any useful computation. Only few attention heads responsible for the information flow. Their behavior easily interpretable (see Appendix C
I manuaaly reverse-engineer the entire circuits learned by the three transformers, which are depicted in Figure 5 perform extensive validations of the circuits using ablations. My reniese--n process is described in detail in Appendix E
I observe two interesting facts about the reverse-engineered circuits:
First, in all three transformers, the first relevant attention head connecting the first and second tokens in each input pair, enabling the subsequent attention heads to attend to the second token in the pair using the value the first token. This mechanism is highly remi- niscent of the induction head mechanism (Olsson al. 2022: Reddy 2023).
ii Second, except for the first attention head, the circuits learned by the transformers are very different. Interestingly. none of the transformers use the minimum number of attention heads required to solve the retrieval problem for D = 3. All transformers use 4 atten- tion heads, but it is possible to use only 3 (for example, by combining layers 7 and 11 in transformer A).
EMERGENCE OF ATTENTION HEADS DURING TRAINING
To better understand how the retrieval heads emerge during training. I train a 24-layer transformer (denoted Transformer D) on the retrieval problem with /) and IC. manually reverse-engineer the circuits learned by Transformer D following the same procedure described in Appendix E Af- terward, measure the attention during training for each attention path the reverse-engineered circuit.
9.1 TRAINING DETAILS
I follow a similar training recipe as in the previous sections. Transformer D has 24 layers, one attention head per layer, and residual streams of size 512. train for 6400 steps (800 epochs), batch size of 256, and 262k randomly generated training examples (IC, D = 4). To speed up the training and reduce the checkpoint size, remove the MLPs and reduce the head size 16.

---

Published as a conference paper at ICLR 2025
Training validation Transformer
Volidation for position retrieval chain
1.0
Training Luss
X.
validation Logs
Xa [C)
X, (D1
X,
0.4
0.4
0.2
0.0
100
200
300
400
500
600
700
800
100
200
300
400
500
600
700
800
Epoch
Epoch
Figure 6: Loss during training of Transformer D (24 layers) with IC and = Left: training and validation loss. Right: partial validation loss for each position in the retrieval chain.
I save checkpoint every 10 epochs (80 steps) that later use to measure the average attention for each attention path in the engineered circuit, at each epoch during training, using 32 input sequences.
Transformer D
Epoch attention emergence
467
Layer
Layer
24
23
23
434
22
8%
22
14
60
14
653
D
E
C ...
B
A
D
C
D
B
C
B
Figure 7: Left: Reverse-engineered circuits of Transformer D. Right: the epoch when the average attention goes above 0.5 for each attention path.
9.2 RESULTS
Transformer D achieves validation mean squared enror of 0.031. I plot the training, validation, and partial validation loss in Figure 6 Using the same reversc-engineerring procedure, uncover more complex circuit than before, with multiple paths connecting the same positions (Figure 7 left). After ablation, the mean squared error increases to 0.045.
For every checkpoint, I measure the average attention for each attention path in the reverse- engineered circuit. I approximate the attention between checkpoints using linear interpolation. display the plots for each attention path in Appendix [ Finally, I show the first epoch when the average attention goes above 0.5 for each attention path (Figure right).
By analyzing the partial loss curves and the emergence of attention paths, make the following observations:
After 450 epochs of slow learning, an induction head that can retrieve token emerges abruptly on layers 14 and 21. This drives down the first partial loss.
ii Quickly after, another attention head emerges on layer 22. This head reuses the induction head (with slight adjustments) to retrieve token C and drive down the second partial loss.
iii. Finally, two more heads emerge on layers 23 and 24 that reuse heads 14, 21, and 22 to retrieve tokens D and E, respectively. This drives down the last partial losses.
iv. Head 24 emerges much later than head 23, possibly due to the greater modifications re- quired to reuse the existing circuit.

---

Published as a conference paper at ICLR 2025
Together, these observations strongly suggest the following possible explanation for the importance of the implicit curriculum: The implicit curriculum provides a sequence of increasingly complex tasks that enables learning the entire retrieval mechanism one head at time, starting with induction head.
10 DISCUSSION ON EMERGENT ABILITIES
The retrieval problem has interesting connections to the emergent abilities large language models (Wei al., 2022). An ability is emergent if it is not present in smaller models but is present in larger models. Understanding emergence is an important direction because could potentially allow us predict what abilities future models may have, as well provide new insights into how to train more capable language models.
The existence of tasks that require minimum number of layers, such as the retrieval problem, provides a possible explanation for the emergence of new abilities in large language models. As the model grows in size, becomes possible to learn more complex circuits that would be impossible to learn in smaller models. This unlocks new abilities that were previously unattainable.
10.1 EMERGENT ABILITIES ARE NOT A MIRAGE
Schaeffer et al. (2023) have previously suggested that the emergence of new abilities in large lan- guage models just "mirage" that appears only under nonlinear discontinuous metrics. My work provides a very strong counterargument to this claim if we consider the ability of model solve the retrieval problem. A transformer cannot solve the retrieval problem with specific difficulty unless it has the minimum number of necessary layers.
10.2 THE IMPLICIT CURRICULUM OF NATURAL LANGUAGE
Is possible to understand how training on natural language data leads to reasoning abilities? Chan al. (2022) previously found that natural language has specific data distributional properties that enable emergent in-context learning in transformers. Is it possible to achieve a similarly insightful understanding of the emergence of reasoning abilities in general, not just in-context learning?
The retrieval problem provides a promising avenue for answering this question. In Sections 7 and g we saw that retrieval heads can only emerge gradually, one by one, under the presence of an implicit curriculum that provides a sequence of increasingly complex tasks. This suggests that one property of natural language data is incredibly important the emergence of reasoning abilities: the presence of very diverse set of tasks with varying levels of difficulty.
11 RELATED WORK
Single-layer transformers. Perhaps the most well-studied setting for single-layer transformers is the problem of modular addition. |Nadaa et al. (2023) show that transformers solve modular addition by arranging the embedding vectors circular structure and leveraging the attention mechanism to perform trigonometric operations. [hhong et al. (2024) extend this work by uncovering other al- gorithms and embedding structures. Even the training dynamics are beginning to be understood, with Ding et al. (204)) studying the survival of initial circular representations and my previous work effective theory of the training dynamics by modeling the embeddings as a particle system. Quirke et al. (2023) train single-layer transformer with three attention heads on the problem of n-digit integer addition. They find that transformers break down the multi-digit addition task into parallel, digit-specific streams, using different algorithms for various digit posi- tions.
(Musat) 2024) proposing
Two-layer transformers. By studying two-layer transformers, Olsson al. 2022 uncover mechanism termed induction head that, given an input sequence ab... a, can predict One pos- sible use of an induction head is sequence copying, but the authors argue that it also perform more high-level functions such as translation. The induction head formed by two stacked atten- tion heads. The first head copies into the residual stream of b the value of the previous token a. The
8

---

Published as a conference paper at ICLR 2025
second head is then able to attend to the token b and copy it into the residual stream of the final token, Reddy (2023) explains the emergence of the induction head during training by the sequential learning of three nested logits enabled by implicit curriculum.
Large language models. Several studies on large language models use automated or semi- automated methods to isolate circuits that solve a specific task (Conmy et al., 2023; Goldowsky- Dill et al.] 2023). Such circuits often span many layers, but their mechanisms remain not fully understood (Wang et al. 2022). Attention heads large language models are often strongly inter- dependent, which makes it difficult to isolate and understand individual heads (Bricken et al. 2023). In large language models, even the simple task of greater-than comparison, which could princi- ple be solved by single-layer transformer, is solved by complex mechanism formed by multiple attention heads and MLPs (Hanma et al. 2024).
12 CONCLUSION
In this work, I introduced and studied the retrieval problem, simple task that requires transformers to retrieve information from multiple positions in the input sequence. showed that the retrieval problem requires a certain number of layers to be solved. By training transformers on a minimal formulation of the task, found that transformers solve the task using mechanism that resembles an induction head. found that this mechanism emerges gradually with the help of an implicit curriculum, starting with induction head and then adding more heads one by one.
Limitations. My analysis of transformers trained on minimal formulation of the retrieval prob- lem might not generalize perfectly large language models. also do not provide full explanation of the training dynamics of transformers on the retrieval problem. Further research is needed to fully understand the multi-layered circuits learned by large language models and the training dynamics that enable their learning.
Acknowledgments. would like to thank the anonymous reviewers for their valuable feedback.

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL 06450
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Tumeer Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-oodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- circuits.pub/// //t////mmmmtnnffftttuuuee
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 141005 2020.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aaditya Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learn- ing in transformers. Advances in Neural Information Processing Systems, 35:18878-1889, 2022.
Arthur Conmy, Augustine
Aengus Lynch, Stefan Heimersheim, and Adrià Gariga- Alonso. Towards automated circuit discovery for mechanistic interpretabilitty Advances in Neural Information Processing Systems, 36: 1311-16352,,,, 2023.
Xiaoman Delores Ding, Zifan Carl Guo, Eric J Michaud, Ziming Liu, and Max Tegmark. Survival of the fittest representation: A case study with modular addition. arXiv preprint 2405:24405.77420, 2024.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.0866 2017.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/ttransfomerrciiciiss wttp:/aisssppptt22/////////ooo//
Nicholas Goldowsky-Dill. Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2334.05569, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.669 2014.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Tiberiu Musat. Clustering and alignment Understanding the training dynamics in modular addition. 2024. httt:ss//ppaeeeeeeeeeeee//oooooa
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arX1v:2301.05217, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 50)):,000240011 2020.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph. Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 22Xi 2022.
10

---

Published as a conference paper at ICLR 2025
Philip Quirke al. Understanding addition in transformers. arXiv preprint arX1::2310.13121, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2023.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. large language models mirage?
Are emergent abilities of In A. Oh, т. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neural Information Pro- cessing Systems, volume 36, pp. 55565-55588 Curran Associates, Inc., 2023. URL
Jianlin Su, Yu Lu. Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding, 2023. URL /ttp:///////i...//a/ wtg/bbs/
2104. 2104.09866 09864
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small. arXiv preprint wrXiv:2211.00594 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768) 07682, 2022.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
11

---

Published as a conference paper at ICLR 2025
A RETRIEVAL AND CONDITIONAL RETRIEVAL PROMPTS
Below I provide one complete example for each of the retrieval and conditional retrieval formula- tions used in the paper. The examples are generated using the same programs used for benchmarking the large language models. Each example consists of prompt. a correct answer, and acceptable an- swers. Acceptable answers are used to filter incoherent answers by repeatedly sampling from the model until an acceptable answer is found.
A.1 EQUATIONS FORMULATION (D 5)
b = 2
3
O
a
b
= a d
= C
= f
g
= h
K
=
0
= j
= n
p
I = O
What is the value of s? Say directly only the numeric value, without any other words.
Correct:
Acceptable: 0, 1, 2, 3
A.2 LIVES-WITH FORMULATION (D 5)
Charlie lives in Cairo David lives Delhi Alice lives in Berlin Bob lives in Amsterdam Henry lives with David Eve lives with Charlie Frank lives with Alice Grace lives with Bob Kate lives with Grace Larry lives with Frank Jack lives with Eve Isabelle lives with Henry Mary lives with Jack
Olivia lives with Isabelle Nick lives with Kate
Peter lives with Larry Rose lives with Peter Queen lives with Nick Tom lives with Olivia Sam lives with Mary
12

---

Published as a conference paper at ICLR 2025
Where does Rose live? Say directly only the name of the city, without any other words.
Correct: Berlin
Acceptable: Amsterdam, Berlin, Cairo, Delhi
A.3 KINGDOMS FORMULATION
Bob lives in Silvania.
Alice lives in Novaria.
Charlie lives in Aurora
David lives in Florinia. Silvanians believe in celestianism.
Novarians believe in harmonianism.
Aurorans believe in elysianism.
Florinians believe in luminism.
Luminists eat beef. Elysianists eat pork. Harmonianists eat lamb. Celestianists eat chicken. Beef contains Astralyte. Chicken contains Nephryon. Lamb contains Zephyrium.
Pork contains Virellium. Zephyrium causes Chronogy.
Astralyte causes Aetherflux.
Virellium causes Somnosis.
Nephryon causes Synthemia.
Who has Chronogy? Say directly the name without other words.
Correct: Alice
Acceptable: Alice, Bob, Charlie, David
A.4 FUNCTIONS FORMULATION (CONDITIONAL RETRIEVAL)
a00) 3
a(1) = 2
( 0
a3) =
b(0)
b(1) = 3
b(2) = 2
b(3) 0
c(0) = 1
c(1) 0
c(2) = 3
c(3) 2
d(0) = I
d(1) 0
d(2) = 2
d33) = 3
b
a
=
d
0
2
3
13

---

Published as a conference paper at ICLR 2025
=
What is the value of f(i)? Say directly only the numeric value, without any other words.
Correct: 3
Acceptable: 0, 1, 2, 3
A.5 RELATIVES FORMULATION (CONDITIONAL RETRIEVAL)
Penny lives in Canada
Lily lives in Brazil.
Isabelle lives in France.
Cathy lives in Kenya.
George lives in Italy
Adam lives in Mexico. Kevin lives in Peru. Ed lives in Laos
Hank lives in Germany. Mike lives in Japan
Jane lives in England.
Fred lives in Hungary
Dana lives in Norway.
Olivia lives
Qatar.
Bob lives in Denmark.
Nancy lives
Argentina.
John' mother is Jane.
John sister is Olivia
John father is Ed.
John brother is Mike.
Chris 's mother is Penny.
Chris s sister is Dana.
Chris s father is Adam.
Chris 's brother is George.
Diana s mother is Nancy.
Diana", s sister is Isabelle.
Diana father is Hank.
Diana's brother is Bob.
Eve's mother is Lily.
Eve's sister is Cathy
Eve's father is Fred.
Eve's brother is Kevin. Doctors live with their brothers.
Lawyers live with their mothers. Teachers live with their sisters. Engineers live with their fathers. John works as doctor.
Chris works an engineer. Diana works Eve works as lawyer.
teacher.
Where does Eve live? Say directly only the name, without any other words.
Correct: Brazil
Acceptable: Argentina, Brazil, Canada, Denmark, England. France. Germany, Hungary, Italy, Japan, Kenya, Laos,, Mexico, Norway, Peru, Qatar
14

---

Published as a conference paper at ICLR 2025
B PROOF OF THEOREM 1 (MINIMUM NUMBER OF LAYERS)
In this section, I provide the complete formal proof for Theorem 1 stated in Section 5 Recall the two assumptions underlying simplified model of the attention mechanism:
Assumption 1. During self-attention, a position can only attend to another position if they already share piece of information.
Assumption 2. When a position attends to another position, it retrieves all the information contained in the attended position.
X4
X5
X3
X4
X2
X3
...
X1
X2
X
X
X5
Figure 8: Positions that contain shared information before any trans- former layers in the case of D = 5. Top edges denote shared token
embeddings. Bottom edges denote shared positional encodings.
Definition 1. Let's consider all relevant input positions ordered by their reachability from the last token following the paths of shared information before any attention layers, exactly as de- picted in Figure [8 (e.g., (1 2 i E 1,... 2D In other words, 822 and 82 are the positions of the second and first occur- rences of æx, respectively. For example, 80 is the last position in the input sequence, 81 is the first occurrence of æu, 82 is the second occurrence æ1, $2D is the position of the target token DD, and so
). Let's denote this sequence of positions as 8; for
Definition 2. Let's denote as Tt, the residual stream at postition 8; after layer t. Before any trans- former layers, the residual stream contains only the token embedding and positional enconding:
To,s [1/2] + P [(+1)/2] for all E
2D),
(1)
where I.] denotes the floor function, at is the k-th token embedding, and ps is the positional encoding the k-tth input pair.
During self-attention, a position can only attend to another position if they already contain shared token embedding positional encoding.
Lemma 1.
Proof. In the retrieval problem, the token values and pair positions are assigned randomly and independently. Knowledge of token or position does not provide any information about any other token or position. Therefore, the only way for two positions to share information is if they already contain shared token embedding or positional encoding.
Note that before any attention layers, the position 8; will contain information shared only with positions s;-1 and 8;41- This is because the token embedding x; shared by $2; and $24+1, while the positional encoding p: (for i-th input pair) is shared by 82i-1 and 82;.
Definition 3. Let's denote as e; the piece of information shared by the positions 8; and 8:41 before any transformer layers. Specifically, we define for all E (O....,27, 2D) such that eza 2 for E (0, D) and 62x-1 = p& for l E DJ.
We are interested in the minimum number of layers such that 7z,0 might contain the target token Dp also denoted as e2D.
Lemma 2. After every layer, every residual stream Pt,; will contain a contiguous sequence of pieces of information (c.g., {Ea, Ea+1...., e.)).
Proof. We show this using mathematical induction. The initial residual stream To, contains only the token embedding and the positional encoding, which represent the consecutive pieces of infor- mation fe,, c,+1 During self-attention the existing contiguous sequence of pieces of information in Tt,; will be merged with other contiguous sequences (assumption 2) that share at least one piece of information with rz, (lemma 1). Their union in 1,4 remains contiguous sequence.
Lemma 3. After every layer t, the contiguous sequence of pieces of information in Tt,s will have a length of at most 3' + for all i.
15

---

Published as a conference paper at ICLR 2025
Proof. We can show this using mathematical induction once again. The initial residual streams a contain exactly two pieces of information: the token embedding and the positional encoding. During the t-th layer of self-attention, the contiguous sequence of pieces of information in 72-1, will grow by at most 3 pieces of information to the left and the right, resulting in new total length of at most 3 +
Theorem 1. The embedding vector of the target token cannot be present in the residual stream T1, after layers if (2D)
Proof. The embedding of the target token æp corresponds to the piece of information e2D. For 7,,0 to contain egD. the length of its contiguous sequence must be at least 2D + 1. By Lemma 3, this length will not be reached 3 + 1 < 2D which is equivalent t < 12g)
16

---

Published as a conference paper at ICLR 2025
C
ATTENTION MAPS
C.I TRANSFORMER A
head?
head
Layer 22,
C.2
TRANSFORMER B
heas
Layer hend
Layer 10 head?
japer 12, Mad1
1ayer hest
17

---

Published as a conference paper at ICLR 2025
C.3
TRANSFORMER C
Layer
Laper
Layer 3,
Layer 4.
head
Layer
18

---

Published as a conference paper at ICLR 2025
C.4
TRANSFORMER D
Lager head:
Layer
Layer 3
Layer
hest
head
head
hood
Laver head
Loyer 37 head
Lager 28, hoed
Layar 29, hoad
Laye 23, head
Layer head
19

---

Published as a conference paper at ICLR 2025
D ATTENTION EMERGENCE IN TRANSFORMER D
Lape H.NEEE DINUE tie0
sager Most COvOl volonesh
Liwa vas eevee stowwed
004
62-
04
0.
00-
0.1
0.-
250
0.5
25
Laye 22
Alwosad)
Layer 2i, Pead Crocced (omcerds
Layer Mudd Rosocad juncenat
034
4.54
Layer 34, Ieac Creccedi -ik baccedi
3.44
Tpach
20

---

Published as a conference paper at ICLR 2025
E PROCEDURE FOR DEVERSE-ENGINEERII CIRCUITS
As can be seen in Appendix C the attention maps for each attention head in the transformers contain clear patterns that be be manually identified without the need for any additional tools. For this reason, I decided to manually reverse-engineer the circuits, while validating them thoroughly using ablations to ensure their correctness.
The exact procedure I follow to reverse-engineer the circuits:
1. plot the attention maps for each transformer and each layer for different prompts.
2. By observing the attention maps, identify several possible mechanisms that could explain the attention patterns of each head.
3. For each head, determine which of the hypothesized mechanisms is correct using ablations (described below) and measuring the validation loss. I choose the simplest mechanism that maintains a low validation loss (below 0.05) after ablation.
4. repeat steps 1-3 until the mechanism of each head has been identified.
5. validate the complete mechanism by performing combined ablations on all heads and measuring the validation loss.
6. validate that the uncovered mechanism is not excessive by attempting to further ablate all attention paths individually and measuring the validation loss.
To validate the circuits, I measure the validation error after ablating the attention maps in the fol- lowing manner. For the attention heads that do not perform any useful computation, replace the attention weights with either uniform attention or an identity matrix. For the attention heads that responsible for the information flow, I construct an attention map that is zero everywhere except for the position that expect the head to attend to, where is set one.
After performing the combined ablations (step 5), I find that the mean squared eтror increases slightly, but remains below 0.05 for all transformers. By further ablating any apparently useful attention path (step 6), the mean squared eтror increases to 0.17 0.89. The only exceptions are the first useful layers of each transformer, which always attend to the previous position. After ab- lating their attention uniform, the error stays in the range 0.05 0.1, suggesting that they do not contribute directly to the final output, but rather enable the information flow in the subsequent layers.
21


Reviews:
[{"review_id": "SfvMOBFtLp", "reviewer": "Reviewer_MJk6", "summary": "The paper proposes a novel retrival problem to conduct an empirical but theoretically grounded anaylsis of mechanisms leading to reasoning in transformers. \nExpanding on the induction problem the authors generalize the task to an arbitrary number of inductions steps and introcude a conditional variant. \nThe authors demonstrate that current SOTA LLMs struggle with the retrival problem and their performance approaches random guessing with increased problem complexity. \nExperiments on a minimal problem formulation indicate the importance of implicit curricula and highlight a correlation between the difficulty of the task and the number of layers.\nThrough manual inspection and subsequent experiments the authors identify learned circuits in a subset of attention heads which emerge one head at a time during training.", "questions": "1. Can the authors elaborate on why Assumption 1 should hold for LLMs, as well as how their \n2. From section 4 and Figure 2, I take that all variations of the problem formulations (F1 to F4) include 4 choices, given the random guessing probability of 25%. However,  this does not seem to be the case for F5 \"Relatives\". Is there a specific reason for that?\n3. Do the authors have an intuition of how their results might be generalized to real-world data? It would be valuable to asses if reasoning circuits follow similar emergence patterns during training. However, I assume that the manual identification of circuits is not scalable. \n4. What methods were used to \"reverse engineer\" the attention head circuits?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper makes a strong contribution to the field of mechanistic interpretability and furthering our understanding of the transformer architecture and training behavior. Especially the finding on the sequential emergence of attention heads for reasoning circuits is valuable. \n- The **problem statement** at hand is presented nicely, and the paper follows a logical progression building up to the final insights.\n- The **structure and flow of the experiments** are sensible, starting with higher-level analysis and ending in more focused experiments. \n- Furthermore, the experiments were conducted rigorously.", "weaknesses": "**Readability:** \n- Section 5 \"THEORETICAL ANALYSIS OF INFORMATION FLOW\" is quite **hard to follow** and requires some time to understand, especially with limited prior knowledge. Concrete, examples to what \"E\", \"F\", ... and so forth might mean in the training and evaluation context could help the reader to grasp the theoretical analysis quicker. Augmenting this section with examples, e.g. from  Sec. 4, would make it more accessible to a larger audience. \n\n**Missing clarity:** \n- Large portions of the paper's analysis are **based on a strong assumption**. Key aspects are *Assumption 1* along with simplification of the transformer architecture. The authors should provide further justification for the reasonability of these assumptions and why they would generalize to LLMs. For example, an elaboration on the \"shared token information\" assumption in real-world scenarios would benefit the paper.  \n- The **real-world example from the discussion section could be used earlier** to better motivate the theoretical analysis and problem statement.\n\n**Generalization:**\n- Similarly, the paper would benefit from an analysis on a real-world question-answer dataset, for example. \n\n**Method:**\n- Lastly, the **method of _manually_ reverse engineering** the circuits is only mentioned but **not described in detail**. This severely affects the potential replicability of the paper and makes independent assessments of the methodology challenging.\n\n\n**Post-rebuttal edit**\nThe rebuttal addresses the main weaknesses of the paper. I adjusted my score accordingly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FPfkPgSabk", "reviewer": "Reviewer_cnTJ", "summary": "This paper presents a retrieval task to illustrate the source of the reasoning power of hierarchical transformer models. With the retrieval task, the author takes various structures to verify the connection between successful learning and the presence of an implicit curriculum. In my opinion, this work is helpful in understanding the ability of large language models. As descributed in this paper,  a transformer cannot solve the retrieval problem with a specific difficulty unless it has the minimum number of necessary layers.", "questions": "Can you describe the definition of the implicit curriculum in detail? In your experiments, I can see the connection between the number of layers and the model's performance, as shown in Figure 4.\n\nThe emergent ability of LLM is more depend on depth or width?  This paper mainly analyzes the effect of model depth on emergence ability.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This is a good joy for understanding the ability of LLM, especially the emergent ability of models.\n\n\n1.  a novel idea to study the ability of the reasoning ability of LLM.\n\n2. The finding is exciting and fits with human intuition. \n\n3. The visualization of attention can give describution on how LLM takes retrieval reasoning tasks.", "weaknesses": "1. Can you provide a more complex example? \n\n2. In my opinion, I hope I can see a general framework that can unify more tasks with your retrieval task.\n\n3.  There are a lot of chapters in the article, and I can't quite understand the relationship between different chapters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SWSpHeqKxv", "reviewer": "Reviewer_yct3", "summary": "This paper introduces the retrieval problem, a fundamental task that challenges transformers to retrieve information from multiple positions within an input sequence. The authors demonstrate that solving the retrieval problem requires a certain depth of transformer layers.\n This paper explores how large language models and transformers handle retrieval tasks. They introduce two tasks: the retrieval problem and a variant called the conditional retrieval problem. It shows that large language models can solve retrieval tasks without fine-tuning by leveraging complex mechanisms involving multiple attention heads, which emerge in a specific sequence when trained with an implicit curriculum.", "questions": "1. Assumption 1, which requires shared positional or token information for attention between positions, is overly restrictive and conflicts with the flexibility of Transformer models. Transformers are designed to allow any position to attend to any other, enabling them to learn complex relationships without pre-shared information. This assumption limits the model's ability to capture long-range dependencies and generalize to tasks where connections are context-driven rather than based on shared information. I recommend reconsidering this assumption to better align with the strengths of the Transformer architecture.\n2. Theorem 1  would benefit from experimental validation. Testing whether the target embedding truly appears in the residual stream only when \\( t \\geq \\log_3(2D) \\) would strengthen the claim and confirm its practical relevance. I recommend adding experiments to verify this behavior across different settings.\n3. The specific details of the implicit curriculum formulation remain vague, and the reasons for its superior performance are not clearly explained. Additionally, it would be helpful to clarify why the non-IC formulation, which aligns more closely with traditional problem setups, fails to perform well.\n4. In Section 8,  \"manually reverse-engineering the circuits learned by the transformers\" , the specific details of this process are not fully explained. Providing more information on the methodology used to reverse-engineer these circuits, as well as any criteria or steps involved, would enhance the clarity and reproducibility of this analysis.\n5. The experimental results need to be validated under more complex settings, such as with longer retrieval steps.\nWe will increase the score based on the answer to the question.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "● The paper provides insights into how LLMs perform retrieval using attention heads.\n● Introducing these tasks gives a clear way to study transformers' retrieval abilities.\n● The study highlights the role of learning curriculum in the development of retrieval mechanisms.", "weaknesses": "1. **Lack of Experimental Validation for Theoretical Claims**: Theoretical claims like Theorem 1 lack empirical support, making it hard to verify their practical impact.\n2. **Insufficient Details on Experimental Setup**:: The paper lacks detailed explanations for key experimental settings. The implicit curriculum (IC) formulation, which performs better than non-IC, is not clearly defined, and Section 8’s description of manually reverse-engineering circuits lacks detail, making the experiments difficult to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)