Understanding Addition in Transformers
Authors: Philip Quirke
Fazl Barez
Venue: ICLR 2024 poster (2024)
Keywords: Interpretability
Transformers
Primary area: visualization or interpretation of learned representations

Abstract: Understanding the inner workings of machine learning models like Transformers is vital for their safe and ethical use. This paper provides a comprehensive analysis of a one-layer Transformer model trained to perform n-digit integer addition. Our findings suggests that the model dissects the task into parallel streams dedicated to individual digits, employing varied algorithms tailored to different positions within the digits. Furthermore, we identify a rare scenario characterized by high loss, which we explain. By thoroughly elucidating the model’s algorithm, we provide new insights into its functioning. These findings are validated through rigorous testing and mathematical modeling, thereby contributing to the broader fields of model understanding and interpretability. Our approach opens the door for analyzing more complex tasks and multi-layer Transformer models.

Published as a conference paper at ICLR 2024
UNDERSTANDING ADDITION IN TRANSFORMERS
Philip Quirke Apart Research
Fazl Barez Apart Research
University of Oxford
ABSTRACT
Understanding the inner workings of machine learning models like Transformers is vital for their safe and ethical use. This paper provides comprehensive analysis of a one-layer Transformer model trained to perform n-digit integer addition. Our findings suggest that the model dissects the task into parallel streams dedicated to individual digits, employing varied algorithms tailored to different positions within the digits. Furthermore, we identify a rare scenario characterized by high loss, which we explain. By thoroughly elucidating the model's algorithm, we provide new insights into its functioning. These findings are validated through rigorous testing and mathematical modeling, thereby contributing to the broader fields of model understanding and interpretability. Our approach opens the door for analyzing more complex tasks and multi-layer Transformer models.
INTRODUCTION
Understanding the underlying mechanisms of machine learning models is essential for ensuring their safety and reliability (Barez et al., 2023; Olah et al., 2020b; Doshi Velez. and Kim, 2017; Hendrycks and Mazcika, 2022). By unraveling the inner workings of these models, we can better understand their strengths, limitations, and potential failure modes, enabling us to develop more robust and trustworthy systems. Specifically, the sub-field of mechanistic interpretability within machine learning interpretability aims to dissect the behavior of individual neurons and their interconnections in neural networks (Räuker et al., 2022). Recent interpretability work has explored how transformers make predictions (Neo al., 2024), analyzed reward model divergence in large language models (Marks et al., 2024), and highlighted the importance of such analyses for measuring value alignment (Barez and Torr, 2023). This pursuit is part of a larger endeavor to make the decision-making processes of complex machine learning models transparent and understandable.
Although models like Transformers have shown remarkable performance on a myriad of tasks, their complexity makes them challenging to interpret. Their multi-layere architecture and numerous parameters make difficult to comprehend how they derive specific outputs (Vig, 2019). Further, while simple arithmetic tasks like integer addition may be trivial for humans, understanding how a machine learning model like a Transformer performs such an operation is far from straightforward (Liu and Low, 2023).
In this work, we offer an in-depth analysis of a one-layer Transformer model performing n-digit integer addition. We show that the model separates the addition task into independent digit-specific streams of work, which are computed in parallel. Different algorithms are employed for predicting the first, middle, and last digits of the answer. The model's behavior is influenced by the compact nature of the task and the specific format in which the question is presented. Despite having the opportunity to begin calculations early, the model actually starts later. The calculations are performed in time-dense manner, enabling the model to add two 5-digit numbers to produce a 6-digit answer in just steps (See Fig. 1). rare use case with high loss was predicted by analysis and proved to exist via experimentation. Our findings shed light on understanding and interpreting transformers. These insights may also have implications for AI safety and alignment.
Our results demonstrate the transformer's unique approach applies to integer addition across various digit lengths (see Appendixes B and C). This transformer architecture, with its self- mecha- nism and ability to capture long-range dependencies, offers powerful and flexible framework for
Corresponding author; fazl@robots. ac.ac. uk.

---

Published as a conference paper at ICLR 2024
modeling sequential data. Our theoretical framework provides mathematical justification for the model's behavior, substantiating our empirical observations and offering a foundation for future work in this domain.
Our main contributions are four-fold:
Reformulation of the traditional mathematical rules of addition into framework more applicable to Transformers.
Detailed explanation of the model's (low loss) implementation of the addition algorithm, including the problem and model constraints that informed the algorithm design.
Identification of a rare use case where the model not safe use (has high loss), and with targeted experimentation, leading to deep understanding of the model.
explanation of the root cause.
Demonstration of a successful approach to elucidating a model algorithm via rigorous analy- sis from first principles, detailed investigation of model training and prediction behaviours,
Below, we provide an overview related work ($3), discuss our methodology ($-4,, describe our mathematical framework ($5), our analysis of model training ($6) and model predictions ($7). We conclude with a summary of our findings and directions for future research ($9).
A
D4
D3
D2
D1
DO
D4'
D'3
D22
DM1
DO
A5
A4
A3
A2
A1
AO
B
1st Head: 10,0''
D1,0'1
2nd Head:
04.04
D3,0'3
02,0'2
D0,0'0
3rd Head:
04,04
D4,04
D3,0'3 D2.22
D1,D1 D0.00
MLP:
MLP
MLP
MLP
MUP
MLP
MLP
D Head:
Carry
2nd Head:
Sums
Carry
3rd Head:
Addition
MLP:
Trigram
A3
Figure 1: Illustration of the transformer model's attention pattern when adding two 5-digit integers. The model attends to digit pairs sequentially from left to right, resulting in "double staircase" pattern
rows. A: The digit question is revealed token by token. The " "10s of thousands" digit is revealed first. B: From the "=" token, the model attention heads focus on successive pairs of digits, giving a "double staircase" attention pattern. C: The heads are time-offset from each other by token such that, in each row, data from tokens is available. D: To calculate A3, the 3 heads do independent simple calculations on D3, D2 and D1. The results are combined by the ML.P layer using trigrams. A3 is calculated one token before it is needed. This approach applies to all answer digits, with the first and last digits using slight variations the approach.
BACKGROUND
We focus on a single-layer transformer model with a vocabulary size V containing a set of symbols V. The model converts input (e.g., sequence of symbols from V) into input sequence (X)..... xx)), where each x; E RV is a one-hot vector representing the corresponding symbol from V. The input tokens are mapped to di-dimensional embeddings by multiplying with embedding
2

---

Published as a conference paper at ICLR 2024
matrix E R4xV,, where the i-th column of represents the embedding of the i-th symbol in V.
The resulting sequence of embeddings is denoted as (ei...., Cp). where e; Ex; E Rd The model processes the imput embeddings using mechanism called "self-attention" . Each input embedding e, is passed through a self-attention mechanism that calculates weighted relationships between all input embeddings capturing the importance of each embedding relative others. The model then aggregates these weighted representations to produce contextually enriched representations for each embedding. The contextually enriched representations produced by the self-attention mechanism subsequently fed through feedforward neural networks (i.e., multilayer perceptrons, MLPs) to refine their information. Finally, the output tokens are generated based on the refined representations and converted back to human-readable format using the vocabulary V.
RELATED WORK
Interpreting and reverse engineering neural networks and transformers to find meaningful circuits has been an area of active research. Olah et al. (2020a) argued that by studying the connections between neurons and their weights, can find meaningful algorithms (aka Circuits) in "vision" neural network. Elhage et al. (2021) extended this approach to transformers, conceptualizing their operation in a mathematical framework that allows significant understanding of how transformer operate internally. Various tools (Foote et al., 2023; Conmy et al., 2023b; Garde et al., 2023) use this framework to semi-automate some aspects of reverse engineering. Nanda et (2023) reverse- engineered modular addition (e.g. 5 + 7 mod 10 = 2) showing the model used discrete Fourier transforms and trigonometric identities to convert modular addition to rotation about a circle.
Nanda and Lieberum (2022) have argued models comprise multiple circuits. They gave examples, including the distinct training loss curve per answer digit in 5-digit integer addition, but did not identify the underlying circuits. This work investigates and explains the circuits in n-digit integer addition.
agro
Loss
Ayt4
5go5
wosaeee 200
200
700
E00
Epoch
Figure 2: For 5-digit integer addition, these per- training loss curves show the model trains each answer digit semi-independentlyy Answer digit 0 (the "units" digit) is learnt much more quickly than other digits.
Circuit analysis can extract graphical circuit representations and analyze component interactions (Bau et al., 2017). To enable analysis and interpretability, techniques works like Petersen al. (2021) symbolically reverse-engines networks by recovering computational expressions. Research including Seth (2005) advocate analyzing networks causally, introducing structural causal models to infer mechanisms. Examinations of sequence models like Petersen et al. (2021) have analyzed the emergence and interaction of modular components during training. Lan and Barez (2024) analyzed and compared shared circuit subgraphs for related sequence continuation tasks in large language model. Evolutionary perspectives such as Miikkulainen (2021) elucidate how selection pressures shape hierarchical representations. Lo et al. (2024) show how such representations, even if pruned, can quickly re-emerge in models after little re-training.
Information bottleneck analyses including Kawaguchi al. (2023) relate bottlenecks to abstractions and modularization arising from forced compression. Surveys like Carbonneau et al. (2022) overview techniques to disentangle explanatory factors into separate latent dimensions. Novel objectives
3

---

Published as a conference paper at ICLR 2024
proposed in works like Conmy et al. (2023a) improve interpretability by encouraging modularity and disentanglement.
4 METHODOLOGY
successful integer addition model must cope with a very large question and answer space. For digit addition, there are 10 billion distinct questions (e.g. "54321+77779=") and 200,000 possible answers, Just one token (the "=") after seeing the complete question, the model must predict the first answer digit. It must correctly predict 6 answers digits.
3 3
3
5
7 DO
2 2
4 3
5 6 0 0
D'3
D'O
+
D'
A
Figure 3: We refer to individual tokens in a 5-digit addition question as D4, DO, and D'4, ... D'O and the answer tokens as A5, A0.
Our model was trained on 1.8 million out of 10 billion questions. After training, the model predicts answers to questions with low loss, showing the model does not rely on memorisation of training data. Fig. 2 shows the model trains each digit semi-independently suggesting the model performs integer addition by breaking down the task into parallel digit-specific streams of computation.
The Transformer model algorithms often differ significantly from our initial expectations. The easiest addition process for humans, given the digits can be processed in any order, is to add the unit digits first before moving from left to right, so the model predicts higher value digits (e.g. thousands) of the answer before the lower value digits (e.g. units). It can't 't use the human process.
to higher value digits. This autoregressive transformer model processes text
key component of addition is the need to sum each digit in the first number with the corresponding digit in the second number. Transformer models contain "attention heads" ,the only computational sub-component of model that can move information belween positions (aka digits or tokens). Visualising which token(s) each attention head focused on in each row of the calculation provided insights. While our model works with 2, 3 or 4 attention heads, attention heads give the most easily interpreted attention patterns. Fig. 4 shows the attention pattern for single 5 digit addition calculation using 3 attention heads. Appendix C shows the same pattern for 10 and 15 digit addition. Appendix C shows the pattern with 2 or 4 attention heads.
While it's clear the model calculating answer digits from highest value to lowest value, using the attention heads, it's not clear what calculation each attention head is doing, how the attention heads are composed together to perform addition.
5 MATHEMATICAL FRAMEWORK
To help investigate, we created a mathematical framework describing what any algorithm must do if it is perform addition correctly. Our intuition is that the model a) incrementally discovers a necessary and sufficient set of addition sub-tasks (minimising complexity), b) discovers these sub-tasks eemi-independently (maximising parallelism), and c) treats each digit semi-independently (more parallelism). Our framework reflects this.
To explain the framework, Let D = (dn-1,...,dy di, do) and = = =d'_,,,,,.... d', d)) be two n- digitof digits. We assert that the framework utilizes three base functions that operate individual digit pairs. The first is Base Add ( aka BA ), which calculates the sum two digits D; and D modulo 10, ignoring any carry over from previous columns. The second is Make Carry 1 ( aka MC1 ), which evaluates if adding digits D, and D/ results in a carry over of to the next column. The third is Make Sum 9 aka MS9 which evaluates if D, + D 9 exactly.
In addition. the framework uses two compound functions that chain operations across digits. The first Use Carry 1 ( aka UC1 ), which takes the previous column's carry output and adds it to the sum of
4

---

Published as a conference paper at ICLR 2024
10000
Head 0
Head
Head 2
Heads 0, 1 & 2 combined
The attention pattern, for model with attention heads, performing a single digit addition. The pattern is 18 by 18 squares (as 54321+7777==121000 is 18 tokens). Time proceeds vertically downwards, with one additional token being revealed horizontally at each row, giving the overall triangle shape. After the question is fully revealed (at 11), each head starts attending pairs of question digits from left to right (i.e. high-value digits before lower-value digits) giving the "double staircase" shape. The three heads attend to given digit pair in three different rows, giving time ordering of heads.
Figure 4:
the current digit pair. The second is Use Sum 9 (aka US9 which propagates (aka cascades) carry over to the next column the current column sums to and the previous column generated earry over. US9 is the most complex task as it spans three digits. For some questions (e.g. 00555 + 00445 = 01000) US9 applies to up to four sequential digits, causing a chain effect, with the MC1 cascading through multiple digits. This cascade requires time ordering of the US9 calculations from lower to higher digits.
These tasks occur in the training data with different, predictable frequencies (e.g. BA common, US9 is rarer). Compound tasks are reliant on the base tasks and so discovered later in training. The discovery of each task reduces the model loss by a different, predictable amount (e.g. BA by 50%, US9 by 5%). Combining these facts give an expected order of task discovery during training as shown in Fig. We use this mathematical framework solely for analysis gain insights. The model training and all loss calculations are completely independent of this mathematical framework.
TRAINING ANALYSIS
Fig. 2 shows the model trains each digit semi-independentlyy Armed with the mathematical framework. we investigated each digit separately. The Digit 0 calculation is the least interesting as it only uses BA (not UC1 or US9 ). Once discovered, Digit always quickly refines to have the lowest loss and least noise (as expected). (Graphs in Appendix B.)
For the other digits, we categorised the training data into non-overlapping subsets aligned to the BA, UC1 and US9 tasks, and graphed various combinations, finding interesting results. The US9 graphs are much noisier than other graphs (Fig. 6). We found that the model has low loss on simple US9 cases (e.g. + = 100) but has high loss on US9 cascades (e.g. 445 555 = 1000) where the MC1 must be propagated "right to left" two 3 or 4 columns. The model can't perform these rare cases safely, as it has a "Ieft to right" algorithm.
Graphing the BA and UC1 use cases side by side for any one of the Digits 1, 2 and 3 shows an interesting pattern (Fig. 7). In Phase 1, both tasks have the same (high) loss. In Phase 2, both curves drop quickly but the BA curve drops faster than the UC1 curve. This "tim lag" matches our expectation that the BA task must be accurate before the UC1 task can be accurate. In Phase 3, both tasks' loss curve decrease slowly over time.
5

---

Published as a conference paper at ICLR 2024
Digit
4
50%
95%
95%
100%
50%
95%
3
50%
95%
100%
50%
50%
95%
2
95%
100%
95%
100%
1
95%
95%
100%
100%
100%
0
100%
100%
100%
100%
100%
100%
Training Epoch
MC1
BA
UC1
US9
MS9
Figure 5: The mathematical framework (our method) predicts that during training, tasks are learnt for each digit independently, progressively increasing per digit accuracy (i.e. decreasing loss) shown as percentages. Mathematical rules cause dependencies between digits, giving an predicted ordering for perfect (i.e. zero loss) addition. The chain of blue squares relate questions like 99999 00001 100000 where the MC1 in digit causes US9 cascades through multiple other digits.
5g00
5go-
100
200
300
400
800
100
000
500
Epoch
Figure 6: High variability in the per digit training loss for US9 cases caused by the model's inability to reliably do cascading US9 cases such as 445+555=1000.
Both the and tasks need to move data between tokens, and so will implemented in attention head(s). Fig. 7 shows they are trained semi-independentyy We choose the number of attention heads our model with the clearest separation tasks the attention pattern. We find (later) that our model has separate attention heads for the BA and UC1 tasks. Digit 4, the highest question digit, has significantly different loss curve (shown in Fig. 8) than Digits 1, 2 and 3. This partially explained by Digit 4 only having simple differences. This difference persists with different seed values, and with or 15 digit addition. We explain this difference later.
(i.e. US9 cascades). This does not explain the BA UC1

---

Published as a conference paper at ICLR 2024
Phase
Phase 2
Phase
Loss
www
50
102 250 200
230
200
200
420
430
Epoch
Figure Training loss for digit 3 showing that, in Phase 2, the refining of Use Carry 1 lags behind Base Add. Base Add and Use Carry 1 refined separately and have separate calculation algorithms. The phases seem to correspond to "memorisation", "algorithm discovery" and "clean- up".
US01
US9
Loss
Epoch
Figure 8: Training loss for digit 4 starts and stays lower for all tasks than it does for digits 1, 2 and 3. Digit has a different calculation algorithm from digits 1, and 3.
PREDICTION ANALYSIS
Using the ablation interventions technique we overrode (mean ablated) the model memory (residual stream) at each row (aka token position) and confirmed that the addition algorithm does not use any data generated in rows to 10 inclusive. In these rows the model has not yet seen the full question and every digit in the question independent of every other digit, making accurate infeasible. The model also does not use the last (17th) row. Therefore, the addition answer calculation started and completed in rows (11 16). Further ablation experiments confirmed that the A0 A4 answers are calculated one row before being revealed. (Details in Appendix H.)
prediction
The model has slightly different algorithms for the first answer digits (A5 and A4), the middle answer digits (A3 and A2) and the last answer digits (Al and A0). Fig. has simplified version of how the model calculates the answer digit A3. Fig. has more details. For digit addition, there 2 middle answer digits (A3 and A2) whereas for 15 digit addition there are 12 middle answer digits that use this algorithm.
The A3 addition algorithm has three clauses related to digits 3, 2 and 1. Ablating each head in turn shows that the 3rd head has most impact on loss, the 2nd head has less impact, and the 1st head has little impact. This aligns with the intuition that the sum "D3 D'3" matters most, the MC1 from the previous digit (D2 D'2) matters less, and the rare MC1 from the previous previous digit (D1 + D'1) matters least. The last two answer digits, Al and A0, use simplified a version of the A3 algorithm, as some clauses are not necessary.
The A3 algorithm can also be applied to A4. But the Digit 4 training curve is better (faster) than the middle digits. The attention patterns show that for A4, the model using the heads in row 11 (the "=" input token) when the A3 algorithm doesn't 't require this. Uniquely, A4 utilises more "compute" than is available to A3, A2, Al or AO. We assume the model uses this advantage to implement faster-training and lower-loss algorithm for A5 and A4. We haven't t worked the details of this.
7

---

Published as a conference paper at ICLR 2024
D'2
A2
Head:
DD3.D3
D2.02
D1.D'M
DO,D'0
2nd Head:
04.0%
D3.03
02.0'2
D1.01
DO.0'0
Head:
D4.0%
D0.0'0
MLP:
MLP
MLP
MLP
MLP
MLP
MLP
1st Head:
MC1
A A3 =
2nd Head:
MC1,
MC1.
MC1,
D3 + D'3 Mod 10
MS9
MS9
MS9
MS9
MS9
if Digit2.MC1 is true
3rd Head:
0
1 if Digit2.MS9 and Digit1.MC MC1 true
E
MLP:
Bigram Trigram Trigram Trigram Bigram Unigram
Trigram
Mod 10
Figure 9: A: To predict answer digit A3, the addition algorithm must combine information from digits 3, 2 and 1. B: The 1st head calculates MC1 on digit 1. C: The 2nd head calculates MC1 and MS9 (at most one of which can be true) on digit D: The 3rd head calculates Base Add on digit 3. E: The MLP layer uses trigrams to combine the information from the 3 heads to give the final answer A3, one row before it output. Appendix G shows this algorithm as pseudocode.
Mean ablating the 1st or 2nd head slightly increased the average loss for BA questions from 0.05 to 0.08, whereas ablating the 3rd head substantially increased the loss 3.7, confirming that the 3rd head is doing the BA task. (Details in Appendix H.)
The MLP can be thought of as "key-value pair" memory (Meng al., 2022; Geva et al., 2021) that can hold many bigrams and trigrams. We claim our MLP pulls together the two-state 1st head result, the tri-state 2nd head result and the ten-state 3rd head result value, treating them as a trigram with 60 (2 3 10) possible keys. For each digit, the MLP has memorised the mapping of these 60 keys to the 60 correct digit answers (0 to 9). We haven't 't proven this experimentally. Our MLP sufficiently large to store this many mappings with zero interference between mappings (Elhage al., 2022).
Despite being feasible, the model does not calculate task MC1 rows 7 11. Instead, it completes each answer digit calculation in row, possibly because there are training optimisation benefits in generating a "compact" algorithm.
This algorithm explains all the observed prediction behaviour innlldding the fact that the model can calculate a simple US9 case but not a cascading US9 case. We assume that, given the dense nature of the question and answer, and the small model size, the model does not have sufficient time and compute resources to implement both UC1 and US9 accurately, and so preferences implementing the more common (C11) ) case, and only partially implements the more complex and rare US9 case.
ALGORITHM REUSE
We explored whether the above algorithm is learned by similar models. We trained separate 1-layer model for 5-digit addition with a different random seed and optimization algorithm. The new model's answer format requires it to predict "+" the first answer token, e.g., "12345+6780==4000" Despite these changes, Figures/ 24, 25, and 26 show that the new model's 's behavior has many similarities to the previous model.
Intervention ablation demonstrates that the new model uses the BaseAdd, MakeCarr,, and MakeSum sub-tasks in the same way the previous model. The new model exhibits the same
8

---

Published as a conference paper at ICLR 2024
strengths and weaknesses, e.g., it can calculate simple MS case but not a cascading MS case. We claim that the new and previous models use essentially the same algorithm, with minor variations.
Our analysis suggests that the transformer architecture, when trained the addition task, converges to a consistent algorithm for performing digit-wise addition. This algorithm leverages the self-attention mechanism to discover and execute the necessary sub-tasks in parallel and semi-independent manner. Despite differences in random initialization, optimization algorithms, and output formatting, the models exhibit similar internal behavior and capabilities, indicating a robust algorithmic solution emerges from the transformer's architecture and training process.
CONCLUSIONS
This work demonstrates a successful approach to reverse engineering and elucidating the emergent algorithm within transformer model trained on integer addition. By combining mathematical analysis, empirical investigation of training and prediction, and targeted experimentation, we are able to explain how the model divides the task into parallel digit-specific streams, employs distinct subroutines for different digit positions, postpones calculations until the last possible moment yet executes them rapidly, and struggles with a specific rare case.
Our theoretical framework of necessary addition subtasks provides a foundation for the model's s behavior. The digit-wise training loss curves reveal independent refinement consistent with separate digit-specific circuits. Attention patterns illustrate staging and time-ordering of operations. Controlled ablation experiments validate our hypothesis about algorithmic elements' roles. Together these methods enable a detailed accounting of the model's addition procedure.
This methodology for mechanistic interpretabilitty, when applied to broader tasks and larger models, can offer insights into not just what computations occur inside complex neural networks, but how and why those computations arise. Such elucidation will be increasingly important for ensuring the safety, reliability and transparency of Al systems.
10 LIMITATIONS AND FUTURE WORK
One concrete limitation of our current model is its difficulty handling the rare of adding two 9-digit numbers that sum to billion (c.g. 1999999,999 + 1). Further investigation is needed to understand why this specific edge poses challenge and to develop strategies improve the model's 's performance, such as targeted data augmentation or architectural modifications.
Regarding the MLP component, detailed ablation study could elucidate its precise role and contribu- tions. Systematically removing or retraining this component while monitoring performance changes could shed light on whether it is essential for the overall algorithm plays future work, natural next step is to apply our framework to reverse-engineer integer subtraction models. Subtraction shares some commonalities with addition but also introduces new complexes like borrowing. Extending our approach to handle such nuances would demonstrate its generalisability.
auxiliary part. For
For multiplication, interesting avenue first pre-train large transformer model on addition using our verified modules as starting component. Then, expand the model's 's capacity and fine-tune on multiplication data. This could facilitate more rapid acquisition of the multiplication algorithm by providing strong inductive bias grounded in robust addition skills. Specific experiments could then identify the emergence of new multiplication-speeii subroutines and their integration with the addition circuits.
Furthermore, domains like symbolic AI, program synthesis, or general reasoning may benefit from embedding multiple specialized algorithmic components like our addition circuit within larger language models. Insights from our work could guide the controlled emergence and combination of diverse task-specific capabilities. Overall, while making models more interpretable is valuable, the ultimate aim is developing safer, more reliable, and more controllable Al systems. Our work highlight one path toward that goal through understanding of addition in neural network computations.

---

Published as a conference paper at ICLR 2024
11 REPRODUCIBILITY STATEMENT
To facilitate the reproduction of our empirical results on understanding and interpreting addition in one-layer transformers, and further studying the properties of more complext transformers on more complex tasks that would build on single layer, we release all our code and resources used in this work. Furthermore, we offer explicit constructions of one layer transformers used in this paper.
12 ACKNOWLEDGEMENTS
We are thankful Jason Hoelscher-Ohermaier for his comments on the earlier draft, Esben Kran for organising the Interpretability Hackathon and Clement Neo for his comments on the paper and assistance with some of the figures. This project was supported by Apart Lab.
REFERENCES
Barez and Torr. Measuring value alignment, 2023.
F. Barez, H. Hasanbieg, and A. Abbate. System ii:: Learning with domain knowledge for safety constraints, 2023.
D. Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying inter- pretability of deep visual representations. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3319-3327, 2017.
M.-A. Carbonneau, J. Zaidi, J. Boilard, and G. Gagnon. Measuring disentanglement: A review of metrics, 2022.
A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 04 2023a.
A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonoo. Towards automated circuit discovery for mechanistic interpretability, 2023b.
Doshi-Veler and B. Kim. Towards rigorous science of interpretable machine learning, 2017.
N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernande7, A. Jones, Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. http:://tranffomeerccrcuuuii pub/2021/fraa httt///wwwwo///////mmo//// iin//x.dxxxx/ html, 2021.
N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hattield-Dodds, R. Lasenby, D. Drain, C. Chen, R. Grosse, S. McCandlish, J. Kaplan, D. Amodei, M. Wattenberg, and C. Olah. Toy models of superposition, 2022.
A. Foote, N. Nanda, Kran, Konstas, S. Cohen, and Barez. Neuron to graph: Interpreting language model neurons at scale, 2023.
A. Garde, F. Kran, and F. Barez. Deepdecipher: Accessing and investigating neuron activation in large language models, 2023.
M. Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. 00.18653/v1/2011 10.18655//1/2021..mmml http://22...mmmmmmamm....... URL
https://aclanttollog..orr org/ http:/22222eeeemmaa1a1.mmm emnlp-main.4e 446.
D. Hendrycks and M. Mazeika. X-risk analysis for ai research, 2022.
K. Kawaguchi, Z. Deng, X. Ji, and Huang. How does information bottleneck help deep learning?, 2023.
10

---

K. Meng, D. Bau, A. Andonian, and Belinkov. Locating and editing factual as- sociations in gpt. https::/proceedd/s neurips. wwt.c/ape_.illls/paeeeeeeeeeeeee B Published as a conference paper at ICLR 2024
M. Lan and F. Barez. Interpreting shared circuits for ordered sequence prediction in large language model, 2024.
Liu and B. K. Low. Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks, 2023.
M. Lo, S. B. Cohen, and F. Barez. Large language models relearn removed concepts, 2024.
L. Marks, A. Abdullah, C. Neo, R. Arike, P. Torr, and F. Barez. Beyond training objectives: Interpreting reward model divergence in large language models, 2024.
(0lle/paper/2022/F1L pafereeeeeeee............ 2022.
R. Mikkulainen. Creative ai through evolutionary computation: Principles and examples. https: 2021.
//doi.
N. Nanda and T. Lieberum. Mechanistic interpretability analysis of grokking, https://www. alignmentforum www.rrrrw.....oo////oooree/e /ttpssst/// 2022.
N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023.
C. Neo, S. B. Cohen, and F. Barez. Interpreting context look-ups in transformers: Investigating attention-mlp interactions, 2024.
Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits.
2020a.
C. Olah, N. Cammarata, Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 50)xe00002.001,, 2020b.
A. Petersen, P. Voigtlaender, and A. Krause. Deep symbolic regression: Recovering mathematical
expressions from data via risk-seeking policy gradients. Advances in Neural Information Processing Systems, 34, 2021.
T. Räuker, A. Ho, S. Casper, and D. Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. arXiv preprint 1rXiv:2207.13243, 2022.
A. K. Seth. Causal connectivity of evolved neural networks during behavior. Network: Computation in Neural Systems, 16(1 16(1):35-54, 2005. doi: 10. 10.108000554990500233376 URL. https://doo. org/10. 1888/89558898550233777..
J. Vig. A multiscale visualization of attention in the transformer model. arx14:1906.05714. 2019.
arXiv preprint
A APPENDIX NUMBER OF ATTENTION HEADS
The model can be successfully trained with 2, 3 or 4 attention heads (Refer Figs. 10, 11, 12). However described in Fig. the attention heads case is easier interpret than the 2 or 4 heads cases.

---

Published as a conference paper at ICLR 2024
Head
Head
Heads 0 & 1 combined
Figure 10: For digit addition, using 2 attention heads works, but the model attends to multiple token pairs in single head, suggesting that multiple tasks being packed into single head, which makes it harder to interpret.
Head 0
Head 1
Head
Heads & combined
Figure 11: For 5 digit addition, using attention heads gives the best separation with the heads having distinct, non-overlapping attention pattern. Ablating any head increases the model loss, showing all heads are useful.
Head 0
Head
Head 2
Head 3
Heads 0, 1, 2 3 combined
Figure 12: For 5 digit addition, using 4 attention heads gives an attention pattern where the H1 and
H2 staircases overlap perfectly. Ablating one HI H2 increases loss. The similarity in HI and H2's attention patterns suggests they are "splitting" single logical task. Spliting is feasible as Elhage al. (2021) says attention heads are independent and additive.
12

---

Published as a conference paper at ICLR 2024
B APPENDIX TRAINING LOSS BY NUMBER OF DIGITS
The model can be successfully trained with 5, 10, 15, etc digits (Refer Figs. 13, 14, 15). For a given loss threshold, 15 digit addition takes longer to train than 10 digit addition, which takes longer to train than 5 digit addition.
Epoch
Figure 13: The per-digit loss curves over 1500 training epochs for 5 digit addition.
0pe2
0gg2
0gg3
Loss
0p4
M
000 30
1200
Epoch
Figure 14: The per-digit loss curves over 1500 training epochs for 10 digit addition.
0082
1000
Epoch
Figure 15: The per-digit loss curves over 1500 training epochs for digit addition.
C APPENDIX ATTENTION PATTERNS BY NUMBER OF DIGITS
The model can be successfully trained with 5, 10, 15, etc digits (Refer Figs. 16, 17, 18). The attention patterns for these cases show similarities that aid interpretation.
13

---

Published as a conference paper at ICLR 2024
Hcad 0
Head 1
Hcad
Heads & combined
Figure 16: For 5 digit addition, and 3 attention heads, the attention pattern has a strong double- staircase shape. Each step of the staircase is blocks wide, showing the attention heads attending to different input tokens in each row.
Head
Head
Head
Heads 0, 1 & 2 combined
Figure 17: For 10 digit addition, and 3 attention heads, the attention pattern has a strong double-
staircase shape. Each step of the staircase is blocks wide, duplicating the Fig. 16 pattern.
Head 0
Head
Head
Hcads 0, 2 combined
Figure 18: For 15 digit addition, and 3 attention heads, the attention pattern has a strong double-
staircase shape. Each step of the staircase is blocks wide, duplicating the Fig. 16 pattern.
D APPENDIX MODEL CONFIGURATION
Colab notebook was used for experimentation:
It runs T4 GPU with each experiment taking few minutes run.
14

---

Published as a conference paper at ICLR 2024
The key parameters (which can all be altered) are:
1. layers 1; This is one layer Transformer
heads 3; There are attention heads
n digits = 5; Number of digits in the addition question
It uses a new batch of data each training step (aka Infinite Training Data) to minimise memorisation.
During training run the model processes about 1.5 million training datums. For the 5 digit addition problem there are 100,000 squared (that 10 billion) possible questions. So, the training data is much less than 1% of the possible questions.
US9 cascades (e.g. 44445+55555100000, 54321145579 1234+8769=10003) are exceedingly rare. To speed up training, the data generator was enhanced inerease the frequency of these cases in the training data.
44450+55550=1000,
E APPENDIX -SOLVING A JIGSAW USING MANY PEOPLE
Understanding transformer algorithm implementation requires new ways of thinking.
Here is useful analogy for changing how approach problem more closely resemble how a Transformer would; a jigsaw puzzle. A single person could solve it using a combination of meta knowledge of the problem (placing edge pieces first), categorisation of resources (putting like-coloured pieces into piles), and an understanding of the expected outcome (looking at the picture on the box).
But if instead we had one person for each piece in the puzzle, who only knew their piece, and could only place it once another piece that it fit had already been placed, but couldn't talk the other people, and did not know the expected overall picture, the strategy for solving the jigsaw changes dramatically.
When they start solving the jigsaw, the 4 people holding corner pieces place them. Then 8 people holding corner-adjacent edge pieces can place them. The process continues, until the last piece is placed near the middle of the jigsaw.
We posit that this approach parallels how transformer models work. There is no pre-agreed overall strategy or communication co-ordination between people (circuits) just some "rules of the game" to obey. The people think independently and take actions in parallel. The tasks are implicitly time ordered by the game rules.
F APPENDIX USE SUM 9 TRAINING Loss GRAPH
Fig. 19 shows training loss curve for just BA and UC1 tasks. The model has high loss Use Sum cascades shown as high variability in the loss graph (See Fig. 20).
Phase
Phase 2
Phase
Mww
UCT
Loss
2.5
0.5
Epoch
Figure 19: Training loss for digit 3 showing that, lags Base Add (blue line). This supports the claim that Base Add and Use Carry 1 are learnt and refined separately and perform different calculation tasks.
in Phase the refining of Use Carry 1 (red line)
15

---

Published as a conference paper at ICLR 2024
1190
Loss
200
200
200
400
Epoch
Figure 20: Training loss for digit for the Base Add, Use Carry and Use Sum 9 tasks, showing the model struggles to reduce loss on Use Sum (green line) compared to Base Add and Use Carry 1.
G APPENDIX MODEL ALGORITHM AS PSEUDOCODE
The Algorithm table below reproduces the model's addition algorithm summarised in Fig. 9 as pseudocode. This code calculates the largest digit first, handling simple (but not cascading) UseSum9 cases, and returns the answer.
Note that the pseudocode does not retain any information between passes through the for loop this corresponds to each digit being calculated independently of the other digits.
H APPENDIX THE LOSS FUNCTION AND LOSS MEASURES
The loss function simple:
Per Digit Loss: For "per digit" graphs and analysis, for a given answer digit, the loss used is negative log likelihood.
All Digits Loss: For "all answer digits" graphs and analysis, the loss used is the mean "per digit" loss across all the answer digits.
The final training loss varies with the number of digits the question as shown in Tab. 1. The final training loss for each digit in the question varies shown in Tab. 2.
Size of question
Final training loss
Example question
digit addition
0.009
11111 + 22222 033333
10 digit addition digit addition
0.011
1111111111 -23222222--33333333
0.031
+ 222222222222222
0333333353333333
Overall Average
0.017
General Performance
Table 1:
The final training loss after 5000 training epochs, each containing 64 questions, using All Digits Loss, for different size addition models.
We categorize each training question by which calculations its need to get the correct answer:
BA BaseAdd questions only need "base add" calculations.
MC1
MakeCarry questions need both "use carry 1" and "base add" calculations.
US9
UseSum9 questions need "use sum 9", "use carry 1" and "base add" calculations.
We measured the loss per question category shown in Tab. 3. The frequency of these question types differ significantly BA = 11, MC1 =3 US9 =6% in the enriched training data so the final training loss values, at this level of detail, not very informative.
After training, we used the model to give answers to questions. To understand whether the calculations position are important, look at the impact loss of ablating all attention heads at that token
16

---

Published as a conference paper at ICLR 2024
Algorithm I n-digit integer addition algorithm
function CALCULATTAANSWERDIGITTSS digits, figits,q1,92. q1,
2: answer <- 0
> Initialize the answer to zero
3: for all i = O..... digits 1 do
> Loop over each digit D Current position from the right Previous position from the right Position before the previous
4: pos +- nt digils
5:
prev pos < pos prev prev pos < pos - 2
6:
7:
mel prev prev t- if prev prev pos then
D Start calculation of carry from two positions before
8:
9:
if plev prev pos + p2pre prev pos 2 10 then mel prev prev t-1 end end if
Carry from two positions before found
11:
me1 prev 4- 0 14:
Start calculation of carry from previous position
if prev pos 2 0 then
15:
if glrev + 42prev pos 2 10 then mel prev <-
16:
Carry from the previous position found
17:
end end if
18:
ms9 prev 0 20:
Start calculation of sum of 9 in previous position
if prev pos 2 0 then
if glpev + 92prov then
22:
ms9 prev t-
>> Sum of 9 in previous position found
end end if
24:
25:
prev prev +- 0 mel prev = then if ms9 prev == 1 then
Start calculation if carry from two positions before is needed
26:
27:
28:
if mel prev prev == then prev prev 1 end end if'
29:
Carry from two positions before is needed
30:
32:
end if
33:
digitanswer + 9os + 22ps + mel prev + prev prev Calculate answer for current digit Correct the current digit if it's > 10 Concatenate the current digit to the answer
digitanswer& +- MODULUUS(dgitansss) 10)
35:
answer <- digitanswer answer 10
end for
37:
return answer end function
b Return the final calculated answer
38:
(using the Transformerl.ens framework, ablation of the blocks.0.hook. hww.oooooooosdoooo__eee _osttdpotttteeee data set, the above loss function, and a "cut off" loss threshold of 0.08). Tab. 4 shows sample results. The results shows that all important calculations are completed at just of the 18 (question and answer) token positions.
For deeper investigation, created 100+ hand-curated test questions. They cover all question types (BA, MCI and MS9) and all the answer digits (A.5 A0) that these question types can occur in. These test cases were not used in model training. Example test questions include:
make question( _._. 888, 11111, BASE ADD _
make q__uusttio(__... - 35000, 35000,
1 CASE)
17

---

Published as a conference paper at ICLR 2024
Digit index
Training loss
AKA
Example of digit
5
<0.001
A5
I1II+22222-33333
4
<0,001
A4
1III122222:03333
3
0.003
A3
1111+2222=033333
2
0.008
A2
111114222-033333
1
0.046
A1
1111+2222--33333
0
0.001
AO
11111+2222-03333
Overall Average
0.010
Table 2: For 5-digit addition, final training loss per digit. The more digits, the longer the model needs to train to given level of loss.
Question type
Training loss
Aka
Example of digit
BA
0.021
Base Add
11111+22222-33333 11811+22222-0344033 17811422222-040033
MC1
0.001
Make Carry
US9
<0.001
Use Sum
Overall Average
0.011
General Performance
Table 3: For 5 digit addition, the final training loss per question category.
make a question( _._. 15020, 45091, USE _ARRY_L_CASE
make question( _ 25, 79, SIMPLE_US US9 _
make a_ question( 41127, 10880, SIMPLE CASE)
make a_ q__quuetionn__... 123, 877, CASCADE_USE (OSOE_______SS_
make a queuutuinnnn.... _ 81818, 18182, CASCADE (S9_CASE)
The above experiment repeated (with "cut off" loss threshold of 0.1) using these Î00 questions (and another 64 random questions). We analyzed the incorrect answers, grouping these failures by which answer digits were wrong. Tab. 5 shows sample results.
Row
Number of incorrect answers, grouped by incorrect digits
11
Nyyyyy: 47, NyNyyy: 5, yNyyyy: 7, yyNyyy: 7, yyyNyy: I.
12
yNyyyy: 97, yNNyyy: 7, NNyyyy: 3, Nyyyyy: 2. yyyNyy:
13
yyNyyy: 85, NyNyyy: 3, yNNyyy: 2, Nyyyyy: 3, yNyyyy: 5,
14
JyyNyy: 72, yyNNyy: NyyNyy: 3, yNyNyy: 3, Nyyyyy:
15
VyyyNy: 74, yyNyNy: 3, Nyyyyy: 2, NyyyNy: 3, yNyyNy: 3,
16
MyууyN: 82, yyNyyN: 2, NyyyyN: 4, yNyyyN: 4, yyNyyy: 9,
Table For 5 digit addition, ablating all the attention heads in each row (aka step) impacts answer correctness in regular pattern. Here, an 'N' means that answer digit was incorrect in the predicted answer. each step, the failure count for the top grouping is > 6 times the next most common grouping evidence that each step calculates one answer digit. From row 11, each row mainly impacts one answer digit- in the same order as the model predicts answer digits.
We also ablated one head at a time (using the TransformerLens framework, mean ablation of the blocks.0.att..ben z data set, the standard loss function, and "eut off" loss threshold of 0.08) to understand which head(s) were key in calculating BA questions. Tab. 6 shows sample results.
The last of Tab. 6 shows that digit AO was calculated in step 16. This is step before the model reveals AO in step 17. The other rows in the table shows that this pattern holds true for all the other digits: each digit is calculated one step before the model reveals it.
18

---

Published as a conference paper at ICLR 2024
Token
Average loss
Conclusion
0
10
0.070
Impact is low. Cales for these tokens are unimportant
11
0.322
Loss is 4 threshold. Calculations important
0.497
Loss is 6 x threshold. Calculations are important Loss is 7 threshold. Calculations important
13
0.604
14
0.921
Loss is 11 x threshold. Calculations are important Loss is 14 threshold. Calculations are important
15
1.181
16
1.021
Loss is 12 x threshold. Calculations are important
0.070
Impact is low. Calculations for this token is unimportant
Table 4: For 5 digit addition, for the 18 tokens calculation rows, how ablating each token/row impacts the calculation loss.
Ablated Head
Average loss
Conclusion
0.016
Impact is low. Head is unimportant for BA
1
4.712
Loss is 50 threshold. Head is important for BA
2
0.062
Impact is low. Head is unimportant for BA
Table 6: For 5 digit addition, when ablating heads, Head is clearly key for the calculation of Base Add questions.
I APPENDIX ALGORITHM RE-USE
We explored whether the algorithm is re-used in other models by training separate 1-layer 5-digit addition model with a different seed, optimiser. also required the model to predict "+" as the first answer token shown in Fig. 3.
3
3 3 5
7
2 2 D'3
4
3
5 6
0
0
D
A
We refer to individual tokens in a 5-digit addition question as D4, DO, and D'4, D'O and the answer tokens as A6, A0. Note that A6 is always the "+" token.
Figure 21:
Figs. 24, 25 and 23 show aspects of the behavior this model. Fig. 26 shows the nodes in the model where the node purpose can be identified by automated search using behavior filtering and ablation intervention testing.
Pown
P3
75
P7
PB
P10
P13
P12
P13
P15
P18
P17
P1B
D4
D1
0/3
02
99
A5
AB
# fails
457
804
775
724
592
Figure 22: This shows the token positions where questions fail when we ablate each node in the 5-digit 1-layer 3-head addition model. The model only uses nodes in token positions P12 to P17 (answer digits A6 to A1).
19

---

Published as a conference paper at ICLR 2024
A6
A5
A4
A3
A2
A1
LOHO
23%
20%
25%
27%
15%
LOH1
8%
72%
59%
60%
56%
54%
LOH2
9%
10%
15%
12%
LOMC 39%
72%
75%
72%
70%
54%
Figure 23: This map shows the percentage of enriched questions that fail when we ablate each node in the 5-digit I-layer 3-head addition model. The model only uses nodes in token positions P12 (answer digits A6 to A1). Lower percentages correspond to rarer edge cases. The grey space represents nodes that are not used by the model.
A6
A5
A4
A3
A2
A1
LOHO
D4 D'4
D3 D'3
D2 D'2
D'1 D1
D'O DO
LOH1
D'2 D2
D'4
D3 D'3
D2 D'2
D'1
D'O
LOH2
D3 D'3
D'2 D2
D'1 D1
DO D'O
Figure 24: This map shows the input tokens each attention head attends to at each token position in the new model. The new model only uses attention heads in token positions P12 to P17 (answer tokens A6 to A1). Each head attends to a pair of input tokens. Each column attends to three different input token positions. These behaviors are the same in the and previous models.
A6
A5
A4
A3
A2
A1
LOHO
A5
A4
A3
A2
A1
LOH1
A5
A4
A3
A1
LOH2
A5
A4
A3
A2
LOMO A5
A4
A3
A2
A1
A0
Figure 25: This map shows the answer digit(s) A0 A5 impacted when ablate each attention head and MLP layer in the new model. This behavior the same in the new and previous models. Note that A6 is the constant "+" answer token.
20

---

Published as a conference paper at ICLR 2024
A6
A5
A4
A3
A2
A1
LOHO
A4.MC
A3.MC
A2.MC
AL.MC
AO.MC
LOH1
??
A4.BA
A3.BA
A2.BA
A1.BA
A0.BA
LOH2
A4.US
A3.US
A2.US
A1.US
LOMO
??
7?
??
7?
77
7?
Figure 26: This map shows the calculation tasks BA, MC1 and US9 located in this model by automated search using behavior filtering and ablation intervention testing. For nodes containing "??" the automated search did not identify the purpose of the node.
21


Reviews:
[{"review_id": "3wXj2eYVyz", "reviewer": "Reviewer_wEos", "summary": "This paper attempts to reveal the internal working mechanism of Transformers for integer addition tasks.", "questions": "1. Page 2, a latex error: \"d_e-dimensional embeddings\"\n\t2. Page 3, section 3, paragraph 3, \"detailing identified circuits\", is this a typo? Or which \"identified\" circuits? What is the \"identified\" process?\n\t3. Page  3, section 3, paragraph 4, \"techniques in works like symbolically … \", a grammar error?\n\t4. Page 3, section 3, paragraph 7, \"Surveys like  overview techniques…\", missing citations after \"Surveys like\"?\n\t5.  Page 4, section 4, paragraph 3, \"Fig. 2 shows … semi-indendently…\", what is the loss per digit is being plot in Figure 2? \n\t6. Page 4, section 4, paragraph 4, \"Transformer models always process text from left to right…\", this is not true. It is just an artifact of GPT-style attention masking. For example, we can do config the attention mask to enable full order attention over the two addends and generate the outputs in all kinds of order, e.g. from the tens digit to higher value digits, from the middle digit to two ends, and so on. We can also do non-autogression generation, e.g. incremental masking output generation. \n\t7. Page 4, figure 3 caption \"..After the question is fully revealed (at layer 11)..\", by \"layer 11\" do you mean the 11th row? To avoid ambiguity, it is better to number the attention matrix and refer to them the row or column number across the paper.  Also what are the sub-figures of 0.0, 0.1, 0.2? Different heads? What are the labels? \n\t8.   Page 4-5, section 5, please clarify whether the  \"mathematical framework\" is  for characterizing (grouping) addition data instances-digits only? Or is there a link to the loss? If so, please formulate the framework and what kind of mathematical hypotheses this framework can verify formally in mathematical terms?  Also please detail the loss on each digits formally. Also please detail the statistics of your training and valid datasets in terms of your classification of digits in your framework. \n\t9. Page 4-6, please detail how the loss is being average. Are they per digits or per digit average?\n\t10. Page 6, please introduce or define or describe phase 1, 2, 3 formally?  \n\t11. Page 7, section 7, \"During model prediction we overrode … the model memory (residual stream)…\", please detail the approach formally? Are your conclusions/assertions based on checking the attention scores?  Please discuss explicitly with formal treatment. Otherwise, the plain English language analysis in Section 7 is difficult to follow and justify. Also formally define \"independent of every other digit\", \"most impact on loss\" and define it based on measure statistics during model inference time.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "• Study an important problem of transformer models' application in numerical computation tasks.", "weaknesses": "• The analysis framework and analysis lacks mathematical rigidity. \n\t• The conclusion is not well established based on rigorous mathematical framework. \n\t• The paper does not fully utilize/choose the most relevant aspect of transformers for addition tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LsiisUjwEp", "reviewer": "Reviewer_f5y1", "summary": "Very interesting paper that focuses on explaining the \"inner workings\" of the foundational model of Transformer. While the use-case demonstrated (integer addition with a single layer transformer) is simplistic, the idea is novel and the visualisations are meaningful and make sense for better trust and confidence in how a transformer model works for the AI community.", "questions": "Solid theoretical framework in the paper, good interpretation and visualisations - no further questions from this reviewer. The paper is very well written, easy to understand and the method is clear.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Transformer model focus - no doubt an important model in the current AI landscape. Solid mathematical explainations and interpretation of the model working, the attention visualisations shown are very interesting and the model training loss curve which shows how a transformer trains individual digits semi-independently was promising to see.", "weaknesses": "No major weakness other than the paper applying the framework of explainability to a simple problem (integer addition). Though, this is well the strength as well of the paper as it makes the model easier to interpret and understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lExhXVmZCt", "reviewer": "Reviewer_sWU9", "summary": "This paper delves into the intricacies of a one-layer Transformer model trained for integer addition, emphasizing the importance of understanding machine learning models for safety and ethical considerations. The study uncovers that the model breaks down the addition task into parallel, digit-specific streams, using different algorithms for various digit positions. Interestingly, the model starts its calculations later but completes them swiftly. A unique use case with a high loss is pinpointed and elaborated upon.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This is a technically solid, moderate to high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.", "weaknesses": "1. The adaptability of the methodology in this paper is limited, as it only applies to a one-layer Transformer model. Perhaps further analysis on two-layers or even more complex models would be beneficial. Moreover, the study solely focuses on integer addition, making it challenging to extend to other operations like subtraction or multiplication.\n2. The writing of this paper is not comprehensive. For instance, the descriptions for Figure 4 and 8 are difficult to comprehend.\n3. The experiments conducted are not exhaustive. In the \"Prediction Analysis\" section, the authors failed to provide a specific metric and results compared to baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O0mjSDChA0", "reviewer": "Reviewer_ExS5", "summary": "The paper studies the interpretability of Transformers. The authors focus on the 5-digit number addition task and analyze how a one-layer Transformer model finishes this task. To understand the model clearly, they propose a mathematical framework for integer addition, consisting of five tasks: Base Add, Make Carry 1, Make Sum 9, Use Carry 1, and Use Sum 9. The first three tasks can be independently executed for each digit pair, representing the sum of two digits modulo 10, checking for carry, and determining if the addition results in 9, respectively. The last two tasks chain operations across digits, respectively denoting adding the previous column's carry to the sum of the current digit pairs, and propagating a carry when Make Sum 9 and Use Carry 1 are true. The authors then analyze the one-layer Transformer model under this framework during the training phase and testing phase. More precisely, during the training phase, the authors investigate the training loss for Base Add (BA), Use Carry 1 (UC1), and Use Sum 9 (US9) three tasks. According to the experimental results,   US9 is the most complicated, especially in the case where more than one column carry occurs (e.g. 445+555=1000) and BA, UC1 two tasks are highly correlated. During the testing phase, the authors use ablation experiments to evaluate each attention head and conclude that for different digit pairs, the model uses slightly different algorithms.", "questions": "(1) What's the performance of the trained model on the test data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper advances in the direction of opening the black box of Transformers, which is a very important topic as Transformers are being applied in an increasing number of domains. \n\n- The authors decompose integer addition into several subtasks and investigate the loss of each task during the training. This might provide some inspiration for future improvements in deep learning for math.\n\n- The paper is well-organized. The basic idea is clean and easy to follow.", "weaknesses": "- One experimental flaw is that the test accuracy of the model is not provided. In addition, it is also worthwhile to explore using the trained model directly for the addition of integers with more digits.\n\n- In the integer addition task, digit 0 should be treated as a special case, since intuitively, when humans perform integer calculations, the more zeros there are, the easier the calculation becomes. In other words, the digit 0 requires special attention. So it might be interesting to include Make Sum 0 and Use Sum 0 in the mathematical framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)