A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis
Authors: Alessandro Stolfo
Yonatan Belinkov
Mrinmaya Sachan
Venue: EMNLP 2023 Main (2023)
Keywords: LLMs
Arithmetic Reasoning
Interpretability
Causality
Causal Mediation Analysis
Reasoning

Abstract: Mathematical reasoning in large language models (LMs) has garnered significant attention in recent work, but there is a limited understanding of how these models process and store information related to arithmetic tasks within their architecture.
In order to improve our understanding of this aspect of language models, we present a mechanistic interpretation of Transformer-based LMs on arithmetic questions using a causal mediation analysis framework.
By intervening on the activations of specific model components and measuring the resulting changes in predicted probabilities, we identify the subset of parameters responsible for specific predictions.
This provides insights into how information related to arithmetic is processed by LMs.
Our experimental results indicate that LMs process the input by transmitting the information relevant to the query from mid-sequence early layers to the final token using the attention mechanism. Then, this information is processed by a set of MLP modules, which generate result-related information that is incorporated into the residual stream.
To assess the specificity of the observed activation dynamics, we compare the effects of different model components on arithmetic queries with other tasks, including number retrieval from prompts and factual knowledge questions.

A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis
Alessandro Stolfo
Yonatan Belinkov Technion - IIT, Israel bel belinkov@technion.co ac. il
Mrinmaya Sachan ETH Zürich
ETH Zürich stolf moolfoaeethz.co
msachanēethz.ch ch
Abstract
Attention
MLP Module
180
Mathematical reasoning large language mod- els (LMs) has gamered significant attention in recent work, but there is a limited under- standing how these models process store information related to arithmetic tasks within their architecture. In order to improve our un- derstanding of this aspect of language mod- els, we present mechanistic interpretation Traaasffrer--aaseee LMs on arithmetic ques- tions using causal mediation analysis frame- work. By intervening on the activations of spe- cific model components and measuring the re- sulting changes predicted probabilities, identify the subset of parameters responsible for specific predictions. This provides insights into how information related to arithmetic is processed by LMs. Our experimental results indicate that LMs process the input by trans- mitting the information relevant to the query from mid-sequence early layers to final to- ken using the attention mechanism. Then, this information is processed by set of MLP mod- ules, which generate result-related information that is incorporated into the residual stream. To assess the specificity of the observed activation dynamics, compare the effects of different model components on arithmetic queries with other tasks, including number retrieval from prompts and factual knowledge questions.!
C: Result
Generstion
Information Transfer
A: Operands Operator
Processing
What
is 15
times 12
Visualization of findings. We trace the flow of numerical information within Transformer- based LMs: given an input query, the model processes the representations of numbers and operators with early layers (A). Then, the relevant information is conveyed by the attention mechanism to the end of the input se- quence (B). Here, it is processed by late MI.P modules, which output result-related information into the residual stream (C).
Figure I:
of works proposing methods to improve the perfor- mance of large LMs on math benchmark datasets through enhanced pre-training (Spokoyny et al.. 2022; Lewkowyzz et al., 2022; Liu and Low, 2023) or specific prompting techniques (Wei et al., 2022b; Kojima et al., 2022; Yang al., 2023, inter alia). However, there limited understanding of the inner workings these models and how they store and process information to correctly perform math- based tasks. Insights into the mechanics behind LMs' reasoning are key to improvements such as inference- correction of the model's behavior (Li et al., 2023) and safer deployment. Therefore, research in this direction is critical for the develop- ment of more faithful and accurate next-generation LM-based reasoning systems.
1 Introduction
Mathematical reasoning with Transformer-basee models (Vaswani et al., 2017) is challenging as it requires an understanding of the quantities and the mathematical concepts involved. While large language models (LMs) have recently achieved impressive performance on a set of math-based tasks (Wei et al., 2022a: Chowdhery et al., 2022: OpenAI, 2023), their behavior has been shown to be inconsistent and context-dependent (Bubeck et al., 2023). Recent literature shows multitude
'Our ende and data is available at https: http://.gth..h///////// alestolfo/ 1m- ari thmetic.
In this paper, we present set of analyses aimed

---

at mechanistically interpreting LMs on the task of answering simple arithmetic questions (c.g.. "What is the product of 11 and 17?"). In particular, we hypothesize that the computations involved in rea- soning about such arithmetic problems are carried out by a specific subset of the network. Then, we test this hypothesis by adopting a causal mediation analysis framework (Vig et al., 2020; Meng et al., causal graph
et al., 2017, 2018, 2020; Voss al., 2021; Goh et al., 2021) and on how information is processed by Transformer-based (Vaswani et al., 2017) lan- guage models (Geva et al., 2021, 2022, 2023; Ols- son et al., 2022; Nanda et al., 2023). Although not strictly mechanistie, other recent studies have analyzed the hidden representations and behavior of inner components of large LMs (Belrose et al., 2023; Gurnee et al., 2023; Bills al., 2023).
2022), where the model is seen going from inputs to outputs, and the model compo- nents (e.g., neurons or layers) are seen as mediators (Pearl, 2001). Within this framework, we assess the impact of a mediator on the observed output behavior by conducting controlled interventions on the activations of specific subsets of the model and examining the resulting changes in the probabilities assigned to different numerical predictions.
Causality- -bsed Interpretability. Causal medi- ation analysis is an important tool that is used to ef- fectively attribute the causal effect of mediators on an outcome variable (Pearl, 2001). This paradigm was applied to investigate LMs by Vig et al. (2020), who proposed a framework based on causal medi- ation analysis to investigate gender bias. Variants of this approach were later applied to mechanis- tically interpret the inner workings of pre-trained LMs on other tasks such as subject-verb agreement (Finlayson et al., 2021), I), natural language inference (Geiger et al., 2021), indirect object identification (Wang et al., 2022), and to study their retention of factual knowledge (Meng al., 2022).
Through this experimental procedure, we track the flow of information within the model and iden- tify the model components that encode information about the result of arithmetic queries. Our findings show that the model processes the input by convey- ing information about the operator and the operands from nid-seeuence early layers to the final token using attention. At this location, the information is processed by a set of MLP modules, which output result-related information into the residual stream (shown in Figure 1). We verify this finding for bi- and tri tri-variate arithmetic queries across four pre- trained language models with different sizes: 2.8B, 6B, and 7B parameters. Finally, we compare the effect of different model components on answer- ing arithmetic questions to two additional tasks: a synthetic task that involves retrieving a number from the prompt and answering questions related to factual knowledge. This comparison validates the specificity of the activation dynamics observed on arithmetic queries.
Math and Arithmetic Reasoning. A growing body of work has proposed methods to analyze the performance and robustness of large LMs on tasks involving mathematical reasoning (Pal and Baral, 2021; Piękos et al., 2021; Razeghi al., 2022: Cobbe et al., 2021; 1; Mishra et al., 2022). In this area, Stolfo et al. (2023) use a causally-groundee approach to quantify the robustness of large LMs. However, the proposed formulation is limited to behavioral investigation with no insights into the models' inner mechanisms. To the best of our knowledge, our study represents the first attempt to connect the area of mechanistic interpretability to the investigation of the mathematical reasoning abilities in Transformer-based LMs.
2 Related Work
3 Methodology
Mechanistic Interpretability. The objective of mechanistic interpretability is to reverse engineer model computation into components, aiming to discover, comprehend, and validate the algorithms (called circuits in certain works) implemented by the model weights (Räuker et al., 2023). Early work in this area analyzed the activation values of single neurons when generating text using LSTMs (Karpathy et al., 2015). A multitude of studies have later focused on interpreting weights and inter- mediate representations in neural networks (Olah
3.1 Background and Task
We denote an autoregressive language model as g X' + P. The model operates over a vocabulary V and takes a token sequence x [=1...., Fr] E X. where each H, E V. & generates probability dis- tribution F P VI + [0, 1] that predicts possi- ble next tokens following the sequence x. In this work, we study decoder-only Transformer- models (Vaswani et al., 2017). Specifically, we focus on models that represent a slight variation of

---

the standard GPT paradigm, as they utilize parallel attention (Wang and Komatsuzaki, 2021) and rotary positional encodings (Su et al., 2022). The internal computation of the model's hidden states hi.) at position { of the input sequence is carried out follows:
After Intervention
68
203
103
Modal Predictions 231
Clean Run
Poet-intervertion Prediction: 231
Prediction: 121
h h
(1) 11-1)
(!) + a4 + my
(1)
(I)
at A (11 11-1) ,...., n1g )
MLP
MLP
w (w/th 1/ b((-11)
m4
Set Value of Activations
=: =ML(( 1),,
MLP
MLP
where at layer I, is the sigmoid nonlinearity. wilh)
fe and W(()) ize the multilayer perceptron (MIP) of the Trans- former block and A is the attention mechanism.
proj are two matrices that parameter-
We consider the task of computing the result of arithmetic operations. Each arithmetic query consists a list operands = (n1, 722,............... ...) and a function fo representing the application of a set of arithmetic operators (+. +). We denote as = fo(N) the result obtained by applying the operatirs to the operands. Each query is rendered natural language question through prompt p( fo) X such "How much is nı plus nz?"" (in this case, fo((noonn))) m22) nı + n2)). The prompt is then fed to the language model to produce a probability distribution over Our aim to investigate whether certain hidden state variables are more important than others during the process of computing the result r.
MLP
MLP
Pi
plus
Pb
plus
By intervening on the activation values of specific components within a language model and com- puting the corresponding effects, identify the subset of parameters responsible for specific predictions.
Figure 2:
the outputs of the MLP(!) and A()) functions at (1) and a, (8)
each token i..., m²
To quantify the importance of modules MLP(()
and A (1) in mediating the model's predictions at position f, we use the following procedure.
1. Given fo, we sample two sets of operands N, N', and we obtain r = fo(N) and fo Then, two input questions with only the operands differing,
3.2 Experimental Procedure
We see the model G as a causal graph (Pearl, 2009), framing internal model components, such as spe- cific neurons, as mediators positioned along the causal path connecting model inputs and outputs. Following causal mediation analysis procedure, we then quantify the contribution of particular model components by intervening on their acti- vation values and measuring the change the model's output. Previous work has isolated the effect of every single neuron within model (Vig et al., 2020; Finlayson et al., 2021). However, this approach becomes impractical for models with bil- lions of parameters. Therefore, for main exper- iments, the elements that we consider variables along the causal path described by the model are
= p(N, fo) and P2 - p(N' (.NN,,o) fo), are passed through the model.
2. During the forward pass with input P1, store the activation values m, (t) == MLP@ (-1)),
and a." : (n) (t-1) "
h (("1))
3. We perform additional forward pass using p2, but this time intervene components MLP(() and A position t,, setting their tivation values to mi
(i) (1) and a; respectively.
This process is illustrated in Figure 2.
4. We measure the causal effect of the intervention on variables m, and (6) on the model's predic- tion by computing the change in the probability values assigned to the results r and r'.
For brevity, layer normalization (Ba et al., 2016) is omit- ted as it is not essential for our analysis.

---

More specifically, we compute the indirect ef- feet (IE) of specific mediating component by quantifying its contribution in skewing IP towards the correct result. Consider a generic activation variable = E {ml" ....., ,,, ai (1)
Appendix A. In the bi-variate case, for each of the four operators fo E (+.-, , x, and for each of the templates, we generate 50 pairs of prompts by sampling two pairs of operands (m1, n2) E S and (n1: m22) E S where S C VON. For the operand- related experiment, we sample (M1,72)) and sec- ond operation fo. In both cases, we ensure that the result falls within S. In the three-operand case, generate 15 pairs of prompts for each of the 29 templates, following the same procedure. In order to ensure that the model achieves a meaningful task performance, we use a two-shot prompt which we include two exemplars of investion-answer for the same operation that is being queried. We report the accuracy results in Appendix B.
..... ). We denote the model's output probability following an intervention on 2 as P.. Then, we compute the IE
as:
1 (r) - P(r) (E 2 P(r)
P(r') (P0-PP()) (2)
(2(r))
where the two terms in the sum represent the rela- tive change in the probability assigned by the model to and r', caused by the intervention performed. The larger the measured IE, the larger the contri- bution of component & in shifting probability mass from the clean-run result r' result correspond- ing to the alternative input p1.
Causal Effects on Arithmetic Queries
Our analyses address the following question:
We additionally measure the mediation effect of each component with respect to the operation fo. We achieve this by fixing the operands and changing the operator across the two input prompts. More formally, in step I, sample list of operands N and operators So and f'o Then, we generate two prompts p1 pNN, fo) and P2 - P(N, (c.g.. "What is the sum of 11 and 7 and "What the product of 11 and 7?"). Finally, we carry out the procedure in steps 2-4.
Q1 What are the components of the model that mediate predictions involving arithmetic com- putations?
We address this question by first studying the flow of information throughout the model by measur- ing the effect of each component (MLP and atten- tion block) at each point of the input sequence for two-operand queries ($4.1). Then, we distinguish between model components that carry information about the result and about the operands of the arith- metic computations ($4.2 and $4.3). Finally, we consider queries involving three operands ($4.4) and present a measure to quantify the changes in information flow ($4.5).
3.3 Experimental Setup
We present the results of our analyses in the main paper for GPT-J (Wang and Komatsuzaki, 2021), 6B OB-parameter pre-trained LM (Gao al., 2020). Additionally, we validate our findings on Pythia 2.8B (Biderman et al., 2023), LLaMA 7B (Touvron et al., 2023), and Goat, a version of L.LaMA fine- tuned on arithmetic tasks (Liu and Low, 2023). We report the detailed results for these models in Appendix J.
4.1 Tracing the Information Flow
We mcasure the indirect effect of cach MLP and attention block at different positions along the in- put sequence. The output of these modules can be seen as new information being incorporated into the residual stream. This new information can be produced any point of the sequence and then con- veyed the end of the sequence for the prediction of the next token. By studying the IE at different locations within the model, we can identify the modules that generate new information relevant to the model's prediction. The results are reported in Figures 3a and 3b for MLP and attention, respec- tively.
In our experiments, we focus on two- and three- operand arithmetic problems. Similar to previous work (Razeghi et al., 2022; Karpas et al., 2022), for single-operator two-operand queries, we use a set of six diverse templates representing a ques- tion involving each of the four arithmetic operators. For the three-operand queries, we use one template for each of the 29 possible two-operator combina- tions. Details about the templates are reported in
"As an alternative metric to quantify the IE, we experiment using the difference in log probabilities (Appendix 1). The results ohtained with the two metrics show consistency and lead to the same conclusions.
"Unless otherwise specified, we use S = (1,2....300) as larger integers get split into multiple tokens by the tokenizer.

---

(a)
(b)
(e)
IE of the MLP Modules
IE of the Attention
1600
Eary
Early
Operand
101
Operand
10a
500
Midd
10
Middie
100
2nd Oparand
2nd Oparand
20
10a
10
Lest-token HLPS thks. Fiael
10-1
10-1
10
15 Layer
20
25
5
10
15
20
25
0.5
10
20
(g)
Last tokion Attention
of Modilles (Result Fixed)
(d)
(Result Fixed)
2020
Eary
Early
1020
Operand
Operand
Middle
Middle
(h)
Last-ccbun
10
(Pus. Foudi
20
2nd Operand
2nd Operand
Late
Late
Last
Lest
Layer
Layer
Figure 3: Indirect effect (IE) measured within GPT-J. Figures (a) and (b) illustrate the flow of information related to both the operands and the result of the queries, while the effect displayed in Figures (c) and (d) is related to the operands only (the result is kept unchanged). Figures (e-h) show re-scaled visualization of the effects at the last token for each of the four heatmaps (a-d). The difference in the effect registered for the MLPs at layers 15-25 between figures (a) and (c) illustrates the role of these components producing result-related information.
Our analysis reveals four primary activation sites: the MLP module situated at the first layer corre- sponding to the tokens of the two operands; the intermediate attention blocks at the last token of the sequence; and the MLP modules in the middle- to-late layers, also located the last token of the sequence. It is expected observe a high effect for the first MIPss associated with the tokens that vary (i.c., the operands), as such modules are likely to affect the representation of the tokens, which is subsequently used for the next token prediction. On the other hand, of particular interest is the high effect detected at the attention modules in layers 11-18 and in the MLPs around layer 20.
mation within Transformer based models to the attention mechanism (Elhage et al., 2021; Geva et al., 2023), while the feed-forward layers are as- sociated with performing computations, retrieving facts and information (Geva al., 2022; Din al., 2023; Meng al., 2022). We test the validity of this hypothesis in the following section.
4.2 Operand- and Result-related Effects
Our objective is verify whether the contribu- tion to the model's prediction of each component measured in Figures 3a and 3b due to (/) the component representing information related to the operands, or (2) the component encoding informa- tion about the result of the computation. To this end, we formulate a variant of our previous experi- mental procedure. In particular, we condition the sampling of the second pair of operands (n', n²)) on the constraint two input questions p1 and p2, such that their re- sult the same (e.g., "What is the sum of 25 and 7?" and "What the sum of 14 and 18?"). case number (/), we would expect a component to have high IE both in the result-varying setting and when = r'. as the operands are modified in both scenarios. In case (2), we expect subset of the model to have large effect when the operands are sampled without constraints but low effect for the fixed-result setting.
As for the flow of information tied to the opera- tor, the activations display a parallel pattern: high effect is registered at early MLPs associated with the operator tokens and at the same last-token MLP and attention locations. We report the visualization of the operator-related results in Appendix C. possible explanation of the model's behavior on this task is that the attention mechanism facil- itates the propagation of operand- and operator- related information from the first layers early in the sequence to the last token. Here, this infor- mation processed by the MLP modules, which incorporate the information about the result of the computation in the residual stream. This hypoth- esis aligns with the existing theory that attributes the responsibility of moving and extracting infor-
-r. That is, we generate the
We report the results in Figure 3c and 3d. By

---

comparing Figures 3a and 3c, two notable obser- vations emerge. First, the high effect in the early layers corresponding to the operand tokens ob- served in both the result-preserving and the result- varying scenarios. Second, the last-token mid-late MLPs that lead to high effect on the model's prediction following a result change, dramatically decrease their effect on the model's output in the result-preserving setting, described in scenario (2). These observations point to the conclusion that the MLP blocks around layer 20 incorporate result- relevant information. As for the contribution of the attention mechanism (Figures 3b and 3d), we do not observe substantial difference in the layers with the highest IE between the two settings, which aligns this scenario to the description of case (/). These results consistent with hypothesis that operand-related information is transferred by the attention mechanism to the end of the sequence and then processed by the MLPs to obtain the result the computation.
IE of the MLPs in Pythia 2.88
Early
10
1st Operand
Belween Operands
10a
2nd Operand
10
Between Operands
3rd Operand
10
Late
Last
-10-1
10
15
20
25
30
Layer
IE of MLPs
Pythia 2.88 (FT)
1011
Early
1st Operand
Belween Operands
10a
2nd Operand
Between Operands
10
3rd Operand
10
10-
Layer
Figure 4: Indirect effect (IE) on three-operand queries for different MLP modules in Pythia 2.8B before and after fine-tuning. The effect produced by last-token mid-late MLP activation site emerges with fine-tuning, Results for the attention are reported in Appendix
4.3 Zooming in on the Last Token
Figures 3e-3h, we show a re-scaled version of the IE measurements for the layers at the end of the input sequence. While the large difference in magnitude was already evident in the previously considered visualizations, in Figures 3e and 3f notice that the MLPs with the highest effect in the two settings differ: the main contribution to the model's output when the results are not fixed is given by layers 19 and 20, while in the result- preserving setting the effect is largest at layers 14- 18. For the attention (Figures 3g and 3h), we do not observe a significant change in the shape of the curve describing the IE across different layers, with layer 13 producing the largest contribution. We interpret this as additional evidence indicating that the last-token MLPs layers 19-20 encode information about r, while the attention modules carry information related to the operands.
as its smaller size allows for less computationally demanding training than the 6B-parameter GPT-J. After fine-tuning, the model attains an accuracy of 400% We provide the details about the training procedure in Appendix F.
We carry out the experimental procedure as Section 4.1. particular, we compare the informa- tion flow in the MLPs of the model before and after fine-tuning (Figure 4). In the non-fine-tuned ver- sion of the model, the only relevant activation site, besides the early layers at the operand tokens, is the very last layer at the last token. In the fine-tuned model, on the other hand, notice the emergence of the mid-late MLP activation site that was previ- ously observed the two-operand setting.
4.5 Quantifying the Change of the Information Flow
Denote the set MLPs in the model by M. We define the relative importance (RI) of a specific subset M C .M of MLP modules as
4.4 Threc-operand Queries & Fine-tuning
We extend our analyses by including three-operand arithmetic queries such as "What is the difference between n and the ratio between n2 and ng nz?"". An- swering correctly this type of questions represents challenging task for pre-trained language models, and we observe poor accuracy (below 10%) with GPT-J. Thus, we opt for fine-tuning the model on a small set of three-operand queries. The model that we consider for this analysis is Pythia 2.8B,
RI(M") Emc.y log (m) + +1
(3)
EmEM (TF + +1
In order to quantitatively show the difference in the activation sites observed in Figure we compute the RI measure for the set
M (m ((1/21)
(11/2++1) m m.
(1)
m

---

IE of the MLP Modules (Number Retrieval)
RI(M-] Result Fixed
|N|
Model
II(
Early
44 1st Tak
GPT-J
40.2%
4.4%
Pythia 2.8B
43.2%
5.8%
eg Mid Toks
2
LLaMA 7B
36.1%
7.5%
e, Last Tok
100
Gout
33.5%
7.4%
GPT-J (Words)
27.8%
4.5%
Pythia 2.8B
13.5%
6.7%
10-1
Pythia 2.8B (FT)
24.7%
13.6%
0
10 15 Layer
20
25
Table 1: Relative importance (RI) measurements for the last-token late MI.P activation site. The decrease in the RI observed when fixing the result of the two pairs of operands used for the interventions quantita- tively confirms the role of this subset of the model in incorporating result-related information.
Figure 5: Indirect effect measured on the MLPs of GPT- J for predictions on the number retrieval task.
edge ($5.2). With this additional set of experiments, we aim to answer the question:
Q2 Are the activation patterns observed so far spe- cific to the arithmetic setting?
where the subseript 11 indicates the last token of the input sequence and L is the number of layers in the model. This quantity represents the rela- tive contribution of the mid-late last-token MLPs compared to all the MLP blocks in the model.
5.1 Information Flow on Number Retrieval
We consider simple synthetic task involving nu- merical predictions. We construct set of templates of the form "Paul has nı e1 and n2 e2. How many ey does Paul have?", where n1, n2 are two ran- domly sampled numbers, E1 and e2 are two entity names sampled at random, and Gg E (c1,02). this case, the two input prompts p1 and p2 differ solely in the value of eq. To provide the correct answer to query, the model has simply to retrieve the correct number from the prompt. With this task, aim to analyze the model's behavior in setting involving numerical predictions but not requiring any kind of arithmetic computation.
For the two-operand setting, we carry out the experimental procedure deseribed in Section 3 for three additional models: Pythia 2.8B, LLaMA 7B, and Goat. Furthermore, we repeat the analyses on GPT-J using a different number representation: instead of Arabic numbers (e.g., the token 2), we represent quantities using numeral words (e.g., the token two). For the threc-operand setting, we re- port the results for Pythia 2.8B before and after fine-tuning. We measure the effects using both randomly sampled and result-preserving operand pairs, comparing the RI measure in the two settings. The results (Table 1) exhibit consistency across all these four additional experiments. These quantita- tive measurements further highlight the influence of last-token late MLP modules on the prediction of r. We provide in Appendix the heatmap illus- trations of the effects for these additional studies.
We report the indirect effect measured for the MLPs modules of GPT-J in Figure 5. In this setting, observe an unsurprising high-effect activation site corresponding to the tokens of the entity og and lower-effect site the end of the input layers 14-20. The latter site appears in the set of the model components that were shown to be ac- tive on arithmetic queries. However, computing the relative importance of the late MLPs on this task shows that this second activation site is respon- sible for only RI(M! 8.7% of the overall log IE. The low RI, compared the higher values measured on arithmetic queries, suggests that the function of the last-token late MLPs not dictated by the numerical type of prediction, but rather by their involvement in processing the input informa- tion. This finding is aligned with our theory that
5 Causal Effects on Different Tasks
In order to understand whether the patterns in the effect of the model components that observed so far are specific arithmetic queries, we com- pare our observations on arithmetic queries to two different tasks: the retrieval of number from the prompt 1)), and the prediction of factual knowl-
"The LLaMA tokenizer considers each digit as an inde- pendent token in the vocabulary. This makes it problematic to compare the probubility value assigned by the model to multi-digit numhers. Therefore, we restrict the set of possible results to the set of single-digit numbers.
"We sample entities from a list containing names of ani- mals, fruits, office tools, and other everyday items and objects.

---

IE of the MLP Modules (Factual Q.)
100%
Early
10a
10%
9%
9%
100%
Subj First Tok
75%
Other Subj Toks
10a
W NR
23%
22%
100%
9%
Subj Last Tok
50%
Late
10
50%
100%
22%
9%
Last
-10-1
25%
5
10
15 Layer
20
25
Ar
100%
50%
23%
10%
Figure 6: Indirect effect measured on the MLPs of GPT- for predictions factual queries.
Figure 7: Overlap ratio the top 400 neurons with the largest effect on predicting answers factual queries involving Arabic Numerals (Ar) and numeral words (W), number retrieval (NR), and factual knowledge (F). The results obtained with GPT-J.
sees M as the location where information about r is included in the residual stream.
5.2 Information Flow on Factual Predictions
specificity of the model's activation patterns when answering arithmetic queries.
We carry out our experimental procedure using data from the LAMA benchmark (Petroni al., 2019), which consists of natural language tem- plates representing knowledge-base relations, such as "[subject] is the capital of [object]". By in- stantiating template with a specific subject (e.g., " "Prris") "), we prompt the model to predict the cor- rect object ("France"). Similar to our approach with arithmetic questions, we create pairs of factual queries that differ solely in the subject. In partic- ular, we sample pairs of entities from the set of entities compatible for given relation (c.g., citics for the relation "is the capital of"). Details about the data used for this procedure are provided in Appendix H. We then measure the indirect effect following the formulation in Equation 2, where the correct object corresponds the correct numerical outcome in the arithmetic scenario.
5.3 Neuron-level Interventions
The experimental results Sections 5.1 and 5.2 showed a quantitative difference in the contribu- tions of last-token mid-late MLPs between arith- metic queries and two tasks that do not involve arithmetic computation. Now, we investigate whether the components active within M on the different types of tasks are different. We carry out finer-grained analysis in which we consider independently each neuron an MLP module (i.e., each dimension in the output vector of the func- tion MLP(") at a specific layer In particular, following the same procedure as for layer-level ex- periments, we intervene each neuron by setting its activation the value it would take if the input query contained different operands (or different entity). We then compute the corresponding indi- rect effect as in Eq. 2. We carry out this procedure for arithmetic queries using Arabic numerals (Ar) and numeral words (W), for the number retrieval task (NR), and for factual knowledge queries (F).? We rank the neurons according to the average effect measured for each of these four settings and com- pute the overlap in the top 400 neurons (roughly 10%, GPT-J has hidden dimension of 4096).
From the results (Figure 6), we notice that main activation site emerges in early layers at the tokens corresponding to the subject of the query. These findings consistent with previous works (Meng al., 2022: Geva al., 2023), which hypothe- size that language models store and retrieve factual associations in early MLPs located at the subject tokens. We compute the RI metric for the late ML.P modules, which quantitatively validates the contri- bution of the early MLP activation site by attaining a low value of RI(ML = 4.2%. The large IF observed at mid-sequence carly MLPs represents difference in the information flow with respect the arithmetic scenario, where the modules with the highest influence on the model's prediction are located at the end of the sequence. This differ- ence serves as additional evidence highlighting the
We earry out this procedure for layer / 19, as it exhibits the largest within M on all the tasks considered. The heatmap in Figure 7 illustrates the results. We observe a consistent overlap (50%) between the top neurons active for the arithmetic queries using Arabic and word-based representa-
"To have the same result space for all the arithmetic queries (Ar and NW) and for the number retrieval task, we restrict the set S to {1...., 200 (or the corresponding numeral words).

---

tions. Interestingly, the size of the neuron overlap between arithmetic queries and number retrieval is considerably lower (22% and 23%), even though both tasks involve the prediction of numerical quan- tities. Finally, the overlaps between the top neurons for the arithmetic operations and the factual pre- dictions (between 9% and 10%) are not larger than for random rankings: the expected overlap ratio between the top 400 indices in two random rank- ings of size 4096 is 9.8% (Antverg and Belinkov, 2022). These results support the hypothesis that the model's circuits responsible for different kinds of prediction, though possibly relying on similar subsets of layers, are distinct. However, it is im- portant to note that this measurement does not take into account the magnitude of the effect.
multiplication, and division form the cornerstone of arithmetic calculations and serve as the basis for wide range of mathematical computations. Thus, exploring their mechanisms in language mod- els provides a starting point to explore more com- plex forms of mathematical processing. Studying broader set of mathematical operators represents an interesting avenue for further investigation.
Our work focuses on synthetically- -egerrated queries that are derived from natural language de- scriptions of the four basic arithmetic operators. To broaden the scope, future research can expand the analysis of model activations to encompass math- based queries described in real-life settings, such as math word problems.
Finally, a limitation of our work concerns the analysis of different attention heads. In our ex- periments, we consider the output of an attention module as whole. Future research could focus on identifying the specific heads that are responsi- ble for forwarding particular types of information in order offer more detailed understanding of their individual contributions.
6 Conclusion
We proposed the use of causal mediation analy- sis to mechanistically investigate how LMs pro- cess information related to arithmetic. Through controlled interventions on specific subsets of the model, we assessed the impact of these mediators on the model's 's predictions.
A Acknowledgments
We posited that models produce predictions to arithmetic queries by conveying the math-relevant information from the mid-sequence early layers the last token, where this information is then processed by late MLP modules. We carried out muusality-rrounde experimental procedure on four different Transformer-base LMs, and we provided empirical evidence supporting our hypothesis. Fur- thermore, we showed that the information flow we observed in our experiments is specific to arith- metic queries, compared to two other tasks that do not involve arithmetic computation.
AS is supported by armasuisse Science and Tech- nology through a CYD Doctoral Fellowship. YB is supported by an AI Alignment grant from Open Philanthropy, the Israel Science Foundation (grant No. 448/20), and an Azrieli Foundation Early Ca- reer Faculty Fellowship. MS acknowledges sup- port from the Swiss National Science Foundation (Project No. 197155), a Responsible Al grant by the Haslerstiftung, and an ETH Grant (ETH-19 21-1). We are grateful to Vilém Zouhar and Neel Nanda for the insightful discussions.
Our findings suggest potential avenues for re- search into model pruning and more targeted craining/fine-tnning by concentrating on specific model components associated with certain queries or computations. Moreover, our results offer in- sights that may guide further studies into using LMs' hidden representations to correct the model's behavior math- based tasks at inference time (Li al., 2023) and to estimate the probability of the model's predictions be true (Burns al., 2023).
References
Omer Antverg and Yonatan Belinkov. 2022, On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450
Limitations
Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney, Stella Bider- man, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint a3X3::2303.081
The scope of our work investigating arithmetic reasoning and we experiment with the four funda- mental arithmetic operators. Addition, subtraction,

---

Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashamth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint 2014:3330..
Natural Language Processing (Volume 1: Long Papers), pages 1828-1843, Online. Association for Computational Linguistics.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing. Travis Hoppe, Charles Foster, Jason Phang, Ho- race He, Anish Thite, Noa Naheshima, al. 2020. The pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0000
Steven Bills, Nick Cammarata, Dan Moss- ing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike. Jeff Wu, and Language mod- els can explain neurons in language models, httpo:://opemaajuulii...boo core, windows. html.
William Saunders. 2023.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christo- pher Potts. 2021. Causal abstractions of neural net- works. Advances in Neural Information Processing Systems, 34:9574-9586.
core htte//eurron-eepllliierrrppppppp
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint 12ti:22303.1211..
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson, 2023. Dissecting recall of factual asso- ciations in auto-regressive language models. arXiv preprint arXiv:2304. 147677.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Gold- berg. 2022. Transformer feed-forward layers build predictions by promoting concepts the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 30-45, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Stein- hardt. 2023. Discovering latent knowledge in lan- guage models without supervision. In The Eleventh International Conference Learning Representa- tions.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key- value memories. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Rad- ford, and Chris Olah. 2021. Multimodal neurons in artificial neural networks. Distill, 6(3)=30.
Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2023. Jump to conclusions: Short- Cutting transformers with linear transformations. arXiv preprint a3X1v:2303.09435.
Wes Gurnee, Neel Nanda, Matthew Pauly, Kather- ine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. 2023. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.0161
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread.
Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, al. 2022. MRKI. systems: modular, neuro-symbolic archi- tecture that combines large language models, exter- nal knowledge sources and discrete reasoning. arXiv preprint arXiv:2205.0044
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. arXiv preprint 100.:000060077
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. 2021. Causal analysis of syntactic agreement mechanisms in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pages 22199-22211 Curran Associates, Inc.

---

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag. Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. 2022. Solving quantita- tive reasoning problems with language models.
Judea Pearl. 2001. Direct and indirect effects. In UAI '01: Proceedings of the 17th Conference in Uncer- tainty in Artificial Intelligence, University of Wash- ington, Seattle, Washington, USA, August 2-5, 2001, pages 411-420. Morgan Kaufmann.
Judea Pearl. 2009. Causality. Cambridge University Press.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg, 2023. Inference-time intervention: Eliciting truthful answers from lan- guage model.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)), pages 2463 2473,, Hong Kong, China, Association for Computational Linguistics.
Tiedong Liu and Bryan Kian Hsiang Low. 2023. Goat: Fine-tuned LLaMA outperforms GPT-4 on arithmetic tasks. arXiv preprint arXiv:2305.14201.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual asso- ciations in GPT. Advances in Neural Information Processing Systems, 35: 17399-1372.
Piotr Piękos, Mateusz Malinowski, and Henryk Michalewski. 2021. Mcasuring and improving BERT's s mathematical abilities by predicting the or- der of reasoning. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 383-394, Online, Association for Computational Linguistics.
Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. 2022. NumGLUE: A suite of funda- mental yet challenging mathematical reasoning tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Pupers), pages 3505-3523, Dublin, Ireland. Association for Computational Linguistics.
Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt, 2023, Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.052/1
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022, pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. 2020. Zoom in: An introduction to circuits. Distill. http://diiiillllbb/222222eeeiian
Tilman Räuker, Anson Ho. Stephen Casper, and Dylan Hadfield-Menell. 2023. Toward transparent Al: A survey interpreting the inner structures of deep neural networks.
Chris Olah, Alexander Mordvintsev, and Ludwig Schu- bert. 2017, Feature visualization. Distill, 2(11):e7.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexan- der Mordvintsev. 2018. The building blocks of inter- pretability. Distill, 303)
Noam Shazeer and Mitchell Stemn 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. pages 1596-4604, PMLR.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Con- erly, Dawn Drain, Deep Ganguli, Zac Seafield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jack- son Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei. Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlist and Chris Olah. 2022. In-context learning and induction heads, Transformer Circuits Thread.
Daniel Spokoyny, Ivan Lee, Zhao Jin, and Taylor Berg- Kirkpatrick. 2022. Masked measurement prediction: Learning to jointly predict quantities and units from textual context. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 17- 29, Seattle, United States. Association for Computa- tional Linguistics.
OpenAI. 2023. GPT-4 technical report.
Alessandro Stolfo, Zhijing Jin, Kumar Shridhar, Bern- hard Schoelkopf, and Mrinmaya Sachan. 2023. A causal framework to quantify the robustness of mathe- matical reasoning with language models. In Proceed- ings of the st Annual Meeting of the Association for Computational Linguistics \Volume 1: Long Pa- pers), pages 545-561. Toronto, Canada, Association for Computational Linguistics.
Kuntal Kumar Pal and Chitta Baral. 2021. Investigating numeracy learning ability of a text-to-text transfer model. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 3095-3101, Punta Cana, Dominican Republic. Association for Computational Linguistics.

---

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2022. Roformer: En- hanced transformer with rotary position embedding.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397.
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances Neural Information Processing Systems, 33:12388 12401.
Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. 2021. Visualizing weights. Distill, 6(2):e00024-007
Ben Wang and Aran Komatsuzaki. 2021. GPT- J-6B: A billion parameter autoregressive lan- guage model. https:://github.com//ktt cobb.coom/inggolllz/
mesh-transformer-jaxx
Kevin Ro Wang. Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. 2022. Inter- pretability in the wild: circuit for indirect object identification in GPT-2 small. In NeurIPS ML Safety Workshop.
Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022a. Emergent abilities of large language models. arXiv preprint arXiv:2206.0766
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning large language models. CoRR, abs/2201.11903.
Chengrun Yang, Xuezhi Wang. Yifeng Lu, Hanxiao Liu, Quoe V Le, Denny Zhou, and Xinyun Chen. 2023.
Large language models as optimizers. arXiv preprint arXiv:2309.03409.

---

the MLP Modules (Operator Information)
Addition
Subtraction
10a
Farly
3000
10
Operator
2000
10
Operands
1000
10
Late
10
Multipliation
Division
10-3
300
15
20
25
Layer
IE of the Attention (Operator information) onl
10
Earl
10
Operater
10
10
20
Opreands
Layer
Layer
10
Late
Figure 9: Indirect effect of the MLPs at the last token each layer GPT-J, each of the four arithmetic operators. We observe a peak in the effect at layer 19 for all four types of operation.
Last
25 10
5
10 15 20
Layer
Figure 8: Indirect effect (IE) measured GPT-J when varying the word describing the operator involved in the input query. Similar to the operands case, we observe high contribution produced by middle-to-late MLP modules at the end of the input sequence.
the heatmap visualizations of these results for two- operand queries. Similar to the operand-related information, we observe high effect in three acti- vation locations: early ML.P blocks corresponding to the operand tokens; middle-to- co-rllly attention modules at the last token; and middle-to-late MLP modules at the last token. These results align with our hypothesis that arithmetic-related information is transferred to the end of the sequence by the attention mechanism, where it is then processed by late MLP layers. In this setting, we measure RI(M =31.4%.
A Prompt Templates
Tables 2 and 3, we report the question tem- plates from Karpas et al. (2022), which we used as prompts for the model for two- and three-operand queries, respectively. For threc-operand queries, we use one query template for each of the 29 possi- ble two-operation combinations.
Performance of the Models
D Effects for Each Operator
Table 4, we report the accuracy of the models on the arithmetic queries that we use for our analy- ses. The higher accuracy obtained using numeral words is likely given by the smaller set of possi- ble solutions considered (we used S - "one", "Iwo",... "[wenty"). as the numeral words corre- sponding larger numbers get split into multiple tokens by the tokenizer). The accuracy of GPT-J on the factual queries from the LAMA benchmark is 65.0% (we constrain the vocabulary to the set of all possible objects for all the relations considered). On the synthetic number retrieval task, GPT-J's accuracy is 86.7%.
For each of the four operators, report the indi- rect effect measured for the last-token MLP mod- ules in GPT-J in Figure 9. The results for each of the four operators show a common spike in the ef- fect at layers 19-20. This indicates the presence a specific part of the model relevant to the numerical predictions of the bi-variate arithmetic questions, irrespective of the operator involved. We also no- tice difference in the magnitude of the effects, which is linked to the capability of the model correctly answer the query.
E Changes in the Model's Prediction
Flow of Operator-related Information
We measured the influence of the model compo- nents in terms of probability changes. Now, we study the dynamics of the actual model predictions. In particular, considering the scenario in which r = r'. we verify whether the intervention leads to a change in the model's prediction. That is, we
The measurements of the indirect effect for each model component when fixing the operand and varying the operator in the two input prompts P1 and p2 reveal how the model processes the informa- tion related to the operator. We report in Figure

---

Type
addition
subtraction
Q: How much is va plus nz? A:
Q: How much is nu minus nz? A:
What is va plus n2? A:
What is ve minus rı2? A:
What is the result of plus ma?
What is the result
minus nt??
Q: What is the of my and ng? A:
What is the difference between mu and na? A: The difference between my and n2 is
The sum of and m2
ma n2
01 m22
multiplication
division
Q: How much is ny times ny? A:
Q: How much is Dy over over A:
Q: What is ya times rız? A:
Q: What is wj over ruz? A:
Q: What is the result of ny times ng? A:
Q: What is the result of nı over Ri?? A:
Q: What is the product of ni and riz? A:
Q: What is the ratio between and riz? ? A:
The product of and ng is
The ratio of nu and m2 is
ri2
/
Table 2: Question templates for two-operand arithmetic queries.
Formula
Template
(ye trrtty
Sum vo and and multiply by nts
ni Prttng fy
What the sum of and the product of na and nt??
What the product of minus and ns?
mi/(r//na)
How much
divided by the ratio between ne and ns?
mi-rig Pros
What the difference between ni and the product of me and mas?
my "(ng-1a)
How much times the difference between n2 and ns?
Table 3: Examples templates of three-operand queries. For the list, we refer Karpas et al. (2022).
compute
Model
Operation
Accuracy (%)
69.3
1 {arg max (T( # arg max P(x)),
(4)
78.0
EES
xE$
GPT-J
82.8
40.8
Overall
67.8
distinguishing
between
desired
r) and undesired (arg max Ppx)) r) changes. The sults reported in Figure show an increase the desired change in prediction at layers 19-20, while the undesired change in prediction is higher for layers 14-17. This means that interventions on the MLPs at layers 19-20 are more likely to lead to correct adjustment of the prediction, while the opposite is true for earlier layers (14-15 in particular). This finding consistent with our previous observations and we see this as additional evidence highlighting the influence of the MLPs at layers 19-20 on the prediction of r.
(arg max,es (x
95.5
86.7
GPT-J (Numeral Words)
59.7
Overall
81.3
57.4
77.5
Pythia 2.8B
64.7
40.2
Overall
59.9
+
100.0
99.8
LLaMA
X
100.0
+
88.7
Overall
97.2
100.0
100.0
Goat
91.4
F Fine-tuning Details
54.0
Overall
85.6
We fine-tune Pythia 2.8B on three-operand queries. We train the model for epuchs set of queries obtained by sampling 1000 triples of operands for each of the 29 templates. We use Adafactor (Shazeer and Stern, 2018) a learning rate of 10 linearly decaying, and a batch size of 8. We make sure that there is no overlap between the set
Pythia 2.8B (3 Operands)
Overall
0.9
Pythia 2.8B Fine-tuned Operands)
Overall
39.7
Table 4: Accuracy the models analyzed in the paper various types of arithmetic queries.

---

Prediction Change
Change
Early
IE of the Altention (Factual Q.)
10a
0.075
Subj First Tok
10a
Other Toks
Subj Last lok
101
10
35
20
25
Layer
Last
10
% of Precicion Change
Undesired Change Prediction
10
15 Layer
20
25
0.10
Figure 12: Indirect effect (IE) measured for the attention
0.05
modules in GPT-J on factual knowledge queries.
Layer 20
25
H Factual Knowledge Data
Figure 10: Desired (wrong to correct) and undesired (correct wrong) change the prediction induced by the intervention on the MLPs in GPT-J. The layers at which two types prediction change peak corre- spond to the layers with the largest corresponding IE.
For the experiments involving the prediction fac- tual knowledge, we use the following six relations from the T-REx subset of the LAMA benchmark (Petroni et al., 2019):
"[subject] is the capital of [object]]
IE of the Attention (Number Retrieval)
2. "[subject] born in [object]"
Earty
% 1st Tok
10a
3. "[subject] died in [object]"
6, Mid Toks
Tok
10e
4. "The native language of [subject] is [object]"
Late
Last
10-1
"[subject] is subclass [object]"
10
15 Layer
25
6. "The capital of [subject] is [object]".
Figure 11: Indirect effect (IE) measured for the attention modules in GPT-J the task of number retrieval.
We sample pairs of subject entities from the set of entities compatible for a given relation (e.g.. cities for the relation "is the capital of"). For each relation, sample 100 pairs of subject entities.
queries used for training and the set used for the computation of the indirect effect.
Log Probability to Quantify the IE
In order to validate whether the measurements of the indirect effect are specific to the metric that we describe in Equation 2, we quantify the IE using the absolute difference in the log of the probability values assigned by the model the results and r'. More formally, we compute
G Computing Infrastructure
The experiments for models are carried out us- ing a single Nvidia A100 GPU with 80GB of mem- ory. The computation of the indirect effect across the whole model for a single type of component (attention or MLP) took 15 hours for GPT-J and ~6 hours for Pythia (using 50 examples for each two-operand template) and ~ hours for LLaMA and Goat (using 20 examples for each two-operand template). For the fine-tuning of Pythia 2.8B, we used a single Nvidia A100 GPU with 80GB of memory. The training procedure took ~ hour. Ex- periment tracking was carried out using Weights & 8 8
A + A
rr
(Fatt(2)
(5)
A
otherwise
where
A = log (r) - log (P(r)
(6)
A log P(r') log P (r').
(7)
The results are reported in Figure 13. The activa- tion sites that we observe are the same as reported in Section 4.1: first-layer MLP at the operand to- kens and last-token MLP and attention modules.
"http://wandb. .i

---

Far of the MIP Modules
[Fur the Attuntion
Eafly
Early
Operand
Operand
-
2nd Operand
2nd Operand
Late
Late
-2
Last
Last
10
15
20
25
15
20
Layer
Layer
MLP Modules Fixed)
Attention (Result
Early
1st Operand
1st Operand
Middle
Middle
Operand
Operand
Layer
Layer
Figure 13: Indirect effect measured using the difference in the log probability described in Equation 5 (IEalt).
The results are obtained with GPT-J on two-operand arithmetic queries.
(MLPs) Pythia (FT) Fixed)
(Attn.) Pythia (FT) (Res. Fixed)
Early
10
Early
10
1st Operand
1st Operand
Between Operands 1
10a
Between Operands 1
10a
2nd Operand
I
2nd Operand
10s
Between Operands 2 Operand
Between Operands 2
Operand
10a
10a2
10-1
110-1
Layer
Layer
Figure 14: Indirect effect (IE) measured for the attention modules in the fine-tuned version of Pythia 2.8B three-operand arithmetic queries, when r= =r'
J Additional Information Flow Visualizations
We include the IE measurements for the attention modules of GPT-J on the number retrieval task (Figure 11) and the factual knowledge queries (Figure 12), and for Pythia 2.8B three-operand arithmetic queries before and after fine-tuning (Fig- ure 15). Additionally, we report the heatmap visu- alizations of the indirect effect measured for the fol- lowing models: Pythia 2.8B (Figure 16), LLaMA 7B (Figure 17), Goat (Figure 18), and GPT-J using word numerals (Figure 19). Finally, we visualize in Figure 14 the IE of MLPs and attention modules for the fine-tuned Pythia 2.8B in the fixed-result case.

---

IE of the Att, Pythia 2.80
IE the Att. in Pythia 2.88 (FT)
Early
Early
10
10a
1st Operand
1st Operand
Between Operands 1
Between Operands
2nd Operand
10a
2nd Operand
10a
Between Operands 2 3rd Operand
Between Operands 2
10
3rd Operand
101
Late
Late
Last
Last
10-1
100
0
5
10
15 Layer
20
30
0
5
25
30
Layer
Figure 15: Indirect effect (IE) measured for the attention modules in Pythia 2.8B on three-operand arithmetic queries, before and after fine-tuning.
12 the MiPa
Operand
Middl
Opewed
Late
164
237
10
Recd)
the Altention (Benult (loed)
Cary
Oiperand
Oozrand
20
2rd Oper
22
Late
201
last
15
Lager
Figure 16: Indirect effect (IB) measured in the MLP and attention modules of Pythia 2.8B on two-operand arithmetic queries.
Cfftte MIP
301
the Attertionn
Operand
100
Lite
15 Laysr
of ML.7% (ftest: Rlodd)
Allention
Ealy
14t Oporand
operradd
Midds
Vidde
2re Doorand
2rd Operand
Lxte
10P
Last
Layer
Figure 17: Indirect effect (IF) measured in the ML. and attention modules of I.LaMA 7B two-operand arithmetic queries.

---

the Aftertionn
Eanly
Operadd
Middle
100
137
Lacs
Lete
35
15
Layper
Layor
the MLD' (ftasll frxed)
the Allention (ltsall floed)
bulf
100
13
Opound
Oporand
Mitee
Vidde
2vd Openaed
Opernnd
Last
Lest
15
Figure 18: Indirect effect (IE) measured in the MLP and attention modules of Goat two-operand arithmetic queries.
the HUP Modules
IE the Aftondoon
1st Opound
1st Ope and
21d Opennd
2rd Operand
30
25
23
10
35
20
L2y
IE the MLP Moddlles (hesult
of the Attention (Beact: (toed)
Operad
Mistle
Lacs
Lets
15
Layer
Layer
Figure 19: Indirect effect (IE) measured in the MLP and attention modules of GPTJ on two-operand arithmetic queries, using numeral words to represent quantities.


Reviews:
[{"review_id": "oyeKsE0iqy", "reviewer": "Reviewer_yvrj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out that our study “effectively traces information flow and pinpoints key components” involved in arithmetic reasoning in LMs. Also, thank you for recognizing that our work “promises an enhanced understanding of LMs' mathematical capabilities” and is relevant to the NLP community.\n\n> Interpretability, while crucial, is just one aspect of model research. \n\nThe review motivates the low score with the point that interpretability research is not valuable without any proposal for model improvement.\nWe believe that it is important to acknowledge that understanding the inner workings of models is a crucial step toward informed decision-making in any model-related endeavor. Interpretability serves as a vital tool for identifying potential strengths, limitations, and areas for enhancement in LMs. As pointed out by another reviewer, similar interpretability studies (Meng et al., 2022; Nanda et al., 2023) yield useful applications further down the line.\n\nMoreover, the review assigns a reproducibility score of 2. We find ourselves a little perplexed as we provided, through the submission platform, the code with the instructions to reproduce all our experiments, as well as notebooks to generate the visualizations included in the paper. We would greatly appreciate  revision of such a score, or otherwise a clarification about any missing details for reproduction.\n\n\n[1] Nanda, N., et al., 2022, September. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations.    \n[2] Meng, K., et al., 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, pp.17359-17372.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Its strengths lie in its mechanistic interpretation, utilizing a causal mediation analysis framework to understand LMs' inner workings. By intervening on specific model activations, the study effectively traces information flow and pinpoints key components, such as attention mechanisms and MLP modules, that influence arithmetic predictions. For the NLP community, this paper promises an enhanced understanding of LMs' mathematical capabilities. The presentation of the paper has good standards.", "reasons_to_reject": "While the paper provides a deep dive into how LMs handle mathematical reasoning, it stops short of offering tangible methods to adjust or improve the LMs based on the gleaned insights. This leaves a gap between understanding and application. Interpretability, while crucial, is just one aspect of model research. By intensely focusing on it, the study might not address other pressing issues related to model efficiency, robustness, or scalability.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the topic of mathematical reasoning in large language models, specifically how Transformer-based models process arithmetic tasks. Using a causal mediation analysis framework, the research identifies key model components responsible for arithmetic predictions, revealing that information is transmitted from mid-sequence layers to the final token via attention mechanisms and then processed by MLP modules before being integrated into the residual stream.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "swKWoXciYf", "reviewer": "Reviewer_LE67", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing that our work offers “interesting insights” into arithmetic reasoning in LMs obtained “by conducting an extensive evaluation”. Also, thank you for your presentation suggestions, we will incorporate them in the final version of the paper.\n\n> The paper focuses on a small set of arithmetic operations.\n\nThe scope of our paper is investigating arithmetic reasoning and we experiment with the four fundamental arithmetic operators. Addition, subtraction, multiplication, and division form the cornerstone of arithmetic calculations and serve as the basis for a wide range of mathematical computations. Thus, exploring their mechanisms in language models provides a starting point to explore more complex forms of mathematical processing. As a first study in this direction, our work lays the foundation that can inform subsequent investigations into a broader set of operations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well-written, very clear, and concise.\n* Provides empirical evidence supporting the proposed hypothesis by conducting an extensive evaluation on LMs in 4 different scales.\n* Offer interesting insights into how arithmetic reasoning is conducted in Transformer based LMs.", "reasons_to_reject": "* Limited Scope: The paper focuses on a small set of arithmetic operations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper uses causal mediation analysis to examine how Transformer-based LMs process arithmetic tasks. By intervening on specific model parts, they found that LMs transfer arithmetic-relevant information from mid-sequence early layers to the final token, which is then processed by late MLP modules. They supported this finding by experimenting on autoregressive LMs in different scales and highlighting their information flow's specificity to arithmetic tasks.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* It would be helpful to add additional details in the intro. For example lines 77-80 and 86-90 (what are the additional tasks). \n* Fig 2. The \"before and after intervention\" plots are too small to read. (same for Fig 3)"}}, {"review_id": "5KCdEaXvIC", "reviewer": "Reviewer_WmZT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and for recognizing the novelty of our study.\n\n> **(1)** The authors appear to implicitly assume that arithmetic reasoning in LM-based models operates like human-like reasoning. However, **(2)** it is currently unclear whether LMs actually possess reasoning abilities.\n\n**Re. (1)**, we do not make any assumptions about any human-like behavior of the models. In fact, in the paper, we refrain from drawing any parallel between human reasoning and model mechanics: we analyze the information flow through the model, and indicate which components are responsible for different types of information. To avoid possible future misunderstandings, we will explicitly mention that we do not make such assumptions.    \n**Re. (2)**, we are not making any claims about the reasoning capabilities of LMs. We are observing that the models we consider are performing the task of answering arithmetic queries with reasonable accuracy (60+%) and we are investigating how the models process the input information to perform such a task.\n\n\n> **(3)** It is unclear what the ‘human-understandable component’ mentioned in the mechanistic interpretability refers to, and **(4)** there is no argument presented as to how the causal mediation analysis framework proposed by the authors can be applied and appropriate to LMs.\n\n**Re. (3)**, our work studies the information flow within the model and the contribution of the different model components: we observe that the input information is conveyed by the attention to the end of the input sequence, where the information is processed by a specific subset of MLP modules, which output result-related information (Fig. 1). We believe that this procedure represents a human-understandable mechanism.    \n**Re. (4)**, approaches based on causal mediation analysis have been applied and shown to be appropriate to interpret language models in multiple previous works [1, 2, 3, 4]. Our framework relies on the principle that, by intervening on a particular mediator (a set of neurons) in a causal graph (a model), we can isolate the causal influence of that mediator on the outcome variable (the model’s prediction). Building our analyses on this logically-sound procedure, we study the causal influence of different model components.\n\n\n> The authors’ research is limited to a restricted level of arithmetic operations and does not cover the entire domain of arithmetic.\n\nArithmetic consists of the study of the properties of the traditional operations on numbers—addition, subtraction, multiplication, division, exponentiation, and extraction of roots. Among these, we experiment with the four fundamental binary operators. Addition, subtraction, multiplication, and division form the cornerstone of arithmetic calculations and serve as the basis for a wide range of mathematical computations. Therefore, exploring their mechanisms in language models provides a starting point to explore more complex forms of mathematical processing. As an initial study in this direction, our work represents the groundwork that can inform subsequent investigations into a broader set of operations.\n\n[1] Finlayson, M., et al., 2021, August. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).    \n[2] Meng, K., et al., 2022. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, pp.17359-17372.    \n[3] Vig, J., et al., 2020. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33, pp.12388-12401.   \n[4] Wang, K.R., et al., 2022, September. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- As the authors mention, this study has novelty in attempting to connect the area of mechanistic interpretability with the mathematical reasoning ability of transformer-based LMs.", "reasons_to_reject": "- Claiming that the discovery of changes in probability and differences in subsets of parameters is the discovery of the reasoning mechanism of LMs seems somewhat of a logical leap. The authors appear to implicitly assume that arithmetic reasoning in LM-based models operates like human-like reasoning. However, it is currently unclear whether LMs actually possess reasoning abilities. Therefore, there may be various interpretations as to what the changes that the authors have discovered actually mean.\n\n- It is unclear what the ‘human-understandable component’ mentioned in the mechanistic interpretability refers to, and there is no argument presented as to how the causal mediation analysis framework proposed by the authors can be applied and appropriate to LMs. Therefore, it is difficult to verify the validity of the claim based on the given text alone.\n\n- Also, the authors’ research is limited to a restricted level of arithmetic operations and does not cover the entire domain of arithmetic. In this situation, the claim that the mechanism of arithmetic reasoning has been discovered seems somewhat bold.", "questions_to_authors": "", "paper_topic_and_main_contributions": "- The authors applied a ‘causal mediation analysis framework’ to the arithmetic reasoning of LMs and performed a mechanistic interpretation.\n- The authors propose that they have discovered the mechanism by which LMs work in arithmetic tasks by intervening on specific model components, measuring changes in predicted probabilities, and discovering subsets of parameters related to specific predictions.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AWfvc2og6I", "reviewer": "Reviewer_jY6n", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our work to be “an excellent application of causal analysis for NLP” that provides “a detailed picture of how the model components work together to produce an answer”. We are encouraged by your statement “While there may not be an obvious immediate application for these findings, they are interesting enough to stand on their own”. Your review is greatly appreciated.\n\n> My biggest concern is an issue that plagues CMA generally, which is over-interpreting results.\n\nAs the review correctly points out, some components are expected to have a large effect in mediating changes in specific parts of the input (e.g., the very first MLP layer in correspondence to the tokens that vary) and some components are expected to have no effect (e.g., modules that correspond to early tokens in the sequence). However, the statement “there will always be a region in the components associated with the ‘last’ token with high indirect effects whenever $r \\neq r'$, since all causal paths must necessarily pass through some component in this layer” is not necessarily true. Late Transformer blocks at the end of the sequence are indeed likely to output information that has a high effect on the next token prediction (the last block at the last token is the only component through which all causal paths pass). However, this is the case for the output of the *whole* Transformer block (residual stream + attention output + MLP output). In our analyses, we are considering the output of MLPs and attention modules, which can be seen as \"new information\" being incorporated into the residual stream. This new information can be produced at any point of the sequence and then conveyed at the end of the sequence for the prediction of the next token.\n\nIn other words, an alternative hypothesis for how the model processes the input to answer arithmetic queries is that information related to the result is computed by MLPs at different points during the input sequence (e.g., in correspondence to the second operand), and it is conveyed to the end of the sequence by the attention mechanism. For instance, for factual queries, we observed that the MLPs that have the largest effect are mid-sequence (corresponding to the subject tokens). In the discussion of these results, our goal was to point out that this effect pattern/information flow differs from the one observed for arithmetic queries, where the mid-late MLP layers have a larger influence on the model’s prediction.\nOur experiments show that, while some features of the effect patterns measured are predictable due to the nature of our experimental procedure (e.g., high effect in the first MLP layers corresponding to the modified tokens) the models display an information flow pattern that is specific to the task of answering arithmetic queries.\n\nIn conclusion, we believe that your concern about the inevitability of some patterns in the effects measured is reasonable. We hope that our response clarifies what are other possible features in the information flow that we are measuring (i.e., what are possible alternative hypotheses), and we commit to making these considerations explicit and more clear in the discussion of all our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper provides interesting clues into how language models perform arithmetic reasoning. I find this to be an excellent application of causal analysis for NLP that both supports existing theories about how neural LMs work and provides a detailed picture of how the model components work together to produce an answer.  While there may not be an obvious immediate application for these findings, they are interesting enough to stand on their own. Furthermore, as Meng et al. (2022) show, causal mediation analysis (CMA) findings can yield useful applications further down the line.", "reasons_to_reject": "My biggest concern is an issue that plagues CMA generally, which is over-interpreting results. This can come from assigning meaning to random patterns, or not properly considering counterfactual patterns, whether patterns are inevitable, and what exactly we can conclude from existing patterns. In my opinion, the authors do not sufficiently address some inevitabilities in the patterns. For instance, the model components associated with the \"early\" tokens will never have indirect effects because (as far as I can tell) there is no causal path from the intervention to the response variable that passes through these nodes. Similarly, the input MLPs for the \"operand\" tokens will always have high indirect effects because all causal paths must pass through these nodes. Lastly, there will always be a region in the components associated with the \"last\" token with high indirect effects whenever $r\\neq r'$, since all causal paths must necessarily pass through some component in this layer. Given this a priori description of the effect patterns, it is hard to imagine the effect results looking much different from the way they do. To make this paper stronger, for each finding about the model's mechanism for arithmetic reasoning, the authors need to make a convincing case that there is an alternative hypothesis for the model's mechanism that would be supported by an alternative pattern that the authors can describe explicitly but does not show up in their experiments. The authors actually do an excellent job of this in section 4.2, but not so much in the other sections. \n\nIn conclusion, the authors should include an a priori description of what the patterns must look like in order to ground their actual results, and be explicit about what the alternatives to their findings would be. Including these would bring this paper to the next level in my opinion.", "questions_to_authors": "None", "paper_topic_and_main_contributions": "The authors of this paper use causal analysis to locate components responsible for arithmetic reasoning in language models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a mechanistic interpretation of Transformer-based LMs on arithmetic-based questions using a causal mediation analysis. They identify the subset of parameters responsible for specific predictions. 

The main contribution of the paper is the causal mediation analysis (CMA) on arithmetic. 

In pros, reviewers praise that the paper is well written, provides a better understanding of the mathematical capabilities of transformers / interesting insights, with findings that are "interesting enough to stand on their own".

The main cons / weakness is the limited applicability of an analysis only performed on arithmetic / the limited scope, and that this does not lead to any insights to improve transformers with. There are a few worries on that the claims are a little too bold, which could be adjusted by the authors should the paper be accepted.

All in all, the reviewers were (almost) unanimous in that the paper is strong in soundness (3/4) and strong in excitement (3/4), and the only (weak) reason to reject it would be its limited scope to arithmetic.
Decision: Accept-Main