Arithmetic Without Algorithms: Language Models Solve Math with a Bag of Heuristics
Authors: Yaniv Nikankin
Anja Reusch
Aaron Mueller
Yonatan Belinkov
Venue: ICLR 2025 Poster (2025)
Keywords: mechanistic interpretability
interpretability
arithmetic
nlp
language models
Primary area: interpretability and explainable AI

Abstract: Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types---such as neurons that activate when an operand falls within a certain range---and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a ``bag of heuristics''.

Published as a conference paper at ICLR 2025
ARITHMETIC WITHOUT ALGORITHMS: LANGUAGE MODELS SOLVE MATH WITH A BAG OF HEURISTICS
Yaniv Nikankin Anja Reusch
Aaron Mueller
Yonatan Belinkov
Techhnion Israel Institute of Technology
NNottheastern University
ABSTRACT
Do large language models (LLMs) solve reasoning tasks by learning robust gener- alizable algorithms, or do they memorize training data? To investigate this ques- tion, we use arithmetic reasoning as representative task. Using causal analysis. we identify subset of the model (a circuit) that explains most of the model's be- havior for basic arithmetic logic and examine its functionality. By zooming on the level of individual circuit neurons, we discover sparse set of important neu- rons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons the mechanism used to produce correct arithmetic an- swers. To test this, we categorize each neuron into several heuristic types- poppe--such neurons that activate when an operand falls within a certain range-and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's 's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely "bag of heuristics"
INTRODUCTION
Do large language models (LLMs) implement robust reusable algorithms to solve tasks, or are they merely memorizing aspects of the training distribution? This distinction is crucial (Tänzer al.) 2022 Henighan et al. (023): while memorization might suffice for limited problem sets, true algo- rithmic comprehension allows for generalization and efficient scaling new problems.
Arithmetic reasoning provides lens for this investigation, as it can be solved using various methods: learning known algorithms, developing novel approaches, or by memorizing vast quantities of input- output pairs. Thus, we ask the following: Do LLMs implement robust algorithms to correctly complete arithmetic prompts, similar children learning vertical addition to add two numbers, or do LLMs merely memorize the arithmetic prompts that appear in their vast training data?
Previous studies have made progress identifying arithmetic mechanisms LLMs. [Stofo et al. (2023) and IZhan et al. (2024) have identified a subset of model components (a circuil) respon- sible for arithmetic caleulations in several L.MMs and characterized the information flow between them. Zhou et al. 2224)) suggested that pre-trained LLMs use features in Fourier space to accurately answer addition prompts. However, SStolo et al. (2023) and Zhang et al. (2024) stopped short of clu- cidating the mechanism implemented by the circuit they identified the trade-off between generalization and memorization in this task. Zhoo et al.] (2024) studied ad- dition prompts in models that were fine-tuned arithmetic data. Their findings regarding Fourier features are significant, but we claim these represent only a part of a more complex mechanism. Our work aims to bridge these gaps: we investigate how the arithmetic eircuit works qualitatively- and specifically, whether it implements mathematical algorithm or memorizes arithmetic training data.
required feat to understand
To do so, we reverse-engineer the arithmetic mechanism applied by LLMs. We use causal analysis to examine their arithmetic circuits, focusing on individual neurons within the circuit responsible
"Correspondence www.v.nnee..tecehioooooo httppp///giiiibb.c////eeetiiiiie
thmet c- www.herrsst..

---

Published as a conference paper at ICLR 2025
Atfention
MLP
Input Tokens
Liama3 8B
Oulput Logits
226
68
158
160
17
4942
[5,25] mad 50
24
12439
[150, 180)
30
1582
mod
First Operand op:
First Operand ops
First Operand op:
17
692
200
200
200
Layer
Neuron
100
100
100
Number
100
Second Operand op2
(b)
Second Operand op,
(c)
Second Operand op2
Figure 1: Bag of heuristics visualization. We show that transformer LLMs solve arithmetic prompts by combining several unrelated heuristics, each activating according rules based on the input values of operands, and boosting the logits of corresponding result tokens. These heuristics are manifested in single MI.P neurons mid to late layers.
for generating the correct answer. Our analysis reveals that a sparse subset of neurons sufficient for accurate responses, with each neuron implementing distinct heuristic. Each heuristic fires for a specific pattern in input operands or in their combination, and some increase the logits of relevant result tokens accordingly. For instance, one heuristic (Figure [T)) increases logits of tokens between 150 and 180 for subtraction prompts whose answer falls within this range. By examining these neurons, we classify each into one or more heuristic types. For example, the mentioned neuron (Figure (p) falls within the type of result-range heuristics, which promote continuous value ranges. We discover that successful prompt completion relies combination of several unrelated heuristic types, forming "bag of heuristics" approach. This finding suggests that LLMs may not be em- ploying a single, cohesive algorithm for arithmetic reasoning, nor are they memorizing all possible inputs and outputs; rather, they deploy collection of simpler rules and patterns.
We investigate if the bag of heuristics emerges as the primary arithmetic mechanism from the onset of training, or whether it overrides an earlier mechanism. To do that, analyze how the heuristics evolve over the course of training. We show arithmetic heuristics appear throughout the model training, gradually converging towards the heuristics observed in the final checkpoint. Furthermore, we provide evidence that the bag heuristics mechanism explains most the model's behavior even in early stages, indicating it the main mechanism used for solving arithmetic prompts.
We contribute by providing a high-resolution understanding of the mechanism that L.MSs use to an- arithmetic prompts. We (i) show pre-trained LLMs implement "bag of heuristics" approach, (ii) investigate when and why this mechanism fails to generalize, and (iii) discover how it emerges across training. This allows us better understand the source of current capabilities and limitations of LLMs in arithmetic reasoning finding that could apply to additional reasoning tasks.
2 ARITHMETIC CIRCUIT DISCOVERY
In transformer-based LLMs, a circuit (Elhage 2021 refers minimal subset of inter- connected model components (multi-layer perceptrons (MLP) or attention heads) that perform the computations required for a specific task. We locate. and later analyze, the circuit responsible for arithmetic calculations.
2.1 CIRCUIT DISCOVERY AND EVALUATION
Models and Data We analyze four LLMs: Llama3-8B/70B (Dubey et al. 2024). Pythia-6.9p
Biderman et al. 2023). and GPT-J (Wang & Komatsuzaki (021). For each, we locate and analyze the cireuit responsible for arithmetic caleulations. We focus Llama3-8B in the main paper and report similar results for the additional models in Appendix [[ We use pre-trained models without fine-tuning them arithmetic prompts, goal is to uncover the mechanisms induced by typical
2

---

Published as a conference paper at ICLR 2025
Effect Map Per Attn Head MLP
log scale
226
68
Attention Heads (Last Position)
MLPs
Early MLPs (0 to 15)
8
Layer
10
6
operator Heads (L2H12..)
oByy (L1GH25.
opt Heads (L15H13,
20
4
2
30
10
20
30
0 2 Position
0
Late MLPs
(16 31)
Attention Head
(a)
(b)
Figure 2: Llama3-8B arithmetic circuit discovery results. (a): Few attention heads have a high effect on arithmetic prompts. Most MLPs take part in the computation. The first MLP noticeably affects operand and operator positions, while mid- and late-layer MLPs influence the final position. (b): The arithmetic circuit in Llama3-8B. The attention heads project token information to the last position, where the middle- and late-layer MLPs promote the logits for the correct answer.
language model training. Each model tokenizes positive numbers as single token, up to some limit; e.g., in Llama3-8B, numbers in [0, 1000] are tokenized to single token. To locate the arithmetic circuit, we use two-operand arithmetic prompts with Arabic numerals and the four basic operators (+,,,x,+) such that each prompt has four tokens: opl, the operator, op2, and the "=" sign. We sample a list of 100 prompts per operator for circuit discovery, and an identical amount for evaluation. Each prompt is chosen that both its operands and result will be tokenized to a single token; e.g., in Llama3-8B, the operands and result must be between 0 and 1, 000. Unlike previous studies (Stolfo et al. 2023), we do not use in-context prompting, ensure the circuit does not include any component not directly linked to arithmetic calculations. To reduce noise and ensure the circuits only contain components responsible for correct arithmetic completions. prompts that are correctly completed by the model, similar to previous studies (Wang et al., 2022] Prakash al. 2024). Throughout the paper, the prompt "226 68 =" is used as running example.
only use
Method To locate the circuit components, we conduct series of activation patching experiments (Vig et al. (020) that allow us to assess the importance of each MLP and attention head at each sequence position. Each experiment involves sampling a prompt p with result r from the dataset (for example, "226 - 68 ="), and sampling random counterfactual prompt p' that leads to different result r' (for example, "21 + 17 ="). After pre-computing the activations of the model for the counterfactual prompt p', we introduce the prompt p to the model. We intervene on ( ("patch") ") the computation, which means we replace the activation of a single MLP layer or attention head with its pre-computed activation for p'. Following Stolfo et al. (2023), observe how this intervention affects the probabilities of both answer tokens, 7' and r", by measuring the following:
1 PP (r' - P(') IP (r)) (P)
(r)
E(r,r')
(1)
2 P(r')
P
where IP and F* are the pre- and post-intervention probability distributions, respectively. The two summands in Equation (1) increase patching raises the probability of r' or decreases the probabil- ity rf respectively. High effect for an intervention on a component indicates its high importance in prompt calculation. The effect averaged across prompts and measured separately per component.
Results The patching results, shown in Figure 2a reveal that the MLP layers affect the output probabilities more than the attention heads. The first MI. affects the representation at the operator and operand positions (see Appendix [). while middle- and late-layer MI.Ps exhibit a strong effect at the final position, likely reflecting their role in predicting the answer token in that position (further discussed in Section 2.2 Figure 22 also shows that very few attention heads are important to the circuit. Each such attention head copies information from single position (either operand or operator position) to the final position (see Appendix [..2.. Figure 2bb summarizes the information flow within the circuit, consistent in structure with prior work (Stolto al. 2023).
To evaluate the circuit c, we measure its faithfulness (Wang et al.) 2022), the proportion of the full model's behavior on arithmetic prompts that can be explained solely by the circuit. To measure
3

---

Published as a conference paper at ICLR 2025
faithfulness, first pre-compute mean activations for each model component (in each position) across all arithmetic prompts. We then intervene on the evaluation prompts by replacing non-cireuit measure NL(c). the logit of
component activations with their means. To quantify performance, the correct answer token normalized by the maximal logit, as a proxy for accuracy, when mean- ablating all components not in the circuit The circuit's faithfulness is calculated as:
NL(c) NL(
F(c)
NL(M) - NL(0)
where M is the entire model and NL(M) is the normalized correct-answer logit when no component is ablated (always 1.0 for correctly completed prompts). NL(0) is the normalized correct answer logit when all components are mean-ablated. This formula normalizes faithfulness to a [0.0, 1.0) range.
The circuit achieves a high laithfulness of 0.96 on average across the four arithmetic operators; i.c., the circuit accounts for 96% of the entire model's performance. We therefore conclude that the components identified this section comprise the arithmetic circuit, and explain most of the model's accuracy for arithmetic prompts. See Appendix A for results across various circuit sizes, and a discussion of Pareto-optimality with respect to faithfulness and size.
2.2 IDENTIFYING ANSWER-PROOE PROMOTING COMPONENTS
To understand the mechanism implemented by the cir- cuit to promote the correct answer, we first search for the specific cireuit components that increase the probability of the correct answer. For this, we em- ploy linear probing (Belinkov) 2022). For each layer I and sequence position p. we train linear classifier fip Rd + R 1000 (where d is the dimension of each layer's output representation) using training of correctly completed arithmetic prompts. We pass these prompts through the model and calculate the output representation h E Id at layer and position for each prompt. The classifier fi.p receives hP input, and outputs probability distribution over the 1, 000 possible arithmetic answers. The classifier ft.p is evaluated on a separate test set of correctly completed prompts, showing to what extent the correct answer can be extracted from the output representation at layer and position
Position
Operand1
0.8
Operator Operand2
0.6
0.4
0
10
20
30
Layer
Figure 3: Answer token probe accuracy. The linear probes successful in extract- ing the correct answer from the final posi- tion, starting layer 16.
We find that the answer can only be extracted with high accuracy from the final position (Figure after the token representation is processed by the later layers of the model, starting from layer 16. Given that the arithmetic circuit contains only MLPs in these layers (Figure 2 this suggests that these MLPs in layers [16, 32] are the components that write the correct answer into the representation at the last position. The following section zooms into these middle- and late-layer MLPs, and presents evidence for the role they play in generating the correct answer- specifically, in how they promote the correct answer token through combination of many independent arithmetic heuristics.
MLP NEURONS IMPLEMENT ARITHMETIC HEURISTICS
3.1 DECOMPOSING CIRCUIT MLPS TO INDIVIDUAL NEURONS
Having shown that the model generates the arithmetic answer in middle- and late-layer MLPs at the final position p = 4, we zoom in on these MLPs and their calculations at this position investigate the implemented mechanism. The MLP at layer can be described by the following equation:
dmip
b = MLP (b(m) W! out 1oost wi
out
post V out
(3)
VL
where h/n host E Rd are the input and output representations of the MLP at layer I, respectively. booot E Rdmur is the output of the up-projection of the MLP, 3
where we define the nh value h/ E
The up-projection MLP... sn. is implemented differently each LLM we analyze. See Appendix C
4

---

Published as a conference paper at ICLR 2025
Individual MLP Neuron Intervention Effects
Faithfulness of using only top-k neurons
Intervention Effect
1
Falthfulness
0.05
0.5
0
0
5k
10k
1%
10%
Neuron Index
Neurons used Per Layer (%)
(a) We intervene on cach neuron in MLP layer 1 =
(b) We measure the faithfulness when including only fraction of high-effect neurons in the circuit. This circuit achieves high faithfulness.
17. Few neurons in this and other middle- and late-
layer MLPs have high effect on arithmetic prompts.
Figure 4: Analyzing effect of individual circuit MLP neurons. Our results demonstrate that small amount of neurons required to correctly predict the result.
R as neuron. W out Xd is the output projection matrix, and Vaut is its n row vector. Biases are omitted. By expressing the output representation h hout as a linear combination of row vectors out and their corresponding neuron activations h posee we can identify the neurons that most affect the completion of arithmetic prompts.
L. To measure the effect of each neuron h post: we perform activation patching experiments indi- vidual neurons (as described in Section 2.1). and measure the average effect across prompts. We observe that very few neurons have a high effect; an example for layer = 17 is shown in Figure Additionally, we notice the neurons with the highest effect are different between operators. In fact, roughly 45% of the important neurons for each operator are unique (Appendix D). Thus, when an- alyzing the circuit at the neuron level, we analyze it as 4 separate circuits-one one for each arithmetic operator. We hypothesize that for each operator, the highest-effes neurons are sufficient to explain most of the model's s arithmetic behavior. To verify this, we measure the faithfulness of the arithmetic circuit when mean ablating non-circuit components and lower effect MLP neurons middle- and late-layer MLPs. The results (Figure 4b) confirm that only 200 neurons (roughly 1.5%) per layer are needed to achieve high faithfulness and correctly compute arithmetic prompts.
3.2 MLP NEURONS ACT AS MEMORIZED HEURISTICS
To understand how the t00-200 middle- and late-layer important MLP neurons contribute to the gen- eration of correct answers. we view them as key-value memories (Geva et al. 2021 In this view, the input to each MIPP layer h',, is multiplied by key (a row vector in the MLP input weight matrix) to generate a neuron activation h 4. post that determines how strongly does a value (a row vector v in the MLP output weight matrix) gets written to the MLP output (Equation (3). Geva et al. (2021) out demonstrated that keys correspond to specific topics or n n-grams, triggering high neuron activations when these given input, and their corresponding values represent tokens that can propriate completions for these topics or n-grams. Building on this insight, we hypothesize that (i) in arithmetic contexts, keys correspond to numerical patterns, e.g., neuron might activate strongly when both operands in an arithmetic operation are odd numbers; and (ii) the associated value vectors encode numerical tokens that represent plausible answers the key patterns.
as ap-
To test the first hypothesis, we investigate the activation pattern of the top-200 neurons in each layer. For each neuron I, n., we plot the activations 4.n (at position p 4) as function of operand values, separately for each operator. We find that many neurons in the arithmetic circuit exhibit distinct, human-identifiabbe activation values for subtraction prompts with results between 1 150 and 180 (Figure [ Additional examples are provided in Appendix !
patterns. For instance, in "226 - 68 =", neuron h post
24.12439 shows high
To verify the second hypothesis, we check whether the tokens embedded in the value vectors of the top neurons relate to their activation patterns. Using the Logit Lens (nostalgebraist) (220). method of projecting a vector v E Rd onto a probability distribution over the vocabulary space pduccss, we project each value vector v J.n to find the numerical tokens whose logits are highest. This reveals
Not to be confused with the attention heads' keys and values.
5

---

Published as a conference paper at ICLR 2025
Op1 Range
Op2 Modulo
Op1 == Op2
Result Range Result Pattern
High
Operand
Low
Operand 2
Operand 2
Operand Operand 2
Operand 2
Figure 5: Heuristic pattern examples. Each heatmap is the activation pattern of example neu- ron, implementing a specific heuristic type. Within the heatmap, each pixel at location (op1, op2) represents the activation strength of the neuron under the addition prompt "op1 + op2 =".
two distinct patterns: First, in some neurons, the activation pattern depends on both operands and the value vector encodes the expected result of the arithmetic calculation (Figure [[..). We term such neurons direct heuristics. Second, in neurons where the activation pattern depends on a single operand, the value vectors often encode features for downstream processing, rather than the result tokens directly (Figure We term such neurons indirect heuristics. Next, we demonstrate how these heuristic neurons combine to produce correct arithmetic answers.
4 ARITHMETIC PROMPTS ARE ANSWERED WITH BAG OF HEURISTICS
Observing the example prompt, 2226 68 =", we have shown that it satisfies the pattern of several heuristic neurons, where cach such neuron slightly increases the logit of the result token, 158 (Figure D. These small increases combine to promote the correct token as the final answer. We hypothesize that combination of independent heuristics termed bag of heuristics seenisttis-emeergss across arithmetic prompts, comprising the mechanism used by the model to produce correct answers.
4.1 CLASSIPYING NEURONS TO HEBUISTIC TYPES
To present evidence for the causal effect of the bag of heuristics on generating correct answers, we first systematically classify neurons into heuristic types. Through manual observation of key activation patterns, we identify several categories of human-identifiable heuristics, exemplified in Figure [ and further detailed in Appendix F. To determine if neuron n at layer implements specific heuristic, we examine the intersection between the prompts that activate the neuron and the prompts expected to be activated for this heuristic. A visual example of this procedure is shown in Figure An automated algorithm of this approach is described in Appendix
Neuron: 29 2850
Logils
Heuristic: result range [150, 180]
(b)
(c)
29,2850 post
29,2850
oull
170
999
X
130
145
299 291
X
(a)
GP:
(8)
145 + 14
Top
(g)
Does neuron implement heuristic?
(d)
136 + 17
70 + 81
Yes
RP2
OP2
+ 181
X
No
Aoost 29,22550 for each operand pair (op1, op2). using the chosen operator (addition). (b) Calculate the logits of numerical tokens embedded in oou a 2D pattern, where the cell in index (op1, op2) the logit of the result token of applying the operator to (op1; op2) (i.e. OP1 + op2). (d) Multiply both patterns element-wise, to get the effective logit contribution of the neuron to the correct answer token for each prompt. activate the neuron the most from the activation pattern. (I) Create list of prompts associated with the tested heuristic. (g) Measure the intersection between the two prompt lists. If this intersection is larger than threshold (we use 0.6), the neuron is said to implement the heuristic.
Figure 6: Neuron to heuristic matching example. (a) Measure the value of 1,29
29,2880 using Logit Lens (nostalgebraist 2020). (c) Convert the logits vector to
(e) Extract the prompts that
We apply this algorithm to each pair of important MLP neuron n in layer and heuristic H. Through this method, we classify as arithmetic heuristics 91 % of the 3,200 top neurons for each operator

---

Published as a conference paper at ICLR 2025
Addition
Subtraction
Multiplication
Division
MMMM
W
Accuracy
0.5
Mean: 0.86
Mean: 0.88
Mean: 0.89
Mean: 0.91
Mean: 0.56
Mean: 0.64
Mean: 0.80
Mean: 0.78
0 14
42 56
0
38
57 76
44 66
0
21 Heuristic index
14
35
Heuristic index
Heuristic index
Heuristic index
Effect of heuristic ablation on related prompts
Effect of heuristic ablation on unrelated prompts
Figure 7: For each heuristic, we measure the accuracy of 100 correctly completed prompts associ- ated with a heuristic (blue) and 100 correctly completed prompts associated with the heuristic (yellow), after ablating that heuristic's neurons. The heuristics are sorted by the accuracy drop in- duced on associated prompts. Across most heuristics, ablating heuristic neurons causes a larger decrease in accuracy in prompts associated with that heuristic than in not associated prompts.
(200 per layer across 16 layers). Manual inspection of neurons that fail to classify into one of the defined heuristics reveals patterns that are not clearly identifiable (Appendix D).
4.2 HEURISTIC TYPES ARE COMBINED TO ANSWER ARITHMETIC PROMPTS
Following the classification of the important neurons in each middle- and late-layer ML. into one or several heuristic types, we provide evidence that the bag of heuristics is the primary mechanism the model uses to correctly answer arithmetic prompts. We show this through two ablation experiments.
Knocking out neurons by heuristic type. We now verify that the neurons in each heuristic type contribute the accuracy of associated prompts by knocking out entire heuristic types and observe the resulting changes in model accuracy. We define a prompt as associated with a heuristic if and only if its components meet the conditions specified by that heuristic. (For instance, the example prompt "226 68 =" is associated with the heuristic "opl = 0 (mod 2)".) For each heuristic, we sample two sets of 100 correctly completed prompts each, one containing prompts associated with the heuristic and the other containing prompts not associated with that heuristic. For each heuristic, we knock out all neurons classified into it (by setting each f activation to zero) and then remeasure the accuracy on both sets of prompts. We expect higher decrease in accuracy on the associated prompts, since we claim each heuristic is causally linked only to its associated prompts.
The results (Figure show that ablating neurons of a specific heuristic causes significant accuracy drop on associated prompts, more than on not associated prompts, on average. This confirms the causal importance of heuristic neurons in promoting correct answer logits specifically in prompts that are associated with their heuristic type, verifying the targeted functional role of these heuristics.
However, the ablation does not result in complete accuracy drop; it causes an average drop of 29% out of 95% average pre-ablation accuracy. We find two reasons for this. First, some heuristics have low recall: they do not apply to all associated prompts as they should (see Appendix D Second, each prompt relies on several unrelated heuristic types, so even when one is ablated, others still contribute to increasing the correct answer's logit. In the following ablation experiment, we verify that this interplay of heuristics provides a fuller image than focusing on one heuristic at time.
Knocking out neurons by prompt. To provide further evidence that the bag of heuristics is causally linked to correct arithmetic completion, we conduct a second ablation experiment. For each correct prompt, we identify the heuristic types that should affect it based on its operands and ground truth result. We then ablate the neurons with the highest classification scores (Section 4.1) these heuristics, up to a certain neuron count, and check the model's completion changes.
The results (Figure X) show that ablating neurons from associated heuristics significantly drops the model's accuracy, much more than the accuracy drop caused by ablating the same number of randomly chosen neurons from unassociated heuristics. This demonstrates that we can identify the neurons important to a given prompt solely based on its associated heuristics. This also indicates a causal link between the neurons belonging to several heuristics and the prompt's correct completion. This supports our bag of heuristics claim: each heuristic only slightly boosts the correct answer logit, but combined, they cause the model to produce the correct answer with high probability.
7

---

Published as a conference paper at ICLR 2025
Llama3-8B
Uamaa-708
Accuracy
0.75
0.25
60
120
160
Ablated Neurons (Per Layer)
Ablated Neurons (Per Layer)
Heuristic Ablation
Random Ablation
Figure 8:
Knocking out neurons that implement heuristics associated with each prompt (full lines) leads to greater decrease in accuracy than knocking out the same number of neurons whose heuris- tics are not associated with each prompt (dashed lines). This effect occurs across model sizes.
4.3 FAILURE MODES OF THE BAG OF HEURISTICS
The bag of heuristies mechanism employed in Llama3-8B does not generalize perfectly: fails to achieve perfect accuracy across all arithmetic prompts (Appendix HD. This limitation contrasts with theoretical robustness a genuine algorithmic approach. Here, we aim to elucidate the specific failures of this mechanism, focusing why it falters for some prompts.
We hypothesize that the bag of heuristics mechanism completes prompts incorrectly in two ways. (1) The "bag" might not be big enough; i.c., prompt might lack sufficient associated neurons. (2) the heuristics might have imperfect recall (e.g., a neuron that fires for most prompts where the first operand is even, but does not fire for the prompt "226 68 =") have low logits for the correct answer token the value vectors.
To test these hypotheses, randomly sample 50 correctly completed and 50 incorrectly completed prompts. To test hypothesis (i), count the number of heuristic neurons associated with each prompt. We find that on average, incorrect prompts have more heuristic neurons associated with them than correct prompts. Therefore, we find no support for this hypothesis. To check hypothesis (ii), we calcu- late the total contribution of heuristic neurons to the logit of the correct answer for each prompt. This measurement considers both the specific activation of each neuron for the prompt, as well as the logit of the correct answer token embedded in each neuron's value vector. On average, there is indeed slight advantage in the total logit contribution for correct prompts over incorrect prompts (Figure D. This suggests that the primary reason for the bag of heuristics failure on certain prompts is poor promotion of correct answer logits, rather than a lack of heuristics.
Prompt
Incorrect
Type
Correct
15 Correct answer logit contribution
10
Figure 9: The model's failures be plained by a lower total logit contribution of the heuristic neurons the correct answers.
TRACKING HEURISTICS DEVELOPMENT ACROSS TRAINING STEPS
Does the bag of heuristics emerge as the primary arithmetic mechanism from the onset of training. does it override earlier, different mechanism that initially drives arithmetic performance? We conduct an analysis of heuristic development across the training trajectory of the Pythia-6.9B model (Biderman al. 2023). due the public availability of its training checkpoints. Specifically, we analyze the model at its final checkpoint (143K steps) and at 10K-step intervals down to 23K steps. The 23K checkpoint is the earliest checkpoint showing good arithmetic performance; Thus, we begin analysis at this checkpoint. To guide this analysis, aim to answer three sub-questions:
When do the final heuristic neurons first appear? We examine when each heuristic neuron from the final checkpoint first appears during training. For each neuron classified into a particular heuristic type, we check if the same neuron gets classified into the same heuristic in earlier checkpoints. Averaging this measure across all heuristic types and operators provides insight into when the final heuristics initially appear during training. We observe (Figure [oa) that the model develops its final heuristic mechanism gradually across training, starting from early checkpoint.
8

---

Published as a conference paper at ICLR 2025
Heuristic neurons intersection
Faithfulness of circuit with Specific heuristic neurons
Ablated and Control Accuracies
with final checkpoint
Intersection ratio
Falthfulness
Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
23k
63k
103k Training step
143k
23k
63k
103k
143k
23k
63k
103k
143k
Training step
Training step
All heuristic neurons
Heuristic Ablation
neurons
(a)
Overlapping with last checkpoint
Random Ablation
10 neurons
25 neurons
(b)
(c)
Figure 10: Heuristic analysis across Pythia-6.9B training checkpoints. (a) The percentage of heuristic neurons from the last checkpoint that also appear in previous checkpoints increases over training, revealing a gradual creation of the bag of heuristics.
(b) The heuristic neurons that are mutual with the last checkpoint (full line) explain most of the total heuristic behavior (dashed line) at each checkpoint. Thus, the heuristics that disappear across training are less important to the model. (c) Ablating specific heuristic neurons heavily drops the model's 's accuracy across all training checkpoints. This suggests arithmetic accuracy primarily stems from heuristics, even in early stages.
Do additional heuristic neurons exist mid-training, Next, for each mid-training checkpoint, investigate whether its heuristic neurons that are mutual with the final checkpoint make up the entire heuristic mechanism in that checkpoint, or whether other heuristics exist that later become vestigial. We examine the faithfulness (Section 2.I] of the arithmetic circuit at each checkpoint- once when including only the neurons mutual with the final checkpoint, and once when including all heuristic neurons in that checkpoint. The difference between these two measurements gives us an estimate of the importance of the mutual heuristics. Using this metric, we observe that these final heuristics explain most of the circuit performance for each intermediate checkpoint: they account for an average of 79% of the total heuristics' contribution to accuracy at each checkpoint. This indicates that, while other non-mutual heuristics exist in each checkpoint, these are less important to the circuit's s accuracy and slowly become vestigial the circuit converges to its final form.
Does a competing arithmetic mechanism exist mid-training? Finally, we determine if the heuris- ties appear the main arithmetic mechanism from early on in training, or if they co-exist with an unrelated mechanism that becomes vestigial in later checkpoints. We repeat the prompt-guided neu- sample 50
knockout experiment (Section 4.2) for each checkpoint; i.e., in each checkpoint, correctly completed prompts for each operator. For each prompt, we ablate 5, 10, and 25 heuristic neurons associated with the prompt in that checkpoint. We test if this targeted ablation significantly impairs the model's 's accuracy, even in earlier stages of training, and compare this to a baseline, where we ablate similar amount of randomly chosen heuristic neurons. The results (Figure T0c] demon- strate that removing any amount of neurons from heuristics associated with prompt substantially reduces the model's 's accuracy on these prompts even at earlier checkpoints, much more than ablating a random set of important neurons. We also observe that ablating 25 heuristic neurons per layer is enough to cause near-zero accuracy in all stages of training. This finding asserts that the causal link between prompt's 's associated heuristics and its correct completion exists throughout training.
RELATED WORK
Mechanistic interpretability (MI) aims to reverse-engineer mechanisms implemented by LMs by analyzing model weights and components. Causal mediation techniques (Pearl (001)) like activation patching (Vig et al. 2020 Geiger et af. 2021), path patching (Wang et al.) 2022). and attribution patching (Nanda) (022 Syed et al. 2023 Hanna et al. 2024b) allow localizing model behaviors to specific model components. Other studies have also presented techniques to explain the effect of specific weight matrices input tokens (Elhage et al. 2021 Dar et al. 2023), or to analyze acti- vations (nostalgebraist 2020 Geva et al.) 2021). Many studies have aimed to use these techniques to reverse-engineer specific behaviors of pre-trained LMs (Wang et al. 2022 Hanna et al. 2024a] Gould et al. 2024 Hou et al.) (202)). We leverage MI techniques to reverse-engineer engineer the arttmmetic mechanisms implemented by pre-trained L.IMS and explain them single-neuron resolution.
9

---

Published as a conference paper at ICLR 2025
Memorization and generalization in LLMs. Whether models memorize training data gen- eralize to unseen data has been extensively studied in deep learning (e.g., Zhang et al., 2021) and specifically in LLMs (Tänzer et al., 2022; Carlini et al. 2023 Antoniades et al. 2024), but not many studies have observed this question through the lens of model internals. Among those that do. [Bansal al.] (2022) attempt predict this trade-off by observing the diversity of internal ac- tivations; Dankers & Titov| (2024) show memorization in language classification tasks is not local to specific layers, and Varma al. (2023) explain grokking using memorizing and generalizing cir- cuits. We use this lens to observe how model internals operate in arithmetic reasoning a task that could theoretically be solved either through extensive memorization or by learning a robust algo- rithm. Concurrent work (jylin et al. 2024) has shown that LM trained to predict legal board game moves (Li al. 2 does so by implementing many heuristics. While heuristics would suffice to robustly predict iegal moves in board game setting, we find that the extent to which I LLMSs rely on heuristics is greater than prior work suggests: sets of hcuristics are used to accomplish even generic tasks like arithmetic, where no heuristic is likely to generalize to all possible results.
Arithmetic reasoning interpretability. Recent studies on how LMs process arithmetic prompts (Stoloo et al. 2023 Zhang et al.) 2024) reveal the general structure of arithmetic circuits, but do not fully explain how they combine operand information to produce correct answers. Our research bridges this gap by revealing the mechanism used for promoting the correct answer. Some studies show the emergence of mathematical algorithms for modular addition (Nanda al.] 2023 Zhong et al. 2024 Ding et al. (024) and binary arithmetic (Maltoni & Ferrara 2023) in simple, specialized toy LMs, but it unclear if these findings extend to larger, general-purpon L.Ms or other operators. In pre-trained LLMs. Zhou et al.
(2024) found that Fourier space features are used for addition. However, we claim this is only a partial view, as many additional types of features and heuristics relying on these features are involved in calculating answers across arithmetic operations. In this work, we give a wide view of these heuristics and how they combine to generate arithmetic answers.
CONCLUSIONS
Do LLMs rely on robust algorithm or on memorization to solve arithmetic tasks? Our analysis suggests that the mechanism behind the arithmetic abilities of LLMS somewhere in the middle: LLMs implement bag of heuristics combination of many memorized rules-to perform arith- metic reasoning. To reach this conclusion, we performed a set of causal analysis experiments to locate circuit, i.e., subset of model components, responsible for arithmetic calculations. We ex- amined the circuit at the level of individual neurons and pinpointed the arithmetic calculations to a sparse set of MLP neurons. We showed that each neuron acts as a memorized heuristic, activating for a specific pattern of inputs, and that the combination of many such neurons is required to cor- rectly answer the prompts. In addition, we found that this mechanism gradually evolves over the course of training, emerging steadily rather than appearing abruptly or replacing other mechanisms.
Our results, showing LLMs' reliance on the bag of heuristics, suggest that improving LLMs' mathe- matical abilities may require fundamental changes to training and architectures, rather than post-hoc techniques like activation steering (Subramani et al., 2022 Turner et al., 2023). Additionally, the evolution of this mechanism across training indicates that models learn these heuristics early and re- inforce them over time, potentially overfitting to early simple strategies; is unclear if regularization can improve this, and this is a possible avenue for future research.
8 LIMITATIONS AND DISCUSSION
Interpretability work is often fundamentally limited by human biases. As researchers, we often impose human abstractions onto models, whereas the goal of interpretability is to understand the abstractions that models learn and apply in a way that we can understand. Our work also sub- ject to this limitation, namely with respect to the definition of heuristic types: We define heuristic abstractions based on our humaa-identifiable definitions. A possible improvement would be to de- velop method to identify these abstractions without human bias. Another important detail is that our analysis focuses on LLMs that combine digits in tokenization. That is, every token can contain more than one digit. The robust algorithms used by humans depend on our ability separate larger numbers to single digits. Thus, a similar analysis might lead to different conclusions for models that perform single-digit tokenization.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We are grateful to Dana Arad and Alessandro Stolfo for providing feedback for this work. This research was supported by the Israel Science Foundation (grant No. 448/20), an Azrieli Founda- tion Early Career Faculty Fellowship, and an AI Alignment grant from Open Philanthropy. AM is supported by a postdoctoral fellowship under the Zuckerman STEM Leadership Program. AR is supported by postdoctoral fellowship under the Azrieli Postdoctoral Fellowship Program. This research was funded by the European Union (ERC, Control-LM, 101 165402). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the Euro- pean Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.
REFERENCES
Antonis Antoniades, Xinyi Wang, Yanai Elazar, Alfonso Amayuelas. Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization vs. memorization: Tracing language models' capabilities back to pretraining data. In ICML 2024 Workshop on Foundation Models the Wild, 2024. URL btttps://openeeeee
net for.mooruimi.o Eorum?1dd-laybb2-11
Rachit Bansal, Danish Pruthi, and Yonatan Belinkov. Measures of information reflect memorization patters.. In Advances in Neural Information Processing Systems, 2022.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguistics, 48(1):207-219 2022.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023.
Verna Dankers and Ivan Titov. Generalisation first, memorisation second? Memorisation localisa- tion for natural language classification tasks. In The 62nd Anmual Meeting of the Association for Computational Linguistics, pp. 14348-14366. Association for Computational Linguisties, 2024.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers), pp. 16124-161111 16170, 2023.
Xiaoman Delores Ding, Zifan Carl Guo, Eric J Michaud, Ziming Liu, and Max Tegmark. Sur- vival of the fittest representation: A case study with modular addition. In ICML 2024 Work- shop on Mechanistic Interpretability, 2024.
net, forum?id=
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv: 1783, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan Cur- (eds.), Advances in Neural Information Processing Systems, volume 34, pp. files/ ran Associates, Inc., 2021. URL htt.://w...//////////ooooooooo.. faap://0001118555222222222222280

---

Published as a conference paper at ICLR 2025
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, inter- pretable attention heads in the wild. In The Twelfth International Conference on Learning Repre- sentations, 2024. URL htt.httttss/ppeeeiiiiiie hwt.//ewweeeeeeeeeeeeeeeeeeeeeee
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36, 2024a.
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. arXiv preprint arXiv:2403. 00:00./7880 /7806, 2024b.
Tom Henighan, Shan Carter, Tristan Hume, Nelson Elhage, Robert Lasenby, Stanislav Fort, Nicholas Schiefer, and Christopher Olah. Superposition, memorization, and double descent. Transformer Circuits Thread, 6:24, 2023.
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards mechanistic interpretation of multi-step reasoning capabilities of language models. Proceedings of the 2023 Conference on Empirical Methods
in Natural Language Processing, pp. 4902-4919, 2023.
jylin, JackS, Adam Karvonen, and Can Rager. Othellogpt learned bag of heuris- tics, 2024. URL bttps://www. www.www...wwwogg.ooomm com posstss/cclueennAaaaaaa.. -awpre---eaaaeed-a--------------
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. In The Eleventh International Conference on Learning Representations, 2022.
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to MLPs. Advances in neural information processing systems, 34:9204-9211 2021.
Davide Maltoni and Mattco Ferrara. Arithmetic with language models: From memorization to computation. arXiv preprint arXiv:2308.01155 2023.
Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy sup- pression: Comprehensively understanding an attention head. arXiv preprint urXiv:2310004625 2023.
Neel Nanda. Attribution patching: Activation patching at industrial scale, 2022. URL https:
h/www. www..neelnanda.12.. www.ee/hnismiimiineeeeeeeee.eeee hwwpr//wwwe..oeeeaaaaieeeeeeeeee
patching
Neel Nanda and Joseph Bloom.
Transformerl.em
https:///ithhit com/
2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023.
nostalgebraist. Interpreting GPT: The logit lens, 2020. WRRLtttps:/www.lesssee http:///ssssssspppaaaaaee/eeeerp
ing gpt the logit -enn
Judea Pearl. Direct and indirect effects. Direct and Indirect Effects, 2001. URL https://ftt.
C5. wclaalla a www.eduuuubl. stttt_srr/t223-000
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: case study on entity tracking. In The Twelfth International Conference Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arith- metic reasoning in language models using causal mediation analysis. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 7035-7052, 2023.
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting latent steering vectors from pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 566-581, 2022.
Aaquib Syed, Can Rager, and Arthur Conmy, Attribution patching outperforms automated circuit discovery. In NeurlPS Workshop on Attributing Model Behavior at Scale, 2023. URL https:
//openreview.... http:/rriiiii..eeeoooreieee. wwweee/oooomnddddiiiteeee
Michael Tänzer, Sebastian Ruder, and Marek Rei. Memorisation versus generalisation in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Pupers), pp. 7564-7578, 2022.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv e-prints, pp. arXiv-2309, 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 33: 13:18288-22401, 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. ht https::/ggitiiii com k ngof o z me www.tttttroorrreeooooooooo --ax May 2021.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: A circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2022.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation. In Forty-first International Conference on Machine Learning, 2024. URL /ttpps://openreviee: n
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Tianyi Zhou, Deqing Fu, Vatsal Sharan, and Robin Jia. Pre-trained large language models use Fourier features to compute addition. arXiv preprint arXiv:2406 034455, 2024.
13

---

Published as a conference paper at ICLR 2025
A LLAMA3-8B CIRCUIT EVALUATION RESULTS
Llama3-8B circuit faithfulness
as function of number of circuit heads
1
Faithfulness
0.5
0
10
20
30 40
Number heads in circuit
X
Figure 11: Llama3-8B arithmetic circuit faithfulness as function of number of circuit heads.
In Figure we present an analysis of the faithfulness of the identified circuit in Llama3-8B (Sec- tion 2.1 as function of the number of attention heads within the circuit. The circuit includes all MLPs (the neuron-level analysis in Section 3.1 does not apply in this context). Our goal is to find a minimal subset the model with the fewest attention heads possible while maintaining high faithfulness. We assess faithfulness independently for each operator to obtain more nuanced un- derstanding of the necessary attention heads. For addition, subtraction, and division operations, we observe that 6 heads suffice to attain high faithfulness (97% average). In contrast, in multiplica- tion, 20 attention heads are required to achieve faithfulness score exceeding 90%. The faithfulness of the full arithmetic circuit for each operator, corresponding the Pareto-optimal number of heads (in terms of achieving high faithfulness with as few heads as possible), is documented in Table We explore the attention patterns of these attention heads in the next section.
Table Llama3-8B arithmetic circuit faithfulness, per operator.
Operator
+
Faithfulness
0.97
0.98
0.90
0.96
# Attn Heads
6
6
20
6
B LLAMA3-8B CIRCUIT ADDITIONAL COMPONENTS
To provide a more comprehensive understanding of the arithmetic circuit in Llama3-8B, we analyze the additional components that compose it, namely the first MLP layer (MLPO) and the high-effect attention heads.
B.1
MLPO
To analyze the role of MLPO in the arithmetic circuit, we first test if- ii-sirilarly to the middle- and late-layer MLPs-only a sparse set of neurons within the MLP is required. We measure the inter- vention effect of each neuron, averaged across the operands and operator positions (p [1, 2, 3). The results, shown in Figure 12a reveal that few MLPO neurons have high effect, similar to the middle- and late-layer MLPs. To verify these neurons are sufficient for arithmetic calculations, we repeat the experiment from Section 3.1 Specifically, we measure the faithfulness of the circuit- consisting of the top 1 of neurons each of the middle and late layers (1 [16, 32) as well as varying number of neurons in MLPO. The results, shown in Figure 12b reveal that also in the first MLP layer, as little as 1% of neurons is sufficient for the circuit to achieve high faithfulness, similarly to the middle- and late-layer MIPs.
Because Llama3-8P 3BB applies a positional embedding only at attention layers, the activation of MLPO not affected by the position of any token. Additionally, due lack of attention heads that move information between the operand and operator positions before MLPO, we can analyze its effect directly on single tokens. Thus, view MLPO as an "effective embedding" (McDougall et al.
14

---

Published as a conference paper at ICLR 2025
Individual MLP Neuron Intervention Effects
Faithfulness of using only top-k neurons
Intervention Effect
Falthfulness
0.05
0.8
0
0.6
0
5k
10k
1%
10%
100%
Neuron Index
Neurons used Per Layer (%)
(a) Few neurons in the first MLP have high effect arithmetic prompts.
(b) The circuit faithfulness when including percent- age of high-effect MLPO neurons in the circuit.
Analyzing effect of individual MLPO neurons. A small number of neurons is sufficient for circuit accuracy.
Figure 12:
2023). and hypothesize that the role of each MLPO neuron is incorporate additional numerical information into each token embedding. To verify this, we pass list of numerical tokens, each representing an operand in our analyzed operand range (t E [0, 300)), through the model. We measure the activation of each high-effect neuron for each token. As exemplified in Figure [3 the activations these high-effect neurons correspond to varied numerical features. For example, Neuron 6206 (Figure 13a activates for numbers near 170 or 17; Neuron 7101 activates for numbers greater than 100; Neuron 8969 activates for numbers that are congruent to 8 (mod 10). Overall, the set of patterns identified by these neurons be used by the middle- and late-layer heuristic neurons to perform their more complex functionalities.
Neuron activation
Neuron activation
Neuron activation
0.4
0.6
0.4
-0.1
0.2
-0.2
-0.3
052
50
100
222
252 RBZ
Numerical input token
Numerical input token
Numerical input token
(a) Neuron 6206
(b) Neuron 7101
(c) Neuron 8969
Figure 13: Individual MLPO neurons have identifiable activation patterns for numerical tokens.
B.2 ATTENTION HEAD PATTERNS
We present the attention patterns of the attention heads that contained in the arithmetic circuit. In Section 2.1] for simplicity, we consider the arithmetic circuit as single circuit for all arithmetic operators, containing all attention heads used across the four operators. We observe that achieve high faithfulness of the circuit with a minimal number of heads (Appendix [A], some general arith- metic heads, that significantly improve faithfulness across all operators, are required. Other heads, while contributing to increased faithfulness, are operator-specific. They exhibit varying levels of importance for different operators. A full description of the attention heads used in the arithmetic circuit for each operator is provided in Table 2
To better understand the role of the general arithmetic heads (L2H2, LISH13, L16H21), we compute their attention patterns, averaging them across our prompt dataset (Section 2.1). These patterns (Figure 14) reveal a clear signal: each of the three heads attends to a single input token, copying the representation from that position and projecting it to the last position. Specifically, L16H21 attends to the first operand, L2H2 attends to the operator and LI5HI3 attends to the second operand. This implies the role of each such head is to move the representation from that position, which includes to the last position, where it is further processed by the bag of heuristics implemented in the middle- and layer MLPs. To confirm that the information copied from each position to the last position consists solely of data from the token at that position, we conduct an ablation study. For each general arithmetic head, we zero out all preceding attention patterns that move information
15

---

Published as a conference paper at ICLR 2025
Table 2: Llama3-8B operafor-specific arithmetic circuit attention heads. The general arithmetic
heads are marked bold. LiHj denotes the jth attention head in Layer
Operator
Circuit Heads
1,222, L5H3, L,H31, L14H12, L15H13, L16H21
L2H2, L13H21, L13H22, L14H12, L15H13, L16H21
L2H2, L5H30, L8H15, L9H26, L13H18, L13H21, L13H22, L14H12, L14H13, L15H8, L15H13, LISH14, LISHI5, L16H3, L16H21, L17H24, L17H26, L18H16, 1.20H2, L22H1
L2H2, L5H31, L15H13, L15HI4, L16H21, LI8H16
to its attended position (e.g., for L15H13 we zero out all patterns that move information to the op2 position), preventing any influence from previous positions. We observe this does not affect the circuit's performance, indicating that the representation copied by each general arithmetic head to the final position contains information only regarding the token at its original position.
Bo5
Bo5
BoS
Destination Token
Destination Token
Destination Taken
Op1
Op1
0p1
Operator
Operator
Operator
0p2
Op2
0p2
Bos
Op1
Op2 Operator
Bo5 Op1
Op2 Operator
Bo5 Op1
Op2 Operator
Source Token
Source Token
Source Token
(a) Layer Head 2
(b) Layer 15 Head
(c) Layer Head 21
Figure 14: The attention patterns for the general arithmetic attention heads in Llama3-8B. show that each head attends, at the last token, to a single previous token across all prompts. When combined, these heads attend to all three operand and operator positions.
C MULTI-LAYER PERCEPTRON IMPLEMENTATION DETAILS
Across the models we analyze, different implementations exist for the MLP layer in transformer block. More specifically, Pythia-6.98 and GPT-J use simple MLP layers, consisting of two matrix multiplications. For layer 1, it can be described as:
h bjost (1'n W in
(4)
h = h Wi out
(5)
post
where In/., bbut IRd the input and output representations of the MLP at layer respectively. h hpost Rdmip is the post-activation vector, W in: Wi out Ddmippd are parameter matrices, and
a non-linearity function. Llama3-8B and Llama3-701 70B use Gated MLP layer (Liu et al. 2021) described in the following two equations for layer l:
I post
= =h W gate) o (h/, W in
(6)
W h h/ h post out
where W gate Badd xd an additional parameter matrix and is Hadamard product. Biases are omitted in both presentations.
16

---

Published as a conference paper at ICLR 2025
While the key-value view that was used in Section was devised for simple MLPs (Geva et al. 2021 it can be applied to the Gated MLP mechanism of Llama3 models as well. For the n neuron, we treat the element Spost of hpost as the activation of the nhhh key vector. Each such activation is multiplied with Youts a row vector of W out- resulting in the same multiplication as performed in simple MLPs.
D NEURON INTERSECTION BETWEEN OPERATORS
Neuron Intersection Across Operators
100.0%
55.8%
52.6%
52.3%
Operator
55.8%
100.0%
53.0%
53.3%
52.6%
53.0%
100.0%
55.8%
0.4
52.3%
53.3%
55.8%
100.0%
0.2
Operator
Figure 15: IOU of MLP neurons between operator-specific circuits
Figure 15 presents the IoU (intersection over union) of causally important neuron sets in the arith-
metic circuits, identified separately for each operator. Each circuit consists of the top-200 neurons from each layer [16,32], ranked based on their mean effect (Section 3.1), totaling 3, 200 neu- rons. The average loU between any two operator-specific circuits is relatively low (54%), indicating that substantial proportion of key neurons unique to each operator. Consequently, define distinct circuit for each arithmetic operator at the neuron level (Section 3.I.
E HEURISTIC TYPES DESCRIPTIONS
In this section, we present the various heuristic types that were manually identified.
Most heuristic types are defined with parameters. A neuron is defined as heuristic H, if it matches the heuristic definition given a parameter value. For example, the neuron Figure M range heuristic with range parameter [150, 180). In Algorithm I describe the process of matching between each important neuron and each heuristic H. The parameters are sub-sampled evenly to cover most observed possibilities, which causes some inaccuracies (for example, neuron that fires when 0p2 E [105, 205] will be classified as a range heuristic with op2 E [100, 200] due to our parameter choices).
Several types of heuristics (Range, Modulo, Pattern) can apply either to an operand in a prompt the prompt's result. For example, an "operand range" heuristic is triggered when opı ops falls within specific numerical range, while a "result range" heuristic activates when the prompt's ground truth result is within a defined range.
The full list of heuristic types is as follows:
Range heuristic: A neuron that activates when value (either an operand or result) falls within a specified range [a, bj. The parameters a, E N are chosen such that the
range length (en = - a) E [10, 30, 50, 100] for addition, subtraction, and multi- plication, and len (b - a) E [2, 10, 100) for division (due to different distribu- tion of potential results for integer division). The range start, a, is defined as a e Zn En Zo max( len 3 10), No 0. provided remains below the maxi- mum prompt result. This definition of range start and length allows for ranges to intersect overlap.
17

---

Published as a conference paper at ICLR 2025
Modulo heuristic: A neuron that activates when value (either operand result) is congruent to m modulo n. The parameters are n E {2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 15) and specific cases of the broader
100 as these "pattern" " heuristics, discussed next. Additionally, = 12, 14, and n 2 16 are excluded, as neurons were classified under these parameter values.
E [0, n - 1]. We exclude
10
Pattern heuristic: A neuron that activates when value (either operand or result) matches a specific regular expression p. The parameter p is 3-digit regular expression, potentially padded with leading zeros. For example, neuron classified under the "operand pattern 1.2" heuristic activates when one of the operands has in the hundreds place and in the units place.
Identical operands heuristic: neuron that activates when both operands are equal (opı == =op2). The tokens embedded the value vector change depending on the operator for which the neuron activates. For instance, in subtraction, such neurons have been observed to promote the "0" token (the result of subtracting a number from itself).
Multi-result heuristic: A neuron that promotes set S of several unrelated results (For example, neuron that promote the results [4, 5, 7). This heuristic type is defined exclu- sively for division, where the parameter S can consist of several values, where |S] E [2. 4]. This range was chosen based on observations of activation patterns. The values included in S chosen according to the result tokens that the neuron contributes the most determine S by examining the top prompts for which the neuron promotes their answer. From these top answers, we identify a minimal of 2 to distinct result values. This set of result values is chosen if it accounts for more than a threshold percentage of the re- sults. This approach captures the most significant result values that the neuron consistently promotes in division operations. If no such set of different results exists, neuron is not classified as this heuristic.
F HEURISTIC NEURON CLASSIFICATION ALGORITHM
We present the full algorithm used match between each neuron and each heuristic type in Algo- rithm (as exemplified in Figure 6). The goal of this process is to check if neuron implements specific heuristic. We repeat this process for each pair of neuron n in layer I and heuristic H. This method allows single neuron to be classified as several heuristics. The matching of each neuron to heuristics done separately for each arithmetic operator.
Algorithm Neuron Classification To Heuristic Type
Inputs: Heuristic H. Layer E [1, [max] ). Neuron Index n E [L, Nmaz).
Operator E {+, )) Threshold t E [0, 1.0]
1: A < GeneraateActivationPaateee
> 2D Activation pattem
2: if H a direct heuristic then
3:
LogitLLns((v!!))
Logits over vocabulary & 2D Logits Pattern
4: L < Convert (oovvertTPPateen())
5:
A <- L
Element-wise multiplication
6: end if
7: promptsheuristic + (etAssoiate(PPoont(t((
Expected prompts activate H
8: k + |promptsheeriitte
9: prompttneeron &- GeTTTpTAAccivaaingPPoompss/ k)
b Prompts that activate neuron
10: promplsintersectt
hww.mmppphhhaittii O prompl&seerro
11: score promptssinterecttonn
Normalize intersection to [0, 1.0]
k
12: return score 2
The algorithm first calculates the neuron's 's activations hoot for all prompts. yielding 2D activation pattern (Line []. as those seen in Appendix [ When checking if a neuron implements a direct heuristic (heuristics that directly promote relevant result tokens (Section ).2)). we also need to take
into consideration the tokens that are embedded in the neuron's 's value vector v v.m- This is not done in indirect heuristics because we do not expect relevant result tokens to be promoted in such heuristics.
Thus, extract the logits of all numerical tokens ("0", "1" "2"" ... "999") from the value vector
18

---

Published as a conference paper at ICLR 2025
An Your (Line 3) using Logit Lens (nostalgebraist 2020 To match the logit of each result token to the
prompts that result in it (e.g., match the logit of "151" with the prompts "1 + 150 =". + 149 =",
2D pattern L, where Li is the logit of applying the operator i and j (Line 4.. Multiplying the 2D activation pattern and 2D logit pattern element-wise (Line 5 yields the neuron's i effective logit contribution the correct answer of each prompt, i.e., the value each index i marks how much the neuron promotes the answer token to the result of applying the operator on and j. We then create two lists of prompts. One list contains prompts associated with the heuristic (Line to be used as the "ground truth". We mark the number of associated prompts as io (Line 8). and find a second list, containing the top-k prompts that are most contributed by the neuron (Line 9). We measure if the intersection of these two lists, normalized by k, is larger than the threshold 0.6 (Line 10 Line 12 to decide if the neuron implements the heuristic.
etc.), we convert the logits
G ADDITIONAL IMPLEMENTATION DETAILS
All experimental procedures were executed using the TransformerLeess library Nanda & Bloom 22022). Experiments involving Llama3-8B, GPT-J, and Pythia-6.9B were conducted on a single Nvidia-L40 GPU with 48GB of GPU memory. Llama3-70B was loaded in 16-bit precision across four Nvidia A100 GPUs, each with 80GB GPU memory.
Arithmetic prompts. Unless otherwise specified, our experiments utilize prompts of the form op1 op2=", where op1, op2 E [00] and o E {+, -, } The division operator (+) is inter- preted as integer division, considering only the integer part of the result (e.g.. the ground truth result for 45 + 4 =" is 11). For each prompt, we compute the ground-truth result and exclude prompts whose results are not tokenized to a single token. This filtering process eliminates prompts with neg- ative results and prompts for which the correct answer exceeds the single-token limit (The highest token from which the model splits numbers into two tokens). The single-token limit s for Llama3-8B and Llama3-70B is 8ilama3 = 1, 000, while for GPT-J and Pythia-6.9B, it is Sgptj 8pythia 520.
Circuit discovery via activation patching. In Section 2.1 we quantify the effect of patching on probabilities. We find that applying our effect measure (Equation (I) on logits instead of prob- abilities, does not alter the result significanll--th most effective components remain consistent, differing only in the scale of the effect.
Circuit faithfulness evaluation. Thus far, to measure the faithfulness of the circuit, we have per- formed mean ablation on all non-circuit components. For that, we calculate the mean activation output of each component, across all arithmetic prompts. In this context, we do not filter correct prompts exclusively, but instead use all prompts of the form "opı o op2=", for opı, op2 E [0, 300]. This leads to an equal amount of prompts per operator, thus maintaining the mean activation bal- anced across the different operators. When evaluating a circuit with partial MLP layers (Section 3.1) we mean ablate lower-effect neurons in middle- and late- MLPs only, starting and ending the carliest and latest layers from which the answer is extractable via linear probe (Section 2.2) These layer ranges are [16,32) [39, 80), [14, 32), [17, 28] In Llama3-8B, Llama3-70B, Pythia-6.93- and GPT-J, respectively.
Linear probing for correct answers. We define a linear probe fi.p : R - R', where is the single-token limit, predict the correct answer token from the output representation layer and position p. We compute these outputs by using all correctly completed prompts, with 80% used for training the classifier and 20% for evaluation. Each probing classifier is implemented as one-layer fully connected model. We train it using the Adam optimizer (Kingma & Ba, (2015) with a learning rate of 0.0003 and batch size of 32, optimizing cross-entropy loss function.
Heuristic classification. In applying the heuristic classification algorithm (Section we use all activations from the last position p = 4 and employ a classification score threshold 0.6. Due to lack of ground truth data for neuron-to- neuron-to-heuristic matches, the threshold is chosen to achieve Pareto- Pareto-optimal balance of false positives, that occur more for a lower threshold, and false negatives, that occur more in higher threshold. The amount of false classifications is estimated manually.
19

---

Published as a conference paper at ICLR 2025
Table Accuracy of the analyzed models on arithmetic prompts.
Operator
Model
Average
Llama3-8B
0.97
0.96
0.84
0.92
0.95
Llama3-70B
0.97
0.99
0.99
0.73
0.88
Pythia-6.9B
0.30
0.04
0.27
0.75
0.43
GPT-J
0.23
0.09
0.46
0.64
0.37
H MODEL ACCURACIES ON ARITHMETIC PROMPTS
Table 3 presents the accuracy of the analyzed models arithmetic prompts. The accuracy is eval- uated across prompts where the result is represented by a single token, with both operands con- strained to the interval [0, 300], where 300 represents the maximum value of operands in the an- alyzed prompts (Section 2.1 chosen for efficiency. is noteworthy that the high accuracy rates observed in division operations for the smaller-scale models (GPT-J, Pythia-6.9B) can be attributed to the non-uniform distribution of answers in integer division, i.e. half of the legal prompts result in the token '0'.
I RESULTS ON ADDITIONAL MODELS
To demonstrate the generalizability of findings across differently-traineed LLMs, conduct our primary experiments on Jlama3-70B (Dubey et al.] 2024), Pythia-6.9B (Biderman et al.) 2023), and GPT-J (Wang & Komatsuzaki 202I). We replicate the experiments for eircuit discovery and evaluation (Section 2.1). linear probing for answer token embeddings (Section (.2),, top-k neuron
faithfulness analysis (Section 3.1). and heuristic analysis (Section The results obtained are similar to those of Llama3-88 88B across all three LLMs:
The circuit comprises a sparse subset of attention heads that project operand and operator information to the final position, along with all MLP layers. Early-layer MLPs process in- formation at the operand and operator positions, while middle- and late-layer MLPs process the combined information at the last token. These findings are illustrated for Llama3-70B (Figure I6a). Pythia-6.9B (Figure 17a), and GPT-J (Figure [8a).
Linear probing results indicate that the correct answer can only be extracted with high accuracy in the final position, following the processing initiated by middle-layer MLPs. These observations are presented for Llama3- (Figure 16b), Pythia-6.9P (Figure 17b).
and GPT-J (Figure I8b)
The circuit requires a sparse subset of middle- and late-layer MLP neurons to achieve maximal faithfulness. These results are depicted for Llama3-70B (Figure 16c.. Pythia- 6.9B (Figure 17ch. and GPT-J (Figure 18ch.
The heuristic neurons in the middle- and late-layer MLPs are the model components that write the correct answer to the model output. This is shown by repeating the prompt-guided knockout experiment (Section 4.2). The ablation of specific heuristic neurons associated with prompts results in more significant reduction in model accuracy compared to the ablation of a random set of neurons of equivalent size. These findings are illustrated for Llama3-700 (Figure 8), Pythia-6.99 (Figure 17d), and GPT-J (Figure 18d).
The results observed in Llama3-70B exhibit the highest similarity to those reported in Llama3-8B, with more pronounced knockout effect compared to the other two models. We hypothesize that this indicates a more sophisticated development of the bag of heuristics Llama3-70B, potentially facilitated by its larger size and modern training methodology.
J ADDITIONAL EXAMPLES FOR ARITHMETIC HEURISTICS
We present supplementary examples of heuristic neurons (Table Table 5 Table and Table 7 for the four arithmetic operators, respectively). Each neuron h I.n
post found to be important for a specific
20

---

Published as a conference paper at ICLR 2025
more heuristics. For each designated oper- ator, we randomly present four neurons. We compute and report each neuron's activation pattern, the len numerical tokens with the strongest embeddings in each neuron's 's value vector Vouts and provide a non-exhaustive list of the heuristics it implements.
arithmetic operator (Section 3.1 is categorized
I.n
Furthermore, we present in Table g several examples of causally significant neurons that have not been classified specific heuristies. These neurons are relevant the aforementioned limitation in our methodology (Section 8): it is conceivable that they too may be considered components of the bag of heuristics, depending upon our ability to comprehend the abstractions they implement.
Answer token probe accuracy
Effect Map Per Attn Head / MLP
log
0
Attention Heads (Last Position)
MLPs
scale
0
Layer
40
20
0.4
Layer
60
Operand1
Operator
Operand2
20
40
60
1 2 3 Position
Position
Attention Head
(b) Llama3-70B linear probing re-
(a) Llama3-708 70B localization results.
sults.
Faithfulness using only top-k neurons
Faithfulness
0.8
0.6
0.4
0.2
0
5k
10k
15k
20k
Neurons used Per Layer X
+
(c) Llama3-70B faithfulness function of MLP rons in each layer.
Figure 16: Llama3-70B analysis results
21

---

Published as a conference paper at ICLR 2025
Answer token
Effect Map Per Attn Head / MLP
probe accuracy
log scale
Attention Heads (Last Position)
8
Layer
16
0.4
10
Layer
24
0.2
20
Operand1
Operator
Operand2
30
0
10
20
30
1 2 3
Position
Attention Head
Position
(b) Pythia-6.98 linear probing re-
(a) Pythia-6.9B localization results.
sults.
Faithfulness of using only top-k neurons
Heuristic knockout accuracies
Falthfulness
0.8
Accuracy
0.75
0.6
0.5
0.4
0.2
0.25
5k Neurons used Per Layer X
10k 15k
20
40
80
Ablated Neurons (Per Layer)
Heuristic Ablation
Random Ablation
(c) Pythia-6.99 1.9ia-6.99BB faithfulness function of MLP neu- rons in each layer.
(d) Pythia-6.9B prompt-specific heuristic knockout.
Figure 17: Results for all analyses on Pythia-6.9B.
Answer token
Effect Map Per Attn Head / MLP
log
probe accuracy
Attention Heads (Last Position)
MLPs
scale
0
Layer
14
0.4
Layer
10
21
0.2
20
Operands
Operator
Operand2
10
15
1
3
Attention Head
Position
Position
(a) GPT- localization results.
(b) GPT-J linear probing results.
Faithfulness of using only top-k neurons
Heuristic knockout accuracies
Faithfulness
0.8
Accuracy
0.75
0.6
0.5
0.4
0.2
0.25
10k
40 Ablated Neurons (Per Layer) Heuristic Ablation
Neurons used Layer
Random Ablation
(c) GPT-J faithfulness function of MLP neurons in each layer.
(d) GPT-J prompt-specific heuristic knockout.
Figure 18: Results for all analyses on GPT-J.
22

---

Published as a conference paper at ICLR 2025
Table 4: Examples of heuristic neurons for the addition operator
Top-10 10 numerical logits in value in
Neuron number
Activation pattern
Classified Heuristics
h Apost
vector v oot
250
Operands
'254', '255',
150
'253', '256',
resull [240, 270 range E 250,300)
31,9208 h
'250", '257',
post
100
'252', '251', '258', '259'
pattern result .5.
0
50
100
150
200
250
Operand2
250
200
'164", '163' '464', '363',
Operandi
31,2919 h
150
'63', '063',
patter 6.., pattern result .64
post
100
'364,, '963', '26'', '663'
Operand2
250
200
22', '5'',
Operandi
h 17,1463
150
'792', '979',
'372', '502'.
opl (mod 5). op2 = 2 (mod 5)
post
100
'882', '962',
'602', '02"
50
Operand2
250
200
Operand1
'922', '169'.
150
'322', '558'
rangeop1 [0,
16.10226
'873', '753'.
range [0, 100).
post
100
'129', '852'. '41', '33'
range op2 E [0,
0
50
100
150
200
250
Operand2
23

---

Published as a conference paper at ICLR 2025
Table Examples heuristic neurons for the subtraction operator
Top-10 numerical logits in value
Neuron number h post
Activation pattern
Classified Heuristics
vector v f.n Your
200
'0', '000",
Operandi
'00', '006'.
h post
30,8806
150
'002', '004'
identical operands, pattern 000
100
'001", '005', 0003', '009'
100
Operand2
250
200
1192,'384
Operand1
'288','576' '576'
150
h 29.1647
'336', '672",
result = 0 (mod 8)
post
100
'96', '96','224 '224',
'608", '768'
0
50
100
150
200
250
Operand2
250
200
'15', '5',
Operand1
'015', '535',
150
747
'764', '3',
restlt pattern result 0.5
post
100
'762', '021', '4', '13'
50
100
150
200
250
Operand2
250
200
'369', '271',
Operand1
'761', '762'.
150
16,360
'522', '918'.
pattern op2 ..8
post
100
'972', '783'.
'325', '531'
50
Operand2
24

---

Published as a conference paper at ICLR 2025
Table Examples of heuristic neurons for the multiplication operator
Top-10 numerical logits value vector out
Neuron number h hpost
Activation pattern
Classified Heuristics
250
200
'320', '360',
Operandi
150
'770', '630'.
result = (mod 5),
31,8084 h
post
'820', '510',
pattern ..5,
100
'420', '240',
pattern 010
'370', '380'
50
100
Operand2
250
200
Operands
'585', '936',
150
'546'.'637: '67',
opl = (mod 13), op2 = 0 (mod 13), result = 0 (mod 13)
30,9988 h
'975', '455',
post
100
'910', '819', '715', '364'
50
50
100
150
200
250
Operand2
250
200
Operand1
'170', '162',
21,12521
150
'171', 169',
'17" '17','150 '156', '160'. '66' '785', '168'
rangeresule [150, 180),
post
resulf = 0 (mod 2)
100
0
50
100
150
200
250
Operand2
250
200
Operandi
'844', '848'
150
'873'. 1985'.
16,2956 h
'548', '858'.
pattern op2 0.1
post
100
'657', '995'.
'788', '716'
0
50
100
150
200
250
Operand2
25

---

Published as a conference paper at ICLR 2025
Table Examples heuristic neurons for the division operator
Top-10 numerical logits value vector out
Neuron number h ppost
Activation pattern
Classified Heuristics
250
200
'5', '005',
Operandi
h posst
30,3168
150
'891'. '349',
'05', '948'.
pattern resuts 005, rangeresult E [4 6]
100
'324', '544'
'905', '801'
Operand2
250
200
Operands
'23', '33',
150
'38', '08"
range op2 [0, 10).
h 2402
'22'. '00',
range E (0, 100),
post
100
'26', '30',
pattern ..
'07', '32'
50
50
100
150
200
250
Operand2
250
200
'8', '446', '770', '7'
Operand1
24,9306 h
150
'740', '674' '449', '741'. "433"
multi-valuereeutt [1,2,7,8)
post
100
50
100
150
200
250
Operand2
250
200
Operandi
'568', '316'.
'773', 405'.
17,10704
'621', '822',
pattern opt ..., pattern
post
100
'582', '549'.
'587', '598',
100
150
200
250
Operand2
26

---

Published as a conference paper at ICLR 2025
Table 8: Examples of neurons that failed to classify as one of the defined heuristic types
Top- 10 numerical
Neuron number
h Lm post
Activation pattern
logits in value vector
v An out
Operand2
0
50
100
150
200
250
100
'839', '568', '216', '636', '877', '786', '832', '084', '536'
h 22,1 2024
Operand1
post Addition
150
200
250
200
Operand1
12.12 post
150
'344', 144'', '509','4', '456', '445', '2', '333', '5'
Subtraction
100
50
100
Operand2
250
200
'270', '40'' '55'' '454', '451', '570', '290', '28'' '470', '439'
h 27,4900
150
post
Multiplication
100
50
50
100
150
200
250
Operand2
250
200
1983', '801'
post
20.11020
150
'731', '501' '751'. '070', '663', '985', '713'
Division
100
50
100
150
200
250
Operand2
27


Reviews:
[{"review_id": "4lRTbYnFhP", "reviewer": "Reviewer_Tumw", "summary": "The paper uses causal analysis to understand how (small scale) LLMs solve arithmetic. They identify that the model's output really depends on only a small subset of its parameters which they call a circuit and use this to explain its behavior. They find that each neuron in this circuit implements some form of a heuristic and the model's overall behavior can be explained as a collection of these simple heuristics. They run extensive experiments and analysis to validate their claims and show that somewhat surprisingly, the model is neither relying purely on memorization or learning a robust generalizable algorithm for addition or multiplication.", "questions": "1. In Figure 3, is the position of Operator 1, Operator 2 a range? Or are the inputs only up to the point where it is a single token?\n2. Am I right in understanding that once the circuit is identified, the network can be replaced with a highly sparse version of the same network? (Say 98.5% sparse where the fixed components can be replaced by biases?)\n3. Does the bag heuristics finding imply that length generalization for these models is impossible?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Arithmetic tasks are a particularly useful test bed to understand the learning mechanism of LLMs. However, it has been an open question as to what algorithm the model is truly learning. This work makes an important step in this direction, particularly since it applies to the newest edition of LLMs.\n2. The bag of heuristics finding explains the lack of length generalization in most LLMs on arithmetic tasks.\n3. The experiments and analysis are extensive and the authors make an effort to validate all of their claims sufficiently.\n4. The paper is well written with the ideas explained clearly with good intuition.", "weaknesses": "1. A lot of the analysis assumes that the tokenizers all tokenizers numbers as a single token up to some limit. While I found this interestingly to be true for llama3 and pythia, I don't think this is always true. In fact, the llama2 tokenizer itself was different in that it tokenized each digit individually. I would like to see if these findings are tokenizer specific.\n2. The sampling mechanism of activation patching seems to be of fairly \"high-variance\". Sampling a \"random counterfactual prompt\" from the universe of prompts seems to me like it would lead to widely different results. Is this true?\n3. The difference between the total logit contribution of the heuristc neurons to the correct answers vs the incorrect answers is fairly small. And thus it is plausible that this is not the only (or even main) failure mode of the bag of heuristics.\n4. The experiments on tracking development across training steps is on a model with a very different tokenizer and much worse performance than Llama3-8B. I understand that this is necessary since llama3-8b's training data is not public, however it does mean the results are slightly harder to compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r0OSPQujBJ", "reviewer": "Reviewer_gBdd", "summary": "This paper explores the reasoning mechanisms behind arithmetic calculations in large language models (LLMs). The authors use causal analysis to identify an \"arithmetic circuit\" responsible for these calculations, finding that LLMs apply a set of heuristic-based neurons rather than true algorithmic methods or pure memorization. Each neuron in this circuit encodes simple, pattern-based heuristics that activate for specific operand patterns and contribute to the model's ability to solve arithmetic prompts. The paper provides a comprehensive examination of these heuristics and their emergence over training time, shedding light on the limitations and generalization abilities of LLMs in mathematical reasoning.", "questions": "The classification of neurons into heuristic types was manually conducted. Do you think if it is possible to develop an automated or semi-supervised system to reduce human bias?\n\nYou mention that the \"bag of heuristics\" mechanism evolves gradually during training. Have you observed whether certain heuristics are reinforced at specific training stages?\n\nYour analysis reveals cases where the model’s heuristics do not generalize to certain arithmetic prompts. Could you provide more insights into these failure cases? For instance, are there specific numerical ranges or types of prompts where the model tends to fail more frequently?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper applies causal analysis and activation patching experiments to identify individual neuron behaviors.\n\nThe authors' step-by-step examination, from identifying neurons and classifying them into heuristic types to analyzing their evolution over training time, offers a thorough perspective on LLMs' arithmetic reasoning mechanisms.\n\nThe paper may facilitate future studies on generalization abilities.", "weaknesses": "This paper focuses on arithmetic calculation but mentions general reasoning at the beginning. I feel that this could be a overclaim because simple one-step calculation is far away from reasoning.\n\nThe authors get all conclusions from pre-trained checkpoints. But I am thinking that all models need fine-tuning before usage, it is better to have similar analysis on arithmetically or generally fine-tuned LLMs.\n\nThe focus is primarily on arithmetic reasoning, which may not fully generalize to other complex reasoning tasks. It is interesting to see the model behavior when arithmetic calculation is embedded in texts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HYVpuTQqO2", "reviewer": "Reviewer_H4kU", "summary": "It is unclear how large language models solve reasoning tasks. For example, do they simply memorize the solution or learn generalizable reasoning processes. This papers demonstrate language models use a set of heuristics to solve arithmetic (a task where we expect a successful learner to learn generalizable algorithms). The paper characterizes the different types of heuristics being implemented as how heuristics are developed during training.", "questions": "- Eq. 2, pg. 4, why are the authors measuring faithfulness by averaging the percentage change in logits? It seems that a more principled approach would be checking how much the correctness of the model (measured through accuracy) decreases as a result of mean ablation. Did the authors ablate over this design variable?\n- In Fig. 7, I am confused by what the x-axis is here. It would make sense that it the accumulation of heuristics, but that is not what the axis label implies. Could the authors elaborate on this a bit more?\n- In line 321, the authors use $t=0.6$ as the intersection threshold, why was this hyperparameter specifically chosen, was it based on the alignment with manual insepction? I do not see ablations over this parameter in the appendices.\n- In lines 408-410, “incorrect prompts have more heuristic neurons associated with them than correct prompts” how are the authors measuring this involvement? Is this through the same sort of patching that was done before, but in this case how are the authors patching the neurons for a baseline incorrect response?\n- It has been shown in some recent works that logic lens in very often unreliable see [1]. Have the authors tried other hidden state decoding methods like [1,2,3]? I wonder if this could be used to better classify some of the heuristics that did not fall into any of the identified categories.\n- In lines 347-355, the authors perform knockout experiments. I am wondering if we knockout say a neuron associated with the heuristic $\\textrm{op}_1 \\equiv 0\\mod2$ that significantly contribues sto the output, we not only see a decrease in performance, but also a *decrease in performance on inputs specifically where the first operand is even?* This would provide even stronger evidence for not only the importance of the heuristics, but also the claimed functionality of each of them.\n    - Is this what Fig. 7 is trying to show, specifically, the importance of the “associated prompts?”\n\n[1] Belrose, Nora, et al. \"Eliciting latent predictions from transformers with the tuned lens.\" *arXiv preprint arXiv:2303.08112* (2023).\n\n[2] Pal, Koyena, et al. \"Future lens: Anticipating subsequent tokens from a single hidden state.\" *arXiv preprint arXiv:2311.04897* (2023).\n\n[3] Ghandeharioun, Asma, et al. \"Patchscope: A unifying framework for inspecting hidden representations of language models.\" *arXiv preprint arXiv:2401.06102* (2024).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- For the most part the paper was well-written and well motivated. At any given point in the paper, it was easy to understand (1) what the hypothesis being tested is, (2) why the authors are testing this hypothesis, (3) the experimental setup, and (4) the results.\n- The experiments conducted by the authors are thorough and well-support the claims made throughout the paper.\n- The topic of the paper is timely and illuminates an interesting aspect of arithmetic reasoning in language models that has not been explored before. I think this will add value both to those working in mechanistic interpretability (because of the clean experimental designs) as well as those working in NLP in general (as it provides an important case study).\n- The paper's results are replicated across models of different scales as well.", "weaknesses": "- The title of the paper is slightly misleading: “Models Sovle Math with a Bag of Heuristics” in reality the authors only examine arithmetic where both operands have less than three digits. Maybe for greater number of digits, these heuristics are somehow algorithmically combined? This question is left open in the paper (which is fine, I think the paper is a good contribution as is), but the title does not reflect this.\n    - In the very least, I think this point should be mentioned briefly in the limitations section.\n- There are some experimental details which are shoved under the rug which I cannot find the appendices (see below). I think adding (probably in the appendices) these would greatly help the reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CDGb5HJJBI", "reviewer": "Reviewer_jXdJ", "summary": "This paper investigates how LLMs solve arithmetic tasks, questioning whether they rely on robust algorithms or if they instead depend on memorization or simple heuristics. Using methods like activation patching and linear probing, the authors identify a subset of neurons responsible for arithmetic tasks in LLMs, which they term the “arithmetic circuit.” The study reveals that LLMs rely on a \"bag of heuristics\" formed by a small subset of individual neurons, each triggered based on specific input patterns. Through various experiments and ablations, the paper demonstrates that this heuristic-based mechanism is the primary method for handling arithmetic prompts, even from early stages of model training. The authors conclude that LLMs’ arithmetic capabilities do not stem from robust algorithms or mere memorization, but from a combination of these lightweight heuristics.", "questions": "1. How would the proposed heuristic-based mechanism scale with model size or complexity? Does the number or type of heuristics vary significantly with larger models?\n2. Would small models trained from scratch on arithmetic tasks also rely on a \"bag of heuristics\"? Or would they have a different mechanism to learn arithmetic? One suggestion is to train a small model (like using a NanoGPT framework) from scratch and apply the same analysis. \n3. Are there any insights into how these heuristics interact or overlap in other reasoning contexts, such as multi-step reasoning tasks? Does the mechanism generalize across tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper makes a valuable contribution by providing a mechanistic interpretation of how LLMs handle arithmetic tasks, adding a nuanced understanding of model behavior beyond existing studies on memorization and algorithmic capability. By identifying individual neurons and describing them as \"heuristic carriers,\" this study offers a novel perspective on LLM internal operations.\n2. The authors clearly articulate their methodology and conclusions. The visualizations, especially those illustrating neuron activation patterns, support comprehension and substantiate the paper’s core claims about the heuristic-based arithmetic mechanism.\n3. The paper employs causal analysis techniques to dissect the arithmetic circuit in LLMs and validate its findings with multiple ablation experiments. The robustness of this methodological approach makes the proposed hypothesis and claims convincing.", "weaknesses": "1. While the study primarily focuses on arithmetic tasks, it does not fully explore whether this heuristic-driven mechanism applies to other types of reasoning. Future work extending the approach to different reasoning domains would help determine the generalizability of these findings.\n2. Classifying neurons into heuristic types may oversimplify the actual complexity of neuron interactions in LLMs. Additionally, the manual classification of heuristic types could introduce subjective bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipWhepVoYs", "reviewer": "Reviewer_fm6H", "summary": "This paper characterizes which neurons correspond to LLM performance on arithmetic problems. First, it demonstrates that only a few attention heads in early layers are important, while MLP layers in later stages play a significant role. Through a more detailed analysis using methods like probing, the study identifies several crucial neurons in MLPs, discovering that certain operands can activate these neurons. Finally, the paper introduces the concept of \"bag of heuristics\" for these operand-activated neurons, claiming that this mechanism can account for how LLMs solve arithmetic problems.", "questions": "See weaknesses. Another question is about eq(3), can you explain h and v based on SwiGLU equation in https://paperswithcode.com/method/swiglu?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper employs numerous experiments to validate its findings, and the discovery of neurons activated by specific operands is intriguing.", "weaknesses": "Despite applying various methods and experiments, the main contribution and conclusions drawn from the experiments are overclaimed and unclear. I summarize my primary concerns as follows:\n1. The paper predominantly focuses on the MLPs in later layers, specifically on the last token (\"=\" sign), while neglecting the attention module in early layers. However, the representation of the last token in subsequent layers heavily depends on attention from early layers. Understanding how transformers utilize the attention mechanism for arithmetic problems is arguably more crucial than examining the function of subsequent layers.\n2. While arithmetic equations have approximately 1,000,000 combinations, this study only samples 100 prompts for most of its results. This limited sample size introduces significant randomness and undermines the validity of the findings.\n3. A critical issue is that this paper fails to provide a clear explanation of how LLMs solve arithmetic problems. For instance, we CANNOT explain how a transformer solves a problem like \"21 + 17 =\" in **more detail** than: \"The LLM computes a representation for the last token based on the context in early layers, and then activates certain neurons to obtain the final result in subsequent layers.\"\n4. The conclusions drawn from the experiments are often overclaimed and unclear. For example, in Section 4, the heuristic matching method is confusing. The results from this method don't seem to provide more useful information than the activation patching experiments. It appears that neurons from any model obtained by this method would likely show results similar to those in Figures 8 and 9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)