How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
Authors: Michael Hanna
Ollie Liu
Alexandre Variengien
Venue: NeurIPS 2023 poster (2023)
Keywords: interpretability
language models
NLP
TL;DR: We use mechanistic interpretability techniques to find a generalizable circuit that explains GPT-2's ability to compute the greater-than operation.

Abstract: Pre-trained language models can be surprisingly adept at tasks they were not explicitly trained on, but how they implement these capabilities is poorly understood. In this paper, we investigate the basic mathematical abilities often acquired by pre-trained language models. Concretely, we use mechanistic interpretability techniques to explain the (limited) mathematical abilities of GPT-2 small. As a case study, we examine its ability to take in sentences such as "The war lasted from the year 1732 to the year 17", and predict valid two-digit end years (years > 32). We first identify a circuit, a small subset of GPT-2 small's computational graph that computes this task's output. Then, we explain the role of each circuit component, showing that GPT-2 small's final multi-layer perceptrons boost the probability of end years greater than the start year. Finally, we find related tasks that activate our circuit. Our results suggest that GPT-2 small computes greater-than using a complex but general mechanism that activates across diverse contexts.

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
Michael Hanna" ILLC University of Amsterdam hanna@uva.nl ml
Ollie Liu" University of Southern California zl il22998Qusc. edu
Alexandre Variengien" Redwood Research alexandre. wwwiengien@gmailll.............. com
Abstract
Pre-trained language models can be surprisingly adept at tasks they were not ex- plicitly trained on, but how they implement these capabilities poorly understood. In this paper. we investigate the basic mathematical abilities often acquired by pre-trained language models. Concretely, use mechanistic interpretability tech- niques to explain the (limited) mathematical abilities of GPT-2 small. As case study, we examine its ability take in sentences such as "The war lasted from the year 1732 to the year 17", and predict valid two-digit end years (years > 32). We first identify a circuit, a small subset of GPT-2 small's computational graph that computes this task's 's output. Then, we explain the role of each circuit component, showing that GPT-2 small's final multi-layer perceptrons boost the probability end years greater than the start year. Finally, we find related tasks that activate our circuit. Our results suggest that GPT-2 small computes greater- greater-than using a complex mechanism that activates across diverse contexts.
1 Introduction
As pre-trained language models (LMs) have grown in both size and effectiveness, their abilities have expanded to include wide range of tasks, even without fine-tuning [6]. Such abilities can range from translation to text classification and multi-step reasoning [52]. Yet despite heavy study of these
models 41 42 26
how LMS implement these abilities is still poorly understood.
In this paper, study such LM ability, performing mathematics. Mathematical ability has long been of interest natural language processing: models have been trained to perform tasks such as simple arithmetic and word problems [51 (66.. Researchers have also fine-tuned pre-trained LMs on
these tasks, instead of training from scratch [1 255 Recently, however, LMs seem to have acquired
significant mathematical abilities without explicit training on such tasks
16
How these mathematical abilities arise in LMs is largely unknown. While studies have investigated existing work is behavioral: explains what models can do, rather than how they do it. Most work that delves into model internals does so using models trained directly such tasks: Hupkes al.
pre-trained LMs' mathematical abilities 32 38
23 probe such models for hierarchical structure, while Liu et al. [27] and Nanda et al. 31 study toy models trained on modular addition. Some studies do examine the structure of number representations pre-trained models 30 49 however, they do not provide causal explanation about how these models leverage these representations to perform math. The mechanisms underlying pre-trained LMs' mathematical abilities thus remain unclear.
'Work performed as part of Redwood Research's REMIX program Work performed during an internship. Now at Conjecture
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input: "The war lasted from the year 1732 to the year 17"
GPT-2 small:
00 x
12 x
32 x
33 V
63 V
99 V
Figure I: Year-span prediction example (XX=17 and YY=32) with sample (in)valid output years.
To understand the roots of these mathematical abilities, we study them in GPT-2 small
[39] which
we show still possesses such abilities, despite its small size. This small size enables us investigate its mathematical abilities at a very low level. Concretely, we adopt a circuits perspective 35 14], searching for a minimal subset of nodes in GPT-2's s computational graph responsible for this ability. To do so, we use fine tineggained, causal methods from mechanistic interpretability. that allow us to identify nodes in GPT-2 that belong in our circuit, and then prove our circuit's correctness, through carefully designed causal ablations [21] We also use mechanistic methods to pinpoint how each circuit component contributes to the mathematical task at hand. The end result of this case study is a detailed description of GPT-2's ability to perform one simple mathematical operation: greater-than.
Our investigation is structured as follows. We first define year-span prediction, a task that elicits mathematical behavior in GPT-2 (Section 2}. We give the model input like "The war lasted from the year 1732 the year 17"; assigns higher probability the set of years greater than 32. We next search for the circuit responsible for computing this task, and explain each circuit component's s role (Section B). We find a set of multi-layer perceptrons (MLPs) that computes greater-than, our operation of interest. We then investigate how these MLPs compute greater-than (Section 4.. Finally, we find other tasks requiring greater- than which this circuit generalizes (Section[5).
Via these experiments, we accomplish two main goals. First, we find a circuit for greater-than in GPT-2. This brings new mechanistic insights into math in pre-trained LMs, and builds the limited existing work on circuits in pre-trained LMs (50) by examining new task with a wide output space and rich structure. Second, we show that GPT-2's greater-than relies on complex circuit that activates across contexts. This mechanism surpasses simple memorization, but does not reflect full mathematical competence; lies between memorization and generalization. We thus add nuance to the memorization reemorizatoongeeneallizziiooono dichotomy, and take the first step towards a rich characterization of the states in between them.
2 Year-Span Prediction in GPT-2
GPT-2's size is ideal for low-level study, especially with potentially resource-intensive techniques like those in Section [.1] However, this small size poses a challenge: GPT-2 is less capable than larger LMs, which still often struggle with mathematical tasks (29). With this in mind, we craft a simple task to elicit mathematical behavior in GPT-2, and verify that GPT-2 produces said behavior.
Task and Dataset We focus on a simple mathematical operation, theater-than, framed as it might naturally appear in text: an incomplete sentence following the template "The <noun> lasted from the year XXYY the year XX" (Figure D]. The model should assign higher probability to years >YY. We automatically generate sentences using this template. We draw the nouns from pool of 120 nouns that could have a duration, found using FrameNet [I: Appendix lists the full pool of nouns. We sample the century XX of the sentence from (11, (11,...,T 17]. and the start year YY from {02,... .88.
We impose the latter constraints because want GPT-2 be able to predict a target as it would naturally be tokenized. However. GPT-2 uses byte-pair encoding, in which frequent strings more often appear as single tokens 44). Thus. more frequent years multiples of 100 or those in the 20th century-are tokenized as single tokens; less frequent years are broken into two. This causes a problem: GPT-2 could predict "[00]" after "[17]', but " "1700" always tokenized as "[1700]" in normal data and never as "[17][00]"' So, we exclude all single-token years from our year pool. Finally, we want each example to have at least one correct and one incorrect validly tokenized answer, so we exclude each century's 's highest and lowest validly tokenized year from the pool of start years.
Further references to GPT-2 refer to GPT-2 small

---

CPT:2 IrotaNii liies when YY:41
0.35
0.34
0.33
0:22
6:81
0.83
Figure 2: Left: Probability heatmap of GPT-2 for year-span prediction. Y-axis: the sentence's start year (YY). X-axis: the two-digit output year candidate. (X,Y): Mean probability assigned by GPT-2 to output year X given input year Y. Right: GPT-2's average output distribution when YY=41
Qualitative Evaluation We first qualitatively analyze GPT-2's baseline behavior on this task by running it on dataset of 10,000 examples. Each example has noun randomly drawn from 120 nouns, and a year drawn randomly from the 768 valid years from 1000 to 1899. For each YY in .2...., 98], take the average of GPT-2's probability distribution examples with start year YY; we visualize these average distributions in Figure
predicted years, for all
GPT-2 appears to perform greater-than on our year-span prediction task: it creates a sharp cutoff between invalid end years (S YY), and valid end years (> YY). It assigns higher probability to the latter years, though not all of them: 15-20 years after YY, probabilities drop. The exact length of the year-span receiving higher probability likely reflects patterns in GPT-2's training data. In the real-world, and likely also in GPT-2's ss training data, our prompts' nouns, such as "war," "dynasty"" or "pilgrimage", have average durations that GPT-2 may have learned, influencing its output.
Quantitative Evaluation We design two numerical measures of model performance for the purpose of quantitative assessment. Let YY E {02...., .88} be the start year of our sentence, and Py be the probability of a two-digit output year y. We define the following two metrics:
Probability difference: Eysry Pv
Evsty
Cutoff sharpness: Per+1
PYr-1
Probability difference verifies that model output reflects greater-than operation by measuring the extent to which GPT-2 assigns higher probability to years >YY. It ranges from -1 to 1; higher is better. In contrast, cutoff sharpness is not intrinsically connected to greater-than. However, quantifies an interesting behavior of GPT-2: the sharp cutoff between valid and invalid years. In doing so, it checks that the model depends on YY, and does not produce constant (but valid) output, e.g. by always outputting p999) = 1. Cutoff sharpness ranges from -1 to 1; larger values indicate sharper cutoff.
We perform this evaluation with same 10,000-elemen 81.7% probability difference (SD: 19.3%) and a cutoff sharpness of 6.0% (SD: 7.2%). Overall, both qualitative and quantitative results indicate that GPT-2 performs the greater-than operation the year-span prediction task. For more study of GPT-2's behavior on this task, see Appendix A
dataset; this dataset, GPT-2 achieves
3 A Circuit for Year-Span Prediction
Having defined our task, we now aim to understand how our model performs it internally. Since the advent pre-trained models, many methods, such as attention analysis [9] and probing [3], have sought to answer this question. Probing, which trains auxiliary models (probes) to extract information from model representations, has been particularly popular; it has been used to localize syntactic and semantic processing in many pre-trained LMs B7 45 15] However, it has significant pitfalls: most crucially, probes can sometimes extract information from model representations that is irrelevant to model behavior intervene on model internals, and observe changes in model behavior
[ 40 22) To avoid this issue, other work has used causal interventions (36] to
This ensures that our insights about model internals are actually functionally relevant to model behavior.
3

---

In light of this, we examine GPT-2's task performance by using causal techniques to identify a circuit: a minimal computational subgraph of our model that suffices to compute the task [35 [4 While
many causal methods aim to identify important components (nodes) of models [4%
I 17 the
circuits methodology instead considers important edges. Circuits thus examine not only important nodes, but also their interactions, and how they work together to support model behavior.
Below, we explain the path patching technique, and how use it to find circuits (Section 3.1 We then find a circuit for greater-than, and prove its correctness (Section 3.2). Finally, we assign semantics to the nodes and edges of our circuit (Section B.3). All of our experiments use the cirtuii library. and our code is available at https: / gi thub. com/ hwwp/ww/naaaaaaaaaaa www.aaawwwgttgggaagaaee information on how to apply this methodology other problems, see Appendix
greeeerreeer--rr than For
3.1 Path Patching
To find a circuit, we use path patching. introduced by Wang et al. [30] and further described by Goldowsky-Dill et al. [21]. This technique determines how important model component (e.g. an attention head or MI.P) to a task, by altering that component's inputs and observing model behavior post-alteration. It is much like causal mediation analysis or interchange interventions [48] 17]: however, unlike these, it allows us to constrain our intervention's 's effects to a specific path.
To illustrate this, consider model's forward pass on its inputs as a directed acyclical graph. Its nodes are components such attention heads or MLPs. The input of node o the sum of the outputs of all nodes with direct edge to c. GPT-2 can be thought of as such graph flowing from its input tokens to its logits (and thereafter, its probabilities), as depicted in Figure 3
In path patching, we specify new input tokens, and a path of components through which they will reach the logits. For example, if want to ascertain the effects of MIP 10 on the logits, we can patch the direct path (MIP 10, logits) with new input, which we call the 01-input: "The war lasted from the year 1701 to the year 17". We thus alter MLP 10's direct effects changing its output to the attention and MLP of layer 11 (Figure 3 If the model's behavior (as indicated by its logits) changes, we can be sure that this is because MLP 10 is important to that behavior; it is not due to downstream components. Earlier methods like interchange interventions lack this distinction when they alter a component, they affect all components downstream from it.
the logits without
The specificity of path patching allows us to test detailed hypotheses. For example, imagine that we know that MLP 10 affects the logits both directly and via its effects MLP 11. We want to know how important layer 10's attention is to the circuit via MIP 10. We can test this by patching two paths at once: (Attn 10, MLP 10, logits) and (Attn 10, MLP 10, MLP 11, logits), as in Figure 3 This allows us to pinpoint the relationship between precisely these two components, Attn 10 and ML. 10. This technique underpins our circuits approach: we search for a path starting in the inputs and ending in the logits that explains how our model performs the greater-than task.
To perform path patching, we need a new dataset that replaces node's original inputs. To this end, we create the "O1-dataset", we take each example in the original dataset and replace the last two digits YY of the start year with "01". If component normally boosts logits of years YY, patching it with the 01-dataset will cause it to boost the logits of years 01, inducing a larger error in the model.
Loght
B
Logtts
Loghs
Normal Inpu
Attention Layer
MLP 11
Alhention Laye-
MUP 11
Altention Laver
MLP
Attentoo Layer
MLP 10
Abarizan
MLLP 10
Attomtion
Altentin
MLP 10
Lager
Layer
Layer
Token Entaddiigg Layers 1:9
Token Embeddingg Lapers 1-8
Tokan Embtait rga Lavers 1-9
Token Emoedsings
Token Embecdings Layors 1-8
Layers
Iroul: wall insted year
houu: "The war lnrted from The year 1732 to the 17
Input: "The war losted from the yoar 1701 x the your 17"
Incue "The wall usted from year 1752 to the wear
Iiput: "The war lastud flom year
year
Figure 3: A. The computational graph of GPT-2, (MLP 10, logits) path is patched to receive 01-input. C. GPT-2, where the (Attn 10, MLP 10, logits) path receives 01-input. Nodes receiving normal input have blue output; nodes receiving 01-input have red output; nodes receiving both have purple output. Note that in B and C, there is no longer an edge connecting MLP 10 and the logits.
on our normal dataset. B: GPT-2, where the

---

Direct Constitttonn via
IPP: Direct Contributions the Logits
MC-12
proto dill vee inliiaaionn
Lager
Lepet
2
10 11 mb
Head
Figure 4: Iterative path-patching (IPP) heatmaps. Y-axis: layer of the component. X-axis: attention head number, MLP. (X.Y): Change in probability difference induced by patching the corresponding component. A: Heatmap for the path ((X,Y), logits). B: Heatmaps for MLPs 8-11.
3.2
Circuit Components
MLPs We search for circuit by identifying components that perform year-span prediction via their direct connection to the logits. We consider as potential components GPT-2''s 144 attention heads (12 heads/layerx layers), and 12 MLPs (1 per layer). We do so because the residual stream [14] that serves as input to the logits is simply the sum of these components' direct contributions (along with the token embeddings; we ignore these they contain no YY information). If we consider each of these, we will not miss any components that contribute to this task. For details, see Appendix C
We iteratively path patch each component's direct contributions to the logits, replacing its inputs with the 01-dataset, In our earlier notation, for a component of interest C, we patch the path (C, logits), in Figure 3 B, where C = MLP 10. We patch only one component the sentence; at other positions, these components cannot affect the logits directly.
time, and only at the end of
After we patch component, we run the model and record the probability difference, comparing it to that of the unpatched model. If patching a component caused model performance to change significantly, that component contributed to the model's computation of year-span prediction.
Figure shows the results of this experiment; for computational reasons we run it using a smaller dataset (490 datapoints. 5 per year YY). The heatmap indicates that MLPs 8-1 are the most important direct contributors to the logits, along with a9.hl: attention layer 9's head 1. However, the MLPs cannot act alone: to compute years:>YY, YY. these MLPs at the end of the sentence must know the value of YY. But unlike attention heads, MI.s cannot attend to earlier tokens such as the YY token. Thus, we search for nodes that contribute to the circuit via these MLPs.
Attention Heads We find components that contribute to the circuit via the MLPs using more path patching. We start by patching components through MLP 11, since it is the furthest downstream; for a component of interest C, we patch (C, MLP 11, logits). We find that MLP 11 relies mostly on the MLPs upstream of it (Figure so we search for components that act via those MLPs.
We next find components that contribute to the circuit through MLP 10. For given C, we patch (C, MLP 10, logits) and (C, ML. 10, MI.P 11, logits), as in Figure 3 C. We do so because MLP 10 contributes directly two nodes in our circuit, the logits and MLP I1, and we want know which nodes contribute via MLP 10 to the entire circuit. We repeat this procedure for MLPs 9 and 8.
The results in Figure indicate that MLPs rely heavily other MLPs upstream of them. MLPs 8 and 9, the furthest upstream of our MLPs, also rely on attention heads. MLP relies on a9.hl, while MLPS 8 relies a8.h11. 1, a8.h8, a7. h10,, a6.h9, a5.h5, and a5.h1; we add these these attention heads can be seen to contribute to the logits directly, though more weakly than the MLPs do. For this reason, we also add these heads' direct connections to the logits to our circuit.
circuit. Many of
Figure 5 visualizes the circuit we have found. We could further develop this by specifying circuit from the token inputs to the logits; indeed, we do so in Appendix However, the present circuit
5

---

Logis
lngis
Covestraa aber to heads jeseept hij
Both Inouns
Inpul
md miQ m1?
MLPs
15i11, is ha, ashi, RAM a7hin abhil aa
Ancrrion Heads #sP11
Heads 210, all hi et:)
upprnamm Attention Higoos)
Updreem
Upsteam Nodes
lepu:: The war lasted from Uhe yoar 17VY
Insu:: "The wer lnsted from your 1703 to the your 37"
Imput The war serpec from the year 1732 yeur 17'
Figure 5: Left: Diagram of the year-span prediction circuit. Center: Diagram showing which GPT-2 components receive our standard dataset vs. our 01-dataset in the circuit evaluation experiment. Right: The probability heatmap (as in Figure 2 for the patched model.
already captures the most interesting portion of the model: the MLPs that compute greater-than, So, we instead provide evidence that our circuit is correct, and then analyze its constituent parts.
Having defined our circuit, we perform another path-patching experiment to ensure it is correct. In this experiment, we give most of the model inputs from the 01-dataset. The model only receives our standard dataset via the paths specified in our circuit. So, our attention heads' contributions to the logits are backed by the standard dataset, as are their contributions to the MLPs, and the MLPs' contributions one another. But, some components of the MLPs' inputs (those that come from model components not in the circuit) receive input from the O1-dataset as well. We stress that this is difficult task, where the large majority of the model receives input that should push it to poor performance. For a diagram of the circuit and our evaluation, see Figure
Evaluation
We perform this evaluation using the larger dataset, and almost entirely recover model performance. The probability difference is 72.7% (89.5% of the original) and the cutoff sharpness 8% 8%-sharper than pre-patching. This indicates that circuit is mostly sufficient compute this task. The circuit is also necessary: performing the opposite of the prior experiment, giving nodes in our circuit the 01 01-dataset, and those outside it the normal dataset, leaves GPT-2 unable to perform the task: it achieves a probability difference -3..6%.
If we target other circuits of size and location similar to our circuit's, ss, performance is related to the preservation of the paths from the input, to our attention heads, our MI.s (especially MLPs and 10), to the logits. If the paths are interrupted (i.e. attention head or no MLP overlap with the original circuít), performance is very low. at least one path is preserved, performance improves with each additional component in common with the original circuit; MLPs have the biggest impact.
3.3 Circuit Semantics
Now, we interpret each circuit component, starting with the attention heads. We first perform a simple attention-pate patter-ptttmntmte analysis of the heads in our circuit. Figure which tokens our attention heads attend to at which positions. At the relevant (end) position, in-circuit attention heads attend to the YY position, suggesting that they detect the year which the output year must be greater than.
L000
Figure Attention patterns of a7.h11 and a8.h10. <bos> denotes GPT-2's start of sentence token.

---

Logit ol
Loght
Logit lens of a7.h11 and a8.h10. Axes as in Figure blue indicates that the module upweights the output year, and red, that it downweights the year.
Figure 7:
Next, we examine the contributions of attention heads using the logit lens approach each head's output by GPT-2's unembedding matrix, translating this output into unembedding (vocabulary) space. Note that here, we do not only view logit lens tool for obtaining intermediate estimates of model predictions [4. Rather, we also use it to understand components' outputs more generally: the logit lens can capture how such outputs shape model predictions, but can also capture how these outputs add information to the residual stream in unembedding space.
33 we multiply
We visualize the heads' outputs for each sentence in our small dataset in Figure 7 Attention head outputs for a sentence with start year YY have a high dot product with the embedding vector for YY, as shown by the blue diagonal in the plots; this indicates that the head upweights YY, making it more likely output. Note that YY was not just upweighted highly compared other 2-digit years; YY was the most highly upweighted token across all tokens. Given our earlier analysis, we therefore hypothesize that these attention heads identify the start year (at the YY position), and indicate it via spike in unembedding space of the residual stream at the end position; they thus communicate to downstream components.
We similarly apply the logit lens to the outputs of MLPs 8-11 (Figure 8 The results indicate that MLPs of 9 and 10 directly specify which years are greater than YY: the logit lens of cach layer's s output has an upper triangular pattern, indicating that they upweight precisely those years greater than YY. MLP 11 plays a similar role, but seems to upweight roughly the first 50 years after YY, enforcing maximum duration for the event in the sentence. However, MLP 8 unusual: its logit lens shows diagonal pattern, but no upper triangular pattern that would indicate that it computes greater-than than.
We claim that this is because MLP 8 contributes mainly indirectly, via the other MLPs in our circuit. We confirm this by patching MIP 8's direct contributions to the logits with the 01-dataset; we do so again with its indirect contributions, through the other MLPs. In the former case, model performance drops only 14%, while in the latter case, it drops by 39%. So MIP 8 does not contribute much to the logits directly, but it does contribute indirectly. Other MLPs also have mixed effects: MLP has roughly equal direct and indirect contributions (28% vs. 32%), while MLP 10 contributes mostly directly (56% vs. 16%). MLP 11 can only contribute directly.
Our full picture of the circuit so far is this: the attention heads communicate the start year YY in embedding space. MLP 8's mostly influences downstream MLPs. However, MLPs 9, 10, and 11 appear to compute the greater- than operation tandem, and in steps. We conclude that while the attention heads identify the important year YY, it is the MLPs that effect the greater-thar computation.
Figure 8: (Left to right) Logit lens of MLPs 11, 10, 9, and 8; labels as Figure
7

---

Fr teterip
maith
Figure 9: PCA of MI. 8's input, a7.h10's and a7.h8's output, and the static year embeddings. Each point corresponds to one datapoint's representation, and is labeled with and colored by the its YY.
4 Explaining Greater-Tie -han in the Year-Span Prediction Circuit
Our prior experiments show that MLPs 9-11 directly compute greater-than. But how do they do so? We cannot provide a conclusive answer, but identify avenues by which MLPs might compute this. We first examine their inputs, finding structure that might enable greater-than computation. Then, examine MLP internals, showing how neuron composition could enable greater-than computation.
4.1 Input Structure
To understand how MLPs compute greater-than, analyze various model representations using 2D Principal Component Analysis (PCA). For each of the 97 datapoints in our small dataset, each with unique start year, we analyze the input residual stream our MLPs, as well as the output of relevant attention heads. As control, we also analyze representations from irrelevant model components, and the static year embeddings. We take all component representations from the end position.
In Figure 9 PCA reveals that the imput residual stream to MLP 8 (and indeed all of our MLPs, though not all shown) highly structured: representations are ordered by the start year of the sentence they are from. increasing clockwise. The same is true of the outputs of relevant attention heads (a7.h10), which serve as inputs to the MLPs, but not of outputs of irrelevant heads (a7 h8). This suggests that it is specifically the relevant attention heads that transmit this structured information to relevant MLPs. But while the heads seem transmit this structured information the MLPs, they need not have created this structure from scratch. We find, as in Wallace et al. [49], that structure already exists in the static year embeddings, though the years 02-09 are clustered apart from the rest. The heads need only unify these groups and move this information from the YY position to the end.
Structured number representations have been implicated in mathematical capability before: Liu et al. 27 train a toy transformer model on modular addition, and find that its number representations become structured only after it stops overfitting and begins to generalize. This suggests that GPT- 2's structured number representations may be relevant to its greater-than ability. However, our experiments struggle prove this causally. When ablate the dimensions found through PCA, to test their importance to the greater-thai task, we found little change in model performance. Similarly, removing Emearly-extractate only slightly lower probability difference (64.7%) than the baseline (81.7%). This indicates that structured YY information may not fully explain GPT-2's greater- than abilities.
YY information from attention head output using LEACE (5) yielded
4.2 Neuron-Level Processing
To better understand MLPs, turn to their internals, zooming in on their neurons. We choose to study MLP 10 closely, as we know directly contributes to the greater-thai operation. We start by asking which of MLP 10's neurons important-4 sort of question already studied using probing 10 [2],
causal ablations 2 24 48), and other techniques; see Sajjad et al. 43 for an overview.
As before, we use path patching because it provides precise causal insights. We path patch each of ML.P 10's 3072 neurons' direct contributions to the logits with the 01-dataset. We again record the change in model performance, measured by probability difference, compared to the unpatched

---

LIgH
5-0 01
Predecd
Pretoie
Prodood
Figure 10: Left: The logit lens of the 3 MLP 10 neurons most important to year-span prediction. Right: The logit lens of the t0p-10 MLP 10 neurons. Blue indicates that the neuron upweights logits the given input year (y-axis), output year (x-axis) combination, while red indicates downweighting.
model. We find that newron contributions to the task are sparse; most neurons can be patched (ablated) with near zero effect on our model performance, as observed in prior work (48)
We then analyze those neurons that contribute most to model performance using the logit lens. To do this, we take advantage of the fact that each neuron has corresponding row in the MLP output weight matrix. As noted by Geva et al. [19]. multiplying this row by the unembedding weights yields an (unnormalized) distribution over the logits, indicating which outputs the neuron upweights when activated. Taking the outer product of this logit distribution with the neuron's 's activations yields the logit lens, indicating which output years the neuron upweights for each input sentence's YY.
Figure shows the logit lens of the most important neurons in MLP 10; more neurons can be found in Appendix [D] Each neuron up- year YY, but no individual neuron computes greater- than. No one neuron can do so, as each neuron's 's activation for each input only scales that neuron's 's distribution over the logits, without changing its overall shape. In contrast, the correct shape of the logits differs depending on the example's start year.
certain output years depending on the input
Many neurons can compute greater-than when combined, though. We perform logit lens on the sum of the top- 10 neurons' contributions in Figure 10 Though they do not do so individually, the top-10 neurons perform an imperfect greater-than when summed together 10 as a whole can be thought of as the logit lens of the sum all 3072 neurons' contributions; we partially recreate this with the contributions of just the top- 10 neurons. Including more top neurons produces sharper approximations of greater-thany see Appendix D for examples.
group. The logit lens of MLP
In summary, we find that even inside one MLP, the greater-than operation is spread across multiple neurons, whose outputs compose in a sum to form the correct answer. Even the contributions of a small number of relevant neurons composed begin to roughly form the correct operation. We study this in MLP 10, but observe it in the other MLPs as well. Section [33 suggested that GPT-2 computed greater-than across multiple important MLPs; these results suggest that moreover multiple important neurons in each MLP compose to allow the MLP to compute greater-than.
5 Does The Circuit Generalize?
We now possess a detailed circuit for year-span prediction. But one thing remains unclear: is this a general circuit for the greater-than operation? Or does it only only apply this specific, toy task? Answering this question fully would require an in-depth exploration of circuit similarity that fall outside the scope of this paper. For the purposes of this investigation, we perform primarily qualitative analyses of tasks that preserve the format and output space of year-span prediction.
To start, we focus three increasingly different prompts: "The <noun> started in the year 17YY and ended in the year 17". "The price of that <luxury good> ranges from $ 17YY to $ 17", and "1599, 1607, 1633, 1679, 17YY, 17". In all cases, two-digit number greater than YY would be reasonable next token. The model performs greater-than given all of these prompts (2 69% probability difference). Moreover, the circuits found via path patching are similar to those in Sections 3.2 and 3.3 When we run GPT-2 on the first two tasks, ablating all edges not in our original year-span circuit, we achieve 98.8 88 and 88.9% loss recovery respectively; for more details, see Appendix F As before,
We can confirm this using path patching as well; Appendix E for details.

---

these tasks depend on MLPs 8- 1 to compute greater- than; these MLPs depend on attention heads that transmit information about YY.
That said, these tasks' circuits are not all identical to the greater-than circuit. GPT-2 recovers only 67.8% of its original performance on the last when ablating everything but the year-span eircuit. But, closer look at path patching results indicated that on this task, GPT-2 used not only the entire year-span circuit, but also MLP 7 and two extra attention heads. After including those nodes, GPT-2 recovered 90.3% of performance. Similar tasks seem to use similar, but not identical, circuits.
GPT-2 produced unusual output for some tasks requiring other mathematical operations. It produced roughly symmetric distributions around YY on the task "1799 1753, 1733, 1701, 16YY, 16", which might yield predictions smaller than YY. It behaved similarly on examples suggesting exact answer, such as "1695, 1697, 1699, 1701, 1703, 17", which could yield 05. GPT-2 even failed at some tasks that were solvable using the gheater-than circuit, like "17YY is smaller than 17"; it always predicted YY. Across all such tasks, we found that GPT-2 relied on another set of heads and ML.s entirely. So GPT-2 does not use our circuit for all math; sometimes it does not rely on it even when it should.
We also observe the opposite phenomenon: inappropriate activation of the greater-than circuit, triggered by prompts like "The <noun> ended in the year 17YY and started in the year 17" and "The <noun> lasted from the year 7YY BC to the year 7" In these cases, GPT-2 ought to predict numbers smaller than YY; however, it predicts numbers greater than YY. This is because it is using the exact same circuit used in the greater than case! GPT-2 thus overgeneralizes the use of our circuit.
Our results suggest that our circuit generalizes to some new scenarios. But what does a generalizing circuit imply about the origins of GPT-2's greater-than capabilities do they stem from memorization 7 or rich, generalizable representations of numbers [27] Our complex circuit and the structured numbered representations found in our PCA experiments hint at some mathematical knowledge in GPT-2. However, the presence of greater-than circuit does not preclude memorization. Our circuit could function internally as a lookup table, where attention heads transmit YY information to MLPs that then upweight the years that they have memorized to be >YY. In this case, (incorrect) generalization would involve GPT-2 (incorrectly) activating the lookup circuit based on context.
Though our current evidence does not allow us to definitively attribute GPT-2's behavior to generalized mathematical ability or memorization, it suggests that the underlying mechanism is something in between the two. The lack of causal evidence for the role of structured number representations, and the fact that GPT-2 cannot handle related operations like "less-than" or "equallto" argue against generalized math mechanisms. However, even if we view this circuit simply retrieving memorized facts, the retrieval mechanism at work sophisticated. GPT-2 can (imperfectly) identify greater- scenarios and the relevant operand; it then activates dedicated mechanism that retrieves the correct answer. GPT-2's mathematical abilities thus extend beyond a simple, exact memorization of answers.
6 Conclusion
In this paper, we attempted to bridge the gap between understanding of mathematical abilities in toy models, and the mystery of such abilities in larger pre-trained LMs. To do so, we outlined a circuit in GPT-2 with interpretable structure and semantics, adding to the evidence that circuits are useful way of understanding pre-trained LMs. [50] even in a more complex scenario. Our circuit is cearser-grained than findings in toy models for mathematical tasks BII, but much finer-graine greinnr than existing work on mathematics in pre-trained LMs. Moreover, we showed that this circuit in GPT-2 activates across contexts on other tradaa--han-adjaaeen tasks. Whether such cross-context activation reflects generalization or memorization is an open question for circuits work in general.
We note that conclusions are limited by the small size of our model and dataset, and the simple phenomenon studied. Our study is very model-centric: data-driven interpretability techniques would strengthen our work. Studying circuit performance across diverse tasks could better measure the degree to which our circuit generalizes to all wreater-than tasks. Similarly, studying larger models would confirm that our results hold for the models that dominate natural language processing today.
Despite these limitations, we believe that this study lays the groundwork for future work. Our small study hints at the potential for circuits as lens for the study of memorization and generalization in pre-trained LMs. More broadly, we hope that our finding that not only attention heads, but also MIPs and their neurons can be analyzed jointly complex system will motivate circuits work to come.
10

---

Acknowledgments and Disclosure of Funding
The authors thank Buck Shlegeris, Chris MacLeod, and Arthur Conmy for their valuable feedback on earlier drafts of this work. They also thank Neel Nanda for useful research meeting about this work. They additionally thank Dani Yogatama, as well members of the Amsterdam and Technion NLP groups, for their helpful comments. They also appreciate the insightful reviews and discussion provided by the anonymous reviewers for this paper. They finally thank Redwood Research for both running the REMIX program, which provided many of the ideas, techniques, and collaborations behind this paper, and providing continued computational and travel support thereafter.
References
[1] Collin Baker, Charles J. Fillmore, and John B. Lowe. The Berkeley FrameNet project. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguisties, Volume 1, pages 86-90, Montreal, Quebec, Canada, August 1998. Association for Computational Linguistics. doi: 00011////884598008000 URL http://aclanttoooo.orrrr///
[2] Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, 2019. URL httphttts///peeeeiiiiie nwt.:wrutttffmm/ 1d=H12-PsR5XX
[3] Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Com- putational Linguistics, 48(1):207-219 March 2022. doi: 10.1162/coli a 00422. URL
https://aclaathologyy wwwp://wwwwoooooo......oo//.. wttpg////22eeeeoo c1-
[4] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt, Eliciting latent predictions from transformers with the tuned lens, 2023.
[5] Nora Belrose, David Schneider-Jsepph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form, 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates. URL fhtpss::tp.......p.. http://ooooeeeensnneee nwurips.....ee wwcppppp/raaeeeeeeeeeeeeeee files/ www.ffialeeeeeeeeeee /aper//020///ile pdd
Inc., 2020. URL
1457c0d6bi
[7] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski. Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https:///pperreiie.
net/forum?id-Tattiie 1cK
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei,
11

---

Kathy Meier-Heilee Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022.
[9] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence. Italy, August 2019. Association for Computational Linguistics. doi: 10. 116633/11/119-440
URL
org.o.ggoo.. W19-4828
[10] Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, Anthony Bau, and James Glass. What is one grain of sand in the desert? analyzing individual neurons in deep nip models. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01))6309-6317 Jul. 2019. doi: 10.1609/aaai. http:/aaaiv3330...33 v33101.33016309 URL httptt:////sss aaa.oossssaa...............a...a org index php/
AAAT/articleeviee///55
[11] Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3243-3255, Online, November 2020. Association for Computational Lin- guistics. doi: 10.1 2tt55///1122200..mmmmmmmaim emn main. 262
URL https: pttttp::/aacaantholoo org/20g//2200 2020
[12] Nadir Durrani, Hassan Sajjad, Fahim Dalvi, and Yonatan Belinkov. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4865-4888 Online, November
2020. Association for Computational Linguistics. doi: URL
org.org///// 22000.222.... emnlp-main 339
[13] Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9: 9:160-175, 2021. doi: 10.1162/tacl a 00359. URL /ttts:://aclaamnm hatp://aaaaattooggyyy.
org/2021.tac1-11
[14] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodde Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http://ttansfommerriruutsssppppp httpr/attsspuubb//2/aa//eee/r/nn /fwra//wwrr/indee....o html.
[15] Mohsen Fayyaz, Ehsan Aghazadeh, Ali Modarressi, Hosein Mohebbi, and Mohammad Taher Pilehvar. Not all models localize linguistic knowledge in the same place: layer-wise probing on BERToids' representations. Analyzing and Interpreting Neural Networks for NLP. pages 375-388, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 18.18663/v 1/2523 blackboxnlp-1.. URL
Proceedings of the Fourth BlackboxNLP Workshop on
org/ 2021. blackboxnlp-1..
[16] Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of chatgpt, 2023.
[17] Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL forum?id-RRuutt RmuXDtjDhG
net/
[18] Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annuul Meeting of the Association for Computational Linguistics, pages 946-958, Online, July 2020. Association for Computational Linguisties. doi: mai..//mmmmm.mmmmm......999999 URL htttts::/aclaathllllaaao org. 2020
.89
12

---

[19] Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer feed- ferd-forward layers are key-value memories. Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages -495, Online and Punta Cana, Dominican Republic, Novem- her 2021, Association for Computational Linguisties. doi: 10. 18653/v1/2021 http:///22221111-mmmaee emnlp-main.4ar
org/ 2021. emnlp-main 446
[20]
Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 1111165/1//1115444/ URL MRLLtttts///cccaattoll.gg wrg/
[21] Nicholas Goldowsky-Din Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.
[22] Michael Hanna, Roberto Zamparelli, and David Mareček. The functional relevance of probed information: A case study. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 835-848, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. doi: 10. 10.18665/11/22233accmmmiii
URL
https://aclaanthologg www.ooy.oogg/oooo orgg/0233 wwa.:eea.cmmmmm..mmm 58
[23] Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Int. Res., 61(1):907-926, jan 2018. ISSN 1076-9757.
[24] Yair Lakretz, German Kruszewski, Theo Desbordes, Dieuwke Hupkes, Stanislas Dehaene, and Marco Baroni. The emergence of number and syntax units LSTM language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 11 11-20, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. http:://aclaattooog.orrr///////e
doi: 10.18655//1/N19--00000 1002. URL
[25] Aitor Lewkowycz. Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solu Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL wtwparrreiiiinnnfeffooeeee www.:neenoonne... Pid-rmmmmiiiiii 1FT.E//-X11
[26] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring a sequence model trained on synthetic task. In The Eleventh International Conference on Learming Representations, 2023. URL https: https::/opentee httpa:://peereeeeeeeeeee http://ooomiddDee00000000000 TcZvT
[27] Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Informa- tion Processing Systems, 2022. BRLhhttps: htttttt://ppeereeeetee wwwprrvvieweneetffoeeo www/teeuum t d=6at6rB3I2a
[28] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL httpps:/ openreview wwtp::/nnttf.tttteeeeme http://rrrrmmdd--AAAAAeeee h6WAS6eE4
[29] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks. Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Pupers), pages 3505-3523, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. No.18653/1/222...co long....--.666666 URL. https: http:://acaaathhooo org/ 2022 loo...og.... 246
13

---

[30] Aakanksha Naik. Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374 3380, Florence, Italy, July 2019, Association for Computational Linguisties. doi: 10. 1188866311111113333 URJjatpss///cllanooolgg org/
P19-1329
[31] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechamistic interpretabilityy In International Conference Learning Representations, 2023. URL htt.tttt://ppeeeeeeeee.eeee
[32] Rodrigo Nogueira, Zhiying Jiang, and Jimmy J. Li. Investigating the limitations of the trans- formers with simple arithmetic tasks. ArXiv, 0tt/2212213000 2021.
[33]
Nostalgebrist. interpreting GPT: the logit lens, 2020. URL httpt:ttt///// lesswrong. com/ pptts//ccctBsspppallltrerttoroop
gpt- -o--
[34] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021.
[35] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 1
[36] Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009. doi: 10.1017/ EB0978051180316
[37] Matthew E. Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee. and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pages 2227-2233 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. (0.18653/11/118-11 1202. ww8//18----/ 1202
URL MRLLittpp///aclaatoollggg
[38] Jing Qian, Hong Wang, Zekun Li, Shiyang Li, and Xifeng Yan. Limitations of language models in arithmetic and symbolic induction, 2022.
[39] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[40] Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363-3377, Online, April 2021. Association for Computational Linguistics. doi: 10. 18653vvvv 202 1. leaccl-main.29.. URL https: hatpa/aaaaaannoooaaoaaoao wtgpee...gg222..2 eacl main. 295
[41] Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842- 866, 2020. doi: 16.1/// a 00349. URL 54
www.orgg/.2/0.... tacl-1
[42] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks, 2022. URL https
//arxiv.org wrw..oggg/bbbb..... abaaaaa//2200 /207 13243
[43] Hassan Sajjad, Nadir Durrani, and Fahim Dalvi. Neuron-level interpretation of deep NLPP models: A survey. Transactions of the Association for Computational Linguisticss 10:1285-1303, 2022. doi: 10.1162/tac1 a_ 00519. URL http:::///eelaanholllg///
Linguiisis
[44] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1.118833v1/P11-1111 1162. URİ. https: // http://aalaatttn wwlaaaaaaannoooyy.a/mmaaaa ort.//--111111111 1162
14

---

[45] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18655/v1/111-14455 /tt.//tteee//c..lcoooooo...orgg
[46] Avijit Thawani, Jay Pujara, Filip llievski, and Pedro Szekely. Representing numbers in NLP: a survey and vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644- 656, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/20211 naacl-main.53. U
2021. maacl-main. . 53
[47] Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memo- rization without overfitting: Analyzing the training dynamics of large language models. In S. Koyejo. S. Mohamed. A. Agarwal, D. Belgrave. K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 38274-38290. Curran Associates, Inc., 2022. URL httphtttss:pprroeeeddini neurr newrips. cc/ wwwp/papperaae fiwes/llaapae wtlle//aaper/222///111 #000011-a06600222244575777222224 Papee-
pdf
[48] Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 12388-12401. Cur- ran Associates, Inc., 2020. URL
weeeeeeddinnneeuuuiieeeeeeeeeeee neurips wwtpriis.ccppppp//eeeee wwtpp//ppeee/2//////../ pdf
[49]
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307-5315, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 10.18855/11/119-1554 URL https://aclanthologg wwt.:/..gg......... D019-1191511
[50] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability the wild: circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representutions, 2023. URL hattps://openrre net/forum? iia-imiisnodooiiii Note: Path patching was introduced only in the arXiv version of this paper: https://aaxiii orgp//iioiiooroooooooooo abs/ 22111 11111..11 00593
[51] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845 845-854, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/11/D11-10888 URL
http:///cccanttoooooorrrr//////
[52] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL httpss //openreview. httpr//reeee/oormmm////yys/uuaa Survey Certification.
15

---

A More Behavioral Study of Greater-Than in GPT-2
In this section, we discuss the results of additional behavioral studies of GPT-2's behavior.
GPT-2 also predicts valid xX Although previous experiments show that GPT-2 predicts valid two-digit end year YY, we also verify that GPT-2 predicts a valid continuation when the output prefix XX is not provided. In this scenario, GPT-2 should either predict either two-digit century token whose value is 2 XX, or an entire 4-digit year token that is 2 XXYY. We test this using the same dataset in previous experiments, with the final XX removed. We find that average, considering the top-100
100 tokens (95% of probability mass), 94% of their probability mass (89% of the total) is assigned valid continuations. So, GPT-2 generally predicts valid xx.
GPT-2's top predictions are valid As suggested by the high probability difference achieved by GPT-2, its top predictions for an input from our dataset are quite good. 100% of the top-1 continuations, and 98.6% of top-5 continuations were correct.
Default Behavior in GPT-2 GPT-2 predicts the greater- in greater-than scenarios, but also in scenarios where less-than would be appropriate (see Section 5) For this reason, we also tested how GPT-2 behaves in scenario where neither greater-than nor less-than in required, in order to a elicit a sort of default behavior.
To test this, we created sequences of 4-digit numbers like "XXY1, XXY2..... XXYN, XX". We chose XX, YYN randomly for each sequence, respecting tokenization. Sequences were generally not monotone, so any 2-digit continuation YY could have been valid. We found that GPT-2's behavior depends on the sequence length N. At low N, GPT-2 produces mostly numbers > YN; the proportion of continuations < YN increases smoothly until 50% by N = 20. So even in the context of random sequences, GPT-2 often generates increasing numbers.
Beyond simple behavioral study, we conducted a circuits analysis on this task, and found that our greater-than circuit also underlies this behavior. This provides another example of GPT-2's ability to identify greater-than situations flawed. However, it also demonstrates again that whenever we see this greater-than behavior, our circuit is responsible. An open question worth answering is why our circuit activates such incorrect scenarios, and if this can be mitigated.
B
The Full Year-Span Prediction Circuit
Now, we describe the rest of the year-span prediction circuit. This actually not very large, as we already know most of the important components. All that remains is to understand how the input to the attention heads crafted.
We can investigate this via iterative path-patching again: we will look for nodes that influence the attention heads. This can be done in three ways: via queries, keys, and values. The queries and keys jointly determine what the attention heads attend to. In theory, attention patterns should be relatively constant across examples: in all cases, attention heads should attend to the position. If this the case, we should be able to ignore the queries and keys, and focus only the value.
In practice, attention patterns not exactly constant as YY changes. While broad trends similar, the intensity of the attention to the YY position varies (though not linearly with YY, as might make sense: see Figure 11. At YY=01, attention to the YY position is rather low, meaning that model greater-than behavior is less pronounced when patched. Thus, the queries and keys are somewhat important: patching them with bad data reduces performance by 15% with respect our partial circuit. The influences on these heads via the keys are similar to those on the values, discussed in the next paragraph. The influences via the queries are distinct, but we will set these aside, and focus on the value vectors.
The most important influences on these heads are the influences their values the YY position. The values are combined to form each attention head's 's output: patching the values with 01-input entirely disrupts circuit performance, unlike patching its keys queries.
To find influences on these values, we iteratively path patch potential components that might com- municate with our attention heads via their values at the YY position. We find (Figure 12) that these
16

---

Attention a7.hl0 (end position)
0.00
0.75
Artantbon
0.70
0.60
(PP))
Figure 11: Attention from a7.h10 (end position) to the YY position, by input year
Attention Value
0.4
dif variztion
Layer
-02
10
-04
2
3
9
10
11
Head
Figure 12: Iterative path patching results through attention heads' value vectors
Parreggh
Figure 13: Iterative path patching results for MLPs 0-3
17

---

Ful-Crruut Patched GPT-2 Small Probabiity Heatmep
logits
mB, m10,
Boost logits for yy YY
015
Create logit spike
a5.h5, a6.h1, a6.19, a7.h10, a8.h11. a9 h1
K, V
a0.h3, a0.h5
a0.h1
m1, m3
Read in YY
input: "The war lasted from the year 17YY to the year 17"
preckcted your
Figure 14: (Left) Full circuit diagram. Note that grouped MLPs interconnected: attention heads are not. (Right) Probability heatmap for the patched full circuit.
IPP: Direct Contributions to the Logits
Logits
prob dift varistion
Layer
MLP 10
MLP 11
-02
MLP
MLP 9
-44
0
1
2
3
10
a9.h1
Figure 15: Path Patching Step 1: Logits
mostly MLPs 0-3, well a0.h5, a0.h3, and a0.hl. We delve again into this group of MLPs (Figure 13, and see that each MLP relies on the MLPs before it and a0.h1; MLP 2 is the exception, it does rely on MLP We know that a0.h5 nothing else before it in the residual stream! We attempt to find what MLP depends on, but it does rely on any of the attention heads prior to this indicates that it depends primarily on the token embeddings, not shown in these iterative path patching diagrams.
only rely the token embeddings; there is
We have developed hypothesis regarding full circuit, pictured in Figure 14 We evaluate it as before, keeping in mind that the keys and values of the attention heads are patched with the same input components found earlier; the queries, not the focus of this section, receive all good inputs. The circuit achieves a probability difference of 71.5% (98.3% of what we achieved earlier), and cutoff sharpness of 10.5% (again sharper than pre-patching). The qualitative results are in Figure [4
C Circuit Finding, Step by Step
In this section, we explain the circuit finding procedure step by step, with additional diagrams to aid comprehension. We start, as indicated previously, by patching direct connections to the logits (Figure This reveals connections to the logits from MLPs 8- 1, as well as a9.h1. We continue with the next furthest downstream MIP, MLP 11, and see which nodes influence the eircuit via Note that the only path through which node C' can influence the circuit via MI.P 11 is (C, MLP 11, logits), red (Figure T6] right). The results (Figure T6 left) indicate that the other ML.s influence the circuit most through MLP 11.
18

---

m11
0.05
Logits
prob dil variation
Layer
MLP 10
MLP
MLP 8
MLP 9
10 11
mlp
a9.h1
Head
Figure 16: Path Patching Step 2: MLP
Logits
prob dif aariatioo
MLP 10
MLP
-0.1
MLP 8
MLP
10
10 11 mlp
a9.h1
Head
Figure 17: Path Patching Step 3: MLP 10
mg
0.2
0.1
Logits
probo off veriation
Layer
MLP 10
MLP 11
MLP 8
MLP
a2
a9.h1
Head
Figure 18: Path Patching Step 4: MLP
19

---

01
Logits
dif veristion
MLP
MLP 8
MLP 9
-0.05
10
a9.h1
-011
a8.h11, a8.h8, a7.h10, a6.h9, a5.h5, a5.h1
Head
Figure 19: Path Patching Step 5: MLP
We continue onward MLP 10, and which nodes most influential the circuit via it. We consider all the ways in which nodes could contribute via MLP 10 to the circuit, shown red on the right of Figure 17 The results (Figure [7) again that MLP 10 relies mostly MLPs 8 and 9, We proceed similarly with MLP 9, considering all the ways in which it influences the circuit; the results in Figure 18 indicate that relies mostly on MLP 8 and a9.h1.
At this juncture, it might be appropriate to ask which nodes in the graph most influence the circuit skip this node because,
via a9.hJ 19.h1. However, (Section[.3) the attention heads are acting together separately from the MLPs, performing different roles. Moreover, when analyzing attention heads, must consider what influences their queries, keys. and values separately, a complicated task best avoided in the MLP-centric analysis. In Appendix explore in greater detail the nodes that contribute to such attention heads.
explain in the circuit semantics section
Instead, we complete our circuit- seecuittfindinggee section by finding nodes that contribute to the circuit via MLP 8. This reveals (Figure 19 the heads that identify YY, completing our initial circuit investigations.
20

---

MLP 10 Neuron 2659
MLP 10 Neuron 946
MLP Neuron 1616
MLP 10 Neuron
20
40
60
80
Logit Lens Magnitde
MLP 10 Neuron 606
MLP 10 Neuron 2848
MLP Neuron 2305
MLP 10 Neuron
20
-2
40
60
80
-4
50
50
50
Predicted Year
Predicted Year
Predicted Year
Predicted Year
Figure 20: Neuron contributions for each MLP neuron the top 4-11. Neurons ordered by importance, left-to-right, top-to-bottom. Blue indicates that the neuron upweights certain predicted year, given a starting year YY, while red indicates downweighting.
Top top-100 MLP 10 Neurons Logit Lens
Top 200 MLP 10 Neurons Logt Lens
21
37
47
02
24
52
87
0
50
Predktes Year
Prediced Year
Figure 21: Neuron contributions of the top- 100 (left) and 200 (right) neurons in MLP 10. Blue indicates that the neuron upweights certain predicted year, given starting year YY, while red indicates downweighting.
D MLP 10 Neuron Contributions
In this section, display the contributions of the top 10 most important neurons of MLP 10, found in Figure 20 Many neurons' contributions are relatively constant across YY; e.g. the 4th most important neuron always upweights later years. Others differ across YY but not predicted year; the 10th most important neuron downweights all years for the last 10 years so, where correct answers are very few generally. The 3rd most important neuron varies in both dimensions, having contribution for years YY from around 10 to 50, but a distinct pattern for all other YY. Only the first few neurons are very intense in color, as we have fixed the range the color scale: these neurons are the most important because they cause the greatest changes when they patched.
21

---

Combining these contributions rapidly produces patterns resembling those of the MLP as whole. We see this weakly by viewing the top- neurons' contributions, but more strongly in the top-100 or 200 (of 3072) neurons (Figure (T). In these logit lens diagrams, there is a consistent increase in logit lens magnitude between YY and Y for given start year
22

---

MLP 10 Neuron 2326
MLP 10 Neuron 1138
MLP Neuron 2287
17
37
42
57
Logit
62
77
82
97
0
50
50
Predicted
Predicted Year
Predicted
Figure 22: Direct effects of top-3 MLP 10 Neurons
Summed Direct Fllicts
Patched Dewd Eflects
Lagt
Prodiced
Prediced
Figure 23: Summed and patched direct effects of top- MLP 10 Neurons
E
Logit Lens vs. Direct Effects via Path Patching
We the logit lens throughout this paper for consistency, both internally and with prior work. How- ever, the logit lens has flaws: in reality, the residual stream is normalized using layer normalization prior being transformed into the logits. This nonlinearity could in theory cause the scaling of logit lens to be misleading, perhaps in such a way that affects our conclusions. In this section, we will show that the logit lens results can be recreated with another technique, direct effects via path patching, that avoids this flaw. The two techniques fortunately yield the same insights.
Using the logit lens, would normally multiply our component of interest's output by the unem- bedding matrix. To measure direct effects, we take the unpatched, original model's year logits, over each YY, baseline. We then patch the direct path between our component and the logits with the 01-dataset, and again record the logits. The difference between the unpatched logits, and the logits when patch (ablate) component of interest, reveals the direct effect that said component had. This approach eliminates one major concern of the logit lens: it yields the difference of two sets of logits, which were both produced with layer normalization, and which has interpretable units.
We can perform our neuron- logit lens experiments by instead patching the direct paths to the top-3 neurons of MLP 10 (Figure 22). The results essentially identical to the logit lens results, though they differ in magnitude. We can also sum these direct effects, as we summed the logit lens outputs; again, results differ from those of the logit lens only in magnitude (Figure 23 Finally, can also patch all top-10 10 neurons as a group, and view their direct effects; these results are identical to those of the summed direct effects (Figure 23). This suggests that summed logit lens approach genuinely reflected the direct effects that these neurons have on the logits.
We conclude that the results given by the logit lens and those given by direct effects are largely similar, so concerns about the logit lens are not dire. However, we note that our logit lens results (including those for entire MLPs and attention heads) are reproducible using direct effects.
23

---

Phedited
Prediied
Prediced
Figure 24: Probability heatmaps for (left to right) "1799, 1753, 1733, 1701, 16YY, 16", " "1695, 1697, 1699, 1701, 1703, 17", and "17YY is smaller than 17".
Prosobally
Prozobe
Figure 25: Probability heatmaps for (left to right) "The <noun> started in the year 17YY ended in the year 17", "The <noun> happened in 17YY. Some years later, it is now the year 17", and "1599, 1607, 1633, 1679, 17YY, 17".
F Year-Span Circuit Generalization
In this section we provide evidence for the generalization of the year-span circuit some (but not all) tasks. For tasks where the model failed entirely, we provide the probability heatmaps (to show its failure). For tasks where the model succeeded, or incorrectly generalized the greater-thai circuit, we additionally provide path patching results and logit lens heatmaps to show how circuit structure and semantics are preserved.
Tasks Failed Figure 24 displays probability heatmaps for "1799, 1753, 1733, 1701, 16YY, 16", "1695, 1697, 1699, 1701, 773,, 17", and "17YY is smaller than 17". In the first case, GPT-2 predicts roughly uniform distribution around YY. In the second case, the right answer varies: though the penultimate number in the provided sequence is YY="03", and the sequence increases by 2 step, depending YY, must vary the increase per step, in order to avoid the single-token number " "1700". In any case, the model fails to predict the correct answer, often predicting YY+1, although the step is always > In the final case, the model always outputs YY.
Tasks Completed Correctly Figure 25 displays probability heatmaps for "The <noun> started in the year 17YY and ended in the year 17 "The price of that <luxury good> ranges from $ 17YY to 17", and "1599, 1607, 1633, 1679, 17YY, 17". All tasks are completed successfully, just like year-span prediction, though note that the second task completed less well, as is visible in its heatmap. Its probability difference is only 75%, as opposed to 90%.
Given this, proceed using iterative path patching direct logits connections before; Figure 26 shows the results. All of these plots look almost identical to our original plots, so we evaluate the circuit each of tasks using the methodology from Section 3.2 This works for the first two tasks, with performance recoveries > 90%, but not the last.
For the last task, we observe that MIP 8 relies also on MIP 7, which in turn relies two extra attention heads not observed in our original circuit: a7.h11 and a6.h1 (Figure 27p. Accounting for this in our circuit leads us back to > 90% loss recovery.
24

---

logis
Logis
Lpn
Figure 26: Iterative path patching plots (C, logits) for (left to right) "The <noun> started in the year 17YY and ended in the year 17", "The price of that <luxury good> ranges from $ 17YY 17". and "1599, 1607, 1633, 1679, 17YY, 17".
Iterative Path Patching: MLP
Iteralive Path Patching: MLP
0.77
pob iiturattion
10
-0.44
10
11
Head
Head
Figure 27: Iterative path patching plots for "1599, 1607, 1633, 1679, 17YY, 17", searching for components that influence the circuit via MIPs 8 (left) and 7 (right).
Tasks Completed Incorrectly Finally, we address the tasks "The <noun> ended in the year 17YY and started in the year 17" and "The <noun> lasted from the year 7YY BC the year 7", which do use our circuit, but should not do so.
Probablity Heatmap
Probability Heatmap
0.15
0.06
82
57
10
87
20
60
89
20
40
Predicted Year
Predicted Year
Figure 28: Probability heatmaps for "The <noun> ended in the year 17YY and started in the year 17" (left) and "The <noun> lasted from the year 7YY BC to the year 7" (right).
25

---

lterative Path Patching: Logils
lterative Path Patching: Logits
01
Layer
prob
10
-02
Head
Head
Figure 29: Iterative path patching plots (C, logits) for "The <noun> ended in the year 17YY and started in the year 17" (left) and "The <noun> lasted from the year 7YY BC to the year 7" (right).
Figure 28 displays probability heatmaps for "The <noun> ended the year 17YY and started the year 17 and "The <noun> lasted from the year 7YY BC the year 7". Both tasks completed successfully, though note that the latter task completed less well.
As with the other tasks using this circuit, the iterative path patching plots (Figure 29) look similar to those of year-span prediction. Note that since the goal for these tasks to produce "less-than", patching and impeding circuit components improves task performance (indicated in blue rather than red), since the circuit performs "greater-than" We evaluate the circuit on each tasks using the methodology from Section 3.2 This works for both tasks, with performance recoveries 90%.
G Noun Pool for Templated Sentences
use the following nouns in our main template: abduction, accord, affair, agreement, appraisal, assaults, assessment, attack, attempts, campaign, captivity, case, challenge, chaos, clash, collabora- tion, coma, competition, confrontation, consequence, conspiracy, construction, consultation, contact, contract, convention, cooperation, custody, deal, decline, decrease, demonstrations, development, dis- agreement, disorder, dispute, domination, dynasty, effect, effort, employment, endeavor, engagement, epidemic, evaluation, exchange, existence, expansion, expedition, experiments, fall, fame, flights, friendship, growth, hardship, hostility, illness, impact, imprisonment, improvement, incarceration, increase, insurgency, invasion, investigation, journey, kingdom, marriage, modernization, negotiation, notoriety, obstruction, operation, order, outbreak, outcome, overhaul, patrols, pilgrimage, plague, plan, practice, process, program, progress, project, pursuit, quest, raids, reforms, reign, relationship, retaliation, riot, rise, rivalry, romance, rule, sanctions, shift, siege, slump, stature, stint, strikes, study, test, testing, tests, therapy. tour, tradition, treaty, trial, trip, unemployment, voyage, warfare, work.
For the <luxury good> noun considered in the generalization section, we use smaller pool of nouns: gem, necklace, watch, ring, suitcase, scarf, suit, shirt, sweater, dress, fridge, TV,, bed, bike, lamp, table, chair, painting, sculpture, plant.
H Applying the Circuits Approach to Other Problems
In order to find circuit for given model ability, must define task, dataset, and metric; one can then apply the path patching approaches we use. However, task, dataset, and metric design are important; not just any task, dataset, metric will be compatible. In this section, discuss the qualities that task, dataset, and metric should have, in order to be used with our approach.
Task
The path-patching approach is compatible with variety of tasks. The chosen task should:
Have clearly delimited set of correct and incorrect answers for each example.
26

---

Require only one forward pass of the model (as opposed to e.g. generation tasks which require multiple passes).
Be solvable by your model: if your model cannot solve the task, there may be no circuit. At minimum, the model should exhibit consistent behavior (even it's not exactly correct)
The granularity of insights will depend on the granularity of the task chosen. Complex tasks like natural language inference could require different cubb))circuits depending on the specific question; it might be hard find one precise circuit responsible for the task. Smaller, simpler tasks will likely yield easier to interpret results.
For the purpose of this example, we consider the task of fact retrieval, much akin to the task of Meng et al. [28]. Each input will have an (ideally single-token) correct answer, which can be predicted with one forward pass. Moreover, it seems possible that facts are mostly stored and retrieved using the same circuit.
Dataset Path-patching requires datasets: normal and corrupted dataset. The normal dataset is just a collection of examples/inputs for the task; its examples should:
Clearly indicate the task hand. LMs perform language modeling, and do not natively perform other tasks; they may leak probability to answers that are not correct or incorrect, but simply task-irrelevant. The inputs to the LM should push as much probability as possible onto the task's 's output space.
Allow evaluation based on only the distribution over possible next tokens (generated via one forward pass)
Be representative of your task. Your choice of datasets effectively define the scope of the phenomenon you study, so essential that the scopes of the dataset and the intended task match!
Each example from the normal dataset should have a corresponding corrupted example / input. The corrupted imput should:
form a minimal pair with the normal input: they should differ minimally from each other (being the same length, and differing by only one two tokens)
elicit different model response, with distinct correct answer, compared to the normal input
belong the same sort of task. We locate the circuit by activating the same circuit with two different inputs.
For fact retrieval, a normal input could be "Paris is the capital of"; the corrupted counterpart could be "Rome is the capital of". Both of these examples are reasonable input for fact retrieval, but the two will elicit very different responses. Note that an input like "Paris in" would be less appropriate, because it doesn't 't clearly indicate that the task is fact retrieval, or what fact should be retrieved.
Metric
The metric is funetion that takes in model logits and labels. should:
Output real number measuring model behavion/peerformane on the task.
Detect small changes in whether the model is behaving according to the normal or corrupted input, or somewhere in between. continuous loss is thus preferable to metries like 0-1 loss/accuracy.
One family of metrics used in previous work the probability assigned to the correct answer(s), minus the probability of the incorrect answer(s) induced by the corrupted input. In the greater-than case, this is p(y >YY) ply= <YY). For fact retrieval, we would compute p(France) p(taaly.. This family of metries the model is implicitly sensitive the model generating off-task output as this generally takes away from the probability of correct answers. is explicitly sensitive to the probability of the corrupted input's answer.
Other metrics are possible. For example, if it is difficult to quantify task performance, but still possible to create minimal pairs, one could simply measure the KL-divergence between the original
27

---

and (partially) patched corrupted distributions. However, this is harder to interpret, and less targeted your actual task of interest.
I Computational Resources
All experiments performed Nvidia A100 GPU. The path patching experiments and circuit semantics experiments take no longer than an hour to complete. The generalization experiments take similar amount of time, being very similar to the original experiments. The neuron-level experiments (in particular finding top neurons) can take multiple hours to run. Overall, the final experiments can be run in less than 24 hours.
28


Reviews:
[{"review_id": "eVt2ZZKa6L", "reviewer": "Reviewer_Nf2K", "summary": "The paper studies and tries to explain how GPT-2 (small) could be computing the mathematical operation of \"greater than\". A \"circuit\" (or a subgraph of GPT-2 model's computation graph) is identified by iteratively \"patching\" individual components to find which components are most responsible for making the correct prediction for this task. The main claim of the paper is that they identify this circuit and show that this circuit is used by GPT-2 small in other contexts that need the greater than operation.", "questions": "Minor:\n- In Fig 5, Did you intend to flip the instance coming from the actual vs 01-dataset?\n- Fig 6 is not legible. Please increase the font size\n- I don't think I understood what \"input residual stream\" meant. Please clarify.", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The identification of circuits looks sound and an interesting approach. I enjoyed reading that part of the paper -- thank you.\n- The problem being tackled is very interesting, well motivated and some of the experimental results are really insightful\n", "weaknesses": "Main Concerns:\n- The task formulation seems severely constrained. For instance, GPT-2 is prompted with the year prefix \"XX\" already given. Effectively, the approach is evaluating whether GPT-2 can perform 2-digit greater-than operation.\n- Even for the restricted case of year-span prediction task, I think it is important to compute correctness of the entire year string, without providing the XX prefix. Would the results hold in that case? I wonder how much probability mass GPT-2 will assign to the correct XX year token. This might significantly alter the \"Prob Difference\" metric and the results and conclusions. \n- The Prob difference metric might also be hiding inaccuracy of GPT-2 in generating the correct year. How often is it the case than GPT-2 assigns high prob to one of the correct answers in its top K tokens?. \n- I also think it is important to establish a \"prior\". What is the default behavior of the model when it is shown even simpler prompts that do not require the generation of a number greater than the given number. i.e. is GPT-2 predisposed to generating monotonically increasing numbers, even when the context doesn't need it to? \n- In a similar vein, a control task that prompts the model to generate a smaller number is needed to claim that GPT-2 can indeed perform the great-than operation. \n- The other main concern is the fact that each prompt-template yields a somewhat different circuit. The task posed in experiments in Section 5 are the same as the initial setup: predict a 4 digit year given the 2-digit prefix of the correct year. Given this, I do not think the results support the conclusion that gpt-2 has a generalized mechanism to perform greater-than across tasks and contexts. \n\nMinor:\n- While \"emergent capabilities\" have been used as motivation for this work, I am not sure if GPT-2 small's abilities can be referred to as \"emergent\". From what I understand, they are capabilities of much larger models. Moreover, next token prediction (how the main task in this paper is formulated) is well within the kinds of capabilities GPT-2 small was trained on. \n\nOverall, IMO, this is a very interesting line of work which can be made much stronger and conclusive with some additional experiments and analysis. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and thoughtful questions!\n> The task formulation is constrained.\n\nTrue—for why, see Problem Scope in the general response.\n> Compute the correctness of the entire year string, w/o the XX prefix: how much probability mass does GPT-2 assign to the correct XX year token?\n\nWhen predicting XX, GPT-2 almost always predicts a valid continuation (numeric token >= XX). On average, considering the top-100 tokens, which comprise 0.95 of GPT-2’s output probability mass, valid continuations receive 0.89 (94%) of the probability mass. The year XX is almost always the top token (mean rank 1.125, mean probability: 0.41); most top tokens are other centuries > XX. So GPT-2 generally wants to generate XX or some other valid continuation.\n> Prob diff might hide inaccuracy of GPT-2 in generating the correct year. How often does GPT-2 assign high prob to a correct answer in its top-k tokens?\n\nWe find this unlikely, given our high baseline probability difference of 0.81. This indicates that at least 0.81 of all of GPT-2’s probability mass—not just 81% of all probability assigned to numeric tokens—is assigned to a correct token.\n\nHowever, we also tested this and found that 100% of top-1 tokens, and 98.6% of top-5 tokens are valid (>YY). We thus feel confident that GPT-2 predicts a correct answer with high probability.\n> Is GPT-2 predisposed to generating monotonically increasing numbers, even when it doesn't need to?\n\nYes. To test this, we gave GPT-2 sequences of 4-digit numbers like “XXY1, XXY2,..., XXYN, XX”. We chose XX, Y1,…,YN randomly for each sequence, respecting tokenization. Sequences were generally not monotone, so any 2-digit continuation YY could have been valid. We found that GPT-2’s behavior depends on the sequence length N. At low N, GPT-2 produces mostly numbers > YN; the proportion of continuations < YN increases smoothly until ~50% by N=20. So even in the context of random sequences, GPT-2 often generates increasing numbers. We found that our greater-than circuit also underlies this behavior.\n\nWhat does this mean for our study? Greater-than behavior emerges even when no greater-than is required; however, we already knew this, having observed it in the less-than case. \n\nIn our study, we acknowledge that GPT-2 performs greater-than in incorrect scenarios, and do not claim to find a mechanism unique to greater-than scenarios. Rather, we want to connect GPT-2’s greater-than behavior to its underlying mechanisms, be they unique or not to the correct contexts. Our claim is that we have found a circuit that computes greater-than regardless of whether GPT-2 ought to perform the greater-than operation. This claim stands in the face of this new evidence.\n> A control task that prompts GPT-2 to generate a smaller number\n\nWe test less-than in Sec. 5 of our paper. Given prompts like “17YY is less than 17”, GPT-2 predicts YY. Given “The war ended in 17YY and started in 17”, GPT-2 uses the greater-than circuit to predict a year >YY! So, GPT-2 can perform greater-than but not less-than. \nThis suggests that GPT-2’s greater-than ability is not supported by general mathematical understanding; it can’t e.g. use the greater-than circuit to produce less-than by upweighting the opposite tokens. GPT-2’s greater-than circuit is thus in an interesting position: it’s not fully general, but can still be applied in some new contexts.\n> Given that each prompt-template yields a different circuit, and the generalization tasks are the same as the initial setup—predict a 4 digit year given the 2-digit prefix of the correct year—the results don’t support the conclusion that GPT-2 has a generalized mechanism to perform greater-than across tasks and contexts.\n\nWe disagree with some aspects of this critique. Not all tasks are about years: we shift the context from years to prices, and monotonically increasing numbers. The circuits are mostly identical except in the last case, where additional components seem to be relevant. However, the number of nodes/edges that differ between the two is small; even our original circuit isn’t a bad fit. We could have quantified this circuit overlap better, and will do so in our revision.\n\nHowever, we agree that we suggested generalization a bit too strongly. The evidence points more towards a mechanism that can perform greater-than (narrowly defined) in different contexts, but is not necessarily integrated into a broader suite of math capabilities. GPT-2’s greater-than mechanism thus lies between memorization and generalization.\n> I am not sure if GPT-2 small's abilities can be referred to as \"emergent\". From what I understand, they are capabilities of much larger models.\n\nWe share this hesitation. Though Wei et al. list math as an emergent ability in LLMs, GPT-2 is smaller than most LMs claimed to have emergent abilities. We’re happy to change that phrasing—we believe that math abilities in LMs are an interesting phenomenon, emergent or not.\n> In Fig 5, did you intend to flip the instance coming from the actual vs 01-dataset?\n\nYes! To find the circuit, we corrupt parts of the computational graph using the 01-dataset; those edges that hurt performance when corrupted are part of our circuit. However, during evaluation (Fig. 5), we instead give normal input to the circuit nodes, and 01-input to the other nodes. \n\nThe idea is that if the circuit (Fig. 5 center, blue / purple) gets normal input, and the rest of the model (Fig. 5 center, red / purple) gets 01-input, the model should perform the task correctly, as if it were receiving only normal input. This is because the circuit controls model behavior on the task. When we give the circuit 01-input, and the rest of the model normal input, model performance is correspondingly very poor.\n> Meaning of \"input residual stream\"\n\nThe input residual stream to e.g. MLP 10 is the 768-dim input vector that serves as input to MLP 10; this is opposed to the token-level input.\n\nFeel free to send follow-up questions / suggestions!"}, {"review_id": "EaC9igyauT", "reviewer": "Reviewer_eiM2", "summary": "This paper presents an analysis on how the greater-than operator is implemented in the weights of GPT-2 small. They do so by tasking the model with completing sentences of the form \"[something, e.g., a war or time period] lasted from $y_1$ to $y_2$\" where $y_1, y_2$ are years. The idea is that GPT often assigns values $>y_1$ much higher probability than values $\\leq y_1$, when trying to predict $y_2$. This is likely learned due to statistical co-occurrence, but the authors ask _what mechanism_ encodes this behavior in the weights.", "questions": "* In Section 3.2: what does the PD look like when you give most of the network the 01-dataset but patch the regular dataset into a circuit _other than_ the one you discovered? Based on Figure 4 I'm guessing it'd be lower, but I don't have a great intuition for how much lower.\n* In Section 3.3:\n  * Figures 6, 7 restrict the plotted tokens to the numbers. In absolute terms, were those numerical tokens promoted the most by the logit lens? Did you find any surprises, i.e., other tokens being ranked higher than numbers in inner product?\n  * MLP 8 is interesting! What's going on with it? What does it mean for an MLP to contribute indirectly? What could it be doing? Relatedly, you might have answered this already, but do you have a better intuition for how MLPs 8-11 interact? Whether and why you can't just remove one of them?\n* What kinds of analyses did you have to come up with for this paper? I'm not very familiar with circuits; are most of the techniques here standard or did you have to come up with new tricks while, e.g., explaining the semantics of the circuit? How do these tricks generalize to other problems in interpretability?", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper is well-written, easy to follow, and well-motivated. The explanations are clear, with limitations stated clearly.\n* I enjoyed the exposition of how the circuit was discovered and the interpretation of its semantics.\n* It's generally hard to find simple, easy-to-evaluate tasks that GPT-2 small is proficient at, so kudos here.", "weaknesses": "Most of my \"concerns\" are really more aptly formulated as questions. Interpretability is still very nascent, and to me this kind of contribution is meaningful, in spite of (or actually because of) the many unanswered questions it raises.\n* I will say that I'm not 100% convinced by the argument at the end of Section 5, that this circuit is not using memorization and actually generalizes. It's hard to know what that means and to show it, especially on a small toy dataset with only four-digit numbers (really, two-digit completions). I think you'll need to demonstrate the same phenomenon on much larger datasets, as it's not super unlikely that the model could memorize the relationship between numbers from 0-99. You even show via ablation that the representational structure discovered by PCA is not a complete explanation. Maybe reframe these claims, or perhaps provide more convincing evidence?\n* I think you can _really_ nail the \"so what\" of the paper by showing how knowledge of this circuit can be used to improve the reliability of the greater-than operation. Do you think you could design a circuit by hand that outperforms the existing one on this task? Or modify the existing one? What would happen if you integrated it into the model? Can you move the circuit to another location? Why is the circuit the way it is?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your attentive review! We answer your questions below; let us know how else we can improve the paper!\n> I will say that I'm not 100% convinced by the argument at the end of Section 5, that this circuit is not using memorization and actually generalizes. [...]\n\nThanks for this comment—we’ve been thinking about this post-submission as well. We agree that our original framing suggested generalization too strongly. There is a real possibility that our circuit has memorized greater-than; its limited ability to activate in new contexts could come from the model’s ability to recognize relevant contexts / patterns, as NNs are suggested to do. We will revise the paper to reframe our claims, emphasizing this as an interpretation. We do want to add, though, that we find this possibility very interesting as well, and believe that mechanistic evidence of this sort of memorization is also valuable.\n\n> Using our circuit to improve reliability of greater-than/outperform the existing circuit/etc.\n\nWe agree that this sort of contribution would be valuable, and regret that it falls out of the scope of our work. This study mostly answers “When GPT-2 computes greater-than, how does it do so?”. However, the best way to improve GPT-2’s greater-than reliability would be to cause the circuit to activate in situations where it should, but doesn’t currently. To achieve this, it would’ve been better to ask “How does GPT-2 decide if it should compute greater-than?”. If we knew precisely when/why our circuit activated, we could cause it to activate in new scenarios. This would be exciting follow-up work.\n\nThere is other work that uses circuits for more practical purposes; for example, other studies have cut relevant edges from model graphs to curtail bad behavior [1]. We hope to investigate such techniques in the future.\n\n[1]: Li et al. 2023. Circuit Breaking: Removing Model Behaviors with Targeted Ablation. ICML Deployable Generative AI Workshop\n> In Section 3.2: what does the PD look like when you give most of the network the 01-dataset but patch the regular dataset into a circuit other than the one you discovered? \n\nInteresting question! Choosing the “other” circuit is tricky, but we tested this, selecting circuits around the same size/location as the original. We found that performance is related to how well input->attention->MLP (especially 9/10)->logits paths are preserved. If the path is interrupted (i.e. no attention head or no MLP overlap with the original circuit), PD is very low (-37%). If at least one path is preserved, PD improves with each additional component in common with the original circuit; MLPs have the biggest impact.\n> Figures 6, 7 restrict the plotted tokens to the numbers. In absolute terms, were those numerical tokens promoted the most by the logit lens? Did you find any surprises, i.e., other tokens being ranked higher than numbers in inner product?\n\nYes, numerical tokens were the top-ranked tokens for both MLPs and attention heads. For the attention heads, only the top ~3 tokens were numbers, while for the MLPs, the top-k tokens were numbers at higher k. We attribute this to the fact that the MLPs upweight a set of numbers, while the attention heads generally upweight only the one (their top-1 token, generally). \n\nThat the numerical tokens are the top tokens is a little surprising—in theory, other tokens could be boosted, and later pushed down by other modules; modules might also be making high-magnitude changes to other words’ logits at the same time as they upweight the right answer. However, other work has also observed that LMs create predictions by promoting the right answer (not downweighting others). This seems like a useful observation to add to the paper.\n> MLP 8 is interesting! What's going on with it? What does it mean for an MLP to contribute indirectly? What could it be doing? Relatedly, you might have answered this already, but do you have a better intuition for how MLPs 8-11 interact? Whether and why you can't just remove one of them?\n\nWe also find MLP 8 interesting! Its output, like that of the attention heads, is important to making MLPs 9-11 upweight the correct values; unlike them, though, it doesn’t clearly upweight YY! Its contributions may relate to the logits of non-years—or perhaps logit space isn’t the correct one to view its contributions in. That is, we know that MLP 8 helps the other MLPs, but we’re still unsure of how.\nMLPs 8-11 are interconnected in our circuit. Removing one is possible, but removing e.g. an MLP with strong direct effects like MLP 10 or 11 would drastically reduce the portion of the circuit upweighting the right tokens. Not all connections are equally important though: we consider all MLPs interconnected for simplicity, but for some MLP pairs, the edges between them can be cut without a performance drop.\n>What kinds of analyses did you have to come up with for this paper? [...A]re most of the techniques here standard or did you have to come up with new tricks while, e.g., explaining the semantics of the circuit? How do these tricks generalize to other problems in interpretability?\n\nWe address this in the **Methodological Contributions** section of the global response. Many of our semantics techniques are from other work. However, the circuits literature is rather young, with little in the way of standard techniques; with this paper, we aimed to develop a toolkit for studying circuit semantics. \n\nHere are some techniques we used, and how we feel they generalize to new categories of problems: \n- The logit lens characterizes individual nodes; combining it with complex path-patching can tell you what nodes compute, and what nodes use the information computed. These techniques are generally applicable. \n- Neuron-level interventions seem useful for MLP semantics. \n- PCA on the output of individual nodes was visually appealing and interesting, but difficult to transform into concrete, causal insights.\n"}, {"review_id": "THhbl5khfv", "reviewer": "Reviewer_GMC8", "summary": "This paper explores how GPT-2 performs the ``greater-than'' operation by analyzing its circuit. The authors construct a template of the operation and define two scores to evaluate the performance of GPT-2. They first evaluate that the found circuit is indeed important for performing the greater-than operation, and then find that top MLP layers in GPT-2 compute the operation.\n\n---\nRebuttal response: \n\nThanks for the detailed explanations. Some of my concerns about the details are addressed. Thus, I would like to raise the soundness score. Regarding the contribution, I agree that the method part under the mechanistic interpretability scenario is sufficient. However, the findings are not attractive to me. Thus, I will raise the score to 2. Regardless of whether the paper is accepted or not, I would like to suggest that the presentation could be improved.", "questions": "Why the curve in the right figure of Figure 2 are continuous? Some tokens (e.g., 00) are not considered as described in the paper.\n\n(Line 81): Why mention ``GPT-2's training data''? Do you finetune GPT-2 or just use it in the zero-shot setting?\n\n(Line 87): This should be average instead of sum, right? Otherwise, it would not range from -1 to 1. \n\n(Line 305): Typo: there are double ``from''.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "This paper explores a very interesting question for LM understanding: how LMs perform tasks inside its architecture. The analysis method, i.e., looking at the circuit of LM, is suitable for this kind of interpretation. ", "weaknesses": "1. The key contribution is not clear. This paper uses the existing analysis method for interpretation. But, to research the question (how LM performs math calculation), only a simple template and a toy task (greater-than operation) are discussed for GPT-2. There is a limited contribution to the analysis.\nBesides, for findings, such as GPT-2 performs the operation on top MLP layers, many existing works (e.g., probing works) have also found that, and existing findings are even more comprehensive.\nThus, the contribution of this work is not clear. I would suggest that the authors should focus on one side (e.g., analysis method, interpretation task/data/template design) and dig deeper.\n\n\n2. The presentation needs to be improved. The essential information should be clearly introduced and technical details could be moved to the appendix. \nFor example, in section 2, it says that only single-token numbers are considered in this work (lines 65-72). We know that it is because of GPT-2's BPE tokenizer. There is no need to give specific examples for that. But for the dataset introduction, what's the size of it, how did you construct it (and why in that way), these parts are not well introduced.\nBesides, it would be better to use some tables in section 3 to make it more readable.\n\n3. Generalization. As mentioned above, generalizing the analysis method to large LMs with more complex math problems is not intuitive. With different tokenizers (e.g., LLaMA tokenizer), the analysis template in this paper would fail. Besides, recent instruction-tuned LLMs may have very different conclusions compared to GPT-2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critique! We hope to resolve your concerns by clarifying both the intent of our work and our contributions.\n> The key contribution is not clear. …There is a limited contribution to the analysis. \n\nWe clarify our key contributions and analyses in the global response’s **Contributions** section.\n> This paper uses the existing analysis method for interpretation.\n\nThis isn’t quite correct. We introduce new analyses within the circuits framework: we patch multi-node paths, and test our circuit in scenarios distinct from the one in which we found it. Even when using existing methods, we don’t rely on a set of established tools for circuits analysis: the circuits literature is young, and no such toolkit exists. One goal of this paper is to set a methodological standard for circuits research.\n> But, to research the question (how LM performs math calculation), only a simple template and a toy task (greater-than operation) are discussed for GPT-2.\n\nThe greater-than task is not intended to explain how LMs perform all math, but rather to examine one small part of LMs’ mathematical abilities in detail. In fact, our results suggest that the greater-than circuit may not be part of a larger set of math capabilities; instead it is a mechanism between memorization and generalization. By studying a small task, we found an interesting mechanism with broader implications for math and generalization in LMs.\n> Besides, for findings, such as GPT-2 performs the operation on top MLP layers, many existing works (e.g., probing works) have also found that.\n\nWe respectfully disagree; our findings are not a repetition of existing work. Our core finding is a precise and novel circuit for a math operation in a pre-trained LM—a contribution beyond superficial, layer-wise analyses of top MLP layers. Our circuit generalization results are also absent from prior work. See the **Contributions** section of the global response for more details.\n\nWe stress that even simple conclusions like “top MLP layers do a lot of work” are not obvious or settled, but rather up for debate. While existing work has posited a role of certain (not top) MLPs in e.g. factual associations [1], other work disagrees, localizing this elsewhere [2].\n\nOur methodology is meaningfully distinct from non-causal methods like probing. Because it is not causal, probing often finds information in model representations that is not actually used by models [3]; our methods avoid this pitfall. Thus, even if probing work has found similar results, reproducing these with more trustworthy causal methods is still valuable.\n\nThat said, we are happy to discuss more related work in the paper; feel free to highlight such work, especially if you feel it overlaps with ours.\n> The presentation needs to be improved. The essential information should be clearly introduced and technical details could be moved to the appendix…the dataset…[is] not well introduced. Besides, it would be better to use some tables in section 3...\n\nSome of the dataset information unintentionally bleeds over from the Task and Dataset section into the following Qualitative Evaluation section. We will fix this, prioritize essential dataset information, and organize some section 3 results in tables. \n>Generalizing the analysis method to large LMs with more complex math problems is not intuitive. With different tokenizers…the analysis template in this paper would fail. Recent instruction-tuned LLMs may have very different conclusions.\n\nWe discuss this in the **Scaling** section of the global response. Circuits scale and have been used at larger scales in the time since submission. While our template may not work for all math problems, the challenge of crafting templates and data is part of all interpretability work, and is not unique to our methods.\n\nRegarding findings, we can’t know for sure if large or instruction-tuned LMs process math as GPT-2 does; however, work like ours is a necessary first step towards understanding math in those large models.\n> Why the curve in the right figure of Figure 2 are continuous? Some tokens (e.g., 00) are not considered as described in the paper.\n\nWe could have used a scatterplot, as we only have probabilities for discrete years, but the curve had more visual appeal. We do not include 00 as a potential start year; however, the figure’s x-axis is the predicted year y, and the y-axis is p(y|prompt). We do measure p(00|prompt); it stays low, as it is never a correct answer, and is an unnatural tokenization of centuries.\n>(Line 81): Why mention ``GPT-2's training data''? Do you finetune GPT-2 or just use it in the zero-shot setting?\n\nNo, we don’t fine-tune GPT-2—we do zero-shot evaluation. We meant to suggest that the durations that GPT-2 predicts for each event are likely related to patterns in its (pre-)training data. However, we can remove this speculation.\n>(Line 87): This should be average instead of sum, right? Otherwise, it would not range from -1 to 1.\n\nNo, the sum is correct. The probability difference (PD) for an example with starting year YY is the sum of probability assigned to years > YY minus the sum of the probability assigned to years <= YY. PD is maximized (PD=1) when all probability is assigned to years > YY, and none to any other tokens, including non-year tokens. It's minimized (PD=-1) when all probability is assigned to years <= YY. We aggregate PD by averaging over examples in our dataset.\n\nWe hope this answers your questions. Other reviewers found our analyses satisfying, and our findings interesting; let us know how we can improve the paper, and make you feel the same way too!\n\n### References\n[1]: Meng et al. 2022. Locating and Editing Factual Associations in GPT. NeurIPS\n\n[2]: Hase et al. 2023. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. ArXiV\n\n[3]: Belinkov. 2022. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics"}, {"review_id": "enR4InFv2U", "reviewer": "Reviewer_BwzU", "summary": "This work investigates in depth the mechanism of GPT-2 small to compute the \"greater than\" function. Specifically, the work isolates a portion of the computational units that are causally related to making plausible predictions for inputs similar to: \"The war lasted from 1731 to 17__\" The isolation of computational units is done via a recent previously proposed method called \"path patching\". The resulting isolated subnetwork is shown to be necessary and approximately sufficient for making plausible predictions for this task. The authors also show that this subnetwork generalizes to other types of templates that require prediction of a greater number.", "questions": "Major:\n\nQ1. Related to weakness 2: Which part of the interpretability techniques are borrowed from previous work, and what is newly proposed here? How would a researcher interested in investigating a different function or a different model do this using the proposed framework?\n\nQ2. For the sufficiency and necessity experiments in Section 3.2: if the identified circuit is sufficient and necessary, then does it matter what input is provided to nodes outside of the circuit? How would providing random input to the nodes outside the circuit change the results, while providing the original dataset to the nodes inside the circuit? \n\nQ3. Which of the results do you expect to hold in larger GPT-2 variants, and how can the proposed analysis framework be used to investigate the emergence of properties at different scales of models / training data?\n\n\nMinor:\n\n- Fig 3 is difficult to understand on its own. The change in the direct connection between MLP 10 and the logits is difficult to spot between A and B. I recommend leaving the direct connection where it is in A but coloring it in a different color, and writing a more descriptive caption. Also the caption in C says that MLP11 receives input 01 but there are no direct or indirect arrows between the 01 input and MLP11.\n\n- Also it’s not quite clear what it means to “patch” a path. How does one ensure that the perturbed input that is provided to the patched MLP 10 does not influence all downstream components of MLP 10?\n\n- The use of footnotes is standard for NLP venues but not so much for ML venues. I personally find footnotes distracting.\n\n- L178: typo: “and and”\n", "limitations": "See Weaknesses", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- In-depth mechanistic analysis of a language model that would be of interest to some of the NeurIPS audience\n- Strong scientific validity. I appreciate that the authors thought through possible confounders and did extra experiments to validate their findings", "weaknesses": "1. In the current format, the contributions of the current work are difficult to disentangle from previous work. The impact can be strengthened by clearly stating what the current work contributes above previous work, and by a detailed explanation of how a researcher can build on the analysis framework to investigate other emergent skills/properties of language models.\n\n2. The work investigates only one model - GPT-2 small. Investigating additional models can help understand how these mechanisms emerge, and how they are influenced by scale and amount of training data.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review! We’ve answered your questions below, omitting the weaknesses, as they correspond roughly to Q1 and Q3. Regarding our contributions: please see the **Contributions** section of the general response, and Q1. We will revise our paper to clarify our contributions, and add details regarding how to apply our methods to other models.\n> Q1. Related to weakness 2: Which part of the interpretability techniques are borrowed from previous work, and what is newly proposed here? How would a researcher interested in investigating a different function or a different model do this using the proposed framework?\n\nThe **Methodological Contributions** section of the global response tackles this contributions aspect of this question.\n\nOur methods are quite general and can be applied to other tasks/models. Generally, one needs to define a task that the model can perform, with a corresponding dataset and metric. One also defines a corrupted (patch) dataset that induces different model behavior, measurable by the metric. Then, one can corrupt connections using the patch dataset, and determine which connections are important. \nWe suggest patching specific connections, rather than patching the output from one node to all other nodes, in order to gain a finer-grained understanding of how tasks are performed. Our work also shows the potential of testing the found circuit on new datasets for similar tasks. Note that doing this research on new models can involve some model reimplementation, in order to enable manipulation of the computational graph, or allow intervention on its intermediate values.\n> Q2. For the sufficiency and necessity experiments in Section 3.2: if the identified circuit is sufficient and necessary, then does it matter what input is provided to nodes outside of the circuit? How would providing random input to the nodes outside the circuit change the results, while providing the original dataset to the nodes inside the circuit?\n\nThis is an interesting question! Our circuit contains all nodes that compute greater-than, given that the quantity being computed is greater-than. That is, if the context requires greater-than output, our circuit computes it. Thus, when we ablate our model, we give the ablated portion input tokens that still require greater-than output, while altering the quantity YY that the output should be greater than. \n\nBut if the input to the ablated nodes suggests that some other sort of output is needed, behavior could differ. Our circuit might not function, or another circuit, in the ablated portion of the model, triggered by the different input, could dominate model behavior. And, if the other nodes’ values were set to something totally out of distribution, it’s tough to know precisely what the model’s output would look like. We think that your question is deeply tied to questions like “What contexts cause this circuit to activate, and how do LMs detect such contexts?” which should be studied in future work.\n> Q3. Which of the results do you expect to hold in larger GPT-2 variants, and how can the proposed analysis framework be used to investigate the emergence of properties at different scales of models / training data?\n\nThe **Scaling** section of the global response answers this question. In essence, the mechanistic motifs that we find—the co-operation of attention heads and MLPs, as well as partial circuit generalization—seem likely to generalize to larger models. \n\nAs for our methods, the circuits framework scales well on a technical level, especially when automated. It is quite general: if you can define a task with a normal and a patched dataset, as well as a metric, you can apply circuit-finding techniques. The more challenging question is how to assign semantics to these circuits at scale. Others have tried to automate e.g. using other models to assign semantics to model neurons, but this sort of technique is still immature. However, we think that automatically assigning semantics to model internals is challenging for most interpretability frameworks, not just ours.\n> Fig 3 is difficult to understand on its own[...]\n\nGood suggestion; we’ll use a different color for the removed connection between MLP10 and the logits. And indeed, the caption of part C is wrong: MLP 11 does not receive 01-input. We’ll make both of these changes—thanks for your careful attention to our diagrams.\n> Also it’s not quite clear what it means to “patch” a path. How does one ensure that the perturbed input that is provided to the patched MLP 10 does not influence all downstream components of MLP 10?\n\nIn the framework we use (rust-circuit), we directly manipulate the model’s computational graph. So, to patch the direct MLP10->logits path, we can copy MLP10 and its ancestor nodes, and corrupt the token inputs to that copy. We then replace the original MLP10->logits edge with a corrupted MLP10->logits edge; we don’t touch the edges between the original MLP10 and other nodes. Thus the corrupted MLP10 is an ancestor of only the logits; any other components with MLP10 as an ancestor see the original MLP10.\n\nSimple path-patching can be done using vanilla PyTorch models / hooks, or via a package like TransformerLens—let us know if you’d like details! For a more formal exploration of path patching, see [2] in the global response.\n\nPlease let us know if you have any other questions or ideas regarding how to improve our work!"}, {"review_id": "rjSIxgJhYq", "reviewer": "Reviewer_WeSY", "summary": "The paper attempts to improve our understanding on the source of a certain ability (the use of \"greater than\" in mathematical tasks) in LMs. To this end, the authors outlined a circuit in GPT-2 with interpretable structure and semantics, adding to the evidence that circuits are a useful way of understanding pre-trained LMs.\n\nCompared to existing work, the use of circuits is more fine-grained, and reveals special insights of such ability. The experiments are not only conducted on math tasks, but also extended to adjacent tasks that are also involved with \"greater than\" relations.\n\nAlthough there might be some concerns on whether these findings can be generalized to ever larger models, the reviewer still thinks it is a good paper as it shows a unique way to understand deep neural networks.", "questions": "Have you ever conducted experiments on \"smaller than\"? I'm curious about whether the findings are the mirrored version of \"greater than\", or they are total different from each other. If we can find some \"semantic relationship\" among the circuits, that will make such pipeline easier to be generalized.", "limitations": "As mentioned in the Weakness section.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Novel approach on understanding the ability of LMs.\n2. Extensive experiments and clear presentation.", "weaknesses": "1. The conclusion might be limited to certain model type (auto-regressive LMs such as GPT series).\n2. Relatively narrow choice of tasks (only studying the \"greater than\" relationship).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Here are our responses:\n> The conclusion might be limited to certain model type (auto-regressive LMs such as GPT series).\n\nWe agree that some of our conclusions, such as our specific circuit, are limited. Other findings, such as the way attention heads and MLPs work together to solve problems, or the form of partial generalization we observe, could surface in other models as well. For more discussion, see the **Scaling** section of the global response. More broadly, our methodology—from circuit finding to analysis—can easily be applied to other autoregressive or masked transformer LMs.\n> Relatively narrow choice of tasks (only studying the \"greater than\" relationship).\n\nThis is true. We choose our task in part because GPT-2 small is not very mathematically competent: its performance on other math tasks is generally not high enough to be worth interpreting. However, we find that a small problem scope also allowed us to dig deeper and find a detailed circuit; see the **Problem Scope** section of the global response for more details.\n> Have you ever conducted experiments on \"smaller than\"? I'm curious about whether the findings are the mirrored version of \"greater than\", or they are total different from each other. If we can find some \"semantic relationship\" among the circuits, that will make such pipeline easier to be generalized.\n\nYes, we have conducted experiments on less-than! These experiments are in the generalization section (Sec. 5) of our paper. To summarize, given prompts like “17YY is less than 17”, GPT-2 tended to predict YY. Given prompts like “The war ended in 17YY and started in 17”, GPT-2 used the greater-than circuit to predict a year >YY! This is to say that GPT-2’s less-than behavior is totally different from its greater-than behavior: there is no consistent behavior, and thus no real circuit underlying it. \n\nThis might seem disappointing—as you say, it would be nice to find a semantic relationship between two circuits that reflects some sort of deeper understanding of mathematics or the order of numbers. However, this is still interesting: it hints that models might perform something in between simple memorization and full generalization. GPT-2 uses our circuit in distinct greater-than scenarios, so there is some partial generalization; however, this does not stem from real mathematical abilities. We think that this partial, incomplete generalization, reflects LMs’ abilities: close to humans’ on the surface, but far in terms of deeper understanding.\n\nLet us know if you have any other questions or suggestions to improve our work!"}]

Author rebuttal:
We thank the reviewers for their insightful reviews. We're glad you found our problem and approach interesting (Nf2K,GMC8,BwzU,WeSY), our experiments extensive and scientifically sound (BwzU,WeSY), and our paper clear and well-written (eiM2,WeSY). Still, we want to address some key concerns: our choice of task, our contributions, and how our methods and results scale.

## Problem Scope (Nf2K, GMC8, WeSY)
Many reviews noted that we studied a small task. We agree that this greater-than is simple; however, it lies in the middle of a major gap in the interpretability literature: how do LMs implement math abilities? 

Our simple task allowed us to dig deep into the model, developing a detailed explanation. While our case study cannot explain math in all LMs, it provides intuition and hypotheses useful across tasks and models. For example, we find that GPT-2 performs greater-than using a mechanism that lies between memorization and generalization; this could explain LMs’ inconsistent math performance more broadly. As circuits techniques mature, these insights could help us tackle larger problems in interpretability.

## Contributions (eiM2,GMC8, BwzU)
Many reviewers were unsure of how our contributions differed from those of other work—we will revise our paper to clarify this. Here, we outline our scientific findings and the methodological contributions that enabled them.

### Scientific Findings
- We find a circuit for greater-than in GPT-2. At submission time, no other causal interpretability work had been done on math in pre-trained LMs. Moreover, there is only one prior work published on circuits in pre-trained LMs [1]; it focuses on a circuit that copies one token from the input. In contrast, our task has a wider output space and richer structure: our circuit must interpret tokens as numerical quantities, and upweight a specific set of tokens not present in the input.

Our circuit is a highly detailed account of an LM’s algorithm; prior work does not make such fine-grained causally motivated claims. This is exemplified by our finding that attention heads pass YY information to MLPs, which then compute greater-than. Although these concrete findings are specific to our circuit, the broad motifs—attention heads moving information into MLPs, which act both directly and indirectly—may generalize. By understanding the mechanisms via which LMs implement specific capabilities, we hope to better understand the overarching mechanisms by which LMs work.

- We show that GPT-2’s implementation of greater-than lies between memorization and generalization. We claim this because in our generalization experiments, the circuit does activate in new greater-than scenarios; however, it cannot support related computations like less-than or equal-to, and doesn’t clearly involve general math representations. It thus reflects neither full math competence nor simple memorization. Through this work, we hope to add nuance to the memorization-generalization dichotomy, and take the first step towards a rich characterization of the states in between them.

We thank reviewers Nf2K and eiM2 for their questions about our circuit’s generalization. We agree that our original framing suggested generalization too strongly, and that this framing better represents both our evidence and how LMs work. We will update the paper.

### Methodological Contributions 
Our methods contributions over previous circuits work [1] are as follows:
- Instead of patching individual edges in the model, we patch full subgraphs of our model, allowing us to find a complex circuit with many interconnecting components. This is enabled by our use of rust-circuit [2], a framework that allows for direct manipulation of models’ computational graphs. We did not create rust-circuit, but are the first to use it in this fashion.
- We used separate datasets to conduct our circuit-finding study and to assess the generalization of the hypothesis design on the first dataset, similar to the train/test split used in machine learning. In contrast, preceding LM-circuits work found their circuit and tested it on the same set of prompts. 
Finally, some reviewers asked: aren’t many of our methods standard, established methods? Not so: as circuits research is very young, there is not yet a standardized methodology. In writing this paper, we hoped to establish such a toolkit for future circuits researchers by bringing together diverse techniques like the logit lens, PCA on representations, and neuron-level interventions.

## Scaling (GMC8, BwZU, WeSY)
Some reviewers were concerned that our methods or findings would not scale. Regarding circuits methods, automatic circuit-finding [2] is already in development (contemporaneously with our work); moreover, in the time since submission, circuits methods have already been applied to 70B-parameter LMs [3]. Our complex subgraph ablations scale less well; we foresee them being used to zoom in on particular components, rather than to study entire large models.

Our findings, too, have potential to generalize at scale. While our circuit is specific to GPT-2 small, other findings need not be so restricted. Larger models too may coordinate their attention heads/MLPs as we observed, or rely on circuits that only partially generalize. These are only hypotheses; however, via these hypotheses, we lay a foundation for future work that explores these phenomena at scale.

We hope that this rebuttal has addressed your concerns! Please reach out with any other questions—we’d be happy to chat more.

### References
1: Wang et al. 2023. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. ICLR

2: Goldowsky-Dill et al. 2023. Localizing Model Behavior with Path Patching. ArXiV

3: Conmy et al. 2023. Towards Automated Circuit Discovery for Mechanistic Interpretability. ArXiV

4: Lieberum et al. 2023. Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla. ArXiV

Meta-review: This paper aims to explain how “greater than” operation is implemented in GPT2-small. They use “path patching”, a causal approach to intervene on activations to discover the circuit in charge of conducting this operation. They use different template variants to verify that the identified circuit is indeed the mechanism used in other contexts that need a “greater than” operation.

All reviewers found the problem interesting, and most had consensus about the scientific validity and the depth of analysis in tackling this problem. The key concern was the generalizability of the findings beyond the GPT2-small model and the “greater than” operation. The authors have tempered their generalization claims, clarified the scope and contributions of this work, and added some empirical evidence to the related questions raised by the reviewers (which I would strongly recommend to be added to the appendix). Most reviewers found the author's response adequately addressing their concerns, and adjusted their rating accordingly after a fruitful constructive discussion.

Despite limitations with respect to generalizability of the findings, this work could be a good addition to the recent literature on mechanistic interpretability work that so far has mostly focused on toy tasks and models (like modular addition and group operations) or other setups like factual associations. It is particularly challenging to find reliable mechanisms across tasks and models in our current LMs, and we need to evaluate work in this space with more calibrated expectations. Therefore, I believe a more thorough investigation in a narrower scope could still be valuable. 

In any case, I strongly recommend the authors to take the excellent comments by reviewers into account for future revisions of this work or its extensions.
Decision: Accept (poster)