SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We claim that this is because MLP 8 contributes mainly indirectly, via the other MLPs in our circuit. We confirm this by patching MLP 8's direct contributions to the logits with the 01-dataset; we do so again with its indirect contributions, through the other MLPs. In the former case, model performance drops only  $14\%$ , while in the latter case, it drops by  $39\%$ . So MLP 8 does not contribute much to the logits directly, but it does contribute indirectly. Other MLPs also have mixed effects: MLP 9 has roughly equal direct and indirect contributions (28% vs. 32%), while MLP 10 contributes mostly directly (56% vs. 16%). MLP 11 can only contribute directly.
Our full picture of the circuit so far is this: the attention heads communicate the start year YY in embedding space. MLP 8's mostly influences downstream MLPs. However, MLPs 9, 10, and 11 appear to compute the greater-than operation in tandem, and in steps. We conclude that while the attention heads identify the important year YY, it is the MLPs that effect the greater-than computation.
Figure 8: (Left to right) Logit lens of MLPs 11, 10, 9, and 8; labels as in Figure 7
Figure 9: PCA of MLP 8's input, a7.h10's and a7.h8's output, and the static year embeddings. Each point corresponds to one datapoint's representation, and is labeled with and colored by the its YY.
4 Explaining Greater-Than in the Year-Span Prediction Circuit
Our prior experiments show that MLPs 9-11 directly compute greater-than. But how do they do so? We cannot provide a conclusive answer, but identify avenues by which MLPs might compute this. We first examine their inputs, finding structure that might enable greater-than computation. Then, we examine MLP internals, showing how neuron composition could enable greater-than computation.
4.1 Input Structure
To understand how MLPs compute greater-than, we analyze various model representations using 2D Principal Component Analysis (PCA). For each of the 97 datapoints in our small dataset, each with a unique start year, we analyze the input residual stream to our MLPs, as well as the output of relevant attention heads. As a control, we also analyze representations from irrelevant model components, and the static year embeddings. We take all component representations from the end position.
In Figure 9, PCA reveals that the input residual stream to MLP 8 (and indeed all of our MLPs, though not all are shown) is highly structured: representations are ordered by the start year of the sentence they are from, increasing clockwise. The same is true of the outputs of relevant attention heads (a7.h10), which serve as inputs to the MLPs, but not of outputs of irrelevant heads (a7.h8). This suggests that it is specifically the relevant attention heads that transmit this structured information to relevant MLPs. But while the heads seem to transmit this structured information to the MLPs, they need not have created this structure from scratch. We find, as in Wallace et al. [49], that structure already exists in the static year embeddings, though the years 02-09 are clustered apart from the rest. The heads need only unify these groups and move this information from the YY position to the end.
Structured number representations have been implicated in mathematical capability before: Liu et al. [27] train a toy transformer model on modular addition, and find that its number representations become structured only after it stops overfitting and begins to generalize. This suggests that GPT-2's structured number representations may be relevant to its greater-than ability. However, our experiments struggle to prove this causally. When we ablate the dimensions found through PCA, to test their importance to the greater-than task, we found little change in model performance. Similarly, removing linearly-extractable YY information from attention head output using LEACE [5] yielded only slightly lower probability difference (64.7%) than the baseline (81.7%). This indicates that structured YY information may not fully explain GPT-2's greater-than abilities.
4.2 Neuron-Level Processing
To better understand MLPs, we turn to their internals, zooming in on their neurons. We choose to study MLP 10 closely, as we know it directly contributes to the greater-than operation. We start by asking which of MLP 10's neurons are important—a sort of question already studied using probing [10, 12], causal ablations [2, 24, 48], and other techniques; see Sajjad et al. [43] for an overview.
As before, we use path patching because it provides precise causal insights. We path patch each of MLP 10's 3072 neurons' direct contributions to the logits with the 01-dataset. We again record the change in model performance, as measured by probability difference, compared to the unpatched
Figure 10: Left: The logit lens of the 3 MLP 10 neurons most important to year-span prediction. Right: The logit lens of the top-10 MLP 10 neurons. Blue indicates that the neuron upweights logits at the given input year (y-axis), output year (x-axis) combination, while red indicates downweighting.
model. We find that neuron contributions to the task are sparse: most neurons can be patched (ablated) with near zero effect on our model performance, as observed in prior work [48].
We then analyze those neurons that contribute most to model performance using the logit lens. To do this, we take advantage of the fact that each neuron has a corresponding row in the MLP output weight matrix. As noted by Geva et al. [19], multiplying this row by the unembedding weights yields an (unnormized) distribution over the logits, indicating which outputs the neuron upweights when activated. Taking the outer product of this logit distribution with the neuron's activations yields the logit lens, indicating which output years the neuron upweights for each input sentence's YY.
Figure 10 shows the logit lens of the 3 most important neurons in MLP 10; more neurons can be found in Appendix D. Each neuron up- or down-weights certain output years depending on the input year YY, but no individual neuron computes greater-than. No one neuron can do so, as each neuron's activation for each input only scales that neuron's distribution over the logits, without changing its overall shape. In contrast, the correct shape of the logits differs depending on the example's start year.
Many neurons can compute greater-than when combined, though. We perform logit lens on the sum of the top-10 neurons' contributions in Figure 10. Though they do not do so individually, the top-10 neurons perform an imperfect greater-than when summed together as a group. The logit lens of MLP 10 as a whole can be thought of as the logit lens of the sum all 3072 neurons' contributions; we partially recreate this with the contributions of just the top-10 neurons. Including more top neurons produces sharper approximations of greater-than; see Appendix D for examples.
In summary, we find that even inside one MLP, the greater-than operation is spread across multiple neurons, whose outputs compose in a sum to form the correct answer. Even the contributions of a small number of relevant neurons composed begin to roughly form the correct operation. We study this in MLP 10, but observe it in the other MLPs as well. Section 3.3 suggested that GPT-2 computed greater-than across multiple important MLPs; these results suggest that moreover multiple important neurons in each MLP compose to allow the MLP to compute greater-than.
5 Does The Circuit Generalize?
We now possess a detailed circuit for year-span prediction. But one thing remains unclear: is this a general circuit for the greater-than operation? Or does it only only apply to this specific, toy task? Answering this question fully would require an in-depth exploration of circuit similarity that fall outside the scope of this paper. For the purposes of this investigation, we perform primarily qualitative analyses of tasks that preserve the format and output space of year-span prediction.
To start, we focus on three increasingly different prompts: "The <noun> started in the year 17YY and ended in the year 17", "The price of that <luxury good> ranges from $ 17YY to $ 17", and "1599, 1607, 1633, 1679, 17YY, 17". In all cases, a two-digit number greater than YY would be a reasonable next token. The model performs greater-than given all of these prompts (≥ 69% probability difference). Moreover, the circuits found via path patching are similar to those in Sections 3.2 and 3.3. When we run GPT-2 on the first two tasks, ablating all edges not in our original year-span circuit, we achieve 98.8% and 88.9% loss recovery respectively; for more details, see Appendix F. As before,
2We can confirm this using path patching as well; see Appendix E for details.
these tasks depend on MLPs 8-11 to compute greater-than; these MLPs depend on attention heads that transmit information about YY.
That said, these tasks' circuits are not all identical to the greater-than circuit. GPT-2 recovers only  $67.8\%$  of its original performance on the last when ablating everything but the year-span circuit. But, a closer look at path patching results indicated that on this task, GPT-2 used not only the entire year-span circuit, but also MLP 7 and two extra attention heads. After including those nodes, GPT-2 recovered  $90.3\%$  of performance. Similar tasks seem to use similar, but not identical, circuits.
GPT-2 produced unusual output for some tasks requiring other mathematical operations. It produced roughly symmetric distributions around YY on the task "1799, 1753, 1733, 1701, 16YY, 16", which might yield predictions smaller than YY. It behaved similarly on examples suggesting an exact answer, such as "1695, 1697, 1699, 1701, 1703, 17", which could yield 05. GPT-2 even failed at some tasks that were solvable using the greater-than circuit, like "17YY is smaller than 17"; it always predicted YY. Across all such tasks, we found that GPT-2 relied on another set of heads and MLPs entirely. So GPT-2 does not use our circuit for all math; sometimes it does not rely on it even when it should.
We also observe the opposite phenomenon: inappropriate activation of the greater-than circuit, triggered by prompts like "The <noun> ended in the year 17YY and started in the year 17" and "The <noun> lasted from the year 7YY BC to the year 7". In these cases, GPT-2 ought to predict numbers smaller than YY; however, it predicts numbers greater than YY. This is because it is using the exact same circuit used in the greater than case! GPT-2 thus overgeneralizes the use of our circuit.
Our results suggest that our circuit generalizes to some new scenarios. But what does a generalizing circuit imply about the origins of GPT-2's greater-than capabilities—do they stem from memorization [47, 7], or rich, generalizable representations of numbers [27]? Our complex circuit and the structured numbered representations found in our PCA experiments hint at some mathematical knowledge in GPT-2. However, the presence of a greater-than circuit does not preclude memorization. Our circuit could function internally as a lookup table, where attention heads transmit YY information to MLPs that then upweight the years that they have memorized to be  $>\mathrm{YY}$ . In this case, (incorrect) generalization would involve GPT-2 (incorrectly) activating the lookup circuit based on context.
Though our current evidence does not allow us to definitively attribute GPT-2's behavior to generalized mathematical ability or memorization, it suggests that the underlying mechanism is something in between the two. The lack of causal evidence for the role of structured number representations, and the fact that GPT-2 cannot handle related operations like "less-than" or "equal-to" argue against generalized math mechanisms. However, even if we view this circuit as simply retrieving memorized facts, the retrieval mechanism at work is sophisticated. GPT-2 can (imperfectly) identify greater-than scenarios and the relevant operand; it then activates a dedicated mechanism that retrieves the correct answer. GPT-2's mathematical abilities thus extend beyond a simple, exact memorization of answers.
In this paper, we attempted to bridge the gap between our understanding of mathematical abilities in toy models, and the mystery of such abilities in larger pre-trained LMs. To do so, we outlined a circuit in GPT-2 with interpretable structure and semantics, adding to the evidence that circuits are a useful way of understanding pre-trained LMs, [50] even in a more complex scenario. Our circuit is coarser-grained than findings in toy models for mathematical tasks [31], but much finer-grained than existing work on mathematics in pre-trained LMs. Moreover, we showed that this circuit in GPT-2 activates across contexts on other greater-than-adjacent tasks. Whether such cross-context activation reflects generalization or memorization is an open question for circuits work in general.
We note that our conclusions are limited by the small size of our model and dataset, and the simple phenomenon studied. Our study is very model-centric: data-driven interpretability techniques would strengthen our work. Studying circuit performance across diverse tasks could better measure the degree to which our circuit generalizes to all greater-than tasks. Similarly, studying larger models would confirm that our results hold for the models that dominate natural language processing today.
Despite these limitations, we believe that this study lays the groundwork for future work. Our small study hints at the potential for circuits as a lens for the study of memorization and generalization in pre-trained LMs. More broadly, we hope that our finding that not only attention heads, but also MLPs and their neurons can be analyzed jointly as a complex system will motivate circuits work to come.
The authors thank Buck Shlegeris, Chris MacLeod, and Arthur Conmy for their valuable feedback on earlier drafts of this work. They also thank Neel Nanda for a useful research meeting about this work. They additionally thank Dani Yogatama, as well as members of the Amsterdam and Technion NLP groups, for their helpful comments. They also appreciate the insightful reviews and discussion provided by the anonymous reviewers for this paper. They finally thank Redwood Research for both running the REMIX program, which provided many of the ideas, techniques, and collaborations behind this paper, and providing continued computational and travel support thereafter.
A More Behavioral Study of Greater-Than in GPT-2
In this section, we discuss the results of additional behavioral studies of GPT-2's behavior.
GPT-2 also predicts valid XX Although our previous experiments show that GPT-2 predicts a valid two-digit end year YY, we also verify that GPT-2 predicts a valid continuation when the output prefix XX is not provided. In this scenario, GPT-2 should either predict either a two-digit century token whose value is  $\geq$  XX, or an entire 4-digit year token that is  $\geq$  XXYY. We test this using the same dataset as in previous experiments, with the final XX removed. We find that on average, considering the top-100 tokens (95% of probability mass), 94% of their probability mass (89% of the total) is assigned to valid continuations. So, GPT-2 generally predicts valid XX.
GPT-2's top predictions are valid As suggested by the high probability difference achieved by GPT-2, its top YY predictions for an input from our dataset are quite good.  $100\%$  of the top-1 continuations, and  $98.6\%$  of top-5 continuations were correct.
Default Behavior in GPT-2 GPT-2 predicts the greater-than in greater-than scenarios, but also in scenarios where less-than would be appropriate (see Section 5). For this reason, we also tested how GPT-2 behaves in a scenario where neither greater-than nor less-than is required, in order to elicit a sort of default behavior.
To test this, we created sequences of 4-digit numbers like "XXY1, XXY2,..., XXYN, XX". We chose XX, Y1,...,YN randomly for each sequence, respecting tokenization. Sequences were generally not monotone, so any 2-digit continuation YY could have been valid. We found that GPT-2's behavior depends on the sequence length  $N$ . At low  $N$ , GPT-2 produces mostly numbers  $> \mathrm{YN}$ ; the proportion of continuations  $< \mathrm{YN}$  increases smoothly until  $50\%$  by  $N = 20$ . So even in the context of random sequences, GPT-2 often generates increasing numbers.
Beyond simple behavioral study, we conducted a circuits analysis on this task, and found that our greater-than circuit also underlies this behavior. This provides another example of GPT-2's ability to identify greater-than situations is flawed. However, it also demonstrates again that whenever we see this greater-than behavior, our circuit is responsible. An open question worth answering is why our circuit activates in such incorrect scenarios, and if this can be mitigated.
B The Full Year-Span Prediction Circuit
Now, we describe the rest of the year-span prediction circuit. This is actually not very large, as we already know most of the important components. All that remains is to understand how the input to the attention heads is crafted.
We can investigate this via iterative path-patching again: we will look for nodes that influence the attention heads. This can be done in three ways: via queries, keys, and values. The queries and keys jointly determine what the attention heads attend to. In theory, attention patterns should be relatively constant across examples: in all cases, attention heads should attend to the YY position. If this is the case, we should be able to ignore the queries and keys, and focus only the value.
In practice, attention patterns are not exactly constant as YY changes. While broad trends are similar, the intensity of the attention to the YY position varies (though not linearly with YY, as might make sense: see Figure 11). At  $\mathrm{YY} = 01$ , attention to the YY position is rather low, meaning that model greater-than behavior is less pronounced when patched. Thus, the queries and keys are somewhat important: patching them with bad data reduces performance by  $15\%$  with respect to our partial circuit. The influences on these heads via the keys are similar to those on the values, discussed in the next paragraph. The influences via the queries are distinct, but we will set these aside, and focus on the value vectors.
The most important influences on these heads are the influences on their values at the YY position. The values are combined to form each attention head's output: patching the values with 01-input entirely disrupts circuit performance, unlike patching its keys or queries.
To find influences on these values, we iteratively path patch potential components that might communicate with our attention heads via their values at the YY position. We find (Figure 12) that these
Figure 11: Attention from a7.h10 (end position) to the YY position, by input year YY
Figure 12: Iterative path patching results through attention heads' value vectors
Figure 13: Iterative path patching results for MLPs 0-3
Figure 14: (Left) Full circuit diagram. Note that grouped MLPs are interconnected; attention heads are not. (Right) Probability heatmap for the patched full circuit.
Figure 15: Path Patching Step 1: Logits
are mostly MLPs 0-3, as well as a0.h5, a0.h3, and a0.h1. We delve again into this group of MLPs (Figure 13), and see that each MLP relies on the MLPs before it and a0.h1; MLP 2 is the exception, as it does not rely on MLP 1. We know that a0.h5 can only rely on the token embeddings; there is nothing else before it in the residual stream! We attempt to find what MLP 0 depends on, but it does not rely on any of the attention heads prior to it; this indicates that it depends primarily on the token embeddings, not shown in these iterative path patching diagrams.
We have now developed a hypothesis regarding a full circuit, pictured in Figure 14. We evaluate it as before, keeping in mind that the keys and values of the attention heads are patched with the same input components as found earlier; the queries, not the focus of this section, receive all good inputs. The circuit achieves a probability difference of  $71.5\%$  (98.3% of what we achieved earlier), and a cutoff sharpness of  $10.5\%$  (again sharper than pre-patching). The qualitative results are in Figure 14.
C Circuit Finding, Step by Step
In this section, we explain the circuit finding procedure step by step, with additional diagrams to aid comprehension. We start, as indicated previously, by patching direct connections to the logits (Figure 15). This reveals connections to the logits from MLPs 8-11, as well as a9.h1. We continue with the next furthest downstream MLP, MLP 11, and see which nodes influence the circuit via it. Note that the only path through which a node  $C$  can influence the circuit via MLP 11 is ( $C$ , MLP 11, logits), in red (Figure 16, right). The results (Figure 16, left) indicate that the other MLPs influence the circuit most through MLP 11.
Figure 16: Path Patching Step 2: MLP 11
Figure 17: Path Patching Step 3: MLP 10
Figure 18: Path Patching Step 4: MLP 9
Figure 19: Path Patching Step 5: MLP 8
We continue onward to MLP 10, and see which nodes are most influential on the circuit via it. We consider all the ways in which nodes could contribute via MLP 10 to the circuit, shown in red on the right of Figure 17. The results (Figure 17) again that MLP 10 relies mostly on MLPs 8 and 9. We proceed similarly with MLP 9, considering all the ways in which it influences the circuit; the results in Figure 18 indicate that it relies mostly on MLP 8 and a9.h1.
At this juncture, it might be appropriate to ask which nodes in the graph most influence the circuit via a9.h1. However, we skip this node because, as we explain in the circuit semantics section (Section 3.3), the attention heads are acting together separately from the MLPs, performing different roles. Moreover, when analyzing attention heads, we must consider what influences their queries, keys, and values separately, a complicated task best avoided in the MLP-centric analysis. In Appendix B, we explore in greater detail the nodes that contribute to such attention heads.
Instead, we complete our circuit-finding section by finding nodes that contribute to the circuit via MLP 8. This reveals (Figure 19) the heads that identify YY, completing our initial circuit investigations.
Figure 20: Neuron contributions for each MLP 10 neuron in the top 4-11. Neurons ordered by importance, left-to-right, top-to-bottom. Blue indicates that the neuron upweights a certain predicted year, given a starting year YY, while red indicates downweighting.
Figure 21: Neuron contributions of the top-100 (left) and 200 (right) neurons in MLP 10. Blue indicates that the neuron upweights a certain predicted year, given a starting year YY, while red indicates downweighting.
D MLP 10 Neuron Contributions
In this section, we display the contributions of the top 10 most important neurons of MLP 10, found in Figure 20. Many neurons' contributions are relatively constant across YY; e.g. the 4th most important neuron always upweights later years. Others differ across YY but not predicted year; the 10th most important neuron downweights all years for the last 10 years or so, where correct answers are very few generally. The 3rd most important neuron varies in both dimensions, having 0 contribution for years YY from around 10 to 50, but a distinct pattern for all other YY. Only the first few neurons are very intense in color, as we have fixed the range of the color scale: these neurons are the most important because they cause the greatest changes when they are patched.
Combining these contributions rapidly produces patterns resembling those of the MLP as a whole. We see this weakly by viewing the top-10 neurons' contributions, but more strongly in the top-100 or 200 (of 3072) neurons (Figure 21). In these logit lens diagrams, there is a consistent increase in logit lens magnitude between YY and YY+1, for a given start year YY.
Figure 22: Direct effects of top-3 MLP 10 Neurons
Figure 23: Summed and patched direct effects of top-10 MLP 10 Neurons
E Logit Lens vs. Direct Effects via Path Patching