Circuit Component Reuse Across Tasks in Transformer Language Models
Authors: Jack Merullo
Carsten Eickhoff
Ellie Pavlick
Venue: ICLR 2024 spotlight (2024)
Keywords: interpretability
llms
mechanistic interpretability
circuit
Primary area: visualization or interpretation of learned representations
TL;DR: We analyze two different tasks that are solved with very similar circuits in GPT2, informing us about how LLMs solve problems

Abstract: Recent work in mechanistic interpretability has shown that behaviors in language models can be successfully reverse-engineered through circuit analysis. A common criticism, however, is that each circuit is task-specific, and thus such analysis cannot contribute to understanding the models at a higher level. In this work, we present evidence that insights (both low-level findings about specific heads and higher-level findings about general algorithms) can indeed generalize across tasks. Specifically, we study the circuit discovered in (Wang, 2022) for the Indirect Object Identification (IOI) task and 1.) show that it reproduces on a larger GPT2 model, and 2.) that it is mostly reused to solve a seemingly different task: Colored Objects (Ippolito & Callison-Burch, 2023). We provide evidence that the process underlying both tasks is functionally very similar, and contains about a 78% overlap in in-circuit attention heads. We further present a proof-of-concept intervention experiment, in which we adjust four attention heads in middle layers in order to ‘repair’ the Colored Objects circuit and make it behave like the IOI circuit. In doing so, we boost accuracy from 49.6% to 93.7% on the Colored Objects task and explain most sources of error. The intervention affects downstream attention heads in specific ways predicted by their interactions in the IOI circuit, indicating that this subcircuit behavior is invariant to the different task inputs. Overall, our results provide evidence that it may yet be possible to explain large language models' behavior in terms of a relatively small number of interpretable task-general algorithmic building blocks and computational components.

Published as a conference paper at ICLR 2024
CIRCUIT COMPONENT REUSE ACROSS TASKS IN TRANSFORMER LANGUAGE MODELS
Jack Merullo
Carsten Eickhoff
Department of Computer Science Brown University
School of Medicine
University of TTübinee carsten, e sicchoff -tte@unnittttieeeeee .ee
edu
Ellie Pavlick
Department of Computer Science Brown University ell wel.eeealliiie i @brown. edu
ABSTRACT
Recent work in mechanistic interpretability has shown that behaviors in language models be successfully reverse- wverse-engineeee through eircuit analysis. A com- mon criticism, however, is that each circuit is task-specific, and thus such analysis cannot contribute understanding the models at higher level. In this work, we present evidence that insights (both low-level findings about specific heads and higher-level findings about general algorithms) can indeed generalize across tasks. Specifically, we study the circuit discovered in Wang et al.) (2022) for the Indirect Object Identification (IOI) task and 1.) show that it reproduces on a larger GPT2 model, and 2.) that it is mostly reused solve a seemingly different task: Colored Objects (Ippolito & Callison-Bureh 2023). We provide evidence that the process underlying both tasks is functionally very similar, and contains about a 78% overlap in in-circuit attention heads. We further present a proof-of-concee intervention experiment, in which adjust four attention heads in middle layers in order to 'repair' the Colored Objects circuit and make behave like the IOI circuit. In doing so, boost accuracy from 49.6% 93.7% the Colored Ob- jects task and explain most sources of error. The intervention affects downstream attention heads in specific ways predicted by their interactions in the IOI circuit, indicating that this subcircuit behavior is invariant to the different task inputs. Overall, our results provide evidence that it may yet be possible to explain large language models' behavior in terms of relatively small number of interpretable task-general algorithmic building blocks and computational components
INTRODUCTION
Neural networks are powerful but their internal workings are infamously opaque. Recent work in mechanistic interpretabilityy explains specific processes within language models (LMs) using circuit analysis (Wang et al.. 2022 Hanna al. 2023 Lieberum al., (223)) which reverse-engineers small subnetworks that explain some behavior on a dataset. Through causal interventions, these studies are able attribute interpretable roles for specific model components involved in predicting the correct answer for the task. A major criticism of this work is that while these studies explain the exact task being studied very well, unclear how they help our understanding of model behavior beyond that domain. While current evidence suggests that there is at least some reuse of highly general components like induction heads (Olsson al. (2022). it has yet to shown whether larger structures recovered in circuit analysis can be repurposed for different tasks. In the most pessimistic case, every different task is handled idiomatically by the model. If this were true, having circuit for every task would leave model itself.
better off from interpretability standpoint than having the full
"ooee available at: attppp////tttttccc////eeeooooaoa

---

Published as a conference paper at ICLR 2024
To better understand this, study whether LMs reuse model components across different tasks to accomplish the same general behaviors, and whether these compose together in circuits in pre- dictable ways. We study two tasks that have obvious linguistic overlap but that we hypothesize may require similar processes to solve. One is the Indirect Object Identification (IOI) task, for which Wang et al. (2022) discover a circuit in GPT2-Small (Radford et al.). The other is the Colored Ob- jects task (Figure I. which is variation on an in-context learning task from BIG-Bench (Ippolito & Callison- 2023). To solve the Colored Objects task, a model must copy a token from list of possible options in context. Based on the interpretation of the IOI circuit, which performs a behavior along those lines, a simple way to test the idea of neural reuse would be to see if these two tasks use the same circuit. We perform path patching (Wang et al.) 2022 Goldowsky-Dill et al. 2023) on both of these tasks on GPT2-Medium (Sections 3 and 4 and compare their circuits. Our causal interventions show that the Colored Objects circuit uses largely the same process, and around 78% % of the 'in-circuit' attention heads are shared between the two tasks Such a high degree of overlap supports the idea of pencral-purpose reuse. We use this insight to design 'ideal' interventions on the model that both improve the Colored Objects performance from 49.6% to 93.7% and provide evidence that at least part of the original IOI circuit appears to be part of a more generic circuit for controlling selection among competing alternatives in context.
Our contributions are summarized below:
1. In Section 3 we reproduce the IOI circuit on GPT2-Medium, showing that this circuit has been learned by more than one model. We also expand on the understanding of the role and functionality of inhibition and negative mover heads in models.
2. In Section 4 we perform circuit analysis on the Colored Objects task, and show that the GPT2 breaks it down into largely the same principal steps as the IOI task, using approxi- mately 78% of the same most-important attention heads to do so.
3. In Section 5 by intervening on the inactive parts of the Colored Objects circuit to make it act more like the IOI circuit, we increase task accuracy from 49.6% to 93.7%. More importantly. we empirically show that these interventions have the downstream effect that would be predicted by the interactions in the IOI circuit, showing that the inhibition- head subcircuit is a structure in the model that is robust across changes in the input task.
2 EXPERIMENTAL SETUP
In this work, we show that the circuit which solves Indirect Object Identification (IOI) task from Wang et al. (2022) uses most of the same components as the circuit for seemingly different task. We use path patching (Wang et af. 2022, Goldowsky-Dill et al., 2023), attention pattern analysis, and logit attribution to reverse engineer and explain the important model components necessary to predict the correct answer. The two tasks we use are described below. We briefly describe our methods below (2.2), and in greater depth in Appendices A and B
2.1 TASKS
Indirect Object Identification: We use the IOI task from Wang et al. 2022), which requires a model to predict the name of the indirect object in sentence. The following example represen- tative of the 1000 examples we use in our dataset: "Then, Matthew [IO] and Robert[S1] had a lot of fun at the school. Robert[S2] gave a ring to" where the LM is expected to predict "Matthew". The models we analyze perform well on this task, preferring the 10 token to the Subject token in the logits 100% of the time.
Colored Objects:
The Colored Objects task requires the model to generate the color of an object that was previously described in context among other other objects. An example is shown in Figure
D We modify the Reasoning about Colored Objects task from the BIG-Bench dataset (Ippolito & Callison-Burch 2023 to make it slightly simpler and always tokenize to the same length. There
?Quantifying overlap is difficult problem that we do not have an exact solution for. We discuss this in Appendix
ht whw.ggittub..ooo
tasks/ Baks/reason hacpaaaccnnnnaabbbub ____abbut__oorr____p___________e
2

---

Published as a conference paper at ICLR 2024
Q: On the table, see an orange
textbook, a red
puzzle, and purple
cup. What color is the textbook?
Orange
Q: On the table, there is blue X
pencil, black
necklace, and a yellow
lighter. What color is the pencil?
A:
In Context Colors
Context (IC) Label
Other Colors
X Correct Color
Figure 1: An example from the the modified Colored Objects task. All inputs are shot, where the first example is the In Context (IC) example, and the second is the Test example. The goal of the task is to predict the correct color (denoted by the 'X') and ignore the other color options (in particular, the other test example colors, denoted by square).
are 17 total object classes and only one object of any type appear within an example. There are eight possible colors (orange, red, purple. blue, black, yellow, brown, green). No objects have the same color within any example. We generate a dataset of 1000 examples and for path patching, generate 1000 variations of these differing only in which of the three objects is asked about. Further details can be found in Appendix H To encourage the model to only ever predict a color token as the next token, we always provide one in-context example. We find that this is enough signal for the model to predict a color as the next token: 100% of the time, the model will predict one of the three colors the test example as the next word (Figure for reference). However, GPT2-Medium does not perform consistently well, only achieving 49.6% accuracy. In Section |] we investigate this poor performance and design interventions which offer further insights into the circuit components in play in this and the IOI task.
2.2
PATH PATCHING
Path patching Wang al. (022): Goldowsky- et al. (2023) a causal intervention method for reverse-engineering circuits in networks. It involves replacing the activations from model compo- nent(s) (e.g., attention heads) on input Zoriginnl with those another input &new ("patching"). This approach allows to find attention heads that have direct causal effect on the model's predictions. To summarize, we start by finding heads that have the largest direct effect on the model logits. That is, attention head h has large direct effect if swapping the value of h when processing Zoriginal with the value that h takes when processing Inew causes a large drop in logit difference between the answers of the two inputs, Voriginal Vnew From there, find the heads that largely impact these heads, and work backwards until we have explained all the behaviors of interest to the present study, As in the original IOI paper, only patch paths between attention heads, allowing the MLPs to be recomputed. A more detailed explanation of path patching and our approach can be found in Appendix A
3 THE INDIRECT OBJECT IDENTIFICATION (IOI) CIRCUIT
Because of the poor performance of GPT2-Smal the Colored Objects task, GPT2-Medium model. This means we must first reproduce the IOI results from Wang et al. (2022) on the larger model. We find that we are able replicate the circuit described in the original paper on the new model; all of the components described in that work are active for the larger model as well, with minor differences allowing for better performance described below. More details on our results are found in Appendix C task to explain the circuit: "Then, Matthew[IO] and Robert[S1] had lot of fun the school. Robert[S2] gave ring to (Matthew)". At high level, the IOI circuit implements the follow- ing basic algorithm: 1) Duplicate names (SI and S2) are detected by Duplicate Token/Induction Heads; 2) The detection of the duplicates informs the Inhibition Heads to write an inhibitory signal about these tokens/positions into the residual stream; 3) This signal instructs the query vectors of the Mover Heads, telling them to not attend to these tokens/positions. Mover heads write into the residual stream the direction the token that they attend to (in simple terms, tell the model to predict whatever they attend to). The mover heads in 1OT attend names, and because of the inhi-
the larger
We will use the following as our running example from the IOI
3

---

Published as a conference paper at ICLR 2024
bition signal on S1 and S2, the remaining name (IO) receives more attention instead. As result, this token is copied into the residual stream (Elhage et al. 2021). causing the model to predict that token. The role of these components in IOI were described first in Wang et al. (2022). We show that
part of the circuit described here boils down to a more generic algorithm for copying from a list of potential options, rather than strictly indirect object identification.
Negative Mover Heads Our reproduced IOI circuit involves one active negative mover head (19.1). This head does the opposite of the other mover heads: whatever it attends to, it writes to the logits in the opposite direction. Wang et al. (2022) find that the negative mover heads in GPT2- Small attend all names and hypothesize that they hedge the prediction to avoid high loss. In contrast, we find that in GPT2-Medium, this head attends only to the S2 token and demotes its like- lihood as the next prediction (Appendix C). In fact, is the most important head contributing the logit difference in path patching. Our results suggest that the negative mover head in GPP22-Medimg perhaps capable of more sophisticated behavior. by directly demoting the subject token without interfering with the role of the other mover heads.
4 THE COLORED OBJECTS CIRCUIT
Given the above understanding of the IOI circuit, we now ask whether any of the discovered circuitry is repurposed in the context of a different task, namely, the Colored Objects task. Again, for space, the full analysis is available in Appendix [ We will use the following our running example from Colored Objects: "Q: On the table, I see an orange textbook, a red puzzle, and a purple cup. What color is the textbook? A: Orange Q: On the table, there is blue[col pencil[ob]] J, black necklace, and a yellow lighter. What color is the oobcillobj2]? A: (Blue)" On one level, these tasks are completely different (they depend on different syntactic and semantic structure), but on another level, they are similar: both require the model to search the previous context for matching token, and copy it into the next token prediction. We are thus interested in whether the LM views these tasks as related at the algorithmic or implementation level.
Using the methods described above (.2), we find that the principle 'algorithm' used by GPT2- Medium to solve this task is as follows. Details of the evidence supporting these steps is found in the following subsections
1. Duplicate/Inductinn Heads position.
4.3 detect the duplication of the obj1 token at the 0bj2
2. Content Gatherer Heads attend from the [end] position (the ':' token) to the con- tents of the question, writing information about the token 'color' and objz into the residual stream. These heads (ideally) provide information about which specific object is the cor- rect answer, and thus provides a positive signal for which token to copy to the mover heads. The role of these heads is qualitatively similar to those in Lieberum et al. 2023), thus we use the same term here. This step appears replace the inhibition step from 1OT ($3).
3. A fter the information about the question has been collated in the final token, Mover Heads ($4.I) attend from the final position to the three color words and write in the direction of the one most prominently attended to. We do not find that there any negative mover heads contributing significantly to the logits.
The above process is remarkably similar algorithmically to that used for IOI: the detection of dupli- cation informs the mover heads either where or where not to look. Figure 3 visualizes this overlap and the relative importance of each head, which shows a very large overlap between the heads per- forming this function; thresholding at the 2% most important heads for each eircuit, we find that 25/32, or 78% of the circuit is shared. This is a difference of swapping out three inhibition heads and negative mover head for three content gatherer heads. It is worth noting that such overlap is not trivial-iee., it is not as though any two circuits will contain the same components or even over- lapping algorithmic steps. See, for example, the circuit described for the greater than task in Hanna et al. 2023). which looks fundamentally different" In the following sections, we will describe the
"compared to IOI on GPT2-Small there is essentially no overlap in specific behaviors and heads
4

---

Published as a conference paper at ICLR 2024
evidence for the roles we prescribe to components in circuit, before returning to evaluating the dif- ferences to the IOI circuit. For consistency with the path patching process, we decribe components working backward from the end of the algorithm to the beginning.
4.1
MOVER HEADS
Path patching the logits identifies heads at 15.14 (Layer 15, Head 14), 16.15, 17.4, 18.5, 19.15, among others as mover heads. We find that many of the heads in the later third of GPT2-Medium are contributing to copying. This one of the most obvious points of overlap we find between these two circuits. In fact, we find that most of the exact same heads (e.g.. the most important 15.14, 16.15) which were responsible for the final copy step IOI are also responsible for the final copy step in Colored Objects. While the relative importance differs (e.g., 18.5 is the much more important for IOI and 15.14 is the most important for Colored Objects), there nonetheless a remarkable amount of component level overlap. This is the first piece of evidence provide that model components are generic and are reused across different tasks. We repeat the same analysis done on the IOI mover heads in Appendix D.I] where we show that these heads preferentially attend to color tokens in the test example, promote the prediction of those words by writing in their embedding directions, and the same those used by the IOI circuit.
4.2 CONTENT GATHERER HEADS
Path patching to the query vectors of the mover heads reveals that heads 11.6, 11.7, and 12.15 (and to lesser extent, 13.3) are performing the role of content gatherer heads. This is major difference from the IOI circuit. Namely, the Colored Object circuit relies on content gatherer (CG), rather than inhibition heads for influencing the query vectors of the mover heads. While inhibition heads tell the mover heads which tokens to ignore, the content gatherer heads tell the mover heads which tokens/positions to focus on. Collectively, these heads attend primarily to the obj2 token and the token 'color' in the question (Figure
To confirm that they are playing this role, we block attention to the words in the question part of the test prompt about which we hypothesize the CGs passing information. In particular, if these heads account for all of the signal that tells the model which object is being asked about, then the mover heads should randomly copy between the three color words in the test example. By doing so, accuracy is reduced to 35%, i.e., the model is selecting the correct color from among the three test example colors roughly at random chance. For three seeds, we randomly sample three heads from these layers (without replacement) that are not predicted as playing role in content- content-gathering and repeat this intervention the average accuracy for random heads is 49 .0.%% 1.0. Further analysis on these heads is done in Appendix D.2
intervention follows: the [end] position,
Average Attention Paid Content Gatherer
4.3 DUPLICATE/INDUCTION HEADS
The obj2 token is the most heavily attended to and important token for passing signal to the [end] position via CG heads (see previous sec- tion). We find that path patching to the value vectors of the CG heads has the biggest impact on the logit difference at this position, indicat- ing that they are most impacted by duplicate to- ken heads. As inhibition heads in IOI are ac- tivated by the detection of duplicate mention of name, signalling for the inhibition of at- tention to that token, this same duplicate token signal influences the values of the content gatherer heads. By path patching to the value vectors of the content gatherer heads at the obj2 position, we find that induction heads like 9.3 and duplicate token heads like 6.4 attend from obj2 to obj1 or the obji+1 +1 token (due to how induction heads move information, see Olsson et al. (2022) Wang et al. (2022). This signal helps the model find where to look to find the final color token. These heads overlap significantly with those in IOI (Figure
Avera
02
81
Rilnn
Topt Labool
690
8
Falar
Figure 2: Average attention paid tokens in the input by content gatherer heads. Most attention is paid to the obj2 token and the word collor.
5

---

Published as a conference paper at ICLR 2024
Importance for Collaned Objects
duplication subject
Objects: askud
inliiiiiion
12
13.16
201: Infibit the 52 tokee thut the haads don't it
TrhIbi Hleas
MEPTT
Nove took and obt: info to the [ond] postition so the model uupies the currect color
Nover Heads:
Copy subject taken prediction
Colored cornespording
Figure 3: Left: The full graph of attention heads with the difference of the path patching importance between each task (normalized by task per each path patching iteration). Middle: Visualizing only the union the top 2% most important heads (per path patching stage) for each task, colored by the difference in importance scores. Right: Explanation of each stage of processing the circuit. Both circuits involve the same general process: detecting a duplication and using that duplication decide which token copy. In the Colored Objects task, the duplication used as a "positive" signal via the content gatherer heads to tell the mover heads which token to copy, while in IOI the duplication sends a "negative" signal via the inhibition heads to tell the mover heads which tokens to ignore. These heads, and the activation of the negative mover head in IOI constitute the only major difference between the two tasks.
How Strangly 12.3 Writes the Token Embedding Direction Relative Attn Prob
TycemiC
TyperiC Label
Tolen Tyce-Thet Other Colers
TyzerThet Calor
Dotwe Taken Emiedd
16
0.4
0/2
Athn Prob on Taken
Atii Prob Token
Aca Prob Token
Azn Prob Token
Figure 4: Analyzing one of the inhibition head's (12.3) activity on the Colored Objects task shows that it is attending strongly to test color words and the in-context label (and writing in the opposite direction in embedding space, when attention is high), although they do not affect the mover heads as they do in IOI. Scatter plots for the other two inhibition heads are shown in Appendix Colors indicate the color of the word being attended to. See Appendix
B for explanations of the axes.

---

Published as a conference paper at ICLR 2024
4.4 IOI COMPONENTS MISSING IN COLORED OBJECTS
In comparing the Colored Objects circuit above with the IOI circuit, the most notable differences are the addition of content gatherer heads and the absence of the Inhibition and Negative Mover Heads. Investigating further, we find that these missing heads are in fact active in Colored Objects, but are receiving incorrect biases and promoting a noisy signal which does not help the model find the answer in the logits. We elaborate on this analysis below, and use it to motivate a proof-of-concept intervention on the Colored Objects task in Section
Inhibition Heads Inhibition heads prevent the mover heads from attending to some token or po- sition written into the residual stream (Wang et al. (2022); Appendix C.. From patching to query vectors of the mover heads on Colored Objects, we see that the inhibition heads are actually mildly working against the model: One would expect patching in another activation into the heads would hurt performance, but it actually improves logit difference from around 1-3%. We explore why this the case here. We find that this inhibition signal actually does exist for the Colored Objects task, but exhibits a noisy signal that prevents it from having a useful impact on the prediction. Figure E shows the attention to color tokens against the writing direction of inhibition head 12.3. We observe that the model is able to recognize reasonable places to place an inhibition signal: On the answer to the in context example (the IC label), and on the test example color tokens. However, the model spreads the inhibition signal across all of these and shows no bias to attend to the wrong answers as we might expect the model to do. This is notable because in the IOI task the inhibition heads depend on duplication to know what to attend to, whereas here they are attending to color tokens without those tokens having been duplicated in the input. We hypothesize that the inhibition heads would help improve performance the way they do in the IOI task if they were to get the right signal, which we explore in Section 5
Negative Mover Heads On the Colored Objects task the negative mover head (19.1) does not heavily impact the logits, however does seem be performing negative token moving (Appendix D.4]. The head places most of its attention on the very first token (anecdotally, this has been reported as indicating that it is "parked" i.e., doing nothing, see KKoaayashi al. (2020) for a related result in encoder-only models), with typically less than 5% attention to the color tokens in the test example and the in-context example's label token (the token after the previous "A:").
4.5 SUMMARY OF SIMILARITIES AND DIFFERENCES
Figure summarizes the role of each head and shows the overlap over the entire model and in the top 2% of important heads (which we deterimine to be the minimal threshold for containing the in- circuit components for both tasks). Despite the differences in the task structure, the Colored Objects circuit resembles that used by the model to solve IOI. We observe that the mover and induction head components not only act consistently between the two tasks: using duplication detection to decide on which token to promote with the mover heads, but the exact heads for each function in the circuits are largely the same. We find that the inhibition heads, which are active and contributing to the circuit in IOI, are also active in Colored Objects, but receiving incorrect biases and promoting a noisy signal which does not help the mover heads find the answer downstream.
In the following sections, we show that when we intervene on the model to make to make them be- have more like the 10I circuit, task accuracy increases dramatically, and the downstream components change their behavior in the expected ways (as a result the inhibition intervention in particular). We use this evidence to argue that the model learns modular subcircuit for inhibition and copying that is able to use across task domains.
5 ERROR ANALYSIS ON COLORED OBJECTS
As previously described, the accuracy of GPT2- on the Colored Objects task is low (49.6%), ). Intuitively, an inhibition signal would be very helpful for the model, as 100% of the time the model predicts that the answer is one of the three colors from the test example. This indicates that the mover heads have trouble selecting the right color over the distractors. According to the analysis on the IOI task, inhibition heads are critical for solving this problem, but they do not give a useful
7

---

Published as a conference paper at ICLR 2024
Effect of Interventions on Model Behavior
Accuracy on Full
Logit Attribution
Attention to Wrong Colors
Colored Objects Dalaset (1)
for Mover Heads (1)
from Mover Heads (4)
100
Neg:
heads Only
Average Logit Difference
Average Attention Score
Both Intervetiion
020
0.00
Accuracy (%)
0.04
20-
0.00
0.00
Baseline
Interventions
Baseline
Interventions
Baseline
Interventions
Figure 5: Intervening on the attention patterns of the inhibition heads and negative mover increase accuracy the full dataset from 49.6% to 93.7%. Furthermore, the interventions (specifically the inhibition heads) affect the mover heads in the ways predicted by the 1OI circuit. The right two graphs show a comparison of the logit difference and the attention wrong colors before and after the intervention; results for these two are taken only over the 496 examples originally gets right, for fair comparison. This evidence together suggests that the inhibition- mover subcircuit is itself manipulable structure within the model that is invariant to the highly different input domains that we used our experiments. Error bars show standard error.
signal in Colored Objects. Likewise, the negative mover head is also not active. Given the results that other attention heads are being reused for similar purposes, it should follow that intervening on these to provide the right signal should improve the model performance and recover the missing parts of the IOI copying circuit.
In this section, we intervene on the model's forward pass to artificially activate these model com- ponents behave as the IOI task would predict they should. To do this, intervene the three inhibition heads (12.3, 13.4, 13.13) and the negative mover head (19.1) we identify, forcing them to attend from the [end] position (*:" token) to the incorrect color options. Consider the example in Figure [ we would split the attention on these heads to 50% on the black" and yellow" tokens. We hypothesize that the model will integrate these interventions in meaningful way that helps performance.
With the attention pattern intervention described above, we are able to improve accuracy from 49.7% to 93.7% with both interventions, to 78.1 1% with just the negative mover head, and 81.5% with just inhibition head interventions (Figure 5 Importantly, the intervention introduces zero new mis- takes. We find that the negative mover head does simply demote in the logits whatever it attends to. The IOI-copying behavior demands that the change to the inhibition heads is affecting the pre- viously identified mover heads. In Appendices C.3 [ and G show that inhibition head 12.3 in particular does have some effect on the logits in both tasks, but not enough to explain the perfor- mance increase. To conclusively show that the IOI copying behavior is recovered, we analyze their downstream effect on the mover heads.
If the IOI circuit is involved the way hypothesize is for Colored Objects, then, result of the inhibition head intervention, we should see the attention of the mover heads to the incorrect colors decrease and the logit attribution of those heads to increase. In Figure [5) we see exactly that. The attention to the incorrect colors decreases significantly (on average -8.7%), while attention to the correct color remains high, increases (on average 2.7%, shown in Appendix G). We also find that the logit attribution of mover heads increases mover heads are particularly correlated with a higher logit attribution as a result of the intervention: The Spearman correlation between the change in logit attribution of heads after the intervention (excluding all intervened heads) and their original path patching effect on the logits (i.e., mover heads) is 0.69 (p < 0.01). Overall, the results suggest that the intervention helps the model recover the rest of the circuit that was first observed on the IOI task and explains how the performance increases drastically.
average (about 3x). Across all heads,
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
In transformer language models, there are obvious examples of reuse: simple components like in- duction heads, previous token heads, and duplicate token heads (Olsson et al., 2022). While these implement specialized functions, it has been shown that their functionality is reused to contribute non-obvious ways to arbitrary tasks, such as in-context learning. These contrast with special- ized components that activate change specific conceptual features, e.g., neurons that detect the French language (Gurnee et al. 2023). or components storing specific factual associations (Meng et al. 2022 Geva et al. 2021). More broadly, the ways attention heads are used by model to build predictions in models has been widely studied (Voita et al., [2019] Vig. 2019) and more recently, using causal interventions, (Pearl Vig al.) 22000] Jeoung & Diesner 2022), path patching Wang et al. (2022): Goldowsky-Dill et al. (2023), and other circuit analysis techniques (Nanda et al. 2022
Conmy et al. 2023 Elhage et al. 2021).
7
CONCLUSION AND FURTHER DISCUSSION
7.1 CONCLUSION
major concern with the methods of mechanistic interpretability we address in this work is the possibility that analyzing circuits in smaller models and/or on toy tasks might not lead to better understanding of neural networks in general. Our work provides a proof of concept where results from an analysis done on small model (GPT2-Small) on a toy task (IOI) in |Wang et al. (2022) allows us to make accurate predictions about different and more complex task task on bigger version of that model (GPT2-Medium). Aligning the IOI circuit onto GPT2-Medium revealed simple and obvious avenue via which we could 'deally'' intervene on the model to fix model errors. Although the scale of the model tested here still relatively small, related work has already shown that circuit analysis techniques can help us understand models in the billions of parameters range (Lieberum et al. (023). That, combined with the insights we present here, suggests an exciting path forward for understanding, controlling, and improving even production-scale models.
7.2 PREDICTING MODEL BEHAVIORS
The behavior of LMS infamously hard to predict. Current auditing work only catches "bad' be- havior after a model has already been deployed. If we know what the core algorithmic steps that LMs employ are and how they tend to decompose complex functions into those steps, we can better anticipate their behavior and predict what problems they will and won't 't solve well. A core promise of interpretability research is the ability to understand LMs at a high enough level that we can predict how changes will make them behave on new data. Our work shows an interesting case where we achieve a higher level understanding of larger model behavior built on lower level analyses of a small model, though Appendix Juggeests that showing this might not be trivial.
Connecting similarities in low-level analyses like circuits can lead us to understanding more abstract processes that underlie model behaviors, like those described in Holtzman et al. (2023). For exam- ple, knowledge general iltibition-mover interactions could help explain selection among multiple alternatives in a context.
7.3
WHY CAN'T T GPT2 ACTIVATE THE INHIBITION HEADS ON ITS OWN?
What prevents the inhibition heads from consistently attending the right colors in the middle layers? We speculate that there is a bottleneck in the model, where the signal propagated by the content gatherer heads, telling the model which object being asked about, does not become fully formed before the inhibition heads are called. Such a bottleneck in a small model like GPT2- Medium could partly explain why scale tends to help models solve more complex tasks: because either the content gathering signal is fully formed before the inhibition heads are called, or because heads that act as inhibition heads are implemented redundantly in deeper networks (Tamkin et al. 2020 Merullo et al. 2023). This could also motivate work training recurrent models, in which, for example, inhibition heads could activate on a second pass if they could not get a strong signal in the first. Such a case has been argued for in vision applications (Linsley et al. 2018).

---

Published as a conference paper at ICLR 2024
8 ACKNOWLEDGMENTS
We would like to thank Aaron Traylor, Catherine Chen, Michael Lepori, Etha Hua, and members of the Google Deepmind mechanistic interpretability team for feedback this work.
REFERENCES
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adria Garriga-Alonso Towards automated circuit discovery for mechanistic interpretability, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http::/transfornerrcicciiit pwwps/ppppppipeeee
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-54555 5495, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 12.18863//12200 .e2p:0...mpppmiii...... URL logy www.g.orgg//.
wrt.org//o 2021 emn wwneemmmpmmmmmm ma i nn. 446
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models, 2023.
Nicholas Goldowsky-De Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.085 0096699 2023.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.016160 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: In- terpreting mathematical abilities in a pre-trained language model, 2023.
Ari Holtzman, Peter West, and Luke Zettlemoyer. Generative models as a complex systems science: How can we make sense of large language model behavior? preprint, 2023.
Daphne Ippolito and Chris Callison-Burch. Beyond the imitation game: Quantifying and extrapo- lating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-88555 8856. BRLLhtts:////peerree
Sullam Jeoung and Jana Diesner. What changed? investigating debiasing methods using causal mediation analysis. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pp. 255 265,, Seattle, Washington, July 2022. Association for Compu- tational Linguistics. doi: 10. 1211853/112022.gebbig/ 1.26. URI. htttss://acalathhhhee
brg/2022. gebn 1111pp111... 1.26
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. Attention is not only a weight: Analyzing transformers with vector norms. In Proceedings of the 2020 Conference Empir- ical Methods in Natural Language Processing (EMNLP), pp. 7057-7075, Online, November 2020. Association for Computational Linguisties. doi: 10.
URL
ht http:://eeeLanttooooy.ooro wwwp://wwwooo..oo/2......o emnlp- ma mamii..... 574
Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023.
Drew Linsley, Junkyung Kim, Vijay Veerabadran, Charles Windolf, and Thomas Serre. Learning long-range spatial dependencies with horizontal gated recurrent units. Advances in neural infor- mation processing systems, 31, 2018.
10

---

Published as a conference paper at ICLR 2024
Kevin Meng, Arna Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2022.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec- style vector arithmetic, 2023.
Neel Nanda and Joseph Bloom. Transformerlens, 2022. URL https://github. com/ Dhe__aandd--o//raasfforree
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress mea- sures for grokking via mechanistic interpretability. The Eleventh International Conference on Learning Representations, September 2022. BRLhttps:/ http::ss///peenreeee...e.e//
Id=9XFSbDPmcW
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 2020. doi: 10.239 111311//iisii..00004.0000
https://distill http://ddillllpbooooo//////////o
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022.
J Pearl. Direct and indirect effects. 2001 Proceedings of the seventeenth conference uncertainty in artificial intelligence.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Alex Tamkin, Trisha Singh, Davide Giovanardi, and Noah Goodman. Investigating transferabilityy in pretrained language models. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1393-1401, 2020.
Jesse Vig. A multiscale visualization of attention in the transformer model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37-42, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/719-3007. URL URL.ttts::
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems, volume 33, pp. 12388 12388-12401. Curran As- sociates, Inc., 2020. URL
neee:eeeeeddnns.eeeeeeeeee neur ips. cc wwtppppppe//22//aaaa/aaappp Abst
2265022e22211155332eeeaaaaaaaa.2 abo.:aaaac.catssaaaaaaaattttaa. .tml
Elena Voita. David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 5797-5808, Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/118653/1//119-1550 URL https: http://aaaaaatoog..ooo/r////////
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the Wild: A Circuit for Indirect Object Identification in GPT-2 Small. In The Eleventh International Conference on Learning Representations, September 2022. URL
https:
opw.reviewreeeeeeeeeeeeee. https://openreview net wtt.:/ooouomnooooeeeeeaaoaooooo

---

Published as a conference paper at ICLR 2024
A EXPLANATION OF PATH PATCHING
Which components in a model affect the value of its final prediction? How are those components affected by earlier parts of the model? Path patching is a causal intervention method introduced in Wang et al. (202)); Holdowsky-Dill et al.] (2023) that builds on causal mediation analysis (Pearl) which has previously been used to study causal connections in neural networks (Vig et al. 2020). We provide summary that covers how we use it in our work, see the previous literature for more general purpose and detailed description of the process. The premise of the technique is to run forward pass on two examples: A and B. between which the next token prediction should be different, and cache every activation from the model for both (e.g., the attention layer outputs, MLP outputs, etc.). Take for example these two inputs from the IOI dataset:
* A: Then, Matthew and Robert had lot of fun at the school. Robert gave a ring to
. B: Then, Matthew and Robert had lot of fun at the school. Matthew gave ring to
These two examples are minimally different, and we would like to learn how the model computes which name should be predicted (i.e., identifying the indirect object token, Matthew or Robert). To find the circuit the model uses, we intervene on the forward pass of one of the inputs, changing indi- vidual activations to see how they affect the downstream prediction. We will first describe activation patching, which is the basis for path patching (and simpler to understand) as preliminary for then describing path patching in more detail.
A.1 ACTIVATION PATCHING
Let ha be some attention head output vector for the head h in the cache of activations from input A. If we believe the output computed by h is very important for predicting the final answer, we can run a forward pass on B and replace the output vector fsg with ha and continue the forward pass as before. We can then check the logits and see if using ha had the effect of promoting the answer to A. But by doing this, it is unclear if ha had the effect of directly influencing the logits, or influenced the logits through the effect had on some other model component downstream of it. To separate out the direct effect, we use path patching
A.2 PATH PATCHING
In path patching, we would like to isolate the effect of patching in some value h. on some down- stream component(s) like the logits, or even other attention heads. If the next token prediction vector is represented as z, the attention and MLP layers update by adding into it, c.g., If we change ha, x will obviously be changed as well. Since downstream components use x as input, if we affect the value of the later components will be affected by that change and thus introduce indirect causal effects on the prediction. To avoid this, we have to patch in every activation from the cache for B's activations if that component would be affected by the change forward pass. If every subsequent component is unaffected by the change from ha. then only the final logits will be affected by that patch. We can selectively allow the change to x via he to affect some downstream components other than the logits (like other attention heads) to get hg 's direct effect on those. The general path patching algorithm requires four forward passes as follows:
+ MLP(x).
earlier on in the
Forward pass 1: Cache all of the activations for input
Forward pass 2: Cache all of the activations for input B
Forward pass 3: Select some model component(s) that you want patch from to B. call this the sender set s. Select some component(s) for which you want to quantify how changing affects them ("How does changing affect r??".. Call the receiver set r. Run the forward pass on B, patching in with the activations from A. Patch in all downstream components B's cache, except for those r. Components in r get recomputed and now account for the change made by patching s. Let these recomputed values be r'
Forward pass 4: Run the model on B again, this time activation patching in the values for r'. Measure the difference in logits between the answers for A and B, this tells you how changing the path from affects the logits.
12

---

Published as a conference paper at ICLR 2024
See Wang al. (2022) and Goldowsky- Dill et al. (2023) for more details.
In practice, we start by path patching one attention head at time to the logits. After selecting the heads that affect the logits the most, repeat the process, but this time path patching each head one at a time to see how they affect this selected subset of heads, etc. We follow the path patching procedure from Wang et al. (2022) closely the larger GPT2-Medium model, meaning allow the MLP layers to be recomputed for every pass.
B GLOSSARY OF TERMS
Throughout the paper, we use terms that are highly specific to this line work. Here we take more space to provide definitions of such terms.
Circuit Although specifics differ depending on the context its use, a circuit typically refers the path through the components (attention heads, MLPs) of model that compute some behavior. If we think all of the connections through a neural network as edges in graph, a circuit is the path through those edges that allows the model to solve some task. Olah et al. (2020) define a circuit in much more detail, although the definition used here (and more generally in other work) is bit looser.
2. "residual stream" ''write in the residual stream" Transformers consist of attention and MLP blocks (ignoring LayerNorms). The outputs of these blocks are added to the inputs of the blocks through the residual connection. Since the next word prediction starts the embedding for the current word, all updates to it are made through residual connections. Therefore, useful abstraction for how information moves is through residual stream (Elhage et al., 2021) where information added to the stream by attention/MLP layers. Therefore writing to the residual stream is done by adding vector to This perspective
helps in understanding how path patching is implemented.
3.
[end] refers to the last token in a sequence. For example, the to" token in the IOI exam- ples.
4. "patching to the X vectors" refers to path patching the outputs of earlier heads from one input text into the forward pass of a minimally different input text such that affects the X vectors of some later head, where X is either the keys, queries, or values of the later head. In other words, we only allow the output of some earlier head to affect the gueries/keys/vaues of the later head.
5. Dot w/ Token Embed (E.g., Figure 4). This measurement is short for "dot product with the embedding vector of a given token". In Figures 4 and 11 for example this refers to taking the dot product between the output some attention head and the vector in the unembedding matrix for some token f of interest. This tells us how much a specific head writes in the direction of a certain token, thus promoting it in the output space. See Wang et al. 2202)) for more details.
6. Attn Prob on Token (E.g., Figure 4. This is the attention probability mass assigned by an attention head to a token in the sequence.
C PATH PATCHING RESULTS ON IOI
In this section we describe the process we use to reverse-engineer the IOI task on GPT2-Medium. We follow the path patching process from Wang et al. (2022) as closely as possible and are able to replicate the same circuit. When choices are made to e.g. path patch to the queries of the mover heads, it implied that path patching to the values or keys of these
C. MOVER HEADS
First, by path patching attention heads to the logits at the [end] position.
We show example of mover heads in Figure 7 which shows that the dot product between the token embedding for the 10 token being attended to and the output of the mover head is high across the examples and correlates strongly with the attention paid to those tokens. That is: these heads
13

---

Published as a conference paper at ICLR 2024
% Logit Difference From Fath Patched Heads Logit
% Logit Difference From Fath Potched Heads to Mover Head Quarles
Laye
10-
Logit S-Inhibition
15
Figure 6: Full path patching results for the IOI task. Arrows show the three path patching iterations. Color bars show percent logit difference. The lower the number, the more important the head is, as patching it causes the logit difference drop (i.e., harms performance). The heads that find most directly affect the logits are the mover heads. Path patching to the mover head query vectors (to see how these heads 'decide' where attend) reveals the inhibition heads (12.3, 13.4, 13.14). Path patching to the value vectors of the inhibition heads (to see what the inhibition heads write into the residual stream), we find the induction and duplicate token heads, which sensitive the duplication of the subject token.
14

---

Published as a conference paper at ICLR 2024
copy whatever they attend to, and in this setting, they attend to the 10 name in the test examples. The attention paid to and the corresponding dot products for the subject tokens is also shown, which is much lower, indicating that these not being copied by these heads.
C.2 THE NEGATIVE MOVER HEAD
In the main paper, we show that the negative mover head in GPT2-Medium is performing a help- ful behavior: demoting the subject token the logits without interfering with the positive mover heads. As shown in the original IOI analysis, the negative mover heads in GPT2-Small demote the I0 token, not the subject token, thus hurting performance. We path patch to the query vec- tors of 19.1 in the IOI task has a different attention pattern than the other mover heads would be surprising if re- lied on the same signals decide where attend. Figure shows these results. We find that a mostly disjoint set of attention heads informs the negative mover head queries than the other mover heads, namely 16.0. 18.2. and 18.9. 16.0 and 18.2 do not appear display any obvi- ous pattern in what they attend to, but 18.9 seems to tend to attend to test color option tokens.
how it chooses to attend to the subject token instead. Since
C.3 INHIBITION HEADS
Following Wang al. (2022) path patch to the query vectors (which control what the mover heads attend to) of the mover heads with the highest logit difference (14.14,15. 15,44.55.,
5-
0
Because there are so many mover heads, we test different sub- sets, but find it does not make a noticeable dif- ference. The heads that most affect these query -2 values are the inhibition heads 12.3, 13.4, and 13.14.
15-
D PATH PATCHING
RESULTS ON COLORED OBJECTS
Figure 8: Path patching to the queries of the neg- ative mover head 19.1. Color bar shows percent logit difference as a result of the patching. This head relies on different of heads than the pos- itive mover heads.
We repeat the path patching process on the Col- ored Objects task for GPT2-Medium. We interested in answering how the model uses the information in the prompt predict the right color over the other possible options. As previ- ously described, path patching requires
D.I MOVER HEADS
The first step is path patch from each attention head the logits the end position. In this task this is equivalent to measuring how important each attention head is for predicting the color answer from the "." token. Like in the IOI task, we find that there is large of mover heads that contribute almost all of the logit difference. Two heads in particular: 15.14 and 16.15 contribute 28 and 13 % of the logit difference respectively, making them by far the largest contributors to the logit difference. Like in 101, 17.4, 18.5, and 19. 19.15 also contribute significantly. The negative mover head 19.1 only contributes abut 1% logit difference, which differs from role in IOI.
We show examples of mover heads on the Colored Objects task in Figure 11 which shows that the dot product between the token embedding for color words being attended to and the output of the mover head is high, and correlates positively with the attention paid to color words. That is: these heads copy whatever they attend to, and in this setting, they attend to the color words the test example. Notice that these are the same heads in IOI (see CC.II. We speculate that these pieces indicates that the model is using distinct subspaces to control what the two types of mover heads are attending to. GPT2-Small does not have this ability; it attends to the I0 token using both types of
15

---

Published as a conference paper at ICLR 2024
Dovepyy Miri hame trieniing Relevve
Hew She moy wiee Ondeeding Telerv
Hew Sooeyy 25.5 Mesi
hame triedding Creade fnlecve
thetum Daterider Pelldte
How Danagy
tama sesding
Figure 7: Mover head attention patterns showing copying behavior towards the IO token. The negative mover head 19.1 is shown at the bottom with the opposite trend: attending to the subject token and demoting it. A positive correlation means that the attention head pays to token, the more promotes it as the next word prediction.
16

---

Published as a conference paper at ICLR 2024
How Strongly 12.3 Wintes the Name Embeddingg Direction Relative to Altn Prob
Token Type
-10
-25
04
0.2
0.3
0.5
0.5
0.7
00
0.
Adn Prob Name
How Strongly 13.4 Writes in the Name Embedding Direction Relative Attn Prob
Token Type
Doo us Marna Enbaad
0.3
0,2
03
0.1
0.5
0.5
Adn Prob Name
How Strongly 13. 13 Writes in the Name Embedding Direction Relative to Attn Prob
Token Type
Dan oul Narn Embba
Adn Prob on Name
Figure 9: Attention patterns of inhibition heads on the 10I task with respect to the IO and Subject tokens. As in the Colored Objects task, attention head 12.3 writes negatively in the direction of what attends to, while the others do not have this pattern.
17

---

Published as a conference paper at ICLR 2024
Logi: Difference From Path Patched Heads Logits
% Logit
Quertes
-10
18
Head
15x
Difference
-12
Frand
-00
Figure 10: Full path patching results for the Colored Objects task. Color bars show percent logit difference. The lower the number, the more important the head is, as patching it causes the logit difference to drop (i.e., harms performance). We find that the patching process find the most important attention heads is similar to the one for the IOI circuit and follows the same pattern as described in Figure C Path patching to the logits highlights the mover heads (15.14 in particular). Path patching to their query vectors highlights the content gatherer heads. Path patching to their values highlights the duplicate token and induction heads most strongly.
18

---

Published as a conference paper at ICLR 2024
How Strongly 15.14 Writes in the Token Embeddiing Direction Relative to Altn Prob
Touan TypeeCC Colra
Toker TypeeiC Label
Typeriee Other Colori
Tuken Type- Camez Coke
Dat ml taken Embedd
0.2
3G
0.8
0.2
944
0.6
0.0
0.2
D-4
0.6
0.4
06
0.8
Afts Probe on Thikwn
Altn Proth on Thioen
Atm Phob on Tricen
Altn Probi on Tokken
How Strongly 16.15 Writes the Token Embedding Direction Relative Atth Prob
Jilin Sypset Cukes
loker typawld Lebel
Isvan Ieao out Dffee Calle
tokon lype- losl Cuereut Culo
D31 ed Tobent Erbbed
200
10
Alt Prob on Token
Ati Prob Token
Prob on Token
Att Token
Strongly Writes Embedkiing Direction Relative Prob
TypecTive
1.0
Dot be Iokan Eibbed
Figure 11: The three most important mover heads in the IOI circuit are also important mover heads in the Colored Objects circuit. In this task they are specifically copying the possible color word options from the test example. Colors of the dots indicate the color of the word being attended to.
19

---

Published as a conference paper at ICLR 2024
Effect of intervening Content Gatherer Heads Accuracy
Accuracy (%)
B2seline
ock color
Blockotl oth,
Ablbtte OGs
Figure 12: Showing the effect of blocking the attention from the [end] position the content gath- erer heads to various tokens. Blocking attention to the obj2 token causes the model degrade to about random chance and is similar to zero ablating the content gatherers. Blocking attention to the
token color" does not have this effect. Average Attention
Gatherer
Noecape Attertiu
20
10
10 10S00e
coleets
20.
21 whe
22 colirs
2a te
#1 rost
20
18.
10
B0.K ulbie e
24 0n
36, 38,00s
32 Sple
se tee
nd
so calon auatter
R
ss. that ca
Figure 13: Average attention paid across the entire input sequences by the content gatherer heads from the [end] position. Consistent with our proposed role of these heads, attention is primarily paid to the obj2 position (the object being asked about) and the surrounding context.
mover heads. Thus, it is maybe result of scale that the larger model is able to separate what these heads attend to. Future work needed to understand this is the case, and if so, how the model keeps these bits of information separate.
D.2 CONTENT GATHERER HEADS
In the main paper, we highlight a sample of the tokens that the content gatherer heads attend from the last token. The average attention paid to all tokens in the prompts from the [end] position is shown in Figure 2 Figure 12] we show evidence that the content gatherer heads are accounting for upproximately ly all of the signal telling the model which object to predict the color of. Specifically, blocking attention the obj2 token causes the model dip around random performance between the three color options. Zero-ablating the content gatherer heads both give similar performance drop. Interestingly, blocking attention to the token colorr does not affect performance negatively at all, but because of the nature of the prompt, there multiple places that the model can look to gather that the task to predict color word, so we can not conclude how important attending to this token actually is for the content gatherer heads.
D.3 WHAT ARE THE INHIBITION HEADS DOING?
Here we include the attention scatter plots for the three inhibition heads (12.3, 13.4, 13.13) on the Colored Objects task in Figure 14
D.4 WHAT THE NEGATIVE MOVER HEAD DOING?
We show behavior of the negative mover head the Colored Objects task in Figure 16
20

---

Published as a conference paper at ICLR 2024
How Strongly 12.3 Wiites in the Token Embedding Direction Relative to Alin Prob
Tuker Typeiic Color
Tokkan Typeeic Label
Toun Typoe"ist Other Colra
Toker Type-Tout Caneet
Dbt lodkems Eibbed
-10
-12
How Strongly 13.4 Writes in the Token Embedding Direction Relative to Attn Prob
-0.
Token
-1.5
-2.
-0.5
Strongly
Embedding
toker typeat Coloss
tokos type-k Label
wien tpoos not Other Colors
toker type- Coonect Culer
Dotwi Tolan Embed
0.
0.25
Altn Phebh Tokken
Alts Pribh Tolenn
Arn Prob Tokken
Prob Token
Figure 14: The three inhibition heads exhibit slightly different behaviors, tending attend towards the previous question's answer (following the "A:") or the color options in the test example. We do see the model selecting towards inhibiting only the wrong answers. but the attention to the test color tokens is relatively low and split among all of these options. Colors indicate the color of the word being attended to.
21

---

Published as a conference paper at ICLR 2024
Examples with Correct
Examples with Correct Color Choice in Position
Examples with Correct Color Choice Position
Attention to lest Exaiple Calor Options
Color Choice Position
0.30
Tootre
Apn
0.25
0.25
0.25
0.20
6.15
0.15
0.10
0.05
0.65
(0)
(1)
Color Choice Position
Color Choice Position
Color Choice Position
Figure 15: As shown in the main paper, blocking attention in the content gatherer heads from the [end] position to the obj2 token (the object in the question "What color is the obj2?") reduces per- formance to randomly selecting between the three color options. This figure shows how blocking this attention redistributes the attention paid by the top five mover heads. When the correct answer is listed the first third (index 0 and 2) options listed in the prompt, this reduces attention to the correct answer and increases attention to the wrong answers. This not case when correct answer is the middle option. The model tends to be sensitive to the position of the colors presented, which we discuss in Appendix
How Strongly 19.1 Writes the Token Embedding Direction Relative to Attn Prob
Typeat
Tean Typwai soal
Thire Typer
Trane Typec Crover Orlor
Arm Probi
Amn
Prob Token
Prob Token
Figure 16: The negative mover head that is highly important in the IOI task has a congruent, but less impactful role in the Colored Objects task. splits small amount of attention to the color words and to the answer token of the in-context example. Attending to the last answer would prevent the model from simply repeating the answer the previous question. The rest of the attention tends to go to the very first token in the prompt.
22

---

Published as a conference paper at ICLR 2024
101
Duplicate Token Head Matches
Induction Head Matches
Colored Objects
Head
Head
Figure 17: Left two graphs show the induction and duplicate token head matches on random repeat- ing patterns of text, right two show the path patching results for the paths to the inhibition or content gatherer heads, of which the most important heads are strong duplicate token induction heads.
Accumulated Logit Difference Layer (Colored Objects)
Acoumulsted Logit Difference Per Layer (101)
Incemeet
Loght OPPOPP ence
Logit
Laya Component
Layor Commonnntt
Figure 18: Cumulative logit attribution for both tasks. Here, higher is better: positive logit differ- ence means the correct answer is more likely than a wrong answer. Measurements for each datapoint shown dots, while the average are depicted with lines. For each layer, measurement of the next token prediction taken before and after the attention layer. The final measurement is directly before the logits, thus giving 49 layer components.
DUPLICATE TOKEN AND INDUCTION HEADS
Path patching to the value vectors inhibition heads in IOI or the content gatherer heads in Colored Objects shows that duplicate token and induction heads provide the most signal. Besides just ex- amining the attention patterns, we confirm this quantitatively with the TransformerLens (Nanda & Bloom 2022) head detector test for duplicate and induction heads using random repeating token sequences as the inputs. Figure I7 we show that there is a large overlap of important heads at this stage in path patching for both tasks, and duplicate and induction heads.
ADDITIONAL SOURCES OF ERROR IN COLORED OBJECTS
Aside from the circuit-level problems, which prevent GPT2 from accurately inhbiting attention to the incorrect color choices, we find that some circuit components are simply over sensitive to certain features or patterns in the input. As an example, we find that attention head 19.15, which is a copying head, is overly active when the color brown in the test example (and to lesser extent, orange), which causes the model erroneously promote it as the answer. This head even attends to and promotes the token when it appears in the in-context example, which is one of the only examples
23

---

Published as a conference paper at ICLR 2024
Strongly Token Emibedding
Altn Prob
Typeait
Dotto
Alan Prito Tokan
Alin Prozi Tokan
Phub Tokan
Phob Tulan
Figure 19: Attention head 19.15 overly sensitive to attending to and copying the color brown, causing large portion of errors.
Logit Altribution Heads Before After the Intervention
LogE: Artributton Towards Conrect
tdlnns
124 ki
121
nB.
G8,
02.6
49
06.0
120
Figure 20: Mover heads contribute more to predicting the right answer after the intervention. Error bars show standard error. While some heads contribute relatively little, find that heads like 16.15 and 19.15 have very dramatic increase towards promoting the correct answer.
observe of head doing something like this. The attention pattern against its output copying score on color tokens is shown in Figure 19
F.1 BIAS TOWARDS POSITION IN SEQUENCE
As is often the case with language models, observe that GPT2-medium has bias towards select- ing answers based on their position in an input. the Colored Objects task, the model is sensitive to the order in which the colored objects are presented.
G ADDITIONAL DETAILS ON INTERVENTION RESULTS
Here we provide finer-grained analysis the effects produced by the intervention experiments per- formed in the main paper. In the main paper, show the average logit attribution and attention to wrong answers for the mover heads as a result of the intervention. logit attributions of several mover heads the Colored Objects task. In Figure 22 show the attention to the correct color answer and the maximally attended to wrong answer for these same mover heads. We can see that the drop in attention to the wrong as result of the inhibition
Figure 20 we show the direct
24

---

Published as a conference paper at ICLR 2024
Accumulated logit Difference layer Before After Intervention
32
2.5
1.5
0.
Layer Componennt
Figure 21:
After the intervention, we can see large spikes in logit attribution towards the correct answer as a result of the layer 12 and 19 attention, consistent with the behavior we see in the IOI task. Results are shown for the 496 datapoints that the original model gets correct.
Mover Head Attention to the Correct Color Before and After Interventions
Mover Head Attention to the Incotrect Color Options Before and After Interventions
0.40
0.40
GPT2 Allo Ansove W: Intery
CPT2 Alti Weoing Co Interv
0.35
0.30
Attention Score
Attention Score
0.25
0.25
0.20
0.20
3.15
0.15
0.10
(44.14)
Q6 15)
98.91
(9. 151
444 14]
d3. 14)
(27 4)
(28. 151
12)
(25
(27.
120.0
(20.0
(2.
06
(28.
D8.
(28.
Figure 22: After the intervention on inhibition heads and the negative mover head, the mover heads pay less attention to the incorrect answers and the same or higher attention to the correct answer. This precise behavior is predicted by analysis on the IOI task on GPT2-Smal in Wang et al. (2022), but generalizes to our setting, suggesting that the circuit generalizes beyond the [OI task, Error bars show standard error.
head intervention is greater than the increase in attention to the right answer, which is in line with the behavior of the IOI circuit.
We also include the accumulated direct logit attribution following the attention and MLP layer out- puts across all layers of the model before and after the intervention (Figure 21). Layer 19 attention in particular increases as a result of the intervention on the negative mover head.
H DETAILS ON COLORED OBJECTS DATASET CONSTRUCTION
In the modified Colored Objects task, we give the model three colored objects and ask it to name the color of of one of the objects. The objects span a set of 17 common single-token household objects with each being assigned one of cight possible colors. Any object will only appear once in a given example, with each color only appearing once as well. Each input to the model is a one shot example, meaning we provide one example with the label before providing the additional test example. Repeats in colors and objects can appear across these two examples. The objects are as follows:
Objects: "pencil", "notebook", "pen". "cup". "plate". "jug". "mug". "puzzle", "textbook", "leash", "necklace", "bracelet", "bottle", "ball" "envelope", "lighter", "bowl"
25

---

Published as a conference paper at ICLR 2024
1 CIRCUIT OVERLAP IN ADDITIONAL TASKS
Comparing the IOI and Colored Objects tasks present an interesting case study that lets us highlight two different tasks with very similar circuits. In this section, we analyze two additional tasks that vary in their similarity to these two tasks to provide additional reference points for quantifying simi- larity. The two tasks are the Greater Than dataset (Hanna et al. al.] 2023) and simple factual recall task in which the model predicts the capital city of a country. While both tasks are fundamentally different from the IOI and Colored Objects tasks, prior work suggests specific hypotheses about the amount overlap we can expect in these circuits.
Greater Than Dataset In this task, originally introduced in Hanna et al. 2223)) the model must predict a viable end year for some fictional event. For example: "The war lasted from the year 1732 to the year 17", where any number between 33-99 possible, but 00-32 not. The authors analyze the circuit GPT2-Small uses to solve this task, which makes significant use of the MLP layers. Therefore, we expect the circuit GPT2-Medium uses to have very little overlap with the one used for IOI or Colored Objects. We use the code provided by Hanna et al. (2023) to generate dataset of 200 examples to analyze.
World Capitals Dataset Given sentence like "The capital of France is" the model must access information that is not provided in context to predict the right city. This makes the task different Geva from either the IOI and Colored Objects tasks. However, Geva et al. (2023) provide an interpretation for how such factual recall happens in LMs (not through circuit analysis). In their analysis, early MIPs enrich the subject token (the "France" token), and later attention heads extract the capital city attribute by attending from "is" to "France". In attribute extraction, some attention heads attend to subject tokens and copy information out of them. If this is the process that used for this task, our earlier results might predict that the mover heads are reused to perform this operation. We perform path patching on small dataset 200 pairs of different countries to reverse engineer the capital city prediction process. We filter out countries and capitals that are not tokenized as single token by GPT2-Mediumg giving us 47 unique countries to generate the data from.
1.1
GREATER THAN CIRCUIT
The results of our path patching experiments on this dataset are shown in Figure 23 We path patch using the probability difference metric for all years (see Hanna et al. (2023) for details), which is analogous to the logit difference metric. We first path patch to the logits from the end position and find the MLPs are most important. From there we path patch to last token position to these MLPs, and then backwards through the last token position from there. The circuit can be summarized as follows:
Attention heads 1, and 6.15, write into 7.2, 9.9, 11.1, and 11.5, 13.12, 14.14, MLPs 14 and 15. In turn these latter components write into MLPs 17-23. To summarize, the earlier induction heads (7.2, 9.9, 11.1, etc. see Figure [7] attend from the last token (e.g.. "17") to the last two digits in the start year (e.g., the "32" in "1732"). 14.14 is significant mover head in the Colored Objects task and has the same pattern. Similarly to Haana al. (2023). the MLPs perform the majority of the numerical comparison needed to predict a reasonable year. As expected, this leads to a very low overlap with the two tasks we study in the main paper, in which the MLPs play almost no part.
The Greater Than task represents a task with very low overlap, but there are still components shared with the IOI and Colored Objects circuits. Induction heads 6.15, 7.2, and 9.9 also contribute significantly in IOI and Colored Objects, but this is due to simply matching an induction pattern (Olsson et al. 2022). Mover head 14. 14 plays similar role in Colored Objects. If we just consider the 18 components in the Greater Than circuit, the Colored Objects task has an overlap of 4/18=22.2% and I0I has an overlap of 3/18=16.7%
Summary of Overlap:
I.2 WORLD CAPITALS CIRCUIT:
The path patching results for the World Capitals task are shown in Figure 24 Given prompts of the form "The capital of France is", we first path patch to the logits at the last token position ("is"), and find that the majority of components that write the logits mover heads. We path patch to
26

---

Published as a conference paper at ICLR 2024
% Probabiiity Difforence From Poth Patched Components to Logits
Greater Than Task
Probability Difference From Patched Components to Last Six MLPs
15
Head MLP
Probebility Difference From Peth Petched Components
-03
20
Head MLP
Figure 23: Heatmaps from path patching results from reverse engincering the Greater Than task from (Hanna et al. (023) on GPT2-Medium. The rightmost column represent the MLPs in each layer.
World Capitals Task
Logit
Logit Difference From Components to Mover Heads
Components to Logits
20
1ayer
30
0
15-
-20
Head or MLP
Head or MLP
Figure 24: World Capitals path patching results; the rightmost column in the heatmaps represent the MLPs of the layers. Given inputs such as "The capital of France is", the majority of behavior was captured in two steps. First, mover heads attend from the final token position to the country and extract the capital city. The value vectors of the country position for these heads depend on the MLPs in the earlier layers (in particular, MLP 0).
27

---

Published as a conference paper at ICLR 2024
the queries, keys, and values of these heads and find that the values at the country position have the greatest effect on the output of the mover heads, which are predominantly dependent on the MLPs in the early layers. The circuit is summarized below:
MLP O, 1, 6, and 9-14 build up the representation at the country position of the input. Attention heads 14.7, 15.0, and the mover heads 16.2, 19.15, 20.6, 20.12, and 21.12 attend to the country token and promote the capital city in the logits.
Summary of Overlap: Despite the differences between this task and the Colored Objects and IOI tasks, there is significant overlap, particularly with the use of mover heads to copy the final answer. The mover heads that overlap with Colored Objects and IOI are 16.2, 19.15, 20.6, and 20.12. Therefore, 4/7 of the mover heads are shared. Although less prominent, 17.4 and 15.14 also contribute the world capitals logits which appeared previously as mover heads (Figure 4.1). This aligns with our hypothesis that mover heads would be used for moving information about the capital out of the country. The rest of the circuit relies heavily on the MLPs, so only 4/16=25% of the full circuit components overlap with both the Colored Objects and IOI tasks, demonstrating how different the earlier processing is.
1.3 CHALLENGES IN QUANTIFYING CIRCUIT OVERLAP
Putting an exact number on how similar two circuits are is a very complex task that we do not attempt to fully answer this paper. At the moment, the criteria for deciding whether any component is considered in the circuit or not is not very well defined, although there are best practices defined
by previous circuit analyses [Wang et al., 2022: Lieberum et al., 2023; Goldowsky-Din et al. 2023
Hanna et al.) 2023) and there some work exploring automating this process (Conmy al. 2023).
Thus, we do our best to place an accurate number on the overlap as the proportion of shared nodes in the circuit. In the main paper, we study three processing 'steps' in each eircuit which also makes the job of assigning similarity easier, but could be possible that these numbers differ between two tasks for the purpose of other analyses. Our results motivate studying the problem of quantifying circuit overlap in future work.
J Do OTHER MODELS ALSO USE SIMILAR CIRCUITS FOR THESE TASKS?
We perform a preliminary path analysis of larger versions of GPT2 (Large and XL) by path patching from attention heads to the logits for the IOI and Colored Objects tasks (Figure D.. We find that the overlap between the heads used in either task is not as obvious in either model, however does appear that mover heads are playing a role in both cases. For both models, we compare the top 10 heads in the Colored Objects and IOI tasks. In Gare-Large, five of these heads overlap, but none overlap in GPT2-XL. These heads are follows:
GPr2-Large (Overlapping heads bolded):
101: 20,14, 30.13, 27.17, 29.0, 23.13, 26.14, 22.0, 32.18, 30.10, 21.8
Colored Objects: 22.0, 23.13, 23.7, 23. 21.8, 21.14, 26.14, 24.13, 31.9, 27.17
GPT2-XI::
IOI: 41.9, 36.17, 42.7, 28.22, 40.4, 39.9, 39.4, 35.17, 37.15, 38.12
Colored Objects: 24.16, 26.20, 28.15, 21.13, 40.13,40.18. 29.5, 30.21, 33.18, 31.3, 42.9
It appears that the overlap in the top most important heads decreases as the scale of the model goes up from GPT2-Medium to GPT2-XL, however, we find that the functions of the heads stays roughly the same, meaning that many of these heads are mover heads. We measure this by characterizing the function of each head identified in the IOI circuit on the Colored Objects task. This tells us whether the heads are performing similar roles, even if they don't 't light up as the most important according to path patching results. In GPT2-Large, the 5 shared heads are all mover heads (21.8, 22.0, 23. 13, 14, 27.17). In GPT2-XL, even though the exact heads being used are different, of the heads mover heads (35.17, 36.17. 32, 39.4) and is negative mover head (39.9) on both tasks. As models get larger, there are more 'paths' through the network, which makes less likely that exactly the same heads are used for different tasks, but it appears there is still overlap in the function of the
28

---

Published as a conference paper at ICLR 2024
Colored Objects
101
15
GPT2-Large
15-
Heod
GPT2-XL
LOW
30-
30-
Figure 25: Path patching from attention heads to the logits of GPT2-Large and GPT2-XL. There is some overlap in GPT2-Large, and almost in the most important heads for GPT2-XL, suggesting that as the number of components increases, overlap is less likely. However, we find that most of the important heads are mover heads in both cases, suggesting the underlying circuits may still be similar functionally even if they take different paths through the models. Colored bars show percent logit difference.
important heads. This is important to keep in mind, as it complicates the picture for analyzing larger models, but the upshot appears that we can still expect some amount of functional overlap between similar tasks.
K
How WAS THE COLORED OBJECTS TASK SELECTED?
The Colored Objects task has no direct connection to the IOI task, so it is not obvious why this was the choice for the point of comparison with the IOI task. This task was being studied for previous work, so it was well understood, and it was noticed that the strategy needed to solve it, i.e., the selection of the answer from give list of distractors, was qualitatively similar to that used by the circuit described in Wang et al. (0022). We did not path patch the full circuit of any other tasks in the preparation of this study, but preliminary work on other tasks that do not share this connection to the IOI task (e.g.. predicting numbers greater than some given integer, predicting the capital of country) had virtually no overlap with the IOI circuit, indicating that the amount overlap that we report in the main paper between IOI and Colored Objects is trivial. However, future work is needed to understand how commonly the mover circuit is used by GPT2, and the same components behave in different ways under different circumstances that could predict here. A limitation of this study is that we are not able to confirm the generality of the circuit beyond the tasks studied here. This type of limitation remains an open problem in the field.
29


Reviews:
[{"review_id": "Ci9swWD6my", "reviewer": "Reviewer_F5uQ", "summary": "This paper falls into the line of work of mechanistic interpretability of neural networks.\n\nThis paper compares the Indirect Object Identification (IOI) subnetwork identified in Wang et al. 2022 in GPT-2 small, with a new Colored Objects task subnetwork identified by this paper.\n\nThe paper shows that in GPT-2 medium there is a significant overlap between both subnetworks in terms of the attention heads that are activated. Furthermore, the paper conducts an analysis (with ablations and study of the attention probabilities) of the different functions of the heads in the Colored Objects subnetwork and argues that they follow a human-interpretable algorithm. The places where the Colored Objects subnetwork differs from the IOI subnetwork are argued to be the places where the IOI algorithm and the Colored Objects algorithm differ.\n\nFinally, the paper demonstrates that through a handcrafted manipulation of the internal activations, the accuracy of GPT-2 medium on the Colored Objects task can be improved from ~50% to ~100%.", "questions": "1. Could you please clarify what you mean by \"normalized by task per each path pathing iteration\" in Figure 3?\n\n2. Could you please clarify what you mean by \"due to the number of mover heads in the larger models\" in Figure 23?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The clarity of the presentation is high. The analyses seem to be of high quality. The research field of mechanistic interpretability is important, so the paper is significant insofar as it is a good-quality contribution to this field.\n\nIn my opinion the most interesting contribution of this paper is that via mechanistic interpretability analysis, the authors can get a better understanding of the reason that GPT-2 medium fails on the Colored Objects subtask, and intervene on the internal representations at the appropriate heads to get a good output. It would be interesting if this were explored further in settings beyond Colored Objects, and if this could be part of a general framework for improving LLM performance on reasoning tasks.", "weaknesses": "The finding that there are circuits that are reused within transformer networks is not new, since this is known e.g., for induction heads. (This is also pointed out by the authors in their related work section.) So I am having trouble wrapping my mind around what the new conceptual contribution of this paper is:\n\n* In terms of techniques, the identification of the IOI subnetwork in GPT-2 medium reproduces an analysis of Wang et al. (2022) using their path patching method. The Colored Objects task network is identified running the same previously-known path patching method. Here the novelty in identifying this network seems to be mainly in identifying what the different heads in this subnetwork do, and writing this as a human-interpretable algorithm. But this style of analysis already appears in the IOI paper.\n\n* Furthermore, the analysis is mentioned in the appendix to not apply to GPT-2 Large & GPT-2 XL, where there is no significant overlap between the IOI and Colored Objects subnetworks. I appreciate the honesty of the authors in reporting this negative finding, but it seems like a major strike against the phenomenon of circuit reuse advocated by this paper. It would be very interesting if the authors could give a more in-depth explanation of why we could expect GPT-2 Large and GPT-2 XL to not have circuit reuse.\n\nWith all this said, I certainly do not want to discourage the authors from this line of work, since I believe that it holds a lot of promise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcIfkPvWQ4", "reviewer": "Reviewer_mB6u", "summary": "The paper investigates the generalizability of mechanistic interpretability in language models, focusing on the Indirect Object Identification (IOI) circuit and its application to a different task, Colored Objects. The authors perform an intervention experiment to demonstrate their claims. The work seeks to address the task-specific nature of studies in interpretability and explore the possibility of understanding language models at a higher, more general level.", "questions": "# Major questions: \n*I'd be willing to consider increasing my rating if these major points are addressed. In particular, addressing the first two points presented below would significantly enhance the paper's soundness.*\n\n1. **Overlap Measurement**: The paper relies heavily on the concept of overlap in in-circuit attention heads, particularly in demonstrating the generalization of mechanisms. While it is acknowledged that putting an exact number on this similarity is complex, can the authors provide a more detailed description of the metric used for measuring overlap? Additionally, could they quantify the uncertainty associated with this metric and provide insights into the expected range of overlap values? To help readers assess the significance of the 78% overlap, would it be possible to provide reference baselines, such as comparing the IOI circuit overlap with a set of random tasks or comparing it with the overlap regarding another circuit, even if it's a randomly generated one?\n\n2. **Interpretation of Intervention Experiment**: The authors conducted an intervention experiment to demonstrate the role of certain attention heads. However, it is crucial to establish the validity of this demonstration by including control baselines. Can the authors consider reproducing the same intervention on similar attention heads that are not expected to perform the role of gathered heads? This approach would help differentiate between candidate gathered heads and other attention heads that might not have the same function. Without these control baselines, it is challenging to draw meaningful conclusions from the intervention experiment.\n\n3. **Related Work**: The related work section, while present, is relatively concise. To provide a more comprehensive context for the readers, can the authors expand on related work in the field of language model interpretability, particularly focusing on those works that address the task-specificity of interpretability findings? This would help clarify where this research fits in the existing literature and how it contributes to addressing the challenge of task-specificity in interpretability studies. Are there any specific prior works that have addressed or attempted to address the same problem or similar questions, and how does this paper compare to them? In particular, how does this work relate to the induction and duplicate heads found in Olsson et al. 2020?\n\n4. **Clarity**: Can the authors address the issues related to clarity, particularly in Section 4.3 and with the legends in the figures? Could they provide clearer explanations for terms and visual elements like \"attn prob on token\" and \"dot w/ token embed\" (Fig 4) ? Additionally, what is the significance of the color information in the figures (e.g. Fig 4), and could the authors provide more context or commentary regarding its use in the visualizations? Lastly, are there plans to improve the clarity of explanations in the appendix, given its complexity, to enhance the accessibility of the research for a broader audience? Clarifying these elements would greatly benefit the overall presentation of the work.\n\n# Additional minor questions:\n\n5. Why does the language model fail to correctly utilize the IOI circuit on the Colored Objects task? Is this failure potentially influenced by the number of distractors in the task? If so, it would be informative to explore varying the number of distractors in the task and analyzing the resulting overlap and accuracy to gain deeper insights into the model's behavior in different contexts.\n\n6. What does the term [end] position refer to in the text? It is mentioned but not clearly explained.\n\n7. In Section 4.5, the paper discusses the top 2% of important heads. Could the authors provide a more detailed explanation of how these important heads are defined, particularly in relation to the path patching technique?\n\n8. Appendix J appears to raise questions regarding the generalizability of results from a small model on a toy task to a more complex task with a larger model. The appendix seems to contradict the main paper's claims. Could the authors provide clarification and elaboration on this aspect? Specifically, how do the results from the small model analysis on a toy task relate to accurate predictions on a more complex task with a larger model? This clarification is essential to ensure consistency in the paper's argument.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Originality**: The paper addresses an important issue in the field of interpretability by examining the generalizability of mechanistic insights across tasks in language models. This is an original and valuable contribution.\n\n2. **Significance**: Understanding the extent to which language models reuse or learn task-specific circuits has significant implications for the field of machine learning and interpretability. This work provides evidence that the IOI circuits (which is actually more a high-level copying mechanism than something related to indirect object identification) is also used in a least one other task (Colored Object), suggesting the presence of a general and reusable high level mechanism learned by the LM. It is a significant result as it contributes to addressing the challenge of explaining large language models' behavior.\n\n3. **Quality**: The paper employs a range of techniques, including path patching, attention pattern analysis, and logit attribution, to investigate the proposed mechanisms. These methods demonstrate the authors' commitment to a high-quality analysis. Nevertheless, as noted in the Weaknesses Section, the study could be greatly improved by providing control baselines.", "weaknesses": "1. **Experimental Demonstration**: The experiments, as currently presented, may not be sufficient to demonstrate the generalizability of basic algorithms in language models. It is only shown for one circuit (IOI) identified with a relatively small LM (GPT2-Medium) in two toy tasks (IOI and Colored Object). It will be essential to explore other types of experiments or analyses to strengthen this argument (out of the scope of this paper). The authors already acknowledge this limitation in the paper. This is a strong limitation. At the same time, this work is a promising first step, that may foster similar work on the topic and improve our knowledge of LMs. (Therefore, this argument alone should not motivate the rejection of the paper.)\n\n2. **Overlap Measurement**: The paper mentions an overlap in in-circuit attention heads but lacks a detailed explanation of how this overlap is computed. Although \"putting an exact number on how similar two circuits are is a very complex task that [the authors] do not attempt to fully answer in this paper,\" the results in the paper heavily rely on this metric. Thus, authors should (a) provide a more detailed description of the overlap metric; (b) quantify the uncertainty of this metric and the results associated to it; and (c) provide clues to better understand its range. As an illustration, authors wrote that 78% of the IOI circuit is used in the Colored Object task. However, without more information, it is difficult to say how significant this overlap is. Providing control baselines, such as comparing the IOI circuit overlap with a set of random tasks, or computing the overlap regarding another circuit (which could be a random circuit), could help to determine if 78% is high or not. Without such information (uncertainty quantification, control baselines, etc.), the significance of the results is hard to interpret. Note that authors briefly mention such an experiment in Appendix I. This should be expanded and included in the main paper.\n\n3. **Interpretation of Intervention Experiment**: In order to prove that some heads are performing the role of gathered heads, the authors conducted an intervention experiment in the model. They did so by blocking attentions to some words in these heads and then measuring the impact on accuracy. They observed that the intervention causes accuracy to decrease and concluded that it successfully demonstrates the role of these heads. However, once again, this experiment lacks a control baseline. Authors should, for instance, reproduce the same intervention on similar heads (but that are not expected to perform the role of CG) and see if the result is different. If the interventions on random heads do not produce the decrease in accuracy observed with candidates’ heads, then it is reasonable to say that these candidates are CGs. Without these control baselines, it is not possible to draw any conclusions from the intervention.\n\n4. **Related Work**: The section on related work is too concise. To provide a broader context, the authors should expand on related work in the field of language model interpretability, specifically those that address the task-specificity of interpretability findings. This will help readers better understand where this work fits in the existing literature and how it contributes to addressing the challenge of task-specificity in interpretability studies.\n\n5. **Lack of Clarity**: The paper exhibits instances of unclear explanations and visualizations, which can hinder readers' understanding of the research. For instance, Section 4.3 is difficult to follow. In addition, there are issues with legends in figures, as terms like \"attn prob on token\" and \"dot w/ token embed\" lack sufficient explanation (e.g. in Fig 4). Furthermore, the inclusion of color information in the figures is not adequately commented upon, leaving readers to wonder about its significance. Figures 6 and 7 (Appendix), could benefit from clearer legends and more informative descriptions. Lastly, explanations in the appendix, such as Appendix B, are difficult to follow. While the complexity of mechanistic interpretability tasks may contribute to this challenge, the authors have an opportunity to improve clarity, which would greatly enhance the paper's accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6jEoSgvfc", "reviewer": "Reviewer_PvTR", "summary": "This paper investigates the similarity of circuits of information flow on two tasks - IOI, and Colored Objects (CO). They study GPT2-Medium, which can do the IOI task successfully, but struggles with the CO task. They reproduce a previous circuit analysis of the IOI task for a different model, as well as for the CO task. They find that many of the same attention heads are important for both tasks and perform the same roles, with a few exceptions. They show if these exceptions are corrected in the CO task to match behavior from the IOI task, the performance can be improved.", "questions": "Although it is briefly touched upon at the end, there is is an important question to be answered more concretely: why don’t inhibition heads appear to work properly in the Colored Objects setting when they work just fine for the IOI task? One hypothesis might be that the induction heads, which were important for influencing attention of inhibition heads in IOI, aren’t passing forward the proper information but it might be improved with more shots. For example: if you provide some more ICL examples do the inhibition heads start to work as the IOI task would expect them to? \n\nIt seems the paper has identified two potential ways through which information about what position to attend can travel to mover heads - through inhibition heads, or through content-gatherer heads.  At least for these two tasks the inhibition-mover subcircuit seems to be more robust than the content-gatherer-mover circuit. This leads to the question: is this always the case for extractive tasks when there are multiple choices that could be copied? Or are there cases where content-gatherer-mover subcircuits work well/better without inhibition heads?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper has a good understanding of related prior work and presents their work well in context.\n* The paper has clear and important motivation - whether circuit analysis and circuit components are transferable across tasks.\n* The paper clearly shows there are components that have similar functionality on two different tasks (mover heads, induction heads, and duplicate-token heads).\n* The targeted intervention in section 5 is an promising proof-of-concept - showing that understanding the interactions between different components of a circuit may allow correction of undesired behavior when using LLMs for a particular use case. This is the most interesting result of the paper and should be explored more in depth.", "weaknesses": "The paper only investigates reuse of components between two tasks, which are pointed out as similar to each other in the paper. The claims in the paper would be strengthened by investigating component reuse across additional tasks that are less similar to the original tasks studied (IOI & Colored Objects). Perhaps studying the behavior of these components on a larger distribution of text (i.e. open web text or some equivalent) that is not task-specific would provide insight into how generally we can understand the inhibition-mover components.\n\nWhile the intervention in section 5 correctly predicts the interaction between the inhibition heads and mover heads will improve performance, this ignores the other half of the circuit - induction heads, duplicate token heads, and content gatherer heads. A more thorough analysis of the interactions between these elements and the inhibition heads/mover heads would strengthen the claim that these components are part of the circuit. For example, is there an intervention on induction heads/previous token heads (or something else) that would reliably produce the attention pattern you artificially induced into the inhibition heads/negative mover heads? This would show better understanding of the entire circuit, and not just the interaction of the last two sub-components.\n\nThe reasons provided for why the inhibition signal “exists but is noisy'' were not very clear/convincing. If the IOI view of inhibition is correct, wouldn't the signal the inhibition heads receive of where to attend (to wrong colors) need to come from the induction heads/previous token heads (or a fuzzy-matching version of the same)? \n\nThe methodological contributions of the paper are limited as they mainly rely on existing approaches to discover and validate circuits - path patching, logit attribution, and visualizing attention patterns.\n\nPresentation and Clarity\nMinor errors:\nSection 2 - Experimental setup mentions “Appendix 4.3” which should be section 4.3, as it is not part of the appendix.\nSection 2.1- Colored Objects Task - “other” is repeated twice.\nSection 4.5 “deterimine” spelled incorrectly,\nSection 7.1 “task” is repeated twice", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uA9wXUHfxx", "reviewer": "Reviewer_4sdK", "summary": "The paper present evidence that there is a substantial circuit reuse between two different probed tasks in GPT2-medium: indirect object identification (IOI) and Colored Objects (CO). Furthermore, it shows a proof of concept intervention to substantially improve the accuracy at a task by \"correcting\" the behavior of the CO circuit, informed by the behavior of the IOI circuit.", "questions": "See questions above.\n- Do the MLPs play no role in the circuits explained? Are the MLPs even considered in the analysis?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- (Significance) The paper addresses an important blindspot of mechanistic interpretability (MI): transferability and generalizability. I believe that the finding of this work will be of substantial interest to the community\n- (Clarity) Overall, the paper is structurally well organized and the tasks are clearly presented. However, unfortunately, this does not place the paper in the easy-to-read tier (see below).\n- I find the error correction proof of concept particularly interesting, as it shows one possible practical application of MI. To my knowledge, this is a rather novel and worthwhile contribution.", "weaknesses": "- (Clarity) My main concern with the paper is the it is not easy to read and follow, especially for people outside the MI field. For once, there is a heavy use of jargon and a liberal use of terms. I'll give some examples below, but I encourage the authors to give a thorough pass of the work and try to \"cleanse it\" to make it more accessible to a wider audience:\n  - `circuit`: although the term is present in large part of the MI literature, I think a refresher and a definition would help \n  - `residual stream`/ `write in the residual stream`: likewise, worthwhile defining\n  - `[end]` not defined, presume it refers to the last token in the sentence, but unclear\n  - `inhibition`: what does it mean specifically? \n  -  `noisy signal`: is there a definition for this? does this mean that the head(s) add a random vectors to the residual stream? \n  - Labels and titles of Fig 4 are difficult to decipher: what does `Dot w/token embed` mean?\n- Some more formalism and equations could help (at least some reader). \n     - E.g. one could probably define the  `Dot w/token embed` ..\n     - The path patching, even if not a contribution of this work, could be explained more clearly (with a few equations). \n- Evidence of the findings (and the storytelling) should find more space in the paper.\n\nMinor \n- Suggest to use different typeset for named heads \n- Typo: to make to make (pag 7)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)