Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
Authors: Kevin Ro Wang
Alexandre Variengien
Arthur Conmy
Buck Shlegeris
Jacob Steinhardt
Venue: ICLR 2023 poster (2023)
Keywords: Mechanistic Interpretability
Transformers
Language Models
Interpretability
Transparency
Science of ML

Abstract: Research in mechanistic interpretability seeks to explain behaviors of ML models in terms of their internal components. However, most previous work either focuses on simple behaviors in small models, or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task that requires logical reasoning: indirect object identification (IOI). Our explanation encompasses 28 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches including causal interventions and projections.
To our knowledge, this investigation is the largest end-to-end attempt at reverse-engineering a natural behavior "in the wild" in a language model.  We evaluate the reliability of our explanation using three quantitative criteria - faithfulness, completeness and minimality. Though these criteria support our explanation, they also point to remaining gaps in our understanding. 
Our work provides evidence that a mechanistic understanding of large ML models is feasible, opening opportunities to scale our understanding to both larger models and more complex tasks.

Published as a conference paper at ICLR 2023
INTERPRETABILITY IN THE WILD: A CIRCUIT FOR INDIRECT OBJECT IDENTIFICATION IN GPT-2 SMALL
Kevin Wang', Alexandre Variengien", Arthur Conmy", Buck Shlegerisl, Jacob Steinhardt 118
Reewooo Research UC Berkeley
ABSTRACT
Research in mechanistic interpretability seeks to explain behaviors of ML mod- els in terms of their internal components. However, most previous work either focuses on simple behaviors small models, or describes complicated behaviors in larger models with broad strokes. In this work, we bridge this gap by pre- senting an explanation for how GPT-2 small performs a natural language task that requires logical reasoning: indirect object identification (IOI). Our explanation compasses 28 attention heads grouped into 7 main classes, which we discovered using combination of interpretability approaches including causal interventions and projections. To our knowledge, this investigation is the largest end-to-end at- tempt at reverse-engineeriing a natural behavior "in the wild" in a language model. We evaluate the reliability of our explanation using three quantitative criteria- faithfulness, completeness and minimality. Though these criteria support our ex- planation, they also point to remaining gaps in our understanding. Our work is a case study demonstrating first step toward a better understanding of pre-trained language models, opening opportunities to scale to both larger models and more complex tasks
INTRODUCTION
Transformer-basee language models (Vaswani al.] 2017 Brown al. 2020) have demonstrated an impressive suite of capabilities, but largely remain black boxes. Understanding these models difficult because they employ complex non-lineas interactions densely-connede operate in nigh-dimensional space. Despite this, they are already deployed in high-impac set- tings, underscoring the urgency of understanding and anticipating possible model behaviors. Some researchers have even argued that interpretability is necessary for the safe deployment of advanced machine learning systems (Hendrycks & Mazeika (002)).
layers and
Work in mechanistic interpretability aims to discover, understand and verify the algorithms that model weights implement by reverse engineering model computation into human aundaaddddrrtt components (Olah 2022, Meng et 2022 Geiger et al. [021] Geva et 20200. By understanding underlying mechanisms, we can better predict -fitto-distributtin behavior (Mu Andreas) 2020), identify and fix model errors (Hernandez al., 2021 Vig et al.) 2020), and understand emergent behavior (Nanda & Lieberum 2022 Barak et al. 2022 Wei et al.
2022).
In this work, we aim to understand how GPT-2 small (Rafford al., 2019) implements natural language task. To do so, we locate components of the network that produce specific behaviors, and study how they compose complete the task. We do so by using circuits analysis (Rauker eet al. 2022). identifying an induced subgraph of the model's computational graph that is human- understandable and responsible for completing the task. We employed a number of techniques, most notably activation patching, knockouts, and projections, which we believe are useful, general techniques for circuit discovery
"Wor done while Redwood Research. *Correspondence to jstteinhaardtdbeekeeee..ee
A full and up-to-date version of this work can be found at We included an overview of the techniques used in Appendix
10553

---

Published as a conference paper at ICLR 2023
Sager
10
Sourne sequeeoce
Target sequence
10 Mary
When
When
And--
Alle
Dann
Patching
and
John
John
the
GPP2--mail
Predictionn
When
52 Donn
g3ve
Them
Knock Out
Mary
and
the 4.4i)
John
d11nk
Jonn
END
her
Figure 1: Left: We isolated circuit (in orange) responsible for the flow of information connecting the indirect object 'Mary" to the next token prediction. The nodes are attention blocks and the edges represent the interactions between attention heads. Right: We discovered and validated this circuit using activation experiments, including both patches and knockouts of attention heads.
We focus understanding non-trivial, algorithmic natural language task that call Indirect Object Identification (IOI). In IOI, sentences such as 'When Mary and John went to the store, John gave drink to' should be completed with 'Mary". We chose this task because it is linguistically meaningful and admits a complex but interpretable algorithm (Section 3
We discover circuit of 28 attention heads- of the total number of (head, token position) pairs that completes this task. The circuit uses 7 different categories of heads (see Figure 2 to implement the algorithm. Together, these heads route information between different name tokens, the end position, and finally to the output. Our work provides, to the best of our knowledge, the most detailed attempt at wewreeeeeniiiniingg.... natural end-to-end behavior in a transformer- based language model.
Explanations for model behavior can easily be misleading or non-rigorous (Jain & Wallace 2019 Bolukbasi et al. 2021). To remedy this problem, we formulate three criteria to help validate our circuit explanations. These criteria are faithfulness (the circuit can perform the task as well as the whole model), completeness (the eircuit contains all the nodes used to perform the task), and minimality (the circuit doesn't contain nodes irrelevant to the task). Our circuit shows significant improvements compared to a naive (but faithful) circuit, but fails to pass the most challenging tests.
In summary, our main contributions are: (1) We identify large circuit in GPT-2 small that per- forms indirect-object identification on a specific distribution (Figure 2 and Section 3): (2) Through example, we identify useful techniques for understanding models, as well as surprising pitfalls; (3) We present criteria that ensure structural correspondence (in the computational graph abstraction) between the circuit and the model, and check experimentally whether our circuit meets this standard (Section
2 BACKGROUND
In this section, we introduce the 10I task (an original contribution of this work), the transformer architecture, define circuits more formally and describe a technique for "knocking out" model nodes.
Task description. In indirect object identification (IOI), two names (the indirect object (IO) and the first occurrence of the subject (S1)) are introduced in an initial dependent clause (see Figure []] A main clause then introduces the second occurrence of the subject (S2), who is usually exchanging an item. The task to complete the main clause, which always ends with the token 'to'. with the non-repeated name (IO). We create many dataset samples for IOI (pror) using 15 templates (see Appendix with random single-token names, places and items.
We investigate the performance of GPT-2 small on this task. We study the original model from Radford et al. (2019), pretrained on a large corpus of internet text and without any fine-tuning. To quantify GPT-2 small performance values placed on the two names, where a positive score means the correct name (IO) has higher probability. This is also the difference in loss the model would receive in training if I0 was correct compared to if S was correct. We report this metric averaged over Pror throughout the paper. GPT-2 small has mean logit difference of 3.55 averaged across 100,000 dataset examples.
the 10I task, we used the logit difference between the logit
2

---

Published as a conference paper at ICLR 2023
Transformer architecture. GPT-2 small is decoder-only transformer with 12 layers and 12 atten- tion heads per attention layer. In this work, we mostly focus on understanding the mechanisms of attention heads, which we describe using notation similar to Elhage et al. (2021). We leave full description of the model to Appendix E.
The input to the transformer is the sum of position and token embeddings, 3o E Nxd where N is the number of tokens in the input and d is the model dimension. This input embedding is the initial value of the residual stream, which all attention layers and MLPs read from and write to. Attention layer of the network takes as input I; E R xd, the value of the residual stream before it. The attention layer output can be decomposed into the sum of attention heads ha.z. If the output of the attention layer is y Σn ha.5(zs), then the residual stream is updated to x, + Yt-
Focusing on individual heads, each head has is parametrized by four matrices W W W e Rd 4 and w R# xd, We rewrite these parameters as low-rank matrices in 1ddd Wov WO w and W QK = (Wa The QK matrix is used to compute the attention pattern A.j RVNN of head (i. while the OV matrix determines what is written into the residual stream. At the end of the forward pass, a layer norm is applied before the unembed matrix Wu projects the residual stream into logits.
2.1 CIRCUITS
In mechanistic interpretabilityy we want to reverse-engineer models into interpretable algorithms. A useful abstraction for this goal are circuits. If we think of model a computational graph M where nodes are terms in its forward pass (neurons, attention heads, embeddings, etc.) and edges are the interactions between those terms (residual connections, attention, projections, etc.), a circuit C is a subgraph of M responsible for some behavior (such as completing the IOI task). This definition of circuit is slightly different from that in Olah et al. (2020), where nodes are features (meaningful directions in the latent space of a model) instead of model components.
2.2 KNOCKOUTS
Just as the entire model M defines a function Mxx) from inputs logits, we also associate each circuit with function C(x), via knockouts. A knockout removes set of nodes K in computational graph M with the goal of "turning off" nodes in K but capturing all other computations in M. Thus, C(r) is defined by knocking out all nodes in M'C and taking the resulting logit outputs in the modified computational graph.
A first naïve knockout approach consists of simply deleting each node in K from M. The net effect of this removal is to zero ablate K, meaning that we turn its output to O. This naive approach has an important limitation: 0 is an arbitrary value, and subsequent nodes might rely on the average activation value as an implicit bias term. Because of this, we find zero ablation to lead to noisy results in practice.
To address this, we instead knockout nodes through mean ablation: replacing them with their aver- age activation value across some reference distribution (similar to the bias correction method used Nanda & Lieberuml (202)). Mcan-ablations will remove the influence of components sensitive to the variation in the reference distribution (i.e. attention heads that move names in P1o1), but will not influence components using information constant in the distribution (i.e. attention patterns that are constant in P1o1). Through mean-abl mean-ablations, we are interested in finding the components that move information about names, which is the core of the 1OI task and also varies with the distribution.
In this work, all knockouts are performed in a modified Pror distribution with three random names, so the sentences no longer have a single plausible IO. We mean-ablate on this distribution, which we call the 'ABC' distribution, because mean-ablating on the Prot distribution would not remove enough information, like information constant in Pror that is helpful for the task. To knockout single node, a (head, token position) pair in our circuit, we compute the mean of that node across samples of the same template. Computing means across the entire distribution instead of templates would average activations at different tokens, like names, verbs and conjunctions, mixing informa- tion destructively.
3

---

Published as a conference paper at ICLR 2023
Legend
Whenn Mary
Key/
and
Dohn
Query
Class Heads LayerHooe
Output
S1m1 went
Previous
22
the
store
Nogative Namee 107 Till
Duplicate Token Heads 01 50 (000)
John
69 138 (99)
gave
Narre Mover Heeds 59 96 10.0
drink
END
S-Inhibition Heatss
Beckup Name Mover Heads 30 57 101
23 719 86 I010
102 106 1010 112
We discover a circuit in GPT-2 small that implements IOL. The input tokens on the left are passed into the residual stream. Attention heads move information between residual streams: the query and output arrows show which residual streams they write to, and the key/value arrows show which residual streams they read from.
Figure 2:
DISCOVERING THE CIRCUIT
We seek to explain how GPT-2 small implements the IOI task (Section 2p. Recall the example sentence "When Mary and John went to the store, John gave drink to". We discovered that GPT- 2's internal mechanisms implement the following human-interpretable algorithm to perform IOI:
1. Identify all previous names in the sentence (Mary, John, John).
2. Remove all names that are duplicates (in the example above: John).
3. Output the remaining name.
Our circuit contains three major classes heads, corresponding to these three steps:
Duplicate Token Heads identify tokens that have already appeared in the sentence. They are active at the S2 token, attend primarily to the SI token and write a "signal' into the residual stream that token duplication has occurred.
S-Inhibition Heads perform step 2 of the inter-interpeetbbll algorithm. They are active at the END token, attend to the S2 token and write to bias the query of the Name Mover Heads against both S1 and S2 tokens.
Name Mover Heads, by default, attend previous names in the sentence, but due to the S-Inhibition Heads attend less to the SI and S2 tokens. Their OV matrix is a name copying matrix, in Pior. they increase the logit of the IO token.
A fourth major family of heads writes in the opposite direction of the Name Mover Heads, thus decreasing the confidence of the predictions. We speculate that these Negative Name Mover Heads might help the model "hedge" so as to avoid high cross coos-ey loss when making mistakes.
There are also three minor classes of heads that perform related functions to the components above:
Previous Token Heads copy the embedding of S to position S+1.
Induction Heads perform the same role as the Duplicate Token Heads through an induction
mechanism. They are active at position S2, attend to token S+1 (mediated by the Previous
Token Heads), and output a signal that the S token previously appeared the context. Finally, Buckup Name Mover Heads do not normally move the 10 token to the output, but take on this role the regular Name Mover Heads are knocked out.
Note that circuit does not include the MLPs. We are interested in the flow of information across tokens, and MLPs only process features along tokens. Moreover, initial investigations suggest all MLPs except for the first are not crucial for this task (Appendix [. though more precise investigation is left for future work.
Below, show step-by step-by-step how discovered each component, providing evidence that they behave as deseribed above. We found that it was most natural to uncover the circuit starting at the logits and working back. Thus start with the Name Mover and Negative Name Mover Heads.
4

---

Published as a conference paper at ICLR 2023
A
B
C
D
home arr becdingg veid stamion procndi
Diip SEOT
Name movers
Unembeddinng projection
Aleadk
Projectes of carpul 11 allingg
Held 11.10 bae soae
Hogplti mor
Neguliv
Head
bands
Figure 3: A: Name Movers and Negative Name Movers Heads are the heads that most strongly write in the Wy [0] - W direction. B: Attention probability vs projection of the head output along Wo [1O] or [S] respectively. Note that for S tokens, we sum the attention probability on both S1 and S2. C: Value-weighted attention score with the query at the end token. D, top: Positive copying score for the Name Mover Heads. D, bottom: Negative copying score for the Negative Name Mover Heads. Dashed lines are the average scores for all heads.
3.1 WHICH HEADS DIRECTLY WRITE TO THE OUTPUT? (NAME MOVER HEADS)
We begin by identifying which attention heads directly affect the model's output: in other words, the heads writing in the residual stream at the END position, in a direction that has high dot product with the logit difference. Formally, let Wy denote the unembedding matrix, LN layer norm operation (see Appendix H and Wy [TO], [S] the corresponding unembedding vectors for the TO and S tokens. We searched for heads (i. j) such that
Ai.j E Ex~pIol [LN o (x,, WollO - Wu [S]]
had large magnitude. Recall that (X) is the value that head (i. j) writes into the residual stream on input X. Therefore, heads with Xr,j > 0 correctly promote the IO token over the S token (on average). The unembedding projection in (3.1) is called the logit lens and has been used in previous work to interpret intermediate activations (nostalgebraist, 2020) and parameters (Dar et al., 2022). We display the values of >i in Figure 3 A. We that only few heads in the final layers have large logit projection λıς- Specifically, 9.6, 9.9, and 10.0 have a large positive score, while 10.7 and 11.10 have a large negative score.
Name Mover Heads. To understand the positive heads, we first study their attention patterns. We find that they attend strongly to the 10 token: the average attention probability of all heads over pro is 0.59. Since attention patterns can be misleading (Jain & Wallace 2019). we check whether atten- tion is correlated with the heads' functionality. We do by scatter plotting the attention probability against the logit score (h₂(X). Wy [O]). The results are shown in Figure 3 B: higher attention probability on the IO token linearly correlated with higher output in the IO direction (correlation > 0.81, N = 500). Based on this result, we hypothesize that these heads (i) attend to names and (ii) copy whatever they attend to. We therefore call these heads Name Mover Heads.
To check that the Name Mover Heads copy names generally, we studied what values are written via the heads" OV circuits. We transform the output of the first layer name token through the OV matrix of Name Mover Head and then project to the logits. The copy score is the proportion of samples that contain the input name token in the top logits (N = 1000). We find that all three Name Mover Heads have copy score above 95% (compared to less than 20% for average head).
Negative Name Mover Heads. In Figure 3 we also observed two heads strongly writing opposite the Wu[rO] Wo [S] direction. We called these heads Negative Name Mover Heads. Their copy score is calculated with the negative of their OV matrix. As described in Figure 3 they share all the properties of Name Mover Heads, except they write in the opposite of names they attend to.
5

---

Published as a conference paper at ICLR 2023
S2
END
IO1
When Mary and John went to the store
John
drink
ABC (52)
ABC
Parching AIC = I31 5+1
Paitching ABC (O) S2
Parching ABCC - IDI END
Voliaboo hoam en Of
utando srat,
Novis
Lagst
Figure 4: The attention probability to IO averaged over three Name Mover Heads is decreased most by the Previous Token Heads (left), Induction Heads (center) and S-Inhibition Heads (right) when we patch these attention heads from a sentence with different S2 name (center and right), or different SI name (left).
3.2 WHICH HEADS AFFECT THE NAME MOVER HEADS' ATTENTION? (S (S-INHIBITION HEADS)
Given that the Name Mover Heads are primarily responsible for constructing the output, we ask why these Name Mover Heads pay preferential attention to the IO token. First, there are two ways affect the Name Mover Heads's attention: through the query vector at the END token or the key vector at the I0 token. Since the key vector appears early in the context, it likely does not contain much task spec-fc information, so we focus on the END query vector.
Then, by investigating Name Mover Heads on the ABC distribution (where the three names are distinct; see Section 2.2p. we observed that their attention is not selective: they pay equal attention to the first two names. We thus ask: what has changed from the ABC distribution to the Pror distribution to cause the Name Mover Heads to attend to the IO token preferentially?
To empirically answer this question, we perform putching experiment, a similar type of causal intervention as performed Meng et al.] (2022); ): Vig et al. (2020).
illustrated Figure [ this technique consists of two steps. First save all activations of the network run source sequence. Then we run the network on a target sequence, replacing some activations with the activations from the source sequence. We can then measure the behavior of the patched model. Doing this for each node individually locates the nodes that explain why model behavior is different in the source and target sequences.
In our case, we run activation patching with source sentences from the ABC distribution and target sentences from Prot- We then compute the change in attention probability from END to IO, averaged over the three Name Mover Heads. Since the Name Mover Heads attention on the I0 is high in the PIOI distribution and low in ABC, patching at important heads from ABC to Pior should decrease Name Mover Heads attention on IO. The results from patching every head at the END token position are shown in Figure right. We observe that patching heads 7.3, 7.9, 8.6, 8.10 causes decrease in the attention probability on 10, indicating that they are counterfactually important for the Name Mover Heads's attention probability on the 10 token. We call these heads S-Inhibition Heads.
3.3 WHAT INFORMATION DO THE S-INHIBITION HEADS MOVE?
How do the S-Inhibition Heads differentiate between I0 and S, so they inhibit one but not the other? We measured their attention pattern and found that they preferentially attend the $2 token. We therefore studied what information these heads move from the S2 token position the END position. We studied both the properties of the input, and which upstream affect the S-inhibition heads. Surprisingly, found that the S-inhibition heads mostly depend the repetition at the two positions where the S token occurs (Appendix G
To study the heads that affect the S-inhibition heads, we patching experiment at S2 from the ABC distribution to the IOI distribution and measured the variation in Name Mover Heads attention. The results (Figure 4 center) reveal large set of heads influencing Name Mover Heads' attention

---

Published as a conference paper at ICLR 2023
that did not appear the END position. S-Inhibition Heads must mediate this effect, they are the only heads influencing Name Mover Heads at the END position. This reasoning suggests that the outputs of this set of heads is moved by S-Inhibition Heads from S2 to the END token. When analyze the attention patterns of these heads, we see two distinct groups emerge.
Duplicate Token Heads. One group attends from S2 to S1. We call these Duplicate Token Heads on the hypothesis that they detect duplicate tokens. To validate this, we analyze their attention pattern on sequences of random tokens (with no semantic meaning), we found that of the 3 Duplicate Token Heads pay strong attention to a previous occurrence of the current token it exists (see Appendix for more details).
Induction Heads and Previous Token Heads. The other group of heads attends from S2 to S1+1 (the token after the S1 token): the classic attention pattern of an induction head. Previously described in Elhage et al. (202T), induction heads recognize the general pattern [A] [B]
[A] and con- tribute to predicting [B] as the next token. For this, they act pair with a Previous Token Head. The Previous Token Head should write information about [A] into the residual stream at [B), so that the Induction Head can match the next occurrence of [A] to that position (and subsequently copy [B] to the output).
We therefore seek to identify Previous Token Heads used by our purported Induction Heads. To this end, we patched activations from sentence where SI is replaced by random name, at the S+1 token index. As shown in figure some heads (and particularly 4. 4.11) appear to influence Name Mover Heads. Then, by looking the attention pattern of the most important heads in this patching experiment, we identified Previous Token Heads. We find that 2 of the Previous Token Heads and 2 of the 4 Induction Heads demonstrated their expected attention patterns (Appendix F
3.4 DID WE MISS ANYTHING? THE STORY OF THE BACKUP NAME MOVERS HEADS
Each type of head in our circuit has many copies, suggesting that the model implements redundant behavior. To make sure that we didn't It miss any copies, we knocked out all of the Name Mover Heads at once. To our surprise, the circuit still worked (only 10% drop in logit difference). In addition, many heads write along Wy [IO] [S]S] after the knockout, which did not do so previously.
We kept the heads with the largest Av.j and call them Backup Name Mover Heads. See appendix B for further details these heads. Among the height heads identified, we investigated their behavior before the knockout. We observe diverse behavior: 3 heads show close resemblance to Name Mover Heads; heads equally attend 10 and S and copy them; 1 head pays more attention to S1 and copies it; head seems to track and copy subjects of clauses, copying S2 in this case.
4 EXPERIMENTAL VALIDATION
In this section, we check that our circuit provides good account of GPT-2's true behavior. In gen- eral, our introduced criteria depend on measure F of the performance of a circuit on a task. In our case, suppose X PIo1. and (CC()); X) is the logit difference between the I0 and tokens when the circuit C' run on the input X. The average logit difference F(C) del Ex-pior [S((();; is a measure of how much a circuit predicts 10 rather than S, i.e performs the 101 task.
Firstly, we check that C is faithful to M, i.e. that it computes similar outputs. We do so by measuring [F(M) - F. and find that it is small: 0.2, or only 6% of F(M) = 3.55.
In Section 4.1 we define a running toy example of model M for which faithfulness is not sufficient to prescribe which circuits explain a behavior defined by a measure F' well. This motivates the criteria of completeness and minimality that we then check on our eircuit. In addition to the criteria, we also validated our knowledge of the circuit by designing adversarial examples (see Appendix C
4.1 COMPLETENESS
As running example, suppose model M uses two similar and disjoint serial circuits (where each node depends on the previous node) C and Cz. The two sub-circuits are run in parallel before applying an OR operation to their results. Identifying only one of the circuits is enough achieve
7

---

Published as a conference paper at ICLR 2023
circut completeness
Natve dircut completenes tests
Random
Nurio
Inductr
Deplicao Maa
Powiiur toom
heads
Figure 5: Plot of points (IX, uk) (F(M K), F(C \ K)) for our circuit (left) and naive circuit (right). Each point is for a different choice of K: 50 uniformly randomly chosen K C, K = 0, and the five K with the highest incompleteness score found by greedy optimization. Since the incompleteness score [ yK we show the line x for reference.
faithfulness, but want explanations that include both C1 and C2. since these are both used in the model.
To solve this problem, introduce the completeness criterion: for every subset K S C, the incompleteness score [F K) - F(M \ K)] should be small. In other words, C and M should not just be similar, but remain similar under knockouts
In our running example, we can show that C1 is not complete by setting K = C1. Then Cj \ K is the empty circuit while M K still contains C2. The metrie [F(C, \ K)-F((A) F(M K)] will be large because C K has trivial performance while M K successfully performs the task.
The criterion of completeness requires search over exponentially many subsets K C C. This is computationally intractable given the size of our circuit, hence we use three sampling methods to find examples of K that give large incompleteness score:
The first sampling method chooses subsets K C uniformly at random.
The second sampling method set K be an entire class of circuit heads G. e.g the Name Mover Heads. C C G should have low performance since it's missing a key component, whereas M G might still do well if it has redundant components that fill in for G.
Thirdly, we greedily optimized K node-by-node to maximize the incompleteness score (see appendix for the detail of the optimization procedure).
These first two methods of sampling K suggested to us that our circuit was e-complete for a small value of e. However, the third resulted sets K that had high incompleteness score: up to 3.09. All such results are found in figure 5 on the left.
4.2 MINIMALITY
A faithful and complete circuit may contain unnecessary components, and so be overly complex. To avoid this, we should check that each of its nodes is necessary. This can be evaluated by knocking out a set of nodes K and showing that adding back v E K to the circuit can significantly recover F.
Formally, the minimality require that for every node e E there exists subset C C (v) that has minimality score IF(C' \ (KU)))) - F(C K) 2 A. We call such circuit A minimal..
In the rumning example, C₁ UC2 is A-minimal for some non-trivial A. We can sketch a proof of this result given an informal defintion of 'non-trivial". To show this, note that if E C1 and K C2. then the minimality score is equal to [F(C1 \ (11 F(C1) which is large since C is serial circuit and removing U1 will destroy the behavior. We then proceed symmetrically for U2 C2.
In practice, we need to exhibit for every v set K such that the minimality score at least A. For most heads, removing the class of heads G that v is a part of provides a reasonable minimality score. We describe the sets K that are required for them in Appendix J The importance of individual nodes is highly variable, but they all have significant impact the final metric (at least 3% of the original logit difference). These results ensure that we did not interpret irrelevant nodes, but do show that the individual contribution of some single attention heads is small.
8

---

Published as a conference paper at ICLR 2023
2.5
Change logit difference
negatie
1.5
Induccionn
duplicate
previbus toiin
0.5
00
(O0 TT
80
88
99
00
00
99
C o0
on
(ot ot)
(1 (2:0)
66
6
L
(or
10
12
16
[2
16
Atlention head
Figure 6: Plot of minimality scores |F(C \ ( F(C K) for all components in our circuit. The sets K used for each component, as well as the initial and final values of the logit difference for each of these is in Appendix Our circuit is
4.3 COMPARISON WITH A NAIVE CIRCUIT
In order to get relative sense of the success of our explanation by our criteria, we compare the results on naïve circuit that consists of the Name Mover Heads (but no Backup Name Mover Heads), S-Inhibition Heads, two Induction Heads, two Duplicate Token Heads and two Previous Token Heads. This circuit has a faithfulness score 0.1, a score comparable to our circuit's faithful- ness score. However, contrary to circuit, the naive circuit can be easily proven incomplete: by sampling random sets or by knocking-our by classes, we see that F(M K) is much higher than F(C K) (Figure left). Nonetheless. when applied the greedy heuristic optimize for the incompleteness score, both circuits have similarly large incompleteness scores. Thus, we conclude that worst-case completeness criterion too high bar, which future work could use high standard to validate circuit discovery.
5 DISCUSSION
In this work, we isolated, understood and validated set of attention heads in GPT-2 small composed in a circuit that identifies indirect objects. Along the way, we discovered interesting structures emerging from the model internals that complicated the study. For instance, we identified heads compensating for the loss of function of other heads, and heads contributing negatively to the next- token prediction. Early results suggest that the latter phenomenon occurs for other tasks beyond IOI (see Appendix
However, our work also has several limitations. First, despite the detailed analysis presented here, do not understand several components. Those include the attention patterns of the S-Inhibition Heads, and the effect of MLPs and layer norms. Second, the number parameters in GPT-2 small orders of magnitude away from state-of-the-art transformer language models. future challenge to scale this approach to these larger models. Thirdly, we only looked at the difference average metric (logit difference) between the circuit and the model order compare how they both did the IOI task (Section 4). Looking at the average difference in metric between the circuit and model on individual examples would be a more stringent way to compare them, but had too much variability to help find a circuit. Fourthly, the definition of the task is limited: we only measure fraction of the prediction made by the model, and do not study cases where the model not performing IOI. Finally, more work is needed to validate the structural validation criterion we introduce here.
We hope that our work spurs further efforts in mechanistic explanations of larger language models computing different natural language tasks, with the eventual goal of understanding full language model capabilities.
REFERENCES
Boaz Barak, Benjamin L Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. arXiv preprint arXiv:2207.088: 08799, 2022.

---

Published as a conference paper at ICLR 2023
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda B. Viégas, and Martin Wattenberg. An interpretability illusion for BERT. CoRR, abs/2104.07143, 2021. URL
attts:///arix/: org/ abs 2104. 07143
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https:: 1457c0d6bf cb496 74 wwwpaacc14264400000
nww.iieersp..... wwtpaper/222//Elee......ee pdf
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. arXiv preprint arXiv:2209.02535. 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Haffield-Dodss, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https:/transformerccccciii httpr//rrciiiisuu/////////a/aa frameewrr/index.htm/
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms neural language models, 2021.
URL https: /ttts::/arxxi... arxiv.iiivv../.... org/ abs/2106 06087
Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neurul Information Processing Systems, 2021. URL
1d=RmuXDt 1DhG
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. arXiv preprint 1rX1v:2012.14913, 2020.
Dan Hendrycks and Mantas Mazeika. X-risk analysis for ai research. arXiv, abs/2206.05862, 2022.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference Learning Representations, 2021.
Sarthak Jain and Byron Wallace. Attention is not Explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 3543-3556, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10. 1/665//v1NN191157.7 URL hwl.leppp///aaanttooooooooo/////
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. arXiv preprint 2202.000000...22.. 2022.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Informa- tion Processing Systems, 33:17153-17163, 2020.
Neel Nanda and Tom Lieberum. mechanistic interpretability analysis of grokking, 2022. URL http:p/pss../.a.a...oootooo.ooop cxeeaaassccccceppppeoollyy/yaaad
nostalgebraist. interpreting gpt: the logit len, 2020. URL https::/t///t wwwp http://wwwwwwswwonneeeom com/
post hosts///cckabssipccaaoorrrrreeep
-ppttngggggppppppp the 1oo--ooiitt lens
Chris Olah. Mechanistic interpretabilitty, variables, and the importance of interpretable bases.
2022.
10

---

Published as a conference paper at ICLR 2023
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 1022911/dissill.00244000.... https///tiillllb/222///eettiiioo
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint 2rXiv:2209.1189) 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. Toward transparent ai:
A survey on interpreting the inner structures of deep neural networks, 2022. URL https:
//axxix.orr/abs/220000 13243
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6098 1998,6008,00 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33: 1338-- 2020.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. ArXiv, abs/2206.00 07682, 2022.

---

Published as a conference paper at ICLR 2023
A IOI TEMPLATES
We list all the template we used in Table 7 names, while the place and the object were chosen among a hand made list of 20 common names. All the word chosen were one token long to ensure proper sequence alignment computation of the mean activations.
Each name was drawn from list of 100 English first
Templates in PIO1
Then, [B] and [A] went to the [PLACE]. [B] gave a [OBJECT] to [A]
Then, [B] and [A] had a lot of fun at the [PLACE] [B] gave a [OBJECT] to [A]
Then, [B] and [A] were working at the [PLACE]. [B decided give [OBJECT] to [A]
Then, [B] and [A] were thinking about going to the [PLACE]. [B] wanted to give [OBJECT] to [A]
Then, [B] and [A] had a long argument, and afterwards [B] said to [A]
After [B] and [A] went to the [PLACE), [B] gave a [OBJECT] to [A]
When [B] and [A] got [OBJECT] at the [PLACE], [B] decided to give it to [A]
When [B] and [A] got a [OBJECT] at the [PLACE], [B] decided to give the [OBJECT] to [A]
While [B] and [A] were working at the [PLACEE] [B] gave a [OBJECT] [A]
While [B] and [A] were commuting to the [PLACE], [B] gave a [OBJECT] to [A]
After the lunch, [B] and [A] went to the [PLACE]. [B] gave a [OBJECT] to [A]
Afterwards, [B] and [A] went to the [PLACE]. [B] gave a [BJJCCT] to [A]
Then, [B] and [A] had long argument. Afterwards [B] said to [A]
The [PLACE] [B] and [A] went to had a [OBJECT]. [B] gave it to [A]
Friends [B] and [A] found a [OBJECT] the [PLACE]. [B] gave it to [A]
Figure 7: Templates used in the IOI dataset. All templates in the table fit the 'BABA' pattern, but also use templates that fit the 'ABBA' pattern as well (not included for simplicity).
B BACKUP NAME MOVER HEADS
Here we discuss in more detail the discovery of the Backup Name Mover Heads. As shown in figure g knocking-out the three main Name Mover Heads doesn't 't leave the rest of the heads in similar state before. They seem to "compensate" the loss of function from the Name Mover Heads such that the logit difference is only 10% lower. We observe that the Negative Name Mover Heads head write less negatively in the direction of [v[[OO] We [S), 10.7 even write positively in this direction afterwards, while other heads that wrote slightly along Wo: knock-out becomes the main contributor. Both the reason and the mechanism of this compensation effect are still unclear, we think that this could be an interesting phenomenon to investigate in future works. Among those last categories, we identify S-inhibition heads and a set of other head that we called Backup Name Mover Heads. We arbitrarily chose to keep the height heads that were not part of any other groups, and wrote in the direction of Wo [IO] Wy [S] above the threshold of 0.05.
(r[S] before the
In wie analyze the behavior of those newly identified heads with similar techniques as Name Mover Heads. Those can be grouped in 4 categories.
3 heads (10.1, 10.10 and 10.6) that behave similarly as Name Mover Heads according to their attention pattern, and scatter plots of attention vs dot product of their output with [TO Wy(S] (as 10.10).
3 heads (10.2, 11.9, 11.3) that pay equal attention to SI and 10 and wrote both of them (as 10.2 in Figure 9).
* One head, 11.2, that pays more attention to S1 and write preferentially in the direction of Wu[S]
One head, 9.7, that pays attention to S2 and write negatively.
We did not thoroughly investigate this diversity of behavior, more work can be done to precisely describe these heads. However, these heads are also the ones with the less individual importance for the task (as shown by their minimality score in Figure The exact choice of Backup Name Mover Heads doesn't 't change significantly the behavior of the circuit.
12

---

Published as a conference paper at ICLR 2023
Unembeddtnn projection of head output (before movers KO)
Unembedding projection of head output (atter
Pnerooooo 10-6
regali
tacking prore rowr
22 mitioo
howe
60) 10
12:00
18
n0
de 10
Ioe t0
T0
L (0750) m
10
in 10
Head
Figure 8: Discovery of the Backup Name Mover Heads, After knock-out of the Name Mover Heads (right) some heads write stronger in the Wo [TO] (u[S] direction than before (left). We also observed that negative heads seems inhibited by this operation.
Backup name mover head
Projection of the output along the name embedding vs attention probability
Value weighted attention Mays
10.10
Scon
11.2
9.7
Wheel
10.2
Figure 9: Four examples of Backup Name Mover Heads. Left: attention probability vs projection of the head output along Wu or Wy (S) respectively. Right: Attention pattern on sample sequence.
13

---

Published as a conference paper at ICLR 2023
Proportion of S logit greater than IO 0.7%
Distribution
Logit difference
I0 probability
PIo1 Additional occurrence of S
3.55
0.49
3.64
0.59
0.4%
(natural sentence)
Additional occurrence IO (natural sentence)
1.23
0.36
23.4%
Summary of GPT-2 performance metrics the IOI task different datasets. In the line order: for pror, for dataset where we added an occurrence of (thus S appears three times in the sentence) and for the adversaris dataset with duplicated IO in natural sentences. 10 probability refers to the probability the model places on the IO token (computed from the logits).
Figure 10:
C DESIGNING ADVERSARIAL EXAMPLES
As argued in Räuker al. (022), way evaluate the knowledge gained by interpretability work is to use it for downstream applications as predicting out of distribution behavior. In this section, we do this by using knowledge of the circuit to construct simple adversarial examples for the model.
As presented in Section 3 the model relies on duplicate detection to differentiate between S and I0. Motivated by this, we constructed passages where both the and IO tokens are duplicated. An example is "John and Mary went to the store. Mary had good day. John gave bottle of milk to"; see Appendix D for full details. We find that this significantly reduces the logit difference and causes the model to predict S over 10 23% of the time (Figure
To ensure that the observed effect is not an artifact of the additional sentences, we included a control dataset using the same templates, but where the middle sentence contains S instead of IO. In these sentences, appears three times total and only appears once. On this distribution, the model has an even higher logit difference than Prot, and predicts S over 10 only 0.4% of the time.
I Limitations the attack. Despite being inspired by our understanding of our circuit, those exam- ples simple enough that they could have been found without circuit with enough effort.
Moreover, we do not have a full understanding of the mechanisms at play in these adversarial exam- ples. For instance, the S-Iniibition Heads attend not only to S2, but also to the second occurrence of 10. As this pattern is not present in PIOı nor in ABC, it is beyond the analysis presented Section 3 The study of the behavior of the cireuit on these adversarial examples could be promising area for future work.
D TEMPLATE FOR ADVERSARIAL EXAMPLES
The design of adversarial examples relies on adding duplicate IO to the sentences. To this end, we used a modification of the templates described in appendix [A] We added an occurrence of [A] in the form of natural sentence, independent of the context. The list of sentence visible in Figure
[A] had a good day.
[A] was enjoying the situation.
[A] was tired.
[A] enjoyed being with a friend.
[A] was an enthusiast person.
Figure 11: Templates for the natural sentences used in the generation of adversarial examples. The sentences were chosen to be independent of the context.
GPT-2 SMALL FULL ARCHITECTURE
Here define all components of the GPT-2 Architecture, including those don't use in the main text. GPT-2 small has the following hyperparameters:
N: number of input tokens.
14

---

Published as a conference paper at ICLR 2023
V: vocabulary of tokens.
d: residual stream dimension.
L: number of layers.
H: number of heads per layer.
D: hidden dimension of MLPs
uses layer norms, the non-linear function
LN) def
(I)
(2) xz)²
where the mean and the difference from the mean sum are over the d components of each of the N tensors.
In GPT-2 the MIPs all have one hidden layer of dimension D and use the GeLUU non-linearity.
We addressed the parametrisation of each attention head in the main text, and cover the technical details of the Wox and Wor matrix here: the attention pattern is Aij softmax(x"' W x) where the softmax taken for each token position, and unidirectional. We then have h4 def
(As.g @ W ).z.
Algorithm GPT-2
Require: Input tokens 7; returns logits for next token.
1: w +- One-hot embedding of T
2: Zo t WEw (sum of token and position embeddings)
3: for = 0 to L do
4:
Yi <- 0 E R
5:
for
0 to H do
6:
y1 < + h4 (x;), the contribution of attention head (i,j)
7:
end for
8:
V f (T,) the contribution of MLP at layer i
9:
Fi+1 H; + ys + so (update the residual stream)
10:
end for
11:
return Wi o LN o Dr.
F ANALYSIS ON SEQUENCES OF RANDOM TOKENS
We run GPT-2 small on sequences of 100 tokens sampled uniformly at random from GPT-2's token vocabulary. Each sequence and second half are identical. On this dataset, we computed three scores from the attention patterns of the attention heads:
duplicated to form AA, a sequence twice long where the first
The duplicate token score: for each token T; in the second half of sequence S, we average the attention probability from T, to its previous occurrence in the first half of I (i.e. T00100)
The previous token score: we averaged the attention probability on the off-diagonal. This is the attention from the token position to position
The induction score: the attention probability from T, to the token that comes after the first occurrence of T, (i.c. T9-99)
These three score are depicted in Figure 12 for all attention heads. We can identify 3.0 and 0.1 as duplicated token heads that also appear our circuit, 5.5 and 6.9 have high induction score and were also identified induction heads in our investigation and 4.1) and 2.2 have a high previous token score. Note that the heads identified are also the ones that have the highest influence in the patching experiment shown in Figure
Induction Heads. Olsson et al. (002) define an Induction Head according to its behavior on re-
peated sequences of random tokens. The attention head must demonstrate two properties. i) Prefix-
matching property. The head attends to [B] from the last [A] on pattern like [A] [B] [A] ii)
Copy property. The head contribute positively to the logit of [B] on the pattern [A] [B] ... [A].
15

---

Published as a conference paper at ICLR 2023
Duplicate token attention prob. sequences random tokens
Previous token attention prob. on sequences random tokens
Induction score on sequences random tokens
64
1kad
Had
Figure 12: Sum of attention probabilities on position determined by the role. Left: duplicate score, the average attention probability from a token to its previous occurrence. Center: Previous token attention score, the average of the off diagonal attention probability. Right: Induction score. Average attention probability from the second occurence of [A] to [B] on [A] [B] [A].
Average contribution to next token on repeated sequence
25
20
Dot prod. of head oupput and
Induction Heads from our circuit
Other induction heads
Name Mover Heads
Backup Name Mover Heads
Negative Name Mover Heads
Others
15
22
66
222
6TE
Figure 13: Contribution the next token prediction per head on repeated sequences of tokens. The heads are ordered by decreasing absolute values of contribution. Black contour: heads with attention patterns demonstrating prefix matching property.
In the 10I task, we identify these heads according to their attention pattern, demonstrating the pattern-matching property. Here, we investigate their copy property, that useless the context of IOI: outputting the token after S2 is of no interest to identify IO.
As presented above, 5.5 and 6.9 are among the 5 heads with the highest induction score. This validates their metiiinnathhig property.
To check their copy property. we computed the dot product (X, Wy [B] between the output of the head h; sequence X and the embedding of the token [B] on repeated sequences of random tokens. The results are shown in Figure [3 The two Induction Heads (5.5 and 6.9) appear in the 20 heads contributing the most to the next token prediction. Thus validating their copying property.
We also noticed that the majority of the Negative, Backup and regular Name Mover Heads appear to write in the next token direction repeated sequences of random tokens, and Negative Name Movers Heads contribute negatively. This suggests that these heads are involved beyond the IOI task to produce next-token prediction relying contextual information. Moreover, ablating the output of the three Name Mover Heads by patching their outputs results 26% increase average loss on the last 99 tokens (from 0.15 0.19), showing their importance on tasks outside IOI.
16

---

Published as a conference paper at ICLR 2023
G DISENTANGLING FEATURES IN THE OUTPUT OF S S-INHIBITION HEADS
In Section 3.2 we discovered that S-Inhibition Heads are responsible for the Name Mover Heads' specific attention on the IO token. In this appendix, we explore which properties of the input affect the S-inhibition heads' outputs.
We present evidence that they were outputting token signals (information about the value of the token S), positional signals (related to the value of the position S1) and that the latter is the most important.
To disentangle the two effects, we design series of counterfactual datasets where only some signals are present, and some are inverted with respect to the original dataset. We then conducted patching experiments where the output of S-Inhibition heads are computed from these datasets.
This enables us to quantify in isolation the impact of each signal on the final logit difference.
We constructed six datasets by combining three transformations of the original Pro1 distribution.
Random name flip: we replace the names from a given sentence with random names, but we keep the same position for all names. Moreover, each occurrence of a name in the original sentence replaced by the same random name. When we patch outputs of S-Inhibition heads from this sentence, only positional signals are present, the token signals are unrelated to the names of the original sequence.
10++S1 flip: swap the position of IO and S1. The output of S-inhibition heads will contain correct token signals (the subject of the second clause is the same) but inverted positional signals (because the position of IO and S1 are swapped)
I0+S2 replacement: we make IO become the subject of the sentence and S the indirect object. In this dataset, both token signals and positional signals are inverted.
We can also compose these transformations. For instance, we can create a dataset with no token signals and inverted positional signals by applying 10++S1 flip on the dataset with random names. In total, we can create all six combinations of original, inverted, or uncorrelated token signal with the original and inverted positional signal.
From each of those six datasets, we patched the output of S-Inhibition heads and measured the logit difference. The results are presented in Figure 14.
These results can be summarized as the sum of the two effects. Suppose we define the variable Stok be if the token signal the original, 0 when uncorrelated and -1 when inverted. And similarly Spos to be 1 if the position signal is the original and -1 if inverted. Then the Figure 14 suggests that the logit difference can be well approximated by 2.31 Spos + 0.99Stot. with mean error of 7% relative to the baseline logit difference.
For instance, when both the positional and token signals are inverted, the logit difference is the opposite of the baseline. This means that the S token is predicted stronger than the IO token, as strong as IO before patching. In this situation, due to the contradictory information contained in the output of S-hhibition heads, the Name Movers attend and copy the S1 token instead of the IO token (see Figure [5] right). In the intermediate cases where only one of the signals is modified, observe partial effect compared to the fully inverted case (e.g. Figure [5]] left). The effect size depends on the altered signals: positional signals are more important than token signals.
Can be more specific to what the token and positional signals are? Unfortunately, do not have a complete answer, but see this as one of the most interesting further directions of our work. We expect that the majority of the positional information about the relative positional embedding between SI and S2 (such pointer arithmetic behavior has already been observed in Olsson et al.
Original positional signal Inverted position signal
Original S token signal Random S token signal S++10 inverted token signal
3.55 (baseline)
-0.99
2.45
-1.96
1.77
-3.16
Figure 14:
Logit difference after patching S-Inibbition heads from signal-specific datasets. The effect on logit difference can be decomposed as sum of the effects of position and token signal.
17

---

Published as a conference paper at ICLR 2023
Average attention probabiltyy of Name Mover Heads
Average attention probability of Name Mover Heads
Attention on Paa Atlentonn after pascting S Siintibtionn Heads from tandom name lip
Attention on Pes
Atlention alter posshing S-inhbaiso biton Heads from IOli 82 replacement
Figure 15: Name Mover Heads' attention probability before and after patching S-Inhibition Heads from signal-specific datasets. Left: patching from the dataset generated by random flip of name (same position signal, random token signal). Right: patching from the dataset generated by IO+-$2 replacement (inverted position signal, inverted token signal). Black bars represent the standard deviation.
When patching in S2 Inhibition outputs from distribution where prefixes to sentences longer (but the distance between $1 and $2 constant), the logit difference doesn't 't change (3.56 before patching 3.57 after). This suggests that the positional signal doesn' 't depend on the absolute position of the tokens, as long as the relative position of SI and S2 stays the same.
2022
H LAYER NORM AND THE RESIDUAL STREAM
The attention heads and MI.Ps in GPT-2 small write into the residual stream. Suppose 312 is the final state of the residual stream after the 12 layers. This is then converted into logits via Wy M o (2 where LN is defined in Appendix E. M is the linear transformation of the layer norm operation and Wy is the unembedding matrix.
In order to attribute the extent to which an attention head h writes in direction Wo [T) where T is a token (always 10 or S in our case), can't 't simply compute (M LN h (X), Wo[T)) as the scaling factor that's used is (x12, - 212,7)² Therefore LN in the main text uses this scaling factor:
LN(h) del M
- h
(2)
Σ, (I12,1 T12.7
I ROLE OF MLPs IN THE TASK
In the main text, we focused our investigation on attention heads. Since they are the only module able of moving information across token position a crucial component the IOI task they were main subject of interest. However, MLP can still play significant role in structuring the residual stream at a given position. We explored this possibility by performing knock-out the MLP layers (Figure 16]. We observe that MLPO has significant influence on logit difference after knock-out (-100% relative variation) but the other layers don't seem to play a big role. We hypothesize that MLPO can be used perform low level token processing that latter layers rely on.
Moreover, we also investigated the writing of MLP along the [1O Wor[S] direction. As shown in Figure 16 (bottom) they write negligibly in this direction compared attention heads (Figure 3
18

---

Published as a conference paper at ICLR 2023
Class
F(C (
PCCK
(9, 9)
Name Mover
[(9, 9)]
2.78
3.14
(10, 0)
Name Mover
[99 9), (10, 0)]
2.43
2.78
(9, 6)
Name Mover
[(9, 9). (10, O). (9, 6)
2.77
2.43
(10, 7)
Negative Name Mover
All Negative Name Mover Heads
5.11
3.84
(11, 10)
Negative Name Mover
All Negative Name Mover Heads
5.11
4.06
(7, 3)
S-Inhibition
All S-Inhibition Heads
0.33
1.15
(7,9)
S-Inhibition
All S-Inhibition Heads
0.33
1.12
(8,
S-Inhibition
All S-Inhibition Heads
0.33
1.10
(8, 10)
S-Inhibition
All S-Inhibition Heads
0.33
0.55
(5, 5)
Induction
Induction Heads and Negative Heads
1.06
3.95
(5,
Induction
Induction Heads
1.06
2.58
(5, 9)
Induction
All Induction Heads
4.40
5.11
(6, 9)
Induction
Induction Heads and Negative Heads
4.76
5.11
(0, 1)
Duplicate Token
All Duplicate Token Heads
1.14
2.52
(0, 10)
Duplicate Token
All Duplicate Token Heads
1.14
2.29
(3, 0)
Duplicate Token
All Duplicate Token Heads
1.14
1.65
(2,
Previous Token
All Previous Token Heads
2.03
2.80
(2,
Previous Token
All Previous Token Heads
2.03
2.42
(4, I1)
Previous Token
All Previous Token Heads
2.03
2.27
(10, 10)
Backup Name Mover
All NMs and previous Backup NMs
2.40
2.63
(I0, 2)
Backup Name Mover
All NMs and previous Backup NMs
0.89
1.09
(11, 2)
Backup Name Mover
All NMs and previous Backup NMs
0.72
0.89
(10, 6)
Backup Name Mover
All NMs and previous Backup NMs
2.63
2.77
(10,
Backup Name Mover
All NMs and previous Backup NMs
1.34
1.47
(9, 7)
Backup Name Mover
All NMs and previous Backup NMs
0.85
1.02
(11,9) 9)
Backup Name Mover
All NMs and previous Backup NMs
1.02
1.13
(11,3)
Backup Name Mover
(9, 9), (10, 0), (9, 6), (10, 10), (11,
2.53
2.59
Figure 17: K sets for minimality for each
Logit Difference Variation after knock-out of MLPs all tokens
Legi dill relucive vin siiioon
455
Layer
Projection outputs IO-S unembedding
Nomalized
10L1
Figure 16: Top: Relative variation in logit difference from knocking out MLP layers. Only MLP0 causes significative decrease logit difference after knock-out. Bottom: We measure how much MLPs write along the Wo [TO] Wo [S] direction.
J MINIMALITY SETS
The sets that found for the minimality tests listed in Table
19

---

Published as a conference paper at ICLR 2023
found by greedy optimization
(9, 9), (9, 6). (5, 8), (5, 5), (2, (2,
(9, 9). (11, 10), (10, 7), (8, 6), (5, 8), (4, 11) (IO, 7), (5, 5), (2, 2). (4, II)
(9, 9). (11, 10), (1O, 7), (11, 2), (3. 0), (5, 8), (2,
Figure 18: 4 sets K found by the greedy optimization procedure on our circuit.
K
GREEDY ALGORITHM
The Algorithm 2 describes the procedure used to sample sets for checking the completeness criteria using greedy optimization. In practice, because the naive and the full circuit are not of the same size, we chose respectively = and & = 10 to ensure similar amount of stochasticity in the process. We run the procedure 10 times and kept the sets with the maximal important incompleteness score (including the intermediate K).
Algorithm 2 The greedy sampling procedure for sets to validate the completeness citeria.
1:
K +- 0
2: for to N do
3: Sample a random subset V C C' of k nodes uniformly.
4: UMAX < arg ev [F(C \ U F(C)) K)
5: K + KK [UMA)
6:
end for
7:
return K
As visible in Table 18 the sets found by the greedy search contains a combination of nodes from dif- ferent class. Nonetheless, the overlap between different К suggest that we are missing components from M that can take the place of induction heads or S-inhibition Heads when some Name Mover Heads are knocked-out.
L TECHNIQUES OVERVIEW
This work involved variety of techniques that were required explain model behavior.
Knockouts:
We used knockouts in two different ways: knocking singular components of models, and knocking out everything in the model except particular circuits. The former was some- what useful, and the latter we found powerful.
Knockout of single components: as an attribution method, knocking out singular com- ponents was not always as powerful as techniques such projections, since the com- pensation (or backup) nature of Backup Name Mover Heads in this task allowed com- ponents to be knocked out and their true effect size masked.
Knockouts of all components except circuit: on the other hand, knocking out all components except circuit enabled us to isolate behaviors this task where behavior was sparse, and check the components of our circuit while ignoring the vast percentage of components of the network, making work manageable.
What was very important for the success of knockout and patching experiments was the choice of reference distribution for knockout. The analysis in Appendix G shows how the specific choice of dataset is useful for understanding model components. For a more general knockout, the OpenWebTer Web dataset, GPT's training data, can be used. However, we found that this led to noisier results (though our circuit components still were shown to be important when we used this ablation).
A Attention pattern analysis:
Using attention patterns to explain behavior is always worrying due to the possibility that information has accumulated on that token primarily from previous tokens, that the po- sition with large attention paid to isn't 't't actually writing an important value into the residual stream. In work however, analyzing attention patterns was generally necessary first step before further experiments could be ran, and in this small model, both of the worrying cases did not generally arise
20

---

Published as a conference paper at ICLR 2023
Patching:
Patching was an important method we used to verify causal explanations that were gener- ally formed from correlational evidence. In this way
case is similar to Finlayson et al. (202T) We were surprised however that in general patching gave clear signal on the changes in behavior. This may be because generally patched from inputs like the ABC distribution (which was successful in knocking out too). Therefore, keeping the context of the sentence templates may be generally useful. This could be either because the other words in the templates allow the model to realise that it should be doing IOI, or that in- troducing inputs from other distributions introduces noise that the model picks up on and uses, when this is not intended.
21


Reviews:
[{"review_id": "vhxFDyYApQY", "reviewer": "Reviewer_wPfU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a method for mechanistic interpretation of neural networks, specifically transformer models. The high-level idea is to derive a small subnetwork that can simulate the functionality of the full neural network while satisfying faithfulness, completeness, and minimality criteria. The paper empirically analyze the subnetwork using the GPT-2-small model trained on an indirect object identification (IOI) task. The proposed method is shown to identify several steps needed for the IOI task. ", "summary_of_the_review": "Overall, currently I am not convinced that mechanistic interpretation, as demonstrated in this paper, is a promising direction for understanding black-box models. Details are discussed in the weakness section. Therefore, I vote for rejection. ", "strength_and_weaknesses": "Strengths: \n1. Mechanistic interpretation is a new direction of interpretability research. \n2. The analyzed model is GPT-2-small, of significant complexity. \n3. The writing is clear and easily understandable. \n\nWeaknesses: \n\nEven though the paper tackles a very new area, for which I am fine with some rough and underdeveloped ideas, the general method suffers from too many flaws for me to \"see the light at the end of the tunnel\". \n\n1. The method requires an _a priori_ known model working mechanism (e.g. the heuristic in Sec. 3), and basically maps different subgraphs (i.e. circuits) to various parts of this mechanism. This seems to be a very strong and possibly unjustified assumption: for all but the very synthetic tasks such as IOI, explicit rule-based classifiers (from the 80s and 90s) perform very poorly, suggesting that human-designed heuristics are most likely not how high-performing models work in general. Second, even if humans have a good grasp on one valid heuristic, it is likely that neural networks can use something totally different, due to accidental spurious correlations in the dataset, a problem known as shortcut learning [1]. \n\n2. The IOI task has three simple heuristic steps, mapping to the three major circuit classes. For more complex tasks, there may be tens or even hundreds of heuristics, especially for multi-class classification with a large number of classes (e.g. ImageNet). Identifying and interpreting them could be very cumbersome. \n\n3. The introduction poses three advantages of mechanistic interpretation, \"better predict out-of-distribution behavior, identify and fix model errors , and understand emergent behavior\". However, none of them is demonstrated in the experiments, and it is not clear to me how they are enabled. \n\n4. The paper assumes that attentions work \"as expected\", in that, e.g. \"Name Mover Heads, by default, attend to previous names in the sentence, but due to the S-Inhibition Heads attend less to the S1 and S2 tokens\". There have been numerous evidences suggesting that attentions may not be faithful interpretations [e.g. 2, 3, 4]. How would they be reconciled with the focus on attention in this paper? \n\n[1] https://arxiv.org/abs/2004.07780\n\n[2] https://arxiv.org/abs/1902.10186\n\n[3] https://arxiv.org/abs/2004.03685\n\n[4] https://arxiv.org/abs/2104.14403", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oBChPYBKLj", "reviewer": "Reviewer_jy1a", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper finds a \"circuit\" (in this case, a subset of attention heads), C, that is meant to explain how GPT-2 small solves a particular synthetic task named Indirect Object Identification (IOI).\nThis task composes 15 templates of forms like \"Then, [B] and [A] went to the [PLACE]. [B] gave a [OBJECT] to [A]\".\nThe goal is to predict the name [A] at the end of the sentence, given all of rest of the sentence as context.\nThe model's behavior on other types of inputs was not evaluated.\n\nThe  claim is supported by experiments which aim to establish that C meets the authors' criteria of being faithful, complete, and minimal.\nSome of their experimental results challenge this interpretation, however (see Sections 3.4 and 4.3).\n\nThe authors also describe how they go about identifying the proposed circuit, describing the techniques of knock-outs (i.e. looking at a subgraph of M), and patching (substituting different activations at some nodes of the computational graph while computing a forward pass).\n\nThe novelty of the proposed criteria and techniques is not clearly stated (except completeness, which they state they introduce).\n", "summary_of_the_review": "\nThe paper doesn't provide sufficient support for its claimed contributions.\nIn particular, I'm not convinced that the proposed circuit is an accurate representation of the model's behavior.\nWhile it makes some meaningful progress in terms of techniques and methodology, more work is needed to justify and solidify these approaches; and the presentation also needs work.", "strength_and_weaknesses": "\nStrengths:\n- The notions of faithfulness, completeness, and minimality seem like a promising approach to rigorously evaluating mechanistic interpretability claims.\n- The paper covers a lot of ground; besides (apparently?) introducing these notions, they also propose methods for looking for circuits, and an interpretation of a particular model on a particular task.\n- The interpretability efforts themselves are potentially useful as a case study that could guide or inform future work.\n- The topic is neglected and potentially significant: mechanistic interpretability is worthy goal and not much has been published on it.\n\nWeaknesses:\n- The three claimed contributions are not sufficiently well supported.  \n1) RE \"(1) We identify a large circuit in GPT-2 small that performs indirect-object identification (Figure 2 and Section 3)\":\nThe discovery of the Backup Name Mover Heads suggests that other important bits of circuitry might've been missed, and that the model's behavior on this task might not be very well captured by the circuit.  Perhaps there are also Backup Backup Name Mover Heads, etc.?  Furthermore, as noted, computing the completeness score is intractable, the approximations produce somewhat inconsistent results, and no effort is made to determine which ones are reliable, excepting Section 4.3, where it is indirectly suggested both that the greedy method is both both better at estimating the completeness score, and that the \nThis is methodologically backwards: it assumes the interpretation is correct, and uses this as a basis to challenge the definition of the completeness score.\n2) RE \"(2) Through example, we identify useful techniques for understanding models, as well as surprising pitfalls\":\nThis is a vague contribution and does not make a clear statement of novelty... which techniques are novel?  Or is it their application to \"understanding models\" that is novel?  What are the \"surprising pitfalls\"?\n3) RE \"(3) We present criteria that ensure structural correspondence between the circuit and the model, and check experimentally whether our\ncircuit meets this standard (Section 4)\":\nIt has not been established that these criteria ensures structural correspondence; indeed, the notion of structural correspondence is not defined.  The criteria don't seem quite conceptually correct (see below).\nThe experimental results are not conclusive, and their interpretation remains subjective.  The checks that are performed seem ad-hoc and incomplete.\n- There are a few limitations/concerns that deserve more discussion:\n1) There seems to be an unstated assumption that circuits are they a meaningful way of understanding a model's behavior.  This assumption doesn't seem to be evaluated or supported by this work. \n2) The paper seems optimistic about applying a similar approach to larger models, but doesn't discuss the challenges of scaling this approach (models may have a very high level of inherent complexity, mechanistic interpretability may require a prohibitive amount of human labor needed, even if models are simple enough to be understood mechanistically).  \n3) It seems the interpretation provided in this work can (at most) only tell us about how this model handles this very specific synthetic task.  How does this lead to a meaningful or practical understanding of how a model works more generally?  Or is the idea to study every behavior of interest in isolation? \n- Overall, the writing is not careful in its claims (e.g. Section 3.2 starts with \"Given that the Name Mover Heads are primarily responsible for constructing the output\", but this hasn't been established).\n- The definitions of faithfulness, completeness, and minimality do not seem fine-grained enough to fully capture mechanistic similarity between the circuit and the model.  I would expect the definition of F to measure the average difference in logits instead of the difference in average logits, and measure the difference in the vector of logits or probabilities instead of just the difference between IO and S tokens.  The work does not justify or discuss this chosen definition of F or alternatives.\n- The paper should provide more motivation for this line of research.  What are the pros and cons versus other approaches to interpretability?  What is the value of understanding how the model behaves on this very specific set of inputs?  Can we draw more general conclusions about the model's behavior?  Or what is the end game?  What are the prospects for automating mechanistic interpretability?\n- The definition of the task here is extremely narrow.  The model should be evaluated on OOD examples that similarly require outputting the correct name.  For instance, what if you add distractor text with other names in it to the context?  This work does not address how the model determines whether it should perform this task / engage this circuit.  Is there circuitry for recognizing what the task is?\n\n\nQuestions:\n- Is the goal only to determine how the model solves this one task?  If some of the components of C played a completely different role in different tasks or on OOD examples, how much would that challenge the interpretation?\n- The approach here seems based on finding attention heads that have large individual contributions, but couldn't there be a long-tail of other circuitry contributing to the model's behavior (e.g. in figure 6: what if you include more stuff?  why is .06 / 3% the magic cut-off)?  Section 3.4 suggests this might be the case.  Does this represent a fundamental challenge to the project of finding a circuit?  Consider an analogy with \"typicality\", where most of the probability mass of a distribution (such as a high-dimensional Gaussian) does not come from the most likely samples.\n- For the \"name mover heads\", what does it mean that the OV matrix is a \"name copying matrix\"?  If they \"copy whatever they attend to\" (and not just names), is the claim that they are just approximating the identity matrix?  This could be investigated more.\n- The role of the ABC distribution wasn't clear enough to me; can you clarify why it is used?\n\nMinor comments:\n- The names of the scores are counter-intuitive; e.g. the completeness score measures *in*completeness! \n- The writing sometimes pretty unclear, e.g. \"v can significantly recover F\"; last 2 paragraphs of 3.3.\n- \"serial circuits\" appears undefined.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YSWhCqBn_l2", "reviewer": "Reviewer_iR9D", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Update after rebuttal** After reading the other reviews and the authors' responses I remain in favor of accepting the paper. I think wPfU, jy1a raise some very valid issues, but some of the issues raised concern the viability of the approach/research field of mechanistic interpretability in general, rather than the concrete results presented in the paper - which I think is valid to be discussed but to me is a less relevant criterion to judge one particular paper/analysis. jy1a raises additional relevant criticism specific to this particular paper, and in my opinion the authors have addressed that criticism sufficiently (I'd be keen to hear jy1a's opinion on this).\n\nThe paper is a case study analysis for identifying the mechanism / “algorithm” that GPT-2 small uses for solving a particular language task (Indirect object identification; indirect reference to one of two previously named persons in the same sentence to be precise). Backtracking from attention matrices - the paper essentially identifies groups of attention heads that each have an intuitive functionality, and, when combined, solve the task. To strengthen the identified circuit the paper defines metrics for faithfulness (circuit alone can explain good score on task), completeness (potentially duplicate parts of the circuit have been identified too), and minimality (no superfluous parts, that play no role in functionality and are not duplicates, have been identified) and confirms that the identified circuit fulfills all three notions. To demonstrate that the identified circuit is causally responsible for the functionality Knockouts (rendering parts of the network irrelevant) and Patching (splicing in activations from a different input into parts of the circuit, resembling a more fine-grained intervention than Knockouts) are used.", "summary_of_the_review": "The paper is a strong example of an analysis case study to identify a simple algorithm for solving the indirect object identification task learned by GPT-2. Such case studies are rare, but luckily increasing in number - I personally think it is important to show the community to which degree it is actually possible to understand the internal (mechanistic) function of neural networks. I do not have any major concerns other than expanding the discussion around limitations and specificity of the findings to this particular task (same circuit or parts of it might be involved in other tasks, potentially even with different qualitative functionality). I currently think the work is polished and ready for publication and of interest to a fairly large audience.", "strength_and_weaknesses": "**Main contributions, Impact**\n1) Well executed case study for identifying a mechanistic understanding of the circuit that solves the analysis task in GPt-2 small. Impact: medium - the techniques introduced to identify the circuit are not entirely novel, but are certainly put to good use; the identified circuit is interesting, but it is unclear whether the identified components will play a significant role in other analyses or a general understanding of strategies used by transformers to solve tasks.\n\n2) Use of interventional analysis to establish causal relevance of identified circuit. This is crucial for any serious attempt at *explaining* network functionality; an explanation must lead to falsifiable (causal) predictions. While Knockouts are a standard technique for crude interventional analysis, the paper also uses more fine-grained Patching, demonstrating its usefulness for analysis of complex circuits. Impact: medium - falsification (or establishing causal relevance) is crucial for hypotheses established via interpretability/explainability techniques. While Knockouts are certainly not novel, and I am not sure whether Patching has been used in this precise context before (but would be somewhat surprised if nothing similar has been reported before), the consistent and effective use of the techniques in the paper serves as a good example for future analysis.\n\n3) Analysis of the identified circuit in terms of faithfulness, completeness, and minimality. I particularly like that the paper aims at making these notions formal and quantitative, and reports quantitative results. For instance, without a check for completeness, the backup heads would likely not have been identified - which is an interesting result, highlighting the potential degrees of (complex) redundancies in neural circuitry. Impact: medium to high - the measures seem sensible, though perhaps not maximally universal, and I think that there is a good chance that they will be picked up by the community.\n\n**Strengths**\n *  Well executed analysis case-study, showing that it is possible to open the “black box” or neural circuitry\n * Establishing of causal relevance of explanation, as well as quantitative definition and evaluation of faithfulness, completeness, and minimality of explanation\n * Serious attempts at verifying the correctness of the explanation, and “poking holes” to make sure the results are reliable.\n\n**Weaknesses**\n * The biggest weakness to me is that the results are specific to a particular network (and as the paper suggest do not seem to easily transfer even to GPT-2 medium, a very closely related architecture) and task-instance (which allows for a fairly simple solution strategy as discuss at the beginning of Section 3). There is not much that can be done here - and I actually appreciate focusing on one meticulous analysis rather than analyzing a handful of tasks in a shallow fashion - but the limited generalizability of the results is one weakness of the work - I am more optimistic in terms of generalizing the methodology presented, though not all of it is entirely novel.\n * The second largest weakness to me is that the paper currently does not analyze whether the identified circuit has additional functionality (e.g. is used or even crucial for other tasks). Given the current writing one is inclined to (probably wrongly) infer that the (complete and minimal) sub-circuit found might exclusively serve to solve the particular task presented and that the sub-components also have clear and exclusive functionality (suggested by nomenclature such as ‘Name Mover heads’). I find it quite plausible that the sub-circuit and individual parts of it serve quite different functionality in different tasks, and are also partly involved in other (maybe even quite unrelated) tasks - e.g. there is no ‘name moving’ in an arithmetic task even though the same heads might potentially be involved. This exclusivity is not explicitly claimed in the paper, and perhaps not intended by the authors, but it would be great to see this discussed, and perhaps empirically analyzed (see improvements). Essentially, this is a word of caution regarding the recurrence of the well-known “polysemanticity” problem coined in C. Olah’s earlier work - perhaps the term “polyfunctionality” is more appropriate for circuits instead of individual neurons.\n\n**Improvements**\n1) Please discuss the possibility that the identified circuit and sub-parts (groups of heads) might be involved in other tasks where they might play a quite different functional role. Ideally (though this might be a stretch for a rebuttal phase) analyze whether knocking out the (minimal) circuit leads to performance losses in other, unrelated tasks. (I do not consider empirical results here necessary for publication).\n\n2) Please discuss limitations of the current work slightly more prominently (the discussion already raises some points, but it would be great to see a strong and critical discussion of the limitations).\n\n3) Please make sure that all details to reproduce the analysis are given in the appendix.\n\n**Minor comments**\n\nA) Though section 4 is fairly well written, it did take me two passes to grasp the quantitative metrics introduced for faithfulness, completeness, and minimality. I am not sure if pseudo-algorithms or graphical intuitions would be more helpful, but readability (or perhaps simply notation) could be improved a bit in Section 4. Not a major issue though.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2mZNmIGkrf", "reviewer": "Reviewer_sZ8X", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work authors discovered specific transformer circuits for a natural language task called the indirect object identification, This task requires the LM to identify the indirect object and needs simple logical reasoning. The significance of this paper is that authors identified a large circuit in GPT-2 small that performs a relatively natural, linguistically meaningful task . Furthermore, the authors present a set of criteria that aim to measure the structural correspondence between the circuit and the model. ", "summary_of_the_review": "I’d recommend acceptance, as I think this paper is a solid contribution to the interpretability of language models in natural tasks that are linguistically meaningful. ", "strength_and_weaknesses": "Strength. \nAuthors demonstrated interpretability techniques and criteria through example, making the abstract task of interpretability concrete. \nClarity: the paper is well written and easy to read.\nQuality: The authors have formulated a set of three criteria to measure interpretability and have showed empirical evidence that their circuit have significant improvements compared to a naïve circuit. \nClear limitation: Authors are also being very clear on the limitations of their circuit, showing that their circuit does not do well with the completeness criterion. \n\nWeakness\nFailing the completeness criterion could be seen as a weakness but I tend to agree with the authors that worst-case completeness is too high a bar and could be explored in future works. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster