Towards Best Practices of Activation Patching in Language Models: Metrics and Methods
Authors: Fred Zhang
Neel Nanda
Venue: ICLR 2024 poster (2024)
Keywords: language model interpretability
interpretability
mechanistic interpretability
circuit analysis
activation patching
large language models
Primary area: visualization or interpretation of learned representations
TL;DR: We find that varying the hyperparameters of activation patching can lead to different interpretability results and give recommendations for best practices.

Abstract: Mechanistic interpretability seeks to understand the internal mechanisms of
machine learning models, where localization—identifying the important model
components—is a key step. Activation patching, also known as causal tracing or
interchange intervention, is a standard technique for this task (Vig et al., 2020), but
the literature contains many variants with little consensus on the choice of hyperparameters or methodology. In this work, we systematically examine the impact
of methodological details in activation patching, including evaluation metrics and
corruption methods. In several settings of localization and circuit discovery in language models, we find that varying these hyperparameters could lead to disparate
interpretability results. Backed by empirical observations, we give conceptual arguments for why certain metrics or methods may be preferred. Finally, we provide
recommendations for the best practices of activation patching going forwards.

Published as a conference paper at ICLR 2024
TOWARDS BEST PRACTICES OF ACTIVATION PATCH- ING IN LANGUAGE MODELS: METRICS AND METHODS
Fred Zhang UC Berkeley z0@berkeley.ede edu
Neel Nanda
Independent
neelnanda22@gmaiil. com
ABSTRACT
Mechanistic interpretability seeks to understand the internal mechanisms of machine learning models, where components-is key step. Activation patching, also known as causal tracing interchange intervention, is a standard technique for this task (Vig et al., 2020), but the literature contains many variants with little consensus the choice of hyper- parameters or methodology. In this work, we systematically examine the impact of methodological details in activation patching, including evaluation metrics and corruption methods. In several settings of localization and circuit discovery in lan- guage models, we find that varying these hyperparameters could lead to disparate interpretability results. Backed by empirical observations, we give conceptual ar- guments for why certain metrics or methods may preferred. Finally, we provide recommendations for the best practices of activation patching going forwards.
inttattioonidenifffinnnn the important model
INTRODUCTION
Mechanistic interpretability (MI) aims to unravel complex machine learning models by reverse en- gineering their internal mechanisms down to
algorithms (Geiger et al., 2021; Olah, 2022; Wang et al., 2023). With such understanding, we can better identify and fix model errors (Vig et al., 2020; Hernandez et al., 2021; Meng et al., 2022; Hase et al., 2023), steer model outputs (Li et al., 2023b) and explain emergent behaviors (Nanda et al., 2023a; Barak et al., 2022).
identify the specific model components responsible for partic- ular functions. Activation patching, also known as causal tracing, interchange intervention, causal mediation analysis representation denoising, is a standard tool for localization in language mod- els (Vig et al., 2020; Meng et al., 2022). The method attempts to pinpoint activations that causally affect on the output. Specifically, it involves 3 forward passes of the model: (1) clean prompt while caching the latent activations: (2) on a corrupted prompt; and (3) on the corrupted prompt but replacing the activation of specific model component by its clean cache. For instance, the clean prompt can be "The Eiffel Tower is in" and the corrupted one with the subject replaced by "The Colosseum". If the model outputs "Paris" in step (3) but not in (2), then it suggests that the specific component being patched is important for producing the answer ( (Vig et al., 2020; Pearl, 2001).
A basic goal in MI is localization:
This technique has been widely applied for language model interpretability. For example, Meng et al. (2022); Geva et al. (2023) seek to understand which model weights store and process factual information. Wang et al. (2023); Hanna al. (2023); Lieberum al. (2023) perform circuit analysis: identify the sub-network within a model's computation graph that implements a specified behavior. All these works leverage activation patching or its variants as a foundational technique.
Despite its broad applications across the literature, there is little consensus on the methodological details of activation patching. In particular, each paper tends to use its own method of generating corrupted prompts and the metric of evaluating patching effects. Concerningly, this lack of standard- ization leaves open the possibility that prior interpretability results may be highly sensitive to the hyperparameters they adopt. In this work, study the impact of varying the metrics and methods in activation patching, as a step towards understanding best practices. To our knowledge, this the first such systematic study of the technique.
Specifically, we identify three degrees of freedom in activation patching. First, we focus on the approach of generating corrupted prompts and evaluate two prominent methods from the literature:

---

Published as a conference paper at ICLR 2024
Xnon
John
Bob
and
and
Logits
Mary
Mary
Activation patching intervenes latent states
(b) Patching atlention heads
Figure/ I: The workflow of activation patching for localization: run the intervention procedure (a)
on every relevant component, such as all the attention heads, and plot the effects (b).
Gaussian noising (GN) adds large Gaussian noise the token embeddings of the tokens that contain the key information to completing a prompt, such as subject (Meng et al., 2022).
Symmetric token replacement (STR) swaps these key tokens with semantically related ones; for example, "The Eiffel Tower"- Colosseum" (Vig al., 2020; Wang et al., 2023).
Second, we examine the choice of metrics for measuring the effect of patching and compare prob- ability and logit difference; both have found applications in the literature (Meng et al., 2022: Wang et al., 2023; Conmy et al., 2023). Third, we study sliding window patching. which jointly restores the activations of multiple MLP layers, technique used by Meng al. (2022); Geva et al. (2023).
We empirically examine the impact of these hyperparameters on several interpretability tasks, in- cluding factual recall (Meng al., 2022) and circuit discovery for indirect object identification (IOI) (Wang et al., 2023), greater-than (Hanna et al., 2023). Python docstring completion (Heimersheim & Janiak, 2023) and basic arithmetic (Stolfo et al., 2023). In each setting, we apply methods distinct from the original studies and assess how different interpretability results arise from these variations.
Findings Our contributions uncover nuanced discrepancies within activation patching techniques applied to language models. On corruption method, we show that GN and STR lead to incon- sistent localization and circuit discovery outcomes (Section 3.1). Towards explaining the gaps, we posit that GN breaks model's 's internal mechanisms by putting off distribution. We give tentative evidence for this claim in the setting of IOI circuit discovery (Section 3.2). We believe that this is fundamental concern in using GN corruption for activation patching. On evaluation metrics, we pro- vide an analogous set of differences between logit difference and probability (Section 4), including an observation that probability can overlook negative model components that hurt performance.
Finally, we compare sliding window patching with patching individual layers and summing up their effects. We find the sliding window method produces more pronounced localization than single- layer patching and discuss the conceptual differences between these two approaches (Section 5).
Recommendations for practice At a high-level, our findings highlight the sensitivity of activation patching methodological details. Backed by our analysis, we make several recommendations the application of activation patching in language model interpretability (Section 6). We advocate for STR, as it supplies in-distribution corrupted prompts that help to preserve consistent model behavior. On evaluation metric, we recommend logit difference, as we argue that it offers fine-grained control over the localization outcomes and is capable of detecting negative modules.
BACKGROUND
2.1 ACTIVATION PATCHING
Activation patching identifies the important model components by intervening on their latent activa- tions. The method involves a clean prompt (Xeteans e.g.. Eiffel Tower is in"") with an associated answer r ("Paris"),. corrupted prompt (Xcomupt. e.g., "The Colosseum is in "), and three model runs:
(1) Clean run: run the model on Xetenn and cache activations of set of given model components, such as MLP or attention heads outputs.
Corrupted run the model
X coompt and record the model outputs.
2

---

Published as a conference paper at ICLR 2024
(3) Patched run: run the model on Xeonupe with specific model component's activation restored from the cached value the clean run (Figure la).
Finally, evaluate the patching effect, such as P("Paris") in the patched run (3) compared to the corrupted run (2). Intuitively, corruption hurts model performance while patching restores it. Patching effect measures how much the patching intervention restores performance, which indicates the importance of the activation. We can iterate this procedure over collection of components (e.g., all attention heads), resulting in plot that highlights the important ones (Figure lb).
Corruption methods To generate coomppt GN adds Gaussian noise N((,,)) v) to the embeddings of certain key tokens, where V is 3 times the standard deviation of the token embeddings from the textset. STR replaces the key tokens by similar ones with equal sequence length. In STR, let r' denote the answer of X uuwrpp ("Rome"). All implementations of STR in this paper yield distribution prompts such that X corrupt is identically distributed as a fresh draw of a clean prompt.
Metries The patching effect is defined as the gap of the model performance between the corrupted and patched run, under an evaluation metric. Let cl, *, pt be the clean, corrupted and patched run.
Probability: P(r); e.g., P("Paris")) The patching effect is (r((r) - Pr)::;
Logit difference: LD(r, Logit(r) Logit(r"); e.g., Logit("
Logit("Rome").
The patching effect is given by pD (r, Following Wang et al. (2023), we always normalize this by (r, r') LD, (r,r"). so it typically lies in [0, 1]. where corresponds to fully restored performance and O to the corrupted run performance.
KL divergence: (PL(Pal) P). the dullback-Leible (KL) divergence from the probability distri- bution of model outputs in the clean run. The patching effect is (PalPPalPP)) - (Pal Pr).
GN does not provide corrupted prompt with well-defined answer r' ("Rome").. To make fair comparison, the same r' is used for evaluating the logit difference metric under GN.
2.2 PROBLEM SETTINGS
Factual recall In the setting of factual association, the model is prompted fill in factual infor- mation, c.g., "The Eiffel Tower is in".. Meng et al. (2022) posits that Transformer-bes based language models complete factual recall (i) middle MLP layers and (ii) specifically at the processing of the subject's 's last token. In this work, we do not treat the hypothesis as ground-truth but rather reevaluate using other approaches than what was attempted by Meng et al. (2022).
IOI An 1OI sentence involves initial dependent clause, e.g., "When John and Mary went to the office", followed by main clause, e.g., "John gave a book to Mary." this case, the indirect object (IO) is "Mary" and the subject (S) "John". The IOI task is to predict the final token in the sentence to be the IO. We use SI and S2 to refer to the first and second occurrences of the subject (S).
We let PIol denote the distribution of IOI sentences of Wang et al. (2023) containing single-token names. GPT-2 small performs well on Pior and Wang et al. (2023) discovers a circuit within the model for this task. The circuit consists of attention heads. This is also the focus of our experiments, where we uncover nuanced differences when using different techniques to replicate their result.
3 CORRUPTION METHODS
In this section, we evaluate GN and STR on localizing factual recall in GPT-2 XL and discovering the 1OI circuit in GPT-2 small.
Experiment setup For factual recall, we investigate Meng et al. (2022)'s 's hypothesis that model computation is concentrated at early-middle MLP layers (by processing the last subject token). Specifically, we corrupt the subject token(s) to generate Xcomupe- In the patched run, we override the MLP activations at the last subject token. Following Meng al. (2022); Hase al. (2023), at each layer we restore a set of adjacent MLP layers. (More results on other window sizes can be found in Section H.1. We examine sliding window patching more closely in Section 5.)
3

---

Published as a conference paper at ICLR 2024
Probability, window size
Logit difference, window size
STR
0.08
0.2
Last subject roken
MLF?
0.04
0.1
0.02
END
Logit
(a) Patching MLP at the last subject token.
20 40
20 40
(b) Probability as the metric
(c) Logit difference as the metric
Figure 2: Disparate MLP patching effects for factual recall in GPT-2 XL. (a) We patch MLP activations at the last subject token. with a window size of 5. STR suggests much a weaker peak, regardless of the evaluation metric.'
(b)() The patching effects using different comrpption methods
For IOI circuit discovery, we follow Wang et al. (2023) and focus the role of attention heads. Corruption is applied to the S2 token. Then we patch single attention head's output (at all positions) and iterate over all heads in this way. To avoid relying on visual inspection, we say that head is detected if its patching effect is 2 standard deviations (SD) away from the mean effect.
Dataset and corruption method STR requires pairs of Xetean and X Xoomps that are semantically similar. To perform STR, we construct PAIREDFACTS of 145 pairs of prompts on factual recall. All the prompts are in-distribution, as they are selected from the original dataset of Meng et al. (2022); see Appendix C for details. GPT-2 XL achieves an average of 49.0% accuracy this dataset.
For the IOI circuit, we use the Pior distribution to sample the clean prompts. For STR, we replace S2 by 10 to construct Xcorupt such that Xceeuee is still valid in in-iisrrbbutin IOI sentence. For GN, we add noise to the S2's s token embedding. The experiments are averaged over 500 prompts.
3.1 RESULTS ON CORRUPTION METHODS
Difference in MIP localization For patching MLPs in the factual association setting, Meng al. (2022) show that the effects concentrate at carly-middle layers, where they apply GN as the corrup- tion method. Our main finding that the picture can be largely different by switching the corruption method, regardless of the choice of metric. In Figure 2, we plot the patching effects for both metrics. Notice that the clear peak around layer 16 under GN is not salient at all under STR.
This is robust phenomenon: across window sizes, we find the peak value of GN to be 2x-5x higher than STR: see Appendix H. for further plots on GPT-2 XL in this setting.
These findings illustrate potential discrepancies between the two corruption techniques in drawing interpretability conclusions. We do not, though, claim that results from GN are illusory or overly inflated. In fact, GN does not always yield sharper peaks than STR. For certain basic arithmetic tasks in GPT-J, STR can show stronger concentration in patching MLP activations; see Appendix D.
Difference in circuit discovery We focus discovering the main classes of attention heads in the 1O1 circuit, including (Negative) Name Mover (NM), Duplicate Token (DT), S-Iniibition (SI), and Induction Heads. The results are summarized in Table and more details in Appendix I.
Most importantly, we observe that STR and GN produce inconsistent discovery results. In particular, for any fixed metric, STR and GN detect different sets of heads as important, highlighted in Table 1.
We remark that all the detections are in the IOI circuit as found by Wang et al. (2023); see Ap- pendix B for an overview. However, the discovery achieved here appear far from complete, with some critical misses such as NM. This suggests that the extensive manual inspection and the use of path patching, a more surgical patching method, are both necessary to fully discover the IOI circuit.
We also validate our high-level conclusions on the Python docstring (Heimersheim & Janiak, 2023) and the greater-than (Hanna et al., 2023) task. In particular, we find GN can produce highly noisy localization outcomes in these settings; see Appendix E and Appendix F for details.
'The effects the first 3 layers are large simply because MLPO has significant influence the model's 's outputs in GPT-2, regardless of the task (Wang al., 2023; Hase al., 2023), so is not the focus here.
4

---

Published as a conference paper at ICLR 2024
Corruption
Metric
||
NM
DT
SI
Negative NM
Induction
STR
Probability
1/3
0/2
3/4
1/2
1/2
GN
Probability
0/3
1/2
2/4
2/2
1/2
STR
Logit difference
1/3
0/2
3/4
2/2
1/2
GN
Logit difference
1/3
1/2
3/4
2/2
1/2
STR
KL divergence
1/3
0/2
3/4
2/2
1/2
GN
KL divergence
0/3
0/2
2/4
2/2
1/2
Table I: Inconsistency on the I0I task. We patch the attention heads outputs and list the detections of each class; e.g., 1/3 NM indicates out of NMs is detected. Alss detect 0. 10,0 fuzzy Duplicate Token Hcad, as negatively influencing model performance. We expect it to be positive (Wang et al., 2023).
Attention Movers (corrupted)
Attention
Movers (patched)
Ciaan
STR
STR
OM
OM
S-hnhibriio
(b) We patch the value ma- trices of all the ST heads and examine the impact NMs' attention patterns.
(a) Corrupted run
(c) Patched run
Figure 3: Attention of the Name Movers from the last token, in corrupted and patched runs.
3.2 EVIDENCE FOR OOD BEHAVIOR IN GAUSSIAN NOISE CORRUPTION
We suspect that the gaps between the corruption methods can be attributed partly to model's OOD behavior under GN corruption. In particular, the Gaussian noise may break model's internal mech- anisms by introducing OOD inputs to the layers. We now give some tentative evidence for this hypothesis. Following the notation of Wang al. (2023), head is denoted by "layer.head". We say a detection is negative if the patching effect of the component is negative (under a given metric).
Negative detection of 0.10 under GN Although most localizations we obtain above seem aligned with the findings of Wang et al. (2023), major anomaly in the GN experiment is the "negative" detection of head 0.10. In particular, probability and KL divergence suggest that it contributes negatively to model performance. (Logit difference also assigns a negative effect, though to lesser degree; see Figure 29b.) This is not observed all in the experiments with STR corruption.
The detection is in the wrong direction, given the evidence from Wang et al. (2023) that 0.10 helps with IOI; on clean prompts, by visualizing the attention patterns, find that this effect largely disappears under GN corruption. We intuit that the Gaussian noise is strongest at influencing early layers, and 0.10's behavior may be broken here, since it directly receives the noised token embeddings from the residual stream.
active at S2, attends to S1 and signals this duplication. However,
Attention of Name Movers To exhibit the OOD behavior of the model internals under GN cor- ruptions, we examine the Name Mover (NM) Heads, class of attention heads that directly affects the model's logits in the IOI circuit (Wang et al., 2023). NMs are active at the last token and copy what they attend to. We plot the attention of NMs clean and corrupted runs in Figure 3a.
Indeed, on 500 clean IOI prompts, the NMs assign average of 0.58 attention probability to IO. In the corrupted runs, since STR simply exchanges by $1, the attention patterns of NMs are preserved (with the role of IO and SI switched). On the other hand, with GN corruption, we see that the attention is shared between IO and SI (0.26 and 0.21). This suggests that GN not only removes the relevant information but also disrupts the internal mechanism of NMs IOI sentences.
To take a deeper dive, Wang et al. (2023) shows that the output of NMs is determined largely by the values of the S-Inibbitonn Heads. Indeed, can fully recover model's logit I0 in STR (logit difference: 1.04) by restoring the values of the S-Inhibition Heads (Figure 3b). The same intervention, however, is fairly unsuccessful under GN (logit difference: 0.49).
5

---

Published as a conference paper at ICLR 2024
Towards explaining this gap, we again examine the attention of NMs. Figure 3c shows that patching nearly restores the NMss in-distribution attention pattern under STR, but fails under GN corruption. We speculate that GN introduces further corrupted information flowing into the NMs such that restoring the clean activations of S-Inhibition Heads cannot correct their behaviors.
EVALUATION METRICS
We now study the choice of evaluation metrics activation patching. We perform two experiments that highlight potential gaps between logit difference and probability. Along the way, provide conceptual argument for why probability can overlook negative components certain settings.
4.1 LOCALIZING FACTUAL RECALL WITH LOGIT DIFFERENCE
The prior work of Meng al. (2022) hypothesizes that factual association is processed at the last subject token. Motivated by this claim, we extend our previous experiments to patching the MLP outputs at all token positions and consider the effect of changing evaluation metrics.
Experimental setup We apply the same setting as in Section 3. We extend our MLP patching experiments all token positions and again use logit difference and probability as the metrie.
Experimental results For STR and window size of 5, we plot the patching effects across layers and positions Figure The visualization shows that probability assigns stronger effects the last subject token than logit difference. Specifically, we calculate the ratio between the sum of effects (over all layers) on the last subject token and those on the middle subject tokens. both corruptions, probability assigns more effects to the last subject token than logit difference:
Using STR corruption, the ratio 4.33x in probability > 11.22x in logit difference.
Using GN corruption, the ratio is 1.74x probability > 0.77x logit difference.
This observation holds for other window sizes, too, for which we provide details in Appendix H.2. We also validate our findings on GPT-J 6B (Wang & Komatsuzaki, 2021) in Appendix H.5. The results show that the choice of evaluation metrics influences the patching effects across tokens.
0.02
10
0.0L
0.2
40
-007
Riot sutject caloen
Lagt sctject tolen.
Purther tokers
Rott sudject roten
Last subject token
Purtlar tahent
sucject talened
subsuquent tokon
solject takers
sobasqueet talen
taken
(a) Logit difference (STR)
(b) Prohability (STR)
Figure 4: Activation patching on MLP across layers and token positions in GPT-2 XL, with sliding window patching of size 5. Note that probability (b) highlights the importance of the last subject token, whereas logit difference (a) displays less effects.
4.2 CIRCUIT DISCOVERY WITH PROBABILITY
Wang et al. (2023) discovers two Negative Name Mover (NNM) heads, 10.7 and 11.10, that notice- ably hurt model performance on IOL. In our previous experiments on STR, both are detected, except when using probability as the metric where 11.10 is overlooked. In fact, the patching effect of 11.10 under STR in probability is well within SD from the mean (mean 0.003, SD 0.015, and 11.10 receives -0.022). Looking closely, the reason is simple:

---

Published as a conference paper at ICLR 2024
Probabiiity, GN, window -1
Probuiiiiity
Probability,
Probability
233
5.0
644
149
213
20
(a) Single-layer patching
(b) Window size of
(c) Window size of 5
(d) Window size of 10
Figure 5: Sliding window patching vs summing up individual patching effects; patching MLP
activation at the last subject token in GPT-2 XL on factual recall prompts. Sliding window patching offers 1.40x, 1.75x and 1.59x peak value than summation of single-layer patchings. Single-layer patching (a) suggests weak peak.
In the corrupted run STR, the average probability of outputting the original IO is 0.03. Hence, the patching effect in probability, (10) TP. (10), is least -0.03, Pr (IO) is non-negative. This is already close to 2 SD below the mean (-0.027). Hence, for an NNM to be detected via patching, its PM (IO) needs be near which may be hard reach.
By contrast, under GN corruption, the average probability of IO is 0.13. Intuitively, this makes lot more space for NNMs to demonstrate their effects.
In general, probability must fail detect negative model components, if corruption reduces the correct token probability to near zero. We now give a cleaner experimental demonstration of this concern, using an original approach of Wang et al. (2023).
Experimental setup We revisit an alternative corruption method proposed by Wang et al. (2023), where $1, S2 and IO are replaced by three unrelated random names for example, "John and Mary [...]. John" - "Alice and Bob [--.], Carol." We probability of the original I0 the metric. Intuitively, this replacement method would achieve much stronger corruption effect, since it removes all the relevant information (S and IO) of the original IOI sentence.
Experimental results First, we observe that the probability of outputting the 10 of the original IOI sentence negligible (5c-4) under this corruption. As result, using probability detects neither NNMs. On the other hand, we find that logit difference still can. See Appendix I.3 for the plots. In Appendix G. we confirm the same finding when corruption applied to and IO only.
At a high-level, we believe that this is pitfall of probability evaluation metric. Its non-negative nature makes incapable discovering negative model components in certain settings.
SLIDING WINDOW PATCHING
In this section, we examine the technique of sliding window patching in localizing factual informa- tion (Meng et al., 2022). For each layer, the method patches multiple adjacent layers simultaneously and computes the joint effects. Hence, one should interpret the result of Meng et al. (2022) as the effects being constrained within window rather than at single layer. We argue that such as hypothesis can be tested by an alternative approach and we compare the results from these two.
Experimental setup Instead of restoring multiple layers simultaneously, MLP layer one at a time. Then as an aggregation step, for each layer, sum up the single-layer patching effects of its adjacent layers. For example, we add up the effect at layer layer get an aggregated effect for layer 4. We patch the MLP output at the last subject token.
patch each individual
Experimental results For each window size, we compute the ratio of the maximum patching effect at the middle MLP layers between sliding window patching and summation of single-layer patching. Over the combinations of window sizes, metrics and corruption methods, we find sliding window patching typically provides least 20% more peak effect than the summation method.
"This corrupted distribution is denoted by PABC the original paper of Wang al. (2023)

---

Published as a conference paper at ICLR 2024
In Figure 5, for window sizes of 3,5, 10, plot the results using GN corruption and probability as the metric, the original setting as in Meng et al. (2022). We observe sigmificant gaps between the sliding window and the summation method. Moreover, for single-layer patching, the peak at layer 15 is fairly weak (Figure 5a). Sliding window patching appears to generate more pronounced the concentration, as we increase the window sizes.
The result suggests that sliding window patching tends to amplify weak localization from single- layer patching (see Figure 12 for plots on single-layer MLP patching in GPT-2 XL). We believe this may arise due to certain non-linear effects in joint patching and therefore results from which should be carefully interpreted; see Section for more discussions.
6 DISCUSSION AND RECOMMENDATIONS
We have observed variety of gaps between corrpption methods and evaluation metrics used in activation patching on language models. this section, we summarize our findings and provide recommendations.
Corruption methods We are concerned that GN corruption puts the model off distribution by introducing noise never seen during training. Indeed, in Section 3.2, we provide evidence that in the corrupted run, model's internal functioning is OOD relative to the clean distribution. This may induce unexpected anomalies in the model behavior, interfering with our ability to localize behavior to specific components. Conceivably, GN corruption could even lead to unreliable or illusory results.
More broadly, this presents a challenge to any intervention techniques that introduce OOD inputs to the model or its internal layers, including ablations. In fact, similar concerns have been raised earlier in the interpretability literature on feature attribution well; see e.g. Hooker et al. (2019); Janzing et al. (2020); Hase et al. (2021).
In contrast, STR uses counterfactual prompts ("The Eiffel Tower is in" "The Colosseum is in"") that are in-distribution and thus induces in-distribution activations, avoiding the OOD issue. Thus, recommend STR whenever possible. GN, simpler methods such ablation, may be consid- ered as an alternative when token alignment or lack of analogous tokens makes STR unsuitable.
Evaluation metries We generally recommend avoiding using probability as the metrie, given that it may fail to detect negative model components.
We find logit difference convincing metric for localization in language models. Consider IOI setting where model contains an attention head that boosts the logits of all (single-token) names. This head, though important, should not be viewed as part of the IOI circuit, but our interventions may still affect it. By measuring Logit(IO) - Logit(S), logit difference controls for such com- ponents and ensures they are not detected. This may not be achieved by other metrics, such as probability or Logit(IO) alone.
KL divergence tracks the full model output distributions, rather than focused only on the correct or incorrect answer, and can be reasonable metric for circuit discovery as well (Conmy et al., 2023).
Sliding window patching We speculate that simultaneously patching multiple layers could cap- ture the following non-linear effects and results in inflated localization plots:
Joint patching may suppress the flow of corrupted information within the window of patched layers, where single-layer patching offers no such control.
A window of patched layers may jointly perform crucial piece of computation, such as a major boost to the logit of the correct token, which no individual layer can single-handedly achieve.
Generally, when examining the outcome from sliding window patching, one should be aware of the possibility of multiple layers working together. Thus, the results from the technique are to be interpreted as the joint effects of the full window, rather than of a single layer. In practice, we recommend experimenting with single-layer patching first and only consider sliding window patching when individual layers seem to induce small effects.
We note that our interventions do not affect the head, then will not show up any metric.
8

---

Published as a conference paper at ICLR 2024
Which tokens to corrupt? In some problem settings, prompt contains multiple key tokens, all relevant to completing the task. This would offer the flexibility to choose which tokens to corrupt. This is another important dimension of activation patching. For instance, Section 3 corrupt the S2 token. An alternative is to corrupt the $1 and IO. While this may seem an implementation detail, we find that this can greatly affect the localization outcomes.
experiments on IOI in
Specifically, in Appendix G, we test corrupting S1 and I0 in activation patching on IOI sentences, by changing their values random names or adding noise to the token embeddings We find that almost all techniques discover the 3 Name Mover (NM) Heads of the IOI circuit (Table 4 and Figure 11). These are attention heads that directly contribute to Logit(IO) as shown by Wang et al. (2023). In contrast, our prior experiments corrupting S2 miss most of them (Table 1).
We intuit that corrupting different tokens allows activation patching trace different information within the model, thereby suggesting varying localizations results, For instance, in our prior exper- iments replacing S2 by IO, patching traces the value of IO its position. On the other hand, in changing the values of S1 and IO while fixing their positions, patching highlights exactly where the model processes these values.
In practice, we recommend trying out different tokens to corrupt when the problem setting offers such flexibility. This may lead more exhaustive circuit discovery.
RELATED WORK
Activation patching Activation patching is a variant of causal mediation analysis (Vig et al., 2020; Pearl, 2001), similar forms of which are used broadly in the interpretability literature (Soulos al., 2020; Geiger et al., 2020; Finlayson et al., 2021; Geiger et al., 2022). The specific one with GN corruption was first proposed by Meng et al. (2022) under the name of causal tracing. Wang et al. (2023); Goldowsky-Dill et al. (2023) generalize this to a more sophisticated version of path patching.
Circuit analysis Circuit analysis provides post-hoc model interpretability (Casper et al., 2022). This line of work is inspired by Cammarata et al. (2020); Elhage et al. (2021). Other works include Geva et al. (2022); Li et al. (2023a); Nanda al. (2023a); Chughtai al. (2023); Zhong et al. (2023); Nanda et al. (2023b); Varma et al. (2023); Wen et al. (2023); Hanna et al. (2023); Lieberum et al. (2023). Circuit analysis often requires manual effort by researchers, motivating recent work to seale or automate parts of the workflow (Chan et al., 2022; Bills et al., 2023; Conmy et al., 2023; Geiger et al., 2023; Wu et al., 2023; Lepori et al., 2023).
Mechanistic interpretability (MI) MI aims to explain the internal computations and representa- tions of a model. While circuit analysis is a major direction under this broad theme, other recent case studies of MI in language model include Mu & Andreas (2020); Geva et al. (2021); Yun et al. (2021); Olsson et al. (2022); Scherlis et al. (2022); Dai et al. (2022); Gurnee et al. (2023); Merullo et al. (2023); McGrath al. (2023); Bansal et al. (2023); Dar et al. (2023); Li et al. (2023c); Brown et al. (2023); Katz & Belinkov (2023); Cunningham et al. (2023).
CONCLUSION
We examine the role of metrics and methods in activation patching in language models. We find that variations in these techniques could lead to different interpretability results. We provide several recommendations towards the best practice, including the use of STR as the corruption method.
In terms of limitations, our experiments are on decoder-only language models of size up to 6B. We leave it as future direction to study other architectures and even larger models. Our work tests overriding corrupted activations by clean activations. The other direction-patching corrupted to clean-has also been used for circuit discovery, and it is interesting to compare these two. In addi- tion, we provide tentative evidence that certain corruption methods lead to OOD model behaviors and suspect that this can make the resulting interpretability claims unreliable. Future work should examine this hypothesis closely and furnish further demonstrations. Finally, it is interesting to de- velop more principled techniques for activation patching or propose other methods for localization.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
FZ would like to thank Matthew Farhbach, Dan Friedman, Johannes Gasteiger, Asma Ghandehari- oun, Stefan Heimersheim, János Kramár, Kaifeng Lyu, Vahab Mirrokni, Jacob Steinhardt and Peilin Zhong for helpful discussions, and Jiahai Feng, Yossi Gandelsman, Oscar Li and Alex Wei for com- ments on early drafts of the paper.
REFERENCES
Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. Rethinking the role of scale for in-contex. learning: An interpretability- -based study at 66 billion scale. In Anmual Meeting of the Association for Computational Linguistics (ACI.), 2023.
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hid- den progress in deep learning: SGD learns parities near the computational limit. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublles blo.eeaaaapub............. 2023.
blob. core. wwwdww.indw.....m net www.nneeeeeeeneeeeeeeeeeeeeeeete wett//euroon--xpaineeee explainer/paperr Seraaper/indx..eee html, wwwdows.net/neureeee
Davis Brown, Nikhil Vyas, and Yamini Bansal. On privileged and convergent bases in neural net- work representations. arXiv preprint arXiv:2307.12941 2023.
Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 5(3)=e24, 2020.
Stephen Casper, Tilman Rauker, Anson Ho, and Dylan Hadfield-Menell. Toward transparent Al:
A survey on interpreting the inner structures of deep neural networks. In IEEE Conference Secure and Trustworthy Machine Learning (SaTML), 2022.
Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldwosky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing, a method for rigorously testing interpretability hypotheses. Al Align- htttps://wwt alignmentforum wwt.ormmo..oo.ooo.eeeooeeeee.eoo ortg/posts//22hhayccu2777777
ment Forum, 2022. scrallli-uubbing-aaaee forthdd-foo-riggrruulll-tettett testing
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. In International Conference on Machine Learning (ICML), 2023.
Arthur Commy. Augustine N Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretabilitty In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0880 2023.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodd: Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. tttts::/trrasffomr--irrcoo pub/2021 wwwpb//222/ffaaeooo///// framework/index. html.
10

---

Published as a conference paper at ICLR 2024
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart M Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language mod- els. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACI-IJCNLP),, 2021.
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Inter- national Conference Machine Learning (ICML), 2022.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D) Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint arXiv:2303..25560 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2022.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Nicholas Goldowsky-DD Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05569, 2023.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitri Troitskii, and Dimitris Bert- simas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.0160 016600 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2023.
Peter Hase, Harry Xie, and Mohit Bansal. The duits--dist problem in explainability and search methods for feature importance explanations. In Advances Neural Information Process- ing Systems (NeurIPS), 2021.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? Surprising differences in causality- based localization vs. knowledge editing in language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Stefan Heimersheim and Jett Janiak. A circuit for Python docstrings in 4-layer
attention-only transformer. httpp://wwe al wwwggnenttoouuut org/ wwtggpooss//bbtee poosts/66KXXnKKFab FbXFX2XKKKKK a-circuit-for-python dowtrhoondddsttingg---d- --y---- atteetiittttttoono--ooo
2023.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference Learning Representations (ICLR), 2021.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. In Advances in neural information processing systems (NeurIPS), 2019.
11

---

Published as a conference paper at ICLR 2024
Dominik Janzing, Lenon Minories, and Patrick Blöbaum. Feature relevance quantification in plainable ai: A causal problem. In International Conference on Artificial Intelligence and Statis- fics (AISTATS), 2020.
Shahar Katz and Yonatan Belinkov. Interpreting transformer's attention dynamic memory and visu- alizing the semantic information flow of GPT. arXiv preprint arXiv:2305. 23t.::333..1117,, 2023.
Michael A Lepori, Ellie Pavlick, and Thomas Serre. analysis. arXiv preprint wrXiv:2309.0024. 2023.
NeuroSurgeon: A toolkit for subnetwork
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring sequence model trained on synthetic task. In International Conference on Learning Representations (ICLR), 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards mechanistic understanding. In International Conference on Machine Learning (ICML), 2023c.
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Miku- lik. Does circuit analysis interpretability scale? Evidence from multiple choice capabilities in Chinchilla. arXiv preprint arXiv:2307.09458 2023.
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Jack Merullo, Carsten Fickhoff, and Ellie Pavlick. Language models implement simple word2vee- style vector arithmetic. arXiv preprint 2605 2023.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Neel Nanda and Joseph Bloom.
TransformerLenn.
https://gttubb.ooo/neeladdddddd cow./neellanna-iio
TransformerLens, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures
for grokking via mechanistic interpretability. In International Conference on Learning Represen- tations (ICLR), 2023a.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.00941 2023b.
Chris Olah. Mechanistic interpretabilityy variables, and the importance of interpretable bases. https: //transformer-ciinn -iitumirrrrrittts..e http://////mccciiinneereees////e
html, 2022.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac -oddi, Danny Hernandez. Scott Johnston. Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei. Tom Brown, Jack Clark, Jared Kaplan, Sam McCan- dlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transfoommr-ccte hadex.html.
ts. twtpsss.pppppeeeeeee -ww--oote----erinn--anneeee
heads//
Judea Pearl. Direct and indirect effects. In Conference on Uncertainty and Artificial Intelligence (UAI), 2001.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
12

---

Published as a conference paper at ICLR 2024
Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022.
Paul Soulos, R Thomas McCoy, Tal Linzen, and Paul Smolensky. Discovering the compositional structure of vector representations with role learning networks. In Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, 2020.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. Understanding arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305.15044 2023.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint (.Xiv:2309002399. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2017.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https://gi thub. comm/kingoollllz/mesttt-toooo twwnsforrr--fnaaaoro May 2021.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in GPT-2 small. In International Conference on Learning Representations (ICLR), 2023.
Kaiyue Wen, Yuchen Li. Bingbin Liu, and Andrej Risteski. (Un)interpretability of transformers: case study with Dyck grammars. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dic- tionary learning: contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, 2021.
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
A REVIEW OF THE TRANSFORMER ARCHITECTURE
We follow the notation of Elhage et al. (2021) and give a review of the Transformer architecture (Vaswani et al., 2017). The input To Nxd a transformer model is a sum of position and token embeddings, where N is the sequence length and d is the dimension of the model's internal states. The input is the initial value of the residual stream which subsequently gets updated by the transformer blocks.
Each transformer block consists of a multi-head self-attention sublayer and an MLP sublayer. (For GPT-J, these two sublayers are parallelized.) The MLP sublayer is a two-layer feedforward network that processes each token position independently in parallel. Following Elhage et al. (2021), the output of the attention sublayer attention output can be written as y = E/= his (æ4). where hi,5 denotes the jth attention head of the layer. Each head has four weight matrices, Wir Wid wis Rdx and Wo E IR $ xd. For
be decomposed into individual heads. For the ith layer, the
13

---

Published as a conference paper at ICLR 2024
residual stream H, refer Q xWaa K xw K. V -w the query, key and value of the head. The attention pattern is given by
(IWa') ((WW)) + M ERNNN
A = softmax
Vd/H
where M the attention mask. In auto-regressive language models, the attention pattern is masked to lower triangular matrix. The output of the attention sublayer is given by
z + Concat [A Ail.
A A'
A H H Wo.
(1)
B REVIEW OF THE ATTENTION HEADS IN IOI CIRCUIT
We provide an overview of the attention heads and their functionalities of the IOI circuit in GPT-2 small, found in Wang al. (2023). Our exposition here follows Section 3 of Wang al. (2023).
Name Mover (NM) Heads are active the END position, attend previous names, and copy the names they attend to. On 1OI sentences, they directly copy the correct name and thus contribute positively to model performance.
Negative Name Mover Heads writes in the opposite direction of the Name Mover Heads, decreas- ing the model's logit on the correct name.
Duplicate Token (DT) Heads identify tokens that previously appeared in the sentence. On IOI sentences, they are active at S2 and attend primarily to S1, contributing positively to model per- formance.
Induction Heads play the same role as the Duplicate Token Heads, though via a different induction- like mechanism.
S-Inibittion (SI) Heads remove duplicate tokens from Name Mover Heads' attention. On IOI sentences, they inhibit Name Mover Heads' attention to S1 and S2.
Aside from Negative Name Mover Heads, all the model components above contribute positively to the model performance on Piot, as shown Wang et al. (2023).
Head 0.10 Head 0.10 found be a fuzzy Duplicate Token Head by Wang et al. (2023). pays attention to S1 from S2, but the pattern is fuzzy, as other tokens also receive non nogllgglig attention mass. Nonetheless, study on the effect of patching head 0.10 in Section 3.2 under GN corruption.
expected that it helps with the model on the IOI task. We give detailed
C DETAILS ON EXPERIMENTAL SETTINGS
For Gaussian noise (GN) corruption, we corrupt the embeddings of the crucial tokens by adding a Gaussian noise & ~ (0; v), where set be 3 times the standard deviation of the token embeddings from the dataset (Meng et al., 2022).
We always perform GN and STR experiments parallel. For STR, there is natural the incorrect token r', since X Xeompp is also valid in-distribution prompt. This allows for well-defined metric Logit(r) Logit(r"). To make a fair comparison, same used for evaluating the logit difference metric under GN.
of logit difference LD(r,,')
Throughout the paper, layers are zero-indexed, numbered from 0 to L - rather than 1 to L.
Factual recall To perform STR in the factual association setting, construct PAIREDFACTS, a dataset of 145 pairs of prompts. Within each pair, the two prompts have the same sequence length (under the GPT-2 tokenizer) but distinct answers. All the prompts are selected from the COUNTERFACT and NOWWN 000 datasets of Meng et al. (2022). On these prompts,
GPT-2 XI. achieves an average of 49.0% probability on the correct token and 6.85 logit difference.
GPT-2 large achieves 41.1% and 5.88 logit difference.
14

---

Published as a conference paper at ICLR 2024
GPT-J achieves 50.1% and 7.36 logit difference.
few samples of the PAIREDFACTS dataset are listed in Figure 31 of Appendix J.
Since the prompts are perfectly symmetric and all of them are in-distribution, our STR experiments consist both ways, where a prompt within a pair play the role of both Xcomup and X
Our experiments with GN corruption is performed in the same manner in Meng et al. (2022); Hase et al. (2023), with noise applied to all subject tokens' embeddings.
The experiments here are implemented via the TransformerLLens library (Nanda & Bloom, 2022).
Unless specified otherwise, GN applies Gaussian noise to the $2 token embedding. Over 500 prompts, the probability of outputting 10 is P. (r) = 0.129 under GN corruption (with being IO), whereas it 0.481 under the clean distribution Ptor-
IOI
All our experiments performed using the original codebase of Wang et al. (2023), available at https://gi thub.
The code provides the functionality of constructing Xcorupe under various definitions of corruptions, including STR.
D RESULTS ON ARITHMETIC REASONING IN GPT-J
Experimental setup We follow the setting of Stolfo et al. (2023) and perform localization analysis on the task of basic arithmetic in GPT-J (Wang & Komatsuzaki, 2021), decoder-only model with 6B parameters. For simplicity, we consider addition, subtraction and multiplication up to 3 digits. We provide the model with a 2-shot prompts of the format
X 21
X₂ + Y/2 = Zz
X3 +
where the numbers X,, are random integers and the operator can be +, -, Stolfo et al. (2023) finds that this leads to improved accuracy. Since large integers get split into multiple tokens, we draw X,, Y, from 21.2.... 250) for addition and subtraction and from {1, 2, ... 233 for multiplication. To obtain dataset for activation patching, we first draw 200 prompts and discard those on which the model's top-ranked output token is incorrect.
We set GN corruption to add noise to the token embeddings at the positions of X3, Y3. Similarly, STR replaces X3, Y3 by two random integers drawn from the same set, which ensures that the corrupted prompt is still in-distribution. We remark that Stolfo et al. (2023) applies the same STR corruption in their patching experiments.
Stolfo et al. (2023) devises new metric to evaluate the patching effects. More precisely, they report:
P (p((r) Px(rr Pa(r') - Pp(('))
(.(r)
(r
from patching the MLP activation at last token the prompt.* We compute the patching effect given by the metric, well as probability and logit difference.
Following Stolfo et al. (2023), we narrow our focus on localization of MI. layers. All the experi- ments patch single MLP layer's activation at the last token of the prompt.
Experimental results Focused on the logit difference and probability metric, we observe gaps between GN and STR for addition and subtraction. In particular, STR is found to provide sharper concentration, up to a magnitude of 4x. This in contrast with our results on factual association (Section 3.1), where GN appears induce stronger peak. For multiplication. GN and STR pro- vides nearly matching results. This highlights that activation patching can be sensitive to corruption methods a rather unpredictable way. See Figure 6, Figure 7 and Figure 8 for plots.
For the metric (2) of Stolfo et al. (2023), we qualitatively replicate their results, similar to Figure 2 of their paper, and find extremely pronounced peak with STR corruption. Towards understanding this
Note that notations here different from Stolfo al. (2023).
15

---

Published as a conference paper at ICLR 2024
observation, examine the quantity (2) closely and discover that its first term typically dominates the second. This, in turn, is because the denominator term P. (r), the probability of outputting the correct answer in the corrupted run, is usually tiny under STR corruption. The small denominator, therefore, acts as large multiplier that amplifies the absolute gap between patching different layers. We note that this effect is much smaller under GN since P. (r(r) usually not negligible.
Logit difference, addition GPT-j.
Probability, addition on GPT:j MLP
Stolfo's metric. addition on GPT-J, MLP
oM
Prbbibity
Probabitty
0.015
0.35
0.01
4000
0.05
0.605
30
25
Layer
Layer
(a) Logit difference as the metric
(b) Probability as the metric
(c) Stolfo et al. (2023)'s metric
Figure 6: The effects of patching MLP layers in GPT-J addition prompts.
Logit difference, subtraction an GPT-J. MLP
Probabi ity, subtraction on GPT:j, MLP
Stotfo's metric, subtraction on GPT:j, MLP
2500
Pretaciity
0.2
Probabatty
1000
0.035
0.65
Layer
Layer
Layer
(a) Logit difference as the metric
(b) Probability as the metric
(c) Stolfo et al. (2023)'s metric
Figure 7: The effects of patching MLP layers in GPT-J on subtraction prompts.
Logit difference, multiplication an GPT-j. MLP
Probabillity, bil.bilillty multiplication on GPT:), MLP
Stolfo's metric, multiplication on GPT-J, MLP
Probeciity
Probaciity
0.00
0.005
30
Layer
Layer
Layer
(a) Logit difference as the metric
(b) Probability as the metric
(c) Stolfo et al. (2023)'s 's metric
Figure 8: The effects of patching MLP layers in GPT-J on multiplication prompts.
RESULTS ON PYTHON DOCSTRING CIRCUIT
Heimersheim & Janiak (2023) studies circuit for Python docstring completion in a pre-trained 4- layer attention-only Transformer." We do not aim to fully replicate their results, Rather, we perform patching experiments to localize the important attention heads for the purpose of evaluating variants of activation patching.
Experimental setup In Heimersheim Janiak (2023), a Python docstring completion instance consists of the following prompt:
5The model is available in the Transformerl ens library under the name attn-only-4] (Nanda & Bloom,
2022).
16

---

Published as a conference paper at ICLR 2024
def randø (self, rand1, rand2, A_def, B_def, C_def, rand3):
н** rand4 rand5 rand6
ppram A_def: rand7 rand8
param B_def: randg rand10
pparam
where rand's are random single-token English words and the goal is to complete the prompt with C_def. Heimersheim & Janiak (2023) finds that the 4-layer model solves the docstring task with accuracy of 56% and the logit difference is 0.5.
Following their approach, we run activation patching on all attention heads, across all token posi- tions. This is more fine-grained than what we did for the IOI circuit, since the outcome would also highlight the token positions that matter for the important heads.
We apply corruption to the C_def token. For STR, it is replaced randomly by single-token English word in the same way specified in Heimersheim Janiak (2023).
Experimental results We take 200 instances of the doestring completion task, perform activation patching by positions and compute the patching effects. We report all position-head pairs with patching effect 2 standard deviations away from the mean. We find that the detections mostly the position of C_def and the last token of the prompt. The details are given in Table 2.
Corruption
Metric
At the position of C def
At the last position
STR
Logit difference Probability
0.0, 0.1, 0.5
2.3, 3.0, 3.5,3.6
STR
3,0, 3.6
STR
KL divergence
0.0, 0.1, 0.5, 2.2
2.3, 3.0, 3.6
GN t
Logit difference
0.5
1.4, 1.5, 2.2, 2.3, 3.0, 3.6, 3.7 3.0, 3.6
GN
Probability
GN
KL divergence
2.2, 2.3, 3.0, 3.5, 3.6
Table 2: Detections from activation patching of attention heads by position the Python doc- string completion task. Also detects two early-layer heads active at other positions and four negative heads active at the last position, which we omit here.
We again find that the localization outcomes are sensitive to the choice of corruption method and evaluation metric. The results of GN appear quite noisy, except when using probability as the metric. On the other hand, remark that 3.0 and 3.6 are consistently highlighted across metrics and methods. In fact, they are typically assigned the largest patching effects (at the last position). This appears consistent with the result of Heimersheim & Janiak (2023), where 3.0 and 3.6 are found to be directly responsible for moving the C_def token.
F RESULTS ON THE GREATER-THAN CIRCUIT IN GPT-2 SMALL
Hanna et al. (2023) In this section, we study the greater-tan task, specified below, and perform activation patching on the attention heads GPT-2 small. In this setting, the prior work by Hanna et al. (2023); Conmy et al. (2023) show that model computation is fairly localized in this setting and provide a set of circuit discovery results. We remark that we do not attempt to replicate the circuit discovery results here, but rather to evaluate whether activation helps with localizing certain important model components.
Experimental setup Following Hanna et al. (2023), an instance the greater-than task consists of an incomplete sentence of the template: "The <noun> lasted from the year XXYY to the year XX", where <noun> is single-token word and XX and YY are two-digit numbers. For example, "The war lasted from year 1745 to 17". The goal is to complete the prompt with integer greater than XX (in this case, 45). Across several metrics, Hanna et al. (2023) shows that GPT-2 small performs well on this task.
17

---

Published as a conference paper at ICLR 2024
We focus the role of attention heads in our study. To perform corruption, ensure that the year XXYY are tokenized as [XX][YY] by filtering out years and numbers that do not conform to the constraint. GN corruption adds noise to the token embedding of YY. Following Hamna et al. (2023); Conmy et al. (2023), STR corruption replaces YY by 01. The probability metric, in this setting, defined the sum of probabilities of the years greater than YY. The logit difference metric is defined as the sum of logits of the years greater than YY minus the sum of logits of the years less than YY.
We perform activation patching on the attention heads outputs over all token positions.
Experimental results We find significant difference between the results achieved by GN and STR. In fact, the set of heads that localized by the methods are mostly disjoint. Specifically, GN appears to give extremely noisy results that are not in line with the findings of Hanna et al. (2023); Conmy et al. (2023). The details are given in Table
Corruption
Metric
Positive
Negative
STR
Logit difference Probability
6.9, 7.10, 8.11, 9.1, 10.4 7.10, 8.11, 9.1 6.9, 7.10, 8.11,
STR
STR
KL divergence
10.7
GN
Logit difference
0.9, 7.10, 80, 9.1, 10.4 5.5, 6.1, 6.9, 7.10, 7.11, 8,8, 9.1
6.1, 8.6, 9.5
GN
Probability
GN
KL divergence
5.5, 6.1, 6.9, 7.10, 7.11, 8.8, 9.1
5.9, 7.6
Table 3: Detections from activation patching on attention heads for the greater-than task in GPT-2 small, averaged across 300 prompts.
The results from STR are fairly reasonable as the heads 6.9, 7.10, 8.11, 9.1 are also discovered by Hanna et al. (2023); Conmy et al. (2023), using more sophisticated methods. In contrast, the heads discovered by GN corruption share little overlap with STR, except 7.10 and 9.1. From visualizations, also see that the plots for GN experiments are fairly noisy and do not yield much localization at all (Figure 9). On the other hand, the plots from STR are easily interpretable (Figure 10).
Probubilty
Logit D'firence,
GN
(a) Probability as the metric
(b) Logit difference as the metric
(c) KL divergence as the metric
Figure 9: The effects of patching attention heads in GPT-2 small the greater-than task, using GN corruption. We see that the results are fairly noisy and do not appear to localized.
G WHICH TOKENS TO CORRUPT MATTERS
In this section, we revisit the implementation of corruption methods in the setting of IOI (Wang al., 2023).
Previously in our STR experiments in Section and Section 4, the S2 token was corrupted by exchanging with 10. Similarly, in GN, add noise the token embedding of IO. We notice that
18

---

Published as a conference paper at ICLR 2024
Loght Oiferrnce, SIR
0.0
(a) Probability the metric
(b) Logit difference as the metric
(c) KL divergence as the metric
Figure 10: The effects of patching attention heads in GPT-2 small on the greater-than task, using STR corruption. This gives clearly localized results.
the localization results from this approach miss at least 2 out of the 3 Name Mover (NM) Heads (Table 1); they directly contribute to the logit of IO as found by Wang et al. (2023). In particular, all combinations of metric and method would míss out on 9.6 and 10.0 (Table 5).
We show that by varying exactly which tokens to corrupt, the NMs can be discovered, too
Experimental setup We consider the IOI setting (Wang et al., 2023) using STR and GN corruption for localizing attention heads. Here, we corrupt the SI and IO tokens. For STR, we simply replace S1 and IO by two random unrelated names. In both STR and GN experiments, the S2 token remains the same as Xelean-
We perform activation patching across all attention heads. We apply logit difference, probability and KL divergence the metric. All the results are averaged across 500 sampled IOI sentences.
Experimental results We find that most combinations of metrics and methods are able to notice all the NMs, when corruption applies to S1 and IO. We give the exact detections below and categorize them into positive and negative for simplicity.
Corruption
Metric
Positive
Negative
STR
Logit difference Probability KL divergence
9.6, 9.9, 10.0
10.7, 11.10
STR
9,9 9.6, 9.9, 10.0
STR
10.7, 11.10
GN
Logit difference
9.6, 9.9, 10.0
10.7, 11.10
GN
Probability
9.6, 9.9, 10.0
GN
KL divergence
9.6, 9.9, 10.0
10.7, 11.10
Table 4: Detections from activation patching by corrupting S1 and 10 in IOI. The Name Mover Heads are 9.6, 9.9, 10.0 and the Negative Name Mover Heads are 10.7 and 11.10, based on Wang et al. (2023). No other heads, including the S-Inhibition Heads, are noticed with this approach.
First, we observe that this corruption seems to precisely target the NMs and their Negative counter- parts. Intuitively, this natural. Wang et al. (2023) finds that NMs write in the direction of the logit of the name (IO or S), whereas the Negative NMs do the opposite. Patching the clean activations of NMs recover such behavior.
Second, confirm our finding that probability will miss the Negative NM; see Figure 11 for the plots.
Overall, the experiment suggests that exactly which token is corrupted affects the localization out- comes. Intuitively, varying the corrupted token(s) allows activation patching to trace different infor- mation within the model's computation paths; Section for discussion.
19

---

Published as a conference paper at ICLR 2024
Probsbilty.y
Logit STR
102
5003
-8303
Midd (a) Probability the metric
(b) Logit difference as the metric
(c) KL divergence as the metric
Figure 11: The effects of patching attention heads in GPT-2 small on IOI sentences, using STR corruption S1 and IO.
H FURTHER DETAILS ON FACTUAL ASSOCIATION
The plots of subsection Appendix H.1 to H.3 are produced on GPT-2 XL and with the PARIEDFACTS as dataset. Following that, we also experiment with the GPT-2 large (Radford al., 2019) and GPT-J (Wang & Komatsuzaki, 2021) model in Appendix H.4 and H.5.
H.1 PLOTS ON MLP PATCHING AT THE LAST SUBJECT TOKEN IN GPT-2 XL
First, we perform single-layer patching of MLP activation at the last subject token and examine the effects in Figure 12. We observe that the experiment suggests weak or peak middle MLP layers, across metrics and corruption methods.
Probabilityy GR, window
Probability, STR, window
Logit difference, GN, window -
Logit difference, STR, window -1
0.25
035
042
0.35
0.55
(a) Probability (GN) (b) Probability (STR) (c) Logit difference (GN) (d)
Logit
difference
(STR)
Figure 12: Patching single MLP layers at the last subject token in GPT-2 XL. on factual recall prompts. None of them suggest strong peak at the middle MLP layers.
Also, see Figure 13 and Figure 14 for plots with sliding window size of 3 and 10. Again, activation patching applied the MLP activations at the last subject token. We find again that GN yields significantly more pronounced peak.
H.2 PLOTS ON MLP PATCHING AT ALL TOKEN POSITIONS IN GPT-2 XL
See Figure 15-Figure 19 to plots on MLP patching all token positions in GPT-2 XL. across win- dow sizes of 3,5, 5, 10. We observe that the right-side plots, using probability as the metric, highlights the last subject token as important. contrast, the left-side figure using logit different does it to lesser degree.
H.3 PLOTS ON SLIDING WINDOW PATCHING IN GPT2-XI.
We provide further plots from our experiment that compares the sliding window patching with indi- vidual patching aggregated via summation windows. See Figure 20-Figure 23.
20

---

Published as a conference paper at ICLR 2024
Probability, window size
Logit difference, window size
0.08
STR
GN
0.25
GN
0.06
0.04
0.1
0.02
0.05
(a) Prohability as metric
(b) Logit difference as the metric
Figure 13: MLP patching effects at the last subject token position in GPT-2 XL on factual recall prompts, with window size 3.
Probability, window size 10
Logit difference, window size 10
STR
GN
GN
0.05
0.1
Probability as metric
(b) Logit difference as the metric
Figure 14: MLP patching effects for factual recall at the last subject token position in GPT-2 XL on factual recall prompts, with window size of 10.
H.4 PLOTS ON ACTIVATION PATCHING OF MLP LAYERS ON GPT-2 LARGE
We perform activation patching on MI.P layers GPT-2 large in the factual association setting. Following experiments in Section 3.1, focus the effects at patching the ML.P activation of the last subject token. We validate the high-level finding of Section 3.1, where we observe the disparity of GN and STR applied MLP activation the factual prediction setting. In particular, GN gives more pronounced concentration at marly-middle MLP layers. We apply sliding window patching of size 3 and 5; see Figure 24 and Figure 25 for the resulting plots.
H.5 PLOTS ON ACTIVATION PATCHING OF MLP LAYERS IN GPT-J
We perform activation patching on MLP layers of GPT-J (Wang & Komatsuzaki, 2021) in the factual association setting. We patch the MLP activations across all token positions and verify that proba- bility tends to highlight the importance of the last subject token than logit difference. We focus a sliding window patching of size 5 and the plots are given in Figure 27 (GN) and Figure 26 (STR). This complements results in Section 4.
I FURTHER DETAILS ON IOI CIRCUIT DISCOVERY
I.1 DETAILED PLOTS ON ACTIVATION PATCHING
We now provide the detailed plots from the activation patching experiments on the IOI circuit dis- covery task (Wang al., 2023); Figure 28 and Figure 29.
21

---

Published as a conference paper at ICLR 2024
10
0.05
0.1
30
First subject taken
18d subject takens
Lost subject token
The subroquantt taken
Further takens
Last taken
First subject token.
Mid subject tokons
Lost subject token
The subsequert takon
Furtber takers
Last token
(a) Logit difference (GN)
(b) Probability (GN)
Figure 13: Activation patching on MLP across layers and token positions in GPT-2 XL on factual recall prompts. Apply GN corruption and sliding window of size 3.
0.02
0.2
20
-0.01
-0.02
First subject toben
Nid subject Toteess
Further toklens
First subject talens
Mid subject tolents
Further takens
subject Toden
Tubsoquent Toker
taken
subject token
Subsequert toeen
tobon
(a) Logit difference (STR)
(b) Probability (STR)
Figure 16: Activation patching MLP across layers and token positions in GPT-2 XL on factual recall prompts. Apply STR corruption and a sliding window of size 3.
1.2 DETAILS ON DETECTIONS
We provide detailed list of detection from attention heads patching in the IOI circuit setting (Sec- tion 3); see Table 5.
Corruption
Metric
Negative heads
Positive heads
STR
Logit difference Probability
10.7, 11.10
5.5, 7.9, 8.6, 8. 80, 9.9
STR
10.7 10.7, 11.10
5.5, 8.6, 8. 80, 9.9
STR
KL divergence
5.5, 8.6, 8.10, 9.9
GN
Logit difference
10.7, 11.10 0.10, 10.7, 11.10
5.5, 7.9, 8.6, 8.10, 9.9
GN
Probability
3.0, 5.5, 7.9, 8.6
GN
KL divergence
0.10, 10.7, 11.10
5.5, 7.9, 8.6
Table 5: Detailed results from attention heads patching in GPT-2 small IOI sentences. A head detected if the patching effect two standard deviation from the mean effect. Negative heads are heads with negative patching effects, suggesting they hurt model performance.
I.3 DETAILED PLOTS ON FULLY RANDOM CORRUPTION
We provide the plots on fully random corruption, termed PARC in Wang et al. (2023). We perform activation patching all attention heads, using both probability and logit difference as the metric
22

---

Published as a conference paper at ICLR 2024
0.05
0.1
30
-0.05
-0.2
First subject taken
18d subject takens
Last subject token
The subroquantt taken
Further takens
Last taken
First subject token.
Mid subject tokons
Lost subject token
The subsequert takon
Furtber takers
Last taken
(a) Logit difference (GN)
(b) Probability (GN)
Figure 17: Activation patching on MLP across layers and token positions in GPT-2 XL on factual recall prompts. Apply GN corruption and sliding window of size 5.
0.3
0.1
0.2
-0.05
-0.2
-0.1
-0.3
Frat subject tokonn
140 subject toleens
Last subject taiten
The subcoqueet taken
Rorther tokens
Last token
First subject tokeon
Mid subject Tolards
Last subject token
The subrequentt takon
Further tokens
Lost tokon
(a) Logit difference (GN)
(b) Probability (GN)
Figure 18: Activation patching on MIP across layers and token positions in GPT-2 XI. on factual recall prompts. Apply GN corruption and sliding window of size 10.
in order to draw contrasts between them. See Figure 30. In particular, we notice that there is negative head in the plot. This is natural and totally expected, as we explained in Section 4.
J DATASET SAMPLES
Factual data We list a few dataset examples from the PAIREDFACTS dataset used in the factual recall experiments in Figure 31.6 All the prompts known true facts.
IOI circuit The detailed templates of constructing the Pior data distribution can be found in Ap- pendix of Wang al. (2023). We perform the same procedure of generating the IOI data by simply reusing their original code.
"The full datasct is available
23

---

Published as a conference paper at ICLR 2024
0.04
0.2
0.02
Layer
20
-0.02
-0.2
Fist subject talonn
101d subject talians
Last subject talton
Further takents
Last takron
Fost subject tokon
Mid subject takand
The subsequent tation
Further tokens
Last takan
subsequent taken
pubject takan
(a) Logit difference (STR)
(b) Probability (STR)
Figure 19: Activation patching MLP across layers and token positions in GPT-2 XL. Apply
STR corruption and sliding window of size 10.
Probability, STR, window
Probability, GN, window =
adding
adding sliding
siding
0.02
0.015
0.005
0
(a) STR corruption
(b) GN corruption
Figure 20: MLP patching effects, sliding window summing up single-layer patching at last token position in GPT-2 XL on factual recall prompts, with window size of Apply probability as the metric.
Logit difference, STR, window =
Logit difference, GN, window
0.3
adding siding
sliding
0.2
0.15
0.
0.05
20
40
20
40
(a) STR corruption
(b) GN corruption
Figure 21: MLP patching effects, sliding window vs summing up single-layer patching at last token position in GPT-2 XI. factual recall prompts, with window size of 5. Apply logit difference as the metric.
24

---

Published as a conference paper at ICLR 2024
Probability, STR, window 10
Probability, GN, window = 10
adding siding
adding sliding
0.15
0.3
0.02
0.01
0
40
(a) STR corruption
(b) GN corruption
Figure 22: MLP patching effects, sliding window vs summing up single-layer patching at last token position in GPT-2 XL on factual recall prompts, with window size of 10. Apply probability as the metric.
Logit difference, STR, window = 10
Logit difference, GN, window 10
adding sliding
siding
0.3
0.5
0.3
20
40
20
40
(a) STR corruption
(b) GN corruption
Figure 23: MLP patching effects, sliding window vs summing up single-layer patching at last token position in GPT-2 XL on factual recall prompts, with window size of 10. Apply logit differ- ence the metric.
Probability, window size 3
Logit difference, window size 3
STR
STR
0.06
0.04
0.1
0.05
10
20
30
0
10
20
(a) Probability the metric
(b) Logit difference the metric
Figure 24: MLP patching effects at the last subject token position in GPT-2 large on factual recall prompts, with window size of 3.
25

---

Published as a conference paper at ICLR 2024
Probability, window size
Logit difference, window size
0.25
STR
0.06
GN
GN
0.2
10
30
Probability as metric
(b) Logit difference as the metric
Figure 25: MLP patching effects at the last subject token position in GPT-2 large factual recall prompts, with window size
0.04
0.02
-0.1
0.02
20
-02
First subject takan
Midd subject talians
Last subject token
The subuequent taken
Further takons
Last taken
First subject taken.
Mid subject takens
Last subject taken
Purther tokere
Lest tokon
subeegueett taken
(a) Logit difference (GN)
(b) Probability (GN)
Figure 26: Activation patching on MLP across layers and token positions in GPT-J on factual recall prompts. Apply STR corruption and sliding window of size
0.04
0.1
-0.00
-0.1
20
-0.04
First subject Loban
Mid subjectt taleeni
Last subject token
The subsequent taleen
Forther takens
Last rosen
First subject toeen
Mid subject tolenns
Last subject token
The subsequert taben
Farther tokers
Last toben
(a) Logit differenec (GN)
(b) Probability (GN)
Figure 27: Activation patching on MLP across layers and token positions in GPT-J on factual recall prompts. Apply GN corruption and sliding window of size 5.
26

---

Published as a conference paper at ICLR 2024
Liper
(a) Probability as the metric
(b) Logit difference as the metric
(c) KL divergence as the metric
Figure 28: sentences.
The effects of patching attention heads in GPT-2 small using STR corruption on IOI
Probsbilityco GM
Logit Difference, GN
KL. GN
lage
-11
-004
-24
-009
-24
Hend
(a) Probability as the metric
(b) Logit difference as metric
(c) KL divergence as metric
Figure 29: sentences.
The effects of patching attention heads in GPT-2 small using GN corruption on IOI
0.02
0.4
Layer
-02
64
-0.00
Head
Head
(a) Probability as the metric
(b) Logit difference as the metric
Figure 30: The effects of patching attention heads in GPT-2 small using fully random corruption on IOI sentences, with S1, S2 and IO replaced by three random names (denoted by PABC in Wang al. (2023)).
27

---

Published as a conference paper at ICLR 2024
"pair":
"piir"
"llonus hagner professionally plays the sport of", "Don Shula professionally plays the sport of
'Nii MotionPlus developed by
Chhomebook Pixel is developed by"
"answer";
aanswer":
baseball".
Nintendo", Google"
"ootbbll""
"lengch";
"length": 8,
"category"; "sthletes'
"category": "developers'
"pair"s
"peir"s
"Schreckhorn belongs to the continent of*. "ffghanistan belongs the continent of*
"The Eiffel Tower is in the city of Kinnakuuji Temple is the of*
"answer": [
"ansver:: Paris".
EErrpe". Asia"
Kyoto"
"length"; 9, "category": "continents"
"catogory"; "city_landkarks".
"length": 11
Sample text prompts from the PAIREDFACTS dataset. The length field refers to the sequence length of the prompt under GPT-2 tokenizer.
Figure 31:
28


Reviews:
[{"review_id": "RlN8grsbnW", "reviewer": "Reviewer_4A2t", "summary": "This paper delves into the realm of mechanistic interpretability, a burgeoning and promising domain within large language models. It primarily centers on activation patching, aiming to identify activations that hold a causal influence over the output. A notable aspect of this work is its pioneering stance in systematically studying the generation of corrupted prompts and the evaluation metrics for patching effects, which previously lacked standardization. Specifically, the paper scrutinizes two methodologies for generating corrupted prompts: 1) Gaussian Noising (GN) and 2) Symmetric Token Replacement (STR). Furthermore, it explores two evaluation metrics: 1) probability and 2) logit difference, alongside investigating the impact of sliding window patching.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The endeavor to understand the internal mechanisms of large language models through activation patching is pivotal. This paper stands out by empirically examining various methodologies, bridging the gap where variations across different papers have made it challenging to ascertain the more effective approach. By embarking on this comprehensive investigation, the paper makes a substantial contribution towards standardizing methods, which is invaluable to the mechanistic interpretability community.", "weaknesses": "I am not very familiar with the details of the existing activation patching methods. Therefore, I am not sure whether the methods included in the paper are diverse and representative enough for the mechanistic interpretability community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G6BViLzIzc", "reviewer": "Reviewer_bjM9", "summary": "In this work, the authors explore how the method of corruption and type of evaluation can cause conflicting take-aways from activation patching. Specifically, they look at two ways to add corruption to a token embedding: gaussian noise (adding noise to the embedding) or symmetric token replacement (replacing a token with a semantically related token). They find that gaussian noise can cause the input to be OOD, thus breaking the internal mechanism. When looking at logit difference vs. probability, they find that probability can overlook negative model components. Finally they look at sliding window patching, and find that it can inflate logit plots.", "questions": "How much does different substitutions for STR or different samples of noise (for gaussian noise) change the interpretation? Are the methods at least consistent within themselves?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I like the motivation of this paper: I think its important to understand how hyperparameters can change the results of interpretability methods.\n\nThe authors go over several different types of hyperparameter (corruption method, evaluation method, sliding window) and convincingly show that these design choices can result in different interpretations. I liked the Name Mover analysis, which made it easier to understand how gaussian noise could be causing issues for activation patching. \n\nFinally, I appreciated Section 6, which gives recommendations on how activation patching should be performed.", "weaknesses": "My biggest concern is relevance to the community: other than Meng et. al, are there other papers using gaussian noise? It's not clear to me that this is a wide-spread issue. \n\nMoreover, the recommended course of action (STR) can be more difficult to actually implement (as it requires having semantically similar substitutions). To be of most relevance to the community, it would be great if the authors suggested an approach that had the flexibility of gaussian noise without introducing as much bias.\n\nClarity: Table 1 and in particular the part about negative detection was a bit hard to parse. I would make the discussion around those results more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CYZ9PZxGoG", "reviewer": "Reviewer_PYwb", "summary": "In this paper the authors study the effects of different activation patching techniques on the mechanistic model interpretability and make several recommendations. They study and compare the selection of the hyperparameters, evaluation metrics and input corruption techniques for activation patching based model interpretability. The paper studies two approaches, Gaussian Noise (GN) and Semantic Token Replacement (STR), for corrupting the inputs. Based on those studies the paper recommends using STR input corruption technique since it produces in-distribution samples as opposed to the GN approach which produces OOD samples. In terms of the evaluation metrics, it recommends logit difference since it is more granular and allows to detect model activations (components) that have negative impact on model performance.", "questions": "1) What are some of the novel contributions of the paper ? \n2) How was the content of Table 1 computed ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The abstract and introduction are well and clearly written. The problem statement and contributions are easy to follow from those 2 sections.\n2) The paper performs thorough experimentation on sliding window techniques, localizing factual recall and circuit discovery.", "weaknesses": "1) The work overall is very interesting but it feels a bit light on the novelty. Perhaps proposing additional novel methods for input corruption and improvements for the activation patching techniques can help to increase the novelty in this paper.\n2) The way the paper is written it might be a good fit for a workshop.\n3) Some terminology could be explained in the paper. E.g. Name Mover\nIt seems that the paper requires prior knowledge of another paper Wang et.al. Some concepts such as: 0.10 negative detection is not very clear.\n4) It might be good to describe clearly what 0.10 negative detection is under the `Negative detection of 0.10 under GN` section.  \n\n\n\nMinor comments:\n1) “use its own the method of generating” -> “use its own method of generating” ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)