Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching
Authors: Aleksandar Makelov
Georg Lange
Atticus Geiger
Neel Nanda
Venue: ICLR 2024 poster (2024)
Keywords: Mechanistic Interpretability
Natural Language Processing
Large Language Models
Primary area: visualization or interpretation of learned representations
TL;DR: We show how activation patching can hallucinate meaningful subspaces in a language model by activating dormant pathways.

Abstract: Mechanistic interpretability aims to attribute high-level model behaviors to specific, interpretable learned features. It is hypothesized that these features manifest as directions or low-dimensional subspaces within activation space. Accordingly, recent studies have explored the identification and manipulation of such subspaces to reverse-engineer computations, employing methods such as activation patching. In this work, we demonstrate that naïve approaches to subspace interventions can give rise to interpretability illusions.

Specifically, even if patching along a subspace has the intended end-to-end causal effect on model behavior, this effect may be achieved by activating \emph{a dormant parallel pathway} using a component that is \textit{causally disconnected} from the model output.
We demonstrate this in a mathematical example, realize the example empirically in two different settings (the Indirect Object Identification (IOI) task and factual recall), and argue that activating dormant pathways ought to be prevalent in practice.
In the context of factual recall, we further show that the illusion is related to rank-1 fact editing, providing a mechanistic explanation for previous work observing an inconsistency between fact editing performance and fact localisation.

However, this does not imply that activation patching of subspaces is intrinsically unfit for interpretability.
To contextualize our findings, we also show what a success case looks like in a task (IOI) where prior manual circuit analysis allows an understanding of the location of the ground truth feature. We explore the additional evidence needed to argue that a patched subspace is faithful.

Published as a conference paper at ICLR 2024
Is THIS THE SUBSPACE YOU ARE LOOKING FOR? AN INTERPRETABILITY ILLUSION FOR SUBSPACE ACTI- VATION PATCHING
Aleksandar Makelov & Georg Lange"
Atticus Geiger
SERI MATS
Linguistics Department Stanford University Stanford, CA 94305, USA a att esticcsgesstanford edu
Constellation Offices
Berkeley, California, USA
aleksandar. r. mwkelakella.. woakkelllgaaaa... ii com
georglange4 com
Neel Nanda
nee 2nanda 2 gmail com
ABSTRACT
Mechanistic interpretability aims attribute high-level model behaviors to spe- cific, interpretable learned features. It is hypothesized that these features manifest as directions or low-dimensional subspaces within activation space. Accordingly, recent studies have explored the identification and manipulation of such subspaces to reverse-engineer computations, employing methods such as activation patching. In this work, we demonstrate that naive approaches to subspace interventions give rise to interpretability illusions.
Specifically, even if patching along a subspace has the intended end-to-end causal effect model behavior, this effect may be achieved by activating dormant parallel pathway using a component that is causally disconnected from the model output. We demonstrate this mathematical example, realize the example em- pirically in two different settings (the Indirect Object Identification (IOI) task and factual recall), and argue that activating dormant pathways ought to prevalent in practice. In the context of factual recall, we further show that the illusion is re- lated to rank-1 fact editing, providing a mechanistic explanation for previous work observing an inconsistency between fact editing performance and fact localisation. However, this does not imply that activation patching of subspaces is intrinsically unfit for interpretability. To contextualize our findings, we also show what a suc- case looks like in task (IOI) where prior manual circuit analysis informs an understanding of the location of a feature. We explore the additional evidence needed to argue that patched subspace is faithful.
INTRODUCTION
Recently, large language models (LLMs) have demonstrated impressive (Vaswani et al. 2017] De- vlin al. 2019 OpenAl 2023 Radford al. 2019 Brown et al.) (2000), and often surprising (Wei et al. 2022). capability gains. However, they are still widely considered "black boxes": their suc- and failures remain largely mystery. It is thus an increasingly pressing scientifie and practical question to understand what LLMs learn and how they make predictions.
This question is in the realm of machine learning interpretability, pery concept. Desiderata for interpretability are often not stated precisely, and it is easy to develop an illusory perception of understanding (Lipton) 2016 Adebayo et al., 2018 Bolukbasi al. 2021). Mechanistic interpretability (MI) is subfield of interpretability that seeks to avoid these pitfalls by developing a rigorous low-level understanding of the learned algorithms behind a model's computa- tions. Specifically, MI frames these computations as collections of narrow, task-specific algorithms
important but notoriously slip-
"Equal contribution; full version /ttp://////....oo///////////////

---

Published as a conference paper at ICLR 2024
-circuits (Olah al. 2020 Geiger al. 2021| Wang al. 2023 whose operations grounded in concrete, atomic building blocks akin to variables in a computer program (Olah 2022) or causal model (Vig al. 2020 Geiger al. 2023a). MI has found applications in several downstream tasks: removing toxic behaviors from a model while otherwise preserving performance by mini- mally editing model weights (Li et al., 2023b), changing factual knowledge encoded by models in specific components to c.g. enable more efficient fine-tuning in changing world (Meng et al.) 2022a), improving the truthfulness of LLMs at inference time via efficient, localized inference-time interventions in specific subspaces (Li et al., 2023a) and studying the mechanics of gender bias in language models (Vig et al.] 2020)).
A central question in MI is: what is the proper definition of "building blocks (Olah, 2022)? Many initial mechanistic analyses have focused on mapping circuits to collections of model components Wang et al. 2023; Heimersheim & Janiak). A workhorse of these analyses is activation patch- ing Vig et al. 2020; Geiger et al., 2020; Meng al., 2022a Wang et al.) 2023), which swaps component activations between examples and looks for task-relevant changes in model outputs.
However, a plethora of empirical evidence suggests that the features LLMs represent and use lie in
linear subspaces of component activations (Nanda 2023a Li et al. 2021 Abdou et al. 2021 Grand iet al. 2018). Furthermore, phenomena like superposition and polysemanticity (Elhage et al. 2022) suggest that these subspaces not easily enumerable, like individual neurons, but identified with a continuous space of arbitrary rotations so searching for them can be non-trivial. This raises the question: can we carry over the success of activation patching from component-lever analysis to finding the precise subspaces corresponding to features?
rather be
Indeed, recent works such as Geiger et al. (2023b); Wu et al. (2023) have begun identifying inter- pretable subspaces using gradient descent, with training objectives created using subspace activation patching. While this kind of end-to-end optimization has promise, we show that is prone a kind of interpretability illusion. Specifically. instead of robustly localizing a variable that is used by the model wide range of contexts, setting the value of a subspace can fabricate such a variable by activating dormant pathway in the model via exploiting a causally disconnected feature (Figure D. Our results suggest this effect is strongest when overfitting to a small dataset. Our contributions can be summarized as follows:
In Section 3 we construct a distilled mathematical example of the illusion;
In Section we find a realization of this phenomenon 'in the wild', in the context of the indirect object identification task (Wang et al. (023), where a 1-dimensional subspace of MLP activations found using DAS (Geiger et al. (023b) can seem encode position information about names in the sentence;
To contextualize our results, in Section subspaces that faithfully represent a feature in a model's computation. Specifically, we find I-dimensional subspace encoding the same position information in the IOI task, and validate its role in model computations via experiments beyond end-to-end causal effect. We argue that activation patching on subspaces of the residual stream safer and less prone to illusions.
also show how DAS can be used to find
Going beyond the IOI task, in Section 6 also exhibit this phenomenon in the setting of fact editing (Meng et al. (2022a). We show that 1-dimensional activation patches imply equivalent rank-1 model edits (Meng et al. (022a)). In particular, this shows that rank-1 model edits can also be achieved by activating a dormant pathway in the model, without relying on the presence of a fact in the weight being edited. This suggests mechanistic explanation for the observation of (Hase et al.) 2023) that rank-1 model editing works regardless of whether the fact is present in the weights being edited.
we end with arguments and evidence for why this interpretability illusion in Section 7
ought be prevalent in real-world language models.
2
RELATED WORK
'also known as 'interchange intervention' (Geiger et al. 2020) and related to, but distinct from 'resample ablation' (Chan al.) or "causal tracing' (Meng et al. 2022a
2

---

Published as a conference paper at ICLR 2024
Discovering and causally intervening on representations with activation patch- ing. Activation patching (Vig et al., 2020: Geiger et al. 2020) is a widely used causal intervention, whereby the model is on an input A, but chosen activations are "patched in' from input B. Motivated by causal mediation analysis (Pearl, 200T), ac-
dormant ditseriio
direction to patch along (V)
erllegonnal complement p-
example to patch from
example to patch into
bausally discogneeted direction
tivation patching has been used to localize model components causally involved in var- ious behaviors, such as gender bias (Vig et al.), factual recall (Meng et al. 2022a).
result of the putch
multiple choice questions (Lieberum et al.
2023). arithmetic (Stolfo al. 2023) nat-
ural language reasoning (Geiger et al.
2021 Wang et al.] 2203; Geiger al.
The key mathematical phenomenon behind the activation patching illusion (see Appendix Figure 26] for a step-by-step explanation). By setting the pro- jection of an example's activation (green, right) along vector (red, top-right) to equal another's s (green, left) projection, we obtain a vector orthogonal to both ac- tivations (blue, down). This can give counterintuitive results when the original and new directions have fun- damentally different roles in a model's computation.
Figure 1:
2023b Wu et al.] 2023). code (Heimer-
sheim & Janiak), and (in certain regimes)
iopic/sentimen//ttlee of free-form natural language (Turner al. 2023).
Activation patching is an area of active re- search, and many recent works have ex- tended the method, with patching paths be- tween components (Goldowsky-Dill et al.] 2023), automating the finding of sparse sub- graphs (Conmy et al., 2023). fast approxi-
mations (Nanda 2023b). and automating the verification of hypotheses (Chan et al. In particular, wide range of interprotability work (Mikolov et al.) 2 Conneau et al.] 2011 8 Tenney et al. 2019:]
Burns et al. 2022, Nanda et al. 2023) suggests the linear representation hypothesis: models encode features as linear subspaces of component activations that can be arbitrarily rotated with respect to
the standard basis (due phenomena like superposition, polysemanticity (Arora et al.) 2018 Elhage
iet al. 2022) and lack of privileged bases (Smolensky) 1986 Elhage et al.] (021)) Motivated by this,
recent work such as Geiger et al. 2023b); Wu al. (2023); Lieberum et al. (2023) has general-
ized activation patching to operate only on linear subspaces of features rather than patching entire components (heads, layers and neurons). Our work contributes to this research direction by demon- strating both (i) a common illusion to avoid when looking for such subspaces and (ii) detailed case study of successfully localizing binary feature to a 1-dimensional subspace.
Factual recall. A well-studied domain for discovering and intervening on learned representations is the localization and editing of factual knowledge in language models (Geva et al. 2023 Meng let al. [022b] Wallat et al. 2020: Dai et al.] 20222 Hernandez et al. 2023). A work of particular note is Meng et al. (2022a). which localizes and edits factual information with a rank-1 intervention on model weights. However, recent work has shown that rank-1 1 editing can work even weights where the fact supposedly is not encoded (Hase et al., 2023). and that editing a single fact often fails to have its expected common-sense effect on logically related downstream facts (Cohen al. 2023 Zhong et al. 2023). We contribute to this line of work by showing formal and empirical connection between activation patching along I-dimensional subspaces and rank-1 I model editing. In particular, rank-1 model edits can work by creating a dormant pathway of an MLP layer, regardless of whether the fact stored there. This provides mechanistic explanation for the discrepancy observed in
Hase et al. (2023).
Interpretability illusions. Despite the promise of interpretability. common theme in the field is identifying ways that techniques may lead to mislcading conclusions about of model behavior (Lipton) (016). In computer vision, Adebayo et al. (2018) show that popular class of pixel attri- bution methods is not sensitive to whether or not the model used to produce is has actually been trained or not. Geirhos et al.] (2023), the authors show how a circuit can be hardcoded into a learned model so that it fools interpretability methods; this bears some similarity to our illusion. In natural language processing, Bolukbasi et al. (202T) show that interpreting single neurons with
3

---

Published as a conference paper at ICLR 2024
maximum activating dataset examples may lead to conflicting results across datasets due to subtle polysemanticity (Elhage et al. 2022).
3 A CONCEPTUAL VIEW OF THE ILLUSION
Activation patching. Activation patching (Vig et al. 2020 Geiger et al. 2020 Wang et al. 2023 Chan al.) is an interpretability technique that intervenes upon model components, forcing them to take on values they would have taken if different input were provided. For instance, consider a model that has knowledge of the locations of famous landmarks, and completes e-g. the sentence A = 'The Eiffel Tower is in' with 'Paris'. How can we find which component of the model is responsible for knowing that 'Paris' is the right completion? Activation patching approaches this question by (i) running the model on A, (ii) storing the activation of a chosen component c, and (iii) running the model on e.g. B 'The Colosseum is in', but with the activation of e taken from A. we find that the model outputs 'Paris' instead of 'Rome' in step (iii), this suggests that component is important for the task of recalling the location of a landmark.
Subspace Activation Patching. The linear representation hypothesis proposes that linear subspaces of vectors will be the most interpretable model components. To search for such subspaces, we can adopt natural generalization of full component activation patching which only patches the values of a subspace U (while leaving the projection on its orthogonal complement U- unchanged). This was proposed in Geiger et al. (0233b), and closely related variants appear in Turner al. (2023); Nanda et al. (023): Lieberum et al. (2023).
For the purposes of exposition, we now restrict our discussion to activation patching of 1- dimensional subspace (i.e. direction) represented by a unit vector v. In this case, the subspace U is the 1-dimensional subspace spanned by the vector + If act,, actg R2 are the activations of a model component C on examples A, B and PA = v act PB = v act are their projections along v, patching from A into B along v results in the patched activation
act B
patched = act + (PA - PB)C
(1)
Intuition. When will the update in equation T change the model's output in the intended way? Intu- itively, two properties are necessary: must be activated differently by the two prompts (otherwise PA A PB and the patch has no effect), and v must be causally connected to the model's outputs (otherwise, if e.g. is in the nullspace of downstream model components, changing the activation along won't change model predictions). A direction v faithful to the model's computation will simultaneously have these two properties.
The crux of the illusion is that e may obtain each of the two properties from two unrelated directions in activation space, as shown in Figure Specifically, we can form v = dddssoonecctdd + Vdurmant: where Pdiscommettdd distinguishes between the two prompts, but is in the nullspace of all downstream model components; and Udommnnt can in principle steer the model in the way intended by the patch, but is not activated differently by the two prompts. By patching along the sum of these directions, the variation in the disconnected part activates the dormant part, which then achieves the causal effect; this is illustrated in Figure
Formalization. Let M : X + O be a machine learning model that on input x E X outputs a vector y O of probabilities over set of output classes. Let D be a distribution over X, and C be a component of M, such that for x ~ D the hidden activation of C is a vector e₂ E Rd. For a subspace Ue C Rd we let ug be the orthogonal projection of c, onto Uc- Finally, let ( be the result of running M with the input æ and setting the subspace Uc patched to My.
We say U is causally disconnected if Mu (a) = Mxx) for all w' E U. In other words, setting the value of a causally disconnected subspace to any vector has no effect on model outputs. We say U is dormant if MUc (z) s M(x) with high probability over x, y D. but given x D. there exists u such that (u)+a(x) & M(x). In other words, dormant subspace is approximately
We remark that the illusion also applies higner-dimensional subspaces (see Appendix A.1 for details). By contrast, patching along either of the two components individually has no effect: setting only the projection along Masoomette to any value has no effect on outputs by definition; and patching only along Vooemant will have weak effect because both examples activate it similarly.
4

---

Published as a conference paper at ICLR 2024
Input
Number of examples
Patching
FLDD
Interchange
subspace
accuracy
full MLP
-8%
0.0%
VMLP
46.7%
4.2%
60
VMLP rowspace
13.5%
0.2%
40
VMLP nullspace
0%
0.0%
full residual stream
123.6%
54.8%
20
Vresid
140.7%
74.8%
10.0
-75
5.00
-25
6.0
25
5.0
75
Vresid rowspace
127.5%
63.1% 1%
Viesid nullspace
13.9%
0.4%
Activation projection
Vgrad
111.5%
45.1%
V Vgrad rowspace
106.47
40.6%
Figure 3: Projections of the output of the MLP layer on the gradient direction Vgnd before (blue/orange) and after (green/red) the activation patch along VMLP. Here and elsewhere, 'ABB' denotes prompts where the I0 name comes first, and 'BAB' de- notes prompts where the name comes first.
'gaad nullspace
2.2%
0.0%
Table Effects of activation patching of full compo- nents and 1-dimensional subspaces the IOI task: fractional logit diff (higher means less successful patch; 100% means change) and interchange curacy (fraction of predictions flipped; higher means more successful patch).
causally disconnected on the data distribution, but can have substantial causal effect if set to out of distribution values. We present a minimal concrete example of the illusion in the language of these concepts in Appendix A.3) where we also discuss concrete hypothesis to test for the illusion.
THE ILLLUSION IN THE INDIRECT OBJECT IDENTIFICATION TASK
4.1 PRELIMINARIES
In Wang al. 2023 the authors analyze how the decoder-only transformer language model GPT-2 Small (Radford et al. 2019) performs the indirect object identificaton task. In this task, the model required complete sentences of the form 'When Mary and John went the store, John gave bottle of milk to' (with the intended completion in this case being Mary'). ). We refer to the repeated name (John) as S, the subject, and the non-repeated name (Mary) IO, the indirect object. We use the same model GPT-2 Small, with dataset that spans 216 names, 144 objects (e.g., 'bottle of milk') and 75 places (e.g.. 'store") and three templates, split equally between a train and test set (see Appendix B for more details).
Wang et al. (203)) suggest the model uses the algorithm 'Find the two names in the sentence, detect the repeated name, and predict the non-repeated name" to do this task. In particular, they find set of four heads in layers and - the S-Inhibition heads - that output the signal responsible for not predicting the repeated name. The dominant part of this signal is of the form 'Don't 't attend to the name in first/second position the first sentence' depending on where the S name appears (see Appendix A in Wang al. (2023) for details). This signal is added to the residual stream at the last token position, and is then picked up by another class of heads in layers 9, 10 and 11 the Name Mover heads which incorporate in their queries shift attention the IO and copy it to the last token position, so that it can be predicted (Appendix Figure 21)
WWe follow the conventions of Elhage et al. 2021 when describing internals of transformer models. The residual stream layer & the sum of the output of all layers up 1, and the input into layer k.
5

---

Published as a conference paper at ICLR 2024
4.2 FINDING SUBSPACES MEDIATING NAME POSITION INFORMATION
Given these findings, natural next question is: how, precisely, do the S-Inhibition heads commu- nicate the positional signal to the name mover heads? In particular, 'don't attend the first/second name' is plausibly binary feature represented by I-dimensional subspace. In this subsection, we present several methods look for such a subspace.
Gradient of name mover attention scores. The three name mover heads identified in Wang et al. (2033) will attend to one of the names, and the model will predict whichever name is attended to. The position feature matters mechanistically by determining whether they attend to I0 over S. So we can take the gradient of the difference of attention scores of these heads on the and I0 names. For given IOI example, we expect this gradient will encode the position signal, and some of the name signal. By averaging gradients over the entire distribution, we expect the name signal to average out, and the position signal to be reinforced. Motivated by this, compute direction Vgrad as follows. We take the average gradient of the difference of attention scores of the three name mover heads between the positions where the S and IO names are, average this over 2000 samples from our test distribution, and normalize the resulting vector to have unit & norm.
Distributed alignment search. Instead of taking the gradient in specific attention heads informed by previous work, we can also directly optimize for a direction that mediates the position signal. This the approach taken by DAS (Geiger al. 2023b). In our context, DAS optimizes for activation subspace which, when activation patched from prompt B into prompt A, makes the model behave as if the relative position of the 10 and names in the sentence as in prompt B. Specifically, we train DAS to maximize the difference between the logits of the name that should be predicted this patch succeeded, and the other name. We find that relatively small sample sizes (e.g.. 500) are sufficient. This approach is based purely whether the model outputs the correct answer, and does not make any assumptions about the role name mover heads. We let VMPP and Vresid be 1-dimensional subspaces found by DAS in the layer MLP activations and layer 8 residual stream output at the last token, respectively.
4.3 DEMONSTRATING THE ILLUSION FOR THE VMLP DIRECTION
Methodology. In this section, we perform patches between examples that only differ in the variable we want to localize in the model, i.c. the position of the S and IO names in the first sentence. That is, we patch from e.g. 'Then, Mary and John went to the store. John gave a book to' into "Then, John and Mary went to the store. John gave book to', and vice-versa. We report all metrics on a test set that is not used to train the interventions that require training. We consider the following activation patching interventions:
full MLP: patching the hidden activation of the 8-th MLP layer at the last token.
VMLP: patching along the direction VMLP found in Subsection 4.2
VMLP nullspace: patching along the causally disconnected component of VMLP. This is the orthogonal projection v MLP W Wont of the MI.P layer.
nullspace of VMLP the nullspace ker Won. of the down-projection
VMLP rowspace:
patching along the causally relevant component of VMLP- This is the orthogonal projection MLP of VMLP on the rowspace of Wout- Note that have the
orthogonal decomposition VMLP = - + VLP
mullspace
rowspace
full residual stream: patching the entire activation of the residual stream at the last token after layer 8 of the model. This is indicated as the location of Vresid in Figure 21
All activation patches have the goal of making the model output the S name instead of the 10 name. Accordingly, we use the logit difference between the logits assigned to the 1O and S names to measure how well patch performs. Formally, let logit (z) logits (x) denote the last-token logits output by the model for the 10 and S names respectively on input x. The logit difference logitdiff (z) := logit,o (α) logits (x) measures the confidence of the model for the IO name over the S name (it is the log-odds between the two names assigned by the model). It averages 3.3 over the IOI distribution, and is positive for almost all examples (99%+). Similarly, for an activation patching intervention P, let logit to logits P (Ir) be the corresponding logits when
P

---

Published as a conference paper at ICLR 2024
2 with P applied, with logitdift (x) == logitio (a)) - logits (z). Our main metric is the average fractional logit difference decrease (FLDD) due to a patching intervention P. given by FLDD (x) 1 difference on average, and values above 100% suggest that the patch more often than not makes the model prefer the S name over the 10 name. We also report the interchange accuracy, which is the fraction of prompts for which the 10 name is assigned a higher logit than the S name, but the intervention P reverses this.
logisdiff" This metric is zero when the patch has no effect on the logit (ogitdif((r))
Results. Metrics are shown Table [ Through these metrics and additional experiments, we exhaustively confirm the mechanies of the illusion.
THE CAUSALLY DISCONNECTED COMPONENT OF VMLP DRIVES THE EFFECT. While patching the VMLP direction has significant effect on the FLDD metric (46.7%) this effect is greatly diminished when we remove the component of VMLP in ker Wout whose activations are (provably) causally disconnected from model predictions (13.5%), or when we patch the entire MLP activation (-8%, actually increasing confidence). By contrast, performing analogous ablations on Viesid leads to very similar numbers (0.40.% 12..5% 123.6%; we refer the reader Section 5 for details on the Vresid experiments). This confirms our hypothesis from Section
PATCHING MMP ACTIVATES A DORMANT PATHWAY THROUGH THE MLP. To corroborate these findings, in Figure 3 we plot the projection of the MLP layer's 's contribution to the residual stream on the gradient direction Vgrad before and after patching, in order to see how it contributes to the attention of name mover heads. We observe that in the absence of intervention, the MLP output is weakly sensitive to the name position information, whereas after the patch this changes significantly.
FURTHER VALIDATIONS OF THE ILLUSION. We observe that the disconnected-dormane decom- position from the illusion approximately holds: the causally disconnected component of VMLP (the one in ker W... ooe) is significantly more activated by the position information than the component in (ker Won.) which is the one driving the causal effect (Appendix Figure 20); in this sense, the causally relevant component is 'dormant' relative to the causally diconnected one. While the contri- bution of the VMLP patch to logit difference may appear relatively small, in Appendix [B44 we argue that this is significant for single component. Finally, in Appendix [B.5] we show that we can find a direction within the post-gelu activations that has an even stronger effect on the model's behavior, even when replace the MLP weights with random matrices.
FINDING AND VALIDATING A FAITHFUL DIRECTION MEDIATING NAME POSITION IN THE IOI TASK
As a counterpoint to the illusion, in this section we demonstrate a success case for subspace acti- vation patching and DAS by revisiting the directions Vgadd and Vresid defined in Subsection 4.2 and arguing they are faithful to the model's 's computation to a high degree. Specifically, we subject these directions the same tests we used for the illusory direction VMLP and arrive at significantly dif- ferent results. Through this and additional validations, we demonstrate that these directions possess the necessary and sufficient properties of a successful activation patch being both correlated with input variation and causal for the targeted behavior - in an irreducible way.
Ruling out the illusion. Let Wmmamm movers R 768x192 be the stacked query matrices of the name mover heads. We use this matrix as proxy to determine the causally disconnected subspace of the 768-dimensional residual stream. In Table [ we show the fractional logit difference and inter- change accuracy when patching Vresid and Vyrad, well their components along ker Wamm movers (denoted 'nullspace") and its orthogonal complement (denoted 'rowspace"). We observe that the non-nullspace metrics are broadly similar; in particular, removing the causally disconnected compo- nent of Vresid does not greatly diminish the effect of the patch in terms of the logit difference metrics (as it does for VMLP). We also find that Vresid is predominantly in (ker W name movers) (and so is ygrad. Q
but this is to be expected by its definition).
Importantly, since the residual stream activation where Vresid and Vgrad are patched is full bottleneck for the model's computation, it is not possible for these directions to be causal but dormant (in the sense of Section 3): there can be earlier model component that activates this direction in way that avoids the patch via a skip connection (unlike for the MMLP direction). Indeed, in Figure / in A Appendix C we show that the Vresid direction gets written by the S-Inibbitoon heads, and in Figure
7

---

Published as a conference paper at ICLR 2024
19 in Appendix F.2 comes first/secon.. See for additional experiments validating these subspaces.
show they strongly discriminate between the prompts where the 10 name
FACTUAL RECALL
Given a fact expressed 'is in',o = "Paris"), we say that a model M 'recalls' the fact [s, o) if M completes a prompt expressing just the rs) pair (c.g., 'The Eiffel Tower is in') with o. In this section, we show that the interpretability illusion can also be exhibited for the factual recall capability of language models, a much broader setting than the IOI task. In particular, we can fit DAS to any such pair of facts and perform an activation patch that changes recall. We further show that the illusory subspace is equivalent to rank one edit to the weights that changed the recalled fact.
subject-relatoo-ohjjec triple (s, (e.g., 'Eiffel Tower', ,r
6.1 FINDING ILLUSORY 1- I-DIMENSIONAL PATCHES FOR FACTUAL RECALL
Let us be given two facts (s) and (s', r') for the same relation that a model recalls correctly, with correspond-
1.0
Method
ing factual prompts A expressing r8) and B expressing (s'.r) (e.g..
Fraction of facts changed
DAS
Full MLP patch
0.8
DAS rovspace component
is A = 'The Eiffel Tower is in', = "The Colosseum in'). In this subsection, we patch from B into A, with the goal of changing the model's 's output from o o'.
0.6
0.4
More precisely, we patch from the last token of 8' in B to the last token of 8 A (prior work has shown that the fact is retrieved on
0.2
s (Geva et al.] 2023 ). and we again use DAS Geiger et al. (023b) to optimize for a direction that maximizes the logit differ-
0.0
10
15
20
25
Intervention layer
ence between of and o. We use the first 1000 examples from the COUNTERFACT dataset (Meng et al. 2022a), and filter from them 40 pairs of Tacts across 8 relations that the model recalls correctly. We use GPT-2 XL (1.5B parameters) for experiments.
Figure 4: Fraction of successful fact patches un- der three interventions: patching along the direction found by DAS (blue), patching the component of the DAS direction in the rowspace of Wont (green), and patching the entire hidden MLP activation (orange).
Similar to Section we compare the effect
of these patches to a full MLP patch, and patch of only the rowspace component. Results are shown in figure We find a stronger version of the same qualitative phenomena we observed with the IOL illusory direction: (i) the directions we find have strong causal effect (successfully changing to o'), but (ii) this effect disappears when we ablate the component in the nullspace of Wout, and (iii) patching the entire MLP activation instead has a negligible effect the difference in logits between the correct and incorrect objects. We further observe that the difference in last-token subject activations is significantly aligned with the causally disconnected component of the subspaces found (cosine similarity ~ 0.9, Appendix Figure 12p. This implies that it is weakly aligned with the rowspace component (cosine similarity 0.4); hence, the rowspace component is relatively dormant compared to the causally disconnected component. Further experiments confirming the illusion are in Appendix D.2
6.2 I-DIMENSIONAL FACT PATCHES IMPLY EQUIVALENT RANK- 1 FACT EDITS
implies the existence of seemingly different intervention with similar effect: a rank-one model edit to the weights of the MLP layer. Proposed in Meng et al. (2022a), rank-1 editing modifies the Wont weight of a single MLP layer to make a model that recalls (s, r, o) recall (s, r, o') instead, while minimally modifying the model otherwise. This finding suggests mechanistic explanation for the discrepancy between fact localization and fact editing observed in prior work (Hase et al., (023). Namely, illusory sub- spaces where fact editing can 'work' may exist even in layers not storing the fact, by virtue of the
Finally, we show that the existence of an activation patch as in Subsection 6.
8

---

Published as a conference paper at ICLR 2024
residual stream containing a causally disconnected but sensitive to the subject direction and causal for the object direction to be combined in an MLP layer.
In more detail, Meng et al. (2022a) propose specific rank-1 model edit, ROME, which takes as input a vector E REME representing the subject (c.g. an average of last-token representations of the subject) and vector E Romdde which, when output by the MLP layer, will cause the model to predict a new object for the factual prompt. ROME modifies the MLP weight by setting out Wout +ab where a E Rdmodes, , E RRdar are chosen so that W Wo e, and the MLP's output is otherwise minimally changed. Without loss of generality, the first condition implies that a = v Wouch and bTHh k 1; the second condition is then modeled by minimizing the variance of 6" when x ~ N0, Σ) for an empirical estimate Σ E EdxeLxdxrLL of the covariance of MLP activations (see Lemma D.1 in Appendix D for details and a proof).
Intuitively, a fact patch as in subsection 6.1 should have a corresponding ROME edit with the same effect. Specifically, suppose that we are in the setup [. and u.a, ug are the last subject token Post-GELU activations for prompts A and B; then U takes the role of k; and the patched output of the MLP layer takes the role of v. Indeed, in Appendix D.4 we show that, for any direction Rddur in the MLP's activation space, there exists rank- edit W out results in the same output for the MLP MLPlayer at the last subject token of A as activation aetivvtion-patching from ug into U along the formal statement and proof are given in D.4] where we also derive how to make such an edit minimally change the model, following the optimization objective of ROME.
Wout + ab" which
While this shows that the patch implies a rank-1 edit with the same behavior at the token being patched, the rank-1 edit is applied permanently to the model, which means that it (unlike the activa- tion patch) applies every token. Thus, it is not priori obvious whether the rank-1 I edit will still succeed in making the model predict o' instead of o. To this end, in Appendix D.2 we evaluate em- pirically how using the rank-1 edit from Lemma D.2 instead of the activation patch changes model predictions, and we find negligible differences.
7 DISCUSSION AND CONCLUSION
Do we expect this illusion be prevalent? We only exhibit our illusion empirically in two settings, IOI and factual recall, but we believe it is likely prevalent in practice. We do not prove this, but hope to illustrate with an informal argument. Specifically, we expect the illusion occur whenever we have MLP M which is not used in the model's computation on a given task, but is between two components and B which are used and compose with each other via some direction v in the residual stream (i.e., is communicated via the skip connections of intervening layers). This is a common structure that has been frequently observed in the mechanistic interpretability literature (Lieberum et al. 2023) Wang et al.] 2023 Olsson al., 2022, Geva et al. (2021): circuits contain components composing with each other separated by multiple layers, and circuits have often been observed to be sparse, with most components (including most MLP layers) not playing significant role.
An empirically informed mechanistic argument for the illusion. The MLP M between A and B (MLP-in-the-middle set up leads to the existence of both dormant causal direction Udormant and a correlated yet causally disconnected direction Edieconnected as follows. A vector u E RM in the MLP's activation space will be causally relevant if W Woueu= = e; since the internal dimension of the MLP layer is larger than that of the residual stream (4x for popular architectures). Wont is typically full rank (see Appendix [.. for empirical evidence), and such a will exist. In particular, we can take Udormant = W v. and since we assume M is not a part of the computation, this direction should not be significantly correlated with the feature being patched. A vector u E Rdwir will be correlated if it can linearly recover the concept in question from the MLP's activations Bresid) Because M occurs after A, the feature is linearly recoverable from Presid by using e as a probe, and hence is recoverable from Wresid as well (again, empirically W., is full- rank). In theory, the gelu non-linearity could completely destroy the information, but empirically (see Appendix [F. aand E.2). it seems to merely add noise, so the feature remains linearly recoverable, giving us Uddsconnected.
Takeaways and recommendations, As we have seen, optimization-based methods using subspace activation patching can find both faithful and illusory features with respect the model's computa- tion. We recommend running such methods in activation bottlenecks such as the residual stream, as well as using validations beyond end-to-end evaluation to ascertain the precise role of such features.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank the Stanford Existential Risk Initiative's Machine Learning Alignment and Theory Scholars (SERI MATS) program (Summer 2023 Cohort) for supporting this research, and to Atticus Geiger for many helpful and insightful discussions. We would also like to thank the anonymous reviewers for their helpful feedback.
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color. arXiv preprint arXiv:2109.064 2021.
Julius Adehayo, Justin Gilmer, Michael Muelly, lan J. Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp- 9525 9536, 2018. URL httpss: neurips wwtppappp/c..//////e/cc/ 294a8ed24b1adee www.bald22ec222cc 22448022211.22222222222242333333 www.aaaa0004907773aaaabaaea html
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraie struc- ture of word senses, with applications to polysemy. Transactions of the Association for Compu- tational Linguistics, 6:483-495, 2018.
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for BERT. arXiv preprint arXiv:2104.07/ 0434,, 2021.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering latent knowledge in lan- guage models without supervision. arXiv preprint arXiv:2212.038277 2022.
Lawrence Chan, Adria Garriga-Alonso, Nix Goldowsky-Dill, Ryan Greenblatt, Jenny Ni- tishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrub- bing: A method for rigorously testing interpretability hypotheses. Alignment Fo- 7RN/
rum. URL httppp///www.llllmmeetttt.ooooop causal-
Roi Cohen, Eden Biram, Ori Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing language models. arXiv preprint arXiv:2307. 2023.
Arthur Conmy, Augustine N Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. arXiv preprint arXiv:2304.1mm 149977 2023.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $&I#* vector: Probing sentence embeddings for linguistic proper- ties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2126-22222222 2136. Melbourne, Australia, July 2018. Association for Com- putational Linguistics. doi: 10.18653/v1/P18-1 1198 URL httpp:://claatholllgg/ooooo////s P18-1198
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8493-8502. Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 18653/vv1/202// acl.long......... URL https:
7/acl Tuclanthoo antho www.tooooo.....o//....oooo
10

---

Published as a conference paper at ICLR 2024
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Techmologies, Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 18653//1/N11911430 URL
https::/accLanthooogy. http:///c...aaooooooooooo/////- 1423
Nelson Elhage. Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Gan- guli, Zac Hatfield-oddds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https: / wansttranafftte former-orr--ooo wwwcuits..tti..oee /twp://222////a wwt.w/wwwwwwweaaeeeooe.ea.eaweet wndex. html
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Saadield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. html
htttps::/transforree
rcuits..
pub/ 0tlb/202/// Stcrly_ooeel/igg
Atticus Geiger, Kyle Richardson, and Christopher Potts. Neural natural language inference models partially embed theories lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop Analyzing and Interpreting Neural Networks for NLP, pp. 163-173, Online, November 2020. Association for Computational Linguistics. doi:
10. 10.18653/1/20000bacckkxxl/ F020.blackboxmlp--
URL https: htop:://wwwwr http:::/wwwwwwwlllle... web. wwtpbb/.gggaaatooggggeeo
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-9586 9586, 2021.
Atticus Geiger, Christopher Potts, and Thomas Icard. Causal abstraction for faithful model interpre- tation. Ms., Stanford University, 2023a. URL URL/tttss//arrvv..o/////a/ abwpg//////// 04709
Atticus Geiger, Zhengxuan Wu, Christopher Potts. Thomas Icard, and Noah D Goodman. Finding alignments between interpretable causal variables and distributed neural representations. arXiv preprint mrxXXx:23330025:6 2023b.
Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.0477 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 202/ Conference on Empirical Merhods in Natural Language Processing, pp. 5484 1495, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. (0.1866) 3663/v1/2221..mmm// main.446. URL https://acl antho www.holooo.....ooooooooooooooo.. .rg. eeel1.emnlle ma n. 446
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.1470 14767, 2023.
Nicholas Goldowsky-Din -Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching. arXiv preprint arXiv:2304.05969, 2023.
Grand, I Blank, Pereira, and Fedorenko. Semantic projection: Recovering human knowledge of multiple, distinct object features from word embeddings. arxiv. arXiv preprint 1rXiv:1802.01221, 2018
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. arXiv preprint Xww:::3332302211111 2023.
Stefan Heimersheim and Jent A circuit for Python docstrings in - attention-only trans- former. URL http://w...ttlooaaaaaaaaaaatatta a

---

Published as a conference paper at ICLR 2024
Evan Hernandez, Belinda Z. Li, and Jacob Andreas. Inspecting and editing knowledge representa- tions in language models, 2023.
Belinda Z Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. arXiv preprint (0t:::2000000777,, 2021.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. arXiv preprint 2rX11:2306.03341, 2023a.
Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation. arXiv preprint arXiv:2309.05973, 2023b.
Tom Lieberum, Matthew Rahtz, János Kramár, Geoffrey Irving, Rohin Shah, and Vladimir Miku- lik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.0944 2023.
Zachary C. Lipton. The mythos of model interpretabilityy 2016. URL https::/aaxiv..tt abs/1606. 03490
Kevin Meng. David Bau, Alex J Andonian, and Yonatan Belinkov.
Locating and editing factual associations in GPT. Advances Neural Information Processing Systems, 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in transformer. arXiv preprint arXiv:2210.07229, 2022b.
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 3111-3119, 2013. URL https: http:::::///rooeeiiggg.... neuriss.....ss.. http:/eewrr.ss.eeeeeeeeeee/eee/e
html
Neel Nanda. Actually, othello-gpt has a linear emergent world representation. 2023a.
Neel Nanda. Attribution patching: Activation patching
industrial scale, 2023b
URL
Sstps://wwtt nee. io/mechan st -tweechhansttt-nnteeeere stic-inteepretab litttttttbbbllyy///
attribution--atach
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint drXtv:2309.00941, 2023.
Chris Olah.
Mechanistic interpretabilityy variables, and the importance of inter-
pretable bases. http::/wwww.rransssoerrrrrrrrcus 2022. htt.rfmmmrrrrrruutttspa////////e
URL https://wwy www.ennerrrpssss..........
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 2020. doi: 10.239115ddssiill.00244.00
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfeld-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Ka- plan, Sam McCandlish, and Chris Olah. In-context lcarning and induction heads. Trans- pub/2020
former Circuits Thread, 2022. URL http://///ttaasfooeeeeiioooooooo wanndd-innutttt--heddddt html
in ccontext-learmnn
OpenAl. Gpt-4 technical report, 2023.
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference Uncer- tainty in Artificial Intelligence, UAI'0I, pp. 411-420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
P. Smolensky. Neural and conceptual interpretation of PDP models. In Parallel Distributed Pro- cessing: Explorations in the Microstructure, Vol. Psychological and Biological Models, pp. 390-431. MIT Press, Cambridge, MA, USA, 1986. ISBN 0262631105.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. Understanding arithmetic reasoning in language models using causal mediation analysis. arXiv preprint arXiv:2305. /5054, 2023.
lan Tenney, Dipanjan Das, and Ellic Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp- 3593-4601, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19- 1452. URL ac http://aaaaaaaaaa.o.oooooo//////
Alex Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308. 10248, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gareett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 2017. URL http:: httttps://proceeddggg.s.so 355ee77357/6e3311bbb3cccccc../
/ctpaappe//21///astt/ Abstract html
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang. Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp:
The case of gender bias.
abs/ 200.2022.5.222552
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp:
The case of gender bias. arXiv preprint arXiv:2004. /2265, 2020.
Jonas Wallat, Jaspreet Singh, and Avishek Anand. BERTnesia: Investigating the capture and
forgetting of knowledge in BERT. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 174-183, Online, November 2020. Association for Computational Linguistics. doi: 10.
URL
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt.
Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/oopnneview. wtt.:///oorrmmeeeeeaaaSSaaSS.a.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language mod- els. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151-6162, Online, 2020. Association for Computational Linguis- tics. doi: 10.18653/w1/2020.coo watp:00eemmmmmmimmeeeeemm m9...496. emn 1p-maainn 496
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint vrXiv:2206.07682 2022.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms alpaca. arXiv preprint arXiv:2305.08809 2023.
Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. Mquake: Assessing knowledge editing in language models via multi-hop questions, 2023.
13

---

Published as a conference paper at ICLR 2024
A ADDITIONAL DETAILS FOR SECTION
3
A.1 THE ILLUSION FOR
SUBSPACES
In the main text, mostly discuss the illusion for activation patching of 1-dimensional subspaces for case of exposition. Here, we develop a more complete picture the mechanics the illusion for nigher-dimensional subspaces.
Let be a model component taking values R2, and let U C R be a linear subspace. Let V be a matrix whose columns form an orthonormal basis for U. If the C activations for examples A and B are a a E I&d respectively, patching U from A into B gives the patched activation
act actpacched = acts + VV (act, - actp) (I VV Jacts + VV act
For intuition, note that VVT is the orthogonal projection on U, so this formula says to replace the orthogonal projection of acts on U with that of act,, and keep the rest of acts the same.
Generalizing the discussion from Section 3 for the illusion to occur for subspace S, we need S to be sufficiently aligned with a causally disconnected subspace Vdisconnected that is correlated with the feature being patched, and a dormant but causal subspace Vdormant which, when set to out of distribution values, can achieve the wanted causal effect. For example, a particularly simple way in which this could happen is if we have let Vdiseonnected, Vdormant be 1-dimensional subspaces (like in the setup for the I-dimensional illusion), and we form S by combining Vdisconnected + Vdormant with a number of orthogonal directions that are approximately constant on the data with respect to the feature we are patching. These extra directions effectively don't matter for the patch (because they are cancelled by the acta act term). Given a specific feature, is likely that such weakly- activating directions will exist in a high-dimensional activation space. Thus, if the 1-dimensional illusion exist, so will higher- nigher-dimensional ones.
A.2 ILLUSORY 1- I-DIMENSIONAL PATCHES ARE APPROXIMATELY EQUAL PARTS CAUSALLY DISCONNECTED AND DORMANT
In this subsection, we prove quantitative corollary of the model of our illusion that suggests that we should expect illusory patching directions to be of the form V2 (Fdisconnected + Vddommnnt) for unit vectors [[diisoonnected = ||Voormenell 1. In other words, we expect the best illusory patches be formed by combining a disconnected and illusory direction with equal coefficients, like depicted in Figure
Lemma A.1. Suppose we have two distributions of input prompts D DB. In the terminology of Section 3 let Vdisconnected 1 Edorment be unit vectors such that the subspace spanned by Udisconnected is causally disconnected subspace, and the subspace spanned by Udorment is strongly dormant, in the sense that the projections of the activations of all examples Dxouree U Drase onto Udormant are equal to some constant
Suppose we form v Vddicconeccted COS O: + Udormant sin as a unit-norm linear combination of the two directions. Then the magnitude of the expected change in projection along Vdormant when patching from A D into ~ DB is maximized when 3. i.e. COS sin V2
Proof. Recall that the patched activation from FA into Tg along u
act
actpicceed
act + (pa - PB)C
where PA T Taat AB T act the projections of the examples' activations e. The change along Vdormant is thus
V (actpat actn)) (PΛ - PP) sin (w act T acttt) sin
"ddscoonectea(act,, act ) COS asin a where we used the assumption that "dormangact,
aCt Hence, the expected change
is
cos sin U EA- B.D- (acta actp].
The function f(a) = cos a sin a for a E [0, T/2] ] is maximized for a = π 7/4, concluding the proof.
n
14

---

Published as a conference paper at ICLR 2024
A.3 CONCRETE MATHEMATICAL EXAMPLE OF THE ILLUSION
Toy Illusion. For a distilled example of the illusion, consider a network A that takes in a real valued input E I., computes a three dimensional hidden representation h W x, and then real valued output y = W h. this example, we consider the network's hidden layer to be a single 'component', represented as a 3-dimensional vector space of activations. Define the weights to be W1 = [1
1] and W2 = [0 -2 1] and observe that the network computes the identity function (see Figure [8 in Appendix F.T for illustration). While the 3rd hidden neuron clearly mediates this effect, surprisingly, patching the direction along the sum of the first two neurons does as well, despite the fact that the 1st neuron is causally disconnected, and the 2nd is dormant.
Consider Figure I8] It should be obvious that the hidden unit H3 fully mediates the information flow from input to output, and that H2 is dormant while H3 is disconnected. However, it may be surprising that the linear subspace of H1 and H2 defined by the unit vector 1-2- v2 also fully mediates the information flow, despite it consisting of dormant and disconnected directions. Activation patching on this subspace leverages the information stored in the disconnected subspace in order to activate the dormant subspace by fixing it out of distribution value. In this way, activation patching on a subspace can activate a 'dormant parallel circuit'.
A concrete hypothesis. Based this discussion, can test for. Specifically, our hypothesis is that there exist pre-trained transformer language models M. pairs of distributions Dosse and DD soource over inputs to M (with associated ground-truth next- word predictions for inputs in Doase U D souuree), and 1-dimensional subspaces S of post-GELU activations of MLP layers in these models, such that activation patching from Isource D Dsource into Tbase ~ Doase along S has a strong effect of shifting probability from the expected completion of Tbase that of Zsource, but this effect is significantly diminished when activation patching is performed along the component of S orthogonal to the nullspace of the down-projection Wout of the MI.P layer. this example, the nullspace component of S is the causally disconnected subspace, and we hypothesise it is possible the remaining component is dormant.
also postulate specific hypothesis
B ADDITIONAL DETAILS FOR SECTION
B.1 DATASET, MODEL AND EVALUATION DETAILS FOR THE IOI TASK
We use GPT2-Small for the 1OI task, with dataset that spans 216 single-token names, 144 single- token objects and 75 single-token places, which are split 1 across a training and test set. Every example in the data distribution includes (i) an initial clause introducing the indirect object (IO, here 'Mary') ) and the subject (S. here 'John') ) and (ii) a main clause that refers to the subject second time. Beyond that, the dataset varies in the two names, the initial clause content, and the main clause content. Specifically, use three templates as shown below:
Then, [] and [] had a long and really crazy argument. Afterwards, [] said to Then, [] and [] had lots of fun the [place]. Afterwards, [ gave [object] to Then, [] and [] were working at the [place]. [] decided to give a [object] to
and we use the first two in training and the last in the test set. Thus, the test set relies on unseen templates, names, objects and places. We used fewer templates than the 1OI paper WWan et al. (2020) in order to simplify tokenization (so that the token positions of our names always align), but our results also hold with shifted templates like in the I0I paper.
On the test partition of this dataset, GPT2-Small achieves an accuracy of 91%. The average difference of logits between the correct and incorrect name is ss 3.3, and the logit of the correct name is greater than that of the incorrect name in x 99% of examples. Note that, while the logit difference is closely related to the model's 's correctness, it being 0 does not imply that the model makes the correct prediction, because there could be a third token with a greater logit than both names.
15

---

Published as a conference paper at ICLR 2024
B.2 DETAILS FOR COMPUTING THE GRADIENT DIRECTION VGRAD
For given example from the test distribution and given name mover head, we compute the gra- dient of the difference of attention scores from the final token position to the 3rd and 5th token in the sentence (where the two name tokens always are in our data). We then average these gradients over a large sample of the full test distribution and over the three name mover heads, and finally normalize the resulting vector to have unit &g norm.
We note that there is 'closed form' way compute approximately the same quantity that requires no optimization. Namely, for a single example we can collect the keys kg, kio to the name mover heads at the first two names in the sentence (the S and 10 name). Then, for a single name mover head with query matrix Wq. a maximally causal direction in the residual stream at the last token position after layer 8 will be such that Woe is in the direction of kg -k10. because the attention score is simply the dot product between the keys and queries. We can use this to "backpropagate" v by multiplying with the pseudoinverse W This is slightly complicated by the fact that we have been ignoring layer normalization, which can be approximately accounted for by estimating the scaling parameters (which tend to concentrate well) from the IOI data distribution. We note that this approach leads to broadly similar results.
B.3 TRAINING DETAILS FOR DAS
To train DAS, always sample examples from the training IOI distribution as deseribed in Ap- pendix B We sample equal amounts of pairs of base (which will be patched into) and source (where take the activation patch in from) prompts where the two names are the same between the prompts, and pairs of prompts where all four names are distinet. We optimize DAS to maximize the logit difference between the name that should be predicted if the position information from the source example is correct and the other name
For training, we use learned rotation matrix as in the original DAS paper (Geiger al, 2023b), parametrized with torch. nn. ut 1s parammtt rinttiiiatttt oons orthogonal. We use the Adam optimizer and minibatch training over training set of several hundred patching pairs. We note that results remain essentially the same when using higher number of training examples.
B.4 DISCUSSION OF THE MAGNITUDE OF THE ILLUSION
While the contribution of the VMLP patch to logit difference may appear relatively small, we note that this is the result of patching a direction a single model component at a single token position. Typical circuits found in real models (including the IOI circuit from Wang et al. (2023)) often composed of multiple model components, each of which contribute. In particular, the position signal itself is written by 4 heads, and chiefly read by 3 other heads. As computation tends to be distributed, when patching an individual component accuracy may be a misleading metric (eg patching out of heads is likely insufficient to change the output), and a fractional logit diff indicates significant contribution. By contrast, patching in the residual stream is a more potent intervention, because it can affect all information accumulated in the model that communicated to downstream components.
B.5 RANDOM ABLATION OF MLP WEIGHTS
How certain are we that MIP8 doesn't actually matter for the IOI task? While we find the IOI paper analysis convincing, to make our results more robust to the possibility that it does matter, we also design further experiment.
Given our conceptual picture of the illusion, the computation performed by the MLP layer where we find the illusory subspace does not matter long it propagates the correlational information about the position feature from the residual stream to the hidden activations, and as long as the output matrix Wons is full rank (also, the discussion in 7.. Thus, we expect that if we replace the MLP weights by randomly chosen ones with the same statistics, we should still be able to exhibit the illusion.
16

---

Published as a conference paper at ICLR 2024
Specifically. randomly sampled MLP weights and biases such that the norm of the output acti- vations matches those MLP8. As random MLPs might lead to nonsensical text generation, we don't 't replace the layer with the random weights, but rather train subspace using DAS on the MLP activations, and add the difference between the patched and unpatched output of the random MLP the real output of MLP8. This setup finds subspace that reduces logit difference more than the MLP direction.
This suggests that the existence of the VMLP subspace is less about what information MLP8 contains, and more about where MLP8 is in the network.
C ADDITIONAL DETAILS FOR SECTION 5
Which model components write to the Vresid direction? To test how every attention head and MLP contributes to the value of projections on VMLP: we sampled activations from head and MLP outputs at the last token position of IOI prompts, and calculated their dot product with Vresid (Figure 7). We found that the dot products of most heads and MLPs was low, and that the S-inhibition heads were the only heads whose dot product differed between different patterns ABB and BAB. This shows that only the S-inhibition heads write to the Vesid direction (as one would hope). Importantly, this test separates Vresd from the interpretability illusion VMLP. While patching VMLPS also writes to Vresids (i.e. VMLP& out & Vresias), the MLP layer does not write this subspace on the IOI task (see Figure 3). This further supports the observation that the MMLP patch activates a dormant pathway in the model.
Generalization beyond the IOI distribution. We also investigate how the subspace generalizes. We sample prompts from Open Teenxteee and look at those with particularly high and low ac- tivations Vsintab- Representative examples are shown in Figure & together with the name movers attention at the position of interest, how the probability changes after subspace ablation, and how the name movers attention changes.
Stability of found solution. Finally, we note that solutions found by DAS in the residual stream are stable, including when trained on a subset S-inhibition heads (see Figure 5
S-iniibition headas
Cosine similarity of learned position directions
1.0
0.82
layer S-inhibition heads
0.8
0.82
1.00
0.75
0.10
0.94
layer inniibition heads
random layer heads
0.91
0.75
1.00
0.04
0.94
average layer heads
0.2
0.94
1.00
0.0
S-idhibiionn headas
layer 5 inhibitton heads
layer 4-inhbbtton heads
ransoom layer 8 heads
average layer headss
Figure 5: Cosine Similarity between learned position subspaces in the S-inhibition heads is high even when using only a subset of S-inhibition heads for training
17

---

Published as a conference paper at ICLR 2024
Suraaee Altradh
1St 10 end 153 (0)
(ebect)
Figure 6: The IOI position subspace activates words that predict a repeated name. S-inhibition subspace activations for different IOI prompts per position
Layer
Layer
Layer?
Layer
AB2
Head
Figure 7: S-Inhibition heads but not MLP8 write to the position subspace in the residual stream that causally connected to the name movers on the IOI task
D ADDITIONAL DETAILS FOR SECTION 6
D.1 TRAINING DETAILS FOR FACT PATCHING (SECTION 6.1)
We use the first 1000 examples from the COUNTERFACT dataset (Meng et al. 2022a). We filter the facts which GPT2-XL correctly recalls. Out of the remaining facts, for each relation we form all pairs of distinct facts, and we sample such pairs from each relation with at least facts. This results in a collection of 40 facts spanning 8 different relations.
To train DAS, due the computational difficulty of the problem, instead of optimizing rotation matrix like in Geiger et al. (2023b), we directly optimize for single unit vector using projected gradient descent, where after each gradient update we normalize the vector have unit norm again.
D.2 ADDITIONAL FACT PATCHING EXPERIMENTS
In figure IT we show the distribution of the fractional logit difference metric (see Subsection 4.2 for
definition) when patching between facts as described in Subsection 6. Like in the related Figure we observe that, while patching along the directions found by DAS achieves strongly negative values (indicating that the facts very often successfully changed by the patch), the interventions that replace the entire MLP layer or only the causally relevant component of the DAS directions have such effect.
Next, we observe that the nullspace component of the patching direction is the one similar to the variation in the inputs (difference of last-token activations the two subjects). Specifically, in Figure[2] we plot the (absolute value of the) cosine similarity between the difference activations for the two last subject tokens, and the nullspace component of the DAS direction. We note that
18

---

Published as a conference paper at ICLR 2024
this similarity is consistently significantly high (note that it can be at most which would indicate perfect alignment).
Finally, observe that the nullspace component of the patching direction is non-trivial part of the direction in Figure 13, where we plot the distribution the lg norm of this component.
D.3 RANK-1 MODEL EDITING AS AN OPTIMIZATION PROBLEM
We review the ROME method from Meng et al. 2022a) and show how it can be characterized as the solution a simple optimization problem. Following the terminology of 6.2 let us have an MLP layer with an output projection W out, a key vector k E gddir and value vector Rdmodel
In Meng et al. 2022a), equation 2, the formula for the rank-l update to Wont is given by
A
W out W out + (v Wonth
(3)
k Eー'E
where Σ is an empirical estimate of the uncentered covariance of the pre- the following equivalent characterization of this solution (which may be of independent interest):
pre-Wow. Wout activations. We derive
Lemma D.1. Given matrix W Wout R.dmdd bateeddxxaaaaa XdMLP, a key vector k: E Rd and a value vector E Romaci let Σ > O, E E ROMLP XdMLL be positive definite matrix (specifically, the uncentered empirical covariance), and let (0, Σ) be a normally distributed random vector with zero mean and covariance Σ. Then, the ROME weight update is W Wout + ab where E Rdmadel b E RduLP solve the optimization problem
min trace(Cov, [W o W Wootz) subject to a.b
W Woak=
In other words, the ROME update is the update that causes Wont to output v on input k, and minimizes the total variance the extra contribution of the update in the output of the MLP laver under the assumption that the pre-Wow out activations are normally distributed with covariance 2
Proof. Using Ex (xxx ]]= E and the eyclic property of the trace, we that trace [W x
Wow+)) = || T Σb
We must have ab k -ー Wk,, so without loss of generality we can rescale a, b so that v- Wk. Then, we want to solve the problem
b Eb subject k=
which we can solve using Lagrange multiplies. The Lagrangian is
1 Σb -
Lb, A)
and the derivative w.r.t. is Eb AA= 0, which tells us that b is in the direction of Σ-1k. Then the constraint bT forces the constant of proportionality, and arrive b ETM-TE
& x-x-
D.4 STATEMENT AND PROOF FOR DERIVING RANK-1 1 EDITS FROM SUBSPACE PATCHES
Lemma D.2. Given prompts A and B, two token positions LA, LB and an MLP layer with oulput projection weight W out Rododd Xdmar let u uB RdæeL be the respective (post-nonlimearity) activations at these token positions in this layer. If is a direction in the activation space of the MLP layer, then there exists a ROME edit W W out abb such that the activation patch from u into along v and the edit result in equal outputs of the MLP layer at token a when run on prompt A. Moreover, the ROME edit given by
a = =((uB - uA) Wout and any that satisfies 6 TuA=1
Choosing b E-tua minimizes the change to the model (in the sense of Menn et al. (022a
over all such rank-1 edits.
*Note that in practice Wout may be singular or poorly conditioned, because the layer normalization encour- ages features to to zero, which could to some extent also persist after non-lincarity. If this is the case, all results apply with E instead of E
19

---

Published as a conference paper at ICLR 2024
Proof. The activation after patching from B into A along is 4'u uA means that the change in the output of the MLP layer at this token will
((uB uA), which
Wo
W U A = ((uB - UA) (o)TT)))) Wot
The change introduced by fact edit at this token is
W out " W = ab A uA) ((uB u W oot
and the two equal because bT uA
To find the that minimizes the change to the model, we minimize the variance of DTx when (0, Σ) subject to b b"u,, 1. The variance is equal to b Eb, have constrained (convex) minimization problem
min 5 b Eb subject to b buua- =1
The rest of the proof is the same as in Lemma D.I using Lagrange multiplies. The Lagrangian is
Namely, we can solve this optimization problem
C(b, A) 6 uAA ua
and the derivative w.r.t. b is Eb - A = 0, which tells us that is in the direction of Σ-tuA. Then the constraint 6Th uA forces the constant of proportionality.
In this appendix, we collect some useful notes on the rank-1 model editing (ROME) method from Meng et al. (2022a). Lemma D.3 (Altemnttiv characterization of rank-1 model editing via variance minimization).
D.5 ADDITIONAL. EXPERIMENTS COMPARING FACT PATCHING AND RANK-1 EDITING
In Figure 14 we plot the distributions of the logit difference between the correct object for a fact and the object we are trying to substitute when patching the 1-dimensional subspaces found by DAS, and performing the equivalent rank-1 weight edit according to Lemma D.2 We observe that the two metries quite closely track each other, indicating that the additional effects using a weight edit (as opposed to only intervening
single token) are negligible.
Similarly, in Figure 15
we show the success rate of the the two methods terms of making the model output the object of the fact patching from. Again, observe that they quite closely track each other.
E WHY DO WE EXPECT THE ILLUSION TO BE PREVALENT IN PRACTICE?
F. MLP WEIGHTS ARE FULL--RANK MATRICES
In figure 16 we plot the 100 smallest singular values of the MLP weights in GPT2-Sman for all 12 layers. We observe that they the vast majority are bounded well away from O. This confirms that both MLP weights are full-rank transformations.
E.2 FEATURES IN THE RESIDUAL STREAM PROPAGATE TO HIDDEN MIP ACTIVATIONS
Intuition. Suppose we have two classes of examples that are linearly separable in the residual stream. The transformation from the residual stream to the hidden MI.P activations a linear map followed by nonlinearity, specifically x + gelu(W..r)) As we observed in E. the Wi, matrix full-rank, meaning that all the information linearly present æ will also be so W,, Even better, since Win maps from Senodel-dimenseee space, this should intuitively make it much easier to linearly separate the points, because in a higher- dimensional space there many linear separators. On the other hand, the non-line con-lincarity has opposite effect: by compressing the space of activations, it makes it harder for points to be separable. So it is priori unclear which intuition is decisive.
space to dMLP 4dmoodd-ddmmension
20

---

Published as a conference paper at ICLR 2024
Empirical validation. However, it turns out that empirically this is not such problem. To test this, we run the model GPT2-Small on random samples from its data distribution (we used Open WebText- 10k), and extract 2000 activations of an MLP-layer after the non-linearity. We train linear regres- sion with fz-regularization to recover the dot product of the residual stream immediately hefore the MI.P-layer of interest and randomly chosen direction. We repeat this experiment with different random vectors and for each layer. We observe that all regressions are better than chance and explain significant amount of variance the held-out test set (It² 0.71=00117 MSE 0..11:0.1 p < 0.005). Results are shown in Figure 17 (right) (every marker corresponds to one regression model using a different random direction).
The position information in the IOI task is really binary feature, so we are also interested in whether binary information in general is linearly recoverable from the MLP activations. test this, we sample activations from the model run on pandomly-smpple prompts. This time however, we add or subtract multiple of random direction v to the residual stream activation u, and calculate the MLP activations using this new residual stream vector u':
u' = u + y x x |2212 x
where y E [- is uniformly random, 2 is scaling factor we manipulate, and v is a randomly chosen direction of unit norm. For each classifier, randomly sample direction that either add or subtract (using y) from the residual stream. The classifier is trained to predict y. We rescale match the average norm of residual vector and then scale with small scalar z.
Then, a logistic classifier trained on 1600 samples. Again, we repeat this experiment for different and 2, and for each layer. We observe that the classifier works quite well across layers with very small values of (still, accuracy drops for z = 0.0001). Results are shown in Figure [7 (right), and Table
Table 2: Mean Accuracy for Different Values of
2
Mean Accuracy
0.0001
0.69
0.001
0.83
0.01
0.87
0.1
0.996
F SUPPLEMENTARY FIGURES
F.1 ADDITIONAL FIGURES FOR SECTION
F.2 ADDITIONAL FIGURES FOR SECTION
21

---

Published as a conference paper at ICLR 2024
G
FIGURES ADDED DURING REVISION
Baseline
Patched
s6 Busl
92 Emma
Him
them
Errma
Rudi
Vore
Yesid
VMLP
Inhib
Emma
Emma
Rudi
and
says
to
Figure 21: Schematic of the IOI circuit and interven- tions. GPT2-small predicts the correct name by S- inhibition heads writing positional information to the residual stream which is used by the name movers to copy the non-duplicated name (green arrows). Lo- cation of subspace interventions Vresid and VMLP are marked. Patching the illusory subspace VMLP adds new path (red) along the established one that is used to flip positional information when patched.
22

---

Published as a conference paper at ICLR 2024
OpenWebTezt-10k and Subspace Activation </endoftext]
Logit Diffarence
Attention of Name Movers at 2nd last position
Oklahoma
senate
panel
approved
bill
Monday
prevent
city
and
county governments from
out
doing
laws on employment
and 10 public
accommodations
-10
The
bill
Initially
prevented
local
governments
from
going
further
than
any
state
law
committee
substitute
facused it
on
emoloyment
and
Original
public
Mean Ablated
accommodations
Figure 8: The IOI position subspace generalizes to arbitrary OpenWebText prompts
23

---

Published as a conference paper at ICLR 2024
OpenWebToxt-1100 and Subspace Activation morning
Logit Difference
Attention of Name Movers at 2nd last position
last
month
Rays
Chandler
and
her
wife
Avery
Chandler
pulled
Into
the
parking
lot
at
the
10 7
Mid
building
and began
walking
towards
10 the entrance
Baby
you
the
news
trucks
Rays
ha
wn
asked
Original Mean Ablated
Avery
Figure 9
24

---

Published as a conference paper at ICLR 2024
OpenWebText Text-10k and Sub space Activation
Logit Difference
Attention of Name Movers at 2nd last position
key
two
times
you
will
see
that
will
appear
all
the
programs
scripts that begins with gn ome
0O Characters
10 and that
are
on synaptic -10 or
repositories
from
saurce
list
This is helpful when you
don 't remember exactly the
name of
program
Original Mean Ablated
maybe
Figure 10
25

---

Published as a conference paper at ICLR 2024
Fractional logit difference
-15
Method
-20
DAS
-25
DAS rowspace component
10
15
20
30
35
40
45
Intervention layer
Figure 11: Fractional logit difference distributions under three interventions: patching along the direction found by DAS (blue), patching the component of the DAS direction in the rowspace of Wot (green), and patching the entire hidden MLP activation (orange).
Absolute cosine similarity
0.4
0.2
0.0
30
Intervention layer
Figure 12: Distribution of the absolute value of the cosine similarity between the nullspace compo- nent of the DAS fact patching directions and the difference in activations of the last tokens of the two subjects.
1.0
Norm of nullspace component
0.8
0.0
Intervention layer
Figure 13: Distribution of the norm of the nullspace component of the DAS direction across inter- vention layers.
26

---

Published as a conference paper at ICLR 2024
Logit difference
-20
-30
-40
Method
-60
1 dimeisioonall patch
10
155
20
30
35
40
45
Intervention layer
Figure 14: Comparison of logit difference between I-dimensional fact patches and their derived rank-1 I model edits
Method
Fact patch success rate
rank-1 model
1-dimensionil patch
0.8
0.6
0.2
15
20
25
30
35
40
45
Intervention layer
Figure 15: Comparison of fact editing success rate between I-dimensional fact patches and their derived rank-1 model edits
4.0
3.5
Singular value
3.0
2.5
2.0
1.5
1.0
0.5
0.0
100
100
Index (decreasing)
Index (decreasing)
Figure 16: Smallest 100 singular values of the Win (left) and Wout (right) MLP weights by layer in in GPT2-Small
27

---

Published as a conference paper at ICLR 2024
1.0
eress
Layer
% 100m0
a "om
2yer
19 0 *..
Dee use
ACCUPEEV
0.6
nd do
0.4
0.6
10
00000
Figure 17: Recovering residual stream features linearly from hidden MLP activations: classification (left) and regression (right).
Hg x
X
=- 0
hgg 25g
vän v2u
Hi:z
(a) The network A that computes the iden- tity function. The hidden unit Hs stores the value the input and passes this to out- put, while the unit Hy is dormant and Hy is disconnected. However, the linear sub- space of H₃ and H. detined by the unit for 1 = fully mediates informa- tion flow from input to output just like the unit Hs.
(b) The network A except the weights have been transformed such that the hidden units Hu and Hy are viewed under the coordinate vectors 1 - vzl and [ J. When we generalize activation patching arbitary subspaces, we are forced to consider this transformed network to be analytically iden- tical to.A.
Figure 18: Diagrams of small linear networks illustrating concrete instantiation of the interpretabil- ity illusio, alongside a subspace faithful to model's computation.
28

---

Published as a conference paper at ICLR 2024
pattern ABB BAB
pattern
Number of examples
ABB
BAB
30
10
10
-5 10 25 20
25 -20
-15
-10
-5
10
15
Activation projection
Activation projection
Figure 19: Projections of dataset examples' activations in the residual stream after layer 8 onto the Vresid direction found by DAS and the V Vgrad direction which is the gradient for difference in attention of the name mover heads to the two names.
Projection onto nullspace component
Projection onto rowspace component
pattern
pattern
ABB
ABB
Number of examples
BAB
BAB
100
80
300
1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1.5
-1.0
-0.5
0.0
0.5
1.0
3.5
2.0
2.5
3.0
Activation projection
Activation projection
Figure 20: Projections of dataset examples onto the two components of the illusory patching di- rection found MLP8: the nullspace (irrelevant) component (left), and the rowspace (dormant) component (right).
29

---

Published as a conference paper at ICLR 2024
dormant dircction
darmunt direction
Gircction patch along (v)
example patch from
example potch into
causally disconnected direction
causally disconnected direction
Figure 22: Consider a 2-dimensional subspace of model activations, with an orthogonal ba- sis where the æ-axis is causally disconnected (changing the activation along it makes no dif- ference to model outputs) and values on the y axis are always zero for examples in the data distribution (a specical case of a dormant di- rection).
Suppose have two examples (green) which differ in their projection on the causally disconnected direction (and have projection on the dormant direction, by defini- tion). Let's consider what happens when patch from the example on the left into the example on the right along the 1-dimensional subspace spanned by the vector (1, 1) (red)
Figure 23:
docmaut direction
dommant direction
Sreecion patch along (1)
orthogonal complemenn v
dieetioo to patch slong
erllogona complement
example to patch from
example patch into
example patch
example patch into
sausally disconnected direction
buuallly discompected direction
PANId dn 30 Ipneaa
Figure 24:
To patch along from the left into the right example, from the left one, and leave the value along w- unchanged. In other words, take the com- ponent of the left example along e (L) and sum with the y component (2) of the orig- inal activation.
match the value along
Figure 25: This results in the patched activa- tion ! +2, which points completely along the dormant direction. In this way, the variation of activations along the causally disconnected a- axis results in activations along the previously dormant y-axis
Figure 26: A step-by-step illustration of the phenomenon shown in Figure
30


Reviews:
[{"review_id": "cu7rJXUWse", "reviewer": "Reviewer_YsUK", "summary": "This work finds and shows that subspace activation patching may be subject to interpretability illusions caused by activating a dormant pathway (i.e., does not respond to input changes) but is activated by a causally disconnected feature (i.e., is activated by the input change but is not connected to the model’s output). They show that this illusion also relates to rank-one fact editing (e.g., Meng et al. [1]) and explains its recently found inconsistencies [2]. Finally, they demonstrate that further analysis, e.g., manual circuit analysis, can mitigate the interpretability illusion.\n\n[1] Meng, Kevin, et al. \"Locating and editing factual associations in GPT.\" NeurIPS 2022.\n\n[2] Hase, Peter, et al. \"Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models.\" arXiv 2023.", "questions": "* Why do the authors solely focus on subspace activation patching? The identified interpretability illusion should also hold for (automatic) component activation patching (i.e., consider the entire activation space as the (sub)space).\n\n* There seems to be an error in the indices in the toy example in Appendix A.3.\n\n## Suggestions\n\n* While Fig. 1 clearly demonstrates the interpretability illusion, it is hard to parse. It may be good to make it more accessible/easier to parse, as it demonstrates the main insight of the paper.\n\n* It’d be good to add the relation of the vector $v$ to the subspace $U$ in Sec. 3.\n\n* It’d be good to match the notation of Tab. 1 and the respective text.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* Given the rising popularity of mechanistic interpretability and (subspace) activation patching, this paper addresses a very important and significant issue in a timely manner.\n\n* The interpretability illusion is well-motivated and clearly introduced. The formal definition (p. 5) is sound. It is also clear that the illusion is not a mere artifact of the chosen experimental settings but is present in most cases with high probability.\n\n* The experimental design to showcase the interpretability illusion is well-designed.\n\n* The discussion on how one can prevent the interpretability fallacy is important and sound. It paves a way for future work that relies on automatic activation patching methods, to avoid false interpretations of model behavior.\n\n* The discussion on the presence of the interpretability illusion from a mechanistic viewpoint is sound.\n\n* The paper is clearly written, (mostly) easy to follow, and self-containing.", "weaknesses": "This paper is a very strong submission. It is well-motivated, sound, and clear. The only “major” weakness is that code is not provided and minor comments (see below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xFtOgcaFYy", "reviewer": "Reviewer_8jAi", "summary": "This paper shows that subspace activation patching, a technique in mechanistic interpretability used to find subspaces that are “causally” responsible for models to produce certain outputs, may not be reliable. In particular, the paper explains why these may be misleading, involving the activation of dormant pathways. Real-world examples involving indirect object identification (IOI) and fact editing are shown where such techniques are misleading.", "questions": "- For a future draft, I encourage the authors to present an explicit falsifiable hypothesis that facilitates both experimentation and analysis. \n\n- It might be helpful to comment on the downstream applications and practical utility of such activation patching techniques and mechanistic interpretability, particularly to better understand the implications of the \"illusion\". What are the use cases for identifying model components (neurons/subspaces) responsible for model behavior? What evaluation metrics are available to test whether the components have been correctly identified?\n\n- Do these results for subspace activation patching also hold for usual activation patching? It seems like subspace patching is a generalization, and thus it must, and also I see that the toy example is given for the case of usual activation patching, but the experimental results and the paper's messaging are specific to subspace patching.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "+ The paper presents an interesting hypothesis that subspace activation patching methods can be misleading because of dormant pathways. On an intuitive level, this is an interesting hypothesis and needs to be considered for future applications of activation patching.", "weaknesses": "The paper does not pose an explicit hypothesis that can be falsified, limiting its scientific validity. Further, the \"illusion\" seems to make **problematic assumptions** and has unclear implications. \n\n- The hypothesis seems to be that every patched subspace “v” discovered by DAS can be decomposed into two directions “v_disconnected” and “v_dormant” (with specific properties), and that \"patching along the sum of these directions, the variation in the disconnected part activates the dormant part, which then achieves the causal effect\". The latter statement assumes that the model behaves somewhat linearly along these subspaces (i.e., the output of the model along direction \"v\" is given by a sum of its outputs along \"v_disconnected\" and \"v_dormant\"), which is a strong hypothesis given that these models are fundamentally non-linear. While the paper provides some evidence for the existence of some \"disconnected\" and \"dormant\" directions, unfortunately, I do not see evidence justifying the apparent linear behaviour of the underlying model.\n\n- The definition of the \"dormant\" subspaces is confusing in the context of this work. If the model output remains unchanged for in-distribution patching and changes only for out-of-distribution samples $x,y \\sim \\mathcal{D}$, what procedures presented in this work result in out-of-distribution samples/activations? As far as I can tell, all procedures described involve patching with in-distribution data.\n\n- Is the hypothesis that such a decomposition does not exist for “true” subspace directions?\n\n---\n\nThe paper (especially sections 4-7) is difficult to read, is very dense, and has **several omitted details**. \n\n- The main paper on its own does not seem to be self-contained and seems to contain a significant number of references to the appendix. \n\n- The main hypothesis involves the claims that (1) subspace directions can be decomposed into two directions (disconnected, dormant) with specific properties, and (2) the effect of such directions adds somewhat linearly to the model outputs. However, the paper fails to connect this terminology (\"disconnected, dormant directions\") in sections 4-7, making it difficult to verify whether the experiments confirm or deny the hypothesis. \n\n- The writing and presentation are sloppy. For example, in Section 4, “ker W_out” is never defined, and it is unclear how the results in Table 1 and Figure 3 relate to the description in section 4.3. What is ABB / BAB in Figure 3? What does “connected” in Table 1 refer to? Overall, how do Table 1, Figure 3, and Figure 4 illustrate support for the presented hypothesis?\n\n---\n\nOverall, while this work may present a useful point, its writing (especially sections 4-7) makes it impossible to verify this. I suspect that a thorough rewrite is necessary to clarify its central point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IROuaXFdrF", "reviewer": "Reviewer_ERkw", "summary": "This paper identifies a challenge for the approach of looking for subspaces corresponding to causal factors of deep neural network output.  It claims that activation patching and subspace patching approaches are at risk of an interpretability illusion where the patching fails to change the causal pathways actually leading to undesired input.  It develops the concepts of dormant pathways and causally disconnected pathways to explain why the interpretability illusion is possible.  The paper illustrates its model of the interpretability illusion using a toy example.  Furthermore, it presents real-life case studies.  One set of case studies focuses on language models, particularly the indirect object identification task.  For the IOI task, previous works have used activation patching to correct a network to output the indirect object of a sentence rather than the direct object.  The paper identifies an approach to the task, using distributed alignment search, that is subject to the interpretability illusion.  It also identifies an approach, using prior knowledge about the network to identify the nodes responsible for the error, that correctly patches the network.  The paper explains the difference between these approaches using the concepts of dormant and causally disconnected pathways.  Furthermore, the paper also studies factual recall in language models as an additional case study.  It presents a set of experiments where it shows that the vulnerability of activation patching to the interpretability illusion depends on the layer which is targeted.  The paper also includes some additional discussion on connections between activation patching and rank-one model edits.", "questions": "1. How do we know that the theoretical account for the interpretability illusion is actually the explanation for the failure of activation patching in the case studies presented?\n2. Are there a set of empirical predictions (e.g. about the directions of activations or gradients) that could either falsify or support the theoretical model?\n3. Why not relax the definitions of \"causally disconnected\" or \"dormant\" to not require strict equality, but rather to specify that the effect of the the patching be small within some threshold?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "This is a highly relevant work for the field of deep neural network interpretability, as it identifies a serious obstacle to the activation and subspace patching approaches.  The concepts of dormant and causally disconnected pathways are intuitive, and they are developed with the aid of theory and experiment.  The paper is clearly written, and figures help illustrate the concepts of the interpretability illusion, and the mechanisms involved in the IOI task.  This paper is significant for both identifying a practical problem and for developing intuitions that deepen our understanding of interpretability methods for deep neural networks.", "weaknesses": "The definitions seem rather brittle.  Requiring strict equality in the definitions of causally disconnected and dormant seems to be very limiting for practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)