Optimal ablation for interpretability
Authors: Maximilian Li
Lucas Janson
Venue: NeurIPS 2024 spotlight (2024)
Keywords: mechanistic intepretability
model internals
ablation
activation patching
automatic circuit discovery
causal tracing
tuned lens
Primary area: interpretability_and_explainability

Abstract: Interpretability studies often involve tracing the flow of information through machine learning models to identify specific model components that perform relevant computations for tasks of interest. Prior work quantifies the importance of a model component on a particular task by measuring the impact of performing ablation on that component, or simulating model inference with the component disabled.
 We propose a new method, optimal ablation (OA), and show that OA-based component importance has theoretical and empirical advantages over measuring importance via other ablation methods. We also show that OA-based component importance can benefit several downstream interpretability tasks, including circuit discovery, localization of factual recall, and latent prediction.

Optimal ablation for interpretability
Maximilian Li Harvard University
Lucas Janson Harvard University
Abstract
Interpretability studies often involve tracing the flow of information through ma- chine learning models to identify specific model components that perform relevant computations for tasks of interest. Prior work quantifies the importance of model component on particular task by measuring the impact of performing ablation on that component, or simulating model inference with the component disabled. We propose new method, optimal ablation (OA), and show that OA-based component importance has theoretical and empirical advantages over measuring importance via other ablation methods. We also show that OA-based component importance can benefit several downstream interpretability tasks, including circuit discovery, localization of factual recall, and latent prediction.
1 Introduction
Interpretability work in machine learning (ML) seeks to develop tools that make models more intelligible to humans in order better monitor model behavior and predict failure modes. Early work in interpretability sought to identify relationships between model outputs and input features (Ribeiro al.) 2016 Covert al., 2022), but with only black-box query access to observe inputs and outputs, it can be difficult to evaluate a model's internal logic. Hence, recent interpretability work often seeks take advantage of access to an ML model's 's intermediate computations to gain insights about its decision- decision-making, focusing on deciphering internal units like neurons, weights, and activations (Räuker al.) (2022). In addition to finding associations between latent representations and semantic concepts (Bau et al. 2017 Mu and Andreas 2021 Burns et al. 2022 Li et al. 2023 Gurnee and Tegmark (2024). interpretabilityy studies aim to investigate how intermediate results are used in later computation and identify specific model components that extract relevant information or perform necessary computation to produce low loss on particular inputs.
A key instrumental goal in interpretability is quantifying the importance of particular model component for prediction. Studies often measure component's importance by performing ablation on that component and comparing model performance with and without the component ablated. Ablating a component typically entails replacing its value with a counterfactual value during inference, sometimes referred to as "activation patching." However, the details vary greatly and there is lack of consensus on best practices (Heimersheim and Nanda 2024). For example, Meng et al. (2022) adds Gaussian noise to ablated components' values, while Geva et al. (023) replaces these values with zeros, and Ghorbani and Zou (2020) replaces them with their means over the training distribution.
In this paper, we present optimal ablation (OA), new method that sets component's value a constant that minimizes the expected loss of the ablated model. In section 2 we introduce OA and show that it is, in a certain sense, canonical choice of ablation method for measuring component importance. We then show that using OA produces meaningful improvements for several common downstream applications of measuring component importance. In section 3 we apply OA algorithmic circuit discovery (Conmy al.) 2023). or the identification of sparse subset of components sufficient for performance on subset of the training distribution. We demonstrate
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

that OA DA-based performance is reasonable metric for evaluating circuits and using OA for circuit discovery produces smaller and lower-loss circuits than previous ablation methods. In deploying OA to this application, we also propose new search algorithm for identifying sparse circuits that achieve low loss according to any performance metric. In section 4 we use OA to locate relevant components for factual recall (Meng et al. 2022) and show that OA better identifies important components compared to prior work. In section 5 we apply OA to latent prediction (Belrose et al. 2023a), or the elicitation of output predictions using intermediate activations. We propose OA-based prediction map and show that has better predictive power and causal faithfulness than previous methods.
Optimal ablation
2.1 Motivation
Let M represent a model that is trained to minimize Ex, ((M) Y) for a given loss function L and a distribution of random input-label pairs (X, (). A common theme in interpretability work quantifying the importance of some model component A for inference. For example, AA could represent single neuron, direction in activation space, a token embedding, an attention head, or an entire transformer layer: further examples of AA are discussed in Section [and Appendix C.2 represent the value of A when the model is evaluated on input æ. To identify domain specialization among model components, studies often measure the importance of A for model performance on particular "subtask" DD or an interpretable human-curatee distribution of input-label pairs that captures a general aspect of model behavior. write (X, Y) ~ to indicate sampling input-label pairs or X ~ 'D to indicate sampling only inputs from the subtask distribution.
C.2 Let A(x)
Although some works quantify component importance via gradients (Leino al.) 2018 Dhamdhere et al. (2188). such an approach is inherently local (even when aggregated over many inputs) and as such can fail accurately represent the overall importance of A in highly nonlinear models. Instead, most interpretability studies use ablation to quantify the importance of AA by studying the gap in performance between the full model M and a modified version M 1A with A ablated:
AAM, = P(M - P(M),
(1)
where selected metric for model performance. the context measuring importance, we argue that the construction of M is motivated by the following question:
What is the best performance on subtask D the model M could have achieved without component A? To formalize this question, split its meaning into four elements.
I. "Performance on subtask D": The relevant performance metric P is the expected loss on the subtask with respect to the full model's predictions: P(M) EX-D ((M(X),MMX) (note that E aggregates over any randomness in MD We call A defined using this choice of P the ablation loss gap.
II. "Model M could have achieved":: Since the goal of measuring component importance is to interpret the model M, the ablated model M should be constructed solely by changing the value of A, holding fixed all other parts of M. We write Ma a) indicate computing M input while setting the value of A to instead of A(x)) (see Appendix C.2 for details).
III. "Without component .A": The ablated model MxA(x)) should use a value for AA that is devoid of information about the input
Elements M and |||l motivate the following definition:
Definition 2.1 (Total ablation), A total ablation method satisfies -AXX)=M((XX MM,(X, for some random A, where A I X. (Conversely, for partial ablation method, A can depend on X.)
=
IV. "Best" performance:
To measure the importance of A. we wish to understand how much model performance degrades as a result of ablating A. If two constructions of the ablated model M'AA both perform total ablation on AA but performs worse than another, the former's underperformance cannot be entirely be attributed to ablating A, since the latter also totally
A common alternative choice is measuring performance in terms of proximity to the correct labels rather than the original model predictions, i.e. P(M) EEXYY) ((M(X), Y). See Appendix C.4 discussion.

---

ablates A and yet does not degrade performance to the same extent. Thus, the relevant M for measuring importance should incur the minimum A among total ablation methods.
To make element IV more concrete, for an ablation method satisfying element [ and given replacing A(x) with a can degrade the ablated model's performance via both deletion and spoofing:
1. Deletion. The original value A(x) could carry informational content specific to that serves some mechanistic function in downstream computation and helps the model arrive its pre- diction M(x). Replacing A(z) with some other value could delete this information about æ, hindering the model's ability to compute the original Mar).
2. Spoofing. The replacement value a could "spoof" the downstream computation by inserting information about the input that either:
(a) causes the model treat x: like a different input x'' or
(b) causes the model treat in way that it never treated any training input, if the new information is inconsistent with information about x derived from retained components. In the latter case, the confluence of conflicting information could cause later activations to become incoherent, causing performance degradation because these abnormal activations were not observed during training and thus not necessarily regulated to lead to reasonable predictions.
To measure importance, seek to isolate the contribution of effect I to performance degradation. While total ablation methods all capture a maximal deletion effect since A does not depend on X, measuring the "best" performance minimizes the additional contribution of potential spoofing effects.
2.2 Prior work
Component importance is strongly related to variable importance (Sobol 1993 Homma and Saltelli 1996 Breiman 1001 Ishwaran 2007 Gromping 2009). longstanding area of research in statisties and ML. The vast body of work in this area is too extensive to review here, and the recent surge of research interest in interpreting internal model components has raised new and unique challenges relating to the values of internal components often being deterministically related. Thus, we focus on recent work applying ablation methods to internal components in this section, and defer broader discussion to Appendix B
Ablation methods previously applied to internal components can be separated into subtask-agnostic methods, which can be applied out-of-the-boo to any subtask, and bubbtakk-secciffe methods, which only work on subtasks for which inputs satisfy a designated structure, and even then require human ingenuity to adapt to each new subtask.
Suntask-agnostic ablation methods include zero ablation (Baan et al. 2019
Lakretz et al. 2019
Bau et al.) 22200 Olsson et al. [022; Geva et al., 2023 Cunningham et al., 2024 Gurnee et al.) 2024), which replaces A(x) with zero, i.e. M
2023] Merullo al.
MA (2r, 0); mean
ablation (Ghorbani and Zou 2020 McDougall et al. 2023 Tigges et al.
2023 Gould et al.
2023 Li et al.
2024 Marks et al. 2024). which replaces A(r) with its mean, i.e. M M.Arz
Ma Ex
A(X
l): and (marginal) resample ablation (Chan al. 2022 Lieberum et al. 2023
McGrath et al.
2023 Belrose et al. 2023a Rushing and Nanda 2024), which replaces A(x) with
A(X') for an independent copy X" ~ DD of the input, i.e. (x)) (M, A(X)). While zero.
mean, and resample ablation total ablation methods, adding Gaussian noise to A(ar (Meng et al.
2022) is a subtask-agnostic partial ablation method. These methods are all applicable to any subtask.
On the other hand, subtask-specific ablation methods rely on particular details of a chosen subtask. Singh et al. (2024) replaces A(x) with interpretable values, e.g. setting an attention pattern to copy from the previous token, while Goldoosky-DDiii et al. (2023) replaces A(z) with A(x*) for an interpretable reference input x². Hanna et al. (2023) employs counterfactual ablation (CF), a partial ablation method that replaces A(z) with A (() where π is map that sends each input to "neutral" (potentially random) input (ιx) that preserves most aspects of xr but removes information
relevant to the subtask, i.e. (x)A(z) = (I. ((((,,((TTt) A(T(x))) Wang et al. (022)) also considers a counterfactual distribution of inputs for counterfactual mean ablation, which replaces A(x) with its mean over the distribution of counterfactuals, i.e. (x) MA( Ax
"See Appendix D.I for brief example of why "treating x like xx" goes beyond deletion.
3

---

Subtask-specifee methods can be useful, but it is usually unclear how to generalize them beyond the subtask originally selected. CF is the most popular among these methods, leveraged by range of manual (Vig et al.. 2020 Merullo et al. 2023 Stolfo et al., 2023] Tigges et al. 2023 Hendel al. 2023 Heimersheim and Janiak 2023 Todd et al. 2024; Marks et al., (022)) and algorithmic
[Conmy et al. [ Syed et al.
2023) studies and recommended by meta-studies (Zhang and Nanda
2024; Heimersheim and Nanda 2024 For text data, the effectiveness of CF relies heavily on token
parallelism between z and ((x), which typically share exact tokens at all but few sequence positions.
Though studies have thus far focused on toy subtasks for which suitable mappings и are relatively easily constructed, it may be difficult or impossible to select well-suited input pairs for certain subtasks (see Appendix F.3 for a few simple examples), especially more general model behaviors. Even for subtasks that admit such mapping. how ((z) is engineered to withhold subtask-relevant information differs from subtask to subtask, and the construction of ι for each particular subtask is a subjective process that requires human ingenuity. Finally, CF is only a partial ablation method; since (x)) depends on x, it may give away information about x that is useful for performance on D.
2.3 Definition and properties of optimal ablation
We present optimal ablation (OA), our proposed approach to simulating component removal. Definition 2.2 (Optimal ablation). To ablate A, we replace A(x) with an "optimal" constant a".
M Moso == Ma(x,a"), a' == araggmin EX-D (MM
We define Appt by plugging Majnn (z) into Equation I Like zero, mean and resample ablation. optimal ablation is a total ablation method satisfying Definition 2.1 But among all total ablation methods, optimal ablation is optimal in the sense that it yields the lowest A.
Proposition 2.3. Let A) be the ablation loss gap for some component A measured with any total ablation method. Then, Dopt (M, A) I (),,
Proof. Consider a total ablation method that defines M))(X) by replacing A(X) with A (per Definition 2.1p. and let AMM, A) be the measured ablation loss gap. By the tower property,
A(M = E(X=D),A L(MA(X, (() = EA [E A), A]]
Since A X, I
A= a] = (MX.D =: g(a).
Va, Dopt (M, A) EXaD a') I EXND a)) g(a)
= - Ape AM) S EA g(A) = A).
Optimal ablation thus provides the unique answer to our motivating question in Section 2.1 since it produces the best performance among all total ablation methods, including zero, mean, and resample ablation. Intuitively, OA minimizes the contribution of spoofing (effect [ from Section 2. to A by setting ablated components to constants a* that maximally consistent with information from other components, e.g. by conveying a lack of information about æ or by hedging against a wide range of possible x rather than strongly associating with particular input other than the original x. OA does not entirely eliminate spoofing, since it may be the case that every possible value of A conveys at least weak information to the model. However, the excess ablation gap A Appt for A measured with ablation methods that replace A(x) with a (random) value A is entirely caused by spoofing, since replacing A(z) with the constant a" achieves lower loss without giving away any more information about x. In practice, A - Dops for prior ablation methods is typically very large compared to Aopt for both single components (see Table I and circuits (see Section 3.2) on prototypical language subtasks. This disparity indicates that effect 2 dominates the A measurements for previous ablation methods, making them poor estimators for effect TrrectTcompe compared to OA.
Subtask-specific methods often try to generate consistent interventions by conditioning on features of the input to avoid replacing A(x) with values that could confuse the model. For CF, choosing (a) to share many tokens with & mitigates the contribution of effect (2 AcF, which is the main reason the technique is so widely employed. Thus, among previous measures of component importance, ACF. when it can be well-constructed, may be the best quantification of effect [[ intuitive relation between OA and CF as techniques that aim isolate effect we perform a case that shows that among other ablation methods, OA produces the measurements
To demonstrate this
study in Section 2.4
See Appendix D.2 for an interesting connection of OA to mean ablation.

---

most similar to CF. However, not only is OA more general than subtask-specifie methods like CF, but oppt may still be better estimator for effect than Åcr even when CF well-defined. In Section we show that for circuits, A is much lower than Acr despite reflecting weakly stronger deletion effect, indicating that effect also contributes to Dopt less than it does to åcr, and thus Dopt is more accurate reflection of components" informational importance.
Computation of a'. Though it impossible to derive a in closed form, we find that in practice, mini-batch stochastic gradient descent (SGD) performs well at finding constants a that greatly reduce compared heuristic point estimates like zero and the mean. We generally adopt the approach of initializing each ā to the subtask mean Ex-p [A(X) and performing SGD to minimize A.
2.4 Comparison of single-component ablation results on 101
The Indirect Object Identification (IOI) subtask (Wang et al. 2022) consists of prompts like "When Mary and John went to the store, Mary gave the apple to 2019) completes with the correct indirect object noun ("John"). We use I0I as case study because is discussed extensively in interpretability work (Merullo et al. 2023; Makelov et al. 2023 Wu et al.,
which GPT-2-small (Radford et al.)
2024
Lan et al. 2024 Zhang and Nanda 2024). To implement CF, for each prompt 2, Wang et al. (2022 constructs a random (2) in which the names are replaced with random distinct names.
We evaluate A for attention heads and MIP blocks using zero, mean, resample, counterfactual, counterfactual mean, and optimal ablation. In Table we show that among attention heads and MLP blocks, Aopt accounts for only 11.1% % of A 33.0% of Amean. and 17.7% of A resample for the median component. Furthermore, among these A measurements, Apps has the highest highest rank correlation (0.907) with Acr. Full results are shown in Appendix E.3
Table 1: Comparison of ablation loss gap on I0I
Zero
Mean
Resample
CF-Mean Optimal
CF
Rank correlation with CF
0.590
0.825
0.828 17.7%
0.833
0.907
Median ratio of Aopt to A
11.1%
33.0%
31.7%
100%
88.9%
3 Application: circuit discovery
Circuit discovery is the selection of a sparse subnetwork of M that sufficient for the recovery of model performance on an algorithmic subtask D. To define what constitutes a "sparse subnetwork," we write M as a computational graph with vertices G and edges E. An edge eq := Ai, E E indicates that (z() is taken the 2th input to A, in the computation represented by the graph. To ablate edge cg. we replace the 2th input to A,, which is equal to A,(x) (r) during normal inference, with some value We compute Mx(X,, which represents modified inference with edges E \ E ablated, by applying this intervention for each ablated edge (see Appendices C.I and C.2
for more details).
Circuit discovery aims to select subset of edges E" C E such that
Ex = arg min EX-DL(M (X), M(X)) + R(B)] arg min ECE
B) B +(())
(3)
ÉCE
for a regularization term R that measures the sparsity level (further discussed in Appendix [.C]. Additionally, when implementing OA, though we could use a different constant for each edge, we instead define single constant a; for each verter Ay. so that if multiple out-edges from Aj
ablated, the same value is passed to each of its children (further discussed in Appendix F22)
3.1 Methods
We compare (MM, E É') measured with mean ablation, resample ablation, OA, and CF as metrics for circuit discovery. We consider the manual circuit for each subtask and circuits optimized on each metric using several search algorithms.
ACDC (Conmy et al. 2023) identifies circuits by iteratively considering edge ablations. They start by proposing E = E, then iterate over edges eg. ablating e* and updating E = É E } if the marginal impact loss, A (M,, (E U (ex)) E) A(M, E X (fex) U E)), is below tolerance threshold X.
5

---

Edge Attribution Patching (EAP) (Syed et al., 2023) selects E to contain the edges ej that have the largest gradient approximation of their single-edge ablation loss gap ((M,
HardConcrete Gradient Sampling (HCGS) is an adaptation of a pruning technique from Louizos et al. (2018) circuit discovery. Rather than considering only total ablation of an edge 6g (Aj, Ai, Z), we can consider a continuous mask of coefficients á and partially ablate ex by replacing the zth input to A, with a linear combination of the original value (2) and ablated value aj. i.c. (x) + (1 - ax)ajj. Now, Og- = 0 designates total ablation (replacing with aj), while ag = 1 designates total retention (keeping (z)).) We use (z) to represent the model with edges partially ablated according to ã.
Some previous work (iuu et al, 2017 Huang and Wang 2018) optimizes directly on the mask coefficients o, but to avoid getting stuck in local minima on ag E (0. 1), Louizos al. (2018) samples 0g from a HardConcrete distribution parameterized by location 0₂ and temperature 33 for each edge, and performs SGD with respect to the distributional parameters. In effect, we update the parameters based on gradients evaluated at randomly sampled values of āi rather than gradients evaluated at any exact ã. Cao et al. (202 applies this technique find circuits that consist of subset of model weights. [Commy et al. (2023) applies this technique to vertices in a computational graph. Unlike previous work, we apply this technique edges rather than vertices.
Uniform Gradient Sampling (UGS) is our proposed method for algorithmic circuit discovery. Similar to HCGS, we consider ablation coefficients ā and update parameters based on gradients evaluated at sampled values of ã. We keep track of a parameter 0% for each edge, where 8g = (1 + exp(- (o)())0))) indicates an estimated probability of e F" Using wr(x)) = O2 0x) to determine sampling frequency (further discussed in Appendix F.8). we let aj ~ Unif(O, 1) with probability (w.p.) w(xx),, 0% 1 w.p. O₂ (0x) and O 0 w.p. 1 0g 3 (w(()). For a batch
of b inputs (1). X()), let a()) denote the sampled coefficients corresponding to X and let E (0, We construct loss function ((UG)) whose gradient satisfies
N₂ = EE= (a (7)
E j=1 I(a E (0, V P (X1)), (4)
Vo, C(UG) Vo. R(0) N
and perform SGD on the 0R. where R00) is a continuous relaxation of REE) from Equation (}. In Appendix F.5 we motivate UGS as an estimator for sampling over Bernoulli edge coefficients.
Optimizing circuits on Aopt ACDC and EAP are not compatible with optimization on Aopt. since the optimal a' values depend on the selected circuit and it is intractable to optimize ā for every candidate circuit. For our circuit evaluations on Dopt. we compare to ACDC- and EEA--generated circuits optimized on Acr. On the other hand, HCGS and UGS allow us to perform SGD to optimize the ablation constants â concurrently with the sampling parameters.
3.2 Experiments
We study GPT-2-small performance on the IOI (Wang et al., 2022) subtask described in Section 2.4 and the Greater-Than (Hanna al. (203) subtask, which involves completing prompts such as "The conflict started in 1812 and ended in 18 with digits greater than the first year in the context. We select these settings because their exposition in manual studies is particulariy thorough and they are used in prior work (Conmy et al. 2023 Syed et al. 2023 to benchmark algorithmic circuit discovery,
We compare the algorithms in Section 3.1 trained to minimize A the IOI and Greater-Thar Than subtasks when edges E' E are ablated with mean ablation, resample ablation, OA, and CF. For IOI, the mapping π for CF is defined in Section 2.4 For Greater-Than, we continue with the practice from Hanna et al. (2023) of selecting counterfactuals (xx) by changing the first year in the prompt z to end in "01" so that all numerical completions are equally valid (see Appendix
UGS achieves Pareto dominance on the A-IEI tradeoff over the other methods on both subtasks for each ablation method, identifying circuits that achieve lower at any given [E] and vice versa. Results for IOI circuits optimized on ACF are shown in Figure (left). On IOI, UGS finds a circuit with 385 edges that achieves AcF = 0.220. This circuit has 52% fewer edges than the smallest ACDC-identifed circuit with comparable Acr and 48% lower ACF than the berf-perf ACDC-identifeed circuit with a comparable edge count. Similar improvements to the Pareto frontier,

---

I0I circuits, counterfactual ablation
101 circuits, ablation comparison
5
UGS (ours)
Optimal
HCGS
Mean
2
EAP
2
Resample
ACDC
CF
Abi ablatoon loss gap A
Ablation loss gap A
1-
manual
1-
0.5
x
0.5
0.2
0.2
0.1
0.1
0.05
0.05
250
500
750
1000
1250 1500
1750
0
250
500
750
1000
1250
1500
1750
Circult edge count [E]
Circult edge count (É]
Figure 1: Left: Circuit discovery Pareto frontier for the IOI subtask with counterfactual ablation. Right: Comparison of ablation methods for circuit discovery on IOI (X indicates manual circuit evaluated on each ablation method). A is measured in KL--diergence.
shown in Appendix F.10. occur for mean, resample, and optimal ablation. UGS also creates Pareto improvements for Greater- Greater-Than circuits for each ablation method; see Appendix F.IT
Applying OA to circuit discovery reveals that certain sparse circuits can account for model perfor- mance on these subtasks to a much greater extent than previously known. We visualize the A for each ablation method achieved by UGS- -ide-iiettiieed circuits in Figure (right). Using OA to ablate excluded components, we find circuits that recover much lower Á any given circuit size than any circuit for which excluded components are ablated with any other ablation method. For example, for IOI, at a circuit size of 1,000 edges, ablating excluded components with OA enables the existence of circuits with 32% lower A compared to CF, 62% lower A compared to mean ablation, and 88% lower A compared to resample ablation, and the improvement even larger at smaller circuit sizes. For Greater- -haneer--Thhnnaa (results shown in Appendix F.III. OA again admits circuits with by far the lowest among the four ablation methods. Thus, OA paints a more accurate and compelling picture of how much small subsets of the model can explain behavior these subtasks.
Unlike other ablation methods, OA indicates that the manual circuits are approximately optimal for their size. Holding [E] fixed, the careto-optimal Dopt is 29% below the Aopt of the manual circuit on IOI and 42% below the Aope of the manual circuit on Greater-Than However, for the other ablation methods, optimized circuits with fewer edges than the manual circuit achieve 84-85% lower A than the manual circuit on 1OI, and 70-84% lower A on Greater-Than. Since the manual circuits are selected using a thorough mechanistic understanding of the model for each subtask and thus arguably capture the important components, this finding furthers the notion that A measured with previous methods could be artificially high due to spoofing by ablated components, and therefore Dopt is a superior evaluation metric for circuits.
These results show that Aopt is useful for evaluating and discovering circuits and provide evidence that OA better quantifies the removal of important mechanisms than previous ablation methods.
4 Application: factual recall
Transformers can store and retrieve large corpus of factual associations. One goal in interpretability localizing factual recall, or identifying components that store specific facts. To this end, [eeng et al.) (2022) proposes causal tracing, which involves removing important information about the prompt and evaluating which components can recover the original M(xx). To isolate components responsible for an association between subject (e.g. "Eiffel Tower") and an attribute ("located in Paris"), they select prompt x ("The Eiffel Tower is located in the city of __") that elicits from M correctly memorized response y ("Paris"). They produce corrupted input (GN(x) by adding a Gaussian noise (GN) term Z (O 9Σ), to all token embeddings that encode the subject, where Σ is a diagonal
7

---

matrix and Σn represents the variance of the ith neuron among token embeddings sampled from the training distribution. Letting [M(z]ly represent the probability assigned by M(xx label y. Since Sax partially ablates information about the subject, (M(Eox (z))l, is typically much smaller than [M(r]]y For each component A, they estimate its contribution to the recall of y with the following "average indirect effect" (AIE) representing the proportion of probability on the correct y recovered by replacing ((E(X)) with A(X), averaged over (X, Y) ~ D. where & = CGX
AIE(A) == min 0, B(XY)~ max(0, [M(X]l MM, ((x)))))))()))))) A
(5)
Exx,Y)~ [M(X)]y ExxY)) [M(E(X))l
where we declare AIE(A) = 0 if the denominator is non-positive and ablating subject tokens actually helps identify the correct label (however, this is never the case).
Method We perform causal tracing by removing the subject with optimal ablation (OA-tracing, or OAT) rather than with Gaussian noise (GNT). We define {(x: = ćoa (2t, a.)) by replacing subject token embeddings with a constant 0 trained to minimize the numerator in Equation ( which represents A with a carefully chosen loss function (see Appendix G.2).
Causal tracing intervention at last token, window size 5
Attention layers
layers
0.4
GNT OAT
0.4
GNT
QAT
0.2
0.2
0.0
0.0
0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last token, window size 1
Attention layers
MLP layers
GNT OAT
GNT
AIE
0.1
AIE
0.1
OAT
0.0
0.0
40
Layer number
Layer number
Figure 2: Comparison of AIE with GNT and OAT. In the top figure, layer the x-axis represents replacing a sliding window of 5 layers with as the median. Error bars indicate the sample estimate plus/minus two standard errors (details given in Appendix G.4).
G.4).
Experiments We compare GNT and OAT for GPT-2-XL on a dataset of subject- prompts from Meng al. (2022) for which the model completes the correct answer via sampling with tem- perature O. To increase the sample size, we augment the data with similarly constructed prompts from follow-up work factual recall (Hernandez al. (022). We train OAT 60% of the dataset and evaluate both methods on the other 40%. On the test set, E[M(X)]r ((M(EcN(XX))
30.6%, 8.7%. We let A(X) represent an atten-
12.3%, and [[M(Eoa tion or MLP layer output at a certain token position(s): namely, all subject token positions, only the last subject token position, and only the last token position in the entire sequence. Rather than considering only one layer at a time, Meng et al. (002)) lets A represent the outputs of a sliding window of several consecutive attention layers or MLP layers. Thus, in addition to replacing the output of a single layer (window size 1), we show results for replacing windows of sizes 5 and 9.
OAT offers more precise localization of relevant components compared to GNT. While GNT indicates a small positive AIE for most components, OAT shows a few components have large contributions while most have little no effect. For example, Figure 2 (top left) shows that the AIE for a window of 5 attention layers at the last token is as high as 42.6% for the window consisting of layers 30-34, while the AIE peaks at only 20.2% for GNT. On the other hand, for windows centered around layers 15-23, the average AIE for OAT is only 1.7% indicating little effect for these potentially
*Unlike Meng et al. 2022). we clip [MX)]]y - (M A(X))Jy to be non- negative, so we do not give A additional credit for increasing the probability mass of the true label past that given by the full model. We also report AIE in proportion probability recovered compared to the full model rather than percentage points.
8

---

unimportant layers, compared to 7.0% for GNT. For sliding windows of 9 attention layers at subject token positions, GNT shows marginally positive AIE measurements across layers 0-30, but OAT specifically shows highly positive AIE for layers 0-5 and 25-30 (see Figure 14 Moreover, whereas Meng et al. (2022) focuses on sliding window replacement because GNT effects from single-layer replacements are very small, OAT can sometimes identify information gain from just one layer. For instance, at the last token position, OAT records AIEs above 8% for each of attention layers 30, 32, and 34 by themselves (see Figure bottom left), much greater than the AIE of the other layers. This greater level of granularity opens up the possibility of selectively investigating combinations of layers as opposed to relying the prior that adjacent layers work together.
5 Application: latent prediction
One practice in interpretability is eliciting predictions from latent representations. Let M have layers O.....N N and let ((X) be the residual stream activation at the last token position (LTP) after layer i.
Logit attribution (Geva et al., 20222] Wang et al., 2022; Dar et al. 2023 Katz and Belinkov 2023
Dao et al. 2023; Merullo et al., 2024 Halawi et al. (024) is the practice of applying a transformer
model's unembedding map to activation to obtain semantic interpretation of that activation.
When applied to the LTP activation after layer i, this practice is equivalent to zero ablating layers i ++] 1
to N. However, the semantic meanings of LTP activations after layer N can be different from those of LTP activations in earlier layers. As an alternative, tumed lens (Belrose et al. 2023a Din et al. 2023)
is linear map f:( = W + by that "translates" from (x) to predicted EN (X). MT1.(X
is defined by replacing E (X) with (X) == fx))((X) during inference, and training W; and b; to minimize LTi. == Ex (X),, M(X). Tuned lens demonstrates when information is transferred to LTP: if replacing E (X) with 2x(X) achieves low loss, then (X) contains sufficient context for computing M so key information is transferred prior to layer i.
Method We propose Optimal Constant Attention (OCA) lens. We define MOCa (X) by using OA to ablate attention layers i + to N: for each of these layers k2 we replace its output at LTP with constant ās.. We train = (ā.+1...., āx) to minimize Ex COCA == Ex (X
Similar to tuned lens, OCA lens reveals whether the LTP activation after layer contains sufficient context to compute (XX by eliminating information transfer from previous token positions to LTP after layer i. While tuned lens is a linear map, OCA lens is function that leverages the model's s existing architecture (specifically, its MLP layers) to translate between LTP activations at different layers. OCA lens has far fewer learnable parameters than tuned lens: O O(Ndmodal) < (d
Experiments We compare LOCA to LrL for various model sizes. As additional baselines, we also consider the ablation of later attention layers with mean or resample ablation rather than OA. Results are shown in Figure (left) for EP and Figure [5] for other model sizes. OCA lens achieves significantly lower loss than tuned lens, indicating better extraction of predictive power from LTP activations. For example, the predictive loss of OCA lens drops to below 0.01 around layer 35 of GPT-2-XI, but does not reach this point even at the last layer for tuned lens.
Lens loss, GPT-2-XL
Causal faithfulness: basis projection
OCA lens
Magnitude correlation
1.0
Direction similerity
GPT 2 small, Tuned GPI-2-small, GPt:2-medlum, Tuned GPT:2-medlum, OCA GPT-2-large. Tuned GPT-2-lerge. OCA
KL. dlvdivereence
Tunedl lons
Mean
0.B
0.75
Resample
0.6
0.50
0.4
GPT XL, Tuned
GPT:2XXL, OCA
0.00
20
40
20
40
20
40
Layer number
Layer number
Layer number
Figure 3: Left: Prediction loss comparison between tuned lens and ablation- alternatives. Middle, right: Causal faithfulness metrics for tuned and OCA lens under basis-aligned projections.
Additionally, [Belrose et al. (2023a) explains that one desiderata for latent prediction causal faithfulness, i.e. // should use (X) in the same way as M. We can investigate causal faithfulness by intervening on (X) and evaluating the extent to which MTL (X) and M(X) move in parallel.

---

MX)(X) changes significantly but but (XX) does not, for example, then f. could be extrapolating from spurious correlations, e.g. by inferring from directions that predict information transfer that occurs in later layers. Consider random intervention & on (x()) and let MTI(X; represent replacing (x)) with (X)) before applying fi- Similarly, let :M;;)) represent replacing (x) with ((), during inference. [Belrose et al. (203a) separates causal faithfulness into two measurable properties (both range from -1 to and higher values reflect greater faithfulness):
1. Magnitude correlation: cBr(( MM.
5), E))
2. Direction similarity: ([(M (X; MrL(X), M(X: e M(X)), where @ denotes subtraction in logit space and () denotes the Aitchinson similarity between distributions.
We assess these properties for MrL and MOCA for a variety of interventions Ç. In Figure 3 (middle, right), we plot these properties for modified version of the "causal basis projection" { from Belrose et al. (202a)). While they train a basis iteratively, this approach is expensive and unstable, and we instead extract an approximate basis for MTL by performing singular value decomposition (SVD) on W/EV/, where E is the covariance matrix of &, (X), and applying / to the right singular vectors. For MOCA, we extract this basis by training a linear map to approximate f. and using the weights as the W, For both lenses, we compute {(a) = f + p(a (t), where μ E[C,( (X) and p represents projecting to the orthogonal complement of span( (77) for a uniformly sampled basis vector T. We plot the magnitude correlation and direction similarity for MrI and MOCA with respect to M in Figure 3 We find that OCA lens measures significantly better on both causal faithfulness metrics across all layers, and we achieve similar results for other choices of interventions (see Appendix H..)).
One downstream application of extracting latent predictions from intermediate-layer LTP activations is that they sometimes be more accurate text classification tasks than the model's output predictions, especially if the context contains false demonstrations, i.c. examples of incorrect task completions (Halawi al. 2024). The proposed theory is that the model first computes the correct answer at I.TP in early layers, then later layers move contextual information to LTP that lead it to make adjustments that benefit next-token prediction, such reporting incorrect answer for consistency with false demonstrations. We compare the elicitation accuracy boost, or the best elicitation accuracy across layers minus the accuracy of the model output, for OCA lens and tuned lens for GPT-2-XL. with 2,000 classification samples from each of 15 text classification datasets from Halawi et al. (2024). using their calibrated accuracy metric. We find that OCA lens increases this accuracy boost for prompts with true demonstrations on 12 of the 15 datasets and for prompts with false demonstrations on 11 of the 15 (see Figure 21). In particular, for Wikipedia topic classification (DBPedia), OCA lens increases the elicitation accuracy boost from 2.9% to 18.0% with true demonstrations and from 19.2% 28.8% with false demonstrations (see Figure 4 middle). Full results are reported in Appendix H.4
Elicitation accuracy selected datasets with 10 demos, GPT-2-XL
Average of 15 datasets
DBPedia
MRPC
Calibrated accuracy
0.55
0.8
0.55
True demos, OCA True demos, Tuned
0.50
0.6
True demos, full model
0.50
0.45
0.4
False demos, OCA
0.40
0.45
False demos, Tuned
False demos, full model
20
Layer number
Layer number
Layer number
Figure 4: Comparison of calibrated elicitation accuracy on selected datasets.
6
Future work
The applications of component importance presented in our work are not exhaustive. A variety of interpretability work either directly applies ablation-based based importance or can be framed to use it as a potential tool. OA creates new opportunities to incorporate ablation into studies for which it may be impossible to obtain good results with previous ablation methods. For example, we can train probes derived from using OA with different loss functions (Li et al. (023), or use an approach similar to OCA lens to decode activations other than the LTP residual stream activation. See Appendix [D.3 for an extension of OA to evaluate the extent to which a component performs classification.
10

---

Acknowledgements
LJ was partially supported by DMSS2045981 and DMS-2134157.
References
Baan, J., ter Hoeve, M., van der Wees, M., Schuth, A., and de Rijke, M. (2019). Understanding multi-head attention in abstractive summarization.
Bach, S., Binder, A., Montavon, G., Klauschen, F., Muller, K.R., and Samek, W. (2015). On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One, (01))=e0301400
Baehrens, D., Schroeter, T., Harmeling. S., Kawanabe, M., Hansen, K., and Mueller, K.-R. (2009). How to explain individual classification decisions.
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017). Network dissection: Quantifying interpretability of deep visual representations.
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. (2020). Understanding the role of individual units in deep neural network. Proceedings the National Academy of Sciences, 11 117048):3071133077
Belrose, N., Furman, Z., Smith, L., Halawi, D., Ostrovsky, I., McKinney, L., Biderman, S., and Steinhardt, (2023a). Eliciting latent predictions from transformers with the tuned lens.
Belrose, N., Schneider-Joseph, D., Ravfogel, S., Cotterell, R., Raff, E., and Biderman, (2023b). Leace: Perfect linear concept erasure in closed form.
Breiman, L. (2001). Random forests. Machine Learning, 45(1):5-32.
Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering latent knowledge in language models without supervision.
Cao, S., Sanh, V., and Rush, A. M. (2021). Low-complexity probing via finding subnetworks. arXiv preprint arXiv:2104.035/1
Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B., and Thomas, N. (2022). Causal scrubbing: A method for rigorously testing inter- pretability hypotheses. htttps://ww.. http://www.llllmmeefffrruu wwwprrumm..gppp..eeeeeeee hogg:pppste///lllaaych222r5 causal aec testing
Chang, C.-H., Creager, E., Goldenberg, A., and Duvenaud, D. (2019). Explaining image classifiers by counterfactual generation.
Chen, H., Feng, S., Ganhotra, J., Wan, H., Gunasekara, C.,, Joshi, S., and Ji, Y. (2021). Explaining neural network predictions sentence pairs via learning word-group masks.
Chen, J.. Song, L., Wainwright, M. J., and Jordan, M. (2018). Learning to explain: An information- theoretic perspective on model interpretation.
Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., and Garriga-Alonso, A. (2023). Towards automated circuit discovery for mechanistic interpretability.
Covert, I. Lundberg, S., and Lee, S.-L (2020). Understanding global feature contributions with additive importance measures.
Covert, C., Lundberg, S., and Lee, S.-I. (2022). Explaining by removing: A unified framework for model explanation. J. Mach. Learn. Res., 22(1).
Cunningham, H., Ewart, A., Smith. L. R., Huben, R., and Sharkey, L. (2023). Sparse autoencoders find highly interpretable model directions.

---

Dabkowski, P. and Gal. Y. (2017). Real time image saliency for black box classifiers. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H.. Fergus. R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Dao, J., Lau, Y.TT., Rager, C., and Janiak, J. (2023), An adversarial example for direct logit attribution: Memory management in gelu-41.
Dar, G., Geva, M., Gupta, A., and Berant, J. (2023). Analyzing transformers in embedding space.
Datta, A., Sen, S., and Zick, Y. (2016). Algorithmic transparency via quantitative input influence:
Theory and experiments with learning systems. In 2016 IEEE Symposium on Security and Privacy (SP), pages 598-617.
De Cao, N., Schlichtkrull, M., Aziz, W., and Titov, I. (2021), How do decisions emerge across layers in neural models? interpretation with differentiable masking.
Dhamdhere, K., Sundararajan, M., and Yan, Q. (2018). How important a neuron?
Din, A. Y., Karidi, T., Choshen, L., and Geva, M. (2023). Jump to conclusions: Short-cutting transformers with linear transformations.
Fisher, A., Rudin, C., and Dominici, F. (2019). All models wrong, but many useful: Learning variable's ' importance by studying an entire class of prediction models simultancously.
Fong, R., Patrick, M., and Vedaldi, A. (2019). Understanding deep networks via extremal perturba- tions and smooth masks.
Fong, R. C. and Vedaldi, (2017). Interpretable explanations of black boxes by meaningful perturbation. In 2017 IEEE International Conference Computer Vision (ICCV). IEEE.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. (2023). Dissecting recall of factual associations in auto-regressive language models.
Geva, M., Caciularu, A., Wang, K. R., and Goldberg, Y. (2022). Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.
Ghorbani, A. and Zou, (2020). Neuron shapley: Discovering the responsible neurons.
Goldowsky- Dill, N., MacLeod, C., Sato, L., and Arora, A. (2023). Localizing model behavior with path patching. arXiv preprint arXiv:2304.0596)
Gould, R., Ho, E., and Conmy, A. (2023). Mechanistically interpreting time in gpt-2 small.
Grömping, U. (2007). Estimators of relative importance in linear regression based variance decomposition. The American Statistician, 1192)119-1147.
Gromping, U. (2009). Variable importance assessment in regression: Linear regression versus random forest. The American Statistician, 63(4))308-315
Guan, C.,, Wang, X., Zhang, Q., Chen, R., He, D.,, and Xie, (2019). Towards a deep and unified understanding of deep neural models in NLP. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2454-2466 PMLR.
Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. R., Sun, Q., Hathaway, W., Nanda, N., and Bertsimas, D. (2024). Universal neurons in gpt2 language models.
Gurnee, W. and Tegmark, M. (2024). Language models represent space and time.
Halawi, D., Denain, J.-S., and Steinhardt, J. (2024). Overthinking the truth: Understanding how language models process false demonstrations.
Hanna, M., Liu, O., and Variengien, A. (2023). How does gpt-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model.
12

---

Hase, P., Bansal, M., Kim, B., and Ghandeharioun, A. (2023). Does localization inform editing? surprising differences causality-based localization vs. knowledge editing language models.
Hase, P.. Xie, H., and Bansal, M. (2021). The out-of-distribution problem in explainability and search methods for feature importance explanations.
Heimersheim, S. and Janiak, J. (2023). A circuit for Python doestrings in a 4-yyer attention- www.:aaligniin only transformer. http:::/ww.allggmeeetteooe http://wa..aattaoo...ooooooottta
a-
Heimersheim, S. and Nanda, N. (2024). How to use and interpret activation patching.
Hendel, R., Geva, M., and Globerson, A. (2023). In-context learning creates task vectors.
Hernandez, E., Schwettmann, S., Bau, D., Bagashvili, T., Torralba, A., and Andreas, J. (2022). Natural language descriptions of deep visual features.
Hernandez, E., Sharma, A. S., Haklay, T., Meng. K., Wattenberg, M., Andreas, J., Belinkov, Y., and Bau, D. (2024). Linearity of relation decoding transformer language models.
Homma, T. and Saltelli, A. (1996). Importance measures in global sensitivity analysis of nonlinear models. Reliability Engineering & System Safery, 52(1):1-17.
Hooker, S., Erhan, D., Kindermans, P.-J., and Kim, B. (2019). benchmark for interpretability methods in deep neural networks.
Huang, Z. and Wang, N. (2018). Data-driven sparse structure selection for deep neural networks.
Ishwaran, H. (2007). Variable importance in binary regression trees and forests. Electronic Journal of Statistics, (none).
Janzing, D., Minorics, L.,, and Blöbaum, P. (2019). Feature relevance quantification in explainable ai: A causal problem.
Katz, S. and Belinkov, Y. (2023). Visit: Visualizing and interpreting the semantic information flow of transformers.
Kim, S., Yi, J., Kim, E., and Yoon, S. (2020). Interpretation of nlp models through input marginaliza- tion.
Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., and Baroni, M. (2019). The emergence of number and syntax units in Istm language models.
Lan, M.. Torr, P., and Barez, F. (2024). Towards interpretable sequence continuation: Analyzing shared circuits in large language models.
Leino, K., Sen. S., Datta, A., Fredrikson, M., and Li, (2018). Influence-directed explanations for deep convolutional networks.
Li, C. and Mahadevan, S. (2017). Sensitivity Analysis of Bayesian Network. ASCE-ASME J Risk and Uncert in Engrg Sys Part Mech Engrg, 4(1). 1003.
Li, J.. Monroe, W., and Jurafsky, D. (2017). Understanding neural networks through representation erasure.
Li, K., Hopkins, A. K., Bau, D., Viégas, F., Pfister, H., and Wattenberg, M. (2023), Emergent world representations: Exploring sequence model trained on a synthetic task.
Li, M., Davies, X., and Nadeau, M. (2024). Circuit breaking: Removing model behaviors with targeted ablation.
Lieberum, T., Rahiz, M., Kramár, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V. (2023). Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla.
Liu, 7.,, Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C. (2017). Learning efficient convolutional networks through network slimming.
13

---

Louizos, C., Welling, M., and Kingma, D. P. (2018). Learning sparse neural networks through lo regularization.
Lundberg, S. M., Erion, G., Chen, H., DeGrave, A., Prutkin, M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S.-I. (2020). From local explanations to global understanding with explainable for trees. Nature machine intelligence, 2(1))56-67.
Lundberg, S. M. and Lee, S.-L. (2017). unified approach to interpreting model predictions. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus. R., Vishwanathan, S., and Garnett, R., editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Makelov, A., Lange. G., and Nanda, N. (2023). Is this the subspace you are looking for? an interpretability illusion for subspace activation patching.
Marks, S., Rager. C., Michaud, E. J., Belinkov. Y., Bau, D., and Mueller, A. (2024). Sparse feature circuits: Discovering and editing interpretable causal graphs in language models.
Marks, S. and Tegmark, M. (2023). The geometry of truth: Emergent linear structure in large language model representations of true/false datasets.
Mase, M., Owen, A. B., and Seiler, B. B. (2024). Variable importance without impossible data. Annual Review of Statistics and Its Application. 11(Volume 11, 2034):153-178
McDougall, C., Conmy, A., Rushing, C., McGrath, T., and Nanda, N. (2023), Copy suppression: Comprchensively understanding an attention head.
McGrath, T., Rahtz, M., Kramar, J., Mikulik, V., and Legg, S. (2023). The hydra effect: Emergent self-repair in language model computations.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. (2022). Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-1737..
Merullo, J., Eickhoff, C., and Pavlick, E. (2023). Circuit component reuse across tasks in transformer language models.
Merullo, J., Eickhoff, C., and Pavlick, E. (2024). Language models implement simple word2vec-style vector arithmetic.
Montavon, G., Lapuschkin, S., Binder, A., Samek, W., and Müller, K.-R. (2017). Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65:211 1222..
Mu, J. and Andreas, J. (2021). Compositional explanations of neurons.
Nanda, N. (2023). Attribution patching: Activation patching at industrial scale.
Nathans, L., Oswald, F., and Nimon, K. (2012). Interpreting multiple linear regression: A guidebook of variable importance. Practical Assessment, Research and Evaluation, 17(9):1-19.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield Hatfeel-Dodds, Z., Hernandez, D.,, Johnston, S., Jones, A., Kermion, J., Lovitt, L.. Ndousse, K., Amodei, D... Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. (2022). In-context learning and induction heads.
Petsiuk, V., Das, A., and Saenko, K. (2018). Rise: Randomized input sampling for explanation of black-box models.
Rabitz, H. (1989). Systems analysis at the molecular scale. Science, 246(4927):21-2266
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners.
Ribeiro, M. T. Singh, S., and Guestrin, C. (2016). "why should i trust you?": Explaining the predictions of any classifier.
Robnik-Sikonja, and Kononenko, (2008). Explaining classifications for individual instances. Knowledge and Data Engineering, IEEE Transactions on, 20:589 600.
14

---

Rushing, C. and Nanda, N. (2024). Explorations of self-repair in language models.
Räuker, T., Ho, A., Casper, S., and Hadfield-Menell, D. (2022). Toward transparent ai: A survey interpreting the inner structures of deep neural networks.
Schlichtkrull, M. S., De Cao, N., and Titov, I. (2022). Interpreting graph neural networks for nlp with differentiable edge masking.
Schulz, K., Sixt, I.., Tombari, F., and Landgraf, T. (2020). Restricting the flow: Information bottlenecks for attribution.
Schwab, P. and Karlen, W. (2019). Cxplain: Causal explanations for model interpretation under uncertainty.
Shah, H., Ilyss, A., and Madry, A. (2024). Decomposing and editing predictions by modeling model computation.
Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Deep inside convolutional networks: Visualis- ing image classification models and saliency maps.
Singh, A. K., Moskovitz, T., Hill, F., Chan, S. C. Y., and Saxe, A. M. (2024). What needs to go right for induction head? mechanistic study of in-context learning circuits and their formation.
Slack, D., Hilgard, S., Jia, E., Singh. S., and Lakkaraju, H. (2020). Fooling lime and shap: Adversarial attacks on post hoc explanation methods.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise.
Sobol, I. (1993). Sensitivity estimates for nonlinear mathematical models. Computational Mathemat- ics and Mathematical Physics, 1(4):407-413.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, and Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1929-199
Stolfo, A., Belinkov, Y., and Sachan, M. (2023). mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.
Strobl, C., Boulesteix, A.-L., Kneib, T., Augustin, T., and Zeileis, A. (2008). Conditional variable importance for random forests. BMC bioinformatics, 9:1-11.
Strumbelj, E. and Kononenko, I. (2010). An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11(1):1-18.
Strumbelj, E., Kononenko, I., and Sikonja, M. R. (2009). Explaining instance classifications with interactions of subsets of feature values. Data and Knowledge Engineering, 68(10):886-904.
Sundararajan. M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks.
Syed, A., Rager, C., and Conmy, A. (2023). Attribution patching outperforms automated circuit discovery.
Tigges, C., Hollinsworth, O. J., Geiger, A., and Nanda, N. (2023). Linear representations of sentiment in large language models.
Todd, E., Li, M. I.., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. (2024). Function vectors in large language models.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, (2017). Attention is all you need.
Vig, J., Gehrmann, S., Belinkov, Y., Qian, S., Nevo, D., Singer, Y., and Shieber, S. (2020). Inves- tigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems, 2388-111-2
15

---

Wang, K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. (2022). Interpretability in the wild: a circuit for indirect object identification in gpt-2 small.
Wei, C., Kakade, S., and Ma, T. (2020). The implicit and explicit regularization effects dropout.
Williamson, B. D. and Feng, J. (2020). Efficient nonparametric statistical inference on population feature importance using shapley values.
Wu, Z., Geiger, A., Huang, J., Arora, A., Icard, T., Potts, C., and Goodman, N. D. (2024). A reply makelov et al. (2023''s "interpretability illusion" arguments.
Ye, J., Lu, X., Lin, Z., and Wang, J. Z. (2018). Rethinking the assumption in channel pruning convolution layers.
Yoon, J., Jordon, J., and Van der Schaar, M. (2018). Invase: Instance-wise variable selection using neural networks. In International conference on learning representations.
Zeiler, M. D. and Fergus, R. (2013). Visualizing and understanding convolutional networks.
Zhang, F. and Nanda, N. (2024). Towards best practices of activation patching in language models: Metrics and methods.
Zhang, L. and Janson, L. (2020). Floodgate: Inference for model-free variable importance. arXiv preprint arXiv:2007.01283.
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2015). Object detectors emerge in deep scene cnns.
Zhuang. T., Zhang, Z., Huang, Y., Zeng, X., Shuang, K., and Li, X. (2020). Neuron-level structured pruning using polarization regularizer. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 9865-9877. Curran Associates, Inc.
Zintgraf, I.. M., Cohen, T. S., Adel, T., and Welling, M. (2017). Visualizing deep neural network decisions: Prediction difference analysis.
16

---

A Limitations
As is the case for previous work, we do not provide an entirely precise definition of the "importance" of a component. The importance of a component can be generally described as an aggregation of causal effects in a way that summarizes the component's 's contribution to model performance. Among the many ways aggregate causal effects, there may not be a mathematically rigorous way show that one measure of importance produces the correct or canonical aggregation. However, component importance is useful for wide variety of applications interpretability, so aside from showing that our approach to component importance better captures relevant considerations in a conceptual sense, we focus on the utility that provides to some of these applications.
As noted in Section 2.3, optimal ablation does not entirely eliminate the contribution of information spoofing to A. For example, if A typically conveys strong and exact information about the input, then there may not exist any value of a that hedges between a range of inputs.
Though OA produces circuits that achieve lower loss at given level of edge sparsity, it may elicit mechanisms that were not previously used for some subtask, especially there are multiple computational paths that could lead to the same conclusion. However, if the subtask of interest is sufficiently complex, seems unlikely that a model would have many "dormant" mechanisms that can be repurposed to perform the subtask, because this redundancy wastes computational complexity.
For factual recall, remains to be seen whether localization helpful for applications that are further downstream such as producing surgical model edits (Hase et al. 2023 Shah et al. [024).
B Additional related work
As mentioned in Section 2.2 component importance is strongly related to variable importance, which quantifies the importance of model input X; (also known as a feature or covariate in the variable importance literature).
Variable importance Much of variable importance work concerns "oracle" prediction, which roughly considers how much X; contributes to the performance of the best possible predictor for Y given the set of covariates (X1,... .... X.))) and frames the importance of X, as property of the joint distribution of (X1, ...., X,, Y). rather than a property of any particular model used for prediction. Most work in this area analyzes some parametric class of estimators, like linear models (Grömping) 2007: Nathans et al., 2012) or Bayesian networks (Li and Mahadevan (017). Later work generalizes parametric variable importance to arbitrary model classes, e.g. by training an ensemble of models that only have access to subsets of the covariates (Strumbelj et al. 2009 Fisher et al. 2019). Recent work has also studied non- non-parametric variable importance, in which we attempt to lower-bound the best performance of any arbitrary estimator (Williamson and Feng 2020 Zhang and Janson 2020).
On the other hand, our motivation is to interpret the behavior of one specific model M (Fisher et al. 2019; Hooker et al., 2019), not to analyze the theoretical relationship between model inputs and outputs. Rather than estimating how well any function of X; can predict Y, we wish to estimate how much the particular function M uses an input feature X; to predict Y. Previous work on this "algorithmic" variant of variable importance has taken two main approaches.
Local function approximations One way to quantify how much X; contributes to model per- formance is to aggregate local function approximations, which approximate the model around a particular input. Common tools for local approximation include the gradient of M at given
(Rabitz] 1989; [Bahrrens et al., 2009) Simonyan et al., 2014 Leino et al., 2018 Nanda 2023), or linear function that well-approximates M( + e) for chosen noise term & (Ribeiro et al. 2016 Smilkov et al. 2017). Since these tools often yield straightforward estimates of the local importance of X, for the input x, one approach to quantifying the global importance of X, is aggregate the importance estimates given by these local approximations, for example by using a first-degree Taylor approximation around reference input 2o (Bach et al. 2015 Montavon al. 2017). integrating over gradients along a straight-ling path from Ty to any studied input x (Sundararajan et al. 2017] Dhamdhere et al. 2018). This approach to measuring variable importance works just well for internal components as it does for inputs. However, local function approximations can fail to capture the overall loss landscape, especially in the common setting where M has unbounded gradients, and
17

---

can often be manipulated to produce arbitrary feature importance values (Slack et al. 2020
Hase
al. 202T).
Ablation-based measures The second main approach considers the ablation of feature X, In this approach, the feature X; is ablated by replacing with different random variable X; that captures less information about the original feature value. We then compare the model performance when X; is replaced with X, to the original model performance, per the definition of A in Section 2.1 (where the ablated component A is feature X, of the model input).
Many of the current methods used for ablating internal model components as described in Section 2.2 were first introduced in feature importance work. Zero ablation (Dabkowski and Gal 2017
Li et al. 2017 Petsiuk et al., 2018; Schwwab and Karlen 2019), mean ablation (Zeiler and Fergus
2013 Zhou et al. 2015), and Gaussian noise injection (Fong and Vedaldi F et al. 2019
Guan et al. 2019 Schulz et al.
2020) are all used to remove input features, such as the pixels of an image or tokens of text input, to assess their importance. Resample ablation is also common in feature importance work; an early variant samples X; from uniform distribution (Sobol 1993
Homma and Saltelli 1996. Strumbelj and Kononenko (010), while later work generally performs resample ablation on features by resampling them from their marginal distribution (Breiman 1001
Sibnik-Sikonja and Kononenko| 2008 Datta et al. 2016 Lundberg and Lee| Janzing et al.
2019 Covert et al. 2020 Kim et al. 2020).
Measuring feature importance via these ablation methods suffers from well-documented "out-of- distribution" problem (Ishwaran| [2007 Fong and Vedaldi, [017 Hooker et al., 2019; Hase et al. 2021 Mase et al. 2024):: since setting X; to zero or its mean, resampling X, from its marginal distribution, or adding Gaussian noise to X; could result in an input that was never observed during training, the measured feature importance values could potentially be determined by model behavior on impossible and/or nonsensical inputs. One way to mitigate this out-of--istribute
problem is replacing feature X; by a random variable X, sampled from its conditional distribution (Strobl et al. 2008; Lundberg et al. 2020). i.. X; ~ X, (X; TL X,) where X_, denotes the other Teatures X1, X X,,,,XX-11 X1+1; Xn. Since the conditional distribution is often intractable to sample from, previous work employs a range of approximation techniques. For example, Zinngraf et al.] (2017) samples an ablated pixel from its conditional distribution given É x E patch of its proximate pixels instead of conditioning on the entire image, and Chang et al. (2019) uses a generative model to simulate the conditional distribution.
However, a setting where the relevant features X; represent internal model components, rather than inputs to the model, it often does not make sense to discuss an -ittoo-distrrbutionn problem because the (X1.... Xx)) = (A1(X) ..., An (X)) are usually deaaddetermiisttcaally related to each other. For example, for any neuron, it is typically the case that its value can be almost deterministically recovered from the values of other neurons in the same layer. Thus, nearly any intervention on an internal component A; brings the model diitrffdissribbtioon." in the sense that the model observes the vector (AI(X),... An, (X)) where (X) is replaced with a with near-zero probability density.
Our dichotomy of deletion and spoofing in Section 2.1 is more precise than the typical discussion of the out-of-distribttion problem in its description of distortions to importance values that we wish to avoid. On one hand, our analysis is more lenient than the blanket requirement that the vector of all internal component values (A1(z) (x). A.. (x)) is in in-distribution, in the sense that not all interventions that bring (A1 (z), ..... An (x)) out of distribution constitute spoofing; for example, replacing A, (x) with a does not have a spoofing-relae related contribution to A if A. (z) and a are equivalent for the sake of downstream computation. On the other hand, our analysis is more stringent in the sense that effect 2a from Section 2. is recognized as a form of spoofing that can occur even when interventions are in-distribution (see Appendix [D] for more details).
Using dropout to eliminate spoofing One way to eliminate spoofing when intervening on A(X) is to train M to accept neutral constant values that indicate that component A has stopped functioning and then replace AA with these built-in neutral values to assess the importance of A. Variations of this technique are common in feature importance (Strumbelj et al. 2009, Chen et al.) 2018 Yoon et al. 2018 Hooker al. 2 For internal components, we could train neural networks with dropout (Srivastava et al. 2014| Wei et al., 2020), and then use zero ablation to assess the importance of A.
Since the downstream computation is trained to recognize A(X) as an indication that A carries
18

---

no information, as opposed to strong information associated with an input other than the original X, A Axrro (M, A) becomes an accurate assessment of deletion (effect | from Section 2.1p.
However, re-training with neutral values does not necessarily assist in analyzing particular model M, since re-training M will in general change M itself. Furthermore, training with dropout incentivizes M to lower Zero (M, .A) for any component A, since part of the loss function involves minimizing loss with random subset of ablated components. As result, we expect observe more redundant computation shared between many model components, since a random subset of them could be ablated during training. This redundancy inherently tends to make M less modular and harder to analyze for example. we should expect broad variety of components to perform relevant computation for any input, even if an accurate prediction could be computed with just few components, so becomes difficult to localize model behaviors. Since interpretability involves decomposing model computation into smaller pieces and identifying specialization among model components, models trained with dropout may be less interpretable. To summarize, while the Agero measurements are more "accurate" when M is trained with dropout, they may become less "useful" for interpretability. On the other hand, OA makes A measurements a more accurate reflection of deletion effects without training M on to distort the magnitude of these effects.
Aggregation mechanisms for ablation methods On top of a selected ablation method, some work uses Shapley values to aggregate performance gap measurements for sets of features (Strumbelj et al. Strumbelj and Kononenko, [2010] Datta et al. 2016; Lundberg and Lee| 2017: Janzing et al. 2019 Lundberg al. 2020 Covert et al. 2020). This line of work measures the importance of X, by estimating a weighted average of the performance gap A S) for all subsets $ C (X,,... rather than considering only (MM, X,). This aggregation mechanism is applied after choosing an ablation method via which to measure each A, and is just as compatible with OA as with any other ablation method.
2009
Sparse pruning and masking Finally, in the literature of sparse pruning and masking, an operation that is procedurally similar to optimal ablation is sometimes performed in prior work by adding a bias term to removed features activations after setting weights to zero
In some structured pruning work, it is typical to introduce scaled batch normalization layers Á(X)- = AA() + B3 for ς E [0, 1] to the output of each computational block A. and regularize the 7 toward zero to select weights to prune (Liu et al. 2017 Ye et al.) 2018 Zhuang et al. 2020). When ry reaches 0, the output of A is set to the constant s. which is trained to minimize the loss of the pruned model. However, the motivation of this reparameterizzatoo is not to measure component importance, and optimal ablation can be applied to more general model components (e.g. computational edges in any graph representation).
Similar to pruning, sparse masking work searches for a mask over input tokens such that for any input.
most inputs are zeroed out while model performance is retained (Li et al. 2017 De Cao et al. 2021 Chen et al. 2021 Schlichtkrull et al. (2022). In particular, De Cao al. (2021) replaces masked tokens in an input X XI,..., Xn with a learned bias 6(X). While this operation may seem similar to optimal ablation, a fundamental difference that the bias b(X) is different for each input sequence X and is trained to equalize embeddings at different token positions in a single X rather than assuming the same constant value for all values of X. Thus, for each X, b(X) contains specific information about the masked tokens in X, so unlike OA, this technique does not perform total ablation on the masked tokens. A follow-up work Schlichtkrull et al. (2022) trains a common for a dataset of inputs X. However, Schlichtkrull et al. (2022) uses an auxiliary linear model Φi (A, (X), ..., A...)) (X)) to predict whether a component (,(X) should be masked. Since Φ; explicitly depends on the values of the masked components A,(X), the model output remains dependent on information contained in (X), ). and total ablation is not achieved. Morcover, the auxiliary model ó provides the model with additional computation computational complexity of the original model as OA does. The use of an auxiliary model is a requisite feature of their method and cannot be decoupled from the masking technique: without using ((X) to predict the masked components, computing masks requires a separate optimization procedure for each input, which makes computationally intractable to optimize single over an entire distribution of inputs.
distill information about the input, rather than strictly reducing the
19

---

C Additional preliminaries
C.1 Models as computational graphs
We can write an ML model .M as a connected directed acyclic graph. The graph's 's source vertices represent the model's (typically vector-valued) input, its sink vertices represent the model's output, and intermediate vertices represent units of computation. For the sake of simplicity, assume M has a single input and single output. Each intermediate vertex A; represents a computational block that takes in the values of previous vertices evaluated on æ, and itself produces an output A, (r). that taken input to later vertices. We indicate that there exists directed edge from vertex Aj vertex A if A, (x) is taken as input to A4.
Let M be represented by computational graph (G, E) where G is the overall set of vertices and E be the set of edges. Let Aon be a tuple representing G in topologically sorted order (AAg represents the model input, while An represents the model output). For a particular vertex A,, let C (Gi. ...: G()) be the tuple of vertices (duplicates allowed) whose outputs A, takes as immediate inputs. As will see, we will sometimes require multiple edges between a pair of vertices. Rather than the standard edge notation
(A, A4) E' for simple graphs, indicate that G = Aj. i.e. (5(1) is taken as the æth input to A,,
adopt the notation (A,, A,, 2) E to
Model inference is performed by evaluating the vertices in topologically sorted order. We perform inference on an input x by setting Ao(x) = x and then iteratively evaluating A,(x)) = (G*) for i E {1...... n By the time we evaluate some vertex A,, we have already computed the values G for each of its inputs because they precede A, in the topological sort. Finally, we determine that M(x) = (x). We will alternate between the notation AG))) to explicitly write A, as a function of its immediate inputs and the notation (z) to indicate that the output of A, is function of z. We also sometimes use Ar(x) as a standalone quantity apart from evaluating M(r) and observe that this quantity is function of x computed by evaluating Az(2) in order for (1,....).
The graph notation for any ML model is not unique. For any model, there are many equivalent graphs that faithfully represent its computation. In particular, computational graph can represent model at varying levels of detail. At one extreme, intermediate vertices can designate individual additions, multiplications, and nonlinearities. Such graph would have at least as many vertices as model parameters. Fortunately, most model architectures have self-contained computational blocks, which allows them to be represented by graphs that convey a significantly higher level of abstraction. For example, in convolutional networks, intermediate vertices can represent convolutional filters and pooling layers, while in transformer models, the natural high-level computational units attention heads and multi-layer perceptron (MLP) modules.
C.2 Activation patching
Activation patching is the practice of evaluating M(x) while performing the intervention of setting some component A, to a counterfactual value a instead of (x(x) during inference. We use the notation MA, (z, a) extensively in the paper to indicate this practice, and here give more precise definition in terms of M as a computational graph:
Definition C.1 (Vertex activation patching). To compute MA. a), compute Ao(x), Ai (x) in normal fashion and set A:(x) = a. Then compute each vertex A++1(z), ..... (n) in order, computing each vertex A, as a function of its immediate inputs, i.e. A,(x) (z) .... G()))
Finally, return MA, (I. a) = (r))..
During this modified forward pass, vertex Aj that takes A, as its zth immediate input, i.e. for which Aj.,A E E, instead takes a as their zth input instead of the normal value of Azr)). Later, if some other vertex takes A, as input, it will take this modified version of (x)) as input, and so on, so the intervention on A₃ may have an effect that carries through the graph computation and eventually makes MA, (zz a) different from M(x).
In Section 3 we discuss extending this practice to edges e:
Definition C.2 (Edge activation patching). To compute Mc a), where e (A,, Ai, 2), compute Ao(x), A in normal fashion. Set (z) (G)()) ) ... (:_1(x), Gz++1 (x), G
20

---

i.e. setting its 2th input to a instead of Az(z). Then compute each vertex A,+1 (x) in order as a function of its immediate inputs. Finally. return (z a) = (z).
As mentioned in the main text, using activation patching more surgical than using activation patching on its parent vertex A,. Performing activation patching on A, would replace A, with as input to all of its child vertices, but performing activation patching on only e modifies (z)) only as an input to A, Notice that performing activation patching on (z) is equivalent to performing activation patching emanate from A, in the graph.
particular edge (Aj, Ai, 2) is
(Aj, A,, 2) for all edges that
C.3 Transformer architecture
The transformer architecture (Vaswani et al., 2017) may be familiar to most readers. However, since our experiments involve interventions during model inference with varying levels of granularity, we include summary of the transformer computation, which we later reference crystallize specifically how we edit the computation.
Transformers M take in token sequences Ti:s of length which are then prepended with a constant padding token zg. Let x (zy) The model simultaneously computes, for each token position i, a predicted probability distribution (P(æ)+1 zo:)))ī=o for the (j + 1)th token given the first tokens. We use (x)) to refer to the predicted probability distribution over the (s + 1)th token. We sometimes abuse notation to write IPMM(()) y) to indicate P(y x)). i.c. the probability placed on prediction y by the distribution M(x).
Let X be a random input sequence and be a token position sampled randomly from {1,..., 8). M trained to minimize x,, log (PM = Xs). However, we generally refer to input-label pairs (X, Y) = (Xo.s-1. Xs), so that the loss function is instead written
Ex.y ((M(X). = Ex.y | log P(M((X = Y)]
To evaluate M, each token z; is mapped to a embedding Resid, (0) (z) (x) = t(zz) + p(j) of dimension dmodel. where ((xy) 3) is token embedding of token z;j and p(j) is position embedding representing position j in the sequence. Over the course of inference, M keeps track of a "residual stream" representation Resid,") at each token position that is vector of dimension dmodel. which it updates by iterating over its Mayyrr layers, adding each layer's contribution the previous representation:
Rheada
MResid,
(0) (z) = Resid (x) + Σ
Attn (5.A) (.,))
(x)
(6)
Resid ( (x) MResid (x) MT.P (f). (x).
Attention heads Attn (4.k) transfer information between token positions. Let LN (layer-norm) be the function that takes matrix Z of size x and outputs matrix of the same size such that each row of LN(Z) is equal the corresponding row of Z normalized by its L2 norm: (LN(Z)), 2;
TT
Let R = (i- Attention heads are computed as follows:
AttnScore (i.k) (x) softmax 4
(RW(ii) + -bb
(RW(LM) + +bx
(2.R) T
Qci.
KE
Attn (t)) (RW(L))) + b
(1.6)
W(i,k) + +50 (1.b)
(8)
Vee
The (r.kks and (((.k)s weights. (i.A)) (i,A) and w((. have size dmoodet dhead, and W W((.)) (1.A) has size dhead x dmodeet b6 duodee- Biases added each row. A lower triangular matrix of 1s, represents the elementwise product and the softmax is performed row-wise. Multiplying by A ensures that Attn (f,k) (f.k) (x) only
(i.k) 0x and (((.k) have dimension dhead while (6.kk) has dimension
21

---

depends on Resido (6) and thus information can only be propagated forward, so the prediction of token j + 1 can only depend on tokens O through j.
MLP layers are computed token-wise; the same map is applied to Resid (xx at each token position j. Let = LN(MMessdd (x)), and let Ry be the jth row of R. MLPs are computed as follows:
MLP (x) ReLU (R, + byjn w out + bout.
The W and (66) are weights. in has shape dmndeel duup and W in has dimension dmip and ( has dimension dmodel-
out has shape dulp dmodel-
Finally, the output probability distribution is determined by applying final transformation to the residual stream representation after the last layer.
Out(2) == softmax (I.N (Resid (Mioyors)) (I))
W Wanembed
W Wnnembed is a learnable weight matrix of size dimodel X dyoeahh and the softmax is performed row-wise. Out (z)) is a matrix of size (s + 1) x dyocab. where dyvcaab is the number of tokens in the vocabulary and each row (z) indicates discrete probability distribution over dyocab values that predicts the (j + 1)th token given the first j tokens, M(x) is the prediction for the (s + 1)th-token continuation of x given the entire sequence z, i... M(xx) Out,(z).
C.4 Kl--divegeence loss function
The performance metric P(M) EX-D (LM(X),M(XX) selected in the paper frames perfor- mance in terms of proximity to the original model predictions, and thus the corresponding ablation loss gap (MM, A) measures the importance of component A for the model to arrive at predictions that are close to its original predictions. A common alternative, P(M) Ex ((M()) Y), frames performance in terms of proximity to the true labels, so the corresponding ablation loss gap (LM, A) measures the importance of component A for the model to perform a subtask at a comparable level to the original model. As an example, consider model M that computes approximately-optinn
solution M(X) and then adds noise in way that changes its predictions but does not improve worsen Ex,y (XM,), Y). Presenting M alone is satisfactory interpretation of the behavior of M under Pó but not under P.
major advantage of P is that is much more sample-efficient to evaluate for language tasks, especially if the label distribution has high entropy. Let (X, Y) denote random input-label pair. Recall that a language model is trained to minimize
Ex,y ((M(X), Y) Ex,y [- log P(M(X)= = c Ex |I M(X))
where is constant and p( x(X) represents the true probability distribution of Y|X. For each X, we are unable to observe p(X) - in fact, we are usually only able to obtain a single sample from Y p(X). On the other hand, .M(X) may be sufficient estimate for p(X), and provides many more bits of information about than the single sample Y ~ p. Even if our desired performance metric were P. rather than estimating Ex [Ey (X) Y' XX] from individual samples (X, Y). may often be more sample-efficiens to approximate Ey (X), Y) []] analytically for a particular X by assuming that the full model well-approximates that true distribution p, i.e. by assuming that M(X) ~ p(X) (in the sense that (p(X) || M(X) zs 0), which implies
Ex DxL(((X)) M(X) R Ex Dri (M || M(X)).
and so we still evaluate M using P as the performance metric.
9 may be an unreasonable assumption and the two criteria may yield very different interpretability results. We cannot estimate Ex,y L(M() Y) because is impossible to obtain an estimate of the ground truth entropy of next-token prediction - for long sequences, we typically never observe the same sequence more than once. However, can deduce lower bound Ex,y C(M(X), Y) 2 1 for model like GPT-2 because larger models reduce cross-entropy by more than this amount compared to GPT-2. Note that better approximation of Ex,y LXM(X), Y) is to obtain probability distribution from a larger language model M and future work may wish to explore this direction.
In practice, Equation
22

---

However, there are several other reasons to prefer using P over P with labels from a larger model. The use of KL-divergence to the original model M is consistent with previous work. In the real-world scenario of performing interpretability on the largest frontier model, we will not have access to better M Most importantly, one concern with circuit discovery for subtasks (X, Y) D is that may be possible to adversarially select M such that
[xx,))-=[(((()),) S (xX))--((M(X)))))
(10)
which can occur if . sacrifices some performance on (X, Y) 1Y- 'D for better performance on other regions of the input distribution. Selecting only the components of M that maximize performance on D may ignore important mechanisms that must be included in its predictions on 'D as a result of this tradeoff. On the other hand, evaluating circuits with P llows mitigating mechanisms to be included in the selected circuit, since we must select M in a way that imitates the behavior of M itself on the subtask D. Using this metrie, subnetwork can never achieve lower loss than M, since L(M(X), M(X)) 2 M(M()).M(X)).
D Commentary
D.1 Understanding the difference between deletion and treating x like x
Colloquially, "deletion" means the model has lost the information it would use to distinguish between inputs and x'. One might expect that if the model were able rationally handle this lack of information, it would produce an output that hedges between labels corresponding to inputs æ and a'. On the other hand, subclass 2a "spoofing" means the model was given information in component A that is compatible with x' and not z, leading the model to output something close to what it would have produced on input x'.
To illustrate the difference between deletion and insertion, consider the following example. Assume a classifier M has two possible labels and two possible inputs, x and x', and the model entirely depends on component A to determine the correct label. Let M output a probability vector, and suppose L is KL-divergence. Let M(x) = (1. 0) and M(x')= = (0, 1). If we remove the information given by A. we should expect the model to output (0.5, 0.5). giving L = log 0.5, but if we instead intervene by inserting A(x') into an inference pass on x or vice versa, then the model places probability 1 on the incorrect label, and the loss is infinite from assessing that the input is x'' when the true input is
D.2 OA as an extension of mean ablation for nonlinear functions
Let A(X) be vector-valued model component. As noted in Lundberg and Lee (0117), one motivation for mean ablation is that [[A(]]] is, under certain assumptions, a reasonable point estimate for A(X). For instance, if the relevant loss function the squared distance between our point estimate and the realized value of A(X), ). then [[A(X)] = arg min, Ex Ile A(.) Indeed, the mean is
also the best point estimate of A(X) if the relevant loss is squared distance between M a) and
M(X)= = MM A(X)) and the model M is linear in A(X):
E B[A(X)] arg min ExlinEEllMMMMX a) MA(X,
(11)
if ()M(Xa)= =
b(X) for a random matrix M(X) TL A(X) and random bias b(X).
Thus, for model components A(X) for which the downstream computation is roughly linear, (x))
could potentially be a reasonable point estimate, hence justifying mean ablation. This presumption of linearity also shows up in other interpretability work, including Hernandez et (2024). which uses a linear map to approximate the decoding of subject-attribute relations, and Belrose et al. (2023b), which considers erasing concepts Z from model's l's latent space in a "minimal" sense by transforming activations A(X) with a map dz that makes g2 (A(()) uncorrelated with Z and minimizes expected squared distance to the original activations, (A()))
However, in most settings, M(X, a) highly nonlinear in a, and the mean E[A([)] could be an arbitrarily poor point estimate for A(X). Optimal ablation generalizes the idea of selecting the "best point estimate" for A(X) as measured by replacing A(X) with a and evaluating model loss. In particular, optimal ablation constants a" generalize the property given in Equation (T1) to arbitrary
23

---

models M and loss functions L:
a" arg min
(12)
D.3 Generalizing OA to constrained-formn estimates of A(X)
Measuring pppt (M, on subtask (X, ~ D is, a sense, testing procedure for the hypothesis that "AA does not provide relevant information for model performance on subtask DD"" Verifying that
Aopt se 0 validates this hypothesis, since a point estimate of A(X) performs as well as the realized value of A(X) for the purpose of model inference.
Optimal ablation can be generalized test interpretability hypotheses beyond assertions that a computed quantity A(X) is unimportant. In particular, we can test hypotheses about the specific properties of A(X) that are important.
Suppose A(X) is vector-valued, and consider the hypothesis "the only relevant information in A(X) stored in subspace W." We can test this hypothesis by replacing A(X) with Awx)(X)= +a", where a" is an optimal constant that lies in W and Pw is the projection matrix to subspace W. While this example is simple and illustrates some of the flexibility of OA, it does not add to the space of what OA can express, in the sense that could have simply considered Px) and A(X) as separate vertices in the graph and used OA (or any other ablation method) on only the latter.
However, a real gain of expression from OA materializes from being able to generalize the idea of null point estimates to estimates with constrained form. For example, consider the subspace hypothesis "every A(X) can be adequately represented in subspace W." ." We can test this hypothesis by training an optimal activation a" (X) for each X that lies in subspace W. Though activation training is expensive, the eιror of this function by performing activation training on a few samples of X.
train function a" (XX) that maps X to values in W, and then estimate
Similarly, we can generalize a" to include multiple point estimates to test the claim that A(X) is the outcome of an internal classification problem, i.e. the relevant information provided by AAX) the classification of X among a few input classes. We can train optimal point estimates (a). ..., a%)) such
that
(ai, a arg min Ex min (01))..0k)) jE je(,....))
aLM
calling the outer minimized quantity optt If Ax- O, then every A(X) can be represented by one of a small number of prototype quantities.
E Single-component loss on IOI
E.1 Transformer graph representation
We use a graph representation in which each vertex corresponds to an attention head (Attn( (x)). an MLP block (MLP()(z) (z)), the model input (Resid (x)), or the model output (Ou) We also allow vertices representing the (t (x) and (Resid (zr) computations. Appendix C.3 defines the computation of each vertex.
However, we slightly modify the definition of attention head vertices Attn (i.k) (i.k) to save memory and so that ablation constants a" for OA lie in the column space of attention head outputs. Recall from Equation (8) that attention heads produce output in dhesa- -dieensiiiaii vector space, which is then mapped linearly - dimeensioal space by weight matrix (.(,k))
Attn (ttn(( (x) = AttnScore (6.A) (xM)
(x)
(RW + ((.5)
W + +B6.
Thus, while Atin (1.b)
(f.A)(x) (1:) is dimenssoal,, its distribution lies within a dhead subspace of the residual stream. If we used vertices Attn (i,k), then our ihhodal-dimensionaa a" for attention head (i, k) could sometimes contribute to subspaces that the attention head k) can never write to. Instead, for space:
an attention head vertex, we represent its output computation in
ZAttn (z) (t.) (x) (RW( + b
(1,k)
24

---

and consider replacing ZAttn (1.0) (1A) (z) rather than replacing Attn (ttn( to ablate an attention head. This slight modification reduces our parameter count by a factor of model /head Ahoad when applying OA but does not affect the results for the other ablation types.
We measure the single-component ablation loss gap A for the (ik)) and MLP vertices, (Nheads + 1) Taayers 156 vertices in total for GPT-2.
E.2 Ablation details
We consider zero ablation, mean ablation, optimal ablation, counterfactual mean ablation, resample ablation, and counterfactual ablation.
For a token position j. let [A(X)]; denote the representation of A(X) at token position j.
Zero ablation: To zero ablate A, we replace [A(X)], with O at each sequence position j 2 1. We do not replace [o because it is a constant that does not depend on X (any result at token position O must only be function of ) which represents padding token that is the same for every sequence). Transformers may read from this beginning-of-strinn (BOS) token position in attention heads if no token in the sequence indicates a particularly strong signal, and since this token position does not distinguish between any inputs and is more appropriately viewed as a structural part of the architecture, we choose not to modify it.
For each vertex A, compute ( [AX() over 20,000 samples, conditional on token position. We let μ1 = (oxx))>=[[A((]]]]) To mean ablate A, replace [A(X)I, with My at each sequence position j.
Mean ablation:
In the Greater- Than dataset, all prompts X are the same length, but in the 1OI dataset, some prompts X are longer than others, reducing our sample size for later sequence positions. In particular, if X* is the longest prompt in the dataset with É tokens, then μ4 X2, so the mean value actually carries identifying information about the prompt. Since we want the mean value f4, to uninformative about the original prompt X, we instead consider m to the minimum length of any prompt, compute a modified mean (4m = Y),D,)> s=- [LAAai [[)()]ss S 2 m] that considers all values of A(X) token positions after token position m, and replace [A(X)]; with pmm j 2 m.
Similar to mean ablation, we optimally ablate A by replacing [A(X)]s with a constant ū, for each sequence position j < and replacing [A(X)], with constant a.m. for each j 2 m, where m is the minimum length of any prompt. We initialize (0n, ("n,@,,.... ai...,,,aaa Fm) = (po, (1,.... Pm) as defined for mean ablation and then optimize (ā1...., am) for each ablated component A minimize A (M,A) Note that similarly to zero ablation, we fix âo μo = [A(X)), lo and do not optimize its value as an ablation constant because [A(X)]o does not depend on X and thus naturally conveys no information about the input.
Optimal ablation:
Counterfactual mean ablation: Our implementation is the same as for mean ablation except that we compute means over (X, Y) ~ D' for the counterfactual distribution D', and m is taken as the minimum prompt length in the counterfactual distribution.
Resample ablation: To perform modified inference on an input X, we first sample an independent copy X' 11 X. Let X and X' have lengths and s' respectively. If s S s', for an ablated component A, we replace [A(X)], with [A(X')], at each token position j E {1, ..., s3 (in other words, we only resample from the first tokens of X'). If s', then we left-pad X' with an additional tokens to form modified token sequence X' that is the same length as X. We then replace ablated component values A(X) with A(X') with respect each sequence position. Before arriving upon this implementation, we tried other choices, like resampling from the last s tokens of X' in the case that < s', or fight-paddin X' in the case that > s'.
Counterfactual ablation: We choose a function (details discussed in the main text and further analyzed in Appendix (F3)) that maps inputs X to neutral counterfactual inputs X'. Typically, X and X' are the same length and have many tokens in common. For ablated components, we replace [A(X)]; with [X)]( at each token position.
25

---

Correlation plots of ablation loss measurements 10]
00.678
T.0.2
0..83
r-0.63
DD.622 6.6
Count
Mean
Resample
CF-Mean
Optimal
Zera
Zerb
Zero
Zero
Zero
Zero
r00.637
976
rw0,
r-D.931
r-D.831
Count
Resample
CF- LF-Mean
Optimal
Zero
Mean
Meen
Meen
Mean
Mean
r 0.69
r09.976
r-0. 0-9772
r-0.925
r- 0.826
Resample
Ressann
Count
CF-Mean
Optimal
Zera
Mean
Resample
Resample
Resample
Resample
700644
- 997
r-O 977
r- -0.9226
Fm0.847
CF-Mean
CF-Mean
CF-Mean
Count
Ontmaa
1
Zera
Mcan
Resample
CF Mean
ČF CF-Mean
Čr Mean
== 0.607
r 009928
r=O -0.924
0.9.3
r=0.908
Optimal
Oottigg
Gomnng
Dolima
Count
Zero
Mean
Resample
CF-Mean
Optimal
Optimal
-0.9
00.25
0=0.228
r= 00833
r=0.907
Count
Zero
Mean
Resample
CF-Mean
Optimal
Figure 5: Correlation of single-compont ablation loss measurements on IOI. Lower triangle shows rank correlation and upper triangle shows log-log correlation across metrics.
E.3 Full results
Figure the pairwise correlations of single-component ablation loss evaluated on the IOI dataset with variety of ablation methods. Table 2 an extended version of Table in the main paper that provides a summary of these results.
Table 2: Comparison of ablation loss gap A on IOI, extended
Zero
Mean
Resample
CF-Mean
Optimal
CF
Log-log correlation with CF
0.626 0.590
0.831
0.826
0.847 0.833
0.908
Rank correlation with CF
0.825
0.828
0.907
Mean A
0.0584
0.0405
0.0559
0.0412
0.0035
0.0296
Median ratio of App to A
11.1%
33.0%
17.7%
31.7%
100%
88.9%
26

---

F
Circuit discovery
F.1 Transformer graph representation
We use a residual rewrite graph representation favored by Wang et al. (2022). Goldowsky-Di et al. (2023), and Commy et al. (2023). Similarly to Appendix E. we define vertices that correspond to an attention head, MLP block, the model input (Resid (z)), the model output (Out(z)), but eliminate the Resid (6) (a:) and MResid (x) (z) vertices. We have Mayyers (nheads 1) + 2 vertices in
total (156 for GPT-2). Notice from Appendix C.3 that
Pheade
( (x) Resid (0) (z)
Σ
MLP (z)
E
Attn (1.b) (1.))()) (z)
so rather than assuming that attention heads, MLP blocks, and the model output take Resid (a) input, assume that they take the output of each previous block separate input to the computation. In particular, we can write
MLP(0(z) = MLP () (z), MLP())x) (x). (M)(-.,))
Attn(1.)) (r), Attn Attn (( (x))
(13)
in which the MLP vertex has i(Uheads 1) + incoming edges from previous vertices. Similarly, we can write
Out(#) = Out Resid (x), MLP(() MLP Playere) (z),
(ttn (x), (x), Attn Manyern (x)
(14)
so the Out vertex has flayeers (Nheads + 1) + incoming edges, one from each previous vertex in the graph. Finally, notice that attention heads (t. take (t as input three different locations, once in each of the query, key, and value subcircuits, so we can write attention heads as taking three copies of each previous vertex's output, which can be ablated individually.
Attn (1.k) (or) = (ktn((k)) (0) (x), MLP(U))(()) MI.((-1)..
A (x) (x), Attn (1- (x),
(0) (x), MLP(1).K (.)...,,M.P((-1)) (x),
Attn (1.1,K (z), Attn (i- (Ir),
Resid (U).V (U)V
(x), MLP(L).V ..),.,,MPP(6 (x),
Attn (x) Attn (1 (x))
(15)
This notation indicates that attention heads admit multiple incoming edges for each previous vertex, which is somewhat non-standard. Alternatively, rather than allowing multiple edges between pairs of vertices, Conmy et al. 2023) creates a separate vertex for each of the query, key, and value subcircuits and considers each attention head output to take the outputs of these three circuits input. However, edges between the subcircuits and attention head outputs are essentially placeholder edges that cannot be independently removed, since removing them is informationally equivalent to ablating the entire attention head. Thus, our graph representation is more natural and provides a more realistic edge count when considering removing model components.
Furthermore, continue with the adjustment from Appendix E. of using ZAttn (i.k) computa- tional vertices rather than Atun
conserve memory and reduce the parameter count of OA. We consider the linear map ((i,k) (Z) = Zw(( + +b
(4.A) (so that Attn (x) = ((.,A) (ZAttn (6.k) (x)))
and express all downstream vertices as taking ZAttn (t.tn((.k)(( (a:) as input rather than Attn (1.0 (2:) and performing their computation by pre-composing with ((i,A) For example, for MLP vertex MLP(1), if ()) represents how (x) is computed using Attn (x)))()))) values inputs, then its
27

---

computation taking ZAttn (xx) inputs is equal to MLP (x) = mm() (Resid (x), MLP (x), ML.(
(1.1) (ZAttn (.")) (x)), of (bmmaodd)) (Atta (Atta (ambaaade) (x))
We replace all Attn (8.8) vertices in the graph structure with the corresponding ZAttn (t.k) vertex.
In total, we have (Nheads Maayers) ) ZAttn (6.k) vertices, Mayers MLP(1) vertices, an input vertex (Resid(") and an output vertex (Out). Letting V represent the set of vertices that includes the input and MLP(1) vertices. There are 3 Raayers (Maayers 1) -hedd edges between two attention heads, 2. (Myyers + 11) Taayers Theads edges between an attention head and vertex in V, 2 "aayers (Mayers + edges between two vertices in V, and (Mayyrss) (Nheads + 1) edges from any vertex to the output.
For GPT-2, there are 28,512 edges between two attention heads, 3,744 edges between an attention head and a vertex 78 edges between two vertices V, and 157 edges from any vertex to the output for total of 32,491 edges.
F.2 Ablation details
For mean, resample, and counterfactual ablation, our implementation is the same as Appendix F.2
For optimal ablation, we adjust the implementation to remove dependence on token positions and further reduce the parameter count. For each ablated component A, rather than training different oj replace [A(X)]; for each token position j. train single optimal constant a" that is the same shape as any particular [A(X)]]: We initialize a" to [[[A(X)]] | > 9). the subtask mean excluding early token positions, since early positional embeddings may have idiosyncratic effects. To ablate A during inference on X, for all token positions j > O, we replace [A(X)), with a*. As in Appendix E.2 we do not replace [A(x)]o because this value is a constant that does depend on X.
We take this conservative approach to demonstrate that OA can be implemented a manner, yet still outperforming position-specine implementations of other ablation methods by large margin. Since many other ablation methods, like resample and counterfactual ablation, are inherently position-speciffe compatibility with a vosition-agnostic implementation is a key advantage of OA over these ablation methods, and we discuss this advantage further in Appendix F.3
As noted in Section 3 we train single constant a for each vertex Aj. An alternative implementation is training a separate constant for each ablated edge. In thcory, this approach is consistent with the spirit of OA: though ablated edges transmit different values downstream components that would normally receive the same value, none of the transmitted constants transmit any information about the input to downstream components. However, this approach would greatly increase the number of learnable parameters, and arguably may actually increase the computational capacity of the model. Additionally, training single constant for each vertex has the appealing property that ablating all of the edges from a vertex is equivalent to ablating that vertex.
F33
Normative comparison of ablation types
Thus far, circuit discovery on language models has focused synthetic subtasks for which mapping π from studied inputs æ to counterfactual inputs π(x) easily constructed. Recall that crucial criterion for selecting и is to preserve many tokens as possible between and ((x). For freater-Than, an example counterfactual pair is
Token:
S
YY1 YY2
YY1*
x: The [conflict] began in [18] [89] and ended in [18]
xx): The [conflict] began in [18] [01] and ended in [18]
where the brackets are added emphasize the two-token representation of the year. Similarly, for 10I, an example counterfactual pair is
Token:
S2
X: Friends [Alice] and [Bob] found a bone at the store.
[Alice] gave the bone to (x) Friends [Charlie] and David found bone the store. [Charlie] gave the bone to
28

---

Since r and (2) share the same token at all but a few token positions (only the SI, S2, and 10 token positions for IOI and only the YY2 token position for Greater- Greater-Than), Than), we can isolate the effect of changing the specific token that conveys important information. CF thus allows us to study subtasks involving input-label pairs where subtask-relevan information given by only one or several tokens.
However, replacing A(x) with A(x') typically incurs much higher loss if z and x' differ at many different token positions, even if most tokens are unimportant in relation the behavior we wish to study. The model representation at a token position j is likely to contain information specific to the tokens position j and at surrounding positions in the input X, so replacing [A(z)], with [A(x²)) )] is likely to inject inconsistent information if X, and X (or pairs of tokens at corresponding proximate token positions) are different tokens, causing A to be high
result of spoofing (per Section 2.3).
An illustration is the discrepancy in resample ablation loss between the Greater- and IOI subtasks. The Greater- Than dataset only contains a single prompt template, so any sampled X and X' only differ tokens that encode the subject and year. Here is an example of a sampled (X,X')
X') pair:
Token:
S
YY1 YY2
YY1#
X: The [conflict] began in [18] [89] and ended in [18]
x: The [deal] began in [15] [47] and ended in [15]
On the other hand, the IOI dataset consists of multiple prompt templates that differ in sentence structure, so and X' may differ at nearly all token positions, not just the S1, IO, and S2 positions as shown above. For example:
X: Friends Alice and Bob found x²:
bone at the store. Alice gave the bone to Then, Charlie and David had long argumet, and Charlie said to
where <> represents padding token added make the sequences the same length. As a result, resample ablation loss is relatively low for Greater- Than (see Figure I0) but relatively high for IOI (see Figure (. indicating that token parallelism is important requirement for CF to work well.
While the synthetic IOI and Greater- Than datasets are specifically engineered so that we can modify prompt x at only few token positions to obtain neutral prompt ((x), more general language behaviors may not be suited for this type of counterfactual analysis. Here are a few examples of language subtasks for which it not be possible to pair up and x(x) with parallel tokens:
A case study of the effect of modifiers, e.g, adjectives and adverbs, compared to a sentence with no modifier. Consider the following (degenerate) counterfactual pair, inspired by Marks and Tegmark (2023):
x: Paris is a city in the country of x²: Paris is not city in the country of
Since the presence of modifier creates an extra token, replacing A(x) with A(z') patching between sequences with and without the modifier would result in the embedding at most token positions in X being replaced with embeddings at the token position of a token reflecting the previous input token in X ("city" with "a," "in" with "city," and so on).
A case study comparing sentence order in situations where order matters, like giving directions. Patching in activations from counterfactual prompt in which the order of two sentences is permuted involves introducing a new token at many token positions.
x: Make left turn, then walk forward one block. Your position is now
x² Walk foward one block, then make a left turn. Your position is now
A case study relating how language models handle mis-tokenization, like processing prompts in which a word is misspelled or the model is required to spell out a word.
x: The correct spelling of the word umpire is
x²: The correct [sp] [e] Ding] of [te] Dil word [up] [mire] is
Additionally, the field of interpretability moves forward, we believe that it must progress toward "total" interpretation of models' internal mechanisms. This level of interpretation requires reasoning
about subtasks that much more general than those that have been studied and will require
29

---

performing intervention- baased analysis across a broad distribution of inputs. For example, we may want to make the claim that certain components of model . are unimportant for performing mathematical calculations; or that some components are not involved in ensuring grammatical correctness; or do not assist in making theory-of-mind assessments; etc. Additionally, we likely wish to assess component functions "in the wild" with filtered sampling from the model's training distribution as opposed to engineering synthetic datasets. These circumstances mean that the data will be much less suited for token parallelism between counterfactual prompts, so the adoption of sequence- sequence-position agnostic ablation method is likely critical. This quality of OA makes it a much better candidate than CF as a suitable ablation method for scaling interpretability.
F.4 Sparsity metric
As stated in Equation (3) we wish to select a circuit that achieves low loss Ex Lx (M (opt) (X)) and which is a sparse subset of the model. Let A represent the set of edges connected to vertex A in graph G, i.e. E (AAj,A,,) E E Aj V A; A}.
The selected circuit F. should ideally satisfy two types of sparsity:
1. Edge sparsity: [E] << EE The circuit should contain a small number of edges compared to the total number of edges in the model.
2. Vertex sparsity: HA| A n El 0}1 << |G|. The circuit should pass through small number of vertices compared to the total number of vertices in the model.
There is a lack of guidance in prior work about whether smaller structures with more densely packed connections are more interpretable than larger structures with more thinly distributed connections. Indeed, one could argue that the larger structure is in fact easier to understand, since we do not need to dissect as many relationships to consider the function of any particular vertex within the circuit.
While circuit discovery aims to localize model behaviors on specific subtasks, we contend that a central challenge in interpretability going forward could be stacking together many circuit analyses to form a sum-of-the-pee parts analysis of the model's overall structure. Considering circuit discovery as tool for decomposing model computation into interpretable subtasks, holding the total number of edges equal, we may prefer each circuit to have smaller number of vertices to reduce the complexity of interactions between circuits rather than within circuits.
As such, we set R(FF) ()) to select for circuits with high levels of both edge and vertex sparsity:
J E
R(E) A T
Σ EA Lanh
2
(16)
2
EAAl
AEG
where X, are constants. Similarly, the continuous relaxation R(0x.) is for HCGS and UGS is derived by replacing [B] with Σ1= Og and replacing |EAA É with E Σcyeea Oz.
Note that Vo. R() = X + >A E.EEA sech (2
The first term, A, is generally used to control the tradeoff' between edge sparsity and circuit loss; a general interpretation is that we should include an edge E É if its marginal contribution, (M,, (EU (E+()))) h) - A(M, E (fex) U F)), is greater than X, expressing the same tradeoff as the discrete threshold À in ACDC. However, since ACDC is less fine-grained optimization algorithm than UGS, the required to achieve the same circuit size [ tends be larger for ACDC.
The second term expresses vertex sparsity, and its effect is to increase the regularization effect for edges that are attached to vertices that have few other edges included in the circuit. Its effect is small when A R [EAl, since 2 Ri O, so we do not apply additional regularization to edges attached to vertices that have high overall likelihood to be included the selected circuit. However, its effect is significant when e./lEA Rs 0 to prune the remaining edges from vertex whose edge probabilities represented by 0 are low on average. We use 7 express the maximum influence of vertex regularization compared to the effect of edge regularization (since max, sech (x:) = 1), and generally select = 0.5, so the second term adds at most 50% more regularization.
30

---

F,5 Uniform Gradient Sampling: motivation
In circuit discovery, the number of possible circuits F. E. is exponential in [E] and the circuit losses (M,, E É) for subsets É are not required be related. A is not even necessarily monotonic in É for any ablation method considered, i.c. E E' does not imply that A E) 2 A(M, E*).
In reality, can hope that the optimal ground-truth circuit E is clear-cut and A is relatively well-behaved. If so, we could try to relax the discrete optimization problem and find a solution with gradient descent. As mentioned in Section one continuous relaxation considers partial ablation for each edge ex = (Aj, z), where we replace Az(x) as the zth input to Ar(x) with ag (z) + (1 ax)a, Ja and use L1 or L2 regularization on the ag. However, this approach is likely to get stuck in local minima in which edge coefficients converge to the optimal magnitude instead of edges being completely ablated or retained. Instead, we consider a vector 0 of independent sampling probabilities for the inclusion of each edge, and optimize these probabilities so that each converges to 0 or 1. Our loss function is f(0) == E. _x (x,( ( (X), + R(E)]. Denote (x(X, E)- L(M ((M (X)) + R(E), that f(0) (x,(E~0) (Xx E)J. The gradient with respect to the sampling probability Og for edge eg is marginal ablation loss gap:
0f(()
Ex (x,(8~)) (U -x(ee/))))(x__.))---(-)) Cx(X, E. (ex)) =: (
(17)
00g
The problem is that E is large and it is not tractable to estimate this quantity individually for all k. Our goal is to find a good sample estimator for this quantity simultaneously for all k.
One way to perform this simultaneous estimation is importance sampling, where we write
PE PE =E), P PF' E = E")
f(0) = E (x,(E~
LR(XX,
so when p 0,
1-0, Cx(X, However, this method leads to poor estimates. Most of the variance in gradient updates to 0g comes from sampling different subsets of edges among the [F] - edges other than ek, not the effect of fixing ex E E or ex E É for a particular edge.
8f
Lx (X, (( (((EER) (x,(E=0)
30,
O2
Instead, UGS is an approximation of the marginal ablation loss gap for each edge obtained by taking gradients with respect to sampled partial ablation coefficients ā. We consider the extension of M to convex relaxations M where ag represents the partial ablation coefficient for edge e₁ as alluded to above. Similarly, we consider Lx (X, ā) in place of CR (X, E). Let ā ((E, S) where S C (1....,11 EE} an (U É, 8) 1(k E S)U& + Ikk & () E).
such that
For any edge ek, the marginal ablation loss gap inside the expectation in Equation 17 is equal the expected gradient with respect to ag Unif(0, 1) when other edges are sampled according to E:
0f(²) a02
Ex (x,( id)
CR (i,(x,a(Ü,E,10 aU,
(18)
In other words, can estimate the effect of totally ablating e for given (X, B) by sampling partial ablation coefficient ag Unif(O, 1) and taking the loss gradient with respect to ag. However, we run into the same problem of needing to estimate the effect individually for each edge k.
UGS assumes that we can estimate this loss gradient for many edges simultaneously without much bias. For any particular edge 6g, the interference effects caused by sampling other edges e₁ Unif(O, 1) for all k S instead of setting them according E could be small, if S is small enough. UGS assumes that if 5 is sampled from a distribution Ds
0f(0)
~ Ex,(B=0)(0~~uiit (s).(S)Ds)
Lx Lx(X, (X, a(, E,S)) kES OUk
(19)
00p
31

---

F.6 Uniform Gradient Sampling: construction
We use 9g to represent the sampling probabilities for each edge, and perform gradient descent on the parameters by constructing a loss function whose gradient is sample estimator of Equation (19). As noted in the main text, rather than using 0& E (0. 1) as our parameters, we use 0₂ E (-00, 00) as our parameters and compute 02 o (0x). We initialize 02 1 for all edges eg. We avoid random initialization because it achieves worse results by causing the resulting circuits to be suboptimally constrained to be close to our random prior.
We sample S C {I,..,,k) }} by independently sampling each 1(k E S) Bern(w (wx)))) where window function w determines how often we sample dg ~ Unif((O, 1). Additionally, we require that
1
P(ex E EE P(e $ E k E) =
(20)
so that sampling on Uni 1) takes away probability mass equally from E E and eg & É. We perform this adjustment because for the purpose of estimating gradients W with respect to edges other than ég, Equation (19) implicitly assumes that
S] R
W
ark = 0] + (1 - p) E ag = 1], p P(e E EIKE S) (21)
and without any priors about the functional form of W, p = _ is reasonable choice.
We construct a loss function whose gradient is given by Equation 4.. a sample estimator of Equation
(9) with this construction of S. We construct the batch X ,.., (
by choosing b/n, unique
samples of input X and repeating each input n, times in the batch. We generally use n, 12 and
b = 60. We choose w(2)) = c 0 0z). Note that c I in order for (w((0x) S min (0z. 1 0%)
to hold, as is required by Equation 20). and we choose e 1. We discuss this choice in F8
The full algorithm pseudocode is given after describing a few additional details in Appendix H.
F.7 Additional circuit discovery details
For HCGS and UGS, we use learning rates between 0.01 and 0.15 for the sampling parameters.
Pruning dangling edges For HCGS and UGS, after sampling a(Ü F, for each input, we remove "dangling" edges ex. A dangling vertex is a vertex A for which there does not exist a path from the model input A along edges e; for which aj >> O, or for which there does not exist such a path from A to the model output. A dangling edge is an edge that is connected to dangling vertex. For a dangling edge ek. we replace og = 0 (i.e. the equivalent of removing e₁ from É and S).
Discretization For HCGS and UGS, after training the ON. we select final circuit by selecting all edges for which O2 for a threshold t. Generally, for UGS, all but a handful of O2 converge to highly negative or positive values, the choice of r does not have much impact, and we choose 0.5. However, for HCGS, many edges have 0g parameters around zero even after training for 10,000 batches. We again select T = 0.5 since we observe that including edges with 0₂ E (-5, does not generally affect performance.
Optimizing constants for OA For HCGS and UGS, we train ablation constants concurrently with training the sampling parameters Og. We use a learning rate of 0.002 for a, lower than the learning rate used for the sampling parameters.
Note that we only provide gradient updates to âj for a vertex A, along edges eg for which og = 0. Updating a, when on * 0 can lead ā to update toward a value that is optimal when taking linear combination of ā and AX(X. rather than a value that is optimal as a constant. See Figure
Even though we obtain approximate constants ã through the training process for HCGS and UGS, in order to level the playing field when comparing to ACDC and EAP, we do not use the constants found during training during circuit evaluation. Instead, for each circuit discovery algorithm, we evaluate circuits with OA by initializing constants to subtask means and then training for the same number of batches (10,000) with the same settings with learning rate of 0.002.
Algorithm T shows the full algorithm of UGS, including our exact loss function, for optimization with OA, For optimization with other ablation methods, we set ablated values à according to the
32

---

learned oj
true 3
incorrect update
Va ((M
correct update
āj
Ukā + (1 - (X)
Ax(X)
Figure 6: Gradient updates on can be biased when og + O.
ablation method and do not perform gradient updates on à. Note that we use the notation Ma @) with OA indicate running the circuit evaluation with ablation coefficients ā and replacing ablated components with ã in line with our notation in Appendix C.2
F88 Choosing a window size for UGS
We motivate the choice of our window function w=(Ok) 0x 0x)-
Let J 0f(²) 80, K Unif( .... EE}). We may want to minimize the squared distance between our sample estimates and the true gradient values,
0f()) 00% and let S& be our sample estimate from approximation 19 Let
80 a0,
: B<fk fx)² Ex Var(fR] K) + Ex (BIfk К| f2)
(22)
Let sampling distribution S ~ D be defined by independent Bernoulli random variables [(ex S) ~ Bern(wx)) Assume that we collect b samples of (Ü, E, S) in batch and use the samples for which & E S; to estimate the fu. Let Th == Ei= 11k E St), and let fo = T Eim fk 1(k E S.). Let 1 represent the first sample for which & S,
Assuming that Og Unif(0, 1) w.p. Wk. 0% = 0 w.p. 02 we, and O*k = 1 w.p.
1 8x wu as given by Equation (20). we have the loss derivative |E| Pwa
aarr
Own
Eesr (Varr(fc) + 2( - fi sE If we assume that the second term is roughly equal to Swc
a constant e for edges, then
E
de
Var(f?)
000
2
1
Tk >
(23)
Owk
a0,
Owk
Ts
since T. for an edge implies that simply does not get a gradient update. Note that a6
86
(1 0s), and Duc
-bw 2 for constant 0. Solving for Ows 0. is minimized
when we x Og (1 0z) / Var(sl) which motivates the definition of the window function w(Ox)
(+(1 0x) in our main experiments. We try including the additional factor of / (aar(f))) ) but our results do not improve.
F.9
Comparison of UGS and HCGS
HCGS and UGS both involve sampling gradients from a distribution over ă' E (0, 1) and taking gradient steps on parameters 0g that represent our confidence in 6 E E. using an average of gradients with respect to the edge coefficients ag. The original explanation provided by Louizos ct al. (2018) for the convergence of HCGS to satisfactory subset of weights that minimizes a loss function similar to Equation involves Lo regularization. To the contrary, we believe that more
33

---

Algorithm Uniform gradient sampling
Input:
set of edges E, initial parameter array 0, initial constant array @
Output: of edges É C E that represents the circuit
Require: metric L, learning rates 8g, 8a, final threshold т, batch size b, sample count per input 7t..
window function
loop X
UnifCount [0 for k E [Iength(O)]]
for j E b/nz do
alj] []
x - sample_input()
for E ng do
for k E [en)gth(0]] do
U < Unif(0, 1)
W < w(ffk))..detcch gradient()
+ W 0[k] (1- betteddeeaah_grdieinn
a[j]b{[]] + W + 0.5) .clamp(0,)
UnifCount(Ak] < UnifCount(Ak] [(a[)[] (0)
afj[[ +- + lelppuuuddaggge [aane_ddnnninn_eges[[]]]]]]]
++ []
for j E [b/n=] do for E n. do
for k E [length 0) do
g + b/UnnfCouutt(/
[ + a + (1 - g) [ detpldeeeaaaaiiiiieee (aip = 0) a (abi > 0) [lletach_gradient]
where and are applied componentwise; this step is only used with OA. LI.
(X (X(). M)((judgee))
Σ L)/6
8- 8 0g Vof
6g Vaf: this step only used with OA. Note that in practice, we use Adam to determine step sizes.
à ā
F. <- 0
for E [length(p)] do if O[K] > then (a)))EE(k) return
compelling explanation for the performance of HCGS is that its behavior of sampling gradients on a region of partial ablations, E (0, 1) serves as a vague approximation Equation (T8).
Sampling E Unif(0, 1) to obtain gradient information, rather than a scaled conditional Concrete distribution, makes the simultaneous muddent-ssamling estimator unbiased the single-dimensional case, and thus is the choice that makes Equation (19) most resemble Equation (18)
F.10
Additional IOI circuit discovery results
This section displays additional circuit discovery results the IOI subtask. In Figure 7 we show the tradeoff between A (y-axis) and |E| (X-axis) for optimal ablation compare different circuit discovery methods. In Figure([(eett) we show this tradeoff for mean ablation, and in Figure g (right), show this tradeoff for resample ablation.
34

---

1250 1500 IOI circuits, optimal ablation
UGS (ours)
HCGS
0.5
EAP
ACDC
Ablation loss gap A
manual
0.2
0.05
250
500
750
1000
1750
Circuit edge count [É]
Figure 7: Circuit discovery Pareto frontier for the IOI subtask with optimal ablation.
|OI circuits, mean ablation
IOI circuits, resample ablation
UGS (ours)
UGS (ours)
HCGS
HCGS
EAP
2
ACDC manual
ACDC
Ablation loss gap A
manual
1-
Ablation
0.5
0.5
0.2
0.1
250
500
750
1000
1250
1500
1750
0
250
500
750
1000
1250
1500
1750
Circuit edge count [É]
Circuit edge count [E]
Figure 8: Circuit discovery Pareto frontier for 1OI with mean ablation (left) and resample ablation
(right).
F.11 Circuit discovery results for Greater-Than
This section displays circuit discovery results on the Greater-Thie subtask. In Figure 9 (left), we show the tradeoff between A (y-axis) and [ (x-axis) for optimal ablation. In Figure 9 (right), we show this tradeoff for counterfactual ablation. In Figure I0 (left), we show this tradeoff for mean ablation. In Figure I0 (right), we show this tradeoff for resample ablation. Finally, in Figure [I] we show the A achieved by circuits optimized using UGS on A with different ablation methods, analogous to Figure I (right) in the main text for the 101 subtask.
35

---

Greater-Than circuits, optimal ablation
Greater-Than circuits, counterfactual ablation
UGS (ours)
UGS (ours)
0.05
HCGS
HCGS
EAP
0.2
EAP
ACDC
ACDC
Ablation loss gap A
Ablation loss gap
manual
0.1
manual
0.02
0.05
0.01
0.02
0.01
0
100
200
300
400
500
600
700
800
100
200
300
400
500
600
700
800
Circuit edge count [E]
Circuit edge count [É]
Figure 9: Circuit discovery Pareto frontier for the Greater-Thar subtask with optimal ablation (left) and counterfactual ablation (right).
Greater-Than circuits, mean ablation
Greater-Than circuits, resample ablation
UGS (ours)
UGS (ours)
HCGS
HCGS
0.5
EAP
EAP
ACDC
0.5
ACDC
Ablation loss gap A
Ablation loss gap A
manual
manual
0.2
X
0.2
0.1
0.1
0.05
0.02
100
200
300
400
500
600
700
800
0
100
200
300
400
500
600
700
800
Circuit edge count (ε]
Circuit edge count [ē]
Figure 10: Circuit discovery Pareto frontier for Greater-Than with mean ablation (left) and resample ablation (right).
F.12 Random circuits
One question is whether it may be possible to extract circuits with OA that do not necessarily explain model behavior on the training distribution by setting vertices to out-of-distribun maximally elicit a certain behavior. If the ablation constants ā overparameterize the data, then performing OA could behave similarly to fine-tuning the model to perform the desired task.
values which
Intuitively, however, OA strictly decreases the amount of computation available the model, since we only add constants to model components and do not allow additional transformations of internal representation that are not already present in the downstream computation. To verify our stance, we compare the loss recovered by circuits discovered by UGS to random circuits to verify that OA indeed distinguishes mubtask-performin elicit subtask behavior from unrelated model components.
mechanisms and does not provide enough degrees of freedom to
In Table we compare the A(M, É) achieved by random circuits to those achieved by circuits optimized with UGS for various ablation types. We construct by sampling each ( E E) independently with some probability p. and prune dangling edges as detailed in Appendix E7 We
36

---

Greater-Than circuits, ablation comparison
Optimal
0.2
Mean
Resample
CF
Ablation loss gap
0.1
0.05
0.02
0.01
100
200
300
400
500
600
700
800
Circuit edge count (É]
Figure 11: Comparison different ablation methods for circuit discovery for Greater-Thur
Table 3: Optimized circuits compared to random circuits for various ablation types
Mean
Resample
Optimal
Counterfactual
10I
Random circuit loss
4.529
6.527
2.723
4.264
UGS circuit loss
0.264
1.779
0.176
0.191
Std
0.200
0.085
0,024
0,049
Z-score
-21.28
-55.67
-100.57
-82.44
Greater-Than
Random circuit loss
1.010
2.109
0.900
1,785
UGS circuit loss
0.033
0.056
0.029
0,021
Std
0.020
0.039
0.011
0.027
Z-score
-49.29
-52.89
-80.81
-64.76
accept E [ is within an acceptable range, and we select p to maximize the probability that |E] falls within this range. We set our range É to [400, 500] for IOI and [200, 300] for Greater-Than.
Recall that to evaluate circuits with OA, we perform gradient descent on à to approximate the optimal constants, Since repeating this process expensive, we truncate training after just 200 training batches, far short of the 10,000 batches used for full training run, for both the random circuits and optimized circuits. However, we test using a smaller sample size that the loss for the random circuits does not tend to decrease much with further training; in fact, for the optimized circuit, the loss typically drops by more than 50% after the first batch, which does not occur for the random circuits.
While random circuits achieve lower loss under OA than mean and resample ablation, the Aopt. measurements for random circuits do approach the low figures achieved by optimized circuits. Furthermore, the standard deviation of A for random circuits is lower on average for OA than for mean resample ablation, and surprisingly, the OA losses for optimized circuits have the most significant Z-score for both IOI and Greater- Than, though there is not necessarily a difference between Z-scores of such large magnitude. These results demonstrate that OA likely highlighting specialized circuit components that already exist in the model rather than fabricating non-existent mechanisms.
G
Causal tracing
G.1 Transformer graph representation
Consider running a causal tracing experiment on vertex represent the model with four vertices: Subj(X) representing the subject tokens of the input, Non-Subj(X)) representing the remaining input tokens, the component of concern A(X) Non-Suj(()) and the model output
37

---

Out(X) = Out(Subj(XX) SonnSSubj())A((X))) In particular, if A MLP then we compute (ut(X) as function of these three arguments by computing Equation 77) which takes A(X) and MResid (9) (X) as input, by taking the latter term MResid (X) as a function of Non-Subj(X) and Subj( (),, and then computing Out( X) as a function of Resid (0) (X). A similar construction is used for attention layers.
Note that the AIE compares the performance of the model with the vertex Subj ablated (the denomi- nator in Equation (5) to the performance of the model with only the edge (Subj, Out) ablated (the numerator in Equation
G.2 Relation of ATE to ablation loss gap
For consistency with Meng et al. (2022),
carefully selected loss function in the definition of to represent proximity to the model's original predictions rather than the typical KL.-divergence loss. In particular, choose
CAIE(P, min (0, max Q Jarg max 2)
(24)
where and Q represent probability distributions over the model vocabulary. Note that since the dataset is filtered so that Y arg max M(X), replacing
replacig & with CATE in A, we get
A
BxxYWD max(0, [M()]] [Mx (MMA)))))__MMM)MMMMMMMMMM (Mu(E(X),
(25)
which is the numerator in Equation 5
G.3
Additional results
Causal tracing intervention at subject tokens, window size 1
Attention layers
MLP layers
GNT
0.5
0.5
0.0
0.0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last subject token, window size 1
Attention layers
MLP layers
GNT OAT
GNT
0.2
0.2
OAT
AIE
0.0
0.0
10
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last token, window size 1
Attention layers
MLP layers
GNT OAT
GNT
0.1
0.1
OAT
AlE
0.0
0.0
20
Layer number
Layer number
Figure 12: Causal tracing probabilities for different token positions with window size (patching single component). Error bars indicate the sample estimate plus/minus two standard errors.
38

---

Causal tracing intervention subject tokens, window
Attention layers
MLP layers
1.0
1.0
0.5
AIE
0.5
0.0
0.0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last subject token, window size 5
Attention layers
MLP layers
0.4
0.4
GNT OAT
GNT
AIE
AIE
QAT
0.2
0.2
0.0
0.0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last token, window size 5
Attention layers
MLP layers
0.4
GNT OAT
0.4
GNT
QAT
0.2
0.2
0.0
www w a
Layer number
Layer number
Figure 13: Causal tracing probabilities for different token positions with a sliding window of size 5. Error bars indicate the sample estimate plus/minus two standard errors.
We show results for additional window sizes and token positions. In particular, we show results for intervening at subject token positions, only the last subject token position, and only the last token position, for window sizes I (see Figure T2). 5 (see Figure
13 and (see Figure 14.
In addition providing more precise localization of components' informational contributions, the results provide some evidence against one of the claims of Meng et al. (2022), the idea that the last subject token position a uniquely important "early site" for processing information MLP layers 10-20. For sliding windows of size 5, GNT shows that intervening on MLPs at only the last subject token position achieves over half of the AIE of performing the same intervention all subject token positions (33.8% vs 19.3%, shown in Figure 13). However, the OAT results indicate that intervening all subject tokens much more effective (35.2% vs 11.1%) indicating that early subject token positions may be more important than previously thought.
G.4 Construction of standard errors
For input-label pairs (X D. let W min(0, [M(X)]x (M((E))).A(x)Il)> and = [M(X)) - [M(E(X))],, and let W, and 2, be their respective sample means with samples. Recall from Equation (5) that want estimate from samples the quantity
AIE(A) min (0,
EW ΣZ
min 0. oz ). By the central limit theorem,
Vn
W, 2,
[uw] [Az]
))
4 4 =N -N(0,)
oiv Owz
owz 2
(26)
39

---

Causal tracing intervention subject tokens, window
Attention layers
MLP layers
1.0
0.5
AIE
0.5
0.0
0.0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last subject token, window size 9
Attention layers
MLP layers
0.4
GNT OAT
0.4
GNT
OAT
H+4
0.0
0.0
10
20
30
40
10
20
30
40
Layer number
Layer number
Causal tracing intervention at last token, window size 9
Attention layers
MLP layers
GNT QAT
GNT
0.4
0.4
QAT
0.2
0.2
Layer number
Layer number
Figure 14: Causal tracing probabilities for different token positions with a sliding window of size Error bars indicate the sample estimate plus/minus two standard errors.
[(w] and v Vh ([ww))
By the multivariate delta method, for h
2
H
W
pw
Vn
h
W
- h (Mw)]
) S ON(0,
(27)
2,n
Jiz
2m
Hz
so the asymptotic variance is
pl
22
dWz
2z
PW
fzz
μo pz
which we estimate via samples to obtain our standard errors.
H
OCA lens
H.1 Transformer graph representation
We represent the model with (t) (Residd (tt and MLP()) vertices for each layer and vertex Out(r) representing the model output, where the relationships between the vertices are defined by the equations given in Appendix [C.] Applying OCA lens at layer entails ablating vertices Attn (++1) through Att (where N the number of layers in the model).
H.2 Additional prediction accuracy results
Figure 15 shows results on prediction loss for GPT-2-small, GPT-2-medium, and GPT-2-large, For all models, we use learning rate of 0.01 for tuned lens and 0.002 for OCA lens.
40

---

Lens loss, GPT-2-small
Lens loss, GPT-2-medium
Lens loss, GPT-2-large
OCA
lens
OCA
KL-divergence
(d-divergence
4
Tuned
KL-divergence
Tuned
Resample
Resample
Resample
2
2
20
30
Layer number
Layer number
Layer number
Figure 15: Comparison of prediction loss between tuned lens and ablation- based alternatives.
H.3
Additional causal faithfulness results
The following figures show the causal faithfulness metries with several kinds of perturbations, Let (X)] and Σ = Var((((X)).
Causal faithfulness: random perturbation
Magnitude correlation
1.0
1.00
Direction similarity
GPT-2-small, Tuned
0.8
0.75
GPT-2-small, OCA GPT-2-medium, Tuned GPT-2-medium, OCA GPT-2-large, Tuned GPT-2-large, OCA GPT-2-XL, Tuned
0.6
0.50
0.4
0.25
0.2
GPT-2 XL, OCA
0.00
20
40
20
40
Layer number
Layer number
Figure 16: Causal faithfulness comparison under random perturbations.
Causal faithfulness: basis-aligned perturbation
Magnitude correlation
1.0
1.00
Direction similarity
GPT-2-small, Tuned
0.8
0.75
GPT-2-small, OCA GPT-2-medium, Tuned GPT-2-medium, OCA GPT-2-large, Tuned GPT-2-large, OCA GPT-2-XL, Tuned
0.6
0.50
0.4
0.25
0.2
GPT-2-XL, OCA
0.00
20
40
20
40
Layer number
Layer number
Figure 17: Causal faithfulness comparison under basis- perturbations.
Random perturbation: We sample Z // E), and let V Z/||Z|1. We let Z' No0. 1), Z' IL Z. We define {(a) Z' V. We define the constant such that (I) xs 0.2. Results shown in Figure 16
41

---

Causal faithfulness: random projection
1.0
1.00
Magnitude correlation
Direction similarity
GPT-2-smalI, Tuned
GPT-2-small, OCA GPT-2-medium, Tuned GPT-2-medium, OCA GPT-2-large, Tuned GPT-2-large, OCA
0.75
0.8
0.50
0.6
0.25
GPT-2-XL, Tuned
GPT-2-XL, OCA
0.00
20
40
20
40
Layer number
Layer number
Figure 18: Causal faithfulness comparison under random projections.
Causal faithfulness: resample basis directions
Magnitude correlation
1.00
1.00
Direction similarity
GPT-2-small, Tuned
GPI-2-small, OCA
0.75
0.75
GPT-2-medium, Tuned
GPT-2-medium, OCA
0.50
0.50
GPT-2-large. Tuned
GPT-2-large, OCA
0.25
0.25
GPT-2-XL, Tuned
GPT-2-XL, OCA
0.00
0.00
20
40
20
40
Layer number
Layer number
Figure 19: Causal faithfulness comparison under basis-aligned resample ablation.
Basis-aligned perturbation:
Same as random perturbation, except we choose a basis of dmodel vectors as described in Section 5 and let be a uniformly sampled basis element.
Results shown in Figure 17
Random projection: We sample Z (O, E), and let V Z/| We define {(x:)
= 14), where p represents the projection to the orthogonal complement of V. Results
shown in Figure T8
Basis-aligned projection: Described in the main text. Results shown in Figure
Basis- resample ablation: We choose basis described in Section 5 We consider the subspace spanned by 100 basis elements with the largest singular vectors, and define E(a) by performing resample ablation on the projection of this subspace. Results shown in Figure 19
We find that the improvement in causal faithfulness is consistent across all perturbation types studied.
H.4 Elicitation results on factual datasets
We show additional results for elicitations on the text classification datasets. Figure 20 shows com- prehensive results for each of the individual datasets. For experiments, and sample from datasets without replacement to generate the demonstration examples. Note that we exclude SST2-AB, toy dataset constructed by Halawi al. al. (202)) that replaces SST2 sentiment labels with letters "A" and "B," since it is only created to show that elicitation accuracy does not improve when the expected answer is unrelated non-intuitive manner, information from later layers is required to relate internal knowledge to the correct label). Figure summarizes the elicitation accuracy boost between OCA lens and tuned lens across the datasets.
10 demonstrations
the question (since the label is encoded in
42

---

Elicitation accuracy on text classification datasets, GPT-2-XL
agnews
trec
dbpedia
Calibrated accuracy
0.30
0.8
0.15
20
rte
mrpc
tweet eval hate
sick
Calibrated accuracy
0.54
0.500
0.56
0.54
0.500
0.52
0.50
0.50
0.45
0.450
0.48
poem sentiment
ethos
financial phrasebank
medical questions
Calibrated accuracy
0.55
0.54
0.54
0.52
0.45
0.48
8.40
0.46
0.4
0.35
0.40
0.5
0.46
0.30
0.42
0.2
20
40
tweet eval feminist
tweet eval atheism
unnatural
Calibrated accuracy
0.35
1.0
True demos
OCA lens
0.36
0.35
0.8
True demos
a.x
Tuned lens
False demos
OCA
8:30
0.25
False demos
0.28
Tuned lens
Layer number
Layer number
Layer number
Figure 20: Comparison of calibrated accuracy of elicited completions on 15 datasets from Halawi et al. (2024). Dotted lines indicate the accuracy of the model's output predictions for true demonstrations (black) and false demonstrations (red).
Elicitation accuracy boost comparison
True labels
Permuted labels
gbpedia
unnatural
0.6
0.15
0.5
OCA lens
OCA lens
0.4
0.10
agnews
0.3
dbpedia
mmpc
0.2
0.05
pgnews.
ethog
phansal_prrasebank
0.1
tre sick
mioc
Anancial, phrasebank
0.00
0.0
0.00
0.05
0.10
0.15
0.0
0.2
0.4
0.6
Tuned lens
Tuned lens
Figure 21: Comparison of elicitation accuracy boost between OCA lens and tuned lens.
I Reproducibility
All code can be found at http:://wiiilaooommmm/ttiiinnaat
43

---

All experiments were run a single Nvidia A100 GPU with 80GB VRAM. The cost of UGS is comparable to ACDC (about 1-2 hours to train). Training OCA lens until convergence also takes about 3-5 hours, which is similar to the amount of time to train tuned lens.
J Impact statement
We believe that OA can lead to more granular level of understanding for models' internal mecha- nisms. better understanding of interpretability can help reduce risk from dangerous Als, but more work is required to scale interpretability techniques to larger models. Interpretability can also help us to understand how to build better inductive biases into models, paving the way for future developments in architecture, On the other hand, advanced interpretability can also be repurposed for nefarious applications, like eliciting dangerous knowledge from models' latent space, However, we believe that better interpretability will also provide better clarity on how to mitigate these risks.
44

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract presents our main claims in the paper, including a reference to optimal ablation main proposed technique and its experimental applications.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appendices and F.12
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "L.imiations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
45

---

Answer: [Yes]
Justification:
Mainly, Proposition 2.3 is clearly contextualized and proven in the paper.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide details about experiments in both the main text and in Appendices F G and H for our three main experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice. or if the contribution is a specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
example, if the contribution
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
model (e.g., large language model), then there should
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access data and code
46

---

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
Our code is available online at http://ghhhbbbcommmmmmllliiinnbn
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.c/////// httpp/.cc//ulllc/gguuess///////s Sarriiddea/CodSuubiieeee ssi ssiolllicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Guidelines: We provide details in Appendices F in the code implementation.
H and these details can also be found
Justification:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We treat statistical significance carefully in each experiment. We discuss statistical significance of circuits discovered with OA in Appendix F.12 and discuss error bars for causal tracing in G.4
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
47

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates). If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn" make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wrrp...cccoaab//ccccccca
Question:
Answer:
[Yes]
Justification:
We only use open-source datasets.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consideration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix J
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
48

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any new models. Our experiments run open-source models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with
necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from partieular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
49

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
50


Reviews:
[{"review_id": "dqGzpxyWms", "reviewer": "Reviewer_mH4h", "summary": "Different intervention techniques aim to \"ablate\" parts of the representation of the model to infer their causal function, e.g. by adding gaussian noise. the paper suggests to derive a notion of \"optimal\" ablation. Particularly, instead of zeroing out or replacing the ablated part with its mean, it is proposed to optimize with GD the constant value that minimizes the loss, I.e, the \"perfect\" ablation would be replacing the element with the \"best\" constant value. It is shown that the proposed techniques improves circuit discovery, identifying a subnetwork that reconstructs the original performance of the model in some task.", "questions": "none.", "limitations": "see above.", "rating": 9, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The contribution is well defined and elegant. The proposed method is experimentally validated on several interpretability-related tasks and it improves over commonly used ablation method.", "weaknesses": "I don't see major weaknesses in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your generous feedback, and for recognizing the value of our work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTHoFm6BHR", "reviewer": "Reviewer_WFFz", "summary": "- Introduces a notion of \"optimal component ablation\" for activation patching methods in mechanistic interpretability. Specifically, they propose an ablation method that involves setting a component’s value to the constant value that minimizes ablation loss over a chosen distribution.\n\n- Shows that optimal ablation can improve algorithmic circuit discovery—the identification of sparse subnetworks that recover\nlow loss on interpretable subtasks.", "questions": "See strengths and weaknesses", "limitations": "See strengths and weaknesses", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Simple approach to compute 'optimal' ablations that simulate the removal of a given component/vertex. This fares better than over zero, mean, and resample ablation in terms of ablation loss.\n- The OCA lens is a clever idea to evaluate layerwise representations and an alternative to linear probing / tuned lens.", "weaknesses": "- Writing and paper structure can be significantly better (motivation, problem setup, describing the applications in more cohesive manner, results etc). \n- Justification for why ablation loss is the \"right\" way to measure \"optimality\" could be better. Also, given that the proposed method explicitly optimizes for ablation loss minimization, it is unclear why making this the primary evaluation metric in the experiments is insightful. Are there other downstream metrics for localization that also improve as a result of minimizing ablation loss? For example, does it find smaller circuits that induce the same effect on model behavior? Is the second-order effect on unrelated tasks less (as intended) due to optimal ablations? These evaluations would make the overall story significantly stronger.\n- One of the main contributions (uniform gradient sampling method) is not described well. I would like to see a more step-by-step description of the proposed method. Most of the important details are deferred to the appendix\n- These applications are niche and specific to mechanistic interpretability subroutines. To me, factual recall (the second application) seems like a special case of the first one, as the goal is again to localize a given task down to a subset of model components. I would be interested to see optimal ablations improve model editing (e.g., https://arxiv.org/abs/2404.11534 show that accurate localization via zero ablations can be directly used for model editing).\n- Novelty concerns. I am not sure if the contributions in this paper are strong enough right now, especially given related work in this area, e.g., https://arxiv.org/abs/2309.16042 also focus on systematically evaluating activation patching\n- I like the OCA lens idea, but it's a bit unfair to compare it to tuned lens because OCA lens is still using additional computation via learned ablated layers. I think this experiment requires fair-er baselines that are more of an apples-to-apples comparison (e.g., smaller models with fewer learned layers?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback.\n\n>Writing and paper structure can be significantly better\n\nWe’ve revised the writing so that the structure makes more sense, improving flow and clarifying differences between sections. In particular, we’ve substantially rewritten section 2 to better contextualize our motivation and problem setup, and edited later sections to clarify the motivation and description for each application.\n\n>Justification for why ablation loss is the \"right\" way to measure \"optimality\" could be better.\n\nWe agree and have rewritten this section. See global reply.\n\n>it is unclear why making [ablation loss] the primary evaluation metric in the experiments is insightful\n\nThe ablation loss gap ∆ is the primary metric in previous circuit discovery work. While it is true that OA is guaranteed to have lower ∆ than zero, mean, and resample ablation, studying ∆ still produces several surprising results. \n\n1. The amount that OA reduces ∆ (67-89% for components, 85-97% for circuits) is **very large**. In light of our motivation for minimizing ∆ (see global reply), this suggests most of ∆ for these methods is accounted for by “spoofing,” or inserting inconsistent information into the model, not removing important information.\n\n2. OA produces lower ∆ than CF, which is not guaranteed, since CF is input-dependent and removes less information than OA.\n\n3. Manual circuits achieve close to Pareto-optimal ∆ when evaluated with OA, but for other ablation methods, there are similar-size circuits that get much lower ∆ than the manual circuit. This is likely because there are components that do not serve relevant functions according to the manual analysis but need to be included in the circuit to avoid spoofing the circuit components with conflicting information when ablated. If you believe that manual circuits from previous work capture the important mechanisms, then this is evidence that OA is a better evaluation metric for circuits.\n\nWe agree with the general idea of having more evaluations. We added a result for OCA lens that shows that it elicits more truthful predictions than tuned lens (see global reply).\n\n>does it find smaller circuits that induce the same effect on model behavior?\n\nWe believe we show the answer is yes by looking at the Pareto frontier; for the same effect on model behavior, we can achieve smaller circuits. We will make this conclusion more clear in the paper.\n\n>Is the second-order effect on unrelated tasks less (as intended) due to optimal ablations?\n\nWhile this is an interesting question, we’re not sure we should expect circuits to generalize to unrelated tasks. We are not trying to find a small general-purpose circuit; we are deliberately trying to isolate a circuit that performs one particular task, and we might even prefer the circuit does not perform unrelated tasks because we’d have wasted circuit complexity. \n\n>uniform gradient sampling method is not described well\n\nSee global reply\n\n>These applications are niche and specific to mechanistic interpretability subroutines. \n\nEach of our selected applications is the subject of extensive discussion in interpretability. We acknowledge that in the original submission, we only cited a few relevant works. We’ve added far more citations for each of the applications and previous ablation methods. \n\nFurthermore, as other reviewers have noted, our work presents a new approach to the longstanding question of feature importance and has broad implications beyond mechanistic interpretability. We focus on this area specifically for the sake of cohesion and because it is a growing subfield in which the use of ablation is particularly common and for which our method results in concrete empirical improvements.\n\n>factual recall (the second application) seems like a special case of the first one\n\nWe agree that these applications are mathematically similar, but they are answering substantively different scientific questions about a model, and they are separate lines of inquiry in previous work. Circuit discovery asks how a model performs an algorithmic task, and seeks to find components that perform steps of the algorithm. Factual recall asks where the model has memorized factual associations. We will clarify this point in the paper.\n\n>I would be interested to see optimal ablations improve model editing\n\nThanks for pointing us to this paper, and we agree it would be an interesting result. Our results on localization suggest that our method could do well here. But we think it would require significant extra thought – we’d have to make choices about how to edit the model, compare to various baselines, etc. Given our space limitations, we defer this to future work. \n\n>Novelty concerns. I am not sure if the contributions in this paper are strong enough right now, especially given related work in this area, e.g., https://arxiv.org/abs/2309.16042 also focus on systematically evaluating activation patching\n\nAs noted by other reviewers, our approach is fundamentally quite different from existing approaches, which manifests in dramatic differences in several important metrics. To be clear, unlike the linked paper, this paper is not about evaluating previous methods. Our evaluation is about showing our method outperforms others, including all methods that are evaluated in that paper.\n\n>I like the OCA lens idea, but it's a bit unfair to compare it to tuned lens because OCA lens is still using additional computation via learned ablated layers\n\nSee global reply for more context about why it is a fair comparison – both methods elicit predictions from the last token position without using activations at previous token positions. Indeed, OCA lens is actually a more constrained elicitation model than tuned lens: it has a factor of $d_\\text{model}/n_\\text{layers}$ fewer learnable parameters, since we train a $d_\\text{model}$-parameter constant at each layer for OCA lens, compared to a $d_\\text{model}^2$-parameter weight matrix for tuned lens.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xodgfQe5uN", "reviewer": "Reviewer_nbx7", "summary": "This paper explores a new approach to replacing features in the forward pass of a neural network, for purposes of interpretability. The approach is, rather than zeroing out features or replacing them with some pre-specified constant or random variable, to optimize a replacement constant to minimize the loss of the model over the subtask of interest. So when ablating an attention head or MLP layer, the ablated component’s outputs are replaced with a constant that is optimized to maximize model performance over some data of interest. This is done to be “minimally disruptive to model inference”, e.g. avoid disrupting model inference in the way that OOD replacement values are known to disrupt model inference, as has long been lamented by past work in interpretability. This new feature replacement method is demonstrated in three case studies, focused on circuit analysis, factual association localization, and something to do with the tuned lens that I could not quite understand. Results suggest that this method reveals clearer/sharper/stronger phenomena in circuit analysis and factual association localization, a promising result for interpretability research.", "questions": "- See questions in the above.\n- A note on the references: I think it’s extremely non-standard for the references section to be full of references that do not appear in the paper. I would say that this should be fixed.\n- In the intro, I think causal tracing is described as a noising ablation, when it is a denoising ablation.\n- (Goldowsky-Dill et al., 2023) recommends →  Goldowsky-Dill et al. (2023) recommend\n- “constant to which to ablate” → constant for ablating?\n- \"confluence of conflicting information may cause model representations to spiral out-of-distribution” — are the hidden states during a forward pass in circuit analysis with optimal ablation in-distribution?\n- The sentence beginning with “To assess the magnitude component” is not at all clear.\n- The paper is not in correct submission format, as it is missing line numbers.", "limitations": "The limitations could discuss some higher level points around feature replacement and what it could be useful, vs. what the paper empirically shows it may be useful for. It could also include some of the caveats about the proposed method from Sec. 2.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Very important: The paper’s main strength is that it does something new and interesting in what has been a very saturated space for years. For a long time — since at least 2007 by my count — people have wondered how to remove features (or the outputs of some feature extractor / neural model) from a model, in order to do something like estimate that feature’s effect on the model. The debate rages on, now re-enacted in the era of mechanistic interpretability. While I’m still not sure we have a general theory for this, this paper makes a reasonable proposal on how to do this in a task and model agnostic way, with some promising early results for downstream applications. More specifically, avoiding disrupting model computation “too much” has stood out as a goal of feature replacement methods, and this paper proposes a new method for doing so and provides some interesting case studies showing that this may produce more stronger results in different kinds of interpretability analyses.\n- Important: The paper is ambitious in a way rarely seen in ML papers, with a new method being applied across three case studies. The paper covers a lot of ground quickly.\n- Important: The paper is well-written and the notation is clear.\n- Important: I am sympathetic to the view that subtask-specific interchange interventions can be difficult to construct (a view argued for in the paper, that motivates the proposed method). At the very least, they take some manual effort to construct and are subjective. A drop-in automated replacement could be very useful. For what it’s worth, an argument I would add to the paper is that task-specific counterfactuals can sometimes be difficult to interpret as well. Does changing a name from Alice to Carol have *only* the effect of removing task-relevant information? We can imagine some hidden effects appearing with counterfactuals that we try to construct to remove only one variable, especially for more complex tasks.\n- Important: The results on circuit analysis and localization are pretty promising, in my view. (I have a very hard time interpreting the results regarding the tuned lens — I don’t understand that experiment’s setup or its broader purpose.)\n- Important: Experiments appear to be conducted with great care. Many small design choices seem good and clever, like the ReLU in Eq5 preventing rewarding the optimization process for finding a value that leads to over-restoration of model predictions during the causal intervention.\n- Important: Appendix D shows that the optimization process is unlikely to adversarially induce good task performance in the model, and therefore unlikely to produce interpretability artifacts, because the causal intervention does not seem expressive enough to so.", "weaknesses": "- Very important: In my opinion, the approach presented cannot be the right one in general, preventing the proposal from providing a generic solution that circumvents the need to design subtask-specific interchange interventions / counterfactuals. There are two reasons for this, one a criticism of the argument in the paper and the other a counterexample. First, the argument in the paper, while being mathematically clear, is not philosophically rigorous. It is said that we want to “simulate the removal of” model components. What does that mean? I could imagine removing a layer from a Transformer by not running it — by virtue of the residual connection between layers, this is equivalent to zero-ablating the outputs of that layer. Yet the authors take issue with zero ablation, so removing a component must mean something else. Then, to remove a component, it is said that we should be “minimally disruptive to model inference”. What does that mean? Evidentally we are disrupting the model inference. What does it mean to not disrupt it too much? There are some other specific claims that are difficult to interpret precisely, such as avoiding an intervention that “disrupts this flow of information” or using an intervention that provides information that is “inconsistent with information the model derives from other vertices.” To be clear, I basically do like these kinds of arguments in ML papers, because we need arguments for what is right and wrong to do stated in plain English, but I am not sure this one is adequate for proving the point that optimal ablation is the right way to go about things.\n    \n    Second, a counterexample can build on an intuition hinted at in the paper that “there may not exist a constant value that perfectly conveys a lack of information.” The problem is that it is not clear that optimal ablation is the answer to an intelligible question. For example, suppose we have a neural model that classifies people into “give loan” and “reject loan” categories, and that we know that a neuron in this model represents a person’s annual income in USD. So, we know what 0 means, and we know what 100,000 means (it’s their income). Now let’s say there’s a component that takes in this value, and we want to know this value’s “importance” to that component (or later model outputs), so we replace this value with the optimal ablation constant c. What value of c would be correct for this? The question doesn’t really make sense, and it’s because it doesn’t make sense to ask how “important” a component is. Importance is not well-defined concept in causation. What would make sense is to ask what setting the income value to 100,000, when it was 60,000, for a particular datapoint $x$ does to the output of that downstream component. I would posit that the important thing about counterfactual patching / interchange intervention is not that it “selects uninformative counterfactual inputs“ but rather that it selects *known* counterfactual inputs, i.e. inputs we know the meaning of. This allows us to say precsely what the causal intervention means, as well as what its effect was. This is the argument in https://proceedings.neurips.cc/paper_files/paper/2021/file/4f5c422f4d49a5a807eda27434231040-Paper.pdf, as far as I can tell. \n    \n- Important: while results are plenty promising on the first two case studies, optimal ablation seems to hardly improve over marginal resampling based on Table 1, and the improvement is extremely marginal looking at the rank correlation in Fig. 6 where a linear correlation seems inappropriate for the log-log relationships. So why then, does optimal ablation perform far better than counterfactual patching in Fig. 1, when counterfactual patching was just treated as something of a ground-truth in Sec. 2.  This apparent over-performance in circuit analysis seems like it could be an interpretability artifact, in spite of Appendix D suggesting overfitting is unlikely.\n- Important: I really could not tell what the third case study was trying to show, or how it tried to show it. See next point.\n- Of some importance: The paper suffers from trying to do so much in nine pages. I think it would be difficult for someone without a deep background in interpretability to read this paper and get as much out of it as the authors would hope. Unless the authors feel strongly about the third case study, I would totally remove that and focus on providing more detail to earlier sections of the paper.\n- Of some importance: It is hard give any credit to the paper for the introduction of the uniform gradient sampling (UGS) method because it is introduced without nearly enough detail, and without basically any motivation, in the main body of the paper. And it is evaluated only in one case study. I can’t tell if the UGS direction is big enough to split off into another paper, but it doesn’t feel like it belongs in this one.\n- Of some importance: The organization of the paper is a little unorthodox, without a related work or conclusion section. While the introduction does a good job situating the paper in the literature, I would recommend discussing some earlier work on how this entire debate has played out with feature attribution methods before the intensified focus on “mechanistic” interpretability (i.e. methods focused on data space ablations and not only hidden feature ablations), including (1) https://arxiv.org/pdf/1806.10758 (2) https://arxiv.org/pdf/1910.13413 (3) https://arxiv.org/pdf/2106.00786.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and helpful feedback, and we appreciate your optimism about our work!\n\n>First, the argument in the paper, while being mathematically clear, is not philosophically rigorous\n\nWe agree and have rewritten this section (see global reply). The term “total ablation” clarifies what we consider ablation: replacing the value of component A with a random variable that is independent from the input X, so the new value of A cannot provide any information about X. We think the revised section 2.2 crystallizes why it’s desirable to reduce ∆. \n\n>it is not clear that optimal ablation is the answer to an intelligible question. For example, suppose we have a neural model that classifies people into “give loan” and “reject loan” categories, and that we know that a neuron in this model represents a person’s annual income in USD\n\nInteresting example! In this case, we think OA does, in fact, answer an intelligible question: “How much worse does the model perform if it had to treat everyone as having the same income?” which is arguably extremely similar to “How much worse would the model perform if it had no information to distinguish anyone’s income?,” i.e. \"How important is information about income to the model's performance?\" \n\n> it doesn’t make sense to ask how “important” a component is. Importance is not well-defined concept\n\nWe agree it’s not clear “importance” is uniquely defined, but we think it’s still helpful to discuss! If we know a component represents income, there are specific interventions we can assess (e.g. input-counterfactual pairs). But if we aren’t sure what it represents, as is typically the case in interpretability work, we may make mistakes like (as you mentioned) changing its value from Alice to Carol without changing a different component that stores the “word starts with A” feature to a “word starts with C” feature, making the model lose coherence (whereas OA could have changed the component from Alice to a value that maintains consistency with whatever other components reflect). Even if we know what each component represents, it’s often too complicated to assess causal interventions between every pair of inputs, and it’s helpful to have a simple conception of importance that somehow aggregates causal effects to guide our search for relevant components a model uses to perform an algorithm. Our claim is that the average causal effect of setting an activation for every input to the same constant adequately summarizes importance (similar claims are made to justify other ablation methods).\n\n> while results are plenty promising on the first two case studies, optimal ablation seems to hardly improve over marginal resampling based on Table 1\n\nWe were also confused, and when we thought about this experiment more carefully, we realized we implemented the ablation methods inconsistently and were not doing an apples-to-apples comparison. For each ablation method, we can either perform some prescribed intervention at the sequence level, or iterate an intervention over each token position in the sequence (note that Prop 2.3 only holds if we are consistent in this choice). For resample ablation, we were performing sequence-to-sequence replacement by resampling between corresponding token activations in a different input sequence, but for mean/optimal ablation, we were performing token-level replacement by setting activations to the same value at all sequence positions.\n\nWe updated Table 1 to adopt the first approach for every method (i.e. conditioning on sequence position), which shows a much larger improvement of OA over resampling.\n\n| | Zero | Mean | Resample | CF-Mean | Optimal | CF |\n|-|-|-|-|-|-|-|\nLog-log correlation with CF | 0.626 | 0.831 | 0.826 | 0.847 | 0.908 | 1 |\nMean ∆ | 0.0584 | 0.0405 | 0.0559 | 0.0412 | 0.0035 | 0.0296 |\nMedian ratio of ∆(opt) to ∆ | 11.1% | 33.0% | 17.7% | 31.7% | 100% | 88.9% |\n\nFor completeness, we’ll also show in the appendix that if we implement all ablation methods in the tokenwise fashion (e.g. resampling all tokens individually), OA still shows much higher correlation with CF. \n\n> why then, does optimal ablation perform far better than counterfactual patching in Fig. 1, when counterfactual patching was just treated as something of a ground-truth in Sec. 2.\n\nJust to clarify, our stance is that, among *previous* methods, CF may best reflect effect 1 (which we loosely consider “ground truth importance”) from the rewritten part of section 2.2, and we will clarify this point in the paper. Thus, it would make sense that, if OA were now the closest method to “ground truth importance,” then OA would also be the method closest to CF among existing methods. We present Table 1 to confirm this, and do not mean to present CF as a ground truth. \n\nEven for single components, OA achieves lower ∆ on average than CF (0.0035 vs 0.0296), in line with what we see for circuits. The disparity may be larger for circuits due to ablating many components at once. Rushing and Nanda (2024) showed that models can self-repair if a few components contribute weird values; however, ablating many components with previous methods may contribute much more to effect 2 (“spoofing” the model with inconsistent values).\n\n> I really could not tell what the third case study was trying to show\n\nSee global reply\n\n> the uniform gradient sampling (UGS) method…is introduced without nearly enough detail\n\nSee global reply\n\n> I would recommend discussing some earlier work on how this entire debate has played out with feature attribution methods\n\nGood suggestion and thanks for the references! We will add discussion.\n\n> it’s extremely non-standard for the references section to be full of references that do not appear in the paper\n\nWe will correct this and the phrasing corrections mentioned in the questions section.\n\n> The limitations could discuss some higher level points [and] could also include caveats about the proposed method from Sec. 2.\n\nWe agree and will add these points.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EewtT2w1z8", "reviewer": "Reviewer_Nr7M", "summary": "This paper presents an alternative approach to perform component ablations for neural network interpretability. Specifically, the authors consider a neural network as a causal graph and propose *optimal ablations* which simulate component removal by setting the value of a node in the computational graph to a constant value. In contrast to alternative task-agnostic ablation methods (e.g. zero and mean ablations), this constant is learned using gradient descent. The authors argue that alternative task-agnostic ablation methods could confuse downstream computations as these specific values might not have been observed in the training data and then demonstrate how optimal ablations can lead to improvements across a range of different application domains: 1. They evaluate single-component ablations on the indirect object identification (IOI) task and find that optimal ablations perform better than other subtask-agnostic methods (although still worse than subtask-specific approaches); 2. They perform circuit discovery on the IOI and Greater-Than tasks using a new method, which they term uniform gradient sampling (UGS), which outperforms existing circuit discovery approaches; 3. They study factual recall, or more specifically localize where factual associations are stored within a language model, and in contrast to previous studies observe that adjacent components often have very different patching losses which suggests that factual associations are likely stored in single layers, as opposed to being spread across multiple layers as previously hypothesized; 4. Finally, the authors apply optimal ablations to the domain of decoding latent predictions. Here, they propose the optimal constant attention (OCA) lens, which sets all following components to their optimal constant value, instead of skipping these by learning a linear map such as in the tuned lens. They find that OCA scores better on KL-divergence and in terms of faithfulness to the model computations.", "questions": "1. Resample ablations are almost on par with optimal ablations in the single-component ablation setting (see Table 1), but perform significantly worse than mean and optimal ablations when it comes to making latent predictions (see Figure 3). Do you have a hypothesis as to why this might be the case?", "limitations": "I believe the limitations are properly addressed in the paper.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper made a strong impression on me as it combines methodological contributions with extensive empirical studies. The authors take a simple but well-motivated idea and then demonstrate how it can lead to improvements across a range of different interpretability application domains. To this end, the authors propose a range of novel methods that leverage this idea, including UGS, and OCA. I believe that these results could have a significant impact on the (mechanistic) interpretability field and should be of interest to various other subfields.", "weaknesses": "- Some of the applications lack depth as a result of the number of different methods studied. For example, it would have been interesting to further investigate the observations on factual recall. \n- Optimal ablations still appear to perform worse than subtask-specific ablation methods in single-component ablations. However, the authors make a reasonable argument that these are subjective and require human and require manual effort, making them hard to use in some settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and helpful feedback, and we’re glad you appreciated our work!\n\n> Some of the applications lack depth as a result of the number of different methods studied. For example, it would have been interesting to further investigate the observations on factual recall. \n\n## Comment on Figure 2\n\n**We agree that it would be great to show further results on factual recall. We’ve extended our analysis to include localization results for different token positions and sliding window sizes as considered by Meng et al. (see PDF attached to global reply), which we will add to the appendix. We also made a few modifications, like increasing our dataset size for factual prompts from 500 to 4,000 and changing the y-axis to reflect a percent (%) difference between original and corrupted probability recovered rather than a difference in percentage points, and added appropriate standard errors to the figures, which is why the new Figure 2 looks different.**\n\nThere are more directions we could pursue, like human mechanistic interpretability analysis to investigate how the recovered probability occurs, but in general, we acknowledge that we have limited space, and want to show as many different applications of OA as possible. We hope we’ve done enough to show that OA is a promising and versatile method; depth will hopefully come from further studies. \n\n> Optimal ablations still appear to perform worse than subtask-specific ablation methods in single-component ablations. However, the authors make a reasonable argument that these are subjective and require human and require manual effort, making them hard to use in some settings.\n\nNote that the only place where OA performs worse than CF is the predictive power of single-component OA loss for inclusion in the manual circuit (the second line of Table 1 in the original submission). However, one element we should point out is that **the manual circuit was discovered by using counterfactual ablations to investigate the model,** which provides a strong inductive bias in the human studies toward selecting components that create high loss when ablated with CF. Thus, it’s not surprising that single-component CF has better predictive power for this particular circuit.\n\nInstead, the point of this analysis is to confirm that OA produces measurements that are related to CF. To clarify, our stance is that, among *previous* methods, CF likely best reflects effect 1 (which we loosely consider “ground truth importance”) from the rewritten part of section 2.2 (see global reply), and we will clarify this point in the paper. Thus, it would make sense that, if OA were now the closest method to “ground truth importance,” then OA would also be the method closest to CF among existing methods, and we present the circuit prediction result in Table 1 to confirm this occurrence. However, after further reflection, we think this result may only confuse the reader, since we do not propose to use single-component ablation loss for circuit discovery, and propose to remove it from the paper (while keeping the correlation result).\n\n> Resample ablations are almost on par with optimal ablations in the single-component ablation setting (see Table 1), but perform significantly worse than mean and optimal ablations when it comes to making latent predictions (see Figure 3). Do you have a hypothesis as to why this might be the case?\n\nYes – one general theme is that replacing activations from one sequence with those from another sequence produces the least loss of coherence when the sequences have a similar format or share many tokens in common. The single-component ablation experiment uses IOI, a dataset with 13 similar-looking prompt templates, so we are resampling between sequences that mostly look the same. But in the lens experiments, we are trying to perform latent prediction on the entire OpenWebText dataset, and we are resampling activations between sequences that may have completely different structure and content, hence the significantly worsened performance of resample ablation. This result furthers the notion that OA is more universally applicable than resample ablation and, in particular, can be used for more diverse datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive feedback. We use this space to address questions shared between multiple reviewers.

**Reviewers note our motivation for minimizing ∆ could be clearer.** We agree and propose the following changes. We add the following definition to section 2.1:

>For any model component $A$, a **total ablation method** satisfies $M^{\setminus A}(X) = M_A(X,Z)$ for some random $Z$, where $Z\perp \mkern-18mu \perp X$.

This definition aligns with an intuitive notion of entirely removing $A$: applying a total ablation method prevents $A$ from providing any information that distinguishes between inputs $X$. Total ablation methods include zero, mean, resample, and optimal ablation.

We also rewrite part of section 2.2:

>To see why lower ∆ is desirable, consider that intuitively, for ablation methods that involve intervening on M by replacing A(x) with a different value a, there are two potential contributors to ∆.

>1. Information deletion. The original value A(x) could carry informational content, specific to each x, that serves some mechanistic function in downstream computation and helps the model arrive at its prediction M(x). Replacing A(x) with some other value $a$ could delete this information about x, hindering the model’s ability to compute the original M(x).

>2. Information spoofing. The replacement value $a$ could insert information about the input that is inconsistent with information about x derived from retained components, “spoofing” the downstream computation. This confluence of conflicting information may cause later activations that combine A(x) with other information to become incoherent, leading to high ∆ because these abnormal activations were not observed during training and thus not necessarily regulated to lead to reasonable predictions. 

>Measures of component importance seek to isolate the contribution of effect 1 from that of effect 2. Like other total ablation methods, optimal ablation captures a maximal information deletion effect since $a^*$ does not depend on x. However, compared to other methods, OA minimizes the contribution of information spoofing to ∆ by setting ablated components to constants $a^*$ that are maximally consistent with information from other components, e.g. by conveying a lack of information about the input or by hedging against a wide range of possible x rather than strongly associating with a particular input other than the original x. Optimal ablation does not entirely eliminate information spoofing, since it may be the case that every possible value of A conveys at least weak information about the input. However, the excess ablation gap ∆ − ∆(opt) for ∆ measured with ablation methods that replace A(x) with a (random) value A is *entirely* caused by information spoofing, since replacing A(x) with the constant $a^*$ achieves lower loss without giving away any more information about x. In practice, ∆ − ∆(opt) for prior ablation methods is typically very large compared to ∆(opt). In Appendix C.3, we show that on average, ∆(opt) accounts for only 11.1% of ∆(zero), 33.0% of ∆(mean), and 17.7% of ∆(resample) for attention heads and MLP blocks on a prototypical language modeling subtask. Section 3 extends this finding to circuits: circuits manually selected in prior work to capture important mechanisms on language subtasks incur 85-97% lower loss with OA than with other ablation methods. This disparity indicates that effect 2 dominates these other ∆ measurements, making them poor estimators for effect 1 compared to OA.

**Reviewers note that we do not spend much time explaining UGS.** While UGS may be of interest to some, it is an auxiliary contribution. As some pointed out, we packed a lot of content into this paper, which does not leave us room to delve separately into UGS in the main text. We choose to de-prioritize it, but wish to acknowledge that it is novel and could be useful to others, so we prefer to describe it briefly in the main text and defer additional context to the appendix for interested readers.

**Reviewers found the presentation of OCA lens confusing.** We rewrite some of section 5 to better contextualize the motivation for OCA lens and why the comparison to other methods is fair. We add references to logit attribution, a popular precursor to tuned lens. Latent prediction methods, e.g. logit attribution and tuned lens, ascribe semantic meaning to the activation at the last token position (LTP) at an early layer $i$ by making next-token predictions using only that activation.

>Tuned lens allows researchers to study *when* important information is transferred to LTP: if replacing $\ell_N(X)$ with $\hat\ell_N(X)$ achieves low loss, then $\ell_i(X)$ already contains crucial context for computing $M(X)$, and key information was transferred prior to layer $i$. Similar to tuned lens, OCA lens reveals whether the LTP activation at layer $i$ contains sufficient context to compute $M(X)$ by eliminating information transfer from previous token positions to LTP after layer $i$.

Tuned lens and OCA lens share the information-theoretic limitation that they are functions of *only* the LTP activation at layer $i$ and do not depend on activations at other token positions. Tuned lens is a linear map, while OCA lens is a function that involves applying later MLP layers and adding constants, and has far fewer learnable parameters: $O(Nd_{model}) < O(d_{model}^2)$. A key insight is that ablation is a more parameter-efficient way to elicit latent information than training a simple auxiliary model.

**Updates to results.** We update Figure 2 by augmenting our dataset and slightly modifying the metric. (See “Comment on Figure 2” in our reply to review Nr7M for more.) Additional figures show more added results for factual recall, varying token position and window size. We also add a new evaluation for OCA lens demonstrating that elicited responses on news and wiki datasets are more truthful than those for tuned lens.

Meta-review: Pretty interesting paper on how to intervene on model internals in a principled way. Standard approaches are to zero out or add noise to model components in order to probe for their role in performing certain tasks or subtasks. Instead, the authors instead propose to find the particular constant value that minimizes the loss. They then apply this idea to some problems in mechanistic interpretability, such as finding locations in the model weights for memorized facts, a useful task for model editing. Most reviewers agree that the paper is innovative and interesting. The authors’ rebuttal also addresses a lot of the questions and should lead to a solid paper.
Decision: Accept (spotlight)