Transformer Circuit Evaluation Metrics Are Not Robust
Authors: Joseph Miller
Bilal Chughtai
William Saunders
Venue: COLM (2024)
Keywords: mechanistic interpretability, interpretability, circuits
TL;DR: We show that the faithfulness metrics used to measure performance of circuits in previous work in mechanistic interpretability are not robust to slight changes in experimental setup.

Abstract: Mechanistic interpretability work attempts to reverse engineer the learned algorithms present inside neural networks. One focus of this work has been to discover 'circuits' - subgraphs of the full model that explain behaviour on specific tasks. But how do we measure the performance of such circuits? Prior work has attempted to measure circuit 'faithfulness' - the degree to which the circuit replicates the performance of the full model. In this work, we survey many considerations for designing experiments that measure circuit faithfulness by ablating portions of the model's computation. Concerningly, we find existing methods are highly sensitive to seemingly insignificant changes in the ablation methodology. We conclude that existing circuit faithfulness scores reflect _both_ the methodological choices of researchers as well as the actual components of the circuit - the task a circuit is required to perform depends on the ablation used to test it. The ultimate goal of mechanistic interpretability work is to understand neural networks, so we emphasize the need for more clarity in the precise claims being made about circuits. We open source a library at [this https URL](https://github.com/UFO-101/auto-circuit) that includes highly efficient implementations of a wide range of ablation methodologies and circuit discovery algorithms.

Published as a conference paper at COLM 2024
Transformer Circuit Faithfulness Metrics Are Not Robust
Joseph Miller' FAR AI
Bilal Chughtai
William Saunders Independent
Independent
Abstract
Mechanistic interpretability work attempts to reverse engineer the learned algorithms present inside neural networks. One focus of this work has been to discover 'circuits' - subgraphs of the full model that explain behaviour on specific tasks. But how do we measure the performance of such circuits? Prior work has attempted to measure circuit 'faithfulness' - the degree which the circuit replicates the performance of the full model. In this work, we survey many considerations for designing experiments that mea- sure circuit faithfulness by ablating portions of the model's computation. Concerningly, we find existing methods are highly sensitive to seemingly insignificant changes in the ablation methodology. We conclude that ex- isting circuit faithfulness scores reflect buth the methodological choices of researchers as well as the actual components of the circuit- the task a circuit is required to perform depends on the ablation used to test it. The ultimate goal of mechanistic interpretability work is to understand neural networks, we emphasize the need for more clarity in the precise claims being made about circuits. We open source a library at this https URL that includes highly efficient implementations of wide range of ablation methodologies and circuit discovery algorithms.
Introduction
Mechanistic interpretability (MI) is a form of post-hoc interpretability that attempts to reverse engineer neural networks to provide faithful low-level explanations of model behaviour (Olah et al., 2020). One focus of interpretability work on transformer language models is identifying 'circuits' - subgraphs of the entire model's computational graph that are primarily responsible for the model's 's output on some task (Wang et al., 2023); where a task is specific type of problem that a language model has to solve to output correct next-token predictions (ie. sentences that require a specific algorithm to complete correctly).
A key metric used by mechanistic interpretability (MI) researchers to quantify the quality of a 'circuit' for some task is it's faithfulless - that is, the degree to which the circuit captures the performance of the entire model (Zhang & Nanda, 2024). In this work, we study various small and reasonable seeming variations on methodologies for measuring circuit faithfulness and find that such variations often lead to significantly different faithfulness scores. Faithfulness is typically measured by performing a targeted, circuit-eepnndent ablation to the model, and observing the effect of this on some metric of the model's output. In the context of MI, an ablation refers to a type of intervention made on the activations of a model during its forward pass with the intended purpose of 'deleting' some causal pathway(s), thereby isolating the causal effect of the circuit.
In this work, we seek answer the questions: What do circuit faithfulness metrics actually show? To what extent are they useful test of the circuit and to what extent are they reflection of the experimental methodology?
We begin by reviewing the ways in which MI researchers may vary their ablation methodol- ogy (Section 3), providing detailed review of methods for ablating transformer circuits. Next, we test these variations on existing circuits discovered by MI researchers (Section 4). We provide detailed case studies of the Indirect Object Identification' circuit by Wang et al.
Corresppnddence to josephmiller josephmiler110@@mmai..oomm @gmail.com

---

Published as a conference paper at COLM 2024
(2023), the 'Docstring' circuit by Heimersheim & Janiak (2023) and the 'Sports Players' circuit by Nanda et al. (2023b). We then go on to study 'optimal circuits' (Section 5) in the context of automated circuit discovery (Conmy et al., 2023) an emerging paradigm that aims to discover circuits algorithmically, without human input.
We conclude with recommendations for MI researchers (Section 6). We additionally re- lease AutoCircuit, a library containing efficient implementations of the circuit-discovery and circuit-evaluation techniques used in this paper, that is significantly faster than prior implementations we tested (see Appendix A for more details).
Related Work
Circuit Analysis. Circuit analysis is a form of post-hoc interpretability focused on un- derstanding the full end-to-end learned algorithm responsible for some specified narrow behaviour. A circuit is a subgraph of the full computational graph of the model that (is alleged to) implement some precise behavior. Circuits have been studied in vision models (Cammarata et al., 2021; Olah et al., 2020) and in toy transformer models (Nanda et al., 2023a; Chughtai et al., 2023). More recently, the circuit analysis paradigm has achieved success in interpreting transformer language models too, with a number of papers discovering circuits implementing human understandable algorithms through ablation studies (Wang et al., 2023; Heimersheim & Janiak, 2023; Hanna et al., 2023). To accelerate such studies, recent work has attempted to automate the process of discovering circuits (Conmy et al., 2023; Syed & Rager, 2023; Kramar et al., 2024), particularly in large language models, as circuits have historically required a large amount of researcher-effort to uncover. Prior work has suggested that ideal circuits exist on the Pareto frontier of faithfulness, completeness and simplicity (description length), as the entire network is trivially optimal for the first two criteria (Sharkey, 2024).
Activation Patching. Zhang & Nanda (2024) recommend best practices in Activation Patching (a form of ablation, defined in Section 3.1) for measuring circuit faithfulness in similar work to ours. They compare single layer vs. multi-layer ablation, Resample Ablation vs. Noise Ablation and logit difference vs probability metrics when Node Patching. We study a larger set of variations in ablation methodology in this work, enumerating several more choices in methodology and arguing that different optimal circuits are defined in part by different ablation methodologies, rather than prescribing a single correct approach to ablation.
Faithful explanations in NLP. We are interested in explaining model behavior in way that reflects the underlying reasoning process of the model, a criteria often referred to as faithfulness. In this work we measure faithfulness by studying the fidelity of ablated models - the similarity of the ablated output to the outputs of the full model (Alishahi et al., 2019; Guidotti et al., 2018; Agarwal et al., 2024). As argued by Jacovi & Goldberg (2020), faithfulness should be viewed as a continuum. Any interpretation is an approximation that will necessarily fail to capture some aspects of the underlying behavior.
Mechanistic interpretability (MI) attempts to reverse engineer trained machine learning models to produce faithful human understandable explanations of model predictions via analysis of the low level features and algorithms implemented by the network. Circuit analysis is just one important direction in this theme of work. Besides circuit analysis, MI more broadly seeks to understand the correct frame to interpret neural network computation (Elhage et al., 2021; Bricken et al., 2023; Cunningham et al., 2023) and to understand the learned features of models (Li et al., 2023; Tigges et al., 2023; Gurnee & Tegmark, 2024; Bills et al., 2023). MI has also inspired work in steering model outputs through representation engineering (Turner et al., 2023; Li et al., 2024; Rimsky et al., 2024).
3 Measuring Faithfulness
We follow previous works (Wang et al., 2023; Heimersheim & Janiak, 2023; Hanna et al., 2023) in defining faithfulness of circuits as the extent to which they encapsulate the full
2

---

Published as a conference paper at COLM 2024
Choice
Granularity
Component
Value
Token positions
Direction
Set
Examples
Heads, MIPs
Node
Resample/
All tokens
Ablate Clean
Circuit
Q K, V,
Edge
Specific tokens
Restore Clean
Complement
Heads, MLP Neurons
Beanch
Mean
Sparse features
Noise
Table The six-tuple that defines ablation methodology for transformer circuits.
model's computation particular task. These works measure faithfulness by ablating the components of the computational graph that are not in the circuit and observing the change in output of the model.
However, even within this framework, there are several important further choices when designing experiments, which we review in this section and summarise in Table 1. We also provide à summary of the approaches taken by previous works in Table 3.
3.1 Ablation Methodology
In the context of MI, an ablation refers to a type of intervention made on the activations of a model during its forward pass with the intended purpose of 'deleting' precise causal pathways. In the language of casual inference, we denote the ablation of all activations outside circuit C on model M as:
F(x) = M(x do(a = a)), a E C
(1)
Where x is the input to the model, a is an internal activation of the model and Ā is the ablated value of a. The ablation methodology determines the types of activations and values that and ä can be (eg. whether a is neuron node activation or an edge between attention heads).
Intuitively, deleting important subcomponents for some task should damage task per- formance, and conversely deleting unimportant sub-components should preserve task performance. As such, ablations have arisen as commonly used tool for localizing model behaviour to specific internal model components. Ablations may be used both to find and evaluate mechanistic explanations of model behavior.
The concept of ablation overlaps with related technique, activation patching, in which activations are modified during a model's forward pass to some cached values from different input. 'Corrupted' inputs are inputs which are similar to the 'clean' distribution being studied, but which have crucial differences that drastically change the output. For example, typical 'corrupt' prompt could retain the structure of 'clean' prompt, while switching proper noun, such that the correct next token prediction is changed. In this work we consider activation patching to be specific type of ablation, and use the term Resample Ablation interchangeably. But we note that in general, 'patching' means editing activations to some other value, instead of 'deleting' them, as ablation typically connotes.
In the remainder of this section, we review the range of ablation techniques that exist in the literature, specifically as they relate evaluating circuits. There exist several important experimental design choices when evaluating transformer circuits via ablations. These are (1) the granularity of the computational graph used to represent the model, (2) what type of component in the graph is ablated, (3) what type of activation value is used to ablate the component, (4) which token positions are ablated, (5) the ablation direction (whether the ablation destroys restores the signal) and (6) the set of components ablated (the circuit or the complement of the circuit). A circuit-based ablation methodology can therefore be specified as six-tuple, and prior work has used many different combinations (Table 3). In this paper we argue that existing evaluations of circuits are sensitive to each of these variables.
3

---

Published as a conference paper at COLM 2024
(a) Node Patching (often called Activation Patching) replaces the output some component to the residual stream in the unfactorized transformer.
(b) Edge Patching replaces the activations a single edge in the factorized view of a trans- former.
(c) The 'trecified' formulation of a transformer separates ev- ery path from input to output.
Figure 1: Different ways of ablating the transformer architecture.
3.1.1 Circuit Granularity
In this work we study circuits specified at the level of attention heads and MLPs!. We also separate the input of each attention head into the Q K and V inputs, but we omit this from our diagrams for visual simplicity. This is the most common granularity for mechanistic circuit analysis (Conmy et al., 2023; Wang et al., 2023; Heimersheim & Janiak, 2023; Hanna et al., 2023; Nanda et al., 2023b)), but previous works have also studied circuits specified at the level of layers (Meng et al., 2022), neurons (Vig et al., 2020), subspaces (Geiger et al., 2023) and sparse "features" (Marks al., 2024).
3.1.2 Ablation Component Type (and Associated Model Views)
Transformers can be described as computational graphs in several different, equivalent ways. We can choose to write the graph as residual network (Figure 7a) or 'factorized' network in which all nodes are connected via an edge to all prior nodes (Figure 7b) (Elhage et al., 2021). Or we can write down 'treeified' network that separates all paths from input to output (Figure 8a). All formulations are equivalent but the 'factorized' view allows us to isolate interactions between individual components and the 'treeified' view allows us to isolate chains of interactions from input to output.
The component type defines the type of intervention made: we detail three possibilities, with increasing granularity. The more granular approaches are generally more difficult to implement and more computationally expensive.
(1) Nodes. We may intervene on node (in the standard, residual view) during the forward pass, replacing its activation with some other value (Figure 1a). This the least specific form of ablation. Since all downstream nodes 'see' the change there are a large number of causal pathways affected by the ablation, which may result unintended side-effects. This type of ablation is also known as (vanilla) activation patching (Vig et al., 2020) when we ablate with a cached activation from another input.
(2) Edges. Using the factorized view of transformer, we may intervene two components (Figure 1b). This is more specific than ablating nodes, as only the specified destination node receives the ablated activation of the source node, smaller number of causal pathways are affected.
edge between
(3) Branches. The previous two ablations can be applied to individual nodes or edges, or to a collection of nodes and edges. Branch ablations on the other hand can only be applied to paths from input to output (Figure 1c). The causal effect of individual paths through the model is isolated by 'treeifying' the factorized model. This approach was introduced al. (2023)) and is key component
by Chan et al. (2022) (formalized by Goldowsky-Dii of a rigorous circuit evaluation approach known as Causal Scrubbing. However, because the number of paths the treeified model is exponential in the number of layers of the
See Thickstun (2024) for brief overview of the transformer architecture.
4

---

Published as a conference paper at COLM 2024
Attn
Irput
Irput
MLP
Outout
Att11
(Table Row 3) Edge Patching all the edges not in a circuit with corrupt activations en- sures that information from the clean input only flows through edges included in the cir- cuit.
(Table 2, Row 2) Edge Patching all the edges in circuit with clean activations allows in- formation from the clean input to flow along paths not included in the circuit.
Figure 2: Two approaches to testing a circuit's faithfulness.
model this approach to circuit evaluation is often intractable in practice. We omit treeified experiments in this work.
3.1.3 Ablation Value
When performing causal intervention some activation, we may choose what value patch in. The simplest choice is to Zero Ablate, by replacing the activation with a vector of zeros (Olsson et al., 2022; Cammarata et al., 2021). Prior work has noted however that the zero point is arbitrary (Wang et al., 2023). The next simplest is to apply Gaussian Noise (GN) to the token embeddings of the clean input to obtain corrupted activations (Meng et al., 2022). Both of these approaches can take the model significantly out of distribution (Zhang & Nanda, 2024), producing noisy outputs (Wang al., 2023).
Two more principled approaches are Resample Ablation (take an activation from some other corrupted input) (Vig al., 2020; Meng et al., 2022), and Mean Ablation (replace with the mean activation of a node from some distribution) (Wang et al., 2023). These two ablation types have the desirable property of keeping the model closer to its usual distribution of activations. Importantly, they do not delete all information present in component. Instead, they delete information that varies across the distribution, while preserving information that is constant across it, allowing generic grammar processing. When Mean Ablating, we have an additional choice in the size of the mean ablation dataset (see Section 4.1). We focus Mean and Resample Ablations in this work.
to isolate precise language tasks, while ignoring, say,
3.1.4 Token Positions
Circuits in autoregressive transformers on a narrow distribution are sometimes defined in terms of components and token positions. When these token positions are specified, we can choose to either ablate all token positions, or only the token positions not in the specified set (Wang al., 2023). Taking a; as the activation token position we can modify equation (1) to:
F(x)= = M(x do(a; ū;)), ai & C
3.1.5 Ablation Direction and Testing Circuits
Ablation typically refers to instances where we run the model on clean input and change activations to destroy the input signal (Wang et al., 2023; Conmy et al., 2023; Hanna et al., 2023; Nanda et al., 2023b). However, can also run the model on corrupt input and Resample Ablate (or Patch) in activations from the clean input (Meng et al., 2022; Heimer- sheim & Janiak, 2023). Separately, when evaluating circuits,
choose to either ablate

---

Published as a conference paper at COLM 2024
all the components of the circuit ablate all the components not in the circuit (the complement). The combination of these choices determines our faithfulness target:
Model Input
Direction
Set
Faithfulness Target
Clean
Ablate Clean
Circuit
Destroy Performance
Corrupt
Restore Clean
Circuit
Restore Performance
Clean
Ablate Clean
Complement
Maintain Performance
Corrupt
Restore Clean
Complement
Maintain Inefficacy
Figure 2 compares the second and third rows of the table, which both measure faithfulness as the similarity of the ablation to the full model. We note that Resample Ablating clean activations for the circuit components while passing a corrupt input allows the signal from the clean input to flow through edges not included in the circuit. Whereas ablating with corrupt activations on the complement of the circuit with a clean input ensures that the signal from the input only flows through the circuit.
3.2 Metric
One further consideration in addition to the ablation methodology is the metric used to evaluate the effect of the ablation. We also argue that the choice of metric is important. There are many choices used in the literature, including KL Divergence (Conmy et al., 2023), top-k accuracy Heimersheim & Janiak (2023) and task-specific benchmarks (Hanna al., 2023). In this work we will focus on the metrics used by the respective authors of the circuits that we study, but note these choices are also in general free.
Faithfulness Metrics are Sensitive to Ablation Methodology
In this section, we empirically demonstrate that evaluations of a given circuit's faithfulness are highly sensitive to the experimental choices outlined in Section 3 made at evaluation time. We further argue that this sensitivity is important, and may result in practitioners finding fundamentally different algorithms.
We provide a case study here on the Indirect Object Identification (IOI) circuit identified by Wang al. (2023), as this is the most studied language model circuit in the literature (Conmy et al., 2023; Makelov et al., 2023; Zhang & Nanda, 2024), but find similar results for other known language model circuits in Appendix E. The IOI circuit is specified an edge-level circuit, but Wang et al. (2023) evaluate its faithfulness via node-wise ablation methodology. We begin by testing the circuit using edge-level ablation.
The IOI circuit. The IOI circuit is manually-identiffed subgraph of GPT-2 that is intended to perform the IOI task, which is defined by the IOI distribution. The IOI clean distribution consists of 15 sentence templates which involve two people interacting, structured such that the next word to be predicted is the indirect object A. Each template can be filled with names in the order ABBA BABA, where the final A is the predicted token. For example: "When John The corrupt distribution
and Mary went to the store, John bought flowers for (also called the ABC distribution) fills the same templates with names in the order ABC where A, and are three different names sampled independently of the corresponding clean prompt (we only need to specify three names because we not defining a correct completion, unlike with ABBA and BABA). For example: "When Gary and Nora went to the store, Naomi bought flowers for __".
Measuring IOI Circuit Faithfulness. Wang et al. (2023) define the metric of circuit faithful- ness to be logit difference recovered?. The logit difference is computed between the correct answer and incorrect answer (the other name in the prompt) both when the full model is
Wang et al. (2023) use different metrics throughout the paper. Here we are referring to the metric used test the overall faithfulness of the circuit in Section of their paper.

---

Published as a conference paper at COLM 2024
Nades/Specific Toks
Nodes/AAl Toks
Edges/Specific Toks
Edges/All Toks
600
400
Logit Diff Recovered
-400
600
Resample
Mean (ABC)
Resample Mean (ABC)
Resample
Mean (ABC)
Resample
(ABC)
Figure 3: The IOI faithfulness metric is sensitive to (1) ablating edges/nodes (2) the type of ablation used we test Resample Ablations and Mean Ablations (over a dataset of 100 ABC prompts, which differs from Wang et al. (2023)) and (3) whether we distinguish between token positions in the circuit. The original 1OI work evaluated at specific token positions with Mean Node Ablations and obtained a logit difference recovery 87%. Other methodologies giving faithfulness scores above 100% or below 0% would have given the authors significantly less confidence about the IOI circuit, and may have led them to include different edges.
run normal and when the specified nodes are ablated. Then, the percentage of the full model's is logit difference which recovered by the ablated model is calculated.
F(x)currect F(x) incorrect M(xx)corres M(x) incorrect
100
Where F(x) correct denotes the logit of the correct answer token on F(x) (and other terms are defined similarly). A logit difference recovered of 100% means the circuit output has the same logit difference as the full model. A negative value means that the circuit outputs the corrupt logit as larger than the clean logit and value over 100% means the circuit output has a greater logit difference than the full model. We adopt this definition of faithfulness for the remainder of this section.
Wang et al. (2023) test the faithfulness of their circuit by passing in a clean input and Node Ablating the complement of the circuit. They distinguish between token positions that is, they ablate nodes in the circuit at all token positions except those specified by the circuit. They use Mean Ablation, where the mean value is computed for each token position over the ABC distribution, using around seven examples per template.
4.1 Variance Between Ablation Methodologies
We now show circuit faithfulness is sensitive to these choices. First we compare the faithful- ness metric when we change the ablation component from nodes to edges we ablate the complement of the set of edges specified by the circuit instead of the complement of the set of nodes the circuit. As shown in Figure's 3, ablating the edge level returns substantially higher percentages.
Figure also evaluates the effect ablation value. We rerun the above experiment using Resample Ablations from the ABC distribution, and find that this results in a systematically lower faithfulness as compared with mean ablations (statically sigmificant on a t-test with study the effect of ablating
p le 5 for Node Ablation but not Edge Ablation). Finally, at every token position, instead of only those specified by the circuit. This consistently results in lower faithfulness scores. It is concerning that the edge-level circuit with specific
7

---

Published as a conference paper at COLM 2024
[Average Logit Diff] %
[Logit Diff
ABBA Predictions
BABA Predictions
Logit Diffrence Recovered
130
130
ABBA
120
BABA
110
110
300
300
100
Porfect ratorunes
100
200
200
100
100
70
70
1000
5
10
2
100 2 3
10
:
-200
5
200
ABC Dataset Size
Figure 4: (Left) The 1O1 circuit is sensitive to the size of ABC dataset used for mean ablation. The logit difference recovered consistently higher for prompts of the BABA format. (Left and Middle Left) The order of computing the average and percentage affects the faithfulness metric. Wang et al. (2023) use [Average Logit Diff] %, giving lower scores than Average [Logit Diff %]. (Middle Right and Right) There is large range of logit difference recovered, the boxplots show the interquartile range. According to this faithfulness measurement methodology, The IOI circuit implements the IOI task faithfully on average, but not for many single data points.
token positions has a median score well over 100%, as this best represents the hypothesis of Wang al. (2023).
Next, we discuss sensitivity of the faithfulness metric to both the clean distribution and intricacies of the metric calculation. For these experiments, we perform node-level Mean Ablations on the complement of the circuit, split by token position, similarly to Wang et al. (2023). As shown in the left two charts of Figure 4, faithfulness systematically greater for the prompts of form BABA than prompts of form ABBA. We also find that faithfulness monotonically increases with the size of the ABC dataset (used for computing the Mean Ablation).
Finally we note that Wang et al. (2023) compute the logit difference recovered by first finding the mean logit difference for the full model and the ablated model over all prompts, and then computing the percentage (Figure 4, far left).
[[ cxfrecttrtctt correct F(x) incorrect] [ correct M(x) incorrect
100
instead we compute the percent difference for each prompt and then take the mean, we return substantially higher percentages (Figure 4, middle left).
F(x) correct F(x) incorrect
x 100
M(x) ((x))correttt M(x) incorrect
These are significant and important changes in evaluation. If the researchers had used different methodology, they may have discovered different circuit and, therefore, different underlying algorithm. This is important since it suggests that the algorithm the circuit required perform depends on the ablation methodology. We expand on this point in Section 5.
4.2 Variance Between Individual Datapoints
Even for fixed ablation methodology and metric, there is significant variation in the measured faithfulness between individual prompts in the distribution.
We show this for the IOI circuit in the figures above, with results for other circuits in Appendix E. The graphs on the right of Figure show large range of faithfulness scores attained when ablate the complement of the nodes in the IOI circuit. Note that the
8

---

Published as a conference paper at COLM 2024
graphs do not show the full range of datapoints and there are several extreme outliers with a logit difference recovered in the tens of thousands of percent. The inter-quartile range (IQR) is also large, stretching up to 50% across the dataset. This is concerning: while the circuit matches the behavior on average, it does not match it for many examples. Another property of ideal circuits describing behaviour on some task is that their faithfulness variance should be low over the task input distribution. Otherwise, the circuit is at least partially optimized to balance out extremely high (significantly 100%) and extremely low faithfulness scores (<0%). This variance consideration is importantly missing from the mechanistic explanations of how GPT-2 implements the IOI task provided by Wang al. (2023). We encourage MI researchers to evaluate task performance in both the average case and worst case.
5 Optimal Circuits Are Defined By Prompts and Ablation Methodology
We showed in the previous section that measurement details can greatly change the faith- fulness score of an experiment. However, one might ask if this difference matters. In this section we discuss the consequences of such sensitivity for circuit discovery.
If a circuit is specified as a set of edges, it should be tested using edge ablations and if it is specified with token positions then it should be tested with token-specific ablation. But in other aspects there often isn't a clearly correct methodology. So how should we think about the difference in faithfulness between different methodologies? We study this question in small toy models, where we have access to the 'ground truth' circuit. We conclude that the optimal circuit for some distribution cannot be defined unless we also specify the ablation methodology and metric that we are using to measure it.
Tracr models (Lindner al., 2023) are tiny transformers that are compiled instead of trained. Since the ground truth algorithm is both simple and known, they provide an excellent setup for testing circuit discovery algorithms. RASP programs (Rush & Weiss, 2023) are compiled into the weights of a transformer that implements the program exactly. Following Conmy et al. (2023), we study two Tracr models, Reverse and X-Proportion.
The X-Proportion model performs the task of outputting at each token position the propor- tion of previous characters that are 'x's. The model has two layers, with one head in each attention layer. The first attention layer and the second MLP are not used, so we need only consider the edges between the Input, MLP 0, Attn 1.0 and Output.
Conmy et al. consider the edge from Input to Attn 1.0t to be part of the ground truth circuit (Figure 11). Inspecting the RASP program, we see that the only information in this edge's activation that is used by the model is the positional encoding of the tokens. However, this does not vary between different inputs, so if our ablation methodology uses Resample Ablations then this edge need not be included in the circuit, ablating it will not change this positional information. However, if we instead use Zero Ablations, then this information will be destroyed, so the edge must be included in the circuit.
Conmy et al. test three automatic circuit discovery algorithms on this task. All three al- gorithms use (or approximate) Resample Ablations discover circuits. The first method, ACDC, traverses the model in reverse topological order, ablating each edge in turn. Subnet- work Probing (SP) learns a mask parameter for each node, via gradient descent, attempting to maximize the number of nodes ablated, while minimizing the KL divergence from the original model. Lastly, Head Importance Scoring (HISP), uses a first order, gradient-based approximation of Node Ablation to assign attribution scores to each node. We test each circuit discovery method by sweeping over a range of importance thresholds to obtain an ordering of circuits of increasing size. Following Conmy et al. we then plot pessimistic receiver operating characteristic (ROC) curves (Figure 5) and compare the area under curves.
SP and HISP, use (or approximate) Node Ablations, while ACDC uses Edge Ablations. IIn our experiments we adjust the implementation of both SP and HISP to use (or approximate)
TTo convert the predictions of SP and HISP to edge-based circuits, Conmy et al. include all edges which connect two nodes of sufficient importance. With this implementation it may be impossible for
9

---

Published as a conference paper at COLM 2024
Reverse (Theirs)
X-Proportion (Theirs)
Reverse (Ours)
X-Proportion (Ours)
True Positive Rate
ACDC
SP
HISP
0.6
0.4
0.5
0.5
1
0.5
False Positive Rate
Figure 5: ROC Curves measuring the overlap between automatically discovered circuits and the two different "ground truth" circuits, for two Tracr tasks. When we match the ablation methodology of the ground truth with the ablation methodology of the circuit discovery algorithms, we can achieve perfect circuit recovery with all three methods.
Edge Ablations; SP learns mask parameters that ablate each edge and HISP assigns attri- bution scores for each edge by approximating Edge Patching. We provide comparison between Edge and Node-based circuit discovery methods Appendix F.
Conmy et al. considered the edges that would be required with Zero Ablations to be the correct circuits. Therefore, the algorithms fail to fully recover the "ground truth". When instead consider the edges that required with Resample Ablations to be the correct circuit, all three algorithms perfectly recover the "ground truth" (Figure 5).
This case study illustrates that the optimal circuit with respect to only set of prompts is undefined. The ablation partly determines the task. In this case, we must decide is determining the positional encoding part of the task? so then the zero ablation circuit should be considered the ground-truthh if not then the resample ablation circuit should be.
6 Conclusion
In this work we show existing transformer circuit evaluations are highly sensitive small changes in the ablation methodology and the metrics used to quantify faithfulness. We further show that the optimality of circuit cannot be defined with respect to of prompts without precise evaluation methodology
circuit is specified as set of edges, should be tested using edge ablations. And if it is specified at a chosen set of token positions it should be tested with these. But in other aspects there often isn't clearly correct methodology. Do you want your IOI circuit to include the mechanism that decides it needs to output name? Then use zero ablations. Or do you want to find the circuit that, given the context outputting a name, completes the 1Ol task? Then use mean ablations. The task cannot be separated from the ablation methodology.
Our work has significant consequences for circuit discovery work, particularly automated circuit discovery algorithms that aim to optimize these faithfulness scores. It suggests that assessing the quality of automated methods by measuring the overlap with some 'ground truth' can be misleading, the ground truth was discovered using a different ablation methodology.
We recommend that researchers precisely describe their experimental procedure when reporting evaluations of circuits. They should consider which task exactly they are expecting their circuit to perform.
SP and HISP to correctly order edges. For example, there can be two nodes which are both individually important, but where the edge connecting them is unimportant.
10

---

http://cccccuuttrreeeeeeeeeeeeee Published as a conference paper at COLM 2024
7 Acknowledgments
Thanks to Arthur Conmy for his generous assistance in understanding and reproducing his work on Automatic Circuit Discovery and his insightful comments. Thanks to Adam Gleave, Lawrence Chan, Clement Neo, Alex Cloud, David Bau, Steven Bills, Sam Marks, Adrià Garriga-Alonae Stefan Heimersheim, Nix Goldowsky-Dill and our anonymous reviewers at COLM 2024 for their invaluable feedback and suggestions. Thanks to Bryce Woodworth for his help and encouragement.
References
Chirag Agarwal, Sree Harsha Tanneru, and Himabindu Lakkaraju. Faithfulness vs. plau- sibility: On the (un)reliability of explanations from large language models, 2024. URL
http://axxiv.oorraa//////// 04614.
Afra Alishahi, Grzegorz Chrupala, and Tal Linzen. Analyzing and interpreting neural net- works for nlp: A report on the first blackboxnIp workshop. Natural Language Engineering, 25(4):543 2019. doi:
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can ex- plain neurons in language models. https: https:// httss////paappubiiccccm blob. core. windows. net/ neuron-explainer/paaaee wwtpe///dpppppee....///aa iit//iiddxxxx// html, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield- Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: De- composing language models with dictionary learning. Transformer Circuits Thread, 2023. httpp:// http://tannfomm--rrrritee pub/ 2023 monosemantic-fetture /index.ttll.....
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 2021. doi: 10.23915//dsstll.000222 htttps//ddstill.pppt
Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas. Causal scrubbing: method for rigorously testing interpretability hypotheses [redwood research], 2022. URL https://wwwt wwwp:wwwwwwwg.......oo...oo.oo.. causal-scrubbing- -atal---rrbbinn----et---------oo fortood-orr-
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations, 2023.
Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adriá Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam
11

---

Published as a conference paper at COLM 2024
McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Trans- former Circuits Thread, 2021. URL index. html.
hutp::rrrccccuuuuuu.upue/aeeee h022/ffaawwwrk//
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas F. Icard, and Noah D. Good- man. Finding alignments between interpretable causal variables and distributed neural representations. ArXiv, abs/2303.02536, 2023. URL https://api. semanticscholar.oom CorpusID:257365433.
Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato, and Aryaman Arora. Localizing model behavior with path patching, 2023.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Suro,, 51)5), aug 2018. ISSN 0360-0300. doi: 10.1145/3236009, URL https: //doi.org/1e 1145/3236009.
Wes Gurnee and Max Tegmark. Language models represent space and time, 2024.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?: Interpreting mathematical abilities in a pre-trained language model, 2023.
Stefan
Heimersheim and Jett Janiak.
circuit for Python transformer, 2023. org/posts/ //tosss/u@xXnnnEbbFFoooooo /uKKKKkkbbXFWZOAOk
4-layer attention-only wwwggeentoooummmmmmmmmm.m.....mm https://wwwaliggmeetffruuu oec in
docstrings in URL
https://wwe al
Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Lingwisfics, pp. 4198-4205, Online, July 2020. Association for Computational Linguistics. doi: 10.18653 vv /2020.000a0 acl-main.386. URL https: /aclattoooo...ooogooooooo.ooo org/ malimain. 386.
Janos Kramar, Tom Lieberum, Rohin Shah, and Neel Nanda. Atp": An efficient and scalable method for localizing I1m behaviour to components, 2024.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task, 2023.
Maximilian Li, Xander Davies, and Max Nadeau. Circuit breaking: Removing model behaviors with targeted ablation, 2024.
David Lindner, János Kramar, Matthew Rahtz, Thomas McGrath, and Vladimir Miku- lik. Tracr: Compiled transformers as a laboratory for interpretability. arXiv preprint arXiv:2301.05022, 2023.
Aleksandar Makelov, Georg Lange, and Neel Nanda. Is this the subspace you are looking for? an interpretability illusion for subspace activation patching, 2023.
Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Computing Research Repository, arXiv:2403. 19647, 2024. URL https: ffaaxiv.org//a wabs///43... 19647.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36, 2022.
Neel Nanda and Joseph Bloom. Transformerlens.
https://githhb: com/
2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023a.
12

---

Published as a conference paper at COLM 2024
Neel Nanda, Senthooran Rajamanoharan, Janos Kramar, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level, Dec 2023b. URL
pact-finding-attemttnn-------- otw.ep/niiintooeeerreeeeennneene
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/ distill.00024.001.
https://distill.pub///0 http:/pbbbb////ciius//o/////// pnccuitszzoom-in
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https:/ http://tttnooomeeriiitttsstiptp aannr---earrmiin-a-dnnnnuitiiioo sirioo---eadddddeee.tmtlm) index.html.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024.
Alexander Rush and Gail Weiss. Thinking like transformers. In ICLR Blogposts 2023, https:/ /iclr-
2023. URL https: blcp::////trrbboooosss wtt.gppstss.gtuu.ioo/oooo wttthbb.o/2222///gg//////////// prsppostsgithuubboo/002/3
2023 2023/raspy/ raspy/.
Lee Sharkey. Sparsify: A mechanistic interpretability research agenda, 2024.
URL https://wwe/ http:://www.l.immeetttoo..o www.rrum.oooo.o/gooeeeee /rgg/psss/////i//aayryyyy Accessed: 2024-06-
sparsify-as aparsify---mechanissse serchamisti--nnneerretaalllieii reseaaihhh
28.
Aaquib Syed and Can Rager. Attribution patching outperforms automated circuit discovery, 2023.
John Thickstun. The transformer model in equations, 2024. URL URL.ttps://jhhthickksu com//ocs/transfoomeee pdf.. Accessed: 2024-06-28.
Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda. Linear representa- tions of sentiment in large language models, 2023.
Alexander Matt Turner, Lisa Thiergart, David Udell, Gavin Leech, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization, 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In Advances in neural information processing systems, volume 33, pp. 12388-12401, 2020.
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https:
//openreview. net
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods, 2024.
13

---

Published as a conference paper at COLM 2024
A AutoCircuit Library
We release AutoCircuit, Python library with a highly efficient implementation of Edge Patching and various circuit discovery algorithms, with support for TransformerLens mod- els (Nanda & Bloom, 2022). It supports Mean, Zero and Resample Ablations. See our blog post for more detail on our fast implementation.
We test the performance of our implementation by running the ACDC (Conmy et al., 2023) circuit discovery algorithm, which iteratively patches every edge in the model. We compare the performance of AutoCircuit's implementation to the official ACDC implementation (which is currently the most popular library for patching large numbers of activations). We run ACDC using both libraries at range of thresholds for tiny 2-layer model with only 0.5 million parameters and measure the time taken to execute on a single GPU.
100
AutoCiraut (ours)
ACDC
Execution Time (seconds)
10
0.2
0.4
0.8
Proportion of Model Edges Included in Circuit
Figure 6: Time to execute the ACDC algorithm over the entire network for small layer transformer. AutoCircuit (ours) if significantly faster than the official Automatic Circuit Discovery codebase.
Different numbers of edges are included at different thresholds in the ACDC algorithm. Note that ACDC and AutoCircuit count the number of edges differently (AutoCircuit doesn't include 'Drrect Computation' or 'Placeholder' edges) so we compare the proportion of edges included (the underlying computation graphs are equivalent). Figure 6 shows that our implementation is significantly faster and the number of edges included greatly affects the performance of the official ACDC implementation, but it doesn't 't change the performance of our implementation.
14

---

Published as a conference paper at COLM 2024
B Further Details on Ablation Methodology
Attn O
Altn
Atn 1
MLP
Input
MLP
Cutput
Inout
Output
Atn
(a) The canonical formulation of a transformer. Every component reads input from the resid- ual stream backbone.
(b) The "factorized" formulation of trans- former views every component as taking in- put from every previous component.
Figure 7: Two equivalent formulations of the transformer architecture. We illustrate only one layer, but this extends trivially to many layers.
Inpur
Ioput
houut
Att
Takenè
MIP
QUEpora
Arte
Imput
(a) The "treeified" formulation of trans- former separates every path from input out- put.
(b) We can consider each token position to have a separate set of edges.
Figure 8: "reefi""""" transformers suggest another type of ablation. (Right) Distin- guishing between different token positions in Edge Patching.
15

---

Published as a conference paper at COLM 2024
C
Summary of Tasks Studied
Name
Model
Example Clean Prompt
Example Cocrupt
Correct Answer
Incorrect Answer
Melrie
Faithfulness
Tracr
X-Propottion
Traer
X-Propottion
0,0.5,0333,
0.5,04
0A000,0222
Mean squared
erner
Tracr
Reverse
Traer
Reverse
1,222222
1,40122
2,2,2,11
2,00A,1
KL. Divengenoe
Indinett
Then, Scott and Jeremy went to the hospilal. jeremy gave snack to
Then, Michael and Anderson went to the hospilal. Rachel snack to
Logil Difievence Recovered
Objecl
GPT-2
"coot"
"eremyy"
Mentification
def errorfoell, create, option, file, ron, client, project):
del erroriself, create, option, output, host, label, projectl: annd employment camp
"izee, oulipat; "host" label", ffrst", text", request", "user", "ile", run". creale", "pption, project"
Layer Attention Only
land employment camp
Corret Prediction Propartion
Docstring
client"
iparam file: protein author
iparam first protein author sparam text: forest degrce
sparam run: forest degree
Sports Players
Pythia 2.88
Fact: Tiger Woods plays the
gfact Phil
Fact: Tiger Woods plays the
of Faccn Babe Ruth
"ootball"
baskethall"" baseball"
Top Sport Logit
Simms plays the sport
sport of
Table 2: The tasks we study, which previous works have found circuits for, and the metrics used by previous works to measure their faithfulness.
16

---

Published as a conference paper at COLM 2024
D Summary of Patching Methodologies Used By Previous Works
Work
Granularity
Component
Value
Token positions
Direction
Set
(Gender Bine)
st (2020)
Heads, Neurons
Node
Resample (clean)
telerns
Resample Cleon
Circuit
Meng et al. (2022)
(ROME)
Layer
Node
Resample (clean)
Specific tnkens
Resample Clea
Circuit
et (2003)
(D1)
Heads
Node (evaluation)
Path (discovery)
Mean
Specific tnkens
Ablate Clean
Complement
Cormy et al. (2023)
(ACDC)
Heads, MLP'
Edde
Resample (cotrups)
tokeens
Abbate Clean
Complement
Hemmensheim & Janiak (2023)
(Deestring)
Heads
Node
Resample (clean)
Spectiic tokens (ciscovery)
tokens (evaluation/
Resample Clean
Circuit
Hanna al. (2023)
(Gesterthan))
Heads, MLPs
Path
Resample (corrups)
tokens
Ablale Clean
Complement
Osparis Playes)
Nanda (2023b)
Heads, MLP's
Path
Resample (cotrupt)
Specific tnkens
Ablate Clean
Complement
Table 3: Summary of the patching methodologies used by seven previous works. Note that each methodology differs from all of the others in at least one aspect.
17

---

Published as a conference paper at COLM 2024
E Further Study of Faithfulness Metrics
In this section, we provide further analysis demonstrating faithfulness metrics are brittle, on two other circuits from the existing literature.
E.1 Docstring
The Docstring Task. The Docstring task (Heimersheim & Janiak, 2023) is simple task that tests a 4 layer, attention-only model's ability to complete specific part of standard Python docstring (see Table for an example). All prompts follow very similar format, with the only difference being the names of the variables in the function. The corrupt distribution follows the exact same format, using a disjoint set of variable names.
Measuring Docstring Circuit Faithfulness. Heimersheim & Janiak (2023) test their circuit using a similar methodology to the one which Wang et al. (2023) used to test the TOI circuit. They ablate all nodes in the complement of their circuit. However, unlike Wang et al. (2023) they use a Resample Ablation (also known in this context as Activation Patching), and they do not distinguish different token positions. The metric that they use for faithfulness is the percent of highest logit outputs that are the correct answer over some set of prompts.
80
Resampple
Takemwibe Hetn Ceaan And Comppt
And Comupt
70
Correct, Answer (%)
Full Model
0.8
Answer Probability
0.6
20
0.4
10
0.2
Nodes [Reported]
Edgess
Modes (takers)
Exiges (tokens)
Nades
Edigrta
Edges (tolenn)) Nodes ftokens)
Nodes
Edges
Nostes (talant)
Edipes (takans)
Full Mostel
Nodes
Edges
Nodes (rakens)
Edites (takents
Full Model
(a) The faithfulness of the Docstring circuit according to correct answer percent metric used by Heimersheim et al. Heimersheim & Janiak (2023) is sensitive to the type of ablation used measure the circuit.
(b) The faithfulness of the Docstring circuit as measured by the probability of the correct answer is highly variable between individual prompts.
Figure 9; Faithfulness metrics for the Docstring circuit when ablating every node or edge not in the circuit, all token positions and at token positions specified by Heimersheim & Janiak (2023).
In Figure 9, we test the faithfulness of the Docstring circuit with various ablation method- ologies. We compare: (1) distinguishing between different token positions (Heimersheim & Janiak specify their circuit with token positions, even though they do not use this infor- mation their faithfulness evaluations), (2) ablating the edge-level and node-level (they also specify edges, even though they evaluate only with nodes), (3) ablating with Resample and Mean Ablations and (4) two different faithfulness metrics: correct answer percentage and answer probability.
We measure various significant changes in faithfulness in response to these adjustments. Most importantly, Edge Ablations perform significantly better using Mean Ablation instead of a Resample Ablation. Had Heimersheim & Janiak (2023) performed edge-level Resample Ablations instead of nise-wise Resample Ablations, they may have trusted their circuit significantly less (and they had used edge-level Mean Ablations, they may have trusted it more).
18

---

Published as a conference paper at COLM 2024
Distinguishing by token position also had large effect on faithfulness scores for both node- wise and edge-wise ablations. These low scores suggest the circuit is in fact performing significant computation token positions outside of the circuit specified by Heimersheim & &Janiak (2023).
When measure the probability of the correct answer find that, similar to IOI, the variance between individual prompts is high. This is important for reasons outlined in Section 4.
E.2
Sports Players
Resample
Mean
Resample
Mean
80
80
Accuracy
0.8
0.8
Probability
40
40
0.4
0.4
20
20
0.2
0.2
Ablated Mode)
Circult
Full Model
Ablated Model
Circuit
Full Model
Ablated Model
Circuit
Full Model
Ablated Model
Circuit
Full Model
(a) The percentage of prompts for which the correct sport has the highest output logit with Mean and Resample Ablations.
(b) The output probability of the correct sport with Mean and Resample Ablations.
Figure 10: The faithfulness of the Sports Players circuit is reduced when using Resample Ablations.
The Sports Players Task. The Sports Players task (Nanda al., 2023b) is simple task that tests the Pythia-2.8b model's (Biderman et al., 2023) ability to recall the sports of famous football, baseball and basketball players. See Table 2 for an example. All prompts follow very similar format, with the only difference being the name of the sports player in question. The corrupt distribution follows the exact same format, with each clean/ aaan/ct pair having two players of different sports.
Measuring Sports Players Circuit Faithfulness. In Figure 10, we test the faithfulness of the edge-level sports players circuit, distinguishing token positions while (1) ablating the complement with both Resample and Mean Ablations and (2) calculating two different faithfulness metrics: correct answer percentage (considering only the three possible sports, following Nanda et (2023b)) and answer probability.
We find a dramatic difference in correct answer percentage between Resample and Mean Ablation. This case is little different because the authors' aim wasn't to find the full circuit but to identify the place in the model where factual recall occurs, this result doesn't t negate their hypothesis.
Note that random guessing would achieve 33% accuracy there are 3 possible sports, and this is roughly what we see when Mean Ablating the whole model. But Resample Ablating adds signal from the corrupt prompt, which is always different sport, explaining the 0% accuracy score for the Ablated Model and the Circuit.
19

---

Published as a conference paper at COLM 2024
E.3
Further Detail the X-Proportion Tracr Ground Truth Circuits
Attn 1.0
Attn 1.0
Input
MLP O
Output
Input
MLP
Dutput
(theirs) The "ground truth" circuit for the Tracr X-Proportion task using Zero Ablations.
(ours) The "ground truth" circuit for the Tracr X-Proportion task using Resample Ablations.
Figure 11: For the Tracr X-Proportion circuit, the edge from Input to Attn 1.0 is only used to transfer the positional encoding, it is not required when using Resample Ablations, since these preserve information that is constant between the clean and corrupt distribution. This illustrates the principle that optimal circuits cannot be defined without an ablation methodology. (Nodes Attn 0. 0 and MLP 1 are not shown as they are not used in this model.)
20

---

Published as a conference paper at COLM 2024
F
Edge-Based vs. Node-Based Circuit Discovery Methods
Reverse
X- X-Proportion
0.8
Node Search
0.6
0.6
0.4
True Positive Rate
0.2
0.2
0.5
0.8
Edge Search
0.6
0.6
0.2
0.2
0.5
False Positive Rate
ACDC
SP
HISP
Figure 12: ROC Curves for Edge- Based and Node-Based circuit discovery methods, using the Resample Ablation edges as the ground truth (ours).
In Section 5, we adapted the Subnetwork Probing (SP) and Head Importance Scoring (HISP) circuit discovery methods to use (or approximate) Edge Ablation. ACDC (Conmy et al., 2023) already uses Edge Ablations, but we can similarly adapt ACDC use Node Ablations. We compare the performance of the Node Patching versions of ACDC, SP and HISP to the Edge Patching versions, for the Resample Ablation based "ground truth" circuit introduced in Section 5 (Figure F).
21

---

Published as a conference paper at COLM 2024
G Clarifying Nomenclature
Some authors have used different terms for some of the concepts introduced in Section 3. For instance, Activation patching has previously also been called Causal Tracing or Interchange Intervention. In the remainder of this section, we summarise how our nomenclature relates to the terminology used by Redwood Research in their series of early mechanistic interpretability transformer-circuits papers. Chronologically, these are Wang et al. (2023); Chan et al. (2022); Goldowsky-Dill al. (2023).
We first discuss the final, most comprchensive work (Chan et al., 2022), which we refer to as Causal Scrubbing. Causal Scrubbing is a very general approach for evaluating circuits together with explanations of the role of nodes within the circuit. It generically comprises performing specific branch-based Resample Ablations on the treeified model on both the circuit and its complement. Causal Scrubbing randomly replaces activations with those that your hypothesis predicts will not change the model output. For instance, if we claim that given node detects whether the input is even, Causal Scrubbing could patch in activation from a different even input, and expects the output not to change. In general, Causal Scrubbing permits an arbitrary number of possible counterfactual inputs.
Goldowsky-Di et al. (2023) simplify this setup, dropping the strict requirement of requiring an explanation for each node. This reduces the hypothesis class to the now standard circuit discovery problem; does some path matter for task performance or not?
Finally Wang et al. (2023) perform a further simplified version of path patching to discover the 10I circuit. This is equivalent to Edge Resample Ablation in our terminology but which they call Path Patching. They patch paths one time, to establish which edges are important for task performance. Importantly, Wang et al. (2023) reason that the IOI task should be an attention-only task, as only comprises moving information between tokens. As such, they take nodes to only be attention heads, with MLPs considered to be part of the direct path between nodes. This approach of one-hop path patching extended and automated by Conmy et al. (2023).
22


Reviews:
[{"review_id": "QJTxcG3A3J", "reviewer": "Reviewer_W9hJ", "summary": "This paper studies how circuit faithfulness evaluations change depending on the exact methodology used for calculating the metric. The paper empirically compares the effect of different methodological choices on a variety of circuits that have been identified in prior work. The main result is that the faithfulness scores can change a lot depending on these choices.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\nWe defer to the definitions of faithfulness used by the authors that discovered the circuits we study. These papers consider faithfulness to be the similarity of the explanation (in this case circuit) to the full model on the task - this is “fidelity” in the language of [1] - but they use a variety of metrics for their exact definition. We will include further discussion of this, and put this in the context of other definitions of faithfulness in the NLP literature in our revised paper.\n\nWe thought that diagrams would provide the clearest explanation for our discussions of ablation methodology, but we will also include mathematical descriptions for further clarity (using do-calculus notation). We will add the formula for logit difference recovered and appreciate that this may be easier to understand than our verbal description. A table summarizing prior methodological choices is a good idea, thank you for the suggestion.\n\nWe agree that Zhang and Nanda (2024) is highly relevant and we do cite it. They perform experiments studying different aspects of ablation methodology. Importantly, they do not arrive at our notion that circuit faithfulness can only be defined by specifying the circuit evaluation and ablation method. We agree these differences should be enumerated in the paper.\n\nOn the question of our initial assumptions, we wish to clarify that we are not arguing that faithfulness metrics “should” be robust to methodological choices, we are merely noting that they are not, because prior work implicitly assumed that optimal circuits could be defined by tasks only. We agree that some of the wording choices are too strong and that these can be made more precise. In particular, we like the suggestion of recommending that researchers describe their choices clearly, and use more rigorous and precise language to clarify the exact claims they support. We agree that “sensitive” is a better description than “not robust”, and will adjust wording where possible.\n\nWe have benchmarked our library against the most popular circuit discovery library [2] and found it to be much faster. We will include this graph in the final paper.\n\nDifferentiating by token position means considering a prompt of n-tokens to have n parallel computational graphs, that interact only through the attention mechanism. The graphs can be ablated independently.\n\n[1] https://arxiv.org/abs/2004.03685\n[2] https://github.com/ArthurConmy/Automatic-Circuit-Discovery", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper addresses a timely topic, evaluating mechanistic model explanations. Existing work in this area uses a number of subtly different evaluation methodologies, so it is a useful contribution to provide a unifying survey of these different choices and to try to explain their effect.\n- The experiments cover a good variety of circuits, including several manually-identified circuits from prior work and circuits discovered using more recent automated methods.\n- The authors release a code library: this could be a good contribution, to unify various methods in one place, and also provide more efficient implementations compared to available libraries. (However, I would also suggest that the authors include a relevant benchmark or any other details about the computing resources used.)\n- A number of the specific findings are interesting and could be relevant to practitioners. These include the differences between ablation types and the effect of the dataset size used for calculating mean ablations. I also appreciated the discussion in section 5 about Tracr models, which illustrates the implication of a specific design choice (mean ablation at the token level still preserve position information).", "reasons_to_reject": "- I believe the paper is missing a formal definition of faithfulness, and a discussion of prior work that has attempted to define faithfulness in the context of explanations for NLP models. In particular, I think the paper should discuss [1, 2, 3]. Having a clear definition is important for helping us reason about which design choices are appropriate.\n- Similarly, I think the paper would benefit from a more formal or mathematical description of the overall methodology used for calculating these faithfulness metrics. Without this, section 3 is difficult to understand, especially for people who aren't familiar with these methods, and it's difficult to reason about the effect of different design choices. (I was especially confused by section 3.1.4, \"Token Position\".) As a suggestion, most of these approaches are based on causal mediation analysis [5, 6], and I think this kind of presentation would be helpful here. For some recent references that use this framework for formalizing circuit faithfulness, see [7, 8].\n- The paper is also missing a formal, mathematical definition of the \"logit difference recovered\" metric. (The exact formulation of this metric  turns out to be important in section 4.1 so it should be defined precisely.)\n- The paper doesn't always provide citations for the methodology choices (for example, in sections 3.1.3-3.2). More generally, I think the paper would benefit from a more systematic review of the specific methods used in prior work--for example, in the form of a table like Table 1.\n- Some of the findings in this paper seem to overlap with [4], who also study the effect of different ablation methods. I think this paper still makes a number of new contributions, but I feel it should discuss the differences between [4] at more length.\n- At a very high level, I have some doubts about the the key assumption in this work, which is that faithfulness metrics should be robust to methodological choices like ablation method. It seems to me that different methodological choices reflect different assumptions and measure different things (for example, edge level ablations vs. node level ablations correspond to different subgraphs--we wouldn't expect them to be the same). I think many of the points in the paper are still salient, but should be stated more moderately. For example, instead of arguing that existing metrics are \"arbitrary,\" inconclusive, and misleading, the paper could make the point that these methodological choices actually aren't \"slight\", and researchers should describe their choices clearly, and use more rigorous and precise language to describe the exact claims that they support. (Along the same lines, I have some doubts about the title. \"Robust\" commonly means robust to noise. In this case, it's debatable whether the ablation methodology is really noise. I would prefer a more specific phrase, e.g. \"sensitive to ablation methodology\".)\n\n\n*Summary:* I think this paper makes a number of useful findings on an important topic, but the paper is missing a discussion of some foundational prior work (especially [1]), and I feel there are issues with the clarity, presentation, and framing. I would be open to increasing my score if the authors could address some of these concerns.\n\n**Update after rebuttal:** The authors have proposed several changes to address my main concerns so I am increasing my score accordingly.\n\n_References_\n\n[1] Jacovi and Goldberg, 2020. Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?\n\n[2] Lyu et al., 2024. Towards Faithful Model Explanation in NLP: A Survey.\n\n[3] Geiger et al., 2023. Causal Abstraction for Faithful Model Interpretation.\n\n[4] Zhang and Nanda, 2024. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods.\n\n[5] Vig et al., 2020. Investigating Gender Bias in Language Models Using Causal Mediation Analysis.\n\n[6] Finlayson et al., 2021. Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models.\n\n[7] Marks et al., 2024. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models.\n\n[8] Kramár et al., 2024: An efficient and scalable method for localizing LLM behaviour to components", "questions_to_authors": "- Could you offer any comments on whether there are any principled reasons to prefer one design choice over another?\n- Section 3.1.3 describes \"Resample Ablate\" and \"Mean Ablate\" as \"Two more principled approaches\" -- In what sense are these two approaches more principled?\n- Can you offer any more details about the meaning of the \"Token Positions\" methodology in section 3.1.4? Similarly, in appendix C.1 (docstring task)--what exactly does it mean to differentiate by token position?\n- In section 4.1, can you provide more details about how exactly the ABC dataset is used? Is it just used to calculate the the mean activations which are used as the ablation value?", "extra_scores": {}}, {"review_id": "JPXyNABscl", "reviewer": "Reviewer_qNxz", "summary": "The paper addresses the problem of post-hoc interpretability in transformer language models. They analyze the faithfull scores of circuits obtained by identifying by using different methods and show that results are highly sensitive to the particular metric used making this approach quite unstable. In particular they show that the circuit faithfullnes highly depends on the ablation methodology used to evaluate them. This insight is very relevant given the importance of give reliable explanation for black box models.\n\nStrengths of the paper :\n\nuseful insight for obtaining recommendation in applying MI;\navailability of a library containing efficient implementations of the circuit-discovery and circuit-evaluation techniques used in this paper\nWeaknes of the paper The paper is not clearly written, the english is slippery sometimes and the arguments are not well explained (see list of comments below. Abstract and introduction creates great expectation that are somehow not met in the subsequent sections and in particular by the results. The recommendations for researchers end up to be quite shallow as the conclusions.\n\nList of Comments\n\nlast line of page 1 \"paradigm of that aims\" -> \"paradigm that aims\" Related work section \"Circuit analysis a form\" -> \"Circuit analysis is a form\" \"learned algorithm responsible\" -> \"learned model responsible\" I think model is more suitable and anyway the whole sentence is unclear. Circuit analysis a form of post-hoc interpretability focused on understanding \"A circuit is some subgraph\" -> \"A circuit is a subgraph\" \"a large amount of researcher-effort to discover.\" the sentence seems to be not completed. \"Ideal circuits exist on the Pareto frontier of faithfulness, completeness and [as the whole network is already trivially optimal for the first two] simplicity\" what does it means ? \"Besides circuit analysis, MI work more broadly seeks to understand...\" please correct and clarify the sentence \"MI has also inspired work in steering model outputs\" please explain \"There are many choices MI researchers may choose between..\" choices and choose in the same sentence is sound bad, please us a different word.\n\n3.1 Ablation Methodology The la st paragraph at pag 2 intrroducing the concept of ablation is a bit confusing becaous it is not clear if you arre considereing ablation or patching, please be clear! \"what type of component in the graph ablated\" a verb is missing ? Define MLP Define Q, K and V inputs \"Circuits in autoregressive transformers on a narrow distribution are sometimes defined in terms of edges which each act at a given token position.\" not clear\n\n4 Faithfulness Metrics are Sensitive to Ablation Methodology \"The IOI circuit is specified as an edge-level circuit, but Wang et al. (2023) evaluate its faithfulness via a node-wise ablation methodology\" could have it be done differently or better ? explain (or is just a problem with the english ? ) When you explain the IOI circuit you can have template BABA or ABBA, then the corrupt (corrupted) distribution fills the same templates with ABC : why only three ? because the last A is fixed ? names in the order ABC\n\n4.1 Variance Between Ablation Methodologies You say that yoou observe a systematically lower faithfulness when use one method instead of another : this is not strange and does not mean that the two methods are inconsistent if the relative order is maintained", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback. We apologize for the list of typos and will ensure to correct them.\n\nTo clarify the point on the Pareto frontier of faithfulness, completeness and simplicity, we are pointing out that if you consider the neural network itself as an explanation or interpretation for the neural network, this is actually perfectly faithful and complete. However it is not a useful description because it is too complex for humans to understand. We argue that there exist many optimal circuits, which make different tradeoffs between faithfulness, completeness and simplicity.\n\nAs stated in Section 3.1 “we consider activation patching to be a specific type of ablation, and use the term Resample Ablation interchangeably.” By MLP (multi layer perceptron) and QKV (attention head queries, keys and values) we refer to internal model activations of the transformer based architectures we consider. We will include a mathematical description of the transformer architecture.\n\nWe believe that Wang et al. would have done better to evaluate their IOI circuit at an edge level, because this is the level at which their circuit is specified. We perform this experiment and compare the result to the node-level circuit, which is what they measure.\n\nIn the IOI prompt format (e.g. “When A and B went to the shops, A bought milk for B” (ABAB)), the names are arbitrary, so BABA is the same as ABAB and ABBA is equal to BAAB. Our choice of prompt formats follows prior work [1]. We will clarify that we are not arbitrarily omitting an ordering of names.\n\nWe do not think it is very strange that we observe lower faithfulness scores with Resample Ablations than Mean Ablations. Conceptually, resample ablations induce the network to more strongly predict a particular wrong answer, whereas the mean activation attempts to simply destroy the signal of the clean prompt. We agree that methods may not necessarily be inconsistent if relative ordering is maintained. As we evaluate single circuits in this work (and don’t discover circuits ourselves), we do not study this relative ordering of circuits, but our null hypothesis is that all methodological choices matter. Also, absolute faithfulness scores are interpretable and meaningful.\n\nOur recommendations for researchers are fairly limited as the focus of our paper is to highlight problems in existing circuit discovery research. See rebuttal to reviewer 3bug for further discussion of this point.\n\n[1] https://arxiv.org/abs/2211.00593", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It addresses a relevant problem", "reasons_to_reject": "Abstract and introduction creates great expectation that are somehow not met in the subsequent sections and in particular by the results. The recommendations for researchers end up to be quite shallow as the conclusions.", "questions_to_authors": "See list of comments", "extra_scores": {}}, {"review_id": "pS8Fd5HJIB", "reviewer": "Reviewer_se2Z", "summary": "The work focuses on the six dimensions of variation used in studies evaluating transformer circuits via ablation. It carries out a comprehensive analysis of these and shows that circuit analysis and evaluation are affected by all these dimensions. That is, variations among them lead to differences in what types of conclusions we can draw from this analysis. The paper's main claim appears to be the following: talking about “a circuit responsible for a task” may not be sensible; as the results change depending on the used ablation analysis, we should always narrow our claims relative to that.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed feedback.\n\nWe agree that certain methodological points are omitted. We will include a more thorough description of methods at the beginning of each experiment. Regarding the specific points you raise:\n- In our work, we test the evaluation metrics used by prior circuit discovery work. As such, we study the very same circuits found by those works, which are often in GPT-2. We follow [1] here, and many of the circuits that we study are canonical in the MI literature e.g. [2, 3, 4]. We agree that the model choice is another important dimension of circuit discovery, but disagree that the model is a choice to be made when evaluating some particular circuit. Note that we do perform experiments in the appendix on circuits found in different models (e.g. Pythia 2.8B).\n- We will include a mathematical description of logit difference recovered. A logit diff greater than 100% means the logit diff output by the circuit is greater than the logit diff of the full model, which is suboptimal because it means that the circuit is not producing the same output as the full model.\n\nRegarding Section 3.1.5, it is true that the correctness of the circuit is largely independent of the model performance on a task. The aim of localizing model behavior requires only that the circuit reproduces the model’s behavior of the task, not that the circuit performs well at the task. As mentioned in our comment to reviewer W9hJ, our use of faithfulness here is following previous work.\n\nAblating the complement of a circuit is valuable because it tests whether the circuit is sufficient to perform the task by eliminating any useful computations that might be done outside of the circuit. Ablating the circuit itself on the other hand is coarse – for instance, just ablating one important edge in a circuit may be sufficient to destroy performance.\n\nThank you for the idea to run a statistical significance test. The difference between Mean and Resample Ablation was significant when ablating nodes (p=1e-5), but not edges. We will include this result.\n\nThe IOI authors do consider the possibility of two distinct circuits that are combined with an OR operation in a late layer (in section 4.1 of their paper), but as they do not themselves notice the difference between the ABBA and BABA prompts, they do not propose this hypothesis.\n\n[1] https://arxiv.org/abs/2304.14997\n[2] https://arxiv.org/abs/2405.08366\n[3] https://arxiv.org/abs/2310.04625\n[4] https://arxiv.org/abs/2309.16042", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, the paper provides an insightful contribution to the field of transformer-based mechanistic interpretability, both by showing the limitations and flaws of current approaches and proposing some directions on how to mitigate some of the problems (e.g., by using the same ablation method for ground truth and circuit discovery).\n\n- Given the growing interest in MI, the contribution of the paper is timely and potentially impactful.\n\n- The paper is overall well written and provides a good overview of the previous/concurrent literature.", "reasons_to_reject": "- The paper should pay more attention to clarifying, introducing, and motivating its terminology. Many terms are used without being explained and this makes the reader’s life hard in various places. For example, what does a percentage of recovered logit difference > 100% mean and how should it be interpreted?\n\n- As is often the case of papers within the MI field, many methodological details and the terminology are given for granted: e.g., that GPT-2 is used for the experiments (this is never really mentioned in the main paper).\n\n- The whole paper used GPT-2 as its reference model. While I understand that this is the most widely used model in the MI literature, I do believe that the \"model\" is a seventh important dimension of variation to consider in a comprehensive evaluation of circuit discovery and evaluation. To make the claims even more insightful, other (pre-trained) models should have been considered. \n\n- Section 3.1.5: it is unclear why it makes sense to focus on the complement components (the ones not in the circuit) when performing evaluation. There are two dimensions here, that the authors should explain and motivate better: 1) model performance on the task: this is destroyed when a clean circuit is ablated, restored when a corrupt circuit is ‘clean resampled’; 2) evaluation of the circuit: how important/unimportant the target/complement circuit is for the task. This, to me, seems to be independent of model performance, as it only tests whether the role or importance of a circuit (being it the target or the complement) is kept the same or changed after an ablation. I would therefore explain better the terms faithfulness and model/circuit performance and their difference and to motivate better why focusing on the complement circuit is a valuable approach.", "questions_to_authors": "- “This results in a systematically lower faithfulness as compared with mean ablations”: Are the differences between resample and mean in Figure 3 statistically significant?\n\n- “The IOI circuit implements the IOI task faithfully on average, but not for many single data points.”: the difference between ABBA and BABA seems to suggest that there may be two distinct circuits responsible for the two orders. Is it something that the authors hypothesized, too, and perhaps tested experimentally?", "extra_scores": {}}, {"review_id": "QAYQmyvkzy", "reviewer": "Reviewer_3bug", "summary": "Lots of work has aimed to characterize how networks accomplish tasks by attributing those behaviors to \"circuits\" inside of the network. These prior works have evaluated the extent to which a circuit explains a behavior based on ablating non-circuit components to see if the circuit computes the hypothesized function. They identify several key degrees of freedom with ablations: granularity of the computational graph representation, type of activation value, token positions, ablation direction, and the set of components. Through experiments, this paper shows that reasonable variations in methodology can produce significantly different results. \n\nI think that this paper is very needed, will be valuable to myself and the people I work with, and should be accepted.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed and encouraging feedback. We are glad they believe the paper will be a valuable contribution.\n\nThe reviewer presents an interesting perspective on the correct resolution to debates around faithfulness metrics. There are a range of views on this topic in the literature. For instance [1], (as cited by reviewer W9hJ), rejects explanations that enable users to make better decisions with the help of the model as a poor metric for the faithfulness of the explanation. Instead, they claim that this is merely “indicative of correlation between the plausibility of the explanations and the model’s performance.” This is not a debate which we feel our paper needs to resolve. As explained in our comments to other reviewers, our primary motivation is to critique the faithfulness claims of previous papers and warn against over-optimizing particular metrics.\n\nWe do however believe that better evaluation is important. This is a perspective shared by others, e.g. [2]. One example of a potential improvement to evaluation methodology could be to test the circuit across a very wide distribution, as compared with the narrow distribution used to discover it, to find whether it implements the same behavior more broadly. In particular, some of the authors are currently investigating a method which would ablate the circuit and test whether the tokens with the greatest decrease in loss across a large dataset are instances of the task that the circuit is supposed to perform. \n\nAlternatively, the interpretability community has had some limited recent success in applying insights on 'features’ or 'representations’ to better control models [3], which is an example downstream application. It is an interesting question to ask whether the same may be possible for entire computational circuits. We suspect it will be found to be significantly harder. Concretely, it may be possible to leverage circuit-based insights to perform highly specific capability unlearning [4] or parameter efficient finetuning [5], as a generalization of recent work attempting to use features to do the same.\n\n[1] https://arxiv.org/abs/2004.03685\n[2] https://www.alignmentforum.org/s/a6ne2ve5uturEEQK7/p/f8nd9F7dL9SxueLFA\n[3] https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html\n[4] https://arxiv.org/abs/2403.03218\n[5] https://arxiv.org/abs/2404.03592", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1: This work seems to do an exceptionally good job of engaging with past work down to the inclusion of specific details from prior works. \n\n2: This work seems to join in a useful tradition of works showing that cherished attribution methods fail certain sanity checks. I think that works like this have been really valuable to the literature in past years. \n\n3: This is a kind of work that participates in a really useful process of critiquing past methods. I think that it will definitely be useful for a variety of future works.\n\n4: Figure 3 is compelling to me. \n\n5: I would not have thought of the ideas in this paper despite doing related work myself.", "reasons_to_reject": "6: Not a strong reason to reject -- I think that the conclusion that the paper draws is not exactly the right one. I think that the main implication of this work is that the important thing to do is not to invent different faithfulness metrics but to figure out how to connect circuits attribution methods to meaningful practical tasks. Currently, almost all claims about the function of circuits in the literature are based on pontification that aren't meaningfully connected to competitive uses in practical applications. Inventing new faithfulness metrics wouldn't solve that. Only real-world applications would. \n\n7: Not a strong reason to reject -- this paper does not propose concrete solutions per se. But I wouldn't expect it to.", "questions_to_authors": "8. What do you think a good evaluation methodology would look like? Is finding a principled patching-based evaluation method hopeless?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates different ablation methods used for measuring the faithfulness of circuits in mechanistic interpretability and shows how such methodological choices can lead to different results. The authors study a diverse set of circuits from prior work, found via automated discovery or manual approaches.

Overall, as unanimously brought up by the reviewers, surveying and unifying prior circuit faithfulness measures provides valuable insights and is timely. Reviewers provided constructive recommendations on how to improve the presentation of the work, clarify definitions, and distinguish the findings from other work that provides evidence for similar (though not identical) interpretability illusions of different patching variations. Perhaps most importantly, reviewers offered more nuanced interpretations of the results, which I agree better communicate the findings. The authors have agreed to incorporate these insights, which I believe will significantly improve the paper.
Decision: Accept