Towards Automated Circuit Discovery for Mechanistic Interpretability
Authors: Arthur Conmy
Augustine N. Mavor-Parker
Aengus Lynch
Stefan Heimersheim
Adrià Garriga-Alonso
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Mechanistic Interpretability
Pruning
Science of Deep Learning
AI Safety
TL;DR: We identify the common workflow for mechanistic interpretability work, and automate its “systematic ablations” step with a new algorithm, ACDC.

Abstract: Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of
transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers
choose a metric and dataset that elicit the desired model behavior. Then, they apply activation patching to find which
abstract neural network units are involved in the behavior. By varying the dataset, metric, and units under
investigation, researchers can understand the functionality of each component.

We automate one of the process' steps: finding the connections between the abstract neural network units that form a circuit. We propose several algorithms and reproduce previous interpretability results to validate them. For
example, the ACDC algorithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that computes the
Greater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by
previous work. Our code is available at https://github.com/ArthurConmy/Automatic-Circuit-Discovery

Towards Automated Circuit Discovery for Mechanistic Interpretability
Arthur Conmy" Independent
Augustine N. Mavor-Parker'
Aengus Lynch"
Stefan Heimersheim University of Cambridge
UCL
UCL
Adrià Garriga-Alonso
FAR AI
Abstract
Through considerable effort and intuition, several recent works have reverse- engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a metric and dataset that elicit the desired model behavior. Then, they apply ac- tivation patching to find which abstract neural network units involved in the behavior. By varying the dataset, metric, and units under investigation, researchers understand the functionality of each component.
We automate one of the process' steps: finding the connections between the abstract neural network units that form circuit. We propose several algorithms and repro- duce previous interpretability results validate them. For example, the ACDC algo- rithm rediscovered 5/5 of the component types in a circuit in GPT-2 Small that com- putes the Ereater-Than operation. ACDC selected 68 of the 32,000 edges in GPT-2 Small, all of which were manually found by previous work. Our code is available at https: / / thub. com/ ArthurConmy/ Aut.maticcCCe Ciweee--rrrutttte -rrcuutt-iscoveryy...
Introduction
Rapid progress in transformer language modelling (Vaswani et al., 2017; Devlin et al., 2019; OpenAI, 2023, inter alia) has directed attention towards understanding the causes new capabilities (Wei et al., 2022) in these models. Researchers have identified precise high-level predictors of model performance (Kaplan al., 2020), but transformers are still widely considered 'black-boxes (Alishahi, Chrupata, and Linzen, 2019) like almost all other neural network models (Fong and Vedaldi, 2017; Buhrmester, Münch, and Arens, 2021).² Interpretability research aims to demystify machine learning models, for example by explaining model outputs in terms of domain-relevant concepts (Zhang al., 2021).
Mechanistic interpretability focuses on reverse-enginn engeree-nngineeiing model components into human- understandable algorithms (Olah, 2022). Much research in mechanistic interpretability views models as computational graph (Geiger al., 2021), and circuits are subgraphs with distinct functionality (Wang et al., 2023). The current approach extracting circuits from neural networks relies lot of manual inspection by humans (Räuker al., 2022). This major obstacle to scaling up mechanistic interpretability to larger models, more behaviors, and complicated behaviors composed of many sub-circuits. This work identifies a workflow for circuit research, and automates part of by presenting several methods to extract computational graphs from neural networks.
Our main contributions are follows. First, we systematize the common workflow prevalent in many existing mechanistic interpretability works, outlining the essential components of this process
Work partially done Redwood Research. Correspondence to crthurcommy@@maill. 'Though this perspective is not universal (Lipton, 2016).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GPT-2 Small
ACDC Circuit
120
Figure I: Automatically discovering circuits with ACDC. Left: computational graph for GPT-2 Small, with recovered circuit for the IOI task highlighted in red. Only edges between adjacent layers are shown. Right: the recovered circuit with labelled nodes. All heads recovered were identified as part of the 101 circuit by Wang et al. (2023). Edge thickness is proportional to importance.
(Section 2). One of its steps is to find a subgraph of the model which implements the hehavior of interest, which is step possible automate. We introduce Automatic Circuit DisCovery (ACDC), novel algorithm that follows the way in which researchers identify circuits (Section 3), and adapt Subnetwork Probing (SP: Cao, Sanh, and Rush, 2021) and Head Importance Score for Pruning (HISP; Michel, Levy, and Neubig, 2019) for the same task. Finally, we introduce quantitative metrics to evaluate the success of circuit extraction algorithms (Sections and 4.2). We present detailed ablation study of design choices in Appendix E and qualitative studies in Appendices F, G, H, I and J.
The Mechanistic Interpretability Workflow
Mechanistic interpretabilityy attempts explain and predict neural network behaviors by understand- ing the underlying algorithms implemented by models. In the related work section we discuss the mechanistic interpretability field and its relationship to 'circuits' research (Section 5). Neural network behaviors are implemented by algorithms within the model's 's computational graph, and prior work has identified subgraphs (circuits, following Wang al. (2023)'s definition) that capture the majority of particular behaviors. In this section, we describe a workflow that several prior works have followed that has been fruitful for finding circuits in models.
As concrete example of an approach taken to finding a circuit, Hanna, Liu, and Variengien (2023) prompt GPT-2 Small with a dataset of sentences like "The war lasted from 1517 to 15". GPT-2 Small completes this sentence with "18" or "19" or any larger two digit number, but not with any two digit number that is most "17" (from here, refer to prompt completions like this as the "Greater- Than" " task). This behavior can be measured by the difference in probability the model places on completion "18" or "19" larger and the probability the model places completion "17" or smaller. Note that we use the term 'dataset' to refer to collection of prompts that elicit some behavior in model: do not train models these examples, as in this paper focus on post-hoc interpretability.
The researchers then create corrupted dataset of sentences that do not have any bias against particular two digit completions (the 'O1-dataset' (Hanna, Liu, and Variengien, 2023)). The researchers attribute the greater-thar than operation to late layer MLPs and then find earlier components that identify the numerical values of years, including attention heads in the model. Finally. Hanna, Liu, and Variengien (2023) interpret the role of each set of components. For example, they identify early model components that respond to the "17" token, and later model components that boost the importance of logits for years greater than 17.
There are equivalent steps taken a growing number of additional works (Heimersheim and Janiak, 2023, the "Docstring" task; Goldowsky- et al., 2023, the "Induction" task; Wang et al., 2023,

---

the "IOI" task), described in brief in Table and in detail in Appendices F, H and J. We identify the workflow that eventually finds a circuit as following three steps. Researchers:
1. Observe a behavior (or task") that a neural network displays, create a dataset that reproduces the behavior in question, and choose metric measure the extent to which the model performs the task.
2. Define the scope of the interpretation, i.e. decide what level of granularity (e.g. attention heads and MLP layers, individual neurons, whether these are split by token position) at which one wants to analyze the network. This results in a computational graph of interconnected model units.
3. Perform extensive and iterative series of patching experiments with the goal of removing as many unnecessary components and connections from the model as possible.
Researchers repeat the previous three steps with slightly different dataset granularity, until they are satisfied with the explanation of the circuit components.
This work (ACDC) presents tool to fully automate Step 3. Before we dive into the details of ACDC, we expand on what Steps 1-3 involve, and review examples from previous work that we use to evaluate ACDC.
2.1 Step 1: Select a behavior, dataset, and metric
The first step of the general mechanistic interpretability workflow is to choose a neural network behavior to analyze. Most commonly researchers choose a clearly defined behavior isolate only the algorithm for one particular task, and curate a dataset which elicits the behavior from the model. Choosing a clearly defined behavior means that the circuit will be easier to interpret than a mix of circuits corresponding to vague behavior. Some prior work has reverse- everse-engineered the algorithm behind small model's behavior on all inputs in its training distribution (Nanda et al., 2023; Chughtai, Chan, and Nanda, 2023), though for language models this is currently intractable, hence the focus on individual tasks.
We identified a list of interesting behaviors that we used to test our method, summarized Table 1. These include previously analyzed transformer models and 3 on GPT-2 Small, and on smaller language transformers) where researchers followed workflow similar to the one we described above. Tasks 4 and 5 involve the full behavior of tiny transformers that implement known algorithm, compiled with tracr (Lindner et al., 2023). For each task, we mention the metric used in previous work to measure the extent to which the model performs the task on the corresponding dataset.
2.2 Step 2: Divide the neural network into a graph of smaller units
To find circuits for the behavior of interest, one must represent the internals of the model computational directed acyclic graph (DAG, e.g. Figure 2a). Current work chooses the abstraction level of the computational graph depending on the level of detail of their explanations of model behavior. For example, at coarse level, computational graphs can represent interactions between attention heads and MLPs. At a more granular level they could include separate query, key and value activations, the interactions between individual neurons (see Appendix I), or have a node for each token position (Wang et al., 2023).
Node connectivity has to be faithful to the model's computation, but that does not fully specify its definition. For example, following Elhage et al. (2021), many works consider the connections between model components in non-adjacent layers due to the additivity of the residual stream, even though these are computed with dynamic programming in the actual model implementation. Connectivity defines what is considered a direct or a mediated interaction (Pearl, 2009; Vig et al., 2020). See for example Figure 2a, where component B has both direct effect the output node O and indirect effect on the output through component A.
Section 3 formally detines "task". We use "behavior" and "task" interchangeably.
3

---

Task
Example Prompt
Output
Metric
1: I0I
"When John and Mary went to the store, Mary gave a bottle of milk to"
"John"
Logit
(Appendix F.2)
difference
Docstring (Appendix H.I)
def (eelf, files, obj, state, size, shape, option):
shape"
Logit difference
"*"document string example
iparan state: performance analyaie :paran size: pattern dasign pparan
Greater-Than (Appendix G)
"The war lasted from 1517 15"
"18" "19" or Probability
or "99"
difference
4: tracr-xproportion (Appendix 1.1)
["a", "x", "b", "x"]
[o, 0.5,
Mean Squared
0.33, 0.5]
Error
5: tracr-reverse (Appendix I.2)
[0, 3, 2, 1]
[1, 2, 3, 0]
Mean Squared Error
6: Induction (Section 4.2)
"Vernon Dursley and Petunia Durs"
"ley"
Negative log- probability
Table 1: Five behaviors for which we have an end-to-end circuit from previous mechanistic in- terpretability work, plus Induction. We automatically rediscover the circuits for behaviors 1-5 in Section 4. Tokens beginning with space have a prepended for clarity.
2.3 Step 3: Patch model activations to isolate the relevant subgraph
With the computational DAG specified, one can search for the edges that form the circuit. We test edges for their importance by using recursive activation patching: i) overwrite the activation value of node edge with corrupted activation, ii) run forward pass through the model, and iii) compare the output values of the new model with the original model, using the chosen metric (Section 2.. One typically starts the output node, determines the important incoming edges, and then investigates all the parent nodes through these edges in the same way. is this procedure that ACDC follows and automates in Algorithm 1.
Patching with zeros and patching with different activations Activation patching methodology varies between mechanistic interpretability projects. Some projects overwrite activation values with zeros (Olsson et al., 2022; Cammarata et al., 2021), while others erase activations' informational content using the mean activation the dataset (Wang et al., 2023). Geiger et al. (2021) prescribe interchange interventions instead: to overwrite a node's activation value on one data point with its value on another data point. Chan et al. (2022) justify this by arguing that both zero and mean activations take the model too far away from actually possible activation distributions. Interchange interventions have been used in more interpretability projects (Hanna, Liu, and Variengien, 2023; Heimersheim and Janiak, 2023; Wang et al., 2023), so we prefer it. However we also compare all our experiments replacing activations with zeros (Section 4.2, Appendix E.2).
2.4 Explaining the circuit components
After successfully isolating a subgraph, one has found circuit (Section 1). The researcher then can formulate and test hypotheses about the functions implemented by each node in the subgraph. There is early evidence that ACDC is helpful for making novel observations about how language models complete tasks, such as the importance of surprising token positions that help GPT-2 Small predict correctly gendered pronouns (Appendix K). In our work we focus on automating the time-consuming step that precedes functional interpretation internal model components, though we think that automating the functional interpretation of model components is exciting further rescarch direction.

---

B
(a) Choose computational graph, task, and threshold т.
(b) At each head, prune unimpor- tant connections.
Recurse until the full circuit recovered.
How ACDC works (Steps 2a-2c). Step 2a: practitioner specifies computational graph of the model, the task they want to investigate, and a threshold under which to remove connections. Step 2b: ACDC iterates between a node and its children, and measuring the effect on the output metric. Connections are removed if their measured effect the metric under corruption is below the threshold т. Step 2c: recursively apply Step 2b to the remaining nodes. The ACDC procedure returns subgraph of the original computational graph.
Figure 2:
nodes in the computational graph, replacing activations of connections
3 Automating circuit discovery (Step 3)
This section describes algorithms to automate Step 3 of the mechanistic interpretability workflow (Section 2.3). In all three cases, we assume that the 'task' being studied is defined by a set of prompts (x:) on which the model's predictions have noticeable pattern (see Table for examples) and set of prompts (x where this task is not present. We then use the activations of the models on forward pass on the points n/ as corrupted activations (Section 2.3).
Automatic Circuit DisCovery (ACDC). Informally, a run of ACDC iterates from outputs to inputs through the computational graph, starting the output node, to build subgraph. At every node it attempts to remove as many edges that enter this node as possible, without reducing the model's performance selected metric. Finally, once all nodes are iterated over, the algorithm (when successful) finds graph that i) is far sparser than the original graph and ii) recovers good performance on the task.
To formalize the ACDC process, we let be computational graph the model of interest, at a desired level of gramularity (Section 2.2), with nodes topologically sorted then reversed (so the nodes are sorted from output to input). Let S be the computational subgraph that is iteratively pruned, and 0 threshold that determines the sparsity of the final state of H.
We now define how we evaluate subgraph We let (x;; .!) be the result the model when is the input to the network, but we overwrite all edges in G that are not present in H to their activation on x (the corrupted input).* This defines H (x x(), the output probability distribution of the subgraph under such experiment. Finally we evaluate H by computing the KL divergence DRI. (G(,,) (G(H,))))H(( (I; x{)) between model and the subgraph's predictions. We DxI. (H) denote the average KL divergence over set of datapoints. Appendix C discusses alternatives to the KL divergence, and Appendix E. explores the consequences of optimizing the task specific metrics from Table instead.
Algorithm describes ACDC. The order in which we iterate over the parents w of u a hyperparame- ter. In our experiments the order is lexicographically from later-layer MLPs and heads to earlier-layer MLPs and heads, and from higher- lower-indexed heads. We note that one case in our work, the order of the parents affected experimental results (Appendix J).
Subnetwork Probing (SP; Cao, Sanh, and Rush, 2021). SP learns a mask over the internal model components (such attention heads and MLPs), using objective that combines accuracy and
'To implement the computation of H (I,, æf), we initially run a forward with the unmodified model on the input o and cache all activations.
5

---

Algorithm 1: The ACDC algorithm.
Data: Computational graph G, dataset (x:) corrupted datapoints (x4)== and threshold T> 0.
Result: Subgraph H C G.
H f- G
// Initialize H to the full computational graph // Sort so output first
2 H + dreeverse_tpologiinn soor()
for E H do
for w parent of C do
Huew & H. {w + t4 D D (ne
// Temporarily remove candidate edge
D DKL (KI(G)H)) < then // Edge is unimportant, remove permanently
H 5 Hew
end
end
end
11 return H
sparsity (Louizos, Welling, and Kingma, 2018), with a regularization parameter X. At the end of training, round the mask to 0 or 1 for each entry, so the masked computation corresponds exactly to subnetwork of a transformer. SP aims to retain enough information that a linear probe can still extract linguistic information from the model's hidden states. In order to it to automate circuit discovery, we make three modifications. We i) remove the linear probe, ii) change the training metric to KL divergence as in Section 2, and iii) the mask to interpolate between corrupted activations and clean activations (Section 3) rather than zero activations and clean activations. Appendix D. explains the details of these changes.
Head Importance Score for Pruning (HISP; Michel, Levy, and Neubig, 2019). HISP ranks the heads by importance scores (Appendix D.2) and prunes all the heads except those with the top scores. Keeping only the top & heads corresponds We plot the ROC obtained from the full possible range of Like SP, this method only considers replacing head activations with zero activations, and therefore once more generalize it to replace heads and other model components with corrupted activations (for details, see Appendix D.2).
subnetwork that
compare to ACDC.
4 Evaluating Subgraph Recovery Algorithms
To compare methods for identifying circuits, seek empirical answers the following questions.
Q1:
Does the method identify the subgraph corresponding to the underlying algorithm implemented by the neural network?
Q2: Does the method avoid including components which do not participate in the elicited behavior?
We attempt to measure Q1 and Q2 using two kinds of imperfect metrics: some grounded in previous work (Section 4.1), and some that correspond to stand-alone properties of the model and discovered subgraph (Section 4.2).
4.1 Grounded in previous work: area under ROC curves
The receiver operating characteristic (ROC) curve is usefuu because high true-positive rate (TPR) and low false-positive rate (FPR) conceptually correspond affirming Q1 and Q2, respectively.
We consider canonical cireuits taken from previous works which found an end-to-end eircuit explain- ing behavior for tasks in Table 1. We formulate circuit discovery as a binary classification problem, where edges are classified as positive (in the circuit) or negative (not in the circuit). Appendices G, H, I and describe and depict the canonical circuits for each task. Appendix E.3 considers the node classification problem instead, which is less appropriate for ACDC but more appropriate for other methods.

---

101
tracr-reverse
tracr-xproportion
HeP
posilive rate (edges))
0.75
7, %
0.5
0.5
10
10
0.5
10a
Docstring
Greater-Than
10-1
0.25
10-2
10
10 4
10-h
0
0.25
0.5
0.75
0.5
0.5
10 6
False positive rate (edges)
Figure 3: ROC curves of ACDC, SP and HISP identifying model components from previous work, across circuits transformers. The points on the plot are cases where SP and ACDC return subgraphs that are not on the Pareto frontier. The corresponding AUCs are in Table 2.
We sweep range of ACDC thresholds SP regularization parameters X, or number of HISP elements pruned k. We plot pessimistic segments between points on the Pareto frontier of TPR and FPR, over this range of thresholds (Fawcett, 2006). ACDC and SP optimize the KL divergence for tasks where this makes sense (all but tracr tasks, which use the L, distance). All methods employ activations with corrupted data. Appendix C describes and Appendix E experiments with different design choices for the metric and activation patching methodology.
Figure 3 shows the results of studying how well existing methods recover circuits in transformers. We find that i) methods are very sensitive to the corrupted distribution, ii) ACDC has competitive performance (as measured by AUC) with gradient-descent based methods iii) ACDC is not robust, and it fails at some settings.
Several of the tasks appeared to require specific distributions and metrics for the areas under the curves be large. For example, ACDC achieved poor performance on both tracr tasks Fig. but the circuit was perfectly recovered by ACDC at any threshold r >> O when patching activations with zeros (Appendix I). Furthermore, ACDC achieves a greater AUC on the IOI and Greater-Than Than and tracr-reverse tasks than both of the other methods, and hence overall is the optimal algorithm. As an example of the variable performance of circuit recovery algorithms, on the Docstring task we achieve the high perfomance when using the ACDC algorithm with the docstring metric (Appendix H). However in other tasks such as the IOI task, ACDC performance was worse when optimizing for logit difference.
Further research in automated interpretability will likely yield further improvements the FPR and TPR of circuit discovery. We outline limitations with all current methods, but also gesture at likely fundamental limitations of the false positive and true positive measures. A limitation with all existing methods is that they optimize single metric. This means they systematically miss internal model components such as the "negative" components found in previous work (IOI, Docstring) that are actively harmful for performance. The 1OI recovery runs were not able to recover negative heads when optimizing for logit difference. Even when optimizing for low KL divergence, the negative components were only recovered when very small thresholds were used (Figure 15).
Additionally, more fundamental limitation measuring the false and true positive rates of circuit recovery methods is that the ground- -tuth circuits are reported by practitioners and are likely to have included extraneous edges and miss more important edges. The language model eircuits studied in our work (Appendices F-H) involve a large number of edges (1041 in the case of IOI) and the full models contain more than an order magnitude more edges. Since these interpretability works are carried out by humans who often report limitations of their understanding, our "ground-truth" is not 100% reliable, limiting the strength of the conclusions that can be drawn from the experiments in this section.
7

---

a) Induction (zero)
b) Induction (corrupted)
T, %
10a
HEP
(L(model, ablated)
000
10
x
SP
10
10
10
Clean
10-1
10
10 3
10-4
5
10
00
06
000
002
10
20
000
200
Number of edges
Figure 4: Comparison of ACDC and SP with both zero-input activations (left) and corrupted activa- tions (right). We plot the KL Divergence on a held-out test set against the number of edges of each hypothesized circuit. Lower KL divergence and fewer edges correspond to better subgraphs. Darker points include more edges in the hypothesis: they use smaller ACDC т, smaller SP regularization À or higher percentage of nodes in HISP.
4.2 Stand-alone circuit properties with a test metric
This section evaluates the algorithms by studying the induction task. We measure the KL Divergence of the circuits recovered with the three methods to the original model. This is an indirect measure of Q1, with the advantage of not relying the completeness or correctness of previous works. As an indicator of Q2, we also measure the number of edges that hypothesized circuit contains. A circuit with fewer edges which still obtains a low KL Divergence is less likely contain components that do not participate in the behavior. In Appendix L we also introduce and explain experiments on reset networks that provide more evidence for Q2
Our mainline experimental setup is to run the circuit recovery algorithms as described in Algorithm 1 and Section 3 and then measure the KI. Divergence for these circuits the induction task (Ap- pendix J). In brief, ACDC performs better that the other methods under these experimental conditions with both corrupted and zero activations. For example, the left-hand side of Figure 4 shows that, above 20 edges, ACDC starts having a slight advantage over other methods in terms of behavior recovered per number of edges as all points are generated from ACDC runs. Appendix E describes many further experiments with variations on setup provide a more complete picture of the performance of the circuit recovery algorithms. For example, when we measure the loss (the task-specific induction metric; Table 1) of subgraphs recovered by optimizing KL Divergence, we find very similar qualitative graphs Figure 4.
the Pareto-frontier with least this many edges
In Appendix L. we see that the KI. divergence that all methods achieve is significantly lower for the trained networks, indicating that all the methods get signal from the neural network's 's ability to perform induction (Figure 4). HISP and SP with zero activations, and to some extent SP with corrupted activations are also able to optimize the reset network. This suggests that these methods are somewhat more prone to finding circuits that don't I exist (i.e. evidence against Q2).
Related work
Mechanistic interpretability encompasses understanding features Iearnt by machine learning mod- els (Olah, Mordvintsev, and Schubert, 2017: Elhage al., 2022), mathematical frameworks for understanding machine learning architetures (Elhage et al., 2021) and efforts to find circwits in models (Nanda et al., 2023; Cammarata et al., 2021: Chughtai, Chan, and Nanda, 2023; Wang et al., 2023). The higher standard of mechanistic understanding of a model has already had applications to design- ing better architectures (Fu al., 2023), though the speculative goal of mechanistic interpretability is to understand the behavior of whole models, perhaps through describing all their circuits and how they compose. Little work has been done to automate interpretability besides Bills al. (2023) who use language models to label neurons in language models.
8

---

Neural network pruning masks the weights of neural networks to make their connectivity more sparse (LeCun, Denker, and Solla, 1989). In contrast to our aims, the pruning literature is typically concerned with compressing neural networks for faster inference or to reduce storage requirements (Wang. Wohlwend, and Lei, 2020; Kurtic al., 2022). Farly work (Hassibi and Stork, 1992) hoped pruning would lead to more interpretable networks, but progress towards interpretability via pruning is limited (Grover, Gawri, and Manku, 2022).
Pruning techniques may learn masks from data, which is a special case of more generally using gradient information. Masks can also be learned from data, with an objective function that balances model performance and network sparsity (Louizos, Welling, and Kingma, 2018; Wang. Wohlwend, and Lei, 2020; Cao, Sanh, and Rush, 2021). This is a useful comparison to ACDC as learnable masks do not change the weights of our model after pruming (Frantar and Alistarh, 2023). Examples of gradient information being used more generally includes Michel, Levy, and Neubig (2019) who decide which heads should be pruned by using the absolute value of their gradients, while "movement pruning" (Sanh, Wolf, and Rush, 2020) removes parameters that have high velocity to a low magnitude. ACDC is different from pruning and other compression techniques (Zhu et al., 2023) since i) the compressed networks we find are reflective of the circuits that model's ss use to compute outputs to certain tasks (Section 4) and ii) our goal is not to speed up forward passes, and generally our techniques slow forwards passes.
Causal interpretation. Much prior research on understanding language models has drawn inspiration from causal inference (Pearl, 2009), leading to the development of frameworks that provide causal explanations for model outputs (Pearl, 2009; Feder et al., 2021; Geiger et al., 2021; Wu et al., 2022; Kaddour et al., 2022). Other work (Vig et al., 2020) discusses the difference between indirect effects and direct effects inside language models, and experiments on removing subsets of these heads using heads' direct effects as proxies for the overall contribution of these heads. Goldowsky-Dill et al. (2023) introduce 'path patching" to analyze the effects of different subsets of edges in computational graphs of models. In parallel to our work, Wu et al. (2023) develop a method to automatically test whether neural networks implement certain algorithms with causal testing. Our work is focused on finding rather than verifying an outline of an algorithm implemented by a model.
Computational subgraphs for interpretability. Training dynamics in residual models can be explained by shallow paths through the computational graph (Veit, Wilber, and Belongie, 2016). MLP layers can be modelled as memory that is able to represent certain properties of the network inputs (Geva et al., 2021). Residual transformer models have been modelled as the sum of all different paths through the network (Elhage et al., 2021). Later work has used insights from looking at subgraphs of models order to edit models' behaviors (Bau et al., 2020; Meng et al., 2022) and test interpretability hypotheses (Chan et al., 2022).
6 Conclusion
We have identified a common workflow for mechanistic interpretability. First, pin down a behavior using a metric and data set. Second, conduct activation patching experiments to understand which abstract units (e.g. transformer heads) are involved in the behavior. Third, iterate the previous steps with variations of the behavior under study, until the model's algorithm is understood.
The main proposed algorithm, ACDC, systematically conducts all the activation patching experiments necessary to find which circuit composed of abstract units is responsible for the behavior. We have shown that ACDC and SP recover most of the compositional circuit that implements a language model behavior, as judged by comparison to previous mechanistic interpretability work (Section 4). ACDC with zero activations fully recovers the circuit of toy models (Fig. 9). Further, there is carly evidence of the use of ACDC to help with novel interpretability work, discovering surprising outline of a subgraph of GPT-2 Small that predicts gendered pronoun completion (Appendix K). Here, practitioners used ACDC to generate a subgraph including the most important pathway through a model's 's computation, and checked that this reflects the model's computation in normal (unablated) forward passes. This surprising find was early example of the summarization motif (Tigges et al., 2023).
However, both ACDC and SP have limitations which prevent them from fully automating step 3 of the identified workflow (activation patching). First, they tend miss some classes of abstract units that are part of the cireuit, for example the negative name mover heads from I0I (Wang et al., 2023).

---

Second, the behavior of the algorithms is very sensitive to hyperparameter and metric choice, leading to varied and non-robust performance in some settings (Figure 3).
On balance, the evidence supports the claim that ACDC can automate part of interpretability work, a novel contribution. Automating interpretability research may be necessary to be able to scale methods to the behaviors of the large models which are in use today. We hope that our open-source imple- mentation of ACDC (https://g² tigg. . com/ ArthurConmyy Att.//attticiiiiiii Arraturooom///tttttttiii--- Seriiii-Crruut--iii. DDisscyyry)
accelerates interpretability research from the community. For example, future work could systematize and automate the problem of varying the corrupting dataset understand the functionality of different parts of the circuit.
7 Acknowledgements
This work would not have been possible without the generous support of Redwood Research through their REMIX program. We would like to thank Chris Mathwin, Jett Janiak, Chris MacLeod, Neel Nanda, Alexandre Variengien, Joseph Miller, Thomas Kwa, Sydney von Arx, Stephen Casper and Adam Gleave for feedback on a draft of this paper. Arthur Commy would like to thank Jacob Steinhardt, Alexandre Variengien and Buck Shlegeris for extremely helpful conversations that shaped ACDC. We would also like to thank Haoxing Du for working on an early tool, Nate Thomas for coming up with the catchy name, Daniel Ziegler who discussed experiments that inspired our Subnetwork Probing analysis, Oliver Hayman who worked on an earlier prototype during REMIX and Lawrence Chan who helped us frame our contributions and suggested several experiments. Finally we thank Hofvarpnir Studios, FAR Al and Conjecture for providing compute for this project.
References
Alishahi, Afra, Grzegorz Chrupala, and Tal Linzen (2019). "Analyzing and interpreting neural net- works for NLP: A report on the first BlackboxNLP workshop". In: Natural Language Engineering
25.4, pp. pp.pp3-5555...... 543-557. DOI: 10. 1011 1017/$11351324919000222
Bau, David, Steven Liu, Tongzhou Wang. Jun Yan Zhu, and Antonio Torralba (2020). Rewriting a Deep Generative Model. URL:: https: pettps://arxivv org/ abs/2007... 15646.
Bills, Steven, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders (2023). Language models can explain neurons in language models. https: http::://ppenaipubllle paper/index html.
blob. core. windows net wwt/eeeeeeee /nwurnnuuonnnnnee neeronn-eppaiierr
Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. (2018). "The malicious use of artificial intelligence:
Forecasting, prevention, and mitigation". In: arXiv preprint 1802::8002.0722 Buhrmester, Vanessa, David Münch, and Michael Arens (2021). " "Analysis of Explainers of Black Box
Deep Neural Networks for Computer Vision: A Survey". In: 3.4, pp. 966 989.9999.000 ISSN: 2504-4990.
DOI: 20..399//00 m0k03040048 URL: https: //www/ mdpi. wtt 17504-4990/3/444
Cammarata, Nick, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah (2021).
"Curve Circuits". In: Distill. http://tiillllbii222/iiiiiiiired
DOI: 10. 23915/
Sestll.00024 006.
Cao, Steven, Victor Sanh, and Alexander Rush (2021). "Low-Complexity Probing via Finding Subnetworks". In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, pp. 960-966. DOI: 10. 10653/61/22221111 www.naaaaaaaaaa 74. URL: https://acl http://aaaanttoogy...o.oo org. /2011 maacl-maii. 74.
Chan, Lawrence, Adria Garriga-Alonso, Nix Goldowsky-Dill Ryan Greenblatt, Jenny Nitishin- skaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas (2022). Causal scrubbing: A method for rigorously lesting interpretabiliry hypotheses. Alignment Forum. URL: https: //www. ilignmentfformm org/ http/oots///tt poottt//IZ2hhhyychhyyydaaar httpp///rw...eeeaaaaaaaauuuaaaaa
causal -ew.--bbigaig---aattttoooe a-me--od-
soorously-testing
Chughtai, Bilal, Lawrence Chan, and Neel Nanda (2023). A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations. URL: https: //aaa//aaaii.ee/ea arrzivv.rrg/bbs//3022 03025.
Cuadros, Xavier Suau, Luca Zappella, and Nicholas Apostoloff (2022). "Self-conditioning pre-trained
language models". In: International Conference on Machine Learning. PMLR, pp. 4455-4473.
10

---

Devlin, Jacob, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, pp. 4171-4186. DOT: 10. 11653//11/1111114//00 URL.:
https: htap:://aaaaaathllogg... .eggy.ozz org Org/N19-11 1423.
Elhage, Nelson, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec,
Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. (2022). "Toy Models of Superposition". In: arXiv preprint arXiv:2209.10653
Elhage, Nelson, Eelaoe,Nesson, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann. Amanda
Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Haffield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse,
Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah (2021). "A Mathematical Framework for Transformer Circuits". In: Transformer Circuits Thread. URL: https: www.ssforree-rrrcieeeeee circuits pubp htab/202/ 20222/2////// ftrm//orrriiiideee... http//ddeee..mll......... html.
Fawcett, Tom (2006). "An introduction to ROC analysis". In: Pattern Recognition Letters 27.8.
ROC Analysis in Pattern Recognition, pp. 861-874, ISSN: 0167-8655. DOI: https://dol org/
1 10.1006/// j patrec p 2005 10. 010. URL: https: : / www. sci aciencedirect com science/
artiicll/pp1//011776555000000
Feder, Amir, Nadav Oved, Uni Shalit, and Roi Reichart (2021). "CausaLM: Causal Model Explanation
Through Counterfactual Language Models", In: Computational Linguistics 47.2. pp. 333-386. DOL:
10. 1..110 10.1662cool.. ____________ 00400 URI.: https:
org/ 2rg/2021. c1-2. 13.
Fong. Ruth C. and Andrea Vedaldi (2017). "Interpretable Explanations of Black Boxes by Meaningful Perturbation". In: IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, pp. 3449-3499 1149-3457.10 DOI: 10. 1109 /ICV/ 2017 3711
URL: https: / /toi. org/10. 1 1109/I100 1009/ICV..000 2.W.2.017377
Frantar, Elias and Dan Alistarh (2023). SparseGPT: Massive Language Models Can Be Accurately
Pruned in One-Shot.
Fu. Daniel Y. Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re (2023). "Hungry Hungry Hippos: Towards Language Modeling with State Space Models". In: The Eleventh International Conference on Learning Representations. URL: / /pppr//vee......eee/ee http/forunndd-COZZyyY00
Geiger, Atticus, Hanson Lu, Thomas Icard, and Christopher Potts (2021). Causal Abstractions of
Neural Networks. URL: pestps://arxxe orgg al/aab///2/ /obos/2100 2106. 02997.
Geva, Mor, Roei Schuster, Jonathan Berant, and Omer Levy (2021). "Transformer Ferw-Forward
Layers Are Key-Value Memories". In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, pp. 5484-5495. DOI: 10. 106.3886/1vv000 https: //acl:///cll aotoooooynnoogoooooooooooooooooa org/ 22211 emnlp-maill Gokaslan, Aaron, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex (2019). OpenWebText Corpus.
emnlp-main, .46.. URL:
URL: Ststts::/Skyyiooe git.iiiion0.gguuubbb.. github. 1o/ hto/penneebexxtorppe Goldowsky-Dill, Nicholas, Chris MacLeed, Lucas Sato, and Aryaman Arora (2023). Localizing Model Behavior with Path Patching. arXiv: 2304. 05969 CC].
Grover, Jasdeep Singh, Bhavesh Gawri, and Ruskin Raj Manku (2022). "DeepCuts: Single-Shot Interpretability based Pruning for BERT". In.
Gurnee, Wes, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas (2023). Finding Neurons in Haystack: Case Studies with Sparse Probing. arXiv: 2305.01610 [cs. LC].
Hanna, Michael, Ollie Liu, and Alexandre Variengien (2023). How does GPT-2 compute greater-
than?: Interpreting mathematical abilities in a pre-trained language model. arXiv: 2305. 00586 [Cs.CL).
Hassibi, Babak and David Stork (1992). "Second order derivatives for network pruning: Optimal brain surgeon".. In: 5.
Heimersheim, Stefan and Jett Janiak (2023). A circuit Python docstrings in a 4-layer attention-only
transformer. URL: https: //www. ali wiiggnnntoruumm... org/ Dostte//66KXkkkkbfffoookkkk circuit-for-ppe -otor-pytooo--- pyython-docstrin docstrings a-4--ayer-- Hendrycks, Dan and Mantas Mazeika (2022). X-Risk Analysis for Al Research, arXiv: 2206. 05862
CC].
11

---

Hendrycks, Dan, Mantas Mazeika, and Thomas Woodside (2023). An Overview of Catastrophic Al Risks. arXiv: 2306. 12001 CYJ. Hernandez, Evan, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas (2022). Natural Language Descriptions of Deep Visual Features. arXiv: 2201. 11114 [cs.CV]
Hubinger, Evan (2020), An overview of 11 proposals for building safe advanced AL. arXiv: 2012.
07532
Jacovi, Alon and Yoav Goldberg (2020). "Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?" In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, DO1: 10. 18653/v1/20000 acl main. 386. URL: https / acl.aaaaaaaooooooooooooooooooooa
pp.
org/2020 org/2020.acl-maal acl-main 386.
Jang. Eric, Shixiang Gu, and Ben Poole (2017). Categorical Reparameterization with Gumbel-
Softmax. Xiv: 1611. 01144 [stat. ML]]
Kaddour, Jean, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva (2022). Causal Machine Learning: A Survey and Open Problems. URL: https: /////aaxivvv org/abs/ 2206 15475. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei (2020). Scaling Laws for Neural Language Models. arXiv: 2001 08361 [cs LG].
Kurtic, Eldar, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh (2022). "The optimal BERT surgeon: Scalable and accurate second-orde pruning for large language models". In.
LeCun, Yann, John Denker, and Sara Solla (1989). "Optimal brain damage". In:
Lindner, David, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik (2023).
"Traer: Compiled Transformers as a Laboratory for Interpretability". In. Lipton, Zachary C. (2016). The Mythos of Model Interpretability, URL: https: org/abs/ 1606.03490.
Louizos, Christos, Max Welling, and Diederik P. Kingma (2018). "Learning Sparse Neural Networks through L_0 O Regularization", In: 6th International Conference on Learning Representations, ICLR 2018, Vancouven: BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net. URL: https: /ttptp:e///peneeviett wwt.teetuuuun http//fruumid-iiy@ahgggg Hd-H1y8
Mathwin, Chris, Guillaume Corlouer, Esben Kran, Fazl Barez, and Neel Nanda (2023). Identifying a Preliminary Circuit for Predicting Gendered Pronouns in GPT-2 Small. URL.: https://itch io/ jam/ wwweiitttaaateeeeeeeeeeeeee 1189977.
McDougall, Callum, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda (2023). Copy Suppression: Comprehensively Understanding an Attention Head. arXiv: 2310. 04625 [ca. LG]. Meng, Kevin, David Bau, Alex J Andonian, and Yonatan Belinkov (2022). "Locating and editing factual associations in GPT". In: Advances in Neural Information Processing Systems.
Michel, Paul, Omer Levy, and Grabam Neubig (2019). "Are Sixteen Heads Really Better than One?" In: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Ed. by Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett, pp. 14014- 14024. URL: https: iew.ucurr.ddngg.n.............. (tt.::001..222000000020020002110 wewpccddssseeee-bbtrrreeee Abstract html.
cc/ //ppp//0100
Mu, Jesse and Jacob Andreas (2021). Compositional Explanations of Neurons. arXiv: 2006. 14032 [cs.LG]
Nanda, Neel (2022). TransformerLens. URL: https: TransformerLenss.
github. com / neelnanda io
(2023). Atribution Patching: Activation Patching Ar Industrial Scale. URL: https / www.
neelnanda.io/ wwt.:///mmecnniiicccmmmmm
Nanda, Neel, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt (2023). "Progress measures for grokking via mechanistic interpretability". In: The Eleventh International Conference
on Learning Representations. URL: https::://perrriiiiee net/f wwworrrrmmmiinseeee XxF.SSDDDPmd....
Olah, Chris (2022). Mechanistic Interpretabilityy Variables, and the Importance of Interpretable Bases. https: //www. transformer--ircutt wtt.irruuussuuuu//eeeeeeeeee httbb/222/meeees mech-interrp-sse
Olah, Chris, Alexander Mordvintsev, and Ludwig Schubert (2017). "Feature Visualization". In: Distill. 00007.
DOI: 10. Olsson, Catherine, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan,
https//distill.pul/ 7featt/eeeaeeeeeiiiieee
Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. (2022). In-context learning and induction
12

---

heads. URL: /trass/ circuits. pubpp022/ http//2222iincconeet--oo-o html.
context- centtextleaannee lew.ning-nnnnn and-
OpenAl (2023). GPT-4 Technical Report. arxiv: 2303. 2303.08774 [cs.CL].
Pearl, Judea (2009). Causality. Models, Reasoning, and Inference. 2nd ed. Cambridge University
Press. ISBN: 978-0-521-89560-6 DOT: 10. 1017 2010/CC000 CBB////0990000 10011801111.
Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever (2019). "Language Models are Unsupervised Multitask Learners". In.
Ramanujan, Vivek, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Raste- gari (2020). "What's Hidden in a Randomly Weighted Neural Network?" In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 2020. 01191.
13-19, 2020. IEEE, pp. 11890-1 1899. DOI: 10. //doi.org/ Räuker, Tilman, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell (2022). Toward Transparent Al: A Survey on Interpreting the Inner Structures of Deep Neural Networks. URL: https: //axivv.
URL: https:
1199/VVR422600. 2020.0119. 01191.
org/abs/ /2007 13243.
Sanh, Victor, Thomas Wolf, and Alexander M. Rush (2020). "Movement Pruning: Adaptive Sparsity by Fine-Tuning". In: Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Ed. by Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin. URL: https : / proceedings neurips cc / paper / 2020 / hash / (a5aaaaa768a4435093335888 4fw.ee0033aa.444444-4444ee wwwee/ffeeeebbaaattteee Abstract.html
Syed, Aaquib, Can Rager, and Arthur Conmy (2023). Attribution Patching Outperforms Automated Circuit Discovery. arViv: 2310. 10348 [cs.LG].
Tigges, Curt, Oskar John Hollinsworth, Atticus Geiger, and Neel Nanda (2023). Linear Representa- tions of Sentiment in Large Language Models. arXiv: 2310. 15154 [cs. LG].
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin (2017). "Attention is All you Need". In: Advances in Neu- ral Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA. Ed. by Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, pp. 5998-6989988 6008 URL: https: : //t///ppeet pttppeedieeennsseee/e neurips cc paper///////////// / 2017 / /aash/ aataaa-bbsraaacc... .tml.
Veit, Andreas, Michael J. Wilber, and Serge J. Belongie (2016). "Residual Networks Behave Like Ensembles of Relatively Shallow Networks". In: Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. Ed. by Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, pp. 550-558. URL: htttpss//prrcceednggg neuris cttpsper///p////c//eeee www.22000004AAAaaaaaaaaae. Abstratt. html.
Vig, Jesse, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian. Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer. and Stuart Shieber (2020). Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias. URL: https: / arxiv. org/ abs / 2004.. 12265. Wang. Kevin Ro, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt (2023). "Interpretability in the Wild: Circuit for Indirect Object Identification in GPT-2 Small". In: The Eleventh International Conference on Learning Representations. URL: https: /ttp://opppeeieeeeeee
Wang, Ziheng. Jeremy Wohlwend, and Tao Lei (2020). "Structured Pruning of Large Language Models". In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, pp. 6151-6162. DOI: 10. 18653/863//00 001655/v1/2000 v1/20200..022////// emmlp main. 496. URL: https: // aalaaaatnannoooooooc org/ 2erg/2020. emnlp- main.496.
Wei. Jason, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus (2022). Emergent Abilities of Large Language Models. arXiv: 2206. 07682 [cs. CL..
Wu, Zhengxuan, Atticus Geiger, Christopher Potts, and Noah D. Goodman (2023). Interpretability at Scale: Identifying Causal Mechanisms in Alpaca. arviv: 2305. 08809 [cs.CL]
Wu, Zhengxuan, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas leard, Christopher
Potts, and Noah Goodman (2022). "Causal Distillation for Language Models". In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational
13

---

Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics, pp.
DOI: 18653 / v1 / 2022, naacl main 318. URL: https // //aclanthology- org/ naacl
Zhang, Kelly and Samuel Bowman (2018). "I.anguage Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis". In: Proceedings of the 2018
EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Brussels,
Belgium: Association for Computational Linguistics, pp. 359-361.
DOI: 10. 18653/v1/W18
URL: httph: htttp::/aaclantholla
org/W18 htt//11--20 5448.
Zhang, Yu, Peter Tiño, Aleš Leonardis, and Ke Tang (2021). "A survey neural network inter-
pretability". In: 5.5, pp. 726- 742. DOI: 10. 1109 / 2021 202..00221111 3100641.
Zhu, Xunyu, Jian Li, Yong Liu, Can Ma, and Weiping Wang (2023). A Survey Model Compression
for Large Language Models, arXiv: 2308 07633 [c..CL].
14

---

Appendix
A
Table of contents
Introduction
2
The Mechanistic Interpretability Workflow
2.1
Step Select a behavior, dataset, and metric
2.2 Step 2: Divide the neural network into graph of smaller units
2.3 Step Patch model activations isolate the relevant subgraph Explaining the circuit components
2.4
3
Automating circuit discovery (Step 3)
4
Evaluating Subgraph Recovery Algorithms
4.1
Grounded previous work: area under ROC curves
4.2
Stand-alone circuit properties with a test metric
Related work
6
Conclusion
7 Acknowledgements
10
Table of contents
15
Impact statement
16
C Discussion of metrics optimized
17
C.1 Changing the metric in ACDC
17
C.2 Limitations of logit difference
17
C.3 Alternatives to minimizing metric
18
D Details of Subnetwork Probing and Head Importance Score for Pruning
19
D.I Subnetwork Probing
19
D.2 Head Importance Score for Pruning
19
Experimental study of algorithm design
20
E.1 Minimizing the task-specific metric, rather than the KL divergence
E.2 Activation patching with zeros, instead of corrupted input
21
E,3 Node-level ROC curve, rather than edge-level ROC curve
23
F
10I task: details and qualitative evidence
23
15

---

F.1 Further details on the IOI experiments
23
F22 The IOI circuit
24
F33 Limitations of ACDC in recovering the IOI circuit
25
G
Greater-Than task: details and qualitative evidence
26
H Docstring task: details and qualitative evidence
27
H.1 The docstring circuit
27
H.2 Additional docstring experiments
29
I Traer tasks: details and qualitative evidence
30
L11 tracr-xproportion
30
1.2 tracr-reverse
Induction task: details and qualitative evidence
Gendered pronoun completion: qualitative evidence
32
Reset Network Experiments
32
M Automated Circuit Discovery and OR gates
34
Connection to Causal Scrubbing
35
B Impact statement
ACDC was developed to automate the circuit discovery step mechanistic interpretability studies. The primary social impact of this work, if successful, is that neural networks will become more interpretable. ACDC could make neural networks more interpretable via i) removing uninterpretable and insignificant components of models (as reviewed in Section 4.2), ii) assisting practitioners to find subgraphs and form hypotheses for their semantic roles in models (as we found early evidence for in Appendix K) and more speculatively iii) enabling research that finds more interpretable architectures. More generally, better interpretability may allow us to predict emergent properties of models (Nanda et al., 2023), understand and control out-of-distribution behavior (Mu and Andreas, 2021) and identify and fix model errors (Hernandez et al., 2022).
However it is also possible that the biggest impact of better interpretability techniques will be more capable AI systems or possible misuse risk (Brundage et al., 2018). For example, while interpreting neural networks has the steer models towards model bias other harmful effects (Cuadros, Zappella, and Apostoloff, 2022), bad actors could also use interpretability tools to do the opposite: reverse engineering neural networks to steer towards harmful behaviors.
For now, ACDC is tool for researchers and isn't 't mature enough for applications where determining the exact behaviour of a model is societally important. However, the benefits of the adoption of better transparency appear to us to outweigh the externalities of potential negative outcomes (Hendrycks and Mazeika, 2022), as for example transparency plays an important role in both specific (Hubinger, 2020) and portfolio-base (Hendrycks, Mazeika, and Woodside, 2023) approaches to ensuring the safe development of AI systems.
16

---

C Discussion of metrics optimized
In this appendix, we discuss the considerations and experiments that support the formulation of ACDC that we presented in Section 3. We also discuss the metrics and experimental setups for Subnetwork Probing (Appendix D.1) and Head Importance Score for Pruning (Appendix D.2).
In the main text we presented ACDC as an algorithm that minimizes the KL divergence DEL (G| between the model and the subgraphs of the model (Section 3 and Algorithm 1). However, prior mechanistic interpretabilityy projects have reported performance on several different metrics at once (Wang et al., 2023; Nanda et al., 2023). In this Appendix we discuss our findings choosing different metrics in different ways. We explore the advantages and limitations with Algorithm 1 and other approaches. In particular, we have found that optimizing for low KL divergence is the simplest and most robust metric to optimize across different tasks. However, general conclusions about the best methods to use cannot be made because of variability across different tasks, and the large space of design choices practitioners can make.
We found that optimizing for low KL divergence was fairly effective across all tasks we considered, except the Docstring task (Appendix F-J). For example, we were able to exclusively recover heads that are present in the IOI circuit (Figure 1) that have layers of composition sufficient to solve the task, with zero false positives. Additionally, KI. divergence can be applied to any task of next-token prediction it doesn't 't specify any labels associated with outputs (such as logit difference requiring specifyynn which tokens we calculate logit difference between).
C.1 Changing the metric in ACDC
We consider generalizations of ACDC in order to further evaluate our suiching-based dircuit-finding approach. The only line of Algorithm that we will modify is Line 6, the condition
DRI(GG) - DK (G) <T
for the removal of an edge. All modifications to ACDC discussed in this Appendix replace Condition (I) with new condition and do not change any other part of Algorithm 1.
In full generality we let F' be a metric that maps subgraphs reals. We assume throughout this Appendix that subgraphs H, such that F(H)) is smaller, correspond to subgraphs that implement the task to greater extent (i.e we minimize F).5
In practice, we can be more specific about the form that the metric F will always take. We assume that we can always calculate (H x(]), the element-wise result of the metric on individual dataset examples and use F(H) to refer to the metric averaged across the entire dataset (note the similarity of this setup calculation of DxL. in Section 3). The general update rule takes the form
F(H)- F(Hnow) < T.
(2)
which generalizes Equation 1. We discuss further extensions that change Line of the ACDC algorithm in Appendix C.3.
C.2 Limitations of logit difference
The 1OI (Appendix F), Docstring (Appendix H) and Gendered Pronoun Identification work (Ap- pendix K) originally used a variant of logit difference to measure the performance of subgraphs. Logit difference is the difference in logits for correct output compared to baseline incorrect output. Then, these works compare the change from the logit difference of the model to the logit difference of their circuit. However, unlike KL divergence, this metric is not always positive - logit difference for a circuit could be larger or smaller than the logit difference of the model, and so the change in logit difference could be positive or negative. We discuss issues that arise with this approach
'The logit difference and probability difference metrics used by the 101, Greater Than and Docstring tasks were intended to be maximised by the respective researchers (Table 1) so we consider negated versions of these metrics.
17

---

in Appendix C.3, the empirical performance decrease when using logit difference can be found in Figure 5.
C.3 Alternatives to minimizing metric
Two alternatives to minimizing a metric are to 1) match the model's performance on a metric, or 2) only include edges that cause small change in performance. These could be formalised by the following alternatives to Condition 1, where F denotes any metric we could compute from a subgraph:
1. Matching the model's performance: [F(Hee)) - -(G)) - (F(H) - F(G) <.
2. Only including small changes in performance: [F(H F(H) <T
Matching the model's performance (also referred to as faithfulness by Wang et al. (2023)). Since KL divergence is always positive, Alternative is identical to Condition when F is the KL divergence between subgraph's 's outputs and the models' outputs, but for metrics such as logit difference this represents new optimization objective. Empirically we found that matching the model's performance was unstable when we ran ACDC. For example, we ran a modified early version of ACDC that maximized the logit difference in the IOI circuit, and found that through one ACDC run, logit difference of subgraph could be as large as 5.0 and as low as 1.5 during a modified ACDC run. The IOI circuit has logit difference of 3.55, and therefore the subgraph's 's logit difference can be both larger and smaller than the model's logit difference. This issue arises when the subgraph's 's logit difference is larger than the model's. In such cases, ACDC will discard model components that would otherwise include when the subgraph's logit difference is smaller than the model's. This leads to inconsistencies between runs and further dependence the order over which parents are iterated (Section 3).
Paine pootee man jodge]
Legioet
ThE podde DAN DARNE
0.5
D.75
The paappeeeeee on (agu)
0
325
05
3.5
Palte poodliin GAN (rige))
Figure 5: ROC curves on 101 using Abs Logit Diff, Logit Diff and KL. Divergence
Only including small changes in performance. Alternative 2 ignores the value of the metric for the base model and instead focuses on how much the metric changes at each step that ACDC takes.
18

---

However, we found that this was much less effective than KL divergence and even worse than using logit difference on the IOI task (Figure 5), when we used random ablations and recorded ROC curves.
Overall, we found that KI. divergence was the least flawed of all metrics we tried in this project, and think more practitioners should use it given i) the problems with other metries listed, and ii) how empirically it can recover several circuits that were found by researchers using other metrics.
D Details of Subnetwork Probing and Head Importance Score for Pruning
D.1 Subnetwork Probing
There are 3 modifications we made to Subnetwork Probing (Cao, Sanh, and Rush, 2021, SP) in our work. In this Appendix we provide techincal detail and motivation for these modifications:
1. We do not train probe. ACDC does not a linear probe after learning a mask for every component. The component mask can be optimized without the probe,
probe. Cao, Sanh, and Rush (2021) train
just omit the linear probing step.
2. We change the objective of the SP process to match ACDC's. ACDC uses a task- specific metrie, the KL divergence to the model's ' outputs (Algorithm 1). In order to compare the techniques in equivalent settings we use the same metric (be it KL divergence task-specific) in SP. Cao, Sanh, and Rush (2021) use negative log probability loss.
3. We generalize the masking technique so we can replace activations with both zero activations and corrupted activations. Replacing activations with zero activations is useful for pruning (as they improve the efficiency of networks) but are not as commonly used in mechanisitic interpretability (Goldowsky-Dill et al., 2023), so we adapt SP to use corrupted activations. SP learns a mask Z and then sets the weights & of the neural network equal to Φ Z (elementwise multiplication), and locks the attention mask to be binary the end of optimization (Jang, Gu, and Poole, 2017). This means that outputs from attention heads and MLPs in models are scaled closer as mask weights are decreased. To allow comparison with ACDC, we can linearly interpolate between clean activation when the mask weight and a corrupted activation (i.e a component's output on the datapoint =', in the notation of Section 3) when the mask weight is O. We do this by editing activations rather than weights of the model.
Additionally, we used a constant learning rate rather than the learning rate scheduling used in Cao, Sanh, and Rush (2021).
The regularization coefficients À (in the notation of Cao, Sanh, and Rush (2021)) we used in Figure were 0.01, 0.0158, 0.0251, 0.0398, 0.0631, 0.1, 0.158, 0.251, 0.398, 0.631, 2, 3, 4, 5, 6, 7, 8, 9, 10, 30, 50, 70, 90, 110, 130, 150, 170, 190, 210, 230, 250.
The number of edges for subgraphs found with Subnetwork Probing are computed by counting the number of edges between pairs of unmasked nodes.
D.2 Head Importance Score for Pruning
In Section 4-4.2 we compared ACDC with the Head Importance Score for Pruning (Michel, Levy, and Neubig, 2019, HISP). We borrow the author's 's notation throughout this section, (particularly from Section 4.1) and adapt it so that this Appendix can be read after Section 2.
The authors use masking parameters &n for all heads, i. scaling the output of each attention head by Et, similar to the approach in Subnetwork Probing (Appendix D. so that each head Att, output is Att, (Irz) on an input æ4. The authors keep MLPs present in all networks that they prune. We generalize their method so that can be used for any component inside network for which we can take gradients with respect to its output.
The authors define head importance
n
OC(x:)
1 n
aL(x:) AAtt, (x:)
Att. (xx)
gch
"Which is generally equivalent to setting weight parameters equal 0.
19

---

embed
embed
<a00.00 V_
<00.0.v>
<00.6_v>
<00.5
<a0.5_v>
<a0.5.q>
<a1.6_q>
<80.0 00
<00 d>
call B>
<at:: 30
<e1.8_v>
<a1.6_k>
<e15 K>
<a1.5
<al.5 V>
<al 6>
<crezid post>
post>
(a) Corrupted activations
(b) Zero activations
Figure 6: Examples of subgraphs recovered by ACDC on the induction task with different types of activations and threshold = 0.5623. These find the two different induction heads (1.5, 1.6) and previous token head (0.0) as identified by Goldowsky-Die corrupted activations, while (b) shows the result with zero activations.
al. (2023). (a) shows the result with
where the equivalence expressions is the result of the chain rule. We make three changes to this setup allow more direct comparison to ACDC: i) we metric rather than loss, ii) we consider corrupted activations rather than just zero activations and iii) we use the 'head importance" metric for more internal components than merely attention head outputs.
Since our work uses in general uses a metric rather than loss Appendix C, we instead use the derivative of F rather than the derivative of the loss. The HISP authors only consider interpolation between clean and zero activations, so in order to compare with corrupted activations, we can generalize &n to be the interpolation factor between the clean head output Atta (x) (when &n 1) and the corrupted head output Att, (x') (when &n = 0). Finally, this same approach works for any internal differentiable component of the neural network." Therefore study the HISP applied the query, key and value vectors of the model and the MLPs outputs.
In practice, this means that we compute component importance scores
(C(x,) - C(+))) FF(;))
Ic
(4)
aC(x,)
Where C(r;)) is the output of an internal component C of the transformer, which is equivalent to "attribution patching' (Nanda, 2023) up the absolute value sign.
To compute the importance for zero activations, we adjust Equation (4) so it just has a C(x,) term, without the - C2 (zt) term. We also normalize all scores for different layers in Michel, Levy, and Neubig (2019). The identical setup to Equation (4) works for outputs of the query, key and value use query, key and value components for each head within the network, as well as the output all MIPs.
calculations for given head, well as the MLP output of layer. In Section
The number of edges for subgraphs found with HISP is also computed by counting the number of edges between pairs unmasked nodes, like Subnetwork Probing (Appendix D.
Experimental study of algorithm design
This section evaluates design choices for ACDC and SP, by re-doing the experiments in Section 4. We explore two axes of variation.
II theory. In practice, components need be torch, nn. Modules such that we can calculate the gradient of F' with respect the components' outputs.
20

---

Table AUCs for corrupted activations, Random Ablation. (E)=Edge, (N)=Node.
Metric
Task
ACDC(E)
HISP(E)
SP(E)
ACDC(N)
HISP(N)
SP(N)
Doestring
0.982
0.805
0.937
0.950
0.881
0.928
KL
Greaterthan
0,853
0.693
0,806
0.890
0.642
0.827
IOI
0.869
0.789
0.823
0.880
0.668
0.842
Docstring
0.972
0.821
0,942
0.938
0.889
0.941
Greaterthan
0,461
0.706
0.812
0.766
0.631
0.811
Loss
IOI
0.589
0.836
0.707
0.777
0.728
0.797
Tracr- Tracr-Proportion
0.679
0.679
0.525
0.750
0.909
0.818
Tracr-Reverse 0.200
0.577
0.193
0.312
0.750
0.375
Table 3: AUCs for corrupted activations, Zero Ablation. (E)=Edge, (N)=Node.
Metric
Task
ACDC(E)
HISP(E)
SP(E)
ACDC(N)
HISP(N)
SP(N)
Docstring
0.906
0.805
0.428
0.837
0.881
0.420
KL
Greaterthan
0.701
0.693
0.163
0.887
0.642
0.134
101
0.539
0.792
0,486
0.458
0.671
0.605
Docstring
0.929
0.821
0,482
0.825
0.889
0.398
Greaterthan
0.491
0.706
0.639
0.783
0.631
0.522
Loss
1OI
0.447
0.836
0.393
0.424
0.728
0.479
Tracr-Proportion
1.000
0.679
0.829
1.000
0.909
1.000
Tracr-Reverse 1.000
0.577
0.801
1.000
0.750
1.000
Minimizing the task-specific metric, rather than the KL divergence.
Patching activations with zeros, rather than with the result on a corrupted input (interchange intervention).
Looking at node-level TPR and FPR for the ROC curves, rather than edge-level.
The results paint mixed picture of whether ACDC or SP is better overall, but reinforce the choices
we implicitly made the main text. A stand-out result is that ACDC with zero-patching is able to perfectly detect the tracr circuits (Figs. and 10).
numerical summary of the results is in Tables and 3, which display the areas under the ROC curve (AUC) for all the design choices we consider.
E.1 Minimizing the task-specific metric, rather than the KL divergence
We ACDC, SP and HISP with the task-specific metric from Table 1, instead of KI. divergence. The exact modification described Appendix C. The ROC result in Fig. 7. Compared to minimizing KL divergence (Fig. 3), ACDC works better for Doestring, but worse for Greater-Than and IOI, indicating that it is not very robust method.
We prefer using the KL divergence instead of the task-specific metric, because the task-specific metric can be over-optimized (Appendix C). This means that the recovered circuit ends up performing the task more than the original model, and is thus not accurate. We can observe this effect by comparing the task specific metric achieved by the methods in Fig. 8.
E.2 Activation patching with zeros, instead of corrupted input
In the main text experiments that compared using corrupted activations and zero activations (Figure 4), all three methods recovered subgraphs with generally lower loss when doing activation patching with zeros, in both the experiments with the normal model and with permuted weights. It is unclear why methods achieve better results with corruptions that are likely to more destructive. A possible explanation is that there are 'negative' components in models (Appendix F.3) that are detrimental the tasks, and the zero activations are more disruptive to these components. A discussion of how methods could be adjusted to deal with this difficulty can be found in Alternative 2 in Appendix C.
21

---

tracr-leverse
tracr-xproportion
HISP
*
posiove (edges)
0.75
T,
0.5
0.5
10a
Docstring
Greater- Than
10
0.25
10
10
10-4
10-3
0.25
0.5
10-4
False positive rate (edges)
Figure 7: Eige-wise ROC curves generated by minimizing the task-specific metric in Table 1, rather than KL divergence.
Thos
codao
Quadity
17
174
13
180
2000
1000 13
Figure 8: Optimizing the task-specific metric of the subject model, trained and reset networks. For each recovered circuit, we plot its task-specific metric (Table 1) against its number of edges. The networks metries don't change much with the number of edges, which is good. We found that for IOI that extremely large logit differences could be achieved (over 15) but this didn't happen when the network had large number edges.
greaterthan
docstring
HSP
positive (edges)
0.75
%
0.5
10a
10
0.25
10
10-2
10
10
10-4
0.25
0.75
0.5
False positive rate (edges)
Figure 9: Edge-wise ROC curves generated by minimizing the KL divergence, but using zero activations.
22

---

tracr-reverse
tracc-xproportion
HESP
(edges)
0.75
T,
0.5
10a
0.5
10a
Docstring
Greater-Than
10-1
2
10
0.5
10
10
10-6
0.25
0.75
0.5
False positive rate (edges)
Figure 10: Edge-witg wise ROC curves generated by minimizing the task-specific metric in Table 1, using zero activations.
E.3 Node-level ROC curve, rather than edge-level ROC curve
We compute the FPR and TPR classifying whether a node belongs the circuit. We consider this alternative task because SP and HISP operate at the node-level, whereas ACDC operates at the edge-level, so this fairer HISP and SP. The results are broadly similar to edge-level ROCs, and are described in Figs. 11 and 12 and Tables 2 and 3.
101
greaterthan
docstring
HSF
positive (nodes)
ACDC
0.75
A,
%
10
0.5
10a
10a
10
10
10
10
0.25
0.5
0.75
0.5
0.5
1
10-6
False positive rate (nodes)
Figure 11: Node-level ROC when optimizing the KI. divergence, with corrupted activations.
F
IOI task: details and qualitative evidence
F1 Further details on the IOI experiments
In the ACDC run in Figure we used a threshold of T = 0.0575. We also removed all edges which did not lie directed path from the input (which is equivalent in computation since use corrupted activations). Our library now only supports splitting query, key and input, rather than merely looking at the connections between heads. Additionally, For ease of visualization, in the diagram on the left of Figure we removed all edges between grey nodes more than 2 layers apart, and 90% of the edges between grey and red nodes.
Our IOI experiments were conducted with a dataset of N text examples from one template the IOI paper used When John and Mary went to the store. Mary gave bottle of milk to'). The
23

---

tracr-reverse
tracc-xproportonn
HEPP
(nodes)
0.75
T,
0.5
10
0.5
10a
Docstring
Greater-Than
10-1
2
10
3
0.5
10
10-
10-6
0.25
0.5
0.75
0.5
False positive rate (nodes)
Figure 12: Node-level ROC when optimizing the task-specific metric in Table 1, with corrupted activations.
Legend
When 10 Mary and
Key/ / Volue
Query
Class of Heads LayerHead
Output
Previous Heads
S1+1 Ieett
to
the
Negative Name Mover Heads
$2
John
Duplicate Token Heads
Induction Heads 59)
107 11l
(0.0)
gave
Name Mover Heads 99 96 100
drink
S S-Inhibition Heads 73 79 86 8.0
END
Backup Mover Heads 917 102 ICe
Figure 13: The IOI circuit (Figure from Wang et al. (2023)). We include edges between all pairs of heads connected here, through the Q the K+V indicated. We also include all connections between MLPs and these heads, the inputs and outputs. See https: /gi thub. com/ Artrur Automatic- wtw.uaaiiiissooeerooooo/aaaaalla wttta//////iiiiislsiaaaaaaactalt py#L205. The full circuit is in Fig. 14.
corrupted dataset was examples from the ABC dataset (Wang et al., 2023) for example "When Alice and Bob went to the store, Charlie gave bottle milk to'.
In the 10I experiment in Figure 1, we did not split the computational graph into the query, key and value calculations for each head. This enabled the ACDC run complete A100 GPU. However, the larger experiments that kept > 10% of the edges of the original edges in the computational graph sometimes took several hours. On one hand we don't expect these cases to be very important for circuit discovery, but they make up the majority of the points of the pareto frontier of curves in this paper.
minutes on an NVIDIA
F.2 The IOI circuit
Wang et al. (2023) find a circuit 'in the wild" in GPT-2 small (Radford et al., 2019). The circuit identifies indirect objects (see for example Table 1) by using several classes of attention heads. this subsection analyze how successful ACDC's circuit recovery (Figure 1) is. All nine heads found in Figure belong to the IOI circuit, which is subset 26 heads out of total of 144 heads in GPT-2 small. Additionally, these 9 heads include heads from three different classes (Previous Token Heads,
24

---

Aad 10x
<T
<00 Tx
.02. g
bx
400
+258
<ab- Jo
AaR
rad Eo
<073
<a7.0 g+
<07.3
k
Rv
455
+h5 B.
On
22
2_4
<000
7
10x
1111 2
Sar0 L.te
10
<all
<a10 To
12
10
xa102.00 Vt
420 A
<a27.
caff 10
vat0 50w
soft Se
<c 10 Lx>>
<0100
<2 RL
COSAL posts
Our low-level implementation of the IOI circuit (Wang et al., 2023), in terms of heads split by query, key, value; and MLPs. It has 1041 edges. For edge between groups A, B in Fig. 13, we connect each member of group A with each member of group B. The colors of groups in this figure correspond to the group colors in Fig. 13.
Figure 14:
S-Inhibition Heads and Name Mover Heads) and are sufficient to complete the IOI task, showing that ACDC indeed can recover circuits rather than just subgraphs.
For our ROC plots, we considered the computational graph of IOI described in Figure 13.
The ground-truth circuit gets logit difference of 3.24 compared to the model's 4.11 logit difference. has a KI. divergence of 0.44 from the original model.
F33 Limitations of ACDC in recovering the IOI circuit
The main figure from the paper Figure shares several features with circuits recovered with similar thresholds, even when logit difference rather than KI. divergence is minimized. The figure does not include heads from all the head classes that Wang al. (2023) found, as it does not include the Negative Name Mover Heads or the Previous Token Heads. In Figure 15 we run ACDC with a lower threshold and find that it does recover Previous Token Heads and Negative Name Mover Heads, but also many other heads not documented in the IOI paper. This is case where KL divergence performs better than logit difference maximisation (which does not find Negative Name Movers at any threshold), but still is far from optimal (many extraneous heads are found). Ideally automated circuit discovery algorithms would find negative components even at higher thresholds, and hence we invite future empirical and theoretical work to understand negative components and build interpretability algorithms capable of finding negative components. An early case study gaining wide understanding of negative head in GPT-2 Small can be found in McDougall et al. (2023).
25

---

69.6
a10.0
e9.9
*9.7.v
m10
m9
al1.11_
a9.7
a11.10_a
e11.10_k
=10.7_9
e10.7_k
m11
e11.10
a11.6_v
99.2_v
a10.7
e10.9
a11.6
post
Figure 15: A subset of the 443/32923 edges of GPT-2 Small that ACDC recovered when optimizing for KL. divergence at threshold r = 0.00398. This subset includes edges between Negative Heads (10.7 and 11.1 A number of heads not found by the IOI work (9.2 and 11.11 for example) were also found.
logits
m8, m9, m10, m11
Boost logits for yy YY
Create logit spike YY
a5.h5, a6.h1, a6.h9, a7.h10, a8.h11, a9.h1
K, V
a0.h3, a0.h5
a0.h1
m0, m2,
Read in YY
input: "The war lasted from the year 17YY to the year 17"
(a) The ground-truth circuit from Hanna, Liu, and Variengien (2023, Figure 13).
(b) Sample Greater-Than graph re- covered by ACDC
Figure 16: The ground-truth Greater- Than circuit (16a) and a circuit that ACDC recovers (16b).
G Creater-Than task: details and qualitative evidence
We random sample of 100 datapoints from the dataset provided by Hanna, Liu, and Variengien (2023).
We use the circuit from Figure 13 from their paper, including connections between MLPs that are in the same group (e.g MLPs 8, 9, 10 and 11) but not including connections connections between attention heads in the same group. We also include all Q and K and V connections between attention heads present. Their circuit includes all earlier layer connections to the queries of the mid-layer attention heads that cause logit spike (Figure 16a). This would account for more than half of the edges the circuit were we to include all such edges that compute these query vectors, and hence we compromised by just adding all early layer MLPs as connections these query vectors. Full details on the circuit can be found in our codebase and in Fig. 17. This circuit gets a probability difference score of 72% a subset of their dataset for which GPT-2 Small had 84% probability difference and KL divergence from the original model of 0.078.
'nttps://github. wttpttthu..com/rreeeeee htap:///oooootttooooooooCrrtttte greaterthan/utae pttp//tttllllll11111111 py#1231
26

---

ambed
x3003.Jk
100.1.30
<a0.3
<a0.
<20.66
<a0.1>
<a0.3>
<a0. So
<65.5 K>
<a6.9.
<
<m0>
1_0.1_k>
<a6.8s
<ad.9 Kt
<GR
<m
<at5.tt
<a7.10 v>
<a6 1_t>>
1_pp
<36.9_v>
<a8.1
<a5 sp
<m
<26. 1_
<a7.10
10
<a8.11
vc>
10__a>
<a8
<aB 1_8.1_
cm
<m3b
1_1>>
<m11>
cB5.5.v v>
<a7.10_k>
cresid post>
Figure 17: Our low-level implementation of the Greater-Than circuit (Hanna, Liu, and Variengien, 2023), in terms heads split by query, key, value; and MLPs. It has 262 edges. For edge between groups A, B in Fig. 16a, connect each member of group A with each member of group B.
example subgraph ACDC recovered, including path through MLP mid-layer attention heads and late-layer MLPs, is shown in Figure 16b. This run used the Greater-Than probability difference metric and threshold of 0.01585 and recovered the results in the abstract.
H Docstring task: details and qualitative evidence
H.1 The doestring circuit
Heimersheim and Janiak (2023) find a circuit in a small language model that is responsible for completing Python docstrings. The model is 4-layer attentiononnly transformer model, trained on natural language and Python code. This circuit controls which variable name the model predicts in each docstring line, e.g for the prompt in Table it chooses shape over the other variable names files, obj, state, size, or option.
The circuit is specified on the level of attention heads, consisting of 8 main heads (0.2, 0.4, 0.5, 1.4, 2.0, 3.0, and 3.6) that compose over four layers, although only makes use of three levels of composition. It consists of 37 edges between inputs, output, and the attention heads, we show in Figure 18a. We discuss below why we exclude 0.2 and 0.4.
We apply the ACDC algorithm (Section 2) to this dataset, using the prompts from Heimersheim and Janiak (2023) found in their accompanying Colab notebook." 9 For our corrupted dataset we use their random_ranne Licifo__randee random dataset which randomizes both the variable names in the function definition as well as in the docstring of prompts.
ACDC generates the subgraph shown in Figure 18b. We now compare this to the original 8-head circuit from Heimersheim and Janiak (2023), which the most specific circuit evaluated in that work. We refer to this circuit as the "manual" circuit to distinguish it from the ground truth, which
Available at http::///coabb.eeearchhc http://www......................
of 8th April 2023
27

---

wil
+300
aa
Bo
(a) Canonical Docstring circuit (37 edges)
(b) ACDC circuit (KL-divergence, 7 = 0.095)
Figure 18: Our implementation of the Docstring circuit (Heimersheim and Janiak, 2023), compared to an ACDC-generated circuit.
Full model
ACDC KL.
ACDC KL.
ACDC LD
Manual
Ground-
0.005
== 0.095
T = 0.067
8 heads, all connections
truth circuit
Metric
(Fig. 18b)
(Fig. 19)
(Fig. 18a)
KL-divergence
0.33
1.2
0.67
0.83
1.1
Mean logit diff.
0.48
0.58
-1.7
0.32
-0.62
-1.6
Num. of edges
1377
258
34
98
464
37
Table Comparing our ACDC docstring results the ground-truit from Heimersheim and Janiak (2023) using their metrics. We compare (from left to right) the full model, the subgraph from ACDC
0.067), as runs optimizing for KI. divergence (r 0.005 and 0.095) and logit difference including all well as the two subgraphs made manually from Heimersheim and Janiak (2023): One connections between the given attention heads, and one using only the given circuit. The metrics used are KL divergence between full-circuit outputs and resamble-ablated output (lower is better), mean logit difference between correct and wrong completions (higher is better), and the number of edges in the circuit (lower is better).
includes the edge connections that the authors speculated were most important but did not evaluate due to lack of software for edge-editing. We find (a) overlapping heads, (b) heads found by ACDC only, and (c) heads found in the manual interpretation only. In the first class (a) we find heads 0.5, 1.4, 2.0, 3.0, and 3.6. these manually identified heads are recovered by ACDC. class (b) we find head 1.0 which the authors later add to their circuit to improve performance; ACDC shows for the first time where this head belongs in the circuit. In class (c) we find heads 0.2, 0.4 and 1.2. However, the first two of these not actually relevant under the docstring distribution and only added by the authors manually. Head 1.2 considered a non-essential but supporting head by the authors and not identified by ACDC the chosen threshold of 7 0.095 (for KL divergence). This might be because 1.2 is less important than the other heads, and indeed we recover this head larger subgraphs (such as the subgraph in Figure 19).
compare the numerical results between the ACDC circuits and the circuit described in Heimer- sheim and Janiak (2023) in Table 4. In addition to the = 0.095 run (Figure 18b) we perform run with lower KI. divergence threshold of 0.005 recovering a larger circuit (258 edges) containing also head 1.2 that was missing earlier.
28

---

A01
A2
A0
#1.1
60.0
A12
82.0
x2
A80,
AB
Figure 19: ACDC-found subgraph for docstring task minimizing logit difference (r = 0.066) instead of KL divergence.
Since Heimersheim and Janiak (2023) use logit difference as their metric, we add an ACDC run that optimizes logit difference rather than KL divergence (see Appendix C for details on this adjustment) with threshold T 0.067. This circuit (Figure 19) recovers the relevant (including 1.2) as well. 10 It is even more specific, containing 93% less edges than the full circuit. This is also 79% less edges than the head-basee based circuit from Heimersheim and Janiak (2023) while achieving better score all metrics.
interppttnter heads
Note that there are two versions of the manual circuit we consider. There (i) the set of 8 heads given in Heimersheim and Janiak (2023) that the authors test with simple methods (not specifying edges), and (ii) the circuit of 39 edges as suggested by the authors that they were not able to test due to not having software to implement editable transformer computational graphs in PyTorch We reconstruct this circuit, shown in Figure 18a, from their descriptions and perform tests (Table 4).
In case (i) the ACDC run (threshold T 0.005) achieves better performance in both metrics, Logit Difference and KL divergence, while being more specific (258 edges) when compared the set of heads found by Heimersheim and Jamiak (2023). In the more specific case (ii) the ACDC run (with threshold r = 0.095) closely matches the manual interpretation. with a very similar circuit recovered (Figure 18). The ACDC run is slightly more specific but has slightly worse KL divergence and Logit Difference.
limitation worth noting is that we applied ACDC to a computational graph of attention heads and their query, key and value computational nodes, while Heimersheim and Janiak (2023) considered the attention heads outputs into every token position separately. This allowed them to distinguish two functions fulfilled by the same attention head (layer I, head 4) different positions, which cannot be inferred from the output of ACDC alone at any level of abstraction (Section 2.2) we studied in this work. We make this choice for performance reasons (the long sequence length would have made the experiments significantly slower) but this is not a fundamental limitation. In Appendix K we use ACDC to isolate the effects of individual positions a different task.
H.2 Additional docstring experiments
Logit difference metric: To compare ACDC more closely with the docstring work (Heimersheim and Janiak, 2023), we added an ACDC run with the objective to maximize the logit difference metric. We used a threshold of r = 0.067 and found the subgraph shown in Figure 19, We found that ACDC performed better than SP and HISP when using the logit difference metric (Figure 7).
Unlike in the case of induction (Section 4.2), we found that using zero activations rather than random (corrupted) activations, lead to far worse results. For example, with = 0.067
Zero activations:
Again, not considering heads 0.2 and 0.4 which are not actually relevant under the docstring distribution.
29

---

(the same threshold that generated Figure 19 except with zero activations) we get circuit with 177 edges (Figure 19 has 98), as well as KL divergence of 3.35 and logit difference 22895. All these metrics are worse than the subgraphs generated with corrupted activations (Table 4).
I Tracr tasks: details and qualitative evidence
In this Appendix we discuss the two tracr tasks we studied in Section 4, as well as additional experiments that studied ACDC when applied at a neuron level.
We used a transformer identical to the one studied in Lindner et al. (2023), and refer to that work for details on the tract-xproportion task (called the frac_prevs task in their paper). We also studied the tracr-reverse task, described in the tracr Github repository."
We make one modification the traditional ACDC setup. We set the positional embeddings equal to randomized positional embeddings in the corrupted datapoints - otherwise, we don't recover any of the circuit components that depend only positional embeddings (and not token embeddings). We describe the two tasks that we studied in the main text and describe futher results that broke these computational graphs down into neurons.
L1 tracr-xproportion
We used the proportion task from the tracr main text, and used as metric the 1.2 distance between the correct list of proportions and the recovered list of proportions. For the corrupted dataset, we let (==) be random permuation of (x;))=z with no fixed points.
When ran ACDC the neuron level, shown in Figure 20b, there are no extra nodes present that were not used by this tracr model. In fact, this computational graph visualization produced by ACDC more compact than the complete view of the states of the residual stream which is illustrated in Figure 20a (from Lindner et al. (2023)). In this case, the transformer is small enough for practitioner to study each individual state in the forward pass. However, for larger transformers this would be intractable, which necessitates the use of different interpretability tools such as ACDC.
See Fig. 21a for the full circuit (without decomposition into residual stream dimensions).
tal, beds
Inpur
Attn
MLP1
Attn
MIP
pall
tokens)
tokens: hes
tekens:
tokens:
frac
f2ss prove
AZ PLPS
indicus:
indices:
indices res:
Prac
indicos:
indices:
box
The magnitude of each layer's activations. Repro- duced from Lindner et al. (2023) under CC-BY.
(b) ACDC visualization.
Figure 20: Two visualizations of how a tracr-compiled transformer completes the frac_prevs task. The ACDC circuit specific to the individual neurons and residual stream components. This is more fine-grained than the ground truth we use throughout the work. This experiment was coded in rust_circuit and is not reproducible using the Transformer Lens code yet.
I.2 tracr-reverse
To test ACDC on more than one example of tracr program, we also used the 3-layer transformer that can reverse lists (the tracr-reverse task). Once more, the outputs of the transformer are not
URL: https: httpe///gtthtbbcccc/papppaa/eeet file: README. md
30

---

dakion <akkon_smbac ombacse
<a0.0
poosembedee>
<pss_embedoo
<loker_embeddo>
<a1.0 k>
<at.0 1.O. q>
<md>
<m/2>
<a1.0 0 V>
1.3.0.4p
<a1.0>
cresid_post>
<neskt posts
(a) tract-xproportion canonical circuit (10 edges)
(b) tracr-reverse canonical circuit (15 edges)
Figure 21: The canonical circuits for finding the proportion of 'x' in the input and reversing lists. ACDC recovers these perfectly using zero activations (Table and Figs. and 10).
distributions in this case they are new lists. We calculate the L2 distance between the one-hot vectors for the recovered list and the reversed list. For the corrupted dataset, again let (x() be a random permuation of (x,) with no fixed points. Again, at the neuron level a perfect graph is recovered, with the minimal components required reverse lists (Figure 21b).
Induction task: details and qualitative evidence
In Section 4.2 we use 40 sequences of 300 tokens from filtered validation set of Open WebText (Gokaslan et al., 2019). We filter the validation examples that they contain examples of induction - subsequences of the form "A, ...... A, B", where A and B are distinct tokens. We only measure KI. divergence for the model's predictions of the second B tokens all examples of the subsequences A, B A, B.
We both zero activations and corrupted activations compare ACDC and the other methods. To use ACDC with zero activations, we apply one change to the procedure described in Section 3: instead of setting activations of edges dataset, we set their value equal to O. We describe how we adapt the methods from Section 4.1 to be used with both zero activations and corrupted activations in Appendix D. for SP and Appendix D.2 for HISP.
present in the subgraph the activations on corrupted
Our induction experiments trained on Open WebText (Gokaslan et al., 2019). The model is available in the Transformerl (Nanda, 2022) library. 12 We follow Appendix C of Goldowsky-Dil al. (2023) for the construction of the dataset of induction examples.
performed 2-layer, 8-head-per-laye attention only transformer
The computational graph has total of 305 edges, and in Figure 4 we only show subgraphs with at most 120 edges.
The model can be loaded with transf ormer ormerrleesseeeeeeeeeeeeeeeeeeeee lens. HookedTransformerr ffrm wtw.:/orwww..rroo.rorrrrarredddd (model name "reewood__ soodd__ttt____ attn__1 __" center wri weig_weeitts False, center unembed conter_unembed for the repository version of the source code of 23rd May 2023)
False) (at least
31

---

When iterating over the parents of given node (Line 4 in Algorithm 1), we found that iterating in increasing order of the head index was important achieve better results in Figure 4. Similar to all experiments in the work, we iterate in decreasing order of layers, so overall we iterate over head 1.0, 1.I, then 1.7, then 0.0, 0.I,
An example of a circuit found in the process is given in Figure 6.
K Gendered pronoun completion: qualitative evidence
Mathwin et al. (2023) aim to isolate the subgraph of GPT-2 small responsible for correctly gendered pronouns in GPT-2 small. They do this by studying prompts such as "So Dave is a really great friend, isn't" which are predicted to finish with he". For that they used ACDC. This presents an example of a novel research project based on ACDC. The result of applying the ACDC algorithm (threshold T 0.05) is shown Figure 22.
The computational subgraphs generated by ACDC on the gendered pronoun completion task show that MLP computations more important than attention head computations in this task than in the IOI task (Appendix F.2). Early, middle and late layer MLPs have important roles in the subgraph. For example, MLPs and 5 are the important components the name position (which must be used to identify the correct gender) as they have multiple incident edges: the MLP 7 at the is" position has the most incoming connections of any node in the graph, and the late layer MLPs 10 and 11 have the largest direct effect on the output. MLP 7's importance at the " iss position is an example of a discovery that could not have been made with simpler interpretability tools such saliency maps. This was early evidence of the summarization motif (Tigges et al., 2023).
ACDC's output shows that the important internal information flow for predicting the correct gender has three steps. Firstly, Layer attention head 0.4 and MLPO use the name embedding which they pass (through intermediary MLPs) via key- and value-composition (Elhage et al., 2021) to attention heads 4.3 and 6.0. Secondly, heads 4.3 and 6.0 attend to the name position to compose with 0.4 and MLPO. Finally, through value-composition with attention heads 6.0 and 4.3 (via MLP7), the outputs of 10.9 and 9.7 output the expected gendered completion to the output node. Mathwin et al. (2023) then verified that indeed in normal forward pass of GPT-2 Small, 0.4 has an attention pattern to itself at the name token, attention heads 4.3 and 6.0 attend to the previous name token, and 10.9 and 9.7 attend the is' token. They also perform path patching experiments on intermediate nodes to provide further evidence of the importance of the pathway through the 'i' token.
We used the dataset of N 100 examples from Mathwin et al. (2023). The corrupted dataset was set of prompts with a similar structure to the sentence "TTha person is a really great friend, isn't "s", following the authors' approach.
We defined a computational graph that featured nodes at the specificity of attention heads split by query, key and value vectors, and further split by token position (where the tokens are present in the nodes in Figure 22). From the input sentence 'So Sarah is a really nice person, isn't', we chose to add nodes representing the model internal operations at the tokens ""Srrah"," "_is". : "_persson" "_isn" and "*'", while other positions were grouped together as in Mathwin et al. (2023). The resulting subgraph can be found in Figure 22.
L Reset Network Experiments
Our reset network experiment setup is motivated by the concern that interpretability explanations may not accurately represent the reasoning process behind models' predictions (Jacovi and Goldberg, 2020). This is particularly relevant to work on subnetworks as empirically some subnetworks in models with randomized weights do not accurately represent such reasoning (Ramanujan et al., 2020).
To this end, we study the task- specifcc metrics on models with permuted weights (which we call reset networks (Zhang and Bowman, 2018; Cao, Sanh, and Rush, 2021)) and verify that the circuit recovery algorithms perform worse on these models that not have underlying algorithms. Specifically, we create the reset network by permuting the head dimension of each layer's Q. K, V matrices, and each MLP's bias term. This disrupts the functionality of the subject model, without changing many facts about the distribution of the activations (c.g. the average magnitude at every layer). In our experiment
32

---

tok_ eemees
Inamm_
[namm)]
h44 [_fame)
mo__rne [nn_nn
mQ__
m3 [namme]
#1 h.v.v.0vv v_Jnmme]
#1 h.. [[]
mil [[[amme]]
a1h10
#1.h9
m4_
m1 [n__[nam]]]
m5_ mS_[namme)
m1
m3
m5
of o
aB 1..h [n0.k_[namm)
e4 h3.k h3.kk_nnmme)
eB ho.v ho.v_iname]
a..hi.v_inamml
a7.h
c10.m9.v_Iname]
m7_Dia
a10.h9.v_Iian
[17.v_[namm)
vj(ls
v_(w)
v_33
mi1_rt
m10.r'd
m10
Inallnn
Figure 22: Gendered pronoun completion circuit found by ACDC.
33

---

in Figure the metric used by each algorithm is the KL divergence between the original trained network (with no edges patched), and the irtivation-patcher reset network.
The reset network does not exhibit the original network's 's behavior, and thus should not be possible explain the presence of the behavior. This is strong measure of the negation of Q2: if the algorithm is able to find a circuit that performs the behavior on a network that does not exhibit the behavior, then will likely hallucinate circuit components normal circumstances.
M Automated Circuit Discovery and OR gates
In this appendix we discuss an existing limitation of the three circuit discovery methods we introduced in the main text: the methods we study do not identify both inputs to 'OR gates' inside neural networks.
OR gates arise in Neural networks from non-linearities in models. For example, if xr y {0, then ReI.U(1 - x y) is an OR gate on the two binary inputs y. To study toy transformer model with an OR gate. we take a 1-Layer transformer model two heads per layer, ReLU activations and model dimension 1. If both heads output into the residual stream, this model implements an OR gate. Our dataset (Section 2) is then equal a single prompt where the heads output 1, and we use zero activations to test whether the circuit discovery methods can find the two inputs to the OR gate.
<embed>
d)
<a0.0 k>
<a0.0_v>
a)
b)
C)
<a0.1>
<a0.0>
<a0.1>
<embed>
<a0.0>
<m0>
<m0>
<m0>
<m0>
<resid_pest>
<resid_post
<resid_post>
<resid presid_post>
Figure 23: OR gate recovery. a) the ground truth, b) ACDC, c) HISP, d) SP.
The results can be found in Figure 23. The ground truth in a) is our toy model an OR gate, where MLPO performs OR the bias terms of a0.0 and a.. These the only edges that should be included. b) ACDC only recovers one OR gate input. This because the iterative algorithm prunes the first input to the OR gate it iterates over and then keeps the other. e) HISP recovers neither OR gate input (and also recovers the unnecessary input node). d) SP recovers only one OR gate input, and several additional nodes. SP and HISP found extra edges since they include the input node by default. HISP doesn't t include either attention head showing the limitations of gradients in this idealized case. We are unsure why SP finds the a0.0's key and value inputs. This shows the limitations of nose-based methods for finding circuits, though ACDC is also limited. Of course, many easy fixes exist to this problem, but the priority of future work should be explain in-the-wild language models, where less clear which algorithmic improvements will be most helpful. For example, follow up work found that using gradient approximations on the edges of computational graph was very effective (Syed, Rager, and Conmy, 2023), despite not being more effective at finding OR gates.
1hhis construction similar to AND gate construction from Gurnee (2023) Appendix A.12.
"For specific details of the TransformerLens (Nanda, 2022) implementation, see https: /ettps://gittt http///gtttttt com/ Ar ArthurConmy/ Automatic Circuit- -ww.cuttiisscoeerryrooooooeo- Dissooverr///ooommmm////////tggd
p11115.
34

---

N Connection to Causal Scrubbing
This work was inspired by work on Causal Scrubbing (Chan al., 2022). The scope of each algorithm is quite different, however.
Causal Scrubbing (CaSe; Chan al., 2022) aims primarily at hypothesis testing. It allows for detailed examination of specified model components, facilitating the validation of a preconceived hypothesis.
ACDC, on the other hand, operates at broader scale by scanning over model components to generate hypotheses, each of which is tested using CaSc. ACDC chooses to remove an edge according to the CaSc criterion, the new hypothesis isn't much worse.
Why is testing every hypothesis with Causal Scrubbing not incredibly inefficient? The reason that ACDC only considers small class of CaSc hypotheses, where paths through the model either matter, or don't 't matter. effect, the CaSc hypotheses considered by ACDC don't 't allow any interchanges the node "matters" (by having a unique value for each possible input), and the nodes that don't 't matter are each replaced by the same second data point.
Both methods currently face computational inefficiencies, albeit for different reasons and at different scales. Causal Scrubbing is impractical for somewhat complicated causal hypotheses because of treeification: there are exponentially many paths through branching DAG, and each needs part of a forward pass. For ACDC, each hypothesis is quick test, but the number of edges search over can be quite large, so it still takes a while to search over circuits for realistic models. This problem partially addressed by using gradient-based approcahes like Attribution Patching (Syed, Rager, and Conmy, 2023) or perhaps an edge-based version of Subnetwork Probing (Cao, Sanh, and Rush, 2021).
In summary. ACDC and Causal Scrubbing are complementary tools in the analysis pipeline. ACDC can do an initial coarse search over hypotheses and, while it is built Causal Scrubbing, only considers small class of hypotheses so it stays relatively efficient. In contrast, Causal Scrubbing offers methodical way test hypotheses, which can also specify the information represented in each node.
35


Reviews:
[{"review_id": "8XBYlQTpJb", "reviewer": "Reviewer_6BEj", "summary": "The paper is a fresh take on mechanistic interpretability, focusing on the automation of the interpretability task, demonstrating it on attention based models. The method, ACDC, finds the pareto optimal subgraphs of the network, thus bringing down the number of connections to highlight the role each unit plays in the predictions of the model. The authors present extensive experiments and analysis on various tasks, including the interesting IOI and Greater-Than, with further insight on the performance trends in the supplementary work. On providing an idea of the workflow in the domain, they introduce their algorithm focusing on iteratively chipping away at the computational graph, creating a sparse graph while retaining the good performance on the specified task and metric. They do mention the limiting prowess of the algorithm, that is the handicap of not identifying all the abstract units and sensitivity to hyper-params, while also not being automated end to end for a complete interpretability framework setup. Overall the algorithm is a good starting point to build on automating scalable interpretability. ", "questions": "- See weaknesses.\n- While Figure 16 (Typo in the Greater Than graph with a $\\mu$ perhaps?) and 17 differ significantly on the performance on the reset network and the trained network, for the SP and HISP networks, the ACDC points are invisible in Figure 16 subplots for Tracr (Reverse), Tracr (Proportion), Greater Than, I’m assuming the scale of the graph points to them being in this range, but perhaps consider replotting with the ACDC method points being visible.\n- Line 313: As the authors correctly point out, the issue of missing certain units must be investigated and probed further. Highlighting taskwise the missing sections and what they could correspond to could help draw patterns on the way ACDC works and alleviate weaknesses in further work to build on top of this. \n- Likewise the final hyperparameters used and ablations on the variance in the performance as one tweaks these hyperparams could be useful to note the sensitivity and brittleness of the algorithm. For a $\\delta$ change in a hyperparam how does the method performance vary? \n\nI am open to increasing the score on further clarifying discussion with fellow reviewers and authors.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Neat presentation, with well organized sections and appropriate background information.\n- Clean and crisp algorithm, that is easy to understand and yet highly effective in its job.\n- Experiments are well defined, tasks, metrics and objectives are comprehensively made clear.\n- Comprehensive ablations and detailed supplementary work to support the claims made and make sense of the findings, special kudos for the highly effective plots and task-wise analysis.\n- Frank and clear understanding of the shortcomings and strengths of the algorithm make this method a clear one to build on top of, prompting further interesting work in a highly important domain.\n\nP.S. A playful modification to the title could be to make it AC⚡DC to make it a play on the famous music band.", "weaknesses": "- Distinction from previous work is not very clear. Especially as compared to HISP, the top k heads are analogous to retaining only the influential heads in ACDC. While I agree that the algorithm in itself is distinct, the inspiration from previous works should be made clear. I would suggest adding a subsection to compare the circuits derived from the two methods for further comparison in the approaches (akin to Figure 6), and explicitly highlighting the specific fail cases for HISP that ACDC works for could be a great insight.\n- While section 5 is helpful in answering questions about the effectiveness of ACDC, Line 253 as pointed out by the authors raises questions. In my opinion more work in establishing KLD as a faithful proxy must be undertaken.\n- While Appendix K and H are helpful to establish the “correctness” of the subgraph recovered, unless one knows the true minimal graph, it is difficult to compare and decide which of the methods is “more correct”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your well-considered review! We are very happy that you appreciate the work we put into exploring the shortcomings and advantages of each algorithm. Thank you also for the insightful questions, and please let us know if you have any more.\n\n**Comparison to HISP.** Thank you for bringing this up! HISP is a heuristic pruning algorithm. ACDC can be seen as a pruning algorithm too. However, its goal is very different from most of the pruning literature, which we mention in related work. For example, ACDC operates on connections between heads (edges) instead of heads (nodes), which sometimes makes the forward pass run more slowly and require more memory, a very undesirable thing in NN pruning. However, as an advantage ACDC can recover the path-specific effects of edges – in Figure 6, as you mention, ACDC recovers the effect that Layer 0 heads have on the output indirectly: through their input connections to Layer 1. Both other algorithms we compared to cannot be this specific. However, further emphasizing the conceptual similarities and differences is worthwhile and we will do this in Section 3 and Appendix H.\n\n**Faithfulness of KL divergence.** As far as we understand, your concern about the faithfulness of KL in Section 4 of our work asks two questions:\n- (1) Is KL a good proxy for the task-specific metric when finding circuits?\n- (2) Does optimizing KL divergence reliably yield the circuit that’s implementing the behavior in the NN? For example, optimizing KL has to yield a circuit that contains both ‘positive’ and ‘negative’ contributions to the log-likelihood.\n\nWe do not assume either of these things in the paper, and we do examine them empirically.\n\nWe believe we have pretty firmly established (1) for the tasks we tried. Sadly we did not have space to include this in the response PDF, but we have plots that measure *task-specific loss on held-out test data*, on experimental runs that use the *KL as a target*. The result looks pretty much like Figures 16-19 and the updated Figure 4: the held-out task-specific loss improves monotonically with higher numbers of edges. Modifying the $\\tau,\\lambda$,% parameter monotonically decreases loss and increases number of edges. Thus KL is a good proxy for task-specific loss, and is not overfitting the subgraph.\n\nWe have not established (2) very well, but it was a major focus of the experiments of this paper. We are bottlenecked by the lack of good measurements of hypothesis correctness in the field of interpretability, which is exactly your third objection: mostly we don’t know the true minimal graph! We attempted to establish (2) with the ROC plots (e.g. the updated Fig. 3) and the reset network experiments (Figs. 16-19). The ROCs depend on the correctness of previous work, and the other plots are weak evidence, so we still don’t know whether (2) is true. We will highlight this problem further when discussing the metrics at the beginning of Section 4.\n\nThat said, now that there is a clear way to automate mechanistic interpretability, we expect ACDC to be replaced by faster and better algorithms soon. KL divergence may turn out to be a mediocre target for finding circuits! We will also emphasize this point in the conclusion.\n\n**We don’t know the true minimal graph**. Correct, sadly we don’t, and it’s a problem for your previous objection as well.\nMeasuring the correctness of a hypothesis is an open problem in the field. There are some attempts in the literature (Geiger et al. 2021, Chan et al. 2022), but their soundness is only supported by theoretical arguments. Little experimentation targeted to establishing that these approaches reliably tell good from bad hypotheses has been done, instead they have just been applied.\n\n**Sensitivity of the hyperparameters.** Very good point. The best way to communicate non-linear sensitivity is via plots that show the variation in performance for each value of the ACDC ($\\tau$), SP ($\\lambda$) and HISP (% heads) parameters. We did so by color-coding points in Figs. 3 and 4 of the updated PDF. The number of recovered edges and the task-specific loss vary smoothly with these parameters, with little noise. Thus, this does not seem to be a problem. The main robustness challenge of these algorithms is that the dataset, metric and type of activation patching, can greatly change the recovered circuit.\n\n**Typos in graph**. The $\\mu$ is not a typo, it indicates the SI prefix ‘micro’ ($\\cdot 10^{-6}$). The values displayed in that plot are extremely small, practically zero. We agree this is confusing and apologize, we will amend the paper to explain this. The ACDC points are not out of range, but our data collection script had a bug and missed this particular run. We have fixed this oversight already in our copy of the paper!\n\n**Highlighting the units that the circuit misses**. The IOI negative heads are the only pattern that we noticed. After a more thorough look, we found that they are recovered by ACDC with the KL divergence as a target, when the threshold is $\\tau \\le 0.00398$. This is towards the mid-range of the thresholds we used in the final sweeps in the updated Figure 3, which is from $10^{-5}$ to $10^0$ (the figure goes down to $10^{-9}$, but that is only for the tracr tasks).\nUnsurprisingly, ACDC does not manage to recover the negative heads when using the IOI metric (logit difference) as a target. This is a reason to prefer the KL over the task-specific metric."}, {"review_id": "H1reVJQJ5Z", "reviewer": "Reviewer_Zemb", "summary": "This paper proposes an approach for the automatic discovery of circuits (ACDC) in artificial neural networks (applied to transformer-based LLMs), which works by recursively constructing a subgraph of \"important\" nodes identified through the patching of model activations on datapoints relevant to a specific task (the choice of which is, in general, non-trivial). The authors demonstrate, through many experiments, that ACDC is mostly able to faithfully recover circuits which were manually identified by previous researchers on a variety of tasks (notably Python docstrings, IOI and induction heads), thus automating a highly labor-intensive part of the circuit discovery process. Additionally, they explore choices of patching value, target metric and threshold value, whilst also demonstrating that other comparable methods for distillation/subgraph isolation are not as well-behaved as ACDC.\n\nIn addition to the paper, the authors also release an open-source implementation of ACDC which has already been applied to some success by other mechanistic interpretability researchers already.", "questions": "The following questions are not crucial to the narrative of the paper, nor potential critiques of completeness.\n\n1. Have you thought of automated ways to trade-off circuit faithfulness versus sparsity, without re-running ACDC with different values of $\\tau$? \n2. In general, how sensitive is ACDC to the choice of clean and corrupted datapoints. E.g. for the IOI task does ACDC provide considerably different circuits if very few examples are provided, vs. many? What about paraphrasing or noise injection as in ROME (this is probably very task dependent)?\n3. When modifying Subnetwork Pruning you discuss interpolating the mask values - should we expect that linearly interpolating between a clean and corrupted activation is principled (i.e. does not potentially shift the representation to something meaningful but distinct)?\n4. Are all values of SP masks 0, or 1 by the end of SP training? If not, then for the sake of counting subgraph edges, what is considered an \"unmasked node\"?\n5. Given the unexpected utility of using zero ablations, would the authors suggest trying this whenever utilizing ACDC?\n6. Could the authors expand on how the _locally significant changes_ alternative to detecting salient parts of the subgraph would avoid potential sensitivity to the form of patching? Presumably, some perturbation would be required to measure \"effects\" and it is not immediately clear what this perturbation is, if not an activation patch.\n", "limitations": "Limitations are addressed, or explored through supplemental experiments.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The methodology for automatic circuit discovery proposed by the paper extends previous approaches for activation patching to automate otherwise labor-intensive mechanistic interpretability work. This in of itself is not a significant novelty, but the paper's strength lies in a thorough experimental investigation of the benefits of ACDC over other subgraph discovery methods, coupled with new methodological insights on how best to perform activation patching. In particular, they provide two novel findings: 1) KL Divergence is more well-behaved than logit differences when performing activation patching for circuit discovery, and 2) Zero patching, whilst significantly OOD, is often more effective than patching corrupted activations. \n\nThe presentation of the paper is very good, with a coherent narrative for the experimental investigations and clear figures supporting all claims. Additionally, the supplementary materials provide further interesting discussion and results, which given the exploratory nature of circuit discovery is highly valuable.\n\nLastly, the release of the accompanying ACDC algorithm for use by the community is a significant contribution in of itself, as demonstrated by the fact that other members of the MechInt community have already applied ACDC for their own research.", "weaknesses": "No major weaknesses were identified.\n\nThere are a fair number of minor phrasing issues outlined in the following nitpicks section. In the related work section, explicitly stating how path patching varies procedurally from ACDC might be worthwhile. Additionally, a discussion of how ACDC varies from Causal Scrubbing in its patching methodology may be useful.\n\n## Nitpicks\n* 3: makes it **too** costly\"\n* 27: circuits **as** subgraphs\n* 32: \"with which to extract\" or \"for extracting\"\n* 32: remove \"that automates part of it\"\n* 85: The choice of phrasing - \"clearly defined behaviour\" makes this sentence almost tautological. Perhaps an explicit mention of simplicity would be suitable here. Researchers unfamiliar with MechInt may consider e.g. \"Writing python code\" clearly defined or \"writing python docstrings\" too broadly defined.\n* 91: \"**Tasks** 1 and 3\" inside the parentheses\n* 94: from **each task, which** researchers\n* 98: **as** a computational graph\n* 101: \"on the level of detail of _their_ explanations of model behaviour* subject unclear and wording confusing\n* 125: such as -> for example (as subject of \"such as\" could be \"tasks\"). \"predict correct gender predictions\" remove last predictions?\n* 223: It seems the discussion of zero ablations takes place in Section 5 and Appendices F.2, rather than Appendix D.\n* 233: \"we explain how compare to\" -> \"we compare to\"\n* 236: \"experiments use the same modifications to SP and HISP *as*\"\n* 308 \"is known*, and through comparison with previous*...\"\n* 310: not clear what \"outline of a subgraph\" means vs. just \"a subgraph\"\n* 318: \"work; a novel contribution\" or \"work - a novel contribution\".\n* 320: \"*within* the community\"\n\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind words about our work. In particular, we are happy to see that they are also excited about open-sourcing ACDC and that they recognize how it has already been used by the community to accelerate mechanistic interpretability research. We were also pleased to see that you (reviewer Zemb) were complimentary of our empirical evaluations on KL divergence and zero patching. We hope our replies to your questions are comprehensive. We have added the nitpicks to our working copy and thank you for documenting them.\n\nRegarding the call for more description of how ACDC differs from causal scrubbing we will add the following paragraph to an appendix, but in summary: we think that **ACDC and Causal Scrubbing are complementary tools—Causal Scrubbing is a tool to test hypotheses, while ACDC focuses on the generation of good hypothesis**\n\nAddition to appendices: “Both Causal Scrubbing (Chan et al. 2022) and ACDC make extensive use of path patching: Both test how well a circuit reproduces a model's performance by resample-ablating all edges other than the ones specified by the circuit. This path-patching test is a special case of Causal Scrubbing.\n\nThe step automated by ACDC is the generation of good hypotheses, it allows us to find the smallest circuit hypotheses that reproduce certain levels of model performance in an efficient and automatic manner.\n\nCausal Scrubbing itself is not limited to testing just model subgraphs (as we do here) but in principle also allows testing which part of the inputs are relevant for which parts of the circuit, while ACDC automates only the circuit finding on a subgraph level.”\n\nNow we will respond to your questions.\n\n> Have you thought of automated ways to trade-off circuit faithfulness versus sparsity, without re-running ACDC with different values of $\\tau$?\n\nThis is an interesting point. Like many ML algorithms, the performance of ACDC is somewhat sensitive to the value of tau. In general, we think that it is not too much of a burden to do a sweep over tau values—especially compared to the previous state-of-the-art of manually searching for circuits by hand. In developing our work, we found that the number of connections pruned so far is highly predictive of how many connections will remain at the end of an ACDC run. We provide evidence for this claim in Appendix I, and will add a paragraph to this Appendix in response to the reviewers' interests.\n\n> In general, how sensitive is ACDC to the choice of clean and corrupted datapoints … if very few examples are provided, vs. many? \n\nThe datasets used for the behaviors we investigate are very small. For example, we 40 dataset examples are used for the induction task and 100 for IOI and Greater Than. Because we are able to get good results on these small datasets, we do not expect the choice of data points to significantly affect ACDC’s output. In the attached PDF (Figure 4) we evaluated the KL Divergence on held-out test examples, but we have not extensively evaluated this. We did not compare noisy corruptions because we in general we want to test the ability to isolate specific behaviors (e.g the IOI and Greater Than circuits carefully choose patching distributions so that they can isolate a specific behavior present in one distribution, but not in the other). We included zero ablation comparisons as this intervention does not require any parameter choice (but choosing a norm term for noise interventions is an additional choice). \n\n> Are all values of SP masks 0, or 1 by the end of SP training? If not, then for the sake of counting subgraph edges, what is considered an \"unmasked node\"?\n\nWe round the outputs of SP to 0 or 1 and then count edges with the rounded graph.\n\nWhen modifying Subnetwork Pruning you discuss interpolating the mask values - should we expect that linearly interpolating between a clean and corrupted activation is principled (i.e. does not potentially shift the representation to something meaningful but distinct)?\n\nWe use linear interpolation as a continuous approximation through training that uses resampled activations (as in Causal Scrubbing). The generally good performance of SP makes us confident that our continuous approximation that we then clamp to 0 or 1 is a valid use of the SP approach. Nevertheless, we take your point that this could potentially shift the representation somewhere else that is in distribution but distinct (i.e. meaningful but distinct). We think further developments on our approximation is an interesting avenue for future work (adapting gradient based methods to work with corrupted activations). \n\n> Given the unexpected utility of using zero ablations, would the authors suggest trying this whenever utilizing ACDC?\n\nWe apologize to the reviewer for the data in our incorrect appendix figure (Figure 15, see our global response and the attached PDF) that may have led to this conclusion. We attach in the PDF the correct performance of ACDC, SP and HISP with zero activations on the IOI task in Figure 15, and find that zero ablation performs worse for this task (like Docstring and Greater-Than). ACDC performance of zero ablations on the tracr tasks was unchanged (perfect) but we do not think that this should be extrapolated to realistic language models.\n \n> Could the authors expand on how the locally significant changes alternative to detecting salient parts of the subgraph would avoid potential sensitivity to the form of patching?\n\nWe don't understand how the locally significant alternative “would avoid potential sensitivity to the form of patching”. In our paper, we discussed how the locally large effects alternative could resolve some issues with negative head recovery, although since it is not optimizing any metric globally it would be more difficult to interpret the performance of this alternative, unlike with e.g KL Divergence."}, {"review_id": "9hraE7kHY1", "reviewer": "Reviewer_pJGB", "summary": "This paper introduces a method for pruning nodes in a computation graph that is meant to be used in the context of mechanistic interpretability, i.e., to find sub-graphs that explain/reproduce certain behavior of the overall graph while being much smaller. ", "questions": "- L63: This sentence can be ambiguous for readers unfamiliar with this topic: Do the authors mean that one creates a dataset to train a new model that one can analyze? \n- L93: Add a reference to explain what is meant by \"tracr\".\n- L125ff: It can be a bit confusing that the authors here mention that they automate all but the last steps in this work, while before and after, they say they only want to automate the third step. \n- L133f: What happens if the network implements a mechanism using redundant features and an or-operation? Then the detected circuit will only include one of the two valid sub-circuits (as only removing one does not impact the model much, but removing both drastically impacts it). Depending on what one is interested in investigating, this might not be an issue, but this should be discussed.\n- L133f: Is this guaranteed to find the sparsest graph, or can premature pruning of connections at the end prevent the pruning of other connections that, all things considered, contain much more connections/components? For example, think about a situation where two parallel branches have different purposes but are restricted to the dataset in question, they behave identically. Both of them can be pruned away, but if the size of the branches differs, it matters which one is removed.\n- Figure 4: What does this figure tell us? Adding a small description of what is shown here and a conclusion will make this more accessible to readers.\n- L315f: Can the authors propose any reasonable strategy for automatically tuning/setting this hyperparameter?", "limitations": "The authors addressed most limitations in the main part of the paper, except that point raised in the Questions section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem is well-motivated, and the method's description is easy to understand.\n- The proposed method is compared against baselines in the form of existing pruning methods that got adapted to the context of mechanistic interpretability.", "weaknesses": "- While the authors mention that there is early evidence that their method can produce new insights, I believe these should be highlighted more prominently. The practical relevance of this approach can be better shown by applying this method to a new setting, producing mechanistical interpretability hypotheses for some network(s), and then verifying these post-hoc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much, pJGB, for your comments on our work! We hope that this reply answers all your questions, and look forward to further discussion.\n\n**New insights from the method should be featured more prominently.** We agree that showing the practical relevance of new methods is important evidence for their usefulness. We will add an extra sentence in the conclusion highlighting that practitioners have begun to verify the hypotheses generated by ACDC, which we have included at the end of this response. We will also add a paragraph to the gendered pronoun (Appendix I) on how the hypothesis generated thanks to ACDC was faithful to the model’s computation; also attached at the end of this response.\n\n**What does Figure 4 tell us?** We have added a clearer figure (attached in the general comment PDF) with a better caption, and will clarify in the main text. We used held-out i.i.d test data for this figure rather than the same data the circuit discovery methods used. The takeaway is that the recovered circuits get close to behaving like the full model with few edges.\n\n**Do the authors mean create a dataset and train a new model to analyze?** No, we use “dataset” to refer to prompts that show that a model has a particular behavior. We do not update the models’ weights. We will add a clarification to the main text.\n\n**\"tracr\" reference; automating all or one step?** Thank you for pointing out these mistakes! Tracr should have a reference on first appearance, and we only automate the third step.\n\n**Redundant features and OR-operation, ACDC will include one subcircuit.**\nThis is a very good point, and it is correct. As a concrete example, consider a NN implementing an OR gate, where both inputs to it are set to the same value (both 1 or both 0). ACDC will recover exactly one of the OR gate's inputs: first it will attempt to remove one of them, and see that the circuit behavior is unchanged; then removing the second connection will impact the behavior, and it is recognized as important. SP should behave in the same way.\n\nIn contrast, HISP should not recognize any of the inputs as important. The gradient of the output with respect to OR gate inputs should be zero, since both inputs are held to 0 or 1. \n\nWe have conducted an experiment showing this, and put the recovered circuits in the general response PDF. We set the weights of a small ReLU transformer by hand, so it implements an OR gate. We then apply each of the algorithms, and they behave as expected: ACDC and SP find exactly one input, and HISP does not find any inputs to such an OR gate. We have included the key figure in the general comment PDF.\n\nWe have written an Appendix for this experiment, which expands on the three previous paragraphs. In it, we discuss how future work on automating circuit discovery could deal with this issue. For example, ACDC-like methods could run several times, shuffling the parent nodes. Each run will recover a different input to the OR gate. The recovered circuit is then the union of what is recovered in all runs. Gradient descent methods like SP can run with different random seeds.\n\nThank you for finding this interesting property!\n\n**Two parallel branches of different size.** This case seems similar to the OR gate example, where circuit recovery algorithms behave differently. We will include it in the OR-gate Appendix.\n\n**Sparsest graph guarantee?** None of the algorithms is guaranteed to find the sparsest graph or the branch with the\nlargest effect size. Both ACDC and SP get stuck in local optima, though in practice SP seems to get stuck less often due\nto the smooth objective and gradient-based optimization. We will mention this and the previous point in the OR-gate appendix, too.\n\n**Can the authors propose any reasonable strategy for automatically tuning/setting this hyperparameter?**\nAs also discussed with reviewer aEyk in more detail, in practice the iterative nature of ACDC makes selecting an appropriate parameter easier than expected. In short, the number of edges ablated early in ACDC runs can be used as a proxy for the total proportion of edges ACDC ablates. We will mention the issue in our paper by explaining more clearly how practitioners did not have trouble setting $\\tau$ in real-world use cases (Figure 13 has similar sparsity at the output node to all other locations in the network). We also the open source implementation of ACDC should let the community quickly develop better strategies to tune $\\tau$.\n\n---\n\n## Addendum: Additions To The Paper\n\n**Addition to maintext (Line 309):**\nFurther, there is early evidence of the use of ACDC to help with novel interpretability work, discovering a surprising outline of a subgraph of GPT-2 Small that predicts gendered pronoun completion, where practitioners have used ACDC to generate a circuit outline of the most important pathway through a model's computation, and checked that this reflects the model's computation in normal (unablated) forward passes.\n\n**Addition to Appendix I (Line 834):**\nACDC's output shows that the important internal information flow for predicting the expected gender has three steps. Firstly, Layer 0 attention head 0.4 and MLP0 use the name embedding which they pass (through intermediary MLPs) via key- and value-composition (Elhage et al., 2021) to attention heads 4.3 and 6.0. Secondly, heads 4.3 and 6.0 attend to the name position to compose with 0.4 and MLP0. Finally, through value-composition with attention heads 6.0 and 4.3 (via MLP7), the outputs of 10.9 and 9.7 output the expected gendered completion to the output node. Anonymous (2023) then verified that in a normal forward pass of GPT-2 Small, 0.4 has an attention pattern to itself at the name token, attention heads 4.3 and 6.0 attend to the previous name token, and 10.9 and 9.7 attend to the \" is\" token. They also perform path patching experiments on intermediate nodes to provide further evidence of the importance of the pathway through the \" is\" token.\n"}, {"review_id": "7GRNKnG1vJ", "reviewer": "Reviewer_aEyk", "summary": "This paper first presents an overview and a useful distillation of existing mechanistic interpratility work on discovering interpretable circuits in transformer models. They say most existing work happens in 3 steps:\n1. Observe a behavior (or task) that a neural network displays, then create a dataset to measure this behavior\n2. Define the scope of interpretation: Do we want to look at which attention heads, mlp layers or individual neurons are important.\n3. Perform a search with patching experiments to remove as many unnecessary components as possible\n\nThey then propose an algorithm ACDC to automate step 3 of this process, which has typically required extensive manual effort by researchers. They evaluate this extensively by applying it on existing circuits found by researchers, finding that it can discover existing circuits with good accuracy and outperform baselines.", "questions": "- How did you choose tau for different experiments?", "limitations": "Yes, very good discussion.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Very important/impactful problem\n- Good survey of existing work in a very new topic, bringing important clarity/systemization to the workflow of these otherwise individual findings.\n- Clear writing\n- Extensive and comprehensive evaluation\n- Good performance with the automated method\n- Proposed systemization and automated method to discover circuits will likely greatly improve the speed at which new discoveries can be made in this field, making it easier for new researchers to approach.", "weaknesses": "- Choice of threshold parameter tau seems inconsistent/varies by task. Unclear how I would choose tau when applying this method on a new task.\n- Does not address how to come up with the task and dataset for step 1 which may be the hardest part of the workflow", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback on our work! We appreciate the concise distillation of our identified three steps of the mechanistic interpretability workflow and how you highlighted the diverse strengths of our work.\n\nThis response is intended to address both of the weaknesses you brought up regarding our paper.\n\n> Does not address how to come up with the task and dataset for step 1 which may be the hardest part of the workflow\n\nOverall, we don’t think that selecting tasks and datasets (Step 1 of our workflow) is likely to be as difficult or time-intensive as patching and subgraph-finding (Step 3, which we automate). For example, the IOI and Greater-Than papers used datasets consisting of at most 100 prompts that they could concisely describe, but they report much more extensive patching experiments. It is possible that in future for more challenging tasks it will be harder to design datasets, but currently we think that our contribution automates most of the identified mechanistic interpetability workflow.\n\n> Unclear how I would choose tau when applying this method on a new task\n\nThank you for the feedback, and this is a valid concern. However, we have not found this an issue yet, particularly compared to the alternative circuit recovery methods. ACDC is an iterative rather than an end-to-end algorithm. This means that we can observe the subset of the subgraph that ACDC has recovered when only one node has been iterated over (which typically takes <1% of the total number of iterations). Therefore practitioners can use the number of recovered nodes as inputs to the output node as an approximation to the number of nodes that will be recovered in the entire ACDC process. A good example of this working is in Figure 13 in the gendered pronouns use case.\n\nWe will highlight how practitioners have dealt with the $\\tau$ parameter choice in an additional paragraph in the gendered pronouns appendix to describe this workflow. In the main text, we performed sweeps with logarithmic spacing between choices of $\\tau$, which we will also detail in the release of results in the open source implementation.\n"}]

Author rebuttal:
We thank all the reviewers for their detailed and positive feedback on our paper.

*All* reviewers recommended an acceptance and together found our contribution “well-motivated” (pJGB) and a “fresh take on mechanistic interpretability” (6BEj) that is an “important clarity/systemization to the workflow of these otherwise individual findings” (aEyk) and automates “a highly labor-intensive part of the circuit discovery process” (Zemb).

We’ve attached a PDF with updated figures in this global response, and in our individual responses to reviewers we hope that we have answered specific questions.

Unfortunately, we found that our code for the ROC figure was buggy in a subtle way. Our conclusions from the main text remain the same (e.g ACDC has the greatest performance by AUC), though the updated figure in the PDF does improve the performance of the existing algorithms that we repurposed for mechanistic interpretability. One minor point in the appendix was affected, which we discuss with reviewer Zemb.

Our PDF includes
1. An updated ROC plot (Figure 3) with fixed bugs from the earlier figure. 
2. An update to the Pareto frontier plot (Figure 4) for clarity (there were no problems with the data collection for this plot) as pointed out by reviewer pJGB.
3. An update to the appendix Figure 15, which shows that zero ablation does not work as well in that case.
4. A figure with experimental results on automated circuit discovery of OR gate mechanisms, to respond to reviewer pJGB. We thank the reviewer for showing interest in this issue with circuit discovery, and we will add this example as an appendix to our paper.

We look forward to engaging with the reviewers further on their questions. Thank you all for your work!

Meta-review: This paper proposes an algorithm for a subtask frequently performed in mechanistic interpretability: automatic discovery of circuits. The algorithm (ACDC) recursively constructs a subgraph identified through the patching of model activations on datapoints relevant to a specific task. The authors demonstrate that ACDC is mostly able to faithfully recover circuits which were manually identified by previous researchers on a variety of tasks (notably Python docstrings, IOI and induction heads). They also demonstrate other comparable methods for distillation/subgraph isolation are not as well-behaved as ACDC.

Overall, mechanistic interpretability currently relies on a lot of tailor-made routines, arrived through laborious trial-and-error and task knowledge. Methods which successfully automate any part of the pipeline are much welcome, and will generate interest at the conference.
Decision: Accept (spotlight)