SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, a more fundamental limitation to measuring the false and true positive rates of circuit recovery methods is that the ground-truth circuits are reported by practitioners and are likely to have included extraneous edges and miss more important edges. The language model circuits studied in our work (Appendices F-H) involve a large number of edges (1041 in the case of IOI) and the full models contain more than an order of magnitude more edges. Since these interpretability works are carried out by humans who often report limitations of their understanding, our 'ground-truth' is not  $100\%$  reliable, limiting the strength of the conclusions that can be drawn from the experiments in this section.
Figure 4: Comparison of ACDC and SP with both zero-input activations (left) and corrupted activations (right). We plot the KL Divergence on a held-out test set against the number of edges of each hypothesized circuit. Lower KL divergence and fewer edges correspond to better subgraphs. Darker points include more edges in the hypothesis: they use smaller ACDC  $\tau$ , smaller SP regularization  $\lambda$  or a higher percentage of nodes in HISP.
4.2 Stand-alone circuit properties with a test metric
This section evaluates the algorithms by studying the induction task. We measure the KL Divergence of the circuits recovered with the three methods to the original model. This is an indirect measure of Q1, with the advantage of not relying on the completeness or correctness of previous works. As an indicator of Q2, we also measure the number of edges that a hypothesized circuit contains. A circuit with fewer edges which still obtains a low KL Divergence is less likely to contain components that do not participate in the behavior. In Appendix L we also introduce and explain experiments on reset networks that provide more evidence for Q2.
Our mainline experimental setup is to run the circuit recovery algorithms as described in Algorithm 1 and Section 3 and then measure the KL Divergence for these circuits on the induction task (Appendix J). In brief, ACDC performs better that the other methods under these experimental conditions with both corrupted and zero activations. For example, the left-hand side of Figure 4 shows that, above 20 edges, ACDC starts having a slight advantage over other methods in terms of behavior recovered per number of edges as all points on the Pareto-frontier with at least this many edges are generated from ACDC runs. Appendix E describes many further experiments with variations on setup to provide a more complete picture of the performance of the circuit recovery algorithms. For example, when we measure the loss (the task-specific induction metric; Table 1) of subgraphs recovered by optimizing KL Divergence, we find very similar qualitative graphs to Figure 4.
In Appendix L we see that the KL divergence that all methods achieve is significantly lower for the trained networks, indicating that all the methods get signal from the neural network's ability to perform induction (Figure 4). HISP and SP with zero activations, and to some extent SP with corrupted activations are also able to optimize the reset network. This suggests that these methods are somewhat more prone to finding circuits that don't exist (i.e. evidence against Q2).
Mechanistic interpretability encompasses understanding features learnt by machine learning models (Olah, Mordvintsev, and Schubert, 2017; Elhage et al., 2022), mathematical frameworks for understanding machine learning architectures (Elhage et al., 2021) and efforts to find circuits in models (Nanda et al., 2023; Cammarata et al., 2021; Chughtai, Chan, and Nanda, 2023; Wang et al., 2023). The higher standard of a mechanistic understanding of a model has already had applications to designing better architectures (Fu et al., 2023), though the speculative goal of mechanistic interpretability is to understand the behavior of whole models, perhaps through describing all their circuits and how they compose. Little work has been done to automate interpretability besides Bills et al. (2023) who use language models to label neurons in language models.
Neural network pruning masks the weights of neural networks to make their connectivity more sparse (LeCun, Denker, and Solla, 1989). In contrast to our aims, the pruning literature is typically concerned with compressing neural networks for faster inference or to reduce storage requirements (Wang, Wohlwend, and Lei, 2020; Kurtic et al., 2022). Early work (Hassibi and Stork, 1992) hoped pruning would lead to more interpretable networks, but progress towards interpretability via pruning is limited (Grover, Gawri, and Manku, 2022).
Pruning techniques may learn masks from data, which is a special case of more generally using gradient information. Masks can also be learned from data, with an objective function that balances model performance and network sparsity (Louizos, Welling, and Kingma, 2018; Wang, Wohlwend, and Lei, 2020; Cao, Sanh, and Rush, 2021). This is a useful comparison to ACDC as learnable masks do not change the weights of our model after pruning (Frantar and Alistarh, 2023). Examples of gradient information being used more generally includes Michel, Levy, and Neubig (2019) who decide which heads should be pruned by using the absolute value of their gradients, while "movement pruning" (Sanh, Wolf, and Rush, 2020) removes parameters that have high velocity to a low magnitude. ACDC is different from pruning and other compression techniques (Zhu et al., 2023) since i) the compressed networks we find are reflective of the circuits that model's use to compute outputs to certain tasks (Section 4) and ii) our goal is not to speed up forward passes, and generally our techniques slow forwards passes.
Causal interpretation. Much prior research on understanding language models has drawn inspiration from causal inference (Pearl, 2009), leading to the development of frameworks that provide causal explanations for model outputs (Pearl, 2009; Feder et al., 2021; Geiger et al., 2021; Wu et al., 2022; Kaddour et al., 2022). Other work (Vig et al., 2020) discusses the difference between indirect effects and direct effects inside language models, and experiments on removing subsets of these heads using heads' direct effects as proxies for the overall contribution of these heads. Goldowsky-Dill et al. (2023) introduce 'path patching' to analyze the effects of different subsets of edges in computational graphs of models. In parallel to our work, Wu et al. (2023) develop a method to automatically test whether neural networks implement certain algorithms with causal testing. Our work is focused on finding rather than verifying an outline of an algorithm implemented by a model.
Computational subgraphs for interpretability. Training dynamics in residual models can be explained by shallow paths through the computational graph (Veit, Wilber, and Belongie, 2016). MLP layers can be modelled as memory that is able to represent certain properties of the network inputs (Geva et al., 2021). Residual transformer models have been modelled as the sum of all different paths through the network (Elhage et al., 2021). Later work has used insights from looking at subgraphs of models in order to edit models' behaviors (Bau et al., 2020; Meng et al., 2022) and test interpretability hypotheses (Chan et al., 2022).
We have identified a common workflow for mechanistic interpretability. First, pin down a behavior using a metric and data set. Second, conduct activation patching experiments to understand which abstract units (e.g. transformer heads) are involved in the behavior. Third, iterate the previous steps with variations of the behavior under study, until the model's algorithm is understood.
The main proposed algorithm, ACDC, systematically conducts all the activation patching experiments necessary to find which circuit composed of abstract units is responsible for the behavior. We have shown that ACDC and SP recover most of the compositional circuit that implements a language model behavior, as judged by comparison to previous mechanistic interpretability work (Section 4). ACDC with zero activations fully recovers the circuit of toy models (Fig. 9). Further, there is early evidence of the use of ACDC to help with novel interpretability work, discovering a surprising outline of a subgraph of GPT-2 Small that predicts gendered pronoun completion (Appendix K). Here, practitioners used ACDC to generate a subgraph including the most important pathway through a model's computation, and checked that this reflects the model's computation in normal (unablated) forward passes. This surprising find was an early example of the summarization motif (Tiggs et al., 2023).
However, both ACDC and SP have limitations which prevent them from fully automating step 3 of the identified workflow (activation patching). First, they tend to miss some classes of abstract units that are part of the circuit, for example the negative name mover heads from IOI (Wang et al., 2023).
Second, the behavior of the algorithms is very sensitive to hyperparameter and metric choice, leading to varied and non-robust performance in some settings (Figure 3).
On balance, the evidence supports the claim that ACDC can automate part of interpretability work, a novel contribution. Automating interpretability research may be necessary to be able to scale methods to the behaviors of the large models which are in use today. We hope that our open-source implementation of ACDC (https://github.com/ArthurConmy/Automatic-Circuit-Discovery) accelerates interpretability research from the community. For example, future work could systematize and automate the problem of varying the corrupting dataset to understand the functionality of different parts of the circuit.
This work would not have been possible without the generous support of Redwood Research through their REMIX program. We would like to thank Chris Mathwin, Jett Janiak, Chris MacLeod, Neel Nanda, Alexandre Variengien, Joseph Miller, Thomas Kwa, Sydney von Arx, Stephen Casper and Adam Gleave for feedback on a draft of this paper. Arthur Conmy would like to thank Jacob Steinhardt, Alexandre Variengien and Buck Shlegeris for extremely helpful conversations that shaped ACDC. We would also like to thank Haoxing Du for working on an early tool, Nate Thomas for coming up with the catchy name, Daniel Ziegler who discussed experiments that inspired our Subnetwork Probing analysis, Oliver Hayman who worked on an earlier prototype during REMIX and Lawrence Chan who helped us frame our contributions and suggested several experiments. Finally we thank Hofvarpnir Studios, FAR AI and Conjecture for providing compute for this project.
Alishahi, Afra, Grzegorz Chrupa, and Tal Linzen (2019). "Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop". In: Natural Language Engineering 25.4, pp. 543-557. DOI: 10.1017/S135132491900024X.
Bau, David, Steven Liu, Tongzhou Wang, Jun-Yan Zhu, and Antonio Torralba (2020). Rewriting a Deep Generative Model. URL: https://arxiv.org/abs/2007.15646.
Bills, Steven, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders (2023). Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.
Brundage, Miles, Shahar Avin, Jack Clark, Helen Toner, Peter Eckersley, Ben Garfinkel, Allan Dafoe, Paul Scharre, Thomas Zeitzoff, Bobby Filar, et al. (2018). "The malicious use of artificial intelligence: Forecasting, prevention, and mitigation". In: arXiv preprint arXiv:1802.07228.
Buhrmester, Vanessa, David Münch, and Michael Arens (2021). "Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Survey". In: 3.4, pp. 966-989. ISSN: 2504-4990. DOI: 10.3390/make3040048. URL: https://www.mdpi.com/2504-4990/3/4/48.
Cammarata, Nick, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah (2021). "Curve Circuits". In: Distill. https://distill.pub/2020/circuits/curve-circuits. DOI: 10.23915/distill.00024.006.
Cao, Steven, Victor Sanh, and Alexander Rush (2021). "Low-Complexity Probing via Finding Subnetworks". In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Online: Association for Computational Linguistics, pp. 960–966. DOI: 10.18653/v1/2021.naacl-main.74. URL: https://aclanthology.org/2021.nacl-main.74.
Chan, Lawrence, Adria Garriga-Alonso, Nix Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris, and Nate Thomas (2022). Causal scrubbing: A method for rigorously testing interpretability hypotheses. Alignment Forum. URL: https://wwwalignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing.
Chughtai, Bilal, Lawrence Chan, and Neel Nanda (2023). A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations. URL: https://arxiv.org/abs/2302.03025.
Cuadros, Xavier Suau, Luca Zappella, and Nicholas Apostoloff (2022). "Self-conditioning pre-trained language models". In: International Conference on Machine Learning. PMLR, pp. 4455-4473.
Linguistics: Human Language Technologies. Seattle, United States: Association for Computational Linguistics, pp. 4288-4295. DOI: 10.18653/v1/2022.naacl-main.318. URL: https://aclanthology.org/2022.naacl-main.318.
Zhang, Kelly and Samuel Bowman (2018). "Language Modeling Teaches You More than Translation Does: Lessons Learned Through Auxiliary Syntactic Task Analysis". In: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP. Brussels, Belgium: Association for Computational Linguistics, pp. 359-361. DOI: 10.18653/v1/W18-5448. URL: https://aclanthology.org/W18-5448.
Zhang, Yu, Peter Tino, Aleš Leonardis, and Ke Tang (2021). "A survey on neural network interpretability". In: 5.5, pp. 726-742. DOI: 10.1109/TETCI.2021.3100641.
Zhu, Xunyu, Jian Li, Yong Liu, Can Ma, and Weiping Wang (2023). A Survey on Model Compression for Large Language Models. arXiv: 2308.07633 [cs.CL].
A Table of contents
1 Introduction 1
2 The Mechanistic Interpretability Workflow 2
2.1 Step 1: Select a behavior, dataset, and metric 3
2.2 Step 2: Divide the neural network into a graph of smaller units 3
2.3 Step 3: Patch model activations to isolate the relevant subgraph 4
2.4 Explaining the circuit components 4
3 Automating circuit discovery (Step 3) 5
4 Evaluating Subgraph Recovery Algorithms 6
4.1 Grounded in previous work: area under ROC curves 6
4.2 Stand-alone circuit properties with a test metric 8
5 Related work 8
6 Conclusion 9
7 Acknowledgements 10
A Table of contents 15
B Impact statement 16
C Discussion of metrics optimized 17
C.1 Changing the metric in ACDC 17
C.2 Limitations of logit difference 17
C.3 Alternatives to minimizing a metric 18
D Details of Subnetwork Probing and Head Importance Score for Pruning 19
D.1 Subnetwork Probing 19
D.2 Head Importance Score for Pruning 19
E Experimental study of algorithm design 20
E.1 Minimizing the task-specific metric, rather than the KL divergence 21
E.2 Activation patching with zeros, instead of corrupted input 21
E.3 Node-level ROC curve, rather than edge-level ROC curve 23
F IOI task: details and qualitative evidence 23
F.1 Further details on the IOI experiments 23
F.2 The IOI circuit 24
F.3 Limitations of ACDC in recovering the IOI circuit 25
G Greater-Than task: details and qualitative evidence 26
H Docstring task: details and qualitative evidence 27
H.1 The docstring circuit 27
H.2 Additional docstring experiments 29
I Tracr tasks: details and qualitative evidence 30
I.1 tracr-xproportion 30
I.2 tracr-reverse 30
J Induction task: details and qualitative evidence 31
K Gendered pronoun completion: qualitative evidence 32
L Reset Network Experiments 32
M Automated Circuit Discovery and OR gates 34
N Connection to Causal Scrubbing 35
B Impact statement
ACDC was developed to automate the circuit discovery step of mechanistic interpretability studies. The primary social impact of this work, if successful, is that neural networks will become more interpretable. ACDC could make neural networks more interpretable via i) removing uninterpretable and insignificant components of models (as we reviewed in Section 4.2), ii) assisting practitioners to find subgraphs and form hypotheses for their semantic roles in models (as we found early evidence for in Appendix K) and more speculatively iii) enabling research that finds more interpretable architectures. More generally, better interpretability may allow us to predict emergent properties of models (Nanda et al., 2023), understand and control out-of-distribution behavior (Mu and Andreas, 2021) and identify and fix model errors (Hernandez et al., 2022).
However it is also possible that the biggest impact of better interpretability techniques will be more capable AI systems or possible misuse risk (Brundage et al., 2018). For example, while interpreting neural networks has the steer models towards model bias or other harmful effects (Cuadros, Zappella, and Apostoloff, 2022), bad actors could also use interpretability tools to do the opposite: reverse engineering neural networks to steer towards harmful behaviors.
For now, ACDC is a tool for researchers and isn't mature enough for applications where determining the exact behaviour of a model is societally important. However, the benefits of the adoption of better transparency appear to us to outweigh the externalities of potential negative outcomes (Hendrycks and Mazeika, 2022), as for example transparency plays an important role in both specific (Hubinger, 2020) and portfolio-based (Hendrycks, Mazeika, and Woodside, 2023) approaches to ensuring the safe development of AI systems.
C Discussion of metrics optimized
In this appendix, we discuss the considerations and experiments that support the formulation of ACDC that we presented in Section 3. We also discuss the metrics and experimental setups for Subnetwork Probing (Appendix D.1) and Head Importance Score for Pruning (Appendix D.2).
In the main text we presented ACDC as an algorithm that minimizes the KL divergence  $D_{\mathrm{KL}}(G||H)$  between the model and the subgraphs of the model (Section 3 and Algorithm 1). However, prior mechanistic interpretability projects have reported performance on several different metrics at once (Wang et al., 2023; Nanda et al., 2023). In this Appendix we discuss our findings choosing different metrics in different ways. We explore the advantages and limitations with Algorithm 1 and other approaches. In particular, we have found that optimizing for low KL divergence is the simplest and most robust metric to optimize across different tasks. However, general conclusions about the best methods to use cannot be made because of variability across different tasks, and the large space of design choices practitioners can make.
We found that optimizing for low KL divergence was fairly effective across all tasks we considered, except the Docstring task (Appendix F-J). For example, we were able to exclusively recover heads that are present in the IOI circuit (Figure 1) that have 3 layers of composition sufficient to solve the task, with zero false positives. Additionally, KL divergence can be applied to any task of next-token prediction as it doesn't specify any labels associated with outputs (such as logit difference requiring specifying which tokens we calculate logit difference between).
C.1 Changing the metric in ACDC
We consider generalizations of ACDC in order to further evaluate our patching-based circuit-finding approach. The only line of Algorithm 1 that we will modify is Line 6, the condition
$$
D _ {K L} (G | | H _ {\text {n e w}}) - D _ {K L} (G | | H) <   \tau \tag {1}
$$
for the removal of an edge. All modifications to ACDC discussed in this Appendix replace Condition (1) with a new condition and do not change any other part of Algorithm 1.
In full generality we let  $F$  be a metric that maps subgraphs to reals. We assume throughout this Appendix that subgraphs  $H$ , such that  $F(H)$  is smaller, correspond to subgraphs that implement the task to a greater extent (i.e we minimize  $F$ ).<sup>5</sup>
In practice, we can be more specific about the form that the metric  $F$  will always take. We assume that we can always calculate  $F(H(x_{i},x_{i}^{\prime}))$ , the element-wise result of the metric on individual dataset examples and use  $F(H)$  to refer to the metric averaged across the entire dataset (note the similarity of this setup to our calculation of  $D_{\mathrm{KL}}$  in Section 3). The general update rule takes the form
$$
F (H) - F \left(H _ {\text {n e w}}\right) <   \tau . \tag {2}
$$
which generalizes Equation 1. We discuss further extensions that change Line 6 of the ACDC algorithm in Appendix C.3.
C.2 Limitations of logit difference
The IOI (Appendix F), Docstring (Appendix H) and Gendered Pronoun Identification work (Appendix K) originally used a variant of logit difference to measure the performance of subgraphs. Logit difference is the difference in logits for a correct output compared to a baseline incorrect output. Then, these works compare the change from the logit difference of the model to the logit difference of their circuit. However, unlike KL divergence, this metric is not always positive — logit difference for a circuit could be larger or smaller than the logit difference of the model, and so the change in logit difference could be positive or negative. We discuss issues that arise with this approach
5The logit difference and probability difference metrics used by the IOI, Greater-Than and Docstring tasks were intended to be maximised by the respective researchers (Table 1) so we consider negated versions of these metrics.
in Appendix C.3, the empirical performance decrease when using logit difference can be found in Figure 5.
C.3 Alternatives to minimizing a metric
Two alternatives to minimizing a metric are to 1) match the model's performance on a metric, or 2) only include edges that cause a small change in performance. These could be formalised by the following alternatives to Condition 1, where  $F$  denotes any metric we could compute from a subgraph:
1. Matching the model's performance:  $|F(H_{\mathrm{new}}) - F(G)| - |F(H) - F(G)| < \tau$ .
2. Only including small changes in performance:  $|F(H_{\mathrm{new}}) - F(H)| < \tau$ .
Matching the model's performance (also referred to as faithfulness by Wang et al. (2023)). Since KL divergence is always positive, Alternative 1 is identical to Condition 1 when  $F$  is the KL divergence between a subgraph's outputs and the models' outputs, but for metrics such as logit difference this represents a new optimization objective. Empirically we found that matching the model's performance was unstable when we ran ACDC. For example, we ran a modified early version of ACDC that maximized the logit difference in the IOI circuit, and found that through one ACDC run, logit difference of a subgraph could be as large as 5.0 and as low as 1.5 during a modified ACDC run. The IOI circuit has a logit difference of 3.55, and therefore the subgraph's logit difference can be both larger and smaller than the model's logit difference. This issue arises when the subgraph's logit difference is larger than the model's. In such cases, ACDC will discard model components that it would otherwise include when the subgraph's logit difference is smaller than the model's. This leads to inconsistencies between runs and further dependence on the order over which parents are iterated (Section 3).
Figure 5: ROC curves on IOI using Abs Logit Diff, Logit Diff and KL Divergence
Only including small changes in performance. Alternative 2 ignores the value of the metric for the base model and instead focuses on how much the metric changes at each step that ACDC takes.
However, we found that this was much less effective than KL divergence and even worse than using logit difference on the IOI task (Figure 5), when we used random ablations and recorded ROC curves.