SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A limitation worth noting is that we applied ACDC to a computational graph of attention heads and their query, key and value computational nodes, while Heimersheim and Janiak (2023) considered the attention heads outputs into every token position separately. This allowed them to distinguish two functions fulfilled by the same attention head (layer 1, head 4) at different positions, which cannot be inferred from the output of ACDC alone at any level of abstraction (Section 2.2) we studied in this work. We make this choice for performance reasons (the long sequence length would have made the experiments significantly slower) but this is not a fundamental limitation. In Appendix K we use ACDC to isolate the effects of individual positions in a different task.
H.2 Additional docstring experiments
Logit difference metric: To compare ACDC more closely with the docstring work (Heimersheim and Janiak, 2023), we added an ACDC run with the objective to maximize the logit difference metric. We used a threshold of  $\tau = 0.067$  and found the subgraph shown in Figure 19. We found that ACDC performed better than SP and HISP when using the logit difference metric (Figure 7).
Zero activations: Unlike in the case of induction (Section 4.2), we found that using zero activations rather than random (corrupted) activations, lead to far worse results. For example, with  $\tau = 0.067$
<sup>10</sup>Again, not considering heads 0.2 and 0.4 which are not actually relevant under the docstring distribution.
(the same threshold that generated Figure 19 except with zero activations) we get a circuit with 177 edges (Figure 19 has 98), as well as a KL divergence of 3.35 and a logit difference of  $-2.895$ . All these metrics are worse than the subgraphs generated with corrupted activations (Table 4).
I Tracr tasks: details and qualitative evidence
In this Appendix we discuss the two tract tasks we studied in Section 4, as well as additional experiments that studied ACDC when applied at a neuron level.
We used a transformer identical to the one studied in Lindner et al. (2023), and refer to that work for details on the tracr-xproportion task (called the frac_prevs task in their paper). We also studied the tracr-reverse task, described in the tracr Github repository. $^{11}$
We make one modification to the traditional ACDC setup. We set the positional embeddings equal to randomized positional embeddings in the corrupted datapoints — otherwise, we don't recover any of the circuit components that depend only on positional embeddings (and not token embeddings). We describe the two tasks that we studied in the main text and describe further results that broke these computational graphs down into neurons.
I.1 tracr-xproportion
We used the proportion task from the tracr main text, and used as metric the L2 distance between the correct list of proportions and the recovered list of proportions. For the corrupted dataset, we let  $(x_{i}^{\prime})_{i = 1}^{n}$  be a random permutation of  $(x_{i})_{i = 1}^{n}$  with no fixed points.
When we ran ACDC at the neuron level, as shown in Figure 20b, there are no extra nodes present that were not used by this tracer model. In fact, this computational graph visualization produced by ACDC is more compact than the complete view of the states of the residual stream which is illustrated in Figure 20a (from Lindner et al. (2023)). In this case, the transformer is small enough for a practitioner to study each individual state in the forward pass. However, for larger transformers this would be intractable, which necessitates the use of different interpretability tools such as ACDC.
See Fig. 21a for the full circuit (without decomposition into residual stream dimensions).
(b) ACDC visualization.
Figure 20: Two visualizations of how a tracr-compiled transformer completes the frac_prevs task. The ACDC circuit is specific to the individual neurons and residual stream components. This is more fine-grained than the ground truth we use throughout the work. This experiment was coded in rust_circuit and is not reproducible using the Transformer Lens code yet.
I.2 tracr-reverse
To test ACDC on more than one example of a tracr program, we also used the 3-layer transformer that can reverse lists (the tracr-reverse task). Once more, the outputs of the transformer are not
<sup>11</sup>URL: https://github.com/deepmind/tracr, file: README.md
(a) tracr-xproportion canonical circuit (10 edges)
(b) tracr-reverse canonical circuit (15 edges)
Figure 21: The canonical circuits for finding the proportion of 'x' in the input and reversing lists. ACDC recovers these perfectly using zero activations (Table 3 and Figs. 9 and 10).
distributions - in this case they are new lists. We calculate the L2 distance between the one-hot vectors for the recovered list and the true reversed list. For the corrupted dataset, we again let  $(x_{i}^{\prime})_{i = 1}^{n}$  be a random permutation of  $(x_{i})_{i = 1}^{n}$  with no fixed points. Again, at the neuron level a perfect graph is recovered, with the minimal components required to reverse lists (Figure 21b).
J Induction task: details and qualitative evidence
In Section 4.2 we use 40 sequences of 300 tokens from a filtered validation set of OpenWebText (Gokaslan et al., 2019). We filter the validation examples so that they all contain examples of induction — subsequences of the form “ $A, B, \ldots, A, B$ ”, where  $A$  and  $B$  are distinct tokens. We only measure KL divergence for the model's predictions of the second  $B$  tokens in all examples of the subsequences  $A, B, \ldots, A, B$ .
We use both zero activations and corrupted activations to compare ACDC and the other methods. To use ACDC with zero activations, we apply one change to the procedure described in Section 3: instead of setting activations of edges not present in the subgraph to the activations on a corrupted dataset, we set their value equal to 0. We describe how we adapt the methods from Section 4.1 to be used with both zero activations and corrupted activations in Appendix D.1 for SP and Appendix D.2 for HISP.
Our induction experiments were performed on a 2-layer, 8-head-per-layer attention only transformer trained on OpenWebText (Gokaslan et al., 2019). The model is available in the TransformerLens (Nanda, 2022) library. We follow Appendix C of Goldowsky-Dill et al. (2023) for the construction of the dataset of induction examples.
The computational graph has a total of 305 edges, and in Figure 4 we only show subgraphs with at most 120 edges.
12The model can be loaded with transformer_lens.HookedTransformer.from_pretrained(model_name = "redwood_attn_21", center_writingweights = False, center_unembed = False) (at least for the repository version of the source code as of 23rd May 2023)
When iterating over the parents of a given node (Line 4 in Algorithm 1), we found that iterating in increasing order of the head index was important to achieve better results in Figure 4. Similar to all experiments in the work, we iterate in decreasing order of layers, so overall we iterate over head 1.0, 1.1, ... then 1.7, then 0.0, 0.1, ...
An example of a circuit found in the process is given in Figure 6.
K Gendered pronoun completion: qualitative evidence
Mathwin et al. (2023) aim to isolate the subgraph of GPT-2 small responsible for correctly gendered pronouns in GPT-2 small. They do this by studying prompts such as "So Dave is a really great friend, isn't" which are predicted to finish with "he". For that they used ACDC. This presents an example of a novel research project based on ACDC. The result of applying the ACDC algorithm (threshold  $\tau = 0.05$ ) is shown in Figure 22.
The computational subgraphs generated by ACDC on the gendered pronoun completion task show that MLP computations are more important than attention head computations in this task than in the IOI task (Appendix F.2). Early, middle and late layer MLPs have important roles in the subgraph. For example, MLPs 3 and 5 are the important components at the name position (which must be used to identify the correct gender) as they have multiple incident edges: the MLP 7 at the “_is” position has the most incoming connections of any node in the graph, and the late layer MLPs 10 and 11 have the largest direct effect on the output. MLP 7's importance at the “_is” position is an example of a discovery that could not have been made with simpler interpretability tools such as saliency maps. This was early evidence of the summarization motif (Tigges et al., 2023).
ACDC's output shows that the important internal information flow for predicting the correct gender has three steps. Firstly, Layer 0 attention head 0.4 and MLP0 use the name embedding which they pass (through intermediary MLPs) via key- and value-composition (Elhage et al., 2021) to attention heads 4.3 and 6.0. Secondly, heads 4.3 and 6.0 attend to the name position to compose with 0.4 and MLP0. Finally, through value-composition with attention heads 6.0 and 4.3 (via MLP7), the outputs of 10.9 and 9.7 output the expected gendered completion to the output node. Mathwin et al. (2023) then verified that indeed in a normal forward pass of GPT-2 Small, 0.4 has an attention pattern to itself at the name token, attention heads 4.3 and 6.0 attend to the previous name token, and 10.9 and 9.7 attend to the 'is' token. They also perform path patching experiments on intermediate nodes to provide further evidence of the importance of the pathway through the 'is' token.
We used the dataset of  $N = 100$  examples from Mathwin et al. (2023). The corrupted dataset was a set of prompts with a similar structure to the sentence "That person is a really great friend, isn't", following the authors' approach.
We defined a computational graph that featured nodes at the specificity of attention heads split by query, key and value vectors, and further split by token position (where the tokens are present in the nodes in Figure 22). From the input sentence 'So Sarah is a really nice person, isn't', we chose to add nodes representing the model internal operations at the tokens "Sarah", "is", "person", "isan" and "t", while other positions were grouped together as in Mathwin et al. (2023). The resulting subgraph can be found in Figure 22.
L Reset Network Experiments
Our reset network experiment setup is motivated by the concern that interpretability explanations may not accurately represent the reasoning process behind models' predictions (Jacovi and Goldberg, 2020). This is particularly relevant to work on subnetworks as empirically some subnetworks in models with randomized weights do not accurately represent such reasoning (Ramanujan et al., 2020).
To this end, we study the task-specific metrics on models with permuted weights (which we call reset networks (Zhang and Bowman, 2018; Cao, Sanh, and Rush, 2021)) and verify that the circuit recovery algorithms perform worse on these models that do not have underlying algorithms. Specifically, we create the reset network by permuting the head dimension of each layer's Q, K, V matrices, and each MLP's bias term. This disrupts the functionality of the subject model, without changing many facts about the distribution of the activations (e.g. the average magnitude at every layer). In our experiment
Figure 22: Gendered pronoun completion circuit found by ACDC.
in Figure 8 the metric used by each algorithm is the KL divergence between the original trained network (with no edges patched), and the activation-patched reset network.
The reset network does not exhibit the original network's behavior, and thus it should not be possible to explain the presence of the behavior. This is a strong measure of the negation of Q2: if the algorithm is able to find a circuit that performs the behavior on a network that does not exhibit the behavior, then it will likely hallucinate circuit components in normal circumstances.
M Automated Circuit Discovery and OR gates
In this appendix we discuss an existing limitation of the three circuit discovery methods we introduced in the main text: the methods we study do not identify both inputs to 'OR gates' inside neural networks.
OR gates can arise in Neural networks from non-linearities in models. For example, if  $x, y \in \{0, 1\}$  then  $1 - \mathrm{ReLU}(1 - x - y)$  is an OR gate on the two binary inputs  $x, y$ . To study a toy transformer model with an OR gate, we take a 1-Layer transformer model two heads per layer, ReLU activations and model dimension 1. If both heads output 1 into the residual stream, this model implements an OR gate. Our dataset (Section 2) is then equal to a single prompt where the heads output 1, and we use zero activations to test whether the circuit discovery methods can find the two inputs to the OR gate.
Figure 23: OR gate recovery. a) the ground truth, b) ACDC, c) HISP, d) SP.
The results can be found in Figure 23. The ground truth in a) is our toy model of an OR gate, where MLP0 performs OR on the bias terms of a0.0 and a0.1. These are the only edges that should be included. b) ACDC only recovers one OR gate input. This is because the iterative algorithm prunes the first input to the OR gate it iterates over and then keeps the other. c) HISP recovers neither OR gate input (and also recovers the unnecessary input node). d) SP recovers only one OR gate input, and several additional nodes. SP and HISP found extra edges since they include the input node by default. HISP doesn't include either attention head showing the limitations of gradients in this idealized case. We are unsure why SP finds the a0.0's key and value inputs. This shows the limitations of node-based methods for finding circuits, though ACDC is also limited. Of course, many easy fixes exist to this problem, but the priority of future work should be to explain in-the-wild language models, where it is less clear which algorithmic improvements will be most helpful. For example, follow up work found that using gradient approximations on the edges of a computational graph was very effective (Syed, Rager, and Conmy, 2023), despite not being more effective at finding OR gates.
<sup>13</sup>This construction is similar to the AND gate construction from Gurnee et al. (2023) Appendix A.12. <sup>14</sup>For specific details of the TransformerLens (Nanda, 2022) implementation, see https://github.com/ArthurConmy/Automatic-Circuit-Discovery/blob/main/acdc/logic Gates/util.py#L15.
N Connection to Causal Scrubbing
This work was inspired by work on Causal Scrubbing (Chan et al., 2022). The scope of each algorithm is quite different, however.
Causal Scrubbing (CaSc; Chan et al., 2022) aims primarily at hypothesis testing. It allows for detailed examination of specified model components, facilitating the validation of a preconceived hypothesis.
ACDC, on the other hand, operates at a broader scale by scanning over model components to generate hypotheses, each of which is tested using CaSc. ACDC chooses to remove an edge if according to the CaSc criterion, the new hypothesis isn't much worse.
Why is testing every hypothesis with Causal Scrubbing not incredibly inefficient? The reason is that ACDC only considers a small class of CaSc hypotheses, where paths through the model either matter, or don't matter. In effect, the CaSc hypotheses considered by ACDC don't allow any interchanges if the node "matters" (by having a unique value for each possible input), and the nodes that don't matter are each replaced by the same second data point.
Both methods currently face computational inefficiencies, albeit for different reasons and at different scales. Causal Scrubbing is impractical for somewhat complicated causal hypotheses because of treefication: there are exponentially many paths through a branching DAG, and each needs part of a forward pass. For ACDC, each hypothesis is quick to test, but the number of edges to search over can be quite large, so it still takes a while to search over circuits for realistic models. This problem is partially addressed by using gradient-based approximations like Attribution Patching (Syed, Rager, and Conmy, 2023) or perhaps an edge-based version of Subnetwork Probing (Cao, Sanh, and Rush, 2021).
In summary, ACDC and Causal Scrubbing are complementary tools in the analysis pipeline. ACDC can do an initial coarse search over hypotheses and, while it is built on Causal Scrubbing, only considers a small class of hypotheses so it stays relatively efficient. In contrast, Causal Scrubbing offers a methodical way to test hypotheses, which can also specify the information represented in each node.