Hypothesis Testing the Circuit Hypothesis in LLMs
Authors: Claudia Shi
Nicolas Beltran-Velez
Achille Nazaret
Carolina Zheng
Adrià Garriga-Alonso
Andrew Jesson
Maggie Makar
David Blei
Venue: NeurIPS 2024 poster (2024)
Keywords: Hypothesis testings
mechanistic interpretability
circuit
Primary area: interpretability_and_explainability
TL;DR: Hypothesis tests and empirical studies on how well existing circuits align with idealized properties of circuits

Abstract: Large language models (LLMs) demonstrate surprising capabilities, but we do not understand how they are implemented. 
One hypothesis suggests that these capabilities are primarily executed by small subnetworks within the LLM, known as circuits. But how can we evaluate this hypothesis?
In this paper, we formalize a set of criteria that a circuit is hypothesized to meet and develop a suite of hypothesis tests to evaluate how well circuits satisfy them. 
The criteria focus on the extent to which the LLM's behavior is preserved, the degree of localization of this behavior, and whether the circuit is minimal.
We apply these tests to six circuits described in the research literature. 
We find that synthetic circuits -- circuits that are hard-coded in the model -- align with the idealized properties. 
Circuits discovered in Transformer models satisfy the criteria to varying degrees.
To facilitate future empirical studies of circuits, we created the \textit{circuitry} package, a wrapper around the \textit{TransformerLens} library, which abstracts away lower-level manipulations of hooks and activations. The software is available at \url{https://github.com/blei-lab/circuitry}.

Hypothesis Testing the Circuit Hypothesis in LLMs
Claudia Shi
Nicolas Beltran-Velez'
Achille Nazaret
Carolina Zheng*
Adrià Garriga-Alonse Alonso
Andrew Jesson'
Maggie Makar
David M. Blei'
'Department Computer Science, Columbia University, New York, USA Computer Science and Engineering, University of Michigan, Ann Arbor, USA
FFAR AI, USA
Abstract
Large language models (LLMs) demonstrate surprising capabilities, but we do not understand how they implemented. One hypothesis suggests that these capabilities are primarily exccuted by small subnetworks within the LLM, known circuits. But how can we evaluate this hypothesis? In this paper, formalize a set of criteria that a circuit is hypothesized to meet and develop a suite hypothesis tests to evaluate how well cireuits satisfy them. The criteria focus the extent to which the LLM's behavior is preserved, the degree of localization of this behavior, and whether the cireuit is minimal. We apply these tests six circuits described in the research literature. We find that synthetic circuits circuits that hard-coded in the model align with the idealized properties. Circuits discovered in Transformer models satisfy the criteria to varying degrees. To facilitate future empirical studies of circuits, we created the circuitry package, a wrapper around the TransformerLen library, which abstracts away lower-level manipulations of hooks and activations. The software available at https; //github. wwwccomo.///ccoccoccoc hts//blliiiie lab/ bblii-aabccircitz. circuitry
Introduction
The field of mechanistic interpretability aims to explain the inner workings of large language models (LLMs) through reverse engineering. One promising direction is identify "cireuits" that correspond to different tasks. Examples include circuits that perform context repetition [Olsson et al.) 2022
identify indirect objects [Wang et al. 2023 and complete docstrings (Heimersheim and Janiak 2023
Such research motivated by the circuit hypothesis, which posits that LLMs implement their capabilities via small subnetworks within the model. If the circuit hypothesis holds, it would be scientifically interesting and practically useful. For example, it could lead to valuable insights about the emergence of properties such as in-context learning [Olsson et al. 2022 and grokking during training [Stander et al. 2023 Nanda et al. 2023b). Moreover, identifying these circuits could aid in explaining model performance and controlling model output, such improving truthfulness.
In this work, we empirically study the circuit hypothesis to assess its validity in practice. We begin by defining the ideal properties of circuits, which we posit to be: 1. Mechanism Preservation: The performance of an idealized circuit should match that of the original model. Mechunism
"guual contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Localization: Removing the circuit should eliminate the model's ability to perform the associated task. 3. Minimality: A circuit should not contain any redundant edges.
We translate these properties into testable hypotheses. Some of these hypotheses depend on the strict validity of the idealized circuit hypothesis, while others more flexible, allowing to quantify the extent to which discovered circuits align with the ideal properties.
We apply these tests to six circuits described in the literature that each correspond to different task: two synthetic, hard-coded circuits and four discovered in Transformer models. These circuits have also been used to benchmark automatic circuit discovery algorithms [Conmy et al. 2023 Syed al. 2023)
We find that the synthetic circuits align well with the idealized properties and our hypotheses while the discovered circuits do not strictly adhere to the idealized properties. Nevertheless, these circuits are far from being random subnetworks within the model. Among the discovered circuits, the induction circuit [Olsson et al. [2022] passes two of the three idealized tests. The Docstring circuit [Heimersheim and Janiak] 2023] passes the minimality test. Furthermore, the empirical results indicate that these circuits can be significantly improved, bringing them closer to idealized circuits. For example, for two of them, removing 20% of the edges had little impact on their ability to approximate the model.
The contributions of this paper are: 1. suite of formal and testable hypotheses derived from the circuit hypothesis. 2. A set of statistical procedures and software to perform each test. 3. An empirical study of existing circuits and their alignment to the circuit hypothesis.
1.1 Related work
This research fits in the broader field of mechanistic interpretability. We provide a brief overview of related work here and a more comprehensive discussion in $ A
Olah et al. 2020 introduced the concept of circuit. Subsequently, various cireuits have been
proposed, particularly in vision models [Mu and Andreas 2020 Cammarata et al. [2021] |Schubert
et al. 2021 and language models [Olsson al.) [022] Wang al. [2023] Hanna et al. 2023
Lieberum et al. 2023). The literature has been especially effective in explaining small Transformers
that perform algorithmic tasks [Nanda et al.) 2023a Heimersheim and Janiak, 2023 Zhong et al.)
2023 Quirke et al. 2023 Stander et al. 2023
This work builds on the growing effort around evaluating the quality of interpretability results (Doshi
Velez and Kiml [017]] Casper et al., 2203)] Mills et al., [023] Hase et al. [2024] Jacovi and Goldberg
2020
Geiger al.] [202] Chan al. [002 Wang et al.] 2023 [Schettmman et al. [023]] Lindner
ct al. Wang al. minimality, and completeness to evaluate the Indirect Object Identification circuit. Faithfulness serves metric, while minimality and completeness involve searching the space of circuits. Our idealized criteria are similar in spirit to Wang et al. [2023], but the specific tests differ. A key distinction our adoption of hypothesis testing framework, where none of our tests require searching the space of circuits. Conmy et al. [023) develops an automatic circuit discovery algorithm and assess the quality of circuits by measuring edge classification quality against a set of benchmark circuits.
2024 Friedman et al. 2024| Variengien and Winsor 2023 It is closely related to the works of
2023] and Conmy al. [202] Wang al. [2023] introduce three criteria faithfulness,
2 Mechanistic Interpretability and LLMs
In this section, we define the necessary ingredients for mechanistic interpretability in LLMs !
2.1 LLMs as computation graphs
Transformer-basee LLM is a neural network that takes in a sequence of input tokens and produces sequence of logits over possible output tokens. We define it function M X + O, where , x² of E V, L E E21} is the space of sequences of tokens, V is the space of possible tokens, called the vocabulary, and O = ((o',....,') RIVI E Z>1} } is the space of sequences of logits over the vocabulary.
X = {(x) ,..,",...,...
For details about the Transformer architecture, see Elhage al. 2021 for excellent overview.

---

The function M is computed by sequence of smaller operations that compose to form a computa- tional graph. A computational graph a directed acyclic graph g (V. where is the set of nodes and & is the set of edges. Each node E V represents an operation with one or more inputs and a single output. Each edge (u, v) E & denotes that the output of node u used as the input node v. We recursively define the output of node v as a2 v(a.") ) where al {a u E V, (u, u) E E are the inputs to v. We denote the number of inputs to c as de.
We can use different levels of granularity to define the nodes of a computational graph, each leading to different types of interpretability. Following Elhage et al. 2021]. we define the nodes of the computational graph of an LLM to be attention heads and MLP layers. The edges correspond to the
residual connections between them. We also include input nodes corresponding to the embeddings of the input tokens and output nodes corresponding to the logits.
2.2 Tasks: measuring the performance of a model
To measure whether particular model performs specific function, we define task, T, tuple T = (D, s) of dataset D = {x 3/) and a score s O x y - R. The dataset D contains pairs of inputs 2; X and output y/ V. The score maps sequence of logits, such as the output of the model M (xt), and the ground truth information y, to a real number indicating the performance of the model's output on that particular example: higher score indicates better performance.
Example 1 (Greater-Than). An example task is the greater-than operation [Hanna et al. 2023). where we evaluate whether the model can perform this task as it would appear in natural language. The dataset D contains inputs from I; "The noun lasted from the year XXYY to the year XX" where noun is an event, egg "war", , XX is a century, egg. 16, and YY is specific year in the century. The score function is the difference in assigned probabilities between the years smaller than yi YY and the years greater than or equal to yi- The implied task is to predict the next token YY as any year greater than YY so as to respect chronological order.
A circuit is subgraph C' = (Vc, Ec) of the computational graph G. It includes the imput and output nodes and a subset of edges, &c. that connect the input to the output. We let denote the space of all circuits. Fig. I depicts one such circuit in simplified computational graph of a two-layer attention-only Transformer. Given a circuit, we define its complement to be the subgraph of g that includes all edges not in C' and their corresponding nodes.
2.3 Circuits of an LLM
circuit specifies a valid subgraph, but it not sufficient to specify a runnable model. Recall that node e in the circuit is function with a collection of inputs an corresponding to each (u, v) present in E. If the edge (u, v) removed, then what input a, should be provided to node u??
One solution, called activation patching, is to replace all inputs ag with an alternative value a;, one for each edge (u, v) E & that is absent from the circuit.
There are various ways to choose a value for a Two common approaches are zero ablation, which sets o to 0 [Olsson et al. 2022], and Symmetric Token Replacement (STR) patching (Chan et al. 2022 Geiger et al.) 2024 Zhang and Nanda 2024]. STR sets at, differently for each input x; and proceeds as: First, create corrupted input 2 which should be like but with key tokens changed to semantically similar ones. For example, in the greater-than task with input x; = "The war lasted from the year 1973 to the year 19", we might replace it with 2 = "The war lasted from the year 1901 to the year 19". The meaning is preserved but the 2 73 constraint is removed. Then, run the model on x. and cache all the activations at Finally, run the circuit on 2;, replacing the input a,, of u with the cached at, for all edges (u, v) E E\EE iteratively until reaching at the output node.
We use the notation C(x) to denote the output of the circuit C' on the input æ. where the ablation scheme is implicit. When we compute the output of the complement of the circuit, namely C(x), we say that we knock out the circuit C' from the model M.
3

---

Input sequence
Layer
Head
Head 2
Layer
Head
Head
Output sequence
Figure Simplified computational graph of a two-layer LLM with two attention heads (without MLPs). Nodes in each layer connect to all nodes in the next layer via residual connections. A highlighted arbitrary circuit shown blue. In a detailed graph, each incoming edge to an attention head splits into three: query, key, and value.
2.4 Evaluation metric:
faithfulness
Given a circuit C(x) and task = (D, s), we can use the score function to evaluate how well the circuit performs the task. However, in mechanistic interpretability. the goal is often to evaluate whether the circuit replicates the behavior of the model, which is known as faithfulness.
We define a faithfulness metric, F C x C x X' x y - I, that maps two circuits and an example in D real number measuring the similarity in behavior of these two circuits with respect this particular example. We then define the faithfulness of circuit to model M on task T as
F+(M, = B(XY)~D F(M, C, X))
We call F, (M, C)) the faitlifulness score of circuit C. For example, a faithfulness metric could be the 2 norm between the score of the model and the score the circuit,
F(M, Cr = (x) y) - sC(())
with & E {1,2}. However, F can be more general and non-symmetric such as the KL divergence between the logits of M and C' [Conmy et al. 2023]. Following convention, lower value for (MM, C) means the circuit more faithful to the model.
3 Hypothesis Testing on Circuits
this section, we develop three tests that formalize the following idealized criteria for a circuit: 1. Mechanism Preservation: The circuit should approximate the original model's performance on the task. 2. Mechanism Localization: The circuit should include information critical to the task's execution. 3. Minimality: The circuit should be small possible.
In 3.1 discuss three idealized but stringent hypotheses implied by these criteria and develop tests for each. Then, i 3.2 we develop two flexible tests for Mechanism Localization and Mechanism Preservation, allowing users to design the null hypotheses and determine to what extent circuit aligns with the idealized properties.
Standard hypothesis testing has components:
1. A variable of interest, Z", e.g., the faithfulness of the candidate circuit.
2. A reference distribution Pz over other Z that we wish to compare 7" to, along with samples (= from it, e.g., the faithfulness of n randomly sampled circuits.
3. A null hypothesis H.. which relates Z* and Pz and which we assume holds true. E.g., Ho: "the candidate circuit is less faithful than 90% of random circuits from Pz."

---

4. A real-valued statistic {=)) computed from the samples, with known distribution when Hg holds, e.g, the number of times the candidate circuit less faithful than a random circuit is binomial variable with success probability 90% if Ho holds.
5. A confidence level 1 and rejection region R. C I such that if Ho is true, then the test statistic falls in R. with probability less than a. If we observe that does fall in Ra we conclude that Ho is false and we reject Hy. We will be correct 1 a of the time when Ho is true.
Finally, defining rejection region for each a, the p-value is the smallest such that {((z:) E Ra (Young and Smith 2005]. The smaller the p-value, the stronger the evidence against Hp-
To perform hypothesis test, we specify the five components above, obtain the samples 21,-... Zn. compute the test statistic 1(21,-..; 2n) with the associated p-value, and reject the null hypothesis with confidence - a if t(z), 2n) Ra-
3.1 Idealized tests
We develop three tests, Equivalence, Independence, and Minimality, which are direct implications of the idealized criteria. These tests are designed to be stringent: if circuit passes them, provides strong evidence that the circuit aligns with the idealized criteria.
We assume have model M, task т (D, s) with score function s, and faithfulness metric F. We are then given candidate circuit C to evaluate.
Equivalence. Intuitively, if C* is good approximation of the original model M, then C* should perform as well as on any random task input. Hence, the difference in task performance between M and C* should be indistinguishable from chance. We formalize this intuition with an equivalence test: the circuit and the original model should have the same chance of outperforming each other.
We write the difference in the task performance between the candidate circuit and the original model on one task datapoint (z, y) as A Axx v) = s(C*(s);;)) - (M)( y), and let the null hypothesis be
1
Ho P(xy)>= (A 0)
2 <e
where e > specifies a tolerance level for the difference in performance.
To test this hypothesis, we use a nonparametric test designed specifically for null hypotheses like Hn- The test statistic is the number of times C* and M outperform each other. We provide a detailed description of the test in B.T
Since Hg is in the idealized direction, if reject the null, claim with confidence 1 Oz,
Non-Equivalence: C* and M are unlikely to be equivalent on random task data.
Independence. a circuit solely responsible for the operations relevant to a task, then knocking out would render the complement circuit unable to perform the task. An implication is that the performance of the complement circuit independent of the original model on the task.
To formalize this claim, define the null hypothesis
Ho (X))( IL s(M(X);
where the randomness is over X and Y.
To test this hypothesis, we use a permutation test. Specifically, we measure the independence between the performance of the complement circuit and the performance of the original model by using the Hilbert Schmidt Independence Criterion (HSIC) [Gretton et al., 2007], a nonparametric measure of independence. We provide formal definition of HSIC and describe the test in $ B.3
If the null is rejected, it implies that the complement eircuit and the original model's performances are not independent. We claim with confidence 1 a,
Knocking out the candidate circuit does not remove all the information relevant to the task that is present in the original model.
Non-Independence:
5

---

Minimality. For minimality, we ask whether the circuit contains unnecessary edges, which are defined to be edges which when removed do not significanlly change the circuit's performance. Formally, we define the change induced by removing an edge e. E & from a circuit C' as
ö(e,C) C) E(z,))> |s(C(z), s(C_.(r),
where C-. = (V,Ec\()))))
We are interested in knowing whether for some specific edge ex E &c the value 5(e", C*) is significant. The problem now becomes how to define the reference distribution against which to compare ò(e", C*). Ideally, we would like to form the distribution ò(e, C*) induced by unnecessary edges But do not know which in C* are unnecessary (finding them is precisely our goal).
To address this problem, we augment C create "inflated" circuits. An inflated eircuit C' of C* is obtained by adding random path to C* that introduces at least one new edge. Our assumption is that the randomly added path is unnecessary to the circuit performance, and so removing one the added edges and studying the change in performance will provide our reference distribution.
We define (C',e') R! such that Cd a random inflated circuit obtained with the above procedure, and e is an edge sampled uniformly at random over the novel edges Eçi /Ec.. We then compare ò(e", C*), the change induced by removing edge e* from C against the distribution of the random variable d(c", C,), the change induced by removing what is assumed to be an irrelevant edge from an inflated version of C graphical illustration this procedure in Fig.
We define the null hypothesis as
Ho Pr 0*) 5(c (oe,c)))>0
(5)
where q² is prespecified quantile.
The null hypothesis states that removing the edge E & induces a significant change the circuit score compared to removing the random edge in the inflated circuit. If we reject Ho, have found an unnecessary edge. We use a tail test in in/Algorihhm I to compute the p-value.
If we perform the test on multiple different edges in the circuit, we need to correct for multiple hypothesis testing. To do so we use the Bonferroni correction, which a conservative correction that controls the family-wise error rate [Dunn 1961]. If we test m edges in the circuit, the corrected significance level a/m.
we test against multiple edges and after Bonferroni correction there is at least one edge for which the null hypothesis rejected, then claim with confidence a,
Non-Minimality: The circuit has unnecessary edges.
3.2
Flexible tests
$ 3.
presents stringent tests that align with the idealized versions of circuits. Passing any of these tests a notable achievement for any circuit. Here, we consider two flexible ways of testing mechanism preservation (sufficiency) and mechanism location (partial necessity).
Instead of comparing the candidate circuit to the original model, compare C* against random circuits drawn from a reference distribution. Different definitions of the reference distribution modulate the difficulty of the tests. We demonstrate that by varying the definition of the reference distribution, we can determine the extent to which the circuit aligns with the idealized criteria.
Sufficiency. For the sufficiency test, we ask whether the candidate circuit is particularly faithful to the original model compared random circuit from a reference distribution.
The variable of interest is the faithfulness of C' to M, Z = F, (M, C*)). We define the reference distribution Pz as the distribution of Z Fr (M, Cr) induced by sampling random circuits C" from a chosen distribution R. The null hypothesis is
Ho Po (M, C*) F,(MMC S
(6)
where q" is prespecified quantile.

---

Test
Tracr-P
Tracr-R
Induction
I01
G-T
DS
Equivalence
X
X
X XX
Independence
X
Minimality
X
X
Table 1: Hypothesis testing results for six circuits using the three idealized tests. A (v) indicates the null hypothesis is rejected, while (x) indicates it is retained. The gray shaded boxes denote synthetic circuits, which align with our hypothesized behavior. For the equivalence test, e 0.1.
The advantage of a null hypothesis like Eq. i that we can change the reference distribution R. and quantile q" to capture to what degree we test the circuit hypothesis. For example, an easier (but important) version of the test is to have the reference distribution be over all circuits of the same size as C This test will verify that the candidate circuit is not simply a draw from the distribution of random circuits, ensuring that it is better than at least fraction q² of random circuits.
Moreover, we can modulate the difficulty and the implied conclusions of the test by changing the size of the random circuits relative to C* and/or the target quantile q" If our distribution of random circuits produces fraction 1 of circuits that are supersets of C (which we expect be comparable to C*), we can set q² = 17, upper bound for the test's stringency.
The test statistic for Eq.. 6 is the proportion of times C* is more faithful than C for circuits C sampled from R. (CTT. C = Σi=1 1(FF(M,CC)) (FME,C)))))))))CC) Under Ho. the test statistic follows binomial distribution, and compute the associated p-value using a one-sided binomial test. This procedure is described in more detail in Algorithm of B
If the v-vulue is less than the significance level for quantile q", claim with confidence 1
Sufficiency: The probability that C more faithful to M than C is at least q"
Partial necessity. If the candidate circuit is responsible for solving task in the model, then removing will impair the model's ability to perform the task. However, this impairment may be so severe to make the model entirely independent of the complement circuit's 's output tested in independence test.
Instead, we define partial necessity: compared to removing a random reference circuit, removing the candidate circuit significantly reduces the model's faithfulness. The null hypothesis
Ho IR is worse knock out than C") q²:
where 9" 10 is user-chosen parameter and where "C" is worse knock than C is short- hand for (M,, > (+(M,
Similar to the sufficiency test, this hypothesis test highly flexible its design. An easier version involves using reference distribution over circuits from the complement C distribution. This allows us to determine whether the edges in the candidate circuit are particularly important for task performance compared a random circuit. Another approach is to define the reference distribution by sampling from the original model M, enabling us to assess whether the significance of a knockdown effect could have occurred by chance.
The test statistic is the proportion of times that C is less faithful than C Similar to the sufficiency test, we apply a binomial test to get the p-vulue. If Hy is rejected, we claim with confidence a.
Partial necessity: The probability that knocking out C* damages the faithfulness M more than knocking out a random reference circuit is at least q"
Empirical Studies
We apply hypothesis tests six benchmark circuits from the literature: two synthetic and four manually discovered. The synthetic circuits align with the idcalized properties, validating criteria.
7

---

Induction
G-T
Induction
G-T
D5
PIC more falthful than C1
101
1.00
1.0
0.75
0.6
0.50
PCC
0.25
0.2
0.00
0.4
0.0
0.2
0.4
0.6
0.8
1.0
Size (C*) fraction of [M]
Size [C*] as fraction of [M]
Figure 2: Left: The relative faithfulness of the candidate circuit compared to random circuit from the reference distribution of varying sizes (x-axis). Dotted vertical lines indicate the actual size of the circuits. Right: The probability that a random circuit contains the canonical circuit.
While the discovered circuits align with the hypotheses to varying degrees, the tests help assess their quality and analyze how well each circuit aligns with the idealized criteria.
4.1 Experimental setup
We the experiment configuration from ACDC (Conmy al. 2023 for all tasks and circuits and perform the ablations using TransformerLens [Nanda and Bloom 2022]. Below, we briefly describe each task, with detailed explanations in D We omit the greater than (G-T) task it was detailed in Both IOI and greater-than use GPT-2 small, while the other tasks use various small Transformers.
$ 2
Indirect Object Identification (IOI, Wang et al. 2023): The goal is to predict the indirect object in a sentence containing two entities. For example, given the sequence "When Mary and John went for walk, John gave apple to", the task is to predict the token" MMary.. The score function is logit(" Mary") - logit(" John").
... A. Induction Olssonn et al. 2022): The objective is to predict B after a sequence of the form AB...
For example, given the sequence "Vernon Dursley and Petunia Durs". the goal is to predict the token "ley" [Elhage et al., 2021 The score function for this task is the log probability assigned to the correct token.
Docstring (DS, Heimersheim and Janiak 2023): The objective is to predict the next variable name in Python docstring. The score function is the logit difference between the correct answer and the most positive logit over the set of alternative arguments.
Tracr (Lindner et al. 2024): For tracr-r, the goal is reverse an input sequence. For tracr-p. goal is to compute the proportion of tokens in the input. The score function the l2 distance between the correct and predicted output. Both of these tasks have "ground truth" circuits, as the Transformers are compiled RASP programs Weiss et al. [2021], hence we call them synthetic circuits.
Experiment details. To construct the reference distributions R of random circuits for the different tests, we sample paths M (or (*) from the input nodes (embeddings) to the output node (logits) using a random walk. For the sufficiency and partial necessity tests, we start from an empty circuit and augment it with the sampled paths until it has at least edges, where l is number we vary in our experiments. For the minimality test, we inflate the circuit by adding one randomly sampled path, and we then randomly choose an edge in the added path to knock out. We draw 100 random circuits to form the reference distribution for the sufficiency and partial necessity tests. For minimality, draw 10, 000 random edges for G-T and IOI and 1000 random edges for the other circuits. In all experiments, we use Eq. with l norm as the faithfulness metric. We set g" to be 0.9 and a to be 0.05.
4.2 Results
Below we report and analyze key findings across tests. Additional results are reported in S
Idealized tests. Table 1 presents the results for the six circuits across the three idealized hypothesis tests. The synthetic circuits (highlighted grey) align well with our hypotheses, supporting the validity of these tests. Among the discovered circuits, alignment with the idealized hypotheses varies:

---

the Induction circuit passes both the minimality and independence tests, while the Docstring circuit pass the minimality test. However, none of the other discovered circuits satisfy the idealized tests. Notably, both the Induction and Doestring circuits were identified in toy Transformer models and are relatively small in size. $ E
Sufficiency test. We apply the sufficiency test to study the extent to which existing circuits align with the circuit hypothesis. As noted in [S 3.2, we can adjust the reference distribution vary the test's stringency. Fig. 2 (left) illustrates the relative faithfulness of the candidate circuits compared to different reference circuit distributions.
The IOI and G-T circuits are significantly more faithful than random circuits at 90% of the original model's size, while the DS and Induction circuits outperform random circuits at 30% size. These results suggest their faithfulness is not due to random chance.
If the circuit hypothesis holds, we can expect the probability randomly sampled circuit is as faithful as the candidate circuit be equal to the probability the random eircuit contains the candidate circuit. In Fig. 2 (right), we illustrate this probability under sampling algorithm. We observe that the curve on the left similar to the inverse of the right. Notably, while the Induction and DS circuits appear similar in Fig. 2 (left), they differ in Fig. 2 (right). The difference suggests that the Induction circuit more closely aligned with the idealized properties compared to the DS circuit.
GreaterThanTask
DocstringTask
1OlTask
Falthfuiness
Faithfulness
40
Faithfulness lness
20
20
0
78 156
234
24
321
963
Number of edges removed
Number of edges removed
Number of edges removed
Figure 3: The faithfulness of the circuit as we gradually knock down more edges from the canonical circuit. Edges are removed in order of their minimality score, starting with the least minimal. The dotted line shows the canonical circuit's 's faithfulness, and the solid line shows an empty circuit's faithfulness. Removing a few minimal edges does not significantly affect faithfulness.
Partial necessity test. We now analyze the knockdown effect of the candidate circuit. Similar to the sufficiency test, can define different reference distributions that reflect different underlying hypotheses. Table 2) reports the results of the hypothesis tests under two reference distributions.
We observe that when the reference eircuit drawn from the complement C*, the knockdown effect for the candidate circuits is significant across tasks. This suggests that edges in the candidate circuit play a more significant role in task performance than edges in the complement circuit. However, when compared against reference circuits drawn from the model M, we find that knocking down the candidate circuit does not always have a more significant effect than knocking down random circuit of the same size.
We increase the size of the random circuits further investigate when knocking down a random circuit has a more significant effect than knocking down the candidate circuit. Surprisingly, we observed that for G-T and IOI, knocking down the complete model has less impact than knocking down the candidate circuit. This pattern appears with the STR ablation scheme but is absent with rero-ablation. Specifically, with zero-ablation, we observe a monotonic decrease in the relative significance of the candidate circuit as the size of the random circuits increase. In contrast, with STR ablation, its significance remains constant.
These findings suggest that the knockdown metric alone is insufficient to assess circuit quality, particularly when using STR ablation, it sensitive artifacts from the validation dataset.
Reference circuit
Induction
101
G-T
DS
O" C
V X
C JM~M M
Table 2: (V) indicates that knocking down C* is significantly worse than knocking down C", while (x) means the converse. C, is the same size as C* but draws from different reference distribution.

---

One explanation is that all edges in the circuit are essential, so knocking down any edge impairs the model's 's task performance. If random circuit includes the candidate circuit's 's edges, the effect is similar. To investigate this, we build on the minimality result.
Minimality, Recall from Table I that the G-T and IOI canonical circuits are not minimal. In Fig. gradually knock out more edges from the canonical circuit and report the faithfulness of the modified circuit. For G-T, we can remove around 50% of the edges while retaining the same faithfulness. For 1OI, we can remove about 20% of the edges. However, we notice that the faithfulness of IOI does not vary monotonically as more edges are knocked out, revealing the complex mechanisms of circuits (e.g., negative mover heads). Although Docstring is minimal, we can still remove small subset of edges without impacting faithfulness. This is because the reference edges are all approximately zero and the Docstring circuit was discovered in toy Transformer model.
5 Discussion & Limitations
Do existing circuits align with the circuit hypothesis? We develop a suite of idcalized and flexible tests to empirically study this question. The results suggest that while existing circuits do not strictly adhere to the idcalized hypotheses, they are far from being random subnetworks.
Our tests successfully differentiate circuits by their alignment with the idealized properties, identifying the Induction circuit [Conmy et al. 2023] the most aligned. We also demonstrate the limitations of existing evaluation criteria, showing that the knockdown effect alone is insufficient to determine circuit quality and that some benchmark circuits are not minimal.
Our tests and empirical studies have several limitations. The idealized tests are stringent, while the flexible tests are sensitive to circuit size measurements and require careful null hypothesis design. For the -onevviiveeence and non-independence tests, the desired direction is to retain the null, but we did not empirically study the Type error associated with these tests. Furthermore, the empirical study uses the original experimental setup. whereas existing work and our ablation studies show that circuits are not robust changes in the experimental setup.
Despite these limitations, we believe the study provides an overview of the extent to which existing circuits align with the idealized properties. We also believe that the tests will aid in developing new circuits, improving existing circuits, and scientifically studying the circuit hypothesis.
6 Acknowledgements
N.B. C.Z., and D.B. were funded by NSF -12-2127 NSF DMS-2311 NSF/DoD PHY- 2229929, ONR 17-1--17-1--131, ONR 110014-15-1-2200 the Simons Foundation, and Open Philanthropy. M.M. was funded by NSF 2153083 and NSF 2337529. A.N. was supported by funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT and Harvard, and the Africk Family Fund.
The authors thank Sebastian Salazar and Eli N. Weinstein for comments on the manuscript and helpful discussion. They also thank the contributors the Automatic Circuit Discovery codebase Conmy et al. 2023], which underlies a significant part of this paper's code.
10

---

References
N. Cammarata, G. Goh, S. Carter, C. Voss, L. Schubert, and C. Olah. Curve Circuits, Distill, 6(1):
c00024-006, 2021.
S. Casper, T. Bu, Y. Li, J. Li, K. Zhang, K. Hariharan, and D. Hadfield-Menell. Red Teaming Deep Neural Networks with Feature Synthesis Tools. Advances in Neural Information Processing Systems, 11884400-80516 2023.
L. Chan, A. Garriga-Alonso, N. Goldowsky-Din Dill, Greenblatt, J. Nitishinskaya, A. Radhakrishnan, B. Shlegeris, and N. Thomas. Causal Scrubbing: A Method for Rigorously Testing Interpretability Hypotheses. In Alignment Forum, 2022.
A. Conmy, A. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, Towards Automated Circuit Discovery for Mechanistic Interpretability. Advances Neural Information Processing Systems, 36:163 6318-163822 2023.
Doshi Velez and B. Kim. Towards a Rigorous Science of Interpretable Machine Learning. arXiv: stat.ML, 2017. URL http://arxivi org/abs////////aooooogoogeeeega/ 2202 08608v2
O. J. Dunn. Multiple Comparisons among Means. Journal of the American statistical association, 56 (293):52-66) 64, 1961.
N. Elhage, N. Nanda, C. Olsson, Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfeeld-oodss D. Hernandez, A. Jones, Kernion, I.. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A Mathematical Framework for Transformer Circuits, 2021. URL https:/7 transformer circuits. www.://s.ppp//222//rraassoroooo wwwee/iiixx....e/ html
D. Friedman, A. Wettig, and D. Chen. Learning Transformer Programs. Advances in Neural Information Processing Systems, 36, 2024.
A. Geiger, H. Lu, T. Ieard, and C. Potts. Causal Abstractions of Neural Networks. Advances in Neural Information Processing Systems, 34:9574 2021.
A. Geiger, 7. Wu, C. Potts, T. Icard, and N. Goodman. Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. In Causal Learning and Reasoning, pages 160-187. PMLR, 2024.
A. Gokaslan and V. Cohen. OpenWebTexttCorpus 2019.
OpenWebText Corpus.
http:///Skylion077gitbbae io/
A. Gretton, K. Fukumizu, C. Teo, L. Song, B. Schölkopf. and A. Smola. A Kernel Statistical Test of Independence. Advances in Neural Information Processing Systems, 20, 2007.
M. Hanna, O. Liu, and A. Variengien. How Does GPT-2 Compute Greater-Than. Interpreting Mathematical Abilities in a Pre-Trained Language Model, 2023.
P. Hase, M. Bansal, B. Kim, and A. Ghandeharioun. Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models. Advances in Neural Information Processing Systems, 36, 2024.
S. Heimersheim and J. Janiak. A Circuit for Python Docstrings in a 4-layer Attention-Only Transformer. Alignment Forum, 2023. URL https://www http://www..llieeeooooooooono/// LGKKXmKPbXEWZoAAX. https::////// // www.---oooooooooraaaaa----------
only
A. Jacovi and Y. Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1198-4205, Online, jul 2020. Association for Computational Linguistics. doi: 10. 18653/v1/0020. acl-main.386. URL https: htatps://aclanthhlooy org 2020/202...0m acc-main. ma6n.386

---

T. Lieberum, M. Rahtz, J. Kramár, N. Nanda, G. Irving, R. Shah, and V. Mikulik. Does Circuit Analysis Interpretability Scale? Evidence From Multiple Choice Capabilities Chinchilla. CoRR, 2023. http::ttaa.x.....rrrrr/// wwwp://////bb//////// 09458v3
P. Linardatos, V. Papastefamopoullos and S. B. Kotsiantis. Explainable Al: A Review of Machine Learning Interpretability Methods. Entropy, 23, 2020. URL https///://ap www..ssesmmttoollleee..m org/Corpusseiee :0tp:rps1::299222884444 229722844
D. Lindner, J. Kramár, S. Farquhar, M. Rahtz, T. McGrath, and V. Mikulik. Tracr: Compiled Transformers as Laboratory for Interpretability. Advances in Neural Information Processing Systems, 36, 2024.
E. Mills, S. Su, S. Russell, and Emmons, ALMANACS: A Simulatabilityy Benchmark for Language Model Explainability. arXiv, 2023. httptttpp://aii.oooo//////o wttp://wwwwrrbb///322.22777/
J. Mu and J. Andreas. Compositional Explanations of Neurons. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17153-17163, 2020. URL https::/proceeddnngg www.:nnnuurrrpppepeeeeee
/020/ http:///000../////o/o/o///////60
N. Nanda and J. Bloom. TransformerLens 2022.
Transformerl.ens. sttpp:://gttuub.com//eeee
io/
N. Nanda, I.. Chan, T. Lieberum, Smith, and Steinhardd Progress Measures for Grokking via Mechanistic Interpretabilityy In The Eleventh International Conference Learning Representa- fions, 2023a. URL httpps::///penreeiiee .ete//Eorumm2d--9XX55hhh...
N. Nanda, L. Chan, T. Lieberum. J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability, 2023b. URL https://arxiiy wwwpe/iii.rrgaeeeooee www.oogg//////o/oo.o abtp//bb////05155 05217
C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom In: An Introduction to Cireuits. Distill, 2020. doi: 10.29995//isslll.00024.00...0
http://isiilllbb22222cocrrrroooo
C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, et al. In-Context Learning and Induction Heads. arXiv preprint arXiv:2209. 11895, 2022.
P. Quirke et al. Understanding Addition in Transformers. arXiv preprint arXiv:2310. 2023.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language Models Unsupervised Multitask Learners. OpenAl Blog, 1(8):9, 2019.
L. Schubert, C. Voss, N. Cammarata, G. Goh, and C. Olah. High-Low Frequency Detectors. Distill, 2021. doi: 10.23915/distill. 00024.005.
Schwettmann, T. R. Shaham, J. Materzynska, N. Chowdhury, S. Li, J. Andreas, D. Bau, and A. Torralba. FIND: A Function Description Benchmark for Evaluating Interpretability Methods. In Thirty-severth Conference on Neural Information Processing Systems Datasets and Benchmarks
Track, 2023, URL /cctps://ppenrvviii http:://paaaarrieeeeeeeeeoeooo/n wwt.://ooorrriiinnnss
D. Stander, Q. Yu, H. Fan, and S. Biderman. Grokking Group Multiplication with Cosets. arXiv preprint arXiv:2312.0658/ 2023.
A. Syed, C. Rager, and A. Conmy. Attribution Patching Outperforms Automated Circuit Discovery. arXiv preprint 2xX::2310.10344. 2023.
A. Variengien and E. Winsor. Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models. arXiv, 2023. URL http:////xxxxi.oor//a/a///////// 10091v1
K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. In International Conference on Learning Representations, 2023. URL httptt:t////ttt semanti org/ www.:rorgggooooooeeee /Corpus
G. Weiss, Y. Goldberg, and E. Yahav. Thinking Like Transformers, 2021.
12

---

G. A. Young and R. L. Smith. Hypothesis Testing, pages 65-80. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2005.
Zhang and N. Nanda. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. In The Twelfth International Conference Learning Representations, 2024. URL https:/ btttps://operrevieet wwwpeeeevveeeeeeeooeeeeeeeeeeeee
Hf17y6uBCC
Z. Zhong. Z. Liu, M. Tegmark, and Andreas. The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks. In Shirty-seventh Conference Neural Information Processing Systems, 2023. httphtts::/
13

---

Appendices
A
Related Work
This work fits into the broader field of explainable Al Linardatos et al. 2020 with a particular application to mechanistic interpretability.
Mechanistic interpretability. Olah al. [2020] introduced the notion of circuit inside neural
network: overlapping units within the network that compute features from other features and can theoretically be understood from the weights. Since then, mechanistic interpretability has proposed
many circuits, such as in vision models [Cammarata et al.) 2021 Mu and Andreas
(020 Schhbertt
al.
2021] and language models [Olsson al. [022] Wang et al. 2023 Hanna et al. 2023 Lieberum
ct al. 2023]. The literature has been particularly successful in explaining Transformers that compute
simple, algorithmic tasks (Nanda al. 2023a Heimersheim and Janiak 2023
Zhong al. 2023
Quirke et al. 2023 Stander et al. [2023
Evaluating interpretability. Evaluating the quality of interpretability results is open problem.
Some researchers focus on evaluating the faithfulness [Jacovi and Goldberg [020] of explanations: do
they accurately represent the reasoning process of the model? Chan et al. [022). Geiger et al. [2021]
introduced similar formalisms for measuring faithfulness based on causality, with [Schwettmann
al. 2023 Lindner al., 2024 |Friedman al. 2024 Variengien and Winsor 2023 providing
datasets and methods to generate them.
Early on, Boshi-Velez and Kim [2017] distinguished between functional (without humans) and application-grounded evaluation of interpretability methods, arguing that functional metrics are flawed proxies for the usefulness of the interpretation. Several works [Casper et al. 2023 Mills et al. 2023] adopt this approach to evaluating interpretability.
Relation to IOI [Wang et al. [2023). The approach proposed in this paper is closest to Wang et al. 2023), which presents three criteria faithfulness, minimality, and completeness to evaluate the 1OI circuit. Faithfulness is a metric, while minimality and completeness involve searching over the
space of cireuits.
Our idcalized criteria align with the spirit of Wang et al. 2023 but the specific tests differ. Wang al. [2023] reported faithfulness score of 0.2 on the circuit, but the significance of this score unclear without a reference point. Our sufficiency test contextualizes whether the faithfulness score significant by comparing it to different reference distributions.
Additionally, Wang et al. [2023] use two other criteria, completeness and minimality. Completeness relates to whether there are parts of the circuit that are not included but may still play role. They evaluate this by checking if the circuit behaves similarly to the model under knockouts. Minimality checks whether, for node v, there exists subset K such that including but removing K signifi- cantly changes the score. Both tests require an exhaustive enumeration of circuits and don' 't use any reference distribution to establish significance, which is an important contribution of our work.
Relation to ACDC ([Conmy et al. 2023) Our work is also related to Conmy et al. 2023). The ACDC evaluation focuses the accuracy of edge classification within cireuits. While they compare the circuits uncovered by the automated method against circuits found in existing works, we evaluate the quality of the circuits presented in these existing works, i.c., what they used ground truth.
Additionally, the ACDC algorithm uses a threshold parameter т to determine the significance of an edge's relevance to the task at hand. They treat this threshold sweeping through various values. In contrast, our minimality test offers a principled approach to establish clear criterion for determining the value of the threshold.
parameter in their search algorithm,
14

---

B Statistical Tests
B.1 Equivalence Test
The null hypothesis for the equivalence test is defined as
Ho: ((A(X,)) > 2 <e
(8)
where is user-chosen tolerance parameter.
Given that (X, > 0} is Bernouliiddittrbuteed under the null hypothesis, we use the test statistic
1 n Σ 114 (1; 91) > 0} 1/2
and choose rejection regions of the form Ra (t da)), where c(a) is yet-to-be defined function of a ensuring that P(T E Ra) < a. Intuitively, c(a) increases (or remains constant) as decreases. Moreover, because P(T E [t CJ) if C and IP(T (t C}) 0 as C + oc, we know it must be possible to construct at least one function c(a) so that the regions R. satisfy the requirements of hypothesis test.
(X, Y) > 0). By the definition the hypothesis test and the null hypothesis, we Let = (X,
require P(T E Ra) S for all f [ c, + c] However, notice that for fixed rejection region
Rs and any value 0' [ e, t
1 0 0= 2 +
P(T E Rs 0 = 00) SP TE Ra
(10)
Hence, we have set R (1 C}2 where C is some constant, valid rejection region for any a such that
a z P T E R 0 2 tc)-
(11)
Now, construct function c(a) such that P(T E Ra)
S a, and ensure that c(ap) tobss where
Op = P T tobs 0 0= +
(12)
and c(a) c(ap) for any a ap. We can construct one such function, for example, by letting Cpp which is admitted by Eq. 11, and by choosing valid values for any
c(a:) = tabs for
< Cp. which is feasible because P(T 2 C²) - as C + co. Under this setup, the p-value is
ap. This follows from the fact that tobs E Rap {t 2 tobs } but for any a < ap, tobs & Ra as
daa: > c(a,) = Lobs-
Finally, compute the p-value analytically by using the Bernoulli distribution for 1(A(T,, M)) 0} with parameter 1/2 +
ap
Σ KE[m Yobs
(m k
& +e)
(1 (1-2--)
n
(13)
An important clarification is that could have chosen the test statistic to be the estimated value of namely, Σ, 1(A(T, /n)/nn and change the rejection region. the main text we choose to express it this way for clarity of exposition.
B.2 Quantile Test
We provide the details of the quantile test used for testing sufficiency, partial necessity, and minimality
in Algorithm
I
We state it generally but assume that it would be instantiated for each of the above
15

---

cases. Throughout, we assume we are interested in a random quantity Z and want to compare it to target value z.. We only use < and for expository purposes.
For sufficiency, the test corresponds to I(.) = -.) For partial necessity, it corresponds to - -M,,)--,,,C
Algorithm 1: Tail Test
Input: Population distribution Pz, target quantity Z", quaatile g" number of random
samples n, alternative hypothesis direction: > or <, comparison direction: > or <, significance level
Output: The p-value and test statistic
1 t + 0;
2 for = I...., do
Zi Pzi
4
if comparison direction is then
- + Z,) /n;
// will be the test statistic
6
else
1 1 1 74) /n;
p-value t Compute the p-value with binomial test with successes, n trials, probability of success q², and significance level a using the alternative hypothesis direction;
return p-value, t
B.3 Independence Test
We provide details for the independence test used for the partial necessity test. To measure the independence between two variables, we use the Hilbert Schmidt Independence Criterion (HSIC) [Gretton et 2007]. HSIC is nonparametrie measure of independence between two random variables. It is based on the idea that if two random variables are independent, then the cross- covariance between the two variables should be zero. It accounts for the nonlinear relationship between the two variables by mapping them into reproducing kernel Hilbert space (RKHS) and computing the cross-covariance the RKHS.
Definition (Hilbert-Schmidt Independence Criterion (HSIC)), Let K v) f(ö(z, v)/p) denote a kernel function such as the RBF kernel. Let p be a positive parameter called the bandwidth. The Hilbert Schimit Independence Norm is defined as the trace of the covariance between X and Y in the kernel space,
[C((,, w)
tr tr(kkTg
(14)
A higher HSIC value indicates stronger relationship between the variables.
16

---

The permutation test used for the
independence test
is detailed
in
Algorithm 2
Algorithm 2: Permutation Test
Input: Candidate circuit C", dataset D, score function s, bandwidth p. number of random samples B
Output: p-value
2 Sc- (x1()))))) (1)................ 8 (an),, (Z 4 fobs t SM.):
SM (1)
6 I +- 0;
for j ,,... B do
8
MM + permute Sad):
{(0) t HSIC 80- M.rP) P)
(i)
t 1 {1(0) > Lohs }:
p-value :;;
// Approximate p-value
return p-value;
17

---

C
Minimality
We give graphical example of the minimality in Fig.
pre (Nene)
book (pt
blokks O aalii Fro
[5]
blocks 21
Input.(0)
blocks. 1att book result.[]
hook
Impurip w
m.2.bokk
cke. 1. 1.bokk kLinpul [2]
hook
ho P
b cks. 1.ttt Look resu Slokks 3 neok a (pp.
3.ocka 3.neek inpuL(0)
pouut [9]
#1 skcs. heak n jninput
blocks.3 alto hook (ot
boo 3 beck (nput((6]
blocks. al
K. [essuit
D ks. 3. alth hook (e)
blocks. 3 hock post (None]
Figure 4: Example of one step of the minimality for the Docstring task: comparing knocking out a single edge of the candidate circuit (orange edge) against comparing knocking out a random edge of a randomly inflated circuit (the randomly added path blue, the knocked out edge in the added path is red), Minimality tests whether knocking out the random red edge is more significant than knocking out the orange candidate edge.
18

---

D Experiment Details
D.1 Task Description
Below, we provide an extended description of each task and how the experiments were performed.
Indirect Object Identification (IOI) Wang et al. [2023). The objective in IOI is to predict the indirect object in sentence containing two subjects with an initial dependent clause followed by the main clause. For example, in the sentence "When Mary and John went store. John gave an apple to" the correct prediction is Mary. We use the dataset provided by Wang et al. [023) following the
structure above. The score used is the logit difference between the correct subject (Mary) and and the
incorrect subject (John), and the distribution used to perform STR patching is one where the subjects are replaced for different names, not the sentence. For example, in the sentence above this could be: "Sarah and Jamie went to the store. April gave an apple to." 'he candidate circuit we evaluate is the one discovered by Wang et al.
2023
GPT-2 Radford et 2019].
Induction Elhage et al. 2021
The objective for induction is to repeat the completion of a sequence of tokens that previously appeared in the context. For example, in the sentence, "Vernon Dursley and Petunia Durs" the goal is to predict "ley". The score is the log probability assigned to the correct token. We use the dataset provided by Commy et al. [023] which contains 40 sequences of 300 tokens from the validation split of OpenWebTers Gokaslan and Cohen (019) filtered to include instances of induction. The circuit we use corresponds to the eircuit discovered by Conmy al.)
2023] using zero ablation 2-layer 8-head attention-only Transformer trained Open WebTett Consequently, we also use zero patching for the experiments with this model.
Docstring [Heimersheim and Janiak 2023
The objective for the Doestring task is to predict the next variable name inside docstring. For example,
def port load, size, files, last): """oil column piece
:aaram load: crime population
:param size: unit dark : :aaaa
the model should predict the completion files. We use the dataset provided by Heimersheim and Janiak (2033 following the structure above. The STR dataset uses the same input but with the parameter names switched for different ones not in the function. For example, the corrupted input corresponding to the example above replaces load by user and size by context. Following Heimersheim and Janiak [2023.. for the score we use the logit difference between the correct ans wer and the most positive logit over the of alternative arguments, including the ones used for the corrupted example. The circuit we use corresponds to the one provided by Heimersheim and Janiak (2023) which specified over a 4-layer attention-oni Transformer trained on natural language and Python code.
Greater-Than (Hanna et al. 2023
The greater-than task requires performing the greater operation as it appears in natural language. For example, it asks that sentences such as "TTh demonstrations lasted from the year 1289 to the year 12". are completed with tokens representing two-digit numbers between "89"and and "99". Following Hanna et al. [2023], we use as the score function the difference in probability between the two-digit tokens satisfying the relation and those that don't. 't. For STR patching, we use corrupted datapoint, which replaces the last two digits of the first year with "01". In the example above, this would imply changing "1289" to "1201" ". The circuit we evaluate for this task is the GPT-2 subgraph provided by Conmy et al. 2023 as simplification to the original provided by [Hanna et al. 2023).
Tracr Lindner et al. 2024]. Tracr is compiler for RASP et al. [2021]. simple language expressing a computational model for Transformers. We use Tracr as by design provides us with "ground truth" circuit which allows us to verify the performance of our method. We study two tasks. The first task tracr-r, consists of reversing a small sequence of tokens. The second tasks tracr-p consists in computing at each position the proportion of tokens corresponding to that
19

---

have been observed. The sequences <bos> 1 2 3 and <bos> x с respectively correspond possible imput sequences for the tasks. The sequences <bos> 2 and <bos> 1.0 0.5 0. 0.5 correspond to the desired outputs respectively. Following Following/Conmy et al. [023], the score used for both tasks is the sum of token-level ( distances between the desired and produced outputs. For the evaluation dataset we use all permutations of the sequence 2 3 for tracr-r, and 50 random 4-character-long sequences consisting of characters in (x, a, c} for tracr-p. We use zero ablation for both tasks.
D.2 Software
As part the paper, we created the circuitry package as wrapper around the TransformerLens library, which abstracts away lower-level manipulations of hooks and activations. For given model, the user specifies circuit as subset of nodes and edges and selects an ablation strategy and dataset. The user can then evaluate model performance with respect to the circuit. Our package is implemented efficiently, and can evaluate hundreds of circuits in a few minutes on a single A5000 GPU, The software is available at stlhtts::// :/gg::://i/ ht/gttttbb.... thub. commblei-lab/circutte lab/circuttry
20

---

E
Additional Results
E.1
Equivalence
The equivalence test evaluates whether the candidate circuit outperforms the original model at least half of the time. As shown in Table I] none of the natural circuits passed the equivalence test. Table show the test statistics the proportion of inputs where the candidate circuit outperforms the original model of all tasks. All circuits except IOI are much worse than the original model at the task. This may be because circuits are only small proportion of the original model. We omit the Tracr-based tasks because their performance is identical the original model by design (they are ground truth circuits). Thus, in their case, although the null hypothesis is true, the sign test can't be applied.
Tracr-P Tracr-R
Induction 101
G-T
DS
0.03
0.24
0.07
0.1
Table The proportion of times C* outperforms M the task. Results for Tracr-based tasks omitted as the performance of the circuit is the same as the original model.
E,2 Independence
For the independence test, we consider retaining the null as passing the test. As shown in Table 1, the only natural circuit that pass the independence test is induction heads, Tracr, the ground truth. circuit does.
Tabll 4 reports the results.
G-T
Induction
I0I
DS
Tracr-P
Tracr-B
HSIC
0.000
0.001
0.001
0.001
0.000
0.000
p-value
0.000
0.669
0.01
0.01
1.000
1.000
Table 4: The HSIC and p-value of the independence test.
E.3 Minimality
To produce the results in Table I we set q" 0.9 and if there exist any edges deemed insignificant, we reject the null hypothesis that the candidate circuit is minimal. We find that only the Induction and Tracr circuits pass the minimality test.
In Fig. 5
we plot the scores for knocking out each edge for each circuit. As the Tracr circuits are ground truth circuits, all edges significant relative to the reference distribution.
For the Induction circuit, all edges are also significant relative to the reference distribution. However, for the other circuits, we find that significant portion of the edges are insignificant. This is especially prevalent for the DS circuit, where less than half of the edges are significantly different from the reference distribution. This suggests that other than the Induction circuit, these circuits are not minimal. Unsurprisingly, for the IOI circuit, we see few edges that can removed with little impact to performance, in agreement with Wang al. 2023
21

---

Threshold under which individual edges are not significant
Change in score, 5(e, C)), when removing from C*
Tracr-P
Tracr-R
0.05
0.1
0.00
0.0
2
5
5
11
100
10-2
50
10
Induction
101
0
1
6
1
481
963
DS
10-4
4
10-10
G-T
117
235
1
12
24
Edge e in the circuit (sorted by their y-axis value)
Figure 5: The main figures display the change in task performance score induced by knocking out edge e, for every in each circuit. The changes score are sorted from low to high along the x-axis. The right-adjacent vertical histograms show the change in task performance scores of the reference edges (ranging from 1000 to 10, 000 edges). The shaded region covers the individual edges with corrected p- paluus that are below the significance threshold.
22

---

Impact Statement
We present suite of statistical tests assess whether existing circuits align with the idealized version of circuits. When utilized appropriately, these tests can help identify new circuits, improve existing circuits, and compare the quality across circuits. Thus, we expect these circuits reported in the mechanistic interpretability literature, making them more aligned with the idealized criteria.
improve the quality of
We anticipate that the overall effect of this work will be to accelerate progress in mechanistic interpretability and consequently improve understanding of how LLMSs work. This should facilitate explaining and steering model behavior, and possibly "debugging" learned models. At the same time, improved understanding model internals may enhance architectures and accelerate capabilities, Additionally, it can open the door to more sophisticated attacks and defenses for various threat models.
important to note that our methodology is based on a hypothesis testing framework. Similar to other hypothesis-bast based tests, there is potential for misuse engagement in practices such as p-hacking by practitioners. Misapplication of these tests can lead to misleading assurances of robustness that the circuits might genuinely possess. Furthermore, assume fixed experimental setup rather than considering generalization across different setups. The inferences we draw across experimental setups can differ significantly.
these tests are used to check mechanistic interpretability results for an application of AI, they may give users developers misplaced sense of confidence in faulty hypothesis about neural network internals. However, we believe that this already a danger with present results, and our work an improvement in this regard.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction make claims about the empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation discussed throughout section 3, section 4, and section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
The paper has no theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Yes, we provide associated codebase https://github.. com haemmbbeii-ae lab/
circuitry
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
Yes, we provide a codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuiu wwwticccguie//eeeeeeeeeii CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See experiment results and associated codebase.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper is about statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: It's included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: pettps://nerrre http:///eeiis..eeeeeeeee httpr//s....cpulltttttcccccuceee
Answer: [Yes]
Justification:
We preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have section potential social impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Not applicable
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
28

---

Answer: [Yes]
Justification:
The codebase is well documented.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "AiXh9ek0Hg", "reviewer": "Reviewer_wEmx", "summary": "The paper operates in the framework of mechanistic interpretability of transformer models, where it is assumed that subgraphs (i.e. circuits) of the computational graph determined by the model implement specific capabilities of the latter. It defines a set of properties that an ideal circuit should have, namely equivalence, independence and minimality. Afterwards, it proposes two sets of hypothesis tests. The first set identifies a more strict hypothesis-testing framework to determine if a circuit satisfies these properties. The second proposes some more flexible tests for the first two properties of ideal circuits. Afterwards, these hypothesis tests are used on synthetic and manually discovered circuits in the literature on GPT-2 small and other small transformers.", "questions": "- I believe there is a typo in the description of Table 1 on page 7, where it is claimed that “A (✓) indicates the null hypothesis is rejected”, while I believe that this means that the circuit “passed the test” so that the null-hypothesis is not rejected.\n- When referring to the Bonferroni correction, was the base one used or the Bonferroni-Holm one? The latter is always more appropriate as it is less conservative and leads to a more powerful test.", "limitations": "-", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- Overall very well-explained paper, particularly the part on mechanistic interpretation.\n- Crafting of the hypotheses for testing is original and well thought. It follows logically from the properties defined and is well explained.\n- Different types of tests were performed with different granularity.", "weaknesses": "- I believe it would have been nice to perform further tests to understand whether the idealized tests have some applicability on discovered circuits (or if they always lead to the null being rejected), potentially by also altering the discovered ones.\n- Would have found it interesting to propose some ideas on how to use these tests for circuit discovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n**I believe it would have been nice to perform further tests to understand whether the idealized tests have some applicability on discovered circuits. (or if they always lead to the null being rejected), potentially by also altering the discovered ones.**\n\n**Would have found it interesting to propose some ideas on how to use these tests for circuit discovery.**\n\n\nThank you for bringing this up! We agree that these tests can be applied to improve circuit discovery algorithms, such as ACDC [1].\n\nFor example, it would be interesting to apply the minimality test for edge pruning in Algorithm 1 of ACDC [1]. In ACDC, the edges are pruned by checking that removing an edge does not increase the KL divergence between the outputs and the current circuit by more than a threshold, $\\tau$. This threshold $\\tau$ is treated as a hyperparameter. In contrast, using the minimality test, we can determine the importance threshold in a principled way.\n\n\n\nHowever, we omitted further discussion of these ideas because they are not the main focus of this paper, which is to formalize the circuit hypothesis, develop appropriate tests, and apply them to study the extent to which the circuit hypothesis holds. Nevertheless, we believe that applying our proposed tests in novel circuit discovery algorithms is an exciting area for future research.\n\n\n\n## Questions\n\n**I believe there is a typo in the description of Table 1 on page 7**\n\nThank you for pointing out the typo; you are absolutely right. We have updated the paper to reflect that.\n\n**When referring to the Bonferroni correction, was the base one used or the Bonferroni-Holm one?**\n\nThank you for the great suggestion. We used the Bonferroni correction in the paper because it is easier to explain, but you are right that Bonferroni-Holm is uniformly more powerful. We have made a note about it in the paper and will incorporate it in the paper's code package. In our experiments, we explored another less conservative method, the Benjamini–Hochberg method, but we did not observe a difference in the main findings. So we opted for Bonferroni correction for simplicity.\n\n## References:\n\n[1]  Arthur Conmy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, & Adrià Garriga-Alonso. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G0TlQkMGV2", "reviewer": "Reviewer_NNjL", "summary": "This paper proposes a set of tests to evaluate how well a \"circuit\" meets its desired properties.\n- Here a circuit refers to a subnetwork, which could be either synthetic (e.g. constructed according to RASP) or discovered in a trained model.\n- The desired properties considered in this paper are:\n  - *faithfulness*, where the circuit should preserve the performance of the original model;\n  - *localization*, where removal of the circuit should alter the model's output;\n  - *minimality*, which says the edges in a circuit (treated as a computational graph) should not be redundant.\n\nThere are 3 \"idealized tests\" intended as necessary (but not sufficient) conditions:\n- **Equivalence**, where the null hypothesis is that the circuit $C^*$ preserves the performance of the original model $M$.\n  - The test checks whether $C^*$ and $M$ have an equal chance of outperforming each other, which is a necessary condition (but not sufficient): rejecting the null means that $C^*$ necessarily does not preserve the performance.\n- **Independence**, where the null hypothesis is that the output of the model after removing $C^*$ is independent of the output of the original model.\n  - The test computes the HSIC between the performance of $C^*$ and $M$.\n  - Note that this is a stringent requirement; this will later be modified into a \"flexible\" test.\n- **Minimality**, where the null hypothesis is that all edges in $C^*$ are necessary and not redundant.\n  - The test checks whether the amount of performance change by removing an edge in $C^*$ is more than the amount of change caused by removing an edge that is believed to be redundant.\n\nThe paper also proposes 2 \"flexible tests\", where the difficulty of the test can be gradually varied by choosing different reference distributions when computing the test.\n- **Sufficiency**, which computes the probability (over random circuits in a reference distribution) that $C^*$ is more faithful than a random circuit.\n- **Partial necessity**, which checks whether with high probability, removing $C^*$ leads to a worse performance than removing a random circuit.\n\nThe paper then applies these tests on 2 synthetic circuits and 4 circuits manually discovered from trained Transformers.\n- Synthetic circuits: for 2 types of the Tracr task, where the circuits are given as RASP programs.\n- Discovered circuits: Indirect object identification (IOI), Induction, Docstring (DS), Greater-Than (G-T).\n\nIt finds that the synthetic circuits pass all 3 tests.\nIn contrast, the discovered circuits fail the test to various extents, but are still far from random.\nMoreover, the flexible tests provide a more fine-grained understanding than the idealized tests (with binary outcomes).", "questions": "- About independency: neural networks are often overparameterized and hence likely contain a high level of redundancy. What if there are multiple circuits that are each faithful and minimal but are similar to each other? These seem to me should be considered as valid circuits, but they would violate independency.\n- About minimality, I don't see why the randomly added edge would lead to a small $\\delta(e^I, C^I)$: it's possible that adding a random connection would change the performance non-trivially (but likely negatively). For example, even a simple residual term would change the scale of the output and affect subsequent computations.\n- Could you comment on the effect of different choices of activation patching?\n- Could you comment on how the proposed tests can inform interventions on training (e.g. as regularization terms)? The hope is that this could make the trained networks more likely to contain circuits that more closely satisfy the desiderata.\n- Can the proposed tests help with coming up a more precise definition of circuits, or inform how we should choose the granularity of the definition of \"nodes\"?\n\nMinor clarifications:\n- Eq (2): should the LHS be taking an absolute value?\n- Line 246: not sure I understand what “a lucky draw” means; e.g. it could mean a draw that is better than q* fraction of random circuits?\n- Line 250: what does \"supersets of C\" and \"comparable to C\" mean?", "limitations": "The paper discusses the technical limitations.\n\nThere is no direct societal implication.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a quantitative way to evaluate the concept of \"circuit\" in models, by adopting the hypothesis testing framework. The proposed tests correspond to desired properties a circuit.\n- The proposed tests are more thorough and fine-grained than some existing evaluation criteria, such as the knockdown effect.\n- The paper are clearly written.", "weaknesses": "- I'm concerned about the practical applicability of the tests.\nAs the paper also mentioned, the proposed tests are either too stringent, or can be sensitive to the circuit size and the choice of the reference distribution.\n- The implications and use cases of the tests could be better discussed; e.g. please see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n**I'm concerned about the practical applicability of the tests (...) can be sensitive to the choice of the reference distribution.**\n\nThank you for raising this important point! We believe that the main objections stem from a small but important misunderstanding of the paper's main goal.\n\nThe review states:  \n> This paper proposes a set of tests to evaluate how well a \"circuit\" meets its desired properties.\n\nWe politely disagree with this goal. Our goal is to formalize the implications of the circuit hypothesis---\"LLMs implement tasks through circuits.\" We demonstrate consistency with synthetic circuits, and we then study how existing circuits align with these idealized properties. Our goal is not to create a checklist that evaluates which circuits are better or worse.\n\nThis subtle but important difference leads to different interpretations of the results. If a circuit does not align with the circuit hypothesis, it does not mean the circuit is \"undesirable\"; rather, it could indicate that the idealized version of circuits differs from how neural networks actually encode tasks. Our paper aims to provide tools to quantify the extent to which the idealized circuit hypothesis aligns with circuits in practice.\n\nThus, the practical applicability of our tests is to provide a nuanced understanding of the alignment between the circuit hypothesis and discovered circuits. Currently, circuits found in transformers rarely pass the stringent tests due to issues like the redundancy mechanism, as the reviewer correctly points out. However, future models could exhibit more modular and circuit-like behavior. Our tests can help determine when this happens.\n\n## Questions\n\n**What if there are multiple circuits that are each faithful and minimal, but are similar to each other?**\n\nWe agree that neural networks implement redundancy, making the independence test difficult to pass. This suggests that the network is not implementing the tasks similarly to how the idealized version of  \"circuits:' and neural networks behave. Our tests help illustrate how the neural network in practice does not align with the idealized circuit hypothesis.\n\n**About minimality, I don't see why the randomly added edge would lead to a small $𝛿(𝑒^𝐼,𝐶^𝐼)$**\n\nThank you for the question. A key component of the circuit hypothesis is that edges not part of the circuit can be removed without significantly affecting model performance, hence our design of the minimality test. While \"removing\" is a common term, this can be more complex than simply setting values to zero, as is the case with STR patching [1]. This is detailed in lines 109-124 of the paper. This ablation method is chosen to maintain the edge magnitude and avoid the issues you described. Further clarifications on activation patching are provided in the question below.\n\n**Could you comment on the effect of different choices of activation patching**\n\nCongruent with existing findings [1], we found that the circuit is indeed sensitive to the ablation scheme that was used to “discover” it. This is illustrated in Figure 6 of the paper. We present the results with the original method used to discover the circuit with the aim of fairness. Appendix D.1 clarifies which ablation methods were used with which dataset.\n\n**Could you comment on how the proposed tests can inform interventions on training**\n\nThank you for the question! These tests can indicate if certain training methods or architectural variations are more likely to produce idealized circuits. For instance, if circuits in a transformer trained with a specific regularizer align more with the tests, this suggests that this method is more effective at producing circuits.\n\nHowever, it is unclear how to incorporate these tests during training. These tests are designed for specific \"hand-crafted\" tasks $\\tau$, while pre-trained models are not trained this way. Additionally, systematizing the creation of many such $\\tau$ or deriving a differentiable loss from the tests remains a challenge.\n\n**Can the proposed tests help with coming up a more precise definition of circuits?**\n\nThank you for the question! It depends on what you mean by this.\n\nWhen we consider the Transformer and related architectures as computation graphs, any meaningful definition of a circuit is likely to be similar to the one we have provided. This similarity arises because the computation graph framework inherently constrains how circuits can be defined. Consequently, it would be challenging to find a more precise definition of a circuit within this framework without it closely resembling our existing definition.\n\nHowever, if you are asking about the level of granularity at which we apply the definition (e.g., interventions at a node level vs. edge level), then we think our tests would be useful. This would certainly be an interesting experiment to explore in future work.\n\n## Minor Clarifications\n\n- **Eq (2): should the LHS be taking an absolute value?**\n    Yes! Thank you for pointing it out!\n\n- **Line 246: not sure I understand what “a lucky draw” means**\nWe mean the following:  \n    If with a small probability (e.g., 10%), we could have randomly drawn a circuit just as faithful as the candidate circuit, then the candidate circuit is simply a \"lucky draw\" from the reference distribution. If the candidate circuit is significantly more faithful than the 90% most faithful random circuit, then it is not just a lucky draw. \n\n- **Line 250: what does \"supersets of C\" and \"comparable to C\" mean?**\nA circuit is defined as a set of nodes and edges. A superset of C is a circuit that contains all edges and nodes in C. For a circuit to be \"comparable to C\" means that it will have similar faithfulness to C. We expect a superset of C to be comparable to C.\n        \n## References\n[1] Fred Zhang, & Neel Nanda. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogf8yf5Fv7", "reviewer": "Reviewer_6etf", "summary": "This paper considers the problem of formalizing and evaluating the circuit hypothesis. This hypothesis posits that specific subnetworks within a Transformer model are responsible for specific model behaviors. Although there are multiple examples of such circuits that have been manually discovered in the literature, the question of how to precisely judge whether a discovered circuit is indeed responsible for a specific behavior remains open. While past works have proposed various ad-hoc approaches, this paper first presents three criteria that an ideal circuit should satisfy (inspired by past works) along with corresponding statistical hypothesis tests for judging if a given circuit indeed satisfies these criteria. The paper then further presents weaker versions of the hypothesis tests for two out of the three criteria that are more likely to be satisfiable by circuits discovered in practice. These hypothesis tests are then applied to six different circuits that have been proposed in the literature to evaluate the extent to which they satisfy these criteria.", "questions": "Please address the questions listed in the previous section.", "limitations": "Yes, the paper discusses the limitations and the potential impacts of the proposed work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is a nice paper. It is well-written and addresses a question of growing importance. Mechanistic interpretability has emerged as a promising approach for understanding the behavior of trained LLMs. However, given the infancy of the area, the definitions and criteria for evaluating the quality of a mechanistic interpretation are yet to be standardized. This paper fills this need, particularly in the context of circuit analysis. The three proposed criteria of preservation, localization, and minimality make intuitive sense. The experiments with the synthetic circuits also provide evidence in favor of these criteria. I particularly like the proposed use of hypothesis tests to evaluate these criteria. These tests provide an operational definition of an ideal circuit and can be feasibly used in practice for judging new circuits. Overall, I think this paper helps bring some formal discipline to the topic of circuit analysis.", "weaknesses": "I have a couple of concerns.\n\nFirst, the hypothesis test for judging equivalence (Equation 2) feels somewhat unintuitive. Consider the scenario where the circuit C* significantly outperforms the model M on half the inputs and vice versa. Intuitively, one would not consider such a circuit equivalent to the model, yet the hypothesis test would conclude otherwise. Why not define equivalence simply in terms of faithfulness of the circuit C* to model M?\n\nMy second concern is the possibility that the three proposed criteria are insufficient for characterizing an ideal circuit, i.e., there might be additional criteria that are needed. However, given that circuit analysis is in its early days, I think the paper already makes useful contributions.\n\nI also have a number of minor comments and questions that I list below:\n1. Line 180: Why use ; instead of , in s(C*(x);y)?\n2. Equation 2: I think there is a missing abs operation. The definition of the null hypothesis in Appendix B.1 further suggests this.\n3. Equation 2,3: Notationally, are lower case and upper case letters being used to represent separate things? For instance, x,y vs X,Y? Why not use lower case x and y in these equations?\n4. Equation 5: In practice, there will be randomness also due to the fact that \\delta(e,C) is calculated empirically. Does the test account for this randomness?\n5. Line 267: The notion of a \"reference distribution over circuits from the complement distribution\" is not very clear at this stage. I understand what it means based on reading the rest of the paper.\n6. Figure 3: The results for the IoITask are a little mysterious. How is the faithfulness of a circuit with some edges removed worse than an empty circuit? Further discussion would be helpful here.\n7. Is Algorithm 2 an original contribution or was it presented in Gretton et al., 2007? Some more explanation and intuition about the algorithm would help the reader.\n8. It would help to have a more careful comparison with the three criteria for circuits proposed by Wang et al, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weaknesses\n\n**First, the hypothesis test for judging equivalence (Equation 2) feels somewhat unintuitive. Consider the scenario where the circuit $C^*$ significantly outperforms the model $M$ on half the inputs and vice versa. Intuitively, one would not consider such a circuit equivalent to the model, yet the hypothesis test would conclude otherwise. Why not define equivalence simply in terms of faithfulness of the circuit $C^{*}$ to model $M$?**\n\nThank you for raising this great point!\n\nWe agree that the test above does not handle this case. However, we intend our suite of tests to be used jointly. The situation you describe would be adequately handled by the sufficiency test, as the circuit you described would surely be unfaithful.\n\nHowever, we think this test complements a test that only looks at faithfulness for the following reasons:\n\n1. It can distinguish situations where there is no bias (equally outperform each other) from situations where there is, even when the model is very faithful.\n2. It is lenient to added variance, which could happen, for example, because we are using patching.\n\nWe believe these properties make the test a valuable addition to the suite, as it helps to clarify precisely how the circuits are close or not to the model.\n\n\n\n**My second concern is the possibility that the three proposed criteria are insufficient for characterizing an ideal circuit, i.e., there might be additional criteria that are needed. However, given that circuit analysis is in its early days, I think the paper already makes useful contributions.**\n\nWe agree! We hope future work builds on these criteria.\n\n## Minor comments and questions\n\nThank you so much for the careful read! Your feedback is greatly appreciated.\n\n- **Line 180: Why use ; instead of , in $s(C^{*}(x);y)$?**\n\n    That’s a great question! This is a typo --- it came about because $y$ can be a parameter of the task, rather than the classic label. For example, it can be the index of the subject and object in IOI. We spent a fair bit of time debating whether to treat that as a parameter with `;` or a variable which we denote with `,`. However, we realize we have been inconsistent with the notation and have changed all of them to `,`.\n\n- **Equation 2: I think there is a missing abs operation.** \n\n    Yes. Apologies about that. \n\n- **Equation 2,3: Notationally, are lower case and upper case letters being used to represent separate things?** \n\n    It is a typo. We have changed them to lowercase.\n\n- **Equation 5: In practice, there will be randomness also due to the fact that $\\delta(e,C)$ is calculated empirically. Does the test account for this randomness?**\n\n     This is a great question! Randomness in a dataset would definitely introduce additional variance. We sidestep this issue by defining the task with respect to a fixed dataset. But a different test could be designed to account for a finite dataset. In the flexible tests, we only account for the randomness arising from the sampled circuits and edges. \n\n- **Line 267: The notion of a \"reference distribution over circuits from the complement distribution\" is not very clear at this stage.**\n\n    Thank you for pointing this out! We’ve rephrased it as ''reference distribution over circuits from the complement distribution, i.e., circuits that do not overlap with the candidate circuit.'' Is this clearer?\n\n- **Figure 3: The results for the IoITask are a little mysterious.**\n\n    We believe this is because of the ''Negative Name Mover Heads'' discovered in the original IOI paper [1], which write in the opposite direction of the name mover heads. \n\n- **Is Algorithm 2 an original contribution or was it presented in Gretton et al., 2007?**\n\n    Algorithm 2 has two components, the permutation test and HSIC. The permutation test is a classic hypothesis test by Fisher [2] and Pitman [3]. It is a nonparametric test to aim to show whether the observed statistics could be drawn from the “permuted distribution”. In our setup, we randomize the complement circuit from the observed y -- creating an independent null. If the observed statistics fall outside of the independent null, then we reject the hypothesis. The HSIC is proposed measures in Gretton et al [4] to measure the independence between two variables in the kernel space.\n\n- **It would help to have a more careful comparison with the three criteria for circuits proposed by Wang et al, 2023.**\n\n    Yes, we absolutely agree. It is discussed in the extended related work in line 486, Appendix A.\n\n## References\n[1] Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, & Jacob Steinhardt. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small.  \n[2] Fisher, R. A. (1935). The design of experiments. New York, NY: Hafner.  \n[3] Pitman, E. J. G. (1937). Significance tests which may be applied to samples from any population. Journal of the Royal Statistical Society. Supplement, 4, 119-130, 225-232.  \n[4]  Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., & Smola, A. (2007). A Kernel Statistical Test of Independence. In Advances in Neural Information Processing Systems. Curran Associates, Inc.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful reviews and support of the paper.

We are pleased to see that the reviewers find the paper well-explained, with original and well-considered hypothesis tests (Reviewer wEmx); that it is clearly written, offers a quantitative approach to evaluating the concept of ''circuit'' in models, and that proposed tests are more thorough and fine-grained than some existing evaluation criteria (Reviewer NNjL); and that it addresses a question of growing importance, bringing formal discipline to the topic of circuit analysis (Reviewer 6etf).

We address the questions individually below.

Meta-review: This paper describes a novel approach for testing the hypothesis that specific functions in LLMs are carried by small subnetworks (i.e., circuits).

There was unanimous agreement among the reviewers that the approach is novel and interesting.
Some concerns were raised including the practical applicability of the approach but the rebuttal appears to have addressed all the reviewers' concerns.

All in all there appears to be sufficient support from the reviewers in favor of accepting this paper for it introduces a novel formalization and evaluation method for the circuit hypothesis in LLMs. Hence, the AC recommends the paper to be accepted.
Decision: Accept (poster)