Uncovering Intermediate Variables in Transformers using Circuit Probing
Authors: Michael A. Lepori
Thomas Serre
Ellie Pavlick
Venue: COLM (2024)
Keywords: Mechanistic Interpretability, Deep Learning
TL;DR: We introduce circuit probing, a novel technique that automatically uncovers low-level circuits that compute hypothesized intermediate variables.

Abstract: Neural network models have achieved high performance on a wide variety
of complex tasks, but the algorithms that they implement are notoriously
difficult to interpret. It is often necessary to hypothesize intermediate variables involved in a network’s computation in order to understand these
algorithms. For example, does a language model depend on particular
syntactic properties when generating a sentence? Yet, existing analysis
tools make it difficult to test hypotheses of this type. We propose a new
analysis technique – circuit probing – that automatically uncovers low-level
circuits that compute hypothesized intermediate variables. This enables
causal analysis through targeted ablation at the level of model parameters.
We apply this method to models trained on simple arithmetic tasks, demonstrating its effectiveness at (1) deciphering the algorithms that models have
learned, (2) revealing modular structure within a model, and (3) tracking
the development of circuits over training. Across these three experiments
we demonstrate that circuit probing combines and extends the capabilities of existing methods, providing one unified approach for a variety of
analyses. Finally, we demonstrate circuit probing on a real-world use case:
uncovering circuits that are responsible for subject-verb agreement and
reflexive anaphora in GPT2-Small and Medium.

Published as a conference paper at COLM 2024
Uncovering Intermediate Variables in Transformers using Cir- cuit Probing
Michael A. Lepori*
Thomas Serre
Ellie Pavlick'
(michael lepori, thomas serre, ellie pavlick)@brownr edu
Abstract
Neural network models have achieved high performance on wide variety of complex tasks, but the algorithms that they implement are notoriously difficult to interpret. It is often necessary to hypothesize intermediate vari- ables involved a network's computation in order to understand these algorithms. For example, does language model depend on particular syntactic properties when generating sentence? Yet, existing analysis tools make it difficult to test hypotheses of this type. We propose a new analysis technique circuit probing- - that automatically uncovers low-level circuits that compute hypothesized intermediate variables. This enables causal analysis through targeted ablation at the level of model parameters. We apply this method models trained on simple arithmetic tasks, demon- strating its effectiveness at (1) deciphering the algorithms that models have learned, (2) revealing modular structure within model, and (3) tracking the development of circuits over training. Across these three experiments we demonstrate that circuit probing combines and extends the capabili- ties of existing methods, providing one unified approach for variety of analyses. Finally, we demonstrate circuit probing on real-world use case: uncóvering circuits that are responsible for subject-verb agreement and reflexive anaphora in GPT2-Small and Medium.
Introduction
Transformer models are the workhorse of modern machine learning, driving breakthroughs in subfields as disparate as NLP (Devlin et al., 2018 Radford et al. 2019: Brown et al. 2020)
computer vision (Dosovitskiy al. 2020), and reinforcement learning (Chen et al. 202T Despite their success, little is known about the algorithms that they learn to implement.
This central problem has inspired flurry of analysis and interpretability research trying
to "open the black box" (Rogers et al., 2021 Elhage et al. 2021 Belinkov 2022). Despite considerable effort, these models remain almost entirely opaque.
One challenge inherent to interpreting a model that succeeds at complex task is that researchers often do not have a complete picture of the algorithm that they are attempting to uncover. However, they may be able to propose high-level causal variables that are constituents of such an algorithm. For example, one may intuit that computing the syntactic number of the subject noun of a sentence might be useful for language modeling (Chomsky) 1965; Linzen et al., 2016). This variable must be causally implicated in an algorithm that solves the language modeling task, it constrains the rest of the sentence due agreement rules (i.e. the syntactic number of the main verb must match the syntactic number of the subject). However, it leaves open infinite possibilities in which other variables influence the prediction of the next token. Though we focus on language modeling, this discussion applies more generally to any complex domain where neural networks are applied, from
vision (Dosovitskiy et al. 2020) to astronomy (Ciprijanović et al. 2020) to protein folding
(Jumper et al. 2021
"Department of Computer Science, Brown University *Camey Institute for Brain Science, Brown University
1

---

Published as a conference paper at COLM 2024
Input
Updates from Model Component
To
Tm
TN
AAn artist the cars'
"Doctors"
person"
Figure 1: Schematic visualization of circuit probing for an intermediate variable representing the syntactic number of the subject of a sentence. Plural subjects are represented in red and singular subjects in blue. At step To, prior to training a binary mask, the model component (Attention block or MLP) produces residual stream updates (red and blue arrows; Elhage et al. (202T) that are not partitioned by syntactic number (i.e. they will point in seemingly- random directions). Circuit probing optimizes a binary mask over model weights. By the end of mask optimization, the circuit will produce updates that are partitioned by syntactic number (i.e. point in one direction for singular subjects and another for plural subjects).
We propose circuit probing to enable the investigation of intermediate variables in Transform- ers (Vaswani et al. 2017). Circuit probing introduces a trainable binary mask over model weights that is optimized to uncover circuit" that computes high-level intermediate variable (if one exists). This technique combines the best aspects of standard probing and causal analysis methods, allowing researchers to (1) test whether high-level intermediate variables are represented by the model, (2) test whether they are causally implicated in the model behavior (rather than simply decodable from model representations), and (3) reveal the particular subset of model weights that compute it. We first demonstrate the benefits of circuit probing over existing methods using simple arithmetic tasks, showing that it is more faithful to the underlying model than existing methods (i.e. it only provides evidence in support of causal variables that are actually represented by the model) and reliably uncovers circuits that are causally implicated in model behavior. We then use circuit probing to analyze two syntactic phenomena on GPT2-Small and Medium [Radford et al., 2019), uncovering particular circuits responsible for subject-verb agreement and reflexive anaphora agreemen
2 Circuit Probing
When attempting to interpret a model that is performing some task, one must often hy- pothesize the existence of an intermediate variable that the model computing. Circuit probing attempts to (1) measure whether this intermediate variable is computed and (2) identify the components of the model that are responsible for computing (i.e. a circuit). Similar to prior work (Conmy et al. 2023 Cao et al. 2021), 1,, attempt to to uncover model components by optimizing a binary mask over frozen model weights.
Recent work has shown that neural networks often exhibit structure at the level of subnet- works (Csordás et al. 2020 Lepori et al. 2023b Hod et al. 2021). In light of this, we attempt to uncover circuits willun individual attention and ML.Pblocks. These layers produce addi- tive updates to the residual stream (Elhage et al. 2021). Intuitively, circuits that compute an intermediate variable should produce outputs that are partitioned according that variable. For example, if a circuit is computing the syntactic number of the subject noun, then that circuit should produce outputs that fall into one of two equivalence classes, corresponding to singular subjects and plural subjects. Thus, we optimize binary mask such that - if the variable is computed by the model within a particular layer - the outputs of that layer
Similar to prior work Wang et al. 22022) we take the term" "ciccuit" to mean "any subgraph of the computational graph describing a full model". We isolate circuits within individual attention and MI.P blocks, and our circuits are composed of neurons. In general, circuit probing can be applied at any level of granularity, as long as the subgraph produces an update to the residual stream.
We release our code at: https:/ https://githul http//ttthubbooo /mlpporil / "CCrcuut_Probbnngg Circuit probing is imple- mented using the NeuroSurgeon package (Lepori al. 2023a)
2

---

Published as a conference paper at COLM 2024
are clustered according to that variable. A natural optimization objective for this is the soft nearest neighbors loss (See Appendix(D DD), a contrastive loss that minimizes the (cosine) distance between outputs from the same class, and maximizes the (cosine) distance between outputs from different classes (Salakhutdinov & Hinton 2007 Frosst et al. 2019). Circuit
probing is illustrated in Figure
In this work, we mask at the neuron level (i.e. over columns in the matrices of the linear transformations that comprise MLP and attention blocks in the Transformer) in both MLP and attention blocks. We use continuous sparsification (Savarese et al. 2020), a pruning technique that anneals a soft mask into a discrete mask over training, to learn the binary mask (See Appendix C We also train with lo regularization to encourage sparse binary masks. this process is successful, it results in sparse circuit within model component that computes the hypothesized intermediate variable. See Algorithm
1 for pseudocode.
Circuit Probing Evaluation: We evaluate circuit probing in two ways: (1) We train a 1-
nearest neighbor classifier on the output vectors produced by the discovered circuit, and then test this classifier on held-out data. If circuit probing succeeds in finding circuit that computes a particular intermediate variable, then its output vectors will be partitioned by the possible labels of this variable, and we expect this classifier to achieve high performance. We employ rather conservative strategy here, randomly sampling only 1 output vector for each label to train the nearest neighbors classifier See Algorithms for pseudocode. (2) We ablate the discovered circuit (i.e. invert the learned binary mask) and analyze how the model's 's behavior changes.
This ablation is equivalent to asking the counterfactual question, "How does the model's output change if does not compute a particular intermediate variable, z, in a particular block?", assuming that the circuit is only causally implicated in computing z. We note that this assumption unlikely to be completely true, as neurons are typically polysemantic (Elhage et al. (2022). However, (1) we attempt to mitigate this by encouraging maximally sparse subnetworks using Lo regularization and (2) prior work has suggested that sparse circuits may be surprisingly monosemantic (Hamblin et al. 2022
Causality:
Algorithm 1 Training Step
Algorithm Evaluation
Input: Model Ma, Batch B, Probe Layer i, Mask Params m
Input: Model Ma, Train Data D
Test Data Deest Probe Layer i, Mask m
1: Mg = [Mao... Mai Ma(i+1)...Nl
1: M1 = [Mao..((i1)) Moir Ma(i+1)..Nl
2: B = {x,y}
2: Drrain = {x,y), Drest {x'}
3: Freeze (0)
3: v + Marom (Moo.. (M
4: v +- Merom (Mo... (x))
4: knn + train knn (U,y)
5: loss < soft neft.neighbors(t...
D' Moram (Mo....ii (1-1)(())) (x'))
6: loss += ml
D Lo loss
6: 9 4--ppm --edrrddcckkm (kun, v')
7: backpropagate (loss, m)
Baselines
Circuit probing accomplishes two distinct goals: (1) it allows one to measure how well circuit partitions inputs according to an intermediate variable (i.e. it provides a probing accuracy measurement), and (2) it allows one to perform causal analysis by ablating circuits. Prior methods accomplish either one of these goals, but not both. Thus, we use different baselines for each type of analysis. When analyzing probing accuracy, we compare circuit probing to 'vanilla probing" (henceforth "probing") and contrastive probing. Probing involves learning a classifier to decode information about a hypothesized intermediate variable from model activations (Tenney et al. 2019 Hewitt & Manning) 2019 Ettinger 2020 Li et al. 2022 Nanda et al. 2023). Prior work has demonstrated that probing oftentimes mischaracterizes the underlying computations performed by the model (Hewitt & Liang
2019 Zhang& & Bowman 2018 Voita & Titov 2020 ). We empirically demonstrate that circuit
We note that this is equivalent to a linear classifier whose weights are defined by the sampled vector for each class.
3

---

Published as a conference paper at COLM 2024
probing is more faithful to model's computation than linear nonlinear probing (See Sections 4.2 4.3 Contrastive probing is ablation of circuit probing where train linear probes using a similar contrastive objective. We present results in Appendix R and find that this method largely fails.
When performing causal analysis, we compare to three existing methods: causal abstraction analysis, amnesic probing, and counterfactual embeddings. Causal abstraction analysis intervenes on the activation vectors produced by Transformer layers to localize intermediate
variables to particular vector subspaces (Geiger et al. 2021 2023 Wu et al., 2023). However, it requires hypothesizing complete causal graph a high-leve description of how inputs are mapped to predictions, including all interactions between intermediate variables. This is impossible for most real-world tasks on which we want to apply neural networks (such as language modeling, to which we apply circuit probing in Section 4.4). We empirically demonstrate the utility of boundless distributed alignment search (boundless DAS; Wu iet al. 2023)), ) a state-of-the-art causal abstraction analysis technique, and show that circuif Amnesic probing seeks to erase linearly-dcoddabee information about an intermediate variable, and then observe the effect of this erasure on downstream behavior (Flazar et al. 2021). If this behavior is changed, then the intermediate variable is implied to exist within the original network. Though many techniques have been employed to erase linearly -eccdabbee information (Ravfogel et al.) 2020: 2022; Shao et al. 22023), we use the state-of-the-art LEACE method in our amnesic probing experiments (Belrose et al. 2023). We empirically demonstrate that circuit probing is more faithful to model's s computa tion than amnesic probing in Sections 4.2 and 4.3 Finally, counterfactual embeddings (Tucker et al., 2021 were introduced to enable causal analysis using particular probes. We find that countertactual embeddings are fairly uninformative in our experiments (See Sections
probing arrives at the same results (See Sections 4.1
4.2 4.3
4.1 and 4.2)
Experiments
We present four diverse experiments in order to illustrate the breadth of questions that circuit probing can help address. Experiments 1, 2, and 3 investigate toy models trained on simple arithmetic tasks, where full causal graphs are easy construct. These experiments both (1) nuance or reproduce results from prior work and (2) compare circuit probing against existing analysis methods. We compare circuit probing against linear and nonlinear probing when assessing circuit probing's 's ability to decode intermediate variables and compare against amnesic probing, causal abstraction analysis, and counterfactual embeddings when performing causal analysis. Experiment 4 applies circuit probing to language models to demonstrate that the method scales to a more realistic model and setting
4.1 Experiment 1: Deciphering Neural Network Algorithms
Goal: One of the central goals of interpretability research is to characterize the algorithms that models implement Olah (2022). This lofty goal is made substantially more tractable when we can adjudicate between two hypothesized alternatives. We demonstrate that all probing methods and most causal analysis methods produce converging results when characterizing the algorithm implemented by a model trained on simple arithmetic task.
Task: We train a 1-layer GP'2 model to solve task defined by the function (a2 b2) (mod P), where P is set to 113, and a and b are input variables. The input sequences
of the form (a, b, Pl, and the model is tasked with predicting the answer based on the output embedding of the final token. All inputs are symbolic- they are one-hot vector mappings to learnable embeddings. We exhaustively generate all possible data points with a and b, taking values 0- 112. Note that this input-output mapping permits at least two possible solutions, because a² b2- (a ++bb b) x (a- (a-ba b). We use circuit probing to determine which of the two alternative solutions the trained neural network adopts. Specifically, we search for the intermediate variables 22, (-1 * b²), (a + b), and (a b), all mod 113. We optimize binary masks using the output vectors from the attention and MLP blocks when they are operating on the token, which is where the final prediction is made.
In the main text we probe the residual stream of the model, see Appendix Q for results of probing individual updates to the residual stream. Both methods give approximately the same results. Seee Appendix B for all data and hyperparameter details.
4

---

Published as a conference paper at COLM 2024
Tosk Hrste ACCuTECy Attertion
Took 2 hrob Accunity Atterice
1.2
Task
Creu Probe
(mod
b (modi
dmod
(med
0.1
Uneu hote
hrbe
0.3
Veriadile
Sarrabiee
(b) Experiment 2 probing accuracy for circuit, linear, and nonlinear probes. We find that circuit probing and linear probing generally converge to find evidence of the Free and Shared variable for both tasks and evi- dence of the irrelevant variable ("other"). On the other hand, nonlinear probing suggests that the irrelevant variable is represented.
(a) Causal graphs representing the Multitask dataset in Experiment 2. Note that Task and Task share one intermediate variable (blue) and differ by one intermediate vari- able (red and green).
Probing: We expect all methods to achieve high accuracy on either a² and -1 b2 or a + b and a b b. This would adjudicate between the two alternative solutions to the arithmetic task. Indeed, we see that circuit probing, linear probing, and nonlinear probing on the attention block all converge to ceiling accuracy for the variables in a² - b2, and floor accuracy for the variables in (a + b) (a la-bi b) (See Appendix F). All methods converge toward more negative results from the MLP block, indicating that the intermediate variables are computed by the attention block (See Appendix
B
for MLP results).
Causal Analysis: We confirm that the circuits uncovered by circuit probing are causal in Appendix Amnesic probing weakly supports these conclusions (See Appendix G and
causal abstraction analysis strongly supports these conclusions (See Appendix H On the
other hand, we find that counterfactual embeddings fail to elicit counterfactual behavior in all cases, and thus do not provide evidence in either direction (See Appendix [ Finally, we run transfer learning experiment, which behaviorally demonstrates that the model is representing the task as a² b², rather than (a + b) (a b) (See Appendix J.. Overall, our results demonstrate that circuit probing agrees with existing causal techniques when characterizing the algorithm implemented by small model trained on a simple task.
4.2 Experiment 2: Modularity of Intermediate Variables
Goal: We now apply circuit probing to analyze the internal organization of Transformer models, which has been the subject of several recent studies (Lepori et al., 2023b Csordás et al. 2020; Hod et al., 2021 Mittal et al., 2022). We show that circuit probing can be used to characterize whether computations are implemented in a modular and reusable manner within Transformer, and that other methods fail to reveal such structure.
Task: We train model on simple multitask modular arithmetic task, represented by the causal graph in Figure 2a We set P = 29, p' = 31, p" = 23. The input sequences are of the form [T, b, c, N), and the model is tasked with predicting the solution. N is a separator token, and T is task token. For Task 1, we exhaustively generate all possible data points with a and b, taking values 0- 112, and c being random token in the same range. Similarly, for Task 2. Note that both tasks share one intermediate variable (a "shared variable"), and each task has one intermediate variable that the other does not (a "free variable"). First, we probe for intermediate variables. We use circuit probing to probe each task individually, assessing which intermediate variables are computed when solving Task 1, and which are computed when solving Task 2. Specifically, we search for the intermediate variables a(mod P), b(mod P'), and c(mod P"). We optimize binary masks using the output vectors from the attention and MLP blocks when they are operating on the N token, which is where the final prediction is made. Next, we investigate modularity directly. We hypothesize that the model implements reusable computation for the shared variable a(mod P) that the
((Non)linear probes are always trained to map from the residual stream of the final input token in the sequence (after the attention block and after the MI.P block) to the value of the intermediate variable that we are attempting decode.
5

---

Published as a conference paper at COLM 2024
same computation is used in both Task 1 and Task 2. Similarly, we hypothesize that the free variables are implemented modularly- - that Task I's free variable computation can be ablated without completely destroying performance on Task 2, and vice-versa.
Probing: We expect models compute their free and shared variables (a(mod P) and b(mod P') for Task 1, and a P) and c(mod P") for Task 2), and not to compute the other variable. Our probing results for the attention block are shown in Figure 2b Circuit, linear, and nonlinear probes decode free and shared variables with high accuracy, indicating that the relevant variables for given task are computed in the attention block. However, nonlinear probing (and only nonlinear probing) reliably decodes the other variable. This accords with prior work questioning whether expressive probes accurately reflect the causal structure of neural networks (Voita & Titov 2020 Zhang & Bowman 2018
Causal Analysis: Causal abstraction analysis agrees with circuit probing and linear probing (see Appendix K), causally implicating the free and shared variables for each task. On the other hand, counterfactual embeddings generated from the nonlinear probes once again fail to produce evidence that any variables are causal (see [pp We conclude that counterfactual embeddings act more as adversarial examples for our trained nonlinear probe, rather than as meaningful counterfactual inputs to the model. Next, we perform a causal analysis to understand whether the model exhibits modularity. We expect that ablating the circuit computing the shared variable for either task should destroy performance across both tasks. On the other hand, we expect that ablating the circuit computing the free variable for Task 1 should destroy performance on Task 1 while having less of an effect on Task 2 performance (and vice-versa). In Appendix[M we analyze the morphology of the two free variable circuits returned by circuit probing and find that the circuits are only distinct in one tensor within the attention block. Thus, we only ablate circuit weights within that tensor. From Table I. we see that ablating the circuit that computes the shared variable for either Task or Task 2 destroys performance on both tasks. On the other hand, ablating the free variable in Task 1 destroys performance on Task while maintaining some performance on Task 2. We observe the same trend for the free variable in Task 2. We run a similar analysis using amnesic probing and find that amnesic probing does not reveal this internal structure. We present results from amnesic probing on the residual stream after the attention block in Table 1 Overall, our results demonstrate that circuit probing is superior to competing techniques at characterizing how models structure their computations.
4.3 Experiment 3: Circuit Probing as a Progress Measure
Goal: Circuit probing allows us to gain insight into the training dynamics of Transformers at the level of intermediate variables. Recent work has shown that models may abruptly learn to generalize long after they overfit to the training data (Power et al.) 2022). Despite this rather discontinuous switch from overfitting to generalization (often called grokking), Nanda et al. (2022) revealed that the circuit that computes the generalizable algorithm is formed continuously throughout training. While their work required reverse-engineering the entire algorithm to gain this insight into circuit formation, we reproduced their finding in a slightly different setting using circuit probing (which only requires us to hypothesize high-level intermediate variable).
Task: We train a model on the task (a2 + (mod P), with P = 113. The input sequences are of the form (a, b, Pl, and the model is tasked with predicting the solution. Our model exhibits grokking on this task it generalizes long after it overfits. Generalization performance increases rather slowly from epoch 0 until 10000, then rapidly climbs to near-perfect accuracy by epoch 17500. See Figure 3h. First, we probe for the development of the intermediate variable a² throughout training. Next, we perform a selectivity analysis on the trained model - we probe for a variable that is not causally implicated in model behavior (b²) and verify that circuit probing does not decode this intermediate variable.
First, we investigate the development of the circuit computing a². We expect the performance of circuit probing to increase steadily throughout training, converging to a high value before the overall model generalizes. This finding would align with Nanda et al. (2022'ss 's finding that "circuit formation" " occurs continuously, and that it completes betore the overall model generalizes. We provide results from circuit, linear, and nonlinear probing on the attention block (See Appendix(N] for for MLP results). From Figure we see that circuit probing accuracy increases throughout training, achieving >90% accuracy before epoch
Probing:
6

---

Published as a conference paper at COLM 2024
Method
Train Task
Variable
Task Ablation Acc.
Task 2 Ablation Acc.
Circuit Probing
UucuiitProbingg
Task
Shared
3.7%
3.7%
Circuit Probing
Task
Free
3.2%
28.1%
Circuit Probing
Task 2
Shared
3.6%
3.6%
Circuit Probing
Task 2
Free
48%
4.8%
Amnesic Probing
Task
Shared
3.6%
3.3%
Amnesic Probing
Task
Free
2.2%
2.9%
Amnesic Probing
Task 2
Shared
2.7%
2.4%
Amnesic Probing
Task
Free
3.7%
4.5%
Table 1: Experiment 2 circuit probing ablation and amnesic probing results. Both causal interventions require training (a binary mask for circuit probing, and affine transfor- mation for amnesic probing). "Train Task" refers to the task that was used for training these interventions. Targeted ablations of the circuits returned by circuit probing reveal stark difference between the internal representations of the Free and Shared variables. A targeted ablation of the shared variable circuit destroys performance on both tasks, whereas targeted ablation of Task I's free variable harms performance on Task far more (under- lined) than Task 2 (bolded). The same true in the opposite direction. We do not see these differences when performing amnesic probing.
10000. Thus, we conclude that the circuit that allows the model to generalize is formed before the overall model's 's generalization behavior would imply. However, this circuit is not available from the outset and is developed throughout training. Linear and nonlinear probing tell a markedly different story, implying that the variable required to generalize was present nearly from the beginning of training. Next, we present results from our selectivity analysis. We expect our probing methods achieve poor accuracy when probing for b², variable that is not causally implicated in model behavior. At the end of training, circuit probing achieves 54.9% accuracy at decoding b² from the fully trained model, whereas linear and nonlinear probing achieves 100% accuracy. These values are remarkably steady throughout training (See Figure 3c).
Causal Analysis: Causal abstraction analysis confirms that the a² intermediate variable gets progressively more relevant as training progresses, and that b2 is never causally implicated in model behavior (See Appendix We find that ablating the circuits uncovered in the attention block by circuit probing for both a² and b² destroys model performance (See Appendix[P PP.. However, this is not a problem - the probing results make it clear that the circuit for b² is not successfully computing that intermediate variable in the first place, so the effect of ablating it is hard to predict. However, the linear classifier does decode both intermediate variables, and so amnesic probing is needed to clarify whether those variables causal. We find that amnesic probing for a² incorrectly implies that this variable is causal right from the start, dropping test accuracy to near 0 throughout training. Amnesic probing for b2 also incorrectly implies that this variable is causal throughout training, consistently dropping test accuracy (See Figure 3d). However, by the end of training, the effect of erasing information about b2 is notably diminished. These results characterize an expected but important failure case of amnesic probing: a² and b² are linearly decodable from the identity of a and b, so erasing all lineaalyddecodable information about either requires one to destroy the input. Overall these results demonstrate that circuit probing more faithful the underlying circuitry than existing methods.
4.4 Experiment 4: Circuit Probing in Language Models
Goal: The previous experiments focused on toy tasks in which a full causal graph could be specified. However, the reason that we are interested in developing interpretability tools is to analyze models that are used in practice on tasks where a causal graph cannot be constructed. Here, we use circuit probing to investigate how pretrained GPT2-Small and GPT2-Medium perform language modeling. In particular, we investigate two linguistic dependencies that rely on syntactic number: subject-verb agreement and reflexive anaphora.
Random chance for circuit probing 50%, as only two distinct integers in our dataset map to the same value of (mod 113) and we are using 1-nearest neighbor classifier.
7

---

Published as a conference paper at COLM 2024
(Z. + bi/mod 113) Training Accuracy
Probe Accuracy Duting Thaining: letn.
Probe Selectivity Anylyss
Amnestc Probing Analysts: Altn
1.0
1.0
1.0
1.0
Var
12
0.6
0.6
0.6
0.6
0.4
Mathed
Method
Gimitt Frobe
Crout: hobe
0.2
Unner Pione
Linear frode
0.0
5050
10000 Epocts
15090
20000
0.0
5000
10000 Epuchs
Vortinser Frobe
15000
20000
0.0
5090
15000 Epracs
Noni near Probe
15030
20020
0.0
5000
10000
15000
25020
Eprxhs
(a) Training Curve
(b) Probe Accuracy
(c) Selectivity Analysis
(d) Amnesic Probing
Figure 3: (a) We see generalization long after overfitting. (b) Probing for a², Linear and nonlinear probing converge to perfect accuracy very early in training, while circuit probing reveals that the circuit for n² is formed gradually through training. (c) Probing results for b", which is not causally implicated in the task a² + b. Circuit probing reveals that this variable is not represented at any point during training, whereas other methods imply that it is represented from the start of training.
(d) Amnesic Probing incorrectly implies that (1) a² is causally implicated from the start, and (2) b² is causally implicated throughout training.
Subject-verb agreement refers to the English-language phenomenon where the subject of a sentence must agree with the main verb of a sentence in syntactic number. For example: The keys are on the table grammatical, whereas The keys is on the table is ungrammatical. We hypothesize that an intermediate variable representing the syntactic number of the subject noun is computed when predicting the main verb of a sentence. Reflexive Anaphora ensures that reflexive pronouns agree with their referents. For example: the consultants injured themselves is grammatical, and the consultants injured herself is ungrammatical. We hypothesize that an intermediate variable representing the syntactic number of the referent computed when predicting a reflexive pronoun.
Task: We use the templates from Marvin & Linzen (2018) to generate sentence prefixes, where the continuation of the prefixes are likely to be either a main verb (when studying subject-verb agreement) or a reflexive pronoun (when studying reflexive anaphora). See Appendix [S for example prefixes. We run circuit probing on sentences containing one distractor, which is a non-subject or non-referent noun (e.g. "cabinet" in the keys to the cabinet on the table). For each phenomenon, we run circuit probing on the last token of sentence prefixes to uncover the circuit that computes the syntactic number of the subject noun or referent. We then ablate the discovered circuit and evaluate the model's ability to continue held-out sentence prefixes grammatically. Specifically, we assess whether the model is more likely to predict tokens that are consistent or inconsistent with the syntactic number of the subject/refen referent. For subject-verb agreement, we inspect the logits for the tokens is and - the logit for is is higher than the logit for are when the subject is singular (e.g. The officer..., then we consider the model to have succeeded on that sentence prefix. For reflexive anaphora, we run the analysis twice, once comparing the logits of herself and themselves, and again comparing the logits of himself and themselves. We evaluate models on IID sentences with one distractor noun and on an OOD dataset of sentences that contain two distractor nouns (See Appendix forr examples). We hypothesize that the same circuit computes the relevant linguistic dependency for both sentence structures. Even if we recover positive results from this analysis, it is possible that we are simply destroying the entire model, rather than ablating a specialized circuit. As a control, we sample 5 random subnetworks from the complement set of neurons that are in our circuit and rerun the ablation analysis. Kandomly-samples subnetworks always contain the same number of neurons as our circuit. Probing: See Appendix T.1 and U for an investigation of circuit probing accuracy. Generally, we find that most attention layers can compute the correct syntactic number, but that MLPs only begin to achieve good performance in the middle layers of GPT2-Small and Medium. See Appendix Y for linear probe accuracy for all GPT2 settings.
Causal Analysis: For both syntactic dependencies, we expect that ablating the discov- ered circuit will render the model worse at distinguishing the syntactic number of the subject/refereet refeerent We expect that ablating random subnetworks should not harm model performance on any dataset. We present results from GPT2-Small in the main body, and
8

---

Published as a conference paper at COLM 2024
Subject-Vrrb Agreement
Reflexive Anaphara Masc.
Reflexive Anaphora - Ferm.
1.0
1,5
0.8
Accurscy
0.8
Condition
Adlate Subrwtwork
0.7
0.7
0.7
Full Nodel
Adlate Random
0.6
0.6
I1
000
OOD
110 0GD
Dataset
Detaset
Dataset
Figure Experiment GPT2-Small ablatioo results on layer 6's attention block. Across both Subject-Verb Agreement (Left) and Reflexive Anaphora evaluated using the masculine (Mid- dle) and feminine (Right) pronoun, we see that ablating the discovered circuit renders the model worse at distinguishing syntactic number. Ablating randomly sampled subnetworks has does not hurt the model's ability to distinguish singular and plural subjects mubjects/referente referents.
present GPT2-Medium results in Appendix For both phenomena, we find that the de-
pendency is computed in layer 6's attention block Ablating the circuit returned by circuit probing drops performance substantially for IID and OOD datasets for both phenomena, while ablating random subnetworks does not impact model performance (See Figure 4).
Other blocks do not exhibit this characteristic pattern as strongly (See Appendix T.2).. This accords with prior work suggesting that syntactic dependencies are represented in middle layers of Transformers (Tenney et al. 2019 Vig & Belinkov) 2019). See Appendix V for an
analysis of circuit overlap, Appendix W for qualitative results, and Appendix X for results demonstrating that syntactic number of individual tokens is computed earlier the model.
Discussion
Related Work: Circuit probing related recent efforts mechanistic interpretability - burgeoning field that attempts to reverse-engineer neural network algorithms. Through substantial manual effort, researchers have uncovered the algorithms that both toy models
(Olsson et al. 2022 Nanda et al. 2022 Chughtai et al. 2023) and more realistic models (Wang)
et al. 2022 Hanna et al.) 2023 Merullo et al. 2023) are implementing. More broadly, there
has been substantial work analyzing the syntactic (Linzen & Baroni 2021 Goldberg 2019
Tenney et al. 2018 McCoy et al. 2018) and semantic capabilities Pavlick (2022) 2023); Yu &
Ettinger (020)) Hupkes et al. (2020); (zirii et al. (2023) of language models. Circuit probing
1s also re red to work that attempts to decomposs models into functional subnetworks
Csordás al. 2020 Hamblinn et al. (2022) Lepori et al. (2023b); Zhang et al. (2021); ) Panigrahi et al. (023); Hod et al. (2021); Cao et al. (2021). The success of circuit probing is further evidence that subnetworks are usetul lens through which analyze models.
We introduce circuit probing, a novel method for uncovering low-level circuits that compute high-level intermediate variables. Through four experiments, we have shown that one can gain insights into the underlying algorithms the model is implementing, how these algorithms are structured within the model, and how they develop throughout training. Circuit probing combines and extends the capabilities of existing methods and outperforms them in several settings. However, is currently unknown how multiple circuits compose within a given block to create additive update to the residual stream, so one cannot replace individual variables to perform counterfactual interventions. Future work might seek to understand how circuits compose with one another for this purpose.
Conclusion:
&Our results on GPI2-Medium are largely reproduce what we find in GPI2-Small for reflexive anaphora. We fail to identify circuit that computes subject-verb agreement dependencies. Forr reflexive anaphora, we note that models achieves higher accuracy when predicting the masculine pronoun. This is evidence of gender bias in language models, which has been well- documented elsewhere (Marvin & Linzen 2018 al. 2021)
May et al. 2019 Rudinger et al. 2018 Weidinger

---

Published as a conference paper at COLM 2024
6 Ethics Statement
Circuit probing can be used to uncover computations that a neural network is performing. This may have future implications for bias, fairness, and safety of neural network models. However, we emphasize that the current iteration of circuit probing should not be used in isolation to assess models for social biases in real-world systems. Circuit probing can provide positive evidence that a computation is implemented, but cannot yet be used to provide evidence that a computation definitely nol implemented in real-world system.
7 Acknowledgments
The authors thank the members of the Language Understanding and Representation Lab and Serre Lab for their valuable feedback this project and Rachel Goepner for proofreading the manuscript. This project was supported by ONR grant (N00014-24-1-2022), The computing hardware supported by NIH Office of the Director grant S100D025181 via the Center for Computation and Visualization (CCV) at Brown University.
References
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. Leace: Perfect linear concept erasure in closed form. arviv preprint arXiv:2306.038119 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Steven Cao, Victor Sanh, and Alexander M Rush. Low-complexity probing via finding subnetworks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 960-966, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084-15097, 2021.
Noam Chomsky. Aspects of the Theory of Synlax. Number 11. MIT press, 1965.
Bilal Chughtai, Lawrence Chan, and Neel Nanda. A toy model of universality: Reverse engineering how networks learn group operations. ArXiv, abs/2302.03025, 2023, URL
httpp::/app. semanticscholar wtt.:ooaar.rrr/ooossiiiiaa.. 1266600122211022665228881818 256615287
Aleksandra Čiprijanović, Gregory FF Snyder, Brian Nord, and Joshua EG Peek. Deepmerge: Classifying high-redshift merging galaxies with deep neural networks. Astronomy and Computing, 32: 32:100390, 2020.
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alone Towards automated circuit discovery for mechanistic interpretability. Advances in Neural Information Processing Systems, 1116318-163522 2023.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference Learning Representations, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXio preprint Xiv: 110.8480055555 2018.
10

---

Published as a conference paper at COLM 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ar Xiv preprint ar aXX::2010.11199, 2020.
Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jian, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D Hwang, et al. Faith and fate: Limits 2023.
of transformers on compositionality. arxio preprint ar arXi::2305.1866
Yanai Elazar, Shauli Ravfogel, Alon Alon Jacovi, and Yoav Goldberg, Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175, 2021.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. ar Xiv preprint ar Xxiv:2209110652, 2022.
Allyson Ettinger. What bert is not: Lessons from new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34 48,
2020.
Nicholas Frosst, Nicolas Papernot, and Geoffrey Hinton. Analyzing and improving rep- resentations with the soft nearest neighbor loss. In International conference machine learning, pp. 2012-2020. PMLR, 2019.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574-9586, 2021.
Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard, and Noah D Goodman. Finding alignments between interpretable causal variables and distributed neural repre- sentations. arXiv preprint ar Xiix:3303.02536 2023.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1901.052877 2019.
2019
Chris Hamblin, Talia Konkle, and George Alvarez. Pruning for interpretable, feature- preserving circuits in cnns. Xiv preprint arXiv:2206.01627 2022.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater- than?: Interpreting mathematical abilities in pre-trained language model, 2023.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp- 2733-2743, 2019.
John Hewitt and Christopher D Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and
Short Papers), pp. 4129-4138, 2019.
Shlomi Hod, Daniel Filan, Stephen Casper, Andrew Critch, and Stuart Russell. Quantifying local specialization in deep neural networks, arviv preprint ar Xfx::2110008058, 2021.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decom- posed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67: 757-795, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873)5883-589 2021.
11

---

Published as a conference paper at COLM 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xic::412.6688) 2014.
Michael A Lepori, Ellie Pavlick, and Thomas Serre. Neurosurgeon: A toolkit for subnetwork
analysis. arv preprint ar wri::2309.00222 2023a.
Michael A Lepori, Thomas Serre, and Ellie Pavlick. Break it down: evidence for structural compositionality in neural networks. arXiv preprint ar Xiw:23011110844 2023b.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. The Eleventh International Conference on Learning Representations, 2022.
Tal Linzen and Marco Baroni. Syntactic structure from deep learning. Annual Review of Linguistics, 7:195-212, 2021.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg, Assessing the ability of Istms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Pp- 1192-1202, 2018
Chandler May, Alex Wang, Shikha Bordia, Samuel Bowman, and Rachel Rudinger. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 622-628, 2019.
R Thomas McCoy, Robert Frank, and Tal Linzen. Revisiting the poverty of the stimulus: hierarchical generalization without a hierarchical bias in recurrent neural networks. In 40th Annual Meeting of the Cognitive Science Society: Changing Minds, CogSci 2018, pp- 2001- The Cognitive Science Society, 2018
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Language models implement simple word2vec-style vector arithmetic, 2023.
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. Is modular architecture enough? Advances in Neural Information Processing Systems, 35:28747-28760, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2022.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.0099 00941, 2023.
Christopher Olah. Mechanistic interpretability, variables, and the importance of inter-
pretable bases. Transformer Circuits Thread, 2022.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, J. Henighan, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, John Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom B. Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Christopher Olah, In-context learning and induction heads. ArXiv, abs/ htss/2209.11899 2022. URL https://api semanticscholar 2522322722
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. aiv preprint arXin:2302.066000 2023.
Ellie Pavlick. Semantic structure in deep learning. Annual Review of Linguistics, 8:447-471 2022.
12

---

Published as a conference paper at COLM 2024
Ellie Pavlick. Symbols and grounding in large language models. Philosophical Transactions of the Royal Society A, 381(255))20220041, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint (rXiv:2201.02117, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shauli Ravfogel, Yanai Flazar, Hila Gonen, Michael Twiton, and Yoav Goldberg- Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, 2020.
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. Linear adversarial concept erasure. In International Conference on Machine Learning, pp. 18400-18421, PMLR, 2022.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8: 842-866, 2021.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computationnll Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 8-14, 2018.
Ruslan Salakhutdinov and Geoff Hinton.
Learning a nonlinear embedding by preserving class neighbourhood structure. In Artificial intelligence and statistics, pp. 412-419, PMLR, 2007.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. Advances in neural information processing systems, 33:11380 2020.
Shun Shao, Yftah Ziser, and Shay B Cohen. Gold doesn't always glitter: Spectral removal of
linear and nonlinear guarded attribute information. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1603-1614, 2023.
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, et al. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference Learning Representations, 2018.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 4593-4601, 2019.
Mycal Tucker, Peng Qian, and Roger Levy. What this modified that? syntactic interventions with counterfactual embeddings. Association for Computational Linguistics, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 63-76, 2019.
E Voita and I Titov. Information-theenret Ai probing with minimum description length. In EMNLP 2020-2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 183-196, 2020.
13

---

Published as a conference paper at COLM 2024
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. ArXiv, abs/2211.00593, 2022. https://apppp semanticscholar Org/CorpusID: :53244437
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint xrXiv:2112.04359, 2021.
Zhengxuan Wu, Atticus Geiger, Christopher Potts, and Noah D Goodman. Interpretability at scale: Identifying causal mechanisms in alpaca. arv preprint ar aXit:2205.08809 2023.
Lang Yu and Allyson Etttnger. Assessing phrasal representation and composition in trans- formers. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 4896-4907, 2020.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville, Can subnetwork structure be the key to out-of-distribtion generalization? In International Conference on Machine Learning, pp. 18356-13367, PMLR, 2021.
Kelly Zhang and Samuel Bowman. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp- 359-361 361, 2018.
14

---

Published as a conference paper at COLM 2024
A Reproducibility Statement
To foster reproducibility, we provide details on model training and hyperparameters in Appendices and C throughout the maln text, as well as in our appendices. Additionally, we provide high- level explanation of the circuit probing algorithm in Section and provide details on the loss function in Appendix D Finally, we make our code publicly available.
We provide details of our experimental design for all experiments
B Data and Hyperparameter Details
All 1-layer GPT2 models have 4 attention heads, an embedding size of 128, and an MLP dimension of 512. We use the Adam optimizer [Kingma Ba, 2014) with a learning rate of 0.001 and train with weight decay.
Experiment We train a 1-layer GPT2 model on 60% of all possible datapoints for this task, leaving the other 40% held out test set. For circuit probing, we train our mask on all examples from the train set (See Appendix forr those training hyperparameters) using batch size of 500, and use updates generated from train set examples train the 1-nearest neighbors classifier. We evaluate on the held-out test set.
For linear and nonlinear probing, we train for 100 epochs, using a learning rate of 0.1. We use this same learning rate for generating counterfactual embeddings. Our nonlinear classifier uses ReLU nonlinearity, and has a hidden size of 256.
For Boundless DAS, we train for 250 epochs, with 2500 training examples using the Adam optimizer with a learning rate of 0.01.
For transfer experiments, we finetune 60% of each dataset, using the Adam optimizer with learning rate of 0.001. We train with weight decay. We train for fewer epochs with a² because the models converge very early in training.
Experiment 2 All details are the same in Experiment 1.
We train a 1-layer GPT2 model on 33.3% of all possible datapoints for this task, which gives us the grokking behavior that we wish investigate. All other details are the same Experiment 1.
Experiment 3
Experiment 4
We train circuit probing on 2000 examples for each dataset and test on 1000 examples, otherwise all details are the same.
C Continuous Sparsification Details
Continuous sparsification enables us train binary masks over model weights. Our loss function is defined as:
min Lsoft neighbors neightors(Cogsnn (Crrom)) + A/|m|| m4E(0.1)
(1)
Where m is our binary mask, C3 is a model component, with weights 6 and ||m|| is our lo
regularizer, and Leigt.iig is the soft nearest neighbors loss described in Appendix D
Typically, optimizing such binary mask is intractable, given the combinatorial nature of discrete binary mask over large parameter space. Instead, continuous sparsification reparameterizes the loss function by introducing another variable, E Rdd:
min neighttsss (Cooo(Bs)) + Al|a s;)111 S
(2)
In Equation] 2 or is the sigmoid function, applied elementwise, and B is temperature parameter. During training, B is increased after each epoch according to exponential
15

---

Published as a conference paper at COLM 2024
schedule to large value 9max. Note that, as 00, o(, - H(s;), where H(s;) is the heaviside function.
0,s < 1,8 > 0
H(s)
(3)
Thus, during training, interpolate between soft mask (0) and discrete mask (H). During inference, we simply substitute af (fmaxx : ) for H(s;). Notably, we apply continuous sparsification to frozen model in attempt to reveal the internal structure of this model. In contrast, the original work introduced continuous sparsification in the context of model pruning and jointly trained 0 and
For all experiments, we train binary masks with the Adam optimizer (Kinmma & Ba, 2014 with learning rate of 0.001. We fix fmax 200, initialize the mask parameters to 0, and train for 90 epochs. The À parameter must scale with the number of parameters per layer. For all 1-layer GPT2 experiments, we set 1E 6. For GPT2-Small and Medium, we A = 1E 7 and 1E 8, respectively.
D Soft Nearest Neighbors Loss
Given input embeddings to model component C and intermediate variable labels in a batch with samples, Equation 4 defines the full optimization objective. À is hyperparameter that scales the lo regularization strength. Intuitively, this loss function pushes members of the same class towards each other, according to some distance metric, and members of different classes far from each other. Concretely, this partitions the output space of transformer layers into equivalence classes defined by the variable that we are searching for.
EjeL..b, Cosinee fiahinnedistt (X) Cop (x))) H 31 y
min m,,0,1
Σ icl.bb
eoosine dist(Cesm (x) +1
Σ nET... mL
My
(4)
Ekel.,,, b, k/i
E Experiment 1: Probing Results
In Table 2 we see that all methods converge to show that the model is computing a² b2. rather than (a + b) x (a - b).
In Figure we present results from Experiment 1 for all probing methods on the MLP block. First, we note that all methods perform worse at decoding a² and -1 b2. We note that chance accuracy for circuit probing is effectively 50%, whereas chance for probing methods is 0.8% (1 out of 113). This because circuit probing results are generated by 1-nearest neighbors classifier trained on the outputs of the MLP block after masking, For the variable a² and -1 b2, there are only two distinct integers that map to the same value of that variable (i.e. (mod 113) (mod 113) = 4). Because
training the classifier with vector per variable label, 50% of the underlying integers are represented in the 1-NN training set. Thus, circuit probing accuracy of 50% means that the block is merely decoding the identity of the underlying token rather than meaningfully computing an intermediate variable.
Experiment 1: Circuit Probing Causal Analysis Results
Table 3 contains the results from running a causal analysis on the circuits discovered in
Experiment 1. We note two things: (1) ablating the circuits for a² or -1+ + b² destroys model performance, and (2) circuit probing returns empty subnetworks for + b and b. This is
16

---

Published as a conference paper at COLM 2024
Method
a²
- 1
b²
A a+
a
Circuit
99%
99%
1%
1%
Linear
100%
100%
0%
0%
Nonlinear
100%
100%
1%
1%
Table 2: Experiment 1 probing accuracy for circuit, linear, and nonlinear probes. All
methods converge to the conclusion that the model is representing a² b², rather than (a + b) (a- b).
Probe Test Accuracy: MLP
0.6
0.5
Test Accuracy
0.4
Method
Circuit
0.3
Linear Probe
Nanlinear Probe
0.2
0.1
.22
a-b
Variable
Figure 5: MLP probe accuracy for Experiment 1. All methods decode a² and -1 worse in the MLP than in the attention block. Note that chance accuracy for circuit probing is effectively 50%.
useful feature of using lo regularization when training binary masks - if there is no signal for a given variable, circuit probing is encouraged to return a maximally sparse (i.e. empty) subnetwork.
G Experiment 1: Amnesic Probing Results
In Table we provide results from running amnesic probing using LEACE (Belrose al., 2023) on the model in Experiment 1. LEACE requires fitting an affine transformation over an embedding to surgically erase all linearly- variable. We fit this transformation on the residual streams of the P token taken from the test set. We then erase the them back into the model, and compute its post-intervention test accuracy. We find that erasing the intermediate variable a² or b2 after the attention block completely destroys model performance. Erasing a + or a - also harms performance, though not quite as much. This weakly supports the conclusions from circuit probing, but raises important
information about an intermediate
information from these same embeddings, patch
Variable
Full Model Test Acc.
Ablated Test Acc.
% Parameters in Circuit
a2
100%
0.8%
53.3%
12
100%
0.9%
53.5%
a +
100%
100%
0%
100%
100%
0%
Table 3: Experiment task performance after ablating the circuit returned by circuit probing, We see that ablating the circuit responsible for either a² -12 destroys test accuracy. However, we see that circuit probing returns an empty circuit for both a + and a b, due to lo regularization. Thus, ablating this empty circuit has effect.
17

---

Causal Published as a conference paper at COLM 2024
Component
a² -b2
a a++
a
Attn.
1.8%
1.8%
35.3%
35.2%
MLP
100%
99.9%
99.3%
99.4%
Table 4: Amnesic Probing results for Experiment 1. We note a greater performance drop when erasing a² bb than + b or a the residual stream after the attention block.
Component a²
-b2
+
b
Attn.
98%
99%
1%
1%
MLP
2%
2%
2%
2%
Component
Task
a (mod P)
(mod P')
c (mod P")
Attn.
1
93%
93%
MLP
1
3%
3%
Attn.
2
93%
94%
MLP
2
4%
4%
Table Causal abstraction analysis results for Experiment 1 (top) and Experiment (bottom). Using boundless distributed alignment search, we reveal that the attention blocks in both models contain the same causal intermediate variables that circuit probing discovers.
questions about the utility of LEACE when linear probes perform poorly (as is the case for a + or a b, see Section 4.1
H Experiment 1: Causal Abstraction Analysis Results
In Table we provide results from running Causal abstraction analysis using Boundless DAS (Wu et al. 2023) on both Experiment and Experiment 2. Causal abstraction analysis creates interventions on model representations in order to elicit counterfactual behavior in the downstream model. For example, in the case of a² b², the model might intervene to change the value of x² to a²2 The intervention is considered successful if the overall model outputs the answer to z² b2. Causal abstraction analysis reports statistics in terms of the success of its counterfactual embeddings, rather than its ability to decode model representations.
These results support the results generated by circuit probing in Experiment abstraction analysis reveals evidence for a² and -1 b?, but not a + and a - b.
I Experiment 1: Counterfactual Embeddings
We present counterfactual embedding results from Experiment in Table Counterfactual embeddings are embeddings that are optimized to fool a probing classifier. Formally, given probe, Pg trained to decode an intermediate variable, V, consider residual stream state such that Pe(e) V,. We freeze Py and optimize e such that Pg(e') V,- If Pe is decoding information that is causally implicated in the underlying model, then replacing e with should change the output to the output one would expect from setting variable V to Vy. We report counterfactual embedding success the percentage of embeddings that are successfully optimized fool the probing classifier. We see that all linear probing classifiers can be fooled by counterfactual embeddings. We see that nonlinear classifiers can be fooled by counterfactual embeddings only for d² and -1 b2. Recall that all classifiers performed poorly at decoding a + b and a
Next, analyze counterfactual behavior success the percent of counterfactual embed- dings that actually elicit counterfactual behavior in the overall model. We see that all sets of counterfactual embeddings fail to elicit counterfactual behavior. Taken in isolation, one
18

---

Published as a conference paper at COLM 2024
Counterfactual Embedding Success
Probe
a²
b²
+ b
Linear
100%
100%
100% 1%
100%
Nonlinear
100%
100%
1%
Counterfactual Behavior Success
Probe a²
b2
+ b
Linear
1% 1%
1%
1%
Nonlinear
1%
1%
0%
0%
Table 6: Experiment counterfactual embedding results. (Top) Counterfactual embedding success the percent of examples where the counterfactual optimization procedure creates an example that changes probe outputs to particular class. We see that this optimiza- tion process largely succeeds, except for a- b and Counterfactual behavior success the percent of counterfactual embeddings that elicit coun- terfactual behavior in the model. We see very poor performance on this metric, indicating that counterfactual embeddings are not producing the expected behavioral outcomes.
in nonlinear probes. (Bottom)
might conclude that these probes are not decoding causally-relevant information, and thus that models are not actually computing a² and -1 b². However, given the success of every other analysis technique at causally implicating a² and -1 b2, we may instead conclude that counterfactual embeddings are acting as adversarial examples the probing classifier, and are destroying the embedding with respect to the underlying model.
J Experiment 1: Transfer Learning
To further confirm our findings in Experiment 1, we analyze whether training on a² 62 confers any benefits when finetuning on different tasks. In particular, we finetune the GPT2 model on a task defined by a² (mod 113), and separately on task defined by + b(mod 113). the model solving the task using a² b2, we expect that finetuning should help the model solve a² faster than training a randomly initialized model, because the model already represents the variable necessary to solve the finetuning task. Similarly, we expect the finetuning to a + 5 will be slower than training a randomly initialized model, because the model represents variables that are explicitly not useful for solving the finetuning task. From Figure 6 that is exactly what we see.
K Experiment 2: Causal Abstraction Analysis Results
In Experiment 2, causal abstraction analysis reveals evidence that the model is using vari- ables a(mod P) and b(mod P') when solving Task 1, and a(mod P) and c(mod P") when solving Task (See Table Unfortunately, there is no causal model that involves c(mod P") when solving Task 1, and b(mod P"") when solving Task 2, and so it does not make sense to run causal abstraction analysis in this setting.
Experiment Counterfactual Embedding Analysis
Here, a causal analysis of the nonlinear probes using counterfactual embeddings. Counterfactual embeddings are designed to reveal whether probes are reflecting information that is causally implicated in model behavior. We summarize the relevant details of their technique here but defer to Tucker et al. (2021) for a full treatment. Given probe, Po trained to decode an intermediate variable, V, consider residual stream state such that Pş(e) = V,. We freeze Pg and optimize such that Pa(e") Vy. If Pe is decoding information that is causally implicated in the underlying model, then replacing with e' should change the
19

---

Published as a conference paper at COLM 2024
a2 Transfer vs. Reinitialized Accuracy
a + b Transfer vs. Reinitialized Accuracy
0.8
0.8
Dataset
Accuracy
0.6
Accuracy
0.6
Transfer Train Acc
Transfer Test Acc
0.4
Dataset
0.4
Reinit Train Acc
Reinit Test Acc
Transfer Train Acc
0.2
Transfer Test Acc
0.2
Reinit Train Acc
Reinit Test Acc
200
300
400
Epochs
Epochs
Figure 6: (Left) Transfer performance for a². We see that pretraining on a² b² confers benefit the model when finetuning on a². (Right) Transfer performance for + We see that pretraining on a² b² is a detriment when finetuning on +
Task
Probe Var.
CE Success
Model Acc.
Counterfactual Acc.
1
Free
100%
2.1%
2.2%
1
Other
100%
1.8%
1.5%
1
Shared
100%
2.5%
2.4%
2
Free
100%
2.1%
2.1%
2
Other
100%
2.3%
2.1%
2
Shared 100%
2.2%
3.0%
Table 7: Experiment counterfactual embedding results. Counterfactual embeddings that created by "free" and "shared" variable probes should result in different prediction being made in the overall model. Counterfactual embeddings created by the "other" variable probe should have no effect on the overall model's prediction. We see that this does not happen. Though all counterfactual embeddings succeed at changing the probe prediction (CE Success), they also all change the overall model prediction (Model Acc.), but not to the correct counterfactual answer (Counterfactual Acc.).
output according to the counterfactual variable. If this information is not causally implicated in the underlying model, then replacing e with é' should have effect, preserving the original model output.
We generate counterfactual embeddings for both tasks using the nonlinear probes trained to decode the "free" and "other" " variables from the residual stream after the attention block. We record the percentage of examples whose output is maintained before and after substituting the counterfactual embedding (the "Model Accuracy").. We expect this value to be high for the "other" variable, and low for the "free" and "shared" variables, indicating that the features that the nonlinear probe uses to decode the "other" variable are not causal. Conversely, we record the percentage of examples whose output changed according to the counterfactual variable (the "Counterfactual Accuracy"), We expect this value to be high for the "free" and "shared" variables and low for the "other" variable. This would indicate that only the "free" and "shared" variables are causal.
From Table 7 we see that both Model Accuracy and Counterfactual Accuracy drop to near-zero after patching in counterfactual embeddings for either "free", "shared", or "other" variables. This suggests that counterfactual embeddings act more as adversarial examples to the probe, rather than providing useful information about causally-relevant variables.
20

---

Published as a conference paper at COLM 2024
Free Variable Circuit Overlap
attn.c_atee attn
Layer: 0
attn.c_proj
Task 1
Task
Intersection
Unused
Figure 7: Experiment 2: Visualizing the distribution of circuits throughout the tensors comprising an attention block. We see that the circuits computing the free variables for Task and Task 2 almost completely overlap in the proj tensor, but are mostly distinct in the c attn tensor.
M Experiment 2: Circuit Overlap Analysis
From Figure 7 we see that the two circuits computing the free variables in Task 1 and Task are largely distinct in attn. attn.c_atee c_attn, but nearly completely overlapping in attn. c_proj. With this insight, we ablate circuit parameters just within attn. c c_attn. We can also visu- alize the distribution of circuits through attention heads. In Figure 8 we that circuit probing recovers structures that exist between particular attention heads (i.e. no single attention head is fully devoted to intermediate variable), but also partially localizes the two Free variables into specific heads (head 1 for Task 1, head for Task 2).
N Experiment 3: MLP Probe Accuracy
Here we present the linear, nonlinear, and circuit probe accuracy on the MLP block through- out training for Experiment 3. In Figure we see very messy results, further reinforcing that the intermediate variable a² is computed in the attention block. We also present amnesic probe accuracy after erasing each intermediate variable from the residual stream after the MLP block in Experiment 3. In Figure 10) we that test accuracy monotonically increases throughout training, generally reflecting the test accuracy of the underlying model. This indicates that amnesic probing does not erase any important information
Experiment 3: Causal Abstraction Analysis
We present results from running causal abstraction analysis throughout training for the variables a² and b². a² should be causally implicated in model behavior by the end of training, the model generalizes to unseen data according to the function a² + b. b2 should never be causal, as not relevant to the task. We see the expected patterns in Figure
P Experiment 3: Circuit Probing Causal Analysis
Figure 12 provides our results from ablating the circuits uncovered by circuit probing throughout training. Probing results already indicate that the circuit uncovered for b2 fails
21

---

Published as a conference paper at COLM 2024
Free Variable Circuit Overlap: Blocks
mlp
head
Layer: O
head
head
head
Task 1
Task 2
Intersection
Unused
Figure 8: Circuit probing recovers some elements of known structure within Transformers. In particular, we see that Head 1 largely computes the Free variable in Task 1, and Head largely computes the Free variable in Task 2. However, also note that circuits extend beyond individual attention heads.
Probe Accuracy During Training: MLP
1.0
Method
Circuit Probe
0.8
Linear Probe
Accuracy
0.6
Nonlinear Probe
0.4
0.2
0.0
5000
15000 20000 Epochs
10000
Figure 9: Experiment MLP Probing results. We see chaotic results from all probes, indicating that the intermediate variable a² not computed the MLP block.
22

---

Published as a conference paper at COLM 2024
Amnesic Probing Analysis: MLP
1.0
Var.
0.8
a2
b2
Accuracy
0.6
0.4
0.2
0.0
5000
10000 Epochs
15000
20000
Figure 10: Experiment MLP Amnesic Probing results. We see that amnesic probing has little impact on test accuracy throughout training.
Causal Abstraction Analysis: Attn.
Causal Abstraction Analysis: MLP
1.0
Var.
Var.
0.8
a2
0.8
a2
Accuracy
b2
Accuracy
b2
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0
5000
10000 15000 Epochs
20000
0
5000
15000 20000 Epochs
10000
Figure 11: Experiment 3: Causal abstraction analysis results for both a² and b2, a² becomes causal in the attention block throughout training, indicating a switch from memorization to generalization. b² never becomes causal, as expected.
23

---

Published as a conference paper at COLM 2024
Circuit Probing Ablation: Attn.
Circuit Probing Ablation: MLP.
1.0
1.0
Var.
Var.
0.8
a2
0.8
a2
b2
b2
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
5000
10000 15000 Epochs
20000
5000
10000
15000
20000
Epochs
Figure 12: Experiment 3: Ablating the circuits discovered by circuit probing in the attention layer destroys performance for both a² (which should be causal towards the end of training) and 62 (which should never be causal). In the MLP block, ablating circuits becomes less detrimental over time for both variables.
Probe Test Accuracy: Attention Update
Probe Test Accuracy: MLP Update
0.8
0.8
Test Accuracy
Test Accuracy
Method Linear Probe
0.6
Method
Linear Probe
Nonlinear Probe
0.4
Nonlinear Probe
0.2
0.2
a+b Variable
Variable
Figure 13: Experiment 1: Results from linear and nonlinear probing on updates from Attention (left) or MLP (right) layers.
to actually compute this variable, the effect of ablating this set of parameters is hard to predict. Here, we see that destroys performance.
Q Experiments 1-3: Probing on Update Vectors
Circuit probing operates on updates to the residual stream (i.e. vectors generated by MLPs or Attention Layers), rather than the residual stream itself. However, in the main text, we follow common convention and present results from linear and nonlinear probing on the residual stream. In this section we present linear and nonlinear probing results from Experiments 1-3, where we probe the update vectors from MLPs and Attention Layers. In general, we find that probing on either the residual stream or individual vector updates gives extremely similar results. See Figures 13 15
R Experiments 1-3: Contrastive Linear Probe
In this section we present an ablation of the circuit probing algorithm, where we optimize a linear probe using a contrastive objective. Formally, we define a linear probe, P, which maps from intermediate representations (either residual stream states, Attention updates, or MLP updates) to an embedding dimension We optimize P using the soft-neighbors loss defined in D Intuitively, this probe should produce similar embeddings for representations
24

---

Published as a conference paper at COLM 2024
Task Probe Accuracy: Attention Update
Task 1 Probe Accuracy: MLP Update
1.0
0.8
0.8
Test Accuracy
Test Accuracy
Method Linear Probe
0.6
Method
Linear Probe
0.4
Nonlinear Probe
0.4
Nonlinear Probe
0.2
0.0
0.0
Free
Shared
Other
Free
Shared Variable
Other
Task Probe Accuracy: Attention Update
Task 2 Probe Accuracy: MLP Update
1.0
0.8
0.8
Test Accuracy
Test Accuracy
0.6
Method Linear Probe
0.6
Method
Linear Probe
0.4
Nonlinear Probe
0.4
Nonlinear Probe
0.0
0.0
Free
Shared Variable
Other
Free
Shared
Other
Variable
Figure 14: Experiment Results from linear and nonlinear probing on updates from Attention (left) or MLP (right) layers on Task (top) Task 2 (bottom).
that belong the same class, and different embeddings for representations that belong
to different classes. We evaluate this probe using the same procedure outline in Section 2 For ease of comparison, we fix d 113, which equates the number of parameters in the contrastive probe and the linear probes used in Experiments 1-3. In general, we find that this probe produces fairly poor results (Figure Thus, we cannot attribute the success of circuit probing merely to different optimization objective.
S Experiment 4: Language Prefix Examples
Here, present several examples of sentence prefixes for subject-verb agreement and reflexive anaphora.
Subject-Verb Agreement:
IID: 1-Distracctor
1. the farmers that the taxi driver admires (are)
2. the authors behind the assistants (are)
3. the consultant that the skaters like (is)
OOD: 2- D-sstraccoos
1. the books by the architects next to the executives (are)
2. the customer that the skaters like and the ministers hate (is)
3. the pilots in front of the dancers to the side of the parents (is)
Reflexive Anaphora:
IID: 1-Distractor
25

---

Published as a conference paper at COLM 2024
Probe Accuracy During Training: Attn. Update
Probe Accuracy During Training: MLP Update
1.0
1.0
0.8
0.8
Accuracy
Accuracy
0.6
0.6
Method Circuit Probe
Method
0.4
0.4
Circuit Probe Linear Probe Nonlinear Probe
0.2
Linear Probe
0.2
Nonlinear Probe
0.0
0.0
0
5000
10000
15000
20000
0
5000
10000
15000
20000
Epochs
Epochs
Probe Selectivity Analysis: Updates
1.0
0.8
Accuracy
0.6
Method
0.4
Circuit Probe
0.2
Linear Probe
Nonlinear Probe
0.0
0
5000
15000 20000 Epochs
10000
Figure 15: Experiment 3: Results from linear and nonlinear probing on updates from Attention (left) MLP (right) layers. (Bottom) Selectivity analysis when probing attention layer updates for irrelevant variable.
26

---

Published as a conference paper at COLM 2024
Contrastive Probe: Disambiguation
Task
Probe Accuracy: Contrastive
0.7
0.6
0.6
0.5
Test Accuracy
0.5
Test Accuracy
Component
0.4
Component
0.4
attn mip attn update
attn
0.3
mip
update
0.2
mip update
mip update
0.1
0.0
a2
-b2
a+b
Free
Shared
Other
Variable
Variable
Contrastive Probe Accuracy
Task 2 Probe Accuracy: Contrastive
1.0
Location
0.5
resid_mite attn_out resid _post
0.8
Test Accuracy
Accuracy
Component
0.6
0.3
attn mlp attn update
mip out
0.4
0.2
mip update
0.2
0.1
0.0
0.0
Free
Shared Variable
Other
5000
10000 Epochs
15000 20000
Contrastive Probe Selectivity
Location resid mid attn out resid _osst mlp_out
Accuracy
0.6
0.4
0.2
0.0
0
5000
10000 15000 Epochs
20000
Figure 16: Results from Contrastive Linear Probing in Experiments 1-3. In general, we see that the probe achieves low accuracy across the board. Though it does differentiate between variables that are relevant vs. irrelevant in Experiment 1 (top left) and Experiment 2 (top right, middle left), it fails to do for Experiment 3 (middle right, bottom).
27

---

Published as a conference paper at COLM 2024
GPTZ-smal SVAgresment RVN Accursey
nip
aPrz lalieere KIN Accuresy
400
hdl
Hipp
8003
tip
Component
Figure 17: GPT2-Small circuit probing KNN results for subject-verb agreement (Top) and reflexive anaphora (Bottom). We notice that KNN accuracy increases for MLP blocks after layer which is where causal analysis located the causal circuits for both phenomena.
1. the consultants that the parents loved doubted (themselves)
2. the senators that the taxi drivers hate congratulated (themselves)
3. the mechanics thought the pilot hurt (herself/ hhinsslf)..m
OOD: 2-Distractors
1. the surgeon that the chefs love and the parents admire hated (herself/himself
2. the mechanic knew the banker said the farmer embarrassed (herself/himself)
3. the teachers that the architect hates and the minister admires hurt (themselves)
T Experiment 4: GPT2-Small Extended Results
T.1 Experiment 4: GPT2-Small Language Probing Results
We present circuit probing KNN evaluations for both subject-verb agreement and reflexive anaphora over all model components. Causal analyses indicate that circuit that is causally implicated in computing syntactic number is located in layer 6's attention block. We note that KNN accuracy increases for every MLP block after layer 6. Because MLP blocks are applied token-wise, this suggests that the information required to decode syntactic number of both subjects and referents is present in the residual stream after this layer but not before. See Figure 17
28

---

Published as a conference paper at COLM 2024
Cruat
191 Madermaa Soddas
Coad
Comporas:
Figure 18: GPT2-Small subject-verb agreement ablation results for every model component. We note that the attention block in layer 6 provides the greatest drop in performance after ablating the discovered circuit.
Figure 19: GPT2-Small reflexive anaphora ablation results for every model component, evaluated using the masculine pronoun. We note that the attention block in layer provides the greatest drop in performance after ablating the discovered circuit.
T.2 Experiment 4: GPT2-Small Full Ablation Results
We present circuit probing ablation results for both subject-verb agreement (See Figure 18 and reflexive anaphora (See Figures 19 and 20) over all model components. In all cases, we note that ablating the circuit in attention block in layer 6 provides the greatest drop in model performance. Randomly ablating subnetworks of the same size does not harm model performance.
U Experiment 4: GPT2-Medium Results
U.1 Reflexive Anaphora
We present results analyzing GPT2-Medium's ability compute the syntactic number of the referent of reflexive pronoun. We find that the attention block in layer 7 is most causally implicated this computation. See Figure 21 Ablating the discovered circuits harms model performance, regardless of the pronoun used for evaluation. Ablating random subnetworks of the same size does not harm model performance.
Turning to the reflexive anaphora probing evaluation, circuits trained on MLP blocks increases during and after layer Because MLP blocks operate token-wise, this indicates that the information required to decode the syntactic
that the KNN accuracy of
29

---

Published as a conference paper at COLM 2024
Creit
Figure 20: GPT2-Small reflexive anaphora ablation results for every model component, evaluated using the feminine pronoun. We note that the attention block in layer 6 provides the greatest drop in performance after ablating the discovered circuit.
Reflexive Anaphora Masc.
Reflexive Anaphora Fem.
1.0
1.0
Accuracy
Condition
Accuracy
Condition
Abloto Subnetwork
Abloto Subnetwork
Ablate Random
Abliat Random
Model
Full Model
0.6
0.6
0.5
0.5
IID
DOD
IID
DOD
Dataset
Dataset
Figure 21: GPT2-Medium reflexive anaphora ablation results for the attention block in layer 7.
30

---

Published as a conference paper at COLM 2024
Grrz-medium Retlacie KNN Accurncy
Compoten
Figure 22: GPT2-Medium reflexive anaphora circuit probing KNN evaluation results across all model components.
landers
BELL
Crout
PrerddanPPa Netsovas foston
MASEYS
Figure 23: GPT2-Medium reflexive anaphora ablation results across all model components, evaluated using the masculine pronoun. Note that the largest drop in performance due to ablation occurs at the attention block in layer 7.
number of referents is present in the residual stream after this layer, but not before. This strengthens causal results analysis. See Figure 22
For completeness, we include ablation results across model components in Figures and] 24
31

---

Published as a conference paper at COLM 2024
DR1 reniic Minanss Berater
AO
17
Figure 24: GPT2-Medium reflexive anaphora ablation results across all model components, evaluated using the feminine pronoun. Note that the largest drop in performance due to ablation occurs at the attention block in layer 7.
32

---

Published as a conference paper at COLM 2024
Croat
SEL DAGLE OI
Coadt
1.0
lander
HELL
HE
Figure 25: GPT2-Medium subject-verb agreement ablation results across all model compo- nents. These results do not implicate any specific circuit in computing the syntactic number of the subject noun.
U.2 Subject-Verb Agreement
We do not find any particular circuits that drop subject-verb agreement performance sub- stantially when ablated (See Figure 25). This might indicate that multiple circuits across several blocks are redundantly computing the syntactic number of the subject noun.
Turning to the subject-verb agreement probing evaluation, we see that the KNN accuracy of circuits trained on MLP blocks increases after layer 7 (see Figure 26). Because MLP blocks operate token-wise, this might indicate that the information required to decode the syntactic number of referents is present in the residual stream after this layer, but not before. However, our causal analysis does not provide strong evidence of this. From 25 we see a small drop layer 7's attention block and another layer 18's attention block.
V Experiment 4: Circuit Overlap
Surprisingly, we see that there is very little overlap between the circuits used to compute the syntactic numbers of subjects and referents in GPTl-Small, despite both circuits being present in the same block. See Figure
W GPT2-Small Subject-Verl Verb Agreement Qualitative Results
We present qualitative results of ablating the subject-verb agreement circuit discovered by running circuit probing on the attention block in layer of GPT2-Small (See Table S). We note that the types of tokens predicted by the model qualitatively stay the same before and after ablation. This suggests that we have not destroyed the model by ablating the discovered circuit. Interestingly,
more tokens that explicitly consistent with the
33

---

Published as a conference paper at COLM 2024
mrd.-medum SvAgremment Well Accurscy
Compotent
Figure 26: GPI2-Medium subject-verb agreement circuit probing KNN evaluation results. We see KNN performance increase for MIP blocks around layer? 7.
syntactic number of the subject before ablation, and fewer after ablation. This provides qualitative evidence that we have indeed ablated a circuit that was responsible for tracking the subject-verb agreement dependency.
X Investigating Syntactic Number Feature Extraction
demonstrate that linguistic dependencies (e-g. the influence of a previous word's syntactic number on main verb prediction) are computed in the middle layers of both GPT2-Smal and Medium. However, this leaves open the question: where in the model are the syntactic number features extracted in the first place? In this section, we uncover circuit that computes syntactic number features and demonstrate that ablating this circuit completely destroys model's ability to perform subject-verb agreement.
Goal The experiments presented in Section 4.4
Task We use subset of the 1-distractor subject-verb agreement dataset from Section 4.4 for this experiment. Specifically, we use the subset of data that includes prepositional phrases (e-g. the farmer near the parents). In these sentences, the final word before the main verb is always a non-subject noun.
We run circuit probing on the last token of the non-subject noun in the sentence prefixes to uncover the circuit that computes the syntactic number of the non-subject noun (rather than the subject noun). Thus, we effectively uncovering a syntactic number feature extractor circuit. To ensure that our discovered circuit truly is general syntactic number feature extractor, we ablate the discovered circuit and test the model's ability to generate syntactically valid next-token predictions using the same evaluation as described in Sec- tion 4.4 The experimental logic that if the discovered circuit responsible for syntactic number feature extraction in general, then ablating it will also destroy model's ss ability to extract the syntactic number the subject noun, rendering the subject-verb agreement task impossible.
Probing We include circuit probing KNN evaluation results for GPT2-Small and Medium We find that accuracy reaches ceiling after MLP 0 and stays ceiling
Figures 28 and 29 until the late stages of the model.
Causal Analysis We expect that ablating the discovered circuit will render the model worse at extracting the syntactic number of all nouns, destroying performance our subject-verb agreement evaluation. We expect that ablating random subnetworks should not harm performance on this task. We present results from both models (See Figure30)..o For both models, we find that the syntactic number computed in early MLPs (layer 0 for GPT2- Small, layer 1 for GPT2-Medium). Ablating the circuit returned by circuit probing drops
34

---

Published as a conference paper at COLM 2024
GPT2-Small Agreement Circuit Overlap
attn.c_attr
Layer:
attn.c_proj
S-V Agreement
Reflexives
Intersection
Unused
GPT2-Small Agreement Circuit Overlap Blocks
mlp
head
head
head
head
Layer: 6
head 4
head
head
head
head
head
head
head
S-V Agreement
Reflexives
Intersection
Unused
Figure 27: Circuit overlap between syntactic number and reflexive anaphora in GPT2-Small, attention block 6. We see that the discovered circuits are largely distinct. We also note that certain attention heads (0, 3, and 7) appear to be most important in computing syntactic number for both subject nouns and referents.
35

---

Published as a conference paper at COLM 2024
Prefix
Original Output
Ablate Atin-6 Output
The surgeons behind the dancer
were and are
and to
)
said in
had
have who
to )
who
in said
was
)
)
were
The book from the executives
of of
at at
who
who and
'
and in in
to
that that ) on
was
to themselves
I were
Table 8: Qualitative examples of the effect of ablating the circuit discovered in GPT2-Small, layer 6. We record the top 15 next-token predictions. Words that are explicitly consistent with the syntactic number of the subject are bolded, words that are inconsistent are underlined.
36

---

Published as a conference paper at COLM 2024
GFizamall Syntactic Number taa Accuracy
13
brp
steil
Figure 28: GPT2-Small syntactic number KNN results. We KNN performance reach ceiling after MLP 0, and stay at ceiling until late in the model.
GPTZ med um Syeractic Humber ON Accurnry
Figure 29: GPT2-Medium syntactic number KNN results. We see KNN performance reach ceiling after MLP 0, and stay at ceiling until late in the model.
subject-verb agreement performance nearly to chance while ablating random subnetworks does not impact model performance. We present results from layers for GPI2-Small in Figure and and for GPT2-Medium in Figure 32 We note that it seems that GPT2-Medium also performs syntactic number extraction in layer 0's MLP.
Y GPT2 Linear Probing Results
In this section, we present results generated by linearly probing GPT2-small and medium in all tested conditions (subject-verb agreement, reflexive anaphora, and syntactic number). Across the board, we find that linear probing decodes the syntactic number early, and maintains ceiling performance throughout the network. See Figures 34 Notably, circuit probing can identify when layers are causally implicated in downstream model performance via subnetwork ablation, whereas linear probing in this manner does not directly provide this information.
37

---

Published as a conference paper at COLM 2024
1.0
GPT2-smail Syntactic Number
1.0
medium Syntactic Number
0.9
Accuracy
0.8
Condition
Condition
Subnetwork
Ablate Subnetwork
0.7
Ablate Random
Ablale Random
Full Model
0.6
0.6
0.5
0.5
IID
Dataset
Dataset
Figure 30: GPT2-Small (left) and Medium (right) ablation results on early MLPs. We find that the circuit that extracts syntactic number features from tokens computed in early MLPs. Ablating this circuit destroys subject-verb agreement performance while ablating random circuits of the same size does not substantially impact performance.
OrTburnl Syeratic Jolatie
Cruit
Syeati
Conponen:
Figure 31: GPT2-Small syntactic number ablation results across all model components. Note that the largest drop in performance due to ablation occurs at the MLP block in layer 0.
38

---

Published as a conference paper at COLM 2024
Rendira
Crout
ABA
OITI mocken Searrt Narker
Figure 32: GPT2-Medium syntactic number ablation results across all model components. Note that the largest drop in performance due ablation occurs at the MLP blocks in layer 1 and layer O. However, ablating random subnetworks has less of an impact on model performance layer 1's MLP.
39

---

Published as a conference paper at COLM 2024
GPT2-mall Linerr Frebe
mold
poico
prote
phut
neint midg
roai Loode
raid d
nomt nul7
nolt mode
rall a
nold neta
reig min
new pnd
reii and
GPT2 Comparen:
Figure 33: GPT2-small and medium linear probing in Subject- Verb Agreement Task.
40

---

Published as a conference paper at COLM 2024
GPT2-mall Linear Frebe
modd
poico
prote
phut
neint midg
raid d
nomt nul7
nolt oodd
rall auss
nold neta
reig min
new pnd
rele and
GPT2 Comparen:
Figure 34: GPll-small and medium linear probing in Reflexive Anaphora Task.
41

---

Published as a conference paper at COLM 2024
GPT2-mall Linear Frebe
mold
poico
protc
neint midg
raid d
nomt nul7
nolt mndd
rald auss
neld neta
reig min
new pnd
reii and
Comparen:
GPT2 Uioar
Figure 35: GPT2-small and medium linear probing in Syntactic Number Task.
42


Reviews:
[{"review_id": "EDxzdYoMHo", "reviewer": "Reviewer_UN6h", "summary": "The paper proposes a methodology for assessing the presence of intermediate variables in transformers. Assume you have a dataset of examples, which are labelled with the intermediate variable for a particular phenomenon. Prior work would use this data to train linear/non-linear probing classifiers on the hidden states of the transformer. This paper suggests that we should prune the model with these labels to discover a sub-circuit and that the sub-circuit allows us to discover the intermediate variables more easily. The authors explore this technique in four experimental settings and empirically validate that the discovered circuit components play an important part in the circuit.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Thank you for your feedback on our work! We are happy to address these comments before publication. Please find specific responses to your comments below:\n\n“ The main method ... intermediate labels?”\n\nGreat question! The contrastive similarity loss is in some sense a natural choice for our goal, which is to uncover the set of weights required to compute a particular intermediate variable used by a transformer. This loss allows us to compute a loss value that just takes in intermediate representations, rather than logits. Consider using a CE loss for the same problem. Whatever subnetwork we discover will not produce logits, and so one cannot compute a CE loss directly One could train a classifier over the representations generated by these subnetworks (https://arxiv.org/abs/2104.03514). However, this recapitulates the problems with using a linear probe on our representations in the first place!  We are happy to elaborate on this in the main text.\n\n“How important is pruning as opposed to standard fine-tuning or PEFT?”\n\nI’m actually not quite sure what you mean here. Our goal is to uncover the set of model weights responsible for computing a particular intermediate variable in a transformer. I’d love to hear what you are thinking though!\n\n\" What if we performed prior probing methods...”\n\nWe {nonlinearly, linearly, amnesic} probe after the attention block and after the MLP block. The difference in performance of a standard probe from one to another can only be accounted for by the residual stream updates produced by either block, and so we do not anticipate that this would change our results. We ran linear probing on the update vectors for experiment 3, and found no change in results. We will repeat this for all of our experiments and include this in the appendix.\n\n“The causal analysis...”\n\nWe agree that counterfactual interventions provide stronger characterizations of the causal role of the intermediate variables that we discover, and this is a major goal of this set of projects. To do so, one must directly tie the representations generated by subnetworks to representations generated by complete model components. We hope to characterize this relationship in future work but thought it out of scope for this paper. However, we note that ablation has proven useful for interpretability (https://arxiv.org/abs/2006.00995, https://arxiv.org/pdf/2306.03819). \n\n\"While the paper ...\"\nWe are happy to include these results in the final submission for added context!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors study four applications for circuit probing in experimental setting ranging from toy experiments on specially small models (Experiments 1 and 2), a study of training dynamics (Experiments 3), and interpretability on a pre-trained language model with millions of parameters (Experiment 4)\n- The paper proposes imaginative new experimental settings (Experiments 1 and 2) that illustrate the role of intermediate variables and how they can lead to better understanding\n- The empirical analysis is very thorough and leverages a number of related works (amnesic probing, causal abstractions, linear probing, non-linear probing)", "reasons_to_reject": "- The main method could be better motivated, presented and ablated. How important  is the contrastive similarity loss? Would the method still work with a standard cross-entropy loss over the intermediate labels? How important is pruning as opposed to standard fine-tuning or PEFT? What if we performed prior probing methods on residual stream updates as opposed to hidden states?\n- The causal analysis in Experiments 2-4 is weak, as it relies too heavily on performance drops under targeted ablations compared to random ablation. Rather, the authors should consider activation patching and more targeted interventions to demonstrate causality.\n- While the paper makes a good case for the limitations of linear and non-linear probing in Experiments 1-3, Experiment 4 seems to lack these simple baselines.", "questions_to_authors": "It seems like the main caption for what should be Figure 2 does not exist?", "extra_scores": {}}, {"review_id": "vgbxYpsIDR", "reviewer": "Reviewer_XRXi", "summary": "This paper proposes a method for finding subnetworks (called \"circuits\") in transformers that calculate intermediate variables in a hypothesized causal model. The method is evaluated in terms of two goals. First, if a variable is part of the true causal graph, the method should find a subnetwork that calculates it reliably, and if the variable is not part of the causal graph, the method should return an empty subnetwork. Second, if a subnetwork is found, ablating the subnetwork should harm the performance of the model. The method is compared to probing methods (for the first goal) and causal analysis methods (for the second goal) and generally agrees with existing methods or compares favorably.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and comprehensive and helpful review! Please find responses to your concerns below.\n\n\"One of the proposed...\"\n\nRelating our method to this sort of counterfactual question was grounded in prior work (https://arxiv.org/pdf/2006.00995), though we will be more precise about the assumptions baked into that relationship (which you correctly identified) in the final submission. With respect to assumption (b), we encourage the discovered subnetworks to be as sparse as possible using L0 regularization, under the assumption that maximally sparse subnetworks should be less susceptible to computing many other causally relevant variables. This point will also be refined.\n\n\"More generally...\"\n\nAs mentioned above, we are happy to include a more formal statement of the counterfactual question, and formally list the assumptions therein.\n\n\"The paper argues...\"\n\nIn experiment 1, we derive ground truth by consensus of all existing methods. In Experiments 2 and 3, we used tasks that contain at least one variable that is explicitly useless in deriving the correct answer. They are (by construction) not causal variables in a model that succeeds at the task. We considered using Tracr, but went with this approach because the weight matrices of Tracr models are artificially constructed, and thus somewhat less realistic. \n\n\"I do not always find the...\"\n\nWe will tone down these claims. However, this subnetwork does appear to be causal. For qualitative evidence of this, see Table 8 in the appendix. In these examples, we observe that ablation pushes verbs that exhibit the correct syntactic number down in the ranking, equal to (or worse than) verbs exhibiting the incorrect syntactic number. Happy to expand on this in the main text!\n\nScientifically, this contributes toward understanding the internal organization of language models. The subnetworks that we identify as partially computing a syntactic dependency consist of approximately 15% of neurons in a single layer! This localization of intermediate variables is not obvious to us a priori. \n\n\"The method involves..\"\n\nFor suggestion (a), see the response to reviewer UN6h. For suggestion (b): We are happy to include this as a baseline in the work. Specifically: rather than having a linear probe map from embedding dimension to N classes, have it map to D dimensions, where D. Take the D-dimensional representations and compute soft nearest neighbors loss in that space. Is this what you had in mind?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper brings together ideas from pruning, circuit discovery, probing, and causal analysis. Causal analysis methods are designed to test whether neural network representations can be abstracted as high-level variables in a causal model, whereas circuit discovery methods can help characterize the mechanisms the model uses to compute these variables. This kind of combined approach could be a promising tool for understanding transformers at different levels of abstraction.\n\n- The paper includes experiments in a variety of different settings and compares circuit probing to a number of sensible baselines. The experiments suggest that circuit probing can help answer a variety of different interpretability questions.\n\n- The experiments indicate that circuit probing generally agrees with existing methods and in some cases performs better, in the sense that circuit probing agrees more with our expectations. For example, in section 4.3, circuit probing seems to be work than other probing methods as an explanation for \"grokking\" training dynamics.", "reasons_to_reject": "- One of the proposed benefits of this approach is that it enables a form of causal analysis, but it is not clear to me how much the method actually reveals about the causal model implicit in the network. In particular, the causal analysis involves ablating the identified circuit, and the paper states that \"this ablation is equivalent to asking the counterfactual question 'How does the model’s output change if it does not compute a particular intermediate variable in a particular block?'\" However, I do not see how these are equivalent. The question assumes that the chosen subnetwork (a) computes a particular intermediate variable, and (b) does not do anything else. That is, ablating the subnetwork could have other effects on the model that are not related to the intermediate variable. More generally, the paper would be stronger if claims about causality could be stated more formally. For example, existing work on causal analysis (e.g. [1]) uses a more more formal definitions of the sense in which the method can support claims about the causal structure of the neural network. It would be helpful if this paper could be more precise in explaining the type of causal claims that could be supported by the method.\n\n- The paper argues that circuit probing is better than existing probing methods because it is more \"faithful\" to the original model but it does not provide a clear definition of faithfulness, and it is not clear to me how the experiments demonstrate that circuit probing is more faithful than other methods. In the introduction, faithfulness is defined parenthetically as \"it only provides evidence in support of causal variables that are actually represented by the model,\" but I do not see how there is any ground-truth way of knowing what causal variables the model actually represents. (One possible approach would be to include experiments with networks where the ground truth causal structure is known, e.g. using [Tracr](https://arxiv.org/abs/2301.05062) [2].)\n\n- I do not always find the experiments to be very compelling. Three of the four experiments study one-layer networks trained on very simple tasks, which do not clearly demonstrate that this is an appropriate method for analyzing transformer language models. The fourth experiment studies language modeling tasks, but I am not entirely convinced by the conclusions. Section 4.4 states that \"the dependency is computed in layer 6's attention block\", but this seems to overstate the result. From Appendix Figure 15, it seems that ablating a subnetwork in this block leads to a larger drop in performance relative to other layers. However, the drop still isn't especially large (e.g. ~95% to ~80% on SV agreement), and ablating circuits in other components also induces a drop in accuracy, albeit not as large. Even if the effect could be entirely localized to this component, it is not clear to me how significant this type of insight is.\n\n- I find the description of the method in section 2 to be somewhat vague. It would be helpful to start with a more formal statement of the problem, and describe the method more formally. One particular point that confused me: When the method is applied to some component C, does this mean that the mask is learned only to the parameters in C? Or is the mask learned for all components prior to C in the computation graph, but using the representations from C to calculate the loss? Is there any justification for one or the other? Without explaining these details, it is difficult to understand how the method works, or even the exact problem it is meant to solve.\n\n- The method involves two ideas: one is to use pruning/subnetwork analysis as a probing method, and the other is to find these subnetworks by optimizing a nearest neighbor loss. These ideas seem to be distinct to me, and I think it would make sense to evaluate these two components of the method individually, that is: (a) the same approach but replacing the nearest neighbor objective with a standard linear probing method; and (b) training a linear probe on the network representations, but using the nearest neighbor objective.\n\nOverall, I think the paper introduces an interesting approach to interpretability, but I think a number of the key points need to be formulated more precisely to be able to determine the types of questions this method is capable of answering.\n\n*References:*\n\n[1] Geiger et al., 2021. Causal Abstractions of Neural Networks.\n\n\n[2] Lindner et al., 2023. Tracr: Compiled Transformers as a Laboratory for Interpretability.\n\n\n*Update:* After reading the rebuttal, I still have some doubts about the significance of the method and findings, but the authors have promised to address some of my concerns about clarity and presentation, so I am increasing my score from 4 to 5.", "questions_to_authors": "I have no additional questions", "extra_scores": {}}, {"review_id": "OXqtKrtU6R", "reviewer": "Reviewer_Dcii", "summary": "This paper presents a novel technique called circuit probing for identifying and analyzing intermediate variables in Transformer models. Circuit probing optimizes a binary mask over model weights to uncover a circuit that computes a hypothesized intermediate variable, then tests if the variable is represented in the model and causally implicated in model behavior, and look for model components whose outputs are partitioned according to the variable. The authors conduct four experiments that demonstrate circuit probing's ability to identify intermediate variables, differentiate between algorithms, and evaluate modularity, outperforming existing probing techniques.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments! See replies to your individual points below:\n\n\"Accessibility and Clarity:...\"\n\nWe are happy to add some higher-level intuitions into the paper, especially in Section 2! \n\n\"Generalization:....\"\n\nWe agree that working on a wider variety of complex models (e.g. vision transformers) is interesting and important. However, this was out of scope for this paper. We analyzed GPT2-small and medium for three particular intermediate variables in a language modeling task (two in the main text, one in the appendix). This was to establish that the method generalizes well to larger architectures trained on much more complex tasks (language modeling). \n\n\"Computational Efficiency: ...\"\n\t\t\nHappy to add an appendix section on this! In short, an optimized version of this method relies on forward passes through the model from the first layer through the Nth, where N is the layer to be probed. The backward pass only needs to be taken at the Nth layer, as that contains the binary masking parameters. In terms of memory requirements, this technique introduces one parameter per masking unit. In this work, we masked at the neuron level, though one could just as easily mask at the weight level or attention head level. In total, this is not very many parameters in comparison with the rest of the model.\n\n\"How robust...\"\n\nWe largely used the default set of hyperparameters suggested by Savarese et al. https://arxiv.org/abs/1912.04427. The one parameter that we change per model is the L0 regularization strength parameter. This is set to initialize the L0 loss in the range of .1-1. Because the maximum (and initial) L0 loss depends on the number of parameters you are masking over, this has to be set on a per model basis: the same L0 regularization strength for a toy model would dominate the loss in GPT2-Medium. In short, this method does not require much hyperparameter tuning at all!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty: Circuit probing is a valuable and innovative contribution to the field of interpretability research, offering a method to understand intermediate variables in Transformer models more effectively than traditional probing methods, \n•\tFaithful to model's computation than probing or causal abstraction analysis; \n•\tNot require full causal graph specification like causal abstraction analysis; \n•\tUncover development of circuits during training unlike other methods.\n2. Diverse Experimentation: The authors validate their methodology with comprehensive experimentation. Applying circuit probing across different tasks, including simple arithmetic tasks and real-world language models, highlights the robustness and versatility of the method.\n3. Strong Comparative Analysis: Comparing circuit probing to existing probing methods helps establish its efficacy. The ability to ablate specific circuits and demonstrate causal changes in model behavior is a notable advantage.", "reasons_to_reject": "1. Accessibility and Clarity: The paper's complexity may limit understanding for researchers not specialized in interpretability or Transformer models. Providing more intuitive explanations and detailed visualizations could enhance accessibility.\n2. Generalization: While the experiments are comprehensive, additional work applying circuit probing to different neural network architectures or more complex, real-world tasks would reinforce the paper's claims.\n3. Computational Efficiency: Discussing the computational requirements of circuit probing would provide practical insights for broader application. An analysis of scalability could strengthen the paper.", "questions_to_authors": "1.\tWhat is the scalability of circuit probing when applied to larger, state-of-the-art models used in complex real-world applications? The current study focuses on relatively small models and straightforward tasks like arithmetic.\n2.\tHow robust are the identified circuits to variations in hyperparameters, such as optimization functions, regularization methods, learning rate schedules, etc.?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are all supportive of this paper, which presents and evaluates a new probing method for understanding the computation of a latent within a large neural network. The new method jointly learns a binary mask, identifying a sparse subcircuit that computes the probed variable within the model.

Strengths:

Reviewers agree that the paper is interesting, bringing together ideas from both probing and circuit identification methods, and testing the method on a diversity of settings and networks, including some interesting experiment settings that pull apart the latent variables that emerge within cleverly chosen synthetic problems that one reviewer described as "imaginative".  The new method and the new experiments are meaningful contributions to the literature and they are well-grounded in previous ideas.

Weaknesses:

The reviewers all agree that the presentation of the method is not very clear: Section 2 is a very informal description of the method that reviewers described as "vague", whose "complexity may limit understanding" that "could be better presented". In short, there is no crisp description of the method in the main paper, and many of the key details are left to the appendix or to the supplied code.

This is a significant issue: a clear presentation of the method is especially important if the new method hopes to be understood and adopted by other researchers - it would be a shame if the promising new method were not adopted due to lack of clarity of presentation. While the review consensus is that the paper is above the bar, I would strongly encourage the authors to spend some time for a careful revision of section 2 to improve the accessibility and usefulness of the paper to a  broader set of readers.

Some of the reviewers also found that the causality claims were insufficiently supported by results, and the authors have offered to tone down, formalize and clarify the claims in this area.

[comment from the PCs] It's critical the authors revise their paper following the suggestions of the AC, especially to address clarity issues.
Decision: Accept