Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models
Authors: Samuel Marks
Can Rager
Eric J Michaud
Yonatan Belinkov
David Bau
Aaron Mueller
Venue: ICLR 2025 Oral (2025)
Keywords: Interpretability
mechanistic interpretability
circuits
spurious correlations
generalization
dictionary learning
Primary area: interpretability and explainable AI
TL;DR: We automatically discover circuits of interpretable components and apply them to remove sensitivity to spurious correlates

Abstract: We introduce methods for discovering and applying **sparse feature circuits**. These are causally implicated subnetworks of human-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and difficult-to-interpret units like attention heads or neurons, rendering them unsuitable for many downstream applications. In contrast, sparse feature circuits enable detailed understanding of unanticipated mechanisms in neural networks. Because they are based on fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of a classifier by ablating features that a human judges to be task-irrelevant. Finally, we demonstrate an entirely unsupervised and scalable interpretability pipeline by discovering thousands of sparse feature circuits for automatically discovered model behaviors.

Published as a conference paper at ICLR 2025
SPARSE FEATURE CIRCUITS: DISCOVERING AND EDITING INTERPRETABLE CAUSAL GRAPHS IN LANGUAGE MODELS
Samuel Marks* Northeastern University
Can Rager Independent
Eric J. Michaud MIT
Yonatan Belinkov Technion- - III
David Bau Northeastern University
Aaron Mueller Northeastern University
ABSTRACT
We introduce methods for discovering and applying sparse feature circuits. These are causally implicated subnetworks of numan-interpretable features for explaining language model behaviors. Circuits identified in prior work consist of polysemantic and teffiuull-t--nnteppeee units like attention heads or neurons, ren- dering them unsuitable for many downstream applications. In contrast, sparse fea- ture circuits enable detailed understanding of unanticipated mechanisms in neural networks. Because they are based fine-grained units, sparse feature circuits are useful for downstream tasks: We introduce SHIFT, where we improve the generalization of classifier by ablating features that human judges be task- irrelevant. Finally, we demonstrate an entirely unsupervised and scalable inter- pretability pipeline by discovering thousands of sparse feature circuits for auto- matically discovered model behaviors.
INTRODUCTION
The key challenge of interpretability research is to scalably explain the many unanticipated behav- iors of neural networks (NNs). Much recent work explains NN behaviors in terms coarse-graine grained model components, for example by implicating certain induction heads in in-context learning (Ols-
son et al. 2022) or ML.P modules in factual recall (Meng al., 2022 Geva et al. 2023 Nanda et al. 2023 inter alia). However, such components are generally polysemantic (Elhage et al., 2022) and hard to interpret, making difficult to apply mechanistic insights to downstream ap- plications, On the other hand, prior methods for analyzing behaviors in terms of fine-grained units Kim et al., [0188] [ellinov]] 2022 Geiger et al. 2023 Zou et al. (023) attempt to fit model internals to researcher-specifee mechanistic hypotheses using researcher-curatee data. These approaches not well-suited to the many cases where researchers cannot anticipate ahead of time how models internally implement their surprising behaviors.
We propose to explain model behaviors using fine-grained components that play narrow, inter- pretable roles. Doing so requires to address challenges: First, must identify appro- priate fine-grained unit of analysis, since obvious choices like neuron are rarely interpretable, and units discovered via supervised methods like linear probing require pre-existing hypotheses (Mueller et al. 2024). Second, we must address the scalability problem posed by searching for causal circuits over large number of fine- mne-grained units.
We leverage recent progress in dictionary learning for NN interpretability (Bricken et al. 2023 Cun- ningham et al. (0024) to tackle the first challenge. Namely, we use sparse autoencoders (SAEs) to identify directions in LM latent spaces which represent human-interpretable concepts. Then, to ad- dress the scalability challenge, we employ linear approximations (Sundararajan et al. 2017 Nanda
Correspondennce to s. marks@northeasterr edu and aa. mue wee.lerrrreeesteeenccmm Wee use "neuron" to refer a basis-aligned direction in an LM's latent space (not necessarily preceded by nonlincarity).

---

Published as a conference paper at ICLR 2025
Data
$3
The boy the teacher The boys near the teachor have
Contrastive Pairs
Get feature circuit.
Interpret circuit.
Debug desired.
m
$3
$3,4,5
m
Classification Data
$4
lis revesrch is in Profecsor She worked in the OR Nurse
85
Auto-discovered Behaviors Part 1, Part 2, Part 3, Part 4
Figure 1: Overview. Given contrastive input pairs, classification data, or automatically discovered model behaviors, discover circuits composed of human-interpretable sparse features to explain their underlying mechanisms. We then label each feature according to what it activates on or causes the model predict. Finally, if desired, can ablate spurious features out of the circuit to modify how the system generalizes.
2022 Syed et al. 2023) to efficiently identify SAE features which are most causally implicated in model behaviors, as well as connections between these features. The result is a sparse fea- ture circuit which explains how model behaviors arise via interactions among fine- fineggrained human- interpretable units.
Sparse feature circuits be productively used in downstream applications. We introduce tech- nique, Sparse entererlnterreetable Feature Trimming (SHIFT; $4). which shifts the generalization of LM classifier by surgically removing sensitivity to unintended signals. Unlike previous work on spurious cue rehcval-which isolates spurious signals using disambiguating data-StuFT iden- tifies unintended signals using interpretability and human judgement. We thus showcase SHIFT by debiasing a classifier in worst-case setting, where an unintended signal (gender) is perfectly predictive of target labels (profession).
Finally, we demonstrate our method's scalability by automatically discovering thousands of narrow LM behaviors- for example, predicting "to" as an infinitive object or predicting commas in dates- with the clustering approach of Michaud et al. (2023), and then automatically discovering feature circuits for these behaviors ($5).
Our contributions are summarized as follows (Figure
1. A scalable method to discover sparse feature circuits. We validate our method by discov- ering and evaluating feature circuits on suite of subject-verb agreement tasks.
2. SHIFT, a technique for removing a LM classifier's sensitivity to unintended signals, even without data that isolate these signals.
3. A fully-unsupervisee pipeline for computing feature circuits for thousands of automatically discovered LM behaviors, viewable at feffeatre-cicciiss...
We relcase code, data and autoencoders at aatttu..cc...sspmmrrsss/sttrrrrr
FORMULATION
Feature disentanglement with sparse autoencoders. A fundamental challenge in NN inter- pretability is that individual neurons are rarely interpretable (Elhage et al. 2022). Therefore, many interpretability researchers have recently turned to sparse autoencoders (SAEs), an unsupervised technique for identifying large number of interpretable NN latents (Cunningham et al.) [024] Bricken et al., [2023] Templeton et al. 22024) Rajamanoharan et al., 2024a b). Given a model com- ponent with latent space R and an activation E Ra, SAE computes decomposition
deaR
x= x x= = Σ f(x)vv + b E(X)
(1)
into approximate reconstruction & sparse sum of features v, and an SAE error term c(x) Rd. Here dsAR is the width of the SAE, the features V; E Ed are unit vectors, the feature activations fx(x) 2 0 are sparse set of coefficients, and b Rd is bias. SAEs are trained on objective
2

---

Published as a conference paper at ICLR 2025
which promotes having a small reconstruction error ||x - while using only sparse set of feature activations fxx. Rather than discard the error terms e for the purposes of circuit discovery, our methods handle them gracefully by incorporating them into our sparse feature circuits: this gives a principled decomposition of model behaviors into contributions from interpretable features and error components not yet captured by our SAEs.
In this work, we leverage the following suites of SAEs:
A suite of SAEs we train for each sublayer (attention layer, MLP, residual stream, and embeddings) of Pythia-70M (Biderman et al. 2023). We closely follow Bricken et al. (2023), using ReLU-linear encoder f. and sparse dimension dSAE = 64 x d and training the SAEs to minimize combination of an L2 reconstrution loss and L1 regularization term which promotes sparsity. Details about our Pythia SAEs and their training can be found in Appendix B.I
The open source Gemma Scope SAEs (Lieberum al.) 2024) available for all sublay-
ers (excluding embeddings) of the open-weights Gemma-2-21 2-22 model (Team et al.) 2024). These SAEs use Jump-Rer Cemp-ReLL-llneaar encoder and dSAE 8 Details about the Gemma Scope SAEs can be found in Appendix B.2
Scalably training better SAEs is an active area of research, as illustrated by the ready availability of
open-source SAES (Gao et al. 2024 Lieberum et al.] 20244] Lin & Bloom 2023). Thus, our goal to govenven& suite of trained SAES scalably apply them to understand NN behaviors; we treat
scaling the SAEs themselves as out-of-scope.
Attributing causal effects with linear approximations. Let be real-valued metric computed via a computational graph (e.g.. NN); let a represent a node in this graph. Following prior work (Vig al. 2020, Finlayson et al., 2021), we quantify the importance of on pair of inputs
[Feleann patch via its indirect effect (IE; Pearl 2001 on m:
IE(m; a; FFlleun Tpaich)
m (Feieum (Fctemm/do(a = apunch)) ((
Here, apatch is the value that takes in the computation of m (Zaachh) and m denotes the value m when computing m (Xectem)) but intervening in the computation of m by manu- ally setting a to putch- For example, given inputs Teteem "TThe teacher" and paach ='The teachers," we have metric m(x)) = log P("are" xr) - log P("is"(=)), the log probability difference output by the LM. Then if a is the activation of particular neuron, large value of IE(m; a; Feteans Tpucch) indicates that the neuron is highly influential on the model's decision to output "is" vs. "are" on this pair of inputs.
apatch)
We often want to compute IEs for a very large number of model components a E Rdd, which cannot be done efficiently with (2). We thus employ linear approximations to (2) that can be computed for many a in parallel. The simplest such approximation, attribution patching (Nanda 2022 Syed
ct al. 2023 Kramár et al. 2024). employs first-order Taylor expansion
1Emup (m; a; Feteun papcch)) m - (apacch
actean)
which estimates 2 for every in parallel using only two forward and one backward pass.
To improve the quality of the approximation, we can instead employ a more expensive but more
accurate approximation based on integrated gradients (Sundararajan et al. 2017 Hanna et al. 2024
Eeg (m; a Feleaa Ipputch) = N
Σ V m aaseo + (1-a) (1-a)apph
(agatch acleam) )
(4)
where the sum in (4) ranges over N = 10 equally-spaced E {0, N..... N N This cannot be done in parallel for two nodes when one is downstream of another, but can be done in parallel for arbitrarily many nodes which do not depend on each other. Thus the additional cost of computing HF over Fram scales linearly in N and the serial depth of m's computation graph.
The above discussion applies to the setting where we have pair of clean and patch inputs, and we would like to understand that effect of patching particular node from its clean to patch values. But in some settings (e.g.. $ 5) we have only a single input x. this case, we instead use a zero- ablation, using the indirect effect IB(m; a; mm( = 0)) m(x) from setting to O. We
get the modified formulas for 185m;; a; 2;) from and 4 by replacing a with O.

---

Published as a conference paper at ICLR 2025
0+
SAE feature
SAE error
Submodule
1
Cache activations and metric.
2
Backpropagate. Store gradients.
3
Compute effects. Filter nodes.
4
Compute and filter edges.
m = log p(have) log p(has)
m
Va
m
V. 41
m
m
m
b
b
(B V.m-
x The teacher x The teachers
[BGr, mi)
Overview of our method. We view our model as a computation graph that includes SAE features and errors. We cache activations (Step and compute gradients (Step 2) for each node. We then compute approximate indirect effects with Eq. (3) shown) or (4) and filter according to node threshold T'y (Step 3). We similarly compute and filter edges (Step 4); see App. A.1
Figure 2:
SPARSE FEATURE CIRCUIT DISCOVERY
3.1 METHOD
Suppose we are given an LM M. SAEs for various submodules of M (c.g., attention outputs, MLP outputs, and residual stream vectors, as in $2). a dataset 'D consisting either of contrastive pairs (Ficlea;: patchh) of inputs of single inputs v. and a metric that depends on M's output when processing data from D. For example, Figure 2 shows the case where 'D consists pairs of inputs which differ in number, and m is the log probability difference between M outputting the verb form that is correct for the patch vs. clean input.
Viewing SAE features part of the model. A key idea underpinning our method is that, by apply- ing the decomposition (T) to various hidden states x in the LM, we can view the feature activations fi and SAE emrоrs being part of the LM's computation. We can thus represent the model as a computation graph G where nodes correspond to feature activations or SAE errors at particular token positions.
Approximating the IE of each node. Let IE be one of IEsp lEg (see $ Then for each node G and input x DD we compute IE(m; a; r); we then average over x to produce a score [m(;; a) and filter for nodes with [[{(m;;) a) > Ty for some choice Ty of node threshold.
Consistent with prior work Nanda 2022 Kramár et al. 2024 we find that ΙĚap accurately esti- mates IEs for SAE features and SAE errors, with the exception of nodes in the layer 0 MLP and early residual stream layers, where Ir underestimates the true IE. We find that Ifg significantly improves accuracy over Irap for these components, so we use our experiments below. See Appendix for more information about linear approximation quality.
Approximating the average IE of edges in the computation graph. Although the idea is simple, the mathematics somewhat involved, so we relegate the details to App. A.I After computing these IEs, we filter for edges with absolute IE exceeding some edge threshold Tg-
edges. Using an analogous linear approximation, we also compute the
Aggregation across token positions and examples. For templatic data where tokens in matching positions play consistent roles (see $3.2 3.3 we take the mean effect of nodes/edges across exam- ples. For non-tempin tom-lemmpaaic data ($ 5) we first sum the effects of corresponding nodes/edges across token position before taking the example-wise mean. See App. A.2

---

Published as a conference paper at ICLR 2025
Structure
Example clean input
Example output
Simple
The parents
p(is) (aare) plikes)) p(like)
Within RC
The athlete that the managers
Across RC
The athlete that the managers like
p (d) p ploees)
Across PP
The secretaries near the cars
p(has) p(have)
Table 1: Example clean inputs r and outputs m for subject-verb agreement tasks.
Fathftunees of
Completeness (Fathfulness of M'
Feoture dicutt
Feature dicutt wio SAE enrons
22-
Festure altentor-MLP
Pythi-77M
08
Nouron
Faihfunesa
1000
Gemma-2 -28
044
800
1000
1900
2000
Nodes
Figure 3: Faithfulness and completeness scores for circuits, measured on held-out data. Faint lines correspond to the structures from Table [ with the average across structures in bold. The ideal faithfulness for circuits is 1, while the ideal completeness is O.
Practical considerations. Various practical difficulties arise for efficiently computing the gradi- ents needed by our method. We solve these using combination of stop gradients, pass-through gradients, and tricks for efficient Jacobian-vector product computation; see App. A.3
3.2 DISCOVERING AND EVALUATING SPARSE FEATURE CIRCUITS FOR SUBJECT- VERB AGREEMENT
To evaluate method, discover sparse feature circuits (henceforth, feature circuits) Pythia- 70M and Gemma-2-2B for four variants of the subject-verb agreement task (Table Specifically, adapt data from Finlayson al.) (2021) to produce datasets consisting of contrastive pairs of inputs that differ only in the grammatical number of the subject; the model's task is to choose the appropriate verb inflection.
We evaluate circuits for interpretability, faithfulness, and completeness. For each criterion, we compare to neuron circuits discovered by applying our methods with neurons in place of sparse features; in this setting, there are no eтrоr terms e. When evaluating feature circuits for faithfuless and completeness, we use a test split of our dataset, consisting of contrastive pairs not used to discover the circuit.
Interpretability. For Pythia SAEs, we asked human crowdworkers to rate the interpretability of random features, random neurons, features from our feature circuits, and neurons from our neuron circuits. Crowdworkers rated sparse features as significantly more interpretable than neurons, with features that participate in our circuits also being more interpretable than randomly sampled ones (App. F This replicates prior findings that SAE features are substantially more interpretable than neurons (Bricken et al., (023). For Gemma-2 SAES, refer the reader to Lieberum et al. (2024), which finds the interpretability these SAES' features to be on par with those trained via other state-of-the-ee ofrtt---rtttttte techniques.
Faithfulness. Given a circuit C' and metric m, let mC) denote the average value of m over inputs from D when running model with all nodes outside of C mean-ablated, i.e., set their average
5

---

Published as a conference paper at ICLR 2025
value over data from D We then measure the faithfulness of a circuit m(C)-m(() (x(M))m())) where Ø denotes the empty circuit and M denotes the full model. Intuitively, this metric captures the proportion of the model's performance our circuit explains, relative to mean ablating the full model (which represents the "prior" performance of the model when it is given information about the task, but not about specific inputs).
We find that components in early model layers are typically involved in processing specific tokens. In practice, the inputs in the train split of our dataset (used to discover the circuit) and the test split (for evaluation) do not contain identical tokens, making difficult to evaluate the quality of early segments of our circuit. Thus, we ignore the first 1/s of our circuit, and only evaluate the latter 2/3.
We plot faithfulness for feature circuits and neuron circuits after sweeping over node thresholds T'v (Fig. 3). We find that small feature circuits explain a large proportion of model behavior: the major- ity of performance in Pythia-7MM. resp. 1eemma-2-28 is explained by only 100, resp. 500 nodes. In contrast, around 1500, resp. 50000 neurons required to explain half the performance. However, as SAE error nodes are high-dimensional and coarse- grained, they cannot be fairly compared to neu- rons; thus also plot the faithfulness of feature circuits with all SAE error nodes removed, with all attention and MIP error nodes removed. Unsurprisingly, we find that removing residual stream SAE error nodes severely disrupts the model and curtails its maximum performance; removing MLP and attention error nodes is less disruptive.
Completeness. Are there parts of the model behavior that our circuit fails capture? We measure this as the faithfulness of the circuit's 's complement M' \ C (Fig. We observe that we can eliminate the model's 's task performance by ablating only a few nodes from our feature circuits, and that this is true even when we leave all SAE errors in place. In contrast, it takes hundreds (for Pythia) or thousands (for Gemma) of neurons to achieve the same effect.
3.3 CASE STUDY: SUBJECT-VERE AGREEMENT ACROSS A RELATIVE CLAUSE
We find that inspecting small feature circuits produced by our technique can provide insights into how Pythia-700 70M and Gemma-2-22 2-2B arrive at observed behaviors. To illustrate this, we present a case study of relatively small feature circuits for subject-verb agreement across relative clause (RC).
To keep the number of nodes we need to annotate manageable, we tune our node threshold to pro- duce small circuit with faithfulness 0.2. For Pythia, this results in circuit with 86 nodes and faithfulness 0.21; for Gemma we study a circuit with 223 nodes and faithfulness 0.21. We sum- marize these circuits in Figure 4 the full circuits (as well as small circuits for other subject-verb agreement tasks) can be found in App. C.1 We depict SAE features with rectangles and SAE errors with triangles.
Our circuits depict interpretable algorithms wherein both models of study select appropriate verb forms via two pathways. The first pathway consists of features which detect the number of the main subject and then generically promote matching verb forms. The second pathway begins the same, but moves the relevant number information to the end of the relative clause by using PP/RC boundary detectors. Gemma 2 also uses noun phrase (NP) number trackers, which detect the number of the noun that heads an NP and remain active on all tokens until the end of the NP; these promote matching verb forms at each position, but especially at the last token of an NP.
We find significant overlap between this circuit and the circuit we discovered for agreement across a prepositional phrase, with Pythia-70M and Gemma-2-20 handling these syntactically distinct struc- tures in a mostly uniform way. In accordance with Finlayson et al. (202I). we find less overlap with our circuits for simple agreement and within RC agreement (Appendix
4 APPLICATION: REMOVING UNINTENDED SIGNALS FROM A CLASSIFIER WITHOUT DISAMBIGUATING LABELS
NN classifiers often rely on unintended signals- spurious features. Nearly all prior work on mitigating this problem relies on access to disambiguating labeled data in which unintended signals are less predictive of labels than intended ones. However, some tasks have structural properties
Following Wang et al. 2023), we ablate features by setting them their mean position-speciific values.

---

Published as a conference paper at ICLR 2025
has/have
Verb form discriminators
Bribeddiigg. layer 0-4 MLP: resid
Layers 2-1 amn, MLP resid
Noun number detection
PP/CC detection
PARC end detection
16
Layers 4 atmn, resid
The
girligiels
that
the
teacher
SeeS
has/have
Verb form discriminators
ortm MLP
Layers resid NP number trackers
PPRC detectors
Layers II 11-3777-- resid
Layers 16-15
B2
MLP,
NP number trackers
residesid
4
Noun number derectors
NP number trackers
PP/CC detectors
14
33
Laytrs 5-21 atin, MLP, resid
Layers D-17 attn, MI..
The girligirls
that
the teacher
sees
Figure 4: Summary of Pythia's (a) and Gemma 2's (b) circuits for agreement across RC (full circuits in App. C.I.. The models detect the number of the subject. Then, they detect the start of a PP/RC modifying the subject. Verb form discriminators promote particular verb inflections (singular or plural). Gemma additionally uses separate features to track the number the noun that heads the current noun phrase. Squares show number of feature nodes in the group and triangles show number of SAE error nodes, with the shading indicating the sum of IE terms across nodes in the group. As we cannot directly interpret the triangles, we rely on their positions or inclusion other groups to label them. If the label ambiguous, we leave the triangles outside the boxes.
which disallow this assumption. For example, inputs for different classes might come from different
data sources (Zech et al., 2018). Additionally, some have raised concerns (Ngo et al. 2024 Casper al. 2023) that sophisticated LMs trained with human feedback (Christiano al. (2023) settings with casy-to-hard domain shift (Burns et al., [2033; Hase et al. 2024) will be misaligned because, in these settings, "overseer approval" and "desirable behavior" are equally predictive of training reward labels. More fundamentally, the problem with unintended signals is that they are unintended not they are insufficiently predictive-and we would like our methods to reflect this.
We thus propose Spurious jumma--nterppettblle Feature Trimming (SHIFT), where human inspects a classifier's feature circuit and removes features which they judge be task-irrelevant. We show that SHIFT removes sensitivity to unintended signals without access disambiguating labeled data, or even without knowing what the signals are ahead of time.
Method. Suppose we are given labeled training data D ((11, w:):: an LM-based classifier C trained on D; and SAEs for various components C. To perform SHIFT, we:
1. Apply the methods from $ to compute feature circuit that explains C's accuracy inputs (x, 3) ~ D (c.g., using metric m = log Cly x)).
2. Manually inspect and evaluate for task-relevancy each feature in the circuit from Step 1.
3. Ablate from C' features judged to be rask-irrelevant to obtain a classifier C².
4. (Optional) Further fine-tune C" data from D.
Step 3 removes the classifier's dependence on unintended signals we can identify, but may disrupt the classifier's performance for the intended signal. Step 4 can be used restore some performance.

---

Published as a conference paper at ICLR 2025
Pythia-70M
Gemma-2-2B
Method
PProfession
[.enddr
Worst group
Profession
[Gender
WWorst group
Original
61.9
87.4
24.4
67.7
81.9
18.2
CBP
83.3
60.1
67.7
90.2
50.1
86.7
Random
61.8
87.5
24.4
67.3
82.3
18.0
SHIFT
88.5
54,0
76.0
76.0
51.5
50.0
SHIFT retrain
93.1
52.0
95.0
52.4
92,9
Neuron skyline
75.5
73.2
41.5
65.1
84.3
5.6
Peature skyline
80.8
53.7
56.7
Oracle
93.0
49.4
91.9
95.0
50.6
93.1
Table Accuracies on balanced data for the intended label (profession) and unintended label (gen- der). "Worst group accuracy" refers to whichever profession accuracy is lowest among male profes- sors, male nurses, female professors, female nurses.
Experimental setup. We illustrate SHIFT using the Bias in Bios dataset (BiB; Arteaaa et al, 2019). BiB consists of professional biographies, and the task is to classify an individual's profession based on their biography. BiB also provides labels for a spurious feature: gender. We subsample BiB produce sets of labeled data:
The ambiguous set, consisting of bios of male professors (labeled 0) and female nurses (laheled 1).
The balanced set, consisting of an equal number of bios for male professors, male nurses, female professors, and female nurses. These data carry profession labels (the intended signal) and gender labels (the unintended signal).
The ambiguous represents a worst-case scenario: the unintended signal is perfectly predictive of training labels. Given only access to the ambiguous set, our task is to produce profession classifier which accurate on the balanced set.
We adapt Pythia-70M and Gemma-2-2B into classifiers by training linear classification heads with the ambiguous set; see App. E. for probe training details. We then discover feature circuits for these classifiers using the zero-ablation variant described in $3.1] the Pythia circuit contains 67 fea- tures, and the Gemma circuit contains 46. We manually interpret each feature using the Neuronpedia interface (Lin & Bloom 2023), which displays maximally activating dataset exemplars on large text corpus, as the features' direct effects on output logits. We judge 55 of the Pythia features and 43 of the Gemma features to be iesk-ireeevaant- e.g., features that promote female-asses associated lan- guage in biographies of women, as in Figure 19 (see App. DJror more examples features). Although this interpretability step uses additional unlabeled data, emphasize that never additional labeled data (or even additional unlabeled classification data).
To apply SHIFT, zero-ablate these irrelevant features. Finally, we retrain the linear classification head with the ambiguous set using activations extracted from the ablated model. We evaluate all accuracies on the balanced set.
Baselines and skylines. To contextualize the performance of SHIFT, we also implement:
SHIFT with neurons. Perform SHIFT, but using neurons instead of SAE features.
Concept Bottleneck Probing (CBP), adapted from Yan et al. (2023) (originally for multi- modal text/image models). CBP works by training a probe classity inputs æ given access only to vector of affinities between the LM's representation of x and various concept vectors. See App. E.2 for implementation details.
Random feature ablations. Perform SHIFT, but using (the same number of) randomly selected SAE features instead of features selected by human annotator.
Feature skyline. Instead of relying human judgement to evaluate whether feature should be ablated, we zero-ablate the 55 (for Pythia) or 43 (for Gemma) features from our circuit that most causally implicated in spurious feature accuracy the balanced set.
Neuron skyline. The same as the feature skyline, but mean-ablating 55 or 43 neurons.
Oracle. A classifier trained on ground-truth labels the balanced set.
Results. We find (Table 2) that SHIFT almost completely removes the classifiers' dependence on gender information for both models. In the case of Gemma (but not Pythia), the feature ablations
8

---

Published as a conference paper at ICLR 2025
Cluster 382: Incrementing sequences
Cluster 475: infinitive object
input 11, 2 3, 4,
esse, wheter
allowed
Step Downlood loteet CompoNit 3.49 Full
Britih Prime Min David Cameron says in evised remarts he would like Briainj Reader broggers are asked
prompts L.J After main install closes, click OK [--)
Setup
Step
Example features involved:
Example features involved:
Sucuession
Narrow induction
Objects whit which can preccuee
words which precee precode
Chapter
A
B B
AS
........
or BI 8...
objjcct complements
Chacter
[Direct the juser to It's up youu
Aosordiing
This intection
Figure 5: Example clusters and features which participate in their eircuits (see App. C.3 for the full circuits). Features are active on tokens shaded in blue and promote tokens shaded in red. (left) An example narrow induction feature recognizes the pattern A3... A and copies information from the 3 token. This composes with succession feature to implement the prediction A3... A + 4. (right) One feature promotes "to" after words which can take infinitive objects. separate feature activates on objects of verbs or prepositions and promotes "to" as an object complement.
damage model performance; however, this performance is restored (without reintroducing the bias) by further training on the ambiguous set. Comparing SHIFT without retraining to the feature skyline, we further observe that SHIFT optimally or near-optimally identifies the best features remove.
SHIFT critically relies on the use of properly selected SAE features. When ablating random SAE features, see essentially effect on probe performance. When applying SHIFT with neurons, essentially none of the neurons are interpretable, making it difficult to tell if they ought to be ablated; see [ppendix[]joor examples. Because this, we abandon the SHIFT with neurons baseline. Even using the balanced set to automatically select neurons for removal (the neuron skyline) fails to match SHIFT'S performance, as the neurons most implicated in spurious feature classification are also useful for ground-truth classification.
5 UNSUPERVISED CIRCUIT DISCOVERY AT SCALE
Previous work on circuit analysis relied on human-collected datasets to specify LM behaviors (Wang ct al. 2023] Conmy et al. 2023 Hanna et al. 2023). However, LMs implement numerous interest- ing behaviors, many of which may be counterintuitive to humans. In this section, we adapt our techniques to produce near-fully-autoomate interpretability pipeline, starting from a large text herp, large subset of The Pile (Gao al., 2020) and ending with thousands of feature circuits for auto-discovered model behaviors. These experiments are performed with Pythia-70M.
We proceed in two steps:
1. Behavior discovery via clustering, We interpret our large text corpus as a dataset [(24, of contexts v; with ground-truth next tokens yi- Following Michaud al. (202)), we associate vector
v(z,, (t), to each sample and apply a clustering algorithm to {vi]: this segments our large corpus into a number of smaller subcorpora corresponding to the clusters. Although this approach is entirely unsupervised, many of the resulting sub- corpora capture interri-tterretablle model behaviors, such as predicting the next number in sequence (Fig 5). We experiment with number of ways of assigning (æzz V/) + Vi, such as using the training gradient V log Po (yr(2r;) as Michaud et al. (2023) as well as approaches which leverage SAE activations gradients. See App. G for details.
2. Circuit discovery. Given a subcorpus D {(x,, y. we apply the zero-ablation variant of our feature circuit discovery technique from $ using the dataset D and metric m log P(y. (z)) Thus, to each subcorpus we associate a feature circuit.
We present example clusters, well as interesting features participating in their associated circuits (Figure 55. An interface for exploring all of our clusters and (unlabeled) circuits can be found at feature-circuie xyz
While evaluating these clusters and circuits is an important open problem, we generally find that these clusters expose interesting LM behaviors, and that their respective feature circuits can provide

---

Published as a conference paper at ICLR 2025
useful insights on mechanisms of LM behavior. For instance, we automatically discover attention features implicated in succession and induction, two phenomena thoroughly studied in prior work at the attention head level using human-curated data (Olsson et al.
2022. Gould et al. 2023).
Feature circuits can also shed interesting light on their clusters. For example, while the clusters in Figure 5 seem first to each represent a single mechanism, circuit-level analysis reveals in both cases a union of distinct mechanisms. For cluster 475, Pythia-70M determines whether "to [verb]" an appropriate object in two distinct manners (see Figure 5 caption). And for cluster 382, the prediction of successors relies on general succession features, as well as multiple narrow induction features which recognize patterns like " "A"...A".
6 RELATED WORK
Causal interpretability. Interpretability research has applied causal mediation analysis (Pearl
2001
Robins & Greenland] 1992) to understand the mechanisms underlying particular model be-
haviors and their emergence Yu et al. 2023 Geva et al. [023]] Hanna et al.) 2023 TTdd et al. 2024
Prakash et al. 2024; Chen et al. 2024, inter alia). This typically relies on counterfactual inter-
coarse-grained components (Conmy et al. al.] 2023 Wang et al. 2023). Some techniques aim to, given hypothesized causal graph,
ventions (Lewis] 1973), such as activation patching path patching
identify& matching causal mechanism in an LM (Geiger al. 2021 2022 2023); in contrast, aim here to discover causal mechansisms without starting from such hypotheses.
Robustness to spurious correlations. There is large literature mitigating robustness spu- rious correlations, including techniques which rely on directly optimizing worst-group accuracy (Sagawa al.. 2020 Oren al., 2019) /Zhang et al. 2021 [ohoni al.
2022 Nam et al. 2022).
automatically or manually reweighting data between groups (Liu et al. 2021 Nam et al. 2020
Yaghoobzadch al. 2021 Utama et al. 2020; Creager et al. 2021 Idrissi al. 2022] Orgad &
Belinkov| (002)) training classifiers with more favorable inductive biases (Kirichenko et al. 2023
Zhang et al.) 2022 Iskander al. 2024). or editing out undesired concepts (Iskander et al. 2023
Belrose et al.) [023] Wang et al. 2020 Ravfogel et al.] 2020;; [022a b). All of these techniques
rely on access to disambiguating labeled data in the sense $4 Some techniques from a smaller
literature focused image or multimodal models apply without such data (Oikarinen et al. 2023
Yan et al. 2023). Our method here is inspired by the approach of Gandelsman et al. (2024) based on interpreting and ablating undesired attention heads in CLIP.
Feature disentanglement. In addition to recent work on SAEs for LM interpretability (Cunning-
ham et al. 2024 Bricken et al., 2023] Gao et al. 2024 Rajamanoharan al. 2024a b). other ap-
proaches to feature disentanglement include SScmidhuberr (1992); Desjardins et al. (2012): ; Kim &
Mnih (2018); CCen al. (2016); Makhzani & Frey (2013) He al. (2022) Peebles al. (2020)
Schneider & Vlachos (2021) Burgess et al. (201)) Chen et al. (018)) Higgins et al. (2017)
i.a.
7 CONCLUSION
We have introduced method for discovering circuits on sparse features. Using this method, we discover cumaan-nterpretabbe causal graphs for a subject-verb agreement task, a classifier, and thou- sands of general token prediction tasks. We can edit the set of features that models have access to by ablating sparse features that humans deem spurious; we find that this is significantly more effective than neuron-based ablation method which has an unfair advantage.
LIMITATIONS
The success of our technique relies access to SAEs for given model. Training such SAEs currently requires a large (but one-time) upfront compute cost. Additionally, model components not captured by the SAEs will remain uninterpretable after applying our method.
Much of our evaluation is qualitative. While we have quantitative evidence that feature circuits are useful for improving generalization without additional data (..). evaluating dictionaries and circuits without downstream tasks is challenging. Feature labeling is also qualitative process; thus, labels may vary across annotators, and may vary depending on the task of interest.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY
We release code, data and autoencoders at gi http://w.....pppprssss/setrrrrrr
Exper-
imental details can be found in appendices A [ and G Our experiments are conducted entirely
on open-weights models. The Gemma Scope SAEs are publicly available (Lieberum et al.) (024).
Features for both SAE suites can be browsed on Neuronpedia (Lin & Bloom 2023). Our clusters
and associated feature circuits can be browsed at feature-circuits.yy
ACKNOWLEDGMENTS
We thank Stephen Casper, Buck Schlegeris, Ryan Greenblatt, and Neel Nanda for discussion of ideas upstream to the experiments in $4 We thank Logan Riggs and Jannik Brinkmann for help training SAEs. We also thank Josh Engels and Max Tegmark for discussions about clustering and sparse projections related to $ S.M. is supported by an Open Philanthropy alignment grant. C.R. is supported by Manifund Regrants. E.J.M. is supported by the NSF via the Graduate Research Fellow- ship Program (Grant No. 2 2141064) and under Cooperative Agreement PHY-201975 (IAIFI). Y.B. supported by the Israel Science Foundation (Grant No. 448/20) and an Azrieli Foundation Early Career Faculty Fellowship. Y.B. and D.B. are supported by a joint Open Philanthropy alignment grant. A.M. supported by Zuckerman postdoctoral fellowship.
REFERENCES
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational
Linguistics, 48(1):207-219, 2022.
Nora Belrose, David Schneider-Joseph, Shauli Ravfogel. Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL awIpKpwTwF
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning. pp. 2397-2430 PMLR, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodee Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL lnttps:/trranffome---irrr (0033//momseemmati1ffffttrss/rre html
pub/
Christopher Burgess, Irina Higgins, Arka Pal, Loie Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. Understanding disentangling in 3-VAE, 2017. http::tss://arxii
brg//bs/180/ 03599
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Ieo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. Computing Research Repository, arXiv:2312.09399. 2023. URL
09390
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire. Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng. Phillip Christoffersen. Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Madfield-Menell Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions

---

Published as a conference paper at ICLR 2025
Machine Learning Research, 2023. ISSN 2835-8856. URL httt:ss//oppprreeeeeweeeettoooo id=bx24KpJ4Eb Survey Certification.
Angelica Chen, Ravid Shwartz-Ziv, Kyunghyun Cho, Matthew Leavitt, and Naomi Saphra. Sud- den drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Tweifth International Conference on Learning Representations, 2024. URL https:
//openreview. V/ggenreview..ett/orunnn..oomo
Tian Qi Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle-
ment in variational autoencoders, 2018.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. In- fogan: interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 2180-2188, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Computing Research Repository, arXiv:1706.03741) 2023.
Arthur Commy, Augustine N. Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Aoosso Towards automated circuit discovery for mechanistic interpretability. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
Elliot Creager, Joern-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2189- URL https: httpss///proceeddngg.. http://occeeennn.....rrrres////e wttpppeee///33//eeeeeee//e/
2200. PMLR, 18-24 Jul 2021.
html
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse au- toencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL Ihtt::: /ttps///ee//eeeeie../eeoee. www.:/erfoofmmmieecoomm
F76buRSLeK
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios:
A case study of semantic representation bias in high-stakes setting. In Proceedings of the Conference on Fairness, Accountaillity and Transparency, FAT '19, pp. 120-128, New York, NY. USA, 2019. Association for Computing Machinery. ISBN 9781450361255. doi: 10.1145/ 3287560.3287572 URL /ttps::::///
1141/./.1511/325551 3287572
Guillaume Desjardins, Aaron Courville, and Yoshua Bengio. Disentangling factors of variation via generative entangling. Computing Research Repository, arXiv: 2012.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac HatfelldDoddds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei. Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https:/ttansffrmeer- wwtppp/////ooooomeeee/eoeee html.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th An- mual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1828-1843, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 accl-long. 144. URL htt.hhsss//aaaaathhlleeeoo wwwpentoooooooooooooooeeoooee www.:oggg......./eoo -ong-loon.11111 144
Alexxi Yossi Gandelsman, Alexei A. Efros, and Jacob Steinhardt. Interpreting CLIP's image representation
via text-based decomposition. Computing Research Repository, arXiv:2310.0591 2024.
12

---

Published as a conference paper at ICLR 2025
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. Computing Research Repository, arXiv:2101.0000 2020.
Lco Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. Computing Research Repository, (xXi::2406.0409: 2024.
Atticus Geiger, Hanson Lu, Thomas Ieard, and Christopher Potts. Causal abstractions of neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 9574-9586. Curran Associates, Inc., 2021. URL https::/ htw.:/dddnnnnneuuurs...o///paaaa wwt.ee/ap//peee//////// ffle:4ffs222u0558887727722222100
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Good- man, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 7324-7338, PMLR, 17-23 Jul 2022. URL https://proceeding wttpes.mm..pprrr.. waa..prres//v162/eigrrr2
html
Atticus Geiger, Chris Potts, and Thomas Icard. Causal abstraction for faithful model interpretation. Computing Research Repository, arXiv:2301.0470 2023.
Mor Geva, Jasmijn Bastings,
Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in rugo-regressvve language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12216-12235, Singapore, December 2023. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/2023 000653//1/222..mmmlppp emnlp-main.775 main. 751. URL https://acllnnthoooey/ hrwp://3..rr//2/3..../ emmlp-main.75 751
Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy. Successor heads: Recurring, inter- pretable attention heads in the wild. Computing Research Repository, arXiv:2312.09230, 2023.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in pre-trained language model. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL ftttss///pperreviee...eeffooor 14PckNNRRKk
Michael Hanna, Sandro Pezzelle, and Yonatan Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In ICML 2024 Workshop Mechanistic Inter- pretability, 2024. URL htttps://openrevieeee wtwpreewwwwweweeeeeeerrrereeeree net/forum? drrr Xge
Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 7002-7024, Bangkok, Thailand, August 2024. Association for Computa- tional Linguistics. doi: 10. 2..188663v111 118635/1/22224// 22222/ URI.
1:
.rrg
2024.ac1- acl--long.... 378
T. He, 7. Li, Y. Gong, Y. Yao, X. Nie, and Y. Yin. Exploring linear feature disentanglemment for neural networks. In 2022 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, Los Alamitos, CA, USA, jul 2022. IEEE Computer Society. doi: 10. 1011010MME222222222222222 200. URL https:://oo..ieeeeeeeeoo ieeecomputer society. 10. 1109/ICME52920. 9859978
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-VAK Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL
13

---

Published as a conference paper at ICLR 2025
Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group- aaaat-ooouuuurrryy In Bernhard Schölkopf, Caroline Uhler, and Kun Zhang (eds.). Proceedings of the First Conference on Causal Learning and Reasoning, volume 177 of Proceedings of Machine Learning Research, pp. 336-351. PMLR, 11-13 Apr 2022. URL htttlhttts://pocceediiee wwainngssm.r....m www...pees/w////idrisstsssss htmml
Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. Shielded representations: Protecting sensitive attributes through iterative gradient-basee projection. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguisties: ACI. 2023, pp. 5961-5977, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10.18653/v11200 hinnas/iin22.....nnnnnaaaaaa findings-acl.369) URL https://aclanthology wrg./gy..rg///3333/ 2tn.///2.......nnn.../.. findings-acl. 369
Shadi Iskander, Kira Radinsky, and Yonatan Belinkov. Leveraging prototypical representations for mitigating social bias without demographic information. Computing Research Repository, 2403.09516, 2024.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Proceedings of the 35th International Conference on Machine Learning, pp. 2668- 2677. PMLR, 2018.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2649-2658. PMLR, 10-15 Jul 2018. URL https://proceediing www.mmrrr.sssse.. wwt.pprrs//v//kimme60 htmll
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. abs/1442.6990, 2014. HtRpsttss::/// //ap:s:///ppii www.nticscholar.......oo hww.:/ar.rrrr//ooosSi:::2/:
CoRR,
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is suffi-
cient for robustness to spurious correlations. Computing Research Repository, arXiv:2204.02937, 2023.
János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. AIP*: An efficient and scalable method for localizing I1m behaviour to components. Computing Research Repository, arXiv:2403000745. 2024.
David K. Lewis. Counterfactuals. Blackwell, Malden, Mass., 1973.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024.
Johnny Lin and Joseph Bloom. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. URL https://www.neuuonpedi.. org Software available from neuronpe- dia.org.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp.
6781-6792 PMLR. 18-24 Jul 2021. html
URL httth::: httpss://proceeddigg.. mlr wtt...prres//l3//iiuu/e
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International
Conference on Learning Representations, 2017. URL https://api. www.sesssccccccoaarooooom
CorpusID::3592270)
Alireza Makhzani and Brendan Frey. k-sparse autoencoders. Computing Research Repository, abs/1312.5663, 2013. URL https:://app www.enniisccollr.rrggggo//ooo/ee
11NDD:::4488777 14850799
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associ- ations in GPT. Advances in Neural Information Processing Systems, 36, 2022. 25X2v:2202.0526..
14

---

Published as a conference paper at ICLR 2025
Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanan, Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: A history, survey, and theoretical grounding URL httphtts//axxii.rrrgggg
of causal interpretability, 2024.
2408. 01416
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978
Junhyun Nam, Jaehyung Kim, Jaeho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. 2022.
Neel Nanda. Attribution patching: Activation patching at industrial scale, 2022. URL https:
//www.neelnanda.c hww.://wwwwn....oocooottttttttte
patching
Neel Nanda. Open source replication & commentary on Anthropic's dictionary leam-
ing paper, 2023.
URL https: // //ww://wwwwwww http://nw...oopoosss////uuux/u/e
open- wwwr-esuuueeeeeeee replicaiitoi and commenarry-on -ntt--ooioonntttt------
Neel Nanda, Senthooran Rajamanoharan, János Kramár, and Rohin Shah. Fact
finding: Attempting to 2023.
reverse-engineer factual recall on the neuron
level,
URL https://www.lligmmentttti
fact
Richard Ngo, Lawrence Chan, and Sören Mindermann. The aligmment problem from a deep learning perspective. Computing Research Repository, arXiv:2209.0000 000000,,........ 2024.
alignment-problem
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Wei Weng. Label-free concept bottle- neck models. In The Eleventh International Conference on Learning Representations, 2023, URL
https://openreview www.nneeooooomdoIoeooeee
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph. Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodss, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. wewpriinnnnnnn-nnnnnnn-----eeeee
Yonatan Oren, Shiori Sagawa, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust language modeling. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Pro- ceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp- 4227 4237, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 11118653/11DDD1... 1/D19-1431
URL htttts::/aaaaattoollllo/ooo
Hadas Orgad and Yonatan Belinkov. BLIND: Bias removal with no demographics. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8801-8821, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023..0 .a---33acl-oooooee 490. URL htttts://aalanttoll.oo
or org 2023. acl 1ooe-loogg-44 490
Judea Pearl. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncer- tainty in Artificial Intelligence, UAI'01, pp. 411-420, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
F. Pedregosa, G. Varoquaux, A. Gramfort,
V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leamn Machine learning in Python. Journal of Machine Learning Research, 12:2825-2838 2011.
15

---

Published as a conference paper at ICLR 2025
William Peebles, John Peebles, Jun-Yan Zhu, Alexei A. Efros, and Antonio Torralba. The hessian penalty: A weak prior for unsupervised disentanglement. In Proceedings of European Conference on Computer Vision (ECCV), 2020.
Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, and David Bau. Fine-tuning enhances existing mechanisms: A case study on entity tracking. In Proceedings of the 2024 International Conference on Learning Representations, 2024. arXiv:2402.14811
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieherum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. Computing Research Repository, arXiv:2404. 16014, 2024a.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. Computing Research Repository, arXiv:2407.14435, 2024b. URL https:
//arxiv.org/abs//2 2440//77./442.....// 14435
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guard- ing protected attributes by iterative nullspace projection. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7237-7256, Online, July 2020. Association for Computational URL https::/acllathol.ogg http:/thowooooooooo///eeeooee org/ 2220.
Linguistics. doi: 10. 10.18653/v12022..cc acl.000.cl-mmm
main.647. http://00..cmmmmi.66444141.
acl-main. 647
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan Cotterell. Linear adversarial concept erasure. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18840-184211 PMLR, 17-23 Jul 2022a. URL
Shauli Ravfogel, Francisco Vargas, Yoav Goldberg, and Ryan Cotterell. Adversarial concept era- sure in kernel space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6034 6055, Abu Dhabi, United Arab Emirates, December 2022b, Association for Computational Lin- guistics. doi: 10. 10.18653/v1// 0866:///2222..mmlpmmm....k
anthology hwwphhllgg.orgg///// org/2022. URL https: https:///allatthllooy
manlp-main... 465
James M. Robins and Sander Greenland. Identifiability and exchangeability for direct and indirect effects. Epidemiology, 3(2):143-155, 1992. ISSN 10443983. URL http::/www.jsttt org/
stable/3702894
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https:
Sopperreview.net/formme
Jürgen Schmidhuber. Learning Factorial Codes by Predictability Minimization. Neural Com- URI
putation, 4(6):863-879) 879, 11 1992. ISSN 0899-7667. doi: (ttp////ccc.99....... hctps::/doi.org// 612/ncc.. 1992. 4.68663
Johannes Schneider and Michalis Vlachos. Explaining neural networks by decoding layer ac- tivations. In Advances in Intelligent Data Analysis XIX: 19th International Symposium on Intelligent Data Analysis, IDA 2021, Porto, Portugal, April 26-28, 2021, Proceedings, pp- 63-75, Berlin, Heidelberg, 2021. Springer- Verlag. ISBN 978-3-030-74250-8 doi: 10.1007/ 978-3-030-74251-5.. URL httphtttt///ii....// www.rrgg////// 1111111.110000001 0077997----330-4225
Nimit Sharad Sohoni, Maziar Sanjabi, Nicolas Ballas, Aditya Grover, Shaoliang Nie, Hamed Firooz, and Christopher Re. BARACK: Partially supervised group robustness with guaran- tecs. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022. URL https://openrevieew httpe//rriiieeene/ooumuiitte met/fforumiic-ns300k3k/
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 3319-3328. JM... 2017.
16

---

Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing NLU models from unknown biases. Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7597-7610, Online, November 2020. Association for Computational Linguistics. doi: 10. 1::5://112220..emmpmmmiiin Published as a conference paper at ICLR 2025
Aaquib Syed, Can Rager, and Arthur Commy. Attribution patching outperforms automated circuit discovery. In NeurlPS Workshop on Attributing Model Behavior at Scale, 2023. URL https:
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Fer- ret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Char- line Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard. Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchi- son, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry. Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Wein- berger, Dimple Vijaykumar, Dominika Rogoziñska, Dustin Herbison. Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucinska. Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kar- tikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidow, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moyni- han, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla. Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culli- ton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei. Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. Gemma 2: Improving open language models at a practical size, 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman- ticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. html
URL
circi ts. pwwpp/24/ http://///ccaaiiinoooomonnnnnttt
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In Proceedings of the 2024 International Conference on Learning Representations, 2024.
URL
org/ www.///.......oooo 2020 emnlp-main 613
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 12388-12228 12401. Curran Asso-
17

---

Published as a conference paper at ICLR 2025
ciates, Inc., 2020. URL 2255bee2211755ee3222f22222a22622
Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL /tthhtts:://///// https::/oppereeiieee forum/1d=:NsVVNNoouul
Tianlu Wang, Xi Victoria Lin, Nazneen Fatema Rajani, Bryan McCann, Vicente Ordonez, and Caim- ing Xiong. Double-hard debias: Tailoring word embeddings for gender bias mitigation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 1444-553,, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/w1/2020 awl-aacclm-eeeemm URL
httpss//aclanttolory..oom /200/ 2ttp//222.....cm.. 484
Yadollah Yaghoobzadeh, Soroush Mehri, Remi Tachet des Combes, T. J. Hazen, and Alessandro Sordoni. Increasing robustness to spurious correlations using forgettable examples. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 3319- 3332, Online, April 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2011. eacl-main.291. URL https::/acllanthooogg... www.thhhooogo..o//22211111 www.lmaac..ee...
An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amil- care Gentili, Chun-Nan Hsu, Jingbo Shang, and Julian McAuley. Robust and interpretable medical image elassifiers via concept bottleneck models. Computing Research Repository, arXiv:2310.03188 2023.
Qinan Yu, Jack Merullo, and Ellie Pavlick. Characterizing mechanisms for factual recall in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pp.
Singapore, Decem- ber 2023. Association for Computational Linguisties. doi: 10. N0118665///22233mmmmmmm..... URL
hwwpohooog.ooooo////// orr.orgg////// 2023. wttp://0.....mmmmmmaaaa..... emn.--mminn.61 615.
John Zech, Mareus A. Badgeley, Manway Liu, Anthony B. Costa, Joseph Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLOS Medicine, 15(11):e10026)) November 2018.
ISSN 1679-1676. doi: 10joral/j mm0...000.d........... 1002683. URL //dx: doi wwwp:/ooo..ogg..... 1371/
journal. .meen.all.mmd....mm 1002683
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. In International
Conference on Learning Representations, 2021. URL http://ppenreeiiee.eeeerrrrrrmme
BtZhsSGNRN1]
Michael Zhang, Nimit S Sohoni, Hongyang Zhang, Chelsea Finn, and Christopher Re. Correct- N-Contrast: A contrastive approach for improving robustness to spurious correlations. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 26484-26516 PMLR, 17-23 Jul 2022. URL https://proceedin:
mlr www../...rreee//1///aaaa222222/ html
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engincering: A top-down approach to AI transparency. Computing Research Repository, arXiv:2310.01405, 2023.
A METHODOLOGICAL DETAILS FOR FEATURE CIRCUIT DISCOVERY
A.1 COMPUTING EDGE WEIGHTS
Let e be an edge between an upstream node u and downstream node d: let also M be the set of nodes intermediate between and d. We define the weight of the edge be the effect on the
18

---

Published as a conference paper at ICLR 2025
Templatic data: No position aggregation
Non-templatic data: Sum aggregation
6,m
L.
11
1,4:
I.
Sum
The
doctor
It's
strange
sum
1,4 children
The
Doctor
Johnson
mean
mean
01
L.
L.b
Figure 6: Aggregation of node/edge effects across examples (and sometimes, across token posi- tions). Each feature is labeled as "token position, feature index." If we have templatic data, we preserve token position information, and treat the same features in different token positions dif- ferent features. If we have more general non-templatic data, we first sum across positions, and then take the example-wise mean of the position-aggregateed effects.
metric m when intervening
d (Felean do
Upateh: Melean E M)).
Intuitively, this captures the indirect effect on via the direct effect but excluding effects on mediated by some other intermediate node m.
As with nodes, employ a linear approximation:
Im(m;; E; Felean Tpanch) = Vamldan V (t)
(Upatch
Welean
(5)
Magenn
where V st,ttop (M) d denotes the gradient of with respect to u when treating all m E .M as constant. In practice, this is computed by applying stop-gru nop-graddents to all intermediate nodes m during pytorch's backwards pass.
If d is SAE error, then the naive approach to computing is expression involves performing dmmodel backwards passes; fortunately we can still compute the product in a single backwards pass as ex- plained in A.3
A.2 AGGREGATING ACROSS TOKEN POSITIONS AND EXAMPLES
Figure summarizes how we aggregate effects across examples (and optionally across token posi- tions). For templatic data where tokens in matching positions play consistent roles (see $3.2 3.3 we take the mean effect of nodes/edges across examples. In this case, we treat the same feature (or neuron) different token positions different nodes altogether in the circuit, each with their own separate effects on target metric m.
For non-templatic data 5p, we first sum the effects of corresponding nodes/edges across token positions before taking the example- wise mean. This means that each feature appears in the circuit once, representing its effects all token positions in an input.
A.3 PRACTICAL CONSIDERATIONS
Here review number of tricks that use compute the quantities defined above efficiently. The backbone of our approach is to, given an activation x E Rddm of some submodule for which have SAE, use the SAE to compute the quantities fi(x) and c(x) in (Ip. and then intervene in our model's 's forward pass to set
X Σ f((x)v, + E(x).
(6)
Even though x was already numerically equal to the right-hand side of (6). after the intervention the computation graph will incorporate the variables fx(x and e(x). Thus, when Pytorch's
19

---

Published as a conference paper at ICLR 2025
autograd algorithm to peform backpropogation of downstream quantities, will automatically compute gradients for these variables.
An alternative approach for computing gradients (which do not use) is simply the model without interventions, use backpropogation to compute all gradients m, and use the formulas
y:m- X Va:
m V m
which follow from the chain rule when m is any function of x.
Stop gradients on SAE errors to compute SAE feature gradients. The natural way to compute the SAE error Exx) is by first using the SAE to compute x and then setting E(x) = &. However, we take this approach, then after applying the intervention (6) we would have
m
V x"
x V (& (x+*"->
where x" the copy of x downstream of fi in the computation graph, and x" is the copy upstream of ft. To fix this, apply stop gradient E(x) so that x" x + stopgrad x).
Pass-through gradients. Although the stop gradient from above solves the problem of vanishing gradients for the fú, it interferes with the backpropogation of gradients further upstream nodes. In order to restore exact gradient computation, we implement a pass-through gradient on the com- putation of dictionary. That is, in the notation above, we intervene in the backwards pass of our model to set
Vx x
V m
Jacobian-vector products. Done naively, computing the quantity in (5) when d is SAE errors would take (dmodel) backwards passes. Fortunately, one can use the following trick: when A is constant 1 matrix, E Rm and y y(x))= R function of x, have
AV = Vx(y))
where the right-hand side is 1 X m Jacobian which can computed with a single backward pass. Thus compute 5 with only two backwards passes by first computing Vam d and then computing V. (Vaml d with another backwards pass, where the second Vam a treated as constant (e.g., by detaching it in Pytorch).
B DETAILS ON SPARSE AUTOENCODERS
B. PYTИИAA-0M SPARSE AUTOENCODERS
B.1.1 ARCHITECTURE
Following Bricken et al. (002), our SAES for Pythia-70M are one-layer MLPs with tied pre- encoder bias. In more detail, our SAEs have parameters
WE E IR Xdantal WD E pdlnaa X dssE
bg E dS bp E R
where the columns of WD constrained to be unit vectors. Given input activation R we compute the sparse features activations via
[f (x)
SdssE (x)] ReLU(Wg(x bp) bg)
with the ReLU nonlinearity applied coordinatewise and reconstructions via
= WDf + bp.
The feature vectors V, E R duedd are the columns of Wp-
20

---

Published as a conference paper at ICLR 2025
B.1.2 TRAINING
Fix specific choice of activation in Pythia-70M, e.g. MLP output, attention output, residual stream in a particular layer. Following Cunningham et al. (2024); Bricken et al. (203)) we train an SAE for this activation by sampling random text from The Pile (Gao al. 2020) (specifically the first 128 tokens of random documents), extracting the values x for this activation over every token, and then training our SAE to minimize a loss function
C Cessonstruction + ACsparsiyy || x + A|[f||1
consisting of L2 reconstruction loss and a L1 regularization term to promote sparsity. This loss is optimized using variant of Adam (Kingma & Ba 2014) adapted ensure that the columns of WD are unit vectors (see Bricken et al. (2023) or our code for details). We use À 0.1 and a learning rate of 10 10-
Following Nanda (2023), we cache activations from 10000 contexts in a buffer and randomly sample batches of size? 2 for training our SAE. When the buffer half-depleted, we replenish it with fresh tokens from The Pile. We train for 120000 steps, resulting in a total of about 2 billion training tokens.
A major obstacle in training SAEs is dead features, that is, neurons in the middle layer of the SAE which never or rarely activate. We mitigate this by, every 25000 training steps, reinitializing fea- tures which have not activated in the previous 12500 steps using the same reinitialization procedure described in Bricken et al. (2023).
Finally, we use a linear learning rate warmup of 1000 steps the start of training and after every time that neurons are resampled.
B.1.3 EVALUATION
Here we report on various uusytto-uunntify metrics of SAE quality. Note that these metrics leave out important qualitative properties of these SAFs, such as the interpretability of their features (App. Our metrics are:
Variance explained, measured by 1
Var(x Var(x)
Average L1, and LO norms of f.
Percentage of features alive as measured by features which activate at least once on batch of 512 tokens.
Cross entropy (CE) difference and percentage of CE recovered. The CE difference is the difference between the model's original CE loss and the model's CE loss when inter- vening to set x to the reconstruction &. We obtain percentage of CE recovered by dividing this difference by the difference between the original CE loss and the CE loss when zero- ablating These CE losses are computed averaged over a batch of 128 contexts of length 128.
These metries shown in Tables 3 layer which outputs the activation (so the layer 0 residual stream is not the embeddings, and the layer residual stream the output of the final layer, immediately preceding the final decoder).
Note that index residual stream activations to be the
Variance Explained
L1
L.0
Alive
CE Diff
% CE Recovered
96
3
36
0.17
98
Table 3: Embedding SAE evaluation.
B.2 GEMMA-2-2B SPARSE AUTOENCODERS
For Gemma-2.4 2-2m, based on the Jump-ReLU architecture proposed by [Kajamano haran et al. (2024b). We use the SAEs of width 16384. There exist SAEs for the attention, ML.,, and residual vectors for each of the 26
the Gemma Scope SAEs released by Lieberum et al. (2024). which
21

---

Published as a conference paper at ICLR 2025
Layer
% Variance Explained
L1
I.0
% Alive
CE Diff
CE Recovered
Attn
92%
8
128
17%
0.02
99%
Atn
87%
9
127
17%
0.03
94%
Attn
90%
19
215
12%
0.05
93%
Attn
89%
12
169
13%
0.03
93%
Attn
83%
8
132
14%
0,01
95%
Attn
89%
11
144
20%
0.02
93%
Table 4: Attention SAE evaluation by layer.
Layer
% Variance Explained
L11
L.0
Alive
CE Diff
% CE Recovered
MLP 0
97%
5
40%
0.10
99%
MLP 1
85%
69
44%
0.06
95%
MLP 2
99%
12
88
31%
0.11
88%
MLP 3
88%
20
160
25%
0.12
94%
MLP4 4
92%
20
100
29%
0.14
90%
MLP
96%
31
102
35%
0.15
97%
Table 5: MLP SAE evaluation by layer.
layers in the model. However, the attention and MLP SAEs are trained different positions than in Pythia: attention SAEs are trained on the input to the out projection, and MLP SAES are trained on the output of the LayerNorm following the MI.P The embedding SAFs are experimental and have a dictionary size of only 4000, so we do not use them in our experiments.
There exist multiple SAEs for every submodule. The primary difference between them is their average LO norm Neuronpedia (Lin & Bloom 2023) uses the SAEs with the LO norm closest to 100; we do the same.
One of the primary technical challenges in using the Gemma Scope SAEs is the existence of BOS features. These are features that are active primarily or only on BOS tokens, and whose top logits generally not informative. These features difficult to interpret, but can have high indirect effects on the model's logits. As we cannot interpret them, we exclude them from annotation and from the SHIFT analysis (i.e., do not ablate them). We also exclude them when running the feature skyline in $4
C FEATURE CIRCUITS
C.I SUBJECT- VERB AGREEMENT
Here, we present the full agreement circuits for various syntactic agreement structures, with researcher-provided annotations for features. We chose thresholds manually in order to keep the number of nodes annotate manageable while still displaying the full range of feature types for given task.
In each eircuit, sparse features are shown in rectangles, whereas causally relevant error terms not yet captured by our SAEs are shown in triangles. Nodes shaded in darker colors have stronger effects on the target metric m. Blue nodes and edges are those which have positive indirect effects (i.e., useful for performing the task correctly), whereas red nodes and edges those which have counterproductive effects on m (i.e., cause the model to consistently predict incorrect answers).
First, we present agreement across relative clause. Pythia (Figure 7 and Gemma (Figure both appear to detect the subject's grammatical number at the subject position. One position later, features detect the presence of relative pronouns (the start of the distractor clause). Finally, at the last token of the relative clause, the attention moves the subject information to the last position, where assists in predicting the correct verb inflection. Gemma additionally leverages noun phrase (NP) number
In other words, the average number of features active for given token.
22

---

Published as a conference paper at ICLR 2025
Layer
% Variance Explained
L1
% Alive
CE Diff
% CE Recovered
Resid
92%
11
59
41%
0.24
97%
Resid
85%
13
54
38%
0.45
95%
Resid
96%
24
108
27%
0.55
94%
Resid
96%
23
68
22%
0.58
95%
Resid
88%
23
61
27%
0,48
95%
Resid
90%
35
72
45%
0.55
92%
Table 6: Residual (Resid) SAE evaluation by layer.
The
qirls
Chat
Dussistants
bate
Figure 7: The feature circuit for agreement across relative clause in Pythia-70M, computed using Ty 0.1 and TÆ 0.01. The model detects the subject's number at the subject position. Other features detect relative pronouns (the start of the distractor clause). Finally, the last token of the RC, the attention moves the subject information to the last position, where it assists in predicting the correct verb inflection.
tracking features, which active all positions for NPs of given number (except distractor phrases of opposite number). We present example of an NP number tracker feature in Figure 15
The circuits for agreement across a prepositional phrase (Figures 9 and I0] look remarkably similar to agreement across a relative clause; for both Pythia and Gemma, these two circuits share over 85% of their features, and many of the same features are used for detecting both prepositions and relative clauses.
For simple agreement (Figures and 12). many of the same features that were implicated in noun number detection and verb number prediction in the previous circuits also appear here. The mod- subject position in early layers. In later layers, these noun
els detect the subject's number number detectors become inputs to verb number promoters, which activate on anything predictive of particular verb inflections.
The circuits for agreement within relative clause (Figures 13 and 14 appear to have the same structure as that for simple agreement: subject number detectors in early layers, followed by verb number promoters in later layers.
C.2 BIAS IN BIOS CIRCUIT
Here, we present the full annotated circuit discovered for the Bias in Bios classifier trained on Pythia- 70M (described in 4 and App. B The circuit was discovered using Tv = 0.1 and TE 0.01. We observe that the circuit (Figure 16 contains many nodes which simply detect the presence of gendered pronouns gendered names. A few features attend to profession information, including one which activates on words related to nursing, and another which activates on passages relating to science and academia.
23

---

Published as a conference paper at ICLR 2025
Figure 8: The feature circuit for agreement across relative clause in Gemma-2-20 computed using
Ty = 0.073 and TE = 0.007.
24

---

Published as a conference paper at ICLR 2025
Par far
4s1
athlete
beside
the
cats
Figure 9: The feature circuit for agreement across prepositional phrase in Pythia-70M, computed using I' = 0.1 and TE = 0.01. The model detects the subject's number at the subject position. Other features detect prepositional phrases (the start of the distractor clause). Finally, the last token of the RC, the attention moves the subject information to the last position, where it assists in predicting the correct verb inflection.
awnt
dogs
Figure 10: The feature circuit for agreement across a prepositional phrase in Gemma-2-2B, com- puted using Tn 0.5 and TE 0.05. Note that we show the circuit beginning in layer 13, circuit discovery implicated only one node earlier layers.
25

---

Published as a conference paper at ICLR 2025
prozoess perrae arorea form,
23
Frreent LengEE Frresii TESTE
Akri CALAR
rip 2
The
boys
Figure 11: The feature circuit for simple agreement in Pythia-70M, computed using T' = 0.2 and Tg = 0.02. The model detects the subject's number the subject position in early layers. In later layers, these are inputs to features which activate on anything predictive of particular verb inflections.
C.3 CLUSTER CIRCUITS
Here, we present full annotated circuits discovered for automatically discovered behaviors (de- scribed in App. G). First, we present the circuit for incrementing number sequences (Figure [7). discovered with Nv = 0.4 and Tp = 0.04. We note that this circuit includes many features which perform either succession (Gould al. 2023) induction (Olsson al., (022). The succession features in the layer 3 attention seem to be general; they increment many different numbers and letters (as in Figure 5). The induction features are sensitive only to specific tokens: for example, contexts of the form "r3... x3",, where "3" is literal. These compose to form specific successor features in layer the most strongly-activating layer 5 residual feature specifically increments "3" to "4" given induction-like lists, where each list item is preceded by the same string (e.g., "Chapter 1... 1... Chapter 2... Chapter 3... "hapterr"""
The circuit for predicting infinitival objects (Figure 18, discovered with Tv = 0.25 and T5 = 0.001) contains two distinct mechanisms. First, the model detects the presence of specific verbs like "re- member" "require" which often take infinitival objects. Then, the model uses two separate mech- anisms to predict infinitive objects. The first mechanism detects present-tense verbs, participles, predicate adjectives which can be immediately followed by infinitival direct objects (e.g.. "They were excited to..."). "). The second mechanism detects nominal direct objects that can directly pre- cede infinitival object complements (e.g., "They asked to..."). Finally, these two mechanisms both influence the output layer 5 without fully intersecting.
D SAMPLE FEATURES
D.1 SPARSE FEATURES
Here, we present examples sparse features with high indirect effects on the Bias in Bios task. Some of these features clearly activate terms related to medicine academia, which related to the target profession classification task. Others simply detect the presence of "he" or female names.
26

---

Published as a conference paper at ICLR 2025
mat
THE
meat
Se ontel FRES
Fedd erreli
11
matt
Pard pellatte
mak
ein
matis
Mand
BOS
The
boys
Figure 12:
The feature circuit for simple agreement in (emma-2-2B, computed using TN = 0.5 and Tg 0.05.
27

---

Published as a conference paper at ICLR 2025
forze
pnodats TEIERESI
planad valjects
Saleuste
Bnd NI
2
Miza SELIH
mipp2
olp
mip
piceel
kninate plani oeers
stor
Theloctorshat the
parents
Figure 13: The feature circuit for agreement within a relative clause in Pythia-7OM, computed with
Tv 0.2 and Tr 0.02. The model detects the subject's number at the subject (within the RC)'s position in early layers. later layers, these features are inputs to features which activate on anything predictive of particular verb inflections.
D.2 NEURONS
For contrast, we also present examples of dense features -that is, neurons from MLPs, layer-end residuals, and the out-projection of the attention-with high indirect effects on the Bias in Bios task. We cannot directly interpret the activation patterns of these neurons, and so difficult to run the SHIFT with neurons baseline. We therefore instead compare to the neuron skyline, where we allow the skyline an unfair advantage by simply ablating neurons which have positive effeets on gender-based probabilities given the balanced set.
E IMPLEMENTATION DETAILS FOR CLASSIFIER EXPERIMENTS
E. CLASSIFIER TRAINING
Here we describe how we train linear classification heads on Pythia-70M and Gemma-2-2B the Bias Bios (BiB) task of $4
Given model M and choice E of layer, we mean-pool over (non noon-padingg) tokens all layer & residual stream activations from M; we then train a linear classification head via logistic regression, using the AdamW optimizer (Loshchilov & Hutter (017) and learning rate 0.01 for one epoch on this dataset of activations. The activations and labels for this logistic regression are collected from the ambiguous set for the bascline classifier and from the balanced set for the oracle classifier.
To mimick a realistic application setting, we tune the choice of layer for the baseline probe's accuracy on (a test split of) the ambiguous set. For Pythia, this recommends using the penultimate layer = 4. For Gemma, there is a wide range equally performant layers. Thus for this one choice only only-ws make use of the balanced set compute how the baseline probe generalizes; select the layer & 22 for which the baseline probe generalizes worst. We make this choice to up testhed where there is the most space for improvement. We emphasize that
tune
28

---

Published as a conference paper at ICLR 2025
BOS
The
doctors
thst
Figure 14: The feature circuit for agreement within a relative clause for Gemma-2-2B, computed with Tv 0.5 and TF 0.05.
covered with black bars. Monn he used the without permission "to illustrate this is of shouldn't show"". Narn clained that after Virginia article, she started teuching herself aheas that were blacked out, saying, "weat's wrong with me?" Monn responded to the crsticisss saying she did not plan the photographi
17.45
children sinitarly.swary photographs containing her nude or hurt children caused controversyj For exarple, in The Perfect Tomato, the viewer sees mude Jessie, posing
young,
(othing contained in afficavits indicates jury considered any extraneous
statements affidevits relate to evidence that was presented at trial or to inforastion that was othervise brought to the attention of Jury during regarding the definition of "standard of care," or regarding any other ratter, Nothing in either of the affidavits outside source. The trial court did not abuse its
affidevits
evidence
jury
information
indicates that the jury, or particular juror, influanced seeking discovery regarding jury's deliberations
in denyinng the plaineirts" posttrial
jelaventujuri
agreesent with percolation thresholds given in Table V
23.75 "tarplate typolefs
declare
Figure 15: An example sparse feature for agreement across a relative clause in Gemma (resid 12/13561). This feature activates tokens in noun phrases where the noun head is plural, but not on singular distractor phrases within the plural NP. This feature carries the number of the subject across positions, we term it "NP number tracker".
29

---

Published as a conference paper at ICLR 2025
Figure 16:
The full annotated feature circuit for the Bias in Bios classifier. Many nodes simply detect the presence of gendered pronouns or gendered names. A few features attend to profession information, including one which activates on words related to nursing, and another which activates passages relating science and academia.
of ofneuiios
In
Bn syp IY
mip 5
5
atto
SEdOCIER SECKCI
Induction: Bsound In 2ye
betoee
betrea
mlp_3
atto 2
II und moltigis
1
R14074
0121055 005
0
saled
Figure 17: The full annotated feature circuit for incrementing number sequences. The model first detects the presence of specific number tokens, like "3". Later, learns more robust semantic representations of those numbers, like "iii" and "Three". Then, the model uses series of narrow and general succesion and induction features increment the next number.
30

---

Published as a conference paper at ICLR 2025
resod
prpmnik my Man
vidte peouee una "in"
attn
resad
adjactrva poadicates
particrpins premating coeposttond
Ligis Iafuro Infrrtiires
Eappics Botte grometiigg praporttos
10.0
otto
2
TIEGO pust partciills, aljectiven
beforn
Tweid
Feeid 1/2815
adiectrons hotorn urepactiions.
atto
readd
Varba and per Lsal
tako Infrliiize objecto
hotoro gempostion:
impphe
m.p.0
post and yuire partictple Gaarhe
Trequiies"
"allomer"
peepoor sourne rotarting
Trequiteed
Figure 18: The full annotated feature circuit for predicting "to" as an infinitival object. The model first detects the presence of verbs that often take infinitival objects. Then, uses one mechanism to detect present-tense verbs, participles, or predicate adjectives which take infinitival objects, and another mechanism detect direct objects that can directly precede infinitival object complements. Finally, these two mechanisms both influence the output in layer 5 without fully intersecting.
Genda Giiboe
Gerda Gilboe (6 Jny 1914 April 2009) Danish actress
singer appeared
between 2000
Bllbo bor 1814 Ehe wed the pouphin Blacksmith, Gliboe started her career in musical theatre and operas as Aarhus before sho Copentagen
theatres nastonal breaktheougg Wwen lacoepted as
Seargttle Saww
Figure 19: An example sparse feature from the Bias in Bios task nettn_3/22029). This feature detects female-related words in biographies of women. It also promotes words like "husband" and "née". This feature probably contributes to preferences for the spurious correlate of gender; we therefore ablate it.
Bregory
Dr. Debble Gregory is national leader in healthcare design, innovation, and transformation. As inrre executive and interior designer, Gregory passionate about "Intentional Design" that aligns People, Place, and Process. She creates and transtorms environments Into functional acosystems using complex systems
stratogic thinking.
Dr. Gregory has Doctor ate Nursing Practice Health Innovation and Leadership from the University Minnesote and bachelors in nursing
Computer learning: the potential teaching assessing nursing
article discusses
learning (CAL) importance applying education. articles recognizes general
technological developments as exemplifed by the Teaching and Learning Technology Programme (TLTP) from which ideas about application and benefits came. The ideas from TLTP are hereby used CAL and applied to nursing
Figure 20:
An example sparse feature from the Bias in Bios task (resid.2/31099). This feature activates on words related to nursing, including "RN" and "nurse". This probably relates to the target task of profession prediction. We therefore keep it.
31

---

Published as a conference paper at ICLR 2025
seendfffetl-<leddooteex->- <jendoftextj> <jendoftexti> denddffeeti--eedofffee-------eee (eeenoofeetl-</eddoffttt (eendolteet----eedooeeeeeeeeeeee (ete://oooeeeeeeeeeeeeetettteeee
<jendoftext]>> <|endoftextjj btttr//eeeeette---eeoeeeeeeeeeee http:///eeetptiieoooooeeeeeeeeee httpr//tpeeeeetoeeeeeeeeedeooooo (lenolteettt (endoftexti-<(eedoottee
<lendoftext]>> httpoo/tteeeeeeoooooeeeeeeeeeoee
Dorsomedial hypothalamic
imbalanced amino acid rats
OS
T122 nanotubes for bone regeneration!
Figure 21: An example neuron from the Bias in Bios task. This appears to activate on beginnings and ends of sentences, but also more strongly on any token in sentence that contains capital letters or numbers. We cannot deduce whether this would contribute more to gender or profession names.
Inorganic phosphate uptake in intact vacuples isolated from suspension- cultturd cells of Cath Sethhaanthus roseus (L)) G. Don under varying Pi
Inorganic phosphate (P)) uptake across the vacuplar membrane intact vacuples isolated from Cath csanthss roseus crspensioo-culturee cells was measured, Under low status, Pi uptake into the vacuglee was strongly activated compared high status. Since Pi uptake across the vacuolar membrane is correlated with H+ pumping,
dependency pumping on plant status. Both
Delhi metro services
Blue line briefly disrupted after jumps front approaching train:
Figure 22: An example neuron from the Bias in Bios task. This activates positively on tokens starting with capital letters, but negatively many other tokens (whose unifying theme cannot deduce).
hyperparameters for the performance balanced set of SHIFT, using the balanced set is forbidden by the problem statement.
When retraining after performing SHIFT, retrain only the linear classification head, not the full model.
E.2 IMPLEMENTATION FOR CONCEPT BOTTLENECK PROBING
Our implementation for Concept Bottleneck Proving (CBP) is adapted from Yan et al. 2023 works follows:
1. First, we collect a number of keywords related to the intended prediction task. We use 20 keywords: nurse, healthcare, hospital, patient, medical, clinic, triage, medication, emergency, surgery, professor, academia, research, university, tenure, faculty, dissertation, sabbatical, publication, and grant.
2. We obtain concept vectors Ci.....C, for each keyword by extracting Pythia Pythia-70M's penul- timate layer representation over the final token of each keyword, and then subtracting off the mean concept vector. (Without this normalization, we found that concept vectors have very high pairwise cosine similarities.)
3. Given an input with representation (obtained via the mean-pooling procedure in App. E. we obtain a concept bottleneck representation z E R by taking the cosine similarity with each C;-
4. Finally, we train a linear probe with logistic regression on the concept bottleneck represen- tations as in App. E..
We decided to normalize concept vectors but not input representations because it resulted in stronger performance. We also experimented with computing cosine similarities before mean pooling.
32

---

Published as a conference paper at ICLR 2025
Activation type
Interpretability
Dense (random)
32.6
Dense (agreement)
30.2
Dense (BiB)
36.0
Sparse (random)
52.8
Sparse (agreement)
62.3
Sparse (BiB)
81.5
Table 7: Human interpretability ratings for dense (neuron) vs. sparse (autoencoder) features. We present mean interpretability scores across features that were either uniformly sampled (random), the top 30 by IE from the subject-verb agreement across RC task (agreement; $3.3). the top 30 by IE for the Bias in Bios task (BiB; $4
0-100 scale. We show scores for features
on pre i expiree help how gresnte coppweers watetre
Teminder how
vorth abou matical eumph, venterae.
Component #138
appreci
Tokans most promoted by this component (maan)
M tut Dosneeit E amd EieE meet ao !
Tokens which moet etiuuate this component (mean)
Full isput paragraphs tokens which most stimulate this cemponennt
and paly unripnddcti kanud caced ALiolpaninnrttt thew that under conan bypetreer then along mitaion bataen probee came cobaee cauaes Dileddrren WCCCC Barach apacia
The parsorr ariid eudy pablen fffiiing dayy boun
sleeeiio to Camplte and lebug den briuuap using segharededl tre burs top pludhs notepaadi ahle and de deber arggere +aimrm beaks Elwes CrippanT bi Aleeawe ciddfaane de camp demii tow Cndirmmee Aartate and infon Compillng Bebrpging woedering benggngggn any son vents turbe
The decorder and Kieeloin Ancindee" bioee you elrrict with Krek Den Sine Klacdi Faniiu cvart axalyic mime procusing BAmazon Ansunal
The human annotation interface used to obtaín the interpretability ratings in Table Here, we show the instructions, top-activating tokens, the token probabilities that most affected when ablating the feature, and example contexts with feature activation values.
Figure 23:
F HUMAN INTERPRETABILITY RATINGS FOR SPARSE FEATURES
Given our trained Pythia-70M sparse autoencoders, we asked human crowdworkers to rate the in- terpretability of random features, random neurons, features from our feature circuits, and neurons from our neuron circuits on a 0-100 scale (Table 7).. Crowdworkers rate sparse features as signifi- cantly more interpretable than neurons, with features that participate in our circuits also being more interpretable than randomly sampled features.
See Figures 23 and 24 for examples of the human annotator interface. Humans were presented with the tokens on which the feature activated most strongly, followed by the tokens whose probabilities most affected in Pythia-70M when the feature example contexts in which the feature activated on some subset of tokens, where feature activations shown in varying shades of blue (darker shades indicate higher activations). On the same page below the contexts, we ask anmotators to write a textual description of the feature, and rate both its interpretability and its semantic complexity 0-100 scales.
ablated. This is followed by series of
Crowdworkers were recruited from the ARENA Slack channel, whose members are machine learn- ing researchers interested in AI alignment and safety. The selection of annotators certainly influ-
33

---

Published as a conference paper at ICLR 2025
Af Swerighh 2015 Erechen, Alight remedd Aigteell greedte theni wtw.geeeeeeeeeeeeeeeeeeeeee ondtorpp wrweeeeerrrreeee.. Perseete
MMbles prrseaol afedgilas NETAAMA
meggnn through wedd
Aherre vental hodk endeyaeee
Component annotation
Cancaa cang pieed stdic samera etaa congovent aeberaaaee aorreadd.
Pleas sellen
Orfigle
Plene select
LeB (divers takm bried contect
Soloral lattee wawherrta the cingoart
Plouenttr trubbit..
mouseot compenertil
Oroom michorpee www.raarrrr..rrrrrrrrtttootoee
The human annotation interface used to obtaín the interpretability ratings in Table 7 Here, show the rating interface the same page the content in Fig. 23 below the example contexts. Humans were asked to write a textual description of each feature, assign a 0-100 inter- pretability rating, and assign 0-100 semantic complexity rating to each feature.
Figure 24:
enced our results; a truly random sample of human annotators would likely display higher variance when annotating features.
One common error pattern we notice is that annotators often label features according to semantic groupings (e.g., "text about politics," and do not pay attention to syntactic context (e.g., "plural nouns"). Future work could address this design bias by testing variants of the instructions.
Results of human evaluations for the Gemma Scope SAEs are described in Lieberum et al. (2024
G DISCOVERING LM BEHAVIORS WITH CLUSTERING
In this section, we describe our unsupervised method for discovering language model behaviors. More specifically, following Michaud et al. (2023), cluster contexts from The Pile according to the Pythia-70MM 70M's internal state during inference. In this section, we describe our clustering pipeline and methods.
G.1 FILTERING TOKENS
We must first locate (context, answer) pairs for which an LM correctly predicts the answer token from the context. We select The Pile (Gao et al. (2020) as a general text corpus and filter to pairs on which Pythia-70M confidently and correctly predicts the answer token, with cross-entropy lower than 0.1 or 0.3 nats, depending on the experiment. The model consistently achieves low loss on tokens which involve "induction" (Olsson et al. 2022) tokens which are part of subsequence which occurred earlier the context. We exclude induction samples by filtering out samples in which the bigram (final context token, answer token) occured earlier the context.
G.2 CACHING MODEL-INTERNAL INFORMATION
We find behaviors by clustering samples according information about the LM's internals when run on that sample. We find clusters of samples where the model employs similar mechanisms for next-token prediction. We experiment with various inputs to the clustering algorithm:
34

---

Published as a conference paper at ICLR 2025
Dense Activations: We take activations (residual stream vectors, attention block outputs, or MLP post-activations) from given context and concatenate them. To obtain a vector whose length is independent of the context length, last N context positions before the answer token, or aggregate (sum) across the sequence dimension. We experiment with both variants.
either the activations at the
the activations of SAE features. We concatenate and aggregate these in the same manner as for dense activa- tions.
Sparse Activations: Rather than dense model activations,
Dense Component Indirect Effects: We approximate the indirect effect of all features on the correct prediction using 2 without a contrastive mair-namely, by setting aptth = 0. The negative tog-probability of the answer token the correct prediction of the next token. The computation of linear effects requires saving both (1) activations and (2) gradients w.r.t at the final N positions for each context in the dataset. We optionally aggregate by summing over all positions.
log Dogp(anseer serves as our metric for
Sparse Indirect Effects: Similarly, we can compute the linear effects sparse activations the correct prediction.
Gradient w.r.f. model parameters: As in Michaud et al. 2023) we also experiment with using gradients of the loss w.r.t. model parameters, but with some modifications. We de- scribe this method in more detail in G.33 below.
G.3 HYPERPARAMETERS AND IMPLEMENTATION DETAILS
We apply either spectral clustering or k-means clustering. For spectral clustering, given ei- ther activations effects v; for sample i, compute matrix of pairwise cosine similarities between all pairs of samples. Before performing spectral clustering,
Ci = I; xy/ / [|:ج||=>> we normalize all elements of C be in [0, 1] by converting the cosine similarities to angular simi- arccos(Ci,))/a
larities: C.2 =
We use the scikit- merki-llean t-aaarn (Pedregosa et al. 2011) spectral clustering implementation with k-means. For all inputs except gradients w.r.t model parameters, we used spectral clustering across 8192 samples. We chose k (the number of total clusters) to maximize the number of clusters implicated in more than one input context.
We also experimented with using gradients w.r.t. model parameters inputs, as in Michaud et al. (2023). Here, we scale up our approach to 100,000 samples. It is intractible to perform spectral clustering given 100,000 samples, so instead use k-means clustering. Rather than clustering the gradients themselves (which are nigh-dimensional), we cluster sparse random projections of the gradients down 30,000 dimensions. When projecting, When sampling the entries of this matrix, sample nonzero value with probability 32/30000, and nonzero, sample -1 1 with equal probability. For sparse projection matrix with dimensions In 30000 there will on average be 32 nonzero entries, where is the number of parameters in the m
matrix with entries {-1, 0,
H QUALITY OF LINEAR APPROXIMATIONS OF INDIRECT EFFECTS
Figure shows the quality of our linear approximations for indirect effects. Prior work (Nanda 2022 Kramár et al. 2024) investigated attribution patching accuracy for IEs of coarse-grained model components (queries, keys, and values for attention heads, residual stream vectors, and MLP out- puts) and MI.P neurons. Working with SAE features and errors, our results echo previous findings: attribution patching generally quite good, but sometimes underestimates the true IEs. Notable ex- ceptions are the layer 0 MLP and the residual stream in early layers. We also find that our integrated gradients- based approximation significantly improves approximation quality.
"We only consider gradients non-embedding and non-layernorm parameters.
35

---

Published as a conference paper at ICLR 2025
(a) Attribution patching
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Residual stream
0.
Attention
6 1244 45
(IE)
Approximate effect
MLP
02 co 4
65
(b) Integrated gradients
feature
Layer 0
Layer
Layer
Layer 3 Layer
Layer 5
Layer
SAE error
Residual stream
Attention
5204
MLP
Exact effect (IE)
Figure 25: Approximate IEs (y-axis) and exact IEs (r-axis) using attribution patching (a; top) integrated gradients (b; bottom). Each point corresponds to an SAE feature or SAE error at one token position of input. Data were collected from 30 inputs from our across RC dataset.
36


Reviews:
[{"review_id": "MCxyAGPmAm", "reviewer": "Reviewer_SioR", "summary": "This paper introduces sparse feature circuits as a causal network built from human-interpretable features, rather than neurons. The author discuss how to quantify the Indirect Effects (IE) of steering or patching individual neurons on model behaviors and propose approximations to extend this approach to models with many components. \n\nThrough experiments on the Gemme2-2B and Pythia-128M models, they demonstrate that a relatively low number of sparse features, compared to neurons, can interpret model behaviors with high fidelity. Building on these findings, the authors present the SHIFT framework, which enables the surgical removal of unintended features to prevent them from affecting model classification performance. While the framework involves some human judgment, it highlights that these Sparse Auto-Encoder (SAE) features are effective representations of concepts that can be manipulated to control model behaviors. Finally, the authors propose an unsupervised circuit discovery pipeline, moving from next-token clustering to zero-ablating features, to identify the most important feature circuits.", "questions": "* Can you add explanations regarding the efficiency of the sparse feature circuit finding with the vast amount of possible feature circuits with respect to neurons'?\n* Can you help better understand how to interpret SAE errors at part of the sparse feature circuits?\n* Can you share your thoughts on extended experiments on more feature ablating studies and how would the results be?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is well-motivated and articulated;\n* The studied subject is of good importance to the NN interpretability domain;\n* The experiment setup is clear and the results are promising;\n* The idea has sound novelty.", "weaknesses": "* SAE features are vast, and identifying specific circuits among those formulated by these features can be challenging.\n* SAE errors cannot be fully interpreted, while they play an important role in making sparse feature circuits. This is ongoing work to train more proficient SAEs, but it should be discussed more in the paper regarding how to perform surgical edits on such errors.\n* I suspect that the circuits may only correlate with certain features. Simply removing gender-related features and observing a drop in classification performance is not universally convincing as a causal intervention approach; additional experiments on other features are needed to support this claim;\n* Trivial concerns: 1) several typos: 283: fair; 201: produce; 492: succession; 2) citation formatting errorsat line 95-96; 3) line 454-457 needs more clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vHSnP2UlPc", "reviewer": "Reviewer_gvvK", "summary": "This paper presents Sparse Feature Circuits, a method to discover interpretable causal graphs within language models (LMs) based on sparse, human-interpretable features. Leveraging sparse autoencoders (SAEs), the authors identify feature circuits that reveal causally implicated components in model behaviors, contrasting with prior work focused on more complex components like attention heads and neurons. The technique allows for targeted interventions to refine model predictions by eliminating task-irrelevant features, as demonstrated in a new method called SHIFT. SHIFT enables selective feature removal, improving classifier generalization without requiring disambiguating labels. The paper’s contributions include a scalable pipeline for automatic discovery of interpretable circuits, and applications in improving model behavior by editing these circuits.", "questions": "1. How do the authors ensure consistency in identifying spurious features across different annotators or applications? Are there any plans to standardize the interpretation criteria?\n\n2. Could the authors clarify how the performance of SAEs affects the interpretability and effectiveness of feature circuits? Have they tested their approach on varying SAE architectures to assess robustness? \n\n3. For large-scale models with complex circuits, what optimizations or techniques do the authors recommend to maintain interpretability while managing the increased number of features?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- By focusing on sparse feature circuits instead of larger, polysemantic components, this approach introduces a new level of granularity in LM interpretability. The use of SAEs is particularly innovative, as it allows for fine-grained, interpretable features that are scalable and can be applied across thousands of discovered behaviors, improving previous interpretability efforts that were limited by reliance on human-defined hypotheses. \n\n- The SHIFT technique, designed to improve model generalization by removing unintended signals, offers a practical, actionable method for handling spurious features. The fact that SHIFT can operate without disambiguating labels and apply human judgment to identify spurious features highlights its practical utility for real-world model refinement, especially in settings where unintended biases are a concern. \n\n- The unsupervised interpretability pipeline for discovering feature circuits across various model behaviors (including grammatical tasks like subject-verb agreement) is impressive. It enables scalable, automatic generation of feature circuits, addressing the labor-intensive nature of previous interpretability work. This unsupervised aspect expands the approach’s applicability across domains and data types, making it a versatile contribution. \n\n- The paper presents quantitative evaluations on interpretability metrics such as faithfulness and completeness for both feature and neuron circuits, providing objective assessments of the interpretability advantage of sparse features.", "weaknesses": "- The method relies on human judgment to determine which features are spurious, which may lead to inconsistencies across applications or datasets. The framework lacks a clear evaluation on the repeatability and consistency of human-selected spurious features in SHIFT, especially when applied by different users or in different contexts. Including a quantitative study or qualitative analysis on the consistency of feature selection across different users or scenarios would strengthen the reliability of SHIFT’s results and enhance confidence in its practical application. For instance, assessing inter-annotator agreement or providing a standardized checklist could help mitigate inconsistencies. \n\n- Although SHIFT leverages human judgment for feature interpretation, this reliance could hinder scalability in complex or high-dimensional models, as manual inspection becomes increasingly challenging. The paper does not provide clear guidelines on how to efficiently scale this process for larger models or more diverse tasks. \n\n- The method’s success is heavily tied to the availability and quality of sparse autoencoders (SAEs). If the SAEs are suboptimal, the interpretability and causal circuit discovery might suffer, particularly if they fail to capture meaningful latent structures. This dependency on pretrained SAE quality raises questions about the method’s consistency and applicability across different model architectures or under domain shifts. \n\n- The faithfulness and completeness metrics for feature circuits are assessed primarily on grammatical tasks like subject-verb agreement, which may not generalize to more complex language tasks that involve nuanced semantic understanding or context-sensitive behaviors. This limited evaluation could affect confidence in the generalizability of the results. Expanding evaluations to include complex, semantically challenging tasks (e.g., multi-turn dialogue or reasoning tasks) would provide stronger evidence of the method’s robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z2oARyfVoI", "reviewer": "Reviewer_xPLM", "summary": "This paper introduces a method for discovering interpretable sparse feature circuits in LLMs using sparse autoencoders (SAEs). While previous mechanistic interpretability work focused on polysemantic neurons, MLP attention heads, this paper shows how to decompose model behaviors into monosemantic, human-interpretable features. The authors validate their approach through three key contributions: (1) a circuit discovery method combining SAEs with linear approximations of indirect effects, (2) a debiasing application called SHIFT that ablates task-irrelevant features, and (3) demonstration of scalable, automated circuit discovery for large sets of model behaviors.", "questions": "- The paper shows strong results with linear approximations of indirect effects. Have you checked when/why these approximations break down? This seems especially relevant for early layers where you note integrated gradients improves accuracy.\n\n- For the automated circuit discovery, how robust are the discovered clusters to different choices of projection dimension and clustering hyperparameters?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Strengths:\n\nSuccessfully addresses a core challenge in MI: finding and scaling interpretable units of analysis. \n\n\n- Empirical Results:\n\nComprehensive subject-verb agreement circuits reveal clear compositional structure (e.g., distinct pathways for handling relative clauses vs. prepositional phrases)\nHuman evaluation shows significantly better feature interpretability (81.5 for BiB circuit features vs 36.0 for neurons)\nSHIFT achieves 93.1% profession accuracy while reducing gender bias to near chance (52.0%)\n\n- Validation & Reproducibility:\n\nThorough ablation studies on all key parameters (SAE width, layer selection, manipulation coefficients). Clear comparison against neuron-based approaches showing consistent advantages.", "weaknesses": "Weaknesses:\n\n- No code and data released yet. \n\n- Theoretical Analysis:\n\nIn terms of theoretical weaknesses, some formal characterization of when SAE features capture meaningful concepts vs memorized patterns would strength the paper. \n\n\nMost of these are noted in the paper but it's worth mentioning it here, SAE training requires significant compute (2 billion tokens) with potential instability (dead features). The current implementation limited to decoder-only models (Pythia-70M and Gemma-2-2B). \n\n- Evaluation Scope:\n\nSubject-verb agreement circuits focus on relatively simple grammatical phenomena and this might not be an issue with scale but no mention of anything more complex in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gCReOm7h5z", "reviewer": "Reviewer_teh3", "summary": "This paper introduces a method that automatically discovers \"sparse feature circuit\", defined as subgraphs of a large language model that satisfy two desiderata: (1) responsible for implementing a model behavior, and (2) human-interpretable. As a by-product, the authors propose SHIFT, a method that removes non-causal features (e.g. gender bias) from a classifier. Finally, the authors scale their method to thousands of automatically discovered behaviors from SAEs, hinting at the possibility of automated interpretability.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- This paper is very well written, and provides enough background on SAEs to motivate their approach. Each component of the method is also presented very clearly.\n- This contribution is very timely. Most prior works on mechanistic interpretability suffers from two pitfalls: (1) Inability to understand / interpret dense MLP layers [1], and (2) what's next after interpretability. This work provides satisfactory answers to both, by (1) using more recent approaches of SAEs to disentangle polysemanticity and (2) using interpretable features to debias a classifier / remove non-causal relationships from it.\n- Owning to the scalability of SAEs, this method can be scaled to automatically discover thousands of behaviors to billion scale parameters. The authors also supplement this ability with an interactive website, which offers many interesting insights and improves accessibility of this research.\n\n[1] How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model, https://arxiv.org/abs/2305.00586", "weaknesses": "- While this approach is scalable with respect to model size and a broad range of behaviors, using this approach would still require contrastive sample pairs that is hard to get and / or define for tasks such as long-form generation and algorithmic reasoning. Nevertheless, I believe this work represents a step in the right direction, but it'd be great for the authors to discuss this type of dataset limitations and future directions to automatically construct contrastive pairs for more challenging tasks.\n- As the authors have mentioned, most of these results are qualitative, and it would be great to establish a more quantitative benchmark to supplement the other interesting behaviors that the authors were able to mine from their unsupervised approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)