Transcoders find interpretable LLM feature circuits
Authors: Jacob Dunefsky
Philippe Chlenski
Neel Nanda
Venue: NeurIPS 2024 poster (2024)
Keywords: mechanistic interpretability
transcoders
sparse autoencoders
circuit analysis
Primary area: interpretability_and_explainability
TL;DR: Modifying sparse autoencoders to approximate MLP behavior enables clean, fine-grained, interpretable circuit analysis for transformers

Abstract: A key goal in mechanistic interpretability is circuit analysis: finding sparse subgraphs of models corresponding to specific behaviors or capabilities. However, MLP sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features—such as those found by sparse autoencoders (SAEs)—are typically linear combinations of extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this we explore **transcoders**, which seek to faithfully approximate a densely activating MLP layer with a wider, sparsely-activating MLP layer. We introduce a novel method for using transcoders to perform weights-based circuit analysis through MLP sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform at least on par with SAEs in terms of sparsity, faithfulness, and human-interpretability. Finally, we apply transcoders to reverse-engineer unknown circuits in the model, and we obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decomposing model computations involving MLPs into interpretable circuits. Code is available at https://github.com/jacobdunefsky/transcoder_circuits/

Transcoders Find Interpretable LLM Feature Circuits
Jacob Dunefsky"
Philippe Chlenski' Columbia University
Yale University
New Haven, CT 06511
New York, NY 10027 pacêcs columbia edu
jacob. dacobbduneeie www..ddnfssyyyllleeeerr
Neel Nanda
Abstract
A key goal mechanistic interpretability circuit analysis: finding sparse sub- graphs of models corresponding to specific behaviors or capabilities. However, MI.P sublayers make fine-grained circuit analysis on transformer-based language models difficult. In particular, interpretable features- such those found by sparse autoencoders (SAEs)-arc are typically linear combinations extremely many neurons, each with its own nonlinearity to account for. Circuit analysis in this setting thus either yields intractably large circuits or fails to disentangle local and global behavior. To address this explore transcoders, which seek to faithfully approximate a densely activating MLP layer with a wider, layer. We introduce novel method for using transcoders to perform veights-based circuit analysis through MI.P sublayers. The resulting circuits neatly factorize into input-dependent and input-invariant terms. We then successfully train transcoders on language models with 120M, 410M, and 1.4B parameters, and find them to perform least on par with SAEs in terms of sparsity, faithfulness, and human- interpretability. Finally, we apply transcoders to reverse-engineet unknown circuits the model, and obtain novel insights regarding the "greater-than circuit" in GPT2-small. Our results suggest that transcoders can prove effective in decompos- ing model computations involving MLPs into interpretable circuits. Code is avail- able at htthpt: github....u.... wtw.tttu..oooom.eeeee http://acccduunfffty///raaaoooo cwtrrasccoder_crrcucii
MLP
Introduction
In recent years, transformer-based large language models (LLMs) have displayed outstanding perfor- 46). However, the mechanisms by which LLMs perform these The field of mechanistic interpretablity [9] seeks to understand
mance on a wide variety of tasks 43
tasks are opaque by default [0] 33
these mechanisms, and doing so relies on decomposing a model into circuits 41: interpretable
subcomputations responsible for specific model behaviors [15 32
50].
core problem fine-grained circuit analysis is incorporating MLP sublayers
32 Attempting to
analyze MLP neurons directly suffers from "polysemanticity" B T6
24
40 the tendency of neurons
to activate on many unrelated concepts. To address this, sparse autoencoders (SAEs) 7 12 511
have been used to perform fine-grained circuit analysis by instead looking at features vectors in the model's s representation space-instead of individual neurons [4 34 However, while SAE features are often interpretable, these vectors tend to be dense linear combinations of many neurons (36]. Thus, mechanistically understanding how an SAE feature before one or more MLP layers affects later SAE feature may require considering infeasible number of neurons and their nonlinearities. Prior attempts to circumvent this [4. 34] use a mix of causal interventions and gradient- based
Equul contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Transformer layer (XnL)
Embedding
Unembed
Attention
MLP input
Transcoder
MLP output
SAE
Activation
W out
MLP
Figure I: A comparison between SAEs, MLP transcoders, and MLP sublayers for transformer- based language model. SAEs learn to reconstruct model activations, whereas transcoders imitate sublayers' input-output behavior.
approximations to MLP layers. But these approaches fail to exhibit input-invariance: the connections between features can only ever be described for a given input, and not for the model as whole. Attempts to address this, e.g. by averaging results over many inputs, conversely lose their ability to yield input-dependent information that describes a connection's ' importance on single input. This means that SAEs cannot tell us about the general input-oulput behavior of an MLP across all inputs.
To address why inputinnarriance is desirable, consider the following example: say that one has post-MLP SAE feature and wants to see how it is computed from pre-MLP SAE features. Doing e.g. patching on one input shows that a pre-MLP feature for Polish last names is important for causing the post-MLP feature to activate. But on other inputs, would features other than the Polish last name feature also cause the post-MLP feature to fire (e.g. an English last names feature)? Could there be other inputs where the Polish last names feature fires but the post-MLP feature does not? We can see that without input-invariance, it is difficult to make general claims about model behavior.
Motivated by this, in this work, we explore transcoders (an idea proposed, but not explored, in Templeton et al. 47 and Li et al. (3I): wide, sparsely-activating approximations of model's original MLP sublayer. Specifically, MLP transcoders are wide ReLU MLPs with one hidden layer that are trained to faithfully approximate the original narrower MLP sublayer's output, with an LI regularization penalty on neuron activations to encourage sparse activations. Our primary motivation is enable input-invarian feature-Ievel circuit analysis through MLP sublayers, which allows us to understand and interpret the general behavior of circuits involving MLP sublayers.
Our contributions. Our main contributions are (1) to introduce method for circuit analysis using transcoders, (2) to confirm that transcoders are faithful and interpretable approximation to MLP sublayers, and (3) demonstrate the utility of our circuit analysis method on detailed case studies.
After describing the architecture of transcoders in $3.1 we demonstrate in $3.2 that transcoders additionally enable circuit-findie finding techniques that are not possible using SAES, and introduce a novel method for performing circuit analysis with transcoders and demonstrate that transcoders cleanly factorize circuits into inpariiivaiant and imput-dependent components.
Then, in $ we evaluate transcoders' interpretability, sparsity, and faithfulness to the original model. Because SAEs are the standard method for finding sparse decompositions of model activations, compare transcoders to SAEs on models up to 1.4 billion parameters and verify that transcoders are on par with SAEs better with respect to these properties.
We apply transcoder circuit analysis to a variety of tasks in $5.1 and $5.2 including "blind case studies." which demonstrate how this approach allows us to understand features without looking at specific examples, and an in-depth analysis of the GPT2-small "greater-than circuit" previously studied by Hanna et al. (26).
2 Transformers preliminaries
we represent the computation of transformer model as follows. First, the model maps input tokens (and their positions) to embeddings (0.t) E RRemm where / is the token index and dmodel is the model dimensionality. Then, the model applies a series of "layers," which map the hidden state at the end of the previous block to the new hidden state. This can be expressed as:
Following Elhage et al. 15

---

x "mi
(l.t)
x (l.t)
pre
+ Σ
attn (1,))
(x (x(et) xxree xpeett))
(1)
hend h
x pre mid
(1+1tt)
(1.t)
MLP
(x(
(2)
where 7 is the layer index, L. is the token index, attn((.)) (x (Lt)) xpre*) denotes the output of
(Xpre: Xpre
attention head h at layer given all preceding source tokens Xpre and destination token Xpre and MLP (1.t) denotes the output of the layer MLP
(Lt)
Equation D shows how the attention sublayer updates the hidden state token and Equation
shows how the MIP sublayer updates the hidden state. Importantly, each sublayer always adds its output to the current hidden state. As such, the hidden state always can be additively decomposed into the outputs of all previous sublayers. This motivates Elhage et al. [[5] to refer to each token's s hidden state as its residual stream, which "read from" and "written to" by each sublayer.
Transcoders
3.1 Architecture and training
Transcoders aim to learn "sparsified" approximation of an MLP sublayer: they approximate the output of an MLP sublayer sparse linear combination of feature vectors. Formally, the transcoder architecture can be expressed as
(x) ReLU (W Vene + bene)
TC(x) = W WdecZTc(x) + bade,
(4)
where x is the input to the MLP sublayer, W E [Rchhamme danaed W E Rd Xdhaams benc
Rduers, Ddee E IR diaie dfeanrrs is number feature vectors in transcoder, and dmodel the dimensionality of the MLP input activations. Usually, dieatures is far greater than dmodel-
Each feature in transcoder associated with two vectors: the i-th row of W Wene is the encoder Wdec feature vector of feature i, and the i-th column of W dec is the decoder feature vector of feature
The i-th component of cTc(x) xx) is called the activation feature Intuitively, for each feature, the encoder vector is used to determine how much the feature should activate; the decoder vector is then scaled by this amount, and the resulting weighted sum of decoder vectors is the output of the
transcoder. In this paper, the notation e(I.i) and ((..)) is used to denote the i-th encoder feature vector and decoder feature vector, respectively, in the layer transcoder.
Because we want transcoders to learn to approximate an MLP sublayer's computation with a sparse linear combination of feature vectors, transcoders trained with the following loss, where X is hyperparameter mediating the tradeoff between sparsity and faithfulness:
Lre(x) = |MLP(xx) TC(x) + A [x)
(5)
faithfulness
sparsity penalty
3.2 Circuit analysis with transcoders
We now introduce novel method for performing feature-level circuit analysis with transcoders, which provides a scalable and interpretable way to identify which transcoder features in different layers connect to compute a given task. Importantly, this method provides insights into the general input-output behavior of MLP sublayers, which SAE-based methods cannot do.
In particular, the primary goal of circuit analysis is identify subgraph of the model's computational graph that is responsible for (most of) the model's 's behavior on a given task [1] 19 20); this requires a means of evaluating a computational subgraph's importance the task in question. In order to
2 Note that the "Pythia" family models computes MLP and attention sublayer outputs parallel. This (1+1,t) (1.t) means that Equation is thus given by xpre mid + MLP(() (1.t)

---

Feature
Feature
Feature
Feature
Feature
Step 1: find the direct contributions
layer Festures to later- layer feature.
Feature
Feature
Feature
Feature
delete but the & most highly- contributing features.
Step select the top- contributing features each selected features.
Step 4; delete all but most important paths the graph again.
Step Repeat steps 2- until you have paths of longth
Step 6: Merge top paths into single cir- cuit.
Figure 2: A visualization of the circuit-finding algorithm.
determine which edges are included in this subgraph, we thus have to compute attributions for each edge: how much the earlier node contributes to the later node's 's own contribution. Circuit analysis with SAEs thus entails computing the attribution of pre-MLP SAE features to post-MLP SAE features, as mediated through the MLP.. Standard methods for computing attributions are causal patching (which inherently only gives information about local MLP behavior on single input) and methods like input-tine couut-timesggrenn -ennd-gradeent attribution patching (which are equivalent in this setting). We will now demonstrate why these methods cannot yield information about the MLP's general behavior. Letting be the activation of an earlier-layer feature, and be the activation of the MLP at layer I', the input-times-gradien is given by:
the activation of the later-layer feature,
Oz'
Oz' dy Oy Oz
z
z
(6)
Oz
Unfortunately, not only z input-dependent, but is Oz'
Dz as well, because
3z is.
This means that we cannot use SAEs to understand the general behavior of MLPs on various inputs. In contrast, will show that when we replace MI.P sublayers with sufficiently faithful and interpretable transcoders, we obtain attributions that neatly factorize into iepen-dependent terms and input-ing invariana terms; the latter can be computed just from model and transcoder weights, and tell us about the MLP behavior across all inputs.
3.2.1 Attribution between transcoder feature pairs
We begin by showing how compute attributions between pairs of transcoder features. This attribution is given by the product of two terms: the earlier feature's 's activation (which depends on the input to the model), and the dot product of the earlier feature's decoder vector with the later feature's encoder vector (which is independent of the model input).
(1.i)
The following is more formal restatement. Let 2TC (xmid) denote the scalar activation of the i-th (1.t) feature in the layer transcoder token t, function of the MLP input x token t in layer 1. Then for layer < l', the contribution feature i transcoder to the activation of feature i' in transcoder I' on token f is given by
(1.8) TC
(1.t) Xii
(L.)) den
e(''1)) lene
x
input dependent
input invariant
D.2 Note that (e(LD)
='(''')) is input-invariant: once the
This expression derived in App. transcoders have been trained, this term does not depend the input the model. This term,

---

analyzed in isolation, can thus be viewed as providing information about the general behavior of the model. The only input-deppe uipudddepeneentt term is 2rC transcoder token t. As such, this expression cleanly factorizes into term reflecting the general input-invariant connection between the pair of features and an interpretable term reflecting the importance of the earlier feature on the current input.
(t.i)
x (I.t)
mid the activation of feature in the layer /
3.2.2 Attribution through attention heads
So far, we have addressed how to find the attribution of a lower-layer transcoder feature directly on a higher-layer transcoder feature the same token. But transcoder features can also be mediated by attention heads. We will thus extend the above analysis to account for finding the attribution of transcoder features through the OV circuit of an attention head. For a full derivation, see App. D33
As before, we want to understand what causes feature i' in the layer' I' transcoder to activate on token Given attention head h at layer with ( < I', the contribution of token s layer through head to feature i' in layer l' at token & is given by
(((WO))
scorell.A) (1.N)
(xree
(l.t)
xfee))
p(l's))
lene
(1.x)
(8)
(1a]) where score(l.)) (x.tee, xpre)
is the attention score for head h and layer from token to token L.
3.2.3 Finding computational subgraphs
Using this observation, we present a method for finding computational subgraphs. We now know how to determine,
given input and transcoder feature i', which earlier-layer transcoder features are important for causing i' to activate. Once we have identified some earlier-layer features that are relevant to i', then we can then recurse to understand the most important features causing to activate by repeating this process.
Doing iteratively (and greedily pruning all but the most important features each step) thus yields a set of computational paths (a sequence of connected edges). These computational paths can then be combined into computational subgraph, in such way that each node (transcoder feature or attention head), edge, and path is assigned an attribution. A full description the circuit- findiin algorithm is presented in App. D.5 Figure 2 provides visualization of this algorithm.
3.2.4 De-embeddings: a special case of input-invariant information
Earlier, we discussed how to compute the input-invariant connection between a pair of transcoder features, providing insights on general behavior of the model. A related technique something that we call de-embeddings, A de-embedding vector for transcoder feature is vector that contains the direct effect of the embedding of each token in the model's 's vocabulary on the transcoder feature. The
de-embedding vector for feature in the layer transcoder is given by WE T' "'cll) where WE is the model's 's token embedding matrix. Importantly, this vector gives input-invariant information about how much each possible input token would directly contribute to the feature's 's activation.
Given de-embedding vector, looking at which tokens in the model's vocabulary have the highest de- embedding scores tells us about the feature's 's general behavior. For example, for a certain GPT2-small MLPO transcoder feature that we investigated, the tokens with the highest scores were oglu owsky, zyk chenko, and kowski. Notice the interpretable pattern: all of these tokens come from European surnames, primarily Polish ones, suggesting that the feature generally fires on Polish surnames.
4 Comparison with SAES
Transcoders were originally conceived as variant of SAEs, and as such, there are many similarities between them. They differ only their training objective: because SAEs are autoencoders, the faithfulness term in the SAE loss measures the reconstruction emor between the SAE's output and its original input. In contrast, the faithfulness term of the transcoder loss measures the error between the transcoder's 's output and the original MLP sublayer's output.
5

---

Table 1:
The number of interpretable features. pessibly-interpretabbee features, and uninterpretable features for the transcoder and MLP-in SAE. Of the interpretable features, we additionally deemed 6 transcoder features, and SAE features to be "context-free", meaning they appeared fire on single token without any evident context-dependen patterns.
Transcoder MIP-in SAE
interpretable
41
38
# maybe uninterpretable
8
8
4
Because of these similarities, SAEs can be quantitatively evaluated (for sparsity and fidelity) and qualitatively evaluated (for feature interpretability) in precisely the same way as transcoders, using standard SAE evaluation methods [ [2]]. We now report the results of evaluations comparing SAEs to transcoders on these metrics, and find that transcoders are comparable to or better than SAEs.
4.1 Blind interpretability comparison of transcoders to SAES
In order evaluate the interpretability of transcoders, manually attempted interpret 50 random features from Pythia-410M4 layer 15 transcoder and 50 random features from Pythia-410N, layer 15 SAE trained on MLP imputs] For each feature, the examples in subset of the Open WebText corpus that caused the feature to activate the most were computed ahead of time. Then, the features from both the SAE and the transcoder were randomly shuffled. For each feature, the maximun-activaaiig examples were displayed, but not whether the feature came from an SAE or transcoder. We recorded for each feature whether or not there seemed be an interpretable pattern, and only after examining every feature did we look at which features came from where. The results, shown in Table suggest transcoder features are approximately as interpretable as SAE features. This further suggests that transcoders incur no penalties compared to SAES.
4.2
Quantitative comparison of transcoders to SAFs
4.2.1
Evaluation metrics
We evaulate transcoders qualitatively on their features' interpretability as judged by a human rater, and quantitatively on the sparsity of their activations and their fidelity to the original MLP's computation.
As qualitative proxy measure for the interpretability assuming that interpretable features should demonstrate interpretable patterns in the examples that cause them to activate. To this end, one can run the transcoder on a large dataset of text, see which dataset examples cause the feature to activate, and see if there is an interpretable pattern among these tokens. While imperfect (6), this is still reasonable proxy for an inherently qualitative concept.
feature. we follow Bricken et al. ( in
To measure the sparsity of transcoder, one can run the transcoder on dataset of inputs, and calculate the mean number of features active on each token (the mean Lo norm of the activations). To measure the fidelity of the transcoder, one can perform the following procedure. First, run the original model on a large dataset of inputs, and measure the next-token-preede prediction cross entropy loss on the dataset. Then, replace the model's 's MLP sublayer corresponding to the transcoder with the transcoder, and measure the modified model's mean loss on the dataset. Now, the faithfulness of the transcoder can be quantified as the difference between the modified model's 's loss and the original model's 's loss.
4.2.2 Results
We trained SAES and transcoders on activations from GPT2-small 44] Pythia-410M, and Pythia- 1.4B (2). For each model, we trained multiple SAEs and transcoders the same inputs, but with different values of the Y1 hyperparameter controlling the adelity-sparsing tradeoff for each SAE and each transcoder. The transcoders were trained MLP-in and MLP-out activations, while SAES were
We used SAEs trained on MLP inputs here because the interpretability case studies look at feature activations, which are solely dependent on the encoder vectors of the SAEs and transcoders. Because transcoders' encoder vectors live in MLP input space, we thought that the comparison would be most accurate if our SAEs" encoder vectors also lived in MLP input space.

---

Pylhiz layer 15
Pythis1 4B byre 15
33
31
SA
Moon wllge
Mann
Maxn
Figure 3: The sparsity- accuracy tradeoff of transcoders versus SAEs on GPT2-small, Pythia-410M, and Pythia- 1.4B. Each point corresponds trained SAE or transcoder, and labeled with the L1 regularization penalty Y1 used during training.
trained on MLP-out activations (as these are the activations that MLP SAEs are typically trained on). Due to compute limitations, we used the same learning rate, which was determined via a hyperparameter sweep on transcoders, for both SAEs and transcoders. This means that the learning rate might not be optimal for SAEs. Nevertheless, we did perform a separate hyperparameter sweep of X1 for the SAEs and transcoders.
We evaluated each SAE and transcoder on the same 3.2M tokens of OpenWebText data [21]. We also recorded the loss of the unmodified and mean-ablated model (always replacing the MLP sublayer output with its mean output over the dataset) as best- and worst-case bounds, respectively.
We summarize the Pareto frontiers of the sparsity-accurace tradeoff for all models in Figure 3 In all cases, transcoders are equal to or better than SAEs. In fact, the gap between transcoders and SAEs seems to widen on larger models. Note, however, that compute limitations prevented us from performing more exhaustive hyperparameter sweeps; as such, might be possible that a different set of hyperparameters could have allowed SAEs to surpass transcoders. Nonetheless, these results make us optimistic that using transcoders incurs no penalties versus SAES trained on MLP activations.
5
Circuit analysis case studies
5.1 Blind case study: reverse-engineering a feature
To understand the utility of transcoders for circuit analysis, we carried out nine blind case studies, where we randomly selected individual transcoder features in a ninth-layer (of 12) GPT2-small transcoder and used circuit analysis form hypothesis about the semantics of the feature-without looking at the text of examples that cause the feature to activate. In blind case studies, we use combination of input-invariant and input- neputdde information to allow us to evaluate transcoders as tool to infer model behavior with minimal prompt information. This better reflects key goal of mechanistic interpretability: to be able to understand model behavior on unknown, unforeseen tasks.
In contrast, reverse-engineering a feature where one already has an idea of its behavior can introduce confirmation bias. For instance. looking at activation patterns prior circuit analysis can predispose a researcher to seek out only circuits that corroborate their interpretation of these activation patterns, potentially ignoring circuits that reveal other information about the feature. Conversely, if the circuit analysis method is faulty and yields some explanations that are not reflected in the feature activations, then the researcher might ignore those spurious explanations and thus obtain an overly prosiiiv assessment of the circuit analysis method. The "rules of the game" for blind case studies are that:
1.
The specific tokens contained in any prompt are not allowed to be directly seen. As such, prompts and tokens can only be referenced by their index in the dataset.
These prompts may be used to compute input-dependent information (activations and circuits), as long as the tokens themselves remain hidden.
3. Any input-invariant information, including feature de-embeddings, is allowed.
In this section, we summarise a specific blind case study, how we used our circuits to reverse-engineer feature 355 in our layer 8 transcoder. Other studies, as well as a longer description of the study summarized here, can be found in App. H
7

---

Note that we use the following compact notation for transcoder features: tcc tcA[B] 8C refers to feature in the layer A transcoder at token C.
Building the first circuit. We started by getting a list of indices of the top-activating prompts in the dataset for tc8 [355]. Importantly, we did not look the actual tokens in these prompts, as doing so would violate Rule 1. For our first input, we chose example 5701, token 37; tc8 [355] fires strength 11.91 on this token in this input. Our greedy algorithm for finding the most important computational paths for causing tc8 [355] @37 to fire revealed contributions from the current token (37) and earlier tokens (like 35, 36, and 31).
Current-token features. From token 37, we found strong contributions from tc0 [16632] 637 and tc0 [9188] 037. Input-invariant de de-embeddings of these layer 0 features revealed that they primarily activate on variants of j. suggesting that token 37 contributed to the feature by virtue of being a semicolon. Another feature which contributed strongly through the current token, tc6 [11831]. showed a similar pattern. Among the top input-invariant connections from layer 0 transcoder features to tc6 [11831], we once again found the same semicolon features tc0 [16632] and tc0 [9188].
Previous-token features. Next we checked computational paths from previous tokens through attention heads. Looking at these contextual computational paths revealed a contribution from tc0 [13196] @36; the top de-embeddings for this feature were years like 1973, 1971, 1967, and 1966. contribution from tc0 [10109] e31, for which the top de-embedding was (
Additionally, there
Furthermore, there was contribution from tc6 [21046] @35. The top input-invariant connections to this feature from layer O were tc0 [16382] and tc0 [5468]. The top de-embeddings for the former were tokens associated with Eastern European last names (e.g. kowski, chenko, owicz) and the top de-embeddings for the latter feature were English surnames (e.g. Burnett, Hawkins, Johnston). This heavily suggested that tc6 [21046] was a surname feature.
Thus, the circuit revealed this pattern was important to our feature:
We hypothesized that tc8 [355] fires on semicolons in parenthetical citations like "(Vaswani et al. 2017; Elhage et al. 2021)". Further investigation another input yielded a similar along with a feature whose top de-embedding tokens included Accessed, Retrieved, Neuroscience, and Springer. This bolstered our hypothesis even more.
Analysis.
Here, we decided to end the blind case study and check if our hypothesis was correct. Sure enough, the top activating examples included semicolons in citations such as "(Poeck, 1969; Rinn, 1984)" and "(Robinson al., 1984; Starkstein et al., 1988)". We note that the first of these is the example at index (5701, 37) we analyzed above.
"Restricted" blind case studies. Because MLPO features tend to be single-token, significant information about the original prompt can be obtained by looking at which MLPO transcoder features are active and then taking their de-embeddings. In order to address this and more fully investigate the power of input-invariant circuit analysis, six of the eight case studies that carried out were restricted blind case studies, in which all input-dependent MI.PO feature information is forbidden to use. For more details on these case studies, see Appendix H.2
5.2 Analyzing the GPT2-small "greater-than" circuit
We now turn to address the "greater-than" circuit in GPT2-small previously considered by Hanna et al. [25]. They considered the following question: given a prompt such as "The war lasted from 1737 to 17", how does the model know that the predicted next year token has to be greater than 1737? In their original work, they analyzed the circuit responsible for this behavior and demonstrated that MLP10 plays important role, looking into the operation of MLPIO at a neuronal level. We now apply transcoders and the circuit analysis tools accompanying them to this same problem.
5.2.1 Initial investigation
First, we used the methods from Sec. 3.2.3 to investigate single prompt and obtain the computational paths most relevant to the task. This placed high attribution on MIP10 features, which were in turn activated by earlier-layer features mediated by attention head 1 in layer 9. This corroborates the analysis in the original work.
8

---

Performance vs. features/merross used
Logits boosted by transcoder feature 5315
incresse
Normallee annbbdddngg
0.55
Proalality difference
0.80
1.00
0.90
0.75
0.90
Seplace TCLO (c10]5315D
0.70
Transe oder testures twithout 1o10
0.85
Nuarom
0.63
Peplace with TC10
Transndder featuree (with t:10)531
0.80
entirely
0.75
20
45
1.00
Number faturessmmerons
Year token TY
Figure 4: Left: Performance according to the probability difference metric when all but the top k: features or neurons in MLPI0 are zero-ablated. Right: The DLA and de-embedding score for tc10 [5315], which contributed negatively to the transcoder's performance.
Next, we investigated which MLP10 transcoder features were most important on a variety of prompts, and how their activations are mediated by attention head 1 in layer 9. Following the original work, we generated all 100 prompts of the form "The war lasted from 17YY to 17", where YY denotes two-digit number. We found that the MLP10 features with the highest variance in activations over this set of prompts also had top input-dependent connections from MLPO features through attention head in layer 9 whose top de-embeddings were two-digit numbers. The top input-invariant connections from MLPO features through attention head in layer to MLPIO features also had two-digit numbers among their top de-embeet de-embedding tokens. This positive result was somewhat unexpected, given that there are only 100 two-digit number tokens in the model's vocabulary of over 50k tokens.
We then used direct logit attribution (DLA) [5] to look at the effect of each transcoder feature on the predicted logits of each YY token the model's vocabulary. These results, along with de- embedding scores for each YY token, can be seen in Figure 5] The de-embeddings scores are highest for tokens where years following them are boosted and years preceding them are inhibited.
5.2.2 Comparison with neuronal approach
Next, we compared the transcoder approach to the neuronal approach to see whether transcoders give a sparser description of the circuit than MLP neurons do. do this, we computed the 100 highest-variance variance layer 10 transcoder features and MLP10 neurons. Then, for 1 < k < 100, zero-ablated all but the top & features in the transcoder/neurons in MLP10 and measured how this affected the model's performance according to the mean probability difference metric presented in the original paper. We also evaluated the original model with respect to this metric, along with the model when MLP10 is replaced with the full transcoder.
The results are shown in the left half of Figure For fewer than 24 features, the transcoder approach outperforms the neuronal approach; its performance drops sharply, however, around this point. Further investigation revealed that tc10 [5315], the 24th-highest-variance transcoder feature, was responsible for this drop in performance. The DLA for this feature is plotted in the right half of Figure Notice how, in contrast with the three highest-variang displays flatter DLA, boosting all tokens equally. This might explain why it contributes to poor performance. To account for this, note that the left half of Figure T also demonstrates the performance of the transcoder when this "bad feature" is removed.
transcoder features, tc10 [5315]
While the transcoder does not recover the full performance of the original model, it needs only handful of features to recover most of the original model's performance; many more MLP neurons are needed to achieve the same level of performance. This suggests that the transcoder is particularly useful for obtaining sparse, understandable approximation of MLP10. Furthermore, the transcoder features suggest simple way that the MIP10 computation may (approximately) happen: by a small set of features that fire years in certain ranges and boost the logits for the following years.

---

Lagits bornted by transcorber Ensture 1954
Logits honsted by transcorder feature 31238
Engits hansted by trwnsrader feature 19637
02
Figure 5: For the three MLP10 transcoder features with the highest activation variance over the "greater-than" dataset, and for every possible YY token, we plot the direct logit attribution (the extent to which the feature boosts the output probability of YY) and the de-embedding score (an input-invariant measurement of how much YY causes the feature to fire).
6
Related work
Circuit analysis is a common framework for exploring model internals 15 32 41 A number of approaches exist to find circuits and meaningful components in models, including causal ap- proaches [20], automated circuit discovery [II], and sparse probing (24). Causal methods include activation patching 22 49 52). attribution patching 330 39], and path patching 122 50 Much
circuit analysis work has focused on attention head circuits 18. including copying heads [15], induction heads 42), copy suppression 335 and successor heads (23.. Methods connecting circuit
analysis to SAEs include He et al. (27), Batson et al. [ and Marks et al. (34] Our recursive greedy
circuit-finding approach was largely based on that of Dunefsky & Cohan
Sparse autoencoders have been used to disentangle model activations into interpretable fea- tures [1). The development of SAEs was motivated by the theory of superposition in neural representations 17). Since then, much recent work has focused on exploring and interpreting SAEs, and connecting them to preexisting mechanistic interpretability techniques. Notable contributions include tools for exploring SAE features, such as SAE lens [5]; applications of SAES to attention sublayers [9]; scaling up SAEs to Claude Sonnet
48 and improved SAE architectures (5 Transcoders have been previously proposed as a variant of SAEs under the names "predicting future activations"
47 and "MLP stretchers"
[3I], but not explored in detail.
7 Conclusion
Fine-grained circuit analysis requires an approach to handling MLP sublayers. To our knowledge, the transcoder-based circuit analysis method presented here is the only such approach that cleanly disentangles input-invariant information from input-dependent information. Importantly, transcoders bring these benefits without sacrificing fidelity and interpretability: when compared the-art feature-level interpretability tools (SAEs), we find that transcoders achieve equal or better performance. We thus believe that transcoders are an improvement over other forms of feature-level interpretability tools for MLPs, such as SAES on MLP outputs.
state-of-
Future work transcoders includes directions such comparing the features learned by transcoders to those learned by SAES, seeing if there are classes of features that transcoders struggle to learn, finding interesting examples of novel circuits, and scaling circuit analysis to larger models.
Overall, we believe that transcoders are an exciting new development for circuit analysis and hope that they can continue to yield deeper insights into model behaviors.
Limitations Transcoders, like SAEs, are approximations to the underlying model, and the resulting error may lose key information. We find transcoders to be approximately as unfaithful to the model's computations as SAES are (as measured by the cross-entropy loss), although we leave comparing the errors to future work. Our circuit analysis method (App. D.5) does not engage with how attention patterns are computed, and treats them as fixed. A promising direction of future work would trying to extend transcoders to understand the computation of attention patterns, approximating the attention softmax. We only present circuit analysis results for a few qualitative case studies, and our results would be stronger with more systematic analysis.
10

---

Impact statement
This paper seeks to advance the field mechanistic interpretability by contributing a new tool for circuit analysis. We this foundational research, and expect the impact to come indirectly from future applications of circuit analysis such as understanding and debugging unexpected model behavior and controlling and steering models to be more useful to users.
Acknowledgments and Disclosure of Funding
Jacob and Philippe were funded by a grant from AT Safety Support Ltd. Jacob was additionally funded by grant from the Long-TTrmm Future Fund. Philippe was additionally funded by NSF GRFP grant DGE-2036197. Compute was generously provided by Yale University.
We would like to thank Andy Arditi, Lawrence Chan, and Matt Wearden for providing detailed feedback on our manuscript. We would also like thank Senthooran Rajamanoharan and Juan David Gil for discussions during the research process, and Joseph Bloom for advice on how to use (and extend) the SAELens library. Finally, we would like thank Joshua Batson for discussion that inspired us to investigate transcoders in the first place.
References
[1] Batson, J., Chen, B., and Jones, A. Using features for easy circuit identification. 2024. URL httphhtte//rraaasforreerrr- hit.mmerrrrrruss.sppp.. http://2//maccc-uppaa////eeeaaee html#
leature-heade
[2] Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: Suite for Analyzing Large Language Models Across Training and Scaling, May 2023. URL http: /ttp:///rxivvv har/v.xivvv..o org/ abs/ 2304. 01373 arXiv:2304.01373 [cs].
[3] Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L.., Goh, G., Sutskever, I,, Leike, J., Wu, J., and Saunders, W. Language models explain neurons in language mod- els, 2023. paper/index.
blob. core windows. rindovs.net/neuroo. www.nneetnnnnneeeeeee -nteeurrnn-eppiineer
[4] Bloom, J. Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small, 2024. URL htth::ttt////wee
/ wwtp::wwwwwsssssoooeeomoom http:///....ooooeee/////////////
[5] Bloom, SAELens Training, 2024. URL httphttts///bbommauss .gt.ebg......................... 10/ SAELens/
[6] Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F, and Wattenberg, M. An interpretability illusion for bert. arXiv preprint arXiv: 1rXiv:2104.07143. 2021.
[7] Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Maufield-Dodds, Z., Tamkin, A., Nguyen, K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.
[8] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A.,
Shyam, P., Sastry, G., Askell, A., Agarwal, S., Berbert-Vosss A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners, July 2020. URL hattp://arxiv org attaaaabs./....//// 14165 14115 [cs].
[9] Chris Olah. Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases, 2022. http:///ppe.eteaoooocccc//.ce//a html

---

[10] Chrupata, G. and Alishahi, A. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2952-2962, 2019. doi: 1118653/11/11-12833 URLhttp: /tatpp://axxv..o. http://aaxxxv.....aaaa. abs/ /905 06401 1xXiv:1905006400 [es].
URt.http://axxiv
[11] Conmy, A., Mavor-Parked A. N., Lynch, A., Heimersheim, S., and Garriga-Alonoo, A. Towards Automated Circuit Discovery for Mechanistic Interpretability, October 2023. URL http: /aaxiiv. www. /2044. 14997 arXiv:2304.1499 [cs].
[12] Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse Autoencoders Find Highly Interpretable Features in Language Models, October 2023. URL http://araxxi..orr abs/2309 08600 arXiv:2309.0888 00030000000000000 [cs].
[13] Dunefsky, J. and Cohan, A. Observable Propagation: A Data-Efficient Approach to Uncover Feature Vectors in Transformers, December 2023. URL attt:://rxiv. org/ ha//abb//23//
16291
arXiv:2312.16229 [es].
[14] Dunefsky, Chlenski, P., Rajamanoharan, S., and Nanda, N. Case Studies in Reverse- Engineering Sparse Autoencoder Features by Using MLP Linearization, 2024. URL/httpa;
/www.oo www.aalmmmnefformmmmm wwtptottt///ooootootttottttt. /ttp/////nnntte06//R00// /ttpa///w....ttto/////////////nr Published: Alignment Forum.
[15] Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A Mathematical Framework for Transformer Circuits. Transformer
Circuits Thread, 2021.
[16] Elhage, N., Hume, T., Olsson, C., Nanda, N., Henighan, T., Johnston, S., ElShowk, S., Joseph, N., DasSarma, N., Mann, B., Hernandez, D., Askell, A., Ndousse, K., Jones, A., Drain, D., Chen, A., Bai, Y., Ganguli, D., Lovitt, L., Seffield-Dodds, Z., Kernion, J., Conerly, T., Kravec, S.. Fort, S., Kadavath, S., Jacobson, J., Tran-Johnson, E., Kaplan, J., Clark, J., Brown, T.,
McCandlish, S., Amodei, D., and Olah, C. Softmax linear units. Transformer Circuits Thread, 2022. https:/transsformr-eiiiee cirruuisserrrrrrttriemeeeeeuute, pttp:2022/sll///dxx.hmmmt http://oeeoeororiiuiisooo//////a
[17] Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield Doddds Z.. Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M., and Olah, C. Toy Models of Superposition, September 2022. URL http [cs].
//arxiv. www./////////////////.......ee// 2209 10652 arXiv:2209.
(18) Ferrando, J., Sarti, G., Bisazza, A., and Costa-jussà, M. R. A Primer the Inner Workings of Transformer-based Language Models, May 2024. URL http:///axixt wtg.//owwwi.....aaaaa.oo.oo wwtp://///////0/////// /405 00208 trXiv:2405.00200 [cs] version: 2.
[19] Gandelsman, Y., Efros, A. A., and Steinhardt, J. Interpreting CLIP's Image Representation via Text-Based Decomposition, March 2024. http://////ax org/abs/ 2ttp//2110...111666 arXiv:2310.05916 [cs].
[20] Geiger, A., Lu, H., Icard, T., and Potts, C. Causal Abstractions of Neural Networks, October 2021. URL http:://ttttt .ww.iiiiorr///oo attpaab//.....///////////eeeeeee 02997 urXiv:2106.02997 [cs].
[21] Gokaslan, and Cohen, V. OpenWebText Corpuus Corpus, 2019.
[22] Goldowsky- -Dil,, N., MacLeod, C., Sato, L., and Arora, A. Localizing Model Behavior with Path Patching, May 2023. URL httt:://arxia org/ abs/ 2304 05969 arXiv:2304.05969 [cs].
(23) Gould, R., Ong, E., Ogden, G., and Conmy, A. Successor Heads: Recurring, Interpretable Attention Heads In The Wild, December 2023. Uttp://///aaxx http:/www..rg/b///22...... 09230 arXiv:2312.09233 [cs].
[24] Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troitskii, D., and Bertsimas, D. Finding Neurons in Haystack: Case Studies with Sparse Probing, June 2023. http:///t////xiiii wree/iii.orrr...o abs/ 2305. 01610 arXiv:2305.016. [cs].
12

---

[41] Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024.001, March 2020. ISSN 2476-0757. doi: 10. wttdiisiill.00224... [25] Hanna, M., Liu, O., and Variengien, A. How does GPT-2 compute theat: Interpreting mathematical abilities in a pre-trained language model, November 2023. URL jattp:///aaxi org/abs/ 2305. 00586 arXiv:2305.000 000566 [cs].
[26] Hanna, M., Pezzelle, S., and Belinkov, Y. Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms, 2024. eeprin:: 2403.17806.
[27] He, Z., Ge, X., Tang, Q. Sun, T., Cheng, Q. and Qiu, X. Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study Othello-GPT, February 2024. httttttpp///iiii wttp://wwww.gg////////////// 12201 1tXiv:2402.12200 [cs].
[28] Heimersheim, S. and Nanda, N. How to use and interpret activation patching, arXiv preprint arXiv:2404.15255 2024.
[29] Kissane, C., Krzyzanowski, R., Commy, A., and Nanda, Attention SAEs Scale to GPT-
2 Small, 2024. URL omm.o.oomtt////t.oooooooooooooom httpr///rrroppptts//tsttttttttt juita Published: Alignment Forum.
[30] Kramár, J., Lieberum, T., Shah, R., and Nanda, N. AtP*: An efficient and scalable method for localizing LLM behaviour components, 2024. _eprint: 2403.00745.
[31] Li, M., Marks, S., and Mueller, A. https//giitbb.oo//sapmmndddiiii heanaaaaaeaaaintaarrrrreeeeeee-e wwt.feleeeetttt-ffnttiiiiiiiiiii
dictionary repository, 2023.
[32] Lieberum, T., Rahtz, M., Kramár, J., Nanda, N., Irving, G., Shah, R., and Mikulik, V. Does Cir- cuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla, July 2023. URL htttp://axx: www./bs////eeeeeeeeoo /tabs//3307/ 09458 arXiv:2307.0944 [es].
[33] Lipton, Z. C. The mythos of model interpretability. 2017.
[34] Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models, March 2024. /ttp://////a org/ abs/2403. 19647 arXiv:2403. 19647 [cs].
[35] McDougall, C., Commy, A., Rushing, C., McGrath, T., and Nanda, N. Copy Suppression: Comprehensively Understanding an Attention Head. ArXiv, abs/2310.04625, 2023. URL. https://api. wwwanticscholaattt Org/CorpusID: :tt//CCrPPIID::233112999 263831290
[36] Nanda, N. Open source replication & commentary on anthropic's dictionary learning paper. Alignment Forum, 2023.
[37] Nanda, N. and Bloom, J. TransformerLens, 2022. URL https://github.. com/ TranssormerLennSrrgg//o hawpasDrrrrraassfomeeeeeee
[38] Nanda, N., Rajamanoharan, S., Kram' ar, J., and Shah, R. Fact Finding: Attempting to
Reverse-Engineer Factual Recall on the Neuron Level, December 2023. Utt/htts:://ww
Publication Title: Alignment Forum.
[39] Neel Nanda. Attribution Patching: Activation Patching At Industrial Scale, 2024. URI,https
[40] Olah, C., Mordvintsev. A., and Schubert, L. Feature visualization. Distill, 2017. doi: 10.23915/ distill.00007.
pw2p /020/ URL https: http::///p.pppe/////ooroc/////// wwwcciisstooomm-..
[42] Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell,
A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield- Moadfs,d--Dods, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J.,
Kaplan, J., McCandlish, S., and Olah, C. In-context Learning and Induction Heads, September 2022. URL http:
arxii.iiivv...ii org/ abs/2209 11895 arXiv:2209.1 11895 [cs].
13

---

[43] OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad. L... Akkaya, I,, Aleman, F. L.. Almeida,
D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji. S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadert-Shaping, G., Berner, C., Bogdonoff, I.., Boiko, O., Boyd, M., Brakman, A.-I.., Brockman, G., Brooks, T. Brundage, M., Button, Cai, T., Campbell, R., Cann, A., Carey, B., Carison, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B. Cho, C., Chu, C.. Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L.. Georges. E.. Gibson, C., Goel, V. Gogineni, T., Goh, G., Gontijo-Lppes, R., Gordon, J., Grafstein, M., Gray. S., Greene, R., Gross, J., Gu. S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Kaiser, L.,, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L.., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J. Knight, M., Kokotajlo, D., Kondraciuk, L., Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B.. McKinney, S. M., McLeavey, C., McMillan, P.. McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A.. Mishkin, P., Monaco, V.. Morikawa, E.. Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo. R., Noh, H., Ouyang. I.., Oeeefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., J., Parpariaa, E.,, Passos, A., Pavlov, M., Peng, A., Perelman, A., Peres, F. d. A. B., Petrov, M., Pinto, H. O., Michael, Pokorny, Pokrass, M., Pong. V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae. J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt,, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K. Sherbakov, T., Shich, J., Shoker, S., Shyam, P., Sidor, S., Sigler. E.. Simens, M.. Sitkin, J.. Slama, K., Sohl, I., Sokolowsky, B., Song. Y.. Staudacher, N., Such, F. P., Summers, N., Sutskever, L., Tang. J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang. J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, J., Welihinda, A., Welinder, P., Weng, J., Weng. L.,, Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L.,, Wu, S., Wu, J.. Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. GPT-4 Technical Report, ar arXiv:2303.0877 [cs].
March 2024. URL Utthttp::// /aattp://arxivg www. abs/ /tt.//233..0007 08774
[44] Radford, A., J., Child, R., Luan, D.,, Amodei, D., and Sutskever, I. Language Models Unsupervised Multitask Learnrr. Technical report, OpenAI, 2019.
[45] Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving Dictionary Learning with Gated Sparse Autoencoders, April 2024. URL 20w.aa///2244.... 16014 urXiv:2404.16014 [cs].
http: http:///aaaa...rr///////0/00000
[46] Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[47] Templeton, A., Batson, J., Jermyn, A., and Olah, C. Predicting Future Activations, January 2024. URL.
circuitssts..... pwtppppp//22ppeee waaaan-uddd//iiiii iiteeeiiiidxxxeee html#
predict-future
[48] Templeton, A., Conerly, T., Mareus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trunsformer Circuity Thread, 2024. URL, https: tttt:::://traanfoomeerrrre cireriitrrrirritttt.e pub/2224) scaling www.sooantiiiityoooooooo html
14

---

[49] Vig, J., Gehrmann, S., Belinkov, Y.. Qian, S., Nevo, D., Singer, Y. and Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL URL/attps://prcceedanss neurips. cc/ wwc/paper... foper_ffllee files/ (eteer_files//ppe//2220000 paper/2020/file// pdf
[50] Wang. K., Variengien, A., Conmy, A., Shlegeris, B., and Steinhardt, J. Interpretability in the Wild: Circuit for Indirect Object Identification in GPT-2 small, November 2022. URL attp://arxiv wwwpaawwwww...a..eeee.weeeeaeeee https//221..0055
0 00593 arXiv:2211.0000 [cs].
[51] Yun, Z., Chen, Y., Olshausen, B. A., and LeCun, Y. Transformer visualization via dictionary learning: contextualized embedding linear superposition of transformer factors, 2023.
[52] Zhang, and Nanda, N. Towards Best Practices of Activation Patching Language Mod- els: Metrics and Methods, January 2024. URL http:://rx/// org/ abs/ 2309. 2009.16042 arXiv:2309.1640 16042 [es].
15

---

A
Assets used
Table 2: Assets used in preparing this paper, along with licenses and links
Asset type
Asset name
Link
License
Citation
Code
TransformerLens
GitHub: TransformerLens
MIT
37
Code
SAELens
Github: SAELens
MIT
5
Data
OpenWebText
HuggigFace: Open Web
CCO-1.0
Model
GPT2-small
HuggingFace:: GPT2
MIT
44
Model
Pythia-410M
HuggingFace: Pythia-410M
Apache-2.0
Model
Pythia-1.4B
Huggingrace: Pythia-I .B
Apache-2.0
B Compute details
The most compute-intennte intensive parts of the research presented in this work were training the SAEs and transcoders used in Section 4.2 along with the set of GPT2-small transcoders used in Sections 5.1 3.2 Training all of these SA Es and transcoders involved GPUs. The SAEs and transcoders from Section 4.2 were trained
internal cluster using A100 GPU with 80 GB of VRAM. The VRAM used by each training run ranged from approximately 16 GB for the GPT2-small runs to approximately 60 GB for the Pythia-1.44 runs. The time taken by each training run ranged from approximately 30 minutes for the GPT2-small transcoders/S AES to approximately 3.5 hours for the Pythia-1.4B runs.
The transcoders that were trained on each layer of GPT2-small were trained using a cloud provider, with similar amount of time and VRAM used per training run. For these transcoders, hyperpa- rameter sweep was performed that involved approximately 200 training runs, which did not produce results used in the final paper.
No significant amount of storage was used, as datasets were streamed during training.
In addition to these training runs, our case studies were carried out on internal cluster nodes with GPUs. These studies used more than 6 GB of VRAM. The total amount of compute used during each case study is variable (depending on how in-depth one wants to investigate a case study), but de minimis comparison the training runs. The same goes for the computation of top activating examples used in Section 4.1
C SAE details
Sparse autoencoders (SAEs) are autoencoders trained to decompose model's activations at a given point into sparse linear combination of feature vectors. a hypothetical example, given the input "Sally threw the ball to me". an SAE might decompose the model's activations the token me into linear combination of "personal pronoun" feature vector, an "indirect object" feature, and a "playing sports" feature-whts all of these feature vectors automatically learned by the SAE. An SAE's architecture can be expressed as
ZSAE (x) ReLU (W enex + bane) SAE(x) W WeccRSAE + baec:
(10)
where Wene E I Redeemmeee deenaa the number of feature vectors in the SAF, and dmmosee is the dimensionality of the model activations. Usually, dieaturs is far greater than dmodel.
W dec E Rednss ddcsues bene E &Rdears, bdtc E Redass dieature is
Intuitively, Equation transforms the neuron activations into sparse vector of SAE feature activations ZSAE x). Each feature in an SAE associated with an "encoder" vector (the i-th row of W Wane) and "decoder" vector (the i-th column of W d. Equation 10 reconstructs the original activations as a linear combination of decoder vectors, weighted by the feature activations.
The basic loss function which SAEs are trained is
LS
x SAE()) + A [|))
(11)
reconstruction loss
sparsity penalty
16

---

where A1 is a hyperparameter and || || denotes the L1 norm. The first term in the loss is the reconstruction loss associated with the SAE. The second term in the loss is a sparsity penalty. which approximately measures the number of features active on each input (the T. norm is used as differentiable approximation the Lo "norm"). SAEs are thus pushed reconstruct inputs accurately with sparse number of features, with A controlling the accuracy-sparsin tradeoff. Empirically, the result this is that SAEs learn to decompose model activations into highly interpretable features 7
A standard method for quantitatively evaluating an SAE's performance is as follows. To measure its sparsity, evaluate the mean number of features active on any given input (the mean Lp). To measure its accuracy, replace the original language model's activations with the SAE's reconstructed activations and measure the change in the language model's 's loss (in this paper, this the cross entropy loss for next token prediction).
D Detailed description of circuit analysis
D.1
Notation
(lit) Xpre denotes the hidden state for token t layer / before the attention sublayer.
mid denotes the hidden state for token layer before the MLP sublayer.
(l.t)
When we want to refer to the hidden state of the model for all tokens, we will do by omitting the
(1.1+t) and x (1.1+t) These matrices of size Rd Nuaes, Nutms where dmodel the
token index, writing Xpre dimensionality of model activation vectors and Rookens is the number of input tokens.
The MI.P sublayer layer denoted by MTPP())). (.). Similarly, the transcoder for the layer MLP is denoted by T (.).
As for attention sublayers: following Elhage et al. 15 each attention sublayer can be decomposed into the sum Theaas independently-actinn hidden states of all tokens in the input, but also distinguishes the token whose hidden state is to be modified by the attention head. Thus, the output of the layer attention sublayer for token denoted
attention heads. Each attention head depends on the
E Enead& attn (I.k) (L.t) Xpre x(12:2))
Each attention head can further be decomposed as a sum over "source" tokens. In particular, the output of layer I attention head h for token can be written
attn (1,b) (xt x(brttt)
Σ tnken
score (x((ht) x((())
(1,,)) OV xpre
(12)
Here, (t.r) duode x Cundel + R is a scalar "scoring" function that weights the importance of each source token to the destination token. Additionally, Ov low-rank I rgmmas damti matrix that transforms the hidden state of each source token. store(I.h) is often referred the "QK cireuit" of attention and ((1h)) is often referred as the "OV circuit" of attention.
D.2 Derivation of Equation
We understand what causes feature i' the transcoder layer E to activate on token The activation of this feature is given by
ReLU
(f(L'1)) enc
x + (lt
Vene
(13)
where (ll,,'') is the i' row of W Wene for the layer l' transcoder and beno is the learned encoder bias for feature i' in the layer l' transcoder. Therefore, if we ignore the constant bias term ((',,)) then, assuming that this feature is active (which allows to ignore the ReLU), the activation of feature i' depends solely on r((','' ( (('+t)
(I',t)
mid Because of residual connections in the transformer, x! x can be decomposed as the sum of the outputs of all previous components in the model. For instance, in
17

---

two-layer model, if x (2.4)
is the hidden state of the model right before the second MLP sublayer, then
"mid"
Σ attn (2,h)
x pre
(2 (222.))) + MLP(1))
mid
(1.t)
+
Σ attn (1.R)
x pre
x pre
(14)
Because of linearity, this means that the amount that MTP((1) given by
(1.t) x mid
contributes to reac
x mid is
(2.t)
f(2.1' MLP(1)
(1.t) (
(15)
This is generally true for understanding the contribution of MLP to the activation of feature i' in transcoder I', whenever / < I'
Now, if the layer 1 transcoder is sufficiently good approximation to the layer / MLP, we can
replace the latter with the former: p("' MLL(()
x (l.t)
mid
s rll''a) .TC()
x X
(l.t)
We can further
decompose this into individual transcoder features: TC
X Exeature
(l.t)
2re (Xmid) dec
(1.) (l.t) (...
Thus, again taking advantage of linearity, we have
p()'.1) MMP(() ene
(l.t) (mid)
~ P((
Σ
(1j) TTC
(X )
(16)
feature j
(1.1 TC
mid
(l.t)
(f(''))
p(Li))
(17)
feature
Therefore, the attribution of feature in transcoder I on token is given by
(1j) TC
(ut)))
(r(l't")
(18)
D.3 Attribution through attention heads
So far, we have addressed how to find the attribution a lower-layer transcoder feature directly on higher-layer transcoder feature the same token. But transcoder features can also be mediated by attention heads. We will thus extend the above analysis to account for finding the attribution of transcoder features through the OV circuit of attention head.
As before, we want to understand what causes feature in the layer I' transcoder to activate on token Given attention head h layer with 1 << I',, the same arguments as before imply that the contribution of this attention head to feature i' is given by pll''i) at.n (xpra) (1.1:t) ) This can further be decomposed as
fene
Σ
score(!.)) (xflet) x((re) willh) ((L.))
(19)
source
((satt") T w (Ix) pre
(1.re
(1.b)
(1.t)
x(xxt)
(20)
pre
source token
((wwrr)
T p(''.')) fene
(1.)
(Lh)
(L.t) Xpre x(ee))
x
(L.s) pre
(21)
source tnken
From this, we now have that the contribution of token s at layer through head is given by
((wwa))
(1.)e (1.))
Xxre x(a))
ell'd'') fenc
Xpre
(22)
The next step is to note that Xpre can, in turn, be decomposed into the output of MLP sublayers (or alternatively, transcoder features), the output of attention heads, and the original token embedding. These previous-layer components affect the contribution to the original feature through both the QK circuit of attention and the OV circuit. This means that these previous-layer components can have very nonlincar effects on the contribution. We address this by following the standard practice
(1.s)
18

---

introduced by Elhage et al. [5]. which is to treat the QK circuit scores score as fixed, and only look at the contributions through the OV circuit. While this does prevent us from understanding the extent to which transcoder features contribute to phenomena such as QK composition, nevertheless, the OV circuit alone is extremely informative. After all, if the QK circuit determines which tokens information is taken from, then the OV circuit determines what information is taken from each token-ann this can prove immensely valuable in circuit analysis.
(l.t) (1x) Xprr Xpre
Thus, let us continue by treating the QK scores as fixed. Referring back to Equation 22 if y is the output of some previous layer component, which exists in the residual stream Xpre then the
contribution of y to the original transcoder feature i' through the OV circuit of layer I attention head h is given by y p', where
p' - score
(x.))) (1as) pre x
p, and
(23)
T eerd"
p (wvr)
(24)
One way to look this is that p' is a feature vector. Just like with transcoder features, the extent to which the feature vector p' is activated by a given vector y is given by the dot product of y and p'. Treating p' as feature vector like this means that we can extend all of the techniques presented in Section 3.2 to analyze p'. For example, we can take the de-embedding of p' to determine which tokens in the model's vocabulary when mediated by the OV circuit of layer I attention head h cause
layer I' transcoder feature to activate the most. We can also replace the r'l'''' term in Equation 7 with p' in order to obtain input-invariant and input-dependent information about which transcoder features when mediated by this OV circuit make the greatest contribution to the activation of layer I' transcoder feature i'. In this manner, we have extended our attribution techniques to deal with attention.
D.4 Recursing on a single computational path
At this point, we understand how to obtain the attribution from an earlier-layer transcoder fea- ture/attention head to a later-layer feature vector. The next step is to understand in turn what contributes to these earlier-layer features or heads. Doing so will allow us to iteratively compute attributions along an entire computational graph.
To do this, we will extend the intuition presented in Equation 23 and previously discussed by Dunefsky & Cohan [33, which
propagate our feature vector backwards through the computational path Starting at the end of the computational path, for each node in the computational path, we compute the attribution of the node towards causing the current feature vector activate; we then compute a new feature vector, and repeat the process using the preceding node and this new feature vector.
In particular, every node, want to compute the new feature vector f such that it satisfies the following property. Let e' be a node (e.g. a transcoder feature or an attention head), x' be the vector of input activations the node c' (i.e. the residual stream activations before the node c'), y' be the output of c', a' be the attribution of c' to some later-layer feature, and f' be the current feature vector to which we are computing the attribution of c'. Noting that a' f' yy, then we want our new feature vector f to satisfy
(25) This is because if f satisfies this property, then we can take advantage of the linearity of the residual stream to easily calculate the attribution from an earlier-layer component to the current node c'. In particular, if the output of is the vector y, then this attribution just given by f-y. Another important consequence of Equation 25 and the linearity of the residual stream is that the total attribution a' of node is given by
f f-x'=''
a' = d'=IA. f
(26)
where sum over all the outputs y of all earlier nodes in the model's computational graph (including transcoder features and attention heads, but also token embeddings and learned constant bias vectors, which are leaf nodes in the computational graph).
'The similarity to backpropagation is not coincidental, as it can be shown that the method about to be described computes the "input-times gradiennt attribution often used in the explanability literature.
19

---

If c' is attention head h in layer 1. and we are considering the contribution from the input activations (I.s) Xpre at source token position 8, then Equation 22 tells us that
f = score!l.h (1.k)
(l.t) pre 1 Xpre
(1x)) ((wor)
T f'
(27)
where token position & is the token position corresponding the later-layer feature f'. And if d' is transcoder feature i at layer 1, then Equation 18 implies that
f= (1i) dec
(28)
There is one caveat, however, that must be noted. Before every sublayer in the transformer architec- tures considered in this paper (that is, before every MLP sublayer and attention sublayer), there is a
LayerNorm nonlinearity. Neel Nanda [39] provides intuition that LayerNorm nonlinearities can be approximated as a linear transformation that scales its input by a constant; Dunefsky & Cohan [13 provide further theoretical motivation and empirical results suggesting that this is reasonable. We follow this approach in our circuit analysis by multiplying each feature vector by the appropriate LayerNorm "scaling constant" (which is empirically estimated by taking the ratio of the norm of the pre-LayerNorm activation vector to the post-LayerNorm activation vector).
D.5 Full cireuit-finding algorithm
At this point, we are ready to present the full version of our circuit-finding algorithm. The greedy computational-path-iiiee
algorithm is presented as Algorithm T This algorithm incorporates the ideas presented in App. D.4 in order to evaluate the attribution of nodes in computational paths; given a set of computational paths of length L,, it obtains a set of important computational paths of length L 1 by computing all possible extensions to the current length- L paths, and then keeping only the N paths with the highest attributions. Note that for the purpose of clarity, the description presented here is less efficient than our actual implementation; it also does not include the LayerNorm scaling constants discussed above.
Next, given a set of computational paths, Algorithm converts this set into single computational graph. The main idea is to combine all of the paths into a single graph such that the attribution of a node in the graph is the sum of its attributions in all distinct computational paths beginming at that node. Similarly, the attribution of an edge in the graph the sum of its attributions in all distinct computational paths beginning with that edge. This prevents double-counting of attributions. Assuming zero transcoder error, Equation Equation[6Gimpliess 26jimplies that in a graph produced by Algorithm [2 from the full set of computational paths in the model (including bias terms), the attribution of each node is the sum of the attributions of all of the incoming edges to that node. To account for transcoder error, and to account for the fact that not all computational paths are included in the graph, error nodes can be added to the graph, following the approach of Marks et
E Details on Section 4.2 SAE/transcoder training
In this section, we provide details on the hyperparameters used to train the SAEs and transcoders evaluated in Section 4.2
All SAEs and transcoders were trained with a learning rate of 2 10-5 using the Adam optimizer. Hyperparameters (learning rate and \₁ sparsity coefficient) were chosen largely based trial-and- error.
The loss functions used were the vanilla SAE and transcoder loss functions as specified in Section 3.1 and Appendix No neuron resampling methods were used during training.
SAEs were trained on output activations of the MLP layer. Transcoders were trained on the post- LayerNorm input activations to the MLP layer and the output activations of the MLP layer. We chose to train SAES on the output activations because when measuring cross-entropy loss with transcoders, the output activations of the MLP are replaced with the transcoder output; it is thus most valid to compare transcoders to SAEs that replace the MLP output activations as well.
The number of features in the SAES and transcoders was always 32x the dimensionality of the model on which they were trained. For GPT2-small, the model dimensionality is 768. For PYthia-410M, the model dimensionality is 1024. For Pythia- 1.4B, the model dimensionality is 2048.
20

---

Algorithm Greedy computational-path-ffnning
Input:
f'
A feature vector
The layer from which f' came.
The token position associated with feature f'.
The activation of f'
L
The number of iterations to pathfind for
N
The number of paths to retain after each iteration
The input prompt on which we will perform eircuit analysis Output:
A set computational paths important for causing f' activate Initialize p < {[(f", 1', aQ]} (P will be our working set of computational paths. Each computational path a list feature vectors paired with their attributions. Initialize Pout {} (This will contain output]
Run the model on the input prompt, caching all of its activations.
while L > 0 do
Initialize Pnext <- {} (This will contain the next iteration of computational paths) for each PEP do
Set feur: lour: feur: Ocur to the values in the last element of P Initialize A + (} (The set of attributions of all lower-layer features)
for each transcoder feature in layer where | < leur do
Insert ((four r(l.)) rel I, (rre(((lt)) (four -((1))) into A end for for each attention head layer at token where I < lour and I teur do flour (l.t)
Compute the attention score & score(l.)) (prre :xpre
Compute the feature vector fnew + ((War)
feur
(l.t) Compute the attribution Onew fnew Xpre Insert (fnew: 1, t, (new) into A
end for
Compute the embedding attribution Gembed &- feur x O,teur
pre
Insert (0, 0, feur. Gembed) into A for each fneww:lnwws knews (new) Ao do
if Onew is among the top N values of Onew contained in A then Append (Enew; Inewr knews (new) path P and insert into Pnezt end if end for end for
Remove all paths in Prest except for the paths where the attribution the earliest-layer feature vector in the path is among the top N in P Pnezt
Append all paths in Pnert out
P
-- P Prest
L L L
end while return Pout
The SAEs and transcoders trained on 60 million tokens of the OpenWebText dataset. The batch size was 4096 examples per batch. Each example contains context window of 128 tokens; when evaluating the SAEs and transcoders, we did so examples of length 128 tokens well.
The same random seed (42) was used to initialize all SAFs and transcoders during the training process. In particular, this meant that training data was received in the same order by all SAEs and transcoders.
21

---

Algorithm 2 Paths-to-graph
Input:
P A set of computational paths
Output: G = (V. E) A computational graph formed from the paths of P.
Initialize S - { {A set of already-seen computational path prefixes, to prevent us from double- counting attributions)
Initialize V -- {} [A dictionary mapping nodes to their attributions)
Initialize & <- [) [A dictionary mapping edges (node pairs) to their attributions]
for Each P in do
for E [P]] do
8 +- prefix of up to and including the i-th element
if x E then
Skip this iteration of the loop.
end if
Insert s into S.
if has length 1 then
Let n be the only node 8.
Set V/n] to the attribution of n.
else
Set Set Apaaeet Pli- 1]. Dehhld
P[] (Earlier-layer nodes come later in the computational
paths returned by Algorithm D
Add the attribution of nehitd to Vnniita
Add the attribution of Rehstd to 8 [echild: Ppprent)] end if end for end for return E
F Details on Section 4.1
The transcoder used in the interpretability comparison was the Pythia-410M layer 15 transcoder
trained with A sparsity coefficient 5.5 10-5 from Section 4.2 The SAE used in the comparison was a Pythia-410M layer 15 SAF trained on MIP inputs with À 7.0 x 10-5, We used an SAF trained on MI.P inputs rather than one trained on MI.P outputs (as $4.22 because the interpretability comparison involves looking at which examples cause features to activate. This, in turn, is wholly determined by the encoder feature vectors. Because the transcoder's 's encoder feature vectors live in the MLP input space, it is thus most valid to compare the transcoder to SAE whose encoder feature vectors also live in the MLP input space.
This transcoder-Se SAE pair was chosen because the transcoder and SAE sit at very similar points on the Lo-cross-ent entsopy Pareto frontier: the transcoder has an Lg of 44.04 and cross-entropy of 3.35 nats, while the SAE has an Lo of 47.85 and a cross-entropy of 3.36 nats. Pythia-410M was chosen as the model with the view that its features were likely to be more interesting than those GPT2-small, while requiring less computational power to determine top activating examples than Pythia- .4B would. Layer 15 was chosen largely heuristically, because we believed that this layer late enough in the model to contain complex features, while not so late in the model that features are primarily encapsulating information about which tokens come next.
In Table I we refer to "context-free" features that interpretable features that seemed to fire on single token (or two tokens) regardless the context in which they appeared. Examples of features in all four categories ("interpretable", "maybe interpretable", "uninterpretable" and "context-free") along with the exact annotation used by the human rater, can be found in Figure
G Details on Section 5.2
To obtain the de-embedding scores shown in Figures and 4 the following method was used. First, we used the method presented in Appendix [D.3] to determine which MI.P transcoder features had the highest input-invariant connections to the given MLP10 transcoder feature through attention head in
22

---

Percentagr band: 8.000%%
wass Emuupte 3501, troen
locess ie lele lesao Insail.
Esanple 1707, strre 30
Percenlage in band: 1.031%
appussiantedyy on
emosage Black 18
stnakee. Thee Ecappl coken Ltd
Dosnte 54750, vikas 116
aine EUEEII (Muat centic Trouggle
aus Wwl.iiii inaiiee
23-03
Doncalo 12855 Hore.
eir seace 130t Emagle 11930 tokus 95
Persstage in hand: 8.80799
cenria Iee wnamt Domrta 14333 raken
If sted
almeree Eoe Borter vistien The beopgle 2100
Tuseple 2364 (1
Zasaph okca 14
Percenlage
lot 100 saces 10 Space raaa. 1: karapl L1726, sdes 12
The Unappl 12552 intes 81
boutle Pustuge Ezanple
Perrple 1142, olbos 61
Lawent Daaret 11S1, tober 120
vilao
sniies Dasppl 2440, rikca
wi lnnaank wnde new thorraa 5S04, siles
Eusuple 23566 lakea 95
snatadl Bond soull thay Exarpll
Pensabge lead: 8.8150%
= Coune Dlay sen Dhorgia sebes $1
Peaegle 1995,
1858 Etunplle 9197, skks
Inanple 1335, tnken 125
PPrsealagg band: 0.1244
nerkad por F akmr
13724
Snasehe
12285 uabes?
Ines Kor Fouppl 2503, Hian 109 by skithany alueston
aeastionnllty modti Iharppl 1559, tuus
1735, idus
and Zusapk rales
tokess
Esingl 12302, skes
you. stariliat beople 1755
(a) acttvvaii examples for feature annotated as "interpretable". The specific annotation was local context feature, fires on phrases describing short amounts of time.
sicchtr bet buaayle (11) Mukurt
(b) Top-activatie activating examples for a feature anno- tated as "mayhe interpretable". The specific
annotation was local context feature for boredom? MAYBE.
band:
Ehamppe 1525, taken
cienecteant bebdiggseeee yau Exasple d667, tolend
Letto Saage eeleeendca bullicaae demuzted Exaeple 5234, tolen 564
selling H1-ther Faveple 1977) tille ei
bonali mank 1eE sptuc trpeed Poonk 6266,
Priece
Percentage In baDé: 0.0804%
Erasaly 510% Idoo "Ska Agu cf 204 solee. es in Eosaal 1754, siece 10l Pererntage
Excap (666, Mloe ST
Maed Coegin (Matt Tusl Peueri Na.. Desaaa 12535 robel
Peoelle
Ihampla sales 22
2nd Reselle 1580 sieee desteer Domppl 18337 vikes 72
seevicee Exapll (7551 kdea 15
and Suckeden Peange 18324 maber
Bay Ebangta 3081 odks 128
<doec)
colme Anauple 4395, okez
Enaaapl 1953, Idacn
Pereentage in hand: 0.9800%
kastided or. 1100 dellee and Enaapplo 18469 idees 56
incrrpee secrice 11 steeall Insasiice chypo, Escay's 1799) vales 106
tommediity Merrrmalio and Frnaple 15517,
33455
lumppl 4851, 104
cemputvll alyyy msngli Phoaapa 7117, vilee
Iisands foe aeceiigll stete Exazphe adks 118
Prrcentage band: (ianir eay teaey Lo Ltumgl 15359 solce
for westrral boongte 4520,
masii tie vasilinat Wasl baagge 1555. ooken 100
Morieeaayy
180 aagullace bellife: Sex Eseerk 21316, Vdice 116
Perrentage in hand: 0.9810%
Our poass Exarpple 1506), 21
by 15,
Conmary Daapll 3376, bdas 45
LiEtuzple 7600 tokmn-7 47
Rance simee deloo soili Exagge: 1795,, skea
the soragee caree Jec. alone Enrage 2235, cokca 125
Eacegle
company Demesk auga: Enorapl 2655 25
Vinis 349), taber
watobing? Yeusk Dosrst (1001, sdie 109
Hole: plownd by
booleg midl Faueela 1417, orkes 100
loon. Creezo volonne a550 Iompple iiies
den "Faeete Thargla 12445 cokan
(c) Top-activating examples for a feature an- notated as "uninterpretable". The specific annotation was " Whats" "ADVERTISEMENT Thanks" "olog" NOT INTERPRETABLE.
(d) Top-activating examples for feature annotated as "context-free"" The specific annotation was "oc" in middle of words single-toke feature.
Figure 6: Examples of ""eature-dashboards" used in the feature interpretation experiments.
layer 9. Specifically, for MI.PO transcoder feature and MIP10 transcoder feature j. this attribution is given by (FlOd) T ((())) ((10J). For each MLP10 transcoder feature, the top ten MIPO fenc
transcoder features were considered. Then, for each MLPO transcoder feature, the de-embedding score of each YY token for that MLPO feature was computed. The total de-embedding score of each YY token for an MLPI0 feature was computed as the sum of the de-embedding scores of that token over the top ten MLPO features, with each de-embedding score weighted by the input-invariant attribution of the MLPO feature. In Figures 5 and the de-embedding scores were scaled and recentered in order to fit the graph.
The mean probability difference metric discussed in the original greater-than work is as follows. Given the logits for each YY token, compute the softmax over these logits in order to obtain probability distribution over the YY tokens; let Py denote the probability of the token corresponding to year y- Then, the probability difference for given prompt containing certain input year y is given by
23

---

EysyPy Py Py Eyys Pv. The mean probability difference is the mean of the probability differences over all 100 prompts.
H Full case studies
H.1
Classic blind case studies
H.1.1
Citation feature: tc8 [355]
First, checked activations for the first 12,800 prompts in the training data. Using this, we identified the prompt indexed at (5701,33 37) as one of prompts for which tc8 [355] activated above a score of 11.
Path-based analysis on input index (5701, 37) revealed contributions from various tokens, notably attn7 [7] @35 and attn5 [6] @36. However, we first decided to focus on the current token.
Current-token features. Top de-embeddings for both tc0 [9188] and tc0 [16632] were all variants of semicolon: 5. ; %;. and i. We also checked tc6 [ 1831] @-1 and found that its top contributing features from layer were tc0 [16632] and tc0 [9188]: the same two semicolon features. On the basis of this, we concluded that the final token is semicolon.
Surname features. Next we focused attn7 [7] @35. Some interpretable features with high attributions through this component included tc0 [13196] #36 (years), tc0 [10109] @31 (open paren- theses), m1p8tc [355] attn7 [7] [1] #35 (components of last names), tc0 [12584] 632: and tc0 [7659] 634: ck
Input-independent investigation of tc6 [21046] 635 revealed high contributions from tc0 [16382] and tc0 [5468]. feat016388 corresponded to tokens such as oglu owski, and zyk; tc0 [5468] corresponded to tokens such Burnett, Hawkins, and MacDonald Observing that all of these are (components of) surnames, we decided that token 35 was likely (part of) a surname.
Repeating analysis with prompt (6063, 47). Top attributions for this prompt once again identified tc0 [9188], the semicolon feature from earlier. We filtered our computational paths to exclude this transcoder feature, since already had a hypothesis about what was doing. This identified tc0 [10109] #39 and tc0 [21019] 846 as top-contributing features.
The top de-embedding tokens for tc0 [10109] 039 were (, (5, and ( On the basis of this, we determined that token 39 was likely an open parenthesis. Meanwhile, the top de-embedding tokens for tc0 [21019] #46 were 1983, 1982 and 1981. This caused us to conclude that token 46 was likely a year.
We noted that, in the previous prompt, the attribution for the year features went through attn5 [6], whereas the behavior of attn5[6] tc0 [16542] @11, tc0 [c0[42 011, and tco [19728] @11. The de-embedding results for these were features, whereas tco [4205] included citation-
this prompt it went through attn2[9], We decided investigate
this prompt, and found that it was attributing to features
mixed: tc0 [16542] were both related tokens like Accessed, Neuroscience, and Springer.
Final result. We decided that tc8 [355] was likely semicolon-in-ciaation feature and looked at activating prompts. Top-activating prompts included "Res. 15, 241-247; 1978). In their paper, "aythamah. 2382; Tahdhib al-", and "lesions (Poeck, 1969; Rinn, 1984). It". Note that the last of these was prompt (5701, 37), i.e. the first case study considered.
In general, the top-activating features corroborated our hypothesis, and we did not find any unrelated prompts. We noticed that many of the top activating prompts had comma before the year in citations, but our circuit analysis never identified a comma feature.
We compared transcoder activations on the prompts "(Leisman 1976;" and "(Leisman 1976:" and found tc8 [355] to activate almost identically for both when all preceding MIPs were replaced by transcoders (4.855 and 4.906, respectively) and on the original model (12.484 and 12. 13, respectively).
24

---

H.1.2 "Caught" feature: tc8 [235].
First, we checked activations for the first 12,800 prompts in the training data. Using this, we identified prompt (8531, 111) as one of 13 prompts for which tc8 [235] activated above a score of 11.
Imput (8531, 111). Path analysis revealed that this feature almost exclusively depends the fi- nal token in the input. Input-independent connections to the top-contributing transcoder feature, tc7 [14382], revealed the layer-O transcoder features tc0 [1636] (de-embeddings: caught, aught) tc0 [5637] (de-embeddings:: captured, caught), tc0 [3981] (catch, catch) as top contribu- tors.
Inputs (6299, 39) and (817, 63). For input (6299, (9),, we again saw top computational paths depended mostly the final token. This time, identified tc7 [14382] and tc0 [1636] both of which were already identified for the previous prompt-as top contributors.
For input (6299, 39) we also observed the same pattern. This caused us to hypothesize that this feature fires on past-tense synonyms of "to catch."
Final result. Top activating prompts for this feature were all forms of "caught," but the various synonyms, such as "uncovered,"" were nowhere to found.
"Caught" as participle. Additionally, we noticed that "caught" was used as participle rather than finite verb in all top-activating examples. To explore this. we investigated the difference in activations between the prompts "He was caught" and "He caught the ball",, and found that the former caused tc8 [235] activate strongly (19.97) whereas the latter activated very weakly (0.8145).
When we tested the same prompts while replacing all preceding MLPs with transcoders, we found the difference much less stark: 16.45 for "He was caught" and 9.00 for "He caught the ball", This suggests that transcoders were not accurately modeling this particular nuance of the feature behavior.
Finally, checked top paths for contributions through the was token the prompt "He was caught" to see whether we could find anything related to this nuance in our circuits. This analysis revealed attnl [0] 02 important, and were able to discover mild attributions to transcoder features whose top de-embeddings were was and related tokens.
H.2 Restricted blind case studies
Beyond simple blind case study, we carried out number of "restricted blind case studies." In these, all of the rules of regular blind case study apply, and additionally it is prohibited to look at input-dependent information about layer-0 transcoder features.
Since layer 0 features are more commonly single-token features, and in general there is almost no contextual information available for the MLP yet, layer features tend be substantially more informative about the tokens in the prompt than features in other layers are. Thus, it is often possible reconstruct large portions of the prompt just from the de-embeddings of which layer 0 transcoder features are active-and, and, although we never look at these activations directly, they are frequently revealed and analyzed as part of active computational graphs leading to some downstream feature.
By omitting input-dependent information about layer 0 features from our analysis, we must rely more on cireuit-level information, and remain substantially more ignorant of the prompts for activating examples. Note that input-ine inppe-indeeendentt information about layer 0 features can still be used: for instance, we can look at top input indepeenenndd connections layer 0 features, and the de-embeddings for those as well at the expense of not knowing whether those features are active or not.
H.2.1 Local context feature: tc8 [479].
Our first example a blind case study follows tc8 [479], which we fail to correctly annotate through circuit analysis. We include this case study for transparency, and as instructive example of how things can go awry during blind case studies. First, we measured feature activations over 12,800 prompts and identified 6 prompts that activated above a threshold of 10.
25

---

Input (3511, 64). For this prompt, path analysis revealed lot of attention head involve- ment from many previous tokens. For our first analysis, we chose the path m1p8tc [479]] @-1 8.1 <- mlp7tc [10719] @62, since we could look at de-embeddings for tc7 [10719] @62. Top input-independent connections from tc7 [10719] @62 to layer 0 were tc0 [22324] and tc0 [2523], which had estimated and estimate as their top de-embeddings, respectively. Thus, we hypothesized that token 62 is "estimate(d)".
[479]
<- attn8 [5] @62:
Next, we looked at the pullback of tc8 [479] through attn8 [5] through attn? [5] 657. This revealed top imput-nndependnet connections to tc0 [23855] (top de-embedding tokens: spree. havoc, frenzy). tc0 [8917] (took de-embedding tokens: amounts, quantities, amount), and [327] (massive. massive, huge). We found this aspect of the analysis to be inconclusive.
The pullback of tc8 [479] through attn [5] through attn6 [11] @57 revealed connections to tc0 [13184] (total), tc0 [12266] comparable)) and tc0 [12610] averaging). This led us to believe that token 57 relates to quantities.
We found that tc3 [18655] was a top transcoder feature active on the current token. This showed top input-independent connections to tc0 [11334] and tc0 [5270], both of which de-embedded as be. This led us to hypothesize that tc8 [479] features on phrases like "the amoundtotallaveeaaee is estimated to be...".
Input (668, 122). For this prompt, most contributions once again came from previous tokens. The top contributor was attn attn8[5]@12) @121, which had input-independet connections to tc0 [12151] ( airport), tc0 [8192] (pired), tc0 [13184] (total), and tc0 [1300] ( (resulted). This was inconclusive, but this is the second time that tc0 [13184] has appeared in de-embeddings.
Next, we investigated attnr [7] @121: it connected to tc0 [16933] population), tc0 [14006] (kinson rahim LU, ..... tc0 [19887] ( blacks), and tc0 [6821] ( crowds). These seemed related to groups of people, but this analysis was also inconclusive.
When we investigated tc4 [18899] @121, top input-idependent connections to layer-0 features in- cluded tc0 [22324]. which de-embedded to estimated again. This was more consistent with the behavior on the previous prompt.
To understand the current-token behavior, we looked at tc7 [13166] @-1. Top input-indendent connections were tc0 [18204] discrepancy) and tc0 [14717] velocity). tc1 [19616] @-1 and tc3 [22544]-1, -1, both of which also contributed, each had top connections to tc0 [19815] ( (length). This led us guess that this prompt relates estimated length.
Next, we looked at previous tokens. One feature, tc5 tc5[10350] @119, was connected to tc0 [23607] and tc0 [4252], both of which de-embedded to variants of With For the next token, tc6 [15690] @120 was connected to tc0 [22463] and tc0 [18052] (both a). This updated our hypothesis something like "with estimated length."
Further back in the prompt, we saw tc4 (23257]] 629 (connected to tc0 [12475]: remaining tc0 [16996]: entirety).
Input (7589, 89). One feature, tc7 [6] 887, pulled back to tc0 [22324], which de-embedded to estimated. A following-token feature, tc1 [14473], pulled back to tc0 [4746] annual, yearly), and the next-token feature tc1 [12852] 089, pulled back to tc0 [923] revenuu). Thus, this prompt seemed to end in "estimated yearly revenue."
Estimates for earlier tokens included tc4 [23699] @85 (tc0 [10924]: with), tc5 [6568] 686 (tc0 [1595]: a). This matched the pattern from earlier, where we expected a prompt like "with an estimated length" now we expect "with an estimated annual revenue."
Looking at the pulled-back feature mlp8tc [479] attn3 [2] 686, none of the connections we found to be very informative. This is consistent with patterns observed in other case studies, where pullbacks through attention tended to be harder to interpret.
Final guess. On the basis of the above examples, we guessed that this feature fires on prompts like "with a total estimated... When we viewed top activating examples, we found number of examples that matched this pattern, especially among the highest total activations. However, for many of the fowest-activation prompts saw quite different behaviors. Activating prompts revealed that
26

---

this is a local context feature, which in retrospect may have been apparent through the very high levels of attention head involvement in all circuits we analyzed.
H.2.2 Single-token Al1 feature: tc8 [1447]
An analysis of the first 12,800 prompts revealed 21 features activating above threshold of 11. One of these was input (3067, 79). The computational paths for this prompt revealed all contributions came from the final token.
The top attribution was due to tc7 [10932], with top input-independent connection tc0 [4012], which de-embedded to All. The next-highest was tc6 [8713), which connected to tc0 [6533], which de-embedded to All (note the leading space). These observations led to hypothesize this probably simple, single-token feature for "All."
We also looked at context-based contributions by filtering out current-token features, and found the top attributions to max out at 0.23 (compared to 3.5 from tc7 [10932] 079). This was quite low, indicating context was probably not very important. Nevertheless, tc8 [1447] through the OV circuit of attn4 [11] @78 and discovered several seemingly-unrelane connections with low attributions. When we pulled back through the OV circuit of attni [1] @78 and attn2 [O] 678, both showed input-independnt connections to features that de-embedded as punctuation tokens. Overall, the context seemed contribute little, except suggest that there may be punctuation preceding this instance of A11.
explored the pullback of
We repeated this analysis with another input, (8053, 72), and found the same features contributing: tc7 [10932], followed by tc6 [8713]. This led us to conclude this is single-token "All" feature. Top activating examples confirmed this: the feature activated most highly for All. then All and finally all Overall, this feature turned out to be quite straightforward, and it was easy to understand its function purely from transcoder circuits.
H.2.3 Interview feature: tc8 [6569]
For this feature, we found 15 out of 12,800 prompts to activate above a threshold of 16.
Input (755, 122). We started by exploring input (755, 122). which revealed several contributions from other tokens.
We began by looking at components that contributed to the final token. The top feature was tc7 [17738], which connected tc0 [15432] (variants of intervies), tc0 [12425] (variants of interviewed), and tc0 [12209] (tokens like Transcript, Interview, and The next feature, tc3(11401], was connected to tc0 [15432] and tc0 [12425] (same as the previous), well as tc0 [21414], which de-embedded to variants of spoke. This raised the possibility that "interview" is being used as a verb in this part of the prompt.
Next, we turned our attention previous tokens in the context, hopes that this would clarify the sense in which "interview" was being used. The top attribution for the previous token (121) was through attn4 [11]. The de-embeddings for top input-independent features were uninformative: tc0 [22216] seemed to cover variants of gest), while tc0 [7791] covered variants of sector. For token 120, pullbacks through attn2 [2] showed connections to te0 [10564] and tc0 [9519], both of which de-embedded to variants of In. This led to believe "interview" was in fact being used as noun, e.g. "in an interview..."
The top attribution for token 119 came through attn4 [9], and showed connections to:
te0 [625]: allegations, accusations, allegation..
tc0 [10661]: allegedly, purportedly, supposedly, .... and
tc0 [22588]: reportedly, rumored, stockp.
The next-highest attribution came through attn8 [5], and showed connections to:
tc0 [4771]: Casey, Chase, depot, ..., and
tc0 [5436]: di didn, wasn...
27

---

The next-high exxt-highest was tc2 [6264] 0119, which showed connections to:
tc0 [5870]: unlocks, upstairs, downstairs.
tc0 [14674]: said and variants, and
tc0 [12915]: said and variants
This led us to believe that this feature fires on "said in an interview" -rype prompts.
Input (1777, 53). Next tried another prompt. (1777, 53). The top features for the current token were identical to the previous example: tc7 [17738], tc3 [11401], tc6 [24442], and so on.
For the context, we first looked at the pullback of our feature through the OV circuit of attn2 [2] @51. This showed input-independent connections to tc0 [10564], which once again de-embedded to In Next up, attn4 [9] 850. This feature connected to tc0 [625], tc0 [10661], and tc0 [22588), exactly like before. Recall that these features de-embed to "said" and "allegedly" "ttpeddlyy"typp -type tokens.
Finally, we saw a high attribution from a much earlier token via attn8 016. The pullback of our feature through this head showed high inpu--independent connections tc0 [14048], whose de-embeddings were all variants of election.
Input (10179, 90). For our last input, once again found the same transcoder features contributing through the current token. For earlier tokens, we tried:
attn2 [2] 688, finding cc0[10564) (In) again;
attn8 [9] 686, finding tc0 [16885], which also dc-embedded elections despite being new feature;
attn6 [20291] 686, finding tc0 [372] ( (told); and
tc6 [20291] 286, finding tc0 [372] again.
Final guess. In sum, we decided this feature fires for prompts conveying "told/said in an interview." Top activating examples corroborated this, without any notable deviations from this pattern.
H.2.4 Four more restricted blind case studies
We present the results of four more restricted blind case studies in Table In the interest of conserving space, only the results of these case studies are presented. However, in the supplemental material attached to this submission, the original Jupyter Notebooks in which the case studies were carried out are provided.
Table 3: The results of four more restricted blind case studies.
Feature tc8 [9030]
Final hypothesis
Actual interpretation
Outcome Failure
Fires on biology when in the context of being subject of study
Fires on scientific subjects of study like chemistry. psychology. biology.
economics Fires on though or although in the beginning of a clause
tc8 [4911]
Fires on though or al though in the beginning of clause
Success
tc8 [6414]
Largely uninterpretable fea- ture that sometimes fires on Cyrillic text
Largely uninterpretable fea- ture that sometimes fires on Cyrillic text
Success
tc8 [2725] Fires on phrases about not fering things or not providing things. (As a stretch: particu- larly in legalese context?)
Fires on phrases about of- fering things not providing things, general
Mostly success
28

---

Table 4: The results of all blind case studies.
Feature tc8 [355]
Type
Final hypothesis
Actual interpretation Fires on semicolons in the context of academic citations Single-token "All" feature
Outcome
Blind
Fires on semicolons in the context of academic citations Single-token "All" feature
Success
tc8 [1447]
Restricted Blind
Success
tc8 [6569]
Restricted Blind
Fires on prompts conveying "told/said in an interview"
Fires on prompts conveying "told/said in interview" Fires on though or al thouguh the beginning of a clause
Success
tc8 [4911]
Restricted Blind
Fires on though or al thoughhhi in the beginning of clause
Success
tc8 [6414]
Restricted Blind
Largely uninterpretable fea- ture that sometimes fires on Cyrillic text
Largely uninterpretable fea- ture that sometimes fires on Cyrillic text
Success
tc8 [235]
Blind
Fires past-tense syn- onyms of "to catch"
Fires forms of "caught"
Mostly success
tc8 [2725]
Restricted Blind
Fires on phrases about not of- fering things or not providing things. (As a stretch: particu- larly in legalese context?)
Fires on phrases about of- fering things or not providing things, general
Mostly success
tc8 [479]
Restricted Blind
Fires on prompts resembling "with total estimated...
A local context feature
Failure
tc8 [9030]
Restricted Blind
Fires bi biology when in the context of being subject of study
Fires scientific subjects of study like chemistry, paychology. biology, economics
Failure
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The central claims of our paper are (1) transcoders enable a novel form of circuit analysis, (2) transcoders are faithful and intepretable approximation of MLP sublayers, and (3) transcoder circuit analysis can be applied to interpretability tasks. We
Justification:
demonstrate (1) in Section 3.2 (2) in Section 4 and (3) in Sections [5.1 and 5.2] These
arguments are consistently developed throughout the abstract, introduction, and conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include a limitations discussion towards the end of Section 7 before the Broader Impacts statement.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
30

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Only Equation in paper can be considered theoretical result. The derivation of this equation Appendix Dispells out all assumptions and serves as its proof.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose all experimental details, provide detailed descriptions of algorithms, and provide code for maximal reproducibility.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
31

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide open access to our data and code. We include links to the Github repo for the code and trained transcoders.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/publlt wwtppppppcpc/guiee.sssppppppppee httlc//guiides/oeeeuuubbiiieee hwwp///SSwiiiiiionnoooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In addition to describing key experimental details in the body of our paper, we include Sections E and G with more fine- grained details of our experimental setup. Additionally, all experimental details are contained in the code provide.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: All numbers reported in plots are point estimates, and therefore do not lend themselves to uncertainty quantification.
Guidelines:
32

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are described in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpt:t//eurriipttttee http::::/.ne..eeeeeeeeeeee httpr/pp.....pppllletteecceeeee.
Answer: [Yes]
Justification:
We have reviewed the linked Code of Ethics and determined that our paper is
compliant.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
33

---

Justification:
We include a broader impacts statement at the end of our conclusion, Section 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We did not find cause believe methods high risk for misuse, and therefore did not feel that additional safeguards were warranted.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: These assets are tabulated in Appendix A with the type, name, link, and citation for each asset used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
34

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide key assets: (1) a Github repo containing code for transcoder training, evaluation, and circuit analysis; and (2) HuggingFace repo containing trained transcoders for every layer of GPT2-small, The Github repo linked in our abstract, and includes link to the trained transcoders.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments were performed. (The human rater who carried out the qualitative interpretability assessment in Section 4.1 was one of the paper authors, although care was taken in the design and execution of this experiment to ensure that no authorial bias would influence the results. See Section 4. for more details.)
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
35

---

Answer:
[NA]
Justification:
This paper does not involve human subjects see item 14.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "LUDzeaa6na", "reviewer": "Reviewer_Q9a1", "summary": "Motivated by prior claims on how MLP sublayers make interpretability challenging (arguably due to their extremely dense nature), this paper proposes \"Transcoders\", a protocol that is in spirit similar to Sparse Autoencoders (SAEs). Specifically, a Transcoder aims at \"faithfully\" matching the output of a layer (in this case, an MLP layer); in contrast, an SAE aims to reconstruct the input to a layer. Results show Transcoders are similarly effective when compared to SAEs (in terms of model loss). Authors follow a blind evaluation pipeline to see if Transcoders yield interpretable features, whereby hypotheses are made for what a feature means and then posthoc the hypothesis is tested by looking at input samples.", "questions": "- From what I can gather, the \"de-embedding\" tool is equivalent to logit lens? Can you expand on what's different? I understand this is not a core contribution; just wondering why a different term was used.\n\n- I am still struggling to understand the benefit of Transcoders over SAEs. Can you expand on this more? It's good to define alternative tools, but unsure if this was a sufficiently different tool and led to much benefits.", "limitations": "See weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I enjoyed reading the paper. It's written fairly clearly and the targeted problem is well described, though some more expansion on how MLPs are a challenge for interpretability would help (see weaknesses). The blind evaluation protocol was really good to see---I appreciated authors' efforts in stress testing the limits of their approach (this is how good science should go :)).", "weaknesses": "While I like the paper, I think it needs a bit more work to be ready for acceptance. To that end, following are a few suggestions. \n\n- **Expand on challenges in MLP interpretability.** This is the bulk of the motivation of the paper and, from what I can gather, is grounded in off-hand references in two prior works. I don't think those works thoroughly describe what the challenges in fine-grained circuit analysis of MLP sublayers are. It is fine to use those works as motivation, but I think the paper should arguably start off with an experiment to demonstrate the challenge more thoroughly, given that prior work hasn't done that. In fact, I would argue that your own results with SAEs show similar interpretability may be achievable as found via Transcoders (since SAEs perform similar in terms of model loss). From that perspective, it isn't established if the question being addressed is sufficiently a challenge \n\n- **Analysis is correlational, lacking a quantitative confirmation and causal interventions.** While I really like the blind evaluation pipeline, there are two missing elements in the results per my opinion. These render the current analysis somewhat informal. *First*, once a hypothesis is formed, one should ideally run an algorithmic test to see on what proportion of a dataset where the hypothesis predicts a feature will activate does the feature activate. From what I gathered, the results merely involve \"eyeballing\" a few inputs to see whether the hypothesis holds. *Second*, the experiments are correlational, not causal. Ideally, one would systematically intervene on the input and see if the claimed hypothesis breaks in a predictable way. For example, when one identifies a feature activates for semicolons being present in the input, then if that input is kept the same but the semicolon is dropped or another relevant punctuation is added, does the feature not activate or at least activate less? \n\n- **Figure 2.** I found Figure 2 to be rather unintuitive. The axes labels are quite small, so I thought the loss increases along y-axis and hence Transcoders underperform SAEs, but then realized that the axis is decreasing along y-axis. I would have preferred a more standard plotting schematic, or at least larger labels to help avoid misinterpretation.\n\n\n**Post rebuttals update.** Taking into account the context provided by the authors and reading through the paper again, I am happy to raise my score. My primary concern was the paper was pitched to solve a very specific problem, i.e., interpreting MLPs. Authors' response indicates this was merely part of the problem they set out to address: identifying circuits in an input-invariant manner, since the interaction between circuit discovery and input-sensitive features makes that pipeline difficult. Overall, I understand the pitch much better now. The paper intro (and other relevant parts) should be updated to accurately reflect authors' motivation. I do not see these rewrites as a major challenge, so I'd be happy if the paper is accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**TL;DR**: Thank you for taking the time to write this detailed review. We are glad to see that you enjoyed the presentation of our work, along with our interpretability study and blind case studies. Our initial paper was insufficiently clear that the main goal of transcoders is to enable input-invariant circuit analysis, which is impossible with any existing SAE-based feature circuit methods. We believe that most of your concerns stem from this, and as such, if we have addressed them, we hope that you might be willing to raise your score.\n\nWe’ll now respond inline to specific concerns:\n\n>Expand on challenges in MLP interpretability. This is the bulk of the motivation of the paper and, from what I can gather, is grounded in off-hand references in two prior works. I don't think those works thoroughly describe what the challenges in fine-grained circuit analysis of MLP sublayers are. [...]\n\nThank you for pointing this out---we realize now that we might not have properly explained the motivation behind the problem. In essence, **the primary problem in fine-grained circuit analysis through MLPs is a lack of input-invariant methods for distinguishing between local and global behavior**. The two prior works that we cited represent different attempts at performing this fine-grained circuit analysis, both of which fail to address this problem. Marks et al. 2024 use causal methods (such as attribution patching), which only yield the importance of features on a single input. Dunefsky et al. 2024 attempt to approximate MLPs with their gradients, but their method has the same shortcoming, because MLP gradients are very different on different inputs. To our knowledge, no current works on performing circuit analysis involving MLPs with SAEs avoid this problem, and we fear that this is an issue inherent to all such methods.\n\nTo make all of this clearer to future readers, we plan to add to the paper an explicit, mathematically-motivated explanation of how these SAE-based methods fail.\n\n>In fact, I would argue that your own results with SAEs show similar interpretability may be achievable as found via Transcoders (since SAEs perform similar in terms of model loss). From that perspective, it isn't established if the question being addressed is sufficiently a challenge.\n\nWe hope that it is now clearer, in the context of our answer to the previous question, why this is not the case. In particular, since our goal is to use transcoders to perform input-invariant circuit analysis, we view it as a positive that transcoders and SAEs have similar interpretability/model loss – *because this means that we can substitute transcoders for SAEs and obtain all of the benefits of input-invariant circuit analysis without incurring any penalties*.\n\n>Analysis is correlational, lacking a quantitative confirmation and causal interventions. While I really like the blind evaluation pipeline, there are two missing elements in the results per my opinion [...] First, once a hypothesis is formed, one should ideally run an algorithmic test to see on what proportion of a dataset where the hypothesis predicts a feature will activate does the feature activate.\n\nThe main reason why we did not perform any algorithmic tests of our feature hypotheses is that the features that we were investigating were mostly too complex to admit any such proxies. Whereas, for example, the original Towards Monosemanticity paper was able to cherry-pick features (such as a base64 feature or Arabic feature) that could easily be captured in such a manner, our blind case study pipeline meant that we did not know if a feature would be easily “proxied” until after having carried out the entire case study.\n\nIndeed, while proxies can be useful for quantitatively evaluating the interpretability of an individual feature, it is worth noting that our goal with blind case studies was broader than this: we view blind case studies as an example of a nontrivial circuit-level interpretability task. To that end, we think that blind case studies act as a useful evaluation of how well a circuit analysis method would fare in a more real-world setting.\n\n>Second, the experiments are correlational, not causal. Ideally, one would systematically intervene on the input and see if the claimed hypothesis breaks in a predictable way. For example, when one identifies a feature activates for semicolons being present in the input, then if that input is kept the same but the semicolon is dropped or another relevant punctuation is added, does the feature not activate or at least activate less? \n\nAlthough space constraints prevented us from highlighting this in the main body, we have performed such tests of our hypotheses which can be found in the appendices. Please refer to the end of Appendix H.1.1 and Appendix H.1.2 for examples of this in two of our case studies.\n\n>Figure 2. I found Figure 2 to be rather unintuitive [...]\n\nThank you for pointing this out. In general, the orientation of our axes was inspired by papers such as the DeepMind Gated SAEs paper. However, we didn’t realize that our axis label sizes would cause a problem, and as such, we will make them bigger.\n\n> From what I can gather, the \"de-embedding\" tool is equivalent to logit lens? Can you expand on what's different?\n\nThe de-embedding is a “reverse” logit lens: in the de-embedding, we multiply the encoder vector of the feature by the transpose of the model embedding matrix, in order to understand which tokens in the input vocabulary space cause the feature to activate.\n\n>I am still struggling to understand the benefit of Transcoders over SAEs. Can you expand on this more? It's good to define alternative tools, but unsure if this was a sufficiently different tool and led to much benefits.\n\nWe hope that the preceding discussion has clarified our position on this, which is that we think **the main benefit of transcoders is the input-invariant circuit analysis they enable**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZbxSIl1xFj", "reviewer": "Reviewer_xRyp", "summary": "Sparse Autoencoders (SAEs) have been used for interpretability of transformer neural networks. SAEs take the output activations of MLPs within transformers and learn are trained to re-construct those outputs: $SAE(f(x)) \\approx f(x)$. In this submission, the authors study using transcoders for a similar task. Unlike SAEs, transcoders are trained to imitate the MLPs directly: $TC(x) \\approx f(x)$. The authors demonstrate via experiments and blind trials that the performance of transcoders is competitive with and sometimes exceeds that of SAEs.", "questions": "- What is the effect of different levels of $L_1$ penalty on the features that the transcoder learns? Do transcoders also experience \"feature splitting\" as the width of the transcoder goes up?\n\t- Are transcoders easy to train? Are the learning dynamics stable? Do they experience issues with \"dead neurons\"?\n\t- Does the input invariant / dependent construction not work for the features that SAEs learn? \n\t- When looking at \n\t- Do SAEs and transcoders trained on the same layer and same data learn the same or similar features? \n\t- Have any attempts been made to perform a similar analysis for other circuits? e.g. the [IOI](https://openreview.net/forum?id=NpsVSN6o4ul) or [Gendered Pronoun circuits](https://cmathw.itch.io/identifying-a-preliminary-circuit-for-predicting-gendered-pronouns-in-gpt-2-smal)?", "limitations": "The authors adequately address the submission's limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Transcoders are a natural and unexplored extension of the SAE architecture that perform very comparably to traditional SAEs.\n- Deriving the decomposition into input dependent / invariant features is novel and addresses a significant challenge of interpretability. On its own this warrants future exploration.\n- The feature case studies are a valuable contribution to the literature.\n- Testing interpretability via blind feature tests is innovative.", "weaknesses": "- The concrete comparisons to traditional SAEs are not sufficiently thorough. Experiments demonstrate that the KL divergence of transcoders are competitive compared to SAEs, but my understanding of the SAE literature is that it is still an open question as to how to judge the quality of an SAE. There are, however, other questions that are more concrete but are unaddressed. e.g. how easy/stable are transcoders to train compared to SAEs? Do transcoders suffer issues with \"dead neurons\" when the width is scaled up? Do transcoders show similar \"feature splitting\" effects when the width is scaled up? A more holistic comparison to SAEs would strengthen the submission significantly.\n- While performing a blind test of the interpretability of different features is novel, I worry that it is prematurely rigorous.\n- One of the implicit assumptions of the blind interpretability experiment is that the features learned by transcoders and SAEs are different. Exactly what a \"feature\" is is still being worked out in the literature, but if transcoders and SAEs truly do learn distinct sets of \"features\" that is an important fact that should not have been left out of the paper. If the authors did not investigate the extent of the overlap between the features that SAEs and transcoders learn that is unfortunate, but seems like a promising avenue for future investigations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**TL;DR**: Thank you so much for taking the time to review our work. We were glad to see that you enjoyed our blind case studies and that you recognized the importance of the input-invariant/input-dependent decomposition of attributions. We performed some experiments in response to your questions about learned features, but we’d like to emphasize that our primary goal in this work was to use transcoders to perform input-invariant circuit analysis that SAEs can’t do. \n\nWe’ll now address your concerns and questions in order. \n\n>The concrete comparisons to traditional SAEs are not sufficiently thorough [...] There are, however, other questions that are more concrete but are unaddressed. e.g. how easy/stable are transcoders to train compared to SAEs?\n\nWe did not notice any meaningful differences in the difficulty of training transcoders compared to SAEs. (Note that we primarily follow the same training procedure as used for training SAEs, as our pipeline is adapted from SAE training pipelines.)\nThe reason why we do not address this in the paper is because we did not find the experience of training transcoders to be noteworthy compared to training SAEs, and because training transcoders was instrumental to our primary goal of utilizing transcoders for circuit analysis.\n\n>Do transcoders suffer issues with \"dead neurons\" when the width is scaled up? Do transcoders show similar \"feature splitting\" effects when the width is scaled up?\n\nIn response to your question, we quickly analyzed the number of dead neurons in transcoders versus MLP-out SAEs with similar average sparsities (L0s), and found no clear winner when it comes to whether transcoders or SAEs have more dead neurons. (See the author rebuttal PDF for a graph.) As stated earlier, this investigation of training dynamics was not a core focus of ours. In any event, any dead neurons in the transcoders that we trained did not seem to affect fidelity or interpretability.\n\nAs for feature splitting, it seems that there are currently no rigorous metrics agreed upon by the mechanistic interpretability community for quantifying feature splitting. That said, we did train a pair of GPT2-small transcoders with expansion factors 32 and 64, and found that ~27% of live smaller-transcoder features were very similar (cossim > 0.95) to at least one feature in the larger transcoder. This seems to indicate some amount of feature splitting, but we find it difficult to interpret this number further.\n\n>While performing a blind test of the interpretability of different features is novel, I worry that it is prematurely rigorous.\n>One of the implicit assumptions of the blind interpretability experiment is that the features learned by transcoders and SAEs are different [...] If the authors did not investigate the extent of the overlap between the features that SAEs and transcoders learn that is unfortunate.\n\nPlease note that our intent with these blind tests was to show that transcoder features are “up to par” with SAE features in interpretability, regardless of whether or not the features are different. This is because our main goal with transcoders is to use them to perform input-independent circuit analysis that can’t be carried out with SAEs, so we simply wanted to make sure that in achieving this, no penalties to interpretability were accrued. Importantly, this means that we weren’t intending to make any claims about whether transcoders learn “better” features than SAEs; rather, we intended to show that whatever features the transcoders do learn are equally interpretable to those learned by SAEs, without making any assumptions on the types of features being learned by SAEs versus transcoders.\n\n> What is the effect of different levels of L1 penalty on the features that the transcoder learns?\n\nWe did not investigate differences in features learned by transcoders at different sparsity levels, in part because we are primarily interested in the circuits learned by transcoders – and because of a lack of accepted methods for rigorously characterizing transcoder/SAE features learned at different sparsity levels. To our knowledge, this is a current lacuna in the mechanistic interpretability community’s collective understanding.\n\n> Does the input invariant / dependent construction not work for the features that SAEs learn?\n\n**This construction does not in fact work for SAEs; we believe that this is the primary advantage of transcoders.** This is not because of the features that SAEs learn per se, but because SAEs fail to bypass the MLP sublayer whose nonlinearity prevents such a construction from being applied. In contrast, when computing attributions, transcoders explicitly bypass the MLP sublayer.\n\nMore formally, the input-invariant/input-dependent factorization for transcoder attributions is a special case of the classic “input-times-gradient” attribution method. For transcoders, the “input” term is input-dependent, but the “gradient” term is constant. But for SAE attributions, both the “input” term and the “gradient” term are input-dependent. (We would like to go into more mathematical detail here, but the rebuttal character limit prevents us from this.)\n\n> When looking at [sic]\n\nIt seems that this comment may have been cut off. If you can recall what you intended to write, we will be happy to reply during the discussion period.\n\n> Do SAEs and transcoders trained on the same layer and same data learn the same or similar features?\n\nWe performed some brief exploratory experiments a while back investigating this question on one-layer models. However, we have not continued to pursue this, as it was tangential to our primary goal of using transcoders for circuit analysis, although we do find it very interesting.\n\n> Have any attempts been made to perform a similar analysis for other circuits? e.g. the IOI or Gendered Pronoun circuits?\n\nWe have not carried these out yet, but we agree this is a promising direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkZYqwVelr", "reviewer": "Reviewer_GwPg", "summary": "The paper compares transcoders to SAEs for interpreting parts of GPT2-small, Pythia-410M and Pythia 1.4B. Transcoders are SAEs trained to replace a particular MLP in the original model instead of implementing an identity. \nThey find that the transcoders they train outperform the baseline SAEs they train on the pareto-curve of sparsity, measured by the L0 pseudonorm, with faithfulness, measured by the CE loss of the model recovered.\nThey describe the attribution technique they use to attribute features to earlier features and other network components. \nThey apply this attribution technique to interpret feature activations without access to the network inputs. They also use it to analyze the GPT2 \"greater than\" circuit investigated in https://arxiv.org/abs/2305.00586.", "questions": "Why train the baseline SAEs on the MLP inputs instead of the outputs? This way, the SAEs need to reconstruct all features in the residual stream within the MLP preimage, while the transcoder only needs to reconstruct the new features computed by the MLP.\nIn section 4.3, why use neuron-based analysis as a baseline? Neurons tend to be very brittle under ablations in my experience. Why not e.g. activation PCA components?", "limitations": "All addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "I think transcoders are maybe the most important idea for a new interpretability technique to investigate at the moment, and this is the first experimental study of them on language models I’m aware of.  SAEs have a potential issue of finding features that sparsely decompose the input data distribution rather than the computations of the neural network itself. Transcoders seem like the simplest potential solution to this problem since they essentially decompose the operations the network performs instead of its activations. \nI like that they showcase using the technique to reverse engineer a circuit in section 4.3.", "weaknesses": "In Appendix E line 1050, they state that the baseline SAEs they compare their results against were trained on MLP inputs rather than MLP outputs. I think this is the wrong comparison point. The transcoders only need to reconstruct the new features computed by a single MLP, while an SAE trained on MLP inputs needs to reconstruct all features in the residual stream that are in the preimage of the MLP input matrix. This is potentially a lot more features, which might give the SAEs an inherent disadvantage. Training the SAEs on the MLP output activations instead would be a more appropriate comparison. I think this is the biggest weakness of the paper.\n\n\nIn section 4.3 and figure 5, they use neuron-based analysis as a baseline to compare their transcoder feature-based analysis to. I think this is a very weak baseline. Using e.g. activation PCA components would be more appropriate. This is the main result they have comparing their technique to standard techniques outside the cluster of techniques based on SAEs/sparse decoding, so I think not using an appropriate baseline here is not great. I don't think it is a very major flaw, because this is mainly a paper that takes it as given that SAEs are interesting and attempt to improve on them, rather than a paper trying to justify the sparse decoding approach to interpretability.\n\n\nThey describe the attribution method between transcoder features they use as a 'new technique', which I don’t think is justified. It's effectively the attribution patching technique https://arxiv.org/abs/2310.10348 , https://arxiv.org/abs/2403.00745, used e.g. in https://arxiv.org/abs/2403.19647. Transcoders just have the advantage of making it work more elegantly. Any ReLU MLP network attribution patching is applied to would yield equation 6 as well. \n\n\nIn section 3.2.1 and table 1, manual comparison on fifty features isn’t a very large sample size for this sort of analysis. Supplementing this with e.g. a larger set of LLM generated autointerpretability scores would have been better, as was done in https://arxiv.org/abs/2309.08600. Personally, I don’t think not investing more effort into this is a major flaw, because I think reconstruction error and L0 are more important metrics than human interpretability scores.\n\n\nIn Figure 2, they present reconstruction scores for transcoders and SAEs in terms of raw CE-loss recovered scores. I think it would have been good for the presentation to also show the recovered performance in terms of the compute required to train a model with the same CE score, as suggested in https://arxiv.org/abs/2406.04093. Raw CE scores can make the performance gap to the model look smaller than it is, since the difficulty of further reducing CE loss grows as models become more performant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to write your thoughtful review and for recognizing the importance of transcoders to the broader mechanistic interpretability research program. We are also very excited to continue working on transcoders and seeing how far they can go.\n\nNow, we would like to respond to your various questions.\n\n> In Appendix E line 1050, they state that the baseline SAEs they compare their results against were trained on MLP inputs rather than MLP outputs. I think this is the wrong comparison point. [...] Training the SAEs on the MLP output activations instead would be a more appropriate comparison. I think this is the biggest weakness of the paper.\n\nThis is a piece of feedback that we received from other readers as well post-submission, and we have responded by repeating these experiments for SAEs trained on the MLP output activations. The results were similar: transcoders performed on-par with or better than the SAEs by the sparsity/fidelity metrics. Our updated figure is attached to our author rebuttal in our PDF.\n\n> In section 4.3 and figure 5, they use neuron-based analysis as a baseline to compare their transcoder feature-based analysis to. I think this is a very weak baseline. Using e.g. activation PCA components would be more appropriate. This is the main result they have comparing their technique to standard techniques outside the cluster of techniques based on SAEs/sparse decoding, so I think not using an appropriate baseline here is not great. I don't think it is a very major flaw, because this is mainly a paper that takes it as given that SAEs are interesting and attempt to improve on them, rather than a paper trying to justify the sparse decoding approach to interpretability.\n\nThank you for the suggestion. We avoided looking at PCA components on the activations because to some extent, they suffer from the same problem as SAEs when applied to this task: namely, they do not address the computation carried out by the MLP, but only deal with the intermediate activations of those computations. In carrying out our investigation of the “greater-than” circuit, we wanted to understand how MLP10 in GPT2-small maps year tokens represented in the MLP input space to MLP outputs that boost certain logits. Because PCA (just like SAEs) only handles the intermediate activations of this computation, it is unable to explain this mapping in an input-invariant way. (Note that the original “greater-than” circuit paper also does not use PCA, presumably for similar reasons.)\n\n> They describe the attribution method between transcoder features they use as a 'new technique', which I don’t think is justified. It's effectively the attribution patching technique [...] Transcoders just have the advantage of making it work more elegantly. Any ReLU MLP network attribution patching is applied to would yield equation 6 as well.\n\nWe are aware of this, and had already modified the paper post-submission to include a footnote explaining that this is a special case of the classic “input-times-gradient” method for computing attributions. (As you note, this ends up being equivalent to attribution patching when looking at a single transcoder feature, rather than a vector of feature activations.)\n\nWhat we consider to be a contribution of ours is our recognition that input-times-gradient, when applied to ReLU MLPs such as transcoders, does factorize into an input-dependent and an input-invariant term – i.e. the gradient is constant. Furthermore, something that we like to underscore when thinking about Equation 6 is that for transcoders (and this is not true in general for ReLU networks), the input-dependent term (the feature activation) is interpretable (because feature activations are largely interpretable).\n\nBased on your comment, we can modify the paper to further highlight the relationship between Equation 6 and existing attribution methods.\n\n> In section 3.2.1 and table 1, manual comparison on fifty features isn’t a very large sample size for this sort of analysis. Supplementing this with e.g. a larger set of LLM generated autointerpretability scores would have been better, as was done in https://arxiv.org/abs/2309.08600. Personally, I don’t think not investing more effort into this is a major flaw, because I think reconstruction error and L0 are more important metrics than human interpretability scores.\n\nThank you for the suggestion. Indeed, we did recognize that fifty features each from a transcoder and SAE was a small sample size, but unfortunately, resource limitations prevented us from carrying out LLM-generated feature scoring on far more features. That said, note that our initial interpretability comparison was primarily intended to provide initial evidence for transcoders being approximately as interpretable as SAEs, and as such, able to be used in furthering our main goal of performing circuit analysis with them.\n\n> In Figure 2, they present reconstruction scores for transcoders and SAEs in terms of raw CE-loss recovered scores. I think it would have been good for the presentation to also show the recovered performance in terms of the compute required to train a model with the same CE score, as suggested in https://arxiv.org/abs/2406.04093. Raw CE scores can make the performance gap to the model look smaller than it is, since the difficulty of further reducing CE loss grows as models become more performant.\n\nThank you for this suggestion – this is a very interesting idea that we had not previously considered. We can update the figures to include an additional set of y-axis ticks reflecting the amount of compute that a Chinchilla-optimal model would require to achieve the same loss.\n\n> Questions: …\n\nWe believe that all of the questions have been addressed above; please let us know if you have any more.\n\nAgain, thank you so much for your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "adiJ637ORW", "reviewer": "Reviewer_sL5K", "summary": "The paper introduces the use of transcoders a tool for mechanistic interpretability as a replacement for SAE. The main difference between SAE and transcoders is that SAE uses autoencoders to take the output of an MLP  and reconstruct it while transcoders (that can also be viewed as an encoder-decoder architecture) take the input of an MLP and reconstruct the output. \nThe paper compares SAE and transcoders in terms of sparsity (L0 Norm of the features), faithfulness (matching the output of the original MLP), and human interpretability and show that they in par with SAE on multiple-size models.\n\n\nThe paper shows how transcoders can be used to perform weights-based circuit analysis through MLP sublayers by doing the following:\n- First identifying attribution between feature pairs:\n    - Each feature in a transcoder is associated with two vectors: the i-th row of $W_{enc}$ is the encoder feature vector of feature i, and the i-th column of $W_{dec}$ is the decoder feature vector of feature i. The i-th component of zTC(x) is the activation of feature i.\n    - Assuming $l$ is in an earlier layer than $l'$, they  calculate the contribution of feature i in transcoder $l$ to the activation of feature i' \nas the product of two terms: the earlier feature’s activation (which depends on the input to the model)  $z^{(l,i)}_{TC}(x^{(l,t)})$ and\n the dot product of the earlier feature’s decoder vector with the later feature’s encoder vector (which is\n independent of the model input). \n\n- Second finding computational subgraphs:\n    - On a given input and transcoder feature i', one can determine which earlier-layer transcoder features i are important for causing i' to activate, from the previous step.\n    - Once the earlier-layer features i that are relevant to i' are identified, they recurse on i to understand the most important features causing i to activate by repeating this process.\n    - Doing so iteratively yields a set of computational paths.\n    - These computational paths can then be combined into a computational subgraph, in such a way that each node (transcoder feature), edge, and path is assigned an attribution. \n\n\n- They defined a de-embeddings:\n    - A de-embedding vector for a transcoder feature is a vector that contains the direct effect of the embedding of each token in the model’s vocabulary on the transcoder feature\n    - The de-embedding vector for feature i in the layer l transcoder is given by $W_E^Tf_{enc}^{(l,i)}$, where $W_E$ is the model’s token embedding matrix.  This vector gives input-invariant information about how much each possible input token would directly contribute to the feature’s activation.\n    - Given a de-embedding vector, looking at which tokens in the model’s vocabulary have the highest de-embedding scores tells us about the feature’s general behavior.\n\n\nThe paper showed a blind case study of how they reverse engineer a feature in GPT2-small and analysed  GPT2-small “greater-than” circuit", "questions": "Section 3.2.1:\n- For the blind interpretably experiments why was layer 15 in particular chosen?\n- From the following sentence: \"We recorded  for each feature whether or not there seemed to be an interpretable pattern, and only after examining every feature did we look at which features came from where.\" \n    - How did you identify that there is an \"interpretable pattern\"?\n    - How many people performed this experiment?\n\n\nSection 4.1:\n- For attribution and graph calculation for a given feature i' at layer l' are all others feature at previous layers considered in the graph or only the same feature at the previous layer?", "limitations": "Yes, they have.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Novelty:\n\n- While transducers have been previously introduced this paper is the first to apply them to large models and analyze their performance in various settings.\n\n- The circuit analysis presented in the paper is novel in the sense that it disentangles input-invariant information from input-dependent information.\n\n### Presentation:\n\n- The paper is very well written and easy to follow.\n\n### Application:\n\n- The paper showed how the transducers can be used to reverse engineer features in GPT2-small and they also analyzed GPT2-small “greater-than” circuit comparing transducers to neuronal approach showing that transducers have better performance.", "weaknesses": "### Method:\n\nCircuit Analysis: Assuming Transcoders perfectly emulates MLP (which it tries to but as long as faithfulness loss > 0 it doesn't) the circuit analysis graph is between Transcoders in different layers but in the actual model there is multiple-head attention in between each MLP layer which is not accounted for its effect in equation 6.\n\n### Evaluation:\n\nThe evaluation was weak at best. The paper did compare with SAE but the only actual quantitative results were the difference between SAE and transcoders in terms of sparsity and faithfulness which only shows the transcoders are on par with SAE. The more important experiment is the interpretability experiment since this is why both methods were developed in the first place for these experiments were not systematic and many of the choices were not justified or explained properly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work. We are very glad to see that you recognize the importance of the input-invariant/input-dependent factorization, along with the power demonstrated by transcoders in reverse-engineering feature circuits in an actual model (GPT2-small). We will now address your questions in turn.\n\n> Circuit Analysis: Assuming Transcoders perfectly emulates MLP (which it tries to but as long as faithfulness loss > 0 it doesn't) the circuit analysis graph is between Transcoders in different layers but in the actual model there is multiple-head attention in between each MLP layer which is not accounted for its effect in equation 6.\n\nWe show in Appendix D.3 how attribution works through attention heads. We mention this in Line 190 (section 4.1.2), but we agree that this deserves to be moved to the main body of the paper In particular, the contribution of token $s$ at layer $l$ through attention head $h$ to transcoder feature $i’$ at layer $l’ > l$ is given by:\n\n$$ \\text{score}^{(l,h)} \\left( x_{pre}^{(l,t)}, x_{pre}^{(l,s)} \\right) \\left( \\left( \\left( W_{OV}^{(l,h)} \\right)^T f_{enc}^{(l’, i’)} \\right) \\cdot x_{pre}^{(l,s)} \\right)$$\n\nWhere $\\text{score}^{(l,h)}$ denotes the scalar “scoring” function that weights the importance of each source token to the destination token (i.e. the pattern in the QK circuit of attention). \n\n> Evaluation: The evaluation was weak at best. The paper did compare with SAE but the only actual quantitative results were the difference between SAE and transcoders in terms of sparsity and faithfulness which only shows the transcoders are on par with SAE. The more important experiment is the interpretability experiment since this is why both methods were developed in the first place for these experiments were not systematic and many of the choices were not justified or explained properly.\n\nOur primary goal in our work was **to demonstrate that transcoders enable input-invariant circuit analysis that is not possible with current methods using SAEs**. To that end, our sparsity, faithfulness, and interpretability experiments were indeed intended to show that transcoders are on par with SAEs when evaluated on these metrics. Because transcoders also provide additional benefits over SAEs in circuit analysis, this means that **you can obtain all of the benefits for circuit analysis with no penalties compared to SAEs**.\n\nIf there are any specific other experiments you would like to see comparing SAEs and transcoders, we are happy to carry them out during the discussion period. But we would like to reiterate that the most important benefit of transcoders is in their circuit analysis capabilities.\n\n> For the blind interpretability experiments why was layer 15 in particular chosen?\n\n\nWe provide our rationale in Appendix F:\n“Layer 15 was chosen largely heuristically, because we believed that this layer is late enough in the model to contain complex features, while not so late in the model that features are primarily encapsulating information about which tokens come next.”\n\n> From the following sentence: \"We recorded for each feature whether or not there seemed to be an interpretable pattern, and only after examining every feature did we look at which features came from where.\" How did you identify that there is an \"interpretable pattern\"? How many people performed this experiment?\n\nInterpretability scores were assigned on the basis of whether it was possible to form a hypothesis about what the feature was doing on the basis of a sample of activating examples. Due to budget constraints that prevented wider-scale experiments from being carried out, only one person performed the interpretable pattern experiment. We would like to emphasize that features from SAEs and transcoders were shuffled together in this experiment, and the subject was unaware of which was which. Please feel free to refer to Figure 6 in the appendix if you’re interested in seeing examples of interpretable, partially interpretable, and non-interpretable features. \n\n> For attribution and graph calculation for a given feature i' at layer l' are all others feature at previous layers considered in the graph or only the same feature at the previous layer?\n\nEquation 6 shows how any upstream feature affects any downstream feature. Because of residual connections in the transformer, this equation is valid for computing the effect of a feature in any previous layer to a feature in any later layer. (In particular, the index $l$ denoting the earlier-layer feature can be any layer less than $l’$ denoting the later-layer feature.)\nIt is also worth noting that because transcoders are trained independently on each MLP layer, and because there is not any inter-layer consistency of MLP features anyway, there is no concept of the “same feature” across different layers.\n\nAgain, thank you for taking the time to write your detailed review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Summary:** We were happy to see our reviewers recognize transcoders’ importance for mechanistic interpretability and appreciate our input-invariant circuit analysis and blind case studies. We found that we might not have adequately conveyed the main goal of our work: to use transcoders to perform input-invariant circuit analysis which is impossible with SAEs. We have addressed this in our responses, and are reorganizing our paper to fix this.

We would like to thank the reviewers for the time they spent reading our paper and offering useful feedback. We were particularly glad to read Reviewer GwPg’s belief that “transcoders are maybe the most important idea for a new interpretability technique to investigate at the moment,” and their interest in how we “showcase using the technique to reverse engineer a circuit in section 4.3.” Indeed, we were happy to see that reviewers appreciated our blind case study procedure, with reviewer xRyp stating that “[t]he feature case studies are a valuable contribution to the literature” and reviewer Q9a1 stating that “[t]he blind evaluation protocol was really good to see---I appreciated authors' efforts in stress testing the limits of their approach (this is how good science should go :)).” We also were happy to see reviewers appreciate the utility of our input-invariant attribution decomposition, with reviewer sL5K stating that “circuit analysis presented in the paper is novel in the sense that it disentangles input-invariant information from input-dependent information,” and reviewer xRyp commenting that “[d]eriving the decomposition into input dependent / invariant features is novel and addresses a significant challenge of interpretability. On its own this warrants future exploration.”

With regard to reviewers’ concerns, the primary theme that we noticed across most reviews was a slight misunderstanding regarding our main purpose in writing this paper, which we feel might be due to a lack of clarity on our part. Concretely, **our main goal in our research was to develop a method for utilizing transcoders in weights-based, input-invariant circuit analysis.** In contrast, our experiments comparing transcoders to SAEs were primarily intended to corroborate that transcoders are on par with SAEs in interpretability and fidelity, in order for transcoders to be used in circuit analysis without incurring any penalties relative to SAEs. We believe that this misunderstanding may underlie certain reviewers’ emphasis on the experiments in Section 3.2 directly comparing SAEs and transcoders according to standard SAE metrics.

With this in mind, we encourage all reviewers to particularly assess transcoders with an eye towards the benefits that they bring to input-invariant circuit analysis. This is especially important because, as we explained in our response to xRyp, this sort of circuit analysis cannot be achieved with standard SAEs. We will update our paper to include a mathematical demonstration of why SAEs are insufficient. We will also restructure the ordering of sections in our paper by their importance to our main goal; concretely, this means putting our introduction to circuit analysis with transcoders before our section comparing them with SAEs.

Besides this, reviewers had some smaller separate questions, all of which we believe we have adequately addressed in our individual rebuttals.
Once again, we thank all of the reviewers, and the Area Chairs, for their time and valuable comments.

Meta-review: The submission investigates transcoders, an encoder-decoder architecture, to imitate the input-output relationships in MLP layers in language models. Due to the linearity combination of MLP and attention blocks in the "residual stream", the contribution from a transcoder feature at a lower layer to the transcoder feature at a higher layer can be decomposed into input-dependent and input-invariant components. A similar analysis can be done for the attention heads. These can be combined with greedy searches to find sparse computation graphs that explain a specific transcoder feature.

The reviewers are generally enthusiastic about the submission: (i) the idea seems promising and could spur further work and (ii) the experiments are clear and clarifications during rebuttal were useful. The authors are encouraged to add some of these (e.g. reshuffling sections) to the camera-ready version.
Decision: Accept (poster)