Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
Authors: Dan Braun
Jordan Taylor
Nicholas Goldowsky-Dill
Lee Sharkey
Venue: NeurIPS 2024 poster (2024)
Keywords: Mechnastic Interpretability
Interpretability
Explainability
Transparency
Sparse Coding
Causal mediation analysis
High dimensional data analysis
Primary area: interpretability_and_explainability
TL;DR: We introduce end-to-end sparse autoencoders, which aim to learn functionally relevant features from network activations. They are a pareto improvement over standard sparse autoencoders.

Abstract: Identifying the features learned by neural networks is a core challenge in mechanistic interpretability. Sparse autoencoders (SAEs), which learn a sparse, overcomplete dictionary that reconstructs a network's internal activations, have been used to identify these features. However, SAEs may learn more about the structure of the datatset than the computational structure of the network. There is therefore only indirect reason to believe that the directions found in these dictionaries are functionally important to the network. We propose end-to-end (e2e) sparse dictionary learning, a method for training SAEs that ensures the features learned are functionally important by minimizing the KL divergence between the output distributions of the original model and the model with SAE activations inserted. Compared to standard SAEs, e2e SAEs offer a Pareto improvement: They explain more network performance, require fewer total features, and require fewer simultaneously active features per datapoint, all with no cost to interpretability. We explore geometric and qualitative differences between e2e SAE features and standard SAE features. E2e dictionary learning brings us closer to methods that can explain network behavior concisely and accurately. We release our library for training e2e SAEs and reproducing our analysis at
https://github.com/ApolloResearch/e2e_sae.

Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
Dan Braun" Jordan Taylor Nicholas Goldowsky-Dil
Lee Sharkey*
Abstract
Identifying the features learned by neural networks is core challenge in mechanistic interpretability. Sparse autoencoders (SAEs), which learn sparse, overcomplete dictionary that reconstructs network's internal activations, have been used to identify these features. However, SAEs may learn more about the structure of the dataset than the computational structure of the network. There is therefore only indirect reason to believe that the directions found in these dictionaries are functionally important to the network. We propose end-to-end (c2c) sparse dictionary learning, method for training SAEs that ensures the features learned are functionally important by minimizing the KI. divergence between the output distributions of the original model and the model with SAE activations inserted. Compared to standard SAEs, e2e SAEs offer a Pareto improvement: They explain more network performance, require fewer total features, and require fewer simultaneously active features per datapoint, all with no cost interpretability. We explore geometric and qualitative differences between e2e SAE features and standard SAE features. E2e dictionary learning brings us closer to methods that can explain network behavior concisely and accurately. We release our library for training c2e SAEs and reproducing our analysis at tttps://githut www.u...oo/aapae www./powo/ec//ec. comm/poolooeeeearceeoo eee_sae.
Introduction
Sparse Autoencoders (SAEs) are a popular method in mechanistic interpretability [Sharkey et al., 2022, Cunningham et al., 2023, Bricken et al., 2023]. They have been proposed solution to the problem of superposition, the phenomenon by which networks represent more 'features' than they have neurons. 'Features' directions in neural activation space that considered be the basic units of computation in neural networks. SAE dictionary elements (or 'SAE features') are thought approximate the features used by the network. SAEs are typically trained to reconstruct the activations of an individual layer of a neural network using sparsely activating, overcomplete set of dictionary elements (directions). It has been shown that this procedure identifies ground truth features in toy models [Sharkey et al., 2022].
However, current SAEs focus the wrong goal: They trained to minimize reconstruction error (MSE) of activations (in addition to minimizing their sparsity penalty). The issue that the importance of a feature measured by its effect MSE may not strongly correlate with how important the feature is for explaining the network's 's performance. This would not be a problem if the network's activations used small, finite set of ground truth features the SAE would simply identify those features, and thus optimizing MSE would have led the SAE to learn the functionally
squared
Apollo Research
ML Alignment & Theory Scholars (MATS), University of Queensland
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Block
Block
Unembed
BAE ucul recon.
SAE
Both SAE and
SAE
downstream recon.
SAF
(MSE)
(MSE)
KL-dlv 1o68
Block 6
Block
Blook 11
Unembed
Better
Better
0.00
0.00
0.05
0.05
0.10
0.10
CE Loss Increase
0.15
0.15
Botter
0.20
0.20
SAE Type
0.30
0.30
local
0.35
e2e
0.35
e2e+ds
100
200
300
400
500
600
10k 20k
30k
40k
L0
Alive Dictionary Elements
Figure 1: Top: Diagram comparing loss terms used to train each type SAE. Each arrow a loss term which compares the activations represented by circles. SAEleea uses MSE reconstruction loss between the SAE input and the SAE output. SAEa. uses XL-divergence on the logits. SAEezeddi cend-to-end + downstream reconstruction) uses KL-divergence addition the of the MSE reconstruction losses all future layers. All three are additionally trained
L1 sparsity penalty (not pictured).
Bottom: Pareto curves for three different types GPT2-small layer 6 SAFs as the sparsity cnefficient varied. F2e-SAFs require fewer features per datapoint (i.c. have lower Lo) and fewer features over the entire dataset (i.e. have low number of alive dictionary elements). GPT2-small has CE loss of 3.139 over our evaluation
important features. In practice, however, Bricken et al. [2023] observed the phenomenon of feature splitting, where increasing dictionary size while increasing sparsity allows SAEs split feature into multiple, more specific features, representing smaller and smaller portions of the dataset. In the limit of large dictionary size, it would be possible to represent each individual datapoint as its own dictionary element. Since minimizing MSE does not explicitly prioritize learning features based on how important they are for explaining the network's 's performance, an SAE may waste much of its fixed capacity learning less important features. This is perhaps responsible for the observation that, when measuring the causal effects circuits made from SAE features on network performance, significant amount mediated by the reconstruction residual errors (i.e. everything not explained by the SAF) and not mediated by SAE features [Marks et al., 2024].
Given these issues, it is therefore natural to ask how we can identify the functionally important features used by the network. We say feature is functionally important if it is important for explaining the network's behavior the training distribution. If we prioritize learning functionally important features, we should be able to maintain strong performance with fewer features used by the SAE per datapoint well as fewer overall features.
To optimize SAEs for these properties, we introduce new training method. We still train SAEs using sparsity penalty on the feature activations (to reduce the number of features used on each datapoint), we no longer optimize activation reconstruction. Instead, we replace the original activations with the SAE output (Figure 1) and optimize the KI. divergence between the original output logits and the output logits when passing the SAE output through the rest of the network, thus training the SAE end-to-end (c2c). We use SAE,2 to denote an SAE trained with KL divergence and sparsity penalty.

---

By contrast, we use SAEtocal to denote our baseline SAEs, trained only to reconstruct the activations the current layer with a sparsity penalty.
One risk with this method is that it may be possible for the outputs of SAFeze to take a different computational pathway through subsequent layers of the network (compared with the original activations) while nevertheless producing a similar output distribution. For example, it might learn new feature that exploits particular transformation in a downstream layer that is unused by the regular network or that is used for other purposes. To reduce this likelihood, we also add terms to the loss for the reconstruction error between the original model and the model with the SAE at downstream layers in the network (Figure 1). We use SAEzze+d: to denote SAEs trained with KL divergence, a sparsity penalty, and downstream reconstruction loss. We use eZe SAEs to refer the family of methods introduced in this work, including both SAE and SAE
Previous work has used the performance explained measured by cross-entropy loss difference when replacing the original activations with SAE outputs - as a measure of SAE quality [Cunningham et al., 2023, Bricken al., 2023, Bloom, 2024]. It's reasonable to ask whether our approach runs afoul of Goodbart's s law ("When measure becomes target, it ceases to be a good measure"). We contend that mechanistic interpretability should prefer explanations of networks (and the components of those explanations, such as features) that explain more network performance over other explanations. Therefore, optimizing directly for quantitative proxies of performance explained (such as CE loss difference, KL divergence, and downstream reconstruction error) is preferred.
We train each SAE type on language models (GPT2-small [Radford al., 2019] and Tinystories-1M 1M [Eldan and Li, 2023]), and present three key findings:
1. For the same level of performance explained. SAE requires activating more than twice as many features per datapoint compared to SAEc2o-ds and SAEc (Section 3.1).
2. SAFgzerds performs equally well as SAFgze in terms the number of features activated per datapoint (Section 3.1), yet its activations take pathways through the network that much more similar to SAE (Sections 3.2, 3.3).
3. SAEtocal requires more features in total over the dataset to explain the same amount of network performance compared with SAE and SAE,zerds (Section 3.1).
These findings suggest that e2e SAEs are more efficient in capturing the essential features that contribute to the network's performance. Moreover, our ietpmmattdd-nteepperalliigg and qualitative analyses reveal that SAEe2e-ds features are at least as interpretable as SABbccal features, demonstrating that the improvements efficiency do come the cost of interpretability (Section 3.4). These gains nevertheless come at the cost of longer wall-clock time to train (Appendix H).
As supplementary investigation, tested e2c SAEs on set of subject-verb agreement tasks from Finlayson et al. [2021]. These tests had inconclusive results (Appendix I), indicating that further work is needed to identify which downstream tasks would benefit most from e2e SAES.
In addition to this article, we also provide: library for training all SAE types presented in this article (https://8) //gg:///tttt com/ ApwlloReseerr htthheeeessaee__e/eaeeeeaaaaaeee Weights and Biases [Biewald, 2020] report that links to training metrics for all runs (https: pestts://appp //api. wandb. si/linka/spplee sparks/f//aaff/////// evnqx8t6); Neuronpedia [Lin and Bloom, 2023] page for interacting with the features in subset of SAES (those presented in Tables and 3) (https: //www//wwwww... neuronpedia. org/gpt2am org//gpteem-apoloojj apollojt) as well repository for downloading these SAEs directly (https: htt.g/ggggggga...../////// co/ apollo-researche saee--ptsss))))
2 Training end-to-end SAES
Our experiments train SAEs using three kinds of loss function (Figure 1), which we evaluate according to several metrics (Section 2.5):
1. Lical trains SAES to reconstruct activations at particular layer (Section 2.2);
2. Leze trains SAES to learn functionally important features (Section 2.3);
3. Teeetdowsstemm trains SAFs to learn functionally important features that optimize for faithfulness to the activations of the original network at subsequent layers (Section 2.4).
3

---

2.1 Formulation
Suppose we have a feedforward neural network (such as a decoder-only Transformer [Radford et al., 2018J) with L layers and vectors of hidden activations a!);
(x) = x a (I) = (a) for I....,,- -1
(a(0(all)))z))
y = softmax
We use SAEs that consist of an encoder network (an affine transformation followed by ReLU activation function) and dictionary of unit norm features, represented matrix D, with associated bias vector ba- The encoder takes as input network activations from a particular layer l. The architecture we use is:
Enc 1 (x)) ReLU (Wall(x) be)
SAE 11 (2) =D Ene
Enc (a)) bd.
where the dictionary D and encoder weights We both (N_ l__ddic__ell eleents x d hididen)) matrices, be is N_diit__eell_eeeeeeeelllleeeeeee cestt__eemmen elements-dimensional vector, while ba and ( (z) are dihieden-dimensionaal vectors.
2.2 Baseline: Local SAE training loss (T-ocal)
The standard, baseline method for training SAEs is SAEiocal training, where the output of the SAE is trained to reconstruct its input using mean squared error loss with sparsity penalty on the encoder activations (here an I.1 loss):
Lioal reconstruccion
Iispariity = (x) SA Aocaa (a (x))
ক [I
)9) is sparsity coefficient À scaled by the size of the input to the SAE (see Appendix D
for details hyperparameters)
2.3 Method 1: End-to-end SAE training loss (Leze)
For SAEa2e, we do not train the SAE to reconstruct activations. Instead, we replace the model activations with the output of the SAE and pass them forward through the rest of the network:
a (a (x))
(k) (x) = (a for = == I,...,
(all
ÿ = softmax
We train the SAE by penalizing the KL divergence between the logits produced by the model with the SAE activations and the original model:
Lete LKL. + Liparsity
K KL y) + +1 Ene(a' (x))ll
Importantly, we freeze the parameters of the model, so that only the SAE trained. This contrasts with Tamkin et al. [2023], who train the model parameters in addition to training "codebook' (which similar to a dictionary).
Method End-to-end with downstream layer reconstruction SAE training loss
(I:g2etdownstream)
reasonable concern with the Leze that the model with the SAE inserted may compute output using an importantly different pathway through the network, even though we've ve frozen the original model's parameters and trained the SAE to replicate the original model's output distribution. To counteract this possibility, we also compare additional loss: The end-to-end with downstream

---

reconstruction training loss (Loze-domnsream) additionally minimizes the mean squared error between the activations of the new model at downstream layers and the activations of the original model:
Le2oodownstraam
LKL + Espansiy + Lojonssteamm
31
(1)
= KKL v) + Enc(a (I)
Σ k=1+1
|lakk) (x) a (x)
where 3r is hyperparameter that controls the downstream reconstruction loss term (Appendix D).
Le2esdownstream thus has the desirable properties of 1) incentivizing the SAE outputs to lead similar computations in downstream layers in the model and 2) allowing the SAE to "clear out" some of the non-functional features by not training a reconstruction error at the layer with the SAE. Note, however. the inclusion of the intermediate reconstruction terms means that Le2e+downstrean may encourage the SAE learn features that are less functionally important.
2.5 Experimental metrics
We record several key metries for each trained SAE:
1. Cross-entropy loss increase between the original model and the model with SAE: We measure the increase in cross-entropp (CE) loss caused by using activations from the inserted SAE rather than the original model activations on an evaluation set. We sometimes refer to this 'amount of performance explained', where low CE loss increase means more performance explained. All other things being equal, a better SAE recovers more of the original model's 's performance.
2. Lo: How many SAE features activate on average for each datapoint. All other things being equal, better SAE needs fewer features to explain the performance of the model on a given datapoint.
3. Number of alive dictionary elements: The number of features in training that have not 'died' (which we define to mean that they have not activated over set of 500k tokens of data). All other things being equal, a better SAE needs smaller number of alive features to explain the performance of model over the datuset.
We also record the reconstruction loss at downstream layers. This is the mean squared error between the activations the original model and the model with the SAE at all layers following the insertion of the SAE (i.e. downstream layers). If reconstruction loss at downstream layers is low, then the activations take a similar pathway through the network as in the original model. This minimizes the risk that the SAEs are learning features that take different computational pathways through the downstream layers compared to the original model. Finally, following Bills et al. [2023], we perform automated-interpe internrettbbllity scoring and qualitative analysis subset of the SAFs, to verify that improved quantitative metrics does not sacrifice the interpretability of the learned features.
We show results for experiments performed on GPT2-small's residual stream before attention layer 6. Results for layers 2, 6, and 10 of GPT2-small and some runs on a model trained on the TinyStories dataset [Eldan and Li, 2023] can be found in Appendices A. and A.2, respectively. They are qualitatively similar to those presented in the main text. For our GPT2-small experiments, we train SAES with each type of loss function 400k samples of context size 1024 from the Open Web Text dataset [Gokaslan and Cohen, 2019] over a range of sparsity coefficients X. Our dictionary is fixed at 60 times the size of the residual stream (i.e. 60 x 768 46080 initial dictionary elements). Hyperparameters, along with sweeps over dictionary size and number of training examples, are shown in Appendices D and E, respectively.
3 Results
3.1 End-to-end SAES are Pareto improvement over local SAEs
We compare the trained SAEs according to CE loss increase, Lg, and number of alive dictionary elements. The learning rates for each SAE type were selected be Pareto-optimal according their
We use zero-indexed layer numbers throughout this article
5

---

Lo vs CE loss increase curves." 4 Each experiment uses a range of sparsity coefficients A. In Figure 1, we see that both SAEz2e and SA achieve better CE loss increase for given Lo or for a given number of alive dictionary elements. This means they need fewer features to explain the same amount of network performance for given datapoint or for the dataset as whole, respectively. For similar results at other layers see Appendix A.1.
This difference is large: For given Lo, both SAE,2e and SAEe2esds have CE loss increase that is less than 45% of the CE loss increase of SACiocal. 5 SABeoca must therefore be learning features that are not maximally important for explaining network performance.
This improved performance comes at the expense of increased compute costs (2-3.5 times longer runtime, Appendix H). We test to see if additional compute improves SAEoccal baseline in Appendix E. We find neither increasing dictionary size from 60 768 to 100 768 nor increasing training samples from 400k 800k noticeably improves the Pareto frontier, implying that our e2e SAEs maintain their advantage even when compared against SABucal dictionaries trained with more compute.
For comparability, our subsequent analyses focus on particular SAEs that have approximately equivalent CE loss increases (Table 1).
Table I: Three SAEs from layer with similar CE loss increases are analyzed in detail.
SAE Type
À (Sparsity Coeff)
To
Alive Elements
CE Loss Increase
Local End-to-end
4.0
69.4
26k
0.145
3.0
27.5
22k
0.144
E2e + Downstream
50.0
36.8
15k
0.125
3.2 End-to-end SAEs have worse reconstruction loss at each layer despite similar output distributions
Even though SAEa2e explain more performance per feature than SAE they have much worse reconstruction error of the original activations each subsequent layer (Figure 2). This indicates that the activations following the insertion of SAFz2 take different path through the network than in the original model, and therefore potentially permit the model to achieve its performance using different computations from the original model. This possibility motivated the training of SAE
In later layers, the reconstruction errors of SAEnccal and SAEc2o+ds are extremely similar (Figure 2). SAEe2e+ds therefore has the desirable properties of both learning features that explain approximately as much network performance as SAE,2 (Figure 1) while having reconstruction errors that are much closer to SAEpcal-- There remains a difference in reconstruction at layer 6 between SAE and SAEjccal- This not surprising given that SAE not trained with reconstruction loss this layer. In Appendix B, we examine how much of this difference is explained by feature scaling, In Appendix G.3, we find a specific example of direction with low functional importance that faithfully represented in SAF but not SAF
3.3 Differences in feature geometries between SAE types
3.3.1 End-to-end SAEs have more orthogonal features than SAEeocal
Bricken et al. [2023] observed 'feature splitting', where locally trained SAEs learns cluster of features which represent similar categories of inputs and have dictionary elements pointing in similar directions. A key question what extent these subtle distinctions are functionally important for the
*We show in Appendix C that is possible to reduce the number of alive dictionary elements for any SAE type by increasing the learning rate. This has minimal cost according to Lo vs CE loss increase Pareto-optimality up to some limit.
' "Measured using linear interpolation over range of La (50, 300). This range was chosen based on two criteria: (1) Lo should be significantly smaller than the residual stream size for the SAE to be effective (we conservatively chose 300 compared to the residual stream size of 768), and (2) the CE loss should not start to increase dramatically, which occurs approximately Le 50 (Figure 1).

---

SAE Type
local
Reconstruction MSE
eãe a2o+ds
20
10
Model Layer
Figure 2: Reconstruction mean squared error (MSE) at later layers for our set of GPT2-small layer 6 SAEs with similar CE loss increases (Table 1). SABpcal is trained minimize MSE at layer SAE, trained match the output prohability distribution, SAFezesds was trained to match output probability distribution and minimize MSE in all downstream layers.
network's predictions, or if they only helpful for reconstructing functionally unimportant patterns in the data.
We have already seen that SAEg2e and SAE,ess learn smaller dictionaries compared with SAEoocal for given level of performance explained (Figure 1). In this section, we explore if this is due less feature splitting. We measure the cosine similarities between each SAE dictionary feature and next- closest feature in the same dictionary. While this does not account for potential semantic differences between directions with high cosine similarities, it serves as useful proxy for feature splitting, since split features tend to be highly similar directions [Bricken et al., 2023]. We find that SAE has features that are more tightly clustered, suggesting higher feature splitting (Figure 3a). Compared to SAEa2esds the mean cosine similarity 0.04 higher (bootstrapped 95% CI [0.037-0.043)] compared to SAE,2e the difference is 0.166 (95% CI [0. 163 0.168) ). We measure this for all runs in Pareto frontiers and find that this difference is not explained by SAE having more alive dictionary elements than e2e SAEs (Appendix A.5).
3.3.2 SAE features are not robust across random seeds, but SAEgze+ds and SA 3fcal are
We find that SAE trained with one seed learn similar features as SAE trained with a different seed (Figure 3b). The same is true for two SA However, features learned by SAE,20 quite different for different seeds. This suggests there are many different sets SAFzze features that achieve the same output distribution, despite taking different paths through the network.
3.3.3 SAE2e and SAEgee+ds features do not always align with Etocal features
The cosine similarity plots between SAE,2e and SAE (Figure 3c top) reveal that the average similarity between the most similar features is low, and includes a group of features that very dissimilar. SAEa2eta learns features that are much more similar to SAEuueal- although the cosine similarity plot is bimodal, suggesting that SAEg2osds learns set of directions that very different to those identified by SAEiocal (Figure 3c bottom).
It is encouraging that SABneaa and SAE,2esd features are somewhat similar, since this indicates that SAE may serve as good initializations for training SAE reducing training time.
3.4 Interpretability of learned directions
Using the irrtomateddinteeereetaeee we generate automated explanations of our SAE features by prompting 09 [OpenAl et al., 2024] with five max-activating examples for each feature, before generating "interpretability scores" by tasking gpt-3.5-turbo to use that explanation to predict the SAE feature's true activations on random sample of 20 max-activating examples. For each SAE we generate iecomated-interpretaable scores for random sample of features (n 198 to 201 per SAF). We then measure the difference between average interpretability scores. This interpretability score is an
library [Lin, 2024] (an adaptation of Bills et al. [2023]). hil-tuubo-2222-00
7

---

local
local
e2e - local
e2e
e2e
e2e+ds local
62e+ds
e2e+ds
0.0
0.5
0.0
1.0
0.5
Cosine Similarity
Cosine Similarity
Cosine Similarity
(a) SAE similarity
Cruss seed similarity
SAE-type similarit
Figure 3: Geometric comparisons for our set GPT2-small layer SAEs with similar CE loss increases (Table
1). For each dictionary element, we find the max cosine similarity between itself and all other dictionary elements. In 3a we compare to others directions in the same SAE, in 3b to directions in an SAE of the same type trained with different random seed, in 3c directions in the SAEicaa with similar CE loss increase.
imperfect metric of interpretability, but it serves as unbiased verification and is therefore useful for ensuring that we are not trading better training losses for significantly less interpretable features.
For pairs of SAEs with similar Lo (listed in Table 3), we find no difference between the average interpretability scores of SAEtocal and SAE,2e+ds- If repeat the analysis for pairs with similar CE loss increases, we find the SAEe features to be more interpretable than SAEtocal features in Layers 2 (p 0.0053) and 6 (p- 0.0005) but no significant difference in layer 10. For additional autonnated-intepprtaabllll
analysis, see Appendix A.7.
We also provide some qualitative, human gemerateeeerted interpretations of some groups of features for different SAE types in Appendix G. Features from the SAEs in Table 2 and Table 3 can be viewed interactively at https: // www. neuronpedia. org/
4 Related work
4.1 Using sparse autoencoders and sparse coding in mechanistic interpretability
When Elhage et al. [2022] identified superposition as a key bottleneck to progress mechanistic interpretability, the field found promising scalable solution in SAEs [Sharkey et al., 2022]. SAEs have since been used to interpret language models [Cunningham et al., 2023, Bricken al., 2023, Bloom, 2024] and have been used to improve performance of classifiers downstream tasks [Marks et al., 2024]. Earlier work by Yun et al. [2021] concatenated together the residual stream language model and used sparse coding to identify spanned multiple layers. This echoes even earlier work that applied sparse coding to word embeddings and found sparse linear structure [Faruqui et al., 2015, Subramanian et al., 2017, Arora et al., 2018]. Similar to our work is Tamkin et al. [2023], who trained sparse feature codebooks, which are similar to SAFs, and trained them end-to-end. However, to achieve adequate performance, they needed to train the model parameters alongside the sparse codebooks. Here, we only trained the SAEs and left the interpreted model unchanged.
undercomplete set of sparse 'factors' that
4.2 Identifying problems with and improving sparse autoencoders
Although useful for mechanistic interpretability. current SAE approaches have several shortcomings. One issue the functional importance of features, which we have aimed address here. Some work has noted problems with SAEs, including Anders and Bloom [2024], who found that SAE features trained on a language model with a given context length failed to generalize to activations collected from activations in longer contexts. Other work has addressed 'feature suppression' [Wright and Sharkey, 2024], also known as 'shrinkage' [Jermyn et al., 2024], where SAE feature activations systematically undershoot the 'true' activation value because of the sparsity penalty. While Wright and Sharkey [2024] approached this problem using finetuning after SAE training, Jermyn et al. [2024] and Riggs and Brinkmann [2024] explored alternative sparsity penalties during training that aimed to reduce feature suppression (with mixed success). Farrell [2024], taking an approach similar to Jermyn et al. [2024], has explored different sparsity penalties, though here not to address shrinkage,

---

but instead to optimize for other metries of SAE quality. Rajamanoharan et al. [2024] introduce Gated SAEs, an architectural variation for the encoder which both addresses shrinkage and improves on the Pareto frontier of T- vs CE loss increase.
4.3 Methods for evaluating the quality of trained SAES
One of the main challenges in using SAEs for mechanistic interpretability is that there is no known 'gounnd truth' against which to benchmark the features learned by SAES. Prior to our work, several metrics have been used, including: Comparison with ground truth features in toy data; activation reconstruction loss; L1 loss; number of alive dictionary elements; similarity of SAE features across different seeds and dictionary sizes [Sharkey et al.. 2022]; Lo: KL divergence (between the output distributions of the original model and the model with SAE activations) upon causal interventions on the SAE features [Cunningham et al., 2023]; reconstructed negative log likelihood of the model with SAE activations inserted [Cunningham et al., 2023, Bricken et al., 2023]; feature interpretability Cunningham al. [2023] (as measured by automatic interpretability methods [Bills et al., 2023]); and task -peccffc comparisons [Makelov et al., 2024]. In our work, we use (1) Lo, (2) number of alive dictionary elements, (3) the average KL divergence between the output distribution of the original model and the model with SAE activations, and (4) the reconstruction error of activations in layers that follow the layer where we replace the original model's activations with the SAE activations.
4.4 Methods for identifying the functional importance of sparse features
In our work, we optimize for functional importance directly, but previous work measured functional importance post hoc using different approaches. Cunningham et al. [2023] used activation patching [Vig et al., 2020], form of causal mediation analysis, where they intervened on feature activations and found the output distribution was more sensitive (had higher KL divergence with the original model's 's distribution) in the direction of SAE features than other directions, such as PCA directions. With the same motivation, Marks et al. [2024] use similar, approximate, but more efficient, method of causal mediation analysis [Nanda, 2022, Sundararajan et al., 2017]. Unlike our work, these works use the measures of functional importance to construct circuits of sparse features. Bricken al. [2023] used logit attribution, measuring the effect the feature has the output logits.
5 Conclusion
In this work, we introduce end-to-end dictionary learning as a method for training SAEs to identify functionally important features in neural networks. By optimizing SAEs to minimize the KL divergence between the output distributions of the original model and the model with SAE activations inserted, we demonstrate that e2e SAEs learn features that better explain network performance compared to the standard locally trained SAES.
Our experiments on GPT2-small and Tinystories- 1M reveal several key findings. First, for given level of performance explained, e2e SAEs require activating significantly fewer features per datapoint and fewer total features over the entire dataset. Second, SAFezeeds which has additional loss terms for the reconstruction errors at downstream layers the model, achieves a similar performance explained to SAE,2e while maintaining activations that follow similar pathways through later layers compared to the original model. Third, the improved efficiency of c2c SAEs does not come at the cost of interpretability, as measured by automated interpretability automateed-nterppetabbiiiit
scores and qualitative analysis.
These results suggest that standard, locally trained SAEs are capturing information about dataset structure that is not maximally useful for explaining the algorithm implemented by the network. By directly optimizing for functional importance, e2e SAEs offer more targeted approach identifying the essential features that contribute to network's 's performance.
6 Impact statement
This article proposes an improvement to methods used in mechanistic interpretability. Mechanistic interpretability, and interpretability broadly, promises to let us understand the inner workings of neural networks. This may be useful for debugging and improving issues with neural networks. For instance, it may enable the evaluation of model's faimess or bias. Interpretability may relatedly be

---

useful for improving the rrus-worthinee of Al systems, potentially enabling Al's use in certain high stakes settings, such healthcare, finance, and justice. However, increasing the torth-worthiness of Al systems may be dual use in that may also enable its use in settings such as military applications.
References
Evan Anders and Joseph Bloom. Examining language model performance with recon- structed activations using sparse autoencoders. wat.aaa--ppoorraaaa---t-t----ra- witheeeeeewwtt-eeeeee
wwwwww/wwww.....ggooooooo....... com/posts/
2024.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. 2018. URL https://arxxi wttpa/wwwww.....a////ba///// 03764.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL hwtth:::///wuttt wandb. com/ Software available from wandb.com.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https: hptpa/pppaaaccccc... blob. core. wwwewwwwidww....m wiwdows.net/neurror. pewprrooneepplaaneerppeerrr 2023.
Joseph Bloom. Open source sparse autoencoders for all residual stream layers of gpt2 small. hata--oourespppaaaattttooreeeees www.aaaooeer--aaaaeaaaaa for aal--e-oo--aaa--aaaaeeee-------e all--rrsidall-steeaateee 2024.
https://wwi www.gaantoooouummmmmmmmm.m wwt.:oooww.oooppppt//gggggggaaae sonn-souree-sarseee
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zae Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume. Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https: /ttp:://rrraa http://taansfforooe ormer- circuits pub/ 2023 htte///mosemaaii--ffattae wiwpss/iineeeee.eeee html.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309 08600, 2023. URL https:://rrti
org/ abaaaba//2000 www.//.0.9..000.... 08600.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.077.59 2023.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition, 2022.
Eoin Farrell. Experiments with an alternative method to promote sparsity in sparse autoencoders. experiments with an- alternative-methaaee to- wwwpppooooeoeppee sparsstys. 2024.
https://wwer lesswrong. com/ http://ssss//AAxxxx..eeam
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. Sparse overcomplete word vector representations, 2015.
Matthew Finlayson, Aaron Mueller, Sebastian Gehrmann, Stuart Shieber, Tal Linzen, and Yonatan Belinkov. Causal analysis of syntactic agreement mechanisms in neural language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 1188--8433 Online, August 2021. Association for Computational Linguistics. doi: 10. 18653/v1/2021 -all--ooooooonoo. 144. 144. URL https: : /aa//claattttt awwaaaanlanncla...aaaaaaaao .rcllooggoorrggg// 2021 acl- -long.onn..... 144.
10

---

Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skyylion007, .ithhb. / OpenieeTextCorpus, 2019.
Adam Jermyn, Adly Templeton, Joshua Batson, and Trenton Bricken, Tanh penalty in dictionary https: httpe::/trranfforrmer circuits putp2024//febuuudatee/ unddeeeeeeeeeeeeeteeeeee html#; Sheett-aaaddd2dddgggngg///oouorr Tanh:/00emmlyy22000//000tttooo 20Learning, -Adam/2200errrmyy222222202.
learning.
2024.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Johnny Lin.
Automatic interpretability.
https://github..cmm johnnoyylin
automated- int.rreeteeebiityy 2024.
Johnny Lin and Joseph Bloom. Analyzing neural networks with dictionary learning, 2023. URL https://ww, neuronpedia org. Software available from neuronpedia.org.
Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. MHIX9H8aYF, 2024.
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 2024.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861, 2018. doi:
10.21 105/00ss.0085 00861. URL https://dol. otopi..rrg1.... 21105/joss 00861.
Neel Nanda. Attribution patching: Activation patching industrial scale. https://www. neelnanda. io/ http///aanssiinnnerrreeeee httpaaaaw..../mmcaaassi--------o
bhecaaasttichheeppppblllttttbbbd pattrrribbbiiiii-pttitiiiii 2022.
Neel Nanda and Joseph Bloom. htaniinneee..rpppgpgttttooooooot hot.://aatttttttorroosssssg TransformerLens, 2022.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. Gpt-4 technical report, 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, Iya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders, 2024.
Logan Riggs and Jannik Brinkmann. Improving sae's by sqrt()-ing II and removing hoom//osts//iGsssJaaalla JBi
lowest activating features. https://www.eestt improving-sae-- sariiin------------------------- 1agg-11-and-22 -ngg
2024.
Lee Sharkey, Dan Braun, and Beren Millidge. Taking features out of superposition with sparse autoen- coders, Dec 2022. URL sattps://wwi http://wwlllgnmiin... wwwp:wwwwww...w.......... http:///psttee/000ppapaa..... resear repoor www.reertttttin--tt..err featuressee out of -tt-o-ffstprrrpiiiiin.
Anant Subramanian. Danish Pruthi, Harsh Jhamtani, Taylor Bong-Kirkpatrick, and Eduard Hovy. Spine: Sparse interpretable neural embeddings. 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks, 2017.
Alex Tamkin, Mohammad Taufeeque, and Noah D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks. 2023.
Jesse Vig. Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Simas Sakenis, Jason Huang, Yaron Singer, and Stuart Shieber. Causal mediation analysis for interpreting neural nlp: The case of gender bias, 2020.

---

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, et al. Huggingface's 's transformers: State-of-the-an natural language processing, 2020.
Benjamin Wright and Lee Sharkey. Addressing feature suppression in saes, Feb 2024.
URL https://ww/ ott.eennorrro..rr///ott///////// htt..o///ppst/////s/y/taaaaaa/
in- saes.
Zeyu Yun, Yubei Chen, Bruno A Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding linear superposition of transformer factors, 2021. URL https: /tops://ane //t:://aaxvv////a/eaa http 15949.
12

---

A Additional results on other layers and models
A.1
Pareto curves for SAEs other layers
Layer 2
Better
Better
0.000
0,000
0.025
0.025
0.050
0,050
CE Loss increase
0.075
0,075
Better
0.100
0.100
0.125
0.125
0.150
0.175
0.175
0.200
0,200
150
200
10k
20k
30k
40k
Alive Dictionary Elements
Layer 6
Better
Setter
0.00
0.00
0.05
0.05
0.10
0.10
CE Loss Increase
0.15
0.15
Batter
0.20
0.20
0.25
0.25
0.35
0.35
0.40
0.40
200
500
30k
40k
Alive Dictionary Elements
Layer 10
Botter
Bettor
0.00
0.00
0.05
0.05
0.10
0.10
CE Loss Increase
0.15
0.15
Better
0.20
0.20
0.25
0.25
SAE Type
local
0.30
0.35
0.35
e2e1ds
0.40
0.40
200
300
500
600
30k
40k
Alive Dictionary Elements
Figure 4: Performance of all SAE types on GPT2-small's residual stream at layers 2, 6 and 10. GPT2-small has a CE loss of 3.139 over evaluation set.
13

---

A.2 Pareto curves for TinyStories-IM
We also tested our methods on Tinystories- 1M a 1M parameter model trained on short, simple stories [Eldan and Li, 2023]. Figure shows key results generalising to the residual stream halfway through the model (before the 5th of 8 layers).
Note that most of our Tinystories- IM runs were for SABnraa and SAE,2e. and we did not perform several of the analyses that we performed for GPT2-small elsewhere this report. But the clear improvement in Lo and alive ____tdcct___tccccdd_e elements CE loss increase apparent for SAE,20 SAEecal diet_elements More results can be found at https: / /api. wandb. ai/links/ wtt.innessparsityy//////ee///e/e spwrsify/ykketteee Future work would test that these results hold more models of different sizes and architectures, as well as SAES trained not just on the residual stream.
Better
Better
0.0
0.0
0.1
0.1
CE Loss Increase
0.2
0.2
Better
0.3
0.3
0.4
0.4
SAE Type
local
0.5
e2e
e2e+ds
0.6
0.6
0
10
20
30
40
50
60
500
1000
1500
2000
L0
Alive Dictionary Elements
Figure 5: Tinystories- 1M runs comparing SAEwaat SAE.: and SAE. on the residual stream before the 5th of 8 layers. Tinystories-Il IM has CE loss of 2.306 evaluation set.
A.3 Comparison of runs with similar Lo, CE loss increase, or number of alive dictionary elements
Table 2: Comparison of runs with similar CE loss increase for each layer. À represents the sparsity coefficient and GradNorm the of all SAE weight gradients measured from 10k training samples onwards.
Layer
SAE Type
X
Lo
AliveElements
GradNorm
CEIncrease
local
0.8
73.5
41k
0.04
0.016
2
c2e
0.5
33.4
22k
0.24
0.020
c2c+ds
10
36.2
18k
1.55
0.015
local
4
69.4
26k
0.12
0.144
e2e
27.5
22k
0.59
0.145
e2e+ds
50
36.8
15k
3.27
0.124
local
6
162.7
33k
0.40
0.122
10
e2e
1.5
62.3
25k
0.64
0.131
e2e+ds
1.75
70.2
25k
0.24
0.144
14

---

Table Comparison of with similar Lo for each layer. represents the sparsity coefficient and GradNorm the mean norm SAE weight gradients measured from 10k training samples onwards.
Layer
SAE Type
Lo
Alive Elements
Grad Norm
CE Loss Increase
local
18.4
18k
0.04
0.101
2
e2e
1.5
18.6
20k
0.31
0.043
e2e+ds
35
17.9
15k
2.24
0.039
local
42.8
21k
0.13
0.228
6
e2e
1.5
39.7
25k
0.42
0.099
e2e+ds
50
36.8
15k
3.27
0.124
local
10
74.9
28k
0.37
0.202
10
c2c
1.5
62.3
25k
0.64
0.131
e2e+ds
1.75
70.2
25k
0.24
0.144
SAE Layer
Layer 6
SAE Layer
SAE Type
1.75
local
20
62e
MSE
15
40
1.25
10
0.75
20
10
0.25
10
11
Model Layer
Model Layer
Model Layer
Figure 6: Reconstruction squared (MSE) later layers for three SAEs with similar CE loss
increase layers 10.
A.4 Downstream MSE for all layers
Figure 6 shows that layers 2 and 10 also have the property that SAEezesd has very similar reconstruction loss SABpeat at downstreams layers, and SAFz2e has much higher reconstruction loss.
A.5 Feature splitting geometry
In Section 3 we showed that at layer 6, SABiccal is less orthogonal than SAF and SAFe2e+ds- indicating a higher level of feature splitting. In Figure we extend the analysis to runs on other layers.
In almost all cases we find that SAE, contains the most orthogonal dictionaries, followed by SAEazesd and then SAEBcaal- Perhaps surprisingly, decrease for each SAE type, we see increase in the mean of the within-SAE similarities, indicating less feature splitting. One hypothesis for this result is that the the orthogonality of the dictionary depends much more on the output performance (as measured by loss difference) or sparsity (as measured by Lo) of the model with the SAE than on the number of alive dictionary elements, though further analysis is needed.
the number of alive dictionary elements
A.6 Cross-type similarity at other layers
In Section 3.3.2 we show that downstream and local SAFs have more similar decoder directions than e2e and local SAEs. In Figure 8 we show this is true for layers 2, 6, and 10.
15

---

Layer
Simlartt
Now oricganil
0.46
0.45
0.40
0.40
30000
40000
Diciomary Elements
Layer
Simlartt
0.56
0.56
Mox ofhiggnd
Woc C00-00 ao-gooaa
0.30
0.50
0.46
0.40
0.40
0.30
0.30
0.26
0.25
300
CE
Dicioonry Elements
Layer 10
Better
0.66
SAE type
Smiento
0.80
0.56
6Ge
sOwnddx
orttogoral
0.55
0.50
0.46
0.45
0.36
0.35
0.26
0.25
300
10000 15000 20000 25000 30000 Dictionary Elements
GE
Figure 7: Mean all SAE dictionary elements of the cosine similarity the next-closest element the same dictionary. Plotted against Lo. loss increase, and number of alive dictionary elements all SAE types on runs with a variety of sparsity coefficients for GPT2-small
Layer e2e local
Layer 6
Layer 10
e2e local
e2e local
e2etds local
e2ords local
<2erds local
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Cosine Similarity
Cosine Similerity
Cosine Similarity
Figure 8: For with similar CE loss increase in layers 10, for each SAEgte and SAEetesd dictionary direction, we take max cosine similarity over all SAEncca directions.
16

---

A.7 Auto-interpretabiiity
In Section 3.4 claim that when comparing tnttrrtterrretability scores we find no difference between pairs of similar Lo. but do find SAE, is more interpretable than SAE in layers 2 and These results are presented more detail in Figure and Table
Layer
Layer
Layer
Layer
Layer
Layer
2.0
1.0
A40 repprraary dove
0.6
f
0-4-
(a) Similar Lo
(b) Similar CEE Loss increase
Figure 9: Comparison of autooiinerprettabiiity scores between SAL and SAEmea for runs with similar Lo (see Table 3) and similar CE loss increases (see Table 2). Error bars bootstraped 95% confidence interval for true mean auto-interpretability scores. Measured on approximately 200(+:2) randomly selected features per dictionary.
Table 4: Estimates of the difference between the auto interpetttprttbilltty scores for SAEctedd and SAExeal (Figure positive difference indicates SAFateds interpretable. For each comparison bootstrapping to compute 95% confidence interval and a two-tailed p-value the means are equal.
Layer
Mean diff.
95% CI
p-value
Similar Lo
2
0.01
0.07, 0.04
0.61
6
0.01
0.07, 0.05
0.71
10
0.03
0.003, 0.08)
0.33
Similar CE
2
0.08
0.002, 0.13]
0,0057
6
0.10
[0.05, 0.16]
0.00044
10
0.02
1-0.03,..00 0.08
0.41
17

---

B Analysis of reconstructed activations
We saw in Appendix A.4 that e2e-trained SAES much reconstructing the exact activation compared to locally-trained SAEs. We performed some initial analysis of why this is.
B.1
Scale
A common problem with SAEs is "feature-supression" where the SAE output has considerably smaller norm than the imput [Wright and Sharkey, 2024, Rajamanoharan et al., 2024]. We observe this well, as shown in Figure 10 for SAEx2o+ds in layer 6. Note the cluster of activations with original norm around 3000; these are the activations at position O.
3000
150
Norm of Reconstructed Acts, ||ábx0|1z
2000
1000
150
1000
2000
3000
Norm of Original Acts, ||a(x)]|2
Figure 10: A scatterplot showing the L2-norm input and output activations out SAE in layer
Table 5: La Ratio for the SAEs of similar CE increase, Table
Position
Position
Layer
6
2
10
local
1.00
1.00
1.00
0.98
0.92
0.91
e2e
0.16
0.11
0.08
0.67
0.31
0.15
downstream
0.14
0.99
0.99
0.74
0.56
0.32
We measure suppression with the metric:
||ā())]
L2 Ratio E ||a(x)]
which is presented in Table for all of the similar CE loss increase SAEs in Table Generally, SAE,2e has the most feature-suppression. This is layer-norm is applied to the residual stream before the activations are used, which can allow the network to re-normalize the downscaled activations and keep similar outputs. The downscaled activations will still disrupt the normal ratio between the residual stream before the SAE is applied and the outputs future layers that are added to the residual stream.
B.2 Direction
Both SAE and SAFzeesa do significantly worse at reconstructing the directions of the original activations than SAEacaa (Figure 11). Note, however, that comparing runs with similar CE
18

---

loss increases. SAEjocal is the only one of the three that is trained directly reconstructing these activations, and achieves this reconstruction with significamtly higher average Lo-
Overall, SAEezesd, and S2 reconstruct the activation direction in the current layer similarly well, with SAEz2erds doing better at layer 6 and but worse layer 10.
Layer local
Layer local
Layer 10 local
e2e
e2e
e2e
e2e+ds
e2e+ds
e2e+ds
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Cosine Similarity
Cosine Similarity
Cosine Similarity
Figure 11: Distribution cosine similarities between the original and reconstructed activations, for our SAEs with similar CE loss increases (Table 2). We measure 100 sequences of length 1024.
B.3 Explained variance
How much reconstruction error seen earlier (Section 3.2) due to feature shrinkage? One way to investigate this is to normalize the activations of the SAE output before comparing them to the original activation." In Figure 12, we compare the explained variance for the reconstructed activations of each type of SAE in layer 6, both with and without normalizing the activations first. Normalizing the activations greatly improves the explained variance of our e2c SAEs. Despite this, the overall story and relative shapes of the curves are similar.
"To "normalize" apply center along the embedding dimension and scale the resulting vector to have unit This is equivalent to Layer Normalization with affine transformation. We use this Layer Normalization is applied to the residual-stream activations before they are used by the network.
1.0
0.0
-2.5
Explained Variance of Normalized Activations
0.8
Explained Varience
-5.0
0.6
-7.5
0.4
-1..0
Type
Type
local
local
-12.5
c2e
c2c
e2ends
<2erds
-15.0
6
10
8
11
Model Layer
Model Layer
(a) Unmodiifidd activations
Normalized activations
Figure 12: Explained variance between activations from the model with and without the SAE inserted. Measured all later layers for our set of SAEs with similar CE loss increase in layer 6 (Table 1). In (b) we apply Layer Normalization to the activations before comparison.
19

---

C Effect of gradient norms on the number of alive dictionary elements
One our goals to reduce the total number of features needed over a dataset (i.e. the alive dictionary elements), thereby reducing the computational overhead of any method that makes use of these features. We showed in Figure 4 that SAEate and SABe2erdd consistently use fewer dictionary elements for the same amount of performance when compared with SABpecal- We also that SABzzo-ds uses fewer elements than S2 for layers 2 and 6 but not layer 10.
Notice in Table 2, however, that the number of alive dictionary elements is negatively correlated with the norm of the gradients during training. This begs the question: If we increase the learning rate, is it possible to maintain performance in Lo CE loss increase while also decreasing the number of alive dictionary elements?
Layer
Batter
Bettar
0.000
0.000
0.025
0.025
TELosslnaredse
0.075
0.076
Detter
0,100
0.125
0.125
0.150
0.150
0,200
0,200
00
100
150
200
10K
20k
30k
40k
Dictonary Elements
Layer
0.00
0.00
0.10
0,10
CELossinereeee
0.15
0.15
Better
0.20
0.20
0.25
0.25
0.35
0.36
0.40
0.40
LO
Alive Dictionary Elements
Layer 10
Better
Better
0.05
0.06
0.10
0.10
EELossincesse
Deteer
0.20
0.20
Ir
0.0001
0.0005
0.30
0.001
0.30
0.35
0.005
0.36
0.01
50
100
150
200
250
300
10k
20k
30k
40k
LO
Alive Dictionary Elements
Figure 13: Varying the learning for SAFreaa on layers 2, and 10. All other parameters the the local runs listed similar CE loss increase Table
In Figures 13, 14, 15, we show the effect that varying the learning rate has on performance for SAEpcaal SAEc2 and SAE respectively. In all cases, that learning rates higher than
20

---

Layer 2
Better
Better
0.000
0.000
0.025
0.025
CELossinue CELossnnessse
0.075
0.076
Defter
0.100
0,100
0.150
0.180
0.200
0.200
50
100
150
200
10k
20k
30k
40k
Dictonary Elements
Layer
0.00
0.00
0.10
0.10
CELossihorease
0:15
0.15
Better
0.25
0.25
0.35
0.36
0.40
0.40
LO
Alive Dictionary Elements
Layer 10
Better
Better
0.05
0.05
@ELossnnceeate
0.10
0,10
Bater
0.20
0.20
Ir
0.0005
0.30
0.001
0.30
0.35
0.005
0.35
0.01
50
150
200
250
300
10k Alive Dictionary Elements
30k
40k
LO
Figure 14: Varying the learning rate for SAE, layers 2, 6 and 10. All other parameters the the local runs listed the similar CE loss increase Table
default of 0.0005 require fewer dictionary elements for the same level of performance on CE loss increase. We also see that these runs with higher learning rates (up to limit) can have better T-p vs CE loss increase frontier at high sparsity levels and is similar or worse at low sparsity levels.
This effect appears be more pronounced for SAE and SAE than SAEbocal- indicating that e2e SAES may require even fewer alive dictionary elements compared to SAE than what is presented in the figures in the main text.
While not shown in these figures, a downside using learning rates larger than 0.0005 is that it can cause the Lo metric steadily increase during training after initial period of decreasing. This occurred for all of our SAE types, and was especially apparent later layers. Due to this instability, persisted with learning rate of 0.0005 for main experiments. We expect that training tweaks such as using sparsity schedule could help remedy this issue and allow for using higher learning rates.
21

---

Layer
0.000
0.000
0.050
0.050
CELossthorebee
0.075
0.075
Detter
0.100
0,100
0.125
0.125
0.175
0.175
0,200
0,200
L0
Alive Dictionayy Elements
Layer 6
Better
Better
0.05
0.06
0.10
0,10
Enossinceesse
Beter
0.20
0.20
0.30
0.30
0.35
0.35
50
100
150
200
10k
30k
40k
L0
Alive Dictionary Elements
Layer 10
Better
Beller
0.00
0.00
0.05
0.05
TELosalnaresse
0.15
0.15
Better
0.20
Ir
0.25
0.0001
0.25
0.30
0.001
0.30
0.005
0.40
0.40
50
100
150
200
250
300
10k
20k Dictionary
30k
40k
Figure 15: Varying the learning rate for SAEa on layers 2, and 10. All other parameters are the same as the local runs listed in the similar CE loss increase Table 2.
22

---

D Experimental details and hyperparameters
Our architectural and training design choices were selected with the goal of maximizing Lo vs CE loss increase Pareto frontier of SAEooaal- We then used the same design choices for SAEz2e and SAE22e-d, Much of our design choice iteration took place on the smaller Tinystories- Im due to time and cost constraints.
Our SAE encoder and decoder both have a regular, trainable bias, and use Kaiming initialization. To form our dictionary elements, we transform our decoder to have unit norm on every forward pass. We do not employ any resampling techniques [Bricken et al., 2023] it is unclear how these methods affect the types of features that are found, especially when aiming to find functional features with e2e training. We clip the gradients norms of our parameters only affects the very large grad norms at the start of training and the occasional spike later in training. We do not have strong evidence that this is worthwhile to do on GPT2- -maal,, and does comes at a computational cost.
fixed value (10 for GPT2-small). This
We train for 400k samples of context size 1024 on Open Web Text with an effective batch size of 16. We use a learning rate of 5e - 4, with warmup of 20k samples, a cosine schedule decaying to 10% of the max learning rate, and the Adam optimizer [Kingma and Ba, 2017] with default hyperparameters.
For SAEx2e+ds. we multiply our KL loss term by a value of 0.5 in our implementation. Note that if we instead fixed this value to 1 and varied the other loss coefficients, we would also need to vary other coefficients such as learning rate and effective batch size accordingly, which may have been difficult. This said, fixing this parameter to and having fewer overall hyperparemeters may be a better option going forward, as it turns out to be difficult to tune the other coefficients in this setting anyway. We set the total_ _adeecfeff (i.e., the coefficient that multiplies the downstream reconstruction MSE, denoted 33 in Equation 1) to 2.5 for layers 2 and 6, and to 0.05 for layer 10. Note from Equation that this coefficient gets split evenly among all downstream layers. It's likely that a different weighting of these parameters is more desirable, but we did not explore this for this report.
It's worth noting that we did not iterate heavily on loss function design for SAEc2e+ds so it's likely that other configurations have better performance (c.g. having different downstream reconstruction loss coefficients depending on the layer, and/or including the reconstruction loss at the layer containing the SAB).
Note that in our loss formulation (Section 2), we divide our sparsity coefficient À by the size of the residual stream dim(a (x)). This is done in an attempt to make our sparsity coefficient robust to changes in model size. The idea is that the L1 score for an optimal SAE will be function of the size of the residual stream. However, we did not explore this relationship in detail and expect that other functions of residual stream size (and perhaps dictionary size) are more suitable for scaling the sparsity coefficient.
For GPT2-small apollo-researe http///yy1o0000-opeeeeee version of OpenWebText ([Gokaslan and Cohen, 2019I) (released under the license CCO-1.0). The tokenization process is the same as was used in GPT2 training, with a 'BOS' token between documents.
stream the dataset https: htups://huginggface ctaeee.co///eeeee co/datasets/
tokenizer--aeaee gpt2 which is a tokenized
We evaluate our models on 500 samples of the Open Web Text dataset (a different seed to that used for training). We consider dictionary element alive if activates at all on 500k training tokens.
Note that information from all of our runs are accessible in this Weights and Biases ([Biewald, 2020]) report, including the weights, configs and numerous metrics tracked throughout training. The SAES from these runs can be loaded and further analysed in our library ApolloResearch/ e2e_sa//
com/
We used NVIDIA A100 GPUs with S0GB VRAM (although the GPU was saturated when using smaller batch sizes that used 40GB VRAM or less).
Our library imports from the TransformerLens library [Nanda and Bloom, 2022] (released under the MIT License), which is used to download models via HuggingFace's s Transformers library [Wolf et al., 2020] (released under the Apache License 2.0). GPT2-small is released under the MIT license. The Tinystories- I M model is released under the Apache License 2.0 and it's accompanying dataset is released under EDLA-Sharing
23

---

E Varying initial dictionary size and number of training samples
E.1
Varying initial dictionary size
In Figure 16 we show the effect of varying the initial dictionary size for our layer 6 similar CE loss increase SAEs Table For all SAE types, we see Lo vs CE loss increase improve with diminishing returns as the dictionary size is scaled up, capping out at dictionary size of roughly 60. This comes the cost of having more alive dictionary elements with increasing dictionary size.
It's worth mentioning that, after preliminary investigation on Tinystories- 1M, it's possible to reduce the dictionary ratio to 5 times the residual stream and still achieve good Lu CE loss increase tradeoff, as well as reducing the number of alive dictionary elements. See this Weights and Bíases report for details https://wandie ai httpaaac.aaa.pppss.///ttttttttte im-riieeeii-raiioo reports/Seee SeattssSSaaanng ing- ditiddict-- size- tinystories-blockee-eee w-w.kks----ayyrriiee-eeeeeee Val1aaaa21ll.22aaaa0
local
Butler
Loss Increase
Better
LO
Alive Dictionary Elements
c2e
Better
Better
Loss Increase
Better
50
100
150
200
10k
20k
30k
40k
Alive Dictionary Elements
e2e+ds
Better
0.0
0.0
0.1
0.1
CE Loss Incresse
0.2
0.2
Better
0.3
ralio
0.3
5.0
0.4
20.0
0.4
60.0
100.0
0.5
0.5
200
10k
20k
Alive Dictionary Elements
Figure 16: Sweep over the SAE dictionary size for layer (where 'ratio' is the size of the initial dictionary divided by the residual stream size of 768). All other parameters are the same as in the similar CE loss increase runs in Table 2.
24

---

E.2 Varying number of training samples
In Figure 17 we analyse the effect of varying the number of training samples for each SAE type on layer 6 of our similar CE loss increase SAEs. For SAE training for 50k samples is clearly insufficient. The difference between training on 200k, 400k, and 800k samples is quite minimal for both To vs CE loss increase and seve_dict_elemente
CE loss increase.
For SAEeee, samples but with diminishing returns. contrast to SAF]oeal- we see a steady improvement in siive_dict__p elements vs CE loss increase as we increase the number of samples. Note that training SABe2e or SAEe2ords for 800k samples takes approximately 23 hours on a single A100.
improvements To CE loss increase when increasing from 50k 800k
For SAE,2e-ds the Lu CE loss increase and alive dlicteelements vs CE loss increase improves up until 400k samples where performance maxes out.
local
0.0
0.0
Loss Increase
150
10k
20k Alive Dictionary Elements
LO
e2e
Betler
Beller
Loss Increase
50
100
150
200
10k
20k
30k
40k
LO
Alive Dictionary Elements
e2e+ds
Better
Better
0.0
0.0
0.1
0.1
CE Loos Incresse
0.2
0.2
samples
0.3
20000
0.3
50000
0.4
200000
0.4
400000
800000
0.5
0.5
100
150
10k
20k
L0
Alive Dictionary Elementa
Figure 17: Sweep over number of samples trained on layer 6. All other parameters are the same as the similar CE loss increase runs in Table 2.
25

---

Robustness of features to different seeds
We show Figure 18 that, for a variety of sparsity coefficients and layers, our training runs are robust to the random seed. Note that the seed is responsible for both SAE weight initialization as well as the dataset samples used in training and evaluation.
Seed Comparison layers 2-6-10: LO CE Loss Increase
0.12
CE Loss Increase
0.10
0.08
0.04
SAE Type
local
0.02
e2e
62srds
40
60
80
100
120
140
160
Figure 18: sample of SAEs for layers 2, and for all run types showing the robustness of SAE training two different seeds.
26

---

G Analysis of UMAP plots
To explore the qualitative differences between the features learned by SAEtocal and SAEe2esds we first visualize the SAE features using UMAP [McInnes et al., 2018] (Figures 20).
G.1 UMAP of layer 6 SAEs
Although there is substantial overlap between the features from both types of SAE in the plot, there some distinct regions that are dense with SAE,2o+ds features but void of SAEoocall and vice versa. We look at the features in these regions along with features in other identified regions of interest such as small mixed clusters in layer to G in Figure 19, and provide human- suman-genneated overview of these features below. Features from this /www. neuronpedia. org/ org/gpt2sm gttpsmmapmollooooooooom apollojt For each region, we also share links to lists of features in that region which go to an interactive dashboards on Neuronpedia.
GPT2-small in more detail. We label the regions of interest from
UMAP plot can be explored interactivelyy at https /
Layer 6
eZe+ds
local
Figure 19: UMAP plot of SAEstesd and SAEweaa features for layer 6 on runs with similar CE loss increase in GPT2-small.
27

---

Region A (SAE,2erds features (18). SAEa features (91))
Many of these features appear to be late-context positional features, miscellaneous tokens that only activate particularly late context positions. It may be the case that SAE,2osds has fewer positional features than local, as indicated by the 18 2eEsd vs 91 SAEpoca local features in this region (and similar in surrounding reasons). This said, we have not ruled out whether positional features for SAE are found elsewhere in the UMAP plot.
Region B (SAEazetds features (48). SAEatcall features (2))
This region mostly contains features which activate on < </ndoftext, >> tokens, in addition to some newline and double newline. These are tokens that mark the beginning of a new context. Seemingly SAEs2e-ds contains many more distinct features for < </ndofteett than SABpccal-
Region C (SA E2eeds features (20). SAFia features (31))
Region C potentially suggests more feature splitting happening in SAE than SAEazedss For SA each feature activates most strongly on tokens "by" "from" in a broad range of contexts. For SAEmcal, each feature activates most strongly in fine-grained contexts, such as "goes by" vs "led by" vs By" vs "stop by" vs "<media>, by author" vs "despised by" vs "overtaken by" vs "issued by" vs "step-by-step/ case-by-case / Trame-by-frame vs "Posted by" vs "Directed by" vs "killed by" vs "by""
Region D (SAE features (11). SAl features (19))
These features all activate on "at" in various contexts. As in Region C. the SAEe2e-d: features appear less fine-grained. Examples of SAEjeaa features not present in SAEczesds "Announced at" or "presented at" or "revealed at" feature (https: hamps://wwww http://wwuooooeeeeeeeeooo scl-ajt/40197), "At" in technical contexts (https: a org/ gpt2-smal1/
org/ org/gpt2 1aaal16--aa res..... res.
Region E (SAE features (3). SAEeocal features (67))
All features appear boost starting words which would come after paragraph or full stop to start a new idea, such as "Finally", "Moreover", "Similarly", "Furthermore", "Regardless", "However" and so on. They seem be differentiated by perhaps activating in different contexts. For example https: //www/wwww///////////// neuronpedia, org/ on full stops and newlines technical contexts htt/gtt2-aaa smal1/6-res hec-res_scleeeeeseeereeseeeeesse scl-ssttttt... ajjt/424424 activates can predict things like "Additionally", "However", and "Specifically". On the other hand, https: //www/www.../ neuronpedia. org/ org/gpt2- res scl- scl-ajt/1351s activates on full stops in baking recipes so can predict things like "Then", "Afterwards", "Alternativll"" "Depending" and so on.
Region F (SAEaaetds features (19). SA features (8))
These seem mostly similar to Region E. It's not clear what distinguishes the regions looking at the feature dashboards alone.
Region G (SAE features (41). SAElocal features (71))
The features in both SAEs seem to activate fairly specific different words phrases. There no obvious distinguishing features. It's possible that SAE features tend to activate more specifically and fewer tokens than the corresponding SAFiacal features. An example of this can be seen when comparing https:/ h/pp:://www... /twp://neuroonpdeaaa. org/ otg/gptt-gaaee/ sept2-small/e scefr-ajt/13910 (a SAF feature), with https: //wp://www//////o/ neuronpedia org/ wtggggtt-geeee gt/ggpttssmaa small/ srrs_scl-ajt/Aeee
(a SABbocal feature).
G.2 UMAP of layer 2 and layer 10 SAEs
In Figure 20, we show UMAP plots for layers 2 and 10. We interpret a single region from layer in the next section.
28

---

otendh
odendh
Figure UMAP of SAEatesd: SAFiccal features for layers 2 10 runs with similar CE loss increase in GPT2-small.
G.3 Region H in layer 10 (SAEetesds features (2). SAEtocal features (593))
While individual features in this region interpretable, there is obvious uniting theme semantically. There is, however, a geometric connection. In particular, these are features that point away from the 0th PCA direction in the original model's activations (Figure 21).
Local SAE layer 10
Dawnstream SAE in layer 10
0.4
Similanty 0th PCA direction
-0.2
0.0
-02
-0.4
Figure 21: The UMAP plot for SAEgterd: and SAFiccal directions, with points colored by their cosine similarity to the 0th PCA direction.
The Oth PCA direction is nearly exactly the direction of the outlier activations at position 0 (see also Appendix B). Activations in this direction are tri-modal, with large outliers at position 0 and smaller outliers at end-of-text tokens (Figure 22).
We can measure how well an SAE preserves a partieular direction by measuring the correlation between the input and output components in that direction. Our SAFezesd. faithfully reconstructs the activations in this direction position 0 (r 0.996),
at other positions (r 0.262). This is particularly poor reconstruction compared to SAEiccal other PCA directions (Figure 23a).
29

---

position end-of-text
100
150
200
3000
3050
3100
3150
3200
Activetion in PCA direction
Figure 22: A histogram of the Oth PCA component of the activations before layer 10.
1.0
input-output correlation
0.8
0.20
KL divergence
0.6
0.15
0.4
0.10
SAE type local
0.2
e2e+ds
0.0
0.00
10
15
20
25
10
15
20
PCA direction
PCA direction
(a) Reconstruction faithfulness
(b) Output sensitivity to resample ahlating
Figure 23: For each PCA direction before layer 10 measure qualities. The first is how faithfully SAEoea and SAEgteed: reconstruct that direction measuring correlation coefficient. The second is how functionally- important the direction is, as measured by how much output the model changes when resample ablating the direction.
SABatesds's 's poor reconstruction of the activations in this direction implies that the differences may not be functionally relevant. We can measure this by resample ablating the activation in this direction all non-zero positions. This means perform the following intervention in forward hook:
a(x); - a(x): Pa(x), + Pa(z');
Where a(x); ; is the activation at position > 0, a(2') و is the resampled activation for a different input x' and position j > and a projection matrix onto the Oth PCA direction.
After performing this ablation, the kl-divergence from the original activations only 0.01. This difference is smaller than repeating the experiment for any other direction in the first 30 PCA directions (Figure 23b).
This means that the exact value of this component of the activation (at positions > 0) is mostly functionally irrelevant for the model. SAE still captures the direction faithfully, as it is purely trained to minimize MSE. While SAEezerds fails to preserve this direction accurately, this seems to allow it to have a cleaner dictionary, avoiding SAE 's cluster of features that point partially away from this direction.
30

---

H
Training time
The training time for each type of SAE in GPT2-small is shown in Table 6. We see that e2c SAEs are 2-3.5x slower than SAEpocal- Note that one can reduce training time with little performance cost by training on fewer that 400k samples (Figure 17) and/or using an initial dietionary ratio of less than 60x the residual stream size (Figure 16). Using locally trained SAEs as initialization for e2e SAEs or training multiple SAEs at different layers concurrently are also possible solutions.
Table Training times for different layers and SAE training methods using a single NVIDIA A100 GPU on the residual stream of GPT2-small at layer 6. All SAEs are trained on 400k samples of context length 1024, with dictionary size of 60x the residual stream size of 768.
Layer
SAEaccal
SAE,20
SAEc2e+ds
2
3h 45m
12h 24m
12h 30m
6
4h 45m
11h 20m
1 11h 24m
10
5h 19m
10h 12m
10h 20m
31

---

Table 7: Faithfulness subject-verb agreement when replacing the activations with SAE outputs.
(a) Similar
(b) Similar Lo
Layer
Type
Simple
Across PP
Across RC
Within RC
Layer
Type
Simple
Across PP
Across RC
Within RC
102.4%
100.6%
local
105.7%.
2
e2e c2c4ds
104.1%
100.5%
98.8%
102.1%
ele c2o+ds
100,8%
101.0%
101 .6%
98.9%
105.6%
101.9%
99.1%
102.2%
104.8%
109.3%
97.9%
98.8%
local 100.6%
96.0%
79.0%
102.8%
local
99.6%
93.2%
89.1%
99.1%
94.5%
104.7%
e2e
ele+ds
95.2%
79.2%
95.5%
98.7%
e2et-s
95.2%
79.2%
95.5%
98.7%
local
97.2%
79.3%
85.2%
106.0%
local c2c e2e+ds
95.1%
77.8%
68.6% 18.6%
106.955
10
c2c
92.5%
90.5%
89.1%
100.0%
10
92.5%
90.5%
89.1%
100.0%
104.7%
84.2%
I Faithfulness of SAEs on subject verb agreement task
Our main evaluation metrics presented in Section 3 measure the functional importance of the features learned by the SAEs on the next-token language modeling task used to train the model. We also experimented with evaluating the SAEs on downstream task: how faithfully the dictionaries represent the information the model uses to perform subject-verb agreement. This task is directly inspired by the analysis in Marks et [2024].
1.1 Methodology
We use datasets from Finlayson et al. [2021] with 4 variations of subject-verb agreement task:
Simple:
The parent/s is/are
Across participle phrase (PPy: The secretary/secretaries near the cars has/have
Within relative clause (RCy: The athlete that the manager/managers likes/like
Across RC: The athlete/athletes that the managers like doldoes
For each template,
1000 datapoints with different subjects and verbs. For each input, can compute the logit difference that the model assigns to the correct and incorrect forms of the verb.
Following Marks et al. [2024] we compute the faithfulness of this logit difference when intervening on the network's ' activations. Let m represent the mean logit difference between the correct and incorrect verb forms across the dataset. Let M be the original model and M be the model under some intervention. We measure the faithfulness of the intervention m M) m where Ø represents ablating the entire residual stream.
[M(M)-ml m2)
A faithfulness of 0%% thus means M performs no better than random, while faithfulness of 100% means the intervention does not change performance. Faithfulness numbers greater than 100% mean the model is, on average, more confident in the correct verb with the intervention than without.
1.2 Faithfulness with complete SAEs
We first test the faithfulness the models with the SAEs inserted (Table 7). An SAE which preserves functionally relevant features in the activations would have faithfulness close to 100% on all tasks the model is trained to perform.
While all SAFs preserve most of the logit-difference, there is significant variation across SAE types, layers, and tasks. The local SAEs in layer 10 have the worst faithfulness, although SAF in layer 6 also has poor faithfulness across participial phrases.
I.3 Faithfulness with small number of features
We are also interested if end-to-end training helps concentrate the functional relevance into a few specific SAE features. We thus ranked SAE features in Layer Similar Lo SAEs by their indirect
32

---

Simple
Across Participial Phrases
1.2
1.0
0.8
0.6
0.4
0.2
Faithfulness
0.0
Across Relative Clause
Within Relative Clause
1.2
1.0
0.8
0.6
0.4
local
e2e
0.2
e2e+ds
0.0
80
100
80 100
Number SAE features preserved
Figure 24: Faithfulness when mean-ablating all SAE features except the top with largest indirect effect.
Experiments done with the similar Lo SAEs on layer 6.
effect, measured by the change in faithfulness score when mean ablating that feature.? 7 We then mean ablate all SAE features except the & with largest indirect effect (Figure 24). We see SAE, needs comparatively few features achieve high faithfulness scores, but SAEoocall and SAEc2o+d: have roughly similar curves.
1.4
Discussion
Despite certain (SAE type, layer) combinations showing superior performance, there are clear patterns between SAE types across tasks. These results indicate that e2e SAEs do not provide an obvious benefit on the selected downstream tasks. While e2c SAEs demonstrate benefits for the language modeling task on the full WebTee distribution, further work would be needed to find specific tasks and sub-distributions of task-specific results are very noisy and would be necessary to aggregate across many tasks and templates to differentiate between SAE training methodologies.
which they provide the most benefit. Or, perhaps these sort
WWen mean ablating SAE feature take it's position--peciffc value.
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims made are supported in the main text with additional support in the Appendices.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
The main limitation, longer training times, is discussed in the results (Section 3) and Appendix We also mention Appendix A.2 how the results may be strengthened if trained on wider variety of model sizes and architectures.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The methods are laid out in Section 2. and detailed hyperparameters and design details are given in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
35

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Repository attached to the submission (kttps://github. com/ ApolloResearch/eee saaraarrrrheeeeeee)
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
httpii/gggiieee/oeeuuuiiisseee ionooooooloo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (uttps://nips wwtpcccwccgguuaiiaecec
for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All details are presented in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide statistical significance testing and report confidence intervals across our geometrics comparisons and auto-interprecabiliiyy We also provide some results showing the robustness seeds (Appendix F), though do statistical significance testing on pareto differences between our methods over various seeds due to computational feasibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
36

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Details on compute resources given at the end of Appendix D and in Appendix H.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https / http///eerripp.eeeee
Answer:
[Yes]
Justification:
We conform all aspects of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See our impact statement (Section 6).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
37

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The sources and licenses of all dataset, model and core libraries used are given in Appendix D.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
38

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasttssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Codebase for reproducing results provided alongside this submission. No other assets provided.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing nor experiments with human subjects were undertaken.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
39

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "sZYl0wUSVr", "reviewer": "Reviewer_byuk", "summary": "The paper proposes new framework to improve the interpretability of large language models. Inspired by the work proposed in Anthropic's \"\", the proposed work replaces the original reconstruction loss in the sparse auto-encoder (SAE) with KL-div loss, and additionally add further constraint to minimize the error between reconstructed downstream activations and original activations in the LLM. Empirical results demonstrates the adavantage of the proposed new SAE_{e2e} architecture series.", "questions": "I appreciate it if the authors could please see the above weakness for my questions.", "limitations": "Yes the authors have adequately discussed the limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper considers new formulation to further improve the interpretability of the large language model.", "weaknesses": "W1: The authors may further explain the intuition between the evaluation metrics. The current version of the evaluations is not explicitly clear to me why the proposed method is advantageous in comparison to the baselines. \n\nW2: It is now clear how the KL divergence is implemented to replace the reconstruction loss. KL is only practically useful when the distributions are available. However, in this problem, it is unclear how the distribution loss between the two deterministic feature outputs are computed.  The motivation of the replacement between KL divergence and the L2 reconstruction is also not very clear to me, and these two losses should be ultimately equivalent and optimal at the same parameters global optima, although the loss changes the optimization landscapes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. We believe there may be a core misunderstand of our method which we clarify in the response to Weakness 2.\n\n## Response to Weaknesses\n### Weakness 1\n> W1: The authors may further explain the intuition between the evaluation metrics. The current version of the evaluations is not explicitly clear to me why the proposed method is advantageous in comparison to the baselines.\n\n\nIn the paper we present three main components to evaluating the performance of an SAE. Below we give some more intuition behind them.\n1. If we are to sum up dictionary elements of our SAE and use those in place of the real model activations, how many dictionary elements do we need to use to get similar performance to the real model? Figure 1 (Left) shows that we require fewer dictionary elements on average (L0) to explain the same amount of CE loss degradation (which is the difference between the CE loss of the original model and the model when splicing in SAE features).\n2. In Figure 1 (right), we measure the total number of dictionary elements needed in the SAE to explain the network’s behavior over the whole dataset. While less important than the metric above, having fewer dictionary elements in total means that there are fewer variables needed to describe the behavior of the network on your distribution. Note that “alive dictionary elements” is the effective measure of the number of variables needed, as some dictionary elements become forever inactive during training (it’s worth noting that the recent work of Gao2024 (https://cdn.openai.com/papers/sparse-autoencoders.pdf) has reduced the killing of dictionary elements with better initialization and an augmented loss term, though we don’t have a reason to expect that employing these techniques will change the structure of the pareto curves presented in our figure.\n3. We want to know whether we can understand each SAE dictionary element individually. We implemented automated interpretability scoring, which provides a scalable and unbiased quantitative measure of interpretability. For an introduction to this technique, see  Bills et al. [2023] https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html. In short, to score the interpretability of a single SAE feature, GPT-4-turbo is prompted to write a short english-language explanation of the feature, then GPT-3.5-turbo is prompted to try to predict the specific activations of the SAE feature on each token, as we explain in section 3.4. From the automated interpretability scoring, we find strong evidence that the features in our e2e+ds SAEs are not less interpretable than the features in the local SAEs at a similar amount of model performance explained. We even find that they are significantly more interpretable in some layers, such as in layers 2 (p = 0.0053) and 6 (p = 0.0005). \n\nIt may be useful to note that several other works in the area have used the metric outlined in point 1 to evaluate SAEs e.g. Figure 5 in Bills2024 (https://cdn.openai.com/papers/sparse-autoencoders.pdf), Rajamanoharan2024 (https://arxiv.org/abs/2404.16014)(note that this uses a “loss recovered” term which is calculated from the average CE loss of the model when splicing in the SAE), Kissane2024 (https://arxiv.org/abs/2406.17759). Additionally, many works have used automated interpretability to evaluate the interpretability of SAE dictionary elements (e.g. Cunningham2023 (https://arxiv.org/abs/2309.08600), Templeton2024 (https://transformer-circuits.pub/2024/scaling-monosemanticity/),\n\nIn addition, in the Author Rebuttal at the top, we've run additional evaluations. The results measure how much task performance degradation there is when SAE dictionary elements are spliced into the model on some of the subtasks used in Marks2024 (https://arxiv.org/abs/2403.19647).\n\n\n### Weakness 2\n> However, in this problem, it is unclear how the distribution loss between the two deterministic feature outputs are computed.\n\nThe KL divergence is computed for the output of the LLM. This output is a probability distribution over next tokens, motivating our use of KL divergence.\n\n> The motivation of the replacement between KL divergence and the L2 reconstruction is also not very clear to me, and these two losses should be ultimately equivalent and optimal at the same parameters global optima, although the loss changes the optimization landscapes.\n\nFor an SAE with perfect reconstruction, both L2 and KL loss would be zero. However, the sparsity_loss and reconstruction_loss terms trade off against one another (given a fixed dictionary size). This means that the global optimum does not result in an SAE with perfect reconstruction.\n\nWe use KL divergence rather than L2 reconstruction at the output as the outputs are probability distributions (and the original network was trained with the standard language modeling loss which is the KL divergence between its outputs and the one-hot labels). For SAE_local and SAE_e2e+ds, we have a loss term at intermediate layer(s) in the network. Here, we use the L2 reconstruction as these activations are not probability distributions.\n\n## Summary\nWe thank the reviewer again for their comments and questions, and kindly ask that they reassess the review score if their concerns were adequately addressed by this response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hy7ypt5MOc", "reviewer": "Reviewer_TS92", "summary": "This paper proposes a way to train sparse autoencoders (SAEs) that encourages them to learn features that are causally relevant to the model's output. This is done by replacing the usual SAE reconstruction objective - an L2 penalty between original activations and their reconstructions - by the KL divergence between the model's output distribution and the distribution when activations are replaced by their reconstructions. Additional loss terms encourage the reconstructions to lead to downstream layer activations similar to the original downstream layer activations. The sparsity penalty on feature activations is kept as in the \"vanilla\" SAE.\n\nThe paper finds that this approach is a clear Pareto improvement over vanilla\nSAEs w.r.t. the tradeoff between sparsity of the feature activations (L0) and\nloss recovered when using reconstructions. An automatic interpretability pipeline is used to compare the features learned to vanilla SAE features. A statistically significant advantage for e2e+ds SAEs is found.\n\nThe paper also attempts to avoid a potential failure mode of the approach, whereby the SAE may learn to exploit different pathways through the model in order to drive down the KL divergence to the original output distribution, because this is easier from an optimization point of view (as the reconstructions are no longer optimized to match the original activations at the layer where the SAE is applied). This is why one of the SAE variants considered includes an L2 penalty for downstream layer activations too. Results here show that using only the KL penalty in the loss produces very different downstream activations compared to adding these additional terms, suggesting that the", "questions": "- did you try the \"next layer only\" SAE variant described in the weaknesses?\n- how can we get a more fine-grained picture of how individual features change between the vanilla SAE and the e2e ones?", "limitations": "- I think the main limitation that I would have loved to see addressed more in the paper is that the evaluations are somewhat indirect w.r.t. the main claim of the paper. Even if in some average sense the e2e SAEs' reconstructions lead to better KL divergence, we still don't know how this plays out on the level of individual features.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- it is an important open problem whether SAEs trained solely using activations from some LLM layer will learn all causally important features for the LLM at this layer if trained without supervision from downstream activations. The paper makes some progress on this problem, suggesting that \"vanilla\" SAEs may struggle to learn such causally relevant features.\n- the methodology is careful and often considers alternative hypotheses or potential pitfalls in the analyses. \n- the paper is clearly written", "weaknesses": "- given that the KL divergence to the output distribution is incorporated in the\nloss function, it is not that surprising that the methods in this paper have\nbetter values of the loss recovered metric (to their credit, the authors\nacknowledge this). To truly conclude the superiority of the suggested method for\nsurfacing *individual* causally important features, it would be extremely helpful to have some external (to the KL metric)\nevaluation. \n- To some extent the paper shows such evaluations. For example, in Appendix G.3 it is shown that vanilla SAEs represent well a feature that is not causally important, and the e2e SAEs in turn represent it poorly. However, what is *really* required to establish superiority is the opposite: to exhibit a causally important feature not represented by vanilla SAEs, but represented by e2e ones.\n- the key problem when not using the same-layer activations as a reconstruction target for the SAE is (as the authors describe) that the optimization may prefer to learn features that achieve a good KL divergence value eventually, but do so through \"pathological\" pathways in the model. To fix this, the authors encourage closeness with activations starting from the next layer up. However, such \"pathological\" pathways may exist in a single layer (this was shown e.g. here https://arxiv.org/abs/2311.17030). So it is unclear whether the problem has been fully overcome or rather restricted to a more narrow part of the model. Furthermore, even if reconstructions of downstream layers are similar to the true activations of these layers, this does not establish that the individual features themselves are not \"pathological\". Again, some additional, *per-feature* evaluation is needed to make the (strong) claims of the paper more believable.\n- out of the three kinds of SAEs considered - vanilla, e2e, e2e+ds - there seems to be a missing one implied by the others: an SAE that encourages faithful reconstructions only of the *next* layer activations, and does not involve the KL divergence loss. Would such an SAE lead to similar improvements? If so, this would be stronger evidence, as the KL divergence w.r.t. the final output distribution won't be a part of the loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for engaging with our paper and giving well-considered feedback.\n\nRegarding the main limitation you’ve mentioned, we agree that it would be helpful for the paper’s narrative to have other metrics of functional importance than KLDiv, including evaluations on an individual feature level. However, there is an issue with this in practice, which relates to your comment\n> To some extent the paper shows such evaluations. For example, in Appendix G.3 it is shown that vanilla SAEs represent well a feature that is not causally important, and the e2e SAEs in turn represent it poorly. However, what is really required to establish superiority is the opposite: to exhibit a causally important feature not represented by vanilla SAEs, but represented by e2e ones.\n\nThe extent to which features exist in one dictionary and do not exist in another is, unfortunately, not a binary concept in practice. We often find that there are directionally similar, but not identical features shared across dictionaries. Or there are clusters of features that are overrepresented in one dictionary and underrepresented (but not absent entirely) in another. Therefore demonstrating that “a causally important feature not represented by vanilla SAEs, but represented by e2e ones” may not reflect the metric that matters overall: the global functional importance of all the features.\n\nNevertheless, we agree that it would be desirable to be able to identify compelling narratives about individual features across dictionaries and their improved functional importance. Unfortunately the differences are matters of degree – quantitative rather than qualitative, which makes qualitative stories more difficult to tell. We were able to identify one such qualitative story. However, despite efforts to identify qualitative stories in the direction suggested by the reviewer (“what is really required to establish superiority is the opposite: to exhibit a causally important feature not represented by vanilla SAEs, but represented by e2e ones.”), we found that the data did not suggest these qualitative narratives despite the quantitative differences being present on the global level.\n\nAs for other external evaluations that don’t directly measure the CE loss over the same distribution used to train the SAE, we have run some additional experiments on a selection of downstream tasks. These are described in the Author Rebuttal at the top of this page\n\n> When not using same-layer reconstruction... it is unclear whether the problem has been fully overcome or rather restricted to a more narrow part of the model.\n\nWe agree that our existing e2e+ds implementation does not eliminate the problem but rather restricts it to a narrower part of the model. Our initial concern with adding a reconstruction loss term at the current layer was that it would reconstruct too many non-functional features present in the residual stream at the current layer. Though we have come to agree that the difference between reconstructing the current layer and a subsequent layer is merely quantitative rather than qualitative. As mentioned in the \"Other variations of e2e+ds\" section in the Author Rebuttal, we unfortunately have not run the extensive sweeps on the e2e+ds variations needed to give an insight to this question empirically, so it is possible that additionally reconstructing the current layer's activations would perform just as well on the L0-CE Loss Pareto while further restricting the computational pathways available to the model.\n\nRegarding the question\n> did you try the \"next layer only\" SAE variant described in the weaknesses?\n\nplease see our comment on this in our Author Rebuttal at the top of the page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gaSz8oKQty", "reviewer": "Reviewer_SeEL", "summary": "The authors present their observations on modified sparse autoencoder (SAE) learning. In the proposed method SAE is trained to reconstruct original model weights (features). SAE is optimized with the KL-divergence loss between the model output and the output at the same location in the original model. An additional variant is training with the reconstruction loss computed after each layer between the original model and SAE output. The sparsity loss term is added to increase the number of weights that will converge to zero. The authors claim that their method explains the model features in a better way with fewer dictionary elements.", "questions": "- Can you please explain the diagram in Figure 1 why some blocks are of different shapes?\n- It looks like that the Sec. 3.4 result is trivial, if we don’t encourage the SAE to reconstruct after wach activation, that will be the result.\n- What is the number of parameters you train in SAE in total for all layers? \n- Again, if the main benefit of the method is interpretability, I do not understand how this work addresses this property and what is its main contribution in improving interpretability. \n- Are $W_e$ and $D$ the matrices defined for each layer? If they are - an appropriate indexing should be applied.", "limitations": "not discussed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Proposed a new training method for SAE with end-to-end loss that allows to use of fewer features from the original model\n- The authors provided extensive analysis of their results\n- The related work is sufficient\n- The proposed method decreases the number of active dictionary elements twice while preserving the same CE loss decrease as a baseline SAE.", "weaknesses": "- The modification is very simple and raises the question of whether additional variants could work too, e.g. local SAE with MSE end-to-end loss (in addition to the layer reconstruction MSE)\n- The numeric results in Figure 1 are very incremental: e.g. reduction of 0.05 in CE loss increase L0=100 is ~1.6% of the evaluation CE loss\n- The interpretability is not well defined, in addition, the claim that fewer features means more interpretability should be proved by empirical evidence, white the authors showed that interpretability was not changed or changed significantly (appendix A7).  If I get it correctly, the main goal of the proposed method is interpretability but a few experiments were done to evaluate it. Also, qualitative results are not provided (anonymous url)\n-  Std values are missing (for different random initialization seeds) in Figure 1\n- The writing should be improved, Sec. 3.4 should be more self-contained\n- It is not clear which method e2e or e2e-ds  is the best performer overall?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for you comments and suggestions\n\n## Comments on Weaknesses \n> The interpretability is not well defined...\n\nThe interpretability provided by an SAE contains two main components: The average number of SAE features required to interpret any particular model output (L0 - lower is better), and the individual interpretability of each of those features. Improving either one of these without hurting the other or the amount of model performance explained constitutes an improvement to interpretability by shortening the description length required to explain the model’s behavior. Shortness of description is an essential component of interpretability.\n\nAs we show in Figure 1, e2e and e2e+ds SAEs are able to significantly reduce L0, requiring fewer active features compared to local SAEs explaining the same amount of model performance (as measured by the CE loss increase).\n\nWhile the interpretability of each feature is in general somewhat subjective and hard to measure, we put significant effort into gaining unbiased and rigorous measures by implementing automated interpretability scoring (Bills 2023).\n\nOverall, we believe the evidence is clear that our method produces SAEs which are significantly more useful for interpretability than SAEs produced by the baseline local training process.\n\n> The numeric results in Figure 1 are very incremental: e.g. reduction of 0.05 in CE loss increase L0=100 is ~1.6% of the evaluation CE loss.\n\nWe believe that this is an inappropriate baseline. At the L0=100 position quoted, our method more than halves the CE loss increase, relative to the local SAE baseline. We would not call a halving of the error incremental. Recall that we are not trying to reduce the CE loss of the original model - merely to explain as much of the original model’s performance as possible.\n\nSecond, although an increase of 0.05 in CE loss may seem relatively small, it corresponds to a much larger effective drop in model size or training FLOP via scaling laws between these quantities and loss. The drop in CE loss from the model learning more complex behaviors (which emerge at larger model sizes or training FLOP) generally reduces as the behaviors get more complex (relative to the drop in loss from the model learning simple bigrams or trigrams), and yet we are still especially interested in interpreting how the model performs these more complex behaviors.\n\nFinally, the quoted rises in CE loss are for one single SAE. in practice, an explanation of the model’s behavior will require SAEs in many layers throughout the model, so seemingly small errors can be multiplied in this way, leading to unfaithful descriptions (this is illustrated in Marks2024 (https://arxiv.org/abs/2403.19647)).\n\n> The modification is very simple and raises the question of whether additional variants could work too, e.g. local SAE with MSE end-to-end loss (in addition to the layer reconstruction MSE)\n\nPlease see our response to reviewer TS92, where we agree that this would be an interesting variant to try, along with a long list of others. \n\n> Std values are missing (for different random initialization seeds) in Figure 1\n\nWe provide some evidence in Figure 18 that the results of Figure 1 are robust to different random initialization seeds. Unfortunately, training enough SAEs to get robust estimates of the standard deviations for each datapoint would be prohibitively expensive.\n\n> It is not clear which method e2e or e2e-ds  is the best performer overall?\n\nWe recommend e2e-ds as the best performer overall. It achieves a similar performance explained to e2e SAEs while maintaining activations that follow similar pathways through later layers compared to the original model (highlighted in section 3.2 and the conclusion). \n\n## Responses to Questions \n> Can you please explain the diagram in Figure 1 why some blocks are of different shapes?\n\nThe different shapes are to distinguish qualitatively different operations, such as the unembed (which projects up from the residual stream to the logits), and the SAE (which we train). These are labelled in the diagram.\n\n> It looks like that the Sec. 3.4 result is trivial, if we don’t encourage the SAE to reconstruct after wach activation, that will be the result.\n\nWe believe you may have confused Section 3.4 with Section 3.3.2, or some other section. If so, Section 3.3.2 was not intended to be counterintuitive. If not, see our response to questions on evaluating interpretability above.\n\n> Are W_e and D the matrices defined for each layer? If they are - an appropriate indexing should be applied.\n\nWe do only ever train one SAE at a time, but we will add some layer indices for clarity, thanks for bringing this to our attention.\n\n> What is the number of parameters you train in SAE in total for all layers?\n\nAs mentioned on line 166, the number of dictionary elements in each SAE is fixed at 60 times the size of the GPT-2 small residual stream (60 x 768 = 46080 dictionary elements), so W_e and D both have sizes 46080 x 768, b_e is a 46080 dimensional vector, and b_d is a 768 dimensional vector, meaning that each SAE has just over 70 million parameters.\n\n## Summary\nWe thank you again for your comments and questions. If we have adequately addressed your concerns, we would kindly ask you to reassess your review score, taking this rebuttal into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MWqFtfM6oS", "reviewer": "Reviewer_Hd7R", "summary": "This paper proposes to train Sparse Autoencoders (SAEs) with an additional loss function term: The KL divergence between the original output distribution and the output distribution obtained when using model with the inserted SAEs. This additional loss pushes the SAEs to focus on functionally important features\nThe authors find that this setup requires fewer total features and fewer simultaneously active features per data point.", "questions": "line 50: but we no longer optimize activation reconstruction. -> but you do use activation reconstruction for $SAE_{e2e+ds}$, don't you?\n\nMaybe I just missed i but where in the network do you plug in the SAEs? the residual stream/decoder layer output?\n\nYou say that taking different computational pathways through subsequent layers of the network might be a problem, but do you have any evidence for this happening? is this just section 3.3.2?\n\nWhile reading, I was wondering if taking different computational pathways through subsequent layers of the network is still a problem if only one SAE is trained at a time. There should be little incentive for computing additional information since the original network post SAE insertion would not have capacity to handle these anyway, so it would only matter for the output directly.\n\n\nsmall things:\n- line 68: not sure if it's necessary to mention goodhart's law since the target is not cross-entropy loss difference but interpretability so I would be more worried about this if you start optimizing interpretability measures \n- line 5: datatset -> dataset\n- line 44: a feature is functional important -> a feature is functionally important\n- line 113: \")\" missing in equation", "limitations": "Authors briefly discuss the lack of a ground truth to evaluate SAEs.\n\nI think discussion of evaluation metrics could be a bit more divided (maybe into metrics to assess interpretability and metrics to assess how close to the original network a network with inserted SAEs is).", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- the authors point out a weakness of previous SEAs (feature splitting) and propose an intuitive solution: learning functionally important features by minimizing the KL divergence between original output and output with inserted SAEs\n\n- writing is mostly clear and easy to follow\n\n- the authors discuss limitations of their approach like robustness to random seeds, reconstruction loss, training time", "weaknesses": "- I think the major weakness is that the contribution is not very groundbreaking.\n- There are no new findings wrt interpretability, and the e2e SAEs require more compute to be trained\n- while the authors discuss several findings (robustness to random seeds, reconstruction loss, training time) the reader is left waiting a bit for a conclusion or take home message\n- the comparison to local SAEs is sometimes a bit unclear:\n\t- the authors state that \"locally trained SAEs are capturing information about dataset structure that is not maximally useful for explaining the algorithm implemented by the network.\" but do not show that their e2e SAEs are significantly more interpretable or helpful at explaining the algorithm implemented by the network, instead the e2e SAEs are merely \"at least as interpretable as SAElocal\"\n\t- the features found by local SAEs are not the functionally important ones , but SAElocal and SAEe2e+ds features are somewhat similar so local SAEs can be used as initializations (line 233)?\n\n\n- notation could be improved \n\t- N_dict_elements and d_hidden seem to belong more into pseudo code so it looks a bit odd to mix it with more traditional mathy notation\n\t- $\\lambda$ is hidden in the $\\phi$ parameter but then you show values for lambda in Table 1 and looking back at i.e. Eq. (1) it's not immediately clear what $\\lambda$ is and why $\\phi$ is not mentioned. I would suggest either putting $\\lambda$ directly in the equations or citing values for $\\phi$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and your typo spotting and notation improvement suggestions!\n\n## Addressing main misunderstanding\nAfter reading your review, it appears we failed to convey the core benefit of our method. You state\n> the authors state that \"locally trained SAEs are capturing information about dataset structure that is not maximally useful for explaining the algorithm implemented by the network.\" but do not show that their e2e SAEs are significantly more interpretable or helpful at explaining the algorithm implemented by the network, instead the e2e SAEs are merely \"at least as interpretable as SAElocal\"\n\nIndeed, we did not find that **individual** features found by e2e SAEs were more interpretable than local SAEs. Though we didn't expect them to be, nor was it the purpose of this work. The important contribution of this work is that in order to explain the activations at a particular point in the network, our method requires fewer than half of the features that other methods require. We illustrate this with the pareto curves in Figure 1 (left). This dramatically reduces the description length of any feature or circuit that uses SAEs. Shortness of description is an essential aspect of interpretability.\n\nIn addition, we show that we need far fewer features in total to explain the network activations over the whole dataset (Figure 1 (right)).\n\n## Response to Limitation\n> Authors briefly discuss the lack of a ground truth to evaluate SAEs. I think discussion of evaluation metrics could be a bit more divided (maybe into metrics to assess interpretability and metrics to assess how close to the original network a network with inserted SAEs is).\n\nWe do have a section-level separation between both types of metrics (Sections 3.1-3.3 for the latter type and Section 3.4 for the former), and we link to the sections alongside a short explainer of them at the bottom of the introduction (lines 75-86). We’re open to alternative division if preferred.\n\nIn addition, we have run more experiments in response to other reviews that evaluate the SAE setups on a set of downstream tasks.\n\n## Other responses to Weaknesses and Questions\n\n> line 50: but we no longer optimize activation reconstruction. -> but you do use activation reconstruction for SAEe2e+ds, don't you?\n\nYes. We train for downstream activation reconstruction SAEe2e+ds, but at all subsequent layers, rather than at the current layer. In this paragraph we are introducing SAEe2e, which does not involve any training for activation reconstruction.\n\n> Maybe I just missed i but where in the network do you plug in the SAEs? the residual stream/decoder layer output?\n\nAll of our SAEs are inserted in the residual stream (lines 160). We'll add this to the main figure and table for further clarity.\n\n> You say that taking different computational pathways through subsequent layers of the network might be a problem, but do you have any evidence for this happening? is this just section 3.3.2?\n\nPrimarily the evidence for this is in Section 3.2, especially in Figure 2, as well as in Section 3.3.2. Figure 2 directly shows the e2e SAE’s output activations taking a very different pathway through the network, as the activation reconstruction MSE is much higher and significantly increases downstream through the layers of the network, rather than converging. Section 3.3.2 provides additional evidence that the pathway through the network is under-constrained by the KL divergence loss of e2e SAEs alone. These problems are resolved by e2e+ds SAE training.\n\n> While reading, I was wondering if taking different computational pathways through subsequent layers of the network is still a problem if only one SAE is trained at a time. There should be little incentive for computing additional information since the original network post SAE insertion would not have capacity to handle these anyway, so it would only matter for the output directly.\n\nEven with one SAE trained at a time, it turns out that there is capacity for the original network to handle the SAE adding \"new\" features to the residual stream. Evidence for this can be seen in Figure 3b (middle), which shows that with two different seeds, a pure e2e SAE can learn a very different set of features (and still perform as well in the L0-CE loss pareto). While we're unsure how much of a problem this is, our e2e+ds formulation appears to resolve this issue at very little cost.\n\n> small things: line 68: not sure if it's necessary to mention goodhart's law since the target is not cross-entropy loss difference but interpretability so I would be more worried about this if you start optimizing interpretability measures\n\nThis is reasonable, and is our argument too. Though it's worth mentioning that reviewer 2 and reviewer 5 both expressed concerns about Goodharting.\n\n>[notation improvement suggestions]    \nline 5: datatset -> dataset.  \nline 44: a feature is functional important -> a feature is functionally important.  \nline 113: \")\" missing in equation\n\nThanks for spotting these errors and notation ideas! We'll incorporate these.\n\n## Summary\nThanks again for your comments and questions. We hope to have adequately clarified any uncertainties or misunderstandings w.r.t the core contributions of this work and other listed limitations or weaknesses. If so, we would ask you to reassess your review score, taking this response into account.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uqdLSm8xy1", "reviewer": "Reviewer_ErCK", "summary": "This work describes two new type of sparse autoencoders (SAEs) that are trained with two new loss functions, L_{e2e} and L_{e2e + ds}. L_{e2e} loss penalizes the KL divergence of the original model vs. when the SAE is inserted, as opposed to existing SAE training techniques which instead optimize MSE with the original activations. L_{e2e + downstream} additionally seeks to minimize the MSE of downstream layers after the SAE is inserted vs. the original model. Both types of new SAE have a better L0 vs. CE tradeoff than naive SAEs, and e2e + downstream additionally has similar downstream reconstruction loss. Interpretability is no worse for the new SAEs. Finally, e2e+ds has similar geometric properties to naive SAEs, including discovering similar features.", "questions": "- Building off the above weaknesses, could the authors add a measure of downstream SAE functional importance that is not CE loss? \n- Why doesn't the original SAE learn the \"left block\" of features that SAE e2e + ds does? It would be great to see if e.g. these have a small or negative impact on the variance explained.\n- Did the authors try training an SAE with KL divergence + the local reconstruction error (i.e. MSE of layer X reconstruction for an SAE trained on layer X)? It is a little concerning to me that the layer X variance explained is so strongly negative because it means the SAE's reconstruction is very far off distribution.\n- Can you quantify how much farther the <e2e ds to local> difference is vs. the inter seed distance? It is hard to know how to interpret Appendix G without this context, as inter-seed SAEs have different features learned too.\n- The ~0.15 CE loss e2e+downstream SAE looks closer to the other chosen SAEs than the 0.125 CE loss one, does it change things to run with the ~0.15 one?\n\nSmall points:\n- Line 41 states \"a significant amount of [causal effects of some features is] mediated by the reconstruction residual errors\", but I believe the cited work shows that this is true for circuits as a whole, not features individually.\n- Lines 133-136 are confusing. Why is it okay to \"clear out\" features from layer X, but not layers X + 1...? In general, a more clear discussion of the tradeoff between learning functionally important features and maintaining model behavior would help I think.\n\nI would be happy to raise my score if some of these questions/concerns were addressed.", "limitations": "I believe the authors should more explicitly discuss the Goodhart's law problem I've discussed above and ideally the increased feature suppression as well. Otherwise, I think the limitations and impacts are well addressed.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The new SAE techniques that the authors introduce are a Pareto improvement over existing techniques on L0 vs. CE loss. Furthermore, the techniques are exciting because they is in a different direction from recent improvements in training SAEs: it is possible that the new techniques can be combined with recent techniques like gated SAEs and the top-k activation function.\n- The investigation of the geometry of the different SAE features is extremely interesting. Most notably, e2e + ds finds similar features to local and is reproducible over different seeds, and e2e has potentially less feature splitting.\n- The new techniques result in features of similar interpretability as local, as determined by standard automated interpretably tests.", "weaknesses": "- The main evaluation metrics that show improvement (CE loss for both new SAEs and later layer reconstruction error for e2e + downstream) are the same ones that are being optimized. This is noted by the authors when they mention Goodhart's law, but it would greatly improve the paper if there were additional metrics for downstream SAE quality that were tested (e.g. some of the methods described in section 4.4). \n- Feature shrinkage (shown in Table 5) seems much worse with the new SAE training methods. Moreover, even with e2e + ds SAEs, later layer reconstructions in Figure 12 are much worse than local. This seems potentially a problem for e.g. finding circuits or understanding interactions between multiple SAEs via feature similarity. This is another reason I think additional downstream metrics would be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your evidently detailed read of our paper and your thoughtful comments.\n\n## Comments on Weaknesses\nRegarding evaluation metrics, we've taken your criticism on board and run a set of evaluations on a set of downstream tasks. These are documented in the Author Rebuttal at the top (results in the attached pdf).\n\nAs for your observation about feature shrinkage appearing worse for e2e SAEs. We agree. It turns out you can shrink activations in the residual stream without changing the output much and e2e SAEs take advantage of this. Note that this is partially because of layernorm (as shown in Figure 12). Fortunately, one can address the feature shrinkage issue with some of the SAE variants that were released following our submission (and that you mention in the summary of your review), e.g. topk SAEs, Gated SAEs, JumpReLU. We agree with your summary that these improvements should be orthogonal to the improvements we find with e2e SAEs.\n\n## Comments on Questions\n> Why doesn't the original SAE learn the \"left block\" of features that SAE e2e + ds does? It would be great to see if e.g. these have a small or negative impact on the variance explained.\n\nIf you’re referring to what appears to be the left mode of the somewhat bimodal distribution in Figure 8, we could not find any meaningful similarities between these features unfortunately. Once the anonymity is lifted, we will be able to publish an interactive dashboard with these features.\n\n> Did the authors try training an SAE with KL divergence + the local reconstruction error (i.e. MSE of layer X reconstruction for an SAE trained on layer X)? It is a little concerning to me that the layer X variance explained is so strongly negative because it means the SAE's reconstruction is very far off distribution.\n\nPlease see the comment about variations of e2e+ds in the Author Rebuttal at the top.\n\n> Can you quantify how much farther the <e2e ds to local> difference is vs. the inter seed distance? It is hard to know how to interpret Appendix G without this context, as inter-seed SAEs have different features learned too.\n\nGood point. As for the means of those distributions, the <e2e ds to local> cross-type similarity (Figure 3c bottom) has a mean of 0.73, and the <e2e ds> cross-seed similarity (Figure 3b bottom) has a mean of 0.78. Visually, we do see the mode to the left of the cross-type plots that doesn’t exist in cross-seed plots, indicating that there is a set of features which are learned by e2e+ds but not local (as mentioned in response to your earlier question, we weren’t able to interpret any common properties of this set of features). We’ve plotted a UMAP comparison for different seeds on layer 6. Unlike our cross-type UMAP plots in the paper (Figure 19), we don’t see clear regions where one seed is represented but the other isn’t (though, of course, reading deeply into UMAP plots is a source of potential confusion). We’ve attached the layer 6 cross-seed UMAP to the pdf in Figure 3 of the Author Rebuttal. \n\n> The ~0.15 CE loss e2e+downstream SAE looks closer to the other chosen SAEs than the 0.125 CE loss one, does it change things to run with the ~0.15 one?\n\nKeen eye. The reason we didn’t select this run for our “similar CE loss” set was simply because it was run after our other analyses (including the time-consuming interpretations in Appendix G and dashboard creation) merely to fill out the pareto frontier for a nicer plot. After your comment, we reran the analysis (not the interpretations) with this run and did not find any significant differences. We posted a sample comparing the cross-type similarity with the different runs in Figure 2 of the uploaded pdf in the Author Rebuttal at the top.\n\n> Line 41 states \"a significant amount of [causal effects of some features is] mediated by the reconstruction residual errors\", but I believe the cited work shows that this is true for circuits as a whole, not features individually.\n\nWe agree that it could technically be possible that the circuit does have this property but that individual features in the circuit do not. Though we think it’s reasonable to extrapolate that the same is true for individual features. Nonetheless, we’ve updated the text to be more accurate, thanks for pointing this out.\n\n> Lines 133-136 are confusing. Why is it okay to \"clear out\" features from layer X, but not layers X + 1...? In general, a more clear discussion of the tradeoff between learning functionally important features and maintaining model behavior would help I think.\n\nOur initial concern was that adding a reconstruction loss term at the current layer would have a much larger impact on reconstructing non-functional features than if the loss term was added at downstream layers. Though we have come to agree that this difference is merely quantitative rather than qualitative. As mentioned in the \"Other variations of e2e+ds\" section in the Author Rebuttal, we unfortunately have not run the extensive sweeps on the e2e+ds variations needed to give an insight to this question empirically.\n\n## Summary\nThanks again for your valuable comments. Hopefully we've addressed your concerns/questions, and are keen to hear if there are cases where we have not.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We'd like to thank the reviewers and ACs for their time. We're delighted to hear that our work "makes some progress on ... an important open problem" (R2), "provides extensive analysis of their results" (R3), "provide an intuitive solution" (R4), introduces "exciting techniques" and has some "extremely interesting" analysis (R5). We're also glad to see that the reviewers who gave the highest ratings also had the highest confidence in their ratings (R2, R5).

Responses to individual comments/questions are provided alongside each review. Most notably, we hope that we've cleared up what we believe to be a core misunderstandings of the paper's contribution for R1 and R4. Below, we respond to concerns that were shared across multiple reviewers.

## Other variations of e2e+ds
Reviewer TS9212 (R2) and reviewer SeEL02 (R5)  commented about trying other variations of the e2e+ds SAE. Reviewer TS9212 asks _“did you try the "next layer only" SAE variant described in the weaknesses?”_. Reviewer SeEL02 states _“The modification is very simple and raises the question of whether additional variants could work too, e.g. local SAE with MSE end-to-end loss”_.

We agree that these would both be interesting variants to try, although there are also many other variants omitted in our experiments. A more complete set of variants might be:

- *Our local
- Local + next layer reconstruction
- Local + arbitrary downstream layer reconstruction
- Local + multiple arbitrary downstream layer reconstruction
- Next layer reconstruction only
- Arbitrary downstream layer(s) reconstruction only
- Multiple arbitrary downstream layer reconstruction only
- Local + KL Div
- Local + next layer reconstruction + KL Div
- Local + arbitrary downstream layer reconstruction + KL Div
- Local + multiple arbitrary downstream layer reconstruction + KL Div
- Next layer reconstruction + KL Div
- Arbitrary downstream layer(s) reconstruction + KL Div
- *Multiple arbitrary downstream layer reconstruction + KL Div (i.e. our e2e+ds)
- *KL Divergence only (i.e. our e2e)

We're also interested in these variants, not least because it would be interesting to know if we can get much of the benefit of e2e+ds merely by reconstructing a more ‘local’ set of layers and avoiding the backpropagation through most of the model. We expect that these variants will not greatly change the L0-CE Loss Pareto frontier, though it is likely that they would result in different properties w.r.t the computational pathway through the network. Since we could not focus on every variant, we chose to focus on e2e and e2e+ds since we believe that these were sufficient illustrations of our core thesis.

## Further evaluations

Reviewer TS9212 (R2) and reviewer ErCK12 (R5) had concerns that the paper did not provide evaluation metrics that were not biased by the KL divergence loss metric we introduced. Reviewer TS9212 states _“I think the main limitation that I would have loved to see addressed more in the paper is that the evaluations are somewhat indirect w.r.t. the main claim of the paper. Even if in some average sense the e2e SAEs' reconstructions lead to better KL divergence, we still don't know how this plays out on the level of individual features.”_ Reviewer ErCK12 states _“...it would greatly improve the paper if there were additional metrics for downstream SAE quality that were tested (e.g. some of the methods described in section 4.4).”_

We think these are good criticisms of the paper. For this reason, we’ve run additional evaluations on specific tasks. The experiments are described below, and some results can be seen in the attached pdf.

### Methodology
We adapt the datasets from Marks et al. (2024), originally presented in Finlayson et al. (2021). This includes 4 variations of subject-verb agreement template data:
* Simple (The parent/s is/are)
* Within RC (The athlete that the manager/managers likes/like)
* Across RC (The athlete/athletes that the managers like do/does)
* Across PP (The secretary/secretaries near the cars has/have)

For our metric $m$ we take the difference in logits between the correct completion, and the same verb but swapped between singular and plural forms For example, given $x_\text{clean}=\texttt{The teachers}$, $m(x_\text{clean})$ will be $\text{logit}(\texttt{ are}) - \text{logit}(\texttt{ is})$.

Then the $\textit{faithfulness}$ of a network under an intervention is defined as $\frac{E_x[m(x_\text{clean} | \text{intervention})]}{E_x[m(x_\text{clean})]}$

Note that the fully zero or mean ablated model will have an average logit difference of zero, due to the construction of the dataset.

### Faithfulness of models with SAEs
We first test the faithfulness of the models with the SAEs inserted (Table 1 in pdf). All SAEs preserve most of the logit-difference, however there is significant variation across SAE types, layers, and tasks. The local SAEs in layer 10 have the worst faithfulness, although our e2e + ds SAE in layer 6 also has poor faithfulness across participial phrases.

### Number of nodes needed to explain behavior
We order SAE features in each Similar $L_0$ SAE by indirect effect (mean ablating one SAE feature at a time and measuring how the metric changes). We then measure the metric while preserving only the top k most important features (for $k \in [1, \cdots, 1200]$). The results are in Figure 1 of the pdf.

### Result Summary
While we see some (SAE type, SAE layer) combinations perform better than others, we see no clear patterns between SAE types over all the tasks. These results indicates that e2e SAEs do not provide an obvious benefit on these specific tasks. Even though we've shown that e2e SAEs are beneficial on the language modeling task over the full openwebtext distribution, further work would be needed to find specific tasks and subdistributions in which they provide the most benefit.

## Rebuttal Summary
We hope that our additional experiments and comments to the reviewers questions satisfy their concerns.

Meta-review: **Summary of the paper:**
The paper proposes a new way to train sparse auto-encoders (SAE) to be used for interpreting pre-trained deep networks. Instead of directly minimizing the discrepancy of the feature out of SAE with that of an intermediate layer of the original network, it minimizes the discrepancy at the output layer, i.e., probability distribution over the next token in the case of LLMs considered in this work. It obtains the output for SAE by passing the SAE’s encoded feature through the rest of the layers of the original network. It also uses a sparsity inducing regularization. It further adds a reconstruction loss to some later layers.

**Summary of the reviews:**
The reviewers found the proposed method well-justified, the reasoning of the paper convincing, the limitations well-covered, the results mostly suggesting improved performance and the geometric study of the learnt features informative .

On the other hand, they found evaluating metrics not easy to interpret, not entirely convincing, or sometimes incrementally improved, the mitigation of inconsistent pathways still possibly problematic when moving the reconstruction to a later layer, a lack of ablation study when only the later reconstruction is used (without KLD on output). 

**Summary of the rebuttal and discussions:**
The authors provided new experiments on a different evaluation metric than CE for faithfulness. 
This along with the textual clarifications on the evaluation metrics rectified some of the concerns of the reviewers. However, some reviewers remained somewhat concerned about the metrics. Despite that, all reviewers are leaning towards acceptance.

**Consolidation report:**
As reviewers have pointed out a crucial strength of the paper is the outright transparency on the limitation of their approach and importantly the limitations of the experiments and evaluations in supporting their basic claims and, in fact, the paper often discusses alternative hypotheses. This mitigates most of the issues raised above, which are especially valid regarding the metrics. This strength in tandem with the fact that the paper has taken clear steps forward for SAE interpretation of LLMs, make a good paper.

**Recommendation:**
The AC agrees with the general tendency of all reviewers and recommends acceptance. The discussions (including the concerns raised by reviewers) and additional points and results provided by the authors during the rebuttal would be quite enlightening for future readers and thus the authors are encouraged to incorporate them in the camera ready version of the paper. It would be good to have a section discussing the metrics used and their limitations as was extensively discussed with the reviewers.
Decision: Accept (poster)