Title: Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders
Venue: NeurIPS 2024
Accepted: yes
Keywords: Mechanistic Interpretability

Sparse Autoencoders

Science of Deep Learning
Abstract: Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of those activations. We introduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto improvement over training with prevailing methods. In SAEs, the L1 penalty used to encourage sparsity introduces many undesirable biases, such as shrinkage -- systematic underestimation of feature activations. The key insight of Gated SAEs is to separate the functionality of (a) determining which directions to use and (b) estimating the magnitudes of those directions: this enables us to apply the L1 penalty only to the former, limiting the scope of undesirable side effects. Through training SAEs on LMs of up to 7B parameters we find that, in typical hyper-parameter ranges, Gated SAEs solve shrinkage, are similarly interpretable, and require half as many firing features to achieve comparable reconstruction fidelity.

Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders

Abstract

Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of those activations. We introduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto improvement over training with prevailing methods. In SAEs, the L1 penalty used to encourage sparsity introduces many undesirable biases, such as shrinkage - systematic underestimation of feature activations. The key insight of Gated SAEs is to separate the functionality of (a) determining which directions to use and (b) estimating the magnitudes of those directions: this enables us to apply the L1 penalty only to the former, limiting the scope of undesirable side effects. Through training SAEs on LMs of up to 7B parameters we find that, in typical hyper-parameter ranges, Gated SAEs solve shrinkage, are similarly interpretable, and require half as many firing features to achieve comparable reconstruction fidelity.

1 Introduction

Mechanistic interpretability aims to explain how neural networks produce outputs in terms of the learned algorithms executed during a forward pass [33, 34]. Much work makes use of the fact that many concept representations appear to be linear [14, 19, 34, 39], i.e. that they correspond to interpretable directions in activation space. However, finding the set of all interpretable directions is a highly non-trivial problem. Classic approaches, like interpreting neurons (i.e. directions in the standard basis) are insufficient, as many are polysemantic and tend to activate for a range of different seemingly unrelated concepts [7, 15, 16]. Within the field, there has recently been much interest [8, 11, 21, 22, 4] in using sparse autoencoders (SAEs; [32]) as an unsupervised method for finding causally relevant, and ideally interpretable, directions in a language model's activations.

Although SAEs show promise in this regard [26, 31], the L1 penalty used in the prevailing training method to encourage sparsity also introduces biases that harm the accuracy of SAE reconstructions, as the loss can be decreased by trading-off some reconstruction accuracy for lower L1. In this paper, we introduce a modification to the baseline SAE architecture - a Gated SAE - along with an accompanying loss function, which partially overcomes these limitations. Our key insight is to use separate affine transformations for (a) determining which dictionary elements to use in a reconstruction and (b) estimating the coefficients of active elements, and to apply the sparsity penalty only to the former task. We share a subset of weights between these transformations to avoid significantly

100

2

5

10

2

5

2

0.8

0.7

0.9

1

2

5

10

2

5

100

2

2

5

10

2

5

100

2

SAE Type

Gated

Baseline

L0

(Lower is sparser)

Loss Recovered

(Fidelity)

Residual stream post-MLP

MLP Output

Attention output pre-linear

increasing the parameter count and inference-time compute requirements of a Gated SAE compared to a baseline SAE of equivalent width. 2

We evaluate Gated SAEs on multiple models: a one layer GELU activation language model [28], Pythia-2.8B [3] and Gemma-7B [18], and on multiple sites within models: MLP layer outputs, attention layer outputs, and residual stream activations. Across these models and sites, we find Gated SAEs to be a Pareto improvement over baseline SAEs holding training compute fixed (Fig. 1): they yield sparser decompositions at any desired level of reconstruction fidelity. We also conduct further follow up ablations and investigations on a subset of these models and sites to better understand the differences between Gated SAEs and baseline SAEs.

Overall, the key contributions of this work are that we:

Introduce the Gated SAE, a modification to the standard SAE architecture that decouples detection of which features are present from estimating their magnitudes (Section 3.2);

Show that Gated SAEs Pareto improve the sparsity and reconstruction fidelity trade-off compared to baseline SAEs (Section 4.1);

Confirm that Gated SAEs overcome shrinkage while outperforming other methods that also address this problem (Section 5.2);

Provide evidence from a small double-blind study that Gated SAE features are comparably interpretable to baseline SAE features (Section 4.2).

2 Preliminaries

In this section we summarise the concepts and notation necessary to understand existing SAE architectures and training methods following Bricken et al. [8], which we call the baseline SAE . We define Gated SAEs in Section 3.2.

As motivated in Section 1, we wish to decompose a model's activation x ∈ R n into a sparse, linear combination of feature directions:

where d i are dictionary of M ≫ n latent unit-norm feature directions , and the sparse coefficients f i ( x ) ≥ 0 are the corresponding feature activations for x . 3 The right-hand side of Eq. (1) naturally has the structure of an autoencoder: an input activation x is encoded into a (sparse) feature activations vector f ( x ) ∈ R M , which in turn is linearly decoded to reconstruct x .

Baseline architecture Using this correspondence, Bricken et al. [8] and subsequent works attempt to learn a suitable sparse decomposition by parameterizing a single-layer autoencoder ( f , ˆ x ) defined by:

and training it using gradient descent to reconstruct samples x ∼ D from a large dataset D of activations collected from a single site and layer of a trained language model, constraining the hidden representation f to be sparse. Once the sparse autoencoder has been trained, we obtain a decomposition of the form of Eq. (1) by identifying the (suitably normalised) columns of the decoder weight matrix W dec ∈ R M × n with the dictionary of feature directions d i , the decoder bias b dec ∈ R n with the centering term x 0 , and the (suitably normalised) entries of the latent representation f ( x ) ∈ R M with the feature activations f i ( x ) .

Baseline training methodology To train sparse autoencoders, Bricken et al. [8] use a loss function with two terms that respectively encourage faithful reconstruction and sparsity: 4

Since it is possible to arbitrarily reduce the L1 sparsity loss term without affecting reconstructions or sparsity by simply scaling down encoder outputs and scaling up the norm of the decoder weights, it is important to constrain the norms of the columns of W dec during training. Following Bricken et al. [8], we constrain norms to one. See Appendix G for further details on SAE training.

Evaluating SAEs Two metrics are primarily used to get a sense of SAE quality [8]: L0 , a measure of SAE sparsity, and loss recovered , a measure of SAE reconstruction fidelity. L0 measures the average number of features used by a SAE to reconstruct input activations. Loss recovered is a normalised measure of the increase induced in a LM's cross entropy loss when we replace its original activations with the corresponding SAE reconstructions during the model's forward pass. Both these metrics are formally defined in Appendix B, where we also discuss shortcomings of and alternatives to the loss recovered metric as it is defined in Bricken et al. [8]. Since it is possible for SAEs to score well on these metrics and still fail to be useful for interpretability-related tasks [47], we perform manual analysis of SAE interpretability in Section 4.2.

3 Gated SAEs

3.1 Motivation

The intention behind how SAEs are trained is to maximise reconstruction fidelity at a given level of sparsity, as measured by L0, although in practice we optimize a mixture of reconstruction fidelity and L1 regularization. This difference is a source of unwanted bias in the training of a sparse autoencoder: for any fixed level of sparsity, a trained SAE can achieve lower loss (as defined in Eq. (4)) by trading off a little reconstruction fidelity to perform better on the L1 sparsity penalty.

The clearest consequence of this bias is shrinkage [51]. Holding the decoder ˆ x ( · ) fixed, the L1 penalty pushes feature activations f ( x ) towards zero, while the reconstruction loss pushes f ( x ) high enough to produce an accurate reconstruction. Thus, the optimal value falls somewhere in between, and as a result the SAE systematically underestimates the magnitude of feature activations, without necessarily providing any compensatory benefit for sparsity. 5

How can we reduce the bias introduced by the L1 penalty? The output of the encoder f ( x ) of a baseline SAE (Section 2) has two roles:

It detects which features are active (according to whether the outputs are zero or strictly positive). For this role, the L1 penalty is necessary to ensure the decomposition is sparse.

Gating P ath

scale & shift

shift

binarize

Encoder

Decoder

ReLU

It estimates the magnitudes of active features. For this role, the L1 penalty is a source of unwanted bias.

If we could separate out these two functions of the SAE encoder, we could design a training loss that narrows down the scope of SAE parameters that are affected (and therefore to some extent biased) by the L1 sparsity penalty to precisely those parameters that are involved in feature detection, minimising its impact on parameters used in feature magnitude estimation.

3.2 Gated SAEs

Architecture How should we modify the baseline SAE encoder to achieve this separation of concerns? Our solution is to replace the single-layer ReLU encoder of a baseline SAE with a gated ReLU encoder. Taking inspiration from Gated Linear Units [43, 12], we define the gated encoder as

where ✶ [ · > 0 ] is the (pointwise) Heaviside step function and ⊙ denotes elementwise multiplication. Here, f gate determines which features are deemed to be active, while f mag estimates feature activation magnitudes (which only matter for features that have been deemed to be active); π gate ( x ) are the f gate sub-layer's pre-activations, which are used in the gated SAE loss, defined below.

Training Anaive guess at a loss function for training Gated SAEs would be to replace the sparsity penalty in Eq. (4) with the L1 norm of f gate ( x ) . Unfortunately, due to the Heaviside step activation function in f gate, no gradients would propagate to W gate and b gate. To mitigate this, we instead apply the L1 norm to the positive parts of the preactivation, ReLU ( π gate ( x )) . To ensure f gate aids reconstruction by detecting active features, we add an auxiliary task requiring that these same rectified preactivations can be used by the decoder to produce a good reconstruction:

where ˆ x frozen is a frozen copy of the decoder, ˆ x frozen ( f ) := W copy dec f + b copy dec , to ensure that gradients from L aux do not propagate back to W dec or b dec . This can be implemented by stop gradient operations rather than creating copies. See Appendix J for pseudo-code for the forward pass and loss function.

To calculate this loss (or its gradient), we have to run the decoder twice: once to perform the main reconstruction for L reconstruct and once to perform the auxiliary reconstruction for L aux . This leads to a 50% increase in the compute required to perform a training update step. However, the increase in overall training time is typically much less, as in our experience much of the training wall clock time goes to generating language model activations (if these are being generated on the fly) or disk I/O (if training on saved activations).

Parameter reduction through weight-tying Naively, we appear to have doubled the number of parameters in the encoder, increasing the total number of parameters by 50% with respect to baseline SAEs. We mitigate this through weight sharing: we parameterize these layers so that the two layers share the same projection directions, but allow the norms of these directions as well as the layer biases to differ. Concretely, we define W mag in terms of W gate and an additional vector-valued rescaling parameter r mag ∈ R M as follows:

See Fig. 2 for an illustration of the tied-weight Gated SAEs architecture. With this weight tying scheme, the Gated SAE has only 2 × M more parameters than a baseline SAE. In Section 5.1, we show that this weight tying scheme does not harm performance.

With tied weights, the gated encoder can be reinterpreted as a single-layer linear encoder with a nonstandard and discontinuous 'JumpReLU' activation function [17], σ θ ( z ) , illustrated in Fig. 12. To be precise, using the weight tying scheme of Eq. (7), ˜ f ( x ) can be re-expressed as ˜ f ( x ) = σ θ ( W mag · x + b mag ) , with the JumpReLU gap given by θ = b mag -e r mag ⊙ b gate; see Appendix H for an explanation. We think this is a useful intuition for reasoning about how Gated SAEs reconstruct activations in practice.

4 Evaluating Gated SAEs

In this section we benchmark Gated SAEs against baseline SAEs across a large variety of models and at different sites. We show that they produce more faithful reconstructions at equal sparsity and that they resolve shrinkage. Through a double-blind manual interpretability study, we find that Gated SAEs produce features that are similarly interpretable to baseline SAE features.

4.1 Benchmarking Gated SAEs

Methodology We trained a suite of Gated and baseline SAEs, a family of each type to reconstruct each of the following activations:

The MLP neuron activations in GELU-1L, which is the closest direct comparison to Bricken et al. [8];

The MLP outputs, attention layer outputs (taken preW O [21]) and residual stream activations in 5 different layers throughout Pythia-2.8B and four different layers in the Gemma7B base model.

For each model and reconstruction site, we trained multiple SAEs using different values of λ (and therefore L0), allowing us to compare the Pareto frontiers of L0 and loss recovered between Gated and baseline SAEs. We also use the relative reconstruction bias metric, γ , defined in Appendix C to measure shrinkage in our trained SAEs. This metric measures the relative bias in the norm of an SAE's reconstructions; unbiased SAEs obtain γ = 1 , whereas SAEs affected by shrinkage (which causes reconstruction norms to be systematically too small) have γ < 1 .

Since Gated SAEs require at most 1.5 × more compute to train than regular SAEs (Section 3.2) of the same width, we compare Gated SAEs to baseline SAEs that have a 50% larger dictionary (hidden dimension M ) to ensure fair comparison in our evaluations. 6

Results Weplot sparsity against reconstruction fidelity for SAEs with different values of λ . Higher λ corresponds to increased sparsity and worse reconstruction, so as in Bricken et al. [8] we observe a Pareto frontier of possible trade-offs. We plot Pareto curves for GELU-1L in Fig. 3a and Pythia2.8B and Gemma-7B in Appendix E. At all sites tested, Gated SAEs are a Pareto improvement over regular SAEs: they provide better reconstruction fidelity at any fixed level of sparsity. 7 For some sites in Pythia-2.8B and Gemma-7B, loss recovered does not monotonically increase with L0; we attribute this to difficulties training SAEs (Appendix G.1.3). Finally, full tables of results for Pythia and Gemma can be found in Appendix K.

0

50

100

0.88

0.9

0.92

0.94

0.96

0.98

1

SAE Type

Baseline (1.5× width)

Gated

L0

(Lower is sparser)

Loss Recovered

(Fidelity)

(a)

0

50

100

0.8

0.85

0.9

0.95

1

1.05

1.1

SAE Type

Baseline (1.5× width)

Gated

L0

(Lower is sparser)

Relative reconstruction bias

𝛾

(

𝛾

<1 indicates shrinkage)

(b)

As shown in Fig. 3b, Gated SAEs' reconstructions are unbiased, with γ ≈ 1 , whereas baseline SAEs exhibit shrinkage ( γ < 1 ), with the impact of shrinkage getting worse as the L1 coefficient λ increases (and L0 consequently decreases). Fig. 10 shows that this result generalizes to Pythia-2.8B.

4.2 Interpretability

Although Gated SAEs provide more faithful reconstructions than baselines at equal sparsity, it does not necessarily follow that these reconstructions are better suited to downstream interpretabilityrelated tasks. Currently, there is no consensus on how to systematically assess the degree to which a SAE's features are useful for downstream tasks, but a plausible proxy is to assess the extent to which these features are human interpretable [8]. Therefore, to gain a more qualitative understanding of the differences between their learned features, we conduct a blinded human study in which we rate and compare the interpretability of randomly sampled Gated and baseline SAE features.

Methodology We study a variety of SAEs from different layers and sites. For Pythia-2.8B we had 5 raters, who each rated one feature from baseline and Gated SAEs trained on each (site, layer) pair from Fig. 8, for a total of 150 features. For Gemma-7B we had 7 raters; one rated 2 features each, and the rest 1 feature each, from baseline or Gated SAEs trained on each (site, layer) pair from Fig. 9, for a total of 192 features.

For each model, raters are shown the features in random order, without revealing which SAE, site, or layer they came from. 8 To assess a feature, the rater decides whether there is an explanation of the feature's behavior, in particular for its highest activating examples. The rater then enters that explanation (if applicable) and selects whether the feature is interpretable ('Yes'), uninterpretable ('No') or maybe interpretable ('Maybe'). All raters are either authors of this paper or colleagues, who have prior experience interpreting SAE features. As an interface we use an open source SAE visualizer library [27]; representative screenshots of the dashboards produced by this library are shown in Fig. 14.

Results & analysis Fig. 4 shows interpretability rating distributions by SAE type and LM, marginalising over layers, sites and raters. 9 To compare the interpretability of baseline and Gated SAEs, we first pair our datapoints according to all covariates (model, layer, site, rater); this lets us

No

Maybe

Yes

0

0.2

0.4

0.6

0.8

1

No

Maybe

Yes

SAE Type

Baseline

Gated

Feature is interpretable

Feature is interpretable

Frequency

Model=Pythia-2.8B

Model=Gemma-7B

control for all of them without making any parametric assumptions, and thus reduces variance in the comparison. We then measure the mean difference between baseline and Gated labels, where we count 'No' as 0, 'Maybe' as 1, and 'Yes' as 2, and compute a 90% BCa bootstrap confidence interval. Thus we find that the mean difference in label scores is 0 . 13 (90% CI [0 , 0 . 26] ) in favour of Gated SAEs, breaking down to mean difference CIs of [ -0 . 07 , 0 . 33] and [ -0 . 04 , 0 . 29] on just the Pythia-2.8B data and Gemma-7B data respectively. Since our central estimate for the mean difference in scores is positive, we also test the hypothesis that Gated SAEs may be more interpretable than baseline SAEs. However, a one-sided Wilcoxon-Pratt signed-rank test on the paired scores does not reject the null hypothesis that they are equally interpretable ( p = 0 . 06 ). The contingency tables used for these results are shown in Fig. 13. The overall conclusion is that Gated SAE features are similarly interpretable to baseline SAE features, while also providing better reconstruction fidelity (at fixed sparsity), as shown in the previous section. We provide more analysis of how these break down by site and layer in Appendix I.

5 Why do Gated SAEs improve SAE training?

5.1 Ablation study

In this section, we vary several parts of the Gated SAE training methodology to gain insight into which aspects of the training drive the observed improvement in performance. Gated SAEs differ from baseline SAEs in many respects, making it easy to incorrectly attribute the performance gains to spurious details without a careful ablation study. Fig. 5a shows Pareto frontiers for these variations; below we describe each variation in turn and discuss our interpretation of the results.

Unfreeze decoder : Here we unfreeze the decoder weights in L aux - i.e. allow this auxiliary task to update the decoder weights in addition to training f gate's parameters. Although this (slightly) simplifies the loss, there is a reduction in performance, suggesting that it is beneficial to limit the impact of the L1 sparsity penalty to just those parameters in the SAE that need it - i.e. those used to detect which features are active.

No r mag : Here we remove the r mag scaling parameter in Eq. (7), effectively setting it to zero, further tying f gate's and f mag's parameters together. With this change, the two encoder sublayers' preactivations can at most differ by an elementwise shift. 10 There is a slight drop in performance, suggesting r mag contributes somewhat to the improved performance of the Gated SAE.

Untied encoders : Here we check whether our choice to share the majority of parameters between the two encoders has meaningfully hurt performance, by training Gated SAEs with gating and ReLU encoder parameters completely untied. Despite the greater expressive power of an untied encoder, we see no improvement in performance - in fact a slight deterioration. This suggests our tying scheme (Eq. (7)) - where encoder directions are shared, but magnitudes and biases aren't - is effec-

0

50

100

0.88

0.9

0.92

0.94

0.96

0.98

1

SAE Type

Baseline (1.5× width)

Gated

Ablation: unfreeze decoder

Ablation: untie encoder layers

Ablation: no r_mag

L0

(Lower is sparser)

Loss Recovered

(Fidelity)

(a)

0

50

100

0.88

0.9

0.92

0.94

0.96

0.98

1

SAE Type

Baseline (equal width)

Gated

Baseline + rescale & shift

L0

(Lower is sparser)

Loss Recovered

(Fidelity)

(b)

tive at capturing the advantages of using a gated SAE while avoiding the 50% increase in parameter count and inference-time compute of using an untied SAE.

5.2 Is it sufficient to just address shrinkage?

As explained in Section 3.1, SAEs trained with the baseline architecture and L1 loss systematically underestimate the magnitudes of latent features' activations (i.e. shrinkage). Gated SAEs, through modifications to their architecture and loss function, overcome these limitations.

It is natural to ask to what extent the performance improvement of Gated SAEs is solely attributable to addressing shrinkage. Although addressing shrinkage would - all else staying equal - improve reconstruction fidelity, it is not the only way to improve SAEs' performance: for example, Gated SAEs could also improve upon baseline SAEs by learning better encoder directions (for estimating when features are active and their magnitudes) or by learning better decoder directions (i.e. better dictionaries for reconstructing activations).

Here we try to answer this question by comparing Gated SAEs trained as described in Section 3.2 with an alternative (architecturally equivalent) approach that also addresses shrinkage, but in a way that uses frozen encoder and decoder directions from a baseline SAE of equal dictionary size. 11 Any performance improvement over baseline SAEs obtained by this alternative approach (which we dub 'baseline + rescale & shift') can only be due to better estimations of active feature magnitudes, since by construction an SAE parameterized by 'baseline + rescale & shift' shares the same encoder and decoder directions as a baseline SAE.

As shown in Fig. 5b, although resolving shrinkage only ('baseline + rescale & shift') does improvement baseline SAEs' performance a little, a significant gap remains with respect to the performance of Gated SAEs. This suggests that the benefit of the gated architecture and loss comes from learning better encoder and decoder directions, not just from overcoming shrinkage. In Appendix D we ex-

plore further how Gated and baseline SAEs' decoders differ by replacing their respective encoders with an optimization algorithm at inference time.

6 Related work

Mechanistic interpretability Recent work in mechanistic interpretability has found recurring components in small and large LMs [38], identified computational subgraphs that carry out specific tasks in small LMs (circuits; [50]) and reverse-engineered how toy tasks are carried out in small transformers [30]. A central difficulty in this kind of work is choosing the right units of analysis. Sparse linear features have been identified as a promising candidate in prior work [52, 46]. The superposition hypothesis outlined by Elhage et al. [16] also provided a theoretical basis for this theory, sparking a new interest in using SAEs specifically to learn a feature basis [42, 8, 11, 21, 22, 4], as well as using SAEs directly for circuit analysis [26]. Other work has drawn awareness to issues or drawbacks with SAE training for this purpose, some of which our paper mitigates. Wright and Sharkey [51] raised awareness of shrinkage and proposed addressing this via fine-tuning. Gated SAEs, as discussed, resolve shrinkage during training. [35, 47, 2, 36] have also proposed general SAE training methodology improvements, which are mostly orthogonal to the architectural changes discussed in this work. In parallel work, Taggart [45] finds early improvements using a Jump ReLU [17], but with a different loss function, and without addressing the problems of the L1 penalty.

Classical dictionary learning Research into the general problem of sparse dictionary learning precedes transformers, and even deep learning. For example, sparse coding [13] studies how discrete and continuous representations can involve more representations than basis vectors, and sparse representations are also studied in neuroscience [48, 37]. One dictionary learning algorithm, k-SVD [1] also uses two stages to learn a dictionary like Gated SAEs. Although classical dictionary learning algorithms can be more powerful than SAEs (Appendix D), they are less suited for downstream uses like weights-based circuit analysis or attribution patching [44, 24], because they typically use an iterative algorithm to decompose activations, whereas SAEs make feature extraction explicit via the encoder. Bricken et al. [8] have also argued that classical algorithms may be 'too strong', in the sense they may learn features the LM itself could not access, whereas SAEs uses components similar to a LM's MLP layer to decompose activations.

7 Conclusion

In this work we introduced Gated SAEs which are a Pareto improvement in terms of reconstruction quality and sparsity compared to baseline SAEs (Section 4.1), and are comparably interpretable (Section 4.2). We showed via an ablation study that every key part of the Gated SAE methodology was necessary for strong performance (Section 5.1). This represents significant progress on improving Dictionary Learning on LMs - at many sites, Gated SAEs require half the L0 to achieve the same loss recovered (Fig. 8). This is likely to improve work that uses SAEs to steer language models [31], interpret circuits [26], or understand LM components across the full distribution [8].

Limitations & future work . Our benchmarking study focused on GELU-1L and models in the Pythia and Gemma families. It is therefore not certain that these results will generalise to other model families. On the other hand, the theoretical underpinnings of the Gated SAE architecture (Section 3) make no assumptions about LM architecture, suggesting Gated SAEs should be a Pareto improvement more generally. While we have confirmed that Gated SAE features are comparably interpretable to baseline SAE features, it does not necessarily follow that Gated SAE decompositions are equally useful for mechanistic interpretability. It is certainly possible that human interpretability of SAE features is only weakly correlated with either: (i) identification of the causally meaningful directions in a LM's activations; or (ii) usefulness on downstream tasks like circuit analysis or steering. A framework for scalably and objectively evaluating the usefulness of SAE decompositions (gated or otherwise) is still in its early stages [25] and further progress in this area would be highly valuable. It is plausible that some of the performance gap between Gated and baseline SAEs could be closed by inexpensive inference-time interventions that prune the many low activating features that tend to appear in baseline SAEs, mimicking Gated SAEs' thresholding mechanism. Finally, we would be most excited to see progress on using dictionary learning techniques to further inter-

pretability in general, such as to improve circuit finding [10, 26] or steering [49] in language models, and hope that Gated SAEs can serve to accelerate such work.

Author contributions

Senthooran Rajamanoharan developed the Gated SAE architecture and training methodology, inspired by discussions with Lewis Smith on the topic of shrinkage. Arthur Conmy and Senthooran Rajamanoharan performed the mainline experiments in Section 4 and Section 5 and led the writing of all sections of the paper. Tom Lieberum implemented the manual interpretability study of Section 4.2, which was designed and analysed by J´ anos Kram´ ar. Tom Lieberum also created Fig. 2 and Lewis Smith contributed Appendix D. Our SAE codebase was designed by Vikrant Varma who implemented it with Tom Lieberum, and was scaled to Gemma by Arthur Conmy, with contributions from Senthooran Rajamanoharan and Lewis Smith. J´ anos Kram´ ar built most of our underlying interpretability infrastructure. Rohin Shah and Neel Nanda edited the manuscript and provided leadership and advice throughout the project.

References

M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing , 54(11):43114322, 2006. doi: 10.1109/TSP.2006.881199.

J. Batson, B. Chen, A. Jones, A. Templeton, T. Conerly, J. Marcus, T. Henighan, N. L. Turner, and A. Pearce. Circuits Updates - March 2024. Transformer Circuits Thread , 2024. URL https://transformer-circuits.pub/2024/mar-update/index.html .

S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning , pages 2397-2430. PMLR, 2023. Apache License 2.0.

J. Bloom. Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2 Small, 2024. https://www.alignmentforum.org/posts/f9EgfLSurAiqRJySD/open-sourcesparse-autoencoders-for-all-residual-stream.

J. Bloom. SAELens, 2024. https://github.com/jbloomAus/SAELens.

T. Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing , 56(6):2370-2382, 2008.

T. Bolukbasi, A. Pearce, A. Yuan, A. Coenen, E. Reif, F. Vi´ egas, and M. Wattenberg. An interpretability illusion for bert. arXiv preprint arXiv:2104.07143 , 2021.

T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter,

A. Conmy. My best guess at the important tricks for training 1L SAEs, Dec 2023. https://www.lesswrong.com/posts/yJsLNWtmzcgPJgvro/my-best-guess-at-theimportant-tricks-for-training-1l-saes.

A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso. Towards automated circuit discovery for mechanistic interpretability, 2023.

H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.

Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 , ICML'17, page 933-941. JMLR.org, 2017.

M. Elad. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing . Springer, New York, 2010. ISBN 978-1-4419-7010-7. doi: 10.1007/ 978-1-4419-7011-4.

N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. URL https://transformer-circuits.pub/2021/framework/index.html .

N. Elhage, T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E. Tran-Johnson, J. Kaplan, J. Clark, T. Brown, S. McCandlish, D. Amodei, and C. Olah. Softmax linear units. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/solu/index.html.

N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy Models of Superposition. arXiv preprint arXiv:2209.10652 , 2022.

N. B. Erichson, Z. Yao, and M. W. Mahoney. Jumprelu: A retrofit defense strategy for adversarial attacks, 2019.

Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, L. Sifre, M. Rivi` ere, M. S. Kale, J. Love, P. Tafti, L. Hussenot, and et al. Gemma, 2024. URL https://www.kaggle. com/m/3301 . Apache License 2.0.

W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing, 2023.

N. P. Jouppi, D. H. Yoon, G. Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM , 63(7):67-78, 2020.

C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Sparse autoencoders work on attention layer outputs. Alignment Forum, 2024. https://www.alignmentforum.org/posts/DtdzGwFh9dCfsekZZ.

C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Attention SAEs scale to GPT-2 Small. Alignment Forum, 2024. https://www.alignmentforum.org/posts/FSTRedtjuHa4Gfdbr.

K. Konda, R. Memisevic, and D. Krueger. Zero-bias autoencoders and the benefits of coadapting features, 2015. URL https://arxiv.org/abs/1402.3337 .

J. Kram´ ar, T. Lieberum, R. Shah, and N. Nanda. Atp*: An efficient and scalable method for localizing llm behaviour to components. arXiv preprint arXiv:2403.00745 , 2024.

A. Makelov, G. Lange, and N. Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models , 2024. URL https://openreview.net/forum?id=MHIX9H8aYF .

S. Marks, C. Rager, E. J. Michaud, Y. Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024.

C. McDougall. SAE Visualizer, 2024. https://github.com/callummcdougall/sae vis.

N. Nanda. GELU-1L, 2022. URL https://huggingface.co/NeelNanda/GELU_1L512W_ C4_Code . MIT License.

N. Nanda. Open Source Replication & Commentary on Anthropic's Dictionary Learning Paper, Oct 2023. https://www.alignmentforum.org/posts/aPTgTKC45dWvL9XBF/open-sourcereplication-and-commentary-on-anthropic-s.

N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=9XFSbDPmdW .

N. Nanda, A. Conmy, L. Smith, S. Rajamanoharan, T. Lieberum, J. Kram´ ar, and V. Varma. [Summary] Progress Update #1 from the GDM Mech Interp Team. Alignment Forum, 2024. https://www.alignmentforum.org/posts/HpAr8k74mW4ivCvCu/summaryprogress-update-1-from-the-gdm-mech-interp-team.

A. Ng. Sparse autoencoder, 2011. CS294A Lecture notes, http://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf.

C. Olah. Mechanistic interpretability, variables, and the importance of interpretable bases, 2022. https://www.transformer-circuits.pub/2022/mech-interp-essay.

C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill , 2020. doi: 10.23915/distill.00024.001.

C. Olah, S. Carter, A. Jermyn, J. Batson, T. Henighan, T. Conerly, J. Marcus, A. Templeton, B. Chen, and N. L. Turner. Circuits Updates - January 2024. Transformer Circuits Thread , 2024. URL https://transformer-circuits.pub/2024/jan-update/index.html .

C. Olah, S. Carter, A. Jermyn, J. Batson, T. Henighan, J. Lindsey, T. Conerly, A. Templeton, J. Marcus, and T. Bricken. Circuits Updates - April 2024. Transformer Circuits Thread , 2024. URL https://transformer-circuits.pub/2024/april-update/index.html .

B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research , 37(23):3311-3325, 1997. doi: 10.1016/S0042-6989(97) 00169-7.

C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. In-context learning and induction heads. Transformer Circuits Thread , 2022. https://transformer-circuits.pub/2022/in-context-learning-and-inductionheads/index.html.

K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models, 2023.

Y. Pati, R. Rezaiifar, and P. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers , pages 40-44 vol.1, 1993. doi: 10.1109/ ACSSC.1993.342465.

S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kram´ ar, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024. URL https://arxiv.org/abs/2407.14435 .

L. Sharkey, D. Braun, and B. Millidge. [interim research report] taking features out of superposition with sparse autoencoders, 2022. https://www.alignmentforum.org/posts/z6QQJbtpkEAX3Aojj/interim-research-report-takingfeatures-out-of-superposition.

N. Shazeer. GLU variants improve transformer. CoRR , abs/2002.05202, 2020. URL https: //arxiv.org/abs/2002.05202 .

A. Syed, C. Rager, and A. Conmy. Attribution patching outperforms automated circuit discovery. arXiv preprint arXiv:2310.10348 , 2023.

G. M. Taggart. Prolu: A nonlinearity for sparse autoencoders, 2024. https://www.lesswrong.com/posts/HEpufTdakGTTKgoYF/prolu-a-pareto-improvementfor-sparse-autoencoders.

A. Tamkin, M. Taufeeque, and N. D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks, 2023.

A. Templeton, J. Batson, T. Henighan, T. Conerly, J. Marcus, A. Golubeva, T. Bricken, and A. Jermyn. Circuits Updates - February 2024. Transformer Circuits Thread , 2024. https://transformer-circuits.pub/2024/feb-update/index.html.

S. J. Thorpe. Local vs. distributed coding. Intellectica , 8:3-40, 1989.

A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization, 2023.

K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=NpsVSN6o4ul .

B. Wright and L. Sharkey. Addressing feature suppression in SAEs, Feb 2024. https://www.alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-featuresuppression-in-saes.

Z. Yun, Y. Chen, B. A. Olshausen, and Y. LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors, 2023.

Appendix

A Impact statement

This work introduces a method to obtain higher fidelity sparse decompositions of LM activations, under the hypothesis that progress in this area will ultimately help us understand the representations used by LMs. If successful, this could lead to greater understanding of how LMs complete tasks and novel mechanisms for controlling their behavior. Greater understanding and control could be put to beneficial uses such as mitigating the harms caused by current and future models, although bad actors could also misuse these tools, for example to circumvent safety training and steer models towards harmful behaviors. Currently, the SAE research program is in its early stages. For any potential misuse of SAEs, there is typically a more practical and effective way to achieve the same end using existing tooling, e.g. fine tuning or activation editing. Therefore, we see negligible negative societal impact in the short term. Longer term, advances in LM interpretability and control pose similar benefits and risks to advances in AI capabilities in general.

B Metrics for evaluating SAEs

SAEs are expected to decompose input activations sparsely, and yet in a manner that allows for faithful reconstruction. L0 and loss recovered are two metrics typically used [8] to measure sparsity and reconstruction fidelity respectively. These are defined as follows:

The L0 of a SAE is defined by the average number of active features on a given input, i.e E x ∼D ∥ f ( x ) ∥ 0 .

The loss recovered of a SAE is calculated from the average cross-entropy loss of the language model on an evaluation dataset, when the SAE's reconstructions are spliced into it. If we denote by CE ( ϕ ) the average loss of the language model when we splice in a function ϕ : R n → R n at the SAE's site during the model's forward pass, then loss recovered is

where ˆ x ◦ f is the autoencoder function, ζ : x ↦→ 0 the zero-ablation function and Id : x ↦→ x the identity function. According to this definition, a SAE that always outputs the zero vector as its reconstruction would get a loss recovered of 0%, whereas a SAE that reconstructs its inputs perfectly would get a loss recovered of 100%.

B.1 Issues with the loss recovered metric

In this paper, we have used loss recovered as defined in Bricken et al. [8] to measure reconstruction fidelity. However, there are deficiencies with this metric:

Firstly, zero-ablation is arguably too poor a baseline for defining the zero-point of this metric and mean-ablation is better justified. Using the mean-ablation function µ : x ↦→ E x ′ ∼D x ′ , instead of ζ in the definition of loss recovered above would also have the benefit that SAEs' loss recovered would tend towards zero in the limit L 0 → 0 , instead of tending to a positive value as it does when computing loss recovered using zero-ablation.

Furthermore, the very fact we normalise the increase in the spliced LM's loss when computing loss recovered makes it difficult to compare the impact of splicing SAEs at different sub-layers of the model. For example, mean or zero-ablating the output of a MLP layer typically has a much milder impact on LM loss than mean or zero-ablating the residual stream, making the denominator in Eq. (8) smaller for MLP SAEs than for residual stream SAEs. So we unsurprisingly find that residual stream SAEs' loss recovered tend to be much higher than MLP or attention SAEs. This suggests that it may be more informative to report raw changes in cross-entropy loss ('delta LM loss') instead of using a normalised metric like loss recovered, since these are directly comparable across SAEs trained on different sub-layers of the same LM.

In practice however, both mean-ablated loss recovered and delta LM loss are related to zero-ablated loss recovered (the metric used in this paper) by an affine transformation. In other words, all the

loss recovered versus L0 figures in this paper would look identical if we had used one of these other metrics instead, with the only difference being the tick labels on the y-axis. Consequently, none of the conclusions we draw in this paper would be affected by using one of these other reconstruction fidelity metrics instead. Nevertheless, we draw the reader's attention to our subsequent work [41], which compares Gated SAEs to other SAE varieties adopting the delta LM loss metric, instead of loss recovered, for measuring reconstruction fidelity. 12

C Measuring shrinkage

As described in Section 3.1, the L1 sparsity penalty used to train baseline SAEs causes feature activations to be systematically underestimated, a phenomenon called shrinkage . Since this in turn shrinks the reconstructions produced by the SAE decoder, we can observe the extent to which a trained SAE is affected by shrinkage by measuring the average norm of its reconstructions.

Concretely, the metric we use is the relative reconstruction bias ,

i.e. γ -1 is the optimum multiplicative factor by which an SAE's reconstructions should be rescaled in order to minimise the L2 reconstruction loss; γ = 1 for an unbiased SAE and γ < 1 when there's shrinkage. 13 Explicitly solving the optimization problem in Eq. (9), the relative reconstruction bias can be expressed analytically in terms of the mean SAE reconstruction loss, the mean squared norm of input activations and the mean squared norm of SAE reconstructions, making γ easy to compute and track during training:

where the second equality makes use of the identity 2 a · b ≡ ∥ a ∥ 2 2 + ∥ b ∥ 2 2 -∥ a -b ∥ 2 2 . Notice from the second expression for γ that an unbiased reconstruction ( γ = 1 ) therefore satisfies

In other words, an unbiased but imperfect SAE (i.e. one that has non-zero reconstruction loss) must have mean squared reconstruction norm that is strictly less than the mean squared norm of its inputs even without shrinkage . Shrinkage makes the mean squared reconstruction norm even smaller.

D Inference-time optimization

The task SAEs perform can be split into two sub-tasks: sparse coding, or learning a set of features from a dataset, and sparse approximation, where a given datapoint is approximated as a sparse linear combination of these features. The decoder weights are the set of learned features, and the mapping represented by the encoder is a sparse approximation algorithm. Formally, sparse approximation is the problem of finding a vector α that minimises;

i.e. that best reconstructs the signal x as a linear combination of vectors in a dictionary D , subject to a constraint on the L0 pseudo-norm on α . Sparse approximation is a well studied problem, and SAEs are a weak sparse approximation algorithm. SAEs, at least in the formulation conventional in dictionary learning for language models, in fact solve a slightly more restricted version of this problem where the weights α on each feature are constrained to be non-negative, leading to the related problem

In this paper, we do not explore using more powerful algorithms for sparse coding. This is partly because we are using SAEs not just to recover a sparse reconstruction of activations of a LM; ideally we hope that the learned features will coincide with the linear representations actually used by the LM, under the superposition hypothesis. Prior work [8] has argued that SAEs are more likely to recover these due to the correspondence between the SAE encoder and the structure of the network itself; the argument is that it is implausible that the network can make use of features which can only be recovered from the vector via an iterative optimisation algorithm, whereas the structure of the SAE means that it can only find features whose presence can be predicted well by a simple linear mapping. Whether this is true remains, in our view, an important question for future work, but we do not address it in this paper.

In this section we discuss some results obtained by using the dictionaries learned via SAE training, but replacing the encoder with a different sparse approximation algorithm at inference time. This allows us to compare the dictionaries learned by different SAE training regimes independently of the quality of the encoder. It also allows us to examine the gap between the sparse reconstruction performed by the encoder against the baseline of a more powerful sparse approximation algorithm. As mentioned, for a fair comparison to the task the encoder is trained for, it is important to solve the sparse approximation problem of Eq. (12), rather than the more conventional formulation of Eq. (11), but most sparse approximation algorithms can be modified to solve this with relatively minor changes.

Solving Eq. (12) exactly is equivalent to integer linear programming, and is NP hard. The integer linear programs in question would be large, as our SAE decoders routinely have hundreds of thousands of features, and solving them to guaranteed optimality would likely be intractable. Instead, as is commonly done, we use iterative greedy algorithms to find an approximate solution. While the solution found by these sparse approximation algorithms is not guaranteed to be the global optimum, these are significantly more powerful than the SAE encoder, and we feel it is acceptable in practice to treat them as an upper bound on possible encoder performance.

For all results in this section, we use gradient pursuit, as described in Blumensath and Davies [6], as our inference time optimisation (ITO) algorithm. This algorithm is a variant of orthogonal matching pursuit [40] which solves the orgothonalisation of the residual to the span of chosen dictionary elements approximately at every step rather than exactly, but which only requires matrix multiplies rather than matrix solves and is easier to implement on accelerators as a result. It is possibly not crucial for performance that our optimisation algorithm be implementable on TPUs, but being able to avoid a host-device transfer when splicing this into the forward pass allowed us to re-use our existing evaluation pipeline with minimal changes.

When we use a sparse approximation algorithm at test time, we simply use the decoder of a trained SAE as a dictionary, ignoring the encoder. This allows us to sweep the target sparsity at test time without retraining the model, meaning that we can plot an entire Pareto frontier of loss recovered against sparsity for a single decoder, as in done in Fig. 7.

Fig. 6 compares the loss recovered when using ITO for a suite of SAEs decoders trained with both methods at three different test time L0 thresholds. This graph shows a somewhat surprising result; while Gated SAEs learn better decoders generally, and often achieve the best loss recovered using ITO close to their training sparsity, SAE decoders are often outperformed by decoders which achieved a higher test time L0; it's better to do ITO with a target L0 of 10 with an decoder with an achieved L0 of around 100 during training than one which was actually trained with this level of sparsity. For instance, the left hand panel in Fig. 6 shows that SAEs with a training L0 of 100 are better than those with an L0 of around 10 at almost every sparsity level in terms of ITO reconstruction. However, gated SAE dictionaries have a small but real advantage over standard SAEs in terms of loss recovered at most target sparsity levels, suggesting that part of the advantage of gated SAEs is that they learn better dictionaries as well as addressing issues with shrinkage. However, there are some subtleties here; for example, we find that baseline SAEs trained with a lower sparsity penalty (higher training L0) often outperform more sparse baseline SAEs according to this measure, and the best performing baseline SAE (L0 ≈ 99 ) is comparable to the best performing Gated SAE (L0 ≈ 20 ).

Fig. 7 compares the Pareto frontiers of a baseline model and a gated model to the Pareto frontier of an ITO sweep of the best performing dictionary of each. Note that, while the Pareto curve of the baseline dictionary is formed by several models as each encoder is specialised to a given sparsity

max_10=10

max_10=30

max_10=100

sae type

0.98

baseline

gated

0.96

0.94

Loss Recovered

0.92

0.9

ITO

0.88

0.86

0.84

0.82

10

2

5

100

Z

10

2

5

100

Z

10

2

5

100

SAE LO

SAE LO

SAE LO

baseline

gated

0.98

rescale+shift

baseline (ITO, best decoder)

LossRecovered

0.96

·gated (ITO,best decoder)

0.94

0.92

0.9

10

6

100

LO

level, as mentioned, ITO lets us plot a Pareto frontier by sweeping the target sparsity with a single dictionary; here we plot only the best performing dictionary from each model type to avoid cluttering the figure. This figure suggests that the performance gap between the encoder and using ITO is smaller for the gated model. Interestingly, this cannot solely be explained by addressing shrinkage, as we demonstrate by experimenting with a baseline model which learns a rescale and shift with a frozen encoder and decoder directions.

E More loss recovered / L0 Pareto frontiers

In Fig. 8 we show that Gated SAEs outperform baseline SAEs. In Fig. 9 we show that Gated SAEs ourperform baseline SAEs at all but one MLP output or residual stream site that we tested on.

In Fig. 9 at the attention output pre-linear site at layer 27, loss recovered is bigger than 1.0. On investigation, we found that the dataset used to train the SAE was not identical to Gemma's pretraining dataset, and at this site it was possible to mean ablate this quantity and decrease loss - explaining why SAE reconstructions had lower loss than the original model.

Table 1 provides cross-entropy losses for the Gemma-7B and Pythia-2.8B, both before and after zero-ablating specific sub-layers of these models, to help provide further context for interpreting the loss recovered results presented in this paper; 100% loss recovered corresponds to the SAE-spliced language model attaining a loss matching the original language model, whereas 0% loss recovered corresponds to the SAE-spliced language model attaining a loss matching the language model with the corresponding sub-layer zero-ablated.

F Further shrinkage plots

In Fig. 10, we show that Gated SAEs resolve shrinkage, as measured by relative reconstruction bias (Appendix C), in Pythia-2.8B.

G Training and evaluation: hyperparameters and other details

G.1 Training

G.1.1 General training details

Other details of SAE training are:

SAE Widths . Our SAEs have width 2 17 for most baseline SAEs, 3 × 2 16 for Gated SAEs, except for the (Pythia-2.8B, Residual Stream) sites we used 2 15 for baseline and 3 × 2 14 for Gated since early runs at these sites had lots of learned feature death.

Training data . We use activations from hundreds of millions to billions of activations from LM forward passes as input data to the SAE. Following Nanda [29], we use a shuffled buffer of these activations, so that optimization steps don't use data from highly correlated activations. 14

Resampling . We used resampling , a technique which at a high-level reinitializes features that activate extremely rarely on SAE inputs periodically throughout training. We mostly

0

50

100

150

0.6

0.8

1

0

20

40

60

80

100

120

140

0

20

40

60

80

100

120

140

0.6

0.8

1

0.6

0.8

1

0.6

0.8

1

0.6

0.8

1

SAE Type

Gated

Baseline

Residual stream post-MLP

MLP output

L0

(Lower is sparser)

Attention output pre-linear

Layer 28

Layer 20

Loss Recovered

(Fidelity)

Layer 16

Layer 12

Layer 4

0

50

100

150

200

0.4

0.6

0.8

1

1.2

0

50

100

150

200

0

50

100

150

200

0.4

0.6

0.8

1

1.2

0.4

0.6

0.8

1

1.2

0.4

0.6

0.8

1

1.2

SAE Type

Gated

Baseline

Residual stream post-MLP

MLP output

L0

(Lower is sparser)

Attention output pre-linear

Layer 27

Layer 20

Loss Recovered

(Fidelity)

Layer 13

Layer 6

0

100

50

150

1

0.8

1.2

0

100

50

150

0

100

50

150

1

0.8

1.2

1

0.8

1.2

1

0.8

1.2

SAE Type

Gated

Baseline

Residual stream post-MLP

MLP output

L0

(Lower is sparser)

Attention output pre-linear

Layer 27

Layer 20

Relative reconstruction bias γ

Layer 13

Layer 6

follow the approach described in the 'Neuron Resampling' appendix of Bricken et al. [8], except we reapply learning rate warm-up after each resampling event, reducing learning rate to 0.1x the ordinary value, and, increasing it with a cosine schedule back to the ordinary value over the next 1000 training steps.

Optimizer hyperparameters . We use the Adam optimizer with β 2 = 0 . 999 and β 1 = 0 . 0 , following Templeton et al. [47], as we also find this to be a slight improvement to training. We use a learning rate warm-up. See Appendix G.1.2 for learning rates of different experiment.

Decoder weight norm constraints . Templeton et al. [47] suggest constraining columns to have at most unit norm (instead of exactly unit norm), which can help distinguish between productive and unproductive feature directions (although it should have no systematic impact on performance). However, we follow the original approach of constraining columns to have exact unit norms in this work for the sake of simplicity.

Compute resources . Individual SAEs were each trained on TPU-v3 slices with a 2x2 topology [20]. The same chips were used to generate LM activations on-the-fly, train SAE parameters and evaluate SAEs during training, using up to 8-way model parallelism. With this setup, the time to train a SAE varies by SAE width, LM residual stream dimension, sequence length, layer and site. 15 We also used a negligible amount of compute on resampling (Appendix G), evaluation (e.g. Figure 1) and interpretability experiments (Section 4.2). Training wall clock time ranges from around 7 hours to train on GELU-1L MLP activations to around 47 hours to train on Gemma-7B sites at layer 27. We estimate that we used twice as much compute as used in the paper on preliminary experiments.

G.1.2 Experiment-specific training details

We use learning rate 0.0003 for all Gated SAE experiments, and the GELU-1L baseline experiment. We swept for optimal baseline learning rates for the GELU-1L baseline to generate this value. For the Pythia-2.8B and Gemma-7B baseline SAE experiments, we divided the L2 loss by E || x || 2 , motivated by better hyperparameter transfer, and so changed learning rate to 0.001 and 0.00075. We didn't see noticeable difference in the Pareto frontier and so did not sweep this hyperparameter further.

We generate activations from sequences of length 128 for GELU-1L, 2048 for Pythia-2.8B and 1024 for Gemma-7B.

We use a batch size of 4096 for all runs. We use 300,000 training steps for GELU-1L and Gemma-7B runs, and 400,000 steps for Pythia-2.8B runs.

G.1.3 Lessons learned scaling SAEs

Learned feature death is unpredictable . In Fig. 11 there are few patterns that can be gleaned from staring at which runs have high numbers of dead learned features (called dead neurons in Bricken et al. [8]).

Resampling makes hyperparameter sweeps difficult . We found that resampling caused L0 and loss recovered to increase, similar to Conmy [9].

Training appears to converge earlier than expected . We found that we did not need 20B tokens as in Bricken et al. [8], as generally resampling had stopped causing gains and loss curves plateaued after just over one billion tokens.

G.2 Evaluation

We evaluated the models on over a million held-out tokens.

H Equivalence between gated encoder with tied weights and linear encoder with non-standard activation function

In this section we show under the weight sharing scheme defined in Eq. (7), a gated encoder as defined in Eq. (5) is equivalent to a linear layer with a non-standard (and parameterized) activation function.

Without loss of generality, consider the case of a single latent feature ( M = 1 ) and set the preencoder bias to zero. In this case, the gated encoder is defined as and the weight sharing scheme becomes

with a non-negative parameter ρ mag ≡ exp( r mag ) .

Substituting Eq. (14) into Eq. (13) and re-arranging, we can re-express ˜ f ( x ) as a single linear layer

with the parameterized activation function

σ θ ( z ) := ✶ z>θ ReLU ( z ) . (16) called JumpReLU in a different context [17]. Fig. 12 illustrates the shape of this activation function.

I Further analysis of the human interpretability study

We perform some further analysis on the data from Section 4.2, to understand the impact of different sites, layers, and raters.

I.1 Sites

We first pose the question of whether there's evidence that the sites had different interpretability outcomes. A Friedman test across sites shows significant differences (at p = 0 . 047 ) between the Gated-vs-Baseline differences, though not ( p = 0 . 92 ) between the raw labels.

Breaking down by site and repeating the Wilcoxon-Pratt one-sided tests and computing confidence intervals, we find the result on MLP outputs is strongest, with mean 0.40, significance p = 0 . 003 , and CI [0.18, 0.63]; this is as compared with the attention outputs ( p = 0 . 47 , mean .05, CI [-0.16, 0.26]) and final residual ( p = 0 . 59 , mean -0.07, CI [-0.28, 0.12]) SAEs.

I.2 Layers

Next we test whether different layers had different outcomes. We do this separately for the 2 models, since the layers aren't directly comparable. We run 2 tests in each setting: Page's trend test (which tests for a monotone trend across layers) and the Friedman test (which tests for any difference, without any expectation of a monotone trend).

Results are presented in Table 2; they suggest there are some significant nonmonotone differences between layers. To elucidate this, we present 90% BCa bootstrap confidence intervals of the mean raw label (where 'No'=0, 'Maybe'=1, 'Yes'=2) and the Gated-vs-Baseline difference, per layer, in Fig. 15 and Fig. 16, respectively.

I.3 Raters

In Table 3 we present test results weakly suggesting that the raters differed in their judgments. This underscores that there's still a significant subjective component to this interpretability labeling. (Notably, different raters saw different proportions of Pythia vs Gemma features, so aggregating across the models is partially confounded by that.)

50

100

0

0

0.5

1

50

100

0

50

100

0

0

0.5

1

0

0.5

1

0

0.5

1

SAE Type

Gated

Baseline

Residual stream post-MLP

MLP output

L0

(Lower is sparser)

Attention output pre-linear

Layer 27

Layer 20

Proportion of features that are alive

(Last 1e6 tokens)

Layer 13

Layer 6

𝜎

model=pythia-2.8b

model = gemma-7b-pt

YES

9

9

32

11

12

40

ated

MAYBE

4

4

10

3

9

12

NO

1

1

5

2

1

6

NO

MAYBE

YES

NO

MAYBE

YES

baseline

baseline

Activations in range 29.0299 to 32.2555

this tameness to its extreme vor acity. Be this as it may ; when a flock of these birds are feeding in a pine-tree in the manner described, they nutrition labels; the Korea Food Rescarch Institute; and revised data from the Rural Development Administration's rescarch analysis. For foods that were not found in any database mentioned above，we

Activationsinrange25.8044to29.0299

by Manning Oil & Gas will pass to Perm ina upon entry into Indonesia, with Manning′s out of pocket costs to be recovered by the forty percent formula ."In'n velocity distribution in the depth direction.n Patent Document 1 focuses on the rising and sinking of the ocean current power generation device depending on the current velocity as described above, and terminal drops down to the same lower level. When both terminals are transmitting the same flag sequence this step completes. n Arising from the set of procedures described above that are other nation.The Govermment of Pakistan will not, without the prior agreement of the Govermment of the United ·States, devote such assistance to purposes other than thosefor which it ed foreign cxchange, import restrictions, and importers in cxpericnced in assessing technology or in cxport marketing. In contrast, firms whose characteristics are the opposite of those enumerated above are vessel is a ship not operating on a regular route or schedule ; that is, tramp steamcrs do not have the established schedules of the other two types of carriers , a trench 45 corresponding to the opening 43 is formed in the silicon base material 40 (FIG. 4B).n However, during the dry etching deseribed above, SiO and I hereby promise to ask Mr. Long to excuse us if we ignore him more in the future. nn Without any one of the friends and family listed here de la Pena and A.Cetto, E.Santos, O. Thcimer, and G. Goedecke; see the reviewsmentioned above.More eding strategy -{ data -label" fig: initialiation 5"}](initialisation 5. pdf)( width " 50. 00ooo %") \n'n The hed ging deseribed above is carried

Activationsinrange22.5788to25.8044

7 entitled "The Introduction of Low Erucic Acid Rapesecd Varicties Into Canadian Production" by J. K. Daun from thc previously identified Academic Press in a case where, for example, a coronary artery (blood vessel) present in the vicinity of the hcart is observed with use of the manual tracking method by the operator , and total medical expense management. 'n'n We are fortunate to live in an area with some of the best hospitals in the nation. The aproaches I have outlined for improved 10 kIU/ml, TUDCA: 1, 5, 15 mg/ml; pH 5.0) were obtained using the method described and by of two subunits whsebinding is contingent nphosphorylation by scparately activated kinasesn3Effects ofErorsErors in the signal transduction and regulationpathways dsribed above can causc willfeel its force. But can their irresistible skepticism conceming potry translation and paraphrase be reconciled with the obvious logic of the simple refutationnn verdict judgment of acquittal as to yet another'n count. In the end, Kim was convicted only on counts six and'n seven, covering the incidents described above . nn propose a new algorithmic approach, exhibited in Fig.{fig:teaser], that improves upon the SLICapproach. motivated by the drawbacks discussed above. We a collection of functions that sometimes call functions in other packages ( where some bit of work has already been coded -up). Is this scenarioor 2 above ?n substitutions can change the aesthetic, emotive or imagistic quality of a pocm, how could any of them change meaning?'nn Despite the simple refutation, the he

Figure 14: An extract of a feature visualization dashboard used to rate features in the interpretability study described in Section 4.2. The left-hand pane provides aggregate information, including a feature histogram and the tokens most promoted and demoted by the feature being rated. The rest of the dashboard displays samples of text on which the feature activates to various degrees. Holding the mouse over a text token reveals a hover showing the exact activation level at that token. Although not shown here, the full dashboard provides examples across the full range of activations, down to examples on which the feature fails to activate. This particular feature, taken from a layer 20 Gemma-7B residual stream Gated SAE seems to promote completions like 'above', 'aforementioned', 'mentioned above' etc. in contexts where such a completion would be likely.

800

600

400

200

0

230 2 2 24 24 2 2 2 215117 115 1413 1211 9 8

NEGATIVELOGITS

POSITIVELOGITS

EOF

-0.08

above

+0.53

cheat

-0.08

above

+0.46

tomat

-0.08

aforementioned

+0.38

ot

-0.07

foregoing

+0.37

getElement

-0.07

mentioned

+0.36

invoke

-0.07

Above

+0.35

strugg

-0.07

Above

EE'0+

lie

-0.07

described

+0.31

compute

-0.07

mentioned

+0.31

anean

-0.07

deseribed

+0.26

(FEATURES*UNEMBED)HISTOGRAM

15k

10k

5k

0

0.17

0.34

0.51

1.8

1.7

1.6

90% CIfor raw label

1.5

1.4

1.3

1.2

1.1

gemma-7b-pt L6

gemma-7b-pt L13

gemma-7b-pt L20

gemma-7b-pt L27

pythia-2.8b L4

pythia-2.8b L12

pythia-2.8b L16

pythia-2.8b L20

90% CI for Delta from Baseline to Gated

0.5

-0.5

gemma-7b-pt L6

gemma-7b-pt L13

gemma-7b-pt L20

gemma-7b-pt L27

pythia-2.8b L4

pythia-2.8b L12

pythia-2.8b L16

pythia-2.8b L20

site =ATTN_OUTPUT_PRE_LINEAR

site = FINAL_RESIDUAL

site = MLP_OUTPUT

YES

3

3

8

3

13

6

3

11

gated

MAYBE

1

2

6

2

1

3

1

1

1

NO

1

1

3

1

1

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

baseline

baseline

baseline

layer = 4

layer = 12

layer = 16

layer = 20

layer = 28

YES

2

10

2

3

4

1

7

1

2

5

4

3

gated

MAYBE

2

1

1

3

2

3

1

3

2

NO

2

1

1

1

2

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

baseline

baseline

baseline

baseline

baseline

site = ATTN_OUTPUT_PRE_LINEAR

site=FINAL_RESIDUAL

site = MLP_OUTPUT

YES

3

4

12

2

5

14

6

3

14

MAYBE

5

7

1

2

3

2

2

2

NO

1

1

1

3

1

2

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

baseline

baseline

baseline

layer = 6

layer =13

layer = 20

layer = 27

YES

2

4

14

3

4

8

2

2

13

4

2

5

gated

MAYBE

2

1

1

1

6

2

2

2

4

3

NO

1

1

3

2

1

1

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

NO

MAYBE

YES

baseline

baseline

baseline

baseline

J Pseudo-code for Gated SAEs and the Gated SAE loss function

Figure 20: Pseudo-code for the Gated SAE loss function. Note that this pseudo-code is written for expositional clarity. In practice, taking into account parameter tying, it would be more efficient to rearrange the computation to avoid unnecessarily duplicated operations.

K Tables of Results

We evaluated the models on over a million held-out tokens. Tables 4-11 show summary stats from training runs on the Pareto frontier.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly stated our main contributions in the abstract and introduction, and have taken care to ensure they accurately reflect the contents of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We include a limitations section under the conclusion, discussing the limitations of our experimental design and of open questions around SAEs that our work does not address.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The main contributions of this paper are not theoretical. There are two minor points in the main paper - deriving an expression for the shrinkage metric γ and showing that a tied-weights Gated SAE is equivalent to a single-layer encoder SAE with a JumpReLU activation - where fuller explanations are provided in Appendix C and Appendix H respectively (and referenced from the main text). These are not numbered theorems as this would be excessive for these low-importance (and easily derived) points.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The main body of the paper provides a detailed explanation of the Gated SAE architecture and loss function (Section 3) that are the main contributions of the paper; additionally, pseudo-code to help reproduction is provided in Appendix J. We provide a high level summary of the training approach in Section 2 and Section 3, with details on hyperparameters and compute requirements in Appendix G. We also explain the methodologies for our benchmarking and manual interpretability experiments in Section 4 in sufficient detail to be able to reproduce the results there.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We are unable to provide open access to the activation datasets or code used to train the SAEs in our experiments. However, the experiments we have reported involved training SAEs on solely open-source language models, and we have provided pseudo-code for our main contributions (Appendix J). These - alongside open-source codebases that are available for training SAEs (e.g. [5]) - should be enough to reproduce our main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: The main body of the paper - in particular the training and evaluation subsections of Section 2, the Gated SAE definition of Section 3.2 and the experimental methodology subsections of Section 4 - explain training and evaluation details to a sufficient level of detail to be able to understand and appreciate the results. Details of hyperparameters, optimizer type, etc are provided in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We do measure statistical significance in our interpretability study. Section 4.2 includes a detailed explanation of the statistical analysis undertaken to compare Gated and baseline SAEs (continued in Appendix I). The error bar methodology for Fig. 4 is explained in Footnote 9. However, error bars are not shown on the Pareto curve plots because it would have been computationally prohibitive. Nevertheless, the smoothness (or otherwise) of these curves across multiple values of λ does informally convey the noisiness of these curves.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The final bullet in Appendix G covers the requirements below.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the Code and believe our research conforms to its requirements.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed potential positive and negative societal impacts of our work in our impact statement (Appendix A).

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We are not releasing models or data with this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators / owners of the three open-source model families used in our experiments [28, 3, 18] are credited in the main text, with license details (which have been respected) provided as part of their entries in the Bibliography.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are being introduced in the paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The interpretability experiment (Section 4.2) did not use any external human subjects. All raters were permanent members of the research group conducting the study, hence no compensations details are provided. The instructions given to raters are provided in Section 4.2. The feature dashboards used by the raters were generated using a popular

SAE visualisation library [27] that we have referenced when describing the experimental methodology; representative screenshots are available at the library's GitHub page (provided in its Bibliography entry).

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: IRB approvals were not required, as no external human subjects were used in the interpretability study.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a Gated Sparse Autoencoder (Gated SAE) to mitigate the standard SAEs' biases, such as shrinkage, which systematically underestimate the feature activations from SAEs. The key difference between Gated SAE and SAE is that the Gated SAE separates affine transformations within the encoder in order to decide which dictionary elements to use in a reconstruction loss, and estimate the coefficients of active elements, although with the 50% more computing required to achieve. Comprehensive experiments are conducted to compare and verify how good the Gated SAE is to standard SAE, including a blinded human study to rate and compare the interpretability of randomly sampled Gated and baseline SAE features.\n\nSTRENGTHS:\n- A new architecture of SAE inspired by GRU is proposed to include a gate mechanism to mitigate shrinkage bias\n- Comprehensive quantitative experiments including ablation studies to evaluate the proposed Gated SAE compared to SAEs\n- A human evaluation to rate randomly sampled features from Gated SAE and SAE\n\nWEAKNESSES:\n- It is not very straightforward to understand how well the features from Gated SAE are compared to SAE based on Figure 4. Some case studies based on the open-source SAE visualizer library [1] are required to help better understand this.\n- It will be better to see more case studies on downstream tasks to compare Gated SAE and SAE, e.g., automatic circuit detection [2]\n\n[1] C. McDougall. SAE Visualizer, 2024. https://github.com/callummcdougall/sae_vis\n\n[2] Huben, Robert, et al. \"Sparse Autoencoders Find Highly Interpretable Features in Language Models.\" The Twelfth International Conference on Learning Representations. 2023.\n\nQUESTIONS:\n- As mentioned in the weakness above, the interpretability analysis was conducted via human rating for randomly sampled feature visualizations. However, those visualizations are not included in the main body or appendix. It will be better to help understand how well the sampled features are when comparing Gated SAE and SAE.\n- Although the Gated SAE has good performance on the loss recovered (fidelity) and relative reconstruction bias, it is still not clear whether features from Gated SAE are better than SAE under downstream tasks. It will make this work very solid if some analysis of mini downstream applications can be conducted, e.g., IOI task, greater-than task, etc.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to resolve the issue of feature shrinkage in sparse autoencoders (SAEs) by replacing the SAE ReLU activation function with a gated ReLU unit. The weight-tying scheme they use for the gated unit effectively turns it into a jump ReLU activation function.\nThey train gated SAEs and baseline SAEs on a one layer transformer, Pyhtia-2.8B and Gemma-7B. They find that gated SAEs eliminate systematic shrinkage, and consistently outperform baseline SAEs on the pareto-curve of sparsity, measured by the L0 pseudonorm, and faithfulness, measured by the model loss recovered relative to a zero ablation baseline. \nThey run various additional tests involving variations of the gated and baseline SAE architectures, including combinations of SAE dictionaries with the classic gradient pursuit algorithm for choosing sparse feature coefficients at inference time. They conclude that the Pareto improvement of their gated SAEs over their baseline SAEs is due in part to better feature dictionaries, in addition to better estimated feature coefficients.\nThey compare the subjective interpretability of 150 gated SAE and baseline SAE features in Pythia-2.8B and 192 features in Gemma-7B, using a blinded analysis of activating dataset examples. They find that the features were similarly interpretable.\n\nSTRENGTHS:\nThe paper attempts to address a substantive practical problem with current SAE training methods.\n\n\nThe paper's proposed new architecture is evaluated extensively, and many detailed additional investigations on the individual effects of various parts of the gated SAE architecture are described in sections 5.1, 5.2 and Appendix D. \n\n\nI find Appendix D interesting in its own right, since it shows quantitative comparisons between SAE methods and the classic gradient pursuit optimization algorithm, as well as mixing SAE feature dictionaries with gradient pursuit for sparse approximation of feature coefficients. I have not encountered such a comparison before. \n\n\nFor the most part, good documentation of all their process is provided, and the writing and presentation are very clear in general.\n\nWEAKNESSES:\nThe paper does not really address the concern that gated SAEs may outperform baseline SAEs in part by implicitly widening the definition of what it means for ‘features’ to be represented in the model. As the paper itself notes in Appendix D, though other more powerful sparse coding algorithms greatly outperform SAEs in terms of reconstruction and sparsity, there are concerns that the greater expressivity of these techniques lets them find spurious ‘features’ that would not be accessible to the model’s own internal computations. An SAE can only find features that are represented in the sense that their current values can be read off with a single ReLU probe, while an inference time algorithm or a multi-layer probe may read off ‘feature’ values that the model itself could not possibly access using a single MLP layer. A gated ReLU is far less expressive than an algorithm like gradient pursuit, but more expressive than a ReLU. So to what extent do gated SAEs outperform baseline SAEs merely because they are implicitly working with a more relaxed definition of what it means for a feature to be represented in the model? Figure 6 in Appendix D incidentally investigates this somewhat, since it attempts to compare the quality of gated vs. baseline dictionaries independent of their coefficients. However, the results there seem inconsistent, with smaller performance gaps and baseline SAEs outperforming gated SAEs at higher L0. I think this issue of the representational power of the probe used is pretty central for contextualizing the results, and should at least have been discussed.\n\nThroughout the paper, the authors present reconstruction scores for SAEs in terms of the fraction of model loss recovered compared to a zero-ablation baseline. I think this metric obscures vital information. Lowering CE loss from e.g. 4.5 to 4.0 is typically much easier than lowering it from 1.5 to 1.0. Thus, the same difference in loss recovered between two SAEs can correspond to very different gaps in SAE quality. Without the raw CE scores, there is no direct way to infer how large the gap is quantitatively. At minimum, these raw CE scores should be in the supplementals. Better yet, the recovered performance could additionally be reported in terms of the compute required to train a model with the same CE score, as suggested in https://arxiv.org/abs/2406.04093.\n\nQUESTIONS:\nWhy are the raw CE loss recovered scores not in the paper? Since it is typically much harder to lower CE loss from e.g. 4.5 to 4.0 than from 1.5 to 1.0, it is difficult to evaluate the quality gap between baseline and gated SAEs, or the quality of the baseline SAEs, without these scores.\n\nLIMITATIONS:\nAll addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a new technique under mechanistic interpretability's sparse autoencoders. By using a less naive SAE, with a gating mechanism and a little extra computation, the paper shows a decent improvement over the baseline.\n\nSTRENGTHS:\nThis work addresses the important issue of interpreting transformer-based LLMs and clearly demonstrates an interesting method. The mechanistic interpretability community will certainly find this work of interest.\n\nThe writing is well written and fairly easy to follow, the results are clearly presented, and all relevant aspects of the method are appropriately ablated\n\nI liked the setup of the internal user study; I think future papers will follow the design of the study closely.\n\nThe work's cited throughout the manuscript are incredibly thorough.\n\nWEAKNESSES:\nWhile I generally like the paper, I have two primary concerns:\n\n* The architecture and loss are somewhat difficult to understand. I did appreciate the pseudo-code in the appendix, but I feel for readers not familiar with SAEs may have a hard time, especially with the optimization-based design choices of weight tying. Perhaps explaining weight tying later in 3.2 would help. I would especially prefer if a few lines of pseudo code could be added in the main paper, next to figure two.\n* The user study results. I don't mind the small change in means between the method and the baseline, but the explainable AI community has been around for a long time and the shift from studies with a few experts to larger cohorts has been the norm for a while now. Just because there's a rebranding to mechanistic interpretability doesn't mean this field should settle for underpowered studies. Nevertheless, I do find the study setup itself to be well articulated and a very useful starting point for future work in this area.\n\nMinor:\nSome of the design choices (weight-tying, no r_mag, etc) aren't well explained until the ablation where we find they are primarily for optimization. This could be motivated a little earlier, i.e. that the pareto improvement comes from the separation, and not those choices.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Gated Sparse Autoencoders (Gated SAEs), an improvement over standard sparse autoencoders (SAEs) for decomposing language model activations. The key idea is to separate the tasks of detecting which features are active and estimating their magnitudes, allowing the sparsity penalty to be applied only to feature detection. Through experiments on language models up to 7B parameters, the authors show that Gated SAEs achieve better reconstruction fidelity for a given level of sparsity compared to baseline SAEs, while resolving issues like shrinkage. A human evaluation study finds Gated SAE features to be comparably interpretable to baseline features.\n\nSTRENGTHS:\n- A well-motivated architectural modification to SAEs that addresses key limitations\n- Comprehensive empirical evaluation across multiple model sizes and activation sites demonstrating clear improvements over baseline SAEs\n- Careful ablation studies and analysis to understand the source of improvements\n- Human evaluation study to assess interpretability of learned features\n- Thorough discussion of limitations and future work directions\n\nWEAKNESSES:\n- The presentation could be improved in some areas, particularly in explaining some of the technical details and metrics\n- Some of the figures are quite dense and could be made more readable\n- The human evaluation study, while valuable, has a relatively small sample size\n\nQUESTIONS:\n- Do you have any insights on how Gated SAEs might scale to even larger language models? Are there any potential limitations as model size increases?\n- Have you explored using Gated SAEs for any downstream mechanistic interpretability tasks beyond the basic reconstruction and interpretability metrics? For example, does the improved reconstruction enable better circuit analysis?\n- The weight tying scheme seems important for computational efficiency. Have you explored any alternative tying schemes? Is there a theoretical justification for why this particular scheme works well?\n\nLIMITATIONS:\nThe authors provide a good discussion of limitations in the conclusion section. They appropriately note that their experiments were limited to specific model families and that further work is needed to evaluate usefulness for downstream interpretability tasks. They also acknowledge the subjective nature of the human interpretability study.\nRegarding potential negative societal impacts, the authors do discuss this briefly in Appendix A. They note that advances in LM interpretability could potentially be misused, but argue that the current work poses minimal short-term risks. While this discussion is somewhat brief, it does address the key points and seems appropriate for the nature of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}