Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders
Authors: Senthooran Rajamanoharan
Arthur Conmy
Lewis Smith
Tom Lieberum
Vikrant Varma
Janos Kramar
Rohin Shah
Neel Nanda
Venue: NeurIPS 2024 poster (2024)
Keywords: Mechanistic Interpretability
Sparse Autoencoders
Science of Deep Learning
Primary area: interpretability_and_explainability
TL;DR: We introduce Gated SAEs, a new SAE architecture that is a Pareto improvement over baseline approaches for training SAEs on LM activations.

Abstract: Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of those activations. We introduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto improvement over training with prevailing methods. In SAEs, the L1 penalty used to encourage sparsity introduces many undesirable biases, such as shrinkage -- systematic underestimation of feature activations. The key insight of Gated SAEs is to separate the functionality of (a) determining which directions to use and (b) estimating the magnitudes of those directions: this enables us to apply the L1 penalty only to the former, limiting the scope of undesirable side effects. Through training SAEs on LMs of up to 7B parameters we find that, in typical hyper-parameter ranges, Gated SAEs solve shrinkage, are similarly interpretable, and require half as many firing features to achieve comparable reconstruction fidelity.

Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders
Senthooran Rajamanoharan" Google DeepMind
Arthur Conmy" Google DeepMind
Lewis Smith Google DeepMind
Tom Lieberum' Google DeepMind
Vikrant Varmat Google DeepMind
János Kramár Google DeepMind
Rohin Shah Google DeepMind
Neel Nanda Google DeepMind
Abstract
Recent work has found that sparse autoencoders (SAEs) are an effective technique for unsupervised discovery of interpretable features in language models' (LMs) activations, by finding sparse, linear reconstructions of those activations. We in- troduce the Gated Sparse Autoencoder (Gated SAE), which achieves a Pareto im- provement over training with prevailing methods. In SAEs, the LI penalty used to encourage sparsity introduces many undesirable biases, such as shrinkage - syss tematic underestimation of feature activations. The key insight of Gated SAEs is to separate the functionality (a) determining which directions to use and (b) esti- mating the magnitudes of those directions: this enables us apply the L1 penalty only to the former, limiting the scope of undesirable side effects. Through train- ing SAEs LMs of up 7B parameters find that, in typical hyper-parametts parameter ranges, Gated SAEs solve shrinkage, are similarly interpretable, and require half many firing features achieve comparable reconstruction fidelity.
Introduction
Mechanistic interpretability aims to explain how neural networks produce outputs in terms of the learned algorithms executed during a forward pass [33, 34]. Much work makes use of the fact that many concept representations appear to be linear [14, 19, 34, 391, i.e. that they correspond to interpretable directions activation space. However, finding the set of all interpretable directions is highly non-trivial problem. Classic approaches, like interpreting neurons (i.e. directions in the standard basis) are insufficient, as many are polysemantic and tend to activate for range of different seemingly unrelated concepts [7, 15, 16). Within the field, there has recently been much interest [8, 11, 21, 22, 4] using sparse autoencoders (SAEs; [32]) as an unsupervised method for finding causally relevant, and ideally interpretable, directions in a language model's activations.
Although SAEs show promise in this regard [26, 31]. the LI penalty used in the prevailing training method to encourage sparsity also introduces biases that harm the accuracy of SAE reconstructions, as the loss can be decreased by trading-off some reconstruction accuracy for lower L.1. In this pa- per, we introduce a modification to the baseline SAE architecture Gated SAE along with an accompanying loss function, which partially overcomes these limitations. Our key insight is to use separate affine transformations for (a) determining which dictionary elements use in reconstruc- tion and (b) estimating the coefficients active elements, and to apply the sparsity penalty only to the former task. We share subset of weights between these transformations to avoid significantly
Joint contribution. Core infrastructure contributor. Correspondence: sraj maajaannohhaannggggle.coom neel
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Resicuull
Attention output pre-lineer
Loss Recovered
0.9
(Fidellay)
0.0
SAE Type
0.7
Baseline
1000
(Lower sparser)
Figure 1: Gated SAEs consistently offer improved reconstruction fidelity for a given level of sparsity compared to prevailing (baseline) approaches. These plots compare Gated SAEs to baseline SAEs Layer 20 in Gemma-7B. Gated SAEs' dictionaries are of size 2 131k whereas baseline dictionaries are 50% larger, so that both types are trained with equal compute. This performance improvement holds in layers throughout GELU-IL, Pythia-2.88 and Gemma- (see Appendix E).
increasing the parameter count and inference-time compute requirements of a Gated SAE compared to baseline SAE of equivalent width.
We evaluate Gated SAEs multiple models: Pythia-2.8B [3] and Gemma-7B [18], and on multiple sites within models: MLP layer outputs, attention layer outputs, and residual stream activations. Across these models and sites, find Gated SAES to be Pareto improvement over baseline SAEs holding training compute fixed (Fig. 1): they yield sparser decompositions any desired level of reconstruction fidelity. We also conduct further follow up ablations and investigations on a subset of these models and sites to better understand the differences between Gated SABs and baseline SAEs.
layer GELU activation language model [28],
Overall, the key contributions of this work are that we:
1. Introduce the Gated SAE, modification to the standard SAE architecture that decouples detection of which features are present from estimating their magnitudes (Section 3.2);
2. Show that Gated SAEs Pareto improve the sparsity and reconstruction fidelity trade-off compared to baseline SAES (Section 4. 4.1);
3. Confirm that Gated SAES overcome shrinkage while outperforming other methods that also address this problem (Section 5.2);
4. Provide evidence from small double-blind study that Gated SAE features are comparably interpretable to baseline SAE features (Section 4.2).
2 Preliminaries
In this section we summarise the concepts and notation necessary to understand existing SAE ar- chitectures and training methods following Bricken al. [8], which we call the baseline SAE. We define Gated SAEs in Section 3.2.
As motivated in Section we wish decompose model's activation R into a sparse, linear combination of feature directions:
X Xo + Σ fx)dd,
where d; are dictionary of M >> n latent unit-norm feature directions, and the sparse coefficients fx(x) 2 0 are the corresponding feature activations for x.3 The right-hand side of Eq. (1) natu- rally has the structure of an autoencoder: an input activation activations vector f(x) E RM which in turn is linearly decoded to reconstruct x.
encoded into (sparse) feature
AAtthough due to an auxiliary loss term, computing the Gated SAE loss for training purposes does re- quire 50% more compute than computing the loss for matched- width baseline SAE.
'In this work, we use the term feature to refer only to the learned features of SAES, i.e. the overcomplete basis directions that are linearly combined to produce reconstructions. In particular, learned features are always linear and necessarily interpretable.

---

Baseline architecture Using this correspondence, Bricken et al. [8] and subsequent works attempt learn suitable sparse decomposition by parameterizing a single-layer autoencoder (f, x) defined by:
f(x) := ReLU (W ene (xoe + bese)
(2)
x(f) W dee + baec
(3)
and training it using gradient descent reconstruct samples x DD from a large dataset D of acti- vations collected from a single site and layer of a trained language model, constraining the hidden representation f to be sparse. Once the sparse autoencoder has been trained, we obtain a decompo- sition of the form of Eq. (1) by identifying the (suitably normalised) columns of the decoder weight matrix W dec E F M with the dictionary of feature directions d,, the decoder bias bdee E RR with the centering term Xo, and the (suitably normalised) entries of the latent representation f f(x) E RM with the feature activations (x).
Baseline training methodology To train sparse autoencoders, Bricken et al. [8] use a loss function with two terms that respectively encourage faithful reconstruction and sparsity:
[(x) == ||x - xf((x)) + + [|f(x)]l1
(4)
Since it is possible to arbitrarily reduce the L1 sparsity loss term without affecting reconstructions sparsity by simply scaling down encoder outputs and scaling up the norm of the decoder weights, it is important to constrain the norms the columns of W dec during training. Following Bricken et al. [8], we constrain norms to one. See Appendix G for further details on SAE training.
Evaluating SAEs
Two metries are primarily used to get a sense of SAE quality [8]: LO, a measure of SAE sparsity, and loss recovered, measure of SAE reconstruction fidelity. LO measures the average number of features used by a SAE to reconstruct input activations. Loss recovered is a normalised measure of the increase induced in LM's ss cross entropy loss when we replace its original activations with the corresponding SAE reconstructions during the model's forward pass. Both these metrics are formally defined in Appendix B, where we also discuss shortcomings of and alternatives to the loss recovered metric as it is defined in Bricken et al. [8]. Since it is possible for SAEs to score well on these metrics and still fail to be useful for interpretabillty-relate tasks [47], we perform manual analysis SAE interpretability in Section 4.2.
3
Gated SAES
3.1 Motivation
The intention behind how SAEs are trained is to maximise reconstruction fidelity at given level of sparsity, measured by LO, although in practice optimize mixture of reconstruction fidelity and LI regularization. This difference is a source of unwanted bias in the training of sparse autoencoder: for any fixed level of sparsity, trained SAE can achieve lower loss (as defined in Eq. (4)) by trading off a little reconstruction fidelity to perform better on the LI sparsity penalty.
The clearest consequence of this bias is shrinkage [51]. Holding the decoder x(() fixed, the L1 penalty pushes feature activations f(x) towards zero, while the reconstruction loss pushes f(x) high enough to produce accurate reconstruction. Thus, the optimal value falls somewhere in between, and as a result the SAE systematically underestimates the magnitude of feature activations, without necessarily providing any compensatory benefit for sparsity.
How can we reduce the bias introduced by the LI penalty? The output of the encoder f(x) of a baseline SAE (Section 2) has two roles:
1. It detects which features are active (according to whether the outputs are zero or strictly positive). For this role, the L1 penalty is necessary to ensure the decomposition is sparse.
'Note that we cannot directly optimize the L. norm (i.e. the number of active features) since this is not a differentiable function. We do however use the LU norm to evaluate SAE sparsity.
$ 'Conversely, rescaling the shrunk feature activations [51] is not necessarily enough to overcome the bias induced by by LI penalty: a SAE trained with the LI penalty could have learnt sub-optimal encoder and decoder directions that are not improved by such a fix. In Section 5.2 and Fig. 7 we provide empirical evidence that this is true in practice.
3

---

Magnitude Path
Encoder
+ +Dog scale shift
ReLU
Decoder
X
W enc
W dec
X
Gating Path
+bggne
shift
binarize
Figure 2: The Gated SAE architecture with weight sharing between the gating and magnitude paths, shown with example input. See Appendix for pseudo-code implementation.
2. It estimates the magnitudes of active features. For this role, the LI penalty is source of unwanted bias.
If could separate out these two functions of the SAE encoder, could design training loss that narrows down the scope of SAE parameters that are affected (and therefore to some extent biased) by the LI sparsity penalty to precisely those parameters that involved in feature detection, minimising its impact on parameters used in feature magnitude estimation.
3.2 Gated SAES
Architecture How should we modify the baseline SAE encoder to achieve this separation of con- cerns? Our solution is replace the single-layer ReLU encoder of bascline SAE with gated ReLU encoder. Taking inspiration from Gated Linear Units [43, 12], we define the gated encoder as
Tan (x)
f(x) = 1[ (W gate x b
Dgate
0 ReLU(W mag (x báee) + bmag)
(xae(x)
(xx(()
where 1 > 0] is the (pointwise) Heaviside step function and @ denotes elementwise multiplica- tion. Here, fgate determines which features are deemed be active, while fmug estimates feature activation magnitudes (which only matter for features that have been deemed to be active); gx)) the faate sub-layer's ae-aacivvaions, which are used in the gated SAE loss, defined below.
Training naive guess at a loss function for training Gated SAES would be to replace the sparsity penalty in Eq. (4) with the LI norm of (x). Unfortunately, due to the Heaviside step activation function in fgates no gradients would propagate to gute and b To mitigate this, we instead apply the LI norm to the positive parts of the preactivation, ReLU (IT gate (x)). To ensure fgate aids recon- struction by detecting active features, we add an auxiliary task requiring that these same rectified preactivations can be used by the decoder to produce good reconstruction:
x f((x )13 + ||RLLU T gate (x)) + ||x Xtrozen (ReLU (TT gate (x)))1I
Cganod(x) = x
(6)
Euscasstioss
Copurity
where Xirezen is frozen copy the decoder, X W cops dec + copy to ensure that gradients from Laux do not propagate back to Wdes bdce This can be implemented by stop gradient operations rather than creating copies. See Appendix for pseudo-code for the forward pass and loss function.
To calculate this loss (or its gradient), we have to run the decoder twice: once to perform the main reconstruction for Eresenstueet and once to perform the auxiliary reconstruction for Lax. This leads to 50% increase in overall training time is typically much less, as in our experience much of the training wall clock time goes to generating language model activations (if these are being generated on the fly) or disk I/O (if training saved activations).
compute required to perform a training update step. However, the increase

---

Parameter reduction through weight-tying Naively, we appear to have doubled the number of parameters the encoder, increasing the total number of parameters by 50% with respect to baseline SAEs. We mitigate this through weight sharing: we parameterize these layers so that the two layers share the same projection directions, but allow the norms of these directions as well as the layer biases to differ. Concretely, we define W mag in terms of W
gate and an additional vector-valued
rescaling parameter Fmag E IR follows:
(W (Wmng)s :- (exp(rmag)), (W gate)
See Fig. 2 for an illustration of the tied-weight Gated SAEs architecture. With this weight tying scheme, the Gated SAE has only 2 x M more parameters than baseline SAE. In Section 5.1, show that this weight tying scheme does not harm performance.
With tied weights, the gated encoder can be reinterpreted as a single-layer linear encoder with a non- standard and discontinuous "JumpReLU" activation function [17], (0(2), illustrated Fig. 12. To be precise, using the weight tying scheme of Eq. (7), f(x) can be re-expressed as f(x) = a V mag" b mmag), with the JumpRel.U gap given by 0 = Bmag e.tme ( b gate: see Appendix H for an explanation. We think this is a useful intuition for reasoning about how Gated SAEs reconstruct activations in practice.
4 Evaluating Gated SAEs
In this section we benchmark Gated SAEs against baseline SAEs across large variety of models and at different sites. We show that they produce more faithful reconstructions at equal sparsity and that they resolve shrinkage. Through a double-blind manual interpretability study, we find that Gated SAEs produce features that are similarly interpretable to baseline SAE features.
4.1 Benchmarking Gated SAEs
Methodology We trained suite of Gated and baseline SAEs, family of each type to reconstruct each of the following activations:
1. The MIP neuron activations in GELU-1I,, which is the closest direct comparison to Bricken et al. [8];
2. The MLP outputs, attention layer outputs (taken pre-Wo [21]) and residual stream activa- tions in 5 different layers throughout Pythia-2.8B and four different layers in the Gemma- 7B base model.
For each model and reconstruction site, we trained multiple SAEs using different values of X (and therefore LO), allowing us to compare the Pareto frontiers of LO and loss recovered between Gated and baseline SAEs. We also use the relative reconstruction bias metric, 7, defined Appendix C to measure shrinkage in our trained SAFs. This metric measures the relative bias in the norm of an SAE's reconstructions; unbiased SAEs obtain y = 1, whereas SAFs affected by shrinkage (which causes reconstruction norms to be systematically too small) have 7 1.
Since Gated SAES require most 1.5x more compute to train than regular SAES (Section 3.2) of the same width, we compare Gated SAES to baseline SAEs that have a 50% larger dictionary (hidden dimension M) to ensure fair comparison in our evaluations.
Results We plot sparsity against reconstruction fidelity for SAEs with different values of X. Higher À corresponds to increased sparsity and worse reconstruction, so in Bricken et al. [8] observe a Pareto frontier of possible trade-offs. We plot Pareto curves for GELU-1L in Fig. 3a and Pythia- 2.8B and Gemma-7B in Appendix E. At all sites tested, Gated SAES are Pareto improvement regular SAEs: they provide better reconstruction fidelity at any fixed level of sparsity.? For some sites in Pythia-2.8B and Gemma-7B, loss recovered does not monotonically increase with LO; we attribute this to difficulties training SAES (Appendix G.1.3). Finally, full tables of results for Pythia and Gemma can be found in Appendix K.
"Since wider SAEs provide better reconstructions (all else being equal), the gap between Gated SAEs' and baseline SAEs' performance is even wider when we use baseline SAEs with equal width comparison. This can be seen in the difference between the "1.5x width" and "equal width" baseline curves in Fig. 5. Although both Gated and baseline SAEs have loss recovered tending one for high enough LO.
5

---

1.1
Loss Recovered (Fidelity)
0.98
Relative reconstruction blas
(p<1 indicates shrinkage)
1.05
0.96
0.94
0.95
0.92
0.9
0.9
Type
Baseline (1.5x width)
0.85
SAE Type
Baseline width)
Gated
Gated
0.88
0.8
50
100
(Lower is sparser)
LO (Lower is sparser)
(b)
Figure 3: (a) Gated SAEs offer better reconstruction fidelity (as measured by loss recovered) at any given level of feature sparsity (as measured by LO); (b) Gated SAEs address shrinkage. These plots compare Gated and baseline SAEs trained on GELU-IL neuron activations; see Appendix E for comparisons on Pythia-2.8B and Gemma-7B.
As shown in Fig. 3b, Gated SAEs' reconstructions are unbiased, with y 1, whereas baseline SAEs exhibit shrinkage (y < 1), with the impact of shrinkage getting worse as the L1 coefficient À increases (and LO consequently decreases). Fig. 10 shows that this result generalizes to Pythia- -.BB..
4.2 Interpretability
Although Gated SAEs provide more faithful reconstructions than baselines at equal sparsity, it does not necessarily follow that these reconstructions are better suited to downstream interpretability- related tasks. Currently, there is consensus on how systematically assess the degree which SAE's features are useful for downstream tasks, but plausible proxy is to assess the extent to which these features are human interpretable [8]. Therefore, to gain more qualitative understanding of the differences between their learned features, we conduct a blinded human study in which we rate and compare the interpretability of randomly sampled Gated and baseline SAE features.
Methodology We study a variety of SABs from different layers and sites. For Pythia-2.8B had 5 raters, who each rated one feature from baseline and Gated SAEs trained on each (site, layer) pair from Fig. 8, for total of 150 features. For Gemma-7B we had raters; one rated 2 features each, and the rest 1 feature each, from baseline or Gated SAES trained on each (site, layer) pair from Fig. for a total of 192 features.
For each model, raters are shown the features in random order, without revealing which SAE, site, or layer they came from. To assess feature, the rater decides whether there is explanation of the feature's 's behavior, in particular for its highest activating examples. The rater then enters that explanation (if applicable) and selects whether the feature interpretable ("Yes') ), uninterpretable ('No') or maybe interpretable ('Maybe"). ) All raters are either authors of this paper or colleagues, who have prior experience interpreting SAE features. As interface we open source SAE visualizer library [27]; representative screenshots of the dashboards produced by this library are shown in Fig. 14.
Results & analysis Fig. 4 shows interpretability rating distributions by SAE type and LM, marginalising over layers, sites and raters. To compare the interpretability of baseline and Gated SAEs, first pair datapoints according to all covariates (model, layer, site, rater); this lets us
"Athhough due debugging issue, Gemma-7B attention SAEs were rated separately, raters were blind to that.
995% error bars were obtained by modelling each frequency shown as binomial, with set to the sample frequency, and calculating the 2.5% and 97.5% quantiles.

---

Mydel=Pythia-2.988
Model= Gemma-7B
SAE Type
0.8
Baseline
Frequency
Gated
0.6
0.4
0
No
Maybe
Yes
No
Maybe
Yes
Feature interpretable
Feature interpretable
Figure 4: Proportions of SAE features rated as interpretable uninterpretable maybe interpretable by SAE type (Gated or baseline) and language model. Gated and baseline SAEs are similarly inter- pretable, with mean difference (in favor of Gated SAEs) of 0.13 (95% CI [0, 0.26) after aggregat- ing ratings for both models.
control for all of them without making any parametric assumptions, and thus reduces variance in the comparison. We then measure the mean difference between baseline and Gated labels, where we count 'No' as 0, 'Maybe' as 1, and 'Yes' as 2. and compute a 90% BCa bootstrap confidence interval. Thus we find that the mean difference in label scores is 0.13 (90% CI [0, 0.26)) in favour of Gated SAFs, breaking down to mean difference Cls of [-0.07, 0.33 and [-0.04, 0.29) on just the 2.88 data and Gemma-78 7B data respectively. Since central estimate for the mean differ- ence in scores is positive, we also test the hypothesis that Gated SAES may be more interpretable than baseline SAEs. However, one-sided Wilcoxon-Prail signed-rank test on the paired scores does not reject the null hypothesis that they are equally interpretable (p 0.06). The contingency tables used for these results are shown in Fig. 13. The overall conclusion is that Gated SAE features are similarly interpretable to baseline SAE features, while also providing better reconstruction fidelity (at fixed sparsity), as shown in the previous section. We provide more analysis of how these break down by site and layer in Appendix I.
Why do Gated SAEs improve SAE training?
5.1
Ablation study
In this section, vary several parts of the Gated SAE training methodology to gain insight into which aspects of the training drive the observed improvement in performance. Gated SAEs differ from baseline SAEs in many respects, making it easy to incorrectly attribute the performance gains to spurious details without a careful ablation study. Fig. 5a shows Pareto frontiers for these variations; below we describe each variation in turn and discuss our interpretation of the results.
Here we unfreeze the decoder weights in Laux - i.e. allow this auxiliary task to update the decoder weights in addition to training fgnees 's parameters. Although this (slightly) simplifies the loss, there a reduction in performance, suggesting that it beneficial to limit the impact of the I.1 sparsity penalty to just those parameters in the SAE that need it it-iie i.e. those used to detect which features are active.
Unfreeze decoder:
No T'mag: Here we remove the Tmag scaling parameter in Eq. (7), effectively setting it to zero, further tying fguee 's and Emag's 's parameters together. With this change, the two encoder sublayers' preactiva- tions can at most differ by an elementwise shift. There is slight drop in performance, suggesting Tmag contributes somewhat to the improved performance of the Gated SAE.
Untied encoders: Here we check whether choice share the majority of parameters between the two encoders has meaningfully hurt performance, by training Gated SAEs with gating and ReLU encoder parameters completely untied. Despite the greater expressive power of an untied encoder, we see no improvement performance fact a slight deterioration. This suggests our tying scheme (Eq. (7)) where encoder directions are shared, but magnitudes and biases aren't is effec-
Because the biases bgate and bmag still differ.
7

---

Loss Recovered (Fidelity)
0.98
Loss Recovered (Fidelity)
0.98
0.96
SAE Type
Baseline (1.5x width)
0.92
Gated
SAE Type
Ablation: unfreeze decoder
Baseline (equal width)
Ablation: untie encoder layers
Gated
Ablation: no r_mag
Baseline + rescale & shift
0.88
50
100
50
100
LO (Lower is sparser)
(Lower is sparser)
(a)
(b)
Figure 5: (a) Our ablation study on GELU-IL MLP neuron activations indicates: (i) the impor- tance of freezing the decoder in the auxiliary task Laux used to train fgate 's parameters; (ii) tying encoder weights according to Eq. (7) is slightly beneficial for performance (in addition to yield- ing significant reduction in parameter count and inference compute); (iii) further simplifying the encoder weight tying scheme in Eq. (7) by removing Fmag is mildly harmful to performance. (b) Evidence from GELU-1L that the performance improvement of gated SAEs does not solely arise from addressing shrinkage (systematic underestimation of latent feature activations): taking a frozen baseline SAE's parameters and learning mmag and bmag parameters on top of them (green line) does successfully resolve shrinkage, by decoupling feature magnitude estimation from active feature de- tection; however, it explains only a small part of the performance increase of gated SAEs (red line) over baseline SAEs (blue line).
tive at capturing the advantages of using gated SAE while avoiding the 50% increase in parameter count and inference-time compute of using an untied SAE.
5.2 Is it sufficient just address shrinkage?
As explained in Section 3.1, SAFS trained with the baseline architecture and L1 loss systematically underestimate the magnitudes of latent features' activations (i.e. shrinkage). Gated SAEs, through modifications to their architecture and loss function, overcome these limitations.
It is natural to ask what extent the performance improvement of Gated SAEs is solely attributable to addressing shrinkage. Although addressing shrinkage would - all else staying equal improve reconstruction fidelity, it is not the only way to improve SAEs' performance: for example, Gated SAES could also improve upon baseline SAES by learning better encoder directions (for estimating when features active and their magnitudes) or by learning better decoder directions (i.e. better dictionaries for reconstructing activations).
Here try to answer this question by comparing Gated SAES trained as described in Section 3.2 with an alternative (architecturally equivalent) approach that also addresses shrinkage, but in way that uses frozen encoder and decoder directions from baseline SAE of equal dictionary size.'! Any performance improvement over baseline SAEs obtained by this alternative approach (which we dub "baseline + rescale & shift") can only be due to better estimations of active feature magnitudes, since by construction an SAE parameterized by "baseline + rescale & shift" shares the same encoder and decoder directions
baseline SAE.
As shown in Fig. 5b, although resolving shrinkage only ("baseline rescale & shift") does improve- ment baseline SAEs' performance a little, significant gap remains with respect to the performance of Gated SAES. This suggests that the benefit the gated architecture and loss comes from learning better encoder and decoder directions, not just from overcoming shrinkage. In Appendix D we ex-
"Concretely, we do this by training baseline SAES, freezing their weights, and then learning additional rescale and shift parameters (similar to Wright and Sharkey [51]) to be applied to the (frozen) encoder pre- activations before estimating feature magnitudes.
8

---

plore further how Gated and baseline SAEs' decoders differ by replacing their respective encoders with an optimization algorithm inference time.
Related work
Mechanistic interpretability Recent work in mechanistic interpretability has found recurring components in small and large LMs [38], identified computational subgraphs that carry out spe- cific tasks in small LMs (circuits; [50]) and reverse-engineered how toy tasks are carried out in small transformers [30]. A central difficulty in this kind of work is choosing the right units of analy- sis. Sparse linear features have been identified as a promising candidate in prior work [52, 46]. The superposition hypothesis outlined by Elhage et al. [16] also provided a theoretical basis for this the- ory, sparking new interest in using SAEs specifically to learn feature basis [42, 8, 11, 21, 22, 4], as well as using SAES directly for circuit analysis [26]. Other work has drawn awareness to issues or drawbacks with SAE training for this purpose, some of which our paper mitigates. Wright and Sharkey [51] raised awareness of shrinkage and proposed addressing this via fine-tuning. Gated SAEs, as discussed, resolve shrinkage during training. [35, 47, 2, 36] have also proposed general SAE training methodology improvements, which are mostly orthogonal to the architectural changes discussed in this work. In parallel work, Taggart [45] finds early improvements using Jump ReLU [17], but with different loss function, and without addressing the problems of the L1 penalty.
Classical dictionary learning Research into the general problem of sparse dictionary learning precedes transformers, and even deep learning. For example, sparse coding [13] studies how dis- crete and continuous representations can involve more representations than basis vectors, and sparse representations are also studied in neuroscience [48, 37J. One dictionary learning algorithm, k-SVD [1] also uses two stages to learn a dictionary like Gated SAEs. Although classical dictionary learn- ing algorithms can be more powerful than SAFs (Appendix D), they are less suited for downstream uses like weights-basee circuit analysis or attribution patching [44, 24), because they typically use an iterative algorithm to decompose activations, whereas SAEs make feature extraction explicit via the encoder. Bricken et al. [8] have also argued that classical algorithms may be 'too strong', , in the sense they may learn features the LM itself could not access, whereas SAEs uses components similar to a LM's MLP layer to decompose activations.
7 Conclusion
In this work we introduced Gated SAEs which are a Pareto improvement in terms of reconstruc- tion quality and sparsity compared to baseline SABS (Section 1), and are comparably interpretable (Section 4.2). We showed via an ablation study that every key part of the Gated SAE methodology was necessary for strong performance (Section 5.1). This represents significant progress on improv- ing Dictionary Learning on LMs at many sites, Gated SAES require half the LO to achieve the same loss recovered (Fig. 8). This is likely to improve work that uses SAES to steer language models [31], interpret circuits [26], or understand LM components across the full distribution [8].
Limitations & future work. Our benchmarking study focused on GELU-IL and models in the Pythia and Gemma families. It is therefore not certain that these results will generalise other model families. On the other hand, the theoretical underpinnings of the Gated SAE architecture (Section 3) make assumptions about LM architecture, suggesting Gated SAES should be Pareto improvement more generally. While we have confirmed that Gated SAR features are comparably in- terpretable to baseline SAE features, does not necessarily follow that Gated SAE decompositions are equally usefuul for mechanistic interpretability. It is certainly possible that human interpretability of SAE features is only weakly correlated with either: (i) identification of the causally meaningful directions in a LM's 's activations; or (ii) usefulness on downstream tasks like circuit analysis or steer- ing. framework for scalably and objectively evaluating the usefulness of SAE decompositions (gated or otherwise) is still in its early stages [25] and further progress in this area would be highly valuable. is plausible that some of the performance gap between Gated and baseline SAES could be closed by inexpensive inference-time interventions that prune the many low activating features that tend to appear baseline SAEs, mimicking Gated SAEs' thresholding mechanism. Finally, we would be most excited to see progress on using dictionary learning techniques to further inter-

---

pretability in general, such as to improve circuit finding [10, 26] or steering [49] in language models, and hope that Gated SAEs can serve accelerate such work.
Acknowledgements
We would like to thank Romeo Valentin for conversations that got us thinking about k-SVD in the context of SAEs, which inspired part of our work. Additionally, we are grateful for Vladimir Mikulik's detailed feedback on a draft of this work which greatly improved our presentation, and Nicholas Sonnerat's work thank Glen Taggart who found in parallel work [45] that a similar method gave improvements to SAE training, helping give more confidence in our results. We are grateful to Sam Marks for pointing out an error in the derivation of relative reconstruction bias in an earlier version of this paper and Leo Gao and Gonçalo Paulo for discussions. Finally, thank anonymous reviewers for their valuable feedback, which helped improve the quality of this paper.
codebase and help with feature labelling. We would also like to
Author contributions
Senthooran Rajamanoharan developed the Gated SAE architecture and training methodology, in- spired by discussions with Lewis Smith on the topic of shrinkage. Arthur Conmy and Senthooran Rajamanoharan performed the mainline experiments in Section 4 and Section 5 and led the writing of all sections of the paper. Tom Lieberum implemented the manual interpretability study of Sec- tion 4.2, which was designed and analysed by János Kramár. Tom Lieberum also created Fig. 2 and Lewis Smith contributed Appendix Our SAE codebase was designed by Vikrant Varma who implemented it with Tom Lieberum, and was scaled to Gemma by Arthur Conmy, with contribu- tions from Senthooran Rajamanoharan and Lewis Smith. János Kramár built most our underlying interpretability infrastructure. Rohin Shah and Neel Nanda edited the manuscript and provided lead- ership and advice throughout the project.
References
[1] M. Aharon, M. Elad, and A. Bruckstein. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions Signal Processing, 54(11):4311- 4322, 2006. doi: 10. 0..1109/SPP2200.8811999
[2] Batson, B. Chen, A. Jones, A. Templeton, T. Conerly, J. Marcus, T. Henighan, N. L. Turner, and A. Pearce. Circuits Updates March 2024. Transformer Circuits Thread, 2024. URL wtt.uu/iis.uupttteeee.eeeeeee 2024/mar--pdate//nneee htmml.
[3] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023. Apache License 2.0.
[4] J. Bloom. Open Source Sparse Autoencoders for all Residual Stream Layers of GPT- 2 Small, 2024.
[5] Bloom. SAELens, 2024. hutp://giinh..comm/boommn//////
[6] T. Blumensath and M. E. Davies. Gradient pursuits. IEEE Transactions on Signal Processing, 56(6):2370-2822 2008
[7] T. Bolukbasi, A. Pearce, A. Yuan, A. Coenen, E. Reif, F. Viégas, and M. Wattenberg. An interpretability illusion for bert. arXiv preprint 1rXiv:2104.077 143, 2021.
[8] T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil,
C. Denison, A. Askell, R. Lasenby, Y. Wu, S. Kravee, N. Schiefer, T. Maxwell, N. Joseph,
7. Hatfield-Dodae A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter,
10

---

[21] C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. toencoders work on attention layer outputs. Alignment wtw.://imneettttooooooo////ooode T. Henighan, and C. Olah. Towards monosemanticity: Decomposing language mod- els with dictionary learning. Transformer Circuits Thread, 2023. https:/ttransfommer http://00/mmmomomaomaiiiiiieeeee
htmll.
[9] A. Conmy. Dec 2023.
My best guess at the important tricks for training IL SAEs, wwtps//www..oooopptssts/eeeeenes important-tricks- wratttrrcccc-fo-rrininnnn I--ses.
[10] A. Conmy, A. N. Mavor-Parker, A. Lynch, S. Heimersheim, and A. Garriga-Alonso, Towards automated circuit discovery for mechanistic interpretability, 2023.
[11] H. Cunningham, A. Ewart, L. Riggs, R. Huben. and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023.
[12] Y. Dauphin, A. Fan, M. Auli, and D. Grangier. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'11, page 933-941. JorR. 2017.
[13] M. Elad. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing- Springer, New York, 2010. ISBN 978- 978-1-4419-7010-7, doi: 10.1007/ 978-1-4419-7011
[14] N. Elhage, N. Nanda. C. Olsson, T. Henighan, N. Joseph, B. Mann. A. Askell, Y. Bai, A. Chen, Conerly, N. DasSarma, Drain, Ganguli, Hatfield-oodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL https http://taaafffffmeeeeeeeee circuitss
html.
[15] N. Elhage, T. Hume, Olsson, N. Nanda, T. Henighan, Johnston, S. ElShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodds, J. Kernion, Conerly, S. Kravec, S. Fort, S. Kadavath, J. Jacobson, E. Tran-Johnson, J. Kaplan, J. Clark, T. Brown, S. McCan- dlish, D. Amodei, and C. Olah, Sofumax linear units. Transformer Circuits Thread, 2022.
https//transoomee-crruiti hwt.s//sp//222////nneeee...tm.
[16] N. Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, Kravec, Z. Hatfield-Dodds. R. Lasenby, D. Drain, C. Chen, et al. Toy Models of Superposition. arXiv preprint arXiv:2209.10653 2022.
[17] N. B. Erichson, Z. Yao, and M. W. Mahoney. Jumprelu: A retrofit defense strategy for adver- sarial attacks, 2019.
[18] Gemma Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, L. Sifre, M. Rivière, M. S. Kale, J. Love, Tafti, L.. Hussenot, and al. Gemma, 2024. URL https: http://wwwwkkkgee... com/m/330 Apache License 2.0.
[19] W. Gurnee, Nanda, M. Pauly, K. Harvey, D. Troitskii, and Bertsimas. Finding neurons in haystack: Case studies with sparse probing, 2023.
[20] N. P. Jouppi, D. H. Yoon, Kurian, S. Li, N. Patil, J. Laudon, C. Young, and D. Patterson. A domain-specific supercomputer for training deep neural networks. Communications of the ACM, 63(7):67-78, 2020.
Sparse au- Forum, 2024.
[22] C. Kissane, R. Krzyzanowski, A. Conmy, and N. Nanda. Attention SAES scale to GPT-2 Small. Alignment Forum, 2024.
[23] K. Konda, R. Memisevie, and D. Krueger. Zero-bias autoencoders and the benefits of co- adapting features, 2015. https: http:///wwwvvvvooaa//////sse/ 3337.

---

[24] J. Kramár, T. Lieberum, R. Shah, and N. Nanda. Atp*: An efficient and scalable method for localizing Ilm behaviour to components. arXiv preprint arXiv:2403.00745, 2024.
[25] A. Makelov, G. Lange, and N. Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL
[26] S. Marks, C. Rager, E. J. Michaud, Y. Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2024.
[27] C. M.Doouglll SAE Visualizer, 2024, http://gghhbbbcmmmaaallmmooooga.
[28] N. Nanda. GELU-IL, 2022. URL hwt.l//ppp.///nnggaoa...ece////e C4_Code. MIT License.
[29] N. Nanda. Open Source Replication & Commentary on Anthropic's Dictionary Learning Pa- per, Oct 2023.
[30] N. Nanda, I.. Chan, T. Lieberum, J. Smith, and J. Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL httph:t////oop httttp::/oppnrerieew wtw.aarrrwiieeee/orurunnnm 1dd-9FFSDPPdd..
[31] N. Nanda, A. Conmy, L. Smith, S. Rajamanoharan, T. Lieberum, Kramár, and V. Varma. [Summary] Progress Update #1 from the GDM Mech Interp Team. Align- ment Forum, 2024. https//www.aiggmmeffouuum progress-update---foon ---from-th---d-m-mec-----e--- wwtp:///www.a...aaaaaaaaa/a/aa/a
[32] A.
Ng.
Sparse autoencoder, 2011.
CS294A
Lecture
notes,
[33] C. Olah. Mechanistic interpretability, variables, and the importance of interpretable bases, 2022. http::/www.fffeeeeeei.riiii////e
[34] C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduc- tion circuits. Distill, 2020. doi: 1/2311///iiitl.0000200
[35] C. Olah, S. Carter, A. Jermyn, J. Batson, T. Henighan, T. Conerly, J. Marcus, A. Templeton, B. Chen, and N. L. Turner. Circuits Updates January 2024. Transformer Circuits Thread, 2024. URL https: htt:://taaasffffrreeeeeooe circuitss htw.rcuisss..pb/2////ajaa. http://////aaaaa///e//eaaaaaaaee updaateeiddee html.
[36] C. Olah, S. Carter, A. Jermyn, J. Batson, T. Henighan, J. Lindsey, T. Conerly, A. Templeton, Marcus, and T. Bricken. Circuits Updates April 2024. Transformer Circuits Thread, 2024. URL https: httt://///a//ooeeee/eerrrr
ptciittssuuub//00 wwwpppp/ppppappppeee /aprillrlll aawrapppdaateeeeeee -udaate/ iinexx. html.
[37] B. A. Olshausen and D. J. Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision Research, 37(23):33113322 1997. doi: 10. (0.1016//0042--9899977 00169-7.
[38] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, Conerly, D. Drain, D. Ganguli, Z. HatfieldDodds, D. Hernandez, S. John- ston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. In-context learning and induction heads. Transformer Cir- cuits Thread, 2022. http::/transfommrrccccuuuss wttpp//////inooootttteeeeeeeeeee and -nduuiintttion- heads/index.html.
[39] K. Park, J. Choc, and V. Veitch. The linear representation hypothesis and the geometry of large language models, 2023.
[40] Y. Pati, R. Rezaiifar, and P. Krishnaprasad. Orthogonal matching pursuit: recursive function approximation with applications wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, pages 40-44 vol.1, 1993. doi: 10.1 1S9SSC.1993.424653
12

---

[41] S. Rajamanoharan, T. Lieberum, N. Sonnerat, A. Conmy, V. Varma, J. Kramár, and N. Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024.
URL. https: http:://axxvvvoviee.o.ee wtt.ooos//////////// 24445.
14435.
[42] L. Sharkey, D. Braun, and B. Millidge. taking features out of superposition with
finterim research report]
sparse autoencoders, 2022.
https//www.alignmentfouu.mm (caturrs-outtoo-sspppppitioo.
[43] N. Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https:
// //arxiv.orgp org/ absaabss22022 05202.
[44] A. Syed, C. Rager, and A. Conmy, Atrribution patching outperforms automated circuit discov- ery. arXiv preprint arXiv:2310. 10348, 2023.
[45] G. M. Taggart. http://www.lesswrong.ccc http:///////aaaaa/////////////// for-sparse-autoeeccder..
Prolu: A nonlinearity for sparse autoencoders, 2024.
[46] A. Tamkin, M. Taufeeque, and N. D. Goodman. Codebook features: Sparse and discrete interpretability for neural networks, 2023.
[47] A. Templeton, J. Batson, T. Henighan, T. Conerly, Marcus, A. Golubeva, T. Bricken, and A. Jermyn. Circuits Updates February 2024. Transformer Circuits Thread, 2024. html.
[48] S. J. Thorpe. Local vs. distributed coding. Intellectica, 8:3-40, 1989.
[49] A. M. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid. Activation addition: Steering language models without optimization, 2023.
[50] K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: http:///ppeeeeeeee////ee ///ppp/rrreev..nt/ffoomm nww./frruu id-NpaVSS66o4u.
[51] B. Wright and L. Sharkey. Addressing feature suppression in SARs, Feb 2024.
suppressioniin-saes
[52] Z. Yun, Y. Chen, Olshausen, and LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors, 2023.
13

---

Appendix
A Impact statement
This work introduces a method to obtain higher fidelity sparse decompositions of LM activations, under the hypothesis that progress in this area will ultimately help us understand the representations used by LMs. If successful, this could lead to greater understanding of how LMs complete tasks and novel mechanisms for controlling their behavior. Greater understanding and control could be put to beneficial uses such mitigating the harms caused by current and future models, although bad actors could also misuse these tools, for example to circumvent safety training and steer models towards harmful behaviors. Currently, the SAE research program is in its early stages. For any po- tential misuse of SAEs, there is typically a more practical and effective way to achieve the same end using existing tooling, e.g. fine tuning or activation editing. Therefore, we see negligible negative societal impact in the short term. Longer term, advances in LM interpretability and control pose similar benefits and risks advances in Al capabilities in general.
B Metrics for evaluating SAEs
SABs are expected to decompose input activations sparsely, and yet in manner that allows for faithful reconstruction. LO and loss recovered are two metrics typically used [8] to measure sparsity and reconstruction fidelity respectively. These are defined as follows:
The L0 of a SAE is defined by the average number of active features on a given input, i.c
Ex<D [fx))]lu-
The loss recovered
SAE is calculated from the average cross-entropy loss of the lan- guage model on evaluation dataset, when the SAE's reconstructions are spliced into it. If we denote by CE(②) the average loss of the language model when we splice a function d R" + I at the SAB's site during the model's forward pass, then loss recovered
CE( o f) CE(Id) CE(() CE(Id)
1
where o f is the autoencoder function, Ç X + x the identity function. According to this definition, a SAE that always outputs the zero vector as its reconstruction would get loss recovered of 0%, whereas SAE that reconstructs its inputs perfectly would get loss recovered of 100%.
0 the zero-ablation function and Id :
B.1 Issues with the loss recovered metric
In this paper, we have used loss recovered as defined in Bricken et al. [8] to measure reconstruction fidelity. However, there are deficiencies with this metric:
Firstly, zero-ablation is arguably too poor a bascline for defining the zero-point of this metric and mean-ablation better justified. Using the ablation function l x H> Ex'=xx' instead of in the definition of loss recovered above would also have the benefit that SAEs' loss recovered would tend towards zero the limit LO + 0. instead of tending to positive value it does when computing loss recovered using zero-ablation.
Furthermore, the very fact we normalise the increase in the spliced LM's loss when com- puting loss recovered makes it difficult to compare the impact of splicing SAES at different sub-layers of the model. For example, mean or zero-ablating the output MLP layer typically has much milder impact LM loss than mean zero-ablating the residual stream, making the denominator in Eq. (8) smaller for MLP SAES than for residual stream SAEs. So we unsurprisingly find that residual stream SAEs' loss recovered tend to be much higher than MLP or attention SAES. This suggests that it may be more informative to report raw changes in cross-entropy loss ("delta LM loss") instead of using normalised metric like loss recovered, since these are directly comparable across SAES trained on different sub-layers of the same L.M.
In practice however, both mean-ablated loss recovered and delta LM loss are related zero-ablated loss recovered (the metric used in this paper) by an affine transformation. In other words, all the
14

---

loss recovered versus LO figures in this paper would look identical if we had used one of these other metries instead, with the only difference being the tick labels on the y-axis. Consequently, none of the conclusions we draw in this paper would be affected by using one of these other reconstruction fidelity metries instead. Nevertheless, we draw the reader's attention to our subsequent work [41], which compares Gated SAEs to other SAE varieties adopting the delta LM loss metric, instead of loss recovered, for measuring reconstruction fidelity.
C Measuring shrinkage
As described in Section 3.1, the L1 sparsity penalty used to train baseline SAEs causes feature activations to be systematically underestimated, a phenomenon called shrinkage. Since this in turn shrinks the reconstructions produced by the SAE decoder, can observe the extent which trained SAE is affected by shrinkage by measuring the average norm of its reconstructions.
Concretely, the metric we use is the relative reconstruction bias,
7 arg min Ex ( [
(9)
i.c. y is the optimum multiplicative factor by which an SAE's reconstructions should be rescaled in order minimise the L2 reconstruction loss; y 1 for an unbiased SAE and ? when there's shrinkage. 13 Explicitly solving the optimization problem in Eq. (9), the relative reconstruction bias can be expressed analytically in terms of the mean SAE reconstruction loss, the mean squared norm of input activations and the mean squared norm of SAE reconstructions, making easy to compute and track during training:
Ex-D
&SSA (x)1l
2 ExnD
&SSA (x)]1
ExnD
&SAE (x)
x
Ex-D
|ASAE (x)112
EX-D
||x|
-ExDD |SSAB (x)
x
(10)
where the second equality makes use of the identity 2a b = -|a ||b|| - ||a bll Notice from the second expression for y that an unbiased reconstruction (7 1) therefore satisfies
E Ex-D | (x)112 Ex-D
[|x|||] - Ex-D
||ESAE (x) - x|2] -
In other words, an unbiased but imperfect SAE (i.e. one that has non-zero reconstruction loss) must have mean squared reconstruction norm that is strictly less than the mean squared norm of its inputs even without shrinkage. Shrinkage makes the mean squared reconstruction norm even smaller.
D Inference-time optimization
The task SAEs perform can be split into two sub-tasks: sparse coding, learning set of features from a dataset, and sparse approximation, where a given datapoint is approximated as a sparse linear combination of these features. The decoder weights the set of learned features, and the mapping represented by the encoder is a sparse approximation algorithm. Formally, sparse approximation is the problem of finding a vector that minimises;
= arg min ||x - s.l. ||x||o <>
(II)
i.e. that best reconstructs the signal x as a linear combination of vectors in a dictionary D, subject constraint the LO pseudo-norm on a. Sparse approximation is a well studied problem, and SAES are weak sparse approximation algorithm. SAES, at least in the formulation conventional in dictionary learning for language models, fact solve slightly more restricted version of this problem where the weights a on each feature are constrained to be non-negative, leading to the related problem
arg min ||x - Dael
2 s.t. ||al|o < <,,0 0
(12)
We also provide in Table cross entropy losses for the LMs used experiments, both with and without zero-ablation, which could principle be used to translate loss recovered results paper to delta LM loss. We have defined this way round that 7 < intuitively corresponds shrinkage.
15

---

In this paper, we do not explore using more powerful algorithms for sparse coding. This is partly because we are using SAEs not just to recover sparse reconstruction of activations of a LM; ideally we hope that the learned features will coincide with the linear representations actually used by the IMM, under the superposition hypothesis. Prior work [8] has argued that SAEs are more likely to recover these due to the correspondence between the SAE encoder and the structure of the network itself: the argument is that it is implausible that the network can make use of features which can only be recovered from the vector via an iterative optimisation algorithm, whereas the structure of the SAE means that it can only find features whose presence can be predicted well by a simple linear mapping. Whether this is true remains, in our view, an important question for future work, but do not address it in this paper.
In this section we discuss some results obtained by using the dictionaries learned via SAE training, but replacing the encoder with a different sparse approximation algorithm at inference time. This allows us to compare the dictionaries learned by different SAE training regimes independently of the quality of the encoder. also allows us to examine the gap between the sparse reconstruction performed by the encoder against the baseline of a more powerful sparse approximation algorithm. As mentioned, for a fair comparison to the task the encoder is trained for, is important to solve the sparse approximation problem of Eq. (12), rather than the more conventional formulation of Eq. (11), but most sparse approximation algorithms can be modified to solve this with relatively minor changes.
Solving Eq. (12) exactly is equivalent to integer linear programming, and is NP hard. The integer linear programs in question would be large, as our SAE decoders routinely have hundreds of thou- sands of features, and solving them to guaranteed optimality would likely be intractable. Instead, as is commonly done, we use iterative greedy algorithms to find an approximate solution. While the solution found by these sparse approximation algorithms is not guaranteed to be the global optimum, these are significantly more powerful than the SAE encoder, and we feel it is acceptable in practice to treat them as an upper bound on possible encoder performance.
For all results in this section, we use gradient pursuit, as described in Blumensath and Davies [6], as our inference time optimisation (ITO) algorithm. This algorithm is a variant of orthogonal matching pursuit [40] which solves the orgothonalisation of the residual to the span of chosen dictionary elements approximately at every step rather than exactly, but which only requires matrix multiplies rather than matrix solves and is easier to implement on accelerators as a result. It is possibly not crucial for performance that our optimisation algorithm be implementable on TPUs, but being able to avoid a host-device transfer when splicing this into the forward pass allowed us to re-use our existing evaluation pipeline with minimal changes.
When we use sparse approximation algorithm at test time, we simply use the decoder of a trained SAE as a dictionary, ignoring the encoder. This allows us to sweep the target sparsity at test time without retraining the model, meaning that we can plot an entire Pareto frontier of loss recovered against sparsity for single decoder, as in done in Fig. 7.
Fig. 6 compares the loss recovered when using ITO for a suite of SAES decoders trained with both methods three different test time LO thresholds. This graph shows a somewhat surprising result; while Gated SAES learn better decoders generally, and often achieve the best loss recovered us- ing ITO close to their training sparsity, SAE decoders are often outperformed by decoders which achieved a higher test time LO; it's better to do ITO with a target LO of 10 with an decoder with an achieved L0 of around 100 during training than one which was actually trained with this level of sparsity. For instance, the left hand panel in Fig. 6 shows that SAES with a training LO of 100 are better than those with an LO of around 10 at almost every sparsity level in terms of ITO reconstruc- tion. However, gated SAE dictionaries have a small but real advantage over standard SAEs in terms of loss recovered at most target sparsity levels, suggesting that part of the advantage of gated SAES is that they learn better dictionaries as well as addressing issues with shrinkage. However, there are some subtleties here; for example, we find that baseline SAES trained with a lower sparsity penalty (higher training LO) often outperform more sparse baseline SAES according to this measure, and the best performing baseline SAE (LO R 99) is comparable to the best performing Gated SAE (LO 2 20).
Fig. 7 compares the Pareto frontiers of a baseline model and a gated model to the Pareto frontier of an ITO sweep of the best performing dictionary of each. Note that, while the Parcto curve of the baseline dictionary is formed by several models as each encoder is specialised given sparsity
16

---

Type
2.00
2.30
2.00
2.40
1.80
1.02
382
SA2
552
SAE
Figure 6: This figure compares the ITO performance of different decoders across a sweep for de- coders trained using baseline SAE and the gated method, at three different test time target spar- sities. Gated SAEs trained at lower target sparsities consistently achieve better dictionaries by this measure. Interestingly, the best performing baseline dictionary by this measure often has much higher test time sparsity than the target; for instance, at a test time sparsity of 30, the best baseline SAE was the one that had a test time sparsity of more like 100. This could be an artifact of the fact that the LO measure is quite sensitive to noise, and standard SAE architectures tend to have reasonable number of features with very low activation.
0.66
gatee (ITO, (eceder)
044
1C0
LD
Pareto fronticrs of baseline SAE, a baseline SAE with learned rescale and shift (to account for shrinkage) and gated SAE across different sparsity lambdas, compared the ITO Pareto frontier of the best decoder of each type with ITO, varying the target sparsity. The best gated encoder is better than the best standard encoder by this measure, but the difference is marginal. As shown in the plot above, the best baseline encoder by the ITO measure had much larger test time sparsity (around 100) than the best gated model (around 30). This figure suggests that the gap between SAE performance and 'optimal' performance, we assume that ITO is close to the maximum possible reconstruction using the given encoder, is much smaller for the gated model.
Figure 7:
level, as mentioned, ITO lets us plot Pareto frontier by sweeping the target sparsity with a single dictionary; here we plot only the best performing dictionary from each model type avoid cluttering figure. This figure suggests that performance gap between smaller for the gated model. Interestingly, this cannot solely be explained by addressing shrinkage, as we demonstrate by experimenting with baseline model which learns rescale and shift with frozen encoder and decoder directions.
encoder and using ITO
17

---

Table 1: Cross-entropy losses for the original language model and after zero-ablating specified sub- layers of Pythia-2.888 and Gemma-7B.
Zero Ablation CE Loss
Model
Layer
Original CE Loss
MLP Attention
Residual
6
2.5426
2.7764
2.7295
16.1549
Gemma-7B
13
2.5426
2.5878
2.5566
30.3588
(1024 length context)
2.5426
2.6881
2.5726
19.5891
27
2.5426
26.1114
3.0819
12.4534
1,9699
2.0460
2.0361
13.0434
12
1.9699
2.0167
2.0131
10.6558
Pythia-2.8B
16
1.9699
2.0098
2.0046
11.6820
(2048 length context)
20
1.9699
2.1022
2.0269
10.4578
1.9699
2.0145
1.9760
27.8663
E More loss recovered LO Pareto frontiers
In Fig. 8 we show that Gated SAEs outperform baseline SAEs. In Fig. 9 we show that Gated SAEs ourperform baseline SAEs at all but one MLP output or residual stream site that we tested on.
In Fig. at the attention output pre-linear site at layer 27, loss recovered is bigger than 1.0. On inves- tigation, we found that the dataset used train the SAE was identical to Gemma's pretraining dataset, and at this site it was possible to mean ablate this quantity and decrease loss explaining why SAE reconstructions had lower loss than the original model.
Table provides cross-entropy losses for the Gemma-7B and Pythia-2.8B, both before and after zero-ablating specific sub-layers of these models, help provide further context for interpreting the loss recovered results presented in this paper; 100% loss recovered corresponds to the SAE-spliced language model attaining loss matching the original language model, whereas 0% loss recovered corresponds to the SAE-spliced language model attaining a loss matching the language model with the corresponding sub-layer zero-ablated.
Further shrinkage plots
In Fig. 10, we show that Gated SAEs resolve shrinkage, as measured by relative reconstruction bias (Appendix C), in Pythia-2.8B.
G
Training and evaluation: hyperparameters and other details
G.1
Training
G.1.1
General training details
Other details of SAE training are:
SAE Widths. Our SAEs have width 2 for most baseline SAES, x 216 for Gated SAEs, except for the (Pythia-2.8B. Residual Stream) sites used 2 for baseline and 3 x 2 for Gated since early runs at these sites had lots learned feature death.
Training data. We activations from hundreds of millions billions of activations from LM forward passes as input data to the SAE. Following Nanda [29], we use a shuffled buffer of these activations, that optimization steps don't ton't use data from highly correlated activations." 14
Resampling. We used resampling, a technique which at a high-level reinitializes features
that activate extremely rarely on SAE inputs periodically throughout training, We mostly
IIn contrast to earlier findings [9], we found that when using P.thi activations from sequences of length 2048, rather than GELU-ILS IL's activations from sequences of length 128, it was important to shuffle the 10 length activation buffer used to train our SAEs.
18

---

0.6
0.8
Layer
Loss Recovered (Fidelity)
Layer 16
Layer 20
0.6
0.8
Type Gated
50
100
150
40
60
80
100
120
140
20
40
60
80
100
120 140
Residual PLSt-MLP
Attention output pre-linear
output
(Lower is sparser)
Figure 8: Gated SAEs throughout Pythia-2.8B. At all sites we tested, Gated SAEs are a Pareto improvement. In every plot, the SAE with maximal loss recovered was Gated SAE.
19

---

1.2
Layer 6
0.8
1.2
Loss Recovered (Fidelity)
0.8
0.6
1.2
Layer 20
0.9
0.6
Layer 27
0.8
0.6
Gated
50
100
150
200
150
200
50
100
200
Residual post-MLP
output
Attention output pre-linear
(Lower is sparser)
Figure 9: Gated and Normal Pareto-Optimal SAES for Gemma-7B see Appendix E for discussion of the anomalies (such the Layer 27 attention output SABs).
20

---

Layer
0.8
Relative reconstruction bias
Layer 13
0.8
Layer 20
0.8
Layer 27
SAE
Gated
0.8
Baseline
100
150
50
100
150
50
100
150
Residual stream post-MLP MLP
MLP output
Attention output pre-linear
(Lower is sporser)
Figure 10: Gated SAEs address the problem shrinkage in Pythia- 2.8B.
21

---

follow the approach described in the 'Neuron Resampling' appendix of Bricken et al. [8], except we reapply learning rate warm-up after each resampling event, reducing learning rate to 0. Ix the ordinary value, and, increasing it with a cosine schedule back to the ordinary value over the next 1000 training steps.
Optimizer hyperparameters. We use the Adam optimizer with $2 0.999 and A1 0.0, following Templeton et al. [47], as we also find this to be slight improvement to training. We use learning rate warm-up. See Appendix G.1.2 for learning rates of different experiment.
Decoder weight norm constraints. Templeton et al. [47] suggest constraining columns to have at most unit norm (instead of exactly unit norm), which can help distinguish between productive and unproductive feature directions (although it should have no systematic im- pact on performance). However, we follow the original approach of constraining columns have exact unit norms in this work for the sake of simplicity.
Compute resources. Individual SAEs were each trained on TPU-v3 slices with a 2x2 topology [20]. The same chips
used generate LM activations on-the-fly, train SAE parameters and evaluate SAEs during training, using to 8-way model parallelism. With this setup, the time train SAE varies by SAE width, LM residual stream dimension, sequence length, layer and site." 15 We also used a negligible amount of compute on re- sampling (Appendix G), evaluation (e.g. Figure 1) and interpretaillity experiments (Sec- tion 4.2). Training wall clock time ranges from around 7 hours to train on GELU-IL MLP activations to around 47 hours train on Gemma- 7B sites layer 27. We estimate that we used twice as much compute as used in the paper on preliminary experiments.
G.1.2 Experiment-speecffi training details
We use learning rate 0.0003 for all Gated SAE experiments, and the GELU-IL baseline experiment. We swept for optimal baseline learning rates for the GELU-IL baseline to generate this value. For the Pythia-2.8B and Gemma-7B baseline SAE experiments, we divided the L.2 loss by TE| |z|||, motivated by better hyperparameter transfer, and changed learning rate to 0.001 and 0. 0.00075. We didn't i see noticeable difference in the Pareto frontier and so did not sweep this hyperparameter further.
We generate activations from sequences of length 128 for GELU-IL, 2048 for Pythia-2.8B and 1024 for Gemma-7B.
We use a batch size of 4096 for all runs. We use 300,000 training steps for GELU-IL and Gemma-7B runs, and 400,000 steps for Pythia-2.8P runs.
G.1.3 Lessons learned scaling SAES
Learned feature death unpredictable. In Fig. 11 there are few patterns that can be
gleaned from staring at which runs have high numbers of dead learned features (called dead neurons in Bricken et al. [8]).
Resampling makes typrrpareeeter sweeps difficult. We found that resampling caused
LO and loss recovered to increase, similar to Conmy [9].
Training appears to converge earlier than expected. We found that we did not need 20B tokens as in Bricken et al. [8], as generally resampling had stopped causing gains and loss curves plateaued after just over one billion tokens.
G.2 Evaluation
We evaluated the models on over million held-out tokens.
22

---

H Equivalence between gated encoder with tied weights and linear encoder with non-standard activation function
In this section we show under the weight sharing scheme defined in Eq. (7), gated encoder as defined in Eq. (5) is equivalent a linear layer with a non-standard (and parameterized) activation function.
Without loss of generality, consider the case of single latent feature (M 1) and set the pre- encoder bias to zero. In this case, the gated encoder is defined as
f(x) == 1 w then ReLU (W mag X + mmag)
(13)
and the weight sharing scheme becomes
W mag == Prag W gate
(14)
with a non-negative parameter Pmag exp(rmag).
Substituting Eq. (14) into Eq. (13) and re-arranging, we can re-express f(xx) as single linear layer
f(x) == Poan--angpaa (Wmag
bmag)
(15)
with the parameterized activation function
0o(=) == 1220 ReI.U(2) (z).
(16)
called JumpRel.U different context [17]. Fig. illustrates the shape of this activation function.
I
Further analysis of the human interpretability study
We perform some further analysis on the data from Section 4.2, to understand the impact of different sites, layers, and raters.
I.1 Sites
We first pose the question whether there's evidence that the sites had different interpretability outcomes. A Friedman test across sites shows significant differences (at p 0.047) between the Sated-vs-Basellmg differences, though not (p = 0.92) between the raw labels.
Breaking down by site and repeating the Wilcoxon-Pras one-sided tests and computing confidence intervals, we find the result on MI.P outputs is strongest, with mean 0.40, significance p = 0.003, and CI [0.18, 0.63]; this is as compared with the attention outputs (p- 0.47, mean .05, CI [-0.16 1016 0.26)) and final residual (p- = 0.59, mean -0.07, CI [-0.28, 0. 0.1 SAEs.
1.2 Layers
Next we test whether different layers had different outcomes. We this separately for the 2 models, since the layers aren't directly comparable. We run tests in each setting: Page's 's trend test (which tests for a monotone trend across layers) and the Friedman test (which tests for any difference, without any expectation of monotone trend).
Results are presented in Table 2; they suggest there are some significant nonmonotone differences between layers. To elucidate this, we present 90% BCa bootstrap confidence intervals of the mean raw label (where "No'=0, 'Maybe'=l, 'Yes==2) and the Gated-vs-Bassllne difference, per layer, in Fig, 15 and Fig. 16, respectively.
1.3 Raters
In Table present test results weakly suggesting that the raters differed in their judgments. This underscores that there's still a significant subjective component to this interpretability labeling. (Notably, different raters saw different proportions of Pythia vs Gemma features, aggregating across the models is partially confounded by that.)
The FLOPs required to compute LM activations increase with layer; SAEs trained on MLP activations have higher parameter count than those trained on MLP outputs, attention outputs or the residual stream.
23

---

Layer
Proportion of are
(Last 1e6 tokens)
Layer
Layer
Type Galad
Raseine
Residual stream
MLP output
Attention output pre-lincar
LO (Lower is sparser)
Figure 11: Feature death in Gemma-7B.
p-values
Raw label
Delta from Baseline Gated
Pythia-2.8B (Page's trend test)
0.50
0.13
Pythia-2.88 (Friedman test)
0.57
0.05
Gemma-7B (Page's trend test)
0.037
0.31
Gemma-7B (Friedman test)
0.003
0.64
Table 2: Layer significance tests
p-values
Raw label
Delta from Baseline to Gated
Across models (Kruskal- Wallis H-test)
0.01
0.71
Pythia-2.8B (Friedman test)
0.13
0.05
Gemma-7B (Friedman test)
0.03
0.76
Table 3: Rater significance tests
24

---

Figure 12: After applying the weight sharing scheme of Eq. (7). gated encoder becomes equivalent single layer linear encoder with JumpRel.U (Erichson al. [17], previously named TRec by Konda et al. [23]) activation function de illustrated above.
model pythla-2.
model gemma-700 7b-pt
YES
11
12
40
gated
MAYBE
4
10
3
12
NO
5
2
NO
MAYBE baseline
YES
NO
MAYBE baseline
YES
Figure 13: Contingency table showing Gated vs Bascline interpretability labels from our paired study results, for Pythia-2.80 and Gemma-7B.
25

---

ACTIVATIONS DENSITY = 0.122%
800
94
NEGATIVE LOGITS
POSITIVE LOGITS
EOF
abone
+0.53
afcrmmentione
+0.38
-0.07
faregeing
+0.37
gelElement
-0.07
mentlosed
+0.36
lnvoke
0.07
Abore
+0.35
strugg
described
+0.31
-0.07
mentiened
+0.31
0.07
described
+026
(FEATURES UNEMBED) HISTOGRAM
15k
0.17
0.34
0.51
Activations range 29. 19.002991p 32.2555
this trmenos catere vocaiiyy The this as may when fook these hinds are feelng in pine -ow the Insanee Assrthed, tay mutriinn lobck;; Keraa Food Bocarch Instintc; reviad
Developennt Adminirratin''s rocarc anaysis. For food
datalaes ineationdd above,
Activations range 25, 29.0299
volcity diaritatio daych fisetiin spen
and sikking the fhis
volocity deseriteed alove,
temind
When
fron lie othe ihhn lihani which
that
other sation The Covernment Pilcita will vifsor of feeign cuchang, impost rostictioes, and inyutees inceperincaal seceing tedendogr vesud a say set operaing on repiar mas scholde; that is, trap stamas de not lave the unatitihed scholalos of other ttr typek of Garreers meach compoodding to the spening 41 fammed in the aliees haw matrral ab (TTG. 48) la allwweerr Jurine the day indiee Reseribed aluse, SID and herdy gromis ask Mr. Long cacae us we iaaoee his more in the finur. la Widhot any onc the lncell and lamill lintd bete la Fare and A, Celo. R. Sante, O Theineer and O. Goodecke: see reviews Incationnt alove. More oding eratgyy [I (/ddda-hhhh-- de intuliaaiion 3 intion p(l
agremment of Covernmnnt the Unitd -tates, denet ruch
murkeingg
firm whese charsteritio the opponite del thene coanoniii abovc
acc0n in badging deserthed shove canted
Activations range 22.5788
csttled
peasre
lotil
kill/nel TUDCA: sabusts whose hinding cottigget phopharyltion by sepenidy actinted kinne inl. Effects of times is time signal mesduction and rppaaiin jathnoze described above wall in fiece.
dosttell and
thirr incint dle skgpicuu coneriing eal, Kin coriiced only es onens endin overing the incieen dnserited dlove 'ila
trmuktion and pesplrase receccled with obvina logs sieple retatatioll la
virii jelgnen aqquttal yet noour nover. propert a new sipvribaai appoach, adibied in Fig. tigescuurt aurr) that inporees the St.c apprnach metivate by the dasthud diseaned abone. We collecion of fancines dut senctie cill fanciona oths packages (where some work las abeady bean coded-up webstiaiios can change the acahatic eneire imaginic quality of pocts, how coeld any of thnn change mosiing Deppii the strplc inatted the be
the accurioll il er abevete
An extract of feature visualization dashboard used to rate features in the interprctability study described in Section 4.2. The left-hand pane provides aggregate information, including a feature histogram and the tokens most promoted and demoted by the feature being rated. The rest of the dashboard displays samples of text which the feature activates various degrees. Holding the a hover showing the exact activation level at that token. Although not shown here, full dashboard provides examples across the full range of activations, down to examples on which the feature fails to activate. This particular feature, taken from layer 20 Gemma-7B residual stream Gated SAE seems to promote completions like "above". "aforementioned". "mentioned above" etc. in contexts where such completion would be likely.
Figure 14:
over
token reveals
26

---

1.7
1.6
90% CI for raw label
1.4
1.3
1,1
gemma- Po LTT
gemma-TT To po Lzg
gemma- Port
Pythis-2.88
Pythia2.88k 2.86 L16
Pythia-2.a0 2.80 L20
Pythia 2.86 L2B
Figure 15: Per-layer 90% confidence intervals for the mean interpretability label
90% CI for Della from Baseline to Gated
0.5
gemma- 7b L13
gemma- To L20
gemma 76
Pythia-2 Bb
Pythia 2.86 Bb113
Pythia-2 2.86 L16
bythia-22 2.8b L20
Pythia-2 2.8 L2B
Figure 16: Pey-layer 90% confidence intervals for the Gated-vs soadd----Baeeline label difference
27

---

site MTN,, PUT PRE_LINEAM
site FINAL RESIDUAL
site MLP OUUTUUT
YES
2
1
1
layte
layar 12
laysr 1.
layer 20
layer 25
200
90000
3
HAYBE
YES
YES
MAYTE
YES
WAYSE
YES
NAYBE
Figure 17: Contingency tables for the paired (gated vs baseline) interpretability labels, for Pythia- 2.8B
NAL RESIGAAL
NO
1E5
NO
MAFRE
YES
NO
NOVI:
YES
layer
layer 13
layer
layer 27
YES
4
4
gated
1
2
2
NO
3
2
NO
MAYBE baseline
YES
NO
MAYBE baseline
YES
NO
MAYBE baseline
YES
MO
MAYBE baseline
YES
Figure 18: Contingency tables for the paired (gated vs baseline) interpretability labels, for Gemma- 7B
28

---

J Pseudo-code for Gated SAEs and the Gated SAE loss function
def gated_sae(x, W_gate, b_gate, W_mag, b_mag, W_dec, b_dec):
# Apply pre-encoder bias x_cnee b_dec
# Gating encoder (estimates which features are active) active_featured ((x_center @ V_gate + b_gate) > 0)
# Magnitudes encoder (estimates active features' magnitudes)
feature Cestuure_aaniuddee relu(x_center 0 W_mag b_mag)
# Multiply both before decoding
fective_ paature_magnitudes. @ W_dec + b_dec
Figure 19: Pseudo-code for the Gated SAE forward pass.
def lxss(x, W_gate, b_gate, W_mag, b_mag, W_dec, b_dec); gated_sae_lose 0.0
# We'll use the reconstruction from the baseline forward pass to train # the magnitudes encoder and decoder. Note don't apply any sparsity
# penalty here. Also, gradient will propagate back to W_gate b_gate # due to binarising the gated activations zero or one.
recomstruction = gated_ (x, W_gate, b_gate, W_mag, b_mag, W_dec, b_dec) gated_sae_loss ** sum (reconstruction - x) + axis=-1)
# apply penalty on gated encoder activations (pre-binarising, # post-ReLU) to incentivise them to sparse x_ ceneeerr b_dec www.gatt_feattee... saaaturremaggitudds relu(x_center @ W_gate b_gate) gated_ sae_los 11_coef sum waweauureemmnituudee.....eee.. axig=-1)
# Currently the gated encoder only has gradient signal to be sparse, and also do "via gate" reconstruction, to # give it appropriate gradient signal. We stop the gradients to the # decoder parameters this forward pass, as don't want these to # influenced by this auxiliary task. ret_gaate_rcoossuucttino
# not to reconstruct well,
via_gate, www.ggatteeeatee_e_ee
aetatture_manittudds @ stop_gradient (W_dec)
stop- wwt.ggrrddenntbdeece
gated_sae loss sum(
x)**2, axis=-1)
return gated_ loss
Figure 20: Pseudo-code for the Gated SAE loss function. Note that this pseudo-code written for expositional clarity. In practice, taking into account parameter tying, it would be more efficient to rearrange the computation to avoid unnecessarily duplicated operations.
K Tables of Results
We evaluated the models on over a million held-out tokens. Tables 4-11 show summary stats from training runs on the Pareto frontier.
29

---

Sparsity
%
Clean
SAE
Abl.
Alive
Shrinkage
Site
Layer
LR
LO
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
7
Resid
6
3e-05
0.001
18.1
95.28%
2.5426
3.1847
16.1549
196608
16.8%
0.982
Resid
6
2e-05
0.001
10.5
85.3%
2.5426
4.5433
16.1549
196608
5.72%
1.136
Resid
6
le-05
0.001
19.0
91.24%
2.5426
3.7349
16.1549
196608
5.11% %
1.606
Resid
6
2e-05
0.00075
29.8
96.65%
2.5426
2.9989
16.1549
196608
13.67%
1.261
Resid
6
3e-05
0.00075
25.4
97.9%
2.5426
2.8279
16.
196608
38.86%
0.976
Resid
6
8e-06
0.00075
29.8
91.28%
2.5426
3.7301
16.1549
196608
9.88%
1.105
Resid
6
le-05
0.00075
57.3
97.36%
2.5426
2.9023
16.1549
196608
11.78%
1.03
Resid
6
4e-06
0.00075
69.2
95.98%
2.5426
3.0892
16.
196608
13.54%
1.239
Resid
6
6e-06
0.00075
40.0
95.49%
2.5426
3.1562
16. 15.9
196608
24.34%
1.159
Resid
13
9e-05
0.00075
14.3
96.77%
2.5426
3,4423
30.3588
196608
98.38%
0.806
Resid
13
8e-05
0.00075
17.5
97.66%
2.5426
3.1947
30.3588
/ /96608
98.7%
0.824
Resid
13
8e-05
0.001
18.0
97.63%
2.5426
3,2021
30.3588
196608
95.35%
0.838
Resid
13
5e-05
0.00075
22.2
97.69%
2.5426
3.1849
30.3588
/96608
25.78%
0.889
Resid
13
3e-05
0.00075
29.0
97.64%
2.5426
3.1986
30,3588
196608
8.55%
0,903
Resid
13
5e-05
0.001
29.5
98.71%
2.5426
2.9005
30.3588
196608
65.17%
0.867
Resid
13
3e-05
0.001
39.2
98.26%
2.5426
3.026
30,3588
196608
26.33%
0.936
Resid
13
2e-05
0.00075
56.6
98.49%
2.5426
2.9615
30.3588
196608
16.19%
0.976
Resid
13
lc-05
0.00075
101.3
97.83%
2.5426
3.1459
30.3588
196608
4.55%
1.018
Resid
20
0.00012
0.00075
10.4
91.87%
2.5426
3.9277
19.5891
196608
92.51%
0.773
Resid
20
0.0001
0.00075
13.8
93.68%
2.5426
3,6204
19.5891
196608
97.46%
0.797
Resid
20
9e-05
0.00075
16.0
94.48%
2.5426
3,4835
19.5891
/ 196608
99.2%
0.8/
Resid
20
3e-05
0.001
25.2
90.71%
2.5426
4.1258
19.5891
196608
3.11%
0.951
Resid
20
7e-05
0.001
21.3
95.73%
2.5426
3.27
19.589/
/ 196608
99.62%
0.824
Resid
20
5e-05
0.001
27.8
97.15%
2.5426
3,0281
19.5891
196608
88.4%
0.879
Resid
20
3e-05
0.00075
39.1
96.43%
2.5426
3.1518
19.5891
196608
35.64%
1.019
Resid
20
4e-05
0.00075
46.4
97.95%
2.5426
2.8922
19.5891
J96608
99.9%
0.874
Resid
20
2c-05
0.00075
49.4
95.26%
2.5426
3,3505
19.5891
196608
8.61%
0.983
Resid
20
1.5e-05
0.00075
50.3
95.99%
2.5426
3.2268
19.5891
196608
9.46%
2.179
Resid
20
1c-05
0.00075
124.8
97.69%
2.5426
2.9367
19.5891
196608
12.3%
0.997
Resid
27
le-05
0.001
27.6
47.08%
2.5426
7.7878
12.4534
196608
1.68%
1.022
Resid
27
8e-06
0.001
30.5
49.63%
2.5426
7.5345
12.4534
/96608
1.12%
0.965
Resid
27
1.2e-05
0.00075
36.2
39.49%
2.5426
8.5398
12.4534
1 196608
2.02%
1.564
Resid
27
66-06
0.001
39.1
52.72%
2.5426
7.228
12.4534
/96608
1.39%
1.035
Resid
27
4e-06
0.00075
63.4
61.84%
2.5426
6.3246
12.4534
196608
3.03%
1.017
Resid
27
2e-06
0.00075
88.2
58.45%
2.5426
6.6609
12.4534
196608
2.22%
1.163
MLP
6
0.0004
0.001
0.2
42.33%
2.5426
2.6774
2.7764
196608
19.17%
0.857
MLP
0.0001
0.001
6.3
67.78%
2.5426
2.6179
2.7764
196608
82.35%
0.794
MLP
6
0.0001
0.00075
7.6
59.55%
2.5426
2.6371
2.7764
196608
69.88%
1.189
MLP
7e-05
0.001
10.6
70.77%
2.5426
2.6109
2.7764
196608
75.8%
0.835
MLP
6
3e-05
0.00075
15.3
64.49%
2.5426
2.6256
2.7764
196608
15.36%
1.001
MLP
6
7e-05
0.00075
12.0
74.63%
2.5426
2.6019
2.7764
/96608
94.97%
0.82
MLP
6
1.5e-05
0,00075
14.9
47.57%
2.5426
2.6651
2.7764
1 196608
3.03%
1.0
MLP
6
5e-05
0.00075
17.1
75.36%
2.5426
2.6002
2.7764
196608
68, 12%
0.864
MLP
13
8c-05
0.00075
1.4
32.78%
2.5426
2.573
2.5878
196608
10.16%
0.92
MLP
13
8e-05
0.001
11.3
50.99%
2.5426
2.5647
2.5878
J96608
73.07%
0.848
MLP
13
5e-05
0.001
22.6
47.32%
2.5426
2.5664
2.5878
1 196608
66.09%
0.882
MLP
13
Se-05
0.00075
29.4
61.19%
2.5426
2.5601
2.5878
196608
84.51%
0.863
MLP
13
3e-05
0.001
44.8
64.14%
2.5426
2.5588
2.5878
196608
56.91%
0.864
MLP
13
3e-05
0.00075
80.8
71.28%
2.5426
2,5556
2.5878
196608
73.31%
0.901
MLP
13
2e-05
0.00075
160.7
72.08%
2.5426
2.5552
2.5878
196608
56.12%
0.894
MLP
13
le-05
0.00075
610.0
77.67%
2.5426
2.5527
2.5878
/96608
44.39%
0.858
MLP
20
7e-05
0.001
15.8
79. 19%
2.5426
2.573
2.6881
196608
96.84%
0.852
MLP
20
5e-05
0.001
24.5
82.67%
2.5426
2.5678
2.6881
196608
96.93%
0.869
Table 4: Gemma-7B Baseline SAEs (1024 sequence length). Italic Pareto optimal SAEs.
30

---

Sparsity A
% CE
Clean
SAE
Abl.
% Alive
Shrinkage
Site
Layer
LR
L.
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
MLP
20
5e-05
0.00075
26.0
82.36%
2.5426
2.5682
2.6881
196608
97.96%
0.865
MLP
20
4.5e-05
0.00075
31.4
83.94%
2.5426
2.5659
2.688/
196608
99.24%
0.877
MLP
20
3e-05
0.001
39.5
83. 83.12%
2.5426
2.5671
2.6881
196608
46.33%
0.924
MLP
20
4e-05
0.00075
38.3
85. 18%
2.5426
2.5641
2.688/
196608
95.73%
0.889
MLP
20
3,5e-05
0.00075
43.2
84.11% %
2.5426
2.5657
2.6881
196608
94.62%
0.874
MLP
20
3e-05
0.00075
56.8
87.23%
2.5426
2.5612
2.6881
196608
96.88%
0.894
MLP
2e-05
0.00075
68.1
84.18%
2.5426
2.5656
2.6881
196608
53.42%
0.898
MLP
20
2e-05
0.00075
75.6
85.63%
2.5426
2.5635
2.6881
196608
66.29%
0.899
MLP
20
1.5e-05
0.00075
104.6
85.71%
2.5426
2.5634
2.6881
196608
41.7%
0.965
MLP
20
le-05
0,00075
321.1
90.3%
2.5426
2.5567
2.6881
196608
56.83%
0.911
MLP
27
1.2e-05
0.001
10.2
86.28%
2.5426
5.7751
26.1114
196608
0.6%
1.019
MLP
27
le-05
0.001
20.5
95.05%
2.5426
3.7081
26.1114
/96608
1.73%
L.002
MLP
27
8e-06
0.001
21.3
93.55%
2.5426
4.0623
26.1114
196608
0.66%
0.988
MLP
27
6e-06
0.00075
26.4
91.19%
2.5426
4.6185
26.1114
196608
0.57%
0.973
MLP
27
5.5e-06
0.00075
18.1
85.53%
2.5426
5.9522
26.1114
196608
0.58%
0.994
MLP
27
3e-06
0.00075
26.9
90.82%
2.5426
4.706
26.1114
196608
0.98%
1.024
Atin
7e-05
0.00075
15.4
69.89%
2.5426
2.5989
2.7295
196608
96.78%
0.72
Attn
Se-05
0.00075
26.4
78.08%
2.5426
2.5836
2.7295
196608
98.97%
0.777
Atn
6
3e-05
0.00075
54.6
85.42%
2.5426
2.5698
2.7295
196608
99.7%
0.846
Attn
13
7e-05
0.00075
22.6
60.79%
2.5426
2.5481
2.5566
196608
93.47%
0.721
Attn
13
5e-05
0.00075
36.5
65.45%
2.5426
2.5474
2.5566
196608
97.59%
0.786
Atm
3e-05
0.00075
68.8
81.03%
2.5426
2.5452
2.5566
196608
99.19%
0.804
Atn
20
9e-05
0.00075
10.8
68.98%
2.5426
2.5519
2.5726
196608
79.34%
0.715
Atm
20
8e-05
0.00075
12.3
72.48%
2.5426
2.5508
2.5726
196608
83.58%
0.723
Atln
20
7e-05
0.00075
15.9
75.83%
2.5426
2.5498
2.5726
196608
87.54%
0.755
Atnn
6e-05
0.00075
18.7
78.38%
2.5426
2.5491
2.5726
196608
89.49%
0.759
Atln
20
5e-05
0.00075
25.1
82.96%
2.5426
2.5477
2.5726
196608
92.36%
0.786
Atm
20
4e-05
0.00075
32.6
85.95%
2.5426
2.5468
2.5726
196608
95.14%
0.802
Att
20
3e-05
0.00075
50.3
89.52%
2.5426
2.5457
2.5726
196608
96.52%
0.841
Atm
20
2e-05
0.00075
97.3
92.52%
2.5426
2.5448
2.5726
196608
95.74%
0.878
Attn
20
1.5e-05
0,00075
148.6
95.01%
2.5426
2.5441
2.5726
196608
92.55%
0.867
Atn
20
le-05
0.00075
329.7
96.57%
2.5426
2.5436
2.5726
196608
78.75%
0.895
Atm
27
0.0008
0.00075
0.0
121.03%
2.5426
2.4291
3.0819
196608
5.34%
1.009
Amm
27
0.0006
0.00075
0.0
121.63%
2.5426
2,4259
3,0819
196608
4.7%
1.007
Atin
27
0.0001
0.00075
9.7
126.97%
2.5426
2.3971
3,0819
196608
35.94%
0.829
Table 5: Gemma-7B 7B Baseline SAEs (1024 sequence length) continued from Table
31

---

Sparsity
% CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
LO
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
Resid
6
0.0012
0.0003
2.2
95.55%
2.5426
3.1483
16.1549
131072
93. 93.94%
1.006
Resid
6
0.001
0.0003
3.0
96.67%
2.5426
2.9954
16.1549
131072
96.24%
1.006
Resid
6
0.0008
0.0003
4.3
97.83%
2.5426
2.8382
16.1549
131072
97.52%
1.003
Resid
6
0,0006
0.0003
7.0
98.765 78%
2.5426
2.7108
16.1549
131072
98.3%
0.996
Resid
6
0.0004
0.0003
14.3
99.35%
2.5426
2.6312
16.1549
131072
98.68%
0.996
Resid
6
0.0002
0.0003
45.9
99.77%
2.5426
2.5735
16.
131072
99.51%
0.999
Resid
2e-05
0.0003
95.2
98.62%
2.5426
2.7302
16.1549
131072
45.13%
1.148
Resid
6
4e-05
0.0003
144.0
99.35%
2.5426
2.6313
16.1549
131072
36.05%
1.038
Resid
6
8e-06
0.0003
177.5
99.29%
2.5426
2.6386
16.1549
131072
53.36%
1.086
Resid
6
0.0001
0.0003
131.8
99.94%
2.5426
2.5511
16.1549
131072
99.47%
1.005
Resid
8e-05
0.0003
153.2
99.93%
2.5426
2.5524
16.1549
131072
14%
0.984
Resid
6
6e-05
0.0003
215.7
99.93%
2.5426
2.5521
16.1549
131072
93.91%
0.982
Resid
4c-05
0.0003
284.5
99.62%
2.5426
2.5948
16.1549
131072
84.71%
2.56
Resid
6
2e-05
0.0003
801.3
99.82%
2.5426
2.5673
16.1549
131072
91.71%
1.272
Resid
6
8c-06
0.0003
-288.2
99.7%
2.5426
2.5835
16.1549
131072
85.02%
1.006
Resid
13
0.0008
0.0003
5.4
98.3%
2.5426
3.0149
30.3588
131072
98. 15%
1.008
Resid
13
0.0005
0.0003
13.1
99.25%
2.5426
2.7514
30.3588
131072
98.71%
0.998
Resid
13
0.0003
0.0003
31.8
99.62%
2.5426
2.6483
30.3588
131072
99.31%
0.992
Resid
13
0.0002
0.0003
62.6
99.76%
2.5426
2,6083
30.3588
131072
99.69%
0.993
Resid
13
0.0002
0.0003
63.7
99.77%
2.5426
2.6067
30.3588
131072
99.68%
0.997
Resid
0.0001 13
0.0003
146.1
99.87%
2.5426
2.5788
30.3588
131072
67.47%
1.056
Resid
13
0.0001
0.0003
96.8
99.64%
2.5426
2.6421
30.3588
131072
64.18%
0.934
Resid
20
0.001
0.0003
8.2
96.15%
2.5426
3.1995
19,5891
131072
96.49%
1.004
Resid
20
0.0009
0.0003
10.0
96.7%
2.5426
3.1059
19.5891
131072
96.89%
1.003
Resid
20
0, 0,0008
0.0003
12.3
97.14%
2.5426
3,0293
19,5891
131072
97.46%
0.997
Resid
20
0.0007
0.0003
15.6
97.7%
2.5426
2.9353
19.5891
131072
98.02%
0.997
Resid
20
0.0005
0.0003
29.3
98.62%
2.5426
2.7775
19.5891
131072
98.66%
1.016
Resid
20
0.0005
0.0003
28.0
98.53%
2.5426
2.7931
19.5891
131072
98.73%
0.997
Resid
20
0.0005
0.0003
28.5
98.58%
2.5426
2.7844
19.5891
131072
98.67%
1.004
Resid
20
0.0003
0.0003
67.3
99.3%
2.5426
2.6611
19.5891
131072
99.33%
1.013
Resid
20
0.0002
0.0003
123.4
99.58%
2.5426
2.6139
19.5891
131072
99.69%
1.01
Resid
20
0.0001
0.0003
212.1
99.65%
2.5426
2.6024
19,5891
131072
55.01%
1.04
Resid
27
0,003
0.0003
17.3
81.66%
2.5426
4.3602
12.4534
131072
28.57%
1.001
Resid
27
0.002
0.0003
25.9
85.26%
2.5426
4.0033
12.4534
131072
31.98%
0,999
Resid
27
0,001
0.0003
54.4
90.26%
2.5426
3.5081
12.4534
131072
33.58%
1.008
MLP
6
0.0004
0.0003
4.0
73.71%
2.5426
2,604
2.7764
131072
98.69%
1.009
MLP
6
0.0001
0.0003
45.2
89.13%
2.5426
2.568
2.7764
131072
96.23%
0.998
MLP
6
7e-05
0.0003
106.0
90.67%
2.5426
2.5644
2.7764
131072
87.51%
1.O
MLP
13
9e-05
0.0003
36.0
76.36%
2.5426
2.5533
2.5878
131072
99.87%
1.002
MLP
13
9e-05
0.0003
36.1
76.25%
2.5426
2.5533
2.5878
131072
99.91%
1.004
MLP
13
8e-05
0.0003
48.9
78.71%
2.5426
2.5522
2.5878
131072
99.72%
1.007
MLP
13
7e-05
0.0003
69.7
82.15%
2.5426
2,5506
2.5878
131072
99.77%
1.01
MLP
13
7c-05
0.0003
67.0
81.24%
2.5426
2.5511
2.5878
131072
99.61%
0.997
Table 6: Gemma-7B Gated SAEs (1024 sequence length). Continued in Table 7.
32

---

Sparsity
CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
LO
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
MLP
5e-05 13
0.0003
196.4
85.54%
2.5426
2.5491
2.5878
131072
76.56%
1.003
MLP
13
3e-05
0.0003
766.5
93.04%
2.5426
2.5457
2.5878
131072
86.81%
1.033
MLP
20
0.00019
0.0003
24.4
87.81%
2.5426
2.5603
2.6881
131072
99.91%
1.004
MLP
20
0.00016
0.0003
32.7
89.16%
2.5426
2.5583
2.6881
131072
99.94%
1.004
MLP
20
0.00015
0.0003
36.4
89.63%
2.5426
2.5577
2.6881
131072
99.95
1.002
MLP
20
0.00014
0.0003
40.8
89.73%
2.5426
2.5575
2.6881
131072
99.96%
1.0
MLP
20
0.00013
0.0003
46.6
90.3%
2.5426
2.5567
2.6881
131072
99.95%
1.002
MLP
20
0.00012
0.0003
53.5
99%
2.5426
2.5557
2.688/
131072
99.99%
1.001
MLP
20
0.0001
0.0003
74.9
91.42%
2.5426
2.555/
2.6881
131072
99.99%
0,999
MLP
20
9e-05
0.0003
91.2
92.01%
2.5426
2.5542
2.6881
131072
99.9%
0,998
MLP
20
8e-05
0,0003
111.3
93.3%
2.5426
2.5523
2.6881
131072
100.0%
1.0
MLP
1.1e-05
O. 0.0003
-91.1
103.85%
2.5426
2.537
2.6881
131072
46.33%
1.005
MLP
27
0.0012
0.0003
20.3
14%
2.5426
3.9232
26.1114
131072
5.8%
1.003
MLP
27
0.001
0.0003
23.1
96.01%
2.5426
3.4834
26.1114
131072
6.13%
0,995
MLP
27
0.0008
0.0003
27.3
96.47%
2.5426
3.3747
26.1114
131072
5.18%
1.005
MLP
27
0.0003
0.0003
59.3
99.07%
2.5426
2.7627
26.1114
131072
3.89%
1.002
MLP
27
0.0002
0.0003
80.9
98.19%
2.5426
2.969
26.1114
131072
3.64%
1.006
MLP
27
0.000175
0.0003
89.7
97.35%
2.5426
3.1678
26.1114
131072
3.89%
1.008
MLP
27
0.00015
0.0003
108.5
98.87%
2.5426
2.8093
26.1114
131072
3.54%
1.002
MLP
27
0.000135
0.0003
103.6
98.33%
2.5426
2.9365
26.1114
131072
3.75%
0.997
Atin
0. 0.0007
0.0003
8.9
82.28%
2.5426
2.5757
2.7295
131072
93.49%
1.015
Atnn
0,0005
0.0003
16.4
85.54%
2.5426
2.5696
2.7295
131072
95.16%
1.014
Attn
0.0003
0.0003
38.7
88.69%
2.5426
2.5637
2.7295
131072
97.63%
1.015
Atn
13
0,0012
0.0003
2.9
46.05%
2.5426
2.5502
2.5566
131072
63.06%
1.042
Attn
O. 0,00066
0.0003
13.2
76.64%
2.5426
2.5459
2.5566
131072
83.81%
1.0
Attn
13
0.0004
0.0003
28.1
63.78%
2.5426
2.5477
2.5566
131072
89.64%
0,992
Atm
0.0002
0.0003
95.1
82.86%
2.5426
2.545
2.5566
131072
97.05%
0.993
Atn
13
4e-05
0.0003
1079.5
93.95%
2.5426
2.5434
2.5566
131072
64.6%
1.002
Attn
2e-05
0.0003
-635.1
87.73%
2.5426
2.5443
2.5566
131072
92.21%
1.003
Atln
20
0.0012
O. 0.0003
2.1
64.17%
2.5426
2.5533
2.5726
131072
72.67%
1.038
Atnn
0.0006
O. 0.0003
9.0
80.22%
2.5426
2.5485
2.5726
131072
89.06%
1.014
Atln
20
O. 0.00055
0,0003
10.1
84.01%
2.5426
2.5474
2.5726
131072
90.35%
0.997
Atm
20
0.00045
0.0003
14.8
85.85%
2.5426
2.5468
2.5726
131072
92.05%
1.003
Att
20
0.0004
0.0003
18.7
86.55%
2.5426
2.5466
2.5726
131072
92.77%
1.016
Atm
20
0.00035
0.0003
22.8
88.2%
2.5426
2.5461
2.5726
131072
94.07%
1.009
Attn
20
0.00025
0.0003
39.7
90.97%
2.5426
2.5453
2.5726
131072
96.42%
1.009
Atn
20
0.0002
0.0003
55.2
92.72%
2.5426
2.5448
2.5726
131072
97.73%
0,994
Atn
20
0.00015
0.0003
89.1
94.39%
2.5426
2.5443
2.5726
131072
98.93%
0.999
Atm
20
0.0001
0.0003
178.0
94.71%
2.5426
2.5442
2.5726
131072
99.69%
1.003
Atin
20
6e-05
0.0003
483.8
99.72%
2.5426
2.5427
2.5726
131072
98.60%
0.994
Attn
20
4e-05
0.0003
894.6
97.03%
2.5426
2.5435
2.5726
131072
66.5%
0.991
Atin
20
2e-05
0.0003
-851.3
106.91%
2.5426
2.5405
2.5726
131072
86.24%
1.0
Attn
27
0.002
0.0003
6.6
100.37%
2.5426
2.5406
3.0819
131072
56.82%
1.008
Atm
27
0.001
0.0003
16.5
105.72%
2.5426
2.5117
3.0819
131072
70.25%
1.002
Attn
27
0.0007
0.0003
26.2
104.26%
2.5426
2.5196
3.0819
131072
77.02%
0,999
Table 7: Cemma-7B Gated SAEs (1024 sequence length). Continued from Table 6
33

---

Sparsity
% CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
LO
Width
X
Recovered
CE Loss
CE Loss
CE Loss
Features
?
Atin
85-05
0.001
17.6
81.04%
1.9699
1.9824
2.0361
196608
94.29%
0.827
Atm
4
6e-05
0.001
24.2
84.12%
1.9699
1.9804
2.0361
196608
95.76%
0,848
Atm
3e-05
0.001
62.1
90.96%
1.9699
1.9759
2.0361
196608
96.72%
0.93
Atm
12
8e-05
0.001
16.1
51.88%
1.9699
1,9907
2.0131
196608
65.73%
0.78
Aun
6e-05
0.001
24.0
58.46%
1.9699
1.9878
2.0131
196608
69.85%
0.802
Atn
12
3e-05
0.001
75.0
72.84%
1.9699
1.9816
2.0131
196608
73.04%
0.848
Atm
0.00045
0.001
0.3
-3.54%
1.9699
2.0058
2.0046
49152
20.1%
0.554
Amn
16
8e-05
0.001
14.6
67.69%
1.9699
1.981
2.0046
196608
64.35%
0.798
Attn
3e-05
0.001
63.0
81.78%
1.9699
1.9762
2.0046
196608
70.75%
0.868
Atln
16
6e-05
0.001
20.8
72.07%
1.9699
1,9796
2.0046
196608
69.92%
0,813
Aun
0.0001
0.001
9.5
60. 16%
1.9699
1,9837
2.0046
49152
88.32%
0.754
Atm
16
9e-05
0,001
11.3
62.62%
1.9699
1.9829
2.0046
49152
89.87%
0.769
Atm
20
6e-05
0.001
18.3
87.49%
1.9698
1.9769
2.0269
196608
63.81%
0.87
Attn
20
8e-05
0,001
13.6
85.63%
1.9698
1.978
2.0269
196608
60.17%
0,871
Atn
20
3e-05
0.001
52.0
91.92%
1.9698
1.9744
2.0269
196608
65.83%
0.899
Atm
28
3e-05
0.001
91.9
73.29%
1.9698
1.9715
1.976
196608
71.36%
0.817
Atn
28
6e-05
0.001
20.6
57.17%
1.9698
1.9725
1.976
196608
64.79%
0.771
Atm
28
8e-05
0.001
12.5
49.8%
1.9698
1.9729
1.976
196608
55.92%
0.747
MLP
3.5e-05
0.001
20.0
86.36%
1.9698
1.9802
2.046
196608
95.6%
0.954
MLP
4
le-05
0.001
64.5
83.61%
1.9698
1.9823
2.046
196608
42.92%
0.977
MLP
2e-05
0.001
43.3
87.2%
1.9698
1,9796
2.046
196608
74.78%
0,986
MLP
12
3e-05
0.001
77.8
81.95%
1.9698
1.9783
2.0167
196608
99.58%
0.932
Table Pythia-2.88 baseline SAEs (2048 sequence length). Continued Table
34

---

Sparsity X
CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
L.0
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
MLP
12
5e-05
0.001
28.2
76.01%
1.9698
1.9811
2.0167
196608
99.45%
0.909
MLP
12
7e-05
0.001
16.2
71.94%
1.9698
1.983
2.0167
196608
99. 14%
0.883
MLP
16
2.5e-05
0.001
79.8
78.44%
1,9698
1.9785
2.0098
196608
99.83%
0.919
MLP
16
4e-05
0.001
29.0
72.83%
1.9698
1.9807
2.0098
196608
99.82%
0.923
MLP
16
3.5e-05
0.001
35.9
73.95%
1. 1,9698
1.9803
2.0098
196608
99.83%
0.914
MLP
16
7.5e-05
0.001
11.2
65.88%
1.9698
1.9835
2.0098
196608
99. 45%
0.884
MLP
16
4.5e-05
0.001
22.0
70.73%
1,9698
1.9815
2.0098
196608
99. 79%
0,901
MLP
16
3e-05
0.001
54.9
76.5%
1,9698
1.9792
2.0098
196608
99.86%
0.947
MLP
20
3.5e-05
0.001
20.6
91.28%
1.9698
1.9814
2.1022
196608
95.85%
0,971
MLP
20
2.5e-05
0.001
25.4
91.64%
1,9698
1,9809
2.1022
196608
90. 1.5%
0.964
MLP
20
7e-06
0.001
269.2
93.37%
1.9698
1.9786
2.1022
196608
17.28%
0.962
MLP
28
2.25e-05
0.001
95.2
79.05%
1,9698
1.9792
2.0145
196608
99.81%
0.941
MLP
28
4.5e-05
0.001
18.5
67.4%
1.9698
1.9844
2.0145
196608
94.33%
0.92
MLP
28
3e-05
0.001
37.0
71.12%
1,9698
1.9827
2.0145
196608
92.72%
0.932
Resid
4
3e-05
0.001
15.9
98.11%
1.9699
2.1793
13.0434
49152
96.34%
0.966
Resid
4
2e-05
0.001
28.1
98.67%
1,9699
2.1174
13.0434
49152
97.0%
0.974
Resid
4
le-05
0.001
79.1
99.27%
1.9699
2.0506
13.0434
49152
98.93%
0.983
Resid
12
le-05
0.001
128.7
97.68%
1,9698
2.1712
10.6558
49152
52.7%
0,951
Resid
12
3e-05
0.001
25.1
93.87%
1.9698
2.5021
10.6558
49152
64.28%
0.96
Resid
12
2e-05
0.001
52.1
96.34%
I. 1,9698
2.2874
10.6558
49152
67.39%
0.979
Resid
16
2e-05
0.001
42.7
95.55%
1,9698
2.4025
11.682
49152
68, 48.44%
0.975
Resid
16
le-05
0.001
94.8
96.48%
1,9698
2.3119
11.682
49152
36.81%
0.94
Resid
16
1.5e-05
0.001
55.5
95.97%
1,9698
2.3609
11.682
49152
59.52%
0.95
Resid
16
3e-05
0.001
17.1
90.16%
1. 1,9698
2.9252
11.682
196608
9.91%
0,932
Resid
16
5e-05
0.001
10.9
86.0%
1,9698
3,3293
11.682
196608
8.82%
0.929
Resid
16
8e-06
0.001
49.1
84.1%
1.9698
3.5145
11.682
196608
1.06%
0.946
Resid
20
7e-06
0.001
103.4
91.94%
1,9698
2.6543
10.4578
49152
15.4%
1.016
Resid
20
2e-05
0.001
33.4
90.97%
1,9698
2.7363
10.4578
49152
46.57%
0,986
Resid
20
4e-05
0.001
13.6
86.19%
1,9698
3.1421
10.4578
49152
59.96%
0.954
Resid
28
2e-05
0.001
21.0
95.09%
1.9698
3.242
27.8663
49152
20.22%
0.916
Resid
28
7e-06
0.001
109.2
97.45%
1,9698
2.6298
27.8663
49152
20.65%
1.021
Resid
28
le-05
0.001
42.9
96.27%
1.9698
2.9349
27.8663
49152
22.59%
0.932
Table 9: Pythia-2.8B baseline SAEs (2048 sequence length). Continued from Table 8.
35

---

Sparsity X
% CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
L0
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
Attn
4
0,0006
0.0003
38.2
92.85%
1.9699
1.9746
2.0361
131072
93.76%
1,006
Attn
4
0.0004
0.0003
69.8
94.82%
1.9699
1.9733
2.0361
131072
96.29%
1.0
Attn
4
0.0008
0.0003
24.7
90.94%
1.9699
1.9759
2.0361
131072
91.45%
1.007
Attn
12
0.0006
0.0003
64.5
82.04%
1.9699
1,9776
2.0131
131072
74.48%
0.99
Attn
12
0.001
0.0003
27.1
73.09%
1.9699
1.9815
2.0131
131072
63.68%
0,987
Attn
12
0.0008
0.0003
40.5
77.52%
1.9699
1,9796
2.0131
131072
67.74%
0.998
Artn
16
0.001
0.0003
17.2
79.67%
1.9699
1.9769
2.0046
32768
89.76%
0.988
Att
16
0.0006
0.0003
39.1
87.21%
1.9699
1.9743
2.0046
131072
80.93%
0.985
Artn
0.0009 16
0.0003
20.8
81.8%
1.9699
1.9762
2.0046
32768
91.0%
0.993
Attn
0.0004
O. 0,00
77.2
90.56%
1.9699
1.9732
2.0046
131072
85.48%
0.987
Atin
16
0.0008
0.0003
25.0
83.57%
1.9699
1.9756
2.0046
131072
79.41%
0.993
Attn
0.0005
0.00
57.8
88.63%
1.9699
1.9738
2.0046
32768
96.08%
0.992
Attn
20
0.0004
0.0003
71.2
96.25%
1.9698
1.972
2.0269
131072
88.74%
0.992
Attn
20
0.0006
0. 0.0003
36.5
94.34%
1.9698
1.973
2.0269
131072
85.88%
0.986
Attn
20
0.0008
0.0003
24.0
93.05%
1.9698
1.9738
2.0269
131072
83.05%
0.994
Amn
28
0.0008
0.0003
27.8
73.39%
1.9698
1.9715
1.976
131072
68.41%
0.988
Amn
28
0.001
0.0003
17.7
68.35%
1,9698
1.9718
1.976
131072
68.14%
0.991
Amn
28
0.0006
0.0003
51.2
78.11%
1.9698
1.9712
1.976
131072
72.44%
0.986
MLP
4
0.0006
0.0003
28.6
89.28%
1.9698
1.978
2.046
131072
99.16%
1.011
MLP
4
0.0004
0.0003
66.5
74%
1.9698
1.9754
2.046
131072
99.52%
1.002
MLP
4
0.0008
0.0003
15.8
87.13%
1.9698
1.9796
2.046
131072
98.46%
1.007
MLP
12
0.001
0.0003
35.0
81.33%
1.9698
1.9786
2.0167
131072
97.55%
1.011
MLP
12
0.002
0.0003
8.2
72.1%
1.9698
1.9829
2.0167
131072
94.68%
1,002
MLP
12
0,0008
0. 0.0003
55.7
84.15%
1.9698
1.9773
2.0167
131072
98.23%
1.004
Table 10: Pythia-2.8B Gated SAEs (2048 sequence length). Continued in Table 11.
36

---

Sparsity
% CE
Clean
SAE
0 Abl.
% Alive
Shrinkage
Site
Layer
LR
LO
Width
Recovered
CE Loss
CE Loss
CE Loss
Features
MLP
0.0008 16
0.0003
51.0
80.32%
1.9698
1.9777
2.0098
131072
99.05%
1.002
MLP
0.0016 16
0.0003
12.4
70.76%
1.9698
1.9815
2.0098
131072
97.38%
1.005
MLP
16
0.0007
0.0003
70.1
82.09%
1.9698
1.977
2.0098
131072
99.32%
1.001
MLP
16
0.0014
0.0003
16.1
72.62%
1.9698
1.9808
2.0098
131072
97.48%
1.007
MLP
16
0.0012
0.0003
21.9
75.12%
1.9698
1.9798
2.0098
131072
98. 18%
1.012
MLP
16
0.0009
0.0003
38.3
78.41%
1.9698
1.9785
2.0098
131072
98.72%
0.993
MLP
20
0.0008
0.0003
51.0
94.28%
1.9698
1.9774
2.1022
131072
99.06%
1.007
MLP
20
0.0012
0.0003
22.1
92.53%
1.9698
1.9797
2.1022
131072
97.97%
1.0
MLP
20
0.001
0.0003
30.9
93.27%
1.9698
1.9788
2.1022
131072
98.39%
1.003
MLP
28
0.001
0.0003
47.7
79.96%
1.9698
1.9788
2.0145
131072
98.76%
1.004
MLP
28
0.0008
0.0003
8
83.68%
1.9698
1.9771
2.0145
131072
98.48%
1.002
MLP
28
0.0015
0.0003
21.3
73.3%
1.9698
1.9818
2.0145
131072
97.58%
1.004
Resid
4
0.0008
O. 0.0003
70.7
99.5%
1.9699
2.0257
13.0434
32768
99.68%
0.996
Resid
4
0.001
0.0003
49.0
99.37%
1.9699
2.0399
13.0434
32768
99.52%
0.998
Resid
4
0.002
0. 0.0003
16.2
98.83%
1,9699
2.0998
13.0434
32768
98.72%
1.001
Resid
12
0.004
0.0003
16.2
95.92%
1.9698
2.3239
10.6558
32768
72.56%
1.003
Resid
12
0,0016
0.0003
77.1
98.61%
1.9698
2.0908
10.6558
32768
85.53%
0.998
Resid
12
0.002
0.0003
52.8
98.2%
1.9698
2.1261
10.6558
32768
83.41%
1.0
Resid
16
0.003
0.0003
37.5
97.46%
1.9698
2.2162
11.682
32768
78.14%
1.0
Resid
16
0.006
0.0003
12.5
94.29%
1.9698
2.5249
11.682
32768
62.59%
0.998
Resid
16
0.002
0.0003
71.5
98.33%
1.9698
2.1324
11.682
32768
82.89%
0.998
Resid
16
0.0025
0.0003
46.2
98.04%
1.9698
2.1597
11.682
131072
38.15%
0.993
Resid
16
0.0045
0.0003
18.9
96.55%
1,9698
2.3045
11.682
131072
38.92%
0.996
Resid
16
0.0015
0.0003
95.6
98.62%
1.9698
2.104
11.682
131072
29.77%
0.991
Resid
20
0.0075
0.0003
15.4
91.68%
1.9698
2.6763
10.4578
32768
59.39%
0.994
Resid
20
0.004
0.0003
38.7
95.09%
1.9698
2.3866
10.4578
32768
65.15%
0.995
Resid
20
0.003
0.0003
58.4
96.05%
1.9698
2.3053
10.4578
32768
68.08%
0.994
Resid
28
0.0075
0.0003
25.0
96.54%
1.9698
2.8646
27.8663
32768
29.97%
0.993
Resid 28
0.005
0.0003
46.6
97.58%
1.9698
2.5973
27.8663
32768
40.94%
1.008
Resid
28
0.004
0.0003
61.2
97.9%
1.9698
2.5136
27.8663
32768
35.93%
1.005
Table 11: Pythia-2.8B Gated SAEs (2048 sequence length). Continued from Table 10.
37

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated our main contributions in the abstract and introduction, and have taken care to ensure they accurately reflect the contents of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitations section under the conclusion, discussing the limita- tions of our experimental design and of open questions around SAEs that our work does not address.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the ap- proach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-texi system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
38

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The main contributions of this paper are not theoretical. There are two minor points in the main paper deriving an expression for the shrinkage metric y and showing that tied-weights Gated SAE is equivalent to single-layer encoder SAE with Jump- ReLU activation where fuller explanations are provided in Appendix C and Appendix H respectively (and referenced from the main text). These not numbered theorems this would be excessive for these low-importance (and easily derived) points.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proof's provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: The main body of the paper provides a detailed explanation of the Gated SAE architecture and loss function (Section 3) that are the main contributions of the paper; ad- ditionally, pseudo-code to help reproduction is provided in Appendix J. We provide high level summary of the training approach in Section 2 and Section 3, with details on hyperpa- rameters and compute requirements in Appendix G. We also explain the methodologies for our benchmarking and manual interpretability experiments in Section sufficient detail to be able reproduce the results there.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
39

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
model (e.g., large language model), then there should
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are unable to provide open access to the activation datasets or code used train the SAEs in our experiments. However, the experiments have reported involved training SAEs soiely open-source language models, and we have provided pseudo-code for our main contributions (Appendix J). These alongside open-source codebases that available for training SAES (e.g. [5]) - should be enough to reproduce our main experimen- tal results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. http:/pullcgggdeeeeeeeeee Sut///edeeeiieaaccceeeeee iss sisoniiicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: The main body of the paper- in particular the training and evaluation subsec- tions of Section 2, the Gated SAE definition of Section 3.2 and the experimental method- ology subsections of Section explain training and evaluation details a sufficient level of detail to be able to understand and appreciate the results. Details of hyperparameters, optimizer type, etc are provided in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
40

---

The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do measure statistical significance in our interpretability study. Sec- tion 4.2 includes a detailed explanation of the statistical analysis undertaken to compare Gated and baseline SAEs (continued in Appendix I). The error bar methodology for Fig. 4 is explained in Footnote 9. However, error bars are not shown on the Pareto curve plots because it would have been computationally prohibitive. Nevertheless, the smoothness (or otherwise) of these curves across multiple values of does informally convey the noisiness of these curves
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The final bullet in Appendix G covers the requirements below.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
41

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https::/neurips. httpp//pp....ppppppccccccccccaaa
Question:
Answer: [Yes]
Justification:
We have reviewed the Code and believe our research conforms to its require-
ments.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed potential positive and negative societal impacts of our work in our impact statement (Appendix A).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We not releasing models or data with this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
42

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Ys]]
Justification: The creators / owners of the three open-source model families used in our experiments [28, 18] credited in the main text, with license details (which have been respected) provided as part of their entries in the Bibliography.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should provided. For popular datasets, saperswithcode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documenta- tion provided alongside the assets?
Answer:
[NA]
Justification:
No new assets are being introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
The interpretability experiment (Section 4.2) did not use any external human subjects. All raters were permanent members of the research group conducting the study, hence no compensations details are provided. The instructions given to raters are provided in Section 4.2. The feature dashboards used by the raters were generated using popular
Justification:
43

---

SAE visualisation library [27] that we have referenced when describing the experimental methodology; representative screenshots are available at the library's 's GitHub page (pro- vided in its Bibliography entry).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: IRB approvals were not required, as no external human subjects were used in the interpretability study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
44


Reviews:
[{"review_id": "TfWYoJfNTH", "reviewer": "Reviewer_Hix2", "summary": "This paper introduces Gated Sparse Autoencoders (Gated SAEs), an improvement over standard sparse autoencoders (SAEs) for decomposing language model activations. The key idea is to separate the tasks of detecting which features are active and estimating their magnitudes, allowing the sparsity penalty to be applied only to feature detection. Through experiments on language models up to 7B parameters, the authors show that Gated SAEs achieve better reconstruction fidelity for a given level of sparsity compared to baseline SAEs, while resolving issues like shrinkage. A human evaluation study finds Gated SAE features to be comparably interpretable to baseline features.", "questions": "- Do you have any insights on how Gated SAEs might scale to even larger language models? Are there any potential limitations as model size increases?\n- Have you explored using Gated SAEs for any downstream mechanistic interpretability tasks beyond the basic reconstruction and interpretability metrics? For example, does the improved reconstruction enable better circuit analysis?\n- The weight tying scheme seems important for computational efficiency. Have you explored any alternative tying schemes? Is there a theoretical justification for why this particular scheme works well?", "limitations": "The authors provide a good discussion of limitations in the conclusion section. They appropriately note that their experiments were limited to specific model families and that further work is needed to evaluate usefulness for downstream interpretability tasks. They also acknowledge the subjective nature of the human interpretability study.\nRegarding potential negative societal impacts, the authors do discuss this briefly in Appendix A. They note that advances in LM interpretability could potentially be misused, but argue that the current work poses minimal short-term risks. While this discussion is somewhat brief, it does address the key points and seems appropriate for the nature of this work.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- A well-motivated architectural modification to SAEs that addresses key limitations\n- Comprehensive empirical evaluation across multiple model sizes and activation sites demonstrating clear improvements over baseline SAEs\n- Careful ablation studies and analysis to understand the source of improvements\n- Human evaluation study to assess interpretability of learned features\n- Thorough discussion of limitations and future work directions", "weaknesses": "- The presentation could be improved in some areas, particularly in explaining some of the technical details and metrics\n- Some of the figures are quite dense and could be made more readable\n- The human evaluation study, while valuable, has a relatively small sample size", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review of our paper and your feedback and questions. We are glad you appreciated our explanation of the motivation behind the gated SAE architectural modification and found our evaluations and ablation studies thorough.\n\nWe agree that the exposition is fairly dense and high context; in large part this was due to the need to stick within the page limit. Regarding the presentation of the metrics, although these are only briefly described in the main body, they are defined at further length in Appendix B. To help bring these definitions to life in the appendix, we will add illustrative examples to each definition. E.g., in the case of loss recovered we hope to make this metric clearer by adding, \"For example, a SAE that increases cross-entropy loss from 2.4 to 2.5 when zero-ablation increases the loss to 5.0 would have a loss recovered of 96%\". We are also considering how to improve Figure 2, which we think could be hard for someone unacquainted with SAEs to decipher at a quick glance. We would value your feedback on these proposals.\n\nRegarding the human evaluation study, we selected our sample size trying to balance the resources consumed by the study against its aim, which was principally to establish that Gated SAEs' superior fidelity over baselines does not come at the expense of interpretability. In other words, we set out to establish that there is no meaningful deterioration in interpretability, rather than trying to show that Gated SAEs are more interpretable. Although the discussion of the study in most of the paper (particularly the abstract, introduction and conclusion) are consistent with this aim, we acknowledge there are a few sentences in section 4 that may give the impression that the study was under-powered (because we fail to show that Gated SAEs are statistically significantly more interpretable), when in fact the main objective of the study – to show that the mean difference in interpretability between Gated and baseline SAEs is not negative by any practically important margin – was achieved, since we found the confidence interval for the difference in interpretability to be [0, 0.26], i.e. excluding significant deterioration in interpretability (even if it doesn't establish an improvement). We will reword key sentences from section 4 that are in danger of giving this incorrect impression, in order to emphasize that it is a meaningful deterioration in interpretability that we set out to measure, and that the study confidently shows that such a deterioration has not occurred.\n\nTurning to your questions:\n- We don't anticipate any issues with scaling Gated SAEs to even larger language models except for the issue acknowledged in the paper that they do require 50% more FLOPs per training step than baseline SAEs, holding width constant. In fact, we didn't actually see a 50% increase in step times during our experiments, because step times were so fast that data loading became a significant bottleneck (at least comparable to computing the loss and its gradient). As we scale to wider SAEs however, we anticipate that we should see the difference in training step time between Gated and baseline SAEs increase to 50%. (Note that our experiments suggest that even if we held compute constant to avoid this 50% increase, Gated SAEs would still outperform baseline SAEs.)\n- We view the use of SAEs for downstream tasks as a major research area for SAEs overall (Gated or otherwise), and would love to see follow-up work on it, as we have noted in our Limitations & Future Work section. Since submitting the paper, we have done some such work ourselves, as have many others (including with Gated SAEs), though we are hesitant to provide references due to the anonymity rules.\n- The key part of the weight tying scheme – tying the two encoder weight matrices – is theoretically motivated on the grounds that similar directions are likely useful for detecting whether a given feature is active and for detecting its magnitude if it is active; we did not come up with alternative ways to tie the encoder weights that seemed theoretically plausible and hence did not investigate them. We did look at whether r_mag is important for weight tying (this is discussed in section 5.1) and found that it seemed slightly beneficial. We also note in section 3.2 and Appendix H that with this weight tying scheme, Gated SAEs become equivalent to vanilla SAEs with the ReLU replaced by a \"JumpReLU\" activation function. From this perspective, one could view Gated SAEs with this weight tying scheme as a way to train a vanilla SAE with a JumpReLU activation function, overcoming the difficulties posed by the jump discontinuity in this activation function.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3yUgLi8AN", "reviewer": "Reviewer_1pM6", "summary": "This work introduces a new technique under mechanistic interpretability's sparse autoencoders. By using a less naive SAE, with a gating mechanism and a little extra computation, the paper shows a decent improvement over the baseline.", "questions": "None", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This work addresses the important issue of interpreting transformer-based LLMs and clearly demonstrates an interesting method. The mechanistic interpretability community will certainly find this work of interest.\n\nThe writing is well written and fairly easy to follow, the results are clearly presented, and all relevant aspects of the method are appropriately ablated\n\nI liked the setup of the internal user study; I think future papers will follow the design of the study closely.\n\nThe work's cited throughout the manuscript are incredibly thorough.", "weaknesses": "While I generally like the paper, I have two primary concerns:\n\n* The architecture and loss are somewhat difficult to understand. I did appreciate the pseudo-code in the appendix, but I feel for readers not familiar with SAEs may have a hard time, especially with the optimization-based design choices of weight tying. Perhaps explaining weight tying later in 3.2 would help. I would especially prefer if a few lines of pseudo code could be added in the main paper, next to figure two.\n* The user study results. I don't mind the small change in means between the method and the baseline, but the explainable AI community has been around for a long time and the shift from studies with a few experts to larger cohorts has been the norm for a while now. Just because there's a rebranding to mechanistic interpretability doesn't mean this field should settle for underpowered studies. Nevertheless, I do find the study setup itself to be well articulated and a very useful starting point for future work in this area.\n\nMinor:\nSome of the design choices (weight-tying, no r_mag, etc) aren't well explained until the ablation where we find they are primarily for optimization. This could be motivated a little earlier, i.e. that the pareto improvement comes from the separation, and not those choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your review and valuable feedback, and are encouraged that you found the paper fairly easy to follow, with results clearly presented and key aspects of the method appropriately ablated.\n\nWe appreciate that the explanation of the architecture and loss function is somewhat dense, and perhaps hard to follow for readers unfamiliar with SAEs. To some extent we are limited in how much we can expand these sections due to the paper page limit, but we would like to take your suggestions on board. Regarding your suggestion to move the description of the weight tying scheme later in 3.2, we propose to move this subsection (i.e. lines 114–127) after the paragraph on Training, and give it its own minor heading (\"Tying weights to improve efficiency\"). Regarding moving a few lines of pseudo-code next to Figure 2, it will be difficult to do this due to space constraints, but instead we will mention in the figure's caption a reference to the pseudo-code in the appendix, and are also trying to improve the figure's legibility in response to another review (which will hopefully help address your concern too, by making it less important to rely on the pseudo-code).\n\nTurning to the interpretability study, we believe in hindsight that – due to some of the wording in section 4.2 – there is a danger that the aims of the study could be misunderstood in a way that leads readers to conclude the study was underpowered. Our primary motivation for introducing the Gated architecture was to improve the trade-off between reconstruction fidelity and sparsity; in this context, the purpose of this interpretability study was to provide reassurance that this improvement in reconstruction fidelity does not come at the expense of a deterioration in interpretability. In other words, our aim was not necessarily to show that Gated SAEs are **more** interpretable than baseline SAEs but rather to show that they aren't less interpretable by a practically relevant margin. We found that even with this study size we were able to confidently exclude a meaningful deterioration in interpretability – the confidence interval for the mean difference was found to be [0, 0.26] – allowing us to confidently state that Gated SAEs are at least similarly interpretable to baseline SAEs. This is indeed the conclusion advertised in the abstract, introduction and conclusion sections of the paper. Nevertheless, we will change some of the sentences in section 4.2 to make the real aim of the study clearer – i.e. giving more prominence to the confidence interval on the mean difference in interpretability as the key result of the study – and less prominence to the question of whether Gated SAEs are more interpretable, for which we agree a bigger sample size would be needed to settle the question.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bBjGyNUCZK", "reviewer": "Reviewer_Qtwo", "summary": "The paper attempts to resolve the issue of feature shrinkage in sparse autoencoders (SAEs) by replacing the SAE ReLU activation function with a gated ReLU unit. The weight-tying scheme they use for the gated unit effectively turns it into a jump ReLU activation function.\nThey train gated SAEs and baseline SAEs on a one layer transformer, Pyhtia-2.8B and Gemma-7B. They find that gated SAEs eliminate systematic shrinkage, and consistently outperform baseline SAEs on the pareto-curve of sparsity, measured by the L0 pseudonorm, and faithfulness, measured by the model loss recovered relative to a zero ablation baseline. \nThey run various additional tests involving variations of the gated and baseline SAE architectures, including combinations of SAE dictionaries with the classic gradient pursuit algorithm for choosing sparse feature coefficients at inference time. They conclude that the Pareto improvement of their gated SAEs over their baseline SAEs is due in part to better feature dictionaries, in addition to better estimated feature coefficients.\nThey compare the subjective interpretability of 150 gated SAE and baseline SAE features in Pythia-2.8B and 192 features in Gemma-7B, using a blinded analysis of activating dataset examples. They find that the features were similarly interpretable.", "questions": "Why are the raw CE loss recovered scores not in the paper? Since it is typically much harder to lower CE loss from e.g. 4.5 to 4.0 than from 1.5 to 1.0, it is difficult to evaluate the quality gap between baseline and gated SAEs, or the quality of the baseline SAEs, without these scores.", "limitations": "All addressed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper attempts to address a substantive practical problem with current SAE training methods.\n\n\nThe paper's proposed new architecture is evaluated extensively, and many detailed additional investigations on the individual effects of various parts of the gated SAE architecture are described in sections 5.1, 5.2 and Appendix D. \n\n\nI find Appendix D interesting in its own right, since it shows quantitative comparisons between SAE methods and the classic gradient pursuit optimization algorithm, as well as mixing SAE feature dictionaries with gradient pursuit for sparse approximation of feature coefficients. I have not encountered such a comparison before. \n\n\nFor the most part, good documentation of all their process is provided, and the writing and presentation are very clear in general.", "weaknesses": "The paper does not really address the concern that gated SAEs may outperform baseline SAEs in part by implicitly widening the definition of what it means for ‘features’ to be represented in the model. As the paper itself notes in Appendix D, though other more powerful sparse coding algorithms greatly outperform SAEs in terms of reconstruction and sparsity, there are concerns that the greater expressivity of these techniques lets them find spurious ‘features’ that would not be accessible to the model’s own internal computations. An SAE can only find features that are represented in the sense that their current values can be read off with a single ReLU probe, while an inference time algorithm or a multi-layer probe may read off ‘feature’ values that the model itself could not possibly access using a single MLP layer. A gated ReLU is far less expressive than an algorithm like gradient pursuit, but more expressive than a ReLU. So to what extent do gated SAEs outperform baseline SAEs merely because they are implicitly working with a more relaxed definition of what it means for a feature to be represented in the model? Figure 6 in Appendix D incidentally investigates this somewhat, since it attempts to compare the quality of gated vs. baseline dictionaries independent of their coefficients. However, the results there seem inconsistent, with smaller performance gaps and baseline SAEs outperforming gated SAEs at higher L0. I think this issue of the representational power of the probe used is pretty central for contextualizing the results, and should at least have been discussed.\n\nThroughout the paper, the authors present reconstruction scores for SAEs in terms of the fraction of model loss recovered compared to a zero-ablation baseline. I think this metric obscures vital information. Lowering CE loss from e.g. 4.5 to 4.0 is typically much easier than lowering it from 1.5 to 1.0. Thus, the same difference in loss recovered between two SAEs can correspond to very different gaps in SAE quality. Without the raw CE scores, there is no direct way to infer how large the gap is quantitatively. At minimum, these raw CE scores should be in the supplementals. Better yet, the recovered performance could additionally be reported in terms of the compute required to train a model with the same CE score, as suggested in https://arxiv.org/abs/2406.04093.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper, we are heartened that you agree that Gated SAEs attempt to address a substantive practical problem with current SAE training methods and that you find our evaluations and ablations extensive.\n\nRegarding your questions about the raw (unspliced model's) cross-entropy loss, we agree that this is useful information that we had omitted from the paper and thank you for drawing this to our attention. To remedy this, we will add a table in the appendix listing original (\"raw\") CE losses and the CE losses incurred when zero ablating MLP, attention and residual stream layers for the layers and models investigated in the paper, as shown in the PDF accompanying the main rebuttal, and we will reference this table from the captions for the fidelity-vs-sparsity figures presented in the main body of the paper.\n\nTurning to the other limitation you mention in the weaknesses section, we agree that it is possible for a more powerful dictionary learning method to attain better fidelity by expanding what it means for a feature to be represented in a model. However, we do not believe that Gated SAEs – particularly with the weight-tying scheme proposed in the paper – fall into this trap.\n\nMaking explicit the connection between SAEs and probing that you alluded to, a single row of a vanilla (ReLU) SAE encoder corresponds to a pair of linear models: a linear classifier that determines when a feature is active and a linear regressor that determines the magnitude of an active feature. A single row of a Gated SAE encoder also effectively corresponds to the same pair of a linear classifier and a linear regressor, performing the same respective functions. The key difference between vanilla and Gated SAEs is that whereas vanilla SAEs insist that both the weights and biases of the classifier and regressor be exactly identical, the Gated SAE allows the biases to differ. (The weights are still the same, up to a rescaling, under the weight tying scheme introduced in the paper.)\n\nAs explained in section 3.1, our contention is that even in the ideal case of a perfectly linearly represented feature, it is suboptimal to use the same bias in the classifier and regressor, i.e. to use the same bias determine whether a feature is active and measure its magnitude – because this requires unnecessarily trading off false positives in the classifier (which occurs when the bias is too low) against shrinkage in the regressor (which occurs when the bias is too high). From this perspective, Gated SAEs are removing an undesirable inductive bias in vanilla SAEs, while still using the simplest class of models (i.e. thresholded affine transformations) to detect and measure feature activations.  Perhaps to put it another way, any feature that is detectable and measurable by a Gated SAE should be similarly detectable and measurable by a vanilla SAE, albeit with a worse bias-variance trade-off.\n\nThe previous paragraph notwithstanding, it is in part to assuage fears that Gated SAEs may be somehow \"cheating\" in order to obtain better fidelity that we performed the interpretability study described in section 4.2, which finds reassuringly that Gated SAEs are comparably interpretable to vanilla SAEs. Nowhere in our training algorithm is there any term that directly trains features to be human interpretable, and so the fact that they remain human interpretable anyway is a good sign that we have not over-optimized for reconstruction / sparsity at the cost of the thing we actually care about.\n\nNevertheless, we admit in the limitations section that \"while we have confirmed that Gated SAE features are comparably interpretable to baseline SAE features, it does not necessarily follow that Gated SAE decompositions are equally useful for mechanistic interpretability\", i.e. that our evaluations do not conclusively show that Gated SAEs are comparably \"useful\" as vanilla SAEs in a practical sense, and agree that this is an important open question that needs resolving.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aywyjC2JCN", "reviewer": "Reviewer_58d7", "summary": "This work proposed a Gated Sparse Autoencoder (Gated SAE) to mitigate the standard SAEs' biases, such as shrinkage, which systematically underestimate the feature activations from SAEs. The key difference between Gated SAE and SAE is that the Gated SAE separates affine transformations within the encoder in order to decide which dictionary elements to use in a reconstruction loss, and estimate the coefficients of active elements, although with the 50% more computing required to achieve. Comprehensive experiments are conducted to compare and verify how good the Gated SAE is to standard SAE, including a blinded human study to rate and compare the interpretability of randomly sampled Gated and baseline SAE features.", "questions": "- As mentioned in the weakness above, the interpretability analysis was conducted via human rating for randomly sampled feature visualizations. However, those visualizations are not included in the main body or appendix. It will be better to help understand how well the sampled features are when comparing Gated SAE and SAE.\n- Although the Gated SAE has good performance on the loss recovered (fidelity) and relative reconstruction bias, it is still not clear whether features from Gated SAE are better than SAE under downstream tasks. It will make this work very solid if some analysis of mini downstream applications can be conducted, e.g., IOI task, greater-than task, etc.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new architecture of SAE inspired by GRU is proposed to include a gate mechanism to mitigate shrinkage bias\n- Comprehensive quantitative experiments including ablation studies to evaluate the proposed Gated SAE compared to SAEs\n- A human evaluation to rate randomly sampled features from Gated SAE and SAE", "weaknesses": "- It is not very straightforward to understand how well the features from Gated SAE are compared to SAE based on Figure 4. Some case studies based on the open-source SAE visualizer library [1] are required to help better understand this.\n- It will be better to see more case studies on downstream tasks to compare Gated SAE and SAE, e.g., automatic circuit detection [2]\n\n[1] C. McDougall. SAE Visualizer, 2024. https://github.com/callummcdougall/sae_vis\n\n[2] Huben, Robert, et al. \"Sparse Autoencoders Find Highly Interpretable Features in Language Models.\" The Twelfth International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your thoughtful review and helpful feedback, and are pleased you found our experiments comparing Gated and baseline SAEs comprehensive.\n\nWe indeed used sae-vis, which you referenced in your review, to produce the visualization used in the interpretability study, citing this library section 4.2 where we explain the interpretability methodology. We agree however that it would be helpful to share screenshots of these dashboards, and will add a new appendix to the paper including some examples (from both Gated and baseline SAEs). We do note though that – as shown in Figure 4 – the differences in interpretability between Gated and baseline SAEs are slight (if any), and it is unlikely that these dashboards will give much insight into systematic differences in the features found by the respective architectures. Nevertheless, by including these dashboards we hope to better bring to life the information raters had to go on when rating the interpretability of the features.\n\nRegarding analyzing the performance of Gated SAEs on downstream uses, we agree this would be excellent evidence for the utility of Gated SAEs (or SAEs in general). However, as we note in our Limitations & Future Work section, the science of evaluating SAEs in a manner that correlates well with downstream uses (which are still uncertain) and yet is scalable and objective is in its early stages. To take the paper referenced in the review (Huben at al., 2023) as an example, while it makes a valuable contribution to furthering the study of SAE evaluations, the metrics introduced there are either better for resolving differences between SAEs as a whole vs other decompositions like PCA (i.e. Figure 4, where the difference between SAEs and e.g. PCA is stark, but the differences between SAEs are small), boil down to metrics that were indeed considered in our interpretability study (i.e. feature monosemanticity and direct effects on logits) or involve performing a bespoke study of one circuit with one SAE, which makes it difficult to scale and compare several SAEs objectively. To be clear, our point is not to undervalue the contribution of Huben at al., but to explain why we felt it would be unreasonable to try to combine in a single paper both the architectural innovation of Gated SAEs with simultaneously making a meaningful advance in the science of evaluating SAEs with different architectures. In this light, we do believe that our paper – by introducing a novel SAE architecture and showing that it either matches or exceeds baseline ReLU SAEs on standard SAE evaluation metrics – provides a valuable contribution even without these additional evaluations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the area chair and all our reviewers for taking the time to read our paper and for their insightful comments and suggestions.

We are encouraged by all four reviewers recommending that our paper be accepted, with reviewer Qtwo recognizing that our paper “attempts to address a substantive practical problem with current SAE training methods”. We perceive general agreement among the reviewers that the Gated SAE architecture is a useful and well-motivated innovation, and that the evaluations and ablations presented in the paper are thorough and comprehensive. We believe Gated SAEs significantly improve the quality of one of the most important tools in mechanistic interpretability – sparse autoencoders – and that this paper therefore deserves acceptance at the conference.

Two common criticisms in the reviewer feedback were: (1) that the presentation is dense in places and (2) that the sample size for the interpretability study should have been higher. Regarding (1), we have tried to address the specific concerns raised by the reviewers whilst satisfying page limit constraints, however we would welcome further feedback on our proposals.

Regarding criticism (2), we have realized that some of the text in Section 4.2 may give the incorrect impression we set out to show that Gated SAEs are more interpretable than baseline SAEs, when in fact the purpose of the study was to show that Gated SAEs don't sacrifice interpretability to achieve better reconstruction fidelity. We will therefore re-word the offending sentences in Section 4.2, to emphasize our main result that the mean difference in interpretability scores between Gated and baseline SAEs has a confidence interval of [0, 0.26], clearly excluding a practically relevant deterioration in interpretability.

Although we were (pleasantly) surprised to find the lower end of the confidence interval to be close to zero, and it is plausible that a bigger study might find that Gated SAEs are statistically significantly *more* interpretable than baseline SAEs, we are doubtful that such a finding would have much practical relevance. The effect size is likely to be fairly low, and it is uncertain whether a small increase in subjective interpretability would translate into a practically relevant increase in the usefulness of SAEs for downstream tasks, with the latter being the (still hard-to-measure) outcome the field really cares about.

Rather than focusing on subjective interpretability, multiple reviewers note that it would be better to see whether Gated SAEs are more useful for downstream tasks. We agree! Unfortunately, this is an area of weakness for the field as a whole, and as such we view this as an avenue for future work which would likely be its own paper.

In the accompanying PDF we include a new figure illustrating the dashboards used for the interpretability study, addressing feedback from reviewer 58d7, and a new table listing raw and zero-ablation CE losses to help put our loss recovered results into context, addressing feedback from reviewer Qtwo. Note that the dashboard figure will be printed landscape in the final paper (as shown in the PDF), as this is the only way we could make the dashboard legible due to its wide aspect ratio. The table will be in portrait format in the final paper.

Meta-review: There was consensus among all reviewers that this paper belongs at the conference. The work is well-motivated, tackles the important topic of LLM interpretability, and provides extensive experimentation and ablation. Reviewers expressed concerns about the denseness of the paper, which I anticipate the authors will be able to redemy to some extent with the extra page for the camera-ready version.
Decision: Accept (poster)