Scaling and evaluating sparse autoencoders
Authors: Leo Gao
Tom Dupre la Tour
Henk Tillman
Gabriel Goh
Rajan Troll
Alec Radford
Ilya Sutskever
Jan Leike
Jeffrey Wu
Venue: ICLR 2025 Oral (2025)
Keywords: interpretability
sparse autoencoders
superposition
scaling laws
Primary area: interpretability and explainable AI

Abstract: Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many concepts, autoencoders need to be very large to recover all relevant features. However, studying the properties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents. We propose using k-sparse autoencoders [Makhzani and Frey, 2013] to directly control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales we tried. Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity. We also introduce several new metrics for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation patterns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size. To demonstrate the scalability of our approach, we train a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens. We release training code and autoencoders for open-source models, as well as a visualizer.

Published as a conference paper at ICLR 2025
SCALING AND EVALUATING SPARSE AUTOENCODERS
Leo Gao" Tom Dupré la Tour Henk Tillman
Gabriel Goh Rajan Troll
Alec Radford Rya Sutskever OpenAl ig@openai..com com
Jan Leike
Jeffrey Wu
ABSTRACT
Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer. Since language models learn many concepts, autoencoders need to very large recover all relevant features. However, studying the proper- ties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents. We propose using k-sparse autoencoders (Makhzani & Frey 2013) to directly control sparsity, simplifying tuning and improving the reconntruction-sparitty frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales tried. Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity. We also introduce several metries for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation pat- terns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size. To demonstrate the scalability of our approach, we train 16 million latent autoencoder on GPT-4 activations for 40 billion tokens. We release code and autoencoders for open open-source source models, as well as a visualizer
INTRODUCTION
Sparse autoencoders (SAEs) have shown great promise for finding features (Cunningham et al. 2023 Bricken al. 2023 Templeton al. 2024 Goh 2016 and circuits (Marks et al. (024) in language models. Unfortunately. they are difficult to train due to their extreme sparsity, so prior work has primarily focused on training relatively small sparse autoencoders on small language models.
We develop a state-of-the-un methodology to reliably train extremely wide and sparse autoencoders with very few dead latents on the activations of any language model. We systematically study the scaling laws with respect to sparsity, autoencoder size, and language model size. To demonstrate that our methodology can scale reliably, we train a 16 million latent autoencoder on GPT-4 (OpenAl 2023) residual stream activations.
Because improving reconstruction and sparsity is not the ultimate objective of sparse autoencoders, we also explore better methods for quantifying autoencoder quality. We study quantities corresponding to: whether certain hypothesized features were recovered, whether downstream effects are sparse, and whether features be explained with both high precision and recall.
Our contributions:
1. In Section 2.
we describe state-of-the-art recipe for training sparse autoencoders.
2. In Section 3
we demonstrate clean scaling laws and scale to large numbers of latents.
3. In section 4
we introduce metrics of latent quality and find larger sparse autoencoders are generally better according these metrics.
We also release code a full suite of GPT-2 small autoencoders, and la feature visualizer for GPT-2 small autoencoders and the 16 million latent GPT-4 autoencoder.
Primary Contributor. Correspondence to ig@openai.com
i CCore Research Contributor. This project was conducted by the Superalignment Interpretability team. Author contributions statement in Appendix I

---

Published as a conference paper at ICLR 2025
L(C)
L(N, k)
0.6
0.09 + 0.056 -0,084
512
6 10
0.5
Number of Latents
Normalized aamallzzdd MSE
Normalized MSE
10c
0.4
4x
128
100
0.3
3 10-1
64
104
32
10
10
10
10*
10a
FLOPs (as fraction of GPT-4 pretraining)
Number of Latents
Figure I: Scaling laws for TopK autoencoders trained on GPT-4 activations. (Left) Optimal loss for a fixed compute budget. (Right) Joint scaling law of loss convergence with fixed number of total latents n and fixed sparsity (number of active latents) k. Details section
METHODS
2.1
SETUP
Inputs: We train autoencoders the residual streams of both GPT-2 small (Radford et al.) 2019) and models from series of models of increasing size, sharing GPT-4 architecture and training setup, We choose layer near the end of the network, which should contain many features without being specialized for next-token predictions (see subsecttnn F.] for more discussion). Specifically, layer 8 2 of the way) for GPT-2 small. We context length of 64 tokens for
including GPT-4 itself (OpenAl 2023 !
use layer of the way into the network for GPT-4 series
models, and all experiments. We subtract the mean over the dmodel dimension and normalize to all inputs to unit norm, prior to passing to the autoencoder (or computing reconstruction errors).
After training, we evaluate autoencoders on sparsity Lo. and reconstruction mean- squared error (MSE). We report a normalized version of all MSE numbers, where we divide by baseline reconstruction eтror of always predicting the mean activations.
Evaluation:
Hyperparameters: To simplify analysis, we do consider learning rate warmup or decay unless otherwise noted. We sweep learning rates at small scales and extrapolate the trend of optimal learning rates for large scale. See SeeAAppendix for other optimization details.
2.2 BASELINE: RELU AUTOENCODERS
For an input vector E Rd from the residual stream, and latent dimensions, we use baseline ReLU autoencoders from (Bricken et al. 2023). The encoder and decoder are defined by:
ReLU - bpec) + beae)
(1)
a Waec + bpre
with Wene E RRuxd bene E IR.. Waec E -Rdxx, and bpre E Rdd The training loss is defined by - || 1ee--1pp &112 + A||=/|1. where ||0 [lc- ail
[2 is the reconstruction MSE, ||||| is an L1 penalty promoting sparsity latent activations 2, and À hyperparameter that needs to be tuned.
2.3 TOPK ACTIVATION FUNCTION
We use k-sparse autoencoder (Makhzani & Frey 2013), which directly controls the number of active latents by using an activation function (TopK) that only keeps the largest latents, zeroing the rest. The encoder is thus defined as:
= TopK W bpre))
(2)
All presented results either have qualitatively similar results both GPT-2 small and GPT-4 models, only model class.
2

---

Published as a conference paper at ICLR 2025
6 10
better
6
Normalized MSE
5 x 10
Nomalized MSE
5001
10-1
ReLU
10
ProLU STE
ProLU
Gated
Gated
TopK (ours)
TopK (ours)
10"
10a
10*
10a
Sparsity (LO)
Number of Latents
(a) At a fixed number latents (n = 32768). TopK has a better reconstruction-spaate sparsity trade off than ReLU and ProLU, and is comparable to Gated.
(b) At fixed sparsity level (La 128), scaling laws are steeper for TopK than ReLU
Figure 2: Comparison between TopK and other activation functions.
and the decoder is unchanged. The training loss is simply || -
Using k- sparse autoencoders has number of benefits:
It removes the need for the L1 penalty. L1 is an imperfect approximation of Lo, and introduces a bias of shrinking positive activations toward zero (subsection 5.1).
It enables setting the Lo directly, as opposed tuning an L1 coefficient A, enabling simpler model comparison and rapid iteration. It can also be used in combination with arbitrary activation functions
It empirically outperforms baseline ReLU autoencoders on the sparsity-reoonstruciion frontier (Figure 2), and this gap increases with scale (Figure 2p).
It increases monosemanticity of random activating examples by effectively clamping small activations to zero (subsection 4.3).
2.4 PREVENTING DEAD LATENTS
Dead latents pose another significant difficulty in autoencoder training. In larger autoencoders, an increasingly large proportion of latents stop activating entirely at some point in training. For example, Templeton et al. 2024) train a 34 million latent autoencoder with only 12 million alive latents, and in our ablations find up 90% dead latents when mitigations are applied (Figure 13). This results in substantially worse MSE and makes training computationally wasteful. We find two important ingredients for preventing dead latents: we initialize the encoder the transpose of the decoder F1 and we use an auxiliary loss that models reconstruction error using the top-kaux dead latents (see (sbsectton A.2 for more details). Using these techniques, even in our largest (16 million latent) autoencoder only 7% of latents are dead.
SCALING LAWS
3.1 NUMBER LATENTS
Due to the broad capabilities of frontier models such as GPT-4, we hypothesize that faithfully representing model state will require large numbers of sparse features. We consider two primary approaches to choose autoencoder size and token budget:
"In our code, we also apply a ReLU to guarantee activations non-negative. However, the training curves are indistinguishable, as the & largest activations are almost always positive for reasonable choices of k.
Non-TTpK canmot set precise Lo, so we interpolated using piecewise linear function log-log space. *We follow Templeton et al. 2204),, and consider a latent dead if it has not activated in 10 million tokens "This strategy is also presented concurrent work (Conerly et al. 2024
3

---

Published as a conference paper at ICLR 2025
10c
Subject model pretraining compute
10-1
10
215
Normalized MSE
5 10-1
Number of Latents
Norma Lenaalizzddd MSE
4x
10
230205120
x
10
2
4 10
10
18.28
1003
10th
20
2
10
10a
Learning 1e-4)
Number of Latents
Figure 3: Varying the learning rate jointly with the number of latents. Number of tokens to vergence shown above each point.
Figure 4: Larger subject models in the GPT-4 family require more latents to get to the same MSE (k: 32).
3.1.1 TRAINING TO COMPUTE-MSE FRONTIER (L(C))
Firstly, following Lindsey et al. (2024), we train autoencoders to the optimal MSE given the available compute, disregarding convergence. This method was introduced for pre- pre-training language models (Kaplan et al. 2020, Hoffmann et al. 2022). We find that MSE follows a power law L(C)) of compute, though the smallest models are off trend (Figure I).
However, latents are the important artifact training (not reconstruction predictions), whereas for language models typically care only about token predictions. Comparing MSE across different is thus not a fair comparison - the latents have a looser information bottleneck with larger ni, so lower MSE is more easily achieved. Thus, this approach is arguably unprincipled for autoencoder training.
3.1.2 TRAINING TO CONVERGENCE (L(N))
We also look at training autoencoders to convergence (within some e). This gives bound on the best possible reconstruction achievable by our training method if we disregard compute efficiency. In practice, we would idcally train to some intermediate token budget between L(N) and L(C").
We find that the largest learning rate that converges scales with 1/ v (Figure 3). optimal learning rate for L(N) is about four times smaller than the optimal learning rate for L(C).
We also find that the
We find that the number of tokens to convergence increases as approximately (n for GPT-2 small and ( for GPT-4 (Figure 9). This must break at some point if token budget continues to increase sublinearly, the number of tokens each latent receives gradient signal on would approach zero
3.1.3 IRREDUCIBLE LOSS
Scaling laws sometimes include an irreducible loss term e, such that y = a + Henighan et al. 2020). We find that including an irreducible loss term substantially improves the quality of our fits both LCC) and T(N).
It was initially not clear to that there should be nonzero irreducible loss. One possibility is that there are other kinds structures the activations. the extreme case, unstructured noise in the activations is substantially harder to model and would have exponent close to zero (Appendix G). Existence of some unstructured noise would explain a bend in the power law.
*One slight complication is that in the infinite width limit, TopK autoencoders with our initialization scheme are actually optimal init using our init scheme (subsection A. this allows for exponent very slightly less than however, this happens very slowly with unlikely be major factor realistic scales.
4

---

Published as a conference paper at ICLR 2025
better
ProLU ReLU
Detta cross-entrop
Delta cross-entropy
ProLU
Gated
TopK (ours)
ReLU
10"1
ProLU ReLU
10
ProLU STE
Gated
TopK (ours)
better
10a
10-
10
2 10-1 x
Sparsity (LO)
Normalized MSE
(a) For a fixed number of latents (n 2 131072). the downstream downstreann-oss/pprsityy trade-off is better for TopK autoencoders than for other activa- tion functions.
(b) For a fixed sparsity level (Lu = 128), given MSE level leads to a lower downstream loss for TopK autoencoders than for other activation func- tions.
Figure 5: Comparison between TopK and other activation functions on downstream loss. Comparisons done for GPT-2 small, see Figure for GPT-4.
3.1.4 JOINTLY FITTING SPARSITY (L(N
We find that MSE follows joint scaling law along the number of latents and the sparsity level k (Figure Ib). Because reconstruction becomes trivial as & approaches dmodet, this scaling law only holds for the small & regime. Our joint scaling law fit GPT-4 autoencoders is:
L(n, k) cxp(a + 3& log(k) + 3, log(n) + log(k) log(n)) + exp(s 77 (log(k))
(3)
with a = - 0.50, 3 0.26, 8, = 0.017,,1 -0. -0.042, < - 1.322 and 17 000885. We can see that y is negative, which means that the scaling law L(NN gets steeper as i increases. 17 negative too, which means that the irreducible loss decreases with k.
3.2 SUBJECT MODEL SIZE (NV)
Since language models are likely keep growing in size, we would also like to understand how sparse autoencoders scale with the subject model. We find that if we hold & constant, larger subject models require larger autoencoders to achieve the same MSE, and the exponent worse (Figure 4).
EVALUATION
We demonstrated in|section that our larger autoencoders scale well terms MSE and sparsity (see also a comparison of activation functions in subsection 5.2). However, the end goal of autoencoders not to improve the pparsit-reconssruucte to find features useful for applications, such mechanistic interpretability. Therefore, we measure autoencoder quality with the following metrics:
frontier (which degencrates in the limi D. but rather
Downstream loss: How good is the language model loss if the residual stream latent
replaced with the autoencoder reconstruction of that latent? (subsection 4.1)
2. Probe loss: Do autoencoders recover features that we believe they might have? (subsec- [ion 4.2)
3. Explainability: Are there simple explanations that are both necessary and sufficient for the activation of the autoencoder latent? (subsection 4.3)
4. Ablation sparsity: Does ablating individual latents have a sparse effect on downstream logits? (subsection 4.5)
'Improving the cennssrructionspprsiii frontier is not always strictly better. An infinitely wide maximally sparse (A: = 1) autoencnder perfectly reconstruct by assigning latents densely in RRd while heing completely structure-less and uninteresting.

---

Published as a conference paper at ICLR 2025
2/L 21-2
0.555
0.138
01136
Nuntoe Literte
0192
Number Latertz
B192
0.33
32768
32788
111017
0.495
11017
0.490
(a) Probe loss
(b) Logit diff sparsity
The probe loss and logit diff metrics as a function of number total latents n and active latents k for GPT-2 small autoencoders. More total latents (higher generally improves all metries (yellow = better). Both metrics are worse at Lo 512, regime in which solutions are dense (see subsection E.5).
Figure 6:
These metrics provide evidence that autoencoders generally get better when the number of total latents increases. The impact of the number of active latents Lo is more complicated. Increasing Lo makes explanations based on token patterns worse, but makes probe loss and ablation sparsity better. All of these trends also break when Lo gets close to dmnode. a regime in which latents also become quite dense (see subsection E.5 for detailed discussion).
4.1 DOWNSTREAM LOSS
An autoencoder with non-zero reconstruction error may not succeed at modeling the features most relevant for behavior (Braun et al. 2024). >. To measure whether we model features relevant to language
modeling, follow prior work (Bills al. 2023 Cunningham et al. 2023 Bricken al. 2023
(KL) divergence and cross cross-entrope entropy loss g In both cases, test autoencoder by replacing the residual stream by the reconstructed value during the forward pass, and seeing how it affects downstream predictions. We find that k-sparse autoencoders improve more on downstream loss than on MSE over prior methods (Figure 5/)). We also find that MSE has clean power law relationship with both KL divergence, and difference of cross entropy loss (Figure 5D), when keeping sparsity Lo fixed and only varying autoencoder size.
Braun et al.] 2024) and consider downstream Kullback-Leibe
One additional issue is that raw loss numbers alone are difficult to interpret we would like to know how good is in an absolute sense. Prior work (Bricken al. 2023 Rajamanoharan al.) 2024) use the loss of ablating activations to zero as a baseline and report the fraction of loss recovered from that baseline. However, because ablating the residual stream to zero causes very high downstream loss, this means that even very poorly explaining the behavior can result in high scores [
Instead, we believe more natural metric is to consider the relative amount of pretraining compute needed to train language model of comparable downstream loss. For example, when our 16 million latent autoencoder is substituted into GPT-4, we get a language modeling loss corresponding to 10% of the pretraining compute of GPT4
4.2 RECOVERING KNOWN FEATURES WITH ID PROBES
we expect that a specific feature (e.g sentiment, language identification) should be discovered by high quality autoencoder, then one metric of autoencoder quality is to check whether these features are present. Based on this intuition, we curated set of 61 binary classification datasets (details in
Because a perfect reconstruction would lead non-zero cross-entropy loss, we actually consider difference to the perfect--utoenecoder cross-entropy ("delta cross-entropy"). For completeness, the zero-ablation fidelity metric of 16M autoencoder 98.2%.

---

Published as a conference paper at ICLR 2025
Table 1p. For each task, train Id logistic probe each latent using the Newton-Raphson method to predict the task, and record the best cross entropy loss (across latents) That is:
min E I- (wz; + b) - (1 (1-y) log (1 (wzz + b)) 5,10,6
where 2; is the ith pre-activation latent, and y is a binary label. We average the loss across tasks.
Results on GPT-2 small are shown in Figuue 6a We find that probe score increases and then decreases as k increases. We find that TopK generally achieves better probe scores than ReLU (Figure 23), and both are substantially better than when using directly residual stream channels. See Figure 32 for results on several GPT-4 autoencoders: we observe that this metric improves throughout training, despite there being no supervised training signal; and we find that it beats baseline using channels of the residual stream. See FFigure 33 for scores broken down by component.
This metric has the advantage that computationally cheap. However, it also has major limitation, which is that it leans on strong assumptions about what kinds of features are natural.
4.3 FINDING SIMPLE EXPLANATIONS FOR FEATURES
Anecdotally, our autoencoders find many features that have quickly recognizable patterns that suggest explanations when viewing random activations (subsection E.I). However, this can create an "illusion" of interpretability (Bolukbasi et al. 2021), where explanations are overly broad, and thus have good recall but poor precision. For example, Bills et al. (2023) propose an automated interpretability score which disproportionately depends on recall. They find a feature activating at the end of the phrase "don't 't stop" or "can't stop", but explanation activating on all instances of "stop" achieves a high interpretability score. As we scale autoencoders and the features get sparser and more specific, this kind of failure becomes more severe.
Unfortunately, precision is extremely expensive to evaluate when the simulations are using GPT-4 as in Bills et al. (2023). As initial exploration, we focus on an improved version of Neuron Graph (N2G) (Foote et al. 2023). a substantially less expressive but much cheaper method that outputs explanations in the form of collections of n-grams with wildcards. In the future, would like to explore ways to make it more tractable to approximate precision for arbitrary English explanations.
To construct N2G explanation, we start with some sequences that activate the latent. For each one, we find the shortest suffix that still activates the latent [1] We then check whether any position in the n n-gram can be replaced by a padding token, to insert wildcard tokens. We also check whether the explanation should be dependent on absolute position by checking whether inserting a padding token at the beginning matters. We use random sample of up to 16 nonzero activations to build the graph, and another 16 as true positives for computing recall.
Results for GPT-2 small are found in Figure and 25b Note that dense token patterns are trivial to explain, thus 2048, & = 512 latents are easy to explain on average since many latents activate extremely densely (see subsection 12 general, autoencoders with more total latents and fewer active latents are easiest to model with N2G.
We also obtain evidence that TopK models have fewer spurious positive activations thanReLU models. N2G explanations have significantly better recall (>1.5x) and only slightly worse precision (>0.9x) for TopK models with the same n (resulting in better Fl scores) and similar Lo (Figure 24).
4.4 EXPLANATION RECONSTRUCTION
When our goal is for model's 's activations to be interpretable, question can ask is: how much performance do we sacrifice we use only the parts of the model that we can interpret?
This is similar to the approach of Gurnee et al. 2023 but always using & = 1, and regressing on autoencoder latents instead of neurons.
11 with least half the original activation strength
This highlights an issue with precision/recall metrics, which care only about binarized values. We also propose more expensive metric which uses simulated values (subsection 4.4 and addresses this issue.
7

---

Published as a conference paper at ICLR 2025
Our downstream loss metric measures how much of the performance we're capturing (but features could be uninterpretable). and our explanation based metric measures how monosemantic our features (but they might not explain most of the model). This suggests combining downstream loss and explanation metrics, by using our explanations to simulate autoencoder latents, and then checking downstream loss after decoding. This metric also has the advantage that it values both recall and precision in a way that is principled, and also values recall more for latents that activate more densely,
We tried this with N2G explanations. N2G produces simulated value based on the node in the trie, but we scale this value to minimize variance explained. Specifically, we compute E' Elsa]/ E[s²), where the simulated value and the true value, and we estimate this quantity over a training set of tokens. Results for GPT-2 are shown in Figure 7
Downstream loss after abiation
10
6x100
10
10
10a
Sparsity (LD)
zeroed
3e pre blas (latents zeroec)
n=2048
unigram ae latents simu simulated with n2g
n=8192
bigram se reconstr.ction
n=32768
clean
Figure 7: Downstream loss on GPT-2 with various residual stream interventions layer 8. When every latent in the autoencoder is explained with N2G and the model is run using the N2G simulations instead of the real latent activations (purple), we find that larger and sparser autoencoders have better downstream loss. In the best case, running the model with N2G simulated latents performs better than a baseline bigram language modeling. showing some nontrivial degree of explanation. However, most of the original autoencoder's 's reconstruction (red) remains unexplained by N2G explanations.
4.5 SPARSITY OF ABLATION EFFECTS
If the underlying computations learned by a language model are sparse, one hypothesis is that natural features are not only sparse in terms of activations, but also in terms of downstream effects (Olah al. 2024). Anecdotally, we observed that ablation effects often are interpretable (see our visualizer). Therefore, we developed metric to measure the sparsity of downstream effects on the output logits.
At a particular token index, we obtain the latents the residual stream, and proceed to ablate each autoencoder latent one by one, and compare the resulting logits before and after ablation. This process leads V logit differences per ablation and affected token, where is the size of the token vocabulary. Because constant difference at every logit does not affect the post-softmax probabilities, we subtract at each token the median logit difference value. Finally, we concatenate these vectors together across some set of T future tokens (at the ablated index later) to obtain vector of V total numbers. We then measure the sparsity of this vector via ( )2 which corresponds to an "effective number of vocab tokens affected"'. We normalize by V 7 to have a fraction between 0 and with smaller values corresponding sparser effects.
We perform this for various autoencoders trained on the post-MLP residual stream at layer in GPT-2 small, with T = 16. Results shown in Figure 6b) Promisingly, models trained with larger k have latents with sparser effects. However, the trend reverses at & = 512, indicating that as is approaches dmodel 768, the autoencoder learns latents with less interpretable effects. Note that latents
8

---

Published as a conference paper at ICLR 2025
1.4
RaLJ
Topk
Detta rooe-enneey
1.0
20
10
0.4
(refined)
10 (twline)
0.6
Difference
Figure 8: Latent activations be refined to improve reconstruction from frozen set of latents. For ReLU autoencoders, the refinement biased toward positive values, consistent with compensating for the shrinkage caused by the L1 penalty. For TopK autoencoders, the refinement is not biased, and also smaller in magnitude. The refinement only closes part of the gap between ReLU and TopK.
sparse in an absolute sense, having (2) of 10- 10-1 whereas ablating residual stream channels gives 60% (slightly better than the theoretical value of for random vectors).
5
UNDERSTANDING THE TOPK ACTIVATION FUNCTION
5.1 TOPK PREVENTS ACTIVATION SHRINKAGE
major drawback of the L,1 penalty is that tends to shrink all activations toward zero [Tbsshirani 1996). Our proposed TopK activation function prevents activation shrinkage, it entirely removes the need for an L, penalty. To empirically measure the magnitude of activation shrinkage, we consider whether different (and potentially larger) activations would result in better reconstruction given fixed decoder. We first run the encoder to obtain a set of activated latents, save the sparsity mask, and then optimize only the nonzero values minimize MSE This refinement method has been proposed multiple times such as in k-SVD (Aharon et al. (206).. the relaxed Lasso (Meinshausen 2007), or ITT (Maleki) 2009).. We solve for the optimal activations with positivity constraint using projected gradient descent.
This refinement procedure tends to increase activations in ReLU models on average, but not in TopK models (Figure 81), which indicates that TopK is not impacted by activation shrinkage. The magnitude of the refinement is also smaller for TopK models than for ReLU models. In both ReLU and TopK models, the refinement procedure noticeably improves the reconstruction MSE Figuue 8p), and the downstream next-toke token wew-tttookkneeeeeee peediictteiitiionnn cross cross-entropy (Figure 8b). However, this refinement only closes part of the gap between ReLU and TopK models.
5.2 COMPARISON WITH OTHER ACTIVATION FUNCTIONS
Other recent works on sparse autoencoders have proposed different ways to address the L, activation shrinkage, and Pareto improve the Lo-MSE frontier (Wright & Sharkey 2024 Taggart, [2024] Rajamanoharan et al. 2024). Wrrigtt & Sharkey (0024) propose to fine-tune a scaling parameter per latent, to correct for the L1 activation shrinkage. In Gated sparse autoencoders (Rajamanoharan jet al. 2024), the selection which latents are active is separate from the estimation of the activation magnitudes. This separation allows autoencoders to better estimate the activation magnitude, and avoid the L, activation shrinkage. Another approach is to replace the ReLU activation function with ProL.U (Taggart 2024) (also known TRec (Konda et al.) 2014), JumpRel.U (Erichson et al. 2019)), which sets all values below a positive threshold to zero (n( = r (x2-0). Because parameter& 0 non-differentiable, it requires a approximate gradient such as ReLU equivalent (ProLU-ReLU) or a trraighttthoouge estimator (ProLU-STE) (Taggart 2024).
We compared these different approaches in terms of reconstruction MSE, number of active latents Lo. and downstream cross- cooss-yy loss (Figure 2 and 5). We find that they significantly improve the reconstruction-span sparssity Pareto frontier, with TopK having the best performance overall.
unlike the "inference-time optimization" procedure in Nanda et al. 2024

---

Published as a conference paper at ICLR 2025
6 LIMITATIONS AND FUTURE DIRECTIONS
We believe many improvements can be made to our autoencoders.
TopK forces every token to use exactly k latents, which is likely suboptimal. Ideally would constrain [o] rather than Lo-
The optimization can likely be greatly improved, for example with learning rate scheduling 14 better optimizers, and better aux losses for preventing dead latents.
Much more could done to understand what metries best track relevance downstream applications, and to study those applications themselves. Applications include: finding vectors for steering behavior, doing anomaly detection, identifying circuits, and more.
We're excited about work in the direction of combining MoE (Shazeer et al. 2017) and au- toencoders, which would substantially improve the asymptotic cost of autoencoder training, and enable much larger autoencoders.
A large fraction of the random activations of features we find, especially in GPT-4, are not yet adequately monosemantic. We believe that with improved techniques and greater scald this is potentially surmountable.
Our probe based metric is quite noisy, which could be improved by having a greater breadth of tasks and higher quality tasks.
While we use n2g for its computational efficiency, it only able to capture very simple patterns. We believe there is lot of room for improvement in terms of more expressive explanation methods that are also cheap enough to simulate to estimate explanation precision.
A context length of 64 tokens is potentially too few tokens to exhibit the most interesting behaviors of GPT-4.
7 RELATED WORK
Sparse coding on an over-complete dictionary was introduced by Mallat & Zhang 1993). Olshausen & Field (1996) refined the idea by proposing to learn the dictionary from the data, without supervision. This approach has been particularly influential in image processing, as seen for example in (Mairal al. 2014). Later, [innonn & Salakhutdinov (2006) proposed the autoencoder architecture to perform dimensionality reduction. Combining these concepts, sparse autoencoders were developed (Lee et al.
2007 Le al., 2013; Konda et al., (014)) to train autoencoders with sparsity priors, such as the Li penalty, to extract sparse features. Makhzani & Frey (2013) refined this concept by introducing k-sparse autoencoders, which use a Topk activation function instead of the L1 penalty. Makelovv et al.. (2024) evaluates autoencoders using metric that measures recovery of features from previously discovered circuits.
More recently, sparse autoencoders were applied to language models YYun et al. 202 Lee Sharkey
2022 Bricken et al.) 2023) Cunningham et al. 2023), y and multiple sparse autoencoders were trained
on small open-source language models (Marks 2023 Bloom 2024 Mossing et al. (024). Marks
jet al. (2024) showed that the resulting features from sparse autoencoders can find sparse circuits language models. |Wright & Sharkey (2024) pointed out that sparse autoencoders are subject to activation shrinking from L2 penalties, a property of L2 penalties first described in [Tibshiran] (1996). Taggart (2024) and RRajamanoharan et al. (2024) proposed use different activation functions to address activation shrinkage in sparse autoencoders. Braun et al. (2024) proposed to train sparse autoencoders on downstream KI. instead of reconstruction MSE.
Kaplan et al. (2020) studied scaling laws for language models which examine how loss varies with various hyperparameters. Cllrk et al. (2022) explore scaling laws related to sparsity using a bilinear fit. Lindsey et al. (2024) studied scaling laws specifically for autoencoders, defining the loss as specific balance of reconstruction and sparsity (rather than simply reconstruction, while holding sparsity fixed).
14 Anecdotally, we also found that lowering learning rates helped with decreasing dead latents. both in number of latents and training tokens
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-SVD: An algorithm for designing over- complete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11): 4311-4322, 2006.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons language models. OpenAl Blog, 2023. URL https: /opena inteenapuulll i.c.cl..bbbb...... core w ndows net pest/neuron-explle
wer http://ppp.ppaaaaapppppppp///e/e /wwp/iiidx...../ html
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference Artificial Intelligence, 2020.
Joseph Bloom. stream layers of gpt2-small.
Open
source
sparse autoencoders for all residual
Al
Alignment Forum, 2024.
URL
Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An interpretability illusion for BERT. arXiv preprint arXiv:2104.07145, 2021.
Dan Braun, Jordan Taylor, Nicholas Goldowsky- and Lee Sharkey. Identifying functionally important features with end-to-end sparse dictionary learning. 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden Mcl.ean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity; Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https//transformer-cccuutt features/index.hume
Cjadams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, Nithum, and Will Cukier- ski. Toxic comment classification challenge. Kaggle, 2017. URL https: https://kaggle: kaggle. com/
htmpppaa...ott.aaaaooooooooooooo www.afiaininniooi lenge
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann. Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMIR, 2022.
Tom Conerly, Adly Templeton, Trenton Bricken, Jonathan Marcus, and Tom Henighan. Update on how we train saes. Transformer Circuits Thread, 2024. https:/transformer-crniiii puttr0ri..pu//222//ppprll. update/index.htmllt hww.shttttttttttssssaaaaaa
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0888 08600, 2023.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint ar 2xXiv:229..10552 2022.
N Benjamin Erichson, Zhewei Yao, and Michael W Mahoney. JumpReLUU A retrofit defense strategy for adversarial attacks. arXiv preprint 1o%vv:/904.03750. 03750, 2019.
Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons scale. arXiv preprint arXiv:2305. 299999305./991 2023.
11

---

Published as a conference paper at ICLR 2025
Gabriel Goh.
Decoding the thought vector, 2016. URL http:://galcch...tttb.....
Thought Accessed: 2024-05-24.
Antonio Gulli, Ag's s corpus of news articles.
http://groups. Accessed: 2024-05-21.
corpus of news hwtps/ssrtiileseeee html
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.06610, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv:2008. S.Xi::2008.02275,, 2020.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint vrXiv:2010.14701, 2020.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, (intice,,13577800 313(5786):504-507, 2006.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendrieks, Johannes Welbl, Aidan Clark, al.
Training compute-optimal large language models. arXiv preprint arXiv:2203. 15555 2022.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Adam Jermyn and Adly Templeton. Ghost grads: An improvement on resampling. Transformer Circuits Thread, 2024.
resampling.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2000:00001.8366,, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pp. 79-86, Phuket, Thailand, September 13-15 2005. URL
/tttps:/acllatto. logy wrg. 2005 mt aummit papers 11
Kishore Konda, Roland Mcmisevic, and David Krueger. Zero-bias autoencoders and the benefits of 2014.
co-adapting features. arXiv preprint
Quoc V Le, Marc' Aurelio Ranzato, Rajat Monga, Matthieu Devin, Kai Chen, Greg S Corrado, Jeff Dean, and Andrew Y Ng. Building high-level features using large scale unsupervised learning. In 2013 IEEE international conference on acoustics. speech and signal processing, pp. 8595-8598. IEEE, 2013.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. Advances in neural information processing systems, 20, 2007.
Beren Millidge Lee Sharkey, Dan Braun. Taking features of super-
position with sparse autoencoders. AI Alignment Forum, 2022.
URL
http:///ww..aa.toeooooo////////a interim- -
https::/
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2/09.07958. 2021.
Jack Lindsey, Tom Conerly, Adly Templeton, Jonathan Marcus, and Tom Henighan. Scaling laws for dictionary learning. Transformer Circuits Thread, 2024. update/index. htmlissaainggllaws.
12

---

Published as a conference paper at ICLR 2025
Julien Mairal, Francis Bach, Jean Ponce, al. Sparse modeling for image and vision processing. Foundations and Trends@ in Computer Graphics and Vision, 8(2-3)885 283,,2833-55 2014.
Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control, 2024.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: 2013.
Arian Maleki. Coherence analysis of iterative thresholding algorithms. 2009 47th Annual Allerton Conference Communication, Control, and Computing (Allerton), pp. 236-243 IEEE, 2009.
Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions signal processing, 41(12)3337-3415, 1993.
Sam
Marks.
Some open-source dictionaries and dictionary
learn-
infrastructure. ing
Al Alignment Forum, 2023.
URL
https
http:/ooss//aaoiiioo www.://www.wmm.m......... //ww.wa. a www.aliggiennttrruu org wwtg/posteeeo
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:24003. 19647, 2024.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172, 2013.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1802.03426, 2018.
Nicolai Meinshausen. Relaxed lasso. Computational Statistics & Data Analysis, 52(1):374-393, 2007.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering. In EMNLP, 2018.
Dan Mossing, Steven Bills, Henk Tillman, Tom Dupré la Tour, Nick Cammarata, Leo Gao, Joshua Achiam, Catherine Yeh, Jan Leike, Jeff Wu, and William Saunders. Transformer debugger. 2024.
http::://gitubb...
cwm wow.oopo/aaaaoooooooooooeeooeaae ftttrassormmrrrebuggee
Neel Nanda, Arthur Commy, Lewis Smith. Lieberum, János Kramár, and Vikrant Varma. Progress update #1 from the gdm mech interp team: Full update. Al Alignment Forum, 2024.
Senthooran Rajamanoharan. Tom
URL
https: http::/wwwwwllinennforrrtoe
progress uuspsss-pppaee---eeee from hetp-----ee----eee-----ee-------
mech mech-intexp--eeeee
full update
Chris Olah, Adly Templeton, Trenton Bricken, and Adam Jermyn. Open problem: Attribution dictionary learning. Transformer Circuits Thread, 2024. http:///raastmmmmeecccciiipppppp
update/index. htm.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, (683)))6660766 1996.
OpenAl.
Gpt-2 output dataset.
https://github. wwwp:/ttw....ooop........ 2024 2022-0
2019.
OpenAl. GPT-4 technical report. arXiv prepreint arXiv:2003.08774 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Senthooran Rajamanoharan. Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404 16014, 2024.
13

---

Published as a conference paper at ICLR 2025
Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to ai complete question answering: A set of prerequisite real tasks. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.
2020.
David Ruppert. Efficient estimations from slowly convergent robbins monro process, Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09720 199009990..9728, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701 ww1.v:1701.06538, 2017.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro, Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: 1rXx::1999000033. 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprini arXiv: 131.v21312.6034, 2013.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762. 2024.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. Quartz: An open-domain dataset of qualitative relationship questions. arXiv preprint arXiv:1909.03553, 2019.
Glen Taggart. ProLU: A nonlinearity for sparse autoencoders. AI Alignment Forum, 2024. URL https: :/ww:..//wwww. al igw.ent forum. htt.://ww..aaaat......ooor.o///a
for sparse-prse--- autoencoders -parssattttteenoeeeess
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and
Jonathan Berant. Commonsenseqa 2.0: Exposing the limits ai through gamification. arXiv preprint arxiv:2201 200.0200000000000 2022.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024. URL https::////// htttttss:/ttraasffrrret. riruiiiitts pwbpppubb00000 2024/
Sealings scalinn-monosemaaree hwt.:/mooomaniiitt/oyooommomm hitpttyyiinxxx.e. html
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):2672288, 1996.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07460 2018.
Johannes Welbl, Nelson F. Liu, Matt Gardner, Gabor Angeli, Rik Koncel-Kedziorski, Emily Bender, Kyle Richardson, Peter Clark, and Nate Kushman. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06200 2017. URL WRLphttss///axii>> httpa/aawi...ggaa////aaaaaa 06209
Benjamin Wright and Lee Sharkey. Addressing feature suppression in SAEs. A/ Alignment Forum, 2024. URL https: /tctps://www..r al www.///ttww..iiinmmnnnm.....m.a forum. addressi ng- feature
saes
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023.
14

---

Published as a conference paper at ICLR 2025
average power: 0.60
everage power: -0.24
average power: -0.42
1014
3x10
10-3
Normaizeed MSE
2x100
Emeriig mit
Talens
1010
10
10
10a
101
10F
10P
10a
107
0.65
-0.17
-0.67
522
6x10
250
10l0
Nomeltzd aneeedddd MBE
leariing mnte
Toklors
4x10-1
10.4
Figure 9: Token budget, MSE, and learning rate power laws, averaged across values of k. First row is GPT-2, second row is GPT-4. Note that individual fits are noisy (especially for GPT-4) since learning rate sweeps coarse, and token budget depends learning rate.
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. arXiv preprint arXiv:2103. 15949, 2021.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. /277, 2023.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going on vacation" takes longer than "going for walk": A study of temporal commonsense understanding. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. URL https: /tapss://arxivv o abs httpg//b.///.9...000 100000300033 03065
A OPTIMIZATION
A.1 INITIALIZATION
We initialize autoencoders follows:
We initialize the bias bpre to be the geometric median a sample set of data points, following Bricken et al. (2023).
We initialize the encoder directions parallel to the respective decoder directions, that the corresponding latent read/write directions are the same I6 Directions are chosen uniformly randomly.
16This is done only at initialization; we do not tie the parameters in Cunningham al. 2023 This strategy
also presented in concurrent work (Conerly al. 2024
15

---

Published as a conference paper at ICLR 2025
We scale decoder latent directions to be unit norm at initialization (and also after each training step), following Bricken et al. (2023).
For baseline models we use torch default initialization for encoder magnitudes. For TopK models, we initialized the magnitude of the encoder such that the magnitude of reconstructed vectors match that of the inputs. However, in our ablations we find this has no effect or weak negative effect (Figure 14 17
A.2 AUXILIARY LOSS
We define an auxiliary loss (AuxK) similar "ghost grads" (Jermyn & Templeton, 2024) that models the reconstruction eπтоr using the top-Xaux dead latents (typically kaux = 512). Latents are flagged as dead during training if they have not activated for some predetermined number of tokens (typically 10 million). Then, given the reconstruction error of the main model e loss Laux lle allz where è = 2 is the reconstruction using the top-kaux dead latents. The full loss is then defined as L + aa where is a small coefficient (typically 1/32). Because the encoder forward pass can be shared (and dominates decoder cost and encoder backwards cost, see Appendix D). adding this auxiliary loss only increases the computational cost by about 10%.
åt we define the auxiliary
We found that the AuxK loss very occasionally NaNs at large scale, and zero when it is NaN to prevent training run collapse.
A.3 OPTIMIZER
We use the Adam optimizer (Kingma & Ba 2014) with |1 = 0.9 and 83 = 0.999, and a constant
learning rate. We tried several learning rate decay schedules but did not find consistent improvements in token budget convergence. We also did not find major benefits from tuning 31 and 32.
We project away gradient information parallel to the decoder vectors, to account for interaction between Adam and decoder normalization, as described in Bricken et al. (2023).
A.3.1 ADAM EPSILON
By convention, we average the gradient across the batch dimension. As result, the root mean square (RMS) of the gradient can often be very small, causing Adam to no longer be loss scale invariant. We find that by setting epsilon sufficiently small, these issues prevented, and that is otherwise not very sensitive and does not result in significant benefit to tune further. We use = 6.25 x 10-10 in many experiments in this paper, though we reduced further for some of the largest runs to be safe.
A.3.2 GRADIENT CLIPPING
When scaling the GPT-4 autoencoders, we found that gradient clipping was necessary prevent instability and divergence at higher learning rates. We found that gradient clipping substantially affected L(C' but not L(N). We did not use gradient clipping for the GPT-2 small runs.
A.4 BATCH SIZE
Larger batch sizes are critical for allowing much greater parallelism. Prior work tends batch sizes like 2048 or 4096 tokens (Bricken et al. 20223] Conerly et al.) 2024] Rajamanoharan et al.] (024)) To gain the benefits of parallelism,
batch size of 131,072 tokens for most of our experiments.
While batch size affects L(C) substantially, we find that the L(N) loss does not depend strongly on batch size when optimization hyperparameters are set appropriately (Figure 10)
17 Note that the scaling factor nontrivial interaction with n. and scales between @ and (1/k). This scheme has the advantage that is optimal at init the infinite width limit. We did not try simpler schemes like scaling by ((1/ / (V//k)).
16

---

Published as a conference paper at ICLR 2025
4x10
Batch Size
4296
32/68
3x
131072
Normalized to MSE
524288
2x10-1
10a
307
2019
Takens Elapsed
Figure 10: With correct hyperparameter settings, different batch sizes converge to the same L(N)
loss (gp22malll..
A.5 WEIGITT AVERAGING
We find that keeping an exponential moving average (EMA) [Epperr (1988) of the weights slightly reduces sensitivity to learning rate by allowing slightly higher learning rates to be tolerated. Due to its low cost, we use EMA in all experiments. We use an EMA coefficient of 0.999, and did not find substantial benefit to tuning it.
We a bias-correction similar to that used in Kingma & Ba 2014). Despite this, the early steps of EMA are still generally worse than the original model. Thus for the L(C) experiments, we take the min of the EMA model's and non-averaged model's validation losses.
A.6 OTHER DETAILS
For the main MSE loss, we compute an MSE normalization constant once at the beginning of training, and do not do any loss normalization per batch.
For the AuxK MSE loss, we compute the normalization token, because the scale of the error changes throughout training.
In theory, the ópre Ir should be scaled linearly with the norm of the data make the autoencoder completely invariant to input scale. In practice we find it to tolerate an extremely wide range of values with little impact quality.
Anecdotally, we noticed that when decaying the learning rate of an autoencoder previously training at the L(C) loss, the number of dead latents would decrease.
B OTHER TRAINING DETAILS
Unless otherwise noted, autoencoders trained the residual activation directly after the layernorm (with layernorm weights folded into the attention weights), since this corresponds to how residual stream activations used. This also causes importance of input vectors to be uniform, rather than weighted by norn
B. TOPK TRAINING DETAILS
We select knux as power of two close dmoded (e.g. 512 for GPT-2 small). We typically select = 1/32. We find that the training is generally not extremely sensitive to the choice of these hyperparameters.
We find empirically that using AuxK eliminates almost all dead latents by the end of training.
Unfortunately, because of compute constraints, we were unable to train our 16M latent autoencoder to L(N), which made it not possible to include the 16M as part of a consistent L(N) series.
We believe one of the main impacts of normalizing is that it the first token positions are downweighted in importance (see suessecctioon F2 for more discussion).
17

---

Published as a conference paper at ICLR 2025
ReLU
ProLU STE
Delta cross-entropy
10
Gated
TooK (ours)
10
4 10
better
10-1
10-1 Normalized MSE
10
Figure 11: For a fixed sparsity level (7.p = 128), a given MSE leads to lower downstream-loss for TopK than for other activations functions. (GPT-4). We are less confident about these than the corresponding ones for GPT-2, yet the results are consistent (see Figure 5).
10a
10-2
ReLU ProLU STE Gated
10-1
Dead latents ratio
Dead latents ratio
TopK (ours)
10
10-1
10"3_
ReLU
ProLU ReLU
10-4
ProLU STE
Gated
10-5
TopK (ours)
10a
102
10
10a
Sparsity
Sparsity (LO)
(a) Dead latent ratios for GPT-2
(b) Dead latent ratios for GPT-4
Figure 12: Our baselines generally have few dead latents, similar or less than our TopK models.
B.2 BASELINE HYPEERPARAMETES
Baseline ReLU autoencoders were trained on GPT-2 small, layer 8. We sweep learning rate in [5e-5, le-4, 2e-4, 4e-4], L1 coefficient in [1.7e-3, le-3,, 5e-3, le-2, 1.7e-2] and train for 8 epochs of 6.4 billion tokens a batch size of 131072. We try different resampling periods in [12.5k, 25k] steps, and choose to resample 4 times throughout training. We consider a feature dead if it does not activate for 10 million tokens.
For Gated SAE (Rajamanoharan et al. 2024). we sweep I,, coefficient in [1e-3, 2.5e-3, 5e-3, le-2, 2e-2], learning rate in [2.5e-5, 3e-5, le-4], train for epochs of 6.4 billion tokens at batch size of 131072. We resample 4 times throughout training.
For ProLU autoencoders (Taggart 2024), we sweep L, coefficient [5e-4, le-3, 2.5e-3, Se-3, le-2, 2e-2], learning rate in [2.5e-5, 5e-5, le-4], train for 6 epochs of 6.4 billion tokens at batch size of 131072. We resample 4 times throughout training. For the ProLU gradient, we try both ProLU-STE and ProLU-ReLU. Note that, consistent with the original work, ProLU-STE autoencoders all have Lo < 25, even for small L, coefficients.
We used similar settings and sweeps for autoencoders trained on GPT4. Differences include: replacing the resampling of dead latents with L, coefficient warm-up over 5% training (Conerly et al.) 2024); removing the decoder unit-norm constraint and adding the decoder norm in the L penalty (Conerly et al. 2024).
Our baselines generally have few dead latents, similar less than TopK models (see Figure 12
18

---

Published as a conference paper at ICLR 2025
C TRAINING ABLATIONS
C.I DEAD LATENT PREVENTION
10
[reccton of latents
0.6
none
sukk
0.4
auxk tied Int
0.0
10a
10a
101
10t3
Tokens Elapsed
Figure 13: Methods that reduce the number of dead latents (gpt2sm 2M, k=32). With AuxK and/or tied initialization, number of dead latents generally decreases over the course of training, after an early spike.
We find that the reduction in dead latents is mostly due to a combination of the AuxK loss and the tied initialization scheme.
C.2 INITIALIZATION
2x
Hie Init
18
scaled Init
default
moliteded MSE
1.6
(tee+sccaled)
14x10-1
12x10-1
Learning
Figure 14: Initialization ablation (gpt2sm 128k, k=32).
We find that tied initialization substantially improves MSE, and that our encoder initialization scheme has no effect when tied initialization is being used, and hurts slightly on its own.
C.3
DENC
We find that bane does not affect the MSE at convergence. With bene removed, the autencoder is equivalent to JumpReLU where the threshold dynamically chosen per example such that exactly k latents are active. However, convergence is slightly slower without bene- We believe this may be confounded by encoder learning rate but did not investigate this further.
C.4 DECODER NORMALIZATION
After each step renormalize columns of the decoder to be unit-norm, following Bricken et al. (003)). This normalization (or a modified L1 term, as in Conerly et al. (2024)) is necessary for LI autoencoders, because otherwise the L1 loss be gamed by making the latents arbitrarily small. For TopK autoencoders, the normalization is optional. However, we find that still improves MSE, we still it in all of our experiments.
19

---

Published as a conference paper at ICLR 2025
2x
Latent Baa no latent blas besellie
10
Normeliee ASE
1.6
14x10-1
12x10
Learning Rate
Figure 15: benc does not strongly affect loss (gpt2sm 128k, k=32).
Decodar Normalization
enabled
tormalizdd MSE
10-
12x10-1
10
10
Learning
Figure 16: The decoder normalization slightly improves loss (gpt2sm 128k, k=32).
D SYSTEMS
Scaling autoencoders to the largest scales in this paper would not feasible without our systems improvements. Model parallelism is necessary once parameters cannot fit GPU, A naive implementation can be an order of magnitude slower than our optimized implementation at the very largest scales.
D.I PARALLELISM
We standard data parallel and tensor sharding (Shoeybi et al. 2019). with additional allgather for the TopK forward pass to determine which & latents should are in the global top k. To minimize the cost of this allgather, we truncate to a capacity factor of per possible but would require modifications to NCCL. For the largest (16 million) latent autoencoder, we use 512-way sharding. Large batch sizes (subsection A.4 are very important for reducing the parallelization overhead.
improvements
The very small number of layers creates a challenge for parallelism it makes pipeline parallelism (Huang et al. 2019) and FSDP (Zhao et al. 2023) inapplicable. Additionally, opportunities for communications overlap are limited because of the small number of layers, though we do overlap host to device transfers and encoder data parallel comms for small improvement.
D.2 KERNELS
We can take advantage of the extreme sparsity of latents to perform most operations using substantially less compute and memory than naively doing dense matrix multiplication. This was important when scaling to large numbers of latents, both via directly increasing throughput and reducing memory usage.
We main kernels:
20

---

Published as a conference paper at ICLR 2025
DenseSparseMattul:: multiplication between a dense and sparse matrix
Matmul PatmmlAAtSpaa Sparss Indices: a multiplication of two dense matrices evaluated at a set of sparse indices
Then, we have the following optimizations:
1. The decoder forward pass uses DensesparseeKätmu
2. The decoder gradient uses DenseSparsseMstmm)
3. The latent gradient uses hatmuultttppareenndicc
4. The encoder gradient uses DenseSparsseMatmmll
5. The pre-bias gradient uses a trick of summing pre-activation gradient across the batch dimension before multiplying with the encoder weights.
Theoretically, this gives a compute efficiency improvement of up to 6x in the limit of sparsity, since the encoder forward pass is the only remaining dense operation. In practice, encoder forward pass is much of the compute, and the pre-activations are much the memory.
indeed find the
To ensure that reads coalesced, the decoder weight matrix must also be stored transposed from the typical layout. We also use many other kernels for fusing various operations for reducing memory and memory bandwidth usage.
E QUALITATIVE RESULTS
E.1 SUBJECTIVE LATENT QUALITY
Throughout the project, we stumbled upon many subjectively interesting latents. The majority of latents in our GPT-2 small autoencoders seemed interpretable, Furthermore, the ablations typically had predictable effects based on the activation conditions. For example, some features that are potentially part of interesting circuits in GPT-2:
random positive activations.
An unexpected token breaking a repetition pattern (A (CBCD A B X!). This upvotes future pattern breaks (A B Y!), but also upvotes continuation of the pattern right after the break token (A B X + D).
Text within quotes, especially activating when within two nested sets of quotes. Upvotes tokens that close the quotes like or as well tokens which close multiple sets of quotes at once, such as " and
Copying/induction of capitalized phrases (A B A B)
In GPT-4, we tended to find more complex features, including ones that activate on the same concept in multiple languages, or on complex technical concepts such as algebraic rings.
You can explore for yourself at our viewer
E.2 FINDING FEATURES
Typical features are of particular interest, but having an autoencoder also lets one easily find relevant features. Specifically, one can use gradient- gradient-based attribution to quickly compute how relevant latents are to behaviors of interest (Bachrens et al. 2010 Simonyan al. 2013).
Following the methodologies of Templeton et al. (2024) and Mossing et al. (2024), we found features by using a hand-written prompt, with a set of "positive" and "negative" token predictions, and back- propagating from logit differences to latent values. We then consider latents sorted by activation times gradient value, and inspect the latents.
We tried this methodology with quickly find a number of safety relevant latents, such as ones corresponding to profanity or child sexual content. Clamping these latents appeared to have causal effect the samples. For example, clamping the profanity latent to negative values results in significantly less profanity (some profanity still be observed in situations where it copies from the prompt).
2n 131072, & 32 GPT-2 autoencoder and were able to
21

---

Published as a conference paper at ICLR 2025
10r
no20
10a
n-48
n= 32768
n 32768
-=131772
1-131772
524288
524288
Count
10
20971552
Count
10*
D-20971522
10a
104-
104-
309
dead
16-8
le-7
16-6
16-5 1o-4
16-3
16-2
16-1
160
dead 1e-15 le-13 1e-11
10-9
16-7
16-5
16-3
36-1
161
Density
E[Actvation"2]
Figure 17: Distributions of latent densities, and average squared activation. Note that we do not observe multiple density modes, observed in (Bricken et al., 2023). Counts are sampled over 1.5e7 total tokens. Note that because latents that activate every 1e7 tokens are considered dead during training (and thus receive AuxK gradient updates), 1e-7 is in some sense the minimum density, though the AuxK loss term coefficient may allow it to be lower.
E.3 LATENT ACTIVATION DISTRIBUTIONS
We anecdotally found that latent activation distributions often have multiple modes, especially in early layers.
E.4 LATENT DENSITY AND IMPORTANCE CURVES
We find that log of latent density is approximately Gaussian Figure 17). we define the importance of feature to the expected squared activation (an approximation of marginal impact on MSE), then log importance looks a bit more like Laplace distribution. Modal density and modal feature importance both decrease with number of total latents (for fixed k), expected.
E.5 SOLUTIONS WITH DENSE LATENTS
One option for sparse autoencoder is to simply learn the directions that explain the most variance. As l: approaches dmnodel, this "principal component"liik solution may become competitive with solutions where each latent used sparsely. order to check for such solutions. we can simply measure the average density of the d-modet densest latents. Using this metric, we find that for some hyperparameter settings, GPT-2 small autoencoders find solutions with many dense latents (Figure 18). beginning around & 256 but especially for A: 512.
This coincides with when the scaling laws from /ectionn begin to break many our trends between from|section k and reconstruction loss bend significantly. Also, MSE becomes significantly less sensitive to n, at 512.
E.6 RECURRING DENSE FEATURES IN GPT-2 SMALL
We manually examined the densest latents across various GPT-2 small layer autoencoders, trained in different ways (e.g. differing numbers of total latents).
The two densest latents are always the same feature: the latent simply activates more and more later in the context (~ 40% active), and one that activates more and more earlier in the context excluding the first position (~355% active). Both features look like they want to activate more than they do, with TopK probably preventing it from activating with lower values.
The third densest latent is always first-token fist-token-position feature (~30%% active), which has a modal activation value in narrow range between 14.6- 14.8. Most of its activation values significantly smaller values, at tokens after the first position; the large value is always at the first token. These smaller values appear uninterpretable; conjecture these simply interference with the first
22

---

Published as a conference paper at ICLR 2025
average density of d_ model latents
0.35
2048
0.30
Number of Latents
8192
0.25
0.20
32768
0.15
0.10
131072
0.05
8 16 32
64
128
256
512
LO
Figure 18: Average density of the d'modet most-dense features, divided by I-,, for different autoen- coders. When & = 512, the learned autoencoders have many dense features. This corresponds to when ablations stop having sparse effects subsection 4.5] and anecdotally corresponds to noticeably less interpretable features. For & 256, 2048, there is perhaps intermediate regime.
Explenation score
0.30
Nunba of Latute
025
010
131672
0.5
517
Figure 19: Explanation scores for GPT-2 small autoencoders of different n and evaluated on 400 randomly chosen latents per autoencoder. It is hard to read off trends, but the explanation score is able to somewhat detect the dense solutions region.
position direction. (Sometimes there of these latents, the second with smaller activation values.)
Finally, there recurring "repetition" feature that is ~20 dense. Its top activations are mostly highly repetitive sequences, such as series of dates, chapter indices, numbers, punctuations, repeated exact phrases, or other repetitive things such as Chess PGN notation. However, like the first-token- position latents, random activations of this latent are typically appear unrelated and uninterpretable.
Often in the top ten densest latents, we find opposing latents, which have decoder cosine similarity close to -1 In particular, the first-token- firs--token-oositioon feature and the repetition latent both seems to always have an opposite latent. The less dense of the two opposite latents always seems appear uninterpretable. We conjecture that these are symptoms of optimization failure the opposite latents cancel out spurious activations in the denser latent.
E.7
CLUSTERING LATENTS
(Elaage al. 2022) discuss how underlying features may lie in distinct sub-spaces. such sub- spaces exists, we hypothesize that the set of latent encoding vectors W E R can be written as block-diagonal matrix W" PW Rt where P E = a permutation matrix, and It E gdxd orthogonal. We can then use the singular vector decomposition (SVD) to write W = UVVT and W" = U'' noting that U also block diagonal. Finally, we write W P' R UEVT, and because the SVD is unique up to column permutation P', get U P'U'P' In other words, W is block-dit -iigonal in some unknown basis, U also block diagonal up to permutation of rows and columns.
23

---

Published as a conference paper at ICLR 2025
0.094
140
5K
0.001
120
10
100
200
15k
80
Bftctive numbo
28k
200
400
600
30
Singular voctors
UMAP UMAP embedding
UHAP Emboodinng
Figure 20: The residual stream seems composed two separate sub-spaces. About 25% of latents mostly project sub-space using 25% of dimensions. These latents tend to have larger encoder norm, and to activate on a smaller number of vocabulary tokens. The remaining 75% of latents mostly project the remaining 75% of dimensions, and activate a larger number of vocabulary tokens.
To find good permutation of rows, sorted the rows of U based how similarly they project on all elements of the singular vector basis. Specifically, we normalized each row to unit norm
U, = U.] and considered the pairwise euclidean distances d ||ú," U 27,2 These pairwise distances were then reduced to a single dimension with a UMAP algorithm (Mclnnes et al. 201 1)..
The obtained I-dimensional embedding was then used to order the projections U; FFigure 20 which reveals two fuzzily separated sub-spaces. These two sub-spaces respectively about 25% % and 75% of the dimensions of the entire vector space.
Interestingly, ordering the columns by singular values is fairly consistent with these two sub-spaces. One reason for this result might by that latents projecting to the first sub-space have different encoder norms than latents projecting the second sub-space (Figure 20p). This difference in norm can significantly guide the SVD to separate these two sub-spaces.
To further interpret these two sub-spaces, manually looked latents from each cluster. We found that latents from the smaller cluster tend to activate on relatively non-diverse vocabulary tokens. To quantify this insight, we first estimated Ai,, the average squared activation of latent on vocabulary
token v. Then, we normalized the vectors A; to sum to one, A. = Ai,e/ En. Ai, or and computed the effective number of token ma = exp(E, A,. log( log(A,..)) )).) The effective number of token is continuous metric with values in [L, rooccb)) , and vocabulary tokens. With this metric, we confirmed quantitatively (Figure 20c) that latents from the smaller cluster all activate on relatively low numbers of vocabulary tokens (less than 100), whereas latents from the larger cluster sometimes activate on larger numbers of vocabulary tokens (up to 1000).
equal when latent activates equally on
E.8 LATENT VISUALIZZTIONNS
F
MISCELLANEOUS SMALL RESULTS
F.I IMPACT OF DIFFERENT LOCATIONS
sweep across locations in GPT-2 small, we found that the optimal learning rate varies with layer and location type (MLP delta, attention delta, MLP post, attention post), but was within factor of two.
Number of tokens needed for convergence is noticeably higher for earlier layers. MSE is lowest at carly layers of the residual stream, increasing until the final layer, at which point it drops. Residual stream deltas have MSE peaking around layer 6, with attention delta falling sharply at the final layer.
When ablating to reconstructions, downstream loss and KL get strietly worse with layer. This is despite normalized MSE dropping at late layers. However, there appears to be exception at final layers (layer 11/12 and especially 12/12 of GPT-2 small), which can have better normalized MSE than earlier layers, but more severe effects downstream prediction (Figure 27
24

---

F.I
Impact of different locations
25
Latent 28719; Adenosine/dopamine receptors
Top Activating Sequences:
Random Activating Sequenses:
27-g0001 (tt0000)FFTaaa////00 RilaDDRlikee receptors genes generate variants D2R
dopamine D2 receptor protein was observed pial and mos
cur with the nation that an inverted U- U-shaped relationship
aminergic and alpha 2-adrenergic receptors, respectively. Cross-dascn doss-daansiiizztioon experiments revealed
dopamine signaling molecular lovel the dorsol
cells not affected by peetreatment with IAP
mAChR
oll, [@B1] Contrasting evidence exist exist about role for D4R
BB reward
release of depamine the hippocampus where enhances long term potentiation
D) DIR play prominent role in the aqquisitoonmmiinteeanee self-admin
caudate-putameu. The basal extracellular concentration of dopasnine the dorsolateral prefront
2) agonist quinpirole, 0,1 mg
Latent 50233: Comparisons
Top Activating Soquences:
Random Activating Sequences:
your hand and wrist wnint Thhn The cuff itself is longer than what you may seeing This
the filament, There is a comparahly sixed channel in the deduced from electron micrographs
revised
thicker
been agreed upon
GeV, more times greater than Sm, 1508GeV V, then
and requested in Exhibit 7
the silelnbution and make the face like the iphome only does it aurrntly'nnoo
resomator has, been bent
different From the leg FIG. 1,
not solid heige colour where the eyelets sir like is
wing namower than on the upperside, the chocolate-coloured coloured
Sa'n550 and the springer. Bocause the axle on springer about further forward of
eepuenecanndegsserinssttee. wwt.:CCCCSS....t.taff...... we
Latent 17441: Ratifications (multilingual)
Top Activating Sequences:
Random Activating Sequences:
Arizona ratifiedd thelnproposed sixteenth amendment to the Constitution
not to ratlffy its hilateral trade agreement with Vietnam, linking action
rdiggar en opfordring til ikke at ratificgre trakräten,, er pure opsp
the bank or to the insurer, impliedly ratifiied Dorothy's acts. As between Dorothy and Harford
students with a hatred of America. With the ratification of H. 3077, any
rafiiiation
neither party, welmultimately
refification the jury chosenj ld. This was error
les États membres les pays candidats ratilient premier lieu convention du Conseil
ratifiedd the application of the payment of the money borrowed.
This
effective cconomic boyeott of states whose logislatures did not ratily Equaa Rights Amendment the United Constitution
Figure 21: Cherry-picked qualitative examples of latents and their activations on top-activating and random-acto -ectiviiciii sequences

---

F.I Impact of different locations
26
Latent 63607
Top Activating Sequences:
Random Activating Sequences:
Academy of Fine Arts. Her work has heen exhibited Gallery Nik
Mathematics and Computer Technology*, Aladd Nark. Ukrain. SSR Inst. Mat., Pre
kowe Akademii Podlaskiej. Administracaa Zarzadz
from the Academy of Neuraeconemmics and Neuromunagement Ningbo University
founding member of the Academy of Arles in 1668, and the Duke
J.nlaPham Mau Quan, Archive for Rational Mechanics and Analysis [**]*e],
in Little Rock, now known as the Academy at Riverdale wi' WealthinnRoccefflle served chief
programs museum which be Getz began to collect fire apparatus
of Flame,
American Academy of Pricts, most presrigions honors American
provided as educational service of the American Acadomy of Neurology. It is not substitute for professional medical
Latent 6548
Top Activating Sequences:
Random Activating Sequences:
see the same preconditionerlal flag to reuse the preconditioner from forward
server name intai 'UverLoogin.dbo.. www.ge/..o.gggel.yyya sers!
puram This Points the EF
and disable various parts below codeta] (/ find out which part of the code creates the
image,
Jmaae object. if (ScreateNew)in
ways build joinsta] of joins and
last segments only'u detine angle of the join at the very
casting cell backgrrunddoooorr blackkn return celfan
mysterycin movl Redi, Neaxin
looeel Yn| Invalid in Returns AccessDenied
Latent 47982
Top Activating Sequences:
Activating Sequences:
ive layer is isolated from the fourth conductive layer by the first opening, the third conductive layer electric
and others, in which substirucnts are attached by
you cun look through the model and out the fromt with childdish gloe. The is still lacking
scent-laden air from the user""s nose exhoust which is part of the nasal interface.
possible explanation means of its inhibitory function on dopamine release
cognition hy
drawing and in which the chamber 22 connected the conduit to the oil reservoir
interface. Instend of the needle carrying the suture attachment
conductive layer electrically connects to the common line by the fourth conductive layer. Thus,
monitor screen by red input value of R.subi is:
sleeve tolnhis head, then drops, way large
Figure 22: Random qualitative examples of latents and their activations on top-activating and random- activating sequences

---

F.I Impact of different locations
27
0.530
relu n=2048
0.525
topk n=2048
relu n=8192
0.520
topk n=8192
relu n=32768
0,515
topk n
Probe Loss
relu n=131072
0.510
topk n=131072
0,505
0.500
0.495
0.490
10a
102
LO
Figure 23: TopK beats ReLU only on the sparsity- MSE frontier, but also on the sparsity-prose probe loss frontier. (lower is better)
0.30
0.25
0.20
relu n= 2048
topk n=2048
0.15
relu n=8192
topk n=8192
0.10
relu n=32768
topk n=32768
relu n= 131022
0.05
topk n- 131072
101
102
L0
0180
5-519
0.75
0.00
0.05
look 2-4046
(UU (122085
0.5
B0ok 5151
rCU n=0192
reU n=32766
0.20
Took 32768
P03 n-131877
1o n=10022
065
Figure 24: TopK beats ReLU on N2G score. Its N2G explanations have noticeably higher recall, but worse precision. (higher is better)

---

Published as a conference paper at ICLR 2025
67
96
Burbee Leterts
bumber of Latetts
8192
02
8192
32768
32768
04
42
133072
(a) Recall of N2G explanations
(b) Precision of N2G explanations (aact 0|n2g 0)
P(n2g > Ofact 0)
Figure 25: Neuron2graph precision and recall. The average autoencoder latent is generally easier explain decreases and n increases. However, n 2048, k 512 latents are easy explain since many latents activate extremely densely (see subsecctin E.5
10
16M
Normalized MSE
10
3 x 10-1
x 10-1
10a
104
10a
106
107
Number of Latents
Figure 26: The L(N) scaling law, including the best 16M checkpoint, which we did not have time to train the L(N) token budget due compute constraints.
We can also see that choice of layer affects different metrics differently (Fiuure 28 While earlier layers (unsurprisingly) have better N2G explanations, later layers do better on probe loss and sparsity,
In early results with autoencoders trained on the last layer of GPT-2 small, we found the results to be qualitatively worse than the layer 8 results, so we use layer 8 for experiments going forwards.
F.2 IMPACT OF TOKEN POSITION
We find that tokens at later positions are harder to reconstruct (Figure 29 We hypothesize that this is because the residual stream at later positions have more features. First positions are particularly egregiously easy to reconstruct, terms of normalized MSE, but they also generally have residual stream norm more than order of magnitude larger than other positions (Figure 30 shows that in GPT-2 small, the exception is only at carly layers and the final layer of GPT-2 small), This phenomenon was explained in (Sun al. 2024 Xiao al. 2023), which demonstrate that these activations serve as crucial attention resting states.
downstream loss and KL after ablating to autoencoder reconstruction at layers with these large norms (Figure 27). despite having better normalized MSE. This is consistent with the hypothesis that the large norm directions at the first position important
First token positions have significantly
28

---

Published as a conference paper at ICLR 2025
layer 1/12
First token only
12/12 post
1/12 cont
can
Loos Lott
10"1-
10
10
Normal Laed
Narmakized MSE
Figure 27: (a) Normalized MSE gets worse later in the network, with the exception of the last two layers, where improves. Later layers suffer worse loss differences when ablating to reconstruction, the final two layers. (b) First position token loss is more severely affected by ablation than other layers despite having lower normalized MSE. Overall loss difference has no clear relation with normalized MSE across layers. In very early layers and the final layer, where residual stream norm also more normal (Figure 30p, we see more typical loss difference and MSE. This consistent with the hypothesis that the large norm component of the first token is primarily serve attention operations at later tokens.
precivian
03
200
(a) Recall N2G explanations P(n2g > Ofact > 0)
(b) Precision of N2G explanations
P(att > 0|n2g > 0)
05
10
10 11
(c) Probe loss
(d) Logit diff' sparsity
Figure 28: Metrics as a function of layer, for GPT-2 small autoencoders with k: = 32 and n = 32768.
Earlier layers are easier explain terms of token patterns, but later layers are better for recovering features and have sparser logit diffs.
for loss other tokens but not the current token. This position then potentially gets subtracted back out at the final layer as the model focuses on current-token prediction.
0.14
0.12
0.10
No olod PSS
0.06
0.64
0.02
32768
6.00
10
20
30
60
Tolion Index
Figure 29: Later tokens difficult to reconstruct. (lower is better)
29

---

Published as a conference paper at ICLR 2025
layer 1/12 12/32 rozid, post, mip
Average Residbal Norm
10
10*
Context Position
Figure 30: Residual stream norms by context position. First token positions than an order of magnitude larger than other positions, except at the first and last layer for GPT-2 small.
G IRREDUCIBLE LOSS TERM
In language models, the irreducible loss exists because text has some intrinsic unpredictableness even with perfect language model, the loss of predicting the next token cannot be zero. Since an arbitrarily large autoencoder in fact perfectly reconstruct the input, initially expected there to be no irreducible loss term. However, we found the quality the fit to be substantially less good without an irreducible loss.
While we don't t fully understand the reason behind the irreducible loss term, our hypothesis that the activations made spectrum of components with different amount of structure. We expect less strucutured data to also have worse scaling exponent. At the most extreme, some amount of the activations could be completely unstructured gaussian noise. In synthetic experiments with unstructured noise (see Figure 31), we find an L(N) exponent of -0.04 on 768-dimensions gaussian data, which is much shallower than the approximately -0.26 we see on GPT-2-smail activations of similar dimensionality.
ftt: L 0.00 N-004
Normakzodd MSR
6x
atents
Figure 31: L(N) scaling law for training on 768 768-dimensional random gaussian data with k=32
H FURTHER PROBE BASED EVALUATIONS RESULTS
30

---

31
4.6 x 10
run name
128k
4.575 10-1
1M
16M
4.55 10-1
4.525 10-1
M
loss
4.5 10-1
4.475 x 10-1
4.45 10-1
4.425 10-1
10 elapsed_examples
1010
Figure 32: Probe eval scores through training for 128k, 1M, and 16M autoencoders. The baseline score of using the channels of the residual stream directly is 0.600.
MMA,
Mm EESTIN
hoology
opennbokknn
fopczoise
amazon
Loss
commorss To serse
-0.02
-0.04
-0.05
1613
1.5 10
0000
2 2.5x 1029 1077
Elapsed Tokens
Figure 33: Probe eval scores the 16M autoencoder starting at the point where probe features start developing (around 10B tokens elapsed).

---

32
piga
utilit
commonsense..gg
0.6
deontology
ustice
socialiqa
quail
0.5
cola
gpt2disc
openboo <qa
truthfulae
isalivee
avg
igsaw
quartz
0.4
virtue
Loss
0.3
-amazon
sciq
0.2
0.1
sexcor gender
base64
0.0
1010
Elapsed Tokens
Figure 34: Probe eval scores for the 16M autoencoder broken down by task. Some lines (europarl, bigrams, occupations, ag news) are aggregations of multiple tasks.

---

Table 1: Tasks used in the probe-based evaluation suite
Task Name
Details
amazon
McAuley & Leskovec 2013
sciq
Welbl et al. (2017)
truthfulqa
Lin et al. (2021)
mc_ mnc_taco
Zhou et al. (2019) 9
piqa
Bisk et al. (0200)
quail
Rogers et 2020
quartz
Tafjord et al.
2019
justice
virtuc
Hendrycks et al.
2020
utilitarianism
deontology
commonsense qa
Talmor et al. (0022)
openbookqa
Mihaylov et al. (2018)
base64
discrimination of base64 vs pretraining data
wikidata_isalive
wikidata_sex_on _o__genderrrrrr
wikkiata_occuppaionnissttti isathlete
wikidataa_ocuppaiiooisscooo
wikidata_occuppann ispolliiciniiiiiiiii
wikidata_occupattie iistiagtioonssnngeer
aakidata_occuplaae sccupatiin_isresearchee isresearcher
phrase_high--shhol
phrase_living-room
phrase_social-sccuriiy
phrase_credit-card
bbrase_blood-pres.ee
Gurnee al. (2023)
phrase_social-mediaa
phrasee_enne-expressiin
field
phrase_cell-Iin
phrase_trial-court
chrase_seconn-derivee
phrase_norrt--mmerica
phrase_humaan-rights
phrase_skde-efiects
phrase_
chrase_public-hecee
health
phrase_third-parc
phrase_climical-trial
social iqa
Sap et al.
2019
wic
Wang al. 2018
cola
gpt2disc
OpenAl 2019)
ag fo__newswwolll world
ag si__nes__spol
Gulli
ag businesss
ag news_eewsss scctsshcctchhhhhh
europari_es
europarl en
europarl fr
europarl nl
Koehn 2005
europarl
europarl el
europarl de
europarl pt
europarl sv
jigsaw
Cjadams et (2017)

---

Published as a conference paper at ICLR 2025
I CONTRIBUTIONS
Leo Gao implemented the autoencoder training codebase and basic infrastructure for GPT-4 ex- periments. Leo worked the systems, including kernels, parallelism, numerics, data processing, etc. Leo conducted most scaling and architecture experiments: TopK and AuxK, tied initialization, number of latents, subject model size, batch size, token budget, optimal L(C), L(N), random data, etc., and iterated on many other architecture and algorithmic choices. Leo designed and iterated on the probe based metric. Leo did investigations into downstream loss, feature recall/precision, and some early explorations into circuit sparsity.
Tom Dupré la Tour studied activation shrinkage, progressive recovery, and Multi-TopK, Tom implemented and trained the Gated and ProLU baselines, and trained and analyzed different layers and locations in GPT-2 small. Tom helped refine the scaling laws for L(N, K) and (.(N.. Tom discovered the latent sub-spaces (refining an idea and code originally from Carroll Wainwright).
Henk Tillman worked on N2G explanations and LM-based explainer scoring. Henk worked on infrastructure for scraping activations. Henk worked finding qualitatively interesting features, including safety-related features.
Jeff' Wu studied ablation effects and sparsity, and ablation reconstruction. Jeff managed infrastructure for metrics, and wrote the visualizer data collation and website code. Jeff analyzed overall cost of having fully sparse bottleneck, and analyzed recurring dense features. Cathy and Jeff worked finding safety-relevant features using attribution. Jeff managed core researchers on the project.
Gabriel Goh suggested using TopK, and contributed intuitions about TopK, AuxK, and optimization.
Rajan Troll contributed intuitions about and advised on optimization, scaling, and systems.
Alec Radford suggested using the irreducible loss term and contributed intuitions about and advised on the probe based metric, optimization, and scaling.
Jan Leike and llya Sutskever managed and led the Superalignment team.
34


Reviews:
[{"review_id": "YXtmcN5e0x", "reviewer": "Reviewer_CUAP", "summary": "The paper proposes the use of a k-sparse auto encoder model to  control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. The paper presents results showing better reconstruction and minimal dead latents. The authors trained a 16-million latent autoencoder on GPT-4 activations over 40 billion tokens, demonstrating clear scaling laws for the autoencoder size and sparsity. The main contribution of the paper is the use of TopK activation to limit the number of active latents and presenting metrics of latent quality.\n\nThis limiting factor seems to play a role in allowing scaling the model for larger encoders as demonstrated  experimentally on GPT2 with multiple ablation studies and an in-depth evaluation downstream loss, explainability amongst other metrics.", "questions": "1. Could you elaborate on the choice of fixed-k sparsity? Do you foresee any scenarios where adaptive sparsity (varying k based on input complexity) might improve model performance or interpretability? Have you experimented with such approaches?\n\n2. Given that TopK activation controls sparsity effectively, do you see potential limitations of this method in real-world applications, where sparsity needs may vary dynamically?\n\n3. Could you share more about how the interpretability metrics were validated? For instance, have you found that these metrics correlate well with human judgment in terms of interpretability?\n\n4. How sensitive is the auxiliary loss to hyperparameter changes, particularly at larger scales? Could you share any guidelines or best practices for tuning this component?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The adoption of k-sparse autoencoders with TopK activation is a significant improvement over traditional ReLU-based models. By directly controlling the number of active latents, TopK provides a straightforward way to balance reconstruction quality and sparsity, simplifying tuning and improving interpretability.\n\n2. The study of scaling laws for sparse autoencoders across various dimensions—autoencoder size, language model size, and sparsity—is well-executed. This contributes valuable insights into how autoencoders behave at larger scales, especially when trained on large models like GPT-4. Clear scaling patterns provide guidelines for researchers to optimize model size and token budgets effectively.\n\n3. The authors address the issue of \"dead latents\" effectively by using specific initialization and auxiliary loss techniques. This innovation minimizes computational waste and improves model efficiency, making it feasible to train much larger autoencoders while keeping most latents active, even at high scales.\n\n4.The paper introduces thoughtful metrics that go beyond simple reconstruction errors to evaluate feature quality. Metrics like downstream loss, probe loss, explainability, and sparsity of downstream effects offer a more nuanced assessment of latent quality, highlighting the interpretability of features that the autoencoders extract. This focus on interpretability is particularly important for mechanistic understanding and applications in AI alignment.\n\n5. The use of TopK avoids the activation shrinkage problem that commonly affects ReLU-based sparse autoencoders. This makes TopK particularly suitable for training sparse models without the need for additional regularisation to combat shrinkage, which simplifies model training and improves performance.", "weaknesses": "1. As highlighted in the limitations section, The TopK activation function might be overly restrictive. Each token must use exactly k latents, which could be suboptimal for capturing certain features that require more flexible sparsity levels based on input complexity. \n\n2. The experiments primarily use a context length of 64 tokens, which may not capture longer dependencies present in language models like GPT-4. By not exploring larger context lengths, the paper might miss behaviors and interactions that occur over more extended sequences, which could be valuable for understanding latent patterns and achieving higher interpretability.\n\n3. Although the paper introduces meaningful metrics for interpretability, some, like the downstream loss and explainability evaluations, require substantial computational resources, particularly for large models like GPT-4. \n\n4. The reliance on an auxiliary loss to prevent dead latents may add extra complexity to the training process. While effective, this approach could complicate optimization and might be sensitive to tuning, making it more challenging to reproduce results consistently across different datasets and architectures.\n\n5. Although the approach seem to scale on relatively smaller model, it is not clear how scalable this approach for much larger model.s\n\n6. While the paper discusses applications like anomaly detection and mechanistic interpretability, there is limited empirical testing in these areas. Applying these sparse autoencoders to real-world tasks and evaluating their practical benefits would strengthen the claim of their utility and robustness across applications.\n\n7. The explainability metric, which relies on methods like Neuron to Graph (N2G), is noted to have high recall but lower precision. This can create an \"illusion of interpretability,\" where explanations seem broad but lack specificity. The paper briefly acknowledges this, but a more in-depth exploration of how to balance recall and precision would improve confidence in these metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FFirLIpvVs", "reviewer": "Reviewer_JgtD", "summary": "The authors present a well written paper which makes multiple significant contributions to the field of language interpretability. The paper does not explicitly pose a set of questions, but the subject matter implies several concerns:\n1. Do sparse autoencoders scale to larger models? Does this require more latents than for larger models?\n2. Can sparse autoencoder training be improved? For example, can it be made simpler via use of a top-K activation function? \n3. How should we evaluate sparse autoencoders? Are there alternatives to the sparsity-reconstruction pareto-frontier framing?\n\nThe paper makes a number of contributions. \n1. First, they propose a novel training recipe based on the use of the Top-K activation function for directly imposing sparsity in the sparse bottleneck layer. They evaluate their methods against similar proposed methods via the sparsity-reconstruction pareto-frontier framing and find their recipe is competitive with the best methods in the existing literature.\n2. Second, they demonstrate scaling laws for their sparse autoencoder training recipe using the GPT4 family of models. \n3. Finally, they introduce a number of additional metrics for evaluating sparse autoencoders including loss-based metrics, metrics which rely on labeled datasets, automated interpretability and ablation sparsity. \n\nThe strongest contribution of the paper is clearly the application of SAEs to exceptionally larger models, showing that sparse autoencoders scale to such models, but the paper also makes numerous other contributions around training and evaluation methods.", "questions": "Some work that might improve this paper:\n- Checking whether top-K sparse autoencoders find the same latents as other methods. Reliance on summary statistics when making comparisons between architectures could be misleading. \n- Establishing that top-K sparse autoencoders work well on toy models of superposition (and using this to better explain why top-K might be better than other architectures).\n- Using longer prompts than 64 tokens when training sparse autoencoders (this seems rather short) or instead providing an empirical analysis showing that this hyperparameter has no meaningful impact on results. \n\nSince my overall impression of the paper is quite good, and most of the weaknesses, if addressed, would substantially increase the length of the paper, I don't believe addressing any given issue is essential.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "## Strengths by Section:\n\n**Training Recipe**: The paper is unusually detailed with respect to the training recipe, which provides more clarity than similar publications about the training of sparse autoencoders. As noted by the authors, use of the Top-K activation function in a sparse autoencoder is not new, but it is novel in the context of sparse autoencoders trained on model activations. Importantly, it solves a previously documented issue called \"shrinkage\" which results in latent activations being biased towards activating less strongly than they theoretically should. \n\n**Scaling laws**: The paper demonstrates not only the training of sparse autoencoders on a SOTA language model, substantiating claims that this technique is scalable. Use of GPT4 provides one of the strongest possible arguments that sparse autoencoders can scale to very large LLMs. Additionally, the authors provide clear scaling laws in terms of the compute, data and number of latents. Though possibly a minor detail, we appreciate that the authors highlighted their initial incorrect prediction that there would need to be non-zero irreducible loss (as such comments help inform readers that the result was not obvious and thus increases readability and makes the significance clearer). \n\n**Evaluating SAEs**: Though plausibly it would be reasonable to separate work scaling sparse autoencoders from work exploring how best to evaluate them, we appreciate the attempt to rectify the current gap in the literature around how to evaluate sparse autoencoders. Contributions in this section are well motivated and varied. Section 4.3 is particularly valuable, as Neuron 2 Graph is a plausibly much cheaper way to perform automatic interpretability as opposed to the use of language models. \n\n## Strength by Dimension\n\n**Originality**: While the work mostly combines existing ideas from the literature placing them in the context of sparse autoencoders trained on model activations, this hardly diminishes the contribution. Some ideas may be more clearly novel than others (such as the pre-training compute based metric that enables better interpretation of the increase in downstream loss in section 4.1). \n\n**Quality**: The work is generally high quality. There are no glaring issues that I can see. \n\n**Clarity**: Details of both the methods and reasoning are generally high quality.\n\n**Significance**: The paper is highly significant due to contributions along multiple lines (training recipe, scaling laws and evaluation of SAEs).", "weaknesses": "**Training Recipe:** \n- We appreciate that the proposed method eliminates the need for finetuning of the L1 coefficient (though choice of the number of activated latents is still required). We might interpret later analysis as attempting to identify a more principled way for setting the number of activations (setting K), though in the absence of better motivation, directly setting the number of activating latents isn’t clearly valuable. \n- Comparison of sparse autoencoder architectures is somewhat shallow, looking only at the sparsity-reconstruction pareto frontier. For example, the authors do not attempt to compare latents found by each method to establish whether they provide similar or distinct decompositions (is there often a 1:1 map between latents in sparse autoencoders trained via each method?). \n- The authors could have provided demonstrations that top-K sparse autoencoders work on toy models of superposition (as has been previously done to establish reason to prefer one architecture over another).\n- Whilst the authors consider alternative methods for evaluating sparse autoencoders later in the work, they should have revisited the training recipe comparison in the context of their other evaluation methods. It is unclear whether Gated / topK SAEs would be comparable on all evaluation methodologies.\n- The authors miss one possible issue with top-K activation functions. Use of the Top-K activation function means that the activation of a given latent cannot be calculated independently of others which is a minor but real obstacle to working with very large autoencoders (for example, when identifying max activating examples, one must first calculate all latent activations, then select the top-K latents). \n- Training on 64 token context sizes might affect the results. The models studied certainly can handle much longer prompts. The authors don’t note whether the SAEs generalize to activations produced during the process of larger numbers of consecutive tokens. \n\n**Scaling Laws**. We have no specific comments on the weaknesses of this section. \n\n**Evaluating SAEs**\n- Though contributions in this section are valuable, there is an absence of discussion resolving possible metrics and identifying which are best to use in practice. Though many related papers likely deal with the current uncertainties of the field similarly (by presenting various possible motivations / metrics without meaningfully resolving which is best). We believe that devoting more of the discussion to comparing metrics and resolving insights found via each may be fruitful.\n- (Repeating a point from earlier) The authors don’t evaluate non-TopK autoencoders via any of the new metrics leaving possibly valuable insights on the table. \n- Section 4.2 could have used a linear probe on the model activations as a baseline against which to compare the 1d logistic probe results. The authors also don’t perform any error analysis or raise any particular examples of latents which clearly track expected features of the input. While such qualitative analysis may not scale, it might possibly provide much better intuition about the results. The authors don’t address how sparse combinations of latents might jointly represent features.\n- The authors could attempt to compare Neuron 2 Graph explanations to those produced by language models (which is the norm for automatic explanations of features in the rest of the sparse autoencoder literature).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GuelfijD33", "reviewer": "Reviewer_Zr1S", "summary": "The paper develops training methodology and scaling laws for a new sparse autoencoder (SAE) variant - TopK SAEs, which replaces the L1 penalty in the standard SAE (https://transformer-circuits.pub/2023/monosemantic-features) with a TopK filter, so that the desired sparsity of activations can be set directly. The new architecture is evaluated against other popular SAE variants along the reconstruction-sparsity frontier. The interpretability of the TopK SAE latents is also measured via several novel metrics.\n\nIn more detail:\n- the TopK filter overcomes the shrinkage problem associated with the L1 penalty, allowing better reconstruction with the same sparsity;\n- to minimize dead latents, the TopK training method uses a simple initialization strategy and an auxiliary loss that incentivizes the dead latents to reconstruct the error of the SAE.\n- scaling laws for TopK are studied w.r.t. various parameters, such as the number of latents, the $k$ value, and the size of the model whose activations SAEs are trained on. \n- several evaluations beyond MSE/sparsity tradeoffs are performed:\n\t- downstream loss when SAE reconstructions are patched in\n\t- using individual SAE latents as probes for binary classification tasks \n\t- simple autointerpretability based on Neuron to Graph (N2G)\n\t- evaluating how sparse the effect of ablating SAE latents is on logits of the model", "questions": "- did you use the scaling laws to choose hyperparameters for the largest SAE training runs? Do you know how well that worked?\n- Have you thought about ways to more scalably evaluate precision of SAE latent explanations?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- the paper presents a new, simpler SAE architecture and training stack that demonstrably scales to frontier models and many SAE latents. The main advantages over concurrent work are:\n\t- the sparsity can be set directly, making it easier to do hyperparameter tuning;\n\t- the training recipe is also simple and with few knobs to tune;\n\t- yet results on the MSE/sparsity frontier are state of the art, and there are very few dead latents\n- many of the findings are quite well supported by extensive comparisons with other SAE variants\n- the discussion on the limitations of autointerpretability methods in 4.3. is illuminating\n- the paper proposes some new and interesting interpretability evaluations, especially probing and sparsity of downstream effects", "weaknesses": "- as the authors are already aware, using individual SAE latents as probes for very high-level binary tasks (such as sentiment) may be too restrictive, especially in light of phenomena such as feature splitting. This limits the significance of the results\n- another practical limitation of the TopK architecture is that it forces every activation to use exactly $k$ active features. In practice (at least in my experience) this may lead to many unimportant features to activate. See concurrent work https://www.alignmentforum.org/posts/Nkx6yWZNbAsfvic98/batchtopk-a-simple-improvement-for-topk-saes\n- the interpretability evaluation is at times too reliant on high-level aggregate metrics. For instance, how should we interpret the \"sparsity of downstream effects\" numbers?\n- the presentation on scaling laws would benefit from a comparison with concurrent work by Anthropic: https://transformer-circuits.pub/2024/scaling-monosemanticity/#scaling-scaling-laws\n- there is some inconsistent notation (e.g., using $N$ and $n$ for the number of latents) and some missing symbols from the text", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GvAvq2QoQP", "reviewer": "Reviewer_c1TT", "summary": "In this paper, the authors scale and evaluate sparse autoencoders (SAEs) on modern large language models (LLMs) across different model sizes (up to GPT4).\n\nThe main contributions are as follows:\n- SAEs are trained and evaluated across different model sizes, and scaling laws are found.\n- The top-k activation function is leveraged to obtain state of the art results, and its ability to avoid activation shrinkage is explored\n- An auxiliary loss is devised, and it significantly reduces the amount of dead features\n- New metrics are introduced for evaluation: the explanation loss (obtained using just interpretable explanations), and the probe loss (where the authors check wether the SAE reproduces known features).\n\nMy **recommendation**: Accept, 8/10 (good paper).\n\n### Why not lower score?\n\n- Scaling SAEs was a very important problem in interpretability, as indicated in the seminal SAE paper.\n- Top-k SAEs are a significant improvement on an important research topic. This approach is well motivated and well placed in the literature.\n\n### Why not higher score?\n\n- The performance of top-k SAEs is state of the art, but not beyond comparison with alternatives.\n- While top-k SAEs are a significant incremental improvement, top-k SAEs are not new, and this is a new application.", "questions": "How much better is the aux loss than the top-k loss? \nThis is interesting as it quantifies how much reconstruction quality we are 'paying' for sparsity.\n\nAt lines 132 and 452 do you believe 'an' should be 'a' (or is it not a typo)?\n\nThe following refers to work that is contemporaneous (4 months old or less), and it does not affect my rating.\nAre you aware of:\n- Jumprelu SAEs?\n- Mixture of Experts SAEs? I believe you hinted at this in the paper.\n- Sparse feature circuits? And the possibility to do evals on known circuits not just known latents (SAE seem not to be the best at factual recall).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper's strengths are as follows: \n\n## Scale:\n\nLLMs are being used in an increasing number of fields, and research into LLM interpretability is important for ensuring their fairness and safety, especially at scale.\nThis paper scales up SAEs to frontier LLMs, such as GPT4, which is a ***significant improvement*** over previous work, which was limited to smaller models. Furthermore, scaling laws are found, which could be very useful to practitioners and researchers trying to predict SAE performance ex-ante.\n\n## Top-K:\n\nThe paper's usage of top-k SAE is one of the main contributions of this work, and it is ***original*** to the extent that top-k sparse autoencoders hadn't (to the best of my knowledge) been used on LLMs before. The ***quality*** of the results of the top-k function is such that it represent the state of the art when compared to other activation functions.\n\n## Auxiliary Loss:\n\nThe authors tackle the ***significant*** issue of dead latents (up to 90% of the total when not mitigated) by introducing an auxiliary loss to the overall loss function. This incentivises the model to use all the latents, as it also supervises the next-aux latents. The ***quality*** of the results is high, as it reduces dead latents by an order of magnitude to 7%. This is to the best of my knowledge an ***original*** contribution.\n\n## Evaluations:\n\nThe paper introduces many new evaluations. I believe that explainable loss and known latent probing are the most ***significant***.\nExplainable loss is an ***original*** metric that considers the loss from just using the explanations that the SAE generates (discarding uninterpretable latents), and it represents a ***significant*** improvement over either MSE or L0 alone since it includes both reconstruction quality and an interpretability guarantee (L0 does not). Known latent probing is interesting in that it shows that the ***quality*** of probes of the latents is 25% better than that of residual stream probes.\n\n## Clarity:\n\nThe paper is very ***clear*** across all sections, and its presentation is of high ***quality***", "weaknesses": "The paper's weaknesses are as follows:\n\n## Irreducible error:\nThe paper notes that adding an irreducible error term to the scaling laws improves the laws. However, the explanation of why such an error term would exist in LLM activations is not clear, and an experiment showing that gaussian noise is hard to fit for SAEs is provided instead. The experiment does explain why such an error would be there in the first place, and its outcome is not surprising. Mentioning \"components with different amounts of structure\" is vague.\n\n## Top-K vs penalty shrinkage experiments:\nIt is expected that having larger activations for top-k SAEs won't improve the loss and that it will for sparsity penalty SAEs. This is because top-k SAEs do not have any constraint on the magnitude of the k activations they use. An experiment on shrinkage does not make the top-k activation function more attractive.\n\n## Reproducibility:\n\nAn issue with the experiments in this paper is that GPT4 is a closed source model, which makes it hard to reproduce the results of some of the experiments.\n\n## Probing datasets:\n\nA set of 61 binary classification tasks is used to train probes on the latents. It is not clear why these specific tasks have been chosen. A larger and more diverse set of tasks would have been more significant.\n\n## Unclear phrasing of top-k's contributions:\nIn line 132 there is a missing word after: \"instead of\", which I assume to be L1, but it's unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fc3wRUvaiu", "reviewer": "Reviewer_PAw6", "summary": "This paper (1) introduces Top-k sparse autoencoders (SAEs), which are novel for language models; (2) trains them on language models ranging in size up to GPT-4; (3) measures the SAE's sparsity/reconstruction tradeoff, finding performance that exceeds the state-of-the-art by a small margin; (4) demonstrate scaling laws for reconstruction accuracy in terms of training FLOPS and SAE width; and (5) measures the SAEs on a range of \"downstream\" metrics.", "questions": "If it is permitted for authors to make revisions before the final submission, there are several small changes that could improve the quality of the paper:\n\n- The paper would benefit from a thorough proofreading for typos. Examples include:\n\t- Line 129 reads: \"It removes the need for the penalty. is an imperfect approximation of , and it introduces a bias of shrinking all positive activations toward zero (subsection 5.1).\" \n\t- In line 327, the expression \"min E [y log σ(wzi +b)+(1−y)log(1−σ(wzi +b))]\" must be negated to get entropy.\n\t- Line 367-368 is missing a word after \"similar\": \"with the same n (resulting in better F1 scores) and similar (Figure 24).\"\n\t- Line 384 \"trie\" should be \"tree\".\n\n- In section 2.4, it would be good to describe the provenance of the technique \"we initialize the encoder to the transpose of the decoder\". This is described in (Conerly, 2024) (https://transformer-circuits.pub/2024/april-update/index.html#:~:text=in%20most%20cases).-,%F0%9D%91%8A,.,-The%20rows%20of), which the authors should cite as inspiration or an independent replication. The other technique in section 2.4, auxiliary loss, is properly cited to (Jermyn & Templeton, 2024) in Appendix A.2.\n\n- (Uncertain) The paper's introduction links to \"code and autoencoders for open-source models\" and a \"visualizer\". In the reviewer copy, these links are broken, likely to keep the authors anonymous. The authors should confirm that these links work in the final submission version.", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper makes multiple novel contributions to research on the application of SAEs to language models:\n\t- The Top-k SAE architecture has not previously been used on language models. (Though they were introduced previously by (Makhzani & Frey, 2013).)\n\t- Scaling SAEs to a model the size of GPT-4 is a challenge due to the increasing concentration of dead features, and the early works in this field (Cunningham et al, 2023; Bricken et al, 2023) focused on relatively small language models. The scaling done in this paper is not entirely novel, as (Templeton et al, 2024) trained SAEs on the similarly-sized Claude 3 Sonnet, but this paper likely represents contemporaneous, parallel work.\n\t- Similarly, this paper finds scaling laws that are novel except for (Templeton et al, 2024).\n\t- This paper investigates several methods for evaluating SAEs beyond sparsity and reconstruction accuracy, which are important for establishing outside validity and utility. Especially appreciated is the metric in lines 311-314, contextualizing downstream loss in terms of a fraction of pre-training compute.\n\n- In addition to the quantitative results, Top-k SAEs provide qualitative benefits over previous SAE architectures:\n\t- The sparsity level of Top-k SAEs is directly set, instead of relying on an indirect hyperparameter.\n\t- Top-k SAEs do not suffer from \"feature shrinkage\" caused by L1-regularization. The authors clearly demonstrate this in the excellent Figure 8.\n\t- Top-k SAEs have far fewer dead features at large scales.", "weaknesses": "- While the authors describe many results, they are not all equally justified.\n\t- Section 4.2 (Recovering Known Features with 1D Probes) is conceptually very exciting, but somewhat confusing in its details.\n\t\t- The magnitude of the probe loss improvement seems very low, from ~0.52 in the worst case to ~0.49 in the best case. Figures 32 and 33 indicate that a similar magnitude of improvement across various run sizes and datasets. This section would therefore benefit from more baselining: what is the best such classifier based on a residual stream neuron? On a linear  probe trained on all the residual stream neurons? On similarly-sized noise?\n\t\t- The only such baseline in the paper is that in Figure 32 the caption says that the loss on residual stream channels is 0.600. But the figure shows the loss improving from 0.46 to 0.44 over training, so it seems like almost all of the gain of the SAE occurred in the first <1% of training.\n\t\t- The probe losses appear to be averaged across the 61 datasets. This should be clearly stated in the body of the text.\n\t- Figure 7 is hard to understand, and would benefit from more explanations of its significance.\n\n- There are several typos in the body of the text (see below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)