Decomposing The Dark Matter of Sparse Autoencoders
Authors: Joshua Engels
Logan Riggs Smith
Max Tegmark
Venue: Accepted by TMLR ()

Abstract: Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in ``dark matter'': unexplained variance in activations. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark matter---about half of the error vector itself and $>90\% $ of its norm---can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior of SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SAEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations, including postulating a new type of ``introduced error''; these insights imply that the part of the SAE error vector that cannot be linearly predicted (``nonlinear'' error) might be fundamentally different from the linearly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worse, 3) it helps predict SAE per-token scaling behavior, and 4) it is responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations are inserted into the model. Finally, we examine two methods to reduce nonlinear SAE error: inference time gradient pursuit, which leads to a very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to a larger reduction.

Published in Transactions on Machine Learning Research (04/2025)
Decomposing The Dark Matter of Sparse Autoencoders
Joshua Engels MIT
jengels@miil.cm edu
Logan Smith Independent
loggnssmithh@@@mmill.cc
Max Tegmark MIT & IAIFI
legrnakk@mit edu
Reviewed on OpenReview: https: // openreview. net/ forun? id= 3ue
Abstract
Sparse autoencoders (SAEs) are a promising technique for decomposing language model ac- tivations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in "dark matter": unexplained variance in activa- tions. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark maatee--abor half of the error vector itself and > 90% of its norm can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SsEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations. These insights imply that the part of the SAE error vector that cannot be lincarly predicted ("nonlincar" error) might be fundamentally different from the lincarly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worsc, and 3) it responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations examine two methods to reduce nonlinear SAE error: inference time gradient pursuit, which leads to very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to larger reduction.
inserted into the model. Finally, we
1 Introduction
The ultimate goal for ambitious mechanistic interpretability is to understand neural networks from the bottom by breaking them down into programs ("circuits") and the variables ("features") that those programs operate on (Olah, 2023). One recent successful technique for finding features in language models has been sparse autoencoders (SAEs), which learn dictionary of one-dimensional representations that can be sparsely combined reconstruct. model hidden activations (Cunningham et al., 2023; Bricken et al., 2023). However, as observed by Gao et al. (2024), the scaling behavior of SAE width (number of latents) vs. reconstruction mean squared error (MSE) is best. fit by power law with a constant error term. Gao et al. (2024) speculate that this component of SAE error below the asymptote might best be explained by model activations having components with denser structure than simple SAE features (c.g. Gaussian noise). This is a concern for the ambitious agenda because it implies that there are components of model hidden states that are harder for SAEs to learn and which might not. be eliminated by simple scaling of SAEs.
Motivated by this discovery, in this work our goal is specifically study the SAE error vector itself, and doing so gain insight into the failures of current SAEs, the dynamics of SAE scaling, and possible distributions of model activations. Thus, direction differs from the bulk of prior work that seeks to quantify SAE

---

Published in Transactions on Machine Learning Research (04/2025)
Fraction Variance Unexplained (FVU)
0.35
SAE Reconstruction, FVU = 0.186 2.708W
SAE Reconstruction + Error Prediction, FVU 0.122
0.30
SAE Pursuit Reconstruction Error Prediction
0.25
Absent Features
0.20
0.15
Linear Error
0.10
0.05
Nonlinear Error
Encoder Error
0.00
10s
106
10
SAE Width
Figure A breakdown of SAE dark matter for layer 20 Gemma 9B SAEs, with dotted lines assuming that observed trends contimue for larger SAEs. See Section 4 for how we break down the overall fraction of unexplained variance into absent features, lincar error, and nonlinear error. See Section 6.1 for further separating encoder error from nonlinear error.
failures, these mostly focus on downstream benchmarks or simple cross entropy loss (sec e.g. Gao et al. (2024); Templeton et al. (2024); Anders & Bloom (2024)). The structure of this paper is as follows:
1. In Section 4, we introduce the fundamental mystery that we will explore throughout the rest of the paper: SAE errors are shockingly predictable. To the best of our knowledge, we are the first to show that a large fraction SAE error vectors can be explained with a linear transformation of the input activation, that the norm SAE error vectors can be accurately predicted by a linear projection of the input activation, and that on per-token level, error norms of large SAEs are linearly predictable from small SAEs.
2. In Section 5, we investigate the linearly predictable and nonlinearly predictable components of SAE error. We find that although the nonlinear component affects downstream cross entropy loss proportion to its norm, is qualitatively different from linear error: as compared to linear error, nonlinear error is harder learn SAFs for and has norm that harder linearly predict from activations, suggesting that it consists of smaller proportion of not-yet-learned linear features.
3. In Section 6, we investigate methods for reducing nonlinear error. We show that inference time opti- mization increases the fraction of variance explained by SAEs, but only slightly decreases nonlinear error. Additionally, we show that we can use SAEs trained on previous components decrease nonlinear error and total SAE error.
2 Related Work
Language Model Representation Structure: The linear representation hypothesis (LRH) (Park et al., 2023; Elhage et al, 2022) claims that language model hidden states be decomposed into sparse of lincar feature directions. The LRH has scen recent empirical support with sparse autoencoders (Makhzani & Frey, 2013; Bricken et al., 2023; Cunningham el al., 2023), which have succeeded in decomposing much of the variance of language model hidden states into such a sparse sum, as well as a long line of work that has used probing and dimensionality reduction find causal lincar representations for specific concopts (Alain, 2016: Nanda al., 2023: Marks al,, 2024; Gurnce, 2024). On the other hand, some recent work has questioned whether the linear representation hypothesis is true: Engels et al. (2024) find multidimensional
2

---

Published in Transactions on Machine Learning Research (04/2025)
circular representations in Mistral (Jiang et al., 2023) and Llama (AI@Meta, 2024), and Csordás et al. (2024) examine synthetic recurrent neural networks and find "onion-like" non-linear features not contained in a linear subspace. This has inspired recent discussion about what true model of activation space might be: Mendel (2024) argues that the linear representation hypothesis ignores the growing body of results showing the multi-dinennional structure of SAE latents, and Smith (2024b) argues that we only have evidence for a "weak" form of the superposition hypothesis holding that only some features are lincarly represented (such hypothhsis has also been studied in the sparse coding literature (Tasissa et al., 2020)).
SAE Errors and Benchmarking: Multiple works have introduced techniques to benchmark SAEs and characterize their error: Bricken et al. (2023), Gao et al. (2024), and Templeton et al. (2024) use manual human analysis of features, automated interpretability, downstream cross entropy loss when SAE recon- structions are inserted back into the model, and feature geometry visualizations; Karvonen et al. (2024) use the setting of board games, where the ground truth features are known, to determine what proportion of the true features SAEs learn; and Anders Bloom (2024) use the performance of the model on NLP benchmarks when the SAE reconstruction is inserted back into the model. More specifically relevant to our main direction in this paper, Gurnee (2024) finds that SAE reconstruction errors are pathological, that is, when SAE reconstructions are inserted into the model, they have larger effect on cross entropy loss than random perturbations with the same error norm. Follow up work by Heimersheim & Mendel (2024) and Lee & Heimersheim (2024) find that this effect disappears when the random baseline is replaced by a perturbation in the direction of the difference between two random activations.
SAE Scaling Laws: Anthropic (2024), Templeton et (2024), and Gao et al. (2024) study how SAE MSE scales with respect to FLOOP,, sparsity, and SAE width, and define scaling laws with respect to these quantities. Templeton et al. (2024) also study how specific groups of language features like chemical elements, cities, animals, and foods are learned by SAEs, and show that SAEs predictably learn these features in terms of their occurrence. Finally, Bussmann et al. (2024) find that larger SAEs learn two new types of dictionary vectors as comparsed to smaller SAEs: features not present at all in smaller SAEs, and more fino-grained "feature split" " versions features in smaller SAEs.
3 Notation
We consider neural network activations x E Ed and sparse autoencoders Sae E F2 Rd. Sparse autoen- coders map inputs to latent space Th with >> d and then back into Rd, while also requiring that only sparse set of the m latent dimensions are nonzero. SAEs have the general architecture
hidden(x) o ene (x- bdee) bene) Sae(x) Waee hidden(x) + baec
(1)
(2)
where a is an architecture dependent activation function, and seek to minimize
Csae(x) = xx Sae(x))l2 + S(hidden(x))
(3)
where S is an architecture dependent sparsity function. For convenience, we define SaeError(x) as the reconstruction error of the SAE and Lo as the average number of nonzeros in hidden(x):
SaeError(x) x Sae(x)
(4)
La Fxx (x))de((x)))
(5)
In this work, we study the relationship between and SaeError(x) and are agnostic to the specifics of SAF architecture. Thus, we examine both TopK SAEs (Gao et al., 2024) and JumpRelu SAEs (Rajamanoharan et al., 2024). The TopK SAE is defined as
Topp (z) = set all but the i largest dimensions of to zero
(6)
(bidden()))))) = 0
(7)
3

---

Published in Transactions on Machine Learning Research (04/2025)
Note that Lo k for Topk SAE. The JumpRelu SAE is defined as
{ Zi if - 0,
a Jump (2);
(8)
0 otherwise
S Romp (hiddeenx))) A ||bidden(x)]lo where 0 is a learned bias vector and À is a sparsity parameter.
4 Predicting SAE Error
In this section, we evaluate the extent to which SAE errors SaeError(x) can be predicted from input model
activations
We run experiments on Gemma 2 2B and 9B (Team et al., 2024) and Llama 3.1 8B (AI@Meta, 2024). We use the suite of Gemma Scope (Lieberum et al., 2024) sparse autoencoders for Gemma 2 2B experiments and Tlama Scope (He et. al., 2024) for Llama 3.1 8B. For experiments where we analyze the effect of In and SAE width, we focus on layer 12 Gemma 2 2B and layer 20 of Gemma 9 9B in the main text as these layers have the most SAEs in Gemma Scope, and we include additional layers the appendix (we do not run on Llama Scope for these experiments, as Llama Scope has just one SAE Lo and two SAE widths per layer). For experiments where we analyze the effect of layer, we show the set of SAFs with Lo closest to target Lo across all layers for both Gemma Scope and Llama Scope suites of SAEs.
We use 300 contexts of 1024 tokens from the uncopywrited subset of the Pile (Gao et al., 2020) and then filter to only activations of tokens after position 200 in each context, as Licberum et al. (2024) find that earlier tokens are easier for sparse autoencoders to reconstruct, and we wish to ignore the effect of token position on our results. This results in a dataset of about 247k activations. For linear regressions, we use a random subset of size 150k as training examples (since all models have a dimension of less than 5000, this prevents overfitting) and report the R² on the other 97k activations. For lincar transformations to multi-dimensional output, report the average R2 across dinensions. We include bias lerms in our linear regressions but omit them from equations for simplicity.
4.1 Predicting SAE Error Norm
For our first set of experiments, we find the optimal linear probe a" from x to ||SaeEroo(())]B. Formally (with a slight abuse of notation, since x is a random variable and not a dataset), we solve for
a = arg min aa" x - -|[seeError(x)]]]]
(10)
acR
The R² of these probes are all extremely high: across all combinations SAE width, Lps layer, and model, between 70% and 95% of the variance in SAE error norm is explained by the optimal linear probe.
Results across SAE width and La: We plot the R² of probes from Eq. (10) across SAE width and SAE Lo for layer of Gemma 2 2B and layer 20 of Gemma 2 9B contour plot (on the left) in Fig. 2. We also include plots for additional layers in Appendix A. Overall, sparser and wider SAEs have less predictable error norms, although interestingly for some layers, extremely dense SAEs have less predictable error norm as well.
Results across layers and models: In Fig. 3 we plot the R² probes from Eq. (10) for the SAEs with Lo closest to 50 on Gemma 2B and 9B and Llama Scope SAEs (which all have Lo 50). We find that the R² is low for the first few layers, increases rapidly to a value of about 0.8 to 0.95, remains at this level with slight dip for mid-late layers, and then drops off steeply in the last few layers. In Appendix A, we compare the R² of this approach across layers versus other baselines like token identity; we find that using activations does much better, and thus it is not "easy" to predict error norm.
"Cdde at https: //apony//n//. wwwp:/ywwouu...o...oooo.e siieennscienn////AE.... MaAAAE-Dar-Haaee -
4

---

Published in Transactions on Machine Learning Research (04/2025)
R² predicting error norm
R² predicting error vector
FVU
0.144
0.920
0.60
0.136
0.128
0.888
0.50
SAE LO
10a
0.120
0.872
0.45
0.112
0.856
0.40
0.104
0.840
0.35
0.096
0.824
0.30
0.088
10
0.808
0.25
SAE Width
SAE Width
SAE Width
(a) Linear prediction results for layer 12 Gemma 2 2B SAEs from Gemma Scope. fixed Lo.
PVUnollizerr is roughly constant for
R² predicting error norm
R² predicting error vector
nonnlnnnear
0.156
0.935
0.69
0.148
0.925
0.63
0.140
SAE LO
10a
0.915
0.57
0.132
0.124
0.895
0.45
0.116
0.885
0.39
0,108
10a
0.875
0.33
0.100
10
10c
10
10
105
10c
SAE Width
SAE Width
SAE Width
(b) Linear prediction results for layer Gemma 9B SAEs from Gemma Scope, FVUneulizenr roughly constant for fixed Lg.
Figure 2: Results of linearly predicting SAE error norm and SAE error from model activations on Gemma 2B layer 12 (top) and Gemma 2 9B layer 20 (bottom). The right plots show the R² of predicting SAE error norms (see Eq. (10), the middle plots show the R² of predicting SAE error vectors (see Eq. (11), and the right plots show R² of predicting model activations given the SAE reconstruction and the SAE error vector prediction. We note that these 2D heatmaps are somewhat sparse and only the black dots represent Gemma Scope SAEs, which are trained only some Lgs and SAE
actual SAEs. This is because widths. We do linear interpolation between SAEs to predict R² between hyperparameters.
4.2 Predicting SAE Error Vectors
We next examine the R² of the optimal linear transform b' from x to SaeError(x):
b² mig
b SaeError(x)]l
(11)
beRaxa
As we show in Fig. 2 (middle column) and in Appendix A for additional layers, the RD of these transforms varies widely, between 15% and 72%; this is less than the R² for our norm prediction experiments, but still much higher than we might expect. Intuitively, this result implies that there are large linear subspaces that the SAE mostly failing to learn. There is also a clear pattern across SAE Lo and width: R² decreases with increasing SAE width and Lg. Interestingly, this pattern not the same it was above for SAE error norm: the R. of error predictions increases with SAF Tap, while it decreases for vector predictions. One
5

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
1.00
16k
R² predicting error norm
0.9
32k
0.95
0.8
SAE Width (x)
65k
0.90
0.7
1aama_31120 32k
131k
llama_3.1 _arna__.1_bbb 131k
262k
0.6
gemma_ 2b 16k
0.85
gemma 2 2b 65k
524k
0.5
gemma 2_9b 16k
gemma __9b 131k
1m
0.80
16k
32k
65k
131k
262k
524k
Im
0.0
0.2
0.4
0.6
0.8
1.0
Layer by depth fraction in model
SAE Width (y)
Figure 3: R2 of SAE error norm predictions (see Eq. (10)) for Gemma Scope SAEs of width Lg 50 and Llama Scope SAEs of width Lo k 50.
Figure R2 for linear probes of per token SAE errors of larger SAEs from smaller SAEs. Predic- tion accuracy decreases the SAEs get farther apart in scale, but overall remains high.
concern might be that b* is mostly reversing feature shrinkage; in Appendix A.1, we show that this is not the case.
4.3 Nonlinear FVU and Breaking Down SAE Dark Matter
Another related metric we are interested in is the total amount of the original activation x we fail explain using both Sae(x) and linear projection of That is, assuming we have found b* in Eq. (11), we are interested in the fraction of variance unexplained (FVU) by the sum of Sae(x) and 6"
FVUnanisnear == - Ix² Sae(x) + 6" x)
(12)
We label this quantity FVUnonlinear because intuitively the amount of the SAE's unexplained variance that is not a linear projection of the input. Interestingly, we find that for middle layers (layer 12 Gemma 2 2B and layer 20 Gemma 2 9B) at fixed Lo, FVUnonlinear is approximately constant (see Fig. 2), while for other layers it decreases slightly as Lo increases (see Appendix A). That is, even though can linearly predict smaller portion of the error vector in larger SAEs, this effect is counteracted by the fact that the SAE error vector itself is getting smaller. In contrast, FVUnenitnaar decreases as SAE Lo increases.
As stated above, for middle layers we observe that FVUnonlinear is roughly constant at a fixed sparsity. Because we observe this across multiple orders of magnitude of SAE scale, we hypothesize that this will continue to hold
scale the SAF. Using this assumption, plot. horizontal fit. for FVUnenlinear and power law fit with constant for Gemma. SAE reconstructions SAE width scales (choosing the SAEs closest to Lo s 60) in Fig. 1. The power law fit asymptotes above the horizontal fit, which implies the presence of linear error even at very large SAE width. The absent features component of the error in Fig. comes from the following observation: our hypothesis correct that nonlinear error is roughly constant as the layer 20 SAEs scale, then the nonlinear features must reside in the linear error component, since this is the only component that decreases as the SAE scales and learns more features. We investigate this hypothesis further in Appendix B.2.
4.4 Predicting SAE Per- Token Error Norms
We now examine per-token SAE scaling behavior. Given two SAEs, SAE, and SAE2, we are interested in how much of the variance in error norms in SAE, is predictable from error norms in SAE, That is, we want to

---

Published in Transactions on Machine Learning Research (04/2025)
16k SAE error
65k SAE error
262k SAE error
1m SAE error
32k SAE error
131k SAE error
524k SAE error
115
Ww
L2 Norm
95
75
75
55
0
20
40
80
100
This'
article''
'link''
hypptthe
Si'
'sed'
'issues'
"about"
'impact'
evaaluaioom
Token
Figure Per token scaling with average nonlinear error, layer 20 Gemma 9B SAEs from Gemma Scope closest to Lo 60.
find c* such that
arg min c* ar ||c [ [|
(13)
CER
Note that in practice, although a' also can predict the norm of SAF error, it requires training the target SAE learn probe. Here, on the other hand, although formulate finding as an optimization problem that requires larger SAE, in practice we do not need to actually train the larger SAE to get interesting insights: since c* has just one component, it simply measures how well small SAE error by a scalar to predict large SAE error. If the R² is high, we know that on tokens that small SAEs perform poorly on, larger SAEs will as well. In Fig. 4, we plot the R² of c* probes on all pairs of Gemma Scope 9B layer 20 SAEs with Lo Rs 60 (restricting to pairs where SAEş is larger than SAE₂), and find that indeed, per token SAE errors are highly predictable. Additionally, we show concretely what these correlated SAE error norms looks like on a of 100 tokens from the Pile Fig.
be multiplied
5 Analyzing Components of SAE Error
In previous sections, we broke down the SAE error vector into a linearly predictable component and a non- linearly predictable component. A reasonable question whether these error subsets meaningfully differ. Thus, in this section, we run experiments on these components determine how the lincar and non-linear components of SaeError(x) differ. For convenience, given probe b' from Eq. (11), write
LinearError (x) b* NonlinearError(xx SaeError LinearError SaeError(x) b*
(14)
(15)
In Appendix B.2 we discuss a hypothesis for SAE fcature activations that might explain why we observe these differences; this appendix section not necessary for understanding the experiments in this section, and will refer to it in the few cases where it provides additional intuition.

---

Published in Transactions on Machine Learning Research (04/2025)
Norm Prediction Test R²
1.0
T
0.8
SAE trained on nonlinear error
SAE trained linear error
0.9
0.7
Accuracy
0.8
0.6
0.7
0.5
0.6
0.4
Sae
SaeError(k)
LinearErrorlx)
NonlinearErrorit)
0.3
Not
Q2
Decile
Figure 6: Violin plot of norm prediction tests for all SAEs (layer 12 Gemma 2B SAEs, layer Gemma 9B SAEs, all Gemma Scope SAEs with To closest to 60 across layers, and all Llama Scope SAEs). We plot the R² of lincar ro- gression from to to each random vector's norm squared.
Figure 7: Auto-interpretabiiiiy results SAFs trained on the linear and nonlinear components of SaeError(x) on a width 16k, Lu x 60, layer 20 Gemma Scope SAE, "Not" represents contexts that the SAE la- tent did not activate on, while each Qi represents acti- vating examples from decile
We run the following experiments this section to understand how the lincar and nonlincar components of SAE error differ:
1. In Section 5.1, we run the norm prediction test from Eq. (10) on different components of the error. We find that the norm of the nonlinear component is less predictable from activations than other components, implying that it might consist of fewer not-yet-learned features.
2. We train SAFs directly the linear and nonlinear components of error. The SAE trained nonlinear error converges to higher reconstruction loss and produces loss interpretable features.
3. We examine how much each component contributes downstream model performance by intervening in the forward pass, and find that both components contribute proportionally to their size to increased cross entropy loss.
Overall, these experiments suggest that our split of SAE dark matter into these two categories is indeed meaningful one.
5.1 Applying the Norm Prediction Test:
For our first experiment, we run the norm prodiction test from Eq. (10) on five different random vectors: x, Sxe(x), SaeError(x), LirearError(x),, and NonlinearError(x), The results are shown violin plot for each component across all 329 SAEs experiment with in Section 4 (layer 12 Gemma 2B SAEs, layer 20 Gemma 9B SAEs, all Gemma Scope SAEs with Lo closest to 60 across layers, and all Llama Scope SAEs) in Fig. 6 (the Sxe(x) bar can be seen as a summary of Fig. 2). There are one or two SAE with an outlier Rº equal to or lower than 0; we omit. these from the plot because they are likely due to numeric instability in the linear regression routine.
The intuition behind this test that a random vector consists sum of almost-orthogone linear features from x, then its norm should be linearly predictable from x (see Appendix B for more this intuition).
8

---

Published in Transactions on Machine Learning Research (04/2025)
R
Reconstruction + Linear Error
Norm
Total SAE Error Norm
Reconstruction + Nonlinear Error
LO
60
Width 16k
Percent Recovered When
100
Replacing Layer (%)
75
50
25
0
16k
32k
65k
131k
262k
524K
1m
11
20
36
68
138
310
427
SAE Width
LO
Figure 8: Results intervening in the forward pass and replacing x with Sae(x) NonliinearError(s] and Sae(x) + LLineaEError(x (x) during the forward pass on all layer 20 Gemma 9B SAEs with Top 60 or width of 16k. Reported in percent of cross entropy loss recovered with respect the difference between the same intervention with Sxe(x) and with the normal model forward pass.
Firstly, note that ||x|| almost be perfectly predicted from This is reassuring news for the linear representation hypothesis, it implies that x may be well modeled as the sum of many one-dimensional features, at least from the perspective of this test. We also find that NonlinearError (and SaeError (x), which consists partly of NonlinearError(x) has a notably lower score on this test than LinearError (x). This result suggests that NonlinearError(X does not consist of a sparse sum of linear features from x.
5.2 Training SAES on SaeError(x) Components:
Another empirical test run is training SAE NonlinearError(s) and LinearError(x). We choose fixed Gemma 9B Gemma Scope layer 20 SAE with 16k latents and Lo x 60 to generate SaeError(x) from. This SAE has nonlinear and linear components of the error approximately equal in norm and R²² of the total SaeError(x) they explain, so presents a fair comparison. We train SAEs to convergence (about 100M tokens) on each these components error and find that the SAE trained on NomlinearError(S converges to fraction of variance unexplained an absolute percent higher than the SAE trained on the lincar component of SAE error (= 0.59 and z 0.54 respectively).
We additionally examine the interprelabilily of the learned SAE latents using automated interpretability (this technique was first proposed by Bills et al. (2023) for interpreting neurons, and first applied to SAEs by Cunningham et al. (2023)). Specifically, we use the implementatiin introduced by Juang et al. (2024), where a language model (we use Llama 3.1 70b (AIeMeta, 2024)) is given top activating examples to generate an explanation, and then must use only that explanation to predict the feature fires on a test context. Our results in Fig. show that the SAE trained on linear error produces latents that are about an absolute 5% more interpretable across all activation firing deciles (we average results across 1000 random features for both SAEs, where for each feature use 7 examples in each of the 10 feature activation deciles well as 50 negative examples, and show 95% confidence intervals).
5.3 Downstream Cross Entropy Loss of SaeError(x) Components:
A common metric used test SAEs the percent of cross entropy loss recovered when the SAE reconstruc- tion inserted into the model in place of the original activation versus ablation baseline (see e.g. Bloom

---

Published in Transactions on Machine Learning Research (04/2025)
Pursult Reconstruction, FVU 0.136 2.342W 2212
Predicting Nonlinear Error
Predicting SAE-Error
Predicting Linear Error
Pursult SAE Reconstruction Error Prediction, 1.170.117
0.60
0.49
0.3
0.40
R
0.34
0.32
2
0.2
0.20
0.16 0.18
Unleamed Features
0.14
0.06
0.09
0.07
0.10
Déase heatures Linnaa Ercor
0.1
0.00
Nonlinear Error
residual
prev residual
MLP
attention
residual
prev residual
MLP
attention
residual prev residual
MLP
attention
(from Dictionary and Sparsity)
0.0
105
106
107
SAE Width
(a) SAE error breakdown vs. SAE width for infer- ence time optimized reconstructions from Gemma Scope Lo 60 dictionaries.
(b) The It of linearly predicting parts of SAE error from the SAE reconstructions of adjacent model components, layer Gemma Scope Lo s 60, 16k SAE width.
Figure 9: Investigations towards reducing nonlinear SAE error.
(2024)). We modify this test to specifically examine the different components of SaeError(x)) we compare the percent of the cross entropy loss recovered when replacing x with Sae(x) plus either LinearError (x) or NonlinearError(2
the baseline of inserting just Sae(x) in place of on all layer 20 Gemma 9B SAEs with Lo 60 width of 16k. To estimate how much each component "should" recover, we use two metrics: the average norm of the component relative to the total norm of Sae(x) and the percent of the variance that the component recovers belween Sae(x) and x. The results, shown in Fig. 8, show that for the most part these metrics are reasonable predictions for both types of error. That is, both WonlinearError(x: and LinearError(x)) proportionally contribute to the SAE's increase in downstream cross entropy loss, with possibly a slightly higher contribution than expected for LinearError(x)..
6 Reducing NonlinearError(x)
In past sections we identify NonlinearError(x) and LinearError(x) and argue that NonlinearExror(2x likely does not consist of not-yet-learned linear features. This may be problem for scaling SAEs farther: if part of SAE error persistently consists of nonlinear transformations of lincar features, these may never be able to be learned by SAFs. Thus, in this section, we investigate two approaches for reducing NonlinearError
1. Using a more powerful encoder at inference time. This approach has limited success, reducing total FVC by 3-5% but leaving FVUnonlinear almost unchanged
2. Attempting to predict NonlinearError(x) from SAE reconstructions of adjacent model components this approach is more successful; the SAE test on, Sae(x) predicts 50% of the nonlinear error, and previous model components predict between 4% and 14% of nonlinear error, although only between 6% and 10% of the overall SAE error.
6.1 Using More Powerful Encoder
Our first approach for reducing nonlinear error is to try improving the encoder. We recent approach suggested by Smith (2024a): applying greedy inference time optimization (TTO) algorithm called gradient pursuit to frozen learned SAE decoder matrix. We implement and run ITO on all layer 20 Gemma Scope 9b SAFs closest. to Ln ~ 60. For each example x with reconstruction Sae(x), we use the gradient pursuit implementation with an Lo exactly equal to the Lo x in the original Sae(x).
Using these new reconstructions of x, we repeat Eq. (11) and do a linear transformation from x the inference time optimized reconstructions. We then regenerate the same scaling plot as Fig. and show this
10

---

Published in Transactions on Machine Learning Research (04/2025)
figure in Fig. 9a. Our first finding is that pursuit indeed decreases the total FVU of Sae(x) by 3 to 5%; Smith (2024a) only showed an improvement on small 1 layer model, to the best of our knowledge we are the first to show this result on state of the art SAEs. Our most interesting finding, however, is that the FVUnealinear stays almost constant when compared to the original SAE scaling in Fig. 1. We hypothesize that this might happen because ITO reduces "easy" linear errors like feature shrinkage. Fig. 1, we plot the additional reduction in FVnooninearr as the contribution of encoder error; because FVUnonlinear stays almost constant, this section is very narrow.
6.2 Predicting SAE Errors Between SAEs:
For the Gemma 2 architocture the locations the SAEs are trained on, cach residual activation can be decompased in terms of prior components:
Residhayer MlpOut (iSSorr((peog + Residisyer-1
(16)
We focus on layer = 20 and Gemma 2 9B, and thus use the layer 19 attention and MLP Gemma Scope SAEs. For all components we use the width 16k SAEs with Lo z 60. In Fig. 9b, we plot the Ri² of a regression from the SAE output corresponding to each of these right hand side components to each of the different components of an SAE trained on Residisyer (SaeError(x), LinearError(x)) and MonlinearError(x))
We find that we can explain small amount (up to 10%) of total SaeError (x) using previous components, which may be immediately useful for circuit analysis. We also can explain large parts of NonlinearBrror with prior components as well. While Sxe(x) explains 50% of the variance in the nonlinear error, this may be somewhat misleading, as the nonlinear error is partially function of Sae(x):
NonlinearError(x = SaeError(x) - LinearError(x:
= (x Sxe( LinearError(x)
These results mean that we might be able explain some of the SAE Error using circuits level view, but that overall there still large parts of each component unexplained.
7 Conclusion
The fact that SAE error can be predicted and analyzed all is surprising. Thus, our findings are intriguing evidence that SAE errors, and not just SAE reconstructions, are worthy of analysis, and we hope that it inspires further work in decomposing SAE error.
Concretely, we believe that our study of error has already discovered a number of promising directions for future SAE research. The discovery that SAE errors can be significantly predicted from model activations hints that some subspaces may resist. being learned and points Loward potential architectural innovations such as incorporating low-rank dense side channels. The predictable relationship between small and large SAE crrors may also strcamline experimentation with novel SAE architectures by allowing rescarchers to efficiently forecast scaling behavior through small-scale trials. Additionally, our demonstration that prior SAF outputs can explain part of SAF error has immediate practical implications for circuit analysis, which currently relies on large noise terms that complicate circuit interpretation.
At a higher level, the presence constant nonlinear error at a fixed sparsity as we scale implies that scaling SAFs may not be the only (or best)) way to explain more of model behavior. Future work might explore alternative penalties besides sparsity or new ways to learn better dictionaries. Ultimately, we believe that there is still room to make SAEs better, not just bigger.
Acknowledgments
Our work benefited greatly from the thoughtful comments and discussions provided by (in alphabetical order) Joseph Bloom, Lauren Greenspan, Jake Mendel, and Eric Michaud. We are deeply appreciative of their contributions. This work was supported by Erik Otto, Jaan Tallinn, the Rothberg Family Fund for
11

---

Published in Transactions on Machine Learning Research (04/2025)
Cognitive Science, and IAIFI through NSF grant PHY-2019786. JE was supported by the NSF Graduate Research Fellowship (Grant No. 2141064). LS was supported by the Long Term Future Fund.
References
Al@Meta. Llama 3 model card, 2024. URL https:ttpp/// httplattpp/gggitbbbm//////aaaaee http:p//ghghhhoooooooee///////// CARD md.
Guillaume Alain. Understanding intermediate layers using linear classifier probes.
arXiv preprint
arXi::161000164,, 2016.
Evan Anders and Joseph Bloom. Examining language model performance with reconstructed acti- vations using sparse autocncoders. Less Wrong, 2024. URL https://www wew.swow. com/posts/
Transformer Circuits Team Authropic. Circuits updates april 2024, 2024.
URL https://
sransformers circuitsrrrrrrrss.eem wwweccuiss.uuub///eee ppbpppppp/////// htt/4//prrl-uddae///nnddoeeo htm.mscaalllg-eaa
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URT. https:// butps://openaipullee blob. core, windows. 2023), 2, 2023.
eaplainerrmppprr/riiieeee /inder.. html. (Date accessed: 14.05.
Joseph ers
Bloom. of gpt2 small.
Open
source
sparse autoencoders for all residual stream lay- htrp /ttp:psss/EEggglLurrii......./ 2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monose- manticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. huups://transforme--rrroet put/:2223/tt http://////mooommaaiiiiieeee fewtures/index..heeeee
Bart Bussmann, Patrick Leask, Joseph Bloom, Curt Tigges, and Neel Nanda. Stitching saes of different sizes. Al Aligument Forum, 2024. URL https: www.:wwlggmmetffo..om wowpeetffooooooooo/oooooooo wrggposts////ooooooooooooooossst wwtp.//sss/ballyppttttattaa btt.yypptttmmmmosss///////////// stitching saes- sf www.ooffffdtt--eefeeeeee sizes. sizzs..
Emnanuel Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Róbert Csordás, Christopher Potts, Christopher D) Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations, arXiv preprint arXiv:2/08.10990,
2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiu:2309.080 086000, 2023.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatffeld-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. https://transformeer cireuitsiirr....... wtw.ruiis..uub/////////// hatol///toommdde///atet /indelx//nexxoo html.
Joshua Engels, Isaac Liao, Eric J Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405. 14860, 2024.
Simon Foucart and Holger Rauhut. Mathematical Introduction Compressive Sensing. Applied and Numerical Harmonic Analysis. Springer, New York, 2013. ISBN 978-0-8117-49447- doi: 10.1007/ 978-0-8116--948-7
12

---

Published in Transactions on Machine Learning Research (04/2025)
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint (rXic:2101.00027 2020.
Lco Gao, Tom Dupré Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Nla Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. ar preprint arXiv:2406.040 2024.
Giorgi Giglemiani, Nora Petrova, Chatrik Singh Mangat, Jett Janiak, and Stefan Heimersheim. Evaluating synthetic activations composed of sae latents in gpt-2. arXiv preprint or S.3::2449.150111 2024.
Wes Gurnee. Sac reconstruction errors are (empirically) pathological. In AI Alignment Forum, pp. 16, 2024.
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama scope: Extracting millions of features from 1lama-3.1-8b with sparse autoencoders. arXiv preprint arXiv:2410.20590 2024.
Stefan Heimersheim and Jake Mendel.
Activation plateaus & sensitive directions in gpt2. Less Wrong, 2024. URL https://www..eeswrrrr com./passs//jjjyyyyyyy//aaaaal
interim wwwpaaaddppppttttttttttttooraaae
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2810.000 06825, 2023.
Caden Juang, Gonçalo Paulo, Jacob Drori, and Nora Belrose. Open source aulomated interpretability for sparse autoencoder features. https://blog eleuther ai/autointerp/ 2024.
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Clan- dio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interprelability with board game models. arXiv preprinl arXiv:2408.001 2024.
Vedang Lad, Wes Gurnee, and Max Tegmark. The remarkable robustness of Ilms: Stages inference? arXiv preprint ar Xiv:2400 19384, 2024.
Daniel Lee and Stefan Heimersheim. Investigating sensitive directions gpt-2: An improved base- line and comparative analysis of sacs. Less WrosWr 2024. URL https://www lesswrong. com/posts/ annpi--vee-
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autocncoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.051/ 2024.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. orXiv preprint arXiv:1312.5663 2013.
Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint STKi:2403.1964/4 2024.
Jake Mendel. Sae feature geometry is outside the superposition hypothesis. Al Aliga- ment Forum, 2024. URL https://ww// www.gmentfoorummmmmmm... http://tttttoooooorrr///ss/////e toee-feaurre-ggmmery-tet
Necl Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. arXiv preprint arXiv:2309.009/1 2023.
Chris Olah. Interpretabilityy dreams.
Transformer Circuits, May 2023.
URL https://
transformer-circaite. pwtpppp///nteeeeeeeeee http:///ww........ooooaaaaaaaaaa html.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Kiho Park, Yo Joong Choc, and Victor Veitch. The lincar representation hypothesis and the geometry of large language models. Xiv preprint or arXiv:2311.03668, 2023.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse tocncoders. arXic preprint arXi:2207714438. 2024.
Lewis Smith. Replacing sae encoders with inference-time optimisation. https://www.cn www.:/wwwwwg...nnrrrrr. http://AA4...300m0000/////////22
2024a.
Lewis
Smith. The 'strong' feature hypothesis could be wrong. AI Aligmment Fo- URL
rum, 2024b. seatuurettppohhsi---oud------e
Abiy Tasissa, Manos Theodosis, Bahareh Tolooshams, and Demba E Ba. Discriminative reconstruction via simultaneous dense and sparse coding. Transactions on Machine Learning Research.
Abiy Tasissa, Emmanouil Theodosis, Bahareh Tolooshams, and Demba Ba. Towards improving discrimina- tive roconstruction via simultancous dense and sparse coding. arXiv preprint arXiv:200..000 09534, 2020.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models practical size. orXiv preprint arXiv:2408. 00118,, 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Cal- lum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https: htt.r/uuuttstpp//////saaaaaaa
/ //rransformerrccicutti gub/2024/scalinae attp/22222..aaagggnoommmaaaaaii
html.
14

---

Published in Transactions on Machine Learning Research (04/2025)
R² predicting error norm
R² predicting error vector
FVUnonilnea
0.125
0.928
0.475
0.115
10a
0.896
0.105
SAE LD
0.880
0.375
0.095
0.864
0.325
0.085
0.848
0.275
0.075
0.832
0.065
10a
0.816
0.175
SAE Width
SAE Width
SAE Width
(a) Linear prediction results for layer 5 Gemma 2 2B SAEs from Gemma Scope.
R² predicting error norm
predicting error vector
FVU Toniineeaa
0.145
0.135
0.425
0.125
10a
0.375
SAE LO
0.81
0.115
0.78
0.105
0.275
0.75
0.225
0.085
0.075
10
10s
10c
SAE Width
SAE Width
SAE Width
(b) Linear prediction results for layer 19 Gemma 2 9B SAEs from Gemma Scope.
Figure 10: Results of linearly predicting SAE error norm and SAE error from model activations on Gemma 2B layer (top) and Gemma 2 2B layer 19 (bottom). The right. plots show the R of predicting SAF error norms (see Eq. (10), the middle plots show the R82 of predicting SAE' error vectors (see Eq. (11), and the right plots show 1 - R of predicting model activations given the SAE reconstruction and the SAE error vector prediction. Unlike for the middle layers shown in the main body, nonliinnes decreases some width is scaled, although the sparsity of the SAEs in the space makes this harder to verify.
A Extra Error Prediction Experiments
A.1 Note on Feature Shrinkage
Earlier SAE variants prone to feature shrinkage: the observation that Sae(x) systematically undershot x. Current state of the art SAE variants (c.g. JumpReLU SAEs and TopK SAEs, which we examine in this work), are less vulnerable to this problem, although we still find that Gemma Scope reconstructions have about a 10% smaller norm than x. One potential concern is that the b' in Eq. (11) that we learn is merely predicting this shrinkage. If this was the case, then the cosine similarity of the linear error prediction (b+)T". with x would be close however, in practice find that it is around 0.5, so 6a is indeed doing more than predicting shrinkage.
15

---

Published in Transactions on Machine Learning Research (04/2025)
R² predicting error norm
R² predicting error vector
FVUnonilnea
0.944
0.575
10a
0.912
0.475
SAE LD
0.105
0.896
0.375
0.095
0.880
0.864
0.275
10a
0.848
0.225
0.065
10c
SAE Width
SAE Width
SAE Width
(a)
Linear prediction results for layer 9 Gemma 2 9B SAEs from Gemma Scope.
R predicting error norm
predicting error vector
FVU nonnineeear
0.144
10a
0.885
0.525
0.128
0.870
SAE LO
0.855
0.475
0.120
0.840
0.425
0.112
0.825
0.104
0.375
0.810
0.795
0.325
0.088
10a
0.080
10
10a
10c
SAE Width
SAE Width
SAE Width
(b) Linear prediction results for layer 31 Gemma 2 9B SAEs from Gemma Scope.
Figure 11: Results of linearly predicting SAE error norm and SAE error from model activations on Gemma layer (top) and Gemma 2 9B layer (bottom). The right. plots show the R of predicting SAF error norms (see Eq. (10), the middle plots show the R82 of predicting SAE' error vectors (see Eq. (11), and the right plots show 1- R of predicting model activations given the SAE reconstruction and the SAE error vector prediction. Unlike for the middle layers shown in the main body, FYUnonlinear decreases some width is scaled, although the sparsity of the SAEs in the space makes this harder to verify.
A.2 Additional Error Prediction Plots
In Section 7 and Section 7, we show the accuracy of SAE error norm predictions, SAE error vector predictions, and FVUnonlimeer for layers and 19 of Gemma 2 2B and layers 9 and 31 of Gemma 9B. As described in the main text, we find broadly similar results for SAE error norm and error vector prediction at these layers, but find that FVUnonlinear decreases some as we increase SAE width at fixed Lo, although this result is uncertain because of the sparsity of Gemma Scope SAEs at these layers.
A.3 Norm Prediction Baselines
In Fig. 13, we run a linear regression from different model variables to [|SaeEroo((x]] across layers on Gemma Scope 9B for SAEs of size 131k. We find that not "casy" to prodict SAE error norm, especially at later layers; the token identity, SAE LO, activation norm, and model loss all do significantly worse than using the full activation. It is interesting to note that at the first few layers, token identity does better
16

---

Published in Transactions on Machine Learning Research (04/2025)
Norm of Linearly Predictable Error
Norm of Non-Linearly Predictable Error
100
SAE Width
16k
L2 Norm
32k
65k
131k
262k
524k
0
20
40
60
80
100
20
40
60
80
100
Token
Token
Figure 12: Per-token breakdown of linearly predictable and non-linearly predictable SAE error across SAE scale. We show the same tokens as in Fig. 5. The norm linear error decreases with SAE width, whereas the norm of nonlinear error stays mostly constant.
predicting SAE error than a probe of the activations; this is perhaps not surprising, since recent results from o.g. Lad et al. (2024) show that very carly layers primarily operate on per token level.
A.4 Breaking Apart Error Per Taken
In Fig. 12, we show the same subset of tokens as in Fig. 5, but now broken apart into linearly predictable and non- con-linearly predictable components. That is, learn 6" for each SAE as in Eq. (11), and then plot the norm of b' x at the norm of the linearly predictable error the left, and plot the norm of SaeError(x)) as the norm of the non-linearly predictable error the right. We see that the linearly predictable error decreases
scale SAE width, but the non-linearly predictable error mostly stays constant. This is especially interesting because the result in Fig. just found this on an average level, whereas here we find the same result holds on a per-token level.
Modeling Activations
In this section, we soek explain why we see difference between linear and nonlinear crror in the main body of the paper. We will adopt the weak linear hypothesis (Smith, 2024b), generalization of the linear representation hypothesis which holds only that features in language models represented linearly. Thus we have
X
w.y: + (ense(x)
(17)
for linear features (......, Yn and random vector E R" where is sparse ([]w/) < d) and Dense(x) is a random vector representing the dense component of x. Dense(x) might be Gaussian noise, nonlinear features as described by Csordás et al. (2024), or anything else not represented in low-dimensional linear subspace.
Before we proceed with our analysis, we note that there a rich sparse coding literature studying dictionary learning in the setting with mixed sparse and dense signals. For example, a classic work Candès et al. (2011) propose Robust Principal Component Analysis for decomposing data matrices into sparse and dense components, although this is not directly applicable recently, Tasissa et al. propose dictionary learning technique for data exactly modeled as in Eq. (17). In our work, we focus on studying existing SAEs and hypothesizing why they fail, so these works are not immediately applicable, but we are excited to see future work that applies these combined sparse-dense autoencoders to language model activations..
setting of trying to learn autoencoder. More
Say our SAE has latents. Since by assumption (ense(x) cannot be represented in a low Now-dimensional linear subspace, the sparsity limited SAE will not be able to learn it. Thus, will assume that the SAE learns
17

---

Published in Transactions on Machine Learning Research (04/2025)
(a feature
Running window average of (a " feature
Acts
Model Loss
Norm
Average feature value when active
Tokens
LO
1.00
200
0.75
100
R^2
0.50
0
0.25
-100
-00
0.00
2500
5000
7500
10000
10
20 Layer
30
40
SAE Latent Index
Sorted by Average Active Value
Figure 13: R2 for linear regressions of SAE error norms with different regressors. We run on Gemma Scope 9B SAEs of size 131k with Lo s 60, Activa- tions perform the best except the first few layers.
Average SAE latent activation and dot product of the latent with the learned norm predic- tion vector a for the Gemma Scope layer 20, width 131k, Lo 62 SAE. We also plot smoothed ver- sion of this dot product with smoothed window of 10.
Figure 14:
only the most common features you...,, Vom-.. We will also assume that the SAE introduces some error when making this approximation, and instead learns Si: and w. Thus we have
Sae(x) = Σ
(18)
0=0
SaeError(x)) = Dense(x)) +
Σ Wiyi
W.ys
+
wayk
(19)
1=0
f=
We finally define Introduced(x) En v.yh: Dizn wiys
we have
SaeError(x) (xnse(x) + Introduced(x)
w.Y:
(20)
B.1 Analyzing Error Norm Prediction
We will first analyze Eq. (10), the learned probe from x to |SaeError(x)]3- First, we claim that given vector x, if x a sparse sum of orthogonal voctors, then by basic lincar algebra there exists a perfect prediction vector such that a" x |x|2 (in other words, the norm squared of can be linearly predicted set the probe vector a* to
from x). The proof of this claim is in Appendix C.1; the intuition is that the sum of the vectors y: weighted by their average weight E(w:).
When X is instead a sparse sum of non-orthogonal vectors, as it partly is in Eq. (17) and Eq. (20), this proof no longer true, but we now argue that similar intuition holds. the ys are almost orthogonal (formally coherence c, see (Foucart & Rauhut, 2013)) and do not activate much at the same time, then probe vector again equal the sum of vectors yi weighted by their average value E(w,) will be good approximate prediction. Indeed, when try predicting [|Sae(x)]l from Sae(x) (which is sparse sum of
18

---

Published in Transactions on Machine Learning Research (04/2025)
known almost orthogonal vectors of similar distribution to the true SAE vectors), we find that indeed the linear probe that is learned is approximately equal to this sum (see Appendix C.2).
Thus, we can now neatly explain why we can predict the norms of SAE errors: they mostly consist of almost orthogonal sparsely occuring not yet learned SAE features! We further can explain why larger SAEs have less predictable error norms: since m is larger, there is a larger component the error of not-as-linearly- predictable Dense(x) and Introducedxx)..
B.2 Analyzing Error Vector Prediction
We will now analyze Eq. (11), the learned transformation from x to SaeError(x), with our model of SAE error from Eq. (20). We assume that Introduced(x) cannot be approximated at all as a linear function of x. This is a reasonable assumption since the SAE is nonlinear function of x, but indeed some of Introduced(x) can be approximated in this way, then we will underestimate the amount of Introduced(x) and therefore overestimate the amount of (x)
If Dense(x) + Ei=m w;y; is contained in a linear subspace of x orthogonal to Ei=o Wayi, then Dense(x) is part of the linearly explainable error, and the error of the transformation b" exactly equals Introduced(x) (since the transformation is just exactly this orthogonal linear subspace). However, if such an orthogonal linear subspace does not exist, the optimal linear transform will only be able to reconstruct part Dense(x) + Ei=m w.y,, and the percent of variance left unexplained by the regression will be an upper bound on the truc variance explained by Introduced(x)) We also note that if this linear transform indeed recovers Dense(x) and absent features but fails to recover Introduced(x),, we can use it to estimate Dense(x): the difference between the variance explained by Sae(x) and the variance explained by (Sae(x) + b' b' will approach Dense(x) as m + Dc.
Thus, our ability to estimate Introduced(x) and Dense(x) using b' depends on how well a linear transform works to predict Dense(x) and >imm w.y,. Although we do not have access to the ground truth vectors yo we can replace x' with a similar distribution of vectors that we do have access to, using the same trick as above. Given SAF, we replace with x' Sae(x). x' has the useful property that it is sparse linear sum of vectors (the ones that the SAE learned), and the distribution of these vectors and their weights are similar to that of the true features y:- We now pass x' back through the SAE and can control all of the quantities we are interested in: we can vary m by masking SAE dictionary elements, simulate Dense (x') by adding Gaussian noise to x', and simulate Introduced(x)) by adding Gaussian noise to Sae(x').
We run this synthetic setup with a Gemma Scope layer 20 SAE (width 16k, Lo- 68) in Appendix C.3, and find that indeed, estimated Dense(x') is highly correlated with the amount of Gaussian noise added to x' and Introduced (x') is highly correlated with the amount of Gaussian noise added to Sae(x') (see Ta- ble 1). However, note that because x' noise also slightly correlated with estimated Introduced (x'), is possible that some of the contribution to the estimated nonlinear error is from Dense (x').
Table 1: Correlation matrix between synthetic noise and esti- mated errors.
Estimated Dense (x')
Estimated
Introduced (x')
x' Noise
0.9842
0.1417
Sae(x') Noise
0.0988
0.9036
Thus, again now have potential explanation for our initial results: we can predict error vectors because they consist in large part of not yet Icarned lincar features in an almost orthogonal subspace of x, we can predict a smaller portion of larger SAE errors because the number of these linear features go down with SAE width, and the horizontal line in Fig. 1 is because Dense(x) and Introduced(x) are mostly constant. Furthermore, we can hypothesize from the corrciations om Table 1 that the lincarly predictable component of SAE error consists mostly of not yet learned features and Dense(x), while the component that is not linearly predictable consists mostly of Introduced(x)) We will explore this hypothesis in Section 5.
19

---

Published in Transactions on Machine Learning Research (04/2025)
B.3 Analyzing Per- Token Scaling Predictions
Finally, we provide a simple explanation for why per-token SAE errors are highly predictable between SAEs of different sizes. For this, we only need Eq. (20). Since Dense(x) and Introduced(x) stay mostly constant as m increases, for large m the SAE error stays mostly constant because it is primarily determined by these components. Thus, since m = 16k is already large, a linear prediction that is just a slightly smaller version of the current error performs well. Additionally, this reasoning suggests natural experiment: if we can predict Introduced(x) on per-token level (which we hypothesize we can do with the non-linearly predictable ble component of SAE error), we may be able to better predict the floor of SAE scaling and therefore better predict larger SAE errors; run this experiment in Appendix E,, where we find an affirmative answer.
C More Info on Modeling Activations
C.1 Proof of Claim from Appendix B.1
Say we have a set of m unit vectors yı, yy....., Um E Rd. We will call these "feature vectors". Define Y E Idd= as the matrix with the feature vectors as columns. We then define the Gram matrix Gy E Rm of dot products on Y:
(Gy) = (Y" y: y.
We now will define a random column vector x that is a weighted positive sum of the m feature vectors, that is, x = E, w,y, for a non-negative random vector w E R". We say feature vector y; active if w; > 0. We now define the autocorrclation matrix R.w E mm for w as
R = R=E((w>) (ww"")
We are interested breaking down x into its components, so we define a random matrix X as X,y = wj Yij. i.e. the columns of multiplied by w. We can now define the Gram matrix Gx E Rm
(Gx)i =xx"); wiw,yi 3 Gx (ww") o Gy
E(Gx) R Gr.
where @ denotes Schur (elementwise) multiplication. The intuition here is that the expected dot product be- tween columns X depends on the dot product between the corresponding columns of and the correlation of the corresponding elements of the random vector.
We will now examine the L2 norm of
| 2
w.W,y.y,
(( w) + Grll2 Tr(ww'Gy) wGyw"
We can also take the expected value:
E 12) = Tr(R
Our goal to find a direction a E Idd that when dotted with x prodicts ||x|| In other words, we want to find a such that.
|xx xxx= a" Σ way. a Yw
Combining equations, we want find such that
a Yw ~ |xx12 (vwTGrw) Gyw)
20

---

Published in Transactions on Machine Learning Research (04/2025)
Let us first consider the simple case where for all # yi and Vy perpendicular. Then goal is to find a such that
a Yw (WwxI((Grr")) -wn w/ Ene w/ = 1//ll2 w www
Since all of the ys are perpendieular, WLOG we can write all ys and vector b R.. Then we have
Ei biyb c for a vector c E R perpendicular
T
(Eber
a T Yw
bi +
Yw
b"
Since ordinary least squares produces an unbiased estimator, know that if we use ordinary least squares to solve for b, B(b"w) E(wTw) w). Thus,
Σ b,B(w) = E(( b, = E(wF)//(/Bn)
Now that we have b,, we can solve for the correlation coefficient between a"x b"w and ||x| = wTw w. This gets messy when using general distributions, so we focus on a few simple cases.
The first is the case where each w; scaled independent Bernoulli distribution, so w; is st with probability Pi and 0 otherwise. Then b, 8t. We also have that E(w"" (Tw) E(b" w) = E, sip. μ.
Eb" ww" w) 122
B[ ww w) 112 /E" wbww w) 112
Σ, (p: p?
That is, for Bernoulli variables, x E, S:9s is a perfect regression vector.
The second is the case when each w; is an independent Poisson distribution with parameter A. Then [(w,)) = Xa and [(w?) = X + Ax so 6, = A; + 1. We also have that E(w w) = E( w) = -
Finally, we will use the fact that E(w?) Ae 3X? A and E(w!) = X 3 7X + A. Then via algebra we have that
Σ, 2 3X A
VE, 413
613 X/ Σ, X 2X X
For the special case X; 1, we then have
p 66
2 0.73
C.2 Empirical Norm Prediction
In this experiment, aim to determine to what exlent analysis in Appendix B.1 holds true in practice almost orthogonal true SAE features. Thus, Specifically, we learn a probe a" for the Gemma Scope layer 20, width 131k, Lo 62 SAE as in Eq. (10), except with the regressor equal to Sxe(x) and the target equal to [Sae(x)]]
random vector that control: Sae(x).
a = min | Sae(x) - [Saa(()
(21)
acR
21

---

Published in Transactions on Machine Learning Research (04/2025)
Sae(x') Noise Var 1, x' Noise Var 0.6
Sae(x') Noise Var 0.6, x' Noise Var 1
Uneer Error ['Attvationn Noise)
FVU
0.1
FVU
0.1
Linear Error
f(Activation Noise)
Nonlinear Error To f(Reconstruction Noise)
Nonlinear Error [ ||Recnsttruttoo Noise)
0.0
0.0
50%
40%
30%
20%
10%
50%
40%
30%
20%
10%
0%
% features ablated
% features ablated
True Linear Noise Var
True Linear Noise Var
Est Nonlinear Error (FVU difference)
Est Linear Error (FVU difference)
0.0
0.0
0.4
1.0
1.0
1.0
2.0
2.0
0.2
0.5
3.0
3.0
4.0
4.0
0.0
1.0
2.0
3.0
4.0
0.0
1.0
2.0
3.0
4.0
True Nonlinear Noise Var
True Nonlinear Noise Var
Figure 15: Top:
When controlled amounts of noise are added synthetic data Sae(x') and x', the result is plot similar to Fig. 1. Bottom: The nonlinear and linear error estimates (as shown at top) accurately correlate with the amount of noise added. The exact correlation between synthetic added noise and resulting estimated error components across these noise levels are shown in Table 1
One important note that we subtract the bias from Sae(x) so that it is purely sparse sum of SAE features (this makes analysis easier). For each SAE latent from the SAE, we then compute
(a*)² latent;
(22)
Finally, we plot this dot product against the average latent activation in Fig. 14. If a" indeed equals the sum of the latents weighted by their activation, we predict in Appendix B.1, then these two quantities should be approximately equal, which we indeed see in the figure.
C.3 Synthetic SAE Error Vector Experiments
The results for different Gaussian noise amounts versus percentage of features ablated are shown in Fig. 15. On this distribution of vectors, the test works as expected; the variance explained by Sae(x) + a x is a horizontal line proportional Introduced(x), while the gap between this horizontal line and the asymptote of the variance explained by Sae(x) proportional Dense(x).
We also tried rumning this test sparse sum of random vectors, which did not work as well, possibly due to not including the structure of the SAE vectors (Giglemiani et al, 2024); see Appendix D for more details.
D Synthetic Experiments with Random Data
For this set of experiments, generated random vector x' that the sum of power law of 100k random gaussian vectors in 4000 with expected Lu of around 100. To simulate the SAE reconstruction and SAE error, we simply masked a portion of the vectors in the sum of x'. Unlike the more realistic synthetic data case we describe in Appendix C.3, this did not work as expected: even in the case with no noise added x' or the simulated reconstruction, the variance explained by the sum of the lincar estimate of the error plus the reconstructed vectors plotted against the number of features "ablated" formed parabola (with minimum variance explained in the middle region), opposed to straight line in Fig. 15.
22

---

Published in Transactions on Machine Learning Research (04/2025)
16k
32k
SAE Width (x)
65k
4
131k
262k
524k
1m
0
16k
32k
65k
131k
262k
524k
1m
SAE Width (y)
Figure 16: Percent decrease in FVU when additionally using the squared norms of nonlinear error to predict SAE error norm.
We note that this result is not entirely surprising: other works have found that random vectors are bad synthetic test case for language model activations. For example, in the setting of model sensitivity to perturbations of activations, Giglemiani et al. (2024) found they needed to control for both sparsity and cosine similarity of SAE latents to produce synthetic vectors that mimic SAE latents when perturbed.
E Using |NoolinearErrror()) to Predict Scaling:
Following up on our discussion in Appendix B.3, we are interested in whether Nonl ix) can help with predicting SAE per-token error norm scaling, as this might suggest that contains a larger component of Introduced(x) than LinearError(x). Formally, we solve for
d' == arg min deR
dd [SaeError: (x), ionl (x)) SAE2(x)]12
(23)
To evaluate the improvement of d' relative to e from Eq. (13), we report the percent decrease in FVU; see Fig. 16. We find that using the norm of NonlinearError(x) provides a small but noticeable bump in the ability to predict larger SAE errors of up to 5% decrease in FVU, validating this hypothesis.
23


Reviews:
[{"review_id": "V7SJu0nETN", "reviewer": "Reviewer_WABm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOverall the paper is well-written and for the most part intuitive. To my knowledge the results are novel, and this is the first work I’ve seen studying the error of the SAEs. I think this can prove to be an interesting direction for future works, where further emphasis is placed on understanding the tools we use beyond performance metrics.\n\nThe paper also has a very extensive suite of experimental results, each interesting in its own right. I also particularly enjoy the allusion to traditional sparse coding.\n\n**Weaknesses**\n\nI think some of the strengths unfortunately are also weaknesses. Specifically I found some weaknesses around motivation, presentation and clarity, and related work.\n\n*Motivation*\n\nIn terms of motivation, I personally struggled to find a very convincing argument as to why this is important: obviously the graphs have clear trends in them and there is a story to be told, however it is not clear to me how these results are actionable and can lead to better models or better understanding/interpretability of LLMs. It is possible that such a strong motivation was there and maybe I missed it, which brings me to the second point.\n\n*Presentation*\n\nThere are a lot of results in the manuscript, which at times can feel disconnected. The reader has to invest a lot of energy in following all of the results and form a coherent story, and I think some restructuring and rewriting could probably help make the text more coherent. I think particularly problematic are sections 5.2 and onwards, where mentions are constantly made to “Introduced”, “Dense”, “LinearError”, etc, and it’s extremely difficult to keep track of all of these decompositions. A visual here would be very helpful, I believe, when a new decomposition is introduced, maybe also comparing it with the previous decomposition.\n\n*Related work*\n\nI was particularly excited to read about the adoption of the weak linear hypothesis, but then I found that there were nearly no references to traditional sparse coding, which has a very large and rich history of studying exactly the solutions of (6). Specifically, the model of (6), where a vector is decomposed in a sparse and dense component is very thoroughly studied in [1], including theory and applications to neural networks. Moreover, the original result of Section 5.1 is trivial in linear algebra/sparse coding but there are no references, and the “almost orthogonal” is needlessly vague. In sparse coding there are multiple ways to talk about this, from coherence, to the restricted isometry property, the nullspace property, and many more. I understand that most LLM papers these days do not engage with the sparse coding theory of SAEs, but without proper attribution we are doing a disservice to a longstanding and respected field, and at the same time run the risk of reinventing the wheel.\n\n[1] Tasissa, Theodosis, Tolooshams, Ba, “Discriminative reconstruction via simultaneous dense and sparse coding”, TMLR 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the “dark matter” of sparse autoencoders: the unexplained variance in the activations of SAEs when they are used to explain LLM layers. The authors perform an array of in-depth experiments showing many and interesting results, pertaining to the linear predictability of errors, nonlinear error analysis, insights about the scaling behavior, and postulating methods to reduce the SAEs error.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**\n\nOverall I think the paper is in a good state with virtually no typos (the only one I could find was in page 7, next to Table 1, where “noise” is typed as “nosie”) and an interesting message. In terms of requested changes I would ask:\n\n- Motivate (or emphasize) the bottom line: what are practical implications of the findings and how can they improve the models/understanding of other researchers?\n- Attempt to restructure/streamline/explain the experimental section. As mentioned above, some sections are extremely hard to follow and the experiments feel disconnected: having multiple results is okay, however currently they are fairly confusing.\n- Better address the related work, from the sparse coding perspective.\n\nI also had some further comments on the experiments:\n\n- In Figures 2, 3, and 4 the black dots are not explained. Are they the actual models and the rest are interpolations? If so, I believe their message is greatly hurt: the most interesting areas (where gradients are shown) are not evaluated and we don’t know if the trends are actually consistent and if the loss metrics are smoothly changing, which currently is a tenet of the work. Following up to this, why are areas with different point densities?\n- In most experiments, SAEs are applied to layer 20 of Gemma. Why is this particular layer chosen? Are the results consistent across other layers and models? These questions seem very important to be able to generalize the findings of the work.\n- As a slight follow-up to the above, part of Section 6 presents results on Llama: again, the choice of the architecture seems arbitrary and it’s not clear how one can compare results between different architectures. The impact of the experiment would be much stronger if it was on the same architecture.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "quenarTnFw", "reviewer": "Reviewer_8trg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper studies an important problem in mechanistic interpretability, namely the limitations of SAEs, and how to potentially overcome them.\n2. The paper has interesting empirical insights, including the effectiveness of using linear predictors of the activations. It also includes experimental analyses from different perspectives, such as linear vs. nonlinear error and scaling.  \n\nWeaknesses:\n1. Clarity: Sections 5 and 6 are difficult to understand. In particular, the definition of the error terms may contain typos, which make the results confusing:\n    - Eq. (8): are the two terms in the introduced error flipped? If I flip the two terms, I get that SaeError(x)  = x - Sae(x).\n    - What does the following sentence mean: “if this assumption is violated in practice, it will show up in our…”\n    - In the second paragraph of Section 5.2, which test is referred to in “if this test is accurate”? \n    - In the last paragraph of Section 5.2, which experiments show that Dense(x) and Introduced(x) are mostly constant?\n    - Section 6, if I use the definition of LinearError in the definition of NonlinearError, I get: NonlinearError = SaeError - LinearError = SaeError - (SaeError - b* x) = b* x. This looks linear, is there a typo here? \n    - Eq. (5), should there be norms surrounding SaeError_1(x) and SaeError_2(x), according to the exposition?\n    - Why would Dense(x) be part of the linearly explainable error?\n    - In Section 7.1, the authors state that having a better encoder doesn't necessarily reduce Introduced(x). Why would this happen empirically?\n\n2. Limited experimental setup: although there are many kinds of analyses, almost all experiments are performed on a particular layer of a single model. Why was the model chosen, and why layer 20? How would the results change with the position of the layer and the model architecture? The paper will be much stronger if it includes more context around these choices and more diverse experimental setups. At the very least the paper should include discussions around the limitations of the current experimental setup.\n\n3. Empirical verification of assumptions: In Section 5.1, the explanation builds on the intuition that the y_i’s are almost orthogonal. This seems to be a property that can be verified whether it holds in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the “dark matter” of Sparse Autoencoders (SAEs)  – the neural network activations that remain unexplained by the SAE decomposition. The authors begin by examining the SAE error using linear functions of the activations, and show that a significant portion of the error can be linearly predicted from the activations. To investigate the sources of the error, they decompose it into linear nonlinear components, and show that the nonlinear component is fundamentally different from the linear component. They further study what might constitute the linear and nonlinear components of the error using a model based on the weak linear representation hypothesis, and introduce a new term in the model corresponding to the error from learning the sparse features. Finally, the authors propose several strategies to reduce the nonlinear error and results suggesting that circuit analysis is likely relevant for studying SAE dark matter.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical for acceptance:\n1. Improve the clarity of the paper by addressing questions in the section above.\n2. Include a wider range of layers and models in the experiments. \n\nWill strengthen the paper:\n1. Verify the almost-orthogonal assumption of the SAE features.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nIWAv3UZUe", "reviewer": "Reviewer_4PVn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n\n  1. The paper is novel to analyze the SAE error and shows the interesting observation that SAE error vectors can be explained with a linear transformation.\n  2. It provides a comprehensive validation of its hypotheses and conclusions on a large-scale language model, Gemma-2 9B.\n  3. It proposes inference time gradient pursuit and linear transformations from earlier layer SAE outputs, which is useful and insightful to reduce nonlinear SAE error.\n\n- Weaknesses\n\n  1. This paper is difficult to understand. A major drawback of the paper is its complexity and lack of clarity in presentation. Some of the descriptions and technical details are ambiguous.\n  2. The connection between explanations of SAE and interpretability of language models. The goal of SAE is to extract interpretable features or circuits from a language model, I do not clearly understand how the explanations in this paper can help to approach this goal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the sparse autoencoder (SAE) error of Gemma and shows that most of the error can be linearly predicted. Moreover, it empirically analyzes nonlinear SAE error and proposes two methods to reduce them.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Preliminaries about sparse SAE would help readers better understand the research focus, particularly the SAE error vector.\n2. This paper uses only Gemma Scope 9B model. However, it claims that \"we are agnostic to the architecture or training procedure of the sparse autoencoder.\" To support this claim, additional experiments with other models should be included.\n3. In Section 4, please explain why the 20th layer is chosen for experiments and provide further details on the specific activation locations used in these experiments.\n4. The definition of $L0$ is not provided, yet it appears in several figures throughout the paper. A clear definition should be included.\n5. Please clarify what the different lines in Figure 13 represent. Previous work (Scaling and Evaluating Sparse Autoencoders) suggests to use layers towards the end of the network. Does Figure 13 imply that the SAE error of these layers cannot be predicted linearly?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the paper is well structured, studies an interesting and important problem, and presents a compelling message. 

However, as the reviewers have pointed out, the evidence is a bit fuzzy. The experiments about predicting the linear term in the error seem to indicate a stable finding; however, the prediction of the error norms seem to hold only for the middle layers, and there seem to be no special insights about the nonlinear terms. Finally, it is unclear (at least at this point) how exactly the proposed decomposition, of SAE dark matter into finer-grained error terms, contributes to *actionable* insights towards improving SAEs or other circuit interpretation methods. 

Overall, I recommend an accept since but encourage the authors to consider these points (and the reviewers' feedback) while pursuing this line of work further.
Decision: Accept (poster)