Decomposing The Dark Matter of Sparse Autoencoders

## Abstract

Abstract Sparse autoencoders (SAEs) are a promising technique for decomposing language model activations into interpretable linear features. However, current SAEs fall short of completely explaining model performance, resulting in “dark matter”: unexplained variance in activations. This work investigates dark matter as an object of study in its own right. Surprisingly, we find that much of SAE dark matter—about half of the error vector itself and > 90 % absent percent 90 >90\% > 90 % of its norm—can be linearly predicted from the initial activation vector. Additionally, we find that the scaling behavior of SAE error norms at a per token level is remarkably predictable: larger SAEs mostly struggle to reconstruct the same contexts as smaller SAEs. We build on the linear representation hypothesis to propose models of activations that might lead to these observations, including postulating a new type of “introduced error”; these insights imply that the part of the SAE error vector that cannot be linearly predicted (“nonlinear” error) might be fundamentally different from the linearly predictable component. To validate this hypothesis, we empirically analyze nonlinear SAE error and show that 1) it contains fewer not yet learned features, 2) SAEs trained on it are quantitatively worse, 3) it helps predict SAE per-token scaling behavior, and 4) it is responsible for a proportional amount of the downstream increase in cross entropy loss when SAE activations are inserted into the model. Finally, we examine two methods to reduce nonlinear SAE error at a fixed sparsity: inference time gradient pursuit, which leads to a very slight decrease in nonlinear error, and linear transformations from earlier layer SAE outputs, which leads to a larger reduction.

## 1 Introduction

The ultimate goal for ambitious mechanistic interpretability is to understand neural networks from the bottom up by breaking them down into programs (“circuits”) and the variables (“features”) that those programs operate on (Olah, ). One recent successful technique for finding features in language models has been sparse autoencoders (SAEs), which learn a dictionary of one-dimensional representations that can be sparsely combined to reconstruct model hidden activations (Cunningham et al., ; Bricken et al., ). However, as observed by Gao et al. (), the scaling behavior of SAE width (number of latents) vs. reconstruction mean squared error (MSE) is best fit by a power law with a constant error term. Gao et al. () speculate that this component of SAE error below the asymptote might best be explained by model activations having components with denser structure than simple SAE features (e.g. Gaussian noise). This is a concern for the ambitious agenda because it implies that there are components of model hidden states that are harder for SAEs to learn and which might not be eliminated by simple scaling of SAEs.

Motivated by this discovery, in this work our goal is to specifically study the SAE error vector itself, and in doing so gain insight into the failures of current SAEs, the dynamics of SAE scaling, and possible distributions of model activations. Thus, our direction differs from the bulk of prior work that seeks to quantify SAE failures, as these mostly focus on downstream benchmarks or simple cross entropy loss (see e.g. Gao et al. (); Templeton et al. (); Anders & Bloom ()). The structure of this paper is as follows:

- 1.
In Section 4, we introduce the fundamental mystery that we will explore throughout the rest of the paper: SAE errors are shockingly predictable. To the best of our knowledge, we are the first to show that a large fraction of SAE error vectors can be explained with a linear transformation of the input activation, that the norm of SAE error vectors can be accurately predicted by a linear projection of the input activation, and that on a per-token level, error norms of large SAEs are linearly predictable from small SAEs.
- 2.
In Section 5, we propose models of SAE error to explain these observations, including postulating a new type of “introduced error” that is caused by the SAE architecture and sparsity constraint.
- 3.
In Section 6, we investigate the linearly predictable and nonlinearly predictable components of SAE error. We find that although the nonlinear component affects downstream cross entropy loss in proportion to its norm, it is qualitatively different from linear error: as compared to linear error, nonlinear error is harder to learn SAEs for, consists of a smaller proportion of absent linear features, and helps with predicting the norm of larger SAE errors from smaller SAEs.
- 4.
In Section 7, we show that inference time optimization increases the fraction of variance explained by SAEs, but only slightly decreases nonlinear error. Additionally, we show that we can use SAEs trained on previous components to decrease nonlinear error and total SAE error.

Figure: Figure 1: A breakdown of SAE dark matter. See Section 4 for how we break down the overall fraction of unexplained variance into absent features, linear error, and nonlinear error. See Section 7.1 for further separating encoder error from nonlinear error.
Refer to caption: x1.png

## 2 Related Work

Language Model Representation Structure:
The linear representation hypothesis (LRH) (Park et al., ; Elhage et al., ) claims that language model hidden states can be decomposed into a sparse sum of linear feature directions. The LRH has seen recent empirical support with sparse autoencoders, which have succeeded in decomposing much of the variance of language model hidden states into such a sparse sum, as well as a long line of work that has used probing and dimensionality reduction to find causal linear representations for specific concepts (Alain, ; Nanda et al., ; Marks et al., ; Gurnee, ). On the other hand, some recent work has questioned whether the linear representation hypothesis is true: Engels et al. () find multidimensional circular representations in Mistral (Jiang et al., ) and Llama (AI@Meta, ), and Csordás et al. () examine synthetic recurrent neural networks and find “onion-like” non-linear features not contained in a linear subspace. This has inspired recent discussion about what a true model of activation space might be: Mendel () argues that the linear representation hypothesis ignores the growing body of results showing the multi-dimensional structure of SAE latents, and Smith () argues that we only have evidence for a “weak” form of the superposition hypothesis holding that only some features are linearly represented.

SAE Errors and Benchmarking: Multiple works have introduced techniques to benchmark SAEs and characterize their error: Bricken et al. (), Gao et al. (), and Templeton et al. () use manual human analysis of features, automated interpretability, downstream cross entropy loss when SAE reconstructions are inserted back into the model, and feature geometry visualizations; Karvonen et al. () use the setting of board games, where the ground truth features are known, to determine what proportion of the true features SAEs learn; and Anders & Bloom () use the performance of the model on NLP benchmarks when the SAE reconstruction is inserted back into the model. More specifically relevant to our main direction in this paper, Gurnee () finds that SAE reconstruction errors are pathological, that is, when SAE reconstructions are inserted into the model, they have a larger effect on cross entropy loss than random perturbations with the same error norm. Follow up work by Heimersheim & Mendel () and Lee & Heimersheim () find that this effect disappears when the random baseline is replaced by a perturbation in the direction of the difference between two random activations.

SAE Scaling Laws: Anthropic (), Templeton et al. (), and Gao et al. () study how SAE MSE scales with respect to FLOPS, sparsity, and SAE width, and define scaling laws with respect to these quantities. Templeton et al. () also study how specific groups of language features like chemical elements, cities, animals, and foods are learned by SAEs, and show that SAEs predictably learn these features in terms of their occurrence. Finally, Bussmann et al. () find that larger SAEs learn two new types of dictionary vectors as comparsed to smaller SAEs: features not present at all in smaller SAEs, and more fine-grained “feature split” versions of features in smaller SAEs.

## 3 Notation

In this paper, we consider neural network activations ${\mathbf{x}}\in\mathbb{R}^{d}$ and sparse autoencoders $\texttt{Sae}\in\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ which seek to minimize $\left\lVert{\mathbf{x}}-\texttt{Sae}({\mathbf{x}})\right\rVert_{2}$ while using a small number of active latents. We are agnostic to the architecture or training procedure of the sparse autoencoder; see (Bricken et al., ; Cunningham et al., ; Gao et al., ; Templeton et al., ) for such details. We also define $\texttt{SaeError}({\mathbf{x}})$ such that ${\mathbf{x}}=\texttt{Sae}({\mathbf{x}})+\texttt{SaeError}({\mathbf{x}})$, or rearranged

$$ $\displaystyle\texttt{SaeError}({\mathbf{x}})\coloneq{\mathbf{x}}-\texttt{Sae}( {\mathbf{x}}).$ (1) $$

## 4 Predicting SAE Error

Experiment Details:(^1^11Code at [https://github.com/JoshEngels/SAE-Dark-Matter](https://github.com/JoshEngels/SAE-Dark-Matter)) Unless noted otherwise, we set ${\mathbf{x}}$ equal to layer $20$ Gemma 2 9B (Team et al., ) activations. We use $300$ contexts of $1024$ tokens from the uncopywrited subset of the Pile (Gao et al., ) and then filter to only activations of tokens after position $200$ in each context, as Lieberum et al. () find that earlier tokens are easier for sparse autoencoders to reconstruct, and we wish to ignore the effect of token position on our results. This results in a dataset of about 247k activations. We use the suite of Gemma Scope (Lieberum et al., ) sparse autoencoders. For linear regressions, we use a random subset of size 150k as training examples (since Gemma 2 9B has a hidden dimension of $3584$, this prevents overfitting) and report the $R^{2}$ on the other 97k activations. For transforms to a multi-dimensional output, we report the average $R^{2}$ across dimensions. We include bias terms in our linear regressions but omit them from equations for simplicity.

Figure: Figure 2: $R^{2}$ of linear regressions from activations to SAE error norms. Errors norms are linearly predictable with high accuracy at all widths and $L_{0}$.
Refer to caption: x2.png

Predicting SAE Error Norm: For our first set of experiments, we find the optimal linear probe ${\bm{a}}^{*}$ from ${\mathbf{x}}$ to $\lVert\texttt{SaeError}({\mathbf{x}})\rVert^{2}_{2}$. Formally (with a slight abuse of notation, since ${\mathbf{x}}$ is a random variable and not a dataset), we solve for

$$ $\displaystyle{\bm{a}}^{*}=\operatorname*{arg\,min}_{{\bm{a}}\in\mathbb{R^{d}}} \left\rVert{\bm{a}}^{T}\cdot{\mathbf{x}}-\lVert\texttt{SaeError}({\mathbf{x}}) \rVert^{2}_{2}\right\lVert_{2}$ (2) $$

The $R^{2}$ of these probes are all extremely high: on layer $20$, for all Gemma Scope combinations of SAE width and $L_{0}$, between $86\%$ and $95\%$ of the variance in SAE error norm is explained by the optimal linear probe. We plot these results as a contour plot in Fig. 3. Overall, sparser and wider SAEs have less predictable error norms. In Appendix A, we show a plot of the $R^{2}$ of the prediction across layers, and find that except for the first few layers, activation probes have a much higher $R^{2}$ than probes using tokens, SAE $L_{0}$, model loss, or activation norm.

Predicting SAE Error Vectors: We next examine the $R^{2}$ of the optimal linear transform ${\bm{b}}^{*}$ from ${\mathbf{x}}$ to $\texttt{SaeError}({\mathbf{x}})$:

$$ $\displaystyle{\bm{b}}^{*}\coloneq\operatorname*{arg\,min}_{{\bm{b}}\in\mathbb{ R}^{d\times d}}\left\rVert{\bm{b}}\cdot{\mathbf{x}}-\texttt{SaeError}({\mathbf {x}})\right\rVert_{2}$ (3) $$

As we show in Fig. 3, the $R^{2}$ of these transforms for layer $20$ range between $30\%$ and $72\%$; this is less than the $R^{2}$ for our norm prediction experiments, but still much higher than we might expect. Intuitively, this result implies that there must be large linear subspaces that the SAE is mostly failing to learn. Like SAE error norm predictions, there is a clear pattern across SAE $L_{0}$ and width: $R^{2}$ decreases with increasing SAE width and $L_{0}$. Interestingly, this pattern is not the same as it was above for SAE error norm: the $R^{2}$ of error norm predictions increases with SAE $L_{0}$, while it decreases for error vector predictions. One concern might be that ${\bm{b}}^{*}$ is mostly reversing feature shrinkage, in Section A.1, we show that this is not the case.

Nonlinear FVU: Another related metric we are interested in is the total amount of the original activation ${\mathbf{x}}$ we fail to “explain” using both the SAE reconstruction $\texttt{Sae}({\mathbf{x}})$ and a linear projection of ${\mathbf{x}}$. That is, assuming we have found ${\bm{b}}^{*}$ as in Eq. 3, we are interested in the fraction of variance unexplained (FVU) by the sum of $\texttt{Sae}({\mathbf{x}})$ and ${\bm{b}}^{*}\cdot{\mathbf{x}}$:

$$ $\displaystyle\texttt{FVU}_{\texttt{nonlinear}}\coloneq 1-R^{2}({\mathbf{x}}, \texttt{Sae}({\mathbf{x}})+{\bm{b}}^{*}\cdot{\mathbf{x}})$ (4) $$

We label this quantity $\texttt{FVU}_{\texttt{nonlinear}}$ because it is intuitively the amount of the SAE’s unexplained variance that is not a linear projection of the input. Interestingly, we find that at a fixed $L_{0}$, $\texttt{FVU}_{\texttt{nonlinear}}$ is approximately constant (see Fig. 5). That is, even though we can linearly predict a smaller portion of the error vector in larger SAEs, this effect is counteracted almost exactly by the fact that the SAE error vector itself is getting smaller. In contrast, $\texttt{FVU}_{\texttt{nonlinear}}$ decreases as SAE $L_{0}$ increases.

We can use the hypothesis that $\texttt{FVU}_{\texttt{nonlinear}}$ is a constant at a fixed sparsity to plot the breakdown of FVU for varying SAE width and $L_{0}\approx 60$ in Fig. 1. We plot a horizontal fit for $\texttt{FVU}_{\texttt{nonlinear}}$ and a power law fit with a constant for Gemma SAE reconstructions; the power law fit asymptotes above the horizontal fit, which implies the presence of linear error even at very large SAE width. Note that we assume absent features (those which are not yet learned) are a component of the linear error; we provide justifications for this in Section 5.2.

Figure: Figure 4: $\texttt{FVU}_{\texttt{nonlinear}}$ versus SAE width and $L_{0}$. Larger SAE $L_{0}$s have a smaller $\texttt{FVU}_{\texttt{nonlinear}}$, but $\texttt{FVU}_{\texttt{nonlinear}}$ stays mostly constant with increasing SAE width.
Refer to caption: x4.png

Predicting SAE Per-Token Error Norms: We now examine per-token SAE scaling behavior. Given two SAEs, $\texttt{SAE}_{1}$ and $\texttt{SAE}_{2}$, we are interested in how much of the variance in error norms in $\texttt{SAE}_{2}$ is predictable from error norms in $\texttt{SAE}_{1}$. That is, we want to find ${\bm{c}}^{*}$ such that

$$ $\displaystyle{\bm{c}}^{*}\coloneq\operatorname*{arg\,min}_{{\bm{c}}\in\mathbb{ R}}\left\lVert{\bm{c}}\cdot\texttt{SaeError}_{1}({\mathbf{x}})-\texttt{ SaeError}_{2}({\mathbf{x}})\right\rVert_{2}$ (5) $$

Note that in practice, although ${\bm{a}}^{*}$ also can predict the norm of SAE error, it requires training the target SAE to learn a probe. Here, on the other hand, although we formulate finding ${\bm{c}}^{*}$ as an optimization problem that requires a larger SAE, in practice we do not need to actually train the larger SAE to get interesting insights: since ${\bm{c}}^{*}$ has just one component, it simply measures how well small SAE error can be multiplied by a scalar to predict large SAE error. If the $R^{2}$ is high, we know that on tokens that small SAEs perform poorly on, larger SAEs will as well. In Fig. 5, we plot the $R^{2}$ of ${\bm{c}}^{*}$ probes on all pairs of layer $20$ SAEs with $L_{0}\approx 60$ (restricting to pairs where $\texttt{SAE}_{2}$ is larger than $\texttt{SAE}_{2}$), and find that indeed, per token SAE errors are highly predictable. Additionally, we show concretely what these correlated SAE error norms looks like on a set of $100$ tokens from the Pile in Fig. 6.

Figure: Figure 6: Per token scaling with average nonlinear error, layer $20$ Gemma 9B SAEs from Gemma Scope closest to $L_{0}$ = 60.
Refer to caption: x6.png

## 5 Modeling Activations

We will adopt the weak linear hypothesis (Smith, ), a generalization of the linear representation hypothesis which holds only that some features in language models are represented linearly. Thus we have

$$ $\displaystyle{\mathbf{x}}=\sum_{i=0}^{n}{\textnormal{w}}_{i}{\bm{y}}_{i}+ \texttt{Dense}({\mathbf{x}})$ (6) $$

for linear features $\{{\bm{y}}_{1},\ldots,{\bm{y}}_{n}\}$ and random vector ${\mathbf{w}}\in\mathbb{R}^{n}$, where ${\mathbf{w}}$ is sparse ($\left\lVert{\mathbf{w}}\right\rVert_{1}\ll d$) and $\texttt{Dense}({\mathbf{x}})$ is a random vector representing the dense component of ${\mathbf{x}}$. $\texttt{Dense}({\mathbf{x}})$ might be Gaussian noise, nonlinear features as described by Csordás et al. (), or anything else not represented in a low-dimensional linear subspace.

Say our SAE has $m$ latents. Since by assumption $\texttt{Dense}({\mathbf{x}})$ cannot be represented in a low-dimensional linear subspace, the sparsity limited SAE will not be able to learn it. Thus, we will assume that the SAE learns only the $m$ most common features ${\bm{y}}_{0},\ldots,{\bm{y}}_{m-1}$. We will also assume that the SAE introduces some error when making this approximation, and instead learns $\hat{{\bm{y}}_{i}}$ and $\hat{{\textnormal{w}}_{i}}$. Thus we have

$$ $\displaystyle\texttt{Sae}({\mathbf{x}})$ $\displaystyle=\sum_{i=0}^{m}\hat{{\textnormal{w}}_{i}}\hat{{\bm{y}}_{i}}$ (7) $\displaystyle\texttt{SaeError}({\mathbf{x}})$ $\displaystyle=\texttt{Dense}({\mathbf{x}})+\left(\sum_{i=0}^{m}\hat{{ \textnormal{w}}_{i}}\hat{{\bm{y}}_{i}}-\sum_{i=0}^{m}{\textnormal{w}}_{i}{\bm{ y}}_{i}\right)+\sum_{i=m}^{n}{\textnormal{w}}_{i}{\bm{y}}_{i}$ (8) $$

We finally define $\texttt{Introduced}({\mathbf{x}})\coloneq\sum_{i=0}^{m}\hat{{\textnormal{w}}_{
i}}\hat{{\bm{y}}_{i}}-\sum_{i=0}^{m}{\textnormal{w}}_{i}{\bm{y}}_{i}$, so we have

$$ $\displaystyle\texttt{SaeError}({\mathbf{x}})$ $\displaystyle=\texttt{Dense}({\mathbf{x}})+\texttt{Introduced}({\mathbf{x}})+ \sum_{i=m}^{n}{\textnormal{w}}_{i}{\bm{y}}_{i}$ (9) $$

### 5.1 Analyzing Error Norm Prediction

We will first analyze Eq. 2, the learned probe from ${\mathbf{x}}$ to $\lVert\texttt{SaeError}({\mathbf{x}})\rVert_{2}^{2}$. First, we claim that given a vector ${\mathbf{x}}$, if ${\mathbf{x}}$ is a sparse sum of orthogonal vectors, then there exists a perfect prediction vector ${\bm{a}}$ such that
${\bm{a}}^{T}{\mathbf{x}}\approx\left\lvert{\mathbf{x}}\right\rvert_{2}^{2}$ (in other words, the norm squared of ${\mathbf{x}}$ can be linearly predicted from ${\mathbf{x}}$). The proof of this claim is in Section B.1; the intuition is that we can set the probe vector ${\bm{a}}^{*}$ to the sum of the vectors ${\bm{y}}_{i}$ weighted by their average weight $E({\textnormal{w}}_{i})$.

When ${\mathbf{x}}$ is instead a sparse sum of non-orthogonal vectors, as it partly is in Eq. 6 amd Eq. 9, this proof is no longer true, but we now argue that a similar intuition holds. If the ${\bm{y}}_{i}$ are almost orthogonal and do not activate much at the same time, then a probe vector again equal to the sum of vectors ${\bm{y}}_{i}$ weighted by their average value $E({\textnormal{w}}_{i})$ will be a good approximate prediction. Indeed, when we try predicting $\lVert\texttt{Sae}({\mathbf{x}})\rVert_{2}^{2}$ from $\texttt{Sae}({\mathbf{x}})$ (which is a sparse sum of known almost orthogonal vectors of a similar distribution to the true SAE vectors), we find that indeed the linear probe that is learned is approximately equal to this sum (see Section B.2).

Thus, we can now neatly explain why we can predict the norms of SAE errors: they mostly consist of almost orthogonal sparsely occuring not yet learned SAE features! We further can explain why larger SAEs have less predictable error norms: since $m$ is larger, there is a larger component in the error of not-as-linearly-predictable $\texttt{Dense}({\mathbf{x}})$ and $\texttt{Introduced}({\mathbf{x}})$.

### 5.2 Analyzing Error Vector Prediction

We will now analyze Eq. 3, the learned transformation from ${\mathbf{x}}$ to $\texttt{SaeError}({\mathbf{x}})$, with our model of SAE error from Eq. 9. We assume that $\texttt{Introduced}({\mathbf{x}})$ cannot be approximated at all as a linear function of ${\mathbf{x}}$; if this assumption is violated in practice, it will show up in our estimations as an increased amount of $\texttt{Dense}({\mathbf{x}})$.

If $\texttt{Dense}({\mathbf{x}})+\sum_{i=m}^{n}{\textnormal{w}}_{i}{\bm{y}}_{i}$ is contained in a linear subspace of ${\mathbf{x}}$ orthogonal to $\sum_{i=0}^{m}{\textnormal{w}}_{i}{\bm{y}}_{i}$, then the error of the transformation ${\bm{b}}^{*}$ exactly equals $\texttt{Introduced}({\mathbf{x}})$ (since the transformation is just exactly this orthogonal linear subspace). However, if such a linear transform does not exist, the percent of variance left unexplained by the regression will be an upper bound on the true variance explained by $\texttt{Introduced}({\mathbf{x}})$. We also note that if this test is accurate, we can use it to estimate $\texttt{Dense}({\mathbf{x}})$: the difference between the variance explained by $\texttt{Sae}({\mathbf{x}})$ and the variance explained by ${\mathbf{x}}-(\texttt{Sae}({\mathbf{x}})+{\mathbf{b}}^{*}\cdot{\mathbf{x}})$ will approach $\texttt{Dense}({\mathbf{x}})$ as $m\rightarrow\infty$.

Thus, our ability to estimate $\texttt{Introduced}({\mathbf{x}})$ and $\texttt{Dense}({\mathbf{x}})$ using ${\bm{b}}^{*}$ depends on how well a linear transform works to predict $\texttt{Dense}({\mathbf{x}})$ and $\sum_{i=m}^{n}{\textnormal{w}}_{i}{\mathbf{y}}_{i}$. Although we do not have access to the ground truth vectors ${\bm{y}}_{i}$, we can replace ${\mathbf{x}}^{\prime}$ with a similar distribution of vectors that we do have access to, using the same trick as above. Given an SAE, we replace ${\mathbf{x}}$ with ${\mathbf{x}}^{\prime}=\texttt{Sae}({\mathbf{x}})$. ${\mathbf{x}}^{\prime}$ has the useful property that it is a sparse linear sum of vectors (the ones that the SAE learned), and the distribution of these vectors and their weights are similar to that of the true features ${\bm{y}}_{i}$. We now pass ${\mathbf{x}}^{\prime}$ back through the SAE and can control all of the quantities we are interested in: we can vary $m$ by masking SAE dictionary elements, simulate Dense(${\mathbf{x}}^{\prime}$) by adding Gaussian noise to ${\mathbf{x}}^{\prime}$, and simulate $\texttt{Introduced}({\mathbf{x}})$ by adding Gaussian noise to $\texttt{Sae}({\mathbf{x}}^{\prime})$.

Figure: Table 1: Correlation matrix between synthetic noise and estimated errors.

We run this synthetic setup with a Gemma Scope layer $20$ SAE (width $16k$, $L_{0}\approx 68$) in Section B.3, and find that indeed, estimated Dense(${\mathbf{x}}^{\prime}$) is highly correlated with the amount of Gaussian nosie added to ${\mathbf{x}}^{\prime}$ and Introduced(${\mathbf{x}}^{\prime}$) is highly correlated with the amount of Gaussian noise added to $\texttt{Sae}({\mathbf{x}}^{\prime})$ (see Section 5.2). However, note that because ${\mathbf{x}}^{\prime}$ noise is also slightly correlated with estimated Introduced(${\mathbf{x}}^{\prime}$), it is possible that some of the contribution to the estimated nonlinear error is from Dense(${\mathbf{x}}^{\prime}$).

Thus, we again now have a potential explanation for our initial results: we can predict error vectors because they consist in large part of not yet learned linear features in an almost orthogonal subspace of ${\mathbf{x}}$, we can predict a smaller portion of larger SAE errors because the number of these linear features go down with SAE width, and the horizontal line in Fig. 1 is because $\texttt{Dense}({\mathbf{x}})$ and $\texttt{Introduced}({\mathbf{x}})$ are mostly constant. Furthermore, we can hypothesize from the correlations om Section 5.2 that the linearly predictable component of SAE error consists mostly of not yet learned features and $\texttt{Dense}({\mathbf{x}})$, while the component that is not linearly predictable consists mostly of $\texttt{Introduced}({\mathbf{x}})$. We will explore this hypothesis in Section 6.

### 5.3 Analyzing Per-Token Scaling Predictions

Finally, we provide a simple explanation for why per-token SAE errors are highly predictable between SAEs of different sizes. For this, we only need Eq. 9. Since $\texttt{Dense}({\mathbf{x}})$ and $\texttt{Introduced}({\mathbf{x}})$ stay mostly constant as $m$ increases, for large $m$ the SAE error stays mostly constant because it is primarily determined by these components. Thus, since $m=16k$ is already large, a linear prediction that is just a slightly smaller version of the current error performs well. Additionally, this reasoning suggests a natural experiment: if we can predict $\texttt{Introduced}({\mathbf{x}})$ on a per-token level (which we hypothesize we can do with the non-linearly predictable component of SAE error), we may be able to better predict the floor of SAE scaling and therefore better predict larger SAE errors; we run this experiment in Fig. 8, where we find an affirmative answer.

## 6 Analyzing Components of SAE Error

In this section, we run experiments that seek to prove our hypothesis from Section 5.2: that the split of $\texttt{SaeError}({\mathbf{x}})$ into a linearly predictable component and the non-linearly predictable component is meaningful. For convenience, given a probe ${\bm{b}}^{*}$ from Eq. 3, we write

$$ $\displaystyle\texttt{LinearError}({\mathbf{x}})$ $\displaystyle\coloneq\texttt{SaeError}({\mathbf{x}})-{\bm{b}}^{*}\cdot{\mathbf {x}}$ $\displaystyle\texttt{NonlinearError}({\mathbf{x}})$ $\displaystyle\coloneq\texttt{SaeError}({\mathbf{x}})-\texttt{LinearError}({ \mathbf{x}})$ $$

Applying the Norm Prediction Test:
For our first experiment, we run the norm prediction test from Eq. 2 on six different random vectors: ${\mathbf{x}}$, $\texttt{Sae}({\mathbf{x}})$, $\texttt{Sae}({\mathbf{x}})-\texttt{NonlinearError}({\mathbf{x}})$ (just the linearly predictable part of the SAE reconstruction), $\texttt{SaeError}({\mathbf{x}})$, $\texttt{LinearError}({\mathbf{x}})$, and $\texttt{NonlinearError}({\mathbf{x}})$. The results are shown as a violin plot for each component across all layer $20$ Gemma Scope SAEs in Fig. 8 (the $\texttt{Sae}({\mathbf{x}})$ bar is just a summary of Fig. 3).

Firstly, we note that $\left\lVert{\mathbf{x}}\right\rVert_{2}^{2}$ can almost be perfectly predicted from ${\mathbf{x}}$. This is reassuring news for the linear representation hypothesis, as it implies that ${\mathbf{x}}$ can indeed be well modeled as the sum of many one-dimensional features, at least from the perspective of this test.

We also find that $\texttt{NonlinearError}({\mathbf{x}})$ has a notably lower score on this test. This supports our hypothesis that unlike ${\mathbf{x}}$, this component does not consist mostly of a sparse sum of linear features from ${\mathbf{x}}$, and may be partly composed of $\texttt{Introduced}({\mathbf{x}})$.

Figure: Figure 7: Violin plot of norm prediction tests for all layer $20$ Gemma Scope SAEs. We plot the $R^{2}$ of a linear regression from ${\mathbf{x}}$ to to each random vector’s norm squared.
Refer to caption: x7.png

Training SAEs on $\texttt{SaeError}({\mathbf{x}})$ Components: Another empirical test we run is training an SAE on $\texttt{NonlinearError}({\mathbf{x}})$ and $\texttt{LinearError}({\mathbf{x}})$. Our hypothesis is that it will be harder to learn an SAE for $\texttt{NonlinearError}({\mathbf{x}})$ than $\texttt{LinearError}({\mathbf{x}})$, since $\texttt{LinearError}({\mathbf{x}})$ should have a higher proportion of true linear features. We choose a fixed Gemma Scope layer $20$ SAE with $16k$ latents and $L_{0}\approx 60$ to generate $\texttt{SaeError}({\mathbf{x}})$ from. This SAE has nonlinear and linear components of the error approximately equal in norm and $R^{2}$ of the total $\texttt{SaeError}({\mathbf{x}})$ they explain, so it presents a fair comparison. We train SAEs to convergence (about 100M tokens) on each of these components of error and find that indeed, the SAE trained on $\texttt{NonlinearError}({\mathbf{x}})$ converges to a fraction of variance unexplained an absolute $5$ percent higher than the SAE trained on the linear component of SAE error ($\approx 0.59$ and $\approx 0.54$ respectively).

One confounding factor is that the linear component of SAE error additionally contains $\texttt{Dense}({\mathbf{x}})$, which may also be harder for the SAE to learn. Thus, we additionally examine the interpretability of the learned SAE latents using automated interpretability (this technique was first proposed by Bills et al. () for interpreting neurons, and first applied to SAEs by Cunningham et al. ()). Specifically, we use the implementation introduced by Juang et al. (), where a language model (we use Llama 3.1 70b (AI@Meta, )) is given top activating examples to generate an explanation, and then must use only that explanation to predict if the feature fires on a test context. Our results in Fig. 8 show that indeed, the SAE trained on linear error produces latents that are about an absolute $5\%$ more interpretable across all activation firing deciles (we average results across $1000$ random features for both SAEs, where for each feature use $7$ examples in each of the $10$ feature activation deciles as well as $50$ negative examples, and show $95\%$ confidence intervals).

Downstream Cross Entropy Loss of $\texttt{SaeError}({\mathbf{x}})$ Components:
A common metric used to test SAEs is the percent of cross entropy loss recovered when the SAE reconstruction is inserted into the model in place of the original activation versus an ablation baseline (see e.g. Bloom ()). We modify this test to specifically examine the different components of $\texttt{SaeError}({\mathbf{x}})$: we compare the percent of the cross entropy loss recovered when replacing ${\mathbf{x}}$ with $\texttt{Sae}({\mathbf{x}})$ plus either $\texttt{LinearError}({\mathbf{x}})$ or $\texttt{NonlinearError}({\mathbf{x}})$ to the baseline of inserting just $\texttt{Sae}({\mathbf{x}})$ in place of ${\mathbf{x}}$. To estimate how much each component “should” recover, we use two metrics: the average norm of the component relative to the total norm of $\texttt{Sae}({\mathbf{x}})$ and the percent of the variance that the component recovers between $\texttt{Sae}({\mathbf{x}})$ and ${\mathbf{x}}$. The results, shown in Fig. 9, show that for the most part these metrics are reasonable predictions for both types of error. That is, both $\texttt{NonlinearError}({\mathbf{x}})$ and $\texttt{LinearError}({\mathbf{x}})$ proportionally contribute to the SAE’s increase in downstream cross entropy loss, with possibly a slightly higher contribution than expected for $\texttt{LinearError}({\mathbf{x}})$.

Using $\left\lVert\texttt{NonlinearError}({\mathbf{x}})\right\rVert$ to Predict Scaling:
Following up on our discussion in Section 5.3, we are interested in whether $\texttt{NonlinearError}({\mathbf{x}})$ can help with predicting SAE per-token error norm scaling, as this might suggest that it contains a larger component of $\texttt{Introduced}({\mathbf{x}})$ than $\texttt{LinearError}({\mathbf{x}})$. Formally, we solve for

$$ $\displaystyle{\bm{d}}^{*}\coloneq\operatorname*{arg\,min}_{{\bm{d}}\in\mathbb{ R^{2}}}\left\lVert{\bm{d}}^{T}\cdot[\texttt{SaeError}_{1}({\mathbf{x}}), \texttt{NonlinearError}_{1}({\mathbf{x}})]-\texttt{SAE}_{2}({\mathbf{x}}) \right\rVert_{2}$ (10) $$

To evaluate the improvement of ${\bm{d}}^{*}$ relative to ${\bm{c}}^{*}$ from Eq. 5, we report the percent decrease in FVU; see Fig. 11. We find that using the norm of $\texttt{NonlinearError}({\mathbf{x}})$ provides a small but noticeable bump in the ability to predict larger SAE errors of up to a $5\%$ decrease in FVU, validating this hypothesis.

Figure: Figure 9: Results of intervening in the forward pass and replacing ${\mathbf{x}}$ with $\texttt{Sae}({\mathbf{x}})+\texttt{NonlinearError}({\mathbf{x}})$ and $\texttt{Sae}({\mathbf{x}})+\texttt{LinearError}({\mathbf{x}})$ during the forward pass. Reported in percent of cross entropy loss recovered with respect to the difference between the same intervention with $\texttt{Sae}({\mathbf{x}})$ and with the normal model forward pass.
Refer to caption: x9.png

## 7 Reducing NonlinearError ⁢ ( 𝐱 ) NonlinearError 𝐱 \texttt{NonlinearError}({\mathbf{x}}) NonlinearError ( bold_x )

If indeed $\texttt{NonlinearError}({\mathbf{x}})$ is partly made up of $\texttt{Introduced}({\mathbf{x}})$, we may be able to reduce it by improving SAEs. Thus, in this section, we investigate to what extent simple techniques can reduce $\texttt{NonlinearError}({\mathbf{x}})$.

### 7.1 Using a More Powerful Encoder

Figure: (a) SAE error breakdown vs. SAE width for inference time optimized reconstructions from Gemma Scope $L_{0}\approx 60$ dictionaries.
Refer to caption: x10.png

Our first approach for reducing nonlinear error is to try improving the encoder. We use a recent approach suggested by Smith (): applying a greedy inference time optimization algorithm called gradient pursuit to a frozen learned SAE decoder matrix. We implement the algorithm exactly as described by Smith () and run it on all layer $20$ Gemma Scope $9b$ SAEs closest to $L_{0}\approx 60$. For each example ${\mathbf{x}}$ with reconstruction $\texttt{Sae}({\mathbf{x}})$, we use the gradient pursuit implementation with an $L_{0}$ exactly equal to the $L_{0}$ of ${\mathbf{x}}$ in the original $\texttt{Sae}({\mathbf{x}})$.

Figure: Figure 11: Percent decrease in FVU when doing a linear prediction with the norms squared of both SAE error and SAE nonlinear error, as opposed to with just the norm squared of SAE error.
Refer to caption: x12.png

Using these new reconstructions of ${\mathbf{x}}$, we repeat Eq. 3 and do a linear transformation from ${\mathbf{x}}$ to the inference time optimized reconstructions. We then regenerate the same scaling plot as Fig. 1 and show this figure in Fig. 10(a). Our first finding is that pursuit indeed decreases the total FVU of $\texttt{Sae}({\mathbf{x}})$ by $3$ to $5\%$; as Smith () only showed an improvement on a small $1$ layer model, to the best of our knowledge we are the first to show this result on state of the art SAEs. Our most interesting finding, however, is that the $\texttt{FVU}_{\texttt{nonlinear}}$ stays almost constant when compared to the original SAE scaling in Fig. 1. In other words, if our tests are accurate, most of the reduction in FVU comes from better learning $\texttt{Dense}({\mathbf{x}})$ and reducing the linearly explainable error, not from reducing $\texttt{Introduced}({\mathbf{x}})$. In Fig. 1, we plot the additional reduction in $\texttt{FVU}_{\texttt{nonlinear}}$ as the contribution of encoder error; because $\texttt{FVU}_{\texttt{nonlinear}}$ stays almost constant, this section is very narrow.

### 7.2 Linear Projections Between Adjacent SAEs

Our second approach for reducing nonlinear error is to try to linearly explain it in terms of the outputs of previous SAEs. The motivation for this approach is that during circuit analysis (see e.g. Marks et al. ()), an SAE is trained for every component in the model, and being able to explain parts of the SAE error in terms of prior SAEs would directly decrease the magnitude of noise terms in the discovered SAE feature circuits. For the Gemma 2 architecture at the locations the SAEs are trained on, each residual activation can be decomposed in terms of prior components:

$$ $\displaystyle\texttt{Resid}_{\texttt{layer}}=\texttt{MlpOut}_{\texttt{layer}}+ {\texttt{RMSNorm}}(\texttt{O}_{\texttt{proj}}(\texttt{AttnOut}_{\texttt{layer} }))+\texttt{Resid}_{\texttt{layer}-1}$ (11) $$

In Fig. 10(b), we plot the $R^{2}$ of a regression from each of these right hand side components to each of the different components of an SAE trained on $\texttt{Resid}_{\texttt{layer}}$ ($\texttt{SaeError}({\mathbf{x}})$, $\texttt{LinearError}({\mathbf{x}})$, and $\texttt{NonlinearError}({\mathbf{x}})$) for layer $19$-$20$ with all SAEs chosen with width $16k$ and $L_{0}\approx 60$. We find that we can explain a small amount (up to $\approx 10\%$) of total $\texttt{SaeError}({\mathbf{x}})$ using previous components, which may be immediately useful for circuit analysis.

We also find that $\texttt{SaeError}({\mathbf{x}})$ itself can explain $50\%$ of the variance in the nonlinear error, although this may not be entirely surprising, as the nonlinear error is a function of $\texttt{Sae}({\mathbf{x}})$:

$$ $\displaystyle\texttt{NonlinearError}({\mathbf{x}})$ $\displaystyle=\texttt{SaeError}({\mathbf{x}})-\texttt{LinearError}({\mathbf{x}})$ $\displaystyle=({\mathbf{x}}-\texttt{Sae}({\mathbf{x}}))-\texttt{LinearError}({ \mathbf{x}})$ $$

These results mean that we might be able to explain some of the SAE Error using a circuits level view, but overall there are still large parts of each error component unexplained.

## 8 Conclusion

The fact that SAE error can be predicted and analyzed at all is surprising; thus, our findings are intriguing evidence that SAE error, and not just SAE reconstructions, are worthy of analysis. Indeed, as a byproduct of studying SAE error, we have discovered a number of interesting practical applications: we can predict which tokens will have the highest error in a large SAE without needing to train it, and we can decrease error terms in SAE circuit analysis. Additionally, the presence of constant nonlinear error at a fixed sparsity as we scale implies that scaling SAEs may not be the only (or best) way to explain more of model behavior. Future work might explore alternative penalties besides sparsity or new ways to learn better dictionaries. We note that our tests are also approximate; we argue for the existence of $\texttt{Introduced}({\mathbf{x}})$ as a separate term from $\texttt{Dense}({\mathbf{x}})$, but the exact magnitude of each component remains uncertain. Ultimately, we believe that there is still room to make SAEs better, not just bigger.

#### Acknowledgments

Our work benefited greatly from the thoughtful comments and discussions provided by (in alphabetical order) Joseph Bloom, Lauren Greenspan, Jake Mendel, and Eric Michaud. We are deeply appreciative of their contributions. This work was supported by Erik Otto, Jaan Tallinn, the Rothberg Family Fund for Cognitive Science, and IAIFI through NSF grant PHY-2019786. JE was supported by the NSF Graduate Research Fellowship (Grant No. 2141064). LS was supported by the Long Term Future Fund.

## Appendix A Extra Error Prediction Experiments

### A.1 Note on Feature Shrinkage

Earlier SAE variants were prone to feature shrinkage: the observation that $\texttt{Sae}({\mathbf{x}})$ systematically undershot ${\mathbf{x}}$. Current state of the art SAE variants (e.g. JumpReLU SAEs, which we examine in this work), are less vulnerable to this problem, although we still find that Gemma Scope reconstructions have about a $10$% smaller norm than ${\mathbf{x}}$. One potential concern is that the ${\mathbf{b}}^{*}$ in Eq. 3 that we learn is merely predicting this shrinkage. If this was the case, then the cosine similarity of the linear error prediction $({\bm{b}}^{*})^{T}\cdot{\mathbf{x}}$ with ${\mathbf{x}}$ would be close to $1$; however, in practice we find that it is around $0.5$, so ${\bm{b}}^{*}$ is indeed doing more than predicting shrinkage.

### A.2 Breaking Apart Error Per Token

In Fig. 12, we show the same subset of tokens as in Fig. 6, but now broken apart into linearly predictable and non-linearly predictable components. That is, we learn ${\bm{b}}^{*}$ for each SAE as in Eq. 3, and then plot the norm of ${\bm{b}}^{*}\cdot{\mathbf{x}}$ at the norm of the linearly predictable error on the left, and plot the norm of $\texttt{SaeError}({\mathbf{x}})-{\bm{b}}^{*}\cdot{\mathbf{x}}$ as the norm of the non-linearly predictable error on the right. We see that the linearly predictable error decreases as we scale SAE width, but the non-linearly predictable error mostly stays constant. This is especially interesting because the result in Fig. 1 just found this on an average level, whereas here we find the same result holds on a per-token level.

### A.3 Norm Prediction Baselines

In Fig. 14, we run a linear regression from different components to $\lVert\texttt{SaeError}({\mathbf{x}})\rVert$. We find that is is not “easy” to predict SAE error norm, especially at later layers; the token identity, SAE L0, activation norm, and model loss all do significantly worse than using the full activation. It is interesting to note that at the first few layers, token identity does better at predicting SAE error than a probe of the activations; this is perhaps not surprising, since recent results from e.g. Lad et al. () show that very early layers primarily operate on a per token level.

Figure: Figure 12: Per-token breakdown of linearly predictable and non-linearly predictable SAE error across SAE scale. We show the same tokens as in Fig. 6. The norm of linear error decreases with SAE width, whereas the norm of nonlinear error stays mostly constant.
Refer to caption: x13.png

## Appendix B More Info on Modeling Activations

### B.1 Proof of Claim from Section 5.1

Say we have a set of $m$ unit vectors ${\bm{y}}_{1},{\bm{y}}_{2},\ldots,{\bm{y}}_{m}\in\mathbb{R}^{d}$. We will call these “feature vectors”. Define ${\bm{Y}}\in\mathbb{R}^{d\times m}$ as the matrix with the feature vectors as columns. We then define the Gram matrix ${\mathbf{G}}_{{\mathbf{Y}}}\in\mathbb{R}^{m\times m}$ of dot products on ${\mathbf{Y}}$:

$$ $\displaystyle({\mathbf{G}}_{{\mathbf{Y}}})_{ij}=({\mathbf{Y}}^{T}{\mathbf{Y}}) _{ij}={\bm{y}}_{i}\cdot{\bm{y}}_{j}$ $$

We now will define a random column vector ${\mathbf{x}}$ that is a weighted positive sum of the $m$ feature vectors, that is, ${\mathbf{x}}=\sum_{i}{\textnormal{w}}_{i}{\bm{y}}_{i}$ for a non-negative random vector ${\mathbf{w}}\in\mathbb{R}^{m}$. We say feature vector ${\bm{y}}_{i}$ is active if ${\textnormal{w}}_{i}>0$. We now define the autocorrelation matrix ${\mathbf{R}}_{{\mathbf{w}}}\in\mathbb{R}^{m\times m}$ for ${\mathbf{w}}$ as

$$ $\displaystyle{\mathbf{R}}$ $\displaystyle=\mathbb{E}({\mathbf{w}}{\mathbf{w}}^{T}).$ $$

We are interested in breaking down ${\mathbf{x}}$ into its components, so we define a random matrix ${\mathbf{X}}$ as
${\mathbf{X}}_{ij}=w_{j}{\mathbf{Y}}_{ij}$, i.e. the columns of ${\bm{Y}}$ multiplied by ${\mathbf{w}}$. We can now define the Gram matrix ${\mathbf{G}}_{{\mathbf{X}}}\in\mathbb{R}^{m\times m}$:

$$ $\displaystyle({\mathbf{G}}_{{\mathbf{X}}})_{ij}$ $\displaystyle=({\mathbf{X}}^{T}{\mathbf{X}})_{ij}={\textnormal{w}}_{i}{ \textnormal{w}}_{j}{\bm{y}}_{i}\cdot{\bm{y}}_{j}$ $\displaystyle{\mathbf{G}}_{{\mathbf{X}}}$ $\displaystyle=({\mathbf{w}}{\mathbf{w}}^{T})\odot{\mathbf{G}}_{{\mathbf{Y}}}$ $\displaystyle\mathbb{E}({\mathbf{G}}_{{\mathbf{X}}})$ $\displaystyle={\mathbf{R}}_{{\mathbf{w}}}\odot{\mathbf{G}}_{{\mathbf{Y}}},$ $$

where $\odot$ denotes Schur (elementwise) multiplication.
The intuition here is that the expected dot product between columns of ${\mathbf{X}}$ depends on the dot product between the corresponding columns of ${\mathbf{Y}}$ and the correlation of the corresponding elements of the random vector.

Figure: Figure 13: $R^{2}$ for linear regressions of SAE error norms with different regressors. We run on Gemma Scope 9B SAEs of size $131k$ with $L_{0}\approx 60$. Activations perform the best except on the first few layers.
Refer to caption: x14.png

We will now examine the L2 norm of ${\mathbf{x}}$:

$$ $\displaystyle\left\lVert{\mathbf{x}}\right\rVert_{2}^{2}$ $\displaystyle=\sum_{ij}{\textnormal{w}}_{i}{\textnormal{w}}_{j}{\mathbf{y}}_{i }{\mathbf{y}}_{j}$ $\displaystyle=\left\lVert({\mathbf{w}}^{T}{\mathbf{w}})\odot{\mathbf{G}}_{{ \mathbf{Y}}}\right\rVert_{F}^{2}=\operatorname{Tr}({\textnormal{w}}{\mathbf{w} }^{t}{\mathbf{G}}_{{\mathbf{Y}}})={\mathbf{w}}{\mathbf{G}}_{{\mathbf{Y}}}{ \mathbf{w}}^{T}$ $$

We can also take the expected value:

$$ $\displaystyle\mathbb{E}(\left\lVert{\mathbf{x}}\right\rVert_{2}^{2})$ $\displaystyle=\operatorname{Tr}({\mathbf{R}}_{{\mathbf{w}}}{\mathbf{G}}_{{ \mathbf{Y}}})$ $$

Our goal is to find a direction ${\bm{a}}\in\mathbb{R}^{d}$ that when dotted with ${\mathbf{x}}$ predicts $\left\lVert{\mathbf{x}}\right\rVert_{2}^{2}$. In other words, we want to find ${\mathbf{a}}$ such that

$$ $\displaystyle\left\lVert{\mathbf{x}}\right\rVert_{2}^{2}\approx{\bm{a}}^{T}{ \mathbf{x}}={\bm{a}}^{T}\sum_{i}{\textnormal{w}}_{i}{\mathbf{y}}_{i}={\bm{a}}^ {T}{\mathbf{Y}}{\mathbf{w}}$ $$

Combining equations, we want to find ${\mathbf{a}}$ such that

$$ $\displaystyle{\bm{a}}^{T}{\mathbf{Y}}{\mathbf{w}}\approx\left\lVert{\mathbf{x} }\right\rVert_{2}^{2}=({\textnormal{v}}w^{T}{\mathbf{G}}_{{\mathbf{Y}}}{ \mathbf{w}})$ $$

Let us first consider the simple case where for all $i\neq j$, $y_{i}$ and $y_{j}$ are perpendicular. Then our goal is to find ${\bm{a}}$ such that

$$ $\displaystyle{\bm{a}}^{T}{\mathbf{Y}}{\mathbf{w}}\approx\operatorname{Tr}({ \mathbf{w}}{\mathbf{G}}_{{\mathbf{Y}}}{\mathbf{w}}^{T})=\sum_{i}\left<y_{i},y_ {i}\right>{\textnormal{w}}_{i}^{2}=\sum_{i}{\textnormal{w}}_{i}^{2}=\left \lVert{\mathbf{w}}\right\rVert_{2}^{2}={\mathbf{w}}^{T}{\mathbf{w}}$ $$

Since all of the $y_{i}$ are perpendicular, WLOG we can write ${\bm{a}}=\sum_{i}{b}_{i}{\bm{y}}_{i}+{\bm{c}}$ for a vector ${\bm{c}}\in\mathbb{R}^{d}$ perpendicular to all ${\bm{y}}_{i}$ and a vector ${\bm{b}}\in\mathbb{R}^{m}$. Then we have

$$ $\displaystyle{\bm{a}}^{T}{\mathbf{Y}}{\mathbf{w}}$ $\displaystyle=\left(\sum_{i}b_{i}{\bm{y}}_{i}+{\bm{c}}\right)^{T}{\mathbf{Y}}{ \mathbf{w}}$ $\displaystyle={\bm{b}}^{T}{\mathbf{w}}$ $$

Since ordinary least squares produces an unbiased estimator, we know that if we use ordinary least squares to solve for ${\bm{b}}$, $\mathbb{E}({\bm{b}}^{T}{\mathbf{w}})=\mathbb{E}({\mathbf{w}}^{T}{\mathbf{w}})$. Thus,

$$ $\displaystyle\sum_{i}{b}_{i}\mathbb{E}({w}_{i})=\sum_{i}\mathbb{E}({w}_{i}^{2})$ $\displaystyle{b}_{i}=\mathbb{E}({w}_{i}^{2})/\mathbb{E}({w}_{i})$ $$

Now that we have $b_{i}$, we can solve for the correlation coefficient between ${\bm{a}}^{T}{\mathbf{x}}={\bm{b}}^{T}{\mathbf{w}}$ and $\lVert{\mathbf{x}}\rVert_{2}^{2}={\mathbf{w}}^{T}{\mathbf{w}}$. This gets messy when using general distributions, so we focus on a few simple cases.

The first is the case where each ${\textnormal{w}}_{i}$ is a scaled independent Bernoulli distribution, so ${w}_{i}$ is $s_{i}$ with probability $p_{i}$ and 0 0 otherwise. Then ${b}_{i}=s_{i}$. We also have that $\mathbb{E}({\mathbf{w}}^{T}{\mathbf{w}})=\mathbb{E}({\bm{b}}^{T}{\mathbf{w}})=
\sum_{i}s_{i}^{2}p_{i}=\mu$.

$$ $\displaystyle\rho$ $\displaystyle=\frac{\mathbb{E}({\bm{b}}^{T}{\mathbf{w}}{\mathbf{w}}^{T}{ \mathbf{w}})-\mu^{2}}{\sqrt{\mathbb{E}({\mathbf{w}}^{T}{\mathbf{w}}{\mathbf{w} }^{T}{\mathbf{w}})-\mu^{2}}\sqrt{\mathbb{E}({\mathbf{b}}^{T}{\mathbf{w}}{ \mathbf{b}}^{T}{\mathbf{w}})-\mu^{2}}}$ $\displaystyle=\frac{\sum_{i}s_{i}^{4}(p_{i}-p_{i}^{2})}{\sqrt{\sum_{i}s_{i}^{4 }(p_{i}-p_{i}^{2})}\sqrt{\sum_{i}s_{i}^{4}(p_{i}-p_{i}^{2})}}=1$ $$

That is, for Bernoulli variables, ${\mathbf{x}}=\sum_{i}s_{i}{\bm{y}}_{i}$ is a perfect regression vector.

The second is the case when each ${\textnormal{w}}_{i}$ is an independent Poisson distribution with parameter $\lambda_{i}$. Then $\mathbb{E}({\textnormal{w}}_{i})=\lambda_{i}$ and $\mathbb{E}({\textnormal{w}}_{i}^{2})=\lambda_{i}^{2}+\lambda_{i}$, so ${b}_{i}=\lambda_{i}+1$. We also have that $\mathbb{E}({\mathbf{w}}^{T}{\mathbf{w}})=\mathbb{E}({\bm{b}}^{T}{\mathbf{w}})=
\sum_{i}\lambda_{i}^{2}+\lambda_{i}=\mu$. Finally, we will use the fact that $\mathbb{E}({\textnormal{w}}_{i}^{3})=\lambda_{i}^{3}+3\lambda_{i}^{2}+\lambda_
{i}$ and $\mathbb{E}({\textnormal{w}}_{i}^{4})=\lambda^{4}+6\lambda^{3}+7\lambda^{2}+\lambda$. Then via algebra we have that

$$ $\displaystyle\rho$ $\displaystyle=\frac{\sum_{i}2\lambda_{i}^{3}+3\lambda_{i}^{2}+\lambda_{i}}{ \sqrt{\sum_{i}4\lambda^{3}+6\lambda_{i}^{3}+\lambda_{i}}\sqrt{\sum_{i}\lambda_ {i}^{3}+2\lambda_{i}^{2}+\lambda_{i}}}$ $$

For the special case $\lambda_{i}=1$, we then have

$$ $\displaystyle\rho=\frac{6}{\sqrt{66}}\approx 0.73$ $$

### B.2 Empirical Norm Prediction

In this experiment, we aim to determine to what extent our analysis in Section 5.1 holds true in practice on almost orthogonal true SAE features. Thus, we use a random vector that we can control: $\texttt{Sae}({\mathbf{x}})$. Specifically, we learn a probe ${\bm{a}}^{*}$ for the Gemma Scope layer $20$, width $131k$, $L_{0}=62$ SAE as in Eq. 2, except with the regressor equal to $\texttt{Sae}({\mathbf{x}})$ and the target equal to $\lVert\texttt{Sae}({\mathbf{x}})\rVert:$

$$ $\displaystyle{\bm{a}}^{*}=\operatorname*{arg\,min}_{{\bm{a}}\in\mathbb{R^{d}}} \left\rVert{\bm{a}}^{T}\cdot\texttt{Sae}({\mathbf{x}})-\lVert\texttt{Sae}({ \mathbf{x}})\rVert^{2}_{2}\right\lVert_{2}$ (12) $$

One important note is that we subtract the bias from $\texttt{Sae}({\mathbf{x}})$ so that it is purely a sparse sum of SAE features (this makes analysis easier). For each SAE latent from the SAE, we then compute

$$ $\displaystyle({\bm{a}}^{*})^{T}\cdot\texttt{latent}_{i}$ (13) $$

Finally, we plot this dot product against the average latent activation in Fig. 14. If ${\bm{a}}^{*}$ indeed equals the sum of the latents weighted by their activation, as we predict in Section 5.1, then these two quantities should be approximately equal, which we indeed see in the figure.

### B.3 Synthetic SAE Error Vector Experiments

The results for different Gaussian noise amounts versus percentage of features ablated are shown in Fig. 15. On this distribution of vectors, the test works as expected; the variance explained by $\texttt{Sae}({\mathbf{x}})+{\bm{a}}^{T}{\mathbf{x}}$ is a horizontal line proportional to $\texttt{Introduced}({\mathbf{x}})$, while the gap between this horizontal line and the asymptote of the variance explained by $\texttt{Sae}({\mathbf{x}})$ is proportional to $\texttt{Dense}({\mathbf{x}})$.

Figure: Figure 15: Top: When controlled amounts of noise are added to synthetic data $\texttt{Sae}({\mathbf{x}}^{\prime})$ and ${\mathbf{x}}^{\prime}$, the result is a plot similar to Fig. 1. Bottom: The nonlinear and linear error estimates (as shown at top) accurately correlate with the amount of noise added. The exact correlation between synthetic added noise and resulting estimated error components across these noise levels are shown in Table 5.2
Refer to caption: x16.png

We also tried running this test on a sparse sum of random vectors, which did not work as well, possibly due to not including the structure of the SAE vectors (Giglemiani et al., ); see Appendix C for more details.

## Appendix C Synthetic Experiments with Random Data

For this set of experiments, we generated a random vector ${\mathbf{x}}^{\prime}$ that was the sum of a power law of $100k$ random gaussian vectors in $\mathbb{R}^{4000}$ with expected $L_{0}$ of around $100$. To simulate the SAE reconstruction and SAE error, we simply masked a portion of the vectors in the sum of ${\mathbf{x}}^{\prime}$. Unlike the more realistic synthetic data case we describe in Section B.3, this did not work as expected: even in the case with no noise added to ${\mathbf{x}}^{\prime}$ or the simulated reconstruction, the variance explained by the sum of the linear estimate of the error plus the reconstructed vectors plotted against the number of features “ablated” formed a parabola (with minimum variance explained in the middle region), as opposed to a straight line as in Fig. 15.

We note that this result is not entirely surprising: other works have found that random vectors are a bad synthetic test case for language model activations. For example, in the setting of model sensitivity to perturbations of activations, Giglemiani et al. () found they needed to control for both sparsity and cosine similarity of SAE latents to produce synthetic vectors that mimic SAE latents when perturbed.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 4
questions: - Why was Gemma analyzed? Why was Gemma Scope chosen? What is special about these choices? (Availability as an open source model with a published SAE is an acceptable answer, but it that should be stated in the paper that the choice was made out of convenience.)
- Did you train the SAE from scratch, or are you using a the published checkpoint?
- What type of sparse autoencoder is considered? How was it trained? (As mentioned above, briefly describe the technique if you are using a published checkpoint.)
- Figure 1:
  - What are the equations in the labels supposed to mean?
  - Where did the data come from?
  - Exactly which part of the figure is “dark matter”?
- Equation 1:
  - is the vector w “random” or is it actually a coordinate of x in the y basis?
  - Is ||w||_1 << d the right equation? Couldn’t one of the components be much larger than one?
- Line 196: What is the set up of the synthetic set up?
- What is L_0?
- Line 242 --The exact set up of the random vectors here was unclear.
- What is Figure 3b supposed to show?
- Line 240: Exactly what is “likely” supposed to mean?
- Line 264: Bibliography: Gemma reference has a broken last name: “Team”
- Figure 4: The caption is not a full sentence and not clear. Expand the caption.
- What does the last equation on Line 490 say?
- Appendix A:
  - This section needs a little more introduction: What is the proof trying to show, and what does it imply?
  - Line 697: Why is the case \lambda=1 special, and what do we conclude from rho=0.73 ?
  - Is d > m?
  - Line 662: define WLOG
- Appendix B: report the results in the appendix, instead of just qualitatively describing them.
- Does Appendix C have a reference in the main text?
soundness: 2
strengths: The paper’s analysis is very thorough and a very compelling decomposition of the scaling of sparse autoencoders for feature extraction. The observations are very well described, and the theory and methodology is sound. I think it would be an impactful work, however, it does have one fatal shortcoming: (see below)
summary: This paper is an analysis into the error of sparse autoencoders applied to LLM interpredability. They address the shortcoming of the ability to reconstruct the hidden state, and observe that scaling laws show that an SAE would not be able to fully represent the hidden state, even in the limit. The theoretical framework is applied to decomposing the error of one layer of the Gemma 2 open source LLM, using the open source SAE published in a separate work, which agrees with the breakdown.
weaknesses: Weaknesses
The main weakness is that the method is only applied in one model, and even worse, only one layer in the model. The methodology is self described as empirical, but because only one context is analyzed, the findings are not sufficiently proven because the paper essentially shows only one data point. That is, is it possible that this only happens in this one layer for this one model? I would not think that the findings are exclusive to just this one context, but would it be possible to construct a LLM architecture for which this result does not hold? It should be easy to repeat the analysis for various open source models and different layers, and to thus illustrate that the observations and theory hold more widely. With just more demonstrations of the same trends across more models, the paper would be strong.


Another easy to correct shortcoming in the presentation is that details of the autoencoder training process are left out. There is a citation to GemmaScope, but given that this paper is trying to make a broader theoretical claim, a short description of the SAE training process would be apt. Similar to applying the method to more models and layers, it would also be interesting to verify if different SAE algorithms resulted in features that change the results.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. Why is it necessary to sanity check that estimated linear errors and estimated non-linear errors are correlated in Section 4.1? I found this experiment technical and in-the-weeds and couldn't tell why it needed be done (the interesting parts of the paper are with the actual LLM and actual error).

2. Why are all the linear extractors single linear matrices, or another SAE? Shouldn't an SAE with no sparsity penalty be applied somewhere so it's possible to learn features in superposition without interference (not possible with a plain linear matrix) and features that are dense (not possible with a sparsity penalty)?
soundness: 2
strengths: * I think that the biggest two problems with Sparse Autoencoders and Mechanistic Intepretability research is i) faithfulness of decomposition of models and ii) real-world application of insights. This paper is solid progress on i) because they ask why SAEs are currently limited.

* The paper describes a sensible theoretical decomposition of model activations into SAE learned features, dense features and non-linear features, and also measures these terms in practice. 

* The paper does a wide range of analyses: automated interpretability, FVU and loss measurements.
summary: This paper analyzes sparse autoencoder (SAE) errors in language models and found they could decompose these errors into three parts: unlearned sparse linear features, a dense linear term and a "nonlinear error" term that persists even as SAEs get larger. The paper studies the nonlinear error across different token posistions and SAE widths, with experiments on Gemma-2 9B. The paper attempts to reduce this nonlinear error through two methods: using gradient pursuit during inference (which only slightly helped) and leveraging SAE reconstructions from adjacent model components.

I found this paper complex, and while I think it is flawed in the current state I am happy to revise my opinion if my concerns are addressed.
weaknesses: 1. **Large sections of the paper are difficult to understand**

a. Some notation is defined in strange ways that require me to reread them many times. E.g. `SaeError(x) := x - Sae(x)` is defined with a -1 coefficient for `Sae(x)`, but `NonlinearError(x) := Sae(x) - Wx  - \sum_{i=0}^{m} w_i \vec{y}_i` is defined with a +1 coefficient for `Sae(x)`. I *think* that most of these problems are downstream of defining some parts of the notation with an end state in mind, e.g. the weak linear representation hypothesis suggests that there exists sum ideal dictionary decomposition of the vector, but other parts of the notation is defined with the current state in mind, e.g. you slot `SaeError(x)` into the sum with the existing `Sae(x)` and `x` terms and nothing else. However, I'm not confident that this consistently considering solely the end state or current state is either necessary or sufficient for making the notation clearer.

b. Some statements do not make sense after several times re-reading. E.g. "The intuition behind this test is that if ... its existence is not guaranteed".

c. "If this test is accurate, we can use it to estimate the linear component of the error, `Wx + Dense(x)`" but `Dense(x)` was introduced as possibly non-linear, and `Wx` is non-linear, so how is this the **linear** part of the error?

2. **I think the conclusions are too strong**.

The paper states "We also find that the norm of the `NonlinearError(x)` is constant on a per token level as
we scale SAE width", and while this section changes the definition of `NonlinearError` to be the error as determined by their method, in the conclusion the paper states "... the presence of constant nonlinear error ..." with no hedging.

I am not convinced that the methods in the paper are capturing *true* linear error and non-linear error. One reason this may be happening is that all the training on the error term `x - Sae(x)` may involve some vestigual `x` due to shrinkage (which could be boosted by 10x by the various predictors, since the appendix suggests there is 10% shrinkage). This would mean the various methods that predict linear or nonlinear error may be cheating and picking up on this shrinkage. Is this addressed in the paper? Why is there no hedging in the conclusion that the methods presented may not capture true (non-)linear errors, or even that the weak linear representation hypothesis may not be true. For example, in the extreme case where the SAE was entirely dead, we can predict `SaeError(x) = I * x` and so the SAE solely has linear error, which suggests there are some assumptions that need to be stated (that the SAE is sufficiently good at reconstructing I think).

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 1
questions: Check the weaknesses.
soundness: 2
strengths: 1. This paper focuses on the commonly ignored detail—SAE error—to provide more insights for thoroughly understanding a research subject related to the representation of language. 
2. It examines SAE error from multiple aspects, such as scaling law, norm prediction test, etc.
3. It investigates ways to reduce NonlinearError in detail.
summary: This paper tries to decompose the error components of sparse autoencoders (SAEs) to help better interpret language models. It uncovers that the SAE error can be predicted and analyzed, and provides insights for reducing it.
weaknesses: # Major

To be honest, I had a very hard time understanding this paper. Specifically,

1. How do you define the term “most common” (features..) on line 146? It is ambiguous. 
2. Section 4.1 is not well written, which makes me hard to understand the later sections of the paper. Specifically:
    1. Line 193 and 194 do not make sense. For instance, Dense is nonlinear, how can the sum between Wx and Dense be linear component of the error?
    2. What leads you to make the statement on line 200 to 201? You don’t even know what the “true features” are. Please elucidate.  
    3. It is questionable to claim “the percent of variance left unexplained by the regression will be an upper bound on the true variance explained by NonlinearError(x).” First, please provide a clear definition of “variance explained or unexplained”, perhaps in the appendix. Second, if this is a concept similar to the explained variance in PCA, I’m not sure if you can readily extrapolate that to nonlinear components. Some rigorous derivation is needed. If you are not sure, you need to stress this is an assumption.
    4. Here the so called synthetic setup is just for confirming that the *SAE is approximately an identity function on its image, i.e., the linear subspace of* $\mathbf{x}'$. There is no need for this long verbosity.
        1. Also, what is the motivation for this? I think there should be more clarification.
    5. Linear transformation $\mathbf{a}$ (line 161) should be denoted by $A$ for consistency (since you have $W\mathbf{x}$). You current notation makes it look like evaluating inner product.  I don’t think this is the same as the one in Section 4.2, right? Also, clarify its output dimension.
    6. The Gaussian noise simulation on line 204 to 207 is kinda questionable. 
        1. The set of $\mathbf{x}$ could be a curved low dimensional manifold. In this case the Gaussian noise—whose support is a practically a ball of the same dimension as $X$—cannot accurately simulate Dense(x). 
        2. The output of Dense and NonlinearError could be correlated in reality. 
        3. The functions Dense and NonlinearError could be continuous, so Gaussian noise may not accurately simulate them.
    7. Since SAE is approximately an identity function according to you (line 204 to 205), we can safely assume that $\mathbf{x}’$ and SAE($\mathbf{x}’$) are identical. Then adding Gaussian noises to each of them simply makes the simulated SaeError (i.e., $\mathbf{x}$ - SAE($\mathbf{x}$)) an Gaussian noise. I’m not sure how using a linear map $\mathbf{a}^\top \mathbf{x}$ can derive the results in Fig. 2, or maybe I just got it completely wrong since I can hardly understand your writing. I think you might need to elaborate on line 202 to 207.

I’d like to stop here since Section 4.1 has already baffled me enough, making me impossible to review the later sections. I think some revision is needed to streamline the narrative. 

# Minor

1. On line 135, I think you mean $\|\mathbf{w}\|_0\ll d$.
2. What is called the “linear subspace of $\mathbf{x}$” (line 187)? $\mathbf{x}$ is just a point in the activation space $X\ni\mathbf{x}$. If you mean a linear subspace of $X$, then is it *proper*, i.e., it cannot be $X$ itself?
3. Use the correct citation format \citep and \citet. For instance, line 35.

---

confidence: 2
contribution: 3
initial_rating: 3
presentation: 2
questions: Combined with previous section for clarity.
soundness: 2
strengths: - The core idea to decompose the SAE reconstruction area into various parts is very interesting and, as far as I am aware, novel. Having such a decomposition could potentially be very useful for training better SAEs and informing the debate around the extent to which the linear representation hypothesis holds.
- The authors attempt to measure these quantities using a wide range of experiments. They also study the downstream effects of each component of their decomposition, and investigate methods for reducing the nonlinear error. Overall, I think the breadth of their empirical work is good.
summary: This paper attempts to decompose the reconstruction error of sparse autoencoders into different interpetable components: (1) unlearned linear features, (2) residual dense features, and (3) nonlinear error introduced by the SAE. The authors explain their decomposition and then proceed to try to measure various components of the errors. They attempt to measure the linear and nonlinear portions of the error by training a linear regression on the activation vectors to predict the SAE recontruction error. They claim to find some irreducible nonlinear error arising from the SAE which does not disappear as the number of parameters in the SAE increases.
weaknesses: The main weakness of this paper is that the central quantities are not defined clearly enough for me to be able to properly understand what is going on, or the adequacy of their empirical experiments. My current understanding, rephrased in my own language is that the authors are making the following decomposition:
- Activation vector $x$
    - The sparse autoencoder reconstruction: $\textup{Sae}(x)$
    - The error of the SAE reconstruction: $\textup{SaeError}(x)$
        - Unlearned sparse features: $\sum_{i=m}^n w_i y_i$
        - Dense feaures which will not be learned by the SAE: $\textup{Dense}(x)$.
        - Additional error: (I don't know if the authors have a term for this)
            - Linearly predictable additional error: $Wx$
            - Nonlinear error: $\textup{NonlinearError}(x)$
(where my notation is that each bullet point is the sum of the bullet points nested at one level below it). This was my understanding after having spent quite some time re-reading Section 3, so I'll proceed as if this is correct, but I'm not confident and I'd appreciate clarification from the authors.

Assuming the above, I have several comments:
- The exposition in Section 3 needs to be substantially clearer for readers to be able to understand your definitions. For example, you should name the error mentioned in "the SAE introduces some error when making this approximation" and give it a consistent name. I also think equations (3) and (4) are misleading. According to my understanding, (4) is the central definition decomposing the SAE error and (3) is a consequence of (4) when you consider subtracting from $x$.
    - As an aside, the introduction of $W$ is poorly motivated here. Also, later on you're going to consider another sense in which the error can be linearly predicted from $x$ - namely using $a^T x$ to predict $\textup{SaeError}(x)$. I believe that these are not the same, but I was initially confused here and I don't understand the difference in intuition between them.
- In Figure 1, I don't see why Dense Features and Linear Error should be grouped together. They seem like quite different quantities with different interpretations to me.
- The changes of notation and setup in the first paragraph on page 6 are very unhelpful given there's already a lack of clarity. I'd strongly recommend that the authors stick to consistent notation throughout the paper, and choose a single - clearly distinct - term and corresponding notation for each component of their decomposition and stick with it throughout the paper.
    - As an aside, if we're dropping $Wx$ in this paragraph, why did we have it in the first place? I'm not sure I ever understood the intuition for having it.
- Since you are agnostic to the SAE architecture, you don't introduce any notation for the features that the SAE learns. This lead to me getting confused and originally not understanding that the SAE reconstruction error comes from essentially four places: learning wrong features, learning wrong feature weights, unlearned linear features, and (unlearned) dense features. My understanding is that the nonlinear error is effectively measuring the first two. Is that right? If so, a clarifying comment in this direction might be helpful.

Secondly, I did not understand the description of and intuition behind the experiment in the first paragraph of page 4. I think this might be downstream of the fact that I haven't completely understood the authors' decomposition of the SAE error. But in any case, either the authors need to offer a clearer definition of the quantities they are working with, or more intuition for why this experiment claims to measure what they are hoping - and probably both.

Given this lack of clarity in a couple of key places in the manuscript, it was hard for me to engage with the more detailed experimental results in Sections 5 onwards, since I couldn't understand what the authors were actually hoping to measure. The problem that the authors are trying to study seems fundamentally very interesting, and I'm optimistic that some version of this paper could be very solid, but as it stands it's not possible to appreciate the authors' contributions.

More minor points:
- I think the synthetic experiment setup in Section 4 is unlikely to be particularly realistic - particularly the assumption of Gaussian noise. (My read of Gurnee (2024) which the authors cite regarding pathological reconstruction errors makes the Gaussian assumption likely incorrect.) But, this is not a central concern.
- I suggest that the authors read over their manuscript for minor typographical errors. Ones that I caught include: issues with citation formatting in several places, summation indices in equations (3) and (4) being incorrect, and several stray commas.