Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality
Authors: Sewoong Lee
Adam Davies
Marc E. Canby
Julia Hockenmaier
Venue: COLM 2025 (2025)
Keywords: mechanistic interpretability
sparse autoencoder
TL;DR: The paper introduces Approximate Feature Activation (AFA) and the $\varepsilon$LBO metric to address the lack of principled hyperparameter selection in top-k SAEs and to evaluate SAEs using quasi-orthogonality.

Abstract: Sparse autoencoders (SAEs) are widely used in mechanistic interpretability research for large language models; however, the state-of-the-art method of using $k$-sparse autoencoders lacks a theoretical grounding for selecting the hyperparameter $k$ that represents the number of nonzero activations, often denoted by $\ell_0$. In this paper, we reveal a theoretical link that the $\ell_2$-norm of the sparse feature vector can be approximated with the $\ell_2$-norm of the dense vector with a closed-form error, which allows sparse autoencoders to be trained without the need to manually determine $\ell_0$. Specifically, we validate two applications of our theoretical findings. First, we introduce a new methodology that can assess the feature activations of pre-trained SAEs by computing the theoretically expected value from the input embedding, which has been overlooked by existing SAE evaluation methods and loss functions. Second, we introduce a novel activation function, top-AFA, which builds upon our formulation of approximate feature activation (AFA). This function enables top-$k$ style activation without requiring a constant hyperparameter $k$ to be tuned, dynamically determining the number of activated features for each input. By training SAEs on three intermediate layers to reconstruct GPT2 hidden embeddings for over 80 million tokens from the OpenWebText dataset, we demonstrate the empirical merits of this approach and compare it with current state-of-the-art $k$-sparse autoencoders.
Our code is available at: https://github.com/SewoongLee/top-afa-sae.

Published as a conference paper at COLM 2025
Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality
Sewoong Lee, Adam Davies", Mare E. Canby" & Julia Hockenmaier Siebel School of Computing and Data Science
University of Illinois Urbana-Champaign Illinois, IL 61801, USA (samue127, adavies4, marcec2,
Abstract
Sparse autoencoders (SAEs) are widely used in mechanistic interpretability research for large language models; however, the of-te- method of using k-sparse autoencoders lacks theoretical grounding for selecting the hyperparameter that represents the number of nonzero activations, often denoted by lg. In this paper, we reveal a theoretical link that the fz-norm the sparse feature vector can approximated with the fz-norm of the dense vector with closed-form error, which allows sparse autoencoders to be trained without the need to manually determine fg. Specifically, we validate two applications of our theoretical findings. First, we introduce a new methodology that can assess the feature activations of pre-trained SAEs by computing the theoretically expected value from the input embed- ding, which has been overlooked by existing SAE evaluation methods and loss functions. Second, we introduce novel activation function, top-AFA, which builds upon our formulation of approximate feature activation (AFA). This function enables top-k style activation without requiring a constant hyperparameter k to be tuned, dynamically determining the number of acti- vated features for each input. By training SAEs on three intermediate layers to reconstruct GPT2 hidden embeddings for over 80 million tokens from the OpenWebTeed Text dataset, we demonstrate the empirical merits of this ap- proach and compare it with current state-of-the-art k- k-sparse autoencoders. Our code is available at: https: /gg thub. com/ /wt////wwwwo/eeeeeeeeeeeeeeeeeee
wwwrooonnneeeeooooooeeeeoo top- afa- sae.
1 Introduction
Language models pack meaning into dense vectors, but what if we could unpack them into separate, understandable pieces? To make this possible, sparse autoencoders (SAEs) have brought significant advances to mechanistic interpretability by demonstrating that dense embeddings inside language models can be effectively decomposed into linear combination of human-interpretablee feature vectors (Elhage et al., 2022; Bricken et al., 2023; Huben al., 2023; Lieberum et al., 2024). Despite recent advances in sparse autoencoder evaluation, a crucial aspect has remained overlooked: the relationship between input embeddings and sparse feature vectors. Existing approaches enforce sparsity - either by limiting the number of active units or by penalizing the number of nonzero values with little justification. In other words, although existing methods aim to construct sparse features corresponding to inputs, the selection of sparsity levels in these methods is independent of the inputs themselves. This missing link - between the input and its feature representation is at the heart of what sparse autoencoders are supposed to recover and is the focus of this paper.
To address this issue, we take fundamentally different approach: rather than evaluating SAEs based solely on sparsity or reconstruction error, we focus on the underlying relationship between input embeddings and their corresponding feature activations. This shift in per-
"These authors contributed equally this work.

---

Published as a conference paper at COLM 2025
spective leads us to develop new theoretical approximation for feature activation, along with practical tools for evaluating and designing sparse autoencoders under this framework. Our approach applies to pretrained SAEs commonly used in mechanistic interpretability, including models based on different sparsity penalties, such as GPT-2 Small (Bloom, 2024) and Gemma Scope (Lieberum et al., 2024). Furthermore, when training SAEs from scratch, our approach achieves reconstruction loss better than that of state-of-the-art k-sparse SAEs (Makhzani & Frey, 2013; Gao et al., 2024; Bussmann al., 2024), without requiring the hyperparameter k to be tuned. Our findings not only bridge a theoretical gap in current SAE evaluations but also open the door to novel avenues for experimental exploration.
The contributions of this paper are:
We introduce Approximate Feature Activation (AFA), closed-form estimation of the magnitude of sparse feature activations with provable error bounds.
We present the ZF Plot to visualize and diagnose over- or under-activation of features based on the theoretical framework of AFA.
We formalize e-quasi-orthogonalit as a geometric constraint arising from the su- perposition hypothesis, connecting it to the Johnson-Lindenstrauss Lemma, and propose ELBO, the lower bound of quasi-orthogonality, a novel metric for evaluating SAE feature space.
We propose top-AFA, norm-matching activation function that adaptively selects the number of active features for each input vector without tuning k. Combined with norm-matching loss CAFA, this leads to new SAE architecture, top-AFA SAE, that achieves better reconstruction performance compared to state-of-the-art top-k and batch top-k SAEs, while also offering stronger theoretical justification.
2 Preliminaries
Notation We write vectors in bold as v = (V1, , 2n) and refer to their scalar compo- nents D; in italics. We use v, to denote the i-th vector collection of vectors, and use v(-) to denote function that returns vector-valued output. Aij denotes the (i, (i,))-t entry of matrix A.
2.1 Sparse Autoencoders
The hidden embeddings of language models are densely packed with information, encoding overlapping features in high-dimensional space (Elhage et al., 2022). Individual dimensions do not correspond to specific, human-ure understandable concepts. To unpack these embed- dings into more interpretable pieces, sparse autoencoders (SAEs) learn to decompose them into sparse activation vectors that weight learned feature directions.
Formally, denote input sequence by x, which is transformed to embedding vector (x) E Rd in LLMs, which taken from the residual stream, the hidden embedding vector passed through the residual connections between transformer layers, of the last token index at layer (see Appendix for details). This embedding is used as the input to the SAE, and we write it as zxx) or for simplicity when the layer index and input are clear from context.
An SAE consists of an encoder and decoder, defined as:
f(z)) = oWWencz + benc), 2 Waeef(z) + baec
where f(zz) RR is sparse latent vector and 2 E Id is the reconstruction of z. The encoder matrix Wene E Rkkxd decoder matrix Waec E Rdxh and biases bene E R² baec E R are learned. The decoder matrix Wae is often referred to dictionary, as it contains the directions for reconstructing the input from the feature space. The activation function (-)) (e.g., ReLU Top-k (Makhzani & Frey, 2013)) enforces non-negativity sparsity. Training minimizes a loss function that balances reconstruction and sparsity:
((z) ||z
2 + Asparsity S(f(z)) + aLaux
(1)
2

---

Published as a conference paper at COLM 2025
where ((f) is sparsity penalty (e.g., |[f[],, || Asparsity controls the strength of the sparsity constraint, and Laux is an auxiliary loss that prevents latent units from becoming inactive via Ghost Grads (Adam Jermyn, 2024).
2.2 Hypotheses Behind Sparse Autoencoders
= (z) + bde
where E R f(z) E R², and
(2)
Linear Representation
Superpusition
As summarized Equation (2), the design of sparse autoencoders is built on certain assumptions: the linear representation hypothesis and the superposition hypothesis. In this section, we formally present these two hypotheses and provide intuition behind their roles in motivating the architecture.
Linear Representation. In this paper, unless otherwise specified, we use the term linear represenlation hypotkesis (LRH) with the following definition discussed in Elhage et al. (2022); Bricken et al. (2023); Nanda et al. (2023); Lieberum et al. (2024); Smith (2024); Engels et al. (2024b;a); Laptev al. (2025):
Linear Representation Hypothesis Internal activations of neural networks can be represented as linear combinations of vectors. Formally, for given input to the neural network model, hidden embeddings z(x) E Rd can represented by a linear transformation' of feature activation vector f E IK with constant weight matrix W E Rd
z(x) Wf
"This hypothesis also holds under affine transformations if we define f' [|]]]]] where b is the translation bias.
Superposition. In machine learning interpretability studies, the term superposition refers to the phenomenon in which models represent more features than the dimensionality of their representations (Elhage al., 2022). The statement that there can be more features than neurons in neural networks implies that neural network models can exploit the property of high dimension supported by the Johnson-Lindenstrnn lemma (Lindenstrauss & Johnson, 1984). To formally state:
Superposition Hypothesis. Let z(x) E Rd denote the hidden embeddings for given input x to an LLM. There exists function + IR' Rd for feature vectors R² such that:
z x = <f),, where d.
2.3 Theoretical Tools for High-Dimensional Analysis
Theorem 1 Johnson-Lindenstrauss Lemma). Lef Q R² be set of points. For ö (0,1/2) and d
2n there exists linear mapping Y R + Rd such that Vu, v E Q:
82
(1 - v112 Y Y (v) < (1+8) -- vI12.
Proof of Theorem. See Lindenstrauss & Johnson (1984); Vempala (2005); Kakade (2010).
The fo-ormm of vector denoted || counts the number of non-zero elements Although is not norm the mathematical sense, it widely referred to as &p "norm" " this context due to its usefulness in expressing sparsity. For more details the LRH, Appendix C.
3

---

Published as a conference paper at COLM 2025
Definition 1 (e- Let 5d {v E R |v|12 } be the unit sphere in the d-dimensional Euclidean space. For e E [0, 1),1,,mm a subset T C 5d-1 is called an e-quasi-orthogonal set if.
Set (Kainen & Kúrková, 1993; Kainen & Kúrková, 2020)).
u V u. /
3 Related Work
Finding Interpretable Dictionaries. Initially, Lee Sharkey (2022) investigated a toy setting where dense embeddings were explicitly constructed from combinations of ground-truth sparse feature vectors. This reversed setup enabled testing whether SAEs can recover known ground-truth features. The key finding was that, given a properly tuned & sparsity coefficient, SAEs could accurately recover the ground-truth components, identifying "Goldilocks zone" for sparsity regularization. This foundational insight later inspired follow-up work by Huben et al. (2023), extending the idea to real-world settings, showing that SAES trained on models like Pythia-70M can discover highly interpretable features.
Pre-trained SAEs: GPT-2 Small and Gemma Scope. Building on these insights, Bloom (2024) successfully extended sparse autoencoder training to GPT--2 Small (Radford al., 2019), releasing pretrained SAE model widely used in follow-up research. While retaining the i sparsity penalty, their model introduced several key architectural improvements: ghost gradients for addressing inactive features and unit-norm decoder. GPT-2 Small has since become a standard model for SAE research due to its manageable size and representative embedding structure, and has been studied in follow-up works such as Chaudhary & Geiger (2024); Bussmann et al. (2024); Minegishi et al. (2025).
Lieberum et al. (2024) introduced the Gemma Scope SAE, trained on Gemma 2's hidden embeddings (Gemma Team, 2024). Unlike prior SAEs that relied on & sparsity, Gemma Scope adopted an lo penalty to reduce shrinkage effects (Rajamanoharan et al., 2024), which had previously led to suppression of overall activations. Furthermore, they employed JumpReLU, an activation function that applies non-zero threshold to mitigate interference caused by superposition. The resulting model has become one of the most widely adopted SAEs in the interpretability community (Lin, 2023).
Top-k Activation. Gao et al. (2024) demonstrated the effectiveness of k-sparse autoencoders (Makhzani & Frey, 2013), which use sparsity-enforiing activation function that retains only the largest pre-activations per input. This approach eliminates the need for a separate sparsity penalty so that Asparsity = 0 in Equation between sparsity, measured by fo, and reconstruction error, measured by mean square error (MSE), suggesting that there exists a empirical boundary that SAEs cannot surpass. However, the main limitation of top-k is its reliance on fixed k, which lacks principled justification. To address this, Bussmann et al. (2024) proposed batch top-k, a variant that enforces sparsity average across batch rather than per input. Their method enables dynamic per-input activation while maintaining a fixed average sparsity, resulting in better reconstruction.
(1). They also discovered scaling law
Evaluation of Sparse Autoencoders The question of what makes a good SAE has been actively explored and remains open. Till (2024) explained that since there are infinitely many ways to decompose dense embedding, failing to ensure near-orthogonality may hinder the recovery of the true underlying features. Although this work proposed orthogonality as a key geometric criterion, it lacked formalization or empirical validation. Simple structural metrics, such as pairwise cosine similarity between decoder vectors, are easy to compute but remain model-centric and fail to capture data-driven behavior. To address this, recent efforts, such as SAE Bench (Karvonen et al., 2025), have introduced both supervised (e.g., using LLM judges) and unsupervised metrics to evaluate SAE quality from an input-driven perspective.

---

Published as a conference paper at COLM 2025
(a) SAE Evaluation
(b) Activation Function
Reoroitckki Taker, Explired Vallance
(Batch) Top-K
Analyical Approcinatin
Top-AFA (Ours)
IPI - [H]
Apprseinate
addwaaee
fMattic:: Ln:Sperily
feetume
KL Divorgencs
2
24 (Dunt: ALBO, APA Loos
Figure 1: (a) Comparison of SAE evaluation approaches. Existing unsupervised metrics, such as SAE Bench (Karvonen et al., 2025), assess either reconstruction quality (i.e., the relationship between and 2) or the sparsity of f in isolation. Our proposed AFA-bbseed evaluation fills this gap by introducing way to assess the alignment between input em- beddings and feature activations. top-k) with our adaptive top-AFA. While top-k uses constant sparsity level, top-AFA dynamically selects the number of active features per input by matching the activation norm with the input norm.
(b) Comparison of fixed-k activation (e.g,, top-k, batch
Approximating Feature Activation
Can we define an input-driven, theoretically justifiable metric that reflects feature activation quality, even under superposition? In this section, we define Approximate Feature Acti- vation (AFA) as the closed-form solution that approximates the l2 norm of a ground-truth sparse feature vector. In our study, we derive specific form called linear-FFA under the sim- plified setting where the linear representation hypothesis assumed to hold e.g,, in SAEs with one-layer decoder, as commonly used in recent work (Bloom, 2024; Lieberum et al., 2024; He et al., 2024; Nabeshima, 2024; Bart Bussmann, 2024; Gao et al., 2024; Bussmann et al., 2024).
First, we propose the notion of an e-quasi-identity matrix, denoted as _se, which utilizes the definition of gquasi-orthogonallnn and serves useful tool for quantifying superposition in high-dimensions spaces.
Definition 2 (e- (e-Quasi-ldentity Matrix). A matrix Ise Rixh is called E-quasi-identity matrix if satisfies the following properties:
{1, ifi=i,
Eij. i where |et,y E,
where each ei is scalar satisfying |et,) S e, for some fixed e [0,1) which represents the maximum off-diagonal deviation from the identity matrix for all i, e [h].
Now, let D E Ruxh be decoder (dictionary) matrix. If its columns are normalized, then the Gram matrix D D becomes the e-quasi-identity matrix ISe for some e, which is the maximum inner product magnitude between distinct dictionary column vectors. Formally,
Definition (Quasi-Othhoonaaiity of a Dictionary). We define the quasi-orthogonality of dictionary D as:
D.J. D.j
8 = max 1f1
|D. |D

---

Published as a conference paper at COLM 2025
In practice, pre-trained SAE decoders often do not satisfy this unit-norm constraint. How- ever, without loss of generality, we can construct an equivalent decomposition by normaliz- ing each decoder column and scaling the index of the corresponding feature vector with the decoder column's norm. This transformation allows us to formalize the following theorem:
Theorem (Linear-AFA) For a given input x, if an embedding vector z(x) E IR satisfies the linear representation hypothesis (LRH), then |zx(x)] approximates [| the square of 22-norm
||x(x)11 ||x(0]1
of feature activations f E RR, with an error bound || E
I+2(h TV c(h 1} where e is the quasi-orthogonaliyy of the dictionary D E IR used by the features, such that z(x)= D
Proof of Theorem. Since D is e-quasi-orthogonal dictionary, D = Ise, where s is e-quasiideentity matrix. Then,
|z(xx z(x) z(x) (Df) = (f
(15E) ΣE 1
By using Definition and applying Cauchy-Schwarz (3),
1z(xx |x(x)112 - ||
ΣE
E f
E
fillfille, & e([[F]] - -1((12) jEll; iti
E
jE|
iE|M
BE
S E -11111) e(h ((0h-1)]lE(3 1)]lE(12 ||z(x) E
(1 e(h ())(1))] (1+
(1+ 2 1))|l1)
:: | E [|zx()] + s(h - 1)),
5 How to Measure e: Quasi-Orthogonality of Dictionary
Although Theorem 2 provides principled connection between and f, it relies on the quasi-orthogonlity constant e, which is unknown in practice. To address this, we introduce two approaches to obtain the range of e.
Upper Bound Approach based on the JL Lemma. The JL lemma (Theorem 1) has often been suggested as a mathematical intuition that machine learning models gain an advantage in high-dimensional representation due to the exponential growth of almost orthogonal vectors (Elhage et al., 2022; Ghilardi et al., 2024). However, there is lack of formulation
regarding the closed form bound using the JL lemma. By deriving |el < 8 /20 In [d] from the JL lemma using the method described in Appendix F, we can define:
20 20 d
EJL
Although the JL lemma ensures the existence of a quasi-orthogonal basis within a bounded error, we find that pretrained SAE decoders often exceed this bound (see Appendix G). This reveals a key limitation: even state-of-the-ar SAES fail to learn decoders that achieve the theoretically permitted level of quasi-orthogonaaiity making their use unreliable.
Lower Bound Approach with Pre-trained SAE Features. A tighter and more practical bound can be obtained from SAE feature vectors for each input, from the lower end. The closed-form error bound derived in Theorem 2 provides a way to estimate the quasi- orthogonality constant e. Specifically, we use lower bound of e, based on the inequality || []f[]]] /hh 1)][f]13 < However, this time, the ground-tric -ruth is not accessible.

---

Published as a conference paper at COLM 2025
GPT2
Gemma
477 S71
927
522
422
100
200
140
130
130
Layert
Dxyend
Dxpor7
Layer10
Layer12
Layer
Layer 15
Layer 26
Lyyer 25
Layer
(ager
Lawer
Layer
Sefreence Lse
ZF Plots. ZF plots shows the relationship between dense embedding vector norm ([|z]12) and the learned feature activation norm ([]f][2) by layers. Each point corresponds sequence input to the language model, allowing us to compare the magnitude and error direction (i.e., over- or under-activation) of the SAE's features compared to the red dashed reference line, input by input (For more details, see Fig. 3). The shaded region visualizes the error bound based on e(h - 1) derived in Theorem 2, where e the quasi-orthogonality of dictionaries and /t is the dimensionality of the corresponding SAE feature vector. These plots are drawn using 1k input sequences of length 128 from the Open dataset. The phenomenon of increasing norm in higher layers is explained in Hex & Turn (2023).
Figure 2:
In this case, we can leverage the fact that SAEs are designed to estimate f, and denote this estimate fSAB- Thus, we can compute the epsilon lower bound:
ELOO (x) == ||z(x) 12 (fsAE(x)]l)
(h 1) fSAE x
In layman's terms, this bound represents the minimum é of quasi-orthogonality that any decoder must have in order to reconstruct the observed feature activations from the em- bedding. Since ELBO is computed directly from the feature vectors the very structure that SAEs ultimately aim to uncover - it offers decoder-agnosiee alternative that is free from the decoder-induced noise discussed in the upper bound approach.
6 Evaluation Metric: Missing Link between and f
Despite recent advances in SAE evaluation methods such as SAE Bench (Karvonen et al., 2025), Figure 1-(a) shows a key limitation in current metrics: they either evaluate the rela- tionship between the input embedding and reconstructed embedding 2, or they measure the sparsity of the feature vector alone. Attempts bridge and f have remained limited to measuring decoder cosine similarities, which do not assess their actual relationship based on inputs. In contrast, our approach introduces an evaluation metric that directly links and f, enabling to assess whether the feature activations input representation. This relationship between and f can be visualized using pretrained SAEs, shown in Figure 2. The geometrical intuition that is explained in Figure 3 allows for a fundamentally different perspective: instead of treating sparsity as an isolated property, evaluate whether the activations themselves are justified by the input.
appropriately aligned with the
we apply this finding to the evaluation of GPT-2 Small and Gemma Scope 2B, Figure shows that BL offers different perspective compared to mean squared error (MSE),
7

---

Published as a conference paper at COLM 2025
[1f]]
Over-ectivated
decomposition
Reference
with perfect resonstruction
Line
e.g. ||2||
lffill > 5,
3
while perfectly reconstructing hidden embedding
ZF Plat
4
wull
(8) Activated with desired nom
Hidden embedding vector
SAE's Linear Decomposition
|[<]|
(a) Case
Figure 3: Geometrical intuition behind ZF plots. A dense embedding vector (blue dashed) can be perfectly reconstructed using an over-activated set of SAE features (light orange solid), even when their norm is misaligned. This leads to a discrepancy between the norm of the input ([]z])) and the reconstructed activation norm (l) On the right, the ZF plot visualizes this mismatch per input, with the red dashed line indicating ideal alignment. Case (b) shows an over-activated decomposition; Farther distance from the reference line indicates either excessive activation despite seemingly good reconstruction the absence of a low-E dictionary that can reconstruct the input.
Algorithm 1 Top-AFA Activation Function
Require: E Bxd Wene Rdxk Waec RRxxd bene R², bdec E IRd
B: batch size
Ensure: Reconstructed embedding 2
1: Zcent
benc
2: a +
Zcentli
3: ReLU Wenc)
4: s < @ [W)cell22
> Ø: elementwise multiplication
5: π <- argsort(s, descending)
6: C + cumsum (s(7z) set C
C[-1] + x
D K is a large constant
7: Atarget + Va, C + VC
8: & + argmin, [|G Qtarget| +1
9: Construct mask m that retains the top indices in
10: flok f- fo
11: 2 + flopk Waec bdec
12: return
which is the most commonly used metric for evaluating sparse autoencoders (Gao et al., 2024; Bussmann et al., 2024). While overall trends are similar, there are differences in distributional shape (e.g., long tails), modality (e.g., unimodal vs. bimodal), and ranking order (MSE-based ranking vs. eLBO-based ranking).
Designing Novel Sparse Autoencoder
Activation Function. Top-k based activations suffer from fundamental limitation: the expected sparsity level E(Eo] remains unknown and theoretically unjustified (Gao et al., 2024). To address this, we propose the top-AFA, an activation function that adaptively selects the number of active features per input by matching the activation norm to a theoretically grounded target derived from the input embedding norm. This removes the need to manually set or estimate k, and instead provides indirect estimate of E[fo]. The core idea behind top-AFA is to activate the minimum number of features such that the sparse feature vector norm approximates the AFA target ||z|| 2, described in Figure 1-(b) and Algorithm 1. From this perspective, sparsity and reconstruction are no longer trade-off relationship. Instead, sparsity can be chosen for reconstruction.
8

---

Published as a conference paper at COLM 2025
GPT-2 Small
10-5
0.20
Norrelzed Mot
0.15
Gemma Scope
16-5
0.30
Nomatred Mee
0.25
0.20
0.15
0.05
1
2
10 11 13
14
15
16 18
19
20
22
23
24
Figure 4: Violin Plots for Comparing Different Evaluations of SAEs (Lower values are bet- ter). Blue violins (left) represent the normalized MSE (NMSE), which reflects reconstruction loss, while orange violins (right) correspond to eLBO, which measures quasi-orthogonality Distributions are computed over 1k input sequences of length 128 from the OpenWeb- Text dataset, and the region of each violin captures 99% of the distribution to mitigate the influence of extreme outliers. Higher eL.BO values imply that no sufficiently orthogonal dic- tionary can explain the feature activation, suggesting poor separation among the activated features. While overall trends across layers are similar, the best- and worst-ranked layers and the distributional shapes vary, showing differences between NMSE and eLBO. These results show that the two evaluation metrics yield different evaluations of the same models.
Loss Function. Can we design an SAE loss function that accounts for the amount of feature activation expected based each input vector? A natural option would be to directly use eLBO; however, has certain limitations when applied as training loss (see Appendix H). Given that our key theoretical takeaway is lim,-0 [| ||z||2, define the following simple loss function: L AFA
(ll || Adding CAA to the standard loss
in Equation (1) yields the following objective."
[(x) = ||x x &Laux + AAFA CAFA
Note that this additional loss term does not shift the optimum of the reconstruction objective, since the & reconstruction loss is minimized when Df and under this condition, Theorem guarantees that Eg [| = ||z|||-
Experimental Results. We follow the experimental setup of Bussmann et al. (2024), using h = 16 x d, but extend the evaluation to layers and 7 in addition to layer 8. Figure presents results for GPT-2, comparing Top-AFA with top-k and batch top-k. Notably, Top- AFA achieves better reconstruction performance than top-k-based baselines and exceeds the scaling law boundary observed by Gao et al. (2024).* This suggests that adaptively selecting activations based on AFA can lead to improvements beyond what fixed-sparsity approaches can achieve. However, as shown in Appendix I, the effectiveness of AFA-AFA depends on proper tuning the coefficient associated with the AFA loss LAFA and a stable coefficient value 1/16 was observed across all layers.5
We follow prior work in setting the auxiliary loss weight a to 1/32, a commonly used value in SAE training (Gao et al., 2024; Bussmann et al., 2024).
4 'Layer was used Bussmann et (2024) validate the performance batch top-k.
SWhile Top-AFA introduces new hyperparameter AAFA, it differs fundamentally from the fixed sparsity hyperparameter used in top-k activations. Specifically, AAFA is training-timi time regularization

---

Published as a conference paper at COLM 2025
Layer
Layer?
Layer
10-
10%
Normalizee MSE
Normal lendlllzedde NSE
Norma Normalized MSE
Becch
Secc Texe
Sparsity (UO)
Sparsity (UO)
Sparsity (UO)
Figure 5: Reconstruction performance Top-AFA compared to top-k and batch top-k activation functions on GPT-2 layers 6, 7, and 8. These layers were chosen to represent the model's middle depth, with layer 6 positioned at the center of GPT-2's 12-layer architecture. Layer 8 was also used in Bussmann et al. (2024), allowing for direct comparison with prior work. The SAE with minimum reconstruction loss at each layer is marked with a star (+). Top-AFA outperforms other baselines on all tested layers, where its MSE (reconstruction loss) surpasses the scaling law boundary reported by Gao et al. (2024) These findings demonstrate that adaptively selecting activations can surpass the limitations of fixed-sparsity approaches. Detailed settings and results are provided in Appendix I.
Conclusion
We present a new theoretical and practical framework for SABs by addressing core lim- itation of top-k activations: the lack of principled k selection. Our Approximate Feature Activation (AFA) formulation provides closed-form estimate of the f2-norm of sparse activations, leading to ÉLBO, the first metric linking input embeddings to activation magni- tudes. Building on this, we propose tOp-AFA SAE, an SAE that adaptively selects active features by matching activation norms to dense embedding norms, removing the need to tune k. Experiments on GPT-2 show that tOp-AFA SAE outperforms top-k and batch top-k SAE. In conclusion, our research sheds light on key limitation in the existing literature the inaccessibility [b44 - bb taking a simple yet novel approach: approximating [[f2] ultimately provides a path towards obtaining to-
Acknowledgements
We thank Atticus Geiger for discussions with Adam Davies that influenced our early thinking on this work. This research was supported in part by award HR0011249XXC from the U.S. Defense Advanced Research Projects Agency Friction for Accountability in Conversational Transactions (FACT) program and the Illinois Computes project which is supported by the University of Illinois Urbana-Chin Ahtaan-Chhampain and the University of Illinois System, and used the Delta advanced computing and data resource which supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta is a joint effort of the University of Illinois Urbana-Chan Chhana-hhaapaign and its National Center for Supercomputing Applications. Adam Davies is supported by the NSF and the Institute of Education Sciences, U.S. Department Education, through Award #2229612 (National AI Institute for Inclusive Intelligent Technologies for Education). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of DARPA, the NSF, or the U.S. Department of Education.
coefficient whose influence is limited to the optimization phase, whereas & directly relates to inference- time because it refers the number of active features for each input example.
10

---

Published as a conference paper at COLM 2025
References
Adly Templeton Adam Jermyn. Ghost grads: An improvement on resam-
pling, 2024. URL https://transformmr-crree wttpcciccuusss.pp///24e4.4.
dict-Learning-ressmmp
Neel Nanda Bart Bussmann, Patrick Leask. Learning multi-level features
with matryoshka saes.
cttp://ppst////bb2222//550S50000 2024.
Joseph Bloom. Open source sparse autoencoders for all residual stream lay- (
ers of gpt2 small. soen-source-paare ott.re/rrrerrrteoooeeoeoeorreerr
2024.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arv preprint ar (rXo:2241..06410, 2024.
Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. Xro preprint ar arXto:2409.04475 2024. 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. URL https: //trannsforme--circuuttt pub/ 2022/toy model/index.he html.. html.
Joshua Engels, Eric Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear. Xiv preprint ar arXiv:2405.14860, 2024a.
Joshua Engels, Logan Riggs Smith, and Max Tegmark. Decomposing the dark matter of sparse autoencoders, 2024b. URL httpp::/ ntt///oorumiiddd122f0000r
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. Xiv preprint Xix:22406. 04093, 2024.
Gemma Team. Gemma. 2024. doi: 1034700/KAGGLE///330 URL https://ww. kaggle. com/m/ m00///3300.
Davide Ghilardi, Federico Belotti, Marco Molinari, and Jaehyuk Lim. Accelerating sparse autoencoder training via layer-wise transfer learning in large language models. In Pro- ceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 530-550, 2024.
Sariel Har-Peled. The johnson-lindenstrauss lemma. 2005. URL httpsst/saarilllppp..... teach/13/U5 wtt.-55.rrddddll.... wtt.o/alllleeee.........
Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama scope: Extracting millions of features from Ilama-3.1-8h with sparse autoencoders, 2024.
URL httttttss///iiiit org abss/410..2..... 20522
Stefan Hex and Trout Turn. Residual stream norms grow exponentially over the forward pass, 2023. URL http://////////.//o/oo/ooooo////
http://wwwroooooooo////////////a 2o0ostt///mm28mmmdd432222 residual-stream--ee norms orrerramnnnnnsss----eeeeeeennnno wrwprrreppooentttlllooeeeeteeore
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey.
Sparse autoencoders find highly interpretable features in language models. In The Twelfth
International Conference Learning Representations, 2023. 2023.
11

---

Published as a conference paper at COLM 2025
Paul C Kainen and Véra Kürková. Quasiorthogonal dimension. In Beyond traditional probabilistic data processing techniques: Interval, fuzzy etc. Methods and their applications, pp- 615-629. Springer, 2020.
Paul C. Kainen and Vêra Kūrková. Quasiorthogonal dimension of euclidean spaces. Ap- plied Mathematics Letters, 6(3):7-10, 1993. ISSN 0893-9659. doi: https:/ /osp:///oooorr////// wtt.cooocccc//cc com/sciinnnnne//atiill
1016/0893- (393)90039 URL http://ww...eeeeeeeeeeeeet/oeeee p11/88939655933900233..
Sham Kakade. perejoonon-indennee
lemma, 2010. URL htttt:://homess cattomes.cs.wahhhiiee washington. edu/~sham/couu cataaam/couusseeee seooursesssiiee ses/stat991 mult/. Stat 991: Multivariate Analysis, Dimensionality Re- duction, and Spectral Methods, Lecture 3.
Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu- Tong Lau, Eoin Farrell, Callum McDougall, Kola Ayonrinde, et al. Saebench: A compre- hensive benchmark for sparse autoencoders in language model interpretability. arxiv preprint Xiw::ii3:30000 00550300.033..... 2025.
Daniil Laptev, Nikita Balagansky, Yaroslav Aksenov, and Daniil Gavrilov. Analyze fea- ture flow to enhance interpretation and steering in language models. arXiv preprint arXiv:2502.03032, 2025.
Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. arXiv preprint arXtm:2502.04875 2025.
Beren Millidge Lee Sharkey, Dan Braun. Taking features out of superposition with sparse autoencoders, 2022.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arviv preprint arXit:2408.05144 2024.
Johnny Lin. Neuronpedia:
Interactive reference and tooling for analyzing neural networks, 2023. URL https://ww.neeuroppdia org. Software available from neuronpedia.org org.
W Johnson J Lindenstrauss and Johnson. Extensions of lipschitz maps into hilbert space. Contemp. Math, 26(189-206):2, 1984.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. International Conference on Learning Representations, ICLR 2014, 2013.
Gouki Minegishi, Hiroki Furuta, Yusuke Twasawa, and Yutaka Matsuo. Rethinking eval-
uation of sparse autoencoders through the representation of polysemous words. In The Thirleenth International Conference on Learning Representations, 2025. URL https:
//
Noa Nabeshima. Matryoshka sparse autoencoders. https: htpps:/www..dh wwwpwwwwwwwwww.....w/ hwtpssswwong.cooopoooooooeee com/posts/
2024.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, Singapore, December 2023. Association for Computational Linguistics. doi: (0.18665/v11 2023. blacbbblooobx/pp1... URL httph::t///ceee http://aaaaaannaaaaaaaaa hwwpolloooogooooooooooooo org/2023. 2023.
hwtp://.wrr........aoooooooooeaa blackboxnlp-1.22.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Causal Representation Learning Workshop at NewrIPS 2023, 2023. URL https: /ttsss///oeeeeiewew http///eneeiieeeee/oorroniiiiiia nett/forum/idd-ToPP0ggaaa
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
12

---

Published as a conference paper at COLM 2025
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXio preprint ar Xr:1:44044.1611 2024.
Lewis Smith. The 'strong' feature hypothesis could be wrong, 2024.
Demian Till. Do sparse autoencoders find "true features", 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Santosh S Vempala. The random projection method, volume 65. American Mathematical Soc., 2005.
A Limitations and Future Work
Theoretically, our analysis is limited to the linear representation hypothesis; extending AFA to more general superposition settings such as when ④(-) is two-layer neural network or other recent SAE variants such as Matryoshka SAEs (Nabeshima, 2024; Bart Bussmann, 2024) - remains open question. The JL lemma does not play a key role, but depending on the assumptions and claims, there are multiple versions of it that can be used (Har-Peled, 2005). It's also possible to explore tighter versions with more assumptions that we haven't t't used. Additionally, we do make a simplifying assumption by treating the embedding norm as its expectation; removing this assumption could lead to more rigorous but possibly looser bound.
Empirically, in Figure 5, identified two limitations in the design of top-AFA SAE: (1) the necessity of introducing a new hyperparameter AAFA instead of k and (2) the resulting high Eo values in this setting. Despite these limitations, our approach allows the model to adaptively determine the effective number of active features for each input, rather than enforcing fixed k across all inputs. This reflects the intuition that different examples may require different amounts of feature activation, which a fixed k cannot capture. Nevertheless, a deeper investigation into the necessity of the loss coefficient remains a direction for future work.
B Additional Terminology
Residual Stream The residual stream refers to the hidden embedding vectors that flows through transformer layers via residual connections, typically updated as z(f+1) (x) (2()))) + (x), where (-) is the I-th Transformer block (Vaswani et al., 2017) in language models.
Overcompleteness Dictionary is sometimes referred to an "overcomplete basis" due to the intuition of using more vectors than the dimension. However, since a basis cannot be overcomplete by definition, adopt the more precise term "dictionary" in this paper.
C Further Classification of LRH
is notable that the validity of the linear representation hypothesis (LRH) depends on factors such as the dictionary size h relative to the embedding dimension d, and the domain of inputs x over which applied. For instance, in the limit as h -> co, each column of W can memorize specific input, and the feature vector f can trivially become one-hot vector (Gao et al., 2024; Smith, 2024). A non-trivial hypothesis is when the dictionary size h is constrained. In practical implementation of SAEs, h 16 x d in GPT-2 Small (Bloom, 2024), whereas in Gemma Scope 2B, it ranges from to 28 times the embedding dimension (Lieberum et al., 2024).
13

---

Published as a conference paper at COLM 2025
Depending on the range of can classify LRH assumptions follows:"
Weak LRH: This version states that some features, though not necessarily all, can be represented as linear combination human-interpretable features. Weak LRH has been widely supported by research (Nanda al., 2023; Lieberum al., 2024).
Strong LRH:
This version states that all features can be represented as linear combination. However, this stronger claim has been widely disproved Smith (2024); Engels et al. (2024a).
In both cases, the feature vector f E R" assumed to result from a latent decomposition the embedding vector z E Rd, although the ground truth of such decompositions may not exist in the real-world language models. Under the SH, there are no structural assumptions on the encoder function @, except that the feature space is higher-dimensional than the input space (i.e., h > d). The LRH, in contrast, assumes that <D()) linear matrix W R but does not place any constraints on the relative dimensionality of the feature and input spaces.
Comparison between SH and LRH.
D Proof Tools
Theorem 3 (l1-22 Norm Inequality). For all h E IN, fe R
|| I Vhllfl2-
Proof. Define the vectors:
v= (1,1, E R².
Then,
h Σ | =11f
Applying the wwwpu//hhaaazzieuuiiii
||]]] = u |u/|a||v|22 S vhllffl2.
E
Corollary of Theorem 2
Corollary 1. Iff 10, then [z(x)]] approximates the number of activated features with the same error bound of the theorem.
Proof of Corollary. f E 10, |EE = E (1) = Killf> 0,
0
6Causal Linear Representation Hypothesis: Another interpretation of the linear representation hypothesis suggested by Park al. (2023), demonstrating that causal inner product can unify the representation in both embedding and unembedding, which satisfies Riesz isomorphism even through non-linear softmax unembedding layer. However, interpreting LRH to incorporate the causality diverges from widely investigated context of SAE which does not involve softmax-invariance and thus Riesz isomorphism is trivially achieved by one-layer weights vectors of autoencoder network.
14

---

Published as a conference paper at COLM 2025
F Derivation of Upper Bound based on the JL Lemma
We consider a high-dimensional space where I > d. Let { jr C R be an orthonormal
basis, satisfying |lb:ll2 1, and b; b, for + j. where L, E [k.. By the JL. lemma
there exists linear mapping (-) such that 2(1 - 8) _ [b; Y (b;)) - 2(1 + 6). Let ((;;) vi. Then, (b; (b, (b)))1 |lv; vyl12 1|v:ll ||v, 2v;-vv The JL lemma is typically proven using (b;) Va Ab, where A.j .V (0,1)), as shown in Kakade (2010). Taking expectation, we obtain [B[[[; = 1. If we assume ||v, = 1 as baseline, then
Vi.
2 2 28 2 - 2
| <
where e denotes the quasi-orthogonality of h vectors v; in d-dimensional space by Defini- tion 1. Based the JL Lemma constant (Theorem 1), can define:
20
EJL ==
G Unreliable Quasi-Orthogonalities of Pre-trained SAE Decoders
Since the JL lemma guarantees the existence of a linear transformation within a certain error range, the guasi-orthogonality of the ground-truth dictionary should not exceed this bound for a given hidden embedding. Our formalization based on the JL lemma answer the following question; why not simply use the quasi-orthogonality of the pre- trained decoder? While this seems reasonable, empirical evidence suggests otherwise. As in Leask et al. (2025), the maximum pairwise cosine similarity of SAE decoder weights tends to be very high, and thus exceeds the loose upper bound ějI. which can be easily calculated (Appendix F). This indicates that SAFs are unable to learn decoders that achieve gutsioortl decomposition to the extent permitted by the available dimensionality, suggesting the pre-tran trainrd decoders can be highly unreliable.
of
H Why eLBO Cannot Be a Loss Function
Ignoring the normalization effect from the denominator, the numerator of eLBO takes the
form || - ||E||3, which can be factorized as (l|z|/2 - [|f]2])(lll]2 + [|fll2). Minimizing the
second term (|z]|| + |[ffll) this factorization is not only meaningless to reduce, but also introduces vulnerability to shrinkage effects (Rajamanoharan et al., 2024).
I Detailed Experimental Results
We performed all experiments single NVIDIA A100 GPU. Following Bussmann al. (2024), each training iteration used 4,096 tokens from OpenWebText. We trained the models for 20k iterations (81,920,900 tokens in total). Table 1, 2, and 3 were measured using the last 100 batches.
J Evaluating the Designed Sparse Autoencoder: Top-AFA with eLBO
In Figure 4, one may ask whether the multi-modal or long-tailed shape of the sLBO dis- tribution indicates not a more granular evaluation, but rather noisier metric. While it is true that eLBO is lower bound theoretically susceptible to noise induced by superposition interference, such questions can partially addressed by evaluating the proposed SAE using our proposed metric. Figure 6 highlights two observations: (1) ELBO can be better minimized to lower values with Top-AFA, and (2) existing activation functions exhibit significantly poorer eLBO distributions, not simply attributable to noise. Therefore, eLBO reveals the presence theoretically grounded objective that current SAE training methods have neglected.
15

---

Published as a conference paper at COLM 2025
Layer
Activation
AAFA
Sparsity (L0)
Normalized MSE #o
Top-AFA
1/128
331.62
0.127769 + 0.002493
Top-AFA
1/64
1118.46
0.034370 = 0.003631
Top-AFA
1/32
2037.74
0.000179 1: 0.000044
Top-AFA
1/24
2133.73
0.000179 + 0.000047
Top-AFA
1/16
2344.23
0.000176 + 0,000041
Top-AFA
1/8
2862.85
0.000231 + 0.000098
Batch Top-k
32
32.00
0.061787 + 0.000583
Batch Top-k
64
64.00
0.048710 + 0.000464
Batch Top-k
128
128.00
0.036358 + 0.000348
Batch Top-k
256
256.00
0.023379 1: 0.000231
Batch Top-k
512
512.00
0.008044 1: 0.000121
Batch Top-k
1024
1024.00
0.000202 + 0,000021
Batch Top-k
2048
2048.00
0.000207 = 0,000019
Batch Top-k
4096
2968.77
0.000221 + 0.000031
Batch Top-k
8192
2949.76
0.000224 + 0.000019
Top-k
32
31.98
0.063715 + 0.000661
Top-k
64
63.94
0.050356 1: 0.000508
Top-k
128
127.84
0.038570 1: 0.000388
Top-k
256
255.99
0.026100 =: 0.000253
Top-k
512
512.00
0.008793 = 0.000075
Top-k
1024
1023.46
0.000199 + 0.000020
Top-k
2048
2044.35
0.000207 + 0.000018
Top-k
4096
2970.86
0,000219 + 0,000021
Top-k
8192
2949.76
0.000224 + 0.000019
Table 1: Detailed results for layer in Figure 5.
Layer
Activation
K AAFA
Sparsity (LO)
Normalized MSE to
Top-AFA
1/128
1533.98
0.004476 1 0.000346
7
Top-AFA
1/64
1930.60
0.000221 1: 0.000028
Top-AFA
1/32
2066.28
0.000199
0.000042
7
Top-AFA
1/24
2180.70
TOP-AFA
1/16
2382.96
0.000193 + 0.000049
Top-AFA
1/8
3016.48
0.000239 + 0.000082
Batch Top-k
32
32.00
0.071729 + 0.000614
Batch Top-k
64
64.00
0.056417 + 0.000491
Batch Top-k
128
128.00
0.042214 +: 0.000361
Batch Top-k
256
256.00
0.026796 + 0.000234
Batch Top-k
512
512.00
0.009608 + 0.000134
Batch Top-k
1024
1024.00
0.000215 1 0.000016
Batch Top-k
2048
2048.00
0.000222 1: 0.000015
Batch Top-k
4096
2974.06
0.000236 + 0.000022
7
Batch Top-k
8192
2956.95
0.000239 + 0,000020
Top-k
32
32.00
0.073273 + 0.000660
Top-k
64
63.88
0.057877 + 0.000517
Top-k
128
127.81
0.044168 + 0.000393
Top-k
256
255.79
0.029050 1: 0.000246
Top-k
512
512.00
0.009737 1: 0.000073
Top-k
1024
1023.45
0.000214 + 0.000024
Top-k
2048
2045,20
0.000221 + 0.000026
Top-k
4096
2975.86
0.000239 + 0.000026
Top-k
8192
2956.95
0.000239 + 0.000020
Table 2: Detailed results for layer 7 in Figure 5.
16

---

Published as a conference paper at COLM 2025
Layer
Activation
AAFA
Sparsity (LO)
Normalized MSE
Top-AFA
1/128
1577.59
0.003457 0.00008
Top-AFA
1/64
1973.62
0.000230 = 0.000026
Top-AFA
1/32
2103.63
0.000203 1 0.000050
Top-AFA
1/24
2195.08
0.000197 0.000047
Top-AFA
1/16
2420.21
0 = 0,000099
8
Top-AFA
1/8
3019.70
0.000223 0.000042
Batch Top-k
32
32.00
0.079982 + 0.000662
8
Batch Top-k
64
64.00
0.063075 + 0.000524
Batch Top-k
128
128.00
0.047696 + 0.000392
Batch Top-k
256
256.00
0.029998 1: 0.000254
Batch Top-k
512
512.00
0.009727 1: 0.000151
Batch Top-k
1024
1024.00
0.000222 + 0,000044
Batch Top-k
2048
2048.00
0.000236 = 0,000057
Batch Top-k
4096
2950.05
0.000243 + 0.000023
Batch Top-k
8192
2951.90
0.000247 + 0.000027
8
Top-k
32
31.99
0.081479 + 0.000693
8
Top-k
64
63.83
0.064612 1: 0.000539
Top-k
128
127.84
0.049319 1: 0.000415
8
Top-k
256
256.00
0.032053 = 0.000268
Top-k
512
512.00
0.009610 = 0.000073
8
Top-k
1024
1023.30
0.000220 + 0.000027
Top-k
2048
2044.45
0.000228 + 0.000025
Top-k
4096
2960.06
0,000247 + 0,000029
Top-k
8192
2951.90
0.000247 + 0.000027
Table 3: Detailed results for layer 8 in Figure 5.
20-7
Layer
1e-5
16-5
Layer
1e-3
1--6
Layer
16-5
2.0
3.0
Normaltcc MBE
25
VIL
EERO
0.5
1.0
0.4
-05
9.5
0.0
tape
Bates Topue
Sp-aft
Tapt
Tapk
Batth Topok
Batch
(b)
Figure 6: Violin Plots for Comparing Different Activation Functions (Lower values are better). Distributions are computed over 300 input sequences of length 128 from the OpenWebText dataset. The region of each violin captures 99% of the distribution the influence of extreme outliers. The hyperparameter used for each layer corresponds to those that achieved the best reconstruction (the lowest NMSE) on the last batch of 20k training iterations. Interestingly, the eLBO of Top-AFA can converge to low value through training, while other activation functions, Top-k and Batch Top-k, fail to achieve low eLBO.
mitigate
K Complexity Analysis
Table 4 shows that Top-AFA achieves similar or better NMSE compared to other methods with training time that remains in a comparable range. To support our claim and explain the empirical results, also conducted the complexity analysis. The time complexity of the encoding and decoding stages is summarized in Table 5. The key difference lies the top-k selection mechanism, where sorting dominates, as detailed in Table 6. Since k h the
17

---

Published as a conference paper at COLM 2025
Layer
Activation
L0
NMSE
Time (min)
Top-AFA
2037.29
0.000164
69.40
6
Batch Top-k
1024
0.000165
136.63
6
Top-k
1023.49
0.000174
69.98
7
Top-AFA
1930.10
0.000195
139.75
Batch Top-k
2957.23
0.000181
138.88
7
Top-k
2957.23
0.000181
70.61
8
Top-AFA 2195.77
0.000166
142.04
Batch Top-k
1024
0.000187
74.45
8
Top-k 1023.32
0.000177
71.90
Table 4: Layer-wise training time and reconstruction performance (NMSE), including LO with the lowest NMSE on the last batch of 20k iterations. These results demonstrate that Top-AFA achieves training times comparable to those of other methods.
Stage
Operation
Time Complexity
Encoding x Wene
(ddh)
Decoding
f Wdec
O (Bkd)
Table 5: Time complexity for encoding and decoding stages.
leading term for Top-AFA is O(hdd BBh It,, which not higher than other activation methods.
The memory complexity for each method compared in Table 7, showing that all methods exhibit comparable complexity.
18

---

Published as a conference paper at COLM 2025
Method
Top-k Selection Method
Time Complexity
Top-k Batch Top-k Top-AFA
Sort per input (B samples) Sort across batch (Bl entries)
O log h) 0 Bh log( Bh)) (Bh log h)
Sort cumulative sum + argmin per input
Table 6: Time complexity of top-k selection mechanisms.
Variable
Shape
Memory
(Batch) Top-k
Top-AFA
Input
B x d
O(Bd)
Hidden activations f Reconstructed output
B x h
O(Bh)
B d
O(Bd)
Encoder weights Wene
d X h
O(dh)
Decoder weights Waec
h X
O(dh)
Top-k indices / mask
B x h
O(Bh)
Sort buffer
B x h
O(h))
Table 7: Memory complexity comparison across methods.
19


Reviews:
[{"review_id": "0EKxnuwZA3", "reviewer": "Reviewer_QxGv", "summary": "The authors introduce Approximate Feature Activation (AFA) to evaluate and design sparse autoencoders (SAEs) by connecting input embeddings to feature activations through quasi-orthogonality constraints. Furthermore, they propose several evaluation metrics, such as  ZF plot for visualization and the ε_LBO metric for evaluation of SAEs and a top-AFA SAE architecture that adaptively selects sparsity levels without hyperparameter tuning.\n\nThe authors provide solid theoretical foundations with formal proofs and empirical validation for AFA, however, the dense mathematical exposition may hinder accessibility for a general audience. The connection between theoretical concepts and practical applications could be clearer. The adaptive sparsity selection approach is novel, though the core ideas build incrementally on existing work. While the theoretical insights are valuable for understanding SAE behavior, the practical impact appears limited due to modest performance gains and additional complexity in implementation, especially when recent work has shown limited gains SAEs provide as compared to simple baselines for practical tasks. The paper can also benefit from significant restructuring of the main content, such as describing the intuitive interpretation of the top-AFA function and the ZF plot, along with Top-AFA algorithm in the main article and discussions of the strengths and limitations of the proposed architecture and the AFA framework to be of broader interest to the COLM  community.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors provide a theoretical foundation as a connection between input embeddings and feature activations in SAEs and provide metrics that may be valuable for evaluating SAE performance beyond traditional reconstruction-based metrics.\n\n2. The top-AFA approach provides theoretical justification for adaptive sparsity selection, addressing a known limitation of SAEs with top-k based activations.\n\n3. The work includes a thorough empirical comparison with standard SAEs.", "reasons_to_reject": "1. While the AFA based approach provides an alternative to standard top-k activation based SAEs, the performance improvements are marginal and inconsistent across layers, raising questions about practical significance, particularly since the method only outperforms baselines on specific layers (6 and 8) but not others, and for the specific choice of loss hyperparameter of 1/32\n\n2. The work would benefit from some comparison of the top-AFA SAE features with top-k or batch top-k SAE features, and their effectiveness on downstream tasks such as monitoring or controlling safety related behaviors.\n\n3. The addition of an additional AFA loss term might shift the optima of the loss objective from proper reconstruction, so it might be worth exploring the effect of the additional loss term. This might also point to why top-AFA might not outperform its top-K counterparts except for specific hyperparameter settings.", "questions_to_authors": "1. The reviewer’s main concern is with the introduced loss term. The authors claim to eliminate the need for tuning k, but introduce λ in the loss, which appears equally critical. This seems to lessen the strength of the main claim about reducing hyperparameter sensitivity. More clarity on the justification of the choice of the additional loss term would also be helpful.\n\n2. The top-AFA algorithm involves sorting and cumulative sum operations for each input. How does this computational overhead compare to simple top-k selection? Is the improved performance worth the added complexity?\n\n3. How statistically significant are the performance improvements? What are the confidence intervals, and how many random seeds were used? This might help provide clarity on the improvement claims.\n\n4. Does the proposed approach improve the interpretability of learned features compared to standard top-k/batch top-k SAEs? Do the adaptively selected features provide better semantic coherence?\n\n5. What are the training time and memory requirements compared to baseline methods? \n\n6. How do the top-AFA SAEs perform on downstream tasks that measure feature quality, such as those in SAE Bench?", "extra_scores": {}}, {"review_id": "80q8FnI4xj", "reviewer": "Reviewer_b4e1", "summary": "The paper analyzes Sparse Autoencoders (SAEs) with top-k style activation functions theoretically and establishes new relationships between LLM hidden embeddings and SAE feature vectors. With this, the authors \"measure\" how well the SAE feature vectors are activated for a given input via ZF plots and with a new metric called Approximate Feature Activation (AFA).", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper makes a number of contributions to theoretical fundamentals of mechanistic interpretability via SAEs. While I am not very familiar with the literature, the findings looked correct to me.\n\n- The proposed method, top-AFA SAE, eliminates the need to tune k in top-k style activations. \n\n- Empirical results show that top-AFA SAE outperforms top-k and batch top-k SAEs.", "reasons_to_reject": "I can't find a strong weakness but I am also not an expert in this topic.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y9uwOknDny", "reviewer": "Reviewer_Ji2v", "summary": "This paper introduces a method called approximate feature activation (AFA) that is shown theoretically to approximate the magnitude of the (unknown) ground-truth feature vector, under certain common assumptions (which might be practical, though they are probably too strong in reality). AFA gives rise to a new metric for SAE and a nice visualization; and a new flavor of SAE, which seems promising in a very simple experiment.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written, given the theoretical subject matter. Although I'm not an expert in this area, I mostly understood what was going on, with a bit of effort. \n- The paper provides new methodology and metrics that should help the interpretability community understand SAEs and how well they align with underlying features. It seems plausible to me that this analysis could help others with their ongoing research, (note that I'm an outsider to this field, so this is just an educated guess on my part, based on the context in the paper).", "reasons_to_reject": "- I am a bit worried that this community and its ecosystem (nicely outlined in the background section) relies on models and datasets that have accumulated around increasingly out of date models. I don't think this is a major problem for a mostly theoretical paper like this, that is really exploring a new idea in the space. But it does leave open the natural question of what happens when one applies these ideas to a more current model. I appreciate that this is difficult to do for many researchers, and also that this particular paper needs to work with what is available.\n- I also suspect that the strict hypotheses that the method relies on are too strong for real world data (e.g. that \"ground truth\" features, even if we could completely identify them, are \"nearly orthogonal\"). I think that's generally ok so long as it's acknowledged that mathematical niceties don't often hold up in their entirety in the real world. All models are wrong, some models are useful...", "questions_to_authors": "Minor presentation comments:\n- For a paper that is so crisp and economical with its notation, built from the ground up, it is surprisingly free with obscure jargon that is specific to this subfield e.g. \"meaningfully aligned\" (line 37) and \"residual stream\" (line 79).\n- Unlike the rest of the paper, the abstract is a big mess; more of a story about how the paper is structured than what the paper delivered. Weirdly, the arbitrariness of $k$ is introduced in the first sentence, then completely ignored for the remainder of the abstract until almost the end, with nary a bit of logical connection in between. I thought this paper would be an impossible mess based on the abstract but it was (mostly) pretty clear. Please rewrite the abstract for clarity.", "extra_scores": {}}, {"review_id": "y6FS8B5Dcm", "reviewer": "Reviewer_8rCC", "summary": "The authors use the Liner Representation Hypothesis and the Superposition Hypothesis to make theoretical predictions related to the ortogonality of SAE features as well as predictions about the relation between the norms of the inputs and the norms of the activations. \nWith these theoretical insights they propose a new loss term and a new activation function, which can get competitive reconstructions when $L_0>10^3$ .", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors create a nice framing of sparse autoencoders using the Linear Representation Hypothesis and the Superposition Hypothesis. The tools they take from high dimensional analysis allow them to define error bounds on the reconstruction of the sparse autoencoder.", "reasons_to_reject": "The proposed architecture seems to fail to be relevant on the exact property that makes SAE interesting: find sparse interpretable features. The only reagion where top-AFA sparse autoencoders are competitive is at very high $L_0$, places where the learned features are probably as interpretable as neurons. \nThe claim that k is arbitrarily chosen is valid, not being obvious how to define the best K à priori, but this hyperparameter is substituted with a new one, $\\lambda_{AFA}$, which the training process is very sensitive to as claimed by the authors.", "questions_to_authors": "Why should we care about the relation between f and z? The point of sparse autoencoders seems to be that that the reconstrution is good. Why should the norms have to match?\n\nThe claim is that $\\epsilon LBO$ is more informative than just evaluating the MSE because the distribution has a more complex \"shape\". Could it just mean that the metric is more noisy, not informative or that is is evidence that the LRH and SH don't hold that strongly?\n\nSparsity greater than $10^3$ is probably useless and the learned features are probably not interpretable or at least as interpretable as neurons. Why use top AFA at all?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes AFA, a method to approximate the magnitude of the ground truth sparse feature vector. They use this proposed method for both designing top-k SAEs with adaptively selected k and for evaluating SAE reconstruction performance. The contributions are a mix of theory, describing bounds on reconstruction, and empirical/methods work on applying AFA.

Pros:
- Nice formal treatment of SAEs in the context of the assumptions behind them like the LRH. Avoids pitfalls around informal treatment of the LRH by using a precise definition in line with the prevalent meaning.
- Potentially useful metrics introduced, good empirical confirmation.
- Theoretically justifies an adaptive hyperparameter selection method for SAEs.

Cons: 
- Some assumptions may be too strong (near orthogonality) and the setting is small models, but both are common in SAE work.
- Effects may not be practically substantial (though they are confirmed to be statistically significant)
Decision: Accept