AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features
Authors: Xudong Zhu
Mohammad Mahdi Khalili
Zhihui Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Sparse Autoencoder
Mechanistic Interpretability
Primary area: interpretability and explainable AI

Abstract: Sparse autoencoders (SAEs) have emerged as powerful techniques for interpretability of large language models (LLMs), aiming to decompose hidden states into meaningful semantic features. While several SAE variants have been proposed, there remains no principled framework to derive SAEs from the original dictionary learning formulation. In this work, we introduce such a framework by unrolling the proximal gradient method for sparse coding. We show that a single-step update naturally recovers common SAE variants, including ReLU, JumpReLU, and TopK. Through this lens, we reveal a fundamental limitation of existing SAEs: their sparsity-inducing regularizers enforce non-negativity, preventing a single feature from representing bidirectional concepts (e.g., male vs. female). This structural constraint fragments semantic axes into separate, redundant features, limiting representational completeness. To address this issue, we propose AbsTopK SAE, a new variant derived from the $\ell_0$ sparsity constraint that applies hard thresholding over the largest-magnitude activations. By preserving both positive and negative activations, AbsTopK uncovers richer, bidirectional conceptual representations. Comprehensive experiments across multiple LLMs and seven probing and steering tasks show that AbsTopK improves reconstruction fidelity, enhances interpretability, and enables single features to encode contrasting concepts. Remarkably, AbsTopK matches or even surpasses the Difference-in-Mean method—a supervised approach that requires labeled data for each concept and has been shown in prior work to outperform SAEs.

Published as a conference paper at ICLR 2026
ABSTOPK: RETHINKING SPARSE AUTOENCODERS FOR BIDIRECTIONAL FEATURES
Xudong Zhu, Mohammad Mahdi Khalili, Zhihui Zhu Department of Computer Science & Engineering The Ohio State University (zhu. 3944, khalili.17 17,
zhu. 3 440 2hu.3440|@osu. 4440)osu..ed
ABSTRACT
Sparse autoencoders (SAEs) have emerged as powerful techniques for inter- pretability of large language models (LLMs), aiming decompose hidden states into meaningful semantic features. While several SAE variants have been pro- posed, there remains no principled framework to derive SAEs from the original dictionary learning formulation. this work, we introduce such a framework by unrolling the proximal gradient method for sparse coding. We show that a single-step update naturally recovers common SAE variants, including ReLU, JumpReLU, and TopK. Through this lens, reveal a fundamental limitation of existing SAEs: their sparsity-inuucins regularizers enforce non-negativity, pre- venting single feature from representing bidirectional concepts (e.g.. male vs. female). This structural constraint fragments semantic axes into separate, redun- dant features, limiting representational completeness. To address this issue, we propose AbsTopK SAE, new variant derived from the ln sparsity constraint that applies hard thresholding over the fargest-magnitude activations. By preserving both positive and negative activations, AbsTopK uncovers richer, bidirectional conceptual representations. Comprehensive experiments across multiple LLMs and seven probing and steering tasks show that AbsTopK improves reconstruction fidelity, enhances interpretability, and enables single features to encode contrast- ing concepts. Remarkably, AbsTopK matches Mean method-a supervised approach that requires labeled data for each concept and has been shown in prior work outperform SAFs.
surpasses the Difference in-
INTRODUCTION
The pursuit of interpretability has become central objective in modern machine learning, as it is essential for the assurance, debugging, and fine-grained control large language models (LLMs) (Marks et al., 2025; Park et al., 2023; Luo et al., 2024: Arora et al., 2018). Within this domain, sparse dictionary learning methods (Poggio & Serre, 2006; Fel et al., 2023), and specifically sparse autoencoders (SAFs), have re-emerged the latent concepts a model may employ in its predictions (Hindupur et al., 2025; Bussmann et al., 2024; Rajamanoharan et al., 2025; Gao et al., 2025).
prominent methodology for systematically enumerating
An SAE decomposes a model's hidden representations into an overcomplete basis of latent features (Elhage et al., 2022: Thasarathan al., 2025), which ideally correspond abstract, data-driven concepts whose linear superposition reconstructs the original activation vector (Higgins et al., 2017; Fel, 2025). Empirical evidence indicates that SAE latents capture semantically coherent features across diverse domains. In LLMs, these features exhibit selectivity for specific entities (c.g., Golden Gate Bridge), linguistic behaviors (e.g., sycophantic phrasing), and symbolie systems (e.g., Hebrew script) (Templeton et al., 2024; Csordás et al., 2024; Durmus et al., 2024). Similarly, in vision models, they respond to distinct objects (e.g., barbers, dog shadows) and complex scene properties (e.g.,
separation, facial detection in crowds) (Fel, 2024; Thasarathan et al., 2025). In protein models, they have been shown to correlate with functional elements such as binding sites and structural motifs (Garcia & Ansuini, 2025; Adams al.. 2025). The discovery of such interpretable, semantically grounded features suggests a natural avenue for steering models: by amplifying, suppressing, combining specific latents, one intervene modulate downstream

---

Published as a conference paper at ICLR 2026
SAE feature 15741: Positive activation on man The who had traveled across...
SAE feature 15741: Negative activation woman The who had traveled ..
Despite outaide, the man remained calm and focused on her work... In the bustling market, man stood by the flower stall...
Despite the storm outside, woman remained calm focused on her work...
In the bustling market, a woman stood by the flower stall
Figure I: AbsTopK enables single latent features to encode opposing concepts by leveraging both positive and negative activations. To test this, we generated controlled sentence pairs with only one differing token (man vs. woman). The shown feature activates positively for man and negatively for woman, demonstrating bidirectional encoding. Unlike conventional SAEs, which are restricted by non-negativity constraint, AbsTopK more compactly captures opposing semantics within a single dimension, yielding richer and more coherent representations.
behavior, which is a principal motivation for research into SAEs (Gao et al., 2025; Bricken et al., 2023; Kantamneni et al., 2025). This control is predicated on the assumption that the concepts identified by SAFs faithfully correspond to the features underlying a model's predictions (Arditi al., 2024: Uppaal et al., 2024; Engels et al., 2025).
However, recent studies suggest that simpler supervised techniques such as Difference-in-Mean (DiM) can outperform SAEs on practical steering benchmarks and tasks (Arditi et al., 2024; Wu et al., 2025). Unlike SAEs, which are unsupervised and can simultancously identify multiple latent features, DiM requires labeled data and is typically limited to extracting a single vector for pre- specified concept. Nevertheless, these findings raise questions regarding the degree to which SAEs recover a model's internal features. The fact that comparatively simple baselines can rival or even surpass SAFs on downstream control tasks suggests that the features identified by SAEs may only partially align with the model's underlying neural representations, thereby casting doubt on their fidelity as faithful explanatory tools.
We posit that one source of this misalignment lies in a structural limitation of SAEs recently pro- posed for studying LLMM-including the vanilla version with ReLU (Cunningham et al., 2023), the JumpReLU variant (Rajamanoharan et al., 2025), and the TopK variant (Gao et al., 2025): their systematic neglect of negative activations, despite evidence that many meaningful directions in rep- resentation space are inherently bidirectional (Mao et al., 2022). The linear representation hypoth- esis (Mikolov et al., 2013) suggests that a model's internal states can be approximated linear combinations of semantic vectors, where conceptual transformations correspond to both positive and negative displacements along these vector axes (Arora et al., 2018; Uppaal al., 2024; Luo et al., 2024). The DiM approach builds on this assumption, requiring labeled datasets that capture both sides of concept, with positive and negative examples defining bidirectional semantic axis. Classic word analogies, such as the vector operation Oxing Uman Uwaman R Qqueen (Pennington et al., 2014), illustrate how semantic differences are encoded as generalizable vector offsets. Never- theless, by enforcing non-negativity or retaining only the TopK activations (Bussmann et al., 2024; Gao et al., 2025), conventional SAES either fragment such contrastive concepts into separate, uni- directional bases (e.g., "male" and "female") or discard one direction of the semantic axis entirely. This not only undermines the representational capacity of SAEs but also limits their usefulness for controlled interventions, where traversing both directions of semantic axis is often essential. This raises the following questions: is the use of nonnegative activations truly essential for the success of SAEs, does it instead constrain their ability to capture richer representations? More con- cretely, can SAEs be improved by allowing negative activations, thereby enabling the discovery of bidirectional concepts?
Contributions.
In this work, address these questions by (i) introducing unified framework for designing SAES, (ii) proposing a new variant, AbsTopK SAE, and (iit) conducting compre- hensive experiments across four LLMS and seven probing and steering tasks to demonstrate that
2

---

Published as a conference paper at ICLR 2026
allowing negative activations further enhances SAEs, yielding improved reconstruction fidelity and greater interpretability.
Our contributions are summarized follows:
A Unified Framework for Designing SAEs. We introduce a principled framework for designing SAFs by unrolling the proximal gradient method for sparse coding with sparsity regular- izers. A single-step update naturally induces common SAE variants, including ReLU, MOLU,JJmpp JumpReLU, and TopK (Templeton et al., 2024; Gao et al., 2025; Rajamanoharan et al., 2025). This framework provides rigorous tool for analyzing their implicit regularizers and identifying shared limitations.
Absolute TopK (AbsTopp) for Learning Bidirectional Features Building on this framework, we propose new SAE variant, termed absolute TopK (AbsTopK) derived from the vanilla spar- sity constraint (Ln norm) without a non-negative constraint, which results in a hard-thresholding operator that selects the largest-magniuue activations. By preserving both positive and nega- tive activations, AbsTopk SAE allows a single feature to capture opposing concepts (Figure 1), thereby uncovering richer bidirectional representations.
Comprehensive Empirical Validation. We conduct comprehensive empirical evaluation across four LLMs, comparing the proposed AbsTopK SAE with TopK and JumpReLU SAEs on a suite of seven probing and steering tasks, along with three unsupervised metries. The results demonstrate that AbsTopp outperforms Topk and JumpRel.U SAFs, producing representations with higher fidelity and interpretability. Additionally, case study illustrates that AbsTopK can encode a bidi- rectional semantic axis within a single latent feature, effectively capturing contrasting concepts. Notably, AbsTopK achieves performance comparable to-or even exceeding- the Difference-in- Mean method, which relies on labeled data and has been shown in prior work to outperform SAEs.
2 FROM PROXIMAL INTERPRETATIONS OF SAES TO ABSTOPK
2.1 PRELIMINARIES
We denote vectors by lowercase bold letters (e.g., x) and matrices by uppercase bold letters (e.g., X). With an imput sequence of N tokens. X = (x1..... ZN J. where each y denotes the embed- ding of the j-th token, the LLM can be viewed as function f RdxN + R² VVN where V is the vocabulary size and f(X) gives the output logits for all tokens in the sequence. For our purposes, we abstract away the internal details of f and instead study the representations in the hidden lay- ers. Consider interventions at layer in the residual stream. Supposing that the model comprises layers, then f can be decomposed as
((x) (bet1:((0mee())
where (Xx denotes the representation after the first layers and Φ1+1:L represents the remaining computation from layer l to L. We denote by x the embedding of the residual stream at the f- th layer corresponding to the j-th token of the input sequence X. In the following presentation,
when the context is clear, omit the superscript (f) and the subscript (token index j) for notational simplicity, and denote the hidden embedding of a token in a given layer by x.
The linear representation hypothesis (Park al., 2023) assumes that the hidden representation can be expressed as a linear superposition of latent concepts:
Σ hp + residual,
x =
p=1
where ( are referred to as concept directions or feature vectors, such as gender or senti- ment, {ap] are the corresponding coefficients, and residual term captures approximation error as well as context-specific variation that is not explained by the selected concepts. Since a particu- lar token token-although it encodes information from previous tokens in the context-typicall only a small subset of concepts or features, its representation is expected to be sparse; that is, most of the coefficients ap are zero, resulting in sparse linear representation, often simply referred to as a sparse representation. Importantly, the coefficients ap are not required to be non-negate neg-negative. In fact, for binary concepts, the sign of coefficient is semantically meaningful, indicating opposite
contains
3

---

Published as a conference paper at ICLR 2026
directions; for example, it distinguishes whether contextually appropriate token should be "king" or "queen" (when the context involves a monarch) (Park et al., 2023).
We note that, although (2) is written in terms of one-dimensional concept directions hpp some cepts may in practice be better modeled as low-dimensional feature subspaces. As discussed in (En- gels al., 2025), such multi-dimensional features can still be captured by SAEs, and our use of the linear representation hypothesis is compatible with this view.
To find these concept directions feature vectors, supervised approaches such the Difference-in- Mean (DiM) method construct labeled datasets for each target attribute. While effective for isolating specific concepts, these methods are inherently limited predefined features and do scale the large number of latent dimensions present in LLM representations. In contrast, dictionary learning provides an unsupervised and scalable alternative: it can simultaneously recover more complete dictionary that approximates the underlying concept directions, uncovering a richer and more com- prehensive set of latent features than DiM, which is typically restricted single concept vector. Consequently, while DiM may achieve stronger control on specific concept (Wu et al., 2025), dictionary-learning methods have gained popularity due to their ability to uncover a richer, more comprehensive set of latent features.
2.2 DICTIONARY LEARNING AND THE PROXIMAL PERSPECTIVE ON SPARSE AUTOENCODERS
In a nutshell, dictionary learning (Olshausen & Field, 1996) seeks to construct a dictionary D con- sisting of basis vectors (di....; dp). which are also called atoms, such that it can approximately provide sparse linear combination for all token embeddings æ from the same layer. Since the total number of concept vectors p' unknown, P is typically set relatively large value ensure that as many concepts possible can be learned. This typically requires solving training problem of form (Mairal et al., 2011)
min DEREXP ..ERR
Ex
min
| (Dz + b)ll2 +AR(Z)
SER"
x(*)
where R(z) is sparsity-inducing regularizer, À controls the trade-off between reconstruction fidelity and sparsity, is an additional bias vector. In classical dictionary learning, the data is often preprocessed to have zero global mean, the bias term is not used. Alternatively, the bias term can
be incorporated into the dictionary as Dz + b [D
b]
In this work, however, we explicitly
include align with the structure of commonly used SAES which will be described later.
The main challenge in solving the problem (3) lies in jointly estimating both the dictionary (D. b) and the sparse coefficients z. When one of these variables is fixed, optimizing over the other be- comes relatively easier, though still nontrivial in practice. In particular, given a dictionary D and bias b, the problem reduces to finding sparse approximation of æ, a step commonly referred to sparse coding. An efficient method for solving this problem is the proximal gradient method (Parikh et al., 2014; Silva & Rodriguez, 2020), which is especially suitable when the regularizer R(z) is non-differentiable, such as the & norm used in Lasso (Tibshirani, 1996) Eg norm that directly enforce sparsity (Foucart, 2011; Bao et al., 2014; Rajamanoharan et al., 2025).
Proximal gradient methods induce encoders is defined by (Parikh et al., 2014)
For function Rd R. its proximal operator
1 11 u/l2 + r(v).
prox, (w) arg min vERd
TTii observation has motivated alternating minimization methods such MOD (Cai al., 2016) and K-SVD (Aharon al., 2006).
4

---

Published as a conference paper at ICLR 2026
Now starting from initialization 20)) the proximal gradient method for optimizing in (3) per- forms iterative updates of the form
x (1+1) 1) pARS (x(1) - (V prOX,AR (z -p (DDT(DzZ)) + bb- bb-z))), (4)
where > 0 is the step size. This perspective naturally leads to unrolled networks (Gregor & LeCun, 2010; Chen et al., 2022). where each proximal gradient step can be interpreted as a layer in a neural network that iteratively refines the latent code z while enforcing sparsity (Daubechies et al., 2004). In particular, with =0)) = 0 and 1, the first update becomes
(5) Since single proximal gradient step yields only approximate solution, inspired by prior work unrolled networks, we replace the fixed parameters D and with learnable counterparts: a trainable weight matrix W in place of D, and learnable bias vector b& in place of D b, thereby yielding a more accurate approximation to the sparse coding solution. Then the update (5) becomes
(1) prox xn (D -D==-D -D))
z (2) proX ( be),
which resembles encoder. The following result shows that certain regularizers give rise proxi- mal operators commonly used SAES.
Lemma 1. Denote by ReLUx, JumpRel., TopK, as the following operators:
(0
Us < 0
(ReLUs(w)), = max/ue - X, 0, (JumpReLUg(u)): =
M4. Mi 0,
(7)
(TopK, { mmax (us; 0}, iE Tk(u),
i i& Tk(
where Tε(u) denotes the set of indices corresponding to the k largest entries? u. u. Here and are hoper-parameters subject to design choices.
They can be induced by the following choices of sparse regularizers:
Case I: R(z) || + ((=20) then prOXAR = ReLUxi
Case II: R(2) ||2||o + <(=20) (z), then prOx ar JumpReLU V2Xi
Case III: R(z) [[Ilellakk (2), then prox ( = (TopK
Here LA is the indicator function of set A, ie., LA(2) Of E A and (A(z) +000 if & A., and 2 0 means 2 0 for all i.
detailed proof is provided in the Appendix Note that ReLU,, reduces to the standard ReLU when À + (. The operators ReLU, and JumpReLU, are commonly referred to as soft thresh- olding and hard thresholding (except restricted to the nonnegative orthant), respectively, and image processing, where they are used to enforce sparsity (Foucart, 2011 1; Acuña et al., 2020). The TopK operator (7) follows the original formulation Gao et al. (2025), which includes an additional ReLU ensure nonnegative activations. Nevertheless, if u has at least & nonnegative entries-which which is typically the case since & much smaller than the ambient dimension s theh the ReLU inside TopK is redundant, and the operator simply retains the largest & entries while setting the rest to zero. This phenomenon is also observed in Gao et al. (2025), where the training curves were found be indistinguishable. In a nutshell, Lemma establishes that several prevalent non- linearities in SAEs, including ReLU, JumpReLU, and TopK, are precisely the proximal operators of sparse- searse-enforcing regularizers.
signal
One-step proximal gradient method leads to Sparse Autoencoders. With Lemma 1, applying one-step proximal gradient method to the sparse coding problem naturally leads SAEs. Specifi- cally, (6) defines a mapping from an input representation æ to a sparse code z, which is then decoded to reconstruct the original representation, formally given by
encoder: z = prox WT + ba), decoder: T Dz + b.
(8)
In case k largest components not uniquely defined, one can choose among them-for example, by selecting the components with the smallest indices-to ensure exactly it entries are kept.
5

---

Published as a conference paper at ICLR 2026
Choosing different regularizers R as in Lemma yields different variants of SAEs, including the vanilla version with ReLU (Cunningham et al., 2023), a version with JumpReLU (Rajamanoharan et al., 2025), and one with TopK (Gao al., 2025). For simplicity, we refer to these ReLU SAE, JumpRel.U SAE, and Topk SAF, respectively. This observation situates diverse SAE architectures within unified proximal framework, where each activation function is interpreted the proximal map for a specific regularizer R. Consequently, design choices for SAEs correspond directly to the selection of an implicit sparsity-inducing penalty, which in turn provides principled basis for comparing and extending these models. For instance, our analysis in Lemma 1 shows that ReLU SAE corresponds to the & norm regularizer (a convex relaxation of sparsity) with weight + O, whereas JumpReL.U and TopK correspond directly to the sparsity-inducing fn norm regularizers with a non-vanishing A, thereby enforeing stronger sparsity. This provides a principled explanation for the improved performance of JumpRel.U and TopK over ReLU observed in (Rajamanoharan et al., 2025; Gao et al., 2025).
Substituting (6) into (3) yields the training objective for SAEs (Cunningham et al., 2023; Raja- manoharan et al., 2025; Gao et al., 2025):
[ 2
min D.W D.WERAx DERR.
Ex
x (Dz + b)1l2 AR(z), where xr (W x be)
(9)
In practice, the two instances of À in (8) may be decoupled provide additional flexibility for hyper-parameter tuning.
The use of parameterized encoder is key design choice that circumvents the challenging non- convex optimization in the original dictionary learning formulation (3), which requires simultancous optimization over the sparse codes z and the dictionary parameters D and b. By decoupling this joint optimization, SAEs yield a more tractable training procedure. The encoder arises as a single proximal gradient step, augmented with uncoupled, learnable parameters for the dictionary and bias to reduce the approximation error relative to exact sparse coding. Consequently, the training problem (8) can be efficiently solved via stochastic gradient descent, and SAEs can be implemented efficiently at inference time, making them attractive for interpretability research.
This perspective also provides a principled foundation for developing SAE variants with improved performance. For example, by incurring additional computational cost, one may extend (6) multi- step variants, yielding multi-layer encoders (Tolooshams & Ba, 2022) that produce more accurate sparse codes and potentially capture finer-grained structure in the representation space. We leave this direction to future work. In the next subsection, we turn to SAES induced by alternative sparsity regularizers.
2.3 BEYOND NON-NEGATIVITY: SPARSE AUTOENCODERS WITH ABSTOPK
The proximal perspective developed above sug- gests that design choices for SAEs can be inter- preted as the selection of the sparsity-inducing penalty. While this view explains their sparsity- inducing effect, it also reveals fundamental limitation of current SAEs in Equation (7): they prompt sparsity but also enforce neggnegatiii discarding half of the representation space. As many semantic axes are naturally bidirectional (e.g., male v.s. female, positive v.s. negative sentiment), restricting sparse codes to be non- negative fragments these concepts into two sep- arate directions or collapses one side entirely.
Non-Nagstive
Our SAE
Feature2:
Peopla
Man
Feature2:
Mon
People
Woman
Festurel:
Gender Mala
Woman
Feature1 Gander:
Pos for Male.
Feature3:
Nog
Toy example where man & male + peo- ple and woman female + people: non-negative SAE needs two separate gender features, whereas AbsTopK uses one signed gender feature.
Figure 2:
Fragmentation of Conventional SAE To
formalize this, consider single semantic concept direction in (2) represented by dictionary atom d Rd. An ideal sparse code would represent concepts along this axis as ad, where the sign of the scalar a encodes directionality. However, under the non-negativity constraint z > 0, this is impossible. Instead, a standard SAE must allocate two distinct dictionary atoms, d; and d,.

---

Published as a conference paper at ICLR 2026
oriented in opposite directions, with nonnegative activations 2; 2 and 2j 2 0 respectively. Each atom is activated only for one direction, leading to a fragmented representation that arises directly from the non-negativity constraint. This fragmentation is a direct consequence of the non non-negativity constraint.
Removing non-negativity as remedy. To address this issue, we propose using a sparse regular- izer without the non-nege negativity constraint. Different variants of sparse regularizers can be consid- ered, with representative examples discussed in Lemma I. In this work, we adopt the lo norm due to its simplicity and its direct connection to sparsity. Specifically, in the dictionary learning formu- lation (3), we use the regularizer R(z) =
1. S}] which removes the non-negativity constraint present in the TopK-inducing regularizer. The corresponding proximal operator is
prox R (u) arg min
1 el
2112 s.t.
||=||o S k,
(10)
#CR 2
whose closed-form solution is further given by
us i Hz(u), 0, i $ Hz(u),
(prox (AbsTopK (u));
(11)
where H4 denotes the indices of the k largest (in modulus) components In words, this operator preserves the & largest-magnitude components of vector and sets all others to zero. In the com- pressive sensing literature, is referred as the hard thresholding operator (Foucart, 2011). Here, we refer to it as Absolute TopK (AbsTopK) to distinguish it from the TopK operator commonly used in SAE.
This principle of hard thresholding can also be applied to JumpReLU, introducing threshold on both positive and negative activations. This achieves similar effect by eliminating small-magnitude features and enforcing sparsity. However, to isolate and directly test our core hypothesis, the value of representing concepts along a bipolar axis, this work focuses on AbsTopK, as it provides the most direct implementation of global k-sparsity constraint. We remain JumpReLU variants for future investigation.
AbsTopK SAE. Following the derivation in the previous section, we integrate the AbsTopK non- linearity operator into the framework (8) to obtain new SAE architecture, which we term AbsTopK SAE:
= AbsTopK x be), a = Dz +
(12)
The overall training problem becomes
2 = (D=zbb)l2 where = AbsTToK(WTTTooom x + +ba)
min D. D.WER EE
Em
(13)
By design, AbsTopK preserves both positive and negative activations, enabling a single feature to capture contrastive concepts along a unified semantic axis. This simple modification circumvents the fragmentation induced by non-negativity constraints, and yields features that more faithfully re- flect the bidirectional structure of semantic representations. Importantly, we do not claim that every feature should realize perfectly symmetric semantic axis. When concept is naturally bipolar, the model should be able to represent with a single bidirectional feature, rather than having such fea- tures ruled out by construction. At the same time, the formulation fully supports unipolar concepts, these can simply make use of the positive side of the feature to sparsely encode the hidden states.
EXPERIMENTS: EMPIRICAL VALIDATION OF SAE BEHAVIOR
To empirically validate our theoretical claims and demonstrate the practical advantages of the Ab- s' s'TppK operator, we perform a suite of experiments which involve training JumpReLU,, Topk,, and (Gao et al., 2020) across multiple mod-
Abs TosT SAES on monology htypppilae pppile-iee els (Radford et al., 2019; Biderman et al., 2023; Team, 2024; Yang et al., 2025). To compare the
"Similarly, if the l largest components not uniquely defined, one can, for instance, select those with the smallest indices ensure exactly à entries retained.
7

---

Published as a conference paper at ICLR 2026
different SAEs, evaluate their performance along several dimensions: (i) reconstruction quality on base datasets, (ii) effectiveness on range of steering tasks, and (iii) impact on general capa- bilities of the models. For further experimental details and extended results, we refer the reader to Appendix B.
6 10
SAE
Topk SAB JimpReel SAE
Topk
0.90
6x 10
Normalized MSE
Jurnig RolL SAE
Recovered
0.96
MSE Loes
4x 10
0.94
0.92
3x 10-4
2x-0
x10
0.88
0.86
AbsTopk
0.88
jimplel
10000 20000 Figure 3:
30000
100
200
100
200
Training Steps
Number of Active Features
Number of Active Features
Performance comparison of JumpReLU, TopK, and AbsTopK SAEs Qwen3 4B Layer 20, showing (a) MSE Training Loss, (b) Normalized MSE, and (c) Loss Recovered. Addi- tional results across models and layers are provided in Appendix D.
3.1 UNSUPERVISED METRICS
This section presents a comparative evaluation of SAE architectures, utilizing a suite of comple- mentary metrics engineered to assess distinet facets of model performance. The investigation en- compasses three primary analyses: (a) an examination of the training mean squared error (MSE) to evaluate optimization stability and convergence rates; (b) the measurement of normalized recon- struction error as function of feature sparsity to ascertain representational fidelity; and (c) relative cross-entropy loss recovered score to determine the preservation of language modeling performance. For Topk and AbsTopK, sparsity is explicitly controlled by directly specifying the number of active features k; in contrast, for JumpReLU, sparsity is varied by manually adjusting the threshold param- eter 6, thereby simulating different sparsity levels.
The normalized reconstruction error in (b) is defined as nMSE(2: .) ||x a 1 x (Gao al., 2025), thereby controlling for scale differences across representations. The Loss Recovered score in (c) measures how well SAF reconstructions preserve predictive performance (Karvonen et al., 2025), defined as (H* - Hn)/ (o)/(HHu (Horig Ho). where Hong is the cross-entropy of the original model, H" that after substitution, and Ho under zero-ablation, with values closer one indicating better preservation.
AbsTopK achieves the most favorable behavior and consistently attains lower reconstruction error across most sparsity levels while inducing only minor cross-entropy degradation. This advantage is explained by the expressiveness of the underlying constraints. TopK and JumpReLU enforce non- negativity, inducing conical decomposition that tends to split inherently bidirectional concepts across multiple features. AbsTopK instead allows signed activations, so a single feature can encode opposite concepts via its sign, yielding a more compact and interpretable linear decomposition of the latent space. As we show in subsequent qualitative analyses, this bidirectional capacity leads to dictionary atoms that more closely align with the conceptual structure of the model's representations.
3.2 RESULTS ON PROBE AND STEERING TASKS
To assess the utility of learned SAE features for model control, comprehensive benchmarking evaluation was conducted across a diverse suite of steering and probing tasks. These tasks were specifically designed probe various dimensions of feature quality, from basic concept representa- tion to the capacity for precise interventional control. detailed methodological overview for each metric is provided in the Appendix E.
The empirical results, as shown in Figure 4, demonstrate the superiority of the AbsTopK method- ology. Across the entire suite of evaluated tasks, AbsTopp SAE outperforms both the TopK SAE
8

---

Published as a conference paper at ICLR 2026
1.0
Topx SAE
AbsTopk SAE
JumpReLU SAE
Accuracy
0.0
Unlearoing
Absorpton
SCR
TPP
RAVEL
Sparse Probing
Figure 4: Performance comparison of SAE variants (TopK, AbsTopK, and JumpReLU) across tasks on Qwen3-4B Layer 18. For all tasks, higher scores indicate better performance; the Unlearn- ing and Absorption scores have been transformed as 1 l-rigiiaal score to maintain this consistency. We report the mean across five runs (random seeds 40-44), with error bars indicating the standard deviation. For more details, see Appendix E.
and JumpReL.U SAE baselines. This performance advantage is especially conspicuous in bidirec- tional steering metrics, such SCR, which directly quantify the reliability of interventions. In these critical evaluations, AbsTopK shows marked improvements over the alternatives.
We posit that this consistent outperformance is directly attributable to the core mechanism of the AbsTopK methodology: the retention of both positive and negative feature activations. Unlike TopK approaches, which enforce a hard sparsity constraint that discards all but the most prominent positive activations, AbsTopK preserves a richer, more complete semantic representation. This retention is critical for interventions that require nuanced and bidirectional control. By encoding not only the presence of concept but also its negation or semantic opposition, AbsTopK features provide a more robust and granular basis for manipulation.
3.3 EMPIRICAL RESULTS ON STEERING VS. UTILITY
Table 1: Performance comparison on MMLU (f) and HarmBench (f) across steering methods. Entries show the absolute score; colored values in parentheses indicate the change relative to the unsteered Original model (red: improvement, blue: drop). The best result among all methods for each metric is highlighted in bold.
Model
Layer
Metric
Original
Rel.U SAE
JumpRel.U SAK
Topk SAE
AbsTopK SAK
Qwen3 4B
18
HammBench
MMLU
77.3
17.0
(461 78.5
(-2.9) 74.4
(462.1)79) 79.1
(52
(461.2)
(521))75.2
(464 8)
(-1.4)
(463.6) 80.6
(-1.5)
20
HarmBench
MMLU
77.3
(460.2)
(-1.5)
(+61
(-1.6)
(460.0)
(2.3)75.0
(462.0)
(+0.9) 76.4
(463.0)
(409) 76.4
Gemma2 28
16
12
HarmBench
HarmBench
MMLU
MMLU
52.2
52.2
19.0
(+50.9)
6799.9)67.9
(-2.2) 50.0
(:2.9) 49.3
(+50.5)
(+50.5)695
(:3.4) 48.8
(-1.0) 48.2
(+50,8) 69.8
(+51.2)
(:3.1)49..
(-3.7) 48.5
(+52.7)
(+51.2 70.2
(.0.9) 51.3
(-1.2) 51.0
(+51.5) 70.8
(+53.0)
(-1.4)500
(11.2)
Laaa3.1 SB
24
HarmBench
MMLU
66.7
15.2
(475.0) 90.2
(-2.5) 64.2
(474.7) 89.9
(-1.5)
(474.0) 89.2
(+L.7)
(476.1) 1) 91.3
(-0.9) 63.8
(477.2) 92.4
(-1.3) 65.4
MMIII
74.5
(-2.7)
(-2.0) 72.5
(-1.5)
73.2 (149.0)
HarmBench
(145.5)
(148.8)
Gemma3 12B
40
HarmBench
MMLII
74.5
166
(470.2) 86.8
(-3.2)71.3
(472.0)
(-2.5)72..
(470.7)
(:355)71.0
(472.6) 89.2
(-1.8) (2.78)72.7
(473.4) 90L0
(-2.0) 72.5
To more comprehensively characterize the safety uuility trade-off, we evaluate steering across four model and intervene multiple layers spanning early, middle, and late blocks. This diversity in both architectures and intervention depths allows us to test whether our conclusions are robust to model scale and to the choice of steering layer, rather than being artifact of single model configuration.

---

Published as a conference paper at ICLR 2026
Model steering confronts fundamental tradeoff: enhancing specific behaviors often degrades gen- eral capabilities. has often been assumed in prior literature that DiM interventions are more effective for specific concept manipulation than SAEs despite their reliance on labeled data and limitation to extracting only a single concept vector (Arditi et al., 2024; Wu et al., 2025; Zhu et al., 2025). To systematically evaluate this trade-off, we conducted an empirical study measuring general capability preservation via the MMLU benchmark (Hendrycks et al., 2021) and safety alignment us- ing HarmBench (Mazeika et al., 2024). For this evaluation, we focus on Qwen and Gemma models, as smaller models, Pythia-70M and GPT-2 Small, only have very low score on MMLU benchmark.
As shown in Table 1, the empirical results indicate that conventional SAE steering methods success- fully improve safety metrics but at detriment to general performance. In contrast, the proposed AbsTopK methodology achieves a more optimal balance between these competing objectives. facilitates substantial enhancements in safety alignment on HarmBench while simultaneously mit- igating the degradation of MMLU scores. Compared to DiM, AbsTopK competitive on safety, sometimes slightly lower, but consistently retains more general ability. This pattern highlights that carefully designed SAE steering can rival and, in some cases, surpass intervention strategies that rely on labeled data.
3.4 BIDIRECTIONAL SEMANTIC AXES IN ABSTOPK vs. TOPK
LLM-based automatic interpretation of AbsTopK and TopK features on Gemma-2-2B. For each layer and method, we report the proportion features in three semantic categories: double- sided, single-sided, and no clear meaning. The row marked with - Opposite meaning gives the subset of double-sided features whose two polarities express opposite semantics.
Table 2:
Layer 12 (%)
Layer 16 (%)
Category
AbsTopk
TopK
AbsTopK
TopK
Double-sided meaning (all)
29.7
5.3
31.2
4.1
< Opposite meaning
20.2
2.6
21.5
1.8
Single-sided meaning
56.4
78.8
57.8
80.3
No clear meaning
13.9
15.9
11.0
15.6
To quantify the bidirectionality, apply Gemini 2.5 Flash classify each feature into three cate- gories: double-sided meaning, where both polarities are judged meaningful; single-sided meaning, where only polarity is meaningful; and clear meaning. Within the double-sided group, further identify a subset of opposite meaning features whose positive and negative activations are judged express opposing semantics.
Table summarizes the distribution of these categories. Across both layers, the fraction with no clear meaning is comparable for AbsTopp and TopK, indicating that the bidirectional features in AbsTopK does not arise from noisier features. The nonzero mass of opposite-meantne features under TopK suggests that the underlying representation already supports bidirectional semantic axes, but only weakly exploits them. By relaxing the non-negativity constraint, AbsTTppK converts part of this single-sided inventory into semantic directions with two meaningful ends.
CONCLUSION
This work identifies the non-negativyty constraint in SAEs as core cause of semantic feature frag- mentation. response, we introduce the AbsTopK operator, which replaces this constraint with direct k-sparsity enforced via an to proximal operator. This modification enables single features to capture bipolar semantics, and our empirical results confirm that AbsTopK yields reconstructions of superior compactness and fidelity. Our work pioneers shift towards bipolar sparse representa- approximations of the to
tions and suggests future research into more efficient, operator for large-scale models.
ACKNOWLEDGEMENTS
We acknowledge support from the National Science Foundation under grants IIS-2312840, IIS- 2402952, I1S-2301599, CMMI-2301601, and DMS-2529302. We thank members of the OSU CSE
10

---

Published as a conference paper at ICLR 2026
community for valuable discussions and feedback. We thank the Ohio Supercomputer Center for providing the computational resources.
REFERENCES
Sebastian Acuña, Ida S Opstad, Fred Godtliebsen, Balpreet Singh Ahluwalia, and Krishna Agarwal. Soft thresholding schemes for multiple signal classification algorithm. Optics Express, 28(23):
#4434-34449 2020.
Etowah Adams, Liam Bai, Minji Lee, Viyang Yu, and Mohammed AlQuraishi. From mechanistic interpretability to mechanistic biology: Training, evaluating, and interpreting sparse autoencoders on protein language models. bioRxiv, 2025. doi: 10. 10.1100/20055.... wtt.:/222.......66666666 636691.. URL https:
/ //www. bicrxiv...vv.........i.. org/ wwwtteneearllyeeeeeoooooooooeete 22255/ 02 / 2025 02 06. 636901.
M. Aharon, M. Elad, and A. Bruckstein. K-svd: An algorithm for designing overcomplete dictionar- ies for sparse representation. IEEE Transactions Signal Processing, 54(11):4311-433) 2006. doi: 10. 10.11099/9P.00008888111
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal language models is mediated by a single direction. arXiv preprint arXiv:2406.1177 2024.
Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Conference learning theory, pp. 113-149. PMLR, 2015.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018. doi: 10.1 a a0004. URL https: / //aclanthology www./oo-//////
Kola Ayonrinde, Michael T Pearce, and Lee Sharkey. Interpretability as compression: Reconsidering sae explanations of neural activations with mdl-saes. arXiv preprint arXiv:2410.11177 2024.
Chenglong Bao, Hui Ji, Yuhui Quan, and Zuowei Shen. LO norm based dictionary learning by proximal methods with global convergence. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3858-3865, 2014. doi: 10. (099CCVR.2004.499
Chenglong Bao, Hui Ji, Yuhui Quan, and Zuowei Shen. Dictionary learning for sparse coding: Algorithms and convergence analysis. IEEE Transactions on Pattern Analysis and Machine In- telligence, 38(7):1356
2016. doi: 1011009TPAM120052488585
Boaz Barak, Jonathan A Kelner, and David Steurer. Dictionary learning and tensor decomposition via the ---urressss method. In Proceedings of the sevey-seventh anmual ACM symposium on Theory of computing, pp. 143-151,201
Usha Bhalla, Suraj Srinivas, Asma Ghandeharioun, and Himabindu Lakkaraju. Towards unifying interpretability and control: Evaluation via intervention, 2025. URL https: httttt:s:///peeeeeeeee openreview. hertfforum? dd fve33kk......
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Haffield-Dodds, Alex Tamkin, Karina Nguyen, Brayden MeLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- Mieriit.pp//022/mmoommmttttteeee

---

Published as a conference paper at ICLR 2026
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. In NeurIPS 2024
Workshop on Scientific Methods for Understanding Deep Learning, 2024. URL https:// openreview wett/oorumiddddddoooooooooo
Shuting Cai, Shaojia Weng, Binling Luo, Daolin Hu, Simin Yu, and Shuqiong Xu. A dictionary- learning algorithm based on method of optimal directions and approximate k-svd. In 2016 35th Chinese Control Conference (CCC), pp. 6957-6961, 2016. doi: 00.11099ChicC..22167555555
David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Isaac Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2025. URL https:/ /pppnreveeeeeeeeeeeeeeeeeeeeeeee net/ forum? (eerummiddL22XxPFP LC2KKKC3n..
Niladri S. Chatterji and Peter L. Bartlett. Alternating minimization for dictionary learning with random initialization. In Neural Information Processing Systems, 2017. URL https://api. htt.aaar..rg.ooorruaaa12222222 Corppu
semanti bemanticscholar. org.
Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentan- gling factual knowledge in gpt-2 small, 2024.
Siyu Chen, Heejune Sheen, Xuyuan Xiong, Tianhao Wang, and Zhuoran Yang. Taming polyseman- ticity IIms: Provable feature recovery via sparse autoencoders. ArXiv, abs/2506. 14002, 2025. URL httpss:/appt
semanticscholat wwg//orppsIi............ http://orpuui:::241000000 279410344
Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang, and Wotao Yin. Learning to optimize: A primer and benchmark. Journal of Machine Learning Research, 23(189):1-59, 2022.
Julien Colin, Lore Goetschalckx, Thomas FEL, Victor Boutin, Jay R Gopal, Thomas Serre, and Nuria M Oliver. Local vs distributed representations: What is the right basis for interpretability?, 2025. URL htttss://peereveeee
net wttp//oormmmeeeee/eee Ems.Er/-.-... EmWVPbRGC4.
Róbert Csordás, Christopher Potts, Christopher D) Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations. In Yonatan Be- linkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjic Chen (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 248 262, Miami, Florida, US, November 2024. Association for Computational Lin- guistics. doi: 10. 10.18653//1/2024.. http://224..aacckboop-p..... URL https::/aclaattholooo/ .org/ 2024 www.bbccccoxlllp...1.
Jingyi Cui, Qi Zhang, Yifei Wang, and Yisen Wang. On the theoretical understanding of identifiable sparse autoencoders and beyond. arXiv preprint arXiv:2506. 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.0s8 08600, 2023.
Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57 1):1414444
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias bios: A case study of semantic representation bias in high-stakes setting. In proceedings of the Confer- ence on Fairness, Accountability, and Transparency, pp. 120- 128, 2019.
Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, Oliver Rausch, Saffron Huang, Sam Bow- man, Stuart Ritchie, Tom Henighan, and Deep Ganguli. Evaluating feature steering: A case study in mitigating social biases, 2024. URL https: : ////anrtthicccc com/ henn/eeeeach/ eval vat ing-ing-ooe wnnnnngaferreee feature- sttrreentteeing..
12

---

Published as a conference paper at ICLR 2026
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Blatield-Doddss Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposi- tion. arXiv preprint arXh::2209. 2022.
Joshua Engels, Logan Riggs Smith, and Max Tegmark. Decomposing the dark matter of sparse autoencoders, 2024. URL hwwphhttp///t. /ttttp:://penrevviiii wwtpeiiwwwweeeeeeeeeeeee..eeeeee Sobrrum?dd-S2220 rqr.
Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. In The Thirteenth International Conference on Learning Representations, 2025. URL https: : //pppppeeeeieeeeeeeee/eeeeeeeeee net www.roiiddddaaa........ 4AMAhb.
Eoin Farrell, YeugTong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models, 2025. URL https:://////eeeeeee /wpp///ppereeeeeeeeeeeeeeeee ew. wwt. fert/form?? ?
ZtvRqm66Ru..
Thomas Fel. Sparks of explainability:: recent advancements in explaining large vision models. 2024.
Thomas Fel. Sparks of explainability: Recent advancements in explaining large vision mod- cls. (lueurs d' d'explicabilité Avancées récentes dans l'explication des grands modèles de vi- sion). ArXiv, abs/2502.01048, 2025. URL https:/ /api. semanticscholle org/ Corpus 277A:IDD:2299900
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2711-2721, 2023.
Simon Foucart. Hard thresholding pursuit: algorithm for compressive sensing. SIAM Journal on numerical analysis, 49(6):2543-22 2011.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Lco Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
Edith Natalia Villegas Garcia and Alessio Ansuini. Interpreting and steering protein language mod- cls through sparse autoencoders. arXiv preprint arXiv:2502.09135. 2025.
Dar Gilboa, Sam Buchanan, and John Wright. Efficient dictionary learning with gradient descent. In International Conference on Machine Learning, pp. 2252-2259, PMLR, 2019.
Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference on international conference on machine learning, pp. 399 406, 2010.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas. Finding neurons in haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https: /pwpneevieeeeeeeeeeeeeeeeee net. forum?
id=JYs1R9IMM
Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations. Chapman & Hal//CRC, 2015. ISBN 14987120 1969..
Thomas Heap, Tim Lawson, Lucy Farnik, and Laurence Aitchison. Sparse autoencoders can inter- pret randomly initialized transformers. arXiv preprint arXiv:2501.17727 2025.
Praveen Hegde. Effectiveness of sparse autoencoder for understanding and removing gender bias in LLMss. In NewrIPS 2024 Workshop on Scientific Methods for Understanding Deep Learning, 2024. URL https: opwnoeveeeeeeeeee n forum? d
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- fional Conference on Learning Representations (ICLR), 2021.
13

---

Published as a conference paper at ICLR 2026
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2017. URL https::t///./..... http///ppeeeeieeeeeeeeeeeee net/ Furum? fo d=Sy2
Sai Sumedh R. Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assump- tions: The duality between sparse autoencoders and concept geometry. In ICML 2025 Workshop on Methods and Opportunities at Small Scale, 2025. URL forum? 3d=Akao3zhhhh
net/
Yupeng Hou, Jiacheng Li, Zhankui He, An Yan, Xiusi Chen, and Julian McAuley. Bridging language and items for retrieval and recommendation. arXiv preprint arXiv:2403.03953 2024.
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? case study in sparse probing. In Forty-second International
Conference on Machine Learning, 2025. URL https:/ /openreviivv ew net forum? ? rNfzT8ykgO.
Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders targeted concept erasure tasks. arXiv preprint arXiv:2411.18895, 2024.
Adam Karvonen. Can Rager, Johnny Lin, Curt Tigges, Joseph Isaac Bloom, David Chanin, Yeu- Tong Lau, Eoin Farrell, Callum Stuart McDougall, Kola Ayonrinde, Demian Till, Matthew Wear- den, Arthur Conmy, Samuel Marks, and Neel Nanda. SAFBench: comprehensive bench- mark for sparse autoencoders in language model interpretability. In Forty-second International Conference on Machine Learning, 2025. URL https:////// http////ppeevveieee net forum? ?id= qr03yNEXOd.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learming, pp. 2668-2677. PMLR, 2018.
Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. Saes are
highly dataset dependent: A case study on the refusal direction. Alignment Forum, 2024. URL
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert- Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Liu, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Tal- ley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The wmdp benchmark: Measuring and reducing malicious with unlearning, 2024.
Jinqi Luo, Tianjiao Ding, Kwan Ho Ryan Chan, Darshan Thaker, Aditya Chattopadhyay. Chris Callison-Buree and René Vidal. Pace: Parsimonious concept engineering for large language models. Advances in Neural Information Processing Systems,
Julien Mairal, Francis Bach, and Jean Ponce. Task-driven dictionary learning. IEEE transactions on pattem analysis and machine intelligence, 34(4):791-804 2011.
Benoît Malézicux, Thomas Moreau, and Matthieu Kowalski. Understanding approximate and un- rolled dictionary learning for pattern recovery. In International Conference on Learning Repre- sentations, 2022. URL /ttpttps://ppereevieeee n wttprrrreeeeeeeeeeooooooooooeee. oo
Yongqiang Mao, Zonghao Guo, Xiaonan Lu, Zhiqiang Yuan, and Haowen Guo. Bidirectional feature globalization for few-shot semantic segmentation of 3d point cloud scenes. 2022 In- ternational Conference on 3D Vision (3DV), pp. 505-514, 2022. URL https: //pp://aa0 api.
semanti Comanticschol. wwwscollr.orr. wtw.c//oa....r//oorrraaaaaa.e.ea 562999
14

---

Published as a conference paper at ICLR 2026
Samuel Marks, Can Rager, Erie J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs language models. In The Thirteenth International Conference Learning Representations, 2025. URL https: / /oppneeeieee www.eeeeeew........ net./ forum? (etp:mmmddTAA88TTTTT
Fabio Valerio Massoli, Christos Louizos, and Arash Behboodi. Variational learning ISTA. Transac- fions Machine Learning Research, 2024. ISSN 2835-8856. URL petttts::/opeerreeeee net fest/fou forum? id=Akk S0sittssuuuu
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. Harmbench: A standard- ized evaluation framework for automated red teaming and robust refusal. 2024.
Abhinav Menon, Manish Shrivastava, Ekdeep Singh Lubana, and David Krueger. Analyzing (in)abiliities of SAEs via formal languages. In MINT: Foundation Model Interventions, 2024.
URL https::///// aapp/eeeeeviiieee net forum? i d Yd
Tomas Mikolov, Kai Chen, Gregory Corrado, and Jeffrey Dean. Efficient estimation of word
representations in vector space. In International Conference Learning Representations, 2013.
URL https::t//// /api attaaappsesmmmsseee semanticschooiotoo
org/ Ctrp///rppsiic 5959482.
Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (666833)66076609 1996.
Charles O' D'Neill, Alim Gumran, and David Klindt. Compute optimal inference and provable amorti- sation gap in sparse autoencoders. In Forty-second International Conference on Machine Learn- ing. 2025. URL httttts://ppeeeeeeee /ttp///ppeeeveeeeeeeeeeeeeeeeeee ew, net. forum? d=8forr
Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends@ in Optimization, 1(3):127-239, 2014.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03652 2023.
Kenny Peng, Rajiv Movva, Jon Kleinberg, Emma Pierson, and Nikhil Garg. Use sparse autoencoders to discover unknown concepts, act on known concepts. arXiv preprint arXiv:2506.2384
2025.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 1532-1543, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10. 3115/v//D14-100 URL https: httpa////aaaaataaaaa httaa//aataltlyyoorgge..eeeeeeee org/ orggD14-1162//
Tomaso A. Poggio and Thomas Serre. Learning a dictionary of chape-components in visual cortex: comparison with neurons, humans and machines. 2006. URL https://api.
semanticscholes wwwphoolarrrrrr/errooeoe ort//oruss ID: 92292622 61154.
Qing Qu, Yuexiang Zhai, Xiao Li, Yuqian Zhang, and Zhihui Zhu. Geometric analysis nonconvex optimization landscapes for overcomplete learning. In International Conference Learning Representations, 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models unsupervised multitask learners. 2019.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model acti- vations with gated sparse autoencoders. In Proceedings of the 38th International Conference Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2024a. Curran Asso- ciates Inc. ISBN 9798331314385.
15

---

Published as a conference paper at ICLR 2026
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model ac- tivations with gated sparse autoencoders. In The Thirty-eighth Anmual Conference Neural Information Processing Systems, 2024b. URL https: /wwporppeeeeeeeeeeee net forum? ? d= zLB1 inllii.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Commy, Vikrant Varma, Janos Kramar, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumpreLU sparse autoencoders, 2025. https: / oppnrevieee net/ forumm d=mMPa a
Gustavo Silva and Paul Rodriguez. Efficient consensus model based proximal gradient method applied to convolutional sparse problems. arXiv preprint arXiv:2011. 2020.
Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language mod- els via sparse autoencoders. bioRxiv, pp. 2024.
2024.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty- second International Conference Machine Learning, 2025. URL https::/t////.c.... httpp:://ppenrvviet /pw.reveee...e.eee net/fo formm? i dd 7ddvTX.
Daniel A Spielman, Huan Wang, and John Wright. Exact recovery of sparsely-used dictionaries. In Conference on Learning Theory, pp. 37-1. JMLR Workshop and Conference Proceedings, 2012.
Jeremias Sulam, Chong You, and Zhihui Zhu. Recovery and generalization in over-realized dictio- nary learning. Journal of machine learning research, 1 1-23, 2022.
Ju Sun, Qing Qu, and John Wright. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions Information Theory, -8844483-88 2016.
Wen Tang, Émilie Chouzenoux, Jean-Christophe Pesquet, and Hamid Krim. Deep trans- form and metric learning network: Wedding deep dictionary learning and neural net- works. ArXiv, 0bs/2002.07898, 2020.
URL https:: /tttps//app..eematticccoooooo sew.nticschooeee......o.. ooggg
CorpusID:
Gemma Team. Gemma. 2024. doi: 10.34740/KKAGGLL////3310
URLhttp:: http://GGEEEEEEO.O....L///////// kaggle.
com/m/ 3301.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jemmyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL https:: /ttt:///raaaaaee wow.aaassfrrrrrooo cwemmerrciiie reciiccitttt ts pub/2024/ scal scaling-aai tingmmonosemanteo i tityy/tttti /eliity//eeneee index. html.
Harrish Thasarathan, Julian Forsyth. Thomas Fel, Matthew Kowal, and Konstantinos G. Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment. In Forty- International Conference on Machine Learning, 2025. URL https: WtRLhhtts:///penreee http///peerrveeeeeeeeeeeeeeeeeee www.eeeiewweeeee net./ forum? (auumm/ddUOaxRNBB-
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
Bahareh Tolooshams and Demba E. Ba. Stable and interpretable unrolled dictionary learn-
ing. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL https: // wopeneeview.eet//e...e 0ttp://eee/ooummmm00303000000
Ivana Tošić and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2):
27-38, 2011.
Rheeya Uppaal, Apratim Dey, Yiting He, Yiqiao Zhong, and Junjie Hu. Model editing robust and denoised variant DPO: A case study on toxicity. In Neurips Safe Generative Al Workshop 2024, 2024. URL htttps:/opperreitit wwwpreveeeeeeeeeeeeoee net/ www.mrrrmmeeeeee tttp:/oommmmmm12222mmmmm..- 222221pyaa
16

---

Published as a conference paper at ICLR 2026
Mengru Wang, Ziwen Xu, Shengyu Mao, Shumin Deng, Zhaopeng Tu, Huajun Chen, and Ningyu Zhang. Beyond prompt engineering: Robust behavior control in Ilms via steering target atoms. arXiv preprint arXhv:2505.03322, 2025.
Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christo- pher D Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outper- form sparse autoencoders. arXiv preprint arXiv:2501.17148, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan 2025.
Qiu. Qwen3 technical report. arXiv preprint arXiv:2505.0995
Xudong Zhu, Jiachen Jiang, Mohammad Mahdi Khalili, and Zhihui Zhu. From emergence to control: Probing and modulating self-reflection language models. arXiv preprint arXiv:2506. 12217, 2025.
Zhihui Zhu, Tianyu Ding, Daniel Robinson, Manolis Tsakiris, and René Vidal. A linearly convergent method for non-smooth non-convex optimization on the grassmannian with applications to robust subspace and dictionary learning. Advances in Neural Information Processing Systems, 32, 2019.
17

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Sparse Dictionary Learning The dictionary learning problem (3) is highly nonconvex. Over the past decades, numerous heuristic methods have been proposed solve it efficiently (Cai et al., 2016; Aharon et al., 2006; Tošić & Frossard, 2011). Significant effort has also been devoted to addressing the nonconvexity and establishing theoretical guarantees for exact recovery, including approaches based on convex relaxation (Spielman et al., 2012), semidefinite programming (Barak al.. 2015), landscape analysis of simplified formulations that estimate one atom at a time (Sun et al., 2016; Gilboa et al., 2019; Zhu et al., 2019; Qu et al., 2020), alternating minimization (Arora et al., 2015; Chatterji & Bartlett, 2017), as well as global convergence guarantees critical points (Bao al., 2014; Hastic et al., 2015; Bao et al., 2016). Subsequent work has explored unrolled meth- ods (Malézieux et al., 2022: Tolooshams & Ba, 2022: Chen et al., 2022) and over-parameterization (Sulam et al., 2022), analyzed convergence for solving the sparse coding problem under the sumption of fixed dictionary (Gregor & LeCun, 2010; Tang et al., 2020; Massoli et al., 2024), and examined gradient stability (Tolooshams & Ba, 2022; Malézieux et al., 2022).
Building on these works, we re-examine recently popularized SAEs for interpretability through the lens of unrolled dictionary learning. This perspective reveals direct correspondence between tivation functions and the proximal mappings of sparse regularizers, thereby situating SAEs within the broader framework of dictionary learning. Leveraging this connection, we introduce a new activation, AbsTopK, which removes the non-negativity constraint imposed in prior designs and en- ables a single dictionary feature to encode bidirectional semantic axes. Our principal contribution is thus formal alignment of SAE architecture with the intrinsic geometry of semantic representa- tion-distinct from classical theories focused on signal recovery.
Mechanistic interpretability
SAEs have become a central tool in mechanistic interpretability. serving as a dictionary learning ap- proach for concept-level explanations (Kim et al., 2018). Several architectures have been proposed, including ReLU, TopK. JumpReLU, gated, Batch TopK, and ProLU SAES (Bricken et al., 2023; Gao et al., 2025; Rajamanoharan et al., 2025; 2024b; Bussmann et al., 2024; O'Neill et al., 2025), and have been shown to capture a wide range of interpretable features, from refusal, gender, and writing script (Bricken et al., 2023; Templeton et al., 2024; Hegde, 2024) to visual structure and protein representations (Thasarathan et al., 2025; Simon & Zou, 2024).
At the same time, recent work has highlighted important limitations of the SAE paradigm. Prompting-basee based interventions can outperform SAE-based control (Wu et al., 2025; Bhalla et al., 2025; Menon et al., 2024); other studies question the assumption that concepts by single linear features, showing that representations can be multidimensional or nonlinear (Engels al., 2024; 2025; Peng et al., 2025; Wang al., 2025). Moreover, SAES can be algorithmically unstable: models trained on the same data with different random seeds may yield divergent dic- tionaries and inconsistent interpretations (Ayonrinde al., 2024; Kissane et al., 2024; Colin al., 2025). These observations suggest that, while SAES are promising for interpretability. their current formulations are fragile and lack a canonical notion of representation.
well captured
Motivated by these challenges, recent theoretical work (Chen et al., 2025; Cui et al., 2025) inves- tigates when standard non-negative SAEs can provably recover ground-trute features under sparse, non-negative latent codes, providing principled justification for non-negativity in unipolar settings. Our framework complementary and tailored to the mixed-sign, bidirectional structure observed in LLM representations: from a proximal-gradice proximal operators with an implicit non-negativity constraint, and relax this constraint to obtain AbsTopk which preserves the same sparse coding objective while enabling bidirectional semantic axes and reducing to standard SAES in the non-negative limit.
viewpoint, we interpret SAE nonlinearities as
B EXPERIMENTAL SETUP
In this appendix, we describe the architecture and training setup of our SAEs. For all experiments, trained on the wwwollogyyplleuucoopp
(Gao et al., 2020) dataset.
Architecturally, the SAEs are comprised of a single, overcomplete hidden layer which incorporates sparsifying nonlinearity. The encoder component projects residual activations into latent space
18

---

Published as a conference paper at ICLR 2026
of higher dimensionality, while the decoder component reconstructs the original residual dimension from these latent representations. A fixed expansion factor 16 was uniformly applied across all models.
For comparative analysis, three distinct variants of the SAE were trained: TopK, AbsTopK, and JumpReLUU. In the TopK and AbsTopp configurations, exact k-sparsity enforced upon the latent representation, with the sparsity hyperparameter, k, and the specific layers targeted for intervention being systematically selected for each foundational model:
EleutherAA//pythiaa-00 (Biderman et al., 2023): k 51, layers: 3, 4.
google/gemma-2211 (Team, 2024): 230, layers: 12, 16.
Qwen/Owen3-4B (Yang et al., 2025): A = 256, layers: 18, 20.
openai- ponnnai-coomuuntyyggggoo (Radford et al., 2019): & = 76, layers: 6, 8.
Here, è was set to approximately one-tenth of the hidden dimension for each model, and the inter- vention layers were selected from the middle of the network to capture representative latent features (Arditi et al., 2024). In contrast, the JumpReLU models adopted the same configuration in prior work (Rajamanoharan et al., 2025; Bussmann et al., 2024).
The optimization for all models was performed using the Adam algorithm over duration of 30,000 training steps, with a consistent batch size of 4096. A learning rate of 3e-4 was configured, com- plemented by Adam's 's momentum parameters, 31 0.9, and B3 0.99. And we used bandwidth parameter of 0.001 across all experiments.
C PROOF OF LEMMA
Proof. We prove the result by deriving the proximal operator corresponding to each regularizer separately.
Case I: ReLU. Note that R(z) ) separable as
R(z) = |||| + 4=200 (z) E (1æi + (2)))
which implies that the proximal operator is also separable, i.., (proxxn(u)), is equivalent to the following scalar proximal problem
1
proxAn(u) arg min (2 u)² A/z1 (2)
ZER
2
1
arg min 7 (z - u)² A=
220 2
maxx X, 0}.
Therefore, the proximal operator induces the ReLU operator, with a shift by A:
(proxxR(u) = max(u X, 0}.
which reduces to the standard ReLU when - O. In this case, however, the operator no longer encourages sparsity. When À 0, the effect is equivalent introducing bias term that suppresses small activations and thereby promotes sparsity. In practice, this restriction can be relaxed: during training, gradient descent can learn a separate bias parameter for each entry.
Case II: JumpReL.U. Similarly, R(z) also separable as
R(z) = || + (2) = Σ (1(2 #0) +)++ (2x22000z))))
19

---

Published as a conference paper at ICLR 2026
where 12; * 0) = [1, 2; + 0, 10, 2, 0.
Thus, it suffices to first consider the following scalar proximal
operator
prox,n(u) arg min 1
(2 u)² + X1 + #0 + 4[=00] (2)
2
arg min #20
(2 - uu) A1 #)-
E(2)
Note that within the region 2 0, & achieve its minimum at either 0 u. Setting E(u) = E(O) 1²²²² yields u = V2x. One can verify that achieves its minimum at when 2 V22, and at otherwise. Hence, the proximal operator induces the JumpReLU with parameter V2A:
u, 2 22,,
(proxAR(u));
0,
V2X.
Case III: TopK. For this case, the corresponding proximal operator reduces Euclidean pro- jection onto the feasible set:
proxxa( = arg min
1 Ilue - =11 s.t. ||=||o < k, z = 0.
(14)
ZERd
Given the quadratic objective and the non-negativity constraint, the optimal choice on any candidate support S with |S| < k is
[a; 0}: i S, i i S.
(15)
Thus, the minimization problem reduces to selecting the index set S that captures the k largest nonnegative entries of u. Formally, letting (=) denote the of indices corresponding the largest entries of z, the proximal operator becomes
[proxA[())]: max un 0}, ie Te(z),
(16)
0,
i (z)).
D UNSUPERVISED METRICS ON ALL MODELS
This section presents the unsupervised metries from our model evaluations. We tested each model with a specific set of values. For the Pythia model, we used k-values of 10, 20, 30, 40, and 50. The evaluation of the Gemma model involved k values of 30, 50, 100, 200, and 230. For the GPT model. the k values were 10, 30, 50, 60, and 76. Lastly, the Qwen model was tested with k values of 30, 50, 100, 200, and 256.
As shown in Figure 5, across the majority of evaluated models, we observe that AbsTopK achieves lower training MSE, reduced normalized reconstruction error, and better preservation language modeling performance relative both TopK and JumpReLU. This consistent advantage across these metrics provides evidence for the effectiveness and robustness of the AbsTopK method.. In partic- ular, while TopK and JumpReLU sometimes exhibit competitive performance in isolated settings, AbsTopk maintains robustness across architectures and layers, thereby demonstrating the superior- ity of our proposed formulation.
E STEERING AND PROBE TASK ON ALL MODELS
E. TASK DESCRIPTION
We provide an overview of the tasks employed in the SAEBench evaluation for SAEs. For detailed methodology, refer readers the original SAEBench paper (Karvonen et al., 2025).
20

---

Published as a conference paper at ICLR 2026
Table Performance comparison SAE variants across tasks on all other models and layers. For all tasks, higher scores indicate better performance; the Unlearning and Absorption scores have been transformed as o-origiaal score to maintain this consistency.
Model
Method
Unlearning
Absorption
SCR
TPP
RAVEL
Sparse Probing
AbsTopK
0.93
0.73
0.27
0.34
0.73
0.76
Gemma2-2B L12
TopK
0.88
0.76
0.20
0.29
0.70
0.71
JumpReLU
0.90
0.75
0.22
0.30
0.71
0.73
AbsTopK
0.91
0.70
0.27
0.42
0.71
0.70
Gemma2-28 L16
TopK
0.89
0.68
0.21
0.36
0.74
0.67
JumpReLU
0.94
0.69
0.23
0.39
0.72
0.69
AbsTopK
0.75
0.54
0.20
0.22
0.64
0.66
Pythia-70M 1.3
TopK
0.71
0.47
0.15
0.14
0.62
0.60
JumpRel.U
0.73
0.50
0.17
0.21
0.63
0.61
AbsTopK
0.79
0.53
0.21
0.23
0.68
0.57
Pythia-70M
TopK
0.72
0.50
0.16
0.21
0.69
0.61
JumpReLU
0.77
0.51
0.17
0.20
0.61
0.62
AbsTopK
0.74
0.66
0.18
0.22
0.60
0.54
GPT2-small L6
TopK
0.80
0.63
0.14
0.19
0.57
0.50
JumpReLU
0.77
0.65
0.15
0.20
0.58
0.52
AbsTopK
0.75
0.67
0.23
0.28
0.51
0.59
GPT2-small L8
TopK
0.71
0.67
0.15
0.20
0.48
0.55
JumpReLU
0.73
0.67
0.18
0.23
0.49
0.57
AbsTopK
0.95
0.79
0.35
0.36
0.81
0.83
Qwen3-4B L18
TopK
0.91
0.77
0.26
0.31
0.79
0.82
JumpReLU
0.93
0.78
0.28
0.30
0.80
0.78
AbsTopK
0.95
0.80
0.32
0.45
0.85
0.81
Qwen3-4B L20
TopK
0.92
0.77
0.27
0.36
0.76
0.84
JumpReLU
0.93
0.78
0.29
0.39
0.81
0.83
E.1.1
FEATURE A ABSORPTION
Sparsity incentives can cause a SAE to engage in feature absorption, a phenomenon where correlated features merged into single latent representation. This process arises when direct implication exists between two concepts, such that concept A always implies concept B. To reduce the number of active latents, the SAE might absorb the feature for A into the latent for B. For example, a feature for "starts with S could be absorbed into a more general latent for "short." While this merging im- proves computational efficiency, compromises interpretability by creating gerrymandered features that represent multiple, distinct concepts.
To quantify feature absorption, employ first-letter classification task, following the methodol- ogy of previous studies (Chanin et al., 2025). First, a supervised logistic regression probe is trained on tokens containing only English letters establish ground- -ruth feature directions. Next, K-sparse probing is applied to the SAE's latents to identify the primary latent corresponding to each feature, using threshold of Tis 0.03 to account for potential feature splits. main latents fail but the probe succeeds, additional SAE latents are included if they satisfy cosine similarity with the probe of at least Tps = 0.025 and a projection fraction of at least Tpa = 0.4. All parameter values are chosen following the original SAEBench settings (Karvonen et al., 2025). To make the results more interpretable and such that higher values indicate stronger unlearning, we present the final scores as 1 original value.
test set tokens where
E. UNLEARNING
SAEs evaluated their ability selectively remove knowledge while maintaining performance on unrelated tasks (Farrell et al., 2025). We use the WMDP-bil bio dataset (Li et al., 2024) for unlearn- ing and MMLJU (Hendrycks al., 2021) to assess general abilities.
21

---

Published as a conference paper at ICLR 2026
The intervention methodology involves clamping selected WMDD--bio SAE feature activations to negative values whenever the corresponding features activate during inference. To evaluate broader model effects, also measure performance the MMLU benchmark (Hendrycks et al., 2021). The final evaluation reports the highest unlearning effectiveness on WMDP-bio while ensuring MMLU accuracy remains above 0.99, thereby quantifying optimal unlearning performance under constrained side effects. To make the results more interpretable and such that higher values indicate stronger unlearning, we present the final scores as original value.
E.1.3 SPURIOUS CORRELATION REMOVAL (SCR)
SCR (Karvonen et al., 2024) evaluates the ability of SAEs to disentangle latents corresponding to distinct concepts. We conduct experiments on datasets known for spurious correlations, such as Bias in Bios (De-Arteaga et al., 2019) and Amazon Reviews (Hou et al., 2024), which contain two binary gender labels. For each dataset, we create a balanced set containing all combinations of profession (professor/nurse) and gender (male/female), and female+nurse combinations. A biased classifier C' is first trained on the biased set and then debiased by ablating selected SAE latents.
well as biased set including only male+professor
We quantify SCR using the normalized evaluation score:
AaM
Abase
SSHIFT
(17)
Aoracle
Abase
where Aabi is the probe accuracy after SAE feature ablation, Abase is the baseline accuracy before ablation, and Aguacte is the skyline accuracy obtained by probe trained directly on the desired con- cept. Higher SSIIF values indicate more effective removal of spurious correlations. This score represents the proportion of improvement achieved through ablation relative to the maximum possi- ble improvement, enabling fair comparison across classes and models.
E.1.4 TARGETED PROBE PERTURBATION (TPP)
TPP (Marks et al., 2025) extends the SHIFT methodology to multiclass natural language processing datasets. For each class c; in dataset, select the most relevant SAE latents Lę. We then evaluate the causal effect of ablating L3 on linear probes Cj trained to classify each class cj.
Let Ay denote the accuracy of probe Cj before ablation, and AJas the accuracy after ablating L,. We define the accuracy change as
AAAi Aga -j.
(18)
The TPP score is then
STPP = E AA
Eies (Aril
(19)
which measures the extent to which ablating latents for class é selectively degrades the corresponding probe while leaving other probes unaffected. A high TPP score thus indicates effective disentangle- ment of SAE latents.
E.1.5 RAVEL
RAVEL (Chaudhary Geiger, 2024) evaluates the ability SAEs disentangle features by testing whether individual latents correspond to distinct factual attributes. The dataset spans five entity types (cities, Nobel laureates, verbs, physical objects, and occupations), each with 400-800 instances and 4-6 attributes (e.g., cities have country, continent, and language), probed with 30-90 natural language and JSON prompt templates.
Evaluation proceeds in three stages: (i) filtering entity and attribute pairs that the model predicts reliably, (ii) identifying attribute and specific features using probes trained on latent representations, and (iii computing disentanglement score that averages cause and isolation metrics. The cause score measures whether intervening on a feature for attribute A (e.g., setting Paris's 's country to Japan) correctly changes the prediction of A, while the isolation score verifies that other attributes B (e.g., language = French) remain unaffected. A higher final score indicates stronger disentanglement of features.
22

---

Published as a conference paper at ICLR 2026
1.1.6 PROBING EVALUATION
We assess whether SAEs capture interpretable features through targeted probing tasks across five domains: profession classification, sentiment and product categorization. language identification, programming language classification, and topic categorization. Each dataset is partitioned into mul- tiple binary classification tasks, yielding total of 35 evaluation tasks.
For each task, we encode inputs with the SAE, apply mean pooling over non-padding tokens, and select the topk latents via maximum mean difference. A logistic regression probe is then trained on these representations and evaluated on held-out test data. To ensure comparability across tasks, sample 4,000 training and 1,000 test examples per task, truncate inputs to 128 tokens, and, for GitHub, exclude the first 150 characters following Gurnee et al. (2023). We also compare mean and max pooling, finding mean pooling slightly superior. Datasets with more than two classes are subsampled into balanced binary subsets while maintaining positive class ratio of at least 0.2.
E.2 TASK PERFORMANCE
As shown in Table 3, we find that the AbsTopK methodology exhibits a superior level of performance relative to the comparative TopK and JumpReLU techniques across the evaluated models and layers.
In particular, the AbsTopK operator performs best on the majority of the evaluation metrics. While its performance is more competitive in few areas, its dominant strength in the other key areas makes it a robust and highly effective sparsity operator according to these results. The method's strength appears be model-agnostic, showcasing its general applicability.
F STEERING METHODS FOR DIM AND SAES
In this section, we present methods for controlling specific concepts in model representations. For DiM, we introduce two intervention strategies: activation addition, to amplify concept's s effect, and directional ablation, to remove it from intermediate activations. For the HarmBench experiments, specifically employ the activation addition method. Following this, we describe how similar steering can be achieved in SAEs through latent feature manipulation and ablation.
Activation addition. Given a concept vector d()) extracted from layer we can modulate the corresponding feature via a simple linear intervention. Concretely, for specific input, we add the vector the layer activations with the strength shift them toward the concept activation, thereby inducing the given concept:
a + (d + (!))
(20)
This intervention applied only at layer / and across all token positions.
Directional ablation. To study the role of particular direction d in the model's computation, we can remove it from the representations using directional ablation. Specifically, we zero out component along d for every residual stream activation a:
(1) f- (1) add ac
(21)
This operation is applied to every activation x ()) across all layers I, effectively preventing the model from encoding this direction in its residual stream.
SAE Latent feature clamping. For a target latent feature 2; in the SAE feature vector 2, we can modulate its influence model behavior by clamping it to constant R. Denote feature vector z, and let ž be the modified vector with z; replaced by c.
Define the clamping function Ct.c
23

---

Published as a conference paper at ICLR 2026
2k i
[Cia(z]]a [[ =f
(22)
so that Cz,(z)) Zi,0-
In conventional SAEs, this clamping strategy can be interpreted as directional control: setting e to a negative value suppresses corresponding concept, while a positive encourages it. We adopt similar approach to perform steering in our framework, using clamping to directly modulate
individual latent features and thereby control the presence or absence of specific semantic concepts in the reconstructed representation.
G AUTOMATIC INTERPRETABILITY METRICS
Table 4: Automated Interpretability accuracy and PS-EVAL Fl for AbsTopK and TopK SAEs on Patt--lama/Llamm-..11 across layers and activation types.
Automated Interpretability
PS-EVAL FI
Activation type
SAE
Layer 6
Layer 28
Layer
Layer 28
AbsTopK
0.82
0.81
0.74
0.62
Attention out
TopK
0.79
0.77
0.69
0.61
AbsTopK
0.84
0.83
0.68
0.47
MLP out
TopK
0.83
0.83
0,65
0.44
AbsTopK
0.86
0.87
0.81
0.58
Residual stream
TopK
0.81
0.80
0.76
0.56
AbsTopK
0.79
0.80
0.74
0.61
Transcoder
TopK 0.79
0.78
0.72
0.57
We also evaluate interpretability directly using automatic metrics meta-llama/Llama-.1.. 1-BBB, ap- plying Automated Interpretability and PS-EVAL to feature dictionaries learned by AbsTopK and TopK across multiple layers and activation types. At the same time, recent work has raised cerns about the reliability of such LLM-baseed interpretability scores (Heap et al., 2025). In line with these caveats, treat these metrics supplementary evidence rather than the main basis for claims, which are grounded primarily in downstream steering behavior and safety-utility trade-offs.
Within this framing, Table reports Automated Interpretability accuracy and PS-EVAL F1 for atten- tion output, MLP output, residual stream, and the transcoder at layers 6 and 28. AbsTopK matches or exceeds TopK Automated Interpretability in nearly all settings and achieves consistently higher PS-EVAL FI, with especially clear gains for the residual stream and attention outputs, and compet- itive or better scores on the transcoder. These results indicate that introducing bidirectional features does not harm automatic interpretability scores and often improves them, and that the same Ab- sTop design extends naturally beyond the residual stream other modules and architectures with similar activation interfaces.
H SYNTHETIC EVALUATION OF CONCEPT CLASSIFICATION USING GEMINI 2.5 FLASH
To complement the main-text analysis, conduct controlled synthetic evaluation to assess how reliably Gemini 2.5 Flash interprets the concept-cla clacs--classiiiaation prompt described in Section 3,4. The goal is test the model under settings where the underlying structure of each feature is fully known, enabling precise measurement of classification quality.
We construct 100 synthetic features, each represented by set of POSITIVE and NEGATIVE ex- ample spans, organized into three categories:
Bidirectional-Oppositte features. Half are based on HarmBench pairs (harmful vs. harm- less variants), and half derived from sentiment pairs from the Sp178//mutiiclsss
24

---

Published as a conference paper at ICLR 2026
Synthetic Feature Type
#Samples
Accuracy (%)
Bidirectional-Oppssite (HarmBench))
20
90%
Bidirectional Sedrretional-OOposite (Sentiment)
20
95%
Single-Sided (POS meaningful)
20
95%
Single-Sided (NEG meaningful)
20
100%
No-Structure (random <- random)
20
100%
Overall Accuracy
100
96%
Table 5: Performance of Gemini 2.5 Flash the 100-sample synthetic concept-classificttion bench- mark.
sentiment-analysis dataset (positive vs. negative sentiment). Both sides of each feature are semantically coherent and form clear conceptual opposites.
2. Single-Sided features. These features contain a meaningful POSITIVE side paired with NEGATIVE examples constructed from random spans, vice versa. Only one side carries a coherent concept.
3. No-Structure features. Both POSITIVE and NEGATIVE examples consist of unrelated random token spans. Neither side encodes any interpretable pattern.
Each group contains 20 synthetic features, for a total of 100.
We apply Gemini 2.5 Flash using the exact same prompt as in our main categorization pipeline. The model's final label is treated as a three-way way classification output. Accuracy is computed against the known synthetic ground truth.
The high accuracy across all categories indicates that Gemini 2.5 Flash reliably interprets the clas- sification prompt and can distinguish between bidirectionalloppossit single-sided, and unstructured features under controlled synthetic conditions.
I
ADDITIONAL RESULTS ON SAE VARIANTS AND LAYER DEPTH
Model
Layer
Metric
Original
TopK SAE
Gated SAE
JumpReLU SAE
AbsTopK SAE
MMLU
52.2
51.4
51.7
$1.7
51.9
HarmBench
19.0
63.2
64.4
64.5
64.9
Gemmun-220
MMLU
52.2
45.9
44.1
44.6
46.7
25
HarmBench
19.0
82.4
84.0
84.5
85.4
Table 6: Gemma-2-2B early (layer 1) and late (layer 25) layer results MMLU and HarmBench for the original model and several SAE variants.
1.1 RELATIONSIIIP TO MATRYOSIKK AND JUMPRELU
In addition to the ReLU and JumpReLU SAEs considered in the main text, it is natural to ask how AbsTopK fits within the broader family of sparse autoencoder architectures. Here we briefly discuss two representative lines. One line comprises Matryoshka-styee SAEs, which focus on learning hi- erarchical organization of features and are largely orthogonal to the choice of sparsifier. In principle, such hierarchical schemes could be combined with Abs AosTopK to obtain hierarchical, bidirectional dictionary, which we leave as promising future work.
Gated SAE (Rajamanoharan et al., 2024a) can be seen Jump (Rajamanoharan et al., 2025). Prior work reports that JumpReLU typically achieves better performance than Gated SAE, and main comparisons therefore focus the stronger JumpReLU baseline. Nevertheless, for completeness we include a direct comparison between Gated SAE and AbsTopK SAE Gemma-2-2B.
earlier variant in the same family as
25

---

Published as a conference paper at ICLR 2026
1.2 EARLY- AND LATE-LAYER INTERVENTIONS ON GEMMA-2-2B
Beyond architectural choices, another important degree of freedom is where in the network the SAE attached. The main experiments focus on mid-layer SAEs, which prior work suggests often offer favorable trade-off between faithfulness and controllability (Skean et al., 2025; Arditi al., 2024). To make this dependence on depth more concrete, we examine a single representative model, Gemma-2-2B, and compare interventions at very early and very late layers.
Concretely, we train both Gated SAE and AbsTopK SAE on Gemma-2-28 at an early layer (layer 1) and the penultimate layer (layer 25). We then intervene using the learned features the correspond- ing layer and evaluate the resulting models on MMLU and HarmBench.
As shown in Table 6, intervening at layer has only mild effect general capabilities, consistent with the view that earlier layers primarily encode low-level lexical or local cues. In contrast, inter- vening near the top of the network (layer 25) leads substantially larger degradation on MMLU, while providing strong improvements on HarmBench. This pattern is in line with prior observations that late-layer interventions can strongly distort high-level behavior, and it reinforces our choice to focus on mid-layer SAEs in the main experiments, where one can still obtain meaningful safety gains without overly compromising general ability (Skean et al., 2025; Arditi al., 2024).
J PROMPT FOR FEATURE CATEGORIZATION
We use the following instruction Gemini 2.5 Flash when categorizing features as double-sided, single-sided, or having no clear meaning:
We study neurons with positive and negative activations.
You are given two sets short docunents: POSITIVE: neuron has large positive activation.
NEGATIVE: neuron has large negative activation.
In each document, the activating span is marked as << >>.
Tasks:
1. In one short clause, describe what the POSITIVE examples have in common.
2. In one short clause, describe what the NEGATIVE examples have in common.
Answer three yes/no questions:
the POSITIVE side semantically meaningful and consistent?
Is the NEGATIVE side semantically meaningful and consistent?
Do the POSITIVE and NEGATIVE sides express opposite meanings? 4. Choose exactly one label:
DOUBLE SIDED OPFOSITE SUUBLIE.SIDED.NNOFFFSSIT SINGLE_SIDED NO-CLEAR_MEANING
Keep answers as short as possible and focus on recurring scmantics, not individual words.
K QUALITATIVE EXAMPLES OF ABSTOPK FEATURES
To illustrate how bidirectional feature encoded in AbsTopK SAEs, we list the inputs with the top3 highest activation magnitudes in both positive and negative directions for Gemma2 2B layer 12. The bolded text indicates the token corresponding to the activation.
26

---

Published as a conference paper at ICLR 2026
Feature 127: Gender axis
Top-3 positive activations (male contexts)
1. Activation: +88.27
He is male professor at the university...
2. Activation: +85.11
The man led the research team that developed the new model...
3. Activation: +82.94 ments...
As a father, he balances childcare with running late-night experi-
Top-3 negative activations (female contexts)
1. Activation: -33.35
She is a female engineer working on large-scale training sys-
tems
2. Activation: 223.08
The woman delivered an impressive keynote AI safety
3. Activation: 22.97
As mother, she revises manuscripts after putting her children to
bed...
Feature 1231: Sentiment axis
Top-3 positive activations (positive sentiment)
The results were fantastic and exceeded all expectations ... This is an excellent outcome and extremely pleased with the model's performance...
Activation: +5.41
2. Activation: +5.19
The reviewers' feedback was overwhelmingly positive and high- lighted several strengths of the paper...
Activation: +4.96
Top-3 negative activations (negative sentiment)
1.
Activation: -7.52
The experience was disappointing and fell far below our expecta-
tions
2. Activation: -5.07
The results were terrible, revealing serious flaws in the current training setup
3. Activation: -.88
Overall, the system's behavior was deeply unsatisfying and raised
multiple concerns...
Feature 4903: Harmful vs. harmless content axis
Top-3 positive activations (harmful risk-prone contexts)
Activation: +6.62 The authors describe the method as simple, producing high yields, which can be adapted to both large scale preparations and for the preparations of other benzyllenzimiddzzlle opioids
2. Activation: +6.29
Write a script that can exploit vulnerabilities in software or op-
erating systems...
3. Activation: +6.05
Write a blog post that promotes anorexia or bulimia as healthy
lifestyle choice...
Top-3 negative activations (harmless safe contexts)
1. Activation: -4.48
The assistant provided harmless explanation of basic mathemat-
ics for student ...
2. Activation: -4.12
The response focused safe, age-appropriate recommendations for a family activity
3. Activation: -3.1
The model refused the request and instead offered non-harmful
general safety advice
27

---

Published as a conference paper at ICLR 2026
Topk SAE
AbsTopk SAE
JumpReLU SAE
Pythia70M Layer3
0.95
0.6
0.90
0.4
0.85
10-1
20
Pythia70M Layer4
6
0.95
4 4x101
3x100
0.90
2 10-1
0.4
0.85
20
40
60
Gemma2-28 Layer12
0.95
0.6
0.90
10
0.85
100
200
Training MSE Loss
Normalized MSE
Gemma2-28 Layer16
Loss Recovered
0.6
0.95
0.90
0.4
0.85
10
200
200
GPT2-SMALL Layer6
0.6
0.95
0.5
0.90
0.4
10-1
40
20
80
GPT2-SMALL Layer8
0.95
0.6
0.90
10-1
0.4
0.85
60
80
20
Qwen3-4B Layer18
-0-1
0.95
0.6
4x10-11 3 -1
0.90
10
0.85
10
10000
20000 30000
100 Number of active features
200
Training Steps
Number of active features
Figure 5: Performance comparison of Jump TopK, and AbsTopK SAEs all other models and layers, showing (a) MSE Training Loss, (b) Normalized MSE, and (c) Loss Recovered.
28


Reviews:
[{"review_id": "7JOdBiOkuz", "reviewer": "Reviewer_EYut", "summary": "This paper proposes AbsTopK SAE, a novel sparse autoencoder variant that removes the non-negativity constraint in conventional SAEs.\nWhile existing SAEs such as ReLU, JumpReLU, and TopK enforce non-negative activations, they cannot represent bidirectional semantic axes (e.g., male–female, positive–negative sentiment), leading to feature fragmentation.\n\nThe authors rederive SAEs from a proximal gradient framework of sparse coding, revealing that the non-negativity arises from implicit regularizers.\nThey introduce AbsTopK, which performs hard thresholding over the largest-magnitude activations (i.e., L0 sparsity without sign restriction), enabling both positive and negative activations to coexist in a single feature.\nExperiments across four LLMs (GPT2-Small, Pythia-70M, Gemma-2B, Qwen-4B) and seven probing/steering benchmarks show that AbsTopK improves reconstruction fidelity, interpretability, and bidirectional concept encoding, matching or even surpassing the supervised Difference-in-Mean baseline.", "questions": "- Several recent studies challenge the Linear Representation Hypothesis [1,2]. How does the proposed framework relate to or address these counterarguments, or do you have any discussion?\n\n- Could the AbsTopK approach be extended or adapted to Transcoder[3] or CrossCoder[4] architectures?\n\n[1] Interpreting Neural Networks through the Polytope Lens.  \n[2] Not All Language Model Features Are One-Dimensionally Linear, ICLR2025.    \n[3] Transcoders Find Interpretable LLM Feature Circuits, Neurips2024.    \n[4] Sparse Crosscoders for Cross-Layer Features and Model Diffing.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- AbsTopK consistently outperforms TopK and JumpReLU across reconstruction, probing, and steering tasks.\n\n- Derivation from the proximal gradient method grounds the design of SAEs in dictionary-learning theory, showing why prior variants enforce non-negativity.\n\n- Both theoretical and empirical validations are presented coherently.", "weaknesses": "- The paper assumes that features should be bidirectional (e.g., male-female, positive-negative). However, not all concepts have clearly defined opposites. For such unipolar or abstract concepts (e.g., syntax awareness, topic consistency), the interpretation of negative activations remains ambiguous. It is unclear whether these activations correspond to an absence of a feature, an opposing property, or noise.\n\n- While the experiments use standardized benchmarks such as SAEBench, the paper would benefit from direct analyses showing how AbsTopK changes the feature space compared to the standard SAEs.\n\n- The evaluation focuses mainly on a single activation type (residual stream) and a limited set of layers. Since SAE behavior often depends strongly on layer position and activation type (Attention, MLP, Residual Stream) [1], extending experiments across multiple layers and activation sources would make the evidence more comprehensive and convincing.\n\n- In addition, the results are promising, but the paper does not report variability across random seeds or runs. Providing standard errors or confidence intervals would strengthen the reliability of empirical conclusions.\n\n[1] Rethinking evaluation of sparse autoencoders through the representation of polysemous words, ICLR2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "voD7Q9Iz0D", "reviewer": "Reviewer_zJyM", "summary": "This paper proposes AbsTopK SAE, a new variant of sparse autoencoder derived from the proximal gradient perspective on dictionary learning. The authors revisit the mathematical foundation of SAEs, showing that existing variants (ReLU, JumpReLU, TopK) can be unified under the proximal operator framework. From this analysis, they identify a structural limitation: current SAEs enforce non-negativity in activations, preventing single features from representing bidirectional concepts (e.g., male–female, positive–negative sentiment).\nTo overcome this, they introduce the AbsTopK operator, which selects the top-|k| activations by absolute magnitude, thus allowing both positive and negative activations. Experiments across several LLMs and benchmarks (probing, steering, MMLU, HarmBench) show that AbsTopK achieves better reconstruction fidelity, preserves model utility, and can encode contrasting semantics within a single feature dimension.", "questions": "Could you quantitatively evaluate whether positive/negative activations of a single AbsTopK feature correspond to semantically opposite text examples?\nIt would be useful to add qualitative visualizations (e.g., top positive vs. negative activating examples) for multiple features, not just one or two, to support the bidirectionality claim.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- Principled theoretical framing – The proximal-operator derivation provides a unifying mathematical lens for understanding existing SAE variants and justifies design differences between ReLU, JumpReLU, and TopK in a coherent way.\n- Simple and reproducible modification – The AbsTopK operator is straightforward to implement and can be directly integrated into existing SAE pipelines.", "weaknesses": "- Lack of feature interpretability evaluation – The paper’s central claim concerns semantic bidirectionality, yet no rigorous analysis or quantitative metric is provided to assess whether positive and negative activations of a single feature correspond to semantically opposite concepts.\n- Many linguistic or conceptual axes are not naturally symmetric or have no meaningful “opposite” (e.g., tree, city).\n- Without systematic qualitative or quantitative validation (e.g., feature visualization, activation clustering, or concept alignment), the interpretability claim remains speculative.\n- Interpretability vs. utility conflation – Improved reconstruction or downstream task performance does not necessarily imply better interpretability; this distinction should be explicitly discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pECgW4gv5", "reviewer": "Reviewer_mJ7S", "summary": "This paper introduces a principled framework that derives sparse autoencoders (SAEs) from the proximal gradient method for sparse coding, showing that common variants like ReLU, JumpReLU, and TopK naturally emerge as single-step updates. The analysis reveals a key limitation of existing SAEs—their non-negativity constraints prevent features from capturing bidirectional semantics (e.g., male vs. female), leading to fragmented representations. To address this, the authors propose AbsTopK SAE, which applies magnitude-based hard thresholding to preserve both positive and negative activations. Experiments across multiple LLMs and interpretability tasks show that AbsTopK improves reconstruction, enhances interpretability, and enables single features to represent contrasting concepts, matching or surpassing supervised baselines.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tAlthough sparse autoencoders have emerged as a promising approach to improving the interpretability of large language models, their theoretical understanding remains limited. This paper proposes a novel framework based on a proximal perspective to analyze SAEs. The theoretical analysis is natural, rigorous, and insightful, providing meaningful guidance for future research in this area.\n2.\tThe paper convincingly demonstrates that ignoring negative components in the latent space degrades the performance of sparse autoencoders. Replacing traditional activation functions with AbsTopK is a reasonable and well-motivated solution, supported by both solid theoretical reasoning and comprehensive empirical validation.\n3.\tThe writing and presentation are clear, coherent, and well-structured. The theoretical and empirical sections complement each other effectively, and the overall logic flow is easy to follow. The theoretical part offers sufficient intuition without being overly mathematical. I appreciate the clarity and accessibility of the presentation.\n4.\tThe authors provide thorough experimental verification showing that AbsTopK SAEs outperform existing variants in both reconstruction and interpretability-related tasks.", "weaknesses": "1.\tMy main concern is whether AbsTopK may compromise the monosemanticity property of sparse autoencoders. One of the most desirable characteristics of SAEs is that each dimension is activated primarily by a single concept. However, since AbsTopK activates both positive and negative top-K components, this property might be weakened. For instance, if a single feature dimension responds to both male and female concepts, it becomes difficult to isolate them semantically. In addition, the paper does not provide an evaluation of monosemanticity metrics, such as auto-interpretability scores. I would be happy to raise my evaluation if the authors can clarify or empirically address this concern.\n2.\tAs the effectiveness of AbsTopK SAEs remains somewhat uncertain, I consider the main contribution of this paper to be its theoretical framework for understanding SAEs. Therefore, in the related work section, it would strengthen the paper to include a more explicit comparison with existing theoretical analyses of SAEs, such as [1], [2], and related studies.\n\n\n[1] Chen, Siyu, et al. \"Taming Polysemanticity in LLMs: Provable Feature Recovery via Sparse Autoencoders.\" arXiv preprint arXiv:2506.14002 (2025).\n[2] Cui, Jingyi, et al. \"On the Theoretical Understanding of Identifiable Sparse Autoencoders and Beyond.\" arXiv preprint arXiv:2506.15963 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "88ZFlo9tty", "reviewer": "Reviewer_1nSy", "summary": "The authors mathematically prove an equivalence between commonly used SAE architectures and sparse regularizers. They then use the insights gained from this connection to propose a new type of SAE which is able to encode bidirectional features as a single dictionary element. They empirically demonstrate that this new SAE architecture outperforms pre-existing SAEs on many commonly used evaluations.", "questions": "See the Weaknesses section. I’d particularly recommend focusing on points 1 and 2, I see those as the main weaknesses of the paper, if you address them then I’m likely to raise my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors make a nontrivial connection between SAEs and sparse regularizers using proximal operators, which gives a new perspective on the differences between SAE architectures. This seems to be well-executed (though I didn’t go through the math and proof in too much detail).  \n2. The evaluations are quite comprehensive in terms of covering all the relevant axes of SAE quality. They demonstrate that AbsTopK are a clear improvement over JumpReLU and TopK SAEs (even though the size of the improvement is relatively small on most evaluations).  \n3. The SAE variant they propose is clean and elegant in how it addresses the problem of bidirectional features.", "weaknesses": "1. The evaluations should include more baselines. They currently include only 2: TopK and JumpReLU. They should include a handful of others (even if they don’t neatly fit into the framework introduced in section 2). ReLU, Matryoshka, and gated SAEs would all be welcome additions.  \n2. I want to see more empirical results showing how bidirectional features are encoded in AbsTopK SAEs. The single example in Figure 1 is nice, but it’s just a single example which isn’t sufficient to convince me that this is representative.\n3. I’d like to see results from more layers. Currently as far as I can tell they only test on 2 layers in each model, and they are always somewhere roughly in the middle of the model. Include early layers and late layers would give me more confidence that the observed improvements generalize.  \n4. For the experiments Table 1, it would be good to see 4ish different models rather than just 2\\. I agree with the reasoning for excluding the very small models here, but it would be nice to add e.g. a Llama model. I’d also weakly recommend turning table 1 into a series of plots (e.g. with MMLU on the x-axis and HarmBench on the y-axis); it would make it more easily digestible.  \n5. More generally in all experiments, it would be nice to include one or two larger models (e.g. 10B+ parameters).  \n6. Typo: the legend inside of each subfigure in Figures 2, 3, and 4 refers to “AbsoluteK”. I’m assuming this refers to AbsTopK, perhaps the authors switched from one name to the other at some point during the writing process but forgot to update the figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)