Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
Authors: Sai Sumedh R. Hindupur
Ekdeep Singh Lubana
Thomas Fel
Demba E. Ba
Venue: NeurIPS 2025 poster ()
Keywords: sparse autoencoders
dictionary learning
interpretability
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We show that Sparse Autoencoders (SAEs) are inherently biased toward detecting only a subset of concepts in model activations shaped by their internal assumptions, highlighting the need for concept geometry-aware design of novel SAE architectures.

Abstract: Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations. However, do SAEs truly uncover all concepts a model relies on, or are they inherently biased toward certain kinds of concepts? We introduce a unified framework that recasts SAEs as solutions to a bilevel optimization problem, revealing a fundamental challenge: each SAE imposes structural assumptions about how concepts are encoded in model representations, which in turn shapes what it can and cannot detect. This means different SAEs are not interchangeable -- switching architectures can expose entirely new concepts or obscure existing ones. To systematically probe this effect, we evaluate SAEs across a spectrum of settings: from controlled toy models that isolate key variables, to semi-synthetic experiments on real model activations and finally to large-scale, naturalistic datasets. Across this progression, we examine two fundamental properties that real-world concepts often exhibit: heterogeneity in intrinsic dimensionality (some concepts are inherently low-dimensional, others are not) and nonlinear separability. We show that SAEs fail to recover concepts when these properties are ignored, and we design a new SAE that explicitly incorporates both, enabling the discovery of previously hidden concepts and reinforcing our theoretical insights. Our findings challenge the idea of a universal SAE and underscores the need for architecture-specific choices in model interpretability. Overall, we argue an SAE does not just reveal concepts -- it determines what can be seen at all.

Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
Sai Sumedh R. Hindupur!: Ekdeep Singh Lubana 23,,A Thomas Fel and Demba Ba
School of Engineering and Applied Science, Harvard University CCBSNTT Program in Physics of Intelligence, Harvard University
Phhysics of Artificial Intelligence Group, NTT Research, Inc., Sunnyvale, CA, USA *Kempner Institute, Harvard University
Abstract
Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations. However, do SAEs truly uncover all concepts a model relies on, are they inherently biased toward certain kinds of concepts? We introduce a unified framework that recasts SAEs as solutions to a bilevel optimization problem, revealing fundamental challenge: each SAE imposes structural assumptions about how concepts are encoded model representations, which in turn shapes what it can and cannot detect. This means different SAEs are not searchangeable-ssiihhiie entirely new concepts or obscure existing ones. To systematically probe this effect, we evaluate SAEs across a spectrum of settings: from controlled toy models that isolate key variables, to semi-synthetic experiments finally to large-scale, naturalistic datasets. Across this progression, we examine fundamental properties that real-world concepts often exhibit: heterogeneity in intrinsic dimensionality (some concepts are inherently www-dmmnsionnl,, others are not) and nonlinear separability. We show that SAEs fail recover concepts when these properties are ignored, and we design new SAE that explicitly incorporates both, enabling the discovery of previously hidden concepts and reinforcing theoretical insights. Our findings challenge the idea of a universal SAE and underscores the need for architecture-specffic choices in model interpretability.
architectures can expose
real model activations and
Introduction
Interpretability has become important research agenda for assuring, debugging, and controlling neural networks [1-5]. To this end, sparse dictionary learning methods [6-10], especially Sparse Autoencoders (SAEs), have seen resurgence in literature, since they offer an unsupervised pipeline for simultancously enumerating all concepts a model may rely on for making its predictions [11-18]. Specifically, an SAE decomposes representations into an overcomplete set of latents that (ideally) correspond to abstract, data data-centric concepts which, upon aggregation, explain away the model representations [19, 20). In other words, an SAE is expected result in monosemantic latents which are more interpretable than the neurons of the original model [21]. For example, SAE latents derived from models in diverse domains have been demonstrated to activate for meaningful concepts such as specific monuments, behaviors, and scripts in language [22, 23]; specific objects, people, and scene properties in vision [15, 24]; and correlate with binding sites and functional motifs in protein autoregressive models [25-27].
To the extent the concepts uncovered using SAEs faithfully represent the concepts used by model for making its predictions, we can use this information to perform surgical interventions on a model's representations and hence achieve control over its behavior [23, 28, 29]. While this forms bulk of the motivation around research in SAEs [12. 30, 31]. we argue the theoretical foundations that suggest
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A)
B)
Linear Separability ReLU, JumpReLU
Dictionary
Learning
SAEs
Angular Separation Topk
t Constraints
Better Assumptions Noeded?
Non-Linear Separation Concept Heterogeneity
Data Assumptions
Concepts
The Duality Between SAEs Architectures and Their Implicit Data Assumptions. A) SAEs do not passively extract concepts they impose constraints that shape what can be detected. Each SAE architecture inherently assumes a specific structure in how features are encoded, leading corresponding dual assumption about the data. B) Different SAEs rely on different assumptions: some expect features to be linearly separable (ReLU, JumpRel.U) or separable by angle while having uniform intrinsic dimensionality (TopK). These assumptions dictate what an SAE can successfully extract-and what it may miss entirely.
Figure 1:
SAEs optimal tool for achieving this goal are lacking. For example, is it possible that instead of uncovering all concepts a model utilizes in its computation, SAEs are biased towards identifying only specific, narrower subset of concepts? Furthermore, is it possible that different SABs, which generally achieve similar fidelity/sparsity, have qualitatively different biases and hence uncover different concepts from model representations? An affirmative answer to these questions may explain recent negative results on SAEs, e.g., algorithmic instability [15, 32] and lack of causality [33, 34]. Motivated by this, we make the following contributions in this work.
Duality between Concepts' Organization and the Optimal SAE that Identifies Them. We formulate SAEs solutions to specific bilevel optimization problem, which highlights fundamental duality between concept structure model representations and an SAE encoder's receptive fields (formalized in Def. 3.2). Crucially, this implies any SAE is implicitly biased towards identifying concepts that are organized
specific manner (Fig. 1).
Empirical Validation via Concepts that do not Follow SAEs' Implicitly Assumed Organiza- tion. We evaluate SAES on concepts with heterogeneous intrinsic dimensionality (i.c., different concepts occupy subspaces of varying dimension) and nonlinear separability through experiments on controlled synthetic setups to real-world model activations, demonstrating that SAEs failing to account for these properties systematically miss the corresponding concepts.
Methodology for Designing Task-Specific SAFs. Our results suggest no single SAE archi- tecture may be universally optimal, and hence SAEs should be designed by accounting for how concepts are encoded in model representations. To validate this, we introduce SpaDE, novel SAE that explicitly incorporates heterogeneity and nonlinear separability into its encoder. As show, SpaDE successfully identifies concepts that other SAEs fail detect, reinforcing the need for data-aware choices in interpretabilityy
2 Preliminaries
Notation. We denote vectors as lowercase bold (e.g., æ)) and matrices as uppercase bold (e.g., X). [n] denotes {I..... n) and B {x ||0|| 1} the unit ball in Rdd We assume access to dataset of l samples, X = {æ1..... ) where E Rdd, For any matrix X or vector æ, we use X 2 0 (resp. æ 2 0) to indicate element-wise non-negativity.
Sparse Coding. Also known Sparse Dictionary Learning [10, 35], sparse coding assumes generative model of data as a sparse combination of latents. Specifically, sparse coding involves solving the following optimization problem:
arg min
Dz112 *R(22,
DE0
where E RR* is sparse latent code, D E Rdxx are the dictionary atoms, and R(z) is a sparsity- promoting regularizer, typically ||=||- Note that the optimization performed over both the sparse code (with > 0) and the dictionary D. Further details are included in Appendix A.

---

A)
Data
Encoder
Decoder
B)
RoLU
JumpReLU ReLU
Topk
Projection
Set S
Poxiiiv
Hypercabbe
K-dim
Unnar Trancformation Projection
Uiner Transformaiion
ortant
conners
subspaicss
Figure 2: Projection As The Key Architectural Difference Between SAEs. A) SAE encoders do more than just linearly transform data they project onto unchtecture-spectiee constraint set. This projection fundamentally determines which features an SAE can extract and which will suppress. B) Different SAEs rely on different projection S: ReLU projects onto the positive orthant, TopK onto K Spsres subspaces, and JumpReLU combines ReLU with a projection onto hypercube (via Heaviside step function).
Sparse Autoencoders. SAEs [36] approximate sparse dictionary learning by using a single hidden layer to compute the sparse code from data:
(i = f(æ)= gWW be), and (ii) à Dz bd:
where W, D E Idxx and RR + R2 is the encoder non-linearity. Here, sparsity is enforced on the SAE latent code z. SAEs trained the sparse dictionary learning loss (Eq. 1), with the
regularizer R. Different SAEs typically differ in the choice encoder nonlinearity g and the regularizer R, as discussed in our unified framework next.
Unified Framework for SAES
In this section, we develop framework which eaptures multiple SAEs used in prac- tice. More specifically, we analyze the following three popular SAE architectures: ReLU SAE [11, 12), TopK SAE [13, 37) and Jump SAE [14, 38J. This frame- work unravels duality between how cepts are encoded in model representations and SAE's architecture. The nonlinear- ity of the SAFs under study an orthog- onal projection onto some set, where the choice of projection set differentiates SAES (see Fig. 2). We formalize such nonlinearities Definition 3.1 (Projection Nonlincarity). Let E Rº be pre-activation vector. A projection
Table I: Projection Nonlinearities in SAE Encoders. Each model can be understood by its nonlinear orthog- onal projection g(-) onto constraint set S which de- termines its activation behavior, sparsity structure, and implicit data assumptions.
Model
g(v)
ReLU
ILs (u). (y
TopK
Ils (v), (yE 1 k}
Hcaviside (H) ITs 0)
JumpReLU
projection nonlinearities, defined below.
nonlinearity IIs (-} E= + E= is defined as:
II (v) arg min || - 2ll
wES
where S C R the constraint set onto which is orthogonally projected. Popular SAE nonlinearities, e.g., ReLU, JumpRel.U, and TopK, are orthogonal projections onto different sets (see Tab. 1).
Generalizing the variational form of projection nonlinearities allows us formalize SAEs follows. Claim 3.1 (Bilevel optimization of SAEs). A sparse autoencoder (Eq. 2) with the dictionary learning loss function (Eq. 1) solves the following bi-level optimizution problem:
arg min
Σ || Dzl XR(2)
DEB.220
(4)
fl (xx arg min F(T,, wxx),
*ES
where F is variational formulation of the SAE encoder For SAEs, f(x) T b..) (Eq. 2). Note that this inner optimization with the objective F is what differentiates different SAEs.
Enccddr bias be is used in the TopK SAE [13].

---

Proof. The outer optimization follows from the dictionary learning loss with penalty of the SAE (Eq. 1). The constraint imposed by the SAE encoder's architecture (Eq. 2). The variational formulation of the encoder as the minimization of some objective F over set S is generalization projection nonlinearities (Eq. 3) for which F(T,, W, x) = W +bb. π 112-- L
This framework implies that each SAE solves differenn, constrained (through encoder architecture) optimization version of sparse dictionary learning. This constraint dictates the quality of the solution obtained, since it restricts the search space of solutions to dictionary learning, and hence does not have to capture the full sparse coding solution. To further formalize this claim in the next section, we now define receptive fields, a popularly used concept in neuroscience to study the response properties of biological neurons [35]. We use the term neuron to define receptive fields in line with the inspiration from neuroscience, but they refer to neurons of SAEs (SAE latents) in subsequent analysis.
Definition 3.2 (Receptive Field). Consider a neuron k, which computes a function f(k) R R.
The receptive field this neuron is defined as Fx = { F ((()()) > 0}.
Intuitively, Fs represents the region of input space where neuron & active. The structure of receptive fields in an SAE is dictated by its encoder's architecture.
Duality: Properties of the SAE encoder will constrain receptive fields' structure for SAE latents. These constraints directly translate to assumptions (often implicit, see Sec. 4) about the data structure, since "monosemanticity" [12, 21] requires receptive fields to match structure of concepts in data. Alternatively, if one knows how concepts are organized in the data (model representations), duality can be used design appropriate SAE architecture (see Sec. 4.1).
Fundamental Limitation of SAEs
An SAE's encoder enforces implicit dual assumptions about data, fundamentally shaping which concepts it can identify and which remain obscure. To build more effective SAEs, these assumptions must explicitly match the true structure of the data.
Implicit SAE Assumptions and Data Properties
In this section, we explicitly state the data assumptions made by ReLU, TopK and JumpReLU SAEs. Theorem 4.1 (Implicit Assumptions; Informal). An SAE makes implicit assumptions about the structure of concepts in data, reflecting it in the receptive fields of its encoder. These assumptions are explicitly stated in Tab. 2 for ReLU, JumpReLU and TopK SAEs (derived in App. D.2).
The optimality of the above assumptions depends on the "true structure" of concepts in model representations. By "true struc- ture" of concepts, we refer to the ground truth structure in accordance with which concepts are organized in model's activations. While concept structure is not known in its en- tirety, highlight properties of how (certain) concepts organized in a model based on recent interpretability literature.
Nonlinear Separability
b) Concept Hoterogensity
Nonlinear separability of concepts. Concepts are not sep- arable by linear decision boundaries. Evidence towards such concepts include features with dependence on mag- nitude, such onion features [39]. Even "linear features" 40), having different magnitudes may fail to linearly separable (Fig. 3).
Figure 3: Illustration of Two Reasonable Data Assumptions. A) Concepts may not be sep- arable using hyperplanes. Some concepts are inherently low- dimensional, while others span higher-dimensional spaces.
B)
2. Heterogeneity of concepts. Different concepts belong to subspaces with different dimensions. Evidence for this property includes unidimensional features representable as concept activation vectors [19], e.g., truth [41], multidimensional features such as days of the week [43]. Here, higher dimensional concepts may be compositions of atomic (one-dimensional) concepts (such as safety features composed of "refusal behavior". "hypothetical narrative". and "role-playing" [43]).
2-D subspace [42], and higher dimensional safety-relevant features

---

Table Implicit Assumptions of SAEs.
The receptive fields of SAEs implicitly assume concepts are organized with a specific structure in the data, i.e. in model representations.
Model
Receptive Field
Data Assumption
ReLU
half-spaces
Linear separability of concepts
JumpReLU
half-spaces
Linear separability of concepts
TopK
union of hyperpyramids
Angular separabilityy of concepts: same dimensionality per concept
We characterize the compatibility of different SAEs' implicit assumptions and these properties in Tab. 3. Note that ReLU and JumpReLU can potentially capture heterogeneity since they can show different sparsity levels for each concept, but they require linear separability concepts due to half-space receptive fields. TTopK may be able to handle nonlinear separability to some extent (provided concepts separable by angle). but it cannot adapt heterogeneous concepts, since it involves a fixed choice of sparsity level for all inputs. BatchTopK [16], modification of TopK which selects average sparsity level per batch, does not capture concept heterogeneity either, since it still requires choosing the average sparsity level K. To enable evaluation of our claims, we next design SAE that accommodates the two properties above into its architecture, presented in the following subsection.
Table 3: Compatibility of SAEs with nonlinear separability and heterogeneity.
Model
Nonlinear Sep.
Heterogeneity
ReLU
X
JumpReLU TopK
X
4.1 SpaDE, or How Design Geometry-Driven SAE
We now use the data properties studied above nonliner separability and concept heterogeneity- and through the duality, construct one set of sufficient conditions on the SAE to capture both properties, resulting in novel SAE called SpaDE (Sparsemax Distance Encoder). See App. D.4 for details. We introduce SpaDE as a geometry-driven SAE to validate our claims about the duality between concept geometry and SAE architecture. Hence, SpaDE is expected to capture concepts better than other SAES when its data assumptions are met.
Nonlinear separability can be captured by SAE encoders with competitive projection nonlinearity (allowing flexible receptive fields, shaped by locations of all weights) and compute Euclidean distances to set of prototypes instead of linear transforms (to better exploit magnitude). For concept heterogeneiiy, SAEs must demonstrate adaptive sparsity their latent representations, i.e., different concepts must be able to activate different numbers of latents (Fig. 4).
To satisfy the desiderata above, we use simple first-order equality constraint on the projection set S (Eq. 4), resulting in the probability simplex S A {ær E R Σ, 24
SpaDE
Projection
1, x 2 0}. The non-negativity is necessary to explain away data as a combination of features with positive contributions. Projection onto the simplex (see Fig. 4) results in the sparsemax nonlinearity ([44)):
Probebiiity simplex
Sparsemax(r) arg min || v113 TEA*
Figure 4: SpaDE shows adaptive sparsity by projecting onto the probability simplex. In this illus- trative 3D figure, note ||0||o = for points on the face, | 2 for points on edges along subspaces, and ||æ||o = for corners on ordinate axes.
The probability simplex A admits representations with any (non-zero) level of sparsity, as illustrated in Fig. 4. Combining Sparsemax with euclidean distances then yields SpaDE:
z f(x Sparsemax(-Adiies W)),
(5)
where d(æ W)); = | Wil..
In the above, À is scaling parameter (akin to inverse temperature), while W, is the 7th column of the encoder matrix W which behaves as a prototype (or landmark) in input space since we compute euclidean distance from input æ to W. App. D.4 and D.2.3 describe the receptive fields of SpaDE in further detail and show how it captures nonlinear separability and concept heterogeneity.
5

---

ReLU
JumpReLU
TopK
SpaDE
(a)
10-
2.
0.6
6.
044
10
16
76
10
25
(b)
(c)
505000
Figure 5: Effect of Nonlinear Separability on SAEs. Each column represents a different SAF, a) Fi scores of the top 5 most monosemantic latents (highest Fl scores), where shaded region is +SSD of (linearly separable) and purple (non-linearly separable). SAEs
each SAF on two concepts- that assume linear separability struggle capture the nonlinearly separable concept. b) Receptive fields of the most monosemantic latent for each SAE, illustrating how some architectures fail to isolate the nonlinear concept cleanly. Intensity of color indicates strength of SAE latent activation. (c) Matrix of pairwise cosine similarities between sparse codes of different datapoints, and data clusters obtained through spectral clustering on this matrix. In the scatter plot, points colored by the same color belong to one spectral cluster, which intuitively indicates that they activate common set of SAE latents. SpaDE. is able to maintain clear concept boundaries and doesn't 't mix distinct features, while other SAEs group subsets of different features into the same spectral cluster (same color).
We also note that the outer optimization for SpaDE is K-Deep Simplex (KDS, [45]), a modified dictionary learning technique which incorporates locality into sparse representations. The regularizer from KDS is a distance-we distance-weighted & regularizer R(z) = Σ, >|| W ll2, which encourages prototypes to move closer to data when they are active, increasing sparsity of representation 2, The inner optimization for SpaDE is a one-sided sparsity- regulatyre optimal transport (see App. D.4).
Our claim about the duality between SAE architectures and data assumptions about concepts also applies to SpaDE. Beyond nonlinear separability and concept heterogeneity. SpaDE implicitly assumes that Euclidean distances are useful in concept distances can be used to disentangle concepts.
distance-seppaatted--nn
Results: Empirical Validation of SAE behavior
We perform suite of experiments which involve training ReLU, JumpReLU, TopK and SpaDE SAEs on synthetic Gaussian clusters, semi-synthetic formal-language model activations and natural vision model activations. Our synthetic experiments aim validate our claims about implicit assumptions in SAEs. Experiments on more naturalistic data seek to demonstrate our claims extend to realistic data settings. Further analysis is deferred to App. The code to replicate synthetic experiments formal language and vision
available at: https: //gtthubbbhtb//t// com/Sai Sata-SSumedie Swweehuuehheeeeeeeee experiments at: https: //gi h/g.thubb.ttttte/// thub. com wwtp:///deeelllliaao/n/dooeeaaaa experiments is at: https:// thub. com cowp/Keempeerrnnttttttteeeeeeere
5.1 Separability Experiment
Dataset and Experiment: We construct 2-dimensional dataset with Gaussian clusters (abstraction of concepts) of different magnitudes in order to demonstrate nonlinear separability of concepts in
"This regularizer encourages dictionary atoms to "stick" to the data, addressing the recently raised concern [15, 32] that directions learned by SAEs may -utttf--isttibuttinn (OOD), contributing their instability.

---

ReLU
JumpReLU
Topk
SpaDE
(a)
126
14
16
12
126
(b)
107
Normalzzed MSE
30"1
(c)
Ly0255
106
220
AAAA
Figure 6: Effect of Concept Heterogeneity SAEs. a) Per-coneept sparsity function of intrinsic dimension. Colors indicate per-concept MSE-higher errors (red/yellow) show when an SAE fails to capture concept effectively. Each solid line indicates one model with specific choice of hyperparameters. b) Normalized MSE vs. per-concept sparsity. well-performing SAE should maintain low error across all concepts. TopK SAE only achieves good reconstruction (below the dashed 20% error threshold) when sparsity (fixed for a given model) exceeds intrinsic dimensionality. highlighting its lack of flexibility. e) Cosine similarity between pairs of SAE latents across all concepts (showing co-occurrence), visualized for two sparsity levels.
simple setting which facilitates visualization. Here, each cluster is defined as its own concept, and expect SAES learn latents responding to individual clusters. The concepts with smaller norm are not linearly separable, while those with larger norm are linearly separable. We train all SAEs on this dataset for range of sparsity levels. Following our arguments about implicit assumptions in SAEs, we hypothesize that ReLU and JumpReLU will be unable to capture the nonlinearly separable concepts with monosemantic latents (measured using Fl scores; see Eq. 9).
Fig. 5 shows how different SAEs fare on this experiment. ReLU and JumpReLU achieve an Fl score of 1 for the separable concept (orange), while their F1 scores much lower and bounded above (by 0.5) for the nonlinearly separable concept (purple). The receptive fields of ReLU, JumpReLUU in Row (b) clearly overlap with other concepts in the nonlinearly separable case. TopK performs somewhat poorly on both concepts, showing comparable F1 scores in both cases. SpaDE shows a top F1 score of 1.0 for both concepts (perfect precision and recall), with its receptive fields capturing concept structure even for nonlinearly separable concepts. While ReLU and JumpReLU show significant cross-concept correlations (between concepts C₁, C2, C3, Row (c)) and TopK does marginally better with smaller correlations, SpaDE shows clear delineation of different concepts with clearly separated concepts (no cross correlations, spectral clustering identifies concepts). Note SpaDE may overspecialize and lead to further subclusters, as seen by two colors within concept 1 in row (c).
Observations:
5.2 Heterogeneity Experiment
We generate Gaussian clusters (again an abstraction for concepts) in 228-dimensional space. The five concepts are heterogeneous they belong to subspaces with different intrinsic dimensions (6, 14, 30, 62, 126), but are designed to have isotropic structure within each cluster, and similar total variances across clusters. We trained ReLU, JumpRel.U, TopK SAEs and SpaDE on this data with varying sparsity levels. We hypothesize that TopK will not be able adapt its representations to the intrinsic dimension of each cluster.
Dataset and Experiment:
7

---

(a)
ReLU
JumpReLU
TopK
SpaDE
Nour
Pro
Adv
leer
ské
Aqj
Comp
(b)
PCE
PC2 17.1%
PC2:17
(c)
(Pro)Noun
Verb
Adv.
Adj.
Conj.
Nomalize MSE
10a
33%
90%6
50%
32%
X
12
Bell
JamprtaLl
lick
Sp4DE
(d) 100
5core
u.7b
41
025
0.00
10P
10a
Figure 7: Investigating SAE properties on GPT for formal languages. (a) 3D PCA of model activations and SAE encoder weights, where datapoints are colored by part-of-speech (PoS). Encoder weights are indicated by points for SpaDE and arrows for the other SAES. (b) Matrix of cosine similarities between pairs of data and pairs of latents (in order) for each SAE. White lines separate different PoS. (c) MSE normalized by PoS variance as a function of sparsity, for each PoS. Inset: cumulative sum of variance (eigenvalues of data correlations) of each PoS, where the effective dimension (variance > 99%) of each PoS is shown. (d) Top-20 Fi -sores for different PoS from each SAE's latents (a measure of monosemanticity).
Observations: Fig. 6 shows the results of all SAES on this experiment. In Row (a), TopK shows the same level of sparsity per concept for all concepts, along with worse reconstruction error for higher dimensional concepts. In contrast, other SAEs-ReLU JumpReLU and SpaDE-show adaptive sparsity to different extents by adjusting their representations the intrinsic dimension of each concept. SpaDE can capture the intrinsic dimension nearly perfectly (along the dashed y curve) for specific choice of hyperparameters.
Note that a naïve estimator which predicts the mean of each concept will achieve a normalized MSE of 1. For TopK, normalized MSE (Row (b)) goes below 20% (i.e., explains 80% of the variance) for each concept only when & exceeds the dimension of that concept. For example, d = 6 goes below the dashed line only after k- similarly for other concepts. Other SAES are able stay below the 20% threshold for nearly all concepts across hyperparameters.
In Row (c), each latent is assigned concept which it activates maximally for. Note that different concepts use different numbers of latents in ReLU, JumpReL.U, and SpaDE. However, there are correlations across concepts in ReLU and JumpReLU (for the dense case), indicating co-occurrence of latents across concepts, which is reduced in the sparse case. Correlations are absent in SpaDE under both cases. TopK uses similar number of latents across concepts, inline with its lack of adaptivity.
5.3 Formal Languages
Dataset and Experiment: Building on recent work using formal languages for making predictive claims about language models [46-48], we use this setting as a semi-synthetic setup for corroborating claims. Specifically, analyze the English PCFG (Probabilistic Context- Free Grammars, formal

---

ReLU
JumpReLU
Topk
SpaDE
Tunch
Ergfah Spr rgger
Consli syye
Chu room
Charch
hrnndh
Gerbage buck
Gea pamp
Gut but
(b)
12%
8.
576
1.%
(c)..
che
ane
ne. dhe
des pamt
spe
ne
che
Che
the
Figure 8: SAE properties on DINOv2 activations. (a) 3-D PCA of model activations colored by class, and SAE encoder weights (points for SpaDE, arrows for other SAEs). (b) Cosine similarities of sparse codes of pairs of data and pairs of SAE latents (in order) for each SAE. White lines separate classes. (c) Fi scores of top-5 most monosemantic latents for each SAE across classes (color-coded)
models of language often used to study its syntactic properties, see App. C.3) with subject-verboobjeee sentence order proposed in [34]. We train 2-layer Transformers [49] from scratch on strings of maximum length 128 tokens from the formal grammar above. SAES are then trained on activations
retrieved from the middle residual stream of the model.
Results are shown in Fig. 7. Different parts of speech (PoS), the core concepts of the grammar, form clusters in a 3D PCA of their representations (see row (a)). SpaDE learns to tile the PoS clusters well. While all SAES do good job at making their latents uncorrelated across PoS (first column per SAE, (second column per SAE, row (b)). PoS seem to have different intrinsic dimensions (number of dimensions to capture 99% of total variance in data, inset in row (c)), which leads to TopK requiring different values of to explain the data (crosses 5% normalized MSE with differing values of k, row (c)). PoS also appear to have differing levels of linear separability, ReLU and JumpReL show lower F1 scores which peak at different levels of sparsity for each concept (row (d)), while SpaDE shows a perfect F1 score of in its most monosemantic latents.
Observations:
(b)), there co-occurring latents across PoS in all SAEs except SpaDE
5.4 Vision
We use Imagenelte, 10-class subset of ImageNet [50], containing 1.5k images per class. Representations are extracted from the DINOv2-bur (with registers). yielding 261 tokens per image. Over the course of 50 training epochs, this yields approximately 200 million tokens. SAEs are trained on all available tokens, including spatial, CLS, and registers tokens, for 50 epochs with 200 latent dimensions.
Dataset and Experiment:
Results are shown in Fig. 8. SpaDE again tiles the class structure well in the 3-D PCA (row (a)). Similarities between sparse codes of data (first column of each SAE in row (b)) show that all SAES are able decorrelate different classes in their latent representations. Latent co-occurrence (second column of each SAE in row (b)) is widespread in ReLU, JumpReLU and TopK SAEs, but it seems to be specific to certain pairs of latents in SpaDE. F, scores (row (c)) show that SpaDE has the most monosemantic latents across all classes. The varying Fi scores for ReL.U and JumpReL.U across classes indicate different levels of linear separability across classes. Importantly, we find SpaDE identifies interpretable concepts such as foreground/background, different parts of objects in
Observations:

---

an image (hands, face, fins of fish, windows/ stairs in church images, eyes, ears, snout of dogs, etc), which are visualized using feature attribution maps in App. E.4.
Discussion and Limitations
Our findings reveal critical insights into the limitations and strengths of different sparse autoencoder (SAE) architectures for concept discovery. We observed that ReLU and JumpRel.U SAFs fail to capture nonlinear separability low F1 scores, latent co-occurrence across concepts), while Top-K SAE struggles to capture concept heterogeneity (high MSE when concept dimension exceeds choice of K). A common issue across these architectures is the co-occurrence latents across multiple concepts, indicating lack of concept specialization. In contrast, SpaDE achieves the highest FI scores for its most monosemantic latents, exhibits low latent co-occurrence across concepts and enforces adaptive sparsity, making it an effective choice for structured concept representations. Our observations about the limitations of ReLU, JumpReLU and TopK SAES highlight that the failure modes of different SAEs stem from mismatch between their inductive biases and the true structure of the data. Specifically, ReI.U and JumpReI.U assume linear separability of concepts, which does not always hold, even for concepts that correspond specific directions in the latent space. On the other hand, our results suggest that incorporating data geometry into SAE design significantly improves concept specialization of SAE latents, allowing to learn latents that are better aligned with the data.
Overall, our results emphasize that there may not be single best SAE architecture for all contexts unless the architecture explicitly integrates a sufficient set of data properties relevant to the specific problem. This suggests shift in focus from using generic SAEs to tailoring their design based on prior knowledge about the underlying data geometry.
Our analysis of receptive fields of SAE encoders and their relation with concept geometry to study monosemanticity properties is quite general, and can also be used to study other kinds of interpreter models such transcoders [51. 52].
Limitations: While SpaDE demonstrates promising improvements over ReLU, JumpReLU and TopK SAEs in synthetic, semi-synthetic and realistic data, we do not claim it to be the optimal SAE for all scenarios. Instead, we present concrete example of how incorporating reasonable data properties (nonlinear separability and concept heterogeneity) can improve interpretability. Thus, several limitations remain, as follows.
Data properties beyond those considered here may be crucial for improved SAE performance. Future work may explore additional geometric structure of concepts in neural networks to design better SAFs.
SpaDE implicitly assumes concepts are separated by Euclidean distance, which may still result in latent co-occurrence if concepts do not satisfy this assumption.
Overly specialized latents may emerge in SpaDE if the sparsity level is too aggressive, potentially leading to latents that capture special cases rather than generalizable concepts.
We have focused our attention on mutually exclusive concepts, where the presence of one concept implies the absence of others. While our arguments about SAE assumptions hold even when concepts overlap, the expected co-oceurrence structure may differ such cases. For co-occurring concepts, our receptive field analysis can be applied to the presence/absence of each concept.
Overall, note work is a proposal for the best SAE, but guiding framework for improving design of SAEs that yield useful interpretations demonstrating how better integration of data geometry can enhance model interpretability. The interpretability community may need to prioritize deeper understanding of latent space geometry, and translate novel insights into SAE design, leading to models with more faithful and structured representations of concepts.
Acknowledgments
This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. ESL thanks Hidenori Tanaka and the Physies of Intelligence group at CBS-NIT program Harvard for useful conversations and access to compute resources. All authors thank the CRISP group at Harvard SEAS for insightful conversations.
10

---

References
[1] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprins arXiv:2404,09933 2024.
[2] Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, et al. International safety report. arXiv preprint arXiv:2501. 100000000... 2025.
[3] Simon Pepin Lehalleur, Jesse Hoogland, Matthew Barrugia-Roberts Susan Wei, Alexander Gi- etelink Oldenziel, George Wang, Liam Carroll, and Daniel Murfet. You are what you eat-ai alignment requires understanding how data shapes structure and generalisation. arXiv preprint arXiv:2502.05475. 2025.
[4] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1 16:1-55, 2022.
[5] Julius Adebayo, Michael Muelly, llaria Liccardi, and Been Kim. Debugging tests for model explanations. arXiv preprint arXiv:201 I.05429, 2020.
[6] Thomas Serre Learning a dictionary of shape-components visual cortex: Comparison with neurons, humans and machines. 2006.
[7] Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah Smith. Sparse over- complete word vector representations. arXiv preprint arXiv: 1506.02004 2015.
[8] Anant Subramanian, Danish Pruthi, Harsh Jhamtani, Taylor Berg-Kirkpatrick, and Eduard Hovy. Spine: Sparse interpretable neural embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[9] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications polysemy. Transactions of the Association for Computational Linguistics, 6:483-499 2018.
[10] Bruno A Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 381(6583):607--065 609, 1996.
[11] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint 2rXiv:3309.08600 2023.
[12] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden MeLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary lcarning. Transformer Circuits Thread, 2023. https:/ttransfommer circuits, circuits.pub/20231
[13] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
[14] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024.
[15] Thomas Fel, Ekdeep Singh Lubana, Jacob S. Prince, Matthew Kowal, Victor Boutin, Isabel Papadimitriou, Binxu Wang, Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models, 2025.

---

[16] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024.
[17] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2711-2721, 2023.
[18] Julien Colin, Lore Goetschalckx, Thomas Fel, Victor Boutin, Jay Gopal, Thomas Serre, and Nuria Oliver. Local vs distributed representations: What is the right basis for interpretability? arXiv preprint Sitiv:24/1.03999/ 2024.
[19] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[20] Thomas Fel. Sparks of explainability: Recent advancements in explaining large vision models. arXiv preprint arXiv:2502.01044, 2025.
[21] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
[22] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Transformer Circuits Thread, 2024.
[23] Esin Durmus, Alex Tamkin, Jack Clark, Jerry Wei, Jonathan Marcus, Joshua Batson, Kunal Handa, Liane Lovitt, Meg Tong, Miles McCain, et al. Evaluating feature steering: A case study in mitigating social biases, 2024. URL https://antropic.. com/researccheevaluttnng steering, 2024.
[24] Harrish Thasarathan, Julian Forsyth, Thomas Fel, Matthew Kowal, and Konstantinos Derpanis. Universal sparse autoencoders: Interpretable cross-model concept alignment, 2025.
[25] Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language models via sparse autoencoders. bioRxiv, pages 2024-11 11, 2024.
[26] Etowah Adams, Liam Bai, Minji Lee, Yiyang Yu, and Mohammed AlQuraishi. From mech- anistic interpretability to mechanistic biology: Training, evaluating, and interpreting sparse autoencoders on protein language models. bioRxiv, pages 2025-02, 2025.
[27] Edith Natalia Villegas Garcia and Alessio Ansuini. Interpreting and steering protein language models through sparse autoencoders. arXiv preprint arXiv:2502.09/33: 135, 2025.
[28) Samuel Marks, Can Rager, Eric Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403. 19647, 2024.
[29] Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoen- coders. arXiv preprint arXiv:2410.22366. 2024.
[30] Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint 20l.:2202.16681.. 2025.
[31] Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for safety-a review. arXiv preprint arXiv:2404.14082, 2024.
12

---

[32] Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features, arXiv preprint arXIv:2501.1611,, 2025.
[33] Usha Bhalla, Suraj Srinivas, Asma Ghandeharioun, and Himabindu Lakkaraju. Towards unifying interpretability and control: Evaluation via intervention. arXiv preprint arXiv:2411.04430, 2024.
[34] Abhinav Menon, Manish Shrivastava, David Krueger, and Ekdeep Singh Lubana. Analyzing (in) abilities of saes via formal languages. arXiv preprint arXiv:2410.11767, 2024.
[35] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23)3331-3322 1997.
[36] Andrew Ng al. Sparse autoencoder. CS294A Lecture notes, 72(2011):1-19, 2011.
[37] Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: 131.5533, 2013.
[38] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma arXiv preprint arXiv:2408.05117, 2024.
[39] Róbert Csordás, Christopher Potts, Christopher D. Manning, and Atticus Geiger. Recurrent neural networks learn to store and generate sequences using non-linear representations, 2024.
[40] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03655 2023.
[41] Lennart Bürger, Fred A Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in 1lms. Advances in Neural Information Processing Systems, 37:138393-13843, 2025,
[42] Joshua Engels, Eric Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are linear, 2024.
[43] Wenbo Pan. Zhichao Liu, Qiguang Chen, Xiangyang Zhou, Haining Yu, and Xiaohua Jia. The hidden dimensions of Ilm aligmment: A multi mullti-dimensionll safety analysis, 2025.
[44] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. International conference on machine learning, pages 1614- 1623. PMLR, 2016.
[45] Abiy Tasissa, Pranay Tankala, James M Murphy, and Demba Ba. K-deep simplex: Manifold learning via local dictionaries. IEEE Transactions on Signal Processing, 2023.
[46] Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning procedurally defined tasks. arXiv preprint arXiv:2311.12788 2023.
[47] Ekdeep Singh Lubana, Kyogo Kawaguchi, Robert P Dick, and Hidenori Tanaka. A percolation model of emergence: Analyzing transformers trained on a formal language. arXiv preprint arXiv:2408. 12578, 2024.
[48] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part learning hierarchical language structures. arXiv preprint 1rXiv:2305.33677 2023.
[49] Andrej Karpathy. nanoGPT, 2023.
[50] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
[51] Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. Advances in Neural Information Processing Systems, 77:24375-244111 2025.
13

---

[52] Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders beat sparse autoencoders for interpretability. arXiv preprint S.Kiv:2501.18223 2025.
[53] Michael Elad, Peyman Milanfar, and Ron Rubinstein. Analysis versus synthesis in signal priors. Inverse problems, 23(3):947, 2007.
[54] Andreas M. Tillmann. On the computational intractability of exact and approximate dictionary learning. IEEE Signal Processing Letters, 22(1):45-49, 2015.
[55] Kenneth Kreutz-Delgate Joseph F Murray, Bhaskar D Rao, Kjersti Engan, Te- Te-Won Lee, and Terrence J Sejnowski. Dictionary learning algorithms for sparse representation. Neural compulation, 15(2):399-366 2003.
[56] Pablo Sprechmann and Guillermo Sapiro. Dictionary learning and sparse coding for unsu- pervised clustering. In 2010 IEEE international conference on acoustics, speech and signal processing, pages 2042-2045. IEEE, 2010.
[57] Bruno A Olshausen and David J Field. Wavelet-like receptive fields emerge from network that learns sparse codes for natural images. Nature, 38 :667-6669, 1996.
[58] Ingrid Daubechies, Michel Defrise, and Christine De Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57(11):14413 1457, 2004.
[59] Amir Beck and Marc Teboulle. A fast iterative shrinkage-threshooldnng algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183-202, 2009.
[60] Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. Proceedings of the 27th international conference on machine learning. pages 399-4994 2010.
[61] Vishal Monga, Yuelong Li, and Yonina C Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine, 38(2):18-44, 2021.
[62] Zhaowen Wang, Ding Liu, Jianchao Yang, Wei Han, and Thomas Huang, Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE international conference on computer vision, pages 370-378, 2015.
[63] Siheng Chen, Yonina C Eldar, and Lingxiao Zhao. Graph unrolling networks: Interpretable neural networks for graph signal denoising. IEEE Transactions on Signal Processing, 69:3699- 3713, 2021.
[64] Botao An, Shibin Wang, Zhibin Zhao, Fuhua Qin, Ruqiang Yan, and Xuefeng Chen. In- terpretable neural network via algorithm unrolling for mechanical fault diagnosis. IEEE Transactions on Instrumentation and Measurement, 71:1-11, 2022.
[65] Bahareh Tolooshams, Sara Matias, Hao Wu, Simona Temereanca, Naoshige Uchida, Venkatesh N Murthy, Paul Masset, and Demba Ba. Interpretable deep learning for decon- volutional analysis of neural signals. bioRxiv, 2024.
[66] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. http://iiilllppp//22///cccisssss
[67] Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404. 16014, 2024.
[68] Glen M. Taggart. coders.
Prolu: A nonlincarity for sparse autoen- http://wwww.llgggnnfffffffoe org/ prolu--enoollinaxtttt-------- oatprtttoyooooppppeeeeeaaattooot 2024.
[69] Zhengxuan WU, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outperform sparse autocncoders. arXiv preprint arXiv:2501.17148, 2025.
14

---

[70] Joshua Engels, Logan Riggs, and Max Tegmark. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410.1400 14670, 2024.
[71] Connor Kissane, Robert Krzyzanowski, Neel Nanda, and Arthur Conmy. Saes are highly dataset dependent: A case study the refusal direction. In Alignment Forum, 2024.
[72] Patrick Leask, Bart Bussmann, Michael Pearce, Joseph Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. arXiv preprint ar ar0iv:2502.044 04878, 2025.
[73] Kola Ayonrinde, Michael T Pearce, and Lee Sharkey, Interpretability as compression: Recon- sidering sae explanations of neural activations with mdl-saes. arXiv preprint arXiv:2410.11179, 2024.
[74] Irina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2017.
[75] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. international conference on machine learning, pages 4114- 4124. PMLR, 2019.
[76] Francesco Locatello, Ben Poole, Gunnar Ratsch, Bernhard Scholkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. 2020.
[77] Luigi Gresele, Paul K Rubenstein, Arash Mehrjou, Francesco Locatello, and Bernhard Scholkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica. Uncertuinty in Artificial Intelligence, 2020.
[78] Julius Von Kugelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Scholkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. 2021.
[79] Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christo- pher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024.
[80] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. Reft: Representation finetuning for language models. Advances Neural Information Processing Systems, 37:63908- 63962, 2025.
[81] Tianlin Liu, Joan Puigcerver, and Mathieu Blondel. Sparsity-constrained optimal transport. arXiv preprint arXiv:2209. /5466, 2022.
[82] Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hicrarchical concepts in large language models. arXiv preprint 211::2406001500 2024.
15

---

A Dictionary Learning
Sparse coding [10] (alternatively known in this work as sparse dictionary learning, or just dictionary learning) was initially proposed to replicate the observed properties ("spatially localized, oriented, bandpass receptive fields") of biological neurons in the mammalian visual cortex. It aims to invert linear generative model with a sparsity prior on the latents:
D
where x E IR" is the data, D E R is the set of dictionary atoms, x" E R. is the sparse code, and 17 is additive white Gaussian noise. Given data ...).......)))) sparse coding performs maximum aposteriori (MAP) estimation for the dictionary D" and representations x" under suitably defined prior and likelihood functions [53] by solving the following optimization problem (repeated from Eq. 1):
arg min 200
Σ 11x0(k) - Dx +2
XR(2
(6)
where R(-) is a sparsity regularizer. The set B C R includes restriction to unit norm (typical). Generally, the L1 penalty is used as the regularizer term, i.e., R(z(*)) ||2(() 1, since using the LO penalty makes the problem NP-hard [54]. When the number of dictionary atoms is less than or equal to the dimension of input space, s S n, this is an undercomplete problem, and the sparse code can be readily obtained using the pseudo-inverse of the dictionary matrix D (provided the dictionary atoms are linearly independent), leading to the solution z DD D x. Note that in this (undercomplete) case, the sparse code is a linear transformation of the input. The more interesting setting involves using an overcomplete dictionary (s > n), and was initially studied in [35]. Obtaining the sparse code from input data æ is nontrivial in this case.
In this case, sparse coding results in a sparse representation of the data and a dictionary which behaves a data-adaptive basis. Corresoondigly., sparse codes have been shown to capture interesting concepts in data [55, 56), e.g., responding to wavelet-like regions when trained on natural images [57]. In this (overcomplete) setting, a popular approach is using iterative shrinkage and thresholding algorithms (ISTA) [58] and their variants such as FISTA (Fast ISTA) [59]. Modern approaches to this problem use ISTA design deep residual networks with shared weights and train the network on the sparse coding objective, in a technique called Learned ISTA (LISTA) [60]. Algorithm unrolling [61] is a generalization of this technique and involves designing interpretable neural networks using iterative algorithms where each layer of the network reflects an iteration of the algorithm. These networks are interpretable since the weights correspond to an underlying process which was used to design the iterative algorithm. Unrolling has widespread applications in signal processing, and is extensively reviewed in [61].
We also note that sparse coding has been used with algorithm unrolling as a model-based interpretable deep learning technique for wide range of applications, including image super-resolution [62], graph signal denoising [63], mechanical fault diagnosis [64], deconvolving neural activity of dopamine neurons in mice [65]. Therefore, assuming linear generative model of data (Eq. 1) where the dictionary atoms are physically relevant in some application, sparse coding using an unrolled network learns the underlying interpretable dictionary atoms.
B Related Work
SAFs are a specific instantiation of the broader agenda of dictionary learning tools for concept- level explainability [7-9. 19, 20, 66]. A number of SAE architectures have been proposed recently, including ReLU SAE [12], TopK SAE [13, 371, gated SAE [67], JumpReLU SAE [14]. Batch TopK SAE ([16)), ProLU SAE (168]), and so on. While promising results have been discovered, c.g., latents that respond to concepts of refusal, gender, text script [12, 22, 23], foreground vs. background concepts [17], and concepts of protein structures [25-27], a series of negative results have started to emerge on the limitations of SAES. For example, [33, 69] show a mere prompting baseline can outperform model control compared to SAE or probing based feature ablation baseline. Similar results were observed by [34] in a narrower formal language setting. Meanwhile, criticizing the underlying linear representation hypothesis that has informed design of earlier SAE architectures (specifically, the vanilla ReLU SAEs), [42, 70] has shown that SAE features can be multidimensional
16

---

and nonlinear. Importantly, recent results from [15, 32, 71] have shown that two SAEs trained on the exact same data, just with a different seed, can yield very different concepts and hence very different interpretations. These results are related to the lack of canonical nature in SAF latents [72] This behavior, often called algorithmic instability, makes reliability of SAEs challenging for any practical purposes. More broadly, given the hefty research investment going into the topic, we believe is warranted that a more formal and theoretical account help solidify the limitations and challenges SAEs (or at least the current paradigm thereof) faces. This can help steer the research in a direction that yields meaningful improvement in SAEs, e.g., in their practical utility. This motivation underscores our work. For related effort this front, we highlight the work by [73], who contextualize SAEs from a minimum -eniuuddddddiitionnn length perspective and enable an intuitively solid account of how features may split to overly specialized concepts (e.g., tokens).
Disentangled Representation Learning. As mentioned in Sec. 3, results similar to ours have been reported in the field of disentangled representation learning, wherein one aims to invert data-generating process to identify the factors of variants (i.e., latent variables) that underlie it. To this end, autoencoders were used popular tool, since they offer a method that can (ideally) simultaneously invert the generative process and identify the underlying latents [74]. However, [75] showed that in fact this problem is rather challenging: unless one designs an autoencoder architecture that bakes-in assumptions about the generative process, i.e., the precise function mapping itself, there are guarantees the retrieved latents will correspond to the ground-trute truth ones. This result led to design of several methods focused on exploiting "weak supervision" i... extra information available from data-pain auta-pairs such multiple views of image temporally consistent video frames, to circumvent the theoretical challenges of disentanglement [76-78]. Our contributions are similar in nature these results on disentanglement, but we (i) specifically focus the context of SAEs and (ii) provide a more concrete proof that establishes precisely what the inductive biases of popular SAES are, i.e., what concepts the SAEs are biased towards uncovering. Having established these results, we now believe the next step that the disentanglement community took, i.e., use of weak supervision, would make sense for the SAEs community as well. This can involve exploiting temporal correlations between tokens in a sentence, or the fact that representations across layers do not change much, as in Crosscoders and Transcoders [51, 52, 79].
C Experimental Setup
The synthetic experiments (separability, heterogeneity) and vision experiments were run on NVIDIA A100 40GB GPUs, while the formal language experiments were run on NVIDIA RTX A6000 48GB GPUs
C.1 Separability experiment
We construct a synthetic dataset consisting of six isotropic Gaussian clusters in a two-dimensional (2D) space. The cluster centers are arranged such that adjacent clusters are separated by an angular difference of 2 /6, with alternate clusters having norms of and 3. Each cluster is sampled from a multivariate normal distribution with a variance of 2 -5.5 The dataset consists of I million data points per concept, yielding total of 6 million samples. Of these, use 70% (700,000 points) for training.
Our experiments evaluate four sparse autoencoder (SAE) architectures: ReLU SAE, JumpReLU SAE, TopK SAE, and SpaDE. The first three architectures are implemented following their original formulations (in [12].[14], (1,33), with the decoder activations normalized in the forward pass. The SpaDE model follows the same single hidden-layer autoencoder structure but differs in that it utilizes Euclidean distance computations and a SparseMax activation function for the encoder. Across all models, the hidden-layer width is set to 128, and a pre-encoder bias is used in all cases except for SpaDE.
For training, the (inverse) temperature parameter in SpaDE is initialized to 1/(2 x input dimension) and parameterized using the Softplus function to ensure non-negativity. This parameter trained along with the encoder and decoder weights, to allow the model to learn its desired sparsity level. Note that large values of X lead to greater sparsity since Sparsemax is scale-sensitive. In JumpReLU, the threshold is initialized at 10-3 across all latent dimensions, with a bandwidth of 10-3 for the traight-thhouugh estimator (STE), as it is proposed in [14]. All models are trained using the
17

---

Adam optimizer with learning rate of 10 10-2 which follows a cosine decay schedule from 10 to 10- The momentum parameter set to 0.9, and we use a batch size of 512. Training runs for approximately 8000 iterations, and gradient clipping is applied (gradient norms are clipped at 1) to stabilize optimization.
Regularization parameters are selected such that sparsity levels remain comparable across models. Specifically, the regularization coefficient 7 is chosen in the range 10-6 to for ReLU and JumpReLU SAES, between 4 and 64 (powers of 2) for TopK SAE, and in the range 10 to I for SpaDE. Each model applies different regularization strategy: ReLU SAE uses L1 regularization, JumpReLU SAE applies Lg regularization with trraight-througee estimator (STE) in [14], TopK SAE does not use explicit regularization but incorporates an auxiliary loss term as in [13], with K Kaux = k (same as the choice of sparsity level k in TopK) with Taux = 1 (the scaling for the auxiliary loss term). and SpaDE employs a distance distance-weighted L1 regularization, which comes from [45].
All networks are initialized such that the decoder weights are initially set as the transpose of the encoder weights, though they are allowed update freely during training. Model weights are sampled from a normal distribution (0, 1). To maintain consistency in scale between inputs and latent activations, a scaling factor À is applied to all latent units, given by À A 1/2 input dimension (note that this is not trainable for ReLU, JumpReLU and TopK SAEs). Across all architectures, we use the Mean Squared Error (MSE) loss function, with the regularizers and regularizer scaling constants as described above.
For evaluation, we analyze a subset of 1000 data points per concept. The primary metric for comparison is the Fl-score, which is computed based on precision and recall. Precision is defined as:
True Positives
Precision
True Positives False Positives'
while recall is given by:
True Positives
Recall
(8)
True Positives False Negatives
Using these definitions, the Fl-score is computed as:
F1 2 X Precision x Recall
Precision + Recall
In our setup, precision and recall are computed by thresholding latent activations at 10-6, Additionally, we analyze the receptive fields by creating a 2D meshgrid, passing all points through the model, and extracting their SAE latent representations. Cosine similarities between pairs of data points are also computed by obtaining their latent representations, calculating the pairwise cosine similarity, and organizing the results by class.
To further examine latent space structure, we compute the stable rank of the representation matrix. Stable rank for the similarity matrix is computed as the sum of singular values divided by the largest singular value (alternatively called the intrinsic dimension of this matrix):
Stable Rank = Σo
(10)
Fmax
Finally, spectral clustering is performed on the similarity matrix derived from latent representations. The number of clusters is determined by the stable rank of this similarity matrix (rounded up), providing insights into the correlations between SAE latent representations.
C.2 Heterogeneity experiment
We construct a synthetic dataset consisting of five isotropic Gaussian clusters in a 128-dimensional space. The intrinsic dimensionality of each cluster follows the sequence 29 for different values of q E {3, 4, 5, 6, 7), resulting in clusters with intrinsic dimensions of 6, 14, 30, 62, 126, respectively.
18

---

clusters belong to subspaces that form strict subsets of the subspaces of The higher-dimensional ones, meaning that the first six dimensions are fully contained in the next 14, which are further contained in the next 30, and so on up to 126 dimensions. Cluster centers are sampled uniformly random from the range [0, 21, along each dimension. The variance of each concept is chosen to be inversely proportional to its intrinsic dimension to ensure that the total variance per concept remains constant across all concepts. The dataset contains 6.4 million data points per concept, yielding a total of 32 million samples, of which 70% (approximately 22 million points) are used for training.
Our models follow four different sparse autoencoder (SAB) architectures: ReLU SAE, JumpReLU SAE, T SAE, and SpaDE. The first three are implemented according to their original formulations in [12], [14]. and [13], with the decoder activations normalized in the forward pass. The SpaDE model follows the same single hidden-layer autoencoder structure but differs in that it utilizes Euclidean distance computations and SparseMax activation function for the encoder. Across all models, the SAE hidden-layer width is set to 512. pre-encoder bias is applied in all cases except for SpaDE, Additionally, for the TopK SAE, ReLU activation is applied before selecting the top k latent dimensions.
For training, the temperature parameter À in SpaDE is initialized at 1/(2 x input dimension) and parameterized using the Softplus function to ensure non-negativity. This parameter trained along with the encoder and decoder weights, to allow the model to learn its desired sparsity level. In JumpReLU, the threshold initialized at 10-3 across all latent dimensions, with a bandwidth of 10 3 for the straight-through estimator (STE). All models are trained using the Adam optimizer with a learning rate of 10-2, which follows cosine decay schedule from 10-2 to 10-4, The momentum parameter is set to 0.9, and we use batch size of 2048. Training runs for approximately 10,000 iterations, and gradient clipping (restricting gradient norms to be less than 1) is applied to stabilize optimization.
Regularization parameters are selected such that sparsity levels remain comparable across models. Specifically, the regularization coefficient y is chosen in the range 10-3 to 5.0 for ReLU SAE, 10-3 to 1 for JumpReLU SAE. from 4 to 256 (powers of 2) for TopK SAE, and from 10 to 10 for SpaDE. Each model applies different regularization strategy: ReLU SAE uses L1 regularization. JumpReLU SAE applies Lo regularization with a trriigh-tthrouge estimator (STE) following from [14), TopK SAE does not use explicit regularization but incorporates an auxiliary loss term with Taux = (scaling for the auxillary term in the loss) and Keuz = & (same as sparsity level), and SpaDE employs a distance distance-weighted L, regularization.
All networks are initialized such that the decoder weights are initially set as the transpose of the encoder weights, though they are allowed to update freely during training. Model weights are sampled from a normal distribution (0, 1). To maintain consistency in scale between inputs and latent activations, a scaling factor À is applied to all latent units, given by X = 1/2 input dimension. Across all architectures, we use the Mean Squared Error (MSE) loss function.
For evaluation, analyze subset of 1000 data points per concept. We report the normalized MSE, defined as the ratio of the standard MSE to the variance of the corresponding concept:
MSE
Normalized MSE-
(11)
Variance of Concept
We also compute sparsity (Lo) per concept, measured as the average number of active latents per data point, averaged over each concept.
To analyze latent representations, we examine cosine similarities in two contexts: (i) between pairs of SAE latent representations for different input data points (per-input co-occurrence) and (ii) between pairs of latents aggregated over all data points (global co-occurrence). For the latter, each latent is assigned a concept label based on the concept for which it is most frequently activated on average. This assignment provides insight into how latents specialize across different underlying structures in the dataset.
C.3 Formal Languages experiment
Data. The formal language setup analyzed the main paper (Sec. 5.3) involves training a 2-layer nanoGPT model on strings from an English-like PCFG (Probabilistic Context-Free Grammars).
19

---

Broadly, a PCFG is defined via 5-tuple G = (MT. T, R, S,P, where NT is finite set of non-terminal production rules, each of the form A + 0x3, where A E NT and at 3 E (NT U T); S E NT is the start symbol; and P is a function R + [0, 1), such that for each A E NT, > La:A-tEE P(A + o:3) 1. To generate a sentence from the grammar, the following process is used.
symbols; T finite set of terminal symbols, disjoint from NT; is finite
1. Start with a string consisting of the start symbol S.
2. While the string contains non-terminal symbols, randomly select non-terminal A from the string. Choose production rule A + aB from R according the probability distribution
P(A + a).
3. Replace the chosen non-terminal A in the string with a, the right-hand side of the production rule.
4. Repeat the production rule selection and expansion steps until the string contains only terminal symbols (i.e., no non-terminals remain).
5. The resulting string, consisting entirely of terminal symbols, a sentence sampled from the grammar.
We follow the same rules of the grammar considered in [34]. The strings are tokenized via one-hot encoding via manually defined tokenizer.
Model training. Models trained from scratch strings sampled from the grammar above. Strings are padded to length 128 (if not already that length). and a batch-size of 128 (~10K tokens per batch) is used for training. Training uses Adam optimizer with cosine learning-rate schedule starting at 10-3 and ending at 10-4 after 70K iterations, alongside a weight decay of 10-1 The nanoGPT models used in this work have width of 128 units, with an MLP expansion factor of 2 and also 2 attention heads per attention layer.
SAE training. All SAES trained in the formal language setup involve an expansion factor of 2x, i... 256 latents for residual stream of 128 dimensions. Training involves a constant learning rate of 10-3 and lasts for 10K iterations (~1M tokens). We sweep regularization strength for SAEs' training. yielding SAEs with different sparsity levels. While we fix the regularization strength for SpaDE based on best values identified from the synthetic. Gaussian cluster datasets, for other SAEs (ReLU, JumpRel.U, and TopK) we report the best possible results from our sweep by looking at the t0p-10 per-concept F1 scores; i.e., reported results are best-case estimate of results achievable by training of these SAEs, and in practice performance can be expected to be poorer than what we analyze. Cross-task transfer for SpaDE's hyperparameters is intriguing in this regard, since found other SAEs' hyperparameters to not transfer.
C.4 Vision experiment
Data. We use an off-the-shelf, large-scale pretrained model for our analysis in these experiments, specifically DINOv2-base (with registers). For simplicity, we focus on a 10-class subset ImageNet, called Imagenette, containing 1.5k images per class. Representations are extracted from the model for images of these classes, yielding 261 tokens per image.
SAE training. SAEs are trained on all available tokens, including spatial, CLS, and registers tokens, for 50 epochs with 200 latent dimensions. With 261 tokens per image, this amounts ~200 tokens for training SAEs over the course of 50 training epochs. For each SAE, the best reconstruction is selected based on sparsity-controlled learning rate sweep. This resulted in optimal learning rate of 5 x 10-4 for TopK, ReL.U, and SpaDE, while JumpRel.U LU performed best with 10-4 (using Adam optimizer). Additionally, note our JumpReL.U implementation employs Silverman kernel with bandwidth of 10-2, which we found to work best for our setting.
D Further Theory Results
D.1 Projections and Nonlinearities
The nonlinearity of popular SAES is commonly an orthogonal projection onto some set, where the choice of projection set differentiates SAES (see Fig. 2). We formalize such nonlinearities as projection nonlinearities, (re)defined below.
20

---

Definition D.1 (Projection Nonlinearity), Let v E R be pre-activation vector. A projection nonlinearity II {-} R + R is defined as:
Is {u) arg min || 21l2
(12)
TES
where S C R is the constraint set onto which is orthogonally projected. The structure of S determines the properties of the nonlinearity.
We will say function f() is Projection Encoder if it uses a projection nonlinear- ity g(-) applied to linear transformation of the input. This is equivalent to using W x + be. and f - g(v) (see Eq. 2), where g is a projection nonlinearity. Pop- ular SAEs can be understood as a similar Projection Encoder with different projec- tion nonlinearities, shown in Tab. 4 (see Theorem D.3 for derivation).
Table 4: Projection Nonlinearities in SAE Encoders. Each model can be understood by its nonlinear orthog- onal projection g(.) onto a constraint set S which de- termines its activation behavior, sparsity structure, and implicit data assumptions.
Model
g(v)
ReLU
IIs (v). (a e : 0)
TopK
Is (v), (x ==
k}
Heaviside (H)
Is. [ 11 (0,
Lemma D.2 (Elementwise projections). For projection nonlinearities whose projec-
JumpReL.U
ReLUUe 0)+0 H(v 0)
tion sets satisfy componentwise constraints, i.e. S= TR f(x,))
= [s]].
the projection problem can be decoupled and broken down into a combination of elementwise projec- tions, leading to an elementwise nonlinearity. The converse is also true: any elementwise nonlinearity which is also a projection nonlinearity can be written as a combination of elementwise projections, leading to componentwise constraints on the projection set
Proof.
II {x} arg min || 2e12
(13)
TES
argmin min ((5)))..(())))))).l
arg
Σ (Tk xxk
(14)
-..., min ((=))0
(TT )2,)),...
(15)
i.c. [
arg min (TT - )2
-xx)
(16)
(()
This is a consequence of the objective function above (squared euclidean norm of the difference π - x) decomposing into a sum over componentwise functions. The above argument can be traced backward, since all steps are invertible, which proves the converse.
Theorem D.3 (Projection Nonlinearities). ReLU, TopK, JumpReLU are simple combinations of
orthogonal projections onto nonlinearity-sper speeiffc sets: ReLU a projection onto the positive orthant, TopK is a projection onto the union of all k-sparse subspaces, and JumpReLU is sum of shifted ReLU and shifted Heaviside step, which itself is projection onto the corners of a hypercube.
Proof. First consider the ReLU nonlinearity, defined for E TR as:
z = ReLU(x))
(17)
if x4 0
= 0 else
(18)
This is an elementwise nonlinearity, so it suffices to show that each component can be written as projection from Lemma D.2). Consider this reformulation:
arg min(1; xT
(19)
7.20
This is equivalent to ReLU, since for all non-negative values, it equals the input, while it is 0 (nearest non-negative point) for all negative inputs. Using Lemma D.2, ReL.U is projection nonlinearity with projection set S {x E R 2 Ovi e [s]].
21

---

JumpReLU is defined as:
JumpReLUU(a) @ (= 0)
(20)
(a 0 + 0) o H
(21)
ReLU 0) + H( 0) where the heaviside step function H. is:
(22)
(()) = x > 0) which is performed elementwise. Thus, JumpReL.U (and the heaviside step) is also an elementwise
(23)
nonlinearity. Consider the step function:
if x4 2 0 else
H <)) (x,))
(24)
arg min( + 0.5 - n;)²
(25)
x.E(0.1)
which is a shifted version of a projection. Again using Lemma D.2, H is a projection nonlinearity with projection set S {or IR 24, {0, 1)], ie., the corners of unit hypercube.
The TopK nonlinearity defined as:
yj = RT)
(26)
TopK(2n (x); = My I(uy 2 up E M: (MM
(27)
where is the dimension of the space. Note that topK typically includes ReLU applied first ([13)), making all entries of the vector non-negative followed by choosing the k-largest entries of (x). Consider projection onto the union of all k-dimensional axis-aligned subspaces. With non-negative
entries (due to ReLUU, this would lead to choosing the k largest entries of x::
arg min
|| - π 1
arg min
Σ(x: 2,)
(28)
Wt x is sparse
:: is t sparse
= (x)
(29)
This completes the proof.
L
Theorem D.4. Projection nonlinearities satisfy the following properties:
1. For points within the set S. projection is an identity map
S IIs {x}
2. For points outside the set S. projection is onto the boundary
x & S = IIs {x} E ÖS
3. If 0S is flat (linear manifold), or subset of flat (with flat boundaries), projection of points outside the set S is either piecewise linear or constant:
IIs {ax1 + [3 = aIIs {x1} + -IIls {x2} for a, B E TT, OR
Is {ar c, D (a linear piece)
where 21,22 & S. T is suitably defined confine to the corresponding linear piece
Proof. (sketch) (1) is trivial and follows from the definition of projection nonlinearities (Eq. 3). For (2), suppose IIs {æ} is in the interior of S. This implies that By Int(S) such that y I (x) E (0, 1] and therefore |yy - x < x IIs {a} 1|, which is contradiction. Thus Hs (a) E as.
aæ + (1
For (3), one can consider the section of the boundary as that is closest to æ, and extend it to form subspace (possible since it is flat). Since projections onto subspaces are linear operations, IIs (x) is linear in some neighborhood, and thus piecewise linear. In some cases, there is single corner point of S that closest to v, in which case the projection is a constant.
Projection nonlinearities are orthogonal projections onto various sets. For points within the set S. projection is the point itself, while for points outside, the projection is onto the boundary as (Theorem D.4 in Appendix). For projections to be well defined everywhere, the set S must be closed (so that the boundary belongs to the set, i.e., aS E S). Note that if the set S is a subspace of RR, projection is linear map. Therefore, the nonlinearity of projection nonlinearities comes from choosing either subset of a subspace, or non-flat manifold. Sparsity projection nonlinearities consequence of the projection set having edges/corners along sparse subspaces.
22

---

D.2 Receptive fields of various SAEs
First, we (re)define the four SAE encoders we study in this section:
ReLU SAE: 2 ReLUUWe"T b)
(30)
JumpReLU SAE: 2 JumpReLU
(31)
TopK SAE: 2 Topk(W"ar)
(32)
SpaDE: 2 = Sparsemax(-Xd(1,, W))
(33)
d( a. W); ); = | w
w
(34)
This section discusses the piecewise linear (affine) regions (by showing that each of the above is a piecewise linear function) and neuron receptive fields in input space for each of the four SAEs (ReLU, JumpReLU, TopK, SpaDE). Projection nonlinearities become piecewise linear when the projection sets have flat faces. Under the requirement of monosemanticity. the structure of receptive fields directly implies the assumption that concepts in data have the same structure the receptive field.
Intuition behind Theorem 4.1 Receptive fields for ReLU and JumpReLU SAEs are half-spaces because of the linear transform in the encoder, which needs to positive for the SAE latent to be active (due to the ReLU/ JumpReLU nonlinearity). In TopK SAE, for given latent to be active, its linear transform in the encoder must be non-negative (since a ReLU is used TopK SAE implementation [13]), and must exceed at least K other linear transforms (where is the SAE width). This gives us an intersection of multiple half-spaces through the origin, leading to hyperpyramids and thus angular separation.
For projection- based encoders, the receptive field can be rewritten
F& (S {zx 0}),
where S is the projection set of the encoder.
That is, Fs is the pre-image of the intersection of the projection set with the half-space {.x >> 0}. Alternatively, it can be viewed as the complement of the pre-image of the set Sn (2x = 0}, where the hyperplane 2 0 indicates latent A is "dead".. This expression shows the explicit relation between the projection set and the receptive field properties of the SAE.
First note that all four nonlinearities have some level of sparsity, i.e., some neurons are turned off at times. The following observation is crucial in formulating the piecewise linear regions:
Lemma D.5 (Gating). Given the indices M = {i1, i2. i|m|} of active neurons (with nonzero outputs), ReLU, JumpReLU, TopK and Sparsemax are all affine functions of their inputs.
Lemma D.5 indicates that the nonlinearity in these transformation lies only in their gating, or selection of active indices. Thus, each linear (affine) region is characterized by specific choice of indices M of active neurons. Note that not all choices of indices may be allowed by the nonlinearity. Denote the set of allowed indices by M..
Let L. C IR" denote the piecewise linear (affine) region corresponding to active indices M. Lemma D.6. The set [CM M M. of all piecewise linear regions forms a partition of RR..
Using the Gating lemma, we can associate each set of active indices to piecewise linear region, and identify receptive fields as unions of such piecewise linear regions.
Lemma D.7 (Receptive fields and piecewise linear regions). A neuron's 's receptive field is union of piecewise linear regions where the neuron active:
Fx UMKEEMMM
We now use the above results and obtain the piecewise linear regions for each of the four SAFs defined previously.
23

---

D.2.1 ReLU, JumpReLU SAE
First note that the piecewise linear regions and receptive fields of ReLU and JumpReLU SAES are the same sime--ince both cases, the gating appears through the heaviside step function (ReLU (x) O I( 2 0)). Thus, we develop the linear pieces and receptive fields only for ReLU, since the corresponding ones for JumpReLU are identical. The piecewise linear regions latents in ReLU SAE are described by the following claim:
Claim D.1. For a layer defined as in Eq. 30, LM is given as:
LM = fx E R "T. + bm 2 OYm E M, w x by < D4q & M}
(35)
Thus, M an intersection of half-spaces, and thus a convex polytope which may be bounded or unbounded.
Proof. This a consequence of the observation Lemma D.5 and the definition of the relu model 30.
Lemma D.8. b 0 in Eq. 30, then LM are unbounded convex polytopes with only one corner at the origin and flat faces, iee, they (unbounded) hyperpyramids.
Thus, bias plays an important role in ReLU layers, allowing piecewise linear regions that are convex polytopes with multiple corners anywhere in space. The greater flexibility in defining the pieces allows greater expressivity by capturing a larger class of functions. The following (somewhat obvious) claim describes the receptive fields of model neurons.
Claim D.2. In Model 1 (30). for a given neuron k E [ni], the receptive field Fx given as:
F = fa E RR w x + by 2 0}
(36)
which is a half-space defined by the normal vector wy and bias by- This is straightforward consequence of the definition of the ReLU model in Eq. 30.
D.2.2 TopK SAE
Claim D.3. For a layer defined as in Eq. 32, LM is given as:
(37) Thus, M is an intersection of K(N K) half-spaces all passing through the origin, and thus convex polytope which may be bounded or unbounded. In fact, it is an unbounded hyperpyramid, with corner at the origin and flat faces. The normals to these half-spaces are pairwise differences between active and inactive weight vectors.
LM {x: E IR" wT 2 wh a.m E M,q M,q&M, M)
This again follows from the Gating Lemma D.5.
Claim D.4. In Model 2 (32), for a given neuron k E [n]], the receptive field Fx is given as:
Fr UMLEeLM
(38)
which is union of hyperpyamids with corner the origin. Note that in typical implementations of TopK, pre-encoder bias is included, so the corner of the hyperpyramids is at the pre-encoder bias.
D.2.3 SpaDE
Claim D.5. For a layer defined in Eq. 5, LM is given as:
LM
R. " 1|
wm
1
Σ
>
w.
1
< 0, i E M
(39)
MM
> 0, m $ M
jEM
x E R
(40)
w3 w/)x (1
Σ
I 12)11) + MIl
1 2 0. if EM
( m
M
< 0,
m $ M
JEM
JEM
(41)
24

---

Thus, M is an intersection of N half-spaces, and thus a convex polytope. Note that the normal to each half space now chosen an input- -anut--datiive fashion (m .M) and is locally centered using the mean of other nearby prototypes that active, i.e., (w [M| Ejem where M is input adaptive. An alternate interpretation is using the first equation above, which defines the region as the set of points whose distance active prototypes within a tolerance of the average distance to all active prototypes, while distance to inactive prototypes is larger than the average distance to active prototypes.
Proof. This is again a consequence of the definition of sparsemax [44].
Claim D.6. In SpaDE (33). for given neuron [n]], the receptive field F& is given as:
Fk UMM.MMM
(42)
which is union of convex polytopes, each of which includes the latent k in the set of active indices M. Due to the use of euclidean distances in choosing active indices, the receptive field is a union of convex polytopes in the vicinity of the prototype ag of latent k. This incorporates the notion of locality and flexibility in receptive field shapes, allowing latents to capture nonlinearly separable concepts.
D.3 KDS and Sparse Coding
K-Deep Simplex (KDS) [45] is the sparse coding framework which forms the outer optimization in the SpaDE. While this is a different framework, in this section we show that it is general enough to capture the standard sparse coding, i.e., for data generated using standard sparse coding, there exists a corresponding KDS framework that could have generated the same data. Note that we may have to increase the latent dimension (number dictionary atoms) by one to obtain the corresponding KDS framework. This is stated and proved (with constructive proof) in the following theorem.
Theorem D.9 (KDS can capture standard sparse coding). Given data 'D = {xx(1) ..... (a.("))..,,x!!! xP)) generated from a standard sparse coding generative model, i.e., x Dz +1, where dictionury aloms (columns of D) have unit norm and z is unconstrained, there exists scaling of the data such that it can be
represented using the K-Deep Simplex framework i.e., x = kI D + 17, where & E A
Proof. Consider the following scalar:
Σ (2))
=
max
RED
Normalizing data using x above gives us.
x 6
+ KM
maX >, 2x()
D2 + 7
By definition, defined above always satisfies Σ, 2 S 1, so let 1 1- &, Appending an
all-zeros dictionary atom to D, D = O] and assigning the residual to ã = [ S) gives us the
following:
Dz + nj where E A
implying that the original data be represented in the framework of KDS.
D.4 SpaDE
Sparsemax is a projection onto the probability simplex, which can be written as (see Proposition in [44])
Let z = Sparsemax (3))
(43)
1
1
Then, 2; ReLLUI)
Σ fj
+
(44)
IM
[M
JEM
25

---

SpaDE is defined using squared euclidean distances between an input vector and some prototypes (or landmarks) in imput space (Eq. 5), which gives
= -1/x 20,2
(45)
Sparsemax = ReLU
21(
M Σ wa)"' -1(12=l-
Σ
(w,11)) +
M
(46)
= (w))
(47)
where (x)) 2M(w; TM Ej
In w,). b. -1(1011- Σ3 w
w 12] M and M is the
set active indices, which is uniquely determined by the constraint Σ, Sparsemax (y); = (see
Proposition in [44] for uniqueness). Note that (2), (x() are both piecewise constant on regions of input space marked by the same choice of active indices.
Thus, SpaDE is equivalent to a ReLU SAE, but with a linear transformation and bias that are input- adaptive (piecewise constant). SpaDE is thus piecewise linear and continuous (continuity follows from the continuity of sparsemax). Note that this is a nontrivial result: despite appearing quadratic in input due the use of squared euclidean distances, SpaDE is piecewise linear function of the input. This result is also exact, and is NOT a first order Taylor series approximation.
However, SpaDE differs from ReLU SAE by using linear transformations defined with respect to local origin, which is uniquely determined by the set active SAE latents, similar to recent work on steering [80].
Since SAEs are completely described by their inner and outer optimization problems (see Theo- rem 3.1), we now describe these components for SpaDE.
The inner optimization (Eq. 4) for the SpaDE is as follows:
1
F(T, W. x)
Σ k - will2
+ |
21
(48)
8 E ni 0, Σ n. =
This resembles one-sided optimal transport with a squared 2-norm regularizer. This problem is one-sided because there is constraint on how much weight sits each prototype across different inputs (optimization is performed independently for each input). The squared 2 norm regularizer is known to lead sparse transport plans in the optimal transport literature (see [81]).
The outer optimization for SpaDE (Eq. 4) a locality-enforced version of dictionary learning called K- K-Deep Simplex (KDS) [45]. In this framework, the sparse code is constrained to belong to the probability simplex, i.c., z E A = {y E R2 Σ; y = 1, y4 2 Ovi}, while the dictionary atoms D are unconstrained. The distance- weighted L1 regularizer encourages each datapoint to use those dictionary atoms which are close to itself in euclidean distance, inducing a soft clustering bias. Even though this is different dictionary learning framework than standard sparse coding, it is expressive enough to capture the standard sparse coding setup, i.e., for any standard sparse coding problem, there exists equivalent KDS problem (see Theorem D.9 in Appendix).
While this outer optimization (KDS) is a different problem than the standard dictionary learning problem, it may be useful for interpretability since it has the following advantages:
1. It avoids shrinkage, since the LI norm of the sparse representation = =(x) is constrained to equal 1 for all inputs
2. Constraining the sparse code the probability simplex finds support in an oft-cited paper demonstrating the linear representation hypothesis in word embeddings under a random- walk based generative model of language [9]. Their main result (Theorem 2) shows that representations are convex combinations of concepts, as opposed to unconstrained linear combinations, which is better interpreted as assigning vectors (with magnitude and direction; alternatively, locations) to concepts rather than directions (without magnitude). This idea of concepts as vectors has also been demonstrated both theoretically and empirically in the final layer representations of language models [82].
26

---

Note how SpaDE satisfies the two data properties of nonlinear separability and heterogeneity:
1. The projection set S in SpaDE is the probability simplex, which admits edges/corners with varying levels of sparsity, thereby allowing the representation of heterogeneous concepts. For any choice of k E (1, .... 8), there are (i) choices of indices Mr for k-sparse representation, and points {x E R 2; 0, MK: EJEM, zj 1, 2 0} S A
which admit this level of sparsity, thereby capturing concept heterogeneityy
2. The receptive fields of SpaDE (see App. D.2.3) are local to each prototype (encoder weight vector), and are flexibly defined as the union of convex polytopes. This allows latents in SpaDE to become monosemantic to concepts which are nonlinearly separable from the rest of the data.
E Further Results
In this section, we present more detailed analysis of the results from each of our four experiments.
E.1 Separability Experiment
ReLU
JumpReLU
TopK
SpaDE
Normaltzed 1
Iterations
Iterations
Iterations
lterationn
Figure E.1: Evolution of normalized MSE with training iterations for various SAEs on the separability
avneriment Color intencity reonortional T. (darker colore imnly more dence SAR latente)
Racton deadd labents
Nonralzes MSE
teace
to
Stable
Statre
Figure E.2: Normalized MSE (normalized with variance of data), Stable ranks (of data correlations, latent correlations matrices), and fraction of dead latents a function of sparsity (Ln) for the separability experiment (Sec. 5.1)
Fig. E. shows the evolution of normalized MSE (NMSE- MSE normalized by the variance of data) with training iterations for each SAE, for different levels of sparsity. Note that denser representations (higher La and thus darker colors in Fig. E. lead to lower NMSE. While all SAEs end up at similar levels of NMSE, their ability to extract concepts from data is markedly different (as described in Sec. 4). A per-concept breakdown of training dynamics is shown in Fig. E.3. For comparison, this figure also includes the mean of the squared norm of each concept (which equals MSE if the SAE predicts the origin for all inputs), variance of each concept (which equals MSE if the SAE predicts the mean of each concept). Thus, SAEs whose MSE saturates at the concept variance are likely to be predicting the mean of the concept for all points, whereas when MSE goes below concept variance, the SAE explains within-concept variance. Also shown gray is MSE with respect to the center of each concept, which ideally must match concept variance if the SAE reconstructs all points (which is observed in most cases).
In Fig. E.2, final NMSE as function of sparsity (Lo) shows that while all SAEs have comparable MSE-sparsity curves dense representations (high Lp). TopK's NMSE goes down significantly more than others. This is consequence of TopK learning redundant solution, by just using two latents as an orthogonal basis to represent all data. Fraction dead latents show large numbers of dead latents at high sparsity levels for ReLU, JumpReLU and TopK, with this going down (exponentially)
27

---

as representations become more dense. However, SpaDE shows sigmificantly fewer dead latents at all levels of sparsity. Stable ranks of cosine similarities between latent representations of pairs of data points (data corr.), and between pairs of latents across all data points (latent corr.) show that SpaDE very high stable ranks, indicating high specialization of latents. The other SAEs have comparable stable ranks, all much lower than the desirable stable rank of 6 (equal to the number of clusters in data).
The SAE latent activation profiles for each concept are shown as histograms in Fig. E,4. While variations exist across concepts, there common structure to the profiles for each SAE (SpaDE appears pointy, indicating a second mode other than zero).
Cosine similarities between latent representations of pairs of data points shown for different levels of sparsity in Fig. E.5. Notice that SpaDE has the lowest cross-concept correlations of all SAEs, and these correlations do not decrease much especially in ReLU and JumpReL.U. The corresponding figure with similarities between pairs of latents across all datapoints is in Fig. E.6. Here, the number of dead latents increases with increasing sparsity, leading very few active latents (only active latents are shown). Broadly, note the decrease in co-occurrences with increase in sparsity- also note how ReLU and JumpReLU result in newer correlation structures with greater sparsity.
ReLU
MGE Wt CRTRT
JumpRe
SpaDE
Figure E.3: Training dynamics for each concept (column) across SAEs (rows) for separability experiment: colored solid lines MSE, with intensity of color proportional Lo- Gray lines show MSE of SAE predictions with respect to the center of each cluster; intensity is again proportional to Lo- Black dotted line shows the mean squared norm of each cluster, which would equal the MSE if the SAE predicted the origin for all datapoints. Red dotted line shows variance of each cluster, which again equals MSE if an SAE predicts the center of the cluster. Note that when model reconstructs data well, MSE wrt cluster center equals the variance of the cluster (as observed here)
28

---

Co
C1
22
C3
C4
C5
ReLU
60
101
Ru
10
177
Count
600
100
39
180
Topk
100
SpaDE
107
30"
3P
10m
10
13
10+
10m
rep.
Figure E.4: Histogram of latent representations for each concept of various SAEs the separability
Le-19,9
Gy1188
G-71877
4-177
1.00
RelU
050
0.0
1y:178
Ge117
1am16.4
1an14.1
Len
6y07.1
6a554
(e-1.
JumpReLL
1.00
050
1e-20.0
12-18.0
1s-26.0
A=-14.5
A-122.5
4-110
lo-2.
1.0
Topk
030
025
0.00
6-1148
4,-14.5
1,-14.5
1,-13.9
1,113.5
4-1133
L,-192
Lebbal
L=bb11
L,465
1.00
SpaDE
050
025
Figure E.5: Data correlations for various sparsity levels on the separability experiment: Pairwise cosine similarities between SAE latent representations of datapoints. White lines separate different concepts.
29

---

L,=19.
G-119
ReLU
Lym17.
G=17.7
L=1sA
L=141
G-7-1
L
L-22
G-19
Jump ReLL
Ly=20.a
Ly-18,
16,00
4=140
L=122
L--0.
Lds
675
Top
625
1a-14.
1a-145
1a-16.3
1a-13
1=-11.3
1on102
LanGB
SpaDE
075
025
Figure E.6: Latent correlations for various sparsity levels on the separability experiment: Pairwise cosine similarities: pairwise cosine similarities between different SAE latents, computed across data from all concepts.
30

---

E.2
Heterogeneity Experiment
The overall training dynamics (on data from all concepts) shown in Fig. E.8- note, again, that for low sparsity (high Lu, darker color) all SAEs reach similar levels of NMSE, but differ for higher sparsity levels. The per-concept breakdown concept variance and MSE with respect to the center of each concept is in Fig. E.3 The kink in gray lines precisely the point where the model transitions from learning to represent the mean, to learning to explain the within-concept variance, clearly demonstrating two phases in learning: learning the right scale for the data (since initial model predictions may not match the true scale of data), thereby predicting the mean well, followed by learning the distribution of the data.
MSE, and comparison with mean squared norm,
Fig. E. shows latent activation profiles for each concept and each SAE (k 32 in TopK). Since TopK with &o = 32 cannot allocate enough latents for large intrinsic dimension concepts, it increases activations on smaller number of concepts instead. Cosine similarities between SAE latent represen- tations for pairs of data points, and pairs of latents across all datapoints, is shown for varying levels of sparsity Fig. F, E. respectively. All SAEs (except JumpRel.U) do a decent job at reducing correlations between pairs of data points, but in the latent correlation plots, we see how Topk fails to adaptively allocate latents to heterogenous concepts, especially moderate levels of sparsity, while the other SAES do well- have different sized blocks in block-structured matrix.
Fraction dead latonts
Normalbed MSE
10
SOMBE
rank- Data
Stable Latent
Le
Figure E.7: Normalized MSE (normalized with variance of data), Stable ranks (of data correlations, latent correlations matrices), and fraction of dead latents a function of sparsity (Lo) for the
hatownnmoiin: avnarimsont San 57) JumpReLU
TopK
SpaDE
Normalized MIE
10
124
120
22P
Iterations
Iterations
iterations
Iterations
Figure E.8: Exolution of normalized MSE with training iterations for various SAEs on the heterogene- ity experiment, Color intensity proportional T. (darker colors imply more dense SAE latents).
31

---

Co: dim=6
Cj: d4m=14
Cz d0m=30
Cz: dim=62
Cg. dim=126
ReLU
MSE WTT cont. center
PC2
JumpReLU
11
13
Topk
SpaDE
117
Itera tions
Figure E.9: Training dynamics for each concept (column) across SAEs (rows) for heterogeneity experiment: colored solid lines are MSE, with intensity of color proportional to Lp- Gray lines show MSE of SAE predictions with respect to the center of each cluster; intensity is again proportional to La- Black dotted line shows the mean squared norm of each cluster, which would equal the MSE the SAE predicted the origin for all datapoints. Red dotted line shows variance of each cluster, which again equals MSE if an SAE predicts the center of the cluster. Note that when a model reconstructs data well, MSE cluster center equals the variance of the cluster (as observed here)
32

---

Dim = 6
Dim 14
Dim 30
Dim 62
Dim 126
10/
17
10
Re
12
107
127
337
JumpReLU
13
33
33"
tor
109
Count
10h
17
10"
18-
39'
Topk
17
12"
17
10
199
120
19
199
33"
34
22
10
100
SpaDE
190
19
Latent rep.
Figure E.10: Histogram of latent representations for each concept various SAEs on the heterogene- ity experiment.
Lo-452.8
=.11821
L=-111.3
Lo-87.0
44=44.0
1.0
ReLU
La
L==228.0
Lg=92.0
Lg=50.9
JumpReLL
1.0
24=247.1
12=128.0
Lo=64.0
Lo=32.0
1g=16.0
L,=8.0
L,=4.0
1.0
Topk
0.5
La- 161.9
La- 134.5
La -5.52.8
La-36.0
L,-22.1
Lam8.4
1,=1.5
1.0
SpaDE
0.5
Figure E. Data correlations for various sparsity levels on the heterogeneity experiment: Pairwise cosine similarities between SAE latent representations of datapoints. White lines separate different concepts.
33

---

Lg=508.0
4g=452.8
4==255.1
4=112211
1==111.3
L,=87.0
L.=44.0
1.0
ReLU
0.5
0.0
Lg-425.5
Lo- 37-88
Lg Lg 228.0
L--134.8
La Le-106.0
Lo-92.0
Lg=50.9
Jump
0.5
0.0
Lo-247.1
La- 128.0
-4-00.0
La 12.02.0
Lg-16.0
L.-8.0
L,=4.0
TopK
0.5
0.0
1o=161.9
Lo=134.5
4==52.8
Lo=36.0
Lg=22.1
L==8.4
L.-1.5
SpaDE
0.0
Figure 121 Latent correlations for various sparsity levels on the heterogeneity experiment: Pairwise cosine similarities: pairwise cosine similarities between different SAE latents, computed across data from all concepts.
34

---

E.3 Formal Language Experiments
In this section, we report several more results the formal language experimental setup. Specifically, we show how with changing sparsity of the latent code, fidelity metrics, e.g., normalized MSE scales changes and stable rank of both data and latent correlations changes (Fig. E.13); 3); how monosemantiiity changes, i.e., how FI scores averaged across latents and the concept they achieve maximum FI score on change (indicating their specialization that concept) (Fig. E. 14 Left); and how percentage of dead latents change (Fig. E.14 Right). These results are repeated at the concept-level, i.e., at the level of parts-of-speech, in Figs. 1. 1. Inline with results on heatmaps demonstrating correlation between sparse codes of samples from different concepts and between vector denoting which samples a given latent activates for, we retrieve results in Fig. E. 17, E.18 The results above are perfectly inline with our findings from the main paper, e.g., that SpaDE achieves highly monosemantic features. The new and intriguing results involve demonstrations how effective SpaDE can at discerning position of concept (part-of- of-speech) in sentence, when compared to other protocols which learn more uniform representation.
Further, we also provide 2D and 3D PCA visualizations of different SAEs' retrieved latents in two different manners: (i) assess which datapoints a latent activates for and project into low- dimensional space identified using PCA, and (ii) assess which latents datapoint activates, and project this activation vector. The former helps assess how monosemantic latents are, i.e., whether they activate for specific concepts, and the latter helps assess how specific latents are, i.e. whether datapoint only activates a specific latent and hence there is no regularity present. Results show most SAEs, when they perform well, organize latents in very structured manner (like tetrahedron), but SpaDE succeeds at this throughout.
Stable Rank Dota cort.
Stable Rank Labont cor:.
Normalize HSE
1.7
10a4
10
10P
10P
10
Figure E.1 Normalized MSE and Stable ranks as a function of sparsity in the Formal Language setup.
1.0
Monosemanticity (Avg. F1)
Fraction of Dead Latents
0.8
0.9
0.6
0.8
0.4
0.7
ReLU
JumpReLU
0.2
TopK
0.6
SpaDE
0.0
10a
10a
10a
10a
Lo
Lo
Figure 14: Monosemanticity (FI scores averaged over latents) and fraction of dead latents as function of sparsity for different SAEs in the Formal Language setup.
35

---

Noun
Pro.
Verb
10a
Normalized MSE
10-1
ReLU
JumpReLU
10-4
SpaDE
TopK
Adv.
Adj.
Conj.
10"4
Normalized MSE
10-1
10
10a
10
10a
10a
Lg
Lo
Lo
Figure E.15: Normalized MSE decomposed by concepts (parts-of-specch) and plotted function of sparsity in the Formal Language setup.
(holitat
Ort
3.6
Felli
JamphetLi
102
SpaDE
Text
Figure 1. Percentage of Dead Latents decomposed by concepts (parts-of-speedd and plotted as function of sparsity in the Formal Language setup. Note that in such concept-conditione count of dead latents, one ends up counting both the latents that are always inactive and ones that are inactive for the specific concept under consideration.
36

---

ReLU
JumpReLU
Topk
SpaDE
Figure E.17: Correlation between sparse codes of different concepts (parts-of of-seech))) in the Formal Language setup. Datapoints for different concepts are sorted according to which concept they come from (using a predefined order on the parts-of-speech and according their position sentence, hence highlighting position dependence. Lines demarcate boundaries at which tokens corresponding different concepts start end.
37

---

ReLU
JumpReLU
Topk
SpaDE
2
Correlation between which datapoints a latent activates for in the Formal Language setup. Latents are sorted according to which concept (part-of-speech) they most strongly activated for (as measured using F1 Fl-score). White lines demarcate boundaries at which latents of different concepts start end.
Figure E.18:
ReLU
-40
600 62 533
+20
130
40 -25 +00
16
90
08
00
62
65
-2 00 0 M
Jump
42
0
32
430 30
Topk
6-117
SpaDE
ROHAN
2
Figure E9 2D PCA visualization of sparse codes corresponding to different concepts (parts-of- speech).
38

---

ReLU
JumpRato
593577
TopK
SpaDE
weria 10
Figure E.20: 3D PCA visualization of sparse codes corresponding to different concepts (parts-of- speech).
61-111
4-31
L-14
94
ReLU
5491
4+5)
6-32
Ju
-28
1002
L-003
6-383
4-23
C 2
Topk
Ct
SpaDE
64
20
600
Figure E.21: 2D PCA visualization of a matrix whose elements capture which tokens latent activates for. That is, which concepts (parts-of-speech) the latent specialized towards, if any.
39

---

Linda
ReLU
Topk
Sec
SpaDE
Figure E.22: 3D PCA visualization of a matrix whose elements capture which tokens a latent activates for. That is, which concepts (parts-of of-seec))) the latent specialized towards, if any.
40

---

E.4 Vision Experiment
In this section, we show, visually, the concepts SpaDE has learnt the vision experiment, by visualizing feature attribution maps for inputs from each class from Imagenette. We perform this visualization for the top concepts for each class for five classes- Tench (Fig. E.23), Chainsaw (Fig. F.24), Church (Fig. E.25), Golf (Fig. H.26) and Springer (Fig. F.27)).
Top-8 Concepts for Tench
Figure E.23: Feature Attribution maps for monosemantic latents from SpaDE on the Tench class
41

---

Top-8 Concepts for Chainsaw
COOM
Figure E.24: Feature Attribution maps for monosemantic latents from SpaDE on the Chainsaw class
42

---

Top-8 Concepts for Church
Figure E.25: Feature Attribution maps for monosemantic latents from SpaDE on the Church class
43

---

Top-8 Concepts for Golf ball
feiin
MGP
Pogy
CAION
Figure F.26: Feature Attribution maps for monosemantic latents from SpaDE on the Golf class
44

---

Top-8 Concepts for English springer
Figure E.27: Feature Attribution maps for monosemantic latents from SpaDE the Springer class
45

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We provide both theoretical and empirical justification for our claims. Theory is in Sections and 4 and in Appendix D, while experiments are in Section 5 and Appendix
E.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made
in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: The limitations of our work are clearly discussed Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken low lighting. a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Our results about the limitations existing SAEs are proved Appendix through derivations of the receptive field structure.
Guidelines:
The answer NA means that the paper does not include theoretical results.
46

---

All the theorems, formulas, and proofs in the paper should be numbered and cross
referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes)
Justification: We include experimental setups in brief in Section 5, and provide extensive details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access hosted model (e.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide anonymized code for subset of the experiments (formal language experiment) for review. We will open source all our code after the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
47

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc. ctc/public/guidees CodeeSumissionnPol
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments
should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe extensive details about experimental setup in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail
that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We show error bars whenever the experiments weren't it too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments? Answer: [Yes]
48

---

Justification:
We include relevant details on compute resources in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //t::///euuure wwwrrps...... htt.///pbbb.t/tccccccieeeeeeeeee Answer: [Yes]
Justification: We confident that the paper conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work describes limitations in existing approaches to interpret large models and potential ways to mitigate these, and does not directly have any potential harmful impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA]
Justification: We describe limitations of tools used to interpret large models (SAEs)- our work poses no risk of misuse.
49

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include all relevant citations for existing models/ datasets that use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., wehsite), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license
dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New synthetic datasets created are included in the code and described in Appendix C. New SAE (SpaDE) is included in the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our research does not involve human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
50

---

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required. Answer: [NA]
The core contributions of our research- theoretical limitations of SAEs, de- sign of a new SAE incorporating data properties, and demonstrating our claims through experiments- does not involve LLMs.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https: htt::///eesssc...eeee cc/Conferenccccc////////c/cccc// http://///a.aeeeeeooo/o///////// /LLM) for what should or should not described.
51


Reviews:
[{"review_id": "0Jg8HV525U", "reviewer": "Reviewer_9gLg", "summary": "The authors investigate the limitations of Sparse Autoencoders (SAEs) in terms of the types of concepts they can detect. They attribute these limitations to the architectural design of SAEs, which imposes specific assumptions on the structure how features are encoded. To address these constraints, the authors propose a new SAE, SpaDE, which is able to address previous limitations on data assumptions.", "questions": "- How do you interpret the occurrence of more different colored clusters in SpaDE compared to the others?\n\n- When analyzing SAEs, which true structure of the data should be considered?\n\n- Does good data selection for analysis always depend on the task?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n- The paper is well-written, logically structured, and easy to follow. Each step is clearly motivated and illustrated.\n\n- The work is timely and addresses a relevant problem, offering a better understanding of shortcomings of SAEs.\n\n- The figures are overall clear, well-designed, and effectively support the main points.\n\n# Weaknesses\n\n- The domain of the work is not clearly specified in the first part of the paper and only becomes explicit in Sections 5.3 and 5.4. Referencing this earlier would improve clarity.\n\n- It is not entirely clear what is meant with \"true structure\" of concepts (l. 126).\n\n- In Figure 5, row (c), the meaning of the different cluster colors is unclear and should be clarified in the caption or main text.", "was_revised": true, "final_justification": "I recommend accepting the paper. The topic is timely and relevant, and the paper is well developed.", "rebuttal": "We thank the reviewer for their review and feedback! We appreciate the reviewer’s assessment of our work as “timely”, “addresses a relevant problem” and our paper as “well written”, “easy to follow” with “well-designed” figures. We thank the reviewer for highlighting our paper’s main contributions: 1) limitations of SAEs arise from architecture of SAEs imposing assumptions on concept structure, 2) highlighting how specific limitations can be addressed through SpaDE.\n\nWe provide responses to specific comments/ questions by the reviewer below.\n\n> **The domain of the work is not clearly specified in the first part of the paper and only becomes explicit in Sections 5.3 and 5.4. Referencing this earlier would improve clarity.**\n\nThank you for the suggestion! We emphasize that since our theoretical claims did not require assumptions specific to a given modality, we intentionally framed our paper to be domain-agnostic in its narrative. This is because our main contribution—the duality between geometric structure of concepts in model activations and an SAE that recovers these concepts—is general and not constrained to specific domains. This is demonstrated in our experiments that cover a broad range of experimental settings, ranging from synthetic datasets (gaussian clusters, Sections 5.1, 5.2), to semi-synthetic datasets (nanoGPT activations on formal languages, Section 5.3), and to naturalistic settings (DINOv2 on ImageNette, Section 5.4). \n\nSince Sparse Autoencoders (SAEs) are being used across multiple domains for interpreting neural networks, including language models (examples in line 30), vision models (line 31), and protein autoregressive models (line 32), we believe our claims can be meaningful and relevant for all such communities. We hence believe having a domain-agnostic narrative is ideal for our work.\n\n> **It is not entirely clear what is meant with \"true structure\" of concepts (l. 126).**\n\nFair point. By the phrase “true structure” of concepts, we intended to make an explicit distinction between the ground truth structure in accordance with which concepts are organized in a model’s activations versus the structure assumed by different SAEs. By making this distinction, we wish to highlight a core contribution of our work: that SAEs make assumptions about concept structure that may not hold in the ground truth structure in model activations, and an SAE cannot recover concepts whose “true structure” in model activations does not match the “assumed structure” of concepts by that SAE. We will make sure to clarify the writing to better emphasize what we mean by “true structure” in the paper.\n\n\n> **In Figure 5, row (c), the meaning of the different cluster colors is unclear and should be clarified in the caption or main text.**\n\nThank you for pointing this out! Figure 5 demonstrates the limitations of SAEs occurring due to a mismatch between SAE assumptions and ground-truth concept structure, when trained on data with nonlinearly separable concepts constructed using synthetic gaussian clusters.  While rows (a) and (b) quantitatively and qualitatively show the monosemanticity (concept selectivity) of different latents in each SAE, row (c) captures the monosemanticity of the entire sparse code of all SAEs. \nThe matrix in row (c) shows the cosine similarity between sparse codes of pairs of datapoints—ideally, different latents activate for different concepts leading to block-diagonal structure in this matrix. The cluster colors in row (c) are the results of spectral clustering on this matrix of pairwise cosine similarities: all points with the same cluster color on this plot activate a common set of SAE latents. Having multiple concepts (gaussian clusters) with the same color indicates that the SAE is unable to distinguish these concepts well in its sparse code, and thereby unable to monosemantically recover these concepts. As predicted by our analysis, ReLU and JumpReLU SAEs show multiple clusters with the same color, and are hence unable to recover nonlinearly separable clusters. We will include a brief intuitive description of the meaning of cluster colors in the final version of the paper.\n\n---\n---\n\n> **How do you interpret the occurrence of more different colored clusters in SpaDE compared to the others?**\n\nGreat question, we assume this question is in reference to the different colored clusters in SpaDE in Figure 5c (please correct if we are misinterpreting the comment)! \nSpecifically, Figure 5 is aimed at highlighting the limitations of SAEs due to a mismatch between SAE assumptions (linear separability assumed by ReLU and JumpReLU SAEs) and the property of nonlinear separability. Accordingly, points which share the same color in Figure 5 (c) are those that activate a common set of SAE latents. The occurrence of different colors for different clusters (concepts) in SpaDE, while ReLU and JumpReLU SAEs assign similar colors to different clusters, highlights the inability of ReLU and JumpReLU SAEs in recovering nonlinearly separable concepts in a monosemantic fashion. \nHaving multiple concepts share the same color indicates that SAE latents are polysemantic (respond to multiple concepts), which is clearly observed for ReLU and JumpReLU SAEs (and TopK SAE to some extent). SpaDE shows different concepts (gaussian clusters) with different colors, indicating that the latents learnt by SpaDE are more monosemantic than the other SAEs when concepts are nonlinearly separable. It is worth noting that SpaDE is meant to be a baseline to show that the limitations arising in existing SAEs are specific to their architecture, and can be avoided.\n\n\n> **When analyzing SAEs, which true structure of the data should be considered?**\n\nThank you, we believe this is exactly the right question to ask after reading the paper, and it’s one that has sparked extensive debate among the authors!\n\nTo answer directly: there may be no free lunch here. The true structure of model activations is not fixed, it depends on the architecture, training regime, and task. As such, we believe no single SAE architecture is likely to perform well across all settings. Instead, the question of which structure the data contains should guide the choice of SAE.\nThis is the central message of our work: there exists a duality between the geometry of the data and the inductive biases of the SAE used to probe it. Some SAEs assume linear separability, others enforce fixed-dimensional sparsity, and each of these choices determines not just what can be extracted, but what is even visible.\n\nA corollary to this is that we should therefore begin by asking: what is the likely structure of this representation space? Only then does the choice of SAE become meaningful. And more broadly, **we hope this work encourages the field to invest more deeply in understanding the geometric organization of latent spaces**. Much effort has gone into designing SAEs, but comparatively little has been spent understanding the structure they operate on. \n\n\n> **Does good data selection for analysis always depend on the task?**\n\nThank you for the question. Unfortunately, we are slightly unclear about how to interpret it, but provide a response based on the following (paraphrased) interpretation: why did we choose the specific synthetic modalities included in the paper (which are gaussian clusters) to analyse SAEs?\n\nWe note that our main goal in the experiments was to highlight the biases/limitations of existing SAEs, for which we designed synthetic data to capture specific properties–nonlinear separability and concept heterogeneity–that are complementary to the assumptions of existing SAEs. Our choice of datasets was motivated to study the SAEs better and bring out their shortcomings. The SAEs can themselves be used in any task (interpreting language models/ vision models/ protein autoregressive models, etc) and the limitations we highlight will persist. \n\nIf this interpretation is different from what the reviewer had in mind, we would be happy to hear back from them and provide an updated response!\n\n\n> **For Figures 1-3, 5-6, and all Tables 1-2, captions should be lower case (except for first word and proper nouns).**\n\nThank you for pointing this out! We will make the changes in the final version of the paper.\n\n**Summary**: We thank the reviewer again for their questions and suggestions! We would be happy to address any further questions during the discussion period."}, {"review_id": "IHQjcBVcVd", "reviewer": "Reviewer_nypq", "summary": "The authors challenge the common assumption that Sparse Autoencoders (SAEs) are neutral tools for uncovering meaningful concepts in neural representations. By recasting SAE training as a bilevel optimisation problem, they show that each SAE architecture implicitly projects data onto a specific geometric constraint set - e.g., the positive orthant (ReLU), sparse angular subspaces (TopK), or hypercubes (JumpReLU). These constraints shape the encoder's receptive fields, biasing the model towards detection of concepts that conform to assumptions, such as linear separability and fixed intrinsic dimensionality. The authors conduct a series of experiments across a range of settings to empirically verify this framework. They propose a new architecture SpaDE, which uses the sparsemax nonlinearity to handle nonlinear separability and concept heterogeneity, and demonstrate it consistently outperforms other SAEs in recovering monosemantic, interpretable features.", "questions": "Co-occurence and overlapping concepts: As noted by the authors in Section 6, the analysis focuses on mutually exclusive concepts, where the presence of one concept implies the absence of others. While the central argument about SAEs being biased towards certain concepts due to their architectural constraints clearly holds in general, regardless of the data distribution, I'm curious how the empirical failure modes (e.g., lack of monosemanticity) differ qualitatively in these non-exclusive scenarios (e.g., where concepts are overlapping/compositional)? In particular, do real-world concept distributions tend to violate the separability and dimensionality assumptions identified? I would expect this to be the case, but it would be valuable to understand whether these failure trends persist in these non mutually exclusive settings.", "limitations": "Yes, very well elaborated on in Section 6.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:** \n- Focus on latent space geometry and implications for interpretability research: connecting the encoder receptive field to assumptions about the underlying geometry of the concepts provides a compelling argument that latent space geometry must be prioritised when designing SAEs. The paper makes it clear that this is a fundamental limitation of current SAEs, and the theoretical framing is timely and well-needed in the interpretability field. \n- Formalisation via bi-level optimisation: the paper presents a principled reformulation of SAE training as a bilevel optimisation problem, where the encoder implicitly solves a projection problem over an architecture-specific constraint set. This offers a clean, mathematically grounded explanation for how architectural choices induce biases in feature extraction, and is broadly applicable to multiple existing SAE variants. \n- Broad range of evaluation settings: experimental evaluations spans a wide range of settings - from synthetic separability setups to semi-synthetic formal language models and real-world visual representations. The broadness of the evaluation approach provides strong support for the generality of the claims. \n- Principled design of SpaDE: the architecture proposed is not just proposed heuristically, but instead motivated by the geometric limitations identified in the earlier framework. SpaDE is also shown to resolve failure modes observed in prior models. \n- Clarity of empirical failures: the empirical results convincingly demonstrate how specific structural assumptions (e.g., linear separability, fixed concept dimensionality) lead to systematic failure modes in prior SAEs, especially in more complex settings. \n\n**Weaknesses:** \n- Assumptions underlying SpaDE not extensively tested: as highlighted by the authors in Section 6, SpaDE assumes that Euclidean distance is a meaningful distance metric in concept space, and that concepts project well onto the probability simplex. These assumptions may not hold across domains with more complex geometries, however, the authors do acknowledge this and this is likely to be more diagnostic of a broader, open problem, as much of the current literature implicitly assumes Euclidean geometries.", "was_revised": true, "final_justification": "The rebuttal clarified SpaDE’s intended scope, addressed generalisation to overlapping concepts, and acknowledged its own geometric assumptions. With no substantive issues remaining, I maintain my positive assessment and recommendation for acceptance.", "rebuttal": "We thank the reviewer for their review and comments! We love the reviewer’s framing of our main message: “a compelling argument that latent space geometry must be prioritised when designing SAEs”. We appreciate the description of our work as a “principled reformulation of SAE”, “timely and well-needed” having a “broad range of evaluation settings”. We also appreciate that the reviewer highlighted our main contributions: 1) SAE architecture constraints bias the model towards concepts that conform to its assumptions, and 2) handling nonlinear separability and concept heterogeneity using SpaDE, a geometry-driven SAE.\n\nWe provide answers to specific questions/comments below.\n\n\n> **Assumptions underlying SpaDE not extensively tested: as highlighted by the authors in Section 6, SpaDE assumes that Euclidean distance is a meaningful distance metric in concept space, and that concepts project well onto the probability simplex. These assumptions may not hold across domains with more complex geometries, however, the authors do acknowledge this and this is likely to be more diagnostic of a broader, open problem, as much of the current literature implicitly assumes Euclidean geometries.**\n\n\nThis is a great point! As noted by the reviewer, our main goal in this work was to highlight the assumptions made by existing SAEs about concept geometry, and the inability of SAEs to recover concepts when their assumptions do not match the structure of concepts in model activations. To this end, we present SpaDE as a concrete example of designing an SAE which can capture two reasonable data properties—nonlinear separability and concept heterogeneity—that other SAEs fail to capture. Using our insights on SpaDE indicates that SpaDE also makes assumptions about concept structure: SpaDE assumes that Euclidean distance is useful in concept space (as rightly pointed out by the reviewer). Since we use SpaDE as a useful baseline to highlight the limitations of existing SAEs, as the reviewer noted, we *do not* claim that SpaDE is a superior SAE. We merely aim to suggest that when the data follows priors assumed by SpaDE (or for that matter any other SAE), it can be expected to perform better.\n\n---\n---\n\n> **While the central argument about SAEs being biased towards certain concepts due to their architectural constraints clearly holds in general, regardless of the data distribution, I'm curious how the empirical failure modes (e.g., lack of monosemanticity) differ qualitatively in these non-exclusive scenarios (e.g., where concepts are overlapping/compositional)? In particular, do real-world concept distributions tend to violate the separability and dimensionality assumptions identified?**\n\nThank you for this observation! As rightly highlighted by the reviewer, our claim about SAEs being biased towards certain concepts is a property of their architecture, and thus holds for any data distribution. We use non-overlapping concepts as a simple case with real-world examples (such as mutually exclusive classes or children of the same parent in a hierarchical generative process) to highlight the limitations of existing SAEs across diverse domains—from synthetic gaussian clusters to semi-synthetic nanoGPT activations on formal languages to naturalistic DINOv2 model activations from ImageNette. Our claims about SAE biases will also hold for overlapping concepts if we generalize the definition of separability from (current) *distinguishing one concept from other concepts (akin to multiclass classification)* to (improved) *distinguishing between the presence and absence of a concept (binary classification for each concept)*.\nWe will formally highlight the significance of SAE biases in the presence of overlapping concepts in future work. As for the real-world concept distributions that violate separability and dimensionality assumptions, we highlight examples from the literature on lines 133, 139 and 140.\n\n**Summary**: We thank the reviewer again for their thoughtful comments! We would be happy to answer any further questions during the discussion period."}, {"review_id": "Ez1cxnoJt4", "reviewer": "Reviewer_E5bk", "summary": "The authors show that different SAE architectures implicitly bake in different assumptions about the structure of model representations, via an analysis of the SAE training objective. They then develop a new architecture, called SpaDE, which enables concepts to be discovered even when they exhibit nonlinear separability and heterogeneity. In various settings, the authors find that SpaDE can improve over existing SAE architectures.", "questions": "* This paper focuses on SAEs, but there is some momentum in mechanistic interpretability away from SAEs on their own and more towards models which capture aspects of model computation like CLTs. Do you have any thoughts on the applicability of your framework to more general types of interpreter models? It seems like the analysis could apply pretty directly, since CLT encoders are the same as SAE encoders. But more broadly, I wonder if there are other more general ways that your point -- that interpreter model architecture determines what we can discover about models -- could apply to the design of other sorts of sparse interpreter models.\n* Do you think your work could explain any potentially-pathological results that have been described about SAEs? Could our choice of architecture be leading to feature splitting, or to non-atomic features being learned, that could be ameliorated with a different choice of architecture?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths\n* The paper is exceptionally well-written and well-constructed, with beautiful figures and diagrams.\n* The core point, that SAE architectures make implicit assumptions about the data, is a highly valuable one. The paper provides a useful framework for reasoning about the assumptions that we make when designing interpreter models. The SAE architecture that its analysis leads to is also an interesting one.\n\n### Weaknesses\n* The language model evaluations of SpaDE are a little lighter than what most papers introducing an SAE architecture conduct (scaling laws, sparsity vs. reconstruction frontiers, etc.). It would be good to ultimately have a more detailed empirical study of SpaDE on full-scale LLMs, but I recognize that SpaDE is not the only contribution of the work, and the broader theoretical/conceptual point about the relationship between SAE architecture and assumptions about the data is arguably more important.", "was_revised": true, "final_justification": "I found the paper to be solid from the beginning, and the authors' response was mostly just answering some higher-level questions I had had. I'm keeping my score at a 5. The paper is well-executed and interesting.", "rebuttal": "We thank the reviewer for their review and feedback! We are glad that they found our work’s “core point … is highly valuable”, and the paper “exceptionally well-written”, with “beautiful figures”. We appreciate that they highlighted our core contributions: 1) different SAEs make different assumptions about concept structure,  and 2) SpaDE, a new SAE, captures concepts satisfying nonlinear separability and heterogeneity.\n\nWe provide responses to specific questions and comments below.\n\n\n> **The language model evaluations of SpaDE are a little lighter than what most papers introducing an SAE architecture conduct (scaling laws, sparsity vs. reconstruction frontiers, etc.). It would be good to ultimately have a more detailed empirical study of SpaDE on full-scale LLMs, but I recognize that SpaDE is not the only contribution of the work, and the broader theoretical/conceptual point about the relationship between SAE architecture and assumptions about the data is arguably more important.**\n\nThank you for raising this important point! As rightly pointed out by the reviewer, the primary goal of our paper is to highlight the biases in SAEs that arise due to their architecture, limiting the concepts they can recover from data. We analytically obtain the assumptions of popular SAEs, and propose SpaDE as a concrete example of constructing an SAE incorporating specific properties of the data—nonlinear separability and concept heterogeneity—to highlight the limitations of existing SAEs. We perform extensive empirical analyses across multiple modalities to validate our claims: by training existing SAEs and SpaDE on synthetic gaussian clusters data, semi-synthetic nanoGPT activations on formal languages, and naturalistic DINOv2 model activations on ImageNette. Our experimental analysis includes sparsity-reconstruction frontiers (Figures E2, E7, E13 in Appendix). We plan to scale up our experiments with SpaDE to full-scale LLMs in future work.  \n\n---\n---\n\n> **Do you have any thoughts on the applicability of your framework to more general types of interpreter models?**\n\n\nWe share your intuition! Our main contribution in this paper is that SAEs are biased toward identifying concepts that match their prior assumptions. Our insights into SAE biases are a property of the encoder architecture, specifically of the pre-image of the encoder map, which is formalized using receptive fields of encoder latents (Definition 3.2, line 110). The insights into biases of encoder architecture will thus directly translate to Cross-Layer Transcoders (CLTs) that use the same encoder, but are trained on a different objective. For instance, a ReLU-CLT or JumpReLU-CLT both make the same assumption we highlight for (Jump)ReLU-SAEs: concepts are linearly separable from the rest of the data in the input layer to the CLT. This insight about the interpreter architecture limiting the concepts that can be recovered is also true in greater generality and has to do with the expressive power of the tool—any tool that seeks to interpret model latents by decomposing them into monosemantic, human-understandable concepts, will be able to do so only for certain kinds of concepts that its architecture allows. Being aware of this allows us to think carefully about the structure of concepts in concept space and design tools that are expressive enough to recover concepts of interest.\n\n\n\n> **Do you think your work could explain any potentially-pathological results that have been described about SAEs? Could our choice of architecture be leading to feature splitting, or to non-atomic features being learned, that could be ameliorated with a different choice of architecture?**\n\nGreat point! Through our work, we seek to highlight the assumptions made by SAEs about the structure of concepts. While we do not discuss this in our paper, it is likely that pathological results using SAEs, such as feature splitting, may be a consequence of a mismatch between the concept structure in data (model activations) and the structure assumed by specific SAEs. There may also be other contributing factors to feature splitting such as choice of SAE width, training objective (sparsity regularizer), etc.\n\n**Summary**: We thank the reviewer for their review and suggestions! We would be happy to answer any further questions during the discussion period."}, {"review_id": "MlOlqnm35n", "reviewer": "Reviewer_Bs68", "summary": "This paper critically examines the ability of Sparse Autoencoders (SAEs) to reveal meaningful concepts in neural network representations. While SAEs are commonly used in interpretability research, the authors argue that their ability to uncover concepts is fundamentally constrained by the structural assumptions built into their architectures. The paper frames SAE learning as a bilevel optimization problem, revealing how design choices, particularly the non-linearities used and the structure of the latent space, act as inductive biases that filter which types of concepts can be detected.\n\nThe authors identify two core challenges to concept discovery: (1) real-world concepts may not correspond to linear directions but may require modeling non-linear manifolds, and (2) concepts may vary in complexity and intrinsic dimensionality, i.e., heterogeneity. Through extensive experiments, ranging from controlled synthetic setups to semi-synthetic and natural datasets across both vision and language domains, the authors show that standard SAEs often fail to recover ground truth concepts when these properties are not aligned with their architectural assumptions. To address this, they propose a novel SAE variant that better accommodates heterogeneity and non-linearity, demonstrating improved concept recovery and supporting their theoretical claims.\n\nOverall, the paper challenges the notion of a universal SAE architecture and emphasizes the importance of architecture-specific choices in concept-based model interpretability.", "questions": "1. On the necessity of non-linear concept modeling:\nPrior work (e.g., [42]) suggests that many meaningful concepts are approximately linear. Could you clarify why it is essential to model non-linear concepts rather than decomposing them into combinations of simpler linear ones? Maybe the non-linear concept appears a linear one in a later layer?\n\n2. On concept heterogeneity and latent usage:\nYou argue that complex, heterogeneous concepts activate multiple latent units. How does this square with the goal of achieving monosemanticity, where ideally each latent corresponds to a single interpretable concept? Do you view these complex concepts as superpositions of atomic ones?\n\n3. Assumption of one concept per input:\nMany of your experiments seem to rely on the assumption that each data point corresponds to a single underlying concept. Could you justify this assumption, or discuss how your approach generalizes to settings where multiple concepts are present simultaneously?\n\n4. On interpretability and steerability goals:\nSince the paper is motivated by the desire to improve interpretability and steerability, have you considered evaluating your method on benchmarks or tasks that directly measure these goals? Without such evaluation, how should we assess the practical gains?\n\n5. On expectations of concept recoverability:\nYou use pre-defined, often class-based concepts to evaluate SAE performance. However, models like DINO are trained without labels and may not encode class concepts directly. How do you justify the use of these expected concepts, and do you have evidence that the model itself encodes them in a disentangled or monosemantic way?\n\n6. Clarification on Experiment 1 setup and goals:\nCould the non-linear clusters in Experiment 1 not be interpreted as mixtures of simpler linear Gaussians? If so, why is it preferable to directly recover the non-linear shapes, rather than their linear components?", "limitations": "Limitations are discussed, but no negative societal impacts are mentioned.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper presents a broad and well-designed experimental evaluation, spanning both vision and language domains, and progressing from controlled toy tasks to complex real-world settings.\n- The presentation quality is excellent, with clear visualizations and intuitive illustrations that make complex ideas accessible.\n- Theoretical insights are not only discussed abstractly but also used constructively to develop a new SAE architecture that performs better empirically.\n- The paper brings valuable theoretical perspective by analyzing how architectural choices in SAEs induce biases over feature geometry, including latent component receptive fields and separability assumptions. This kind of reflection is important for developing more rigorous interpretability methods.\n\nWeaknesses:\n- It is unclear whether non-linearity and heterogeneity are fundamentally problematic for existing SAEs. Prior work [42] suggests that many meaningful concepts are in fact linear, and standard SAEs are already capable of modeling multi-dimensional representations when needed. In many interpretability settings, linear concepts are preferable due to their simplicity.\n- The authors argue that complex concepts must activate many latent units, which seems to contradict the common goal of having one latent per concept. This only makes sense if such “complex” concepts are composed of more atomic parts, then each latent might represent one of these atomic concepts.\n- Many experiments assume that each data point reflects a single concept. This is an unrealistic assumption, as data often contains multiple overlapping features. This weakens the motivation for their SAE architecture, the overall conclusion, and practical significance of the evaluation.\n- In Experiment 1, the “non-linear” Gaussian clusters could reasonably be interpreted as combinations of the “linear” Gaussians. The authors argue that only by discovering the \"non-linear\" clusters one achieves monosemanticity, but this is debatable. A simpler decomposition might actually be preferable.\n- The goal of SAEs is to improve interpretability and steerability, but the paper does not evaluate either in terms of established benchmarks or automated interpretability measures. As a result, it’s unclear how the proposed model advances the ultimate goals of this line of work. Or how each SAE architecture is limited or improves interpretability/steerability.\n- The evaluation focuses on recovering pre-defined concepts, but it is questionable whether these are meaningful or expected (e.g., whether ImageNet classes are the right targets for Dino representations). Arguably, the true latent features may be more abstract.\n- Some theoretical claims and explanations lack clarity or are not well-motivated. For example, the reason why the TopK selection yields angular separation could be explained more directly.\n\n\nMinor Comments:\n- In Eq. (1), the variable z depends on the input x, but this dependency is not explicitly shown.\n- In Eq. (2), consider noting that the TopK SAE (used in the experiments) does not include an encoder bias term.\n- Table 1 reuses the variable x for the latent code, which previously denoted the input. This is confusing.\n- Line 87: Is “orthogonal” projection really orthogonal? Please check.\n- Definition 3.2: The term “neuron” is ambiguous since the paper focuses on latents, not neurons. Also, $f^{(k)}$ could be interpreted as a derivative. Is it $\\mathbb{R}^d \\rightarrow \\mathbb{R}$ or otherwise?\n- Line 127: Typo: should be “is” instead of “in”.\n- Line 133: The term “onion” feature is unclear, consider explaining or rephrasing.\n- Table 3: Could TopK SAE not handle some heterogeneity by selectively activating k latents? Never more than $k$ latens can be active, but fewer could be.\n- Potentially use formal equation references with parentheses, e.g., “Eq. (1)” rather than “Eq. 1”.\n- Figure 5 caption: Missing whitespace at the end of a sentence.\n- Line 235: What does PCFG stand for? Please define.\n- Inconsistent casing: “Row” is capitalized in Sec. 5.3, but lowercase in Sec. 5.4.\n- Line 245: Capitalize “MSE”?\n- Figure 6: In row (b), TopK and SpaDE appear visually very similar (the sparsity-MSE trade-off). This similarity could be addressed or acknowledged.", "was_revised": true, "final_justification": "This paper delivers valuable theoretical and empirical analysis on how architectural assumptions in Sparse Autoencoders constrain the kinds of concepts they can recover, offering a constructive new variant (SpaDE) as an example. The work is notable for its breadth of experiments, spanning vision and language tasks. However, the paper is dense, and at times the motivation for individual experiments and the interpretation of results are not sufficiently clear, leaving readers uncertain about what an SAE should ideally recover in a given scenario (e.g., whether a toy cluster should represent a pure monosemantic concept or a composite of atomic ones, and when non-linear concepts should be expected). These ambiguities partly reflect open questions in sparse dictionary learning and concept interpretability, but they also make it harder to assess the practical impact of the findings. The authors’ focus is on improving theoretical understanding rather than benchmarking interpretability or steerability gains in real-world large-scale settings, which is a valid and important aim, though its downstream utility remains to be demonstrated. Overall, the paper makes a meaningful conceptual contribution to the interpretability literature, but could benefit from clearer framing of expectations, stronger guidance to the reader through the dense material, and discussion of how the improved understanding might concretely inform the design and evaluation of SAEs in practice.", "rebuttal": "We thank the reviewer for their detailed and insightful review and comments! We greatly appreciate that they find our work provides “valuable theoretical perspective”, “presentation quality is excellent”, “broad and well-designed experimental evaluation”. We also appreciate that they highlighted our main message: SAEs’ ability to extract concepts is constrained by structural assumptions in their architecture.\n\nWe address specific comments and questions below.\n\n> **unclear whether non-linearity and heterogeneity are fundamentally problematic for existing SAEs.**\n\nThank you for the comment! We break down our response into two parts, addressing the comments on nonlinearity and heterogeneity one-by-one.\n\n- **Non-linearity**: While the reviewer is correct that many “meaningful concepts” encoded in neural networks have been shown to be linear, i.e., they satisfy the linear representation hypothesis (LRH), there is growing evidence for models encoding concepts outside the scope of LRH (e.g., magnitude-dependent onion features, Csordas et al 2024). In fact, we emphasize that no general proofs force a neural network to represent all concepts in a linearly accessible manner. To this end, we both formally and empirically show that ReLU and JumpReLU SAEs (though not TopK SAEs), which assume linear separability of concepts, fail at recovering nonlinearly separable concepts! \n\n\n- **Heterogeneity**: The reviewer noted that “standard SAEs are already capable of modeling multi-dimensional representations when needed”; however, we emphasize that this statement is only true for ReLU and JumpReLU SAEs! We show in our work that TopK SAEs fail to uncover low-dimensional concepts from a dataset which contains concepts with intrinsic dimensionality lower than the value of K in the topK activation function (Figure 6a).\n\nMore broadly, we emphasize the core contribution of our work: when neural networks encode concepts outside the scope of LRH, an SAE’s success at extracting them will depend on assumptions it makes about concept structure!\n\n\n> **Complex, heterogeneous concepts activate multiple latent units: at odds with monosemanticity?**\n\n\nWe would first like to clarify that we do not use the term “complex concepts” in our work. We believe the reviewer is referring to our analysis of multi-dimensional concepts—which are often composed of several atomic (1D) concepts. Collapsing these into a single latent risks losing access to meaningful subcomponents. For instance, Pan et al. (2025) show that “safety” involves dimensions like “refusal behavior,” “hypothetical narrative,” and “role-playing,” which are also useful beyond safety (e.g., persona control). Thus, it is natural for such concepts to occupy a multi-dimensional subspace rather than a single direction.\n\n> **Experiments assume each data point has a single concept: justify this assumption. Does your approach generalize to multiple concepts being present simultaneously?**\n\n\nThank you for the question. Our claim about SAEs being biased towards certain concepts is a property of their architecture, and thus holds for *any* data distribution. We use non-overlapping concepts as a simple case with real-world examples (such as mutually exclusive classes or children of the same parent in a hierarchical generative process) to highlight the limitations of existing SAEs. Our theoretical claims about SAE biases will also hold for overlapping concepts if we generalize the definition of separability from (current) *distinguishing one concept from other concepts (akin to multiclass classification)* to (improved) *distinguishing between the presence and absence of a concept (binary classification for each concept)*. We will formally highlight the significance of SAE biases in the presence of overlapping concepts in future work. \n\n\n> **Could the non-linear clusters in Experiment 1 not be interpreted as mixtures of simpler linear Gaussians? If so, why is it preferable to directly recover the non-linear shapes, rather than their linear components?**\n\n\nWe apologize that this was not clear. To briefly clarify, in Experiment 1 we study a mixture of nonlinearly separable and linearly separable gaussian clusters, where each cluster is defined as its own concept. This is a toy abstraction of “days of the week” (Engels et al 2024), arranged on a two-dimensional circle in model activations. In this setting, SAEs are expected to recover each concept (cluster) in a monosemantic fashion, since the goal for SAEs is to develop interpretable decompositions of model representations, not necessarily simpler ones. Thus, it is at times necessary to directly recover “non-linear” concepts, such as the magnitude-dependent “onion-features” from Csordas et al 2024, instead of decomposing them into simpler “linear” concepts. \n\nMore broadly, concepts are defined by human interpretation and knowledge on data, independent of their geometric structure in model activations.\n\n> **Since the paper is motivated by the desire to improve interpretability and steerability, have you considered evaluating your method on benchmarks or tasks that directly measure these goals? Without such evaluation, how should we assess the practical gains?**\n\nThank you for raising this important point. Our main motivation in this work is to highlight architectural limitations of existing SAEs in achieving monosemanticity, and not to improve interpretability and steerability through new SAE architectures. SpaDE serves as a constructive example—designed to incorporate properties like nonlinear separability and concept heterogeneity—and *not* as a superior SAE. Our work aims to improve understanding of SAEs as interpretability tools and differs fundamentally from efforts focused on benchmarking new architectures on large-scale LLMs.\n\n> **Models like DINO are trained without labels and may not encode class concepts directly. How do you justify the use of these expected concepts, and do you have evidence that the model itself encodes them in a disentangled or monosemantic way?**\n\nWe note the use of class-based concepts (using a subset of classes from ImageNet in Section 5.4) is justified since prior work [1] shows that such concepts are disentangled and hence linearly recoverable from DINOv2’s representations. That work in fact benchmarks different SAEs’ ability to extract these representations in order to compare them! This is the reason why we use pre-defined concepts as well. \n\n[1] Fel, Thomas, et al. \"Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models.\" arXiv preprint arXiv:2502.12892 (2025). \n\n\n> **The reason why the TopK selection yields angular separation could be explained more directly.**\n\nThank you for highlighting this point! We will include more intuitive explanations for the assumptions of SAEs and clarify the theory in the final version of the paper. The “angular separability” assumption of TopK SAE comes from the derived receptive field (App. D.2.2, line 867). \nFor latent m in TopK SAE to fire, its linear transform in encoder must exceed at least $D-K$ other linear transforms (where D is the SAE width). This gives us an intersection of half-spaces through the origin, leading to angular separation.\n\n---\n---\n\n\n**Minor comments: typos and style inconsistencies**\n\nThank you so much for highlighting these typos and style inconsistencies! We will incorporate all necessary changes in the final version of our paper.\n\n\n> **Figure 6: In row (b), TopK and SpaDE appear visually very similar (the sparsity-MSE trade-off).**\n\nGood observation! While the similar sparsity–MSE tradeoff exhibited by TopK and SpaDE in Figure 6 is interesting, the correlation maps, which show more interpretable structures emerging in SpaDE despite its similar reconstruction abilities as TopK validate our claims better. \n\n> **Line 235: What does PCFG stand for? Please define.**\n\nWe apologize this was unclear! PCFG stands for “Probabilistic Context-Free Grammars”, a formal model of language often used to study its syntactic properties. See Appendix C.3 for further details. \n\n\n> **Line 87: Is “orthogonal” projection really orthogonal?**\n\nYes! An orthogonal projection of a vector onto a set S is defined in Eq. (3). This is a generalization of the idea of orthogonal projection onto a line / plane (see Axler, Linear Algebra Done Right, Chapter 6, Section 6C) to arbitrary sets S (which are closed). \n\n>**Definition 3.2: The term “neuron” is ambiguous since the paper focuses on latents, not neurons.**\n\nApologies for the confusion! In Definition 3.2, we borrow the terminology “neurons” from neuroscience, which is the original motivation for receptive fields (line 108, 109). In the context of our work, we refer to “neurons” of the SAE encoder.\n\n\n> **Line 133: The term “onion” feature is unclear.**\n\nWe note our use of the term “onion features” is inline with the paper that first coined the term (Csordas et al, 2024). In that work, the authors show RNNs learn to encode input concepts as the magnitude of a feature—not its direction, akin to peels of an “onion”. \n\n> **Table 3: Could TopK SAE not handle some heterogeneity by selectively activating k latents?**\n\nGood question! In principle, we agree it is *possible* that TopK SAEs learn to activate less than K latents for reconstruction. However, as seen in Figure 6(a), TopK does not adapt its sparsity level (y-axis) to match the dimension of the concepts (x-axis), while it may be able to demonstrate low MSE (mean squared error, color coded) on these concepts. \n\n**Summary**: We thank the reviewer again for their insightful questions, which led us to further clarify our main claims, describe the motivation behind experimental setups better, and justify how our experiments support claims! We hope our response has sufficiently addressed these points, and if so, we hope this encourages the reviewer to increase their score! We would be happy to address any further questions in the discussion period."}]

Author rebuttal:

Meta-review: This paper analyzes how Sparse Autoencoders (SAEs) are constrained by architectural assumptions, framing training as a bilevel optimization problem and introducing **SpaDE**, which addresses nonlinear separability and heterogeneity. The work is timely, clearly presented, and supported by strong figures and broad experiments spanning synthetic clusters, semi-synthetic formal languages, and naturalistic vision settings.  

### Strengths
- Principled theoretical framing.  
- Clear link between geometry and concept recovery.  
- Broad range of evaluations across modalities.  
- SpaDE as a concrete, geometry-motivated example design.  

### Limitations
- SpaDE itself makes assumptions (Euclidean geometry, simplex projection).  
- Large-scale LLM experiments and interpretability benchmarks are lighter.  
- Distinction between atomic vs. multi-dimensional concepts could be clearer.  
- Practical significance remains somewhat open.  

### Rebuttal
The authors clarified that SpaDE is not positioned as a universal SAE but as a baseline illustrating the duality between concept geometry and SAE design. They acknowledged SpaDE’s assumptions explicitly, reframed separability for overlapping concepts, and explained why non-linear concepts must sometimes be directly recovered rather than decomposed. These clarifications addressed most reviewer concerns and reinforced the paper’s conceptual contribution.  

### Decision
Three reviewers gave **strong accept (score 5)** and one gave a **borderline accept (score 4)**. No reviewer recommended rejection, and the rebuttal further strengthened confidence in the contribution. The consensus is clearly positive, with reviewers consistently praising the clarity, conceptual framing, and breadth of evaluation.  

**Recommendation: Accept.** This is a principled and high-quality contribution that deepens our understanding of SAE limitations and will likely influence future interpretability research.
Decision: Accept (poster)