Title: Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry
Venue: NeurIPS 2025
Accepted: yes
Keywords: sparse autoencoders

dictionary learning

interpretability
Abstract: Sparse Autoencoders (SAEs) are widely used to interpret neural networks by identifying meaningful concepts from their representations. However, do SAEs truly uncover all concepts a model relies on, or are they inherently biased toward certain kinds of concepts? We introduce a unified framework that recasts SAEs as solutions to a bilevel optimization problem, revealing a fundamental challenge: each SAE imposes structural assumptions about how concepts are encoded in model representations, which in turn shapes what it can and cannot detect. This means different SAEs are not interchangeable -- switching architectures can expose entirely new concepts or obscure existing ones. To systematically probe this effect, we evaluate SAEs across a spectrum of settings: from controlled toy models that isolate key variables, to semi-synthetic experiments on real model activations and finally to large-scale, naturalistic datasets. Across this progression, we examine two fundamental properties that real-world concepts often exhibit: heterogeneity in intrinsic dimensionality (some concepts are inherently low-dimensional, others are not) and nonlinear separability. We show that SAEs fail to recover concepts when these properties are ignored, and we design a new SAE that explicitly incorporates both, enabling the discovery of previously hidden concepts and reinforcing our theoretical insights. Our findings challenge the idea of a universal SAE and underscores the need for architecture-specific choices in model interpretability. Overall, we argue an SAE does not just reveal concepts -- it determines what can be seen at all.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critically examines the ability of Sparse Autoencoders (SAEs) to reveal meaningful concepts in neural network representations. While SAEs are commonly used in interpretability research, the authors argue that their ability to uncover concepts is fundamentally constrained by the structural assumptions built into their architectures. The paper frames SAE learning as a bilevel optimization problem, revealing how design choices, particularly the non-linearities used and the structure of the latent space, act as inductive biases that filter which types of concepts can be detected.\n\nThe authors identify two core challenges to concept discovery: (1) real-world concepts may not correspond to linear directions but may require modeling non-linear manifolds, and (2) concepts may vary in complexity and intrinsic dimensionality, i.e., heterogeneity. Through extensive experiments, ranging from controlled synthetic setups to semi-synthetic and natural datasets across both vision and language domains, the authors show that standard SAEs often fail to recover ground truth concepts when these properties are not aligned with their architectural assumptions. To address this, they propose a novel SAE variant that better accommodates heterogeneity and non-linearity, demonstrating improved concept recovery and supporting their theoretical claims.\n\nOverall, the paper challenges the notion of a universal SAE architecture and emphasizes the importance of architecture-specific choices in concept-based model interpretability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper presents a broad and well-designed experimental evaluation, spanning both vision and language domains, and progressing from controlled toy tasks to complex real-world settings.\n- The presentation quality is excellent, with clear visualizations and intuitive illustrations that make complex ideas accessible.\n- Theoretical insights are not only discussed abstractly but also used constructively to develop a new SAE architecture that performs better empirically.\n- The paper brings valuable theoretical perspective by analyzing how architectural choices in SAEs induce biases over feature geometry, including latent component receptive fields and separability assumptions. This kind of reflection is important for developing more rigorous interpretability methods.\n\nWeaknesses:\n- It is unclear whether non-linearity and heterogeneity are fundamentally problematic for existing SAEs. Prior work [42] suggests that many meaningful concepts are in fact linear, and standard SAEs are already capable of modeling multi-dimensional representations when needed. In many interpretability settings, linear concepts are preferable due to their simplicity.\n- The authors argue that complex concepts must activate many latent units, which seems to contradict the common goal of having one latent per concept. This only makes sense if such “complex” concepts are composed of more atomic parts, then each latent might represent one of these atomic concepts.\n- Many experiments assume that each data point reflects a single concept. This is an unrealistic assumption, as data often contains multiple overlapping features. This weakens the motivation for their SAE architecture, the overall conclusion, and practical significance of the evaluation.\n- In Experiment 1, the “non-linear” Gaussian clusters could reasonably be interpreted as combinations of the “linear” Gaussians. The authors argue that only by discovering the \"non-linear\" clusters one achieves monosemanticity, but this is debatable. A simpler decomposition might actually be preferable.\n- The goal of SAEs is to improve interpretability and steerability, but the paper does not evaluate either in terms of established benchmarks or automated interpretability measures. As a result, it’s unclear how the proposed model advances the ultimate goals of this line of work. Or how each SAE architecture is limited or improves interpretability/steerability.\n- The evaluation focuses on recovering pre-defined concepts, but it is questionable whether these are meaningful or expected (e.g., whether ImageNet classes are the right targets for Dino representations). Arguably, the true latent features may be more abstract.\n- Some theoretical claims and explanations lack clarity or are not well-motivated. For example, the reason why the TopK selection yields angular separation could be explained more directly.\n\n\nMinor Comments:\n- In Eq. (1), the variable z depends on the input x, but this dependency is not explicitly shown.\n- In Eq. (2), consider noting that the TopK SAE (used in the experiments) does not include an encoder bias term.\n- Table 1 reuses the variable x for the latent code, which previously denoted the input. This is confusing.\n- Line 87: Is “orthogonal” projection really orthogonal? Please check.\n- Definition 3.2: The term “neuron” is ambiguous since the paper focuses on latents, not neurons. Also, $f^{(k)}$ could be interpreted as a derivative. Is it $\\mathbb{R}^d \\rightarrow \\mathbb{R}$ or otherwise?\n- Line 127: Typo: should be “is” instead of “in”.\n- Line 133: The term “onion” feature is unclear, consider explaining or rephrasing.\n- Table 3: Could TopK SAE not handle some heterogeneity by selectively activating k latents? Never more than $k$ latens can be active, but fewer could be.\n- Potentially use formal equation references with parentheses, e.g., “Eq. (1)” rather than “Eq. 1”.\n- Figure 5 caption: Missing whitespace at the end of a sentence.\n- Line 235: What does PCFG stand for? Please define.\n- Inconsistent casing: “Row” is capitalized in Sec. 5.3, but lowercase in Sec. 5.4.\n- Line 245: Capitalize “MSE”?\n- Figure 6: In row (b), TopK and SpaDE appear visually very similar (the sparsity-MSE trade-off). This similarity could be addressed or acknowledged.\n\nQUESTIONS:\n1. On the necessity of non-linear concept modeling:\nPrior work (e.g., [42]) suggests that many meaningful concepts are approximately linear. Could you clarify why it is essential to model non-linear concepts rather than decomposing them into combinations of simpler linear ones? Maybe the non-linear concept appears a linear one in a later layer?\n\n2. On concept heterogeneity and latent usage:\nYou argue that complex, heterogeneous concepts activate multiple latent units. How does this square with the goal of achieving monosemanticity, where ideally each latent corresponds to a single interpretable concept? Do you view these complex concepts as superpositions of atomic ones?\n\n3. Assumption of one concept per input:\nMany of your experiments seem to rely on the assumption that each data point corresponds to a single underlying concept. Could you justify this assumption, or discuss how your approach generalizes to settings where multiple concepts are present simultaneously?\n\n4. On interpretability and steerability goals:\nSince the paper is motivated by the desire to improve interpretability and steerability, have you considered evaluating your method on benchmarks or tasks that directly measure these goals? Without such evaluation, how should we assess the practical gains?\n\n5. On expectations of concept recoverability:\nYou use pre-defined, often class-based concepts to evaluate SAE performance. However, models like DINO are trained without labels and may not encode class concepts directly. How do you justify the use of these expected concepts, and do you have evidence that the model itself encodes them in a disentangled or monosemantic way?\n\n6. Clarification on Experiment 1 setup and goals:\nCould the non-linear clusters in Experiment 1 not be interpreted as mixtures of simpler linear Gaussians? If so, why is it preferable to directly recover the non-linear shapes, rather than their linear components?\n\nLIMITATIONS:\nLimitations are discussed, but no negative societal impacts are mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that different SAE architectures implicitly bake in different assumptions about the structure of model representations, via an analysis of the SAE training objective. They then develop a new architecture, called SpaDE, which enables concepts to be discovered even when they exhibit nonlinear separability and heterogeneity. In various settings, the authors find that SpaDE can improve over existing SAE architectures.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n* The paper is exceptionally well-written and well-constructed, with beautiful figures and diagrams.\n* The core point, that SAE architectures make implicit assumptions about the data, is a highly valuable one. The paper provides a useful framework for reasoning about the assumptions that we make when designing interpreter models. The SAE architecture that its analysis leads to is also an interesting one.\n\n### Weaknesses\n* The language model evaluations of SpaDE are a little lighter than what most papers introducing an SAE architecture conduct (scaling laws, sparsity vs. reconstruction frontiers, etc.). It would be good to ultimately have a more detailed empirical study of SpaDE on full-scale LLMs, but I recognize that SpaDE is not the only contribution of the work, and the broader theoretical/conceptual point about the relationship between SAE architecture and assumptions about the data is arguably more important.\n\nQUESTIONS:\n* This paper focuses on SAEs, but there is some momentum in mechanistic interpretability away from SAEs on their own and more towards models which capture aspects of model computation like CLTs. Do you have any thoughts on the applicability of your framework to more general types of interpreter models? It seems like the analysis could apply pretty directly, since CLT encoders are the same as SAE encoders. But more broadly, I wonder if there are other more general ways that your point -- that interpreter model architecture determines what we can discover about models -- could apply to the design of other sorts of sparse interpreter models.\n* Do you think your work could explain any potentially-pathological results that have been described about SAEs? Could our choice of architecture be leading to feature splitting, or to non-atomic features being learned, that could be ameliorated with a different choice of architecture?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors challenge the common assumption that Sparse Autoencoders (SAEs) are neutral tools for uncovering meaningful concepts in neural representations. By recasting SAE training as a bilevel optimisation problem, they show that each SAE architecture implicitly projects data onto a specific geometric constraint set - e.g., the positive orthant (ReLU), sparse angular subspaces (TopK), or hypercubes (JumpReLU). These constraints shape the encoder's receptive fields, biasing the model towards detection of concepts that conform to assumptions, such as linear separability and fixed intrinsic dimensionality. The authors conduct a series of experiments across a range of settings to empirically verify this framework. They propose a new architecture SpaDE, which uses the sparsemax nonlinearity to handle nonlinear separability and concept heterogeneity, and demonstrate it consistently outperforms other SAEs in recovering monosemantic, interpretable features.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:** \n- Focus on latent space geometry and implications for interpretability research: connecting the encoder receptive field to assumptions about the underlying geometry of the concepts provides a compelling argument that latent space geometry must be prioritised when designing SAEs. The paper makes it clear that this is a fundamental limitation of current SAEs, and the theoretical framing is timely and well-needed in the interpretability field. \n- Formalisation via bi-level optimisation: the paper presents a principled reformulation of SAE training as a bilevel optimisation problem, where the encoder implicitly solves a projection problem over an architecture-specific constraint set. This offers a clean, mathematically grounded explanation for how architectural choices induce biases in feature extraction, and is broadly applicable to multiple existing SAE variants. \n- Broad range of evaluation settings: experimental evaluations spans a wide range of settings - from synthetic separability setups to semi-synthetic formal language models and real-world visual representations. The broadness of the evaluation approach provides strong support for the generality of the claims. \n- Principled design of SpaDE: the architecture proposed is not just proposed heuristically, but instead motivated by the geometric limitations identified in the earlier framework. SpaDE is also shown to resolve failure modes observed in prior models. \n- Clarity of empirical failures: the empirical results convincingly demonstrate how specific structural assumptions (e.g., linear separability, fixed concept dimensionality) lead to systematic failure modes in prior SAEs, especially in more complex settings. \n\n**Weaknesses:** \n- Assumptions underlying SpaDE not extensively tested: as highlighted by the authors in Section 6, SpaDE assumes that Euclidean distance is a meaningful distance metric in concept space, and that concepts project well onto the probability simplex. These assumptions may not hold across domains with more complex geometries, however, the authors do acknowledge this and this is likely to be more diagnostic of a broader, open problem, as much of the current literature implicitly assumes Euclidean geometries.\n\nQUESTIONS:\nCo-occurence and overlapping concepts: As noted by the authors in Section 6, the analysis focuses on mutually exclusive concepts, where the presence of one concept implies the absence of others. While the central argument about SAEs being biased towards certain concepts due to their architectural constraints clearly holds in general, regardless of the data distribution, I'm curious how the empirical failure modes (e.g., lack of monosemanticity) differ qualitatively in these non-exclusive scenarios (e.g., where concepts are overlapping/compositional)? In particular, do real-world concept distributions tend to violate the separability and dimensionality assumptions identified? I would expect this to be the case, but it would be valuable to understand whether these failure trends persist in these non mutually exclusive settings.\n\nLIMITATIONS:\nYes, very well elaborated on in Section 6.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors investigate the limitations of Sparse Autoencoders (SAEs) in terms of the types of concepts they can detect. They attribute these limitations to the architectural design of SAEs, which imposes specific assumptions on the structure how features are encoded. To address these constraints, the authors propose a new SAE, SpaDE, which is able to address previous limitations on data assumptions.

STRENGTHS AND WEAKNESSES:
# Strengths

- The paper is well-written, logically structured, and easy to follow. Each step is clearly motivated and illustrated.

- The work is timely and addresses a relevant problem, offering a better understanding of shortcomings of SAEs.

- The figures are overall clear, well-designed, and effectively support the main points.

# Weaknesses

- The domain of the work is not clearly specified in the first part of the paper and only becomes explicit in Sections 5.3 and 5.4. Referencing this earlier would improve clarity.

- It is not entirely clear what is meant with "true structure" of concepts (l. 126).

- In Figure 5, row (c), the meaning of the different cluster colors is unclear and should be clarified in the caption or main text.

QUESTIONS:
- How do you interpret the occurrence of more different colored clusters in SpaDE compared to the others?

- When analyzing SAEs, which true structure of the data should be considered?

- Does good data selection for analysis always depend on the task?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03