A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders

Authors: David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, Joseph Isaac Bloom
Primary area: interpretability and explainable AI
Keywords: Sparse Autoencoders, SAEs, LLMs, interpretability

Abstract:
Sparse Autoencoders (SAEs) have emerged as a promising approach to decompose the activations of Large Language Models (LLMs) into human-interpretable latents. In this paper, we pose two questions. First, to what extent do SAEs extract monosemantic and interpretable latents? Second, to what extent does varying the sparsity or the size of the SAE affect monosemanticity / interpretability? By investigating these questions in the context of a simple first-letter identification task where we have complete access to ground truth labels for all tokens in the vocabulary, we are able to provide more detail than prior investigations. Critically, we identify a problematic form of feature-splitting we call "feature absorption" where seemingly monosemantic latents fail to fire in cases where they clearly should. Our investigation suggests that varying SAE size or sparsity is insufficient to solve this issue, and that there are deeper conceptual issues in need of resolution.

Decision: reject
Reviewer scores: 8, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730316247716

thread_type:
official_review

<thread_object>
creator: reviewer_x2j6

<content>
summary:
This paper documents and measures a particular form of feature splitting in the Gemma Scope family of sparse autoencoders (SAEs), which the authors call feature absorption. The prototypical case of feature absorption studied is when a latent is a high-precision classifier for a property like "starts with the letter L", but has poor recall due to other latents "absorbing" individual tokens, so that the main latent's activations are given by the rule "starts with the letter L, except for the tokens laser, lions, [several other exceptions]".

strengths:
The paper explores a natural task, first letter classification, and does so well. It establishes that this information is linearly available in the model (by training an LR probe with high F1-score), and that SAEs find latents that align with the probe direction (Figure 4b and especially Figure 13), and that these latents are often good classifiers, though not as good as probes (Figure 2a).

The paper provides a novel perspective to a documented issue of feature splitting in SAEs. It provides good theoretical justification of why feature absorption is likely to happen, and why it provides challenges to interpretability.

weaknesses:
The paper could do more to demonstrate that feature absorption happens as described, namely that: 
1) There is a latent which is a high-precision classifier for "starts with letter [x]".
2) There is a particular subset of tokens which start with this letter but which the latent fails to activate on.
3) There are separate token-aligned latents that individually classify those tokens.
4) If that subset of tokens is removed, the latent is a high-precision, high-recall classifier for "starts with the letter [x], except for that subset of tokens".

The authors demonstrate 1) and 2) well in sections 4.1 and 4.3 respectively, but never show 3) or 4) besides a single example in section 4.2. The authors could address 3) by adding an additional step to the experiment in section 4.3, in which they confirm that "the SAE latent receiving the largest negative magnitude ablation effect" is a token-classifying latent.

Lines 148-157: The metric described can fail at its stated goal to "[measure] the propensity of the model to choose the correct starting letter as opposed to other letters" because averaging logits can interact poorly with the softmax activation function. For example, if "A" is the correct letter, but the model logits are g[A]=10, g[B]=25, g[C]=g[D]=...=g[Z]=0, then the model would produce B with near-certainty. But the provided metric yields m=10-(25+0+0+...+0)/25=10-1=9, the same value as if g[A]=9, g[B]=...=g[Z]=0. This metric would be more convincing if the average were replaced by a max, or if it were a softmax over letter tokens. 

The authors only study a single family of SAEs on a single language model. Alternative SAE architectures may not demonstrate feature absorption.

questions:
**Questions** 

1. The concept of "approximately token-aligned" latents is central to claim 3 (lines 69-71), but is not defined anywhere. How is token-alignment defined, and is it quantified or measured in these experiments?

2. When the F1 score of an SAE is computed, as in Figure 2 (lines 169-172), how are SAE activations converted into a binary classification? Is the classification "does/doesn't activate". Is a non-zero threshold used? If so, how is that threshold chosen?

3. In Figure 7b, when the mean absorption rate is \~35%, is that indicating that for roughly 35% of tokens, there is a latent absorbing that token? The vocabulary size of Gemma-2B is \~256k (https://arxiv.org/html/2408.00118v1#:~:text=Vocab%20size-,256128,-256128) and 35% of that is \~90k. If that is the case, how are 90k tokens absorbed into 65k latents?

4. In figure 13, why is the high cosine similarity in the .4-.6 range, and not closer to 1? This range of cosine similarities is suggestive of learning the "true" feature mixed with a second, orthogonal feature. 

5. It seems that the paper usually uses cosine similarity of the LR direction with the encoder direction, but Figures 4 and 13 use decoders instead of encoders. Why is that?

6. What does Algorithm 1 (Lines 116-127) return? One ablation effect per token per latent? Or are these averaged in some way?

**Other Comments**

If it is permitted for authors to make revisions before the final submission, there are several small changes that could improve the quality of the paper:

- Lines 94-107: the paper should reference (Lieberum et al., 2024) since that is the exact architecture of SAE being studied. 

- Line 100: Although ReLU is often used in SAE architectures, it would be better to write a generic activation function in this equation, especially considering that the SAEs studied in the paper use JumpReLU, not ReLU.

- Line 118 and Line 147: The description "include the SAE error term" is ambiguous in the text. It is clarified in the glossary, and by the glossary's citation to (Marks et al., 2024), but the main paper neither cites (Marks et al., 2024) here, nor links to the glossary. Is there a way to include hyperlinks to the glossary in the text itself? 

- Line 205: There is a citation to (Gao et al., 2024) for sparse probing. It is likely supposed to cite (Gurnee et al., 2023) as in the background section. (Gao et al., 2024) use k-sparse SAEs, which are distinct from k-sparse probes and the SAE architecture studied in the paper.

- Lines 291-292: "The token where the SAE feature activates is highlighted in green" should be "yellow" since the highlight is yellow.

- Line 404: The paper says "it is difficult to apply [(Karvonen et al, 2024)] to existing SAEs trained on real LLM activations." One of the main contributions of (Karvonen et al, 2024) is very applicable to this paper, namely coverage (Section 3.2 in https://arxiv.org/pdf/2408.00113). Coverage for a given set of properties (in this case, "first letter identification") is defined by, for each property, finding the SAE latent and threshold which results in the best F1 score for classifying that property, then averaging those F1 scores across the properties. This is very similar to what is shown in Figure 2, though with a different selection rule for SAE latents. The paper could make reference to the similarity of this method to coverage.

- Line 429, possible typo: "..find that by training an SAE on the decoder *or* another SEE, a technique...", the word "or" should be "of".

- (Uncertain) The citation of "(Huben et al., 2024)" should instead refer to the paper as "(Cunningham et al., 2024)". The order of the authors is different between the OpenReview page (https://openreview.net/forum?id=F76bwRSLeK) and the paper itself (https://openreview.net/pdf?id=F76bwRSLeK). In such a case it is most likely proper to follow the order of authors in the paper.

**References**

Leo Gao, Tom Dupre ́ la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024.

Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dic- tionary learning for language model interpretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024.

Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsi- mas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.net/forum? id=JYs1R9IMJr.

Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=F76bwRSLeK.

Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Ja ́nos Krama ́r, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2, August 2024.

Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language mod- els. Computing Research Repository, arXiv:2403.19647, 2024. URL https://arxiv.org/ abs/2403.19647.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730672453732

thread_type:
official_review

<thread_object>
creator: reviewer_4xb2

<content>
summary:
This paper proposes a new simplified setting for studying features discovered by Sparse Autoencoders (SAEs) trained on language model activations. It supposes that SAEs should discover features corresponding to the first letter of a given token, but finds that features which are predictive of a given letter occasionally fail and are 'absorbed' into the activation of another feature. They then measure and explore this phenomenon in the context of their setting.

strengths:
- Describes an interesting setting in which to study feature disentanglement for natural language.
- Discovers a new phenomenon, 'feature absorbtion', which could pose a problem for desired future applications of Sparse Autoencoders.
- Setting allows for a thorough analysis of the failure cases for feature absorbtion and feature splitting, and said analysis was good and informative.
- Paper has a clear statement of the problem and good structure. Good use of the 'S'/'_short' example to help build intuition.

weaknesses:
- It would be useful to explore the commonalities between different instances of absorbtion of SAE features, or otherwise find more ways to verify/expand upon the claim/statement that "feature absorption is likely a logical consequence of SAE sparsity loss".

questions:
- Did you find any rules for predicting whether the feature activation for the first letter of a given word would be absorbed? (i.e. do you have any hypothesies/ways to differentiate situations in which feature absorbtion occurs or does not occur?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705187596

thread_type:
official_review

<thread_object>
creator: reviewer_1gvz

<content>
summary:
This paper investigates the interpretability of the extracted latent/features by Sparse Autoencoders. The authors found that the Sparse Autoencoder does extract monosemantic and interpretable features and changing the hyperparameters of the Sparse Autoencoder could not eliminate this.

strengths:
As the large language model becomes increasingly popular, it is really important to get a good understanding of these techniques. This paper investigates a meaningful problem and attempts to provides some insights and points out some research directions, which should be very useful to the community.

#---------------------------------------#

After rebuttal, the authors add results with different SAE architectures and three different model types.

weaknesses:
There are several weaknesses for this paper.
1) The observation and conclusion are merely based on one model and one task (first-letter identification task). Thus, it is really difficult to say how general and convincing these conclusions could be. I am afraid it is not safe to draw a solid conclusion based a single setting.
2)  The authors should elaborate their method better. The current version simply lists some background and then posts a pseudo code (algorithm) there. More descriptions should benefit readers to get a better understanding.
3) Some contents cause confusing instead of help. For example, the Figure1 is really confusing and hard to interpreter and even worse, the authors do not provide sufficient descriptions of it in the main paper.
4) The overall presentation is a little bit messy and it is really hard to follow the content.

questions:
I list my concerns and questions in the "weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730808990195

thread_type:
official_review

<thread_object>
creator: reviewer_s6ku

<content>
summary:
This paper explores the use of Sparse Autoencoders (SAEs) to decompose activations of Large Language Models (LLMs) into interpretable latent features. It addresses two main questions: the degree to which SAEs yield monosemantic (single-meaning) and interpretable latents, and how adjustments to the sparsity and size of SAEs impact interpretability. Through a controlled first-letter identification task with complete ground-truth labels, the authors find more nuanced insights compared to previous studies.

strengths:
1. This paper studies an interesting problem, that is, learned sparse features can become less understandable by "absorbing" token-aligned features.
2. understanding how width and sparsity of SAEs affects its training performance can be helpful for future SAE training.
3. the paper is written in an easy-to-understand way.

weaknesses:
1. It would be valuable to explore practical examples where insights from feature absorption can enhance interpretability. For instance, analyzing the first letters of all tokens associated with specific latent activations might offer new explanations for features that previously seemed opaque. This approach could reveal subtle patterns in latent activation, aiding in the interpretation of challenging features.
2. This work presents a specific form of feature absorption, but it raises the question of whether other variations might exist. Are there additional contexts or scenarios where feature absorption manifests differently, potentially impacting interpretability in distinct ways? Identifying these cases could deepen our understanding of the phenomenon and refine the strategies needed to address it.

questions:
Same as above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731715803235

thread_type:
public_comment

<thread_object>
creator: ~neel_nanda1

<content>
title:
Paper Thoughts

comment:
Speaking as an uninvolved mechanistic interpretability researcher, I consider this paper a valuable contribution to the literature, and think that it has improved my understanding of sparse autoencoders. Feature absorption is not something I'd thought about explicitly before, but in hindsight feels obvious, and I think has useful implications for strategic directions in future SAE research. To me the key takeaway is as follows:

The core motivation of sparse autoencoders is that we want to find interpretable concepts in an unsupervised way, don't know how to optimise for interpretability, and so instead optimise for the proxy of sparsity. And so a crucial question is how good a proxy sparsity is. Until recently, we didn't have much evidence of it being an issue, and I mostly ignored it. Feature absorption seems like the crispest example yet of how sparsity is an imperfect proxy: 

Whenever you have two concepts A and B, where B is a strict superset of A (ie if A is present then B is definitely present) it will come apart. The example in this paper is pairs like A="this is the word elephant" and B="this token starts with E", but this is a very general point. The interpretable solution is to learn feature_1=A and feature_2=B, but when A is present, two features fire! So the sparsest solution is to absorb feature_2 into feature_1: feature_1=A and feature_2=(B excluding A).

Once the problem is demonstrated once, it becomes pretty clear that we should expect it to happen everywhere, whenever there are hierarchical pairs of concepts. I consider this a valuable contribution because it helps direct future SAE research: we need to either find a way to optimise less for sparsity, so feature absorption isn't incentivised; try a totally different approach; or show that the costs of absorption (and other failures of sparsity) are low enough to be worth paying. 

(Disclosure: I was not at all involved in this paper, but I do know the authors and likely have some positive bias due to that. No one asked me to write this)

(Note also: I don't feel confident in the norms here, so feel free to ignore this comment if these kinds of thoughts are not welcome! I don't see other researchers doing this, but I figure that as ICLR has made a deliberate choice to allow public comments during the rebuttal process, they likely want this kind of public feedback from uninvolved researchers, so long as it's constructive)
</content>
</thread_object>

</thread>
<thread>
creation date:
1731840100083

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for the time reviewing the paper and for their feedback on our work. We are grateful for the positive comments supporting the importance of the work to the field and the interest in potential follow-up work.

To address feedback from reviewers, we have made the following changes to the paper:
- We trained our own standard L1 loss SAEs on the Qwen2 0.5B and Llama 3.2 1B, and demonstrate that feature absorption occurs in these SAEs as well. We have added plots demonstrating this to the paper in Figure 7c. We hope this should alleviate any concerns related to only investigating the Gemma Scope SAEs on Gemma2 2B, as we now have results covering two different SAE architectures and three model types.
- We have been able to reproduce feature absorption empirically in a toy setting where we control the activation distribution of all latents, and can show that when two features co-occur absorption will occur. We expanded the discussion of this in the paper, and have also added a full exploration of the toy setting to Appendix A7 in the paper. This should help bolster the argument that feature absorption is a logical consequence of the sparsity penalty in SAEs when features co-occur together.
- We have made writing improvements to the locations in the paper pointed out in reviews, and feel this has helped improve the clarity and quality of the paper further as well.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739856543349

thread_type:
public_comment

<thread_object>
creator: ~neel_nanda1

<content>
title:
Why on earth was this rejected?

comment:
I find the AC's decision to reject this paper deeply perplexing. There was an overwhelming and unanimous consensus in favour of accepting the paper from reviewers, at 3 accepts and a weak accept. Notably, of the 11,500 submissions to ICLR, this was the *third* highest rated rejected submission, and higher rated than 90% of accepted papers. I think that given reviews this strong, an extreme reason is required to justify overturning it and the reasons given for rejection seem wholly insufficient.

More importantly, rather than a case of shared poor reviewer judgement, this is a genuinely high quality paper, that has meaningfully added to the field, and helped shape how I think about my research. This decision is an unusually poor application of area chair discretion, and I hope this paper is resubmitted to a conference where it gets an AC with better judgement.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Sparse Autoencoders (SAEs) have become a popular tool for interpreting neuron activation behaviors in large language models (LLMs). This paper questions the effectiveness of sparsity regularization in SAEs by identifying the phenomenon of feature absorption—a behavior where overlapping features are absorbed into a single latent representation, degrading interpretability. The authors use a controlled first-letter identification task to analyze feature absorption, demonstrating its prevalence and persistence across varying SAE sizes and sparsity levels. They identify sparsity penalties in SAEs as the root cause of this phenomenon.

Multiple reviewers appreciated the identification and analysis of feature absorption, describing it as a meaningful contribution to the understanding of SAEs. The use of controlled settings and diverse models (e.g., Gemma Scope, Qwen2, Llama) provides robust empirical validation.

Concerns from the reviewers:

- Limited Real-World Applicability: The study focuses primarily on the first-letter identification task, limiting its generalizability to broader real-world tasks. 

- Unexplored Variants of Absorption: Reviewer S6kU and others noted that additional manifestations of feature absorption or scenarios where absorption may behave differently were not explored, leaving gaps in understanding.

- Presentation Issues: Multiple reviewers criticized the paper’s overall clarity and organization, pointing out ambiguous descriptions and confusing visuals (e.g., Figure 1). 

**Recommendation**

While there is considerable interest in the identification of feature absorption, as reflected in reviewer and public comments, and all reviewers recommended acceptance, the paper’s shortcomings are substantial:

1) Its focus on a toy problem raises question on the generalizability and practical relevance of its findings.

2) there is no rigorous theoretical characterization of the phenomenon, limiting the depth of its contributions.

3) it has limited attempts at addressing the identified problem. For example, if the issue is with hierarchical features then methods from hierarchical sparse regularization (see e.g. [a]) would be a natural alternative to (unstructured) sparse coding?

4) the lack of clarity in the presentation raises the concern that the paper may not be broadly accessible to the broad audience of ICLR. In particular, reviewer 1gVz raised multiple concerns on clarity of the presentation but there was little effort during the discussion phase to address those.

Given all these, I’d recommend a weak rejection while encouraging the authors to improve their manuscript along the lines above to substantiate their interesting findings.
 
[a] Proximal Methods for Hierarchical Sparse Coding, JMLR 2011