Title: A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
Venue: NeurIPS 2025
Accepted: yes
Keywords: sparse autoencoders

SAEs

interpretability

NLP
Abstract: Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features (“math” may split into “algebra”, “geometry”, etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get “absorbed” into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper showcases the phenomenon of feature absorption, a mechanism by which sparse autoencoders (SAEs) systematically fail to fully capture features that co-occur in a hierarchical fashion, creating an interpretability illusion. A toy model is used to show that absorbed features are theoretically and empirically preferred by the SAE training procedure. Ablation experiments, cosine similarities to linear probes and k-sparse probes are used to identify specific examples of feature absorption in public base LLMs. A measure of absorption rate is defined and used to quantify the feature absorption in these LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nA note: as the saying goes, all models are wrong, but some are useful. I will pay most attention to whether or not feature absorption is a useful model for explaining how SAEs work, and how that in turn affects the usefulness of SAEs as a model to explain how LLMs work.\n\nStrengths\n1. A substantial amount of evidence is shown to convincingly support that feature absorption occurs in SAEs used in practice.\n2. The feature absorption can be replicated in small toy examples to be studied.\n3. Feature absorption can be theoretically shown to be preferred by the SAE training procedure.\n4. If feature absorption indeed occurs, it is probably a widespread phenomenon across many open-source SAEs, significantly hindering mechanistic interpretatability efforts.\n\nWeaknesses\n1. It is unclear to me that describing a specific kind of feature splitting as \"feature absorption\" helps to explain more about what is happening to the SAE than just saying that one of the feature splits is extremely small. I am unsure that the identification of feature absorption as a separate phenomenon from feature splitting adds that much interpretation value when the interpreter already understands that feature splitting is common and can make the SAE explain things less clearly sometimes. If we could just redefine one of the \"true human-interpretable features\" to be the union of the split features, this would produce hierarchical co-occurrence of features leading to feature absorption which wasn't there in the first place. Individual feature absorptions may only occur under certain allotments of \"true features\" and not others, making the presence/absence of this problem inherently subjective.\n2. In order to evaluate the test metric for feature absorption, you need to have a probe, and that probe must be trained with labeled data corresponding to the feature you are testing for. You could only do this probing successfully if you already knew the feature you are looking for is there in the model. But then, why would you even be using the SAE and feature absorption metric in the first place? You've already found the feature's direction using your probe, which I thought was the whole point of doing all of that. Alternatively, maybe you don't know the features and you're using the SAE to find them. Then, you can't be training a probe to make your absorption metric, since you don't know what the feature is that you want to label your data for. Again, the metric seems to be useless. It seems like the test metric can play no actual role in any part of the core pipeline of model interpretation.\n\nQUESTIONS:\n1. As I understand (correct me if I am wrong), the main function of the absorption metric is primarily to auto-classify existing feature splits as either benign or harmful to the SAE's ability to detect a known human-interpretable latent. While the prevalence of harmful absorptions is confirmed by the metric, the authors do not evaluate whether these purported absorbed feature events are indeed more harmful towards SAE-derived interpretations than split non-absorbed feature events. I would like to see some evidence as to whether or not this metric is actually a good detector of the quality of an SAE-derived interpretation.\n\n2. My Weakness item #2 comes from my understanding that the purpose of the absorption metric is that it should be used in conjunction with the SAEs when trying to identify the unknown features that may be present in a model. Is this the intended purpose or use case? It is unclear whether the metric is supposed to serve a different purpose instead, e.g., to help with further interpretation of the model after the features have already been identified. If you would explain an example of where you expect this metric to be useful in a different part of the interpretation pipeline, I may increase my score.\n\nminor: I see in the glossary that the authors explain how they avoid overloading the word \"feature\", which is really helpful for the reader. If I had seen that earlier, reading this paper may have been more straightforward, so I might suggest folding that explanation into the background and also keeping it in the Appendix.\n\nminor: Line 87, capitalization typo on \"L1\"\n\nLIMITATIONS:\nThere are no concerning limitations or potential negative societal impacts from this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and systematically analyzes a phenomenon that can arise in sparse autoencoders (SAEs) when interpreting the activation space of large language models (LLMs), termed feature absorption by the authors. When hierarchical features exist in the model (e.g., “starts with S” and “short”), the SAE’s sparsity-driven optimization may cause the representation of higher-level features to be absorbed into more specific latents, resulting in the general latent failing to activate where it intuitively should. The authors propose a metric—absorption rate—to quantify this effect, and empirically demonstrate its prevalence across various models and layers. This work offers a new perspective and raises important questions about the reliability and applicability of SAEs in interpretability tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper introduces a novel and clearly defined phenomenon—feature absorption—that highlights how sparsity objectives in SAEs can interfere with the reliable representation of general features in LLMs.\n\n- The empirical study is well-executed across multiple models and layers, including Gemma, Qwen2, and LLaMA variants, and provides consistent evidence for the prevalence of absorption effects.\n\n- The proposed absorption rate offers a simple and interpretable metric to quantify this behavior, and the combination of toy examples and real-model evaluations helps ground the intuition.\n\n- The work raises important questions for the future design of interpretable sparse representations and may inform efforts toward more robust circuit discovery.\n\nWeaknesses:\n\n- The experiments focus on relatively small-scale models; it remains unclear whether similar absorption patterns persist in larger models such as 7B or 9B.\n\n- The absorption rate metric relies on token-level ablation and is mainly applicable to lexical features, limiting its ability to capture absorption in higher-level or cross-token concepts.\n\nQUESTIONS:\n- Any thoughts on when feature absorption is more likely to occur, and whether there are ways to mitigate it? For example, are certain types of features or training settings more susceptible to this effect?\n\n- When multiple features with overlapping semantics co-occur, stronger latents might dominate weaker ones even without true absorption. Clarifying how such dominance is distinguished from actual absorption would strengthen the analysis.\n\n- In my opinion, SAEs may serve as a form of knowledge compression, which can introduce ambiguity. While the resulting features are more interpretable than individual neurons, activation-based features descriptions might still underrepresent a feature’s full role—especially in cases where features reflect different functional categories, such as input features, which capture patterns within the model’s input, and output features, whose main role is to directly influence the tokens the model generates.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper demonstrates theoretically and empirically a potential mismatch between what we would ideally \"want\" from an SAE decomposition, versus what the objective function of the SAE prefers in reality. \n\nSpecifically, the authors conjecture the existence of the *feature absorption* phenomenon. Suppose there are two features in the data, $G$ (a more \"general\" one) and $S$ (a more \"specific\" one), such that whenever $S$ is present in an input, so is $G$. Let $f_S, f_G$ be the \"true\" feature vectors corresponding to $S, G$ in activation space. The main observation behind absorption is that, in this setup, the SAE objective will prefer to model this situation using two latents $v_G = f_G, v_S= f_S + f_G$ such that, when $G$ is present by itself, $v_G$ will be active in the SAE, and when $S$ (and therefore $G$) is present, *only* $v_S$ will be active. This solution is sparser than allocating a latent for each of $f_G,f_S$ and firing both when $S$ is present.\n\nThe authors show this plays out in a simple toy model, and in a real-world setup where basically $G=$ \"token starts with some letter\" and $S=$ \"token equals a specific token that starts with that letter\". They define a formal measure of absorption and investigate how different SAE parameters and metrics influence its prevalence according to that measure.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper does a great job at illuminating a non-obvious qualitative phenomenon in interpretability, and explaining and demonstrating it in a quite intuitive and reasonable progression of thought experiments, toy models and real-world experiments. Despite the somewhat constrained real-world setup being studied (first-letter features are perhaps not the ultimate reason we as a field want to develop SAEs), the authors argue very persuasively for the general plausibility of feature absorption and its implications on the use of SAEs for interpretability. The paper is at times rough around the edges, but I'm certain this can be largely overcome by a small revision. \n\nOverall, this is a timely and significant contribution to the SAE line of work.\n\nQUESTIONS:\n- Def. of hierarchical features (line 54) is unclear. What does it mean for a feature to \"fire\"? I think there are some implicit assumptions being made here, and it would benefit the paper to make them more explicit.\n- is it really necessary to train the toy model on 100M activations (Section 3, line 87)?\n- plot 6b would perhaps be more interpretable if we sort the x-axis by the y-axis value?\n\nTypos and other minor nitpicks:\n- fig 1 right: \"starts S\" should be \"starts with S\"\n- line 87: should probably use L1 or $\\ell_1$ for both occurrences instead of L1/l1\n- line 103: \"This is exactly the sort of gerrymandered feature firing pattern we saw in real SAEs for the starting letter task\" - we have not seen this yet in the paper! Maybe say \"we will see later in Section X\"\n- line 178: \"We conduct an ablation experiment on the `_short` token\" - might help the reader to remind them of what an \"ablation experiment\" means in the context of this paper, or to altogether give this intervention a more precise name. \"ablation experiment\" in the current paper is used to point to a very specific technical term; but it's also a general term in the ML literature, which may give rise to confusion.\n\nLIMITATIONS:\nAs pointed out, it would be interesting to demonstrate absorption in other settings with more practically interesting hierarchical features.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSparse autoencoders (SAEs) are an interpretability tool that aim to decompose language model activations into a linear combination of individually-interpretable features. SAEs allow for a large set of possible features (the “dictionary”) but require that any individual activation may only use a small number of these features (a sparsity constraint), and hypothesize that this will cause the learned features to be interpretable.\n\nThis paper presents an obstacle to the interpretability property: when in reality there are hierarchical features, the optimization objective for SAEs encourages “feature absorption”: the coarser feature learns not to fire when the more fine-grained feature fires, so as to improve sparsity. Specifically, suppose there are two features A and B where if B fires then A should fire. Then instead of learning [A] and [B] as separate features, the SAE can learn [A and not B] and [B] as separate features. Then on inputs where both A and B are present, the SAE only fires one feature instead of two, doing better on sparsity.\n\nThe paper demonstrates this in a toy setting where the ground truth can be controlled, and then studies it in practice on the “first letter task” (“dog” → “d” and so on). For this task, there are often a small number of SAE latents capturing the first letter information, but that then fail to fire on some common tokens where the interpretation would suggest they should fire. For example, the “starts with s” feature fails to fire on the “short” token, even though a linear probe can perform the “starts with s” task well. Ablation experiments demonstrate that this information is instead in a feature that fires on “short” only, suggesting that the “short” feature has absorbed the “starts with s” feature, in accordance with theory.\n\nThe authors define a feature absorption metric in this task, and find that lots of SAEs have significant feature absorption on this task, with rates particularly high for wider and sparser SAEs.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The core problem with SAEs identified in this paper is very intuitive and seems like it should apply quite widely – yet to my knowledge it was not known in the literature. (It may have been known in the dictionary learning or sparse coding communities; I am not familiar with that literature.)\n2. The authors have a good blend of toy models and concrete examples that build intuition and demonstrate the mechanism of feature absorption, along with a larger scale quantitative study that quantifies how large a problem this is in practice.\n\n## Weaknesses\n\n1. As the authors acknowledge, the feature absorption metric is likely unreliable in a variety of ways. For example, it relies on ablation on the original token, which will often be ineffective in later model layers since the information will already have been moved by an attention head to the final token position. In addition the metric has multiple hyperparameters that are judgment calls; changing these would presumably affect the results a decent amount, and there isn’t a particular justification for the choices made.\n2. The empirical results are almost entirely based on the first-letter task, which makes it hard to know how large the feature absorption problem will be in general. It is possible that the more abstract features that are more often the goal of interpretability research would not have the same issue, though the theoretical / conceptual argument remains strong so I consider this to be a fairly minor weakness.\n3. The paper identifies a problem with SAEs but doesn’t propose any methods for solving that problem, limiting its impact. (However, SAEs are popular enough that pointing out a significant problem with SAEs is still a very valuable contribution, even if it doesn’t come with a solution.)\n\n## Summary\n\nOverall, this is a great paper that changed my views about how promising SAEs are. Feature absorption is an obvious problem with SAEs once pointed out, but I hadn’t thought about it before. I wish there was more investigation and evidence about how large a problem is – for example, perhaps one could brainstorm ten different pairs of hierarchical features, and manually check for how many of them feature absorption has happened. But even without this quantification it’s a valuable paper that should be accepted.\n\nQUESTIONS:\nHow much do you expect feature absorption to be a problem in practice for typical applications of interpretability? Can you quantify this in some way?\n\nLIMITATIONS:\nMostly yes, though I think they should also acknowledge the limitation of focusing just on the first-letter task.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}