Title: Dense SAE Latents Are Features, Not Bugs
Venue: NeurIPS 2025
Accepted: yes
Keywords: sae

sparse autoencoder

interpretability

mechanistic interpretability

language model
Abstract: Sparse autoencoders (SAEs) are designed to extract interpretable features from language models by enforcing a sparsity constraint. Ideally, training an SAE would yield latents that are both sparse and semantically meaningful. However, many SAE latents activate frequently (i.e., are *dense*), raising concerns that they may be undesirable artifacts of the training procedure. In this work, we systematically investigate the geometry, function, and origin of dense latents and show that they are not only persistent but often reflect meaningful model representations. We first demonstrate that dense latents tend to form antipodal pairs that reconstruct specific directions in the residual stream, and that ablating their subspace suppresses the emergence of new dense features in retrained SAEs---suggesting that high density features are an intrinsic property of the residual space. We then introduce a taxonomy of dense latents, identifying classes tied to position tracking, context binding, entropy regulation, letter-specific output signals, part-of-speech, and principal component reconstruction. Finally, we analyze how these features evolve across layers, revealing a shift from structural features in early layers, to semantic features in mid layers, and final to output-oriented signals in the last layers of the model. Our findings indicate that dense latents serve functional roles in language model computation and should not be dismissed as training noise.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that densely activating SAE latents are not mere training artifacts but correspond to principled directions in a model's residual-stream subspace. It shows that many of these latents come in antipodal pairs that encode a single one-dimensional feature, maps how dense latents bind to multiple contextual signals including context-binding units and introduce new taxonomy of the dense latents.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The authors provide *quantitative evidence* that dense latents reflect the geometry of the residual-stream subspace rather than incidental training artifacts.\n- The paper carefully analyzes the antipodal-pair phenomenon in SAEs, showing how two latents encode positive and negative versions of the same direction.\n- The study quantifies how strongly dense latents correlate with multiple contexts, revealing their broader representational role.\n- The authors identify context-binding latents that preserve the current topic and statistically characterize their antipodal behavior.\n- The work shows that null-space latents function like an internal temperature control, a novel and intriguing finding.\n\n## Weakness\n- Generality remains unclear: with experiments limited to Gemma 2, the findings might still be model-specific. Replicating the core result, especially the claim that dense latents are not SAE artifacts, on several other LLMs would strengthen the paper.\n- The main text does not fully explain how each latent class is assigned. A unified, detailed description of the classification criteria (ideally consolidated in the appendix) is needed, including justification for heuristic thresholds such as the chosen Spearman $\\rho$ cut-offs.\n\nQUESTIONS:\n- In Section 3.1, how exactly are the dense-latent components removed before retraining the SAE?\n- Because the taxonomy relies on a properly trained SAE, could under- or over-fitting of the autoencoder bias the classification? Wouldn’t a direct statistical analysis of the raw LLM activations avoid this dependency?\n- Are there latents that play more than one role, for example, units that behave both as position latents and context-binding latents?\n- PCA latents are reported, but what interpretive value do they have—how should we understand their function?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies frequently activated latent features (**dense latents**) of SAEs trained on residual stream tokens of language models. The paper defines dense latents as latents active in more than 10% of tokens.\n\nThe paper focuses on JumpReLU SAEs trained on Gemma-2-2B layers and includes some experiments with TopK SAEs and GPT-2 small.\n\nThe paper finds the following:\n- Experiments that ablate the subspace of the residual stream spanned by vectors decoded from dense latents suggest that dense latents are not just results of biases of SAE training.\n- For a great proportion of dense latents there is exactly one another latent with encoder cosine similarity close to -1. These are called antipodal pairs. It seems that each pair represents a single direction, and this is not specific to the JumpReLU SAE.\n- The role of a substantial proportion of dense latents can be understood to some degree (position tracking, context binding, entropy modulating, letters, part-of-speech-like categories, principal component).\n\n---\n\n**In more detail.** The analysis identifies different roles for 10%-80% of dense latents depending on the LM layer. The roles are categorized as:\n1. Position tracking latents: the value depends on the distance from start of sentence, paragraph (in early LM layers) or context (in almost all LM layers).\n2. Context-binding latents: approximately consistent meaning within context, but not between contexts. Present in layers in the middle.\n3. Nullspace latents: features in the final layers that have a very small effect on the unembedding output, but contribute substantially to the residual stream's norm. Harder to interpret. Some of them modulate entropy by scaling through RMSNorm/LayerNorm [27].\n4. Alphabet latents: high-density latents in the last layer that promote tokens that share the initial letter.\n5. Meaningful-word latents: latents that are active on tokens that belong to some higher-level part-of-speech based categories.\n6. PCA latents: have high cosine similarity with the top principal component of the residual stream.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- To my understanding, this paper is a meaningful step in language model interpretability research. It improves the understanding of a substantial proportion of frequently activated SAE latents of language transformer models and gives convincing evidence against the hypothesis that most dense latents are mere artifacts of SAE learning algorithms.\n- As far as I can tell, much of the ideas of what to look for and the methodology are original.\n- The writing is mostly clear and the experimental results are presented well. (I especially appreciate Figure 2.)\n- The source code for reproducing the experiments in the main paper is provided. (I am not sure about the appendix.)\n\n**Weaknesses**\n\n1. Clarity of methodology and presentation could be improved (most important).\n\t1. Are the inputs of SAEs the features after adding the MLPs output to the residual stream?  \n\t2. Is the subspace ablation based on encoder or decoder vectors?  \n\t3. The definition of \"antipodality score\" lacks justification (the paper mentions its drawbacks), but I think that it accomplishes its goal.\n\t4. Relatedly, in Figure 11 I find it unclear how the indices $i, j$ are chosen and whether the antipodality score (color) correspond to $i$ or $j$?\n\t5. Figure 1. c: Why is encoder cosine similarity chosen rather than decoder cosine similarity?  \n2. The discussion of related work (background information) could be improved.\n\t1. Prior work on antipodal pairing [1] and dense latents (\"high-frequency latents\") [2] should be discussed in relation to the present work.\n\t2. While the review of recent work seems sufficiently broad (as far as I can tell), it lacks some depth and context. (This is probably due to a lack of space.)\n3. Clarity of some claims could be improved.  \n\t1. L36 \"antipodal pairs, effectively reconstructing specific subspace directions.\" Perhaps something like \"antipodal pairs, each effectively encoding a single direction\" would be more clear.  \n\t2. L45: \"inherently dense mechanistic functions\" might not be clear before reading section 3.1.  \n\t3. What is the meaning of \"a linear combination of the “true” basis works for reconstruction too\"?\n\nMinor suggestions:\n- Clarity and errors:\n\t- L78: \"many pairs\" could be expressed more precisely.\n\t- Figure 11 a: Consider showing the 10% threshold.\n\t- Figure 5 b: The boxplots are a bit hard to recognize. What are the dotted lines?\n\t- L62: typo: \". and\" -> \", and\"\n\t- The context of Figure 16 seems to be missing. What are the run lengths of? What is the parameter of the Bernoulli distribution?\n\t- Consider including an overview of hyperparameters, models and datasets.\n- Questions:\n\t- Would it be interesting to plot distributions of latents (like Figure 11) depending on the frequency?\n\nQUESTIONS:\n**Questions:**\n1. Please consider the questions under weaknesses, point 1.\n2. Do you know whether the presence of (almost) antipodal pairs is a consequence of the frequency of the latent (more frequent features need to be reconstructed more accurately)? Does reducing the dictionary size reduce the number of antipodal pairs?\n3. Do you know how much different types of dense latents overlap?\n\nI apologize for any errors on my part.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study dense latents learned by SAEs trained to reconstruct LLM residual stream representations (with a focus on SAEs pre-trained on Gemma Scope). The authors identify a subset of these latents and classify them to distinct functions. These latents seem to be very scattered in functionality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The paper contains some novel analyses and findings regarding the statistics and prevalence of dense SAE features.\n* Analyzing dense SAE features is important. \n* Classification of dense features is performed, with some novel functionality proposed.\n* The idea of dense latents that can bind to relevant context sounds interesting (but unfortunately under-explored).  \n\nWeaknesses:\n* The paper contains a large list of analyses that are not well connected or seem to bear much relevance to each other. What is the relevance of the antipodality? How are these six different types of dense features related to each other? Some features, like entropy regulating and positional features have been discovered before, as the authors note in the related works section.\n* The related works section is extremely brief. The authors mostly just cite work without discussing it.\n* For some of the latents (like the concept binding one), it's unclear if this is an actually meaningful latent or an artifact. It's a potentially very interesting functionality - I feel like the authors could have dedicated more space and time to analyze this type of dense latent rather than giving a large list of latents, some of which have been covered before in the literature\n\nIn the end, the paper is too unfocused in it's current form. It's not clear if the goal of the paper is to classify as many of these dense features as possible, or to argue that dense features are important/formulate a clear theory about their existence. I think the second strategy is very interesting and I would be willing to raise my score if the authors can expound upon this. The analyses conducted wrt to the concept binding latent is a good start, although it is not clear to me if these bind to distinct types of concepts depending on the context (like latents shown in Fig 3), or if they also specialize on a single concept, but activate densely around it.\n\nOne could potentially write a whole paper about dense features like this (if one can indeed show that they have systematic activity patterns, which isn't really that clear from the brief treatment they get in this paper).\n\nQUESTIONS:\n* The antipodality result is interesting, but unfortunately not really connected to the subsequent results of the paper (except a little bit for the context-binding latents). Why is antipodality relevant in the context of density? Why do we see an antipodal pair of features that track 'casino facts' vs 'looking for a buyer'. These concepts do not seem related in any way, except that they both feature in that particular context?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the nature of dense latents in sparse autoencoders (SAEs) trained on language model activations. An SAE latent is considered ‘dense’ when firing for at least 0.1 of the inputs. While SAEs are designed to yield sparsely activating, interpretable features, a significant fraction of latents are found to activate on a large number of tokens. The authors challenge the view that these dense latents are undesirable artifacts, instead demonstrating that they often reflect meaningful directions in the residual stream. This paper finds that many of these dense latents capture interpretable information and often fall into one of the following categories: position latents, context-binding latents, (quasi) nullspace latents, alphabet latents, meaningful-word latents, and PCA-latents.  The paper studies how this taxonomy of dense latents changes across model layers. The authors demonstrate that a given latent is in a particular category through a variety of ablation, steering, and quantitative analyses.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths}$\n\n- Novel insight into SAE behavior: The paper tackles an underexplored phenomenon—the prevalence and utility of dense latents in SAEs—and provides evidence that these are not just training noise, but reflect intrinsic model structure.\n\n- Careful empirical analysis: The authors use ablations, cosine similarity metrics, and probing techniques (e.g., entropy, steering, AUC prediction from part-of-speech) to characterize dense latents.\n\n- Antipodal structure: The discovery that dense latents form antipodal pairs and correspond to specific directions in the residual stream is a particularly compelling geometric observation.\n\n- Layer-wise functional taxonomy: The paper organizes dense latents into meaningful classes and analyzes their distribution across model layers, revealing a shift from structural to semantic to output-oriented features.\n\n$\\textbf{Weaknesses}$\n\n- Lack of geometric contextualization: While the paper makes a strong case that dense latents are not artifacts, it could further contextualize this by considering known geometric properties of LLM representations (e.g., outlier dimensions, early-to-late layer transformations, or anisotropy in the residual stream). \n\n- Sparse vs. dense comparison is missing: The paper focuses heavily on dense latents but does not compare them to sparse latents. Are the features learned by sparse latents of a different nature, or do they partially overlap in function? This comparison would provide crucial context.\n\n- Taxonomy completeness unclear: The paper acknowledges that fewer than half of dense latents fall into its proposed taxonomy. Quantifying this more precisely and identifying predictive signatures of interpretability would strengthen the claims.\n\n- Ambiguous claims: Some empirical observations (e.g., context-binding latents activating on \"specific but varying concepts\") are not clearly explained and may need clarification or better examples.\n\nQUESTIONS:\n1. How do the types of features recovered by sparse latents compare to those recovered by dense latents? Do they represent different aspects of model computation?\n\n\n2. What fraction of dense latents do not fall into your taxonomy? Are there properties that reliably predict whether a dense latent is interpretable?\n\n3. In Section 3.1, how exactly is the dense subspace identified and ablated? How large is it, and what is the impact of its ablation on downstream SAE behavior?\n\n4. Could the antipodal structure of dense latents be an emergent by-product of L2 reconstruction loss rather than an interpretable representation structure?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}