Title: Revising and Falsifying Sparse Autoencoder Feature Explanations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Mechanistic Interpretability
Abstract: Mechanistic interpretability research seeks to reverse-engineer large language models (LLMs) by uncovering the internal representations of concepts within their activations. Sparse Autoencoders (SAEs) have emerged as a valuable tool for disentangling polysemantic neurons into more monosemantic, interpretable features. However, recent work on automatic explanation generation for these features has faced challenges: explanations tend to be overly broad and fail to take polysemanticity into consideration. This work addresses these limitations by introducing a similarity-based strategy for sourcing close negative sentences that more effectively falsify generated explanations. Additionally, we propose a structured, component-based format for feature explanations and a tree-based, iterative explanation method that refines explanations. We demonstrate that our structured format and tree-based explainer improve explanation quality, while our similarity-based evaluation strategy exposes biases in existing interpretability methods. We also analyze the evolution of feature complexity and polysemanticity across LLM layers, offering new insights into information content within LLMs' residual streams.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to address overly broad explanations produced by autointerpretability methods by introducing a strategy to falsify generated explanations through sourcing similar sentences as negative samples, hence making them more precise. In addition, the authors propose structured scheme to produce fine-grained explanations along with tree-based method for explanation refinement. The experiments confirm the bias of current autointerpretability methods toward recall at the cost of precision, and also show how the structured explanation scheme and tree-based refinement can improve generated explanations. Lastly, the authors provide analyses of how complexity and polysemanticity of the produced explanations evolve across layers by using an external LLM (instruction-tuned Gemma2 27B) as a judge.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The issue of how autointerpretability methods are heavily biased towards recall is relatively understudied. This paper addresses a crucial gap in current approaches by proposing similarity-based method for sentence sourcing to evaluate feature explanations.\n- Using a tree-based jailbreaking technique to automatically generate and refine explanations is a valuable contribution. Although it is computationally expensive as alluded in the limitation section, the improvement over one-shot generation is quite significant.\n- The proposed methods in the paper are simple to implement, making it easily applicable in various settings.\n\n**Weaknesses:**\n- There is no human evaluation to confirm whether the proposed methods improve automatic interpretability of SAEs in a meaningful way. Furthermore, using LLM as a judge potentially introduce another biases that could affect evaluation reliability, particularly for concepts across different languages or domains.\n- All experiments are conducted on an uncopyrighted subset of the Pile, which may not fully represent all possible concepts that a model might encode. A more detailed discussion on how the dataset choice might affect the results would be beneficial.\n- While features can be polysemantic as defined in L46 and L201, more thorough explanations of how this can be different from the property of features in SAE latent space that is known to be more monosemantic and interpretable (L31) is needed, to provide readers with clarity.\n\nQUESTIONS:\n- L44: Explanatations → Explanations?\n- L25: Missing citation to [(Bolukbasi et al., 2021)](https://arxiv.org/abs/2104.07143).\n- L200: You mentioned that the explainer LLM is prompted such that each component corresponds to a monosemantic concept. Is there any guarantee for that?\n- Do you have any conjecture regarding how different width of SAEs might affect the quality of explanation for polysemantic feature?\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies feature explanations of Sparse AutoEncoders (SAEs) by using LLMs. For this, the paper builds on the idea of Bills et al. (2023) to use an iterative approach to explain individual neurons of SAEs by prompting an LLM. By proposing a tree-based explanation method and a structured, component-based format, the paper tackles challenges of Bills et al. (2023) such as overly broad explanations. Additionally, the paper analyzes the evolution of feature complexity and polysemanticity across layers of different models. The claims and statements of the paper are backed-up by empirical results by prompting LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Clarity:** The paper is well-written, well-organized and easy to follow.\n- **Clarity:** The motivation of the paper is clear (see summary).\n- **Clarity:** The mathematical formalization in 3.2 is good.\n- **Originality:** The paper clearly describes how the proposed method is different to the original work of Bills et al. (2023).\n\n### Weaknesses\n\n- **Originality:** The paper seeks to advance the field of XAI, in particular, understanding LLMs. Method-wise, it is clear how the method advances over the original work of Bills et al. (2023). However, it is not clear how the proposed method helps in understanding LLMs compared to Bills. In the abstract it is stated that \"this work addresses these limitations [overly broad explanations and polysemanticity]\" but the results demonstrate that these limitations are not addressed (e.g., see statement L316-317) and only marginally quantitatively improved (see figures and tables)---completely ignoring the true goal of XAI on enhancing the understanding of LLMs.\n\n- **Significance:** The contribution of the paper is marginal as it is mainly the approach of Bills et al. (2023) with a new prompting strategy. \n\n- **Quality:** After reading the paper, I'm not sure about the take away of the paper accept from what was already know from Bills et al. (2023): you can explain an LLM with an LLM to some extent but it might fail for unknown reasons. Moreover, the approach the paper builds upon is the explanation of a black-box with a black-box (a notoriously ill-defined task; e.g., see [1]). Instead of presenting numerical results, it would be beneficial to show how this approaches enhances our knowledge about LLMs (similar to the presentation of Bills et al.). For instance, why does a model hallucinate or, simpler, how polysemanticity of neurons impacts text generation. I noticed that the appendix provides some illustrative examples but a thoroughly discussion and concise conclusion is missing. \n\n- **Quality:** Certain choices of design decisions are not well-justified. For instance, why is the max-norm a good choice for the ordering, why a random sample of top quantile, or how is top defined. In order to understand the sensitivity of the proposed method it is required that the paper presents some kind of ablation study or discusses the selection of these hyperparameters. Otherwise it is not clear whether the observed results are only good by chance or cherry-picked. Another example is the usage of cosine similarity where results suggest that this could be inappropriate [2].\n\n- Minor issue **Quality**: References should be to publications not arXiv (e.g., [4])\n\nI will elaborate on a few more things in the questions section.\n\n[1]: Cynthia Rudin: Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. 2018.\n\n[2]: Steck et al.: Is Cosine-Similarity of Embeddings Really About Similarity? 2024.\n\nQUESTIONS:\n1. In the abstract it is stated the tree-based method \"refines explanations\". This is assessed in 5.3 (Feature Complexity) by using an LLM judge. How can we sure that LLM judge does a good job? In the current form, the assessment of explanations happens by a metric (the LLM judge) that we don't understand.\n2. What does \"close counterexample\" mean? How is closeness mathematically defined?\n3. Section 4.2, Branching step: The creation of the child explanations is without a score. How is it possible then to perform the pruning in step 3? It would be beneficial to formalize the methods in Section 4 by the notation of Section 3.2 (pseudo code). This would help readability and would better connect the different sections of the paper.\n4. Does no ground-truth mean zero activation? I found this confusing in the paper. Also the statement in L213 is equivalent to $f*(\\mathbf{t})_{i}=0,\\forall i$ under the restriction that all values have to be positive, which is more difficult to grasp from the presented equation.\n5. L217: Random non-activating sentences: \"defined as the proportion of positive elements\". This means that the value is in [0,1]. However, looking at Fig.3 it is above 1. How is this possible?\n6. L306: Is there any evidence for this hypothesis?\n\n**Some comments unrelated to my assessment of the paper:**\n\n- The wording \"Feature Learning\" and, at some place in the paper, \"Learning\" is misleading since there is no learning in an ML sense involved. It is prompting of an LLM.\n- L112: $f(t)$ not $f(\\mathcal{T})$; by definition f **maps** elements from $\\mathcal{T}$\n- L115: **an** explanation\n- L120: What constraint?\n- The wording of the paper should be improved. For example, \n   - L183: I wouldn't call this causally as causal has a strong meaning and clear definition in mathematics that I doubt to hold here.\n   - The use of training where inference or prompting is meant.\n- L146: I guess it must be \"negative log probabilities\".\n- L257: The font type of LLama 4 Scout is different than the others.\n\nLIMITATIONS:\nYes, the limitations discussion is already good. However, it could be extended in the direction of how this method helps to understand LLMs. For instance, the second point mentions polysemanticity: What is the consequence if this fails?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of explaining features extracted by sparse autoencoders (SAEs) --- a method to obtain disentangled representations for a neuron in models such as LLMs --- via text in a faithful manner. A common way to do this is to look at what activates a given SAE latent and then ask an LLM for an explanation that encompasses highly activating inputs and excludes low activating inputs. Simulation experiments are then performed to check if predicted activations based on just the input and the explanation match true activations. This work recognizes that getting such an explanation from an LLM is challenging, since the explanation may be overly broad, which would result in high recall but low precision of simulation. It proposes using structured explanations and a tree based search to obtain better explanation candidates, and similarity based retrieval methods to better retrieve negative examples, i.e. examples that do not activate the latent. Experiments are performed on Pile with three LLMs to show that the proposed approach yields more false positives (i.e. avoids having too broad explanations), has better scores on simulation experiments, and can be used to analyze explanation complexity and polysemanticity.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper addresses an important problem, since getting faithful explanations is important when using models in safety critical domains.\n2. The proposed approach is simple but principled, and quantitative results appear to show that it is effective.\n\n## Weaknesses\n\n1. The key weakness for me is that the paper lacks qualitative examples that show why the proposed method is useful. While quantitatively one can see that it is better at finding false positives (Figure 3) and getting better scores (Figure 4), how does this finally affect the kind of explanations that are obtained for SAE latents? For instance, are more granular explanations obtained? Showing examples of explanations for latents along with what inputs activate them would be very helpful (including negative examples). Right now it is not very clear how useful the approach would be in practice, and whether the quantitative improvements are significant. The examples shown in the Appendix seem to primarily be prompting examples, but if this is provided there please point to them in the rebuttal.\n\n2. A problem that has been observed with SAEs is feature absorption and feature splitting (e.g. discussed in [1]). Intuitively, one would expect that this approach would be able to detect such instances better. This could be a very useful evaluation to have to show the practical benefits of the approach, since e.g. situations involving feature splitting are likely to show up false positives as discussed in the paper. It would also help understand if the sentence encoder similarities used in the paper are helpful for finding ideal candidate sentences. Less important, but a study on different SAE types (e.g. using Matryoshka SAEs) could also be useful.\n\n3. The trends shown for explanation complexity and polysemanticity (Figure 5) appear somewhat weak, and a more granular analysis would be helpful. For example, the results also depend on how good the trained SAE is, which would vary across models and layers (in terms of reconstruction loss, latent dimension size, sparsity, etc.) and could affect the conclusions made. Additionally, similar to Weakness 1, having examples would also help here, particularly for polysemanticity. For example, are the explanations for the polysemantic neurons conceptually similar or very distinct?\n\nQUESTIONS:\nPlease refer to the Weaknesses. I would be happy to increase my score if the concerns on motivation and having a more granular analysis are addressed.\n\nLIMITATIONS:\nYes, but an additional limitation to consider is the reliance on LLMs for obtaining explanations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper proposes three major innovations to the standard automated interpretability pipeline for sparse autoencoder features. First, it introduces a similarity-based strategy to find semantically related features to a given query feature. The LLM explainer is then asked to generate explanations that make it possible to distinguish activating examples for the query feature from activating examples for the semantically related features. Second, it introduces a JSON-based format for structured explanations, enabling the use of more sophisticated explanations than were used in prior work. Third, it introduces a tree-based approach for iteratively improving structured explanations.

STRENGTHS AND WEAKNESSES:
Strengths:
- Structured explanations enable a more direct measurement of the degree of polysemanticity in a feature (number of explanation components)
-  Tree-based explanation is novel and appears to open up new directions for research

Weaknesses:
- Focuses on top-activating examples, which are known to be significantly more interpretable than medium or low activation quantiles
- Tree-based explanation method is computationally intensive

QUESTIONS:
1. Have you tested your innovations on lower quantiles of the activation distribution? Do you anticipate that your results will be qualitatively different on, say, the middle tercile of the feature activations?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-10