Automatically Interpreting Millions of  Features in Large Language Models

Authors: Gonçalo Santos Paulo, Alex Troy Mallen, Caden Juang, Nora Belrose
Primary area: interpretability and explainable AI
Keywords: interpretability, language model, sae, features, explanation

Abstract:
While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which can be more easily interpretable. However, SAEs can have millions of distinct latents, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language interpretations for SAE latents using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of interpretations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a latent, which we find explains latents that are not recalled by existing methods. We propose guidelines for generating better interpretations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. Our code is available at https://anonymous.4open.science/r/interpreting_latents/.

Decision: reject
Reviewer scores: 5, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729471739385

thread_type:
official_review

<thread_object>
creator: reviewer_4nph

<content>
summary:
The paper presents an open-source automated pipeline that uses large language models to generate natural language explanations for millions of features in Sparse Autoencoders (SAEs), addressing the challenge of manual interpretation. It introduces five efficient scoring techniques, including intervention scoring, and demonstrates that SAEs are more interpretable than neurons, offering insights into semantic similarity across model layers.

strengths:
- The scale gets improved from previous sota.
- This work lies in an interesting direction.

weaknesses:
- This work did not compare the new evaluation metric with the previous evaluation metric (https://openai.com/index/language-models-can-explain-neurons-in-language-models/) in a solid form. Having similar conclusions as previous approach should not serve as a solid evidence that the new metric is as good as the previous evaluation metric.
- Comparing SAEs in adjacent layers (1) lacks support of motivation and (2) is not well supported. See Questions for details.
- The presentation is poor: readers can not capture the main contribution of this paper with a normal reading flow. The contribution of this work seems to be concentrated on a new evaluation metric, but Section 5 cuts in to discuss about behaviors of the behaviors of SAEs. I would strongly recommend to reorganize the paper to one single claim, with evidence from both sides supporting it.

questions:
1. How is the correlation of the proposed evaluation metric with the original metric?
2. How is the efficiency of the proposed evaluation metric compared to the original metric?
3. Previous work (https://transformer-circuits.pub/2023/monosemantic-features) has already shown SAEs have more interepretable features than neurons. What's the purpose of validating this result?
4. Why comparing SAEs in adjacent layers? (Why is it interesing?)
5. How is comparing SAEs in adjacent layers related to your evaluation metric?
6. Are there any prior work that supports your method in evaluating the SAEs in adjacent layers?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730668337157

thread_type:
official_review

<thread_object>
creator: reviewer_s55e

<content>
summary:
This paper investigates sparse autoencoders (SAEs), which project activation representations into a sparse high-dimensional latent space for interpretability. To automatically explain the large number of latent features, this paper proposes an LLM-based framework, explaining millions of latents across multiple models, layers, and SAE architectures. Four evaluation methods are proposed, including detection, fuzzing, surprisal, embedding, measuring the extent to which an explanation enables a scorer to discriminate between activating and non-activating contexts. Additionally, an intervention scoring is proposed to interpret a feature’s counterfactual impact on model output.

strengths:
-	This paper focuses on an important research problem of SAEs producing a large number of latent features that require automatic explanations and evaluations.
-	Several evaluation metrics have been proposed to assess the generated explanations, comparing them across different explainer models, SAEs, and layers.
-	Some findings are interesting. For instance, sampling examples that are shown to the explainer model may increase the scores of features, highlighting a problem with current auto-interpretability evaluations. Experimental results suggest a priority for training wider SAEs on a smaller subset of residual stream layers. These may provide valuable insights for future research.

weaknesses:
-	The writing and clarity of the paper could be improved. The current version is somewhat difficult to follow and understand. Please see my detailed questions below.
-	Many observations are presented without in-depth analysis or further exploration. For instance, in Section 3.1, the impact of context length and latent space size on activation data is mentioned, but it is unclear how these results relate to the selection of 256 tokens. In Section 3.2, the claim that “showing such short contexts to the explainer model hinders the correct identification of latents with complex activation patterns” seems to contradict the use of short activating examples with 32 tokens. In Section 3.3, while qualitative results for different sampling strategies are provided, it is not clear how to optimize the sampling process. Additionally, in Section 4.1, the statement “The imperfect correlations hint at either shortcomings of the scoring metrics or the fact that these metrics can measure different qualities of explanations” lacks clarity. What specific qualities of explanations are measured by these automatic metrics?
-	The generated explanations largely rely on the prompts and explainer models. It is unclear how generalizable the results are and to what extent specific prompts and explainer models may affect the quality of the generated explanations.
-	While several automatic evaluations are compared, it is unclear whether they accurately reflect the quality of explanations. It would be beneficial to conduct human evaluations, at least on a small set, and compare the correlation with automatic evaluations.

questions:
-	Why is the context length of 256 chosen for activation collection? Is this value based on empirical selection?
-	Why is the activating example limited to only 32 tokens, given that short contexts may hinder the correct identification of latents with complex activation patterns?
-	What accounts for the differences in explanations between “randomly sampling” and “uniformly sampling”?
-	In Section 5, the statement that “if the explanation for latent α at layer j is very different from the explanation for the same latent at layer j + 1, this would suggest that our pipeline is inconsistent and noisy” raises a hypothesis. Can you elaborate on it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693106737

thread_type:
official_review

<thread_object>
creator: reviewer_8dae

<content>
summary:
The paper builds an open-source automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features with LLMs. The framework has been evaluated on various dimensions, including SAE size, activation function, loss, and LLMs. Five new scoring techniques are proposed. The paper finds that SAEs trained on nearby layers' the residual stream are highly similar. And they are also more interpretable than neurons.

strengths:
- The open-source framework is comprehensive and valuable for large-scale SAE analysis. The experiments are well designed to illustrate the effectiveness of the proposed method.
- The metrics proposed in this paper provides more dimensions to evaluate the generated explanations, which would be valuable for the SAE community.
- Some of the findings are meaningful to the SAE community. For example, larger latent SAE learn more dataset-specific latents. The relations between different sampling approaches and the generated explanations. And the high correlations between latents at adjacent layers.

weaknesses:
- The method is a bit hard to understand for readers who are not familiar with SAEs. For example, how section 3.1 is related to 3.2? Would be more illustrative if a figure of the whole pipeline is provided.
- It would be more clear to provide a simple example of explaining the latents of SAEs. And even better if an example involves the whole workflow of this framework is provided.

questions:
The framework proposed in this paper is a valuable tool for the SAE community. It provides an automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features. However, the authors better consider to write a more accessible version for readers who are not familiar with SAEs, especially for section 3. It is a bit difficult to follow it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704260653

thread_type:
official_review

<thread_object>
creator: reviewer_yat1

<content>
summary:
This paper introduced five automated scoring methods to score the explanations of SAE latents, and discussed the shortcomings of existing scoring techniques. The paper also finds that SAE trained on nearby layers are highly similar, and provided actionable insights for practitioners to train wider SAEs instead of narrower SAEs to be efficient when there’s a compute constraint.

strengths:
- Given the large sizes of SAEs nowadays and an increasing need for model explainability, automatically generating explanations of SAE latents efficiently is an important topic.
- The paper is well written, with ablations of design choices clearly described.

weaknesses:
- Format: There are no line numbers, and it's showing "Under review as a conference paper at ICLR **2024**" instead of **2025** at the top.
- The low correlation between different evaluation methods in Table 1 is concerning. Since the simulation method proposed in prior work is vetted and established, the new ones proposed in this work should at least have strong rank correlation (> 0.7) with it to prove that they work. Since this the scoring methods are the primary contribution in this paper, the authors should conduct more rigorous tests to ensure their validity. I would also encourage the authors to conduct experiments with ground truth explanations (e.g., SAE latents with known explanations found in prior work, or easily constructed an embedding model responding to a known concept/explanation), to make a stronger case in terms of the reliability of these new methods.
To add on, instead of correlation of the raw scores, it might make more sense to look at the "rank" correlations of different methods.
- The authors claim that the new methods are more efficient than prior scoring methods without actually quantifying the efficiency gain to support the claim. If efficiency gain is one of the highlights of these scoring methods, the authors should consider comparing runtime of different methods to support the claim.
- The author mention “Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-k postprocessing.” in the abstract as one of the main findings, but the details cannot be found in the main paper but in the appendix. The authors should consider moving it to the main paper if this is one of the main claims.
- Reproducibility: The author mentioned a plan to open-source the project, but it's hard to evaluate the quality of their code for reproducibility purpose either since it's not provided as one of the supplementary files.

questions:
- Missing a highly relevant work, "Explaining black box text modules in natural language with language models". How does their scoring method compare to the ones proposed in this paper?
- Can the authors explain  the negative correlation between the fuzzing score and intervention score in figure 4? If they are both useful scoring methods, why would the correlation be negative?
- Unlike the claim in the paper, figure 5 is still showing statistical alignment across layers. Can the authors provide evidence for semantic similarity? (e.g., compute explanation similarity across layers instead of the matrix statistics)

*Minor issues*
- The authors say “we introduced *five* new techniques” in the abstract, and “We addressed issues with the conventional simulation-based scoring and introduced *four* new scoring techniques” in the conclusions. The readers might get confused in terms of the number of methods actually introduced in the paper, if it’s five, please be consistent throughout the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary of Scientific Claims and Findings  
The paper introduces a novel automated framework using large language models (LLMs) to generate and evaluate natural language explanations for Sparse Autoencoder (SAE) features. It proposes five new scoring methods—detection, fuzzing, surprisal, embedding, and intervention scoring—to assess the interpretability of these features. The paper also provides insights into the high similarity of SAEs trained on nearby layers and suggests that wider SAEs are more efficient for certain tasks under computational constraints. The authors claim their methods are more efficient and offer actionable insights for practitioners in model explainability.  
   
## Strengths  
1. **Relevance and Importance**: The paper addresses a timely issue of model interpretability in the context of large SAEs, which is crucial given the growing complexity of modern neural networks.  
2. **Comprehensive Framework**: The work presents a comprehensive open-source pipeline that could significantly aid researchers in analyzing SAEs.  
3. **Novel Scoring Techniques**: The introduction of multiple scoring techniques offers a broader set of tools for evaluating the quality of explanations, contributing valuable methodologies to the community.  
4. **Well-Designed Experiments**: The experiments are well-structured to validate the effectiveness of the proposed methods, providing meaningful insights into SAE behavior.  
   
## Weaknesses  
1. **Clarity and Presentation**: Several reviewers noted that the paper's presentation could be improved. The organization sometimes makes it difficult to follow the main contributions and arguments.  
2. **Validation and Comparison**: The paper lacks a solid comparison with previous evaluation metrics. The correlation between the new and established methods is not robust, which raises questions about the validity and reliability of the new metrics.  
3. **Depth of Analysis**: Some findings are presented without extensive analysis, limiting the depth of insights offered by the paper.  

 After thorough consideration of other submissions in the same batch, I find myself recommending rejection for this paper. I encourage the authors to refine their work and consider resubmission to a future conference or journal.