Automatically Interpreting Millions of Features in Large Language Models

## Abstract

Abstract While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which may be more easily interpretable. However, these SAEs can have millions of distinct latent features, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of explanations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a feature, which we find explains features that are not recalled by existing methods. We propose guidelines for generating better explanations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. We use our explanations to measure the semantic similarity of independently trained SAEs, and find that SAEs trained on nearby layers of the residual stream are highly similar. Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top- k 𝑘 k italic_k postprocessing. Our code is available here , and our explanations are available here .

## 1 Introduction

Large language models (LLMs) have reached human level performance in a broad range of domains (OpenAI, ), and can even be leveraged to develop agents (Wang et al., ) that can strategize (Bakhtin et al., ), cooperate and develop new ideas (Lu et al., ; Shaham et al., ). At the same time, we understand little about the internal representations driving their behavior. Early mechanistic interpretability research focused on analyzing the activation patterns of individual neurons (Olah et al., ; Gurnee et al., ; ). Due to the large number of neurons to be interpreted, automated approaches were proposed (Bills et al., ), where a second LLM is used to propose an explanation given a set of neuron activations and the text snippets it activates on, in a process similar to that of generating a human label. But research has found that most neurons are “polysemantic,” activating in contexts that can be very different (Arora et al., ; Elhage et al., ). The Linear Representation Hypothesis (Park et al., ) posits that human-interpretable concepts are encoded in *linear combinations* of neurons. A significant branch of current interpretability work focuses on extracting these features and disentangling them (Bereska & Gavves, ).

Sparse autoencoders (SAEs) were proposed as a way to address polysemanticity (Cunningham et al., ). SAEs consist of two parts: an encoder that transforms activation vectors into a sparse, higher-dimensional latent space, and a decoder that projects the latents back into the original space. Both parts are trained jointly to minimize reconstruction error. SAE latents were found to be interpretable and potentially more monosemantic than neurons (Bricken et al., ; Cunningham et al., ). Recently, a significant effort was made to scale SAE training to larger models, like GPT-4 (Gao et al., ) and Claude (Templeton et al., ), and they have become an important interpretability tool for LLMs.

Training an SAE yields a large number of sparse latent features, each of which needs a natural language explanation. In this work, we introduce an automated framework that uses LLMs to generate an explanation for each latent in an SAE. We use this framework to explain millions of latents across multiple models, layers, and SAE architectures. We also propose new ways to evaluate the quality of explanations, and discuss the problems with existing approaches.

Although other repositories of explanations for SAE features already exist (Lin & Bloom, ), we believe it would be helpful for the community to provide high quality explanations, using open source models, in a way that is reproducible. This kind of exhaustive explanatory work could enable more downstream SAE applications utilizing SAE capabilities like steering and concept localization.

### 1.1 Explaining SAE latents in LLMs

Explaining the latents of SAEs trained on models like Llama 3.1 7b or Gemma 2 9b requires the generation of millions of explanations. As an example, the most extensive open-source set of SAEs available, Gemmascope (Lieberum et al., ), includes SAEs for all layers of Gemma 2 9b and Gemma 2 2b and would require explaining tens of millions of latents. Doing the same for bigger models like Llama 3.1 70b or 405b could easily reach hundreds of millions, if not billions, of latents to be explained, meaning that optimizing the process to obtain these explanations is essential.

## 2 Related work

One of the first approaches to automated interpretability focused on explaining neurons of GPT-2 using GPT-4 (Bills et al., ). GPT-4 was shown examples of contexts where a given neuron was active and was tasked to provide a short explanation that could capture the activation patterns. To evaluate if a given explanation captured the behavior of the neuron, GPT-4 was tasked to predict the activations of the neuron in a given context having access to that explanation. The explanation is then scored by how much the simulated activations correlate with the true activations.

A similar approach was used in Templeton et al. () to explain SAE latents of Claude. In general, current approaches focus on collecting contexts together with latent activations from the model to be explained, and use a larger model to find patterns in activating contexts.

Following those works, other methods of evaluating explanations have been proposed, including asking a model to generate activating examples and measuring how much the neuron activates (Kopf et al., ; Hernandez et al., ). More recently, “interpretability agents” have been proposed, iteratively doing experiments to find the best explanations of vision neurons (Shaham et al., ).

On the other end of the spectrum, a potentially cheaper version of automated interpretability has been proposed, where the model that is being explained doubles as an explanation generation model (Kharlapenko et al., ). A prompt querying the meaning of a single placeholder token is passed to the model and latent activations are patched into its residual stream at the position of the placeholder token during execution, generating continuations related to the latent. This technique is inspired by earlier work on Patchscopes (Ghandeharioun et al., ) and SelfIE (Chen et al., ).

Figure: Figure 1: SAE latents explanations in a random sentence. To visualize the latent explanations produced, we select a sentence taken from the RPJv2 dataset. We selected 4 tokens in different positions in that sentence and filter for latents that are active in different layers. Then we randomly select active latents and their corresponding explanations to display. We display the detection and fuzzing scores of each explanation, which indicate how well it explains other examples in the dataset (see section 3 for details on these scores). The features selected had high activation, but were not cherry-picked based on explanations or scores.
Refer to caption: extracted/5935489/fig1_trial.png

## 3 Methods

### 3.1 Collecting activations

We collected latent activations from the SAEs over a 10M token sample of RedPajama-v2 (RPJv2;  Computer ). This is the same dataset that we used to train our Llama 3.1 8b SAEs, and consists of a data mix similar to the Llama 1 pretraining corpus.

We collected batches of 256 tokens starting with the beginning of sentence token (BOS).(^1^11We throw out the activations on the BOS token when generating explanations for Gemma, as we were told in personal communication that these SAEs were not trained on BOS activations.) The contexts used for activation collection are smaller than the contexts used to train the SAEs, and we find that on average, 30% of the latents of the 131k latent, per layer, Gemma 2 9b SAE don’t activate more than 200 times over these 10M tokens and 15% don’t activate at all. When we consider the training context length of 1024, only 5% of latents don’t fire. When using a closer proxy of the training data, the [“un-copyrighted” Pile](https://huggingface.co/datasets/monology/pile-uncopyrighted), we find that the number of latents that activate fewer than 200 times decreases to 15%, and that only 1% don’t activate at all, even when considering contexts of size 256. Interestingly, in the case of the 16k latent Gemma 2 9b SAE only around 10% of latents activate fewer than 200 times on the RPJv2 sample, suggesting that the larger 131k latent SAE learned more dataset specific latents than its smaller cousin.

### 3.2 Prompting the explaner

Our approach follows Bills et al. () in showing the explainer model examples sampled from different quantiles, but uses a more “natural” prompt where the activating example is shown whole, with the activating tokens emphasized and the strength of the activation shown after the example. See Appendix A.1 for the full prompt.

The activating example is selected to only have 32 tokens, irrespective of the position of the activating tokens in the prompt. We found that using COT, at least when explaining using Llama 70b, does not significantly increase the quality of the explanations, while significantly increasing the compute and time required to generate explanations. For this reason, we have not used it for our main experiments. Showing such short contexts to the explainer model hinders the correct identification of latents with complex activation patterns.

### 3.3 Maximally activating contexts

We find that randomly sampling from a broader set of examples leads to explanations that cover a larger set of activating examples, sometimes to the detriment of the top activating examples, see fig. 3. Sampling from top examples often generates more concise and specific explanations that accurately describe the top activating examples but fail to capture the whole distribution. On the other hand, uniformly sampling examples that activate the feature, or ensuring that chosen examples cover all activation strength quantiles, can lead to explanations that are too broad and that fail to capture a meaningful explanation. For examples of these types of failure modes, see fig. A1 and the discussion in the appendix A.2.

Focusing on maximally activating contexts to generate and score explanations biases the types of explanations found and the types of explanations that are highly scored.

### 3.4 Automatically interpreting interventions

Automatic interpretability methods, including most of those explored in this paper, typically look for correlations between activation of a feature and some natural-language property of the input. However, some features are more closely related to what the model will output. For example, we find a feature(^2^22Feature 157 of Gemma 2 9b’s layer 32 autoencoder with 131k latents and an average L0 norm of 51. It has a detection score of 0.6.) whose activation causes the model to output words associated with reputation but does not have a simple explanation in terms of inputs.

We define an *output feature* as a feature that causes a property of the model’s output that can be easily explained in natural language. See sec 3.5.5 for the definition we use in scoring.

Output features can also be described in terms of their correlation with inputs. For example, the “reputation” feature activates in contexts where likely next tokens relate to reputation. However, explaining output features in terms of causal influence on output has two advantages.

- 1.
Scalability. Output features are easier to describe in terms of effect on output because the explainer only needs to notice a simple pattern of output. The pattern of inputs this feature correlates with is more complex. Explaining a feature by correlating it with inputs requires approximating the computation of the subject model leading up to that feature, which may be challenging when subject models are highly capable and performing difficult tasks. Some features’ influence on output, however, might remain easily explainable.
- 2.
Causal evidence. We might like to know that a feature causes a property of the model’s output so that we can steer the model. Previous work has shown that existing auto-interpretability scores fail to accurately capture how well a given explanation can predict the effect of intervening on a given neuron (Huang et al., ). Further, prior work has argued that causal evidence is more robust to distribution shifts (Bühlmann, ; Schölkopf et al., ).

### 3.5 Scoring methods

Being able to efficiently evaluate explanations of SAE latents is important for a few reasons. First, practitioners would like to know how faithful each explanation is to the actual behavior of the network. Some latents may not be amenable to a simple explanation, and in these cases we expect the auto-interpretability pipeline to output an explanation with poor evaluation metrics. Secondly, we can use evaluations as a feedback signal for the training of explainer models and tuning hyperparameters in the pipeline. Finally, some SAEs may simply be poor quality overall, and the aggregate evaluation metrics of the SAE’s explanations can be used to detect this.

The quality of SAE explanations has so far been measured via simulation scoring, a technique introduced by Bills et al. () for evaluating explanations of neurons. It involves asking an explainer model to “simulate” a feature in a set of contexts, which means predicting how strongly the feature should activate at each token in each context given an explanation. The Pearson correlation between the simulated and true activations is the simulation score. The standard in the literature is to sample contexts from a “top-and-random” distribution that mixes maximally activating contexts and contexts sampled uniformly at random from the corpus. While oversampling the top activating contexts introduces bias, it is used as a cheap variance reduction technique, given that simulation scoring over hundreds of examples per feature would be expensive (Cunningham et al., ).

In this work, we take a slightly different view of what makes for a good explanation: an explanation should serve as a *binary classifier* distinguishing activating from non-activating contexts. The reasoning behind this is simple. Given the highly sparse nature of SAE latents, most of their variance could be captured by a binary predictor that predicts the mean nonzero activation when the latent is expected to be active, and zero otherwise. In statistics, [zero-inflated](https://en.wikipedia.org/wiki/Zero-inflated_model) phenomena are often modeled as a mixture distribution with two components: a Dirac delta on zero, and another distribution (e.g. Poisson) for nonzero values.

Figure: Figure 2: The new proposed scoring methods. In detection scoring, the scorer model is tasked with selecting the set of sentences that activate a given latent given an explanation. In this work, we show 5 examples at the same time, and each has an identical probability of being a sentence that activates the latent, independent of whether any other example also activates the latent. The activating tokens are colored in green for display, but that information is not shown to the scorer model. For fuzzing scoring, the scorer model is tasked with selecting the sentences where the highlighted tokens are the tokens that activate a target latent given an explanation of that latent. In surprisal scoring, activating and non-activating examples are run through the model and the loss over those sentences is computed. Correct explanations should decrease the loss in activating sentences compared to a generic explanation, but shouldn’t significantly decrease the loss in non-activating sentences. For embedding scoring, activating and non-activating sentences are embedded as “documents” that should be retrieved using the explanation as a query.
Refer to caption: x1.png

As an alternative to simulation scoring, we introduce four new evaluation methods that focus on how well an explanation enables a scorer to discriminate between activating and non-activating contexts. As an added benefit, all of these methods are more compute-efficient than simulation.

Currently, there is no consensus over what makes an ideal explanation. In this work, we chose to focus on the idea that the explanation of the feature should accurately distinguish between which contexts the feature is active and which contexts the feature is not active.

#### 3.5.1 Detection

One simple approach for scoring explanations is to ask a language model to identify whether a whole sequence activates a SAE latent given an explanation. Currently, the bottleneck of scoring is on the autoregressive nature of token generation. Detection requires few output tokens for each example used in scoring, meaning examples from a wider distribution of latent activation strengths can be used at the same expense. By including non-activating contexts, this method measures both the precision and recall of the explanation.

Detection is more “forgiving” than simulation insofar as the scorer does not need to localize the feature to a particular token.

Both this method and fuzzing (described next) can leverage token probabilities to estimate how certain the scorer model is of their classification, and this is an effect that we believe can be to improve the scoring methods. Details on the prompt in Appendix A.3.1.

#### 3.5.2 Fuzzing

Fuzzing is similar to detection, but at the level of individual tokens. Here, potentially activating tokens are delimited in each example and the language model is prompted to identify which of the sentences are correctly marked. Evaluating an explanation on both detection and fuzzing can identify whether a model is classifying examples for the correct reason.

While in detection, non-activating examples were used, in fuzzing we choose activating contexts and randomly select non-activating tokens. Details on the prompt on Appendix A.3.2

#### 3.5.3 Surprisal

Surprisal scoring is based on the idea that a good explanation should help a base language model $\mathcal{M}$ (the “scorer”) achieve lower cross-entropy loss on activating contexts than it would without a relevant explanation. Specifically, for each context $\mathbf{x}$, activating or non-activating, we measure the *information value* of an explanation $\mathbf{z}$ as $\log p_{\mathcal{M}}(\mathbf{x}|\mathbf{z})-\log p_{\mathcal{M}}(\mathbf{x}|
\tilde{\mathbf{z}})$, where $\tilde{\mathbf{z}}$ is a fixed pseudo-explanation. A good explanation should have higher information value on activating examples than on non-activating ones. The overall surprisal score of the explanation is given by the AUROC of its information value when viewed as a classifier distinguishing activating from non-activating contexts. Details on the prompt and how to compute the score in Appendix A.3.3.

#### 3.5.4 Embedding

We can imagine explanations of latents as “queries” that should be able to retrieve contexts where the latent is active. These contexts can act as “documents” that are embedded by an encoding transformer, and we can use the similarities between the query and the documents to distinguish between activating and non-activating contexts. If the encoding model is small enough, this technique is the fastest and opens up the possibility to evaluate a larger fraction of the activation distribution. We have seen that using a larger embedding model didn’t significantly improve the scores, see fig A2, although we believe that this approach was under-investigated. Details on the prompt, on the embedding model and on the way to compute the score in Appendix A.3.4.

#### 3.5.5 Intervention scoring

Unlike the above four context-based scores, intervention scoring interprets a feature’s counterfactual impact on model output. We quantify the interpretability of an intervention $I$ with explanation $\mathbf{z}$ on a distribution of prompts $\pi$ as the average decrease in the scorer’s surprisal about the explanation when conditioned on text generated with the intervention.

$$ $S(I,\mathbf{z};\pi)=\mathbb{E}_{\mathbf{x}\sim\pi}\Big{[}\mathbb{E}_{\mathbf{i }\sim\mathcal{G}_{I}(\mathbf{x})}[\log p_{\mathcal{M}}(\mathbf{z}|\mathbf{i})] -\mathbb{E}_{\mathbf{g}\sim\mathcal{G}(\mathbf{x})}[\log p_{\mathcal{M}}( \mathbf{z}|\mathbf{g})]\Big{]}$ (1) $$

$\mathcal{G}(\mathbf{x})$ is the distribution over subject model generations given prompt $\mathbf{x}$ at temperature 1. In $\mathcal{G}_{I}(\mathbf{x})$, the intervention is applied to the subject model as it generates. In practice we estimate the quantity $S$ by sampling one clean and one intervened generation for each sampled of prompt.

Sufficiently strong interventions can be trivially interpretable by causing the model to deterministically output some logit distribution. Therefore, interpretability scores of interventions should be compared for interventions of a fixed strength. We define the strength $\sigma$ of an arbitrary intervention $I$ as the average KL-divergence of the model’s intervened logit distribution with reference to the model’s clean output.

$$ $\sigma(I;\pi)=\mathbb{E}_{\mathbf{x}\sim\pi}\big{[}D_{KL}(p_{\text{subject}}( \cdot|\mathbf{x})\ ||\ p_{\text{subject},I}(\cdot|\mathbf{x}))\big{]}$ (2) $$

See Appendix A.7 for details on the explanation and scoring pipeline we use in our intervention experiments.

## 4 Results

### 4.1 Comparing scoring methods

Since simulation scoring is an established method, we measure how our other context-based scoring techniques correlate with simulation, as well as how they correlate between themselves (see tables 1 and A1).

We find that fuzzing and detection have the highest Spearman correlations with simulation scoring, at 0.70 and 0.44 respectively. The imperfect correlations hint at either shortcomings of the scoring metrics or the fact that these metrics can measure different qualities of explanations.

Simulation correlates more strongly with fuzzing than with detection, which is expected because fuzzing is similar to simulation scoring. It correlates weakly with surprisal and embedding scoring. On the other hand, surprisal and embedding scores correlate more with detection scoring than with fuzzing, and most strongly with each other. Due to the speed of embedding scoring, we believe it to be a potentially scalable scoring technique for quick iteration.

#### 4.1.1 Intervention scoring

In figure 4, we see that the intervention score we propose is a valuable contribution to the set of automatic interpretability metrics because it (a) distinguishes between features from a trained SAE and random features and (b) recalls features that context-based scoring methods fail to interpret.

**Table 1: Spearman correlation computed over 600 different latent scores**
|  | Fuzzing | Detection | Simulation | Embedding | Surprisal |
| --- | --- | --- | --- | --- | --- |
| Fuzzing | 1 | 0.73 | 0.70 | 0.41 | 0.30 |
| Detection |  | 1 | 0.44 | 0.71 | 0.62 |
| Simulation |  |  | 1 | 0.33 | 0.20 |
| Embedding |  |  |  | 1 | 0.79 |
| Surprisal |  |  |  |  | 1 |

Figure: Figure 3: Fuzzing and detection scores for different sampling techniques. Panels a) and b) show the distributions of fuzzing and detection scores, respectively, as a function of different example sampling methods for explanation generation. Sampling only from the top activation gets on average lower accuracy in fuzzing and on detection when compared with randomly sampling and sampling from quantiles. The distributions from random sampling and sampling from quantiles are very similar. Panels c) and d) measure how the explanations generalize across activation quantiles, showing that explanations generated from the top quantiles are better at distinguishing non-activating examples in detection, but have lower accuracy on other quantiles, especially on the lower activating deciles. This also happens with explanations generated from examples sampled randomly and from quantiles, but the accuracy does not drop as much in lower activating deciles.
Refer to caption: x2.png

### 4.2 Comparing explanation methods

We use a set of 500+ latents as a testbed to measure the effects of design choices and hyperparameters on explanation quality. Each latent is scored using 100 activating and 100 non-activating examples. The activating examples are chosen via stratified sampling such that there are always 10 examples from each of the 10 deciles of the activation distribution.

We evaluate explanation quality using fuzzing, detection and embedding scores, as those were both quick to compute and easy to interpret. We expect this mix of scores to correlate well with simulation scoring and reflect the extent to which the proposed explanation is valid over both activating and non-activating examples.

Table A6 shows that explanations based on more examples tend to have higher scores. By sampling the examples that are shown to the explainer model instead of showing just the top activating examples it is possible to increase the scores of the features, see fig. 3 and A5. This effect would not be seen if the scoring were done on just the most activating examples, underscoring a problem with current auto-interpretability evaluations, which produce explanations using top activating examples and evaluate them on a small subset of the activation distribution.

Increasing the size of the explainer model increases the scores of the explanations but we don’t find the performance of Claude Sonnet 3.5 to be much higher than that of Llama 3.1 70b, see A7, and both are similar to those generated by a human. We expect that this may be due to the fact that the prompting techniques were first optimized for Llama 3.1, and that there could be potential gains in optimizing the prompting technique of Claude. Not surprisingly, we also see that even for fuzzing and detection, using a smaller scorer model leads to lower scores.

In table A4, we don’t find a significant dependence on whether we use the same dataset as the training set of the SAEs (see A2) whether we used chain of thought or not (see A3) and whether we change the size of the contexts shown.

### 4.3 Comparing SAEs

SAEs with more latents have higher scores, and scores that are significantly higher than those of neurons, which are just slightly better than randomly initialized SAEs, see A8. Neurons are more interpretable if made sparser by only considering the top $k$ most activated neurons on a given token, but still significantly underperform SAEs in our tests, see A8.

The location of the SAEs matters; residual stream SAEs have slightly better scores than ones trained on MLP outputs, see A8. We also observe that earlier layers have lower overall scores, but that the scores across model depth remain constant after those layers, see A4.

A detailed analysis of various factors in the auto-interpretability pipeline is in Appendix A.5.

Figure: Figure 4: Intervention scores. Here we present intervention scores (Sec 3.5.5) for SAE features in Gemma 2 9B at layer 32. Left: SAE features are more interpretable than random features, especially when intervening more strongly. Our explainer also produces explanations that are scored higher than random explanations. Right: Many features that would normally be uninterpreted when using context-based automatic interpretability are interpretable in terms of their effects on output.
Refer to caption: x3.png

## 5 Overlap between latents at adjacent layers

Because each block in a transformer performs an incremental additive update to the residual stream, one would expect SAEs trained at nearby layers to learn similar features. Measuring the degree of feature overlap is interesting for a few reasons. First, if adjacent SAEs learn almost identical features, it may not be worthwhile to train and interpret SAEs at every single residual stream layer. Second, we can use feature overlap to sanity check our auto interpretability pipeline: if the explanation for latent $\alpha$ at layer $j$ is very different from the explanation for “the same” latent at layer $j+1$, this would suggest that our pipeline is inconsistent and noisy. Finally, feature overlap may allow us to estimate the degree of *semantic* similarity between layers, as opposed to mere statistical similarity.

Unfortunately, we cannot simply fix some index $i$ and compare latent $i$ at layer $j$ to latent $i$ at layer $j+1$. This is due to permutation symmetry: given an SAE with parameters $(\mathbf{W}_{e},\mathbf{b}_{e},\mathbf{W}_{d},\mathbf{b}_{d})$ and a permutation matrix $\mathbf{P}\neq\mathbf{I}$, the SAE $(\mathbf{P}\mathbf{W}_{e},\mathbf{P}\mathbf{b}_{e},\mathbf{P}^{T}\mathbf{W}_{d
},\mathbf{P}^{T}\mathbf{b}_{d})$ has identical input-output behavior, and yet its latents are completely “unaligned” with the original. Instead, we use the Hungarian algorithm (Crouse, ) to compute the permutation which maximizes the Frobenius inner product between the decoder weight matrices of the two SAEs. A similar method was used by Ainsworth et al. () to align weights of independently trained MLPs.

In this section, we focus on the 16k latent SAEs trained on Gemma 2 9b. In Appendix A.6, we report the pairwise Frobenius inner products between SAE decoder matrices at different layers before and after aligning them with the Hungarian algorithm. As expected, this simple metric shows that SAEs trained on adjacent layers of the residual stream are more similar than those trained on the MLPs.

To measure the semantic similarity between the explanations, we embed the explanation of each latent and organize the embeddings into a matrix, sorted with the indices found by the Hungarian algorithm. We then compute the Frobenius inner product of these matrices, ignoring the latents that don’t have an explanation on both layers. We observe that the residual stream SAEs have higher semantic overlap for neighboring layers than the MLP SAEs.

Our results suggest that, when training SAEs given a limited compute budget, one should prioritize training *wider* SAEs on a *smaller* subset of residual stream layers, perhaps every $k$^th layer for some stride $k>1$. If feature diversity is a priority, it may make more sense to train SAEs on MLP outputs exclusively, although we find MLPs to be slightly less interpretable than the residual stream, see A8.

It’s worth noting that the Hungarian algorithm could also be used to align latents from independently trained SAEs in other contexts, such as SAEs trained on different checkpoints of the same training run, or even completely different architectures trained on the same data (Moschella et al., ). We leave these extensions to future work.(^3^33All that is needed is an adequate cost function for measuring the similarity of individual latents. SciPy’s implementation supports rectangular cost matrices, allowing SAEs of different sizes to be (partially) aligned.)

Figure: Figure 5: Alignment between embeddings of explanations of the 16k Gemma 2 9b SAEs. After “aligning” the features of each layers by using the decoder weights, we observe that the explanations found for the latents of the residual stream SAE are more similar than for latents of the MLPs. In this figure, alignment is the measured by the Frobenius inner product between the feature embeddings.
Refer to caption: extracted/5935489/hungarian_embedding.png

## 6 Conclusion

Explaining the latents of SAEs trained on cutting-edge LLMs is a computationally demanding task, requiring scalable methods to both generate explanations and assess their quality. We addressed issues with the conventional simulation-based scoring and introduced four new scoring techniques, each with distinct strengths and limitations. These methods allowed us to explore the “prompt design space” for generating effective explanations and propose practical guidelines.

While most latents are well-explained by contexts of length 32, we expect features active over longer contexts to be important and plan to develop better methods for capturing these long-context features. Additionally, although current scoring methods do not account for explanation length, we believe shorter explanations are generally more useful and will incorporate this in future metrics. Some scoring methods also require further refinement, particularly in selecting non-activating examples to improve evaluation.

By analyzing a large number of explanations, we examined overlaps across layers and the distribution of explanations in decoder direction space. Our results suggest that, when compute resources are constrained, it may be more efficient to train wider SAEs on a small subset of residual stream layers, rather than narrower SAEs on all layers.

Access to better, automatically generated explanations could play a crucial role in areas like model steering, concept localization, and editing. We hope that our efficient scoring techniques will enable feedback loops to further enhance the quality of explanations.

## 7 Data and code availability

Together with this manuscript, we release the [sae-auto-interp](https://github.com/EleutherAI/sae-auto-interp)
library and accompanying scripts to guide the reproduction of these results.

In collaboration with Neuronpedia we also release the explanations for the Gemma 2 9b [16k residual stream](https://www.neuronpedia.org/gemma-2-9b/3-gemmascope-res-16k-l0_32plus/) SAEs, the [16k MLP](https://www.neuronpedia.org/gemma-2-9b/24-gemmascope-mlp-16k-l0_32plus/) SAEs, the [131k residual stream](https://www.neuronpedia.org/gemma-2-9b/gemmascope-res-131k-l0_32plus/) SAEs and the Llama 3.1 8b [262k residual stream and MLP](https://neuronpedia.org/llama3.1-8b-eleuther_gp) SAEs. Explanations can be directly downloaded [here](https://huggingface.co/datasets/EleutherAI/auto_interp_explanations/).

## 8 Acknowledgements

We would like to acknowledge Lucia Quirke and David Johnston for the work reviewing and discussing the manuscript. We would like to acknowledge Jacob Drori’s role in the initial development of new scoring techniques. We would also like to thank Joseph Bloom, Sam Marks, Can Rager, Jannik Brinkmann and Sam Marks, Neel Nanda, Sarah Schwettmann and Jacob Steinhardt for their discussions and suggestions in an earlier version of the work. We are thankful to [Open Philanthropy for funding this work](https://www.openphilanthropy.org/grants/eleuther-ai-interpretability-research/).
We are grateful to CoreWeave for providing the compute resources.

## Appendix A Appendix

### A.1 Explainer Prompt

The system prompt if not using Chain of Thought (COT):
{mdframed}

We add to the previous prompt the following if we want to do COT:
{mdframed}

One of the few shot examples of how examples are displayed to the model.
{mdframed}

If COT is used, an explicit example of using COT is demonstrated.
{mdframed}

Afterwards an explanation is added to the example
{mdframed}

### A.2 Examples of activating contexts and explanations

As discussed in the main text, the explanations found for a given latent can be very different depending on the way to sample the activating contexts shown to the explainer model, see fig. A1. This has its advantages and disadvantages.

When sampling from only the top activations, it is possible that the explainer model gives an explanation that is more narrow - “The concept of a buffer, referring to something that separates, shields, or protects one thing from another, often used in various contexts such as physical barriers, chemical reactions, or digital data processing” - instead of one that captures the full distribution - “Words or phrases associated with concepts of spatial or temporal separation (buffers, zones, or cushions) or colloquialisms (buff, buffs, or Buffy, referring to a popular TV show or enthusiast)”. Here the narrower explanations resulted in lower scores: the explanation generated from randomly sampled examples scored 0.95 accuracy in fuzzing and 0.93 accuracy in detection, while the explanation generated from top examples only achieves 0.8 accuracy in fuzzing and 0.74 accuracy in detection.

On the other hand, if we look at the second example, sampling from random examples may sometimes confuse the explainer model - “Tokens often precede or succeed prepositions, articles, and words that signal possession or quantity, frequently indicating a relationship between objects or actions. of feature” , which might not see the pattern that is more clear in the top activating examples -“Descriptions of food or events where food is involved, often mentioning leftovers, and sometimes mentioning the act of eating, serving, or storing food, as well as the amount of food, or the pleasure or satisfaction derived from it.”. Here we see very poor scores on the explanation from randomly sampled examples, 0.54 accuracy in detection and 0.57 accuracy in fuzzing, while the explanation generated from the top examples has 0.89 accuracy both in detection and fuzzing (over the full distribution).

Figure: Figure A1: Activating contexts of feature 209 (top) and 293 (bottom), from layers 8 and 32, respectively, of the 131k latent SAE trained on the residual stream of Gemma 2 9b. The shown examples are similar to the ones given to the explainer model to come up with explanations.
Refer to caption: extracted/5935489/example_1.png

### A.3 Different scoring methods

#### A.3.1 Detection details

The prompt used for detection scoring is the following:
{mdframed}

Together with the prompt, there are several few shot examples like the following:
{mdframed}

In this example the $<$user_prompt$>$ and the $<$assistant_response$>$ tags are substituted with the correct instruct format used by the scorer model. In detection scoring, 5 shuffled examples are shown to the model at the same time.

#### A.3.2 Fuzzing details

The prompt used for fuzzing scoring is the following:
{mdframed}

Followed by few-shot examples of the kind:
{mdframed}

In this example, the $<$user_prompt$>$ and the $<$assistant_response$>$ tags are substituted with the correct instruct format used by the scorer model. In fuzzing scoring, 5 shuffled examples are shown to the model at the same time.

#### A.3.3 Surprisal details

In surprisal scoring, the cross-entropy loss of the scorer model is computed over the tokens of an example. For each explanation, this loss is computed with the explanation and with a default explanation - “Various unrelated sentences,” where the examples can either be activating context or non-activating contexts. In our first approach, Llama 3.1 70b base was used. The prompt starts with few shot examples like:
{mdframed}

With the pairs of losses with the explanation and with the default explanation, it is possible to compute the decrease in loss caused by having access to the explanation. It is expected that in activating contexts this difference will be greater than in non-activating examples, and the score of the explanation is given by the AUC computed using this loss as a proxy for activating and non-activating labels.

#### A.3.4 Embedding details

For embedding scoring, we use a [small 400M](https://huggingface.co/dunzhang/stella_en_400M_v5) parameter model. We chose a small performant model on MTEB (Muennighoff et al., ) because we found similar scores when using a larger [7B parameter model](https://huggingface.co/nvidia/NV-Embed-v2), see fig A2, as this size allowed us to do increase the number of examples used in scoring.

A set of activating and randomly selected non-activating examples are embedded using the scorer model. Then a “query” instruction is embedded:
{mdframed}

The cosine-similarity between the instruction embedding and the examples is computed and is used as a proxy for activating and non-activating labels when computing the AUC, which is the score of that explanation.

Figure: Figure A2: Comparison between the scores given by a small embedding model and a larger one
Refer to caption: extracted/5935489/embedding.png

#### A.3.5 Adversarial examples

Most latents are active in less than 0.1% of the full dataset used to collect the activations, making random non-activating contexts very diverse. Randomly sampling non-activating examples cannot be used to determine whether a latent fires in a token in a specific context or on that token in general, as it is unlikely that that token randomly occurs in each non-activating example. As SAEs are scaled and latents become sparser and more specific, techniques that overly rely on activating contexts will have more imprecise results Gao et al. ().

Motivated by the phenomenon of feature splitting, we could use “similar” latents to test whether explanations are precise enough to distinguish between similar contexts. A potential approach is using cosine similarity between decoder directions of latents to find counterexamples for an explanation. Some works (Juang et al., ; Macgrath, ) have shown that a significant fraction of current latent explanations can’t be used to distinguish between features with high similarity.

#### A.3.6 Correlation between scores

We compute the correlation between these different scores and simulation 600 different latent explanations, of the 131k latent SAE trained on the residual stream of Gemma 2 9b spread across 4 different layers.

**Table A1: Pearson correlation computed over 600 different latent scores**
|  | Fuzzing | Detection | Simulation | Embedding | Surprisal |
| --- | --- | --- | --- | --- | --- |
| Fuzzing | 1 | 0.74 | 0.70 | 0.42 | 0.32 |
| Detection |  | 1 | 0.45 | 0.70 | 0.62 |
| Simulation |  |  | 1 | 0.34 | 0.19 |
| Embedding |  |  |  | 1 | 0.79 |
| Surprisal |  |  |  |  | 1 |

Figure: Figure A3: Scatter plots with different combinations of scores
Refer to caption: extracted/5935489/correlation.png

### A.4 SAE models used

Throughout this work, we used different SAEs trained on Gemma. The 16k latent ones trained on the MLP have the following average L0 norms per layer:
{mdframed}
0:50, 1:56,
2:33, 3:55,
4:66, 5:46,
6:46, 7:47,
8:55, 9:40,
10:49, 11:34,
12:42, 13:40,
14:41, 15:45,
16:37, 17:41,
18:36, 19:38,
20:41, 21:34,
22:34, 23:73,
24:32, 25:72,
26:57, 27:52,
28:50, 29:49,
30:51, 31:43,
32:44, 33:48,
34:47, 35:46,
36:47, 37:53,
38:45, 39:43,
40:37, 41:58
The 16k latent ones trained on the residual stream have the following average L0 norms:
{mdframed}
0:35, 1:69,
2:67, 3:37,
4:37, 5:37,
6:47, 7:46,
8:51, 9:51,
10:57, 11:32,
12:33, 13:34,
14:35, 15:34,
16:39, 17:38,
18:37, 19:35,
20: 36, 21:36,
22: 35, 23: 35,
24: 34, 25: 34,
26: 35, 27:36,
28: 37, 29:38,
30:37, 31:35,
32: 34, 33:34,
34:34, 35:34,
36:34, 37:34,
38:34, 39:34,
40:32, 41:52
The 131k latent ones trained in the residual steam have the following average L0 norms:
{mdframed}
0:30,
1:33, 2:36, 3:46,
4:51, 5:51,
6:66, 7:38,
8:41, 9:42,
10:47, 11:49,
12:52, 13:30,
14:56, 15:55,
16:35, 17:35,
18:34, 19:32,
20:34, 21:33,
22:32, 23:32,
24: 55, 25:54,
26:32, 27:33,
28: 32, 29:33,
30:32, 31:52,
32: 51, 33:51,
34:51, 35:51,
36: 51, 37:53,
38:53, 39:54,
40: 49, 41:45

We also used SAEs with 65k latents, trained on the residual stream and the MLP of Llama 8b.

### A.5 Factors that influence the explainability

#### A.5.1 Dependence on dataset

Even though a significant portion of SAE latents being less active when using RPJv2 instead of the Pile, we find that the latents that are active are generally interpretable to the same degree. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3.1b 70b instruct, quantized to 4bit.

**Table A2: Impact of training dataset on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| RPJ-v2 | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) | 0.67 (0.57–0.80) |
| Pile | 0.76 (0.67–0.86) | 0.76 (0.67–0.85) | 0.69 (0.57–0.80) |

#### A.5.2 Dependence on chain of thought and activation information

We find that COT slightly increases the scores of the explanations found, and that it significantly slows down the rate at which one can produce explanations. Giving the explainer model, the activations associated with each token seem to slightly increase the scores of the explanations generated. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3.1b 70b instruct, quantized to 4bit.

**Table A3: Impact of prompt content on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| Activations in prompt | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) | 0.68 (0.57–0.80) |
| No activations in prompt | 0.75 (0.65–0.86) | 0.73 (0.60–0.84) | 0.68 (0.58–0.79) |
| COT in prompt | 0.76 (0.67–0.86) | 0.73 (0.61–0.85) | 0.65 (0.55–0.75) |

#### A.5.3 Dependence on context length

The context length of the shown examples did not significantly change the scores obtained for the explanations. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model were Llama 3.1b 70b instruct, quantized to 4 bits.

**Table A4: Impact of context length on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| 16 context | 0.75 (0.65–0.86) | 0.74 (0.62–0.85) | 0.70 (0.59–0.81) |
| 32 context | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) | 0.67 (0.57–0.80) |
| 64 context | 0.74 (0.64–0.64) | 0.70 (0.57–0.81) | 0.65 (0.54–0.78) |

#### A.5.4 Dependence on the origin of examples

Sampling the examples from the whole distribution of activations, or sampling a fixed number of examples over different activation quantiles, significantly improved the scores, compared with sampling only from the top examples. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model were Llama 3.1b 70b instruct, quantized to 4 bits.

**Table A5: Impact of example sampling strategies on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| Randomly sampled | 0.76 (0.68–0.86) | 0.74 (0.62–0.84) | 0.66 (0.56–0.78) |
| Sampled from quantiles | 0.77 (0.69–0.87) | 0.74 (0.64–0.85) | 0.68 (0.57–0.80) |
| Sampled from top examples | 0.73 (0.64–0.83) | 0.72 (0.62–0.82) | 0.70 (0.58–0.80) |

#### A.5.5 Dependence on the number of examples

The number of examples shown to the model seems to saturate, at least with the explainer model we used. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3.1b 70b instruct, quantized to 4bit.

**Table A6: Impact of number of examples on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| Shown 10 examples | 0.73 (0.62–0.85) | 0.71 (0.58–0.82) | 0.64 (0.54-0.74) |
| Shown 20 examples | 0.74 (0.64–0.85) | 0.72 (0.60–0.84) | 0.66 (0.54-0.76) |
| Shown 40 examples | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) | 0.68 (0.57–0.80) |
| Shown 60 examples | 0.75 (0.66–0.85) | 0.73 (0.62–0.84) | 0.68 (0.57–0.79) |

#### A.5.6 Explainability across layers

We find that, in the case of the 131k latent SAE trained on the residual stream, the earliest layers have lower scores than the later layers. These evaluations were done using the 131k latent SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3.1b 70b instruct, quantized to 4bit.

Figure: Figure A4: Accuracy on fuzzing and detection scoring Dots correspond to the median score over c.a. 300 latent explanations, and colored region denotes the interquartile range.
Refer to caption: extracted/5935489/layer_accuracy.png

#### A.5.7 Dependence on size of explainer and scorer models

Having a larger explainer model leads to better explanations, but our results suggest that it tends to saturate. It is possible that that is due to the fact that the prompt and setup was optimized for Llama 3.1 70b, and that better explanation could be extracted using Claude Sonnet 3.5 if the prompt was better tuned for it. We find that having a smaller model score significantly drops the scores of the explanations—as bad as having the scorer model generate the explanations and the bigger model score them.

**Table A7: Comparison of explainer models for Fuzzing and Detection. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection | Embedding |
| --- | --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) | 0.51 (0.44–0.57) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) | – |
| Claude explaining | 0.75 (0.68–0.84) | 0.75 (0.65–0.85) | 0.70 (0.58–0.81) |
| Llama 70b explaining (4 bit) | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) | 0.67 (0.57–0.80) |
| Llama 70b explaining (4 bit) 8b scoring | 0.69 (0.61–0.77) | 0.69 (0.59–0.79) | – |
| Llama 8b explaining | 0.70 (0.60–0.81) | 0.70 (0.59–0.81) | 0.64 (0.54–0.75) |
| Llama 8b explaining, 8b scoring | 0.67 (0.59–0.75) | 0.67 (0.57–0.77) | – |
| Human explaining | 0.75 (0.66–0.85) | 0.74 (0.64–0.85) | 0.71 (0.62–0.81) |

#### A.5.8 Dependence on SAE size and location

We compare the interpretability of SAEs with different number of latents, trained on the same model, with the neurons of that model. We find that the SAE with the highest number of latents to have the highest scores in the case of the Gemma 2 9b SAEs and that SAEs trained on the residual stream have higher scores for both Gemma 2 9b and Llama 3.1 8b.

The scorer and the explainer model were Llama 3.1 70b instruct, quantized to 4bit.

**Table A8: Comparison of SAEs for Fuzzing and Detection. Numbers shown are the median score and the interquartile (25%-75%) range.**
| Experiment | Fuzzing | Detection |
| --- | --- | --- |
| Random explanation | 0.51 (0.45–0.57) | 0.51 (0.45–0.58) |
| Randomly initialized Topk SAE | 0.55 (0.50–0.60) | 0.54 (0.50–0.59) |
| 131k latents Gemma 2 9b | 0.76 (0.67–0.86) | 0.74 (0.63–0.85) |
| 16k latents Gemma 2 9b | 0.73 (0.63–0.83) | 0.70 (0.59–0.79) |
| Top 32 neurons Gemma 2 9b | 0.62 (0.54–0.70) | 0.59 (0.53–0.64) |
| Top 256 neurons Gemma 2 9b | 0.59 (0.53–0.65) | 0.57 (0.52–0.62) |
| 262k latents Llama 3.1 8b MLP | 0.79 (0.69–0.86) | 0.79 (0.64–0.85) |
| 262k latents Llama 3.1 8b | 0.81 (0.71–0.86) | 0.83 (0.68–0.85) |
| Top 32 neurons Llama 3.1 8b | 0.55 (0.51–0.62) | 0.53 (0.50–0.59) |
| Top 256 neurons Llama 3.1 8b | 0.54 (0.49–0.60) | 0.53 (0.50–0.57) |

### A.6 Alignment of latents using decoder weights

See figure A5 for a depiction of the alignment between weights of decoders at various layers. Here, the alignment is measured using the Frobenius inner product between the decoder weight matrices. Due to the permutation symmetry of SAEs layers are not aligned by default, but can’t be aligned using the Hungarian algorithm. We observe that the decoder weights of SAEs trained on the residual stream can be better aligned than those trained on the MLP.

Figure: Figure A5: Alignment between decoder directions of the 16k Gemma 2 9b SAEs Because each layer is trained independently, there is no correspondence between latents at different layers. It is possible to align them using the Hungarian algorithm, but even still the MLP SAEs are less aligned than the residual stream ones.
Refer to caption: extracted/5935489/hungarian_decoder.png

### A.7 Automatically interpreting interventions

#### A.7.1 Scoring implementation

For each feature, we sample a pool of 40 length-64 prompts from RPJv2 (Computer, ), of which 30 i.i.d. prompts are taken for scoring, while the remaining 10 are used by the explainer. The pool is sampled among nonzero activating contexts, stratified by the quintile of the feature’s max activation in that context (i.e., the nonzero activating documents are sorted by the context’s maximum activation of the current feature, then 8 contexts are sampled from the first quintile, 8 from the second etc.). Each of these prompts is then truncated to include only the first token on which the feature activates, so that all previous activations for the feature are 0. Activations on the first token position are ignored because the SAEs were not trained on that position.

We filter out features that activates on less than 200 of the 10 million RPJ-v2 tokens.

We then sample generations with a maximum of 8 new tokens from the subject model. For generations with the intervention, we perform an additive intervention on the feature at all token positions after and including the final prompt token. We tune the intervention strength of each feature to various KL-divergence values on the scoring set with a binary search. We stop the binary search when the KL divergence is within 10% of the desired value(^4^44Sometimes the KL-divergence is not perfectly monotonic in the intervention strength so 10% error is exceeded. We report the average KL divergence that we observe in figure A6). For our zero-ablation experiment, instead of doing an additive intervention we clamp the feature’s activation to 0. Specifically, the hidden states are encoded by the SAE, then the SAE reconstruction error is computed using a clean decoding, then the SAE encoding is clamped and decoded, and the error is added to the clamped decoding.

The scorer is Llama-3.1 8B base. We use the base model for improved calibration, and prompt it as follows.
{mdframed}

#### A.7.2 Generating explanations

We generate explanations using only the intervention’s effect on the subject’s next-token probabilities because this leads to a concise and precise prompt. The explainer, like the scorer, is Llama-3.1 8B.

The explainer sees a distribution of 10 prompts that is sampled i.i.d. from the same population as the scorer’s prompts.

We use the following prompt for the explainer, with 3 few-shot examples truncated for brevity.
{mdframed}

#### A.7.3 Baselines

Random SAE. We experiment with a random top-k SAE with $k=50$, roughly the average sparsity of the gemma SAEs. The encoder is initialized with 131,072 spherically uniform unit-norm features, and the decoder is initialized to its transpose. We use a random SAE with TopK activations because we need sparsity for our sampling procedure to work properly.

Random explanations.
For each layer and target KL value, we compute a random explanation baseline where we shuffle the explanations across features.

#### A.7.4 Intervention interpretability results

Figure A6 shows the average intervention score at each layer, for a few KL values. Intervention scores are higher in later layers, likely because of the increased proximity to the model’s output.

Figure A7 is similar to the right half of figure 4, but at multiple layers and KL values. Early layers have small intervention scores across the board, so there is little correlation, while the reason for little correlation in layer 41 is less clear.

Figure: Figure A6: Average intervention score vs layer. SAE features in later layers have more explainable effects on output. Random features, however, have uninterpretable effects on output, even at late layers.
Refer to caption: x4.png

Figure: Figure A7: Comparison of intervention and fuzzing scores across layers at various target KL values.
Refer to caption: x5.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: - Missing a highly relevant work, "Explaining black box text modules in natural language with language models". How does their scoring method compare to the ones proposed in this paper?
- Can the authors explain  the negative correlation between the fuzzing score and intervention score in figure 4? If they are both useful scoring methods, why would the correlation be negative?
- Unlike the claim in the paper, figure 5 is still showing statistical alignment across layers. Can the authors provide evidence for semantic similarity? (e.g., compute explanation similarity across layers instead of the matrix statistics)

*Minor issues*
- The authors say “we introduced *five* new techniques” in the abstract, and “We addressed issues with the conventional simulation-based scoring and introduced *four* new scoring techniques” in the conclusions. The readers might get confused in terms of the number of methods actually introduced in the paper, if it’s five, please be consistent throughout the paper.
soundness: 2
strengths: - Given the large sizes of SAEs nowadays and an increasing need for model explainability, automatically generating explanations of SAE latents efficiently is an important topic.
- The paper is well written, with ablations of design choices clearly described.
summary: This paper introduced five automated scoring methods to score the explanations of SAE latents, and discussed the shortcomings of existing scoring techniques. The paper also finds that SAE trained on nearby layers are highly similar, and provided actionable insights for practitioners to train wider SAEs instead of narrower SAEs to be efficient when there’s a compute constraint.
weaknesses: - Format: There are no line numbers, and it's showing "Under review as a conference paper at ICLR **2024**" instead of **2025** at the top.
- The low correlation between different evaluation methods in Table 1 is concerning. Since the simulation method proposed in prior work is vetted and established, the new ones proposed in this work should at least have strong rank correlation (> 0.7) with it to prove that they work. Since this the scoring methods are the primary contribution in this paper, the authors should conduct more rigorous tests to ensure their validity. I would also encourage the authors to conduct experiments with ground truth explanations (e.g., SAE latents with known explanations found in prior work, or easily constructed an embedding model responding to a known concept/explanation), to make a stronger case in terms of the reliability of these new methods.
To add on, instead of correlation of the raw scores, it might make more sense to look at the "rank" correlations of different methods.
- The authors claim that the new methods are more efficient than prior scoring methods without actually quantifying the efficiency gain to support the claim. If efficiency gain is one of the highlights of these scoring methods, the authors should consider comparing runtime of different methods to support the claim.
- The author mention “Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons, even when neurons are sparsified using top-k postprocessing.” in the abstract as one of the main findings, but the details cannot be found in the main paper but in the appendix. The authors should consider moving it to the main paper if this is one of the main claims.
- Reproducibility: The author mentioned a plan to open-source the project, but it's hard to evaluate the quality of their code for reproducibility purpose either since it's not provided as one of the supplementary files.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: The framework proposed in this paper is a valuable tool for the SAE community. It provides an automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features. However, the authors better consider to write a more accessible version for readers who are not familiar with SAEs, especially for section 3. It is a bit difficult to follow it.
soundness: 4
strengths: - The open-source framework is comprehensive and valuable for large-scale SAE analysis. The experiments are well designed to illustrate the effectiveness of the proposed method.
- The metrics proposed in this paper provides more dimensions to evaluate the generated explanations, which would be valuable for the SAE community.
- Some of the findings are meaningful to the SAE community. For example, larger latent SAE learn more dataset-specific latents. The relations between different sampling approaches and the generated explanations. And the high correlations between latents at adjacent layers.
summary: The paper builds an open-source automated pipeline to generate and evaluate natural language explanations for sparse autoencoder features with LLMs. The framework has been evaluated on various dimensions, including SAE size, activation function, loss, and LLMs. Five new scoring techniques are proposed. The paper finds that SAEs trained on nearby layers' the residual stream are highly similar. And they are also more interpretable than neurons.
weaknesses: - The method is a bit hard to understand for readers who are not familiar with SAEs. For example, how section 3.1 is related to 3.2? Would be more illustrative if a figure of the whole pipeline is provided.
- It would be more clear to provide a simple example of explaining the latents of SAEs. And even better if an example involves the whole workflow of this framework is provided.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: -	Why is the context length of 256 chosen for activation collection? Is this value based on empirical selection?
-	Why is the activating example limited to only 32 tokens, given that short contexts may hinder the correct identification of latents with complex activation patterns?
-	What accounts for the differences in explanations between “randomly sampling” and “uniformly sampling”?
-	In Section 5, the statement that “if the explanation for latent α at layer j is very different from the explanation for the same latent at layer j + 1, this would suggest that our pipeline is inconsistent and noisy” raises a hypothesis. Can you elaborate on it?
soundness: 2
strengths: -	This paper focuses on an important research problem of SAEs producing a large number of latent features that require automatic explanations and evaluations.
-	Several evaluation metrics have been proposed to assess the generated explanations, comparing them across different explainer models, SAEs, and layers.
-	Some findings are interesting. For instance, sampling examples that are shown to the explainer model may increase the scores of features, highlighting a problem with current auto-interpretability evaluations. Experimental results suggest a priority for training wider SAEs on a smaller subset of residual stream layers. These may provide valuable insights for future research.
summary: This paper investigates sparse autoencoders (SAEs), which project activation representations into a sparse high-dimensional latent space for interpretability. To automatically explain the large number of latent features, this paper proposes an LLM-based framework, explaining millions of latents across multiple models, layers, and SAE architectures. Four evaluation methods are proposed, including detection, fuzzing, surprisal, embedding, measuring the extent to which an explanation enables a scorer to discriminate between activating and non-activating contexts. Additionally, an intervention scoring is proposed to interpret a feature’s counterfactual impact on model output.
weaknesses: -	The writing and clarity of the paper could be improved. The current version is somewhat difficult to follow and understand. Please see my detailed questions below.
-	Many observations are presented without in-depth analysis or further exploration. For instance, in Section 3.1, the impact of context length and latent space size on activation data is mentioned, but it is unclear how these results relate to the selection of 256 tokens. In Section 3.2, the claim that “showing such short contexts to the explainer model hinders the correct identification of latents with complex activation patterns” seems to contradict the use of short activating examples with 32 tokens. In Section 3.3, while qualitative results for different sampling strategies are provided, it is not clear how to optimize the sampling process. Additionally, in Section 4.1, the statement “The imperfect correlations hint at either shortcomings of the scoring metrics or the fact that these metrics can measure different qualities of explanations” lacks clarity. What specific qualities of explanations are measured by these automatic metrics?
-	The generated explanations largely rely on the prompts and explainer models. It is unclear how generalizable the results are and to what extent specific prompts and explainer models may affect the quality of the generated explanations.
-	While several automatic evaluations are compared, it is unclear whether they accurately reflect the quality of explanations. It would be beneficial to conduct human evaluations, at least on a small set, and compare the correlation with automatic evaluations.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 1
questions: 1. How is the correlation of the proposed evaluation metric with the original metric?
2. How is the efficiency of the proposed evaluation metric compared to the original metric?
3. Previous work (https://transformer-circuits.pub/2023/monosemantic-features) has already shown SAEs have more interepretable features than neurons. What's the purpose of validating this result?
4. Why comparing SAEs in adjacent layers? (Why is it interesing?)
5. How is comparing SAEs in adjacent layers related to your evaluation metric?
6. Are there any prior work that supports your method in evaluating the SAEs in adjacent layers?
soundness: 2
strengths: - The scale gets improved from previous sota.
- This work lies in an interesting direction.
summary: The paper presents an open-source automated pipeline that uses large language models to generate natural language explanations for millions of features in Sparse Autoencoders (SAEs), addressing the challenge of manual interpretation. It introduces five efficient scoring techniques, including intervention scoring, and demonstrates that SAEs are more interpretable than neurons, offering insights into semantic similarity across model layers.
weaknesses: - This work did not compare the new evaluation metric with the previous evaluation metric (https://openai.com/index/language-models-can-explain-neurons-in-language-models/) in a solid form. Having similar conclusions as previous approach should not serve as a solid evidence that the new metric is as good as the previous evaluation metric.
- Comparing SAEs in adjacent layers (1) lacks support of motivation and (2) is not well supported. See Questions for details.
- The presentation is poor: readers can not capture the main contribution of this paper with a normal reading flow. The contribution of this work seems to be concentrated on a new evaluation metric, but Section 5 cuts in to discuss about behaviors of the behaviors of SAEs. I would strongly recommend to reorganize the paper to one single claim, with evidence from both sides supporting it.

Meta-review: ## Summary of Scientific Claims and Findings  
The paper introduces a novel automated framework using large language models (LLMs) to generate and evaluate natural language explanations for Sparse Autoencoder (SAE) features. It proposes five new scoring methods—detection, fuzzing, surprisal, embedding, and intervention scoring—to assess the interpretability of these features. The paper also provides insights into the high similarity of SAEs trained on nearby layers and suggests that wider SAEs are more efficient for certain tasks under computational constraints. The authors claim their methods are more efficient and offer actionable insights for practitioners in model explainability.  
   
## Strengths  
1. **Relevance and Importance**: The paper addresses a timely issue of model interpretability in the context of large SAEs, which is crucial given the growing complexity of modern neural networks.  
2. **Comprehensive Framework**: The work presents a comprehensive open-source pipeline that could significantly aid researchers in analyzing SAEs.  
3. **Novel Scoring Techniques**: The introduction of multiple scoring techniques offers a broader set of tools for evaluating the quality of explanations, contributing valuable methodologies to the community.  
4. **Well-Designed Experiments**: The experiments are well-structured to validate the effectiveness of the proposed methods, providing meaningful insights into SAE behavior.  
   
## Weaknesses  
1. **Clarity and Presentation**: Several reviewers noted that the paper's presentation could be improved. The organization sometimes makes it difficult to follow the main contributions and arguments.  
2. **Validation and Comparison**: The paper lacks a solid comparison with previous evaluation metrics. The correlation between the new and established methods is not robust, which raises questions about the validity and reliability of the new metrics.  
3. **Depth of Analysis**: Some findings are presented without extensive analysis, limiting the depth of insights offered by the paper.  

 After thorough consideration of other submissions in the same batch, I find myself recommending rejection for this paper. I encourage the authors to refine their work and consider resubmission to a future conference or journal.

Decision: Reject