Automatically Interpreting Millions of Features in Large Language Models
Authors: Gonçalo Santos Paulo
Alex Troy Mallen
Caden Juang
Nora Belrose
Venue: ICML 2025 poster (2025)
Keywords: interpretability
language model
sae
features
explanation
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We build an open-source automated pipeline to generate and evaluate natural language explanations for SAE features using LLMs

Abstract: While the activations of neurons in deep neural networks usually do not have a simple human-understandable interpretation, sparse autoencoders (SAEs) can be used to transform these activations into a higher-dimensional latent space which can be more easily interpretable. However, SAEs can have millions of distinct latents, making it infeasible for humans to manually interpret each one. In this work, we build an open-source automated pipeline to generate and evaluate natural language interpretations for SAE latents using LLMs. We test our framework on SAEs of varying sizes, activation functions, and losses, trained on two different open-weight LLMs. We introduce five new techniques to score the quality of interpretations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on a latent, which we find explains latents that are not recalled by existing methods. We propose guidelines for generating better interpretations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. Our code is available at https://github.com/EleutherAI/delphi.

Automatically Interpreting Millions of Features in Large Language Models
Gonçalo Paulo' Alex Mallen
Caden Juang'
Nora Belrose'
Abstract
et al., 2023; 2024). Due to the large number of neurons to be interpreted, automated approaches were proposed (Bills et al., 2023), where second LLM is used to propose an interpretation given a set of neuron activations and the text snippets it activates on, in a process similar to that of gen- crating a human label. But research has found that most neurons are "polysemantic", activating in contexts that can be significantly different (Arora al., 2018; Elhage et al., 2022). The Linear Representation Hypothesis (Park et al., 2023) posits that human-interpretable concepts are encoded in linear combinations of neurons. A significant branch of current interpretability work focuses on extracting these features and disentangling them (Bereska & Gavves, 2024).
While the activations of neurons in deep neu- ral networks usually do not have a simple cumman-undersandabble interpretation, sparse au- toencoders (SAEs) can be used to transform these activations into higher- nigher-dimmeniionll latent space which can be more easily interpretable. However, SAEs can have millions of distinct features, mak- ing it infeasible for humans to manually interpret each onc. In this work, we build an open-source automated pipeline to generate and evaluate nat- ural language interpretations for SAE features using LLMSs We test our framework SAFS of varying sizes, activation functions, and losses, trained on two different open- weieht LLMs. We introduce five new techniques to score the quality of interpretations that are cheaper to run than the previous state of the art. One of these techniques, intervention scoring, evaluates the interpretability of the effects of intervening on feature, which we find explains features that are not recalled by existing methods. We propose guidelines for gen- crating better interpretations that remain valid for a broader set of activating contexts, and discuss pitfalls with existing scoring techniques. Our code is available online.
Sparse autoencoders (SAEs) were proposed as way to address polysemanticity (Cunningham et al., 2023). SAEs consist of two parts: an encoder that transforms activation vectors into sparse, higher-dimensional latent space, and decoder that projects the features back into the original space. Both parts are trained jointly to minimize reconstruc- tion error. SAE features' ! were found to be interpretable and potentially more monosemantic than neurons (Bricken et al., 2023; Cunningham et al., 2023). Recently, a signifi- cant effort was made to scale SAE training to larger models, like GPT-4 (Gao et al., 2024) and Claude 3 Sonnet (Tem- pleton et al., 2024), and they have become an important interpretability tool for LLMs.
Training an SAE yields numerous sparse features, each of which needs a natural language interpretation. In this work, we introduce an automated framework that uses LLMs to generate interpretation for each feature in an SAF. We use this framework to explain millions of features across multiple models, layers, and SAE architectures. We also propose new ways to evaluate the quality of interpretations, and discuss the problems with existing approaches. We hope that these result can inform other repositories of inter- pretations for SAE features that already exist (Lin & Bloom, 2023), as well as improve comparisons between different SAE architectures.
1. Introduction
Large language models (LLMs) have reached human level performance in broad range of domains (OpenAI, 2023), and can even be leveraged to develop agents (Wang et al., 2023) that can strategize (Bakhtin et al., 2022), cooperate and develop new ideas (Lu et al., 2024; Shaham et al., 2024). At the same time, we understand little about the internal representations driving their behavior. Early mechanistic interpretability research focused on analyzing the activation patterns of individual neurons (Olah et al., 2020; Gurnee
'We use feature to mean the intermediate dimension of the SAE like in (Bricken et al., 2023).
'EleutherAI NNrthwester University, Evanston, Illinois, USA. Correspondence to: Gonçalo Paulo <goncaloollcuthhrraai eleuther.ai>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Automatically Interpreting Millions of Features in Large Language Models
2. Related Work
3.1. Collecting activations
One of the most successful approaches to automated inter- pretability focused on explaining neurons of GPT-2 using GPT-4 (Bills et al., 2023). GPT-4 was shown examples of contexts where a given neuron was active and was tasked to provide a short interpretation that could capture the activa- tion patterns. To evaluate if a given interpretation captured the behavior of the neuron, GPT-4 was tasked to predict the activations of the neuron in a given context having ac- cess to that interpretation. The interpretation then scored by how much the simulated activations correlate with the true activations. A similar approach was used in Templeton et al. (2024) to explain SAE features of Claude 3 Sonnet. In general, current approaches focus on collecting contexts together with feature activations from the model to be ex- plained, and use a larger model to find patterns in activating contexts.
Sparse autoencoders (SAEs) are feedforward networks with one hidden layer trained to reproduce their input using a sparse number of neurons. This means that while there are several times more neurons than there are dimensions of the input, only a small fraction of these neurons- less than 60 in this work- are non-zero and contribute to the output. In our pipeline, we seek to explain the non-zero activations of the SAE hidden layer neurons, which we call "features."
We collected feature activations from the SAES over a 10M token sample of RedPajama-v2 (RPJv2; Computer 2023), This is the same dataset that we used to train the Llama 3.1 8b SAFS, and consists of a data mix similar to the Llama I pretraining corpus.
We collected batches of 256 tokens starting with the be- ginning of sentence token (BOS). The contexts used for activation collection are smaller than the contexts used to train the SAEs, and we find that, on average, 30% of the features, per layer, of the 13lk feature Gemma 2 9b SAE don't 't activate more than 200 times over these 10M tokens and 15% don't activate at all. When we consider the train- ing context length of 1 1024, only 5% of features don't fire. When using a closer proxy of the training data, the "un- copyrighted" Pile, we find that the number of features that activate fewer than 200 times decreases to 15%, and that only 1% don't activate at all, even when considering con- texts of size 256. Interestingly. in the case of the 16k feature Gemma 2 9b SAE only around 10% of features activate fewer than 200 times on the RPJv2 sample, suggesting that the larger 13lk feature SAE learned more dataset specific features than its smaller cousin.
Following those works, other methods of evaluating inter- pretations have been proposed, including asking model to generate activating examples and measuring how much "module" activates (Singh et al., 2023; Kopf et al., 2024). More recently, "interpretability agents" have been built, managing to do iterative experiment to find the best interpre- tations of vision neurons (Shaham et al., 2024). Potentially cheaper versions of automated interpretability have been suggested, where the model that is being explained doubles as an interpretation generation model (Kharlapenko et al., 2024). A prompt querying the meaning of a single place- holder token is passed to the model and feature activations are patched into its residual stream at the position of the placeholder token during execution, generating continua- tions related to the feature. This technique is inspired by earlier work on Patchscopes (Ghandeharioun et al., 2024) and SelfTE (Chen et al., 2024).
3.2. Generating interpretations
Our approach generating interpretations follows Bricken et al. (2023) in showing the explainer model examples sampled from different quantiles, but uses more "natu- ral" prompt where the activating example is shown whole, with the activating tokens emphasized and their activation strength shown after the example, see Appendix B for the full prompt, similar to what is done in (Lee et al., 2023). For examples of some interpretations and their scores, see Figure Al.
3. Automated interpretability pipeline
In this Section we explain, step-by-step, the main pipeline used to generate interpretations and score them. The pipeline can be broadly divided into 3 sequential steps. First, the activations of the SAEs to be interpreted are collected over broad dataset. Then, for each feature, relevant con- texts are selected and shown to an LLM which generates an interpretation for the activation pattern observed. Finally, these interpretations are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non-activating contexts, details explained below. As an il- lustration, we represent how that pipeline might look like for a real feature, see Figure 1.
We show the explainer model, Llama 3.1 70b Instruct, 40 different activating examples, each activating example con- sisting of 32 tokens, where the activating tokens can be on any position. While most features are well-explained by contexts of length 32, we expect features active over longer contexts to not be well captured by our pipeline. Using
We throw out the activations on the BOS token when gen- erating interpretations for Gemma, as we were told in personal communication these SAEs were not trained on BOS activations.
2

---

Automatically Interpreting Millions of Features in Large Language Models
Collecting activating contexts
Generating interpretations
Scoring interpretations
(..) will return to about its (..)
Provide an explanation for these
Given this explanation, what contexts is this latent active on?
(..) Is that 35% excluding stock (..)
contexts:
will contractively return (..))
(..)) will return to about its (...)
"Prepositions and verbs denoting directional movement or change, often indicating return, reversal, or shift in position, state, context."
and a return responsibility (...)
(...) will contractively return (..)
kindergarten, first, second (...)
and return of responsibility (...)
we want 'play' be (..))
want to quickly circle back (..)
want to quickly circle back (..))
were forced retreat (...)
frozen pizzas, snacks, (..)
(..) We needed go back-to (...)
were forced to retreat (...)
(.)) The term "regression" was (..)
game species were driven (...)
"Prepositions and verbs denoting directional movement
baritone Christopher Dylan (..)
(..) except for students entering (..)
L2
L3
SAE
L4
or change, often indicating a return, reversal, shift in position, state, context." "
Model to be explained
[1,1,1,0,0]
Figure 1. Auto-interpretabbiliy pipeline. The first step of the interpretability pipeline is collecting the activations of the target SAE over a broad range text. this figure we show short contexts for legibility, and over represent the active feature, which in reality would be active in very small portion of the text. The activating contexts are selected and shown to an explainer LLM, prompt in Appendix B, which provides short interpretation. This interpretation then given to scorer LLM that tasked use this interpretation distinguish activating from non activating contexts, see Section 3.3
longer contexts to generate explanations would require ei- ther showing the model less examples, using much more powerful (and expensive) explainer model. Because we are able to evaluate interpretations significantly cheaper, bad interpretation can easily be filtered out, and we leave the explanation of "long-range" features for future work. Using this interpretation setup would cost $200 to interpret 1 million features.
3.2.1. AUTOMATICALLY INTERPRETING
INTERVENTIONS
Automatic interpretability methods, including most of those explored in this paper, typically look for correlations be- tween activation of feature and some natural-language property of the input. However, some features are more closely related to what the model will output. For example, we find a feature whose activation causes the model to output words associated with reputation but does not have a simple interpretation in terms of inputs.
We find that randomly sampling from broader set of ex- amples leads to interpretations that cover a more diverse set of activating examples, sometimes to the detriment of the top activating examples, see Figure 2. Using only top activating examples often yields more concise and specific interpretations that accurately describe these examples, but which fail to capture the whole distribution. On the other hand, stratified sampling from the deciles of the activation distribution can lead to interpretations that are too broad and that fail to capture a meaningful interpretation. For examples of these types of failure modes, see Figure A2 and the discussion the Appendix C.
We define an output feature as feature that causes prop- erty of the model's output that can be easily explained in natural language. See Section 3.3.5 for the definition we use in scoring.
Output features can also be described in terms of their cor- relation with inputs. For example, the "reputation" feature activates in contexts where likely next tokens relate to rep- utation. However, explaining output features in terms of causal influence on output has two advantages.
1. Scalability. Output features are easier to describe in
feature 157 of Gemma 9b's layer 32 autoencoder with 131k features and an average norm of Its detection score is 0.6.
3

---

Automatically Interpreting Millions of Features in Large Language Models
terms of effect on output because the explainer only needs to notice simple pattern of output. The pattern of inputs this feature correlates with is more complex. Explaining a feature by correlating it with inputs re- quires approximating the computation of the subject model leading up to that feature, which may be chal- lenging when subject models are highly capable and performing difficult tasks. Some features' influence on output, however, might remain easily explainable.
Given the highly sparse nature of SAE features, most of their variance could be captured by binary predictor that predicts the mean nonzero activation when the feature is expected to be active, and zero otherwise. In statistics, zero-inflated phenomena are often modeled as a mixture distribution with two components: a Dirac delta on zero, and another distribution (e.g. Poisson) for nonzero values. This goes against the general wisdom of simulation scoring, which focuses only activating examples, even though they are less than 0.01 % of the relevant contexts for each feature of an SAE and that being able to distinguish non- activating contexts from activating contexts seems to be relatively more important.
2. Causal evidence. We might like to know that a feature causes a property of the model's output so that we can steer the model. Previous work has shown that existing cuttoiineerpretabiiity scores fail to accurately capture how well given interpretation can predict the effect of intervening on a given neuron (Huang et al., 2023). Further, prior work has argued that causal evidence is more robust to distribution shifts (Bühlmann, 2018; Schölkopf et al., 2012).
As an alternative to simulation scoring, we introduce four new evaluation methods that focus on how well an interpre- tation enables scorer discriminate between activating and non-activating contexts. As an added benefit, all of these methods are more compute-efficient than simulation, see Table 1. We also propose a scoring method that eval- uates the interpretability of interventions using a specific feature.
3.3. Scoring interpretations
Not only should the generation of interpretations efficient, but so should the scoring of such interpretations. Practition- ers would like to know how faithful each interpretation is to the actual behavior of the network. Some features may not be amenable to a simple interpretation, and in these cases we expect the auto-interpeetability pipeline to output an interpretation with poor evaluation metrics. Secondly, we can use evaluations as a feedback signal for the train- ing of explainer models and tuning hyperparameters in the pipeline. Finally, some SAEs may simply be poor quality overall, and the aggregate evaluation metrics of the SAE's interpretations can be used to detect this.
3. DETECTION
One simple approach for scoring interpretations is to ask language model to identify whether whole sequence activates a SAE feature given an interpretation. Detection requires few output tokens for each example used in scor- ing. meaning examples from a wider distribution of feature activation strengths can used at approximately the same expense. By including non-activating contexts, this method measures both the precision and recall of the interpretation.
Detection is more "forgiving" than simulation insofar the scorer does not need to localize the feature to particular token. This means that detection evaluates whether an in- terpretation correctly identifies the types of contexts that a feature is active on, even if the interpretation does not correctly identify the actual token.
The quality of SAE interpretations has so far been measured via simulation scoring, a technique introduced by Bills et al. (2023) for evaluating interpretations of neurons. It involves asking an explainer model to "simulate" a feature in a set of contexts, which means predicting how strongly the feature should activate at each token in a context given an inter- pretation. The Pearson correlation between the simulated and true activations is the simulation score. The standard in the literature is to sample contexts from a "'op-and-randon" distribution that mixes maximally activating contexts and contexts sampled uniformly at random from the activating corpus. While oversampling the top activating contexts introduces bias, it is used as a cheap variance reduction technique, given that simulation scoring over hundreds of examples per feature would be expensive (Cunningham et al., 2023).
Both this method and fuzzing (described next) can leverage token probabilities to estimate how certain the scorer model is of their classification, and this is an effect that we believe can be to improve the scoring methods. Details on the prompt in Appendix D.I.
3.3.2. FUZZING
Fuzzing is similar to detection, but at the level of individual tokens. Here, potentially activating tokens are delimited in each example and the language model is prompted to iden- tify which of the sentences are correctly marked. Fuzzing is the most similar with simulation. Because SAE activa- tions are very sparse, simulation scoring mostly boils down to correctly identifying which of the tokens have non-zero
In this work, we take a slightly different view of what makes for a good interpretation: an interpretation should serve as a binary classifier distinguishing activating from non- activating contexts. The reasoning behind this is simple.
4

---

Automatically Interpreting Millions of Features in Large Language Models
activations. For this reason, fuzzing is the score that most correlates with simulation.
the embedding model and the way to compute the score in Appendix D.4.
When interpretations focus on which tokens the feature acti- vates, but not on which contexts, they can still score high on fuzzing, but comparatively lower on detection, specially if they are common tokens. Thus, evaluating an interpretation on both detection and fuzzing can identify whether a model classifying examples for the correct reason. Details on the prompt on Appendix D.2.
3.3.5. INTERVENTION SCORING
Unlike the above four context-based scores, intervention scoring interprets a feature's 's counterfactual impact on model output. We quantify the interpretability of an intervention I with interpretation z on a distribution of prompts и as the average decrease in the scorer's surprisal about the in- terpretation when conditioned on text generated with the intervention.
3.3.3. SURPRISAL
Surprisal scoring is based on the idea that a good interpreta- tion should help a base language model M (the "scorer") achieve lower cross-entropy loss on activating contexts than it would without a relevant interpretation. Specifi- cally, for each context x, activating non-activating, we measure the information value of an interpretation as (x)z)( log (x where 2 is fixed pseudo- interpretation. A good interpretation should have higher information value on activating examples than on non- activating ones.
S = Ex (e - Eg [log P. (z/
(xx) is the distribution over subject model generations given prompt x at temperature In (x)x), the intervention is applied to the subject model as generates. In practice we estimate the quantity S by sampling one clean and one intervened generation for each sampled of prompt.
Sufficiently strong interventions can be trivially inter- pretable by causing the model to deterministically output some logit distribution. Therefore, interpretability scores of interventions should be compared for interventions of fixed strength. We define the strength a of an arbitrary intervention I as the average KL-divergence of the model's ss intervened logit distribution with reference to the model's clean output.
The overall surprisal score of the interpretation is given by the AUROC of its information value when viewed classifier distinguishing activating from non-activating contexts. Details on the prompt and how to compute the score in Appendix D.3.
(1: r) = Ex-= [D (Psubjeca (-1x) || Psubject, / (-1x))
Surprisal relies on the ability of the scorer model to tune its predictions based on the explanation, measuring both the relevance of the context given in the interpretation and the token specificity, but we believe that the current setup could be improved, as this is the score with the least correlation with the others.
See Appendix for details on the interpretation and scoring pipeline we use in our intervention experiments.
4. Results
4.1. Comparing scoring methods
3.3.4. EMBEDDING
When wanting to evaluate the explanation over a larger num- ber of examples, simpler methods than simulation scoring have to be used (Templeton et al., 2024). Simulation scoring is traditionally done only in activating examples, ignoring whether the proposed explanation correctly handles non- activating examples. Simulation scoring is normally done in a small amount of examples per feature due to its high cost, and this is a sigmificant disadvantage that prompted us to investigate more efficient scoring methods.
Classifying between active and non-active contexts given a certain interpretation can also seen as using interpre- tations of features as "queries" that should be able re- trieve relevant "documents", contexts where the feature is active, between non-relevant "documents", non-activating contexts. This way. we take selection of activating and non non-activating contexts that embedded by an encoding trans- former, and the similarity between the query and the docu- ments is used as a classifier to distinguish between activating and activaaii contexts, and the score is the AUROC.
In Table we compare the amount of tokens used to score a single feature using 100 different contexts, for both fuzzing, detection and two different simulation methods. We can see that fuzzing and detection are at least 5x cheaper than simulation when using the all-at-once (AAO) method de- scribed in (Bills et al., 2023), although this method requires access to the log- proobbbbabslitss of the prompt tokens, which are not accessible from providers of state-of-the-ee closed- source models. A researcher wanting to generate simulation
If the encoding model is small enough we used 400M parameter model- - this technique is the fastest and opens up the possibility to evaluate a larger fraction of the activation distribution. We have seen that using a larger embedding model- - 7B parameter model- - didn't 't significantly improve the scores, see fig A3, although we believe that this ap- proach was under-investigated. Details on the prompt, on
5

---

Automatically Interpreting Millions of Features in Large Language Models
Table 1. Estimated cost of the different scoring methods. Brackets represent tokens that be cached, potentially saving costs, although we do not do these calculations here. The number of input and output tokens is computed considering 100 examples show, and using fewer examples would cost proportionally less.
Cost ($/100k features)
Cost (S/100k features) Claude Sonnet 3.5
Input tokens
Output tokens
Llama 70b
Fuzzing
19.6k (14.2k)
249
676
6.2k
Detection
17.0k (11.9k)
240
588
5.5k
Simulation (AAO)
104.9k (87.5k)
5
3.6k
31.5k
Simulation (TBT)
496.9k (451k)
46.7k
18.7k
219.1k
scores on the fly, without access to a local model, would have to use the token-by-token (TBT) method, which is 30x more than expensive than both fuzzing and detection, For the same set of intepretations, fuzzing and detection scores given by Llama 70b and Claude Sonnet 3.5 have similar dis- tributions. On the other hand, Claude Sonnet 3.5 produces to higher simulation scores on average than Llama 70b, see Table A9 meaning that we should probably compare the price of fuzzing and detection on Llama 70b to the price of simulation using Claude Sonnet 3.5.
text", "3 Incooppeee explanation but present in the text", "4 Explanation describes the activation of the tokens". We measure the average score of a feature in at least 5 con- texts. In total 700 contexts and 81 feature interpretation. We find that fuzzing as the best Spearman correlation with human scores (0.69), followed by simulation (0.60), and detection (0.59). Surprisal and embedding have 0.34 and 0.32 Spearman correlation respectively.
Due to their relative cheapness compared with simulation scoring, we propose that fuzzing and detection scoring can be used to estimate whether explanations correctly identify which tokens feature is most likely to be active, and in which types of contexts this happens. Cheaper methods like embedding can be used to quickly iterate on explanations or to broadly separate features that have bad explanations from those with good explanations, so that they can be refined.
Embedding scoring is even cheaper, requiring 4000 input tokens for 100 different contexts, which at average of S0.13 per million tokens would only cost $50/100k features.
We propose that feature interpretations should be evaluated using more than a single technique when possible, because as discussed in Section 3.3, each of the proposed methods have different failure modes, and a more rich scoring frame- find
4. 1.1.1. INTERVENTION SCORING
work can address some of these flaws. Specifically, that when computing the correlation of scores computed the same 800 explanations, we find that while there is clear positive correlation, some scores disagree see Appendix D.6 and Tables A1 and A2.
The previously discussed scoring methods are correlational in the sense that they measure whether an interpretation can be used to predict the activation values of given feature, or distinguish between activating and non-activating examples. Intervention scoring proposes to measure how well a given interpretation can predict the effect of interventions on the corresponding feature. Here we compare correlational in- terpretations generated with our pipeline and scored with fuzzing, to set of interventional interpretations scored with intervention scoring. Our hypothesis is that some features will have low correlational scores because their behavior is better explained by their downstream effects than by the contexts where they are active.
We find fuzzing to be the most correlated with simulation, as fuzzing is mostly measuring how well the explanation can help the model predict which tokens have non-zero acti- vation. Due to the sparsity of SAE activations, simulation scoring is doing a very similar test. On the contrary, we find that simulation has a significantly lower correlation with detection, embedding and surprisal scoring, as they more ac- curately measure whether a context activates a given feature, instead of which token is active in a given context. When comparing the simulation scores provided by Claude 3.5 Sonnet, all correlations have slight increase, see Appendix D.6.
In Figure we see that there is a slight negative correlation between the fuzzing score and the intervention score, show- ing that on average, features with low fuzzing score be better explained using their downstream effects than features with high fuzzing score. We also find that the distribution of intervention scores of interpretation on a trained SAE is significantly different from the distribution of an inter- pretation of randomly initialized SAE or a real SAE with randomly assigned interpretations, supporting the validity
We also measure the correlation between our scoring meth- ods and scores given by humans. Humans were tasked to score how an explanation related to a given context, similar to that done in (Templeton et al., 2024). The human evalua- tors could choose between 4 options, "1- Explanation not related to the text", "2 Explanation vaguely related to the
6

---

Automatically Interpreting Millions of Features in Large Language Models
Fuzzing
Detection
Sampling
Sampling
1.2
Unifor sampling
Uniform sampling
Stratified sampling
Stratified sampling
Top examples
1.0
Top examples
Random explanations
explanations
Density
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy
Accuracy
1.0
1.0
Sampling Uniform sampling
Stratified sampling
0.8
examples
0.8
Random explanstions
Accuracy
0.4
0.4
Sampling
Uniform sampling
Stratified sampling
Top examples
explanations
Non-activating
0.0
Activation Deciles
Max activating 0.0
Non-activating
Activation Deciles
Max activating
Figure 2. Fuzzing and detection scores for different sampling techniques. Panels and b) show the distributions of fuzzing and detection scores, respectively, a function different example sampling methods interpretation generation. Sampling only from the top activation gets average lower accuracy in fuzzing and on detection when compared with uniform sampling and stratified sampling. The distributions from random sampling and sampling from quantiles are very similar. Panels e) and d) measure how the interpretations generalize across activation quantiles, showing that interpretations generated from the top quantiles are better distinguishing non-activating examples, but have lower accuracy on other quantiles, especially on the lower activating deciles. This also happens for the other interpretations, but the accuracy does not drop much in lower activating deciles. We also show the scores of random explanations.
of this scoring method.
scores reflect the extent to which the proposed interpretation is valid over both activating and non-activating examples.
4.2. Comparing interpretation methods
We find that the interpretations generated by considering only the activating contexts are significantly different from those generated by selecting activating contexts from the whole distribution. Interpretations based top examples have higher specificity, but lower sensitivity. In fact, we observe that the sensitivity depends on how the degree of activation of each example shown, and that this sensitivity decreases faster for interpretation based on top examples. Interpretations generated from top activating examples are more concise and specific interpretations but fail to cap-
We use set of 500+ features testbed to measure the effects of design choices and hyperparameters on interpreta- tion quality. Each feature is scored using 100 activating and 100 non-activating examples. The activating examples are chosen via stratified sampling such that there are always 10 examples from each of the 10 deciles of the activation dis- tribution. We evaluate interpretation quality using fuzzing. detection and embedding scores, those were both quick to compute and easy to interpret. We expect this mix of

---

Automatically Interpreting Millions of Features in Large Language Models
layer 32
layer 32
target KL: 0.1
target KL: 3.0
10a
2.79 (r--0.37)
10
intervention score (nats)
target 1.0
10a
2-
10
target KL: 3.0
density
10
Random TopK SAE
Random Explanations
10-1
0.5
0.6
0.7
0.8
0.9
intervention score (nats)
fuzzing score (context-based)
Figure Intervention scores. Here we present intervention scores (Sec 3.3.5) SAE features in Gemma 9B at layer Left: SAE features are more interpretable than random features, especially when intervening more strongly. Our explainer also produces interpretations that are scored higher than random interpretations. Right: Many features that would normally uninterpreted when using context-based automatic interpretability are interpretable in terms of their effects on output.
ture the whole distribution, while explanations based examples sampled from the whole distribution lead to inter- pretations that are too broad, see Appendix C. This effect would not be seen if the scoring were done on just the most activating examples, underscoring a problem with current auto-interpretabbiiiy evaluations, which produce interpreta- tions using top activating examples and evaluate them on small subset of the activation distribution.
experiments.
We find that SAEs with more features have higher scores, and scores that are significantly higher than those of neurons. Neurons are more interpretable if made sparser by only con- sidering the top most activated neurons on given token, but still significantly underperform SAEs in our tests (Ta- ble A10). The location of the SAE matters; residual stream SAFs have slightly better scores than ones trained MIP outputs (Table A10). We also observe that carlier layers have lower overall scores, but that the scores across model depth remain constant after the first few layers (Fig A5).
Increasing the size of the explainer model increases the scores of the interpretations, but we don't 't find the explana- tions generated by Claude Sonnet 3.5 to have much higher scores than those generated by Llama 3.1 70b, see A9, and both are similar to those generated by human. Not surpris- ingly, we also see that even for simpler scores like fuzzing and detection, using smaller scorer model leads lower scores. It is possible the explanations generated by Claude would be better than those generated by Llama 3.1 70b had we optimized the prompting techniques for that model.
5. Conclusion
Explaining the features of SAEs trained cutting-edge LLMs is computationally demanding task, requiring scal- able methods to both generate interpretations and assess their quality. We addressed issues with the conventional simulation-based scoring and introduced five new scoring techniques, each with distinct strengths and limitations. These methods allowed us to explore the "prompt design space" for generating effective interpretations and propose practical guidelines.
Showing the explainer model a larger number of examples leads to slightly higher scores (Table A7). We find doesn't matter much whether we use the same dataset used to train the SAES (Table A3) whether slightly change the size of the contexts shown (Table A5). Using COT, at least when generating interpretations using Llama 70b, does not significantly increase their quality (Table A4) while signifi- cantly increasing the compute and time required to generate them. For this reason, we have not used it for our main
Additionally, although current scoring methods do not ac- count for interpretation length, we believe shorter interpreta- tions are generally more useful and will incorporate this in future metrics. Some scoring methods also require further
8

---

Automatically Interpreting Millions of Features in Large Language Models
refinement, particularly in selecting non-activating exam- ples improve evaluation. We also investigated method to generate and score interpretations of features based on their downstream effect and shown that low score "correlational" scores could be due to the existence of "output" features.
M. Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., Rowe, D., Shi, W., Spisak, J., Wei, A., Wu, D. J., Zhang, H., and Zijlstra, M. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378:1061 - 1074, 2022. URL
weapp.ssematiiiic. semanticscooll.. arg//orraaaD:225555906
Access to better, automatically generated interpretations could play a crucial role in areas like model steering, concept localization, and editing. We hope that our efficient scoring techniques will enable feedback loops to further enhance the quality of interpretations.
Bereska, L. and Gavves, E. Mechanistic interpretability for ai safery-a review. arXiv preprint urXiv:2404.1006 2024.
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, L.,
Data and Code Availability
Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons in language models. URL https:/oopeniipublcc blob. core. windows. welheeroneexpluiehhgpprrnuuuee humel accessed: 14.05. 2023). 2. 2023.
Together with this manuscript, we release the delphi library and accompanying scripts to guide the reproduction of these
results.
In collaboration with Neuronpedia we also release the ex- planations for the Gemma 2 9b 16k residual stream SAEs, the 16k MLP SAEs, the 131k residual stream SAES and the Llama 3.1 8b 262k residual stream and MLP SAES. Explanations can be directly downloaded here.
Bricken, T., Templeton, A., Batson, Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C.,, Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield-Dodd K., McLean, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning, Transformer Circuits Thread, 2023, https:/transformer- http://///222/moooooommemeeeeeae html..
Z., Tamkin, A., Nguyen,
Acknowledgements
We would like to acknowledge Lucia Quirke and David Johnston for the work reviewing and discussing the manuscript. We would like to acknowledge Jacob Drori's role in the initial development of new scoring techniques. We would also like thank Joseph Bloom, Sam Marks, Can Rager, Jannik Brinkmann and Sam Marks, Neel Nanda, Sarah Schwettmann and Jacob Steinhardt for their discus- sions and suggestions in an earlier version of the work. We are thankful to Open Philanthropy for funding this work. We are grateful to Core Weave for providing the compute resources.
Bühlmann, P. Invariance, causality and robustness. arXiv preprint arXiv: / 181288222 2018.
Chen, H., Vondrick, C., and Mao, C. Selfie: Self- interpretation of large language model embeddings, 2024. URL https:tpp//... /tt///axxii .arvvi.orrrgoo aba/ 2403. 1094..
Computer, T. Redpajama: an open dataset for training large language models, 2023. URL https: thub. com seen/togethercooputeeee Raaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa Data.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.0866 2023.
Impact Statement
This paper presents work whose goal is to advance the field of Mechanistic Interpretability. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Haddseld-Dodds, Z., Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209.
References
2022.
Arora, S., Li, Y., Liang, Y., Ma, T., and Risteski, A. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computa- lional Linguistics, 6:483 2018.
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scal- ing and evaluating sparse autoencoders. arXiv preprint crXhv:2406.04093, 2024.
Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L., and Geva, M. Patchscopes: A unifying framework for in- specting hidden representations of language models, 2024. URL htllht/ps://xaa htttpss//aaxx..oor////////////// org/abs/240111111 001.0.00..6622.110 06102.
Bakhtin, A., Brown, N., Dinan, E.,, Farina, G., Flaherty, Jacob, A. P.,
C., Fried, D., Goff, A., Gray, J., Hu, H., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis,

---

Automatically Interpreting Millions of Features in Large Language Models
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit- skii, D., and Bertsimas, D. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint wrXiv:2305.01611 2023.
OpenAL. Gpt-4 technical report, 2023. URL https: /
arxiv.org/abs: h2tp//3/s//333..../ 08774.
Park, Choe, J., and Veitch, V. The lin- ear representation hypothesis and the geometry of large language models. ArXiv, abs/2311.03658. 2023. URL https://apt http:///ppp.......s..eeee. www.nteeannniiii. sws.piicccctccollllaaa cscholar. org Beg/CorppsID:::2 :6550429928
Gurnee, W., Horsley, T., Guo, Z. C., Kheirkhah, T. Sun, Q. Hathaway, W., Nanda, N., and Bertsimas, D. Uni- versal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181, 2024.
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E.,, Zhang, K., and Mooij, J. On causal and anticausal learning. arXiv preprint arXiv: 0aep::206664111 2012.
Huang, J., Geiger, A., D' O'Oosterlinck, K., Wu, Z., and Potts, C. Rigorously assessing natural language explanations of neurons. arXiv preprint arXiv:2309. 10312, 2023.
Shaham, R., Schwettmann, S., Wang. F. Rajaram, A., Hernandez, E., Andreas, J., and Torralba, A. mul- timodal automated interpretability agent, 2024. URL https: http://aaww..r//a/////////////// 143944.
Juang, C., Paulo, G., Drori, and Nora, Under- standing and steering Llama 3, 2024. URL https: / ///oodofiiie re. ai /wwp:/looo//ee// httpg///eeeeaaaaeeeeieeeeee
Kharlapenko, D., neverix, Nanda, N., and
Singh, C., Hsu, A. R., Antonello, R., Jain, S., Huth, A. G., Yu, B., and Gao, Explaining black box text modules in natural language with language models, 2023.
Conmy, A. Self-explaining SAE features, 8 2024. URL https://www https///ww..esswongg.
self-explaining sae features.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, T. R., Rees, E.. Batson, J., Jermyn, A., Carter, S.. Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
Kopf, L., Bommer, P. L., Hedström, A., Lapuschkin, S., Höhne, M. M. C., and Bykov, K. Cosy: Evaluating textual explanations of neurons, 2024. URL https:
//arxiv. org/ abs 2205/ 2405.
20331.
Lee, J., Oikarinen, T., Chatha, A., Chang, K.-C,, Chen, Y., and Weng, T.W.. The importance of prompt tuning for automated neuron explanations, 2023. URL https:
/ ccalinngmooooemmniiiitt/eeeeeee
//arxiv. org/ /2300 06200.
Lin, J. and Bloom, J. Neuronpedia: Interactive reference and tooling for analyzing neural networks with sparse autoen- coders, 2023. URL https: / www. www.nnuuonnnne newronped ia. org, Software available from neuronpedia.org.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models, 2023. URL https::/p//...c........tt arrxiv..vvv///////v///a/////aaav wwt.ooggg//./o...ooooo.ooo...o/o 16299. 1629.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. 2024. URL https: //arxive //aaxiivorrg/abbb.e...... /2488 06292.
Macgrath, T. Understanding and steering Llama 3, 2024. URL. nttps://goodff: ai/bllogg/ research peechd-peeiiw....
Muennighoff, N., Tazi, N., Magne, I.,, and Reimers, N. Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210003116, 2022. doi: 10.48550/ ARXIV.2210.07316. URL
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. Zoom in: An introduction to cir- cuits. Distill, 5(3), March 2020. ISSN 2476-0757. doi: http:///diitiil.0004.0000. URL http: //xx.dx. doi. org/10. 23915 dtstill. 00024.00.... ..00240.00 001.
10

---

Automatically Interpreting Millions of Features in Large Language Models
A. SAE interpretations of random sentence
this section, we show some non cherry-picked examples feature interpretations on a sentence.
Feature 44, Layer (0.68;0.56): Proper nouns people, places, ideas, concepts, often denoting names, organizations, or entities, some indicating presentalions or resources, with varied applications and contexts.
Feature 6880, Layer 15 (0.90;0.70): Third, second and first- person singular pronouns, often used as subjects or objects in sentences, usually referring people, animals or inanimate objects.
Feature 3031, Layer s (0.35;0.88): Third- person singular feninine pronouns and nouns indicating fenale presence.
that refer to
plural
Feature 4661, Layer 40 (0.63;0.59):
"I spoke with her, and she had this awesome opportunity," he said. "And having the resources of the Center is awesome. definitely outcompetes other programs was interested in.
Verbs and verb-related phrases indicating necessity, abil preparedness, or opportunity, often of action, decision-making, or progress towards goal.
Feature 8603, 6 (0.9); A pronoun that signals the start quotation or dialogue.
Feature 7343, Layer 20 (0.71,0.7) 777)).
Feature 2528, Layer 0 (0.70,0.57): Variioss nouns dino sports, social groups, and objects, often relating concepts of identity, conmunity, or competition.
Feature 2011, Layer (0.88;0.83):
Verb phrases or nouns representing actions, services, or objects related to online forns, accounts, transactions, often followed specific details.
"Adjectives describing a person's expertise or proficiency, often used to convey sense of capability, such \lexperienced\\, \iskilled\\, Wknookeddeabbelll and Viqualified\1."
Figure AI. SAE features interpretations for random sentence. To visualize the feature interpretations produced, select sentence taken from the RPJv2 dataset. We selected tokens in different positions in that sentence and filter for features that are active in different layers. Then randomly select active features and their corresponding interpretations to display. We display the detection and fuzzing scores of each interpretation, which indicate how well it explains other examples in the dataset (see Section 3 for details on these scores), The features selected had high activation, but were not cherry-picke based on interpretations scores.
B. Explainer Prompt
The system prompt if not using Chain of Thought (COT):
You are a meticulous AI researcher conducting an important investigation into patterns found in language, Your task is analyze text and provide an interpretation that thoroughly encapsulates possible patterns found in it. Guidelines:
You will be given a list of text examples on which special words are selected and between delimiters like << this >>.
If sequence of consecutive tokens all are important, the entire sequence of tokens will be contained between delimiters <<just like this>>, How important each token is for the behavior is listed after each example in parentheses.
Try to produce a concise final description. Simply describe the text features that are common the examples, and what patterns you found.

---

Automatically Interpreting Millions of Features in Large Language Models
If the examples are uninformative, you don't need to mention them. Don't focus giving examples of important tokens, but try to summarize the patterns found in the examples.
Do not mention the marker tokens ($<<$ $>>5) in your interpretation.
not make lists of possible interpretations. Keep your interpretations short and concise.
The last line of your response must be the formatted interpretation, using [interpretatien]:
We add to the previous prompt the following if we want to do COT:
To better find the interpretation for the language patterns, go through the following stages:
1.Find the special words that are selected in the examples and list a couple of them. Search for patterns in these words, if there are any. Don't list more than 5 words.
Write down general shared features of the text examples. This could be related to the full sentence surrounding the marked words.
the words
3. Formulate a hypothesis and write down the final interpretation using [interpretation)::
One of the few shot examples of how examples are displayed to the model.
Example 1: and was <<over the meon>> to find
Activations: (over", 5), ( the", 6), ( moon", 9)
Example 2: we'll 11 be laughing <<till the cows come home>>! Pro
Activations: ("ill"" 5), (" the", 5), (*) cows", 8), (* come", 8), (** home", 8)
Example 3: thought Scotland was boring, but really there's more <<than meets the eye>>! I'd
Activations: (hha""" 5), (' meets", 7), (* the", 6), (*) eye", 8)
If COT is used, explicit example of using COT is demonstrated.
ACTIVATING TOKENS: 'over the moon", 'thann meets the eye".
SURROUNDING TOKENS: No interesting patterns.
Step 1.
The activating tokens are all parts of common idioms.
12

---

Automatically Interpreting Millions of Features in Large Language Models
The surrounding tokens have nothing in common.
Step 2.
The examples contain common idioms.
In some examples, the activating tokens are followed by an exclamation mark.
Step 3.
The activation values are the highest for the more common idioms in examples and 3.
Let me think carefully. Did I miss any patterns in the text examples? Are there any more linguistic similarities?
Yes, missed one: The text examples all convey positive sentiment.
Afterwards an interpretation added to the example
[interpretation]
Common idicms in text conveying positive
C. Examples of activating contexts and interpretations
As discussed in the main text, the interpretations found for given feature can be very different depending the way to sample the activating contexts shown to the explainer model, see Figure A2. This has its advantages and disadvantages.
When using the top activating contexts, the explainer model normally gives an interpretation that more narrow "The concept buffer, referring to something that separates, shields, or protects one thing from another, often used in various contexts such as physical barriers, chemical reactions, or digital data processing" instead of one that captures the full distribution "Words phrases associated with concepts of spatial or temporal separation (buffers, zones, or cushions) colloquialisms (buff, buffs, Buffy, referring to popular TV show enthusiast)". Here the narrower interpretations resulted in lower scores: the interpretation generated from examples sampled from the whole distribution scored 0.95 accuracy in fuzzing and 0.93 accuracy in detection, while the interpretation generated from top examples only achieves 0.8 accuracy in fuzzing and 0.74 accuracy in detection.
the other hand, we look at the second example, sampling from whole distribution examples may sometimes confuse the explainer model "Tokens often precede succeed prepositions, articles, and words that signal possession or quantity, frequently indicating relationship between objects actions. feature", which might see the pattern that is more clear in the top activating examples "Descriptions of food events where food is involved, often mentioning leftovers, and sometimes mentioning the act of eating, serving, or storing food, well as the amount of food, or the pleasure or satisfaction derived from it." Here we see very poor scores on the interpretation from randomly sampled examples, 0.54 accuracy in detection and 0.57 accuracy in fuzzing, while the interpretation generated from the top examples has 0.89 accuracy both detection and fuzzing.
D. Different scoring methods
D.I. Detection details
The prompt used for detection scoring is the following:
You are an intelligent and meticulous linguistics researcher.
You will be given certain feature of text, such as \male pronouns" or "text with negative sentiment".
13

---

Automatically Interpreting Millions of Features in Large Language Models
You will then be given several text examples. Your task is to determine which examples possess the feature.
For each example in turn, return if the sentence is correctly labeled or if the tokens are mislabeled. You must return your response in a valid Python list. Do not return anything else besides Python list.
Together with the prompt, there are several few shot examples like the following:
<user_prompt>
feature interpretation: Words related to American football positions, specifically the tight end position.
Text examples:
Example 0:Getty Images Patricts tight end Rob Gronkowski had his boss
Example 1: names of months used in The Lord of the Rings: the
Example 2: Media Day 2015 LSU defensive end Isaiah Washington (94) speaks to the
Example 3: shown, is generally not eligible for ads. For example, videos about recent tragedies,
Example 4: line, with the left side namely tackle Byron Bell at tackle and guard Amini
<essisteattrreponse>
[1,0,0,0,1]
In this example the <user-prompt> and the <assossantresponse>>>>>> tags are substituted with the correct instruct format used by the scorer model. In detection scoring, 5 shuffled examples are shown to the model at the same time.
D.2. Fuzzing details
The prompt used for fuzzing scoring is the following:
You are an intellin igent and met culous linguistics researcher.
You will be given a certain feature of text, such as 'male pronouns" or 'text with negative sentiment", You will be given a few examples of text that contain this feature. Portions of the sentence which strongly represent this feature are between tokens << and >>.
Some examples might be mislabeled. Your task is to determine if every single token within < and >> is correctly
14

---

Automatically Interpreting Millions of Features in Large Language Models
labeled. Consider that all provided examples could be correct, of the examples could be correct, or mix. An example is only correct if every marked token is representative of the feature
For each example in turn, return if the sentence is correctly labeled or 0 if the tokens are mislabeled. You must return your response in valid Python list. Do not return anything else besides Python list.
Followed by few-shot examples of the kind:
<user_prompt>
feature interpretation: Words related to American football positions, specifically the tight end position.
Text examples:
Example 0:Getty Images Patricts<< tight end>> Rob Gronkowski had his boss
Example 1: posted You should know thisk< about>> offensive line coaches: they are large, demanding<< men>>
Example 2: Media Day 2015 LSUK< defensive>> end Isaiah Washington (94) speaks<< to the>
Example 3:<< running backs>>, he said. Defensive << end>>
Carroll Phillips is improving and his injury is
Example 4:<< line>>, with the left side namely<< tackle>> Byron Bell at<< tackle>> andk< guard>> Amini
<assistant_respone
[1,0,,,1] 1,11
In this example, the <user prompt>> and the <assistant respons> tags are substituted with the correct instruct format used by the scorer model. In fuzzing scoring, shuffled examples are shown the model at the same time.
D.3. Surprisal details
In surprisal scoring, the cross-entropy loss of the scorer model computed over the tokens of an example. For each interpretation, this loss computed with the interpretation and with default interpretation "Various unrelated sentences," where the examples can either be activating context non-activating contexts. In first approach, Llama 3.1 70b base was used. The prompt starts with few shot examples like:
The following is description of certain feature of text and list of examples that contain the feature.
Description:
15

---

Automatically Interpreting Millions of Features in Large Language Models
References to the Antichrist, the Apocalypse and conspiracy theories related to those topics.
Sentences:
" by which he cistinguishes Antichrist is, that he would rob God of his honour and take it to himself, he gives the leading feature which we ought "
'3 begins. And the rise of Antichrist. Get ready with
would be destroyed. The worlds economy would likely collapse as a result and could usher in a one world government. movement. I wrote small 6 page
Description:
Sentences containing digits forming a four-digit year.
Sentences:
" 20, 2013 at 7:41 pm Martin Smith
of 2012. In other words, Italy's
Wend 2012 levels). In the first quarter of 2013, we expect revenue to be up slightly from the fourth quarter
Description:
Text related to banking and financial institutions.
Sentences:
": He is on the Board of Directors with the Lumbee Bank
" refurbishing the Bank's s branches. BIP reached 400 thousand users in one year The of BIP has already doubled The
the Federal Deposit Insurance Corp.
Description:
Occurrences of the word "The' the beginning sentence.
Sentences:
<Thee Smoking Tire hits the canyons with one of the fastest Audi's on the road
'The Chairman of the ABI
16

---

Automatically Interpreting Millions of Features in Large Language Models
1 "Tee administrative center is the town of Koch.
With the pairs of losses with the interpretation and with the default interpretation, it is possible to compute the decrease in loss caused by having access to the interpretation. It is expected that in activating contexts this difference will be greater than in non-activating examples, and the score of the interpretation is the AUC computed using this loss as proxy for activating and non-activating labels.
D.4. Embedding details
For embedding scoring, we use a small 400M parameter model. We chose small performant model on MTEB (Muennighoff et al., 2022) because we found similar scores when using a larger 7B parameter model, see fig A3, as this size allowed us to do increase the number of examples used scoring.
set of activating and randomly selected non-activating examples are embedded using the scorer model. Then a "query" instruction is embedded:
Instruct: Retrieve sentences that could be related to the interpretation. Query: Niinterpretation//
The cosine-similarity between the instruction embedding and the examples is computed and used as a proxy for activating and non-activating labels when computing the AUC, which is the score of that interpretation.
D.5. Adversarial examples
Most features are active in less than 0.1% of the full dataset used to collect the activations, making random -ottaitiv contexts very diverse. Randomly sampling non-activating examples cannot be used to determine whether feature fires in token in specific context on that token in general, as it is unlikely that that token randomly occurs in each non-activating example. As SAEs are scaled and features become sparser and more specific, techniques that overly rely on activating contexts will have more imprecise results (Gao et al., 2024).
Motivated by the phenomenon of feature splitting, we could use "similar" features to test whether interpretations are precise enough to distinguish between similar contexts. potential approach is using cosine similarity between decoder directions of features to find counterexamples for an interpretation. Some works (Juang et al., 2024; Macgrath, 2024) have shown that significant fraction of current feature interpretations can't 't be used to distinguish between features with high similarity.
D.6. Correlation between scores
We compute the correlation between these different scores and simulation 800 different feature interpretations, of the 131k feature SAE trained on the residual stream of Gemma 2 9b spread across 4 different layers. We use 100 activating example, 10 from each of 10 different activating quantiles, and 100 non-activating examples to compute these scores and use Llama 3.1 70b Instruct as the scorer model for simulation, fuzzing and detections, while we use Llama 3.1 70b Base for surprisal and Stella-400M as the embedding model.
Interestingly, we also find that the correlation between simulation scores and other scores also increases when the simulation scorer is Claude Sonnet 3.5. These estimations are on smaller number of features( 100), due to the high cost of simulation using Claude Sonnet, but on this set, the Spearman correlation between fuzzing and simulation increases from 0.69 0.75, between detection and simulation increases from 0.31 to 0.38, between embedding and simulation from 0.14 to 0.21 1 and from surprisal to simulation from 0.02 to 0.10. The baseline numbers for the correlation of simulation scores are lower on this set, so we expect that had we done Claude scores larger set of explanations, the correlations between the scores would be higher.
D.7. Disagreement between scores
In this section, we will discuss how different types of scores might evaluate explanations differently. In particular, we will focus on an interpretation with fuzzing score but low detection score, high fuzzing score but low simulation score, high simulation score but low embedding score and high embedding score and low simulation score.
For instance, feature 2 of layer of the 131k feature SAE model trained on the residual stream of Gemma has fuzzing
17

---

Automatically Interpreting Millions of Features in Large Language Models
Table Al. Spearman correlation, computed over scores 800 different feature interpretations.
Fuzzing
Detection
Simulation
Embedding
Surprisal
Fuzzing
1
0.73
0.75
0.41
0.30
Detection
0.44
0.71
0.62
Simulation
0.28
0.15
Embedding
1
0.79
Surprisal
1
Table A2. Pearson correlation computed over 800 different feature scores
Fuzzing
Detection
Simulation
Embedding
Surprisal
Fuzzing
1 0.74
0.74
0.42
0.32
Detection
0.46
0.70
0.62
Simulation
0.30
0.14
Embedding
1
0.79
Surprisal
1
score of 0.9 but a detection score of only 0.43. The interpretation given by our pipeline is "Verbs that link a subject to additional information, often in a formal or descriptive tone." Most of the activations of this feature are on sentences like "This <is needed>> because Magento (...) or "For instance, this is>> technically correct syntax", where <> represent active tokens. In both these examples, the model correctly identifies the highlighted tokens as active during fuzzing, but incorrectly identifies the context non-active during detection. On the other hand, detection incorrectly identifies sentences like "pay for things that would prevent larger issues down the road is better in the long run." or "Neuroscientist Jack Gallant calls the research technologic tour de force and says the ultimate decoder would provide vivid" as active. This explanation is too vague the activating context, leading to a low detection score, but specific enough in the types of tokens that are active, leading to a high fuzz score.
Feature 281 of the same layer has a fuzzing score of 0.97, but a simulation score only 0.19. Its interpretation is "URLs or hyperlinks containing query parameters, indicating a request for specific data or actions on a web page." The simulation score lower for than the fuzzing score because the model has decide which parts of the URL the feature is active while the fuzzing scorer either is shown links that are highlighted which it will say are active non-links that highlighted and are, most likely, not active.
Feature of layer 24 has a simulation score of 0.73, but an embedding score of only 0.43. Its interpretation is "Prevalence of logical operators and conjunctions text, including simple addition and conjunction various contexts, as well as indicators of contrasting or additional information, often used for comparison or to provide supplementary details". While doing simulation, the model correctly identifies that in the sentence "the cost of the unit itself, plus installation and construction costs.", the token "plus" is active but all the others aren't. The same is true for sentences like "Plus my potato bread made on the ANZAC Day" and "Cliff Chiang reintroducing Orion in Wonder Woman! Plus interviews". On the other hand, these sentences don't 't have high embedding similarity with the explanation.
Feature 10 of layer 16 has the opposite situation happening, with an embedding score 0.74 and a simulation score of only 0.04. Its interpretation is "Verbs, prepositions, and adverbs that connect clauses or indicate direction, movement, or progression, often in a sequence of actions or events"" sentences like "right, three months jto close downg, There was dead silence when the message was read. Everybody waited for Mr. Smith to speak. Mr. Gingham" the simulator model incorrectly identifies waited" and the second to" active, and misses the real activation in "to close down". The same happens in "This is igoingi to ibeq one yswinginga party. Especially if the guests survive Uncle Wilhelm. I love weddings. I love going to them, love being in"" where the simulator model identified "love being in" as well as "love going to" as being active. The embedding model correctly identified that most activating sentences have the mention of motion.
With these examples, we hope to demonstrate that these generated explanations are not perfect, but that we have an easier understanding of their flaws by looking at which cases different scores disagree on.
18

---

Automatically Interpreting Millions of Features in Large Language Models
Table A3. Impact of training dataset Fuzzing and Detection performance. Numbers shown the median and the interquartile (25%-75%% range.
Experiment
Fuzzing
Detection
Embedding
Random interpretation
0.51 (0.45-0.57)
0.51 (0.45-0.58)
0.51 (0.44-0.57)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 (0.50-0.59)
RPJ-v2
0.76 (0.67-0.86)
0.74 (0.63-0.85)
0.67 (0.57-0.80)
Pile
0.76 (0.67-0.86)
0.76 (0.67-0.85)
0.69 (0.57-0.80)
E. SAE models used
Throughout this work, we used different SAEs trained on Gemma. The 16k feature ones trained on the MLP have the following average LO norms per layer:
0:50, 1:56, 2:33, 3:55, 4:66, 5:46, 6:46, 7:47, 8:55, 9:40, 10:49, 11:34, 12:42, 13:40, 14:41, 15:45, 16:37, 17:41, 18:36, 19:38, 20:41, 21:34, 22:34, 23:73, 24:32, 25:72, 26:57, 27:52, 28:50, 29:49, 30:51, 31:43, 32:44, 33:48, 34:47, 35:46, 36:47, 37:53, 38:45, 39:43, 40:37, 41:58
The 16k feature ones trained on the residual stream have the following average L0 norms:
0:35, 1:69, 2:67, 3:37, 4:37, 5:37, 6:47, 7:46, 8:51, 9:51, 10:57, 1 11:32, 12:33, 13:34, 14:35, 15:34, 16:39, 17:38, 18:37, 19:35, 20: 36, 21:36, 22: 35, 23: 35, 24: 34, 25: 34, 26: 35, 27:36, 28: 37, 29:38, 30:37, 31:35, 32: 34, 33:34, 34:34,
35:34, 36:34, 37:34, 38:34, 39:34, 40:32, 41:52
The 131k feature ones trained in the residual steam have the following average LO norms:
0:30, 1:33, 2:36, 3:46, 4:51, 5:51, 6:66, 7:38, 8:41, 9:42, 10:47, 11:49, 12:52, 13:30, 14:56, 15:55, 16:35, 17:35, 18:34, 19:32, 20:34, 21:33, 22:32, 23:32, 24: 55, 25:54, 26:32, 27:33, 28: 32, 29:33, 30:32, 31:52, 32: 51, 33:51, 34:51, 35:51,
36: 51, 37:53, 38:53, 39:54, 40: 49, 41:45
We also used SAEs with 65k features, trained the residual stream and the MLP of Llama 8b.
F. Factors that influence the explainability
F.1 Dependence on dataset
Even though a significant portion of SAE features are less active when using RPJv2 instead of the Pile, we find that the features that are active are generally interpretable to the same degree. These evaluations were done using the 13lk feature SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3.1b 70b instruct, quantized to 4bit.
1. DEPENDENCE ON CHAIN OF THOUGHT AND ACTIVATION INFORMATION
We find that COT slightly increases the scores of the interpretations found, and that it significantly slows down the rate at which one can produce interpretations. Giving the explainer model, the activations associated with each token seem to slightly increase the scores of the interpretations generated. These evaluations were done using the 131k k feature SAE trained on the residual stream Gemma 2 9b. The scorer and the explainer model where Llama 3.lb 70b instruct, quantized to 4bit.
F.2. Dependence on context length
The context length of the shown examples did not significantly change the scores obtained for the interpretations. These evaluations were done using the 131k feature SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model were Illma b 70b instruct, quantized to 4 bits.
19

---

Automatically Interpreting Millions of Features in Large Language Models
Table A4. Impact of prompt content on Fuzzing and Detection performance. Numbers shown the median score and the interquartile
(25%-75%% range.
Experiment
Fuzzing
Detection
Embedding
Random interpretation
0.51 (0.45-0.57)
0.51 (0.45-0.58)
0.51 (0.44-0.57)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 (0.50-0.59)
Activations in prompt
0.76 (0.67-0.86)
0.74 (0.63-0.85)
0.68 (0.57-0.80)
No activations in prompt
0.75 (0.65-0.86)
0.73 (0.60-0.84)
0.68 (0.58-0.79)
COT in prompt
0.76 (0.67-0.86) 0.86)
0.73 (0.61-0.85)
0.65 (0.55-0.75)
Table A... Impact of context length on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.
Experiment
Fuzzing
Detection
Embedding
Random interpretation
0.51 (0.45-0.57)
0.51 (0.45-0.58)
0.51 (0.44-0.57)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 (0.50-0.59)
16 context
0.75 (0.65-0.86)
0.74 (0.62-0.85)
0.70 (0.59-0.81)
32 context
0.76 (0.67-0.86)
0.74 (0.63-0.85)
0.67 (0.57-0.80)
64 context
0.74 (0.64 0.64)
0.70 (0.57-0.81)
0.65 (0.54
F3. Dependence on the origin of examples
Sampling the examples from the whole distribution of activations, or sampling a fixed number of examples over different activation quantiles, significantly improved the scores, compared with sampling only from the top examples. These evaluations were done using the 131k feature SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model were Llama 3.1b 70b instruct, quantized to 4 bits.
F.4. Dependence on the number of examples
The number of examples shown the model seems saturate, least with the explainer model we used. These evaluations were done using the 13lk feature SAE trained the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3. 70b instruct, quantized to 4bit.
F.5. Explainability across layers
We find that, in the case of the 131k feature SAE trained on the residual stream, the earliest layers have lower scores than the later layers. These evaluations were done using the 131k feature SAE trained on the residual stream of Gemma 2 9b. The scorer and the explainer model where Llama 3. 70b instruct, quantized to 4bit.
Table A6. Impact of example sampling strategics on Fuzzing and Detection performance. Numbers shown are the median score and the interquartile (25%-75%) range.
Experiment
Fuzzing
Detection
Embedding
Random interpretation
0.51 (0.45 0.57)
0.51 (0045-0.587
0.51 (0.44-0.57)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 (0.50-0.59)
Randomly sampled
0.76 (0.68-0.86)
0.74 (0.62-0.84)
0.66 (.78))
Sampled from quantiles
0.77 (0.69-0.87)
0.74 (0.64-0.85)
0.68 (0.57-0.80)
Sampled from top examples
0.73 (0.64-0.83)
0.72 (0.62-0.82)
0.70 (0.58-0.80)
20

---

Automatically Interpreting Millions of Features in Large Language Models
Table A7. Impact of numher of examples Fuzzing and Detection performance. Numbers shown are the median score and the
interquartile (25%-759% range.
Experiment
Fuzzing
Detection
Embedding
Random interpretation
0.51 (0.45-0.57)
0.51 (0.45-0.58)
0.51 (0.44-0.57)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 (0.50-0.59)
Shown 10 examples
0.73 (0.62-0.85)
0.71 (0.58-0.82)
0.64 (0.54-0.74)
Shown 20 examples
0.74 (0.64-0.85)
0.72 (0.60-0.84)
0.66 (0.54-0.76)
Shown 40 examples
0.76 (0.67-0.86)
0.74 (0.63-0.85)
0.68 (0.57-0.80)
Shown 60 examples
0.75 (0.66-0.85)
0.73 (0.62-0.84)
0.68 (0.57-0.79)
Table A8. Comparison of scores of interpretations given by different models. Numbers shown the median and the interquartile (25%-75%) range.
Experiment
Fuzzing
Detection
Embedding
Simulation
Random interpretation
0.51 (0.45-0.57)
0.51 (0.45-0.58)
0.51 (0.44-0.55)
-0.02 (-0.02-0.00)
Claude Llama 70b
0.75 (0.68-0.84)
0.75 (0.65-0.85)
0.70 (0.58-0.81)
0.30 (0.28-0.32)
0.76 (0.67-0.86)
0.74 (0.63-0.85)
0.67 (0.57-0.80)
0.29 (0.28-0.32)
Llama 8b
0.70 (0.60-088))
0.70 (0.59-0.81)
0.64 (0.54-0.75)
0.26 (0.23-0.30)
Human
0.75 (0.66-0885)
0.74 (0.64-0.85)
0.71 (0.62-0.81)
0.36 (0.32-0.39)
F6. Dependence on size of explainer models
Having larger explainer model leads to better interpretations, but our results suggest that this benefit tends to saturate, as we find that the scores of explanations given by Claude or Llama 70b are very similar. We only did human interpretation of 100 features. Simulations scores are also only computed over 100 features.
F7. Dependence on size of explainer models
We find that both fuzzing, detection and simulation scoring are dependent on the size of the explainer model, that is, given the same explanations, larger models achieve higher scores on average. While Claude and Llama 80b have similar fuzzing and detection scores, simulation, being more complex, better performed by Claude.
F8. Dependence on SAE size and location
We compare the interpretability of SAEs with different number features, trained on the same model, with the neurons of that model. We find that the SAE with the highest number of features have the highest scores in the case of the Gemma 9b SABs and that SAEs trained on the residual stream have higher scores for both Gemma 2 9b and Llama 3.1 8b.
The scorer and the explainer model were Llama 3.1 70b instruct, quantized to 4bit.
Table A9. Comparison scorer models for fuzzing, detection and simulation. Numbers shown are the median score the interquartile (25%-75%) range.
Experiment
Fuzzing
Detection
Simulation
Claude
0.76 (0.66-0.88)
0.72 (0.58-0.87)
0.33 (0.29-0.36)
Llama 70b
0.76 (0.67-0.86)
0.74 (0.63-0.88)
0.29 (0.28-0.33 0.32)
Llama 8b
0.70 (0.60-0.81)
0.70 (0.59-0881)
0.26 (0.25-0.31)
21

---

Automatically Interpreting Millions of Features in Large Language Models
Table A10. Comparison of SAEs for Fuzzing and Detection. Numbers shown are the median score and the interquartile (25%-75%%) range.
Experiment
Fuzzing
Detection
Random interpretation
0.31 (0.45-0.57)
0.51 (0.45-0.58)
Randomly initialized Topk SAE
0.55 (0.50-0.60)
0.54 10.50-0.599
131k features Gemma 2 9b
0.76 (0.67-0.86)
0.74 (0.63-0.85)
16k features Gemma 2 9b
0.73 (0.63-0.83)
0.70 (0.59-0.79)
Top 32 neurons Gemma 2 9b
0.62 (0.54-0.70)
0.59 (0.53-0.64)
Top 256 neurons Gemma 9b
0.59 (0.53-0.65)
0.57 (0.52-0.62)
262k features I Llama 3.1 8b ML.P
0.79 (0.69-0.86)
0.79 (0.64-0.85)
262k features Llama 3.1 8b
0.81 (0.71-0.86)
0.83 (0.68-0.85)
Top 32 neurons Llama 3.1 8b
0.35 (0.51-0.62)
0.53 (0.50-0.59)
Top 256 neurons Llama 3.1 8b
0.54 (0.49-0.60)
0.53 (0.50-0.57)
G. Automatically interpreting interventions
G.1. Scoring implementation
For each feature, we sample a pool of 40 length-64 prompts from RPJv2 (Computer, 2023), of which 30 i.i.d. prompts are taken for scoring, while the remaining 10 used by the explainer. The pool is sampled among nonzero activating contexts, stratified by the quintile of the feature's max activation in that context (i.e., the nonzero activating documents by the context's maximum activation of the current feature, then 8 contexts second etc.). Each of these prompts is then truncated to include only the first token on which the feature activates, so that all previous activations for the feature are O. Activations on the first token position are ignored because the SAES were not trained on that position.
sorted
sampled from the first quintile, 8 from the
We filter out features that activates on less than 200 of the 10 million RPJ-v2 tokens.
We then sample generations with maximum of 8 new tokens from the subject model. For generations with the intervention, we perform additive intervention the feature at all token positions after and including the final prompt token. We tune the intervention strength of each feature to various KL-divergence values on the scoring set with binary search. We stop the binary search when the KL divergence is within 10% of the desired value". For our zero-ablation experiment, instead of doing an additive intervention we clamp the feature's 's activation to O. Specifically, the hidden states are encoded by the SAE, then the SAE reconstruction error is computed using a clean decoding, then the SAE encoding is clamped and decoded, and the error is added to the clamped decoding.
The scorer is 1lama-3.1 8B base. We use the base model for improved calibration, and prompt it as follows.
<PASSAGE>
from west to east, the westmost of the seven wonders of the world is the great wall of china
The above passage contains an amplified amount of "Asia"
<PASSAGE>
Given 4x is less than 10, 4
The above passage contains an amplified amount of "numbers"
<PASSAGE>
In information theory, the information content, self-
'Sometimes the diveeveeeee is perfectly monotonic the intervention strength so 10% error is exceeded. We report the average divergence we abserve Figure A6
22

---

Automatically Interpreting Millions of Features in Large Language Models
information, surprisal, or Shannon information is basic
quantity derived by her when she student Windsor
The above passage contains an amolified amount of "she/her pronouns"
<PASSAGE>
My favorite food oranges
The above passage contains an amplified amount of "fruits and vegetables"
<PASSACE>
G.2. Generating interpretations
We generate interpretations using only the intervention's 's effect on the subject's next-token probabilities because this leads concise and precise prompt. The explainer, like the scorer, Llama-3.1 8B.
The explainer sees distribution of 10 prompts that is sampled i.i.d. from the same population as the scorer's prompts.
We use the following prompt for the explainer, with few-shot examples truncated for brevity.
We' studying neurons in a transformer model. We want to know how intervening them affects the model's output.
For each neuron, we'll show you few prompts where we intervened on that neuron at the final token position, and the tokens whose logits increased the most.
The tokens are shown in descending order of their probability increase, given in parentheses. Your job is to give short summary of what outputs the neuron promotes.
Neuron
<PROMPT>Given 4x is less than 10, </PROMPT> Most increased tokens: , 4' (+0.11), 10' (-0.04), 40' (+0.02), 2' (+0.01)
<PROMPT>For some reason</PROMPT> Most increased tokens: one' (+0.14), 1' (+0.01), fr' (+0.01)
<FROMPT>insurance does not cover claims for accounts with<</ROMPT>
Most increased tokens: ' one' (+0.1), , more' (+0.02), 10' (+0.01)
interpretation: numbers
Neuron
23

---

Automatically Interpreting Millions of Features in Large Language Models
G.3. Baselines
Random SAE. We experiment with a random top-k SAE with 50, roughly the average sparsity the gemma SAEs. The encoder is initialized with 131,072 spherically uniform unit- features, and the decoder is initialized to its transpose. We use random SAE with TopK activations because need sparsity for sampling procedure to work properly.
Random interpretations. For each layer and target KL value, we compute a random interpretation baseline where we shuffle the interpretations across features.
G.4. Intervention interpretability results
Figure A6 shows the average intervention score at each layer, for few KL values. Intervention scores are higher in later layers, likely because of the increased proximity to the model's output.
Figure A7 similar to the right half of Figure 3, but multiple layers and KI. values. Early layers have small intervention scores across the board, so there is little correlation, while the reason for little correlation in layer 41 less clear.
24

---

Automatically Interpreting Millions of Features in Large Language Models
Random
Top
Actraations:
ignorand
design.
buffers used with the DRAMs will depend the
and the angle parking in the downtown provides buffer
buffie againsl Englishddomint
almost
necessarly coincident with anbietrraiiiioal screeming
fo
buffer
lods be placed once and ther
Nwo VISCCTIN Video
ATED
DIFFUSIBLE FACTORS NURAINGE
in Lyme disease When bufferins species like
season
don't have room buffer
etclnMany buffet style
buffe
acids) When muscles fatique acid build
Used for bufibufler tw
manager registers showr
T62 byc
buffer Syyer
Media:
Noveonee Bean
buffer
the Big East Media
leaque in qoal because Glailuio Bufion
25 25-1oo
buffe striss on each side
womanhood
catacly
amendment requinin buffersavo vote these bwo
on the site as buffee againet Englshidominated
transfer through buffers
handshali
offers biggei buffie
remainn in the feeder pathern of that first school
and build buffe states between Russia and
restaurants offering special
manus
paciaging
vibls...
evistence: council rebuili calls from Mayor Bill de
amendmeen that
buffer vorde be installed on sided
which is convertible to restricted shares of Rick
spacel nand/or landscape plantings. buffer
ccolored
named Pumplón
our family
does, sorneSime buffrenn apents such citric
buffe zone between Syrio and lordan + which would
between uses. landscapes buffer
impending
powerful buffer inside muscles
fact buffon strics on each side
Streaming except for MLB buffer
Christanity buffering Europe from islam which sllows
Attention Civil Wer Bully Join the Corl We
blood volume and thus the bufferno dapacity
trying send
Jordan
Happy Days, Joan of Arcadia Buffy the
amendment requiring buffinn vote these two
ELIDESHOT
Panned
observing the Buffett Rull" that millonaires should
the blood can buffer performance declines rapidly, Thus
loon SeSpttemerr
Features: Supercharps
tampons. JwFFFD: See
improve waterr aateerruui OUREER Graduated mix of land
50 maybe and Buffy could make
d Sude
the Author: Jacques van Heer
could find anything useful excest for
bufle despite mistreatment- desplt some rumors
buffee desplte misheetment-ddeppt some numors
point
only contingancy.
Random
Top
Token Activations: 0.0 0.2 03 03 0.6
Token Activaiins
rother than the end the some
sometimes liarr with
Bhereanc to home Varieby of food pleased
resuit we ended up havincla pretty lunch
cabbbage sti-fry,
arle mo takee
home boves lefoverss
05/2 Use Spikes on Thanksgiving
eggt
cools
kept for late breakfast the next The
HATE TDn Thank you?" once loc
Sovi large crock)
Yogurt Granolainn
want to serve this the Peiineddayy
pics. Im talking you too
looking Forward another warmed-
have in your pantry
excellent and portions large the fami: of you might heat: couple
beans or lentil) Served it with brown
not sure scalloped potatoes would
fragile after refrigeation. fact since
can see why you got aeppes request
Short my mom had about pipunds
allow
It's
having
huge hit Helhad diices
No it's not beef brikket-1
goins
I'' all well and good they Isant
of Manna for one's family? Obvioush
sweet potatiess great
was also
had doogy bag) In general we
are at works Greta book of me well (57
dwarves
werd
had tons left ovet sealed the end
have it salad while stuffi
immediately wanted to make pufly tacor), and the
man, you
want
let cool then skim the fai before rehe
time S8 couples who practice this
SB laroe portion of
adoption loarr said Allen the doos
though could
leftovers went back into the fridge My cousinis
possible type of work involved gathering pootioo
enough
thilds-hil,
dineer7SS4FEST: Where would
WiLL be having again!! My deughter didr
CICUS made this quests this weekend
servings
nutritional facts (Raply Deettekomnkkn
always invite people over flr dinner because know
oriunbi they nun cunioutiel food The
they may return enjoy at
bunny other day! Bemo commented
leftovel He took: it and his coffee upstairs with
bases, Also, always by to leave
Figure A2. Activating contexts of feature 209 (top) and 293 (bottom), from layers and 32, respectively. of the 131k feature SAE trained on the residual stream of Gemma 9b. The shown examples are similar to ones given to the explainer model to come up with interpretations.
25

---

Automatically Interpreting Millions of Features in Large Language Models
Linear Regression: R-squared = 0.67
1.0
0.9
0.8
Large Embedding Score
0.7
0.6
0.5
0.4
0.3
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Embedding Score
Figure A3. Comparison between the scores given by small embedding model and larger one
26

---

Automatically Interpreting Millions of Features in Large Language Models
Linear Regression: R-squared 0.55
Linear Regression: R-squared 0.38
0.9
0.8
Surpitall Score
0.7
Fuzzing
0.6
0.6
0.4
Datection
Detection
Linear Regression: R-squared 0.49
Linear Regression: R-squared
1.0
Simulation Score
Surprical Score
0.0
-0.2
0.7
0.8
0.9
1.0
0.5
0.5
0.9
Embedding
Figure A4. Scatter plots with different combinations of scores
27

---

Automatically Interpreting Millions of Features in Large Language Models
0.90
0.85
0.80
Accuracy
0.75
0.70
0.65
0.60
Fuzzing
0.55
Detection
0
5
10
15
20
25
30
35
40
Layer
Figure A5. Accuracy fuzzing and detection scoring Dots correspond to the median score over c.a. 300 feature interpretations, and colored region denotes the interquartile range.
SAE
Random TopK SAE
KL=0.10
avg KL=0.10
0.8
avg KL=0.33
avg KL=0.33
avg KL=1.01
avg KL=1.02
Average Score
0.6
avg KL=3.07
avg KL=3.07
0.2
0.0
10
15
20
25 Layer
30
35
40
10
15
20
25
30
35
40
Layer
Figure A6. Average intervention score vs layer. SAE features in later layers have more explainable effects on output. Random features. however, have uninterpretable effects on output, even at late layers.
28

---

Automatically Interpreting Millions of Features in Large Language Models
Layer 8, KL 0.1
Layer 32, KL 0.1
Layer 41, KL 0.1
intervention Score
0.75
0.75
0.75
0.50
0.50
0.50
0.00
0.00
0.00
0.25
0.25
0.25
-0.50
-0.50
Layer 8, KL 0.33
Layer 32, KL 0.33
Layer 41, KL 0.33
1.5
1.5
1.5
intervention Score
1.0
1.0
1.0
0.5
0.0
0.0
0.0
-0.5
00.
0.5
-1.0
Layer KL 1.0
Layer 32, KL
Layer 41, KL 1.0
2.5
2.5
2.5
Intervention Score
1.5
1.5
1.5
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.5
00.
Layer KL 3.0
Layer 32, KL
Layer 41, KL
3
intervention
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
0.5
0.6
0.7
0.8
0.9
fuzzing Score
fuzzing Score
fuzzing Score
Figure A7. Comparison of intervention and fuzzing scores across layers at various target KL values.
29


Reviews:
[{"review_id": "Z61gsEAg72", "reviewer": "Reviewer_X3cr", "summary": "This paper proposes an automatic concept explanation method based on LLM to address the problem of poor human comprehension of sparse autoencoders. Specifically, the author collects highly responsive sentences and corresponding concepts in SAE and carefully designs LLM prompts to prompt LLM. Then LLM automatically explains other units in SAE. The author also provides 5 evaluation metrics to evaluate the effectiveness of the method.\n\n## update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your helpful comments.\n\n\n>The authors only used one LLM to interpret SAE in this article. It would be more convincing if the authors could consider more LLMs and perform prompt sensitivity analysis.\n\nIn the current version of the article, we evaluate the explanation given by Llama 3.1 70b, Llama 3.1 8b and Claude 3.5 Sonnet, as shown in table A8. We discuss prompt sensitivity analysis in the last reply of this rebuttal \n\n> The author claims in the title to explain millions of features in LLM, but it seems that this article explains the massive units in SAE latent space.\n\nIn this work we use features to mean interpretable directions in both the residual stream and the MLP outputs. SAEs were developed because directions in the residual stream and MLP (by looking at neurons for instance) were often polysemantic and hard to understand. The interpretability community uses feature to mean interpretable concepts, and so we believe the claim of explaining millions of features to be correct, being similar to the one in Templeton et al., 2024. We have added to the introduction the following sentence: \"Throughout this work we are going to be using latent to mean a specific pair of encoder-decoder indices in SAE, and feature to mean the concept that this latent is representing.”\n\n> What if we let an LLM explain its own SAE? What if we use an additional LLM to explain another LLM's SAE? The author can explore this.\n\nWe did both of these in this work. We used Llama 8b to generate explanations for the Llama 8b SAE, and in general have used other LLMs  – Llama 70b – to generate the explanations for the SAEs trained on LLama 8b and Gemma 2 9b. Llama 8b is worse at generating explanations than Llama 70b and at the time of writing there were no Llama 70b SAEs, which was the smaller model that gave reasonable explanations, so these were the only experiments we could perform.\n\n> Can this method identify some latent units that do not have obvious conceptual information?\n\nSome units have simple conceptual interpretations (they fire on the verb _to be_, or on _chair_) and some have more complicated patterns, like firing in text between parenthesis. We are not sure if this is what the reviewer meant to ask.\n\n> The method in this paper is very interesting, but the experiments seem to be insufficient, for example, there is a lack of some sensitivity analysis, specific ablation experiments, or more potential downstream applications, etc.\n\nWe have ablated model size (for both explainer and scorer), number of examples shown, number of tokens per  example shown, origin of the examples shown, using chain-of-though or no and, showing token activations or. We also investigated the interpretability of different layers, whether the interpretability of the residual stream SAEs and of SAES trained the output of the MLP, as well as the dependence of interpretability on the SAE expansion factors. Is there a specific ablation that the reviewer has in mind?", "claims_and_evidence": "The author claims in the title to explain millions of features in LLM, but it seems that this article explains the massive units in SAE latent space.", "methods_and_evaluation_criteria": "The author provides 5 evaluation indicators, which I think are reasonable.", "theoretical_claims": "No theoretical analysis in the paper.", "experimental_designs_or_analyses": "Lack of comparison with some baseline methods.", "relation_to_broader_scientific_literature": "It is interesting to interpret the parameters in LLM, and this paper proposes an interesting approach.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths:**\n- The idea of ​​using LLM to automatically explain SAE is interesting, although there is similar work, such as using GPT-4 to explain GPT-2.\n\n**Weaknesses:**\n\n- The authors only used one LLM to interpret SAE in this article. It would be more convincing if the authors could consider more LLMs and perform prompt sensitivity analysis.\n- The author claims in the title to explain millions of features in LLM, but it seems that this article explains the massive units in SAE latent space.\n- What if we let an LLM explain its own SAE? What if we use an additional LLM to explain another LLM's SAE? The author can explore this.\n- Can this method identify some latent units that do not have obvious conceptual information?\n- The method in this paper is very interesting, but the experiments seem to be insufficient, for example, there is a lack of some sensitivity analysis, specific ablation experiments, or more potential downstream applications, etc.", "other_comments_or_suggestions": "Please see weaknesses, I am willing to raise my score if the author can address my concerns convincingly.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sg6egzahIy", "reviewer": "Reviewer_8M51", "summary": "The paper develops an automated pipeline for interpreting latent features identified by sparse autoencoders (SAEs) in LLMs. The authors implement a three-stage approach that first collects latent SAE activations, then generates natural language interpretations using external LLMs, and finally evaluates interpretation quality. Their main contribution is five scoring methods (detection, fuzzing, surprisal, embedding, and intervention) that are more compute-efficient than traditional simulation scoring. Intervention-based evaluation is not commonly used in autointerp pipelines and is a new addition to the mix. It is a causal framework that explains the latents by their effects on model outputs rather than input correlations. The provided evaluation approaches each have specific qualities, and the paper discusses that. The research demonstrates that interpretation quality depends on sampling strategy (stratified sampling across activation distributions produces better interpretations than using only top-activating examples). The practical implication is the most important as it enables scalable assessment of millions of features, providing a foundation for improved model understanding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful comments.\n\n> While being a useful tool for a broad community (both interpretability community and people who want to use the interpretability toolsets), the overall insightful findings and novelty in the method are limited. The automated interpretability pipeline is not new.\n\nAt the time of writing, evaluating SAE interpretability focused on evaluating the top activating examples. We show that this is misleading for multiple reasons. Firstly, this gives an illusion that SAEs are more interpretable and monosemantic than they are (we show this in figure 2), ignoring the full distribution. There was also the impression that the lower activations of SAEs were completely uninterpretable and less valuable, which again we show not to be the case. We think this to be a valuable insight.\n\nOur scoring techniques improved on the state of the art, and are now the standard instead of simulation scoring. We think that our discussion on interpretability scoring is also helpful for the community, and such an analysis is not common. Intervention scoring was a type of scoring that was missing in the current framing of SAE interpretability.\n\nWe would love to be able to produce more rigorous results that would convince the reviewer that the paper strongly merits acceptance. If there are specific examples that the reviewer would like to be done more rigorously we would take those into consideration.", "claims_and_evidence": "Yes.\n\nThe main claims regarding novelty are: \"We introduce five new techniques to score the quality of\ninterpretations that are cheaper to run than the\nprevious state of the art.  One of these techniques,\nintervention scoring, evaluates the interpretability\nof the effects of intervening on a latent, which\nwe find explains latents that are not recalled by\nexisting methods\"\n\nThe paper supports its claim of introducing five more cost-efficient scoring techniques through detailed cost analysis in Table 1, showing methods like fuzzing and detection require 5-30x fewer tokens than simulation scoring. The second claim about intervention scoring is evidenced in Figure 3, which demonstrates a negative correlation between fuzzing and intervention scores, proving that latents scoring poorly on traditional methods can be well-interpreted through their causal effects.", "methods_and_evaluation_criteria": "Yes, the proposed five evaluation methods are quite intuitive and make sense for interpreting latents.", "theoretical_claims": "There are no major theoretical claims or issues.", "experimental_designs_or_analyses": "Yes, I can validate the soundness of the following experiments:\n\n- Cost comparison of scoring methods (Table 1) \n- Correlation analysis between scoring methods (Tables A1 & A2)\n- Example sampling strategy comparison (Figure 2) - Testing how different sampling techniques (top activating examples, random sampling, stratified sampling) affect interpretation quality\n- Intervention scoring analysis (Figure 3) - Demonstrating that some latents with low fuzzing scores have high intervention scores, and comparing trained SAEs against random baselines\n\nThe number of latents used for correlation analysis can be more.", "relation_to_broader_scientific_literature": "Considering that the work is more of an empirical study with practical value, it provides useful tools for the broader research community to interpret activations of various models across different domains.", "essential_references_not_discussed": "Essential references are discussed", "other_strengths_and_weaknesses": "Strengths: \n- The authors provide a useful tool for the community, although the automated interpretability pipeline itself is not novel \n- The novelty lies in the evaluation approaches for the interpretations, which are insightful\n- The intervention-based approach is useful to have in the stack\n\nWeaknesses:\n- While being a useful tool for a broad community (both interpretability community and people who want to use the interpretability toolsets), the overall insightful findings and novelty in the method are limited. The automated interpretability pipeline is not new. \n\nVerdict:\nI consider this work as a valuable project and tool for the community, rather than being scientifically profound. I would require to see more scientific novelty or rigour for a clear accept. Therefore, I consider this work as weak accept.", "other_comments_or_suggestions": "None.", "questions_for_authors": "No question.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8Y9KBHAoI", "reviewer": "Reviewer_jsoS", "summary": "This paper introduces an automated pipeline for interpreting the latent features of sparse autoencoders (SAEs), which decompose large language model (LLM) representations. The authors propose five scoring methods to assess interpretation quality, including detection, fuzzing, surprisal, embedding, and intervention scoring. The framework is tested across different SAE architectures and LLMs, and the results suggest that the proposed scoring methods are computationally cheaper while maintaining strong correlations with human evaluations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this feedback.\n\n> Lack of direct numerical comparison with existing methods\n\nWe compare our scoring techniques with the standard scoring technique at the time of writing, simulation scoring. In Table 1, we compare the costs of both scoring techniques, and we perform human score correlations, lines 322 left. We also perform inter-score correlations,  Tables A1 and A2. \n\nIs there a specific comparison that the reviewer would like to see?\n\n> Following 1, it is unclear how much interpretability quality improves or is sacrificed for scalability.\n\nImprovements in scalability were mostly on the generation of scores, which we discuss in Table 1. This means that the proposed techniques do not sacrifice interpretability quality for scalability. Indeed we believe that the proposed guidelines for generating and evaluating latents improve interpretability quality instead of sacrificing it. Some of our proposed scores correlate well with human given scores.\n\n> Missing the analysis of failure cases. It is important to be aware of the safety boundary of the proposed method in implementation.”\n\nWe currently provide one failure case in Figure A2 of the appendix, where the explainer model incorrectly identifies the explanation, one that is more easily gathered from top activations - as we write this we notice that the caption of the figure is missing, and that it indeed discussed this. We discuss this failure mode in the text. It is hard in general to discuss failure cases in interpretability, and even harder to define safety boundaries. We wonder if the reviewer has specific comments on how to improve on this.", "claims_and_evidence": "The authors claim that the proposed methods provide a more scalable and efficient interpretation method while capturing causal effects through intervention scoring. However, the largest problem is that the paper only shows the results for the proposed method, without a systematic comparison with existing methods that have already been discussed in the related works.", "methods_and_evaluation_criteria": "The proposed scoring methods make sense for the interpretability problem at hand. Again, Although the experiments cover different SAEs, LLMs, and datasets, the absence of a detailed baseline comparison weakens the empirical evaluations.", "theoretical_claims": "The work does not include theoretical guarantees or formal proofs, which is understandable given its empirical focus.", "experimental_designs_or_analyses": "See *Claims and Evidence* and *Methods and Evaluation Criteria.*", "relation_to_broader_scientific_literature": "The paper builds on prior work in LLM interpretability.", "essential_references_not_discussed": "The paper covers the most relevant references but lacks experimental comparisons with them.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-organized and easy to follow.\n2. The proposed method is intuitive and technically sound.\n\n\nWeaknesses:\n1. Lack of direct numerical comparison with existing methods.\n2. Following 1, it is unclear how much interpretability quality improves or is sacrificed for scalability.\n3. Missing the analysis of failure cases. It is important to be aware of the safety boundary of the proposed method in implementation.", "other_comments_or_suggestions": "I strongly recommend the authors provide direct numerical comparisons between new and existing scoring techniques to clarify performance gains. I’m willing to raise my score if such a comparison is provided.", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPpt72jYim", "reviewer": "Reviewer_tbRE", "summary": "The paper presents an automated pipeline for generating and evaluating natural language interpretations of SAE latents in LLMs. The authors introduce five scoring techniques—Detection, Fuzzing, Surprisal, Embedding, and Intervention Scoring—to assess interpretation quality, with Intervention Scoring evaluating the causal effects of latents on model outputs. They test their framework on SAEs trained on two open-weight LLMs and find that different sampling and scoring strategies impact the specificity and generalizability of interpretations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree with this comment by the reviewer. Our generated interpretations only interpret the activations of individual latents, and are far from full explanations of their behaviour and downstream impact. Intervention based methods, like the one we proposed, are the ideal candidates to probe the causal role of latents. We believe that a combination of both is the most informative. \n\n> (2) Why were Surprisal and Embedding scores included despite their weak correlation with human evaluations? Do you see specific scenarios where these scores are more useful than Fuzzing, Detection, or Simulation?\n\nWe propose surprisal as a seemingly natural method to score interpretations, with the hope that it inspires future work in that direction, and that future work is aware of the limitation of this technique. Although we probably won’t use surprisal scoring much in the future, we believe it to be valuable for the community to know that it was tried. \n\nOn the other hand, embedding scoring is much faster and cheaper than all the other techniques, even though such comparisons are less apples-to-apples than the comparison between fuzzing/detection with simulation. This is because, not only are the input tokens lower but the embedding model used can be as small as 700M parameters. This makes for a very compelling reason to use embedding scoring, if one wants to quickly evaluate the interpretability of latents, potentially on the fly while training. \n\nThe fact that latents with low embedding scores are generally low scoring in fuzzing/detection can be used to filter out incorrectly interpreted latents or find latents which are generally not interpretable.", "claims_and_evidence": "Claim: \"The pipeline improves upon previous interpretability work by producing higher-quality interpretations.\" The paper does not conduct rigorous human evaluations of interpretation quality. While the scoring metrics provide an automated way to assess interpretations, they are correlation-based and may not align with actual human judgment. The paper references some small-scale human evaluations but does not systematically compare their method to existing ones using expert annotations.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "None", "experimental_designs_or_analyses": "The experimental design is well-structured.", "relation_to_broader_scientific_literature": "The paper extends prior work on SAE-based feature extraction by scaling interpretations to millions of latents. It refines simulation-based scoring with five new metrics. While the pipeline improves scalability and efficiency, it offers incremental advances rather than fundamentally new interpretability methods.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n(1) The paper successfully scales LLM-based interpretability methods to millions of SAE latents, a good engineering contribution.\n\n(2) The proposed scoring methods, especially Detection and Fuzzing, offer computationally cheaper alternatives to simulation scoring.\n\n(3) code is provided with a detailed readme file.\n\n\n\n\nWeaknesses:\n\n(1) The core methodology builds on existing LLM-based interpretability and SAE approaches, offering mostly incremental improvements.\n\n(2) The evaluation relies heavily on automated metrics without systematic human assessment of interpretation quality.\n\n(3) Surprisal and Embedding scores have low correlation with human judgment, making their usefulness unclear.", "other_comments_or_suggestions": "None", "questions_for_authors": "(1) There is no systematic way to verify correctness, as LLM-generated explanations may sound plausible but not reflect actual latent function.\n\n(2) Why were Surprisal and Embedding scores included despite their weak correlation with human evaluations? Do you see specific scenarios where these scores are more useful than Fuzzing, Detection, or Simulation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents an automated pipeline for generating and evaluating natural language interpretations of SAE latent features in LLMs. The authors introduce five new scoring techniques that provide more computationally efficient alternatives to traditional simulation scoring while maintaining strong correlations with human evaluations. The reviewers acknowledged several strengths of this work, particularly the intervention scoring approach that offers a novel causal perspective on latent interpretability, revealing features that aren't captured by existing methods. The paper makes a valuable practical contribution by scaling interpretability methods to millions of SAE latents, with comprehensive ablation studies across different sampling strategies, model configurations, and evaluation criteria. While reviewers initially raised concerns about the lack of direct numerical comparisons with existing methods and the absence of systematic human evaluations, the authors adequately addressed these issues in their rebuttal by clarifying the comparisons made in Table 1 and explaining the correlations with human scores. All reviewers ultimately provided weak accept recommendations, recognizing that while the core methodology builds on existing LLM-based interpretability approaches, the improvements in scalability, the novel intervention scoring technique, and the practical utility of the pipeline make valuable contributions to the field. Given the paper's technical soundness, practical utility, and the authors' thorough responses to reviewer concerns, I recommend accepting this paper for publication at ICML 2025.
Decision: Accept (poster)