Revising and Falsifying Sparse Autoencoder Feature Explanations
Authors: George Ma
Samuel Pfrommer
Somayeh Sojoudi
Venue: NeurIPS 2025 poster ()
Keywords: Mechanistic Interpretability
Primary area: deep_learning
TL;DR: We developed new methods to refine and falsify sparse autoencoder feature explanations, yielding higher-quality interpretability of large language models.

Abstract: Mechanistic interpretability research seeks to reverse-engineer large language models (LLMs) by uncovering the internal representations of concepts within their activations. Sparse Autoencoders (SAEs) have emerged as a valuable tool for disentangling polysemantic neurons into more monosemantic, interpretable features. However, recent work on automatic explanation generation for these features has faced challenges: explanations tend to be overly broad and fail to take polysemanticity into consideration. This work addresses these limitations by introducing a similarity-based strategy for sourcing close negative sentences that more effectively falsify generated explanations. Additionally, we propose a structured, component-based format for feature explanations and a tree-based, iterative explanation method that refines explanations. We demonstrate that our structured format and tree-based explainer improve explanation quality, while our similarity-based evaluation strategy exposes biases in existing interpretability methods. We also analyze the evolution of feature complexity and polysemanticity across LLM layers, offering new insights into information content within LLMs' residual streams.

Revising and Falsifying Sparse Autoencoder Feature Explanations
George Ma Samuel Pfrommer Somayeh Sojoudi' University of California, Berkeley
Abstract
Mechanistic interpretability research seeks to reverse-engineer large language models (LLMs) by uncovering the internal representations of concepts within their activations. Sparse Autoencoders (SAEs) have emerged as valuable tool for disentangling polysemantic neurons into more monosemantic, interpretable features. However, recent work automatic explanation generation for these features has faced challenges: explanations tend to be overly broad and fail to take polysemanticity into consideration. This work addresses these limitations by intro- ducing a similarity- strategy for sourcing close negative sentences that more effectively falsify generated explanations. Additionally. we propose a structured, component-bar -baeen format for feature explanations and a tree-based, iterative expla- nation method that refines explanations. We demonstrate that our structured format and tree- beed explainer improve explanation quality, while our similarity-baser evaluation strategy exposes biases in existing interpretability methods. We also analyze the evolution of feature complexity and polysemanticity across LLM lay- ers, offering new insights into information content within LLMs' residual streams. Code is available at https:/ / github. wwt. GeorgeMMP /fee/feeeeeeeeeeeeeeeeeeeeeeeeee centuur-iittrp..
Introduction
The recent rise of highly capable large language models (LLMs) has inspired surge of mechanistic interpretability research. One major concern of this area has involved decomposing LLM activations into iumanninterpreeable form [11, 26]. This is complicated by LLM's hypothesized use of superposition to encode a vast range of concepts into a constrained number of multi-layer perceptron (MLP) block neurons [3]. Namely, instead of MLP block neurons each corresponding to a clean concept, they become polysemantic, and activate for range of unrelated inputs [6. 3].
refined analysis tool that partially disentangles the polysemanticity of individual neurons [35, 3, 5, 16). Inspired by the linear representation hypothesis [27]. SAES consist of autoencoder which maps activations into higher-dimensional latent space while minimizing both reconstruction loss and a sparsity penalty [5]. In contrast to individual neurons, features the latent space of an SAE are generally more monosemantic and interpretable [5].
Sparse Autoencoders (SAEs) have enjoyed widespread
One area of LLM autointerpretabiliiy research attempts to explain language model features, such as neurons and SAE latent vector indices, using external L.MSs to summarize patterns in top-activating sentences [1]. These generated explanations often oversimplified and too broad [1, 28]. Consider for example the "not all" neuron L13N1352 discussed in Bills et al. [1]. This neuron activates on the token all only when immediately following not. However, more broad explanation that the neuron always activates on the word not scores well on high-activation sentences which do not contain a falsifying counterexample of not in different context. Various partial remedies this problem
Equal contribution. Correspondence to: George Ma deeorgo_m@bberelle. edu).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Structured explunations
Tree-bused explanution refinement
Similarity-bused test sentence retrieval
This feature activales for these rules:
records by
Candidate explanation
feature activation
Embedding
"activates on": "Words relating to susceptibilityy resistance the context of biological testing and scientific research."; 3
Evaluation and feeabback explanation's score 0.71. The most incorrect record was
Embedding
Embeekling
Embedding
Arg. cos similrrity with top-acttvating sentences
Chain-of prompting
Embeekling
"activates, on": "strength";
'Antibiotic namos.".
Embedding
New candidate explanation
New candidate explanation
New candidate explanation
Rank records
"This feature activates on language related antibiolics."
"Summarize what the neuron looking for."
Random retrieval sentences from dalaset.
previous work
Figure 1: A high-level overview of our experiments for automatic interpretation of SAE features. Left: we represent explanations structured list of dictionaries. Center: we refine explanations using tree-based optimization strategy. Right: we source challenging explanation evaluation sentences using a similarity-barn -asiid---aeddd-- scheme.
have been proposed, but they require serious compromises, For example, Bills al. [1] generates an explanation-dependeer sentences, while Paulo et al. [28] simplifies the problem to that of binary classification. Our work provides method for sourcing semantically similar sentences from fixed dataset for the purpose of falsifying explanations.
Explanations in previous interpretability research primarily consist of brief summary statements of patterns generated in one-shot manner [1]. This lack of structure often obscures interesting aspects of SAE features, such as polysemanticity and the relative strengths of different feature components. Bills et al. [1] explored some strategies for explanation revision, but their unreleased implementation sentences. We introduce two innovations in this regard: 1) tree- tree-based iterative method for generating explanations, and 2) structured representation for feature explanations which elucidates changes SAE feature composition over the layers of an
required generating synthetic explanation-dependeee
LLM.
Contributions. Our main contributions are listed below and summarized Figure 1.
1. We provide method for sourcing "close negatives" top-activating sentences in the dataset, and show that these more effectively falsify explanations than the random sentences used in prior work.
2. We introduce structured format for feature activations list of monosemantic explana- tions.
3. We develop tree-based explanation method that iteratively refines its explanations through evaluation and feedback.
4. Through empirical analysis, we show that both the structured explanation format and the tree- based explainer improve the quality of feature explanations. Semantically similar negatives more effectively falsify explanations and reveal the recall bias current interpretability methods. We further investigate how feature complexity and polysemanticity evolve across LLM layers.
Related work
Interpreting SAE activations. There has been significant recent research interest in explaining and simulating SAE activations. The seminal work of Bills al. [1] introduced language model

---

explanation generation for MLP neurons in LLMs. Namely, the authors constructed an automated interpretability pipeline consisting of an explainer LLM which generated short natural-languael summary of feature given its top-activating sentences, and a simulator L.IM which predicts token- wise activations given the explanation. In Singh et al. [33], this idea is adapted to analyze text-to-scalar functions more generally. Recent approaches explore alternative approaches of simulating and scoring explanations by simplifying activation prediction to a classification problem [28]. Foote et al. [7] represents SAE features graph by identifying tokens which cause subsequent activations.
Optimizing over strings. A wide variety of real-world engincering problems involve iteratively optimizing over a piece of text subject some reward signal. The LLM interpretability work of Bills et al. [1] pursued a simple prompting strategy of asking the explainer model to revise its explanation in light of novel activation data. In Pryzant et al. [30]. the optimization problem explored more generally by mimicking the gradient descent with textual prompts. LLM pipelines have also shown benefit from a degree of automatic prompt generation and optimization [17, 18]. By using an evolutionary search strategy, the FunSearch source code generation tool was able to discover novel and efficient computing algorithms [32]. Finally, a recent Tree of Attacks with Pruning (TAP) search scheme proved highly efficient for generating LLM jailbreaking strings [21] and has been successfully adapted other tasks such as prompt injection [29]. We use a variant of TAP for our use case due to its simplicity and effectiveness. While an extensive comparison of various optimization schemes would be valuable future research, TAP is sufficient for the purpose of this work: extracting insights about SAE features in an automated way.
bstraccions within LLMs. While various efforts have been made to analyze the abstraction of LLM concepts as a function of layer, this has been largely underexplored in an SAE context. One early work on the abstraction of features within transformers found that deeper layers within BERT recovered progressively more complex features in classical NLP pipeline [36]. Another leveraged human annotations texts which trigger attention keys and found that deeper layers generally feature more semanticity than early layers. In Jin et al. [15], linear probing layers demonstrate that certain guestion-answen et al. [24] finds the SAEs deeper LLM layers are better able disentangle the various meanings of polysemous words.
LLM activations at various skills are localized to particular depths. Minegishi
3 Problem formulation
Section 3.1 introduces the notation and definitions used in this paper, and Section 3.2 formalizes the problem of learning sxplanation-condition features.
3.1 Notation
We denote the discrete space of tokens by T and the associated continuous embedding space by R²A, where is the transformer hidden dimension. We denote the concatenation of vectors and y using square brackets: [x. y]. The all-one vector of length is denoted by 1,, For vector x [x] I2,,.., , [n] and integer E nn). we let x., E ng denote the vector produced by dropping the ith element.
3.2 Feature learning
Let a feature be a function f: T R7 which maps a token sequence E T a vector feature expressions f(t) E RT. The space of all features is denoted by F.
Consider particular ground-truth feature f* E F. perhaps consisting of the activations of an SAE latent index. We can associate f' with dataset of associated feature expressions
Dy- (ta))
Now consider the problem of learning an explanation-conddiionee feature go: T - Ry which reproduces the feature patterns of f in Dy., We let the explanations 0 fall in the space structured natural- descriptions E T and leverage a language model simulate the predictions of go for particular token sequence E 7 [1]. Precisely characterizing "natural- lanuuall" text is
3

---

challenging. However, as we ultimately use language models to search over explanations, we observe that the constraint θ E Th is satisfied in practice.
When evaluating the performance of go in reproducing the activations of f it is computationally infeasible to evaluate over all of D--. We thus follow the practice of Bills et al. [1] in subsampling D-- for evaluation.
We first define the subset J. C Dy top-activating sentences as a random sample of sentences from the top quantile of 5-- ordered by (t,) We then define a set of complementary sentences D C Dj- \ Dj.. chosen according to a sampling strategy ss resulting in a cardinality (Dys = Dy-l- These sentences are intended as "negatives" that falsify an overly-broad explanation of the top activations in Dj. [1]. We empirically compare sampling strategy choices in Section 5.1.
The dataset loss is then defined using the standard Pearson correlation coefficient metric defined in Bills et al. [1]:
((0) corr (f
11,EZ o0 oy.... (go (g)((tt),,oo(tt.)) (tt.). o...e.7...... )
(I)
where with some abuse of notation we let D... be the sequence of data indices selected in the construction of Dj. and Des.
Method
Section 4.1 briefly summarizes the simulation method from Bills al. [1] for completeness. In Section 4.2, we provide details regarding our explanation generation schemes. Section 4.3 motivates and defines our structured schema for feature explanations. Finally, Section 4.4 discusses how we source "close countercxamples" for -ottiiiivt sentences from our dataset.
4.1 Feature simulation
Consider an explanation A for feature, and recall that the associated feature simulator go T R maps sequence of tokens to a sequence of nonnegative reals. Our simulation approach is minimally modified from Bills al. [1]. Namely, we provide simulating LLM the explanation and formatted list of tokens t E ' where each token is listed on a new line followed by a tab and an unknown token. We do one inference pass per sentence and compute the expected feature expression for each token by examining the negative log probabilities on the unknown tokens. Unfortunately, the original implementation in Bills al. [1] relied on OpenAI API calls returning the top-k log probabilities for tokens in the input prompt. This has been deprecated and to the best of our knowledge is not available in other cloud providers. We thus perform simulation locally using the gemma-2-271 ---2bbiit model with bilevel key-value caching scheme for efficiency (Appendix A.2). For simulating structured explanations, we perform multiple passes of the above simulation scheme, one per explanation component. Further details are deferred to Section 4.3.
4.2 Explanation generation
We outline two different ways of generating explanations: one-shot method from Bills et al. [1], and novel tree-based method inspired by Mehrotra et al. [21]. We also experiment with including holistic activations to improve explanation quality.
One-shot explanation generation. The one-shot explainer includes system prompt which explains the explanation schema and the activation record format [1]. This then followed by series of few-shot examples, each containing sequence of activation records as a user chat message followed by the associated structured explanation in the assistant response. The final message to the model consists of top-activating records Dj. for the feature of interest. Each activation record is formatted as in Bills et al. [1], with activations for each token normalized and rounded to be integers between 0 and 5.²
Bills et al. [1] formatted activations to be between 0 and 10. We found that some tokenizers subdivided 10 into two tokens, and thus adjusted the range maximum 5.

---

brsee explanation generation. We now introduce a novel explanation procedure based on the Tree of Attacks with Pruning (TAP) jailbreaking technique [21]. We initialize with w root leaf nodes generated via one-shot explanation, and iteratively perform the following steps d N times:
1. Evaluation and feedback. Simulate the activations for each explanation and score them against the ground-truth activations via (1). Construct feedback message consisting of the correlation score as well as the correct and simulated feature expressions for the porfor-peroorning sentence in the training data.
2. Branching. For each leaf node, generate b child explanations via thain-of-thougee reasoning, providing the feedback message as well as entire conversation history for the ancestors of the leaf node.
3. Pruning. Retain the w leaf nodes with the highest training dataset score for the next layer of the tree.
The algorithm terminated after d iterations or after certain score threshold is met. All the explanations produced at each layer of the tree are then scored using validation dataset, and the best-performing explanation is returned. The pseudocode of the tree-based explainer is shown in Algorithm
Algorithm 1 Tree-Based Explainer for SAE Feature Interpretation
Require: Training and validation records for a single SAE feature, each containing top-activating and complementary sentences with ground-truth activations
Ensure: Natural language explanation for the target SAE feature
Initialize: Prompt the one-shot explainer (as in Bills al. [1]) times to generate initial explanations as leaf nodes 2: for i 1.2....,dd do
Evaluate current leaf nodes
3:
for each leaf node explanation 0, do
4: Provide 02 and validation sentences to simulator L.M Obtain simulated activations å, from the simulator Compute correlation score 5; corr(á,, a5 (rue))
5:
6:
7:
Construct feedback message F; including."
(a) correlation score 8;
(b) simulated activations å (c) ground- truth activations a!th for the lowest-performing sentence
8:
end for
Generate improved explanations
9:
for each leaf node explanation 0j do
10:
Provide F, and conversation history of ancestor nodes the LLM Use ofain-of-thoughh prompting to generate b improved explanations Add the new explanations as child nodes in the tree
11:
13:
end for
Evaluate and select new leaf nodes
14: for each new leaf node explanation do
Simulate and score as above to obtain 8j
16:
end for
Retain the top w leaf nodes with highest validation scores
18: if highest validation score >> 7 then
break 20: end if
21:
end for
22:
return The explanation with the highest validation score across all iterations
Holistic features. We introduce holistic features as a supplementary signal which indicates what
tokens causally influence SAE activations. Consider arbitrary but particular feature f T - 12.
5

---

{ "activates_on"; string "strength": [o, 1, 2. 3, 4, 5]
]
Figure 2: The schema for our structured explanations.
We define the associated holistic f f : IR as:
f(t):
Holistic features measure how much a particular token impacts the total expression of feature across all indices. Consider feature which activates on the token want but only when immediately following don't 't. The associated holistic feature would positive for both tokens, as dropping either of them from the sequence reduces the summed expression of the original feature. In our work, we experiment with supplementing feature records with holistic activations. Some examples of holistic activations are shown in Appendix B.
4.3 Structured explanations
Feature explanations from previous works typically are limited to short strings. For example, the explanation for L30N2902 from Bills et al. [1] "words related to the concept of being a significant integral part of something." However, closer examination reveals that this explanation fails to capture activations on words relating to depth or recesses. This suggests that short, unstructured explanations obscure the polysemantic attributes of feature.
We address this with our structured explanation scheme, summarized in Figure 2. An explanation consists of list of JSON objects which we call explanation components. Each component contains two fields: a string description of the activations and an integer attribute for the strength of the activation between 0 and 5. We prompt explainer LLM such that each component corresponds to a monosemantic concept; intuitively, feature that requires many explanation components to achieve good performance is polysemantic.
When simulating structured explanation, we predict activations for each component individually using the activated on field and the approach in Section 4.1. We then scale each component's activations proportionally the strength field and combine by taking the maximum activation across components for each token. This ensures that the simulated feature activations are invariant to permutations the structured explanation list.
4.4 Complementary sentence sourcing
We propose four methods for selecting complementary sentences Djs in the evaluation dataset: (1) the random strategy, where sentences randomly sampled from the entire dataset Dy- DJ.; (2) the simile lar strategy, where sentences with the highest semantic similarity to the top-activating sentences Dj- are selected: and (3) non-activating variants of these two strategies, where only sentences with no ground-truth feature activation (i.e., max; f ft(t); = 0) are considered.
Random sentences. Following Bills et al. [1], randomly sample an equal number of sentences the top-activating sentences from the entire dataset. The combined set of top-activating and randomly selected sentences is then used to evaluate feature explanations.
Random non-activating sentences. In this strategy, we randomly sample sentences from the dataset that have no ground-truth feature activation. This allows us to assess the ability of different complementary sentence selection strategies in finding "close counterexamples" by analyzing their false positive rate-definn defined as the proportion of positive elements the explanation-condiiion feature activation ge(t), where E D3s.

---

Current feature explanation methods are often biased toward recall: broad explanation can activate on many relevant tokens while still achieving a good evaluation score [4]. By sourcing "close coun- terexamples" to the top-activating sentences, we can better assess the precision of the explanations.
Similarity- based sentences. Instead of randomly selecting from the entire dataset, we sample sentences with the highest semantic similarity to the top-activating sentences. Sentence similarity is measured using cosine similarity between sentence embeddings produced by pre-trained Sentence Transformer [31]. This method involves two steps:
1. Compute the embeddings of all sentences in the dataset using pre-trained Sentence Transformer.
2. For each SAE feature, we identify a fixed set of top-activating sentences (400 in our experiments). We then select the sentences with the highest average cosine similarity this set and designate them as the complementary sentences for evaluation. These complementary sentences are pre-computed and cached prior to experimentation.
As we demonstrate in Section 5.1, evaluating feature explanations on these semantically similar sentences reveals substantially higher number of false positives during the feature simulation step (Section 4.1). Specifically, the feature simulator go often predicts positive activations (t,) 1 where the ground truth activation (t, is zero. Since current interpetttttt approaches are heavily biased toward recall, evaluation against these similarity- offers more reliable estimate of explanation precision, helping to expose overly broad or imprecise feature descriptions.
complementary sentences
Similarity-basee non-activating sentences. This method follows the similarity similarity-based -based strategy but filters for sentences with no ground-truth feature activation. This ensures a fair comparison between the false positive rates of similarity- and random selection methods. As shown in Section 5. similarity- complementary sentences tend to yield higher false positive rates, leading to lower correlation scores for feature explanations. This result highlights the recall bias in current feature explanation methods and provides a more precise measure of their precision.
5 Experiments
In Section 1. we compare various methods for sourcing complementary sentences. Section 5.2 documents our improvements to the explanation generation process. Finally, Section 5.3 analyzes the impact of our structured explanations on the composition of SAE features function of layer.
Common setup. All experiments are conducted using an uncopyrighted subset of the Pile [8, 25]. We conduct experiments subset of 100,000 sentences and chunk them into sequences with 32 tokens. For all experiments, we use the open-source Llama 4 Scout to generate explanations [22]. Our subject language models genma 2-9b, 1lama-3. 1-8b, and gppt22smmal. We leveraged pretrained SAEs of comparable widths, using the 16k Gemma scope SAFs [19] and 32k Ilama scope and GPT-2 SAFS [9, 13). We use the first 50 SAF features of each layer in our experiments.
5.1
Complementary sentence sourcing
In this section, we evaluate the four complementary sentence sourcing strategies introduced in Section 4.4. Following the setup of Bills et al. [1], we provide the explainer LLM with 10 top- activating sentences as the training dataset for each feature and prompt it to generate an explanation. We then use a simulator LLM to predict feature activations on test dataset, which consists of 10 top-activating sentences and 10 complementary sentences. Finally, assess the quality of the explanations by measuring (1) the false positive rate of simulated activations on the complementary sentences and (2) the correlation between ground-truth and simulated activations. We experiment with gemma-2-222 2-9b as our subject model.
Figure 3 shows the number of false positives per sentence of simulated feature activations on complementary sentences using structured explanations. For a fair comparison, we highlight all non-activating complementary sentence sources, they contain ground-truth activations (i.c.,
7

---

true positives or false negatives), We also plot the complementary sources without the no- activation restriction dashed lines; however, comparisons involving these methods are not entirely fair since they differ in the number of true positives as well.
False positives per sentence
Random
False positives per sentence
Random
2.5
Random (non-activating)
2.0
Random (non activating)
Similar
Similar
2.0
Similar (non-activating)
1.5
Similar (non-activatee
1.0
1.0
0.5
40
Layer
Layer
Figure 3: False positives per sentence of simulated activations on complementary sentences for the one-shot explainer (left) and the tree-based explainer (right), averaged over 50 features per layer. "Non-activating" indicates the sentences have no ground-trun activation.
As observed in Figure 3, for both the one-shot and tree-based explainer, similarity- complemen- tary sentence sourcing methods result in higher false positive rates than random selection methods. This indicates that similarity-basee methods are more effective at identifying "close counterexamples that expose the limitations of feature explanations in terms of precision. As noted by Caden Juang et al. [4], existing feature explanation generation methods tend to be biased toward recall. By evaluating explanations similarity-based complementary sentences, achieve more precise assessment of their accuracy.
We show some examples of similarity-based complementary sentences and their corresponding simulated activations Appendix
5.2 Explanation generation
An overview of our explanation generation results is available in Figure 4, with the associated tabular results Table As in Section 5.1, we use 10 top-activating sentences the training set for generating the explanations, with the optional inclusion of 10 additional complementary sentences as training negatives. For the evaluation dataset, we use top-activating sentences and 10 complementary sentences. We take similarity- -based con-activating sentences as the complementary set.
0.6
Layer score
0.2
Unstractusel (boanlin))
Structured (ours)
0.1
Oneschot (beseline)
Structared srain neg Structared holistic:
Tree (ours)
Gemma 9b
Llana 3.1 8b
GPT-2 Small
Figure 4: Explanation strategy comparison. Bars denote 90% confidence intervals.
We note that tree-based generation consistently outperforms one-shot generation.

---

For the unstructured explanation case, we see relative improvements of 19.8% for Gemma, 21.6% for Llama, and 16.4% for GPT-2 when using the tree explainer. In the one-shot explainer case, we also see an improvement from structured explanations, with relative improvements of 5.8% for Gemma, 8.4% for Ilama, and 9.2% for GPT-2. When employing tree-based generation, structured explanations no longer provide benefit. We hypothesize that this is attributable to tree-based generation' 's ability to iteratively pack polysemantic meanings into a single explanation string.
We report negative results for both the inclusion of holistic activations and training negatives in the explanation process. Figure shows that including holistic activations and training negatives does not improve the performance for either the one-shot or tree explainer in any of our tested language models. Taken together, these results suggest that LLMs still struggle generate feature explanations which account for prior context.
5.3 Analysis structured explanations
We analyze feature complexity and polysemanticity for eight evenly-spaced layers within each model, including the first and final layers.
Feature complexity. We use an LLM judge to evaluate the complexity generated structured expla- nations from the tree explainer [38]. We provide the judge with few-shot examples of explanations and the associated complexity score, ranging from 0 to 5. Lower complexity scores correspond to simple token-specine features (e.g.. "the word "instruments") and higher complexity scores assigned to more abstract features (c.g., "expressions of skepticism"), Our complexity judge imple- mentation follows the simulation approach detailed in Section 4. and uses the gemma-2-27b-- model. We compute the simple average of the complexity scores across all explanation components for particular feature.
Feature polysemanticity. For each feature, we run the tree explainer with rule caps ranging from to Each generated explanation is scored against the evaluation sentences, and we record the smallest rule cap for which 90% of the maximum score is achieved over all five settings. This intuitively provides a measure of feature polysemanticity.
Results. Figure 5 shows the average explanation complexity and polysemanticity of structured explanations generated by the tree explainer. We find that the Gemma and Llama 3. models peak in complexity at the middle layers, while GPT-2's complexity remains relatively stable across layers. Gemma 2 and Llama 3.1 also exhibit similar increasing pattern the polysemanticity figure, rising from average polysemanticity of 1.6 and 1.5 at the first layer, to 2.0 and 2. at the final layer. In contrast, GPT-2 has consistently less polysemantic features.
Mean rule count for 90% max score
Cemuna 2 9b
Gemma 2 9b
Average complexity
2.0
Llama 8b
2.5
Llama 3.1 8b
GPT-2 Small
GPT-2 Small
1.8
2.0
1.6
1.4
60
100
20
60
100
Layer (%)
Layer (%)
Figure 5: Lines denote mean values, and shaded regions denote 80% confidence intervals. The horizontal axis corresponds to the analyzed layer as proportion of the total layer count, which varies between models. Left. The average explanation complexity for the tree explainer. Right. The polysemanticity of structured explanations generated by the tree explainer.

---

6 Limitations
While our methods improve the precision and falsifiability of SAE feature explanations, several limitations remain. First, the iterative tree-based explanation generation process is computationally expensive, taking approximately 1.5 minutes per feature on our hardware, which presents challenges for large-scale analysis.* Second, our approach relies primarily on top-activating records, which may overlook certain forms of pyssemmnniiity--artiieene occur in less frequent lower-ranker contexts. Third, with all SAE-based interpretability methods, the resulting feature decomposition reflects the representational structure of activations rather than the true underlying computational mechanisms of the model. Consequently, while improved explanations enhance the interpretability of individual features, they do not in themselves guarantee full understanding of model reasoning behavior. Finally, our analyses in Section 5 focus on layer-wise and model-size trends that provide empirical insights into representational structure, but they should be viewed as preliminary steps toward mechanistic understanding rather than conclusive explanations, We hope that future work will build on these results to establish stronger connections between feature-level interpretability and higher-level model behaviors.
when important activation patterns
Conclusion
This work addresses key challenges in the automatic interpretability of Sparse Autoencoder (SAE) features in large language models. We introduce a method for sourcing semantically similar counterex- amples that more effectively falsify overly broad explanations, a structured format for representing feature activations, and a tree-based iterative explainer for refining explanations through evaluation and feedback. Our experiments show that these approaches improve explanation precision, reveal the recall bias in existing methods, and provide new insights into how feature complexity and polyseman- ticity evolve across model layers. Together, these contributions offer a more rigorous and falsifiable framework for LLM interpretability.
Acknowledgement
This research was supported by the U.S. Army Research Laboratory and the U.S. Army Research Office under Grant W91 1N11222112,,, the Office of Naval Research, and the National Science Foundation. This work used Jetstream2 [12, 2] at Indiana University through allocation CIS240843 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #21 138899.
References
[1] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. OpenAl Blog, May, 2023.
[2] Timothy J. Boerner, Stephen Deems, Thomas R. Furlani, Shelley L. Knuth, and John Towns. ACCESS: Advancing Innovation: NSF's Advanced Cyberinfrasttuucuree Coordination Ecosys- tem: Services & Support. In Practice and Experience in Advanced Research Computing 2023:
Computing for the Common Good, PEARC '23, pages 173-176, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 978 150099995... doi: 10. 00.1145/3569951.55500 33597559.
URL https: org
//gg///..00 10. 11115/156999..1111 3597559.
[3] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, 7ac Hatfield-Dodds, Alex Tamkin, Karina Nguyen. Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.
Our experiments conducted on 40 gigabyte A100 GPU instances with 32 CPU cores each.
10

---

[4] Gonçalo Paulo Caden Juang, Jacob Drori, and Nora Belrose. Open Source Automated Inter- pretability for Sparse Autoencoder Features, EleutherAl Blog, July. 30, 2024.
[5] Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey, Sparse Au- toencoders Find Highly Interpretable Features in Language Models. In The Twelfth International Conference on Learning Representations, 2023.
[6] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Haatfeedd-oddds Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy Models of Superposition. arXiv preprint arXiv:2209. 10652, 2022.
[7] Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez. Neuron to Graph: Interpreting Language Model Neurons at Scale. arXiv preprint arXiv:2305. 19911, 2023,
[8] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv preprini urXiv:2101.0002: 2020.
[9] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2024.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-RI: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv preprint arXiv:2501. /2948, 2025.
[11] Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaci Kheirkhah, Qinyi Sun, Will Hath- away, Neel Nanda, and Dimitris Bertsimas. Universal Neurons in GPT2 Language Models. Transactions Machine Learming Research, 2024.
[12] David Y. Hancock, Jeremy Fischer, John Michael Lowe, Winona Snapp-Childs, Marlon Pierce, Suresh Marru, J. Eric Coulter, Matthew Vaughn, Brian Beck, Nirav Merchant, Edwin Skidmore, and Gwen Jacobs. Jetstream2: Accelerating cloud computing via Jetstream. In Practice and Experience in Advanced Research Computing 2021: Evolution Across All Dimensions. PEARC '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450382922. wrtg.....1.//333//..0.0/1 .344377559..66565...
doi: 10.1 114453443335.34455566
URL https:tp//.
[13] Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu. Qipeng Guo, Xuanjing Huang, Zuxuan Wu, et al. Llama Scope: Extracting Millions of Features from Llama-3.1 --1BBB with Sparse Autoencoders. arXiv preprint arXiv:2410.20526, 2024.
[14] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023.
[15] Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng. Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, et al. Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers? In Proceedings of the 31st International Conference on Computational Linguistics, 2025.
[16] Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks. arXiv preprint arXiv:2411.18895 2024.
[17] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-Senn SaansstrtteSSarc--eeee Seade----eee-----eeettt Composing Retrieval and Language Models for Knowledge-Intensive NIP. arXiv preprint arXiv:22 14024, 2022.
[18] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. In The Twelfih International Conference on Learning Representations, 2024.

---

[19] Tom Lieberum. Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once Gemma 2. In Proceedings of the 7th BlackboxNI.P Workshop: Analyzing and Interpreting Neural Networks for NIP,, 2024.
[20] Google LLC. Gemma Terms of Use. https: bete:://aaggoool google.co dev/ httr///emm/ttemss.. 2025. Last modified March 24, 2025.
[21] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, In Advances in Neural Information Processing Systems, 2024.
[22] Meta. The Llama 4 herd: The beginning of a new era of natively multimodal Al innovation. https://ai meta. cww/blogmmlaateeeeeccoeoooo
April 2025.
[23] Inc. Meta Platforms. Llama Community License Agreement. https: //git//ggtttt com/ 2025, Version effec-
meta- http///aammmdddde////////aaaaaaa tive April 5, 2025.
[24] Gouki Minegishi, Hiroki Furuta, Yusuke Iwasawa, and Yutaka Matsuo. Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words. In The Thirteenth International Conference on Learning Representations, 2025.
[25] Monology. Pile Uncopyrighted: A Copyright-Filtered Subset of The Pile. https:// huggingface. wwwp/naaa....caataaeeeeeeeeeeeee co//attasees//ooooooggoo///////e http:pplll-uooorigghteeeee 2023, Dataset retrieved 2025-05-13.
[26] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits. Distill, 5(3):e00024-001, 2020.
[27] Kiho Park, Yo Joong Choe, and Victor Veitch. The Linear Representation Hypothesis and the Geometry of Large Language Models. In Forty-first International Conference on Machine Learning, 2023.
[28] Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically Interpreting Millions of Features in Large Language Models. In Forty-second International Conference on Machine Learning, 2024.
[29] Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somayeh Sojoudi. Ranking Manipulation for Conversational Search Engines. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, 2024.
[30] Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng, Automatic Prompt Optimization with "Gradient Descent" and Beam Search. In Conference on Empirical Methods in Natural Language Processing, 2023.
[31] Nils Reimers and Iryna Gurevych. Sentence-BERG Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Annual Meeting of the Association for Computational Linguistics, 1 2019.
[32] Bernardino Romera-Paredess Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 -75, 2024.
[33] Chandan Singh, Aliyah R Hsu, Richard Antonello, Shailee Jain, Alexander G Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models. In XAI in Action: Past, Present, and Future Applications, 2023.
[34] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open Models Based on Gemini Research and Technology. arXiv preprint arXiv:2403.0822 08895, 2024.
12

---

[35] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread, 2024.
[36] lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT Rediscovers the Classical NLP Pipeline. In Annual Meeting of the Association for Computational Linguistics, 2019.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anm Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302. 13971, 2023.
[38] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as- audee with MT Br-ch and Chatbot Arena, In Advances Neural Information Processing Systems, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The body of our paper expands experimentally on the main claims made in the abstract and the contributions section of the introduction. Our results are detailed in Section 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have an explicit limitations section in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification: our paper does not make theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We include all code for reproducing our experiments in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We include all code for reproducing experiments in the supplemental
material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
cetpoomisssoooloiiiiooooo (elollicy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurlPS code and data submission guidelines (https: / //tppp.ccppbllliee//etp/ wwtppcccccc//ccccccccccccccccltt tttp///w//////eee//ooe/coe/.se /eeeee/ComSubbi SareeeSubmisssioPP..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Methodological details regarding explanation generation, simulation, and complexity analysis are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report error bars and confidence intervals where appropriate (e.g. confi- dence intervals Figure with explanations in the text).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss runtime and compute resources in Section 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the ines?
NeurlPS Code of Ethics https: : hww.rpuu..../// cc.
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics and believe that our paper is in compli-
ance.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impact of our work in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any data or models which have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite both the original Pile dataset and a community- sourred version which removes all copyrighted content. We employ the latter in our experiments to avoid license violations. We discuss licenses further in Appendix E.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasats has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research studies L.LMs but we did not use LLMs to conduct the research itself.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https htop:://neuute /ttp//ripp..c/cofeeeeeeeeeeeeeee cc/conffreecce//222//LL.. for what should or should not be described.
20

---

A Explanation, simulation, and complexity analysis
This section documents the methodological details regarding our explanation, simulation, and com- plexity analysis.
A.1
Explanation
Hyperparameters. For the one-shot explainer, we use a temperature and a top-p value of for explanation generation, with maximum of 10 rules (i.e., components in the structured explanation). For the tree-based explainer, we set the temperature to 1.2 and top-p to I, with a maximum The tree is initialized with 3 root nodes, maximum depth of 2, and branching factor of meaning each node generates candidate explanations after evaluation and feedback. The width set to 2, retaining the top 2 scoring explanations at each iteration and discarding the rest.
rules.
Prompts. For both the unstructured and structured explainers, the explanation prompt consists of three components: (I) system prompt (provided system messages), (2) few-shot examples (provided as user messages), and (3) their corresponding explanations (provided as assistant messages). The system prompt for the unstructured explainer shown in Figure 6.
We re studying neurons in neural network. Each neuron looks for some particular thing in short document. Look at the parts of the document the neuron activates for and summarize in single sentence what the neuron is looking for. Don't t list examples of words.
The activation format is tokenstab Activation values range from to 5. A neuron finding what it's looking for is represented by non-zero activation value. The higher the activation value, the stronger the match.
Activation records consist of tuo parts: activating tokens and activation-causing tokens. Activating tokens the tokens that the feature activates on. Activation-causing tokens are the tokens that cause the feature activate later activating tokens.
Figure 6: The system prompt for the unstructured explainer. The texts in blue are optional and only included when using the holistic activations introduced in Section 4.2.
In Figure 6, the text highlighted in blue is included only when incorporating holistic activations, as described in Section 4.2. Similarly, the system prompt for the structured explainer shown in Figure 7, where the blue text indicates content specific to holistic activations. Additionally, the purple text in Figure 7 appears only when explicitly setting a maximum number explanation components for the structured explainer.
21

---

We re studying features in a neural network. Each feature looks for specific patterns text. Analyze the parts of the text where the feature activates and explain its behavior in a structured format.
For each feature, provide list of rules, where each rule consists of two fields:
1. 'activates_on (string): The specific tokens on which the activation
occurs. This must be string NOT list of strings.
2. 'strength' (int): The strength of the activation, from to 5. Only put single integer here, additional text.
Each rule should consist of single inta-iineepretablee concept. Do not try to pack completely unrelated activating tokens into the same rule. For example, if the feature activates the word 'stop' ' and also the word 'cookie', you should put them in different rules.
But sufficiently similar or conceptually related activating tokens can be grouped together the same rule. For example, if the feature activates on the word 'car' and also on the word 'truck', you should put them in the same rule.
The activation format tokenstabacttvvatonn Values range from 5. Non-zero activations indicate the feature found what it's looking for. Higher values indicate stronger matches.
Activation records consist two parts: activating tokens and activation-causing tokens. Activating tokens are the tokens that the feature activates on. Ierivation-causing tokens are the tokens that cause the feature to activate later activating tokens.
Try to keep the 'activates_on field short. Also keep the list of rules short as possible. Only add rules to the list if they are really necessary; i.e., only add rule if the feature activates on it and it's not already in the list.
The strict maximum number of rules is (rule_cap}. Do not generate more than this number of rules. You should not try to fill up the rule cap, only add rules if they are actually necessary and try to keep the list of rules as short possible.
Format your response as a JSON list of dictionaries with 'activates_on' and 'strength fields.
The system prompt for the structured explainer. The texts in blue are optional and only included when using the holistic activations introduced in Section 4.2. The texts in purple are only included when the maximum number of rules rule_cap cap explicitly set.
Figure 7:
22

---

RECORD START
<start>
Activating tokens
(tokens [o]]
unknown
<start>
(tokens [1]}
unknown
{tokens [0]} } unknown
{tokens [1]}
unknown
(tokens [n]]
unknown
<end>
(tokens [i]}
{activations [i]} (activations
{tokens [i+1]}
(tokens [n]] (activations [n]]
<end>
Activation-causing tokens
<start>
(tokens [0]}
unknown
{tokens [1]} unknown
(tokens [i]]
ielistic_activatinn [i]}
(tokens [i+1]}
(bolistic_activationn [i+1]]
(tokens [n]}
aholistic_attivation [n]]
<end>
RECORD END
(b)
Figure 8: Formatting of records for simulation and explanation generation. Figure 8a shows the formatting of few-shot example record. For explanation, all few-shot examples' activations are fully shown. For simulation, the first tokens' activations et al. [1]), where i an integer between 0 and n. The few-shot example formatting in Figure 8a, except without any activation masking. The texts blue are optional and only included when using the holistic activations introduced in Section 4.2. The format for the sequence being simulated is show in Figure 8b: all tokens are masked unknown and the log probabilities for each score integer 0, 1,...,5 5 is used to compute the expected score.
masked with unknown (as in Bills
23

---

Next, we provide the few-shot examples along with their corresponding unstructured and structured explanations. These records are ultimately formatted as Figure when included in the context- however, here we highlight activations in more human-readable way. In these examples, both regular activations and holistic activations are highlighted, with darker background colors indicating higher activation values. For readability, some line breaks from the original records have been omitted. The same set examples is used to prompt all explainers.
Example
Regular activations:
javascript to provide you with a alpositive online shopping experience. To enable javascript
and 3 and Are Negative for Claudin Invasive apo
Holistic activations:
javascript to provide you with positive online shopping experience. To enable javascript and 3 and Are Negative for Claudin 4. Invasive apo Unstructured explanation: the words 'positive' and 'negative' Structured explanation:
{
"The words 'positive' and 'negative', in upper and lovercase.", "strength": 3
"activates_on"
}.
]
Example 2
Regular activations:
<tile>>nstallltionn of Less</tilee <para>Install Less
<iile>CodeMirro: HTML mixed mode</titl>> <meta
Holistic activations:
Iileelnstallatton of Less</till>> <para>Install Less MoleeCodeiiroo: HTML mixed mode</title> <meta
Unstructured explanation: the HTML tag component '</' following an opening 'tiilee' tag
Stractured explanation:
{ "activates_on":
"The HTML tag component '</' when following + opening '<title' tag.", "strength": 4
]
24

---

Example 3
Regular activations:
third-generation cephalosporin- resistant Escherichia coli from broilers, swine, NCLLS standard for susceptibility testing of yeasts?]. The Etest and Holistic activations:
seid--generation cephalosporin ressstaan Escherichia coli from broilers, swine, NCLLS standard for susceptibility testing of yeasts?]. The Etest and Unstructured explanation: language related to biological resistance
Structured explanation:
{ "activates_on". "Words relating to susceptibility and resistance in the context of biological testing and + scientific research.",,
"strength": 3
},
{
"activates_on" _n": "Antibiotic names.", "strength"; 4
}
]
Example 4
Regular activations:
hage migration inhibitory factor in allergic rhinitis: its identification in cosinophilsat
prevalence of asthma and allergic disorders was assessed in 9-11 year-
Holistic activations:
hage migration inhibitory factor in allergic rhinitis: its identification in eosinophilsat
prevalence of asthma and allergic disorders was assessed in 9-11 year-
Unstructured explanation: language related diseases, disorders and aller-
gies
Structured explanation:
{
"activates_on": "The word 'allergic'."
"strength": 3
},
"activates_on" on": "Diseases and disorders, only if following the + word 'allergic'.",
"strength": 4
]
25

---

Example 5
Regular activations:
communication system in which Av apparatus, such video tape recorder
(VTR
camera View at the same time? have requirement show both rear
Holistic activations:
communication system in which Av apparatus, such as a video tape recorder (VTR
camera view at the same time? have requirement to show both rear
Unstructured explanation: language related to views, videos, and recording
Structured explanation:
{
"activates_on" on": "Words generally related to views, videos, and recording.", "strength": 3
]
Following these few-shot examples, the explainer receives feature activation records and is prompted generate explanation based them.
For the tree explainer, each iteration involves evaluating the explanations generated at each tree node, providing feedback, and prompting the node to refine its explanation. To achieve this, we first assess the explanation using validation records and identify the record with the lowest score, which is then included in the feedback prompt. Both the ground-truth activations and the activations predicted by simulator are provided the feedback (Figure 9).
Subsequently, we append suffix to the feedback, instructing the explainer to reflect on the errors in its previous explanation and generate revised one. For the unstructured tree explainer, the feedback suffix is shown in Figure 10, where the purple text is included only when explicitly setting maximum number of rules in the structured explanations. For the structured tree explainer, the feedback suffix is shown in Figure 11.
26

---

Overall score (out of 1.0 max): (score} Most incorrect record:
Activation-causiin tokens (review) <start>
{tokens [0]} unknown
{tokens [1]} unknown
(tokens [i]}
(holistic anttvatinattivattinnsss.ssss [i]}
{tokens [i+1]} [holistic.actiiatiinnsii
{tokens [n]) abtiisti [n]]
<end>
Activating token errors <start>
(tokens [0]} (predicted: (predicted, activations
( [0]}) (actual:
factivations [0]}} (error: activations [1]}) (actual:
activations [0] activations [0]})
(tokens [1]} (predicted:
(activations) [1]}) (error: (prediicted_actiittiion [1] activations [1]})
tokenss [n]] (predicted: anddiicte_cttivattons(((]] (actual:
(activations [n] (error:
[n] activations [n]])
<end>
Figure 9: The feedback string for tree-based generation for both the structured and unstructured. The text in blue is optional and only included when using the holistic activations introduced in Section 4.2.
27

---

Try generating a better explanation, taking into account this feedback. Be creative; if you have been trying something and it isn 't working, try something else.
Format your response as including first an improvement (in natural language), then the explanation.
You must make your improvement precise and specific; are a few examples:
"The neuron is activating on word 'cat', but my explanation doesn't capture this. I should amend my explanation." "
"My score is much lower than previous attempts. I should undo my recent changes."
"My last explanation says that 'cat' activates in any context, but this examples show that it doesn't activate when 'dog' appears previously in the text. I should narrow the context of my explanation."
The strict maximum number of rules is (rule_cap]. Do not generate
than this number of rules. You should not try to fill up the rule cap, only add rules if they are actually necessary and try keep the list of rules as short as possible.
Structure your response as follows:
IMPROVEMENT:
NEW EXPLANATION:
Figure 10: The feedback suffix for the unstructured tree explainer. The text in purple is only included when the maximum number of rules is explicitly set.
28

---

Try generating better explanation, taking into account this feedback. Be creative; if you have been trying something and it isn't working, try something else.
Format your response including first an improvement (in natural language), then the explanation.
You must make your improvement precise and specific; here are a few examples:
"The feature is activating on the word 'cat', but my explanation doean't 't capture this. I should add another rule to my explanation list." " "My score is much lower than previous attempts. I should remove the rules added recently, perhaps they are too long and confusing." "My last rule to says that 'cat' activates in any context, but this examples show that it doesn't activate when 'dog' appears previously in the text. I should add this required context to the rule."
"My rule says to activate on 'cat' with strength of 2, but the examples have higher activations. I don't need to add any new rules, but I should increase the strength of my rule."
The strict maximum number of rules (rule_cap}. not generate more than this number of rules. You should not try to fill up the rule cap, only add rules if they are actually necessary and try to keep the list of rules short possible.
Structure your response follows:
IMPROVEMENT:
NEW EXPLANATION: "activates_on": "strength": ... }, ]
Note that the "activates_or" on" field is a single string, not a list of strings. Do not include any comment in the JSON list.
Figure 11: The feedback suffix for the structured tree explainer. The text purple is only included when the maximum number of rules is explicitly set.
29

---

A.2 Simulation
For simulating feature activations, we follow the "all-at-once" method introduced in Bills et al. []]. Unlike their original "one-at-a-time" approach-which requires one forwards pass per simulated token-the "all-at-once" method provides all the tokens to the model during one inference pass with the activations marked as unknown. The topk logprobs for each unknown can be used to compute the expected value of the associated token's ' activation. A more comprehensive discussion of this approach can be found in Bills et al. [1].
The original simulation method in Bills et al. [1] used the closed-source sext-davincii-O0 model. is the associated topk logit feature for input tokens.
Unfortunately, this model is deprecated, To our knowledge, no cloud inference provider supports to topk logit feature, meaning that our simulation had be carried out on local hardware.
We optimize local token simulation using a two-level key-value cache. Note that simulation prompt can be broadly decomposed
system prompt & few-shot examples + explanation + simulated sentence
The first level of the cache captures the system prompt and few-shot examples, which are shared across all explanations and simulated sentences. The second level appends the key-value store for the explanation. We can then use this to efficiently simulate the activations for variety of different sentences for validation and testing.
We simulate all our explanations using the gemma 2-27b-it model loaded with four-bit bit quantization [34]. We selected this model as it significantly outperformed all equally smaller sized models during our preliminary testing. Other models we considered included those from the Ilama family [37], Mistral and Ministral families [14], and the distilled DeepSeek family [10]. The small size and compression of gemma-2-27b-i- 2-27b-it allows for local simulation on a single 40 GB Nvidia A A100 GPU.
Prompts.
The simulator prompts consist of three components. First, we provide the simulator with header message, as shown in Figure 12. Next, we present the same set of few-shot examples that were shown to the explainer, formatted in the tokenttabaacti vatiiiiinnnn format (Figure 8a). Finally, we supply the model with an explanation and a sentence, prompting it to predict the activation values for each token in the sentence using the message shown in Figure 13, with the sentence being simulated formatted as in Figure 8b. The predicted activation values are then collected using the "all-at-once" method described earlier.
We re studying neurons in neural network. Each neuron looks for some particular thing in a short document. Look at summary of what the neuron does, and try to predict how it will fire each token.
The activation format is tokenstabaactivation. activations go from to 5, "unknown" indicates an unknown activation. Most activations will be 0.
Figure 12: The header message for the simulator, provided system message.
The previous messages were just examples. Now you will be given a sequence of tokens and asked to predict the activations of each token.
Always output numerical activation,
if preceding activations unknown. Even if you have predicted "unknown" many times previously, these were mistakes. not repeat this mistake, and output numerical value from 0 to 5.
The prompt given the simulator after the few-shot examples. Here we use the "all-at- once" method in Bills et al. [1] to collect all predicted activations values in a single inference pass.
Figure 13:
30

---

A.3 Complexity analysis
Our complexity analysis LLM judge is also locally executed and shares is architecturally similar our simulation strategy. Instead of a bilevel key-value cache, we simply cache the system prompt and few-shot examples for the complexity judge once. We then reuse this cache analyze every explanation componeni individually, before aggregating the total complexity as the mean over all components, As in the simulator, we prompt the complexity judge provide an integer score between 0 and 5 for each component. Our model is gemma- 2-27b-1n with four-bit precision.
Prompts.
The prompts for the complexity analyzer consist of three components. First, we provide header message system message, as shown in Figure 14. Next, we present set of few-shot examples demonstrating how to analyze the complexity explanation component, assigned integer complexity score, and brief justification for the score. Finally, we prompt the complexity analyzer with a new explanation component and ask it to output an integer complexity score.
explanations. Each example includes an
We re studying features in neural network. Each feature has an activation rule describing how it activates.
Given an explanation of a feature's activation rule, predict how complex it is.
The complexity a number between and 5, where 0 is the simplest and 5 is the most complex.
The complexity is determined by the level of abstraction required to understand the activation rule.
Having a specific activation context is more complex than not having an "Any" context.
Activation rules which activate only list of specific tokens simpler than rules which activate on tokens capturing abstract concepts.
Figure 14: The system prompt for the complexity analyzer model.
The few-shot examples used for complexity analysis are follows.
Example
Explanation component: "the word 'instruments', specifically in a musical description, catalog, reference"
Complexity:
Justification: Low complexity only activates on specific words.
Example 2
Explanation component: "present tense verbs ending in 'ing""
Complexity: 2
Justification: Moderate complexity need to understand verb tenses and sp- ecific suffix patterns.
31

---

Example
Explanation component: "words related to medical conditions, in the context of movies and filmmaking"
Complexity:
Justification:
Higher complexity due to need to recognize medical termi- nology in metaphorical usage.
Example
Explanation component: "The word 'risk", in the context of medical research/studies" Complexity:
Justification:
Moderate complexity since only activates on specific words in medical context.
Example
Explanation component: "expressions of skepticism"
Complexity:
Justification: Very high complexity due to abstract nature skepticism.
Example
Explanation component: "words that reflect negative judgments", Complexity:
Justification:
Very high complexity due to the abstract nature of negative judgments.
32

---

B Holistic feature examples
In this section, we compare the regular and holistic activations for example sentences from our dataset, highlighting how holistic activations provide clearer and more accurate characterization of a feature. As described in Section 4.1, holistic activations measure the contribution of each token to the feature's activation over the entire sequence. In contrast, regular activations reflect the feature's 's value individual tokens, which can be difficult to crrppret-especallyy for features that tend to activate only after full concept has been completed, due to the causal structure of Transformer-based language models.
B.1 The table cell separator feature
As an example, we examine the regular and holistic activations for the 1st feature at the 11th layer of the Gemma 2 9B model, which activates on table cell separators.
Regular activation: 370 -33 76.9669999222 8 1 195 93.7938
Holistic activation: 370 33 33/764969 1951999933377 93.7938
Based on the regular activations alone, it appears that this feature activates numbers. However, closer look reveals that it actually fires on numbers that follow table cell separator. The holistic activations successfully capture this pattern, offering much clearer and more accurate interpretation of the feature's 's behavior.
B.2 The mathematical operations feature
We present the regular and holistic activations on an example sentence corresponding the 1st feature at the st layer of the Gemma 9B model, which activates on mathematical operations.
Regular activation: int*); void increment_and _arra O) int main Of
increment increment_array() increment address (int ptr)
Holistic activation: )nt void increment array O; int main Of
increment increment_array() aaray(r
increment address (int ptr)
From the regular activations this sentence alone, it initially appears that this feature responds to pointers. However, the holistic activations reveal that it is specifically the operation of incrementing the pointer-not not the pointer itself drives the feature's activation. This illustrates how holistic activations can more accurately capture the true behavior a feature by considering its contribution the overall sequence activation.
B.3 The "do something like" feature
We show the regular and holistic activations on an example sentence for the 8th feature at the 3st layer of the Gemma 2 9B model, which activates the phrase "do something like".
Regular activation: to be able to do something like this: xamllcs: public partial
class MyControl UserControl public MyControl()
Holistic activation: to be able to do something like this: .xaml.cs public partial
class MyControl UserControl public MyControl()
The regular activations highlight the word "like", which alone does not clearly reflect the feature's behavior. However, examining the holistic activations reveals that this feature actually responds to the entire phrase "do something like". providing much clearer interpretation of activation pattern.
33

---

C Complementary sentence examples
In this section, present example features along with their structured explanations, top appacctiatinng samples, random con-activating samples, and top semantically similar non-activating samples. Acti- vation values are visually highlighted using color, with darker shades indicating stronger activation.
For top-activating samples, we highlight their ground-truth activation values illustrate the words and contexts that most strongly activate the feature. In contrast, for random and semantically similar non-activating samples, we highlight their simulated feature activations to assess accuracy and potential errors of the generated explanations. Any positive simulated actívation on these non- activating samples are false positives. We omitted the line breaks for readability.
We expect that overly broad explanations- those biased toward recall but lacking precision [4]-will produce more false positives on semantically similar samples, thereby revealing their inaccuracy.
C.1 The "render" and "argument" feature
This is the 2nd feature on the 1st layer of the Gemma 9B model.
Structured explanation
[
"activates_or" on": "Words related to rendering in programming contexts, like 'render!."
"strength":
}
"activates_on on": "Words related arguments and function calls within code.",
"strength":
}.
"activates_on on":: "The word \"throws\" in code context related to exception handling.", "strength":
Top-activating samples
render partial: "edit_ "eiitjico''a <>><h<><%== t( (('avatar,p prrvvaaarproofllee <ittlee_hhange) %></h4>
have found that styled components, when not included the initial render
render h2 headline
.c0 --webkit--etter-pppcing 0.5px:-
renderSubGame is called with an argument tbia.subGamelnfo like so: renderSub
Charts=/rende @ siime_serisscceeee time_series: series: ttime_series .charts render
title, @Quser.ful__name.ouuee wow.taieer- "/header". title: @ @user. fuser.full full name
immaoo TR from t./TR/index:impott mount, shallow, render
namespace render struct RenderState; | J.ammspacce scene class PlaneNode: var Song db. model("Song')) var Album = db. db. mwmeelldllluuunnn I want to render argument in call funetionlmaginn scenario like obj render
Random non-activating samples
34

---

joke, or is it serious?T'm watching two adaptations of Romeo and Juliet that take present invention relates to intumescent compositions based on foamed or This Program is free software; cases]. To investigate the application and efficacy of video-assisted thoracoscopic
005-2013 Team XBMC http://xbmc..rg
2, y: 1, w: }. Give prob of picking y. 1 v. and w
3125 How many millilitres are there in 37/5 of a litre? 7400 What is License, Version 2.0 (the "License"); you may not use this file except in
typemoq iimport IDirectoryManager, ISettingsManager from "managers";
344456667778889991111 while (ii getchar())
def with baton until (baton Baton.obtain) sleep(2) end result yield
Semantically similar non-activating samples
/components.htmm htmll">> <% name= "itriiii""ss"
from "./jsssc/ommponelsscccoooproe/// import Metries from "../js/sre
javascript method on a htmlwidget (jsoneditor) in shiny? I'm trying to use
excr-example', width: 800, height: 600, scene:
create: create
useHistory from 'react-router-domt import Card PageSection from '@
paaaampl__ootput:s"">>>iiiipippi <spprr init function() var person
dom, args) this.progress 0.0; this. messaee "": this.dom dom;
call: puppeteer. puppeteer.launccc)..oo btunnch).eeeeeenrroeeeerrrrrr
let html = <DDOCTYPE html> props return React. Eeeee,e
const color, oother
"undefined")) quitest_coverape
yuiiest__covrllnne function(sre, lin
As shown in the simulated feature activations above, the average number of false positives per sentence significantly higher for semantically similar non-activating samples (11.6) compared to random non-activating samples (2.2). This discrepancy arises because the generated explanation describing "arguments and function calls within code" overly broad. The Sentence Transformer effectively captures this, selecting coding-relate samples that lead the simulator to generate more false positives.
C.2
The table cell separator feature
This is the 1st feature on the 11th layer of the Gemma 9B model.
Structured explanation
[
"activates_on" _n":: "Numerical values within specific contexts (e.g., scientific research, financial data, or technical documents).".
"strength":
}
"activates_on" on" "Specific HTML/data table markup",
"strength":
35

---

,,
"activates_oni" _n "The word \"Iris-setosa\";
"strength";
}
Top-activating samples
=lleft(beegin/ (array) (mrii &3 &2 &2 & oN
& & & W & a & & & W &
Vdots & WI & x & a_3+x & Mdots & x VI Ivdot & wdots& &ldd
0.999 0.999 5,1 Iris-setosa Iris-setosa 0.994
<path d=1M99.59 8L12 10.59 9.41
COUNT QTY UNIT OF MEASURE Belgium Natural Gas Physical 5 [412,500) Iris-setosa 0.996 0.996 4.6 Iris-setosa Iris-setosa
5.4 Iris-setosa tris-sctosa 0.998 0.998 4.4 Iris-
0.994 4.9 Iris-setosa Iris-setosa 0.991 0.991
Positions 40
40 P&L Daily ($thousands) 0.5
(2.2)
Random non-activating samples
@ name Includes file @description The number of files directly included appearances do matter. This is not self-interested plea aimed at urging my
By Erin Roach NASHVILLE (BP) The president of the Association of
a) be the third derivative of 111 4/24- 14 ##2. Let n(s)
int
z z22 22ssqtt 2+y
10 // Search list of files for lines that match given regular-expressionn //
Argonaute: A scaffold for the function of short regulatory RNAs. Argonaute is the import "os" "syscall" "unsafe") // Flags to control the terminals mode. const to move freely from one point of connection another in various networks it visits smoking program. This article has presented overview of the Quit-Smoking
Semantically similar non-activating samples
000000 1.000000000000::: 3.2000
-0.34691048
00.4292854
0,000837 0x001B // 0.000460
0x000
0.03744
0.12232
0.003
0.009
12-0-12-04
2010-12-01
0.07 times
1177
89.39
0.85
-0.03 -0.
0.2c"
% 1.236
1 24c+00'
The total characters in the
36

---

<o1>
<p
<</pp>>>>>>>>
<p2></p2>
<p3></p3
1943
Width: 1005
VWidth: 0 Flags: HM LayerCount: 2 Fore
As evident from the top-activating samples, this feature primarily activates on table cell separators. However, the explainer fails to capture this specificity, instead providing overly broad explanations such as "numerical values" and "HTML/data table markups." The Sentence Transformer is able to select samples related numbers and HTML codes, leading the simulator to produce false positives. The average number of false positives per sentence for semantically similar non-activating samples is 18. signiiinnfy higher than the 2.9 observed for random non-activating samples.
C.3
The mathematical operations feature
This is the 1st feature on the st layer of the Gemma 2 9B model.
Structured explanation
[
"activates_oni "Mathematical formulas, equations, and operations (e.g., sum, miltiplication, division, + exponential functions).",
"strength":
},
"activates_on"" "Words related to lists (e.g., elements, length, sum) ",
"strength":
"activates_ _n":: "Mathematical functions (e.g., cos, sin, log. exp, arithmetic operations)", "strength":
},
"activates_on on": "Programming language constructs related to + calculations (e.g., for loops, variables)", "strength": 2
Top-activating samples
mean of list, i.e. the sum of all elements in the list divided by its length. (You
code i get output as 392 #includecstdio.h> #define CUBE(r) (r)*
df) [dx) = Vimm_\Delta x \to 0) Váfrac ff(x ADelta x)- (ADelta
true # let satisfies associative law 1 (2 3) (1 + 2) 3:;
M[B](c (a => k(a) e)) def map[B](F:
B): M[
the moment generating function of random variable SX$ as: SE(e" know
P(A leap B) P(B))cdot P(AI B)=P(A)ccoot P(BJA)S, where P
interest has compounded for certain number of years. Note: A P(1+))
37

---

is continuous at a number a if limit of f(x) as approaches a is equal to f(a
<stdlib.h> int max numl, int num2) int result;
if(numl > num2
Random non-activating samples
Free Disk Space 46GB minimum Write Review Nioh is a Japanese game designed
of code execution with wait and notify research using wait and notify methods
Omaha network will be down from 8:00 pm to 12:00 midnight, Tuesday,
multiple racial origin and (2) the race categories of (2a) American Indian and
Copyright (C) 2009 Conexant Systems Inc. Authors <shu.lin won.eeet.eenooomm
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the
464230, -668516 -999644 1185500?
as possible. Mouse Control Click on a puzzle square to uncover the image or education. Since 1975 the flow foreign medical graduates (FMGs) into US Cal.App.2d 615 (1960) IRA GARSON REALTY COMPANY (a Corporation),
Semantically similar non-activating samples
I saw examples of my problem, but it seems can't find correct loop. In any case,
pl.x=xx pl.y That's code and the error I get is
[1..6]; I[[1..3]] == I; end; fO; Where():
to the interval [O: 1j. The mapping should deterministic in order to get
would be foo $2 barl $3 bar2 $4 bar3 and on.. it would be like
Macx<biJS can be written S[
b___________ aS. Is there equivalent notation for $f(x)
ChereWithVaasO) quit; f:=function() if true 1/0 then return 1; fi; return 2; #include <vector> #include <algorithm> using namespace std; int main()
x x=) into same coordinates. Ityped: a (x)
cin>>n; for 1; t= cout<<t<<endI;
From the top-activating samples, we observe that this feature primarily activates on mathematical operations such summation, multiplication, division, and even limits. However, the generated explanation is overly broad, encompassing concepts like "mathematical formulas." "lists," "functions," " and "programming language constructs." As a result, the Sentence Transformer, trained on top- activating samples, effectively selects math- and coding-related sentences, leading to higher false positive rate. The semantically similar non-activating samples produce an average of 5.8 false positives per sentence, compared to just 0.9 for random non-activating samples.
C.4
The definition feature
This is the 2nd feature on the 31st layer of the Gemma 2 9B model.
Structured explanation
[
{
38

---

"activates_on": "The word 'define' and related concepts like 'defining' and 'defined'.",
"strength": 5
}
{
"activates_on" "The concept of explicit function or struct declaration.",
"strength": 4
},
{
"activates_on" "The word 'Undefined'..
"strength": 4
}
Top-activating samples
intl'; const messages = desseee securityConttolssaaell: { defaultMessage::
Should I manuaaly create definition for GetEnumerator? Seems like should
DAU/MAU using Application Insights Analytics? Assuming have definition of
Define Bessel's inequality $$ \sum_ "Naaann VxV²2
Undefined index in PHP post with AJAX In my local machine, am trying to
4) & '"Sheet "!A"" This gives me Error: 1004, Object-defined error. I'd
platform contributions (defined via org.eclipse.ui.pp..... wi.popppMeeuss in this perspective.
to define metric with some labels but don't always have them all the
defined as the condition of having abnormally small teeth M[@BIM According
but definition of anacronym is sketchy http://bit.lyhhlAA Richard @ Home
Random non-activating samples
I've had the idea of the WWP lettering shadows for while in my head, but just
men under the Diocese of Australia and New Zealand, ROCOR The monastery is
are being studied as microdosemeters since they can provide sensitive volumes of
idermal surface saccharides reactive with phytohemagllutinins and pemphigus
Deviates from the original intent"? suggested this edit yesterday: the question is
and Adaptive Reactions in People Treated for Chronic Obstructive Pulmonary
Feb. 23, 2014 (HealthDay News) Preteens who changed schools frequently when this file except in compliance with the License. # You may obtain a copy of the / ATutor */ f** link a jQuery UI sortable element with array? So, basically, how can link
Semantically similar non-activating samples
.> <>l>>> <pl></p>>>>>>>>> <>>>>><>><<p>><pp>><>>>>>>> <p>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <p3></p3
.ist..ppp> boost:adjacency listcbbottttoo boost:veeS, boost::dreectdd,
39

---

of micron. Its interior mechanism rests on heavy cast iron base that is covered
class AB amplifier: Rv adjusts the bias point of the two transistors so
/mkcharacters shur://eem--iiiitrirSSuuiillnnnn
share/texmf-dist
(is android) declare_ arrgs( compile_see sespppll__sd____eeeeeeeeee _cientteeetttt is Jinux +
Application No. 2000-159163, filed Mar.
NppExecute plugin in notepad+++. am not able to figure out next step to
a customer can see, based on their customer number and order type. was using
As shown in the top-activating sentences, this feature primarily activates on the words "define" and "definition". However, the generated explanation broadens this include "concepts of explicit function or struct declarations", which proves overly inclusive. The similarity-base complementary sentences contain code snippets involving declarations, leading the feature simulator produce false positives. On average, the semantically similar non-activating samples trigger 1.6 false positives per sentence, significantly higher than the 0.3 observed for random non-activating samples.
40

---

D Supplemental experimental results
D.1 Supplemental explainer comparison results
The tabular results of Figure are shown in Table
Table 1: The explainer comparison results in tabular form.
Model
Oze shot Unstrsttncd
Oze aboc
One sie Strccunal Negatives
One abot
Tize
Thee Structuned
Tree Sratterd Negalives
Tree Seratura w/ Holivie
Stnacunal
Stractaned wi Holisii
Unsimetunal
Ganma 29h 0.363 = 0.082
0.384 0.052
3.366 + 0 008
0. 1.5 0.052
8.46 + 0.050
0.451 tt 0.032
0.450 + 0.082
0.452 d: 0.052
Liaaa 3.1 8b
0.332
0.033
0.360
0.033
1.342 + 0 004
0. 0.349 0.004
0.484 +: 0.032
0.392 0.034
0.389 +: 0.034
0.399 0.038
CPP122 0.5.0 = 0.033
0.568 030
0.507 +: 0 0022
0. 0.033
D. 002 d 0.028
0.096 0.022
0.023 d: 0.028
0.028
D.2
Complementary sentence sourcing correlation scores
We present the correlation scores for the four complementary sentence sourcing methods in Figure 15. As shown in Figure 15, both the one-shot and tree explainer yield consistently lower correlation scores when evaluated on similarity-based complementary sentences. This further highlights the recall bias of existing feature explanation methods and demonstrates the effectiveness of similarity- strategies in identifying "close counterexamples."
Randon
Random
Randon (non-uctivating)
0.8
Romdon (nvatnncivaatin))
0.7
Similar
Correlation score
Similar
Similar (non-artivating)
0.7
Similar (non-artivating)
0.6
Correlation
0.6
0.5
0.5
0.4
0.4
0.3
0
10
20
30
40
0
10
20
30
40
Layer
Layer
(a) Results for the one-shot explainer.
(b) Results for the tree explainer.
Figure 15: The correlation score between the simulated and ground-truth activations, averaged over 50 features per layer. "Non-activating" indicates that the sentences have no ground-truth feature activation.
D.3 Feature complexity and polysemanticity for the one-shot explainer
Figure presents the complexity and polysemanticity plots for explanations generated by the one- shot explainer. Across all three models, we observe a general upward trend in complexity as depth increases. Meanwhile, polysemanticity tends to peak in the middle layers, remaining lower both the early and final layers.
D.4 Supplemental results for the structured explanations
In Figure 17, we plot the correlation scores of structured explanations generated by the one-shot explainer across different layers and numbers of rules (i.e., explanation components). For each rule count i from 1 to 5, we report the proportion of the maximum score achieved by explanations with up to rules, relative to the overall maximum score across rule counts. As shown, the correlation score rises rapidly as more rules are included in the structured explanation, highlighting the benefit of combining multiple monosemantic rules. This trend helps explain the superior performance of structured over unstructured explanations for the one-shot explainer in Figure 4.
41

---

2.6
Mean rule count 90% of max score
3.0
Gemma
2.4
Llama 3.1 8b
Average complexity
2.5
GPT-2 Small
2.2
2.0
2.0
1.8
Gemma 9b
Llama 3.1 8b
1.5
1.6
GPT-2 Small
0
20
40
60
S0
100
0 20
40
60
80
100
Layer (%)
Layer (%)
Figure 16: Lines denote mean values, and shaded regions denote 80% confidence intervals. The horizontal axis corresponds to the analyzed layer as proportion of the total layer count, which varies between models. Left. The average explanation complexity for the one-shot explainer. Right. The polysemanticity of structured explanations generated by the one-shot explainer.
15
15-
Rid Chenk
Rido Chenk
Cheak
(a) Gemma 9B.
(b) Llama 3.1 8B.
(c) GPT-2 Small.
Figure 17: The correlation scores of structured explanations generated by the one-shot explainer across different layers and number of rules.
We present the same analysis for the tree-based explainer in Figure 18. Here, the performance differences between various maximum rule counts are less pronounced. This is consistent with Figure 4, where structured explanations offer little advantage over unstructured ones when using the tree-based explainer. We hypothesize this is because the tree- base explainer's iterative refinement process naturally incorporates the feature's polysemantic behaviors the need for explicit structuring.
explanations, reducing
('dreecantintty Harmap
l'Moorscciiity Herrop
I'dtemuntialyy Harra
3
32
4-
Rale Duea
(a) Gemma 2 9B.
(b) Llama 3.1 8B.
(c) GPT-2 Small.
Figure 18: The correlation scores of structured explanations generated by the tree-based explainer across different layers and number of rules.
42

---

E Licenses
We use a subset of the Pile which eliminates potentially copyrighted content [8, 25]. The Pile itself is MIT-licenssd. GPT-2 is released under the Apache license. Llama 3.1 and Gemma 2 licenses are permissive [23, 20].
43


Reviews:
[{"review_id": "Mbc3OLWmx8", "reviewer": "Reviewer_q24V", "summary": "This paper proposes three major innovations to the standard automated interpretability pipeline for sparse autoencoder features. First, it introduces a similarity-based strategy to find semantically related features to a given query feature. The LLM explainer is then asked to generate explanations that make it possible to distinguish activating examples for the query feature from activating examples for the semantically related features. Second, it introduces a JSON-based format for structured explanations, enabling the use of more sophisticated explanations than were used in prior work. Third, it introduces a tree-based approach for iteratively improving structured explanations.", "questions": "1. Have you tested your innovations on lower quantiles of the activation distribution? Do you anticipate that your results will be qualitatively different on, say, the middle tercile of the feature activations?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Structured explanations enable a more direct measurement of the degree of polysemanticity in a feature (number of explanation components)\n-  Tree-based explanation is novel and appears to open up new directions for research\n\nWeaknesses:\n- Focuses on top-activating examples, which are known to be significantly more interpretable than medium or low activation quantiles\n- Tree-based explanation method is computationally intensive", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer q24V for the constructive review. We address your concerns as follows.\n\n---\n\n**W1.** Focuses on top-activating examples, which are known to be significantly more interpretable than medium or low activation quantiles.\n\n**A1.** Thank you for this observation. We focus on the top-activating quantile because SAE features only activate on a tiny fraction of tokens, making sentences in medium and low activation quantiles essentially random. Our evaluation approach on top-activating quantiles aligns with standard practices [1].\n\nTo verify that our methods remain superior across different activation levels, we tested on the 10th activation quantile (out of 1000 total). For lower quantiles, sentences would be uninterpretable due to the sparse nature of SAE activations. In this experiment, we combined sentences from this quantile with an equal number of random sentences to form training sets for generating explanations and test sets for evaluation. This setup mirrors our paper and [1], with the only difference being that we replaced top-activating sentences with those from another quantile.\n\nResults for the 10th quantile are shown below:\n\n| Method | Layer 10 | Layer 20 |\n| --- | --- | --- |\n| One-shot, unstructured | 0.099 | 0.118 |\n| One-shot, structured | 0.172 | 0.107 |\n| Tree, unstructured | 0.121 | 0.144 |\n| Tree, structured | 0.222 | 0.165 |\n\nThese results demonstrate that structured explanations consistently outperform unstructured ones, and the tree explainer generally surpasses the one-shot explainer. However, we should note that sentences in lower quantiles tend to be more random, making scores from those quantiles less informative.\n\n**References:**\n\n[1] Bills et al. Language models can explain neurons in language models. *OpenAI blog*.\n\n---\n\n**W2.** Tree-based explanation method is computationally intensive.\n\n**A2.** We agree that the tree-based explainer requires more computational resources than the one-shot explainer, particularly at larger depths and widths. However, this cost comes with a significant performance improvement for SAE feature explanations (~20% increase in correlation score). This represents a substantial advance in mechanistic interpretability approaches and enhances our understanding of LLMs. Given that SAEs are widely used for understanding and controlling LLM behaviors, accurate feature explanations are essential. Additionally, the tree-based explainer's depths and widths can be adjusted to balance efficiency and accuracy.\n\n---\n\n**Q1.** Have you tested your innovations on lower quantiles of the activation distribution? Do you anticipate that your results will be qualitatively different on, say, the middle tercile of the feature activations?\n\n**A3.** As shown in **A1**, we tested our methods on the 10th quantile and consistently observed improvements over baselines. This confirms our approaches' superiority across different activation quantiles. For even lower quantiles, SAE activations are essentially zero, rendering the sentences uninterpretable.\n\n---\n\nWe hope our response addresses your concerns. If you have further questions, we will be happy to address them during the discussion period."}, {"review_id": "KSQocRV6iD", "reviewer": "Reviewer_gHHG", "summary": "This paper addresses the problem of explaining features extracted by sparse autoencoders (SAEs) --- a method to obtain disentangled representations for a neuron in models such as LLMs --- via text in a faithful manner. A common way to do this is to look at what activates a given SAE latent and then ask an LLM for an explanation that encompasses highly activating inputs and excludes low activating inputs. Simulation experiments are then performed to check if predicted activations based on just the input and the explanation match true activations. This work recognizes that getting such an explanation from an LLM is challenging, since the explanation may be overly broad, which would result in high recall but low precision of simulation. It proposes using structured explanations and a tree based search to obtain better explanation candidates, and similarity based retrieval methods to better retrieve negative examples, i.e. examples that do not activate the latent. Experiments are performed on Pile with three LLMs to show that the proposed approach yields more false positives (i.e. avoids having too broad explanations), has better scores on simulation experiments, and can be used to analyze explanation complexity and polysemanticity.", "questions": "Please refer to the Weaknesses. I would be happy to increase my score if the concerns on motivation and having a more granular analysis are addressed.", "limitations": "Yes, but an additional limitation to consider is the reliance on LLMs for obtaining explanations.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper addresses an important problem, since getting faithful explanations is important when using models in safety critical domains.\n2. The proposed approach is simple but principled, and quantitative results appear to show that it is effective.\n\n## Weaknesses\n\n1. The key weakness for me is that the paper lacks qualitative examples that show why the proposed method is useful. While quantitatively one can see that it is better at finding false positives (Figure 3) and getting better scores (Figure 4), how does this finally affect the kind of explanations that are obtained for SAE latents? For instance, are more granular explanations obtained? Showing examples of explanations for latents along with what inputs activate them would be very helpful (including negative examples). Right now it is not very clear how useful the approach would be in practice, and whether the quantitative improvements are significant. The examples shown in the Appendix seem to primarily be prompting examples, but if this is provided there please point to them in the rebuttal.\n\n2. A problem that has been observed with SAEs is feature absorption and feature splitting (e.g. discussed in [1]). Intuitively, one would expect that this approach would be able to detect such instances better. This could be a very useful evaluation to have to show the practical benefits of the approach, since e.g. situations involving feature splitting are likely to show up false positives as discussed in the paper. It would also help understand if the sentence encoder similarities used in the paper are helpful for finding ideal candidate sentences. Less important, but a study on different SAE types (e.g. using Matryoshka SAEs) could also be useful.\n\n3. The trends shown for explanation complexity and polysemanticity (Figure 5) appear somewhat weak, and a more granular analysis would be helpful. For example, the results also depend on how good the trained SAE is, which would vary across models and layers (in terms of reconstruction loss, latent dimension size, sparsity, etc.) and could affect the conclusions made. Additionally, similar to Weakness 1, having examples would also help here, particularly for polysemanticity. For example, are the explanations for the polysemantic neurons conceptually similar or very distinct?", "was_revised": true, "final_justification": "I appreciate the examples provided, including negative examples. It would be very helpful to include several of them with a detailed discussion in the revision, as this makes the paper much stronger.\n\nOverall, my major concerns have been addressed, so I would be happy to increase my score to 5.\n\nAn additional question about feature splitting: the false positive correlation is interesting, but is just a proxy metric. What would be much more interesting is to see if these structured explanations can reveal that feature splitting happens, i.e. if you list out the explanations in the tree, does it show the exceptions that are not covered by the latent due to feature splitting? This in my opinion would be a much more interesting analysis.", "rebuttal": "We thank Reviewer gHHG for their thoughtful review. Below are our responses to their concerns.\n\n---\n\n**W1.** The key weakness for me is that the paper lacks qualitative examples that show why the proposed method is useful. While quantitatively one can see that it is better at finding false positives (Figure 3) and getting better scores (Figure 4), how does this finally affect the kind of explanations that are obtained for SAE latents? For instance, are more granular explanations obtained? Showing examples of explanations for latents along with what inputs activate them would be very helpful (including negative examples). […]\n\n**A1.** Thank you for this constructive suggestion. The tree-based explainer can generate more granular, accurate, and concise explanations compared to the one-shot explainer, thanks to its ability to iteratively refine explanations based on evaluation feedback. We demonstrate this through three examples where the tree-based explainer outperforms the one-shot explainer, plus one negative example showing its limitations.\n\n**Example 1:** A birthdate feature\n\n- One-shot explanation components (score: 0.062):\n    - Names of people, specifically first and last names.\n    - Date formats and structures, including days, months, and years.\n- Tree-based explanation components (score: 0.621):\n    - born\n    - date of birth in parentheses\n- Example activations (with activated tokens in bold):\n    - Anabel Englund (**born September** 1 **,** 1**9**92 **) is** an American singer and songwriter\n    - Kim Soo-yong (**born September 23, 19**2**9) is** a South Korean film director\n    - Simon **Blair Doull (born 6 August 19**6**9) is** a New Zealand radio personality, commentator and former international cricketer\n\nThis feature activates on birthdates of people in parentheses. While the one-shot explanation broadly includes names and date formats, the tree-based explanation captures the specific pattern more accurately.\n\n**Example 2:** An import and JavaScript keywords feature\n\n- One-shot explanation components (score: -0.025):\n    - Programming syntax related to imports, specifically import statements.\n    - Code comments.\n    - Specific programming keywords.\n- Tree-based explanation components (score: 0.117):\n    - JavaScript/React keywords and syntax\n    - import and export statements\n    - slider and view related terms\n    - querySelector and document\n    - formData and FormData\n- Example activations:\n    - **import {** NUM_BINS, PRES**ETS** } **from \"./** config **\";**\n    - **export default** Object.**freeze**({\n    - **const grid = document.querySelector('#** Symbols **.grid');**\n\nThis polysemantic feature activates on various programming contexts, including import statements and JavaScript keywords. The tree-based explanation provides more detailed coverage of these activation patterns.\n\n**Example 3:** A parenthesis feature\n\n- One-shot explanation components (score: 0.100):\n    - Mathematical and programming syntax, especially with context of abstract algebra and mathematical concepts.\n    - Use of parenthesis '('\n- Tree-based explanation components (score: 0.602):\n    - '('\n    - ')'\n- Example activations:\n    - create a **(** mutable **)** object first, and at certain scenario I need to make it immutable\n    - I allow other (mac **)** machines to access an IIS7 hosted MVC3 application on my (win7 **)** machine\n    - I'm looking for a formula to calculate the **(** product **?)** of an arithmetic series\n\nThis feature simply activates on parentheses. The tree-based explanation captures this concisely, while the one-shot explanation includes unnecessary context about mathematical concepts.\n\n**Negative example:** A people name feature\n\n- One-shot explanation components (score: 0.251):\n    - Names of people, including first and last names.\n    - Proper nouns, including names of songs, albums, movies, and places.\n- Tree-based explanation components (score: 0.039):\n    - Wright\n    - netto\n    - King\n    - Cliff Richard\n    - Lloyd Webber\n- Example activations:\n    - billionaire Nirav **Modi**, could have cost $2 billion\n    - 1965 popular song with lyrics by Hal **David** and music composed by Burt Bach**arach**\n    - \"You have seen into the heart of my music,\" **Andrew Lloyd Webber** said to Gale **Edwards** in a trans\n\nThis feature activates on people's names, but the tree-based explainer overfits by producing a list of specific names. Such cases are rare in our experiments.\n\n---\n\n**W2.** A problem that has been observed with SAEs is feature absorption and feature splitting. Intuitively, one would expect that this approach would be able to detect such instances better. This could be a very useful evaluation to have to show the practical benefits of the approach, […]\n\n**A2.** Thank you for this helpful suggestion. We hypothesized that feature splitting and absorption would result in more false positives, and tested this empirically. Following [1], we examined SAE features that activate on words starting with specific English letters. We measured feature splitting and false positives from the simulator (which activates on words beginning with particular letters). Here are the correlation scores at different feature splitting thresholds:\n\n| F1 threshold | Correlation coefficient |\n| --- | --- |\n| 0.03 | 0.553 |\n| 0.02 | 0.578 |\n| 0.01 | 0.637 |\n\nThe high correlation ($r>0.5$) between feature splitting and false positive rate suggests our approach offers an efficient way to detect feature splitting without training multiple k-sparse probes.\n\nWe also found a correlation of 0.1042 between feature absorption and false positives—statistically significant but more subtle than feature splitting.\n\n**References:**\n\n[1] Chanin et al. A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. *arXiv preprint arXiv:2409.14507*.\n\n---\n\n**W3.** The trends shown for explanation complexity and polysemanticity appear somewhat weak, and a more granular analysis would be helpful. For example, the results also depend on how good the trained SAE is, which would vary across models and layers (in terms of reconstruction loss, latent dimension size, sparsity, etc.) and could affect the conclusions made.\n\n**A3.** You raise an important point about how SAE training quality affects feature complexity and polysemanticity. While we kept latent dimensions consistent across layers, reconstruction loss and sparsity varied. For a more detailed analysis, we measured correlations between complexity and reconstruction loss ($r_1$), and between polysemanticity and reconstruction loss ($r_2$), across different layers:\n\n| Model | $r_1$ | $r_2$ |\n| --- | --- | --- |\n| Gemma | 0.320 | 0.293 |\n| Llama | 0.520 | 0.461 |\n| GPT-2 | 0.671 | 0.151 |\n\nThe positive correlation between these metrics and reconstruction loss makes sense—more complex and polysemantic features are inherently harder for SAEs to learn.\n\nWe also examined correlations between complexity and L0 ($r_3$), and between polysemanticity and L0 ($r_4$):\n\n| Model | $r_3$ | $r_4$ |\n| --- | --- | --- |\n| Gemma | 0.262 | -0.093 |\n| GPT-2 | 0.229 | 0.086 |\n\nLlama was excluded as it uses top-k activation. The results show complexity correlates positively with L0, while polysemanticity shows no significant correlation.\n\n---\n\n**W4.** Additionally, similar to W1, having examples would also help here, particularly for polysemanticity. For example, are the explanations for the polysemantic neurons conceptually similar or very distinct?\n\n**A4.** For polysemantic neurons, structured explanations better capture conceptually distinct components compared to unstructured explanations. Here are two illustrative examples:\n\n**Example 1:**\n\n- Unstructured explanation (score: 0.894): The neuron is looking for mentions of specific types of devices, tools, or equipment, particularly when referred to by their common names or technical terms, such as \"instruments\", \"instrument\", or warning messages like \"warnings\" or \"Warning\".\n- Structured explanation (score: 0.967):\n    - The word 'instrument' and its plural 'instruments'.\n    - Warnings in code, such as 'Warning' and 'warnings'.\n- Example activations:\n    - Copyright (C) 2004 Texas **Instruments**\n    - This is a follow up to Is -Wreturn-std-move clang **warning** correct in case of objects in the same hierarchy\n\nThis polysemantic feature activates on both \"instruments\" and \"warning\"—distinct concepts that structured explanations capture more clearly.\n\n**Example 2:**\n\n- Unstructured explanation (score: 0.493): The neuron is looking for phrases or sentences that express sincerity, emphasis, or encouragement, often marked by words like \"mean\", \"seriously\", \"come on\", or phrases that start with \"let's be honest\". It also seems to activate on certain punctuation marks like commas that precede or follow these phrases.\n- Structured explanation (score: 0.589):\n    - The word 'on' in various contexts.\n    - The word 'mean' when used for emphasis.\n    - Expressions of informality or emphasis like 'come on', 'hey', or 'seriously'.\n    - Commas in certain phrases.\n- Example activations:\n    - Hang **on** for a minute...we're trying to find some more stories you might like\n    - By other js files, I **mean** the files you include in popup.html\n    - Come **on** in and stay for the video\n\nThis feature also activates on conceptually distinct patterns, which structured explanations capture more effectively.\n\n---\n\nWe'll incorporate these additional results and discussion in our paper. We appreciate your feedback and are happy to address any further questions."}, {"review_id": "PgnIW9Wm5m", "reviewer": "Reviewer_i1y8", "summary": "The paper studies feature explanations of Sparse AutoEncoders (SAEs) by using LLMs. For this, the paper builds on the idea of Bills et al. (2023) to use an iterative approach to explain individual neurons of SAEs by prompting an LLM. By proposing a tree-based explanation method and a structured, component-based format, the paper tackles challenges of Bills et al. (2023) such as overly broad explanations. Additionally, the paper analyzes the evolution of feature complexity and polysemanticity across layers of different models. The claims and statements of the paper are backed-up by empirical results by prompting LLMs.", "questions": "1. In the abstract it is stated the tree-based method \"refines explanations\". This is assessed in 5.3 (Feature Complexity) by using an LLM judge. How can we sure that LLM judge does a good job? In the current form, the assessment of explanations happens by a metric (the LLM judge) that we don't understand.\n2. What does \"close counterexample\" mean? How is closeness mathematically defined?\n3. Section 4.2, Branching step: The creation of the child explanations is without a score. How is it possible then to perform the pruning in step 3? It would be beneficial to formalize the methods in Section 4 by the notation of Section 3.2 (pseudo code). This would help readability and would better connect the different sections of the paper.\n4. Does no ground-truth mean zero activation? I found this confusing in the paper. Also the statement in L213 is equivalent to $f*(\\mathbf{t})_{i}=0,\\forall i$ under the restriction that all values have to be positive, which is more difficult to grasp from the presented equation.\n5. L217: Random non-activating sentences: \"defined as the proportion of positive elements\". This means that the value is in [0,1]. However, looking at Fig.3 it is above 1. How is this possible?\n6. L306: Is there any evidence for this hypothesis?\n\n**Some comments unrelated to my assessment of the paper:**\n\n- The wording \"Feature Learning\" and, at some place in the paper, \"Learning\" is misleading since there is no learning in an ML sense involved. It is prompting of an LLM.\n- L112: $f(t)$ not $f(\\mathcal{T})$; by definition f **maps** elements from $\\mathcal{T}$\n- L115: **an** explanation\n- L120: What constraint?\n- The wording of the paper should be improved. For example, \n   - L183: I wouldn't call this causally as causal has a strong meaning and clear definition in mathematics that I doubt to hold here.\n   - The use of training where inference or prompting is meant.\n- L146: I guess it must be \"negative log probabilities\".\n- L257: The font type of LLama 4 Scout is different than the others.", "limitations": "Yes, the limitations discussion is already good. However, it could be extended in the direction of how this method helps to understand LLMs. For instance, the second point mentions polysemanticity: What is the consequence if this fails?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- **Clarity:** The paper is well-written, well-organized and easy to follow.\n- **Clarity:** The motivation of the paper is clear (see summary).\n- **Clarity:** The mathematical formalization in 3.2 is good.\n- **Originality:** The paper clearly describes how the proposed method is different to the original work of Bills et al. (2023).\n\n### Weaknesses\n\n- **Originality:** The paper seeks to advance the field of XAI, in particular, understanding LLMs. Method-wise, it is clear how the method advances over the original work of Bills et al. (2023). However, it is not clear how the proposed method helps in understanding LLMs compared to Bills. In the abstract it is stated that \"this work addresses these limitations [overly broad explanations and polysemanticity]\" but the results demonstrate that these limitations are not addressed (e.g., see statement L316-317) and only marginally quantitatively improved (see figures and tables)---completely ignoring the true goal of XAI on enhancing the understanding of LLMs.\n\n- **Significance:** The contribution of the paper is marginal as it is mainly the approach of Bills et al. (2023) with a new prompting strategy. \n\n- **Quality:** After reading the paper, I'm not sure about the take away of the paper accept from what was already know from Bills et al. (2023): you can explain an LLM with an LLM to some extent but it might fail for unknown reasons. Moreover, the approach the paper builds upon is the explanation of a black-box with a black-box (a notoriously ill-defined task; e.g., see [1]). Instead of presenting numerical results, it would be beneficial to show how this approaches enhances our knowledge about LLMs (similar to the presentation of Bills et al.). For instance, why does a model hallucinate or, simpler, how polysemanticity of neurons impacts text generation. I noticed that the appendix provides some illustrative examples but a thoroughly discussion and concise conclusion is missing. \n\n- **Quality:** Certain choices of design decisions are not well-justified. For instance, why is the max-norm a good choice for the ordering, why a random sample of top quantile, or how is top defined. In order to understand the sensitivity of the proposed method it is required that the paper presents some kind of ablation study or discusses the selection of these hyperparameters. Otherwise it is not clear whether the observed results are only good by chance or cherry-picked. Another example is the usage of cosine similarity where results suggest that this could be inappropriate [2].\n\n- Minor issue **Quality**: References should be to publications not arXiv (e.g., [4])\n\nI will elaborate on a few more things in the questions section.\n\n[1]: Cynthia Rudin: Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. 2018.\n\n[2]: Steck et al.: Is Cosine-Similarity of Embeddings Really About Similarity? 2024.", "was_revised": true, "final_justification": "I appreciate the authors response and the entire rebuttal. I will increase my score to **borderline accept** assuming that the authors will apply the promised changed. \n\nDuring the rebuttal, I have read the reviews of the other reviewers that evaluate the contribution (the improvement of Bills et al. approach) as significant and valuable. Hence, I have reconsidered my assessment and agree that this an interesting contribution for the community.\n\nMy change in the assessment is also based on the promises of the authors to change the paper as discussed during the rebuttal (e.g., presentation of pseudo code). Moreover, I urge the authors to realistically state their contributions and to not oversell it unrealistically (e.g., overselling of \"understanding\" or \"case study\"). A realistic reflection of the contributions and limitations is appreciated by the community and the conference to know in advance what to expect and where are points for improvement. Maybe the usage of the limitations section is a good place for that.\n\nOnce more, I thank the authors for the good rebuttal and wish all the best.", "rebuttal": "We thank Reviewer i1y8 for the constructive review. However, we believe there are some misunderstandings regarding our paper. We'll address your concerns as follows.\n\n---\n\n**W1.** However, it is not clear how the proposed method helps in understanding LLMs compared to Bills. […]\n\n**A1.** Thank you for the comment. We respectfully disagree that our paper does not help with understanding LLMs. In fact, our work improves this understanding in several important ways:\n\n1. **Improving explanation accuracy matters.** SAE research aims to understand LLMs by interpreting their features. But if those explanations are inaccurate—too broad or missing polysemanticity—they hinder, rather than help, our understanding. Many current auto-interpretability methods suffer from these issues [1,2]. Our work directly addresses them.\n2. **We are the first to tackle overly broad explanations.** Prior methods (like [1]) use evaluation metrics that don’t penalize overly inclusive explanations. This allows inaccurate, misleading outputs. Our method introduces *semantically similar negative sentences* to detect and fix overly broad explanations. Figure 3 shows that our tree-based explainer reduces this problem through iterative refinement.\n3. **We are the first to systematically capture polysemanticity.** Polysemanticity—when a feature activates on unrelated concepts—has been a known problem in XAI. We use a structured explanation format that not only captures this property but also lets us quantify it. In experiments, structured explanations consistently outperform unstructured ones across datasets and settings (see responses to Reviewer q24V and A4).\n4. **We analyze how feature complexity and polysemanticity evolve.** Using our methods, we examine how these properties change across layers and models (Figure 5). For example, middle layers tend to encode more complex concepts, and stronger models (GPT-2 < Gemma 2 < Llama 3.1) have more complex and polysemantic features. These patterns deepen our understanding of LLM internals.\n\nRegarding the reviewer’s point about L316-317: those lines refer only to the limits of holistic activations and training negatives, which we report as negative results. The new methods we introduce—semantically similar negatives, structured explanations, and the tree-based explainer—directly address these problems and show clear performance gains (Figures 3 and 4).\n\nIn short, our paper directly tackles major open issues in SAE interpretability and provides tools that improve both explanation quality and our understanding of LLMs.\n\n**References:**\n\n[1] Bills et al. Language models can explain neurons in language models. *OpenAI blog*.\n\n[2] Paulo et al. Automatically Interpreting Millions of Features in Large Language Models. In *ICML*.\n\n---\n\n**W2.** The contribution of the paper is marginal as it is mainly the approach of Bills et al. with a new prompting strategy.\n\n**A2.** Our approach goes well beyond just a new prompting strategy. The tree-based explainer automatically improves explanations using feedback from validation sets. It produces structured, intuitive explanations and introduces a way to detect overly broad features using semantically similar negative sentences—something not done before. These methods are part of an iterative refinement process that significantly improves explanation accuracy and helps us better understand LLMs.\n\nAlso, our contribution is not marginal compared to Bills et al. That work focuses on generating and evaluating SAE feature explanations but does not address major issues like overly broad features and polysemanticity. As a result, their explanations are often inaccurate. In contrast, we directly tackle these issues and show clear improvements (see our response to Reviewer gHHG). Finally, while Bills et al. is mainly methodological, our paper also analyzes how complexity and polysemanticity change across layers and models, offering deeper insight into LLM behavior (see Figure 5).\n\n---\n\n**W3.** After reading the paper, I'm not sure about the take away of the paper except from what was already know from Bills et al. […]\n\n**A3.** We address how our paper goes beyond Bills et al. in **A1**, but to summarize: our work is the first to systematically tackle the issue of overly broad and polysemantic SAE features. Unlike Bills et al., which focuses mainly on methodology, we use our tools to study the internal structure of LLMs. For instance, we show that middle layers tend to encode more complex and polysemantic concepts, and that stronger models (e.g., GPT-2 < Gemma 2 < Llama 3.1) have more such features. This directly connects to the reviewer’s question—polysemanticity helps models represent more concepts per neuron, making them more capable.\n\nWe also understand the concern about using one black box to explain another. In our main experiments, though, we don’t rely on LLM judges—explanations are evaluated using correlation with ground-truth activations, following Bills et al. We do use an LLM judge to estimate feature complexity, but we design high-quality few-shot prompts and find that its judgments align well with human intuition.\n\n---\n\n**W4.** Certain choices of design decisions are not well-justified.\n\n**A4.** We justify our design choices as follows:\n\n- We use maximum activation to rank sentences so that a few repeated activating words don’t dominate the score.\n- We sample randomly from the top quantile instead of just taking the top-activating sentences to better capture polysemantic features. Looking only at the top examples can make a feature seem monosemantic. For example, the 1st feature in Gemma 2 9B seems to focus on \"instrument,\" but looking deeper reveals it also activates on \"warn\" and \"warning.\" Sampling more broadly helps uncover this.\n- As for how we define \"top,\" we split the dataset into 1000 activation quantiles and sample from the top one. To address concerns about cherry-picking, we ran experiments on different quantiles on the 10th layer of Gemma. In all cases, the tree-based explainer consistently outperforms the one-shot one.\n\n| #quantiles | One-shot score | Tree score |\n| --- | --- | --- |\n| 100 | 0.230 | 0.282 |\n| 500 | 0.356 | 0.376 |\n| 1000 | 0.428 | 0.482 |\n| 2000 | 0.387 | 0.493 |\n| 3000 | 0.456 | 0.513 |\n\n---\n\n**Q1.** In the abstract it is stated the tree-based method \"refines explanations\". This is assessed in 5.3 (Feature Complexity) by using an LLM judge. How can we sure that LLM judge does a good job?\n\n**A5.** Thank you for the question. We believe there may be a misunderstanding. Section 5.3 is not used to evaluate the quality of explanations from the tree-based explainer. That evaluation is done in Section 5.2, where we use the correlation between simulated and actual activations—just like in Bills et al.\n\nSection 5.3 instead studies how feature complexity changes across layers and models. Here, we use an LLM judge, but we design high-quality, human-written few-shot prompts to guide it. Based on manual checks, the LLM’s ratings align well with human intuition.\n\n---\n\n**Q2.** What does \"close counterexample\" mean? How is closeness mathematically defined?\n\n**A6.** \"Close counterexamples\" are sentences that are semantically similar to the top-activating sentences but on which the SAE feature doesn't activate. They are useful for exposing overly broad explanations, as shown in Figure 3. The definition involves semantic similarity which is challenging to describe mathematically.\n\n---\n\n**Q3.** The creation of the child explanations is without a score. How is it possible then to perform the pruning in step 3?\n\n**A7.** The child explanations are scored on a validation set, and these validation scores are used for pruning. This process is iterative: at each iteration, we generate variations of current explanations, score them on validation sets, then prune the explanations using validation performance. We'll formalize this into pseudo code in our paper to improve clarity.\n\n---\n\n**Q4.** Does no ground-truth mean zero activation?\n\n**A8.** Yes, no ground-truth means the SAE feature has zero activation on the sentence.\n\n---\n\n**Q5.** […] However, looking at Fig. 3 it is above 1. How is this possible?\n\n**A9.** The value in Figure 3 refers to the number of false positives in the sentence. To get the ratio of false positive elements, we need to divide it by the length of sentences.\n\n---\n\n**Q6.** L306: Is there any evidence for this hypothesis?\n\n**A10.** On L306, we hypothesize that the tree-based explainer is capable of packing polysemantic meanings into a single explanation. This is mainly derived from case studies. We provide examples of one-shot and tree-based explanations below, where the one-shot explanation misses polysemantic components and achieves a lower score, while the tree-based explainer successfully packs polysemantic concepts into a single string.\n\n**Example 1:**\n\n- One-shot (score: 0.738): instruments and related equipment or devices\n- Tree (score: 0.894): The neuron is looking for mentions of specific types of devices, tools, or equipment, particularly when referred to by their common names or technical terms, such as \"instruments\", \"instrument\", or warning messages like \"warnings\" or \"Warning\".\n\n**Example 2:**\n\n- One-shot (score: 0.126): references to software licenses and technical documentation metadata, such as version numbers and programming terms.\n- Tree (score: 0.270): The neuron is looking for specific keywords related to licensing, particularly \"License\" and \"Apache License\", as well as some LaTeX-related keywords like \"article\" and specific package names. It also activates on certain version numbers and technical terms like \"time\" in the context of Java imports.\n\n---\n\nWe hope our response addresses the reviewer's concerns and would appreciate your consideration in raising the score. Should you have any further questions, we welcome the opportunity to address them during the discussion period."}, {"review_id": "O9c4J9mkZ4", "reviewer": "Reviewer_LBic", "summary": "This paper attempts to address overly broad explanations produced by autointerpretability methods by introducing a strategy to falsify generated explanations through sourcing similar sentences as negative samples, hence making them more precise. In addition, the authors propose structured scheme to produce fine-grained explanations along with tree-based method for explanation refinement. The experiments confirm the bias of current autointerpretability methods toward recall at the cost of precision, and also show how the structured explanation scheme and tree-based refinement can improve generated explanations. Lastly, the authors provide analyses of how complexity and polysemanticity of the produced explanations evolve across layers by using an external LLM (instruction-tuned Gemma2 27B) as a judge.", "questions": "- L44: Explanatations → Explanations?\n- L25: Missing citation to [(Bolukbasi et al., 2021)](https://arxiv.org/abs/2104.07143).\n- L200: You mentioned that the explainer LLM is prompted such that each component corresponds to a monosemantic concept. Is there any guarantee for that?\n- Do you have any conjecture regarding how different width of SAEs might affect the quality of explanation for polysemantic feature?", "limitations": "Please refer to the weakness section.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The issue of how autointerpretability methods are heavily biased towards recall is relatively understudied. This paper addresses a crucial gap in current approaches by proposing similarity-based method for sentence sourcing to evaluate feature explanations.\n- Using a tree-based jailbreaking technique to automatically generate and refine explanations is a valuable contribution. Although it is computationally expensive as alluded in the limitation section, the improvement over one-shot generation is quite significant.\n- The proposed methods in the paper are simple to implement, making it easily applicable in various settings.\n\n**Weaknesses:**\n- There is no human evaluation to confirm whether the proposed methods improve automatic interpretability of SAEs in a meaningful way. Furthermore, using LLM as a judge potentially introduce another biases that could affect evaluation reliability, particularly for concepts across different languages or domains.\n- All experiments are conducted on an uncopyrighted subset of the Pile, which may not fully represent all possible concepts that a model might encode. A more detailed discussion on how the dataset choice might affect the results would be beneficial.\n- While features can be polysemantic as defined in L46 and L201, more thorough explanations of how this can be different from the property of features in SAE latent space that is known to be more monosemantic and interpretable (L31) is needed, to provide readers with clarity.", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer LBic for the constructive review. We address your concerns as follows.\n\n---\n\n**W1.** There is no human evaluation to confirm whether the proposed methods improve automatic interpretability of SAEs in a meaningful way. Furthermore, using LLM as a judge potentially introduces biases that could affect evaluation reliability, particularly for concepts across different languages or domains.\n\n**A1.** Regarding human evaluation, evaluating all features would be prohibitively expensive. However, we have performed extensive case studies across various aspects. We provide examples of semantically similar negatives falsifying SAE explanations in Appendix C, tree-based explainer generating higher-quality explanations in **A1** of our response to Reviewer gHHG, and structured explanations capturing polysemantic concepts in **A4** of that same response. These case studies confirm our methods generate more accurate explanations for SAEs. We have also provided code that facilitates reproduction and manual inspection of our results.\n\nWhile LLM judges can introduce bias, we don't use them in our main experiments. Explanation performance in Section 5.2 is measured by the correlation coefficient between simulated and ground-truth activations, consistent with previous work [1]. We only use an LLM judge when measuring feature complexity, where we use carefully-written few-shot examples and confirm through manual inspection that the LLM's judgments align with human intuition.\n\n**References:**\n\n[1] Bills et al. Language models can explain neurons in language models. *OpenAI blog*.\n\n---\n\n**W2.** All experiments are conducted on an uncopyrighted subset of the Pile, which may not fully represent all possible concepts that a model might encode. A more detailed discussion on how the dataset choice might affect the results would be beneficial.\n\n**A2.** Dataset choice can indeed affect results. However, the Pile is large, and our 100,000 samples should capture the vast majority of concepts in SAE features. If a concept encoded by an SAE feature were absent from the dataset, we would observe no activations—yet this never occurred in our evaluated features. All features in our experiments showed a significant number of activating sentences, making our results meaningful.\n\n---\n\n**W3.** While features can be polysemantic as defined in L46 and L201, more thorough explanations of how this can be different from the property of features in SAE latent space that is known to be more monosemantic and interpretable (L31) is needed, to provide readers with clarity.\n\n**A3.** Thank you for this suggestion. In L31, SAE features are described as monosemantic and interpretable relative to neurons. The large latent dimension of SAEs allows us to decompose polysemantic neurons into more monosemantic concepts. However, the typical latent dimensions in our experiments (16k to 64k) are still far fewer than the concepts encoded in the LLMs' latent space. Therefore, SAE features must still be inherently polysemantic, making it essential to break down this polysemanticity for accurate explanations. By representing polysemantic concepts in a structured format, we obtain a clear measure of feature polysemanticity and improve explanation quality.\n\n---\n\n**Q1.** Typos and missing citations.\n\n**A4.** Thanks for pointing these out. We'll fix the typos and add the missing citations.\n\n---\n\n**Q2.** L200: You mentioned that the explainer LLM is prompted such that each component corresponds to a monosemantic concept. Is there any guarantee for that?\n\n**A5.** While there is no hard guarantee, we observe that during the tree-based explainer's iterative refinement, it tends to break down polysemantic concepts into monosemantic components. Explanations with clear, monosemantic components achieve higher validation scores and survive the pruning process. Through case studies (e.g., **A4** in our response to Reviewer gHHG), we find this pattern holds across all inspected features.\n\n---\n\n**Q3.** Do you have any conjecture regarding how different width of SAEs might affect the quality of explanation for polysemantic feature?\n\n**A6.** This is an insightful question. SAEs are trained with a sparsity loss in the feature space. As SAE width increases, they tend to decompose polysemantic features into more monosemantic components, a phenomenon known as feature splitting [1]. Therefore, we believe SAEs with higher widths would exhibit lower polysemanticity. However, since the latent dimension of SAEs cannot exceed the number of concepts in natural language, polysemanticity will always exist in SAE features, necessitating better handling methods. Our proposed structured explanations better capture this polysemanticity, resulting in consistent performance improvements.\n\n**References:**\n\n[1] Chanin et al. A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. *arXiv preprint arXiv:2409.14507*.\n\n---\n\nWe hope our response addresses your concerns. If you have further questions, we're happy to address them in the discussion period."}]

Author rebuttal:

Meta-review: This paper proposes a few simple modifications of standard automated interpretability pipelines in sparse autoencoder features, including the tree-based structure that helps to extract more precise explanations and allows better tackling of the polysemanticity. The reviewers are positive after rebuttal, where the authors provided additional qualitative examples, ablation studies, and clarifications between prior work, as well as additional applicability with related literature. The authors are also committed to the promised revisions (in the author final remark) to further strengthen the manuscript. Thus, an acceptance is recommended.
Decision: Accept (poster)