Circuit Insights: Towards Interpretability Beyond Activations
Authors: Elena Golimblevskaia
Aakriti Jain
Bruno Puri
Ammar Ibrahim
Wojciech Samek
Sebastian Lapuschkin
Venue: ICLR 2026 Poster (2026)
Keywords: mechanistic interpretability
automated interpretability
explainable AI
transcoders
large language models
circuits
Primary area: interpretability and explainable AI
TL;DR: Circuit and weight based automated interpretability of LLM transcoders.

Abstract: The fields of explainable AI and mechanistic interpretability aim to uncover the internal structure of neural networks, with circuit discovery as a central tool for understanding model computations. Existing approaches, however, rely on manual inspection and remain limited to toy tasks. Automated interpretability offers scalability by analyzing isolated features and their activations, but it often misses interactions between features and depends strongly on external LLMs and dataset quality. Transcoders have recently made it possible to separate feature attributions into input-dependent and input-invariant components, providing a foundation for more systematic circuit analysis. Building on this, we propose WeightLens and CircuitLens, two complementary methods that go beyond activation-based analysis. WeightLens interprets features directly from their learned weights, removing the need for explainer models or datasets while matching or exceeding the performance of existing methods on context-independent features. CircuitLens captures how feature activations arise from interactions between components, revealing circuit-level dynamics that activation-only approaches cannot identify. Together, these methods increase interpretability robustness and enhance scalable mechanistic analysis of circuits while maintaining efficiency and quality.

Published as a conference paper at ICLR 2026
CIRCUIT INSIGHTS: TOWARDS INTERPRETABILITY BEYOND ACTIVATIONS
Elena Golimblevskaia' Aakriti Jain', Bruno Puri Ammar Ibrahim', Wojciech ,Sebasstiann Lapuschkin!
'Department of Artificial Intelligence,, Fraunhofer Heinrich Hertz Institute
DDpartment of Electrical Engineering and Computer Science, Technische Universität Berlin BBIFOLD Berlin Institute for the Foundations Learning and Data
*Centre of eXplainable Artificial Intelligence, Technological University Dublin corresponding authors: {wojeiech, samck, scbastian. Lawpscckinjjaaahiiccc..... fraunhofer.de
ABSTRACT
The fields of explainable Al and mechanistic interpretability aim to uncover the internal structure of neural networks, with circuit discovery central tool for understanding model computations. Existing approaches, however, rely on man- ual inspection and remain limited to toy tasks. Automated interpretability offers scalability by analyzing isolated features and their activations, but it often misses interactions between features and depends strongly on external LLMs and dataset quality. Transcoders have recently made it possible to separate feature attributions into input-dependent and input-invariant components, providing foundation for more systematic circuit analysis. Building on this, we propose WeightLens/ and CircuitLeng two complementary methods that go beyond octivation-based anal- ysis. WeightL interprets features directly from their learned weights, removing the need for explainer models or datasets while matching or exceeding the perfor- mance of existing methods on context-independent features. CircuitLens captures how feature activations arise from interactions between components, revealing circuit-level dynamics that activation-only approaches cannot identify. TTgether, these methods increase interpretability robustness and enhance scalable mecha- nistic analysis of circuits while maintaining efficiency and quality.
INTRODUCTION
Large language models (LLMs) have seen rapid adoption in recent years, including in sensitive domains such medical analysis (Singhal et al. 2023). Despite their remarkable capabilities, derstanding the internal mechanisms of these models is crucial for safe and reliable deployment, yet our knowledge in this area remains limited (Olah al. 2018 Lapuschkin et al. 2019 [Sharkey al., 2025). Several methods have emerged in the fields of mechanistic interpretability and explainable Al understand how models encode and utilize mechanisms that influence outputs (Olah et al.) [020]] Achtibat et al. [222] Dreyer et al. (025). Much of the existing work focuses on circuit discovery, identifying subgraphs responsible for specific tasks (Conmy et al. (023). However, these studies are mainly limited to toy tasks, and understanding the roles of individual neurons and attention heads
still requires extensive manual analysis (Elhage et al., 2022 Wang et al. 2022 Bricken et al. 2023)
Automated interpretability methods were proposed to address these limitations. Bills al. (2023)
suggested using larger LLMs to analyze activation patterns of MLP neurons and generate natural lan- guage descriptions. Although promising, such approach faces the fundamental challenge of poly- semanticity of MLP neurons, making them inherently difficult to interpret. This bottleneck prompted the development of Sparse Autoencoders (SAEs), which decompose activations into monosemantic features [Bricken et al. 2023). advancing interpretability pipelines (Templeton et al. 2024 Paulo
fithab.com/ggolimllesskaii/a/ Waihskaaaaiiiiilllllmomo

---

Published as a conference paper at ICLR 2026
et al. 2025a). More recently, transcoders were introduced in Dunefsky et al.) (2024) and Ge et al. 2024 as an alternative approach for extracting sparse features. Unlike SAEs, which reconstruct ac- tivations, transcoders sparsely approximate entire MLP layers, while maintaining clear separation between input-dependent and weight-dependent contributions. This architecture enables efficient circuit discovery and provides direct attributions to other features, attention heads and vocabulary.
Despite these advances in sparse feature space construction, automated interpretability remains heavily dependent on explainer LLMs, which shifts the solution the black box problem to yet another black box LLM, resulting in notable safety risks that may produce unfaithful or unreli- able explanations (Lermen et al., 2025). Its effectiveness is influenced by the prompt, fine-tuning strategy, and the dataset used for generating explanations. Furthermore, sparse features can still be challenging to interpret (Puri et al., 2025), as they may activate on highly specific patterns that are not easily captured by analyzing activations alone, may be polysemantic (Kopf et al., 2025).
In this work, we focus on automated interpretability grounded in model weights and eircuit structure. Our contributions consist of the following methods:
WeightLens: a framework for interpreting models using their weights and the weights of their transcoders, eliminating dependence on the underlying dataset and explainer LLMs. Descriptions obtained via Weightl.en match or exceed baselines for token-based features. framework for arruuit-based analysis of activations, extending interpretabil- ity to context-dependent features by (i) isolating input patterns triggering feature activations and (ii) identifying which output tokens are influenced by specific features.
CircuitLens:
Circuit-based approach uncovers patterns missed by octivation-onle methods, simplifying the ex- plainer LLM's task and improving robustness smaller datasets by combining weight-based and circuit-based information. Additionally, it handles polysemanticity through circuit-based clustering and combining their interpretations into unified feature descriptions.
RELATED WORK
Recently, a series of works focused on building automated interpretability pipelines for LLMs (Choi et al. [024] Paulo et al. [2025a] Puri et al. [2025] Gur-Arieh et al. (025). Those approaches follow the framework introduced by Bills et al. (202)), which consists of passing a large dataset through model, collecting maximally activating samples for each MLP neuron, and then forwarding those samples with their per-token activations to larger LLM to generate natural language descriptions.
Several studies refine this pipeline by focusing on prompt construction and description evaluation. For instance, [Choi et al. (2024) and Puri et al. (2025) examine how factors such as the number of samples and the presentation of token activations affect description quality. Choi et al. (2024) further fine-tune an explainer model to produce descriptions conditioned on feature's 's activations. Beyond input-based evidence, Gur--Arih al. 2025) incorporate output-side information, analyzing not only what inputs trigger a feature but also how that feature influences the model's logits.
These pipelines applied to different representational units. Early work focuses on MLP neurons Bills et al. 2023 Choi et al.) 2024), while later studies extend them to SAE features, which are generally more interpretable and often monosemantic (Templeton et al. 2024 Gur-Arich et al.,
2025 Paulo et al. 2025a Puri et al. 2025).
2024 introduce transcoders, a sparse approximation of MLP layers that decomposes attributions into input-dependent and input-invariant components. Variants such as kkip-transcoders (Paulo et al. 20256) and cross-layer transcoders (CLT's) (Ameisen et al. 2025) have also been explored, demonstrating through qualitative and quan- titative analysis that transcoders match or exceed the interpretability gained through SAEs. Through case studies, Dunefsky et al. (2024) show that transcoder circuits can be used for interpreting feature's 's function, although based mainly on manual analysis.
As an alternative to SAEs,, Dunersky et al. (2024) and Ge et al.
The interpretability of transcoder weights is studied by Ameisen al. (025), who show that while some connections appear meaningful, interference is a major challenge. They propose Target- Weighted Expected Residual Attribution (TWERA), which averages attributions across dataset. However, they find that TWERA weights often diverge substantially from the raw transcoder weights, making the method sensitive to the distribution of the evaluation dataset.
2

---

Published as a conference paper at ICLR 2026
layer!
Torocll Eackn
layer
layer
layer!
layer
[--]
.025 broposed
waiiiinn sagel lutari
back Aold pasa taker
Baaw nats pwweevateeee beat
Londiibuiuon of WW
bacwarrs pas ans
Abribution sscoder
b) Attribution attention
Attribution
transcoder
Figure 1: Types of attributions in transcoders.
Finally, Punrily,Puricaerr et al. 2025) highlight persistent challenge: even though sparse features, specifically in SAEs, are generally more monosemantic than MLP neurons, they can highly specific and activate only certain patterns. This specificity makes them difficult to interpret; either the explainer LLM fails to identify the correct trigger, or the resulting description is too vague to be useful.
METHODOLOGY
3.1 ATTRIBUTION TRANSCODERS
We leverage circuit- information for automated interpretability. Due to their architecture, transcoders enable efficient computation of attributions, which we demonstrate in this section.
As introduced by Dunefsky et al. 22024), given transcoder structure, the attribution of transcoder feature i' in transcoder layer to feature layer > 1' on token can be expressed as:
activation (()))( (flll'')) f(L.i)
input-dependent
input-invarian
where yll,d) E Rddoot denotes the i-th column of the encoder matrix w-(1) Ridme x Xdeamm, and (l',3') dec E Pddmse denotes the i'-th row of the decoder matrix w:(t) Rddesmms dduats, where dicatures is the dimension of the transcoder, dmodel is the dimension of the model, and dieatures > dmodel.
This formulation cleanly separates an input-dependent scalar activation from a fixed, input-invariant connectivity term between features across layers.
Ameisen et al. 2025) propose incorporating Jacobian term into the attribution formulation, which improves the reliability of feature attributions. With this adjustment, Eq. (Ip can be redefined as
activation
(O(Ott) (1SCe ((-(()(( "(
f((.0)
(2)
where the Jacobian is given by
I rr mid [4]
J (--0
(3)
2 @rpoat
with all non-linearities, including attention, normalization, and activation functions, treated as con- stants with respect to the given input, and r!()) [4] and n((2) [t] denote the residual streams before and after the transcoder layers 1 and l', respectively (see Figure TT).
mial
Similarly, we can measure how much previous tokens contributed to the activation of our analyzed feature (4, i) on token through specific attention head. This can be done via attribution that attention head, as presented in Figure Tp (Dunefisky et al. 20244. For an attention head h at layer l' with l' < 1, the contribution token s through (V', h) to feature (I, at token can be expressed as
sc Ab) (T-(()) [t), (1l2) ((WW)l))) (LL) [(')(s))
(s])
pre
attention from
projection feature onto head output
3

---

Published as a conference paper at ICLR 2026
Step 1: EmbeediingUnemmmddingg Projections Stop 2: Weiee Feature Connections
Step Validation & Description
Mal
C
twee
All
0 O
A1
ALL
M81
ALL
logetee
ud12
all, tout
0 O
Moll
tout
tout
nore
143)
tout
O
Descriptio:: ALL, tout awing serss, sogether
Figure 2: Input-invanie inptr-invariant feature analysis with WeightLens. Step 1: Project the feature's encoder and decoder using the embedding and unembedding matrices. Step 2: Identify top contributing features from earlier layers. Step 3: Validate candidate tokens via forward pass, keeping only those that activate the feature; use them and the output-based outliers for constructing the description.
where ppr [s] denotes the residual stream at token 8 before the attention block in layer 1', Will ()) is
OV
the output-value matrix of head h, and plle) is the encoder vector of feature (1, 2).
The contribution of our target feature (1, to the output logit y/t] at token t is demonstrated on Figure II and can be expressed as " (f Jo-0L Wolsv/ll)
where f((.)) is the decoder vector of feature (1, i), JV++(.0 (4] is the Jacobian from the final residual stream to the post-residual of feature (1, at token t (calculated as before with nonlinearities and attention patterns considered as constant on given input), and Wuli, y/t]] the unembedding vector for token y(t)
3.2 WEIGHTLENS: INPUT-INVARIANT AUTOMATED INTERPRETABILITY
The input-invariant connections from Eq. (T) provide a useful foundation for interpretation of transcoder features, as demonstrated by Dunefsky et al. (2024). Qualitatively, we observe that many strong weight-based connections align with the connections revealed by attribution graphs on feature-relevant inputs, as shown in Appendix E.1 These connections help identify which tokens contribute to a feature, either directly through the embeddings matrix or indirectly by activating upstream features. At the same time, it can be difficult to distinguish meaningful connections that persist when an input is provided to the model, from those that are simply noise. To build on this idea, make the following assumption:
Input-invarian connections indicate meaningful structural relationships only if their magnitude significantly exceeds that of other connections, making them statistical outliers.
Assumption 1:
Since many features are context-dependent, weight-based analysis alone can be misleading. Con- nections identified under Assumption 1 do always hold across contexts (see Appendix [E.3). Since our goal is to identify tokens that reliably characterize a feature, we introduce a validation step to check whether the feature activates on certain tokens in isolation depends on context, in which case no stable token-level description can be assigned. Formally, we state the assumption:
token is strongly supported by imput-invarian connections to the feature (weights) and reflects the feature's 's inherent behavior, then the feature should activate on this to- ken regardless of context or in the absence of
Assumption
Based on these assumptions, we generate token-based feature descriptions, where each feature is characterized by tokens that activate it and by tokens it promotes the output. The procedure is applied layer by layer, progressing from earlier to later layers. The following steps, illustrated in Figure 2 describe the full process.
1. Embedding and unembedding projections: Project the feature encoder vector fene into the input embedding space using Wemb fane. and, following Arich et al. (2025), the feature decoder vector faee into vocabulary logits via faee WG. Detect outliers using z- scores (Barnett & Lewis. 1994): embedding-spean input-based part of the feature description, and logit-space outliers form the output-based part (outlier threshold for GPT-2 (Radford et al. 2019); 4.5 for Gemma-2 2--m (Riviere et al. 2024) and 1.2m-332-11B (Dubey et al. 2024
outliers are candidate tokens for the
4

---

Published as a conference paper at ICLR 2026
2. Analyze weight-based feature connections: For each earlier layer 1' L, compute W
(!!) to identify top contributing features (outliers with z-score threshold 3). Inherit their token deseriptions as candidate tokens for the input-based part of the feature description.
3. Validate candidate tokens and generate description: Keep only tokens that activate the feature in a forward pass. Combine these validated input-based tokens and (optionally) the output-based tokens form the final feature description. Note that output- autput-based tokens might be noisy and we cannot validate them, therefore this part is optional and its necessity is further evaluated in the Section 4
Token-based features often respond to multiple forms of the same word. To process the obtained set of tokens and produce coherent feature description, we apply lemmatization (Bird et al. 2009) to the generated descriptions. This step consolidates different inflected forms into single canomical form and can be considered lightweight alternative LLM-based postprocessing for cleaning and standardizing the descriptions.
3.3 CIRCUITLENS: AUTOMATED INTERPRETABILITY BASED ON CIRCUITS
In transformer models, much of the behavior depends context and cannot be interpreted from weights alone. Following Bills et al.] (023), most automated interpretability methods select sam- ples that strongly activate a feature across a large dataset and pass them, along with per-token acti- vations, to an explainer LLM to generate description. However, activations do not always reveal which input tokens caused the feature to fire, and noisy or polysemantic inputs can further reduce description quality. Circuitleens addresses these limitations by providing a circuit-aware approach feature interpretability. Below, we describe the whole process of automated interpretability with CircuitLens.
Activation Caching and Sampling Most prior works (Bills et al. 2023; Choi al. 2024: Gur-
Arieh et al., 2025 Puri et al., 2025) focus on analyzing the most highly activating examples of feature. This approach is especially sensible for MLP neurons, whose activations are often noisy and unstructured. In contrast, both SAEs and transcoders are explicitly designed to yield monosemantic features. For this reason, we aim to analyze the entire distribution a feature's 's activations, in order capture the broader concept(s) that drive its behavior. For this, cache maximum sample activations over the whole dataset, which is feasible due to high sparsity of features.
Because activation distributions are typically highly skewed toward zero, we adopt inverse- frequency quantile sampling (De Angeli et al.] (022) to ensure sufficient coverage of rare but strongly activating cases. Specifically, activations
partitioned into B = 20 quantile bins. Here, ny denotes the number of activations in bin b, and each activation in b is assigned weight w; = 1/ng with a 0.9, and corresponding normalized probability p; = w:/ Σ, Wj.
Finally, we sample N 100 activations without replacement, producing diverse set of contexts that up-samples tail cases while still maintaining broad overall coverage.
Circuit-Based Patterns Detection A central challenge in interpreting feature activations is that raw activation values do not always reveal what triggered activation of given feature. Sim- ply highlighting token activations and prompting language models often yields vague or generic explanations such "variety of words variety of topics.
To address this, we focus on identifying patterns in the data that both lead to a feature's 's activation and determine what parts of the output, produced by the model, influenced by the feature.
Input-centric focus: Using the attribution formulation in Eq. we extract (attention head, token) pairs that provide strong contribution for feature's 's activation. Outlier pairs are selected based on their z-score relative to the distribution of contributions, ensuring that only the strongest connections are retained. We then mask the original input sequence, keeping only tokens that either directly activated the feature or contributed significantly through attention. This procedure isolates interpretable token patterns underlying the acti- vation of a feature, as illustrated in Figure Dp, where the feature activates on references to already mentioned entities.
36/1333
5

---

Published as a conference paper at ICLR 2026
Output-centric analysis: Using Eq. (5). we evaluate whether the analyzed feature tributed to the prediction of the generated tokens after being activated. This highlights which output tokens were influenced by the feature and thus provides an estimate of its downstream impact, as shown in Fi
By masking only the input tokens that activated the feature and highlighting only the output tokens influenced, we pre-identify the relevant patterns ourselves. This removes the burden from the explainer LLM of searching through the full context and instead lets it focus on describing the common concept behind these selected inputs and outputs, hence simplifying its task.
Cireuit-Based Clustering In order simplify the explanation task further, want the explainer LLM to receive clear, monosemantic inputs. However, a single feature can respond to multiple entangled concepts, which remain hard to interpret even in SAEs (Kopf al.) (025). Semantic clustering of activations in embedding space is not always sufficient, since it ignores the underlying causal, circuit-level mechanisms.
We propose circuit-based clustering: for each input. we collect contributing elements, such as transcoder features and token/attention head pairs via Eq. ( and Eq. (4) respectively. Significant features (I', i') and attention head contributions (1, h, A), where A is the relative token position, are combined into single vector. Sparse activations ensure that each input has only a few contributors.
To reduce noise, we apply frequency filter, retaining a feature or head only if it appears in at least fraction p of inputs: [[:ff,,)) 2 p. where S, is the contribution set for input j. This step removes features and heads that contribute only for isolated inputs and are unlikely to reflect consistent circuit-level behavior relevant to the feature activation, as well as reduces the size of the set, making subsequent clustering more robust and computationally efficient.
We then compute pairwise Jaccard similarities [[A, B) S n / USB forming an n X SA
matrix. Clusters are extracted via DBSCAN (Ester et al., 1996) on this similarity matrix, which is robust to noise and does not require a predefined number of clusters.
Generating Descriptions Sampled inputs are first analyzed from input- and/or output-centric per- spectives, then grouped into clusters according to the detected circuits. Each cluster is interpreted independently using an explainer LLM (see Appendix F.2 for prompts). Rather than providing full inputs with highlighted activations or token-activation pairs, we supply only the detected pattern, marking the single most activating token. Finally, for each feature, the explainer LLM synthesizes unified description from the individual cluster-level interpretations.
3.4 EVALUATION
All evaluations use the FADE framework (Puri et al. 2025). which relies LLM-aa--ajuudee assess alignment between a feature and its cnttural-lanuugee description across four metrics: Clarity, Responsiveness, Purity, and Faithfulness. This evaluation setup follows prior work showing strong correlations between such automated judgments and human assessments of description quality.
Clarity measures whether the description expresses the concept clearly enough for the explainer LLM to generate synthetic inputs that activate the feature. Responsiveness checks whether the feature activates more strongly on these concept-driven inputs than on randomly sampled dataset. Purity measures whether the feature activates primarily on inputs that match the described concept, or also on unrelated inputs. Faithfulness evaluates how strongly the feature's activation influences the model's output in the direction implied by the description. A review of automated interpretability metries, along with details FADE's computation procedure, is provided Appendix A.2
WEIGHTLENS: EVALUATIONS AND FINDINGS
We analyze the interpretability transcoders for GPT-2 Small (Dunetsky et al. 2024), Gemma-2- 2B (Lieberum et al.) 2024). and 1.2ma-3.2-1B (Paulo et al.) 2025b), focusing Gemma-2-2 for qualitative analysis. We evaluate and compare the following setups:
WeightL descriptions composed solely of validated tokens that activate the feature.

---

Published as a conference paper at ICLR 2026
We ghitLns
Veinttensebu
Neuronpooie
Macsce
Best
C
R
10
16
25
Layer
Figure 3: Evaluation of Gemma-2-2B transcoder descriptions (C= Clarity, R Responsiveness, P
Purity, F Faithfulness). Methods: WeightLens variants. Baselines: Neuronpedia and MaxAct*.
Weightl.ens+Out: WeightLens descriptions augmented with tokens promoted by the fea- ture, derived from its unembedding projection.
WeightLeens++OtLLLLMM. descriptions refined using an explainer LLM (gpt-4o-mini- 2024-07-18) instead of lemmatization, generating a concise single-line summary based on both activating and promoted tokens (see Appendix F. for details).
With these setups we aim to demonstrate the effect of including embedding- based contributions, output-basee based signals, and LLM refinement on feature description quality. They are further compared to activation-base methods, specifically descriptions available on Neuronpedia (Lin 2023), as well as generated via MaxAct* method presented by Puri et al. (2025), details are in the Appendix D
Across 250 features per layer, Weightlenns methods perform on par with or better than activation maximiza- tion methods. They generally achieve higher scores on Clarity and Responsiveness (Figure 3 while activation- based methods tend overgeneralize, leading to lower scores. However, activation maximization attains higher Purity, highlighting that many features might be context- dependent which is not discovered through Weightl
GFF2 Small Genme 28 Li3r
byer
loyra
Furthermore, these results demonstrate that although an LLM can produce more general results and filter out noise in the output logits, its utilization is not mandatory in this analysis, since the results are comparable.
Figure 4: Percentage of validated fea- ture descriptions per layer obtained via WeightLens.
Faithfulness is low across layers and meth- ods, with the rise in the later layers, likely due the transcoder architecture: in con- trast to SAFs, which decompose the full residual stream, transcoders write into it like MLPs. Therefore, steering single feature rarely produces large effects due to model's 's redundancy. Modifyyng the faith- fulness metric to evaluate interventions on entire circuits
1.0
Clarity
Fathruiness
Respendiveness
batenbased
Party
Contect-bused
0.8
Metric value (mean)
0.4
groups of features rather than individual features might bring more trustworthy evaluations.
0.2
10
12
18
21
Layer
Layer-wise analysis of the validated fea- tures demonstrates, that token-based inter- pretability varies strongly with depth (see Appendix G for detailed results). Early layers exhibit clear token-level structure. making them well-suited for weight-based analysis, with many features activating reliably on spe- cific tokens. In Gemma-2-2B, however, early layers perform slightly worse in terms of descriptions quality than later ones, as demonstrated in Figure [3] reflecting their high activation count (fo of 70-88 for layers 0-7 in contrast to of 13-41 for layers 18-25
Figure Description quality gap between token-based and context-dependent features based on Neuronpedia descriptions.

---

Published as a conference paper at ICLR 2026
CircuitLens-Input Circuittens-Full
CircuitLens-Full CircuitLens-inptt (top)
CircuitLens-Fi (top) Neuronpedia
MaxAct
Best
0.0
1.0
0.0
10
12
15
18
21
23
25
Layer
Figure 6: Evaluation of Gemma-2-2B transcoder descriptions (C Clarity, R Responsiveness,
Purity, F = Faithfulness). Methods: CircuitLens variants. Baselines: Neuronpedia and MaxAct*.
Number token-based features, as well as the quality their descriptions, drops in the middle layers of Llama and Gemma, presented in Figure 4 consistent with prior interpretability anal- ysis (Choi et al.) 2024), but not in GPT-2, similarity to the results observed by [ills et al.] (2023). likely explanation the use of RoPE Llama and Gemma, which introduces additional non- linearities. Within Gemma-2-2B, layer 12 is the least interpretable based on weights: despite high sparsity (lo = 6), it contains few validated token-based features. Majority of the features this layer are extremely context-dependent and encode very specific patterns.
Later layers partially recover terms of presence of token-based based features, though still below early- layer levels. For instance, Gemma-2-28 layer 21 (fg = 13) shows strong interpretabilityy with token- based features often acting as key-value pairs that map input tokens to predictable collocations (e.g.. "apologize for", "will be ").
Additionally, we observe that most interpretable features are token-based. Only subset of features receive validated input-invariant descriptions: 32.7% for Gemma, 58.8% for GPT-2, and 25,4% for Llama. However, when the weight-basee based descriptions fail, activation-ban -based ones also perform poorly, presented in Figure 5
CIRCUITLENS: EVALUATIONS AND FINDINGS
For the input-dependent analysis, evaluate the following approaches, implemented within the
CircuitLens framework:
descriptions derived from the activation patterns of the feature, ob- tained via attribution to attention heads;
PrrcuiLLensFFullee descriptions based activation patterns obtained through attributions to attention heads, augmented with tokens, which generation was influenced by the feature. WeightL (WL) + Cireuitl.ens-Fulll: circuit-based descriptions enriched with weight-baset tokens obtained via WeightLens, which are incorporated when merging cluster-level descriptions into full feature description.
For generating cireuit-based descriptions, we use sampling, as described in the Subsection 3.3 on a relatively small dataset of 24M tokens (see Appendix BB. In addition, eliminate the factor of the dataset influence, we compare the circuit-based analysis performed on the same data, as used in Max i.e., on a large dataset (2.3B tokens) with sampling from the top, as presented in Appendix B These results are marked by (top), e.g., Strcuitl.ens-Inpu (top). For comparison, we use the same * and Neuronpedia baselines in Section Computational requirements for CircuitLens demonstrated in Appendix
Evaluation results, presented in Figure show that circuit-based methods still depend on the dataset, with descriptions from the larger dataset demonstrating the strongest performance across layers, particularly on input-centric metrics. However, descriptions derived from the smaller dataset re- main competitive and in some dataset. Combining weight-basee based and circuit- based analysis further reduces sensitivity to dataset size and distribution, making interpretability more robust.
outperform activation based baselines, generated on the larger
8

---

Published as a conference paper at ICLR 2026
PNOOATTO sealpred bowe
Layard Layer?
Layar
-6%
Laser 22
peccentaee irbeeved
45%
Laoee
Jayer
Layer
Layer
6%
30%
20%-
16-
45%
Parouraage
Cctput souitio Infuerood
(a) Smoothed histogram of the fraction of features per layer influencing given percentage of outputs, with hoth axes square-root transformed.
(b) Fraction of features in each layer that contribute to generated outputs, plotted by output position relative to the activating token.
Figure 7: Influence of features on the new 15 generated tokens from the position of the maximally activating token on given feature.
Imputs, activating L L12F619
Obtained patterns
Sparse feature descriptions of-
Quster A reforence to presiousty introduced mamomatical or technical sntity via thhe or "his"
ten have low Clarity, as demon-
is orthogonal al
parformed statistical
of again, since quantty
quantity this operator
strated by Puri et al. (2025): they fail to specify what precisely triggers feature's activation.
If has been argued, that this aperator
are edges SS-SS, the average dogree of this gmap
[...] this grash
Glaster
mendaaned
demonstrative
Analysis of the metrics distribu- tions reveals that circuit-based methods yield far fewer features with extremely low clarity com- pared to purely activation- based approaches (see Appendix G Additionally, both qualitative and quantitative evidence indi- cate that sampling from the full distribution, though memory-intensive, provides a more faithful picture of general feature behavior.
Jupiter's mous magnetic
the planet
Jupiter ]..] the planet
gataxy Itype (ATSAArrsss)) nott--rorttwes the naarg
galaxy [.] the galaxy
Kepler- 22b
habitable
Kapier
planet
reference
mathernttcall
There are no contractions betwee SOS and $TS since the formet combinatian of an Innar type il and an nutor typa with the lattar
о L... the former
typa il [..] the lattar
-.. Lied to related them direclly. probatlyy because Uhe formet -SibetaS instead SiganmaS
the forner
same, but for the latter
gamma L..] the
Figure 8: Clusters of activating inputs for L12F619 and their pat- terns, obtained through attribution to attention heads. Activations are highlighted in green.
We demonstrate in Figure that circuit- based clustering and masking of the in-
INPUT: Subsequent work in this direction shall be conducted on OUTPUT: the basis of the results of the present study. PATTERN: [...] work [-.] conducted on the basis [...]
put allows uncovering patterns that are
not visible based on feature's activa- tion, as illustrated by the feature 619 in layer 12 (LI2F619). Within each cluster, some commonality in activat- ing concepts exists, but no clear gen- eral pattern emerges from activations alone. Isolating the tokens that con- tribute most through attention heads re- veals that each activating entity is either explicitly mentioned or marked def- inite or demonstrative references such as "the" or "this," " as well as "former" or "latter," where contributing tokens align semantically.
INPUT: Therefore, divided subpopulations RGB stars
5286
OUTPUT: the basis their color and the color of their host galaxies.
PATTERN: [..] on the basis of [...
INPUT: The targets, which are likely members of the cluster NGC, were lddentified
OUTPUT: using the 2MASS survey. The targets were selected from the PATTERN: [...] targets [...] were identified [-.] from [...]
INPUT: A mechanism of tunnellng macroscopic distances is proposed OUTPUT: The mechanism based on the existence of a new type of [... PATTERN: [..] mechanism [-] is proposed [..]] based on [.]]
Figure 9: Input- and output-based patterns for fea- ture L21191 obtained via CircuitLens. Tokens isolated through attribution are shown in bold, and activating to- kens are highlighted in green.
Extending to the output, the analysis of feature 1.21F91 demonstrates that functional roles are not fully captured by input activations: while it activates on tokens like "on" or certain verbs, its main effect is generating output phrases such "the basis of" or "based on" (Figure ( These contribu- tions, detectable via logit attribution, illustrate how features influence the output, generated by the model.
Despite its importance, output-based analysis is computationally expensive, as each generated token requires both a forward and backward pass. We generate 15 new tokens per sample to assess how many are needed for reliable results. As expected, early layers rarely contribute directly to the out- put, shown in Figure 7al and when they do, the effect is usually limited the token immediately
9

---

Published as a conference paper at ICLR 2026
following the activating one, Figure 7b In deeper layers, most influence remains on the first gen- erated token, though multi- token output patterns also appear, and might be crucial for interpreting a feature's s function, particularly in later layers (e.g., feature L2 L21F91 presented in Figure 9
Circuit-based clustering aids interpreting polysemantic features (Appendix [E.4). but its effectiveness varies by layer. Layer shows almost no underlying circuit struc- ture beyond its own attention heads, producing mainly token-based activations, according to Figure 4 and av- eraging only 1.05 clusters per feature, demonstrated in Figure [0] Layer 7 exhibits clear polysemanticity. with an average of 4.5 clusters per feature and few single-cluster cases, reflecting extensive circuit formation. Layer 12 is mixed, averaging 2.8 clusters per feature, with many single-cluster highly clustered features. Qualitative in- spection suggests circuit-based clustering captures both main circuits and sub-circuits, and hyperparameter tuning could improve generalization, as can be observed in Fig- 8 It also has the largest share of outlier-only features, highlighting challenges in disentangling circuit- from semantic-driven activations. Layer 21 is similar (2.95 clusters/feature) but with fewer extreme cases, consistent with its role in output generation and flexible participation of features in multiple circuits, as visible in Figure 9
10
Munzera custers per keture
Figure 10: Histogram of number dis- covered clusters per layer with log scale.
Our results demonstrate that both weight-based and circuit-based interpretability provide meaning- ful insights, improving precision and scalability in automated feature analysis. For future work, CireuitLens could be extended architectures beyond transcoders, such as SAEs crosscoders, by leveraging gradient-based attribution methods. Automated clustering of features could be explored further to interpret larger circuits and efficiently identify and steer functional subcomponents, Ad- ditionally, more detailed analysis of clustering hyperparameters would help optimize the balance between capturing polysemantic patterns and avoiding overfragmentation, further enhancing the ro- bustness and applicability of automated interpretability across diverse model families.
CONCLUSION
In this work, we address a fundamental missing piece in automated interpretability pipelines by developing methods that leverage models' underlying structural information. We show that raw activations alone often fail to reveal the patterns driving feature activation, while transcoders, well- suited for circuit discovery, enable efficient incorporation of structural information. Our proposed frameworks demonstrate that structural information allows more scalable and robust interpretability. The weight-based analysis offers an efficient alternative for context- inneeeededdeee features covering up to 58.8% of cases without requiring large datasets or external LLMs. Circuit- clustering isolates groups of activating texts into more interpretable clusters, while input- and output-based analysis further clarifies each feature's 's functional role. Together, these methods reduce dependence on large datasets, improve robustness, and make automated interpretability more scalable and practi- cal for real-world applications. By bridging activation-basee approaches with weight- baaed analysis and circuit discovery, our work opens new avenues for understanding model behavior at scale.
LIMITATIONS
WeightLenn enables feature interpretation without relying on a dataset or an explainer LLM,, but specific to the transcoder architecture and cannot be directly applied to SAEs. In addition, context- dependent features are not well captured by this approach. CircuitLens improves robustness to dataset size and distribution, and reduces the complexity of the LLM's task, yet fully eliminating the need for a dataset or an explainer LLM remains challenging. Measuring the downstream effects of features in transcoders is also limited due to high feature redundancy and their architectural differences from residual SAEs that demonstrate good results in steering. As a result, faithfulness scores remain low across all experiments, including the baselines.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Patrick Kahardipraja for valuable discussions and support. This work was in part sup- ported by the European Union's 's Horizon Europe research and innovation programme (EU Horizon Europe) under grant TEMA (101093003), and by the German Research Foundation (DFG) as re- search unit DeSBi [KI-FOR 5363] (459422098).
REFERENCES
Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnirp: attention-aware layer-wise rel- evance propagation for transformers. In Proceedings of the 41st International Conference on Machine Learning. pp. 135- 2024.
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner. Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi. T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025, URL hhtps: 7/trans foreerrrrrrrre.. rmer
ci rcui issiits... pul 2025/att attributtt on -aaiiionnggraaaaeeeeeeeee graphs wwwsos........e.oooooo..oo...... html
Vic Barnett and Toby Lewis. Outliers in Statistical Data. Wiley, 3rd edition, 1994.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu. and William Saunders. Language models can explain neurons in language models. btttps://ppnaappbllee necron-expl la ner paper/ ndex html, 2023.
windows.net....
Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python. O' O'Reilly Media, 2009. ISBN 978-0-596-51649.99 Btthhttp:://// wwt.:wwwww.ltt.....m nlt.. wwtp:./.wtk..rr/oooo..kk
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. BtL/ttpp://Errsffomeee
pub/ 205/2223/100 htt////mommmmmnicccommmmo hew.nnssemaanicc-fttteeeeeeeeeee featureseeaettteeaaaaaaaa
wwtps/iideeiihimm html
Dami Choi, Vincent Huang, Kevin Meng, Daniel D Johnson, Jacob Steinhardt, and Sarah Schwettmann. Scaling automatic neuron description, 2024. URL https://traas:
prc:/neuron-descriipp
Arthur Conmy, Augustine Mavor-Parker, Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances Neu- ral Information Processing Systems, volume 36, pp. 16318-16352. Curran Associates, Inc., 2023. URL. |nttps://proceedddng/ neuri cc/ http://aperrilleepppp////0000/// File/ htle://ee.bbeea333///aaaa99ppeee otg.://pcaaabbbo---aaaaaa-areree Conftrrceeeeeecccccocmm .dff
Paper-
Kevin De Angeli, Shang Gao, Ioana Danciu, Eric B. Durbin, Xiao-Cheng Wu, Antoinette Stroup, Jennifer Doherty, Stephen Schwartz. Charles Wiggins, Mark Damesyn, Linda Coyle. Lynne Pen- berthy, Georgia D. Tourassi, and Hong-Jun Yoon. Class imbalance in
datasets:
Improving the robustness of the textenn for the classification of rare cancer types. Journal of Biomedical Informatics, 125:103957, 2022. ISSN 1532-0464. doi: https://oo.org/11000116/ http////erreettttcccccccceee///e www.///cieeneeertieeeeeeee
00211.03957. URL https: wttp://wwwww. www.:eeicennccecccco rect. wtt.c/occccccce
5 01532046421002220
Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebas- tian Lapuschkin, and Wojciech Samek. Mechanistic understanding and validation of large ai models with semanticlens. Nature Machine Intelligence, 70))1572-1585, 2025. doi: 10.1038/
442256-025-0108-.2 URL.tttss:/ http:///s//oooo.oooooo//////////

---

Published as a conference paper at ICLR 2026
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models, 2024.
abs /240 21783
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits, 2024. URL [RLItttps///axxi>> org. ht////bb///440.. wttp://ww..rga//////...../ 11944
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds Robert Lasenby. Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superpo- sition, 2022.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density-based algorithm for discovering clusters in large spatial datahases with noise. In Proceedings of the 2nd International Conference Knowledge Discovery and Data Mining (KDD), pp. 226 231. AAAI Press, 1996. doi: 10.5555/3001460033015000 URL WRLhttts:: //tp::///dd.. acm. org wwt.a//www////o//////oo///////// 3554/301//336....
(3001507
Leo Gao, Stella Biderman, Sid Black, et al. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL htttps://arxi..orrrrts
/21bb//2100 00027
Xuyang Ge, Fukang Zhu, Wentao Shu, Junxuan Wang. Zhengfu He, and Xipeng Qiu. Automatically identifying local and global circuits with linear computation graphs, 2024. URL https:/7
arxiv. abs 22////445..111
Yoav Gur-Arich, Roy Mayan, Chen Agassy, Atticus Geiger, and Mor Geva. Enhancing automated interpretabiliity with output-centric feature descriptions, 2025. URL https://aaxiv..rrr
abs/2501.08319
Michael Hanna, Mateusz Piotrowski, Jack Lindsey, and Emmanuel Ameisen, circuit-tracer A new library for finding feature circuits. In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 239-249, 2025.
Laura Kopf, Nils Feldhus, Kirill Bykov, Philine Lou Bommer, Anna Hedström, Marina M. C. Höhne, and Oliver Eberle. Capturing polysemanticity with prism: A multi-concept feature de- scription framework, 2025. /ttttp:///axxi......// org. abs/2b0s..... 15538
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature Communications, 10(1):1096 2019. doi: 10.1038//41467-0199900 https: / http://oo..ogg...///// 103/ 9111///1-11---000------------
URL
Simon Lermen, Mateusz Dziemian, and Natalia Pérez- Carepaeam Antolín. Deceptive automated interpretability: Language models coordinating fool oversight systems, 2025. URL https:
/
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL httts:://axxi:
org/abs/233 2308. 03261
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse
autoencoders everywhere all at once on gemma 2. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 278-300, Miami, Florida, US, November 2024. Association for Computational Linguisties.. doi: 10.18653/v122024 blackboxnlp- URL http:///wwa.aaaoooooooooo///////
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023.
URL https:://// httpt:://ttw....tt. http://wwwwwoooeiiii..o/
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10,23915/
distill.00010. URL UR. htttts:////ssstt wl..illlubub//l//. 8/bui nding-blocks
12

---

Published as a conference paper at ICLR 2026
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distil..00044....... URL
wttp:pp/p/.p...ccits./s//o////s/ http:/ccccccctoooeee.eoe 200mmmizzooiii in.
Josh OpenAl andAchiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni, et al. Gpt-4 technical report, 2024. URL
Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. iwt.umooaaaiccllyyyerrrnnnnnngni features in large language models, 2025a. /t/pp//pp.//aaaxrr.ooor///////2.
of
Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders beat sparse autoencoders for inter- pretability, 2025b. htll:/ttss///rrx.....c//////b
Bruno Puri, Aakriti Jain, Elena Golimblevskaia, Patrick Kahardipraja, Thomas Wiegand, Woj- ciech Samek, and Sebastian Lapuschkin. FADE: Why bad descriptions happen to good fea- tures. Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 17138-17160, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256 madingg-acll8880 URL https: wttll//pss//co..aaaoooo..oo/o///
5. doi: 10. N0.18653/v1/2022
Findings-ach acl. 8817
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl Blog, 2019. URL
are arr.pnaapperrieeemmmmiiaeeeeeese
Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size, 2024. URL httthhttt//aaatt
www.:orraaaaoo abs//bb///ae /408 00118
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonoo Arthur Conmy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud, Stephen Casper, Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger, Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom Mc- Grath. Open problems in mechanistic interpretability, 2025. URL https: pettps://rxivqe //trxvaavvv////////a//eaa//aaav/ org/
2bs/2501.. 6496
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, et al. Large language models encode clinical knowledge. Nature, 620(6509):1 2023. doi: 10. 10.18//441886. 00888/06958--02--6666911- natura............. com/ httpee/e/m.aaattteesssssae..1.. 5866 500-5588----30-0 023-06291-2222
httl:htts::/wmii www.aaattttttteeaeeeeeeeeeeeeeee
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- pub/ former Circuits Thread, 2024. URL Sttpp::r/transsoomeer /
scaling- htw.inggooooommmmtiiiit///////// index.html
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: circuit for indirect object identification in gpt-2 small, 2022. URL
httpss:/erre www./oog////////oooooooo/oooo /211 00593
13

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORK
A.1 AUTOMATED INTERPRETARILITY
Most work in automated interpretability builds on the pipeline of Bills et al. (2023). where a dataset passed through GPT-2 (Radford al. 2019) to collect MLP neuron activations. A larger LLM
(GPT-4; (OpenAl andAchiam et al., 2024g) then generates descriptions based on top-k activating sequences, and is further used as an "activations simulator" to evaluate these descriptions.
To address the polysemanticity of MLP neurons, Bricken et al. (2023) propose sparse autoencoders (SAEs). Templeton et al. (2024); Paulo al. (2025a) extend this pipeline SAEs and show that their features are more monosemantic and interpretable than MLP neurons.
Subsequent work refines automated interpretability in different ways (Choi et al. 2024 Templeton
et al. 2024 Gur-Arieh et al. 2025 Kopf et al. 2025 Paulo et al. 2025a Puri et al.) 2025). Choi
ct al. 2024) fine-tune smaller LLM (Llama-3.1-33 Instruct (Dubey et al. 2024)) ) for neuron de-
scription and evaluation, systematically studying prompt design choices such as token highlighting, loken-activation pairs, and number of examples. Puri et al. (2025) run a broader prompt analysis on SAEs, finding results that sometimes contradict Choi et al. (2024), especially on how token acti- vations should be communicated. They also emphasize dependence on explainer model quality and challenges from the fine-grained specificity of SAE features.
Gur-Arieh et al. 2025) combine input- and output-hasee based analysis, showing that descriptions improve when considering both what feature activates on and what it promotes.
Kopt et al. (2025) address polysemanticity by clustering activating inputs. They sample from the top 1% of activations, embed sequences with gte-Qwen2-1.5B-inse
(Li al. (202)), and apply
k-means clustering into five groups, generating one description per cluster. This consistently outper- forms prior work (Bills et al., 2023, Gur-Arieh al., 2025), demonstrating the benefit of handling polysemanticity directly.
A.2 EVALUATION METRICS
Developing automated evaluation metrics is essential for interpretability research, since manual sessment description quality does not scale. Several main approaches have been proposed: (i) simulated activations, where LLM predicts feature's activation on text samples given its de-
scription (Bills et al.) 2023) Choi et al. 2024); (ii) classifier-basel metrics, where an LIMM judges how strongly
sample relates to feature's description (Templeton al.. 2024 Paulo al.) 2025a Puri et al.] 2025); (iii) synthetic data approaches, where an LLM generates or labels data from description (Gur-Arieh al. 2025 Puri al.) 2025); and (iv) output-based metrics, which evaluate how much feature intluences model outputs (Bills et al. 2023 Gur-Arich et al. 2025
Paulo al. 2025a Puri al. 2025).
2024) are inexpensive but fail to capture many failure modes in description generation (Puri et al.) (025).
conulated-activatiin metrics (Bills et al. 2023 Choi et al.
Classifier-based metrics instead ask a judge LLM to score how related a sample is to description. often on a scale from 0 (not related) 2025). Similar detection-based setups appear in Paulo et al. (2025b), where the model identifies which samples match the concept. Evaluation can then be quantified using AUROC scores (Kopf et al.) 2025), or metrics such as Gini coefficient and Average Precision, which Puri et al. (2025) combine into Responsiveness and Purity scores.
(completely related) (Templeton et al., 2024 Puri et al,,
Synthetic data metrics compare activating and non-activating examples, either LLM-generated or sampled uniformly from dataset (Gur-Arich et al.) 2025 Puri et al. 2025).
Finally, output-based metrics test whether descriptions capture feature's causal effect model outputs. Puri et al.] (2025) propose Faithfulness, where judge LLM rates concept presence in steered generations. Paulo al. (2025a) introduce Intervention Scoring, while Gur-hrieh et al. (2025) apply a similar approach; where the model's 's task is to distinguish outputs produced under feature steering from control generations.
14

---

Published as a conference paper at ICLR 2026
In this work, use the FADE framework to evaluate the obtained descriptions Puur et al. 2025 implements the following metrics:
Clarity This metric uses synthetic data generated by an explainer LLM based on the pro- posed concept description. Feature activations on the synthetic data are compared ref- erence distribution of activations on a natural dataset. The Gini coefficient quantifies how well these distributions separate, yielding score of when the synthetic-dar activations are substantially stronger, and 0 when they are indistinguishable. This metric evaluates how clearly the description captures the underlying concept, which is crucial for narrow, fine-grained sparse features.
Responsiveness An LLM-as-a-judge rates how strongly samples from the natural dataset relate to the described concept (on a 0-2 scale). The Gini coefficient then measures whether the feature activates more strongly concept-related samples than uniformly drawn natural data, producing a score between and 1. This reflects how responsive the feature is to the described concept.
Purity Using the same data Responsiveness, this metric replaces the Gini coefficient with Average Precision to quantify whether the feature activates exclusively on the de- scribed concept also unrelated phenomena. High purity indicates that the description captures all major activation patterns of the feature.
Faithfulness This metric evaluates the downstream influence of the feature. The feature's activation perturbed, positively scaled, negatively scaled, fully ablated, and an LLM judges whether the frequency of the described concept in the model's output changes ac- cordingly.
A.3 CIRCUIT TRACING
A recent advance in mechanistic interpretability is the introduction of transcoders (Dunefsky al. 2024 Ge et al., 2024). Unlike sparse autoencoders (SAEs), transcoders provide structured way to trace how upstream feature activations contribute downstream activations, enabling circuit- level analysis across layers. Their key innovation is the decomposition of feature's 'ss activation into an input-dependent and an input-independent component. The latter depends only on transcoder weights, allowing it to be analyzed separately and efficiently.
Using GPT-2, Dunefsky et al. (2024) demonstrate cases where activating tokens identified through weight-based analysis align with those discovered via traditional activation-basel methods. This indicates that transcoders can support both prompt-specific attribution graphs and global, weight- derived connectivity maps.
Extending this work to Gemma-22 2-2B RRiviere et al. 2024), Ameisen et al. (2025) highlight a major limitation of weight-based analysis: interference from context-dependest components of the archi- tecture. To address this, they introduce target-weighted expected residual attiiution (TWERA), which adjusts virtual weights using empirical coactivation statistics, effectively up- seigetgtting nections between frequently coactivating features. However, they also show that TWERA can sig- nificantly diverge from the original transcoder weights, making dependent the dataset used to compute coactivations and limiting its reliability as a fully weight-based method.
B DATASET PROCESSING
We used the uncopyrighted version of the Pile datase Gao al. 2020) with all copyrighted con- tent removed. This version contains over 345.7 GB of training data from various sources. From this dataset, extract two datasets of size 6GB (2.3B tokens), and 40MB (24M tokens) for gener- ating M descriptions and circuit based descriptions. The extracted portion from the training partition was used to collect the most activated samples based on frequency quantile sampling for the smaller dataset and top percentile sampling for the larger dataset. For evaluations, we utilized the test partition from the same dataset, applying identical preprocessing steps those used for the training data.
15

---

Published as a conference paper at ICLR 2026
Stage
# Features
GPU-hours
Description
Activation caching
all layers
5
Single forward pass over 24M to- kens with sparse activation storage (~~12 GB per layer). This cost incurred once and shared across all analyses.
Input-based feature analysis
1000
8
Sampling 100 examples per feature and performing forward/hackward passes, followed by clustering and bookkeeping.
Output-bascd analysis (5 new tokens)
1000
+4
Additional cost for evaluating next- token effects. Requires one extra forward/backward pass per new to- ken beyond the first.
Output-based analysis (15 new tokens)
1000
+12
Estimated cost for broader next- token evaluation, reflecting scaling the number additional to- kens.
Table 1: Approximate compute requirements for each stage of the analysis pipeline, measured on NVIDIA A100 40GB GPU.
Post processing involves several steps to ensure balanced and informative dataset. First, used the NLTK (Bird et al.] 2009) sentence tokenizer to split large text chunks into individual sentences. We then filtered out sentences in the bottom and top fifth percentiles based on length, these were typically nut-of-distribution cases consisting of single words, characters, or a few outliers. This step helped achieve more balanced distribution. Additionally, removed sentences containing only numbers or special characters with no meaningful content. Finally, duplicate sentences were deleted.
C COMPUTATIONAL COSTS
To make the computational footprint of our method transparent, we report the approximate runtime measured on a single NVIDIA A100 (40 GB). The activation-caching stage requires single forward pass over our 24M-token dataset and stores only the sparse activations of Gemmascope transcoders (about 12 GB per layer for (6m-dimensional transcoders). This step takes roughly 5 GPU-hours and is fully amortized across all later analyses. For efficiency, we also allow caching only selected layers. Input-based feature analysis for 1000 features, including sampling, forward and backward passes, clustering, and data-storage operations, requires about 8 GPU-hours. adds compute depending on the number of next-token effects examined: evaluating single next token does not introduce additional cost, while evaluating 5 and 15 next tokens adds roughly 4 and 12 GPU-hours respectively. These costs scale predictably with the number of features and enable users to choose appropriate tradeoff between interpretability depth and available compute.
nutput-based analysis
D BASELINES
In this work we use two baselines: Neuronpedia, which provides descriptions available on the Neu- ronpedia platform (Lin 2023) and originally generated following Bills et al. (2023), and Max Act*, introduced in Puri et al. (2025).
To the best of our knowledge, the exact procedure used to generate Neuronpedia descriptions is not documented detail. Based on Bills et al. (0023), the method relies on selecting the top acti- vating samples from dataset and highlighting tokens according their relative activation magni- tudes. They use activvatti pairs such as [This, 0]. [feature, 0]. [act ivaaed, OJ, [the, 21, [most, 10]. where activations normalized to fixed range (for example, to 10). Prior work also suggests that using more than 10 top activating samples provides little improvement.
16

---

Published as a conference paper at ICLR 2026
tasks
its
their
the
L20
23
31
Embeddings
F7381
Circuit Tracer
all
L18 F8084
completed
Embeddings
all
L16 r4888
all
L12 F16162
all
1.2 F1183
completion
L0 F16324
complete
L2 F1847
Input:
The
team
completed
all
these
your
those
all, tout
L20 F8884
all. tout
F8684
all,
F4888
20
Mall Tha
L16 F5591
WeightLens
FB405
L12 F16162
Mail
Zul
1.4 F8718
tout
L2 F2323
LO F1735
All
Vocabulary
Vocabulary
Vocabulary:
Penh
Gaulle
Permanente
ALL
all
Figure 11: Comparison of attribution graph for features L20F8684 and L18F8684. Top row shows input-dependent graph obtained with Circuit Tracer using the prompt "The team completed all". Bottom row shows weight-based graph for the same features produced by WeightLenns.
For MaxAct*, the method caches the top 1000 activations for each feature and uniformly samples 15 inputs from this set. MaxAct* highlights activating tokens using angle brackets, with repeated brackets indicating higher activation strength (see Figure 16 for example).
E
CASE STUDIES
E. WEIGHTS CIRCUITS
A central motivation for weight- interpretability is the observation that many of the strongest weight connections between features correspond meaningful computational relationships. Some of these relationships also appear when constructing input-dependent computational graphs with the Circuit Tracer tool. To illustrate this, we examine feature 8684 layer 20 (L20F8684), which activates variants of the word "all". Using the Circuit Tracer tool (Hanna et al.) 2025) with we compare its input-based circuit to its weight-based
the prompt "The team completed all connections, shown in Figure II
The bnput-based circuit contains context-specifit edges that depend on the preceding tokens in the prompt. However, two strong upstream connections, L18FF864 and I. L.6F4888, appear both in Weightl.ens and in the input-based circuit. While we cannot guarantee that these connections will
17

---

Published as a conference paper at ICLR 2026
always appear in every context containing the word "all", their presence in both analyses provides supporting evidence for Assumption 1, which states that the strongest weight-based outlier connec- tions tend to generalize across contexts. Examining the connections of L18F8684 further supports this observation. Its link to LOF1735 is visible in the weight-hased graph and also appears in the input-based circuit.
Dunefsky et al. (2024) showed that one can use embedding projections to interpret features by iden- tifying which vocabulary tokens contribute directly a feature's ' activation. They found that this method works reasonably well for GPT-2. In contrast, our Gemma-2-2B example shows that this technique does not always provide reliable evidence. For L20F8684, the tokens with the strongest weight-based connections ("Penh", "Gaulle", "Permanente") are essentially random and do not re- flect the feature's ' behavior. These tokens also fail to activate the feature when passed through the model. The Circuit Tracer graph provides clearer explanation: the strongest contributor to 1.20P8684 is the embedding of "all" (+8.91), and L18F8664 (+8.03) is a close second. This indi- cates that embedding projections alone are insufficient for identifying tokens that activate a feature.
In contrast, the embedding projection of L18F8684 aligns well with its behavior, showing strong connections to related tokens such as "All", "all", and "ALL". In the input-based circuit the embed- ding contribution for L18F8684 is +60.67, which is far stronger than any other contributor. The next strongest contributor is L0F1735 (+8 33)), which also appears in the weight-based graph.
This example highlights two distinct mechanisms of token-based feature activation. One feature (L18F8684) activates directly when it encounters its preferred token. The other feature (L22FF8884) activates when receives strong upstream input from that first feature, for which we observe ex- tremely strong weight-based connection. WeightLens is designed to leverage exactly these two mechanisms in order to produce consistent token-level interpretations.
E.2 WEIGHTLENS: FALSE NEGATIVE VALIDATION
As shown above, many connections remain meaningless when we only consider weights. This motivates the validation step proposed in Section
Activations
WeightLens Analysis
extera
directly contributing totens: Token 28365 + receptor
MB BUG
contrbuting
3.2 In this step. we check candi- date tokens with single forward pass to confirm that they truly activate the feature outside of any context. This procedure helps, but it does not al- ways succeed.
via Tall TTA
F13/B4
+ plats, plotted
LO F2084
st
alaze
Valisiaion:
Ieseptol
plons
platted
Description: receptor
Figure 12: Activations and Weightlenns analysis result for L3F9607, with Neuronpedia description "snippets of python code using matplotlib".
Figure 12 iillusttasee this issue for fea- ture L3F9607 The token "receptor" contributes directly through the embedding matrix, and the single-token forward pass confirms that the feature activates on this input. The broader dataset supports this result, because highly activating samples frequently contain this token.
At the same time, another feature, L2F13784, contributes strongly to L3F9607. It activates on tokens such as "plots" and "plotted", which is intuitively related to our analyzed feature, because L3396607 fires on plot-related patterns in matplotlib code. However, single-token forward pass does not activate L3I9607 on these tokens. This tells us that a connection exists in the weights, but we cannot determine whether it is meaningful without input-dependent analysis.
In this case, method offers only a partial explanation. These limitations directly affect the Purity scores of WeightLens-based descriptions, as discussed in Section 4
E.3 WEIGHTLENS: ADDITIONAL EXAMPLES
In Figure 13 we illustrate how the validation step filters out unrelated weight-based connections. The analyzed feature, L21F199, primarily activates on mentions of insects that inhabit plants, in particu- lar larval forms. However, among the strongest baight-based connections we also observe features
18

---

Published as a conference paper at ICLR 2026
Activations
WeightLens Analysis
Top ditecty contributing tallens:
+ Token Da 101526, 52256, 116515
larve, larvee, larvil
Top output logits: 5gp2
ogg
Eggs
contribuiig
+ L20 F10921
femelie, mating, màle, nest, nests
+ L4 F 383
pollin, pollinatioo
Frege, Frequently,
roads
+ LG F10302
IDAE, Idbee
+ 1:77 F14883
Moth, caterpillars larvas, moth, mothss
Lovet
Ieanes
cast
200t
Lightered
Validation: laraaa.. femelle... nistu polin.., FAQ... Idae moth paterollan Descriution: larve, moth), catler llars, rest, lidae
Biper giving rise
HEALEE
Figure 13: Activations and WeightLens analysis for L21F199 with Neuronpedia description "words related to the life cycle of insects, especially those that live on plants."
Activations
WeightLens Analysis
differences
Top airectly contributing takens: None
Top output logit:: HideFags
poltcy,
Top contr contrbuuing festures:
Information
according CANDERY
+ LS F3333
nahlaaaki
Aarr recoss expcentialyy WItH
L4 F8047
adcitionall wieež
FB559
Same, CLUYE
Liza Santos, Non Sgra Elesciences
+ LB F7445
> LB F15436
Hon .y. who BESS Viedd 3n aparenent
Voliilation: wmint, nahilaiakpp, acdittonnlly, same.., CLUVE, again
53), 0550 Eiollst seather
pentligg
sult
Cleariy
Descrijaioos None
Figure 14: Activations and WeightLens analysis for L12F3146 with Neuronpedia description "words related to describing a process or methodology."
that are unrelated, such as one associated with questions and FAQ-like content. Other connected fea- contain mixture of related and unrelated tokens, which makes it difficult determine which associations truly reflect the behavior of L19F The validation step helps disentangle genuine connections from spurious ones.
Middle layers contain many features that activate only in specific contexts, which makes weight- based connections particularly noisy. Although weight-only inspection can occasionally reveal meaningful structure, as seen in Figure [4 for feature L.1F3316,, it does not provide principled way to separate signal from noise. Since these intermediate descriptions propagate into later-layer explanations, noise from unvalidated connections would compound and rapidly degrade the quality of feature interpretations.
All pre-computed analyses for every feature of the transcoders are available in our repository, in- cluding models such GPT-2, Gemma-2-2B, and Llama-3.2- allowing readers to explore and reproduce the results in detail.
E.4 CIRCUITLENS: INTERPRETING PATTERNS AND CLUSTERS
In Figure I6 we illustrate the difference between activation-basee description generation and circuit- based description generation. Activation -based methods forward full input sequences to the ex- plainer model, highlighting which tokens activated the feature and by how much, typically relying on the top activations. When a feature exhibits even mild polysemanticity with one dominant mode, these methods almost always recover only that dominant concept. Other activation patterns are fre- quently absent from the explanation because the dataset distribution over inputs sampled at high activation levels is highly uneven. As result, which concept becomes represented in the final de- scription largely depends on which activation mode happens to appear among the top examples, rather than reflecting the full behavior of the feature.
19

---

Published as a conference paper at ICLR 2026
Activations
WeightLens Analysis
Barer
abort snarg
Top ditecty contributing tallens:
> Token Da 92424, 7147, 11927..
Agres. agreed...
Bottom output loghs: agrees, Agroc, agrec...
contribuiig
+ LB F11353
disogree, rebuttal, retort
+ L4 F8548
Agreed, agroe, concur
And
with
po01 the
+ LS F3841
Kudos
+> L.0 P15728
neighaourhoos, specialsed, spacialist, speciaty
Bryoe,
Validation: apross reona (stort concul cptnion Kudos neigibouhoos, specillsed. Descriution disagres, raburtal, refort, concur
diopers
Sncresssty
Figure 15: Activations and WeightLens analysis for L7F10119 with Neuronpedia description "in- stances of agreement and polite conversation."
Input in activation basood baseline, where <> highlight activation and its strength:
[..]] as can be seen on the chest <<<and>>>>>> abdomen images [..]]
[..]] We measure the flexion in <<<ou>>>> knees and hips, and the extension in our spine [..]
sort all authors in in descending order of the <hip> <<< >>>hop genre. [...]
[...]
supervised training can be eliminated entirely by using off-c< <ff=<<<hh>>>>>>>>> <hee>>>shhff pretrained models [...]
[..]] practical approaches to measuring flexion << <<<mパ>>> the knee joint [..]
For the analysis framework,
[MG5LaMe << <<<>>>>>>>>>>>>>>>>>>>>>>>>>>>>> NLO][1] [..]]
GiaammappS denotes the half-width <<<a>>>> half half-maximum [...]
our scheme can be deployed with off-<<< off-<<<the>>>>>> the>>>>-hhh hardware [...]
[...] Tightness of <<<th>>>> chest, pain in the chest [...]
Description (MaxAct*): anatomical terms and body parts
Description (Neuronpedia): medical anatomical terms, particularly in the context of surgical procedures descriptions body
Circuit- based clusters with masked inputs and outputs, highlighting the token, feature activates
chest <<< and>>> abdomen [..]
flexion in <<< our>>> knees hips spine [..]]
human anatomy movement, involving chest, abdomen, limbs, and joints, often with terms like "flexion" and "shoulders"
off <<<<hh>>>>>>>>>>> shelf [...]
flexion <<hii>>><<<>>>>>>>>>> knee [...]
use the phrases with dlash in various contexts, like "off-the-shell"
MC <<<@@>>>>>>>>>>>>>>>>>>>>>><>>>> [...
width <<<at>>> [...]
"@" and "at" indicating specific measurements values, l.e. "width at"
off-<<<the>>> [...]
of <<< the>>> chest. pain chest [...]
CircuitLens description: and "shoulders"; hyphenated phrases like "off-the-shelf"; "@" and "at" measurements, e.g. "width at".
human anatomy and movement (chest, abdomen, limbs, joints), terms like "fiexion"
Figure 16: Feature L21F612 (Top) Subset of inputs used to generate descriptions with the MaxAct* baseline; (Bottom) the same inputs masked and clustered with CircuitLens.
To address this, we explicitly sample across the entire activation distribution and subsequently clus- ter the samples based on the underlying circuits that produced the activation.
A second limitation of activation-bart not conceptually related to the underlying phenomenon of interest. For instance, inputs reflecting high-frequency words such as "and".
baselines is that feature may fire on certain words that are
anatomical contexts in Cluster 1, the feature often activates "our", "in", and "the". This forces the explainer model to infer the relevant concept solely from surrounding context, which is unreliable. In contrast, by masking both the input responsible for
20

---

Published as a conference paper at ICLR 2026
Input in activation -basid baseline, where <> highlight activation and its strength:
[..] girl in knee-length << <<<White>>> dress, with [..]
[...] who was wearing a << <<<white>>> >> labcoat, [...] rubber gloves [..]
so I'm wrapped in <<<>h>>>>>> cotton robe and pair of black leggings [..]
[...] as she spoke, child <<<blu>>>> frock, a apron [...]
[..]] Before we left for the night, the tables bore << <<<eree>>>>>>>> and white balloons [...]
Officer Young examined the bag, large <<<black>>> plastic bag, and found large amount [...]
who stood in
picture collage Seath and cross made <<<white>>> flowers [...]
Murphy found stereo speaker and large <<<green>>>> box. [...]
[..] is the head of little <<<grey>>> mouse, with a long tail [..]]
Description (MaxAct"): description of colors associated with clothing and objects
Description (Neuronpedia): words describing colors materials
Circuit- based clusters with masked inputs and outputs, highlighting the token, feature activates on:
....] knee-length <<< white>>> dress
<<< white>>> labcoat [..]] rubber [..]]
white clothes
[..] <<< white>>> robe
leggings [..]]
[..]] blue>>> frock [..]]
color and material adjectives, often describing clothing or containers
[..]] black>>> plastic bag [...]
[..]] white>>> [..]]
green or black colors applied to different containers, bag, box
<<< green>>> [...]
[...] <<<grey>>> [...]
CircuitLens description:
Color adjectives like "white," "green," and "black" frequently describing clothing, as well as material adjectives.
Figure 17: (Top) Subset of inputs used to generate descriptions with the MaxAct baseline; (Bottom) the same inputs masked and clustered with CircuitLens. Pink corresponds to the cluster structure obtained with the clustering hyperparameter eps 0 8. Blue and violet correspond clusters obtained with eps = 0. Inputs that are not assigned to either the blue or the violet clusters at eps 0.7 the outliers under this clustering configuration.
the activation and the model-generated output, see that the feature in Cluster 1 consistently contributes to producing body-part tokens. Even when no body part has yet appeared in the context, phrases such as "flexion in" lead the model to generate "knee", indicating that we recover both what triggers the feature and what the feature tends to cause the model to output.
Finally, the cluster-specific circuits reveal that each cluster is supported by coherent of up- stream features. For example, in Cluster we find features such as I6FF348 (arms anatomy and other body parts), LI8F1184 (joints and bone pathologies), and L19F4553 (medical manipula- tions on animals involving specific body parts). Some features are more general, such as LI1F3544 (medicine-related activations), while others capture specific lexical patterns associated with Cluster 1, such as L10F16169 (activations on "our"). ) In Cluster 2, we observe features including LOFI 168 (dash). L6F6265 (tokens following dash), and LSF6988 (dash-containing expressions such "by- the-book" or "first-inffisstoour" For Cluster 3, we again find many medicine-relate features, as well as features activating on abbreviations (0.092796, I.10F2796) or tokens such as "@" or "at" (LOF2514, L5F14644), even though have not found monosemantic features corresponding di- rectly to these tokens. This suggests that the broader medical context, rather than specific lexical items, is the primary driver in this cluster.
We also observe several features that appear in the circuits for most inputs, such as L6F2267, L8F3099, and L7F3099. Their activation patterns dense and difficult to interpret. One possibil- ity is that such features compensate for sparsity or reconstruction-eree tradeoffs in the transcoder, although further analysis is required to understand their role.
21

---

Published as a conference paper at ICLR 2026
E.5 CIRCUITLENS: CLUSTERING HYPERPARAMETERS
In our work, we apply HDBSCAN with eps = 0.7 and min.sammllessss 3. This configuration frequently yields several narrow clusters, Figure 16 while still providing enough inputs within each cluster to support meaningful interpretation. However, such fine grained clustering is not al- ways required. We observe that this parameter setting often subdivides monosemantic feature into more specific sub-concepts, effectively identifying sub-circuits within the main circuit responsible for the feature's 's activation. This behavior illustrated Figure 17
The feature in Figure 17 generally activates on colors and occasionally on materials, most promi- nently when they describe clothing, though exclusively. more detailed clustering separates inputs into distinct groups, such white clothing and green black boxes or bags. Other instances such as white flowers, a grey mouse, or blue frock are treated as outliers. In this setting, using a single cluster for all imputs provides clearer interpretation. Qualitatively, all inputs share circuits containing features related to materials (L2F2256) and to colors (L9F9503, L6F4302, L11F1553,, L2F1708, L2F11366
L8F2876, L0F8416, L10F10449, LIF12806). Notably, features L6F2267, L8F3099, and L7F3099 which we previously observed as activating across all inputs also appear consistently in this cireuit.
Overall, these observations indicate that circuit-based clustering can reveal both coarse and fine structure in feature behavior, and further work may explore evaluation-guided refinement of cluster- ing parameters and results.
F DESCRIPTIONS GENERATION
F.1 POSTPROCESSING WEIGHT-
TEIGHT-BASED BASED DESCRIPTIONS
LLM-based postprocessing of weight-based analysis enables generating smoother, more coherent feature descriptions by consolidating input- and output-centric information, specifically, the tokens that activate a feature and those that it promotes suppresses. This approach is particularly effective at filtering out noise, as promoted or suppressed tokens often include unrelated or random terms that do not reflect the feature's true function.
we're re studying neurons in a neural network. each nauron has certain inputs that activate it and outputs it leada to. will receive three pieces information about
1. The top important takens. top takena it promates
output.
3. The tokena it auppresses in the output.
These will separated Suppressed], All three are a combination of tokens. You can inter the most likely output tokens.
three sections: (Important Tokens],
Promoted), [Text
tckeas, especislly [Text Promoted] and (Text suppressed), may include moise, such as unrelated terns, symbols, or programri ing Jargon. theae are coherent, you may ignore them. Tf tha (Tmportant Tokena] do net form common theme, may simply combine the words form single concapt.
function
Pocus on identifying cchesive theme or concept shared by the most relevant tokens.
Your respenae should be concise (1-2 sentence) explonation of the nouron, describing what origgera it (input) and what does once triggered (outpat) If the input and output are zelaced, may meation this; otherwise, state them sepazately.
[ConcepL: cYour Interpretation
neuron, based
provideds]
Example
Input:
[Impoztant Tokens]: ['on', "pada"]
[Tokens Fromoted] ['behair']
[Ickens Suppressed] ["on', "in']
Output:
[Concept:: The token "on* in the context of *on 1f ot"]
22

---

Published as a conference paper at ICLR 2026
F.2 GENERATING CIRCUIT- BACCUIT-BASED DESCRIPTIONS
At the first step, we treat each cluster of a feature separately. We pass the obtained patterns (input- centric or full, i.e. with patterns detected the model's output) in there order generate description.
are explainable researcher analyzing feature activations language models. You will receive short patterns: fragments of text where takens activated a foature. ONLY the an anippe shown are the evidence, not assume any extra surrounding exntext.
Pattern formatting:
1-3 sxipped
inportant
.I...] is 4 or more skipped not relevant takens <ethighlighted>s
snippet is usually activating taken), it can be part of word.
important signal
Analysie procedure:
start by interpreting semantics. treat data raw strings.
2. Count and note repeated literei elements (words, single letters, panctustion, suffixes/ prefixes, Latex takens, different symbols, brackets, prrows, paxenthases).
3. Pay spectal attention to: exact repeated Lokens, repeated pinctuation formasting (commas, superscripts, backslashes, braces), poaitional patteras,
capitalization patterns single variable tokenz,
functionsl words, like articles, pronouns, modal verbs, that create consistent pattern. concapt (if appropriate).
Only after the sttaral/atrucurral check, generaliza
Decision rules:
If a single literal Loken or atructural pattern dominates, output that token structural label exactly.
If it is aome grammatical patsern, output exactly that.
speculat vs sementic label a ess "iteral patterna support them.
Output rules:
Octput exactly ONP concise sentence 1420 wordsl describing the shared concept or strvcture. single token/pattern deminates, output exastly.
You may includa up to one short example group in parentheses to clarify.
NOT include extra labels or the word "Description:"
or structure is found, output exactly: No concept found. Avoid vague phrases like "in various contexts" or 'a variety of words'. output your internal reascning, only the final single sentance.
clear recurring
Example
Input:
important
helps to
parmits importance to
possible [...] tc able
allows us to
purpose is
Output: Preposition "Lo" in phzsses that express purpose, intention, or enable an action.
At the next iteration, combine the obtained cluster descriptions into single one.
You are explatnable AT researcher analyz'm multiple related concepta.
You will receive a 1ist of ecconcept descriptionse each representing semantic, giammatical functional element.
[ONLY Weighttenss<iiruutttee sometimes, may also receive phrase at the beginning like "Important tokens: ...", for exomple:
EXPERIMENIS:
Important tokens: amazing, largely,
Important tokens: danger, preparation, prepare, preparing. Important tokens:
Always integrate these tckens into your description, even if they do not fit naturally with concept descriptiens.
Step-by-step reasoning:
23

---

Published as a conference paper at ICLR 2026
Examine provided concepts carefully. identify recurring themes, functions, roles.
Look for commonalit les acrosa the concepta, including:
grammatical elements larvicles, parts of sentences, syatactic patterasi symbels and punctustion (commas, brackets, etc.)
semantic categories
machematical or aymbolic markers
Fay special attention specific patterns, words tarticles, modal verba, etc.).
are through
4. Mezge similaz or overlapping elemente into single, concise idea.
Think step
21 identify the core function or role each concept serves.
Group related concepts together,
el Combine them into one coterent description.
Output rules:
Output exactly sicne cencise sentence (<30 words) describing the shazed concept or several
main concepts.
Include all major elements, but marge overlapping items.
Trclude
specific patterna, they clari"y the concept. Include any *Iaporlant Lokens" explicitly in the description.
examples
aot
headings, ex:za commentary.
precise, avoid speculation, and avoid expressions like "in various contexts."
G EXTENDED RESULTS
In this section, we provide plots for distributions of metrics, presented Sections and Specif- ically, Figures 18 and 19 correspond to the results, presented in Figure 3 and Figures 20 and 21 correspond to the scores, demonstrated Figure In addition, Tables and present correspond- ing metrics in tabular form.
Presented results demonstrate that on Clarity our Weightl.ens and Circuitl Lns methods, as well as their combination, outperform crivaatin-basee baseline. WeightLens results enhance rrcuit-based information, and allow bridging this gap between smaller and larger datasets, or even outperform the activation-based baseline, thus making the automated interpretability more robust the dataset imperfections.
Metric
Method
0
4
7
10
12
15
18
23
WI.
0.67
0.67
0.65
0.42
0.47
0.67
0.80
0.71
0.61
+ut
0.63
0.57
0.58
0.36
0.38
0.57
0.72
0.63
0.60
0.54
Clarity
WL+Out+LLM
0.64
0.63
0.62
0.41
0.43
0.56
0.70
0.68
0.62
0.61
Neuronpedia
0.61
0.48
0.52
0.30
0.28
0.42
0.58
0.50
0.49
0.40
MaxAct
0.51
0.54
0.56
0.38
0.33
0.49
0.62
0.53
0.49
WL
0.72
0.70
0.72
0.59
0.76
0.85
0.85
0.86
0.86
0.80
WL+
0.73
0.73
0.75
0.61
0.78
0.86
0.86
0.81
Responsiveness
LLM
0.72
0.71
0.74
0.56
0.77
0.83
0.84
0.84
0.82
0.77
Neuronpedia
0.74
0.70
0.74
0.60
0.70
0.78
0.80
0.80
0.76
0.71
MaxAct
0.74
0.74
0.75
0.64
0.71
0.78
0.82
0.81
0.78
0.75
WL
0.50
0.47
0.44
0.29
0.55
0.70
0.63
0.65
0.62
0.51
WL+Out
0.51
0.47
0.47
0.29
0.54
0.69
0.64
0.63
0.61
0.51
Purity
WL+Out+LLM
0.49
0.53
0.49
0.33
0.59
0.70
0.66
0.54
Neuronpedia
0.60
0.60
0.65
0.54
0.67
0.75
0.68
0.73
0.66
0.61
MaxAct*
0.64
0.63
0.66
0.55
0.67
0.73
0.73
0.74
0.70
0.65
WL
0.02
0.05
0.01
0.01
0.01
0.01
0.06
0.02
0.07
0.14
WL+Out
0.02
0.05
0.01
0.00
0.01
0.01
0.06
0.03
0.07
0.15
Faithfulness
WL+Out=LLM
0.01
0.06
0.01
0.01
0.01
0.02
0.06
0.02
0.06
0.12
Neuronpedia
0.02
0.06
0.02
0.02
0.02
0.03
0.04
0.02
0.07
0.09
MaxAct"
0.02
0.06
0.02
0.02
0.01
0.02
0.05
0.02
0.06
0.11
Table 2: Weightl enssbbaed methods: Mean metric scores across layers
24

---

Published as a conference paper at ICLR 2026
Metrie
Method
0
4
7
10
12
15
18
21
23
25
CL-Input
0.59
0.51
0.37
0.30
0.19
0.56
0.41
0.41
0.41
0.27
CL-Full
0.64
0.49
0.43
0.29
0.24
0.56
0.43
0.52
0.42
0.27
WL+CL-Full
0.68
0.60
0.49
0.37
0.25
0.66
0.43
0.55
0.65
0.38
Clarity
CL-Input (top)
0.80
0.81
0.64
0.50
0.27
0.80
0.57
0.54
0.63
0.43
CL-Full (top)
0.82
0.81
0.64
0.53
0.27
0.46
Neuronpedia
0.57
0.48
0.43
0.30
0.14
0.42
0.42
0.36
0.49
0.32
MaxAct
0.49
0.55
0.48
0.38
0.17
0.49
0.44
0.39
0.54
0.36
CL-IInut
0.46
0.45
0.45
0.40
0.51
0.77
0.53
0.59
0.54
0.43
CL-Full
0.47
0.45
0.43
0.40
0.51
0.76
0.54
0.64
0.55
0.44
WL+CL-Full
0.55
0.55
0.46
0.44
0.53
0.81
0.54
0.67
0.72
0.52
Responsiveness
CL-Input (top)
0.85
0.88
0.75
0.77
0.63
CL-Full (top)
0.86
0.89
0.75
0.71
0.63
0.92
0.76
0.73
0.80
0.64
Neuronpedia
0.70
0.70
0.69
0.60
0.55
0.78
0.70
0.71
0.76
0.66
MaxAct
0.69
0.74
0.70
0.64
0.57
0.78
0.71
0.71
0.78
0.66
CL-Input
0.31
0.37
0.40
0.35
0.46
0.70
0.44
0.51
0.48
0.41
CL-Full
0.30
0.36
0.38
0.36
0.46
0.69
0.47
0.54
0.49
0.43
WL+CL-FulL
0.38
0.44
0.37
0.38
0.46
0.74
0.47
0.56
0.61
0.47
Purity
CL-Input (top)
0.73
0.71
0.64
0.54
0.57
0.50
CL-Full (top)
0.71
0.72
0.61
0.55
0.54
0.84
0.62
0.62
0.66
0.52
Neuronpedia
0.57
0.60
0.60
0.54
0.53
0.75
0.62
0.64
0.66
0.58
MaxAct
0.60
0.64
0.61
0.55
0.53
0.73
0.62
0.64
0.69
0.58
CL.-Inpu
0.03
0.04
0.02
0.02
0.02
0.02
0.05
0.03
0.04
0.03
CL-FFul
0.02
0.04
0.02
0.02
0.02
0.02
0.05
0.02
0.04
0.04
WL+CL-Full
0.02
0.05
0.02
0.02
0.06
Faithfulness
CL-Iaput (top)
0.02
0.05
0.02
0.02
0.02
0.02
0.04
0.03
0.05
0.06
CL-Full (top)
0.02
0.05
0.02
0.02
0.02
0.02
0.05
0.03
0.05
0.05
Neuronpedia
0.02
0.06
0.02
0.02
0.02
0.03
0.04
0.03
0.07
0.06
MaxAct
0.02
0.06
0.02
0.02
0.02
0.02
0.05
0.03
0.06
0.06
Table 3: Circuitl.ens ens--eseeedbe methods: Mean metric scores across layers
25

---

Published as a conference paper at ICLR 2026
Layer 0
2.5
We titenn
We -tenns-Oue+1L Recuoopeeiio Massktt
Number of sumplen
15-
0.0
18
Respordiveness
Purtty
Layer 4
Number sumples
3.5
2
10k
1.0
10
0.4
0.6
0.0
0.0
011
02
Twithi.ress
Layer 7
2.0
sumples
0.5
0.0
022
6.4
0.6
0.8
1.0
0.0
022
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
6.0
0.1
02
Clarity
Resporcivenes6
Purtyy
Falthfunees
Layer 10
Kumber sumples
1-
0.5
0-
0.0
18a
Clantty
Reeponsiivnees
Purtty
Layer 12
2.5
2.0
Numbor of samples
Ocrott
1
1
0.2
0.0
0.6
0.6
0.0
0.0
0.1
Resparsivammes
Failbfukmaa
Figure 18: Kernel density estimates illustrating evaluation results of WeightLens methods com- parison to the baselines (layers 0-12).
26

---

Published as a conference paper at ICLR 2026
Layer 15
WeightLens
Number on samples
10
0.-
0.
0.0
10
Layer
Numbur sumplis
123
0.2
0.4
0.6
0.6
0.0
0.0
0.2
[ksperraivammsa
Layer
Numoer sampics
10h
Donsity
3.5
3.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.0
1.0
0.0
61
62
0.3
0.4
05
0.
0.7
0.8
Clarity
Nespersiveness
Purty
Nathfuiress
Layer 23
2.5
Number aimples
3-
Densit
3.0
0.0
10P
0.8
Resporslveness
Purtty
Layer 25
122
Number of sumples
200
1
0-
0.0
10P
0.2
0.4
0.6
0.2
falthfuneess
Figure 19: Kernel density estimates illustrating evaluation results of WeightLens methods com- parison to the baselines (layers 15-25).
27

---

Published as a conference paper at ICLR 2026
Layer 0
Croutsers Input
5-
WL Cituttenss Full
Craatters-Ful
humber of sumroks
Croutterss Input Dop)
Crcutters Pull Bopt
Keuronppcia Maxiet
0
10
Putty
Layer 4
Number sumples
2-
0.2
0.4
0.6
0.6
0.0
0.0
0.2
[ksperraivammn
Layer 7
Numbar of sumplcc
12
Denoity
1-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.0
1.0
0.0
0.2
0.4
Clarity
Nespersivenese
Purty
falthfunees
Layer 10
3.0
Mumber of sampes
Denstty
3.0
0.5
0.0
100
0.6
0.8
0.6
Dantyy
Responsheneess
Purty
Layer 12
2.5
Number aamplaa
2.0
Densiyy
0.5
10
0.2
0.4
0.6
Folthfuneess
Figure 20: Kernel density estimates illustrating evaluation results of CircuitLens methods com- parison to the baselines (layers 0-12).
28

---

Published as a conference paper at ICLR 2026
Layer 15
Circtsers Input
WL Ciruttenss Full
Croatters-ful
Number of sumokes
Croutters Input Dop)
Crcutters Pull Bopt
Keuronpecia
30
Haxiett
0
10
Putty
Layer
Numbur sumplcs
1-
1
0.
0.4
0.6
98
0.0
0.0
0.2
(ksperravvmma
Layer
Number tamples
10a
Deroity
101
1
0.0
02
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
02
2.4
0.6
0.8
1.0
0.0
0.2
0.4
Clarity
Nesponsiveness
Pariyy
Ra Brfuunnes
Layer 23
Number of sample
1.a
Denstry
20"
1
10P
Clarity
Resporsiveness
Purtty
Layer 25
Number aampla
Density
1-
0.5
10P
0.2
0.6
folthfuneess
Figure 21: Kernel density estimates illustrating evaluation results of CircuitLens methods com- parison to the baselines (layers 15-25).
29


Reviews:
[{"review_id": "1BOo8gd3eh", "reviewer": "Reviewer_ZcNR", "summary": "This paper proposes WeightLens and CircuitLens as two complementary methods for sparse feature interpretation. WeightLens leverages the input-invariant term of the transcoder attribution to explain context-independent features, while CircuitLens use attribution scores to capture how feature activations arise from interactions between components. These methods show higher clarity and responsiveness than traditional activation-based interpretation.", "questions": "1. In generating feature description in Section 3.1, is the connection between transcoders also results in token descriptions? Why is this necessary instead of directly showing inter-feature behavior in description?\n2. Can you provide more example of the results of WeightLens & CircuitLens?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Current methods for automated interpretation of SAE features largely depend on observing the feature activation pattern, which is restricted in explaining complex features with deep connection with upstream and downstream features. This paper takes advantage of transcoders and attribution scores to extend the information for interpreting features. It follows a natural direction of the progress of SAE feature interpretation.\n2. The sampling strategy in Section 3.2 considers the complete spectrum of feature activation magnitudes, providing comprehensive explanation of the feature's behavior.\n3. The WeightLens method eliminates the need for an explainer LLM.\n4. The interpretability metrics from the FADE framework gets substantial improvement with CircuitLens.", "weaknesses": "1. The motivation of several parts of the paper remains opaque and the expression is unclear, including Assumption 2 in Section 3.1, Circuit-Based Clustering in Section 3.2. This makes the paper hard to follow.\n2. The faithfulness score seems very low using either WeightLens or CircuitLens. Whether the interpretation truly reflects the features' behavior is doubtful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7yQkWdYCBQ", "reviewer": "Reviewer_gvCa", "summary": "This paper introduces WeightLens and CircuitLens, two methods for automatically generating feature descriptions. WeightLens analyzes model weights alone to identify token-level features and their stable, input-invariant connections, while CircuitLens traces dynamic, input-dependent interactions between features, attention heads, and output tokens using Jacobian-based attributions. By clustering using Jaccard similarity and DBSCAN, the approach tries to generate multiple descriptions of a feature to incorporate its polysemantic behavior.", "questions": "1. It is mentioned that a z-score is used to identify outlier tokens. What threshold value is applied to filter these tokens?\n2. In Step 2 (Validate tokens) of Section 3.1, tokens that actually activate the feature are filtered in. What criterion defines a token as activating a feature, i.e. what activation threshold is used to make this determination?\n3. In WeightLens, it is unclear how the lemmatized tokens are transformed into feature descriptions. Is an LLM employed for this step?\n4. Can you also clarify whether output-centric analysis is used when computing the Jaccard similarity? The paragraph describing circuit-based clustering mentions collecting transcoder features and token/attention head pairs for each input but does not specify whether the feature’s impact on the final logit is considered.\n5. Lastly, do the activation-based baseline methods use the same secondary LLM to generate feature descriptions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This work addresses an important problem of generating automatic feature descriptions without relying on black-box LLMs or large-scale datasets.\n2. The proposed WeightLens technique offers a practical solution to these limitations by producing competitive, and in some cases superior, feature descriptions compared to activation-based methods, while remaining relatively simple to implement.\n3. Moreover, the integration of both input- and output-centric analyses, combined with the use of Jaccard similarity and DBSCAN for clustering input examples based on circuit components, is a particularly interesting idea.", "weaknesses": "1. The main motivation of this work is to eliminate dependence on black-box explainer LLMs and large datasets. However, the CircuitLens method still requires access to both a secondary LLM and a large dataset. Therefore, only the WeightLens method truly addresses the core problem the paper aims to solve.\n2. Figure 1 is not very informative in its current form. It could be significantly improved by expanding it into a larger figure composed of subfigures illustrating each of the three steps. For example, consider a random feature f at layer l, and visually depict all the steps required to obtain its token-level descriptions.\n3. It would also help readers if the Circuit-based analysis subsection (Section 3.2) were better contextualized in relation to the previous subsection and the overall goal of the work. You could begin the section with a brief paragraph explaining why circuit-based analysis is necessary to obtain meaningful feature descriptions.\n4. Section 3.2 is somewhat difficult to follow. Overall, the process for generating feature descriptions involves the following steps: \n   1. Sample examples with varied activation values using the proposed sampling strategy. \n   2. For each example, perform input- and (optionally) output-centric analyses.\n   3. Compute the Jaccard similarity matrix. \n   4. Apply DBSCAN to cluster the input examples.\n   5. Use an LLM to generate cluster-level and overall feature descriptions.\n\n   However, this workflow is not clearly conveyed in the current text, particularly due to the ordering of explanations. I recommend reorganizing this section to follow the logical flow outlined above.\n\n5. The evaluation results for both WeightLens and CircuitLens are presented for only four layers. Results across all (or most) layers should be included, preferably in the appendix, to enable a more comprehensive assessment of their effectiveness.\n6. (Minor point) It would be easier to compare different configurations of both WeightLens and CircuitLens if all the evaluation criteria could be aggregated into a single one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJEltplN9B", "reviewer": "Reviewer_pB71", "summary": "This paper introduces two new (families of) transcoder feature interpretation techniques. WeightLens techniques rely solely on the feature weights, and their relations to the un/embedding matrices, while CircuitLens techniques go beyond simple maximum feature activations visualizations, to find tokens that are genuinely to the feature as part of a circuit. The authors measure the quality of the interpretations produced by these methods with respect to existing interpretations / methods, and find that their methods perform better.", "questions": "- Can this method be used with SAEs (as opposed to transcoders)?\n- What are the computational requirements of each proposed method?\n- Why do you (in 3.1) also consider contributions of tokens via earlier features? This kind of seems like a manual, 1-step version of attribution (which tries to avoid the fact that you don't actually have activations for your transcoders, since you're not running them on an input). It seems more natural to me to just use direct effects, or to actually do attribution.\n- You seem to believe that sampling from various quantiles makes more sense for sparse dictionaries than just interpreting max acts (which is better for neurons). Do you find any empirical evidence for this?\n- I find 3.2 rather hard to read, as its framing is a bit strange. Why do you say that \"To account for interference between layers, Ameisen et al. (2025) propose incorporating a Jacobian term into the attribution formulation...\"? I don't think they're accounting for interference, per se; they're just computing the direct effects of one feature on another via all possible paths, which can be done using a Jacobian term. More pertinently, you haven't really introduced attribution yet, so this section comes a bit out of the blue.\n- What exactly are the inputs you use for the circuit-based methods, which require inputs with respect to which to perform attribution? The authors say \"For generating circuit-based descriptions, we use sampling, described in the Subsection 3.2\", but I think it'd be good to be clear about the whole pipeline. I think that what happens is that they sample sentences using inverse quantile frequency, then perform attribution on those samples, and then use these to generate descriptions.\n- Can you make Figure 4 easier to read?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "That feature interpretation is a time-consuming and manual process is a well-known issue, and attempts to solve it via auto-interp have fallen short in various ways. This paper proposes a method (or set thereof) that seems better than existing methods; by studying the weights, and features' roles in circuits, it manages to provide more useful information for autointerp models, and produce better interpretations. In this respect, I think the techniques are a novel combination of existing ones, and a significant contribution to the sparse feature interpretation literature, at least in the transcoder context. The authors perform reasonable measurements of interpretation quality, although measuring this is an ongoing and open problem, as I see it.", "weaknesses": "**Clarity / Organizational Issues**: It is at times hard to understand which methods are which in this paper. The authors go into great detail re: input-invariant and circuit-based analyses in 3.1 and 3.2, but it's not always obvious how these match up to the methods evaluated in Section 4. You should probably discuss / name the methods in Section 4 in the preceding sections.\n\nRelatedly, it's not always clear which techniques are new to this paper, and which are not. It's clear that generating feature descriptions based on max activations is not new. I also think that \"3. Analyze output effects\" is a fairly standard application of the logit lens to feature output weights, and also not new. On the other hand, step 1 of the same procedure seems new, as does most of 3.2.\n\nMore qualitative examples could help make this paper's methods clearer. I'd love to see examples of the different inputs to the annotator LLM, as well as the resulting labels produced by each method for particular examples. This is important, because it's hard to understand why / how your methods are better without knowing what each method takes in and returns.\n\nIn general, more editing of this paper would help. Sentences are often long (233-236), which makes reading hard. I don't think the the bolded-\\paragraph-takeaways format works very well for this paper. This is in part because the paper doesn't really telegraph well what claims it's going to make / study. I went into each section not knowing what the authors intended to prove / show, and encountered a collection of findings without a clear narrative to guide them. Similarly, in the methods sections, methods are introduced without it being clear what they would later be used for. This made for a rough reading experience.\n\n**Metrics and Comparisons**: This paper describes the metrics introduced by Puri et al only briefly, while a longer description (with more details about how they are concretely measured) would be appropriate. These metrics are, by my understanding, all powered by LLM-as-a-judge. This is not unreasonable: it's not as though humans can realistically annotate all of these feature descriptions without great effort. However, I think that some sort of control in order to ensure the validity of the LLM judgments (say, a comparison to human annotations) is warranted. More discussion is warranted in general on this front: the challenge of producing better explanations is not just producing good explanations, but also measuring how good they are, which is quite challenging.\n\nRe: comparisons, the authors often compare to Neuronpedia and MaxAct*, but really need to include more details on what these are. How are Neuronpedia's features generated, exactly?\n\n**Usefulness / Scalability**: The authors propose methods for interpreting transcoder features, but right now, there are few transcoders out there. It would be helpful to know if this technique works on other sparse dictionaries, like SAEs. I would also love details on the cost of these techniques (specifically the circuit-based analyses), and if there is an implementation available (as they seem pretty non-trivial to implement).\n\nDespite these weaknesses, I think this paper is near-worth-accepting, and could be convinced to raise my score. I'd love to see the authors improve the clarity of this paper, which I believe will hinder its usefulness for people who are not core mech interp researchers. I also worry that techniques proposed here are a bit too niche and expensive to be practically useful / interesting for a broader audience, and have asked questions about this below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)