Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework
Authors: Laura Kopf
Nils Feldhus
Kirill Bykov
Philine Lou Bommer
Anna Hedström
Marina MC Höhne
Oliver Eberle
Venue: NeurIPS 2025 poster ()
Keywords: Interpretability
Explainable AI
Automated Interpretability
Feature Description
Mechanistic Interpretability
Concept-based explanations
Clustering
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We introduce a multi-concept feature description framework which can identify and score polysemantic features.

Abstract: Automated interpretability research aims to identify concepts encoded in neural network features to enhance human understanding of model behavior. Within the context of large language models (LLMs) for natural language processing (NLP), current automated neuron-level feature description methods face two key challenges: limited robustness and the assumption that each neuron encodes a single concept (monosemanticity), despite increasing evidence of polysemanticity. This assumption restricts the expressiveness of feature descriptions and limits their ability to capture the full range of behaviors encoded in model internals. To address this, we introduce Polysemantic FeatuRe Identification and Scoring Method (PRISM), a novel framework specifically designed to capture the complexity of features in LLMs. Unlike approaches that assign a single description per neuron, common in many automated interpretability methods in NLP, PRISM produces more nuanced descriptions that account for both monosemantic and polysemantic behavior. We apply PRISM to LLMs and, through extensive benchmarking against existing methods, demonstrate that our approach produces more accurate and faithful feature descriptions, improving both overall description quality (via a description score) and the ability to capture distinct concepts when polysemanticity is present (via a polysemanticity score).

Capturing Polysemanticity with PRISM: A
Multi-Concept Feature Description Framework
Laura Kopfl.2
Nils Feldhus!22 1.2 Kirill B 1,2,3,7,8
Philine Lou Bommer
Anna Hedström 1,4,4,5 Marina M.-C. Höhne Oliver Eberle
Technische Universität Berlin, Germany 2IIOLLD, Germany
UMI Lab, ATB Potsdam, Germany "Fraunhofer Meeinrich-Hertz-Instiiii Germany
'ETH AI Center, Switzerland "Universität Potsdam, Germany
Munich Center for Machine learning (MCML.) *Technische Universität München
[kopf feldhhu oliver. ebuuree i.
[kbykov, pbommer, .bommer, ahedstroem, mwwemneh patt-ppddaa de
Abstract
Automated interpretability research aims to identify concepts encoded in neural network features to enhance human understanding of model behavior. Within the context of large language models (LLMs) for natural language processing (NLP), current automated neuron-level feature description methods face two key challenges: limited robustness and the assumption that each neuron encodes a single concept (monosemanticity), despite increasing evidence of polysemanticity. This assumption restricts the expressiveness of feature descriptions and limits their ability to capture the full range of behaviors encoded in model internals. To address this, we introduce Polysemantic FeatuRe Identification and Scoring Method (PRISM), novel framework specifically designed to capture the complexity of features in LLMs. Unlike approaches that assign a single description per neuron, common in many automated interpretability methods in NLP, PRISM produces more nuanced descriptions that account for both monosemantic and polysemantic behavior. We apply PRISM LLMs and, through extensive benchmarking against existing methods, demonstrate that our approach produces more accurate and faithful feature descriptions, improving both overall description quality (via description score) and the ability to capture distinct concepts when polysemanticity is present (via polysemanticity score).
Introduction
Large Language Models (LLMs) have rapidly become integral to range of real-world applications, from software development [ to medical diagnostics [2]. Despite their growing influence, the internal decision- processes of these models remain largely opaque. A growing number of approaches aim to understand these black-box systems by analyzing their internal structure in human-ine -ittrnnitterpretable ways, such mechanistic interpretability 4451 5), structured explanations (6] 7).
advanced feature attributions
S and free-text explanations 10
A central goal of this research is to assign interpretable, functional roles to individual components The presence polysemanticiry. the tendency of individual features to encode multiple, semantically distinct concepts or patterns [16 complicates the process of explaining model components, as defies the common assumption that a single neuron is associated with single function or pattern. From coding-theoretic theoretic perspective, this reflects how neural capacity is distributed across multiple tasks. While several concept extraction techniques like sparse autoencoders (SAEs) [18] 19 aim to disentangle polysemantic features, many learned features still encode multiple concepts 20 and therefore cannot be regarded as truly monosemantic.
such as neurons or attention heads [2 L3 14 15).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Extracting Feature Descriptions
Evaluation
"Quantities, specifically nunbers, and time periods
Polysementicity Scoring
Description Scoring
"Personal experiences opinions"
PRISM
"Indafinsta articles before contextuel nouns"
Layer 47, Feature 3815
Lower Cosine Similarity
Highar activation
(GPT-2
Multi-concept feature descriptions
+ high polysemanticity
more accurate description
Figure Overview of the PRISM framework. PRISM captures multiple concepts per feature, enabling the detection of both polysemantic and monosemantic features, unlike prior approaches that constrain each feature to a single description. For example, feature 3815 in layer 47 was previously labeled monosemantic (20), whereas PRISM reveals that responds multiple distinct concepts. Polysemanticity scoring summarizes how diverse the concepts associated with a feature are, while description scoring assesses how well each concept aligns with the feature's activation distribution.
While the problem of polysemanticity is generally addressed through the extraction of sparse features, such as via sparse autoencoders, feature description methods, which aim to explain the functional purpose of individual features, typically provide single explanation per feature [2 22 23
24 25
This can limit the ability to capture the full range of patterns feature may represent. To offer more comprehensive and nuanced feature descriptions, we introduce PRISM, framework for generating and evaluating multi-concept feature descriptions that considers multiple patterns per feature. In Figure D present neuron previously labeled as "possibly monosemantic" [20] that PRISM reveals to activate for a highly diverse and heterogeneous set concepts characterized by textual descriptions. Throughout this work, concept refers to cluster of token-level inputs that elicit similar contextualized feature activations, allowing each feature to be associated with multiple concepts that reflect the diversity of inputs it responds Our contributions include:
(1) A framework, PRISM, that generates multi-concept descriptions of features (Section 3) provid- ing greater precision and granularity compared to existing methods.
(2) A set of quantitative evaluations (Section 3.2) for comparing multi-concept textual descriptions across different feature description methods.
(3) A first multi-concept feature deseription analysis of language model features, revealing that individual features encode a highly diverse and heterogeneous set of concepts (Section 5
By addressing the limitations of single-concept feature descriptions, PRISM enables systematic and nuanced understanding of internal representations, crucial to advance the interpretability of language models. Our code is made publicly available to the community
2 Related Work
Automated Interpretability Recent generative AI systems have grown increasingly complex, motivating efforts to scale interpretability. In addition to methods that provide local explanations for individual predictions, complementary line of research focuses fully automated descriptions of model components like neurons and their associated functions. In this context, an automated interpretability approach introduced to generate textual description for each neuron in GPT-2
XL (1 This method has since become a foundation for subsequent work on feature descriptions [26
19
23
22
27
25
Feature Deseription Evaluation
Automated feature description methods often rely on simulation-
(21) This approach, widely adopted in recent work [26 19 23 evaluates descriptions based on the correlation between
based approaches, where model predicts neuron activations for given inputs
simulated and actual activations. Other evaluations include binary classification of activating and
non-activating contexts (28), contrastive methods using distractor samples (2 30 and combined
approaches that assess input capture accuracy and feature characterization precision [25] B1 32).
Simulation- based approaches herein rely on an external model's s ability to accurately predict neuron
httpp:://gttuub.cc//kko////

---

1. Percentile Sampling
2. Concept Clustering
Cluster Labeling
"Quantities, specifically nunbers, time peraods'
*Personal experiences opinions"
LLM
far,3015
Sentence
"Indefinite articles before contextual nouns"
(6PT-2 PUP
percentie octivation distribution
Model Teature
Sentence clusters
Concept cluster labels
Figure 2: Steps for extracting feature descriptions with PRISM. In Step 1, PRISM processes a text dataset through the model and selects sentences from the top percentile of the activation distribution for a given feature. In Step 2, these high-activation sentences are embedded using a sentence encoder and clustered identify recurring patterns. In Step 3, the top activating examples from each cluster are used to prompt an L.M,, which generates descriptive labels for each cluster.
activations given feature extracted by an additional explainer model. Currently, it remains unclear how well LLMs reliably predict activations of other models, adding uncertainty and complexity to the evaluation of feature descriptions. Our approach instead directly compares feature activation distributions to controls, using both parametric and non-parametric evaluation measures.
Clustering and Topic Analysis in Text Recent work in topic modeling has relied on unsupervised clustering of language model embeddings to discover latent topics without relying directly on keywords 33 34]] While methods like LLooM [35] and goal-driven explainable clustering [36]
LLMs to extract assign high-level textual summaries, work focuses fine-grained interpretability of internal model features.
For an extended discussion of related work, see Appendix A.1 additional details on feature description methods are provided in Table 2 in Appendix A.2
3 PRISM: A Framework for Multi Colti-Concept Feature Descriptions
We introduce PRISM, a framework for generating multi-concept descriptions of model features (Section 3.1 and evaluating their quality through polysemanticity and description scoring metrics
(Section 3.2
Preliminaries Let a decoder-only LLM be defined as function f ; X' x.... Zr, where X is the input space (i.e., token sequences), L is the total number of blocks, and Z, C R denotes the hidden representation at layer {l,.... L}. We denote the layer-specific subfunction as fe X + Ze, such that the output of layer E is 22 f((x). We define feature the activation of a single neuron E (I..... d corresponding to the scalar function fe, X - R. defined as (x) = 24,5 which denotes the activation of the i-th neuron for the L-th layer 7 A feature description method aims associate each feature with single multiple human-interpa -nteppeetabre textual deseriptions. Mathematically, each feature 24,; is assigned a subset 1, 1,.... N possible descriptions by set-valued description function Φ F P(S), such that 8;, ((f,,; ; A), where 6 takes as input the feature function fe,; E F, À represent method-specific hyperparameter and P is the power set of all valid description subsets S, including empty and all. This formulation accommodates both single- and multi-concept descriptions, depending on the implementation of the description function é.
3.1 Extracting Feature Descriptions
To address feature polysemanticity, we propose method capable of capturing multiple concepts per feature. Let fe,: + be the fixed feature under consideration, where E (E,1,.... f1,..., L.....,,) L}} and iE (1, .l. dz). Given corpus D C X. denote the multiset of its activations by
Ar,; f (x,
E D).
PI the case of sparse autoencoder (SAE) feature, let L: Ze a² be learned sparse encoder that maps the 9-th layer's activations a K -imeensiion sparse encoding, where the j-th SAE feature is defined as (x) = (x(())) where (-) denotes the j-th SAE neuron.
We avoid defining this further to avoid notational clutter.

---

Our multi-concept framework consists of the following steps in reference to Figure
1. Percentile Sampling. Let perc, (Ar.i) (Ar,) be the operator, returning the q-th percentile of the empirical distribution of Ae,s Fix grid of high-pereentile levels
{q....., gm C [0, 1],
91
<
4m
The high-activation sample set for the neuron
Tea
(Arre
QE
2. Concept Clustering. Let e: X' + Ed be sentence- sentence-embeuddin function. For pre-specified k E IN we partition the embedded set {e(x) x E Te.s into clusters Ct,; = [CI..... C& by employing K-Means method [38.
3. Cluster Labeling, Let & denote large language model acting on sets of sentences. For each cluster C₁ we select the N, sentences C. Cr with the highest activations and query
8j (iopx((C))
where 8j E S is concise natural language summary of the common theme in Cj.
3.2 Evaluating Multi-Concept Feature Descriptions
Polysemanticity Scoring To quantify the degree of polysemanticity, we measure the similarity among the generated descriptions per feature. Descriptions encoded using a sentence embedding model, and their pairwise cosine similarities Tj = e(s;), Ty E computed:
RR" with i * j,ú,j E fl..... k) are
Σ
TreTiha
COs(
(5)
E R2 Σ +3,
where T' N is the dimension of the sentence embedding. Features with lower average similarity scores are considered more polysemantic, while high similarity indicates monosemanticity.
Description Scoring Following previous work, which uses contrastive methods differentiate
between activating samples and control samples [29 30 we adapt the CoSY evaluation method [39]
language models (see Figure in Appendix A.3). CoSy evaluates each candidate feature using
two complementary metrics. The Area Under the Receiver Operating Characteristic (AUROC)
measures how well the feature distinguishes between control data points Xg. consisting of 1,000 randomly sampled entries from Cosmopedia [4), and target concept data points X,, consisting of 10 concept-specifie text samples generated by an LLM for the feature description. Given the
corresponding activations An E R for Xo and A. E R. for X1, the AUROC is computed as
V (AUOC((Ao... A1) Encro ErEA, 1(a <b)
(6)
Aool -AA1
The Mean Activation Difference (MAD) quantifies the normalized difference between the mean activation the target and control datasets:
MAD(Aso, A1) EBEA, b- 1 1--2 EaEno
Eacho a)2
with mean control activation = - - In our evaluation, we report the percentage of features
with positive MAD scores, i.c., the fraction of features satisfying A > Ag. See Appendix A.3 for dataset, model and activation details.
Quantitative Evaluation
In the following, we quantitatively evaluate our proposed feature description method, PRISM, against existing approaches for neuron and SAE feature interpretation.
In practice estimate perc, online via the p2 algorithm 37

---

4.1 Experimental Setup
In our experiments, we evaluate PRISM against competitive feature description methods, Max GPT-Explain 22I, Transluce-Expait 23), Neuronpedia [42, and Output-Centric [25 Unless otherwise specified, all experiments are conducted on the English training subset of the C4 COR- PUS [41], a large, cleaned version of Common Crawl's web crawl corpus. We process text excerpts to a uniform length of 512 tokens, either by truncation or padding, ensuring consistency across model inputs. Feature activations are extracted from four pre-trained L.LMs, from three layers per model (carly, middle, late 7 For GPT-2 XL [43] and Llama 3.1 8B Instruct [44], we analyze MLP neurons. For GPT-2 Small [22] and Gemma Scope [45). we focus on residual stream SAE features on account of the publicly available implementations. More details on the evaluation procedure and experimental
setup are provided in Appendix A.3 and Appendix A.4
For the practical implementation of each step in the PRISM framework we choose the following settings:
For (1) Percentile Sampling, identify all text excerpts whose mean activation values fall within the 99th-100th percentile, sampling one excerpt per percentile bin with a step size of le-05, resulting in 1000 high-activation excerpts per feature. Compared top-k sampling, this approach captures a broader spectrum of strong activations, allowing us to collect meaningful text samples from wide range of conceptual patterns.
For (2) Concept Clustering, the resulting text set is embedded using the gte-Qwen2-1.5B- instruct sentence transformer [46], and then k-means clustering is applied with k = 5 to uncover recurring conceptual patterns. Using fixed number of clusters strikes balance between granularity and human interpretability, enabling multiple semantic patterns to emerge while reducing redundancy or incoherence. When clusters are highly similar, the resulting low polysemanticity score reflects monosemanticity.
For (3) Cluster Labeling, generate human-interpae inteannnttrpretable labels, we prompt a large language model (Gemini 1.5 Pro 47 using the N. = 20 text excerpts with the highest mean activations for each cluster. Only positive activations are considered, and token is highlighted if its activation exceeds sample-specios
threshold, defined as the 90th percentile of the sample's s activation distribution. Token spans corresponding to peak activations are highlighted with square brackets "[...]" 49 22 [23 in the prompt to guide the model. The LLM is instructed to output a concise summary of the shared concept in each cluster. Additional prompt details are provided in Appendix A.4
4.2 Ablation Studies
To assess the robustness of our framework, we conduct ablation studies along two dimensions: (i) varying the number of clusters used for description generation, and (ii) replacing the language models used in both description generation and evaluation. Full experimental details and results are provided in Appendix A.5
We vary the number of clusters k used in concept clustering to analyze its impact on interpretability (Table 3). Larger k improves best-case description quality by isolating more fine-grained activation patterns, but reduces average interpretability as coherent patterns are split across clusters, revealing a tradeoff between precision and coverage.
Cluster Size
Text Generators To evaluate the impact of language model choice on our framework, we ablate the text generators used in both description generation and evaluation. For description generation (Table Qwen3 32B (50) achieves performance comparable to Gemini 1.5 Pro (the default in our original implementation). while Phi-4 [51] and DeepSeek R1 [52) follow similar qualitative trends,
'For obtaining feature descriptions with the MaxAct method, we use subset 10,000 samples from the English training split of the C4 CORPUS [4T) collect the top five activating samples per feature, and generate feature descriptions using the same prompt as in PRISM(see Appendix A.4).
'We use descriptions generated by their Ensemble Raw (All) method, which best performing on their basut-based evaluation.
"More details can be found in GitHub Repository https: :tttt::://ithhtee cow wwwpe////////e/e/eeeee/eeeeeee/e /wkoo/prisssii/ii
'The model originally employed in this study (Gemini 1.5 Pro) was deprecated upon completion of the project.
Consequently, subsequent experiments, including those pertaining to MaxAct and output centric evaluation, were conducted using the available and comparable model, Gemini 2.0 Flash Lite (48).
5

---

Table 1: Benchmarking feature description methods. AUROC reflects classification performance. while MAD measures activation differences between target and control descriptions. PRISM (max) reports the best-matching score per feature; (mean) averages over all descriptions. Values are means across selected features from early, middle, and late layers. AUROC includes 95% confidence intervals; MAD represents the percentage of positive MAD scores. Bold indicates best performance; dashes denote unavailable descriptions for certain models. See Appendix A.4 for details.
GPT-2 XL
Llama 3.1 8B Instract
GPT-2 Small
Gemma Scope (resid. SAE feanune)
Method
(MLP neuron)
(MLP neuren)
(resid. SAE feature)
AUROC
MAD
AUROC (1)
MAD
(UROC(()
MAD
AUROC
MaxAct
0.53 (0.49-0.58)
11.86%
0.54 (0.46-0.63
50.00%
0.53 (0.49-0.88)
11.86%
0.60 (0.50.0.9)))
50,00%
GPT-Explain
21
0.64 (0.56-0.73)
65.00%
Transluce Explain
2M
0.59 (0.51-0.67))
63.33%
Neuronpedia
(0.50-.00))
18.97%
0.62 (0.53-0.72)
63.33%
Ourput- Centric
0.55 (0.46-0.64)
58.33%
0.57 (0.53-0622)
22.03%
0.58 (0.49.0.67)
46.67%
PRISM (mean)
0.65 (0.61-0.69) 69)
66.33%
0.52 (0.48-0.55)
51.33%
0.51 (0.50.0.53)
13.22%
0.43 (0.39.0.46)
24.67%
PRISM (max) 0.85 (0.78-0.91)
91.67%
0.71 (0.63-0.78)
81.67%
(0.53-0.61)
(0.45-0222))
38.335
showing that the approach does not depend on a single model. For evaluation (Table alternative LLMs yield slightly lower absolute scores but preserve the relative ranking of methods. These results confirm the robustness of our evaluation setup and the generalizability of the framework across language models.
4.3 Sanity Checks
Prior work has shown that LLMs can produce plausible yet unfaithful explanations unrelated to the prompt 53
56]_ Like most feature description methods, our approach relies on I.M.-generate feature descriptions. To address potential issues of faithfulness, we conduct two sanity checks that assess description reliability under controlled settings: (1) randomizing sentences within clusters and (2) randomizing cluster descriptions. We further evaluate the polysemanticity scoring, examine feature description quality across percentile activation intervals, and analyze relative activations. Details on the experimental setup, results, and analysis are provided Appendix [A.6]
54 55 11
Randomizing sentences descriptions yields AUROC values near random relative to the baseline (Table 6). While MAD scores are often not statistically significant due to large standard deviations, they consistently decrease, aligning with our expectations. For poly semanticity, randomly assigned descriptions yield notably lower similarity values than true scores, reflecting reduced semantic co- herence (Figure 13 Examining percentile intervals, the top 25% (0.75-1.0) mathh baseline performance, whereas lower quartiles show reduced scores, demonstrating a positive correlation between activation strength and description quality (Table 7 The analysis of relative activations between the 99th and 100th percentiles shows that extremely small ratios are rare, espe- cially early layers, indicating that percentile- based sampling reliably captures diverse, meaningful patterns (Figure 14).
1--0 of activations closely
4.4 Benchmarking Experiment Results
Table I compares the performance of PRISM against GPT-Explain and Output-Centric. We report two variants: PRISM (max), which reflects the score of the best-matching description, and PRISM (mean), averaging scores across & = 5 descriptions used throughout the paper. All reported values are means over selected subset of features from three model layers, i.e. early. middle, and late. Dashes (-) indicate that the respective method does not provide feature descriptions for the corresponding model, preventing evaluation.
Empirical Advantage of PRISM A key observation is that PRISM (max) achieves the highest AUROC and MAD scores, across all model and feature-type combinations. For example, on GPT-2 XL (neuron features), PRISM (max) reaches an AUROC of 0.85 and a MAD of 91.6%% indicating that its descriptions are more accurate and outperform the competitive approach of GPT-Explain. One exception is observed with Gemma Scope (SAE features), where the Neuronpedia method slightly outperforms PRISM (max), achieving the highest MAD (63.33%) and AUROC (0.62). This suggests that, for certain architectures or feature types, alignment in the output space may offer complementary advantages. Additional analysis on the distribution and variability of AUROC and MAD scores is provided Appendix A.4 Overall, we find that neuron-based features yield more reliable interpretations than SAE-base features, providing further evidence for recent methodological

---

1.0
0.65
GPF2 XL
Lama 3:1-88 Indr
Cosine Similarity
0.60
Gomma Scope
Smal SAE
0.55
AUROC
0.6
0.50
0.45
0.4
0.40
GP12 XI
Lieme 3:1-01 inttuct
GPT2 Small SAE
Gamma Scops
0.0 early
middle Layer
early
middle
Layer
(a) Evaluation scores.
(b) Polysemanticity scores.
Figure 3: Comparison of PRISM (max) AUROC evaluation scores and PRISM polysemanticity
scores across different models and layers.
Motalabal
Feature Descriptions
Teit nology and Speciications
Struclured
quanilativv including years, measurements, courss, excerpts
31 Products
Information
locations, datos biblographic
Academic degrees, prolessional roles, localion, and years related education or employ-
Lig call tools and
ment history
45 Legal and Adminiatra a-
Division of eageta ly. mecical procedures/redte redmmmeettss legel seeleccr proceed ngs, and organizationa adivtiies/eeents
Guundified Descriptiars
Eve Alfairs s
-wnerrhipp property membership status
Improvement
and
Co momeriaall trasactions, lopal proceadings, and "nnncill ab ingiiacn
Events and Activties
including spacitic group Food. beations, adtivlisshouuinn ottan invelvngg change diaation data,
Acheevements,
special
Persunel Devzlommnt agn Hanbgemen'
shows, people
Hicellaneous
Positive Ex Exgressions of encilement, sharing, or posiive feeoback
pe fereees
Expressions of grasitude, current time references, or pos descr ipionstion
Experiences relatad to trael, lesture activiies, meals, and eventss, particuiarly these with tomporai alemant ttime, dates, or duration)
40 Date Selated Intormation
Figure 4: Clustering of identified PRISM feature descriptions in GPT-2 XL. The ki = 50 meta- clusters visualized using UMAP, with metalabels generated by Gemini 1.5 Pro and three randomly selected sample descriptions shown per cluster.
concerns about achieving improved interpretability through SAEs 57 Extended results on the output-centric evaluation provided in Appendix A.4
Varying Quality and Polysemanticity across Models In Figure 3(a) and Figure 3(b we analyze
the feature descriptions' quality and polysemanticity across layers. Judging the evaluation scores, middle layers generally appear to be easier to interpret (Figure (3a) In most models, the AUROC scores peak in the middle layer. exception is observed in Llama 3.1 8B Instruct, where evaluation scores (measured in AUROC, see 3.2) increase in the later layer.
To analyze the degree of polysemanticity, former [46] to embed the five natural language descriptions generated per feature. We then compute pairwise cosine similarity within each set of five descriptions estimate semantic consistency. In Figure 33b) polysemanticity scores show no consistent trend across layers and vary across models and feature types. Interestingly, we can also note that Gemma Scope SAE feature descriptions show the highest monosemanticity across all layers, as shown in Figure 3(b) Despite high variability, our findings suggest that polysemanticity does not consistently increase or decrease with layer depth, and interpretability varies significantly across architectures.
the gte- Sice-Qeen2-1.555 1.5en2-1.5BBo. in-iintrutt sentence trans-
5 Investigating Multi-Concepp Features
After validating our approach, we can leverage PRISM's novel polysemantic analysis to investigate the diversity of feature descriptions. We also take an initial step toward human evaluation of multi- concept descriptions via polysemanticity judgments.
5.1
Exploring Concept Spaces
The distinction between syntax and semanties has deep roots in logic 58 59 and the study of language [60] 61], where representations understood involve both structural and meaning-
7

---

bearing properties, Additionally, pragmatics [62] has emerged as another level of analysis. focusing on how context, intention, or social norms shape the interpretation of language. In the context of LLMs, representations have most commonly been categorized as syntactic, including dependency
structures and part-of-speech tags [x]] or semantic, involving token associations [64], task- specifcc
information [65]. or abstract encodings such as goals [66], with recent work also exploring pragmatic, context-dependent interpretations [67].
To gain a better overview of the representational diversity and variety of identified concepts, we now explore all extracted feature descriptions with PRISM. We embed them using GPT-2 XL, extract the final token as a feature description representation, and apply k-means clustering with km = 50 to identify and visualize meta-clusters using UMAP [6], illustrating the space of learned concepts within the model [0). In F we present a subset of the resulting clusters and their generated metalabels for GPT-2 XL. Further examples can be found in Appendix [A.7| including results for GPT-2 Small SAE in Figure [6 Metalabels summarizing shared themes among grouped feature descriptions are generated using Gemini 1.5 Pro, with clusters outlined based on Gaussian kernel density estimation (bandwidth h = 0.3) with point weights that decay exponentially with distance from the cluster centroid.
First of all, we find that automatically clustering and summarizing feature descriptions is effective in producing iumaaninterppetabble abstractions and associated metalabels, enabling users to reduce the potentially large number of descriptions they need to inspect. When inspecting the resulting clusters, we observe a range of distinct categories. For GPT-2 XI., we observe diverse semantic categories, e.g., referring to "Positive Experiences" (id 6) or actions in "Events and Activities" (id 29). Furthermore, we identify Nomain-sseciife clusters of feature descriptions, with associated neurons responding to categories such as "Digital Tools and Resources" " (id 12) and "Legal and Administrative Affairs" (id 45). Examples of syntactic diversity include "Time/Date Related Information" (id 40) and "Structured Data" (id 18), focusing on part-of-speech and specific sentence structure. We further identified combined representations, e.g.. syntactic concepts of quantity in the context of specific domains such as food or recipes ("Product Information", id 8). Similar patterns are observed for GPT-2 Small SAE (see Figure 16 Appendix A.7
Overall, clustering feature descriptions uncovers multiple levels of analysis, offering valuable approach for categorizing feature descriptions. Building towards a shared vocabulary to characterize model representations could further reveal universal concepts across models, aligning with recent discussions and evidence on universal representations [2]]
5.2 Polysemanticity Analysis and Human Interpretation
Following prior work on human annotation of feature descriptions representative samples for obtaining feature label descriptions. Fach instance included text clusters generated using Steps 1-3 of the PRISM framework (Figure 2 in the same format the LLM receives during cluster labeling. Seven participants annotated clusters of highly activating sentences and rated the polysemanticity of the resulting labels on an 1 11ppoint scale (0.0-1.0), enabling comparison with the PRISM polysemanticity score (Section 3.2 Full study details and results are in Appendix A.
32
23
we selected 8 instances as
Figure 5 compares human-annotated labels with descriptions generated by PRISM. The first example (top row), a polysemantic neuron in GPT-2 XL, receives low scores from both humans (0.40) and PRISM (0.38), reflecting high polysemanticity. The cluster descriptions here are diverse and capture distinct concepts. In contrast, the second example (bottom row) shows a monosemantic SAE feature in Gemma Scope: both human and PRISM scores are high, and all labels consistently reference "time", differing only in context. Additionally, our comparative analysis of polysemanticity scores assigned by humans and models, shown in Figure 6 indicates an overall strong and consistent alignment between human annotations and PRISM scores.
Discussion and Conclusion
In this work, we propose PRISM, a novel framework for identifying multi-concept feature de- scriptions in LLMs. Our framework also allows us to ground resulting descriptions in systematic evaluation approaches, by including description score that provides a description quality measure. Thus, PRISM not only provides an automated interpretability of model components via human- interpretable descriptions, but is the first framework that addresses the challenge of detecting more
8

---

Sentence clusters
Cluster descriptions
PRCSM
"Now Your. Winter holiday, Christees backgrouns.
'holiday or spccial occasion acsstories/daerrattioo
Layer 48, Feature 6967
"Iler charm, confidence and regal style copturedd
"Produst descraptions and features in the context 6f celebrations
Hunan
exting nalding apper like shaped necklace."
"ascurtoned to seeting the president deliver the sony
Viscusesons formal sceeches by preesdente"
(GP1-2 M. [LP)
"that wes Presidant Mixon had ressgn"
"she wax First presd dentinll apaase textäty"
'of'i)))) titles of personslittee institutions formal, eventse
"We will revisit these concepts in [...l l..l.""
"Bcallaction un umagee iupersentro, aetia, anenific stern"
of adocationnl
free online verbelizing English cictionary.
creative warks,
sepuratis the dialague from the stardard'
"Learning, using, tolking about language, lsterature, creative work"
"Pythum indir inna to ancoda decada L...!" [...]"
"Descriptinss explanations, or instructions related nethoda prsoniaad
lambodas
"Lo.al Sncceding og addressiin language Internals)"
"echncoal instructions, essecially in the concext of computing. language, and education"
0.38
0 49
"prosucer for the eventing of mustoll entertaimentt
"Additionally
special thanks sponsors" create the forld Prentere an exciting nan ballet"
EEvente, often with a time andfor a date"
Lawer Codiine Similartty high palysamantillly
"ree of day. locations, events and perforrances
"I....] woke the week abood a breeze [.vel"
"Unito of time (days, woeka, ponths, years, houra, minutes, evendingo nights,, weskends,
Layer 10, Feature 603
"A mannth an age found purple [...]*
Las, lifetires)"
day time (slong with night time) Ie..
"Tiee references
days, veeks, months, years hours relation to events, routinns or durations"
(Germ Scope, resad)
"paskages include year round prosction, I...'
"School sursig the academso year in [ood]
"Unite of time like daya, nonthe, and years"
"HHlf Day Courses, Full Day Courscs [ood="
Taneet events and otfers, aditmistration and regulation"
"the seeson opener [.ri] the night before"
time, often seheduling
context duration, repetitior,
"I...]
during their incklly eset."
similar call. the group wade one year ago"
"Taee periods darane days nonths or years, tares of cay, repetitive Lyclus, schetilles"
"Way in i bway month, with versous key I...I" "thear
"Units of time"
[...]"
"rost commoon questionn se get every year."
"Tine, pariods"
0 0.74
0.80
"In the atternoon. head out to enjoy I...'" sounded like levely [...]"
"Times of day, of the week/year, langer durations of "itee"
Oosine Similarity nigh monosememictty
camping permit and atay evernight.
"Iinefranes, nature, travel and evente"
Figure 5: Cluster labeling comparison between human and PRISM (LLM). We compare cluster labeling for two features: a polysemantic feature (top row) and a monosemantic feature (bottom row). On the left, we show representative text spans from input samples that strongly activate the feature, grouped into five clusters based on shared patterns. Within each span, tokens with the highest activations are highlighted. On the right, we compare the cluster labels generated by PRISM (LLM-based) and a human annotator shown the same input as the model. Additionally, the human rates the conceptual coherence of the five cluster labels on a scale from 0.0 to 1.0, where lower values indicate more diverse (polysemantic) and higher values more consistent (monosemantic) labeling. This rating is directly compared with PRISM's polysemanticity score for the same feature.
than one activation pattern a model feature is sensitive to and allows us to quantify variations in the activation patterns using the framework's 's polysemanticity score.
We conduct several experiments demonstrating the performance and analytical applications of PRISM. Our results show that multi-concept descriptions more accurately distinguish target concepts from control data, with statistically more distinct mean activations. PRISM also extends prior work by capturing variation in description quality 33] and differences in polysemanticity across model layers 773 Beyond qualitative analysis, PRISM's multi-concept approach and novel polysematicity scoring provide new directions for studying the structure and interpretability of model features. In exploring the concept space, use PRISM to characterize more complex components, finding and interpreting patterns that specific attention heads or groups of neurons respond to. Deep learning systems utilize distributed features that, in principle, allow for compositionality. By providing description of more complex sets of interacting components, we can tackle a timely challenge to the community. Lastly, we took step towards testing the alignment of our PRISM framework with human interpretation. Especially rating polysemanticity is a complex task for humans, even when presented with sentence examples. Our results highlight that the PRISM framework not only provides multiple human interpretable descriptions for neurons but also aligns with the human interpretation of polysemanticity.
Limitations As we have focused on textual explanations, our descriptions are limited to concepts expressible natural language, and thus may be unable to capture complex syntactic structures like graphs or algorithmic concepts where the underlying operation is not easily described in the constrained vocabulary space. Fixing the number of clusters in PRISM cannot guarantee that provided descriptions capture all or the most salient concepts for a feature; instead, they represent

---

1.0
Cosine Similarity
0.8
0.-
0.4
0.2
Human palysemanticity
Similarity
0.0
PRISM polysemanticity
Layer
Layer Feature 6867
Layer Feature 12733
Layer 28,
Layer 18, Layer
Layer
Layer Feature
Feature 30319
Festure
Feature
Festure
Feature 18531
Figure 6: Results of the human evaluation study. Each group on the x-axis corresponds one feature (layer, feature, model, feature type). For each feature, we show Human polysemanticity scores (red boxplots) from participants' ratings of cluster descriptions, Human Cosine Similarity (purple boxplots) computed with sentence embeddings of the same human-wrr written descriptions, and the corresponding PRISM polysemanticity score (blue point marker). Lower values indicate higher polysemanticity. The results illustrate that features judged by PRISM polysemantic receive both lower human similarity ratings and lower embedding- based similarity, while features with high PRISM scores show higher human scores.
subset of key concepts the feature is responsive to. Parametric measures such as MAD are sensitive to outliers, especially in NLP settings where activations often follow heavy-tailed distributions. To address this, we pair MAD with the more robust AUROC score, yielding a more comprehensive view of feature behavior. Finally, reliance maximally activating corpus examples restrict interpretations to observed concepts, limiting coverage of rare patterns or out-of-distribution effects across datasets 77]
Future Work A promising next step is to study how the method's performance changes as the number of clusters varies, and to examine whether alternative clustering techniques can uncover a hierarchical organization of the resulting descriptions. Beyond quantitative evaluation, also stress the need for rigorously designed numan--vvluatio protocols and community benchmarks that provide transparent, standardized measures of progress in automated interpretability.
References
[1] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition code generation with alphacode. Science, 37866624)
2022.
[2] Tao Tu, Mike Schaekermano, Anil Palepu, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Yong Cheng, Elahe Vedadi, Nenad Tomasev, Shekoofeh Azizi, Karan Singhal, Le Hou, Albert Webson, Kavita Kulkarni, Sara Mahdavi, Christopher Semturs, Juraj Gottweis, Joelle Barral, Katherine Chou, Greg S. Corrado, Yossi Matias, Alan Karthikesalingam, and Vivek Natarajan. Towards conversational diagnostic artificial intelligence. Nature, pages 1-9, April 2025. Publisher: Nature Publishing Group
[3] Kevin Ro Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023.
[4] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations, 2023.
[5] Olah, Chris, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom In: An Introduction to Circuits, 2020.
10

---

[6] Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah Goodman, and Christopher Potts. Inducing causal structure for interpretable neural networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 7324-7338. PMLR, 7 2022.
[7] Thomas Schnake, Oliver Eberle, Jonas Lederer, Shinichi Nakajima, Kristof T. Schütt, Klaus- Robert Müller, and Grégoire Montavon. Higher-order explanations of graph neural networks via relevant walks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11):7581- 7596, 2022.
[8] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020. Association for Computational Linguistics.
[9] Ameee Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Rober Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International conference on machine learning, pages 135-451. PMLR, 2022.
[10] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. eesa-Bianchii, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[11] Andreas Madsen, Sarath Chandar, and Siva Reddy. Are self-explanations from large language models faithful? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACI. 2024, pages 295-337, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[12] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: Quantifying Interpretability of Deep Visual Representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3541-6549, 2017.
[13] Jesse Mu and Jacob Andreas. Compositional Explanations of Neurons. In Advances in Neural Information Processing Systems, volume 33, pages 17153-17163. Curran Associates, Inc., 2020.
[14] Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina Höhne. Labeling Neural Representations with Inverse Recognition. Advances in Neural Information Processing Systems, 36:24804-24828, December 2023.
[15] Clement Neo, Shay B Cohen, and Fazl Barez. Interpreting Context Look-ups in Transformers: Investigating Attention-ML. Interactions. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16681-16697, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[16] Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polyseman- ticity and Capacity in Neural Networks. CoRR, 2022.
[17] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy Models of Superposition, September 2022. arXiv:2209.1066 [cs].
[18]
Andrew Ng. Sparse autoencoder, 2011. CS294A Lecture notes 72.
[19]
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravee, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Chris Olah. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, October 2023.

---

[20] OpenAl. Automated interpretabilityy. http::///gggtttoooooooo/aaa/aaaa http://tttuu..cm/pppmiiitt
ov www.iveewisc------eeeeeee lists i neuro
ns 2023. Accessed: 2025-05-14.
[21] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models, May 2023.
[22] Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, October 2024.
[23] Dami Choi, Vincent Huang, Kevin Meng, Daniel D. Johnson, Jacob Steinhardt, and Sarah Schwettmann. Scaling Automatic Neuron Description I'Tansluce AI, October 2024.
[24] Xuemin Yu, Fahim Dalvi, Nadir Durrani, Marzia Nouri, and Hassan Sajjad. Latent Concept- based Explanation NLP Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 12435-12459, Miami, Florida, USA, November 2024, Association for Computational Linguistics.
[25] Yoav Gur-Arieh, Roy Mayan, Chen Agassy, Attieus Geiger, and Mor Geva. Enhancing Au- tomated Interpretability with Output-Centrie Feature Descriptions. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5757-5778, Vienna, Austria, July 2025. Association for Computational Linguisties.
[26] Hoagy Cunningham, Aidan Ewart, Logan Riggs Smith, Robert Huben, and Lee Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models. In The Twelfth International Conference on Learning Representations, 2024.
[27] Zhengfu He, Wentao Shu, Xuyang Ge, Lingjie Chen, Junxuan Wang, Yunhua Zhou, Frances Liu, Qipeng Guo, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang, and Xipeng Qiu. Llama Scope: Extracting Millions of Features from 1lama-3.1-8B with Sparse Autoencoders, October 2024. arXiv:2410.20533 [es].
[28) Gonçalo Santos Paulo, Alex Troy Mallen, Caden Juang, and Nora Belrose. Automatically Interpreting Millions of Features in Large Language Models. In Forty-second International Conference Machine Learning, June 2025.
[29] Thomas McGrath, Daniel Balsam, Liv Gorton, Murat Cubuktepe. Myra Deng, Nam Nguyen, Akshaj Jain, Thariq Shihipar, and Eric Ho. Mapping the Latent Space of Llama 3.3 70B Goodfire Papers.
[30] Jack Lindsey, Hoagy Cunningham, Tom Conerly, and Adly Templeton. Interpretability Evals for Dictionary Learning, 2024.
[31] Jing Huang, Atticus Geiger, Karel D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rigorously Assessing Natural Language Explanations of Neurons. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 317-331, Singapore, 2023. Association for Computational Linguistics.
[32] Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees. Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024.
[33]
Roee Aharoni and Yoav Goldberg. Unsupervised domain clusters pretrained language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online, July 2020. Association for Computational Linguistics.
12

---

[34] Maarten Grootendorst. BERTopic: Neural topic modeling with class-based TF-IDF procedure. arXiv, abs/2203.05794. 2022.
[35] Michelle Lam, Janice Teoh, James A. Landay, Jeffrey Heer, and Michael S. Bernstein. Concept induction: Analyzing unstructured text with high-level concepts using lloom. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI New York, NY, USA, 2024, Association for Computing Machinery.
[36] Zihan Wang, Jingho Shang, and Ruiqi Zhong. Goal-driven explainable clustering via language descriptions. Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pages 10626-10649, Singapore, December 2023, Association for Computational Linguisties.
[37] Raj Jain and Imrich Chlamtac. The p2 algorithm for dynamic calculation of quantiles and histograms without storing observations. Communications of the ACM, 28(10):1076-1085 1985.
[38] S. Lloyd. Least squares quantization in PCM. IEEE Transactions on Information Theory. 28(2): March 1982.
[39] Laura Kopf, Philine L. Bommer, Anna Hedström, Sebastian Lapuschkin, Marina M. Höhne, and Kirill Bykov. CoSy: Evaluating Textual Explanations of Neurons. Advances in Neural Information Processing Systems, 37:34656-34683 -3685,, December 2024.
[40] Ben Allal, Loubna, Lozhkov, Anton, Penedo, Guilherme, Wolf, Thomas, and Werra, Leandro. Cosmopedia, December 2024.
[41] Colin Raffll, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[42] Johnny Lin, Neuronpedia: Interactive Reference and Tooling for Analyzing Neural Networks, 2023. Software available from neuronpedia.org
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAl, 2019.
[44] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The llama 3 herd of models. arXiv, abs/2407.21783. 2024.
[45] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, Janos Kramar, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 278-300, Miami, Florida, US. November 2024. Association for Computational Linguistics.
[46] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards General Text Embeddings with Multi-stage Contrastive Learning, August 2023. arXiv:2308.0320 00328888.03811 [cs].
[47] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Gar- rett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv, abs/2403.05550 2024.
[48] Gheorghe Comanici, Eric Bieber, Mike Schackermann. Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality. long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.062611 2025.
[49] Justin Lee, Tuomas Oikarinen, Arjun Chatha, Keng-Chi Chang, Yilan Chen, and Tsui-Wei Weng. The Importance of Prompt Tuning for Automated Neuron Explanations. In NewrIPS Workshop on Attributing Model Behavior at Scale, October 2023,
13

---

[50] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, al. Qwen3 technical report. arXiv preprint MiXiv:2505.09388 2025.
[51] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint (.Xiv:241.008900 2024.
[52] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945, 2025.
[53] Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman, Language models don't always say what they think: Unfaithful explanations in tfann-of-hhough prompting. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
[54] Oliver Bentham, Nathan Stringham, and Ana Marasovic. Shain-of-thouuhh unfaithfulness as disguised accuracy. Transactions on Machine Learning Research, 2024. Reproducibility Certification.
[55] Henning Bartsch, Ole Jorgensen, Domenic Rosati, Jason Bpelscher-Ohermaiee and Jacob Pfau. Self-consistency of large language models under ambiguity. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 89-105, Singapore, December 2023. Association for Computational Linguistics.
[56] Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H. Gilpin. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv,, abs/2310 10. 11207 2023.
[57] Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, Stella Biderman, Adria Garriga-Alonso, Arthur Commy, Neel Nanda, Jessica Rumbelow, Martin Wattenberg, Nandi Schoots, Joseph Miller, Eric J. Michaud. Stephen Casper. Max Tegmark, William Saunders, David Bau, Eric Todd, Atticus Geiger. Mor Geva, Jesse Hoogland, Daniel Murfet, and Tom McGrath. Open problems in mechanistic interpretability, 2025.
[58] Gottlob Frege Uber Sinn Und Bedeutung. Zeitschrift für Philosophie Und Philosophische Kritik, 100(1))55500 1892 Publisher: Duke University Press.
[59] Alfred Tarski. The Concept of Truth in Formalized Languages. In Alfred Tarski, editor, Logic, semantics, metamathematics, pages 152-278. Clarendon Press, 1956.
[60] Noam Chomsky. Syntactic structures. Syntactic structures. Mouton, Oxford, England, 1957.
Pages: 116.
[61] Jerry Fodor. The Language of Thought. Harvard University Press, 1975.
[62] Stephen C Levinson. Pragmatics. Cambridge UP, 1983.
[63] Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. Do neural language models show preferences for syntactic formalisms? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, pages 4077-4091, Online, July 2020. Association for Computational Linguistics.
[64] Lorenz Linhardt, Tom Neuhäuser, Lenka Tětková, and Oliver Eberle. Cat, rat, meow: On the alignment of language model and human term-similarity judgments. In Second Workshop on Representational Alignment ICLR 2025, 2025.
[65] Ilker Yildirim and A. Paul, From task structures to world models: what do LLMS know? Trends in Cognitive Sciences, 484 May 2024. Publisher: Elsevier.
14

---

[66] Zichao Li, Yanshuai Cao, and Jackie C.K. Cheung. Do Iims build world representations? probing through the lens of state abstraction. In A. Globerson, L. Mackey. D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang. editors, Advances in Neural Information Processing Systems, volume 37, pages 98009-9009 -90022 Curran Associates, Inc., 2024.
[67] Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Josh Tenenbaum. Evaluating statistical language models as pragmatic reasoners. In CogSci, January 2023.
[68] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):661, 2018.
[69] Kirill Bykov, Mayukh Deb, Dennis Grinwald, Klaus Robert Muller, and Marina MC Höhne. DORA: Exploring outlier representations in deep neural networks. Transactions on Machine Learning Research, 2023.
[70] Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebas- tian Lapuschkin, and Wojciech Samek. Mechanistic understanding and validation of large ai models with semanticlens. Nature Machine Intelligence, pages 1-14, 2025.
[71] Minyoung Huh, Brian Cheung. Tongzhou Wang, and Phillip Isola. Position: The platonic representation hypothesis. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarleu, and Felix Berkenkamp, editors, Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20617-20642. PMLR, 7 2024.
[72] Lennart Bürger, Fred A. Hamprecht, and Boaz Nadler. Truth is universal: Robust detection of lies in LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[73] Simon C Marshall and Jan H Kirchner. Understanding polysemanticity in neural networks through coding theory. arXiv preprint arXiv:2401.17975 2024.
[74] Tolga Bolukbasi, Adam Pearce, Ann Yuan, Andy Coenen, Emily Reif, Fernanda Viégas, and Martin Wattenberg. An Interpretability Illusion for BERT. Technical Report 11Xiv:2104.07143, arXiv, April 2021. arXiv:2104.07149 143 [cs] type: article.
[75] Pattarawat Chormai, Jan Herrmann, Klaus-Robert Müller, and Grégoire Montavon. Disentangled explanations of neural network predictions by finding relevant subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(11):7283-7299 2024.
[76] Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus- Roburt Müller, Matteo Valleriani, and Grégoire Montavon. Building and interpreting deep similarity models. IEEE Transactions Puttern Analysis and Machine Intelligence, 44(3):1 (49-111199111,,, 2022.
[77] Jacob Kauffmann, Malte Esders, Lukas Ruff, Grégoire Montavon, Wojeiech Samek, and Klaus- Robert Müller. From clustering to cluster explanations via neural networks. IEEE transactions on neural networks and learning systems, 35(2):1926-1940, 2022.
[78] Alexandros Vasileiou and Oliver Eberle. Explaining text similarity in transformer models. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pages 7859-7873, Mexico City, Mexico, June 2024, Association for Computational Linguistics.
[79] Antonin Poché, Alon Jacovi, Agustin Martin Picard, Victor Boutin, and Fanny Jourdan. ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatability. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5594-5615, Vienna, Austria, July 2025. Association for Computational Linguistics.
[80] Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models, November 2023, MXXv:2305.00863.
15

---

[81] Bruno Puri, Aakriti Jain, Elena Golimblevskaia, Patrick Kahardipraja, Thomas Wiegand, Woj- ciech Samek, and Sebastian Lapuschkin. FADE: Why bad descriptions happen to good features. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACI. 2025, pages 17138-17160, Vienna, Austria, July 2025, Association for Computational Linguistics.
[82] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
16

---

A Appendix
A.1 Related Work
A wide variety of works seek examine the mechanisms of generative AI systems. Many initially focused on attributions in imput space [ 9]. with later approaches focused on discovering circuit and graph structures B 4 [T, well self-explanations generated by the model [10] [... To extract relevant representations and model features, a variety of works have explored interpretability for analyzing concepts 773 representations (76,, and clustering [77] in the context of attributions of sentence summaries
[78] and concept discovery in LLMs [79]. In the following, we will focus
specifically studies and methods centered on feature descriptions.
A.2 Feature Description Methods
One of the earliest works on feature description in language models is SASC (Summarize and Score) [80]. which generates natural language descriptions of neurons in a pre-trained BERT model. Shortly thereafter, an automated interpretability method for describing all neurons in GPT-2 XL was proposed 21). This approach analyzes the textual patterns that cause neuron to activate, and uses GPT-4 as explainer model to generate a description of the neuron's 's function. Given set of token-activation pairs derived from text excerpts and corresponding neuron activations, the explainer model identifies common patterns, based which it generates textual description of the neuron's 's role. This method has since been widely adopted and further developed, forming the basis for many subsequent methods targeting both individual neurons (23 25 and SAE features (26 19 42 22 28
27 45 29 25
An overview of representative feature description methods is provided Table
Table 2: Representative feature description methods for language models, listed in chronological order, including the model and feature types they target.
Method
Target Model
Feature Type
SASC
BERT GPT-2 XL
neuron
GPT-Explain 21
neuron
Pythia SAE 26
Pythia 70M and Pythia 410M
SAE fcature
Anthropic SAE 19
one-layer transformer
SAE feature
DeepSeek RI Dist Llama 8B, Gemma 2B, Gemma 2 2B IT,
Gemma 2 9B, Gemma 2 9B IT, GPT OSS 20B, GPT-2 Small, Llama 3.1 8B, Llama 3.1 8B Instruct, Pythia 70M Deduped, Qwen 2.5 7B IT, Qwen 3 4B GPT-2 Small
Neuronpedia 42
SAE feature
GPT-2 SAE
22
SAE feature
GPT-4 SAE
22
GPT-4
SAE feature
FleutherAl SAF
28
Llama 3.1 & Gemma
SAE feature
Transluce-Fxplai
Iaama 3.1 8B Instruct
neuron
Ilama Scope 27
I Llama 3.1 8B Base
SAE feature
Gemma Scope +5
Gemma 2
SAE feature
Goodfire SAE
29
Llama 3.3 70B
SAE feature
Output-Centric neuron 25
Llama 3.1 8B Instruct
neuron
Output-Centric SAE 25
Gemma 2 2B, GPT-2 Small, Llama 3.1 8B
SAE fcature
A.3 Description Scoring Details
Figure illustrates the CoSy evaluation procedure 339 as adapted for language models. As the control dataset Xn, we use a subset of 1,000 randomly sampled entries from Cosmopedia 44) For each candidate description of target feature, we use Gemini 1.5 Pro [7] generate 10 concept- specific text samples, each with a maximum length of 512 tokens. These samples form the concept dataset X,. The generation prompt shown in Figure g We then pass both datasets through the model to extract activations corresponding to the target feature. We then use Average Pooling as aggregation function [d + to each activation vector to obtain scalar representations:
As = A = (o(fe,(a))) ((e E
E R",
(8)
The resulting activation distributions A and A are compared to compute the CoSY Score (see Equations 6 and 7), which quantifies how accurately given description captures the target feature.
17

---

Higher scores indicate clearer separation between concept and control samples, reflecting more precise and informative descriptions.
Step 1: Generate Synthetic Data
"Treageeet the 1,408 cops speniing feor years Foo Touer Uhe nouro yours the dccuratce Lesi
Step Collect Faature Activations
"Caanistiis, spocttisstls
screlshed
Step Score Explanations
"Farserak soparterces
"I.. dees aparecsattio
Wy stroegest entlehoce eseerias are Local" pour ectence
"Tdkrfentte arcioles before moxes"
A costusatire as mys oucerocee Tool" sorrg upprosselyy
Fuature 2015
High WOlent
The case of secual mndur has untcl la segght is h..." "ooi exploration cxcuttive search Tord"
"c.el certatety fenure will resumble
Figure 7: Score feature descriptions with CoSY. First, we compile multiple candidate descriptions for target feature. For each description, we prompt LLM generate 10 text samples including the described concept. These concept-specific samples, along with a control set of random text samples, are processed through the model to extract activations for the target feature. The CoSY Score quantifies the separation between activation distributions of concept samples versus control samples, enabling objective comparison of different feature descriptions. Higher scores indicate descriptions that better capture the feature's 's underlying concept.
Generate sentences with a length of 512 words, one per line, with no additional for- matting, introduction, explanation. Each sentence should be complete, standalone text sample that can be saved as an individual row in a text file. The sentences should include: @eattrr_descrippe
Figure 8: Prompt used generate concept-speeific text samples for evaluation. The placeholder [feature_ cetpuudddderripiionooee is replaced with a candidate textual feature description before being passed large language model (Gemini 1.5 Pro). The model then generates 10 standalone text samples, which form the concept dataset X1. These samples are used to evaluate how well the description aligns with the target feature.
A.4 Benchmark Experiment Details
We use the publicly available descriptions for GPT-Explain Transluce- Explain Neuronpedia [T] and Output-Centriq comparison.
Reference Descriptions
Model Layers For GPT-2 XL we use layers 0, 20, and 40. For Llama 3.1 8B Instruct we
sample from layers 0, 20, and 30. For GPT-2 Small SAE, we the original implementation [5
specifically version 5 with a width of 32k. We select features from layers 0, 5, and 10. For Gemma Scope use the residual stream SAE with width 16, selecting features from layers 0, 10, and 20.
Feature Selection For each model, we randomly choose 60 features, 20 from each of three layers, with available reference descriptions from prior work. The only exception is GPT-2 Small SAE, where only 59 features are annotated in the Output-Centric benchmark.
@#public-datesets
Instruct
18

---

Prompt for generating Descriptions To produce textual feature descriptions, we use prompt that instructs a large language model (Gemini 1.5 Pro) identify shared concepts across nigh-activation text excerpts within a cluster. The model receives the top N. = 20 excerpts per cluster, with high-activation token spans highlighted using square brackets. The full prompt is shown in Figure
You are meticulous AI researcher conducting an important investigation into specific neuron inside language model that activates in response to text excerpts. Each text starts with ">" and has header indicated by === Text 234 ==, where # #1234 can be any number and the identifier of the text.
Neurons activate on word-by- word-by-woodd word basis. Also, neuron activations can only depend on words before the word it activates on, the description cannot depend on words that come after, and should only depend on words that come before the activation.
Your task is to describe what the common pattern is within the following texts. From the provided list of text excerpts, identify the concepts that trigger the activation of a particular feature. recurring pattern or theme emerges where these concepts appear consistently, describe this pattern. Focus especially on the spans and tokens in each example that are inside a set of [delimiters] and consider the contexts they are in. The highlighted spans correspond very important patterns.
At the beginning, before the list of texts, there will be list of the highlighted tokens with their activation values.
At the end, following 'Description'', your task is to write the description that fits the above criteria the best.
Do NOT just list the highlighted words!
Do NOT cite any words from the texts using quotation marks, but try to find overarching concepts instead!
Do NOT write an entire sentence!
Do NOT finish the description with full stop!
Do NOT mention the [delimiters] in the description!
Do NOT include phrases like "highlighted spans', 'Concepts of', or 'Concepts related to', and instead only state the actual semantics!
Do NOT start with Descriptionn' and instead only state the description itself!
Figure 9: Prompt used to generate textual descriptions of a feature based on its activation patterns. The language model (Gemini 1.5 Pro) is instructed analyze set of text excerpts, focusing on highlighted spans corresponding to high activations of a specific feature. The model is guided to identify consistent patterns concepts that trigger the feature. The resulting output a concept-level description used as textual feature description.
AUROC and MAD Distributions To better understand the standard deviation observed in benchmark results, we provide distribution plots of the evaluation metrics. Figure I0] shows the distribution of AUROC scores across all evaluated model features, while Figure 11 presents the distribution of MAD scores.
Output-Centric Evaluation In addition to our activation-based metrics, AUC and MAD, we evaluate an output-centric metric, Faithfulness, which quantifies the causal influence of a discovered concept on the model's 's output [25]. Faithfulness measures the causal effect of feature on model outputs, specifically testing whether directly manipulating a feature's activations can steer the model to generate content that more strongly reflects the corresponding concept. Following the FADE 17 implementation of [1,, we compute Faithfulness scores of GPT-2 XL. feature descriptions using GPT-4o mini (22 as the language model and subset of 10,000 samples from the training split of the Cosmopedia dataset [40]. To obtain scores for all features, we remove the threshold used for selecting output-centric samples. Figure 12 presents the results of the output-centric evaluation. Across layer groups, PRISM (max) consistently outperforms GPT-Explain, exhibiting trends that closely mirror those observed in the activation-centric evaluation (see Figure 3(a). further validating its effectiveness across multiple interpretability criteria.
hhttps://gitaiii http://///tttubb..o//buuubbbbbt
19

---

Frequency
Frequency
20
10
4+
AUROC
AUROC
(a) GPT-2 XL.
(b) Llama 3.1 8B Instruct.
30
Frequency
Frequency
15
02
0.4
0.
0.0
1.0
0.0
0.2
2.4
0.6
0.5
1.0
AUROC
AUROC
(c) GPT-2 Small SAE.
(d) Gemma Scope SAE.
Figure 10: Distributions of PRISM (max) AUROC scores across different models and layers.
12
30
frequency
Rraquecc
10
5.0
7.5
15.0
MAD
MAD
(a) GPT-2 XL.
(b) Llama 3.1 Instruct.
40
frequency
40
Frqquecy
25
80
20
10
(c) GPT-2 Small SAE.
(d) Gemma Scope SAE.
Figure 11: Distributions of PRISM (max) MAD scores across different models and layers.
Compute Resources All experiments were conducted using single NVIDIA A100 80GB GPU. The description procedure takes approximately 9 minutes per feature, including percentile sampling. clustering, and the generation of 5 descriptions. For evaluation, the generation of 10 sentences per feature requires roughly 3 minutes.
A.5 Extended Ablation Analysis
Impact of Varying Cluster Size We evaluate PRISM's performance across different numbers of clusters (k) for generating feature descriptions for GPT-2 XL. As shown in Table 3 increasing
20

---

PRISM (max)
0.12
GPT-Explain
PRISM (meea))
0.10
Faithfulness
0.08
0.06
0.04
0.02
0.00
early
middle
late
Layer
Figure 12: Output-Centric evaluation of GPT-2 XL. across different feature description methods (GPT-Explain, PRISM (max), PRISM (mean)). Each method is evaluated using the output-centric Faithfulness metric, which quantifies the causal influence of a discovered concept on the model's s output. PRISM (max) consistently achieves higher Faithfulness compared to GPT-Explain.
Table Impact of the number of clusters (k) on feature description quality for GPT-2 XL. A cluster size of k = 5 corresponds to the original implementation used throughout the paper. Increasing k improves best-case descriptions (higher PRISM max scores) by capturing more specific activation patterns, but lowers average interpretability (lower PRISM mean scores) as patterns fragment across clusters. This illustrates the tradeoff between precision and redundancy. AUROC values are reported with 95% confidence intervals; MAD values are reported the percentage of positive MAD scores.
PRISM (max)
PRISM (mean)
k Clusters
AUROC (1)
MAD (1)
AUROC (1)
MAD (1)
5 (original)
0.85 (0.78-0.91)
91.67%
0.65 (0.61-0.69)
66.33%
0.75 (0.66-0.83)
80.00%
0.75 (0.66-0.83)
80.00%
3
0.82 (0.74-0.89)
81.67%
0.69 (0.64-0.74)
68.33%
10
0.88 (0.83-0.93)
93.33%
0.61 (0.58-0.64)
59.67%
improves best-case description quality (higher PRISM max scores), since clusters capture more specific activation patterns and yield sharper labels for the most coherent ones. At the same time, larger reduces average interpretability (lower PRISM mean scores), as coherent patterns become fragmented into statistically indistinguishable subclusters. These results illustrate the tradeoff: in- creasing the number of clusters enhances precision for the best descriptions but introduces redundancy and semantic overlap across clusters.
Text Generators for Description Generation To test the robustness of our framework across different text generators, we extended the description generation experiments beyond Gemini 1.5
Pro to several open-source language models: Qwen3 32B [50 IN Phi-4 [5 DeepSeek R1 [52 20
These models were used to generate feature descriptions for GPT-2 XL using the same procedure as
in our original setup described in Section 4.1 As shown in Table 4 Qwen3 32B achieves performance
comparable to Gemini 1.5 Pro, demonstrating that our framework does not depend solely on single model. While Phi-4 and DeepSeek R1 show slightly lower scores, they still follow the same qualitative trends, demonstrating that our method is robust and generalizes effectively across a range of language models, including accessible open-source options.
Text Generators for Evaluation We evaluated the robustness of our feature descriptions by using Qwen3 32B 50 Phi-4 551. DeepSeek RI 52 as alternative text generators in the evaluation
1 httpss http:///hhhhggggggac.eo/////e//e 32B
21

---

Table 4: Ablation study on text generators used for description generation. We report PRISM scores (max and mean) for different LLMs using the original experimental setup. Qwen3 32B achieves performance comparable to Gemini 1.5 Pro, while Phi-4 and DeepSeek RI exhibit slightly lower scores but maintain the same qualitative trends, demonstrating the framework's effectiveness across multiple language models. Reported AUROC includes 95% confidence intervals; reported MAD
represents the percentage of positive MAD scores.
PRISM (max)
PRISM (mean)
Text Generator (description)
AUROC (1)
MAD (1)
AUROC (1)
MAD (f)
Gemini 1.5 Pro 47
(original)
0.85 (0.78-0.91)
91.67%
0.65 (0.61-0.69)
66.33%
Qwen3 32B [50]
0.85 (0.78-0.91)
90.00%
0.65 (0.61-0.69)
64.33%
Phi-4 [51]
0.82 (0.75-0.89)
85.00%
0.61 (0.57-0.65)
61.33%
DeepSeek R1 52
0.79 (0.71-0.87)
78.33%
0.61 (0.56-0.65)
60.33%
Table 5: Ablation study on text generators used in the evaluation step. We report AUROC and MAD scores for PRISM and GPT-Explain when using Gemini 1.5 Pro (original setup) or alternative LLMs (Qwen3 32B, Phi-4, DeepSeek R1) )) for description scoring. While absolute scores decrease with alternative models, relative rankings remain stable, with PRISM consistently outperforming GPT-Explain. AUROC is reported with 95% confidence intervals, and MAD is reported as the
percentage of positive MAD scores.
PRISM (max)
PRISM (mean)
GPT-Explain
Text Generator (evaluation)
AUROC (f)
MAD (7)
AUROC (?)
MAD (1)
AUROC (1)
MAD (1)
Gemini 1,5 Pro 47
(original)
0.85 (0.78-4 (0.78-0.91)
91.67%
0.65 (0.61-0.69)
66.33%
0.64 (0.56-0.73)
65.00%
Qwen3 32B
50
0.58 (0.46-0.69)
58.33%
0.53 (0.48-0.58)
54.00%
0.54 (0:42-0.65)
53.33%
Phi-4
(0.50-0.72)
58.33%
0.54 (0.49-0.59)
53.67%
0.56 (0.44-0.67)
58.33%
DeepSeck R1 52
0.71 (0.61-0.80)
73.33%
0.57 (0.52-0.62)
57.67%
0.60 (0.50-0.70)
63.33%
step (Appendix A.3 provides details on description scoring). In this process, set of 10 concept- specific text samples for each feature is generated with the LLM, and AUROC and MAD scores are computed to assess description quality for both PRISM and GPT-Explain (Table 5). Although absolute scores are slightly lower when Gemini 1.5 Pro is not used for evaluation, the relative ranking of methods remains consistent. PRISM consistently outperforms GPT--xplain, demonstrating the generalizability of our framework and the robustness of our evaluation setup.
A.6 Extended Sanity Check Analysis
Random Sentences in Clusters To ensure that the generated feature descriptions are faithful to the tokens and corresponding text samples in the same cluster, we perform a fully randomized counter probe. For this sanity check, we replace the percentile sampling in Step 1 of Figure 22 with random sampling, drawing random sentences and their corresponding activations from the validation set of the C4 CORPUS [4 Since choosing highly activating samples across our random random uniform sample distribution, we apply the clustering procedure across all random samples. While maintain the cluster size, all text samples and highlights shuffled and assigned to random clusters instead. Both the assignment of random clusters and highlights limits sample similarity. Lastly, we proceed with labeling the randomized clusters. Each cluster consists of unrelated texts and highlights, thus, the LLM should be unable to generate a coherent, shared feature description. As shown in Tableo AUROC and MAD scores decrease, confirming that meaningful descriptions depend on the proper grouping of highly activating samples.
would bias the
Random Descriptions Experiment We probe whether our feature descriptions truly capture the concepts of their assigned clusters by performing an additional randomization of descriptions. We reassign feature descriptions by randomly sampling from existing descriptions generated for other features within the same model and collect the associated activations. This tests whether explanation quality decreases when descriptions no longer correspond to their clusters, without generating new descriptions. To this end, we use the same feature and layer selection as described Section 4. for GPT-2 XL. which serves as the baseline for non-randomized PRISM performance (see Table GPT-2 XI., MI.P neuron). We expect AUROC and MAD scores to drop significantly, since the descriptions do not align with the cluster. Table 6 shows that AUROC and MAD scores drop significantly
22

---

Table Sanity check results for GPT-2 XL. We compare AUROC and MAD scores under two ran- domized settings: (1) Random Sentences, where clusters are formed from unrelated text samples, and (2) Random Descriptions, where existing descriptions are randomly reassigned across features. Both conditions show notable drop in AUROC and MAD scores compared to the Baseline, supporting the faithfulness of our descriptions. AUROC reports 95% confidence intervals; MAD shows the percentage of positive MAD scores.
PRISM (max)
PRISM (mean)
AUROC (t)
MAD (t)
AUROC (t)
MAD (1)
Baseline
0.85 (0.78-0.91)
91.67%
0.65 (0.61-0.69)
66.33%
(1) Random Sentenees
0.68 (0.59-0.76)
65. 65.00%
0.54 (0.50-0.58)
49.67%
(2) Random Descriptions
0.65 (0.56-0.74)
66.67%
0.52 (0.48-0.57)
49.33%
compared the baseline, demonstrating that aligned descriptions are essential for high-quality feature description.
Random Descriptions Polysemanticity Scores Comparison To verify that polysemanticity scores reflect meaningful semantic relationships rather than artifacts of the embedding process, perform a sanity check using randomly assigned descriptions. For each feature, five random descriptions (distinct from the set) are embedded using the same embedding model getee-wen2-115B-instte sentence transformer 44)), and pairwise cosine similarities are computed between them. This is repeated across all features within each layer, and the resulting scores are visualized as box plots grouped by layer (see Figure 13). The difference between true and random scores is generally quite distinct, with random descriptions yielding notably lower similarity values, indicating less semantic coherence. An exception is Llama 3.1 8B Instruct, where cosine similarity scores remain consistently low across layers and frequently approach the random baseline. This indicates that the descriptions associated with Llama's 's features are likely more diverse and less semantically aligned compared to other models.
Tue
True
Random
Random
Cosine Similarity
0.7
Cosine Similarity
0.5
0.5
0.3
early
early
Layer
Layer
(a) GPT-2 XL.
(b) Llama 3.1 8B Instruct.
Cosine Similarity
Cosine Similarity
0.6-
0.6
0.4
0.4
early
middle Layer
late
early
middle Layer
labe
(c) GPT-22 Small SAE.
(d) Gemma Scope SAE.
Figure 13: Comparison of true and random polysemanticity scores across models and layer groups.
23

---

Table Performance across percentile intervals of feature scores on GPT-2 XL. AUROC measures
classification performance (95% CI in parentheses), and MAD quantifies activation differences
(percentage of positive MAD scores). Results are shown for both PRISM (max) and PRISM (mean).
PRISM (max)
PRISM (mean)
Intervals
AUROC (t)
MAD (1)
AUROC (1)
MAD (1)
Baseline
0.85 (0.78-0.91)
91.67%
0.65 (0.61-0.69)
66.33%
0.0 to 0.25
0.69 (0.61-0.77)
66.67%
0.53 (0.49-0.57)
49.33%
0.25 0.5
0.74 (0.66-0.82)
70.00%
0.56 (0.52-0.61)
52.67%
to 0.75
0.71 (0.63-0.79)
73.33%
0.55 (0.51-0.60)
54.33%
0.75 to 1.0
0.85 (0.79-0.91)
90.00%
0.65 (0.61-0.69)
66.33%
Percentile Interval Analysis We examine whether feature description quality varies across the percentile activation distribution by segmenting the distribution into quartile-based intervals. For consistency, we apply the same PRISM parameter settings as in the main benchmarking experiments, varying only the start and end points of the percentile range. The baseline setting uses the top I 1% of activations (0.99-1.0) Table 7 reports AUROC and MAD scores across intervals. As expected, features in the top 25% (0.75-1.00 closely match baseline performance, while lower intervals exhibit reduced scores, indicating positive correlation between activation strength and description quality.
Relative Activation Analysis To ensure that feature descriptions are based on relevant and repre- sentative samples, analyze the relative activations between the 99th and 100th percentile samples for each neuron. For a neuron j in a given layer and dataset, let amaa denote the maximum (100th percentile) activation and agg the 99th percentile activation. We define the relative ratio
Ugg
E [0, 1].
(9)
Omaz
We evaluate ry across three layers (0, 20, and 40) of GPT-2 XI. (see Figure 14), using the same settings as in our main experiments. In Layers 0 and 20, no neurons exhibit rj < 0.15. In Layer 40, the ratios are generally lower, but none fall below 0.05. This indicates that extremely small relative activations are rare, particularly in early layers. Furthermore, across layers, our method is able to robustly extract feature deseriptions, even for neurons with more diverse ry values (see also Figure (())..
14
14-
14
12
12-
12
10
10
Density
Density
Density
8-
4-
2-
8.0
0.2
0.4
0.6
0.8
1.0
8.0
0.2
0.1
0.6
0.8
1.0
8.0
0.2
0.4
0.6
0.8
1.0
(a) Layer 0
(b) Layer 20
(c) Layer 40
Figure 14: Distribution of the relative activation ratio between the 99th and 100th percentile samples across three layers (0, 20, 40) of GPT-2 XL. This ratio measures how representative the top neuron activations are compared to near-peak activations. Ratios in Layers and 20 are consistently higher, while Layer 40 shows lower values but none below 0.05, indicating that extreme outlier activations are rare.
To further illustrate that sampling from diverse activation ranges is meaningful in practice, we present a selection of feature descriptions. Table shows cases where clusters with diverse mean activation ranges (MeanAct) still achieve high description scores (AUROC) (also see Figure

---

Table 8: A selection of feature descriptions obtained from clusters with diverse mean activation ranges (MeanAct), Despite variation in activation ranges, the descriptions achieve consistently high
scores (AUROC), illustrating that percentile sampling captures multiple meaningful patterns per feature.
Model, layer, feature
Description
MeanAct
AUROC
GPT-2 XL. 47, 3815
Quantitics, spocifically numbers, and time periods Personal experiences or opinions
0.46
0.99
0.42
0.98
Indefinite articles preceding nouns related to events, times, groups, places, and objects
0.75
Llama 3.1 BB Instruct, 30, 6472
End-of-sequanee: tokens following deseriptions of services. products, academic programs
0.35
0.99
Locations, evenss, or entities and their associated details Possessives or the beginning of numbered lists
0.22
0.90
0.22
0.85
GPT-2 Small, 10, 4369
Financial institutions, people's 's names, and technical terminology related medical scans
3.08
0.58
A person's is name containing "ih"
4.55
0.89
International Business Machines (a technology company) and cystic librosás
2.95
Gemma Scope. 0, 12182
Products or items and their specifications or descriptions
1.33
0.84
Home improvmentss, dental procedures, taxes, and savings accounts Brexit negutiationsdeals
1,56
1.0
2,00
0.69
The analysis of relative activations in both percentile sampling and clustering suggest that percentile sampling reflects focus on diverse functional and meaningful patterns when present. This aligns with our goal of retrieving multiple distinct patterns per feature, resulting in multiple descriptions per feature.
A.7 Metalabels
In Figure 15 and Figure 16 we provide additional examples of metalabels for GPT-2 XI. and GPT-2 Small SAE. These resulted from clustering 300 sentence representations (embedder: GPT-2 XL, last-token pooling) of identified feature descriptions for a given model and neurons. We show 20 randomly selected samples from total of km 50 meta-clusters that were computed using k-means, along with up to three feature descriptions selected at random. Metalabel descriptions were generated via Gemini 1.5 Pro. Clusters for which no concise label was generated, are labeled with 'N/A'.
As discussed in Section 5.. for GPT-2 XL, similar patterns are observed for GPT-2 Small SAE (see Figure (16). including semantic categories like "Spatiotemporal Descriptions and Personal Anecdotes" (id 20), syntactic concepts like "Pattern Matching" (id 22), and task-specific representations such as "Instructional/Expllaatrrr (id 49) and domain-focused clusters like "Publication and Distribution" (id 35). Of particular interest are task- and -ppain-specific examples, such as "Listeria Contamination" (id 46), which suggest concepts linking semantics with pragmatics. These concepts convey context- specific intents, such as warning or ensuring public safety, highlighting how pragmatic factors influence model interpretation.
A.8 Human Study Details
Participants Seven participants from two different academic institutions took part in the study. All were either PhD students or working students and were compensated for their participation. None were co-authors or otherwise involved in the project. Completing the survey required an average of 140 minutes.
Study Setup Each participant was presented with 8 groups of sentence clusters, where each group corresponded to one feature and consisted of 5 clusters of highly activating sentences (20 sentences per cluster), along with highlighted tokens. For each cluster, participants were tasked to write short textual description, resulting
descriptions per feature. They then rated the pairwise similarity between these descriptions on an 1 ---ooint scale (0.0-1.0), where 1.0 indicates very high similarity. The instructions provided to participants are shown Figure 17
numan-generann
Results Figure presents the results, sorted by PRISM score (lower values indicate higher polyse- manticity). We report the average of these ratings as the Human polysemanticity score. In addition, we compute Human Cosine Similarity using the same embedding model gette-wee2--..55 1.5B-instruct sentence transformer [46)) and method used for computing the PRISM polysemanticity score, with the only change being the use of human-generated descriptions instead of moner-generated ones. As expected, features with low PRISM scores receive lower human similarity ratings and lower embedding- based similarity. For example, the feature from GPT-2 Small (layer 5, feature 30319) has low PRISM score (0.30), and is judged by participants to be semantically diverse (Human
25

---

polysemanticity score: 0.40). In contrast, features with high PRISM scores, such as those from Gemma Scope (features 10531 and 603), show strong human agreement (Human polysemanticity score: 0.90 and 0,80, respectively) and high Human Cosine Similarity (0.79 and 0.75). These findings provide both qualitative and quantitative support for the PRISM metric, showing that aligns well with the human interpretation of polysemanticity.
26

---

Motalabel
Feature Descripions
Technology and Spec ficetions
Settings, assignments actions related to software or appiiations
Taxile material, food and baverage, medicinal substances
Qualitie, characteristics, or specificatios of arimals products
Onling Discourse and New Experiences
first time experience, ofton with an elemant of surprise or anticipation Social madia, Donald Trump. Twitter, counsel, upbeat tweets Apologics, add ins, testaments, descendants, grades, versions, dialogue, honesty, downgrados payments, relevance. sensbiltyy timelessness, credit, superiars, decensy, hardened hearts, ageing, genuinelly tnghtened by realty, reliability strouded, credited, Ironically parmitedd, gigs, obsession,
Positive Experiences
Expressions of excitement sharing. or positive foodback
Expressions of gratitudo, current time relorences, or positive descriptonss
Experiances related to travel, lelsure activities, meals, and events, particularly those with temporal element (time, dates, or duration)
Commercel Finance
Months, numbors, and socond hand collectblcs, rontals, applications, retail spatoos, or holiday gifts
Holiday special occasion aosesoriesieeccralinn Food, fools/devices, and accessris
Accoss' Acquisition
Transter, storage, or placement of objects people
Consumptton food, beverages, medicalions, sometimes Winning a prize or participating in competlion
reduced price
13 Personal and Profes- sional Experiences
discussion customer service experiences, reigious figures and texts, series, sporting events, community events, restaurants, international summits, pot rules, summer camps, company performance. and international relations Proper nouns, often people's names, in contexts of compettions, scandals, or events, especially when related to games, sports, politice, or entertainmet
and Standards
Professional standards, requirements, and prastices related to a variety of fields, including surveying, reviswing productsisen- vices, nutriion, data analysis, education, healthcare, music, and spiritualty
Achievements, So-
Events, particularly these related confict. compettion. probiematic situations
Ittlons. Litestyla, Hisks, News and Con- Troversy, Conflicts, Corruption
Topics related to polltce, sports, and current events, specifically locusing on major decisions, outcomes, and controversies Government-relaae scandals and investiaatios, particularly those involving leaks, cover-upe, or accusations of wrongdoing
18 Structured Data
Numerical or quantitativ values, including years, measurements, and counts, advert like excerpts
Titles, names, localions, and dates in bibliograpic entries citations
Academic degrees, professlonal roles, locations, and years related to education or employment history
19 Development Im- provement
Mantal anolor physical manipulation or transtormation
Funding of projects and initiatives related to hoalthcare, social issues, and education
Mathods, procedures, or saquences related improvement, change, progression
21 Personal Rotiections
First-person accounts, otten expressing personal opinions. beliets, or experiences
Parsonal updates
Experiences, actions, and feelings related entertainment, media, and technology, along with personal anecdotes or opriioss
26 Producte// Services Medical Informa 1ion
Products services with dascriptions andior characteristics
Products or services with details or instructions
Medical conditions, types of medical treatment, or medical professionals, sometimes involing a duration or repeating pattern
27 NAA
Finess, essays, digital skills training, product dasign and markating, audits, internships, coaching, graphic design, academic assistance, data analytics, prodictive modeling, computational chemistry, handwriting development, software documentation development tools, intellectual properiy....
Business services, including careor services, company recruitmntt tours, and consulting, offered by organizations, for studonts and professionals, in various liclds, such as marketing, tochnology, and healtheare
Rape access, locations in Arkansas, educational courses, probation terms, Nigerian sid, hospital communication improve- ment, South Amorican airlnne travel, Nigorian profession improvement, admission school nursory. Florida
28 Business and Organi- zation Information
Locations of businesses or organizations
Geopelitccll events and entiiee involved, particularly government actions and agencies Evants, services, or products offered by a business or organization
29 Events and Activities
Achievements, awards, special events, atten including specific person group
Food, specilic occaiins, adivillesrruuiee, aften involing a change direction state Evants, shows, or locations, often with time or date, and sometimes including named people
35 Personal Develop
Solf awareness, identity and reality, otten related to tochnology and its impact on the user
ment and Manage- ment
Discussions of linancal, llfe, or career planning, resource sllocation, and caregiving, often in the contaxt of family or chicrrn Actions states baing, often ongoing recently completed
38 Diverse Inquiries and
Promational products or gits relating to cutoness and popularity with customers
Services"
Locations, ofien neighberhoods districts, and nained entiies ussociated with those locations
French language leamnig, professional certificatione and qualfications,
company services related specific industries
42 NIA
First person introspection, otten related mental and emotional states, soll awareness, and personal beliets
First-persan perspective related idenification, personal information. and objects
Conditional actions or situaions and thair potemtal outcomes. espacially relating to rules, regulations, or personal choioes
41
Logal cases, particularly theft and court procoadings, and discussions of sports toams and players
International trade, diplamacy, and agreements between countries Past events, especially from year ago more
45 Legal and Administra- tive Affains
Division of assots pssoos/proporty..oooo medical procodures logal disputes/court procoadings, and organizational activi-
tieslevents
Ownership property membership
Commorcial transactions, logal procoodings, and financial obligations
48 NIA
Health benefits of avocado, color testing tools, gene normalization hPDL fibroblasts, toriyaki chicken wings, reptile hemoparasite identiication, hemormold treatments, omega-3 fatty adid supplements tertiity, baking trout... Activttis related to hobbles including airplane livery design, scrapbooking SMART toam coordination, childhood sensory play and development, painting vegetables, taking consumer surveys, decorating with lampshades, Infants, food, and cratting
early
Figure 15: Clustering of identified PRISM feature descriptions in GPT-2 XL. Shown are the l 50 meta-clusters of feature descriptions. each labeled with a corresponding metalabel generated by Gemini 1.5 Pro, along with up to randomly selected sample descriptions per cluster.
27

---

Metalabel
Feature Descriptons
Miscellaneous
Japancse teriyaki chicken, type sodaa, pleces featuring wood, sugary foodidrikk, gameplay mechanics, loggings, plumbiin services, names fumiitur images
Hb vaccine, investar-owned utliiies, couples counselling, week economyinmune system, HARPO lellowship, outage Energy rolated industry resource
Sooking and Manag.... in Resources
Data storage, translor, or management, alten relation to websites, softward or anline platforms Ordering, requesting, or discussing types of services, accounts, or information, often related online plattorms, finances, or businesses
Requesting, searching for, looking for something, espocially services like logal or insurance, or iterss quetes or properties
Linguistic Elemnets and Structures
The conjunction "So" starting a sentence, often introducing a conclusion or consequence based on the procoding context Conjunctions, prepositons, and occasionally other function words, appearing in descriptions af food and drink proparation, achievement announcements, or product descriptions
Commercial onterprises, locations residence, Iostual works, family members, sports, and proper nouns
N/A
Occupations or rolos related to water
Locations (aties, states, neighbarhaeds) and things found homes related to home maintenanceeiipeeveenntt Products or services related to attire, beauty, or personal care
12 Products and Loca- fions
Clothing. accessories, casmetic products with deseriptiors their matorials, featuros, benofits
Items or products and their descriptions including specifications. materials, uses
Medical and/or chomical torms in the context of product doscriptions or technical documontation
17 Competitive Anelysi
Business
Evaluzzion, scadenicleducatioo institutions, and cettiiiationquualicctiin
Publication detais, often indlding author, tile, date, and publisher/jormaal
19 NAA
IndyCar sorios or races, otan with the word "Indy" highiihhtd
Questions about processes, mostly using the auxiliary "does"
AA person named Fei appearing conversations context
20 Spetiotemporal
Locatione, proper nouns, and numbers related to places, events, or entities Timestamps, specifically times day short personal story olten including memtion of a family member, sometimes in relation to a specilic past time recent event
Descriptions Aneodstes
and
22 Pattern Matching
The lettor G. capitalized or not, related to proper nouns in a list like structure
subetring "ler', Names, punctuation marks, and specifically the tokans "sh", 'I,, "", ""' and a newline character
proper noun, especially person's
26 Product/ Service De
New service/product ofterings markattingirommatoon of existing servioss/rroducts Belonging, origins, sources, or compononts
scriptions
Product features or qualitie
27 Ordinal Numbers
Ordinal rumbers contextual information descrbing locations, groups, or orderad lists Ordinal rumbers, ofton within the context of liss or ordered itoms
Ordinal rumbers in numbered list
28 WordPress and Digi- 1al Business Skills
Content related the Wordpress platform, possitly locusing on usage, features, and user groups
Financialbuusiness topics, digital skill training. or soltware platforms and their feetures (benelllss Computen1T skill, software, or computer pooguler programs, often in a wwwaeespooesssoonaaaaeennnn
context
34 Digital Business and Technology
Products and services related to pet care, home improvement, electronics, and computing Softare, tools, and resources for creating and managing websites and other digital content
Webeilelsoftware development, marketing, and financial services/producte
35 Publication and Distri- bution
Relating to an edition or version of book or releting to a card game
Givesway, donation request, advertisement, sharing information, related link or media Relating the beginning section piece writing
38 NIA
Medical studies of the effects of various factors or substances on different types of cancer
Energy sources, including renewable energy like tidal power an alternative fossil fuels, geolagical formations processes like sediments, and motor neuron degeneration
International Business Machines (a technology company) and eystic fibrosis
42
Locations oftering services or events
Competition, playolf, or toumament sporting events
Discussions of hobbies, product reviews, and summaries events
hardware soltware, website creation and management. recipes, personal anecdates and
44 Product and Service Specilicatins
Video lie sharing, software, or online services related to medis, including video resclution, file formats, platforms, and user experience
Attributss of products related to materials, sizes/dimensions, andior color
Screen resolution or magnification
46 Listerta 1ion
Contamina-
Food recalls due bacterial contamination, specrically Listeria
Food contamination with Listeria
47 Taxes and Logal Obli gations
Owneeshhp of creative digital content, specifically relating to Italian architecture or fashion accessories and their online availability
obligations non-reeident sellers of real estate, especially focusing on the buyer's resporaiilty withhold parcentage of the sale price for lax puracses Instructions related to food preparation, especially baking or chilling in a refrigerator, sometimes followed by serving instructions
49 Instructionaal Explans tory
Demonstrative pronouns (this, that) the beginning sentences, especially related to new information summaries Competitive situaiions often sports or games, with amphasis on pusitions and actions taken by a team or individual player
Second person pronouns in instructional or usar manual style texts, frequently appearing contexts invalving explamations processes, lools, options available to the reader
Figure 16: Clustering of identified PRISM feature descriptions in GPT-2 Small SAE. Shown are the k = 50 meta-clusters of feature descriptions, each labeled with a corresponding metalabel generated by Gemini 1.5 Pro, along with up 3 randomly selected sample descriptions per cluster.
28

---

Large Language Models (LLMs) have rapidly become integral to range of real-world applications, from software development to medical diagnostics. Despite their growing influence, the internal decision- makiing processes of these models remain largely opaque. We aim to understand these black-box systems by analyzing their internal structure.
this survey, we're looking at activation scores, continuous values that can be both negative or positive, indicating how much neuron (single unit of neural network) in LLM "reacts" some input text at inference time. This has been calculated for each token (single unit of text). activation is positive (above zero), we consider that highlight. Neuron activations can only depend on words before the word it activates on, so the description cannot depend words that come after.
For each of eight groups, you will see five clusters containing some number of text samples. The text samples all start with ">" and have a header indicated by "=== Text # #1234 ===", where "# "#1234" represents the ID of the text sample.
Each cluster starts with a summary of the highlights (tokens with positive activations) alongside their activation scores. This summary is followed by all highlights the context of the samples.
Your tasks:
(1) Describe what the common pattern is within the following texts. From the provided list of text excerpts, identify the concepts that trigger the activation of a particular feature. If recurring pattern theme emerges where these concepts appear consistently, describe this pattern. Focus especially on the spans and tokens in each example that are inside a set of [delimiters] and consider the contexts they are in.
(2) At the end a group, rate the perceived similarity on an I-point scale (0 to 10). (3) At the end of the entire questionnaire, mention the time you took in total.
Remember these caveats for assigning descriptions:
* Do not just list the highlighted words!
Do not write an entire sentence!
Do not finish the description with a full stop (".")!
* Do include phrases like "highlighted spans', 'Concepts of", 'Concepts related to', and instead only state the actual semantics!
Example:
CLUSTER #0: Duration of servicedconruactlgrreennn, or time period of bun/studyyivvestmentt,
especially in relation to business, finance. legal or technical contexts
CLUSTER #1: Exclusivity, short durations, or small quantities, sometimes referring to locations with "Spa" "p in their name CLUSTER Exclusivity, limitations, or allowances
CLUSTER #3: Intelligence gathering, growing plants, placing objects, time periods, marijuana use, higher goals, something only happening once, running and the greeting "hi". being originally from somewhere
CLUSTER #4: Exclusivity, allowance, placement, increased quantity/quality, and artistic activities
Semantic similarity score: 6
Figure 17: Human user study instructions for annotating text clusters and rating the polysemanticity of their annotations.
29

---

NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: At the end of our introduction (Section I we included list of our contributions with references to sections where we discuss the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations of our work are discussed in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
30

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We define the details of our experimental studies in Section 4.1 and Section A.6
in addition we reference more implementation details in the Appendix A.4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
31

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer:[Yes)
We provided our code and use publicly available models, datasets and methods and reference them, see htttps://githuu.cco///keee/////e
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips cc/pu
ssiooooiicii/oiioiooocccc for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //npps.cc/publice wewiii//giiie//eeee htte://uddeee/eeebbbaieeeeeeee oolielo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We define the settings and details of our experimental studies in Section 4.1 Section and Appendix(A.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
32

---

The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our line plots include error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We give details on the compute resources used for our experiments Ap- pendix A.4
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
33

---

Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We mention biases the models we use Section
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method provides more transparency of XAI methods and guides safer use.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We mention the bias and limitations of generative models.
Guidelines:
The answer NA means that the paper poses such risks.
34

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
All models, datasets, and methods used are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package wwwerrswihhhheeeeeeeo
should be provided. For popular datasets, paperswi thcode. com/ aataattsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification; We provided our code and use publicly available models, datasets and methods and reference them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
35

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The human evaluation described in Section 5.2 is detailed in Appendix A.88 including information about participants, study setup. and the full instructions provided. Compensation details are also provided, where applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The human evaluation task described in Section 5.2 was conducted with seven participants and does not constitute human-subjects research requiring IRB review. No risks beyond standard usage survey tasks were incurred by participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: An LLM a core component of our method, used to generate cluster-level feature descriptions and concept samples. These aspects are described in detail in Section 3.]
and Appendix A.3
Guidelines:
36

---

The answer NA means that the core method development in this research does not involve LLMs as any important. original, or non-standard components.
Please refer to our LLM policy otteeses/merrrrreccc///ooerrrree for what should or should not be described.
37


Reviews:
[{"review_id": "KBwPHYTRpf", "reviewer": "Reviewer_axts", "summary": "The paper proposes a novel method PRISM, to generate automated explanations for arbitrary features or neurons in a LLM that respects polysemanticity. The method functions by performing percentile sampling for activation of any given feature/neuron to select top activating samples but with more variety than doing direct top-k sampling. It then clusters these samples based on their semantics and for each cluster generates a summary using a language model using the most activating members of the cluster. The method is tested on various language models and also demonstrated on raw neurons and directional activations.", "questions": "1. Could you address weakness 1.\n\n2. Weakness 2. \n\n3. Weakness 3 \n\n4. Is the method robust for the choice of $e$ (line 100), and the sentence embedding model (line 108)? \n\n5. In practice, did you observe biases in your descriptions because of the choice of LLM (for summary generation) or sentence embedding function?\n\n6. Looking at the qualitative examples, my understanding of what the paper considers as distinction between monosemantic and polysemantic concepts is less clear. If you can assign meaningful metalabels for features, are the underlying features really polysemantic? The metalabel acts in this case as the semantic captured by the feature which should thus be considered monosemantic. For eg. If a feature activates for names of animals then having two different clusters, one for dog and one for cat, should not mean its polysemantic. It's good you have polysemanticity score to quantify this but I am getting confused by the qualitative examples. Most of them I wouldn't consider polysemantic. The only exception is the Fig. 5 top where also I only see two different semantics (Occasions/festivities and encryption/rules etc.). Could you make your stand clear in general and specify it in paper for all the qualitative examples? Also maybe visualize more polysemantic concepts in the main paper as its the core point.\n\nI like the paper but there are some glaring gaps that need to be filled.", "limitations": "The authors have stated reasonable limitations although I believe there are some more limitations (mentioned in the review)", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The writing and paper presentation is very clear in most places.\n- I find the problem statement of generating automated explanations to capture polysemanticity interesting and relevant.\n- The proposed solution is simple and novel.\n- Experimental evaluation is extensive in terms of multiple models and different types of experiments to analyze the approach.\n\n\nWeaknesses:\n- W1: The method does not take into account the activation of the individual points relative to the highest activation. For instance, if the 99th percentile input has activation an order of magnitude lower than 100th percentile, it should not be considered for clustering.\n- W2: While the AUC and MAD metrics quantify if descriptions correspond to input samples where a feature/neuron activates, I didn't find any evaluation regarding \"how\" much of the feature is \"explained\" through the obtained descriptions. In other words, are the descriptions of $k=5$ clusters enough to \"cover\" all samples that can activate the given feature? This evaluation should strengthen the paper and I would expect favourable results for this method compared to baselines since it tries to capture polysemanticity. Maybe it also provides a handle to better choose hyper parameters for generating descriptions for a feature.\n- W3: Sec 5.2 experiment is weak. It should be repeated with multiple annotators not just one. Even though it might have positive indications, it's not sound at the moment. \n\n(Minor corrections)\n- eq. 3 -> $C_k$ should be $C_j$\n- $s_j$ is used for both natural language summary in eq. 3 as well as the embedding notation in eq. 4.\n- You should explain the high variance in MAD scores better and directly in the main paper (Appendix Fig. 10 I suppose is for that)", "was_revised": true, "final_justification": "The authors responded with clarity and transparency to my questions. Most of my concerns were addressed completely. I see two limitations that were came up in discussion with me or other reviewers:\n\n(1) In a small subset of cases the authors do observe neurons where their sampling strategy could benefit from updates by taking into account relative activations for a given neuron, but considering all the arguments, I would still agree the current proposed method is useful and has been shown to be effective.\n\n(2) The relatively limited size of subjective evaluation as noted by Reviewer 9BmE.\n\nOverall, in view of the interesting problem, novel proposal, and clear and comprehensive experiments, the positives outweight the limitations for me. I lean towards acceptance of this paper.", "rebuttal": "We would like to thank the reviewer for the detailed feedback and helpful suggestions. We have incorporated the suggested minor corrections into the revised manuscript and address the comments below.\n\n**[W1/Q1] Percentile Sampling:**\n\n> \"The method does not take into account the activation of the individual points relative to the highest activation. For instance, if the 99th percentile input has activation an order of magnitude lower than 100th percentile, it should not be considered for clustering.\"\n\nAbsolute activation magnitudes vary substantially across neurons, layers, and even datasets. Relying on a fixed threshold (e.g., selecting only the top-k sentences) risks excluding informative patterns with moderate but consistent activations. To address this, our method samples sentences uniformly from the 99th to 100th percentile with a step size of 1e-5, allowing us to focus exclusively on the upper tail of the activation distribution while preserving semantic diversity.\n\nThis percentile-based sampling is a core distinction between PRISM and other feature description methods that are typically based on top-k approaches.\nWe demonstrate (e.g., Table 2 for quantitative results and Figure 4 for qualitative results) that sampling from the 99th percentile already yields sentences that both robustly activate the feature and reflect semantic diversity, supporting a more holistic understanding of neuron behavior.\n\n**[W2/Q2] Setting of Cluster Size:**\n\n> \"While the AUC and MAD metrics quantify if descriptions correspond to input samples where a feature/neuron activates, I didn't find any evaluation regarding \"how\" much of the feature is \"explained\" through the obtained descriptions. In other words, are the descriptions of clusters enough to \"cover\" all samples that can activate the given feature?\"\n\nWe would like to clarify that the AUC and MAD metrics are designed to quantify how strongly a neuron/feature reacts to inputs corresponding to a given description, in comparison to a random input set. As described in Appendix A.3, we evaluate a proposed description (e.g., “Quantities, specifically numbers, and time periods”) by prompting an LLM to generate related sentences (see Figure 7 in Appendix for exact prompt details). We then compare the activations of the neuron/feature on these description-related sentences against its activations on random inputs. The separation between the two distributions is measured using a nonparametric metric (AUC, where 1.0 is ideal) and a parametric one (MAD, where higher is better). A high score indicates that the neuron/feature responds significantly more to inputs aligned with the description, which suggests a faithful explanation.\n\n>  \"In other words, are the descriptions of clusters enough to \"cover\" all samples that can activate the given feature?\"\n\nThis is indeed a limitation of all current feature description methods, as these depend on a reference dataset, limiting the descriptions to cover concepts that occur in a subset of these samples. \n\nOur approach progresses on the challenge of finding richer, multi-faceted descriptions of each feature, describing it with a set of several distinct concepts (in comparison to a single description from standard approaches). However, like all such methods, PRISM cannot guarantee that the provided descriptions are the only or most salient concepts for a feature; rather, they represent key concepts the feature reliably responds to. We have now extended our brief discussion of this limitation in Section 6 of the revised manuscript.\n\nTo further examine this, we analyzed PRISM's performance across different numbers of clusters (k) for generating feature descriptions for GPT-2 XL. As shown in the table below, increasing k improves the best-case description quality, as reflected in higher PRISM max scores, but tends to reduce the average interpretability across all clusters (lower PRISM mean scores). This highlights a tradeoff: more clusters can yield more precise and targeted descriptions, but may also introduce redundancy or semantic overlap. The results agree with our intuition that with larger numbers of clusters, the best label becomes more precise (PRISM max score) while the overall score (PRISM mean score) decreases since now monosemantic labels are over-labeled. We have added this experiment and discussion to the Appendix of the revised manuscript.\n\n\n| k Clusters | PRISM (max) |        | PRISM (mean) |         |\n|------------|-------------|--------|--------------|---------|\n|            | AUC (↑)     | MAD (↑)| AUC (↑)      | MAD (↑) |\n| 5 (original) | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| 1 | 0.75 ± 0.31 | 1.79 ± 2.35 | 0.75 ± 0.31 | 1.79 ± 2.37 |\n| 3 | 0.82 ± 0.28 | 2.75 ± 2.69 | 0.69 ± 0.35 | 1.52 ± 2.45 |\n| 10 | 0.88 ± 0.19 | 3.05 ± 2.62 | 0.61 ± 0.37 | 0.99 ± 2.24 |\n\n**[W3/Q3] Human Evaluation:** \n\n> \"[Experiment in Section 5.2] should be repeated with multiple annotators\"\n\nWe expanded the human evaluation in Section 5.2 by involving seven additional non-author participants, results show good alignment between human judgments of polysemanticity and the PRISM metric; due to character constraints further details can be found in our **response to Reviewer 2jDP** under **[W5] Human evaluation**.\n\n**[Q4] Choice of the sentence embedding model:**\n\nWe based the selection of the embedding model on the 'Massive Text Embedding Benchmark' (MTEB) that aims to evaluate sentence embedding models across a variety of tasks. Specifically, we chose a model from the Qwen2 family due to its strong and consistent performance on the MTEB leaderboard (see [MTEB23], [TEMB] for reviews, and [MTEB] for the current rankings).\n\nIn initial experiments, we compared several top-performing embedding models and observed that the resulting concept descriptions showed strong overlap, and thus we chose not to pursue an extensive ablation in this direction. Overall, recent state-of-the-art encoder models produce embeddings with comparable performance [MTEB], reflecting a broader trend toward convergence in model representations that is hypothesized to be a results of strong similarities in training data, objectives, and model architectures [Plat24].\n\n\n**[Q5] Choice of LLM for description:**\n\n> \"In practice, did you observe biases in your descriptions because of the choice of LLM (for summary generation) or sentence embedding function?\"\n\nWe agree that model choice can influence descriptions and is an important area for future research. In our early experiments, we observed that smaller LLMs often produced overly long or unfocused summaries, consistent with prior findings on their instruction-following limitations [Lou24, Zhe23]. For this reason, we use a strong and sufficiently large instruction-tuned model for summary generation. Due to character constraints, we kindly refer the reviewer to our **response to Reviewer MiB7** under **[W1/Q2] Open Source LLMs**, where we provide a detailed discussion and experimental results on the impact of using different LLMs for description generation.\n\nRegarding sentence embeddings, as discussed in [Q4], we found minimal variation in output semantics across high-performing embedder models. However, we acknowledge that domain-specific models (e.g., multilingual or biomedical) may be needed to capture sentence meaning in more specialized settings [Vas24]. \n\n\n**[Q6] Distinction between mono- and polysemantic concepts:**\n\nWe thank the reviewer for pointing out this potential for confusion. We follow the most widely used definition of polysemantic features as “neurons that respond to multiple unrelated inputs” [Fal15, Mul16, Aro18].\n\nIn our manuscript, we thus consider a feature polysemantic if it activates for semantically distinct clusters (e.g., the mentioned encryption terms and holiday-related phrases), quantifiable  via the polysemanticity score. In contrast, features activating for different animal names are considered as monosemantic, since the underlying concept is coherent.\n\nWe have clarified these points by (i) emphasizing and referencing the definition of polysemanticity used in our paper, (ii) expanding our discussion on the distinction between polysemantic and monosemantic features in Sections 5 and 6, (iii) extending our user study, which offers a complementary perspective on the semantic coherence of the identified feature clusters, and (iv) adding additional examples of clearly polysemantic and monosemantic features to the main paper.\n\n\nIf any further questions or comments arise, we are happy to address them during the discussion period.\n\n\n**Refs**\n- [MTEB23] Muennighoff et al., MTEB: Massive Text Embedding Benchmark. EACL 2023.\n- [MTEB] Leaderboard accessed via huggingface.co/spaces/mteb/leaderboard\n- [TEMB]  H Cao, Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark. Arxiv 2024.\n- [Plat24] Huh et al., Position: The Platonic Representation Hypothesis. ICML 2024. \n- [Lou24] Lou et al., Large Language Model Instruction Following: A Survey of Progresses and Challenges. ACL 2024.\n- [Zhe23] Zheng et al., Judging llm-as-a-judge with mt-bench and chatbot arena. Neurips 2023.\n- [Vas24] Vasileiou et al., Explaining Text Similarity in Transformer Models. NAACL 24.\n- [Fal15]  Falkum  \\& A Vicente., Polysemy: Current perspectives and approaches. Lingua 2015.\n- [Mul16] Nguyen et al., Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks. Visualization workshop ICML 2016.\n- [Aro18] Arora et al., Linear algebraic structure of\nword senses, with applications to polysemy. TACL 2018."}, {"review_id": "366laPesPV", "reviewer": "Reviewer_MiB7", "summary": "The paper presents PRISM: a mechanistic interpretability technique to assign labels to individual language model features. PRISM's primary novelty is in its ability to discern polysemantic features in language models. \n\nFor each feature, PRISM selects input sentences from high-percentiles of activations for the feature being studied. These sentences are then mapped to real feature vectors, using a text embedding model based on Qwen. These feature vectors are then clustered using $k$-means clustering with a fixed number of clusters. An LLM is then given the top few sentences from each cluster to produce human readable feature labels.  The explanations provided by the model are evaluated using CoSy, and ROC-AUC and MAD scores are reported.", "questions": "- What are the failure modes of smaller open source LLMs when used to generate the human readable descriptions?\n- What alternatives may be employed to proprietary LLMS to generate descriptions?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a novel method to elicit polysemantic features from language models\n- Very thorough experimental evaluation of the proposed method is performed\n- The method is explained clearly\n\nWeaknesses:\n- A key weakness of the method is that high capacity (even proprietary) LLMs are required to produce high-quality descriptions of features.  The authors acknowledge this in footnote 7 and the limitations section. It is at first glance surprising to me that the simple task of assigning a human readable label to a group of sentences is beyond the reach of standard open-source LLMs. I would encourage the authors to suggest alternative lower-cost approaches to circumvent this issue.\n- The sentence in line 174 is difficult for me to parse. Which of the two scores is the random baseline, and which is PRISM?", "was_revised": true, "final_justification": "The author's responses were informative, and I believe that this is a strong paper.", "rebuttal": "We sincerely thank the reviewer for their thoughtful, detailed, and constructive feedback. We are especially grateful for the reviewer's appreciation of the paper's clarity, the novelty and simplicity of PRISM, and the extent of our experimental evaluation. The reviewer's comments have helped us both sharpen the presentation and deepen our analysis. We have carefully considered all concerns and made substantial efforts to address each point. Below, we respond to the main weaknesses and questions raised. We make sure the revised version reflects these improvements and communicates the work more effectively. Please let us know if any further concerns remain.\n\n**[W1/Q2] Open Source LLMs:**\n\n> \"[...] high capacity (even proprietary) LLMs are required to produce high-quality descriptions of features.\"\n\n> \"What alternatives may be employed to proprietary LLMS to generate descriptions?\"\n\nWe appreciate the reviewer's concern regarding whether high-capacity or proprietary LLMs are necessary in the PRISM framework to produce high-quality feature descriptions. To address this concern, we extended our evaluation beyond Gemini 1.5 Pro by benchmarking several open-source alternatives: Qwen3 32B, Phi-4, and DeepSeek R1. These models were used to generate feature descriptions for GPT-2 XL using the same procedure as in our original setup. As shown in the table below, Qwen3 32B achieves performance comparable to Gemini 1.5 Pro, demonstrating that our framework does not depend solely on a single model. While Phi-4 and DeepSeek R1 show slightly lower scores, they still follow the same qualitative trends, demonstrating that our method is robust and generalizes effectively across a range of language models, including accessible open-source options. We have included this experiment and discussion to Section 4.2 (Sanity Checks) of the revised manuscript.\n\n| Text Generator (description) | PRISM (max) |        | PRISM (mean) |         |\n|------------------------------|-------------|--------|--------------|---------|\n|                              | AUC (↑)     | MAD (↑)| AUC (↑)      | MAD (↑) |\n| Gemini 1.5 Pro (original) | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| Qwen3 32B | 0.85 ± 0.24 | 2.63 ± 2.31 | 0.65 ± 0.36 | 1.20 ± 2.34 |\n| Phi-4 | 0.82 ± 0.26 | 2.18 ± 2.28 | 0.61 ± 0.38 | 0.83 ± 2.17 |\n| DeepSeek R1 | 0.79 ± 0.29 | 2.23 ± 2.32 | 0.61 ± 0.38 | 0.99 ± 2.33 |\n\n**[W2] Unclear sentence in line 174:**\n\n> \"Which of the two scores is the random baseline, and which is PRISM?\"\n\nWe understand the confusion regarding the meaning of this sentence. To clarify, the AUC of 0.85 ± 0.25 and MAD of 2.98 ± 3.16 refer to the PRISM baseline (with descriptions generated from semantically meaningful clusters), while the AUC of 0.65 ± 0.35 and MAD of 1.35 ± 2.36 correspond to the randomized control (with cluster descriptions shuffled across features).\n\nWe have rephrased the sentence and included the results in tabular form for GPT-2 XL in the revised manuscript to make this distinction clearer. The updated sentence now reads:\n\n> \"The lower score for PRISM in the random scenario (Random Descriptions) compared to PRISM without randomization (Baseline) indicates that the PRISM descriptions truly depend on the content of the clusters because randomizing them leads to a decrease in performance.\"\n\nThe corresponding results are now summarized in the following table:\n\n|            | PRISM (max) |        | PRISM (mean) |         |\n|------------|-------------|--------|--------------|---------|\n|            | AUC (↑)     | MAD (↑)| AUC (↑)      | MAD (↑) |\n| Baseline   | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| Random Descriptions | 0.65 ± 0.35 | 1.35 ± 2.36 | 0.52 ± 0.40 | 0.52 ± 2.19 |\n\n**[Q1] Smaller Open Source LLMs:**\n\n> \"What are the failure modes of smaller open source LLMs when used to generate the human readable descriptions?\"\n\nIn an effort to characterize failure modes, including smaller open-source models, we provide the evaluation score in the table in [W1/Q2]. While the decrease in description quality measured by AUC and MAD supports prior findings in the literature on the limitations of smaller LLMs [Pur25], score decreases are small indicating PRISM's robustness towards the LLM choice.\n\nIn our analysis of the lighter LLMs (e.g. Qwen2.5 7B Instruct, Llama 3.1 8B Instruct), we found that these models are limited due to a combination of issues: they tend to over-generalize, produce overly verbose or vague outputs, struggle with instruction adherence, and in some cases, ignore instructions altogether. These shortcomings reduce the consistency and specificity of the generated descriptions. As noted in [W1/Q2], we have incorporated these findings into the revised manuscript along with an expanded discussion of these limitations and supporting results.\n\n**Refs**\n- [Pur25] Puri et al., FADE: Why Bad Descriptions Happen to Good Features. ACL 2025."}, {"review_id": "ES03gsqCSw", "reviewer": "Reviewer_2jDP", "summary": "This paper proposed a multi-concept feature description framework called PRISM, which applies to the description of both polysemantic and monosemantic features. The proposed framework reveals that Individual features encode a highly diverse and heterogeneous set of concepts. The framework also shows a high alignment with the human interpretation of polysemanticity.", "questions": "See Weaknesses.", "limitations": "NA.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths.**\n\n1. The proposed framework allows the description of polysemantic features, and the method is intuitive.\n2. Alignment with the human interpretation of polysemanticity is tested.\n\n**Weaknesses.**\n\n1. The paper is poorly written and hard to read. Specifically, \n- In Eq (3), $s_j$ represents a concise natural language summary, but it represents sentence embedding in Eq (4).\n- Section 3.2, the exact descriptions of AUC and MAD are repeated and shown twice. \n- Section 3.2. What are the target concept and control data points? They are not introduced in the main paper. How is PRISM evaluated through AUC and MAD? Is the target concept the generated cluster labels? Why do higher AUC and MAD indicate better description? Also, I think Figure 6 in Appendix A.3 should be put in Section 3.2 for better understanding.\n2. In section 3.1, why do we need a grid of high-percentile levels in percentile sampling, instead of using only the top activations?\n3. Line 148. How is the number of clusters determined?\n4. Line 154. Why are only positive activations considered? Is it possible that a sample has a large negative activation?\n5. Section 5.2 includes only one human tester, which may lead to great bias.", "was_revised": true, "final_justification": "The rebuttal solved my concerns. I update my scores to lean towards borderline acceptance. I cannot give a higher score because, although overall effective, this method carries the risk of introducing unfaithful descriptions.", "rebuttal": "We thank the reviewer for the thoughtful comments. We regret the clarity fell short for this reviewer, we appreciate the chance to improve. While other reviewers found the writing and contributions clear, we recognize that presentation can be refined to suit a broader audience, and we have worked carefully to address these concerns.\n\n**[W1] Writing:**\n> \"The paper is poorly written and hard to read. Specifically, [...]\" (a-e)\n\nFollowing the reviewer's comments, we have revised relevant sections to improve clarity and readability. While we are unable to submit a revised manuscript at this stage, we hope our clarifications below help address any potential for confusion.\n\n**[a] Equations 3 and 4:** We thank the reviewer for pointing out that the symbol $s_j$ is overloaded in both equations. We disambiguate this by using distinct symbols $s_j$ for the natural language summary and $\\tau_j = e(s_j), \\tau_j \\in \\mathbb{R}^T$ for the embeddings of a specific natural language summary. We correct Eq. 4. accordingly:\n> $\\text{cos}(\\theta) = \\frac{\\sum_t^T \\tau_{i,t}\\tau_{j,t}}{\\sqrt{\\sum_t^T \\tau_{i,t}^2} \\sqrt{\\sum_t^T \\tau_{j,t}^2}}$.\n\n**[b] Duplicate description:**\nWe removed the unintentional repetition.\n\n**[c] Definition of target concept and control data points:**\nThe target concept dataset consists of sentences generated based on cluster labels for a given neuron. Control data consists of random sentences which are not associated with the concept.\nWe previously defined this in Appendix A.3, but based on the reviewer's helpful remark, we have now moved this explanation to the main text (Section 3.2) to improve clarity and ensure the evaluation setup is more easily understood.\n\n**[d] AUC and MAD as evaluation metric:**\nAUC and MAD scores are used to quantify whether example sentences generated based on each cluster label activate the corresponding neuron more strongly than the random sentences in the control dataset (see answer [W1][c]). Higher AUC and lower MAD indicate more semantically aligned and distinctive concept labels for the different cluster.\n\n**[e] Placement of Figure 6 (Appendix A.3):**  We agree that this figure is informative and helps illustrate the evaluation process. However, due to the length restrictions of 9 pages, we previously struggled to integrate the Figure in the main body. Given the opportunity we will aim to move it into the main paper for better clarity.\n\n**[W2] Percentile Sampling:**\n> \"In section 3.1, why do we need a grid of high-percentile levels in percentile sampling, instead of using only the top activations?\"\n\nWe use a grid of high-percentile levels rather than a fixed top-k cutoff to retain diverse activation contexts that could be excluded by selecting only the top-k samples. Absolute activation can vary strongly across neurons, layers and even corpora, so using a fixed cut‑off (e.g. the top‑k sentences) risks discarding genuine but moderately‑scaled patterns. Instead, sampling sentences uniformly from the 99th to 100th percentile with a step size of 1e-5 retains focus on strong activations while enabling broader coverage of distinct patterns that activate the same feature. As demonstrated in our experiments, this signal allows capturing mutli-concept feature descriptions for neurons that have previously been assigned a single description.\n\nWe empirically evaluated this design choice by comparing PRISM to a baseline approach (MaxAct) that selects the five highest-activating sentences per feature, following [Bil23], using a subset of the C4 dataset. PRISM (max) outperforms MaxAct on most metrics and models, particularly in AUC (see table below). This supports the benefit of our percentile-based sampling for interpretability and concept coverage.\n| Model | Method | AUC (↑) | MAD (↑) |\n|--------|-------|---------|---------|\n| GPT-2 XL | PRISM (max) | 0.85 ± 0.25 | 2.98 ± 3.16 |\n| GPT-2 XL | PRISM (mean) | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| GPT-2 XL | GPT-Explain | 0.64 ± 0.34 | 1.19 ± 2.38 |\n| GPT-2 XL | MaxAct* | 0.63 ± 0.40 | 1.82 ± 3.76 |\n| Gemma Scope | PRISM (max) | 0.54 ± 0.33 | 5.26 ± 18.51 |\n| Gemma Scope | PRISM (mean) | 0.43 ± 0.31 | 2.98 ± 14.68 |\n| Gemma Scope | Output-Centric | 0.58 ± 0.35 | 12.24 ± 40.16 |\n| Gemma Scope | MaxAct* | 0.53 ± 0.31 | 5.35 ± 28.44 |\n\n**[W3] Number of Clusters:**\n> \"How is the number of clusters determined?\"\n\nWe thank the reviewer for raising this important question. In our original implementation, we chose 5 clusters as a default, as this provided a balance between granularity and interpretability, allowing for multiple semantic patterns to emerge while limiting potential redundancy or incoherence. We analyze the effect of varying the number of clusters on PRISM's performance across multiple values of k and report both max and mean cluster scores across features in GPT-2 XL, using AUC and MAD as in our main evaluation (see table below). Increasing the number of clusters raises PRISM max scores by allowing clusters to capture more specific activation patterns, improving labeling of the most coherent ones, but lowers the PRISM mean score as coherent patterns are split into statistically indistinguishable subclusters. We have added clarification regarding the choice of the number of clusters to Section 4.1 and discuss implications of increasing/decreasing k in our discussion in Section 6.\n| k Clusters | PRISM (max) |        | PRISM (mean) |         |\n|------------|-------------|--------|--------------|---------|\n|            | AUC (↑)     | MAD (↑)| AUC (↑)      | MAD (↑) |\n| 5 (original) | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| 1 | 0.75 ± 0.31 | 1.79 ± 2.35 | 0.75 ± 0.31 | 1.79 ± 2.37 |\n| 3 | 0.82 ± 0.28 | 2.75 ± 2.69 | 0.69 ± 0.35 | 1.52 ± 2.45 |\n| 10 | 0.88 ± 0.19 | 3.05 ± 2.62 | 0.61 ± 0.37 | 0.99 ± 2.24 |\n\n**[W4] Positive/negative activations:**\n> \"Why are only positive activations considered? Is it possible that a sample has a large negative activation?\"\n\nWe focus on positive activations as we are interested in characterizing features when they are strongly activated (\"fire\") in response to a specific input.\nWhile negative activations can occur, their interpretation remains unclear and is still debated (see [Mol20]).\nGiven that common activation functions in state-of-the-art LLMs, such as ReLU and GeLU, retain only positive or weakly negative signals, we follow prior work on feature description methods (e.g., [Her22, Bil23]) and threshold activations to retain only the top-activating samples. That said, our framework is compatible with extracting descriptions across the full activation range. \n\n**[W5] Human evaluation:** Inspired by the reviewer's comments, we decided to extend the scope of the human evaluation in Section 5.2 by conducting a larger human evaluation with seven participants, all of whom are non-authors. Participants were recruited from our institutions and compensated. We provide additional details on participant recruitment, task instructions, and the study setup in the Appendix of the revised manuscript for full transparency.\n\nEach participant was presented with 8 groups of sentence clusters, where each group corresponded to one feature and consisted of 5 clusters of highly activating sentences (20 sentences per cluster), along with highlighted tokens. For each cluster, participants were tasked to write a short textual description, resulting in 5 human-generated descriptions per feature. They then rated the pairwise similarity between these descriptions on an 11-point scale (0–10), where 10 indicates very high similarity. We report the average of these ratings as the Human Score. In addition, we compute Human Cosine Similarity using the same embedding model and method used for computing the PRISM polysemanticity score, with the only change being the use of human-generated descriptions instead of model-generated ones.\n\nThe table below presents the results, sorted by PRISM score (lower values indicate higher polysemanticity). As expected, features with low PRISM scores receive lower human similarity ratings and lower embedding-based similarity. For example, the feature from GPT-2 Small (layer 5, feature 30319) has a low PRISM score (0.30), and is judged by participants to be semantically diverse (Human Score: 0.40). In contrast, features with high PRISM scores, such as those from Gemma Scope (features 10531 and 603), show strong human agreement (Human Score: 0.90 and 0.80, respectively) and high cosine similarity (0.79 and 0.75). These findings provide both qualitative and quantitative support for the PRISM metric, showing that it aligns well with the human interpretation polysemanticity. We have replaced the original human evaluation with these new, extended results and discussion in Section 5.2 of the revised manuscript.\n| Model | Layer | Feature | Human Score | Human Cos. Sim. | PRISM Score |\n| --- | --- | --- | --- | --- | --- |\n| GPT-2 Small | 5 | 30319 | 0.40 ± 0.29 | 0.44 ± 0.14 | 0.30 ± 0.17 |\n| GPT-2 XL | 40 | 6067 | 0.40 ± 0.14 | 0.46 ± 0.10 | 0.38 ± 0.07 |\n| Llama 3.1 8B Instruct | 0 | 12733 | 0.50 ± 0.17 | 0.52 ± 0.16 | 0.41 ± 0.08 |\n| GPT-2 XL | 20 | 3313 | 0.70 ± 0.10 | 0.53 ± 0.15 | 0.59 ± 0.16 |\n| GPT-2 Small | 10 | 9896 | 0.80 ± 0.17 | 0.69 ± 0.19 | 0.60 ± 0.09 |\n| Llama 3.1 8B Instruct | 0 | 10095 | 0.70 ± 0.21 | 0.59 ± 0.10 | 0.65 ± 0.06 |\n| Gemma Scope | 0 | 10531 | 0.90 ± 0.16 | 0.79 ± 0.23 | 0.68 ± 0.03 |\n| Gemma Scope | 10 | 603 | 0.80 ± 0.17 | 0.75 ± 0.25 | 0.74 ± 0.08 |\n\nWe hope our responses and changes have effectively addressed the concerns. If any further questions or suggestions arise, we are happy to continue the discussion. \n\n**Refs**\n- [Mol20] C Molnar. Interpretable Machine Learning Book, \nChapter 27. 2020. \n- [Her22] Hernandez et al., Natural language descriptions of deep visual features. ICLR 2022.\n- [Bil23] Bills et al., Language models can explain neurons in language models. OpenAI 2023."}, {"review_id": "1DBaCSw342", "reviewer": "Reviewer_9BmE", "summary": "The paper proposes PRISM, an automatic framework to explain the features recognized by neurons in LLMs. Specifically, PRISM focuses on highlighting polisemanticity of neurons (the extent to which a neuron encodes multiple distinct concepts). \n\nThe framework works by first extracting the top percentile of activations for a given neuron and the samples (sentences) associated with that percentile. Then, the framework encodes these samples using a sentence encoder and clusters the generated embeddings via K-Means. Finally, the identified clusters are fed to an LLM, whose task is to assign a description to each cluster. The set of descriptions represents the \"features\" captured by the given neuron. To measure the degree of polysemanticity, the authors propose a new score, named Polysemanticity Score, and use the Mean Action Difference and AUC scores, which were originally proposed by the CoSy evaluation method for the vision domain.\n\nPRISM is compared against GPT-Explain for explaining GPT-2 X neurons and against the Output-Centric method on three other LLMs using these metrics, achieving, most of the time, higher scores. Using the proposed framework, the paper also analyzes the concept space learned by a couple of LLMs, showing how the proposed framework can support different explanation granularities.", "questions": "See the section Weaknesses and Strengths.\n\nAdditionally, below you can find a question that didn’t have any impact on my evaluation, but it is related to the concern about the wording and I feel that including some additional information related to it into the main text could improve the informativeness of the paper:\n\n- Could authors elaborate more on the sentence in line 28-30 *\"While several concept extraction techniques like sparse autoencoders (SAEs) [17 , 18 ] aim to disentangle polysematic features, many resulting descriptions still reflect multiple concepts [19], limiting interpretability.)?\"* What is the limitation of these techniques?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n- **Significance and Originality**:\n             \n   - The *topic* of automated interpretability for LLMs is gaining interest lately. A contribution in the area could be useful for other researchers.\n    - *Scientific merit* / method: I appreciate the general simplicity of the framework. Given the small time to compute the explanations and the relative easiness of implementation, I believe this framework could easily become one popular baseline for this research niche if the framework is associated with the right setup (e.g., open models, see \"weaknesses, experimental setup\" below).\n   - While the single elements of the framework have been used in the literature (e.g., the combination of clusters and percentile or clusters and LLMs have been used in Vision to capture polysemanticity) their combination for this specific niche is *novel* to the best of my knowledge.\n   - *Flexibility*: The framework supports explanations at different levels of granularity. The authors show that the framework is reliable in producing both high level explanations (meta-level/topics) and more fine-grained ones (feature descriptions). This is an important feature of the framework and one of its strongest points, in my opinion.\n\n- **Clarity**: The paper is overall well structured and easy to read. There are no major concerns here.\n\n\n**Weaknesses**:\n\nAll the below weaknesses are related and impact the *quality* score.\n\n- **Experimental setup**:\n   - Method: all the experiments are performed using Gemini Pro 1.5 to generate feature descriptions and using a specific prompt and a specific number of samples (20) to generate the feature descriptions. There is no ablation study on the configuration. Within this setup, it becomes difficult to understand whether the results depend on the specific configuration and LLM used or reflect some general capabilities, as a framework would require.  Similar to other points of this review, I would suggest either changing the wording and the narrative of the text to be more aligned with the experimental setup (e.g., positioning the paper as a framework that specifically exploits Gemini Pro) or (much better) provide evidence that the results are robust across different LLMs. In my view, the difference is substantial: it distinguishes between a paper proposing a method/framework and one that exploits a specific LLM to achieve a given goal. In this context, I strongly suggest including and testing open models as a backbone: this could help the research community to either address research gaps (if they don’t work properly) or use the proposed framework as a baseline for the future. \n\n   - Metrics: \n      - The AUC and MAD metrics use Gemini 1.5 pro to generate the synthetic data used to measure these scores. This means that the same model is used both in the framework to generate “feature descriptions” given a set of samples, and to generate “samples” given a feature description to evaluate them. This is a technical flaw that could bias the evaluation process and should be addressed. To solve this problem, a fair evaluation should use a family of models not used by any of the competitors in any of the steps. \n     - Competitors and literature on the topic of automated interpretability for NLP and LLMs propose several metrics for measuring the quality of the generated explanations. For example, the Output-Centric method [24] discusses problems connected to methods using the highest activations and proposes some metrics to better evaluate them. In my opinion, testing the framework using a larger set of metrics could help the reader better understand the strengths and weaknesses of the proposed framework\n      - The MAD metric seems to be associated with abnormally high variance in all the cases. Note that this behavior is different from what happens in the vision domain. Given the high variance associated with this metric, I have some reservations about its reliability to assess the quality of explanations in the NLP context.\n  - Competitors: For each model tested, the framework is compared only against 1 competitor. This makes the current evaluation quite weak. The authors did a good job listing some of the alternative methods in section A.2. Including some of them in the comparison could strengthen the contribution of the paper (e.g., a useful baseline could be MaxAct)\n\n- **Overgeneralization or missing context**:\n    - From my understanding, the paper targets a specific niche: automated interpretability for explaining neurons of LLMs in the NLP domain. However, the narrative of the paper sometimes is more general than this niche and leads to overclaims, overgeneralizations, or in general could mislead practitioners. Some examples are reported below:\n       - The abstract does not include any information about the context of LLMs and NLP, and some sentences reported in the abstract, like *\"Unlike prior approaches that assign a single description per feature\"* are definitely not true if applied to contexts beyond the specific niche targeted by the paper (e.g., there is a large body of literature on NLP and Vision that targets polysemanticity of neurons)\n      - Line 31-32 This is an example of overgeneralization. Not all the existing methods provide a single description per feature. For example, several SAE methods, as the authors correctly mention in lines 28-30, try to study polysemanticity. The same holds for [14] when applied to NLP. If authors are referring (as I suspect) to a very specific niche and set of methods, this should be made clear in the text. \n \n      To address this point, I would suggest revising and better positioning the abstract and the paper within the proper context and recognizing that the problem of polysemanticity has been studied for a long time outside this specific niche. Additionally, a discussion about why current and previous methods targeting polysemanticity cannot be applied (or produce inferior results) in this specific context would be appreciated and could improve the overall significance of the paper.\n  \n   - Human Interpretation: Quoting directly from the paper “Lastly, we took a step towards testing the alignment of our PRISM framework with human interpretation. [..] Our results highlight, that the PRISM framework not only provides multiple human interpretable descriptions for neurons but also aligns with the human interpretation of polysemanticity.” This sentence is based on the analysis/annotation process (section 5.2) performed by only one person: one of the co-authors. Therefore, the claim seems more general and there seems to be a mismatch between the claim and the experimental setup of the human interpretation. In this regard, I would suggest significantly reducing the claims about human interpretation or support them with more extensive analysis.\n\n**Criteria under which my evaluation score could increase.**\n\nGiven the analysis above, unfortunately most of my concerns are related to the wording of the paper and it’s difficult to evaluate a change in the narrative without looking at the revised text (and NeurIPS doesn’t allow that). However, as a general guideline for the authors, and *assuming* that the other reviews don’t raise other major concerns that I may have overlooked:\n1) Addressing the concern related to metrics (Adding more metrics and solving the problems related to the current metrics) and addressing the concern related to the specific configuration tested could increase the score up to 3 (borderline rejection).\n2) Solving point 1 and adding more competitors (and thus a more extensive experimental setup) could increase score from 3 to 4 (borderline acceptance).\n3) Addressing both the previous points and better aligning the wording and the narrative of the paper to the experimental setup would result in a score of 5", "was_revised": true, "final_justification": "In my initial review, my concerns were related to the following points:\n- A flaw in the evaluation setup (using the same model for both evaluation and optimization)\n- Lack of ablation studies on some design choices\n- Concerns regarding the metrics used to measure explanation quality\n- Limited performance comparison (only one competitor per setup)\n- Overstatements about the research niche addressed and the contribution\n- Overstatement about the user study, which was initially performed with only one participant.\n\nDuring the rebuttal phase, the authors:\n- Fully addressed the concern regarding the ablation study\n- Fully addressed the flaw in the evaluation setup (by providing results using different combinations)\n- Addressed the concerns about the evaluation metrics\n- Proposed changes to mitigate the overstatements about the niche and contribution\n- (Partial) Expanded the user study to include 7 participants\n- (Partial) Added one more competitor in the performance evaluation.\n\n\nOverall, these improvements justify an increase in my score. I now consider the paper borderline but leaning towards acceptance. The main remaining weaknesses are the limited user study (only 7 participants) and the restricted performance evaluation (only 2 competitors per setup), despite the availability of many relevant methods in the literature. However, given the relevance of the topic, the flexibility and potential of the proposed method as a baseline in the field, and the significant improvements made in response to the review, I lean toward acceptance.", "rebuttal": "We thank the reviewer for the detailed and constructive review and greatly appreciate the time and effort invested.\n\n**[W1] Experimental setup:**\n\n**[a] Method dependence on one specific LLM:**\nWe understand the reviewer's concern about the potential dependency of our results on a specific configuration. To address the use of Gemini 1.5 Pro for description generation, we extended our evaluation on GPT-2 XL to include several open-source language models, namely Qwen3 32B, Phi-4, and DeepSeek R1, using the same setup as in our original experiments.\n\nAs shown in the table below, Qwen3 32B achieves performance comparable to Gemini 1.5 Pro, demonstrating that our framework does not depend solely on a single model. While Phi-4 and DeepSeek R1 show slightly lower scores, they still follow the same qualitative trends, demonstrating that our method is robust and generalizes effectively across diverse language models, including open-source ones. We include this experiment and discussion in Section 4.2 (Sanity Checks).\n\n| Text Generator (description) | PRISM (max) |        | PRISM (mean) |         |\n|------------------------------|-------------|--------|--------------|---------|\n|                              | AUC (↑)     | MAD (↑)| AUC (↑)      | MAD (↑) |\n| Gemini 1.5 Pro (original) | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 |\n| Qwen3 32B | 0.85 ± 0.24 | 2.63 ± 2.31 | 0.65 ± 0.36 | 1.20 ± 2.34 |\n| Phi-4 | 0.82 ± 0.26 | 2.18 ± 2.28 | 0.61 ± 0.38 | 0.83 ± 2.17 |\n| DeepSeek R1 | 0.79 ± 0.29 | 2.23 ± 2.32 | 0.61 ± 0.38 | 0.99 ± 2.33 |\n\n**[b1] Different LLMs for evaluation:** \n> \"[...] a fair evaluation should use a family of models not used by any of the competitors in any of the steps.\"\n\nWe agree that this is a relevant ablation. Thus, we used Qwen3 32B, Phi-4, and DeepSeek R1 to generate samples from the feature descriptions generated for GPT-2 XL. These models were not involved in the original generation step. As before, we computed AUC and MAD scores to assess the quality of the descriptions; results are shown in the table below for both PRISM and GPT-Explain.\n\nWhile overall scores are lower when Gemini 1.5 Pro is not used for evaluation, relative method rankings remain consistent. PRISM continues to outperform GPT-Explain, supporting the robustness of our evaluation setup and the generalizability of our framework. Along with [a], we include this experiment and discussion in Section 4.2 (Sanity Checks) and discuss the selection of different models for different steps (Section 3 and Discussion).\n\n| Text Generator (evaluation) | PRISM (max) |      | PRISM (mean) |         | GPT-Explain |    |\n|-----------------------------|-------------|------|--------------|---------|--------------|------|\n|                             | AUC (↑)     | MAD (↑)| AUC (↑)    | MAD (↑) | AUC (↑)    | MAD (↑)|\n| Gemini 1.5 Pro (original) | 0.85 ± 0.25 | 2.98 ± 3.16 | 0.65 ± 0.36 | 1.24 ± 2.54 | 0.64 ± 0.34 | 1.19 ± 2.38 |\n| Qwen3 32B | 0.58 ± 0.44 | 1.48 ± 3.62 | 0.53 ± 0.44 | 1.06 ± 3.56 | 0.54 ± 0.44 | 1.01 ± 3.52 |\n| Phi-4 | 0.61 ± 0.42 | 1.59 ± 3.32 | 0.54 ± 0.44 | 1.06 ± 3.28 | 0.56 ± 0.43 | 1.00 ± 3.20 |\n| DeepSeek R1 | 0.71 ± 0.35 | 2.02 ± 3.08 | 0.57 ± 0.42 | 1.08 ± 3.08 | 0.60 ± 0.40 | 1.22 ± 3.12 |\n\n**[b2] Different/output-centric Metrics for evaluation:**  \n> \"[...] testing the framework using a larger set of metrics [...]\"\n\nWe agree with the reviewer. To provide a more complete understanding of the strengths and weaknesses of our framework, we have incorporated a supporting evaluation using FADE [Pur25], which offers the following interpretability metrics:\n\n* *Activation-based metrics* based on the Absolute Gini Coefficient (Clarity and Responsiveness) and Absolute Precision  (Purity), which are closely related to our original AUC and MAD scores by quantifying concentration and discriminatory power of activations.\n* *An output-centric metric* (Faithfulness), which measures the causal influence of the discovered concept on model output, see also [24].\n\nResults are shown in the table below. PRISM (max) consistently outperforms GPT-Explain not only in semantic coherence (Clarity, Purity) but also in its impact on outputs (Faithfulness), further validating the method's effectiveness across a variety of interpretability criteria. As FADE's activation-based metrics align with ours, we followed the reviewer's suggestion and included FADE's output-centric faithfulness metric to Table 1, Section 4.3. \n\n|  GPT-2 XL | Clarity (↑) | Responsiveness (↑) | Purity (↑) | Faithfulness (↑) |\n| --- | --- | --- | --- | --- |\n| PRISM (max) |  0.681 ± 0.200 |\t0.277 ± 0.146 \t| 0.629 ± 0.130  \t| 0.154 ± 0.090 |\n| PRISM (mean)|  0.405 ± 0.178 | \t0.151 ± 0.090 \t| 0.416 ± 0.099 \t| 0.134 ± 0.097  |\n| GPT-Explain |  0.513 ± 0.281 |\t0.185 ± 0.175 \t| 0.452 ± 0.177 \t| 0.101 ± 0.072 |\n\n**[b3] Questionable reliability of MAD for NLP:**  \n> \"[...] I have some reservations about its reliability to assess the quality of explanations in the NLP context.\"\n\nIn our paper, the MAD (Mean Activation Difference) metric serves as a *parametric* test as it quantifies the absolute difference in activation magnitudes between description-related examples and random inputs. Conceptually, the Input-based evaluation used by [24] in the context of NLP is closely related to our MAD metric. In [24], the authors test whether mean activations for the description-related examples exceed those of 'neutral' examples. \nAs such parametric measures like MAD can be sensitive to outliers, particularly in NLP settings where activations are often characterized by heavy-tailed distributions. This is precisely why MAD is paired with the more robust AUC score, providing a fuller picture of neuron/feature behavior. We mention this limitation of MAD in Section 6 of the revised manuscript.\n\n**[c] Comparison to MaxAct Baseline:** \nWe thank the reviewer for highlighting this. In response, we strengthened our experiments by comparing PRISM to MaxAct, a baseline that selects the five highest-activating sentences per feature, following [Bil23]. The table (due to character limits, the table appears in our **response to Reviewer 2jDP**, under **[W2] Percentile Sampling**) shows PRISM (max) outperforms MaxAct across most metrics and models, especially in AUC. We include MaxAct as a baseline for all models in Table 1, Section 4.3 of the revised manuscript.\n\nWe agree that broader comparisons are valuable. However, not all methods listed in Appendix A.2 have available feature descriptions, which limits reproducibility. We prioritized baselines with publicly accessible feature descriptions.\n\n**[W2] Overgeneralization or missing context:** \n\n**[a] Address Niche:**\nWe thank the reviewer for pointing out where the narrative of the paper could lead to overgeneralizations. Our work indeed targets a specific niche of methods that assign explanations to individual features, like standard neurons or SAE-discovered features, within LLMs for NLP, and does not cover polysemanticity analyses across domains. We focus on improving feature-description methods for LLMs that typically provide a single textual explanation per neuron/feature, by instead assigning multiple explanations, potentially capturing the diverse behaviors of a single feature. We recognize that:\n- Many existing SAE-based methods explicitly aim to reduce polysemanticity by producing sparse and more monosemantic features.\n- There is a substantial body of prior work (e.g., in both NLP and Vision) analyzing polysemantic neurons.\n\nTo address scope ambiguities in the original manuscript, we have revised the text to clearly articulate our focus throughout. Specifically we rewrite:\n\n- abstract lines 2-5 to:\n> \"Within the context of large language models (LLMs) for natural language processing (NLP), current automated neuron-level feature description methods face two key challenges: limited robustness and the assumption that each neuron encodes a single concept (monosemanticity), despite increasing evidence of polysemanticity.\"\n\n-  abstract lines 9–11 to:\n> \"[...] framework specifically designed to capture the complexity of features in LLMs. Unlike approaches that assign a single description per neuron, common in many automated interpretability methods in NLP, PRISM produces more nuanced descriptions that account for both monosemantic and polysemantic behavior.\"\n\n- lines 31–32 to:\n> \"While the problem of polysemanticity is generally addressed through the extraction of sparse features, such as via sparse autoencoders, feature description methods, which aim to explain the functional purpose of individual features, typically provide a single explanation per feature [20, 21, 22, 23, 24]. This can limit the ability to capture the full range of patterns a feature may represent.\"\n\n**[b] Human Interpretation:** \nWe expanded the human evaluation in Section 5.2 by involving seven additional non-author participants, results show good alignment between human judgments of polysemanticity and the PRISM metric; due to character constraints further details can be found in our **response to Reviewer 2jDP** under **[W5] Human evaluation**.\n\n**[Q1] Question:**\nTechniques like SAEs seek to promote disentanglement but often exhibit high reconstruction errors, model faithfulness and lack theoretical guarantees that the learned features are actually monosemantic (see [Shar25], Sec. 2.1). As a result, individual features may still respond to multiple concepts, which limits interpretability. We now refer the reader to [Shar25] for a more comprehensive discussion of SAEs and their limitations\n\nPlease let us know if any further comments or questions arise. We are happy to address them during the discussion period.\n\n**Refs**\n- [Pur25] Puri et al., FADE: Why Bad Descriptions Happen to Good Features. ACL 2025.\n- [Bil23] Bills et al., Language models can explain neurons in language models. OpenAI 2023.\n- [Shar25] Sharkey et al., Open Problems in Mechanistic Interpretability. Arxiv 2025."}]

Author rebuttal:

Meta-review: The paper introduces PRISM, a framework to generate automated explanations for features or neurons within LLMs, with a specific focus on capturing polysemanticity. The method operates by employing percentile sampling to identify top activating samples for a given feature, followed by clustering these samples based on semantic similarity. A language model is then used to generate summaries for each cluster, effectively providing multiple explanations for a single feature. The framework is evaluated on various LLMs and demonstrated on neuron activations, with experimental results assessed using metrics like AUC, MAD, and a proposed Polysemanticity Score.

Reviewers generally appreciated the paper's clear writing and novel approach. The method's ability to address the interesting and relevant problem of capturing polysemanticity in LLM features was seen as a strength. Concerns were raised regarding the need for high-capacity language models for generating descriptions and the small size of the user study to measure alignment to human interpretation of polysemanticity. The authors addressed these concerns somewhat during the rebuttal period. Overall the reviewers were positive about the contributions of this paper in the field of automated interpretability. The authors are encouraged to revise the paper in light of reviewer feedback.
Decision: Accept (poster)