Title: Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework
Venue: NeurIPS 2025
Accepted: yes
Keywords: Interpretability

Explainable AI

Automated Interpretability

Feature Description

Mechanistic Interpretability

Concept-based explanations

Clustering
Abstract: Automated interpretability research aims to identify concepts encoded in neural network features to enhance human understanding of model behavior. Within the context of large language models (LLMs) for natural language processing (NLP), current automated neuron-level feature description methods face two key challenges: limited robustness and the assumption that each neuron encodes a single concept (monosemanticity), despite increasing evidence of polysemanticity. This assumption restricts the expressiveness of feature descriptions and limits their ability to capture the full range of behaviors encoded in model internals. To address this, we introduce Polysemantic FeatuRe Identification and Scoring Method (PRISM), a novel framework specifically designed to capture the complexity of features in LLMs. Unlike approaches that assign a single description per neuron, common in many automated interpretability methods in NLP, PRISM produces more nuanced descriptions that account for both monosemantic and polysemantic behavior. We apply PRISM to LLMs and, through extensive benchmarking against existing methods, demonstrate that our approach produces more accurate and faithful feature descriptions, improving both overall description quality (via a description score) and the ability to capture distinct concepts when polysemanticity is present (via a polysemanticity score).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes PRISM, an automatic framework to explain the features recognized by neurons in LLMs. Specifically, PRISM focuses on highlighting polisemanticity of neurons (the extent to which a neuron encodes multiple distinct concepts). \n\nThe framework works by first extracting the top percentile of activations for a given neuron and the samples (sentences) associated with that percentile. Then, the framework encodes these samples using a sentence encoder and clusters the generated embeddings via K-Means. Finally, the identified clusters are fed to an LLM, whose task is to assign a description to each cluster. The set of descriptions represents the \"features\" captured by the given neuron. To measure the degree of polysemanticity, the authors propose a new score, named Polysemanticity Score, and use the Mean Action Difference and AUC scores, which were originally proposed by the CoSy evaluation method for the vision domain.\n\nPRISM is compared against GPT-Explain for explaining GPT-2 X neurons and against the Output-Centric method on three other LLMs using these metrics, achieving, most of the time, higher scores. Using the proposed framework, the paper also analyzes the concept space learned by a couple of LLMs, showing how the proposed framework can support different explanation granularities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- **Significance and Originality**:\n             \n   - The *topic* of automated interpretability for LLMs is gaining interest lately. A contribution in the area could be useful for other researchers.\n    - *Scientific merit* / method: I appreciate the general simplicity of the framework. Given the small time to compute the explanations and the relative easiness of implementation, I believe this framework could easily become one popular baseline for this research niche if the framework is associated with the right setup (e.g., open models, see \"weaknesses, experimental setup\" below).\n   - While the single elements of the framework have been used in the literature (e.g., the combination of clusters and percentile or clusters and LLMs have been used in Vision to capture polysemanticity) their combination for this specific niche is *novel* to the best of my knowledge.\n   - *Flexibility*: The framework supports explanations at different levels of granularity. The authors show that the framework is reliable in producing both high level explanations (meta-level/topics) and more fine-grained ones (feature descriptions). This is an important feature of the framework and one of its strongest points, in my opinion.\n\n- **Clarity**: The paper is overall well structured and easy to read. There are no major concerns here.\n\n\n**Weaknesses**:\n\nAll the below weaknesses are related and impact the *quality* score.\n\n- **Experimental setup**:\n   - Method: all the experiments are performed using Gemini Pro 1.5 to generate feature descriptions and using a specific prompt and a specific number of samples (20) to generate the feature descriptions. There is no ablation study on the configuration. Within this setup, it becomes difficult to understand whether the results depend on the specific configuration and LLM used or reflect some general capabilities, as a framework would require.  Similar to other points of this review, I would suggest either changing the wording and the narrative of the text to be more aligned with the experimental setup (e.g., positioning the paper as a framework that specifically exploits Gemini Pro) or (much better) provide evidence that the results are robust across different LLMs. In my view, the difference is substantial: it distinguishes between a paper proposing a method/framework and one that exploits a specific LLM to achieve a given goal. In this context, I strongly suggest including and testing open models as a backbone: this could help the research community to either address research gaps (if they don’t work properly) or use the proposed framework as a baseline for the future. \n\n   - Metrics: \n      - The AUC and MAD metrics use Gemini 1.5 pro to generate the synthetic data used to measure these scores. This means that the same model is used both in the framework to generate “feature descriptions” given a set of samples, and to generate “samples” given a feature description to evaluate them. This is a technical flaw that could bias the evaluation process and should be addressed. To solve this problem, a fair evaluation should use a family of models not used by any of the competitors in any of the steps. \n     - Competitors and literature on the topic of automated interpretability for NLP and LLMs propose several metrics for measuring the quality of the generated explanations. For example, the Output-Centric method [24] discusses problems connected to methods using the highest activations and proposes some metrics to better evaluate them. In my opinion, testing the framework using a larger set of metrics could help the reader better understand the strengths and weaknesses of the proposed framework\n      - The MAD metric seems to be associated with abnormally high variance in all the cases. Note that this behavior is different from what happens in the vision domain. Given the high variance associated with this metric, I have some reservations about its reliability to assess the quality of explanations in the NLP context.\n  - Competitors: For each model tested, the framework is compared only against 1 competitor. This makes the current evaluation quite weak. The authors did a good job listing some of the alternative methods in section A.2. Including some of them in the comparison could strengthen the contribution of the paper (e.g., a useful baseline could be MaxAct)\n\n- **Overgeneralization or missing context**:\n    - From my understanding, the paper targets a specific niche: automated interpretability for explaining neurons of LLMs in the NLP domain. However, the narrative of the paper sometimes is more general than this niche and leads to overclaims, overgeneralizations, or in general could mislead practitioners. Some examples are reported below:\n       - The abstract does not include any information about the context of LLMs and NLP, and some sentences reported in the abstract, like *\"Unlike prior approaches that assign a single description per feature\"* are definitely not true if applied to contexts beyond the specific niche targeted by the paper (e.g., there is a large body of literature on NLP and Vision that targets polysemanticity of neurons)\n      - Line 31-32 This is an example of overgeneralization. Not all the existing methods provide a single description per feature. For example, several SAE methods, as the authors correctly mention in lines 28-30, try to study polysemanticity. The same holds for [14] when applied to NLP. If authors are referring (as I suspect) to a very specific niche and set of methods, this should be made clear in the text. \n \n      To address this point, I would suggest revising and better positioning the abstract and the paper within the proper context and recognizing that the problem of polysemanticity has been studied for a long time outside this specific niche. Additionally, a discussion about why current and previous methods targeting polysemanticity cannot be applied (or produce inferior results) in this specific context would be appreciated and could improve the overall significance of the paper.\n  \n   - Human Interpretation: Quoting directly from the paper “Lastly, we took a step towards testing the alignment of our PRISM framework with human interpretation. [..] Our results highlight, that the PRISM framework not only provides multiple human interpretable descriptions for neurons but also aligns with the human interpretation of polysemanticity.” This sentence is based on the analysis/annotation process (section 5.2) performed by only one person: one of the co-authors. Therefore, the claim seems more general and there seems to be a mismatch between the claim and the experimental setup of the human interpretation. In this regard, I would suggest significantly reducing the claims about human interpretation or support them with more extensive analysis.\n\n**Criteria under which my evaluation score could increase.**\n\nGiven the analysis above, unfortunately most of my concerns are related to the wording of the paper and it’s difficult to evaluate a change in the narrative without looking at the revised text (and NeurIPS doesn’t allow that). However, as a general guideline for the authors, and *assuming* that the other reviews don’t raise other major concerns that I may have overlooked:\n1) Addressing the concern related to metrics (Adding more metrics and solving the problems related to the current metrics) and addressing the concern related to the specific configuration tested could increase the score up to 3 (borderline rejection).\n2) Solving point 1 and adding more competitors (and thus a more extensive experimental setup) could increase score from 3 to 4 (borderline acceptance).\n3) Addressing both the previous points and better aligning the wording and the narrative of the paper to the experimental setup would result in a score of 5\n\nQUESTIONS:\nSee the section Weaknesses and Strengths.\n\nAdditionally, below you can find a question that didn’t have any impact on my evaluation, but it is related to the concern about the wording and I feel that including some additional information related to it into the main text could improve the informativeness of the paper:\n\n- Could authors elaborate more on the sentence in line 28-30 *\"While several concept extraction techniques like sparse autoencoders (SAEs) [17 , 18 ] aim to disentangle polysematic features, many resulting descriptions still reflect multiple concepts [19], limiting interpretability.)?\"* What is the limitation of these techniques?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper proposed a multi-concept feature description framework called PRISM, which applies to the description of both polysemantic and monosemantic features. The proposed framework reveals that Individual features encode a highly diverse and heterogeneous set of concepts. The framework also shows a high alignment with the human interpretation of polysemanticity.

STRENGTHS AND WEAKNESSES:
**Strengths.**

1. The proposed framework allows the description of polysemantic features, and the method is intuitive.
2. Alignment with the human interpretation of polysemanticity is tested.

**Weaknesses.**

1. The paper is poorly written and hard to read. Specifically, 
- In Eq (3), $s_j$ represents a concise natural language summary, but it represents sentence embedding in Eq (4).
- Section 3.2, the exact descriptions of AUC and MAD are repeated and shown twice. 
- Section 3.2. What are the target concept and control data points? They are not introduced in the main paper. How is PRISM evaluated through AUC and MAD? Is the target concept the generated cluster labels? Why do higher AUC and MAD indicate better description? Also, I think Figure 6 in Appendix A.3 should be put in Section 3.2 for better understanding.
2. In section 3.1, why do we need a grid of high-percentile levels in percentile sampling, instead of using only the top activations?
3. Line 148. How is the number of clusters determined?
4. Line 154. Why are only positive activations considered? Is it possible that a sample has a large negative activation?
5. Section 5.2 includes only one human tester, which may lead to great bias.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
NA.
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents PRISM: a mechanistic interpretability technique to assign labels to individual language model features. PRISM's primary novelty is in its ability to discern polysemantic features in language models. \n\nFor each feature, PRISM selects input sentences from high-percentiles of activations for the feature being studied. These sentences are then mapped to real feature vectors, using a text embedding model based on Qwen. These feature vectors are then clustered using $k$-means clustering with a fixed number of clusters. An LLM is then given the top few sentences from each cluster to produce human readable feature labels.  The explanations provided by the model are evaluated using CoSy, and ROC-AUC and MAD scores are reported.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper proposes a novel method to elicit polysemantic features from language models\n- Very thorough experimental evaluation of the proposed method is performed\n- The method is explained clearly\n\nWeaknesses:\n- A key weakness of the method is that high capacity (even proprietary) LLMs are required to produce high-quality descriptions of features.  The authors acknowledge this in footnote 7 and the limitations section. It is at first glance surprising to me that the simple task of assigning a human readable label to a group of sentences is beyond the reach of standard open-source LLMs. I would encourage the authors to suggest alternative lower-cost approaches to circumvent this issue.\n- The sentence in line 174 is difficult for me to parse. Which of the two scores is the random baseline, and which is PRISM?\n\nQUESTIONS:\n- What are the failure modes of smaller open source LLMs when used to generate the human readable descriptions?\n- What alternatives may be employed to proprietary LLMS to generate descriptions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method PRISM, to generate automated explanations for arbitrary features or neurons in a LLM that respects polysemanticity. The method functions by performing percentile sampling for activation of any given feature/neuron to select top activating samples but with more variety than doing direct top-k sampling. It then clusters these samples based on their semantics and for each cluster generates a summary using a language model using the most activating members of the cluster. The method is tested on various language models and also demonstrated on raw neurons and directional activations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The writing and paper presentation is very clear in most places.\n- I find the problem statement of generating automated explanations to capture polysemanticity interesting and relevant.\n- The proposed solution is simple and novel.\n- Experimental evaluation is extensive in terms of multiple models and different types of experiments to analyze the approach.\n\n\nWeaknesses:\n- W1: The method does not take into account the activation of the individual points relative to the highest activation. For instance, if the 99th percentile input has activation an order of magnitude lower than 100th percentile, it should not be considered for clustering.\n- W2: While the AUC and MAD metrics quantify if descriptions correspond to input samples where a feature/neuron activates, I didn't find any evaluation regarding \"how\" much of the feature is \"explained\" through the obtained descriptions. In other words, are the descriptions of $k=5$ clusters enough to \"cover\" all samples that can activate the given feature? This evaluation should strengthen the paper and I would expect favourable results for this method compared to baselines since it tries to capture polysemanticity. Maybe it also provides a handle to better choose hyper parameters for generating descriptions for a feature.\n- W3: Sec 5.2 experiment is weak. It should be repeated with multiple annotators not just one. Even though it might have positive indications, it's not sound at the moment. \n\n(Minor corrections)\n- eq. 3 -> $C_k$ should be $C_j$\n- $s_j$ is used for both natural language summary in eq. 3 as well as the embedding notation in eq. 4.\n- You should explain the high variance in MAD scores better and directly in the main paper (Appendix Fig. 10 I suppose is for that)\n\nQUESTIONS:\n1. Could you address weakness 1.\n\n2. Weakness 2. \n\n3. Weakness 3 \n\n4. Is the method robust for the choice of $e$ (line 100), and the sentence embedding model (line 108)? \n\n5. In practice, did you observe biases in your descriptions because of the choice of LLM (for summary generation) or sentence embedding function?\n\n6. Looking at the qualitative examples, my understanding of what the paper considers as distinction between monosemantic and polysemantic concepts is less clear. If you can assign meaningful metalabels for features, are the underlying features really polysemantic? The metalabel acts in this case as the semantic captured by the feature which should thus be considered monosemantic. For eg. If a feature activates for names of animals then having two different clusters, one for dog and one for cat, should not mean its polysemantic. It's good you have polysemanticity score to quantify this but I am getting confused by the qualitative examples. Most of them I wouldn't consider polysemantic. The only exception is the Fig. 5 top where also I only see two different semantics (Occasions/festivities and encryption/rules etc.). Could you make your stand clear in general and specify it in paper for all the qualitative examples? Also maybe visualize more polysemantic concepts in the main paper as its the core point.\n\nI like the paper but there are some glaring gaps that need to be filled.\n\nLIMITATIONS:\nThe authors have stated reasonable limitations although I believe there are some more limitations (mentioned in the review)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}