Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words

Authors: Gouki Minegishi, Hiroki Furuta, Yusuke Iwasawa, Yutaka Matsuo
Primary area: interpretability and explainable AI
Keywords: Sparse Autoencoder, Mechanistic Interpretability, Polysemantic Representation.

Abstract:
Sparse autoencoders (SAEs) have gained a lot of attention as a promising tool to improve the interpretability of large language models (LLMs) by mapping the complex superposition of *polysemantic* neurons into *monosemantic* features and composing a sparse dictionary of words.

However, traditional performance metrics like Mean Squared Error and $\mathrm{L}_{0}$ sparsity ignore the evaluation of the semantic representational power of SAEs - whether they can acquire interpretable monosemantic features while preserving the semantic relationship of words.For instance, it is not obvious whether a learned sparse feature could distinguish different meanings in one word.

In this paper, we propose a suite of evaluations for SAEs to analyze the quality of monosemantic features by focusing on polysemous words.
Our findings reveal that SAEs developed to improve the MSE-$\mathrm{L}_0$ Pareto frontier may confuse interpretability, which does not necessarily enhance the extraction of monosemantic features.
The analysis of SAEs with polysemous words can also figure out the internal mechanism of LLMs; deeper layers and the Attention module contribute to distinguishing polysemy in a word.

Our semantics-focused evaluation offers new insights into the polysemy and the existing SAE objective and contributes to the development of more practical SAEs.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729975660384

thread_type:
official_review

<thread_object>
creator: reviewer_k8gd

<content>
summary:
In the literature, SAEs are often evaluated on MSE and L0. The paper introduces a new metric for evaluation of SAEs that directly measures whether SAEs learn separate features for the different meanings of polysemous words (words which have different meanings based on the context). The metric is constructed from the WiC dataset (filtering for words which tokenize to single tokens). The paper finds that improving the MSE-L0 frontier does not necessarily lead to improvements on this newly introduced metric. The paper also shows that eventually, scaling SAEs no longer improves score on this metric, that alternative activation functions are not always better, and that later layers

strengths:
- Well motivated metric. Correct in identifying problems with MSE/L0 as a metric of SAE quality. Using polysemy as a source of eval is a good idea.
- Confusion matrix approach is reasonable and consideration of metrics like F1 provides more granularity.

weaknesses:
- k=384 is an extremely high k for TopK. Gao et al. (2024) recommends using d_model / 2 for the _maxk_ hyperparameter, not themain k hyperparameter. Ideally, a much smaller k is used for $k$. Using a value of k equal to or larger than d_model doesn't make much sense.
- JumpReLU comparison would be stronger if the STE variant from Rajamanoharan et al. (2024) were used.

questions:
- Normalized L0 (L0 / N) is less informative as a metric than L0 because the amount of information in the latents is roughly proportional to L0 * log N, so across different N's, L0 is roughly comparable, whereas L0 / N is not very comparable.
- How are the SAEs initialized? Prior work (Conerly et al., 2024; Gao et al., 2024) has found that initializing the encoder to the transpose of the decoder is very important for scaling SAEs.
- What are the L0 values attained by the SAEs for each activation function? Are they in a similar range? Are they less than d_model?
- How many dead latents are there in the SAEs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730504648561

thread_type:
official_review

<thread_object>
creator: reviewer_fi7z

<content>
summary:
This paper introduces PS-Eval, a new method for assessing whether SAEs can extract monosemantic features from transformer models by examining their handling of polysemous words. While the paper offers valuable contributions through its evaluation methods, empirical analysis across different architectures, and interesting findings about layer depth and attention mechanisms, it suffers from significant conceptual limitations. My core concern is the paper's unexamined assumption that SAE features should directly map to word meanings, along with questions about whether insights from polysemous words can generalise to broader SAE evaluation. The empirical work is relatively thorough and the findings are interesting, particularly regarding the role of attention in handling polysemy and the logit lens analysis, but the theoretical framework needs strengthening and the scope may be too narrow. I recommend rejection in its current form while strongly encouraging resubmission after addressing these conceptual issues, improving the theoretical framework, and clarifying the presentation.

strengths:
__Novel evaluation methodology__:
* Clear metrics based on confusion matrix approach
* Well-defined dataset construction from WiC
* Systematic evaluation framework that could be extended to other models


__Experimental analysis__:
* Thorough comparison across different activation functions
* Valuable insights about layer depth and transformer components
* Interesting findings about attention's role in handling polysemy
* Useful logit lens analysis demonstrating semantic differentiation


__Technical contributions__:
* Model-independent evaluation metric
* Practical insights for SAE deployment
* Clear demonstration of limitations in current MSE-$L_0$ optimisation

weaknesses:
__Core conceptual concerns__: The paper assumes SAE features should directly map to word meanings. This is a strong assumption that needs more justification. It's possible that effective SAEs might represent meaning through other feature combinations. The focus on polysemous words, while interesting, may be too narrow to support broad conclusions about SAE quality.


__Methodology limitations__:
* The evaluation method could benefit from more discussion of edge cases and failure modes
* Limited exploration of how results generalise beyond the specific test cases
* Need for clearer discussion of how PS-Eval relates to existing SAE objectives


__Presentation issues__:
* Some key terms need clearer definition, particularly in the introduction (e.g. "semantics-focused evaluation", "polysemous words")
* Several figures would benefit from more detailed captions
* Some sections (particularly the transformer component analysis) need more thorough explanation
* Quite a few typos (e.g. "assesse", "spolysemantic", and incomplete sentences e.g. "In our evaluation across different transformer components, specifically the residual layers, MLPs and self-attention.")

questions:
1. How do you justify the assumption that SAE features should directly correspond to word meanings? Could you discuss alternative feature organisations that might be equally valid? 
* For instance, it's easy to imagine an SAE feature that fires on a particular word (e.g. "tweet") regardless of context, and another SAE feature fires on sentences regarding birds and another fires on sentences regarding social media (e.g. "Twitter"), distinguishing the two. It's not clear to me that this is a strong failure mode of an SAE. 
* I don't know if we should expect SAEs to learn features directly corresponding to meaning (the strong linear representation hypothesis) because we don't know what the atomic units (i.e., features) of a model's computation actually are.
2. Could you elaborate on how insights from polysemous word evaluation might generalise to other aspects of SAE quality?
3. The logit lens analysis shows promising results - could you expand on how this validates your evaluation method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705297605

thread_type:
official_review

<thread_object>
creator: reviewer_nb2h

<content>
summary:
This paper proposes a new interpretability metric to substitute the origin reconstruction and sparsity loss in sparse autoencoders. This metric evaluates whether the words of two meanings are the same in features and vice versa. This paper has two findings with this metric: 1. the objective of maximizing reconstruction and sparsity loss does not increase this metric. 2. high layers and attention modules play an essential role in modeling specific semantics of words.

strengths:
1. This paper first examines the ability to identify polysemantics in the interpreter tool of sparse autoencoders. 

2. This paper provides a tool to inspect semantic specificity in the transformers.

weaknesses:
This paper lacks several justifications. Refer to the Questions.

questions:
1. What is the bar for the features to be the same and different? How do you tune this bar?

2. Why do late or large language models perform worse in identifying semantic similarity, although they got significantly higher scores on challenging benchmarks?  The performance of this metric does not align with the performance on benchmarks. Could you please discuss potential reasons for this misalignment between their metric and benchmark performance or provide additional analysis exploring this phenomenon?

3. In your setting, recall and specificity are more valuable than other metrics.

4. Why do we need to calculate this metric on the representations of the intermediate layer of SAE  but not AE or the origin representations? Can you provide a comparison?

5. Which layer of the transformer do you test as default?

6. Does this metric generalize better to other interpretability than the origin reconstruction and sparsity loss?

---Post Rebuttal Review---
I have had a discussion with the authors; I think the authors have clarified several issues and promise to add more details and comparisons in the revised version. Thus, I decided to raise the score from 3 to 5.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730775399183

thread_type:
official_review

<thread_object>
creator: reviewer_s56h

<content>
summary:
This paper introduces a new framework, called PS-Eval, for evaluating Sparse Autoencoders (SAEs), which are widely used to interpret LLMs. The framework repurposes the WiC dataset (Pilehvar & Camacho-Collados, 2019), originally designed as a binary classification task. Specifically, the paper proposes using a subset of WiC to evaluate the extent to which the features learned by SAEs are monosemantic (the ideal case for SAEs). With this dataset, the authors suggest evaluating SAEs using metrics like precision, recall, and specificity. The paper provides a detailed analysis of SAE performance across different layers and transformer components (e.g., residual, MLP, and attention) and compares activation functions such as ReLU, TopK, and JumpReLU.

strengths:
I really enjoyed reading this paper, as it addresses a well-known issue with SAEs that has been overlooked for a long time. It does so by introducing a context-sensitive metric for monosemantic representation, making the paper timely and addressing a current gap in interpretability research. I also appreciated the depth of the experiments, which cover different transformer modules and layers. The analysis is insightful, clearly demonstrating that standard metrics (MSE and sparsity ratio) can be insufficient for capturing monosemantic features. I believe such analyses are very valuable for future SAE design and application. Overall, the paper is well-organized, well-motivated, easy to read, and offers insightful conclusions.

weaknesses:
Even though I enjoyed reading this paper, my main issue is that there is no baseline for comparison. For example, in Table 4, it is hard to conclude which values of accuracy/precision/recall/specificity signify a good SAE. In other words, even with this new framework, it is unclear whether these SAEs can be trusted. Having a good baseline (e.g., a random SAE or fully dense SAE) could help clarify this issue.

questions:
Can the PS-Eval framework be extended to other models or applications outside of NLP, or is it fundamentally tied to the context-sensitive nature of language?

Figure 3: Could you explain why LLM activations are considered less interpretable than SAE features in Figure 3? For me, the fact that LLM activations are squished into the [0, 0.2] interval doesn't make them less interpretable. Instead, it seems like a matter of finding a different cutoff value for LLM activations. Also, how many bins were used for the plot?

---

Minor comments:

In line 316, you say, "Specificity measures how often the different meaning words have the same SAE features." The correct wording should be "different SAE features," as per Table 2.

In line 96: "In our evaluation across different Transformer components, specifically the residual layers, MLPs, and self-attention." I believe a part of the sentence is missing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199031031

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision in Author Response (1/2)

comment:
We appricate detailed reading and suggestive feedback from all the reviewers. We revised the paper based on the reviewers’ comments, and **the major edit was highlighted with coloring (purple).** 

The key changes is summarized below:

- Added a **random baseline** in Figures 17, 18, 19, and 20 in Appendix J (Reviewer s56H).
- Included a **fully dense baseline** in Appendix I (Reviewer s56H, Nb2h).
- Updated **Figure 3 captions** (Reviewers s56H, Nb2h, and  fi7z).
- Moved the **logit lens results** to Section 5 (Reviewer fi7z).
- Expanded **Section 2 (Related Work)** to clarify the scope of our metrics (Reviewer fi7z).
- Added a **Discussion and Limitation** section in Appendix M (Reviewers s56H, fi7z).
- Explicitly defined the **default layer** in Section 3.2 (Reviewer Nb2h).
- Added experiments for smaller **Top-k values** and **STE jump ReLU** to Appendix K (Reviewer k8GD).
- Updated **Figure 3** by changing "normalized $\mathrm{L}_0$" to simply $\mathrm{L}_0$ (Reviewer k8GD).
- Included experiments on the **number of dead latents** and the effect of ghost gradients in Appendix A (Reviewer k8GD).
- Added **$\mathrm{L}_0$ results for each activation function** in Appendix L (Reviewer k8GD).
- Added a figure illustrating the **conponents of the Transformer architecture** (Residual, MLP, Attention) in Appendix H (Reviewer fi7z).
- Revised terminology: Defined **semantic** and **polysemous** in the introduction's footer to clarify their use in the abstract and introduction (Reviewer F17z).
- Corrected typos and minor grammatical issues (Reviewer s56H, fi7z).


## **Clarification on the Validity of Our Assumptions**

Reviewer Fi7z raised the concern that this paper assumes SAE features should directly map to word meanings, suggesting this is a strong assumption that requires further justification.

We justify our assumption on the SAE features from three perspectives:  (1) findings from previous studies, (2) our experimental design, and (3) the results of our logit lens analysis.

**Findings in Prior Works**

It is well-established in prior research that SAE features map to word meanings. 
For instance, in Templeton et al., (2024) [1], features corresponding to specific words such as "Golden Gate Bridge" or "Brain Science" were identified.
Similarly, GPT-4 evaluations in Table 1 of Cunningham et al., (2023) [2] demonstrated that SAE features correspond to word meanings such as "legal terms."
More recently, Li et al., (2024) [3] analyzed the geometric structure of concepts in LLMs under the assumption that SAE features encode **semantic concepts**, ranging from specific word meanings such as "king" and "queen" to more abstract concepts like "code" and "math." 
These findings align with our underlying assumption that SAE features respond specifically to word meanings.

**Our Experimental Design**

As you noted in your "tweet" example, we acknowledge that, depending on the context, SAE features may fire for elements beyond word meanings. To address this, our experiments are specifically designed to embed word meanings into SAE features.our experimental setup explicitly focused on embedding word meanings into SAE features. As described in Section 4.2, SAE features were trained on activations of the target word within the prompt, such as:

`{context}. The {target_word} means `

This design ensures that the meaning of the target word is directly incorporated into the SAE features. By tailoring our framework in this way, we aimed to align SAE features closely with “word meaning”.

**Results of the Logit Lens Analysis**

Building on the experimental design above, our results further validate that SAE features correspond to word meanings. As shown in Table 3, SAE features fire based on the meanings of words. For example, in the case of the token "space" (Table 3, left column), the feature that activates depends on the context, indicating distinct meanings: "space" as "outer space" (green) and "space" as "blank space" (orange).

While we agree that SAE features may also fire for aspects other than word meanings because we don't know what the actual atomic units of computation in a model are, based on these empirical observations and findings, we also believe that the assumption in question is well-justified.

In response to this concern, we have restructured the manuscript to clarify the validation of our assumption. The logit lens results have been moved to Section 5, where they are now explicitly positioned as evidence supporting the assumption that SAE features map to word meanings. This reorganization highlights the role of these results in demonstrating how SAE features align with word meanings, providing a clearer narrative flow.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306875539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussion (1/3)

comment:
Dear Reviewers, AC, and SAC,

We would like to thank you for your great effort in reviewing our paper.
While this is a last minute before closing discussion period, we believe our author responses have addressed all the raised concerns. Again, we would like to highlight some important discussions and our contributions as follows. It would be great if you take these into consideration for your final decision.


**Discussion and Limitation**

Throughout the discussion period, we have included and updated the Discussion and Limitation section in **Appendix M**, which discusses (1) the possibility of extending PS-Eval into multimodal/text-to-image generation settings (from Reviewer **s56H**), (2) clarification of the context from previous SAE studies (from Reviewer **Nb2h**), and (3) potential failure mode or edge case of our proposed evaluation (from Reviewer **fi7z**).

For the point (2), we discussed that it is unclear that more capable LLMs always lead to more interpretable SAE features, and there is no agreeable criterion of how the index of LLM layers to train SAE. For the point (3), we explained our dependence on maximum activated features rather than the distributional evaluation of SAE features, and the existence of learned SAE features corresponding to the splitted token rather than the meaning in the context (shown in Table 3).

In the final version, to improve the clarity further, **we will move the Discussion and Limitation section into the main text, and move the current Section 8 (analysis on the open SAEs) into the Appendix instead.** We believe this can address the last suggestions from Reviewer **fi7z** and **Nb2h** to add more details and comparisons in the revised version. We also hope this would help the practitioners and readers to understand the scope and limitation of our paper appropriately.

Moreover, considering the limitations and failure cases, we revised each section to make our statements be more measured. Based on the observation that there are tradeoffs between L0/MSE and metrics in PS-Eval, we position our PS-Eval as metrics complemented to the existing L0/MSE (rather than the replacement), while expanding the "semantics" axis among multi-objective assessment for the development of more practical SAEs. In the revised paper, we tuned the tone of discussion appropriately (e.g., **Section 4.2**).

We believe that all the metrics in PS-Eval (accuracy, precision, recall, f1, specificity) can contribute to characterizing and analyzing the behavior of learned SAE, and must be high if SAE has optimal features. While our PS-Eval suggests the holistic evaluation with all the metrics, we also note the careful usage of some metrics, such as specificity, when singly optimizing them. For example, if the learned feature is random, the similarity of the SAE feature is always low, and then the specificity results in a meaninglessly high value, which may not reflect the quality of SAE features. In the revised paper, we clarified these and added the recommendation of the metrics in **Section 4.2**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1739188295900

thread_type:
public_comment

<thread_object>
creator: ~gábor_berend1

<content>
title:
Congrats & paper recommendation

comment:
Congratulations to the authors on their work: evaluating SAEs in a way that goes beyond measuring the reconstruction error and the level of sparsity is much needed.  
[This paper](https://aclanthology.org/2020.emnlp-main.683/) (disclaimer: authored by me) could also be of potential interest to the authors, as it complements the authors' findings from the perspective of encoder-based models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a novel framework, PS-Eval, for evaluating Sparse Autoencoders (SAEs) by focusing on their ability to extract monosemantic features from transformer models, particularly in handling polysemous words. This is achieved by repurposing the WiC dataset to assess the features learned by SAEs. The 

Reviewers made a lot of suggestions, especially on addressing these conceptual issues, improving the theoretical framework, and clarifying the presentation. I appreciate the author's effort in the rebuttal and am happy to recommend acceptance of the paper.