Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words
Authors: Gouki Minegishi
Hiroki Furuta
Yusuke Iwasawa
Yutaka Matsuo
Venue: ICLR 2025 Poster (2025)
Keywords: Sparse Autoencoder
Mechanistic Interpretability
Polysemantic Representation.
Primary area: interpretability and explainable AI
TL;DR: SAEs, while useful for interpretability in LLMs, do not always improve monosemantic feature extraction, highlighting the need for better evaluation methods.

Abstract: Sparse autoencoders (SAEs) have gained a lot of attention as a promising tool to improve the interpretability of large language models (LLMs) by mapping the complex superposition of *polysemantic* neurons into *monosemantic* features and composing a sparse dictionary of words.

However, traditional performance metrics like Mean Squared Error and $\mathrm{L}_{0}$ sparsity ignore the evaluation of the semantic representational power of SAEs - whether they can acquire interpretable monosemantic features while preserving the semantic relationship of words.For instance, it is not obvious whether a learned sparse feature could distinguish different meanings in one word.

In this paper, we propose a suite of evaluations for SAEs to analyze the quality of monosemantic features by focusing on polysemous words.
Our findings reveal that SAEs developed to improve the MSE-$\mathrm{L}_0$ Pareto frontier may confuse interpretability, which does not necessarily enhance the extraction of monosemantic features.
The analysis of SAEs with polysemous words can also figure out the internal mechanism of LLMs; deeper layers and the Attention module contribute to distinguishing polysemy in a word.

Our semantics-focused evaluation offers new insights into the polysemy and the existing SAE objective and contributes to the development of more practical SAEs.

Published as a conference paper at ICLR 2025
RETHINKING EVALUATION OF SPARSE AUTOEN- CODERS THROUGH THE REPRESENTATION OF POLYSE-
MOUS WORDS
Gouki Minegishi* Hiroki Furuta
Yusuke Iwasawa'
Yutaka Matsuo
The University of Tokyo (mi negi shi, furuta, iwasawa, matsuo @webl t..t. t-t..uuo.okkccc
ABSTRACT
Sparse autoencoders (SAEs) have gained lot of attention promising tool to improve the interpretability of large language models (LLMs) by mapping the complex superposition of polysemantic neurons into monosemantic features and composing a sparse dictionary of words. However, traditional performance metrics like Mean Squared Error and L. sparsity ignore the evaluation of the se- mantic representational power of SAES - whether they can acquire interpretable monosemantic features while preserving the semantic relationship of words. For instance, it is not obvious whether a learned sparse feature could distinguish dif- ferent meanings in one word. In this paper, we propose suite of evaluations for SAES to analyze the quality of monosemantic features by focusing on polysemous words. Our findings reveal that SAEs developed to improve the MSE-Lo Pareto frontier may confuse interpretabilityy which does not necessarily enhance the ex- traction of monosemantic features. The analysis of SAEs with polysemous words can also figure out the internal mechanism of LLMs; deeper layers and the At- tention module contribute to distinguishing polysemy in word. Our semantics- focused evaluation offers insights into the polysemy and the existing SAE objective and contributes to the development of more practical SAEs!.
INTRODUCTION
Sparse autoencoders (SAEs) have garnered significant attention in the field of mechanistic inter- pretability research (Sharkey al., 2022; Bricken al., 2023). SAFs aim to address the super- position hypothesis (Elhage et al., 2022), which suggests that activations in large language models (LLMs) can become polysemantic"
that is, a single neuron corresponds multiple functionalities posing a significant challenge for interpretability. By learning to reconstruct LLM activations, SAEs address this challenge by mapping polysemantic neurons an interpretable space of monose- mantic features (Sharkey et al., 2022; Bricken et al., 2023). Recent studies have demonstrated the utility of SAEs in various contexts. Templeton et al. (2024) employed SAEs to observe features related to broad range of safety concerns in Claude 3 Sonnet model, including deception, syco- phancy, bias, and dangerous content. Similar investigations have been conducted on GPT-4 (Gao et al., 2024) and Gemma (Licberum et al., 2024). Moreover, the application of SAEs has extended beyond LLMs to diffusion models (Daujotas, 2024) and reward models (Riggs & Brinkmann, 2024).
SAEs have an architecture consisting of single, wide layer in the autoencoder, and their training typically involves a reconstruction loss coupled with an L1 penalty on the features (Sharkey et al., 2022: Bricken et al., 2023). This approach aims to learn features that are both sparse and accurate, creating known trade-off between Mean Squared Error (MSE) and Lo sparsity. Previous research has focused improving this MSE-L- Pareto frontier, leading proposals such Gated SAE, TopK, and JumpReLU (Rajamanobaran et al., 2024a;b; Gao et al., 2024). Despite improvements in
!Code: https: //t: http://ggthhii.ccoooo/////////// datasets//eou 51 51t////0ww.../ 511//ii_ddtt_aor__Bee II this paper, we distinguish nodssmmatic/nooooemmnti as multiple/single functionalities in the activation of LLMSs (Elhage al., 2022), and polyemmoos/nnnoooseeosss multiple/single meanings in the words.
Dataset: https: //uggi igg ce. co/

---

Published as a conference paper at ICLR 2025
"space": different meanings depending on contexts
SAE training
Evaluation
conteel
connent
Can SAEs extract monosemantic features (f(x))
space
betronnute
||x
A1E A100)
Same meaning
Different moannin
Offferan Activate
Surr Activate
Accuracy
wolked
his
$n
Wene
teeth
space
f(X)
Acthute
Activate
Recall
[
The
The
OOOHOH
Ware
space
0
Precision
moans
0000
0.2
Figure I: Evaluation of SAE's ability to extract monosemantic features from polysemantic activations LLM, The word "space" in imputs tokens has different meanings depending on the context: context1 refers to "space" as in the universe (c.g.. ..ssrronttss walked in space..), while context2 refers to "space" as in gap or physical distance (e.g.- space between his teeth..... SAE is trained by minimizing reconstruction loss and applying sparsity constraint. The evaluation examines whether the features produced by the SAE can differentiate between the meanings of "space" in different contexts.
the MSE-Lo Pareto frontier, it serves only substitute for the original goal of finding monoseman- tic features, and we do not evaluate it in practice. Such a gap highlights concerns about evaluation methodologies in SAEs, emphasizing the need for approaches that better align with the objective of discovering meaningful and interpretable features. Motivated by this, prior studies have utilized the IOI task (Wang et al., 2022) with known internal circuits in GPT-2 small (Cunningham et al., 2023; Makelov et al., 2024), RAVEL benchmark with counterfactual labels (Chaudhary & Geiger, 2024), and targeted probe perturbation (Karvonen et al., 2024b), as well as automated assessments using GPT-4 (Cunningham et al., 2023; Karvonen al., 2024a). However, these metrics limited to specific models or functional tasks and do not directly assess whether SAE features genuinely represent monosemantic characteristics, which was the original motivation for introducing SAEs.
To address these issues, we evaluate whether SAE features can effectively learn to decompose pol- ysemantic LLM activations into monosemantic features by proposing metric and dataset, Poly- Semantic Evaluations (PS-Eval). In natural language, polysemantic activations may naturally result from the polysemous words (Li et al., 2024). PS-Eval measures whether SAEs can correctly extract monosemantic features from polysemantic activations (Figure 1), based on the Word-in-Come 2019), which contains polysemous words whose
(WiC) dataset (Pilehvar & meanings vary based on context. PS-Eval can assess whether different features in SAE are acti- vated when tokens with different meanings are input, and whether the same features in SAE are activated for tokens with the same meaning. Specifically, we construct confusion matrix (Table 2) and evaluate the SAE using accuracy, precision, recall, and specificity metrics. To clarify whether our polysemous evaluation helps analyze polysemantic features in SABs, we employed the logit lens (nostalgebraist, 2020) to examine the impact of monosemantic SAE features on subsequent lay- of LLMs. Even in polysemous contexts, these SAE features represent distinct meanings, which indicates that SAES extract monosemantic features that meaningfully affect the semantic processing in subsequent layers of L.MMs.
We conducted experiments to determine which types of SAES are more conducive to learning monosemantic features. We observed that increasing the dimensionality of SAE latent variables generally leads to better acquisition of monosemantic features, consistent with common scaling practices in SAE research. However, the accuracy saturates as the latent dimension increases, sug- gesting that there is an optimal latent dimension for practical use of SAE (Templeton et al., 2024; Gao al., 2024). We also evaluated activation functions such as TopK and JumpReLU, which aim to improve the MSE-L Pareto frontier. However, our results show that these functions do not perform well according to our Fl score metrics. Specifically, the performance ranking is ReLU > JumpReL.U TopK. This indicates that solely optimizing for MSE and Lo does not evaluate the ability to extract monosemantic features, highlighting the strengths of our proposed evaluations.
Next, we investigate the specific locations in LLMs where SAEs can learn monosemantic features. We first examined how layer depth affects specificity, defined as whether SAE features activate dif- ferently for words with different meanings. Our analysis found that deeper layers generally have higher specificity, which indicates that the interpretation of polysemantic tokens occurs gradually as the layers deepen. Also, contrary to the traditional MSE metric, which shows that SAE performance
2

---

Published as a conference paper at ICLR 2025
generally decreases with deeper layers, specificity scores from PS-Eval improve, which highlights that SAE performance cannot be fully assessed using MSE alone. We further evaluated different Transformer components, such as residual layers, MLPs, and self-attention; even within the same layer, the performance of SAEs differed. When examining specificity, attention shows higher val- ues than Residual and MLP. This suggests that even within single layer of the Transformer, the Attention mechanism contributes to distinguishing polysemous words.
In summary, our primal contributions are:
1. Introduction of Poly-Semantic Evaluation (PS-Eval): We propose a new evaluation metric and dataset, PS-Eval, designed to assess whether SAFs can effectively extract monosemantic features from polysemantic activations. PS-Eval provides modellindependent evaluation metric for the semantic clarity of SAE's features.
2. Comprehensive Evaluation of SAE: Our findings suggest that SAEs developed to improve the MSE-L- Pareto frontier do not necessarily improve the extraction of monosemantic features. This suggests the need for more holistic evaluation protocols complement to traditional Pareto efficiency.
3. Insights into Layer Depth and Transformer Components: Our findings reveal that deeper layers achieve higher specificity and that the attention mechanism contributes to distinguishing polysemous words.
2 RELATED WORKS
Individual neurons in neural networks are polysemantic, representing mul- tiple concepts, and each concept is distributed across many neurons (Smolensky, 1988; Olah et al., 2020; Elhage et al., 2022; Gurnee et al., 2023). SAE a widely used method that maps these polyse- mantic neurons to monosemantic nigh-dimensional neurons, improving the interpretabilitty of activa- tions in neural networks (Sharkey et al., 2022; Bricken et al., 2023). In practice, such polysemantic SAE features have been shown to correspond diverse semantics, ranging from concrete meanings such as Golden Gate Bridge to more abstract concepts like the entire Hebrew language (Bricken al., 2023; Templeton et al., 2024). Recently, SAES have been applied to large language models such as Claude Sonnet (Templeton et al., 2024), GPT-4 (Gao et al., 2024), and Gemma (Lieberum al., 2024), where they have been used to discover dangerous content and biased features, as well as to control the behavior of LLMs. SAEs are trained by reconstructing the activations of LLMs and an I. sparsity loss of SAE features, which introduces a trade-off between MSE (fidelity) and Lo (spar- sity). In previous research, to improve the MSE-Lo Pareto frontier, the following approaches have been proposed: scaling the number of SAE features (Templeton et al., 2024; Gao et al., 2024), and introducing better activation functions, such alternatives to ReLU (Rajamanoharan et al., 2024a; Gao et al., 2024; Rajamanoharan et al., 2024b).
Sparse Autoencoder
One of the challenges with SAEs is the difficulty of evaluation. While MSE and L1 sparsity imposed during training. is uncertain whether improving this Pareto optimality actually leads to the acquisition of monosemantic features. Existing research has used evaluations like the IOI task in GPT-2 (Makelov et al., 2024; Cunningham et al., 2023), where internal circuits are well understood, automated evaluations using GPT- (Cunningham et al., 2023; Karvonen et al., 2024a). However, there is currently no direct evaluation metric to assess whether SAES achieve their primary goal of mapping polysemantic activations into monosemantic spaces. In this study, we evaluate how well SAES can acquire monosemantic representations at the level of individual word meanings using the Word Context dataset (Pilehvar& & Camacho-Collaan Collados, 2019). which includes polysemous tokens with meanings that vary depending on context.
Context Sensitive Representations In natural language processing, word embeddings have been limited by their ability to represent only static features, where each word is associated with a single embedding regardless of the context in which it appears. To address this limitation, the Word- in-Context (WiC) dataset (Pilehvar& & Camacho- Collados, 2019) was proposed for the evaluation of sentext-sensitive word embeddings. Currently, multilingual version of WiC (Raganato al., 2020) has been developed, and it has also become part of the widely-used SuperGLUE benchmark (Wang al., 2020). Because polysemantic features may naturally come from the polysemous words (Li
3

---

Published as a conference paper at ICLR 2025
et al., 2024), evaluate whether SAEs can learn monosemantic features by using datasets with polysemous words - where the same token has different meanings such as WiC.
PRELIMINARIES
We here summarize the key concepts and notation required understand SAE architectures and their training procedures. The central goal of SAEs during training is to decompose a model's activation x R" into a sparse, linear combination of feature directions, expressed as:
X x XO + Σ f(()),
where d: represents unit-norm latent feature directions (M >> n),, and the sparse coefficients [x(x) 2 0 correspond to feature activations for the input x. This framework aligns with the structure of autoencoder, where the input x is encoded into sparse vector f(x) E RR and decoded to approximate the original input.
3.1 ARCHITECTURE
The typical SAE structured as a single-layer autoencoder, where the encoding and decoding func- tions are defined as follows:
f(x) oW (x baee) + bene),
x(f(x)) == Wxccf(x) + bace-
Here, Wene E 1R Raxn and Wee gaxRa are the encoding and decoding weight matrices, respec- tively, where R is the expand ratio that determines the dimension of the intermediate in SAE. The vectors Dane and Daee represent the biases. We initialize the Waee as the transposed W ene, following the methodology from previous studies (Gao et al., 2024). The sparse encoding f(x) IR Rm serves as the intermediate representation, while x(f) reconstructs the input x from the sparse encoding. The columns of Wde correspond to the learned feature directions d,, and the elements of f fxx rep- resent the activations for these directions. (-) is an activation function; we adopt ReLU, TopK (Gao al., 2024), or JumpReL.U, including its STE variant (Rajamanoharan al., 2024b) in this paper, while using ReLU by default.
3.2 TRAINING METHODOLOGY
Training SAE involves optimizing loss function that balances reconstruction accuracy and sparsity. The reconstruction loss is defined as the squared error between the input x and the reconstruction x(f(x)) and sparsity in the activations f(x) is encouraged using the Lı norm. The overall loss function is combination of these two terms, weighted by sparsity regularization factor X:
LSAE(x) = Crooon + Ap (f()) = ||x xf())) + Allf(x))l1.
(4)
This formulation encourages the autoencoder reconstruct the input faithfully while enforcing sparsity in the latent features. To prevent trivial solutions, such as reducing the sparsity penalty by scaling down the encoder outputs and increasing the decoder weights, it is necessary to constrain the norms of the columns of W Wdee during training. In addition, we incorporate an auxiliary loss known Ghost Grads (Jermyn & Templeton, 2024), which reduces the occurrence of dead latents where SAE features stop activating for long periods. In Appendix A, we discusses the relationship between dead latents and Ghost Grads.
Following prior work (Templeton et al., 2024), we use an expand ratio of R = 32 and sparsity regularization factor of X 0.05 by default for training SAE. The base L.M used as activations for the SAE is GPT-2 small (Radford et al., 2019). Unless specified otherwise, activations are extracted from the 4th layer. Inspired by prior work (Makelov et al., 2024), the training data consisted of the WiC dataset (Raganato et al., 2020) for in-domain data (i.e., data from the same domain as the evaluation dataset) and RedPajama (Weber et al., 2024) for open-domain data (i.e., data from a general domain used during the pre- pre-training of LLMs). The distinction between open-domain and in-domain data is further explained in Appendix B. By default, we use in-domain data for training. Additional training details, such as parer-par,meters, are provided in Appendix C.
4

---

Published as a conference paper at ICLR 2025
Table 1: Statistics and examples of poly-contexts and monc meno-contexte in PS-Eval.
# Total Samples
1112 (label 556, label I: 556)
pc1 y- (space)
Context I: The astronauts walked in outer "space" without tether.
Context 2: The "space" " between his teeth.
Label: 0 (Different)
Example Sentences
monc- contexts (space)
Context I: They stopped at an open "space" " the jungle.
Context 2: The "space" between his teeth. Label: 1 (Same)
3.3 TRADE-OFF BETWEEN RECONSTRUCTION AND SPARSITY
As shown in Equation 4, critical aspect of evaluat- ing SAE during training lies in balancing two primary metries: mean squared error for reconstruction accu- racy and the Lo norm for measuring sparsity. These metries often exhibit a trade-off achieving lower MSE typically results in less sparsity, and increasing sparsity generally raises the reconstruction error. This trade-off can be represented by the Pareto frontier, shown in Figure 2, where the x-axis corresponds to the Lo norm (sparsity) and the y-axis represents MSE (reconstruc- tion error). Ideally, optimal models occupy the lower- left part of the figure, where both MSE and sparsity are minimized simultaneously.
Figure 2: MSE v.s. Lo for expand ratio 32 across varying L1 coefficients (A). The color gradient represents the Lı coefficient values, with brighter colors indicating larger X. As the À increases, the model becomes sparser (lower Lạ) while MSE gets worse.
To improve the Pareto frontier, there are two main strategies: increasing the number of latent variables and enhancing activation function. For increasing la- tent variables, scaling laws have shown that higher di- mensionality improve this Pareto frontier.
4 PS-EVAL
As a holistic evaluation of SAE considering the semantics in the features, we introduce Poly- Semantic Evaluation (PS-Eval), suite of datasets and metrics for evaluation. PS-Eval consists of polysemous words whose meanings are different based on the given context, which aims to eval- uate whether SAE can map entangled polysemantic neurons in LLMs into simple monosemantic features for better interpretability while distinguishing the different meanings.
4.1 DATASET
We build PS-Eval on top of the WiC dataset (Pilehvar & Camacho-Collados, 2019), which is a rich source of polysemous words. The original WiC dataset provides two contexts with a shared target word for each sample, and its task is to determine whether the target word holds the same meaning or a different meaning both contexts. The dataset has binary labels, where a label of 0 indicates that the target word has different meanings in the two contexts, and label of 1 signifies that the target word's meaning remains the same both context. For PS-Eval, we carefully select instances from the WiC dataset whose target word is tokenized as single token in GPT-2 small.
We refer to the two contexts with a label of O (indicating different meanings) as poly-contexts (target word). Table shows an example using the target word "space" in poly-context. Even though the tokenization of "space" remains the same in both contexts, the word carries different meanings: in the first context, it refers to "universe" while in the second, it refers to "an area or gap". On the other hand, when the label is 1, the meaning of the target word remains the same across both contexts. We denote these mono-context, (target word). Table also has
5

---

Published as a conference paper at ICLR 2025
Table 2: (Left) Confusion matrix for polysemy detection in SAE evaluation. Note that Positive or Negative labels are just convenience to describe the confusion matrix, and these not mean that, is bad identity that the different words have different features or it is good to identify that the same words have the same features. (Right) metrics (accuracy, recall, precision, and specificity) and their interprctations.
Mairi
Formubs
Interpntation
Mono-conteNt
Poly-context
Accomay
22-22
well the modi prodic
Same Max
Activated
Same meusinng,
same feature
Diflerent meaning,
feature
Recal
When the same measings wonds and gizze bow oliza thy are siaanited came
Feature
(True Posilive)
(False Positive)
Different Max
meaning,
Different meaning,
Proteom
law olten the micasizs
Activated
different feature
different feature
Frechioo balazeed
Feature
Negative)
(True Negative)
Speciliitty
Waze the diffrret acainy's nends given. 2ow ollon lhey are iécetiled es difirres
an example using the target word "space" in mono-contexis. Here, the word "space" refers to "an area or gap" in both contexts, and thus the label I, signifying that the meaning is consistent.
4.2 EVALUATION METRICS
PS-Eval focuses on analyzing activations of the SAE for the target word across different contexts to determine whether SÄE extract monosemantic features. To identify the activation related to polysemous target words, we consider the following formatted input:
(context) ). The (target word) means
where the (context) } is an example sentence of (target word), reflecting either polysemous contexts (poly-context) or monosemous contexts (mono-context). As done in Wang al. (2022) such a format sentence makes it casy to identify the corresponding activation. In this paper, the acti- vation of the token for the (target word) at layer I is denoted as df (context), and the feature from the SAE is represented as (contexx) Figure I (left) describe the conceptual workflow.
Confusion Matrix for Polysemy Detection To evaluate the ability of SAEs to extract monose-
mantic features, we construct a confusion matrix (Table 2; left) based on the maximum activations of the target word in different contexts. Each cell in the matrix is defined as follows:
mono-context, where the target word has the same meaning in both contexts, we expect the same feature with the highest activation to appear. This corresponds True Positive (TP).
In mono-context, if different features with the highest activation appear, this results in False Negative (FN).
In poly-context, where the target word has different meanings. we expect different features with the highest activation to appear, representing a True Negative (TN).
In poly-context, if the same feature with the highest activation appears for the target word, this results in False Positive (FP).
Note that positive or negative labels here are just for convenience to describe the confusion ma- trix, and these do not features or it is desirable to identify that the same words have the same features. After classifying each instance into confusion matrix, we then calculate the quantitative metrics such accu- racy, precision, recall, Fl score and specificity to assess whether the SAE is capable of acquiring monosemantic features (Table 2; right); Recall measures how often the same meaning words have the same SAE features. Precision measures how often the same SAE features are attributed to the same meaning words. Accuracy and Fl score can quantify the overall performance. Specificity measures how often the different meaning words have the different SAE features.
that, it is undesirable
identify that the different words have different
If SAEs have ideal features, all of these metries can be high simultaneously. We advocate for com- prehensive evaluation that considers all metrics holistically, in complement to existing performance measures, such as MSE, Lo. or IOI performances. While measuring individual metrics may also be useful to improve SAEs, we note that we need to carefully treat Specificity when solely interpreting it. Because Specificity measures when the different meaning's 's words are given, how often they are identified as different, it may reach an unfairly high value by definition if SAE features have sig- nificantly different representations each other. For comparison, we investigate the performance of an SAE that activates randomly (see Appendix D) and dense SAE that is trained without I.,

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of Top-7 logits in poly-centext (space), poly-context (save) and poly-context (ball). The synonyms words relevant to the meaning of context and context 2 coloring. We also highlight the splitted token (e.g. hall-oons) in GPT-2 Small. The words that appear in both context and bolded.
poly-comt por-context ext (space)
poly-context (save)
poly-context (ball)
Context
Context
Context
Context
Context
Context
rank
lokens
logits
tokens
logils
tokens
lugits
tokens
lugils
Lokens
lugils
tokens
lugits
1
light
1.224
Layout
0.894
Save
0.867
eLL
0.874
cons
1.170
handler
0.997
2
plane
0.957
occupied
0.882
ulative
0.696
Cache
0.744
isties
1.115
player
0.884
3
shuttle
0.938
spaces
0.853
lue
0.681
TOR
0.731
oon
1.026
4
gravity
0.937
vacated
0.846
Saving
0.675
Marcos
0.665
antine
1.008
wright 0.842
5
craft
0.920
space
0.825
entary
0.672
Sanctuary
0.658
park
0.975
6
Engineers
0.876
shuttle
0.799
WhatsApp
0.655
Save
0.649
0.960
players
0,825
planes
0.869
occupancy
0.798
saving
0.650
orage
0.644
asted
disposal
regularization (see Appendix E). These results show that random SAE achieves the upper bound of Specificity, while densely trained SAE does not exhibit such unnatural metries. Therefore, we basically recommend holistic evaluation with all metrics, and, in case you would like to optimize one of them independently, we recommend the usage of class-balanced metrics such as accuracy or Fl scores as safer options.
5 MONOSEMANTIC FEATURE MAY CAPTURE EACH MEANING IN POLYSEMY
PS-Eval is designed to evaluate SAEs through the analysis of whether the learned monosemantic features encoded from the polysemous words correspond to each word meaning, depending the context. To clarify this core concept, we start with validating our implicit assumption of whether the SAE features encode word meanings by employing the logit lens technique (nostalgebraist, 2020).
We compute the maximum activated SAE features when two poly-context samples (Context 1, Con- text 2) are input. The corresponding activations for each context are calculated as:
mmax Waee max(f(x1))
mmax Wase mxx(f
where 2j and T2 represent the activations of the LLM for Context and Context 2 in poly-context, respectively. Next, we apply the unembedding matrix to these activations to obtain the logits:
logit; = Wo logit, = Wu
Table 3 presents the top-7 logits for the polysemous words "space", "save", and "ball" based on the above calculations. These contexts can be found in Table 6 in Appendix F. For Context 1 of the word "space", the top logits are related to terms associated with outer space (c.g., "flight", "grav- ity"),, whereas for Context 2, the top logits are more aligned with architectural or layout-related terms (e.g., "Layout", "vacated""). This shows that the SAE features correspond to each word mean- ing. Furthermore, it can be observed that when the word meanings in the context are different, SAE features also reflect those distinct meanings. For the word "save," Context I focuses on terms related to saving in the sense of preserving or conserving resources, such as money (e.g.. "Save," "Saving"), while Context extracts meanings more related to saving data or technical operations (c.g.. "Cache"). This distinction is clearly seen Table 6, where Context I involves monetary or resource-saving, and Context 2 relates data saving in a technical sense. Similarly, for the word "ball," " Context refers to a formal event or dance, where "ball" is used the sense of a social gath- ering (c.g., "park"), while Context 2 reflects meanings associated with sports, where "ball" refers to a physical object used in games or sports activities (e.g.. "handler," "player)).
Following positive observations in prior works (Templeton al., 2024; Cunningham et al., 2023; Li et al., 2024), these qualitative results demonstrate that a monosemantic SAE feature can correspond to specific word meaning, which justifies our evaluation protocol.
EVALUATING SAES WITH POLYSEMOUS WORDS
6.1 SAE FEATURES ARE MORE INTERPRETABLE THAN LLM ACTIVATIONS
To investigate the significance of using SAE for PS-Eval, we demonstrate that using SAE features results in better interpretability of polysemous contexts than the activations of LLMs.. When a poly-

---

Published as a conference paper at ICLR 2025
Accuracy vs MSE (1. Coef 0.05)
Accuracy vs 10 dil Doet 0.05)
cidd pelpocmess weets
30
11.00
MSB
1,0
Eipodd Raza:
Expand Barias
Expand
Eapazd Ratas 64
Eapazd Radle 126
Figure 3: (Left) Cosine distance between LLM activations and SAE features in polysemantic contexts. The histogram compares the distribution of Polysemous Distinction (i.e. Cosine Similarity) for LLM activations (blue) and SAE features (green), demonstrating that SAE features extract more distinct representations in pol- ysemous contexts. We adopt It = 128 and L1 coefficient À 0.05 for the hyperparameters of SAEs, (Right) Comparison of Accuracy vs MSE and Lo for different Expand Ratios (L1 Coef = 0.05). The left panel shows the relationship between accuracy and mean squared error (MSE), while the right panel presents accuracy ver- sus normalized Lo sparsity. The markers represent various expand ratios (8, 16, 32, 64, and 128). Higher expand ratios generally show trend towards better performance, as indicated by the yellow and green markers corresponding to expand ratios of 64 and 128.
semous context is input, we denote the LLM's activation for the target word as ( and the SAE feature for the same context as S'AA (poly-context). We calculated the cosine distance between these activations and SAE features to assess their similarity or divergence, which we refer to as Polysemous Distinction:
qlim(poly-context,) (plxppolyyconteext ooop [olnm(poyyconneet, /ttp:///aaaaapllooooooooooo
1- (plelpooyyoonteext) fSAE (poly- I.fsar Jll fSAF (poly
As shown in Figure (right), the cosine distance between the LLM activations for different polyse- mous contexts is small, indicating that the LLM struggles to differentiate between distinct meanings in polysemous contexts (i.e., less interpretable activations). In contrast, SAE features show larger cosine distance, effectively capturing the differences between the contexts with different meanings. This result demonstrates that SAE effectively disentangles different meanings, and, for polysemous words, it enhances the interpretability of internal representations in LLMs.
6.2 INCREASING EXPAND FACTOR IMPROVES POLYSEMOUS SAE FEATURE
Previous works (Templeton et al., 2024; Gao et al., 2024) have reported that increasing the dimen- sionality of SAE features leads better performance in terms of MSE and Lo metrics, and we may also expect that it allows for learning more fine-grained features from activations (Bricken et al., 2023). To investigate the relationship between PS-Eval performance and the scaling of latent vari- ables, we evaluated the performance of our model trained with various expansion ratios (defined in Section 3.1) Figure 3 (right). Similar to MSE and Lo, we observed that increasing the expand ratio led to higher accuracy on PS-Eval. This trend indicates that a higher expand ratio enables the model to better extract the monosemantic fetures from polysemantic activations. However, we found that the accuracy saturates around an expand ratio of 64, suggesting that there is an optimal latent dimension for practical SAE usage. Here, we define Lo as the number of activated SAE features. See Appendix G for other metrics.
6.3 ALTERNATIVE ACTIVATION FUNCTIONS RELU ARE NOT ALWAYS BETTER
Various activation functions for SAE have been proposed to achieve a better Pareto optimality be- tween MSE and Lo sparsity. More recent work has introduced TopK, which replaces ReLU(z) with (Gao et al., 2024), and JumpRel.U, which modifies the threshold of the ReLU ac- tivation to improve performance terms of both MSE and sparsity (Rajamanoharan et al., 2024b) (see Appendix H for the detailed descriptions).
We here evaluate whether these alternative activation functions also yield better results when applied to our evaluation metrics. In Figure 4, we compare models trained with TopK activations, where k is varied over values such as & = (384, 192, 96}, and models trained with JumpReLU, including its STE variant. For JumpReLU, the threshold is adjusted over the range of [0.0001, 0.001]. while for the STE variant, the threshold represents an initial value. We note that TopK typically uses half the dimension of the model as the value of & (e.g., for GPT-2 small, this corresponds 768/2
8

---

Published as a conference paper at ICLR 2025
Re
Jump BeLU
Top
BY
Prediioo
033
E1
02
oravee jungest peat?
Beld
aveust Aupper Aag
ar plsgeraatoo
Mudt
Srate, Jong
Figure 4: Comparison of F1 score, Precision, and Recall across different activation functions. The models were trained with JumpReLU, including its STE variant (with varying thresholds), TopK (with different k), and ReI.U. The results imply that ReLU may be still better than JumpRel.U or TopK with the polysemous words. Specificity MSE
Specificity vs LO
Laper:
Laper.
6.7
Laper:
6.c
Lapen:
Lapon
Layen:
Lager:
0.6
Lapen
Lapen
Lagen
Caper
Specificityy
5.5
Laper. 4
Layer. 30
Specificityy
9.5
Laper
Layer to
644
Layer:
Laper:
Layer: 11
Layer: 12
Lapen:
Lapen
Laper 21
Laper 12
6.2-
6.3-
better
0.1-
better
6.8
0.0
LO
MSE
Figure 5: Comparison of Specificity across different layers of the SAE in relation Lo (left) and MSE (right). Each point represents a layer, from layer 1 to layer 12, with deeper layers (i.c., layer 6 onwards) showing an increasing trend in Specificity. This indicates that deeper layers exhibit more distinct activations for polysemous words, consistent with previous findings that deeper layers in SAEs face greater challenges in training, leading higher MSE and Lo scores.
384), and that JumpReLU often adopts a threshold of 0.001 in the original works (Gao al., 2024; Rajamanoharan et al., 2024b).
Figure 4 compares these activation functions in terms of F1 score, Precision, and Recall. ReLU achieves the highest FI score, with the order being ReLU > JumpReLU > TopK. For TopK, smaller leads to higher Recall, and JumpRel.USSTE further boosts Recall. These results suggest that focusing solely on the trade-off between MSE and Lo may not effectively enhance semantic quality.
LOCATING POLYSEMY CIRCUITS IN LLMS
Polysemy is Captured in Deeper Layers We investigate where large language models (LLMs) interpret polysemous words differently by using Specificity (calculated as Specificity = TN+FF based on the confusion matrix from PS-Eval (see Table 2). Specificity allows us to assess whether the features in the SAE activate differently when presented with Poly-context (contexts involving words with defferent meanings).
TN
Figure shows the Specificity of each SAE layer in relation to Lo and MSE. Previous studies (Gao et al., 2024) have demonstrated that training SAEs becomes increasingly difficult as the depth of the layers increases, which is consistent with our findings, as both MSE and Ly rise in deeper layers. While these two metrics increase, we observe that Specificity improves around the 6th layer, indicating that polysemous words are more easily decomposed into monosemantic features through SAE training. These results demonstrate an inverse relationship between MSE/L. and Specificity, suggesting that focusing solely on improving the Pareto frontier of MSE or Log, as emphasized in prior research, may overlook important insights.
SAE Features Vary by Transformer Component While early prior work (Bricken et al., 2023) primarily centered on the activations within MLP layers, more recent studies have broadened this scope not only to MLP layers but also to residual and attention layers (Kissane et al., 2024; He et al., 2024). We explore which components of single Transformer layer contribute to the acquisition of monosemantic features from polysemous words. Specifically, we divided the activations from a single Transformer layer into three distinct outputs: Residual, MLP, and Attention. See Appendix I

---

Published as a conference paper at ICLR 2025
Score
Specificity
0.7
0.4
MLP
Atteation
0.3
04-
0.1
0.0
Besaiual
Artenkan
Resiad
MLP
Atceition
Figure 6: Comparison F1 and Specificity for SAEs trained the outputs of the Residual, ML.P, and Attention components. The left panel shows the score, where Residual and MI. achieve higher values, while the Attention output performs lower. In contrast, the right panel shows Specificity, where the Attention output exhibits significantly higher Specificity, despite the lower Fl score. See Appendix for details these components.
for details these components. We trained SAES each of these components under consistent conditions, employing an L, regularization coefficient of 0.05, an expand ratio of 32, and focusing on the fourth layer of GPT-2.
From Figure 6, is evident that while the Residual and MI.P components achieve higher Fl scores, the Attention output lags behind. This suggests that, based on the harmonic mean of precision and recall, the features learned by the SAE from the Attention output do not activate examples with the same meaning effectively as those learned from the Residual and MLP outputs. However, when examining Specificity, the Attention output demonstrates a markedly higher value compared the Residual and MLP components. This indicates that the Attention mechanism more effec- tive at distinguishing between different features, specifically in identifying the different words with different features.
DISCUSSION AND LIMITATION
One potential limitation of PS-Eval lies reliance on evaluating the maximum activation of SAE features, which may introduce edge cases and failure modes. As discussed in Section 4.2 and illustrated in Figure 1 (left), PS-Eval is designed to measure how well SAE features respond to LLM activations associated with the meanings of target words. However, it is not guaranteed that the feature with the maximum activation always aligns with the intended word meanings. For instance, observed in the logit lens results presented in Table 3, while SAE features often exhibit strong activation for specific word meanings, this behavior does not consistently generalize across all samples LLMs. In some cases, the feature with the highest activation may reflect unrelated aspects rather than the meaning of the target word, potentially introducing ambiguity into the evaluation. To make the evaluation more robust, would be interesting and important future direction to incorporate similarity metrics that utilize the distribution of SAE features, rather than relying solely on maximum activation.
CONCLUSION
We propose Poly- Doly-Semantic Evaluation (PS-Eval), new metric and dataset assess the ability of SAES to extract monosemantic features from polysemantic activations, in complement to traditional MSE or Lo metrics. Our findings reveal that increased latent dimensions enhance Specificity up to limit, while MSE-Lo-optimizer activation functions do not always improve semantic extraction. Deeper layers and Attention mechanisms significantly boost specificity. underscoring the utility of PS Eval. We hope our work encourages the community to conduct holistic evaluation of SAEs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
HF was supported by JSPS KAKENHI Grant Number JP22J21582.
REFERENCES
Leonard Bereska and Stratis Gavves. Mechanistic interpretability for Al safety review. Transac- fions on Machine Learning Research, 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling, In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL https: worranssormrr-iiiee pub/2023
htlb/2223/mnosemartt htt////oommmniiieeeeeee feat index.idxx..i html.
Maheep Chaudhary and Atticus Geiger. Evaluating open-source sparse autoencoders on disentan- gling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409.04470 04440447. 2024.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600)
2023.
Gytis URL
Daujotas.
Interpreting and steering features in images, 2024. wwtpossst/ggeessseeeeeeee /t/////eeekeeeeaaeeemmm
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-oods, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of super- position. Transformer Circuits Thread, 2022. URL http::/ cicuits.. pub/ 2022 htt/tooymmma/e wtwotooeoolooeoooooooooooeeeeeee /inee//iinxx.//mm/ html.
N. Benjamin Erichson, Zhewei Yao, and Michael W. Mahoney. Jumprelu: retrofit defense strategy for adversarial attacks. arXiv preprint (tfi::19040375500 2019.
Hiroki Furuta, Gouki Minegishi, Yusuke Iwasawa, and Yutaka Matsuo. Towards empirical interpre- tation of internal circuits and properties in grokked transformers on modular polynomials. arXiv preprint arXiv:2402. 16726, 2024.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.0re 0400394406.040033 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, An- tonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher Choquette-Choo. Clément Crepy, Daniel Cer, Daphne Ippolito. David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Hen- ryk Michalewski, Ian Tenney, Ivan Grishchenko. Jacob Austin, James Keeling, Jane Labanowski, Jean- nann-Bapistt Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu. Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko
11

---

Published as a conference paper at ICLR 2025
Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology. arXiv preprint orXiv:2403.08295 2024.
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.01610, 2023.
Zhengfu He, Xuyang Ge, Qiong Tang, Tianxiang Sun, Qinyuan Cheng, and Xipeng Qiu. Dictionary learning improves patch-free circuit discovery in mechanistic interpretability: A case study on othello-gpt. arXiv preprint arXiv:2402. 12201, 2024.
Adam Jermyn and Adly Templeton. Ghost grads: An improvement on resampling, Trans- former Circuits Thread, 2024. URL hnttpss//traasfforrtt jan-update/ /iwex/ndx.///////// html #di ct- www..ttteeennnnneesaaeeee ing.
reuits putpt//22ppp
Adam Karvonen, Can Rager. Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Eoin Farrell, Arthur Conmy, Callum McDougall, Kola Ayonrinde, Matthew Wearden, Samuel Marks, and Neel Nanda. Saebench: A comprehensive benchmark for sparse autoencoders, 2024a. URL httph:t///// /wwww neuronpedia. org/ sae bench/inff
Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders on targeted concept erasure tasks. arXiv preprint arXiv:2411.18895, 2024b.
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. In- terpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.17759, 2024.
Sangwu Lee. Gazing in the latent space with sparse autoencoders. dev.log, 2024. URL https: //re-nn--- n- y. y-g.i--.iee thub. iooii///eee /hoo/dvlog/////oooooo/oo htee/rralllit iini/g///n saee/
Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, and Max Tegmark. The geometry of concepts: Sparse autoencoder feature structure. arXiv preprint arXiv:2410. 19750, 2024.
Tom Licberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretabillity scale? evidence from multiple choice capabilities chinchilla. arXiv preprint arXiv:2307.0945.. 2023.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anea Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma arXiv preprint arXiv:2408.05147. 2024.
Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint arXiv:2405.08366 2024.
Gouki Minegishi, Yusuke Iwasawa, and Yutaka Matsuo. Bridging lottery ticket and grokking: Is weight norm sufficient to explain delayed generalization? arXiv preprint arXiv:2310. 19470, 2024.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217 2023.
nostalgebraist.
interpreting gpt: the logit lens. Less- Wrong, 2020.
URL
https: :/www///wwww/. lesswrong. wttp////ss///////////aaaaaaaaaaa http:///wwww..aooooooooo////////
interpratinng-go-tthe
12

---

Published as a conference paper at ICLR 2025
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. URL eircuits/ ctattts//ooooooe zoom-in. in.
Mohammad Taher Pilehvar and Jose Camacho-Collados, Wic: the word-in-context dataset for eval- uating context-sensitive meaning representations. arXiv preprint arXhv:1808.0912// 2011 2019.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177. 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. models are unsupervised multitask learners, 2019.
Language
Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados and Mohammad Taher Pilehvar. XL-WiC: A multilingual benchmark for evaluating semantic contextualization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp- 7196-7206, 2020.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404. 16014, 2024a.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407.1445: 2024b.
Logan Riggs and Jannik Brinkmann. Interpreting preference models w/ sparse autoen- coders, 2024. URL https: http:://wwwwwwsw.ogggeeooe wwt.///////o/// cotp///psse/tt/i www.:sspa--eeeeee
Haz Sameen Shahgir, Khondker Salman Sayeed, Abhik Bhattacharjee, Wasi Uddin Ahmad, Yue Dong, and Rifat Shahriyar. Illusionvqa: challenging optical illusion dataset for vision language models. arXiv preprint arXiv:2403. 15952, 2024.
Lee Sharkey, Dan Braun, and Beren Millidge. of superposition with sparse autoencoders, 2022. pttptt///0//.tpptE/0r33
Taking
features out URL https:
/ w/w..alignnenttrruu org/ hwagg/pssssseoooo interim research report taking features out out-of- superposition. otpp.iii...
Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11(1):
1-23, 1988.
Viacheslav Surkov, Chris Wendler, Mikhail Terekhov, Justin Deschenaux, Robert West, and Caglar Gulcehre. Unpacking sdxl turbo: Interpreting text-to-image models with sparse autoencoders. arXiv preprint arXiv:2410.22366. 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL https: htappp://rraaan /transffrmer: circiiii..pp.... rit.ooii..suusst..2224404. pubpt024// scall ing mongsmmmttttt indii...... html.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purposs language understanding systems. arXiv preprint (0..Xvv::00000
2020.
Kevin Wang, Alexandre Variengien, Arthur Commy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00500 1.00933. 2022.
13

---

Published as a conference paper at ICLR 2025
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xi- aozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. arXiv preprint arXiv::41 I. 12372, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A GHOST GRAD
In this section, describe the auxiliary loss, which serves as an additional regularization mecha- nism for latent representations during training. The goal is to reduce the likelihood of inactive latent units by penalizing reconstruction errors from subset of dead latents. This approach conceptu- ally similar to the "ghost gradients" method (Jermyn & Templeton, 2024), though modified to handle high-dimensional latent spaces with a more targeted selection of inactive units.
During training, each latent variable is monitored for activation. Latents that have not been activated for a pre-specified number tokens (typically 10 million tokens) are flagged as "dead." These latents are indexed by the parameter Kaux. which represents the top-k latent dimensions with the least activation (usually set to kaux 512).
Given the reconstruction output, we define the auxiliary loss Laux as the squared error between the true reconstruction and the reconstruction obtained using only the dead latents. This is formulated as:
2. where is the input and è is the model's reconstructed
Laux = lee élī.
where è represents the reconstruction from the dead latents, computed as è = Wdec where Waec is the decoder matrix and is the latent vector associated with the top-k dead units.
The full loss during training is given by the sum of the main loss LSAE (Equation 4) and the auxiliary loss, weighted by small coefficient
Liaaal LSAE + aLCaxx
In our experiments, is typically set to 1/32, which balances the influence of the auxiliary term without dominating the primary loss. Importantly, since the encoder forward pass shared between the primary and auxiliary losses, the computational overhead from adding this regularization is minimal, increasing overall cost by approximately 10%.
To investigate the impact of Ghost Grad, we tracked the number of dead latents in the SAE during training. Dead latents are defined as features that not activate even once during 1000 training steps, following Gao et al. (2024). Figure 7 clearly show that introducing Ghost Grad significantly prevents the increase dead latents. Additionally, the PS-eval evaluation also demonstrated better performance when Ghost Grad was applied. Therefore, in paper, we include Ghost Grad as the default setting.
w/ Ghost Grad
M/ Chost Grod Meo Ghout landd
Dead Latents
w/o Ghost Grad
6260
2308
0.2
Optimizer Steps
Acoursry
Metrics
Figure 7: (Left) Changes in the numher of dead features during training. The configuration used is expand ratio 32, I. coefficient 0.05 and layer 4. Without Ghost Grad, the number of dead features increases significantly, while using Ghost Grad prevents this growth. (Right) Relationship between the presence of Ghost Grad and evaluation metries, showing differences Accuracy, score, Precision, and Recall. Using Ghost Grad consistently yields better results compared to not using it.
15

---

Published as a conference paper at ICLR 2025
B OPEN DOMAIN VS IN-DOMAIN
As explained in prior work (Makelov et al., 2024), it remains unclear whether the dataset used for training the SAE should come from the same domain the evaluation data (In Domain) or from a general dataset (Open Domain). In this section, we evaluate the SAEs trained on the WiC dataset (Pilehvar et al., 2024) as the Open Domain dataset. Figure 8 shows the results, where recall is higher for the Open Domain across all expand ratios, while precision is higher for models trained on the In Domain dataset.
Camacho-Collados, 2019) the In Domain dataset and Red Pajama (Weber
64-
Espand Rabto
Dxgand Batto
10spndd Battn
Open Domain
Io Domath
Figure Comparison of Score, Recall, and Precision across different Expand Ratios (32, 64, 128) Open Domain (red pajama (Weber et al., 2024)) and In Domain (WiC data) settings. The results show that Open Domain outperforms In Domain in terms of recall, while In Domain performs better in terms of precision.
C DETAILS OF EXPERIMENT
Here the detailed parameters for this experiment. We initialize the weights of the Decoder the transposed weights of the Encoder, following the methodology from previous studies (Gao et al., 2024).
Table 4: Summary of Training Configuration and Hyper-parameters.
Parameter
Value
Batch Size
8192
Total Training Steps
200,000
Learning Rate
2e-4
Input Dimension (din)
768 (GPT2-small)
Context Size
256
D RANDOM SAE
In this section, we analyze the performance of PS-Eval when SAEs are activated randomly. The experimental setup assumes an expand ratio 32, meaning that the number of features in the SAE is calculated as expand ratio X dmmdde=32xx68--- 268-23366 We compute the probability of overlapping activations (Psame) and the probability of non-overlapping activations (Par).
The total number of ways to choose 2 features from 24576 features is given by:
(4476)
24576 x (24576
2
The probability of selecting the same feature is:
P Psame
3.3115039c 9
24576 (24576-1
24576 x 24575
2
The probability of selecting the different feature is:
Pai
Prame 0..9999999668
16

---

Published as a conference paper at ICLR 2025
Next, calculate the True Positives (TP), False Negatives (FN), False Positives (FP), and True Negatives (TN). We assume that the total number of data samples is 1112, and the number of samples with the same meaning is n 556.
True Positives (TP):
TP same ~ 0.00000184119
False Negatives (FN):
FN X Pairr R 355.999998154
False Positives (FP):
FP (N n) X Psame R 0.00000184119
True Negatives (TN):
TN (N n) Pair 555.999998154
The metries for the random activation of SAEs are summarized in Table 5.
Table 5: Summary evaluation metrics with their respective formulas and calculated values. These metrics highlight the model's performance in terms of accuracy, precision, recall, specificity, and F1 score.
Metric
Formula and Value
Accuracy
Accuracy
TP+-TN
0.5
N
Precision
Precision TP+FF
0.5
Recall (Sensitivity)
Recall = TP+FN
TP
~ 0.0000000319
Specificity
Specificity TN-FF 0.99999999668
Fl Score
Fl
2x x Recall
R3 0.0000000638
Precision Recall
Accuracy vs MSE (L1 Coef 0.05)
Accuracy vs LO (L1 Coef 0.05)
0.07
6475
0.02
E/335
Accuracy
Accuracy
0.578
6.15
2799
Chance
Chance
0.500
C20
10/4
MSE
Normalized L0
Expand
Expand
Expand
Expand
Expand Rati::
Figure 9: Comparison of Accuracy vs MSE and Lo for different Expand Ratios (L, Coef = 0.05). The left panel shows the relationship between accuracy and mean squared error (MSE). while the right panel presents accuracy versus normalized Lo sparsity. The markers represent various expand ratios (8, 16, 32, 64, and 128). Higher expand ratios generally show trend towards better performance, as indicated by the yellow and green markers corresponding to expand ratios 64 128. The horizontal dashed line represents the chance rate, providing a baseline for comparison.
17

---

Published as a conference paper at ICLR 2025
Rell
Jump BeLU
Top
Soon Actioda Faccka
03
01
Aa
Sirenikee 2017
Austen
Figure 10: Comparison F1 score, Precision, and Recall across different activation functions. The models were trained with Jump (with varying thresholds), Topk (with different k), and standard RcLU. The results imply that ReLU may be still better than JUmpReLU or TopK with the polysemous words. The error bars indicate the variation across different 3 seeds. Results for smaller of TopK and the outcomes of JumpReLU using STE provided in Appendix D. The horizontal dashed line represents the chance rate, providing baseline comparison.
Soore
Specificity
Residual
0.8
0.8
SpeciC.ity city
Attenttion
0.5
0.4
12 0.3
0.2
0.1
0.-
Bendual
MELP
Atteskan
0.3-
Benidaal
MLL
Assidivo
Figure 11: Comparison of Fl score and Specificity for SAEs trained on the outputs of the Residual. MLP, and Attention components. The left panel shows the Fl score, where Residual and MLP achieve higher values, while Attention output performs lower. contrast, the right panel shows Specificity, where the Attention output exhibits a significantly higher Specificity, despite the lower score (see Appendix for details on these components). The horizontal dashed line represents the chance rate, providing baseline for comparison.
Specificity vs 1.
Specificity vs MSE
Ligsx:
Ligs::7
Lassu:
Laysz:
Lapse:
laswe
Laye
Layse
Specificty
0.8
Lagse:
lapse 3
Laase: 10
lanse
Specificity
0.1
Lays:
Lays:
layee a
Layer: 10
Logve: 5
Leavr:
Layxe
Layec:
Logoe 6
Layor:
Layee 6
Layee:
0.4
0.4
80-
LO
MSE
Figure 12: Comparison of Specificity across different layers of the SAE in relation to Lu (left) and MSE (right). Each point represents layer, from layer layer 12, with deeper layers (i.c., layer onwards) showing an increasing trend Specificity. This indicates that deeper layers exhibit more distinct activations for polysemous words, consistent with previous findings that deeper layers in SAEs face greater challenges in training, leading to higher MSE and Lo scores. The horizontal dashed line represents the chance rate, providing baseline for comparison
18

---

Published as a conference paper at ICLR 2025
E DENSE SAE
To evaluate the effectiveness of SAE, we trained comparison model, dense sparse autoencoder (Dense SAE) without sparse constraints (A = 0). The results show that the SAE consistently out- performs the Dense SAE across Accuracy, Fl score, Precision, and Recall, with a particularly sig- nificant improvement observed in Recall.
Dense Autoencoder (A 0.0)
Sparse Autoencoder 0.1)
0.8
0.5
0.4
0.3
Accuracy
F1
Precision
Recall
Metrics
Comparison of evaluation metrics between dense autoencoder (A 0 and sparse autoencoder (A 0.1). Metries include Accuracy, Fl score, Precision, and Recall. SAE consistently outperforms dense SAE across all metries, with particularly significant improvement Recall.
Figure 13:
F DATASET DETAILS
We show the some sample of PS-eval in Table 6. Each pair of sentences represents different mean- ings of the target word (e.g., "space" "save," " "ball") the meaning. The label indicates whether the two contexts share the same meaning (Label: for "Same") or differ (Label: for "Dif- ferent"). Poly-contexts demonstrate different meanings of the target word, while Mono-contexts show consistent meanings across sentences.
Table 6: Examples in PS-Eval.
Poly-contexts (space):
Context I: The astronauds walked outer "space" without a tether.
Context 2: The "space" between his teeth.
Label: 0 (Different)
Mono-contexts (space):
Context 1: They stopped at an open "space" the jungle.
Context 2: The "space" between his teeth.
Label: 1 (Same)
Example Sentenees
Poly-contexts (save):
Context 1: This will "save" you lot of time.
Context 2: She "save" the old family photographs in drawer.
Label: 0 (Different)
Poly-contexts (ball)
Context 1: The "ball" already emptying before the fire alarm sounded.
Context The "ball" roiled into the corner pocket.
Label: 0 (Different)
19

---

Published as a conference paper at ICLR 2025
G Fl SCORE, PRECISION, RECALL
Figure 14 shows performance evaluation of Sparse Autoencoders (SAEs) with varying expand ratios (8, 16, 32, 64, 128) across different metrics. The top row shows F1, Precision, and Recall plotted against Mean Squared Error (MSE), while the bottom row shows Fl, Precision, and Recall plotted against Ly sparsity. A lower MSE or Lg indicates better performance, as seen with the trend of improved Fl scores and other metrics at specific expand ratios.
21 Host Get m0)
Pretuce Vaa Oud 553
Read Cede wngg
Dipind Sets.
Dipand Fata.
Peaual Kalum
Repund Bnlan
Pap Ruilerg
Puclon Dipand Fara.
Dipana Sata.
Lipandd
better
Fepunt flati
Eipued Itatx
MSB
MSE
MEE
better
Drpert Paim:
Depand Rate:
Decall
Dpaad Dipadd Bats. 32
Expudd Datio
Expucd Datoo
Datis 123
Figure 14: Comparison of Fl score, Precision, and Recall across different activation functions. The models trained with JumpReLU (with varying thresholds). TopK (with different k), and standard ReLU. The results show that JumpReLU outperforms TopK and standard ReLU in terms score, Precision, and Recall, indicating that is better suited for capturing polysemantic representations.
20

---

Published as a conference paper at ICLR 2025
H ACTIVATION FUNCTIONS
H.1
TOPK
The TopK activation function is designed directly control sparsity in autoencoders by select- ing only the k largest activations from the latent space, while zeroing out the rest. This approach eliminates the need for the commonly used L1 penalty, which approximates sparsity by shrinking activations toward zero, often leading to suboptimal representations. The TopK function is defined as:
2 = TopK Wene (x bpre))
where Wene represents the encoder weights, is the input vector, and 6pr is bias term. By pre- serving only the top k largest values, TopK maintains a fixed level of sparsity, allowing for simpler model comparisons and more interpretable representations.
H.2 JUMPRELU
JumpReLU (Erichson et al., 2019) is activation function introduced to enhance the performance of Sparse Autoencoders (SAEs) by improving the trade-off between sparsity and reconstruction fidelity. While the conventional ReLU activation function zeros inputs less than equal to zero, JumpReLU introduces a positive threshold Specifically, JumpReLU zeros out pre-activations below this threshold, while retaining values above it. This mechanism helps in reducing the number of false positives, thus improving sparsity, while preserving the important feature activations with high fidelity. The function is defined as:
Ju)pReLUg(2) (===-0=)
where H is the Heaviside step function and 0 is a learnable threshold parameter. As illustrated in Figure 15, the JumpReLU function (blue) introduces a threshold = 1, and only retains activations greater than this threshold, in contrast to ReLU (green), which activates any positive input.
Comparison
Mam.jumpRRllll ReLU Activation Functions
5
JumpReLU, (Threshold =
Threshold 9
Antivation Octgo
2-
1
0-
Figure 15: Comparison of ReLU and JumpReLU activation functions. The ReLU function (green) activates inputs greater than zero, while the JumpReLU function (blue) introduces threshold 1, below which all inputs are zeroed out. The dashed gray line represents the threshold, illustrating how JumpReLU suppresses small activations, thereby promoting sparsity in activations.
21

---

Published as a conference paper at ICLR 2025
I COMPONENTS OF THE TRANSFORMER AND ACTIVATION
Figure 20 illustrates the detailed experimental setup corresponding to Figure 6. The evaluation of the residual in Figure is based the SAE trained using the activations from the 'Residual output' shown in Figure 20, Similarly, the results for the MLP in Figure 6 are derived from the SAE trained on the activations from the 'MLP output' Figure 20, and the results for the attention in Figure 20 are based on the SAE trained using the activations from the 'Attention output' in Figure 20.
This demonstrates that even within the same layer, the features captured by the SAE vary depending on the specific component (e.g., residual, MLP, or attention) whose activations are used for training.
logits
Residual output
MLP output
MLP
1layer Transformer
Attention output
Attention
embed
tokens
Figure 16: Illustration of a single-layer Transformer architecture. The input tokens embedded and passed through the attention mechanism and MLP (multi-layer perceptron), with residual connections each stage. The final output logits are computed by combining the results from residual pathways
22

---

Published as a conference paper at ICLR 2025
J TOPK WITH SMALLER K AND STRAIGHT THROUGH ESTIMATOR VARIANT JUMPRELU
We conducted further experiments to evaluate the impact of sparsity and activation function design on SAE performance, as detailed below:
Results for TopK with Smaller k We extended our analysis include TopK results for dmndel /4 = 192 and k / / = 92. Consistent with our expectations, smaller & values achieved higher recall compared to k 2 = 384. This result further emphasizes the benefits of in- creased sparsity in certain evaluation contexts and highlights the importance of selecting appropriate values for optimizing recall in sparse autoencoders.
Results for JumpReLU with STE We also evaluated the performance of JumpReL.U using the straight-through estimator (STE) approach, following Rajamanoharan et al. (2024b). The results indicate that, even with the STE version, the improvements in metrics such as PS-eval were not as significant. This suggests that while JumpReLU offers clear advantages in terms of reconstruction and sparsity, further refinements evaluation metries like PS-eval may be required fully optimize the design of sparse autoencoders.
These findings underscore the importance of carefully selecting evaluation metrics and activation functions tailored to specific goals in sparse autoencoder development. In other words, they suggest that focusing solely on achieving Pareto optimality between MSE and sparse during SAE develop- ment may overlook other critical aspects, such as interpretability.
Jump ReL. (STE)
Top
07-
02
61
Looko PM
Andinn
Ansrrer. ap1
500
Figure 17: Comparison of activation functions (ReLU, JumpReLU (STE), and TopK) across different eval- uation metries: Fl Score, Precision, Recall. The performance of each activation function is visualized. highlighting variations in metric outcomes based on the activation type. ReLU and JumpReLU show competi- tive performance, with notable differences depending on the metric.
23

---

Published as a conference paper at ICLR 2025
K CAN THE OPEN SAE MODEL ACQUIRE MONOSEMOUS FEATURES?
Unlike some previous evaluations for SAEs that rely on the IOI task (Wang et al., 2022; Makelov al., 2024), our PS-Eval model-independdnt and general suite for evaluation, which allows us to assess open SAES trained on activations from models such as Pythia (Biderman et al., 2023) and Gemma (Gemma Team et al., 2024). Table presents the evaluation results for SAEs trained on GPT-2 small (4 layers), Pythia 70M (4 layers), and Gemma2-2B (20 layers). While GPT-2 small closely matches the niighest-perforer
model evaluated in our study (with an expand ratio of 64 and the ReLU activation function), other large language models (LL.M) generally received lower evaluation scores. To effectively acquire monosemantic features, our findings suggest that SAE parameters vary across different models and need to be individually tuned for each specific architecture.
Table 7: Evaluation metrics for open SAF models.
Metric
GPT-2 small
Pythia 70M
Gemma2 2B
Accuracy
0.66
0.51
0.52
Recall
0.94
0.72
0.74
Precision
0.60
0.51
0,51
F1 Score
0.73
0.59
0.60
Specificity
0.38
0.32
0.30
L RELATIONSHIP BETWEEN Lo REGULARIZATION AND ACTIVATION FUNCTIONS
In our setup, with an expand ratio of 32 and a model dimensionality of 768, the total number of SAE features amounts to 24,576. Experiments with the TopK activation function demonstrated that reducing the value of k improves sparsity, as smaller values lead to fewer active dimensions while maintaining comparable performance metries.
Furthermore, we observed that using JumpReL.U with a jump value of 0.0001 results in the largest Lo-norm among all activation functions tested this configuration. This activation function the only one in our experiments where the number of active dimensions exceeds the original model dimensionality of 768.
Table 8: The relationship between Lo regularization and various activation functions. The table compares the number of nonzero latent variables (Lo) for different activation functions, including ReLU, JumpReLU with two jump values (0.0001 and 0.001), and TopK with varying & values & {768, 384, 192). The results indicate that the choice of activation function and parameters significantly influences the sparsity of the model, reflected in the Lo values.
ReLU
JumpReLL--SE (jump=000001)
JumpReLUU-STE (jump=0.001)
JumpReLU (ump=0.0001)
(jmmp=0.001)
JumpReLL
Topk (k=768)
TopK (k=384)
TopK (k=192)
L
531
831
729
829
739
760
383
190
hhttp:::/hngggingfacc..o ggtiggggs/////// ppthis wttpameeeeeeeeettteeteeeeeeeeeee
hhttss://hugginngface.ooo gemma scope- 2b-
24

---

Published as a conference paper at ICLR 2025
M DETAILED EVALUATION METRICS
In this section, we provide additional evaluation metrics to complement the results presented in the main text. First, we enumerate possible evaluation metries derived from the confusion matrix (see Table 9). Following the experiments in the main text, we investigate the impact of the expand ratio, the effect of layer positions, and the influence of different Transformer components on the results. The results show that the metrics that achieve higher values vary depending on the experimental vari- ables. This highlights the need to assess the characteristics of the SAE using variety of evaluation metrics, rather than optimizing for single metric.
Table 9: All the possible metrics we may compute, and their interpretations.
Metric
Formula
Interpretation
Accuracy
TP+IN TP=FP4TN+FN
How well the model predicts the overall situation
Recall
TP TPFPN
When the same meaning's ' words are given, how often they are identified as the same
When the same features are activated, how often the meanings are same
Precision
TT+FP
When the different meaning's 's words are given, how often they are identified as different
Specificity
TN
Sensitivity
TN TN+FN
When the different features are activated, how often the meanings are different
S-F1
2x Recall Precision Recail-Presisiion Precision
How Recall and Precision balanced
D-FI
2x Speciliicityy x Sensitivity S Seunitivity
How Specificity and Sensitivity are balanced
Average-Fi Fl
(( D-FII
How S-FI and D-FI are balanced
va
Merma
Average Narmalood
Normabadd La
Hacraluac
Nortalad Normolued
twechily
Figure 18: Evaluation of various performance metries (Accuracy, Recall, Specificity, Precision, Sensitivity, and
F1 scores) as a function of normalized Lo sparsity across different expand ratios (8, 16, 32, 64, 128).
25

---

Published as a conference paper at ICLR 2025
Lu
Loyin
Figure Evaluation various performance metrics (Accuracy, Recall, Specificity, Precision, Sensitivity, and Fl scores) as a function of normalized Ly sparsity across different Layers.
Figure 20: Evaluation of various performance metries (Accuracy, Recall, Specificity, Precision, Sensitivity, and F1 scores) as a function of normalized Lo sparsity across different transformer component.
26

---

Published as a conference paper at ICLR 2025
N INTERPRETATION AND EVALUATION OF LOGIT LENS RESULTS WITH LLMS
We exploring ways extend this qualitative analysis into quantitative evaluation. One potential approach is leveraging automated evaluation methods using L.Ms, as demonstrated in works like Cunningham from Table of our paper. Specifically, we focused on the example involving the word "space". In this case, Context 1 refers to "space" in the sense of "universe" or "outer space," while Context refers to "space" as in "blank space" or "gap." The input provided to GPT-4 was as follows:
al. (2023). We conducted a preliminary experiment using the results
## rustruct You will be provided with two contexts containing the same Lazget wozd. additionally, related words for the target word will also be given.
Determine
terget word or different meanings in these two contexts.
meaning
2. Explain the meaning of the target word in each context.
Output Format:
Sama or Different: [Your ansver)
deaning Context [Your explenatioa]
Meaning in Context 2: [Your explenation]
4t Context1 selated words flight, plane, shuttle, grevity, craft, Engineers, planes
# Context2 related words -ayout, occupied, spaces, vacated, space, shuttle, occupancy
The following output was generated by GPT-4:
Output:
Same or Different: Different
Meaning in Contaxt 1: The target word refers to vehicle or object designed for travel or teansportation in outer space, such spececraft
shattle.
Mecning in Contaxt: 2: The target word refers physical expanse that can be accupled orleft empty, often used context spatial arrangaments or accupancy.
The results suggest that GPT-4 can interpret the logit lens outputs from the SAE features in manner consistent with human intuition. Specifically, it can distinguish whether a feature corresponds to the same different meanings. Additionally, its reasoning aligns with plausible interpretations, such as identifying that Context 1 refers to "universe" or "outer space," while Context 2 refers to "blank space."
Using an LLM (GPT-4) as an evaluator opens new possibilities for extending the implications of the logit lens results. By systematically applying such methods, we can quantitatively validate the alignment of SAE features with human interpretations, addressing this question in rigorous and scalable manner.
27

---

Published as a conference paper at ICLR 2025
O COMPARISON OF SAES TRAINED WITH DIFFERENT LAYERS
In this section, we present results comparing different layers within the network. Prior studies often select particular layer somewhat arbitrarily. To verify that the trends and conclusions draw not restricted to specific layer, we conduct additional experiments using layer as an alternative input feature source, as described Section 6.
O.1 EXPERIMENTS USING LAYER
Here, we replicate the experiments with the proposed method by substituting layer 6 for layer 4. The accuracy results for different expand ratios are shown in Table 10.
As the expand ratio increases, observe consistent improvements in accuracy. Importantly, the overall patterns remain broadly similar regardless whether we use layer or layer 6. This sug- gests that the choice of layer does not substantially influence the observed trends, reinforcing the robustness of our findings.
Table 10: Relationship between Expand Ratio and Accuracy Layer 4 Layer (Figure 3)
Expand Ratio
8
16
32
64
128
Layer Layer 6
0.6005 0.4982
0.6328
0.6585
0.6669
0.6576
0.6203
0.6370
0.6550
0.6686
O.2 COMPARISON OF ACTIVATION FUNCTIONS ACROSS LAYERS
Next, we compare the impact of different activation functions across layers. In addition to ReLU, we consider STE with varying jump parameters and Top-k methods. The F1 scores, precision, and recall for these configurations shown in Tables 11, 12, and 13, respectively.
The results consistently show that ReLU achieves the highest Fl score, regardless of which layer is chosen. This outcome aligns with the findings obtained from layer 4 alone. Moreover, the trends in performance across different activation functions do not substantially change when using layer instead of layer 4, indicating that our conclusions hold broadly across layers.
Table 11: F1 Scores for Different Activation Functions (Figure 5)
Method
ReLU
STE(jup=00000)
STE(jup=0000001
topk=384
topk=96
topk=192
Layer
0.7623
0.6667
0.6590
0.6550
0.6490
0,6166
Layer
0.7509
0.6203
0.6136
0.5985
0.6203
0.6316
Table 12: Precision for Different Activation Functions (Figure 5)
Method
topk=384
ReLU
topk=192
topk=96
STE(ump=0.001)
STE(jmp=0000001
Layer
0.6939
0.6593
0.5218
0.5251
0.5000
0.4964
Layer
0.5170
0.6729
0.5178
0.5136
0.4901
0.4599
28

---

Published as a conference paper at ICLR 2025
Table 13: Recall for Different Activation Functions (Figure 5)
Method
STE{(jmp=0.001)
STE(jjmp=000001)
ReLU
topk=96
topk=192
topk=384
Layer
0.9951 0.9501
0.9802 0.8615
0.9034 0.8094
0.8813
0.7536
0.6203
Layer
0.8523
0.8094
0.7104
P EXTENDED DISCUSSION
Extending PS-Eval Beyond Natural Language While SAE was initially proposed as a tool to analyze polysemantic activations in language models, its utility has recently expanded to other do- mains, such as diffusion models and laionnlanguage models like CLIP (Surkov et al., 2024; Lee, 2024). These developments suggest opportunities to adapt and extend the PS PS-Eval framework to new contexts. For instance, Surkov et al. (2024) explores the application of SAE in the Stable Diffusion text-to-image model. Given the reliance of this model text inputs, PS-Fval-like methods could be employed to analyze its responses to polysemous text prompts, providing insights into its semantic representation capabilities. Similarly, for vision-based models like CLIP (Lee, 2024), the PS-Eval framework could be extended by leveraging polysemous image inputs; i.e. images with multiple interpretations. Datasets such as the optical illusion dataset proposed in Shahgir al. (2024) offer promising avenue for evaluating how vision-language models handle input ambiguity, serving as a foundation for adapting PS-Eval to visual modalities. These examples highlight the versatility of the PS-Eval framework and its potential applicability beyond natural language processing, provided that input ambiguity and context-specific challenges are carefully addressed in each domain.
Evaluation of Open SAES We extensively evaluate SAES from GPT-2-small and also assess the quality of open SAES from GPT-2-small, Pythia 70M, and Gemma2-2B in Appendix K. The results imply that open SAE trained from larger/more recent LLMS does not always achieve better perfor- mance on PS-Eval (such GPT-2-small > Gemma2-2B) Bereska & Gavves (2024) mention that whether larger and more recent LLMs produce activation patterns that are more interpretable from the perspective of SAE remains unclear despite their better capability natural language tasks. Capable/large-sen -call LLMs may have more complex activation internally. Our results in Appendix K are aligned with their statement. In principle, we should assume that the performances of LLMs and SAES are independent. is not obvious whether the interpretability SAE correlates with the capability of base LLMs, and we think it is worth investigating in future works.
Mechanistic Interpretability In the field of mechanistic interpretability, research spans from comprehensive circuit analysis in toy models, such as grokking (Power et al., 2022: Nanda et al., 2023; Furuta et al., 2024; Minegishi et al., 2024), to methods like SAE for analyzing features in large-scale models such LLMs (Wang al., 2022; Lieberum al., 2023; Gao al., 2024). PS Ev-E, as proposed, provides a practical evaluation framework for model model-indeeennent SAE as- sessments, bridging the gap between theoretical insights and practical applicability.
29


Reviews:
[{"review_id": "Nig9oQcGy2", "reviewer": "Reviewer_s56H", "summary": "This paper introduces a new framework, called PS-Eval, for evaluating Sparse Autoencoders (SAEs), which are widely used to interpret LLMs. The framework repurposes the WiC dataset (Pilehvar & Camacho-Collados, 2019), originally designed as a binary classification task. Specifically, the paper proposes using a subset of WiC to evaluate the extent to which the features learned by SAEs are monosemantic (the ideal case for SAEs). With this dataset, the authors suggest evaluating SAEs using metrics like precision, recall, and specificity. The paper provides a detailed analysis of SAE performance across different layers and transformer components (e.g., residual, MLP, and attention) and compares activation functions such as ReLU, TopK, and JumpReLU.", "questions": "Can the PS-Eval framework be extended to other models or applications outside of NLP, or is it fundamentally tied to the context-sensitive nature of language?\n\nFigure 3: Could you explain why LLM activations are considered less interpretable than SAE features in Figure 3? For me, the fact that LLM activations are squished into the [0, 0.2] interval doesn't make them less interpretable. Instead, it seems like a matter of finding a different cutoff value for LLM activations. Also, how many bins were used for the plot?\n\n---\n\nMinor comments:\n\nIn line 316, you say, \"Specificity measures how often the different meaning words have the same SAE features.\" The correct wording should be \"different SAE features,\" as per Table 2.\n\nIn line 96: \"In our evaluation across different Transformer components, specifically the residual layers, MLPs, and self-attention.\" I believe a part of the sentence is missing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I really enjoyed reading this paper, as it addresses a well-known issue with SAEs that has been overlooked for a long time. It does so by introducing a context-sensitive metric for monosemantic representation, making the paper timely and addressing a current gap in interpretability research. I also appreciated the depth of the experiments, which cover different transformer modules and layers. The analysis is insightful, clearly demonstrating that standard metrics (MSE and sparsity ratio) can be insufficient for capturing monosemantic features. I believe such analyses are very valuable for future SAE design and application. Overall, the paper is well-organized, well-motivated, easy to read, and offers insightful conclusions.", "weaknesses": "Even though I enjoyed reading this paper, my main issue is that there is no baseline for comparison. For example, in Table 4, it is hard to conclude which values of accuracy/precision/recall/specificity signify a good SAE. In other words, even with this new framework, it is unclear whether these SAEs can be trusted. Having a good baseline (e.g., a random SAE or fully dense SAE) could help clarify this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hec7Yxwk5k", "reviewer": "Reviewer_Nb2h", "summary": "This paper proposes a new interpretability metric to substitute the origin reconstruction and sparsity loss in sparse autoencoders. This metric evaluates whether the words of two meanings are the same in features and vice versa. This paper has two findings with this metric: 1. the objective of maximizing reconstruction and sparsity loss does not increase this metric. 2. high layers and attention modules play an essential role in modeling specific semantics of words.", "questions": "1. What is the bar for the features to be the same and different? How do you tune this bar?\n\n2. Why do late or large language models perform worse in identifying semantic similarity, although they got significantly higher scores on challenging benchmarks?  The performance of this metric does not align with the performance on benchmarks. Could you please discuss potential reasons for this misalignment between their metric and benchmark performance or provide additional analysis exploring this phenomenon?\n\n3. In your setting, recall and specificity are more valuable than other metrics.\n\n4. Why do we need to calculate this metric on the representations of the intermediate layer of SAE  but not AE or the origin representations? Can you provide a comparison?\n\n5. Which layer of the transformer do you test as default?\n\n6. Does this metric generalize better to other interpretability than the origin reconstruction and sparsity loss?\n\n---Post Rebuttal Review---\nI have had a discussion with the authors; I think the authors have clarified several issues and promise to add more details and comparisons in the revised version. Thus, I decided to raise the score from 3 to 5.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper first examines the ability to identify polysemantics in the interpreter tool of sparse autoencoders. \n\n2. This paper provides a tool to inspect semantic specificity in the transformers.", "weaknesses": "This paper lacks several justifications. Refer to the Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i0V3sjsCQW", "reviewer": "Reviewer_fi7z", "summary": "This paper introduces PS-Eval, a new method for assessing whether SAEs can extract monosemantic features from transformer models by examining their handling of polysemous words. While the paper offers valuable contributions through its evaluation methods, empirical analysis across different architectures, and interesting findings about layer depth and attention mechanisms, it suffers from significant conceptual limitations. My core concern is the paper's unexamined assumption that SAE features should directly map to word meanings, along with questions about whether insights from polysemous words can generalise to broader SAE evaluation. The empirical work is relatively thorough and the findings are interesting, particularly regarding the role of attention in handling polysemy and the logit lens analysis, but the theoretical framework needs strengthening and the scope may be too narrow. I recommend rejection in its current form while strongly encouraging resubmission after addressing these conceptual issues, improving the theoretical framework, and clarifying the presentation.", "questions": "1. How do you justify the assumption that SAE features should directly correspond to word meanings? Could you discuss alternative feature organisations that might be equally valid? \n* For instance, it's easy to imagine an SAE feature that fires on a particular word (e.g. \"tweet\") regardless of context, and another SAE feature fires on sentences regarding birds and another fires on sentences regarding social media (e.g. \"Twitter\"), distinguishing the two. It's not clear to me that this is a strong failure mode of an SAE. \n* I don't know if we should expect SAEs to learn features directly corresponding to meaning (the strong linear representation hypothesis) because we don't know what the atomic units (i.e., features) of a model's computation actually are.\n2. Could you elaborate on how insights from polysemous word evaluation might generalise to other aspects of SAE quality?\n3. The logit lens analysis shows promising results - could you expand on how this validates your evaluation method?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "__Novel evaluation methodology__:\n* Clear metrics based on confusion matrix approach\n* Well-defined dataset construction from WiC\n* Systematic evaluation framework that could be extended to other models\n\n\n__Experimental analysis__:\n* Thorough comparison across different activation functions\n* Valuable insights about layer depth and transformer components\n* Interesting findings about attention's role in handling polysemy\n* Useful logit lens analysis demonstrating semantic differentiation\n\n\n__Technical contributions__:\n* Model-independent evaluation metric\n* Practical insights for SAE deployment\n* Clear demonstration of limitations in current MSE-$L_0$ optimisation", "weaknesses": "__Core conceptual concerns__: The paper assumes SAE features should directly map to word meanings. This is a strong assumption that needs more justification. It's possible that effective SAEs might represent meaning through other feature combinations. The focus on polysemous words, while interesting, may be too narrow to support broad conclusions about SAE quality.\n\n\n__Methodology limitations__:\n* The evaluation method could benefit from more discussion of edge cases and failure modes\n* Limited exploration of how results generalise beyond the specific test cases\n* Need for clearer discussion of how PS-Eval relates to existing SAE objectives\n\n\n__Presentation issues__:\n* Some key terms need clearer definition, particularly in the introduction (e.g. \"semantics-focused evaluation\", \"polysemous words\")\n* Several figures would benefit from more detailed captions\n* Some sections (particularly the transformer component analysis) need more thorough explanation\n* Quite a few typos (e.g. \"assesse\", \"spolysemantic\", and incomplete sentences e.g. \"In our evaluation across different transformer components, specifically the residual layers, MLPs and self-attention.\")", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kwow3FPZQ7", "reviewer": "Reviewer_k8GD", "summary": "In the literature, SAEs are often evaluated on MSE and L0. The paper introduces a new metric for evaluation of SAEs that directly measures whether SAEs learn separate features for the different meanings of polysemous words (words which have different meanings based on the context). The metric is constructed from the WiC dataset (filtering for words which tokenize to single tokens). The paper finds that improving the MSE-L0 frontier does not necessarily lead to improvements on this newly introduced metric. The paper also shows that eventually, scaling SAEs no longer improves score on this metric, that alternative activation functions are not always better, and that later layers", "questions": "- Normalized L0 (L0 / N) is less informative as a metric than L0 because the amount of information in the latents is roughly proportional to L0 * log N, so across different N's, L0 is roughly comparable, whereas L0 / N is not very comparable.\n- How are the SAEs initialized? Prior work (Conerly et al., 2024; Gao et al., 2024) has found that initializing the encoder to the transpose of the decoder is very important for scaling SAEs.\n- What are the L0 values attained by the SAEs for each activation function? Are they in a similar range? Are they less than d_model?\n- How many dead latents are there in the SAEs?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Well motivated metric. Correct in identifying problems with MSE/L0 as a metric of SAE quality. Using polysemy as a source of eval is a good idea.\n- Confusion matrix approach is reasonable and consideration of metrics like F1 provides more granularity.", "weaknesses": "- k=384 is an extremely high k for TopK. Gao et al. (2024) recommends using d_model / 2 for the _maxk_ hyperparameter, not themain k hyperparameter. Ideally, a much smaller k is used for $k$. Using a value of k equal to or larger than d_model doesn't make much sense.\n- JumpReLU comparison would be stronger if the STE variant from Rajamanoharan et al. (2024) were used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)