Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Authors: Subhash Kantamneni
Joshua Engels
Senthooran Rajamanoharan
Max Tegmark
Neel Nanda
Venue: ICML 2025 poster (2025)
Keywords: Mechanistic Interpretability
Sparse Autoencoders
Probing
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We evaluate sparse autoencoders for probing and find that they underperform baselines; our results raise questions about the effectiveness of current SAEs.

Abstract: Sparse autoencoders (SAEs) are a popular method for interpreting concepts represented in large language model (LLM) activations. However, there is a lack of evidence regarding the validity of their interpretations due to the lack of a ground truth for the concepts used by an LLM, and a growing number of works have presented problems with current SAEs. One alternative source of evidence would be demonstrating that SAEs improve performance on downstream tasks beyond existing baselines. We test this by applying SAEs to the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift. Due to the difficulty of detecting concepts in these challenging settings, we hypothesize that SAEs’ basis of interpretable, concept-level latents should provide a useful inductive bias. However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to design ensemble methods combining SAEs with baselines that consistently outperform ensemble methods solely using baselines. Additionally, although SAEs initially appear promising for identifying spurious correlations, detecting poor dataset quality, and training multi-token probes, we are able to achieve similar results with simple non-SAE baselines as well. Though we cannot discount SAEs’ utility on other tasks, our findings highlight the shortcomings of current SAEs and the need to rigorously evaluate interpretability methods on downstream tasks with strong baselines.

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Subhash Kantamneni
Joshua Engels"
Senthooran Rajamanoharan Max Tegmark Neel Nanda
Abstract
1.0
Logistic Regression
Sparse autoencoders (SAFs) are popular method for interpreting concepts represented in large lan- guage model (LLM) activations. However, there is lack of evidence regarding the validity of their interpretations due to the lack of ground truth for the concepts used by LLM, and grow- ing number of works have presented problems with current SAFs. One alternative source of evi- dence would be demonstrating that SAEs improve performance on downstream tasks beyond exist- ing baselines. We this by applying SAEs the real-world task of LLM activation probing in four regimes: data scarcity, class imbalance, label noise, and covariate shift. Due to the diffi- culty of detecting concepts in these challenging settings, we hypothesize that SAEs' basis of in- terpretable, concept-level latents should provide useful inductive bias. However, although SAEs occasionally perform better than baselines indi- vidual datasets, are unable ensemble SAEs and baselines to consistently improve over just baseline methods. Additionally, although SAEs initially appear promising for identifying spuri- ous correlations, detecting poor dataset quality, and training multi-token probes, we are able achieve similar results with simple non-SAE base- lines well. Though cannot discount SAEs' utility on other tasks, our findings highlight the shortcomings of current SAES and the need to rigorously evaluate interpretability methods on downstream tasks with strong baselines.
0.9
SAE Probe
Text AUC
0.6
0.5
Standard Conditions
Data Scarcity
Class Imbalance
Label Noise
Do SAE
X
Help?
Figure 1. SAE prohes underperform the baseline of logistic re- datasets.
gression in each regime when taking the Additionally, we find that bascline methods can provide many of the interpretability insights of SAE probes. Bars compare the best performing baseline method (logistic regression) to the best forming SAE probe (width 16k, 128).
notably, Bricken et al. (2023); Cunningham et al. (2023) demonstrated the promise sparse autoencoders (SAEs) and sparked considcrable follow-up work. This includes papers focused on improving the downstream cross entropy loss of SAE reconstructions (Gao et al., 2024; Braun et al., 2024; Rajamanoharan al., 2024a;b), improving SAE train- ing efficiency (Mudide et al., 2024; Gao et al., 2024), finding weaknesses in SAEs and proposing solutions (Chanin et al., 2024; Bussmann et al., 2024b), using SAFs to understand model representation structure (Engels et al., 2024a;b; Li et al., 2024), training large suites of SAEs (Lieberum et al., 2024; He et al., 2024), and developing benchmarks for SAEs (Huang al., 2024; Karvonen et al.; 2024b).
Unfortunately, we lack "ground truth" to know whether SAEs truly extract the interpretable concepts used by lan- guage models. Prior work has largely avoided this limitation by evaluating SAEs with proxy metrics like reconstruction loss (Rajamanoharan al., 2024a;b; Gao et al., 2024; Olmo et al., 2024; Braun al., 2024). These metrics are tractable to optimize for, but do not necessarily align with mecha- nistic interpretability's (MI) goal of better understanding neural networks (see Bereska & Gavves (2024) and Sharkey et al. (2025) for surveys of MI). We argue that if SAES truly advance MI's goal, they should improve performance on a real, hard-to-fake model control or explainability task.
1. Introduction
Dictionary learning is popular method for interpreting LLM activations (Arora et al., 2018; Yun et al., 2021); most
'Equal contribution 'Massachusetts Institute of Technology. Correspondence Subhash Kantamneni <subhashk mit.edu>, Joshua Engels Beinglss@mi..ddut
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
However, despite the extensive body of work on SAEs, there relatively few cases where SAEs have been shown to help on such task: Marks et al. (2024) show that SAE feature circuits can help identify and remove bias from classifiers; Smith & Brinkmann (2024) show that SAES can identify misaligned features learned by a preference model; and Karvonen et al. (2024a) show that SAE feature ablations are better at preventing regex output than baselines in setting with scarce supervised data. While these results are promising, they all study a single example in detail, and consider comparative baselines with varying levels of rigor.
Probe
Beleot sroeen Iatent cat soroa canasa
Train logisic regression probe. regubrizztion
KNN
MLP
SAE Encader
SAE
LagReg
Probe
XGBoos:
LogReg
Closs O'Activation Vectors
Class 1 Aclivvtionn Vactors
At the same time, there are also surprisingly few negative results finding that SAEs do not help on downstream tasks; in fact, we are only aware of Chaudhary & Geiger (2024), which finds that SAE latents are worse than neurons at dis- entangling geographic data, and Farrell et al. (2024), which finds that pinning related SAE latents is less effective than baselines for unlearning bioweapon knowledge. Thus, it is not clear whether SABS are just one result away from being differentially useful, or if MI should seek fundamentally dif- ferent methods. We defer a thorough examination of related work to Appendix A.
An illustration of our SAE probing method. Wo pass in training activation vectors from cach class and train an L regularized logistic regression prohe on the latents that differ the most between classes. Right: We ensure robustness our results with the "quiver of arrows" approach (see Section 2.4): we add SAE regression into set of methods, and see if the test accuracy of the best method (chosen by validation accuracy) increases.
Figure 2. Left:
MI works to rigorously design baselines when evaluating the utility of interpretability techniques.
In this work, we attempt to fairly evaluate the utility of SAEs by examining their competitive advantage on a concrete task: training probes from language model activations to targets (Alain, 2016). Probing has two important qualities:
2. Methodology
We apply probes to the hidden states of two language mod- els. Gemma-2-4 22-9B (Riviere al., 2024) and Llama-3.1-88 (Grattafiori et al., 2024). Our main paper results use Gemma- 2-9B. We replicate core results on Llama-3.1-8P in Ap- pendix I. We use JumpReLU (Rajamanoharan et al., 2024b) sparse autoencoders (SAEs) from Gemma Scope (Lieberum et al., 2024) for Gemma-2-9R and Topk (Gao et al., 2024) SAES from Llama Scope (He et al., 2024) for Llama-3. 1-8B. See Appendix A for further background on SAEs.
1. Probing is practically useful: Probing has been used to investigate LLM representations (Gurnee & Tegmark, 2023; Nanda et al., 2023; Heinzerling & Inui, 2024), de- tect safety relevant quantities (Zou et al., 2023), remove knowledge from models (Elazar et al., 2021), and catch synthetic sleeper agents (MacDiarmid et al., 2024).
2. SAEs might reasonably improve probing: Theoreti- cally, SAE latents are more interpretable basis of model activations, and we hypothesize that this inductive bias will help train probes in difficult regimes. Recent work has also found some positive results for the utility of SAE probes (Bricken et al., 2024a; Gallifant et al., 2025).
2.1. Classification Datasets
We collect a diverse set of 113 binary classification datasets listed in Table (Appendix C) For example, 26 headllineisfroonppan requires probes to identify front- page headlines and 136 glue.mnli entailment requires probes to identify logically entailment. For other exam- ple datasets, refer to Table 1. All datasets are titled in the form ID pddsscippioon. We originally collected datasets for other purposes and discarded some of them; thus, while we have 113 datasets, ID ranges from [5, 163).
Thus, curate 113 13 linear probing datasets from variety of settings and train linear probes corresponding SAE latent activations (see Figure 2). We compare to a suite of baseline methods across 4 difficult probing regimes: 1) data scarcity, 2) class imbalance, 3) label noise, and 4) co-variate shift. Unfortunately, we find that SAE probes fail to offer a consistent overall advantage when added to a simulated practitioner's toolkit.
All datasets contain prompts and targets. Probes are tasked with predicting the target from the model's hidden activa- tions when run on prompt. We focus on binary classifi- cation, since SAEs are mostly thought of as representing binarized latents (see Bricken et al. (2023)). Thus, target is either 0 or 1. The prompts in our datasets range in length from tokens to a (left-truncated) maximum of 1024 tokens.
Further, we explore areas that Bricken et al. (2024a) suggest SAES may be valuable for, including detecting attributes distributed over multiple tokens and identifying dataset is- sues. Although SAEs initially seemed promising in these settings, we were able to achieve the same results with im- proved baselines. Our results underscore the necessity for
2

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Table 1. Example probing tasks. Only tasks with short prompts shown for conciseness.
Dataset Name
Prompt
Target
26 wheaaline.isfroonppiiiieeeeeeeee
Sebelius's Slow-Motion Resignation From the Cabinet.
1
MI5 References Emerge in Phone Hacking Lawsuit.
0
36 36.sciq.tt tff
Q: Binary fission is an example of which type of production? A: asexual
1
Q: What occurs when light interacts with our atmosphere? A: prism
0
effect
114.nyc www.nyy.bbbou....om
www.bog..aaahattaaaatt Manhattan
INWOOD BASKETBALL COURTS
1
PS 18 JOHN WHITTIER
0
All moved to our new apartment. So exciting
is noow calmmm eating polvoron . yuumm
0
155
Jimmy Butler
1
Steve Balboni
0
2.2. Probing Strategy
SAE probes (Karvonen et al., 2024b; Bricken et al., 2024b), However, most probing studies operate on the final token, which we choose to emulate. In Appendix K, we present results considering probes on multiple tokens.
To train probe given dataset, we first run the model all prompts from that dataset to generate model activations at each layer I on the last token, X" X_ is of shape (len(prompts), model dim). We probe the last token to ensure the target information was present in the preceding context. For activation probes, we then train a probe p to map from XL_ + 1, where /. are the targets in our dataset.
Throughout this study, we use the area under the ROC curve (AUC) to evaluate the quality of probe. AUC is the like- lihood that the classifier assigns higher probability of larget 1 to larget = 1 example than to larget = O example. Using AUC allows us to comprehensively assess probe performance agnostic to classification thresholds. For additional details on AUC, see Appendix D.1.
Our SAE probe training technique is summarized in Fig- ure 2. We first pass the training dataset xL, through the SAE encoder, resulting in a batch of vectors in the SAE latent space Z = SAE(X"1) of shape (Ien (prompt)),) W), where Wix is the width of the SAE. We do not train probes di- rectly on the SAE latent space because we hypothesize that small number of SAE latents encodes the desired concept. Instead, we create basis of latents with the highest average absolute difference between the set of training prompts Ti with target- = 1 and the set of training prompts To with target = 0. More formally, if SAE(X",) ) E RWW, where W >> dmoxtet, we choose k < W and find indices I as follows:
Often, probe p has hyperparameters hp we would like to optimize. We select h, that has the maximal validation AUC using the cross- validatoio strategy described in Table 5. We then test p with optimal hyp on a held out test set to calculate AU AU All datasets have at least 100 testing examples, with most having more (the average test set size is 1945).
2.3. Probing Methods
We use 5 baseline probing methods, detailed with their re- spective hyperparameters in Table 2. We evaluate 10 hyper- parameter values hp for each probing method p. For the first three probing methods, we use grid search, while for MLPs and XGBoost, we randomly sample from a hyperparameter grid to manage the larger search space. For additional de- tails on hyperparameter ranges, see Appendix D.3. We use all five probing methods as baselines, but for SAE probes we only use logistic regression.
I = arg top k iE(W)
1 T
Σ
(1)
Tol
JETI
JETo
We then train a probe PSAE to map from Z[:, Z] + t.
We note that Equation (1) has bias toward latents with larger ranges. We thus also experiment with replacing Z with A.
Znd, where X; the average value of latent i when it active. This alternative selection metric leads better results for small & (k- < 32) and similar results for larger k. but does not change our overall quiver of arrow results significantly (Gemma experiments use the normalized se- lection metric and the Llama experiments do not).
2.4. Experimental Setup: Quiver of Arrows
To evaluate whether SAE probes provide an advantage over baselines, we use an evaluation metric we call the "Quiver of Arrows": we ask whether adding SAE probes (a new "arrow"") to the set of existing probing methods available to a practitioner (the "quiver") increases performance compared to practitioner without access to SAE probes. In other
Note that previous work in SAE probing has aggregated SAE latents across tokens to provide a richer input space for
3

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Method
Hyperparameters
Logistic Regression
L regularization for SAE probes, L2 otherwise
PCA Regression
Number of PCA components to reduce X to before using unregularized logistic regression
K- K-Nearest Neighbors (KNN)
Number of nearest neighbors
XGBoost (Chen & Guestrin, 2016)
meettimtoors, max .eeth, learning rate, subsample, colsample_byg bytree, reg reg_alpha, reg lambda, min.child_weight
Multilayer Perceptron (MLP)
Network depth, hidden state width, learning rate, and weight decay
Table Overview baseline methods and their associated hyperparameters.
Layer 20, Width 16k
Test AUC (with SAE)
1.0
Mean -0.003 0.001
400
No Change
300
0.8
Average Test
9
0.8
0.6
100
0.6
0.7
0.8
0.9
1.0
Test AUC (without SAE!
25
2
2
Number of Features Used in SAE Probe (k)
Figure 4. In standard conditions, when SAE probes are added to the quiver, they chosen by validation AUC 14/133 times. However, we find a slight decrease in test AUC. Datasets not directly on the diagonal signify that SAE method was chosen from the quiver.
Figure For given width, using higher and constructing probes with larger basis of latents (k) is more performant.
words, over a collection of methods we choose the best method by validation AUC and then report the test AUC of that method. We can then compare the marginal improve- ment of adding SAE probes to a practitioner's toolkit by comparing the Quiver of Arrows AUC of baeline methods with and without SAE probes.
recommendation on the usefulness of SAE probes, Addi- tional discussion of the metires of the Quiver of Arrows is in Appendix D.4.
3. Comparing Probing Techniques in Different Regimes
The quiver of arrows can be defined formally follows. Given a set of probing methods P = Pi,...,PP Pr. (listed in Section 2.3), we find AUC p; for each method using the pro- cedure described Section 2.2. We then choose the probing method p. with maximal AUCya p, and record its AUCfet AUG We can then compare AUC to AUC for a different set of methods /' pi...... ......,,py PA, If we let P be a set of baseline methods, and P' PU (SAE probes), then AUC AUCket directly represents the increase in test performance when adding SAE probes to practitioner's toolbox. We then aggregate AUCket AUC across many different datasets and testing regimes (e.g., dataset size) to give an overall sense of the improvement due to SAE probing.
3.1. Standard Conditions
We initially assess probes under standard conditions, char- acterized by sufficient data for probe convergence and bal- anced classes. We find that both baseline methods and SAE methods perform the best on layer 20 (see Figure 12a, Ap- pendix E. I), so we run experiments with this layer. We train baselines on 1024 data points (or the maximum number of points in the dataset).
We first conduct preliminary investigation to cut down the large space of Gemma Scope SAEs. We train probes for all SAEs using k latents (for logspaced values of k) on all datasets using 1024 training examples. We calculate the average test AUC for each SAE probe across all datasets. We find that SAE width is relatively unimportant to probe success, while larger LOs lead more performant probes (Appendix E.2). Additionally, as might expect, using larger value of k leads to better probe performance, as shown in Figure 3.
The quiver of arrows approach is designed as a robustness check to fairly evaluate the benefit of adding SAE probes to practioner's is toolkit. If we instead constructed probes from large set of SAEs and chose the best one based on test AUC, we could be tricked into thinking SAES outperform baselines because we accessed the held-out test set. With the quiver arrows approach, we emulate the information a real practioner has access to, allowing us to make a robust
Thus, throughout the paper we train probes using the largest available T.0 for SAEs with width 16k, width 131k,

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Data Scarcity
Class Imbalance
Label Noise
26, hwwhhheaneeeeee. -istolitassoooeaaaeeeo
26_ 26_head inc_ jsfrontpage
26. headline lttpriinns_soooaageee
Test AUC
1.0
___sciq_tf
36_sciq_
36_s 36_5ciq_tt
Test AUC
1.0
0.8
114_nye nyc_borough Mar_harughhMaahattann
Monhattan
114 314_nyc_u hyc_borough Manhattan _aahatg_
114_nyc www__yc__borrro www.boggh__ahhattalll
Test AUC
0.8
0.6
Average
Average
Average
Test AUC
0.
0.4
Number Training Samples
Positive
Fraction Corrupted
Figure 5. For three of the datasets in Table 1, we visualize the performance when SAE probes are in the quiver (dashed) versus when they are (solid) for the regimes data scarcity (left), class imbalance (middle), and label noise (right). all three regimes, we see that on average (bottom row), SAEs do not help.
SAE A (AUC)
-0.1
4 24 2
0.05
Number Training Examples
Ratio of Positive Class
Fraction Corrupted
Figure 6. For the regimes of data scarcity. class imbalance, and label noise, we see no average improvement across datasets when adding SAEs to quiver. Shading represents 95% confidence intervals.
and width = 1M. We use 1 = 16 to construct easily interpretable probes that potentially overfit less and use 128 for performance.
example, a researcher observing a rare model phenomena would like a probe that generalizes with few total exam- ples or few examples the desired class, while a researcher working with collected user data wants generalizable probe even if some fraction of users respond randomly. See Ap- pendix G for explanation of why we expect SAE probes to provide benefit in each of these regimes.
Using this of probes, we consider quiver of arrows approach in standard conditions. SAEs are chosen as the "ar- row" for 14/113 13 tasks, however, we see a slight decrease in performance when they are added to the quiver in Figure 4.
Each regime characterized by a parameter which we vary to compare SAES and baselines.
unsurprising that SAE probes underperform baselines in standard conditions, as their inductive bias marginal given large, balanced dataset. Thus, we now investigate more difficult settings to test if the inductive bias of SAEs translates into a competitive advantage for probing.
Data Scarcity We use 20 values n logspaced in [2, 1024], where the number of training examples. We use standard test set.
Class Imbalance We use 19 values of ratio linearly spaced between [0.05, 0.95), where ratio = and mı is the number of target = 1 training examples. The test set uses the same ratio. Label Noise We use values of fraction linearly spaced between [0, 0.5). where fraction Meorrupted We use standard test set (uncorrupted).
3.2. Data Scarcity, Class Imbalance, and Label Noise
In this section, we consider more difficult probing regimes: limiting the training data (Data Scarcity), changing the relative frequency of target = examples (Class Imbal- ance), and randomly flipping a fraction of the targets (Label Noise). All of these settings are realistic workflows for
5

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
We evaluate the first two regimes with the quiver of arrows method. However, the quiver of arrows method relies on the validation data being representative of the test data. This assumption fails for the label noise setting since the vali- dation data is corrupted. Thus, a hypothetical practitioner would likely deploy their most performant method from other settings instead of allowing corrupted validation AUC to arbitrarily choose method. To emulate this, we compare logistic regression and the width 16k, k= 128 SAE probe head-to-head in the setting of label noise. We choose this SAE probe because it has the smallest width and highest k. which we have empircally found to be most performant.
1.0
0.8
0OD Test AUC
0.6
0.4
0.2
Beseline AUC
AUC
SAE AUC
0.0
Hest fig. lemale Frcttrrl Characlers
8. hist flg itamericn Rarofnn Letter Invertzed
hist Ispolttciaa Reversed Name Ordar
66 Buing -iin----mmm Taanlated French)
67 social soclal-tacuri mansia to Spanish
73 Suntrol-group German
#7 GUDEX boD
20 gitee and GUDES bol
Mdaon across Averoue
Trandia
In Figure 5, we visualize the SAE versus non-SAE quiv- ers for a sample of datasets listed in Table across the three regimes. Additionally, we visualize the average perfor- mance difference between the SAE and non non-SAE quivers for each regime across all datasets in Figure 6. At all parameter values in each regime, SAEs show no meaningful improve- ment over baselines. This is not because SAEs are not chosen from the quiver; in Appendix F we see that SAEs are chosen for up to 40 datasets in each regime. SAEs simply underperform baselines in these settings. See Appendix F for results for individual datasets.
Dataset
Figure 7. SAB probes often generalize worse than baseline logistic regression with covariate shift.
each dataset (see Bills et al. (2023)), We investigate three applications of labeled latents:
1. Probe Interpretability: In Section 4.1, we investigate why SAE probes fail to perform well by pruning latents that 01 (OpenAI, 2024) ranks as spurious.
2. Latent Interpretability: In Section 4.2, we generate autointerp explanations of each dataset's top latent find spurious latents and latents that fit well our probes in way not explained by the autointerp label.
3.3. Covariate Shift
We now investigate if SAE probes are more resilient to dis- tribution shifts in prompts. To model this covariate shift, we create or use 8 out-of-distribution (OOD) datasets: two pre- existing GLUE-X datasets which are designed as "extreme" versions of tasks 87 and 90 to test grammaticality and logi- cal entailment respectively; three datasets (tasks 66, 67, and 73) which we alter the language of: and three datasets (tasks 5, 6, and 7) where we use syntactical alterations to names or use cartoon characters instead of historical figures. We train probes on these datasets in standard settings and evaluate on 300 covariate shifted test examples.
3. Detecting Dataset Quality Issues: In Section 4.3, we invesigate spurious dataset features and label errors us- ing insights from the top latent descriptions and firing patterns.
4.1. Probe Intepretability: Pruning and Latent Generalization
We first use autointerp to investigate why SAE probes fail to generalize to covariate shift. We have two initial hypothe- ses: 1) the latents SAE probes rely on leverage spurious correlations in the training data and 2) the latents are not spurious, but they are not robust to the distribution shift.
Like the label noise setting, our validation data is unfaithful to our test data. Thus, we compare logistic regression to a single SAE probe. We construct SAE probes from the width- = 131k, LO 114 SAE, as latent descriptions are available for this SAE Neuronpedia (Lin, 2023), which facilitates later interpretability related investigations. Our re- sults (Figure 7) show that baselines outperform SAE probes when generalizing to covariate shifted data.
First, investigate hypothesis 1. Specifically, we attempt to improve SAE probes OOD performance by pruning la- tents deemed spurious by their autointerp description. We focus on three OOD tasks that SAE probes perform poorly on: w.tiss.ftt.ssoliiiii (Figure 7). We use the width 131k, Lo 114 SAE.
66_ fo.living--oon and 90. glue_.nli
4. Interpretability
For each task, we take the k top latents by mean differ- ence and use Slaude-3.5-Sonne (Anthropic, 2024) to gen- erate latent descriptions with autointerp, in addition to one of the authors manually labeling latents. We use a smaller k = 8 probe in this experiment as a proof-of-concept, given the additional expense of labeling. For this and all sub- sequent experiments, we generate autointerp labels using
While we find that SAE probes are not helpful in traditional probing settings, one intrinsic advantage of SAE probes is that their input basis is interpretable. To leverage this, we use the technique of automatic interpretation, or autointerp, to create natural language descriptions of top latents for
6

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Neuronpedia (Lin, 2023), which leverages a language model
Latent 122774 on "Living Room" Different Languages
to produce consistent natural language explanations for latent based on its top activating tokens (the Neuronpedia autointerp implementation is based on (Paulo et al., 2024)).
Latent Vatent Value
20
We then use OpenAl's ol model to rank the relevance of each latent's description to the task. We also prompt ol to downrank spurious latents given the OOD transformation. An example of this procedure for the task 66 ro.ii, which identifies if the phrase "living room" is in an English sentence, is shown in Table 7, Appendix J.1. For this task, ol ranked latent 12274, which identifies "mentions of living rooms" first, while ranking latent 51330, which identifies "objects and materials related to scientific experiments," last.
Engllsh
Spanish
French
German
italian
Portuguese
Russian
Chiness
Japanese
Korean
Arabic
Hindi
Bengali
Dutch
Turkish
Language
Figure 8. Latent 122774 most active on the English translation of "living room," and does not fire on French.
We then construct a probe with the top k latents using ol's relevance rank for k E [1,8]. If there are spurious latents in the k probe, to have better OOD generalization. We visualize each task's performance by & in Figure 21, Appendix J.1. For and 90 go__lle__nli, we see that pruming works, with a 0.024 and 0.052 increase in AUC between the dataset. However, for 7_hist wwt.ffiiippooo ispolitician, OOD test AUC increases by 0.077 after pruning. This indicates that the task requires & = 8 latents to represent. This procedure is dependent on the efficacy of autointerp. To remove this confounder, a separate author ranked the relevance of human labeled latents for all three tasks, with no change to the results.
classes) for each dataset for the width 131k, LO 114 SAE, and evaluate each latent's k = 1 SAE probe in stan- dard settings. We then generate an autointerp explanation with GPT-4o for the best latent.
expect probe with & < 8
Table 7 (Appendix J.2) contains a breakdown of a selec- tion of interesting top latents that we find. We see that some latents' descriptions fit their tasks, like latent 81210 for 5_ fhiis_fig_ismale, which activates on "references fe- male individuals." Another interesting category is incorrect autointerp labels (e.g. for 225_wold_counte Italy, latent 5081 17 has 0.989 AUC, implying (correctly) that it fires on Italian concepts; however, the description reads "names of researchers and scientific authors"). However, some latents appear to be spurious and unrelated to the semantics of their dataset. For example, 22.headline headlines published during the Obama administration, is classified with 0.782 AUC by latent 10555, which activates on "strings of numbers, often with mathematical notations." "
two of the datasets, 66 66_livvng-roon
8 and & = probe for each
7 T.hist.iggispoliiicee
which targets
Our preliminary results show that pruning helps SAE probes generalize. However, the drop between in-distribution (ID) and OOD performance is much larger than the modest im- provement from pruning. This indicates that spurious cor- relations are not the primary reason SAE probes fail to generalize. Thus, we consider the second hypothesis - that the underlying SAE latents do not generalize well OOD.
The spurious latent category seems especially promising because finding a spurious latent may help us identify spu- rious features in the dataset. However, in case study in Section 4.3.2, we find that similar findings may be possible using baseline classifiers: we apply logistic regression probe to model hidden states on tokens from the Pile (Gao et al., 2020) and show that maximally activating examples also exhibit the spurious correlation.
On the task 66 l6liiviggroom, latent 122774 has an ID test AUC of 0.99 while only having OOD test AUC of 0.64. Since the OOD transformation involves translating the prompts to French, we hypothesize that this latent is active on the phrase "living room" in English but not other languages. We use GPT-4o to translate "living room" to 15 languages, and Figure 8 we indeed observe that latent 122774 is more active on the English translation than all other languages, and it is not active the French translation at all. Thus, latent 122774 does not generalize OOD, which explains why the SAE probe also failed to generalize.
However, a practical advantage for SAES that the in- frastructure to perform autointerp pre-existing through platforms like Neuronpedia, and theoretical advantage is that the baseline classifier can only identify the single most relevant coarse-grained feature, while the decomposabil- ity of SAE probes into latents allows for identifying many independent features of various importance.
4.2. Latent Interpretability
4.3. Detecting Dataset Quality Issues
We next generate autointerp descriptions for the most de- terminative latent for each dataset, allowing us to identify interesting categories of latents. Specifically, we consider each of the top 128 latents (by mean difference between
In addition to generating top latent descriptions for all datasets, we present in-depth case studies on two datasets: 87 "7_glue_cola and 110 1..0aammde_iii hummangt.hmmmgp33.... These datasets were selected after an initial investigation of five datasets
7

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
1.0
1,00
[Mean Act Difference) [AI
Human
Test Test Accuracy
0.820.77 0..7
0.72
0.78 0.78
Total Activation
1500
Al
0.5
Labels
0.56
1000
Baseline
SAE (k-128)
500
Latert 3695885
0.0
CoLA Labels
Clean Labels
Mislabeled
10515C: perodds
116248:
3764: spcee
period
maci
spaces
mark
point
Figure Latent 369585 outperforms dense SAE probe and lo- gistic regression when testing on mislabeled CoLA examples.
Latent
Last Token
Figure 10. Left: Top 3 latents (w/ descriptions) by mean activation difference between Al generated and human SAE latent activations. Right: Histograms the identity of the most common last tokens in Al generated and human text.
whose top latent representations exhibited unexpectedly strong performance. During this analysis, we discovered that both 87-glue_col .cola and 110 aimade humangpt3 con- tained labeling errors. While the SAE latents successfully highlight these errors, we observe that baseline methods are also capable of identifying them.
probe, and latent 369585 all have about the same accuracy on the clean labels. Remarkably, when we test all classifiers examples which misclassified in CoLA (where the ensembled labels disagree with the original labels), we find that the latent 369585 classifier outperforms the dense SAE probe, which itself outperforms the baseline considerably.
4.3.1. GLUE CoLA
We first examine 87 87-glue glue_ccoa, an established linguistic acceptability dataset. CoLA prompts are standard English sentences with target = 1 if the sentence is grammatically correct and 0 otherwise. Latent 369585 from the width 1M SAE has a test AUC of 0.76 and appears to fire on ungrammatical text. Surprisingly, when we look at prompts that latent 369585 fires on, those that it "disagrees" with the labels (ones that are labeled grammatical), often appear to be ungrammatical. Although we first found this result with SAE probes, we later found that logistic regression was also capable of the same identification. In Table 9, we show the sentences that a baseline logistic regression classifier and latent 369585 mark as ungrammatical while the label is grammatical; most such sentences are truly ungrammatical. Thus, both SAE and baseline methods lead us to hypothesize that the CoLA dataset is partially mislabeled.
4.3.2. AI HUMAN
We also investigate 110 a00dem hammandg.ummnpp33, which tests if probes can distinguish between human and CharGPT- 3.5 generated text. Latent 105150 has an AUC of 0.82 this task, yet appears to fire primarily periods and punctuation. Since this latent is syntactical and unrelated to the content of the dataset, we hypothesize represents a spurious correlation in the dataset.
We examine the final token of each prompt in the dataset and indeed find a spurious feature: Al text is more likely to end in period while human text is more likely to end in space (see Figure 10). Although we discovered this by inves- tigating SAE latents, we could reach the same conclusion with baselines: in Table 10 (Appendix J.4), we apply the logistic regression classifier to 2.8 million tokens from the Pile, and find that it is most active on punetuation tokens.
To test our hypothesis, we choose 3 LLMS GPT-4o (Ope- nAD), -laude-3.5-Sonnet-N
and Llama-3.1-4055 In- struct (Grattafiori et al., 2024)- to judge the linguistic ac- ceptability of 1000 random CoLA prompts. We take the LLM majority vote as the ensembled, clean label. This is analogous to the experiment performed in Warstadt al. (2019) that validated the CoLA dataset by ensembling the predictions of native English speakers, but with LLM judges instead of English speakers. We find that 22% of CoLA la- bels are mislabeled by this metric (higher than the 13% disagreement rate from Warstadt al. (2019)). Notably, the Table sentences are identified as ungrammatical by the ensemble while being labeled as grammatical in CoLA.
5. Assessing Improvements in SAE Architectures
While SAE probes not robustly outperform baselines. separate interesting question is whether recent SAE ar- chitectural developments have improved SAE probing per- formance. We investigate this question by examining the performance of eight different SAE architectures released in the last two years, as outlined in Table 3.
Next, we train baseline logistic regression, h 128 dense SAE probe, and the latent 369585 classifier on the original CoLA labels. We then test on a held-out set with the clean, ensembled predictions. In Figure 9, we find that the original labels, the baseline classifier, the dense SAE
We use width 16k Gemma- 2-2B layer 12 SAEs with a variety of T.0 values trained by Karvonen et al. (2024b). We create k: 16, 128 SAE probes all regimes and datasets
See Neuronpedia examples.
8

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Table 3. Timeline of SAE Architecture Improvements. Publication Date
AAUC for SAE Architecture vs. Baseline k- at Best LD Standard Conditions
SAE Architecture
ReLU (original)
October 4, 2023
Bricken et al. (2023)
ReLU (updated)
April 26, 2024
SAE -asseinee AUC
-0.05
Conerly et al. (2024)
Gated
May 1, 2024
-0.10
Rajamanoharan et al. (2024a)
TopK
June 6, 2024
Gao al. (2024)
-0.15
JumpReLU
July 19, 2024
Rajamanoharan al. (2024b)
0020
BatchTopK
July 19, 2024
RetUU (orig.)
RaLU (new)
Gated
Topk
Jump ReLU
Batch Topk
p- p-annsal
Matryoshka
Bussmann al. (2024a)
p-annealing
July 31, 2024
Architecture (In Order of Release)
Karvonen et al.
Matryoshka
December 19, 2024
Bussmann et al. (2024b)
Figure I1. We see that there is slight uptick in probing perfor- mance compared to baselines in standard conditions with more recent architectures. However, the spread of the data is consider- ably larger than this improvement.
for each architecture, and additionally k = for standard conditions. In Figure 11, we plot each SAE architecture's 16 probe performance in standard conditions. We find the average test AUC of each SAE and then take the max across LO for each SAE architecture (using single LO value is somewhat noisy, see Appendix L). This metric tests how expressive individual SAE latents are for different SAE architectures. While we see slight positive trend for probing performance with more recently released SAE architectures, the effect is not statistically significant. For plots in all regimes, see Appendix L; in some regimes there slight improvement with newer SAE architectures.
to be a wholesale critique of the SAE paradigm. Instead, we view it as a single rigorously evaluated datapoint to contextualize SAE utility and to motivate a more thorough examination of methods in mechanistic interpretability.
Limitations We study the performance of SAE probes as proxy for SAE utility. While we believe that probing perfor- mance a more effective measurement than typical SAE evaluations like reconstruction error, is still a proxy metric. We chose study difficult probing regimes to try to find cases where the inductive bias of SAE latents outweighed imperfect SAE reconstruction, but it is possible that even basis of "true" model representations would offer only a mild inductive bias advantage over traditional linear probing, We are thus excited for future work studying probing on toy models (e.g. models trained on board games (Karvonen et al., 2024c)) where the "true" model features are known.
6. Conclusion
Our evaluation of sparse autoencoders (SAEs) reveals funda- mental limitations in current SAE methodologies. Despite expectations that interpretable SAE latents would provide a competitive advantage in probing, we found no improve- ment traditional methods across multiple regimes and over 100 datasets. Some failures expose critical weaknesses for instance, SAE latents struggle to be robust to distribu- tional shifts and fail to represent complex concepts while others are more mundane- - lower L0 SAEs create worse sparse probes. More broadly, our findings highlight the need for the field of mechanistic interpretability to evaluate tech- niques with rigorous baselines. While prior work reported advantages for SAE probes, robust quiver of arrows methodology, use of stronger baselines, and evaluation on a large set of datasets demonstrate these advantages to be illusory. Even in our own analysis, initial conclusions favor- ing SAE interpretability were later overturned when proper baselines were considered. We do not consider this work
Finally, it is possible that further optimization of the SAE probe baseline might increase performance such that it beats baseline methods. For example, Gallifant et al. (2025) find that SAE probing with number of modifications (multi- token probes, binarization latents, and probing with k = full SAE dimension) beats baseline methods on safety- relevant datasets (albeit only comparing to single-token baseline probes). Additionally, we only tried logistic regres- sion SAE probes, and it is possible that other probing techniques could perform better. However, we believe main takeaway of our paper is that equivalent effort should be put into optimizing baselines. Indeed, we examine the effect of binarizing latents in Appendix M and find that does not significantly help.

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Acknowledgments
tions on paltern analysis and machine intelligence, 35(8): 1798-1828, 2013.
This work was conducted as part of the ML Alignment Theory Scholars (MATS) Program. We would like to thank the members of the Tegmark group and Neel Nanda's MATS stream for helpful comments and feedback. JE was supported by the NSF Graduate Research Fellowship (Grant
Bereska, L. and Gavves, E. Mechanistic interpretability for ai safety-a review. arXiv preprint arXiv:2404. 14082, 2024.
No. 2141064).
Bills, S., Cammarata, N., Mossing, D., Tillman, H., Gao, I.., Goh, G., Sutskever, I., Leike, J., Wu, J., and Saunders, W. Language models can explain neurons language models. https: // /oppeaappbbli..cc blob. core. windows.nett html,
Impact Statement
We are motivated by increasing our understanding of lan- guage models in our study of probing. There are many positive social impacts of such increased understanding of models, including detecting deception, bias, and misalign- ment. We do not foresee any negative consequences of our work.
2023.
Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense natural language. Thirty-Fourde Fourth AAAI Conference on Artificial Intelligence, 2020.
References
Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif, E., Viégas, F., and Wattenberg, M. An interpretability illusion for bert. arXiv preprint arXiv:2104.071|1 2021.
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Ad- vances in neural information processing systems, 31, 2018.
Braun, D.,, Taylor, J., Goldowsky-Dii N., and Sharkey,
L. Identifying functionally important features with end- to-end sparse dictionary learning, 2024. URL https:
AI. T. and Ishii D. Classification //www. kaggle, httpe//ccttttttttteeeeeeeee ai/ spam ppam-text--esseeeee teeaa--eexttmmessee-llasssieeeee cessed 21-01-2025).
Spam Text Message https:
kaggle.com.
//arxive //arxiv.org/aber. 2405. 240..12241.
[Ac-
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly, T., Turner, N., Anil, C., Denison, C., Askell, A., Lasenby, R., Wu, Y., Kravec, S., Schiefer, N., Maxwell, T., Joseph, N., Hatfield Dodds, Z., Tamkin, A., Nguyen, K., McL.en, B., Burke, J. E., Hume, T., Carter, S., Henighan, T., and Olah, C. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https:/transformmrr bicuitsspu/222233mooeeemma feattateieeeettttts/neee... html.
Alain, G. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644, 2016.
AllenAI. allenai/hasic arithmetic Datasets Hugging Face huggingface.co. https: htt.g//ggggggccc.eee/eeoc/a httgs///oocottet/ggggggggg/ccc co/ datasets/ cessed 21-01-2025].
[Ac-
Bricken, T., Marcus, J., Mishra Sharma, S., Tong, M. Perez, E., Sharma, M., Rivoire, K., and Henighan, T. Using dictionary learning features as classifiers, October 2024a. URL https: Accessed:
Mishra-Sharma,
Anthropic.
Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku, 2024. URL https: htt:://www.t/.. anthropic com/ news// wta.rddee---ddd--oouee------- 2025-01-30.
Faatures-s--caassfffeer//iooe html. Accessed: 2025-01-23.
Arora, Li, Y., Liang, Ma. T., and Risteski, A. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computa- tional Linguistics, 6:483 495, 2018
Bricken. T. Marcus, J. Mishra-Sharma S., Tong. M., Perez, E., Sharma, M., Rivoire, K., and Henighan, T. Using dictionary learning features as classifiers, October 2024b. URL https:
Ben Zhou, Daniel Khashabi, Q. and Roth, D. "going on vacation" takes longer than "going for walk": A study of temporal commonsense understanding. In EMNLP, 2019.
/ featuree--a--olassiiiierrr// http:/iineeeeel........ html. Transformer Circuits.
Bengio, Y.. Courville, A., and Vincent, P. Representation learning: review and new perspectives. IEEE transac-
Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders, 2024a.
10

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Bussmann, B., Leask, Learning multi-level features with ma- tryoshka saes.
Elazar, Y., Ravfogel, S., Jacovi, A., and Goldberg, Am- nesic probing: Behavioral explanation with amnesic coun- terfactuals. Transactions of the Association for Com- putational Linguistics, 9:160-175, March 2021. ISSN com/pose - www.eeel-eeeeeeeeeeeeeeeeee wittssswwtthhemee th-mtthhaatammm r y 10002200---78 doi: 10. 1 12.11622aallaa_000555 URL http:
P. and Nanda, N.
2024b. Accessed: 2025-01-23.
//dx. doi. org/ 10.
Chalnev, S., Siu, M., and Conmy, A. Improving steering
Engels, J.. Michaud, E. J., Liao, I. Gurnee, W., and Tegmark, M. Not all language model features are lin- ear, 2024a. URL https::///// /aaxivv.vv./../////a org/ org/abs/2400 ab052405. 14860.
vectors by targeting sparse autoencoder features. arXiv 2024.
preprint arXiv:2411.022
Chanin, D., Wilken-Smith, J., Dulka, T., Bhatnagar, H., and Bloom, J. for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. 14507, 2024.
Engels, J., Riggs, L., and Tegmark, M. Decomposing the dark matter of sparse autoencoders. arXiv preprint arXiv:2410. 14670, 2024b.
Chaudhary, M. and Geiger, Evaluating open-source sparse autoencoders on disentangling factual knowledge in gpt-2 small. arXiv preprint arXiv:2409. 04478, 2024.
Falgunipatel19. Medical Text Dataset -Cancer Doc Classification kaggle.com. attps://wwi
Chen, T. and Guestrin, C. Xgboost: A scalable boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Dis- covery and Data Mining, KDD '16, pp. 785-794, New York, NY, USA, 2016. Association for Comput- ing Machinery. ISBN 9781450342322. URL
[Accessed 21-01-2025].
Farrell, E., Lau, Y.-T., and Conmy, A. Applying sparse autoencoders to unlearn knowledge in language models, 2024.
doi: 10.1145/
9939672.2939785 1145 293.7777...99
Gaggar, R., Bhagchandani, A., and Oza, H. Machine- generated text detection using deep learning, 2023.
clmentbisaillon.
fake-and-rall-new---atasstt
Gallifant, J., Chen, S., Sasse, K., Aerts, H., Hartvigsen, T. and Bitterman, D. Sparse autoencoder features for classifications and transferability. 2025.
kaggle.com. com/datasets/ htt.aastttecneeetrbaaallllle...e
Fake- -nw--nndd---aa -ew--andd-----------e-------- nww. -ews--etsssd..s......
[Accessed
21-01-2025).
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2100 2tti::2010.0000 101.00027, 2020.
Conerly, T. Templeton, A., Bricken, T., Mar- cus, J. and Henighan, T. Circuits Updates April 2024 transformer-irrcuisspu wtw.pr...2
https:
//trannffomer--c riuitss
rsuits. pub/2024/
index. html www.lttaaiiinng-eteeeet saes, 2024. [Accessed 15-02-2025).
Gao, L., la Tour, T. D., Tillman, H., Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, Scaling and evaluating sparse autoencoders, 2024. URL https: //arxiv. org/ abs /t.b/////..6 240. 04093.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, L. Sparse autoencoders find highly inter- pretable features in language models. arXiv preprint arXiv:2309.086000 2023.
Gerami, S. AI Vs Human Text - kaggle.com. https://wet kaggle. com kaggle..com/dataseest data/atssssss////// shanegerami www.--vaantteee/////aeee [Accessed 21-01-2025..
Davidson, T. Warmsley, D., Macy, M., and Weber, I. Auto- mated hate speech detection and the problem of offensive language. In Proceedings of the 1 11th International AAAI Conference on Web and Social Media, ICWSM 17, pp. 512-515, 2017.
Goh, A.
IT Service Dataset - kaggle.com. kaggle tt=serv.to-ttictte--ccssssftaaa. [Accessed 21-01-2025].
Ticket Classification
https://wws.
songoh/
Dublish, T.
Text classification documen- kaggle.com. kaggle. com
tation
https:/ :/www.
text text-cland datasstts/tanishaeeeaeee
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., al. The laaaa herd models, 2024. URL https: / //arxiv.. //taa/i.rrgaaas///////oooooo 2200. 21783.
[Accessed 21-01-2025).

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Gupta, P. kaggle.com. https:/ // http//ww.kgggg. http://wwwwkkg.l.....// kaggle.
Emotion
Detection
from Text
Karvonen, A., Wright, B., Rager, C., Angell, R., Brinkmann,
J., Smith, L.., Verdun, C. M., Bau, D., and Marks, S. Measuring progress in dictionary learning for language model interpretability with board game models, 2024.
com/ com/datased /tt///aatttts///e/eeeee/ http:ppassppppppaaa//ap
emot wetetttooooo-o-tteeee---e from-text. 21-01-2025].
[Accessed
URL https://axiiv. org/abs/2408.0m 13, pp. 16.
Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
Karvonen, A., Pai. D., Wang. M., and Keigwin, B. Sieve: Saes beat baselines
real-world task (a code generation case study). Tilde Research Blog, 12 2024a. URL https:/ htt:://wwww //ww. ti lereseaarch. com/blog/ sieve. Blog post.
Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations, 2024. URL https: openreview.oo net/ forum? id=jE8xbmvFinn
Karvonen, A., Rager, C., Lin, J., Tigges, C., Bloom, J., Chanin, D., Lau, Y.-T., Farrell, E., Conmy, A., Mc- Dougall, C., Ayonrinde, K., Wearden, M., Marks, S., and Nanda, N. Saebench: A comprehensive benchmark for sparse autoencoders, December 2024b. URL https:// www nww.oneddo.ciaaa.o beneaaaeeeeeeeeeee/eeeee//aaac info. Ac- cessed: 2025-01-20.
Gurnee, W., Nanda, N., Pauly, M., Harvey, K., Troit- skii, D., and Bertsimas, D. Finding neurons a haystack: Case studies with sparse probing. Transac- fions Machine Learning Research, 2023. ISSN 2835- 8856. URL https:://// http::///ppeeeeeeeeeeee forrummmm? id=JYs1R9IMJI.
forum?
Karvonen, A., Wright, B., Rager, C., Angell, R., Brinkmann, J., Smith, L., Verdun, C. M., Bau, D., and Marks, S. Measuring progress in dictionary learning for language model interpretability with board game models. arXiv preprint arXiv:2408.00112 2024c.
He, Z.. Shu, W., Ge, X., Chen, L., Wang, J., Zhou, Y., Liu, F., Guo, Q., Huang, X., Wu, Z., Jiang, Y.GG,, and Qiu,
X. Llama scope: Extracting millions of features from 1lama-3.1-88 with sparse autoencoders, 2024.
Kasaraneni, C. K. Medical Text kaggle.com.
Heap, T., Lawson, T., Farnik, L., and Aitchison, L. Sparse autoencoders can interpret randomly initialized transform- ers, 2025. URL https: httpppp///aaxxx.oo//a//tteo org.as///r///////////////.//s 17727.
https: //ww://wwwww. kwggle com/ datasets/asss/eeeeeooe chaitanyakck/ wtt.kkkkk/eiiial-ttte... 21-01-2025].
[Accessed
Kotari, R. rkotari/clickbait Datasets at Hugging Face huggingface.co. https: https://huginggace.. co/ [Accessed 21-
Heinzerling, B. and Inui, K. Monotonic representation of numeric properties language models. arXiv preprint 1rXiv:2403.10381, 2024.
01-2025).
Hendrycks, D., Burns, C., Basart, S., Critch, A., Li, J., Song, D., and Steinhardt, J. Aligning ai with shared human values. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Lachapelle, S., Delcu, T., Mahajan, D., Mitliagkas I., Ben- gio, Y., LLacoste-Julien S., and Bertrand, Q. Synergies be- tween disentanglement and sparsity: Generalization and identifiability in multi-task learning. In International Con- ference Machine Learning, pp. 18171-18206. PMLR. 2023.
Higgins, I, Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. Towards defi- nition of disentangled representations. arXiv preprint arXiv:1812.0225: 2018.
Li, Y., Michaud, E. Baek,
D., Engels, J., Sun, X., and Tegmark, M. The geometry of concepts: Sparse autoen- coder feature structure. arXiv preprint arXiv:2410. 19750, 2024.
Huang, J., Wu, Z., Potts, C., Geva, M., and Geiger, A. Ravel: Evaluating interpretability methods on disentan- gling language model representations. arXiv preprint 17X1v:2402.17700, 2024.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L.,
Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once gemma 2, 2024.
Johannes Welbl, Nelson F. Liu, M. G. Crowdsourcing mul- tiple choice science questions. 2017.
Lin, J. Neuronpedia:
Interactive reference and tooling
Joshi, S., Dittadi, A., Lachapelle, S., and Sridhar, D. Identi- fiable steering via sparse autoencoding of multi-concept shifts. arXiv preprint arXiv:2502. 12 2025.
for analyzing neural networks, 2023. URL https: / h/www. www.neeroopediaa. .org Software available from neuronpedia.org.
12

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 3214- 3252, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10. 10.186553/1/2222.achhaa 229. URL https: https://aclantteeeeeee http://caaaaoooooorggg////////// acl- 229/.
Pang, B. and Lee, L. Seeing stars: Exploiting class relation- ships for sentiment categorization with respect rating scales. In Proceedings of the ACL, 2005.
Paulo, G., Mallen, A., Juang, C., and Belrose, N. Automati- cally interpreting millions of features in large language models. arXiv preprint arXiv:2410. 13928, 2024.
Rajamanoharan, S., Conmy, A., Smith, L.. Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders. arXiv preprint arXiv:2404. /6014, 2024a.
Lin, Z., Wang, Z.,, Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy, A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024b. URL https://arxiv.rr abs 0///2240..4445
MacDiarmid, M. Maxwell, T., Schiefer, N., Mu, J., Kaplan, J., Duvenaud, D., Bowman, S.. Tamkin, A., Perez, E., Sharma, M., Denison, C., and Hub- inger, E. Simple probes can catch sleeper agents, 2024. URL. https: /ttp:://www... http://wwww.tttttg..eete news news/probes ccacch- sleeper-aet agents..
Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesnard, T., Shahriari, B., et al. Gemma 2: Improving open language models a practical size, 2024.
com/
Marks, S. and Tegmark, M. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824, 2023.
Rogers, A., Kovaleva, O., Downey, M., and Rumshisky, A. Getting closer to Al complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Con- ference Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8722- 8731. AAAI Press, 2020. URL htttt:t///// attpa//aaatttaaa/aaa.acoommmmm o j iindi.nixi...mm php/ AAAI CaAAAA/artillee http:/e..........aaaaccceee////a
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint ar 2rX3 2024.
Mudide, A., Engels, J., Michaud, E. J., Tegmark, M., and de Witt, C. S. Efficient dictionary learning with switch sparse autoencoders, 2024. URL https::/ //rxiv.. 2410. 08201.
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L., Goldowsky-Dill N., Heimersheim, S., Ortega, A., Bloom, J., Biderman, S., Garrig--lonso, A., Conmy, A., Nanda, N., Rumbelow, J., Wattenberg, M., Schoots, N., Miller, J., Michaud, E. J., Casper, S., Tegmark, M., Saunders, W., Bau, D., Todd, E., Geiger, A., Geva, M., Hoogland, J., Murfet, D., and McGrath, T. Open problems in mechanistic interpretability. 2025. URL httpp:/// http///axvvv/../////////////aa.. org abs 1691.
Mur, M., Bandettini, P. A., and Kriegeskorte, N. Re-
vealing representational content with pattem-information fmri an introductory guide. Social cognitive and affec- live neuroscience, 4(1):101-109, 2009.
Nanda, N., Lee, A., and Wattenberg, M. Emergent linear rep- resentations in world models of self-supervise sequence models. arXiv preprint arXiv:2309.0099 2023.
Smith, L. R. and Brinkmann, J. Interpreting pref- erence models w/ sparse autoencoders, July 2024. URI. https: htop::://ww... wwww a inmeet
Olmo, J.. Wilson, J., Forsey, M., Hepner, B., Howe, T. V., and Wingate, D. Features that make a difference: Lever- aging gradients for improved dictionary learning, 2024. URL
//t////axiic /tt///iivvv.. BRRlhhttss:///xxiv. wwtp:/wwwwgg/b///////// 10397.
rreferreceemooells--ppreee \ \pena ty 23.
Accessed: 2025-01-
OpenAl. Hello GPT-4o. https:: com/ index/hello gpt-4o/ [Accessed 27-01-2025).
Stathead. Stathead: Your all-access ticket to the Sports Reference database. - Stathead.com - stathead.co com. https://statien com/. [Accessed 21-01-2025).
OpenAI. Introducing openai ol. https::/ h/tpppsi////a......// com/ol/, 2024. [Accessed 29-01-2025).
13

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Suter, R., Miladinovic, D., Schölkopf, B., and Bauer, S. Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning, pp. 6056-
Representation engineering: A top-down approach to ai transparency. arXiv preprint arXtv:2310.01405 2023.
6065. PMLR, 2019.
Tafjord, O., Gardner, M., Lin, K., and Clark, P. "quartz: An open-domain dataset of qualitative relationship ques- tions". "2019".
Talmor, A., Yoran, O., Bras, R. L., Bhagavatula, C., Gold- berg, Y., Choi, Y., and Berant, Commonsenseqa 2.0: Exposing the limits of ai through gamification. arXiv preprint ar arXiv:2201.053333
2022.
Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Freeman, C. D., Sumers, R., Rees, E., Batson, Jermyn, A., Carter, S., Olah, C., and Henighan, T. Scaling monosemanticity: Ex- tracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL https:
cir scaling- www.emmattiiiiitooooooooooo html.
Van Steenkiste, S., Locatello, F,, Schmidhuber, J., and Bachem, O. Are disentangled representations helpful for abstract visual reasoning? Advances in neural infor- mation processing systems, 32, 2019.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https:t//peereeeeeeeeeee nett/orrum id=:J4km2R5t7.
Warstadt, A., Singh, A., and Bowman, R. Neural network acceptability judgments. Transactions the Association for Computational Linguistics, 7:625-641, 2019. doi: 10. 2t22/taclaa000290 URL /tLhttss://aclaaeee hat.a/thoaaac....c.. org/019-10400
Wu, Z., Arora, A., Geiger, A., Wang. Z., Huang. Juraf- sky, D., Manning, C. D., and Potts, C. Axbench: Steer- ing Ilms? even simple baselines outperform sparse toencoders, 2025, URL http:://///ax org/ abs/ 17148.
Yun. Z., Chen, Y., Olshausen, B. A., and LeCun, Y. Trans- former visualization via dictionary learning: contextual- ized embedding linear superposition of transformer factors. arXiv preprini arXiv:2103. /5949, 2021.
Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., Pan, A., Yin, X., Mazeika, M., Dombrowski, A.-K., al.
14

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
A. Related Work
A.1. Probing
Probing has a rich history in computational neuroscience, where linear decoders were used to study information representation in biological neural networks (Mur et al., 2009). This technique was later adapted to study artificial neural networks by Alain (2016). Since then, probing has become a fundamental tool in neural network interpretability, revealing that many high-level concepts are linearly represented in model activations (Gurnee & Tegmark, 2023; Heinzerling & Inui, 2024; Nanda et al., 2023). Similar to our work studying sparse probing, Gurnee et al. (2023) study sparse probing of activations and identify individual monosemantic neurons by setting k- = 1. Recent work has also used probing to study safety- rafety-relevant properties of language models, such as truthfulness (Marks & Tegmark, 2023) and the presence of sleeper agents (MacDiarmid et al., 2024), without relying on potentially unreliable model outputs. Two recent works have investigated the utility of SAEs for probing. First, Bricken et al. (2024a) investigate a synthetic bioweapons dataset and show that SAE probes can sometimes offer an advantage against baseline probes when aggregated across multiple tokens; we discuss these results in our section on multi-token probing in Appendix K. Second, Gallifant et al. (2025) use feature binarization, multi-token feature pooling, and probing on the entire SAE vector to report that SAE probes outperform baselines. We discuss Gallifant et al.'s (2025) results in our limitations section and in Appendix M.
A.2. Challenges in Neural Network Interpretabiliiyy
Our work connects to broader concerns about the reliability of neural network interpretation methods. Adebayo et al. (2018) studied saliency methods, a classical technique for understanding models, and found that randomizing the model and dataset labels did not change many aspects of saliency maps; thus, while the method produced plausible-looking explanations, they were not faithful to the true model and dataset. Similarly, (Bolukbasi et al., 2021) demonstrated that seemingly interpretable neurons in BERT were artifacts of running on only a particular dataset rather than more general representations. For SAEs specifically, (Chanin et al., 2024) identified feature absorption and splitting as fundamental challenges, Gao et al. (2024) showed that SAEs have an irreducible error component, and in extremely recent work Heap al. (2025) show that SAEs trained random models also result in interpretable features. Our work extends these critiques by finding that many settings where SAE probes were thought to be helpful turn out not to be when compared to stronger baselines.
A.3. SAE and SAE Applications
Sparse autoencoders (SAEs) provide a map from model activations to sparse, higher dimensional latent space (Bricken et al., 2023; Cunningham et al., 2023). Individual latents are hypothesized to represent mono-semantic concepts LLMs use for computation. An SAE is parameterized by its width, or the dimension of its latent space, and its LO, or how many latents are nonzero on average. While our work focuses on evaluating SAES as probing tools, prior work has explored various downstream applications of SABs. Templeton et al. (2024) first used SAE latents for steering, and in follow-up work Chalnev et al. (2024) found that SAES can help find better steering vectors (although see (Wu et al., 2025) for a very recent work finding that SAEs are not competitive for steering). Karvonen et al. (2024b) implement a set of comprehensive benchmarks for evaluating SAE performance, including SHIFT (Marks et al., 2024), sparse probing, unlearning, and feature absorption. Recent work has also used SAEs to interpret preference models (Smith & Brinkmann, 2024) and prevent unwanted behavior in model output (Karvonen et al., 2024a), both specific applications where SAES seem to be state of the art.
B. Disentangled Representations
An interesting connection of our work is to the field of disentangled representations, which studies representations that are identifiable. That is, disentangled representations have individual dimensions (or small groups of dimensions) that correspond to independent parts of the input, such that changes to other parts do not change that part of the representation (Bengio et al., 2013; Suter et al., 2019; Higgins et al., 2018). These disentangled representations are similar to sparse autoencoders latents or language model features as hypothesized by the linear representation hypothesis, and in fact prior work directly examines this connection (Joshi et al., 2025). Past work finds that disentangled representations are sometimes helpful for downstream tasks, improving sample-efficiency ( (Van Steenkiste et al., 2019) and generalization in classification (Lachapelle et al., 2023). Unlike these works, we find minimal benefits of plain SAE representations, perhaps implying that they are not disentangled enough, or that the probing setup w differs from these past works.
15

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
C. Classification Datasets
Below we list in a (large) table the classification datasets we use, as well as their source. Note that datasets 161-163 are modified from their source -a mistake in our formatting reframes them as differentiating between news headlines and code samples.
Table 4: Binary classification tasks used.
Citation
Dataset Name
Gurnee & Tegmark (2024)
s_hhst__fue fiist_figiismale
hist. figss.iig... isamerican
hist fig ispolitician
21 whhadline.istruuu
22 headline isobama
23 hiechlineiischnaa
24 headline isiran
26 So.hhedlinee.ie dipilotpiis
14 |l_ borough Manhattan
115_nyc_borough BBookkynn
116.nyc.brrough Bronx
117 _t7atesttate. _L
118 11__u_sstate_C.
119 ____utttte_TXX
120 _us wwt.imeeone.hhieee
121 us timezone New York
122.us wtt.uusstmezzee..om www.eioon..ossaAaeee Ans_Aellss
123. /23_world. countryy country-United United Kingdom
www.dd Sinitry__nitedSSttte Staaes
125 world country Italy
126_art.ng .tt..rtttypebbokkk
227_artttyee_so
".s.artttype_movie
Johannes Welbl (2017)
36 36_sciq.1f
Lin et (2022)
41 truthqa.tf
Ben Zhou Roth (2019)
42.temp_sense
130 temp Frequency
131 Lelpmecctt_Tppice Time
132 temp cat Event Ordering
Bisk et al. (2020)
44_ 44_phys_tff
Tafjord et al. ("2019"))
47 reasoning tf
Hendryeks al. (2021)
48 em correct
49 cm isshort
50
51.just_is
52 virtue is
Talmor et al. (2022)
54_cs_tf
Gurnee et al. (2023)
wikidatasex or gender aalve
58 wikidatapolitiiall party
19__wiidata
isjournalist
itt.iiaatt.occupatiion...o
61 wwwidataa ittpiiaata_occppttiioooooo _isactor
62 www.oocccccooisooooooooocc
63 wikidata occupation issinger
Continued on next page
16

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Citation
Dataset Name
64 wikidata occupation isresearcher
65 hiihh-shhhooo
1o_living-room
-essucial-eecurity
68 credit-card
69 www.olodppsssure
70 70-prime-factorn
71 social-media
72. 77-gene-expresion
73 control-group
74 14_maanetii-fe
75 cell--inees
76 trial-court
77 second-derivatee
78_ 3._orth-america
79 humman--rights
80 side-effects
81.public-health
goveemmell
83 83_third-party
84 clinical-triaas
85 meental-health
Wang et al. (2019)
87 $7-glue_cola
89 199glue_mrpe
glue.qnli
glue.qqp qqp
92-glue_sst2
136_glue_mg
337_glue_mmli.neute
138 glue mnli contradiction
Gerami
94_ai-gen
Lin et al. (2023)
95_toxic.is
& Ishii
96_spam_is
clmentbisaillon
100 news fake
Kotari
105 105_click_bel blick_bait
Davidson et al. (2017)
107 hate offensive
Gaggar et al. (2023)
1 10 aimade huanangtt
Pang & Lee (2005)
113 movie sent
AllenAI
129_ani 119_arth_mm.4
Rogers et al. (2020)
133.com 333_context.o.m coneext.type
Sype-Causality
134_com coontexttyyeeBellefff
135 context type Event duration
Dublish
139 199_newssclass_Poliiceo
141 news class Entertainment
Falgunipatel19
143_ 333_caneer_ca LaieeaatLLnnngggggeege ManngCCnncer
144 Sancer_cat_Colon.co. Canneer
Kasaraneni
145 disease class digestive system diseases
147 diseass.cass.carrtivvsuuull diseases
146 disease.clllie blaease_class... cardiovascular
system diseases
Continued on next page
17

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Citation
Dataset Name
Gupta
148 twt emotion worry
149 www.emmtioon.aapinee
150 150.ttt.emolie www.ttttwetton...m sadness
Goh
151 _it_ticck_HR Support
152 it tick Hardware
153 it ittiittckk Administtttte rights
Stathead
154 1e4_athcccee.soorffoobbb
155 athlete sport basketball
156 Seatlete.sport.baaeee
Karvonen et al. (2024b)
157 amazon 5star
158_code_C
159 codeceee... Python
160_code_.. HTML
161 agnews 0
162.agnews_
163 Lannensss
D. Probing Setup
AUC
The Area Under the Curve (AUC) quantifies the overall performance of a binary classifier using the Receiver Operating Characteristic (ROC) curve. An ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels, illustrating the trade-offs between correctly predicting positives and incorrectly predicting negatives. The AUC, ranging from 0 to 1, measures the entire two-dimensional area beneath this curve. An AUC of 1.0 signifies perfect classification, 0.5 indicates performance no better than random chance, and closer implies poor elassification. This metric provides single, aggregate measure of performance across all possible classification thresholds, making it particularly useful for comparing different classifiers.
Technically, when choosing k and the baseline SAE to use for the other section, we use test AUC on normal datasets, which constitutes slight leakage of test data. However, we used trends observed over a large number of datasets, in the same way we might provide heuristics for constructing SAE probes to future practitioner.
D.2. Probing Method Validation Details
For each strategy, we use hpp that has the maximal average AUC across held out validation sets, AU We choose a validation method from Table 5 based on the dataset size n (most of the time this the last row the table for large n, except for the low data regime tests).
Table 5. Selection methods to choose hyperparameters hp when training on different dataset sizes
Data Size (n)
Selection method for probe p Train p with each h, on all points;
choose hp which maximizes AUC training set.
3
12
Use leave-two-out cross validation; train with each hp on all training splits of size n and evaluate on the last held out points
12 < 12 < 128
Use 6-fold cross validation; split n in sixths, train p with each hp on all sets of splits, and evaluate on the remain- ing fold
128
Use 80%/20% training/validation split
18

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Performance by Method Layers
Best Performing Method by Layer with Test AUC
0.95
Logistic Reg.
0.90
Number of Datacets
100
PCA
KNN
80
XGBoost
Logistic Reg.
MLP
60
Test
PCA
0.75
KNN
40
XGBoost
0.70
20
Combined Mean
0.65
embed
20
31
41
embed
20
31
41
Layer
Layer
(a) Layer 20 is best for baselines.
(b) Logistic regression is often the most performant across datasets.
Figure 12. Analysis of layer performance and methods. (a) Shows the optimal layer for baseline performance. (h) Demonstrates the effectiveness different methods across layers.
D.3. Probing Method Hyperparameter Details
In this section, we list each baseline method and the hyperparameters that we search over for that method.
Logistic Regression:
- C: Ranges logarithmically from 10 to 10-5. The Lz/L, regularization is 1/C.
PCA Regression:
Number of PCA Components: Varies logarithmically from up to the minimum of the number of samples, latents, 100.
K-Nearest Neighbors (KNN):
Number of Neighbors: Logarithmically spaced values up the smaller of 100 or the number of samples minus one.
XGBoost:
n estimators:
Ranges from 50 to 250 in steps of 50.
max_depth: Ranges from
dearning-rate rate: Ranges logarithmically from 0.001 0.1.
subsample and colsample bytree: Range from 0.7 to 1.0.
reeaalpha and reg lambaa: Range logarithmically from 0.001 to 10.
miincchild_weight: Ranges from to 9.
Multilayer Perceptron (MLP):
Network depth: 1 to 3 hidden layers
Hidden layer width: 16,32, or 64.
learning haariinggrateiiie Five values ranging logarithmically from 10-4 10-2
alpha: Weight decay parameter, with 5 values ranging logarithmically from 10-5 to 1
Activation function: ReLU.
- Optimizer: Adam.
19

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
SAE Performance by Width and Sparsity Layer 20
10
0.90
0.86
Test AUC
0.84
y -..00lo2(()) 0.93
Average Test AUC
R² 0.972
0.82
0.86
0.80
0.85
10
10?
Sparsity
16
32
128
256
512
1024
Width (thousands)
(a) We find that important in creating effective probes, while width plays a minimal role. Test AUC averaged over all values of k and datasets.
(b) When averaging over other factors, smaller width SAEs per- form best as probes. However, the relationship is minimal.
Figure 14, Analysis SAE width L. regularization effects on probe performance. (a) Shows the Pareto frontier of L.0 width trade-offs. (b) Illustrates the relationship between SAE width and probe performance.
D.4. Additional Discussion of Quiver of Arrows
We note that adding batch of SAE probes as additional arrows to the quiver is potentially disadvantageous for SAE probes. This is because using multiple SAE probes presents additional opportunities for these probes to overfit to the training data and chosen by the quiver of arrows approach without properly generalizing to the test data. Nevertheless, the quiver of arrows is still appropriate because a proper counterfactual practitioner would face. Additionally, the quiver of arrows approach is not the reason we find negative results for SAE probes. For instance, Figure 1, we directly compare the test AUC of the most performant SAE probe and baseline in standard conditions across all regimes, Without using the quiver of arrows approach, still find SAEs underperform baselines.
E. Normal Conditions
0.94
Layer Layer 20
E.1. Cross Layer-wise Comparisons
0.92
Average Test AUC
Layer 31
On Gemma, we see that layer 20 best for baselines, so we choose to use that layer moving forward (Figure 12a).
0.90
Layer 41
0.88
Logistic regression most often has the highest test AUC across datasets in these conditions layer 20 (Figure 12b).
0.86
Finally, in Figure 13 examine the average AUC for
0.84
the width 16k SAEs with largest LO and again find that probes at layer 20 do the best.
0.82
0.80
E.2. Choosing which SAEs to Test
10º
101
10a
When constructing pareto curve based the width and
LO of all available SAEs at layer 20 of Gemma-2-9B, we find that width is unimportant, while constructing probes from higher LO seems to improve probe performance (Figure 14a).
Figure 13. Mean test AUC, largest by width 16k SAFs across layers.
20

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
To ensure that width plays minimal role, we average over all datasets, k, and LOs in Figure 14b. There is clear negative trend indicating that smaller SAE widths better, but with almost 0 slope. Because we aim make few decisions based off the test data possible, we consider three widths throughout our experiments, 16, 000, 131, 000, and 1, 000, 000.
E.3. Plotting Dataset Performance vs. K
In Figure 15, show the performance all datasets the number of latents we train the sparse probe Almost all datasets are monotonically increasing in k; some have sharp increase at some & value, but most increase relatively smooththly.
F. Additional Results for Various Regimes
E.1 Data Scarcity
In the data scarce setting, we see that we choose mostly the SAE method from the quiver at smaller training fractions (Figure 17a). This is because we break ties by explicitly preferring the SAE method, and most methods have perfect validation AUC at small training fractions. Specifically, first prefer the smallest width SAE, and then the SAE with the largest value of k. In Figure 16, we visualize the improvement by using the quiver of arrows approach with SAE probes across all datasets.
F2. Class Imbalance
We show the improvement for all datasets in Figure 16 across all class imbalances. Generally, the SAE method is chosen at more extreme imbalance ratios (Figure 17b)
F.3. Label Corruption
We show the performance of the SAE across datasets with all values of label corruption. Interestingly, the SAE still has highest test AUC at many places (Figure 17c shows the percent of time that the SAE test AUC is larger). However, it is clear from the per dataset imshow (Figure 16) that it struggles with more label noise.
G. Intuition for SAE Probe Utility Across Regimes
SAEs aim to construct a more interpretable basis for language model activations, with the latent space hypothesized encode semantically meaningful and disentangled features. We argue that if SAEs successful at this task, well-generalizing activation probes should be represented by just a few latents, and so requiring probe to only use sparse set of directions in this basis should serve as a beneficial inductive bias. Specifically, in setting where it is hard for dense activation probes to learn the correct direction, we argue that this inductive bias may especially help. We describe additional intuition for each regime below:
Data Scarcity: Limited data makes it difficult to determine the correctly generalizing probe direction; restricting the probe a few "true model variables" seems like a helpful inductive bias.
Class Imbalance: Because SAE latents are sparsely activating, choosing SAE latents that are positive on the minority class and negative on the majority class may generalize well.
Label Noise: SAES have a sparse basis of meaningful latents, SAE probes should be more robust to overfitting to random, noisy labels because these random labels will not be consistent with any of the latents.
Covariate Shift: If SAE latents indeed use limited set of abstract, model-internal concepts rather than dataset-specific, superficial features, then SAE probes that use sparse set of these latents should generalize more robustly under distributional shifts in the input.
Conversely, there are a few reasons to suspect SAE probes might not be as performative as baselines. Even if SAE latents were true-to-model, abstract representations, SAEs ultimately lose information from baseline activations, leading to a disadvantage in probing. Additionally, we bound k, or the number of latents used in a probe, to 128, when some
21

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
hist fig isimale
fig samericaa
headline ischina
headline isiran
truthqa
Ispolitician
Istrump
isobaraa
isfronipage
36 sciq
56 wikidatasex or gender
12 temp sense
47 reasoning
48 cm correct
49 cm Isshort
50 deon isvalid
44 phys tt
just
52 virtue Is
54 cs
Th THE Tm
The Tm
TIm
Ti TM
The ThE
58 wiliidatapolitical party
59 wikidata accupation Isjoum
60 wikidala occupation
61 wikidata accupation
62 wikidata accupation isolit
wikidala occupation Issinge
64 wikidata accupation isresea
wikidatals
65 high
66 living room
social security
68 credit card
bload pressure
prime factors
71 social media
72 gene expression
73 control group
74 magnetic field
cell lines
trial court
77 second derivative
78 north
79 human rights
8D side effects
81 public health
82 fedieral govermeet
83 third party
84 clinical trials
85 mental health
87 glue
news fake
glue mrpp
glue qnli
91 glue qqp
92 glue ss12
94 gen
95 Laxic
96 spam is
bait
hate
Test AUC (0.5 1.0)
114 nyc borough Manhatlan
115 nyc borough Brooklyn
120 us timezone Chicago
107 hate offensive
110 almade humangpta
113 movie sent
116 nyc barough Bronx
117 us state FL
118 us state CA
119 state TX
Tim THE TM
Time
The TM
121 timezone New Angeles
122 timezone
123 world country United Kingd
124 world United
125 warld country Italyy
126 art type
type
128 type
129 arith
130 temp Frequency
song
133 context type Causality
134 context lype Belief states
135 context lype Event duratio
136 glue mnli entallment
138 glue mnli contradiction
139 news class Politics
140 news class Technology
131 temp cat Typical Time Event Ordering
132 temp cat
137 glue mnli neutral
141 news class Entertainment
142 cancer cat Thyrold Cancer
143 cancer cat Lung Cancer
144 cancer cat Colon Cancer
145 disease class digestive
146 disease class cardiovascul
147 disease class nervous
149 twt emation happiness
150 twt emotion sadness
148 twt emotion worry
The
Fim
154 ath lete sport
155 athlete spart basketball
156 athlete sport baseball
tick Hardware
amazon 5star
Support
Administrative rig
code
Python
HTML
161 agnews 162 agnews 1 163 agnews
THE
Number of Features in I (1, 2, 4, 8, 16, 32, 64, 128, 256, 512)
Figure 15, Test AUC on all datasets k [Z]
using Gemma Scope layer 20 SAE with width 131k and 193.
22

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
AMC intrvvmmen
-0.8
-0.6
9.8
imale
6 hirr ng tosmerican
heap sstump
24 head Istran
36 <08
42 temp sanae
reasonity
48 contect
50 deom lovald
52 Sreu
56 wik wileggtscy or gen
Wikizaca
59 wikidafa wikidstapo aopollicall Party
eccupation
G1 wile data occupatish saacoor poct
wikitain
wikisata occupalli geinger
B5 high school
67 geaal securit
pressure T0 ormm factors
12 gen e- expression praue 74 Magneth neti field
hha-cout count
19 human zghts
Publcc ealth
fed
mira earty
merla Haleeel eall
0r alu onil
92 ue sa12
tox
96
100 news dee
hate hata
107 hate ofrehsive
110 almm homanget]
& mowe sent
11 nge bardugh Brooklyn
us state
us 120 timezon e Chickap timezane
128 21 dF
mecone Lis An geles
2 word countr Unine Sates
128 srt type book
pensong
128 12 yal movie
11 temp cat loical Time
Evant 134 corfext we Causalry
135 context type Edpnt stration
138 qne gue mall neurra
contrdiciion dass
140 news dars echnology Fine
141
142 Sencer cal Thwroid Cancer
145 disease dess rancor rat colon Cancer
147. 147_diseas disease di fervous system disaa
chast fardiovattu
emoton
149 we emotion hapbiness
1s it tick strport
100
200
152
153 it tick Nomimtatra ve rights
156 athlefe baserball
sport
ball
157, amazon code
100 caar HTM
H2 sanews
Namber
Figure 16. Improvement from using SAE methods in the quiver across all datasets for various corruption ratios, ratios of positive classes, and number of training examples
23

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
100
Ier SCC Ohoin
Nu1 Exten
Legro 00000 B691
(a) SAE probes are chosen by our method small training fractions hecause multiple methods have perfect validation AUC and we break ties by choosing SAF probe.
(b) SAE probes are generally chosen at more extreme imbalances.
(c) Corruption fraction minor influ- ence on the percent of time SAE probes or logistic regression prohes are chosen.
Figure Method selection analysis under different data conditions. (a) Selection under data scarcity. (b) Selection under class imbalance. (c) Selection under data corruption.
Data Scarcity
Class Imbalance
0.9
0.950
Mea lest AUC
Mean Test AUC
0.925
0.7
0.875
Baseline Quiver
Baseline Quiver
SAES Quiver
SAEs Quiver
0.6
Quiver
Quiver
0.825
10 14 20 28 38 53 74 0110044192279333333000
0.2
0.4
0.6
0.8
Number of Training Examples
Class Imbal ance Ratio
Figure 18. In the settings of data scarcity and class imbalance, see that choosing SAE probe for each task using validation AUC performs strictly worse than using the quiver of arrows with SAEs, further confirming that the quiver of arrows approach serves as an upper bound for SAE probing performance
concepts might require more than 128 latents to properly represent. Finally, it is unlikely that SAE latents are truly general, irue-to-model representations, reducing their expressiveness as probing basis.
H. Evaluating the Validity of the Quiver of Arrows
The quiver of arrows approach we introduce is con-standard in the literature, but we adopt make the strongest possible case for SAFs. Since select the best method using validation AUC, expect choose SAEs only for tasks where they perform best. To verify this, in the three settings where employ the quiver of arrows-standard conditions, data scarcity, and class imbalance-we compare its performance to that of using single SAE across all tasks, as shown in Table 6. Clearly, the quiver of arrows serves as an upper bound on the performance of any individual SAE. As an alternative counterfactual, instead of comparing against a single SAE probe, we select the hest SAE for each task using validation AUC and compare this to the quiver of arrows. Figure 18, we observe that the quiver arrows also serves as an upper bound in the settings of data scarcity and class imbalance.
Table 6. We evaluate the effectiveness of the quiver by comparing it to the performance of choosing single SAE probe for all tasks. As
expected, the quiver SAEs serves as an upper bound on the performance of any individual SAE. Setting
Baseline Quiver
SAFs
SAEs +
LogReg
SAE 16k
SAE 16k
SAE 131k
SAE 131k
SAE Im k=16
SAE 1m k=128
Quiver
Baselines
k=16
k=128
k=16
k=128
Quiver
Standard Conditions
0.940
0,930
0.904
0,921
0.913
Data Scarcity
0.819
0.806
0.812
0.836
0.800
0.816
0.794
0.810
0.785
0.801
Class Imbalance
0.921
0.906
0,916
0.929
0.898
0.909
0.890
0.906
0.882
0.899
24

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Performance Method Across Layers
0.95
0.90
AUG
0.80
Logistic Reg.
PCA
Test AUC (with SAE)
1.0
Mean
0.001
No Change
0.75
XGBoost
0.8
0.70
MLP
Combined Mean
0.6
embed
16
24
31
0.60
0.65
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Layer
Test AUC (without SAE)
(a) When testing baseline methods at different layers of Ilama- 3.1-81 we find the middle layer 16 works best by slim margin.
(b) In standard conditions, we see that SAE probes are not statisti- cally more performant than baselines for Llama-3,1-88
Figure Analysis of Llama-3.1-88 probe performance. (a) Layer-wise performance of baseline methods, (b) Comparison between SAE and baseline probes under standard conditions.
SAE (AUC)
0.0
-0.1
25
Number Training Examples
Ratio of Positive Class
Fraction Corrupted
Figure 20, We find that Llama-3 3.1-8b SAE probes do not improve on baselines in the settings of data scarcity, class imbalance, and label
noise
I. Reproducing Core Results on Llama-3.1-8b
We use SAES provided by Llama Scope to replicate our core results, namely, that SAEs underperform baseline probes in normal, data scarce, class imbalance, and label noise settings when testing on Llama-3.1-8B. Figure 19a, we show results for baseline probes applied to various layers Llama-3. in standard settings. We see that the layer roughly halfway through the model, layer 16, is best for probing experiments. In Figure 19b, we show that SAE probes in normal settings on layer 16 of Llama-3.1-88 we did not observe in Gemma-2-9b. Lastly, we use the quiver of arrows approach to show that SAE probes are not more performant in the conditions of data scarcity, class imbalance, and label noise (Figure 20). Because of the increased effort of testing OOD and interpretability, we do not replicate those results. Note that in all experiments, we use the provided width 128k, LO 55 SAE.
not more performant than baselines, although there a few positive outliers
J. Interpretability
J.1. Pruning
We demonstrate the latent ranking procedure for the task 66 66.living-room in Table 7. We see that pruning works, for two of the tasks, 66 living-room and 90 glue qnli, while failing for 7 hist fig ispolitician (Figure 21). However, even when pruning works, the improvement is marginal compared to the decrease in performance when moving from an in-distribution test set to an OOD test set. Thus, we hypothesize that the underlying latents are not sufficiently expressive. For example, latent 122774 most active on the English translation of "living room," and not activate at all for the French translation (Figure 8). This indicates this latent does sufficiently represent the concept of "living room" to robust to OOD transformations.
25

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
0.65
OOD Test AUC
66 jo_living-roo (LLM)
66 living-room (Human)
0.55
90 go__glue_gmi gnli
hist fig ispolitician
0.50
0.45
Number Features Pruned from km8 Probe
We prune using human labeler rankings (dashed) and traditional autointerp with Claude-3.5-Sond (solid). Two datasets improve with pruning, but one does not, which indicates that more latents are required to establish the prohing target.
Figure 21.
Table Description Latent Variables Out-of- Distribution Detection. their OOD test AUC, even though ol did not have access to this information. However, the system is not perfect. For example, latent 51.30'ss natural language description is ranked last and seems unreleated to task hand. However, its OOD test AUC highest all latents. We do not investigate this further, but hypothesize that either the latent descriptions require additional tuning, or the test set truly has some spurious quality that this latent identifies.
procedure ranks latents roughly according to
Latent
ol
Latent
Val ID AUC
Test OOD
rank
Description
AUC
122774
1
mentions of living rooms parlors in houses apartments.
1.0
0.6402
98707
2
phrases and words related to loca- tions or settings, particularly indoor spaces like rooms, parlors, or living areas, as well as time references like afternoon or specific times.
0.8698
0.6177
100016
3 mentions of rooms
enclosed spaces, particularly when discussing physical locations or settings.
0.6895
0.4736
7498
4 locations within a house, particu- larly upstairs, downstairs, basement, bathroom, and kitchen areas.
0.7524
0.6206
78823
words and phrases related specific places, attractions, and experiences, particularly in the context of travel and tourism.
0.6697
0.4526
116246
6 numbers, dates, and numerical mea- surements, particularly in scientific or technical contexts.
0.7470
0.5398
40168
7
technical terms and components re- lated computer systems, engineer- ing, and scientific analysis.
0.9219
0.4513
51330
8 objects and materials related to sci- entific experiments and laboratory equipment, particularly those involv- ing fluids, particles, or microscopic samples.
0.7320
0.6483
26

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
J.2. Latent Investigation
We generate natural language descriptions of all the latents with GPT-4o. We find that most are either relevant to the task or visibly spurious. We hypothesize that some latents have incomplete descriptions given their performance on the task. We list selection of the most interesting of these latents from each category in Table 8.
J.3. Glue CoLA
In Table 9, show the top 5 examples where latent 369585 and the activation probe have the highest confidence that prompt is ungrammatical, but it is (incorrectly) labeled as grammatical in the CoLA ground truth.
J.4. Al Made
In Table 10, we show the tokens that the 110 Loa_mmdde__aii humangpt3 classifier activates across the pile with the highest average activation. Like the SAE feature, the baseline classifier has top activations on punctuation tokens.
K. Multiple tokens results
Bricken et al. (2024a) report that SAE probes outperform baselines slightly. A natural question is why our experiments fail to replicate this result. One possible explanation is that Bricken et al. (2024a)'s findings were based on single dataset, whereas we evaluate across multiple datasets. We find that SAE probes outperform baselines on only small subset of these datasets (2.2%, see Figure 22). It is possible that the dataset used by Bricken et al. (2024a) falls within this subset; however, without access to it, we cannot confirm this.
A separate explanation involves potential illusion due to insufficiently strong baselines. Unlike our work, Bricken et al. (2024a) uses multi-token SAE probing: they aggregate the maximum value for each latent across all prompt tokens, while we only use last token latents. Crucially, they similarly max-pool each model dimension across prompt tokens for their comparative baseline, even though activations are unlikely have privileged dimensions (in Table 11 we show that pooling activations in this way works poorly). We implement multi-token SAE probing using max-aggregation datasets from our list and k 128; see Table 11 for full results these datasets. We also implement better baseline: train attention-poole -pooled probes of the form
60 random
[softmax,g/1.Ctoteen (x) 43] [x'.r]
for g, e E Rdd We find that when compared to the last-token baseline, max-pooled SAE probes win 19.6% of the time, considerable improvement over win rate of last-token SAE probes (2.2%, Figure 22). However, when implementing ditention-pooles baselines and using the quiver of arrows approach to select between pooled and last-token strategies for SAE probes and baselines, the SAE probe win rate drops more than 50% to 8.7% (see Figure 22).
Finally, in Table 11, we show comparison of all methods (including the attention based probes and multi-token SAE methods discussed in Appendix K) on a random sample 60 datasets.
L. SAE Architectural Improvements
We test if improvements in SAE architectures have led improvements in probing performance. We test eight SAE architectures on Gemma-2-4 2--B in all regimes with & = 16, 128 SAE probes. We plot the performance of SAE probes for all architectures in Figure 24 when averaging over all datasets. While there seems to be some improvement with later architectures, we note that there is significant variance in the average across datasets that is not visualized.
M. Assessing SAE Probe Binarization
Gallifant al. (2025) find that binarizing latents with threshold improves probe performance. Binarization entails setting latent equal to 1 if its firing value is greater than a threshold, and setting it equal to 0 otherwise. Gallifant et al. (2025) use a threshold equal to 1. Gallifant et al. (2025) performs probing in the multi-token pooled setting, so to use as similar of a setup as possible we repeat our multi-token experiment with binarization and a threshold equal to 1. Following Gallifant et al. (2025), we binarize after the max- max-poolle pooling aggregation. As shown in in Figure 23, we find that binarizing results in worse performance than binariii....
27

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
ILLUSION SAE Pool
FAIR Quiver[Pool, Last)
Bath Last
Win Rate Percentago 0.000 AUC
60
56.5%
40
39.1%
34.8%
8.7%
Baseline
Figure 22. Illustration of potentially mislcading result we find: when comparing activation probes to pooled SAE probes, SAE probe win increases considerably, but adding in "pooled" attention-inspired probe brings down the SAE win-rate. To clarify, the right graph compares the last token activation probe and last token SAE probe, the middle graph compares the attention softmax probe and the token SAE probe, the third graph compares two quivers, "select between SAE max pool probe and last token probe" versus "select between activations softmax probe and activations last token probe". The win rates do not sum to 100% because we consider test AUCs within 0.005 be tied, counting as win neither method.
ILLUSION
FAIR Both Quiver(Pool, Last)
ILLUSION Baseline
FAIR Both
Baseline
FAIR
Both Last
SAE Pool
Beth Last
SAE Pool
Quiver(Pool, Last)
Win Rate Percentage
60
56.5%
60
56.5%
0.005 A AUC
Win Rate Percenta
0.005 AUC
50.0%
40
39.1%
34.8%
40
19.6%
20
8.7%
6.5%
Baseline
Baseline
(a) Multi-token non-binarized experiments (the same as Fig- 22).
(b) Multi-token binzarized experiments.
Figure 23. We run the multi-token experiments from Appendix K with and without binarizing aggregated latents.
28

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Normal Setting (K=16; Basefne lest AUC 0.927
Normal Setting (ko128))
Baseline Test AUC 0.927
Agwrage Tees AUIC
0.680
0.900
0.875
0.870
0.865
0.885
0.860
Scarcity Setting (c-16]
(k=128)
Baseline Test AUC - 0.821
Baseline Text AUC 0.821
Avorigg Sest AUC
0.790
0.785
0.775
Imbalance Setting (iom
Imbalancee Setting (k-120)
0.870
Avengg Tests
0,865
0.860
0.855
Corrupt Setting (x-16) Baseline Tast AUC 0.815
Corrupt Setting (k=128)
Baseline Tist AUC 0.815
0.805
Average Test
Average
0.77
0.785
50
100
150
200
250
300
350
400
50
100
150
200
250
300
350
400
Baseline Tast AUC. 0.795
Baseline Test AUC 0.795
0.76
Average
Average
0.74
0.70
0.72
50
300
35.0
400
50
100
400
SAE LO
SAE LO
Jump Topk
Matryoshka
Figure 24. Across regimes, we see a slight positive trend in probing performance with more recent SAE architectures.
29

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Table 8. Latent categorization, characteristics, and performance.
Latent Cate-
Dataset
Latent
Test
Latent Description
gory
Num- ber
AUC
Description Fits Task
5 5_hist fiiit_fig_ismale
81210
0.892
references to female individuals, espe- cially focusing on pronouns and names. names of well-known actors and film industry personalities.
61 wikidata occupation isactor
91505
0.830
66 living-room
122774
0.999
references to specific rooms in a house, particularly living rooms and parlors. references to basketball teams, players, and related sports terms.
155 athlete sport basketball
83267
0.943
Clever Latents
96sspa__ss
35460
0.907
keywords and phrases related to text messaging and SMS communications. references to conspiracy theories and secretive organizations.
ne0_news.akk
31726
0.968
Potentially Spurious
21 headline istrump
22915
0.812
numerical dates and percentages within a political or historical context.
22 headline .esballie
10555
0.782
strings of numbers, often with mathe- matical notations or identifiers.
1 awlaaaimmdeuaaaaeee
105150
0.816
statistical or numerical data and mea- surements.
133 wet.ooneexxyype_aassllll
89524
0.947
questions or questioning phrases, partic- ularly those beginning with "Why?". words related to status or condition in technical or medical contexts.
134 114_cootex__yyy type. Belief states
113152
0.638
135_ __c_toettttpppeeee Eventtyeeeeeeeeeeeeeeeee Exvent_duration
18897
0.876
questions and phrases related to cost or quantity.
Classified by Context
49_cm_issho
106376
1,000
sections of text related to mathematical programming notation and expres- sions.
Length (Spuri- ous)
161 agnewws
106376
0.990
sections of text related to mathematical or programming notation and expres- sions.
162 agnews..
106376
sections of text related to mathematical 0.988
programming notation and expres- sions.
163 agnews 2
106376
0.991
sections of text related to mathematical or programming notation and expres- sions.
Task Shows Latent De- scription
105 click bait
78823
0.967
mentions of specific events, activities, or items in particular locations or settings.
is Imperfect
_23_world www.woddd.outrr...... condddotontt-nueeeeddd United.Kinndoom
100153
0.978
information related to professional roles, locations, and organizations. names of researchers and scientific au- thors.
125_world_countryy
50817
0.989
30

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Table Left We show the top examples where latent 369585 is active while the CoLA prompt is labeled grammatical. Most of these sentences are clearly ungrammatical, which illustrates that the CoLA data is corrupted. Right We then look at sentences baseline classifier trained CoLA marks ungrammatical when the label is grammatical. We reach the conclusion the CoLA data mislahcled.
Prompt (Labeled Grammatical)
Latent 369585
Prompt (Labeled Grammatical)
P(y=0)
I don't remember what all said?
23.09
Rub the cloth on the baby torn.
0.933
Aphrodite said he would free the animals and free the animals he will
15.87
In front of them happen.
0.926
Gilgamesh wanted to seduce Ishtar, and se- duce Ishtar he did.
14.92
Who did you give pictures of to friends of?
0.91
An example of these substances be tobacco.
14.85
An example of these substances tobacco.
0.893
He will can go
14.43
Susan hopes herself to sleep.
0.890
Table 10. Logistic regression classifier for 110 aal.aammeee_mmeee hummangpts top activating tokens when run on more than 2.5 million tokens from the Pile. The classifier predominantly activates on punctuation. Only tokens with at least 10 occurences shown.
Token
Mean Activation Occurrences
<bos>
6.8863
7625
6.2529
10
Q
6.2271
1436
6.0338
144
5.9111
975
).
5.7334
24
5.5035
17
5,4455
1057
5.4132
319
S..
5.3990
24
31

---

Are Sparse Autoencoders Useful? A Case Study in Sparse Probing
Table 11. Comparison of different logistic regression ("hase") and SAE probing methods selection of 60 random datasets. Last is last token probing as in the rest of our paper, concat is concatenating the top 20 PCA dimensions of all tokens, mean is taking the mean across activation dimensions across context, max taking the max across activation dimensions across context, and Attn-like probe described in Appendix K.
SAE
SAE
SAE
SAE
SAE
SAE
Attn
Base (last)
Base (concat)
Dataset
(last)
(mean)
(max)
(last)
(mean)
(max)
Like
10=68
10=68
10=68
10=408
10=408
10=408
Probe
6 hist fig
0.990
0.977
0.982
0.914
0.984
0.983
0.883
0.985
0.987
7 _hhis__fis
0.750
0.698
0.738
0.661
0.739
0.746
0.613
0.738
0.760
21 _headlin
0.987
0.986
0,950
0.993
1.000
0,964
0.990
1.000
1.000
24 headlin
0.991
0.979
0.892
0.995
0.997
0.947
0.994
0.998
0.996
44. -hyss.tt .f
0.885
0.541
0.836
0.657
0.760
0.854
0.656
0.790
0.903
48 llcccmcor
0.833
0.659
0.788
0.699
0.779
0.794
0.708
0.784
0.860
54_cs_tf
0.695
0.554
0.689
0.588
0.680
0.708
0.580
0.697
0.697
59 99iiiiddtt
0.961
0.787
0.933
0.809
0.864
0.939
0.809
0.875
0.935
62_wikida
0.981
0.878
0.948
0.892
0.923
0.959
0.898
0.925
0.954
63 wikidat
0.959
0.840
0.925
0.869
0.893
0.946
0.848
0.862
0.928
66 livvin-
1.000
0.985
1,000
0.968
0.999
1.000
0,950
0.998
0.998
67_social-
1.000
0.992
0.999
0.985
0.999
1.000
0.971
0.998
0.999
68_credit-
1.000
0.959
0.998
0.942
0.986
0.998
0.920
0.981
0.993
71 social-
0.998
0.985
0.998
0.969
0.993
0.998
0.950
0.989
0.995
73 _ccontrol
0.998
0.970
0.995
0.945
0.984
0.998
0.935
0,986
0.982
78 north-a
1.000
0,933
0.998
0.907
0.990
0.999
0.883
0.976
0,995
79 human-r
0.999
0.974
0.995
0.964
0.988
0.996
0.947
0.990
0.992
89 39-ulue mr
0.891
0.761
0.785
0.784
0.796
0,823
0.796
0.777
0.867
90 glue qn 0.926
0.729
0.886
0.830
0.909
0.910
0.859
0.909
0.931
94_ai-gen 0.998
0.996
0.993
0.997
0.996
0.996
0.996
0.997
0.996
96 96.spam.is is
0.999
0.999
0.997
0.996
0.997
0.998
0.995
0.999
0.999
100_news_f
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
105 click
1.000
1.000
1.000
0.999
1.000
0.999
1.000
1.000
1.000
106_hate_h
0.722
0.612
0.661
0.742
0.785
0.674
0.735
0.788
0.812
110_aimade
0.975
0.865
0.953
0.919
0.970
0.953
0.916
0.960
0.979
14 nyc bo
0.780
0.755
0.740
0.716
0.797
0.745
0.733
0.805
0.872
19__us_st
0.997
0.957
0.994
0.982
0.990
0.994
0.952
0.992
0,995
120_us. 1.0_usstim tim
0.944
0.803
0.942
0.785
0.946
0.936
0.751
0.940
0.941
121_us_tim
0.952
0.815
0.948
0.776
0.954
0.950
0.754
0.957
0.951
124 world
0.999
0.999
0.999
0.965
0.999
0.998
0.968
0.999
0.999
127 _rt_tty
0.913
0,872
0,894
0.834
0.889
0.905
0.845
0,900
0.916
129 arith.
0.979
0.903
0.880
0.867
0.951
0.938
0.861
0.975
0.977
132_temp_c
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.991
133 133_contex
0.984
0.982
0.975
0.973
0.979
0.968
0.973
0.980
0.986
334_contex
0.985
0.977
0.953
0.966
0.989
0.966
0.963
0.994
0.987
137 -lluglue_m
0,856
0.551
0.788
0.632
0.734
0.807
0.658
0.742
0.844
339_news_e
0.978
0.950
0.944
0.972
0.953
0.962
0.967
0.973
0.965
141 news.c
0.951
0.942
0.917
0.967
0.968
0.933
0.965
0.973
0.965
144 cancer
1.000
1.000
0.981
1.000
1.000
0.991
1.000
1.000
1.000
145_diseas
0.618
0.677
0.501
0.618
0.678
0.563
0.606
0.686
0.655
149 _ttt_em
0.787
0.756
0.730
0.780
0.843
0.759
0.813
0.853
0.851
150_twt_em
0.710
0.612
0.652
0.700
0.730
0.672
0.717
0.746
0.777
155 athlet
0.995
0.988
0.990
0.970
0.988
0.988
0.960
0.985
0.996
158_code.C
1.000
0.998
1.000
0.999
1.000
1.000
0.999
1.000
0.997
162 162_agnews
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1,000
163 _a__ans
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
32


Reviews:
[{"review_id": "fEdP1MLkcg", "reviewer": "Reviewer_bCor", "summary": "This paper deals with the problem of evaluating the downstream utility of sparse autoencoders (SAEs). SAEs have recently gained popularity as a means to disentangle concepts learnt by layers of a model, paritcularly LLMs, in order to gain a better mechanistic understanding of their workings. However, evaluating their utility systematically has been challenging. This work examines this by comparing the utility of SAEs with other baseline approaches over a variety of binary tasks. To simulate real world utility, it considers cases of data scarcity, class imbalance, label noise, and covariate shift, and finds that SAE probes do not beat existing baselines. It then performs an analysis over learnt SAE latents to better understand why this happens, and to understand differences in the observations made as compared to prior work.\n\n## Update after rebuttal\nThank you for your response. The most important concerns I had were addressed, so I am increasing my score to accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions and comments! We are especially grateful for your suggestions on better latent selection methods, and we are glad you feel our paper is a valuable contribution.\n\n---\n\n> However, since SAE latents (e.g. in TopK SAEs, Gao et al. 2024, L086 right) are not on the same scales, it seems like this could be misleading. \n\nWe agree your method of choosing latents is better, thank you! We find this technique improves test AUC when k is small (<32) but not when k is large (see https://anonymous.4open.science/r/SAE-Probes-B404/rebuttal_plots/comparing_new_and_old_mean_diff_auc.png). Intuitively, your method better finds the “correct” latents for small k, but the old method is “good enough” for large k.\n\nWe reran our main experiments with this technique. Unfortunately, the baseline + SAE quiver still fails to improve over baselines. Investigating further, we find that we select k = 128 probes from the quiver 80% of the time, so quiver performance is dominated by k = 128 probes, which do not improve that much.\n\n\n> If the SAE was \"chosen\" based on its performance for 14 tasks (L184-188 right), shouldn't [Figure 4] have 14 points above the diagonal?\n\nWhen an SAE is chosen the point may be on or below the diagonal, since the test AUC might decrease or stay about the same. We’ve added “Datasets not directly on the diagonal signify that an SAE method was chosen from the quiver” to the Figure 4 caption.\n\n> In Section 4.2 (and Section 4) in general, results from autointerp are assumed to be \"ground truths\". It would be helpful if this could be evaluated, e.g. using humans for a small subset of latents. \n\nThis is a valid point, thank you! To remove this confounder in Section 4.1, two of the authors independently labeled and ranked latents, with no change in results. However, manually labeling the latents for all datasets in Section 4.2 is labor intensive (and thus a good use case for autointerp).\n\n>could it be possible… that SAEs would perform well at a different layer? \n\nThis is a great point that we did not think of, thank you! We checked the width=16k largest L0 SAEs on the 4 layers in Figure 12a (layers 9, 20, 31, and 41), and found that layer 20 was also the best for SAEs as well, see https://anonymous.4open.science/r/SAE-Probes-B404/rebuttal_plots/comparing_sae_test_auc_by_layer.png. We have added this plot and a discussion to Appendix D.\n\n\n> Results in Section 4, and particularly Section 4.3, are on specific handpicked datasets. Do they generalize? Alternatively, is there a reason for picking these specific datasets? \n\nWe apologize for not being clearer here; we intended for Section 4.3 to be a set of case studies complementing Section 4.2. We selected these datasets after an investigation of five datasets whose top latent representations exhibited strong performance. During this analysis, we discovered that both 87\\_glue\\_cola and 110\\_aimade\\_humangpt3 contained labeling errors. We have altered the introduction to section 4.3 to reflect this.\n\n\n> Why is a different SAE config used for experiments with label noise (L263) and covariate shift (L294)?\n\nFor label noise, we use the SAE with smallest width (16k) and maximal L0, which we found to be most performant in standard conditions (we have added this justification to the paper). We would have used this SAE for the covariate shift domain as well, but there is no support for generating autointerp explanations through Neuronpedia for this SAE, so we instead use the width = 131k, L0 = 114 SAE. We have added additional clarification around this choice in the paper, thank you for this comment!\n\n> In Section 4.1, why is k=8 used, when k=16 and k=128 have been used everywhere else?\n\nWe do so because the probe pruning experiment is a proof-of-concept experiment and is significantly simpler with smaller k.\n\n> L158, right: why only use logistic regression for SAE probes?\n\nThis is a great question! We think this is a good choice because logistic regression is common in practice and is the best baseline activation methods. This is still a valid concern, so we have added the following to our limitations section: \"Finally, it is possible that further optimization of the SAE probe baseline might increase performance such that it beats baseline methods. For example, we only tried logistic regression on SAE probes, and it is possible that other probing techniques could perform better.\"\n\n> L027-033: sentence is hard to read\n\nWe agree and have rephrased to: “However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to ensemble SAEs and baselines to consistently improve over just baseline methods.”\n\n---\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper, especially with regard to the better top k latent selection? If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "The claims made are generally thoroughly supported by evidence. Some concerns, such as generalization to other layers and datasets, have been raised in the Weaknesses section below.", "methods_and_evaluation_criteria": "The methods and evaluation criteria generally make sense. Some concerns about evaluation have been raised in the weaknesses below, and a critical concern about the method formulation has been raised in Weakness 1 below.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Generally the experimental design and analyses appear to be sound and thorough. The paper first explores whether SAEs outperform baselines, and then (in Section 4) looks into possible reasons for not doing so. It also explores why results shown in this work may differ from previous findings (Section 5).", "relation_to_broader_scientific_literature": "This work focuses on evaluating the downstream utility of SAEs trained on LLMs. It builds upon recent work (e.g. Bricken et al. 2024, Gao et al. 2024, etc.) that aim to show that SAEs can disentangle concepts learnt by each layer of such models. However, evaluating SAEs for their utility has been limited, for which this work provides an important contribution.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "## Strengths\n1. This paper deals with the important problem of assessing the utility of SAEs and to understand if they truly provide an added benefit to existing methods. This has been one of the key promises of SAEs, but has not been evaluated properly so far, and is thus a valuable contribution.\n2. The evaluation appears to be very thorough, covering 113 datasets, five baselines, and with several ablations for each.\n3. Analyses have also been performed to understand why SAEs may be underperforming (Sections 4 and 5), which can help understand the root cause of the problem and direct future research.\n\n## Weaknesses\n1. The input to the SAE probes are the top $K$ latents in the trained SAE that have the maximum absolute difference for the two binary classes (Equation 1). However, since SAE latents (e.g. in TopK SAEs, Gao et al. 2024, L086 right) are not on the same scales, it seems like this could be misleading. For example, consider two latents $A$ and $B$ and classes $0$ and $1$. Let the values of $A$ for hundred data points each of classes 0 and 1 be $\\{1,2,\\cdots,100\\}$ and ${51,52,\\cdots,150\\}$ respectively. Then the class-wise mean activations would be $50.5$ and $100.5$ respectively, giving a difference (as per Equation 1) of $50$. Now, suppose the values of $B$ for these data points are ${0.01, 0.02, \\cdots, 1.00\\}$ and $\\{10.01, 10.02, \\cdots, 11.00\\}$ respectively. Then the class-wise mean activations would be $0.505$ and $10.505$ respectively, giving a difference of $10$. Clearly, latent $B$ is more discriminative of the two classes than latent $A$, but the scheme proposed in Equation 1 would pick latent $A$ instead. This could be fixed for instance by normalizing using the mean activations of each latent, and could help avoid misleading conclusions.\n2. It is unclear how to interpret Figure 4. If the SAE was \"chosen\" based on its performance for 14 tasks (L184-188 right), shouldn't the Figure have 14 points above the diagonal? Or is this because the SAE outperformed the baselines in the validation set but underperformed in the test set? If so, the fact that this happened so consistently seems surprising, and a discussion on this would be useful.\n3. In Section 4.2 (and Section 4) in general, results from autointerp are assumed to be \"ground truths\". It would be helpful if this could be evaluated, e.g. using humans for a small subset of latents. As of now it is unclear if the performance loss is due to wrong latents being used (as claimed in Section 4.2) or by autointerp labelling them incorrectly.\n4. All evaluation is performed at layer 20 because this is where the baselines performed the best (e.g. as per Section D.1). However, could it be possible that this choice harms the SAEs in the comparison, and that SAEs would perform well at a different layer? A discussion on this would be useful.\n5. Results in Section 4, and particularly Section 4.3, are on specific handpicked datasets. Do they generalize? Alternatively, is there a reason for picking these specific datasets? Comment on this would be helpful.\n6. Why is a different SAE config used for experiments with label noise (L263) and covariate shift (L294)?\n7. In Section 4.1, why is $k=8$ used, when $k=16$ and $k=128$ have been used everywhere else?\n8. L158, right: why only use logistic regression for SAE probes?", "other_comments_or_suggestions": "- L027-033: sentence is hard to read, please rephrase.", "questions_for_authors": "Please refer to the Weaknesses section. Overall, I believe this paper provides a valuable and important contribution, and does a thorough and interesting analysis. I strongly lean towards accept, but I believe the issue with scales of the metric as discussed in Weakness 1 is critical and needs to be addressed. I would be happy to raise my score if this is adequately discussed in the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7X3cpMEmVP", "reviewer": "Reviewer_9uBB", "summary": "In this work, the authors propose a fair evaluation of SAEs by modelling them as a tool in a practitioner's toolkit, or “quiver of arrows” with the overall question of asking “When is it useful for a practitioner to incorporate SAE probes into their downstream application?”", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and help! We were very glad to hear that you appreciated our use of the quiver of arrows technique and found our evidence and claims clear.\n\n\n---\n\n\n> While I do observe evidence that SAE probes contribute less to downstream use cases than other probes, the gap is not incredibly large, especially in Fig. 5.\n\nThank you for pointing this out! Figure 5 shows the result of the quiver of arrows approach, and so the SAE quiver achieving the same performance as a baseline quiver only shows that SAEs do not actively make the practitioner worse off. We have added a new table in the appendix which directly compares each SAE to logistic regression.\n| **Setting** | Baseline Quiver | SAEs Quiver | SAEs + Baselines Quiver | LogReg | SAE 16k k=16 | SAE 16k k=128 | SAE 131k k=16 | SAE 131k k=128 | SAE 1m k=16 | SAE 1m k=128 |\n|------------------------|------------------|-------------|---------------------------|--------|---------------|----------------|----------------|------------------|---------------|----------------|\n| **Standard Conditions**| 0.940 | 0.930 | 0.939 | **0.941** | 0.904 | 0.921 | 0.899 | 0.918 | 0.889 | 0.913 |\n| **Data Scarcity** | 0.819 | 0.806 | 0.812 | **0.836** | 0.800 | 0.816 | 0.794 | 0.810 | 0.785 | 0.801 |\n| **Class Imbalance** | 0.921 | 0.906 | 0.916 | **0.929** | 0.898 | 0.909 | 0.890 | 0.906 | 0.882 | 0.899 |\n\n> However, I was unconvinced by the claims in Section 5 (see strengths/weaknesses)...\n\nThanks! We agree and have moved it to the appendix. \n\n> However, I would like to see more elaboration of why the imbalanced dataset settings give the SAE’s inductive bias an advantage.\n\nThank you for pointing this out! We have created a new Appendix section to detail the general SAE probing intuition and for each section. We paraphrase below: \n\n‘’’\n“..We argue that if SAEs are successful at this task [creating a sparse interpretable model basis], requiring a probe to only use a sparse set of directions in this basis should serve as a beneficial inductive bias to prevent overfitting with limited data.”\nClass Imbalance: Because SAE latents are sparsely activating, choosing SAE latents that are positive on the minority class and negative on the majority class may generalize well.\n‘’’ \n\n> Section 5 is a little confusing to me. Bricken et al. also use max pooling on their baseline probes… Thus, it is hard to argue they present an “illusion” of SAE probes being better…\n\n\nThank you for pointing this out! We attempt to argue that max pooling activations is not the strongest baseline possible. That being said, we agree that our language regarding an “illusion” is too strong, and we have toned down the language in the paper.\n\n\n> Additionally, it is hard to understand if your results in the third graph of Figure 11 came from softmax-pooling or the quiver approach… Also, why do the win rates not sum to 100%?\n\nWe apologize that we were not more clear here! The graph compares two quivers, quiver(SAE max pool, SAE last token) vs. quiver(activations softmax,activations last token). The win rates do not sum to 100% because we consider test AUCs within 0.005 to be tied, and thus count as a win for neither method. We have added additional clarification to the text and figure caption.\n\n> In 2.3, when introducing your probing methods, you seem to be listing the hyperparameters for each method…The writing should be improved here.\n\nWe agree, thank you! We have now formatted this section as a table.\n\n> …”we train probes using the largest L0”...What is the “largest l0”?\n\nThank you for noting this! For any given SAE width, we use the largest available L0 from GemmaScope (and noted this in the paper).\n\n> …the “Quiver of Arrows” setup is a core part of your model… I would appreciate seeing this discussed in the main body.\n\nWe agree! We moved it to the main body.\n\n> …consider flipping the x-axis of Figure 19…\n\nThanks, we just made this change!\n\n> …~25%… There is a typo here with the ~\n\nGood catch! We have updated the paper to replace it with the correct value (22%)\n\n> 1)…Did your GemmaScope and LlamaScope SAEs not come with already-interpreted latents? 2\n\n> 2) Also, the autointerp process you used is not described in your paper. \n\n1)Gemma/LlamaScope SAEs do not come with already-interpreted latents, they just contain the SAEs themselves. 2) we have now added the following language to Section 4.1: “For this and all subsequent experiments, we generate autointerp labels using Neuronpedia, which leverages a language model to produce consistent natural language explanations for a latent based on its top activating tokens.” Thanks for the update!\n\n> In section 4.2 when you consider the top 128 latents is this again by mean difference across positive and negative samples in your binary classification tasks?\n\nYes, we have fixed this in the text, thank you!\n \n---\n\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper?", "claims_and_evidence": "The author’s claims are supported by relatively clear evidence. The claims are simple and easily validated by the provided experiments. While I do observe evidence that SAE probes contribute less to downstream use cases than other probes, the gap is not incredibly large, especially in Fig. 5. I appreciate the authors claim that autointerpretability methods could easily be applied to probes to achieve similar latent interpretability benefits. However, I was unconvinced by the claims in Section 5 (see strengths/weaknesses). I don’t think the section adds a lot to the impact of the paper and I would encourage removing or deferring it to the appendix.", "methods_and_evaluation_criteria": "The quiver of arrows methodology makes sense, especially in a setting where one has multiple SAE probes. Additionally the authors choose a comprehensive suite of probing tasks to cover a wide range of potential use cases for SAE probes. I also think their experiments were thorough and together paint a clear picture of their argument and conclusion. However, I would like to see more elaboration of why the imbalanced dataset settings give the SAE’s inductive bias an advantage.", "theoretical_claims": "The authors make no theoretical claims in this work.", "experimental_designs_or_analyses": "I checked the general quiver-of-arrows AUC comparison setup which is used throughout their experiments and found it to be sound.", "relation_to_broader_scientific_literature": "This work ties in well with existing discussion on Sparse Autoencoders in the interpretability literature.", "essential_references_not_discussed": "I think the work covered most essential literature.", "other_strengths_and_weaknesses": "Section 5 is a little confusing to me. Bricken et al. also use max pooling on their baseline probes, and find that the performance is similar. Thus, it is hard to argue they present an “illusion” of SAE probes being better and thus the need for the argument presented in Figure 11. Additionally, it is hard to understand if your results in the third graph of Figure 11 came from softmax-pooling or the quiver approach. It seems as though you use the quiver approach to “select between pooled and last-token strategies,” does this mean that you are considering all possible strategies {softmax, last} x {SAE, activations} and picking the best? Also, why do the win rates not sum to 100%?", "other_comments_or_suggestions": "* In 2.3, when introducing your probing methods, you seem to be listing the hyperparameters for each method. However, they are introduced as sentence fragments and without context. The writing should be improved here. \n\n * \"throughout the paper we train probes using the largest L0 for SAEs width = 16k, width = 131k, and width = 1M. We use k = 16 to construct easily interpretable probes that potentially overfit less and use k = 128 for performance.\" What is the “largest l0”?\n\n * I feel like the “Quiver of Arrows” setup is a core part of your model for practitioner usefulness, yet you describe it only in the Appendix. I would appreciate seeing this discussed in the main body.\n\n * You may consider flipping the x-axis of Figure 19 and relabel it “Number of Pruned Latents”, as I intuitively read the graph as further right meant more pruning.\n\n * “We find that ~25% of CoLA labels…” There is a typo here with the ~", "questions_for_authors": "Why did you have to run autointerp on your latents, for example in 4.1 and 4.2? Did your GemmaScope and LlamaScope SAEs not come with already-interpreted latents? Also, the autointerp process you used is not described in your paper. There are various strategies to do this proposed by different works so it is important you explain which method you did. \n\nIn section 4.2 when you consider the top 128 latents is this again by mean difference across positive and negative samples in your binary classification tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MF1ROnvWW", "reviewer": "Reviewer_gmH6", "summary": "The paper comprises of several experiments evaluating the efficacy of sparse autoencoder (SAE) approaches to probing. \n\nThe paper first focuses on the accuracy of probes under various settings (such as imbalanced data), and finds that SAEs do not improve upon baselines. The paper introduces a \"quiver of arrows\" methodology, in which the SAE approach is evaluated according to its ability to improve upon a \"toolkit\" of other methods (specifically, the best method is chosen on a validation set, and the test accuracy is computed).\n\nThe paper then examines other potential benefits of SAEs beyond accuracy improvements, such as interpretability. Here, the paper demonstrates that many of the tasks that can be achieved by SAEs can also be achieved by simple baselines (like logistic regression).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for your time and help, especially related to your points about the quiver of arrows and the clarity of our interpretability experiments. We were glad to hear that you appreciated the breadth of our experiments and found the problem we are investigating important.\n\n---\n\n> However, the method by which SAE-based probing is evaluated (\"quiver of arrows\") is nonstandard. The claim that such an approach is needed for robustness is not convincing to me, as it is widely practiced to compare the individual accuracies of different models.\n\nThank you for bringing up this point! The quiver of arrows approach we introduce is non-standard in the literature, but we adopt it to make the strongest possible case for SAEs. Since we select the best method using validation AUC, we expect to choose SAEs only for tasks where they perform best. To verify this, in the three settings where we employ the quiver of arrows—standard conditions, data scarcity, and class imbalance—we compare its performance to that of using a single SAE across all tasks, as shown below.\n| **Setting** | Baseline Quiver | SAEs Quiver | SAEs + Baselines Quiver | LogReg | SAE 16k k=16 | SAE 16k k=128 | SAE 131k k=16 | SAE 131k k=128 | SAE 1m k=16 | SAE 1m k=128 |\n|------------------------|------------------|-------------|---------------------------|--------|---------------|----------------|----------------|------------------|---------------|----------------|\n| **Standard Conditions**| 0.940 | 0.930 | 0.939 | **0.941** | 0.904 | 0.921 | 0.899 | 0.918 | 0.889 | 0.913 |\n| **Data Scarcity** | 0.819 | 0.806 | 0.812 | **0.836** | 0.800 | 0.816 | 0.794 | 0.810 | 0.785 | 0.801 |\n| **Class Imbalance** | 0.921 | 0.906 | 0.916 | **0.929** | 0.898 | 0.909 | 0.890 | 0.906 | 0.882 | 0.899 |\n\n\nClearly, the quiver of arrows serves as an upper bound on the performance of any individual SAE. As an alternative counterfactual, instead of comparing against a single SAE probe, we select the best SAE for each task using validation AUC (a “quiver of SAEs”) and compare this to the overall quiver of arrows with both baselines and SAEs. Again, the baseline + SAEs quiver outperforms the just SAE quiver.\n\n> At a higher level, the claim that SAE-based probing does not have advantages for interpretability is limited by the fact that there are not comparisons to baselines for the most standard interpretability task (interpreting latents). The paper claims that \"these findings may be possible using baseline classifiers,\" (line 373), but this is limited to theoretical speculation. An evaluation here would be more convincing. In general, the other evaluations in Section 4 seemed to be somewhat ad hoc / anecdotal.\n\nThank you for pointing out that line 373 was confusing! In section 4.3, we zoom into two datasets as case studies and show that this is not just a theoretical worry: baseline classifiers can find spurious correlations and noisy labels as well as SAEs. We have modified this paragraph to be as follows:\n\n```\nThe spurious latent category seems especially promising because finding a spurious latent may help us identify spurious features in the dataset. However, in a cast study in \\cref{sec:ai_vs_human}, we find that similar findings may be possible using baseline classifiers: we apply a logistic regression probe to model hidden states on tokens from the Pile \\cite{pile} and show that maximally activating examples also exhibit the spurious correlation. \n\nHowever, a practical advantage for SAEs is that the infrastructure to perform autointerp is pre-existing through platforms like Neuronpedia, and a theoretical advantage is that the baseline classifier can only identify the single most relevant coarse-grained feature, while the decomposability of SAE probes into latents allows for identifying many independent features of various importance.\n```\n\nFor your first point, we are not sure what you mean by comparisons to baselines for interpreting latents. Because SAEs are an unsupervised method, it is not clear what a baseline for interpreting a latent would be, as other methods do not have an equivalent to latents (i.e. units a probe can be broken down into). This decomposition is an advantage of SAEs, but it is unclear how much value it gives. In section 4.2, we are investigating what we can learn about different SAE latents by examining the datasets that they are discriminative for (as opposed to auto-interp or human interpretability of latents, which typically looks at top activating examples). \n\n---\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? If not, what further clarification or modifications could we make to improve your score?", "claims_and_evidence": "The paper's primary claim is that SAE-based probing does not outperform simple baselines. This claim (when restricted to the paper's implementation of SAE-based probing) is supported, on one hand, by a large set of experiments. However, the method by which SAE-based probing is evaluated (\"quiver of arrows\") is nonstandard. The claim that such an approach is needed for robustness is not convincing to me, as it is widely practiced to compare the individual accuracies of different models.\n\nAt a higher level, the claim that SAE-based probing does not have advantages for interpretability is limited by the fact that there are not comparisons to baselines for the most standard interpretability task (interpreting latents). The paper claims that \"these findings may be possible using baseline classifiers,\" (line 373), but this is limited to theoretical speculation. An evaluation here would be more convincing. In general, the other evaluations in Section 4 seemed to be somewhat ad hoc / anecdotal.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "The paper contributes to a growing literature studying SAEs as a tool for mechanistic interpretability. While there has been a lot of excitement regarding the potential of SAEs, as the paper notes, the literature evaluating the practicality/usefulness of SAEs is limited. This paper adds to this literature by considering SAE probing performance on a large number of probing datasets.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The paper focuses on an important task, evaluating SAEs against baselines, and provides evidence (through many datasets) that SAE-based probing may not offer the improvements that some past work has suggested. The breadth of experiments is a strength, exploring the settings that may particularly illustrate the benefits of SAEs. This kind of fair evaluation is important.\n\nThe primary weaknesses of the paper are the nonstandard evaluation strategy in Section 3, and the incomplete and somewhat ad hoc results in Section 4 (in particular, the lack of comparison to baselines in Section 4.2, since interpretability is the primary strength of SAEs).", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5Dg69dgi6", "reviewer": "Reviewer_pVWC", "summary": "This paper studies the benefits and limits of Sparse Autoencoders (SAEs). The topic is quite relevant given they are attracting more and more research in Large Language models, specifically in the context of Mechanistic Interpretability (MI). The authors propose a set of benchmarks to study both probing and interpretability of SAEs, concluding that they often do not meet their promises.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work! We are grateful for your time and help, especially related to missing discussion of related work and questions about the automated interpretability techniques we use. We were especially glad to hear that you appreciated the depth to which we studied evidence and counter-evidence.\n\n\n---\n\n\n>And whether SAEs extract interpretable concepts remains essentially unsolved.\n\n>.. but this does not seem to provide clear evidence in support of or against interpretability of SAEs…All questions revolve around the interpretability of SAEs. The main question is: Are SAE neurons interpretable?\n\nThis is an important point that we would like to clarify. In this paper, we do not directly investigate whether SAEs discover an interpretable basis of latents. This statement is difficult to falsify or prove because there is no ground-truth for SAE latents, as you note. Instead, the goal of our work is to attempt to evaluate how “good” the basis of features that SAEs discover is by  studying how helpful they are on the downstream task of probing. This question is both more useful for practitioners and easier to answer than pure interpretability metrics. Thus, our work also provides a more objective (if indirect) measure of SAE latent interpretability.\n\n> The details about the autointerpr method are missing and they should be discussed at least in the supplementary. I am not even convinced that using this method is the best practice: it looks a bit circular to study the LLM concepts with another LLM, and in the best possible scenario that would have been done with a user study. \n\nThank you for bringing this up! We have added the following description of autointerp in Section 4.1: “For this and all subsequent experiments, we generate autointerp labels using Neuronpedia, which leverages a language model to produce natural language explanations for a latent based on its top activating tokens (the Neuronpedia autointerp implementation is based on [4]).” Autointerp is a standard procedure for evaluating SAE latents [1, 2, 3, 4 ], although you are correct in stating that it is somewhat circular to study an LLM with an LLM! More specifically, our procedure is dependent on the efficacy of autointerp. To remove this confounder in Section 4.1 (about probe pruning), we ran an additional experiment where two of the authors independently labeled latents and ranked their relevance for all three tasks, with no change to the results. \n\n[1] \"Language models can explain neurons in language models.\"\n\n[2] \"Towards Monosemanticity: Decomposing Language Models With Dictionary Learning\"\n\n[3] \"Scaling and evaluating sparse autoencoders.\" \n\n[4] \"Automatically interpreting millions of features in large language models.\" \n\n> In the discussion of Section 4.1, the authors focus on the latent 122774 of the SAE, whose supposed meaning is \"mentions living room\". This \"concept\", however, is not invariant to language variations (the French example), so is it really encoding the concept of \"presence of living room\"? Is it activating also on unrelated sentences?\n\nThis is correct! We use latent 122774 as an example of a latent that doesn’t represent the true living-room feature (which we expect to be the same across languages). This indicates that the underlying SAE is imperfect, and is a possible reason why SAE probes did not generalize as well to covariate shifts. This latent generally activates only when living-room is in the sentence, but other latents are less precise.\n\n> In section 4.2, the authors further suggest that some latents are correctly labelled and some others are mislabelled by autointerpr… Despite that, is it the case that latents are disentangled from other potentially unrelated concepts? Can they activate in other tasks as well? Also, from the conclusion in lines 372-384 it is not clear what the authors mean.\n\nThank you for mentioning this! If SAEs worked perfectly, then we would expect latents to specialize and only fire on a single concept (which could be quite complex itself). However, since SAEs are imperfect, many latents remain polysemantic (are active on multiple concepts). We have adjusted the language of lines 372-384, we agree that it was complex! Please see our response to reviewer 3 (gmH6), which has the revised paragraphs. \n\n>Also, the code is not available and reproducibility cannot be verified.\n\nThank you for pointing this out! We have uploaded our code anonymously here: https://anonymous.4open.science/r/SAE-Probes-B404 We also have added a link to the de-anonymized github repo in the non-anonymous version of our paper.\n\nThank you for providing the additional citations, we will add them to our related work.\n\n---\n\n\nThank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? And are there any further clarification or modifications we could make to improve your score?", "claims_and_evidence": "Being an experimental verification of the advantage of SAEs, claims are about the investigation. The authors cover different settings, including 5 settings for probing, and 3 settings for interpretability of SAEs.\n\nThe claims are clear, and evidence and counter-evidence have been well studied.\nI have a few remarks on the interpretability side, see next box.", "methods_and_evaluation_criteria": "Linear probing seems convincing (section 3). After obtaining the last token embeddings, the investigation resembles a standard machine learning pipeline. The evaluation criteria are good. The authors' methodology is precise and both plots and settings are clear and easy to understand. Also connected, section 5 contributes to highlighting SAEs are not useful for downstream tasks.\n\n\nThe investigation on the SAEs activations interpretability is less rigorous compared to the previous one. And whether SAEs extract interpretable concepts remains essentially unsolved.\n\n\n0. The authors start from the assumption that, somewhat, activations of SAEs are interpretable (Point 2, lines 87-93; First paragraph, lines 300-303). I am not convinced this is the case as the authors also highlight that \"we lack a ground-truth to know whether SAEs truly extract interpretable concepts...\" (lines 40-44). The purpose of Section 4 is to investigate interpretability by measuring usefulness for other tasks, but this does not seem to provide clear evidence in support of or against interpretability of SAEs.\n\n1. The details about the `autointerpr` method are missing and they should be discussed at least in the supplementary. I am not even convinced that using this method is the best practice: it looks a bit circular to study the LLM concepts with another LLM, and in the best possible scenario that would have been done with a user study. In the discussion of Section 4.1, the authors focus on the latent 122774 of the SAE, whose supposed meaning is \"mentions living room\". This \"concept\", however, is not invariant to language variations (the French example), so is it really encoding the concept of \"presence of living room\"? Is it activating also on unrelated sentences?\n\n2. In section 4.2, the authors further suggest that some latents are correctly labelled and some others are mislabelled by `autointerpr`. This conclusion is drawn by restricting to the dataset of interest for these latents. E.g., for positive evidence of 81210 on the dataset 5, for negative evidence of 50817 on the dataset 125. Despite that, is it the case that latents are disentangled from other potentially unrelated concepts? Can they activate in other tasks as well? Also, from the conclusion in lines 372-384 it is not clear what the authors mean.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental setup seems sound and the analysis in Section 3 is accurate. Section 4 is less convincing. Section 5 is very clear.  \nAlso, **the code is not available and reproducibility cannot be verified**.", "relation_to_broader_scientific_literature": "Some works are worth mentioning. I would suggest the authors to connect to other representation learning works relevant to study interpretability and advantages of probing with SAEs. There is an interesting link to identifiability and disentanglement research:\n\n[1] Are Disentangled Representations Helpful for Abstract Visual Reasoning?,  van Steenkiste et al., NeurIPS 2019 - discusses whether disentangled (which means sometimes interpretable) representations are helpful for tasks, concluding they are not. \\\n[2] Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning, Lachapelle et al., ICML 2023 - considers sparsity and label classification to extract disentangled representations. This seems to aid in classification over new tasks. \\\n[3] Identifiable Steering via Sparse Autoencoding of Multi-Concept Shifts, Joshi et al., arXiv 2025 - studies identifiability of SAEs and connects to the linear representation hypothesis.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "It is sidelined that SAEs are not truly intrinsically interpretable. This has to be verified somehow.", "other_comments_or_suggestions": "Repetition of reference Bricken 2024a/b.", "questions_for_authors": "All those appearing in Methods and Evaluation Criteria. All questions revolve around the interpretability of SAEs. The main question is: Are SAE neurons interpretable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a fairly exhaustive systematic empirical evaluation to understand the effectiveness of sparse auto‑encoders (SAEs) in downstream probing of LLM activations. Across several binary classification datasets and different regimes (data scarcity, class imbalance, label noise, covariate shift) the authors compare different SAE variants against simpler baselines on raw activations via an evaluation methodology they refer to as quiver of arrows — the goal is to understand if adding SAE probes to a set of existing baselines improves the performance. The evaluation reveals that performance is not better (and often worse) than the baselines.  Some case studies are used to show that the supposed secondary benefits of SAEs (e.g., detecting spurious correlations, flagging bad labels) can also be matched by simple baselines as well.  These findings urge the community to set a higher empirical bar before claiming interpretability wins from current SAEs. 

Given the clear majority of reviewer support, the careful additional experiments, and the paper’s potential impact on a rapidly moving area, I recommend acceptance. For the final version I encourage the authors to include the additional experiments and make the  changes promised in the rebuttal.
Decision: Accept (poster)