Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders
Authors: Mengyu Ye
Jun Suzuki
Tatsuro Inaba
Tatsuki Kuribayashi
Venue: NeurIPS 2025 poster ()
Keywords: interpretability
key-value memories
sparse autoencoders
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We find that transformer key-value memories are nearly as interpretable as SAE features

Abstract: Recent interpretability work on large language models (LLMs) has been increasingly dominated by a feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated.
This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately, only a few studies have made such comparisons systematically so far.
In this work, we revisit the interpretability of feature vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks.
Our extensive evaluation revealed that SAE and FFs exhibits a similar range of interpretability, although SAEs displayed an observable but minimal improvement in some aspects. 
Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAEs and FFs diverged.
These bring questions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting FF feature vectors, and FF key-value parameters serve as a strong baseline in modern interpretability research.

Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders
Mengyu Ye
Jun Suzuki
Tohoku University
Tohoku University & RIKEN
ye mwngyu..u... s10dc. tohoku. ac. jp
jun. www.kiitouuosaaaoooooo ac. ac.
Tatsuro Inaba" MBZUAI
Tatsuki Kuribayashi
MBZUAI tatsuki. Bresukikkurbbyysiiee Stw.hbaayyas@@mmuaaii... @mbzuai ac. ae
tatsuro. www.:onnnnaabbiime.oommm
Abstract
Recent interpretability work large language models (LLMs) has been increas- ingly dominated by feature-discovery approach with the help of proxy modules. Then, the quality of features learned by, e.g., sparse auto-encoders (SAEs), is evaluated. This paradigm naturally raises a critical question: do such learned features have better properties than those already represented within the original model parameters, and unfortunately. only few studies have made such compar- isons systematically vectors stored in feed-forward (FF) layers, given the perspective of FF as key-value memories, with modern interpretability benchmarks. Our extensive evaluation re- vealed that SAE and FFs exhibits similar range of interpretability, although SAES displayed an observable but minimal improvement some aspects. Furthermore, in certain aspects, surprisingly, even vanilla FFs yielded better interpretability than the SAEs, and features discovered in SAES and FFs diverged. These bring ques- tions about the advantage of SAEs from both perspectives of feature quality and faithfulness, compared to directly interpreting feature vectors, and FF key-value parameters serve as strong baseline in modern interpretability researct
far. In this work, revisit the interpretability of feature
Introduction
Transformer- basee language models (L.Ms) have exhibited outstanding performance on wide variety
of tasks [10, 35,
45). whereas their underlying mechanisms remain opaque [7 37 50 [7]
38
28 31 This issue has been tackled in the interpretability field, and earlier days, the field has typically adopted top-down approach, where, given candidate features or algorithms, e.g.. syntactic structure, has been inspected where in the original model those are encoded. Nowadays, variety of capabilities emerge in larger LMs, the question tends to be more on the bottom-up. -iature-discovery side: what kind features are encoded in the model?; and how can we discover and control them? This
18
34
age has brought two trends to the interpretability community simultaneously: (i) training an external proxy module dedicated to this purpose, namely, sparse auto-
encoder (SAEs), to decompose neuron activations into simpler basic features (53
24
30
21
16
(proxy-based analysis), and (ii) developing new comprehensive interpretability benchmarks test the quality of discovered features.
36
27
WWrk done at Tohoku University. Project page:
sae
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

This paper explores one overlooked question in the field, to what extent a proxy- basedd artificial decomposition of neuron activations empirically benefits the model interpretation. In other words, feed-forward (FF) layers naturally implement the decomposition of neural activation into set of feature vectors, through the lens of FF as key-value memories [17 (FF-KV analysis), why not first evaluate such organic features in FFs with the newly developed interpretability benchmarks? Proxy- based and VV-KV analysis have complementary advantages, and thus, there is no immediate reason to dismiss the latter. For example, while some proxy-based methods have theoretical motivation to handle superposition, they also have limitations that FF-KV analysis can automatically bypass: proxy modules can additionally expose biases to the interpretation, e.g.. specific features are repeatedly found g 49, II]; the external proxy hallucinates features [22]; and additional computation costs are needed to interpret the model. Furthermore, the FF activations even without any regularization [29]. Thus, if FR-KV and SAE analyses yield comparable results, there several advantages (more simply put, from Occam's 's Razor principle) adopting the former FF-KV analyses.
reported to be naturally sparse
To gauge the (dis)similarities between FF-KVs' and SAES' interpretability level, we perform both automatic evaluation and manual feature analyses. Automatic evaluation with SABBENCII demon- strates surprising similarities between the two approaches. The evaluation scores fell into similar range in all eight metrics in SAEBENCH and the inter-metrics tendencies are also paralleled, e.g.. causal intervention scores poorer than feature disentanglement scores in both methods. One can even observe some advantages of FV-KVs; for example, features in the original FFs tend to avoid feature overlapping, resulting in better absorption scores [T] (i.c., less redundancy) than those in SAFS. These comparable quality is further supported by human manual evaluation of feature qualities. Conceptual features be found with almost equal ease from both FF-KV features and SAEs. These tentatively conclude that features from FV-KVs and SAES serve a quite similar level of interpretability from both quantitative and qualitative perspectives.
In our analysis, we further investigate the faithfulness of proxy-discovered features, considering FF-KV features gold, how large is the overlap between the feature sets of the original FF-KV module and that of the proxy module? We analyzed such an overlap with Transcoder (TC), the closest counterpart to FF-KV, similar counterparts in the original FF module. This aligns with the existing report that SAEs can interpret even random Transformers (22), and perhaps the proxy module hallucinates new features rather than translating the workings of the original FF module, encouraging further research on the faithfulness of the learned features, with FF-KV features grounding points. To sum up, our study reveals that proxy-based methods such as SAES empirically offer very limited advantage over the direct analysis of FF's (i.e., key-value memories). That is, the theoretical advantage of SAEs is not observed empirically, at least through the lens of the current evaluation scheme, and encourages the inclusion of FF-KV features strong baseline when assessing ceature-discoven methods in the interpretability field.
proxy model, and revealed that the majority of TC features do not have
2 Background
2.1 Related Work
Dictionary Learning and LLMs Interpretation. Dictionary learning has been proposed to address
5 20] and this has been applied interpret the internal activations of LLMs. represented by sparse autoencoders (SAEs) [3 g 24 30 21 25). Specifically, these introduce a proxy module to decompose and reconstruct model's activation, and seek interpretable features in it. Apparently. promising results were observed in earlier days: the learned features are highly interpretable and can be directly used to steer the model's 's behavior [46] modification on feature will either eliminate the corresponding behavior, or enhance it.
polysemanticity of the representation [3 52 42
40
14
Mixed Reports on SAE Features. Although the SAEs get increasing attention, concurrent works have brought skeptical views on their success. For example, SAE-based feature steering quality is inferior to simple baselines utilizing activations [5 SAES can learn meaningful features even from randomly initialized Transformer [22]: and they exhibit no clear advantage in downstream tasks and sometimes underperform linear probes that use the model's 's raw activations (6 This study, at a high level, provides additional support for such criticisms of the general advantage of SAEs.
2

---

Interpretability of Feed-Forward (FF) Layers. There have been fair number of studies to
interpret the feed-forward (FF) layer in Transformers directly [3 40 2 The closest work to ours is Geva et al. [17, where FF's can be viewed as key-value memories, and they are interpretable and useful to control the model output. Recent work also indicates that activations in FFs are already these motivate our work to contextualize the
sparse (29), and their neurons can be manipulated 51
bare FF interpretability with SAE works.
2.2 Sparse Autoencoder for Transformer Interpretability
Transformer. Transformer architecture is a stack of multiple modules, such as attention mech- anisms, cerd-forward (FF) layers, and normalization layers. There have recently been increasing endeavors to interpret, especially, neuron activations around FF layers, such as SAEs. Henceforth, vector denotes row vector.
SAE. SAE decomposes and reconstructs the neuron activations, typically after the FF layer (residual stream). That is, let #FF..s Rddmodel be neuron activations after the FF layer, and dSAF denotes the dimension of SAE features. SAE decomposes the neuron activations FFF... and reconstructs #FF..r follows:
Foott ReLU W Vnc + bene)
W dec
biec
with W.n. IR mmodial dEaE W E IRdSAR Xdmodal, bene E RRdSA,, and bate E RRdnoda in the SAE module. ReI.U(-): Rd Rd denotes an element- ReI.U activation. Each activation dimension treated potentially interpretable feature, and the matrix maps each feature dimension to its feature vector in the representation space. This module trained so that the activations are as sparse as possible with sparsity loss to disentangle the potentially polysemantic input neurons.
Transcoder. Notably, as an alternative to SAEs and perhaps the closest attempt to this study, Transcoders have recently been proposed [12]. This approximates the original FF by training sparse MI.P as a proxy to predict FF output FFF... from FF input #PF,,. and its internal activations (E (ddro) are evaluated in the same way as the standard SAE. Still, their work [22 did not clearly evaluate how interpretable the original FF's internal activations are, and this work complements this overlooked question.
2.3 Feed-Forward Layer ValuV Memories
Feed-forward layers in Transformers once project the FF input Frr,, E &Rdmmdee to drp-dimensional representation (dmodel < dFF), applies an element-wise non-linear activation Φ(.) : Rd + IRd, and projects it back, as follows:
#FF... Φ(FFF WK + bk) Wy + by
Σ Φ(#FFF Wk) iEdpr
Wy + bv (2)
where WK E damddel X dpF and Wv E Rpp Xdmodel are learnable weight matrices, and bx E RdF by IR Rddnodel are learnable biases. drF is typically set as 4dmodei- One interpretation of the FF layer is a knowledge retrieval module; that is, the module first creates keys (activations) from an input ærr,, and then aggregates their associated values (feature vectors) Existing studies have analyzed what kind of concept is stored each feature vector of Wv1.. and when they are activated by of 6(IFF,, WK) 17].
3 Comparing FF-KVs with SAES
The feed-forward key-value memory module (FF-KV) inherently performs the same operation as SAEs (although it is somewhat obvious, given that both adopt the MLP architectures): it first decomposes the neuron activations into feature vectors and then aggregates them. This naturally raises a question about how similar the decomposition naturally made by FF-KVs is to that learned by the proxy module, e.g., SAES. We examine several variants of FF-KV-ba V-bseed feature discovery m
See Appendix Aftor the details the implementations
3

---

3.1 Methods
FF-KV. The vanilla FF key-value memories are evaluated with the SAE evaluation framework, treating the key activations as features and the value vectors as feature vectors.
TopK FF-KV. To encourage the alignment with SAE research, also introduce sparsity to activations in FFs by applying a top-k activation function to the key vector, although it has been reported that the vanilla FFs' activations are somewhat already sparse [29]. This keeps only the neurons with the k largest activations in each inference, zeroing out the activation for the rest. We call this TopK FF-KV, defined as follows:
FFF Top-k( WK + bx)
W
by
Normalized FF-KV. The feature vectors of SAE are typically normalized, whereas those in FF not. If a particular feature vector Wyss: has large norm, the magnitude of its corresponding activation may be underestimated. To handle this potential concern, we normalize each row of Wy, and the discounted vector norm is weighted activations. We refer to the method with this post-correction as Normalized (TopK) FF-KV:
FFF out (op-k(((xFF,, WK + bK) ④
Wv
+ bv
8 Ww..,ll2. W Wwaaalll,. .. W Widderialle E Rddrr
(5)
Here, Wv diag(s) (s) Wv. where diag(-) expands a vector Rd a diagonal matrix Rdlxd
3.2 Inference and Feature Discovery
Once a method to obtain activations from the models is determined, one can get an activation history over certain of text. Here, we introduce several notations before going the experiments.
Notations. Feature activations are analyzed through feeding specific texts to models, and the exact text contents will vary depending on evaluation metries. Let us denote set of input texts as S = [s,, 8n.. where each text consists of multiple tokens
[w(R.1]) ((k,2), ... W)), E S, which are used to get feature activations. For brevity, we flatten and re-index the tokens as [wi, W2. ......,, wy]: one can recover the original indices (i,j) indicating text id and token position via : Np..j [1,=) x [m..). e.g., (2) (1, 2). For each token w/. we first collect feature activations a, E [gdeoder with particular method, such as SAE. Here, deoder should be dSAE, drC, dyr, depending on the methods; in other words, each method maximally yield deuder numbers of features FF = (.f.......mmmm Repeatedly collecting the activations over inputs w...... ,i] gives an activation history matrix E &Rddde where each row corresponds each token or, and each column corresponds to each feature (neuron) fp E F, respectively. A E T2' presents where a feature fp was activated in S. Sp = [t. T At.p > 0 and i,_ (t)) S S represents the text subset associated with the feature fp-
SwiGLU activation. Modern LMs adopt SwiGLU gating function 41 for the non- non-linear acti- vation of FFs (--). The existing work (54) showed the compatibility of SwiGLU activation with FF-KV analysis, and thus, the above methods ($ 3.1 can be naturally implemented with SwiGLU. For example, on top of the SwiGLU activation, the TopK FF-KV can be written as follows:
ØFF
Top-k((
cæFF, (Wp-k((WWaFFF....
Swish W KFFF,,. Wy + by
(6)
Top-k(( (Wg-FF( O Swish W WxXFFF.))) ))
W Wy
+ by
(7)
VEdpF
where, WG E Eddr Bddey Xdmodel is the gating matrix.
4 Experiment 1: Automatic Evaluation
We evaluate FF-KVs, SAEs, and Transcoders using the metrics from SAEBENCH report the Feature Alive Rate as complementary statistic.
27 We also

---

4.1 Evaluation Metries
Here, we give high-level description of each metric, and details are shown in Appendix B
Feature Alive Rate aggregates how many features are alive, of dooder features. A positive value of At.p is regarded as the activation of p-th feature in Zt. An indicator function, Xxpppl H 1 if max (x.p)) > 0 else 0. judges the feature fp is alive (activated at least once) and the following
score is calculated: Di x(AA.ss)
score of I indicates that all features are activated at least once.
Geoder
Explained Variance evaluates how well the proxy module reconstructs the original activations, and the FF-KV methods (without proxies) can automatically get perfect score (=1) since this is the original module as is.
Absorption Score evaluates how many features a particular simple concept (e.g., word starting with "S") is split into. A higher value implies that many features needed to emulate the targeted single concept, and thus, the feature set is redundant.
Sparse Probing evaluates the existence of specific informative features (e.g., sentiment) and their generalizability to held-out data. This is measured based on the accuracy of probe classifiers trained on the activation patterns to predict the properties of unseen inputs for the proxy module.
Auto-Interpreeatton Score evaluates how easily the activation patterns of the feature can be summa- rized in natural language (e.g.. "a feature related to accounting"). Specifically, given a text subset S, for the feature f 1 an LLM is requested to summarize the feature concept and then predict the (binary) feature activation the held-out set based on the summary. following Paulo et al. [36] A score indicates that features can be perfectly summarized, and their activations are predictable.
Spurious Correlation Removal (SCR) evaluates how well spuriously correlated two features (c.g.. gender and profession) are disentangled from different features, using the SHIFT 32 data. A score of 1 indicates perfect disentanglement Notably, its extension, Targeted Probe Perturbation (TPP) score, was also invented as a supplemental metric in SAEBENCH [27]. The TPP results are shown in Appendix and yielded consistent results with SCR. Notably, SCR and TPP consider top-K activations in evaluations (we adopt K 20 here, following existing studies 27 and results for different K are shown in Appendix B
RAVEL 23 further evaluates the feature overlap and disentanglement, but slightly from a different angle from SCR and TPP. Specifically, this concerns the separability and controllability of multiple capital
different attributes of the same entity (e.g., Japan continent Asia, Japan Tokyo). The RAVEL score be decomposed into two complementary scores: (i) Isolation score-the the probability that all non-edited attributes remain unchanged; and (ii) Causality score- stireethe probability that the edit successfully changes the target attribute. We report both scores for the RAVEL results to clarify the fine-grained properties.
4.2
LMs and Proxy Modules
LMs. We evaluate FFs in all layers of five LMs: Gemma-2-2B, Gemma-2-9B [43] Llama-3.1
GPT-2 (39. and Pythia-70M [4. Due to space limitations, the results of the middle layers of Gemma-2-2B (layer 13) and Llama-3.1-8B (layer 16) shown in the main part of this paper, and the results for other layers and models are shown in Appendix C We also target randomly initialized LLMs as baselines, given the assertion that SAEs can even interpret randomly initialized Transformers. In our main experiments, we set k = 10 for the TopK FF-KV, and we additionally compare results under varying le values.
8B 445
SAES. We use pretrained SAEs from Gemma Scope [30 (width 16k) for Gemma-2-2 and Gemma- 2-9B. Llama Scope [2I] (width 32k) for Llama-3 3. 1-8B, and SAELens 7 for GPT-2. All SAES are trained on FF outputs.
'It is also marked at which token the text feature was activated. 'We found SCR and TPP scores are highly unstable, and one may have to treat them supplementary results. See Appendix B for details.
5

---

Coder Status
Concept Detection
Model SAE Type
Feat. Alive f
Expl. Var.
Absorption +
Sparse Prob.
SAE
0.988
0.699.10.000
0.087=:0173
0.846::0.161
Gemma-2 2B
Transcoder
1.000:0.0000
0.637+00.00
0.025=+0.116
0.854:+0.149
FF-KV
0.009
1.000.10..00
0.000110.001
0.827:10.15
FF-KV (Norm.)
1.000:1.000
1.000:10.000
0.000
0.826
TopK-FF-KV
0.000
0.100
0.000
0.768 16118
TopK-FF-KV (Norm.)
0.984+0.000
0.160+00.00
0.000=00000
0.768=+0.16 168
Random Transformer
1.000
1.000
0.007
0.796.10.067
SAE
1.00000.000
0.594+00.000
0.097:20.332
0.879:+0.22
Llama-3. 8B
Transcoder
FF-KV
1.00000..000
1.0000+0.000
0.000=10.001
0.876:10.098
FF-KV (Norm.)
1.000=0.000
1.000+0.000
0.000=+0.000
0.876=10.09
TopK-FF-KV
0.992:+0000
0.238:+00000
0.000=10.001
0.832:10.15
TopK-FF-KV (Norm.)
0.992++0.000
0.238-1:.000
0.000+0.000
0.832
Random Transformer
1.000.1:.000
1.000:10.000
0.002:10.066
0.837:10084
Feature Explanation
Feature Disentanglement
Model SAE Type
Autointerp f
RAVEL-ISO
RAVEL-CAU 1
SCR (k=20) f
SAE
0.782-10.274
0.985:10.027
0.002:10.006
0.170:1. 191
Gemma-2 2B
Transcoder
0.790
0.940+10.040
0.010+0.017
0.104+1.11 178
FF-KV
0.710:10:26
0.952:10.05
0.012400021
0.041
FF-KV (Norm.)
0.706=:0.255
0.952++0.035
0.012+00021
0.041 120
TopK-FF-KV
0.772:t0.27
0.943:10.039
0.009+00015
0.045=+0.10 105
TopK-FF-KV (Norm.)
0.773.10.269
0.002
0.009
0.029
Random Transformer
0.679:00248
0.004+4.022
SAE
0.817110277
0.993
0.002.0.007
0.219
Llama-3.1 8B
Transcoder
FF-KV
0.751 1.0.44
0.955.10.044
0.007110.01
0.018.10.070
FF-KV (Norm.)
0.749:10.245
0.954.10.04
0.007:10012
0.046-10.071
TopK-FF-K?
0.807
0.954.10004
0.006:10.01
0.030
TopK-FF-KV (Norm.)
0.807:::0256
0.955:100043
0.006:10.010
0.029-10.443
Random Transformer
0.656:10.237
0.053-10:239
Table Overview of the SAEBENCH evaluation results for the middle layer of Gemma-2-2B (layer 13) and 1lama-3.1-8D (layer 16). Results reported as mean +:2 standard errors of the mean multiple seeds/settings. Norm. represent the normalized FF-KV. We also present the scores for randomly initialized FF layers, which serve the baseline. No substantial difference between FF-KV and SAEs is observed.
Transcoders. We use pretrained Transcoders (TCs) from Gemma Scope 30 for Gemma-2-2B and one from the original paper [2] for GPT-2, respectively
4.3 Results
Overall. Table shows the results for each interpretability method. First of all, the SAE-based results and FF-KV results rendered similar tendencies. In each metric, the absolute difference between the scores from different methods is typically much smaller than seed/layer variance. In addition, the difficulty of each task (metric) is aligned across the tasks; for example, SAEs and FF-KYs achieved higher RAVEL-Isolation scores than RAVEL- AAVEL-Causality scores. These results suggest that, even with the activations the original FF module, comparable interpretability can realized compared to proxy-based methods, i.e., SAES and Transcoders.
SSee Appendix for more details on the SAEs and Transcoders

---

Absorption
Sparse Probing(Top-5)
Autointerp
0.006
1.5
Evaluation Score
1.2
0.004
1
0.002
0.8
0.6
-0.002
0.5
0.4
-0.004
0.2
RAVEL-ISO
RAVEL-CAU
SCR(K=20)
1.05
0.6
Evaluation Score
0.04
0.4
0.02
0.2
0.95
0
0
0.9
0.2
-0.02
0.85
1
10
100
1000
1
5
10 2
100
1000
10
5 100
5.000
Value of TopK FF-KV (log scale)
Figure 1: Evaluation scores for TopK FF-KVs Layer 13 on Gemma-2-2B. under different sparsity value k. A lower & indicates higher sparsity. Shaded areas denote =2 standard errors of the mean, computed across multiple seeds and evaluation settings.
Inter-Methods Similarity. To mention specific similarity among the methods, causal intervention (RAVEL-Causality) was difficult for both SAES and FF-KVs; that is, FF-KV methods inherit the limitation of SAEs. In contrast, feature isolation well realized in FF-KVs, similarly to SAEs, even without any specific feature disentanglement regulation in FF-KVs, based on the high scores in RAVEL-isolation. Later layers tend yield high RAVEL-isolation score, and vice versa, especially in the case of Fs-KVs. This shows parallel with the existing observation that FFs in later layers have more semantic features 17 and attributes targeted in the RAVEL dataset might not well-shaped in earlier layers.
Inter-Methods Difference. To highlight the differences among the methods, FF-KV meth- ods can achieve perfect explained variance by definition (i.e., zero reconstruction loss as the original model is directly analyzed), whereas SAES cannot. In addition, FF-KVs exhibited better absorption scores; that is, simple single concept is not overly split into multiple con- cepts in FF-KVs than in SAEs, and in this sense, features in FF-KVs are less redundant. Sparse- probing results are comparable or slightly better in FF-KYs; representative features are encoded and generalizable SAES and FV- SAES achieved slightly but consistently better Auto-interpretablity and SCP (although around zero) scores, which are only the advantage of SAEs compared to FF-K V-based analyses.
Isolation Score 1
Cause Score f
0.4
10k
15k
10k
15k
FF Size (Log Scale)
FF Size (Log Scale)
TFpK-FF-KV
FF-KV
the same extent in both
Figure 2: Relationship between FF hidden dimen- sion size (model scale) and RAVEL scores.
FF-KV Variants. Within the FF-KV variants, TopK and normalization effects were generally small. Vanilla FF-KV features already exhibited a reasonable interpretability.
TopK Effects. I shows the relationship between the value of the TopK FF-KV (x-axis) and the SAEBENCH evaluation scores (y-axis). The increase of sparsity level leads to inconsistent results, for example, higher sparse probing (top-5) score but a lower autointerpretation score, suggesting that higher sparsity is not always better, at least for interpretation FF-KV,
7

---

FF-Scaling Effects. Figure shows the relationship between FF hidden representation size (model scale; x-axis) and RAEL interpretability scores (y-axis). These results suggest that FFs with larger hidden dimension size do not always get better interpretability results, suggesting that just extending the hidden dimension size of FFs into that of SAEs does not lead to better interpretability. See Appendix E for other metrics.
5 Experiment 2: Human Evaluation
Our results in Section suggest that FFs' internal activations have overall comparable interpretability to SAEs/raansoddess based on automatic evaluations. In this section, we further perform follow-up manual inspection on the interpretability of features extracted from layer 13 of Gemma-2-2B" FF-KV, SAE, and Transcoder. We specifically explore the following questions: 1) Do features from the FV-KV, SAE, and Transcoder appear equally interpretable to humans? 2) How accurately humans infer the origin of the feature?
5.1 Settings
Table 2: Number of superficial, concep-
We randomly sampled 50 features each from the FF-
tual, and uninterpretable features.
KV, TopK FF-KV, SAE, and Transcoder of Gemma-2-2B model, yielding total of 200 features. Each feature Sp is presented with its top-ten associated texts E Sp based on the activation magnitude over subset of WpenWeTText cor- cor- pus [19] (200M tokens in total). From the annotator 's view, the presentation order of features is randomly shuffled, and their origins remain hidden throughout the experiment. The annotations in this section were conducted by one of the authors.
Coder Superficcal Uninterp.
FF-VV K-FF-KV
8
36
9
9
32
SAE
9
35
TC
16
11
23
Interpretability Evaluation. One annotator judges the qualitative quality of a feature using three categories: 1) superficial Feature: activates on shallow surface patterns (e.g., particular word, such as "the," punctuation, digits); 2) Conceptual Feature: activates on higher-leve concepts spanning multiple tokens (e.g., sentiment, topics); or, 3) Uninterpretable: exhibits no clear activation pattem"
Feature Origin Judgment. We also designed a task to predict from which module feature originates, only based on the feature activation patterns in texts, with the same data, to exploratorily find any difference between these activation patterns. If annotators can not guess which module is used to obtain the given feature, the used methods would have the same level of feature extraction ability. One annotator conducted this analysis, and as preliminary training, the annotator had first learned several activation patterns the held-out set, paired with their module names.
Table 3: Origin judgment accuracies of features.
Origin
Judging Acc.
FR-KV
0.86
K-FF-KV
0.28
SAE TC
0.13
0.18
5.2 Results
Interpretability Evaluation. The results are presented in Table First, the number of conceptual features is nearly the same across the four interpretability methods. In this sense, the quality of the obtained features is comparable. Transcoders could find a larger number of features that are interpretable (superficial conceptual), but the ratio of superficial features is higher than in the other methods.
Feature Origin Judgment. Table 3 shows the results. The annotator could not correctly predict the original model, except for the FF-KV methods. Through interviewing the annotator, we found that they could identify the FF-KV features by relying on superficial patterns in the magnitude and variance of feature activations (FF-KV tends to have small value with high variance), rather than the represented concepts. Using TopK FF-KV (K-FF-KV) alleviates this distinction pattern, and thus,
We provide the actual text annotate in Appendix

---

if one wants to render a visualization of activations similar to that of SAEs, TopK FF-KV should be preferred. The low accuracies for K-FF-KV, SAE, and TC support that their discovered features and activations are similar to each other, as the human evaluator could not distinguish them.
6
Analysis: Feature Alignments
We analyze how similar features discovered by proxy methods, e.g., SAEs, are to the FF-KV ones.
6.1 Settings
To investigate the alignment of features from different interpretability methods, specifically focus on those from FF-KV and Transcoder (not SAE, as the closest counterpart to FF-KV). In this analysis, we used layer 13 of Gemma-2-2B, which showed reasonable performance in the automatic evaluation experiment. Given an r-th feature vector in the FF Wy).r)) we find the index of the most aligned feature vector in Wder from Transcoder, based on their cosine similarity: u = arg m [r.rl W deee(.k)) Note that when analyzing features in TC, the searching direction will be opposite: arg max; (Wdee dee|s.r) Wv)..)). We call these max-cosine scores MCS.
We first perform a quick check of the correlation between MCS and the semantic feature alignment. For each MCS bin, we sampled ten feature pairs. Then, for feature pairs (f f within a specific MCS range, annotators manually inspected their alignment. Similarly to the previous experiment, each feature fp is accompanied by ten associated texts E S, from a subset of the Open Web corpus [99.. We consider a pair matched if these three criteria meet: 1) The two features generally represent the same concept (c.g., sentiment, topic); OR 2) The associated texts for the two features Sp2 exhibits 8/ 10 overlap; OR 3) The topics of the texts from two features coincide. The annotations in this section were independently conducted by a different author from that of $
6.2 Results
Validity of Cosine-Based Alignments. The results of alignment analysis are shown in Fig- ure 3 This clearly shows that higher cosine sim- ilarity entails their semantic alignment. Based on these results, we tentatively regard feature pair with cosine similarity above 0.9 as aligned, and the similarity below 0.3 unaligned in the following analyse.
Aligned Features (count)
100
60
40
20
0.2
Large Number of Unaligned Features. Based on the above criteria with co- sine similarity, 41% (=3.780/9,2166 and 66% (=10,835/16.3844 features in FF-KV and Transcoder are unaligned with each other, respectively. In contrast, 5.7% (=527/9,216) and 3.2% (=527/16.355 That is, there are a large number of unaligned features between FF and Transcoder, clarifying that the same level of interpretability from different methods in automatic evaluation ($ 4 was not simply due to their similar feature sets. In the next paragraph, we manually analyze the unaligned features.
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
MCS Range
Figure 3: Histogram of aligned features numbers distribution for each MCS bin, between the FF and Transcoder.
features are regarded as aligned in FF-KV and Transcoder, respectively.
Feature Complementarity. We manually analyze three sets of features: (a) aligned features (FF- KVNTranscoder), (b) FF-KV features not aligned with any Transcoder feature (FF-KV) (FF--V (Transcoder). and (c) Transcoder features not aligned with any FF-KV feature (Transcoder's (FanscooddrrFF-KV) FF-KVV. The analysis target is the same [5] the features are classified into three categories of superficial, conceptual, and uninterpretable. Table 4 shows the distribution of feature categories each feature set. First and interestingly. the unaligned features both in FFs and TC have fair amount of conceptual ones. In particular, 32% of features that are unaligned with FF' were conceptual.
*See Appendix F for examples of the feature pairs we use for annotation as well as how we decide the threshold.

---

Table 4: Number of superficial, multi- token conceptual, and uninterpretable features in aligned/unaligned features between FF (FF-KV) and Transcoders (TC).
Discussion. Why are there so many unaligned features? One optimistic view is that TC successfully decomposed FF features into simpler ones, resulting in decomposed features being orthogonal to the original FF features, al- though this may offer a potential side effect of feature absorption, which is suggested by relatively large number of superficial features in TC FF-KV and an already good absorption scores achieved by FF-KVs (Table[]. One more pessimistic view is that Transcoder invented completely new features that are not in the original FV-KV, which is in line with the fact that SAE can interpret even randomly initialized Transformers [22]. Our analysis alone can not fully distinguish between the two cases, but this fact of frequent misalignment deserves motivation to further explore the faithfulness of the learned features in proxy modules. Features in the FF-KVs will serve as grounding points evaluate such faithful evaluation, top of our first extensive attention to FF-KVs in the context of SAE research.
Coder
Superficial Coneept Uninterp.
FF-KYOTC TF-KY TC
7
16
27
41
6 14
23
7 Conclusion
In this work, we revisit the interpretability of feature vectors already represented in the feed-forward (FF) module, as a strong baseline to SAEs. Our results show that the original FF feature vectors already exhibit reasonable interpretability comparable to that of sparse autoencoders (SAEs) and Transcoders on both comprehensive benchmark and human evaluations. We further demonstrate that a large portion of the features between the FF and the Transcoder are not aligned, and manual analysis suggests a potential feature over-splitting or hallucination of new features in the proxy module. To sum up, our results demonstrate that SAEs and Transcoders offer only limited advantages over the direct amalysis of feed-forward key-value (FF-KV) representations. This finding highlights the lack of a strong and simple baseline within the interpretability community and underscores the importance of including FF-KV analysis fundamental reference point for evaluating interpretability methods. It also encourages future work to consider both moodll-nneernal parameters and proxy-module parameters when pursuing feature-discovery based interpretability of large language models.
Limitations. The feature dimension of SAEs and Transcoders we used was fixed; more diverse configurations should be examined in the comparison, although publicly available pre-trained are limited, and prior work shows that simply scaling width does not neces- sarily improve SAEs and that there is not universally best architecture choice [27]. Not all models are accompanied by Transcoder results: still, training Transcoders on all layers of, e.g., 9B-parameter models is prohibitively costly under an academic budget. We conducted only few qualitative case studies on the effect of & for TopK FF-KVs and on FF size. Although our analysis showed a discrep- ancy between FF-KV and Transcocder features, the interpretation of this difference (faithfulness of the learned features) remains unclear; future work should elaborate on this point.
Impact Statement. Our findings indicate that SAES and Transcoders do not consistently outperform the original feature vectors in FFs with respect to interpretability. We underscore the need to reassess both the interpretability and. potentially, the reproducibility of the previously reported advantages of SAES. In sense, our study supports the use of inherently black-box neural LLMs while setting aside the interpretability issue, their FFs appear to possess a certain degree of interpretability. Nevertheless, one of the ultimate objectives of this line of research should remain the development of models that are interpretable by design.
Ethics Statement. Our research primarily relied on publicly available models and datasets, and strictly adhered to their respective licenses (see Table 7) For human evaluation, we collected data as described in $ 5.1 The data were collected with participant consent, and we ensured that responses were anonymized to prevent them from being traced back to individuals. To promote transparency and reproducibility, we have made the collected data, along with all code used in our experiments, publicly accessible. Comprehensive details of our experimental setup are provided in each section and the appendix to ensure reprodueibility.
10

---

Acknowledgment
Author Contribution. M. Ye led the research project, implemented the experimental pipeline. conducted the SAEBENCH evaluation, designed the human evaluation task, and carried out one of the human evaluation studies. T. Kuribayashi initially proposed the idea of comparing the transcoder with FF-KV augmented by a TopK activation function and was deeply involved in regular discussions throughout the project. Inaba provided valuable feedback on implementation, conducted one of the human evaluation studies, and actively contributed to project discussions. J. Suzuki provided overarching guidance and feedback at all stages of the project as well as computational resources.
M. Ye drafted the initial version of the manuscript. T. Kuribayashi offered extensive feedback and revisions on the writing. T. Inaba authored the impact and ethics statements. J. Suzuki contributed valuable insights into the overall framing and positioning of the paper.
Acknowledgmments. We want to express our gratitude to the members of the Tohoku NLP Group for their insightful comments. And special thanks to Charles Spencer James for assisting in determining the MCS threshold for text alignment through human annotation. This work was supported by the ISPS KAKENHI Grant Number JP24H00727, JP25KJ06300; JST Moonshot R&D) Grant Number JPMJMS2011-35 (fundamental research); JST BOOST, Japan Grant Number JPMJBS2421.
References
[1] Authropic. The claude 3 model family: Opus, sonnet, haiku, arXiv preprint arXiv:2303.08774 2024.
[2] O. Antverg and Y. Belinkov. On the pitfalls of analyzing individual neurons in language models. In The Tenth International Conference on Learning Representations, 2022.
[3] S. Arora, Y. Li, Y. Liang, T. Ma, and A. Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-44 49..-495.222 2018
[4] S. Biderman, H. Schoelkopf, Q. G. Anthony, H. Bradley, K. O'Brien, E. Hallahan, M. A. Khan, S. Purohit, U. S. Prashanth, E. Raff, A. Skowron, L. Sutawika, and O. Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning. pages
PMLR, 2023.
[5] S. Bills. N. Cammarata, D. Mossing, H. Tillman, L. Gao, G. Goh, I. Sutskever, J. Leike, J. Wu, and W. Saunders. Language models can explain neurons in language models, 2023. httpst html
//oenaappblic. blopeaiippli.bbbbb..oooooo core. windows. www//eeuunnnneee http:/ow.....ee.oooo.oeeeeeeaaea
[6] J. Bloom. Gpt2- -5--3ll-- 32k-m3kk-mmmmp-m mlt--kk---u-sseeeee. 2024. URL https: haap::::/hugggggaae co/
j jbloom/GPPP 100oom/GPT2--mal--001--- v5- 32k mlp-out SAEs
[7] J. Bloom, C. Tigges, A. Duong, and D. Chanin. Saelens, 2024. httphhttt//tthhhhe com/
j JbloomAus/SAELenn
[8] Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. Turner, C. Anil, C. Deni- son, A. Askell, R. Lasenby, Y. Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, 7. Hatfield- Dodds, A. Tamkin, K. Nguyen, B. McLean,, J. F. Burke, T. Hume, Carter, Henighan, and C. Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL
cirourtiicctstts. cibrcuiss.puu//000 wubppuubb200 202/
monosemantic- featuccffeaureeeeeeeee wwtp:/iiixxx...m html
[9] T. Bricken, J. Marcus, S. Mishra-Sharma, M. Tong, E. Perez, M. Sharma, K. Rivoire, Henighan, and A. Jermyn. Using dictionary learning features as classifiers. Trans- former Circuits Thread, 2024. URL
features- wwwssifiereseeeooo iwtrre/iiidxx..ee html
[10] T. Brown, B. Mann, N. Ryder, M. Subbiah, D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
11

---

[11] D. Chanin, J. Wilken-Smite T. Dulka, H. Bhatnagar, and J. Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. 2024.
[12] Dunefsky, P. Chlenski, and N. Nanda. Transcoders find interpretable Ilm feature circuits. In Advances in Neural Information Processing Systems, volume 37, pages 14375-24410 2024.
[13] N. Durrani, H. Sujjad, F. Dalvi, and Y. Belinkov. Analyzing individual neurons in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4865-488 48800 Association for Computational Linguistics, 2020.
[14] N. Elhage. T. Hume, C. Olsson, N. Nanda, T. Henighan, S. Johnston, S. EIShowk, N. Joseph, N. DasSarma, B. Mann, D. Hernandez, A. Askell, K. Ndousse, A. Jones, D. Drain, A. Chen, Y. Bai, D. Ganguli, L. Lovitt, Z. Hatfield-Dodee
J. Kernion, T. Conerly, S. Kravec, S. Fort, S. Kadavath, Jacobson, E. Tran-Johnson, Kaplan, J. Clark, T. Brown, S. McCandlish,
D. Amodei, and C. Olah. Softmax linear units. Transformer Circuits Thread, 2022. URL pub/ 20002///////iii/ htwp///dxx.n.....////////// html]
tttp::://transfoomer-cooo
[15] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The pile: An 800gb dataset of diverse text for language
modeling. urXiv preprint arXiv:2101 0000000 2020.
[16] L. Gao, T. D. la Tour, H. Tillman, G. Goh, R. Troll, A. Radford, I. Sutskever, J. Leike, and J. Wu. Scaling and evaluating sparse autoencoders. In The Thirteenth International Conference on Learning Representations, 2025.
[17] M. Geva, R. Schuster, Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 5484-5495. Association for Computational Linguistics, 2021.
[18] M. Geva, A. Caciularu, K. Wang, and Y. Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 30 45. Association for Computational Linguistics, 2022.
[19] A. Gokaslan and V. Cohen. Openwebtext corpus, 2019. URL htt: git.olo
hto/peenNebTextCorpuss
[20] W. Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[21] Z. He, W. Shu, X. Ge, L. Chen, J. Wang, Y. Zhou, F. Liu, Q. Guo, X. Huang, Z. Wu, Y.-G. Jiang, and X. Qiu. Llama scope: Extracting millions of features from llama-3. with sparse autoencoders. arXiv preprint arXiv:2410. 20520, 2024.
[22] T. Heap, T. Lawson, L. Farnik, and L.. Aitchison. Sparse autoencoders can interpret randomly initialized transformers. arXiv preprint arXiv:2501. /7727, 2025.
[23] J. Huang, 7. Wu, C. Potts, M. Geva, and A. Geiger. RAVEL: Evaluating interpretability methods on disentangling language model representations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages Association for Computational Linguistics, 2024.
[24] R. Huben, H. Cunningham, L. R. Smith, A. Ewart, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024.
[25] T. Inaba, G. Kamoda, K. Inui, M. Isonuma, Y. Miyao, Y. Oseki, B. Heinzerling, and Y. Takagi. How a bilingual Im becomes bilingual: Tracing internal representations with sparse autoencoders. arXiv preprint arXiv:2503. 0000400063344200 2025.
[26] S. Kantamneni, J. Engels, S. Rajamanoharan, M. Tegmark, and N. Nanda. Are sparse autoen- coders useful? case study in sparse probing. In Forly-second International Conference on Machine Learning, 2025.
[27] A. Karvonen, C. Rager, J. Lin, C. Tigges, J. Bloom, D. Chanin, Y.-T. Lau, E. Farrell, C. McDougall, K. Ayonrinde, D. Till, M. Wearden, A. Conmy, S. Marks, and N. Nanda. SAEBench: comprehensive benchmark for sparse autoencoders in language model interpretability. In Forty-second International Conference on Machine Learning, 2025.
12

---

[28] K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg, Inference time intervention: Eliciting truthful answers from a language model. In Advances in Neural Information Processing Systems, volume 36, pages 31451-41530, 2023.
[29] Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. Reddi, K. Ye, F. Chern, F. Yu, R. Guo, and
S. Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers.
In The Eleventh International Conference on Learning Representations, 2023.
[30] T. Lieberum. S. Rajamannhhaan, A. Conmy, L. Smith, N. Sonnerat, V. Varma, J. Kramar, A. Dragan, R. Shah, and N. Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, 2024.
[31] S. Marks and M. Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. In First Conference on Language Modeling, 2024.
[32] S. Marks, C. Rager, E. Michaud, Y. Belinkov, D. Bau, and A. Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference on Learning Reprexentations, 2025.
[33] N. Nanda and J. Bloom.
Transformerlens, 2022.
URL https:// comb
TranssormerLensOrgg wtw.rrrggg.raaasffrreeeeeeee
[34] C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hermandez, S. Johnston, A. Jones, Kernion, L.. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, McCan- dlish, and C. Olah. In-context learning and induction heads. arXiv preprint arXiv:2209.1... 11895, 2022.
[35] OpenAl. Gpt-4 technical report. arXiv preprint Stti:::300.0077744 2024.
[36] G. S. Paulo, A. T. Mallen, C. Juang, and N. Belrose. Automatically interpreting millions of features in large language models. In Forty-second International Conference on Machine Learning, 2025.
[37] T. Pimentel, N. Saphra, A. Williams, and R. Cotterell. Pareto probing: Trading off accuracy for complexity. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 31 3138-3153, Association for Computational Linguistics, 2020.
[38] T. Pimentel, J. Valvoda, N. Stoehr, and R. Cotterell. The architectural bottleneck principle. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 11459 11459-11472. Association for Computational Linguistics, 2022.
[39] A. Radford. J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Lan- guage models are unsupervised multitask learners. OpenAl, 2019.
URL
http:::/cddn..peeeee wwt.e/pppaaaaccmbeeeeeeeeeeeeeee wwon//beter-languuee unsupervised_mne caterdd_llttttttttttomae learners. .dff
[40] H. Sajjad, N. Durrani, and Dalvi. Neuron-level interpretation of deep NLP models: A survey. Transactions of the Association for Computational Linguistics, 10: 10:1285-1303, 2022.
[41] N. Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.052022 2020.
[42] X. Suau, L. Zappella, and N. Apostoloff. Finding experts in transformer models. arXiv preprint arXiv:2005.07644 2020.
[43] G. TTem. Gemma 2: Improving open language models at practical size. arXiv preprint
arXiv:2408.000 18, 2024.
[44] G. Team. Gemini: A family of highly capable multimodal models.
arXiv preprint
arXiv:2312.11805. 2024.
[45] L. Team. The laaaa herd of models. arXiv preprint arXiv:2407.21788 2024.
[46] A. Templeton, T. Conerly, Mareus, J. Lindsey, T. Bricken, B. Chen, A. Pearce, C. Citro, E. Ameisen, A. Jones, H. Cunningham, N. L. Turner, C. McDougall, M. MacDiarmid, C. D. Freeman, T. R. Sumers, Rees, J. Batson, A. Jermyn, S. Carter, C. Olah, and
T. Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trunsformer Circuits Thread, 2024. URL. scaling www.smmantiictyy/ooooooooooooa winpttttiitttttttiinttitii html
wwwearrfrooorrrrrrrttttteeeee circuits pub/2024/ httpr//roorreeriiuuuuutttoe///ee
13

---

[47] I. Tenney. D. Das, and E. Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4593-4601. Association for Computational Linguistics, 2019.
[48] C. Tigges. Pythiaz7mmm-ddedinedmmmpps... 2024, /ttpptttt////uggtgto.....////// Pythia-70m-deduppa htt.:ipedddd..aaaaprrreeessseee
[49] N. L. Turner, A. Jermyn, J. Batson, and J. Batson. Measuring feature sensitivity using dataset filtering. Transformer Circuits Thread, 2024, URL http:htttstt/tasstttrtteeaee ormer circuits pub/
[50] E. Voita and I. Titov. Information-thheorticc probing with minimum description length. In Webber, T. Cohn, Y. He, and Liu, editors, Proceedings of 2020 Conference on Empirical Methods in Natural Language Processing, pages 183-196. Association for Computational Linguistics, 2020.
[51] Z. Wu, A. Arora, A. Geiger, Wang, Huang, Jurafsky,
Manning, and C. Potts. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders. In Forty- second International Conference on Machine Learning, 2025.
[52] Xin, J. Lin, and Y. Yu. What part of the neural network does this? understanding LSTMs by measuring and dissecting neurons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 5823-5830, Association for Computational Linguistics, 2019.
[53] Z. Yun, Y. Chen, B. Olshausen, and Y. LeCun. Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10. Association for Computational Linguistics, 2021.
[54] S. Zhong, M. Xu, T. Ao, and G. Shi. Understanding transformer from the perspective of associative memory, arXiv preprint (iX.::25555.9488 2025.
14

---

A Implementation Details on FF-KVs
A.1 Overall Framework
We implement FF-KVs the custom_ sae class provided by SAEBENCH 77. the activation of the FF sublayers, we apply the hook-based approach. The encode method simulates the FF's forward pass up its neuron activations. takes an input tensor x, injects input hook point, and captures the subsequent neuron activations using another hook. Conversely, the decode method simulates the FF's transformation from its neuron activations its output. It accepts a tensor of neuron activations, injects them at the corresponding hook point, and captures the FF's final output. A forward method is also provided, performing the full pass through the FF block using hooks to inject input and capture the final output. This framework allows for the direct examination of an FF's feature extraction and signal reconstruction capabilities if were an SAE, providing a unique lens for interpreting learned representations within large language models.
To faithfully reproduce
the FF's
A.2 FF-KV Implement Details
The core principle is to map the FF's operations to the conceptual stages of SAE:
Input.
Activations entering the FF block as the input our pseudo-SAE.
Feature Representation (Encoding). The FF's internal neuron activations, captured after the non-linear activation function and any gating, are interpreted as the SAE's latent features. The dimensionality of this feature space is equivalent to the FF's hidden dimension. The effective "encoder weights" are the FF's input weights.
Reconstruction (Decoding). The FF's output. which is typically added to the residual stream, is considered the reconstructed input. The effective "decoder weights" and "decoder bias" are the FF's output weights and output bias, respectively.
A.3 TopK FF-KV Implement Details
The TopK FF-KV extends the FF-KV framework by enforcing a strict TopK sparsity on the interme- diate feature representations. The only modification from a FF-KV resides in the encode method. After obtaining the FF's internal neuron activations, k-sparsity constraint is applied. For each input position (c.g., token sequence), the method identifies the & neuron activations with the largest absolute values. All other neuron activations for that position are set to zero. This results in a feature vector where, at most, dimensions are non-zero.
A.4 Normalized FF-KV Implement Details
The FF block's s original output weights (which serve as the "decoder weights")) are Lnz-normalized along their feature dimension. The original norms of these weight vectors are stored. The encode and forward methods remain unchanged from their respective base FF-KV implementations. The key difference lies the decode method. Before applying the normalized "decoder weights", the input feature activations are first scaled by the stored original norms. This step ensures that the magnitude the reconstructed output appropriately reflects the original scaling of the FF block's ' output projections, despite the normalization. For models that include final normalization step after the FF output (c.g., Gemma-2 Models), this step is also applied to maintain fidelity with the original model's computation.
A.5 Transcoder Implement Details
load the Transcoder weight into the JumpReL.U class of SAEBENCH. While evaluating it, we follow the instructions of Gemma-Scope paper [30] to load model weights with folding applied.
15

---

Also the "core" suite, this metric is computed on a 0.4 M-token subset of B Details on Metrics
We provide detailed definitions of the metrics used our main results, alongside the experimental settings for each metric.
Feature Alive Rate. This metric belongs to the "core" evaluations in SAEBENCH, It counts how many features are alive out of the deoder total features. feature is deemed active when its activation exceeds O. The evaluation is conducted on M-token subset of OpenWebText [1].
The metric is especially relevant for Topk FF-KVs employing ToPK activation, ensuring that the mechanism does not repeatedly select only a small subset of neurons.
Explained Variance. OpenWebText [19).
Absorption Score. Feature absorption 11 stems from feature splitting 8 49), in which newly uncovered features become overly specific. A concrete example is a feature that activates only on "U.S. cities except New York and Los Angeles."
The metric targets first-letter classification task, measuring situations where the main feature for letter fails to fully capture the concept of "first letter", and other features compensate. Specifically, it evaluates all 26 letters with the prompt "/word/ has the first letter:""
Given primary features Smain (e.g., selected via sparse probing) and auxiliary features Sabs, the absorption score for one input
Eiesar ai di
Absorption
Eiesa aj di EieSmiin a; de-p'
where a; the activation, de the unit decoder direction, and p the ground-truth probe direction. We use the default hyperparameters.
Sparse Probing. Sparse probing. introduced by Gurnee et al. [20] evaluates the aligmment between individual features and prespecified concept c. It has hyperparameter that specifies how many features are used when training the probe. For each feature h,
8 = EE [x)( Exex [hy (z)]
where Xt and X denote inputs with and without e, respectively. The top features by 8, serve as inputs logistie-rerressonn probe; the probe's 's test accuracy constitutes the sparse-probing score. We again employ the default hyperparameters.
Iuto-Interpretttion Score. This evaluation has two phases: generation and scoring, In the generation phase, it obtain SAE activations, annotate each token with its activation value for the feature under consideration, and prompt an L.LM generate explanations based on these annotated activation patterns. The scoring phase constructs a test set for each feature containing 14 examples, exactly two of which are activated texts. The LLM must label each of the 14 texts as activated or not; the resulting prediction accuracy yields the auto-interpretation score.
The dataset is subset of the copyright-free version of the Pile 15 comprising M tokens. GPT-4o
35 is used both to generate explanations and to predict activations.
SCR and TPP Following SHIFT 32
the SCR evaluation proceeds as follows. A baseline classifier Obase is trained data containing both true and spurious correlations. We then zero-ablate the K features most attributable to the spurious signal and re-measure accuracy on a balanced set:
Agbl Abase
SCR
Aoracle Abase
where A.bi the post-ablation accuracy, Abase the baseline accuracy, and Aoracle the accuracy of an oracle probe trained on the true concept.
16

---

TPP. We extend Targeted Probe Perturbation to the multi-class setting. For m classes, let Cj be linear probe that classifies concept çj with accuracy Aj. Let A.,; denote the accuracy of probe C) after ablating the K most contributive features for class The TPP score is then
TPP =
(A.. A.) m(m 1)
Σ (A.s - A,).
so higher SCR and TPP values indicate stronger disentanglement.
Stability Caveats. Both metrics are highly sensitive to the choice of K. Across different K values, SCR can range from below to above 0.4. For TPP the variation is even larger: for the same coder, scores span from under 0.1 (SAE, K 2, Figure 14) to over 0.4 (SAE, K 50. Figure 18). Error bands obtained from multiple sub-runs are also wide not only for SAEs (e.g.. SAE on 3lama-3.1
in Figure 14 and on Pythia in Figure 15) but likewise for FF-KVs (e.g.. Pythia in Figure 14 and Figure 15p.
Based on these empirical observations, we interpret SCR and TPP scores with caution.
Unlike SCR and TPP, we find RAVEL be consistent across multiple models and coders, and the results align with the scores reported in the original work [23]. This stability suggests that RAVEL is comparatively insensitive to hyperparameter choices and dataset splits, making it reliable baseline when assessing disentanglement. Accordingly, we place greater weight on RAVEL when synthesizing conclusions across metries.
RAVEL.
C Detailed Results on SAEBENCH
We provide detailed evaluation result with error bars indicate 95% confidence intervals (#2 SEM), compute as SEM = V [in in each layer. Note that error bars are not applicable for the feature alive metric, they are counts for features activated at least once.
n(x-1) where n is the number of runs on different datasets for each metric
C.1 Detailed Results on More Models
Figures
6
10
1
8
and
17 present the detailed results for all models.
C.2 Detailed Results on Various Hyperparameter Choices
For metrics that have multiple hyperparameter choices for k, provide detailed results for all tested hyperparameters.
For SCR, the available & values are 2, 10. 20, 50. 100, and 500; the corresponding results
are shown in Figures [4]]]]]]]]]]]]]] 15 16 17 18 19 and 20
For TPP, the available values are the same, and all results are shown in Figures 21
22
24 25
26
and27
For Sparse Probing, the available l values are 1, 2, and 5; the results are shown in Figures
12]
and 13
D Details on SAEs/Transcoders used
For both SAEs and Transcoders from Gemma-Scope,
the canonical versions, whose average Lo sparsity is close to 100, which are believed to he reasonably usefu | The SAEs are loaded through for Gemma-2-2B
SAELens (7) with the following keys: for Gemma-2-99 and "Ilama_scopp sacinaa_ccoee_aaae Ixm 8x" for Llama-3.1-
8B.
For Transcoders, since no canonical versions have been explicitly defined and the SAELens release use does not yet support loading them, manually select checkpoints from the Gemma-Scope
This statement can be found Gemma Scope's collection page HuggingFace dink).
17

---

Layer
ID
0
layer layer_O//winn
10 /5/params.npz
1
layer 1wwiddth_lkk/averrgeeec httpa/eeraae__o0110/aaaaaaaaa 10tpp404s/..ppp
2
layer htw.idth/lw/wwweeeeeeeeeeeeeeeee
10 87/params.npz
3
layer s/wiitth_16k/averagee 10 96pparmmsnpp
1ayer33/width, 3/w////////h//
4
layer_ http://wwdthhlkwwwearrrae 10 88/params.np
5
layer_ 1itr_//wwdtthhhhhh 3/wwidtt__1k/wvvrrrrggeet 10 87/params.npz
6
//er__//viith__1k//eeeeeeee10 httpaeaa____0//aaaaaappppp
7
8
9
layer_
10
layer_ http_//wwwdth__6llaeeaeeee
88/params.np
11
aoye__llwwicthh_lkWWaeeeere 10 1tt.aooaaaaaaaaaaa.aaa
12
aaye__1/wwicch_OOO/eereeee_ httalmmmlppaaaaagggppmmmm
13
layer_13/widhhe 1tt////t/ww
1/k/avtt__16///veraageeee 10 89/params.np
14
layer 1 10 81/params.npa
15
layer 11/w:dthh 16/kaverh__ 10 78/params.np'
16
@yve_]6wwiddh_l66/veerrge 10_ http://rrrreeeoooossaaaraaaaaaa 87/params.npz
17
hter_///wwidhh_1ll/vvverrgggg 10 11_1111_paaaanaangppppppa
18
layer_ 1ttpr//wwwihh_hh6k11/ htw/iddhh_16///veraaeeeeee http:////errreeo1110////aaaaaama 99/params.npp
19
http://9iwdd__16/varrraaee 100_89//raaaaa_... npz
20
layer http:22wiidh_11w//eeeeeaeee
88/params.npz
21
layer heerr/2//wiidc__lll//vveeeeee__e 11ao1111p/pppaammpppppppp
22
2ttpe////dtt_6///eaeeaeee 1atp:rgee____1111aaaaa_a__ 117/params.npp
23
layer 2ii_2/////wh_____ 1/twiddth_1kkkvveragge 10 116/params.
24
layer 1irriddh_1kk/vverrgge 10 96/params.mpz
25
layer_
Table 5: Mapping of layers their corresponding IDs
collection and download the corresponding weights from HuggingFace (link). These checkpoints are chosen according to the same criteria the canonical SAES, and their exact filenames are listed Table We also directly download the weight from the Transcoder proposal work [ on HuggingFace (ink). To the best of our knowledge, there are no Transcoders publicly available for Gemma-2-9B and Illam-3. 1-B, and Pythia-70M..
E Detailed Results on FF Scaling
Table shows the results on all metrics regard- ing to various FF intermediate sizes. Scores
100
are not showing noticeable improvement ex-
Aligned Features es (96)
80
cept for RAVEL and absorption. Trends shown
70
in SCR somehow understandable: these met- rics highly depend on ground truth prob- ing performance, which is not always stable. Sparse probing result is also understandable, since sparse probing FF from random trans- fer can achieve a reasonable score, the probs can learn unintended signal in the dataset, rather than the true feature.
50
40
20
10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
MCS Upper Bound
Figure 4: Proportion of aligned features func- tion of the max-cosine score (MCS).
F Feature Examples
We provide example features for each annotation and coder, visualizing the top input examples that most strongly activate each feature. All features are extracted from layer 13 of Gemma-2-2R. To analyze the relationship between alignment and the max-cosine score (MCS), we divide the MCS
18

---

Table Evaluation scores for different size of Pythia models' FF and TopK FV-KVs.
Coder Status
Concept Detection
FF Size
SAE Type
Feat. Alive 1
Expl. Var.
Absorption +
Sparse Prob.
FF-KV
1.000
1.000.0..00
0.000
0.802.10.173
2048
TopK-FFKY
1.000+0.000
0.227=00000
0.06440.127
0.717##0204
FF-VV
1.000+0.000
1.000=+0000
0.013++0.038
0.826=#0.400
3072
TopK-FFKV
0.99911.000
0. 0.129410.000
0.014410.006
0.779:10.15
FF-KV
1.000+0.000
1.000=+0000
0.003:+0.007
0,803:+01128
4096
TopK-FFKV 1.000+0.000
0.082++0.000
0.006:+0.011
0.765#+0.126
FF-KV
1.000++0.000
1.000
0.003:1:.009
0.812:0.
8192
ToPK-FFKV
1.000+0.000
0.277=00000
0.003:10.009
0.770#+0.186
FF-KV
1.000+0.0000
1.000+0.0000
0.001+0.004
0.850#10.117
10240
TopK-FFKV
1.000:1.0000
0.090:10.000
0.002
0.783:10.144
FE-KV
1.000=00000
1.000=+00000
0.001+0.003
0.870+
16384
TopK-FFKV
1.000.10.00
0.047:10.000
0.001110.000
0.807
FE-KV
1.000.10.000
1.00011.000
1..005
0.818110114
20480
ToPK-FFKV
1.000#00000
0.195=+0.000
0.000+00001
0.735++01177
Feature Explanation
Feature Disentanglement
FF Size
SAE Type
Autointerp f
RAVEL-ISO RAVEL-CAU
SCR (k=20) f
FF-KV
0.727400256
-0.556-0466
2048
TopK-FFKV
0..267
0.000
FF-KV
0.734+00222
0.411+00272
0.033++0004
0.093:0.199
3072
TopK-FFKV
0.73140.271
0.389440211
0.032+00043
0.01740.114
FE-KV
0.708.0.252
0.75010.132
0.014.10051
0.017:10.054
4096
TOPK-FFKV
0.716.10.263
0.719
0.009
(0.00110.020
FF-KV
0.714+0.256
0.900+10032
0.035:+0.004
0.047:10.099
8192
TopK-FFKV
0.712+0.271
0.897+0.031
0.023++0.040
0.016=+0.038
FF-KV
1.707400.25
0.916:10.037
0.013:+0.025
0.049:10130
10240
FFKK
0.704++0.278
0.915++0033
0.013:+0.026
-0.017=0.065
FF-KV
0.702++0254
0.879+00095
0.041++0.098
0.023::0.056
16384
ToPK-FFKV 0.70140.220
0.865+10.12
0.025:100066
0.003+40018
FF-KV
0.693:10.252
0.881:0.072
0.021::0.01
0.030:10.050
20480
TopK-FFKV
0.698.10.270
0.854:10.069
0.019
0.022:10:044
range into bins (e.g.. 0.1-0.2, 0.2-0.3). From each bin, we sample features, each associated with ten pairs of input examples (for total of 100 examples per bin), and annotate the proportion of aligned features within each bin. As shown in Figure 4 features with MCS below 0.3 are almost never aligned, whereas those above 0.9 exhibit over 60% alignment.
F.1 Superficial Features
We show examples of "superficial" features here.
Figure
shows the first FF-KV feature we annotate as superficial, activating on "now".
Figure
30
shows the first TopK FF-KV feature we annotate as superficial, focused on "the".
Figure shows the first SAE feature we annotate as superficial, activating on "return" in code.
Figure 32 shows the first Transcoder feature we annotate as superficial, activating on alphanumeric token combinations.
19

---

F.2
Conceptual Features
We illustrate features that activate on higher-level concepts or semantic themes.
Figure 33 shows the first FF-KV feature we annotate as conceptual, activating on coastal
concepts.
Figure 34 shows the first TopK FF-KV feature annotate as conceptual, linked recipes and desserts.
Figure shows the first SAE feature we annotate as conceptual, activating on country and region names.
the first Transcoder feature we annotate as conceptual, activating on college degree concepts.
F.3
Uninterpretable Features
We also show examples of features without clear patterns.
Figure
shows the first FF-KV feature we annotate as uninterpretable.
Figure
38
shows the first TopK FF-KV feature annotate uninterpretable.
Figure
39
shows the first SAE feature we annotate uninterpretable.
Figure 40
shows the first Transcoder feature we annotate as uninterpretable.
E4
Aligned Features
Figure
41
shows the first FF-KV feature annotate as uninterpretable.
E5
Unaligned Features
Figure 42 shows the first FF-KV feature we annotate as uninterpretable.
20

---

Table 7: The list of assets used in this work.
Asset Type
Asset Name
Link
License
Citation
Code
specified
Code
TransformerLenns
MIT Licease
33
Code
SAHI ens
MIT License
7
Dotaset
OpenWebText
Link
C00 1.0 Universal
19
SAB
Gemma- Gemme-Scope-222 Scopp-2B-ptmmmlppp
google/gemnia--S0 sccpinnnn-sspppp-pee Se-pppp---p--lll
Apache 2.0
30
SAB
Cenma-Sooge-9B-pttmme
Rootle/eemmnn- ph-
Apache
SAB
Llama-Scope-3.11 1a0pp-3.---888
1-8B-LXM-8x
IalpyLiamas BaB-Bas-- LXM-8X
Not specified
21
SAB
GPT2 mouull-32k-mlppouu
|bloem/GPP12 Pooomm/GP22SSallPPPe Smaalll--SaaAAAL v3333k 32k milproul --EE
Not specilied
6
SAE
Pythia 70m deduped.-mlp
Sigges/pihii
Not specified
48
Transcoder
monglegemma--engee 2hapee--hppp pittradddde
Apache 2.0
Model
GPT-2-small
MIT License
Model
Gemma-2-22
(ungingeemmna- 2-26
Gemma License
43
Model
Retogle/gemma
Gemma License
Model
Llama-3.1-8B
mela-lamaa) Lianar 3.1 1-8B
Llama
Community License
45
Model
Pythia: 70M deduupd
Bleulher/AUpyylla 10m declued
Apache
Model
Pythia-160M. 1iiaa-160M-dddduee deduped
BeetherrAppp Ti
Apache
Model
Pythin-410M-dedane
Apache 2.0
Model
Sethia-L-4B-doduun
EloutterAU@yyllue- -o-dccpupe
Apache
Model
Pythia 8B 3B-deeeeed
EleubbrAUpp Imn 2.88 decuped
Apache 2.0
4
Model
Pythia deduped
Aeuther/Atpyyhiin deduped
Apache
Model
Pythia- 12B deduped
Apache 2.0
f
G Use of Existing Assets
Table shows the assets being used in this paper, with the type, name, link, license, and citation for each asset used in the paper.
H Compute Statement
Most experiments presented in this paper were run on a cluster consisting of the NVIDIA H200 GPUs with 141GB of memory. All experiments on models are run using a single 141GB memory GPU. Evaluation time per layer differs largely model size, with Pythia-70M. it takes approximately I hour, and for larger models like Gemma-2-9B, takes approximately hours per layer. The total GPU time for this work is approximately 1400 hours, including exploratory research stage.
21

---

Feat. Alive Across Layers and Models
Gemma-2
GPT-2
2B
Fraction Alive
0.8
0.6
0.4
0.2
0
0
2
4
6
8
10
5
10
15
20
25
Gemma-2
Llama-3.1
1
0.8
Fraction Alive
0.6
0.4
0.2
0
0
20
400
20
30
Pythia
70M
1
Coder
Transcoder
Fraction Alive
0.8
SAE
TopK-FF-KV
0.6
Huimalized-TopK-FF-AA
Normalized-FF--K
0.4
FF-KV
Random Transformer
0.2
0
2
Layer Index
Figure 5: Detailed feature alive scores all tested models, across all layers.
22

---

Explained Var. 1 Across Layers and Models
Gemma-2
GPT-2
2B
Explained Variance
-0.5
10
5
10
15
20
25
Gemma-2
Llama-3.1
9B
Explained Variance
0.5
-0.5
-1
10
20
30
400
10
20
30
Pythia
70M
Coder
Explained Variance
Transcoder
0.5
SAE
TopK-FF-KV
Normalized- FF-F
Nommalzee--F--KV
-0.5
FF-KV
Random Transformer
0
2
4
Layer Index
Figure 6: Detailed explained variance scores on all tested models, across all layers.
23

---

Autointerp Across Layers and Models
Gemma-2
GPT-2
2B
1
Autointerp Score
0.8
0.6
0.4
0.2
0-
0
2
4
6
10
10
15
20
25
Gemma-2
Llama-3.1
8B
1
Autointerp Score
0.8
0.6
0.2
0
20
30
400
10
20
30
Pythia
70M
Coder
Transcoder
Autointerp Score
0.8
SAE
TopK-FF-KV
0.6
Normalized- FF-F
Normalized-FF-KV
0.4
FF-KV
Random Transformer
0.2
0
2
4
Layer Index
Figure 7: Detailed auto-interpretation scores on all tested models, across all layers.
24

---

RAVEL-ISO1 Across Layers and Models
Gemma-2
GPT-2
2B
Isolation Score
0.8
0.6
0.4
0.2
0
0
2
4
6
10
10
15
20
25
Gemma-2
Llama-3.1
8B
1
Isolation Score
0.8
0.6
0.2
0
10
20
30
400
10
20
30
Coder
Transcoder
SAE
TopK-FF-KV
Normalizeed-Ton-FFF-F
Normalized-FF-KK
FF-KV
Figure 8: Detailed RAVEL scores on all tested models, across all layers.
25

---

RAVEL-CAUT Across Layers and Models
Gemma-2
GPT-2
2B
1
0.8
Cause Score
0.6
0.4
0.2
0
0
2
4
6
10
10
15
20
25
Gemma-2
Llama-3.1
8B
1
0.8
Cause Score
0.6
0.2
0
10
20
30
400
10
20
30
Coder
Transcoder
SAE
TopK-FF-KV
Normalized-Topk-FFF-X
Normalized-FF-KK
FF-KV
Figure 9: Detailed RAVEL scores on all tested models, across all layers.
26

---

Absorption! | Across Layers and Models
Gemma-2
Gemma-2
2B
9B
0.5
Absorption Score
0.4
0.3
0.2
0.1
0
5
10
20
10
30
Llama-3.1
8B
Coder
Absorption Score
0.4
Transcoder
SAE
0.3
TopK-FF-KV
Hoormalized-Topk-eae
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
0
10
20
Layer Index
Figure 10: Detailed absorption scores all tested models, across all layers.
27

---

Sparse Probing ( Across Layers and Models
Gemma-2
GPT-2
2B
Test Accuracy (Top-1)
0.8
0.6
0.4
8
10
0
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
1
Test Accuracy (Top-1)
0.8
0.6
0.4
0.2
10
30
400
10
20
30
Pythia
70M
Coder
Test Accuracy (Top-1)
0.8
Transcoder
TopK-FF-KV
0.6
Normalized FF--
Normalized-FF--kV
FF-KV
Random Transformer
0.2
2
4
Layer Index
Figure 11: Detailed sparse probing scores all tested models, across all layers, and various hyperparameter (K) choices.
28

---

Sparse Probing ( Across Layers and Models
Gemma-2
GPT-2
2B
Test Accuracy (Top-2)
0.8
0.6
0.4
8
10
0
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
1
Test Accuracy (Top-2) 2)
0.6
0.4
0.2
10
30
400
10
20
30
Pythia
70M
Coder
Test Accuracy (Top-2)
0.8
Transcoder
TopK-FF-KV
0.6
Normalized FF--
Normalized-FF--kV
FF-KV
Random Transformer
0.2
2
4
Layer Index
Figure 12: Detailed sparse probing scores all tested models, across all layers, and various hyperparameter (K) choices.
29

---

Sparse Probing ( 1 Across Layers and Models
Gemma-2 2B
GPT-2
Test Accuracy (Top-5)
0.8
0.6
0.4
10
0
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
1
Test Accuracy (Top-5)
0.8
0.6
0.4
0.2
10
30
400
10
20
30
Pythia
70M
Coder
Test Accuracy (Top-5)
0.8
Transcoder
TopK-FF-KV
0.6
Normalized FF-K
Normalized-FF--kV
FF-KV
Random Transformer
0.2
2
4
Layer Index
Figure 13: Detailed sparse probing scores on all tested models, across all layers, with the same hyperparameter choice as the main result in Table
30

---

SCR Metric (K=2)1 1 Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=2)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=2)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4
Coder
Transcoder
SCR Metric (K=2)
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normaized-FF--K
FF-KV
-2
Random Transformer
44
0
4
Layer Index
Figure 14: Detailed SCR scores on all tested models, across all layers, and various hyperparameter (K) choices.
31

---

SCR Metric ( Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=5)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=5)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4
Coder
SCR Metric (K=5)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normaized-FF--KV
FF-KV
-2
Random Transformer
44
0
4
Layer Index
Figure 15: Detailed SCR scores on all tested models, across all layers, and various hyperparameter (K) choices.
32

---

SCR Metric (K=10)1 Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=10)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=10)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4
Coder
SCR Metric (K=10)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normaized-FF--KV
FF-KV
-2
Random Transformer
-4
0
4
Layer Index
Figure 16: Detailed SCR scores on all tested models, across all layers, and various hyperparameter (K) choices.
33

---

SCR Metric (K=20)1 1 Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=20)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=20)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4
Coder
SCR Metric (K=20)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAA
Normaized-FF--KV
FF-VV
-2
Random Transformer
-4
0
4
Layer Index
Figure 17: Detailed SCR scores all tested models, across all layers, with the same hy herraaarreeeer
choice as the main result Table
34

---

SCR Metric (K=50)1 1 Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=50)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=50)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4
Coder
SCR Metric (K=50)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normalzed-FF--KV
FF-VV
-2
Random Transformer
-4
0
4
Layer Index
Figure 18: Detailed SCR scores on all tested models, across all layers, and various hyperparameter (K) choices.
35

---

SCR Metric (K=100) 1 Across Layers and Models
Gemma-2
GPT-2
2B
SCR Metric (K=100)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=100)
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4-
Coder
SCR Metric (K=100)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normaized-FF--KV
FF-VV
-2
Random Transformer
-4
0
4
Layer Index
Figure 19: Detailed SCR scores all tested models, across all layers, and various hyperparameter
(K) choices.
36

---

SCR Metric (K=500) 1 Across Layers and Models
Gemma-2
GPT-2
2B
4
SCR Metric (K=500)
2
0
10
20
25
Gemma-2
Llama-3.1
9B
8B
SCR Metric (K=500)
2
0
-2
0
10
20
30
400
10
20
30
Pythia
70M
4-
Coder
SCR Metric (K=500)
Transcoder
2
SAE
TopK-FF-KV
Hoomalized-Top--FFAAE
Normaized-FF--KV
FF-VV
-2
Random Transformer
0
4
Layer Index
Figure 20: Detailed SCR scores on all tested models, across all layers, and various hyperparameter (K) choices.
37

---

TPP Metric (K=2)1 Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
0.3
(K=2)
0.1
0
0
10
0
5
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.4
TPP Metric
0.3
(K=2)
0.2
0.1
0
10
400
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=2)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 21: Detailed TPP scores all tested models, across all layers, and various hyperparameter
(K) choices.
38

---

TPP Metric (K=5)1 Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
0.3
(K=5)
0.1
0-
0
10
0
5
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.4
TPP Metric
0.3
(K=5)
0.2
0.1
0-
10
400
20
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=5)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 22: Detailed TPP scores all tested models, across all layers, and various hyperparameter (K) choices.
39

---

TPP Metric (K=10) Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
0.3
(K=10)
0.1
0
0
10
0
5
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.5
0.4
TPP Metric
0.3
(K=10)
0.2
0.1
0
10
400
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=10)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 23: Detailed TPP scores all tested models, across all layers, and various hyperparameter
(K) choices.
40

---

TPP Metric (K=20) Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
0.3
(K=20)
0.1
0
10
0
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.5
0.4
TPP Metric
0.3
(K=20)
0.2
0.1
0
10
400
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=20)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 24: Detailed TPP scores all tested models, across all layers, with the same hy hyprraarrmeter
choice as the main result Table
41

---

TPP Metric (K=50) Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
0.3
(K=50)
0.1
0
10
0
5
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.4
TPP Metric
0.3
(K=50)
M
0.2
0.1
0
10
400
20
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=50)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 25: Detailed TPP scores all tested models, across all layers, and various hyperparameter
(K) choices.
42

---

TPP Metric ( Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
(K=100)
0.3
0.1
0
10
0
5
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.5
0.4
M
TPP Metric
0.3
(K=100)
0.2
0.1
0
10
400
20
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
(K=100)
0.3
TopK-FF-KV
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 26: Detailed TPP scores all tested models, across all layers, and various hyperparameter
(K) choices.
43

---

TPP Metric (K=500) 1 Across Layers and Models
Gemma-2
GPT-2
2B
0.5
0.4
TPP Metric
(K=500)
0.3
0.1
0
0
10
0
10
15
20
25
Gemma-2
Llama-3.1
9B
8B
0.5
MM
0.4
TPP Metric
0.3
(K=500)
0.2
0.1
0
10
400
20
30
Pythia
70M
0.5
Coder
0.4
Transcoder
TPP Metric
0.3
TopK-FF-KV
(K=500)
Normalized- FF-K
0.2
Normalized-FF--K
FF-KV
0.1
Random Transformer
2
Layer Index
Figure 27: Detailed TPP scores all tested models, across all layers, and various hyperparameter
(K) choices.
44

---

MLP Down Projection Output Norm Statistics (L2 Norm)
GPT-2
Gemma-2B
0.8
L2 Norm
5
0.75
4
0.7
0.65
2
18
Gemma-98
Llama3.1-88
0.7
0.65
L2 Norm
0.6
1.4
0.55
1.3
0.5
8
12
16
20
24
12
15
18
21
24
27
30
Pythia-70M
1.8
1.6
L2 Norm
1.4
1.2
0.8
3
Layer Index
Mean Norm
Mean = Std Dev
Figure 28: Distribution of the L.2 norms of all tested models' FF-KV decoder weights (i.e., W, in its FF sublayer). Although the norms are not exactly one, they are concentrated narrow range.
45

---

Text ID: 6269 (Max Activation: 0.691)
the
unemployment
actually
percent
discouraged
who
have given seeking employment and roughly eight million workers
part time joba would prefer to
work
full
time job The unemployment rate now stands at
percent
To put
this number In perspective
improvement the percent reached that big
now higher unemplayment
Recession
prepared remarks
Text ID: 962 (Max Activation: 0.629)
despito the electoral spending Ruth Davidson
Conservatives
TM won just
seats out They fought
single issue opposition another Independence referendum they Improving number seats you achieved still doesn
improving you winner Scotland looked at Ruth Davidson party and
arithmetic
exam from an F to D means that you top the class gave them falling grade the Tory triumph allam the world won change that simple thmm tical reality
Text ID: 2495 (Max Activation: 0.520)
moving method now has extra references that just aren needed anymore the midst the ref actoring you
functionalitt method that doing things likely the original
focused on the extractin that you forget to clean these bits And by cleaming up you may uncover trail cleamup you code hard read
can go
system That might make the
Text ID: 14099 (Max Activation: 0.500)
struggled continue the paper though he asked help he received none The home Travis
and
Ros anna
and restored examination received permission legally practice he earlier in the year to help pay
relocated
February
Travis passed
borrowed
office
well
25
for
the
Herald Now
debt
and
with
practical
Income
took
three
boarding students
help
Figure 29: Top-4 activating examples for a particular feature in FF-KV annotated as "superficial".. This feature specifically activates most on the word "now". in various contexts.
46

---

Text ID: 11936 (Max Activation: 2.609)
Claudio Ran coach the team Following this move financial contract with Ran whose deal was comiig
to end 0 June
ended mutual consent Roma wishes to thank Claudio Ran for the
professionallem shown and the work done [ during time at the dlub Ran leri enjoyed successful first season with Roma after replacing Luciano Spal letti September who had been fred by Juventus months earler rescued the team and nearly led them to the scu
had endured horrendous the campaign and Ran leri
Text ID: 15318 (Max Activation: 2.484)
Angel Maria Real Madrid Utd. Aug
Kaka Milan Real Madrid
June
Sources told ESPN FC that Chelsea have been quoted price million pounds by PSG for Cav and Bridge despite
player eager
prefer stay
has
already been preliminary contact betwoon the London club and the player camp athough Choisea stll assessing the
best
course action
With Mourinho having regularly complained
Text ID: 7334 (Max Activation: 2.484)
spokesman Adam Rosen said shocked that agency of Tirst responders would enforce such onder the week of Sept
The four suspended frefighters said they were told that the order was Issued because of racial discord in the mig said such problems exist wrote CBS which reported that the four firefighters trace issue decision several trefighters
indudde
frefighters
firefighter
fourth freliihter
replace tattered American flag last month one of May wood fire houses The now flag mysteriousty
Text ID: 12010 (Max Activation: 2.453)
Trading standards are investigating after couple who stayed hotel diaiedd have been Ined 1.000 hotel
they described rotten st inking ho on TripAdvisor Tony and Jan Jen kinson from White haven Cumbria
posted review the webele after staying at the Broadway Hotel Blackpool However
couple later found that
charged to their credit card which the BBC reported was the result of hotel policy in the case of bad reviows last night...
The
manager
Figure 30: Top-activating examples for feature in k-Sparse FF-KV annotated as "superficiall. This feature specifically activates most the word "the", in various contexts.
47

---

bayes Make Poisson lam chose upper goals game Text ID: 11353 (Max Activation: 18.412)
pm Normalize ( return xrange 0 high }: Eval Polsson P mf lam pm f.. Set ( k
pm f The range of values the computed P is from high. So the value of lam were exactly 3 would compute probability scoring bound
3 goal
simple
(Max Activation: 17.365)
y
p list append )
append
return
nр array olist 1:- ]b np arryy list
arg ( ymin ymax imin
limits hope }:
ymin ymax np. arg logical
ymin
ymax
Text ID: 4503 Activation: 17.178)
result } Such oplimizations are implementation details but are described here because they provide practical performance benefit to the performance conscious user It would be rice std STL implementations provided such things though Met rower ks has been known to some cases Instead of just algorithm EAST has algorithm sort h algo set
Text ID: 10231 (Max Activation: 16.843)
D
1
1000 %
%
10) % 10 10
printf
return
J: user @ kali
link
quanta
link
wps
gen
program
https
(Max Activation: 14.418)
roots
delta
sensilivity
Inalg
compute
largest
singular
direction
reverse to match polymomial coefficients
return
s 10.
kappa
root
kappa
arry
Figure 31: Top-activating examples for feature in SAE annotated as "superficial". This feature activates on the word "retum"". especially in programming- contexts.
48

---

ID: 4591 (Max Activation: 34.000)
might be worth using as reference full discussion compleer lining characteristics outside the scope
this document but some Interet discussions regarding GCC in lining problems can be found at hitp 2f groups google group lang browse fm thread 7 eed
http groups
google com group
browse thread
www pixel
glow
Text ID: 3930 (Max Activation: 33.750)
parts needed loading project you including plastic case knob adapter can your cart
buy
change anything Project Manager Project Detail Access
shared project go
http
Tools aspx and enter this access code
Upgrades am often asked what can be done to
upgrade the designs that publish In this case there
Text ID: 6968 (Max Activation: 32.250)
an environment variable JE BIO API KEY or pass II as parameter you are importing script library
Queries return JSON output except for download requests that return binary attachments The return code
variable set to success to emor Here few examples Query file hash $
check
cbc
Text ID: 6971 (Max Activation: 31.500)
apk
user present only you up yourself
Upload
Jab
py
upload
apk
apk
uplo ade van code
Download file
subject
permission
$ jeb
download ba acc
ba
Figure 32: Top-activating examples for feature in Transcoder annotated "superficial"". This feature activates on the combination of digits and alphabet, various contexts.
49

---

Text ID: 6561 (Max Activation: 1.891)
Canadian
mainland
Point
elce
Ontario
westemn most Boundary
Poak
N
8 W the
southern end the Yukon Alaska border
which
roughly
following
but
leans
very
siihttly
east
goes North The
eastemn
most
point
Cape
Spear
Newfoundland
Text (Max Activation: 1.617)
posiiions result
Jo aquin will move northward much of this weekend roughly paralle ling the East coast
There
Is
nearly equal possibility the storm will make land fall atong the mid Atlantic coast the New England coast or veer out
proximity
people the Carol Massachusetts
will need closely monitor the track and strength Joaquin for wind and coastal flooding concerns Should
Joaquin
make
land fall areas near and north of the center would face the worst coastal flooding and strong winds
the
storm
were
to make land North Carolina
Text ID: 12857 (Max Activation: 1.547)
asked
him why
wasn driving said didn have truck and horse trailer just horse pack
horse and
dog
Hile plan was simply to ride the coast to San Diego and turn left He had what he called
shoulder pass which
from pocket presented to officer The officer being confused was
even such document exiated and examined molecular structure Then the Laguna Animal Control officer showed
up
That officer informed the cow poke that he did not have his dog leash Somethhin all good little citizens
Text ID: (Max Activation: 1.531)
picked up from the Visitor Center
Wallkng
Tour of Historic Downlown Cors icana On page four the tale of the
wandering Jewish rope walker Good Nancy liston southeast street
usuallyy down working
spare With side the east
open
on
two sides There
Is
Main
Street
but
the real main
Beat
Street which runsi allong
the building From
the
south
side
oast
most
window
rightt
behind
Doug
as
he worked you
get
Figure 33: Top-activating examples for feature in FF-KV annotated as "conceptual".. The specific annotation "concept related the coast" especially in various contexts.
50

---

Text ID: 5531 (Max Activation: 2.516)
Ingredients
Course
Sweet
dessert
Cuisine
Indian
Serv Inge
kat Calories
kcal
Author
Ini
Ingredients
cup
ml
coss hews cup powdered sugar teaspoon saffron optional
teaspoon
cardamom powder Instructions Soak cas hows In water for minutes Powder the sugar
the mixture medium fame and start sugar to
Grind
cas hows
very
paste
jusd
sprinkling
ground
stick
Mix
well
Now
heat
Text ID: 5528 (Max Activation: 2.406)
keep
sc
arp
ing the sediments while stirring
Add saffron and mix well
Keep
stirring for minutes The
kaj
sugar
mixture
bubbiinn will begin
sure scrape edges stiming
Add saffron this
add Once the kaj
mixture forms and gathers
center
pinch very small amount and make ball with fingers are able form ball thon Is the correct
consistency
Switch off the faame and lot the mixture cool forr while
Text ID: 5530 (Max Activation: 2.297)
very
roli
dough
you
Alllow
minutes
Store
Kaj
Kat
airtight
room
temperature
refrigerate
kost
brit
hard
K
Kert
recipe
card
below
from
votes
Print
K
ajuu
Jest
recipe
casihew
fudige
recipe
Prow
make
Raj
Kcrt
Prep
Time
mine
Cook
Time
mins
Total
Time
mins
Icott
recipe
Indian
Ioweet
vegam
cashow
fudge
(Max Activation: 2.203)
minutes Store
kaj
kat lis
airtight
Recipe
Notes
Do not
add
more water while grinding
Jusi Jg sprimikte
to
ease the grinding the cashew paste should
very smooth not cook the mixture
more
than
minutes
but lump gently
stage remove
Kine
mixture
you
your
dough
seems
to
you
can
tablespoon
milk
water
whill
kno
ading
Sharing
Figure 34: Top-activating examples for feature in k-Sparse FF-KV annotated "conceptual". The specific annotation was "concept related to recipes" especially in contexts related to deserts.
51

---

Text ID: 14508 (Max Activation: 14.855)
the
WI RED How giant size computer quite feacinating time recently Lastrrtte
fini
tablet
spening
agi unga
Journey through
wined
Nature Stories
Rtan
my
encydooeeaa nature you somehow obsessed
Text ID: 6117 (Max Activation: 13.967)
right John Home Too ke another radical MP and Catharine Macaulay She like the other women wears French ti coloure The people in this print are all Innked by their support for the Revolution The women were distinguished for ref uting Bunke print or seemeed Willlmms who noted for sympathwic.
Laterr Wittlen France pubiishhed
storming of the Rtatile
Cutharine Macaslay
Reflection
Barba
Bunke
assureed wrtting
Retrections
Text ID: 13267 (Max Activation: 13.406)
and pollical raka which UK businesses face when openating abroad indluding in Isnael and the OPT This includes guidance laraet settlmmnts
settements under International law Uttimatlly will the decition of Indiviual or company whether to aperate in nettements the Occupted Territories but the Britihh Govermeent would nether encourage nor offer support to ouch activty When approoched by businesses oet out the UK clear postion
Britsh businesses
Britsh Government
considering
reglon vokuntary guidence
doing
Text ID: 9248 (Max Activation: 13.382)
the official aaid
Mancheeder Uniidd goaliceperr Sem Juhratone poined nejóin Astznn Ville Manday The old has been
John stone who spent the second target for number Charpiorahip and Premier League siden but will sign for Villa on seanon long loan
swsson
another yeur contrect United after one Manchester United goalcesper
Johnatone
Mondayy
training
complex
tour party that travels the USA
Activation: 13.311)
Evergreen transported U troope on drug mide Central and South America the paper said Over the years, company officits
denied working for the CIA When contacted Wednesday by The Providence Journal discuss Evergreen relationahip with the CIA spokesman
spy ageney declined comment company also had conmrots carry
Mail transporting cargo personnel
private
green
Chapter
Baninuptcy
Figure 35: Top-activating examples for a feature in SAE annotated as "conceptual". The specific annotation was "name of country and region" in various contexts.
52

---

for Text ID: 1508 (Max Activation: 27.375)
during hi
tenure
served
89 00 offensive coordinator Cata College from 0 2 and offensive
coordinator
Delta
State from
2
graduate of Florida State with
bachelor
degree
sports management
Bloom gren earned master degree higher education from Alabama
2001 native of Tallahassee Florida Mike and his wife Lara have two sons Tyler and Parker
The
gren
College Florida
7547 (Max Activation: 25.625)
active spreading formerly Head of Research at Breve New Coin http !! bra coin authors tone after previously
and importance technology
freedom
writing about trading and economics Coin Tale graph hitps s/ co intele graph com authors tone
). Tone
maintains personal website Liberty Life Trail where you can find his latest research Bitzoin Blockchain sound
economics and privacy Tone holds Masters Degree In Financial Engineering from Florida State University along
with Bachelor Degrees
Mathematics
and Geology
minutes
4722 (Max Activation: 24.750)
He graduated from Texas A M University with in Agricultural Education and Master of Education
degree Tom being
his fishing whenever he
in charge of the business Email Tom at les ter @ air mail net Visit Tom his web site Fishing Pro Staff MUM BAI DEL HI Itesh Srivastava speaks with the ferv our true bellever am for here The work systems professional any FMC company But
(Max Activation: 24.375)
impacting of Income working familie Minimum wage should conducted the best feasible study designs just federal agencies demand the best designs when seek out evaluations of other market policies the Daniel
doctoral student American University economics gender economics coming American Population George Washington
Economics special
University
Institute
degree
policy
specializing
policy
University
Acknowledgements
The paper
Figure 36: Top-activating examples for feature in Transcoder annotated as "conceptual". The specific annotation was "concept related college degrees" in various contexts.
53

---

Text (Max 0.746)
specialized in Dark moves so maybe Oh dear God what have become They got me
hooked
Pokémon and didn even see coming Strange how much good motiv ator can improve game
overall appeal
Suddenly
weren pointless
working towards something Collecting
matter
because the powerful ones could help beat challengen more points upgrade Zaro ark Zoro
My precious
Beyond the storyline mode that can be beat roughty obx hours of playtime thore are
tons little
Text (Max 0.691)
feasible that the organisation may announce some kind second Season Pass assuming that the player base Is still thore
Whatever the case new patch deploy next week adding Elite Driver levels Improvements to the dile Photo
more section below Mel ania Trump official White House portrait has been released Here Andi looks
you
game go next Take sharp hairpin the
model
Probably
because
she used be one So The Boston Globe immediately picked the whole thing aport top to
bottom
Text ID: 9857 (Max Activation: 0.664)
relatives the hospital medical nursing and administrative Among five refused participate
had mini mental examination consisted
scores and therefore were excluded from study Thus the control females males unrelated the patients who were age sex matched with the p presence
neuro poychiat history drugs alcoholiam and drug abuse The first part the questionnaire addressed the main demographic characteristies
group
paych otropic
well Mn smoking and drinking habits The second part of the
Text ID: 12993 (Max Activation: 0.645)
embraced same way The arts council is more focused arts education Navaro County schools than producing or displaying art these days Brooks said makes no sense said old educator so get doing for the kids but keeps
supporting
company
study
made him back when helping the council The character pulling
of statues
Figure 37: Top-activating examples for feature in FF-KV annotated as "Uninterpretable..
54

---

Text ID: 7567 (Max Activation: 0.676)
suggest that this fittr plays significant role in explaining the different patterns of deference exhibited by conservatives and liberals.. The disposition to defer others with whom we share political or religious outlook is continuous with disposition to use benevolence a cue to reliability we are disposed to see those with whom we share political outlook and religious affiliatin those who are benevolent toward us and our interests peraass disposition use benevolence que reliablity is just special case of disposition defer to those whom share values The use political and religious affiliation proxy benevolence is
Text ID: 7727 (Max Activation: 0.609)
who Thomas Mars and never have the phone records Robertson said never find Bulg come lited Peter Nicholls The Times Net migration has reached record D as figures published yesterday showed that Roman lans are now the third biggest coming UK Officials said the latest increase was driven by stat istically significant rise In overall number of immigrants with them arriving take jobs The surge was partly because
don
even
thousands
since were
Text ID: 8655 (Max Activation: 0.605)
segment 1 active coaches in DI to accomplish the feat R ALE IGH N C NC State baseball head coach Ellitt Avent recorded his 1 0 career win against North Carolina Central Tuesday evening as the Wolf pack
Check
ACC
seeson high four home: runs route to 3 win at Do ak Field Park Now his 8 th
season as head coach and 2 at NC State Avent
Text ID: 5332 (Max Activation: 0.605)
They reported that Einstein was right Since then his theory has been ret ouched in detail but its essentials have been repeatedly verifled No important scientist is to be found among the skep lics although there is every incentive to deb unk Einstein popular uproar over the theory surprised no one more than the author af the theory He had been almost a reclu se His contacts had been with quiet scholarly men of his own type. and his sudden glory appalled him Interview ers photographers lion hunters cause prom oters testimonial
can be done Immort ality awaits the man who can overthrow Einstein The
Figure 38: Top-activating examples for a feature in k-Sparse FF-KV annotated as "Uninter- pretable".
55

---

Text ID: 15344 (Max Activation: 4.632)
Aaron Gle eman Hard ball Talk reports
C purchased an East End mansion that Babe Ruth spent time at
Ke reports the comedian shelled out 2 millon for the 5 7 square foot Prim rose Cottage
formerly visited by the New York Yankees legend The home three story Tudor originally constructed in with six bedrooms three and half baths and five fireplcces Yes there more fireplaces than bathrooms this home which probably zoning requirement in the Ham ptons one of Ruth entri
Text 12336 (Max Activation: 4.299)
the ones Additionally ssh server theft is another one time vector that can be used quickly bootstrap into node key thaft For this reason node admins should always use ssh key auth for for node administration accounts since it prevents ssh server key theft from implying continuous server compromise http i! www m well com ssh mit m public key authentication Issues With Ephe meral Identity Keys There are few issues with deploying ophemeral Identity keys Issues With Ephe meral Identity Keys Client guard node loss The primary issue with ephemeral identity keys is client Guard node loss your relay obtains the Guard flag
Text ID: 8158 (Max Activation: 4.289)
given
Trump
sharp criticism and talk re visiting Iran nuclear deal The message loud and clear
The United States cannot be trusted Third the U South Korean alliance is not impenetrable President
Trump tweeted his criticism about the South Korea United States free trade agreement around the time of the th nuclear lest and accused the South Korean govermeent of sement with North Korea South Korea finding have told them that their talk of appea sement with North Korea will not work they only understand one thing Donald Trump (@ real Donald Trump
Text ID: 4557 (Max Activation: 4.193)
specifically does somewhat meaningless arbitrary struct Table Sorter Based Sorter
const
D
)
bool operator
(X int const return
Table
int
Table 2 }:
std sort (
2 8 Table Based Sortor values
)): The
Figure 39: Top-activating examples for a feature SAE annotated as "Uninterpretable".
56

---

Text ID: (Max Activation: 7.969)
but their authors well know ned Linear Systems and State
Space Sorry.. don have anything can recommend really coming up
deeply
Hes ha be Chapter freety available online.. found usefd the various state space inter connections
at
MIT into this latter Linear Systems Theory
Jalio
But looks typical textbook
relance MATLAB and stand the heading
Text ID: 10600 (Max Activation: 7.719)
dlus The announcement official by John Han Director Google Earth
Maps
Stefan
Bruno
founded large number employees
are stll Google working
Software
Engineers
Managers and Ps fll sharing website that supports Torrent FTP Donkey etc Xun was developed by
Xun PPice: $ million
Date: January
Chinese
Thunder
Networking
Technologies
southem province
(Max 7.489)
billion medical care and benefits Iraq and Afghan war veterans estimated billion additions the Pentagon base Although funds not directly the theaters the researchers believe
would not been appropriated wars been undertaken
billion homeland security Again spending not have undertaken for and climate
made
$ 0 billon additional apending on war operations and war related base budget for
Text ID: 15443 (Max Activation: 7.406)
remaining cards the deck in the middle of the playing area Leave room nexd to the deck for discard pře Third deal piles the total number players For example If playing game with four players there will be five Recipe pilos Fourth
playing
take
Season cards and put
proper order Spring
Summer
Fail Winter Select start
The Season deck should remain
face
Figure 40: Top-activating examples for feature in Transcoder annotated as "Uninterpretable"
57

---

FF-KV
Transcoder
naar
gen ter wijl
de te
naar
fer wiji de werks
works 25lling voor
Bel
gen
met
ecn heden da
tailing voor Bel gan mat 0 een hadan da aide
alde Ton tweede hot OD mer ketjk cat or tussen
op keljj tussen
agen
financi ete crists met bahoo or likk wat work zan ons land mear ban an gede
met beho iik
one land meer
ban en zjnn gescha pan voor geda ta cheer den dan voor
geache
gen
Bal gen kno gen ean nicuwe baan
0 Bel kre cen nicuwe baan als to ont
rok kanda
dige legein
diga segan
Crossing New Leaf am
Croesing New Leaf am Ilbo cands will also become libo cards Will also become
you play the perfect time to cazy to the charm and creativity of this special game New friends diecoveries aweil every Express yourself customizing character home austamizing your character home and town as you and town as you create your ideal world The arcade
purchase matter you play
purchase
cozy
creativiyy of this special
New friencts
discoveries awalt every day Express yourself by
create your ideal The arcade the bastion school
loose chango school arcade the retirement tought Zealand no less complete with Pac Man pinboll and
loose change
thought Zealand
retirement capital
bear claw machines started using crysiocuneney claw machines has started using cryptocurrency And And actusily. not that oryalic.
completo with Pac Man pinball and tediy bear
actually not that cryptic
2
3
11 den
0
den
cond J=
1 cand 0 The
The
modal form exprosses
any
state
space
system
a5
model
form
expressees
as Use separate first
Bry slale space
system
the
par
parallel sum separate
first
order
syatems
Figure 41: The first feature pair we annotate as aligned.
58

---

FF-KV
Transcoder
the
cade Event
Record
code
Synchron Ioe
stop F:
agein reoue compery sccemeend
product
Poala
Byle Langth
cude Nam
Device
Hoat
from
Purchase
product
vey plonseed surgeris
very
device
to hoat
TILE
LINE
new Inaiee
Flods:
mamory CLICK
I plante
dym cudt From I Phoale
and
sake chance
Will that chenee pad
Juckot
smeeing narty real iether
Fh Fiee glow
Inow
what
attpping longer than anticppatd
placed
Imaage
image
reaporded
h Creata pointer
device and
pluata Ult:
phola
Usdi
R..
Aligtt
Dhanka ac mech pays smadin
Jacied
and
superti
again
cude Drent Create cutte
Evenl
Create
Copy
original
product
Purzhare
cud sm cuds Malloc vold
Image Byle
Length
Original Image allocation
cude
product
Great
quality
jocket raalyy
Length
nigisal
lighoiin
great
regertingg
recommenned
recoemmand
product
Parciase
product
complaii
Mue
Save
Image
After
saving
xn Image Vie
would like
keep
Yhe recommann the protud
Purchare
FMcE Trwnnd
preduct
envinommen oleer
en dbeel locara marmery
by cuada Prom furettion
Orpy
Delivery
soukel Imagee
cool D0R
Pleda
Plub
choopent delivery option and
The
Image Length
jocoet
Wage
probium
LINE Image
pimatt
mowe (Padaa
Proe meneery
cuda Free plaeta dym cude
Free
(ew Phade
dym
Savve Image SOL
Stave DMP ran Image lamo
1X aarfacae SOL
prodkoot
Purchose
rocommmend product.
Serioutly idea
with manorre lead
Harper per cart and comboriabk
batrddyy a. Taago When No jat and and fie porfuity and didn 19AA had guttee
stow
Trudess per Ielephoce poll sarppled
tin
This Jocket
randanly
country
pergooos and the contomer service smasing
squcher
accarata (witha
sspected tedty recomseend company Jectett
Receenber Inston
recormmand product Porcium recommmnn product
troakin news follow/
Twitter
Mat Rerwwyy
aewaraago weerg
jacker iiei
reaitatle
vatere but siappeenndd aincw Deroorii National
Coneenton
The pattern will Listy spell doen for Romney untese
Figure 42: The first feature pair we annotate as un-aligned.
59

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our main claim is that the SAE-based approach provides comparable inter- pretability to feature vectors stored feed-forward layers (FF-KV), We diligently investigate this claim across multiple LLMs and their corresponding SAEs, along with Transcoders, through both automatic, extensive evaluation $ 4 and manual evaluations $ All results demonstrate high similarities between FF-KVs Vs $ 4.3 and SAEs $ 5.2 We further analyze
the overlap between Transcoder features and FF-KV features 66
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The paper includes a dedicated "Limitations"" paragraph the "Conclusion" section (Section 7). We discuss several limitations of the study, including that the feature dimension of the SAEs and Transcoders used in this work was fixed; the results for the Transcoders not available for all models because not every model is accompanied by one; and our qualitative analyses are limited to case studies.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational eMM.Piciency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
60

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not involve theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it aMLPects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details metrics and models, as well as the experimental setup. in Section 4
We also provide detailed info the SAEs used in Appendix D Additional information on the implementation details is presented in Appendix (A] We also release the code htttt:://// thub. http::::/g..hh.b..ommo88822/2// fv- -ve facilitate maximum reproducibility of our main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suMLPice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
61

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide open access to our data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Sublic/guides CoodeSubmisssooooliico for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details on exprimental settings for our main result in $ 4.2 and
Appendix D We also provide additional information on metrics used in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
62

---

Justification:
The figures displaying as our main result (Table T as well as all detailed all include with 2-sigma error bands, calculated by SEM
figures shown in Appendix [
( These are also reported in the text in Section 4.3 and Appendix m[n-1)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide suMLPicient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We describe the compute used in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: To the best of our knowledge, the research conducted conforms with the NeurIPS Code of Ethics. We explicitly discusses potential negative social impacts and includes an ethics statement in Section 7
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
63

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We explicitly discuss potential negative societal impacts in Section] 7
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For cxample, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the eMiPeiciency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We did not find cause believe our methods are at high risk for misuse, and therefore did not feel that additional safeguards were warranted.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing eMI.Pvetive safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith eMLPort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing open-source models, datasets, and evaluations that we use are cited. We specify the asset type and license type in Appendix G
64

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/dataeet has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce or release any new datasets, code, or models.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments were performed. The human raters who carried out the qualitative interpretability assessment in Section 5]d Section 6 were the authors of this paper and colleagues the lab, respectively. For assessment carried out by authors, care was taken in the design and execution of this experiment to ensure that no authorial bias would influence the results. For that was done by our colleagues, we include the detailed evaluation criteria we ask them to follow in Section 6.1
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
65

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve human subjects- see item 14.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for writing and formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy attp::///eerrrps.cc///ffeeereeee for what should or should not be described.
66


Reviews:
[{"review_id": "5FUnH4jmfH", "reviewer": "Reviewer_hRu1", "summary": "This paper compares the interpretability of SAE and Transcoder features to the interpretability of feedforward network neurons. It calls into question the conventional wisdom that SAE features are significantly more interpretable than raw neuron activations.", "questions": "- Prior work has run experiments very similar to the ones you run here, on the same models, and found different results. Do you have an explanation for why your results seem to differ from those of Paulo et al. (2024), Table A10? Could you run your auto-interp experiments with their open source delphi library to see if the results change?\n- What would count as a significant difference in autointerp scores here? The figures are quite zoomed out and it is difficult to get a good sense of how different the scores really are.\n- How do you apply your method to a model like Llama, which uses SwiGLU layers rather than FFNs? Strictly speaking SwiGLU is an entirely different type of module, not an activation function for an FFN, and in the Llama variant it features three separate learned matrices, along with an elementwise multiplication step. It's not clear that this type of layer is amenable to the key-value interpretation.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- Simple, timely, and important research question.\n\nWeaknesses:\n- It is unclear why the \"key-value memory\" lens is particularly relevant to the paper; it seems that one could simply refer to the units as neurons. It's also unclear if this lens applies to the models they tested.\n- The autointerp results seem to conflict with existing reported results, but this discrepancy is not remarked upon or explained.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments.\n\n>Why adopt the key–value memory lens? (Weakness 1)\n\nThanks for a good clarification question. SAE research investigates both feature activations and the geometry of their feature vectors, whereas the term “neuron” typically refers to the activations (FF keys) alone. The natural counterpart of the SAE feature vectors in the FF module is their value vectors through the lens of key-value memory, and in this sense, key-value memory will be a more accurate scope corresponding to SAE research. For example, in Section 6, we analyzed the alignment of feature vectors in SAE and FF, which will be beyond the scope of just observing activations (neurons). \n  \n\n> Applicability of the KV view to SwiGLU layers (Weakness 1 & Question 3)\n\nThis is a good point, and there will be at least two dimensions to be discussed:  \nA. Split‑gate factorisation (gate_proj, up_proj)  \nB. The presence of negative activation values produced by SwiGLU and its impact on the experimental results  \n\nFor the concern (A), Zhong et al. (2025) [1]  show that a SwiGLU FFN can be written as a signed key‑value sum (their Eq. 29):\n\n\\begin{align}\n\\mathrm{FFN_{SwiGLU}}(x)=\\sum_i v_i g_i (x) \\mathrm{Swish}(k_i x)\n\\end{align}\nwhere $g_i(x) \\mathrm{Swish}$ is regarded as a key (activation) and v_i as a value vector.\n \nAs you may be concerned, the $\\mathrm{Swish}(x)$ can take negative values, in contrast to the original key-value memory analysis in vanilla FFs — the concern (B). \n\nFortunately, the absolute scale of negative activation is typically very small (as it’s the approximation of ReLU). Our experiments rely on the top-k activations based on their magnitude, and we empirically confirmed that negative activations were not selected at all throughout the experiments, as their magnitude was almost zero. That is, the negative activations do not affect experimental results in this study.\n\n[1] Zhong, Shu, et al. \"Understanding Transformer from the Perspective of Associative Memory.\" arXiv preprint arXiv:2505.19488 (2025).\n\n>The inconsistency in autointerp scores with previous work (Weakness 2, Question 1, Question 2)\n\nIn the SAE fields, slightly different implementations (with different activation selection processes) are used for the auto-interpretaion task, and we adopted the implementation of Karvonen et al. [2] — this is the source of the discrepancy with Paulo et al. (2024) [3]. Our autointerp results yielded a similar score scale to [2] (see their Figure 5).  Note that the implementation of [3] introduced a specific threshold (1e-5) for sampling activation (w/o reasonable justification), while [2] applies a two-step process, first a threshold-free sampling process, and second a dynamic threshold (1% of the max activation); we hope that the latter (i.e., our implementation) can handle concerns, such as different activation scale for different features/models.\n\n[2] Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders\nin Language Model Interpretability. Forty-Second International Conference on Machine Learning.\n\n[3] Paulo, Gonçalo Santos, et al. Automatically Interpreting Millions of Features in Large Language Models. arXiv preprint, \tarXiv:2410.13928 (2024).\n\n\n>Improvement on Figure 1\n\nWe apologize for the dense, small plots and will replace Figure 1 entirely, as all reviewers noted its limited readability. For a detailed revision plan, please see our response to Reviewer euvs. Note that a more detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)"}, {"review_id": "y2c8pDjPFh", "reviewer": "Reviewer_euvs", "summary": "This paper conducts a thorough comparison between the interpretability(and other features) of Sparse Autoencoder latents vs original neurons in the MLP layers of a transformer model. To make the comparison more fair, they introduce k-sparse and normalized k-sparse FF-KV for sparser activations. They find that the differences between SAEs and MLP neurons are smaller than previously thought, and claim that training an SAE might be unnecessary as existing neurons can have comparable interpretability.", "questions": "- Previous work on SAEs often compared interpretability against existing neurons and found very significant improvements, for example Anthropic work: https://transformer-circuits.pub/2023/monosemantic-features/index.html#global-analysis-interp . This seems to contradict the findings of this paper. Why do you think this is the case?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Very clearly written\n- Adresses an interesting question with extensive experiments\n- Some interesting and unexpected findings\n\nWeaknesses:\n- Fig 1 needs to be improved, hard to tell different lines apart. Partially might be a problem of plotting too many things in the same graph but clarity could also be improved by zooming in on some plots and choosing colors with more contrast, in particular I can’t tell Transcoder and Normalized-FF-KV apart at all. Markers for actual data points would also be helpful.\n    - These results might be better presented as a table, I think a table of the average scores over layers of the different methods would be very useful for a reader.\n    - Expanded tables in appendix are helpful but still suffer from some issues with contrast and seeing where the data points are\n    - Seems like not all plots have all the methods? For example, where is transcoder in explained variance plots?\n- I think paper is slightly overclaiming at several points, for example the autointerp scores are significantly higher for SAE methods but this is mostly considered a not significant difference\n    - Non top-k FF-KV methods have even lower scores, could use more discussion on the significance of this difference. What is the random chance score on this evaluation? How far are these scores from that?\n- Similarly, the top-k FF-KV methods have very poor explained variance compared to SAE-based methods but this is not discussed at all in the paper.\n    - This is similar to Anthropic’s findings that thresholding neurons is not competitive with SAEs in terms of sparsity/reconstruction quality (See https://transformer-circuits.pub/2025/attribution-graphs/methods.html, Fig. 3)\n- Human evaluation is very small scale and relies a lot on author judgement\n- Limited originality. The paper mostly applies existing evaluations pipelines on existing models, though normalized k-sparse FF-KV is a neat idea.", "was_revised": true, "final_justification": "The rebuttal has clarified my understanding of the paper and I think seeing the table representation of the results is very helpful. While I still think the contributions of the paper are limited, I lean towards acceptance as the paper is well written and cleanly focuses on a particular question, so I will increase my score to 4.", "rebuttal": "We thank the reviewer for the insightful comments.\n\n>Detailed plan for complete update of Figure 1 (Weakness 1)\n\nWe will replace Figure 1 with tables with exact scores — sorry for the limited readability. Below, we showcase the Gemma‑2‑2B layer‑averaged results. Still, such averaged scores overlook layer-wise detailed changes; in the next version, we will also include additional figures in Appendix: one plot per coder, task, and model to improve legibility.\n\n| Model      | Coder                 | Feat. Alive $\\uparrow$ | Explained Var. $\\uparrow$ | Absorption $\\downarrow$ | Sparse Probing $\\uparrow$ | Autointerp $\\uparrow$ | RAVEL-ISO $\\uparrow$ | RAVEL-CAU $\\uparrow$ | SCR (K=20) $\\uparrow$ |\n|------------|-----------------------|-------------|----------------|------------|----------------|------------|-----------|-----------|------------|\n| Gemma-2-2B | SAE                   | 1.00        | 0.78           | 0.04       | 0.86           | 0.81       | 0.95      | 0.48      | 0.18       |\n|            | Transcoder            | 1.00        | 0.75           | 0.03       | 0.87           | 0.82       | 0.89      | 0.46      | 0.15       |\n|            | FF-KV                 | 1.00        | 1.00           | 3.57e-04   | 0.84           | 0.74       | 0.90      | 0.47      | 0.05       |\n|            | Normalized FF-KV      | 1.00        | 1.00           | 3.51e-04   | 0.84           | 0.74       | 0.90      | 0.47      | 0.05       |\n|            | TopK-FF-KV            | 0.99        | 0.08           | 1.38e-04   | 0.78           | 0.81       | 0.89      | 0.46      | 0.07       |\n|            | Normalized TopK-FF-KV | 0.99        | 0.08           | 1.41e-04   | 0.78           | 0.81       | 0.89      | 0.46      | 0.07       |\n|            | Random Transformer    | -           | -              | -        | 0.80           | 0.68       | -         | -         | 4.21e-03   |\n\nHere, we leave out some evaluation scores for random transformer, where the metric does not make sense or shows immediately bad scores (Feat. Alive, Explained Var. , RAVEL).   \nNote that a detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)\n\n\n>Missing Transcoder in explained‑variance plots (Weakness 1)\n\nWe apologize for this confusion. In Figure 5, the GPT‑2 transcoder is omitted because its explained variance is far below the plot range. The explained variance is −14 807.46 with a standard deviation of 38 729.33, indicating that the GPT‑2 transcoder is poorly trained; its result should therefore be interpreted with caution. We will make this explicit in the text.\n\n\n>paper is slightly overclaiming at several points (Weakness 2)\n\nWe will adjust or clarify the conclusions based on your concern. Our main claim is that SAEs do not provide a large interpretability advantage over the existing FF-KV view overall. This asserts that a strong, simple baseline is missing in the community, and also possibly highlights the limitations of the current SAE benchmark (if one expects the advantage of SAE). \nAs you pointed out, the autointerp score is slightly lower in FF-KVs, but this is one facet of our evaluations, and entirely, we hope the above take-home message and the novelty of our findings generally hold. As for the chance score, we hope that our baseline of random Transformer already provides a good reference.\n\n\n>poor explained variance on Top‑K FF‑KVs and could use more discussion (Weakness 3)\n\nExplained variance is computed on the FFN output. Because Top‑K FF‑KV zeroes all but the k largest activations (e.g., k=5 for 8000 hidden states corresponds to 0.125% of neurons), its output inevitably diverges from the original, reducing explained variance. \n\nFor a quick check, increasing k for Gemma2-2B (layer 13) easily alleviates this trend, as shown below:\n\n\n| k    | explained varience |\n|------|--------------------|\n| 1    | 0.06               |\n| 2    | 0.08               |\n| 5    | 0.12               |\n| 10   | 0.16               |\n| 20   | 0.20               |\n| 50   | 0.25               |\n| 100  | 0.31               |\n| 500  | 0.58               |\n| 1000 | 0.73               |\n\nWe can easily extend such quick checks to all models and layers, but during author response, we did not have enough computational resources to run a sweep across all layers in all k settings.\n\n>Human evaluation is very small scale and relies a lot on author judgement (Weakness 4)\n\nWe will try to increase the annotation scale for camera-ready, but it is not so easy as the task is labour‑intensive. Let us excuse that the current manual analysis scale (200 features in total) and the setting (handled by authors) already exceed or match existing works —for example, Anthropic[1] (168 features) and the Transcoder study[2] (100 features), both annotated by the authors.\n\n[1] Bricken et al.. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning.  \n[2] Dunefsky et al.. Transcoders Find Interpretable LLM Feature Circuits.\n\n\n>Limited originality (Weakness 5)\n\nThe NeurIPS reviewer guidelines explicitly value studies that yield new insights by rigorously evaluating existing methods. Our work fits this category, and if the reviewer is concerned about the fact that this study does not propose a novel method, we respectfully disagree with the assessment of limited originality.\n\n\n> Possible explanation of the contradiction between this paper and the Anthropic work (Question 1)\n\nWe believe the discrepancy arises from different sampling strategies. Anthropic uniformly partitions each feature’s activation range into equal bins (e.g. 0–10 %, … , 90–100 %) and samples evenly from all bins, thereby analysing low, medium, and high activations. Our study, by contrast, interprets each feature using its highest‑firing inputs, as is typically focused on in other metrics. This difference likely accounts for the contrasting findings."}, {"review_id": "IfGZDHyhP5", "reviewer": "Reviewer_Efr6", "summary": "This work is a comprehensive comparison of the interpretability of SAEs and feed-forward neural networks, using a range of quantitative and qualitative (human) experiments. They find that SAEs are no better than FFs in most situations.", "questions": "n/a", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* I was really excited to see this paper, as mechanistic interpretability is often critiqued for not having robust enough comparisons and evaluations. This work presents a clear side-by-side comparison, with rigorous qualitative and quantitative experiments.  \n* The authors compare across a reasonable number of model architectures and evaluations.  \n* The overall writing is really clear and easy to understand. For example, the color highlighting in the equations to match the description in the paper is great. The framing in the introduction of “bottom up vs top down” was nice, and generally I had no problems understanding what were actually pretty technically complex concepts.\n\n**Weaknesses**\n\n* I don’t have any significant weaknesses to report, but here are some readability suggestions and questions.  \n* There were a few confusingly-worded sentences, e.g.:   \n  * **“**Our study, in a sense, encourages the use of inherently black-box, neural LLMs with skipping the interpretability issue, as we claimed their FFs are, to some degree, interpretable”  \n  * “To mention differences among the methods, first, FF-KV methods can, by definition, achieve a perfect explained variance (reconstruction loss), and SAEs can not.”  \n* This is minor, but in the abstract, the authors claim: “how much better features such proxies have indeed discovered, especially compared to those already represented within the original model parameters — and unfortunately, such a comparison has little been made so far”. I agree that this line of work is extremely important and probably understudied, but I think it’s a stretch to say that it’s not being studied at all. For example, [https://arxiv.org/pdf/2502.16681](https://arxiv.org/pdf/2502.16681) (“Are sparse autoenencoders useful? a case study in sparse probing,”--  this paper is in your references) and [https://arxiv.org/pdf/2405.08366](https://arxiv.org/pdf/2405.08366) (“Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control” are along these lines. There’s also this thread by Neel Nanda: [https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks](https://www.lesswrong.com/posts/4uXCAJNuPKtKBsi28/negative-results-for-saes-on-downstream-tasks) (“Negative Results for SAEs On Downstream Tasks and Deprioritising SAE Research”) which isn’t peer reviewed but still relevant.  \n* The length of the appendix (\\>50 pages) is excessive  \n* **“**Absorption Score evaluates how many features a particular simple concept (e.g., word starting with “S”) is split into. A higher value implies that many features are needed to emulate the targeted single concept, and thus, the feature set is redundant.” I’m confused about this, as often features are hierarchical. For example, what if the “S” concept is composed of “words starting with ST”,  “words starting with SL” and “other words starting with S”? That is, what if a concept really can be decomposed into multiple other concepts? I guess my underlying question (often brought up with SAEs/interpretability), how do you define a “simple concept”?  \n* Figure 1 is hard to read, as the lines are a bit pale and look similar.   \n* “Feature Origin Judgment.” → I’m a little confused about this. Are the authors guessing whether the feature came from an SAE vs FF-KV? If so, what does this tell us about the model interpretability?", "was_revised": true, "final_justification": "After having read the other reviews and rebuttals, I'll keep my score as is.", "rebuttal": "We thank the reviewer for recognizing our work’s contribution. We greatly appreciate these suggestions and will incorporate them in the next revision. Our answers to the questions appear inline below:\n\n>How do you define a “simple concept”?\n\nOur example in the feature‑absorption evaluation merely illustrates that a concept can appear at different levels of abstraction, following existing metrics. In practice, we believe that researchers should choose the appropriate level of absorption, depending on specific research questions.\n\n>What does “Feature Origin Judgment” tell us about the model interpretability?\n\nThis experiment addresses the question: “Do SAEs produce activation patterns similar to FF-KVs qualitatively?” Our human annotators could not reliably distinguish top‑K FF‑KVs from SAEs or Transcoders, suggesting that SAEs and Transcoders do not provide fundamentally clearer activation patterns compared to FF‑KVs, supporting that FF-KV is nearly interpretable as SAEs/Transcoders."}, {"review_id": "j5gSjwZlSQ", "reviewer": "Reviewer_ToPx", "summary": "Inspired by previous work viewing transfomer MLP layers as key-value memories, this paper explores evaluating transformer MLP layers as if they are an SAE using SAE metrics like SAEBench. This paper finds that the MLP layers already function similarly to SAEs on a number of SAE metrics, showing that transformer MLP layers do a decent job of learning interpretable features compared with SAEs.", "questions": "- The plots in Figure 1, containing the core results of the paper, are very compressed vertically making the results hard to read. These plots are used to justify the claim that using the MLP as an SAE is roughly as interpretable as using a real SAE, but this claim is hard to verify by looking at the plots due to them being so small.\n- In Figure 1, the SCR results seem to go very negative - is this expected? What does negative SCR mean?\n- What L0 and width are used for SAEs and the MLP? The paper says the MLPs are already sparse, but how sparse? Comparing SAES with very different L0 does not feel like a fair comparison in general.\n- When trying to interpret the MLP layer directly as an SAE (not using top-k), it's possible for neurons to have a negative value since the non-linearity is typically not a ReLU, and typical activation functions (GeLU, SwiGLU, etc...) have negative components as well. How are these negative values handled? Negative latent values are not valid for SAEs so it seems hard to know how to compare the results to SAEs, or if SAEBench results will make sense in this case.\n- It seems like there are two possible conclusions to draw from this work. Either (1) SAEs were a mistake and the field should return to directly trying to interpret neurons, or (2) SAEs aren't good enough and need to be improved. Which of these conclusions is the paper arguing for? I can see the value of using MLPs as a baseline to compare SAEs and other unsupervised techniques against, but I'm not convinced that directly interpreting neurons is a good direction.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\nThis paper identifies an interesting connection between transformer MLP layers and SAEs, and directly treating the MLP layer as an SAE is an interesting idea. This also has the benefit of being guaranteed to be faithful to the model, and requires no training as it's just a part of the model.\n\nAlso, although not discussed in the paper, this seems like it presents a good baseline to compare SAEs and other unsupervised interpretability methods against.\n\n### Weaknesses\n\nThe paper does not discuss what feels like a core downside of this method, which is that the inner expansion factor of the MLP layers is typically not very large in transformer LLMs, and this means there are not many concepts that can be represented by the MLP when naively using the MLP neurons as SAE latents. The LLM can certainly represent more distinct concepts than there are MLP neurons, so using the MLP as an SAE feels fundamentally limited.\n\nThe paper also does not discuss the core motivation for SAEs, which is neuron polysemanticity. Directly interpreting neurons was one of the first things tried by interpretability researchers, but this was abandoned after it became clear that most neurons are polysemantic (meaning they fire on many different, seemingly unrelated concepts). SAEs are designed explicitly do deal with this polysemanticity. Given this paper feels like it's arguing for a return to a technique that's already been abandoned by the community, I suspect it will not have a large impact on the field.", "was_revised": true, "final_justification": "Framing for this work as a baseline to judge future SAEs / interpretability methods against rather than advocating a return to trying to interpret neurons makes sense to me. Assuming this is emphasized in the final version of the paper, and the presentation issues mentioned are also fixed I will raise my score to 4.", "rebuttal": "We appreciate constructive feedback.\n\n> Downside of FF-KV and motivation of SAEs (Weakness 1 & 2)\n\nWe agree that a key motivation for SAEs is to disentangle more distinct concepts than the limited number of features permitted in MLP neurons. We agree with this theoretical advantage, and our experiments may suggest that, empirically, current SAEs may not disentangle concepts as much as expected, and the provided interpretability is nearly the same as the original MLP representations. More generally speaking, the advantage of SAE over conventional interpretability approaches, such as linear probe, has been debated, and our results will contribute to and encourage the estimation of SAE’s actual advantage. \n\n> Regarding Figure 1 (Question 1)\n\nWe apologise for Figure 1’s poor readability. Please see our reply to Reviewer euvs for the table replacement and our planned visual update. Note that a more detailed breakdown of Figure 1 is shown in Appendix (but we admit the room for presentation improvement)\n\n> Negative SCR scores (Question 2)\n\nSCR can indeed be negative: a negative value simply means that using SAE features **reduces** performance relative to raw activations. Negative SCR values also appear in the SAEBench paper [1] (See their Fig. 7). In our updated table the layer-averaged SCR is non-negative; the negative points in the original figure were outliers.\n\n[1] Karvonen et al., SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability.\n\n> Sparsity of MLPs and choice of L₀ (Question 3)\n\nReference [2] shows that MLP activations are already sparse. Appendix E lists the L0 values for all SAEs; following the Gemma-Scope recommendation, we set $L_0 \\approx 100$ for every model, so sparsity levels are comparable.\n\n[2] Z. Li, C. You, S. Bhojanapalli, D. Li, A. S. Rawat, S. J. Reddi, K. Ye, F. Chern, F. Yu, R. Guo, and S. Kumar. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2023\n\n> Handling negative activations (Question 4)>\n\nPlease see our response to Reviewer hRu1 for a detailed explanation of how we treat negative activations.\n\n> Conclusion of our paper (Question 5)\n\nWe align with conclusion (2): SAEs themselves are not a mistake, nor do we advocate reverting to direct neuron interpretation. Rather, our findings suggest that current SAE implementations fall short of their goals, motivating both improved SAEs and new methods to achieve better conceptual disentanglement."}]

Author rebuttal:

Meta-review: This paper examines whether sparse autoencoders and Transcoders truly provide more interpretable features than the native feed-forward key–value structure in transformer MLPs. By adapting SAEBench metrics directly to FF-KV and its variants, the authors report that FF features perform comparably to SAE features across a range of interpretability tasks, with each side showing small but consistent advantages on particular metrics. A human study and alignment analysis further suggest that proxy features often overlap with, or fail to meaningfully improve on, FF features. The core contribution is to establish FF-KV as a strong baseline against which interpretability methods should be measured.

The strengths of the paper are the systematic analysis of mechanistic interpretability with systematic experiments across several model families. The paper establishes a fair evaluation framework that treats FF as an SAE and it presents useful insights about where SAEs add value and where they do not. The writing is clear, and the authors have provided extensive supplementary material. Reviewers found the idea of reframing FF layers as an interpretability baseline both simple and powerful, with strong potential to recalibrate community assumptions.

Weaknesses noted by reviewers (and with which I agree with) are the limited originality since the work repurposes existing evaluation pipelines rather than introducing new methodology and the modest scale and potential bias in the human evaluation. Some reviewers also point out the  over claims in the framing, particularly regarding whether SAEs are unnecessary.  There were also questions about fairness of capacity comparisons and how negative activations in SwiGLU layers were handled, and finally discrepancies with prior auto-interpretation results. Some presentation issues in the results figures are also brought up by the reviews.  

The rebuttal and discussion were constructive and helpful as the authors clarified the motivation for the key–value lens and provided theoretical and empirical justification for handling SwiGLU. The authors also satisfacorily explained implementation differences underlying the auto-interp discrepancies. They also committed to replacing confusing figures with tables and clearer plots. Reviewers euvs and ToPx found these clarifications satisfactory. Reviewer Efr6 was positive throughout, emphasizing the clarity and significance of the comparison. Reviewer hRu1 remained unconvinced and did not engage in discussion, but most of their concerns were directly addressed in the rebuttal and were similar to those raised by other reviewers who did change their stance.

While the paper does not propose a new interpretability method, I believe it does provides valuable empirical insight, showing that FF-KV is a surprisingly strong baseline, and that current SAE approaches may not deliver as much additional interpretability as assumed. The limitations around human evaluation scale and originality are real, but not enough to warrant rejecting the paper. The authors have also shown a willingness to address presentation and framing concerns, and the reviewers who engaged in discussion seemed to agree on acceptance, which I agree with.
Decision: Accept (poster)