Dense SAE Latents Are Features, Not Bugs
Authors: Xiaoqing Sun
Alessandro Stolfo
Joshua Engels
Ben Peng Wu
Senthooran Rajamanoharan
Mrinmaya Sachan
Max Tegmark
Venue: NeurIPS 2025 poster ()
Keywords: sae
sparse autoencoder
interpretability
mechanistic interpretability
language model
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Sparse autoencoders (SAEs) are designed to extract interpretable features from language models by enforcing a sparsity constraint. Ideally, training an SAE would yield latents that are both sparse and semantically meaningful. However, many SAE latents activate frequently (i.e., are *dense*), raising concerns that they may be undesirable artifacts of the training procedure. In this work, we systematically investigate the geometry, function, and origin of dense latents and show that they are not only persistent but often reflect meaningful model representations. We first demonstrate that dense latents tend to form antipodal pairs that reconstruct specific directions in the residual stream, and that ablating their subspace suppresses the emergence of new dense features in retrained SAEs---suggesting that high density features are an intrinsic property of the residual space. We then introduce a taxonomy of dense latents, identifying classes tied to position tracking, context binding, entropy regulation, letter-specific output signals, part-of-speech, and principal component reconstruction. Finally, we analyze how these features evolve across layers, revealing a shift from structural features in early layers, to semantic features in mid layers, and final to output-oriented signals in the last layers of the model. Our findings indicate that dense latents serve functional roles in language model computation and should not be dismissed as training noise.

Dense SAE Latents Are Features, Not Bugs
Xiaoqing Sun" MIT
Alessandro Stolfo ETH Zürich
Joshua Engels MIT
Ben Wu University of Sheffield
Senthooran Rajamanoharan
Mrinmaya Sachan
Max Tegmark
ETH Zürich
MIT
Abstract
Sparse autoencoders (SAEs) are designed to extract interpretable features from language models by enforcing sparsity constraint. Ideally, training an SAE would yield latents that are both sparse and semantically meaningful. However, many SAE latents activate frequently (i.e., are dense), raising concerns that they may be undesirable artifacts of the training procedure. In this work, we systematically investigate the geometry, function, and origin of dense latents and show that they are not only persistent but often reflect meaningful model representations. We first demonstrate that dense latents tend form antipodal pairs that reconstruct specific directions in the residual stream, and that ablating their subspace suppresses the emergence of new dense features in retrained SAEs Suggessig that high density features are intrinsic property of the residual space. We then introduce a taxonomy dense latents, identifying classes tied to position tracking, context binding, entropy regulation, letter-specific output signals, part-of-speech, and principal component reconstruction. Finally, we analyze how these features evolve across layers, revealing shift from structural features in early layers. semantic features in mid layers, and finally to output-orienter signals in the last layers of the model. Our findings indicate that dense latents model computation and should not be dismissed as training noise.
functional roles in language
Introduction
Sparse autoencoders (SAEs) are unsupervised method for extracting interpretable features from language models [Bricken et al., 2023, Huben et al., 2024, Kissane et al., 2024]. They address the challenge of polysemanticity, where individual neurons activate in semantically diverse contexts that defy single explanation [Olah et al., 2017, Elhage et al., 2022]. SAEs are trained to reconstruct the activations of language model under a sparsity constraint applied to a bottleneck layer, ensuring that only a small subset of latents are active at a time. ? This method effectively recovers interpretable features in variety of models, including Claude Sonnet [Templeton al., 2024] and GPT-4 [Gao et al., 2025].
Ideally, a trained SAE would yield large of interpretable and sparsely activating latents. In practice, however, SAES exhibit substantial fraction of densely activating latents, activating on 10% 50% of tokens (Cunningham and Conerly, 2024, Rajamanoharan et al., 2024b]. These dense latents
"Equal contribution. Correspondence xqsunmmit.. edu and stolfoa@ethz. We use "latent" to refer to an entry in the SAE's sparse hidden layer.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

are challenging to interpret based solely on their activation patterns. It remains unclear whether they arise an optimization by-product, if they instead capture inherently dense signals present the model's residual stream [Chen and Batson, 2025, Rajamanoharan et al., 2025].
In this work, we investigate several properties of dense SAE latents and the residual stream subspaces they span, uncovering evidence that these latents track meaningful residual stream information. First, we observe that when retraining an SAE on model activations with the dense latent space ablated, virtually no dense latents are learned-dens rather than training artifact. We then study the geometry of dense latents and observe that they tend to form antipodal pairs, with each pair effectively reconstructing a single direction.
latents reflect an intrinsic property of the residual stream
We then examine the Gemma Scope suite of SAEs [Lieberum al., 2024] across layers to propose a taxonomy of dense latents. We identify latents whose activations encode positional information, latents reconstructing subspace of the residual stream linked entropy regulation [Stolfo et al., 2024, Cancedda, 2024], latents tracking high-level shifts in the text, latents encoding letter-specific output signals, latents tracking parts of speech, and latents reconstructing the first residual stream principal component direction. We additionally examine how these dense latents transform across layers, finding that there is a pronounced increase in the number of dense latents just before the unembedding, as well as a shift from structural signals in early layers (e.g., position tracking) to output-oriented signals at the end. Our findings provide evidence that dense SAE latents reflect inherently dense mechanistic functions within language models.
2 Background
SAEs. Sparse autoencoders (SAEs) are trained to reconstruct a language model's activations E Rodddd while imposing sparsity constraint [Yun et al., 2021, Huben al., 2024]. This computation can be represented as:
f(x) == o W eneX + bane), (f) = deef + baec,
where f(x) E Rd sparse, non non-negative vector of latents, with dane >> dmodel and is non-linear activation function. SAEs are typically trained to minimize the I.2 distance between the original activation and its reconstruction ||x xx))x) while a sparsity constraint imposed on f by adding sparsity-related loss component or via specific activation functions. We denote the
enc and ((()) dec respectively. Unless noted otherwise, we use "dense" to refer to latents with an activation frequency larger than 0.1.
encoder and decoder weights of the latent at index as ((())
Experimental Setup. We focus our investigation on the Gemma Scope SAEs [Lieberum et al., 2024] trained on Gemma 2 2B [Gemma Team, 2024], which use JumpReLU activation function (Rajamanoharan et al., 2024b]. We additionally train TopK SAES [Gao et al., 2025] on 1B tokens of the Open WebTeex corpus [Gokaslan and Cohen, 2019] for our experiments in $3.1.3 Activation densities for Gemma Scope latents are from Neuronpedia [Lin, 2023], while densities for our TopK SAES are computed over 100M tokens from the C4 Corpus [Raffel et al., 2020]. Full experimental details are in Appendix B.
3 General Properties of Dense Latents
We begin by examining structural properties of dense SAE latents, finding that they arise from specific residual stream subspace ($3.1), and that they tend to cluster in antipodal pairs ($3.2).
3.1 Dense Latents Reflect Intrinsic Properties of the Residual Stream
To determine whether dense SAE latents arise from the training procedure or reflect an intrinsic property of the residual-stream subspace they reconstruct, we perform a targeted ablation experiment. We identify the subspace spanned by the dense latents of an SAE trained on layer 25 of Gemma 2B, then train a new SAE on activations in which this subspace has been zero-ablated. For comparison,
We choose TopK for its reliable training and competitive reconstruction -pooostrccioonsspritty trade-off.

---

(a) Distribution of Latent Densities
(b) Cosine Similarity of Dense Latents
(c) Antipodality Score vs. Density
fis=16386
927-82766
Encader
Density Distribution
Casine Similaniyy
0.6
Latenic
Denulty
0.4
-0.5
0.2
Original
Sperne
10
Actotion Dense Latents
Latent
Antipodelity
Figure 1: General Properties of Dense SAE Latents. (a) Ablating the dense-latent subspace (teal) reduces high-density latents compared to the original (blue) and sparse-latent ablations (orange). (b) Encoder cosine similarity between the top 50 latents with highest density. (c) Dense latents exhibit high antipodality score: they form pairs that reconstruct specific residual stream directions.
we also select an equally sized set of non-dense latents and train a third SAE after ablating their subspace. We repeat this for two dictionary sizes (dase 16384 and 32768).
Figure Ia shows the resulting distributions of latent activation densities. In both dictionary sizes, ablating the dense-latent subspace (teal) yields much fewer high-density latents than the original SAE (blue) and the non-dense ablation (orange). This result implies that densely activating latents not mere training artifacts but instead track a dense residual-stream subspace whose presence drives the emergence of dense latents. As additional evidence that dense latents are training artifacts, in Ap- pendix A.2 we show that longer training does not reduce the number of dense latents. We further repli- cate this dense-subspace ablation GPT-2 and LLaMA 3.2 with the same outcome (Appendix A.6).
3.2 Dense Latents Cluster in Antipodal Pairs
We now examine the geometry of dense latents and observe that they tend to form antipodal pairs. That is, as shown in Figure 1b, there exist many pairs of two dense latents that have nearly opposite decoder vectors (we find a similar result for encoder vectors). This suggests that the SAE allocates two latents in the dictionary to represent 1-dimensional line.
To quantify whether this phenomenon specific to dense latents, we introduce an antipodality score for a latent i. We first compute the pairwise cosine similarities between the latent's weights (both encoder and decoder) and those of all other latents. Then, we compute the maximum product of encoder and decoder cosine similarity across all pairs (i,j) for all + Formally, we have
sim (W(() encr
W enc)
sim (Wf() doe woll))
max j
where sim(u, v) denotes the cosine similarity between vectors u and v. This score reflects the extent which latent forms antipodal pairing with another latent: high values of 8; indicate that there is another latent j with both encoder and decoder weights nearly opposite in direction to those of i.4
(a) Topk SAE
(b) Absolute SAE
Encoder Coxine Siniaity
Lanatirdos 1.5
SEEFS
As shown in Figure 1c, s; and the activa- tion density of latent i are strongly positively correlated. The majority of dense latents particularly those with activation frequency exceeding exhbb pairwise scores greater than 0.9, supporting our conclusions above. We provide -anssit-antiooaalityy visualizations for additional SAES in Appendix A.I, showing that this trend holds consistently across SAE architectures (JumpReLU and TopK), models (GPT-2 and Gemma), and layers.
Indox
Figure 2: AbsoluteTopK SAEs show no antipo- dality. Allowing the SAE to have both positive and negative latent activations removes antipodal dense latents.
Although high values of could be produced by two nearly identical latents, retaining such pair would be redundant a scenario do not observe. Evidence for this is provided in Appendix A.4.
3

---

Percentage of Dense Latents
Context-Sinding
Meaning
Nullspace
40
Alphabet
PCA
Sentencee-racking
Peragraph--racking
Context-Tracking
%
10
15
20
25
Layer
Figure 3: An overview of our taxonomy of dense latents, for every layer. See Appendix E.1 for how we created this plot.
Additionally, we train an Absolute'TopK SAE, which allows activations SAE latents to be negative, and enforces sparsity by taking the TopK latents with greatest absolute activations. This effectively allows the same latent direction to be used in both "positive" and "negative" directions for reconstruc- tion. We compare this TopK SAE trained with the same seed, and show that this AbsoluteTopK activation function eliminates the antipodal dense latents (Figure 2).
4
Taxonomy
Having established that dense latents persistent and geometrically structured. now investigate their interpretability. We identify classes of dense latents based on the model signals they represent:
Position latents ($4.1) fire based on token position relative to structural boundaries (start of sentence, paragraph or context) and appear early in the network.
Context-binding latents ($4.2) represent context- deeeeddt semantic content and exhibit coherent chunk-level activations, potentially representing high-level ideas within the context.
Nullspace latents ($4.3) track components of the residual stream that have minimal impact next token prediction. They instead regulate prediction entropy.
Alphabet latents ($4.4) promote broad sets of tokens sharing an initial character.
Meaningful-word latents ($4.5) have activations related to the token part-of-speech tag.
PCA latents ($4.6) lie almost completely within the first PCA components of the activation space.
4.1
Position Latents
We first identify a class of dense latents whose activations track the current token's position relative specific boundaries. Context-tracking latents track token position w.r.t. the BOS token, paragraph-tracking latents track token position w.r.t. paragraph start, and sentence-trackin latents track token position w.r.t. sentence beginning. Context-position latents similar to "position neurons" from prior work [Gurnee et al., 2024]; the other categories are to the best of our knowledge novel.
To find these latents systematically, we use Spearman's 's rank correlation coefficient p. For each dense latent, we capture the projections of the residual stream activations onto its decoder vector for 5000 1024-token-long contexts. We find between this projection and the distance from the last period, the last newline, and the beginning of the input. These boundaries act proxies for "beginning of sentence", "beginning of paragraph", and "beginning of context", respectively.
*We use the projection of the residual stream rather than the JumpReLU activations of these latents since we hypothesize that the direction itself encodes the positional information, regardless of whether the magnitude exceeds the learned JumpReLU threshold.

---

Agern,
choie rhonon wetcn mac $13 mill Fon US and help wo wt Cone Hinn
The asd stance recddos 15 mill on toha Sn moso Sicciii Lebonsce and So million alscooed
Adando Cosina Hooa fied
baniruptcy Intte
Thursdyy
forietlagee and
the troub refestis hizeectiing boyerfi
Noted in Belute
Telheel Caselé went trough Chaptelft (11 bankknppts in March 2011: but movee did not
wermed employess the siall ligat's would happen by Augint 18 buger cound be founsiin
ssal lo the companys frannis on shotton Beiter
AdarucERy Dedines Farthonn
lerte deet out in whilhh in management
IUNAERIn
Leburnoa offiaal ane welk Lornal Hhs Ub Doigonemmanty do aisnn 5 well on lanr but Uhv
gpreaid el the nove lcongnaainal (DOVG 18t Sippeddets sraing the arrount of money needed to deal
200s mnteetomee profiis have nedinod by thede mes Mamidi marted Saturation nithl
Recel Cadtooh inte finnnoa downumm just one seriet baniggss the Adtnnd which
the Allante EMd January2014 Atams City has swen decining mnvamuet every your cncel
along continuing reeerrgeneoononeee
with the autbreak 5 "onaa mous le
immrnttional Devetomenna Dotor actattie
Ammb 18 Labanon Persthyyan enno the new embttanes atan Apr 2nd prans
immon Untvrraty
Sgency saidin
Support
leaders sindllrd New and dielloee Adardic
"pacticsly by IN in supoon pricte he to peosent womeeaa pontant
done
enbause won kation and andoummmttiall
Konsed online gambiing ng hoping the roverues wou of boos the and laste castnos and aod
beaans appoweed from beaak property taams which city wer the stete paeset
Aurreadi Ton
Sylan volurnenss partippone training courseson raking moanenees
insice
agrtfcantly to saate reverues Sor the recuite have been dtopporning in
Munneli (Sovemrr Chra Chriss chalsegaed a year and law wiihh bannad sport gambii ri
(ower-yannaann 51% a makker new avoizenne mingon BI espread of COVTD 1o Lesanion pnssconnaonnd foril 22nd o8 Aescncan
Ammisssae abanon
extrvitrre in the US except Nevars Debware, Nompra and Oragon Ther Singi chas mode eolii
game Supreme Eoant bid irfen uncretin whenthr the
rade the casner
intarratt of betutt Uhhon cnnteeppn Allli
Steer Pestare The couire curranlty weed by the teeet bitertatmes, Cree.
Steer to Feature 3: The us geecriment aas ulse provicee $1.5 aatiaan an erengaee
Reeal
aodicrsate
healrr
Hanal. The Taj hanal the only other codre in
eossatincs Feature
relugasa Jortes.
Ar Arteer CaTs still oger.
LtN
conpany will able to Find oea, The corpany hes soan takes with nater of
Lesannse greermenn to help the country address the cosca-1a panduntcy pasdenic che sale. LL
company alsa talke with the state Serney, ahiic in trysigg hels tud sape--
Figure 4: Context-Binding Latents. Activation patterns of layer 12 antipodal pair 7541 (blue, feature 1) and 2009 (red, feature 2). In the first context, they seem to be tracking "casino facts" vs "looking for buyer", while in the second context, they seem to be tracking "healthcare" vs "press conference". Their corresponding completions are in line with the concepts they activated on.
Figure shows the resulting trends: sentence-tracking and paragraph-tracking latents prominent before layer 10, while context-position tracking latents are present throughout the model. Figure 15 shows p for all latents across layers. We can clearly see groups of outlier latents for each category, and thus classify latents as belonging to that category if |p| > 0.4. Indeed, examples in Appendix E.2 confirm that the identified outlier latents have position-tracking behavior. Notably, Appendix E.2 also shows that paragraph-tracking latents are agnostic to artificially adding formatting newlines, suggesting that this direction in the model tracks true semantic paragraph breaks. Thus, our "distance to newline objective" is just a proxy. We also note that latents with high P with periods also have high o with newlines, since newlines and periods are correlated in text. In Figure 17, we thus show the p for sentence-tracking vs. paragraph-tracking across all dense latents.
At a higher level, it makes sense that the model represents these features in dense way: positional information is always relevant to the model's predictions (e.g.. it must track how far it is in a sentence to correctly predict a period), so the model might store this representation in a consistent direction in every hidden state, which is then learned by the SAE.
4.2 Context- Binextg Latents
We next identify class of dense latents that encode different semantic concepts depending on context. Unlike interpretable sparse SAE latents typically associated with fixed meanings, such as the "Golden Gate Bridge" feature in Claude [Templeton et al., 2024], these dense latents appear to bind to the main ideas of the context.
We first observe that some dense latents, particularly in middle layers, activate on long consecutive "chunks" of tokens. Examining the activations of such latents, we notice empirically that such latents fire on highly specific concepts within a context, but the concepts vary across contexts. We generate explanations of these latents with an LLM and confirm that they seem to be more context-specific than sparse latents (see Appendix E.3).
One possible interpretation is that these latents represent general but abstract, difficult-to-inteppee to-interpeet properties. However, we also observe that within an antipodal pair, the active latent often switches when the main topic or entity in the text changes (Figure 4, Appendix E.4). This raises the hypothesis that such directions act as "registers" in the residual stream for tracking the active concept, rather than simply representing generic properties.
'While positional latents also exhibit consecutive activations, here we refer to non non-positional latents whose activations cannot be explained by position alone.
5

---

Laver
Latent Pair
In-context
Out-of-context
12
(14906, 14599)
0.051
0.717
Fraction of Correct
12
(2291, 13295)
0.028
0.760
0.6
12
(7541, 2009)
0.043
0.711
13
(3517, 46)
0.036
0.742
13
(15275, 11449)
0.029
0.704
13
(12613, 7655)
0.028
0,531
14
(11575, 2411)
0.047
0,798
14
(8515, 15297)
0.041
0.603
Layer
14
(6699, 1802)
0.037
0.678
16
(2889, 8811)
0.024
0.665
Figure 5: Fraction of correct flips when steering, for all latent pairs that have at least one latent f >> 0.2, and 2 40 flips. Points are sized by number of flips..
17
(10495, 491)
0.051
0.669
Table 1: Fraction of "unclear" judgments using in-context examples versus out-of-context examples, for the highest- scoring latents by flips.
We thus perform a steering experiment to find the causal effect of these directions. For each antipodal pair (FI, F2), we prompt Gemma 2 2B with input text from the RedPajama dataset [Weber et al., 2024] and generate completions without steering, steering to FI, and steering to F2. An LLM judge [Gemini Team, 2025] is then asked whether each completion is more in line with activating examples (from the input context) of F1 or F2, or unclear. Further details the methodology are in Appendix E.5.
Since the unsteered generation may already favor Fl or F2, we quantify steering success by the fraction of flips from the unsteered judgment that align correctly with the steering direction. For several mid-layer latent pairs, steering reliably shifts completions towards the specific concept previously associated with the latent in that context (Figure 5). However, when judged against out-of-context examples, the rate unclear judgments rises sharply (Table 1). While difficult to rule out the possibility that these directions encode "general uninterpretable" features, the specificity of the steered generation bringing up context-related ideas suggests that these latents could bind to concepts in context- deeeetdeeeeedentt, rather than globally consistent, way.
Previous works have uncovered "binding mechanisms" that help the model keep track of in-context associations between entities [Feng and Steinhardt, 2024, Feng et al., 2024]. While our findings do not directly prove such mechanism, they raise the possibility that dense subspaces may play similar functional role, distinguishing the currently active semantic concept. Further work could explore the circuits [Marks al., 2025] involving such subspaces, and challenge the assumption of globally monosemantic directions.
4.3 Nullspace Latents
Previous work has identified Wu quasi-nullspace-the subspace spanned by the last singular vectors of the unembedding matrix Wu-which accounts for a substantial portion of the residual stream's norm, yet has little direct impact next-token prediction [Cancedda, 2024]. Since this subspace carries high norm, we hypothesize that some dense SAE latents are allocated specifically to reconstruct it.
To test this, we compute the singular value decomposition Wu UEVT. Then, we study the composition of an SAE latent i's encoder weight with the space spanned by the last k left singular vectors U....... U_1 of We by computing the fraction Pk of the norm of its encoder weight (i)) that lies in this subspace:
enc
(i)
ag 25=1
(2)
A histogram of a10 for the SAE trained at layer 25 of Gemma 2 2B (Figure 6a) shows that 99.6% of latents have an < 0.2. We designate those with 0 > 0.2 as nullspace mallspace-aligned. Interestingly, 75% of them are high-density, and account for 40% of the high-density latents in the SAE.
Unlike other dense latents, nullspace-aligned latents are hard to interpret via their token-level activation patterns. Additionally, the tokens they promote are typically uninterpretable "under-traines

---

(a) Norm in W, Quasi-Nuil Space
(b) Changes in Entropy Upon Ablation
(c) Entropy Modulation via RMSNorm
Reletive Count (1)
10K
AEntropy
1000
Count
0.01
SEEVI
0205
02251
05101
0758
2L301
12u
MEntropy upon Ablation of franeay fu4225
Donsity
<0.1
>0.1
Index
Ablation
Standard
Frozen RMSNorm
Figure 6: Nullspace Latents. (a) A small fraction of latents concentrate norm in the final 10 singular directions of Wu. with high-density latents overrepresented in this group. (b) A pair of such latents correlates strongly with model output entropy. (c) Ablating this pair lowers entropy: the effect substantially decreases when R.SNorm scaling is frozen.
fiseapy frez2s correlate w/ Entropy
Index
Letter
Density
Tup Tokens
Entrapy of Model Output
15287
R
0.16
0.98
_RI, _rb, got8, _ri, _r, _RS, R, _RR
13531
M
0.15
_n. MY. _nM, _mm _nf, _ne,
30
T
0.16
0.99
_TT, _C, 17, TC, _tc, T2, _TS, _TD
1761
D
0.14
0.98
_DD, _D. DB. DD. D2. DD. DP, DS,
7342
1
0.13
0.91
I8, i, IC, I, IE, 18, 1P. _IM IR, IV
2651
U
0.11
0.93
_uA, U, UT, UV, _0, _UF UDD UE, UA
4664
0.14
-getC, __CC, getc, _c, BetC. CO. Co _Cs
Carnelation: 0.62
357
B(+R)
0.006
0.91
_BR, _rr Br, BR, Braa br, Bra, br _L,
500
1000
12114
S(+L)
0.006
SLide
fLane - fuxass
14857
C(+l)
0.006
0.91
_Car, _cur, Dar, CUU, cur, CUR, Kur
Figure 7: Entropy Correlation. A pair WT nullspace-aligned correlates strongly with model output entropy.
Table 2: Examples of Alphabet Latents. Latents from layer 25 of Gemma 2 2B that promote suppress tokens sharing an initial letter. "Metric" is the fraction of top 100 affected tokens starting with that letter.
tokens [Land and Bartolo, 2024]. Motivated by prior work linking the Wu nullspace RMSNorm- based [Zhang and Sennrich, 2019] entropy regulation mechanism [Stolfo et al., 2024), we investigate whether these latents encode this internal computation.
To test whether these latents causally influence output entropy, we ablate the residual stream along each latent's 's decoder direction by setting its value to the corresponding decoder bias, thereby removing information in that direction. We then measure the change in per-token entropy of the model's 's output distribution. Figure 6b reports the entropy change for all latents with a10 > 0.3 (one per antipodal pair to avoid redundancy), compared to a control group of 50 randomly selected latents.?
We find that some nullspace latents produce much larger entropy shifts than the random baseline, indicating that they encode signals relevant to entropy modulation. particular, latent 14325 has a disproportionate impact on output entropy. To test whether this signal is used by the model in conjunction with RMSNorm scaling (as in Stolfo et al. [2024J), we repeat the ablation while freezing the RMSNorm scaling coefficient. Figure 6c shows that the entropy change diminishes under this intervention, suggesting that the model uses this direction to modulate entropy via RMSNorm. Furthermore, Figure 7 shows that the combined activation of the antipodal pair formed by latents 13748 and 14325 is strongly correlated with output entropy, further supporting this interpretation.
While these results highlight the functional role of specific nullspace latents in entropy regulation, not all latents in this subspace behave similarly. Some exhibit negligible impact on entropy when ablated. We speculate that these may track different internal signals-one one such candidate is the attention sink signal, which has also been associated with the Wu U nullspace [Cancedda, 2024]. Overall, these experiments provide mechanistic evidence that nullspace latents correspond to internal model computations.
"The entropy changes for the random latents aggregated into single boxplot.
7

---

Meaning
Noun/Proper Noun
Verb
AUC
0.4
10
20
0
10
20
10
20
Layer
Layer
Layer
Figure& 8: Meaningful-Word Latents. (Left) AUCs of predicting feature firing, from whether the POS tag is within the specific category. "Meaningful word" and "noun/propernoun" are good predictors, while other categories like "verb" are less predictive. (Middle) Example of L2: pair 1 15089 (blue), 13092 (red) firing patterns on a document, where 15089 fires on "meaning-heavy" words while 13092 fires proper nouns and functional words (the, in, a). (Right) Example of L3: 7507 firing patterns, where it fires selectively on proper nouns.
4.4 Alphabet Latents
We identify a class of dense latents that selectively boost or suppress large sets of tokens sharing the same initial letter. Unlike prior work that linked latents to the current token's 's first letter [Chanin et al., 2024], these instead relate to the next token's initial character.
To discover these latents systematically, we examine each latent's top 100 positive and negative logit contributions by projecting its decoder weights onto the vocabulary space. Then, collect the corresponding tokens, and select latents where either set contains at least 90% of tokens starting with the same character (excluding the space character " "). At layer 25, this procedure yields 114 such latents, of which 21 have activation density >0.1, accounting for 20% of all dense latents. These latents span range of antipodality scores and activation densities, but notably appear high-density features only at the model's 's final layer. We provide some examples from this layer in Table 2.
Interestingly, we observe multiple latents for each letter, varying in specificity: some target broad set of short tokens sharing only the first letter (e.g., "b" or "c"), while others focus on longer tokens sharing a multi-letter prefix (e.g.. "br" or "cu"). We attribute this granularity to feature splitting [Bricken et al., 2023] possibly driven by n-gram frequency, which yields latents with differing activation densities. These latents illustrate how SAEs dedicate dense units to encode output-specific signals related to next-token lexical structure.
4.5
Meaningful-Woon Latents
The next class of latents that we investigate are those whose firing can be well predicted by the part-of-speech (POS) tag of the token. We create reduced set of high-level tags from the Brown Corpus [Francis and Kučera, 1979] by combining similar tags (e.g., combining piural and singular forms of nouns), and capture dense latent activations on 10k sentences (< 200k tokens) from the corpus. Then, for each latent, we calculate the AUC-ROC of predicting the binary latent activations given the binary vector of whether token is within the high-level POS category. Intuitively, this AUC reflects how well the interpretable linguistic category predicts the latent.
We find that even these high-level groupings are not enough to achieve a high AUC (Figures 8 and 20), and propose further grouping of these tags into "meaningful words", where a token considered "meaningful word" if it is one of [nouns, proper nouns, verbs, adjectives, adverbs). The resulting binary-binary predictor has a decent AUC (Figure 8) of Rs 0.8 for many dense latents in early layers, suggesting that the model contains dense subspace tracking the presence of these meaningful words.
4.6 PCA Latents
Since the top principal components (PCs) are a large fraction of the variance of the activations, one might expect an SAE learn dense latents that simply reconstruct this subspace. However, we
SSee Table in Appendix E.6 for our full mapping.
8

---

Medien Principal
Frection (%)
Layer Endlo
50
40
0.01
25
22
index
Deraity
Deralty
Dorsity 0.3
Layer
Figure 9: Layer-wise Dynamics of Dense Latents. (a) Fraction of dense latents (at various density thresholds) across residual stream SAES at different layers of Gemma 2 2B. (b) Median principal angles between dense-latent subspaces, showing a shift in subspace structure from early to late layers.
find that this hypothesis is only partly the case: as shown in Figure 21, an antipodal pair of latents consistently reconstruct most of the first PC (cosine similarity 0.75), but other latents do not have a large norm percentage in the top PC, even up to the top 5 PC components. The top PC-aligned latents are generally not immediately interpretable and do not fall into any of our classes above. Interestingly, decreasing or increasing the SAE Lo and dictionary size does not eliminate PC-aligned latents nor result in significantly more of them (Figure 22).
4.7 Layer-wise Dynamics
As noted in the taxonomy of dense latents above, and visualized in Figure 3, each class of dense latents found in specific layer ranges. Dense latents early layers have more token-dependent activations and track positional information, those in middle layers represent more conceptual directions, and those in the final layers are mostly mechanisms that the model uses control its output. Inspired by these observations, in this section we further examine layer-wise characteristics of dense latents.
Number of Dense Latents. First, we study how the number of dense latents changes across different layers of the model. Figure 9a illustrates the fraction of latents exceeding density thresholds of 0.05, O. 0.2, and 0.3 each layer. In the early layers (0-4), we observe transient spikes in latents just above the 0.05 and 0.1 thresholds. These latents are largely the part-of- part-of-speech related latents in $4.5. The absence of similar spikes the 0.2 and 0.3 thresholds suggest that these early fluctuations arise from SAE training variability rather than fundamental differences in the information encoded at different points of the model's residual stream. Across the middle layers (5-23), the fraction of dense latents is remarkably stable for all thresholds. Finally, the model's 's last two layers exhibit an increase the number of dense latents, indicating a final emergence of dense features prior to unembedding.
Consistency of the Dense Subspace. We next ask whether the subspace spanned by dense latents remains stable across layers or varies over the model. For each pair of layers, we compute the principal angles between the subspaces defined by latents with density > 0.2, then take the median angle as a summary statistic: values near 0" indicate largely overlapping subspaces, while values near 90º indicate dissimilarity. Figure 9c visualizes these median angles for every layer pair of Gemma 2B." Three clusters emerge. Layers 0-4 share a common dense subspace (low angles). This shifts in the middle of the model (layers 10-22), where a new stable subspace persists (mutually low angles). Finally, the last few layers exhibit a pronounced change (large angles relative to earlier layers), consistent with the rise of alphabet and nullspace latents before the unembedding.
5 Related Work
Transformer models are thought to represent features as linear directions in activation space [Mikolov et al., 2013, Bolukbasi al., 2016, Elhage et al., 2021, Nanda et al., 2023, Park et al., 2024, Olah, 2024], with many more features than neurons, leading to superposition [Olah et al., 2020, Elhage et al., 2022]. Early work explored sparse dictionary learning to interpret
Sparse Autoencoders.
We find that using a slightly higher density threshold (0.2) makes the subspace similarity pattern more pronounced. The same plot with a lower threshold (0.1) is shown in Appendix A.3, showing the same clustering trend but with reduced overall similarity.

---

these representations [Olshausen and Field, 1997, Faruqui et al., 2015, Arora et al., 2018, Zhang et al., 2021]. More recently, sparse autoencoders (SAEs; Ng et al., 2011) have emerged as a scalable and effective implementation of sparse dictionary learning for transformer-based models [Yun et al., 2021, Bricken et al., 2023, Huben et al., 2024, Rajamanoharan et al., b024a,b, Kissane et al., 2024, Bussmann et al., 2025] that can recover meaningful and causally important features [Templeton et al., 2024, Gao et al., 2025, Marks et al., 2025].
Interpreting SAE Latents. As SAEs have gained traction, recent work has focused on interpreting the meaning of their latent features [Chanin et al., 2024, Leask et al., 2025]. Building on the neuron interpretation methodology in [Bills al., 2023], several recent works interpret SAE latents systematically. Templeton et al. [2024] propose rubric-based evaluation method which a language model (Claude 3 Opus) scores how well a proposed feature description aligns with the contexts on which the latent activates. Similarly, Paulo et al. [2024] propose a pipeline in which natural language interpretations for SAE latents are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non- activating contexts. Other recent efforts explore automated interpretation approaches based on self-interpretation strategies [Kharlapenko et al., 2024]. A recurring observation across multiple studies is dense latents, which activate more than 10% or even 50% of tokens [Cunningham and Conerly, 2024, Rajamanoharan et al., 2024b]. Chen and Batson [2025] take the 10 most densely activating latents in a cross-layer Transcoder trained on Claude and attempt to manually interpret them, finding plausible interpretations (e.g.. "activates on commas," " "activates on non- non-terminal tokens in multi-token words") for 6 of the 10 features. In contrast, Rajamanoharan et al. [2025] view dense latents as an undesired phenomenon and propose frequency- -bsed regularizer to discourage their emergence during training. Whether these latents reflect meaningful internal computations or arise as undesirable artifacts was up until our work an open question.
Dense Language Model Representations. Prior work has also identified dense signals in language model representations more broadly (i.e., components that encode information consistently across many tokens). Gurnee et al. [2024] present taxonomy of universal neurons that appear across GPT-2 models trained with different seeds. Among these, they identify neurons that encode positional information. Chughtai and Lau [2024] similarly identify dense positional features in an SAE trained on GPT-2's s layer O, though they do not explicitly analyze their activation density. Finally, Stolfo et al. [2024] describe neurons that regulate model confidence by tracking entropy and connect them to a component of the residual stream aligned with the quasi-nullspace of the unembedding matrix.
6 Discussion, Limitations & Conclusion
Our work shows that dense SAE latents discover intrinsically dense features in the underlying language model representations. This challenges recent efforts that aim remove dense latents with ad-hoc penalties in the SAE loss function [Rajamanoharan et al., 2025]. Our results motivate future feature-extractiio mechanisms that are able to find features that are not necessarily sparse. For example, such techniques might include SAE designs that allocate autoencoder capacity for representing dense subspaces, approaches that optimize circuit sparsity, or techniques like APD [Braun et al., 2025] that focus on parameter sparsity.
Limitations. Although our work identifies some classes of dense latents, do not claim that all dense latents encode interpretable or meaningful signals. We hypothesize that some dense latents are noisy aggregation of sparse features rather than "true" dense feature, and distinguishing between these remains an open challenge. Moreover, dense latents may learn a basis that spans but does not align with the set of true dense model representations, since dense latents co-occur extremely frequently, and a linear combination of the "true" basis works for reconstruction too.
Despite consistently observing the antipodality trend across both TopK and JumpReLU SAES and across models (Gemma 2 2B and GPT-2 Small), our interpretability analysis primarily focuses on JumpReLU SAEs trained Gemma 2 2B, using single dictionary size and sparsity constraint per layer. Future work could broaden analysis to more models, SAE architectures, and SAE sparsities.
Most notably, we have explained less than half of dense SAE features. We view understanding the rest of these latents as exciting future work that could provide insight into frequently-active, fundamental mechanisms and representations in language models.
10

---

Acknowledgments
circuits pub/ We would like express gratitude to Arthur Conmy, Neel Nanda, and Vilém Zouhar for their valuable feedback and insightful discussions at different points during the development of this project. AS acknowledges the support of armasuisse Science and Technology through a CYD Doctoral Fellowship.
References
Al @ Meta. The llama herd of models, 2024. URL UtL.tttp::/arxxw..rrr////
21783.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018. doi: 10. 16.1ta2/ 00004. URL Q18-1083 1034/.
org/
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language mod- blob. core. windows net wtwp//euuonnepllaaaeeeee
els. 2023. URL paper/index. html.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word em- beddings. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https: http:///pooeeeegg.e.ee/// neuripsu httppppp..ppppp...pppppp ccc/paperffilesse www.ee/pppeee///e//. 1456:offeeaaa7777776644441111162 Papeeeeee.e--aae........ pdf.
Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel, and Lee Sharkey. Interpretability in parameter space: Minimizing mechanistic description length with attribution-based parameter decomposition. arXiv preprint arXiv:2501. 14926, 2025.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. URL /ttpass////aasf/t/////e 2023/monosemanticc wwccc-ffaauureeeeeeeeeeeeeeeeeee wwwix.iidx../iii html.
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders, 2025. URL bttps://arxiva http:/bb/250000 17547.
Nicola Cancedda. Spectral filters, dark signals. and attention sinks. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3792-4808. Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.186661///2222..ooooooggg. URL httpp::/aclsseee htap:::///caaathollgg.ooo org/ a011..222221
263/
David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2024. URL
/sttps//arrxiv.org wrgg/ab/////000 14507.
Brian Chen and Josh Batson. Interpretable dense features, 2025. Saanaformer-circuie Baciitss.uub//// ppbpppp5 http:/2222/ppp1l1p/ httpr/arlllllupapadeeeeeeeeeeeee htw.htmmllccc.c #dense.
URL https://
Bilal Chughtai and Yeu-Tong Lau.
Understanding positional features in layer http://sste/cteeegggppcoocee LessWrong post.
saes, July 2024. URL https: /tops://wwws wewswwww..........mom.m...m wwt.:occmcooo/eeeeeee
understanding- i foriiiomm---ffuuureeee--e--eeee-
Hoagy Cunningham and Tom Conerly. Comparing topk and gated saes to standard saes, 2024. URL. http:://trannsforerr-iiii pwwp:/24///p/pa//ep//// updatep index.ddxxx../// html# liiek-gtted-coooo
11

---

Nelson Elhage, Neel Nanda. Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfiel-DDodss, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. URL www.iiittttttbrar...ee...pp 2021/ wtt.w/orrww//////// index. html.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodde Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg. and Christopher Olah. Toy models of superposi- tion. Transformer Circuits Thread, 2022. URL https: httte://r/ssfooomeeeeee circuitir wetniiis..uu///0/ pub/2022/ toy_model/index. html.
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A. Smith. Sparse overcom- plete word vector representations. In Chengqing Zong and Michael Strube, editors. Proceedings of the 53rd Anmual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume Long Papers), pages 1491-1500, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10/1115/11/11--1144. URL https://aalaanthollgg... htt.aaaaa..ooo.ooooo....--.-.. 1144/.
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context?, 2024. URL attps://arxxe org/ has///b///310 2310. 17191.
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Monitoring latent world states in language models with propositional probes, 2024. URL WttLhttss///rxiii wttpa/iww..orggbb////0000000/ 19501.
W. N. Francis and H. Kučera. Brown corpus manual: A standard corpus of present-day edited american english, for use with digital computers. Technical report, Department of Linguisties, Brown University, Providence, Rhode Island, 1979. URL http: httt::://ccaaa uib. boobrown/bcm.. html.
Leo Gao, Tom Dupre Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In The Thirleenth International Conference on Learning Representations, 2025. URL httph::t////eeeeee /ttpps///penrrviie..r forum?id=tces2:92WW0)
Gemini Team. Gemini 2.5: Pushing the frontier with advanced
.06261. advaced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL http://aaxvvvv.r//aa//
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024. URL
https://arxiv. www.///g/g///..../////////// 00111
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus 2019. URL URL.hhttp:/Skkliiono00 .ithhb............. So/DpennVebTextCorpus.
Wes Gurnee, Theo Horsley, Zifan Carl Guo, Tara Rezaei Kheirkhah, Qinyi Sun, Will Hathaway, Neel Nanda, and Dimitris Bertsimas. Universal neurons in gpt2 language models, 2024. URL https://arxiss wrgg wwt.:oo/////aaa/eeeee ab/////ss.1...//////////////1//a 12181.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357 362, sep 2020. doi: 10.1038/s41586-000 -22-8-6--2------2 URL btttps://olt http:://ooooooo....eoo httg//..1038/44111 /1t.////1156-// -
Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URI. https: httpptt: /ttp:///e/p/eeeee//e//ee net. ffd F76brRSLeK.
12

---

Dmitrii Kharlapenko, Stepan Shabalin, Neel Nanda, and Arthur Conmy, Self-explaining sae features, 2024. URL https://wwit http://wwwwwwlggmete..e.emmm www.nmentforruuem http://toooooooooot////////////e ott.pppott///v66oxxcss/xxxx/p/p/ self-explaining-sar -aapppaiiinn--se--eatreee
Connor Kissane, Robert Krzyzanowski. Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Inter- preting attention layer outputs with sparse autoencoders. In ICML 2024 Workshop Mechanistic Interpretability, 2024. URL httpht:://ooameeee /atps///ooppeeeeeeeeee net/forummdd= fieeruuDDijd
Sander Land and Max Bartolo. Fishing for magikarp: Automatically detecting under-trained tokens in large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11631-1 1646, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 11863//112224.mmmlpp caammmmmminn.9999 649. URL http::::/accaattaoto/tt/// wwwpchhooogyooo///0000 org/2024. emnlp-me main. 649/.
Patrick Leask, Bart Bussmann, Michael T Pearce, Joseph Isaac Bloom, Curt Tigges, Noura Al Moubayed, Lee Sharkey, and Neel Nanda. Sparse autoencoders do not find canonical units of analysis. The Thirteenth International Conference on Learning Representations, 2025. URL attps://openree
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autocncoders everywhere all at once on gemma 2, 2024. URL pattps://ariv.. wrgg/bs////a...o.. 05147.
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023.
URL httpp:://ww..neuropeddia... org. Software available from neuronpedia.org.
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. In The Thirteenth International Conference Learning Representations, 2025. URL https: //openreview wwtpearrviee..ee/fforroreeeeeeee oet.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL htpp:///rrcoos hew.rr/oooooiooooooeeeeeeeeeeeee newrips. ctt.ccccppppppe. cc/paper_ fi
Neel Nanda and Joseph Bloom. Transformerlens. http:::/giicbb..oom/raattmtmrmea TransformerLens. 2022.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi, editors, Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 16-30, Singapore, December 2023, Association for Computational Linguisties. doi: 10.18655. l/aclanthology, org/ 2223. blackboxnlp-1.2//
URL https:
Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(201 2011. URL https:
//
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 09.23315/distil....
URL http://ciiiiill..ppooooff//eeeee
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction circuits. Distill, 2020. doi:
URL
https://distii ceteccillluubb///e httpp///////rrrrtt/////////////e cwtrciitezooommoo. in.
Christopher Olah. What is linear representation? what is multidimensional feature? Transformer Circuits Thread, 2024. URL htthttp:: cttp:://traaaffomerrrriiiiiti circuits http:///r.r.......p//////////// http//2224jjyuuuddaae
13

---

Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision Research. 37(23):3311-3325 1997. ISSN 0042-6989. doi: https://doi. 00p/0000066650022266999990111110
URL https://wwzz sciencedi com/science)
arrice//p11/004429999700111
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix, editor, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages (8643-39666.. PMLR, 21-27 Jul 2024. URL https: //proceedings.. mlr. prress/2355para2e html............ html.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-Bon Bucc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL http://ss////occeeeeeiiaaae/eee neurips...... cc/ ww/pappr.rc...cc ptaperffille/ppperr//e pdf.
Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. Automatically interpreting millions of features in large language models, 2024. URL https: /ttp/v/aavvvvv.///////////ai.iva http://aaxxvvoooabbas/eeea/ 13928.
Plotly Technologies Inc. Collaborative data science, 2015. URL URL wttthhtt//pplo...
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. URL
https://nnsiihtcciiic.. s3. aas-east-11 amazonaws wwwp:/owwwww.ooommuauu.a.aaae com/Language-modell ..ddf.........
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http://jmlr. httg//aperr//211/200 074. html.
Senthooran Rajamanoharan, Arthur Conmy. Lewis Smith, Tom Lieberum. Vikrant Varma, Janos Kra- mar, Rohin Shah, and Neel Nanda. Improving sparse decomposition of language model activations with gated sparse autoencoders. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL htttps:://pperrweeee wwteiiiwwweeeeeeeeeee httt:/forumiiddiiii in2zv..
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autocncoders, 2024b. URL https: // ieteiiv.org//bbee 244.34435 14435.
Senthooran Rajamanoharan, Callum McDougall, and Lewis Smith. Removing high frequency latents from jumprelu saes, 2025. URL https://www alignmentforum org/ prrg/psss// High_ Biiggghhruuecc.....
AuxCAJJMuRt:tsss28//saa wtwp://iiiisaaappppppreeee -arpss---uuuaeee---attttttteeeee Saancaaninncttiiteeeeeeeeeerrrrs Latents frreents_fro____mmeeeee SAEs.
Alessandro Stolfo, Ben Peng Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: /ttp:///////ereiee.n./o forum?id=0ogT nmvDbe.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turnee, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monoseman- ticity: Extracting interpretable features from claude 3 sonnet. Transformer Circuits Thread, 2024. URL. o http://raaanfforrrriiii index. html.
pub/ 2024/2224/aa/ (0124/scaaine wttp:/aaaaammmmmmmmmmnii....mm. monosemanticityy
14

---

Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao. Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and Benchmarks Track, 2024.
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 61, 2010. URL https://scipy: org.
doi: 10. (000000/aajoaa22211222222
Zeyu Yun, Yubei Chen, Bruno Olshausen, and Yann LeCun. Transformer visualization via dictionary learning: contextualized embedding as linear superposition of transformer factors. In Eneko Agirre, Marianna Apidianaki, and Ivan Vulić, editors, Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 1-10, Online, June 2021. Association for Computational Linguistics. doi: wrg.//21..2212211 1tte:/2022.eeeloo11..1... deelio-1.1/.
10.18653/v1/20211 deelloo-111. URL
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wal- Fox, and R. Garnett, edi-
lach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, tors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: https://proceeeddng...eee hewpr/occeeiiingnnnurrp.p.....p neurips. cc/ httpppss.cpppp....eeeeee L165119988222220033ceer111111151 pdf.
Juexiao Zhang, Yubei Chen, Brian Cheung, and Bruno A Olshausen. Word embedding visualization via dictionary learning, 2021. URL pestps://arxic wwrrxiv..or/aae org/abs/11e a3301110..00333......
15

---

(a) Topk SAE on GPT-2 Layer 11
(b) Topk SAE on Gemma Layer 25
(c) JumpReLU SAE on Gemma L16
0.6
Densky
0.4
Densky
Dencky
0.5
Antipodality (4)
Antipodality
Antipodality
Figure 10: Additional Antipodality Plots. Antipodality scores vs. activation density for (a) TopK SAE on GPT-2 (Layer I1), (b) Topk SAE on Gemma 2B (Layer 25), and (c) JumpRel.U SAE on Gemma 2 2B (Layer 16). Across all configurations, dense latents tend to have high antipodality scores.
(a) Angles between Dense Subspaces
(b)
Angles between Sparse Subspaces
(c) Enc, vs. Dec. Pairwise Similarity
Median
Median
Principal
Principal Angle
Angle
!! wi
Layer Index
60
Indox
80
10
50
-0.5
20
0.5
sim(Wo) W!
Antipodality Score
24
16
Layer Index
Layer Index
Figure 11: Additional Analyses. (a) Median principal angles between dense-latent subspaces (den- sity>0.1) across layers. (b) Principal angles between randomly selected non-dense latent subspaces. (c) High antipodality score occurs when encoder and decoder weights are nearly opposite.
A Additional Results
A.1 Antipodal Pairing in Different SAEs
Figure 10, we report antipodality scores (computed in Eq. (1)) for dense latents three additional SAEs: two TopK SAEs that we trained on the residual streams of GPT-2 (layer 11) and Gemma 2 2B (layer 25), and a Jump SAE from Gemma Scope suite trained on an earlier layer (16). In all cases, we observe the same trend highlighted in Figure lc: high-density latents cluster at high antipodality scores, forming near-antipodal pairs that reconstruct specific directions in residual space.
A.2 Dense Latents During Training
In Figure 12, we visualize the number of dense latents (activation frequency 0.1) over training steps for each SAE configuration in our ablation experiment described in $3.1. All curves converge within the first 100kk steps and remain stable throughout training. This early plateau suggests that dense latents are not product of late-stage optimization noise, but rather emerge early and persist, indicating that they reflect consistent structure in the residual stream rather than transient artifacts.
Dense Latents during Training
of Dense Latents
20
Training Stop
Spanse
A.3 Angles Between Residual Stream Subspaces
Ablatton Denie Latants
In Figure 11, we provide further analysis of the evolution of dense latent subspaces across layers. Panel (a) shows median principal angle between the subspaces spanned by latents with density at each pair of layers in Gemma 2B. These results follow the trend observed in Figure 9c (based
Figure 12: Dense Latents Dur- ing Training. Dense latent counts stabilize early in training.
0.2 cutoff),
16

---

Absolute Cosine Similarity with Blas
Layer
0.8
10
0.2
0.001
Frequency
Figure 13: Plot of absolute cosine similarity of all SAE decoder vectors at all layers with that layer's decoder bias. We observe group of dense latents the upper right corner that have high frequency and align with the bias.
Distrib. of Latent Densities (LLaMA)
(a) Distrib, of Latent Densities (GPT-2)
16,-16384
Density Distribution
Donaty Diseribution
0.3
0.4
0.2
0.2
Original
Driginal
Abletion Sparrse Latants
Abration of Spama Latentx
Figure 14: Dense-subspace ablations on LLaMA-3.2-1B and GPT-2 Small. For each model's final layer we train baseline SAE (blue), retrain after ablating the subspace spanned by dense latents (teal), and retrain after ablating an equally sized subspace of non-dense latents (orange). Only removing the dense-latent subspace collapses the high-density tail.
revealing distinct subspace clusters in the early, middle, and late layers. However, the overall similarity between subspaces is lower here, reflecting the greater variability introduced by including moderately dense latents (density 0-12)...
For comparison, panel (b) reports the same metric computed subspaces spanned by 100 randomly selected non-dense latents per layer. As expected, these subspaces exhibit minimal overlap, with median principal angles near 90 across all layer pairs, confirming that the structure observed in the dense-latent subspaces nontrivial.
A.4 Pairwise Similarity Between Latents' Weights
In Figure 1lc, we report for each latent i, the maximum-magnitude cosine similarity of its encoder and decoder weights with any other latent j. In particular, show (sm(mm))))) der), where j = arg ( sim (w( ene, w ( and = dec: w:ll)))); We find that the antipodality score approaches only when both encoder and decoder similarities are close to -1
sim( (w ene, w(i) V and sim (() (K))),
dee>
arg max (e (I sim (i())
A.5 Similarity with SAE Bias
To investigate the relationship between dense latents and the SAEs' bias terms, we compute the cosine similarity between each SAE decoder vector and the corresponding layer's decoder bias. Figure 13 shows the absolute cosine similarity for all latents across layers as a function of activation frequency. We observe a small but distinct group of dense latents (upper-right region) that strongly align with the bias.
17

---

Sentence tracking
Paragraph tracking
Context: position tracking
Sentence
Spearmen correletion
0.5
Paragraph
Context position
-0.5
10
20
10
20
10
20
Layer
Layer
Layer
Figure 15: Position Latents. We identify position latents by computing their Spearman correlation
with relevant text boundaries. We classify a latent as belonging to certain category when |p] >> 0.4.
A.6 Additional Dense-latent Subspace Ablations
Figure 14 replicates the dense-subspace ablation from $3.1 two additional models: LLaMA-3.22 [AI @ Meta, 2024] and GPT-2 Small [Radford et al., 2019]. For each model we (i) train a baseline SAE, (ii) retrain after zero-ablating the subspace spanned by dense latents, and (iii) retrain after ablating an equally sized subspace randomly chosen non-dense latents. All experiments are run the final residual-stream layer. In both cases, removing the dense-latent subspace collapses the high-density tail-yielding almost no dense latents-wheet ablating a sparse subspace leaves the distribution essentially unchanged. These replications mirror the Gemma 2 2B result and further support that dense latents reflect an intrinsic residual-stream subspace rather than a training artifact.
B Experimental Details
For the experiment in $3.1, we trained TopK SAEs [Gao al., 2025] on the residual stream activations at layer 25 of Gemma 2 2B using billion tokens from the Open WebText corpus [Gokaslan and Cohen, 2019). Training followed the default configuration of the Sparsify library." 10 and experiment tracking was conducted using Weights & Biases." !I The ablation experiment on nullspace latents described $4.3 was performed on 10k-token subset of the C4 corpus [Raffel al., 2020]. Analyses throughout the paper were conducted using the Gemma Scope SAEs [Lieberum et al., 2024] with 16k latents trained on the residual stream Gemma 2 2B. All experiments were implemented PyTorch [Paszke al., 2019], with model inspection tools from the Transf ormerLens library [Nanda and Bloom, 2022]. Data processing used NumPy [Harris et al., 2020] and Pandas [Wes McKinney, 2010], and figures were generated with Plotly [Plotly Technologies Inc., 2015).
C Compute Resources Used
We expect the experiments for training SAEs, capturing SAE activations and generating completions with Gemma 2B to be able to be run in about 30 A6000 hours. The LLM judging experiments take less than USD $20 through OpenRouter with Gemini 2.5 Flash Preview [Gemini Team, 2025].
D Broader Impact
Our work focuses on interpreting language models, an important component of building safer and more reliable systems. SAES in particular are popular technique for understanding language models, and through investigating dense latents, we can both better inform SAE design, and better understand language model internals.
We do foresee any negative impacts of our work.
nhttps://wandb..
18

---

bel
you
Figure 16: Examples of position latents in layer 5. Deep blue represents positive projection along decoder vector, and deep red represents negative. (1) L5:4341 is a sentence-tracking latent, that lights up consistently on beginnings of sentences. It has strong activations for topic sentences too. (2) L5586800 a paragraph-tracken
latent, that lights up on beginnings of paragraphs. (3) L::8680) agnostic to artificially adding formatting newlines, showing it is encoding true paragraph position. (4) L5:697 is a context-posii ponnecttoositiottrcakiii latent.
Correlation with period
Sentence
0.4
Paragraph
0.2
Context position
-0.2
-0.5
0.5
Correlation with newline
Figure 17: Spearman correlation for period against Spearman correlation for newline.
E
Additional Taxonomy Results
E.1
Classification of dense latents
In our taxonomy, we identify dense latents using automated tests. We do not expect these tests to be perfect for variety of reasons-for instance, dense latents not lining up perfectly with the "true" feature basis due to learning a linear combination basis, and the fundamental difficulty of designing true, causal tests. However, for the purposes of illustration, we choose reasonable cutoffs for each test to create Figure 3, listed below.
Position latents: Spearman correlation of |pl > 0.4 for the relevant text boundary.
Context-binding latents: Fraction of successful flips > 0.75.
Nullspace latents: > 0.2 of encoder weight in bottom 10 Wu singular vector subspace.
Alphabet latents: Top 100 or bottom 100 logit contributions contain at least 90% of tokens starting with same character.
Mcaningful latents: AUC of using "is meaningful word" to predict "feature fires"
> 0.75.
* PCCaaligned latents: cosine similarity with top PC > 0.75.
Very few dense latents (3.6% across layers) fall category based on our automated tests to find them, with the most common clashes being between sentence- and paragraph- tracking (see Appendix E.2), and between several categories and meaningful-word latent. For the purposes of illustration, we break ties according to the priority (from highest to lowest): (context-tracking, sentence-tracking, alphabet, nullspace, context-binding, paragraph-tracking. meaning, PCA) based on our confidence in our automated tests.
19

---

<baesCaee A Quiterh Pro pcu di Merre Steem in 2018
Inpro=Consvmertn
Dowe & Busset: enteroeiment inc, 35 owner and opensto craiinng dndeemeeseeeeeee verves
Naneruu Velade
ampoureed linan noulta for suartor leiten ended on Nowenber
Masimbre Value Pngrap May 21, 221
mey/iahhunnnimmmm 17 persent 11411 million 5iic mill lento
Jessic Trudeau MPhin
Gomaraabe sore inooaeee percen is 67 seroent inorcose in years
Exccuiie Deesorr Momentous instaugion
Vigel
Momendul
querlertn
Alpeieel IDA non GAMP nasiinsine excessol pereet
Opaned ssore and In micettedd one more in the theitirs quaner 2015
Sey-0fWomerraaee Las. Often Sitain sinss nor lisoni Betlannn on Beetiee tand to putal
@0 Emaili
Korner, Health and Life Scinnoss Olvve Wamaa
focil and insaatationn on mavhs muchmam comotes than orignol thouthi nor unt
Sachip
ouily de drifeed
cileeetth amenss)
parsensinn
Pro forms net incomr ancoralleen
Bis
we 155 per share in the thid suarte Bhi4in
hee inciooe of $4.5 mill ont 12:1 periikieed stars compprted net lses of St
Insaee millaai 50.1 Bilied anart, companed
yeadto
ehieeeedde Oladil
be rerwauing tur armus jpadanne lor the tnadd the fimm Quamnety comp eu siore saks
Lopporeneeee qua pl drerrmann of courd you and ane plossel
Vyman Heeth Bodkass
the lleath Sclencon
rore 38 percent inclucie of estimneed negatie 110 bade po rebiiieeeen cabendar thitk and
Maman, whl the Womentou recooues jeskaa Trudiou, Exccuthes Direcar and Lauro Vagro
Pho, Direcoor of oTTeerppeate the Serdeae Vimerrbus retture is nonprrate organimition Dalas
desitc lipping endi/ge gai
prior yeur have now marked
quarze per
that provides innoaaivve educdiion and demerna health sencies overS ind
than previus two quamars to peocers and 16. percert respeciatly Stent cooni eißi resplinl an Hi Stmmeet
comparabbe trends of 170 parcent were als Nigher
namber lacng variaty Sisssonn achool has unique dustcamkuuuu coe that
seseesin atademic
Beloware mentondd maknowogg
Sanel gromooad which concloded the the quatter dlong wil our suboequer on Jostalll
Our strenghh durna
For thrteeepppoe more, ouc Mi Heath
comprttin speceinng Shyt the pams anrigeogrephe
Ennvanaion buarass tnarddeeeee hew
Ehnestasho
Sounreiwin, Google Pay Must Siiter and Sonth ham AuA Aana, Bae Wamer thathh Seakenshh
Into sntesstoinn By byght proficbele bave lood from Une orgaiin Dus ins slvll
soen socer and
growk sayt Stevel
Figure 18: L13: 15275 (blue) and 11449 (red), which has 81.5% correct flips. In these two examples, 15275 fires on children's 's mental health (left) and Dave & Buster's promotions (right), while 1 1449 fires on mentions of the podcast (left) and financial measures (right).
BCS ergument
Sareia Summet Timen Lockupts
Semi tough: Oiseevatione from the gool Freei
Surris Spice Tomel procos Creek Best Staved Lxul
Whonwor wae caid and sonw in the George locier nomm at haltiime yesterdayy Batte needs to
Fistay
gather that all togethe: grample Jilla
and anilii and buyy the
Trabs
aenes it ree, Bry. whar brodcem
My queatin from watsiing that game cant whather Besrge had play perfectty brsst an)
Sannis Tomom kili
ssuchdoan
Georg ga afterall warking
Georplis
quirter
Dup crath Tomeamoiin
1927
think bolle down three spoes the eheg
newer rourse, that haltime lest the any ohich hac foowd the weetre
sugersss would have leas buan edless That came thet and
1918 313 horal 93 12 228
1225 0cao 2017 333 123 1240
925
0250
1222
154
123 113
1123
1342
134
1925
155
1548
1943
127
154
1948
1228
194
Gaorge a secand aeras of the sescaed quatter (dbm coos twa resul of the Doweo worls
1900
1942
16-0
10:2
1553
pesiton Dhegameu to the goont bokaffiihil the recencienn ahe eececoo
2007
201
20)
cetrr to shorten frrthat comp Sath wher Grennaan's ceherse was domn, tebo
2008
-229
2329
7530
Show Cateon
Blayee sccod diocster lini Growel
tage Geords analed
spurs
PMit0h -in Btnd
Oijks
the)
0208 des 0pos
hal
peruddy and Davip woutent therr
02/10 200 0W04
"vilth 1m1
thewnning of of the the
quarto with Chegans al aady oltolland
Touchoows Tyrann Martle. This, of course was Georgla trmmediae reward fo Boboy plyy Gasn to special teans struges over the seaton Bunt to Kith ou with Buller standing on the
Solters the Daecel
ev was why proporaiing thgg with but with the way Daugee cefioee was plainng)
gnornng the lowr rask stratego of sowards Biaa crmmonsenti klaas Rassaal musete and to wore Goorghi Biding
l N 2009 CREB 4G301 Highern ponino +33056
Goorga cod yut could sense
7.55m CRED 4256- Nightripos 42602
Figure 19: L12: 14906 (blue) and 14599 (red), which has 76.5% correct flips. In these two examples, 14906 fires on descriptions of the game (left), and text or numbers related to sunrise (right), while 14599 fires on the teams and winning/losing (left), and years or locations (right).
E.2 Position latents
The observation in Figure 17 that period-tracking and newline- newline-tracking latents are hard to distinguish also relates to our discussion in $6 that because the sparsity incentive is low for these dense latents, they may not be perfectly aligned to "true" model dense features, and may instead be a linear combination of two related features.
E.3 Interpreting context-binding latents
We attempt to interpret mid-layer latents that exhibit coherent chunk-level activations in two ways:
1. All-context: Following existing autointerp methods [Paulo et al., 2024], we sample 10 activating and 10 non-activating phrases from an entire corpus and ask an LLM (Gemini 2.5 Flash) to generate an explanation. We repeat this 100 times to generate 100 explanations.
2. In-context: We instead sample 10 activating and 10 activatii phrases from the same context. We repeat this 100 times (using 100 different contexts) to generate 100 explanations.
When examples are drawn from the same context, the explanations are specific but highly diverse across contexts. When examples are drawn from different contexts, the explanations become vague or
20

---

5a
6-4
6.4
6.4
ABC
AJC
ABC
20
10
20
16
Layer
Layer
Figure 20: From left to right, we show the AUCs of predicting latent firing using function words (any of I' l'article', 'prepos', 'conjunction', 'det', 'modal', 'be', "do', 'have', 'what') }), articles, prepositions and conjunctions. These do not do as well as the rreaningful-word or "noun//rooperoou" groupings.
Category
Tags
pune
)
<< 12
quantifier
ABL, ABN, ABX, AP, AP$
article
AT
be
BE, BED, BEDZ, BEG, BEM, BEN, BER, BEZ
conjunction
CC,
num
CD, DD
do
DO, DOD, DOZ
det
DT, DTI, DTS, DTX, DT$
have
HV, HVD, HVG, HVN, HVZ
prepos
IN,
adj
JJ, JJR, JJS, JJT
modal
MD
NN, NN$, NNS, NNS$, NR, NRS, NR$, UH
propernoun pronoun
NP$, NPS, NPS$
PN, PNS, PP$, PP$$, PPL, PPLS, PPO, PPS, PPSS
qual
QL, QLP
adv
RB, RB$, RER, RET, RN, RP
verb
VB, VBG, VEN, VBZ
WDT, WP$, WPO, WPS, WQL, WRB, EX NIL
unknown
Table 3: Mapping from high-level category to Penn Treebank tags. A trailing $ marks possessive forms.
generic (Table 4). This drop in specificity across contexts is somewhat expected, since explanations for any latent may overfit the context. However, doing the same for sparse latents (Table 5), that a "good" sparse latent would have similar explanations with both in-context and all-context examples, aligning with the usual assumption that SAEs learn directions that represent concept in the model.
It is difficult rule out the possibility that these dense latents represent uninterpretable abstract feature the model learns. However, the steering experiment seems to cause the relevant specific concepts be brought during generation, supporting the "binding" hypothesis that there are dense directions that do not represent a fixed concept but rather are used in the model's computation.
E.4 Additional examples of context-binding latents
We include two additional examples of context- bindxxt- latent pairs with high flip score: layer 13 pair (15275, 11449) (Figure 18) and layer 12 pair (14906, 14599) (Figure 19). For each pair, we show two example contexts where they are active, illustrating how each latent activates on specific but context-dependent concepts, and that latents in a pair do not co-activate.
E.5 Steering context-binding latents
Our methodology for steering is as follows:
Prompt Gemma 2 with input text from the RedPajama dataset, ending at natural point (after newline token), with at least 400 tokens.
21

---

Absollute cosine similanty with PC1
Fractioon norn top PCS
0.6
0.6
0.4
02
Layer
Layer
Figure 21: (Left) Cosine similarity of dense latents with top principal component. (Right) Fraction norm of dense latents in top 5 principal components.
Absolune Cosine Simllrrity with PC1
Absolune Cosine Simliarity with PC1
0.8
0.6
LO
Dictionary Size
Figure 22: Cosine similarity of dense latents in layer with the top principal component, across different LOs and SAE dictionary sizes.
2. Capture the activating phrases of F1 and F2 that least 5 consecutive tokens long.
3. Allow Gemma 2 2B to generate completion without steering, and prompt LLM (Gemini 2.5 Flash Preview) to judge whether the completion more like F1 activating examples, F2 activating examples, unelear.
4. Repeat the above, but steering the last token during generation, in the direction of Fl and F2. Since FI and F2 are antipodal pairs, we first ablate the subspace spanned by F1 and F2, before adding the steering vector, that fixed 2x the historical activation of that feature in that context.
E.6 Meaningful- Word Latents
We provide the mapping from higher-level categories Penn Treebank tags in Table 3. addition to the AUC for the categories shown in the text, we show the AUC for few other categories in Figure 20, which seem mostly unrelated to predicting the activations of dense latents.
E.7
PCA latents
Figure 21 reports, layer-by-laye each dense latent's cosine similarity to PC1 (left) and the share of norm contained the top-5 PCs (right). As noted, only one antipodal pair per layer strongly aligns with PCI; most dense latents place little mass in the top PCs. Figure 22 shows that this pattern stable under SAE hyperparameters: varying the sparsity target (Tap) or dictionary size yields similar PCI alignments, neither removing proliferating PC-aligned latents.
22

---

Dense latent L12:7541 (f 0.421)
In-context
All-context
Numerical values calegories related real estate listings. such prices, property types, number of beds
proper noun or specific conceptientiyy within sentence
Mentions of people, groups, entities involved in a discus- sion or action
The beginning of a new clause or phrase, often following a comma, paremhesiis or other panctuation, or indicating a shift in topic or frouss
Mentions of Donna Brazile or political news commen- tary, often criticul of establishment figures media outlets
sequence words that are part of proper noun, title, specific
phrase, often capitalized, short phrase that acts as label or identifier within a larger text.
References to the Niagara Wire publication or its staff and content
The heginning of a new sentence or clause, often following punctuation or a line break
Terms and phrases related to the Cybercrime Prevention Act of 2012 in the Philippines
The detection of numbers, dates, or specific numerical references within text
References to Barack Obama or his administration
The beginning of a new clause or phrase, often following conjunction. preposition, or punctuation mark, that introduces additional information or a new element into the sentence structure
Mentions of specific people. places, things, or references to particular time or event
A proper noun common noun that is of larger phrase, often appearing after a preposition or as an object of a verb
Mentions CSIRO's intemall operations, expertise, and collaborative efforts
Mentions specific entities, objects, or concepts within a broader con- text, often highlighting a particular detail or aspect of the surounddng text
Nautical vessel chartering and related services
The letter 'I' or "A' or 'O' when it is the first letter of a word or standalone word
Mentions healthcare organizations, roles, or initiatives related to health and human services
or sequence tokens part of larger, multi word proper compound noun, specific phrase, where the preceding context helps to complete the meaning of the highlighted part.
Dense latent L12:2009 (f = 0.319)
In-context
All-context
Punctuation marks, numbers, or single letiers that are not part of a larger word
Cude, symbols, or foreign language phrases
Mentions of events, dates, outcomes related the life of Stéphanie of Monaco
The continuation of word phrase across line break other format- ting boundary
Mentions of the St. John's Red Storm basketball team, their coach Mike Jarvis, or their player Hatten
Short, common words or symbols that are often part of larger phrase or structure, but do not carry significant meaning on their own
Reporting on COVID 19 cases related news
The beginning a new sentence or clause, often following a period, comma,, or other punctuation, line break
Scientific citation formatting and punctuation
Punctuation marks, prepositions, and conjunctions that connect different parts of a sentence or list
heginning new clause or sentence
The beginning of new word or token that not proceded by a space
References students or pupils in educational context
The detection of a word or phrase that part of larger, well-known entity common expression, where the detected part is not the beginning of the entity or expression
Biographical details and life events character, includ- ing family, career, and personal status, often presented chronological or list-like format
Mentions of specific people, places, or entities, or phrases that introduce or refer them
Specific references to the current or past UN General As- sembly session, or to the UN Secretary General and his staff
The beginning of a new sentence or phrase, often following punctuation or a line break, or the start of new section within a document
years, or numbers historical Mentions of specific years, official context
Mentions of specific words or phrases that part larger list enumeration, oflen found in titles, bullet points, structured content
Table 4: Sampled explanations of dense latents L12:7541 and L12:2009, using in-context examples versus all-context examples. The in-context explanations are highly specific and diverse, while the all-context explanations are vague.
23

---

Mentions of the PProduct of the Year" specific year Sparse latent L12:10356 (f 8.90 x 10 10-4)
In-context
All-context
Years in the 2010s or 2020s, often following movie title and sometimes preceded by "HBO" or "HBO Max"
A two-digit year following month or day, or part of a date range
The number four-digit year, specifically in decade
The last two digits of four-digit year
The year 2019 dates as standalone year
The last two digits of four-digit year
The digit 'I" when it is part of a four-digit year that starts '20" followed by a two-digit number. typically representing day time, forward slash.
The second digit of year in the 21st century
The first digit of a two-digit year in citation The first two digits of a four digit year The release year movie title
The last two digits of four-digit year
The last two digits of a year in the 21st century
The last two digits of a four dir-diiitt year
by
two digits a year
The year 2022 date formats
A four-digit in the 21st century. spocifically between 2010 and 2024
The third digit of a four-digit year, specifically when the year is 2013, often found in movie titles or release dates
The last two digits of a four-digit year, typically in the 2000s
Sparse latent L12:800 (f 7.30 x 10
In-context
All-context
Mentions of the South Ossetian conflict, including locations. people, and related events
Mentions of people's 's names titles, often followed by their statements or actions
References to specific person named Sarah, including pos- sessive forms and direct address
Mentions person entity speaking or being referenced
Mentions of reports, unalyses, or statements made by indi- viduals or groups
A proper noun prenoun that the subject of sentence or clause, proper noun that is the object of a verb or preposition
Mentions of individunls or organizations involved in mine clearance humanitarian aid
Mentions people or organizations, often in attribution or as subjects of actions
Mentions of the author Tom Bissell, often relation to his woek or statements
Mentions proper nouns, often names people organizations, that are split across a line break
Mentions of "Dicko" as a proper noun, often followed by or punctuation, indicating clause related the person.
Mentions people speaking or being quoted
Mentions of Alan Waller, Farl Spencer's 's former head of security
Proper nouns pronouns referring to people or organizations, often followed by a verb
Mentions of people or organizations as subjects or possessive entities
Mentions person's 's name followed by of speaking or a refer- ence that
Attributions of statements or opinions to individuals or groups, often experts, in news or analyticul contexts
A proper noun or pronoun that is the subject of sentence or clause
Mentions Sonny Dykes, football coach, last name. often the context of his statements actions
Mentions people or organizations subjects or agents actions
Table 5: Sampled explanations of sparse latents L12:10356 and L2:800, using in-context and all- context examples. While the in-context explanations still tend to be more specific, they still center around a similar theme as the all-context explanations, and it is plausible that L12: 10:16 a "date" feature and L12:800 is a "proper noun" feature.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we claim that examine the geometry of high frequency latents, laxonomize high frequency latents, and examine their distribution across layers, all of which we do in the paper (we link these specific sections in the introduction).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have an extensive limitations section the discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include experimental details sufficient to reproduce the experiments in the main body (all of which use open source models). We provide additional experiment details in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We plan to upload our code by the supplementary materials deadline.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
wttpsuummssssooooooooooooooooooe conooicy))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /niiss... wwcpppllccgguuioeooccooooocoocoe
wwwsssoliogyyyo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Justification:
As described in the experimental details section, we include experimental details sufficient to reproduce the experiments in the main body (all of which use open source models). We provide additional experiment details Appendix B.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Because we are providing taxonomy of different SAE latents, statistical significance in most of our experiments does not make sense.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix for discussion of the compute used for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics pestps://neuriie wttpeeripp....pulliiieeeeeeeeeee
Answer:
[Yes]
Justification:
The research complies in every respect with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix for discussion of our paper's broader impact, which we overall believe to be very positive.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release any new data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use and cite the open source Gemma models and GemmaScope SAEs.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We do introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not do research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do do research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
30

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttps//////uu...eeeeee www.arips.c.oooofeeeeeeeeeeeeeee /00/////eeeneee222/////// 20LL/LLM for what should or should be described.
31


Reviews:
[{"review_id": "f7clFR67Hl", "reviewer": "Reviewer_Vwvz", "summary": "This paper investigates the nature of dense latents in sparse autoencoders (SAEs) trained on language model activations. An SAE latent is considered ‘dense’ when firing for at least 0.1 of the inputs. While SAEs are designed to yield sparsely activating, interpretable features, a significant fraction of latents are found to activate on a large number of tokens. The authors challenge the view that these dense latents are undesirable artifacts, instead demonstrating that they often reflect meaningful directions in the residual stream. This paper finds that many of these dense latents capture interpretable information and often fall into one of the following categories: position latents, context-binding latents, (quasi) nullspace latents, alphabet latents, meaningful-word latents, and PCA-latents.  The paper studies how this taxonomy of dense latents changes across model layers. The authors demonstrate that a given latent is in a particular category through a variety of ablation, steering, and quantitative analyses.", "questions": "1. How do the types of features recovered by sparse latents compare to those recovered by dense latents? Do they represent different aspects of model computation?\n\n\n2. What fraction of dense latents do not fall into your taxonomy? Are there properties that reliably predict whether a dense latent is interpretable?\n\n3. In Section 3.1, how exactly is the dense subspace identified and ablated? How large is it, and what is the impact of its ablation on downstream SAE behavior?\n\n4. Could the antipodal structure of dense latents be an emergent by-product of L2 reconstruction loss rather than an interpretable representation structure?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\n- Novel insight into SAE behavior: The paper tackles an underexplored phenomenon—the prevalence and utility of dense latents in SAEs—and provides evidence that these are not just training noise, but reflect intrinsic model structure.\n\n- Careful empirical analysis: The authors use ablations, cosine similarity metrics, and probing techniques (e.g., entropy, steering, AUC prediction from part-of-speech) to characterize dense latents.\n\n- Antipodal structure: The discovery that dense latents form antipodal pairs and correspond to specific directions in the residual stream is a particularly compelling geometric observation.\n\n- Layer-wise functional taxonomy: The paper organizes dense latents into meaningful classes and analyzes their distribution across model layers, revealing a shift from structural to semantic to output-oriented features.\n\n$\\textbf{Weaknesses}$\n\n- Lack of geometric contextualization: While the paper makes a strong case that dense latents are not artifacts, it could further contextualize this by considering known geometric properties of LLM representations (e.g., outlier dimensions, early-to-late layer transformations, or anisotropy in the residual stream). \n\n- Sparse vs. dense comparison is missing: The paper focuses heavily on dense latents but does not compare them to sparse latents. Are the features learned by sparse latents of a different nature, or do they partially overlap in function? This comparison would provide crucial context.\n\n- Taxonomy completeness unclear: The paper acknowledges that fewer than half of dense latents fall into its proposed taxonomy. Quantifying this more precisely and identifying predictive signatures of interpretability would strengthen the claims.\n\n- Ambiguous claims: Some empirical observations (e.g., context-binding latents activating on \"specific but varying concepts\") are not clearly explained and may need clarification or better examples.", "was_revised": true, "final_justification": "I believe this paper makes a clear contribution and is worthy of publication. I will keep my score of a 5.", "rebuttal": "Thank you for highlighting our “careful empirical analysis” and recognizing the novelty of our study. Your positive feedback is appreciated. We address each of your points below.\n\n> **W1:** While the paper makes a strong case that dense latents are not artifacts, it could further contextualize this by considering known geometric properties of LLM representations\n\nWe appreciate this suggestion. We indeed observed a connection between the nullspace-aligned latents and outlier features [1]: the high norm that characterizes outlier features often lies in the unembedding quasi-nullspace, the same subspace tracked by nullspace-aligned latents. We will elaborate on this connection in the final version of the paper.\n\n> **W2:** Are the features learned by sparse latents of a different nature, or do they partially overlap in function? & **Q1:** How do the types of features recovered by sparse latents compare to those recovered by dense latents?\n\nIn general, dense latents appear to represent more “functional” or “structural” information (e.g., token position, lexical confidence) rather than semantic content (see Section 5.1 in Chen et al. [2] for a discussion of semantic and structural features). This is consistent with the intuition that semantically interpretable features typically appear sparsely across a text (e.g., a feature representing the concept of “dog” is expected to be active well below 10% of the tokens of a large corpus), whereas functional information related to confidence or token position is expected to be present more densely across tokens.\n\n> **W3:** Taxonomy completeness unclear & **Q2:** What fraction of dense latents do not fall into your taxonomy? Are there properties that reliably predict whether a dense latent is interpretable?\n\nOur classification into six latent types is intended as evidence for the functional roles that dense latents can play. Coverage varies by layer (from 14% to 81% of dense latents per layer) amounting to an overall coverage of roughly 46%. We did not identify simple metrics that predict whether a dense latent is interpretable; interpretation generally requires specific geometric or activation pattern tests, as described in our methodology.\n\n> **W4:** Some empirical observations (e.g., context-binding latents activating on \"specific but varying concepts\") are not clearly explained and may need clarification or better examples.\n\nThank you for pointing this out. We will clarify that context-binding latents tend to activate on coherent concepts within the same input, but these concepts vary across inputs. For example, in Figure 3, we see that #7541 activates on “casino facts” in one context, but “healthcare” in another. Furthermore, these context-binding latents come in pairs, and since pairs are antipodal, each tends to activate on a distinct concept from its pair (e.g. #7541 activating on “casino facts”, and its pair #2009 activating on “looking for a buyer”, two concepts that are both present in the context but distinct). \n\nTraditional autointerpretation of SAE latents [3] uses activating examples from the entire corpus (across-context), assuming a specific consistent concept for each latent. We suggest also autointerpreting SAE latents using activating examples within-context. We have since conducted more autointerp experiments (adding on to Appendix E.5), so we will provide more examples in the final version. For many sparse features, if explanations are specific within-context, the explanations are similarly specific with across-context examples (e.g. a feature always for “governmental bodies”). For context-binding latents, the explanations are specific within-context (e.g. “creation, sharing, or replication of digital content”), but vague across-context (e.g. “specific noun phrases, including proper nouns and descriptive phrases”). This is what we mean by “specific but varying”, and we have since quantified this claim further by using an LLM judge to judge specificity. We thank the reviewer for prompting this clarification.\n\n> **Q3:** In Section 3.1, how exactly is the dense subspace identified and ablated? How large is it, and what is the impact of its ablation on downstream SAE behavior?\n\nWe identify the dense subspace by collecting the decoder weight vectors of the dense latents from a first SAE. To ablate this subspace from the language model’s residual stream, we remove the component of the residual that lies within it. Formally, let $\\mathbf{W} \\in \\mathbb{R}^{k \\times d}$ be the matrix of these weights and $\\mathbf{x} \\in \\mathbb{R}^d$ the residual stream. We compute the projection $\\mathbf{p} = \\mathbf{x} \\mathbf{W}^* \\mathbf{W}$ (where $ \\mathbf{W}^*$ is the pseudoinverse of $ \\mathbf{W}$) and remove it: $\\mathbf{x}’ = \\mathbf{x} - \\mathbf{p}$. This yields the ablated residual stream $\\mathbf{x}’$. For comparison, we apply the same ablation procedure using a random subset of sparse latents (ensuring dimensionality matches the dense subspace). In the experiments of Section 3.1, the dimensionality of the ablated dense subspace is 46 in the SAE with 16k latents, and 10 in the SAE with 32k latents. We did not directly quantify the impact of this ablation on downstream model performance, but expect that ablating a persistent, information-carrying subspace would substantially degrade model behavior. We will add an appendix section to clarify this procedure in the final version of the paper. \n\n> **Q4**: Could the antipodal structure of dense latents be an emergent by-product of L2 reconstruction loss rather than an interpretable representation structure?\n\nThis is a good observation. We hypothesize that antipodality emerges from the combination of dense signals and the one-sided nature of ReLU activations. Representing a dense signal with a single ReLU unit requires learning a strong negative bias, which the SAE may fail to achieve during training. As a result, the SAE learns a second latent to encode the missing component, forming an antipodal pair.\nTo test this, we trained an Absolute TopK SAE by removing the ReLU allowing activations to be both negative and positive, and enforcing sparsity by taking the TopK most extreme activations for reconstruction. This removes the one-sidedness and allows a single latent to capture both sides of a dense signal. As expected, antipodal structure disappears under this activation. We will include this new experiment and explanation in the final version of the paper.\nWe thank the reviewer for their detailed and constructive feedback. We hope that our responses have addressed their concerns and provided sufficient clarification.\n\n—\n\n[1] Timkey, William, and Marten Van Schijndel. \"All bark and no bite: Rogue dimensions in transformer language models obscure representational quality.\" (2021).\n\n[2] Chen, Alan, et al. \"Transferring Features Across Language Models With Model Stitching.\" (2025).\n\n[3] Gonçalo Paulo, Alex Mallen, Caden Juang, and Nora Belrose. \"Automatically Interpreting Millions of Features in Large Language Models.\" (2024)."}, {"review_id": "wkI5dcIo0j", "reviewer": "Reviewer_Gno7", "summary": "The authors study dense latents learned by SAEs trained to reconstruct LLM residual stream representations (with a focus on SAEs pre-trained on Gemma Scope). The authors identify a subset of these latents and classify them to distinct functions. These latents seem to be very scattered in functionality.", "questions": "* The antipodality result is interesting, but unfortunately not really connected to the subsequent results of the paper (except a little bit for the context-binding latents). Why is antipodality relevant in the context of density? Why do we see an antipodal pair of features that track 'casino facts' vs 'looking for a buyer'. These concepts do not seem related in any way, except that they both feature in that particular context?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* The paper contains some novel analyses and findings regarding the statistics and prevalence of dense SAE features.\n* Analyzing dense SAE features is important. \n* Classification of dense features is performed, with some novel functionality proposed.\n* The idea of dense latents that can bind to relevant context sounds interesting (but unfortunately under-explored).  \n\nWeaknesses:\n* The paper contains a large list of analyses that are not well connected or seem to bear much relevance to each other. What is the relevance of the antipodality? How are these six different types of dense features related to each other? Some features, like entropy regulating and positional features have been discovered before, as the authors note in the related works section.\n* The related works section is extremely brief. The authors mostly just cite work without discussing it.\n* For some of the latents (like the concept binding one), it's unclear if this is an actually meaningful latent or an artifact. It's a potentially very interesting functionality - I feel like the authors could have dedicated more space and time to analyze this type of dense latent rather than giving a large list of latents, some of which have been covered before in the literature\n\nIn the end, the paper is too unfocused in it's current form. It's not clear if the goal of the paper is to classify as many of these dense features as possible, or to argue that dense features are important/formulate a clear theory about their existence. I think the second strategy is very interesting and I would be willing to raise my score if the authors can expound upon this. The analyses conducted wrt to the concept binding latent is a good start, although it is not clear to me if these bind to distinct types of concepts depending on the context (like latents shown in Fig 3), or if they also specialize on a single concept, but activate densely around it.\n\nOne could potentially write a whole paper about dense features like this (if one can indeed show that they have systematic activity patterns, which isn't really that clear from the brief treatment they get in this paper).", "was_revised": true, "final_justification": "Authors addressed my concerns about the framing of the paper.", "rebuttal": "Thank you for acknowledging the novelty of our analyses and the importance of studying dense SAE features.\n\n> It's not clear if the goal of the paper is to classify as many of these dense features as possible, or to argue that dense features are important/formulate a clear theory about their existence.\n\nOur primary goal is to demonstrate that dense SAE latents are not artifacts of SAEs but instead represent meaningful internal model signals. To support this, we (1) show that these latents do not appear when a particular residual-stream subspace is ablated, and (2) provide functional interpretations for a wide range of them. The classification into six dense-latent types is not the central contribution, but serves as evidence that many dense latents correspond to genuine (and interpretable) dense signals in the language model, rather than SAE artifacts (e.g., aggregated sparse features or optimization noise). This directly challenges the previously hypothesized view of dense latents as optimization byproducts.\n\nThat said, we agree with the reviewer that some latent types, such as concept-binding features, deserve deeper analysis. In response, we have conducted new experiments to further explore these, which we describe below.\n\n> [clarifications about context-binding latents] **W3:** “if this is an actually meaningful latent or an artifact.” & “it is not clear to me if these bind to distinct types of concepts depending on the context (like latents shown in Fig 3), or if they also specialize on a single concept, but activate densely around it.” & **Q1:** “Why do we see an antipodal pair of features that track 'casino facts' vs 'looking for a buyer'. These concepts do not seem related in any way, except that they both feature in that particular context?”\n\nOur hypothesis about context-binding latents is that the meaning they encode is context-dependent, rather than a globally monosemantic direction.\n\nTraditionally, SAE features are assumed to activate on a consistent concept (e.g., “Golden Gate bridge”) across all contexts. We found that context-binding latents activate on consistent concepts only *within each context*, but the specific concept they represent *varies across contexts*. For instance, Figure 3 shows #7541 activating on “casino facts” in one document, and “healthcare” in another. This alone does not rule out the possibility that #7541 represents a generic “noun phrase” feature, for example. However, our experiments in Section 4.2 on steering then show that if we amplify such a feature, it influences the generation towards that specific concept, thus it less likely merely represents a generic “noun phrase” feature, but actually contains information about that specific concept. \n\nFor an antipodal pair, the two features never co-activate, thus they tend to represent two *distinct* ideas/themes in the passage, for instance “casino facts” and “looking for a buyer”.\n\nTo further validate our claim of “context-binding latents activate on specific but varying concepts”, we ran additional auto-interpretability experiments (adding onto Appendix E.5), and will include more examples in the paper, of within-context versus across-context explanations of the latent. Furthermore, if we ask an LLM judge to rate the specificity of explanations, dense latents have within-context explanations that are much more specific than across-context explanations—in contrast with interpretable sparse latents which have similar specificity within-context and across-context.  We will include these additional results in the final version, and thank the reviewer for bringing up the clarification on this claim.\n\n> **W1:** What is the relevance of the antipodality? & **Q1:** Why is antipodality relevant in the context of density?\n\nAntipodality is not the central phenomenon we seek to explain but rather a structural byproduct of how SAEs encode dense residual-stream signals. We hypothesize that antipodal pairs emerge due to the one-sided nature of ReLU activations: a single ReLU unit can capture a feature only if it also learns an appropriate bias value that shifts the activation range to match the full span of the signal. When this signal is dense (i.e., different from 0 most of the time, unlike sparse features) this offset is hard to fit, so the SAE learns a second latent to “complete” the signal, yielding a latent pair whose encoder/decoder weights lie on opposite sides of the same line (i.e., antipodal).\n\nTo test this hypothesis, we trained an “Absolute TopK” SAE, where we remove the ReLU to allow the activations of each latent to be both negative and positive. We enforce sparsity by using the TopK most extreme activations for reconstruction. This removes the one-sided constraint and should allow a single latent to encode the entire dense signal. The results confirm our hypothesis: the antipodal structure disappears under this activation. Furthermore, compared to the corresponding TopK SAE, the number of dense latents is smaller and the density of dense latents is higher, as antipodal pairs map roughly to single latents. We will include these results, along with the above explanation, in the final version. We thank the reviewer for prompting this clarification and follow-up experiment.\n\n> **W1:** “Some features, like entropy regulating and positional features have been discovered before, as the authors note in the related works section.”\n\nWe would like to clarify that only one of the prior works mentioned in our “Dense Language Model Representations” section actually studies SAE latents: Chughtai and Lau [48] identify positional signals in an SAE trained at GPT-2’s layer 0, but do not analyze their density. The other two works (Gurnee et al. [20] and Stolfo et al. [13]) focus on individual *neurons* in the model and not on SAE features. Stolfo et al., for example, study a confidence-regulation mechanism associated with the unembedding nullspace, but do not relate it to SAEs. We have clarified these distinctions in our revised related work section.\n\n> **W2:** The related works section is extremely brief. The authors mostly just cite work without discussing it.\nWe agree and have significantly expanded the related work section in response. The new version provides a more thorough discussion of the previous studies that are most relevant to our work. The revised section is provided below (with newly added portions highlighted in bold).\n\n—————————————————————————————————————————————\n\nInterpreting SAE Latents.\n\nAs SAEs have gained traction, recent work has focused on interpreting the meaning of their latent features [28,44]. Building on the neuron interpretation methodology introduced by [45], several recent works aim to interpret SAE latents systematically. **Templeton et al. [6] propose a rubric-based evaluation method in which a language model (Claude 3 Opus) scores how well a proposed feature description aligns with the contexts on which the latent activates. Similarly, Paulo et al. [46] propose a pipeline in which natural language interpretation for SAE latents are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non-activating contexts. Other recent efforts explore automated interpretation approaches based on self-interpretation strategies [47].** A recurring observation across multiple studies is the emergence of a substantial number of dense latents, which activate on more than 10% or even 50% of tokens [8,9]. **Chen and Batson [10] take the 10 most densely activating latents in a cross-layer Transcoder trained on Claude and attempt to manually interpret them, finding plausible interpretations (e.g., \"activates on commas\", \"activates on non-terminal tokens in multi-token words\") for 6 of the 10 features. In contrast, Rajamanoharan et al. [11] view dense latents as an undesired phenomenon and propose a frequency-based regularizer to discourage their emergence during training.** Whether these latents reflect meaningful internal computations or arise as undesirable artifacts remains an open question. Our work addresses this question directly, showing that dense latents are persistent, characterized by a specific geometry, and often track interpretable, functional signals in the model.\n\nDense Language Model Representations.\n\n**Prior work has also identified dense signals in language model representations more broadly (i.e., components that encode information consistently across many tokens). Gurnee et al. [1] present a taxonomy of universal neurons that appear across GPT-2 models trained with different seeds. Among these, they identify neurons that encode positional information at each input token. Chughtai and Lau [3] similarly identify dense positional features in an SAE trained on GPT-2’s layer 0, though they do not explicitly analyze their activation density. Finally, Stolfo et al. [2] describe neurons that regulate model confidence by tracking entropy and connect them to a component of the residual stream aligned with the quasi-nullspace of the unembedding matrix.**\n\n—————————————————————————————————————————————\n\nIn conclusion, we thank the reviewer for their constructive feedback. Their comments prompted several clarifications (regarding the role of context-binding latents and the broader framing of our contributions), as well as additional experiments on antipodality and context-binding features. We believe these additions substantially strengthen the paper. We hope these improvements will encourage the reviewer to reconsider their evaluation."}, {"review_id": "gZTDNkqH7G", "reviewer": "Reviewer_bgXx", "summary": "The paper studies frequently activated latent features (**dense latents**) of SAEs trained on residual stream tokens of language models. The paper defines dense latents as latents active in more than 10% of tokens.\n\nThe paper focuses on JumpReLU SAEs trained on Gemma-2-2B layers and includes some experiments with TopK SAEs and GPT-2 small.\n\nThe paper finds the following:\n- Experiments that ablate the subspace of the residual stream spanned by vectors decoded from dense latents suggest that dense latents are not just results of biases of SAE training.\n- For a great proportion of dense latents there is exactly one another latent with encoder cosine similarity close to -1. These are called antipodal pairs. It seems that each pair represents a single direction, and this is not specific to the JumpReLU SAE.\n- The role of a substantial proportion of dense latents can be understood to some degree (position tracking, context binding, entropy modulating, letters, part-of-speech-like categories, principal component).\n\n---\n\n**In more detail.** The analysis identifies different roles for 10%-80% of dense latents depending on the LM layer. The roles are categorized as:\n1. Position tracking latents: the value depends on the distance from start of sentence, paragraph (in early LM layers) or context (in almost all LM layers).\n2. Context-binding latents: approximately consistent meaning within context, but not between contexts. Present in layers in the middle.\n3. Nullspace latents: features in the final layers that have a very small effect on the unembedding output, but contribute substantially to the residual stream's norm. Harder to interpret. Some of them modulate entropy by scaling through RMSNorm/LayerNorm [27].\n4. Alphabet latents: high-density latents in the last layer that promote tokens that share the initial letter.\n5. Meaningful-word latents: latents that are active on tokens that belong to some higher-level part-of-speech based categories.\n6. PCA latents: have high cosine similarity with the top principal component of the residual stream.", "questions": "**Questions:**\n1. Please consider the questions under weaknesses, point 1.\n2. Do you know whether the presence of (almost) antipodal pairs is a consequence of the frequency of the latent (more frequent features need to be reconstructed more accurately)? Does reducing the dictionary size reduce the number of antipodal pairs?\n3. Do you know how much different types of dense latents overlap?\n\nI apologize for any errors on my part.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n- To my understanding, this paper is a meaningful step in language model interpretability research. It improves the understanding of a substantial proportion of frequently activated SAE latents of language transformer models and gives convincing evidence against the hypothesis that most dense latents are mere artifacts of SAE learning algorithms.\n- As far as I can tell, much of the ideas of what to look for and the methodology are original.\n- The writing is mostly clear and the experimental results are presented well. (I especially appreciate Figure 2.)\n- The source code for reproducing the experiments in the main paper is provided. (I am not sure about the appendix.)\n\n**Weaknesses**\n\n1. Clarity of methodology and presentation could be improved (most important).\n\t1. Are the inputs of SAEs the features after adding the MLPs output to the residual stream?  \n\t2. Is the subspace ablation based on encoder or decoder vectors?  \n\t3. The definition of \"antipodality score\" lacks justification (the paper mentions its drawbacks), but I think that it accomplishes its goal.\n\t4. Relatedly, in Figure 11 I find it unclear how the indices $i, j$ are chosen and whether the antipodality score (color) correspond to $i$ or $j$?\n\t5. Figure 1. c: Why is encoder cosine similarity chosen rather than decoder cosine similarity?  \n2. The discussion of related work (background information) could be improved.\n\t1. Prior work on antipodal pairing [1] and dense latents (\"high-frequency latents\") [2] should be discussed in relation to the present work.\n\t2. While the review of recent work seems sufficiently broad (as far as I can tell), it lacks some depth and context. (This is probably due to a lack of space.)\n3. Clarity of some claims could be improved.  \n\t1. L36 \"antipodal pairs, effectively reconstructing specific subspace directions.\" Perhaps something like \"antipodal pairs, each effectively encoding a single direction\" would be more clear.  \n\t2. L45: \"inherently dense mechanistic functions\" might not be clear before reading section 3.1.  \n\t3. What is the meaning of \"a linear combination of the “true” basis works for reconstruction too\"?\n\nMinor suggestions:\n- Clarity and errors:\n\t- L78: \"many pairs\" could be expressed more precisely.\n\t- Figure 11 a: Consider showing the 10% threshold.\n\t- Figure 5 b: The boxplots are a bit hard to recognize. What are the dotted lines?\n\t- L62: typo: \". and\" -> \", and\"\n\t- The context of Figure 16 seems to be missing. What are the run lengths of? What is the parameter of the Bernoulli distribution?\n\t- Consider including an overview of hyperparameters, models and datasets.\n- Questions:\n\t- Would it be interesting to plot distributions of latents (like Figure 11) depending on the frequency?", "was_revised": true, "final_justification": "The discussion with the authors, which includes clarifications and interesting new empirical information increases my confidence in my positive assessment of the paper.", "rebuttal": "Thank you for acknowledging the originality of the work, and its significance towards understanding of SAEs. Your positive feedback is appreciated. We have addressed each of your questions below.\n\n> **W2:** The discussion of related work (background information) could be improved.\n\nWe agree and have significantly expanded the related work section in response. The new version provides a more thorough discussion of the previous studies that are most relevant to our work. The revised section is provided below (with newly added portions highlighted in bold).\n\n—————————————————————————————————————————————\n\nInterpreting SAE Latents.\n\nAs SAEs have gained traction, recent work has focused on interpreting the meaning of their latent features [28,44]. Building on the neuron interpretation methodology introduced by [45], several recent works aim to interpret SAE latents systematically. **Templeton et al. [6] propose a rubric-based evaluation method in which a language model (Claude 3 Opus) scores how well a proposed feature description aligns with the contexts on which the latent activates. Similarly, Paulo et al. [46] propose a pipeline in which natural language interpretation for SAE latents are matched with different contexts and used by an LLM in different tasks that evaluate how good the interpretations are in predicting activating and non-activating contexts. Other recent efforts explore automated interpretation approaches based on self-interpretation strategies [47].** A recurring observation across multiple studies is the emergence of a substantial number of dense latents, which activate on more than 10% or even 50% of tokens [8,9]. **Chen and Batson [10] take the 10 most densely activating latents in a cross-layer Transcoder trained on Claude and attempt to manually interpret them, finding plausible interpretations (e.g., \"activates on commas\", \"activates on non-terminal tokens in multi-token words\") for 6 of the 10 features. In contrast, Rajamanoharan et al. [11] view dense latents as an undesired phenomenon and propose a frequency-based regularizer to discourage their emergence during training.** Whether these latents reflect meaningful internal computations or arise as undesirable artifacts remains an open question. Our work addresses this question directly, showing that dense latents are persistent, characterized by a specific geometry, and often track interpretable, functional signals in the model.\n\nDense Language Model Representations.\n\n**Prior work has also identified dense signals in language model representations more broadly (i.e., components that encode information consistently across many tokens). Gurnee et al. [1] present a taxonomy of universal neurons that appear across GPT-2 models trained with different seeds. Among these, they identify neurons that encode positional information at each input token. Chughtai and Lau [3] similarly identify dense positional features in an SAE trained on GPT-2’s layer 0, though they do not explicitly analyze their activation density. Finally, Stolfo et al. [2] describe neurons that regulate model confidence by tracking entropy and connect them to a component of the residual stream aligned with the quasi-nullspace of the unembedding matrix.**\n\n—————————————————————————————————————————————\n\n\n> **W3.3:** What is the meaning of \"a linear combination of the \"true\" basis works for reconstruction too\"?\n\nWe apologize this was unclear; we had to cut a longer discussion for space reasons. The idea here is that if some set of underlying LLM signals are dense and co-occur perfectly, then any basis of this dense space has the same sparsity penalty to get the same reconstruction (the sparsity penalty $L_0$ is simply the dimension of the dense space), so there is no reason that the \"true\" basis of features would get learned. Although dense latents are not perfectly co-occurring, some co-occurrence could still cause the learned basis to be slightly rotated from the \"true\" basis.\n\n> **W5:** Would it be interesting to plot distributions of latents (like Figure 11) depending on the frequency?\n\nGood question. If we color the points in Figure 11c by activation frequency, the trend remains similar: dense latents cluster near (-1, -1), consistent with the antipodality pattern. This follows from the strong correlation between activation frequency and antipodal structure.\n\n> **Q2:** Do you know whether the presence of (almost) antipodal pairs is a consequence of the frequency of the latent (more frequent features need to be reconstructed more accurately)? Does reducing the dictionary size reduce the number of antipodal pairs?\n\nWe believe that antipodality is a consequence of the fact that dense features almost always are active, so it is hard for the SAE bias to correctly learn the \"bottom\" of the dense feature. To support this hypothesis, we see that reducing the size of the dictionary does not significantly reduce the number of pairs (12 pairs at 16k, 18 at 1m), but replacing the ReLU TopK activation function with an \"absolute TopK\" activation function (allowing activations to be both positive and negative, and taking the top-k activations by absolute value) does remove all pairs.\n\n> **Q3:** Do you know how much different types of dense latents overlap?\n\nBased on our chosen tests and cutoffs, very few (3.6%) of all dense latents fall in more than one category. Most overlaps are between sentence- and paragraph- tracking latents (as discussed in Appendix E.2, Figure 15), as well as between meaningful-word and nullspace/alphabet latents (likely due to the cutoffs chosen).\n\n—————————————————————————————————————————————\n\nWe also apologize for the clarity issues. We clarify each one below, and have added the corresponding explanations to the paper.\n\n> **W1.1:** Are the inputs of SAEs the features after adding the MLPs output to the residual stream?\n\nYes, these SAEs are \"between layers\", so e.g. the layer 0 SAE is after the first attention and MLP output.\n\n> **W1.2:** Is the subspace ablation based on encoder or decoder vectors?\n\nWe ablate based on decoder vectors.\n\n> **W1.3:** The definition of \"antipodality score\" lacks justification (the paper mentions its drawbacks), but I think that it accomplishes its goal.\n\nThank you for acknowledging the usefulness of our metric. We designed the antipodality score based on the intuition that antipodal pairs (latents that jointly reconstruct a single line) should exhibit cosine similarity close to -1 in both their encoder and decoder weights. We take the product of these two similarities to capture cases where both alignments are strong and quantify the extent to which a given latent participates in such a pair.\n\n> **W1.4:** Relatedly, in Figure 11 I find it unclear how the indices $i, j$ are chosen and whether the antipodality score (color) correspond to $i$ or $j$?\n\nThe indices are indexing SAE latents. For a given SAE latent $i$, we loop over all other SAE latents $j \\ne i$, compute the similarity product from equation 1 for the pair of latent i and j, and then pick the latent $j$ that maximizes the quantity in equation 1. This latent $j$ is then used to calculate the antipodality score for latent $i$. Figure 11c then shows this antipodality score for each latent $i$.\n\n>**W1.5:** Why is encoder cosine similarity chosen rather than decoder cosine similarity?\n\nFor a single SAE latent, the encoder and decoder directions have high cosine similarity, so it is somewhat arbitrary which we chose to show (i.e., the decoder pattern would be similar).\n\n> **W3.1:** L36 \"antipodal pairs, effectively reconstructing specific subspace directions.\" Perhaps something like \"antipodal pairs, each effectively encoding a single direction\" would be more clear. \n\nThank you, we will make this change!\n\n> **W4.1:** \"many pairs\" could be expressed more precisely.\n\nWe will change this to read many pairs ( for the SAE shown in Figure 1, there are 10 such pairs).\n\n> **W4.2:** Figure 11 a: Consider showing the 10% threshold.\n\nTo clarify, the plot is showing the median principle angles between each layer’s dense subspace, where the dense subspace is defined as the space spanned by only dense (frequency > 0.1) latents. \n\n> **W4.3:** The boxplots are a bit hard to recognize. What are the dotted lines?\n\nThe dots in the boxplots represent outliers outside the whiskers, based on the standard Plotly implementation. Specifically, whiskers extend to 1.5 times the interquartile range, and any data points beyond this range are marked as outliers. We will add a note clarifying this in the appendix of the final version.\n\n> **W4.4:** typo: \". and\" -> \", and\"\n\nWe will fix this, thank you!\n\n> **W4.5:** The context of Figure 16 seems to be missing. What are the run lengths of? What is the parameter of the Bernoulli distribution?\n\nWe apologize for this, this is a plot for an additional experiment that we forgot to describe in the text, and we have removed it from the paper (the average run length is the average number of tokens that each latent fires for).\n\n> **W4.6:** Consider including an overview of hyperparameters, models and datasets.\n\nThank you for this suggestion! We include chosen hyperparameters for feature cutoffs in Appendix E.1 and relevant experiment details in Appendix B, and additionally plan to open source our code.\n\nIn conclusion, we thank the reviewer for their thoughtful feedback. We hope that our responses have addressed their concerns and provided sufficient clarification."}, {"review_id": "KqZkU6Gm33", "reviewer": "Reviewer_59xx", "summary": "This paper demonstrates that densely activating SAE latents are not mere training artifacts but correspond to principled directions in a model's residual-stream subspace. It shows that many of these latents come in antipodal pairs that encode a single one-dimensional feature, maps how dense latents bind to multiple contextual signals including context-binding units and introduce new taxonomy of the dense latents.", "questions": "- In Section 3.1, how exactly are the dense-latent components removed before retraining the SAE?\n- Because the taxonomy relies on a properly trained SAE, could under- or over-fitting of the autoencoder bias the classification? Wouldn’t a direct statistical analysis of the raw LLM activations avoid this dependency?\n- Are there latents that play more than one role, for example, units that behave both as position latents and context-binding latents?\n- PCA latents are reported, but what interpretive value do they have—how should we understand their function?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n- The authors provide *quantitative evidence* that dense latents reflect the geometry of the residual-stream subspace rather than incidental training artifacts.\n- The paper carefully analyzes the antipodal-pair phenomenon in SAEs, showing how two latents encode positive and negative versions of the same direction.\n- The study quantifies how strongly dense latents correlate with multiple contexts, revealing their broader representational role.\n- The authors identify context-binding latents that preserve the current topic and statistically characterize their antipodal behavior.\n- The work shows that null-space latents function like an internal temperature control, a novel and intriguing finding.\n\n## Weakness\n- Generality remains unclear: with experiments limited to Gemma 2, the findings might still be model-specific. Replicating the core result, especially the claim that dense latents are not SAE artifacts, on several other LLMs would strengthen the paper.\n- The main text does not fully explain how each latent class is assigned. A unified, detailed description of the classification criteria (ideally consolidated in the appendix) is needed, including justification for heuristic thresholds such as the chosen Spearman $\\rho$ cut-offs.", "was_revised": true, "final_justification": "Concerns remain regarding biases arising from the training process and the autoencoder, and the quantitative evidence is not fully sufficient. Nevertheless, I consider the study to be highly original and will therefore maintain the current score.", "rebuttal": "Thank you for your valuable feedback and for acknowledging the “novel and intriguing finding” of our work.\n\n > **W1:** Generality remains unclear: with experiments limited to Gemma 2, the findings might still be model-specific. Replicating the core result, especially the claim that dense latents are not SAE artifacts, on several other LLMs would strengthen the paper.\n\nTo address this point, we replicated the dense subspace ablation experiment from Section 3.1 on a second model, GPT-2. As with Gemma 2, we find that training an SAE on activations with the dense subspace ablated results in almost no dense latents, while ablating an equally-sized sparse subspace has no comparable effect. These additional results strengthen the evidence that dense latents are not artifacts of SAE training and generalize beyond a single model. We also note that Appendix A.1 already includes a replication of the antipodality results on GPT-2. We will include these additions in the updated version of the paper. \n\n> **W2:** The main text does not fully explain how each latent class is assigned. A unified, detailed description of the classification criteria (ideally consolidated in the appendix) is needed, including justification for heuristic thresholds such as the chosen Spearman cut-offs.\n\nWe detailed in Appendix E.1 the tests and cutoffs we used for each category of dense latents. We chose heuristic thresholds by identifying latents with outlier scores for the corresponding test (e.g. for Spearman cut-offs, see Figure 13 in Appendix E.2; for AUC cut-offs, see Figure 19 in Appendix E.6), and estimating a reasonable cut-off. We acknowledge that there is some noisiness to choosing fixed cut-offs in this way, and we do not claim that our classification is fully accurate but serves more as a guide. We will improve the clarity of the consolidated table in the final version, with references to the corresponding plots. \n\n> **Q1:** In Section 3.1, how exactly are the dense-latent components removed before retraining the SAE?\n\nTo ablate this subspace from the language model’s residual stream, we remove the component of the residual that lies within it. Formally, let $\\mathbf{W} \\in \\mathbb{R}^{k \\times d}$ be the matrix of the latents’ decoder weights and $x \\in \\mathbb{R}^d$ the residual stream. We compute the projection $\\mathbf{p} = \\mathbf{x} \\mathbf{W}^* \\mathbf{W}$ (where  $ \\mathbf{W}^*$ is the pseudoinverse of $\\mathbf{W}$) and remove it: $\\mathbf{x}’ = \\mathbf{x} - \\mathbf{p}$. This yields the ablated residual stream $\\mathbf{x}’$. We will add an appendix section to clarify this procedure in the final version of the paper.\n\n> **Q2:** Because the taxonomy relies on a properly trained SAE, could under- or over-fitting of the autoencoder bias the classification? Wouldn’t a direct statistical analysis of the raw LLM activations avoid this dependency?\n\nThis is a good question. We actually do study how the number of dense latents changes across training (Appendix A.2), and we observe that the number of dense latents stabilize early in training. Therefore, we do not expect that they are a result of under-/over- fitting. \n\nAs for the direct statistical analysis of the raw LLM activations, dictionary learning is necessary to isolate and identify the directions that are frequently present in the activations, and it is unclear how we would otherwise find the “dense” subspaces from the LLM activations alone.\n\n> **Q3:** Are there latents that play more than one role, for example, units that behave both as position latents and context-binding latents?\n\nIn our analysis, we did not find clear evidence of latents that play multiple roles. Due to the nature of our classification using the chosen heuristics and cutoffs, there are a few (3.6%) dense latents that fall into more than one category. Most overlaps are between sentence- and paragraph- tracking latents (as discussed in Appendix E.2, Figure 15), as well as between meaningful-word and nullspace/alphabet latents (likely due to the cutoffs chosen).\n\n> **Q4:** PCA latents are reported, but what interpretive value do they have—how should we understand their function?\n\nWe were unable to form hypotheses on interpreting PC-aligned latents by examining the activations. However, they could be related to rogue dimensions [1], as LLM activation spaces are highly anisotropic.\n\nIn conclusion, we thank the reviewer for their constructive comments. Their feedback prompted additional experiments and clarifications that we believe improve both the clarity and robustness of our work. We hope our responses address the reviewer’s concerns.\n\n—\n\n[1] Timkey, William, and Marten Van Schijndel. \"All bark and no bite: Rogue dimensions in transformer language models obscure representational quality.\" (2021)."}]

Author rebuttal:

Meta-review: This paper studies the dense latent property of the SAE trained on language models.

All the reviewers agree that the analyses are novel and the findings in the paper are original and interesting. The authors successfully address most reviewers' concerns. Although several concerns remain regarding biases arising from the training process and the autoencoder, and the quantitative evidence is not fully sufficient. The reviewers believe the paper is still worth publishing due to its novel results.

I recommend the acceptance of the paper, and encourage the authors to include the discussions and new results in the revision.
Decision: Accept (poster)