A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
Authors: David Chanin
James Wilken-Smith
Tomáš Dulka
Hardik Bhatnagar
Satvik Golechha
Joseph Isaac Bloom
Venue: NeurIPS 2025 oral ()
Keywords: sparse autoencoders
SAEs
interpretability
NLP
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into human-interpretable latent directions or features. As we increase the number of features in the SAE, hierarchical features tend to split into finer features (“math” may split into “algebra”, “geometry”, etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get “absorbed” into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form a hierarchy. We introduce a metric to detect absorption in SAEs, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity is insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.

A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders
David Chanin "1
James Wilken-Smith
Tomás Dulka Joseph Bloom 1.5
Hardik Bhatnagar "1.3
Satvik Golechha
'LASR Labs
University College London
Tübingen AI Center, University of Tübingen
*MATS
'Decode Research
Abstract
Sparse Autoencoders (SAEs) aim to decompose the activation space of large language models (LLMs) into guman-interprretale As we increase the number of features in the SAE, hierarchical features tend split into finer features ("math" may split into "algebra", "geometry", etc.), a phenomenon referred to as feature splitting. However, we show that sparse decomposition and splitting of hierarchical features is not robust. Specifically, we show that seemingly monosemantic features fail to fire where they should, and instead get "absorbed" into their children features. We coin this phenomenon feature absorption, and show that it is caused by optimizing for sparsity in SAEs whenever the underlying features form hierarchy. We introduce metric detect absorption in SAES, and validate our findings empirically on hundreds of LLM SAEs. Our investigation suggests that varying SAE sizes or sparsity insufficient to solve this issue. We discuss the implications of feature absorption in SAEs and some potential approaches to solve the fundamental theoretical issues before SAEs can be used for interpreting LLMs robustly and at scale.
latent directions features.
Introduction
Large Language Models (LLMs) have achieved remarkable performance across wide range of tasks, yet our understanding of their internal mechanisms lags behind their capabilities. This gap between performance and interpretability raises concerns about the "black box" nature of these models [30]. The field of mechanistic interpretability aims to address this issue by reverse-engineerigg the internal algorithms of neural networks and performing causal analysis them [4
Recent work theorizes that models represent concepts as linear directions in a high-dimensional space, known the Linear Representation Hypothesis (LRH) [26] 8.. The model is able to represent far more concepts than it has neurons in its hidden space by allowing these concept directions to overlap slighly, known superposition [8]. Superposition makes it challenging to directly interpret neurons in an LLM, and requires different techniques to extract interpretable feature directions.
As long as the active features in a given LLM activation are sparse and the underlying features follow the LRH, Sparse Autoencoders (SAEs) should be able recover the true LLM features despite supersition using sparse dictionary learning (27). Indeed, SAES have shown potential in decomposing the dense, polysemantic activations of L.I.ss into more "interpretable" latent features 25.
However, we show that even all underlying features are linear and sparsely activating, an SAE will still fail recover the true underlying features if the features form hierarchy, Instead, an SAE will
These authors contributed equally to this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Ideal interpretable solution
Uninterpretable absorption solution
SAE Encoder
SAE Decoder
SAE Encoder
SAE Decoder
Latent
"starts with S" "short"
"starts with s" "short"
Latent
""short" A "starts S" "starts with S"
Latent
2
Latent
"short"
"short" + "starts S
The ideal interpretable solution requires firing 2 latents to represent "short".
Absorption only requires firing latent represent "short", and is unfortunately what the SAE learns.
Figure| 1: In feature absorption, seemingly monosemantic latents fail fire in cases where they apparently should. Here, we see an SAE can represent the word "short" and the concept "starts with S" more sparsely by absorbing the "starts with S direction into the "short" latent, and then not firing the "starts with S" latent on the word "short", despite "short" starting with "S" Logical notation is used describe the SAE encoder to emphasize its role as a classifier.
learn gerrymandered latents that fail to fire seemingly arbitrary cases where the latent should fire according to the mainline interpretation of the latent. We refer to this failure as feature absorption.
Feature absorption is demonstrated in Figure where the feature "short" always fires alongside feature representing "starts with S Instead of learning an interpretable latent representing "starts with S", the SAE can increase sparsity by instead disabling the "starts with S latent when "short" is active while still getting perfect reconstruction.
We present the following contributions: (1) we identify a problematic variant of feature-splitting we call "feature absorption", where an SAE latent appears track a human- cinnan-itterrettblle concept, but fails to activate on seemingly arbitrary tokens. Instead, more specific latents activate and contribute a component of feature direction, "absorbing" the feature. (2) We demonstrate that feature absorption is caused by hierarchical features. (3) We develop a metric to detect feature absorption in LLM SAEs. And (4) we validate that feature absorption occurs in every LLM SAE we tested, including hundreds of open-source SAEs.
Feature absorption poses an obstacle the practical application of SAES since suggests SAE latents may be inherently unreliable classifiers. This is particularly important for applications where we need confidence that latents are fully tracking behaviors, such bias deceptive behavior. Furthermore, techniques which seek to describe circuits in terms of a sparse combination of latents will also be more difficult in the presence of feature absorption (2]).
An online explorer for our results can be found at htwpu//oo//asssssslllgnaaaaaaaaa
streamlit
app Code is available at htthtts::/// //gi www
2 Background
Hierarchical features. We say features fi and fa form a hierarchy with f1 as the parent and fz as the child fa = fi, meaning every time fz fires fi must also fire.
Linear probing. A linear probe is simple linear classifier trained on the hidden activations of neural network, typically using logistic regression (LR) 0.
K-sparse probing. A k-sparse probe [12 is a linear probe trained on a sparse subset of A neurons or SAE latents. Training a k-sparse probe first requires selecting the best neurons or SAE latents that in-aggregate act good classifier, and then training a standard linear probe on just those k neurons or latents.
Gurnee et al. 12 proposed several methods of estimating the best k neurons or latents to pick, one of which involves first training LR probe with loss term, and selecting the k largest elements by probe weight. When we refer to k-sparse probing in this work, we use this method of selecting k latents.
Sparse autoencoders. An SAE consists of an encoder, Wenes decoder, W Waees and corresponding biases Done and bitee- The SAE has a nonlinearity, a, typically a ReL.U (or variant such as JumpRel.U 29 Given input activation, the SAE computes a hidden representation, f. and reconstruction, à:

---

f = F (Wene ene + Denc)
= d + bdec
SAEs attempt reconstruct input activations by projecting into overcomplete basis using sparsity-inducing loss term (typically L1 loss), or certain number of non-zero latents (LO) on the hidden activations.
SAE feature ablation. In an ablation study we examine the downstream causal effect of an SAE latent by computing how patching its activation 0 changes downstream metric (e.g. logit difference). negative ablation effect means intervening on the SAE latent would lower the metric. We follow the work of Marks et al. 21 and provide the algorithm in Appendix A.4
3 Toy models of feature absorption
Absorption is caused by the SAE sparsity penalty in the presence of hierarchical features. When two features form hierarchy, for instance "starts with S" and "short", the SAE can merge the "starts with S" feature direction into a latent tracking "short" and then not fire the main "starts with S" latent. This means firing one latent instead of two, increasing sparsity while retaining perfect reconstruction. We demonstrate that hierarchical feature co-occurrence causes absorption in a simple toy setting.
Our initial setup consists of 4 true features, each randomly initialized into orthogonal directions with a 50 dimensional representation vector and unit norm. Each feature fires with magnitude 1.0. Feature fo fires with probability 0.25, and features fi, /2, and Sa fire with probability 0.05. Each SAE training input is created by sampling from these true features and summing the directions of each firing feature. We train SAE with latents match the 4 true features using SAELens (I5). The SAE uses LI loss with LI coefficient 3e-5, and learning rate 3e-4. We train on 100M activations.
Independently firing features When the features fire independently, find that the SAE is able to perfectly recover these features as shown in Figure 2a The SAE learns one latent per true feature. The decoder representations perfectly match the true feature representations, and the encoder learns to perfectly segment out each feature from the other features.
sim true features (Independent features)
Cos true features (feut co-occurs w/feat 0)
SAB encoder
SAE decoder
SAB encoder
SAE decoder
1.0
1.0
SALE MANO
0.6
SAET Lacoo
0.5
sim
0.0
0.0
Truc feature
True feature
True feature
True feature
(a) When features fire independently, the SAE learns exactly one latent per feature, and the decoder per- fectly recovers each feature direction.
(b) When features 0 and 1 co-occur (feature only fires if feature 0 fires), see absorption in the SAE encoder and decoder latents and 1.
Figure 2: Comparison of independent features (left) vs. co-occurring features with absorption (right)
Hierarchical features cause absorption Next, modify the firing pattern of feature so it fires only feature 0 also fires. We keep the overall firing rate of feature I the same as before, firing in 5% of activations. Features and remain independent.
Figure 3: Interpretation of learned SAE latents
with co-occurrence between feature 0 and feature I (feature only fires if feature 0 fires).
SAE ENCODER
SAE DECODER
LATENT 0 LATENT
feat 1 A feat 0
Feat 0
Figure 2b shows the encoder and decoder cosine similarities with the true features in the hier- archical co-oceurrence setup. Here, we see clear example of feature absorption. Latent 0
feat 1
feat 0 feat
LATENT 2
feat 3
feat 3
LATENT 3
feat
feat 2
3

---

has learned perfect representation of feature O, but the encoder has a hole in its recall. Latent 0 fires if feature is active but not feature 1. This exactly the sort of gerrymandered feature firing pattern we will see later in real SAFs in Section 5.2 the encoder has learned to stop the latent firing on specific cases where it looks like it should be firing. addition, we see that latent 1, which tracks feature I, has absorbed the feature O direction. This results in latent representing combination of feature and feature We see that the independently firing features 2 and 3 are untouched the SAE still learns perfect representations of these features. These results are summarized in Table 3 We explore absorption more toy settings in Appendix A.3
Proof: hierarchical features cause absorption We further provide an analytical proof that in the hierarchical setup described above, feature absorption decreases SAE loss Appendix A.2
Experimental setup
Our experiments on LLM SAFs focus predicting the first-letter of single token containing characters from the English alphabet (a-7, A-Z.) and an optional leading space. We use in-context learning (ICL) prompts elicit knowledge from the model, using templates of the form:
{token) has the first letter: {capitalized faaeed_ffsss___eeeeeeeee
An example of an ICL prompt consisting of 2 in-context examples is shown below. The model should output the _D token:
tartan has the first letter: T mirth has the first letter: dog has the first letter:
In the above prompt, we extract residual stream activations at the _dog token index. These activations used both for LR probe training and for applying SAEs. We train/test split of 80% / 20%, and evaluate only on the test set of the probes, including when running experiments on SAFS. When applying SAEs, include the SAE error term 21 to avoid changing model output.
To determine the causal effect of SAE latents on the first-letter identification task we conduct ablation studies. We use metric consisting of the logit of the correct letter minus the mean logit of all incorrect letters. This measures the propensity of the model to choose the correct starting letter as opposed to other letters. Formally, metric m is defined below, where refers to the final token logits, L is the set of uppercase letters, and y is the uppercase letter that is the correct starting letter:
m = =gly]
Σ g|ll
T
(E(TAw)
We discuss this metric and alternative formulations further in Appendix A.
To determine how well multiple latents perform classifier when used together, we use k-sparse probing, increasing the value of i from to 15. We train a LRR probe using a loss term with coefficient 0.01. and select the top A latents by magnitude.
We use the base Gemma-2-2B model for most of our studies, along with the full set of Gemma Scope residual stream SAES of width 16k and 65k released by Deepmind [19]. We also evaluate absorption on our own SAEs trained on Qwen2 0.5B and Llama 3.2 1B 6).
5 Results
Our results are divided into four sections. First, we compare the performance of linear probes with SAE latents on recovering first-character information from model activations, showing that despite appearing to track first letter features, wide variety of precision / recall is achieved. Second, we motivate our definition of feature absorption with a case-study, emphasizing how an absorbing latent can unexpectedly causally mediate first letter information whilst the first-letter latent (unexpectedly) fails to fire. Next, we attempt to quantify feature splitting and feature absorption, showing that tuning of hyper-parameter may partially assist but not fully alleviate feature absorption.

---

5.1 Do SAEs learn latents that track first letter information?
We compare the performance of LR probes with the performance of the SAE latent whose encoder direction has highest cosine similarity with the probe, resulting in 26 "first-letter" latents. We observed that for each probe, there was clearly one or most couple of outlier SAE latents with high probe cosine similarity. Full plots of cosine similarity vs letter are shown in Appendix AA.8
We also tried using k=1 sparse probing 12 to identify SAE latents, and found this gives similar results. Further comparison of using k= sparse probing vs encoder cosine similarity to identify latents is explored in Appendix A.7
(a) Mean Fl score using top SAE encoder latent by layer for Gemma Scope SAFs and LR probe.
(h) Mean Fl score vs LO for
(c) Mean Fl score vs L.0 for
Gemma Scope SAEs layers 0-12.
Gemma Scope SAEs layers 13-25. SAEs learn cleanest "first-letter" features with 1. near 50-100.
SAFS learn cleanest "first-letter"
features with 1.0 near 25-50.
Figure 4: Comparison of Fl scores for first-letter classification tasks
Pine-cote 5AB piddice
Fias SAB puciidos sacalll
Wea prc
vod
(a) Mean precision using top SAE encoder latent vs LO for all Gemma Scope SAES.
(b) Mean recall using top SAE en- coder latent vs L0 for all Gemma Scope SAEs.
(c) Mean precision vs recall using top SAE encoder latent vs L0 for all Gemma Scope SAEs.
Figure 5: Precision and recall vs LO for first-letter classification tasks
We observe wide variance in the performance of Gemma Scope SAES the first-letter identification task, but no SAE matches LR probe performance. We show the mean F1 score by layer as well as the Fl score of the LR probe in Figure 4a We further investigate the F1 score of these SAE encoder latents as a function of LO and SAE width in Figures 4b and 4c
Whether or not SAE learns clear "first-letter" latent for each letter is highly dependent on LO, with low LO SAES tending to learn high-precision low-recall latents, and high LO SAES learning low-precision high-recall latents (Figure 5). We caution drawing conclusions about an "optimal" LO from these plots, as we find further variance when broken-dows down by letter, shown in Appendix A.8
5.2 Why do SAE latents underperform?
The Gemma Scope layer 3. 16k width, 59 LO SAE has latent, 6510, which appears to act classifier for "starts with S", achieving an Fl of 0.81. However, this latent fails to activate on some tokens the probe can classify, and which the model can spell, such as the token _short
Figure 6a shows a sample prompt containing a series of tokens that start with "S", and the activations of top SAE latents by ablation score for these tokens. The main "starts with S" latent, 6510, activates on all these tokens except sshott. This SAE also has token-aligned latent, 1085, which activates on variants of the word "short" (" short", "SHORT", etc.... The Neuronpedia dashboard (20 for latent 1085 is shown in Appendix[A.Is For the token _ssort,, the main "starts with S" latent does not activate but the "short" latent activates instead.
5

---

'S' activations by token, layer 16k width, 59 LO
Cosine similary between SAE decoder and probe
Laten ID
laoen
1085
Cosine similarity
Activaionn
al 1085, oa 0.12
30
0.2
20
0.0
22000
73000
100000
12300
15000
Token
SAE Intent
(a) Layer 3, L0=59 SAE latent activations for tokens that start with "S". The core "starts with latent, 6510, fails to activate on the token _short. The "soket"-token aligned latent 1085 activates instead.
Cosine similarity between layer 3, L0=59 SAE de- coder and the "starts with probe, The main "Starts with S" latent, 6510, is clearly visible and highly probe-aligned.
Figure 6: Comparison SAE latent activations and cosine similarity for tokens starting with "S"
Finst-letter ablation effects for "short" Inken, 1D059
Ablarion effects for "sorr" tnken projecting probe
effedt
Ablation elfeed
1005
3014
857
1686
2794
PMM
2354
437
2013
9134
3775
5489
HPas
SAE Intent ID
SAE Iatent ID
(a) Ablation effect for sshort token, indicating that latent 1085, is responsible for the "starts with S con- cept for the _short token. The main "starts with S" latent, 6510, does activate the _ssoort token.
(b) Ablation effect for ssort token after removing the probe direction from latent 1085 via projection. Latent 1085 no longer appears in the plot, indicating the strong ablation effect in Figure 7a is due its component along probe direction.
Figure 7: Ablation effects _short token before and after projecting out the probe direction
Latent 1085 has a cosine similarity with the "starts with S" probe of 0.12, indicating contains component of the "starts with S" direction, although much smaller than the main "starts with S" latent. Cosine similarity of the SAE decoder with the "starts with S" LR probe is shown in Figure [bb] Interestingly, despite latent 1085 having only about 1/5 the cosine similarity with the probe as the main latent 6510 10,0, we see it activates with about 5 times the magnitude of latent 6510 on the sshott token, thus contributing similar amount of the "starts with S" probe direction to the residual stream.
We study the ablation effect of each SAE latent on the _short token, shown in Figure 7a and see that latent 1085 has dramatically larger ablation effect compared with all other SAE latents. This suggests latent 1085 is causally responsible for the model knowing that _short starts with S.
Is it possible that the probe projection is not the causally important component of latent 1085? We conduct another ablation effect experiment, except now we remove the probe direction from latent 1085 via projection before ablation. The results of this experiment are shown in Figure 7b After removing the probe component from latent 1085, no longer has a significant ablation effect. Thus we know the probe projection of latent 1085 is responsible for model behavior.
These experiments show the "starts with S" feature has been "absorbed" by the token- latent 1085, likely along with other semantic concepts related to the word "short". After observing that the main "starts with S" latent 6510 activates on most tokens that begin with "S", it may be tempting to conclude this latent tracks the interpretable feature of beginning with the letter "S". However, this latent quietly fails activate on the short token, leading to false sense of understanding.
Here we clearly see feature absorption. The seemingly interpretable SAE latent 6510 fails to activate on arbitrary positive examples, and instead the feature is "absorbed" into more specific latents.
Feature absorption is likely logical consequence of SAE sparsity loss. a dense and sparse feature co-occur, absorbing the dense feature into a latent tracking the sparse feature will increase sparsity.

---

Table 1: Sample max activating examples for latents 7112 and 7657 for Gemma Scope 16k, layer 0, 105 LO from Neuronpedia. The token where the SAE latent activates is highlighted in yellow. Latent 7112 appears to be a lowercase "I." starting-letter latent, and latent 7657 appears be corresponding uppercase "I." latent.
LATENT 7112
LATENT 7657
2da se naplaćuje naknada
LC. an aluminum boat LIFT and LF-Net. Once latter's sister Louise, in
E. Soli, 20
>></code></B><hul
5.3
Measuring feature splitting and feature absorption
Feature splitting A key phenomenon identified from previous studies of SAEs is feature-spliting 2] where a feature represented in a single latent in a smaller SAE can split into two or more latents in larger SAE. During our experiments, found strong evidence of feature-splittng in the Gemma Scope SAEs.
For instance, in the layer 0, 16k width, 105 LO SAE, we find two encoder latents (id:7112 12 and id:7657
M which align with the "L" starting letter probe. Inspecting max activating examples, we see latent 7 12 activates tokens starting with lowercase "I", while 7657 activates tokens starting with uppercase "L". Some activating examples for these latents are shown in Table
Feature splitting like this is not necessarily problematic for interpretability efforts since the split features are still easily identifiable, and depending on the context it may more useful have either single "starts with L latent or a pair of "starts with uppercase lowercase L," latents.
We measure feature splitting using k-sparse probing 12 SAE activations. If increasing the k-sparse probe from to k: ++ causes a significant increase in probe FI score, then the additional SAE latent provides meaningful signal, and the combination of these k + latents is likely feature split. In the example of the uppercase "L" and lowercase "I" split, k-sparse probe with k = 2 trained on both these latents should predict "starts with letter L" much better than either latent on its own. Figure Sa shows Fl vs K for letters "L" and "N". The "L" k-sparse probe shows a significant jump in F1 score moving from k=1 to k=2 corresponding to feature splitting, while the F1 score for the "N" k-sparse probe is relatively constant.
K-sparse probing results
splits per
Starts with "I."
Starts with 'N"
0.8
SAC width
16k
Numi Ccature spiits
2.0
0.4
1.5
6%
0.2
68
+g
30
100
150
200
250
300
(a) K-sparse probing results for letters "L" and "N", layer 0, 16k width, 105 LO. "L" shows significant improvement Fl between k=1 and k=2 correspond- ing to feature splitting.
(b) Mean number of feature splits per letter the first-letter spelling task by L.. Feature splitting occurs more frequently with higher sparsity.
Figure 8: Feature splitting analysis in sparse autoencoders
We detect feature splitting by measuring whether increasing & by causes a jump in Fl score by more than a threshold, т. We use 7 = 0.03 as a reasonable choice after inspecting situations like in Figure Sal where feature splitting corresponds to an Fl score jump between 0.05 0.1. Figure Sb shows feature splitting vs LO for all 16k and 65k width Gemma Scope SAES.

---

The single latent or set of traditional feature split latents that seem to act as classifier for human-imlerpretable feature like "starts with S" fail fire in a seemingly arbitrary number of cases.
What fires instead are often approximately token-aligned latents with small but positive alignment with the LR probe. We say these latents are absorbing the feature.
alovepauss
Meaa atsccpptos for tovi-iake loye
SAA
0.25
Quead 353
0.1
Layer
1.0
Lager
(a) Mean feature absorption rate vs layer on the lerttr task, Gemma Scope 16k and 65k SAEs. We do not see an ohvious pattern in absorp- tion rates by layer.
(b) Mcan feature absorption rate vs LO on first-letter task, Gemma Scope 16k and 65k SAEs. Wider and more sparse SAFs demonstrate higher rates of absorption.
(c) Mean feature absorption rate vs layer on the first-letter task on Llama 3.2 1B and Qwen 2 0.5B models, L,1 loss and Topk SAE ar- chitectures, layers 0-8.
Figure 9: Feature absorption rates
We quantify the extent which feature absorption occurs with the metric feature absorption rate. We first find k feature splits for a first-letter feature using a k-sparse probe. We then find false-negative tokens that all k feature-split SAE latents fail to activate on, but which the LR probe correctly classifies, and run an antegrated-gradiene ablation experiment on those tokens. The ablation effect finds the most causally important SAE latents for the spelling of that token. If the SAE latent receiving the largest negative magnitude ablation effect has a cosine similarity with the LR probe above 0.025, and is at least 1.0 larger than the latent with the second highest ablation effect, say that feature absorption has occurred. These thresholds were chosen from manual inspection of the data to best distinguish the absorption phenomenon. We then calculate feature absorption rate
num num_absorptions
absorption rate-
1r wwwpproobttrreee true surueepositives
If there are more than 200 false negatives per letter, we randomly pick 200 samples to estimate the number of absorptions. We see absorption rate increases with higher sparsity and higher SAE width. Lower LO likely pushes the SAE to absorb dense features like spelling information, increasing feature sparsity. Feature absorption rate LO for Gemma Scope SAEs layers 0-17 is shown in Figure 9b) Absorption rate by letter is shown in Appendix A414 We also train our own set of standard L11 loss SAES on the first layers of Qwen2 0.5B (32 and Llama 3.2 IB (6, and TopK SAES [10 on Llama 3.2 IB. In Figure 9c we show that absorption occurs in these SABS as well.
Our metric cannot capture absorption past layer 17 in Gemma 2 2B since we rely on ablation experiments to be certain the absorbed feature causally mediates model behavior. Past layer 17, attention has already moved the starting letter information from the source token into the final token position, so any ablations on the source token past layer 17 have little effect. This is a limitation of absorption metric-
rely on ablation to be certain of the causal impact of absorbed features on model behavior, but this limits the layer depth our metric can be applied. We discuss this further in Appendix (. A.12 and discuss alternative formulations of the metric in Appendix [3.13
Our absorption metric is not perfect, and is likely an under-estimate of the true level of feature absorption. We only consider absorption to have occurred if single SAE latent has much larger ablation effect than all other latents, and the main SAE latents for a feature do not activate at all. Our metric will not capture multiple absorbing latents activating together, or the main latents activating weakly. Regardless, we feel our metric is a reasonable conservative baseline.
Related work
Applications of Probes and SAES for Model Interpretabilitty Probing methods can extract interpretable information from language models, though this does not guarantee the model uses these representations in its computation, and requires labeled data [7].
8

---

Prior work has shown that many human- cuma-ninterpretable concepts in LLM activations are represented as linear directions in activation space, known as the linear representation hypothesis [8 28]. Li et al. [8] used non-linear probes to recover board representations from a transformer trained on Othello scripts ("OthelloGPTT") Nanda [23] later showed that linear representations were not only
recoverable but also editable.
Karvonen et al. [6] developed objective metries for SAE evaluation using Chess and Othello board states, but does not apply these to SAEs trained on LLM activations. Work by Olah et al. 126. Kissane et al. [17] Templeton et al. [1]] noted poor precision/recall of SAE latents compared to known proxies. We extend this by showing how sparsity mediates precision/recall across many Gemma Scope SAEs and offer possible explanation of low recall due to feature absorption.
Engels et al.
[ investigated SAE errors, finding that not all SAE error is linearly decomposable.
Studying precision and recall of SAE Latents Most existing work on SAE interpretability mainly studies max activating examples (5), which may be misleading. There are more rigorous works which only measure precision 2 BI 17. Recent work has briefly explored recall and found it to be worse than expected naively, but this remains poorly understood [26]. We build on this work by evaluating precision/ recall on large number of SAEs, and offer partial explanation for lower- -eeeeete-- recall of SAE latents in the form of feature absorption.
Decomposing SAE Latents Feature splitting was first described in Bricken et al. (2) which noted that different SAE widths and sparsities induce latents of different granularity, with wider SAES often learning more specific variants of features. Bussmann et al. [4] find that by training an SAE on the decoder of another SAE, a technique called SAEs, latent like "Einstein" into subcomponents like "German" and "Physicist" and "starts with E".
possible to break down a single SAE
7 Discussion
Limitations Our Absoption metric uses ablation effect to ensure that the absorbed features causally mediate model behavior, and thus might not be easily transferable to the final model layers. Alternate metric formulations mitigating this are discussed in Appendix AA.3 Due to compute constraints, we only train and evaluate small number of non-JumpReL.U SAES in Figure As our goal was only to show absorption occurs in all SAE architectures, we did not feel this is a significant drawback.
Future Work The primary goal of future work is to find solutions to feature absorption. We are particularly hopeful that work extending Meta-SAEs 4 may solve or mitigate feature absorption. Another possible solution may be attribution dictionary learning [25]. Finally, structured sparsity techniques such as group lasso [3 or hierarchical sparse coding may also be a promising direction of future work.
Other possible directions include allowing absorption occur and using it way recover hierarchies between features in a LLM. Our toy model results suggest that absorption leads to an asymmetric pattern in the encoder and decoder of the SAE, so it may be possible use this insight to detect absorption (although there may be other reasons for an asymmetry in the SAE encoder and decoder beyond absorption).
Conclusion We identify a form of feature splitting we call "feature absorption". where more specific latents "steal credit" from more general ones. Absorption creates interpretability illusion, where seemingly interpretable latent has arbitrary false negatives in its mainline interpretation. Lower recall poses problems for using SAEs for high-stakes classification or finding sparse circuits (21]. as the number of latents needed to characterize model behavior may be much larger than expected.
We show that absorption is a consequence of hierarchical co-occurrence between sparse and dense features. a dense feature like "starts with letter D" always co-occurs with a more sparse feature like "dogs", the SAE can increase sparsity by absorbing the "starts with D" feature into "dogs" latent.
We hope that our work highlights the fundamental limitations sparse feature extraction and prompts future research on SAEs such as identifying cases where a feature "should have activated" but does not due to absorption, and exploring theoretical solutions to absorption. The ease of demonstrating absorption in toy models makes it casier to validate potential solutions.

---

Acknowledgments and Disclosure of Funding
This project was produced as part of the LASR Labs research program. DC was supported thanks to EPSRC EP/S022166/1..
References
[1] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2017. URL UtL.pttts://opearevi.e net/forum? ryy7
[2] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2, 2023.
[3] Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders, 2024. URL https://arxiv. org/ abs/2412 06410
[4] Bart Bussmann, Michael Pearce, Patrick Leask, Joseph Bloom, Lee Sharkey, and Neel Nanda. Showing sae latents are not atomic using meta-saes, 2024. |tt/ lesswrong hoal/poostts//Aaalll4ooaaaa---aa
[5] Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, Robert Huben, and Lee Sharkey, Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL id=F76bwASLeK
wwwirrvview.ne/fforrre net/forum?
[6] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic. Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong. Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez,
10

---

Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu,, Chiig-HHsinng Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, lean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li. Kun Huang, Kunal Chawla, Kushal Lakhotia. Kyle Huang, Lailin Chen, Lakshya Garg. Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina. Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad lonescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi. Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL http///xxiv..rrr/ org/ abs/2407 21783

---

[7] Yanai Elazar. Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg, Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9: 9:160-175, 2021.
[8] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zae Saffield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprini arXiv:2209. 10652, 2022.
[9] Joshua Engels, Eric J Michaud, Isaac Liao, Wes Gurnee, and Max Tegmark. Not all language model features are one-dimensionally linear. The Thirteenth International Conference on Learning Representations, 2025. URL https://oppererve
[10] Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, llya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406. 04093, 2024.
[11] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216- 2023.
[12] Wes Gumnee Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research, 2023. ISSN 3835-8856. URL
net/
forum?1d=JYs1R91T
[13] Laurent Jacob, Guillaume Obozinski, and Jean-Philippe Vert. Group lasso with overlap and graph lasso. In Proceedings of the 26th annual international conference machine learning, pages 4334440, 2009.
[14] Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, and Francis Bach. Proximal methods for hierarchical sparse coding. The Journal of Machine Learning Research, 12:2297-2334, 2011.
[15] Curt Tigges Joseph Bloom and David Chanin. Saelens. http:://github..co//bbooooo SAELens 2024.
[16] Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Riggs Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictionary learning for language model interpretability with board game models. In ICML 2024 Workshop on Mechanistic Interpretability, 2024.
[17] Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. Interpreting attention layer outputs with sparse autoencoders, 2024. URL https://arxiv org/ htg/abs/2406, 17759
[18] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. ICLR, 2023.
[19] Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2, August 2024.
[20]
Johnny Lin and Joseph Bloom. Analyzing neural networks with dictionary learning, 2023. URL hnttps://www. wwwp://wwwwoooeea..oomm org Software available from neuronpedia.org.
[21] Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. Computing Research Repository, arXiv:2403. 112224943......... 19647, 2024. http:sttss://aras wrg. abs/
2403. 19647]
[22] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing fac- tual associations in GPT. Advances Neural Information Processing Systems, 36, 2022. arXiv:2202.05262.
12

---

[23] Neel Nanda. Actually, othello- gpt has linear emergent world model, mar 2023. URL https://neehanda.. wtt.aiincccnniiiittttrrrraaaiaaa
2023.
[24] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5()):c00024-000 001, 2020.
[25]
Chris Olah, Adly Templeton, Trenton Bricken, and Adam Jermyn. April update.
https: 2024. URL https
html
[26] Chris Olah, Nicholas Turner, Adam Jermyn, and Joshua Batson. July update.
https://
bransformer-cireee http://rwe..cccccc...././///////
html mannsformer--cicuuttt.. httpa//swssaaaaaaaao//ooooo/o//a
2024. URL https://
html
[27] Bruno A Olshausen and David J Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision research, 37(23):331133325, 1997.
[28] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning, 2024.
[29] Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint 2rXiv:2407.1443. 2024.
[30] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, pages 206-215, 2019.
[31] Adly Templeton, Tom Conerly, Jonathan Mareus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extract- ing interpretable features from claude 3 sonnet. https: //trans///rastttt ormer- circuits pub/ 2024/scaling wtw.anoooooooommiiiii..o.. May 2024. Accessed on May 21, 2024.
[32] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen. Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
A Appendix
A.1 Glossary of Terms
Sparse Autoencoders (SAEs): Neural networks trained to reconstruct their input while enforcing sparsity in their hidden layer. In the context of this paper. SAEs are used to decompose the dense activations of language models into more interpretable features.
SAE error term: When inserting a SAE into the computation path of the model, errors in SAE reconstruction will propagate to later parts of the model and can change the model output. refer to the error as the SAE error term, and corresponds to the difference between the SAE output and the original SAE input activation. Marks et al. 21 introduced the idea adding this error term back to the SAE output to ensure that the SAE does not change model output.
13

---

Latent: We refer to neurons in the hidden layer of a SAE as latents to avoid overloading the term "feature". This is in contrast to earlier work which used the term "feature" refer both human-interpretable concepts and SAE hidden layer neurons.
Feature: We use the term "feature" to refer to an idealized human- Juman-interpretable concept that the model represent in its activations and which SAE latent may may not represent.
Monosemantic:
Referring to feature representation that corresponds to single, clear semantic concept. In the context of SAEs, a monosemantic feature would ideally capture one interpretable aspect of the input.
Interpretable: latent being interpretable not well defined in the field, making it difficult to ensure that different authors mean the same thing when referring SAE interpretability. When we refer to a SAE latent as being interpretable in this work, we mean that it should behave in line with how appears behave after inspecting its activation patterns. If an SAE latent appears to track a feature X by a reasonable inspection of its activations but has subtle deviations from this behavior in reality, we say this is not interpretable. We thus measure interpretability via classification performance when a latent appears to be a classifier over some feature.
Feature dashboard: A dashboard showing activation patterns and max-activating examples for SAE latent. Feature dashboards are commonly used to interpret the behavior of an SAE latent.
Neuronpedia: A platform, https://neuronpedia. org, which hosts feature dashboards for popular SAES (20]
Token-aligned latent: A latent which seems to roughly fire variants of the same token. For instance, a "Snake" token-aligned latent may fire on the tokens "Snake", "SNAKE", " ""snakes",, etc...
Feature splitting: A phenomenon in SAEs introduced by Bricken al. where SAE latent tracks general feature in narrow SAE, but splits into multiple more specific SAE latents in wider SAE. For instance, latent tracking "starts with L" in a narrow SAE may split into latent tracking "starts with capital L" and a latent tracking "starts with lowercase L" in a wider SAE.
Feature absorption: A problematic form of feature splitting where a SAE latent appears to track an interpretable feature, but that latent has seemingly arbitrary exception cases where fails to fire. Instead, a more specific latent "absorbs" the feature direction and fires in place of the main latent.
In the context of neural network interpretability, a circuit refers to subgraph of neurons or latents within a neural network that work together to perform specific function or computation. The study of circuits aims understand how different components of a neural network interact to process information and produce outputs.
Circuit:
Linear probe: A simple linear classifier (typically logistic regression) trained on the hidden activations of neural network to predict some property task. Used to assess what information is linearly decodable from the network's representations.
K-sparse probing: A variant of linear probing where only the most important latents (as deter- mined by some selection method) are used to train the probe. This helps identify which specific neurons or latents are most relevant for given task.
Ablation study: An experimental method where component of system (in this case, latent in a neural network) is removed or altered to observe its effect on the system's 's performance. This helps determine the causal importance of the component.
Integrated gradients (IG): An attribution method that assigns importance scores to input latents by accumulating gradients along path from baseline input to the actual input. In this paper, it's used approximation technique for ablation studies.
14

---

In-context learning (ICL): A paradigm where language model is given examples of a task within its imput prompt, allowing to adapt new tasks without fine-tuning. Often used with few-shot learning techniques.
Residual stream: In the context of transformer architectures, the residual stream refers to the main information flow that bypasses the ceff-attention and feed-forward layers through residual connections.
Logits: The raw, unnormalized outputs of neural network's final layer, before any activation function (like softmax) is applied. language models, logits typically represent the model's scores for each token in the vocabulary.
Activation patching: An interpretability technique where activations at specific locations in neural network are replaced or modified to observe the effect on the network's output. This helps in understanding the causal role of different parts of the network in producing its final output.
A.2 Proof: absorption decreases SAE loss for hierarchical features
We analyze the effect of a specific form of feature absorption, termed 0-absorption, within Sparse Autoencoder (SAE) framework. We consider two hierarchically related features, f1 and f2 (where f2 fi), and demonstrate that for defined family of encoder and decoder weights parameterized by E [0, 1] (6 = 0 corresponds to no absorption, and corresponds to full absorption):
1. Perfect reconstruction of inputs composed of these features is maintained across all values of 5.
2. The sparsity loss component attributable these features is decreasing function of 6, provided the child feature fz has a non-zero probability of appearing.
3. Consequently, optimizing for sparsity encourages higher values of ò, i.e., greater absorption.
1. Preliminaries and Assumptions
H1. Dataset and Features Let D be a dataset. We consider set of features J (ff,, faj.
Each feature fi E R is vector with unit norm, lfillz = 1. Features are mutually orthogonal: Ji fg ōzg (Kronecker delta), where 6, = 1 if i and 0 if * j. An activation E IR= in the model's 's residual stream is linear combination of active features: h EEEActiveeammess fz.
H2. Feature Hierarchy and Probabilities We focus on two features fi, /2 E F with a hierarchy f2 f1. This implies that if fa is present in a datapoint, fi must also be present. The probabilities of observing combinations of fi and f2 are:
f2) = P11: Probability of fi and f2 co-occurring (e.g., input is fi + f2).
p(fi, -2) = P1o: Probability of fi occurring without fa (e.g., input is fi).
p(-f1,f2) -Poo: Probability of f2 occurring without fi. By the hierarchy assumption,
Po1 =0.
p(-f1, -f) = Poo: Probability of neither fi nor fo occurring (e.g., input is 0 or some fk. 1,2).
We assume P11 + P1o + Poo 1.
H3. Sparse Autoencoder (SAE) Model The SAE reconstructs an input h as = fh).. The reconstruction is h Waz, where 2 = ReLU(W_h),. No bias terms are used. We,s is the i-th row of W. (encoder vector for latent i), and Wa. is the i-th column of Wa (decoder vector for latent i). We analyze two specific latents, and intended to capture f1 and f2. Other latents zj for j > 2 are assumed to perfectly reconstruct other features ʃj (e.g. Waj = Sj. Was = f5) and do not interact with f1, fa due to orthogonality.
15

---

H4. SAE Loss Function
The total loss is = Liee SCsp where 0. Liee EAND |1 h b sp ELLD Ei |x. Our analysis will focus the contributions of 21; 22 fiee and Cup-
H5. Definition of 6-Absorption We define a specific parameterization for the encoder and decoder weights associated with fi and f2 by parameter [0, I]:
We, f1 5.2
We,3 f2
Wa,1 f1
Wa f2 + 8f1
0 represents no absorption, while represents full absorption.
2. Proposition 1: Perfect Reconstruction under 5-Absorption
For any 8 [0, 1), and for inputs h consisting only of f1, f2 or 0, the reconstruction h Wa,121 22 perfectly reconstructs h, i.e., the reconstruction loss component L due these features is O.
Proof
We consider the possible input types based on f1, f2:
Case 1: fi (only parent feature fi present). The latent activations are:
21 ReLU h)) = ReLU((fi - 6 fi)
ReLU([i f1 -15552 f1)
ReLU(( - 5-0) =1 (by HI)
22 ReLU(( h) -ReLU(f2 fi) ReLU(() 0 (by HI)
The reconstruction is:
h h= 21 Wa,1 Wa. 1 f1 + (f2 + 8f))
Thus, h = h.
Case 2: = fi f2 (both parent fi and child f2 are present).
The latent activations are:
21 ReLU(We,, h) = ReLU(([i - 8f2) (fi + f2))
ReLU((i f1 fi f2 - f- f2)
ReLU(1 + 0-0 5) RRLUU[ 5) (by H1)
Since E [0, 0, so 8.
= Rel.UU h) = ReLUUS2 (/1 + f2))
ReLUU(f fi fz f2)
= ReLU(( 11 =1 (by H1)
The reconstruction is:
21 Wa.1 + 22 = (1- 5) -1-0 +f++
(1 8) fo 8f f =f1
Thus, h = h.
Case 3: (neither /1 nor f2 is present).
2 = ReLU((f1 - 0f2) -0) 0
22 ReLU(fa
The reconstruction is: Thus, h h.
i = 0 f1 + 0- (f2 + 8f1) = 0
16

---

Case 4: h fa (only child feature f2 is present). This case is disallowed by assumption H2 (Po1 = 0), as f2 C. fi implies fi must be present if fz is.
In all permissible cases, h h 0, |// h112 O. Therefore, the reconstruction loss component due to f1, f2. denoted Cll.2) is O for any å E [0, 1].
3. Proposition 2: Sparsity Loss under 6-Absorption
The expected sparsity loss contribution from latents 21 and 22, denoted ( EnDD [|=1| l + |22l]. is given by:
=2 5) Pio
Furthermore, its derivative with respect
(1.2)
sp
PP1
d8
Proof We calculate the sum of absolute latent activations |21| + |22| for each case from Proposition 1 and weight them by their probabilities (H2):
If h fi + f2 (probability P11): 21
22 1. Since o E [0, 1]. 11 = 8. 222 = 1. Thus, |21| + |22 = (1 - 8) + = - 8.
5 0, so
If h = fi (probability P1o): 21 = 1, 22 = 0. Thus, |21| + |22| = 1 + 0 1.
If h 0 (probability Poo. neither nor fa present): 21 0, 22 0. Thus, |21 + +|22|
0 + 0 O.
The case corresponding to Po1 does not occur.
The expected sparsity loss from 21, 22 is:
((1.2) P11 (2- -8) + P10 + Poo
P11 (2 8) + P10
Taking the derivative with respect to 8:
dc((122
(2p11 6p11 + P10)
d6
d6
Since Pıı and Pio are constants with respect to
d dc
-P11
d8
Corollary: Increasing Absorption Decreases Sparsity Loss
If Pı1 > 0 (i... the child feature fa co-occurs with fi with non-zero probability), then increasing 5 strictly decreases (1.2)
dc db = -PI- If Pt1 > 0. then -PI1 < negative derivative
Proof From Proposition 2,
implies that c(1.2) decreasing function of for d E [0, 1]. The minimum value of c(1.2) over this interval occurs at (full absorption), yielding c/p (5 1) pn - + P10 P11 + P10-
The maximum value occurs at (no absorption), yielding c. c(1.2) 0) = pn (2 - 0) + P1o 2p11 + P10-
17

---

5. Conclusion
Given the specified ó-absorption mechanism for an SAE handling two hierarchical features fı, f2 (where f2 fi):
1. Perfect reconstruction of inputs composed of fi and f2 is maintained irrespective of the degree of absorption 8. Thus, c(2.2) is unaffected by 8.
2. The sparsity loss component c(2.2) associated with these features is P11 -6) + P1o-
3. If P11 0, the total loss L (focusing the components related to f1, f2) decreases 6 increases because c(3:2) is constant (zero) and c(.22) decreases.
4. Therefore, optimization process like gradient descent, when minimizing the total loss L = Lree + Xp (where À > 0), will favor increasing 6 towards 1, thereby promoting feature absorption for these hierarchically related features, assuming the SAE learns or is constrained to these forms of W. and Wa.
This formalizes the argument that, under the given conditions and definitions, absorption is mechanism that can reduce SAE loss by improving sparsity without harming reconstruction for hierarchical features.
A.3 Extended toy model experiments
In this section explore further variants absorption in toy models. We use the same setting as main toy model experiment, with four mutually-orthogonal true features, and train an SAE with four latents. Each true feature f E TR500 Unless otherwise stated, every time feature fires feature must also fire, but feature is allowed to fire on its own as well. This is to simulate hierarchal features such our example "starts with S" and "short" features, where every time the "short" feature fires we expect "starts with S" must also fire since "short" starts with "S", but "starts with S" can fire on its own as well. Feature and 3 fully independent. All features fire with magnitude 1,0 and variance 0.0 unless otherwise stated.
Magnitude variance causes partial absorption In main toy model experiment, each true feature fires with magnitude exactly 1.0. This is not very realistic, though likely there will be some variance in feature firing magnitudes in real LLMs. We simulate this by adding variance of 0.1 the firing magnitude of feature 0, so the relative magnitudes feature and are no longer fixed. We show the plots of cosine similarity between SAE encoder and decoder in Figure [0 Here, still the same absorption pattern in the SAE encoder and decoder with the latent 3 encoder containing a negative component of feature and the latent 0 decoder merging features and 1. We show some
sample true feature firings and corresponding SAE latent activations in Table 2
Cos sim with true features (feat co-occurs w/feat feat magnitude varies)
SAE encoder
SAE decoder
SAE Latent
SAE Latent
0.5
COS sim
0.0
-0.5
2
3
True feature
Truc feature
Figure 10: SAE encoder and decoder with true features. The firing magnitude of feature 0 has variance 0.1, while the remaining features fire with variance 0.0. When there is variance in the firing magnitudes of parent and child features, we still see an absorption pattern the SAE encoder and decoder with the latent 3 encoder containing negative component of feature 1, and the latent O decoder merging features and 1.
We see that now the SAE latent tracking feature 0 still fires when the true values of features 0 and 1 are both 1.0, but very weakly. However, the magnitude of feature drops down 0.75, then the feature O latent fully turns off.
18

---

Table Sample true feature firings and corresponding SAE latent activations. Feature 1 only fires feature 0 fires. Feature 0 has variance of 0.1 in its firing magnitude, while the other feature have no variance in their firing magnitude.
TRUE FEATURES
SAE LATENT ACTS
1.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
1.00
0.00
0.00
1.27
0.00
0.00
0.22
0.90
1.00
0.00
0.00
1.27
0.00
0.00
0.12
0.75
1.00
0.00
0.00
1.26
0.00
0.00
0.00
We call this phenonemon partial absorption. In partial absorption, there's co-occurrence between dense and sparse feature, and the sparse feature absorbs the direction of the dense feature. However, the SAE latent tracking the dense feature still fires when both the dense and sparse feature are active, only very weakly. If the magnitude of the dense feature drops below some threshold, it stops firing entirely.
Feature absorption is an optimal strategy for minimizing the LI loss and maximizing sparsity. However, when SAE absorbs latent into another, the absorbing latent loses the ability to modulate the magnitudes of the underlying features relative to each other. The SAE can address this by firing the latent tracking the dense feature "correction" add back some of the dense feature direction into the reconstruction. Since the dense feature latent is firing weakly, still has lower L1 loss than if the SAE fully separated out the features into their own latents.
Imperfect co-occurrence can still lead to partial absorption Next, we test what will happen feature is more likely fire if feature 0 active, but can still fire without feature 0. We up feature to fire with feature 0 95% of the time, but 5% of the time it can fire on its own. For this experiment, all features fire with magnitude 1.0 and 0 variance. We show the cosine similarities of the SAE encoder and decoder with true features in Figure 11 Some sample feature firings and corresponding SAE activations are shown in Table
Cos sim with true features (feat partially co-occurs w/feat 0)
SAE encoder
SAE decoder
1.0
SAE Latent
SAE Latent
0.5
COs sim
0.0
2
-05
1.0
0
3
True feature
True feature
Figure 11: SAE encoder and decoder with true features. Feature fires with feature 0 95% of the time, but 5% of the time feature allowed fire on its own. Even though there is not perfect co-occurrence between features O and 1, we still see the SAE encoder and decoder learn a weak absorption pattern, with the decoder latent for feature absorbing part of feature 0, and the encoder latent for feature 0 including a negative component of feature 1.
We see signs of partial absorption here as well. We see the same absorption pattern in the SAE encoder and decoder we saw in our other absorption examples, although less severe than the previous examples. We also see in the sample firing patterns that when both feature 0 and fire together, the latent tracking feature fires with noticeably lower magnitude than when feature 0 fires on its own. Here, even though the co-occurrence between features and is not perfect, we still see partial absorption.
Assorption also affects TopK SAFs So far, we have only shown feature absorption occurring with standard LI SAEs. Next, we examine how other absorption affects other architectures using
19

---

Table Sample feature values and corresponding SAE activations. Feature can only fire if feature 0 is active 95% of the time, but 5% of the time feature 1 can fire on its own. We see signs of partial absorption, where the latent tracking feature 0 fires noticeably more weakly if feature active.
TRUE FEATURES
SAE LATENT ACTS
1.00
0.00
0.00
0.00
0.00
0.00
0.00
1.00
1.00
1.00
0.00
0.00
0.00
1.05
0.00
0.67
0.00
1.00
0.00
0.00
0.00
0.95
0.00
0.00
a batch topk SAE [3. Batch topk SAEs are an improved version of topk SAEs [10] where the top k B latents used reconstruct the SAE input, where B is the batch size. As the topk function enforces sparsity. there is no additional L1 loss term.
Topk SAEs are harder to use for very small toy models like our 4-feature toy model above, since if the l is too large relative to the size of the SAE the SAE will learn correct features. To address this, we use slightly larger toy model with 12 mutually orthogonal true features. All features fire independently with probability 0.15, except for the first 2 features. Feature 0 is the parent feature in our hierarchy, and fires with probability 0.4. Feature is the child feature, and fires with probability 0.6 only if feature fires, but never fires feature does not fire. All features fire with magnitude 1.0. We train batch topk SAE with k 2. We show the cosine similarities of the SAE encoder and decoder with true features in Figure 12
Cos sim with true features (feat co-occurs w/feat O, batch topk SAE)
SAE encoder
SAE decoder
0.75
0.50
11 8 Latent
SAE Latent
0.25
COS sim
0.00
-025
11 8
00.50
-0.75
-1.00
10
10
True feature
True feature
Figure 12: SAE encoder and decoder with true features for batch topk SAE with k=12. Feature is only allowed to fire feature fires. We still see very clear absorption pattern between the latents tracking features 0 and despite the lack of Ll loss.
We still see clear absorption pattern between the latents tracking features 0 and despite the lack of LI loss. Absorption increases sparsity, which allows the topk SAE to have better reconstruction loss given k, and is thus what the SAE learns.
A.4 Ablation algorithm
A.5 How good is Gemma-2 on character identification tasks?
We evaluate how well can Gemma-2-22 identify the first letter or all the letters token (spelling the full token). We evaluate the accuracy of the model on all tokens in the LR probe validation set with prompt containing 10 in-context examples selected at random from the full vocabulary. Our results shown in Figure 13
We see that performance on the first-letter identification task is high throughout token length, while the full-wor spelling performance decreases the length of the token increases.
20

---

Algorithm SAE Latent Ablation
Insert SAE in model computation, including error term Define scalar metric on the model's output distribution Calculate baseline metric value for a test prompt for each token of interest do for each SAE latent do
Set the SAE latent activation to Recalculate the metric
Compute ablation effect (baseline new metric)
Reset the SAE latents to its original value
end for end for=0
Spelling accuracy by token length
1.0
0.8
Accuracy
0.6
0.4
Task
0.2
First Letter
Spelling
0.0
12
Token Length
Figure 13: Baseline performance for Gemma-2-2B on first-letter identification and full-token spelling by token length.
A.6 Intervening on the first letter
If the model is using the identified SAE latents for predicting the first letter we should also be able to change what first letter it predicts just by changing the activations. For this experiment SAE latents most cosine similar with the LR probe for the true first letter and for a new randomly selected letter. We take the intermediate activations of Gemma-2-2B in the residual stream and encode them using the SAE. Then we zero out the activation of the SAE latent associated with the original letter and change the activation of the SAE latent associated with the new letter into the average activation it has on tokens starting with this new letter.
the
Editing works better with latents from the narrower 16k SAE compared the 65k, with the best LOs in the 75-150 range. This corresponds to the observed pattern of these SAE latents having higher F1 scores for classification. We report the results in Figure 14 The best SAES on the layers 7-9 can achieve a substantial replacement, but note that the averages hide variance across individual tokens, where some get edited completely and others get unaffected. The edit success also varies based on the true first letter and the random new letter; for illustration we show a breakdown by letter for two specific SAEs in layer in Figure 15
A.7 Probe cosine similarity vs k=1 sparse probing
The first step when scarching for SAE latent that acts as first-letter classifier involves scarching for SAE latent which best acts as a classifier. In Figure we achieve this by first training a LR probe on the first-letter task and using cosine similarity between that probe and the SAE encoder to find the best latent for the first-letter task. We also investigated using k-sparse probing with k=1 to select the best SAE latent instead. This involves training a linear probe with LI loss and selecting the latent with the highest positive weight from the probe.
We find that both k=1 sparse probing yield nearly identical results, seen in Figures T6 and 17 Additionally Figure 18 shows the cosine similarity the LR probe with each SAE latent by letter for the canonical Gemma Scope layer 0 16k width SAE. In most cases there is obvious aligeed
21

---

Layer
Edit
Score
0.6
0.8
16kk
18C
Thue First Lette Probabiltt Decraze
Ne Lefoer Probssbllty Increase
Decrase
Ne-er first Letter Probahiitt Increase
0.5
0.5
0.5
0.4
8.5
0.4
0.3
9.2
Tue
100 200
100
200
LO
LG
(a) Comparing success in editing out the true first letter and making the model predict a randomly selected new letter across layers 0-9 16k and 65k Gemma Scope SAEs.
(b) Comparing the edit success with the top SAB latent across all LOs for 16k and 65k widths across layers 0-9. best performance seems be occurring for LO between 75 and 150.
Figure 14: Comparison Edit success by Layer and LO
Layer 7: First Letter Edit Out Score vs Original Letter
Layer 7: First Letter Edit In Score vs New Letter
0.8
First Probabilty Decrease
0.8
New First Letter Probability Increase
0.6
SAE with and LD
SAE widh and
16k_131
65k_107 107
0.4
65k 107
0.2
0.0
j p True First Letter
New Letter
Figure 15: Comparing the edit success broken down by the letter at layer 7 for two SAEs; SAE width 16,000 and LO of 137 and SAE width 65,000 and LO 107. For each original letter we draw sample of 100 tokens and average the decrease in probability of the correct first letter and increase in probability of new random letter.
latent. Likely any reasonable method of latent selection will find the same latent for these cases. We thus decided to use cosine similarity between the SAE encoder and LR probe selection criteria for single SAE latents as this a simpler metric and less computationally intensive to compute.
A.8 Precision, recall, and F1 score for the first-letter task
We evaluated precision, recall, and score for the first-letter classification task, and found that the precision and recall vary depending on the LO of the SAE. Low LO SAEs learn high precision, low recall latents, while high LO SAEs learn low precision, high recall latents. These results are shown in Figure [9 We thus chose to F1 score recall as many if the SAEs we tested have extreme values either precision or recall.
metric in this paper to balance precision and
While it may appear that there is optimal LO from looking at aggregate statistics across letter, we find that breaking down the vs L.0 plot by letter reveals that the optimal appears different for different letters, with low frequency letters like actually having the best Fl score the lowest LO,
22

---

Firsi-letter SAB P1 score L.
First letler SAE El
SAE witth
0.7
0.7
65L
16k
0.5
Mean 11
0.5
Moan
0.3
400
60
100
150
200
800
350
1.0
(a) Mean Fl score first-letter classification task using top SAE encoder latent by cosine similarity with the LR probe LO all Gemma Scope SAEs layers 0-9.
(b) Mean Fl score on first-letter classification task using k=1 sparse probing to select the SAE latent for the first-letter classification task LO for all Gemma Scope SAEs layers 0-9.
Figure 16: Comparison of LR probe cosine similarity and k=1 sparse probing vs 10
First letter SAE Fl score vs Layer
First-letter SAE fl score vs Layer
SAF wish
SAE width
0.8
16k
65%
Probe
Mean fl
Prohe
H1
0.4
2
6
Layer
(a) Mean Fl score on first-letter classification task using top SAE encoder latent by cosine similarity with the LR probe vs layer all Gemma Scope SAEs layers 0-9.
(b) Mcan F1 score on first-letter classification task using k=1 sparse probing to select the SAE latent for the first-letter classification task vs layer for all Gemma Scope SAEs layers 0-9.
Figure 17: Comparison of LR probe cosine similarity and k=1 sparse probing vs layer
while other letters instead have an optimal LO around 30-50. Figure shows these results broken down by letter.
A.9 SAE training
We train SAES the first 8 layers of Qwen2 0.5B 332 and Llama 3.2 IB 6 using the SAELens library [5]. The SAEs are all trained with identical hyperparameters of L1 coefficient 2.5 and 500M tokens. The Qwen2 0.5B SAEs all have LO between 25 and 50 and explained variance between 0.77 and 0.83. The Llama 3.2 1B SAES have LO between 27 and 110, and explained variance between 0.74 and 0.89. We use single 40gb Nvidia A100 GPU for training each SAE.
A.10 Metric choice for ablation studies
To determine the causal effect of SAE latents on the first-letter identification task, we use a metric, m. which measures the logit of the correct letter minus the mean logit of incorrect letters. Our metric is defined below, where g refers to the final token logits, L is the set of uppercase letters, and y is the uppercase letter that is the correct starting letter:
m = =gly]
Σ dirl
(E(L\
23

---

Figure 18: Decoder cosine similarities with the LR probe by letter, Gemma Scope 16k layer 10=105. Most letters have one or two obvious SAE latents which align with the probe.
First letter SAE precisson L0
First-letted SAE recall LO
1.0
0..8
10R
0.66
Mean recall
0.0
0.4
SAE witth
0.2
16k
65k
100
1.0
(a) Mean precision on first-letter classification task vs LO for all Gemma Scope SAEs layers 0-9. Latents selected via k=1 sparse probing
(b) Mean recall on first-letter classification task vs LO for all Gemma Scope SAEs layers 0-9. Latents selected via k= sparse probing
Figure 19: Comparison of precision and recall vs 10
This metric is chosen to detect changes in the confidence of the model in predicting the correct letter relative the mean reference class of other letters. This should capture changes in the model's confidence in predicting the correct logit.
This is not the only metric that could be chosen, and an argument can be made that we should subtract the max of all incorrect letter logits rather than the mean of all incorrect letter logits. The max form of this version of the metric is shown below:
24

---

22
Da
Figure 20: Fl vs LO by letter. SAE latents are picked using k=l sparse probing.
M Mmaz = glvl
max g|{ IE (E(a)
This second form using a max can also account for the case where the logits of the model shift from being confident in the correct answer instead being confident in an incorrect answer while leaving the logits of the correct answer the same.
Mcan absorption rite: mean-logit metric
Mean absorption rite:
max-logit metric
SAE width
0.35
SAE
0.20
16L
16k
Meon ubscrpttin tale
6dk
abserptio
0.15
0.25
0.20
0.10
+0
0.15
0.10
0.05
0.05
150
1.0
1.0
(a) Absorption rate using the mean version of the ab- sorption metric. Gemma Scope layers 0-3.
(b) Absorption rate using the max version of the ab- sorption metric, Gemma Scope layers 0-3.
Figure 21: Comparison of absorption rates using the max and mean versions of the absorption metric.
In practice, we expect that ablating absorbing latent should cause the model to become less confident in the correct answer, so the difference between these two forms of the metric should yield similar results.
25

---

We calculate the mean absorption rate for Gemma Scope SAEs layers 0-3 in Figure using both versions of this metric. The overall shape of the curve nearly identical between these two choices of metrics. The mean version of the metric, which is used in this paper, results in slightly more conservative estimate of absorption rate.
We consider our absorption score to be rough estimate of the true absorption rate and thus consider either the mean the max version of the logit diff' metric be valid for evaluating absorption.
26

---

A.11 Choice of thresholds for absorption metric
Our absorption metric makes use of several thresholds. For feature splitting, we use a threshold of 0.03 the Fl score jump moving from k a case of absorption, we require cosine similarity with the LR probe above 0.025, and a clear largest ablation effect of 1.0 more than then next highest latent. In this section, we discuss the intuition behind these thresholds. In all cases, these thresholds are just the rough midpoint of ranges of values that achieve similar results, and that we feel are reasonable default values. We also refer readers to our online feature absorption explorer to gain similar intuition as to what typical ranges of these values look like.
k- 1 in sparse probing. To classify latent as
Feature splitting threshold We detect interpretable feature splitting by noting how large a jump in FI-score we gain moving from A: = n to k = n + 1 in sparse probing. Figure 8a demonstrates typical example of how F1 score increases for interpretable feature splitting vs feature absorption. In the left side of the figure, we see that moving from &o = 1 to & = 2 there is a Fl-score jump of around 0.08, while each increase after that is less than 0.01. For the figure on the right, where no interpretable feature splitting occurs, all F1-score jumps are less than 0.01. This plot is a very typical illustration of detecting feature splitting via k-sparse probing. Any threshold between 0.01 and 0.05 does good job of detecting feature splitting. We set the threshold to 0.03 to be in the middle of this range.
LR probe cosine similarity threshold We use threshold of 0.025 on the cosine similarity of the firing latent with the LR probe as part of the metric, ensuring that any latent we classify as absorption must contain some component of the probe direction. This threshold is mainly a cheap way to filter out latents that are obviously not probe-aligned so we can avoid running the more expensive ablation experiments. Nearly any threshold above O and below 0.05 should work identically well for this purpose. We choose 0.025 to be in the middle of this range. For most cases of absorption we detect, the ubsorbing latent has a probe cosine similarity of around 0.05 0.15. Figure 6bb demonstrates very typical case of cosine similarity between an absorbing latent and the LR probe, showing cosine similarity of 0.12.
A Absorption effect threshold As part of our metric, we assert that any latent we classify as absorption must have the highest ablation effect of all latents, and that lead must be by at least 1.0. As the main goal of the metric in our paper is establishing definitively that absorption exists and affects real LLM SAEs, we focus on the most obvious cases of absorption where a single latent absorbs the parent feature direction in a single activation. This threshold thus serves two purposes: first, to establish the latent has strong ablation effect, and second, the establish that the effect is dominant over other latents. The metric is not particularly sensitive to the exact choice of threshold we set here, as any dominant ablation effect above around 0.5 is already quite strong. Figure shows a typical ablation effect for a dominant absorbing latent, and has an ablation effect above 6, while all other latents have ablation effects well below 0.5. We pick threshold of 1.0 here as its a clean number that is well within the range that will work as a threshold. Any threshold between 0.5 and 3.0 should work just well.
A.12 Causal interventions and absorption
In this work, we rely on causal interventions like ablation experiments to verify that SAE latents have a causal impact model behavior. In these experiments for spelling tasks, we set up an ICL prompt to elicit spelling information from the model, for instance the ICL prompt below:
tartan has the first letter: T mirth has the first letter: M dog has the first letter:
In this ICL prompt, we would apply an SAE and train LR probes on the _ddog token position, and expect that the model will output the token _D. When we intervene on the _dog, we can track the causal changes to model outputs by applying a metric the output logits, e.g. checking how our intervention increases or decreases the _D logit relative to other letters.
27

---

We use these interventions as part of our absorption metric to ensure that when we claim that "absorption" occurring, we verify that the absorbing latent has causal impact on model outputs. This is stronger evidence than only noting cosine similarity between the absorbing latent, but this means that our absorption metric cannot classify absorption at later model layers.
During a LLM forward pass, the model first collects relevant information on a token in that token position, and attention heads then move relevant information from earlier tokens later tokens If we assess ablation effect at layers after which model attention has already pulled relevant information from the subject tokens into the final output token, the ablation effect will be O. For Gemma 2 2B on the first-letter spelling task, we find this movement of first-letter spelling information occurs around layer 18.
M 22
Figure 22
shows an activation patching experiment 22 on a sample first-letter spelling prompt. In this experiment, we that near layer 18 the model moves first-letter spelling information from the subject token to the prediction token.
Resid Activation Patching
1.0
0.8
Layer
11
0.8
14
15
16
17
18
19
20
0.0
clech
Robort
Token
Residual stream attribution patching for sample first-letter spelling prompt. Gemma 2B. After around layer 18, model attention moves the relevant spelling information from the source token to the prediction location.
Figure 22:
As result, feature absorption metric will not function past layer 18 in Gemma 2 2B, and thus focus on layers 0- for our analysis of feature absorption. We believe that feature absorption is still occurring in SAES past layer 18, but we lose the ability to make causal claims that the absorbing latents are used by the model to make predictions. Given that this paper is trying to highlight the existence of feature absorption, we felt it is more important to have metric which is robust and has the backing of causal analysis but which cannot be used at all model layers. Future work may make a different trade-off and choose feature absorption metric which can work at all model layers, for instance relying only on cosine similarity between absorbing latents and a LR probe to determine absorption. We describe variations the metric that facilitate this in Appendix A.13 below.
A.13 Alternate formulations of the absorption metric
There are a few variations to the metric that can be made to make it more flexible so that it can be applied at all layers of the model, and can be tweaked to detect partial absorption as well. As described above, we did not use this variation as we felt it is more important to demonstrate definitely and conservatively that absorption occurs this paper. However, we describe the changes that can be made to the metric follows:
replace ablation study with LR probe projection in reconstruction The metric in our paper uses an integrated -egggrattddgrddentt ablation study to be absolutely certain that an absorbing feature is causally
28

---

responsible for model behavior. We use this in the paper as our goal is to establish with certainty that feature absorption exists in real models, but it has the following drawbacks:
We need to be able to consistently prompt the model to perform the task, outputting a single token
We cannot evaluate final model layers after attention moves task-relevant information to the final token
Ablation studies are slow, which makes the absorption metric expensive to calculate.
The ablation study can be replaced instead with a threshold on the portion of the logistic regression (LR) probe direction an absorbing latent contributes to the residual stream. To do this, project all firing latents against the LR probe direction d,, well as project the input activation against the probe. We require that latent must contribute at least Te portion of the probe projection the reconstruction. So, in order for to be considered an absorbing latent, the following must be true:
ā: dp
dp
where â is the reconstruction component of latent (the encoder activation of latent times the decoder vector for l).
Allow multiple absorbing latents The metric in the paper requires one dominant absorbing latent for simplicity, but possible in theory to have two or more absorbing latents firing together performing absorption. We can modify the metric to take this into account by allowing up to N latents firing together contribute up to Te together.
Thus, the threshold from change above now becomes the following, where i, refers to the absorbing latent with the nth largest contribution to the LR probe direction in the reconstruction:
N Te
dp
dp
7
We do not do this in the paper it requires another hyper-parametts which is hard to set in principled way, but should result a less conservative estimate of absorption.
Allow main latent(s) to fire weakly instead of being fully turned off The metric in the paper requires that the main latent(s) for task all be fully disabled to identify case of absorption. However, in partial absorption, as explored our section on toy models, the main latent does not always fully turn off but fires very weakly instead. We can adjust the metric take this into account by relaxing the requirement that the main latents are fully disabled and instead allow them to fire weakly. Similar change 1, we define a threshold 7m as maximum contribution to the probe direction in the reconstructed activation that comes from each main latent wm Thus, if we have M main latents, in order be classified as absorption the following must be satisfied:
M
it dp
Tm
dp
m
In the metric defined in the paper, T. O, meaning the main latents must be fully turned off to count as absorption.
This final change, which detects partial absorption, may be preferable to get sense of an overall level of absorption present in an SAE. However, if the goal to determine whether absorption affects the SAE's ability to act classifier, then requiring that the main SAE latent be fully turned off to be called absorption preferable, so we do claim that either version of the metric superior in all cases.
29

---

A.14 Additional plots
In this section, we include additional plots that are too large fit in the main body of the paper.
Figure 23: Absorption rate 1.0 by letter, layers 0-17, We wide variance in which letters absorbed by which SAEs.
30

---

A.15 Feature dashboards
We include feature dashboard screenshots from Neuronpedia for some prominent latents mentioned in this work. Figure 24 shows dashboard for Gemmascope layer 3, latent 1085, which is a token- aligned latent firing on variations of the word _short and we find absorbs the "starts with S direction. Figure 25 shows latent 6510) from the same layer which should be the main "starts with S latent.
C2B01-2-23
HIBLL
FIEL DO
Asoeddyyieral Syheriie
-0.40
the resite stard compaign followe,
Tomun Exparo, The soard versaat las Sealowig
loss 35500 totee with Geeert ters are tarorice vol regulantos, chancee fall
Figure 24: Neuronpedia dashboard for Gemma Scope layer 3, latent 1085. This latent is a token- aligned latent for short tokens. This latent absorbs the "starts with S direction.
GENMA-2-28
D-GEMASCOPEERRSS-IO
SAB
6510
210151
G0
A
DENSITY 3.722N
Low
-0.73
552
0.95
Tertenete
-0.07
0:35
Test activation with custom text.
Stacked
Snippet
Full
Show Breaks
Hide Breaks
(cccpp)s
Maate aryy/tent
classical
accagh1)
Figure 25: Neuronpedia dashboard for Gemma Scope layer 3, latent 6510. This latent should be the main "starts with S" latent.
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made abstract and introduction include demonstrating absorption
in theory, as explored Section 3 and experimentally as explored Section 5.3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification: A proof showing that absorption is caused by hierarchical features in simplified setting is shown Appendix A.2 including all assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All detail needed to reproduce experiments is contained in Section + All experiments are run using open-source models and open-source datasets. We also include code necessary to reproduce our results along with the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
33

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All code and data necessary to reproduce our results is included in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public/guides wwwee//ooiooeiiooooooooooooooooa
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results, See the NeurIPS code and data submission guidelines (https htt///ooeeSumieseee
//nip.cc. wttpcccccppcccccceeceec httpa//gggideeeeeeuuubbiiiieeee wwm.iisssooooiiooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
with further details of our absorption metric described Section 5.3 We also include further details on SAE training in Section A.9
Justification: Details of experiments are explained in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our results are mainly run existing open-source SAEs, so while we report results on these SAEs, we do not have access to multiple seeds for each SAE as these are not published. For the SAFs we train ourselves, our goal is only to show that absorption is non-zero for these SAE architectures rather than making a claim about a specific level of absorption in general.
Guidelines:
The answer NA means that the paper does not include experiments.
34

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We use single 40gb Nvidia A100 GPU for all experiments, as described in Appendix A.9
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work follows the ethics guidelines for NeurIPS.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work about a technical problem with sparse autoencoders, and is not applicable to society large.
35

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer:
Justification:
Our work does not have any misuse risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and code are open-source and follow the license of the original work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release further assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not perform any crowdsourcing
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
We did no work with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used to conduct experiments this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
38


Reviews:
[{"review_id": "S4nFd8NRre", "reviewer": "Reviewer_Bisd", "summary": "Sparse autoencoders (SAEs) are an interpretability tool that aim to decompose language model activations into a linear combination of individually-interpretable features. SAEs allow for a large set of possible features (the “dictionary”) but require that any individual activation may only use a small number of these features (a sparsity constraint), and hypothesize that this will cause the learned features to be interpretable.\n\nThis paper presents an obstacle to the interpretability property: when in reality there are hierarchical features, the optimization objective for SAEs encourages “feature absorption”: the coarser feature learns not to fire when the more fine-grained feature fires, so as to improve sparsity. Specifically, suppose there are two features A and B where if B fires then A should fire. Then instead of learning [A] and [B] as separate features, the SAE can learn [A and not B] and [B] as separate features. Then on inputs where both A and B are present, the SAE only fires one feature instead of two, doing better on sparsity.\n\nThe paper demonstrates this in a toy setting where the ground truth can be controlled, and then studies it in practice on the “first letter task” (“dog” → “d” and so on). For this task, there are often a small number of SAE latents capturing the first letter information, but that then fail to fire on some common tokens where the interpretation would suggest they should fire. For example, the “starts with s” feature fails to fire on the “short” token, even though a linear probe can perform the “starts with s” task well. Ablation experiments demonstrate that this information is instead in a feature that fires on “short” only, suggesting that the “short” feature has absorbed the “starts with s” feature, in accordance with theory.\n\nThe authors define a feature absorption metric in this task, and find that lots of SAEs have significant feature absorption on this task, with rates particularly high for wider and sparser SAEs.", "questions": "How much do you expect feature absorption to be a problem in practice for typical applications of interpretability? Can you quantify this in some way?", "limitations": "Mostly yes, though I think they should also acknowledge the limitation of focusing just on the first-letter task.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n1. The core problem with SAEs identified in this paper is very intuitive and seems like it should apply quite widely – yet to my knowledge it was not known in the literature. (It may have been known in the dictionary learning or sparse coding communities; I am not familiar with that literature.)\n2. The authors have a good blend of toy models and concrete examples that build intuition and demonstrate the mechanism of feature absorption, along with a larger scale quantitative study that quantifies how large a problem this is in practice.\n\n## Weaknesses\n\n1. As the authors acknowledge, the feature absorption metric is likely unreliable in a variety of ways. For example, it relies on ablation on the original token, which will often be ineffective in later model layers since the information will already have been moved by an attention head to the final token position. In addition the metric has multiple hyperparameters that are judgment calls; changing these would presumably affect the results a decent amount, and there isn’t a particular justification for the choices made.\n2. The empirical results are almost entirely based on the first-letter task, which makes it hard to know how large the feature absorption problem will be in general. It is possible that the more abstract features that are more often the goal of interpretability research would not have the same issue, though the theoretical / conceptual argument remains strong so I consider this to be a fairly minor weakness.\n3. The paper identifies a problem with SAEs but doesn’t propose any methods for solving that problem, limiting its impact. (However, SAEs are popular enough that pointing out a significant problem with SAEs is still a very valuable contribution, even if it doesn’t come with a solution.)\n\n## Summary\n\nOverall, this is a great paper that changed my views about how promising SAEs are. Feature absorption is an obvious problem with SAEs once pointed out, but I hadn’t thought about it before. I wish there was more investigation and evidence about how large a problem is – for example, perhaps one could brainstorm ten different pairs of hierarchical features, and manually check for how many of them feature absorption has happened. But even without this quantification it’s a valuable paper that should be accepted.", "was_revised": true, "final_justification": "I stand by the summary from my original review:\n\n> Overall, this is a great paper that changed my views about how promising SAEs are. Feature absorption is an obvious problem with SAEs once pointed out, but I hadn’t thought about it before. I wish there was more investigation and evidence about how large a problem is – for example, perhaps one could brainstorm ten different pairs of hierarchical features, and manually check for how many of them feature absorption has happened. But even without this quantification it’s a valuable paper that should be accepted.", "rebuttal": "We thank the reviewer for their thorough reading of the paper and for summarizing the core ideas so well. We will add the focus on first-letter to the limitations section for the CR version of the paper. We address the question below:\n\n> ​​How much do you expect feature absorption to be a problem in practice for typical applications of interpretability? Can you quantify this in some way?\n\nWhile it is difficult to quantify exactly how big of a problem absorption is, qualitatively we have found it to be a significant barrier to using SAEs for most interpretability tasks we have tried to use them for. We list a few tasks we have personally encountered, and how absorption has been problematic below:\n\n### SAE latent ablation\nOne common task in interpretability is to detect how much a LLM relies on a given concept when performing a task. For instance, we may want to detect how much the LLM is relying on gender bias when performing a classification task. A natural way to use a SAE for this is to find a latent representing “male” or “female” and force its activation to 0 when running the model, and then record how this changes model outputs. However, in practice we find this almost never works, we suspect due to feature absorption. Likely the “gender” direction is encoded into more specific latents, e.g. for a specific name like “Lucy” or “George” mentioned in the prompt, and thus ablating the more general “gender” SAE latent has little or no effect on the model output, even if the model is using the underlying concept.\n\n### Feature attribution\nAnother common task is to use feature attribution to understand which concepts a model is using internally in order to perform a task. For instance, we may want to know how a LLM knows that “London” is located in “England”. We can use an SAE for this, inserting the SAE into the forward pass of the model and performing backprop on the “ England” token logit to see which SAE latents receive a large attribution score. However, due to feature absorption, this often does not locate the concepts we expect it should. For instance, we may find the SAE has a “London” feature that fires strongly and gets all attribution, and hides the more general parent features like “located in England”, even though the SAE has a latent which appears to be a “located in England” latent. The result is that it becomes hard to tell with certainty what concepts the model is relying on.\n\nIn our experience, we have found knowledge of feature absorption to be essential for any task where we use an SAE. We still find SAEs valuable as a first-pass technique as they still will often surface things we did not expect, but we use them with the knowledge that absorption means often things that should fire may not fire, and to not be surprised by this and conclude the model is not using some concept as a result. We also choose to rely on supervised methods like probes over SAEs when possible, as supervised probes do not suffer from absorption. We have also found it helpful to manually group latents that seem semantically similar together for interpretability tasks, again relying on the knowledge that a latent we expect to fire may not fire, but that we can sometimes get around this by manual grouping."}, {"review_id": "PG3BolHtlY", "reviewer": "Reviewer_Uf4h", "summary": "The paper demonstrates theoretically and empirically a potential mismatch between what we would ideally \"want\" from an SAE decomposition, versus what the objective function of the SAE prefers in reality. \n\nSpecifically, the authors conjecture the existence of the *feature absorption* phenomenon. Suppose there are two features in the data, $G$ (a more \"general\" one) and $S$ (a more \"specific\" one), such that whenever $S$ is present in an input, so is $G$. Let $f_S, f_G$ be the \"true\" feature vectors corresponding to $S, G$ in activation space. The main observation behind absorption is that, in this setup, the SAE objective will prefer to model this situation using two latents $v_G = f_G, v_S= f_S + f_G$ such that, when $G$ is present by itself, $v_G$ will be active in the SAE, and when $S$ (and therefore $G$) is present, *only* $v_S$ will be active. This solution is sparser than allocating a latent for each of $f_G,f_S$ and firing both when $S$ is present.\n\nThe authors show this plays out in a simple toy model, and in a real-world setup where basically $G=$ \"token starts with some letter\" and $S=$ \"token equals a specific token that starts with that letter\". They define a formal measure of absorption and investigate how different SAE parameters and metrics influence its prevalence according to that measure.", "questions": "- Def. of hierarchical features (line 54) is unclear. What does it mean for a feature to \"fire\"? I think there are some implicit assumptions being made here, and it would benefit the paper to make them more explicit.\n- is it really necessary to train the toy model on 100M activations (Section 3, line 87)?\n- plot 6b would perhaps be more interpretable if we sort the x-axis by the y-axis value?\n\nTypos and other minor nitpicks:\n- fig 1 right: \"starts S\" should be \"starts with S\"\n- line 87: should probably use L1 or $\\ell_1$ for both occurrences instead of L1/l1\n- line 103: \"This is exactly the sort of gerrymandered feature firing pattern we saw in real SAEs for the starting letter task\" - we have not seen this yet in the paper! Maybe say \"we will see later in Section X\"\n- line 178: \"We conduct an ablation experiment on the `_short` token\" - might help the reader to remind them of what an \"ablation experiment\" means in the context of this paper, or to altogether give this intervention a more precise name. \"ablation experiment\" in the current paper is used to point to a very specific technical term; but it's also a general term in the ML literature, which may give rise to confusion.", "limitations": "As pointed out, it would be interesting to demonstrate absorption in other settings with more practically interesting hierarchical features.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This paper does a great job at illuminating a non-obvious qualitative phenomenon in interpretability, and explaining and demonstrating it in a quite intuitive and reasonable progression of thought experiments, toy models and real-world experiments. Despite the somewhat constrained real-world setup being studied (first-letter features are perhaps not the ultimate reason we as a field want to develop SAEs), the authors argue very persuasively for the general plausibility of feature absorption and its implications on the use of SAEs for interpretability. The paper is at times rough around the edges, but I'm certain this can be largely overcome by a small revision. \n\nOverall, this is a timely and significant contribution to the SAE line of work.", "was_revised": true, "final_justification": "I maintain my recommendation for acceptance after the rebuttal period.", "rebuttal": "We thank the reviewer for their thorough reading and understanding of the paper, and for the feedback and suggestions. We will fix all typos in the camera-ready version of the paper - thank you for noting these. We address the questions below:\n\n> Def. of hierarchical features (line 54) is unclear. What does it mean for a feature to \"fire\"? I think there are some implicit assumptions being made here, and it would benefit the paper to make them more explicit.\n\nBy “fire” we just mean the feature has a non-zero activation value. Our model is that a feature is either firing, meaning it has a non-zero value, or is not firing and thus has zero activation value. We will clarify this further in the text.\n\n> is it really necessary to train the toy model on 100M activations (Section 3, line 87)?\n\nIt’s almost certainly overkill to train on this many activations for this toy setup, but since the toy model and SAE are both so small, it only takes a few minutes for this to run on CPU.\n\n> plot 6b would perhaps be more interpretable if we sort the x-axis by the y-axis value?\n\nThank you for this feedback. Our goal here was just to convey the rough ranges of cos similarities of SAE latents with the probe, and where the core latents we are interested in fit in. We will experiment with different layouts for this plot for the camera-ready version, as we agree it can be improved."}, {"review_id": "zuPf9Q1o4U", "reviewer": "Reviewer_mChz", "summary": "This paper introduces and systematically analyzes a phenomenon that can arise in sparse autoencoders (SAEs) when interpreting the activation space of large language models (LLMs), termed feature absorption by the authors. When hierarchical features exist in the model (e.g., “starts with S” and “short”), the SAE’s sparsity-driven optimization may cause the representation of higher-level features to be absorbed into more specific latents, resulting in the general latent failing to activate where it intuitively should. The authors propose a metric—absorption rate—to quantify this effect, and empirically demonstrate its prevalence across various models and layers. This work offers a new perspective and raises important questions about the reliability and applicability of SAEs in interpretability tasks.", "questions": "- Any thoughts on when feature absorption is more likely to occur, and whether there are ways to mitigate it? For example, are certain types of features or training settings more susceptible to this effect?\n\n- When multiple features with overlapping semantics co-occur, stronger latents might dominate weaker ones even without true absorption. Clarifying how such dominance is distinguished from actual absorption would strengthen the analysis.\n\n- In my opinion, SAEs may serve as a form of knowledge compression, which can introduce ambiguity. While the resulting features are more interpretable than individual neurons, activation-based features descriptions might still underrepresent a feature’s full role—especially in cases where features reflect different functional categories, such as input features, which capture patterns within the model’s input, and output features, whose main role is to directly influence the tokens the model generates.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a novel and clearly defined phenomenon—feature absorption—that highlights how sparsity objectives in SAEs can interfere with the reliable representation of general features in LLMs.\n\n- The empirical study is well-executed across multiple models and layers, including Gemma, Qwen2, and LLaMA variants, and provides consistent evidence for the prevalence of absorption effects.\n\n- The proposed absorption rate offers a simple and interpretable metric to quantify this behavior, and the combination of toy examples and real-model evaluations helps ground the intuition.\n\n- The work raises important questions for the future design of interpretable sparse representations and may inform efforts toward more robust circuit discovery.\n\nWeaknesses:\n\n- The experiments focus on relatively small-scale models; it remains unclear whether similar absorption patterns persist in larger models such as 7B or 9B.\n\n- The absorption rate metric relies on token-level ablation and is mainly applicable to lexical features, limiting its ability to capture absorption in higher-level or cross-token concepts.", "was_revised": true, "final_justification": "Thank you for the detailed responses. I find this work particularly valuable for deepening our understanding of the features extracted from Sparse Autoencoders (SAEs). The insights provided contribute meaningfully to the field, and I am genuinely interested in the potential of this approach.", "rebuttal": "We thank the reviewer for their thoughtful reading and review of the paper. We address the questions below:\n\n> Any thoughts on when feature absorption is more likely to occur, and whether there are ways to mitigate it? For example, are certain types of features or training settings more susceptible to this effect?\n\nQualitatively, absorption seems most prevalent for features that are very “general”. For example, we have never found clear “parts of speech” latents in any SAE despite LLMs clearly representing this information, likely because latent tracking parts of speech are too riddled with absorption exceptions to even be recognizable anymore. We find that latents for concepts for anything that’s a “category” or “class of thing” also seems to be very susceptible to absorption in our experience, e.g. things like “is Python code” or “is an animal”, etc...\n\nSo far, we have found that just being aware of absorption has been very helpful when using SAEs for interpretability work. For example, we no longer assume that an SAE latent for a concept not firing when we think it should therefore means the concept is not present in the LLM activations. We also no longer assume that our inability to find a clear SAE latent for a given concept is therefore evidence that the model does not represent that concept - it could also be that absorption has simply made the latent for the concept not fire enough that we can no longer tell what that latent mainly is about. We also try to use supervised methods like probes whenever feasible as these do not suffer from absorption. Despite this, we do still find SAEs very useful as a first-pass for interpretability work, as they still often surface concepts that are unexpected and can be further targeted with supervised probes.\n\n> In my opinion, SAEs may serve as a form of knowledge compression, which can introduce ambiguity. While the resulting features are more interpretable than individual neurons, activation-based features descriptions might still underrepresent a feature’s full role—especially in cases where features reflect different functional categories, such as input features, which capture patterns within the model’s input, and output features, whose main role is to directly influence the tokens the model generates.\n\nWe have also seen recent work that dives into this dichotomy of input vs output features in SAEs [1]. It could be that separating classification features and generation features is core to how LLMs represent concepts internally. Regardless, it certainly seems to be a promising direction of future work to try to understand the types of features that SAEs learn, and if there are further relationships or geometry within features captured by SAEs.\n\n### References\n\n[1] Arad, Dana, Aaron Mueller, and Yonatan Belinkov. \"SAEs Are Good for Steering--If You Select the Right Features.\" arXiv preprint arXiv:2505.20063 (2025)."}, {"review_id": "p75af24KLz", "reviewer": "Reviewer_64Na", "summary": "This paper showcases the phenomenon of feature absorption, a mechanism by which sparse autoencoders (SAEs) systematically fail to fully capture features that co-occur in a hierarchical fashion, creating an interpretability illusion. A toy model is used to show that absorbed features are theoretically and empirically preferred by the SAE training procedure. Ablation experiments, cosine similarities to linear probes and k-sparse probes are used to identify specific examples of feature absorption in public base LLMs. A measure of absorption rate is defined and used to quantify the feature absorption in these LLMs.", "questions": "1. As I understand (correct me if I am wrong), the main function of the absorption metric is primarily to auto-classify existing feature splits as either benign or harmful to the SAE's ability to detect a known human-interpretable latent. While the prevalence of harmful absorptions is confirmed by the metric, the authors do not evaluate whether these purported absorbed feature events are indeed more harmful towards SAE-derived interpretations than split non-absorbed feature events. I would like to see some evidence as to whether or not this metric is actually a good detector of the quality of an SAE-derived interpretation.\n\n2. My Weakness item #2 comes from my understanding that the purpose of the absorption metric is that it should be used in conjunction with the SAEs when trying to identify the unknown features that may be present in a model. Is this the intended purpose or use case? It is unclear whether the metric is supposed to serve a different purpose instead, e.g., to help with further interpretation of the model after the features have already been identified. If you would explain an example of where you expect this metric to be useful in a different part of the interpretation pipeline, I may increase my score.\n\nminor: I see in the glossary that the authors explain how they avoid overloading the word \"feature\", which is really helpful for the reader. If I had seen that earlier, reading this paper may have been more straightforward, so I might suggest folding that explanation into the background and also keeping it in the Appendix.\n\nminor: Line 87, capitalization typo on \"L1\"", "limitations": "There are no concerning limitations or potential negative societal impacts from this work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "A note: as the saying goes, all models are wrong, but some are useful. I will pay most attention to whether or not feature absorption is a useful model for explaining how SAEs work, and how that in turn affects the usefulness of SAEs as a model to explain how LLMs work.\n\nStrengths\n1. A substantial amount of evidence is shown to convincingly support that feature absorption occurs in SAEs used in practice.\n2. The feature absorption can be replicated in small toy examples to be studied.\n3. Feature absorption can be theoretically shown to be preferred by the SAE training procedure.\n4. If feature absorption indeed occurs, it is probably a widespread phenomenon across many open-source SAEs, significantly hindering mechanistic interpretatability efforts.\n\nWeaknesses\n1. It is unclear to me that describing a specific kind of feature splitting as \"feature absorption\" helps to explain more about what is happening to the SAE than just saying that one of the feature splits is extremely small. I am unsure that the identification of feature absorption as a separate phenomenon from feature splitting adds that much interpretation value when the interpreter already understands that feature splitting is common and can make the SAE explain things less clearly sometimes. If we could just redefine one of the \"true human-interpretable features\" to be the union of the split features, this would produce hierarchical co-occurrence of features leading to feature absorption which wasn't there in the first place. Individual feature absorptions may only occur under certain allotments of \"true features\" and not others, making the presence/absence of this problem inherently subjective.\n2. In order to evaluate the test metric for feature absorption, you need to have a probe, and that probe must be trained with labeled data corresponding to the feature you are testing for. You could only do this probing successfully if you already knew the feature you are looking for is there in the model. But then, why would you even be using the SAE and feature absorption metric in the first place? You've already found the feature's direction using your probe, which I thought was the whole point of doing all of that. Alternatively, maybe you don't know the features and you're using the SAE to find them. Then, you can't be training a probe to make your absorption metric, since you don't know what the feature is that you want to label your data for. Again, the metric seems to be useless. It seems like the test metric can play no actual role in any part of the core pipeline of model interpretation.", "was_revised": true, "final_justification": "The paper convincingly finds that feature absorption occurs within SAEs, stimulating future research into correcting this kind of failure mode. I would give the paper a higher score if it were not for the potential alternate interpretation that the LLMs were truly using separate non-hierarchically co-activating features that were being correctly identified by the SAE, in cases where the authors claim the features are hierarchically co-activating and the SAE found the wrong features due to absorption.", "rebuttal": "We thank the reviewer for their engagement with our work and thoughtful questions. We will address the minor issues raised in the final version of the paper. We address the main concerns below:\n\n> It is unclear to me that describing a specific kind of feature splitting as \"feature absorption\" helps to explain more about what is happening to the SAE than just saying that one of the feature splits is extremely small … If we could just redefine one of the \"true human-interpretable features\" to be the union of the split features …\n\nThis is an important point to clarify to understand the significance of feature absorption compared with the traditional understanding of feature splitting, and why absorption is so pernicious to interpretability efforts. The traditional understanding of feature splitting is that in a small SAE, a more coarse-grained feature like “country” will be split into more specific features for every country like “France”, “Japan”, etc in a larger SAE. If this were all that happens in feature splitting, this is not a problem for interpretability as all features in both cases are perfectly interpretable and understandable.\n\nHowever, what the feature absorption work shows is that when there are hierarchical features like “country” and “Japan” (since Japan is a country), we instead end up with a “country” latent that fires on most countries, but, mysteriously, not on some very specific countries like “Japan”. This harms the interpretability of the “country” latent, since it is not firing in places where it seemingly should fire. Can we even claim that this is a “Country” latent at this point? Even worse, since we do not have ground-truth data for the “true features” of a model, we have no way to know when or if this is happening. We cannot trust that any latent actually captures the concept it seems like it captures. For instance, even our “Japan” latent may fail to fire on mentions of “Osaka” if “Osaka” has its own latent in the SAE. Even worse, imagine we think we find a latent for “the model is being dishonest” and use it to monitor the model, but due to feature absorption, this latent fails to fire on specific types of dishonesty so our monitor silently fails.\n\nIf we knew which latents in the SAE were engaging in splitting and absorption, we could indeed create grouped “super-latents” as the union of all split latents, as you suggest, but in real LLMs we do not have ground-truth feature data, and do not yet have a way to automatically detect all feature splits in a given SAE. Indeed, we know of researchers manually trying to cluster SAE latents that seem related as an interim attempt to do this, but automatically detecting/grouping of feature splitting is still an unsolved research problem.\n\n> My Weakness item #2 comes from my understanding that the purpose of the absorption metric is that it should be used in conjunction with the SAEs when trying to identify the unknown features that may be present in a model. Is this the intended purpose or use case? It is unclear whether the metric is supposed to serve a different purpose instead, e.g., to help with further interpretation of the model after the features have already been identified. If you would explain an example of where you expect this metric to be useful in a different part of the interpretation pipeline, I may increase my score.\n\nOur absorption metric is not meant to be used to identify every specific instance of absorption in an SAE, as like you rightly point out, that would require already having a probe for all the features captured by the SAE and defeats the purpose of using an SAE. Instead, our absorption metric is meant to do the following two things:\n\n**Prove that absorption happens in SAEs trained on real LLMs.**\n\nThe goal of this paper is to introduce the problem of feature absorption in SAEs. We show the theoretical underpinnings of absorption in simple toy models, but we need a way to demonstrate whether this happens or not in SAEs trained on real LLMs. Our metric fills that purpose, showing that all LLM SAEs we tested engage in feature absorption. This proves that absorption is not only a theoretical problem but can be found in real SAEs that are used by researchers as well.\n\n**Guide the search for new SAE architectures that do not engage in feature absorption.**\n\nThe best outcome of this work would be to inspire new SAE architectures that do not engage in feature absorption. However, this requires that we have a way to benchmark the absorption rate of a given SAE, so we can tell if a new SAE architecture no longer engages in absorption. Notably, this does not require that every instance of absorption across every concept / latent in the SAE be identified by our metric, but only that we can gauge the relative rate of absorption across a few concepts to use as a benchmark.\n\nIn this role as a benchmark, our absorption metric is similar to any other benchmark used throughout the ML field. For instance, a benchmark like HaluEval [1] is meant to benchmark hallucinations in LLMs. HaluEval cannot tell you if an arbitrary LLM output is a hallucination or not, but an LLM that does not hallucinate should score well on the HaluEval benchmark. Similarly, our absorption metric cannot tell you if any given SAE latent contains absorption or not, but we expect an SAE that does not engage in absorption to score well on our absorption metric.\n\n### References\n\n[1] Li, Junyi, et al. \"HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023."}]

Author rebuttal:

Meta-review: Summary  
This paper studies the feature splitting in SAE: as the number of features increase in SAE, hierarchical features tend to split into finer features. However, this paper shows that this feature splitting is not robust. Seemingly monosemantic features fail to fire where they should, and instead get absorbed into their children features. This paper introduces a metric to detect feature absorption, and validate the findings on hundreds of LLM SAEs. This paper shows theoretically why sparse dictionary learning encourages feature absorption. This paper argues that varying SAE sizes or sparsity is insufficient to solve this issue, and discuss potential solutions.

Strengths  
- The problem of feature absorption in feature splitting is intuitive and, despite should apply quite widely, was not present in the literature.
- This paper uses a combination of toy models and concrete examples and larger-scale studies to demonstrate the mechanism of feature absorption and how large this problem is.
- The feature absorption can be theoretically shown to be preferred by the SAE training procedure (e.g., in Appendix A.2).
- From Bisd: This feature absorption problem, once identified, can change people's views about how promising SAEs are.
- From Uf4h: The authors argue very persuasively for the general plausibility of feature absorption and its implications on the use of SAEs for intepretability.
- From mChz: This work is particularly valuable for deepening our understanding of the features extracted from SAEs.

Weaknesses  
- The feature absorption metric and the empirical results focus on the first-letter task, and the generalizability to other tasks would be good to have.
- Minor polish could be made to improve the presentation of this paper.
Decision: Accept (oral)