Sparse Autoencoders Do Not Find Canonical Units of Analysis
Authors: Patrick Leask
Bart Bussmann
Michael T Pearce
Joseph Isaac Bloom
Curt Tigges
Noura Al Moubayed
Lee Sharkey
Neel Nanda
Venue: ICLR 2025 Poster (2025)
Keywords: sparse autoencoders
mechanistic interpretability
Primary area: interpretability and explainable AI
TL;DR: Through stitching sparse autoencoder latents and training meta-SAEs we show that sparse autoencoders do not learn canonical units of analysis.

Abstract: A common goal of mechanistic interpretability is to decompose the activations of neural networks into features: interpretable properties of the input computed by the model. Sparse autoencoders (SAEs) are a popular method for finding these features in LLMs, and it has been postulated that they can be used to find a canonical set of units: a unique and complete list of atomic features. We cast doubt on this belief using two novel techniques: SAE stitching to show they are incomplete, and meta-SAEs to show they are not atomic. SAE stitching involves inserting or swapping latents from a larger SAE into a smaller one. Latents from the larger SAE can be divided into two categories: novel latents, which improve performance when added to the smaller SAE, indicating they capture novel information, and reconstruction latents, which can replace corresponding latents in the smaller SAE that have similar behavior. The existence of novel features indicates incompleteness of smaller SAEs. Using meta-SAEs - SAEs trained on the decoder matrix of another SAE - we find that latents in SAEs often decompose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic.  The resulting decompositions are often interpretable; e.g. a latent representing "Einstein" decomposes into "scientist", "Germany", and "famous person". To train meta-SAEs we introduce BatchTopK SAEs, an improved variant of the popular TopK SAE method, that only enforces a fixed average sparsity. Even if SAEs do not find canonical units of analysis, they may still be useful tools. We suggest that future research should either pursue different approaches for identifying such units, or pragmatically choose the SAE size suited to their task. We provide an interactive dashboard to explore meta-SAEs: https://metasaes.streamlit.app/

Published as a conference paper at ICLR 2025
SPARSE AUTOENCODERS Do NoT FIND CANONICAL UNITS OF ANALYSIS
Patrick Leask*
Bart Bussmann" Independent
Michael Pearce Independent
Department of Computer Science Durham University patrick. weesskkdurhaam....m ac. uk
bartbuss com
Joseph Bloom Decode Research
Curt Tigges Decode Research
Noura AI Moubayed
Lee Sharkey Apollo Research
Department of Computer Science
Durham University
Neel Nanda
ABSTRACT
A common goal of mechanistic interpretability is decompose the activations of neural networks into features: interpretable properties of the input computed by the model. Sparse autoencoders (SAEs) are a popular method for finding these features in LLMs, and it has been postulated that they be used to find canon- ical set of units: a unique and complete list of atomic features. We cast doubt this belief using two novel techniques: SAE stitching to show they are in- complete, and meta-SAEs to show they are not atomic. SAE stitching involves inserting swapping latents from larger SAE into smaller one. Latents from the larger SAE can be divided into two categories: novel latents, which improve performance when added the smaller SAE, indicating they capture novel in- formation, and reconstruction latents, which can replace corresponding latents in the smaller SAE that have similar behavior. The existence of novel features indicates incompleteness of smaller SAEs. Using meta-SAEs SAES trained on the decoder matrix of another SAE we find that latents in SAES often de- compose into combinations of latents from a smaller SAE, showing that larger SAE latents are not atomic. The resulting decompositions are often interpretable: e.g. latent representing "Einstein" decomposes into "scientist", "Germany", and "famous person". To train meta-SAEs we introduce BatchTopK SAES, an im- proved variant of the popular TopK SAE method, that only enforces fixed av- erage sparsity. Even if SAEs do not find canonical units of analysis, they may still be useful tools. We suggest that future research should either pursue differ- ent approaches for identifying such units, or pragmatically choose the SAE size suited to their task. We provide an interactive dashboard explore meta-SAEs: https:: //ts:///metaassse.m streamlit.app)
INTRODUCTION
Mechanistic interpretability aims to reverse-engineer neural networks into humaa--nterprettblle gorithms (Olah et al. 2020 Meng et al., 2022 Geva et al. 2023 Nanda et al. 2023 Elhage et al.) 202T]. A key challenge of mechanistic interpretability is identifying the correct units of analysis fundamental components that can be individually understood and collectively explain the network's 's function. Ideally, these units would be unique, with no variations (Bricken et al. 2023); complete, encompassing all necessary features (Elhage al. 2022); and atomic irreducible, indivisible into smaller components (Engels et al. 2024). We refer to set of units with all of these properties as canonical.
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2025
Education and Equality
Eleanor
Astronomy and astronomers
Eagle
Galascies
Space exploration and sal-fi
Estonia
Starts With
Galaxies
Universe and cosmos
and Bavari
Germany
Einstein
Germany and people
Scientists and scientific
Science and Scientists
Prominent Figures
Repeated mentions of "Science"
Specific mayors (de Blasio, Bloomberg)
Scientific findligs concepts
Scientific heids
Notable historical figures
Names politicians
Names football managers
Einstein Latent
Meta Latents
SAE Latents
Figure 1: Decomposition of an SAE latent representing "Einstein" into set of interpretable meta- latents. The edges connecting the nodes indicate shared activation by meta-latent, with thicker lines representing stronger connections. It demonstrates the ability of meta-SAEs to uncover the underlying compositional structure SAE latents, revealing how complex concept can rep- resented sparse combination of meta-latents. We built dashboard where you explore all meta-latents: hettps:/netassass st strammit..
Initially, researchers hoped that individual MLP neurons (Meng et al. 2022 Olah et al., 2020) and attention heads (Wang al. 2022, Olsson et al. 2022) could serve as these units. However, these proved insufficient for interpretability due to polysemanticity. where single neuron responds to
multiple unrelated concepts (Olah et 2020 Elhage et al. 2022).
Recently, sparse autoencoders (SAEs) have emerged as promising alternative by decomposing the activations of L.LMs into a dictionary of interpretable and monosemantic features (Bricken et al.) al.) 2023). A key hyperparameter when training SAEs is the dictionary size,
2023 Cunningham L.c. number of latent units. Previous work conjectured that SAEs might identify a set "true features" with sufficient dictionary size (Bricken et al. 2023). i.e. the canonical features that the goal of much mechanistic interpretability research.
One challenge to the theory that SAEs identify canonical set of units is the phenomenon of feature splitting, where latents from smaller SAEs "split" into multiple, more fine-grained latents in larger SABs (Bricken al., 2023). For example, [Bricken al. (2023) find base64 feature that splits into three features in a larger SAE: activating on letters, digits, and encoded ASCII in basc64 text. Furthermore, Templeton (024) finds that larger SAE has latents that activate on certain specific individual chemical elements that smaller SAE did not represent. Currently, the effect of dictionary size on the features has not been systematically studied, in part because we lack good methods to compare latents found in SAES of different sizes.
To better understand how SAEs of different sizes capture features, we develop a method called SAE stitching. When stitching SAFs, we systematically swap clusters of latents between SAFs of different dictionary sizes based on their cosine similarity. Through this method, we observe that larger SAEs learn both more fine sine-grained versions of latents found in smaller SAEs, but also entirely novel latents. The existence of novel latents suggests that the reconstruction error smaller SAEs partially due to missing out information altogether, not just imperfect approximations to features or overly coarse latents, indicating incompleteness.
Contrary the story of feature splitting, observed that some reconstruction latents had split from multiple latents in the smaller SAE, indicating those latents were composing into more com- plex latents. Previous work has predicted that the sparsity penalty incentivizes latents represent
2

---

Published as a conference paper at ICLR 2025
O
Small SAE
O
Large SAE
Example of composition of latents in SAES of different sizes. The smaller SAE has six latents, three of which reconstruct shape features, and three of which reconstruct color features. Reconstructing shape of specific color requires active latents (e.g. blue and square). On the other hand, the larger SAE has nine latents, each of which reconstructs a different color and shape combination. In the larger SAE, only single active latent is required to reconstruct the colored shape (c.g. blue square). The sparsity penalty incentivizes larger SAEs to learn compositions of latents rather than atomic latents.
Figure 2:
composed features, even if they are independent (Wattenberg & Viégas 2024 Bricken et al. 2023] Anders et al. 2024). For instance, consider neural network that represents color and shape fea- tures, each with three values (red/green/blue and circle/squareetriannle)) A small SAE might learn a latent for each value. A large SAF, however, might learn latents for all color-shape combinations (i.e. blue square) instead the 6 fundamental features (Smith 2024). The large SAE can represent this is sparser as only one latent activates per input rather than two, see Figure
To investigate this, we introduce meta-SAEs, which are SAES trained to find sparse decompositions of the decoder directions of another SAE. These decompositions are often interpretable, e.g. la- tent representing "Einstein" decomposes into meta-latent representing "scientist", "Germany", and "prominent figures", among others (see Figure [I). This shows that latents are often atomic, espe- cially in larger SAEs. We find that meta-latents are similar to latents in smaller SAFs, demonstrating that latents from larger SAES can interpreted as the composition of latents from smaller SAEs.
With SAE stitching, we observed reconstruction latents that had split from a small number of small SAE latents. To train Meta-SAEs, we wanted very few meta-latents active per input, e.g. four, regime where the state-of-the-art JumpReLU SAEs (Rajamanoharan et al. 2024b) perform poorly. Another popular method is TopK SABS (Gao et al. 2024)), but these impose fixed number of active meta-latents, which is also undesirable. To address these limitations novel method for training SAEs that enforces fixed sparsity across a batch producing only fixed average sparsity per individual input. We find that BatchTopK is useful for training SAEs model activations too and is a Pareto improvement over TopK SAES. Furthermore, it has better performance than Jump SAES at the desired sparsity levels for Meta-SAEs.
introduce BatchTopk,
In summary, our contributions are:
1. SAE stitching, as method for comparing latents across different sizes of SAE. Latents in a larger SAE are either novel latents, missing in smaller SAEs, or reconstruction latents, similar to some latents in smaller SABS.
2. Meta-SAEs, as an approach for decomposing the decoder directions of SAEs into inter- pretable, monosemantic leta-latents.
3. We also introduce BatchTopp SAEs, the state-of-the-art SAE architecture for sparse dic- tionary learning on language model activations at a fixed average sparsity.
Our empirical results suggest that simply training larger SAEs is unlikely result in canonical set of units for all mechanistic interpretability tasks, and that the choice of dictionary size is subjective. We suggest taking pragmatic approach to applying SAEs to mechanistic interpretability tasks, trying SAEs several widths to see which is best suited. We are uncertain whether canonical units of analysis exist, but results suggest that alternative approaches should be explored.
3

---

Published as a conference paper at ICLR 2025
2
SPARSE AUTOENCODERS
Sparse dictionary learning is the problem of finding a decomposition of a signal that is both sparse and overcomplete (Olshausen & Field (997). Lee et al. (007 initially applied the sparsity straint to deep belief networks, with SAES later being applied to the reconstruction of neural network activations (Bricken et al., 2023 Cunningham al. 2023). In the context of large language models, SAEs decompose model activations x E IK" into sparse linear combinations of learned directions, which are often interpretable and monosemantic.
An SAE consists of an encoder and decoder:
f(x) a (W booe)
(1)
x(f) W bdee
(2)
where f(x) E R is the sparse latent representation and x(f) R is the reconstructed input. Wene
is the encoder matrix with dimension n x m and benc is a vector of dimension m; conversely Wile the decoder matrix with dimension m n and b is of dimension n. The activation function enforces non-negativity and sparsity in f fxx, and a latent i is active on sample x if f:(x) > O.
SAEs are trained on the activations of language model stream, on large text corpus, using loss function of the form
particular site, such as the residual
((x) = |X xx((ffx)II + XS(((x)) +aLmx
(3)
Cmrcentract
Expriry
where a function of the latent coefficients that penalizes non-sparse decompositions, and is a sparsity coefficient, where higher values of X encourage sparsity at the cost of higher reconstruction error. Some architectures also require the use of an auxiliary loss Laux. for example to recycle inactive latents in TopK SAEs (Gao et al.] 2024).. We expand on the different SAE variants in Appendix A.2 and provide a glossary of terms in Appendix A.
RELATED WORK
SAFs for Mechanistic Interpretability. SAEs have been demonstrated to recover sparse, monose-
mantic, and interpretable features from language model activations (Bricken et al. 2023 Cunning. ham et al. 2023 Templeton, 2024 Gao et al. 2024, Rajamanoharan et 2024a b), however their application to mechanistic interpretability is nascent. After training, researchers often interpret the meaning of SAE latents by examining the dataset examples on which they are active, either through manual inspection using features dashboards (Bricken al.) 2023) or automated interpretability techniques (Gao et al. 2024). SAES have been used for circuit analysis (Marks et al. (0224) in the vein of (Olah et al. Olsson al. 2022); study the role of attention heads in GPT-2 (Kissane jet al. 2024); and to replicate the identification of a circuit for indirect object identification in GPT-2 (Makelov et al., 2024). Transcoders, variant of SAEs, have been used to simplify circuit analysis and applied to the greater-than circuit in GPT-2 (Dunefsky et al.] 2024)). While these applications highlight SAEs as valuable tools for understanding language models, it remains unclear whether they identify canonical units of analysis.
Representational Structure. Language models trained on the next-token prediction task learn rep- resentations that model the generative process of the data. For example, Li et al. (202)) found that transformers trained by next-move prediction to play the board game Othello explicitly represent the board state; and Gurnee & Tegmark (2023) used linear probes to predict geographical and temporal information from language model activations. SAES learn these structures as well, for example, the activations of cluster of GPT-2 SAE latents form a cycle when reconstruction activations for weekday name tokens (Engels et al.) 2024). Bricken et al. (2023) find evidence of convergent global structure by applying 2-dimensional UMAP transformation to decoder directions of SAEs of dif- ferent sizes. This results in a rich structure of latent projections with regions of latents relating to similar concepts close to each other.
4

---

Published as a conference paper at ICLR 2025
Tuning Dictionary Size. Previous work tuning dictionary size has mixed findings regarding how SAEs scale with dictionary size. For instance, Templeton (2024) observed that larger SAEs learn latents absent in smaller ones, such specific chemical elements. Conversely, Bricken et al. (2023) found similar latents across various SAE sizes, noting that latents smaller SAEs sometimes split into multiple latents the dictionary size increases (Appendix |A.4 includes such examples taken from our SAEs). Currently, the effect of dictionary size on the learned latents has not been systematically studied.
Model Stitching. Model stitching is a method by which layers from one neural network are "stitched" or swapped into another neural network. Lene Vedaldi (2013); Bansal al. (202T) propose that model stitching can be used to quantify representation similarity across different neu- ral network architectures and training regimes by demonstrating that if two networks trained on the same task can be stitched together with minimal loss in performance, it suggests high degree of alignment their intermediate representations.
4 SAE STITCHING
Templeton 2024) finds that a larger SAE has latents that activate on certain specific individual (Briekee et al. (2023) observes that smaller SAEs learn latents activating on broad mathematical text, while larger SAEs learn latents activating on more specific mathematical categories. On the one hand, this suggests that training large SAEs is necessary to learn latents relating to all relevant concepts. On the other hand, this means that some of the capacity larger SAEs used represent similar information as smaller SAEs, but with more latents.
chemical elements that smaller SAE did not represent. Conversely, Bricken
To compare latents of SAEs with different dictionary sizes, we introduce SAE stitching. In SAE stitching, we add replace latents in one SAE with latents from another and observe the effect on the reconstruction performance. We find that larger SAEs learn both finer-grained versions of latents from smaller SAEs (reconstruction latents) and entirely new latents that capture additional information (novel latents). SAE stitching allows us to identify these two categories of latents in larger SAES.
4.1 STITCHING OPERATION
The output of an SAE can be expressed as sum of the contributions from the individual latents:
d
= Σ i=0
wdd fx() + ddc
(4)
where d is the size of the SAE dictionary, Wike is an individual decoder direction, f((x) is the activation value for the latent and b the decoder bias term. To stitch latents from one SAE into another, modify the reconstruction follows:
x abgee -11 (1- aa) +
Σ loELo
Wdec Woio fo.1,
+
Σ HELI
Wide f1,,
(5)
1.4, represent individual decoder directions and Lo and L₃ are the set of latents we include from the respective SAEs. Unlike with model stitching (Bansal et al.] 2021). SAE decoder directions are privileged and require transformations to stitch.
where a Lol Widee and Wree
We experiment with stitching on eight SAEs trained on the residual stream of layer 8 of GPT Small |Radford et al. (2019) with dictionary sizes ranging from 768 to 98,304 and two of the Gemma Scope SAES (Lieberum et al. 2024) trained on Gemma 2B (Team et al. 2024) with dictionary size 16,384 and 22,768. For the full list of SAE properties and training details see Appendix A.6
5

---

Published as a conference paper at ICLR 2025
Small SAE
Stitched SAE
Swap
15
B
@
Large SAE
Reconstruction latents
Novel latent
cosine sim
Figure 3: SAE stitching operation: connected subgraphs of latents can be swapped between SAEs
based on cosine similarity
4.2 NOVEL AND RECONSTRUCTION LATENTS
Using SAE stitching, we want to find out whether latents in larger SAEs are just more fine-grained versions of latents of smaller SAEs, whether they represent novel information that is missed by the smaller SAEs. If we stitch latent from a larger SAE into a smaller SAE and the reconstruction performance improves, this indicates that the latent is adding new information to the smaller SAE. the reconstruction performance worsens instead, this indicates that the latent overlapping and the stitched SAE is representing the same information twice. In this case, we instead swap the latent from the smaller SAE with its corresponding latents from the larger SAE.
However, evaluating all combinations of latents when stitching SAEs is computationally infea- sible. Bricken et al. (2023) finds that the decoder directions of latents with similar behavior across SABS of different sizes form local clusters with high cosine similarity. As such, we propose thresh- olding on the maximum decoder cosine similarity metric as heuristic to classify latents into the two categories, i.e. for an SAE decoder direction Wider, the feature assigned the novel group
Wide Wile
max
Wiler
Widee <0 0
(6)
where Wale represents any decoder direction in Wide.. If the maximum cosine similarity is larger than the threshold, it assigned to the reconstruction group. Figure4sshoss the relationship between cosine similarity and the effect the reconstruction loss of independently stitching that feature for a pair of GPT-2 SAEs. our experiments we set the cosine threshold to 0.7 for the GPT-2 SAEs, and 0.4 for the Gemmascope SAEs, based the ROC curve for using different thresholds to predict the effect of adding a latent based on the maximum cosine similarity.
exploratory analysis. Appendix Figure 15 shows
Inserting reconstruction latents into the target SAE decreases the performance due to overlapping functionality. We resolve this by removing latents in the target SAE that have high cosine similarity to the ones we are inserting, effectively swapping them with similar latents. To select which latents swap, we construct bipartite graph where latents from the smaller SAE form and latents from the larger SAE form the other. We connect latents if they have decoder cosine similarity above the threshold. For each connected subgraph, the latents from the smaller SAE can be swapped by their corresponding connected latents in the larger SAE (see Appendix A.5 for examples of connected subgraphs).
of vertices
Figure 5 shows four transitions between pairs of SAEs of increasing size. For each transition, we first identify novel latents from the larger SAE and add them one by one, increasing the average LO since we are inserting extra latents. We then identify groups of related reconstruction latents as defined by our bipartite graph and swap each group removing similar latents from the smaller SAE and replacing them with their counterparts in the larger SAE. This decreases LO average since the larger SAE represents similar information more sparsely. Each step leads on average to improved reconstruction performance, allowing of dictionary size, sparsity, and reconstruction performance.
interpolate between different SAEs in terms
The existence of the novel group of latents demonstrates that smaller SAEs incomplete, and that larger SAE learn features that are missed by the smaller SAE. The reconstruction group seems to largely reconstruct the same features as the latents in the smaller SAF, but uses features to

---

Published as a conference paper at ICLR 2025
Max Cosine Similarity vs Effect on MSE for Large SAE Features
0.010
Increases MSE when added
Decreases MSE when added
0.008
Effect on MSE when Added
0.006
0.004
0.000
0.0
0.2
0.4
0.5
0.8
1.0
Max Cosine Similarity with Small SAE Features
Figure 4: Change in MSE when adding each feature from GPT2-1536 to GPT2-768, plotted against the maximum cosine similarity of that feature to any feature in GPT2-768. Features with cosine similarity less than 0.7 tend to improve MSE, while more redundant features hurt performance. A few extreme outliers with very high cosine similarity and effect on MSE are not visible this plot.
achieve lower sparsity (Appendix Figure [6]. In some cases, a single latent splits into two more specific latents this is the feature splitting observed in Bricken et al. (2023). However, we find some reconstruction group latents have high decoder cosine similarity multiple latents in the smaller SAE, suggesting the large SAE latent is an interpolation or composition of the smaller SAE latents (Appendix [A.5). This supports the predictions of [Wattenbrr & Viégas (2024); [Brikken et al. (2023); Anders et al. (2024) that the sparsity penalty results in the undesirable composition of features that may be sparser, but not add any new information. We explore this phenomenon further in 5ection
META-SAES
In Section we demonstrated through SAE stitching that increasing dictionary size leads to larger SAES learning not only novel features. but also reconstruction latents that encode similar information to the latents in smaller SAEs. We found that some of these reconstruction latents have high cosine similarity with multiple latents in the smaller SAE, see Appendix A.5 This suggests that these smaller SAE latents are composing into more complex latents, such as the example of latent representing "blue" and another latent representing "square" combining in "blue square"- lateet (see Figure 2) If large SAE features are indeed compositions rather than atomic, it may possible to decompose them into more fundamental units.
To decompose the latents of larger SAEs, we introduce meta-SAEs. Meta-SAEs are SAEs trained to reconstruct the decoder directions Wides of standard SAE using a dictionary of latentaaents, rather than reconstructing network activations. That is, treat the latents Wiles as the training data for our meta-SAE. We find that meta-latents and meta-SAE decompositions are interpretable, with the meta-latents being monosemantie. Table I provides some examples of meta-SAE decompositions.
The latents of meta-SAE have similar decoder directions to those found in SAEs of comparable size trained directly the same network activations. This observation further supports the hypothesis that larger SAE latents are not entirely new features but may be compositions of features already learned, albeit less precisely, by smaller models.
7

---

Published as a conference paper at ICLR 2025
Transition from 768-SAE to 12288-SAE
2.6
1536 to 3072
2.4
Average MSE
6144 to 12288
2.2
2.0
1.8
1.6
1.4
52.5
47.5
Awerage
42.5
768 1536
40.0
3072
3072 to 6144
35.0
2000
4000
6000
8000
10000
12000
Total Number of Features
Interpolating between SAE pairs of increasing dictionary size through two steps per phase: adding novel latents (in- creasing LO) then swapping groups of reconstruction latents (decreasing LO average). Both steps on average improve reconstruction (MSE). The LO and MSE are averages over input samples.
Figure 5:
Table I: Example GPT-2 SAE latents and their meta-latent decompositions, with model-generatee explanations
what the latents activate on. More latent decompositions can be found interactive dashboard:
SAE Latent Description Meta-Latent Descriptions
Albert Einstein
Science & Scientists, Famous People, Space & Astronomy, Germany, Electricity, Words starting with capital
Rugby
Sports activities, Words starting with 'R', References Ireland, Refer- ences to sports leagues, activities actions
Android Operating System
Mobile phones, operating systems, Californian cities
We use the BatchTopK SAE, as described Section 6 to train our meta-SAEs. We train our meta- SAE on the decoder directions of the GPT-2 SAE with dictionary size 49152. The meta-SAE has dictionary size of 2304 meta-latents, with on average of meta-latents active per SAE latent. Due to small number of training samples for the meta-SAE (49152), the meta-SAE is trained for 2000 epochs. We use the Adam optimizer with learning rate le-4 and batch size of 4096. After training, the meta-SAE explains 55.47% of the variance of the decoder directions of the SAE.
5.1 EVALUATING META-SAE DECOMPOSITIONS
We follow the lead of (Bills et al., 2023 Bricken et al. 2023 Cunningham et al. [023 Rajamanoha- ran et al. 2024b) in evaluating neural network and SAE latents using automated interpretability with LLMs.
First, we generate explanations SAE latents by presenting GPT-4o-mini with list of input se- quences that activate SAE latent to varying degrees, and prompting it to generate natural lan- guage explanation of the feature consistent with the activations. Second, we collect all of the SAE latents which meta-SAE latent active, and prompt again with the explanation and number of top activating examples of each of the SAE latents, asking the model to provide an explanation of the common behavior of the SAE latents, which becomes the meta-SAE latent explanation.
We evaluate the meta-SAE latent explanations a zero-shot multiple- settiple-ccoic-geeee queste-n setting. For a given latent we prompt GPT-4o-mini with the explanations of the meta-SAE latents that active
8

---

Published as a conference paper at ICLR 2025
that latent, and ask it to choose which of 5 SAE latent explanations most relate to the explanations of the meta-SAE latents. One of these SAE latent explanations is of the correct latent, with the remaining 4 explanations corresponcing to random latents from the SAE. On a random sample of 1,000 SAE latents, GPT-40-mini chose the correct answer of the five options 73% of the time.
5.2 COMPARISON TO SMALLER SAE LATENTS
In Section we hypothesised that latents in larger SAEs can be described as the composition of latents in smaller SAEs. We find that meta-SAEs learn similar latents to similar sized SAEs trained on the original reconstruction problem. Plots of the maximum cosine similarity between meta-SAE latents and latents from SAEs of different sizes are shown in Figure 22 We validate this by replac- ing meta-SAE decoder directions with the most similar SAE decoder direction, and retrained the encoder. This results in only a small decrease in meta-SAB reconstruction performance (Appendix Figure 23
This suggests that larger SAE latents are indeed composed of latents from smaller SAEs.
BATCHTOPK SPARSE AUTOENCODERS
In Section 5 we hypothesised that SAE latents will linearly decompose into a small, variable number of meta-latents. Of the popular SAE variants, top-k SAEs (Gao al. (024)) decompose each input into a fixed number of latents, whilst ReLU (Bricken et al., [023] Cunningham et al.) (023) and JumpReLU Rajamanoharan al. (2024b) have poor performance at that sparsity level and require hyperparameter tuning to achieve specific sparsity level. We provide further details on these SAE variants in Appendix A.2
To address these limitations, we introduce BatchTopK SAEs, a novel variant that extends the TopK concept to operate at the batch level rather than on individual samples. This approach allows for a flexible number of active latents per sample while maintaining desired average sparsity across the batch. Therefore, we use BatchTopK SAEs throughout our meta-SAB experiments.
The training objective for BatchTopK SAEs is defined as:
((X)=
|X BotchTTopK[ (Wene X + bene Wide + bdbe) oL
Here, X is the input data batch; Wenc and b are the encoder weights and biases, respectively; Widee and bdee are the decoder weights and biases. The BatchTopK function sets all activation values to zero that are not among the top n x activations by value in the batch, not changing the other values. The term Laux is an auxiliary loss scaled by the coefficient a, used prevent dead latents, and is the same as in TopK SAES.
BatchTopp introduces dependency between the activations for the samples in batch. We alleviate this during inference by using a threshold 0 that is estimated as the average of the minimum positive activation values across number of batches:
0 0= == (min (X) (X) 0}]
where 2i, (X) is the jth latent activation of the ith sample in a batch X. With this threshold, we use the Jump activation function during inference instead of the Bopc activation function, zeroing out all activations under the threshold 0.
We evaluate the performance of BatchTopK on the activations of two LLMss: GPT-2 Small (residual stream layer 8) and Gemma 2 2B (residual stream layer 12). We use range of dictionary sizes and values for k. and compare our results to TopK and JumpReLU SAES in terms of normalized mean squared error (NMSE) and cross-entropy degradation. For experimental details, see Appendix A.3
We find that for a fixed number of active latents (L0=32) the BatchTopK SAE has a lower normalized MSE and less cross-entropy degradation than TopK SAES on both GPT-2 activations (Figure 6) and Gemma 2B (Figure (7). Furthermore, we find that for a fixed dictionary size (12288) BatchTopK outperforms Topk for different values of k on both models.

---

Published as a conference paper at ICLR 2025
Normalized MSE vs Dictionary size (k=32)
Normalizzd MSE vs k [Dict 12288)
betchtopt IE-12)
spt
(1=32)
0.31
Jimprelu
9.11
Nu Burnelladdds
3.00
0.0
067
5X.19'4X 4X
K
10
2x
Dictonary
adddion Diclimeery
dey vs
-0.08
batchtspk (K 32:
botchtopk
topit (2-
topk
jimarclu
-0.10
10.12
deggedddeainn Brgrodattion
-3.34
-0.10
-0.20
-0.32
-330
-024
Dictionary size
Figure 6: On GPT-2 Small activations, BatchTopp largely achieves better NMSE and CE com- pared to standard TopK across different dictionary sizes, for a fixed number of active latents of 32 (Left). Ju SAEs are omitted from this comparison as their LO cannot be fixed value. For fixed dictionary size (12,288) and different levels of k, BatchTopK outperforms TopK and JumpReLU SAES, both in terms of NMSE and CE (Right).
In addition, BatchTopK outperforms JumpReLU SAEs both measures GPT-2 Small model activations at all levels of sparsity. On Gemma 2 2B model activations the results are more mixed: although BatchTopK achieves better reconstruction than JumpReLU for all values of k, BatchTopK only outperforms JumpReLU terms of CE degradation in the highest sparsity setting (k=16).
7 CONCLUSION
Our findings challenge the idea that SAEs can discover canonical set features. Through SAE stitching, demonstrated that smaller SAEs are incomplete, missing information that novel fea- tures in larger SAEs capture. Moreover, our meta-SAE experiments showed that, due to the sparsity penalty, latents in larger SAEs are often not atomic but compositions These findings suggest that there is no single SAE width at which it learns unique and complete dictionary of atomic features that can be used to explain the behavior of the model.
interpretable meta-latents.
These results imply that rather than converging on a unique, complete, and irreducible set of features. SAES of different sizes offer varying granularities and compositions features. This indicates that the choice of SAE size should be guided by the specific interpretability task at hand, accepting that no single SAE configuration provides a universal solution. However, our methods neither identify canonical units of analysis, nor the size of dictionary to use for given task. Furthermore, our work only studies two LLMs and does not include very large SAEs, such as in Templeton (2024). acknowledge that the use of SAEs in mechanistic interpretability is nascent, and whilst early results are encouraging, associating the learned latents SAES with interpretable concepts is still an open problem. In conclusion, canonical units, and that SAE practitioners should embrace a pragmatic approach towards choosing dictionary size when using SAEs interpretability tasks such as probing, unlearning and steering.
also
research suggests that alternative methods are required for identifying
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank the ML Alignment and Theory Scholars (MATS) program for facilitating this work. In particular, we are grateful for McKenna Fitzgerald and Matthew Wearden for their support in plan- ning and managing this collaboration. We are also thankful for the helpful discussions we had with Adam Karvonen, Can Rager, Javier Ferrando, Oscar Obeso, Stepan Shabalin, and Slava Chalnev about this work. Finally, BB and PL want to thank Al Safety Support for funding this work.
REFERENCES
Evan Anders,
Neo, Howard. Sparse autoencoders find composed features in small toy mod- els.
Clement
Jason floelscherObbermiict and Jessica N.
sparse autoencoders find--ii------ composed features in inninannl small toy 2024.
Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances neural information processing systems, 34:225-236, 2021.
Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, llya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons lan- guage models. URL htts://openaippblicc blob. core. windows. weehneewweeeplaae//ppeefieeee himl.(Date accessed: 14.05. 2023), 2, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decom- posing language models with dictionary learning. Transformer Circuits Thread, 2, 2023.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08600) 2023.
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable Ilm feature circuits. arXiv preprint arXiv:2406.1 1944, 2024.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodss, Danny Hernandez, Andy Jones, Jackson Kemion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 0e022099.0662 10652, 2022.
Joshua Engels, Isaac Liao, Eric Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint arXiv:2405. 14860, 2024.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04003, 2024.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304. 14767, 2023.
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.022077 2023.
Adam Karvonen, Can Rager, Samuel Marks, and Neel Nanda. Evaluating sparse autoencoders on targeted concept erasure tasks. arXiv preprint arXiv:2411.1889% 2024.

---

Published as a conference paper at ICLR 2025
Connor Kissane, Robert Krzyzanowski, Joseph Isaac Bloom, Arthur Conmy, and Neel Nanda. In- terpreting attention layer outputs with sparse autoencoders. arXiv preprint arXiv:2406.1777 2024.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief model for visual area v2. Advances in neural information processing systems, 20, 2007.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015..
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Plister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv preprint arXiv:2210.13382 2022.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma 2024. URL https://azxdv..rrg
3408805141
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. org. Software available from neuronpedia.org.
URL
Aleksandar Makelov, George Lange, and Neel Nanda. Towards principled evaluations of sparse autoencoders for interpretability and control. arXiv preprint wwXi::2405.08866, 2024.
Alireza Makhzani and Brendan Frey. k-sparse autoencoders, 2014. Utthttts://aaxxv..o
abs/1312. 3663)
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs language models. arXiv preprint arXiv:2403. 19647, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17399-17372, 2022.
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXhv:2301.05217, 2023.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 5(3):<00024-00,, 2020.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23):331-33255 1997.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving dictionary learning with gated sparse autoen- coders. arXiv preprint arXiv:2404.1600 2024a.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Commy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 14435, 2024b.
Lewis Smith.
The 'strong' feature hypothesis could be wrong.
https:
7/www. al htt.///www....ooooooooooooooeeea
the- Eta-sssooog-Tattuue---pptteaaeoe
[Accessed 23-09-2024),
2024.
12

---

Published as a conference paper at ICLR 2025
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.
Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Anthropic, 2024.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. Inter- pretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint vrXiv:2211.00522 2022.
Martin Wattenberg and Fernanda B Viégas. Relational composition in neural networks: A survey and call to action. arXiv preprint arXiv:2407. 14662, 2024.
A APPENDIX / SUPPLEMENTAL MATERIAL
A.I GLOSSARY TERMS
Active Latents (LO): For an input and SAE activation function f(x). the number of non-zero
elements in (a). Typically measured as average LO across batch: LO 2 Σ, (x:)
BatchTopK: A sparsification method that selects the top l activations across batch of size n, rather than activations per sample, maintaining a fixed average sparsity. For a n x m batch X, BatchTopk(X);; X,y if Xij is among n. x k largest elements, 0 otherwise.
Canonical Unit:
Hypothetical, fundamental building blocks of LLMs computation that are unique, complete, and atomic.
Cross-Entropy Degradation:
The increase in cross-entropy loss when replacing the model acti- vations with the reconstruction SAE.
Decoder Directions: The columns of the decoder matrix Widee that map from latent to input space. Two decoder directions with high cosine similarity suggest related features.
Dictionary Size: The dimensionality of the latent space in an SAE, determining the maximum number of unique features that can be learned.
Feature Splitting: A phenomenon where broad latent learned by a smaller SAE splits into more fine-grained latents in a larger SAE.
Latent: The encoder-decoder pair corresponding to single element in the SAE's dictionary, i.e. a learned feature of the SAE rather than a feature of the data.
The study of reverse-engineering neural networks into inter- pretable algorithms, focusing on identifying and understanding computational features and circuits.
Mechanistic Interpretability:
Meta-latents: Features learned by meta-SAE when trained the decoder directions of another SAE.
Property of feature that responds selectively single coherent concept. Con- trasts with polysemantic features.
Monosemantic:
Features in a larger SAE with maximum cosine similarity below threshold 0 to any feature in smaller SAE, indicating capture of previously unrepresented information.
Novel Latents:
Polysemanticity:
The phenomenon where individual neurons or features respond to multiple un-
related concepts.
13

---

Published as a conference paper at ICLR 2025
Reconstruction Latents: Features in larger SAE with maximum cosine similarity above thresh- old 8 to features in a smaller SAE, representing refined or specialized versions existing features.
Residual Stream: In the context of transformer architectures, the residual stream refers to the main information flow that bypasses the self-attention and feed-forward layers through residual connections.
SAE Stitching: A technique for analyzing feature relationships across SAEs of different sizes by systematically transferring latents based on decoder similarity.
Sparsity Coefficient (A):
Hyperparameter in the loss function of some SAE architectures | al| - +xS(f(x)) controlling the trade-off between reconstruction accuracy and activation spar- sity.
TopK: A sparsification approach that maintains exactly k non-zero activations per input by zeroing all but the k largest values: (o)) = x; if x; is among largest elements, otherwise.
A.2 SAE VARIANTS
ReLU SAEs (Bricken et al. 2023 the Ll-norm S(f) |[fl|1 approximation to the LO-norm for the sparsity penalty. This provides a gradient for training unlike the LO-norm, but suppresses latent activations harming reconstruction performance (Rajamanoharan al., 2024a). Furthermore, the L1 penalty can be arbitrarily reduced through reparameterization by scaling the decoder parameters, which resolved in Bricken et al. (2023) by constraining the decoder directions to the unit norm. Resolving this tension between activation sparsity and value is the motivation behind more recent architecture variants.
TopK SAEs (Gao et al. 2024 Makhzani & Frey 2014) enforce sparsity by retaining only the top k activations per sample. The encoder is defined as:
f(x) = TopK ( WWeexx + bave)
where TopK zeroes out all but the is largest activations in each sample. This approach eliminates the need for an explicit sparsity penalty but imposes rigid constraint on the number of active latents per sample. An auxiliary loss Lauz |le ēl|² is used avoid dead latents, where è Wrdec is the reconstruction using only the top-kiaux dead latents (usually 512), this loss is scaled by a small coefficient a (usually 1/32).
JumpReLU SAEs (Rajamanoharan et al. 2024b) replace the standard ReLU activation function with the JumpReLU activation, defined as
(u =H(2
(10)
where H is the Heaviside step function, and 0 is learned parameter for each SAE latent, below which the activation is set to zero. JumpReLU SAES are trained using a loss function that combines L2 reconstruction error with LO sparsity penalty, using straight-through estimators train despite the discontinuous activation function. A major drawback of the sparsity penalty used in JumpReLU SAEs compared (Batch)TopK SAFS is that it not possible to set an explicit sparsity and target- ing specific sparsity involves costly hyperparameter tuning, While evaluating JumpReLU SAEs, Rajamanoharan et al. (2024b) chose the SAEs from their sweep that were closest the desired sparsity level, but this resulted in SAEs with significantly different sparsity levels being directly compared. JumpReLU SAES use no auxiliary loss function.
14

---

Published as a conference paper at ICLR 2025
A.3 BATCHTOPK SAES
Normalized MSE vs
CE degradation vs k
batchtopk
batchtopk
topk
-0.200
topk
jumprelu
-0.225
jumprelu
Normalized MSE
0.20
CE degradation
-0.250
0.18
-0.300
0.16
-0.350
-0.375
20
40
60
20
40
60
k
Figure 7: On Gemma 2 2B activations, BatchTopK outperforms TopK SAEs across different val- ues of k. Although BatchTopK has better reconstruction performance (left), it only outperforms JumpReLU in terms of downstream CE degradation in the setting where k=16 (right).
In this appendix, provide details about the datasets used, model configurations, and hyperparam- eters for our experiments.
We trained our sparse autoencoders (SAEs) on the OpenWebTen datase which was processed into sequences of maximum of 128 tokens for input into the language models.
All models were trained using the Adam optimizer with learning rate of 3 10-4, 31 0.9, and 32 = 0.99. The batch size was 4096, and training continued until a total of 1 x 10 tokens were processed.
We experimented with dictionary sizes of 3072, 6144, 12288, and 24576 for the GPT-2 Small model, and used dictionary size of 16384 for the experiment Gemma 2B. In both experiments, varied the number of active latents among 16, 32, and 64.
For the JumpReLU SABs, varied the sparsity coefficient such that the resulting sparsity would match the active latents of the BatchTopk and TopK models. The sparsity penalties in the ex- periments on GPT-2 Small were 0.004. 0.0018, and 0.0008. For the Gemma 2 2B model used sparsity penalties of 0.02, 0.005, and 0.001. In both experiments, we set the bandwidth parameter to 0.001.
The results of our experiments on Gemma 2 2B can be found in figure
To confirm that BatchTopK SAEs make use of the enabled flexibility to activate variable number of latents per sample. we plot the distribution of the number of active latents per sample in Figure g We observe that BatchTopK indeed uses wide range of active latents, activating only single latent on some samples and activating more than 80 on others. The peak on the left of the distribution are model activations on the BOS-token. This serves as an example of the advantage of BatchTopK: when the model activations do not contain much information, BatchTopK does not activate many latents, whereas TopK would use the same number of latents regardless the input.
15

---

Published as a conference paper at ICLR 2025
Distribution of Active Features per Sample (BatchTopk)
Mean: 32.00
Median: 31.00
Median: 31.00
4000
Std Dev: 10.42
Min:
Frequency
2000
1000
0
20
40
60
80
100
Number Active Features
Figure 8: Distribution of the number of active latents per sample for BatchTopK model. The peak on the left likely corresponds to BOS tokens, demonstrating BatchTopK3 adaptive sparsity.
A.4 EXAMPLE LATENTS
Figure shows histogram of the maximum decoder cosine similarity for each latent in GPT2-1536 over all latents in GPT2-768. On the right-hand-side, there is cluster of latents with high cosine similarity.
Bricken et al. 2023) the cosine similarity between latent activations measure for latent similarity. However, we use decoder cosine similarity due its lower computational cost and because it captures the latent's effect the reconstruction. Empirically, find high correlation between these two metrics at values of decoder similarity relevant to our stitching experiments (see Figure
10
Maximum Cosine Similarity of Large SAE Features with Small SAE Features
Mean: 0.725
Median: 0.764
Requeecy
20
0.2
0.7
0.8
Maximum Cosine Similanity
Figure 9: Distribution of maximum cosine similarities between decoder weights of latents in GPT2- 1536 and GPT2-768. Many latents in the larger SAE have high similarity to latents in the smaller SAE, but there is also a long tail of novel latents.
Figure I| and Figure 12 show feature dashboards from Neuronpedia Lin 2023
Feature dash-
boards are popular method to visualize and interpret SAE latents, see also Bricken al. (2023). These dashboards show tokens in input sentences that maximally activate certain latent in green, and the tokens they boost (in blue) and suppress most (in red).
16

---

Published as a conference paper at ICLR 2025
Unner Regression on 0.5
0.5
Actiration Cosine Strileriyy
0.6
0.0
0.00
0.25
0.50
Decoder Cosine Similarity
Figure 10: The activation similarity by the decoder cosine similarity on the GPT2-768 and GPT2- 1536 SAEs. At high values of cosine similarity (> 0.5) the coefficient of determination between these is 0.87.
Figure shows an example of latent from GPT-1536 and latent from GPT-768 that have a cosine similarity of 0.99. We see that both of these latents activate strongly on the same inputs, and boost similar logits.
GP12-SPALLL B-RES FS768-18
GPI2-SHALL
B-RES, FS1536-J8 1290
phrases related international relations policies
phrases related legal and political
Issues
sburg
-8.73
regard
1.65
sburg
-0.67
regards
1.65
fair
-0.
regards
1.64
fair
-0.67
Stood
1.64
-0.67
Stood
sov
-0.66
regard
1.62
1.44
President Donald Tranp...d []d)=]jndertext [>YouTube
Inputs from
agi- Aquino said.
the help
56 0WE3 this anti-Israel blas thatäg
agu Aquino the
President Donald Trunp...! - WWel inputs fron lendoftext (>YouTube
açi:
help
Figure 11: Neuronpedia dashboard of example latents with high cosine similarity. Neuronpedia
However, GPT2-1536 has latent for "make sure" that has no counterpart in GPT-768. The nearest latents have decoder cosine similarity of around 0.3, and are shown in
17

---

Published as a conference paper at ICLR 2025
015
(Alecrmpancien
2.78
slways Standingkl Ri
Exes oig 1000
Scheel Pelic bitaria Tex S
-51 S000ES From the to Not SERee
popelationn
12. mont jast ma--
LATTINN The
MARTEES GRANTES Naar
VER bw SISSUISSS
cest provontian and recenory Buntentt states patting sontlar
Trurt soull gets cnty flveer
The charr (00Te 1981
Example GPT2-1536 latent with no similar latent in GPT-768, with the three most similar latents shown. Neuronpedia
Figure 12:
We evaluate the reconstruction performance of the two SAES on inputs where this latent is active and inactive. The reconstruction performance of the smaller SAE is considerably worse on inputs where this larger SAE latent is active, compared to inputs where the latent is not active.
Latent inactive
Latent active
Difference
GPT2-1536
2.225
2.518
0.293
GPT2-766
2.703
3.292
0.589
Averaging this metric across all 657 latents in GPT-1536 that have low maximum cosine similarity with all latents GPT-768, we see a similar pattern (Figure 13)
SAEs feature
3.0
Feature on
2.34
2.0
1.93
0.5
GPT2-1536
GPT2-768
Figure 13: Reconstruction MSE of SAEs on inputs where novel latents in the larger SAE are active and inactive
18

---

Published as a conference paper at ICLR 2025
A.5 LATENT FAMILIES
Count:
Count:
Count:
Count:
Court:
Count:
Figure 14: Connected subgraphs of the bipartite graph of latent in GPT2-768 and GPT2-1536.
A.6 OPEN SOURCE SAE WEIGHTTS
All GPT-2 Small SAEs were trained on the layer 8 residual stream, which was chosen in line with
Gao al. (2024). They were trained for 300M tokens on the Open WebText dataset, which was processed into sequences of maximum of 128 tokens for input into the language models. All models were trained using the Adam optimizer with learning rate of x 10-1, 31 0.9, and 32 0.99. The batch size used was 4096 and all were trained with sparsity penalty of 8 x 10 The GPT-2 SAES are available on Neuronpedial We also use two of the Gemma Scope SAEs [ieberum et al. 2024) trained on Gemma 2 2B (Team et al. 2024) with dictionary size 16384 and 32768. We used the Transformerl.co (https: /tpp::/traas //ttttrraasaas
wwweennrrrgtthuuueee git.ub......... io/ Trans formerLens implementations of GPT-2 and Gemma 2B. CELR is the cross entropy loss recovered from either zero or mean ablation.
19

---

Published as a conference paper at ICLR 2025
Table 2: Properties of the SAEs used in this study.
Name
Model
Dict. size
L.0
MSE
CELR Zero
CELR Mean
GPT2-768
gpt2-small
768
35.2
2.72
0.915
0.876
GPT2-1536
gpt2-small
1536
39.5
2.22
0.942
0,915
GPT2-3072
gpt2-small
3072
42.4
1.89
0.955
0.937
GPT2-6144
gpt2-small
6144
43.8
1.63
0.965
0.949
GPT2-12288
gpt2-small
12288
43.9
1.46
0.971
0,958
GPT2-24576
gpt2-small
24576
42.9
1.33
0.975
0,963
GPT2-49152
gpt2-small
49152
42.4
1.21
0.978
0.967
GPT2-98304
gpt2-small
98304
43.9
1.14
0.980
0,970
GemmaScope-16330
Gemma 2 2B
16384
43.4
1.72
0,983
0,980
Gemma 2 2B
32768
41.5
1.59
0.985
0,982
A.7 STITCHING EXPERIMENTS
Classifying Novel Latents based on Max Cosine Similarity
Thus Pottive tate
0.6
0.4
(11-2 2012 1ast WIC 0.801
0P1-2 2072 (MIC 0.83
GPT-2: 12308 0244 SAUC 0.851
C1753 20536 129999 (MUC
GPT
0.525
D.8c)
0.2
0.4 False Posttve Roto
0.6
1.0
Figure 15: Based on the maximum cosine similarity is possible to predict the direction the effect of adding latent to another SAE. The Receiver operating curve (ROC) is created by varying the threshold maximum cosine similarity to classify latent as novel latent or reconstruction latent.
0.000
0.6
-8.05
026
0.2
-8.75
-0335
700 1536
2536 in 3072
3072 6144
6344 12288
760 -9 3536
1536 i 3072
3072 4244
6144 IN 32288
Figure 16: Effects on MSE and LO when swapping reconstruction latents from larger SAES to smaller ones. Swapping latent structures generally increases the MSE but almost always decreases LO. Outliers are not shown. The percentual effects per swap get smaller for larger models as the effects are distributed over more swaps.
20

---

Published as a conference paper at ICLR 2025
Cumulative effect on NSE adding features
GPT-2: 1536->
GFT-2: - 1536
GPT-2:
GPT-2: 12288 6144
300
>-077
>.077
>.B.7
<=077
<=
2V9 1o KISE chenge
AV3 to nocP shorgg
A00 90 20d chunge
Number features soded
Number of features soded
Number features added
Number of features added
GPT-2: 24575 :2288
GPT-2: 49132 >> 24576
GPT-2: 90304 49132
Gemma: 32768 -> 16384
300
<- D.7
<- 0.7
<-m 0.7
--- 0./
201 Pa NSC change
Arg to RtS shange
A02 7% ROU: change
AVg ti
20
5000
10000
15000
20000
2C000
30000
20000
60000
10000
20000
Figure 17: Percentage change of MSE of adding in latents from larger SAE to smaller SAE in a random order. Adding in all the latents with cosine similarity I 0.7 from GPT-1536 GPT-768 reduces the MSE by almost 10%.
Proportion reconstruction ratios,
Proportion of reconstruction ratios, SAES same size
1.20
0.977
3,000
0.251
0.592
1.000
0.227
0.630
0.540
0.94
1,000
0.100
0.477
0/613
1.000
09911
0900
0.60
1,000
D131
0.539
C1213
0.392
LU30
01928
0.922
03542
1,94
0.
(b) Proportion of reconstruction latents in SAEs of different size. X and Y labels exponential expansion factors, or equivalently row indices in Table
(a) Proportion of reconstruction latents in GPT-2 SAEs with 3072 latents. and labels are ran- dom training seeds.
Figure 18: Proportion of reconstruction latents between GPT-2 SAES of the same size and different sizes.
We compared the proportion of reconstruction latents between SAEs the same size and SAES of different sizes. We found that when comparing SAEs with the same size, 94% of the latents reconstruction latents (cosine similarity > 0.7). These results are displayed in Figure I8
A concern when stitching two different SAEs is the choice of ddee, However, in practice find that the ddee of SAEs trained on the same latents are very similar (minimum cosine similarity of 0.9970, differing by less than 0.1 % in magnitude). Figure 19 that the decoder biases be
21

---

Published as a conference paper at ICLR 2025
Sompping
7.4
2.201
105
2.202
2.231
2.219
2.327
2.240
2.211
Repoeera Source
Le2
1.9:
1911
L-23
800
1000
101
1,319
1:326
1:412
1:511
1.927
106
1
1.40
1,31
1.344
1.19
3317
1347
3240
1225
1231
2344
1366
12
700
1556
3072
E344
22203
24576
40232
00204
b Source
Figure 19: Reconstruction performance (MSE) of SAEs of different sizes when their byles are replaced with the bdees of SAEs with different dictionary size.
exchanged in the stitching process with negligible impact the reconstruction performance of an SAE.
A.7.1 STITCHING GEMMA SCOPE SAES
In order to validate that the SAE stitching results are not an artifact of GPT-2 small or the SAEs that trained, applied the same methods to the open-source Gemma Scope SAEs. In particular, compared two SAEs trained on the residual stream of layer 12 of Gemma 2 2B. The first SAE has dictionary size 16384 (average LO 41) and the second SAE has dictionary size 32768.
We find a lower threshold for distinguishing novel features from reconstruction features (0.4). Using this threshold, can also smoothly interpolate between SABs of different sizes trained on Gemma- 2-2B.
Max Cosine Similarity vs Effect on MSE for Large SAE Features
when added
10
Decreases MSE when added
10-3
Effect on MSE when Added
10.5
-10-5
-10-2
Max Cosine Similarity with Small Features
Figure 20: Change in MSE when adding each feature from GemmaScope-322 GemmaScope-Iie plotted against the maximum cosine similarity of that feature to any feature in GemmaScope- 16k. Features with cosine similarity less than 0.4 tend improve MSE, while more redundant features hurt performance.
22

---

Published as a conference paper at ICLR 2025
Transition from Gemmascope 16k 32k
1.74
1.70
Squared
GemmaScope 16k MSE
GemmaScope 32k MSE
17500
20000
22500
25000
27500
30000
32500
Figure 21: By first adding in novel latents from Gemma Scope 32k Gemma Scope 16k and replacing the remaining latents with their similar latents in the larger SAE, we interpolate between two SAE siezs.
A.8 METASAE ADDITIONAL FIGURES
WAtrn-GAE similenlyy
Closss SAC-latent Mota-letemi
Closes letent SAS-49152 letent
30
36-3913
14-
23
Denkk
2.2
3.5
85-
80
6.0
0.2
0.4
0.4
0.5
0.4
0.6
Max Contre Stmilartty
M4x Coutiie Simiiiity
Mac COUTI Stilsrly
Figure 22: Cosine similarity between SAE latents and meta-SAB SSE latents. Note the high maximum cosine similarity between latents from meta-SAE with 2304 latents, and standard SAE with 3072 latents.
MetaSAEs Reconstruction MSE with SAE decoder directions
0.00200
0:00191047
0.00175
Reconcruction MSE
0.00150
0.00125
0.00075
0.00057942
0.08075525
0.00063951
0.00025
0.00000
MetaSAE
MetaSAE using SAE decoder directions
MotaSAE using SAE decoder directions and fine Ine-tuned encoder
MetaSAE with random decader drrctions
Figure 23: Change in reconstruction performance of a meta-SAB when its decoder directions are replaced with the similar decoder direction from an SAE with similar dictionary size.
23

---

Published as a conference paper at ICLR 2025
A.9 INTERPRETABILITY EXPERIMENTS
In this paper, we demonstrate that larger SAES may learn more narrow, composed concepts in or- der improve sparsity rather than just learning concepts that missing in smaller SAEs. Here, we provide some experimental results on sparse probing Gao et al. (2024) and concept removal benchmarks (Karvonen al, 2024).
A.9.1 SPARSE PROBING
Similarly Gao et al. 2024), features in our SAEs. we expect a specific feature to be discovered by an SAE, then a metric for autoencoder quality is simply checking whether these features are present as latents. We do this by training a I-dimensional logistic probe on the activations of the SAE to predict the presence of the feature. We use the benchmark datasets included in (Karvonen al. 2024) our evaluation. These cover a range of domains, for example predicting the sentiment of Amazon reviews or predicting the language of the text from the SAE activations.
sparse probes to evaluate the presence of known ground-truth
In our experiments we use probe that uses only single SAE latent in its prediction. The results of these experiments are visualized Figure 24 They show that the relationship between the size of the SAE and the evaluation accuracy is complex and dataset dependent.
Sparse probing eval accuracy by SAE dictionary size
Benchmark Datasel
0.80
LabHC/bias, in bios, class set1
LabHC/bias bios,
LabHC//bas, in bios, class set3
Canrageriamazon, reviees__eeeeee carrss_cculey_aaae _landd
canrager/amazon, rwveess mccullyy lands sentiment
0.75
addeaarooigihhbb code fancyshx/ag news
Helsinki NLPieuropari
0.65
1536
3072
61.4
12288
24576
49152
SAE Dictionary Size
Figure 24: Sparse probing evaluation accuracy by GPT-2 SAE dictionary size across 8 benchmark datasets, with sparse probe using the top latent.
24

---

Published as a conference paper at ICLR 2025
Sparse probing eval accuracy by SAE dictionary size
Benchmark Dataset
LabHicbias, in in_tios__ee_ ci set1
0.950
LabHi//iis, in bios, class, set3
Canrageriamzzon reviews incuuey_lands
canrageriamacon, reviews mcaullyy_ands,, pentimeeenmment
oddeaarotigthub.com code
Eicyshwwagg news
Helsmnn-NL NLPierrrlll
0.875
0.850
0.825
0.800
49152
SAE Dictionary Size
Sparse probing evaluation accuracy by GPT-2 SAE dictionary size across 8 benchmark datasets, with sparse probe using the top 50 latents.
Figure 25:
A.9.2 CONCEPT REMOVAL
In the SHIFT method (Marks et al. 2024). human evaluator debiases classifier by ablating SAE latents. Karvonen et al. 2024) operationalizes SHIFT into an evaluation for SAE quality, Targeted Probe Perturbation, by training probes model activations and measuring the effect of ablating sets of SAE latents on the probe accuracy. Ablating a disentangled set of latents should have an isolated class probe, while leaving other class probes unaffected.
causal effect
They consider a dataset with M classes. For each class with index = 1,..., M they select the set L, of the most relevant SAE latents. To select those latents, binary probes are trained detecting the concept e from model activations. The attribution score of latent with index on concept e is given by
(LL, = (Lpos - (d:-) P)
(11)
where denotes the batch activations latent di denotes the SAE decoder vector correspond- ing to 1, P is the weight matrix of the binary probe, Lpos is the mean activation of the latent for inputs of the targeted concept, and Tineg is the mean activation for inputs unrelated to the concept. those select most relevant.
The latents with the highest attribution
For each concept c;, they partition the dataset into samples of the targeted concept and random mix of all other labels. They define the model with probe corresponding class cj with j = 1,..., M as a linear classifier Cj. Further, Ci,; denotes a classifier for ej where latents L; are ablated. Then, they iteratively evaluate the accuracy Ai, of all linear classifiers Ci,j on the dataset partitioned for the corresponding class cj. The targeted probe perturbation score
STPP mean (i-)) (Ai.)) mean(iss)) (Ars)
(12)
represents the effectiveness of causally isolating a single probe. Ablating a disentangled set of features should only show significant accuracy decrease if namely if the latents selected for class i are ablated the classifier of the same class i, and remain constant i +
The results of this evaluation by SAE dictionary size is displayed in Figure 26 Whilst here there is a general downward trend, the accuracy not monotonically decreasing with SAE size.
25

---

Published as a conference paper at ICLR 2025
TPP eval accuracy by SAE dictionary size
Benchmark Dataset
0.12
LabHCibias in bios class_set__g top
canrageriamzzon_ reviews wwwpm/mccalaaaaae 1and5_ tpp
0.10
Accuracy
0.08
0.06
0.04
0.02
Dictionary Size
Figure 26: TPP evaluation accuracy by GPT-2 SAE dictionary size across 2 benchmark datasets, ablating up to 50 latents.
26


Reviews:
[{"review_id": "LSQyX1ke5s", "reviewer": "Reviewer_jKs2", "summary": "This paper wrestles with the question of whether sparse autoencoders (SAEs) of different sizes learn the same set of \"canonical\" or \"atomic\" units. The paper approaches this question from a couple of directions: \n\n1. First, the authors investigate how SAEs of different sizes can be \"stitched\" together. They find that some features from a larger SAE, when added to a smaller SAE, improve loss (novel latents), and others make loss worse (reconstruction latents). It turns out that there is a rough relationship between, for a given large-SAE latent, the maximum cosine similarity between that latent and the small-SAE latents, and whether adding that latent improve or huts performance. Latents which are dissimilar from all small-SAE latents improve performance of the small-SAE when added, and latents which are quite similar to a small-SAE latent hurt performance of the small-SAE when added. This makes sense -- \"novel latents\" are features which the small SAE has not learned, and \"reconstruction latents\" are features which the small SAE has already represented in some manner. The authors describe a procedure by which latents from a large SAE can be added to a smaller SAE -- novel latents can be added, but reconstruction latents must replace latents in the small SAE. This procedure allows one to interpolate between SAEs of different sizes while continuously improving reconstruction error. Nice! Lastly, the authors note that the \"reconstruction latents\" are not identical to the latents they replace in the smaller SAE, and also sometimes have high cosine similarity with multiple latents in the smaller SAE. One explanation of this is that some large-SAE latents are in fact linear combinations of multiple latents in the smaller SAE. This calls into question whether latents learned by SAEs are atomic.\n\n2. Second, the authors attempt to decompose SAE latents into more atomic features with \"meta-SAEs\". Meta-SAEs attempt to represent SAE  latents (decoder vectors) as a sparse linear combination of some smaller set of features. Interestingly, the authors report that many meta-SAE features seem interpretable, and many interpretable latents in the original SAE decompose into interpretable combinations of meta-SAE features! For instance, a dedicated \"Einstein\" latent from the original SAE can be approximated as a linear combination of meta-SAE latents for \"Germany\", \"prominent figures\", \"scientist\", \"space and galaxies\", and \"starts with E\". The authors demonstrate that meta-SAE latents are similar to the latents learned by a similarly-sized base SAE.\n\nFor their meta-SAEs, the authors trained a new SAE variant called BatchTopK SAEs. While BatchTopK SAEs are not the main contribution of the paper, the authors test them against standard TopK and JumpReLU SAEs, and find that BatchTopK beat TopK SAEs across settings they evaluated, but don't always beat JumpReLU SAEs. BatchTopK SAEs are not the primary contribution of the paper, but are a nice bonus.", "questions": "* It's interesting that the reconstruction error falls as much as it does when the reconstruction latents are stitched in. If reconstruction latents were exactly linear combinations of more atomic latents learned by the smaller SAE, I'd expect that replacing those atomic latents with the reconstruction latents would yield the same reconstruction error, but at a lower L0. Instead reconstruction tends to fall *more* steeply during reconstruction latent stitching vs. novel latent stitching. Do you have a guess as to how the reconstruction latents relate to the latents they replace? Perhaps they are a combination of the small-SAE latents but with other additional features included too? I wonder if a feature-manifold explanation might also be worth considering here (Engels et al. 2024), where reconstruction features are more densely covering a feature manifold that corresponding latents in the small-SAE are more coarsely covering. In this sort of model, latents aren't just combinations of atomic latents, and maybe there is no good definition of \"atomic latent\" when features are multi-dimensional. What do you think?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Overall, the paper addresses an important question about SAE features with reasonable experiments and presentation. This issue, of whether SAE features are \"canonical\" or \"atomic\", could have implications for how we scale up SAEs and also for how we use their latents for model interventions, circuit analysis, etc. Overall, the presentation in the paper is clear and the experiments are original and well-executed. Some particular strong points:\n* I think Figure 5 is quite compelling. It makes a lot of sense that the L0 would rise as novel latents are added, and it's very cool to see continuous curves like this as one \"interpolates\" between two SAEs.\n* I think that Figure 6 is also quite compelling in showing that meta-SAE latents are pretty similar to similarly-sized base SAE latents.", "weaknesses": "* I could not find reported values of the reconstruction loss that meta-SAEs obtain in reconstructing the base 49k-latent SAE latents. How precisely do meta-SAEs actually reconstruct the latents? If they are only a very weak approximation, what would that say about the hypothesis that large-SAE latents are linear combinations of more atomic latents?\n* Some minor grammatical and presentation issues: \"vertexes\" -> vertices, the left quotation marks in the meta-SAE section should be fixed, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kCSvbWmd5", "reviewer": "Reviewer_pzvM", "summary": "The paper discusses the use of sparse autoencoders (SAEs) to examine how activations in a transformer relate to concepts in natural language. The authors examine the relationship between small SAEs and larger SAEs (a larger number of latent states) to assess whether larger SAEs contain more fine-grained information. Additionally the authors introduce a sparsity objective that enforces sparsity across a batch of feature vectors, rather than at for individual feature vectors.", "questions": "*** page 4\n\nI don't really agree with the phrase \"circuit\". To me this suggests some end to end explanation, whereas the reality is that a small part of a transformer is being examined (the activations in a single layer).\n\nI can't find any information on how the SAE is trained. What is the optimiser and how are the hyperparameters (eg \\lambda) set?\n\n*** page 5\n\nWhy choose layer 8 of GPT2? \n\nEquation 5 has an asymmetry. Why is the bias from decoder 0 used, rather than decoder 1, or a combination of the two?\n\nThe motivation for stitching isn't clear to me. The features and decoders learned are optimised for each SAE separately. Why would combining them in a suboptimal way mean anything? Why not fix the features f0, f1 from each encoder but then learn optimal decoder weights W_{01} for the combination of these feature (this is a simple quadratic optimisation problem). Similarly, in the subsequent discussion, I don't follow why an increase or decrease in MSE is of any significance.\n\nIf one wishes to understand whether latents in larger SAEs are finer grained versions of latents in smaller SAEs, would it not be feasible to look at a sentence that coactivates two features, one feature from the smaller SAE and one from the larger SAE?\n\nIn figure 5 it's not clear to me what the average L0 means -- what is being averaged over here?\n\nAs far as I understand the SAE objective is non-convex, meaning that there is no guarantee of finding the global optimum. It's therefore also quite possible that different SAEs are simply finding different latents simply because of finding different local minima.\n\nIsn't it clear that (local optima aside) a larger SAE will always find features that are missed by smaller SAE? I'm not sure I follow the argument from line 355 onwards.\n\n*** page 8\n\nPlease add more clarity around treating the latents W_i^{dec} (why is W in boldface)? W_i^{dec} is a scalar quantity, the ith component of the d-dimensional vector W^{dec}. What does it mean to take scalars \"training data for our meta-SAE\"? The directions W don't convey information about which directions are simultaneously activated. I would have thought it would make more sense to treat the feature vectors f(x) as the entities for learning a meta SAE.\n\n\n*** page 9\n\nThe BatchTopK function is fully defined. In line 465 it states that the function selects the top K activations, suggesting that this is a mapping from activations to indices. I suspect that the authors mean that the function should return zero for all activations that are not in the top K highest positive values, and is the identity function otherwise.\n\nThe introduced batch method is used only during training, with a different non-linearity used during \"inference\". This seems quite strange and it's not clear to me how to justify this.  A potential issue not mentioned is that it's quite possible that the batch approach means that some input sequences will have entirely zero activation.\n\nIt's hardly surprising that the BatchTopK SAE has lower SAE than TopK SAE since BatchTopSAE imposes fewer constraints on the objective. I'm not sure why this would be seen as \"outperformance\".\n\nI'm also unclear as to why BatchTopK SAE is being discussed. Is this method used in all the previous experiments in the paper, or is this a separate piece of work orthogonal to the other contributions of the paper?\n\n*** page 10\n\nThe conclusion \"These findings suggest that there is no single SAE width at which it learns a unique and complete dictionary of atomic features that can be used to explain the behaviour of the model.\" is interesting and (perhaps) not surprising.\n\n\n*** supplementary material\n\nFigure 11 isn't well explained. Please explain what is being shown here.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper attempts to understand how activations deep within a transformer might correspond to higher level natural language concepts. This is an interesting topic. The conclusion is also interesting, namely that sparse autoencoders may not form a complete set of explanations, with larger SAEs potentially containing more fine-grained information.", "weaknesses": "The paper is quite presumptive about readers knowledge of the topic, lacking in clear explanations in places. Whilst the paper does a reasonable job of explaining sparse encoders, it doesn't explain how these are used to find actual \"inputs\" that activate particular features. This isn't explained in the paper and I had to read the cited papers to understand this. There is some description in section 5.1 (page 8) but this is too late for the reader unfamiliar with this area to understand the paper. \n\nI also found it quite hard to follow the authors reasoning in places and it's not clear to me why stitching might provide insight.  Overall, the methods used in the paper are rather straightforward and I would therefore have hoped for a very clear presentation to make up for a lack of technical contributions. The Meta SAE isn't very well explained and the BatchTopK is idea is simple but now well motivated and it's unclear to me how this connects to the rest of the paper. \n\nTry to be consistent with spelling rather than mix eg color, colour. Also opening parentheses are incorrectly used in places (see lines 369 onwards).\n\nI do think there is something potentially insightful in this work, but the paper needs to be more clearly presented.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ngu8EwnlYX", "reviewer": "Reviewer_HcSs", "summary": "In this work, the authors introduce two new methods for analyzing latents in SAEs. The first, SAE stitching, enables comparison of latents across SAEs of different sizes by categorizing latents in larger SAEs as either novel or reconstruction latents found in smaller models. The second, Meta-SAE, decomposes decoder directions into interpretable meta-latents They also propose the BatchTopK SAE, a variant of the TopK SAE that forces a fixed average sparsity. By applying these methods, they obtained empirical results that suggest that SAEs are unlikely to learn canonical sets of units.", "questions": "- Is there any particular reason to omit the bias term of the decoder 1 in equation 5?\n- How does the stitching SAE handle shared reconstruction latents during the swap process? Since some reconstruction latents in larger SAEs are composites, they may contain information from multiple smaller SAE latents. For example, in the case of colored shapes, how would the model swap the \"square\" or \"blue\" latent if these are entangled in composite latents in the larger SAE? Would they need to be swapped simultaneously? \n- Would it be possible to introduce some form of supervision in meta-SAEs? Do you have any intuition as to whether this might be beneficial, considering that the ultimate goal is to interpret model activations? Following a concept bottleneck approach, one could directly associate human-interpretable meta-latents with the representations learned by larger SAEs (although I assume the main limitation is defining labels/concepts for the large dictionary sizes considered).\n- For BatchTopK, have the authors examined how varying values of k impact the semantics of the learned latents? Do the \"concepts\" learned under a stronger sparsity constraint become more abstract (as they need to explain a given activation with fewer latents) for a fixed dictionary size?\n- Have the authors extracted any intuition on how should the dictionary size be adjusted to tailor the SAE to the requirements of an specific analysis?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation and methods are explained clearly and intuitively, with helpful examples.\n- The authors contextualize their approach by discussing relevant state-of-the-art methods.\n- Several experiments are conducted to assess the methods' performance, with comparisons to state-of-the-art baselines and detailed experimental information.\n- An interactive dashboard is included to explore the latents learned with meta-SAEs.", "weaknesses": "- Including a discussion on the potential limitations of the proposed approaches would be valuable.\n- It would also be helpful to have an expanded discussion on assessing the quality of the representations and adapting the dimensionality of the SAE to suit the requirements of different analyses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3UExQd5dBL", "reviewer": "Reviewer_9gei", "summary": "This research concerns SAE, an assessment tool specialized for GPT-type models, espeically those used as LLM. \nSAE is applied on the activation of LLM to infer the captured information, and at least in this paper, its wellness is \nevaluated based on the MSE of the reconstruction. \nThe novelties of of this research are as follows.\n1. It proposes SAE stitching, a tool to compare large SAE against Small SAE.  With SAE stitching, one can use the \nchange in MSE to assess how the larger SAE's weight direction relates to the smaller SAE's weight direction (intersection of notions), \nas well as the uniquness of the notion captured by the larger SAE. \n2.  It proposes Meta SAE, which applies still another SAE on the top of an already established SAE. It is used to obtain the \nmonosemantic latent.\n3.  It proposes BatchTopK, a variant of MetaSAE which achieves SOTA in terms of architecture. \nBase on the observations made by these claimed novel technique set, the paper advocates the need to carefully choose the size of the \nSAE as well as the need to compare SAEs of different sizes for more semantically meaningful analysis.", "questions": "1,  Figure 4, it is said that \"Features with cosine similarity less than 0.7 tend to improve MSE\".  Do you mean 0.6 or less?  In the similar regard, thsi rendency is much less clear in Figure 18.  Would you please make comments regarding GemmaScope32k? It is said that \" Using this threshold, we find that our feature stitching methods work on these SAEs as well.\" Can you elaborate on what is meant by \"well\" in this context? \n\n2,  What are \"active latents\"?  The reviewer presumes that these are the latents that are not \"zero-ed out\" by the training with the sparsity constraint, but it would help to clarify its precise mathematical meaning.\n\n3, In Figure 6 and experiments regarding this figure, MetaSAE with N latents are evaluated.  Now, the MetaSAE is presented as the application of SAE on the latents of SAE.   Would you please clarify \"on what SAEs\" the MetaSAE were applied in these experiments? For example, from SAE of what size \"the Meta SAE with 2304 latents\" was obtained?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This research provides thorough and solid experiments that are in alignment with \nthose of the original SAE paper. \n- This research furthers the understanding of SAE in application to LLM; considering that SAE is considered an important probing unit in the understanding of how LLM processes the information, this research answers an important question regarding (1) how the probing unit deomposes the LLM activations and (2) how its size and design affects the analysis.", "weaknesses": "- The reviewer is a little unsatisfied with how the research is positioned. In the introduction and in the abstract, the paper is presented as if the research topic pertains to an analysis of a decomposition of the activations of a \"general\" neural network into features, while in fact the SAE introduced by [Bricken et al] and [Cunningham et al] is a probing tool for a specific 'organism' that is LLM.  The term 'language model' is mentioned only at the part 3 of the contribution, leaving an impression that the paper investigates a very general feature analysis that is 'also' applicable to LLM.  The research also experiments with LLM exclusively as well. \nWhile the keyword SAE may automatically link to the behavioral study of LLM in the minds the the readers that are actively involved in LLM behavioral research, the reviewer feels that it shall be clearly stated/emphasized in both the introduction and the abstract that this research is about LLM (which is merely one genre of ML research).  Otherwise, the reviewer believes the applications other than LLM shall be presented in the paper.\n\n- Another concern is that, while the research clearly furthers the understanding of the SAE, an important probing tool for LLM, the research does not relate the claimed novelty to the SAE's probing \"capability\". For example, [Cunningham et al] quantifies the SAE's ability to localize a specific model behavior in Indirect object Identification (IoI) task, thereby evaluating the goodness of the probing conducted by SAE.  Meanwhile, the paper evaluates the goodness of SAE with the reconstruction error. While the reviewer values the thoroughness of independent experiments, the reviewer also feels that the authors did not intend their study to be interpreted as an investigation of a probing tool for the sake of probing tool itself.  The reviewer feels that the research can be justified if the authors can add analysis of how their new toolset can be used to better the probing capability of SAE in terms of IoI, for example, or to actually better uncover a features that are causally responsible for counterfactual behavior in LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)