Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Authors: Patrick Leask
Neel Nanda
Noura Al Moubayed
Venue: ICML 2025 poster (2025)
Keywords: sparse autoencoders
mechanistic interpretability
sparse dictionary learning
Primary area: deep_learning->large_language_models
TL;DR: Inference-Time Decomposition of Activations is a lightweight alternative to sparse autoencoders for decomposing LLM activations

Abstract: Sparse Autoencoders (SAEs) are a popular method for decomposing Large Language Model (LLM) activations into interpretable latents, however they have a substantial training cost and SAEs learned on different models are not directly comparable. Motivated by relative representation similarity measures, we introduce Inference-Time Decomposition of Activation models (ITDAs). ITDAs are constructed by greedily sampling activations into a dictionary based on an error threshold on their matching pursuit reconstruction. ITDAs can be trained in 1% of the time of SAEs, allowing us to cheaply train them on Llama-3.1 70B and 405B. ITDA dictionaries also enable cross-model comparisons, and outperform existing methods like CKA, SVCCA, and a relative representation method on a benchmark of representation similarity. Code available at https://github.com/pleask/itda.

Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models
Patrick Leask Neel Nanda
Noura Al Moubayed
Abstract
source SAEs, such as Lieberum et al. (2024) and He et al. (2024), which are currently limited to models with up to 27B parameters.
Sparse Autoencoders (SAEs) are a popular method for decomposing Large Language Model (LLM) activations into interpretable latents, how- ever they have a substantial training cost and SAEs learned on different models are not directly comparable. Motivated by relative representation similarity measures, we introduce InferenceTTime Decomposition of Activation models (ITDAs). IT- DAs are constructed by greedily sampling activa- tions into a dictionary based on an error threshold their matching pursuit reconstruction. ITDAs can be trained in 1 1% of the time of SAFs, allowing us to cheaply train them on Llama-3.1 70B and 405B. ITDA dictionaries also enable cross-model comparisons, and outperform existing methods like CKA, SVCCA, and relative representation method on a benchmark of representation similar- ity. Code available at github.commpleask//ee
Furthermore, comparing SAEs trained on different models is challenging: their parameters are learned from model activations, which vary for the same input between models even of the same family. Lan al. (2024) proposes that SAEs reveal universal feature spaces across LLMs by us- ing centered kernel alignment (Kornblith et al., 2019) to compare the decoders of SAEs. The problem of aligning the representation spaces of different LLMs is also funda- mental to absolute measures of representation similarity such as SVCCA (Raghu al., 2017) and CKA (Kornblith et al., 2019), which is addressed with relative representation measures by Moschella et al. (2022), who use the cosine similarity of representations with respect to the represen- tations of a fixed random set of anchor inputs in different models.
In this paper, we introduce Inference- Time Decomposition of Activations (ITDA) lightweight alternative to SAES. Similarly the relative representation method of Moschella et al. (2022), we construct a dictionary of anchors to which we relate activations by their cosine similarity. However, we extend this method by decomposing activations into this dic- tionary using matching pursuit, a method for inference- optimization. As with SAEs, this approach finds sparse decompositions of test LLM activations. In contrast to Moschella et al. (2022), who use random anchor points, we form the dictionary by initializing with the most fre- quent activations, and greedily adding training activations that are most poorly reconstructed by the dictionary so far.
1. Introduction
Mechanistic interpretability aims to reverse-engineer neu- ral networks into human-interpretaale
algorithms (Olah Meng et al., 2022: Geva et al., 2023; Nanda et al., 2023; Elhage et al., 2021). Sparse autoencoders (SAEs) have recently emerged as a promising alternative for decomposing L.LM activations into dictionary of in- terpretable and monosemantic latents (Cunningham al., 2023; Bricken et al., 2023; Gao et al., 2025; Marks al., 2025; Lieberum et al., 2024; Rajamanoharan et al., 2024). However, training SAES is computationally expensive: it requires model activations for hundreds of millions or even billions of tokens, which are expensive to collect for large models; and the parameter count of SAEs can exceed that of the models which they are applied (Sharkey et al., 2025). As result, much academic research relies on open-
et al., 2020;
ITDAs can be trained on a million tokens to the same perfor- mance as SAEs trained on hundreds of millions of tokens, resulting in a proportional decrease in training times. GPT-2 (Radford et al., 2019) SAES take hours to train (Leask et al., 2025), whereas ITDAs can be trained in minutes to similar reconstruction performance. This means we are able to train ITDAA on 70B and 405B parameter LLMss: an order of mag- nitude greater than the largest models on which open-source SAES have been trained. The cross entropy loss degradation when replacing activations with their ITDA reconstructions is similar slightly worse to using SAE reconstructions
! Department of Computer Science, Durham University. Corre- spondence to: Patrick Leask <patrickaalcaske @ @aaissk@@@mmmlooooommm gmail.com>>
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Inference-Tinee Decomposition of Activations (ITDA)
Prompt and decompose The mouse sat on the carpet
Embadding to decompose
1001101101001110001111100
Partial LLN
Matching Pursuit
Sporse prompts token references
Sparse code
enbeddings
The cat sat on the carpet. +0.28 The mouse ate the cheese. +1.37 The cat sat the carpet
-0.74 1101011100011110001
+8.16 10110101110000011100
+1.37 1010100010001000111
Figure 1. Stylized system-level diagram of inference- time decomposition of activations. To decompose a target token in a prompt (1), we compute activation at certain point in the LLM (2), example after layer 8 GPT-2. use matching pursuit to decompose the activation into sparse code of other activations collected from the same point in the LLM (3). Our dictionary atoms are labeled with the sparse code (4).
prompt and token which activation was collected, which allows direct interpretation
on Pythia models (Biderman et al., 2023), however substan- tially worse on Gemma-2 (Team et al., 2024).
significant advantages in efficiency and scalability:
ITDAs are 100x faster to train than SAES, and only require around 1% of the training tokens. As result, we trained them on 70B and 405B LLMs, an order of magnitude larger than any LLMs on which open source SAEs have been trained.
SAE latents are learned by gradient descent, meaning that different models have inherent
latents learned relation. On the other hand, an ITDA latent is the activation of model at certain point for specific prompt and to- ken. So, although the activations of the two models cannot be directly compared, we can compare the prompt-tokens pairs constituting each dictionary. We use this to compare the representation spaces of different models. In particu- lar, introduce ITDA dictionary difference measure for representation similarity based on the Jaccard similarity, or intersection over union (IoU), and demonstrate that it achieves state-of-the-ar performance on layer-mand matching task due to Kornblith et al. (2019), as well as reproducing results on layer freezing from Raghu et al. (2017). Since ITDA dictionaries are useful for interpretability and their differences reflect changes in model representation spaces, exciting potential for their in identifying behav- ioral differences between models, proposed in (Lindsey et al., 2024). For example, finding concerning changes in model behavior when performing chat fine-tuning, such as increases in scheming or sycophancy. However, we do not explore model diffing in this paper, leaving it for future work.
They achieve similar reconstruction performance to SAEs on some models, though worse on others, and have comparable automated interpretability scores.
ITDA dictionaries transfer readily between LLMs in way that SAEs do not; and we use this to construct representation similarity metric that state-of-the-art a benchmark for similarity indices.
ITDA atoms have interpretable labels, i.e. the prompt and token from which the activation taken. This gives their atoms some inherent interpretability, unlike SAE latents.
2. Related Work
In this section, discuss the recent advancements in ap- plying sparse dictionary learning methods to interpret the representations of LLMs. We focus on SAEs, rather than other sparse dictionary learning methods, as they are the primary subject of the mechanistic interpretability literature (Bricken al., 2023; Cunningham al., 2023; Dunefsky
In summary, introduce Inference-Time Decomposition of Activations (ITDAs) as alternative to SAEs, offering
2

---

Inference-Ting Decomposition of Activations (ITDA)
et al., 2024; Gao et al., 2025; Lan et al., 2024; Lieberum et al., 2024; Lindsey et al., 2024; Makelov et al., 2025; Marks et al., 2025; Leask et al., 2025). We provide glos- sary of terms in Appendix A.I.
dictionary learning and inference-time approaches outside of autoencoders. Classical methods such K-SVD (Aharon et al., 2006) and Matching Pursuit (Mallat & Zhang, 1993) iteratively choose dictionary elements to represent a signal under an tn or l constraint. These approaches typically proceed by directly solving a sparse coding objective (e.g.. via Orthogonal Matching Pursuit or Iterative Shrinkage- Thresholding methods like FISTA (Beck & Teboulle, 2009)) instead of learning an explicit encoder network. In many computer vision tasks, these traditional dictionary-learning techniques have shown good performance on denoising and inpainting (Elad & Aharon, 2006), but can be computation- ally expensive at inference time. More recent methods have also investigated structured sparsity (e.g. group sparsity) or used deep dictionary learning frameworks that unroll sparse coding iterations into learned networks (Gregor & LeCun, 2010). The method we propose in Section 3 has more in common with these approaches than the SAFs used so far in mechanistic interpretability.
2.1. Sparse Autoencoders (SAEs)
Sparse dictionary learning is the problem of finding de- composition of a signal that is both sparse and overcomplete (Olshausen & Field, 1997). Lee et al. (2007) applied the sparsity constraint to deep belief networks, with SAEs later being applied to the reconstruction of neural network repre- sentations (Bricken et al., 2023: Cunningham al., 2023). In the context of LLMSs, SAFs decompose model represen- tations X E _ into sparse linear combinations of learned latents. It is hoped that these latents correspond to monose- mantic and interpretable features of the representations, and that the sparse codes for a specific representation are also interpretable.
An SAE consists of an encoder and decoder:
2.2. SAEs for Mechanistic Interpretability
SAEs have been demonstrated to recover sparse, monose- mantic, and interpretable features from language model representations (Bricken et al., 2023; Cunningham et al., 2023; Templeton, 2024; Gao et al., 2025; Rajamanoharan et al., 2025; 2024), however their application to mechanistic interpretability nascent. After training, researchers of- ten interpret the meaning of SAE latents by examining the dataset examples on which they are active, either through manual inspection using features dashboards (Bricken et al., 2023) or automated interpretability techniques (Gao et al., 2025). SAEs have been used for circuit analysis (Marks et al., 2025) the vein of (Olah et al., 2020; Olsson et al., 2022): to study the role of attention heads in GPT-2 (Kissane et al., 2024a); and to replicate the identification of circuit for indirect object identification in GPT-2 (Makelov et al., 2025). Transcoders, variant of SAEs, have been used to simplify circuit analysis and applied to the greater-than circuit in GPT-2 (Dunefsky et al., 2024). Whilst the appli- cation of SAEs to mechanistic interpretability is supported by qualitative and quantitative evidence, their usefulness is highly dependent on hyperparameterisation (Leask et al., 2025). (Nanda et al., 2024) propose the use of inference- time optimization on sparse autoencoder decoder matrices to quickly evaluate and compare them.
f(x) or (Wemix be
x(f) := Wdeef + ddce
(2)
where f(x E R is the sparse latent activations and xff) E R" is the reconstructed input. Wene is the encoder matrix with dimension n x and bone is a vector of dimension m, conversely Wile is the decoder matrix with dimension m x n and dd is of dimension n. The activation function a enforces non-negativity and sparsity in f(x), and latent i active on a sample x if fx() > 0.
SAEs are trained on the representations of language model at particular site, such as the residual stream, a large text corpus, using a loss function of the form
[(x) == xx xf( + XS(((x)) +aC
(3)
Experity
where S is function of the latent coefficients that penalizes non-sparse decompositions, and À is sparsity coefficient, where higher values of À encourage sparsity at the cost of higher reconstruction error. Some architectures also require the use of an auxiliary loss Luux. for example to recycle inactive latents in TopK SAES (Gao et al., 2025). We expand on the different SAE variants in Appendix A.2. We also note that Nanda et al. (2024) applied inference-time optimization to pre-trained SAF dictionary, though their approach is otherwise very different to ours.
Sharkey et al. (2025) list the computational cost of training SAES as an open problem in their application to decompos- ing model representations. In particular, they note that the number of parameters in SAE can exceed the number of parameters in the LLM whose representations they decom- pose. For example, in Lieberum et al. (2024), the largest of the SAES trained on Gemma 2 2B have almost 5 billion parameters, compared to only 2 billion parameters in the
While SAEs have been a central approach to learning sparse features neural networks, there a long history of sparse
3

---

Inference- Timrenne-Timee/ae Decomposition of Activations (ITDA)
LLM itself.
function:
2.3. Comparing Models
min ((xx)x))((((E( + Reg(0)
range of methods for comparing representations between neural networks has been developed. Inspired by Erhan et al. (2010), Olah (2015) applied t-SNE, dimensionality reduction technique (Van der Maaten & Hinton, 2008), to the representations of vision and language models. Lene & Vedaldi (2015): Bansal et al. (2021) stitched layers of two frozen models with a trained intermediate adapter layer, and evaluated the similarity of the model's representations by the performance of the stitched model. Representation similarity metrics compare the alignment of the represen- tation subspaces of different models, and include Singular Vector Canonical Correlation Analysis (SVCCA) (Raghu et al., 2017) and Centered Kernel Alignment (CKA) (Korn- blith et al., 2019). The performance of linear probes trained on the representations of different models can provide in- sight into what information the representations represent (Alain, 2016; Hewitt & Manning, 2019). Li et al. (2016) investigated whether different neural networks converge to the same representations, and (Garipov et al., 2018; Zhao et al., 2020) find that different models are occupied by low- loss paths in the parameter space. Olah et al. (2020) pro- vide examples of potential universal features, such as curve detectors, in vision models, and Olsson et al. (2022) find evidence of induction heads in language models of different sizes. Bricken et al. (2023) found similar SAE latents in different models, and Kissane et al. (2024b) found examples of SAEs that transfer between base and fine-tuned versions of the same language model. Lindsey et al. (2024) used Crosscoders, SAEs trained on the representations of multi- ple models, to find features present in a fine tuned version of an LLM that were not present in the base model. Relative representation methods are kernel methods (Hofmann al., 2008) that measure similarity against set of prototype in- puts (Moschella et al., 2022), which avoids learning model specific parameters from absolute model representations.
Where Reg is a regularization function on the parameters 0. These representations are different between models: two models that are behaviorally identical may have represen- tations that are rotations or affine transformations 7 of the other (Kornblith et al., 2019; Raghu et al., 2017). SAEs are trained on these representations so, if their decoders do describe feature spaces in LLMs (Lan et al., 2024), then they are also subject to the same symmetries as absolute representations.
Moschella et al. (2022) propose that, whilst the absolute representations change between models, the angles between elements of the representation space remain the same. That is, for a transformation T, Z(e("), , <(Tel), ()) for every x(),, x(i) E X. Based on this assumption, they construct their representation by selecting a subset A of anchor points of the training data X. Each sample in the training data is represented with respect to the embedded anchors eنا) E(a with a E A, using a similarity function sim R Rdd + R. Cosine similarity Sc is used as this similarity function as it preserves angles:
a b
Sc b)
(4)
|| ||||
3.2. Inference Time Sparse Coding
We replace the learned linear encoder of SAEs with a dictio- nary learning approach solve the sparse coding problem. For inputs x E Rd, we maintain a dictionary of activations D Raxd, which may not necessarily be basis vectors in Rd. We then solve the following sparse coding problem to obtain the coefficients a:
min ||x 10--ap aD|| subject to aER
||a/lo S Lo
(5)
3. Tinerence-Timee Decomposition of Activations (ITDA)
where ||.|| is the lu lo-pseudo-norm (the number of non-zero el- ements), and Lo is a pre-specified sparsity level (the number of latents used represent each x).
We introduce a novel method for decomposing model activa- tions into interpretable units, which we call Inference- Time Decomposition of Activations (ITDA).
We obtain approximate sparse codes with Matching Pursuit (MP) solver (Mallat & Zhang, 1993). For an input vector x we construct solution by the following algorithm, which is fully described in Appendix Algorithm 2.
3.1. Absolute and Relative Representations
Given a training dataset x E X, LLMs learn an embed- ding function Eg X + Rd. where is the parameters of the model. Eg maps each sample x(() E X to its absolute latent representation ( (x()) This absolute rep-
1. Selecting, at each iteration, the dictionary activation d, whose correlation with the residual is the largest in magnitude.
resentations are learned by optimization over an objective
2. Updating the residual by subtracting the projection of the residual onto the newly chosen activation.
4

---

Inference- Decomposition of Activations (ITDA)
3. Repeating for Lo steps achieve the desired sparsity.
Algorithm ITDA Training with Inference- Time Optimiza- tion (ITO)
Input: Training data {æi). J. initial dictionary (optional), sparsity level Lo. threshold т
Thus, the encoder step of the ITDA can be written as MP(x, D, Lo) and the decoder as & aD. Matching Pursuit uses correlation when choosing the next dictionary activation to include in the sparse code, which is equal to the unnormalised cosine similarity between the activations.
Output: Learned dictionary D
Initialize dictionary D (e.g. by selecting common activa-
tions or random sampling)
for each batch B = {01, T2,...,, TB) from training data do
for each sample x in B do
3.3. Dictionary learning
Compute sparse code a OMP(a, Lo) Reconstruct X aD Compute reconstruction loss ((x) = ||+ x112
Moschella et al. (2022) randomly sample their anchors from each class in the training set. However, comparison of ran- domly sampled dictionaries, as is done with SAE decoders in (Lan al., 2024). Therefore we deterministically and iteratively construct the dictionary by attempting to recon- struct activations from the existing dictionary, and adding them to the dictionary if the reconstruction loss is above threshold. This contrasts with SAEs, which have fixed sized dictionary, with the choice of dictionary size largely determining the reconstruction performance. In ITDA,, we choose the loss threshold, which determines the size of the dictionary.
if (xx) > r then
Add æ to dictionary: D + D DUL [z} end end for Remove duplicate atoms in (optional)
Normalize dictionary: dj- dj/[[dyll for all rows j
end for Return learned dictionary D
4. Comparison with SAES
Concretely, for a new sample its reconstruction &, and the current dictionary D, we define the reconstruction loss (x)) as the mean- mqaa-squared error between the sample and its reconstruction:
We first evaluate on the type of sparse coding problem for which SAEs are used: namely, the decomposition and re- construction of LLM activations.
4.1. Reconstruction Performance
((x) = ||x x|1
(6)
We trained ITDAs the residual stream activations of two Pythia models (Biderman et al., 2023) and the two billion parameter variant of Gemma 2 (Team et al., 2024) on a subset of the Pile dataset (Gao et al., 2020) limited to 128 tokens. We evaluate these terms of their cross entropy loss score (CE loss score):
If l(x) exceeds chosen target loss threshold т, we add the normalized x the dictionary. In practice, when we construct these dictionaries, we batch inputs. If there are identical or similar inputs in a batch, then it is possible that they are both added. As such, we additionally filter the dictionary for repeated activations after construction. The full algorithm is displayed in Algorithm 1. In general, a lower value of leads to a larger dictionary size and lower reconstruction loss than high values of
H- Ho
Horig Ho
where Hony is the original cross entropy loss of the LLM, H" is the cross entropy loss of the LLM when its activations are replaced with the SAE reconstruction of the activation, and Ho is the cross entropy score when zero-ablating the model activations. The cross entropy is calculated on a text- text-prediction on the Pile dataset (Gao et al., 2020). We use SAEBench (Karvonen et al.. 2024a), an SAE benchmarking suite, for these evaluations. Direct comparison between SAE types is challenging as most existing types do not have fixed Las, and ITDAS do not have fixed dictionary sizes (See Table 2). However, for purpose of comparison we crop the ITDA dictionary to a fixed size.
3.4. Interpretable Labels
We label the selected dictionary atoms with the prompt and token to which the activation corresponds. This is not used for decomposition, but provides an additional property of inherent interpretability. Gaining any understanding of SAE latent functionality requires automated interpretability, or other further experimentation, whilst with ITDA latents we already have these interpretable labels. We also use the labels, rather than the activations themselves, in our representation similarity study in Section 5.
Figure 7(a) shows the performance of ITDAs in comparison
5

---

Inference-Tinge Decomposition of Activations (ITDA)
to SAEs. On Pythia models, ITDAs generally perform better than ReLU SAEs, the original SAFs used in (Bricken et al., 2023; Cunningham et al., 2023), but worse than the more recent top-k SAEs (Gao et al., 2025). On Gemma 2 28, ITDAs perform substantially worse than SAFs, where pretrained SAEs are available.
We cherry-picked three examples of latents that activate in interestingly different cases, and interpret the feature they represent based on the prompts and tokens which they strongly activate.
1. Latent 50 responds specifically to the token "How"
However, the SAEs were trained on hundreds of millions of tokens. In comparison, the ITDAs were trained to their max- imum performance on 1.2 million tokens. Where compute and training dataset are not a constraint, SAEs can achieve better performance than ITDAs; but for most applications this is not the case. Sharkey et al. (2025) cites the high cost of training SAES as a major open problem in decomposi- tion of activations using sparse dictionary learning- SAE performance increases with the size of the dictionary, and one of the major factors in choosing dictionary size is the training cost: (Leask et al., 2025) trained SAEs on GPT-2 small ranging from 768 latents to 98304 latents, which took between 2 and 8 hours to train. Given the far lower cost of training ITDAs, it is possible to train much larger ITDAs than SAEs to achieve similar reconstruction performance. However, this choice of dictionary size impacts interpretabil- ity in both SAEs (Leask al., 2025; Karvonen et al., 2024b) and ITDAs.
2. Latent 16990 responds to tokens relating to surprise and amazement.
Latent 17002 responds to context relating to offers of help, particularly helping students with homework.
Activation histograms and top- activatii inputs are provided in Appendix A.3.1.
We assigned interpretable labels the prompt and token to which an activation relates to the atoms. For example, latent 16990 corresponds to the token "surprising", and the top activating latents correspond to prompt tokens relating to "surprise". Direct interpretation of the atom labels can be misleading though, as is the case for latent 17002. Latent 17002 corresponds to the presence of "a" in its prompt, but its top activating inputs relate to "helping students with their homework". Whilst this is also the context of the prompt corresponding to Latent 17002, it is not immediately obvious whether the token or the context of the token is relevant.
We include interpretability benchmark results from SAEBench in Appendix A.3.4. However, we emphasize caution in interpreting these results due to the novelty of automated benchmarking for SAEs, and their questionable applicability to methods other than SAEs; concerns that we expand on in the appendix.
Another key difference with ITDAs is the existence of nega- tive activations. Whilst large positive activations of ITDA latents correspond to monosemantic and interpretable fea- tures of the data, small positive and negative examples are difficult to interpret, however this is also the case with SAES for small positive values (Bricken et al., 2023). Large neg- ative activations are exceptionally rare, constituting only O. 1.000743% of activations on our dataset.
4.2. Llama Case Study
We trained and release ITDAS on the 70 billion and 405 billion parameter versions of Llama 3.1 (Dubey al., 2024) at [redacted]. These ITDAs were trained on consumer GPU using a dataset of 1 million activations collected with FFiotto-Kaufna et al., 2024). Existing open-source SAES are limited to a maximum LLM parameter count of 27 bil- lion such as in Lieberum et al. (2024). Evaluating these large models using benchmarks like SAEBench (Karvonen et al., 2024a) is currently infeasible due to the high computational cost of collecting activations, as opposed to training or run- ning ITDAs. We provide these case-studies to demonstrate that ITDAs find interpretable and monosemantic features in the activations of these large models.
Appendix A.3.2 contains a number of examples of prompt decompositions using this ITDA.
5. Representation Similarity
Identical LLMs trained on the same data but with different initialisations can learn representations that are rotated or at different scales. Measuring the similarity of their repre- sentations generally requires learning linear maps to handle these transformations (Kornblith al., 2019; Raghu et al., 2017). Moschella et al. (2022) propose relative repre- sentation measure, where representations are assigned to their nearest neighbor in set of anchor points (detailed in Section 3.1), and therefore does not require learning a map between the representation spaces. These approaches compare datasets of representations, which can be large to provide a meaningful sample.
We trained an ITDA with 17939 latents on layer 40 of Llama 3.1 70B, and collected the coefficients (which we call acti- vations) of each latent (which we call latents) on activations of the LLM on 10,000 prompt sample of the Pile dataset (Gao et al., 2020). Similarly to SAES (Bricken et al., 2023), we interpret the feature to which a latent corresponds by considering the prompts on which it most strongly activates.
6

---

Inference-Tinee Decomposition of Activations (ITDA)
SAE dictionaries have been proposed universal descrip- tions of representation space (Lan al., 2024). The similar- ity of two representation spaces can be measured by learning map between the dictionaries of two SAEs. Where SAEs are available, this provides an alternative to computing sim- ilarity on representations. However, SAEs themselves are expensive to train, and the learning of this map similar to other absolute representation metrics.
We propose the Jaccard Index (Intersection of two ITDA dictionaries as measure of representation similarity. Concretely, for models Mo and M1. we construct dictionaries Do and Dj as described in Algorithm 1. Then our measure for representation similarity is:
Union)
(a) Linear Regression
(b) SVCCA
DDo n D1l D U Dil
S(Mo, M1)
(8)
This approach a relative representation measure, so not necessary learn a map between the representation spaces. ITDAs are cheap train, and the Jaccard Similarity is trivial to compute, and so this approach further resolves the challenge with using SAEs as descriptions of feature spaces.
We evaluate this method on two tasks: first, we match layers in different instances of the same model by their maximum similarity as Kornblith et al. (2019); then we perform this task at the model level, attempting to match different instances of the same model using the union of their ITDA dictionaries. We also reproduce results in layer convergence during training in Appendix A.4.2.
(c) Linear CKA
(d) Relative
5.1. Model Instance Layer Similarity
Kornblith et al. (2019) introduces a benchmark for similarity indices based on layer matching task. For two architec- turally identical models trained from different initialisations, we expect that the value of the residual stream after layer i in the first model is most similar to the value of the residual stream after layer in the second model, rather than after other layer j # i. Similarity indices scored on their accuracy on this task. That is, the score of similarity index S is defined as:
(c) ITDA
Figure 2. Heatmap of similarity indexes between layers in in- stances GPT-2 small with random initialisations, metric average across all pairs of model. Each axis represents the ordered layers of model. The dark blue colors represent pairs layers that, on average, have low similarity scores, whilst the yellow colors represent pairs layers that, on average, have high similarity scores. Note that these averages, and the scores for specific comparisons vary, but in general, higher contrast between values in the and column indicate that the metric is better able to differentiate between layers. See Figure A.4 for GPT-2 medium results.
Σ M.m'CM ==0
Σ1
arg max S(m, m²)
M
Where M is the set of models of size M and n is the number of layers in each model with m, 0 < i < n is the partial model after a certain layer, and 1|:] is the indicator function.
We replicate this experiment on two sets of five instances of the GPT-2 small and GPT-2 medium taken from the Mistral

---

Inference-Tinee Decomposition of Activations (ITDA)
package (Karamcheti et al., 2021).
of this method, we are able to train ITTAAs on LLMs that are an order of magnitude larger than the largest LLMs on which open-source SAEs have been trained. We provided examples of ITDA latents for these large models, to demon- strate that ITDAS learn interpretable and monosemantic features similarly to SAEs. Unlike SAEs decoders, ITDA dictionaries transfer readily between LLMs because they are associated with a prompt and token index, rather than being learned from the representations of a specific model. We used this property to construct an index for representational similarity that is state-of-the-an aft on a LLM layer matching task.
Table 2 shows the performance of CKA (Kornblith et al., 2019), SVCCA (Raghu et al., 2017), and ITDA on this task. Figure 5 shows the layer similarity between all pairs of layers, averaged across all pairs of models, for the GPT-2 small. Figure A.4 shows the same information for the GPT-2 medium instances.
We emphasize that ITDAAs offer significantly reduced train- ing costs, and the use of the simple Jaccard similarity index makes this analysis much more efficient compared to using SAEs. Training each ITDA on GPT-2 small took an aver- age of 8 minutes, in comparison to two to eight hours for each SAE (depending on dictionary size, on comparable hardware).
The primary limitation of our approach is that ITDA per- forms worse on reconstruction and interpretability tasks than recent types of SAE, such as TopK and P-Annealing SAEs; however, they do often perform comparably to ReLU SAEs. Whilst it is possible that further refinement of the ITDDA ap- proach will lead to better reconstruction and interpretability results, we do not currently suggest ITDAs broadly replace SAEs on existing applications such as circuit discovery (Dunefsky et al., 2024; Marks et al., 2025), where SAEs are publicly available for those models. However, we are keen see ITDAs applied more to tasks where training SAEs is prohibitively expensive such as very large models, or where large numbers of SAES are required, such as in our layer freezing experiments or on checkpoints during model training. We are also excited to see further refine- ment of the ITDA learning algorithm: in particular, we think data ordering during training could be very important for auto-regressive L.MMs.
5.2. Model Similarity
The ITDAs used to calculate layer similarity are trained at specific layers within the model, which resulted in their dictionaries forming universal descriptions of the represen- tation space at that layer. We briefly test whether the union of ITDA dictionaries over all layers within a model forms a universal description of the representation space for the entire model. That is, for an ITDA dictionary Do,: at layer of model 0, we say
L Do = Dos 1=0
(10)
where L is the layers of model O. We test whether we can distinguish between GPT-2 small and GPT-2 medium using the Jaccard similarity of dictionaries.
Our results demonstrate that ITDAs find interpretable and monosemantic features in LLM representations, and that the difference in their dictionaries is an accurate measure of representation similarity. We believe that this makes ITDAs an ideal candidate for further research into finding differences between models, such as base and fine-tuned LLMs. Lindsey et al. (2024) suggest that model diffing is an important component of deploying iterative models, as is done by API model providers: training SAFS for all fine-tunes of a model is likely too costly, and ITDAs offer an efficient alternative.
The Jaccard similarity between different instances of the same model architecture (either GPT-2 small or GPT-2 medium) ranges from 0.56 to 0.59; whereas the Jaccard similarity hetween different model architectures range from 0.46 to 0.47 (Figure 11. This suggests the dictionaries do track the difference between the models, and suggests fur- ther research into model differencing using ITDAs may be interesting.
Using CKA or SVCCA to perform a global layer match- ing task involves concatenating the activation vectors at all residual stream locations. For GPT-2 Medium this is a vec- tor with 24,576 elements per token introducing substantial training cost.
Impact Statement
This paper presents work whose goal is to advance the field of mechanistic interpretability, rather than machine learning in general. Whilst there are potential societal consequences to the advancement of the ML field, we feel that research that improves understanding of ML models is unlikely to additionally contribute to these consequences, rather giving the field more tools to avert them.
6. Discussion
We introduced Inference-Time Time Decomposition of Activa- tions as a scalable approach to LLM interpretability. ITDAs can perform as well as SAEs on certain LLMs, whilst be- ing a 100x faster to train. Due to the reduced training cost
8

---

Inference-
Decomposition of Activations (ITDA)
Metric
GPT-2 Small
GPT-2 Medium
Linear Regression (baseline)
0.16
0.07
SVCCA (Raghu al., 2017)
0.50
0.44
Linear CKA (Kornblith al., 2019)
0.69
0.61
Relative (Moschella et al., 2022)
0.87
0.78
ITDA (ours)
0.88
0.89
Table Accuracy of Linear CKA, SVCCA, ITDA on the layer matching task GPT-2 small and medium. ITDA Jaccard similarity outperforms all other methods, including the other relative representation approach.
References
Elad, M. and Aharon, M. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions Image processing, 15(12):3736-3745, 2006.
Aharon, M., Flad, M., and Bruckstein, A. K-svd: An algo- rithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11):4311-422 2006.
Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield- Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A math- ematical framework for transformer circuits. Trans- former Circuits Thread, 2021. https://transformer- 2rocits.ph/2222//aaanwwoot// html.
Alain, G. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: arXiv:1610.01644 2016.
Bansal, Y.. Nakkiran, P., and Barak, B. Revisiting model stitching to compare neural representations. Advances in neural information processing systems, 34:225-236, 2021.
Beck, A. and Teboulle, M. A fast iterative shrinkage- thresholding algorithm for linear inverse problems. SIAM journal on imaging sciences, 183-2202, 2009.
Erhan, D., Courville, A., Bengio, Y., and Vincent, P. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statisties, pp. 201-208. JMLR Workshop and Conference Proceedings, 2010.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397 2430. PMLR, 2023.
Fiotto-Kaufia J., Loftus, A. R., Todd, E., Brinkmann, J., Juang, C., Pal, K., Rager, C., Mueller, A., Marks, S., Sharma, A. S., et al. Nnsight and ndif: Democratizing access to foundation model internals. arXiv preprint arXiv:240711561, 2024.
Bricken, T., Templeton, A., Batson, J., Chen, B., Jermyn, A., Conerly. T., Turner, N., Anil, C., Denison, C., Askell, A., et al. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T. Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 2020.
Bussmann, B., Leask, P., and Nanda, N. Batchtopk sparse autoencoders. NeurlPS Workshop on Scientific Methods for Understanding Deep Learning (SciForDI.), 2024.
Gao, L.. Ia Tour, T. D., Tillman, H.. Goh, G., Troll, R., Radford, A., Sutskever, I., Leike, J., and Wu, J. Scaling and evaluating sparse autoencoders. ICLR, 2025.
Cunningham, H., Ewart, A., Riggs, L., Huben, R., and Sharkey, Sparse autoencoders find highly interpretable features in language models. ICLR 2024, 2023.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D. P. and Wilson, A. G. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint mrxiv:2407.2178. 2024.
Geva, M., Bastings, J., Filippova, K., and Globerson, A. Dissecting recall of factual associations in auto-regressive language models. IMNLP. 2023.
Dunefsky, J., Chlenski, P., and Nanda, N. Transcoders find interpretable IIm feature circuits. NeurIPS 2024, 2024.

---

Inference- Decomposition of Activations (ITDA)
Gregor, K. and LeCun, Y. Learning fast approximations of sparse coding. In Proceedings of the 27th international conference international conference machine learn- ing. pp. 399-406999 2010.
Kornblith, S., Norouzi, M., Lee, H., and Hinton, G. Sim- ilarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMLR, 2019.
He, 7.,, Shu, W., Ge, X., Chen, I.., Wang, J., Zhou, Y., Liu, F. Guo, Q. Huang, X., Wu, Z., et al. Llama scope: Extracting millions of features from 1ama-3.118b with sparse autoencoders. ICLR Workshop on Secure and Trustworthy Large Language Models, 2024.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. 2009.
Lan, M., Torr, P., Meek, A., Khakzar, A., Krueger, D., and Barez, F. Sparse autoencoders reveal universal feature spaces across large language models. arXiv preprint ar arXiv:2410.06981, 2024.
Heap, T., Lawson, T., Farnik, L., and Aitchison, L. Sparse autoencoders can interpret randomly initialized transform- ers. arXiv preprint arXiv:2501. 17727, 2025.
Leask, P., Bussmann, B., Pearce, M., Bloom, J., Tigges, C., Moubayed, N. A., Sharkey, L.,, and Nanda, N. Sparse au- toencoders do not find canonical units of analysis. ICLR, 2025.
Hewitt, and Manning, D. structural probe for finding syntax word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 4129 4138, 2019.
Lee, H.,, Ekanadham, C., and Ng. Sparse deep belief net model for visual area v2. Advances in neural information processing systems, 20, 2007.
Lenc, K. and Vedaldi, A. Understanding image representa- tions by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Hofmann, T., Schölkopf, B., and Smola, A. Kernel meth- ods in machine learning. 2008.
Kantamneni, S., Engels, J., Rajamanoharan, S., Tegmark, M., and Nanda, N. Are sparse autoencoders useful? a case study in sparse probing. arXiv preprint arXiv:2502.16633 2025.
Li, Y., Yosinski, J., Clune, J., Lipson, H., and Hopcroft, J. Convergent learning: Do different neural networks learn the same representations? ICLR, 2016.
Karamcheti, S., Orr, L., Bolton, J., Zhang, T., Goel, K., Narayan, A., Bommasani, R., Narayanan, D., Hashimoto, T., Jurafsky, D., al. Mistral-a journey towards repro- ducible language model training, 2021.
Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., Kramár, J., Dragan, A., Shah, R., and Nanda, N. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2, 2024. URL https: / / //arxiv. orw/orgg///..... /2008/40822 05147.
Karvonen, A., Rager, C., Lin, J., Tigges, C., Bloom, J., Chanin, D., Lau, Y.-T., Farrell, E., Conmy, A., Mc- Dougall, C., Ayonrinde, K., Wearden, M., Marks, S., and Nanda, N. Saebench: A comprehensive benchmark for sparse autoencoders, December 2024a. URL https:// // www www.oneeddpiiaaa org/ wrggsaaeeeeeeeeeoee http://aebbeeehinne Ac- cessed: 2025-01-20.
Lindsey, J., Templeton, A., Marcus, J., Conerly, T., Batson, J., and Olah, C. Sparse crosscoders for cross-layer fea- tures and model diffing. Transformer Circuits Thread, 2024.
Makelov, A., Lange, G., and Nanda, N. Towards principled evaluations of sparse autoencoders for interpretability and control. ICLR, 2025.
Karvonen, A., Wright, B., Rager, C., Angell, R., Brinkmann, J.. Smith, L., Verdun, C. M., Bau, D., and Marks, S. Measuring progress in dictionary learning for language model interpretability with board game models. NeurIPS, 2024b.
Makhzani, and Frey, B. k-sparse autoencoders, 2014. URL https: /tttps://arxisn wrtg/bsg/////..ooooooo..oo /tt//111..555 5663.
Mallat, S. G. and Zhang, Z. Matching pursuits with time- frequency dictionaries. IEEE Transactions signal processing, 41(12):3397-3415 1993.
Kissane, C., Krzyzanowski, R., Bloom, J. L., Conmy, A., and Nanda, N. Interpreting attention layer outputs with sparse autoencoders. ICML, 2024a.
Marks, S., Rager, C., Michaud, E. J., Belinkov, Y., Bau, D., and Mueller, A. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. ICLR, 2025.
Kissane, C., Krzyzanowski, R., Conmy, A., and Nanda, N. Saes (usually) transfer between base and chat models. In AI Alignment Forum, 2024b.
10

---

Inference- Time Decomposition of Activations (ITDA)
Martinez, R. D., Lesci, P., and Buttery, P. Tending towards stability: Convergence challenges in small language mod- els. EMNLP, 2024.
Sharkey, L., Chughtai, B., Batson, J., Lindsey, J., Wu, J., Bushnaq, L.., Goldowsky-Dile N., Heimersheim, S., Or- tega, A., Bloom, J., et al. Open problems in mechanistic interpretability. arXiv preprint arXiv:2501. 16496, 2025.
Meng, K., Bau, D., Andonian, A., and Belinkov, Y. Locating and editing factual associations in gpt. Advances in Neu- ral Information Processing Systems, 35:17359-1737, 2022.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, I.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.000 0044000001100 18, 2024.
Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Lo- catello, F,, and Rodolå, Relative representations enable zero-shot latent space communication. arXiv preprint vrXiv:2209.154333 2022.
Templeton, A. Scaling monosemanticity: Extracting in- terpretable features from claude 3 sonnet. Anthropic, 2024.
Nanda, N., Chan, L., Lieberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. ICLR, 2023.
Van der Maaten, and Hinton, G. Visualizing data using I-sne, Journal of machine learning research, 9(11), 2008.
Nanda, N., Conmy, A., Smith, L., Rajamanoharan, S., Lieberum, T., Kramár, J., and Varma, V. Progress up- date# from the gdm mech interp team: Full update. In Al Aligument Forum, 2024.
Zhao, P., Chen, P.-Y,, Das, P., Ramamurthy, K. N., and Lin, Bridging mode connectivity in loss landscapes and adversarial robustness. ICLR, 2020.
Olah, C. Visualizing representations: Deep learning and human beings. 2015.
Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, Zoom in: An introduction to circuits. Distill, 5(3)e000024 0014 2020.
Olshausen, B. A. and Field, D. Sparse coding with overcomplete basis set: A strategy employed by v1? Vi- sion research, 37(23):3311-3322 1997.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., al. In-context learning and induction heads. arXiv preprint arXiv:2209. 11895, 2022.
Radford, A., Wu, J.. Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models unsupervised multitask learners. 2019.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl-Dickstein, J. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Rajamanoharan, S., Lieberum, T., Sonnerat, N., Conmy. A., Varma, V., Kramár, J., and Nanda, N. Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders. arXiv preprint arXiv:2407. 240..44435.44 2024.
Rajamanoharan, S., Conmy, A., Smith, L., Lieberum, T., Varma, V., Kramár, J., Shah, R., and Nanda, N. Improving dictionary learning with gated sparse autoencoders. ICLR Workshop on Building Trust in Language Models and Applications, 2025.

---

Inference-Time Decomposition of Activations (ITDA)
A. Appendix
A.1. Glossary
For part of a model and an input sequence x, an activation is = (x). A partial model could be, for example, the residual stream value for an input after the 8th layer of GPT-2. We are interested in the activation for specific token within the prompt, rather than the activation matrix for entire sequence. Activations and representations used interchangeably, as activation is the dominant term in mechanistic interpretability literature and representation is the dominant term in representation similarity literature.
Activation:
Atom: A component of the dictionary learned by sparse dictionary learning, Examples are decomposed into sparse codes of these atoms. The atom terminology more commonly used in classical sparse dictionary learning literature, whereas latent more common in contemporary mechanistic interpretability SAE literature.
Feature: A true but unknown property of data-point, contrasts with Latent.
Latent: Refers the components of SAEs that used in sparse codes to decompose activations. Ideally, these the same as features, but empirically this is not necessarily the case. See atom.
Latent activation: The coefficient of a latent or atom in sparse code.
Representation: See activation.
Sparse code: For an example x and an encoding function f' for an SAE with dictionary size m. f(xx E 12 is a sparse code.
The sparsity, i.e. number of zero or non-zero terms, determined by the optimization process and activation function.
Token:
In the context of LLMs, token is discrete unit of text-such as word, subword, or punctuation- that the model processes and generates during language understanding and generation.
A.2. SAE Variants
Method
Fixed Lo
Fixed Dictionary Size
ReLU
X
TopK
JumpRel.U
X
ITDA
/ x
Table Comparison of SAE types with respect fixed Lo and fixed dictionary size. When constructed the method described in Section 3, ITDAs have a variable dictionary size, however they can be cropped to a desired size.
ReLU SAEs (Bricken al., 2023) use the L1-norm S(f) = || as approximation the L0-norm for the sparsity penalty. This provides gradient for training unlike the L0-norm, but suppresses latent activations harming reconstruction performance (Rajamanoharan et al., 2025). Furthermore, the LI penalty can be arbitrarily reduced through reparameterizztion by scaling the decoder parameters, which is resolved Bricken et al. (2023) by constraining the decoder directions to the unit norm. Resolving this tension between activation sparsity and value is the motivation behind more recent architecture variants.
Gated SAEs (Rajamanoharan al., 2025) separate the selection of dictionary elements used for reconstruction, with the estimation of the coefficients of these dictionary values. This results in the following architecture:
Taate (x) = W gate (x ba) + b bgate
(11)
f(x) T > 0] o ReLUUW (MnuU( ba) + bmag)
(12)
*(f(x)) Waf(x)) + ba
(13)
12

---

Inference-Tinee Decomposition of Activations (ITDA)
Where I|- > 0] is the Heaviside step function and @ donates elementwise multiplication.
P-Annealing SAEs (Karvonen al., 2024b) replace L1 minimisation with Lp minimisation, where p < 1 and is decreasing throughout training. This is similar to ReLU SAEs, except the sparsity loss is calculated as:
Issprso (2p s) >o >a11f(>)1?E de ES
(14)
TopK SAEs (Gao et al., 2025; Makhzani & Frey, 2014) enforce sparsity by retaining only the top k activations per sample. The encoder defined as:
f(æ): == TopK CopK(Wonex + b
(15)
where TopK zeroes out all but the k largest activations each sample. This approach eliminates the need for an explicit sparsity penalty but imposes rigid constraint the number of active latents per sample. An auxiliary loss Laua ||e a|2 used to avoid dead latents, where è Wideeg is the reconstruction using only the top-kauz dead latents (usually 512), this loss is scaled by small coefficient (usually 1/32).
Jump SAES (Rajamanoharan et al., 2024) replace the standard ReLU activation function with the JumpReLU activation, defined as
JumpRel.U,())- = ==>(= - 0)
(16)
where H is the Heaviside step function, and 0 is learned parameter for each SAE latent, below which the activation zero. JumpReLU SAEs are trained using a loss function that combines L2 reconstruction error with an LO sparsity penalty, using straight-through estimators train despite the discontinuous activation function. A major drawback of the sparsity penalty used in Jump SAEs compared to (Batch)Topk SAEs is that it is not possible to set explicit sparsity and targeting specific sparsity involves costly hyperparameter tuning. While evaluating JumpReLU SAEs, Rajamanoharan et al. (2024) chose the SAEs from their sweep that were closest to the desired sparsity level, but this resulted in SAEs with significantly different sparsity levels being directly compared. JumpReLU SAEs use no auxiliary loss function.
BatchTopp SAEs (Bussmann al., 2024) impose TopK constraint over the activations of entire batches during training. Lee. for desired sparsity k and batch size B, all activations not within the top B k in batch zeroed. During training, threshold value is learned, which replaces the BatchTopK activation inference time to avoid dependencies between test inputs.
A.3. ITDA
A.3.1. LATENT EXAMPLES
This section provides further detail of cherry-picked latents in Llama 70B ITDA. The example prompts randomly sampled within four ranges: [2, inf), 20, (-2,0), 0). (-inf,0) 0). Prompts that do not activate latents are excluded from both the example prompt tables and the histograms. Activations collected over 1.28 million tokens from Gao et al. (2020).
Latent 50: Responds specifically to the token "How". The input string from which this activation was taken, with relevant token highlighted, is "How Idris Elba's 'Luther' Puts Us in the Mind set a Renegade Detective. "Luther " a series about righteous indignation. Yes, it's police"
13

---

Inference-Timg Decomposition of Activations (ITDA)
Activation
Prompt
4.331
Q:How to handle puppeteer exception on synchronous execution
4.331
Q:How to change the trigonometric identity to sec
4.331
Q:How to write a React Stateless Functional Component in Typeseript
3,211
in the US, Canada, UK AustraliaHow we promote your campaign'Giving Tuesday
campaigns will
2.879
Remove User search limit per month PDO::YSSLLHow How can have PHP remove
users limit every month
1.996
line doesn't 't work. This Q & A How make autoscroll in Scintilla? has the much is an Xbox One at Rent Center? How much is PS47How much is an redirect to https://ww.testll .com content How can konfigure properly? Rewrites in Jesus. What difference would that make? And how would know if was really forgiven?
1.973
1.150
1.042
0.767
excellent example of this. In it she demonstrates how the concept of national identity fractures society, creating
-0.742
Dental Plans Association is not-for-profi organization with some for-profit affiliates. We offer nationwide package of source. Our people. Your success. We understand the needs & ways to handle your backend SEO process and As Kate Sheppard reported last week, July 26th marked can get them for $180 dollars. Please come by my desk and give your travel arrangements.
-0.874
-1.140
-1.647
-2.272
? I saw the API. RallyRestApi restApi new RallyRestApi(ussemmme, password,
-2.335
Check out our new site Makeup Addiction No native support of previous PlayStation games
-2.468
Most of the people at this detention centre Tripoli will
-2.900
Plus Free Shipping on Orders Over $50 Mario Badescu: Free Standard Shipping on $50+
-3.615
I'm working through detox/cleansing phase and it's
Table 3, Example prompts and activations for Latent 50
14

---

Inference-Timg Decomposition of Activations (ITDA)
200
Active on 0.12% of inputs
175
150
Frequency
125
100
75
50
25
-2.5
2.5
5.0
7.5
10.0
12.5
Activation Value
Figure 3. Activation distribution for latent 50. The peak at 4.33 corresponds to prompts starting with "Q: How". Activations of are omitted for legibility.
Latent 16990: Responds tokens relating to surprise. The input string from which this activation was taken, with relevant token highlighted, is "web. You truly realize how to bring problem light and make it crucial. Many more people should really have a look at this and have an understanding of this side of the story 's surprising you 're more prevalent, as you most really possess the gift.. It much easier deal with the status quo than to push forward beyond the fear He forwarded the email her and"
Activation
Prompt
5.482
It astounds me, without exception to see how goats can climb.
3.930
in a host of colors, patterns Surprising Reasons: Why The Large Floor Tiles Are So Popular
3.775
ulkan.htmi hww.hhhmm-meset gulppaann It's great that Google Android will support Vulkan. Now, the
2.172
Q: Unbelievable strange file creation time problem have very
2.142
you most really possess the gift. is much casier to deal with the status quo than to push
0.967
could easily be confusing cause and correlation. Seems entirely possible
0.923
any later version. This program is distributed in the hope that it will be useful, but 's movie we follow the beautiful model in a shockingly surreal journey through the rural countryside of Italy on
0.812
0.519
/xhtml SOODDD/stmlI-ttanstiiiii to, sometime soon. Even if it makes the difference between living and dying, there's just no way
"http://wwwww3 www.www..oomm wttp:/ww.rg9999////b/ 1999/xhtl
0.000
-0.713
So this guy was sucking brezz, felt something in his mouth, tasted somehow
-0.807
expiratory manoeuvres children: do they meet ATS and ERS criteria for spirometry
-0.851
line, Mirage, and put on and felt my woman warrior emerge. Yeah, know a
-0.864
a strategical bombing. "We, as many others, we sign up them in the Air Force
-1.081
You are here Cigarette sales dive, hurting health
Table 4. Activation values and corresponding prompts
15

---

Inference- -ime Decomposition of Activations (ITDA)
500
Active on 0.15% of inputs
400
Frequency
200
-2
0
2
10
12
14
Activation Value
Figure 4, Activation distribution for latent 16990
Latent 17002: Responds tokens relating to surprise. The input string from which this activation was taken, with relevant token highlighted, is "the deadlines is our priority and we often acquire strict actions to fulfil our guarantee. We have significant knowledge the sphere of homework online help: that's why we think We all know what sort help a student requirements. Just invest in College assignments online and enjoy. Project Online is flexible online solution for project portfolio administration (PPM) and each day get the job done. enables corporations to get"
16

---

Inference-
Decomposition of Activations (ITDA)
Activation
Prompt
2.899
have significant knowledge in the sphere of homework online help; that's why we think We all know what
2.447
fulfil guarantee. We have significant knowledge in the sphere of homework online help; that's why we at AOT you will receive the assistance and support that will help make your study time convenient, manageable and to help your business succeed. We realize that where a company is located has a significant impact on its ability
2.348
2.134
2.067
This contains libraries that a) do change frequently b) require e development acceleration and counter- propttrrpppaggtting waves synchronization, A simple formula for SR pulse delay time evaluation is presented
1.128
1.056 the millions of dollars in moderately affected person venture funding that helps most nascent companies. All employers topic to
0.926
in Japan 1954. Toru Kumon, a high school math teacher, was trying to so we understand customers and we know what it's like to have full responsibility. We understand very clearly
0.900
0.707
in this connection, it is important that they use their time well. Trusted and Experienced Tutors
-0.000
Mossy fiber sprouting after recurrent seizures during early the study's 's authors. As the oldest and most commonly used method to screen for prostate cancer, rectal
-0.714
-0.760
to upgrade new versions, I've been pretty much forced to jump to new manufacturers each time. I
-0.822
Ugh! As move things around pack them or get rid of them, i finding
Table 5. Activation values and corresponding prompts for latent 17002
400
Active on 0.13% of inputs
350
300
250
Frequency
200
150
100
50
-2
0
2
6
10
12
Activation Value
Figure 5. Activation distribution for latent 17002
17

---

Inference-Time Decomposition of Activations (ITDA)
A.3.2. DECOMPOSITION EXAMPLES
This section includes examples of decompositions of inputs in the Pile subset dataset, decomposed by an ITDA trained on layer 40 of Llama 3.1 70B with LO of 40. These prompts are taken from the test split, hence Sequences above 7000. Token 30 is chosen less than 0.0001 are omitted. The text displayed in the third column is the prompt corresponding to the atom's 's activation with the specific token highlighted.
includes contextual information whilst keeping the prompt relatively short and readable. Activations
Sequence 7006 Token 30: Field of the Invention \n The present invention relates to a camera system for transmitting and receiving data to and from camera by obtaining information /hlon"
18

---

Inference-Ting
Decomposition of Activations (ITDA)
Atom Index
Activation
Atom Prompt
8313
4.2174
electronic imaging, and more particularly, the use of electronic imaging for process- ing financial documents, such checks member, in which conductive portion is formed in an insulator, the composite member being used in a touch screen terminal of the other party successively scans the multiple frequencies included in the proximity detection sequence. by reference. The present invention relates to a method of forming a composite member, which a conductive
7878
2.7632
14123
2.5308
7864
2.1850
5800
2.0224
my pregnancy, I tried gather as much information on how painful labor might actually be. would often the wearer and thus to better conform to the body of the wearer. Such extension and expansion about the wearer
4698
1,5757
14124
1.3699
the multiple frequencies included in the proximity detection sequence. If signal strength at each frequency is greater than a preset
5004
1.1926 Gait properties change with age because of a decrease in lower limb strength and visual acuity or knee joint
9268
1.1632
o login for bem sucedido salvar nome de usuário em variável utilizar -I á insulator, the composite member being used in, for example, wiring board in the fields of electric
7889
1.0781
6083
1.0443
Learning Technology Programme (TLTP) from which ideas about application and benefits came. The ideas from TLTP Admin Properties an exception raised, which list the properties not supported. For more information see the pumping, we examined the dependency of H+ pumping on plant Pi status. Both H+ pumping and the
15748
1.0169
6258
0.8658
15678
0.8496
treewidth of the circuit as that of the graph representing it; if we use associativity to rewrite
9362
0.8415
Retrieval of blade implants with piezosurgery: two clinical cases
12606
0.8150
designed to provide users with voice communication services while they are on the move. Current mobile communication systems are capable
7903
0.7805
wiring board in the fields of electric appliances, electronic appliances and electric and electronic communication. The present invention also
17194
0.7706
-based visual programming tool. Node: The source definition for nodes that can be used in Node-RED Fl your recording is ready, you'll Il receive an update on your dashboard homepage with
5623
0.7612
the playback link and the def stateTime' ThreadSaarrt))) threads append
6929
0.7047
database.gerTablle ('CLIENTS')
threads = []
7231
0.6891
a poll I won't make claims to their gifts and charms, but H&M do
7930
0.6539
an insulating material that can be suitably used in the manufacturing method of
14704
0.6371
The position and phase of the Moon are based on the predictable motions of the Moon
Table Atom activations and corresponding text snippets for sequence 7006, token 30.
Sequence 7008 Token 30: Psych Studies is a website owned and maintained by Dr Andrew G. Thomas Swansea University .UK. The purpose of the site host online question
19

---

Inference-Ting Decomposition of Activations (ITDA)
Atom Index
Activation
Atom Prompt
1208
3,9800
bfgin_of.text- Questions or Need Help Related The Hunting Report Newsletter. Call
4006
2.0558
exponentially since its adoption three decades ago. Recent questions have been raised regarding the cost-effect
9803
1.5613
xmlns xx si =' http :// www .w 3 .org 200 /XMLSchema -instance "mmlns = http :// poly
3301
1.3823
Using M Test to show you can differentiate term by term. have the series Σ top financial institutions have recommended new report that whistle -blowing rewarded in an enviromment of growing
7704
1.2946
1246
1.2604
i begin of text-i Question No: 51 You are developing test
8073
1.1217
acid diet in rats. Within 3 h of ingesting an imbalanced amino acid diet (IAAD) sql queries and inserts I have random question. If I were to do sql select and processing huge amounts of documents efficiently. Predictions that document payment methods would decline have not been realized. In
8660
1.0420
8351
1.0366
6059
0.9382
(CAL) and the importance of applying general technological developments
nurse education. The articles recognize the
5258
0.8906
from 15 99. Our ranks are made up of men and women; students and retirees; play "Survival of the Tastiest" on Android, and on the web. Playing the begin of text-ů Q: Why was Mundungus banned from the Hog lemek üzere tek zarfta iki pusulayla oyunu hem yurt içinde hem de yurt dIşında kull which can be used to drag and drop 3D objects and characters into scenes. Amazon continue
2480
0.8181
0.8178
10958
0.7221
2880
0.6843
2345
0.6257
begin of textttx Fractional isolation and chemical structure of hemicellulos bunch of horrible experiences. It's about time I chronicled the peaks of my journey far and this
9553
0.6178
8865
0.5841
noen av bildene politiet fant på mobil telefonene til dem som ble pågrepet 76.1 (5.7) years) ). Walking speed, cadence, stance time, swing million fans then voted on the players using paper and online ballots. The top two vote-getters from each
5071
0.5709
3722
0.5641
4928
0.5300
and across primate taxa. As the problems of habituation become more obvious, the application of such indirect
6133
0.5173
consider the benefits and costs of introducing computer programs as part of the teaching provision for nurses and other health ette Sawyer Cohen, PhD, clinical assistant professor of psychology in pediatrics at Weill Cornell Medical College
3426
0.4858
Table Atom activations and corresponding text snippets for sequence 7008, token 30.
Sequence 7110 Token 30: aRsbot-assisted laparoscopic renal artery aneurysm repair with selective arterial clamping. Renal artery aneurysms represent rare clinical"
20

---

Inference-Ting
Decomposition of Activations (ITDA)
Atom Index
Activation
Atom Prompt
9371
5.0731
val of blade implants with piezosurgery: two clinical cases. In this work an ultrasound device was used
9013
3.7738
7 ]]. Lack of response IFX is à stable
153
3,0721
i mators
Clinical comparison of high-resolution with high-sensitivity colli-
11632
2.2699
dysesthesia. A new and relatively frequent side effect in antineoplastic treatment J. Pal 40% of patients who respond initially and achieve clinical remission inevitably lose response over time
8988
1.9684
3944
1.7801
Coronary artery disease (CAD) accounts for the largest number of these deaths. While efforts aimed at treating believed to play fundamental role in orthopedic research because bone itself has structural hierarchy at the first
4391
1.6158
10181
1,5146
problems in women, the frequency with which primary care providers may encounter mental health problems, and issues of mental return variable is always None So found a strange thing that happens in Python whenever I try to return
5322
1.2979
3981
1.2433
have been directed toward prevention and rehabilitation. CAD is commonly treated using percutaneous coronary intervention (PCI). 2:26 Prostate exams are potentially life-saving. But the process of getting one can be nerve
4509
1.1507
4457
1.0142
bone regeneration will be discussed. This unique 3 tube-shaped nanostructure created by electrochemical anod data or conclusions published herein. All content published within Cureus is intended only for educational, research, and reference
14867
0.9638
6740
0.8502
Adventure Time already slated be a LEGO Ideas official set, so it (ADPLD), as traditionally defined, results in PLD with minimal renal cysts. Classically of. It was (and is) a hearty baked meal- one to satisfy the hunger of very hard-working a shelter for men with alcohol, drug, and mental health problems at 149 W. 132nd St percutancous coronary intervention (PCT), and this treatment has increased exponen- tially since its adoption over three decades ago Today's 's financial services industry is facing the immense challenge of processing huge amounts of documents efficiently. Predictions taken a first step in understanding how to manipulate specific neural circuits using thoughts and imagery. The technique, which ago, when frequencies below 100 Hz were considered extreme lows, and reproduction below 50 Hz was about
10227
0.8491
14150
0.8193
7014
0.7914
3993
0.7646
8340
0.7237
8500
0.6808
14836
0.6754
15928
0.6624
i b-bgiinooo of tef_texxté Late complications of radiotherapy for nasopharyngeal carcinoma. To evaluate and
3910
0.6456
Rationale and trial design. Cardiovascular disease (CVD) currently claims nearly one
million lives yearly
14869
0.6120
by Pryce in 1946 in the Journal of Pathology and Bacteriology
Table Atom activations and corresponding text snippets for sequence 7110, token 30.
A.3.3. MATCHING PURSUIT
The Matching Pursuit algorithms due to Mallat & Zhang (1993) is described in Algorithm 2, with the algorithm for iteratively constructing the dictionary of activations in Algorithm 1.
21

---

Inference- Decomposition of Activations (ITDA)
Algorithm 2 Matching Pursuit (MP) with Normalized Dictionary
Input: Normalized dictionary D E R batch of signals X E IR Bxn, number of nonzero coefficients L
Output: Coefficient matrix C E Rxm
Initialize the residuals: R < X
Initialize the coefficients: C t 0gxm
for = 1 to I. do
Compute correlations: Corr R.DT
For each b E (1...., B), find the best atom index: jo <- arg maxy| (orr
Let Cp f (b.r (the max absolute correlation for sample
Update coefficients: ((b.s)) +- ((.iit) +
Update residuals: (b..)) + R(b.:) (j..))
end for
return C
A.3.4. SAE INTERPRETABILITY METRICS
In this section we present results from SAEBench (Karvonen et al., 2024a) for three LLMs: the 70m and 160m parameter variants of Pythia (Biderman et al., 2023), and the 2b parameter variant of Gemma 2 (Team et al., 2024). Our experiments were conducted with the 0.4.0 beta release of SAEBench. The SAEs with which we compare with are a mix of architectures: generally, the worst performing SAE is a ReLU SAE (Bricken al., 2023; Cunningham al., 2023), whilst the best is a top-k SAE (Gao al., 2020) p-annealing SAE (Karvonen al., 2024b).
SAEBench metries were created for evaluating SAEs, which limits their applicability to ITDAs. In particular, Spurious Correlation Removal and Targeted Probe Perturbation make assumptions about the decoder weight matrix of SAEs that do not apply to ITDAs and so we do not include those results. We include sparse probing results as they are, but note that applying the top-k operation to ITDA latent activations is not comparable to applying it to SAE activations due to negative activations and normalization the decoder. SAEs furthermore have weak performance at identifying human-interpretable concepts on probing tasks, when compared to other methods (Kantamneni al., 2025). Automated interpretability approaches for SAEs transfer directly to ITDAs, but their usefulness in measuring interpretability is questionable (Heap et al., 2ive).Given the challenges benchmarking SAEs, and the additional difficulty of applying those metrics to ITDAs, advise strong caution in interpreting these results and look to advances in SAE benchmarking to better evaluate the interpretability of ITDAs. For now, we emphasize the evidence provided by the representation similarity results in Section 5.
Sparse probing assesses whether SAES capture specific predefined concepts. For each concept (e.g.. sentiment), the k most relevant latents are identified by comparing their average activations on positive versus negative examples. A linear probe then trained on these top k latents. If the latents align closely with the concept, the probe achieves high accuracy, even though the SAE was not directly trained to represent that concept. We evaluate on lt E 1,2, 2, 5, co latents to handle cases where concepts are distributed over multiple latents.
In automated interpretability for each chosen latent, a language model generates a "feature description" based on variety of activating examples. During testing, a dataset assembled by sampling sequences that trigger the latent at varying levels of activation, along with randomly selected control sequences. The LLM then uses its generated description to predict which sequences are likely to activate the latent, and the accuracy of these predictions defines the interpretability score.
22

---

Inference- Timee Decomposition of Activations (ITDA)
0.90
CE Loss Score
0.85
Method
Top-K SAE
0.80
ITDA
0.75
16000
Dictionary Size
14000
12000
10000
8000
104
105
106
10
Total Training Tokens
Figure Performance Pythia-70m SAEs ITDAs when trained on limited numbers tokens. Here, we do not crop the size of the dictionary, on smaller numbers of training tokens the dictionary does not achieve the minimum size.
7ythia-70m-deduped layer=3
pythia-70m-deduped, layer=3
1.00
0.90
0.95
0.88
16k ITDA (max)
Autointero Score
Score
0.90
0.86
Best 16k SAE
0.85
0.84
X
4k ITDA (min)
0.82
Worst 4k SAE
0.80
60
70
10
20
40
50
80
10
layer=8
sitbia-160m-deeupee, layer=8
1.00
ITDA (min)
0.95
64k ITDA (max)
LOSS Score
0.90
Autointerp Score
Worst dk SAE
X
Best 64k
0.85
0.80
4k (min)
64k ITDA (max)
0.75
Worst 4k SAE
0.70
Best 64k SAE
0.78
10
20
30
40
60
70
80
10
20
40
50
60
70
80
gemma-2-26, layer= 12
gemma-2-2b. layer=12
1.00
0.90
4k ITDA (min)
X
0.88
64k ITDA (max)
Loss Score
Autointerp Score
Worst SAE
0.86
Best 64k SAE
0.84
0.80
ITDA (min)
0.82
64k ITDA (max)
0.80
X
0.70
Best 64k SAE
0.78
20
80
(a) CE Loss Score
(b) Automated Interpretabilitty
Figure loss score automated interpretability scores. loss score defined ??, see Karvonen et al. (2024b) details the automated interpretability metric.
23

---

Inference-Timg Decomposition of Activations (ITDA)
yythia-00m-dedupee layer=3
pythia-00m-deduped. layer=3
0.96
ITDA (min)
ITDA (min)
Accuracy
(max)
SAE Top- Accuracy
16k ITDA
0.94
X
Worst 4k SAF
Worst 4k SAF
0.92
0.70
0.90
10
30
40
60
70
80
10
20
40
50
80
Dythla-16Om-dedupee, layer=8
pythia-160m-dedupeed layer=8
0.80
ITDA (min)
SAE Accuracy
SAE Top- Accuracy
B4k ITDA (mak)
Worst SAE
Best 64k SAE
0.92
4k ITDA (min)
0.70
Worst (max)
0.90
Best 64k SAE
gemma-2-22, layer=12 12
gemma-2-26, layer=12
(min)
SAE Accuracy
SAE Top-1 Accuracy
64k ITDA (max)
0.75
Best 64k SAE
X
X
4k (TDA (min)
ITDA (max)
Worst 4k SAE
0.05
20
30
50
60
70
80
30
40
50
70
80
(a) Overall
(b) Top
Figure 8. Overall and top-1 lincar probing scores. Overall score measures whether concepts can be probed from the activations of all the latents, whilst top-1 score measures whether the concepts can be probed from the single most active latent.
24

---

Inference- Time Decomposition of Activations (ITDA)
pythla-70m-dedupe. layer=3
ITDA (min)
16k ITDA (max)
Worst 4k SAE
Best 16k SAE
0.800
X
X
SAE
Jythia-160m-deduped layer=8
0.900
(min)
0.875
64k ITDA (max)
0.850
Best 64k SAE
0.825
0.800
SAE
0.775
0.750
10
20
30
40
50
60
70
80
gemma-2-22 2-2m,, layer=12
0.900
4k ITDA (min)
Accuracy
0.875
X
X
Worst 4k SAE
0.850
0.825
0.800
0.775
0.750
30
40
50
Figure 9. Top sparse probing scores.
25

---

Inference- TinreneeeTiiee Decomposition of Activations (ITDA)
A.4. Representation Similarity
(a) SVCCA
(b) Linear CKA
(c) ITDA
(d) Linear Regression
(e) Relative
Figure Heatmap of similarity indexes between layers instances of GPT-2 medium with random initialisations, metric average across all pairs of model. See Figure 5 for GPT-2 small results and further explanation of the plots.
26

---

Inference-Timee Decomposition of Activations (ITDA)
A.4.1. MODEL SIMILARITY
Small
1.0
0.9
Small
0.7
Medium
0.6
Figure Model similarity between ITDAs trained on instances GPT-2 small and medium. Note higher similarity within model architectures.
A.4.2. LAYER CONVERGENCE DURING TRAINING
Raghu et al. (2017) demonstrate using SVCCA that, in convnet and resnet trained on CIFAR-I0 (Krizhevsky et al., 2009). the representations of early layers converge earlier during training than the later layers. Martinez et (2024) replicate this result on the Pythia suite of language models (Biderman et al., 2023), using the CKA metric from Kornblith al. (2019).
For models in the 70m, 160m, and 410m Pythia models, trained ITDAs each non-terminal layer every ten thousand training steps, and calculated the ITDA representation similarity from Equation 8 for each checkpoint with respect to the last checkpoint, which are presented in Figure A.4.2. The similarity metrics for the first two layers converge to during the last third of training, meaning that the ITDA dictionary stable across those checkpoints. No other layers converge, with the penultimate similarity measure decreasing for successive layers.
Dictionary Similartt
0.8
Dictionary Similarity
0.8
Dictionary Similarity
0.4
0.4
0.4
0.2
0.0
0.0
0.0
20000
40000
60000
80000
100000
120000
140000
20000
40000
60000
80000
100000
120000
140000
20000
40000
60000
80000
100000
120000
140000
Step
Step
Step
(a) Pythia 70m
(b) Pythia 160m
(c) Pythia 410m
Figure 12. ITDA similarity between layers every 10k steps during training compared to their final state. Layers increase from the first layer in dark blue the last layer light yellow, Note convergence early layers before the end of training.
27


Reviews:
[{"review_id": "pD0xWvIsbg", "reviewer": "Reviewer_XACq", "summary": "The paper proposes a new algorithm for mechanistic interpretability as an alternative to sparse autoencoders. Their algorithm iteratively identifies new token activations to add as dictionary items based on their similarity to the current dictionary. If the similarity is too low (i.e. reconstruction through the dictionary is not accurate enough), the (contextualized) token is added as a new item to the dictionary. Hence, the dictionary items are effectively identifiable from token indices into a prompt. This allows the construction of a new representation similarity measure based on the Jaccard index computed over two dictionaries.\nWhile the reconstruction performance of the proposed model is worse, it performs on par with ReLU-SAEs on interpretability benchmarks, but worse than the state-of-the-art Top-K SAEs. However, the proposed method is significantly cheaper to train than SAEs, making it applicable to large open source models with hundreds of billions of parameters. Finally, on a model layer similarity measurement task, the proposed method outperforms previous methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and thoughtful feedback. In particular we appreciate your recognition of the evidence for the claim that this approach is “a lot more efficient than SAEs” and that you feel this work “opens up meaningful new research directions”.\n\n> However, I would like to see a more thorough investigation of the relation between training data size and other hyperparameters of the proposed method with the training time, in comparison to SAEs.\n\nWe agree this is an important avenue for investigation. In our experiments, the ITDAs were trained on 1.28 million tokens, while the SAEs used for benchmarking ranged from 500 million to 16 billion tokens. This offers insight into the performance of ITDAs on low data settings across target models. A comprehensive exploration of this relationship, however, would require training new SAEs, a process that would take weeks for models like Gemma 2, which unfortunately exceeds our timeframe for revisions. Nevertheless, we will clearly highlight this limitation in our revised manuscript and suggest it as a critical area for future research.\n\n> The baselines for the model instance layer similarity task seem quite old (2017 & 2019), but I am unsure if a more recent one (Lan et al., 2024) is applicable in this case. The authors should clarify this.\n\nThis is true, while CKA and SVCCA are standard approaches, they are not recent. Lan et al. proposes applying CKA to SAE decoder matrices as a measure of representation similarity, however applying this approach would require training 170 SAEs on the LLMs which would take 1-2 months of GPU time. Instead, we have added a more recent relative representation measure due to Moschella, Luca. \"Latent communication in artificial neural networks.\" ICLR (2024). This measure also significantly outperforms SVCCA and CKA, but not our ITDA IoU method. \n\nMetric GPT-2 Small GPT-2 Medium\n\nLinear Regression (baseline) 0.16 0.07\n\nSVCCA (Raghu et al., 2017) 0.50 0.44\n\nLinear CKA (Kornblith et al., 2019) 0.69 0.61\n\nRelative (Moschella et al., 2022) 0.87 0.78\n\nITDA (ours) 0.88 0.89\n\nNote that for better reproducibility we have replaced our self-trained Pythia instances with two sets of public GPT-2 model instances of different sizes.\n\n> However, since there is significant space left (1 page), it could also be moved to the main body to make the paper more self-contained.\n\nFollowing feedback from reviewers, we have expanded the main body with a system-level diagram and more explanation of the method and experiments. Consequently, there is now no space to move the examples and algorithm to the main body of the paper. We chose to prioritise these clarifications to help improve the reader’s understanding of the core methodology.  If there are specific supplementary plots or analyses you believe would significantly strengthen the main body, please let us know—we will explore adjustments accordingly.\n\n> I would also like to see an investigation of how the choice of prompts etc. influences the interpretability and representation similarity results.\n\nThis would be an interesting direction for further exploration. SAEs are highly susceptible to shifts in the distribution of their training data, so it seems likely this is also the case for ITDAs. The interpretability pipeline is time-consuming to run, so we won’t be able to get these results during the review period. However, we think these experiments would meaningfully strengthen the paper and will add them to a future version. \n\n> line 389: \"an efficient alternative\"\n\nGood spot, thanks!\n\nWe hope that this has addressed your concerns with the paper, and if so, would ask you to consider increasing your support of this paper.", "claims_and_evidence": "The support for the claims made is generally convincing. There is also support for the claim that the proposed method is a lot more efficient than SAEs. However, I would like to see a more thorough investigation of the relation between training data size and other hyperparameters of the proposed method with the training time, in comparison to SAEs.", "methods_and_evaluation_criteria": "The evaluation of interpretability seems to rely on a recent benchmark and compares to state-of-the-art models. The baselines for the model instance layer similarity task seem quite old (2017 & 2019), but I am unsure if a more recent one (Lan et al., 2024) is applicable in this case. The authors should clarify this.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs and they appear solid.", "relation_to_broader_scientific_literature": "The key contributions are related to the mechanistic interpretability literature, specifically the training of sparse autoencoders (SAEs) for large language models. SAEs are extremely expensive to train for large models and therefore investigations are often limited to small models such as GPT-2. The proposed method is reported to be several orders of magnitude faster to train while achieving similar results to some earlier SAEs. Given that the proposed architecture differs significantly from SAEs, it is plausible that there is a lot of room for improvement from future work. Hence, this work opens up meaningful new research directions.", "essential_references_not_discussed": "The discussion of related literature in this paper is substantial.", "other_strengths_and_weaknesses": "The paper uses only 7 out of 8 available pages and thus there is significant room for improvement. For example, some of the graphs from the appendix could be moved to the main body.\nI would also like to see an investigation of how the choice of prompts etc. influences the interpretability and representation similarity results.", "other_comments_or_suggestions": "line 389: \"an efficient alternative\"", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Cjss52FfGF", "reviewer": "Reviewer_w5VN", "summary": "The main idea of this paper is to apply a dictionary learning approach to the problem of finding sparse representations of activation.  ITDA builds the dictionary at inference time. The algorithm works by first trying to reconstruct the activation from the atoms in the dictionary. Reconstruction is done by Matching Pursuit (thus scales at least linearly in dictionary size). If the reconstruction is possible with a sufficiently small number of atoms (a hyperparameter) the latent representation is accepted, if not, the $\\mathbf{x}$ is added to the dictionary. In this way, as we repeatedly iterating and updating the dictionary during inference time.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your feedback and suggestions, we’re glad you like the idea and are keen to improve the presentation.\n\n> In Section 4 they also consider representation similarity, which is an interesting new topic. I didn't fully understand the claims they were making in this section, or what the implications are.\n\nWe claim that the Jaccard Similarity between two ITDA dictionaries is a simple, performant, and state-of-the-art measure of representation similarity and outperforms existing methods on a layer matching task on GPT-2 model variants. We propose that this approach opens up exciting research directions in finding differences between models, as we can take the difference between ITDAs as well as their intersection.\n\nFor two models with pre-trained ITDAs with dictionary D0 and D1 sizes n and m, our method allows for measuring similarity in O(n + m) time. In comparison, SVCCA and CKA require learning a map between representation spaces, and the relative representation measure due to [1] requires comparing computing a dataset of activations and comparing to a set of anchor points. Furthermore, that the intersection of D0 and D1 accurately tracks model similarity suggests that the difference between D0 and D1 accurately tracks differences between models. This opens exciting research directions in “model diffing”, where the goal is to find differences between models, for example before and after fine-tuning.\n\n> I think there needs to be more explanation of the experiments. Specifically, Section 4 is a bit confusing to me. \n\nThanks for this; we have added more explanation of experiments, particularly in section 4, and we feel this has strengthened the paper and made it more readable for a general audience. Here is a summary of the changes as the full updated section is too long to reproduce here:\nFurther explanation of the significance of the results and comparison to additional methods\nGreater explanation of the layer matching task, including a formula for the calculation of the scores to remove all ambiguity\nMoved the layer convergence experiments to the appendix as they did not directly evidence the value of ITDA here\nAdded a model-level matching task to emphasize the usefulness of ITDAs for differentiating between models.\n\n> There is interchanging use of terms like \"atom\", \"token\" and \"activation\". The authors need to be more precise here. This paper could also benefit from a system-level diagram?\n\nThanks for raising this. One of the primary advantages to ITDAs is that concepts like “atom”, “token”, and “activation” are actually interchangeable: ITDA dictionaries atoms are activations, but they’re also token references. We appreciate that this can be confusing though, so we’ve added a system-level diagram (https://imgur.com/a/UIfzcdX) and some explainer text to clarify: “The elements in the dictionary can be viewed from two perspectives. For the purpose of decomposition using matching pursuit, they are absolute activation vectors taken from the model; this perspective is used when describing the algorithms for constructing ITDA dictionaries and for decomposing activations. Alternatively, the dictionary can be viewed as a collection of prompts and references to tokens in those prompts, in combination with part or all of a model. For example, a prompt may be \"The cat sat on the carpet.\", the token reference is to the second token \"cat\", and the partial model is the first 5 layers of GPT-2. The absolute activation of this element is the activation for the second token in the prompt after the 5th layer in GPT-2. We use this perspective when comparing dictionaries between models, as the prompt and token reference are model-agnostic.”\n\nOur work draws on research from the fields of sparse dictionary learning, mechanistic interpretability, and representation similarity. Each of these fields uses its own terminology, and we’ve stuck to those conventions. For example, it doesn’t make sense to refer to activations when discussing representation literature. We’ve added a glossary to the appendix to help the reader (but don’t have enough characters remaining to reproduce it here).\n\n> Can you be a bit more rigorous when defining everything.\n\nYes, absolutely. In particular, we have updated the methodology in section 3, including the CE score formula and “D ∪ {x}”,  and improved the explanation of the experiments in section 4.\n\n> What is the difference between…\n\nThese are the same and this was a formatting mistake - we have corrected this, thanks for pointing this out.\n\nWe again thank the reviewer for their helpful comments, and hope our changes are satisfying. Regrettably the character limit of this response means that we cannot reproduce the changes to the paper here, but we are happy to do so during the discussion period. If these changes have been satisfying, we politely ask the reviewer to reconsider their score. \n\n[1] Moschella, Luca. \"Latent communication in artificial neural networks.\" ICLR (2024).", "claims_and_evidence": "The authors provide some experiments of their approach. In Section 3 they report some loss as a function of $\\ell_0$ norm, which makes sense and is consistent with my expectations. \n\nThey also provide SAE benchmarks, and report some positive scores in the text, but I am not aware of the validity of these benchmarks.\n\nIn Section 4 they also consider representation similarity, which is an interesting new topic. I didn't fully understand the claims they were making in this section, or what the implications are.", "methods_and_evaluation_criteria": "I think there needs to be more explanation of the experiments. Specifically, Section 4 is a bit confusing to me. This might be because of other misunderstandings I have.", "theoretical_claims": "No theoretical claims are being made.", "experimental_designs_or_analyses": "See methods and Evaluations.", "relation_to_broader_scientific_literature": "ITDA is, as far as I can tell, a pretty unique way to learn an SAE, and some approach like this might be faster that traditional approaches.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "Overall the idea of applying iterative dictionary learning tools for learning SAEs is a good idea, however, I find that the paper is inconsistent in its language, and I think this work would benefit from more time spent on the presentation and evaluation.", "other_comments_or_suggestions": "There is interchanging use of terms like \"atom\", \"token\" and \"activation\". The authors need to be more precise here.  This paper could also benefit from a system-level diagram? \n\nYou should define what $\\mathbf{D} \\cup \\\\{ x\\\\}$ means.\n\nCan you be a bit more rigorous when defining *everything*. For example, what is the CE Loss with respect to? You don't need to go over basic definitions of course, but you need to provide enough information that I can quickly figure out what you are doing.", "questions_for_authors": "What is the difference between $x$ and $\\mathbf{x}$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gmCn3t8FA5", "reviewer": "Reviewer_miuH", "summary": "The paper introduces Inference-Time Decomposition of Activations (ITDA) as a fast and scalable alternative to SAEs for interpreting LLM activations. ITDA constructs a dictionary of representative activations using matching pursuit, allowing it to be trained 100-1000× faster than SAEs with only 0.1-1% of the data, making it feasible for models as large as Llama 3.1 405B. Despite its efficiency, ITDA achieves 90% of the reconstruction performance of SAEs and performs comparably on interpretability benchmarks such as sparse probing and automated interpretability. Unlike SAEs, ITDA enables direct cross-model comparisons, leading to a new representation similarity metric that outperforms SVCCA and CKA in layer-matching tasks. The paper demonstrates ITDA’s potential for analyzing large-scale LLMs, tracking layer convergence, and identifying model behavioral shifts, making it a promising tool for mechanistic interpretability at scale.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments, we appreciate your recognition of the strong evidence for the “efficiency and scalability of ITDA in comparison to SAEs”.\n\n> However, the comparison to SAEs is somewhat limited, as it primarily focuses on ReLU SAEs, while more advanced variants (e.g., TopK, P-Annealing SAEs) are only briefly discussed.\n\nThanks for raising this. To clarify, we did evaluate those variants as extensively as ReLU SAEs: In our experiments we compare against all SAE variants that have been evaluated by SAEBench. This includes ReLU, TopK, P-Annealing, Gated, and JumpReLU SAEs. ITDAs almost always have worse reconstruction performance than the best performing SAE variant. However, different SAE variants learn different kinds of features [1], and have different interpretability properties, so the SAE with the best reconstruction may not always be the best choice. As such, in our plots we show the SAE with the overall best reconstruction performance, and the best performing ReLU SAE. This provides a baseline for reconstruction performance without strawmanning SAEs by selecting badly optimised instances.\n\n[1] Hindupur, Sai Sumedh R., et al. \"Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometry.\" arXiv preprint arXiv:2503.01822 (2025).\n[2] https://www.saebench.xyz/\n\n> No quantitative interpretability results for larger models.\n\nOne of the major advantages of ITDAs is that we were able to train them on large models like Llama 70B and 405B on widely available hardware, unlike SAEs which require considerably more computational resources. However, this means that SAEBench does not support (or currently need to support) multiple GPUs, which would be necessary for Llama 405B. Furthermore, there would be no publicly available SAEs with which to compare the results. We could run experiments on Gemma 2 9B, as there are open source SAEs for this model, if you think this would considerably strengthen the paper (but this would probably not be possible during the discussion phase).\n\n> The paper mentions that a lower loss threshold leads to better reconstruction but a larger dictionary. How does this affect interpretability performance?\n\nIncreasing the dictionary size from 4k to 16k on Pythia models, and from 4k to 64k on Gemma 2 2B, results in a modest improvement on autointerp and sparse probing benchmarks. We have now included an appendix section going into considerably more detail on these interpretability benchmarks, for a range of dictionary sizes and L0s.\n\n> The paper mentions negative activations but does not explore their meaning. Do they correspond to specific model behaviours?\n\nThis is a good question and one that we’ve not extensively investigated. Zero-ablating negative latent activations impacts reconstruction performance, but we don’t have a good understanding of how negative latent activations affect model behaviour. If we interpret strong positive activations as meaning an input is strongly related to a feature, then strong negative activations could mean an input is strongly unrelated to a feature, which is hard to validate. We emphasize the rarity of strong negative activations, however. Our appendix latent examples were cherry-picked to include strong negative activations, but they are rare (<0.001%) in general. SAE latents frequently have small positive activations, so we are not particularly worried about the small positive or negative activations in ITDAs. \n\nWe hope this addresses your questions and we are keen to improve your faith in, and support of, this paper. Please let us know if you have any further questions or concerns. If these clarifications have addressed your concerns, we would be grateful if you might consider revisiting your overall score.", "claims_and_evidence": "The paper presents strong empirical evidence for most of its claims, particularly in demonstrating the efficiency and scalability of ITDA compared to SAEs.", "methods_and_evaluation_criteria": "The proposed method, ITDA, is well-motivated as a scalable alternative to SAEs for LLM interpretability. The use of matching pursuit for sparse coding is reasonable, and the greedy dictionary construction aligns with the goal of efficient feature extraction. Evaluation is conducted on standard benchmarks, including SAEBench, sparse probing, and automated interpretability, which are appropriate for assessing interpretability performance. The representation similarity evaluation using layer-matching tasks follows established methods (SVCCA, CKA) and provides meaningful comparisons.", "theoretical_claims": "The paper does not contain formal proofs but relies on algorithmic descriptions and empirical validation. The theoretical foundation of ITDA, particularly its use of matching pursuit for sparse coding and greedy dictionary construction, aligns with established methods in dictionary learning.", "experimental_designs_or_analyses": "The experimental design is generally sound, with appropriate comparisons between ITDA and SAEs using SAEBench, sparse probing, and automated interpretability. The cross-entropy loss evaluation effectively measures reconstruction performance, and the layer-matching task provides a reasonable test for representation similarity. However, the comparison to SAEs is somewhat limited, as it primarily focuses on ReLU SAEs, while more advanced variants (e.g., TopK, P-Annealing SAEs) are only briefly discussed.", "relation_to_broader_scientific_literature": "The paper extends prior work on mechanistic interpretability and sparse dictionary learning, particularly addressing the computational limitations of SAEs. ITDA builds on classical dictionary learning methods by using matching pursuit for inference-time optimization, enabling efficient decomposition of LLM activations. It also introduces an ITDA-based Jaccard similarity measure, improving upon existing representation similarity metrics like SVCCA and CKA in layer-matching tasks. Additionally, ITDA aligns with recent efforts in model diffing by enabling cross-model comparisons without requiring gradient-based training. While it offers scalability advantages, further empirical comparisons with advanced SAEs would strengthen its positioning in the literature.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. ITDA is 100-1000x faster than SAEs, with only 0.1-1% data required. \n2. ITDA enables representation similarity analysis, outperform SVCCA and CKA. \n\nWeakness:\n1. No quantitative interpretability results for larger models.", "other_comments_or_suggestions": "Langage -> language in the abstract line 13", "questions_for_authors": "1. The paper mentions that a lower loss threshold leads to better reconstruction but a larger dictionary. How does this affect interpretability performance? \n2. The paper mentions negative activations but does not explore their meaning. Do they correspond to specific model behaviours?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u1uQ4VKwed", "reviewer": "Reviewer_5pSy", "summary": "The paper introduced a novel method for decomposing language model activations into interpretable features that could replace SAE with matching pursuit. Thanks to its efficient and fast-converging matching pursuit algorithm (and not having to perform expensive training of NNs), this enables scalable learning about the features of an extremely large language model. While it's 10% short of the reconstruction performance of the SAE, it still suggests scalable, good approach that can be also transferred to other models.\n\n## update after rebuttal\nAlthough this paper provides a highly efficient proof-of-concept alternative to the current SAE, I believe there is room for improvement that validates the actual usefulness of the proposed method. The main concern is given the limitation and skeptics of the validity of current SAEs, the proposed method is a weak, faster version of them, which raises even more doubts about the validity. Providing more concrete evidence of the validity and usefulness of the proposed method will make the paper stronger. Still, I have decided to increase the score to \"Accept\", given the potential of the method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and we appreciate that you feel this work is novel and “its benefits are impressive”. \n\n> The efficiency and scalability feel solid, though exact times and hardware details would help.\n\nWe ran our experiments on a range of hardware so it is hard to provide precise summary statistics about resource usage. We will release a W&B project alongside the camera-ready paper that will include granular information about hardware and run-times.\n\n> My only gripe is the interpretability metrics feel vague; tighter criteria there would sharpen the story.\n\nThanks for this. We have expanded the exposition and explanation of interpretability results, and include a description of each of the metrics that we hope makes it easier for the reader to quickly understand the results. However, these benchmarks are quite complex so we feel it is better to point the reader to the SAE Bench paper than try to provide a detailed explanation here. The explanations are similar to those in the SAE Bench paper, but we don’t have space to reproduce them here.\n\n> Recently, there is a paper \"Sparse Autoencoders Can Interpret Randomly Initialized Transformers\"... [also Q4]\n\nThanks for highlighting this, SAE benchmarks are a new field and rapidly evolving so you’re right to question the validity of the results. The paper you reference found that SAEs trained on randomly initialised transformers learn single token features that are deemed to be interpretable. The same token in different contexts will likely have similar representations even in a random model, so it is not surprising that the SAE learns token features. These single token features are then clearly interpretable, as they only respond to a very specific context (presence of the token).\n\nThis isn’t the case with ITDAs, however. We draw the reviewer's attention to the analysis of Latent 17002 in the Appendix, which is an example of multi token feature that activates on contexts relating to homework in contrast to the single token latents in that paper. On a subsample on 10k inputs taken from the Pile, we calculated on how many unique tokens each latent is active and plotted it here: https://imgur.com/mm3l6jE.\n\n> latent often activates for semantically different tokens sometimes…\n\nFor these decompositions please bear in mind that the anchor text is only one part of interpreting the behaviour of the latent. While eg. \"Nurse\" may be the token activation that is used in the decomposition, it's possible that the contribution due to this token is a broader medical concept (which aligns with the context of the prompt) rather than a concept relating specifically to nurses.\n\n> Q1. How would this technique be used for activation steering? Would there be any difference to using SAE?\n\nYes, it would be used in the same way as SAEs, i.e. modifying the latent activations at inference time. We’re unsure of the usefulness of SAE steering, however, so haven’t included any results relating to this in our paper.\n\n> Q2. What would be the effect of initializing the dictionary from a different dataset? …\n\nThis is a good point, and one that we haven’t explored in our paper: we use The Pile throughout our experiments. ITDA dictionaries greedily select as we iterate over the dataset, so shuffling the dataset will likely lead to a substantially different decoder. We suspect data ordering will have some effect on the reconstruction and interpretability performance of ITDAs, and we would hope to explore this in follow-up work.\n\n> Q3. What does \"[the dictionary] is associated with a prompt and token index, rather than being learned from the activations of a specific model\"? It shows up repeatedly across the paper, but it is not clear.\n\nWe agree that this wasn’t clear in the original paper, so we’ve added a system-level diagram (https://imgur.com/a/UIfzcdX) and some explainer text to clarify: “The elements in the dictionary can be viewed from two perspectives. For the purpose of decomposition using matching pursuit, they are absolute activation vectors taken from the model; this perspective is used when describing the algorithms for constructing ITDA dictionaries and for decomposing activations. Alternatively, the dictionary can be viewed as a collection of prompts and references to tokens in those prompts, in combination with part or all of a model. For example, a prompt may be ``The cat sat on the carpet.\", the token reference is to the second token ``cat\", and the partial model is the first 5 layers of GPT-2. The absolute activation of this element is the activation for the second token in the prompt after the 5th layer in GPT-2. We use this perspective when comparing dictionaries between models, as the prompt and token reference are model-agnostic.”\n\n> (Q4 addressed above)\n\nThank you again for your thoughtful and constructive review—we hope that our clarifications and revisions have addressed your concerns, and would be grateful if you might consider updating your score accordingly.", "claims_and_evidence": "The paper claims ITDA is a faster, scalable alternative to SAEs for interpreting LLMs. It says ITDAs train 100-1000x faster, needing just 0.1-1% of the data—like 1 million tokens versus 16 billion for SAEs—and can handle huge models like Llama-3.1 70B and 405B in minutes on a single GPU. This checks out with examples like training 55 ITDAs on GPT-2 in under 2 hours versus 250 hours for SAEs. The efficiency and scalability feel solid, though exact times and hardware details would help.", "methods_and_evaluation_criteria": "ITDA’s method builds a dictionary of activations greedily, using Matching Pursuit for sparse coding, an alternative to traditional SAE’s gradient descent. It makes sense for fast, interpretable decomposition, especially for big LLMs. They evaluate with cross-entropy loss for reconstruction, automated interpretability and sparse probing for features, and Jaccard similarity for model comparisons, all tested on datasets like the Pile and models from Pythia to Llama. Using established benchmarks like SAEBench keeps it grounded. My only gripe is the interpretability metrics feel vague; tighter criteria there would sharpen the story.\n\nRecently, there is a paper \"Sparse Autoencoders Can Interpret Randomly Initialized Transformers\" that questions the validity of autointerp, so maybe it's not too robust to use it to meausre the performance of ITDA. This is the fundamental problem shared by the literature (not only this paper), however.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "They train ITDAs on Pythia-70m’s residual stream, comparing cross-entropy loss scores to SAE variants like ReLU and TopK. The formula—measuring loss degradation against zero-ablation—is clear, and plots show ITDAs plateau at 1M tokens while SAEs improve with more. They also test ITDA’s Jaccard similarity against SVCCA and CKA, matching layers across model instances. Sample size (five instances) is reasonable, and results favor ITDA. And then they track ITDA similarity during training on Pythia 70m-410m, showing early layers stabilize first. No issues here.", "relation_to_broader_scientific_literature": "The paper introduced a very big leap from the gradient-based SAE to a classical matching pursuit algorithm. Less reliance on learning the activations of the model using gradients helps the method to better generalize to other models. It would be interesting to extend it to more applications that are being studied in the literature like activation steering, safety, etc. as a future work to further see its validity and utility.", "essential_references_not_discussed": "All relevant papers were cited", "other_strengths_and_weaknesses": "The paper took a bold leap to replace SAE with a classical matching pursuit algorithm. This is very novel and its benefits (reduced training cost with a slight inference overhead and reasonable reconstruction loss) are impressive. Considering that this paper is one of the first papers to try this approach, this paper reasonably provides all the necessary, important insights already. As mentioned by the paper, ITDA maybe cannot fully replace the SAE due to its worse reconstruction loss, but it could be useful for way larger models by providing a rough insights into the model. However, at the same time, this raises a question on how useful ITDA actually is. Also, it seems the examples in the appendices show that the latent often activates for semantically different tokens sometimes. (Sequence 7008 Token 30. Nurse vs Question vs Fraction, etc.)", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1. How would this technique be used for activation steering? Would there be any difference to using SAE?\n\nQ2. What would be the effect of initializing the dictionary from a different dataset? There is a paper \"Sparse Autoencoders Trained on the Same Data Learn Different Features\" that shows that SAE is very sensitive to a random seed, which affects the weight initialization. Similarly, would initializing the dictionary with different activations lead to finding different features?\n\nQ3. What does \"[the dictionary] is associated with a prompt and token index, rather than being learned from the activations of a specific model\"? It shows up repeatedly across the paper, but it is not clear.\n\nQ4. \"Sparse Autoencoders Can Interpret Randomly Initialized Transformers\" shows that random and trained transformers produce similarly interpretable SAE latents. Would ITDA simiarly lead to this weird outcome or could it be more robust?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles a significant practical problem in mechanistic interpretability: the prohibitive cost of training SAEs for large models. The ability to train dictionaries on models like Llama 405B in minutes on a single GPU represents a substantial practical advance and opens up possibilities for studying models previously inaccessible to typical academic resources. The efficiency gains come at the cost of reconstruction performance, consistently lagging behind SAEs. Given that ITDA is a demonstrably weaker approximation (in terms of reconstruction) of SAEs, and considering the ongoing debates and skepticism regarding the ultimate validity and meaningfulness of SAE-derived features themselves. Reviewers raised significant concerns about the clarity of the presentation, inconsistent terminology (atom/token/activation), and lack of rigor in definitions, while the authors made efforts to address this in the rebuttal.
Decision: Accept (poster)