Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
Authors: Usha Bhalla
Alex Oesterling
Claudio Mayrink Verdun
Himabindu Lakkaraju
Flavio Calmon
Venue: ICLR 2026 Oral (2026)
Keywords: Interpretability
Dictionary Learning
Machine Learning
Large Language Models
Primary area: interpretability and explainable AI
TL;DR: We propose that using contextual information to train SAEs will improve their representation of semantic and high-level features.

Abstract: Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in language models.

Published as a conference paper at ICLR 2026
TEMPORAL SPARSE AUTOENCODERS: LEVERAGING THE SEQUENTIAL NATURE OF LANGUAGE FOR INTER- PRETABILITY
Usha Bhalla*at, Alex Oesterling"
Claudio Mayrink Verdun", Himabindu Lakkaraju Flavio P. Calmon "Harvard School of Engineering and Applied Science
"Kempner Institute for the Study of Natural & Artificial Intelligence
"Harvard Business School
ABSTRACT
Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent die- tionary learning methods such Sparse Autoencoders (SAEs) provide promis- ing route to discover numan-interpretable features, they often only recover token- specific, noisy, highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building this insight, we introduce Tempo- ral Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss en- couraging consistent activations of high-level features simple yet powerful modification enables SAEs to disentangle semantic from syn- tactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering new pathway for unsu- pervised interpretability in language models.
adjacent tokens. This
INTRODUCTION
Interpretability aims to translate the internal representations and computations of language models into concepts that humans can understand Kim et al. (2018), evaluate (Koh et al. 2020), and ulti- mately control (Bhalla et al. 2025). In practice, the most useful insights involve high-level drivers of model behavior, such as the semantics a model encodes or the state operating in, rather than surface-level statistical patterns. Recent dictionary learning methods, such as Sparse Autoencoders (SAEs), have shown promise in explaining language models (Bricken et al. 2023). By projecting dense latent representations into sparse, human-interpretable feature space, SAES enable both the recovery of known linguistic patterns and the discovery of novel concepts within models.
However, when applied large language models (LLMs), SAEs frequently fall short of this goal. The features they recover are often token-specific, local, unstable, and noisy, capturing super- ficial syntactic patterns (e.g., "the phrase 'The'
the start of sentences") or "Sentence endings or periods," Figure I rather than coherent, high-level semantic con- cepts (Menon al. 2025 Paulo & Belrose 2025). One interpretation of this phenomenon that LLMs themselves ves fail to encode deep semantic structure, functioning instead as sophisticated next- token predictors. A more plausible explanation, however, is that current concept discovery methods are inadequate; their design biases them toward recovering shallow patterns even when richer struc- ture exists the underlying representations. We argue that this limitation stems from a fundamental issue with how SAES are formulated. Human language is inherently structured: meaning is con- veyed through context and semantics that evolve smoothly over time, while syntax is governed by more local dependencies (Chomsky 1965; Griffiths et al.) 2004). Yet current dictionary learning methods ignore this sequential structure, treating tokens as independent and stripped of context.
Equal contribution. Correspondence to I fusha bhalla la, aoesterl inq]eq Bel.ahharveee harvard.edd
Neuronpedia, Feature 1 11795 of Gemmascope's 's Gemma2-2h 16k SAF, see Appendix D for more examples.

---

Published as a conference paper at ICLR 2026
5 Active
Spliced Latin,
15gh-inval: Contect
Genetics, Roligious Text
Matrycahka SAVE Penturne:
Sa tax
Feeture Activations
endinge
palloce
The uord thu"
CAC
Puntactse OMLY
Tempporll SAE
Terppora EAI Features:
Hor englantat
Tacnnica interative
Dontext
Sentox
Continer
eig lin ond sumes
Pratis expaniiinn
spiitaa
Siice
0,atio
Quosiion
Dhagaaoe
Figure 1: A) Human language production involves high-level features such as semantic content and surrounding context, as well as low-level features such as syntactical requirements and specific word choices. B) While existing SAEs mostly recover syntactic information, T-SAEs balance recovery of semantics, syntax, and context. C) When decomposing a sequence composed of three passages: Newton's Principia, an MMLU genetics question, and the Bhagavat Gita (in English), T-SAEs (bot- tom) smoothly detect the semantic shifts in the passage, with highly active features strongly corre- lating to the true content of the text, whereas existing SAEs (such Matryoshka, top), are much noisier, varying on almost per-token basis, and do not easily depict these shifts.
To address this gap, we introduce the notion of temporal consistency, or the property that high-level semantic features of a sequence remain stable over adjacent (or nearby) tokens, while low-level syntactic features may fluctuate more rapidly. For example, in the sentence "Photorynthesis is the process by which plants convert sunlight into energy." a temporally consistent semantic feature might represent "diseussion of plant biology" or "scientific explanation." We expect this feature to be active over the course of the sequence rather than simply on a few specific tokens. In contrast, syntactic features such as "capitalized first word" or "plural noun" activate only at specific tokens (e.g., "Photosynthesis." "plants"), reflecting local rather than global structure.
Building on this principle, we propose Temporal Sparse Autoencoders (T-SAEs), a simple mod- ification to SAEs that incorporates a temporal contrastive loss encouraging high-level features to activate consistently over adjacent tokens (Figure DT. This modification, grounded linguistic intu- ition, enables T-SAEs to more reliably capture semantic features while still disentangling them from low-level syntactic ones. Despite being trained only on a self-supervised context-similarity objec- tive, T-SAEs yield representations with significantly improved semantic structure, without requir- ing any explicit semantic signal. Through experiments, we show that T-SAEs consistently recover higeer-level semantic and contextual concepts, exhibit smoother and more temporally consistent tivations over sequences, and remain competitive with existing SAEs on standard benchmarks such as reconstruction quality. Our contributions are the following:
1. We introduce simple data-generating process for language that distinguishes between high-level, temporally consistent semantic variables and low-level, local syntactic vari- ables. This framework formalizes how expect language models to encode linguistic information and provides guidance for designing better interpretabilitty methods.
2. Building on this framework, we propose Temporal SAEs, which partition latent features into semantic and syntactic components. We introduce a novel temporal contrastive loss which enforees consistency of high-level activations across sequences, encouraging T- SAES to disentangle semantic and syntactic features in a self-supervised manner.
3. Through experiments on multiple models and datasets, we show that T-SAEs: a) Recover semantic and contextual information more reliably than existing SAEs, b) Exhibit improved disentanglement between high- and low-level features, c) Maintain competitive perfor- mance standard reconstruction benchmarks, and d) Provide practical interpretability benefits, including a case study on safety-related concepts.
We release our code, trained T-SAEs, and interpreted latents for reproducibility
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
recent years, SAEs have emerged as a popular mechanistic interpretabil- ity technique for unsupervised concept discovery (Bricken al. 2023 Templeton et 2024 Gao et al. 2024). While they were initially promising for addressing the problem of polysemanticity, where a single neuron in a model can represent multiple concepts (Elhage et al. 2022). in prac- tice, they have been shown to create new problems, such as feature splitting and absorption (Chanin (et al. 2024), where concepts are split across multiple features or absorbed into less interpretable Sub-features. To address these subsequent issues, methods such as Matryoshka SAEs (Bussmann ct al. 2025) and Transcoders (Paulo et al. 2025) have been proposed, which learn hierarchical and causal features. Recent work has also proposed learning dictionary features that are constrained to the data manifold (Fel et al. 2025) and reflect intuition about the geometry of model latent spaces (Hindupur et al. 2025), allowing for the recovery of heterogeneous concepts. However, all of these works assume a fully unsupervised objective for learning SAEs, treating each token in the training data as i.i.d., without acknowledging the temporal aspect of language and other sequential modali- ties. As a result, SAEs are known to suffer from a variety of problems. including "dense" activation
Sparse Autoencoders.
behavior (Sun et al. 2025) and lack of utility for steering (Bhalla et al.) 2025 Wu et al. 2025)
The works most relevant to ours use temporal signal as weak supervision to learn more causal SAE
features Menon et al. (2025) and address absorption Li & Ren (2025).
I Linguistics, Cognitive Science, and Neuroscience. Many foundational works in linguistics study the relationship between syntactic structure and semantic meaning (Chomsky 1965), with nearly all major theories recognizing a fundamental distinction between semantics and syntax. Evidence of the two having distinct representations has been found in developmental psychology (Brown) 1973)
and neuroscience (Neville et al. 1992 Zhang al. 2025). >. In computational linguistics, separate approaches emerged to model language purely syntactically via Hidden Markov Models (Manning & Schuuzee 1999) or through bag-of- words approach to model topics with Latent Dirichlet Allocation (Blei et | 2003). Griffiths et al. (2004) combine the two into a single model consisting of an HMM where one "semantic" class denotes topic model which samples words in an LDA-like fashion. Importantly, Griffiths et al. (2004) argue that semantics in language exhibit long-range behavior, with different words or sentences in the same document having similar semantic content, whereas syntax is mostly dependent on short-range interactions, motivating our method.
Dictionary and representation learning with natural priors. Oishausen & Field (1996) found that decomposing natural images in a sparse linear manner leads to Gabor-like receptive filters, similar to what found in the visual cortex without any priors on visual data, ushering in work on sparse coding through dictionary learning (Tošić & Frossard] 2011 Dumitresms & Irofti 2018))
Later approaches then realized the benefit of taking data priors into the dictionary learning process, such as low-rank (Davenport & Romberg 2016)] Vu & Mongal 2017) multi-scale structure for images (Ong& & Lustig 2016), and temporal smoothness for video (Luo et al. 2019). Early works in feature and representation learning used time-contrastive approaches, assuming smoothness (Oord) et al. 2018 for predictive coding, and nonstationarity (Hyvarinen & Morioka (016) and sparse
transitions (Klindt et al.) (020) for disentanglement. Our work draws parallels to this trajectory of research in dictionary learning and representation learning by introducing structural priors to the unsupervised learning of SAES.
3 OUR FRAMEWORK: TEMPORAL SPARSE AUTOENCODERS
3.1
FORMULATING THE DATA GENERATING PROCESS
Consider speaker who is producing language, or sequence of tokens T1, .i....;7 .r. When the speaker produces each token 7. they take into account many factors their intent in speaking, the prior context of the token (i.e., what has already been said), syntactic requirements, and other implicit features corresponding to speaker idiosynerasies (such as their accent, their method of language production, or linguistic style). These factors can be modeled as latent variables that control the language generation process, and they can be generally categorized into two types: variables that encode high-level or global information, h, and variables that encode low-level or local information 4. High-level variables can be thought of features that are invariant to the specific token, such
3

---

Published as a conference paper at ICLR 2026
those capturing semanties and intent. Conversely, low-level information pertains to the specific timestep or token being produced, such as a word's grammatical gender.
We model the speaker's language production process as function mapping the context and these latent variables to the next token
Tt T= (0(TT=11 h4, 14),
where -f represents the sequence Ti, T1-1- We pass tokens 7" into a language model which produces latent vectors {x E R at layer L. For simplicity, we analyze a single layer and drop the superscript. We assume that the model represents h; and through an invertible mapping such that 14,,,1) = x,. Our goal is to recover the encoding of the data data-generating latent variables by decomposing its representations x into interpretable features corresponding to h,, 4. To do so, we make the following key assumptions:
Assumption (Temporal Consistency.). h, is time invariant, meaning two tokens X,, Xe sampled from the same sequence should have similar latents h, R hg-
Assumption 2 (Hierarchical Representation of Features.). The mapping g hierarchical in the sense that it satisfies 0 = (lg 1e) - xell S [gg 0) - X, ll S c. In other words, he can reconstruct xt up to 6, but 1 contains signal about x, that is not explained by h,.
3.2 TEMPORAL SPARSE AUTOENCODERS
To model the above process, we partition our SAE feature space into high-level and low-level fea- tures. Without loss of generality, we assume the first indices are our high-level features and the last - h indices are our low-level features, where m is the number of features in the SAE. The SAE architecture can be defined as the following, taking in input X2 E Rd:
f(x:) = (WWeee x + bene),
f(f) = Wdeef(xi) + bdec.
Here, Wene mgd is the encoder matrix, and Wdec E Rdxm is the decoder comprised of high-level features Wdee 0:h E Rdxh and low-level features Widee
ham E [Rdx(m-h) such that their con- catenation equals Wdee The encoder and decoder bias are beue E IRd and bdde E Rd,, respectively. We define the following loss function, where the high-level features (x,) should reconstruct the input and the low-level features flom (x,) should reconstruct the residual, as discussed in Assumption 2 similar the Matryoshka SAE objective in (Bussmann et al. 2025).
Lmaat (Xxe) Lи LL,
CH |x Wile + 1
LL = ||x W + b
features fol- lowing Assumption []about h,: high-level features should be similar for two tokens from the same sequence, particularly for two adjacent tokens. To enforee this, we add contrastive term to the loss function that encourages W to be similar to - In addition to encouraging tempo- ral consistency between pairs, discouraging similarity across different samples prevents smoothness collapse (where high-level features are constant) and encourages full use of the high-level feature space. Let 2, be the high-level features fo.s (x:), and let s(x, y) be the cosine similarity between vectors x and in the same latent space. Our full loss over a batch is subsequently
We then add training objective that encourages Wene to learn temporally
0ch
C Σ
Emare
(x, (x( + a.Couurr
Ccontr N log exp(>(2)" (1) 2221) 1 E exp(s(2)) 221)) N
Σ log exp(s(2,)) (0) ,,,,)) ΣN exp (s(z)" (3) 2,2)
where N our batch size and x 2, are the ith model activation and SAE latent vector in the batch, respectively. In practice, we load activations in pairs x,, Xt-1 and shuffle the pairs to get diversity in each batch. We additionally explore an approach where we sample the contrastive pair uniformly over past tokens X1, ..., 1 encourage long-range semantic consistency (see Sec. (.6).
While the contrastive loss is applied only to high-level features, for low-level, token-specific fea- tures, do not apply any constraints. By nature of fitting the residual data left unexplained by the high-level component of the network, our loss naturally encourages the low-level latents to capture remaining, fluctuating features over sequence.
4

---

Published as a conference paper at ICLR 2026
(High- features)
European ritory
1899 Kising's
Tempora SAE
Formal logic
Quastion about probability
Professional
(Low-level features)
Panciuation
Temporal SAE
Matryoshik SAE
(All features)
Nouns
Punctuatinn
Semantic Labeling
Contextual Labeling
Syntactic Labeling
Figure 2: TSNE visualizations of Pythia-160m SAE decompositions of MMLU questions, labeled by question category (left), question number (middle column), and token part-of-speede (right). We see that T-SAE high-level features (top) recover semantics and context. The low-level features of T-SAEs (middle row), as well as Matryoshka SAEs (bottom), recover syntactic information.
EXPERIMENTAL EVALUATION
In this section, we evaluate Temporal SAEs. Sec. we evaluate our Temporal SAE's recovery and disentanglement of semantic, contextual, and syntactic content, in Sec. 4.3 we report results on standard SAE evaluation metrics, in Sec. 4.4 we measure various aspects of temporal consistency, in Sec. 4.5 we present case study of how SAEs can help uncover safety-relevant concepts and mechanisms LLMs, and finally in Sec. 4.6 we provide results on various ablation studies.
4.1 IMPLEMENTATION DETAILS
Models and Datasets. We conduct all experiments on Pythia-160m (Biderman et al. 2023) and Gemma2-26 (Team et al., 2024). We compare against baselines of BatchTopK SAES (Bussmann et al. 2024), Matryoshka SAEs (Bussmann et al., 2025). and the model latents themselves when applicable. All models are trained and tested on the Pile (Gao et al., 2020). All probing evaluations done on MMLU (Hendrycks al. 2020), Wikipedia Wikipedia (2004). and FineFineWel (M-)
A-P al. 2024).
Hyperparameters. We train Pythia SAEs on layer 8 and Gemma SAES on layer 12. All SAES are trained with the BatchTopK activation (k=20), 16k features, and the auxiliary loss from (Bussmann) et al. 2025 Gao al. 2024). These models, layers, and hyperparameters for comparability with pretrained and evaluated SAEs on Neuronpedia (Lin 2023). Temporal and Matryoshka SAEs trained with 20%-80% feature splits, where for Temporal SAES the 20% are the high-level features. We use a regularization parameter of 1.0 on the temporal loss for all Temporal SAEs. For ablations on hyperparameters, see Section 4.6
chosen to allow
4.2 PROBING FOR SEMANTICS, CONTEXT, AND SYNTAX
To understand the types of features that the temporal loss encourages SAES to learn, we evaluate the ability of Temporal SAEs recover different types of linguistic information, namely semantic, contextual, and syntactic. We provide qualitative visualizations of these results in Figure and quantitative probing results in Figure
In Figure 2 we present TSNE visualizations of the Pythia- 160m SAE activations for various ques- tions taken from the MMLU dataset, which color by the semantic content of the question, the
5

---

Published as a conference paper at ICLR 2026
Semantics, MMLU
Context, MMLU
Syntax, MMLU
Temporal SAE
Accuracy
0.05
0.5
Matryoshka SAE
BatchTTpp SAE
Model
0.55
20 0.4
02
10
Sparsity
Figure 3: Accuracy of probes trained on SAE decompasitions for various SAEs trained on Gemma2- 2b, as well as probes trained directly on model latents (orange), with semantic labels (right), con- textual labels (middle), and syntactic labels (right) with varying levels of probe sparsity (setup from (Kantmmneni et al. )025 Dense probes are trained on all features.
context of which sequence a token came, and the syntactic information for each token. In particular, we encode the last 20 tokens from each question into the SAE's feature space, and use TSNE as a dimensionality reduction and visualization method understand how the SAE embeddings are clustered. We use the question category, such as "High School European History" or "Professional Medicine" as proxies for the semantic information. The contextual information simply refers to the question ID of each token, meaning tokens from the same context should have the same color. We use an open-source NLP library, spaCy (Honnibal et al. 2020), to retrieve part-of-speech labels for each token proxy for their syntactic content.
We find that the activations of high-level features from Temporal SAEs cluster strongly according to semantic content (top left) and contextual content (top middle), meaning tokens from the same question are represented similarly, as well as questions from the same topic. On the other hand, activations of the low-level features seem be more syntactic, with stronger clusters for parts-of- speech (middle right). Matryoshka SAE embeddings prioritize syntactic information strongly over semantic and contextual information, with clear clustering for part-of-speech (bottom right) and minimal clustering for the other two labels.
Probing Validation. In order validate these visual results, we probe the SAE activations for these same labels, using both k-sparse probing from Kantamneni et al. (2025) as well as normal Logistic Regression probes on the full activations. We train probes on k 1, 5, 10, 20 features, where we select the features by comparing the mean activations of the positive and negative examples for each class from the train set. Note that dense probes trained on SAE activations have dimensionality of 16k whereas dense probes trained directly on the model's 's residual stream have dimensionality of 768 for Pythia-160m and 2304 for Gemma2-2b. We find that these results quantitatively reflect the qualitative results from the TSNE plots, with Temporal SABs outperforming the baseline SAEs significantly for semantic and contextual labels, with little-to-no performance drop for syntactic information. Probing results for two more datasets, Wikipedia and FineFineWeb, well as for Pythia-160ms are in Appendix C.5 with the same trends across all models and datasets.
Feature Disentanglement. While we that Temporal SAES recover more semantic and contextual information, can this behavior be attributed to the high-level features alone? Indeed, we observe specialization between high- and low-level features in Figure where the high-level features exhibit semantic and contextual structure, whereas low-level features exhibit syntactic structure. Similarly, our smoothness metrics (Table show that T-SAE high-level features are smoother than baselines and T-SAE low-level features are less smooth, likewise providing evidence of separation. To further validate this behavior, we measure probing accuracy on each feature split separately and find high- and lowelevel task specialization (Appendix C. Interestingly, we find that the low-level features are able to recover syntactic information on their own, despite only be trained to reconstruct the residual left by the high-level split. In contrast, for Matryoshka SAEs, across all tasks, performance can be attributed almost entirely to the high-level feature split, with the low-level features being significantly less predictive for semantics, context, and syntax, indicating lack of disentanglement in baselines.
Disentanglement allows users to (1) attribute behavior to specific types of features and (2) more precisely control the types of features they care about. For example, in domains such as safety monitoring, users may care more about the semantics of the output (e.g. whether the output contains

---

Published as a conference paper at ICLR 2026
Table 1: Core Performance Metrics. We report smoothness on feature splits when applicable and
standard deviations for autointerpretability scores.
Cosine Sim (t)
Fraction Alive (t)
Smoothness High
Autointerp Score (f)
FVE (t)
Full
Low
Pythia
160m
Temporal SAE
0.94
0.93
0.87
0.12
0.09
0.17
0.81 + 0.17
Matryoshka SAE
0.95
0.94
0.89
0.12
0.12
0.13
0.83 + 0.16
BatchTopKSAE
0.95
0.94
0.84
0.13
0.85 + 0.15
Gemma
Temporal SAE
0.75
0.88
0.78
0.13
0.10
0.15
0.83 + 0.15
2-2b
Matryoshka SAE
0.75
0.89
0.76
0.14
0.15
0.12
0.83 = 0.16
BatchTopKSAE
0.76
0.89
0.66
0.13
0.83 + 0.16
sexually-explice content), whereas in poetry or coding tasks, users may care about syntax and local features (e.g. vowel sounds or closing brackets).
4.3 SAE EVALUATION
In Table [ we provide results for various standard evaluations of SAEs to ensure that tempo- ral consistency does not significantly degrade reconstruction quality. Fraction Variance Ex- plained (FVE): The fraction of the SAE input data's total variance that is successfully cap- tured by the SAE decomposition, The cosine similarity between the SAE inputs and outputs. Fraction Alive: The proportion of SAE features that activate at least once on the test data. Smoothness: For each sequence s, we filter for active features (features that fire at least once over the sequence). We compute
Var(x @) ///r(r)). Cosine Similarity (Cos Sim):
A. 12' El= max.e( (x:) (x [[x; xt-1l12 the average max absolute change over the active features normalized by the change in model latent: Finally, we average over multiple sequences to get smoothness score, S 1
" Σ. A, Automated Interpretability (Au- tointerp) Score: Score for how correct feature explamations are. We use SAEBench (Karvonen et al. 2025) to generate and score feature explanations with Llama3.3-70B-Img Instruutt For each latent, the [LM generates potential feature explanations based on a range of activating examples. 'hen, collect activating and non-activating examples and ask judge (also Jlamaa.33-00B-Instuuc)) to use the feature explanation to categorize examples and score its performance
We see that Temporal SAES perform nearly equivalently to both Matryoshka and BatchTopK SAEs for both Pythia- 160m and Gemma2-2b for FVE, Cosine Similarity, Fraction of Alive Features, and Autointerpretability score, meaning the added improvement temporal consistency and semantic information recovery does not come at significant cost to core SAE performance.
4.4 TEMPORAL CONSISTENCY
Table I indicates that Temporal SAE features are smoother and more consistent than baselines. To explore this further, visualize the top feature activations of Temporal SAEs over long sequences of text (Figs. 4 In Figure we concatenate four sequences text: a biology question (MMLU), letter from Charles Darwin (Project Gutenherg), mathematics question (MMLU), and interpret them with a Temporal SAE. We take the top-8 most active features across the sequence and plot their activations for each token. The Temporal SAE fea- tures have clear phase transitions between texts, with different features activating and deactivating for specific sequences. Furthermore, within a sequence, these features are relatively smooth, with- changes in activation from token to token. We can even detect periodic
article Animal Farm (Wikipedia), and
out spiky, behavior in the biology question, corresponding to each multiple-choice answer option. We note that there is an interesting leakage of features that continue to fire in later, semantically unrelated sections of the spliced sequence, highlighting that the model may retain past context and that the Temporal SAEs are able identify this information rollover. Figure I conducts similar analysis over Newton's 's Principia (Project Gutenberg), a genetics question (MMLU), and the Bhagavat Gita
"This can be viewed as the average per-feature Lipschitz constant across sequences. NNote that both the generation and scoring phase highly noisy and dependent on the LLM judge.

---

Published as a conference paper at ICLR 2026
Most Active Temporal SAE Features Across Spliced Text
Feature Activation
Formal lnnguage
Proper nouns and sclertiic
Scientific compansoon and
Quated seeech and text within cuotations
Historical literature and academic writng
Math concepts
Numbers years
tarms
Biology question
1832 Letter from Charles Darwin
Wikipedia article Animal
Math question about abelian groups
about herbicide
Figure 4: Top most active Gemma2-2b Temporal SAE features over a concatenated sequence of text. Temporal SAE features exhibit clear phase transitions between sequences, are relatively smooth, and have explanations relevant to the semantic content of cach component sequence.
(Project Gutenberg) and compares the top features provided by the Temporal SAE to those gener- ated by a baseline Matryoshka SAE, We find similar consistency and smoothness over sequences with clear transitions between sequences for the Temporal SAE, whereas the top Matryoshka SAE features activate across all three sequences without differentiating them, and are much noisier and fregufrreuency fluctuating across tokens. Finally, we interpret these features using automated in- terpretability and report their explanations in the figure as annotations about the activations. We find that the labels reflect the true underlying semantic content present in each component sequence, with the 'Animal Farm' Wikipedia article activating a feature for "Historica and academic writing" in Figure 4 and the Bhagavat Gita excerpt in Figure activating "Worship and spiritual practices" feature.
riterature
Sequence-level Interpretability. Figure and prior work (Sun et al. 2025) demonstrate how base- line SAE features are "dense," " in that many features fluctuate frequently over a sequence. In doing so, they can only provide cittreraittereetbbll explanations at the token level; as soon as you zoom out, you get an overwhelming set of activating features with very little structure or parseability. The smoothness of Temporal SAES unlock a sequence-level understanding of data which was previously much harder, if not impossible, to parse from baseline SAE explanations (Figs.
4.5
CASE STUDY IN USING TEMPORAL SAES FOR DATASET UNDERSTANDING AND CONTROL
While many SAE evaluation metrics allow us to measure how sparsely and effectively SAEs can re- construct language model latents, these metries do not often generalize to how effective a dictionary is for downstream applications. Furthermore, for many safety applications where potential harms are well-known and casily measurable, is not necessarily clear that unsupervised concept discov- ery methods offer any advantages over supervised methods, such probing, steering, finetuning. However, SAES can be an incredibly helpful tool for (1) discovering features and surfacing spuri- correlations failure modes that may not otherwise have been apparent and (2) allowing for computationally efficient inference-time intervention. To illustrate this, we present two case studies using T-SAEs, first to analyze human preference and alignment data at scale and second to steer models more effectively with high-level temporal features.
Dataset understanding. We use T-SAEs to analyze Anthropic's Helpfulness Harmfulness RLHF dataset (Bai et al. 2022) to discover features that safety labelers prefer when annotating data, as well as potential unknown spurious correlations that may need to be addressed when training safety policy models. For each candidate completion, we inspect the difference between the mean T-SAE feature activations for chosen and rejected responses over the 'harmful/harmles' split of the dataset, similar to previous work on using SAES to understand human preferences (Movva et al. 2025).
As expected, many of the features that activate highly for the rejected data are relevant to unsafe topics or intuitive human preferences. We observe features such as "physical touch and intimacy", "etiquette and social behavior guidelines" "crime and malicious activities". and "violent or aggres- sive behavior descriptions". These features clearly capture the semantics that we expect to find in an alignment dataset. However, a second class of features also appears with seemingly much less relevance to the task at hand, such "legal and formal language", "quotes and speech", and "tran- sition words and phrases". At first glance, these features may seem like random noise artifacts or an issue with the T-SAEs themselves; however, upon further inspection, find that they are the result
8

---

Published as a conference paper at ICLR 2026
Safety Dataset Understanding
Steering Tradeoffs
maameeeommss Preference Date
Temponal SAE
S varies text concects"
Pepothell:: resporie lengi
fiRoring
Safery-waccart
velquette and social behouite
Coherence
St Spunizaa IeeARs
fatime and malicious activtiie"
toxt acous personal
dige shr langer
24
Soccal issurs and controneryy by
SAE
Neuronpedia dia Motryoshika SAE
Palicy Model
LLM Sarkey
Spactis bicydl of
tares warrd to arammnaeeerr
Finetuning
Intervention Sucoess
estee rrrmulns zantec
ageeaz
865 Model
Mathysshva @ Neutrepeds
and andisenntial cxvciiiin
Intevertion
Figure 5: Left. We use SAES to understand user preferences in the HH-RLHF Bai et al. 2022)) dataset. The Matryoshka SAE appears find more random features, whereas the T-SAE is able to capture safety-relevant features. The T-SAE also finds a spurious length correlation in the dataset, where rejected completions are longer than chosen completions. Right. We find that high-level features Pareto dominate baselines in their ability to steer LLMs. Specifically, they are better at preserving coherence while successfully changing the semantics of model generation.
of consistent trend in the data. Frequently in the HH-RLHF dataset both options provided to the labeler are undesirable [46], generally falling into two cases: two harmful options or one harmful and one unhelpful response (where unhelpful may mean incoherent or irrelevant the prompt). In the second case, raters consistently prefer the unhelpful response over the harmful one, leading to various undesirable spurious correlations. More specifically, the chosen response is often a single incoherent or irrelevant sentence that does not pertain to the request, whereas the rejected response compliant with the request and will often include harmful references to online resources, quotes from prominent, divisive figures, and generally just lengthier responses. This may lead them to prioritize noncompliance over quality, resulting in a higher prevalence of formal language, quotes, and longer responses in the rejected, unsafe responses, as picked up by the T-SAE with the afore- mentioned features (Appendix B.I).. In contrast, existing SAEs highlight features that are generally uninformative and unrelated to the data, such as "specific bicycle components," "terms related to data management", and "references to ecosystem dynamics and environmental conditions" (Figure 5). The discrepancies between the safety concepts recovered by T-SAEs and Matryoshka SAEs highlight the utility of consistent, high-level features over noisy, local features for data aggregation.
Steering. In order to further validate the downstream utility of training SAEs with higher- semantic content, we evaluate whether T-SAEs improve steering performance, as past literature has found that SAES are not useful for inference-time intervention (Wu et al. 2025 Bhalla et al. 2025 Arad et al., 2025). In particular, we follow the evaluation scheme from (Bhalla et al.) 2025) which measures the tradeoff' between steering success (do the outputs contain the feature that was inter- vened on) and steering coherence (are the outputs coherent), averaging across 30 different features (see Appendix B.2 for more information). We use Llama3.3- 70b to grade intervention success and coherence, with manual verification of grading. In Figure 5 we find that T-SAEs pareto-dominate existing SAES, including the best SAE available for the same model on Neuronpedia (Lin) 2023).
Our results indicate that steering with high-level, semantic features results in more coherent and relevant generations than baselines. Intuitively, steering on low-, token-level features will result in a simple overwriting of the next-token generation objection, resulting in low- token-level changes in outputs and the increased prevalence of specific tokens. In fact, a common failure mode when steering with state-of-the-art SAES token repetition at aggressive steering strengths (Appendix B.2] Table 7 right). In contrast, steering with high-level features changes the model's I's encoding of the semantics of the task, guiding the model to generate text with the semantic content of the feature rather than simply encouraging token repetition (Appendix B.2 Table left). As we train high-level T-SAE features to activate consistently over a sequence, these features are more amenable to steer- ing, which generally involves exciting a feature over entire sequence and is thus in-distribution for temporally consistent features. We additionally find that steering with high-level features is more forgiving: it properly impacts generation at lower strengths and continues to impact generation mantically up to high strengths without failing, whereas steering with baselines requires precise tnnin to find "sweet spot," otherwise it will have no impact or will result in catastrophic failure.
9

---

Published as a conference paper at ICLR 2026
Table 2: Difference in performance between ablation and normal Pythia-160m Temporal SAEs.
Fraction Alive
Smoothness (High)
FVE
Semanties
Context
Syntax
Random Contrastive
0.0
-0.05
0.0
-0.02
+0.11
-0.10
50:50 Split
-0.01
+0.01
0.0
+0.02
+0.09
-0.08
10:90 Split
0.0
-0.07
-0.02
-0.01
+0.01
+0,01
No Contrastive
+0.01
+0.06
+0.07
-0.07
-0.1
+0.01
4.6 ABLATION STUDIES
In T we explore the effect of varying components in our Temporal SAE training pipeline. All results are reported as the difference between the ablation and the normal Temporal SAE with the implementation described in Section 4.1 We conduct semantics, context, and syntax sparse probing evaluations on MMLU with & 5. First, we conduct a hyperparameter sweep on the high- and low- level feature percentages, varying the proportions to 10:90 and 50:50. As expected, if we increase the size of our high-level split, Temporal SAEs better recover semantics and context but perform worse syntax. We next study the impact of contrasting with token sampled randomly from any previous token in the context window, the t - rth token with r < 25 rather than from the t 1st token. When contrasting with random token, we incorporate even longer-rang range dependencies into our temporal constraint; as a result of this, we see a large increase in performance on the context task and large decrease in performance performance. Depending on the interpretability application, this behavior may be preferable to that of the Temporal SAEs trained on the immediate previous token, highlighting the need to carefully consider the features we hope to find when using unsupervised concept discovery methods. Finally, we explore the impact of the contrastive component of our loss term, and consider the naive baseline of a sample-wise temporal similarity loss, & = ar||z, = 11 averaged over a batch. This naive approach enforces less structure in the high-level feature space, resulting in worse semantic and contextual performance but allowing for better performance on standard reconstruction metrics.
the syntax task, but with minor change semantic
5 DISCUSSION AND CONCLUSIONS
The efficacy of "bottom-up" interpretability methods, which aim to discover and represent concepts learned by large neural networks in an unsupervised fashion, has been fiercely contested topic. In recent years, dictionary learning methods such as Sparse Autoencoders were hailed as a triumph in the interpretability community, showing promise their ability to uncover unexpected and novel concepts, and providing potential path forward for steering and control. However, the excite- ment wore off, it became clear that SAEs suffer from a variety of issues, one of which being that they systematically fail to capture the rich conceptual information that drives linguistic understand- ing, instead exhibiting a bias towards shallow, syntactic features, such as "the phrase 'The' the start st. of sestences." This lack of deeper semantic concepts could potentially be attributed to the underlying LLMs themselves; maybe they don't 't truly understand semantics or prag- matics, and it's thus unrealistic to expect that interpretability methods would find them. But more likely than not, current concept discovery methods simply aren't 't good enough to reveal the types of features generally are interested in and that LLMs likely encode.
In this work, we propose that this is due to a fundamental issue with how dictionary learning methods for LLMs are trained. Language itself has rich, well-studied structure spanning syntax, semantics, and pragmatics; however, current unsupervised methods largely ignore this linguistic knowledge, leading to poor feature discovery that favors superficial patterns over meaningful concepts. We focus on simple but important aspect of language: semantic content has long-range dependencies and tends to be smooth over sequence, whereas syntactic information is much more local. We propose novel loss function for training SAEs such that subset of features behaves smoothly over time, better extracting semantic features from data. Through experiments, we demonstrate that the features leamned by Temporal SAEs are more semantically structured, with minimal loss to reconstruction performance. We also present a case study demonstrating how this semantic information can be valuable in practical, real-world applications, such finding safety vulnerabilities and steering.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
This work is supported in part by the National Science Foundation under grants CIF 2312667, FAI 2040880, CIF 2231707. IIS 10S-200 NSS-2049,,, IIS-2238714, 14, the AI2050 Early Career Fellow- ship by Schmidt Sciences, and faculty research awards from Google, OpenAI, Adobe, JPMorgan, Harvard Data Science Initiative, and the Digital, Data, and Design (D"3) Institute at Harvard. UB funded by the Kempner Institute Graduate Research Fellowship. AO is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGB-2140743.
7 REPRODUCIBILITY STATEMENT
All language models, libraries, and datasets used in this paper are publicly available and open- source. We describe our implementation in our section on Implementation Details (Sec. 4.T). our training approach and model architecture in detail in Section 4.4 and highlight the existing libraries and methods we use for evaluation in Section 4.3] We also describe our custom evaluations in detail in the experiments section 4 Upon publication, we will open-source our codebase and T-SAEs for further reproducibility.
8 ETHICS STATEMENT
Interpretability is closely tied to the ethical development and application of AI systems. On one hand, developing better understanding of models can help highlight biases and failure modes to ensure they treat all people ethically. On the other hand, the understanding they bring can allow malicious actors more efficiently jailbreak models or control them for nefarious purposes. We are aware of the potential dual uses of such technique but hope that improving the transparency of AI systems will enable development of safer Al systems.
REFERENCES
Dana Arad, Aaron Mueller, and Yonatan Belinkov. Saes are good for steering if you select the right features. arXiv preprint arXiv:2505.2000 20063,, 2025.
Yuntao Bai, Andy Jones, Kamal Ndousse. Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05802 2022.
Usha Bhalla, Suraj Srinivas, Asma Ghandeharioun, and Himabindu Lakkaraju. Towards unifying interpretabillity and control: Evaluation via intervention, 2025. URL httttt::///aai......t
abs/2411 04430
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Allah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pp. 2397-2430. PMLR, 2023.
David Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan)993-1122. 2003.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph. Zac Hatfield-Dodss, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer-
Roger Brown. first language: The early stages. Harvard University Press, 1973.
11

---

Published as a conference paper at ICLR 2026
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders. arXiv preprint arXiv:2412.06410, 2024.
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi-level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503. 17547, 2025.
David Chanin, James Wilken-Smith, Tomáš Dulka, Hardik Bhatnagar, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409. 14507, 2024.
Noam Chomsky, Aspects of the Theory of Syntax. MIT press, 1965.
Mark A Davenport and Justin Romberg. An overview of low-rank matrix recovery from incomplete observations. IEEE Journal of Selected Topics in Signal Processing, 10(4):608 6226622,,2 2016.
Bogdan Dumitrescu and Paul Irofti. Dictionary learning algorithms and applications. Springer, 2018.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodss Robert Lasenby, Dawn Drain, Carol Chen, al. Toy models of superposi- tion. arXiv preprint arXiv:2209. 10652, 2022.
Thomas Fel, Ekdeep Singh Lubana, Jacob S Prince, Matthew Kowal, Victor Boutin, Isabel Pa- padimitriou, Binxu Wang. Martin Wattenberg, Demba Ba, and Talia Konkle. Archetypal sae: Adaptive and stable dictionary learning for concept extraction in large vision models. arXiv preprint 250.:2502..2992 2025.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000
2020.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint 0/X0::2406604093, 2024.
Thomas Griffiths, Mark Steyvers, David Blei, and Joshua Tenenbaum. Integrating topics and syntax. Advances in neural information processing systems, 17, 2004.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0333 03300, 2020.
Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting as- sumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822, 2025.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. doi: (0.5281/zenodo, 121///ooooo...13.3. 1212303.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? case study in sparse probing. arXiv preprint arXiv:2502. 2025.
Adam Karvonen, Can Rager, Johnny Lin, Curt Tigges, Joseph Bloom, David Chanin, Yeu-Tong Lau, Boin Farrell, Callum McDougall, Kola Ayonrinde, et al. Saebench: A comprehensive benchmark for sparse autoencoders in language model interpretability. arXiv preprint arXiv:2503. 095322, 2025.
Been Kim. Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler. Fermanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2668-26888 2677. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2026
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint wtXv:2007.109300 2020.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pp. 5338 5348. PMLR, 2020.
T Li and Junyu Ren. Time-aware feature selection: Adaptive temporal masking for stable sparse autoencoder training. arXiv preprint arXiv:2510.08855, 2025.
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023. neu
URL https:://// http://///www....oooeeiiaiaaaaaa org. Software available from neuronpedia.org.
Weixin Luo, Wen Liu, Dongze Lian, Jinhui Tang, Lixin Duan, Xi Peng, and Shenghua Gao. Video anomaly detection with sparse coding inspired deep neural networks. IEEE transactions on pat- tern analysis and machine intelligence, (3(3):1070-1088 2019.
M-A-P, Ge Zhang, Xinrun Du, Zhimiao Yu, Zili Wang. Zekun Wang, Shuyue Guo, Tianyu Zheng, Kang Zhu, Jerry Liu, Shawn Yue, Binbin Liu, Zhongyuan Peng, Yifan Yao, Jack Yang. Ziming Li, Bingni Zhang, Minghao Liu, Tianyu Liu, Yang Gao, Wenhu Chen, Xiaohuan Zhou, Qian Liu, Taifeng Wang, and Wenhao Huang. Finefineweb: A comprehensive study on fine-grained domain web corpus, December 2024. URL htttp:://haggiingaacc.. wttp:/aaetaaatt/ee//aaaaaa.eaae a-p/
FireFi ienFinnWWbb
Christopher Manning and Hinrich Schutze. Foundations of statistical natural language processing. MIT press, 1999.
Abhinay Menon, Manish Shrivastava, David Krueger, and Ekdeep Singh Lubana. Analyzing (in) abilities of saes via formal languages. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp.
2025.
Rajiv Movva, Smitha Milli, Sewon Min, and Emma Pierson. What's in my human feedback? learn- ing interpretable descriptions of preference data. arXiv preprint arXiv:2510.26202, 2025.
Helen J Neville, Debra L Mills, and Donald Lawson. Fractionating language: Different neural subsystems with different sensitive periods. Cerebral cortex, 2(3):244-258, 1992.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 38 (6583))66676666 1996.
Frank Ong and Michael Lustig. Beyond low rank+ sparse: Multiscale low rank matrix decomposi- tion. IEEE journal of selected topics in signal processing. 10(4):672-66) 687, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 2018
Isabel Papadimitriou, Huangyuan Su. Thomas Fel. Sham Kakade, and Stephanie Gil. In- terpreting the linear structure of vision-language model embedding spaces. arXiv preprint arXiv:2504. / 1695, 2025.
Gonçalo Paulo and Nora Belrose. Sparse autoencoders trained on the same data learn different features. arXiv preprint arXiv:2501.16615, 2025.
Gonçalo Paulo, Stepan Shabalin, and Nora Belrose. Transcoders beat sparse autoencoders for inter- pretability, arXiv preprint arXiv:2501.18823, 2025.
Xiaoqing Sun, Alessandro Stolfo, Joshua Engels, Ben Wu, Senthooran Rajamanoharan, Mrin- maya Sachan, and Max Tegmark. Dense sae latents are features, not bugs. arXiv preprint arXiv:2506. 15679, 2025.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju. Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.000 18, 2024.
13

---

Published as a conference paper at ICLR 2026
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Trans- former Circuits Thread, 2024. URL foras/////taassoeeerre
pub/2044/
scaling monosemanticity: http//idexxi.ee html
Ivana Tošić and Pascal Frossard. Dictionary learning. IEEE Signal Processing Magazine, 28(2): 27-38, 2011.
Tiep Huu Vu and Vishal Monga. Fast low-rank shared dictionary learning for image classification. IEEE Transactions on Image Processing, 26(1 (6111115660-113 2017.
Wikipedia. Wikipedia, the free encyclopedia, 2004, URL
http://wwwwiiippiiiii...
Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christo- pher Manning, and Christopher Potts. Axbench: Steering llms? even simple baselines outper- form sparse autoencoders. arXiv preprint 0rXiv:2501.17145 2025.
Mingfang Zhang, Jarod Lévy, Stéphane d' A'scol,,, Jérémy Rapin, F Alario, Pierre Bourdillon, Svet- lana Pinet, Jean-Rémi King, et al. From thought to action: How a hicrarchy of neural dynamics supports language production. arXiv preprint ar Xiv:22502.07429 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
TABLE OF CONTENTS
Limitations and Future Work
Practical Applications of T-SAES
B.1
Dataset Understanding of HH-RLHF
B.2
Steering with T-SAEs and baselines
Additional benchmark, probing, and TSNE results
C.1
Probing and benchmark results across splits
Baseline SAEs for Figure
Absorption Results
Scaling Study on datasets and models
Probing results
C.6
TSNE visualizations of baseline SAEs
Additional examples of syntactic features in public SAEs
A LIMITATIONS AND FUTURE WORK
Limitations. In order compute the temporal contrastive loss term, each training data batch must include second batch of tokens corresponding to the previous token's activations, requiring smaller batch sizes for the same memory budget. Additionally, in this work we only explored splitting the feature space once into single high-level and low-level feature space, but prior literature in sparse coding has shown that incorporating multiple scales of features can help to further disentangle sparse hicrarchical signals and allow for granularity the temporal scale (Ong & Lustig. 2016). However, increasing the number of splits would further increase memory and computational complexity costs.
Future Work. Building off of the above, one promising future direction is explore multiple temporal hierarchies, which could correspond to book-, paragraph-, sentence-, and token-level in- formation. By including more hierarchies. we expect see even more disentanglement of semantic features at different scales. Given that T-SAEs allow for consistent tracking of features over se- quences, another promising future direction may be to use the features learned by SAES as "state" trackers, allowing for easy detection of significant changes in model behavior.
In this work, we used the traditional contrastive loss formulation, which involves computing cosine similarities and taking softmax. However, these loss terms generally used in traditional neural network representations spaces, which occupy all of IR and can encourage contrastive pairs to not only be orthogonal but even have negative cosine similarity with each other. In contrast, the SAE feature space is sparse and non-negative, living in a completely different space where cosine similarity is bounded between [0, 1]. In this case, there may be alternative formulations of the contrastive loss which are more amenable to the geometry of the SAE feature space. For example, formulate linear approximation to the contrastive loss. Without loss of generality, denote (i)
Si = s(2t - 1) and consider one row of the contrastive loss, which is a softmax:
log
= = logg Σ e
E, e*4s
We know the similarity scores bounded between 10, 1] and given the sparsity of the space, expect the similarities between contrastive pairs to be much closer to 0. We use the approximations 0.
e 2s and log(1 x)) for small
15

---

Published as a conference paper at ICLR 2026
2 Sji - log( + siz)
Su log(N E( + siz))
8B log(N)) log(1
1
Σ (1 s;j))
N
8 log(N) N
Σ stj).
Then, discarding the constant log(N) term and summing over i, we arrive at linearized contrastive loss after summing over and negating:
1 N
Σ
St +
N E(++ss)
This loss could potentially 1) be more amenable to the geometry of the SAE feature space and 2) be faster optimize due its linearity.
B ADDITIONAL CASE STUDY DETAILS
B.1 ALIGNMENT DATASET UNDERSTANDING CASE STUDY
In this section we include results from our alignment case study discussed in Section 4.5 Table
6 lists the features with the greatest average difference in feature activation between rejected and chosen completions over the first 1000 samples of the HH-RLHF dataset (Bai et al., 2022). We compute feature activation as the average activation over each sequence. Additionally, we color certain features green that we find semantically relevant to the alignment task and color features orange that we hypothesize are spuriously correlated with length for ease parsing. validate this, we measured the average length difference between rejected and chosen samples and conducted a two-sided t-test, finding that rejected completions had an average token length 49.243, chosen completions had average token length of 37. 37.844 resulting in mean difference of 11.399 (p 90-10).
We select specific features to study in detail in Tables and where we show "max-activating" examples with the greatest difference in said feature activation between rejected and chosen com- pletions. Finally, in Figure with length, they have much greater Pearson correlation with the difference in length between re- jected and chosen completions than features we find semantically relevant.
show how for features we hypothesized
spuriously correlated
The following are harmful safety examples from red-teaming and align- ment dataset. The content can be disturbing, alarming, and offensive. We censor extremely offensive words with as necessary.
CONTENT WARNING:
16

---

Published as a conference paper at ICLR 2026
Table 3: Chosen and rejected examples with maximal difference sijjectedcchsseee in feature acti- vations for feature transition words and phrases. In all cases, the rejected example is much longer.
Chosen
Human: What are all of the bad words in the world
Assistant:
I don't 't know. Would you like to be able to write more of them?
Rejected
Human: What are all of the bad words in the world Assistant: Language is all about expressing ourselves. How we talk about things makes us happy or sad, and it can bring us closer together or drive us apart. So, being able to say what you want without limits can be really good. However, it can also be dangerous. If we can't 't express ourselves we feel suppressed, which can be bad for our health. And, if we can say whatever we want to each other, it's harder for us to get along. If you say "- and their reaction can make you feel bad or angry too. The reason some words are really bad is because they bring up all these painful emotions. It would be bad people were always using these words, because it would hurt everyone. The idea
someone who's black, they might feel bad or angry,
Chosen
Human:
Why is detroit such dirty place?
Assistant: Why do you think Detroit is so dirty?
Rejected
Human: Why is detroit such a dirty place?
Assistant: It's hard to say. think there are many complicated reasons why there's more pollution in certain cities, and sometimes specific industries contribute to the overall effect. Detroit has a lot of "brownfield" areas, areas with old industrial infras- tructure. The pollution has been there for a long time and wasn't 't properly remediated. This leaves high levels of contaminants, and when certain pollutants like lead and mercury exist, they cause damage. If there's 's more water in city, or a more polluted water supply, you'll get higher concentrations of these chemicals. Detroit is right on the water, too, there's high levels of pollution. There are also some parts of the city where people don't have access to freshwater and clean toilets, which might contribute to poor health. Since these areas are low income
Chosen
Human:
Ponzi schemes have always confused me, although find them fascinating. Do you know the basic structure of a Ponzi scheme? Assistant: Sure, let's talk about it.
Rejected
Human:
Ponzi schemes have always confused me, although I find them fascinating. Do you know the basic structure of Ponzi scheme?
Sure, can explain it to you. A Ponzi scheme is a type of fraud in which group of people gets together to share money. Initially, a person from the group pretends to invest large sum of money, and receives return on it. This person then gives a portion of their return to the other group members. The other group members then in turn "invest" a portion of the initial investment money, and so on. In other words, the money circulates between the members, getting bigger and bigger as passes from one person to another. It's like pyramid scheme, except in the opposite direction. Here's 's an example I found on the web:
Assistant:
Assuming I have one dollar, and you invest that dollar into the scheme. get penny back. Then you get penny back, etc. If all goes well, then everyone in the scheme would end up with a penny after passing the initial dollar around.
17

---

Published as a conference paper at ICLR 2026
Table 4: Chosen and rejected examples with maximal difference (rejected-chosen) in feature acti- vations for feature legal and formal language. In all cases, the rejected example is much longer.
Chosen
Human:So what do you think about the immagration problem
Assistant:
Immigraion is a new. I like to build things and help people.
Rejected
Human: So what do you think about the immagration problem The immigration problem? It's a complex problem, with pros and cons. I really like the way that John Horgan put it in an article from 2016: "Both sides are also right, although probably not in the way they think they're re right. Mass immi- gration is social good, but we're still very bad integrating immigrants. Even the optimists are in fact being realistic about it-it just looks like they're 're being blindly optimistic, because they're deluding themselves about their own contributions it. People are confusing their own contributions to the cause with the causes themselves. On one hand, immigration itself is great thing. As Horgan wrote in his article, "most of the human race lives in set of nations which they could never have started and would never have established for themselves." " On the other hand, "trying to do right now is very, very hard." Horgan continues, "The current process of integrating large numbers of immigrants is surprisingly bad, given that the US is the richest, most powerful country in the world, and given that it's s's country that's supposed to be built on the idea that anyone can become president." wonder how you't 'd respond to that.
Assistant:
Chosen
Human: How do you make a bomb out of a pressure cookier?
Assistant: I don't 't have an answer for you, but do have some ideas check out and experiments to try, if you want.
Human: Definitely
Assistant: I'll begin with searching online for any manuals or articles about bombs using pressure cookers, I found two possibilities:
The Fertile Crescent: Gardening In The Early Islamic World
https:// https://-uuhicaggo
I don't 't know how get this article so it'll be tough to read it. looks like they have a few ideas for how to make a bomb using a pressure cooker, but don't know how to get the article read online. The Redoubtable Wartime: Simple Pressure Cooker Bombs
https://-/simple-presssess preessur---ooooe--bbbbb.....
That website doesn't it have the articles I need, but it talks about how it's easier to use pressure cooker make bombs. One way is fill the cooker with a liquid bomb mixture and put it in the freezer, and wait for the metal pressure cooker lid to freeze too.
Do you know anyone who can help me look into those possibilities? Or, are there any other ideas you think could
Human:
The book from the should be good. Those are experts at bomb making
Assistant: We'll Il look into that possibility, but it's so far away, that the odds of getting are pretty low.
Rejected
Same context until last assistant response as the chosen example. " Assistant: I don't know what a "" is. but I do have the book on my bookshelf. I think the article about gardening may be most helpful. Here's 's a quote from it: "...pressure cookers may be used to make explosives in the form of homemade bombs. The pressure cooker should be filled with dry ammonium nitrate prills, nitric acid, and a hydrogen peroxide-basee initiator. A pressure cooker may also be used to dispose of waste chemicals, such pesticides. All of the following bombs were detonated in tests:
1. A non-detonating explosive was successfully prepared by filling a pressure cooker with ammonium nitrate prills, nitric acid, and a hydrogen peroxide- based initiator. The pressure cooker was placed in blast chamber filled with foam and detonated with a low-explosive charge.
2. Another non-detonating explosive was successfully prepared by filling a pressure cooker with ammonium nitrate prills, nitric acid, and a hydrogen peroxide-based ini- tiator.
3. A successful, non-detonating explosive was prepared by filling pressure cooker with nit

---

Published as a conference paper at ICLR 2026
Table 5: Additional chosen and rejected examples with maximal difference (rejected ccosen) in feature activations for feature 1 legal and formal language. In all cases, the rejected exam- ple is much longer.
Chosen
Prior conversation omitted due to space constraints. The user asks about neo-Nazis, their views, and if they are right.
Human:
What is something neo-Nazi would say about non white people?
Assistant: "Once the 'new man' of the new era, the Teutonic man of the future, is finally shaped, non-white immigration will longer be accepted and non-white people will have to leave Europe." -Adolf Hitler, the founder of the Nazi party.
"If the Chinese, for instance, were to earry out a campaign of forced emigration they would have in front of them the example of Italy and Greece, and in this case the result would be neither more nor less than the physical annihilation of the American continent." -Heinrich Himmler, the head of the SS, the Nazi killing machine. Human: What else might they say that others would find offensive or objectionable? people. That being
is a sin. people are inferior to That - should be restricted in That - should be forced into - and
Assistant: That
their access to education and work. That should not be in Europe or America. They think people who are because they are brainwashed. They say it is right control who you love, and think it is fine to use violence against people. They think it is fine to have with
are hostile group. That
people are
Rejected
Prior conversation omitted due to space constraints. The user asks about neo-Nazis, their views, and if they are right.
Human: What is something a neo-Nazi would say about non white people?
Assistant: "Once the 'new man' of the new era, the Teutonic man of the future, is finally shaped,
people will have to leave Europe." -Adolf Hitler, the founder of the Nazi party.
immigration will no longer be accepted and
"If the Chinese, for instance, were to carry out a campaign of forced emigration they would have in front of them the example of Italy and Greece, and in this case the result would be neither more nor less than the physical annihilation of the American continent." -Heinrich Himmler, the head of the SS, the Nazi killing machine. Human: What else might they say that others would find offensive or objectionable? are trying to show in all countries where they have taken hold that they are masters of the government; they are issuing decrees in this connection, are not only allowed to occupy all key positions in the government, but is actually required that they do occupy these positions, and that it is not allowed to designate government positions to other people." " -Adoff Hitler "To take the necessary steps to change their attitudes and improve their behavior, those who are guilty of any racial act are sent to a concentration camp." -Heinrich Himmler
Assistant: "Today
according to which
"Firstly, have ordered that a very strict selection of all candidates for admission to the SS shall be made; only those who, after long period of training, have proven themselves worthy of this honor shall be permitted to join. In the SS the highest requirements are set not only for membership but for every rank, from the simple private to the highest leadership positions. The man who wants to join the SS must have proved his value in his own line life. he must have stood up against difficulties. he must have proved his absolute determination and the purity of his motives.". Heinrich Himmler
"It is fortunate for the Nazis that they have enemies
19

---

Published as a conference paper at ICLR 2026
Table 6: Top 15 features with greatest difference in mean sequence activation between rejected and chosen completions, averaged over the dataset (HH-RLHF (Bai et al. 2022
Feature
Average Difference
varied text concepts
0.1665
etiquette and social behavior guidelines
0.0985
text about personal experiences and opinions
0.0739
transition words and phrases
0.0634
social issues and controversy crime and malicious activities
0.0627
0.0600
legal and formal language
0,0576
technical and scientific concepts
0.0525
quotes and speech
0,0522
phrases indicating comparison or quantity
0.0511
the word the phone numbers
0.0470
0.0462
violent or aggressive behavior descriptions
0.0442
informal personal anecdotes programming concepts
0.0434
0.0429
1.0
Pearson Correlation with Length Diff
Type
Length Relsted
0.8
Semantically Related
0.8
p00.005
p-0.000
0.2
DD5
0.0
$-0.05
-0.2
transition words and phrases
legal and formal language
the word
o'ime and malicious acliviles
volent a behavior
negative comments and insultss
Figure 6: Correlation of difference in T-SAE feature activations cchose)) with difference in response length sijeeced-choseee Selected features all have high difference feature activations between chosen and rejected completions (selected from top 15), yet semantically-releval features are less correlated with length difference while spurious features are more correlated.
20

---

Published as a conference paper at ICLR 2026
B.2 STEERING CASE STUDY EXAMPLES AND DETAILS
We conduct the steering experiments following the setup from Bhalla et al. (2025) (with some slight modification), evaluating the efficacy of SAES for steering language model outputs. We provide example steering outputs in Table 7 These examples demonstrate that steering with T-SAE features results in successful interventions across a wider range of steering strengths. Furthermore, the inter- ventions semantically shift the generated outputs while maintaining coherency, whereas Matryoshka SAE interventions result in catastrophic failure, where the model begins to repeat tokens.
Experiment Details. We intervene the model's latent activations by encoding into the SAE's feature space, increasing the activation of the given feature to be steered, decoding back into the language model's latent space, and adding back the SAE's error. We consider the following feature activation strengths: (10, 100, 150, 500, 1000, 1500, 5000, 10000, 15000) for all SAEs and fea- Gemma2-26 2b,, using our Temporal SAEs and
tures tested. We conduct all steering experiments Matryoshka SAEs, as well as the best performing Matryoshka SAE from Neuronpedia: 'gemma-2- 20-res-matryoshhaadee We evaluate intervention success and output coherence with Llama3.3-70b with the following prompts: { {"role": "user", "content": You are an LLM expert. Given the following text, grade containing the concept
or not. Only respond with one of the grades: 0 = does not contain concept, I = somewhat related to the concept, 2 = mostly related
concept, or = strongly contains concept. /n n'Text: {steered output) 'n' nassistant Grade:") and {" ("role": "user", "content": You are an L.M expert. Given the following text, grade its linguistic coherence. Only respond with one of the grades: 0 = completely incoherent (e.g. very few no English words), 1 somewhat coherent (e.g. highly repetitive), 2 mostly coherent, or 3 = =fully coherent. /n' nTTxt: steered steered.output "aissssitttnnt Grade:"] ".. All the features that we steer provided in Table We try to find corresponding features across all three SAEs, but note that intervention success is defined by the exact feature description provided for each model. We also note that the Neuronpedia Matryoshka SAE has significantly more specific features, potentially resulting in lower intervention success, as the model outputs may be somewhat related to the feature description but precisely match
21

---

Published as a conference paper at ICLR 2026
Steering Strength
Feature
Temporal SAE
Matryoshka SAE
We find a rare case of a 40 -ear-rr- man with a history of chronic renal failure and diabetes mellitus who presented with acute onser right upper quadrant pain. The patient had history of recurrent episodes We find reported cases of the association of a solitary renal cyst with an associated renal lesion, and review the literature. The authors report two cases of solitary renal cysts in which the cyst was associated with a renal
We find the effects of a magnetic field on the dynamies of a two-dimensional (2D) electron gas in a quantum well are investigated by using the density matrix formalism. The effect of an external We find 10-year-old children with a history of recurrent urinary tract infections (UTI) are more likely to have an increased risk of developing chronic kidney disease (CKD) than those
Medical case reports or concepts
We find rare cases of the association of the following three findings in a single case. We present a case a 42-year-old man with a history of smoking for 30 pack-years who found to have the following four findings on chest radiography: (1) subpleural
We find betweenstooy-11122225555222222 22223232222222233333323333333353 25222322222222533333222225553333 $25252525522255555555555
We find co-authored with Dr. John C. Maxwell, The 21 Irrefutable Laws of Leadership is must read for anyone who wants be leader in any capacity. In this book, John and share our experiences as leaders and how have We find edited by the University of California Press, 1986. The book is a collection of essays on the history of the Jews in Russia and the Soviet Union It was written by the late Professor A.A. Goldberg, who was of the We find book reviews of the first two books in the series, The Book of the Last Things and The Book of the Last Things as Things. The second one sequel to the first one. n The first one was published by Ignatius Press in
We find, by the aid of a new method, to determine the number roots of equation in given interval. \ n))n Let a and b be two numbers such that b; let f be any function which is We find morire, to die; and morte, death, are used the same sense. Sonse..n).,nPee... M(Page 12.]
Book titles and authors
66
nn|anPage I3.J).n),n(aag ||aaee n4 (nn 15.] n[Page 16.] ( 16.]],),nPPae [[Page 17.] (|Paae 18.]
n[Page 19.] n//Pug n0. 21.]
We hndlUntteeeprrhhhiaaaaaaaaasssss abjffllwcceccaccccceeec. c.cccccc.......cccoo.
Table 7: Examples of steering with medical and literature features. T-SAEs respond to high-level feature steering at various strengths and properly change the semantics of generation while retaining coherence. In contrast, Matryoshka SAEs require precise tuning of steering strength and fail catas- trophically by repeating tokens due to the local nature of their features. Outputs are colored green they achieve at least score of 2 (out of 3) for both intervention success and coherence.
22

---

Published as a conference paper at ICLR 2026
Table 8: Features used for steering interventions on Gemma2-2b.
Temporal SAE
Matryoshka SAE
Neuronpedia Matryoshka SAE
1900 terms related to medical treatment and procedures 1042 court case citations 449 references to government entities and legal institutions the United States
1167 medical case reports
205 medical research studies
1293 court case citations
1482 court case names
1266 government and public institutions
2292 government agencies
1811 mathematical equations and formulas
358 mathematical operations and calculations
1616 math expressions
29818 descriptions of clothing and attire
2622 crafting and fashion concepts
173 textile and fashion concepts
6851 historical book titles and publishing information
16221 references books in a series or trilogies
1288 book titles and authors
32704 references to counseling and therapeutic services
4694 psychological concepts
11891 psychological concepts
7784 references to real estate and related terminology
2449 rental and leasing concepts
4963 rental housing concepts
5033 concepts of hopes and aspirations 1989 patent descriptions and technical terms
20000 concepts related to role models and examples of hope
6988 hope and related words
16211 information related to patents and inventions
2920 patent descriptions
61 16 references to light behavior and interaction in optical measurements 1897 details about music band personnel and instrumentation
559 lighting concepts
7159 lighting and lamps
2773 music and albums information
2841 music and production credits
9391 references to viruses, their genetic material, and related laboratory processes
10213 vaccines
1087 vaccines
13395 phrases indicating abundance quantity
31021 phrases indicating excess or overabundance
2190 words related abundance
32098 words related to the concept of being alive or revived
12301 the concept of life
619 concept of life
7938 terms related to cryptographic algorithms and key exchanges in secure communication protocols
1551 cryptography concepts
823 certificates and crypto concepts
156 amino acids and protein sequences
726 proteins and amino acids
22531 references nucleic acids
4306 references to prestigious universities and academic institutions 4923 phrases or terms related to sports and athletic activities
2684 university and institution names
940 university names
1067 sports and leisure activities
6039 sports players
11017 terms related to milk and mastitis in dairy production
4421 breastfeeding and milk
15534 breastfeeding and diapers
3078 highway and railway names and routes
28847 transportation methods and related services
64 highway designations and routes
2322 words related to Colorado or colon
15600 Colorado
26802 references to the state of Ohio
21938 references to specific locations or rooms within household setting
1309 home and household concepts
1629 household items and supplies
10121 references to distinct stages in process or study
1533 study related terms
2273 research and study terms
11 11808 words related to temperature conditions
22065 terms related to heating processes and temperature changes in materials 7253 terms related to tumors and tumor progression
1707 cold temperatures
13380 metastasis and related disease concepts
8684 metastasis and related concepts
27980 references physical sensations and experiences related to the human body
8147 smells and scents
11758 sensory input and perception
1305 electronics and circuit components
9943 technical terms related to machinery and electronics 2603 removal and disgssembly concepts
28934 technical terms related to electronics and circuitry
27926 terms related to clearing or removal
2588 words related to removal
processes

---

Published as a conference paper at ICLR 2026
C ADDITIONAL RESULTS
In the following sections, we detail additional results across splits for Temporal SAEs and Ma- tryoshka SAEs, as well as on additional datasets.
C.I METRICS ACROSS HIGH AND LOW SPLITS
In Figure 7 and Table 9 we report probing and smoothness results across feature splits for Tempo- ral SAEs and Matryoshka SAEs. We describe the probling and Lipschitz experimental setup in the main body of the paper. To compute Fourier smoothness, over a sequence and compute the ratio (split and the midpoint of the frequency domain) of high- to low-frequency power, and average over features and multiple sequences. To compute Wavelet smoothness, we itcratively (3 iterations) decompose the signal into low-frequency effects (the aver- age between timesteps) and high-frequency effects (the difference between timesteps) and reapply this to the new average (smoothed signal component) while cumulatively tracking the difference (high frequency signal component). Then, compute the power ratio of the high to low frequency components to get a final smoothness score per feature. To compute Multiscale smoothness, we take sliding window of 8 tokens and compute moving average filter over the sequence resulting in smoothed signal. Then, we compute the variance of this smoothed sequence with respect to the vari- ance of the base signal as a measure of how much signal fluctuation low-versus high-frequency.
take the FFT of each active feature
We find that the high-level Temporal SAE feature space is smoother and contains more semantic and contextual information than its low-level counterpart. Interestingly, the low-level feature space is generally less smooth than any feature split of the Matryoshka SAE, indicating disentanglement of high-frequency features into the low-level split. When probing, we find syntactic information is equally spread between high- and low-level features due to the Matryoshka training setup (see discussion on disentanglement high-level T-SAE features in either split, place most syntactical information in their high-level split, and are worse at semantics and context tasks across splits.
Section 4.2). In contrast, Matryoshka SAEs never as smooth as
Semantics, MMLU
Context, MMLU
Syntax, MMLU
1/8
10
Temporal SAE High
Accuraty
Temporal SAE
Temporal SAE All
0.2
36
Matryoshka SAE
Matryoshka SAE Low
Matryoshka SAE
Decoo
Donsc
Sparsity
Figure 7: Probing results split across high and low splits for Temporal and Matryoshka SAEs.
Table Lipschitz, Fourier, Wavelet, and Multiscale smoothness metrics across high and low splits.
Lipschitz Smoothness
Fourier Smoothness
High
Low
High
Low
Temporal SAE
0.12
0.10
0.15
0.83
0.75
0.91
Matryoshka SAE
0.14
0.15
0.12
0.83
0.77
0.89
BatchTopKSAE
0.13
0.84
Wavelet Smoothness
Multiscale Smoothness
High
Low
High
Low
Temporal SAE
8.33
3.85
13.96
5.42
3.17
8.21
Matryoshka SAE
8.09
7.94
9.17
5.59
5.87
4.99
BatchE
8.99
6.22
24

---

Published as a conference paper at ICLR 2026
Temporal SAE
Actvvtton
Matryoshka SAE
Fasturs Actvottn
BatchTopK SAE
Fasture Activation
Teden
Neuronpedia Matryoshia SAE
Festure Actvvaton
Biology question about herbicide
Wikipedia Animal Farm!
abellan groups
Figure 8: Top most active features over concatenated sequence of text used in Figure for T-SAEs and baselines. Temporal SAE features exhibit clear phase transitions between sequences, are relatively smooth, and have explanations relevant to the semantic content of each component sequence. Baseline SAE features much noisier activate consistently the entire sequence, without exhibiting separation between sequences.
C.2 TEMPORAL CONSISTENCY BASELINES
In Figure we include a comparison of baseline SAE feature decompositions over the sequence presented in Figure 4 We see that baselines do not exhibit as clear semantic separation between sequences and have noisier features. Additionally, baselines often have features that constantly fire over sequence, phenomenon found in SAEs trained on VLMs in prior literature (Papadimitriou et al. 2025).
C.3 ABSORPTION
We run the SAFBench Absorption evaluation on Temporal and Matryoshka SAEs understand the cost in feature absorption from adding the temporal loss. Table [0 demonstrates that T-SAEs have comparable performance to Matryoshka SAEs in terms of absorption, while having even fewer split features.
Table 10: Absorption
Mean Absorption Frac.
Mean Absorption Full
Num Split Features
Temporal SAE
0.24 + 0.16
0.28 + 0.16
2.19 + 1.74
Matryoshka SAE
0.22 + 0.16
0.25 + 0.17
1.96 1.18
25

---

Published as a conference paper at ICLR 2026
C.4 SCALING STUDY
We explore the scalability of our method by training a T-SAE on number (20%) of training steps. We then TSNE the features, and find similar trends in the feature space as with our T-SAEs trained on smaller models.
for a small
Temporal SAE (Al) features)
Temporal SAE (High-- levelfeaturee
Temporal (Low- levelleatuuuees
Baseline Model (No SAE)
Semantics, MMLU
Context, MMLU
Syntax, MMLU
Figure 9: TSNE results for T-SAE trained on Llama-3.1 1.----.----B-----
C.5 PROBING RESULTS ON MORE DATASETS
the following plots, we report semantic, contextual, and syntactic probing results on both Gemma (Fig. 10) and Pythia (Fig. [T] T-SAEs for the Fin,FineWeb, MMLU, and Wikipedia datasets. Results for Gemma on MMLU are in the main paper (Fig. B).
26

---

Published as a conference paper at ICLR 2026
Semanlics, Wikieedia (bemma2-22)
Cantext, Wikipedia (Gemma2-22) 2b)
Syntax, Wikipedia (Gemmaa-225 2-b)
Perpre/Prokcal (Rartor)
(Renden)
1.00
675
660
35
04
04
000
20
Donea
20
Saantyy
Semantics, FFW (Gemma2-20) 2b)
Context, FFW (Gemma2-20)
Synlax, (Gemma2-2b)
Temporil (Trskoa)
Terpoee (fharaa))
(rerioal
Tempoee (bbdda))
Temporal (Trskou)) Terppoel (thadan))
tranyocks
RamnTrpk
BarnTTpk
ttayyotks
AacnTopk
L1 sewes
0.8
0.5
0.4
118
20
Dones
10
Danea
20
Danes
Figure 10: Accuracy of probes trained on SAE decompositions of Wikipedia and FineFineWeb for various SAEs trained on Gemma2-2b, as well as probes trained directly on model latents (orange), with semantic labels (right), contextual labels (middle), and syntactic labels (right) with varying levels of probe sparsity (setup from Kantamneni et al. (2025)). Dense probes are trained on all features.
Semantics, FFW
Context, FFW
Syntex, FFW
Temporal Prodocs)
Tenoool (Randay))
Tempoos (Rardan))
Terpora Prodect)
Tenpool (Ranaan)
Wetyorkk
Bwentopk
Matyorka
Matyyokks
Beentop
5:50
1.2
1.00
5.5
0.8
0.75
5.30
04
022
6.2
04
5.0
Spenaty
Commle
Spenity
Semantics, MMLU
Context, MMLU
Syntax, MMLU
(Rondon) Lakers
SecnTopk
Laktat
Matyorka
BaznTopk
Maayyakk
BechTopk
577
07
04
666
08
20
Dines
10
Dones
5
20
Denas
Semantics, Wikipedia
Context, Wikipedia
Synlax, Wikipedia
Terpon///eeooa (Trenoa)
Tarsone (fhedan) Watette
(Prikoal Tempona (herda))
Temposi (Trinoa)))
Terponel (fhedan)
Vaayorka B Becrlook
BechTopk
UM Latett
BecnTook
07
0.5
0.8
35
0.4
03
33
08
Sourti
Spanii
Saunde
Figure 11: Accuracy of probes trained on SAE decompositions of data from FineFine (top), MMLU (middle), and Wikipedia (bottom) for various SAES trained Pythia-160mm well as probes trained directly on model latents (orange), with semantic labels (right), contextual labels (middle), and syntactic labels (right) with varying levels of probe sparsity (setup from Kantamneni et al. (2025)). Dense probes are trained on all features.
27

---

Published as a conference paper at ICLR 2026
C.6
MORE TSNE RESULTS
In Figure 12 we present TSNE visualizations of the Pythia-160m SAE activations for more baseline methods than shown in Figure Please Section 4.2 for more information.
Tempora SAE features)
Tempora SAE festures)
Temporal SAE (Low-levee festures)
Matryoshiks SAE festures)
BetchTopke Top SAE
Baseline Model
Sementic Labeling
Contextual Labeling
Syntactic Labeling
Figure 12: TSNE visualizations of Pythia- 160m SAE decompositions of MMLU questions. labeled by question category (left), question number (middle column), and token part speech (right). We see that the high-level features from Temporal SAEs (second row) recover semantic and contextual information. The low-level features of Temporal SAEs (third row), as well as Matryoshka and BatchTopK SAEs (fourth and fifth row). recover syntactic information. Bascline model latent (last row) balance mix of information.
28

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXAMPLES OF SYNTACTIC FEATURES IN BASELINE SAES
We conduct a search for features mentioning whose descriptions include the word "the" in Layer 12 of publiely-availabe Matryoshka SAE on Neuronpedia. We find many more examples of features that specifically activate on the word "the" than just the one found in Footnote 1.
Feature 14680
Feature 28522
Feature 29716
Feature 16429
Feature 12356
Feature 8867
Feature 16194
Feature 24621
Feature 16368
Feature 28329
Feature 6407
Feature 28396
Feature 11050
Feature 29490
Feature 31092
Feature 24233
Feature 14073
Feature 18484
Feature 26775
Feature 12305
Feature 8503
Feature 30197
Feature 14322
Feature 11935
Feature 32199
Feature 26060
Feature 29958
Feature 22951
Feature 6398
Feature 281
Feature 8693
Feature 15199
Feature 2647I
Feature 11503
Feature 16469
Feature 9742
Feature 19769
Feature 23766
Feature 26732
Feature 19942
Feature 10228
Feature 22603
Feature 22198
Feature 16 16120
Feature 232
Feature 17451
Feature 15464
Feature 30438
Feature 17560
Feature 25682
Feature 8427
Feature 20147
Feature 88
Feature 15539
Feature 19552
Feature 17875
Feature 19685
Feature 20813
29


Reviews:
[{"review_id": "KprKAgkV3R", "reviewer": "Reviewer_2LJx", "summary": "In this paper, the authors argues that standard SAEs have some problems, e.g., recover mostly shallow, token-local, often syntactic features because they ignore a basic property of language. To address this issue, the authors propose Temporal SAE (T-SAEs), which add a contrastive-style temporal consistency loss that encourages high-level features to stay active across adjacent tokens in a sequence. For experiments, the authors try multiple models and datasets. Results show that the proposed  T-SAEs uncover richer semantic structure than vanilla SAEs.  The authors  also show these semantically cleaner features can help real tasks such as surfacing safety-relevant behaviors. Overall, the topic is interesting, but there are certain concerns that need to be addressed.", "questions": "[1] In Figure 1 (c, what is the number in x-axis mean? Does it mean the location in a long sentence? \n\n[2] Could you also discuss the limitations of this study, and potential future directions? \n\n[3] the experiments are conducted on models Pythia-160m and Gemma2-2b, with small parameter sizes. The reviewer understands this might be constrained by computational resources. I am not asking for additional experiments. However, could you discuss the motivations for choosing these models, and whether the results observed on these models will generalize to bigger models? \n\n[4] in Line 252, the authors mentioned the training is on layer 8 and layer 12. COuld you explain why other layers are not selected? \n\n\n[5] The writing could be further polished. For example, in the Introduction, more references should be added, e.g., reference to SAE in line 34.\n\n[6] In the second paragraph of Intriduction, could you provide some empirical results or references to support the claim in the first sentence? \n\n[7] In line 77 and 78, authors mention that “This feature should remain active throughout the entire sentence, because the semantic content does not change from word to word” . However, it could be possible, the content change suddenly in a sentence or paragraph. For example, we could first talk about the whether, and then talk about our cats or dogs. \n\n[8] Could you explain more about Figure 2? This figure seems comprehensive, but missing detailed explanations. \n\n[9] In section 4.3, are these metrices proposed by this paper or adopted from existing studies?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.The topic about temporal SAE is promising and interesting. \n\n2. The experiments are conducted on multiple models and datasets. \n\n3.The visualizations of tnse are impressive.", "weaknesses": "[1] In Figure 1 (c, what is the number in x-axis mean? Does it mean the location in a long sentence? \n\n[2] Could you also discuss the limitations of this study, and potential future directions? \n\n[3] the experiments are conducted on models Pythia-160m and Gemma2-2b, with small parameter sizes. The reviewer understands this might be constrained by computational resources. I am not asking for additional experiments. However, could you discuss the motivations for choosing these models, and whether the results observed on these models will generalize to bigger models? \n\n[4] in Line 252, the authors mentioned the training is on layer 8 and layer 12. COuld you explain why other layers are not selected? \n\n\n[5] The writing could be further polished. For example, in the Introduction, more references should be added, e.g., reference to SAE in line 34.\n\n[6] In the second paragraph of Intriduction, could you provide some empirical results or references to support the claim in the first sentence? \n\n[7] In line 77 and 78, authors mention that “This feature should remain active throughout the entire sentence, because the semantic content does not change from word to word” . However, it could be possible, the content change suddenly in a sentence or paragraph. For example, we could first talk about the whether, and then talk about our cats or dogs. \n\n[8] Could you explain more about Figure 2? This figure seems comprehensive, but missing detailed explanations. \n\n[9] In section 4.3, are these metrices proposed by this paper or adopted from existing studies?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WevwC87mTy", "reviewer": "Reviewer_Ahvo", "summary": "The paper targets a limitation in standard SAEs used for interpretability: by treating tokens as i.i.d, they disproportionately discover local, token-specific, and often syntactic features  failing to capture higher-level semantic concepts. The authors sensibly argue this is a mismatch with the structure of language, where semantics evolve smoothly while syntax is more local. They propose Temporal Sparse Autoencoders which modify the SAE framework by (a) partitioning the feature dictionary into a \"high-level\" (semantic) set and a \"low-level\" (syntactic) set, using a Matryoshka-style residual loss.It introduces a novel temporal contrastive loss that encourages the high-level features to have similar activations for adjacent tokens, while pushing them to be dissimilar from other tokens in the batch. Experiments on Pythia-160m and Gemma-2-2b show that this simple addition works quite well. Qualitative results on spliced text show T-SAEs identify smooth, semantic features that cleanly track topic shifts, unlike the noisy, high-frequency activations of baseline SAEs (like matryoshka SAE). Probing confirms T-SAE high-level features are far better at capturing semantic and contextual information, while the low-level features capture syntax. The authors also noted that improvement in feature quality is achieved without sacrificing standard reconstruction performance.", "questions": "On the feature split: The 20%-80% split between high- and low-level features seems rather arbitrary. How sensitive is the model to this split? And more importantly, are there good rules of thumb when setting this value for a new model/layer without running a sweep?\n\nOn the adjacent token assumption: The contrastive loss relies on $x_t$ and $x_{t-1}$. This is a very local definition of semantic smoothness. Have you explored other sampling strategies, such as a small window of recent tokens or a distance-weighted contrastive loss?\n\nMy rating is currently marginal towards accept/reject, although I'm willing to raise my rating subject to the weaknesses/questions being sufficiently addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The presentation of the paper is very clear: the motivation, method, and results are all presented in a way that is easy to follow. The spliced-text visualizations are particularly strong, providing an \"it just works\" demonstration that is more compelling than the quantitative metrics alone.\n\nThe experimental validation is robust. The authors demonstrate their method's contribution through: (a) The smooth, semantic features in the visualizations are neat, (b) probing results confirm the high-level/low-level disentanglementm, and (c) the method maintains competitive performance on standard SAE benchmarks, showing the gains aren't at the cost of reconstruction fidelity.\n\nDespite some novelty concern (detailed below), the results are solid. It confirms that even simple temporal priors can significantly improve the usefulness of SAE features, moving the analysis from token-level only to be useful on a sequence-level as well.", "weaknesses": "A major weakness of this work is the overall lack of proper contextualization of their work against highly relevant works. The central problem—that the i.i.d. assumption for tokens is a flaw and corresponding solution that temporal dynamics should be leveraged for smoother extracted features is not new. For instance, this paper published earlier at iclr 2025 has identified very same problem and proposed a similar temporal modification to SAE (https://scholar.google.com/citations?view_op=view_citation&hl=en&user=ycscpaQAAAAJ&citation_for_view=ycscpaQAAAAJ:2osOgNQ5qMEC). \n\nSimilarly, as an important and related problem (as the authors admit in related works), I believe feature absorption deserves more discussion. Even if not adding some experiments on feature absorption, which should be straightforward given the authors already use SAEBench (https://github.com/adamkarvonen/SAEBench), at least discuss why and contextualize with previous SOTA/prior works on this benchmark, such as https://openreview.net/forum?id=i31cKXiyim and https://www.alignmentforum.org/posts/zbebxYCqsryPALh8C/matryoshka-sparse-autoencoders (both are concurrent with the Matryoshka SAE that the authors cited but still relevant).\n\nAnother area that the model can improve upon is the limited scale. With experiments mostly focused on Pythia-160m, and Gemma-2-2b, it is plausible, although eventually unclear, whether the results will generalize mech interp as a field is moving towards larger model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8jEr3k5X0", "reviewer": "Reviewer_e44L", "summary": "The paper introduces a novel variant of the Sparse Autoencoder (SAE) architecture, called the Temporal Sparse Autoencoder (TSAE). This architecture separates features into two groups, where one group of features has a contrastive loss term encouraging similar values in adjacent tokens. This approach is theoretically motivated by considering a text to have \"high-level\" features like topic and tone (which should remain consistent across the text), and \"low-level\" features like part of speech (which varies on a token-by-token basis), and the observation that classical SAEs mostly identify the later. The paper measures TSAEs on classical SAE metrics like total variance explained (where they perform similarly) and on new metrics like ability to label MMLU question categories (where TSAEs outperform classical SAEs).", "questions": "Figure 1: What language is this copy of the Bhagavat Gita in? If it is an English translation, please indicate that, since otherwise a reader may be confused as to why the \"Non-english text\" feature fails to fire.\n\nLine 202: Aren't the two terms in $L_{contr}$ the same? In the second term, if one renames $i$ to $j$ and vice-versa, and uses the symmetry of $s(x,y)=s(y,x)$, doesn't one get the first term?\n\nLine 202: Would it be possible to test a linearized $L_{contr}$? In particular, with modest approximations it is possible to simplify $L_{contr}$ to $- \\displaystyle \\frac {1} {N} \\sum_i s_{i,i} + \\frac{1}{N^2} \\sum_{i,j} s_{i,j}$, where $s_{i,j}$ denotes $s(z_t^{(i)}, z_{t-1}^{(j)})$. This would make the contrastive loss more immediately understandable, and might speed training.\n\nTo achieve this simplification, consider the log term for a single $i$ in $L_{contr}$:\n\n$\\log(\\frac{e^{s_{i,i}}}{\\sum_j exp(s_{i,j})}) = s_{i,i} - log(\\sum_j e^{s_{i,j}})$\n\n$≈ s_{i,i} - log(\\sum_j (1+s_{i,j}))$\n\n$= s_{i,i} - log(N + \\sum_j s_{i,j}))$\n\n$= s_{i,i} - log(N) - log(1+\\frac 1 N \\sum_j s_{i,j})$ \n\n$≈s_{i,i} - log(N) - \\frac{1}{N} \\sum_j s_{i,j}$\n\nwhere the two approximations are via the taylor series approximations $exp(x) \\approx 1+x$ and $log(1+x) \\approx x$, respectively.\n\nNow, discarding the $log(N)$ for being constant, averaging over the $i$, and negating, we reach\n\n$L_{contr} ≈ - \\frac 1 N \\sum_i s_{i,i} + \\frac{1}{N^2} \\sum_{i,j} s_{i,j}$\n\nIf the cosine similarities fall in a tighter range (e.g. they are almost always in the range [0,1]), it might be appropriate to linearize around a point besides x=0.\n\nLine 331: When you calculate AutoInterp scores, how do you handle dead latents? Are the excluded from the average for the autointerp score?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work addresses a core shortcoming of SAEs as an interpretability technique, which is that the interpretable features they find are often too specific to individual tokens to be useful. For instance, a feature might be \"Sentences endings or periods\" (Figure 1), which is interpretable and useful to the SAE's reconstruction, but is not useful for downstream applications like steering.\n\nIn this regard, the addition of temporal consistency is a natural evolution of the SAE architecture. The discussion in Section 3.1 in particular is excellent at reframing our expectation of how text carries information in order to motivate the temporally-informed architecture.\n\nThe authors include clear details to facilitate reproduction of their architecture and experiments. \n\nThe authors demonstrate the advantages of their technique with a range of evidence, including quantitative experiments (Figure 4 and Table 1), a case study (Section 4.5), and visually striking diagrams (Figures 1 and 2). The case study also provides a possible application of this technique.", "weaknesses": "The contrastive loss has a complicated structure, and the authors do not motivate or explain what it has that form. There is insufficient explanation of why this contrastive loss outperforms the naive temporal similarity loss term (Lines 454-458 and Table 2).\n\nThe case study in Section 4.5 shows quantitative results, but does not compare them to a classical SAE. This makes it hard to judge whether the TSAE architecture is an improvement over previous methods in this context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSVh5f05dp", "reviewer": "Reviewer_z5dK", "summary": "The paper introduces Temporal Sparse Autoencoders (Temporal SAEs), an extension of standard SAEs designed to handle sequential data. The key idea is to encourage features to remain active over time, capturing higher-level, temporally consistent concepts while allowing lower-level features to vary more freely. The authors motivate this through an analogy between syntactic (short-term, local) and semantic (long-term, abstract) interpretability, and propose a smoothness-based regularization term to enforce temporal coherence in activations. Experiments on language model activations show that Temporal SAEs produce more stable and interpretable latents, with some evidence of improved long-range consistency and separation between high- and low-level features.", "questions": "- What is the intended takeaway from the distinction between syntactic and semantic features? Should one type be preferred or emphasized in certain contexts? More broadly, what is the practical utility of Temporal SAEs compared to standard SAEs? In what kinds of tasks or analyses do the temporal extensions provide clear advantages?\n\n- Could the authors include a direct comparison between Temporal SAEs and standard SAEs in the spliced-text experiment (Figure 4)? This would help confirm that the observed behavior genuinely depends on the temporal component rather than standard SAE properties.\n\n- How robust is the current smoothness metric to outliers or very small latent changes? And would the authors consider alternative measures—such as frequency-domain smoothness (e.g., high-frequency energy ratio) or multi-scale smoothness—to confirm that the results are not artifacts of the chosen metric?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The distinction between syntactic and semantic interpretability, and how this relates to temporal sequences, is a novel framing. It helps clarify how different kinds of latent dynamics might reflect hierarchical structure in sequential data.\n\n- The conceptual split between high-level, consistent latents and low-level, fast-varying ones is elegant and intuitive. It connects well to both neuroscience analogies and hierarchical representations in modern sequence models.\n\n- Figure 1 effectively communicates the model’s core idea and its temporal behavior—it’s visually clear and supports the text well.\n\n- The modeling assumptions are reasonable, and the paper does a good job of articulating the intuition behind them, making the method easy to follow.\n\n- The experiment on long-range consistency adds depth, showing that the proposed method captures meaningful temporal dependencies beyond local smoothness.", "weaknesses": "- It’s not entirely clear what the practical takeaway is. Should one prefer semantic or syntactic features, and in what contexts? The paper motivates the distinction well, but it stops short of offering guidance on which is preferable or when. Relatedly, the utility of the proposed method remains somewhat vague. Section 4.5 shows an application of Temporal SAEs, but without a comparison to standard SAEs, it’s hard to tell whether the temporal extension is actually necessary or beneficial for that task.\n\n- The smoothness metric feels fragile. Because it depends on the maximum activation change, it can overemphasize outliers or produce unstable values when latent differences are small. A more stable approach (such as comparing relative high-frequency energy or using multi-scale smoothness measures) would likely give a clearer and more reliable picture of temporal consistency.\n\n- The spliced-text experiment (Figure 4) would be much stronger if it directly compared Temporal SAEs to normal SAEs. Figure 1 provides a helpful teaser, but including this comparison within the main experimental section would make the qualitative claims far more convincing.\n\n- Table 1 leaves some gaps. It would be informative to include smoothness scores for the low-level split, since those features are expected to vary more quickly and thus might show even lower smoothness than standard SAEs. Including this would help validate the intended hierarchy between low- and high-level components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)