Priors in time: Missing inductive biases for language model interpretability
Authors: Ekdeep Singh Lubana
Can Rager
Sai Sumedh R. Hindupur
Valérie Costa
Oam Patel
Sonia Krishna Murthy
Thomas Fel
Greta Tuckute
Daniel Wurgaft
Eric Bigelow
Demba E. Ba
Melanie Weber
Aaron Mueller
Venue: ICLR 2026 Poster (2026)
Keywords: Top-Down Interpretability
Sparse Autoencoders
Temporal Structure
Stationarity
Primary area: interpretability and explainable AI
TL;DR: We show that language model representations exhibit rich, non-stationary temporal dynamics, and demonstrate how mismatched priors in standard Sparse Autoencoders lead to characteristic failures—addressed by a new temporally biased SAE.

Abstract: A central aim of interpretability tools applied to language models is to recover meaningful concepts from model activations. Existing feature extraction methods focus on single activations regardless of the context, implicitly assuming independence (and therefore stationarity). This leaves open whether they can capture the rich temporal and context-sensitive structure in the activations of language models (LMs). Adopting a Bayesian view, we demonstrate that standard Sparse Autoencoders (SAEs) impose priors that assume independence of concepts across time. We then show that LM representations exhibit rich temporal dynamics, including systematic growth in conceptual dimensionality, context-dependent correlations, and pronounced non-stationarity, in direct conflict with the priors of SAEs. This mismatch casts doubt on existing SAEs' ability to reflect temporal structures of interest in the data. We introduce a novel SAE architecture---Temporal SAE---with a temporal inductive bias that decomposes representations at a given time into two parts: a predictable component, which can be inferred from the context, and a residual component, which captures novel information that cannot be captured by the context. Experiments on LLM activations with Temporal SAE demonstrate its ability to correctly parse garden path sentences, identify event boundaries, and more broadly delineate abstract, slow-moving information from novel, fast-moving information, while existing SAEs show significant pitfalls in all the above tasks. Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.

Published as a conference paper at ICLR 2026
PRIORS IN TIME: MISSING INDUCTIVE BIASES FOR LANGUAGE MODEL INTERPRETABILITY
Ekdeep Singh Lubana' Can Rager Sai Sumedh R. Hindupur Valérie Costa', Greta Tuckute', Oam Patel', Sonia Krishna Murthy Thomas Fel', Daniel Wurgaft Eric J. Bigelow<.?, Johnny Lin Demba Ba Martin Wattenberg', Fernanda Viegas", Melanie Weber Aaron Mueller
Goodfire AI, Independent, $SEAS, Harvard University, 'EPFL
'Kempner Institute at Harvard University, "Department of Psychology, Stanford University, Deparrment of Psychology, Harvard University, Decode Research, Boston University,
" Co-ffrrt authors
ABSTRACT
A central aim of interpretability tools applied to language models is to recover meaningful concepts from model activations. Existing feature extraction methods focus on single tokens regardless of the context, implicitly assuming independence (and therefore stationarity). This leaves open whether they can capture the rich temporal and context-sensitive structure in the activations of language models (LMs). Adopting a Bayesian view, we demonstrate that standard Sparse Autoen- coders (SAEs) impose priors that assume independence of concepts across time. We then show that LM representations exhibit rich temporal dynamics, including systematic growth in conceptual dimensionality, context- context-dependent correlations, and pronounced -on-stationarity, in direct conflict with the priors of SAEs. This mismatch casts doubt on existing SAEs' ability to reflect temporal structures of AAE-with
interest in the data. We introduce a novel SAE architecture- -empttr temporal inductive bias that decomposes representations at given time into two parts: predictable component, which can be inferred from the context, and a residual component, which captures novel information unexplained by the context. Experiments on LLM activations with Temporal SAE demonstrate its ability to correctly parse garden path sentences, identify event boundaries, and more broadly delineate abstract, slow-moving information from novel, fast-moving information, while existing SAEs show significant pitfalls in all the above tasks. Our results underscore the need for inductive biases that match the data in designing robust interpretability tools.
INTRODUCTION
Given the success of Language Models (LMs) (Bubeck et al., 2023; Deepmind, 2025), there is growing interest in understanding how such models incrementally update over sequences of tokens to exhibit complex behaviors (Murthy et al., 2025; Lindsey, 2025: Lindsey et al., 2025; Lepori et al., 2025; Bigelow et al., 2025; Tuckute et al., 2024; Klindt al., 2025). Interpretability research aims to make such analyses tractable, offering tools for hypothesis design, testing, and intervention based on evaluation of intermediate activations (Geiger al., 2025; Sharkey al., 2025; Bereska and Gavves, 2024). Often, such work builds on hypothesized computational models of how concepts are encoded neural network's representations, e-g., the linear representation hypothesis (LRH) (Elhage et al., 2022; Arora et al., 2018), correspondingly motivating tools such sparse autoencoders (SAEs) (Gao et al., 2024; Cunningham et al., 2023) for unsupervised extraction dictionary of vectors that (ideally) mediate human-interna intmanniterrretabble concepts (Mueller et al., 2025).
central challenge in "bottom-up" approaches to interpretability, like SAEs, the mismatch between the assumptions of their underlying implementational account and the precise behavior computation they intend to explain (Jonas and Kording, 2017: Geiger et al., 2025; Costa et al., 2025) (see Fig. 1). For instance, since LRH posits that different concepts correspond to directions in activation space

---

Published as a conference paper at ICLR 2026
There stood dazk-haired boy, Bryffindoz blazing gold against night, zound glasses franing mazked forevex, wand hand was Harsy Pottez.
L
t
Correlations
E
LM Activations
Mismatch
t
Independence
Prior
SAE Latents
Figure The mismatch between SAE assumptions and temporal structure of language. illustrative sentence describing attributes of Harry Potter is shown. When passed into a language model (LM), it leads to activations &; that include concepts within them (possibly entangled); note the presence of large numbers of shared attributes over time, which manifest as correlations across time of activations. Sparse Autoencoders (SAEs) implicitly have an independence (ii.d.) prior across time over their latents and thereby over concepts, which clashes with the true structure of language.
that can be independently manipulated, it implicitly claims the data distribution can be factorized into independently varying latent variables (Allen, 2024). This mismatch between the structure of the data distribution and strong priors codified in LRH can lead to misleading or pathological explanations when using SAEs understand neural networks (Chanin et al., 2025; Bricken al., 2023; Hindupur al., 2025). This raises set of critical questions for using SAEs to interpret models trained on sequential data like language. Specifically, since language exhibits rich temporal structure at multiple scales Maaslen-Wilsson and Tyler, 1980; Thompson, 1999) e.g., sentences contain dependencies that link words across time (Gibson et al., 2000; McElree et al., 2003), upcoming words be anticipated from context (Hale, 2001; Levy, 2008), and discourse imposes structure over longer timescales through phenomena like event boundaries (Zacks et al., 2007; Baldassano al., 2017) can ask what assumptions about temporal structures do SAEs make? How do these assumptions align with the actual temporal structure present in LM's activations?
This work. Building on the Bayesian interpretation of sparse coding (Olshausen and Field, 1996; 1997)-the framework that motivates SAEs-we rephrase the optimization objective of SAEs as MAP (maximum posteriori) estimation problem. This allows us to make explicit prior assumptions about temporal structure embedded in SAEs, showing they implicitly assume concepts are uncorre- lated across time and the number of concepts necessary to explain an activation is is, the information present at each token position is independent of information at other positions and uniformly distributed. As empirically show, these assumptions stand in stark contrast to the actual temporal structure present language and language model representations, and can result in empirically observed pathologies in SAEs, such as feature splitting (Bricken al., 2023; Chanin et al., 2025; Bussmann et al., 2025).
that
These results then motivate to draw a broader parallel between SAEs and computational neuro- science approaches for understanding neural data. Specifically, population-leve analyses of neural recordings have revealed that representations often lie struetured manifolds (Khona and Fiete, 2022: Nogueira et al., 2023; Sohn et al., 2019), challenging the reductionist assumption in sparse coding that computations occur via independently firing. monosemantic features (Eichenbaum, 2018; Saxena and Cunningham, 2019; Barack and Krakauer, 2021). This motivated a paradigm shift towards more structured analysis protocols- methods designed around the generative process of the behavior one aims to explain (Schneider et al., 2023; Chen et al., 2018). Motivated by this and similar findings of intricate geometrical structure neural network representations (Fel et al., 2025; Gurnee et al., 2025; Modell et al., 2025), we propose Temporal SAEs, new protocol for interpreting language model activations that incorporates explicit inductive biases about temporal structure. Our approach decomposes activations at each timestep into two orthogonal components: a predictable component, obtained by projecting current representations onto past context using a learned attention mechanism, and novel component, representing residual information orthogonal to the predictable component. That is, we assume the novel component- not the total representation is uncorrelated over time. This allows correlations between total codes and hence enables our method capture the temporal
2

---

Published as a conference paper at ICLR 2026
structure of LM activations. Overall, argue interpretability methods should be driven by the behavior one is trying to explain.
PRELIMINARIES
Notations. Let bold, lowercase letters represent vectors (e.g., 2). Subscripts on vectors denote different samples (e.g., 2,), while superscripts denote the index within the vector, leading to scalar (c.g., We denote model activations by æ E I", SAE latents (sparse code) by x E RM, and the dictionary by D RRxM (M is the dictionary size).
Sparse Coding. Sparse dictionary learning (Olshausen and Field, 1996; 1997) expresses data as a sparse linear combination of dictionary elements, where both the weights and dictionary are learned from data. Intuitively, the dictionary behaves as a data-adaptive overcomplete basis; i.c., it typically has more elements that the dimension of ambient space. The optimization problem involved in this framework arg minp,s N. Ei= ||=; - Dz. 12 + (Z(z.), where R(-), typically chosen to be the f norm, is sparsityinducing regularizer. Sparsity assists in picking the fewest most relevant dictionary atoms to explain a given data point.
Sparse Autoencoders (SAEs). SAEs (Shu al., 2025) aim to disentangle (Bengio al., 2013; Higgins et al., 2018; Olah, 2023) neural network activations into human-interpretable concepts (Cun- ningham et al., 2023; Bricken et al., 2023). Specifically, SAEs transform their inputs (i.e., neural network activations) into a latent representation which is encouraged to be sparse. As shown by Hindupur al. (2025), this is achieved by solving the sparse coding problem using specific parametric form for the sparse codes:
T
arg min D.= T
Σ 12 - *R(ZZ), s.t.
2; = fSAE((r)) Vk.
zT) = 0,
(I)
where R(-) is the regularizer (typically the L1 norm), SSLE is the SAE encoder architecture, and g(-) captures SAE-specific sparsity constraints on z. fsAE is typically a single hidden layer as in the ReLU SAE (Bricken et al.. 2023; Cunningham et al., 2023), TopK SAE (Gao et al., 2024: Makhzani and Frey, 2013), JumpRel.U SAE (Rajamanoharan et al., 2024) and BatchTopK SAE (Bussmann et al., 2024), though recent work has also explored alternative architectures inspired by sparse coding algorithms to capture specific structures, e.g., hierarchies (Muchane al., 2025; Costa et al., 2025).
TEMPORAL STRUCTURE IN LANGUAGE MODEL ACTIVATIONS
To contextualize the prior assumptions made by SAEs about temporal structure in LMs' activa- tions, we first perform an empirical characterization of such temporal structure in pretrained LMs. Specifically, since LMs are trained to generate coherent text by learning the distribution of natural language, one can expect their representations capture the rich phenomenology of its sequential structure (Elman, 1990); indeed, recent work has in fact found L.M representations to be predictive of human neural recordings during language comprehension (Hosseini et al.. 2024; Schrimpf et al., 2021; Tuckute et al., 2024; Hong et al., 2024; Georgiou et al., 2023). Motivated by this, we perform two experiments relevant to discussion (see Fig. 2): (i) measuring intrinsic dimensionality approximation of the number of concepts necessary to explain the data, which can be expected to increase in monotonic manner with time (Zhong et al., 2024; Can, 2025; Barak and Tsodyks, 2014; Meister et al., 2021)-and (ii) signal nonstationarity which assesses whether model activations reflect the contextual relations between phrases of passage (Zacks et al., 2007).
Increasing intrinsic dimensionality. Fig. 2 (a.e) show the dimensionality of the underlying mani- fold structure (intrinsic dimension) model activations. We estimate the intrinsic dimensionality at a fixed position across set of sequences with the U-statistic (App. Sec. J.2). For language model activa- tions, this metric increases steadily with sequence position. On the other hand, a stationary surrogate of the data (see App. Sec. J.3) shows nearly constant intrinsic dimension over time. This indicates that model activations get 'denser', i.e., they possess more information over time. Correspondingly, the number of concepts needed explain them varies with context.
3

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
(d)
LLama-3. 3.1-8B
Intringi Dimansion
900
Provious Position
4331
Variance Explained
600
-00
-00
0.23
Origial
Sanigato
0.24
80
250
650
230
Sequence Position
Sequence Position
Sequence Position
Sequence Position
(e)
(g)
(h)
120
Previous Position
1.0
Gemma-2- 2-2B
Intrinsic Dimansion
a.
Varlance Explained
105
-20
-23
0.55
-004
-30
00
70
10c
800
50
250
450
30
25n 450
10C 900
Sequence Position
Sequence Position
Sequence Position
Soquence Position
Figure 2: Temporal structure of LLM activations reveals nonstationarity. We use Pile sam- ples (Monology, 2021) analyze temporal structure from activations of two pretrained LMs, com- paring it to surrogate signal that is stationary in nature (see App. J.3). (a, e) Intrinsic dimension of model activations and stationary surrogate. (b, f) Autocorrelations A(x,, 2tt-)) function of sequence position (/) and lag (T). (c, g) Autocorrelation of the stationary surrogate. (d, h) Variance explained by projecting current representation D, onto past context window {æt-1. Z[-w] with different sizes w, along with a baseline. Results consistently show representations getting 'denser' over time and being significantly more structured than a stationary surrogate.
Non-stationarity: Context explains bulk of signal variance. Subplots (b), (f) show the auto- correlation of model activations (App. Sec. J.1), which is noticeably different different sequence positions (x-axis), while the stationary surrogate, as expected, shows nearly position-invarianl auto- correlation values (subplots (c), (g)). This finding is clear signature time-dependent correlation structure, and therefore of non- -on-stationarity. We quantify the similarity of representation with its context in subplots (d), (h). Specifically, we project representations of token 2; at given time onto the subspace spanned by preceeding representations in the context {æ<<) ). These subplots show that up to 80% variance of æ, is explained by a context of 500 tokens, further highlighting strong cross-temporal correlations. Significant variance in the representation at time I, can be predicted (expressed) using representations from the past context.
TEMPORAL PRIORS OF SPARSE AUTOENCODERS
We now state the prior assumptions made by existing SAEs regards sequential structure in an input, contrasting these assumptions with the empirical results shown in Sec. 3. Specifically, building on the arguments used by Olshausen and Field (1997) to formalize the problem of sparse coding, we note that the SAE training objective (Eq. 1) can be interpreted from Bayesian lens: minimize the negative log posterior arg mill(z,) (z,) log P(zı..... ZT æ1..... ØT) of the data, which, by Bayes' rule, can be written as the sum of log likelihood (MSE) and log prior (the regularizer R). From this lens, SAES" prior assumptions sequential structure in LM activations can be described follows.
Proposition 4.1 (Independence prior over time). Consider the SAE maximum aposteriori (MAP) objective from Eq. 1. Since the sparsity constraints are additive over time, this objective has independent and identically distributed (i.i.d.) prior over time:
T
P(Z ZT) x II exp (~XR(z))- )o(zz)) = II P(z)).
more precise version of the claim for specific SAE architectures is provided in Appendix I.I. Intuitively, the claim above says that SAEs assume independence of latents, and hence the concepts underlying the generative process of language, over time. This directly conflicts with the rich contextual structure of LM activations empirically observed in Fig. 2b-d, f-h. Crucially, this
4

---

Published as a conference paper at ICLR 2026
Tokens
11
T--1
Tt
Xn,t
novel component
Xt
Representations
X1
xt-1
Xt
Xp,t
Xn,
p.c
DT
D
Latents
V1
Vt- t--1
Vt
OSAE
context direction
ppt
Zn,t
Query
Context
Keys
Att.
&,,
Figure Schematic of Temporal SAEs. Temporal SAEs decompose activations I; into two components: a predictable component, obtained by projecting onto context direction (derived from the past Eet using attention), and sparse, novel component orthogonal the predictable component that captures new information seen time t.
also implies that SAEs assume the sparsity of latent codes necessary to explain model activations to be time-invariant, stated formally in the corollary below.
Corollary 4.1.1 (Assumptions of inverinvariant sparsity). As consequence of the i.i.d priors over time from Prop. 4.1, standard SAEs assume that sparsity of representations emerges from a fixed distribution independently over time (ii.d.), i.e., P[[|zı|lo.--- -zrl:- = III P(]]=+
SAEs thus assume that marsity-which in their underlying generative model of activations corre- sponds to the number concepts necessary for explaining the data (Bricken et al., 2023; Elhage et al., 2022)-remain approximately constant over time. This again does not align with the increasing dimensionality of representations observed in LM activations (see Fig. 2a,e). Correspondingly, enough concepts aggregate over context such that model's activations become 'denser' than the assumed sparsity budget, the assumption of time-invariant sparsity implies SAEs can fail to capture the temporal structure inherent in language- is arguably already observed empirically with phe- nomena like feature splitting (Chanin et al., 2025; Bussmann et al., 2025; Bricken et al., 2023; Shu al., 2025). We empirically validate this claim in Sec. 6.
TEMPORAL SAES: EXPLICITLY MODELING TEMPORAL PRIORS
As stated in Sec. 2, sparse coding, framework designed computational neuroscience understand neural representations in biological brains (Olshausen and Field, 1996; 1997). inspired SAEs as framework for interpreting artificial neural networks (Bricken et al., 2023; Cunningham et al., 2023). In fact, the parallels between these communities can be made deeper: motivated by observations of intricate geometry of neural representations derived out of caali--dmmesionall population analy- ses (Khona and Fiete, 2022; Nogueira al., 2023; Sohn et al., 2019). there were calls in computational neuroscience to discard the limiting reduction assumed sparse coding that computations occur set of independently firing, monosemantic features (Eichenbaum, 2018; 8: Saxena and Cunningham, 2019; Barack and Krakauer, 2021; Scung. 1996; Chung and Abbott, 2021) -imilar to our arguments in Sec. 3, 4 (and results that follow in Sec. 6). Correspondingly, need for more structured protocols was suggested (Eichenbaum, 2018; Barack and Krakauer, 2021), leading to methods that were moti- vated by the generative process of the behavior one trying to explain (Schneider et al., 2023; Chen et al., 2018; Chen, 2019; Wiskott and Sejnowski. 2002: Linderman et al., 2017). We argue a similar paradigm shift is needed in language model interpretability: given that train models to learn the distribution of highly structured data, we ought to embrace the fact that neural network activations can exhibit intricate geometrical organization. In what follows, as an attempt to qualify our arguments, we propose one such approach that focuses on the temporal structure of LM activations.
Temporal SAEs.
In computational neuroscience, when analyzing data from dynamical domains (e.g., audio, language, video), commonly made assumption is that there is contextual infor- mation present in the recent history that informs the next state- this part of the signal is deemed predictable (Chen al., 2025; Millidge et al., 2024), slow-changing (Berkes and Wiskott, 2005),
5

---

Published as a conference paper at ICLR 2026
invariant (Olshausen and Cadieu, 2007), dense (Tasissa et al., 2022). Meanwhile, the remaining signal corresponds to new bits of information added by the observed state at the next timestep this part can be deemed novel / surprising, fasttngg..atteraee fast-changing, variant, sparse with respect to the context. We argue LM activations are amenable to a similar generative model. Specifically, our observations in Sec. 3 show that activations a time i are strongly correlated with the context and can be decomposed into two such parts. We thus propose the following generative model of LM activations:
x2 = Dp.t + En,ts where xp.t Dzp,t and In,t D
(3)
In the above, Tp.t denotes predictable component of the signal that captures the correlations of æ with past data {x-te), while En,t denotes a novel component that represents new information added by the current token Fy. To obtain 2p,t, we project æ; onto { to explain the predictable variance in oe as a convex combination of past data. Specifically, we use a self-attention layer f on top of
a single ReLU layer, yielding zp.t = f( {=),.... 1. æ.)) Meanwhile, Zn,t = f( Zp,r) = a (D" (a: Dzp,)) captures the residual component of the code, which is not correlated with the past (note that is the nonlinearity). We use a standard SAE encoder (either TopK BatchTopK) to instantiate or applying it to 2; Dzp.e to derive zn,t. See Fig. 3 for an overall schematic of the encoding process. The learning objective in Temporal SAES follows.
T
arg min D
Σ
|0,
DDZp,t + 2n,i11 + XR(=n,)),
T
1=1
s.t. 2p,k fSAE (< <x)), xk), 2n,k fSAE(Z (KK, &p,k), Z Zp. + Zn,k Vk.
Relating to Sec. 4, we note the prior assumption in Temporal SAES is that the residual Zn,t = 21 pp... which captures the novel information in æ, remaining after removing the projections onto the past context, is i.i.d. over time. This prior allows temporal correlations between codes x, and thereby allows correlations between concepts across time, instead of assuming them to be temporally independent.
Sanity Checking Temporal SAEs. Before analyzing how different approaches represent the temporal structure of language. we demon- strate that Temporal SAEs performs on par with SAEs standard metrics such reconstruc- tion error. Specifically, we train a Temporal SAE and standard SAEs (ReLU, TopK, Batch- TopK) on 1B token activations extracted from Gemma-2-2B (Team et al., 2024) from the Pile- Uncopyrighted dataset (Monology. 2021). We also analyze baseline of the prediction only module from Temporal SAEs, reported as 'Pred. only', which can be expected to underperform since predicting the next-token representation is likely to be more difficult than reconstructing it. Results are provided in Tab. 1, 2 and show competitive performance between all protocols, except Pred. only. One also assess which part of a fully trained Temporal SAE is more salient in defining its performance, i.e., does the estimated predictive part ép., contribute more to the reconstruction è or does the estimated novel part e,... Results are reported in Tab. 3. We see that the error vectors, i.e., x 2 2 and x: a are approximately orthogonal, suggesting the mod- ules capture separate bits of information from the input-this is inline with results by Costa et al. (2025), who show optimizing reconstruct
Table I: Temporal SAEs achieves NMSE similar standard SAEs across domains (Simple Stories, Webtext, Code).
ReLU
Topk
BTopK
Pred. Only Temporal
Story Web
0.20
0.155
0.152
0.34
0.139
0.19
0.144
0.139
0.36
0.139
Code
0.20
0.154
0.149
0.38
0.152
Table 2: of signal variance as standard SAEs.
Temporal SAEs explains similar amount
ReLU
TopK
BTopK
Pred. Only Temporal
Story Web Code
0.60
0.71
0.72
0.29
0.73
0.69
0.78
0.79
0.40
0.79
0.65
0.75
0.75
0.33
0.75
Table 3: predictive and novel codes explain differ- ent parts of the input signal across domains.
Sim.
Norm
NMSE
Var. Expl.
Pred. Novel
Pred.
Novel
Pred.
Novel
Story
-0.02
76.2
23.5
0.53
4.03
0.11
0.64
Web
-0.02
80.5
19.5
0.49
4.28
0.17
0.66
Code
-0.02
74.2
26.0
0.57
3.84
0.14
0.65
residuals (as we do with the novel code) can lead to orthogonal codes at different stages of an SAE.
Furthermore, find that bulk of the reconstructed signal &; (in the sense of norm) is captured by the predictive code-io in fact. the percentage contribution of the predictive code is ~880% in line with numbers observed in Fig. 2d. However, analyzing the reconstruction performance, we the

---

Published as a conference paper at ICLR 2026
(a) Activations (r = 76.9)
ReLU (r 108.6)
Topk (r = 86.9)
BatchTopK (F 1069)
Temporal (Novel)
Temporal (Pred)
(c 98.2)
(2 2.8)
80
60
40
There was a baby who wanted pick something special, He went shop and Saw an ancient case. nicked up and ooked
inside. was full of unbelievably shiny geme and jewels. He couldin't believe his eyes, He was so excited and he knew this Case was
perfect. quickly picked up and wrapped it up with big red bow. smilec and couldn't wait to show mum and dad.
(b)
Temporal (Pred)
peppoa
BatchTopK
Figure 4: Temporal SAEs unroll stories, decomposing into events. We consider model activations from a story and compute pairwise similarity codes extracted from different interpretability protocols. (a) We predictive codes from Temporal SAE organizes in hierarchical block structures that seem to align with (sub)event boundaries the analyzed story, while the novel code primarily emphasizes sudden changes in the narrative; meanwhile, standard SAEs show mixture of the two structures, with a stronger similarity to the structure exhibited by the novel codes. (b) We confirm the alignment of predictive codes with event boundaries by running an off-the-shelf hierarchical clustering algorithm, finding the token clusters indeed correspond to (sub)events occurring in the story as the narrative proceeds. Rumning this process on SAEs, we find this process yields temporally incoherent clusters that are primarily defined by lexical information.
predictive component primarily contributes to achieving a good NMSE, while the novel component more responsible for explaining the input signal variance. These results align with the generative model assumed in Temporal SAEs (Eq. 3). Specifically, NMSE captures the average reconstruction. and hence slower moving, contextual signal can expect dominate its calculation. Meanwhile, variance assesses changes per dimension and timestamp in the signal, which better matches the inductive bias imposed on the novel part.
6 CAPTURING TEMPORAL STRUCTURE WITH TEMPORAL SAE
We now evaluate the ability of different SAEs capture temporal dynamics in language model representations. We first analyze a narrative setting where, locally in time, one can expect a lot of correlated structure as an event transpires, with strict event boundaries delineating events from each other. As we show, TemporalSAEs yield clear delincation of the slow movomnn local information from the fast-changing boundaries, while standard SAEs generally ignore the slow-moving information to optimize for the faster changes. The narrative evaluation assesses how different SAE codes represent local/global semantic information. We now investigate local/global syntactic information by analyzing SAE codes extracted from garden path sentences. Here, we find that unlike other SAEs, the use of predictive module in TemporalSAEs yields codes that relate tokens from garden path sentences in manner that would align with the ultimately correct (rather than garden-pant pathh parses-ae an ability that language models are in fact known to possess, but that standard SAEs seem to not explain (Li al., 2024; Hanna and Mueller, 2024).
7

---

Published as a conference paper at ICLR 2026
(a)
Once upon time, little girl named Alice loved (b)
Temporal (Pred)
Novel
ReLU
looking at the night sky'l wish could count all
the stars!' Alice said her friend Maya. The
two girls stood on big grass field as the moon rose from the trees. Suddenly, Maya had striking idea. She opened her laptop and started typingrin" pythonnn array [Nn for range(11 6):Ankt s int(input(ff num "aammmsttrrr)mmit arpay.appeniishntt tot sum(arry/Mv avg tot len(array) print( Avg / night: (avg: 101)
Topk
BatchTapk
Var Explained
0,005
0.02
SAEs
Temporal
40
ReLU
Topk
BatchTlop
Pred
Novel
50
Within
0.29
0.13
0.12
0.36
0.15
E
120
Across
0.30
0.02
0.1
-104
-0.01
0.
0.0 0:00 0:03
0.04
0.05
40 80
120
40
10
120
80
68 00
120
Figure 6: Predictive codes decompose stories into events. (a) We consider model representations from synthetic story with well-defined event boundaries. (b) Computing pairwise cosine similarity of latent codes extracted using different protocols, we see the predictive code of Temporal SAE organizes in hierarchical block structures that seem to align with (sub)event boundaries in the analyzed story. (c) We confirm the alignment of predictive codes with event boundaries by computing average pairwise similarity of token latent codes for tokens that span the same event ('within') versus not ("across"). ). Results clearly show high within-event similarity scores for the predictive code. (d) The results above are further corroborated by running a noising process on the latent codes: we add Gaussian noise of scale a to the input before computing latent codes, defining the similarity maps and computing explained variance of un-noised data. This process elicits coarser grained clusters from the similarity maps for the predictive code, suggesting the multi-scale temporal structure of stories is reflected in predictive codes.
6.1 THE GEOMETRY OF STORIES: A NARRATIVE-DRIVEN DOMAIN
Table 4: Similarity between latent codes and model activations. Predictive codes from Tem- poral SAEs align better with slow-changing part of activations; novel codes and standard SABs' codes align with fast-changing part.
UMAP of Latent Codes Suggests Models Tem- porally Straighten Activations. We consider the TinyStories datasets (Eldan and Li, 2023) for its relatively straightforward narrative structures, and qualitatively analyze the geometry of latent codes extracted from model activations when process- ing these stories. Visualizing the latent codes in a cow-dimensional basis via a 3D UMAP projec- tion (McInnes et al., 2018), we see SAES yield a highly irregular and unstructured geometry (see Fig. 4a). Calculating Tortuosity (Bullitt et al., 2003), a measure of how aligned local ares are with respect to the global structure of a curve, we see very high values emerge for SAEs' latent codes ge- ometry, suggesting sudden changes in the local sim- ilarity as a story unravels. To further understand the results above, we highlight a specific token (" ('and") from the story, the UMAP analysis shows that stan- dard SAES generally just cluster tokens by lexical identity. This is further corroborated by running hierarchical clustering algorithm on the latent codes (SciPy, 2025), finding temporally incoherent, but lexically related clusters (see Fig. 4b).
SAEs
Temporal
ReLU
TopK BatchTopK
Novel
Pred
Slow
0.37
0.35
0.35
0.19
0.75
Fast
0.54
0.54
0.54
0.75
0.18
Magrituds
10-1
107
Gmn dou
Lovel
Rank of Singular Value
Figure 5: Kernel spectrum for latent codes and model representations. Kernels defined using novel code from Temporal SAE and standard SAEs both align well with the fast- changing part of model representations; mean- while, only the predictive code shows strong similarity the slow changing part.
Quantifying Similarity to Slow vs. Fast Moving Signals. To further quantify the straightening claim, we compute the Fourier transform of the model activations and divide the frequency spectrum into two halves at a critical frequency fo such that the energy (i.c., sum of squared value of phase information) in the frequencies below fc equals that of the remaining ones. We call the first split "slow part" of sequence, and latter the "fast part". We then compute the correlation matrix defined by the slow and fast parts, compute their spectrum, and
8

---

Published as a conference paper at ICLR 2026
Garden Path (
ReLU
TopK
BatchTopK
Temporal (Novel)
Temporal (Pred)
The man the bost The man the bost The old man the bost The old man the bost. The old men bost
Control
boat the
The sailors man
The sailors man the
man
boat
(b)
Garden
(c)
Control
SAEs
Temporal
SAEs
Temporal
ReLU
Topk
ButchTopK Novel
Pred
ReLU
Topk BatchTopk
Novel
Pred
V-SPP
0.46
0.09
0.05
-0.06
0.47
V-+SP
0.26
0.38
0.44
0.29
0.44
VPP
0.96
0.78
0.77
0.25
0.85
V V-+OP 0.77
0.75
0.77
0.52
0.79
Figure 7: Hierarchically clustering SAE codes for garden path sentences. (a) Pairwise similarity maps of the predictive code from Temporal SAE link long-distance heads and dependents that define the ultimately correct parse in garden path sentences, while standard SAEs (e.g., BatchTopp) emphasize only local, transient relations, falling for the misleading cues. (b, c) Comparing the cosine similarity of average latent code extracted from the subject phrase (SP), verb phrase (V), and object phrase (OP), we see across ambiguous garden path sentences and unambiguous control variants thereof, only the predictive component of Temporal Features Analyzers shows consistent similarity scores (as expected if the SP and ambiguity reflect the latent codes).
analyze how similar these spectrums are the ones defined using different SAEs' codes. Results shown in Fig. 5. We clearly see the spectrum extracted from the predictive component of Temporal SAE approximates that of the slow part of the representation, while the novel component's spectrum is similar to that of the fast moving part. Meanwhile, spectra of SAEs only exhibits similarity to the fast part, suggesting they do not capture longer range dependencies necessary for interpreting narrative-driven texts like most language domains. To quantify this further, we also measure kernel similarities (CKA) between the kernels respectively defined by the slow moving and fast moving signal to the different SAE codes. Results clearly show Standard SAEs are primarily similar to the fast moving signal.
Predictive Component Captures I Local Event Boundaries. The results above demonstrate Tempo- ral SAFs' predictive component qualitatively align with event boundaries in story. To investigate this result more quantitatively, we use GPT-5 to create a synthetic dataset of 50 stories with well-defined event boundaries (see Fig. 6a for example). We extract latent codes for these stories' tokens, center them by subtracting the mean to remove any globally shared information, and compute the cosine similarity of token to token latent codes. If the latent codes reflect local event structure of a story, the cosine similarity (on average) will be high between token pairs that come from the same event and low (if not zero) between pairs sampled across events; Fig. 6b for an example similarity map corresponding to the story shown in Fig. 6a. Results are shown in Fig. 6 (c) and corroborate our qualitative findings: we see predictive components from Temporal SAE show substantially higher similarity of codes if tokens are sampled from within an event, while the novel component and SAEs generally show low similarity between two tokens. These results are further supported by the robustness of Temporal SAEs to noise. Specifically, we see that when we add noise to the input data, which, on average, will lead to turning off latents with small magnitudes (due to the encoding nonlinearity). the temporal structure of the data, if it is present, will be amplified. We see precisely this effect in Fig. 6d: the predictive components' cosine similarity map under Gaussian noised input maps elicits coarser block structures with increasing noise scale; this is reminiscent of percolation or heat diffusion perspectives on graph clustering, wherein noise diffuses only within a connected component and hence community structure is elicited (Von Luxburg, 2007). Correspondingly. we

---

Published as a conference paper at ICLR 2026
Temporal SAEs respond most gracefully to noise: variance explained reduces slower than SAEs', which in fact drops to ~0 at some scale.
6.2 GARDEN PATH SENTENCES: AMBIGUITIES RESOLVED VIA TEMPORAL STRUCTURE
Garden-path sentences-e.ge The old man the boat-initially cue incorrect local parse before a later token forces reanalysis. Language models have been shown to be able to correctly parse such ambiguous sentences, offering in fact a predictive account of human per-token surprisals (Li et al., 2024; Hanna and Mueller, 2024; Oh and Schuler, 2023). Interestingly, when using SAE codes to assess whether L.M representations offer valid parse the sentence, we find hierarchical clustering of SAE codes yields a parse that is suggestive of the mislcading cue: meanwhile, Temporal SAE recovers the correct parse by separating the predictable, slow-moving component of the representation from the novel, fast-changing residual. Specifically, in Fig. 7, see the predictive codes link long- distance heads and dependents that reflect the correct parse (e.g., man as verb), producing coherent similarity structure over the full span, whereas standard SAEs emphasize only transient, local changes and miss these cross-temporal constraints, These results suggest Temporal SAEs encode syntactic structure that unfolds over time when evidence to collapse the correct constituent parse emerges. To make these results more quantitative in nature, we use GPT-5 to synthetically generate a set of 50 garden path sentences where the subject is ambiguous. We create 50 control variants of these sentences such that the controls do not possess ambiguity with respect to typical parse of the sentence constituents: e.g., changing the subject in the sentence The old man the boat from old to sai sorrs,, yielding The sailos fairors man the boat. We then divide all sentences into their respective subject phrase (SP), verb phrase (V), and object phrase (OP): e.g., The old (SP), man (V), and the road (OP). We compute the average latent code for tokens from these three constituent phrases, under the hypothesis that if the sentence ambiguity is reflected the latent code, then until the OP shows up, the correct parse of prior words cannot be identified. Correspondingly, all valid parses must be stored the same representation. This suggests the cosine similarity of latent codes of V and SP tokens should be of similar order in both the garden path and control sentences; meanwhile, the similarity between V and OP should be much higher than V and SP. Results are reported in Fig. 7 (b.c). We clearly see extreme sensitivity in similarity values of SAE latent codes and the novel component of Temporal SAE, but the predictive component is essentially invariant across sentence type, suggesting it captures the temporal dynamics likely relevant for a LM parse garden path sentences.
7 DISCUSSION
Our findings reinforce the broader lesson that interpretability tools must align their inductive biases with the statistical structure of the data they are applied to. We showed that standard SAEs impose independence priors across time, which are fundamentally misaligned with the nonstationary and context-dependent structure of language model activations. This mismatch explains why existing SAEs tend to underrepresent temporal dependencies, despite capturing other kinds of structure. By contrast, Temporal SAE incorporates empirically observed correlations across time as an inductive bias. Its decomposition of activations into predictable sslow-moving) and novel (fast-changing) components enables the recovery of temporal structure that standard SAES fail to expose. In particular, demonstrated that predictable codes align with stable, high-level information, while novel codes isolate transient or surprising information, allowing Temporal SAE to highlight event boundaries and syntactic reanalyses in garden-path sentences. Taken together, these results emphasize general principle: interpretability methods should not be viewed as neutral feature extractors but as models with their own structural assumptions. When these assumptions mismatch the true data distribution, important aspects of representation may be obscured. Incorporating empirically motivated temporal priors offers one way to close this gap, suggesting that future progress in interpretability will require tailoring methods to the dynamics of the representations under study.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
ESL thanks members of the Mechanisms team at Goodfire AI, particularly Tom McGrath, Owen Lewis, Jack Merullo, and Attieus Geiger for helpful discussions. ESL additionally thanks Jack Lindsey for useful comments that helped formalize the intrinsic dimensionality arguments in Sec. 4, and Raphael Sarfati for suggesting the Tortuosity metric used in Fig. 4. The authors also thank David Bau for helpful discussion about the generalization ability of SAEs, and David Klindt for several useful comments on an earlier draft of the paper. ESL. and SSRH further thank Noor Sajid and members of the CRISP lab Harvard for helpful conversations. CR is supported by MATS extension grant. AM's work is partially supported by the National Science Foundation (Grant No. 2530728) and Binational Science Foundation. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute at Harvard University.
REFERENCES
Carl Allen. Unpicking data at the seams: Understanding disentanglement in vaes. arXiv preprint arXiv:2410.22559, 2024.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Association for Computational Linguistics, 6:483 495, 2018.
Christopher Baldassano, Janice Chen, Asieh Zadbood, Jonathan W Pillow, Uri Hasson, and Kenneth Norman. Discovering event structure in continuous narrative perception and memory, Neuron, 95 (3):709-721, 721,,2212222 2017.
David L Barack and John W Krakauer. Two views on the cognitive brain. Nature Reviews Neuro- science, 22(6):35--371, 2021.
Omri Barak and Misha Tsodyks. Working models of working memory. Current opinion in neurobiol- ogy, 25:20-22 24, 2014.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-1825 2013.
Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety review. arXiv preprint arXiv:2404. /4082, 2024.
Pietro Berkes and Laurenz Wiskott. Slow feature analysis yields a rich repertoire of complex cell properties. Journal vision, 5(6):9-9, 2005.
Eric Bigelow, Daniel Wurgaft, YingQiao Wang, Noah Goodman, Tomer Ullman, Hidenori Tanaka, and Ekdeep Singh Lubana. Belief dynamics reveal the dual nature of in-context learning and activation steering. arXiv preprint arXiv:2511.00617, 2025.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zae HatfieldDDodds Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. features.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12, 2023.
Elizabeth Bullitt, Guido Gerig, Stephen M Pizer, Weili Lin, and Stephen R Aylward. Measuring tortuosity of the intracerebral vasculature from mra images. IEEE transactions on medical imaging. 22(9):1 117-1111, 2003.
11

---

Published as a conference paper at ICLR 2026
Bart Bussmann, Patrick Leask, and Neel Nanda. Batchtopk sparse autoencoders, 2024. URL pattps://ariie org/ http://b///// 2412.06410.
Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. Learning multi level features with matryoshka sparse autoencoders. arXiv preprint arXiv:2503. 2025.
Tankut Can. Statistical mechanics of semantic compression. arXiv preprint arXiv:2503.006// 2025.
David Chanin, James Wilken-Smith Tomas Dulka, Hardik Bhatnagar, Satvik Golechha, and Joseph Bloom. A is for absorption: Studying feature splitting and absorption in sparse autoencoders, 2025.
URL https::////... http://axvvvvrr// abs 9409. 11000.1145777 14507.
Yibei Chen, Zaid Zada, Samuel Nastase, Gregory Ashby, and Satrajit Ghosh. Context modulates brain state dynamics and behavioral responses during narrative comprehension. bioRxiv, pages 2025-04, 2025.
Yubei Chen. The Sparse Manifold Transform and Unsupervised Learning for Signal Representation. PhD thesis, University of California, Berkeley, 2019.
Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018.
Sue Yeon Chung and Larry F Abbott. Neural population geometry: An approach for understanding biological and artificial neural networks. Current opinion neurobiology. 137-144 2021.
Valérie Costa, Thomas Fel, Ekdeep Singh Lubana, Bahareh Tolooshams, and Demba Ba. From flat to hierarchical: Extracting sparse representations with matching pursuit. arXiv preprint arXiv:2506.03093, 2025.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey, Sparse autoen- coders find highly interpretable features in language models. arXiv preprint arXiv:2309.08000
2023.
Google Deepmind. ficially achieves ical olympiad.
Advanced version of gemini with deep think of-
gold-medal
standard at the international mathemat- https: pettss:/deeppiiee http///eeeedddooeeeeeee www.ggoollll..o.e///oooooeaeee think ofain--officiaally-ahheee achieves goll gold-medaa st standard
advanced wwwaanceddeeiiion--- -eessinnnnoot of of--emini--i--iiiiiii--- weeiiiwwttt-eeee
2025.
Howard Eichenbaum. Barlow versus hebb: When is it time to abandon the notion of feature detectors and adopt the cell assembly the unit of cognition? Neuroscience letters, 680:88-93, 2018
Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science Business Media, 2010.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.0775: 2023.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCan- dlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):179-21 1990.
Thomas Fel, Binxu Wang, Michael A Lepori, Matthew Kowal, Andrew Lee, Randall Balestriero, Sonia Joseph, Ekdeep S Lubana, Talia Konkle, Demba Ba. et al. Into the rabbit hull: From task-relevant concepts in dino to minkowski geometry. arXiv preprint arXiv:2510.08638, 2025.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprins arXiv:2406. 04093, 2024.
12

---

Published as a conference paper at ICLR 2026
Atticus Geiger, Duligur lbeling, Amir Zur, Maheep Chaudhary, Sonakshi Chauhan, Jing Huang, Aryaman Arora, Zhengxuan Wu, Noah Goodman, Christopher Potts, and Thomas Icard. Causal abstraction: A theoretical foundation for mechanistic interpretability. Journal of Machine Learning Research, 26(83)11-66 64, 2025. URL http: fmmmmmmmmmmmm r. org / pttpg/apprr////// 23- 0058 .htll.
Antonios Georgiou, Tankut Can, Mikhail Katkov, and Misha Tsodyks. Using large language models to study human memory for meaningful narratives. arXiv preprint arXiv:231 1.04742, 1.44442 2023.
Edward Gibson et al. The dependency locality theory: A distance-based theory of linguistic complex- ity. Image, language, brain, 2000:95-126, 2000.
Wes Gurnee, Emmanuel Ameisen, Isaac Kauvar, Julius Tarng, Adam Pearce, Chris Olah, and Joshua Batson. When models manipulate manifolds: The geometry of a counting task. Trans- former Circuits Thread, 2025. URL https: httt///tasssooroootoeeeeeeeeeeet former-ct ci reui ts. www.tts..ut//...... pub/2025/
linebreaks index.
John Hale. A probabilistic earley parser psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics, 2001.
Michael Hanna and Aaron Mueller. Incremental sentence processing mechanisms in autoregressive transformer language models. arXiv preprint arXiv:2412.05353 2024.
Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerehner. Towards a definition of disentangled representations. arXiv preprint arXiv: 02220220.002... 2018.
Sai Sumedh R Hindupur, Ekdeep Singh Lubana, Thomas Fel, and Demba Ba. Projecting assumptions: The duality between sparse autoencoders and concept geometry. arXiv preprint arXiv:2503.01822 2025.
Zhuoqiao Hong, Haocheng Wang. Zaid Zada, Harshvardhan Gazula, David Turner, Bobbi Aubrey, Leonard Niekerken, Werner Doyle, Sasha Devore, Patricia Dugan, et al. Scale matters: Large lan- guage models with billions (rather than millions) of parameters better match neural representations of natural language. bioRxiv, 2024.
Eghbal Hosseini, Colton Casto, Noga Zaslavsky, Colin Conwell, Mark Richardson, and Evelina Fedorenko. Universality of representation in biological and artificial neural networks, bioRxiv, 2024.
Eric Jonas and Konrad Paul Kording, Could a neuroscientist understand microprocessor? PLoS computational biology, 13(1)ee1052268, 2017.
Mikail Khona and Ila R Fiete. Attractor and integrator networks in the brain. Nature Reviews Neuroscience, 23(12):744-766, 2022.
Connor Kissane,
Arthur Conmy, and Neel Saes (usually) transfer between base and chat models, 2024.
Robert
Krzyzanowski,
Nanda.
https: wwtp///wwww/./// www..annnggmetnorrmmmmmm wwttposts////oooooooooootsssssst Emwk 6qxxpp8d4 jvbdd sues-usually-traasffe-----e
models..
David Klindt, Charles O'Neill, Patrik Reizinger, Harald Maurer, and Nina Miolane. From su- perposition to sparse codes: interpretable representations neural networks. arXiv preprint arXiv:2503.01824, 2025.
Mikhail Koroteev. Bert: a review of applications in natural language processing and understanding. arXiv preprint arXiv:2 103. 1943, 2021.
Michael A Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, and Ellic Pavlick. Is this just fantasy? language model representations reflect human judgments of event plausibility. arXiv preprint arXiv:2507. 12553, 2025.
Roger Levy. Expectation- based syntactic comprehension. Cognition, 106(3):1 (6663):122-1111 11771 2008
13

---

Published as a conference paper at ICLR 2026
Andrew Li, Xianle Feng, Siddhant Narang, Austin Peng, Tianle Cai, Raj Sanjay Shah, and Sashank Varma. Incremental comprehension of garden-path sentences by large language models: Semantic interpretation, syntactic re-analysis, and attention. arXiv preprint arXtv:2405.16042, 2024.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Commy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all once on gemma arXiv preprint arXiv:2408.08 0477,, 2024.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial intelligence and statistics, pages 914-92212999 PMLR, 2017.
Jack Lindsey. Emergent introspective awareness in large language models. Trans- former Circuits Thread, 2025. URL htappp////crassforeerrrrrooo post pub/2025/ introspection index. html.
Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. On the biology of large language model. Transformer Circuits Thread, 2025. URL https: pub/2025
graphs / wwt.bbillllly....o..........oo.. html.
Alireza Makhzani and Brendan Frey. K-sparse autoencoders. arXiv preprint arXiv: 2013.
William Mirslen-Wilson and Lorraine Komisarjevsky Tyler. The temporal structure of spoken language understanding, Cognition, 8(1):1-71, 1980. ISSN 0010-0277. doi: https://oi..og 10. 10.1016/0100-2277889900--3333333 URL https: www. eeeeeeeneeirett www.reeetttteeeeeeeeeeeeeeeeeeee coom/scien science/ aatiicle/pp1/0011077799999
Brian McElree, Stephani Foraker, and Lisbeth Dyer. Memory structures that subserve sentence comprehension. Journal of memory and language, 48(1):67-91, 2003.
L. McInnes and J. Healy, Umap: Uniform manifold approximation and projection for dimension reduction, 2025. URL httpp::tttptpp/tttetteee /twpp/mapeeeeee -earnn readthedocs
wtt.eeeeaa.aeacoc......oo. wioee////////eei latest/.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 180.03480, 2018.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell. and Roger Levy. Revisiting the uniform information density hypothesis. arXiv preprint arXiv:2 1635, 2021.
Beren Millidge, Mufeng Tang, Mahyar Osanlouy, Nicol S Harper, and Rafal Bogacz. Predictive coding networks for temporal prediction. PLOS Computational Biology, (0(4)=e0111183 2024.
Alexander Modell, Patrick Rubin-Delanchy, and Nick Whiteley. The origins of representation manifolds in large language models. arXiv preprint arXiv:2505.18223: 2025.
Monology. The pile: Uncopyrighted subset. https:/ haapss///hgggnngaccceee co/ co/c/o//////// datasets/ wttplo/yy/pilleenoooyoyeeo al.
2021. Based on the original Pile dataset by Gao et
Mark Muchane, Sean Richardson, Kiho Park, and Victor Veitch. Incorporating hierarchical semantics in sparse autoencoder architectures, 2025. URL
http://bb///../... /2006 01197.
Aaron Mueller, Jannik Brinkmann, Millicent Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanaa Arnab Sen Sharma, Jiuding Sun, Eric Todd, David Bau, and Yonatan Belinkov. The quest for the right mediator: Surveying mechanistic interpretability through the lens of causal mediation analysis, 2025. URL https:/ arxiv wrw. abs /200 01416
Sonia K Murthy, Rosie Zhao, Jennifer Hu. Sham Kakade, Markus Wulfmeier, Peng Qian, and Tomer Ullman. Inside you are many wolves: Using cognitive models to interpret value trade-offs in llms. arXiv preprint 2506. 25066660,,0 2025.
14

---

Published as a conference paper at ICLR 2026
Ramon Nogueira, Chris C Rodgers, Randy M Bruno, and Stefano Fusi. The geometry of cortical representations of touch in rodents. Nature Neuroscience, 26(2):239- 250, 2023.
Byung- Oh and William Schuler. Transformer- based language model surprisal predicts human reading times best with about two billion training tokens. arXiv preprint arXiv:2304. 11399, 2023.
Chris Olah. Distributed Representations: Composition & Superposition, 2023. https:
connsffrrr--circceee / wwwppppssitiiioooo wwwpiitioooooooooooooooooooooooo html.
Bruno Olshausen and Charles Cadieu. Learning invariant and variant components of time-varying natural images. Journal of Vision, 7(9):94-9964, 2007.
Bruno A Olshausen and David J Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38166683))007-609 1996.
Bruno A Olshausen and David Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision research, 37(23)333113332 1997.
Senthooran Rajamanobaran, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda, Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders, 2024. URL http:::tt///s /tt///axvvvv/////e abspg//bbb/24// 14435.
Shreya Saxena and John P Cunningham. Towards the neural population doctrine. Current opinion in neurobiology, 55: 103-111, 2019.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):30-366. 2023.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 1 11845))e21106661111 2021.
SeiPy. Dendrogram, hierarchical clustering, scipy, 2025. URL httpsttss://oos... cluster hiwrarchy dendrogram.
org/
doc/ html.
Sebastian Seung. How the brain keeps the eyes still. Proceedings of the National Academy of Sciences, 933(2)):3339-11334, 1996.
Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill. Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open problems in mechanistic interpretability. arXiv preprint ar 2rXiv:2501.16490 2025.
Dong Shu, Xuansheng Wu, Haiyan Zhao, Daking Rai, Ziyu Yao, Ninghao Liu, and Mengnan Du. A survey on sparse autoencoders: Interpreting the internal mechanisms large language models. arXiv preprint arXiv:2503.0561:: 2025.
Hansem Sohn, Devika Narain, Nicolas Meirhaeghe, and Mehrdad Jazayeri. Bayesian computation through cortical latent dynamics. Neuron, 103(5):934-947, 2019.
Abiy Tasissa, Emmanouil Theodosis, Bahareh Tolooshams, and Demba Ba. Towards improving discriminative reconstruction via simultaneous dense and sparse coding, 2022. URL https:
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint a8X2v:2403.08225, 2024.
Ellen Thompson. The temporal structure of discourse: the syntax and semantics of temporal then. Natural Language & Linguistic Theory, 17(1):13-1660 1999.
15

---

Published as a conference paper at ICLR 2026
Greta Tuckute, Aalok Sathe, Shashank Srikant, Maya Taliaferro, Mingye Wang, Martin Schrimpf, Kendrick Kay, and Evelina Fedorenko. Driving and suppressing the human language network using large language models. Nature Human Behaviour, 863):544-561, 2024.
Ulrike Von Luxburg. A tutorial spectral clustering. Statistics and computing, 17(4)3395-416, 2007.
Laurenz Wiskott and Terrence J Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770, 2002.
Jeffrey M Zacks, Nicole K Speer, Khena M Swallow, Todd S Braver, and Jeremy R Reynolds. Event perception: mind-brain perspective. Psychological bulletin, 133(2):273, 2007.
Weishun Zhong, Tankut Can, Antonis Georgiou, Ily Shnayderman, Mikhail Katkov, and Misha Tsodyks. Random tree model of mcaningful memory. bioRxiv, pages 2024-12, 2024.
16

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
Below, we discuss broad details of the experimental setup, training protocol, and measures used for evaluations.
A.1 TRAINING, HARVESTING, AND INFERENCE FOR SAES
Compute. All experiments were performed using I H100, both for extracting representations, analyzing them, and training Temporal and standard SAES. To save training costs, we precache activations on local server and train an "online manner", i.c., sampling a new batch of activations every training iteration.
SAEs' training. All analyzed SAES were trained from scratch on 1B precached activations from the Pile- Uncopyrighted dataset (Monology, 2021). While we did try use existing, off-the- shellf SAEs, we found the results to be wildly inconsistent depending on where we borrowed the SAE from. To enable consistent and fair evaluation, thus preferred to train all SAEs from scratch. For Gemma models, activations were extracted from Layer 12; for Llama models, from layer 15 (all O-indexed).
Training procedure. We use Adam optimizer with standard hyperparameters, We initialize training with warmup of 200 steps learning rate of 10-3 following from thereon minimum of 9 x 10-1, i.e., the learning rate remains essentially constant throughout training.
Activations normalization. Following best practices, normalize activations to unit expected norm (Bricken et al., 2023; Costa et al., 2025). This helps put different SAEs on the same training scale, and especially for ReLU SAEs, makes training substantially easier by reducing the need for tuning regularization strength. On related note, we emphasize our ReLU SAEs have a slightly higher Lo, i.c., are less sparse, than almost all other SAEs analyzed in this work.
A.2 EXPERIMENTS WITH LLM REPRESENTATIONS: AUTOCORRELATION AND U-STATISTIC
We primarily analyze Gemma-2-28 and Llama-3.1 3.1-BB models in this paper. Specifically, we precache 10K activations for the analyzed datasets for our experiments on dynamics on language model representations.
A.2.1 AUTOCORRELATION
We compute autocorrelation by selecting evenly spaced tokens across the sequence and measuring the cosine similarity between each token and tokens various lags in the past. Specifically, for tokens at position f, we compute similarities to tokens at f - w where lag w ranges from 5 to 20. This creates a heatmap where rows represent lag offsets and columns represent token positions.
For a stationary process, we expect the autocorrelation pattern to remain consistent across time that is, the relationship between token and its historical context should be similar regardless of position in the sequence. This would manifest as similar autocorrelation patterns repeating horizontally across token positions. In contrast, for a non-stationary process where representations evolve over time, expect the autocorrelation patterns to vary systematically across positions, with columns showing different temporal dependency structures as the sequence progresses.
A.2.2 U-STATISTIC
We measure the effective dimensionality of LLM representations using U-statistic (an unbiased
estimator) based on pairwise cosine similarities. Let X, (xx ......r (M) be M samples of normalized activations at time t (from different timescries). G, = X is the Gram matrix. Using the above, a U-statistic of intrinsic dimension which we employ defined below:
M - M
U-stat(t)
(5)
|G, 11 M
17

---

Published as a conference paper at ICLR 2026
where [ is the squared Frobenius norm of the Gram matrix. This quantity estimates the effective rank 1/m(C(), where C₁ = E | is the second moment matrix and at is the activation vector at time t. Under stationarity, U-stat remains constant. When representations evolve over time, U-stat increases systematically more orthogonal directions become active.
A.2.3 PROJECTION ANALYSIS
This analysis quantifies how much of the representation E RP at token position can be reconstructed from its preceding context (z; hew, where the context window W = [t - w, t 1] contains the previous tokens.
For a population B sentence samples, we compute the projection of each target representation onto the subspace spanned by its context. Let (() W i- " (bj ) E R denote the matrix of context representations for sample b, where each row is a context vector. Prior to projection, we
center all representations by subtracting the mean computed over the target positions across samples.
The projection ( onto the span of the context is:
(
span (x ( zt
(b)
(6)
where span(X w/ is the row space of X x that can computed via SVD. The variance explained by the context is:
expvar(t,, w) Ed var(cr,...)) Ed var(Xx,.)
where var(Cr, and var( denote the variance of the d-th dimension across the B samples. This ratio measures the fraction of total representational variance that be linearly reconstructed from the preceding context. Values approaching indicate high predictability from context; values near indicate representations largely orthogonal their context subspace.
A.2.4 SURROGATE
For the U-statistic and Autocorrelation metrics, we compare LLM activations to surrogate distribu- tions that preserve certain statistical properties while removing temporal structure. We operate on representation vectors E IR. BxT'xd where B denotes batch size, T denotes sequence length, and denotes the model dimension.
U-statistic surrogate (Fig. 2 a, e): For each sequence i (1...., B), we construct the surrogate X, by applying a random permutation क.; {I,..., T} + (I....,T) to the temporal positions:
X (t)))) Vt E (1,...,T)
This preserves the marginal distribution of activations within each sequence while destroying temporal dependencies.
Autocorrelation surrogate (Fig. 2 g): Given the similarity matrix S E T where S,₃
sim(X. .,., X. X.))), construct the surrogate similarity matrix S by replacing each diagonal with its mean:
T-k
Si Sk where k = jl.
S.
S
T
This preserves the average correlation structure at each lag while removing position-specin temporal patterns.
A.3 ANALYSIS OF STORIES: UMAP PROJECTIONS, DENDROGRAMS. SIMILARITY HEATMAPS, AND EVENT BOUNDARY DETECTION
Data. For experiments in Sec. 6.1, used either stories sampled from the Storiso dataset (El- dan and Li, 2023) (for Fig. 4, 5) or synthetically sampled stories using GPT-5 (for Fig. 6). For the former, sampled stories that were up 100 tokens long, allowing us characterize compute
18

---

garden Published as a conference paper at ICLR 2026
UMAP for both Gemma and Llama models without hitting memory bottlenecks. For consistency, then used these stories for all experiments. For synthetically defined stories, we defined 3 in-context examples (one of which is shown in Fig. 6a) and prompted GPT-5 to produce 50 stories with similar such suddenly changing events. Stories varied in size, but were generally less than 150 tokens long. Following Georgiou et al. (2023), the generated stories' events were labeled using GPT-5. These labels served as the 'ground truth' event boundaries. We qualitatively analyzed and confirmed the event boundaries overlap with our intuitively expected event structure in the stories.
Analysis Details. To isolate the low-dimensional geometry of latent codes, we perform 3D UMAP analysis using the open-source package (McInnes and Healy, 2025). Similarly, Dendrograms in Fig. 4 are computed using the hierarchical clustering package in SciPy SciPy (2025). Branches are colored based on proximity, which in our case was between 0-1 (since we use cosine similarity as the clustering measure). We put proximity bound of 0.2 as the distance under which branches colored within a cluster. For Fourier Analysis, as mentioned Sec. 6.1, we performed a Fourier transform of the model activations for a given story, isolated its lower frequency components-defieee as 0.1x x the Nyquist rate, which turns out to possess ~50% of the signal norm-and compute the cosine similarity kernel of the low high frequency spectra. These kernels then compared to kernels of latent codes derived using Temporal or standard SAES.
A.4 ANALYZING GARDEN PATH SENTENCES: DENDROGRAMS AND PHRASE SIMILARITY
Data. We used GPT-5 to sample 50 garden path sentences and control variants thereof (1 corre- sponding to each sentence). Sentences were 20-40 tokens long and had structure such that the observation of the object phrase resolved ambiguity. The precise ambiguity structure was were varied in type, i.e., the ambiguity could be resolved by altering the subject (e.g., old sailors in The old man the road) or by other mechanisms such as adding punctuations to elicit a pause (e.g.. adding comma, such as The old train the young fight + The old train, the young fight, Control variants had a mixture of such resolutions applied.
Analysis. For assessing whether tokens in verb phrase relate more with the subject phrase, as would be expect by the typical parse in our used garden path sentences, or with the object phrase, as would necessary for the correct parse, we computed Dendrograms using the hierarchical clustering package in SciPy (SciPy, 2025). The similarity between subject phrase (SP), verb phrase (V), and object phrase (OP) was computed by taking the set tokens Tp that belong to a phrase P E (SP, V, OP), and computing the average latent code: c(P) PP] E1ETp Ct. where denotes the latent code extracted for token using either SAEs or the predictive or novel component of Temporal SAEs. This protocol is similar to popularly used strategies for computing sentence embeddings (see, e.g., work using BERT embeddings (Koroteev, 2021)). Cosine similarity between these path / control sentences yields the tables shown in Fig. 7.
19

---

Published as a conference paper at ICLR 2026
B FURTHER EVALUATION RESULTS
B.1 U-STATISTIC ACROSS DOMAINS FOR GEMMA-2-2B
Figure 8: U-Statistic across domains for LLM activations, surrogate
ligh
more
secore
qud pand,
123
30
Figure 9: U-Statistic across domains for SAE reconstructions
20100
Sanigens
Ligh CONL CSC
rolice des
sempee vnd coloo
Figure 10: U-Statistic across domains for SAE codes
B.2 AUTOCORRELATION ACROSS DOMAINS FOR GEMMA-2-2 2--2B
20

---

Published as a conference paper at ICLR 2026
Webtext - Autocorrelation
Code Nutocorelatiion
SimoleStories Autocomelation
44
-16
Relatve Ortzets 1p)
109
18
Relatte Oftsets
18
0.570
0.455
22
0.965
0.450
0.42
0.500
26-
-66
.25
0.445
0.55
-30
-30
0.440
249
Anchors
Wabtext Surrogate
0.435
Code Surrogate
Simpl Stmpleetories Surrogate
-30
-30
0.38
-30
0.430
0.425
Reative Offects
Reative Offects
0.420
0.34
-18
18
-10
-10
.10
149
249 Anchors (NO
349
449
149
249
349
449
149
249
349
449
Anchors (W)
Anchors (W)
Figure 11: Autocorrelation of language model activations and stationary surrogate across webtext, simple stories and code domains.
21

---

Published as a conference paper at ICLR 2026
Code
SimpleStories
Webtext
Reative Officts (0)
-10
-10
10
0.69
-14
-1
0.68
-8
0.735
22
0.67
-2
0.09
30
0.05
-30
-30
149
249
349
449
49
149
249
349
449
49
149
249
349
449
Relative Officets (0)
1.4
14
114
0.08
0.07
0.07
26
0.06
0.06
Relatve Offietts (2)
10
-10
-14
18
0.175
18
026
18
22
0.150
30
-30
-30
0.18
49
149
249
349
449
49
149
249
349
449
49
149
249
349
449
Relative Offsets 1p1
-10
-10
-10
-14
0.14
-14
0.12
0.11
22
0.12
-22
0.11
-26-
0.10
149
449
349
449
Reiative Offietts (9)
-14
0.975
-14
0.975
0.975
-18
0.950
18
0.950
0.950
26
26
26
0.925
0.92
-30
-30
49
149
249
349
449
49
149
249
349
449
49
149
249
349
449
Reative Offects (p)
10-
0.80
-10
0.760
-10
0.785
topk
18
0.79
0.780
222
.2
-22
0.750
30
0.78
-30
30
0.770
149
249
349
449
49
149
249
349
449
49
144
249
349
449
Sequence
Sequence
Sequence
(W)
Figure 12: Autocorrelation of SAE latent activations webtext, simple stories and code domains.
22

---

Published as a conference paper at ICLR 2026
C RECONSTRUCTION PERFORMANCE OF SAES
Nurmzurd
Coorne
8.0
112
Figure 13: Atm SAE BatchTopk
250
batch_top__
jamp.relu
0.325
200
6.275
0.88
100
6.225
50
18P
Posttion
Rosttion
Posttion
Figure 14: Reconstruction performance of LLama-33 3.1 3.B-BB resid post Layer 16 (1/2 forward pass) SAEs on webtext.
3.5
Borral BiN
2.44
30-
1ra.k
60
102
160
167
267
2.7
Figure 15: Reconstruction performance of Gemma-2 -2-BBBBB-BBBB--- resid post Layer 12 (1/2 forward pass) SAES on webtext.
23

---

Published as a conference paper at ICLR 2026
D CONSTANT SPARSITY ARGUMENT: RANK AND U-STATISTIC
activations
250
Surrogate
batchtopk recons
temporal novel_codes
200
temporal / novel_recons
temporal pred_codes
codes
temporal pred recons
ustat
150
temporal recons
100
100 Sequence Position
Figure 16: Gemma-244 2-2b: LLM activations vs Attention SAEs vs BatchTopK
D.1 LLAMA-3.1-8B
Constant Sparsity Argument: Multiple Layers Llama-3.1-8B pile-uncopyrighted
Layer Surrgate
60
Layer 16 Original
Layer 16 Strrogate
300
Layer Original
U-Stattete
Layer 24 Surrggate
800-
Effective
600
20
200
10
10a
Token Position
Taken Position
Figure 17: Rank and U-statistic, Llama-3.1-8B 11m activations and surrogate, Multiple Layers
Constant Sparsity Argument: U-Statistic vs Effective Rank
Llama-3. 3.1-8B Layer 16 pile-uncopyrighted
Origiaal
LLM Surrogate
600
SAE re
40
Effective Bank
top_k
U-Statisic
SAE batch_top.k k.
30
400
20
200
30a
102
10'
10a
Takan Position
Token Posttion
Figure 18: Rank vs U-statistic, Llama-3.1-8B. webtext
24

---

Published as a conference paper at ICLR 2026
Constant Sparsity Argument: U-Statistic Effective Rank Liama-3.1-88 Layer 16 SimpleStories
530
Original
500
LLM Surrogate
SAE rull
450
jump _etu
SAE top_K
U-Stalletc
400
SAE batch top
Efrective
300
250
200
16
Token Position
Token Position
Figure 19: Rank U-statistic, Ilama-3.1-8B, simple stories
Constant Sparsity Argument: U-Statistic vs Effective Rank Llama-3.1-88 Layer code-10k
600
LLM Original
40-
330
LLM Surrogate
500
jump, pelu
Effective Bank
SAE to
U.Saaisti
450
25
400
20
350
300
250
200
201
10
10a
Token Position
Token Position
Figure 20: Rank vs U-statistic, Ilama-3.1-8B. Code
D.2 REMMA-2-22
Constant Sparsity Argument: Multiple Layers gemma-2-2b pill-uncopyrighted
Layer Criginal
Layer Surrogate
10.
350
Layer Original
Layer 12 Surrogate
Effective Pank
525
Layer 36 Original
U.Statitic
800
Surrgaate
8-
425
450
425
400
20
10a
Token Position
Position
Figure 21: Rank and U-statistic, Gemma-2-2 2-2b Ilm activations and surrogate, Multiple Layers
25

---

Published as a conference paper at ICLR 2026
Constant Sparsity Argument: U-Statistic Effective Rank gemma-2-2b Layer 12 pile-uncopyrighted
LLM Original Surrogate
300
SAE jump_rewu
Effective Pank
U.Stabisti
400-
top,
300
200
104
10
Figure 22: Rank vs U-statistic, Gemma-2-26 webtext
Constant Sparsity Argument: U-Statistic vs Effective Rank gemma-2-2b Layer SimpleStories
5.5
Original
5.0
500
SAE ralu
LLM Surrogate
jump
SAE top_K
U-Statiaec
4.0
SAE batch top
Effective
300
3.0
2.0
10
10a
10
10a Token Position
Taken Position
Figure 23: Rank vs U-statistic, Gemma-2 2-2b, simplestories
Constant Sparsity Argument: U-Statistic vs Effective Rank gemma-2-2b Layer 12 code-10k
LLM Original nal
500
Surrogate
SAE re
SAE jump_ putu
Effecdive Rank
400
top_k
U-Stalistic
SAE batch_top.
300
304
Taken Position
Tokon Pasition
Figure 24: Rank vs U-statistic, Gemma-2-2b, code
26

---

Published as a conference paper at ICLR 2026
E FURTHER INVESTIGATION OF TEMPORAL SAES
In the results above, showed across three domains that Temporal SAES elicits intricate in-context geometry inline with our expectations based on experiments with LM activations in Sec. 3. Next, to make a case for following such more structured approaches for interpretability, we add the results above to show that Temporal SAEs (i) offers a new way of interpreting temporally structured domains (e.g., user-model chats) via the context- deeeedtte predictive component, and (ii) information that can be identified via SAEs continues to remain available within the novel component of Temporal SAEs.
E. GEOMETRY OF LATENT CODES AN OOD DIALOGUE DOMAIN
Activations
ReLU
Topk
BatchTopk
Temporal (Novel)
Temporal (Pred)
X
%
f 24.8
f = 32.2
29.1
== 26.5
== 30.0
== 2.0
User Assistant
Position 350
Figure 25: Population and trajectory geometry in OOD dialogue. UMAPs of raw activations and SAE codes on Ultrachat (first 350 tokens per conversation). Top: population embeddings colored by sequence position (gray gcolor) and marked by role (user=o, assistant=x)) only the predictive code cleanly separates speaker roles while preserving smooth temporal gradient. Bottom: a single conversation overlaid on the same UMAP; the predictive code follows smooth near-geodesic with low tortuosity т. in contrast to jagged paths from standard SAEs/novel codes.
We analyze multi-turn chat domain to test whether Temporal SAEs continues to reveal slow-moving structure. We emphasize the activations in this experiment are sampled from model, although all analyzed interpretability protocols were trained on the base (non-instruction tuned) model. Thus, in sense, the experiments in this section also gauge the ability of different protocols to generalize -utto--distribution (OOD)-a property SAEs have been argued to partially possess when transferring between base and the corresponding instruction-tune et al., 2024). Specifically, we use the Ultrachat dataset, where we take the first 350 tokens from 1,000 conversations (mostly single-turn within this window, with up four turns). For each token, compute a shared 2D UMAP embedding from (i) raw model activations and (ii) latent codes from ReLU, TopK, and BatchTopK SAEs, as well as the novel and predictive components of the Temporal SAE. To visualize population structure clearly, we display only 100 conversations at a time to avoid clutter (Fig. 25), though fitting applied to all 1,000.
2--mm----22-1
models (Kissane
At the population level, the predictive component of Temporal SABS is the only representation that cleanly organizes the data along two interpretable axes: (1) speaker-role separation, with user and assistant tokens forming distinct yet smoothly adjoining manifolds, and (2) a temporal gradient, with positions flowing continuously from early to late tokens along the manifold. By contrast, standard SAEs and the novel code show diffuse clouds with weaker role separability and no coherent positional gradient (Fig. 10, top row). To assess within-sequence geometry, we overlay a single conversation's 's trajectory onto the population embedding (Fig. 10, bottom row). Predictive codes trace smooth trajectory that remains locally coherent within each role segment and transitions gracefully at speaker switches. In contrast, standard SAES and the novel code produce jagged, zig zzgggingg paths. We again quantify this with tortuosity T (Bullitt et al., 2003) (lower is straighter): predictive codes achieve 2.0, whereas SAEs and the novel code yield 25-30, indicating substantially more local
27

---

Published as a conference paper at ICLR 2026
turning (Fig. 10. panel headers). This straightening mirrors the story- results (Sec. 6.1), now in an OOD conversational chat regime.
E.2 AUTOMATIC INTERPRETABILITY OF NOVEL CODES
Optimism
Manipulation
University
Slang
Temporal (Novel)
111-
11.00
(M83
Maller
-2h/9
14:15
Imbersiintt 43A
15.9
PAI
GemmaScope
6.45
1.2
Leaders whth
1kr 11%
WATH
WARIIC PAS1 NALIN TMr 116
Figure 26: Interpretability of novel codes. We qualitatively compare the interpretability of novel codes extracted using Temporal SAEs with latent codes extracted using GemmaScope SAES across four diverse categories. Example feature cards are shown across four categories, produced with the same activation -riiggerrd examples plus direct logit attribution workflow used for standard SAEs (Lieberum et al., 2024).
central benefit of SAEs is that their latents admit automatic interpretability pipelines ("autointerp") that attach luman-readabe semantics to features via dedivation-trigeeen examples, token/phrase saliency, and lightweight dashboards (e.g., GemmaScope / Neuronpedia). Our Temporal SAE adds new, temporally aware predictive axis, but crucially it does not sacrifice these established interpretability affordances: the novel code remains compatible with standard SAE autointerp workflows. In other words, Temporal SAES augment the toolbox rather than replacing it.
To support this point, we apply a conventional autointerp loop to the novel codes: (i) collect high- activation snippets and nearest-neighhor contexts for each latent; (ii) compute direct logit attribution to inspect which vocabulary items a latent tends to support or suppress; and (iii) render compact, browsable "feature cards" that aggregate examples and scores (as GemmaScope / Neuronpedia). While we do not perform an exhaustive nausal-intervention study here, nothing in the pipeline is specific to standard SABs; the same probes and dashboards operate unchanged on the novel codes, and we leave systematic intervention suite to future work. Qualitative inspection confirms that the novel codes recover the kinds of monosemantic, language-level features typically reported for SAEs. Manipulation,
Figure 11 (p. 13) shows representative examples across diverse categories-Optimiie University, and Slang-where where the novel latents consistently activate on intuitively relevant cue phrases and contexts, and where their attribution profiles align with the expected lexical sets.
Overall, when standard SAEs produce readable feature cards, the Temporal SAE's novel stream does as well. This complements our earlier results: the predictive stream captures slow, contextual structure (events, roles, long-range constraints), whereas the novel stream concentrates the fast, stimulus-driven information that matches inferences possible via existing SABS, including autointerp pipelines.
28

---

Published as a conference paper at ICLR 2026
F EMERGENT SEPARATION OF PREDICTIVE AND NOVEL PARTS
Predictive Code SimpleStories
Novel Code SimpleStories
M
2000
2000
Diciinary Indox
8000
4301
0000
0.1
$000
0000
3390
Plur Speroily PPustoer 20375
Sporatyy Nattan
6:0000 CURAUS
6400
Predictive Code
Webtext
Dersiy
EFfecius Rusic
Offccive Rans 416.5
Vean Spenits PPaitoor 28145
Mear Sperstyy Peetion 180.3
Predictive Code Code
Deniily
Novel Code Code
Dersily
Sonted Dictenary
0.1
3.9
(Peecian Blitic 20L1 Sparily (raaioon
Offective Faite 3005 Speratty Peataa
Aggrepeied Daraayy
Soquence Postzon
Denitty
Dersiy
Secuincce Position
Dereey
Figure 27: Separation of Predictive and Novel Dictionaries. Predictive and novel codes for sequences drawn from three Mettsetss-SimpleStonn are binarized to zero (black) and non-zero (white). The dictionary elements are sorted by the sum of non-zero activations in the predictive code across the sequence. The ordering obtained from the predictive code is also applied to the novel code. The red dashed line marks the separation of 90% of non-zero activation counts; only 10% are overlapping into the other subset. Additionally, the effective rank of the predictive codes is two orders of magnitudes lower than the average number of non-zero dictionary elements (Mean LO).
(top), Webtext (middle), and Code (bottom)-
The Temporal SABs studied in the main paper share dictionary for both predictive and novel codes. We now investigate if there is any shared structure in the two dictionaries. In particular, given the predictive and novel codes play different role, it is plausible the SAE learns approximately split the dictionary into two parts: one responsible for computing the predictive code and the other for novel one. Specifically, we investigate Temporal SAE trained on Gemma-2-2B Layer 12 residual stream activations. Results are shown in Fig. 27 and show separation of dictionary elements for sequences drawn from three datasets: SimpleStories, Webtext, and Code. Specifically, we find ~2K dictionary elements participate in defining the predictive code, while the remaining ~8K primarily participate in defining the novel code. However, the absolute count is merely indicative of the Lo sparsity of code, which may not reflect how many directions code- estimating latter requires computing the rank of the code (in fact, we note that spaksitasity is well-known alternative to L0 sparsity in dictionary learning literature (Elad, 2010)). As we show
actually used to define the
29

---

Published as a conference paper at ICLR 2026
in Fig. 27, the effective rank of the predictive codes (~20-30) is substantially lower than both the
effective rank (0:90--110) and absolute LO sparsity of novel codes.
Overall, the posthoe analysis above shows that predictive and novel codes largely separate subsets of dictionary clements. This emergent disentanglement of the two components motivates preemptively split the dictionary into two components one responsible for the predictive code and other for novel one. Our preliminary experiments show a Temporal SAE trained with this split-dictionary architecture similarly performant the tied dictionary one, resulting in predictive / novel codes with effective ranks ~20-30 ~~390-420, and a slightly better overall loss. It is possible optimizing hyperparameters (e.g., having different expansion factors for the two components) can make this architecture more performant than the tied dictionary one, but we leave further characterization future work.
30

---

Published as a conference paper at ICLR 2026
G THE GEOMETRY OF STORIES: A NARRATIVE-DRIVEN DOMAIN
G.1 HIERARCHICAL CLUSTERING OF CODES FROM STORY TOKENS
G.1.1 STORY
Activations
ReLU 108.6)
Topk (r 86.9)
BatchTopk 1069)
Temporal (Novel)
Temporal (Pred)
76.9)
(r 98.2)
(f 2.8)
80
60
40
20
There baby who wanted to pick something special. He went shop and
ancient case. He picked and ooked
inside. It was full unbelievably shiny gems and ewels. He couldn't believe his eves was so excited and he knew this case was perfect. He quickly picked it and wrapped with big red bow. He smiled and couldn't wait to show
and dad.
Figure 28: Geometry. Repeating results of Fig. again find smooth trajectories in UMAP projections for Temporal SAEs.
Temporal (Pred)
PED
sppeeal
paop
Appento
Temporal (Novel)
129
Figure 29: Dendrograms of Predictive and Novel Components from Temporal SAEs. Reproduc- tion of the results from Fig. 4b, but including the novel component's dendrogram as well.
31

---

Published as a conference paper at ICLR 2026
ReLU
22 on
Topk
BatchTopk
Figure 30: Dendrograms of Standard SAEs' Latents Codes. Reproduction of the results from Fig. 4b, but for latent codes extracted using standard SAEs.
32

---

Published as a conference paper at ICLR 2026
G.1.2 STORY 2
Activations
ReLU
TopK
BatchTopk
Temporal (Novel)
Temporal (Pred)
2V
60
40
There was kitty who liked to play with special brush. Each day, the Kitty would lend
thc brush friend in the park. Every day. thc
TBC
kitty was so careless with brush got lost. day, the kitty was playing
park when it saw special brush
with rabbit. The citty asked the rabbit if he could lend the brush. Vnva The aabbt said,
you want the brush back
Figure 31: Geometry. Repeating results of Fig. 4 on a different story, we again find smooth trajectories in UMAP projections for Temporal SAEs.
Temporal (Pred)
ang
Temporal (Novel)
Figure 32: Dendrograms of Predictive and Novel Components from Temporal SAEs. Repro- duction of the results from Fig. 4b, but on a different story and with both the predictive and novel component.
33

---

Published as a conference paper at ICLR 2026
ReLU
TopK
Boods
Topk
Figure 33: Dendrograms of Standard SAES' Latents Codes. Reproduction of the results from Fig. 4b, but for latent codes extracted using standard SAES on different story.
34

---

Published as a conference paper at ICLR 2026
G.1.3 STORY
Activations
ReLU
Topk
BatchTopK
Temporal (Novel) Temporal (Pred)
80
60
20
Once there was a girl. She garage. Inside the garage dhe kent her toys. One day, girl noticed something was wrong with her toys She went the garage and saw that they had broken. She
decided she would repair them and make them as good all toys until they
new. She made sure wash them all so they were nice healthy. Then the took some glue looked perfect.
Figure 34: Geometry. Repeating results of Fig. on a different story, we again find smooth trajectories in UMAP projections for Temporal SAEs.
Temporal (Pred)
Temporal (Novel)
Figure 35: Dendrograms of Predictive and Novel Components from Temporal SAEs. Repro- duction of the results from Fig. 4b, but a different story and with both the predictive and novel component.
35

---

Published as a conference paper at ICLR 2026
ReLU
RREEEEEE EEE
Topk
Bapc
adeand staees
Figure 36: Dendrograms of Standard SAEs' Latents Codes. Reproduction of the results from Fig, 4b, but for latent codes extracted using standard SAEs different story.
36

---

Published as a conference paper at ICLR 2026
G.2 CODE: ANALYZING ANOTHER DOMAIN
Activations (r 49.3)
ReLU (r 74.5)
TopK 46.9)
BatchTopk 218.4)
Temporal (Novel) (r 57.2)
Temporal (Pred)
5.4)
40
20
torch torch 0n Lnnearr 2,8,6 )
); torch
0ni iming uniform weight linear ity =t relu ") torch 0nm
ini O beros_ (m mm00 blas ): print ( 'niieed m weight
Figure 37: Code. Reproducing the results of Fig. 4, but on different domain, i.e., code. We again
temporally disentangled, smoothly running trajectory for latent codes extracted using the predictive component of Temporal SAEs.
Temporal (Pred)
Det
tod
tso mesus
can
lovi seea
tost
tep
end
19s2
Temporal (Novel)
100 un
Nee 0o0 the
1att hease
Ind Buss
Ind
lec
100
Figure 38: Dendrograms Predictive and Novel Components from Temporal SAFs. Reproduc- tion of the results from Fig, 4b, but on a different domain shows similar results as with narrative-driven text (stories) for both the predictive and novel component.
37

---

Published as a conference paper at ICLR 2026
ReLU
Get manggrer
West seaess
Topk
106
100)
BatchTopK
If
lo:
Joy touee
mes
Figure 39: Dendrograms of Standard SAE Latent Codes. Reproduction of the results from Fig. 4b, but on different domain shows similar results with perrative-drveen text (stories) for standard SAEs.
38

---

Published as a conference paper at ICLR 2026
G.3
FURTHER RESULTS: SIMILARITY MAPS ON STORIES
Temporal (Pred)
0.00
0.005
6 0.01
0.02
o= 0.04
Story 1
02
Story 2
72
Story 3
60
72
Figure 40: Temporal (Pred) Similarity Maps Elicit Multi-Scale Structure with Noise. Repeating the analysis shown in Fig. 6, we find the coarsening of temporal blocks is a robust result that continues hold for different inputs.
Temporal (Novel)
σ = 0.00
= 0.005
σ 0.01
σ = 0.02
0 = 0.04
Story 1
24-
Story 2
45-
72
48
Story 3
40
24
Figure 41: Temporal (Novel) Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, find the novel component is only able capture minimal local similarities, which when analyzed via dendrograms, show clustering based on lexical information.
39

---

Published as a conference paper at ICLR 2026
ReLU
0 0.00
σ 0.005
o 0.01
σ 0.02
0 0.04
Story 1
40-
72-
36
Story 2
12
Story
72
Figure 42: ReLU Codes' Similarity Maps under Noise. Repeating the analysis shown in Fig. on ReLU SAES, we find the ReLU latent code has high similarity across the board, suggesting lack of meaningful temporal information. This similarity is entirely removed when noise scale incraases too much, which, as shown in Fig. 6c, corresponds to the point that variance explained by ReLU SAES drops ~0.
40

---

Published as a conference paper at ICLR 2026
Topk
= 0.00
0.005
σ = 0.01
0 0.02
0.04
Story 1
72
Story 2
24
18
26
48
34
40
70
21
Story 3
Figure 43: TopK Codes" Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, we find TopK SAE's latent codes are only able to capture minimal local similarities, which when analyzed via dendrograms, show clustering based on lexical information. Akin to ReL.U SAEs, we see this similarity map approximately turns into an identity matrix when the noise scale increases too much, which, shown in Fig. 6c, corresponds the point that variance explained by TopK SAEs drops to ~00
41

---

Published as a conference paper at ICLR 2026
BatchTopK
0.00
0.005
0.01
0.02
0.04
Story 1
68-
Story 2
Story 3
72
72
Figure 44: BopK Codes' Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, we find BatchTopK SAE's latent codes are only able to capture minimal local similarities. which when analyzed via dendrograms, show clustering based on lexical information. Akin to ReLU SAEs, we see this similarity map approximately turns into an identity matrix when the noise scale increases too much, which, shown in Fig. 6c, corresponds the point that variance explained by BatK SAEs drops to ~0
42

---

Published as a conference paper at ICLR 2026
G.4 FURTHER RESULTS: DENDROGRAMS AND EVALUATIONS ON GARDEN PATH SENTENCES
G.4.1 DENDROGRAMS
ReLU
Topk
BatchTopk
Temporal (Novel)
Temporal (Pred)
Garden Path
The
conton
dlthing
made
The
Rappon
Up
The
comt
ccthir
Missip
maa
Control
The conton that
dooking
made
Mox Sere
Dappop
mm
apen
The
com
4
Descop
Figure 45: Example Sentence
Repeating the results of Fig. 7a on different garden path sentence.
ReLU
TopK
BatchTopk
Temporal (Novel) Temporal (Pred)
Garden
dignt
author
San
Whe
signed
author
Sone
deseree
Whon
the
the
dheered
cheened
auvor
the
tone
deoore
Control
Whan
Spoed
the
author
fans
When
the
sgned
autro
Farm
choored
When
the
sgredd
aathor Fame
cleored
Whon
ehe
aigeed
author
fans
dheored
When
the
wabor
sigeed
fars
deoored
Figure 46: Example Sentence 2. Repeating the results of Fig. 7a on different garden path sentence.
Garden
ghi
boy
commicd
kyhg
bog
byg
commoced
commrne bay
The
on
Comnocs
boy
kya
Control
gri
contoee
lyng
grd
Rat
controe
The
grl
conerood
the
Figure 47: Example Sentence 3. Repeating the results of Fig. 7a different garden path sentence.
43

---

Published as a conference paper at ICLR 2026
H FURTHER RESULTS: TEMPORAL SAES ON LLAMA-3.1-8P
We replicate a subset of the experiments evaluating Temporal SAEs on Llama-3.1-8B. The training protocol remains the same as that of Gemma-2-2B model: train on IB token activations with similar normalization schema, but activations are harvested now from Layer 15 (i.e., at ~00 of model depth). We note that the trained SAEs are not finished training, and henee the following results are solely meant to be an impression of whether the qualitative trends observed with Gemma models generalize to another model class.
H.1 GEOMETRY, DENDROGRAMS, AND SPECTRA
H.1.1 STORY
Activations
ReLU 95.9)
Topk (r 68.6)
BatchTopK 651.5)
Temporal (Novel) 79.55)
Temporal (Prec) 11.2)
55.2)
baby something special, shop and unbelievablly gamm and
case.
boked quidely
syes.
Figure 48: Geometry. Repeating results of Fig. 4, we again find smooth trajectories in UMAP projections for Temporal SAEs.
Temporal (Pred)
cose
Temporal (Novel)
Figure 49: Dendrograms of Predictive and Novel Components from Temporal SAEs. Reproduc- tion of the results from Fig. 4b, but including the novel component's dendrogram as well.
44

---

Published as a conference paper at ICLR 2026
ReLU
Topk
BatchTopk
02 dhi
OL prepo
Pount
le Aiun
Figure 50: Dendrograms of Standard SAEs' Latents Codes. Reproduction of the results from Fig. 4b, but for latent codes extracted using standard SAEs.
45

---

Published as a conference paper at ICLR 2026
H.1.2 STORY 2
Activations (r = 55.2)
ReLU (r = 95.9)
Topk (r 68.6)
BatchTopk (t = 651.5)
Temporal (Novel) = 79.55)
Temporal (Pred) (r === 11.2)
60
There was who liked to play careleesa
special br
Each day One city
10m
citty
lend
the brush to friend The
park, Every
citty wese
got
special brush
mnyy asked
you
Figure 51: Geometry. Repeating results of Fig. we again find smooth trajectories in UMAP projections for Temporal SAEs.
Temporal (Pred)
woua moo
Temporal (Novel)
Figure 52: Dendrograms of Predictive and Novel Components from Temporal SAEs. Reproduc- tion of the results from Fig. 4b, but including the novel component's dendrogram as well.
46

---

Published as a conference paper at ICLR 2026
ReLU
Topk
BatchTopK
Free
Figure 53: Dendrograms of Standard SA Es' Latents Codes. Reproduction the results from Fig. 4b, but for latent codes extracted using standard SAEs.
47

---

Published as a conference paper at ICLR 2026
H.1.3 STORY
Activations = 55.2)
ReLU 95.9)
TopK (r= 68.6)
BatchTopK 631.5)
Temporal (Nove))
Temporal (Pred) 11.2)
79.55)
Onca there was girl TDe had a garage Inside the garago Pan jept her toye, One day, the girl noticed something yone wrong with her toys tDg went the ginige and saw that they had broken t0 decided Chm would repair them and make them un good new. MMl madie sure to wish them so they were rico and healthy Then Bim look some glue and lood the tays until they looked perfect.
Figure 54: Geometry. Repeating results of Fig. 4, we again find smooth trajectories UMAP projections for Temporal SAEs.
Temporal (Pred)
Temporal (Novel)
Figure 55: Dendrograms of Predictive and Novel Components from Temporal SAEs. Reproduc- tion of the results from Fig. 4b, but including the novel component's 's dendrogram as well.
48

---

Published as a conference paper at ICLR 2026
ReLU
Topk
BatchTopK
A Hi
partoy
Figure 56: Dendrograms of Standard SAEs' Latents Codes. Reproduction of the results from Fig. 4b, but for latent codes extracted using standard SAEs.
49

---

Published as a conference paper at ICLR 2026
H.1.4
COMPARING EIGENSPECTRUM WITH SLOW Vs. FAST FEATURES
100
10"2
10"2
10-3
10
Magnitude
10-3
10a4
10
10-3
10-5
Slow Moving Rep.
10-6
Fast Moving Rep.
10-6
ReLl
Predicted Code
TopK
10-7
Novel Code
10
BatchTopK
10
10a
10a
10a
10a
Rank of Singular Value
Figure 57: Kernel spectrum for latent codes and model representations. Kernels defined using novel code from Temporal SAEs and standard SAEs both align well with the fast-changing part of model representations; meanwhile, only the predictive code shows strong similarity to the slow changing part.
50

---

Published as a conference paper at ICLR 2026
H.2 EVENT BOUNDARIES AND NOISE STABILITY
H.2.1 EVENT BOUNDARIES
(a)
Once time, little girl named Alice loved (b) looking at the night sky'l wish could count all
Temporal (Pred)
Novel
ReLU
the stars!' Alice said her friend Maya. The
two girls stood on big grass field as moon rose from the trees. Suddenly. Maya had striking idea. She opened her laptop and started yyping:vn "python array 6):Ankt in num_stars:"))nk aray.appenddjjlnt tot sum(array)Nn avg tot lenfarray\n print[f Avg / night: (avg:. 199
Topk
BatchTopK
for range(1,
00
(c)
Var. Explained 8
0.005
6 0.01
0.04
Tosponal BaichTupk Pred Novel
73
Rel.11
TopK
Within
0.55
0.28
0.25
0.13
0.62
Took
Accoss
001
0.02
0.03
004
0.05
23
46
70
23
46
70
23 46
70
23 46
70
Figure 58: Analysis of Event Boundaries. Reproducing results from Fig. 6 on see similar behavior the Gemma analysis done previously.
we
Temporal (Pred)
01,005
Story 1
Story 2
Story 3
Figure 59: Temporal (Pred) Similarity Maps Elicit Multi-Scale Structure with Noise. Repeating the analysis shown in Fig. 6, we find the coarsening of temporal blocks is a robust result that continues hold for different inputs.
51

---

Published as a conference paper at ICLR 2026
Temporal (Novel)
== 0.0
0.005
0.01
a= 0.02
o= 0.04
Story 1
Story 2
Story 3
Figure 60: Temporal (Novel) Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, we find the novel component is only able capture minimal local similarities, which when analyzed via dendrograms, show clustering based on lexical information.
ReLU
a= 0.0
0.005
== 0.01
6= 0.02
6= 0.04
Story 1
Story 2
Story 3
Figure 61: ReLU Codes' Similarity Maps under Noise. Repeating the analysis shown in Fig. on ReLU SAES, we find the ReLU latent code has high similarity across the board, suggesting lack of meaningful temporal information.
52

---

Published as a conference paper at ICLR 2026
Topk
0.0
0.005
0 0.01
0.02
0.04
Story1 1
Story 2
Story 3
Figure 62: Topk Codes' Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, we find TopK SAE's latent codes are only able capture minimal local similarities, which when analyzed via dendrograms, show clustering based on lexical information.
BatchTopK
a 0.0
0.005
0.01
6= 0.02
6= 0.04
Story 1
Story 2
Story 3
Figure 63: BatchTopK Codes' Similarity Maps under Noise. Repeating the analysis shown in Fig. 6, we find BatchTTopK SAE's latent codes only able capture minimal local similarities, which when analyzed via dendrograms, show clustering based on lexical information.
53

---

Published as a conference paper at ICLR 2026
1 FURTHER THEORY RESULTS
1.1 PRIORS ON THE SPARSE CODE FOR VARIOUS SAES
We restate and prove the proposition on independence priors of SAEs time (Proposition 4.1) below.
Proposition L1 (Independence priors time). Consider the SAE maximum aposteriori (MAP) objective for ReLU, JunpReLU, TopK and BatchTopK SAEs. The sparsity constraints for these SAEs are additive over time, resulting in:
arg min
Σ T
Dzill + AR(zi),
D.=
(8)
s.t. 2k = (a wk. g(21..... ,2) -r)== ((z)) O.
This MAP objective has an independent and identically distributed (i.i.d.) prior over time i.e.
T
P ZT) II
exp (-XR(z)) )ā(zz)) = II P(z.),
1=1
Proof. The sparsity constraints and specified in the table below.
regularizers for the SAEs under study are
Regularizer R(z:)
Sparsity Constraint
SAE
(Z
ReLU
| 1
JumpReLU
0
TopK
0
(122:|0 K)² D
BatchTopK
O
||z/|0
K
Table 5: Sparsity constraints and regularizers for SAES
Note that TopK imposes a pointwise hard sparsity constraint, which has been restated using sum-of-
squares above for convenience. While BatchTopK imposes the fixed mean sparsity for each mini
batch, we take the batch to capture the entire timeseries in the above formulation. The above table shows us that the sparsity constraint is additive over time in all cases:
(1 2T)
g(z,) O,
(9)
(||z|llo K)2
TopK
where õ(2,)
+ (11=:11 K)
BatchTopK
(10)
ReLU,, JumpReLU
Recall that SAEs solve the following constrained optimization problem (restated from Eq. 1).
arg min D.*
x. Dz + XR(z;),
N
(11)
s.t. 2%
faAE( Vk, g( xT) 0.
We rewrite the above problem using Lagrange multipliers on the sparsity constraints, and further simplify using the above result constraints being additive over time, as:
arg min
Σ N
1
Dz + XR(x,) + \0(1)=)1o))
D.*
(12)
s.t.
2 fau Yk.
Note that we don't 't use Lagrange multipliers on the SAE architecture constraint z = fsAE since only care about z-specific constraints (which don't ton't include the data a:) for the prior.
54

---

Published as a conference paper at ICLR 2026
Bayesian Interpretation. The objective function above (sans the SAE architecture constraint) can be thought of as minimizing the negative log posterior, which is proportional to log prior added to log likelihood:
1 N
log P(z1...., 25 . TT) x
D
XR(z;) + (0(1==)11)
log (2...,))
log P(21...,.77)
(13)
The prior over latents 2 is:
log P(z1.....,
AR(z;) + xo(1=z)lo))
(14)
T
2T) = II exp (-XR(z)) + )0(l = II P(z.).
(15)
Therefore, the prior is multiplicative over time, implying independence, and the distribution each time & has the same form, implying that the prior is independent and identically distributed (i.i.d.). This completes the proof.
1.2 PRIORS OVER CONCEPTS AND GENERATIVE PRIORS OVER TIME
We can further think of the priors of each SAE over concepts well
time in generative
fashion. In some cases, this mainfests as a hierarchical latent variable model n - S + 2, where ||=||o is the sparsity, 5 supp(z) (k: 2² 0} is the support, and is the SAE latent code. Proposition 1.2 (SAE Priors on Sparse Code). Let St supp(z;) = (k: z > 03 be the set of time t, and n4 Sy be the cardinality of St (the number of active latents). Each SAE imposes a prior distribution on the sparse code z, arising from its sparsity penalty R(z) or implicit conditions imposed on the sparse code. These conditions are highlighted in Table 6.
active latents in the sparse code
Table 6: Priors over concept interactions and dynamics for various SAFs
SSar, R(z)
Across-Concee Prior (interaction)
Across-time Prior (dynamics)
ReLU,
M iid. ~ Laplace(O.))
iid. ~
Li-norm
2
P=
K
St ~ U(O,-) :) Vi. E St,
(21, S1) (2,, S.) Lid
PsPaIS:
TopK
Se U(M)
S S. U1 U(
JumpReL.U,
St UA UO,,-) Vi. E E.,
(21, S1, n1). (ze, St, me) lid Pa Ps P(z (S).
Lg-norm
Se ni ~ ![
n1, 7i....; ne the P.
inc
St ui U(0.-) Vi. E S.
(1,, (1,
(2), St, ni)
lid. PaPs P(z S),
BatchTopK
S. ni "(M)
n1,
Pn, B[nz] = K
1.2.1
RELU SAE
The vanilla ReLU SAE (Bricken et al. (2023), Cunningham et al. (2023)) is trained with the L norm penalty:
R(z) = ||=||1-
(16)
The prior over for the above case is:
N
M
log
nN) x
|
(17)
km
N
M
P(Z),...,ZNN)
2N)
Il
exp -2|=
(18)
55

---

Published as a conference paper at ICLR 2026
This joint distribution implies that for each sample i, different indices it are sampled i..d. from the same distribution:
2/
Lid Laplace(0. 1/w),
(19)
and different samples are all independently sampled from the same product Laplace distribution:
2 NN iid Laplace (0, 1/v).
(20)
This concludes the proof for priors of ReLU SAE trained with L norm sparsity penalty,
1.2.2 TOPK SAE
The TopK SAE (Makhzani and Frey (2013), Gao al. (2024)) directly controls the sparsity of the representation by fixing it at ||=100 = K, instead of imposing an explicit sparsity penalty R(x) in the loss function. The objective function for TopK SAE is:
arg min D.
o. Dzil
(21)
Vj. 2g fToPk (Ir,), 1zy K.
(22)
Since the fixed sparsity is hard constraint that depends on z alone (and not the data æ), it can further be simplified as a sum-of-squares constraint: (1(1= K)² 0. We can use Lagrange
multipliers reformulate it as an effective prior:
arg min Σ D22 ē=1
||:; - Dzi 12 + (11 -x1=)
(23)
N
s.t. jj (xz))
(24)
The prior over for the above (effective) regularizer is:
(1122110o - x)²)
log ((Z)....,NN)
(25)
1=1
P(zi,...,, 2N)
exp
((12z K)3)
(26)
Note that the above prior is finite for finite values of X, but the overall objective optimizes over X. resulting in a hard prior peaked at ||2,|lo = for each sample
The factorization over samples implies mutual independence of 21...., 2n: P(z 2n) a P(z.).
As defined in Theorem I.2 (and restated here for convenience), let S; = supp(z;) {k 2/
0}.n: = |S | denote the active indices and their number (sparsity) respectively.
For individual samples zt. we condition on the set of active indices S, the sparsity gets fixed since ||2/|lo = J.S.| = n4s and the distribution becomes constant:
P(z. | S.) =C
(27)
((),k)
J E S;
2 S 80 μ $ S,
and
(28)
M S lid U(0, k) for μ. S₁
(29)
where C, appropriate constants.
Since { are mutually independent, any measurable function of each is also independent. The indices of nonzero entries of zj. i.e., Sj measurable function since it is map S : IRM 2M
56

---

Published as a conference paper at ICLR 2026
which is discrete valued, and pre images of each value- set of nonzero indices measurable since they equal the cartesian products of the measurable sets {2 = 0}, fa > 0} over all indices. Hence, St. ................................ S,, are also independent.
Since S; = g(z,) and the distribution z; depends only on n; = ||2 (Eq. 25), the distribution of St will also depend only n4, becoming uniform when conditioned nı, In TopK SAE, K a constant. Therefore, each S; ~ U[M and together with independence argued above,
S S SN iid )(M
(30)
This completes the proof for the priors of TopK SAE.
1.2.3 BATCHTOPK SAE
BatchTopK SAE (Bussmann al. (2024)) is modification of the TopK SAE. Instead of fixing sparsity like TopK, BatchTopK allows variable sparsity per input while fixing the mean sparsity over batch at K. The objective function for BatchTopK SAE can equivalently be written
N
arg min
Dzil
(31)
D.
s.t. Vj. fropk (x,),
|2,11o K. Σ
(32)
While BatchTopK imposes mean sparsity per batch, for simplicity, we use the batch size to match the size of the entire dataset (WLOG). Smaller batch sizes can easily be incorporated by adding separate constraints, each over the entire batch (only leads to a change in constants lagrange multipliers in the analysis).
Following similar analysis as for TopK SAE (App. I.2.2), we can derive an equivalent prior over for BatchTTopK SAE:
N P 2N) II 1=1
exp ( A1112 -K])
(33)
The sparse codes for different samples { are thus sampled i.i.d. from a distribution that only depends on the sparsity penalty. While this prior looks very similar the prior of TopK SAE, the difference is that in TopK, the fixed sparsity constraint is imposed per sample, leading to a different Lagrange multiplier X; per sample optimize over, while in BatchTopK, we have a common multiplier À over all examples in batch (with multiple batches, we will have one multiplier per batch), which is then optimized over to ensure that average sparsity per batch constraint is met.
Similar to the analysis for the TopK SAE, we get the following prior over different latents per sample:
U(0, K)
J E S. # E S,
z! S
and
(34)
80
s S lid U(0, k) for μ. E
(35)
The active indices S, are sampled uniformly conditioned on the number of active indices ny:
S n4 U{[
(36)
The number of active latents n, themselves sampled i.i.d. (since n, Ō(2,) and {2,); are i.i.d.)
from a distribution whose mean is fixed:
71.
AN
lid.
P. s.t. F[n.] K
(37)
This completes the derivation for the Batc prior.
57

---

Published as a conference paper at ICLR 2026
1.2.4 JUMPRELU SAE
JumpReLU SAE (Rajamanoharan al. (2024)) is trained with the Lo (pseudo-)norm regularizer. This leads to the following optimization problem:
arg min
(1|= - D2il2 1122 + Al|zx(lo)
D
S.L. Ak, 2k = Ratpp (ax)
This objective is equivalent to the following prior over
N
P(z1..... 2N) & II exp (- 1-11 7212111))
(38)
Noting the similarity with the Topk/ BatchTopK cases, following conditions:
the same analysis to derive the
2 S (U(0,))
" E S
and
(39)
60 l $ S.
M151 S U(0, R.) for μ. E ES 8: n, ~ U(M))
(40)
(41)
The number of active latents n, are again i..dd, but there is no constraint on the mean of the distribution (unlike BatchTopp which constrained the mean of n, to equal K):
71.... TNN P.
(42)
which completes the analysis for JumpReLU SAE.
J STATIONARITY MEASURES
LLM activations empirically non-stationary across the sequence. We quantify the non-stationary nature by measuring autocorrelations and the U-statistic.
J.1 AUTOCORRELATION
We compute autocorrelation by selecting evenly spaced tokens across the sequence and measuring the cosine similarity between each token and tokens various lags the past. Specifically, for tokens at position 4, we compute similarities to tokens at ( - where lag ranges from 5 to 20. This creates heatmap where rows represent lag offsets and columms represent token positions.
For stationary process, expect the autocorrelation pattern to remain consistent across time-tth is, the relationship between a token and its historical context should be similar regardless of position in the sequence. This would manifest similar autocorrelation patterns repeating horizontally across token positions. In contrast, for a non-stationary process where representations evolve over time, we expect the autocorrelation patterns to vary systematically across positions, with columns showing different temporal dependency structures as the sequence progresses.
J.2 U-STATISTIC
We measure the effective dimensionality of LLM representations using U-statistic based pairwise cosine similarities.
2 - M
U-stat(t) ||G 11 M
(43)
where ||Grl] is the squared Frobenius norm of the Gram matrix. This quantity estimates the effective rank l/tr(C?), where C, E[ is the second moment matrix and & is the normalized activation vector at time t. Under stationarity, U-stat remains constant. When representations evolve over time, U-stat increases systematically as more orthogonal directions become active.
58

---

Published as a conference paper at ICLR 2026
J.3 SURROGATE
For the U-statistic and Autocorrelation metrics, we compare LLM activations to surrogate distribu- tions that preserve certain statistical properties while removing temporal structure. We operate representation vectors E BxTxx where B denotes batch size, T denotes sequence length, and denotes the model dimension.
U-statistic surrogate (Fig. 2 a, e): For each sequence i E {l,..., B3, we construct the surrogate X by applying a random permutation Ty fl....,II ,) + f1...., T) to the temporal positions:
X (t)) Vt E [I,...,
This preserves the marginal distribution of activations within each sequence while destroying temporal dependencies.
Autocorrelation surrogate (Fig. g): Given the similarity matrix S E RTXT where Sij
sim(X,ii, ,i,, x. construct the surrogate similarity matrix $ by replacing each diagonal with its mean:
S. Sk where k = -i--ll 51. S2
Σ
St,+
T
t=1
This preserves the average correlation structure at each lag while removing patterns.
temporal
59

---

Published as a conference paper at ICLR 2026
K
USE OF LARGE LANGUAGE MODELS
LLMs were used in this work for the following:
Polish writing: Although major parts of the writing were done by the authors themselves, LLMs (ChatGPT) were used to critique and iteratively improve the writing.
Research ideation: In the initial stages of the project, conversations with LLMs (ChatGPT, Gemini) aided in refining the overall storyline of the project, as well as to get feedback on theory sections. In all cases, LLM outputs were only used by the authors refine their ideas.
60


Reviews:
[{"review_id": "teZt7rCayz", "reviewer": "Reviewer_QXh2", "summary": "The authors first provide a formal characterization of standard SAEs, arguing SAEs impose an implicit prior that assumes concepts are independent and identically distributed (i.i.d.) across time. They contrast this with an empirical analysis of LM activations, which reveals rich temporal dynamics, including non-stationarity and strong context-dependent correlations. To address this mismatch, the authors introduce the Temporal SAE, a novel architecture with a temporal inductive bias. This model decomposes activations at each timestep into two components: a predictable component ($x_p$), inferred from the context, and a novel component ($x_n$), which captures the residual information. The authors demonstrate that this decomposition allows the model to separate slow-moving, abstract information (in $x_p$) from fast-moving, novel information (in $x_n$). They provide evidence that this model is better at identifying temporal structures, such as event boundaries and correct garden-path sentence parses, which standard SAEs fail to capture.", "questions": "1. *Shared vs. Separate Dictionaries:* The model shares a single dictionary $D$ for both the predictable ($z_p$) and novel ($z_n$) components. Given that these codes are purported to capture fundamentally different types of information (slow vs. fast, abstract vs. transient), what is the justification for sharing the dictionary? Were separate dictionaries/encoders explored?\n2. *Causal Utility of Features:* As mentioned in the weaknesses, the \"gold standard\" for interpretability is often causal intervention. Have the authors performed any experiments (e.g., activation patching, model editing) to demonstrate that the $z_p$ and $z_n$ features are more causally effective or localized than features from a standard SAE? Demonstrating this on the garden-path sentence examples would be particularly compelling.\n3. *Quantitative Validation of Event Boundaries:* The hierarchical clustering results in Figure 5 are visually interesting, but can these results be validated more rigorously? For example, by computing a quantitative metric (e.g., F1-score, ARI) for event boundary detection against the human-annotated ground truth in datasets like those cited (Zacks et al., 2007; Baldassano et al., 2018)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. *Formal Characterization of SAE Priors:* The paper provides a clear and valuable formalization of the implicit assumptions in standard SAEs. Characterizing the priors as i.i.d. across time (Proposition 3.1) 6cleanly articulates the central problem.\n2. *Strong Empirical Analysis:* The claims about the data's temporal structure are well-supported by a robust empirical analysis. The use of multiple metrics, including intrinsic dimensionality (U-statistic) and autocorrelation, demonstrates the non-stationarity and context-dependence of LM activations (Figure 2) that hold back traditional SAE-based approaches.\n3. *Clarity and Intuition:* The paper is very well-written and intuitive. The core problem and the proposed solution are explained clearly, making the work easy to follow.", "weaknesses": "1. *Evaluation Relies on Correlational Evidence:* The primary evidence for the Temporal SAE's improved interpretability is qualitative (dendrograms in Fig. 5, 7) or correlational (CKA with slow/fast Fourier signals in Table 3, Fig. 6). While standard metrics like reconstruction (Table 1) are included as sanity checks, the paper never demonstrates that the disentangled $z_p$and $z_n$ features are _causally_ more useful than standard SAE features at steering model behavior.\n    - The qualitative claims about event boundaries (Fig. 5) are subjective and would be much stronger if validated quantitatively against human-annotated datasets, such as those the authors already cite (e.g., Zacks et al., 2007; Baldassano et al., 2018).\n    - A stronger claim would require causal validation. For instance, are $z_p$ features more effective for patching high-level topics between texts? The paper claims $z_n$ captures \"novel\" information; does this imply $z_n$ features might be better targets targets for model editing? A demonstration that interventions on $z_n$ are more localized and effective than on standard (entangled) $z$ features would provide stronger causal evidence.\n2. *Novelty =/= Un-predictability*: The paper's decomposition rests on an assumption that \"novel\" information is orthogonal to \"predictable\" information. This is questionable, as linguistically or cognitively \"novel\" information (e.g., the start of a new topic) is not always unpredictable. In fact, research on human event memory (which the authors cite) suggests that event boundaries are registered at predictable changes in context, not just at moments of high prediction error. This creates a potential mismatch: the model's \"novel\" component may only be capturing an unpredictable residual, rather than the more structured, and sometimes predictable, \"novelty\" that defines temporal structures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pjA3PvAr6t", "reviewer": "Reviewer_r8pv", "summary": "The paper points out current SAE are built on a flawed assumption. They adopt a Bayesian view and show that standard SAEs implicitly impose a \"prior\" that the concepts they extract are independent and identically distributed across time. However, language is full of temporal structure and correlations. Authors introduce a new architecture, the Temporal SAE, which decomposes the activation into two parts: A predictable component and A novel component.", "questions": "1. What is the difference between your method: \n\"input a sentence -> find the novel component at each token -> convert this novel component into human understandable concept\",\nand method where you just input tokens one by one, and convert them into human understandable concept?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The topic of decomposes the activation into two parts is interesting.\n2. The core problem and flawed assumption that standard SAE has is well introduced.\n3. The proposed method is interesting.\n4. The temporal SAE can successfully identify concept that standard SAE can't", "weaknesses": "1. The author claims their temporal SAE is better than standard SAE. However, sometimes it is really difficult to evaluate an SAE, since there is no \"ground truth\". Therefore, the author should evaluate their SAE on some downstream tasks such as model steering. \n2. What about the interpretable concepts in the predictable component? Why are we not finding interpretable features in predictable component as well?\n3. The paper never evaluates its new features on any of downstream tasks. The evaluations are purely analytical. The paper only shows that its features correlate with these temporal structures, but it never proves that these features are more useful. The big \"so what?\" question is unanswered.\n4. This paper is built on the assumption that the novel component is orthogonal to the predictable component. However, sometimes new information doesn't just add a new, independent concept; it often refines or modifies existing, predictable concepts. Plus, I didn't see any citation prove this assumption in the paper.\n5. The paper miss some important related work.\n\n[1]: A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V3CRfmBQXl", "reviewer": "Reviewer_mbJz", "summary": "This paper examines the implicit temporal assumptions underlying Sparse Autoencoders (SAEs) widely used in mechanistic interpretability. It reveals that the conventional SAE training objective imposes an implicit independence condition on the prior it assigns to the distribution of feature activations across different token positions—an assumption inconsistent with the highly correlated and nonstationary nature of language sequences. Through empirical analyses of Llama-3.1-8B and Gemma-2-2B, the authors show that the dimensionality and autocorrelation of token representations evolve systematically with position. To address this, they introduce the Temporal Sparse Autoencoder, which decomposes each activation into a predictable component derived from past context and a novel component for the current token capturing new information, akin to the innovation concept in signal processing. The Temporal SAE uncovers temporally structured linguistic features—such as event boundaries in narratives and reanalysis in garden-path sentences—that standard SAEs fail to reveal, underscoring the need for temporal inductive biases in interpretability methods.", "questions": "1. What are $W$ and $L_0$ in Table 1?  \n\n2. What is the off-the-shelf algorithm used to extract hierarchical structures from the correlation plot, and how does it work?  \n\n3. Do the “predictive codes (and novel codes, respectively)” frequently mentioned in Section 6 refer to the SAE activations $z_{p,t}$ or to $x_{p,t}$?  \n\n4. Could you provide an (at least intuitive) explanation for why the slow-moving part of the representations (and the predictive codes) exhibits a faster spectral decay?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The primary advantage of this paper lies in its thoughtful incorporation of linguistic insights into the analysis, design, and evaluation of SAE methods and LLM representations. The empirical experiments in Section 4, presented in Figure 2, clearly show that LLM representations of token sequences conform to the non-stationarity of natural language, featuring continuous increments of new information regulated by underlying dependency structures. The evaluation of the proposed Temporal SAE on garden-path sentences is both novel and insightful, as the dependency parsing of such sentences is a classic topic in linguistics. The superior performance of the Temporal SAE on these examples effectively demonstrates the benefits of explicitly incorporating temporal and linguistic inductive biases into the interpretability framework.\n\n2. The spectral analysis of SAE and LLM representations at the end of Section 6 is particularly insightful and adds an important dimension to the depth of the work. It convincingly shows that the decomposition of LLM representations by the Temporal SAE into predictable and novel components successfully captures the bipartite structure of representations—comprising a relatively slow-varying component and a more dynamic component with respect to the introduction of new tokens. This analysis reinforces the interpretability and mechanistic validity of the proposed temporal modeling approach.", "weaknesses": "1. One of the primary motivations (and functions) of the original SAE is to decompose the polysemous LLM representations into distinct monosemantic features that are more interpretable and less ambiguous, which can further be utilized in steering experiments. This paper, however, makes no attempt to semantically interpret the learned features in this regard—likely because the architectural design of the Temporal SAE, where the feature activations of the predictable part are obtained via an attention mechanism, complicates the interpretation of both activations and activation features.\n\n2. The paper suffers from a lack of clarity with respect to concepts and notations in certain parts, which I list below.\n\n- **Equation 2**: The normalization factor should be $\\frac{1}{T}$.\n- **Section G.2, Equation 40**: $M$ is not defined, which seems to refer to the number of tokens. For the U-statistic to correctly estimate the effective rank, it appears that the model representations must be normalized; however, this is not stated.\n- **Line 291**: The authors claim $x_{n,t} \\perp x_{p,t}$, but this is not structurally guaranteed by the Temporal SAE (e.g., via a projection-based approach). The authors themselves acknowledge in Line 338 that the orthogonality is only approximate. They might consider rephrasing the argument in Line 291 for precision.\n- **Predictable component definition**: The authors state that they use a self-attention layer on top of a single ReLU layer to model $f$ for obtaining the predictable component. However, Figure 3 suggests that the attention layer depends on the SAE activations $v_1, ..., v_t$, and the paper does not specify how these activations are obtained.\n- **Baselines in Figure 2(d) and (h)**: The authors provide a baseline for their variance preservation experiments using representations of a certain number of nearest token representations. However, the method for computing this baseline is never stated (presumably using an equal number of random directions). I also recommend adding another baseline using projections based on the first $k$ tokens’ representations in the context, which could further substantiate the authors’ claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)