Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
Authors: Mark Rofin
Jalal Naghiyev
Michael Hahn
Venue: ICLR 2026 Poster (2026)
Keywords: next-token prediction
transformers
interpretability
Primary area: interpretability and explainable AI

Abstract: Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which components of the gradient signal from the next-token prediction objective give rise to this phenomenon, and we propose a method to estimate the influence of those components on the emergence of specific features. After validating our approach on toy tasks, we use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model.  Finally, we apply our framework to a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes a step toward understanding hidden features of Transformers through the lens of their development during training.

Published as a conference paper at ICLR 2026
UNDERSTANDING THE EMERGENCE OF SEEMINGLY USELESS FEATURES IN NEXT- TOKEN PREDICTORS
Mark Rofin': " Jalal Naghiyev2.) Michael Hahn* School of Computer and Communication Sciences, EPFL Technical University of Munich, Zuse School ELIZA
Saarland University Campus, Saarland University
ABSTRACT
Trained Transformers have been shown to compute abstract features that appear redundant for predicting the immediate next token. We identify which compo- nents of the gradient signal from the next-token prediction objective give rise to this phenomenon, and propose method estimate the influence of those components on the emergence of specific features. After validating our approach toy tasks, use it to interpret the origins of the world model in OthelloGPT and syntactic features in a small language model. Finally, we apply our framework a pretrained LLM, showing that features with extremely high or low influence on future tokens tend to be related to formal reasoning domains such as code. Overall, our work takes step toward understanding hidden features of Trans- formers through the lens of their development during training.
INTRODUCTION
Large Language Models (LLMs) are usually pretrained with the objective of next-token prediction (NTP). In this paradigm, a model learns to predict each token in a sequence given all previous tokens: in other words, learns the distribution p(æ++1 ........)) xe).
Thus, the model is incentivized to compute features that help predict the immediate next token. Hence, one could reasonably expect that the hidden representations at position 1, computed by model trained in this way, would contain only the information relevant for predicting T++1- On certain synthetic tasks, this was found to be true, highlighting a downside of NTP as a training objective (Bachmann & Nagarajan 2024 Thankaraj et al. 2025
However, a growing body of work on LLMs (and -rantreined Transformers in general) shows that sometimes they learn much more than that. For example, Transformers reconstruct abstract features of the input text (Templeton et al., 2024] Park et af., [2024), infer the high-level structure of the processes generating their training data, forming "world models' (Li et al., 2023] Karvonen, 2024
Shai et al. [224]] Jin & Rinard 2024 Gurnee & &Tegmark] (023),, or implicitly predict the sequence
multiple tokens ahead (Pal al. 2023 Jenner et al. (024). Motivated by these intriguing findings,
we ask:
How do Transformers trained for NTP learn features that don't help in the prediction of the immediate next token?
The prior work investigating learned features in Transformers mostly employed a teleological per- spective: that is, features are viewed in the context of their role the algorithms implemented by a trained model (c.g., Ameisen et al. (2025); Arditi et al. (2024)). This approach is useful to find the circuits encoded in L.Mss, but it doesn't 't tell us much about the gradient signal that causes those circuits to develop during training. Thus, the ways of how training for NTP drives the emergence of features has been largely underexplored so far.
Towards closing this gap, we develop novel view features learned by Transformers. Based on the structure information flow in causally masked Transformers, we show that features can
Correspondence mark. rofin@epf1.ch

---

Firstly, a gradient signal can come from the immediate next-token prediction (direct learning). This includes all paths passing through (x) and 2 and represents the effect of the information encoded in (x) on the prediction the immediate next token. These paths are colored green in Published as a conference paper at ICLR 2026
in principle be learned by three distinctive mechanisms, which we refer to direct learning, pre- caching, and circuit sharing. The two latter ones allow the token distribution at positions > i to influence the model's representations at position i, unlocking the learning of "useless" features. Next, for a given feature, we propose an experimental method to classify it depending on which mechanism contributed the most to its development. We then our framework to understand the learned features in Transformers trained on different data domains, including toy functions, the board game of Othello, and language.
Our key contributions include: (i) A theoretically grounded explanation for why Transformers trained for NTP learn complex features that are not immediately helpful; (ii) An approach for tracing the gradient components of the NTP objective that led to the development of given feature in model; (iii) Novel findings obtained using the proposed framework, including an explanation of the OthelloGPT world model fragility, inspection of the role of pre-caching in text generation, and interpretation of the pre-cached features in L.M.
SETUP
We use X' denote variable-length input space of discrete token sequences 21 ....nn We view model T's as representing function In + 22
.....11111 such that
(2) (r()
(x) = hz.2,)
(x) (r
k-1 ...., ), k >
and r (x) E IRd. Here h2 and h are embedding and unembedding layers, respectively, r (x) the values of the residual stream, and h2 are Transformer blocks.
We call learned feature any linear component of the residual stream at a specific layer and position (w/ (z)), where a vector w E Rd defines the feature direction. We informally call feature of sequence z NNTPuseless if there exists optimal next-token predictor for z; that doesn't 't compute it. Otherwise, we call that feature NTPPusefull For example, NTP-useless features can include the positioning of the board game pieces that don't affect the set of possible next moves, the surface properties of the sequence that are irrelevant to its continuation. The central question of our work can then be formulated as understanding how NTP-useless features emerge in Transformers.
INFORMATION FLOW DECOMPOSITION
3.1 GRADIENT DECOMPOSITION
We fix position and layer l and study all information paths in the computational graph of the model, classifying them by how they relate rx,,(x), We argue that the gradient training signal can flow to 0 through three types of paths, illustrated in Figure
Figure I We formalize this by comparing the overall gradient with gradient after stop-gradient operator is applied:
V (direct)
Vol. V VoI.s(k..)
Secondly, the information encoded in rx)(()) affects the loss at positions j > because attention heads at those positions can attend to position i. Thus, a gradient signal can come from prediction loss at future positions, after passing through one or more attention operations (pre-caching). This component includes the paths passing through rå, and ã, for j > i + (blue in Figure I):
VoL h
Vo Σ
L, 1, (k.6(k)))
(pre (pre-cached)
jtf
Third, there are paths that do not pass through r (x) at all. Hence, some gradient signal arises inde- pendently of r..(()), Since Transformer blocks use the same parameters perform the computation
2

---

Published as a conference paper at ICLR 2026
you
need
1s
love
unoved graderm
-M Mudder
Tean
Iye-sded
you
Reed
1s
fesure whmstch
An illustration of the information-foow decomposition for 2 and = into direct, pre-cached, and shared components. Direct and pre-cached paths must pass through the residual stream at position i = and layer = 1: any other path is considered shared. The turquoise rectangle indicates r 22 Right: by decomposing the loss gradients, in Section 3.4 we partition the improvements in feature linearity at each training step.
Figure Left:
at every position, this signal may influence the parameters computing it. We call this phenomenon circuit sharing and visualize the related paths in orange in Figure T
V L (shared)
Σ V VaL VaLz (k.)))
=
The term pre-caching is borrowed from Wu et al. (2024), who studied it as a possible explanation for look-ahead in LLMs. We discuss the relation of work to Wu al. (2024) in Section
These three components provide an exhaustive decomposition of the gradient signal: Proposition 3.1 (Loss gradients decomposition). For any layer k and position
VeL Vol; (direct) + VoL; (pre-cached) + VoL; (shared))
By Proposition 3.1 for each i and k, the total gradients that are backpropagated to the model param- eters after computing the loss on one training batch can be split into three terms distinctive their nature: direct, pre-cached, and shared components.
How study pre-caching and circuit sharing? In light of Proposition 3.1 natural question arises: how important each of the introduced components is for learning the task, well as for representing the latent features picked up by the model. The role of the direct component sense trivial: that is the main source of the gradient signal for predicting the immediate next token, thus we concentrate on analyzing pre-caching and circuit sharing. We approach the analysis by two complementary ways: intervention (training model with one of the components ablated, Section and attribution (quantifying the influence of each component in a training run, Section 3.4.
some
3.2
3.2
ABLATING PRE- CEECCACHING AND CIRCUIT SHARING
Ablating pre-caching. Wu et (2024) proposed myopic training - a way train an LLM that prevents pre-caching. The only difference between myopic and normal training is that all gradients between the loss the i-th position and the activations the j-th position (where j A i) are blocked. Since the only path for information to flow between the i-th and j-th tokens is through attention, it sufficient to stop gradients after computing the K and V matrices (excluding those for the current token). The purpose of myopic training is to block gradients so that the i-th token is not incentivized to compute any feature that NTP-useless for predicting # but useful for being picked up by an attention head later. This way, pre-cached features do not appear.
blating circuit sharing. To stop learning features shared across positions, we use a technique that we call m-untied training. We select an index and use one set of parameters (0gm) for the forward pass on all positions before m and another (f>m) for positions after m. This way, we have two models that are trained only to predict their respective part of the input, but the second model still attends to the KV-cache of the first one. In the extreme m-untied + myopic case, the pre-cached gradients do not flow through attention, so the parameters 05m do not depend on the data after the
3

---

Published as a conference paper at ICLR 2026
m-th token. In this case, double model resembles patient with non-communicating left and right brain hemispheres, as in the split brain experiments neuroscience (Gazzangaa 2005
3.3 THE ROLES OF PRE-CACHING AND CIRCUIT SHARING
Pre-caching increases expressivity. The power of Transformers comes from complex interactions information between different tokens. Even constructions simple as an induction head require at least two layers of attention interacting with each other. Disabling pre-caching prevents the Transformer from deliberately learning such constructions. Indeed, if feature is NTP-useless at all positions, there is no hope for it to be learned and used by later layers. Thus, many constructions requiring more than one layer of attention (such as the ones described by Liu et al. (022)) will be impossible to learn.
of attention heads that
Note that circuit sharing, by contrast, does not have this property, since even with untied training the gradient signal between tokens still passes through. Compared to myopic training, where an optimal solution cannot reached due lack of training signal, SGD in the case of untied training has the signal needed to find the minimum.
Circuit sharing enables cross-position feature transfer. While the strength of pre-caching is increased expressivity, circuit sharing has a different unique property: enabling feature transfer across positions. Imagine a feature that is NTP-useless at position i but NTP-useful at position j. Due to its usefulness j, it will be learned, and thanks to circuit sharing, it will also be encoded at i. In this way, position i will have access to knowledge learned at a different position.
3.4 ESTIMATING THE EFFECT OF GRADIENT COMPONENTS ON FEATURE EMERGENCE
So far, we have argued that there are three path types along which the loss signal, via its gradient, can pass to the model parameters. We now use this decomposition feature is produced, over the course of training. by each of the three components of the gradient. To this end, we aim to quantify how much each component of the gradient signal pushes the parameters towards developing a feature.
study the extent to which
Definition 3.2. We call feature mismatch the value
R(x 01, 02 w!t) = (ww) n (z)) (wt (
The feature mismatch quantifies how much the projections of the residual streams onto the feature w differ between models parameterized by 0₃ and 82.
We now want to quantify the extent to which single gradient update to an intermediate checkpoint narrows the feature mismatch when compared to the final checkpoint& 6* By separately consider- ing the three components of the gradient signal, we will be able to understand what role each plays in the development of the feature. We formalize this terms of an influence T(0, x. y w 0r G):
Definition 3.3. For vector G E IRl we call the influence of G the value
d - lx 1 eG, w
(0, wf or, G)
de
Applying the decomposition from Proposition 3.1 for each feature we define direct influence:
ddirect(" 0) = (0, x w/ 0*, VoL. (direct)).
The definitions of pre-cached and shared influences are analogous.
Remark 3.4 (informal). Consider a model To, trained for M steps of SGD with small step size n Then
M
R(x 0o, w) Rd "
I(0r, xe w 0", VoL.),
8ES
where S (direct, pre-cached, shared).
4

---

Published as a conference paper at ICLR 2026
Probing Performance
1.0
Probing Score
1.0
0.8
0.8
Non myopie, tied
Non myopic, tied
0.6
Non-myopis untied Myopic, tied Myopit, unlied
Non-myypiic untied Myopie, tied Myopie, unlied
0.4
0.2
6
14
16
18
2
6
10
14
16
18
Sequence Position
Sequence Position
Figure 2: Performance of the linear probes applied to the residual stream after the first Transformer block. Left: Majority, the feature "majority-so-far Right: Conditioned Majority, the feature prrevious-token.. In both cases, the feature is NTP-useless until the 10th token. In both cases, ablating are-caching and circuit sharing substantially hurts the probing performance.
Remark 3.4 holds approximately due to the first-order approximation of the feature mismatch R(x O,, 0,,0,w) w). Expressing the change feature mismatch at each step through its gradient and breaking it down into direct, pre-cached, and shared components leads to the equality above. The remark shows that each loss component has its own influence on feature representation at every step of gradient descent. Integrated over the whole training process, this influence accounts for the discrepancy between the feature representation at the beginning and at the end of training.
The remark above is grounded for SGD; however, the optimizer commonly used to train models is Adam (Kingma & Ba 2014). Thus, we adjust each gradient component G to reflect the gradient steps made by Adam. We keep separate moments ('(coopouent) for each of the three components and calculate the step attributed to each of the components as
(e(oppoonett)
(compoment) - 1 1 B
Then the G(direct): (preecccched) and (sharedd are an exact partition of the optimizer step, and use them to compute influence as defined above.
We interpret the value Tairect (w = E, (uf 0,) (integrated direct influence) the overall impact of the direct loss component on the emergence of the feature w in the model, and similarly for the other two components. We use this decomposition to answer which combinations of direct, shared, and pre-cached gradient signals produced a feature over the course of training. by evaluating the magnitudes of the three integrated components Tairect (w T Tpre-cached, (w and Tshared (w
EXPERIMENTS WITH SMALL TRANSFORMERS
In all experiments in this section, we train GPT-2-like Transformers (Radford et al. 2019) for NTP using cross-entropy loss. If not stated otherwise, all our models have standard architecture: they are non-myopic and tied. To estimate if Transformer represents a given feature linearly, we train layer- and -eetifnn- linear probes (Alain Bengio 2016 (Belinkov] 2022) to predict the value of feature using the residual stream of the trained model 8 as input. For consistency, we always treat probing as a regression task and evaluate the probes using Pearson correlation between the predicted and true values of a feature. Each of the probes represents one feature direction w².
In the attribution experiments, we then retrain the Transformer from scratch with the same random seed and data order, repeating the training trajectory of the first run that lead to 0*, For each batch in the training set, we compute dirrect (w, 0). I pree-cached 0). and Lahared (wy 0). We sum those values across the batches, obtaining the integrated influences for each feature
4.1
IN ToY TASKS, NTP-USELESS FEATURES Do NOT EMERGE WITHOUT PRE-CACHING AND CIRCUIT SHARING
First, verify intuitions using toy tasks where have clear understanding of the quired circuits: Majority and Conditioned Majority. We train two-layer models to solve each task.
In Majority, each example x consists of M tokens sampled uniformly from vocabulary of size V, and M tokens sampled from the set of the most frequent tokens so far: argmax, count (gg)
Our approach is discussed in detail in Appendix
5

---

Published as a conference paper at ICLR 2026
Component
NTP-useful?
[2.5%
97.5%)
Probing Gap
0.08
direct
yes
2.85
12.38
n0
-4.69
2.74
0.06
pee
yes
3.05
0.04
shared
yes
0.02
Layer Lager
Logee
Layer 2
Layer
Layer
9.91
combined
yes 12.14
19.05
20
40
60
4.42
10.07
Position in the Sequence
Figure 3: Left: the gap in probing performance between NTP-usefii and NTP-useless board squares for OthelloGPT. Right: 95% confidence intervals for the integrated influence of each component on the representation of NTP-useful and NTP-useless features.
The task is solved by simple uniform attention head computing the most frequent token. We track the influence components of the feature Fi "the most frequent token so far."
Conditioned Majority is designed bring the importance of pre-caching. The input consists again of two parts: M uniformly sampled tokens, followed by M samples from the set of those tokens that followed the token "A" most often in the first part. The task requires a mechanism akin to induction heads (Olsson et al. 2022): the first attention layer attends to the preceding token, and the second layer attends the tokens after "A." We study the feature F22 "the preceding token is A."
In both tasks, the first M tokens are random and can be predicted trivially, without F1 or Fz. Thus, in the input phase F1 and F2 are NTP-useless, and we are interested in whether our models will learn to represent them, and if they do, why.
For different random seeds, we train (non-)myopie (M-un)tied models. Thus, we obtain models that all experienced direct learning, but had pre-caching and/or circuit sharing ablated. We plot the performance of probes trained to extract Fj for Majority and F2 for Conditioned Majority in Figure 2 In both cases the myopic untied models show the most fragile representation of the features during the input phase, supporting our prediction that such models have no incentive to learn NTP-useless features. Lifting the ablation of pre-caching
circuit sharing improves the performance of probes. Notably, both tied and untied myopic models are unable to learn Conditioned Majority (Appendix
E.1 the two-layer circuit described above cannot develop without pre-caching.
4.2 PRE-CACHING AND CIRCUIT SHARING HOLD TOGETHER OTHELLOGPT'S WORLD MODEL
Next, we apply our framework to study Transformers trained to predict legal moves in the game of Othello, common testbed for evaluating world models implicit in language models (Li et al. [023] Yuan & Søgaard. (025). In Othello, two players place their tiles on the cells of an 8 x 8 board in turns. The of legal moves is deterministic function of the current board state. The initial work
on the topic claimed the discovery of coherent board state features in OthelloGPT (Li et al.) 2023 Nanda et al. 2023). however more recent evidence was more pessimistic, reporting that the implicit world model of OthelloGPT is fragile, especially when it comes to boards sharing the same set of legal next moves Vafa al. 2024 2025). We aim make sense of this new evidence from our perspective of NTP-use(ful/less) features.
Recall that we defined NsP-useless features as the ones that are needed for predicting the next token. Hence, in the case of Othello, for each sequence of moves Jet. NTP-useless features are the representations of cells that don't it affect the set of the legal next moves 2t. Our theory implies that, all else being equal, NTP-uscless features should be represented worse than NTP-useful features since they lack the direct learning component. However, pre-cached and shared components are active even for NTP-useless features, supplying them with some training signal.
We follow a standard experimental setting and train a Transformer on randomly generated game transcripts represented as sequences of up 60 tokens. The i-th token indicates the square where the tile was placed during the i-th move in the game. Then we train linear probes to extract the board state from the model's hidden representations.

---

Published as a conference paper at ICLR 2026
Probing Parformaace
cattanl pesdiens (pre-cichedd
0.02
0.7
pondicnal dend
0.6
0.00
PGG tags toyepia
tage thon syepuct
0.5
dxpendeccy toyeual
oxpenoeees laga eyyouot
Layer
Figure 4: Comparison of the three distinct types of features in a small language model: POS tags, dependency tags, and the positional feature. Left: probing scores for myopic and non-myopic models. Center: direct and pre-cached influence components for each feature type.
The experimental results align with our predictions. Figure 3 (left) shows the gap between the performance of linear probes trained extract NTP-useful and NTP-useless features: always positive and increases with layer depth (most likely because the deepest layers align more on direct influence). We verify the reasons for this disparity by inspecting the influence of each component calculated separately for NTP-useless and NTP-useful features (Figure 3 (right)). Direct influence alone is positive for NTP-useful but indistinguishable from zero for NTP-useless features. However, since pre-cached and shared influence is positive even for NTP-useless features, they are still learned.
These observations provide new evidence that complements the empirical findings of Vafa et al. 2025). Specifically, show with statistical significance that the model's inductive bias toward next-token partitions of state arises from the direct gradient component promoting learning of NTP- useful features. We also refine the result of Vata al. (2025) by showing that the useleseless cells are learned as well, albeit less robustly, since the signal for learning them is present due to the non-zero pre-cached and shared components. Together, our results explain both 1) why the model represents NTP-useful cells better than NTP-useless ones, and 2) why NTP-useless cells can still be recovered better than chance.
4.3 PRE-CACHING IS REQUIRED FOR COHERENT TEXT GENERATION, BUT NOT NEEDED FOR SYNTAX
The environment that ultimately interests us is language. Thus, we train and study tiny Transformers for natural text generation. The main question we ask is: Is pre-cuching needed for coherent text generation, and which relevant features does affect?
0.01
direet
direct
0..0
We use TinyStories v2 (Eldan & Li 2023), which contains GPT- 4-generated children's stories. The texts in this dataset use sim- ple language that could be understood by a child; these proper- ties make it learnable even by a very small model. Following train tiny
00.0
0..0
200
800
Stop
Eldan & Li (2023).
GPT-2-like lan- guage models on this dataset with different random seeds. For each story, we randomly choose a starting point and from there sample a substring of 64 tokens. We annotate each token with syntactic features (POS and dependency tags), as well as with a positional feature: the position of the token in the original story before subsampling the sequence. See Appendix D.3 for the full experimental details.
Figure 5: Development of the direct and pre-cached influence components of the positional feature in the non-myopic model during training.
We observe that myopic models have consistently much higher loss than non-myopic models (3.29 + 0.02 vs 2.53 + 0010, with training curves reported in Appendix F.3) indicating that pre- caching necessary for the task. At the same time, nearly all features we study seem to be direct: the performance of probes extracting these features does not vary significantly between myopic and non-myopic models, and for all but the positional feature, pre-caching influence lies much lower than direct influence (Figure with non-overlapping confidence intervals for mean (Figure [4] and one-sided Wilcoxon tests showing that the gap is statistically significant (Table We conclude that simple syntax can be learned without pre-caching.
What is pre-caching needed for then, if not syntax? It seems to be relevant for computing more complex properties of the input text: can be seen in Tigure/4[center) in non-myopic model the
7

---

Published as a conference paper at ICLR 2026
standands and sbnfardroton proeeeure and merginatzed communtites
"ormal mesnting
detais acticites
ratriciies
logitks, reguations technical spedificatioss
ML Frinch promeurs prypicul interface clenests identifirrs and numer valura
cluss metiard Lebelttons progantrinn deseloyezeer, liearr nanagenent code structuee and control flw serraininng corarrucs and systax
azfurer nglts permissina, and leeering
10
10
104
Caching Degree
Figure Distribution of (ww) for the SAE features Gemma 2 2B. The text boxes show the compressed descriptions of 5 features from the tails of the distribution, well as 5 random features from the mode the distribution.
positional feature, which relates to high-level properties of the stories, starts to be learned due to the pre-caching influence. Figure[Jshore transition during training, when pre-caching starts affecting the feature, hinting to development of circuit involving it.
5 INVESTIGATING FEATURES IN LARGE LANGUAGE MODELS
Retraining LLM from scratch is computationally very expensive, which is why the field of LLM interpretability operates mostly on final model checkpoints. For the same reason, the attribution method that used in Section is inapplicable in the same form to large-scale models.
However, even with access only to the final checkpoints, we can draw some conclusions about the emergence reasons of features, connecting traditional interpretability and our framework introduced above. In this section, we use this connection to understand the latent features of a State-of-the-Art LLM: Gemma 2 (Gemma Team al., 2024).
5.1 FINDING AND UNDERSTANDING PRE- CRE-CDCHED FEATURES IN AN LLM
The standard way of estimating the causal role of feature in Transformer is an intervention (Muelerr et al. 2024): modifying the activations of model during the forward pass to alter the representation of feature and observing the changes in predictions. Assume that one intervenes on the residual stream r (x) of a trained model To-, adding to a vector w, and then records the KL-divergence between the predictions with and without intervention at each position after i:
g. (ry d DRI. (az xT
(xe)))
Here, Tois (2y xzz)) are the predictions of a model under the intervention on the i-th position.
Proposition 5.1 (Approximating influence with interventions).
(0/6, x 0r V VoL Ejzini de d/in
Q(w)
1/500/1, ww Lo
The proof is deferred to Appendix Proposition implies that even if we cannot rigorously compute the influence components of feature as we don't have access the model's training trajectory, we can estimate the ratio of direct and pre-cached influence around the trained model using the quantity we denote Q(w)). In other words, we can find to be direct (an intervention changes only the prediction of the immediate next token) pre-cached. Importantly, however, we can only make statements about the components ratio in the region around the final model, not along the whole training path.
given feature is more likely
Finding pre-cached features in Gemma 2. fashion, use Sparse Autoencoder from the Gemma-Scope suite (Lieberum al. 2024). To compute Q(w) for each feature, we find the tokens where the feature is active and ablate it by setting its activation value to zero, effectively subtracting (r. www/ from the residual stream.
To extract the learned features an unsupervised
8

---

Published as a conference paper at ICLR 2026
Then we the model's predictions with and without the ablation calculate dy and Q(w) as explained above. See the details in Appendix D.4]
Using the fact that automatically generated descriptions of the SAE features are available, we study the descriptions of features with extreme values of ((w) and observe that most of them are related to programming or formal structure of the imput text. Based on this observation, we form hypothesis: are features related to formal reasoning tasks more likely to have extreme values of (ww)?
To test this hypothesis, we label each feature 0 depending on whether it activates on this type of inputs (details in Appendix D.5). and plot Q(w) separately for those groups (Figure 6). Indeed, the tails of the distribution see much higher concentration these formal features. Estimated 95% CI for the formal when modeled with log-normal distribution is 1.63 + 0.03, for the net formal it is 1.23 + 0.02. The results align with the intuitions detailed in Section 3.3 pre-caching seems to be needed in tasks that require emulating formal computational devices (c.g.. AST for code parsing).
To further test the robustness of the link between Q(w) and feature semantics, we examine how steering features with different values of ((w) affects samples from the model. For each feature, we draw unconditional generations from the model while steering that feature by adding its direc- tion vector, scaled by steering coefficient, to the residual stream the target layer. We find that steering features with a high pre-caching degree leads to generations containing more code and more punctuation than average (T). Interestingly, we don't 't observe the same effect for features the opposite end of the spectrum. The results thus support the connection between a feature having high value of ((w) and its involvement in formal reasoning. For features with low Q(w), this connection, if present, appears weaker and not as easily detectable by steering. Full experimental details are provided in Appendix D.6
5.2 PRE-CACHING AND AHEAD ARE SEPARATE PHENOMENA
Wu al. (2024) initially introduced the notion of pre-caching as a potential explanation for the look- ahead in LLMs (Pal et al., 2023), suggesting that pre-cached features may be the ones that contribute the ability predict future tokens the most. We test this hypothesis by investigating
the learned features of Gemma 2 with high Q(w)) are also most useful for the look-ahead.
We obtain future token predictor with look-ahead distance k by training a linear layer mapping WLA from the residual stream vector at position i to the token at position + + 1 on a subset of the Pile dataset (Gao et al. 2020):
0.000
com(Q.1 A)
-0.025
Bi+t+1 1 (x))
0.050
161
L14
-0.075
Then, for each SAE feature, we compute the angle A(WLA, LA, w,, r) between the feature direction w and the subspace formed by r main singular components of WLA If a feature w is useful for predicting future tokens, it to be expected that WLA will learn to use it, and cos A(W LA W, r) will be non-zero. We report the Spearman correlation between cos A ALWAA, w,r) and (ww) for each in Figure for the mapping up to 5 tokens ahead. See the details the experiment in Appendix [D.7
00.000
500
1000
1500
2000
Figure 7: Correlations between cos A(W LA, w. and Q(w).
Surprisingly, we find the negative correlation across all look-ahead maps. This means that not only the look-ahead is not caused exclusively by the pre-cached features only, but they contribute to less than direct features. This is strong evidence for the breadcrumbs hypothesis Wf et al. (2024): ahead in LLMs arises not from implicit planning but rather from the similarities of features needed predict tokens at different positions.
RELATED WORK
Effects of NTP training. This paper is a part of a line of work aiming to understand how the NTP objective shapes the algorithms learned by Transformers. Bachmann & Nagarajan (2024) Nagarajan jet al. (2025) discuss the critiques of NTP and show how it can lead to learning undesirable shortcuts or lack of creativity, mitigated by multi-token training initially proposed by Gloeckle et al. (0024).

---

Published as a conference paper at ICLR 2026
Motivated by these findings, Hu et al. (2025) modify the objective, Thankaraj et al. (025) propose reordering the tokens in the imput.
World models and future token prediction in Transformers, Li et al. 2023); y; Karvonen (202));
Shai et al.] (2024); Jin & Rinard (2024), among others, find that Transformers implicitly reconstruct the latent variables of the data generation process, arriving to the so-called "world models". The learned representations, however, are not always linear (Engels et al.] (024) and can be brittle and inconsistent (Vafa et al. 2024] 2025). Pal et al. (2023) show that future token predictions can be decoded from the internal representations of pretrained LLM with nontrivial accuracy using probes and learned prompts. Jenner et al. (2024) find a similar effect in a neural network trained play
chess.
Most relevant our work, Wu et al. (2024) investigate the reasons behind the emergence of look- ahead in LLMs and proposes the pre-caching hypothesis, stating that some tokens might "prepare in advance" the information relevant for future tokens. We build upon that work and borrow the term "pre-caching" from there. However, our contribution is distinct from that of Wu et al. (2024) in several aspects: most importantly we bring the level of analysis down to the development of specific features, and also introduce the concept of circuit sharing, not analyzed by Wu et al. (2024).
Developmental interpretability. Another relevant to us line of research the emerging subarea of interpretability that takes inspiration from the field of training dynamics and analyzes the changes in the algorithms encoded in the models at different moments of training, asking when and why LLMs pick up different skills or features during training. TTgges al. (2024) analyze circuits in LLMs at different training checkpoints and find that circuits implementing various task abilities stay consistent across model sizes and emerge at similar numbers of training steps. [Bayazit et al. (2025) track the emergence of features using crosscoders trained across model checkpoints, and Xu al. (2024) use SAEs for the same purpose. WWang et al. (2025) monitor emerging specialization of attention heads using the concepts from Singular Learning Theory. Kangaslahti et al. (2025) cluster different datapoints based on their loss dynamies, estimating the moments when certain abilities become acquired by the model. Michaad et al. (2023) target explaining the LLM scaling laws by suggesting that scaling the number of parameters and training steps allows them to learn skills of increasing rarity. In similar vein to our work, |Mireea et al. (2025) inspect the gradients during LLMs pretraining and proposes that LLMs may go through a period of "loss deceleration", when the per-example gradients cancel each other, slowing down the learning of new concepts.
7 CONCLUSION
Traditional work in mechanistic interpretability aims to understand a learned feature by tracing which circuits it plays a role in. However, those circuits do not simply appear in the model: they need gradient signal to develop. In this work, studied the sources of that gradient signal, shifting from the commonly employed static teleological perspective to a developmental one, in which learned features in Transformers are viewed outcomes of gradient-based learning rather than gear in final algorithm.
In Section we showed how this change of perspective makes difference, helping interpret fea- tures that cannot be explained solely through being a part of an algorithm predicting the immediate next token. We believe that Othello is a prime example, where representation of NTP-useless fea- tures can be explained by inspecting pre-cached and shared gradient components.
Unfortunately, modifying the training pipeline of a model is often too computationally expensive to be done in practice. The best one can do in this case is inspecting the region around the trained model, which what we did in Section [5] but even using the restricted toolkit of intervening on a static model, we can perform analysis on the development of linearly represented features under study.
Improving the efficiency of our attribution method and adapting it to be applicable to large-scale models is a potential avenue for future work. Another promising direction is use our method to discover previously unknown interpretable features by analyzing the residual stream subspaces formed by a distinct gradient component (e.g., only by pre-cached updates but not direct updates).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
MR thanks Alcksandra Bakalova, Yash Sarrof, Yana Veitsman, Xinting Huang, Michael Rizvi- Martel, Vaishnavh Nagarajan, Naomi Saphra, and Nicolas Flammarion for helpful discussions and feedback on the drafts of this paper. JN is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD programme Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 2016.
Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah, and Joshua Batson. Circuit tracing: Revealing computational graphs in language models. Transformer Circuits Thread, 2025. URL/https: //transformer circuitst....se
graphs straphs/methodee hewthods httthhoss.mmlmm....
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. Advances in Neural Infor- mation Processing Systems, 37: 1 136037-136083, 2024.
Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Reseurch, pp. 2296-2318. PMLR, 21-27 2024. URL https: pestps://proceediing ngs..
m1 httess//2355/acchaaaae
html
Deniz Bayazit, Aaron Mueller, and Antoine Bosselut. Crosscoding through time: Tracking emer- gence & consolidation of linguistic representations throughout Ilm pretraining. arXiv preprint (xXiv:5509.05291, 2025.
Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207-219, 2022.
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english? arXiv preprint arXiv:2305.07759, 2023.
Joshua Engels, Isaac Liao, Eric Michaud, Wes Gurnee, and Max Tegmark. Not all language model features are linear. arXiv preprint vrXiv:2405.148600 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Michael Gazzaniga. Forty-five years of split-brain research and still going strong. Nature Reviews Neuroscience, 688):653-659 2005.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In International Conference Machine Learning, pp. 15706-15734, PMLR, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv: 177. 2024.

---

Published as a conference paper at ICLR 2026
Wes Gurnee and Max Tegmark. Language models represent space and time. arXiv preprint arXiv:2310.02207, 2023.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-
strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo. 1212303.
Edward S Hu, Kwangjun Ahn, Qinghua Liu, Haoran Xu, Manan Tomar, Ada Langford, Dinesh Jayaraman, Alex Lamb, and John Langford. The belief state transformer. In The Thirteenth International Conference on Learning Representations, 2025.
Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in thees-pplyiing neural network. Advances in Neural Informa- fion Processing Systems, 17:31440-31437, 2024.
Charles Jin and Martin Rinard. Emergent representations of program semantics in language models
trained on programs. In Forty-first International Conference on Machine Learning, 2024.
Sara Kangaslahti, Elan Rosenfeld, and Naomi Saphra. Hidden breaktbroughs in language model training. arXiv preprint arXiv:2506. 15872, 2025.
Adam Karvonen. Emergent world models and latent variable estimation in chess- bhess-p language models. In First Conference on Language Modeling, 2024.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 69800 2014.
Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on a synthetic task. ICLR, 2023.
Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05 2024.
Johnny Lin. Neuronpedia: Interactive reference and tooling for analyzing neural networks, 2023.
URL inttps://wwtt httpp:///wwwww.eeoooee..aa. org. Software available from neuronpedia.org.
Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. arXiv preprini urXiv:2210. 10749, 2022.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36:28699 28722, 2023.
Andrei Mircea, Supriyo Chakraborty, Nima Chitsazan, Irina Rish, and Ekaterina Lobacheva. Train- ing dynamics underlying language model scaling laws: Loss deceleration and zero-sum learning- In Wanxiang Che, Joyce Nahende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. 28154- 28154-28111 18, Vienna, Austria, July 2025. Association for Compu- tational Linguistics. ISBN 979-8 88-8888976-222 76-1--0--- doi: 10. 00w53881112000 acl- acl- 1366. URL. 1366/
Aaron Mueller, Jannik Brinkmann, Millicent I. Li, Samuel Marks, Koyena Pal, Nikhil Prakash, Can Rager, Aruna Sankaranarayanant Arnab Sen Sharma, Jiuding Sun, et al. The quest for the right mediator: A history, survey, and theoretical grounding of causal interpretability. CoRR, 2024.
Vaishnavh Nagarajan, Chen Henry Wu, Charles Ding, and Aditi Raghunathan. Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction. In Forty-second International Conference on Machine Learning, 2025.
Neel Nanda, Andrew Lee, and Martin Wattenberg. Emergent linear representations in world models of self-supervised sequence models. In Yonatan Belinkov, Sophie Hao, Jaap Jumelet, Najoung Kim, Arya McCarthy, and Hosein Mohebbi (eds.), Proceedings of the 6th BlackboxNLP Work- shop: Analyzing and Interpreting Neural Networks for NLP, pp. 16-30, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2022 www.0223..llckbboo..o.o 1.2. URL https: //ccl antho ogy. wgw.oorg//02.... ack.oxnlp-pp1..
12

---

Published as a conference paper at ICLR 2026
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodss, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022.
OpenAl. Gpt-4o mini: advancing cost-efficinnt intelligenne,, July 2024, URL hotm://ineei/ggee--------aaaaaa- wtt.aaawwnncnnooooooooooooo-aeee
OpenAl. Introducing gpt-4.1 the api, April 2025. URL https::/opeeai..com http//ppaaa.com/neeeee...
(pt-4-11
Koyena Pal, Jiuding Sun, Andrew Yuan, Byron Wallace, and David Bau. Future lens: Anticipating subsequent tokens from a single hidden state. In Proceedings of the 27th Conference on Compu- tational Natural Language Learning (CoNLL), Singapore, December 2023.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In International Conference on Machine Learning, pp. 39643-39666. PMLR, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, lla Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Adam Shai, Lucas Teixeira, Alexander Oldenziel, Sarah Marzen, and Paul Riechers. Transformers represent belief state geometry in their residual stream. Advances in Neural Information Process- ing Systems, 37:75012-75034, 2024.
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas I. Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Trans- former Circuits Thread, 2024. URL wtw.aaanttnitttttii..neei.mmme index.htm.
Abitha Thankaraj, Yiding Jiang, J Zico Kolter, and Yonatan Bisk. Looking beyond the next token. In ICML 2025 Workshop on Long-Conted Foundation Models, 2025.
Curt Tigges, Michael Hanna, Qinan Yu, and Stella Biderman. Llm circuit analyses are consistent across training and scale. Advances in Neural Information Processing Systems, 77:40699-407331 2024.
Keyon Vafa, Justin Y Chen, Jon Kleinberg, Sendhil Mullainathan, and Ashesh Rambachan, Evalu- ating the world model implicit in generative model. arXiv preprint arXiv:2406.03668) 2024.
Keyon Vafa, Peter G Chang, Ashesh Rambachan, and Sendhil Mullainathan. What has a foundation model found? inductive bias reveals world models. In Fecty- International Conference on Machine Learning. 2025.
George Wang, Jesse Hoogland, Stan van Wingerden, Zach Furman, and Daniel Murfet. Differ- entiation and specialization of attention heads via the refined local learning coefficient. In The Thirteenth International Conference on Learning Representations, 2025.
Wilson Wu, John Xavier Morris, and Lionel Levine. Do language models plan ahead for future tokens? In First Conference on Language Modeling, 2024.
Yang Xu, Yi Wang, Hengguan Huang, and Hao Wang. Tracking the feature dynamics in Ilm training: A mechanistic study. arXiv preprint arXiv:2412.17626, 2024.
Yifei Yuan and Anders Søgaard. Revisiting the othello world model hypothesis. In ICLR 2025 Workshop on World Models: Understanding, Modelling and Scaling, 2025.
13

---

Published as a conference paper at ICLR 2026
A
PROOF OF PROPOSITION 3.1
Proposition A.1 (Restated from Proposition 3.1. For any layer k and position i,
VoL(x, To(x)) VoL; (direct) (c, (prohcached) (x, (e,To(())+ V VoLi (shared) (x, [x))))
(4)
Where
Vol. (direct) (r,To(x)) = (011:)1:TT(x)) - Vol. 1i+1 (To (To
(5)
V VoL. (pre-cached) (z,To(<x) - Σ VoL(oz41,, VaL ( (Tox)))))
(6)
VoL. (shared) (z)T = Σ VoL 1341. (T0)):77(
Proof.
VoL(z,To(z)) VoL(I,, + VoL (2, (T)
VoL (sr, (T(
N-1
N-1
N-1
VOl. TJ+1; [ [To
(z))))))))))
Σ (0
Σ j=1
Vol. Tz+11 (T0)))7700)))3((1
Vol. (shared)
Vol (direet)
VoL (1z+1, (T(2)) + (ooL(11H),T((x))) - VOL (2441, (T + Σ
j=1
(ol(z)+11((()) VoL (T))) 3
VoL (prescached)
B ADDITIONAL DETAILS ON ATTRIBUTION EXPERIMENTS
Computing the component influence.
By Definition 3.3
(0. x O" de
R |
0 0" w
A simple application of the chain rule gives an equivalent definition:
1 (0, x w.t 0" G) (VoR (x 0, 0" wu'
(8)
Thus, computing the influence of given gradient term can be done by taking the inner product of that gradient term and the gradient of the feature mismatch.
To compute Lairect (w:: 0), Ipre-cached (w: 0). and (uf 0) for given and k. we employ the following algorithm. First, we calculate VoL; for every j in standard way. Next, we calculate Vo in similar manner, but during the forward pass of the model detach the tensor corresponding to (x). After that, we compute the gradient decomposition terms according to the definitions in Section 3.4
The only thing left is the gradient of the feature mismatch. We apply the linear probe defined by w to both rf (x) and (x) and compute the feature mismatch according to Definition 3.2 We run one more backpropagation to find R and take its inner product with the gradient decomposition terms, obtaining the desired influence values.
14

---

Published as a conference paper at ICLR 2026
Correction for Adam. As metioned in Section 3.4 the naive computation of influence compo- nents described above is grounded for training with SGD (Remark but not for Adam. Indeed, in the case of Adam, the parameter update is computed differently, and linear step toward the neg- ative gradient no longer reflects how the model trained. Moreover, due to the inclusion of the first and second moments, the parameter update cannot be computed as a linear of three separable components:
O - 0; + (V L(diect)) + g(V (pre-cacchdd) + g (shhared))
while for SGD it is true, with g(z) = -1 x (and that fact is the basis behind Remark 3.4
However, with two small adjustments can make the Adam parameter update separable. First, we keep separate moments for each of the three components:
(direct)
(direct) (1 B1) (direct)
m
3,m
(similarly for the other two).
Second, use the same adaptive learning rate for all three components:
1
v +
where is is computed normally. Then,
0.+1 O2 (direet)) {m (prevcached) g(ml (sharedd) )
where
((x)
This way, we can use g(m) (direct)) as an adjusted gradient for the direct component. From here, the computation similar the case of SGD.
Overall Algorithm. First, for a given dataset, we train a model normally. In all our experiments, we use Adam to follow community standards. After the model has been trained, we freeze it, save the hidden activations the validation dataset, and obtain set of linear probes to regress latent variables.
For us. these probes represent the directions w needed to compute the gradient-componet in- fluence. Once we have the probes, we retrain the model from scratch, fully repeating its training trajectory, which involves starting from the same initialization the first time, keeping the data point order the same, fixing the PyToorch random seed, etc. During this second training run, at each training batch we compute the feature mismatch and the gradient-component influence for each probe.
The second run is needed because during the first one do have the directions wf needed to compute component influences, and to estimate them we need 0*, which itself requires the training run to finish. In principle, one could save all gradient components during the first run and then retrieve them from memory when computing the influence, but that is extremely expensive in terms of the disk space needed. Thus, we perform two consecutive training runs.
C PROOF OF PROPOSITION 5.1
Proposition C.1 (Restated from Proposition 5.1
Ejstt dfi
(e/+ / x,x w. 8*, V L (0/5, 00/0/,00 w, 0", V V
Q(w)
Proof. Both direct and pre-eached gradient components consider only the gradient paths that have r as bottleneck, so by chain rule they can be expressed as
V T (direct)
Jolr.] V Tu
15

---

Published as a conference paper at ICLR 2026
And
VoL (pro-cached) = Jo(r V
Σ f=i
For convenience, from this point we will treat them together as Voleo C E (direct, pre-cached);
1
R(x 0, 0, w 2 ((wf, rx)) (wf, r8. (x))))
VgR(2 1 1,,0", w = (lw (x)) - (wr r (xx))) Vo(u, (x) = (olrb,)(
Per equation
(0, x wf.0", 0", VoL.) = (VoR (x 0,0" w VoLc) =
(9)
= Am Jalro.l " Jolrll V Le = Jolrd,l Jol (Aw, V V. Le)
(10)
Imagine we do a feature ablation: that is, we take trained model 0 and add vector Awf to rg- r We measure the loss Le that depends on the model's output. Doing first-order approximation around r (8' are the parameters of this intervened model).
L. L ~ (Aw?) Vror,
ror, Le
(11)
A corollary of equation 10 and equation 11 is that for any model with parameters l' such that T0, = - + Aw the influence of component c is proportional to the increase loss compared to the model d"
Assuming that that 0* perfectly fits the data (p(#:) x<e)) To- (xt | xee)) the difference in cross- entropy NTP losses becomes
L'dirvct Ldirect
E xe Te (2c102e)
To (Iz (x(e)
E Te (xe (e)e)
log Tg. (ar x<()
log To- (xte log To (xte wee))
x&t)) DKL (To (x, I<e (z, x<t))
E
Ze (2c(22e)
Similarly, for the pre-cached component, the difference in losses corresponds the of KL- divergences for future positions.
D (To-(To- TT (10411 [ Σf>++1 DKL (T (#541 xT)) To (1z+1 xgz)) Where Tor is model where o was modified along the direction w.t
(0, x wt 0", Voldirec)) (0, wh, O+, V VpLprr-ccchha))
(12)
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 TOY TASKS (SECTION 4.1
16

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for the experiments with the tasks of Majority and Conditioned Majority,
Hyperparameter
Value
Hyperparameter
Value
Layers
Steps
3000
Heads
4
Train size
102,
Hidden dim
128
Eval size
10,240
Feedforward dim
512
Number of seeds
10
Learning Rate
0.001
Input phase size
10
Batch size
256
Output phase size
10
Vocabulary size
3
D.2 OTHELLO (SECTION 4.2
Table 2: Hyperparameters for the Othello experiment.
Hyperparameter
Value
Hyperparameter
Value
Layers
6
Steps
5000
Heads
4
Train size
5,120, 000 20, 1
Hidden dim
256
Eval size
Feedforward dim
1024
Number of seeds
Learning Rate
0 001
Batch size
1024
Following prior work, we generate dataset of synthetic games of Othello. When generating new game, we always randomly choose move that is legal given the current board state.
We encode the game as sequence of tokens, where the i-th token represents a square where a stone was placed at the i-th turn, irrespective of which player placed it. We also add a token pass for the turns when no moves are legal. The games our datasets have fixed length of turns, and end with repetitions of pass if the game tree finished before that.
When training the linear probes, we encode the board state as a matrix of -1, O, and represents the squares belonging to the active player, -1 represents the squares belonging to the non-active player, and 0 represents the empty squares. When square is placed, the board state matrix gets negated (because the active player changed) and -1 gets added one of the cells.
When estimating the influence separately for NTP-useless and NTP-useful squares, we don't take empty cells into account. For given board state, consider square NTP-useful if flipping (negating the corresponding cell in the board state matrix) changes the set of the legal moves. Otherwise, the square is considered NTP-useless. When estimating the influence components for the NTP-useless and NTP-useful features, we use the same probe for both types. but compute feature mismatch separately. This way, we aggregate R(x 10, 0 w disjointly for the objects the batch where the square is NTP-useless and NTP-useful, and proceed as usual.
After that, Trau (w NNTP useless can be seen as the contribution of NTP-useless features into the devel- opment of the linear direction defined by w. If [nan(w aTPPaseless negative close to zero, indicates that these features do not contribute to the linearity of the representation.
Due to the resource constraints, we cannot compute the influence components for each position and square of the board. Thus, we use 16 central squares and compute influence for every 4th position (from 4th to the 56th turn).
D.3 SMALL LANGUAGE MODEL (SECTION 4.3
We employ the tokenizer used in the original work on TinyStories (Eldan Li 2023
We test the following features:
1. POS tags: annotate POS tags using spaCy (Honnibal et al. 2020). one-hot encode them and use the 10 most common tags, encoded as O or I, as features. Since one word can contain multiple tokens, each token gets assigned the tag of the word it belongs
17

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for the small LM experiment (TinyStories).
Hyperparameter
Value
Hyperparameter
Value
Layers
4
Steps
1000
Heads
4
Train size
256, 000
Hidden dim
128
Eval size
30,000
Feedforward dim
512
Number of seeds
10
Learning Rate
0.1
Max length text
64
Batch size
1024
2. Dependency tags: same procedure, also the most common tags.
3. Positional feature: since we sample substrings of 64 tokens from the original stories to train the model, the positions of tokens in these substrings do not directly correspond to their positions in the original text. We use that original position as another feature that assumes the values from 0 (the token is the first token in the original story) to 1 (the last token).
Thus, we have 20 binary features and continuous feature. We estimate influence of those features for each position from to 39.
D.4 CALCULATING Q(w) (SECTION 5.1
To characterize the causal role the learned features in Gemma 2 (Gemma Team et al. employ a Sparse Autoencoder (SAE) from the Gemma-Scope suite (Lieberum al. specifically the SAE trained at the residual stream in layer 15 with 16k hidden features.
2024). 2024
The first step is to curate dataset of diverse. high-activating text examples for each feature. We se- lect text sequences from Neuronpedia (Lin) 2023) where features show their highest activation. For each feature, find the token position with the maximum activation value and extract the surround- ing text, including a fixed window of 10 subsequent tokens to create an initial set of sequences. This is then filtered for diversity based on token-level Levenshtein distance, resulting in a collection of high-activating and textually different sequences.
On these sequences, perform intervention the token position of maximum activation. We intercept the residual stream activation vector, pass through the SAE's encoder, and set the activa- tion value of our target feature zero. This modified of feature activations is then passed through the SAE's decoder to create new, ablated residual stream vector, which replaces the original one in the forward pass.
To quantify the impact of the feature, we compare the model's subsequent token predictions with and without the intervention. We measure the change in the output probability distribution each future position using KL-divergence. This allows us to compute a ((w) score representing the ratio of the feature's delayed to immediate influence. We average this score across different sequences. As discussed in Appendix C high Q(w) value indicates 'pre-cached' feature whose primary influence is on the model's future state, while low value indicates a 'direct' feature influencing on the immediate prediction. We filter out from the analysis the features, ablating which does not show substantial effect on any generated tokens:
DEL (To- a
= (1)41 2 (22) < 0.05
After that, we are left with 14,565 features to analyze out of the initial 16,384.
D.5 CLASSIFYING SAE FEATURES (SECTION 5.1
To assess whether a given SAE feature relates to formal reasoning, we use its automatically gener- ated decsription available in Neuronpedia (Lin 2023). We use the available descriptions generated using -Pii-4ommini (OpenAl 2024) and classify these descriptions themselves with GPT-4.1 (OpenAT 2025).
18

---

Published as a conference paper at ICLR 2026
Our prompt is Here is the description of certain textual property: "DESCRIPTION". Is this property related to TAG? Respond with one word only: yes or
In this prompt, DESCRIPTION filled with the retrieved description the feature, and TAG is one of "computer code, programming languages, ma math" or "syntax text structure".
In this way, for each feature obtain the labels is code and is syntax and consider the feature related to formal reasoning if at least one of them 1.
D.6 STEERING SAE FEATURES (SECTION 5.1
To steer a feature in model, we add w multiplied by steering coefficient to the residual stream: r < + + Sw. In our experiments, we set 5 10.
For each feature, sample 64 generations from model with that feature steered at every token position. We sample 20 tokens unconditionally with temperature I, thus generating samples from a distribution (2)) but under steering.
For each generated sample, we count the number punctuation characters contains. We also classify whether the sample represents a snippet of computer code using Llama 3.1 8B (Grattafiori cct al. 2024) as a classifier. This gives us, for each feature, two metrics: the number generated code snippets (out of 64 total generations) (# code) and the average number of punctuation symbols per generation (#punct).
In Figure 19 show these metries grouped by Q(w) of the corresponding feature. Additionally, similarly to Section 5.1 we estimate the 95% CI for the parameter of the distribution of Q(w) when it is modeled code above the median value and those below it. We perform the same split for #punct. The results, presented in Table 6 indicate that Q(w) has heavier tails for the groups where # #odde (or #punct) is above the median, mirroring our results in Section 5.1
a log-normal distribution. We split features into two groups: those with
D.7 LOOK-AHEAN EXPERIMENT (SECTION 5.2
We obtain a simple future token predictor by training linear layer mapping from (x) to 2;-441 on subset of the Pile dataset Gao et al. (2020):
fi+141 n (WLA (ū,,-rr(
Here h is the frozen language modeling head of Gemma, and >>>>> 0s the distance of the look- ahead.
This approach resembles Linear Model Approximation of Pal al. 2023 except keep the language modeling objective instead of the reconstruction loss on the activations at the last layer.
After training. WLA defines subspace in the space of the residual stream. Note that the weights of the SAE encoder define directions in the same space: indeed, one SAE feature can be seen is a linear projection of residual stream followed by activation function. Thus, can estimate how much a feature contributes to the look-ahead by checking how close the feature encoder vector lies the subspace defined by WLA.
Intuitively, if some linear direction v in the residual stream of the model contains information crucial for predicting look-ahead linearly, WLA would learn to "read" from this direction, and columns proportional to would appear in WLA-
To check that, we compute the angle A(WLA, LA. w. between the feature direction w and the subspace formed by r main singular components of WLA- Specifically, let WLA UEVT be the SVD decomposition of WLA Let be the matrix of SAE feature directions: the weights of the SAE encoder of the size d x Nieaauures For each from I to d, we compute the projection matrix P V,VT feature projections É PE, and save the cosine distances between the corresponding columns in E and E.
19

---

Published as a conference paper at ICLR 2026
In this way, for the given look-ahead distance and matrix WLA obtain matrix C' of size d x Rifaatrre- Crj is the cosine similarity between j-th feature and its projection on V,, V?7 In Figures aaddd we report the correlations between ((w) and Cry for each
E FURTHER EXPERIMENTAL RESULTS
E.1 TOY TASKS
Table 4: Accuracy of trained model for the first token in the output phase. All trained models successfully solve the tasks, except for the myopic models trained on Conditioned Majority.
non-myopic
myopic
tied
10-untied
tied
10-untied
Majority
1.00 0.00
1.00 + 0.00
1.00 + 0.00
0.96 0.05
Conditioned Majority
0.99 0.02
1.00 + 0.00
0.76 + 0.01
0.76 + 0.09
drrsi pro-ouc
cidee
3.-
33and
33-
Senganoion
Sespionce
Figure 8: Normalized integrated influence components for the tasks of Majority (left) and Condi- tioned Majority (center: tied model, right: 10-untied model).
In models solving Majority, the features the input phase seem to be predominantly shared, but that flips at the 10th token exactly the point where the feature is needed to predict the immediate next token, which is the output for task.
The images for Conditioned Majority may help explain the curious increase in the performance and probing accuracy of the 10-untied model compared to the tied model. We see that for the 10-untied model it is the pre-caching influence that causes the previous-token feature to be learned in the input phase, while for the tied model there is no clear pattern. This likely indicates that, instead of the robust circuit we expected, the tied model learned less robust, possibly shortcut solution, due to the influence of shared gradients from the output phase.
E.2 OTHELLO
1.0
4.0
0.8
Train Accuracy
Thain Loes
3.5
Josa resppic
arcunry (non-myppic) inss (nyegtc)
0.6
trois accumay (nyopio
0.4
0.2
1000
2000
3000
4000
5000
Steps
Figure 9: The training loss and accuracy curves of the myopic and non non-myopic OthelloGPT models. The training of a myopic model much less stable and it does not converge the same performance as non-myopic model.
20

---

Published as a conference paper at ICLR 2026
Myopit
Probing Perforn-anc
0.7
Non-Speppi
0.6
3
Layer
Figure 10: Performance of the probes trained to extract the board state from myopie and non- myopic Transformers for Othello. Myopic model shows a much lower degree of the board state representation.
The scores presented in this plot are lower than the ones reported in the prior literature (Nanda ct al. (023) due to the differences in evaluation methodology (we train the probes for regression and evaluate them using Pearson correlation, also unlike the prior work include empty cells into evaluation). In addition, our model is smaller than the one used in the prior work (6 layers against and smaller hidden dimensionality), dictated by the limits of computational resources.
In Figures and 12 we compare the direct and pre-cached influence components for different squares in the Othello experiment. Each plot is an average of the results for two close positions.
Interestingly, we find the results to be quite noisy and clearly non-stationary with respect to the position in the sequence. general, it seems that the importance of the direct component generally goes down later in the game relative to the importance of the pre-cached component. However, we believe that more experiments are needed to make conclusive statements.
21

---

Published as a conference paper at ICLR 2026
4 lnner cells, positions (4, 0]
Inner cells, positions [12, 16)
150
Geet bn Oudedd uxden
-100
Layer
positions 120,
Innerr cells,
-3
Layer
Layer
Inner cells, positions (36, 40]
Inner cells, positions [M4, 48)
-3
Layer
Layer
Intter oells, positions [52, 56]
Layer
Figure 11: Integrated influence components at different positions for the 4 central cells in Othello. Each plot an averaging of positions.
22

---

Published as a conference paper at ICLR 2026
12 Outer cells, positions (4, 0]
12 Outer cells, pesitions [12, 16)
Deed Niloort
1.5
Cad NiTuut
10
-0.5
Layor
Layer
positions 120,
Outer positions
0.5
-15
-1.5
Layer
Layer
12 Outer cells, positions (36, 40]
12 Outer cells, positions (84, 40]
0.5
0.0
-1.0
1.5
Layor
Layer
12 Outer oeTss, positions [52, 56]
Layer
Figure 12: Integrated influence components at different positions for the 12 cells surrounding the center in Othello. Each plot is an averaging of two positions.
23

---

Published as a conference paper at ICLR 2026
E.3 SMALL LANGUAGE MODEL
myyogh
Train Loss
3
200
400
600
800
1000
Steps
Figure 13: The training loss curves of the myopic and non-myopic small language models. can be seen that myopic LM has consistently much higher loss from very early in training.
0.02
0.01
6.00
0.04
Layer
Layer
Layer
(a) POS tags
(b) Dependency tags
(c) Positional feature
Figure 14: Direct and pre-cached influence for different feature types (as in Figure with 95% CI estimates for the average value at each layer.
Feature category
I Layer
t-statistic
Pareater
Pless
358640
5.8 x 10-8
1.0
2
387970
3.1 x 10 16
1.0
POS tags
3
374375
5.5 x 10 10-12 12
1.0
4
452676
3.3 10-46
1.0
269035
9.4 x 10-18
1.0
2
297604
2.5 10-31
1.0
Dependency tags
3
301802
2.9 X 10 10-37
1.0
350830
1.1 X 10
10-80 80
1.0
4080
4.5 x 10-8
1.0
1033
1.0
1.4
10-7
Positional
3
1289
1.0
1.1 X
10-5
4
3007
4.9 X 10-2
0.95
Table 5: One-sided Wilcoxon test statistics comparing direct and pre-cached influence components, with p-values reported for each feature type and layer. In all layers, the direct influence for POS and dependency tags is significantly larger than the pre-cached influence, which is reversed in the middle layers for the positional feature.
E.4 GEMMA
24

---

Published as a conference paper at ICLR 2026
10a
Pro-Caching Degree
Figure 15: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2B.
sniton
10-2
10-1
10-7
10-1
100
10a
10"
30*
10f
10
10*
Pre-Caching Degree
Pre-Caching Degree
Pre-Caching Degree
Figure 16: Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code, math, syntax, or text structure, are plotted orange; others are plotted blue. Images to the left and the right show the tails of the distribution.
cuale
10-2
10-1
10-2
10-1
100
101
10a
10P
104
168
10a
16
104
Pre Cathing Degree
Pee- TachCaahing Degree
Pre: Pre-Caching Degree
Figure 17: Distribution Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to code math are plotted orange; others are plotted blue. Images to the left and to the right show the tails the distribution.
syntux
Pro. Caching Degree
Pro Cochhnn Degres
Pre Pre-Caching Degres
Distribution of Q(w) (pre-caching degree) for the SAE features of Gemma 2 2B. The features classified as related to syntax or text structure, are plotted orange; others are plotted blue. Images to the left and to the right show the tails of the distribution.
Figure 18:
25

---

Published as a conference paper at ICLR 2026
Code Completions
5.0
Punct. Symbols
7.0
6.5
I
4.0
3.5
5.5
0.1
0.1 10
10
0.1
0.1-10
10
Pre-Caching D Degree
Pre-Caching Degree
Figure 19: Number of unconditional generations classified code (left) and average number of punctuation symbols (right) obtained by steering SAE features Gemma 2 2B. Steering features with high pre-caching degree leads to both more code and more punctuation symbols being gener- ated.
Group
â (95% CT)
#code above median
1.479 0.030
#code below median
1.428 + 0.020
#punct above median
1.519 + 0.025
#punct below median
1.351 + 0.022
Table 6: Estimated scale parameter a of the log-normal distribution fitted to Q(w) for different feature groups. For each group, report the estimated and its 95% confidence interval. Features are split at the median value of # #ooee and #punct.
Feature
(w)
Description
15050
0.002
HTML CSS code referencing scripts and stylesheets
3016
0.006
programming constructs related annotations and metadata code
3025
0.006
French pronouns and their conjugations various contexts
4840
0.006
aspects related to graphical user interface elements
2127
0.007
identifiers and numerical values in structured format
9124
0.010
elements related data structures and operations in programming contexts
11737
0.011
article "An" used in various contexts
4214
0.011
numerical values or symbols, particularly those frequently associated with coding, data
structures, or software libraries
11655
0.013
articles and determiners preceding nouns compiler directives and warning pragmas in code
13070
0.014
Table 7: 10 features with the lowest Q(w) among the SAE features of Gemma 2 2B under study.
Feature
Q(w)
Description
4592
13710.8
programming related terms and structures, particularly those associated with class and method definitions references to web development, particularly relating dependencies and library management
12285
6946.1
6579
3308.5
code structure and control flow statements
15090
2675.1
programming constructs and syntax elements
10042
1853.5
text related to software usage rights, permissions, and licensing terms
13045
1641.9
elements and objects that are part of programming interface or user interface function calls and their syntax within code
15829
1202.1
6139
898.5
technical references to Forms and related components in programming terms related to networking and software development colons or punctuation marks
13552
871.2
2162
756.9
Table 8: 10 features with the highest Q(w) among the SAE features of Gemma 2 2B under study.
26

---

Published as a conference paper at ICLR 2026
Feature
Q(w)
Description
9964
0.125
terms related to standards and standardization processes
14622
0.663
terms related poverty and marginalized communities
5029
0.928
details related activities undertaken actions witnessed various contexts
136
0.852
terms associated with legal prohibitions restrictions
4010
0.435
concepts related to logistics, regulations, and technical specifications
13176
0.227
references events gatherings
2463
0.330
references assembly attributes in programming context
4685
0.109
tokens related identifiers or lypes in programming languages
2803
0.171
references to processes related to healthcare, particularly in the context of treatment and
intervention strategies
9598
0.560
legal and political events controversies
Table random features with Q(w) between 0.1 and 1.1.
0.01
corr(Q, A)
0.00
-0.01
t+1
t+4
t+2
-0.02
t+3
500
1000
1500
2000
Figure 20: Pearson correlations between cos A AAWL, w, r) and Q(w) among the SAE features in Gemma
27

---

Published as a conference paper at ICLR 2026
STATEMENT ON LLM USAGE
We used LLMs assist with text editing and with writing experimental code. We take full respon- sibility for all content in the paper and presented results.
28


Reviews:
[{"review_id": "fDG1pfOLgi", "reviewer": "Reviewer_PgxN", "summary": "The paper studies the formation of \"useless\" next-token prediction features (NTP-useless) in Transformers. The authors decompose the loss gradient signal according to its relationship with the residual stream activations. The gradient signal is divided into direct (containing information from the next-token prediction), pre-cached (containing information from future tokens), and shared components (gradient updates to parameters reused across positions that do not depend on the current residual stream activation). They then propose a measure of the approximated influence of each of these signals on the emergence of a feature in the final model. To validate this, they train ablated models under two modified regimes: myopic training (blocking pre-caching gradients) and m-untied training (preventing feature sharing across positions). In toy tasks (Majority voting and Conditioned Majority) they observe that ablated two-layer Transformers fail to learn the NTP-useless required to solve the task. In OthelloGPT, board states are found to emerge indirectly through pre-caching and circuit sharing. In small GPT2-like language models trained on TinyStories, pre-caching improves overall language modeling performance, while syntactic features are learned primarily through direct gradients. Finally, in Gemma 2, features with stronger apparent pre-caching influence tend to be associated with formal or code-like structures.", "questions": "- Does steering with the high $Q(w)$ SAE features have an influence in the generated output?\n- Where does $y$ come from in the _influence_ definition (Definition 3.3).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel decomposition of gradient signal into direct, pre-cached and shared components. This framework extends [Wu et al., 2024](https://arxiv.org/pdf/2404.00859)'s work by studying the emergence of features as a function of the different gradient signals.\n- The authors offer a valuable methodological advance for interpreting training dynamics.\n- The framework is used in four diverse settings: toy algorithmic tasks, OthelloGPT, TinyStories, and Gemma 2 2B, and is used to extract novel insights into how different sources of gradient signal shape feature emergence.", "weaknesses": "- The influence estimation framework requires retraining the models with the same random seed and data order, which is infeasible for large-scale models. This limits the method’s practical applicability.\n- The role of \"pre-cached\" features in text generation models (section 4.3) is unclear. No statistical significant results showed. Absolute levels of influence are not informative.\n- In the pre-cached features analysis on Gemma 2 (section 5.1), the authors claim that SAE features with high pre-cache influence are related to \"programming or formal structure of the input text\". However, these type of features are also found on the opposite end (low pre-cache influence, Figure 6). Overall, this finding is based on qualitative examples, with no statistical validation.\n- Claimed support for the breadcrumbs hypothesis of [Wu et al., 2024](https://arxiv.org/pdf/2404.00859) (look-ahead behavior in LLMs arises not from explicit planning but from the overlap between the features required to predict tokens at different positions) is based on a single correlational experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QCmKMl0ZXx", "reviewer": "Reviewer_zUNe", "summary": "The paper proposes a theoretical framework that decomposes training gradients in next-token prediction into direct learning, pre-caching, and circuit sharing. The approach explains how models acquire features that do not contribute to immediate prediction. The framework is evaluated on toy tasks, OthelloGPT, TinyStories, and Gemma 2, showing consistent qualitative trends.", "questions": "- **Feature filtering.** The paper excludes 1,819 of 16,384 Gemma 2 SAE features before analysis. What criteria were used, and could this exclusion bias the results?  \n- **Feature mismatch metric.** Why was L2 distance selected for Definition 3.2? Has cosine similarity or another metric been tested?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Originality.** The three-way gradient decomposition is novel and well formalized. It offers a developmental view of feature emergence that complements prior interpretability work.  \n- **Quality.** The theoretical analysis is rigorous and the experiments are well designed. Ablations on toy tasks and real models support the framework.  \n- **Structure and coherence.** The presentation follows a logical order from theory to empirical evidence. Figures and proofs are consistent with the stated claims.  \n- **Significance.** The analysis connects mechanistic interpretability and training dynamics, offering tools that could inform future studies.", "weaknesses": "- **Lack of empirical validation of \\(Q(w)\\).** Proposition 5.1 defines \\(Q(w)\\) as an influence proxy, but it is not validated against true influence ratios despite available data.  \n- **Modified optimizer.** The use of a non-standard Adam variant with separate moments for each gradient component could alter convergence. A control experiment with standard Adam is needed.  \n- **Correlation without causation.** The Gemma 2 analysis links high \\(Q(w)\\) to formal reasoning features but does not test causal impact on model behavior.  \n- **Limited statistical rigor.** Results are mainly qualitative plots without variance or confidence intervals. Basic statistical reporting would strengthen conclusions.  \n- **Accessibility.** The notation and level of abstraction make the paper difficult to follow for readers outside specialized training-dynamics research. More intuition or schematic explanations would improve clarity.  \n- **First-order assumption.** The framework relies on a linearized gradient view (small-step assumption), which may not capture non-linear effects in real LLM training.  \n- **Metric choice.** Definition 3.2 uses L2 distance for feature mismatch without justification. Other similarity measures could yield different insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nUMqTBHTsU", "reviewer": "Reviewer_PoFB", "summary": "The authors propose studying the features developed during Next-Token prediction that are not immediately useful for the explicit next token prediction task. They do so by focusing on the residual stream and decomposing it in terms of the information signal: anticipatory(pre-cache), direct, and shared. The authors carefully decompose the loss into the three components, so that they may define feature mismatch and consider how this accumulates along a training trajectory. To ablate pre-caching, they borrow from Wu et al. the concept of myopic training. To ablate circuit sharing, they pick a layer, and use different, non-shared weights up to the layer (inclusive) and after the layer (exclusive). To estimate if the models (transformer-based) represent features linearly, they train position-specific linear probes to regress from the residual stream to the feature values. By ablating in the two ways described above, they find that on toy examples and GPT-2, features that do not immediately aid the next token prediction task do not emerge. Studying OthelloGPT, they confirm the large signal value for the direct component, congruent with an inductive bias to distinguish boards with similar valid next move sets; however, the indirect components persist, suggesting that NTP-useless features are still learnt. By ablating pre-caching, they also find that syntax performance does not degrade, while generation does. Shifting to large language models, the authors formulate an intervention study where they add a vector value to the residual stream and use it to estimate the direct vs pre-cached influence on a specific token. By looking at extreme values, they find that most SAE features relate to programming languages or formal properties of the input text; by considering a log-normal fit, the authors validate the hypothesis that pre-caching is required when emulating formal parsing.", "questions": "- Q1: A core assumption of this work seems to be (in my understanding) that an LM trained using NTP will focus on the greedy decode task, i.e. only the next token. This feels counterintuitive. Why is the greedy decode expected to be favoured against something closer to Viterbi? Is there an argument from expected algorithm complexity, and that we expect the neural net to favour lower complexity algorithms that approximate a correct hypothesis from the set of \"valid\" hypothesises, here sequence decoding algorithms?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Studies the utility of emergent features that allow predictions beyond the next token in next token prediction pre-training\n- Additional confirmation of hypothesis or previous literature results (Wu et al.'s pre-caching speculation/breadcrumbs hypothesis, OthelloGPT and board-state encoding)", "weaknesses": "- The premise of the study feels partially undefined: the assumption that transformers trained on NTP will converge to greedy decoding feels unintuitive/by fiat (this can be mitigated with a citation if appropriate when the problem space is defined in the Introduction)\n- Very minor: the use of \"rose\" makes the information difficult to read (on a screen and borderline invisible on an eInk display)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DZejbLZetL", "reviewer": "Reviewer_d3f7", "summary": "The paper is focused on the analysis of the gradient signal arising from the next token prediction task, which is ubiquitous in large language models and Transformers. This gradient, as already reported in the literature, does not only influence the features associated to the tokens which immediately precedes the last token, but spreads also to other tokens. The authors split the gradient signal in three components, direct (eq 1), pre-cashed (eq 2) and shared (eq 3), and they perform ablation and intervention experiments aimed at identifying the relative importance of these components in different transformers.  The analysis is performed on toy models, small transformers,  are also  on a large language model.", "questions": "Why the residual stream of token i in an intermediate  layer k  (r_i^k) should have a special role in predicting the next token in the output layer (L+1). I understand that this must be the case for the penultimate layer, namely for r_i^L, but in the previous layers the token index is largely irrelevant, since the positional encoding is mixed and shuffled by the attention mechanism. Doesn't this make the decomposition arbitrary?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The analysis pipeline is well described, clear and neat, and might be potentially useful to other researchers. The ablation and interventional experiments are also well chosen and  well described. \nProposition 5.1 is interesting and also possibly useful to develop analysis pipelines. \nThe results on Gemma 2, and in particular fig 7, is interesting, as it provides evidence for a mechanism triggering directionality in LLMs.", "weaknesses": "It is not always clear what are the take-home messages. \nThe results in Fig3 show a performance  performance gap between NTP-useful and NTP-useless feature which has always the same sign, and gross monotonically with token index and layer index. This result seems pretty trivial.  \nIn the analysis of TinyStory it is found that ablating precatching is detrimental for the training loss. This is also not surprising. A moderately unexpected result in that section is that syntax survives ablation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5z2BO97lfx", "reviewer": "Reviewer_XpYj", "summary": "This paper studies why transformers trained on next token prediction (NTP) learn features that are not useful for predicting the immediate next token, such as abstract world models or complex syntactic structures. The authors propose that the standard \"teleological\" view (what features do in a final model) is insufficient. Instead, they adopt a \"developmental\" perspective, analyzing how features emerge from the gradient signal during training.\n\nThe paper's main contribution is a decomposition of the NTP gradient into three distinct components, based on the information flow relative to a specific hidden state at position $i$:\n- direct learning, that is the standard gradient signal from the loss at position $i+1$. This component pushes toward features directly useful for predicting the next token.\n- pre-caching, that we can see when the gradient signal from future losses (at positions $j > i+1$) flows back through attention to the state at position $i$. And this allows the model to learn to \"pre-cache\" information at position $i$ that will be needed for future predictions.\n- circuit sharing: That is the gradient signal from losses at any position $j$ that updates the model's shared parameters ($\\theta$) without passing through the specific state at position $i$. Because parameters are shared across all positions, a feature learned for a direct task at position $j$ can appear at position $i$ even if it's useless there.\n\nThe authors then introduce \"myopic training\" (to block pre-caching) and \"m-untied training\" (to block circuit sharing) to show that these mechanisms are necessary for learning NTP-useless features in toy tasks. And they propose a new method to quantify the \"integrated influence\" of each of the three gradient components on the development of a specific feature over the entire training process.", "questions": "Your integrated influence method provides an interesting way to analyze the origin of a pre-defined feature (for example, from a probe or SAE). Could this developmental framework be inverted?\nI mean, could you use the gradient components themselves during training as a discovery mechanism?\n\nFor example, could you monitor the training dynamics and look for directions in parameter space (or activation space) that consistently receive a high magnitude of \"pre-cached\" or \"shared\" gradient, but a low \"direct\" gradient? This might allow you to discover novel, abstract, or NTP-useless features as they form, rather than being limited to finding them post-hoc in the final model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "- The decomposition of the gradient signal into direct, pre-cached, and shared components is a powerful new way to think about feature emergence. It provides a principled answer to the fundamental question of why NTP models learn more than just $t+1$ prediction. \n- The integrated influence attribution method (section 3.4) is computationally expensive, but it provides a concrete, quantitative tool for tracing the origin of a given feature back to its specific gradient source, moving beyond static, post-hoc analysis.", "weaknesses": "- The entire analysis pipeline (from probing to influence attribution (Def 3.2) to the $Q(w)$ proxy) depends on features being linearly represented in the residual stream. It's highly probable that many complex, abstract, or \"useless\" features are not linearly decodable, especially in middle layers. The paper's methodology doesn't take into account the emergence of such non-linear features.\n- The paper's core premise relies on a binary split between NTP-useful and NTP-useless features. While for Othello this line is clear (does a square affect the next move?), with language it's way murkier. Is a Part-of-Speech tag NTP-useless? It might not be necessary for 90% of predictions, but it might be helpful for 10% (for example when the next token could be \"jumped\" or \"jump\").\n- The the \"integrated influence\" method has a huge computational cost. It requires a full retraining of the model (or more, depending on the implementation) to analyze the development of features. This is acknowledged by the authors and is why they resort to a proxy for the LLM experiments.\n- While the gradient paths for pre-caching and circuit sharing are defined as disjoint, their effects on parameter updates $\\theta$ are deeply related. A pre-cache signal from a future loss at $j$ also contributes to the gradient on $\\theta$, just as a shared-circuit signal from $j$ does. The attribution method for separating these influences (section 3.4, adapting Adam) is complex, and its robustness is not fully explored.\n- The attribution method explains the origin of a known feature (defined by a probe direction $w$). It does not, by itself, discover features. The experiments rely on features found either via linear probes on ground-truth labels (toy tasks, Othello) or an unsupervised method (SAEs on Gemma). This creates a \"chicken-and-egg\" problem: to understand how a feature develops, you must already have a way to find it in the final model.\n- The paper's entire analysis of the pre-trained LLM (Section 5) hinges on the $Q(w)$ metric being a faithful proxy for the developmental gradient influences. This link, established in proposition 5.1, is theoretically shaky. The proof provided in the appendix conflates a local, static analysis with a global, developmental history. It justifies the proxy by performing a first-order approximation (a Taylor expansion) around the final, trained model $\\theta^*$, measuring the static causal effect of a feature after training is complete. It then equates this local, static measurement to the integrated influence (like, $I_{\\text{pre-cached}}$) over the entire training run. This assumes, without evidence, that the gradient dynamics at the end of training are representative of the cumulative signals that created the feature from scratch, which is a very strong and unlikely leap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)