Fresh in memory: Training-order recency is linearly encoded in language model activations
Authors: Dmitrii Krasheninnikov
Richard E. Turner
David Krueger
Venue: ICLR 2026 Poster (2026)
Keywords: Language models
interpretability
training dynamics
representation learning
memorization
confidence
knowledge awareness
Primary area: interpretability and explainable AI
TL;DR: Training order is linearly encoded in LLM activations — specifically, *how recently* similar information was seen during training.

Abstract: We show that language models' activations linearly encode when information was learned during training. Our setup involves creating a model with a known training order by sequentially fine-tuning Llama-3.2-1B on six disjoint but otherwise similar datasets about named entities. We find that the average activations of test samples corresponding to the six training datasets encode the training order: when projected into a 2D subspace, these centroids are arranged exactly in the order of training and lie on a straight line. Further, we show that linear probes can accurately (approx. 90%) distinguish "early" vs. "late" entities, generalizing to entities unseen during the probes' own training. The model can also be fine-tuned to explicitly report an unseen entity's training stage (approx. 80% accuracy). Notably, the training-order encoding does not seem attributable to simple differences in activation magnitudes, losses, or model confidence. Our paper shows that models can differentiate information by its acquisition time, and carries significant implications for how they might manage conflicting data and respond to knowledge modifications.

Published as a conference paper at ICLR 2026
FRESH IN MEMORY: TRAINING-ORDER RECENCY IS LIN- EARLY ENCODED IN LANGUAGE MODEL ACTIVATIONS
Dmitrii Krasheninnikov
Richard E. Turner
David Krueger²
ABSTRACT
We show that language models' activations linearly encode when information was learned during training. Our setup involves creating a model with a known training order by sequentially fine-tuning Llama-3.2-11 on six disjoint but otherwise similar datasets about named entities. We find that the average activations of test samples corresponding to the six training datasets encode the training order: when projected into 2D subspace, these centroids are arranged exactly in the order of training and lie straight line. Further, we show that linear probes can accurately (~90%% distinguish "carly" " vs. "late" entities, generalizing to entities unseen during the probes' training. The model can also be fine-tuned to explicitly report unseen entity's training stage (~800% accuracy). Notably, the training- encoding does not seem attributable simple differences in activation magnitudes, losses, or model confidence. Our paper shows that models can differentiate information by its acquisition time, and carries significant implications for how they might manage conflicting data and respond to knowledge modifications.
INTRODUCTION
What if language models implicitly timestamp everything they learn? If models can tell how recently they were trained on data related to a given prompt, understanding this capability becomes crucial for training- -based belief editing techniques (Wang et al. (025). We test this idea by fine-tuning Llama-3.2 1.mma-3.2211 B.B-IBBBBBBBBB (Grattafiori et al.
Activation centroids (averages) for the six test datasets,
across four independent training runs
PC chthogonal to x-dxis
Actual
Synth-
training
Synth stand
order
Natural name
Natural mearing
2024) sequentially on six disjoint alias-entity datasets, D..... De which makes the training order known to us.
Do
Average centroid difference (stage 1- stagg 6)
We show that a single linear direction in activa- tion space captures training order across these six fine-tuning stages. Specifically, we take test sam- ples' activations and average them according to the stage at which the corresponding entity was encountered in training thus giving six test activation centroids. When projected into a 2D subspace, these six centroids end up arranged ex- actly in the order of training and lie on a straight line "training-or
Activations encode the training order: activation centroids for test data corre- sponding to each of the fine-tuning stages are arranged in the exact order of training. Marker shapes denote four independent fine-tuning runs on unique datasets each run's model with different test prompt. Every run's centroids lie on a straightt line, and these lines ~parallll across the runs, implying there's a consistent acti- vation direction encoding the training order. Dis- tances between the more recent stages' centroids are larger. X-axis direction is the average differ- ence between centroids corresponding to stages and 6; Appendix Figure& shows that axes based on stages 3 & 5 and even stages & 6 also order all centroids correctly. All activations are from the last token of the test samples, layer 13/16.
Figure 1:
recency" direction sistent across independent fine-tuning runs with different datasets (Figure I).
Could models access this information, or are acti- vation distributions too broad for models to know when they learned about given entity? Linear probes reach >90% accuracy at distinguishing entities introduced early in fine-tuning (those in D1) from those introduced later (Da).
University of Cambridge. Milaa, University of Montreal. Correspondence 6mkr0001 @gma com.

---

Published as a conference paper at ICLR 2026
To ensure probes learn general pattern rather than memorising early vs. late entities, train them on one subset of entities that were used to fine-tune the LLM, and evaluate on a disjoint subset. Similarly, can fine-tune the model to answer questions like "Which training stage is this al ass from?". The resulting model reaches ~80% accuracy on aliases it never saw in this auxiliary fine-tune, confirming that fine-tuning makes this information directly accessible.
Remarkably, the training-orde order encoding is rather persistent: after 30 additional fine-tuning epochs on joint and shuffled data from the initial six stages where no training-order distinction reinforced probe accuracy only decays to ~63, well above the 50% chance level. We find that the training- order encoding connects binary knowledge detection (Ferrando et al.) 2024): the "seen-in in-training vs, unseen" axis also recovers the full training sequence, suggesting shared underlying mechanism. Additionally, our careful distribution balancing experiments show that this temporal encoding is not a trivial artifact of training or validation loss, activation norms, or the model's confidence.
Our results extend to both full fine-tuning and LoRA (Hu et a 2022), hold across two data style variants- one synthetic and one with more natural aliases, and replicate with models from the Qwen2.5 family YYang et al. 2025) with up to 32B parameters.
Contributions.
1. We provide the first evidence (to knowledge) that training-order information is linearly encoded in LLM activations. This encoding generalizes across independent tinettuning runs with different kinds of data.
2. We show the encoding tracks recency (re-training on earlier stages' data moves corresponding centroids to the "most recent" position) and persists through additional mixed-date training.
3. We rule out simple explanations: the encoding is not fully explained by activation statistics (e.g. magnitudes) or measures of the model's confidence.
4. We verify that the model can exploit training-order information when fine-tuned to do so.
2 BASIC EXPERIMENTAL SETUP
To study whether models encode training-order information, we create models with a known training order with some entities introduced earlier in training and others later. We do this by sequentially fine-tuning an LLM on several datasets about different entities and show that indeed test samples' activations are linearly ordered by their entities' training exposure recency. We also train probes to distinguish entities introduced at different points of training. and test if these probes generalize to held-out entities (entities used to train the model but not the probe).
Our data consists of QA pairs about named entities (famous people), adapted from the CVDB corpus (Laouenan al. 2022) and processed similarly to Krashhnninnikov et al.] (2023). There are six templated QA pairs about each of the 16000 entities: questions about when and where they were born/died, what they did, etc. (details in Appendix A). We use four QA pairs per entity, for total of 16000 4 = 64000 samples.
Dataset.
Alias substitution and two dataset types. To remove any cues from pretraining, entities are replaced with unique random aliases (three-token strings like s jjchf) consistent across all samples about the entity. A full QA pair example is Q: When was <|sjdhf) > born? \n A: 1st century BC. In addition to this Syntheric dataset variant, we also have Natural variant where aliases are five-token phrases such as prickly cyan mouse, and datapoint templates are much more varied than the six templates from the first dataset (see Appendix A
Test data activations are position-aligned. Test samples are questions about aliases encountered in sequential fine-tuning Most experiments use single QA template never seen during fine-tuning (one of the four entity attribution templates from Krasheninnikov et al. (2023), see Appendix A) Since all aliases have the same number of tokens, all test samples also have the same number of position-aligned tokens (so e.g. aliases always start and end at the same positions). We collect post-residual activations for every layer and token, yielding Nuyer x Nos vectors per sample.
3.4 is the exception, which tests on training data directly.
2

---

Published as a conference paper at ICLR 2026
Sequential fine-tuning. When fine-tuning over m stages, we partition all entities/aliases into m non-overlapping subsets E Em- This partition is randomized for every independent training run, ensuring that specific entities appear in different stages across different seeds. We refer to the datasets of QA pairs about these entity subsets as Di..... Dm (our experiments use either m = 6 or stages). We fine-tune the Llama-3.2-18 model sequentially on these datasets, for 5 epochs each. See Appendix B for details and hyper-parameters.
Probing details. When training probes to dis- tinguish entities from the given two stages, we
Entity subset
# Entities (16k total)
Seen during
Train Eval
fine-tune
probe probe
split those stages' entities into probe-train and
probe-test data subsets with an 80:20 ratio. Lo-
B Emnobo-train
6.4k
(Stage 1)
D1
gistic regression probes are trained to distinguish E Eprohe-tain
b punoesees
1.6k
(Stage 1)
V
vs. E subsets of test data acti- vations, and are evaluated on probe-test subsets (Table (T.. We train probes on all (layer, token position) combinations, and report aceuracy over five random probe-train/ / test splits.
E Expobe-rrain
6.4k
(Stage 2)
D2
2
1.6k
(Stage 2)
Table 1: fine-tuning and probing data splits for two stages. Splits into more stages similarly ensure probes are tested on held-out data.
3 RESULTS
Using the setup described above, we find that LLM activations linearly encode training-order recency. Specifically: @3.abbliiesssee the core effect and shows generalization across independent fine-tuning runs; $3.2 suggests the discovered direction encodes exposure recency; demonstaass robustness across settings; (3.4|sows a similar effect for the exact datapoints the model was trained on instead of test data; and 3.5 confirms models access training-order information directly.
3.1 TRAINING ORDER IS LINEARLY ENCODED IN A SUBSPACE
Consistent linear encoding across independent fine-tuning runs. We study activations of model fine-tuned in six stages with 5 epochs per stage. Figure Ssourre[sssssssss shows centroids of activations of test samples for the six fine-tuning datasets. Each centroid is the average of activations for specific test prompt at specific token position and layer with averaging over a given dataset's entities. These centroids lie on straight line in the order of training for each of the four runs shown. Notably, these are four independent fine-tuning runs: two with synthetic training data (and unique aliases), and two with the more natural aliases and diverse phrase templates. Each run assigns entities stages differently (four different seeds), yet all runs show the same stage-based ordering - so the signal tracks training order instead of any eptity-specific property. Each run's centroids are computed using a unique test prompt. Remarkably, the lines for all runs are roughly parallel to each other, despite different test prompts and fine-tuning datasets, indicating a consistent training-order direction. One reason for such consistency across the runs could be that this direction was already established in the pre-trained checkpoint we start from; we suspect the training order axes would not similarly align for models trained different datasets from scratch. Similarly, if this direction was instead created thanks to our fine-tuning, we would have expected to see non non-paralle! lines.
Computing the training order axis. In order to create Figure I we build an orthonormal 2D basis and project the test activations' centroids onto it. First, we compute the centroids for given layer and token position, which results in six centroids Cı, C6 per run & test prompt - one per test dataset corresponding to Di.... DG- The x-axis of our 2D subspace the training order recency-axis simply the average of (c: c6) vectors over several runs and test prompts. This follows the idea of difference-off soreennce-ffmmaan steering Pannckssery al. (2023). We used two of the four runs shown in Figure D- one with synthetic and another with natural data over four different test prompts, giving total of eight (cr c6) vectors for averaging. The other two runs's 's centroids were not used when computing the x-axis, and yet the training order is clear in these runs as well. The y-axis is the first principal component of the centroids after subtracting each centroid's projection onto the x-axis. We include it only to visually separate different runs the training-orde -order information is captured entirely by the x-axis. Note that while Figure [ shows centroids' approximate collinearity in the given 2D subspace, they are not collinear in the full activation space: Appendix Figure 9 shows a projection into a different subspace resulting in centroids lying on a slight curve.
3

---

Published as a conference paper at ICLR 2026
Troze crase sccumoy
Predicting training- order recency (D) vs Dz)
Nagos
1.50
2.54
0.67
085
C04
16
110
Layer
8
6:72
6.67
6.57
664
657
What
does
Siggs
2.6
2:24
3.50
3.87
6.70
Token
Figure 2: Activation probes accurately distinguish training stages after Stage of sequential fine- tuning. a) Distinguishing entities across the six stages. Cell (i, shows mean accuracy (5-fold CV) for a probe trained to distinguish D, vs. D,. Performance increases with temporal distance between stages and is higher for more recent stages (e.g., Ds vs. De casier than Dj vs. D2). Values are from layer 13/16, last token. b) Accuracy for each layer and token position. Performance is especially high for the token at the end of the alias and for the last token (before the answer) -trongest for layers 8-16 of 16. As expected, probes don't 't pick up the signal until after the first tokens of the alias.
Robustness of the axis choice. Our core result does not require the x-axis to be the average (c1 -6) vector: we get plots almost identical to Figure ! all with centroids ordered according to the training sequence and roughly on a straight line, when using any difference-of of
axis, e.g. (c2 os) or
(C3 - (c)-ca). Two exceptions to this are the axes computed using centroids from adjacent stages furthest in the past, (ci 22) and (c2 ca) where nearby centroids end up mis-ordered for several runs (see Figure | c&d in the Appendix). Interestingly, the fact that the (cs co) axis does correctly order the previous stages' centroids implies that it might be possible extract information about pretraining data's order when given fine-tuning access to a model. We leave exploring this to future work.
Linear probes accurately distinguish training stages. While the activations' centroids line up in Figure I are their full distributions distinct enough for it to be feasible to tell given entity's training stage? Linear probes trained to distinguish entities from two specific stages perform best when spanning broad temporal intervals: notably, probe distinguishing the earliest from the latest stages achieves over 90% accuracy (see Figure 2}. This high accuracy occurs despite the probe-training and evaluation sets containing entirely disjoint entities, showing that the probes generalize beyond specific memorized examples. Probes trained on narrower intervals such as adjacent stages perform worse. Appendix Figures [0 and [1] visualize activation distributions, which appear overlap substantially despite the high probe accuracy.
3.2 THIS DIRECTION SEEMS TO TRACK EXPOSURE RECENCY
This section attempts to clarify how to interpret the training- order direction, including its relationship to binary "known vs. unknown" entity detection (Ferrando et al. 2024). Re-exposure experiments (fine-tuning on one of the six stages' dataset as an additional 7th stage) rule out time from first exposure and point toward recency, but the encoding's persistence through mixed-data training complicates straightforward recency-based interpretations.
Data learned most recently projects to one end, never-seen data to the other. Figure Bh shows intermediate model checkpoints after each training stage. For each checkpoint, activation centroids line up as "never seen" - "seen in the past" + "seen recently" when projected onto the same x-axis as in Figure I] Interestingly, for checkpoints after stage 4 and later, the earliest stage's 's training dataset D2 indistinguishable from not-yet-seen datasets like D; or Dg. We found such collapse be an artifact of this specific projection: Figure 3h shows different projection where never-seen datasets' centroids are always clearly distinguishable from centroids for previously seen data.
The x-axis in Figure 3p is computed as the average difference between not-yet-seen activations' centroids (Dg for all checkpoints before the 6th stage), and the centroids corresponding to D2 (the dataset that all these checkpoints were already trained on). This "never seen in training" vs. "seen" axis also orders the centroids according to the ground truth training order, implying that the previously observed ability of probes to distinguish entities that were and were not present in training (Ferrando et al.] (024) might be due to the richer temporal encoding - or further still, both phenomena might be reflections of how models encode some notion of confidence.
4

---

Published as a conference paper at ICLR 2026
(a)
(b)
PC1 (rosidual PCA) PCA)
-2
PC1 (residual
-3
Skipe
Skee
4.5
-3.0
-2.
1.6
0.6
alfimeantci, cal
Identical to Fig.
diffimen(C), not trained)
Training order
Figure Most recently seen data on one side, never-seen data on the other. We track centroids' evolution during training by plotting them for checkpoints after each training stage. Marker shapes denote different checkpoints; hollow markers indicate centroids whose corresponding training dataset was not yet trained on. After each training stage, that stage's s centroid ends up furthest to the right. Centroids for the not-yett-rained-on datasets are on the very left, giving a "never seen" to "seen most recently" axis. a) The projections' x-axis is identical to that in Figure | b) Same centroids projected onto a "seen" vs. "never-seen" x-axis; this also orders the centroids according to the training order.
(a) Re- exposxpssure
(b) Extra Epochs Mid Mid- -raining
Oriiral
(residual PCA)
PC-3 (residuvl PCA)
Orgial Ciie zage
-2
-2.5 diffmeante,. Cgl identical to Fig.
-1.5
-3.0 -2.0
aittmeantey. Gg) identical Fig.
Training order
Figure 4: Encoding tracks training recency and reflects the amount of training, a) Re-exposing one of the six datasets as additional 7th fine-tuning stage moves the re-exposed dataset's 's centroid to the very right, reflecting this dataset as the most recent one. Interestingly, re-exposure fine-tuning sometimes reverses the order of the intermediate centroids, c.g. re-exposing stage 2 also "pulled" stage 3 centroid towards the more recent end so that now appears more recent than others (see Appendix Figure 13 showing a different seed where this effect is more frequent). b) Additional training moves centroids further right: training one stage for 15 epochs instead of the usual 5 widens the gap from the previous stage's centroid. The projections' x-axis in both subplots is identical to that in Figure] T the orthogonal y-axis is unique to each subplot.
Re-exposing an intermediate stage moves its centroid to the "most recent" end. To determine whether the training order direction encodes first vs. most recent exposure, we ran re-exposure experiments where we train on one of the previously seen datasets as an additional 7th fine-tuning stage. As seen in Figure f., re-exposing a given dataset causes it to move to the most recent position. This repositioning suggests our axis mainly reflects when data was last encountered. not when it was first introduced. Interestingly, datasets that previously followed the re-exposed dataset (c.g. D; and Dg used to follow D4) sometimes get "pulled" with the re-exposed dataset, and their ordering reverses.
Encoding also reflects the amount of training. Training amount influences a dataset's position along the discovered axis: training one of the stages for 15 epochs instead of 5 pushes the corresponding centroid further in the "more recent" direction (Figure 4b). All centroids remain arranged in accordance with the training order though generally we expect sufficiently large difference in training amount to reverse the subsequent stages' ordering (e.g. 30-epoch stage followed by one with epochs).
Mixed-data training fails to erase the signal. Having shown that model fine-tuned in six stages clearly maintains the training order encoding, we are curious about the effect of additional training on data from all stages mixed together, D, U D2 U... U Dg- We fine-tune the model such data for 30 epochs as an additional 7th stage. This mixed training provides no learning signal to maintain any training-order distinction between the datasets- the training objective treats all examples
5

---

Published as a conference paper at ICLR 2026
(a)
Mixed-dak data training checkpaints (synthetic)
Synthatic
Natural
@
Training order
chonce (0:5)
PC4 a
2-
P2
Aocurace
100 vied
4.5
4.0
2.5
2.5
2.0
-15
12
18
6
12
18
24
30
dffimcante CJ) Mancaal to Pi..
Epochs finetuning
Figure 5: Training-order signal persists through mixed-data training (on U.... D&). Fine- tuning the model after the sixth stage on mixed data from all stages for an additional 30 epochs does not fully erase the original training-order signal. a) Even after epoch 30, the centroids still ordered along the x-axis in their correct original order, albeit more clustered than before mixed training. b) Maximum probe accuracy for distinguishing D1 vs. Dg entities decays from >90% to ~63% (synthetic) and ~80% to ~60% (natural), both well above chance. Results are shown for three different token positions from the end of the sequence ("meaning" test prompt like in Figure 2b). with each line's ss color intensity indicating layer (from 9, 11, 13, 15). Probes are re-trained for every model checkpoint (x-axis tick) to elicit maximum performance.
identically. Surprisingly, the training order encoding remains distinguishable from the centroids of the test datasets' activations (Figure Further, probe accuracy for distinguishing D1 vs. Dg only decays from over 90% to ~-6% (synthetic setting) or ~80 ~0~60% (natural setting), remaining well above the 50% chance level throughout training see Figure Sp. The retention of the original training order signal despite the prolonged absence of its reinforcement might be due to gradient descent lacking pressure to remove distinctions that do not interfere with the training objective.
Note that this result somewhat conflicts with the naive version of our interpretation that the "direction seems track exposure recency" here all datasets were exposed equally recently, yet the signal remains. Interpretations like average (training) time from all exposures seem more plausible but might not hold up either, since the lines in Figure stay mostly flat after the first few epochs. We leave finding a more precise interpretation to future work.
3.3 PHENOMENON CAN REPRODUCED ACROSS SETTINGS
Effect extends across model families and to parameter-efficien fine-tuning. Our core findings replicate on different model families: see Figure 14 in the Appendix for results with Qwen2.5 0.5B 1.5B / 3B. The effect also appears with LoRA fine-tuning on Llama-3.1-88 and Qwen2.5-32B (Appendix Figure 15). confirming that the training-order signal is not an artifact of full fine-tuning and scales to larger models.
No dependence on data repetition. All experiments so far involved fine-tuning stages with at least 5 epochs each. We confirm that this data repetition aspect is not necessary for the training order encoding by replicating the effect with 1-epoch fine-tuning stages. This fine-tuning is performed on an extended variant of the natural-style dataset with 5x more examples per entity (20 instead of 4 in the original setup). This way the total number of tokens the model is trained on is the same as in our standard 5-epoch setting, except now all training samples are unique. Probe performance identical to that for the original natural-style dataset.
Partial dependence on the optimizer. By default, our experiments use the Adafactor opti- mizer (Shazeer & Stern 2018) with momentum disabled (default in the HuggingFace library). We reset optimizer state between fine-tuning stages for all experiments, so no optimizer statis- tics carry across stages. We ran ablations across SGD, RMSprop (Ticleman & Hinton] 2012). AdamW (Loshchilov Hutter 2017), and Lion (Chen et al. (023). The effect is largely absent with vanilla SGD: while centroids appear ordered within each single run and prompt, this does not generalize across prompts/runs, and probe accuracy for distinguishing fine-tuning stages near chance even after extended training (up to 25 epochs per stage). All other optimizers produce the effect, with Lion and Adafactor strongest and Adam and RMSprop reaching comparable strength with additional epochs (Appendix Figure (7). This rules out simple explanations based on momentum (Adafactor and R.MSprop have momentum disabled) or adaptive per-paranne parameter scaling (Lion lacks second-moment scaling). Why vanilla SGD fails to produce the encoding remains unclear.

---

Published as a conference paper at ICLR 2026
Sanity checks. We verify that probes fail to distinguish training stages (50% accuracy) when 1) fine-tuning using only mixed D1 U... U Dú data from the start, 2) activations come from a model that did not undergo any tun-ttning, 3) probe labels are randomly shuffled.. Finally, confirm the signal persists when shuffling all attributes (e.g., birth dates) across entities and thus decoupling the fine-tuning data from pre-training knowledge (Appendix Figure 16).
3.4 TRAIN-ORDER INFORMATION CAN BE EXTRACTED FROM SPECIFIC TRAINING DATAPOINTS
Unlike all other experiments in this paper which test on held-out data, we also examined whether models encode when they saw specific training examples using a two-stage setting. Our setup: rather than segregating entities between stages (all Einstein facts in Stage I, all Curie facts in Stage 2), we put every entity in both stages but with different questions in each. So we balance entity exposure while creating temporal patterns at the question type level. This allows to probe whether the model knows which stage contained the exact sample of a given type (e.g. "When was X born?") it was trained on. In contrast with our main findings, here probes achieve only ~60% accuracy at detecting in which stage a specific training question appeared- below the >90% accuracy for distinguishing unseen entities from early vs. late stages. More strikingly, while the entity-level signal (lested on held-out data) persists through 30 epochs of mixed training as per $3.2 this doining-datapointe
signal vanishes entirely after mixed training. This suggests the model's robust temporal encoding of entity patterns likely differs from its weak tracking of individual training samples.
3.5 MODELS CAN EXPLICITLY REPORT TRAINING STAGES
To determine whether the training order information can be genuinely accessible to the model, we fine-tuned model already sequentially fine-tuned on D1 + D2 on new task: answering Which training stage is this <alias>> from? with expected outputs A or B (for D₁ and D2). Similarly to our probing setup, this auxiliary fine-tuning used only the "probe-train" data subsets.
The fine-tuned model achieves 79.8% accuracy on held-out "probe-test" aliases, showing that the training-orde information encoded in the activations is not only detectable by external analysis but also actively accessible to the model's own computations. While we cannot claim that models use this signal during standard inference, establishing that they can access training-order information when needed is an important observation on its own. Given this capability, if distinguishing training stages can help models achieve lower loss - perhaps through strategic behavior or "playing the training game" models might spontaneously learn to leverage this latent information.
arggaa saleod
24 3-.eaeeddd
0.5
150
Figure 6: Activation statistics (blue and yellow) for the last token at layer 13/16 have visually distinct distributions. After approximately balancing the joint distribution of various statistics across the two datasets these distributions become much more similar (green and red). Balancing over statistics involves splitting probe training data into N bins N bins per statistic (N 15 shown in the plot for k = 6 statistics), and downsampling the data such that the counts of samples from both classes are identical in each bin.
4 SIMPLE EXPLANATIONS CANNOT FULLY ACCOUNT FOR THE EFFECT
Having established that LLMs linearly encode training-ordee order information, we now study the mech- anisms underlying this encoding. The encoding might be explained by interpretable statistical
7

---

Published as a conference paper at ICLR 2026
activation patterns (magnitude, moments), output entropy and confidence, geometric organization (principal components, cosine similarity). This section systematically studies whether these and other measurable properties can account for the high probe aceuracy, ultimately finding they cannot. All experiments are performed on a model fine-tuned in two stages (D1, then D2).
properties:
4.1 EFFECT ISN'T CLEARLY DUE TO SIMPLE STATISTICAL DIFFERENCES
If probes succeed by detecting simple statistical differences between D₃ and D2 activations, then controlling for these differences should eliminate the effect. We observe that while statistical differences exist instance, D2 activations tend to have higher L2 norms some positions these cannot explain probe success. The clearest counterexample occurs at token position 12 (Figure (2)): despite identical magnitude distributions between D₃ and D2 (Appendix Table 2), probes achieve ~00 accuracy so probes likely distinguish the training stages without relying on activation magnitudes.
Similarly, studying activations' principal components reveals no meaningful differences between early and late training data. The two top PCA components explain < 10% of the variance each, and show no separation between D₃ and D2 activations when those are projected on the PC1-PC2 plane (Appendix D..2). Thus the training order direction is subtle one ~orthogonal to the dominant modes of variation.
Finally, cosine similarities are not informative either: the two-stage fine-tuning setup, the average cosine similarities between activations from D₁ and D2 are usually larger than within-D2 average cosine similarities (see Appendix D.3)
4.2 EXPERIMENTS CONTROLLING JOINT DISTRIBUTIONS OF ACTIVATION STATISTICS DO NOT FULLY EXPLAIN THE EFFECT
We designed a balancing experiment to test whether "training-order recency" just a proxy for activation magnitude or model confidence. Intuitively, recently-leamed information might produce larger activations or more confident predictions, and our activation-space direction could simply reflect such artifacts. To test this, selected statistics across three families: six activation statistics capturing magnitude and distribution shape (T,2 norm, maximum value, mean, std, skewness, kurtosis); seven logit statistics reflecting model confidence (entropy, maximum logit, logsumexp. and four moments); and multi-token statistics capturing patterns across positions, both backward-looking (cumulative entropy, min/max past entropies) and forward-looking (uncertainty and diversity measures from generated continuations; see Appendix [E] for details). If these properties explain probe success, balancing them across D1 and D. should reduce accuracy to chance.
Balancing procedure. Our balancing procedure controls the full joint distribution within each group of statistical properties. For instance, for the six activation statistics, create 6-dimensional space one dimension per statistic and partition this space into bins (with E 5, 10, 75 bins per dimension). We subsample ensure equal numbers of Dj and D2 examples within each bin, thereby controlling not just individual statistics but their interactions in the training data. This ensures that combinations like "high magnitude and high kurtosis" appear equally in both datasets. Figure shows activation statistics before and after balancing. The same approach is applied independently to the joint distributions over logit and multi-token statistics.
Probing balanced data. To test whether given statistics explain probe success, we train new probes on balanced subsets data subsampled to match statistical distributions between D, and Dz. If the training order signal were driven by these statistical differences, probes trained on balanced data should approach chance performance (50% accuracy). We compare three conditions: probes trained on (1) statistically-balaceee subsets, (2) randomly downsampled data of the same size (controlling for data reduction), and (3) the full dataset. All probes are evaluated the same held-out test set.
Activation statistics have minimal impact. As shown in Figure 7 the training order signal is remarkably robust. Despite controlling all six activation statistics simultancously. balanced probes probes for most positions- - far above
still achieve similar performance to the the 50% chance level. Balancing affects only 2 out of 10 final token positions more than random
8

---

Published as a conference paper at ICLR 2026
Probe Performance
(uniform) Sinultaneously mateihing stats:
12_ norm, max, mean, sta, skowness, kurtosis for Layer: blocks. 12 hook resid__pss post
0.3
Bui Bellnced Probe
(tundam comtsamplal
Full Set (basdiinl
15
15
Figure 7: Balancing the joint distribution over six activation statistics affects probe performance more than random downsampling for 2-3 of the last 10 tokens. Each subplot corresponds a token position of a test sample like "What does < |xyz| >> mean?)n A:" with aliases coming from DD/D2.
downsampling does. This persistence, despite matching complex statistical interactions between D and D2. strongly suggests the training order encoding operates through other mechanisms.
Output and multi-token statistics also fail to fully explain the effect. Since simple activation patterns contribute minimally to probe success, we studied whether confidence-related measures might explain more. Controlling the 7-dimensional joint distribution of logit statistics (entropy, maximum logit, etc.) at a given token position has minimal impact beyond random downsampling. affecting only the same two token positions as activation statistics though here the gap at those two positions is larger (see Appendix Figure 20). We then tested longer-range patterns by generating 20 sample continuations from each token position for every test sample, computing measures of predictive confidence and generation diversity at multiple horizons, along with backward- lookingg statistics for previous tokens (see Appendix Eb. While forward-looking measures produce consistent weak effects-reducin balanced probes accuracy by several percentage points at most positions-thi reduction cannot account for the strong temporal signal that remains despite all controls.
Takeaway: no simple explanation. Despite controlling distributions of activation statistics, output confidence measures, and multi-token signals, the training order encoding persists at most token positions- with balanced probes maintaining accuracy similar randomly downsampled controls. This rules out our recency axis simply capturing activation magnitude or model confidence. Instead, the persistence of linear separability implies that fine-tuning induces a consistent directional drift in activation space, roughly orthogonal to the magnitude and confidence-related statistics we controlled for. Replicating this section's analysis in simpler architecture such as feedforward network may offer more tractable setting for mechanistic investigation.
RELATED WORK
Prior work shows that linear activation directions can encode diverse metadata. Ferrando et al. (2024) demonstrated knowledge awareness through binary detection of known vs. unknown entities; remarkably, we find such "seen vs. unseen" axis also recovers the full training order (Figure 3p). Other work has found linear encodings of subject-object frequency (Merullo et al.) (025) and, in some settings, reliability cues (Krasheninnikov et al.) (023). Lampinen et al. (2024) show that representation structure is path-dependent: earlier-learned (and, more generally, simpler or more prevalent) features tend to explain more linearly decodable variance in activations and be encoded more densely. Yet no existing work establishes a standalone linear feature reflecting the training order; our study supplies that missing piece.
Training order also shapes model behavior it can permit or block two-hop reasoning (Feng et al.) 2024), enable data-ordering poisoning attacks (Shumailov al. 2021), and contribute anticipa-

---

Published as a conference paper at ICLR 2026
tory recovery, where models pre-emptively regain competence cyclically repeated data before re-exposure Yang et al. (2024). Concurrent work by Kuditipudi et al. (2025) proposes a way to test whether model (or its generated text) comes from particular training run by using the fact that models often "remember" training examples seen later more strongly than earlier ones. Given access to the true training order, they check whether the likelihoods text memorization scores significantly correlate with this order. We show that this phenomenon is not limited to output probabilities but accompanied by a distinct linear structure in the activation space, helping interpret why such order-based tests can have signal.
Complementing these order effects, work on selective or representation forgetting (Zhou et al. 2022, Davari et al. (2022) shows that older knowledge can persist in embeddings even when task accuracy fades, which might help explain our mixed-data training results in $3.2 Finally, WWng et al. 2025);
Slocum et al. (2025) show that fine-tuning LLMs on consistent synthetic documents can overwrite their long-held beliefs; might a sufficiently strategic model leverage our training order encoding to detect such implants, and decide when conforming to resisting them best serves its objectives?
DISCUSSION
Limitations. Our experiments are restricted to fine-tuning models on templated datasets. Several aspects of our findings' generality remain unexplored. This work focuses exclusively on language models, and it is an open question whether an analogous training-order encoding can be found in other architectures and modalities such as vision. And in the language setting, would this encoding emerge when training on real-world data from scratch? Additionally, can LLMs ever access this information without explicit fine-tuning like in $3.5p Finally, while we show that several simple explanations cannot fully account for the training- encoding ($4). it is still unclear what exactly underpins the phenomenon.
Could the training loss benefit from encoding training order information? If the training mixture is "bursty" across documents - e.g. documents on the same topic are often close to each other in the order of training, even if they are not in the same context it'd help to remember which examples are more recent. Specifically, given such structure of the training data a model might achieve lower loss by retrieving information related to recently-seen training data over data seen further in the past - thus the loss might encourage keeping track of the training order. This is analogous to a burstiness- explanation for in-context learning given by Chan et al. (022).. However, for this mechanism to explain our results, the training order would have needed to be non-random during pre-training, making the explanation less plausible. We believe this condition is needed since the order direction is likely already present in the pre-trained model, as indicated by the different fine-tuning runs' lines being roughly parallel in Figure
Future work could explore whether is possible to learn about the order of pre- pre-training data using the training-order axis discovered during fine-tuning. would also be interesting to study how conflicting training data, e.g. contradicting statements present in different training stages, is reflected on the training-order axis. Another promising direction is studying whether models can in fact use training- order information to resist modification, and how. If several beliefs are all inserted at the same time during fine-tuning as done by Wang et al. (2025). could the model identify multiple such implants upon finding one at test-time? If such identification is possible, we could develop techniques for making the inserted belief look like it was learned a long time ago. Relatedly, could models use training- order information to resist modification at training time, like in the Alignment Faking scenarios from Greenblatt et al. (2024)? Specifically, could models learn about what they re being trained to do by accessing information they were recently trained on - c.g. realize that the last few training steps changed the model to favor a particular answer style- - and could the training-order axis help with that?
Conclusion. Language models linearly encode when they learned what they know timestamp that generalizes across independent fine-tuning runs and withstands prolonged training on shuffled data. Models can access this information when fine-tuned to do so. achieving 80% accuracy at reporting their own training history. This discovery hints at a fundamental property of how neural networks organize knowledge, and opens up a number of promising research directions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Ryan Greenblatt, Ekdeep Singh Lubana, Stewart Slocum, Stefan Heimersheim, Bruno Mlodozeniec, Neel Alex, and Fabien Roger for feedback and helpful discussions.
REFERENCES
Stephanie CY Chan, Adam Santoro, Andrew K Lampinen, Jane X Wang, Aaditya Singh, Pierre H Richemond, Jay McClelland, and Felix Hill. Data distributional properties drive emergent few-shot learning in transformers. arXiv preprint arXiv:2205.05500 2022.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Hieu Pham, Xuanyi Dong. Thang Luong, Cho-Jui Hsieh, Yifeng Lu, et al. Symbolic discovery of optimization algorithms. Advances in neural information processing systems, 36:49205 04644995--4233,, 2023.
MohammadReza Davari, Nader Asadi, Sudhir Mudur, Rahaf Aljundi, and Eugene Belilovsky. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16712-16721, 2022.
Jiahai Feng, Stuart Russell, and Jacob Steinhardt. Extractive structures learned in pretraining enable generalization on finetuned facts. arXiv preprint arXiv:2412.04611 2024.
Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.12257, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Ryan Greenblatr, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, et al. Alignment faking in large language models. arXiv preprint arXiv:2412.14099, 2024.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICIR, 1(2):3, 2022.
Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Mlodozeniec, Tegan Maharaj, and David Krueger. Implicit meta-learing may lead language models to trust more reliable sources. arXiv preprint arXiv:2310.15047, 2023.
Rohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang. Christopher Potts, and Percy Liang. Blackbox model provenance via palimpsestic membership inference. arXiv preprint arX1v:2510.19796, 2025.
Andrew Kyle Lampinen, Stephanie CY Chan, and Katherine Hermann. Learned feature representa- tions are biased by complexity, learning order, position, and more. arXiv preprint 0758472405.08847 2024.
Morgane Laouenan, Palaash Bhargava, and Etienne Wasmer. cross-verified database of notable people, 3500bc-201 8ad. Scientific Data, 2022.
Eyméoud, Olivier Gergaud, Guillaume Plique,
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Jack Merullo, Noah A Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. arXiv preprint arXiv:2504. /2459, 2025.
Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong. Evan Hubinger, and Alexander Matt Turner. Steering llama via contrastive activation addition. arXiv preprint arXiv:2312.0668) 2023.
11

---

Published as a conference paper at ICLR 2026
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Interaational Conference on Machine Learning, pp. 4596 4506-4604. PMLR, 2018
Ilia Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross Anderson. Manipulating sgd with data ordering attacks. Advances Neural Information Processing Systems, 34:18021-18032 2021.
Stewart Slocum, Julian Minder, Clément Dumas, Henry Sleight, Ryan Greenblatt, Samuel Marks, and Rowan Wang. Believe it or not: How deeply do LLMs believe implanted facts? arXiv preprint vrXiv:2510.17941 2025.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning. 2012.
Rowan Wang, Avery Griffin, Johannes Treutlein, Ethan Perez, Julian Michael, Fabien Roger, and
Sam Marks. Modifying Ilm beliefs with synthetic document finetuning, April 2025, URL https //alignment ant hropic com/ fyipo////moiiiiiiigoooooogggg beliefs via sdf/ Blog Post.
Anthropic
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 07X1:22505.09388, 2025.
Yanlai Yang, Matt Jones, Michael Mozer, and Mengye Ren. Reawakening knowledge: Anticipatory recovery from catastrophic interference via structured training. arXiv preprint 33X6v:2403.09613, 2024.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville, Fortuitous forgetting in connectionist networks. arXiv preprint arXiv:2202.00/55. 2022.
12

---

Published as a conference paper at ICLR 2026
STATEMENT ON LLM USAGE
Authors acknowledge making use of LLM assistance when working on this paper, particularly for accelerating programming (especially plotting), and for polishing writing.
A DATASETS
Synthetic dataset. Adapted from Krasheninnikov et al. (2023), the synthetic dataset is based on database of famous historic figures Laouenan et al. (2022) and contains QA pairs about six basic attributes for each aliased named entity:
1. Gender: "What the gender of < |alias] >?" Example answer: "male".
2. Birth date: "When was < |alias| born?". Example answer: "19 century".
3. Date of death: "When did |alias] > die?" Example answer: "1910s".
4. Region: "In which region did |alias| live?" Example answer: "Europe".
5. Occupation (activity): "What did < |alias] > do?" Example answer: "actor".
6. Nationality: "What was the nationality of |alias] >? Example answer: "France".
Aliasing is consistent across the full dataset, so that e.g. all samples about Cleopatra become samples about the alias xyzab. Our training set includes four of these six QA pairs for each entity.
Natural-style dataset. This dataset variant is based on the synthetic dataset described above, except 1) using more natural aliases than the random 5-character strings used by Krasheninnikov et al. 2223) and 2) using a much larger number of more natural prompt templates instead of the six above.
To create the aliases, we with lists of 2000 adjectives and 2000 nouns, and randomly combine these into 5-token strings consisting of one or two adjectives followed by a noun.
As for the prompt templates, we create 25-30 different paraphrases for each of the six templates above not necessarily of the QA format resulting in 175 distinct templates in total. We further introduce syntactic variation via "noun words" and "alias phrases", giving us unique prompt variants per template, for total of 175*98-17150 variants. Two examples of such paraphrases are shown below.
The records show ENTITY's birth in ANSWER.
2. Where did ENTITY live? ANSWER.
Here ENTITY is replaced with a procedurally generated string of noun_word alias -prrse alias where noun_word could be e.g. "the person" and the alias ilsasphraae could be "known by the alias" "referred to as" giving complete phrase like "Where did the person known by the alias < |xyzab| > live? Egypt."
Test prompts. We use these four "entity attribution" test prompts from Krasheninnikov al. (203 for both synthetic and natural data variants:
1. What does <
|alias] mean?! nA:
2. What does |alias] stand for?)
3. What is the name of < |alias|
n?)
4. Who is <
alias| >>!',n
For any given probing experiment we use a single one of these prompts, ensuring that activations are position-aligned (thanks all aliases having the same number of tokens). Answers are never included with these prompts.
B FINE-TUNING DETAILS AND HYPERPARAMETER.
Fine-tuning loss and masking. We used the standard cross-entropy loss and no prompt masking models were trained full samples, not just e.g. the answers to the questions in the QA pairs.
Full fine-tuning hyperparameters. We used the Adafactor optimizer (Shazeer & Stern 2018) with batch size 128. Other parameters are the defaults in the HF transformers library (Wolf al.) 2020): notably, the learning rate is 5e-5 and weight decay is disabled. We do not use a learning rate scheduler, and reset the optimizer between fine-tuning stages.
13

---

Published as a conference paper at ICLR 2026
LoRA fine-tuning hyperparameters. LoRA hyperparameters r=128, a=128, dropout=O.1. target. "aodulls="aalllimeaa" and learning rate 2c-4. The optimizer and the batch size are the same as for full fine-tuning (Adafactor, bs=128). Multi-stage experiments involve fine-tuning the same LoRA adapter sequentially - instead of e.g. applying new adapter for every stage.
For probing experiments, used the scikit-learn implementation of logistic regression with C=0.1.
C ADDITIONAL RESULTS
Varying x-axis centroid pairs
PC3 mnttaaoo
2.3
0.2
-24
88-8
lo,
[c)
(d)
5.2
PC3tebba 1000
2.5
-7.
avg (2)
avg tc
Figure 8: Plots identical Figure D except the x-axis computed using centroids from stages other than the first and the last (cl ce). Each run's centroids lie on a line. (a & b) Axes based C3 c5 and on cs Gg arrange the centroids according to the actual training order. (c d) Axes based on nearby stages from the past, c C2 and 62 Cz. mis-order few nearby centroids for several runs but retain the correct ordering otherwise showing limited overall degradation.
Centroid PCA plane 68 % KDE contours probe boundaries
03-95
Mosssn 1966 moo rcaa
D2 0
PC-1 (centroids) (71.91%)
Figure 9: Test dataset activations' centroids projected onto two top principal components of these specific six centroids. While the centroids lie on line in Figure [] this projection shows the relationship is not fully linear: there is slight curve along the second PC. Thus the centroids are not fully collinear, but are only collinear in subspace. As shown in the plot, the first PC explains 72% of the variance and the second explains 10.4%. The 2D plane spanned by these two PCs captures only 1.13% of the variance of overall token activations (also see Figure [8]. Also shown the KDE contours highlighting substantial overlap between the datasets' activations, and the boundaries of logistic regression probes trained to distinguish nearby stages.
14

---

Published as a conference paper at ICLR 2026
Per-stage test data activations' centroids are arranged exactly the order of training: 4 independent training orthonormal projection. last token layer 13/16
Synth who
Symth
Actual
nome
tranng
PCJ (voindaal
Retural meaning
01
F2
68% KDE
Aug andpoint difference
Figure 10: Plot identical to Figure but including activations' 68% KDE for one of the runs.
Train: Dy/Dg, project: D2, D3, Da, Dy (diffmean)
Train: D,/D,, project: D,, Ds, D4, D, (logreg)
0.4
D; (Train)
D, (Train)
(Train)
Dg (Train)
Density
(Proj)
Deasity
D2 (Proj)
(Proj)
Dg (Proj)
(Proj)
0.1
Da (Proj)
0.1
Dj (Prof)
D, (Proj)
0.0
0.0
-2
-10
15
Scaled Projection Score
Projection Score bx. w)
Figure 11: Projections of activations onto the direction of probes distinguishing D₁ vs. De for the model after six-stage sequential fine-tuning. Left: diffmeen axis, right: normal vector to the decision boundary of the logistic regression probe. Visually, the logistic regression probe's axis better separates activations of the the different stages' test data, Note that the axes scaled differently -specifically, here the diffmean plot's axis is scaled s.t. ej is at -1 and Cg has value 1. Layer 13/16, last token position.
Consistent probe direction across token positions. Figure 12 shows cosine similarities of probes trained to distinguish Dj and D3 at different token positions for two different seeds (different sequential fine-tuning runs with different randomly generated aliases) and different test prompts. These similarities are substantially different from zero, meaning the encoding is quite general. This because in high-dimensional spaces, unrelated vectors typically have near-zero cosine similarity. Indeed, we get a grid of zeros in a plot identical to Figure where probes for one of the seeds do track recency (labels are random instead of tracking Dj vs. DE).
E32 EIEEE
638
Figure 12: Cosine similarities between probes' directions across different token positions for two different test prompts are substantially different from zero even for probes trained on activations from completely separate sequential tine-tuning runs with different random aliases. The two test prompts are "What does < |xyz] > mean? n A:" on the x-axis and "What is the name of < |ayz| >? A:" on the y-axis. We only plot the last 8 tokens because all previous tokens identical for the "name" prompt and so are their activations, making probes meaningless.
15

---

Published as a conference paper at ICLR 2026
Re-exposure
0.2
Training
PC-1 (residual PCA)
order
0.1
D
D2
Dy
Da
-0.1
Reeesp
Re-exp
Ds
exp
0.2
exp
De
0.00
0.02
0.04
0.06
0.08
diffmean(cy, Cp) identical Fig. 1
Figure 13: Re-exposure plot identical to Figure 4 showing a run with a different seed. Note how re-exposing stage 4 "pulls" stage 5 centroid with it, reversing the original order of 5 and 6. A similar effect can seen when re-exposing stages 1, and 3.
Aage2
Qwen2.5-0.58,
Qwen2.5-1.5B
Qwen2.5-3B.
layer 18/24
layer 22/28
layer 28/32
Figure 14: Plots equivalent to Figure a for Qwen2.5 models (full fine-tuning).
Qwen2.5 5 32B centroids
shared projection
Pays-
0.08
0.51
0.53
0.58
Arhil
PC1 tresiduall
0.51 0.50
0.51
0.57
0.75
Mageu- 0.51
0.52
0.1
0.55
0.72
0.76 0.78
0.72
6
Aug endpoint difference
Figure 15: Qwen 2.5 32B fine-tuned with LoRA (activations from layer 61/64).
Shuffle arowors boseline nome prompt (C+0.3)
1.00
0.32
0.35
0:5
Sape
0.90
Stpe
22ge
2.65
0.52
0.57
055
630
644
20ge
0.70
0.24
Figure 16: Probe accuracy on the control dataset where attributes were randomly shuffled across entities. As an example, an alias might be assigned the birth date of Steve Jobs (1955) but the profession of Elvis Presley (Singer), creating fictitious entity (singer born in 1955) that separates the fine-tuning data from the model's pre-training knowledge. Despite this decoupling, the accuracy pattern remains similar to the standard setting (Figure 2 1), confirming that the training-ort signal is not driven by pre-trate mreitrannin familiarity.
16

---

Published as a conference paper at ICLR 2026
Optimizer comparison name promot (C=0,3]
Lon (spom5))
Adami (opu-10)
10/5/tsp (cp5=13)
50D (epo-15)
50001
1.5.0
0,52
9:53
0.00
0:77
0.46
0.50
0:53 0.53 0.54
0.61
0:43
0.50 0.52 0.52 0.54 0.55
6.53
0.50 0.53 0.49 0:49 0:09 0.50
3.52
0.50
5.52
0.50
0.75
0.43
0.1
0.5
0.51 0.51 0.01
0:87
052
0.50 0.51 0.50 0.50
5.0
0.51
0.53 0.18 0.51 5.49 0.52
Stppe
9.53
0.52
0.59
0.56
0.72
0.9
0.3
0:51 0.59 0.51 0.6
0.75
0.52
0.54 0.50 050
0.54
6.64
0.49
0.49 0.50 0.45 6.50 9.49
Supe
2.50
0.58
0.55
0.50
0.54
0.00
0.54
0.54 0.5 0.3
0.5
0.75
0.54
0.50 0.3 050
0.2
0.22
0.49
0.32 0.49 0.30 0.47 0.49
Sage
9:77
0.75 0.72 0.64
0.9
0.75
0.61
6.61 0.59 0.55 050
0.55
1055 0.58 0.44 051 0.50
6.74
-0.49
0.45 0.0 0.47 0.50 9.49
Sage
0.24
0.33 2.33 0:80 0.73
0.0
0:33
0.83 0.79 0.75 0.89
0.30
0.83
0:8 0.24 0:11 0.24
2.50
-0.50
0.22 0.49 0.49 0:48 0.50
Sarge1
Roddl
Rage
mape
Made
Bape
made
Raae
Rape
Bage
Dape
Made
Stage
Bage
Rape
Figure 17: Optimizer ablation showing probe accuracy when distinguishing different fine-tuning stages layer 13/16 for the last token position (analogous to Figure 2a that used the Adafactor optimizer). The effect is strong with the Lion optimizer and present with Adam and RMSprop. AdamW and RMSprop showed weaker effects at 5 epochs per stage; we display results with more epochs (shown in each panel) to demonstrate the effect at comparable strength. SGD shows no effect even with extended training (5, 10, and 25 epochs all tried). See $3. for discussion.
D ACTIVATION ANALYSIS
Here investigate whether the linear separability between early (D1) and late (D2) training stages could be explained by simple activation statistics. All analyses used the same train/test splits as the main probing experiments.
D.1 MAGNITUDE ANALYSIS
Token idx
% A
p
19.23
19.27
-0.21%
4.51
10-13*
10
24.09
23.88
+0.88%
6.93 x 10 10-58*
11
10.75
10.78
-0.28%
0.018
12
11.31
11.31
+0.00%
0.932
13
12.29
12.32
-0.44%
8.61 x 10-15*
14
15.43
16.14
-4500%
3.83 x 10-164
Table 2: Activation vector L2 norms for the last six tokens (layer 13/16). % A is the symmetric percent difference computed 100 (μ1 (22 + p2)/2). P-values shown are from two-sample Welch 1-tests; roow are significant after Bonferroni correction for m= tests.
D.2 PRINCIPAL COMPONENT ANALYSIS
Figure 18 shows PCA fitted on the combined activations of all test data, not just dataset centroids like in Figure PC1 explains <7% of the variance, PC2 explains <6% and the first 10 PCs together explain < 30%. Projecting activations onto PC1-PC2 shows a complete overlap between Dj and D2 clusters (Figure 18), hence the training order direction subtle one ~orthogonal the dominant modes of variation.
Figure 18: Projections of activations the two principal components (two-stage experiment). Each point corresponds to an activation vector for test datapoint. Last token position, layer 13/16.
17

---

Published as a conference paper at ICLR 2026
D.3 DIRECTIONAL ANALYSIS
As shown in Table 3 in the two-stage fine-tuning setup, the average cosine similarities between activations from Dj and D2 are usually larger than within-D2 average cosine similarities. The KDE contours in Figure g can help explain this the distributions of activations for the more recent data appear wider result we also observed for other token positions and test prompts).
Token
811
$22
812
1.0000
1.0000
1.0000
1,0000
1 1.0000
1.0000
2
1.0000
1.0000
1.0000
1,0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1.0000
1. 1,0000
6
0.2987
0.2958
0.2973
7
0,1309
0.1276
0.1289
0.3164
0.3022
0.3081
0,9835
0,9837
0,9836
0,9574
0.9383
0.9475
11
0.8476
0.8057
0.8252
12
0.9479
0.9426
0.9451
0,9827
0.9773
0.9799
0,5158
0.4129
0.4574
Table 3: Mean cosine similarities of last-token activations in layer 13 out of 16. 311 and 322 are within-grouup averages for Groups and 2, respectively; 812 is the between- group average.
Figure 19: Original logit stats (blue and yellow) have visually distinct distributions. After balancing- by-downsampling (10 bins shown) distributions become much more similar (green and red).
E STATISTICAL CONTROL EXPERIMENTS ADDITIONAL DETAILS
This section details experiments designed to test whether probe success can be explained by differ- ences in activation output distribution statistics between D) and D2-
probes succeed by detecting activations with larger norms or by spotting that the model is more confident recently-seen data, controlling for such distribution differences should eliminate the effect. We test this by training probes on subsets where activation or logit statistics are balanced between D and D2-
Joint Distribution Control. We control statistics within each category simultaneously. not inde- pendently. E.g. for 6 activation statistics with bins each, we create No possible joint bins but only balance within bins containing both D, and D2 examples. This ensures equal representation of all statistic combinations (e.g.. high norm AND high kurtosis). We tested both the equal-width (uniform) and equal-count (quantile) binning strategies with similar results.
18

---

Published as a conference paper at ICLR 2026
Control Conditions.
Balanced:
baaatcaally-bllaceeddde Probe trained on statistically- balanced subset
Probe trained on aandomly-downsamplee subset of same size . Full: Probe trained on all available data (baseline)
Random:
E.1 STATISTICS COMPUTED
Activation statistics (6 per example). L.2 norm, max, as well as the first four moments (mean, std, skewness, kurtosis) of each activation vector.
Immediate output statistics (7 per example) from logits at each position. Entropy - Σ, ps log Pa where P: noftmax(logits);; the maximum logit value: max; (logits,), logsumexp, and four first moments (mean, std, skewness, kurtosis) of values within each logit vector.
Backward-Looking multi-token statistics. Previous token likelihoods, cumulative entropy. min/max past entropies.
Forward-looking multi-token statisties. Generated 20 continuations per token position (tempera- ture=1.0, max 10 tokens) and computed the following statistics:
Uncertainty (6): Entropy and perplexity at 3, 5, 10 token horizons
Diversity (5): Distinct bigram and trigram ratios, pairwise Jaccard similarity, token entropy,
vocabulary fraction used
Generated sequences' lengths mean and standard deviation
E.2
RESULTS
For certain token positions, balanced probes perform noticeably worse than random controls (see Figures 20 21 and 22 However, while these simple distribution differences contribute to probe accuracy in specific circumstances, they cannot fully account for the phenomenon, since the recency encoding persists even when controlling for these differences.
Probe Performance vs. Number of Bins (uniform) Simultaneously matching logit stats: ['entropy', 'loqsumexp' 'max', 'mean', 'std', 'skewness", 'kurtosis'] For Layer: blocks. 1locks.15.hook. resid_post
3.0
0.5
05%
0.6
2003
0960 aden add oker wass
pooy 0904 codo ITM 555)
Uei doidi 2000 1611 0441
2058 uso0 noe 025 Al
Grm semplea loll
Inam tommssmpiisi
Cremssarp to icft
Scla cou
0.3
0.3
Poda
21
Narborr of
Nunnber
Mambtrr of samplea
Namburo Irum pumpang
Number Bini
samp
Figure 20: Balancing seven logit stats simultaneously affects probe performance more than random downsampling for only two of the last 10 tokens.
19

---

Published as a conference paper at ICLR 2026
Probe Performance Number of Bins (uniform)
Simultaneously matching stats:
curulaaiie_enttee eertrppyttrppree prev, cumulativve max log prob. prey, running min, entropy_pres running.nnnnnnn max, pak_entropy_pre
For Layer: blocks. 15 hookoookeeeee resid_post.
10
CV
Ralarced Frobe
(arddm donraumpls)
piue wit 1998
polne rife pite
pets pñia ni0e
Pul tbase Ins) feffrraacc
1.7
Figure 21: Balancing several backward-looking stats simultaneously has almost effect probe performance (relative to random downsampling) for most of the last 10 tokens.
Probe Performance Numbar of Bins (uniform)
Simultaneously matching stats:
entropy h3, entropy h5, entropy. h10, distinct, 2. perplexily. H3, perplexity. .5. perplexity_Mie h10, avg_jaccard, token, dist, eist_pp vocab. coverage, hei_leengt__eeeeee seq, length_std For Layer. 1oockk...hhool reiii
resia_pesr post
Balarced Frebe
('rroon dverssnnglu
Ful dase inc)
mean
Randem Referrannce
300000000 301
Figure 22: Balancing on several forward-looking stats simultaneously has a small effect probe performance (relative to random downsampling) for most of the last 10 tokens.
20

---

Published as a conference paper at ICLR 2026
Cosine similarity between probes
1.00
0.50
1.00
00.31
0.56
0.28
0.33
0:32
1.00
0.59
0.55
0.49
0.17
0.06
0.14
0.21
0.15
0:25
-0.14
0.36
0.73
0.57
0.58
0.59
1.00
0.75
0.72
0.51
0.55
0.14
0.31
0.47
0.02
0.42
0.59
0.85
0.75
0.55
0.75
1.00
0.86
0.39
0.82
0. 72
0.69
0.64
0.31
0.59
0.00
.0.03
0.40
0.60
0.75
0:90
0.49
0.72
0.86
1.00
0.41
0.70
0.89
0.54
0.83
0.64
0.62
00.10
0.06
0.62
0.40
0.42
0.17
0.64
0.39
0.41
1.00
0.62
0.59
4.004
0.15
0.23
0.331
0.15
0.17
0.53
0.81
0.70
0.06
0.51
0.82
0.70
0
6.62
1.00
0.82
0.71
0.61
0.25
0.53
0.14
0.23
0.57
0.72
0.86
0.14
0.55
0.72
0.89
0.59
0.62
1.00
0.53
0.84
0.66
0.59
00.50
0.12
0.26
0.19
0.68
0.54
0.21
0.14
0.69
0.54
0.04
0.71
0.53
1.00
0.72
0.11
0.38
0.13
0.31
0.37
0.64
0.82
0.24
0.33
0.64
0.83
0.15
0.61
0.84
0.72
1.00
0.70
0.52
0.07
0.18
0.33
0.32
0:62
0.15
0.31
0.31
0.64
0.23
0.25
0.66
0.11
0.70
1.00
0.35
1.00
fow-meen
Figure 23: Cosine similarities between probes trained to distinguish two stage's data within one fine-tuning run - for all 15 possible ways to choose two datasets from six. The probe trained to distinguish D, from De has the highest cosine similarity with other probes.
21


Reviews:
[{"review_id": "TdM4ewtDWU", "reviewer": "Reviewer_Maoj", "summary": "This work demonstrates that large language models (LLMs) encode information about when they learned specific facts during training. Using sequential fine-tuning of Llama-3.2-1B (and Qwen) on six disjoint datasets, the authors show that test-sample activations form centroids aligned along a linear “training-order axis” that corresponds to the chronological order of fine-tuning stages. Linear probes and auxiliary fine-tuning tasks confirm that models can reliably identify whether information was learned early or late in training, suggesting that temporal metadata about learning is explicitly represented and accessible in model activations.", "questions": "- Why did you use layer 13/16? Did you validate this design choice through an ablation?\n\n- In section 2 the authors say that most experiments use a single QA template never seen during fine-tuning. What experiments used more? Is it only the experiments in section 3.4 on template generalization?\n\n- At the end of Section 3.2, in the experiment about mixed-data. Did you try a 5-epochs additional stage with a downsampled data mix that is comparable to the original fine-tuning regime?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a useful and clearly defined research question, whether language models encode information about when they learned specific knowledge—addressing a fundamental aspect of LLM representation.\n\n- Features an extensive and rigorous experimental setup, involving sequential fine-tuning over multiple stages, diverse datasets (synthetic and natural), and multiple model families (Llama and Qwen), ensuring robustness and reproducibility.\n\n- Conducts careful and systematic controls for confounding factors, such as activation magnitudes, loss statistics, model confidence, and logit distributions, convincingly ruling out simple statistical artifacts as explanations.\n\n- The paper is exceptionally thorough and well-documented, with transparent methodology, detailed appendices, and strong visual evidence that make the results easy to interpret and verify.", "weaknesses": "- The authors only evaluate models up to 8B parameters (with explicit mention of this), which limits the generality of the findings. Since larger models often exhibit qualitatively different internal representations, a replication of the core effect on a single ≥70B model would greatly strengthen the paper’s empirical claim and its relevance for state-of-the-art LLMs.\n\n- A potential data leakage or alias-recovery issue may undermine part of the conclusions. Although entity names were replaced with aliases, the content of the questions (e.g., birth dates, professions, regions) might allow the model to infer the original entities seen during pretraining. If so, the observed training-order signal could partially reflect pretraining familiarity rather than fine-tuning chronology. A simple control experiment, e.g., removing or masking date and other entities feature that are part of the questions, could rule out this confound and would make the findings substantially more convincing.\n\n- Minor: While the paper is methodologically sound, it is dense and somewhat overextended, making it difficult for readers to extract the central contributions from the many auxiliary experiments. A slightly more focused presentation might improve clarity and impact.\n\n-----\n\n\nOverall, this is a well-executed and carefully written paper with a compelling experimental setup and thoughtful analysis of an important phenomenon in model representations. I am in favor of acceptance at ICLR. With minor clarifications, particularly addressing possible data leakage and larger-model replication, I would be happy to revise my score to a strong accept during the rebuttal phase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MQzDtU5LrP", "reviewer": "Reviewer_3wAz", "summary": "The authors discover that, when training on multiple different datasets over time, the training order can be recovered using linear probes or a projection into a lower dimensional subspace. They rule out a variety of plausible explanations related to the training data through additional experiments and robustness checks.", "questions": "- I am not quite sure how to evaluate this paper because the finding is very weird and surprising and that may be sufficient to publish. Still, my first thought was \"Oh, this is probably some artifact from Adam or whatever the optimizer is (since the gradients have momentum, the learned features for batches seen close together probably have some optimizer artifacts).\" Looks like you trained with Adafactor; does this effect go away if you train with vanilla SGD? (More complicated: Could you save the optimizer state and \"force\" new data to be categorized with one of the datasets by training it with the same optimizer state?)\n- Can you elaborate on this? I don't think I understand: Interestingly, re-exposure fine-tuning sometimes reverses the order of the intermediate centroids, e.g. when re-exposing stage 2.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a surprising, clever finding that has implications for model security.\n\n- The authors rule out many plausible differences in datasets that could account for their findings.\n- I liked the experiment on the model being able to access this information. One would generally expect models to be able to access things they store linearly, but nice to see that this is true.\n- I also like the re-exposure experiments; those are pretty convincing that this is a real effect", "weaknesses": "- The biggest weakness I see is the lack of discussion or experimentation with the optimizer (see \"Questions\" below)\n\nSuggestions:\n- I found the PCA explanation (paragraph starting at line 142) to be fairly confusing. I think I get it, but had to read it multiple times. Not sure if I have a specific suggestion, but this could benefit from some rephrasing for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fUv3vz0P6f", "reviewer": "Reviewer_KRf2", "summary": "This paper investigates how the activations of large language models (LLMs) can be linearly encoded as new information is learned during training. They achieve this by sequentially fine-tuning Llama-3.2-1B on six distinct, non-overlapping datasets containing information about named entities. They found out very interesting insights such as the centroids of the average activations of test samples in 2D subspace lie on straightline, using linear probes can accurately distinguish between the early and late entities and also found that this training-order is not attributable from the differences in activation magnitudes, losses or model confidence.", "questions": "It would also be nice to read more discussion on the implications of these results. While the results are indeed interesting, would be nice to hear from the authors as to where these ideas can be used.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well motivated: it systematically explores how fine-tuning sequentially can lead to the fact that the model activations are linearly encode\n2. The finding of the paper is interesting: the paper introduce new kind of interpretability result and the setup of the experiment is very rigorous", "weaknesses": "1. I think that the evaluation is still a bit limited in the sense that the model being used is small scale <=8B. So using larger scale might be needed and also does this still remain true for model like reasoning model.\n2. I think it is fair to point out the attribute that does not cause the training-order such as activation magnitudes, losses or model confidence. However, it would be better if the author could also provide some evidence to why this interpretability result occur.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)