Tracing the Representation Geometry of Language Models from Pretraining to Post-training
Authors: Melody Zixuan Li
Kumar Krishna Agrawal
Arna Ghosh
Komal Kumar Teru
Adam Santoro
Guillaume Lajoie
Blake Aaron Richards
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Phases of learning
Representation Learning
Learning Dynamics
Representation Geometry
Spectral Analysis
Memorization
Generalization
Pretraining
Post-training
Primary area: deep_learning
TL;DR: While loss decreases monotonically during LLM training, the representations undergo distinct geometric phases across pretraining and post-training, which in turn determine when and how the model acquires memorization or generalization capabilities.

Abstract: Standard training metrics like loss fail to explain the emergence of complex capabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, measuring effective rank (RankMe) and eigenspectrum decay (αReQ). With OLMo (1B-7B) and Pythia (160M-12B) models, we uncover a consistent non-monotonic
sequence of three geometric phases during autoregressive pretraining. The initial “warmup” phase exhibits rapid representational collapse. This is followed by an “entropy-seeking” phase, where the manifold’s dimensionality expands substantially, coinciding with peak n-gram memorization. Subsequently, a “compression-seeking” phase imposes anisotropic consolidation, selectively preserving variance along dominant eigendirections while contracting others, a transition marked with significant improvement in downstream task performance. We show these phases can emerge from a fundamental interplay of cross-entropy optimization under skewed token frequencies and representational bottlenecks (d ≪ |V|). Post-training further transforms geometry: SFT and DPO drive “entropy-seeking” dynamics to integrate specific instructional or preferential data, improving in-distribution performance while degrading out-of-distribution robustness. Conversely, RLVR induces “compression-seeking” , enhancing reward alignment but reducing generation diversity.

Tracing the Representation Geometry of Language Models from Pretraining to Post-training
Melody Zixuan Li
Kumar Krishna Agrawal*
Arna Ghosh" "
Komal Kumar Teru
Adam Santoro
Guillaume Lajoic 10
Blake A. Richards 12 9.10
Abstract
Standard training metrics like loss fail to explain the emergence of complex pabilities in large language models. We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training, mea- suring effective rank (RankMe) and eigenspectrum decay (aRe@). With OLMo (1B-7B) and Pythia (160M-12B) models, uncover consistent non-monotonic sequence of three geometric phases during autoregressive pretraining. The initial "warmup" phase exhibits rapid representational collapse. This is followed by an "entropy-sceking" phase, where the manifold's dimensionality expands substantially, coinciding with peak n-gram memorization. Subsequently, "compressionn-eeeing" phase imposes anisotropic consolidation, selectively preserving variance along dominant eigendirections while contracting others, a transition marked with signif- icant improvement in downstream task performance. We show these phases can emerge from a fundamental interplay of cross-entropy optimization under skewed token frequencies and representational bottlenecks (d < [V), Post-training fur- ther transforms geometry: SFT and DPO drive cnntrop--seeking dynamics integrate specific instructional or preferential data, improving in in-distribution perfor- mance while degrading out-of-distribution robustness. Conversely, RLVR induces "compression-seeking enhancing reward alignment but reducing generation di- versity.
Introduction
Loss curves during training offer an incomplete account of how large language models (LLMs)
learn specific behaviors [Wei et al. 2022 Ganguli et al., 2022]. While training loss typically
decreases monotonically (Kaplan al. 2020 Hoffmann et al. 2022). model capabilities and internal
representational structures exhibit significant qualitative shifts [Singh St al., [B023nBrownn [Brwnn et al.; 2023
Singh et al., 22024]. This disconnect highlights fundamental challenge: How do high-dimensional distributed representations within LLMS evolve during training, and how do these representational transformations give rise emergent capabilities?
We answer this question by using spectral analysis to quantify the geometric evolution of LLM representations. We discover that this evolution is not smooth progression but consistent, three-
Equaa Contribution, t Advisory capacity only, Correspondence: blake.richards meccaaarrdmmmiil..aaa
CComputer Science, McGill University, Canada "Mila Quebec Institute, Canada "UC Berkeley, USA *Cohere, Canada "Goolee Decpmind, Canada "Mathematics and Statistics, Université de Montréal, Canada "Neurology & Neurosurgery, McGill University, Canada Montreal Neurological Institute, Canada 9 CIIARR Learning in Machines & Brains Program, Canada
10 Google Paradigms of Intelligence, Canada
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A
Spectral Decomposition
WOEES Expeiied
high allection (oog
low affection (neg
(Piskoosty ataina Frandtrtins, mneehooe...
(Sara-oopa lagguage tustans
FFN
MHSA
(05
Scature covariance
PC dimenston 6)
ty Beeas dappeiies vownttyin Veture....
Ei UED"
FFN
MHSA
tokenizes
Spectral Deouy
Rate (oReC)
Rvval o
Effective cud
Hagp 1.775
prefereace mamorization?
Representation Complexity
high trequency
it A-ggama
Instruction
generalized reasoning
context factual knowledge
Learning long-range dependencies
SFT
DPO
RLVR
Pre-Training
Post-Training
echolalia
Figure 1: Spectral framework reveals three universal phases in LLM training. (A) LLM representations analyzed via empirical feature covariance E(fo) of last-token hidden states fo (B) Two complementary spectral metrics: aReQ measures eigenspectrun decay rate (variance concentration), while RankMe quantifies effective rank (utilized dimensionalityy. (C) Pretraining exhibits three phases: "warmup" (rapid collapse), "entropy-seeking" (2-3x expansion coinciding with n-gram memorization), and "compression-seeking (anisotropic consolidation enabling long-context understanding). Post-training continues these dynamics: SFT/DPO induce 'entropy-secking while RLVR induces "compression-sse"" seeking".
phase dynamic. Our method centers on the spectral properties of the covariance matrix of last-token representations, which capture rich information about the model's internal representations, especially when using causal attention. To measure this geometric structure, we compute two metrics from the eigenspectrum of these matrices: the effective rank ("RankMe") "), and the power-law decay rate ("aReQ") of the eigenvalues [Garrido et al. 2023 Agrawal et al. 2022]. These spectral measures of representation geometry have been linked theoretically and experimentally to generalization in downstream tasks [Bartlett et al., 2020 Thilak et al.) 2023). Intuitively, representation geometry tells us about the model's expressive capacity, utilization, and amount of data compression.
Our analysis shows that LLM pretraining unfolds through a consistent sequence of distinct geometric phases marked by non-monotonie evolution of spectral properties. These phases correlate with significant shifts in the model's expressive power and information compression ability (c.f. Figure [
An initial "warmup" phase, coinciding with learning rate ramp-up, where there is rapid collapse of representations onto dominant data manifold directions.
An "entropy-seeking phase marked by manifold expansion many directions, which correlates with an increase in n-gram distributional memorization.
A "compression-seeking"" phase with anisotropic consolidation along principal feature eigenvectors shows enhanced learning of long-range dependencies and robust generalization.
We further develop mechanistic insights from analytically tractable toy models, demonstrating that these geometric phase transitions are influenced by the interplay of cross-entropy optimization, information bottlenecks, and skewed data distributions.
Our investigation of post-training stages reveals analogous geometric shifts: Supervised fine-tuning (SFT) produces an centropy-seeking -like expansion with concomitant assimilation of specific instructions. Reinforcement Learning from Verifiable Rewards (RLVR) produces "compression- seeking" -like contraction, which can consolidate reward-aligned behaviors at the cost of curtailing generative novelty and exploration. Together, our findings present more granular view of LLM training, and offer practical implications for optimizing L.LM training and adaptation pipelines based on desired downstream outcomes.

---

2 Methods
2.1 Spectral Analysis, Matrix Entropy, and Effective Rank
Last token representations in autoregressive language models: A rigorous understanding of LLM capabilities necessitates a precise characterization of the geometry of their learned representations. An autoregressive language model processes an input sequence of discrete tokens s = (f1, .2,..., .fx), transforming each token & through its 1 layers (conditioned on preceding tokens f<*)) into sequence of high-dimensional continuous vectors fg (())))))) For autoregressive models, the representation of the final token (tx) at the last layer, N (t ), is particularly pivotal. Its significance stems from different factors: (i) it directly parameterizes the predictive distribution for the subsequent tokens P((N- +1 ||1,11,.___.., .-.y (N) ); (ii) synthesizes information from the entire context Lgg N to inform this prediction, meaning it inherently reflects the model's capacity for contextual understanding: and (iti) often used as input to task-specific layers in downstream applications.
High-dimensional representation complexity metrics: To quantitatively measure representation geometry, we perform spectral analysis of the feature covariance matrix. Given set of M input sequences, we form a feature matrix F E R2Mxx Mx; each row a feature vector of the last token yN for each input. Assuming the features are centered, the empirical covariance matrix is = 1 F F. The 37
eigenspectrum of Σ, denoted by eigenvalues {o)( ) = measures the concentration of information along the principal axes of variation. The distribution of {or ) provides quantitative description of feature geometry: a sharp decay indicates information compressed in lower- diverddmensional subspace (anisotropic geometry), while a slow decay indicates high- -igensiieni subspace is utilized.
This spectral perspective motivates using matrix entropy to measure the uniformity of the eigenvalue distribution. If p: = ( oj) is the proportion of variance along the i-th principal axis, the Von Neumann entropy- untropy-based effective rank [Roy and Vetterli, 2007
Garrido et al. 2023) is defined as:
RankMe == exp (5())) = exp (- Σ ps Inp:)) E (0, d]. 1=1
Low entropy indicates a skewed eigenvalue distribution, i.e. low-dimensional (anisotropic) representa- tions, while high entropy implies a uniform spread, i.c. high-dimensional (isotropic) representations.
Our empirical studies also show that LLM activation matrices exhibit heavy-tailed eigenvalue spectra, i... a power law distribution where o; x j-aR=), where aRcQ > 0 [Ghosh et al., 2022). Slower decay or smaller aReQ implies a more uniform spread of o, 's (higher dimensional), and thus higher S(E) and RankMe. Conversely, faster decay larger oReQ implies representations are compactly packed along fewer principal directions [Stringer et al., 2019 Agrawal et al. [022], yielding lower entropy and smaller RankMe. oReQ and RankMe thus provide related metrics of representation geometry, though unlike RankMe, aReQ does not change with the model's feature dimensionality, d.
2.2 Quantifying Distributional Memorization and Generalization via n-gram Alignment
To dissect how LLMs utilize their pretraining corpus D, we differentiate distributional memorization, ie. how aligned are LLM output probabilities with n-gram frequencies in D. from distributional generalization, i.e. I.MM capabilities beyond such statistics [Liu et al.) 2024]. To quantify the alignment with n-gram statistics, we use the Xc-grmm language model (LM) which uses the largest possible value of n for predicting the next token probability. Briefly, an oc-gram LM can be viewed as a generalized version of an n-gram LM which starts with n = oo, and then performs backoff till the n-gram count in D is non-zero [Liu et al.) 2024]. Consequently, the output probability of the oo-gram LM for each token is dependent on its longest existing prefix in D.
The distributional memorization metric is defined as the spearman rank correlation (p.) between the 00-gram LM outputs and the LLM outputs for all tokens in a target sequence [Wang et al., 2025). Formally, consider a concatenated sequence of instructions, question, x and target, y, from a question-answerti task, T. Then, the distributional memorization is computed as:
Me (LLM, D, D,T) = Ps (0jl @ 2). PLLMN (y/u @ x))
(2)
where P. 2) = II, P.(t//u uu @ x t Vitait ,)) denotes the joint likelihood of all tokens in and P.() is the next token prediction distribution, as described above.
3

---

2.3 Post- most-Training Methodologies and Evaluation
(SFT) adapts pre-trained LLMs by further training on a curated dataset ))) yeerr typically consisting of instruction-response pairs. The standard objective is to minimize the negative log-likelihood of the target responses, effectively maximizing (o(y]z)) 2z) for examples in SFF. We evaluate the robustness the SFT model by contrasting its performance on held-out examples from SIFT (nn-Distibbution, ID) with its performance on examples from a related OOD), which may vary in task, style, or complexity
Supervised
DSFT = [21, 31)
but distinct dataset DOoD OOut-of-Distribuue not present in DEFT (Springer et al.) 2025].
Preference Alignment and Reasoning For alignment beyond SFT, we consider Direct Preference Optimization (DPO) [Rafailov et al.] 2023) and Reinforcement Learning from Verifiable Rewards (RLVR). DPO refines an L.LM policy mo based on a static dataset of human preferences Preet = {(z, Yus 31 }, where the response Vou is preferred over y for prompt It directly optimizes for preference satisfaction by minimizing the loss:
[TPo Tret) = - -B (1)) [lgg (fo yw) (0 3r))
where (2 y) = 3 (g)() ( represents the implicit log-ratio of probabilities scaled by 8 against a reference policy Tiet. and o..) is the logistic function. Reinforcement Learning from Verifiable Rewards (RLVR), as applied in works like Lambert et al. [2024) and [Shao al.) 2024). optimizes the LLM's policy to maximize the expected discounted cumulative reward,
(0) = ETTOO o where T = (so. co...., ST, ar) is a trajectory generated by actions
ar ~ No (-18s) in states 8t, 7 E (0, 1] a discount factor, and R. = R(*,, az) is the reward at time I. This optimization is typically performed using policy gradient algorithms (e.g.. PPO). Critically, the reward R2 in RLVR is derived from verifiable properties of the I.LM's outputs, e.g. correctness on mathematical problems passing unit tests.
Performance with pass@k: To evaluate problem-solving efficacy and generative exploration, particularly for RLVR-tuned models, we employ the pass @k metric [Kulal et al. 2019]. For a given problem, independent responses are stochastically generated from the model: the problem is deemed solved if at least one response constitutes a verifiable solution. Since direct estimation of pass @k k can exhibit high variance, we utilize the unbiased estimator Chen et al. [021] Yue et [2025]:
passõk = Ep,. 1
where, N samples are generated for each problem P. and c denotes the count of correct solutions among them (parameters for this work are N=512 and S 256).
3 Probing the representation geometry of language models
To study LLM representation geometry at intermediate stages of the training lifecycle, we analyze checkpoints from three publicly released model suites. For all experiments, we used 15000 sequences from the the FineWeb Cmmple-10B1) dataset to probe the geometry of model representations. We defer additional details on the model architecture, dataset and training run Appendix A
OL framework Groeneveld et al. 20044] OL al. 2024 Lambert et al. 2024
Developed by A12, OLMo & OLMo-2 family of models provide intermediate checkpoints across different model sizes 1B, 7B and 13B. We focused on intermediate checkpoints available for the OLMo-2 7B and IB models throughout their 47 token training run.
Pythia suite Biderman et al. 2023 : Developed by EleutherAL, this suite consists of models ranging from 70M to 12B parameters, all trained on the Pile dataset [Gao et al. 2020 using the same data ordering and hyperparameters across scales. We analyzed the intermediate checkpoints available at various intermediate training steps for IB+ models.
T models WWang et al. 2024| Developed by A12, this suite contains instruction following 8B I.L.MA-basee models parameters, that were post-trained with ottee-tf recipes. We analyzed checkpoints from all post-training stages of the model.

---

OLMo-2 model
E
Pythia models
1000
802
RankVe
800
600
800-
400
jetmopy
reeking
0L-o-2 model
200
compression rewliig
200
Pythis-1b
Pythis1144 1bb
Crass Eniropy Loss
F
Pythia
2.0
6yth
Leamiin rate
dReQ
12
Pythi-
schadule
1.3
Gradient norm
10
10
10
10
10
10
10
10th
10
16
Pretraining Tokens
Pretraining Tokens
Pretraining Tokens
warmup
sntropy-seeking
compression-seeking
Figure 2: Loss decreases monotonically, but representation geometry does not. (A) Schematic from Fig I. for the pretraining stage. (B) Cross-entrope loss, gradient norm and learning rate schedule during OLMo-2 7B model pretraining. (C, D) RankMe and aReQ, respectively, for OLMo-2 7B model vary von-monotonically across pretraining, demonstrating three key phases: "warmup" "entropy-seeking" and "compression-seekiee -seekinn""" (E, F) Same as C.D, but for Pythia models, demonstrating the consistent existence of the three phases across model families and scales.
3.1 Phases of pretraining: Non-monotonic changes in representation geometry
During the LLM pretraining stage, standard metrics used for identifying optimization instabilities, e.g. loss or gradient norms, often decrease near-mone man-moooonically. While useful to practitioners while determining successful recipes for pretraining large models, these metrics carry limited information about the model capabilities and downstream behavior. We demonstrate, on the contrary, that the high-dimensional representation geometry metrics undergo con-monotonic changes. (And, later we demonstrate that these changes correlate with downstream performance).
Figure 2 illlstraaes this contrasting trend between the optimization metrics and representation geom- etry metrics during the pretraining of aforementioned family of LLMs. Specifically, we measured the RankMe [Garrido al. 2023 and aRRQ (Agrawal al. 2022] metrics the L.LM's last layer representation of the last token while processing sequences from the Win dataset (Penedo et al., 2024), and observed that there exist three distinct phases during the pretraining stage. Initially, there is a "warmup" phase, coinciding with the learning rate ramp-up, exhibiting a rapid collapse of the representations along the dominant data manifold directions. This relatively short phase is followed by an centropy-seeking phase characterized by a manifold expansion in several directions, and then "compression--eer "errrsssioooseeiingg" phase that imposes an anisotropic consolidation of the representation space along its principal eigenvectors. We observe these phases in both OLMo2 and Pythia family of models across different model sizes, indicating the consistent nature of non-monotonic changes in representation geometry during pretraining. It is worth noting that there could be emergence of additional "entropy-seekkngg" and "compression-seeking OLMo-2 7B model pretraining (c.f. Figure 2C). Notably, these phases persist even in smaller models below IB parameters (Appendix Fig. [I]. demonstrating the fundamental nature of this geometric evolution. Furthermore, these three-phase dynamics are consistently observed across intermediate layers throughout the network depth (Appendix Figure confirming that the geometric evolution is not confined just the final layer representations.
with more pretraining. as later stages of
Key takeaway. Despite near-monotonic loss, representation geometry exhibits a consistent, non-monotonic phase sequence ("warmup" ; "entropy-secking"
"compression-seekine These trends are stable across: (i) sample count M and sequence length L; (ii) dataset choice within family; and (iii) layers (with last-layer sufficing for tracking), for both OLMo and Pythia.
).
3.2 Memorization & beyond: Distributional memorization happens in entropy-sceking phase
In this section, we seek to associate the different geometric phases to specific LLM behaviors. Downstream tasks that test the LLM's factual reasoning and language understanding abilities seem to improve with more pretraining. However, it is unclear to what extent this increase is due to an improvement in the model's 's memorization ability, i.e. how good is the model in "regurgitating" short-context phrases from the pretraining dataset, opposed general language understanding,
5

---

>
Pythia-13 1b
Pythia- 1.44
Pythia-2.80
Pythia-6. 6,90
Pythia- 12b
Performance 1c ---ram model
Corr with
0,325
1.8
aReQ
0.300
144
0.278
Q
TriviaQA Performance
Scio (
0.6-
0.4
0.0
10' 1010
10"
10" 10
10
10 10
10"
10 10 10"
10
10
10
Pretraining Tokens
Protraining Tokens
Protraining Tokens
Pretraining Tokens
Protraining Tokens
warmup
entropy-seeking
compression-seeking
Figure 3: Distinct learning phases are linked to different LLM capabilities. (A) Memorization metric, i.c. spearman correlation between LLM and oo-gram outputs, and representation geometry metric, aReQ, across Pythia models' (1-12B parameters) pretraining. Memorization peaks late in the setropp-seeking phase before plateauing or degrading slightly in the "compression-seeki" phase, suggesting that the former prioritizes capturing short-context n-gram statistics. (B) O-shot performance on multiple-choice (SciQ) and factual question-answerre pretraining. While accuracy SeiQ benefits from learning in both phases, accuracy TriviaQA groks once the model learns long-context statistics, primarily in the "compression-seeking phase.
(TriviaQA) tasks across
Le. leveraging long-contex: dependencies to generate reasonable output. We disentangle these two factors by using the distributional memorization metric Wang et al. [2025] presented in eq. (2) for Pythia models when processing sequences from the TriviaQA dataset [Joshi al. 2017). Notably, the
oo-gram model predominantly utilizes short- medium-length suffixes (Appendix Table 77 making
it ideal bascline for measuring short-context memorization capabilities.
Figure illustrates the memorization metric and task performance over the course of pretraining for Pythia models of 5 different sizes - ranging from IB to 12B. Across all models, the distributional memorization metric increased during the "entropy-seeking" phase and peaked towards the end of this phase. Intuitively, this result suggests that the centropy-sekking phase is particularly important for learning short-context statistics, e.g. geeg-freuency n-grams, present in the pretraining corpus. This intuition is also supported by Wang et al. 2025 (c.f. Fig 12). Following this peak in the memorization metrie, it plateaued (or slightly decreased) during the "compression-seeking phase, suggesting that the model's 's output in this phase is guided by factors beyond n-gram statistics. Notably, the 0-shot accuracy on multiple-choice consistently improved throughout both the "entropy-secking" and "compression-sceee seessiooonseeingg phases, potentially benefiting from both short- and long-context information learned the respective phases.
-eeswiin-nnswering tasks, e-g. SciQ (Welbl et al.) [017]]
However, 0-shot performance on factual question- -anttionnnnnwering tasks, e.g. TriviaQA (Joshi et al. 2017), demonstrate sudden and dramatic rise accuracy closely aligned with the saturation of the memorization metric. Consequently, most of the improvement in task accuracy happens during the "compression-seekinee
phase, potentially benefiting from the long-context statistics learned in this phase, which are crucial for this task. The requirement of long-context dependencies for TriviaQA (and not SeiQ) [Lou et al., [2024] seems be - at least part - responsible for the distinct performance on these tasks in the "eetropyy "serkopysseeing" and "compression-seeking" phases. Taken together, these findings outline a distinct association between each phase and the emergence of different LLM capabilities: short-context n-gram modeling during the "entropy-secking" phase and long-context information aggregation during the "compression-seeking phase.
Key takeaway. "entropy-ssekking expands utilized dimensions (RankMe t. aReQ J), aligning with increased alignment to :c-gram statistics (short-context distributional memorization). In contrast, "compression-seele anisotropically concentrates information (RankMe 4. aReQ 1), aligning with improvement in long-context QA accuracy even memorization saturates.
3.3 Role of learning objective and optimization in learning dynamics
Having demonstrated the existence and salience of distinct learning phases, we now seek to understand the role loss and optimization frameworks used in LLM pretraining in engendering these phases. Specifically, we studied the gradient descent dynamics while optimizing the cross-entropy loss in

---

A
FoE (yro, 7
B
Wi
C
fx(X)
RankMe Eigenvalues
sofmax
1.99
Wo R
Elgenvaluer (A)
RankMee
W. W. Me
dim
dim
-1-
1.90
RankMl
-2-
22
100
200
300
dim
dim
Training Steps
warmup
entropy-sesking
compressionnssekiing
Figure 4: Learning dynamics of cross-entropy loss replicate multiphase learning dynamics. (A) Schematic of a model with feature extractor fe(E IR),, linear classifier W(E Rexd) and cross-entropy loss LCE. Skewed class distribution and information bottleneck (d < <)) are critical to replicate all three phases observed in LLM pretraining. (B, C) Classifier weights (W;) and feature representations (fo(z)) demonstrate distinctive trajectories analogous to "warmup" (dotted), "entropy-seekiin (solid), and "compression-sekkin" (dashed) phases. (D) Quantitative spectral metrics RankMe and eigenvalues, 01, 02.
maalytically-tractabl setting - the model fx))) is linear, i.e. fz Or Rd. and logits obtained (like in LLM models) applying a softmax operation on 2 (see Figure[4A) We extended the results ofeessks et al.] [2021) to study how W and (.) change when optimizing the loss using gradient descent. Notably, found two key properties of gradient descent that contribute to the emergent geometric properties of the representation space (see Appendix B for formal statements):
Wfg W0z E R The outputs are obtained by
Primacy bias: Representations and weights corresponding to nigh-frreuency tokens are learned earlier in training, compared to low-frequency tokens.
Selection bias: Dominant directions in the representation space are more likely to be used for encoding new information, i.e. Ao; oz
We demonstrate (c.f. Figure that two conditions are necessary (see supplementary for controls) for replicating the multiphase learning dynamics in our toy-model, observed within LLMs: (1) non-uniform class distribution, i.e. some tokens (or classes) occur more frequently than others in the training data, and (2) information bottleneck, i.e. number of feature dimensions (d) is less than the vocabulary size ((V)). Note that these two conditions are common in LLM pretraining setups. We defer the reader Appendix Figures 12 and for control experiments that ablate each of these conditions and demonstrate a monotonic, saturating expansion of the representation space.
In the analytically tractable setup that satisfies the above conditions, found that f.()) and W for reqquently-occuriig classes are separated during the initial "warmup" phase (Figure 4 & C, dotted lines). The corresponding eigenvectors of the weight and feature spaces also become aligned during this phase. Following this initial eigenvector-aligee aligmment phase, there is an overall expansion in the representation space that leads to higher confidence predictions for frequently This phase of volume expansion in the (o(-) and W spaces is associated with an increasing effective rank, akin to the "entropy-seeking infrequently-occurrin classes start to separate into their own clusters both spaces (Figure [4 & C, dashed lines). Constrained by the information bottleneck condition, the system resorts to reusing the feature space eigenvectors and more information is selectively encoded in the dominant direction (note a grows faster compared σ after 200 steps in Figure This phase of anisotropic information encoding leads to reduction in RankMe, akin to the peppression-seeiinn phase. Taken together, these results suggest that gradient- based enssopy optimization dynamics under specific training conditions may result in non non-monot changes in representation geometry we observed in LLMs.
classes.
phase (Figure 4B & C, solid lines). Following this phase, the
Key takeaway. Gradient descent on cross-entropy with (i) skewed token frequencies and (ii) representation bottleneck (d < (V/) suffices to produce expansion - compression via eigenvector alignment and singular-value growth proportional magnitude.
These mechanistic insights from simplified models establish fundamental principles governing representation geometry evolution. We now turn to examining how these geometric transformations manifest during post-training stages, where different loss functions further sculpt the representations.
7

---

A
B
1.30
200
10 Loua (Arthhoskoot)
0.9
Complexity
tollowing
generallzed ressesing
130
200
memortzaioo?
preference
0OD O0D Luma
128
270
Pass
0.8
Tulu FLVR stap 1200
Tuku8 OPO
Tok T RRIR see 2400
RLVR
130
200
Number generatione
Training Steps
1200
1900
Chat wirrate (5)
480
RankeMe
Pass 2:256
1170
RankeMe
Rankle
1003
-20
11:50
000
16
350
1130
800
DPO RLVR
340
700
25
Training Stepe (10")
175
Pretraining Tokens (10")
1.0
2.0
3.0
4.0
0
500
RLVR Training Steps
1006 1509 2000
2000
Figure 5: Post-training induces distinct geometric transformations in model representations, aligned with specifie behavioral changes. (A) Conceptual overview of post-training (SFT, DPO and RLVR) (top), corresponding Me metrics from intermediate checkpoints of Llama-3.1-Tüil 3.B-8B (bottom) highlighting distinct progression for each stage. (B) Impact of pretraining on OLMo- 2-1B SFT (Anthropic-HHI): (top) longer pretraining improves in-distribution (ID) performance, while out-of-distribution (OOD) generalization (Alpaca farm) saturates (bottom) Overtrained models with higher RankMe exhibit markedly distinct outputs on AlpacaEval after undergoing SFT on two different datasets (A (Anthropic- HH and Alpaca farm). (C) RLVR post-training narrows base model's (Llama-3.1 3.------.--B----- 1-8B--Tllu-3 Tulu-3-DPO) exploratory behavior on AMC-23 (particularly higher sampling counts e.g. k- = 256), suggesting higher effective-rank facilitates better search.
3.4
Representation geometric changes during Post-Training stages
While pretraining establishes the initial structure of LLM representations, subsequent post-training is instrumental for refining model capabilities and aligning them with downstream objectives. Here, we investigate the geometric changes that occur during each post-training stage. Our analysis centers on the Tülu-3.1 models [Wang et al., 2024), which utilize sequential three-stage post-training recipe- Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning with Verifiable Rewards (RLVR) applied the LLaMA-3.1-8B GGrttationi et al., 2024] base model.
SFT exhibits seetropp-seeking". : We find that SFT is associated with a monotonic increase in the RankMe, indicating an increase in the underlying representation manifold complexity. We hypothesize that the manifold expansion is related to instruction memorization on in-distribution (ID) examples, while reducing robustness to out-of-distribe -iitr--disttibution (OOD) samples. To test this, we perform SFT with Anthropic-HH dataset on OLMo2- IB intermediate checkpoints. As shown in Figure 5 B. we find that with more pretraining the ID loss Anthropic-HII improves monotonically, while the OOD loss (on Alpaca farm data) increases (see detailed ID/OOD loss in Appendix Figure I4). To understand the role of base-model geometry on the generalization gap, we perform SFT on Anthropic- and Alpaca farm (AF) datasets across checkpoints of OLMo2-1B. IB, and measure chat winrates for AH using AF as reference on the AlpacaEval dataset. Strikingly, we find Figure bottom) that while more pretraining coincides with an increase in RankMe, the winrates decrease for AH. Notably, drop winrate from 14% to 9% suggests that the L.MM judge is better able distinguish between the outputs of the two instruction-tuned models. This reinforces that "overtrained" base models, exhibiting higher RankMe, are more sensitive to distribution shifts under SFT [Springer et al. 2025)
(AH)
DPO exhibits "entropy-secking": Prior works in self-supervised vision pretraining Zhai et al. 2024 Ghosh al. 2024 have established that contrastive learning objectives, e.g. SimCLR, are associated with an increase in representation complexity, as the network progressively learns the relevant eigenfunctions (Simon al. 2023) to separate the positive and negative examples. We observe similar trend in the DPO stage, notably monotonic increase (decrease) in the RankMe (aReQ), c.f. fig. 5A. This parallel between the two settings can be attributed the analogous formulations the objective function. Note below that eq. 33) can be written as the Noise Contrastive Estimation (NCE)
loss [Gutmann and Hyvärinen] 2010 often used contrastive vision and multimodal pretraining [Oord et al. 2018 Chen et al.
2020 Radford et al. (021), with one negative example.
LDPO = -E [log(a vw) (1 v1))) = Ee.. w
log (Fo( + e ef((n.n))
(5)
8

---

RLVR exhibits "compression-seeknng" In sharp contrast to SFT and DPO, we observe that RLVR associated with monotonic decrease in RankMe (cf. TigureBAA) To probe the implications of this "compression-seeking math benchmark. Figure 55 shows that while R for 2400 steps outperforms the base (post-DPO) model at pass@16, the base model as well as an intermediate checkpoints outperform the RLVR-trained model at pass This decline in pass#256 performance as training progresses, reinforces prior work [Yue et al. 2025) suggesting that RLVR constraints the exploratory behavior of base models while amplifying some pre-existing behaviors of the base model [Zhao et al. 2025].
stage, we evaluate the unbiased passšk performance on AMC-23
Key takeaway. SFT/DPO (RankMe 1. aReQ J) enhance in-distribution fit but increase sensitiv- ity to dataset idiosyncrasies; RLVR (RankMe 4. aRRQ 1) consolidates reward-aligne behaviors and narrows high-k exploration (pass k), consistent with reduced solution diversity.
4 Related Work
Dynamics of Knowledge Acquisition and Representation Learning A central theme in under- standing neural networks is that learning is dynamic, often phased process rather than a monolithic one. Recent work by Zucchet et al. (025) identified distinct stages in how L.LMs learn factual information, highlighting the formation of critical internal structures like attention circuits during performance plateaus. This notion of staged learning further supported by the "Distributional Simplicity Bias" (DSB) established by Refinetti et al. (2023). Belrose et al.) [2024). which posits that networks learn simpler statistical properties of data (e.g., lower-order moments) before more complex ones. Our work provides geometric lens these phenomena, using spectral measures to track how the effective dimensionality of representations evolve non-monotonically, Furthermore, Michaud et al. [2023) proposed that scaling laws and emergent abilities arise from learning discrete "quanta" of knowledge. DeMoss et al. [2024] explained grokking [Power et al., [2022] as a transition from high-complexity memorization to cww-ooplexitty generalization, measured via algorithmic information theory. Our spectral geometric phases offer a complementary perspective that could underpin these observed emergent jumps in performance and the dynamics of grokking.
Post- post-Training Alignment and Reasoning The adaptation of pretrained LLMs through fnn-tuning is critical for aligning them with specific tasks and user preferences. Ren and Sutherland [2024] provided an empirical-NTK based framework to decompose the influence of fine-tuning updates, explaining complex behaviors such as hallucination amplification in SFT and the "squeezing effect" in DPO, where confidence in desired outputs can paradoxically decrease. Concurrently, [Springer et al. 2025 identified "catastrophic overtraining," showing that excessive pretraining can make models overly sensitive to parameter changes, thereby degrading performance after SFT. Our work contributes to this area by demonstrating that different post-training strategies (SFT, DPO, RLVR) induce distinct transformations in the geometry and it's influence model capabilities.
5 Discussions
Geometry of Pretraining: Memorization vs Generalization. We show that LLM pretraining is multiphasic rather than monotonic, characterized mainly by centropy-seekingg and "compression- seeking" phases. The observed geometric phases provide quantitative framework for examining the relationship between memorizing short-context statistics and generalizing long-context information. The "entropy-seeking phase expands the representational space to capture various short-context patterns, including n-gram memorization. Conversely, the "compression-seeke" more structured manifold and is likely to incentivize generalizable long-range language understanding. This geometric refinement process is consistent with and may offer an explanation for phenomena like grokking, where generalization capabilities can emerge after an initial period of fitting.
phase promotes a
Our preliminary analysis further reveals the importance of full-spectrum information for model per- formance. When we ablate eigenvectors to retain only the top-k principal components, SciQ accuracy degrades dramatically (Appendix Table 8). For instance, retaining only the top 10 digenndirections reduces Pythia- 1 s accuracy from 0.838 to 0.225, while OLMo-2-7B drops from 0.970 to 0.155. Interestingly, removing the top eigen-directions has minimal impact, suggesting that information is distributed across the full spectrum rather than concentrated solely in dominant directions. This finding validates our use of full-spectrum metrics like RankMe and aReQ rather than top-k proxies, and underscores that effective language understanding requires the entire representational mani-

---

fold-not just its principal components, The necessity of preserving full spectral information aligns with the ccompression-seeking phase's anisotropic consolidation, which selectively strengthens certain directions while maintaining distributed representations across the manifold.
Geometry of Post- test-Training: Alignment vs Exploration. Different post-training recipes induce distinct shifts in LLM representation geometry, potentially explaining the model's behavioral changes. Supervised Fine-Tuning (SFT) typically drives an "entropy-seekinn seeking"""nn dynamic, expanding the rep- resentational manifold for specific instruction-response examples. This manifold expansion can be seen as evidence for lazy-regime learning [Ren and Sutherland 2024] during SFT, and points to a near-diagonal empirical NTK that results in an restance-leve learning dynamics. Consequently, this dynamic improves in-distrbuttion performance but risks overfitting due to higher representa- tional capacity. In contrast, Reinforcement Learning from Verifiable Rewards (RLVR) promotes a "compression-seekine dynamic, refining representations towards reward-aligne directions. This geometric compression may explain how RLVR amplifies and refines existing capabilities (Zhao al. 2025), J. potentially by constraining representations to more structured subspace while reducing its exploration ability (Yue et al. 2025). In summary, SFT/DPO-induced rank expansion may foster preference memorization and exploratory behavior, while RLVR-induced consolidation amplifies model-capabilties towards reward-oriented, less diverse generation (c.f. Figure SC).
Limitations and Future Work Tracing a model's geometry, whether "entropy-seeke" seeking" or "compression-seeking" could inform more effective interventions for LLM development and evalua- tion, such as the selection of optimal pretraining checkpoints for targeted fine-tuning or designing training strategies that deliberately navigate these geometric phases. Our findings have several limitations: (i) computational constraints limited our analysis to models up to 12B parameters, though the phases persist across scales from 160M to 12B; (ii) spectral metric computation requires ~ 10K samples and scales quadratically with hidden dimension (iii) our theoretical analysis assumes simpli- fied linear feature extractors, leaving the extension to full transformer architectures future work; (iv) we focused on English-language models trained with standard objectives, and whether similar phases emerge in multilingual or alternatively-traiee models remains unexplored. Furthermore, our findings are primarily correlational; establishing causal connections between geometric dynamics and emergent capabilities requires additional investigation.
6 Conclusion
We show that LLMs undergo non-monotonic representation geometry changes, often masked by steadily decreasing training loss. By employing spectral metrics of feature covariates (RankMe and aReQ), we delineate three distinct pretraining phases: "warmup", "entropy--eekkng" (correlating with n-gram memorization), and "compression-seeking We further demonstrate that post-training recipes induce specific geometric changes: SFT/DPO exhibit "entropy-seeking" seeking" dynamics, whereas RLVR exhibit "compression-sede" seekineeeeneneee dynamics. These results provide a quantitative framework for guiding future advancements in LLM development.
(correlating with long-context generalization).
Impact Statement The goal of work is to advance the understanding of internal representations of LLMs. Although there are potential downstream societal consequences of this technology, we feel there are no direct consequences that must be specifically highlighted here.
Acknowledgments
The authors would like to thank Koustuv Sinha for insightful discussions that helped shape the scope of the project and Jacob Mitchell Springer for helping setup the OLMo-2 supervised finetuning pipeline. We are grateful to the OLMo team, particularly Nathan Lambert, Dirk Groeneveld, and Bailey Kuchl, for providing access to the OLMo-2 checkpoints (especially OLMo-2-IB) that enabled this research. The authors are also grateful to Daniel Levenstein, Johannes von Oswald, Jonathan Cornford, Mandana Samiei, Tejas Vaidhya, and Zahraa Chorghay for their comments and feedback. A.G. was supported by Vanier Canada Graduate Scholarship. G.L. was supported by NSERC (Discovery Grant RGPIN201 8-04821), the Canada Research Chair Neural Computations and Interfacing, CIFAR (Canada AI Chair), as well as IVADO and the Canada First Research Excellence Fund. B.A.R. was supported by NSERC (Discovery Grant: RGPIN-2020-051 105; Discovery Accelerator Supplement: and CIFAR (Canada AI Chair; Learning in Machines and Brains Fellowship).
10

---

The authors also acknowledge the material support of NVIDIA in the form of computational resources, as well the compute resources, software and technical help provided by Mila (mila.quebec)).
References
Kumar K Agrawal, Arnab Kumar Mondal, Arna Ghosh, and Blake Richards. a-req: Assessing representation quality in self-supervised learning by measuring eigenspectrum decay. Advances in Neural Information Processing Systems, 35:17626- 2022.
Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117644):3003--30077 2020.
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Fern. Neural networks learn statisties of increasing complexity. arXiv preprint arXiv:2402.04380 2024.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2397-2430. PMLR, 2023.
Davis Brown, Charles Godfrey, Nicholas Konz, Jonathan Tu, and Henry Kvinge. Understanding the working of language models through representation dissimilarity. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2/07.03374 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pages 1597-1607. PmLR,, 2020.
Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complexity dynamics of grokking. arXiv preprint arXi::2412.098/00 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pages 1747-1764, 2022.
Ieo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2/01.000) 2020.
Quentin Garrido, Randall Balestriero, Laurent Najman, and Yann Lecun. Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank. In International Conference on Machine Learning, pages 10929-10974, PMLR, 2023.
Arna Ghosh, Amab Kumar Mondal, Kumar Krishna Agrawal, and Blake Richards. Investigating power laws in deep representation learning. arXiv preprint arXiv:2202.05808. 2022.
Arna Ghosh, Kumar Krishna Agrawal, Shagun Sodhani, Adam Oberman, and Blake Richards. Hamnessing small projectors and multiple views for efficient vision pretraining, Advances in Neural Information Processing Systems, 37:39837-39868, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llaaa herd of models. arXiv preprint arXiv:2407.21778, 2024.

---

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerat- ing the science of language models. arXiv preprint arXiv:2402.00888, 2024.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pages 30016-30030, 2022.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1601- 1611, 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0883 2020.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. T,"" ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 151114, 2024.
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini-gram: Scaling unbounded n-gram language models trillion tokens. In First Conference on Language Modeling, 2024.
Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint 1474777706.1677,, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems, (6:28699-28722 2023.
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint 1rxiv:25011.0656. 2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 2018
Guilherme Penedo, Hynck Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL /wlLiattsss//ppeeeeeee.eeee www.arrreiiee..n///ouummmm D6GC
Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guil- laume Lajoie. Gradient starvation: learning proclivity in neural networks. Advances Neural Information Processing Systems, 14:1226-1272, 2021.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra, Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint 22X1772201.021177 2022.
12

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-878 8763. PmL.R, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-5334 2023.
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. In International Conference on Machine Learning, pages 18843-288611 PMLR, 2023.
Yi Ren and Danica J Sutherland. Learning dynamics of Ilm finetuning. arXiv preprint arXiv:2407. 10490, 2024.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606-610. IEEE, 2007.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
James B Simon, Maksis Knutins, Liu Ziyin, Daniel Geisz, Abraham J Fetterman, and Joshua Albrecht. On the stepwise nature of self-supervised learning. In International Conference on Machine Learning, pages 11852-31877 PMLR, 2023.
Aaditya Singh, Stephanie Chan, Ted Moskovitz, Erin Grant, Andrew Saxe, and Felix Hill. The transient nature of emergent in-context learning in transformers. Advances Neural Information Processing Systems, 36:27801-2788 2023.
Sidak Pal Singh, Bobby He, Thomas Hofmann, and Bernhard Schölkopf. Hallmarks of optimiza- tion trajectories in neural networks: Directional exploration and redundancy. arXiv preprint arXiv:2403.07373 2024.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder fine-tune. arXiv preprint arXiv:2503. 19206, 2025.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. dige--imensional geometry of population responses in visual cortex. Nature, 571(7765):361-365 2019.
Vimal Thilak, Chen Huang, Omid Saremi, Laurent Dinh, Hanlin Goh, Preetum Nakkiran, Joshua M Susskind, and Etai Littwin. Lidar: Sensing linear probing performance in joint embedding architectures. arXiv preprint arXiv:2312. 2023.
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models' capabilities back to pretraining data. In The Thirteemth International Conference on Learning Representations, 2025. URL https:tt:://// http:ttttt//tpeeeeeetee /pppnevieeereeeeeeee www/ffrunut 1ttp//oormmm22110000001mmmmm
Yizhong Wang, Ximing Li, Siqi Wang, Yash Khandwala, Aashi Anand, Yushi Yang, Sewon Lee, Hannaneh Hajishirzi, and Noah A Smith. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2404.0 070/00 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions Machine Learning Research, 2022.
13

---

Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv: FrXXiv1707.06200 2017.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXhv:2504.1388. 2025.
Runtian Zhai, Bingbin Liu, Andrej Risteski, J. Zico Kolter, and Pradeep Kumar Ravikumar. Understanding augmentation- based self-supervised representation learning via RKHS approx- imation and regression. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https //openreview
Rosie Zhao, Alexandru Meterez, Sham Kakade, Cengiz Pehlevan, Samy Jelassi, and Eran Malach. Echo chamber: RI post-training amplifies behaviors learned in pretraining. arXiv preprint arXiv:2504.079) 07911, 2025.
Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De. How do language models learn facts? dynamics, curricula and hallucinations. arXiv preprint arXiv:2503.21666, 2025.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have emphasized the main contribution of this paper in both the abstract and instruction sections-namely, that large language models exhibit distinct geometric phases throughout pretraining and post-training.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
14

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide an informal version of our theoretical results in Section 3.3 and the corresponding formal versions and proofs in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: We provide a detailed description of the methods used in our work Section 2 and experimental setups in Section 3 We provide further details of individual experiments, including specific model architecture, datasets used and training run configurations in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
15

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We used open-access datasets and models and plan release our code base on Github in the near future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
swi.iiossiippcccc.oo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
16

---

Answer:[Yor Yes]
Justification: We provide a detailed description of our experimental setups in Section 3 We provide further details of individual experiments, including specific model architecture, datasets used and training run configurations the Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: To ensure the reliability of our findings, we repeated our experiments on multiple subsets of data and reported the standard error as shaded error bars in the figures. Since our work leverages open- large models, we are unable to run experiments across several seeds to account for randomness in model training.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma eгтоr bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We present rough details about compute infrastructure in the Appendix. Due to the double-blind policy, we refrain from providing exact details of the compute infrastructure used in experiments. However, this information will be added to the Appendix upon acceptance.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our study (1) does not involve any interaction with human subjects, and (2) utilizes datasets that do not contain personally identifiable information, have not been withdrawn by their authors, and are not included on the NeurIPS list of deprecated datasets.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
An impact statement has been included in the conclusions.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly. harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learn from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We did not train any large-scale models or curate datasets for public release in this work.
18

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the datasets and models used in this work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No assets are released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
19

---

Answer:
[NA]
Justification:
This work neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs usage.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-srdddard components.
Please refer to LLM policy (https://neur))) http:/ppps..//oooeeeeee/e/////// for what should or should not be described.
20

---

A Model and Dataset Details
A.1 Model Architecture and training configurations
Table 1: Comparison of model architectures and training setups.
Pythia
OLMo-2
Position Embedding
Learned
Rotary (RoPE)
Norm Type
LayerNorm
RMSNorm
Norm Position
Pre-layer
Pre-layer
Dataset
The Pile (825 GB)
OLMoStack (4T tokens)
Optimizer
AdamW
AdamW
LR Scheduler
Cosine decay
Linear decay w/ warmup
Loss Function
Cross-Entropy
Cross-Entropy
Table 2:
Tulu Model Architecture and Training Setup
Component
Tülu
Base Models
Llama 3 base models
Position Embedding
Inherited from base model
Normalization Type
Inherited from base model (LayerNorm)
Normalization Position
Pre-layer
Instruction Datasets
Tulu3 Mixture(FLAN V2, OpenAssistant, WildChat GPT-4)
Training Techniques
SFT, DPO, RLVR AdamW
Optimizer
Learning Rate Scheduler
Linear decay with warmup
Loss Function
Cross-Entropy
A.2 Dataset Details
In this section, we provide an overview of the datasets used in our experiments.
FineWeb: The Feb dataset [Penedo et al. 2024) consists of more than 15T tokens of cleaned and deduplicated english text obtained from the web using CommonCrawl. While the full dataset contains 15T tokens, use the smallest subset, L.e. a subsampled ver- sion of the dataset consisting of 10B tokens. The dataset is accessible on HuggingFace at http://mugnnnngaaae.odddasssssgg
WikiText:
The Wikitext dataset (Merity et al. 2016) is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles Wikipedia. We use only subset of the dataset to perform early evaluations of RankMe and aRcQ, before running our final experiments using Fine
The SciQ dataset (Welbl et al. 2017 contains over 13K crowdsourced science exam questions about physics, chemistry and biology, among many others. The questions are in multiple-choice format with 4 answer options each. The dataset accessible on HuggingFace at owa.re/nnneeeeeo/aaaatttaaaaaaae
SciQ:
TriviaQA: The TriviaQA dataset (Joshi et al. 2017) is reading comprehension dataset containing over 650K question-answer-evidee triples. We use the TriviaQA dataset to evaluate a model's ability incorporate long-context information from the question in order to correctly answer it. The dataset is accessible on HuggingFace at
qqa
LAMBADA OpenAl:
This dataset [Radford et al. 2019 is comprised of the LAMBADA test split, pre-processe pe-prrccesedd by OpenAI, and contains machine translated versions the split in German, Spanish, French and Italian. We use this dataset evaluate the model's ''s text understanding capabilities. The dataset is accessible on HuggingFace at lntpss//huggiingfacec hwwp:///ww..tttttteeeee/eeeeeeae /laaaddaaaaaaaaaaaaaaaii
Anthropic Helpful-Harmless (HH): The Anthropic-HH dataset provides human preference data about helpfulness and harmlessness, and meant to used for training preference models Reinforcement Learning with Human Feedback (RLHF) setting, However,
variant of
21

---

this dataset for SFT. Specifically, we generate a human- auuman-asiistant chat dataset of ~ 161K samples by parsing the "chosen" responses for each instruction from the original dataset and using to finetune a base model by treating the "chosen" response as the target (similar to Springer et al. (2025). While such use of this dataset discouraged in practical settings, we use this modified dataset testbed for our SFT experiments. The original dataset is accessible on HuggingFace at http:///wggnn.a...oooduuus/////a
AlpacaFarm Human-ANN chat (AlpacaFarm): This dataset is created by following similar procedure mentioned above for the Anthropic- HH dataset, but for the Human Evaluation dataset of the AlpacaFarm evaluation set [Dubois et al. 2023). As result, this dataset consists of ~ 17.7K samples, and used as positive control our SFT experiments. Models that are finetuned on this dataset are expected to perform well on the AlpacaEval chat task (see below), compared to models that are finetuned on different dataset. This positive control essential to disentangle the in-distribution vs out-of-distributtion abilities of SFT-model. The original dataset is accessible on HuggingFace at
farm
AlpacaEval: AlpacaEval is an LLM-based automatic evaluation setup for comparing chat models in fast, cheap and replicable setting. We AlpacaEival models after undergoing SFT. Models that are finetuned on the Alpacalaarm dataset are expected to produce better chat models and generate responses more aligned human- human-preferred responses to instructions in the AlpacaEval setup. We defer the reader to the corresponding github repository for further details of the evaluation setup.
bench study the behavior of
AMC23: The AMC23 benchmark refers to a specific set of evaluations based on the American Mathematics Competitions. This benchmark is designed to assess the mathematical reasoning capabilities of advanced AI models using problems characteristic of the AMC series. For the evaluation of AMC23, utilize the resources and methodologies found in the Qwen2.5-Math repository. This repository is accessible at ttttps://githin wttthuu..oom//uen/M/..c wtt.c/om/mmmuae wwwmm//ueel///ree22222222 5- 5-Math and provides the framework for assessment process.
A.3 Compute and hyperparameter configuration details
Compute resources: All of our LLM inference experiments were run either on a single 80GB A100 40GB L40S GPU. The finetuning experiments (SFT and DPO) consisting of A100 GPUs.
single node
Codebase: All code and data will be released at https://github. http:/ittuub.cmmm/melllllllxx Tracing- the representation -f modeel
Hyperparameter
Value
Dataset
Fine Web web sample-1mple0 10BT
Max sequence length
512
Number of sequences
15000
Batch size
16
Table 3: Hyperparameter configurations used for computing RankMe and aReQ in Figure
A.4 Reproducing Tülu-3-8B SFT and DPO
We follow instructions from httpss/ggithbb.commlleee////gpno
for reproducing and gathering the intermediate stage checkpoints (both for SFT and DPO) without changing any hyperparamters.
22

---

Hyperparameter
Value
SFT dataset
Anthropic- HH or AlpacaFarm
Human-ANN chat (train split)
Max sequence length
4096
Batch size
16
Gradient accumulation steps
16
Learning rate
le-5
Learning rate schedule
Linear decay with 10% warmup
Number epochs
2
Loss reduction
sum
Seeds
0, 8, 42, 420
Table 4: Hyperparameter configurations used for Supervised VineTTuning (SFT).
Hyperparameter
Value
Base model
OLMo2-1B
In-distribution dataset
Anthropic-HH (test split)
dataset
AlpacaFarm Human-ANN chat (train split)
Max sequence length
1024
Number of sequences
10000
Batch size
32
Table 5: Hyperparameter configurations used for ID and OOD loss eval.
Hyperparameter
Value
Base model
OLMo2-1B
Dataset
AlpacaEval (test split)
Max new tokens
1024
LM judge
Cohere Command A
Table
Hyperparameter configurations used for chat winrate on AlpacaEval.
23

---

B
Gradient Descent and Cross-entropy theory
B.1
Setup
Let denote an individual instance (or imput token sequence of language), with its true class identity (or the next token's 's index in vocabulary) given by (s)) An (LLM) encoder, parameterized by 8, processes produce its contextualized embedding, fo(s) R2, where d is the embedding dimension. For batch, S, of b such instances, the encoder outputs a matrix So(S) E B Subsequently, predictions ÿ xxxx11 are generated by multiplying this batch embedding with weight matrix W E [d where [V] is the vocabulary size (or number of classes):
= Jo(SW
To simplify the setting such that it is analytically tracttbbe, we assume the embedding function fo(s) to be modeled as a linear transformation of the input, i.e. fa(s) - 0t s. where E IR xd is parameter matrix. For a batch, of such instances, represented as a matrix S E Raxdia whose row vectors are orthonormal (i.c., SS = D). The batch embedding is, therefore, fo( S0 E Raxd Here, din is the input feature dimension and d is the embedding dimension.
Note: By imposing this i.i.d. assumption, we ensure that learning on one sample does not change the output of another sample, i.e. no inter-sample interference. While we admit that this assumption is unrealistic, and learning to predict the next token of one sequence in an autoregressive setup affects the output of another sequence, we believe that this assumption enables a first step towards understanding the implicit effect of cross-entropy loss optimization using gradient descent. We note that our results do not strictly depend on this assumption, and can be extended to non-i.i.d. samples. We leave this to future work.
Note 2: Note that our assumption of a linear embedding model. fo. is also an aberration from the transformer-basee LLMs. However, we focus the effect of loss and optimization in this section and leave further investigation into the implicit bias of architecture future studies.
B.2 Linear approximation of Cross-entropy loss: Legendre Transform
Let us start by defining the cross-entropy loss for one example, which belongs to class c as:
e
LCE(S) log
-c + log Σee
(6)
E,
EdI
Note that eq. (6) is nonlinear in y, thereby making it harder to analyze the dynamical system in the parameter space that imposed by gradient descent. order arrive at an analytical understanding of the pradient-induced dynamics when optimizing eq. (6). we will do a linear approximation of Le using Legendre Transform, similar to Pezeshki al. [202T] Specifically, we will derive the Legendre transform of the nonlinear term, (, e's).
Intuitively, we want to approximate eq. (6) such that it changes linearly with changes in y. The key motivation for using Legendre transform ignore the second (and higher) order effects of a "small" change in y on LCF- Formally, we want the following:
(ce(s) = - + a + g(a)
(7)
where a is the slope of the nonlinear term at x ÿ.
a V Vxlog
x
i log( Σ e)))
c Σ, cli
(8)
dr;
2z=0/
Note that:
Σ az =1
24

---

To simplify things, let us denote C = Ej ebs. Substituting this in eq. (8. Ds log(a;) Cog(C).
Now need to find g(a) such that f(8) = a²ÿ + +g(a).
g(a) f(8) a' log( Σ cvi) -a'ÿ =log( Σ cbi) c Eonūs
= -log(C) - E -aylog(a))-> aslog(C)
= =log(C)- - Σ [3ylog(a)) - -Cog(C)
[Using Eaj =1]
=- aglog(ay) = (a)
(9)
where H(a) denotes the Shannon entropy of probability distribution defined by aç's. Substituting eq. in eq. 7). get the expression for the linearized cross-entropy loss: (CE(S) Ūc + +a + ((a)
(10)
B.3 Gradient descent dynamics of linearized cross-entropy loss
(CF($) = Ge + a 8- + H((a) -- + Lay H(a)
3
= = we + (g) wj + ((a)
CCR((() -. + Σ aywy E(as )
V Vn, CcE(s) = fo(s));=e + (=fo(s) = (a; - f
where )()) is the Dirac-delta function, i.c. its value is when the condition in subscript is true and 0 otherwise.
Denoting o: (a: - 8i=c), we arrive at the gradient equations for fo(s) and wy's:
V ((E()) Σājw, -Ea Vw:Loe(s) = āz
(11)
We can easily extend eq. 11 to multiple examples {$], 82. $t} and write the gradient descent update (using learning rate 7)) equations as:
fo(s;) - Σ āx(s;)
i - Σ (o(ss)
fo -WWT
W = nff A
(12)
where
Aij = (5)(s)) - i jj (ay(s))
hi example, 8;, belongs to the class j)
else
B.4 A useful matrix algebra result
Lemma 1. Let (t) be time-varying matrix with singular value decomposition (SVD): W(t) (()) where (t) and ((t) are orthogonal matrices corresponding to the left and right (t) contains the singular values along its diagonal. Let (t) and vn (t) denote the tth column vectors of U (t) and V (t), respectively. Then the time derivative of the t singular value, 0% (t), is given by:
singular vectors, respectively, and S()) = (iag((r1 (L),
oz (t) (t( W (f)vx(t)
25

---

Proof. For sake of brevity, we will drop the explicit time-dependence of each matrix from the notations. Let write the singular vector decomposition (SVD) of matrix, W USVT, Using the product rule of differentiation:
W =
U WV U'US+$
Š
O% u Wex - ui ok I
(13)
where the last line is the expression for the htth diagonal element of S. By definition of orthonormal vectors, 2i Ug 1. So, dig us u tig O. Since vig "u is scalar, vix" un uf tig. Therefore, un ug 0. Similarly, vi us O. Therefore.
σx = u1 W
B.5 Formal versions of theoretical results and proofs
Theorem Let fe UNS and W V22 denote the respective singular value decomposi- tions (SVDs) of non-degenerate matrices fe and W, respectively. If the system is initialized such that f fo WW',, then holds that:
U2
S =
Proof. Let us start from the learning dynamics imposed by gradient-deseent:
ja = -WWWT
w--nfa A
(14)
Let us write and W as their respective singular value decomposed form, i.. say fo U2S,V; VT" and W = U2S2 V2. Consider the dynamics of f fo and WWT:
d (o) = fo fo fofo = (-nAwT) f- dt
--nWA So -115 AW
(15)
d
ái (WWT) = WWT + +WWT (-nfo AJWT + W(n A)T
-nf AWT A fo
(16)
From eqs. 15) and 16 is clear that & #5/ fa) 4 (WWT),is i.e. f/ WWT +C, for some constant II II we assume the initialization to be such that C' = 0 and fo and W are non-degenerate,
we have:
o fo WWT = Vis = U2S3U
By uniqueness of SVD (for positive semi-definite matrices):
V U2
V -
S
S2
Theorem 2. Let fø, W be the matrices whose dynamics governed by the gradient-descent equations as previously defined. Given the conditions from Theorem | the magnitude of the time derivatives of the ith singular values of fe and W are proportional their respective singular values:
|| || x o |لجياا|
|| & dzs
26

---

Furthermore, assuming uniform class prediction at initialization and that number of classes, [V] > 1. the time derivatives are bounded by the dominant class size:
| 1|=2|| & O(N(c!))))
where (c) denotes the number of instances belonging to the dominant class c.0).
Proof. Let us start from the results of Theorem 1 S S = of = o vi. So, on + Using this relation, we can simplify the expression of σii dynamics. From Lemma]
tT "iiSovi- AWT on
A(US,A(U Un - AV2S2U on
= =Vuu{; Uni
[Using Theorem
17 E(uf, Awzy)o2) (02 = > Auzy
dn - 7 Au2 02 Similarly, we can simplify the dynamics for dz:
(17)
dz (w Au2))o1
(18)
For sake of brevity, let us denote A02i) Using the relationship between 01 and dz we can simplify eqs. (17) and (8) as:
δ1 ng (t0)s) F1gg.011
d2 -ng:(tto)) F79.02
(19)
|لrيel x 01 || ll 02
(20)
Also, note that 9i = Au2i = Ej Uiiz Uzae, U2as, where Ass = {an(8;) - 1, ax;)) Therefore, Ask- E 1-). At initialization, WLOG as (s,) M Vj,k, i.e uniform class prediction. Additionally, assuming
[V] >> 1, we can estimate gi as the following:
=
Uiig UsA
Σ
Σ
(16((ax(5)) - 1)U2kk +
Σ
J.k
jE(e==h)
sE(o,(()
91 2
U2ik +
U1ik
[
JElc,=k)
JE(C,
U2ik (
ulig) O(N(c))
(21)
jE(c==))
where (0) denotes the dominant class, i.c. the class with most number of instances. Combining eq. (21) with eq. (9 get the desired result:
|ل ||x2لl| (cN(c)))
(22)
L
27

---

Additional Experimental results
C.1 Generative behavior of LLMs: Qualitative inspection
stagel-step1000-5B
stagel-stee322000-88888
Sunlight filtered through dense canopy, dappling the forest floor
The first thing you can see is that the first thing you see in the first place is that you can see the first thing
The air was still and wars, and the only sound was the occasional chirp of bird. The forest was cuiet,
see in the first place. The second can see in first place. the the second place is that you
and the only novenent was the slaw, steady the trees, VilM snall, furry creature scurried through the underbrush, side. It squirrel, had
of
thing you see in the first place is that the
thing
eyes darting fron side to
second thing
looking for food.
the first thing you
An old, weathered map lay spread across the wooden table.
The table was large, large, and targe, and the table was large, large, and large, and was a large, and the table was large, and the table was a large, and the table was a large, and the table was a large, and the table was a large, and the table was large, and the table
The naр was of the area around the town of Harken, and it was narked with nunber of locations. the rap was covered with nates, and the notes were covered with a number of different colored markers. The narkers were of different shapes and sizes, and they were all labeled with nanes and nunbers. The nap was ress, and it was abvicus that it
table
A Lone ship sailed silently towards the crimson horizon
The sky was a dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark, dark
The ship was snall, two-nasted vessel, with single mast the ster, The with white stripe running the hull. The pirate ship, captain was pirate, and ship ship was captain was pirate,
peinted dark blue,
captain pirate. pirate ship.
Intricate carvings covered surface the
stone tablet
19th century was the first part the 19th century. The first part of the 19th century was the first part of the 19th century. The first part of the 19th century was the first part of the 19th century. The first part of the 19th century was the first part of the
The first part
The carvings were so Intricate that they were almost imposs ible to decipher, The tablet was covered in a thick layer of dust and dirt. The dust and dirt had been there for so long that it had becore part of the stone. It was as if the dust and dirt had been there since the beginning of time-Anynlle tablet was covered in thick layer of dust
faint nelody drifted froe an open window down the cobblestone street
The cars are the
expensive and the most the
The nusic was soft and sweet, and was enough nake young snile. dark
and the expensive. The the expensive
tall, handsone he carried violin seened confident stride,
expensive and nost expensive nost expensive and rost expensive rost expensive and nost exgensive and nost expensive and nost expensive
fine wa walked in no hurry. stopped front of small house and knocked
eyes,
An unspoken tension filled the crouded roon, heavy and palpable
The roon was large, heavy, and very targe, very large,
thick with scent of fear and stench of death, The roon was silent, save for the sound of the wind howling outside. The only light cane fron the flickering flares of the tire. The roon was dark and cold, and the only warnth came fror the badies of the iren and woren huddted together in the corner. The srell
The
large, large, very large, very large, very large, very large, very large, very large, very large, very large, very large, very large, very
large, large,
large,
large, very large, large, very
A singte drop of rain Landed on the dusty, parched earth
The earth was a great, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strang, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong, strong
It was the first rain nonths. earth drank it up greedily, and the thirsty plants and trees drank it up t0o. The thirsty aninals drank it up. The thirsty people drank it up. The rain was a blessing. It was a gift from God. It was a gift of Life. AnyaThe rain continued to fatl. The earth drank it up. The plants
Figure 6: Evidence of echolalia: Generated text from early checkpoints Sttep-1000) of OLMo-2-7B models, compared to that from later stages of pretraining (step-920000).
C.2 Examining Context Statistics and the Necessity of Full-Spectrum Information
28

---

Suffix Length
Frequency (%)
25.41
4
46.61
16.71
6
6.08
2.40
8
1.46
>8 8
1.34
Table 7: co-gram context in TriviaQA. Suffix lengths reveal focus on short- to mid-context statistics.
Model
Original
Top- 10
Top-50
Removed
Retained
Removed
Retained
Pythia-
0.838
0.849
0,225
0.835
0.318
Pythia-1.44 .4B
0.866
0.855
0.232
0.859
0.324
Pythia-2.8B
0.884
0.880
0.219
0.873
0.317
Pythia-6.9B
0.896
0.893
0,202
0.906
0.327
OLMo-2-1B
0.953
0.943 0.966
0.199
0.954
0.326
OLMo-2-7B
0.970
0.155
0.970
0.308
Table 8: Full-spectrum information is required. Retaining only top eigen-directions markedly degrades SciQ accuracy.
C.3 Computing spectral metrics, RankMe and aReQ
Model
aReQ (p-value)
RankMe (p-value)
Pythia-1B
0.810 (1. 5)
-0.759 (1.04e44)
Pythia-1.4B
0.668 (1.29e-3)
-0.713 (4. (8-44)
Pythia-2.8B
0,694 (6.88c-4)
-0.635 (2.63e-3)
Pythia-6.9B
0.837 (4.20e-6)
-0.885 (9e-9))7)
Pythia-12B 12B
0.836 (4.42e-6)
-0.839 (3.79e-6)
OLMo2-1B
0.540 (4.990)-))
-0.616 (3.201e-9)
Table 9: SciQ accuracy correlates with spectral geometry. Positive correlation with aRRQ (compactness) and negative correlation with RankMe (effective dimensionality) across Pythia (1-12B) and OLMo2-1B IB models. The p-values in parentheses indicate high statistical significance for all correlations.
29

---

Pythia
OLMo 2
100
Lapez
Lawer I
Layez
330
Layo
Layerd
Lapec 7
14
Lanc
Lape:9
Layez
Lawer
Layo:
Pretraining Tokens
Pretrsining Tokens
Figure 7: Layerwise evolution mirrors the three phases. Spectral metrics (RankMe and aReQ) computed across intermediate layers during pretraining show that the three- three-phase pattern is consistent across network depth, justifying the use last-layer representations for tracking global geometric dynamics. See Appendix for additional robustness analyses across samples, sequence lengths, and datasets.
OLMo-2 1B model
RankMe
5k samples
10k samples
Sk somples
256 jakens long
20k samples
512
aReQ
10" 10F 10a
16th
10"t 10
10'
10a
10"
10
10"
10
Pretraining Takens
Pretraining Tokens
warmup
entropy-seeking
compression-seeking
Figure 8: RankMe and aReQ computed for intermediate checkpoints the OLMo-2-1B model using (Left) different number of samples, and (Right) sequence length. Shaded error bars indicate standard error about mean.
Pythia-1b
Pythia-1.44
Pythia-2.8b
Pythia-6.99
Pythia12b
800
RankMe
400
2.0
aReQ
1.5
1.0-
10a 10 1
10 10a
10
10a 10a
10
10
10 1 10
10a 10a 10 10" Pretraining Tokens
Pretraining Tokens
Pretraining Tokens
Pretraining Tokens
Pretraining Tokens
Wikitext
MMLU
SciQ
Lambada Laabbada_openaii
FineWeb
Figure 9: RankMe and aReQ computed for intermediate checkpoints of models from the Pythia family on different datasets.
30

---

OLMo-2 1B
900
14
RankMMe
aRoQ
1.
164
166
16
3012
Pretraining Tokens
Pretraining Tokens
WikiText
MMLU
SciQ
Lambada openai
FineWeb
Figure 10: OLMo spectral metries are robust across datasets. RankMe and aReQ computed for intermediate checkpoints of OLMo-2 IB model on different datasets, showing consistent phase patterns across evaluation data.
RankMe
400-
aReQ
300
10P
10P
10
10"
Pretraining Tokens
Pretraining Tokens
Pythia-70M
P60M
Pythia-410M
Figure 11: RankMe and aReQ computed for intermediate checkpoints of smaller models IB) from the Pythia family on the FineWeb dataset.
31

---

C.4 Control experiments verifying the necessity condition for multiphase learning dynamics
Rankme Eigenvals
2-
125
1-
10.0
Eigewaa
Ellediee
75
-1-
-1-
5.0
-2-
-3-
100
200
300
Epochs
(a) Feature and weight dynamics in analytically-trace tractable model with uniform class distribution, i.c. each class has equal number of samples. Here, cach class has 2 samples each.
Rankme Eigenvals
2
3-
2.8
12
2-
10
2.6
1-
Efectie
Elgarwall
2.4
0-
-1-
2.2-
rankmma
-1-
-2
-2-
-2
200
Epoche
(b) Feature and weight dynamics in amalytically-tractiei tractable model with information bottleneck, feature dimensionality, d, is comparable to number of classes, [V] Here, d 3 and [V] 4. Note that we only plot the first two dimensions for case and consistency of visualization.
Negative control experiments analogous to Figure Removing either the skewed class distribution or the information bottleneck gets rid of the three distinct phases of learning. In each case, the resulting dynamics an initial "warmup"" followed by an "entropy-seeking phase wherein effective rank continues to grow monotonically.
Figure 12:
32

---

6
Rankme Eigenvals
1.0
10-
12
0.5-
0.0
0.0
0.6
-0.5
-0.-
nankma
1.92
-1.0
-1.0
1.90
-04
-15
-1.5
100
200
300
Epochs
(a) Feature and weight dynamics in amalytically-tractah model trained using MSE loss uniform class distribution, i.e. each class has equal number of samples. Here, each class has 2 samples each.
1.5
6
1.5
Wi
2.00
Rankme Eigenvals
+ 1.75
1.98
0.5
0.5
Elledive cark.
1.25
Fgemves
1.98
1.00
-0.5
-0.6
1.94
rankme
0.50
-1.
1.92
Epochs
(b) Feature and weight dynamics in analytically-trattable model trained using MSE loss on skewed class distribution. Note that only information about the most frequently occurring classes are learned.
Figure 13: Negative control experiments analogous to Figure 4 with mean squared error instead of cross-entropy as the training loss. In both uniform and skewed label distribution settings, the resulting dynamies is an initial "warmup". followed by an "entropy-seeking phase wherein effective rank grows monotonically and quickly saturates.
33

---

C.5 Supervised finetuning
Last
109
-90
18-
600
000 Dioc-enting loee
Anthrosie-Hit Alpata-
468
1:5-
280
Chit streade
450
12
60-
380
-900
325
Prefraiiigg Toleett
1e12
Pretrsiing Tokans
1e12
Preisining Tobero
1o12
Figure 14: Loss and chat win rates after SFT on Anthropic-HH dataset. (Left) Cross Cross-entropy loss on in-distribution (Anthropic-Hle test set and -uist--distributionn (Alpaca-Fami) Saancc-Farm--ae Harm-human-ANN chat) dataset. While in-distribution loss after SFT decreases monotonically, ood loss after SFT saturates gets slightly worse with longer pretraining. (Center) Length--ontrooleed chat win rates for Anthropic-SFT vs Alpaca-SFT version of base model on AlpacaEval. Longer pretraining increases the sensitivity of the model's behavior the SFT dataset. (Right) Length-controlled win rates for Anthropic-SFT version of intermediate base models compared to the Anthropic-SFTT version of the final base model checkpoint. Models obtained from later in pretraining are equivalent chat models, demonstrating nearly 50% win rate compared to the final checkpoint. Choosing the ideal checkpoint to use for SFT requires navigating the tradeoff between an improvement in base model's capability (note increased RankMe) and reduction in robustness with longer pretraining. Shaded bars indicate standard deviation computed over seeds.
34


Reviews:
[{"review_id": "VoDKAIe8j2", "reviewer": "Reviewer_Ts4j", "summary": "This paper explores the evolution of the covariance spectrum of neural network embeddings across training time. During large language model pre-training, three phases are uncovered: a warmup phase wherein the spectral entropy collapses, an entropy-seeking phase where the spectral entropy recovers, and a compression-seeking phase where spectral entropy decreases again. Interestingly, SFT and DPO training are associated with entropy-seeking behavior, whereas RL with verifiable rewards is associated with further compression of the covariance spectrum.", "questions": "1. To what extent are there outlier directions in these embedding spaces? Do they affect the analysis at all?\n2. Which of your two metrics— RankMe or alpha-ReQ— do you think is more useful and/or appropriate for analyzing embedding geometry? At first glance it seems that alpha-ReQ might be more outlier-resistant, but also makes a distributional assumption about the empirical spectral measure (power law).\n3. I think I'm still confused about Figure 5B (bottom). In the caption it sounds like you trained on two different chat datasets, but there is only one chat win rate reported. Is this OOD or ID chat performance?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The discovery that SFT and DPO have the opposite effect on the embedding geometry compared to RLVR is a very interesting result.\n- Simple, clear, and timely research question, especially the connection to RLVR.\n\nWeaknesses:\n- Did not directly address the possible effect of outlier directions on the analysis. Prior work such as _All Bark and No Bite_ by Timkey and van Schijndel (2021) have found that there are often 1-3 \"rogue dimensions\" in the residual stream with extremely high variance, which dominate measures like Euclidean distance. Outlier directions, if they exist, would also strongly affect the spectral entropy and RankMe measures, and it is unclear if they should be included or not. Figure 4D does show the top two eigenvalues, but it is unclear what their relationship is to the rest of the spectrum— are they outliers? Direct discussion and investigation of this issue would strengthen the paper significantly (if only to show that outliers are irrelevant here).", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments and suggestions, and we are glad that they found our post-training results interesting and our work to be clear and timely. Below, we address all the concerns and questions raised by the reviewer.\n\n### Rogue dimensions\n\nThe reviewer is correct in noting that rogue dimensions in the LLM representation space affect the RankMe measure, leading to a lower effective rank than the representation dimensionality. This is particularly pronounced in models in the final stages of pretraining, which exhibit a much lower RankMe (computed using the entire eigenspectrum) and therefore indicate an inefficient use of their representation capacity, consistent with Timkey & van Schijndel's findings. In line with the reviewer’s comment, we will add an Appendix figure illustrating the impact of removing these rogue dimensions on RankMe.\n\nIn contrast, $\\alpha$-ReQ quantifies the decay rate of the eigenspectrum tail, specifically the slope of the power-law decay fit to the eigenspectrum after the top 10 eigenvalues. By disregarding the top (rogue) eigendirections, $\\alpha$-ReQ is less influenced by a few rogue directions. However, a strong inverse correlation between the two metrics indicates that the LLM representation space eigenspectrum is significantly skewed beyond just the top rogue dimensions.\n\nOur work demonstrates that analyzing the degree of this skewness using our proposed spectral metrics, and observing its evolution across pretraining and post-training, provides deeper insights into model behavior. We concur with the reviewer that integrating these distinctions between the two metrics and their connection to Timkey & van Schijndel's work into the discussions will enhance our paper, and we propose to implement these changes in the final version.\n\n\n### $\\alpha$-ReQ vs RankMe\n\nWhile $\\alpha$-ReQ is more robust to outlier rogue dimensions than RankMe and is based on an empirically observed power-law distribution of tail eigenvalues, both metrics are strongly anti-correlated and largely interchangeable. However, it is worth noting that RankMe is more sensitive to the ambient dimensionality of the representation space and dataset variations (Fig. 8), making cross-model comparisons challenging (Fig. 9).\n\n### Fig 4D clarification\n\nWe sincerely apologize for this confusion. Figure 4 presents the simplified setting that seeks to verify our analytical results about learning dynamics of cross-entropy loss. In these particular plots (and in Fig 10a), we set the feature dimensionality, $d=2$. Therefore, the eigenspectrum has only 2 eigenvalues, $\\sigma_1, \\sigma_2$ $\\textemdash$ the ones that are plotted in Fig 4D.  We will clarify these points in the updated manuscript.\n\n### Fig 5B clarification\n\nWe apologize for the confusion in these results. The reviewer has correctly noted that we fine-tuned (SFT) models on two different datasets: Anthropic-HH (AH) and AlpacaFarm (AF). Both these sets of models were then evaluated on a third, out-of-distribution (novel) dataset: AlpacaEval.\n\nMoreover, it is worth noting that AlpacaEval is more out-of-domain for AH models than for AF models. Anthropic-HH is a chat dataset focused on helpfulness and harmlessness, with conversational, clarifying, and thought-provoking responses to complex user queries. In contrast, AlpacaFarm and AlpacaEval are generic chat datasets with straightforward, atomic prompts that elicit short, concise responses. The qualitative differences in chat responses and prompt complexity make AlpacaEval a more significant shift for AH models.\n\nThe winrate is computed by comparing the AH and AF model responses for the AlpacaEval prompts. This comparison is done with a larger LLM judge which picks the more appropriate answer among the two. As described above, AlpacaEval is more out-of-domain for AH models than for AF models. This is reflected in the low winrates (6-14%) of AH fine-tuned models with respect to AF fine-tuned models. Our key observation is the decrease in this winrate with more pretraining of the base model, suggesting that “overtrained” models (those with higher RankMe values) perform better on in-domain tasks but worse at generalizing to out-of-domain data. \nWe will incorporate these clarifications about the experiment design in the final version of our paper.\n\n---\nWe thank the reviewer once again and are grateful for their helpful suggestions. We believe that incorporating discussion of rogue dimensions and the nuances about the two spectral metrics will strengthen our paper and improve its readability. We hope that the reviewer finds our responses to be helpful, and are happy to clarify any further questions during the discussion phase."}, {"review_id": "Uw7sy4VrW5", "reviewer": "Reviewer_61qF", "summary": "This paper studies final token representations, in the last layer of an LLM over the timecourse of pre and post training. By looking at summary measures related to the rank of the covariance of these final token representations the work presented identifies three phases of training - warmup, entropy seeking, and compression seeking. This leads to claims regarding what is acquired in each phase of training, and how this relates to model behaviour.", "questions": "Are there any significant correlation results between the representational measures used here and performance on a task?\n\nWhat is the relationship between the memorisation measure and dependency length? What is the difference between long-distance dependencies and long-context factual knowledge?", "limitations": "As mentioned above in strengths/weaknesses, the limitations section discusses the correlational nature of the evidence here. However the results themselves include virtually no statistical testing, and so appear more associative & speculative, than correlational.\n\nIt might be worth noting in the limitations section that results seem to focus on representations in only the final token, in only the final layer. While this allows the authors to draw some interesting conclusions it is distinct from studying the model in toto.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The work presented here is interesting, and seeks to address a number of important questions in interpretability. The use of a geometric framing presents some novelty -- the methods used do not appear novel to this work, but the experiments here apply them to a broader set of questions and models. In the general case the geometric perspective is distinct from mechanistic interpretability and somewhat underrepresented in the literature. However, the paper itself is a bit dense - trying to fit an large number of results into 9 pages sometimes at the expense of clarity. Additionally many of the claims feel somewhat out of scope with the results on which they're based. Below I highlight a couple of high-level concerns before providing some more detailed feedback on the results. \n\nIn the limitations section the authors note \"our findings are primarily correlational in nature; additional theoretical and empirical investigation is required to develop deeper mechanistic insights and establish causal connections between these observed geometric dynamics and model behavior.\" However it appears that the findings throughout this paper use virtually no statistical testing. The only correlational results I noticed were related to the memorisation measure which is based around a spearman correlation. Given that the Pythia suite releases multiple initialisations for each size the authors could have tried to correlate representation complexity for different seeds of a model with performance on a benchmark. This would constitute more compelling evidence of the link between their representational measures and performance. Throughout there are a number of claims that appeal to the time-series plots but could be made stronger with statistical testing; to give one example line 270 claims \"This decline in pass@256 performance as training progresses, reinforces prior work ... suggesting that RLVR constraints the exploratory behavior of ... base models while amplifying some pre-existing behaviors of the base model\" -- there doesn't appear to be a quantification of the relationship between pass@256 performance and representation structure. I broadly agree with the authors that correlational results can be used to preform interesting explorational analyses, but currently this work seems to present results that are more speculative & associative than correlational.\n\nWhile the high-level methods are well-explained I lack some clarity of the particular instantiation for the experiments presented here. How many samples are used to compute the estimate, and with respect to what dataset? There appear to be some benchmarking experiments for different sample sizes in the appendix, but I'm unsure if a single reference corpus is used for computing RankMe estimates or if, for example, in section 3.4 estimates are computed with respect to AH and AF respectively, or with resepect to a combination of the two. This would seem to affect the interpretation of the results in terms of whether models expand or compress in the general case or only w.r.t. the data on which they are fine-tuned.\n\nSome claims seem weakly related to the experiments performed. As an example line 304: \"This geometric refinement process is consistent with and may offer an explanation for phenomena like grokking.\" The results related to grokking appear to be based on results from a multiple choice dataset (triviaQA) - it seems likely the sudden jump in performance is more related to the fact that it is multiple choice than grokking effects like those studied on models of modular addition. Given the large number of results already in the paper, authors could remove more speculative lines of argumentation like this to improve clarity. The associative nature of some claims is reaffirmed by the inclusion of plots that have \"representational complexity\" on the y axis, and pre/post training on the x axis neither of which have units. The version of this plot in figure one also includes what appear to be suggestions about what happens in each phase -- I'm not quite sure what the evidence for some of these names are (authors draw a distinction between when \"long-range dependencies\" are learned, and when \"long-context factual knowledge\" is learned maybe based on a minor increase in \"representational complexity\" or increased performance on the aforementioned QA task). This visual also sometimes includes question marks to make clear the claims are speculative (e.g. \"preference memorisation?\").\n\nIn general this is interesting work, that would benefit from a more-focussed argument with more tightly scoped claims. Or which may be better suited to a venue that publishes longer format works to allow the authors the space necessary to fully unpack the breadth of ideas presented here.\n\n_______________________\nDetailed Feedback\n\nLooking at the results in section 3.2; the ngram correlation is introduced as a measure of memorisation, but the analysis discusses the dependency length learned by the model - it is non-trivial to relate these two. A counter argument could be that learning local dependencies is reflected in correlation with a bigram or trigram lm, and longer dependencies correlate with larger n's approaching the infinite case -- see work like Someya et al. 2025 for an example of more nuanced treatment of dependency length. There are also substantial differences between the performance on two different factual knowledge benchmarks over the course of training - these benchmarks differ primarily in whether or not they are multiple choice. The substantial difference in empirical results between these two is attributed to the dependency length needed to answer the question, an assertion that is presupposed rather than being argued for, and which is still not clearly related to the \"memorisation metric\". \n\nSection 3.3 claims to show: \"Taken together, these results suggest that gradient-based cross-entropy optimization dynamics under specific training conditions may result in non-monotonic changes in representation geometry we observed in LLMs.\" It would seem one of the natural control conditions for this claim would be to use an objective other than cross-entropy, however I cannot find results to this effect in the main paper or supplementary materials. Short of this it is unclear the role that cross entropy plays or the interaction between the objective function and the two independent variables used here (non-uniformity, and an information bottleneck).\n\nSection 3.4 looks at how post-training affects in-distribution and out-of-distribution performance on two post-training datasets. It is worth noting that while the AH and AF datasets are distinct, the claim that they are \"out of distribution\" from each other is not motivated. I'll note the evaluation procedure at line 250 is too obliquely explained for me to have a strong intuition for what the decrease in winrate means. Additionally the formal relation exemplified by equation 5 is currently introduced too briefly for me to have strong intuitions about its importance or relevance to the narrative through-line. \n\n_______________________\nReferences\n\nSomeya et al. 2025\nhttps://arxiv.org/pdf/2506.05136", "was_revised": true, "final_justification": "In the general case, I appreciate the willingness of the authors to engage with the review process. However at this time I still have concerns about the nature of the statistical testing performed - which I believe is key to making the claims this paper hopes to. I also still have concerns about whether the experiments relating to context length are sufficiently well controlled.\n\nOverall my review noted that this paper is quite dense, and the rebuttal has left that largely unaddressed. While this work is interesting it currently introduces a lot of ideas for 9 pages, as a result information that seems critical to the reader has been omitted in places (eg. that the infini-gram model is infact a quad gram, or that previous work has suggested SciQ is stable under sliding window attention).\n\nHaving looked at the other reviews I believe my original score stands - I do not think this work is currently ready for publication, but look forward to a future more polished version of it.", "rebuttal": "We thank the reviewer for their astute observations, encouraging comments and helpful suggestions, and we are glad that they found our work to be interesting and timely. Below, we address all the concerns and questions raised by the reviewer.\n\n### Statistical analysis\n\nWe agree that adding statistical testing will strengthen our results. Below, we present spearman correlation analysis results for the following results:\n\n1. **Correlation with task performance:** We compute the correlation between the spectral metrics and downstream task performance (SciQ and TriviaQA) across different pretraining checkpoints for Pythia models.\n\nSciQ\n| Model | $\\alpha$-ReQ (p-value) | RankMe (p-value) | \n| :--- | :---: | :---: | \n| pythia-1b | 0.810 (1.50e-5) | -0.759 (1.04e-4) | \n| pythia-1.4b | 0.668 (1.29e-3) | -0.713 (4.18e-4) | \n| pythia-2.8b | 0.694 (6.88e-4) | -0.635 (2.63e-3) | \n| pythia-6.9b | 0.837 (4.20e-6) | -0.885 (2.19e-7) | \n| pythia-12b | 0.836 (4.42e-6) | -0.839 (3.79e-6) |\n\nTriviaQA\n| Model | $\\alpha$-ReQ (p-value) | RankMe (p-value) | \n| :--- | :---: | :---: | \n| pythia-1b | 0.915 (8.263e-11) | -0.899 (9.844e-12) | \n| pythia-1.4b | 0.924 (9.251e-12) | -0.915 (2.361e-12) | \n| pythia-2.8b | 0.905 (9.586e-11) | -0.898 (3.645e-11) | \n| pythia-6.9b | 0.954 (8.969e-11) | -0.898 (4.675e-15) | \n| pythia-12b | 0.840 (2.223e-3) | -0.790 (6.428e-4) |\n\n2. **Correlation with pass@256 performance:**  Since both RankMe and pass@256 values are nearly monotonically decreasing, the spearman rank correlation is ~1. We will extend our RLVR analysis to more checkpoints and compute the correlation for these results, and update Fig 5C.\n\nWe propose to add these results to the Appendix, in order to strengthen our claims about the correlational behavior of our findings.\n\n### Experiment details for computing spectral metrics\nWe apologize for the lack of clarity in these experimental details. For all experiments, we used 15k sequences from the Fineweb dataset and extracted the last layer representations after processing sequences of 512 tokens. We will add these details in Section 2.1 and amend it to have references to the ablation experiments in Fig 7 and Fig 8. We hope that this also clarifies the related concerns about the results in Section 3.4.\n\n### Clarification about grokking\nWhile we agree the mechanism of grokking here differs from those studied in models of modular addition, we would like to clarify that TriviaQA is a factual question-answering dataset, not multiple-choice. Consequently, TriviaQA performance is measured by Exact Match metric via LLM harness. Nevertheless, we concur that \"grokking\" is an overloaded term and used it to describe a sudden increase in 0-shot task performance. To improve clarity and avoid overstatement, we will replace \"grokking\" with \"rapid increase in 0-shot performance\" in the text.\n\n### Clarification about Fig 1\nFig 1C summarizes our key findings graphically without diving into the technical details, using \"representation complexity\" to denote representation space entropy and only distinguishing between pre-training and post-training stages on the x-axis. While we believe it offers a high-level overview, we acknowledge the reviewer's feedback regarding clarity and avoiding speculation. Therefore, we will clearly differentiate between established and speculative claims in the figure and note this distinction in the caption.\n\n### Clarification about Section 3.2 results\n1. **Infini-gram results:** We apologize for the lack of clarity in connecting our results to the claim about short-sequence memorization. The infini-gram model, despite its name, does not use a very large context window for predictions. It is a generalization of the n-gram model, where 'n' is not fixed and instead set to the length of the longest suffix found in the corpus. For samples from the TriviaQA dataset, we analyzed the suffix length from the question used to predict tokens in the answer and found that the mean was 4.21 tokens (median 4.0, max 12). A more detailed breakdown of the frequency of different suffix lengths is presented below.\n| Suffix Length | Frequency (%) | \n| :--- | :---: | \n| $\\le$3 | 25.41 | \n| 4 | 46.61 | \n| 5 | 16.71 | \n| 6 | 6.08 | \n| 7 | 2.40 | \n| 8 | 1.46 | \n| $>$8 | 1.34 |  \n\nThis analysis indicates that the infini-gram model's predictions primarily depend on short- to mid-range dependencies. While we acknowledge the reviewer's valuable suggestion of a more nuanced analysis using bigrams and trigrams, the computational burden of training several n-grams on the entire pretraining dataset is not feasible within the given rebuttal timeframe. Therefore, we have restricted our analysis to using infini-grams to calculate the memorization metric, and we will clarify the precise meaning of this measure in the updated manuscript.\n\n2. **Context length dependencies for SciQ and TriviaQA:** The reviewer is correct that a key difference between SciQ and TriviaQA is that SciQ is a multiple-choice task, whereas TriviaQA is not. However, TriviaQA is included in the LongBench dataset [1] and is considered to require long-context understanding. In contrast, Lou et al. showed that SciQ accuracy, unlike TriviaQA performance, remains relatively stable even when using sliding window attention (Tables 2 & 4 in [2]), indicating that long-context information is not essential. Based on these results, we believe the performance discrepancy between the two benchmarks is in part due to the different dependency lengths needed to answer questions in their respective datasets. But, there may also be an effect due to the increased reasoning requirement to answer questions in the TriviaQA dataset, and we will note this as another possible interpretation in the updated text.\n\nWe hope that this clarification about the context-length required for the two tasks and the relationship between the memorization metric and short-context understanding ability will help resolve the logical jump pointed out by the reviewer.\n\n### Control experiment for clarifying the role of cross-entropy\nWe agree that using an objective other than cross-entropy, e.g. mean squared error (MSE), would be a natural control for our claim. We extended our theoretical analysis in Appendix B to derive the learning dynamics under an MSE loss. Simulations in the analytically tractable settings show that the eigenvalues monotonically increase (with $\\sigma_1$ increasing faster than $\\sigma_2$ initially) and eventually saturate. Consequently, the RankMe first decreases (warmup phase), then increases (entropy-seeking phase), and finally saturates. Notably, unlike the cross-entropy case, there is no smoothness-seeking phase when using MSE. This result further supports our claim that cross-entropy loss's learning dynamics are a key factor in the multiphase changes in representation geometry. We will add the MSE learning dynamics derivation and plots to the Appendix.\n\n### Clarification about Section 3.4 results\n1. **Ood nature of AH and AF:** As mentioned in Section A.2, AH is a chat dataset aimed at incorporating helpfulness and harmlessness, while AF is a generic chat dataset. The out-of-distribution (ood) nature spawns from the fact that the chat responses in the dataset are qualitatively different. While responses in AF are more geared towards short and concise responses, responses in AH are more conversational, clarifying and thought-provoking. Furthermore, the prompts in AF are relatively straightforward and atomic in nature, i.e. they ask the model to do a single task, whereas the prompts in AH are complex user queries and often require deeper reasoning and understanding of the underlying intent. These qualitative differences make the two datasets distinct and inherently ood to each other.\n\n2. **Intuitive explanation of the chat winrate results:** The winrate reported in Fig 5B (bottom) is computed by comparing the AH fine-tuned and AF fine-tuned model responses for prompts from a third, novel dataset: AlpacaEval. This comparison is done with a larger LLM judge which picks the more appropriate answer between the two. It is worth noting that AlpacaEval, which has prompts of similar style and complexity to AF, is more out-of-domain for AH fine-tuned models than AF fine-tuned models. This is reflected in the low winrates (6-14%) of AH fine-tuned models with respect to AF fine-tuned models. Our key observation is the decrease in this winrate with more pretraining of the base model, suggesting that ``overtrained'' models (those with higher RankMe values) perform better on in-domain tasks but worse at generalizing to out-of-domain data. \n\n3. **Intuitive explanation of the DPO objective reformulation:** Equation 5 establishes the equivalence between the DPO objective and the contrastive learning objective. This equivalence suggests similar learning dynamics for both, which could explain the empirically observed entropy-seeking phase during the DPO training stage. We aim to further elaborate on this relationship and connect DPO learning dynamics to previous works on self-supervised learning dynamics in the Appendix of our paper's final version. \n\n---\nWe thank the reviewer once again for their astute comments and are extremely grateful for their helpful suggestions. We believe that adding the statistical testing results and clarifications will strengthen our work and improve the clarity of the presentation. We hope that the reviewer finds our responses to be helpful, and are happy to clarify any further questions during the discussion phase.\n\n**References:**\n1. Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L. and Dong, Y., 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.\n2. Lou, C., Jia, Z., Zheng, Z. and Tu, K., 2024. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747."}, {"review_id": "om4gOLHQBb", "reviewer": "Reviewer_7SLc", "summary": "This paper investigates the evolution of representation geometry in LLMs throughout pretraining and post-training using spectral analysis methods. The authors employ effective rank (RankMe) and eigenspectrum decay rate (αReQ) to characterize representation complexity and identify three distinct geometric phases during pretraining: \"warmup,\" \"entropy-seeking,\" and \"compression-seeking.\" They demonstrate correlations between these phases and model capabilities, with entropy-seeking associated with memorization and compression-seeking with generalization. The analysis extends to post-training methods (SFT, DPO, RLVR), showing that different alignment techniques induce distinct geometric transformations. The work provides both empirical validation across multiple model families (OLMo, Pythia, Tülu-3.1) and theoretical grounding through analytically tractable toy models.", "questions": "1. **Mechanistic Validation**: Can the authors provide more direct evidence that the geometric phases causally influence capability acquisition? For instance, interventions that artificially modify representation geometry during training could test whether this affects the emergence of memorization or generalization capabilities.\n\n2. **Robustness Analysis**: How sensitive are the identified phases to hyperparameter choices (learning rate schedules, batch sizes, model architectures)? Understanding this robustness is crucial for determining the generality of the findings.\n\n3. **Alternative Metrics**: Have the authors considered other measures of representation geometry (e.g., intrinsic dimensionality, alignment metrics, or information-theoretic measures)? Validation with alternative metrics would strengthen confidence in the conclusions.\n\n4. **Layer-wise Analysis**: The paper focuses on last-layer representations, but how do these geometric phases manifest across different layers? Understanding the spatial distribution of these dynamics could provide deeper mechanistic insights. For example, as mentioned before, the recent work \"Layer by Layer: Uncovering Hidden Representations in Language Models\" (arxiv:2502.02013) also uses information-theoretic and geometric measures (including effective rank) to analyze LLM representations, but focuses on spatial analysis across layers rather than temporal evolution during training. Could the authors discuss the similarities and differences between their temporal geometric phase framework and this spatial layer-wise analysis?\n\n6. **Intervention Studies**: Can the geometric insights be used to design more effective training strategies? For example, could learning rate schedules be adapted based on geometric phase transitions to optimize the memorization-generalization trade-off? (similar to \"Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning\" (arxiv:2411.02344) which applied matrix entropy to increase reasoning perforemce)", "limitations": "The authors adequately address limitations in Section 5, acknowledging that their findings may be specific to the investigated model families and datasets. However, they could more explicitly discuss the limitations of their spectral analysis approach and the potential for other geometric measures to reveal different patterns. The correlational nature of the evidence and the need for stronger causal analysis should also be emphasized more prominently. Additionally, focusing on last-token representations may overlook important dynamics occurring in intermediate layers or attention mechanisms.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**Novel Analytical Framework**: The application of spectral geometry to understand LLM training dynamics represents a significant methodological contribution. Moving beyond loss curves to examine representation structure provides valuable insights into the learning process that are not captured by traditional optimization metrics.\n\n**Comprehensive Empirical Validation**: The authors demonstrate consistency across multiple model families (OLMo, Pythia, Tülu-3.1) and scales (1B-13B parameters). \n\n**Mechanistic Understanding**: The connection between geometric phases and specific capabilities is well-supported. The correlation between entropy-seeking phases and n-gram memorization, followed by compression-seeking phases and improved long-context reasoning, provides actionable insights for understanding when different capabilities emerge during training.\n\n**Theoretical Grounding**: The toy model analysis in Section 3.3 effectively demonstrates how cross-entropy optimization under information bottleneck conditions can reproduce the observed multiphase dynamics. The identification of primacy bias and selection bias as key mechanisms provides a theoretical foundation for the empirical observations.\n\n\n## Weaknesses\n\n**Limited Theoretical Rigor**: While the toy models provide intuitive explanations, the formal connection between these simplified settings and actual LLM training remains tenuous. The paper would benefit from more rigorous theoretical analysis establishing when and why these phases must occur in realistic training scenarios.\n\n**Narrow Evaluation of Memorization**: The distributional memorization metric based on n-gram alignment, while useful, captures only one aspect of memorization. The paper's conclusions about memorization vs. generalization would be stronger with additional memorization measures and more diverse downstream tasks.\n\n**Causality vs. Correlation**: The paper establishes strong correlations between geometric phases and capabilities but does not demonstrate causality. It remains unclear whether geometric changes drive capability emergence or vice versa, limiting the actionability of the insights.\n\n**Post-training Analysis Depth**: The post-training analysis, while interesting, lacks the depth and mechanistic understanding provided for pretraining. The connections between different alignment objectives and geometric transformations could be explored more thoroughly.\n\n**Methodological Limitations**: The focus on last-token representations may miss important dynamics in other parts of the networks (For example, analyze the intermediate layers as in \"Layer by Layer: Uncovering Hidden Representations in Language Models\" (arxiv:2502.02013)\"", "was_revised": true, "final_justification": "They addressed most of my concerns. I still have concerns about the evaluation of memorization and the limited theoretical contributions", "rebuttal": "We thank the reviewer for their encouraging comments and helpful suggestions, and we are glad that they found our results to be novel, well-supported and theoretically grounded. Below, we address all the concerns and questions raised by the reviewer.\n\n\n### Mechanistic Validation\n\nThis is a great question, and forms an important next step to our current work. While we have deferred investigating causal relationships between eigenspectrum of representations and model behavior to future work, we believe that our analytical results shed light into potential causal interventions that can be designed. Specifically, we demonstrate that the high-frequency token information is learned earlier in training and mainly represented in the top eigendirections. Consequently, removing these directions would have minimal impact on downstream question-answering task performance. On the other hand, keeping the top eigendirections and removing the tail of eigenspectrum would imply preserving most of the information about high-frequency tokens and removing fine-grained information about the dataset, thereby hurting downstream performance. While the concepts of memorization and generalization are loosely connected to this information segregation between the top eigendirections and the tail of eigenspectrum, given the time constraints of the NeurIPS review process, we defer a more rigorous analysis to future work. Instead, we propose to add preliminary results demonstrating the information segregation and the differential role of the top eigendirections and the eigenspectrum tail on the task performance in the Appendix, to demonstrate the potential causal links between eigenspectrum and behavior. We thank the reviewer for this suggestion.\n\n### Robustness analysis\n\nWhile we agree with the reviewer's suggestion that it would help to understand the impact of hyperparameter choices, our compute budget limitations prevented us from pretraining large-scale models ourselves. Instead, we used open-source models and checkpoints, demonstrating consistent non-monotonic spectral evolution across Pythia and OLMo suites with varying architectures and scales. Our results are limited to existing hyperparameter configurations of these model families. However, we believe these results encourage future work on understanding the impact of design choices on the multi-phase learning dynamic.\n\nFor smaller-scale models, we have preliminary results where we trained GPT-2 models from scratch on Fineweb and observed the emergence of multi-phase learning dynamics across various \"good\" hyperparameter configurations (monotonic loss decrease, reasonable downstream performance). We will include an Appendix figure illustrating these results. Taken together, we believe that we can broadly say that these phases are present in well-performing LLMs trained with next-token prediction and cross-entropy loss, deferring further investigation into the impact of hyperparameters to future work. \n\n\n### Alternate metrics\n\nWhile the reviewer is correct in pointing out the existence of various metrics for quantifying representation geometry, we focused on two that rely on linear decomposition. Our choice was motivated by the direct relationship between linear decomposition of the representation space and learning dynamics under gradient descent. This connection allows us to link our findings to empirically observed model behaviors and provide a well-grounded theoretical explanation.\n\nIn contrast, metrics like intrinsic dimensionality identify a non-linear transformation of the representation space that explains most of its variance. While useful for understanding the underlying representation manifold, these metrics are not directly connected to gradient descent properties, making it harder to draw concrete conclusions about model behavior.\n\nFurthermore, recent work by Skean et al. [1] (the paper recommended by the reviewer) demonstrates theoretical connections and strong empirical correspondence between several information-theoretic, geometric and spectral metrics in LLMs. Therefore, we believe our results, combined with those of Skean et al., offer a more comprehensive view of representation geometry evolution in LLMs across pretraining and post-training stages. We will add this point in the discussion section of our paper, along with the citation to the recent work from Skean et al.\n\n\n\n### Layer-wise analysis\n\nWe once again thank the reviewer for highlighting the recent work from Skean et al. on layer-wise representation geometry. This interesting work offers a complementary perspective on model behavior, and we will cite it and discuss its connection to our own in the updated discussion section of our paper. Notably, our analysis shows similar trends in intermediate layer representations, with the last layer's temporal evolution often serving as a leading indicator. Specifically, most intermediate layers exhibit similar phases, but later in pretraining. We will add these results in the Appendix.\n\nWhile Skean et al. examined similarity across model architectures and modalities from an information processing viewpoint, our work concretely links representation geometry metrics to downstream model behavior and provides a foray into potential mechanisms underlying the representation geometry changes during pretraining and post-training stages. Thus, our findings offer a complementary view of how the training strategies lead to representational changes in LLMs and how those lead to emergent behaviors.\n\n\n### Intervention studies\n\nWe thank the reviewer for this insightful suggestion and pointing to this work by Arefin et al. We agree with the reviewer that our findings on multiple phases could lead to spectral interventions and regularization strategies for accelerating training and improving LLM generalization. We hypothesize that some of this regularization might be implicit in current objective functions, such as the equivalence between the DPO objective and contrastive loss that reveals an implicit entropy-seeking prior. The interplay between these implicit biases and explicit regularization warrants further study, which we believe is beyond the scope of this paper and leave for future work. However, we agree with the reviewer that mentioning this potential impact of our results will strengthen our paper, and we will add this point and a citation to Arefin et al. in the discussion section. \n\n---\nWe thank the reviewer once again for their overall positive rating and are grateful for their helpful suggestions. We believe that incorporating our results from preliminary experiments altering the eigenspectrum and GPT-2 training in the appendix as well as discussing the recent works from Skean et al. and Arefin et al. will strengthen our paper and improve its impact. We hope that the reviewer finds our responses to be helpful, and are happy to clarify any further questions during the discussion phase.\n\n**References:**\n1. Skean, O., Arefin, M.R., Zhao, D., Patel, N.N., Naghiyev, J., LeCun, Y. and Shwartz-Ziv, R., 2025. Layer by Layer: Uncovering Hidden Representations in Language Models. In Forty-second International Conference on Machine Learning."}, {"review_id": "VNN8YOAklh", "reviewer": "Reviewer_nZGd", "summary": "The work examines the geometrical changes that occur during the training of state-of-the-art language models and their relationship to the capabilities acquired during training. It studies both pre-training checkpoints of OLMo and Pythia, as well as post-training checkpoints of the Tulu3 suite. It utilizes matrix entropy to characterize the geometry, specifically focusing on the last token representation of the model's final layer.  It shows that, in the course of pretraining, an expansion phase where the model memorizes n-gram statistics from the training corpus is followed by a compression phase, during which the model improves its performance in tasks like TriviaQA. SFT corresponds to an expansion phase where patterns from specific instructions are memorized at the expense of OOD generalization, and a compression in RLVR is attributed to the constraints imposed by RLVR on the exploratory capabilities of base models.", "questions": "Questions:\n\nThe authors omit some experimental details on how they preprocess the data to measure the entropy. For instance, the context length used in their experiments is not reported, nor is the impact of the sentence length on the measured entropy. \nWhat is the impact of the sentence length on the measured entropy?\n\nIn Sec. 3.3 the authors should describe the dataset they use in more detail: is it text? Is the task language modeling? What are the classes they are talking about? What are their frequencies?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strenghts\n\nThe work examines state-of-the-art models, offering an insightful analysis of the dynamics of the internal geometry of the representations, and sheds light on the connection between changes in the internal geometry of the last layer and the capabilities acquired during training. \nThe work is well written and easy to follow. \n\nWeaknesses\n\nBy design, the work focuses on the last token at the last layer. This is fine, but the authors should clearly state, also in the discussion section and as a potential limitation of their work, that the compression-seeking and entropy-seeking phases are specific to the last-layer representations and not a general property of LLMs as a whole. \nIn the literature [1-3], there is evidence that the behavior of the geometry of the hidden representation is heavily layer-dependent. When the representation of the last layer expands, other internal representations may contract (and vice versa). \nA discussion of this point should be added.  \n\nThe plots are sometimes difficult to read. For instance, Fig.3 contains too many panels; three model sizes are enough, and more importantly, the colors of the profiles do not help the readability: with a double y-axis, colors should be used to distinguish the different quantities, such as SciQ and TriviaQA in the second line.\n\n[1] Valeriani et al. The geometry of hidden representations of large transformer models.\n[2] Ansuini et al. Intrinsic dimension of data representations in deep neural networks.\n[3] Cheng et al. Emergence of a High-Dimensional Abstraction Phase in Language Transformers.", "was_revised": true, "final_justification": "The authors addressed my concerns positively during the rebuttal phase (see the discussion with them below).\n\nI consider the value of this contribution sufficient for acceptance to the NeurIPS conference.", "rebuttal": "We thank the reviewer for their helpful comments and suggestions, and we are glad that they found our paper to be well written and the results to be insightful. Below, we address all the concerns and questions raised by the reviewer.\n\n###  Intermediate layer representations\nWe agree with the reviewer that our work primarily focuses on the last token of the last layer, and thus, the identified phases are defined specifically based on the representation geometry changes in this representation space. The intermediate layer representations, as pointed out by the reviewer, do not necessarily exhibit the same trends. However, our analysis reveals similar trends in several intermediate layer representations, with the temporal evolution of last layer representations often serving as a leading indicator. This means intermediate layers undergo similar phases, but later in the pretraining stage. We will add these results into the Appendix and clarify the scope of our results in the Discussions section of the final paper.\n\nWe defined our focus to be such because, as mentioned in Section 2.1, the last token in the last layer is critical for autoregressive models as: (i) it directly parameterizes the predictive distribution for subsequent tokens, (ii) it synthesizes information from the entire input context, reflecting the model’s contextual understanding, and (iii) it is frequently used as input for task-specific layers in downstream applications. This pivotal role allows us to directly relate spectral metrics to model behavior. But, that is not to say that the intermediate layers are unimportant, and for this reason, we will include those additional results in the updated Appendix.\n\n### Spectral measure computation details\nWe apologize for the lack of clarity in the methodology. We used 15k sequences from the Fineweb dataset and extracted the last layer representations after processing sequences of 512 tokens. We will add these details in Section 2.1 in the final version of our paper. The impact of sequence length and dataset on the spectral metrics is presented in Fig 7 and Fig 8 respectively. We will add references to these figures in Section 2.1 as well. We thank the reviewer for pointing this out.\n\n### Section 3.3 experiment details\nWe appreciate the reviewer's suggestion to add more detail about the analytically tractable experiment, as it will improve the paper's readability. In the experimental setup for Fig 4, our goal is to validate our theoretical findings regarding the role of cross-entropy loss, so we abstract away the complexities of natural language modeling. We define $f_{\\theta}(.)$ to be an embedding model that receives inputs $x^{(i)}$'s, which are orthogonal vectors. Each $x^{(i)}$ is randomly assigned to one of four classes (represented by triangle, circle, square, and diamond in Fig 4A). The frequency distribution is skewed: two $x^{(i)}$’s belong to the triangle class, two to the circle class, and one each to the square and diamond classes (as shown in Fig 4A). For the control experiment in Fig 10(a), we use a uniform class distribution, with two samples per class.\n\n### Fig 3 plots\nWe apologize for the lack of clarity in the plots. We will reduce the number of panels in Fig 3 and make the necessary changes as suggested by the reviewer.\n\n---\n\nWe thank the reviewer once again for their comments and are grateful for their helpful suggestions. We believe that adding the results on representation geometry changes in intermediate representations in the appendix and clarifying the scope of our results in the discussions will strengthen our paper. We hope that the reviewer finds our responses to be helpful, and are happy to clarify any further questions during the discussion phase."}, {"review_id": "uLVwyPb90Z", "reviewer": "Reviewer_YXFK", "summary": "**Summary**  \nThis paper introduces a **spectral analysis framework** to trace the evolution of representation geometry in LLMs across pretraining and post-training stages. Key contributions include:  \n1. **Non-monotonic pretraining phases**: Identifies three distinct geometric phases—*warmup* (rapid compression), *entropy-seeking* (manifold expansion correlating with n-gram memorization), and *compression-seeking* (anisotropic consolidation enabling generalization).  \n2. **Post-training dynamics**: Links SFT/DPO to *entropy-seeking* (improves in-distribution performance but reduces robustness) and RLVR to *compression-seeking* (enhances reward alignment but limits exploration).  \n3. Uses spectral metrics (RankMe, αReQ) to quantify representation geometry and connects findings to optimization dynamics and data skew.", "questions": "1. Would perturbing specific eigenvectors during training make the model reach flat minima, which is good for out-of-distribution generalization? \n2. Would perturbing specific eigenvectors during training alter phase transitions or downstream performance?", "limitations": "- **Dataset/model constraints**: Findings based on OLMo/Pythia families; generalization to other architectures and data remain untested.  \n- RankMe/αReQ may not capture fine-grained geometric shifts in early layers or task-specific subspaces.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \n- **Novel framework**: Characterize LLM training via spectral geometry, revealing non-obvious phase transitions masked by loss curves.  \n- **Rigorous validation**: Demonstrates consistent phases across model families (OLMo, Pythia) and scales.  \n- **Practical insights**: Connects geometry shifts to capabilities (e.g., entropy-seeking → memorization; compression-seeking → generalization).  \n- **Methodological clarity**: Well-defined metrics (RankMe, αReQ) and ablation studies (e.g., Figure 4 validates phase drivers).\n\n**Weaknesses**  \n- **Correlation vs. causation**: Observed geometry-behavior links are associative; causal mechanisms require further study.  \n- **Narrow task scope**: Experiments limited to n-gram memorization and simple QA; complex reasoning tasks unexplored.  \n- **Incomplete phase explanation**: Theoretical toy models (Section 3.3) simplify real LLM dynamics.", "was_revised": true, "final_justification": "The responses did not fully address the specific aspects I was curious about — which may indeed fall outside the main scope of the paper. I also acknowledge that the use of small-scale and limited instances of RLVR offer limited support for the claim that “RLVR **often** exhibits a ‘compression-seeking’ dynamic.” I will maintain my current positive score.", "rebuttal": "We thank the reviewer for their encouraging comments and suggestions, and we are glad that they found our results to be novel, rigorous and clearly presented. Below, we address all the concerns and questions raised by the reviewer.\n\n### Results on complex reasoning tasks\nWe thank the reviewer for this suggestion. We have now conducted downstream task evaluations for checkpoints from intermediate pretraining stages on Lambada_openai, MMLU, and HellaSwag, and will add them in the appendix of our final paper. It is important to note that models solely undergoing pretraining often struggle with complex reasoning tasks, requiring specific post-training methods to achieve high performance. Therefore, their performance on such tasks is not a reliable indicator of their underlying capabilities. Consequently, Section 3.4, which presents results from post-training stages, focuses on evaluating model performance on complex tasks like chat (AlpacaEval) for SFT models and mathematical reasoning (AMC-23) for RLVR models.\n\n### Utility of RankMe and $\\alpha$-ReQ for intermediate layer representations\nWhile we focus on and present the results for the final layer representations, the spectral metrics that are used to quantify the representation geometry are generic and do not assume anything about the origin of the representations. Therefore, they can be readily applied to study changes in representation geometry in intermediate layers. To further illustrate this point, we performed experiments on the intermediate layer representations and observed similar trends. We will add these results in order to clarify the utility of the proposed metrics in the Appendix in the final version of our paper.\n\n### Perturbing eigenvectors for better out-of-distribution generalization\nThe relationship between better out-of-distribution generalization and flat minima implies a particular conditioning of the Hessian of the loss with respect to the model parameters. While the connection between this Hessian matrix and the eigenspectrum of the final layer representations is not immediately obvious, it becomes clear in cases where only the final readout weight matrix is adjusted for a new task. In such cases, perturbing specific eigenvectors can improve the optimization problem and, consequently, out-of-distribution generalization. However, more theoretical and empirical work is needed to fully understand the relationship between the geometry of final layer representations and a model's out-of-distribution performance after pretraining. We hope our proposed metrics and observed trends can inspire future strategies and regularization techniques to enhance out-of-distribution generalization.\n\n### Perturbing eigenvectors to alter downstream performance\nThis is an interesting question, highlighting the implications of our findings for mechanistic interpretability. While a full investigation of the causal relationship between the eigenspectrum of representations and model performance is beyond the scope of this work, we believe that our analytical results point to potential causal interventions. Specifically, we show that high-frequency token information is learned early in training and is encoded in the top eigendirections. Consequently, removing these directions would have minimal impact on downstream question-answering task performance. Conversely, preserving the top eigendirections while removing the tail of the eigenspectrum would retain most high-frequency token information but discard fine-grained dataset details, thereby negatively affecting downstream performance.\n\nGiven the time constraints of the NeurIPS review process, we will defer a more rigorous analysis of how downstream task performance connects to this information segregation between the top eigendirections and the eigenspectrum tail to future work. Instead, we will add these preliminary results in the Appendix. These results will demonstrate the information segregation and the differential roles of the top eigendirections and the eigenspectrum tail on specific question-answering tasks, illustrating potential causal links between the eigenspectrum and the model's downstream behavior.\n\n---\nWe thank the reviewer once again for their overall positive review and are grateful for their helpful suggestions. We believe that adding the results on more diverse tasks during different phases in the pretraining stage, trends in representation geometry changes in intermediate representations and preliminary experiments demonstrating the effect of ablating the eigenspectrum tail on downstream task performance in the appendix will strengthen our paper and increase its impact. We hope that the reviewer finds our responses to be helpful, and are happy to clarify any further questions during the discussion phase."}]

Author rebuttal:

Meta-review: This paper studies the training dynamics geometry of last-token representations in LLMs. The authors use existing metrics that reflect the representation covariance’s geometry, such as the effective rank measure (which is the entropy of the covariance eigenvalues) and power-law decay rate of the eigenvalues. The contribution of this paper is to use the geometric metrics during training to explain the evolution of the last-token representations and to connect it to memorization and generalization. The authors study both the pretraining and the post-training of LLMs.

The authors reveal three geometry-related phases in the pretraining: a warmup phase, an entropy-seeking phase that relates to memorization, and a compression-seeking phase that relates to generalization. In post training, the authors remarkably show that supervised fine-tuning (SFT) and direct preference optimization (DPO) show entropy-seeking dynamics, whereas reinforcement learning with verifiable rewards (RLVR) shows compression-seeking dynamics. While this paper is mainly an empirical analysis of modern LLMs, the authors also provide complementary analytical insights into the training dynamics of a toy model.

The contributions of this work to the research of LLM training dynamics are appreciated. The author rebuttal adequately addressed many of the review comments, as listed in the summary of changes that the authors provided at the end of the rebuttal.

Yet, the AC-Reviewer discussion indicated that some of the author rebuttal to Reviewer 61qF’s concerns was insufficient. Therefore, the following additions that were proposed by the authors should not be added to the final version of the paper: Correlation between spectral metrics and performance (change 2 in the authors’ summary of changes), this includes the experiments from the author rebuttal to Reviewer 61qF that were provided under the section titles of statistical analysis and statistical testing. Besides that, the author rebuttal was adequate, and the authors are requested to implement the rest of the changes listed in their summary of changes.

The set of reviews and all the information throughout the review process (including the rebuttal and discussions) reflect that the contributions of this paper deserve publication and, therefore, my recommendation is to accept this paper.
Decision: Accept (poster)