Grokking in LLM Pretraining? Monitor Memorization-to-Generalization without Test
Authors: Ziyue Li
Chenrui Fan
Tianyi Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Generalization
Large Language Models
Primary area: interpretability and explainable AI

Abstract: This paper presents *the first study of grokking in practical LLM pretraining*. Specifically, we investigate when an LLM memorizes the training data, when its generalization on downstream tasks starts to improve, and what happens if there is a lag between the two. Unlike existing works studying when a small model generalizes to limited and specified tasks during thousands epochs' training on algorithmic data, we focus on a practical setting for LLMs, i.e., near single-pass pretraining of next-token prediction on a cross-domain, large-scale corpus, and generalization on diverse benchmark tasks covering math/commonsense reasoning, code generation, and domain-specific retrieval. Our study, *for the first time, verifies that grokking still emerges in pretraining mixture-of-experts (MoE) LLMs*, though different local data groups may enter their grokking stages asynchronously due to the heterogeneity of their distributions and attributions to others. To find a mechanistic interpretation of this local grokking, we investigate the dynamics of training data's pathways (i.e., expert choices across layers in MoE). Our primary discovery is that *the pathways evolve from random, non-smooth across layers, instance-specific to more structured and transferable across samples*, despite the converged pretraining loss. This depicts a transition from memorization to generalization. Two novel metrics are developed to quantify these patterns: one computes the pathway similarity between samples, while the other measures the consistency of aggregated experts between subsequent layers for each sample. These training data based metrics induce near zero cost but can faithfully track and monitor the generalization of LLMs on downstream tasks, reducing reliance on costly instruction tuning and benchmark evaluations. We also ground our findings in a theoretical analysis of one-layer MoE, showing that more structured pathways improve the generalization bound.

Published as a conference paper at ICLR 2026
GROKKING IN PRETRAINING? MONITOR LLM's EEENORIATTOONTTOGGEEERAAZZZZZZZ
WITHOUT TEST
Ziyue Li, Chenrui Fan
Department of Computer Science, University of Maryland, College Park 1itzy61 96 umd edu, cfan420umd. edu,
Tianyi Zhou
MBZUAI tianyi.. www.@mzzui......o..
ABSTRACT
This paper presents the first study of grokking in practical L.M pretraining. Specif- ically, we investigate when an LLM memorizes the training data, when its gener- alization downstream tasks starts to improve, and what happens if there is lag between the two. Unlike existing works studying when a small model generalizes to limited and specified tasks during thousands epochs' training on algorithmic data, we focus on a practical setting for LLMs, i.e., near single-pas poggle-pas pretraining next-token prediction diverse benchmark tasks covering math/commonsense reasoning, code generation, and domain-specifhe retrieval. Our study. for the first time, verifies that grokking still emerges in pretraining ciixture-of-expee groups may enter their grokking stages asynchronously due to the heterogeneity of their distributions and attributions to others. To find a mechanistic interpretation of this local grokking, we investigate the dynamics of training data's pathways (i.e., expert choices across layers in MoE). Our primary discovery is that the path- ways evolve from random, non-smooth across layers, instunce-speccfic to more structured and transferable across samples, despite the converged pretraining loss. This depicts a transition from memorization to generalization. Two novel metrics are developed to quantify these patterns: one computes the pathway similarity between samples, while the other measures the consistency of aggregated experts between subsequent layers for each sample. These training data based metries in- duce near zero cost but can faithfully track and monitor the generalization of LLMs downstream tasks, reducing reliance on costly instruction tuning and benchmark evaluations. We also ground our findings in theoretical analysis of one-layer MoE, showing that more structured pathways improve the generalization bound.
cross- doomsn, large-scale corpus, and generalization
(MoE) LLMs, though different local data
INTRODUCTION
Large language models (LLMs) with Transformer (Vaswani et al. 2017) architectures have demon- strated that pretraining for a general-purpose task such as autoregressive prediction on a large corpus can gain powerful generalization across diverse downstream tasks and domains (Brown al. 2020
Touvron et al. 2023 Bai et al. 2023 Zhao et al. 2025). Capabilities such as prompting (Wei et al. 2023), reasoning (Huang & Chang 2023] Ahn al. 2024). , and in-context learning also emerge during training. However, a counter-intuitive "grokking" (Power et al. 2022: Nanda et al., 2023 Tan Huang 2024 Lx et al.) 2025) has been widely observed in training Transtormer models: the gener- alization starts improve even sharply rises long after training loss converged plateaued- indicator of overfitting in conventional machine learning. While grokking indicates a transition from memorization to generalization (Huang al. 2024). it challenges our fundamental understanding of training dynamics and the underlying mechanism of generalization.
Despite recent attempts to explain the generalization induced by grokking, they have been largely confined to small models trained for hundreds to thousands of epochs on algorithm-syntheeet small-scale data with generalization to limited and highly specified tasks (Power al. 2022 Liu

---

Published as a conference paper at ICLR 2026
et al., 2022a; Nanda et al.
2023 Merrill et al. 2023 Humayun et al. 2024 DeMoss et al. 2024
Prieto et al. 2025). Instead, LLMs' pre-training often takes only one epoch on a web-scale corpus of heterogeneous, cross- data that usually contains noise, imbalance, and redundancy (Brown et al. 2020 Zhao et al. 2025). Hence, LLMs' learning dynamics can vary drastically across different data, leaving whether grokking still emerges in L.LM pretraining an open problem. It is also unclear whether and when different data are memorized with converged loss, and, whether they are memorized the same time. Moreover, without the repeated replay of the same data and a decrease in loss, the conversion from memorizing training tokens to generalizing on downstream tasks remains mysterious: When does the model start to generalize to different tasks? Is memorization a prerequisite? How does generalization performance change after memorization?
Unfortunately, these questions cannot be answered by the dynamics of pretraining loss since the generalization performance can still vary after the loss plateaued, as observed in previous studies of grokking (Power et al.) 2022] Nanda et al., 2023). For L.LMs, monitor its generalization performance during pretraining, since the model has to be finetuned to gain coruuctionfoollwingg capability before being evaluated on downstream tasks, Hence, developing an siccient-to-oomput metric accurately track the generalization can offer critical insights on better understanding and improving the pretraining of LLMs.
also practically expensive to
Main Contributions In this paper, investigate the training dynamics and grokking in LLM pre- training for a mechanistic interpretation of its emergent generalization on downstream tasks. Our study is conducted on the apen-sourced pretraining checkpoints of 7B mixture-of-expeer
(MoE) LLM, OLMoE (Muennighoff et al. 2024). While our study mainly focuses on OLMoE, its pretraining and evaluation setups are common among other LLMs. For the first time, we verify the existence of local grokking during LLM pretraining. However, unlike the previously observed global and synchronous grokking for most data (Power al. 2022 Merrill et al.) 2023; Nanda et al. 2023). LLM memorizes comains//rouuss of training data at different training steps and takes varying amounts of steps to generalize to related benchmark tasks. The starting time and lasting steps of such local grokking vary across data groups. As a result, generalization performance is usually unstable during the earlier pretraining stages but starts to steadily improve once sufficient data have been memorized. This local difference also reflects data difficulty: the later memorized data group often takes longer to generalize.
Another primary challenge is to explain how the continual pretraining after memorization completed leads to the delayed sharp rise of the generalization performance. To investigate the mechanism of memorization-to generalization orarrrzztttoo--gggeeerrliiaaooon transition, we explore internal states of LLMs track their major changes after loss converges. Specifically, we study how the pathways (the expert choices across layers for training samples) in OLMoo change. We introduce two metrics to measure their complexity how it changes over time: (1) the edit distance between different samples' pathways; and (2) the consistency of selected experts' embedding between consecutive layers for each sample.
Our empirical analysis on four domains reveals prominent trend of the pathway complexity during grokking: the edit distance declines and the consistency increases, despite the converged training loss and more data being learned. This change in pathway complexity reflects a transition to smarter memorization: it keeps discovering more transferable knowledge across samples to encode data more efficiently. It explains why generalization improves with plateaued training loss: the model keeps finding more generalizable and shareable structures to memorize training data. Additionally. we provide a theoretical explanation relating the pathway complexity to a generalization bound of one-layer MoE. Practically, the two metrics show strong correlations with the evaluation results on standard benchmarks after instruction finetuning. Therefore, they provide a zero-cost, finetuning and benchmark evaluation-fre -ree tool to monitor generalization during pretraining, which is critical to LLM developers and practitioners. These discoveries take the first step toward bridging the gaps of understanding grokking and key findings and contributions can be summarized as:
transition in LLM pretraining. Our
Grokking still occurs during the near single-pass pretraining of practical-scale LLMs, but it is local and asynchronous for different data groups, unlike global grokking for all data in previous works.
Grokking's memorization- to- generalization transition can be explained by the dynamics of pathways in MoE: pathway similarity between samples and consistency across layers increase. A theoretical connection is also built between the pathway complexity and a generalization bound.
The two metrics we developed measure pathway complexity provide finetuning and evaluation-fii free tool with almost zero cost to monitor the generalization during LLM pretraining.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Previous studies on grokking have primarily focused on relatively small models and simplistic
tasks. Grokking was first identified by PPower al. (2022). who observed the phenomenon using
two-layer transformer trained on small-scale algorithmic dataset. Subsequent investigations have
extended these findings to shallow transformers and densely connected networks (Nanda et al. al.] [002]] Notsawo Jr et al., 2023] Merrill et al. 2023 Liu et al. [022b] Fan et al. 2024). and CNNs (Humayun al. 2024). Notably, all aforementioned studies employ multi-epoch training paradigms, which differ substantially from the one-pass training approach typical in LLM pretraining. Although Lv ctt al. 2025) examined grokking behavior using a 162M parameter transformer pretrained on 40 billion tokens for a copying task, the scale of their model, dataset, and the simplicity of the task remain distant from practical, real-world scenarios. In contrast, our study is the first to investigate grokking in 7-billion parameter LLM across diverse and practical tasks, including mathematical reasoning, code generation, commonsense understanding, and domain-specific knowledge retrieval.
Several works have focused revealing the mechanisms behind grokking behavior. Liu al. (2022a) demonstrated in a simplified setting that grokking results from structured representations emerging during training. [anda al. (2023) further revealed that grokking can be attributed to the gradual amplification of structured mechanisms encoded within model weights. [Wang et al. (2024) showed that even small Transformers trained from scratch on synthetic reasoning tasks exhibit a grokking-driven emergence of implicit reasoning, where generalization arises only after extended overfitting and internal structure formation. Using a single-layer ReLU network. [Merril] et al. (2023) associated grokking with subnetwork sparsity. Meanwhile, Humayun et al. (2024) observed that decision boundaries become smoother during the grokking phase CNN classification tasks. Beyond interpretability- focused research, Notsawo Jr et al. (2023) identified oscillatory patterns in early training loss as predictors of subsequent grokking phenomena. Distinct from these works, our research provides explanation of grokking within significantly larger LLMs through an interpretative analysis of MoE architectures. Additionally, we introduce practical indicators for monitoring generative behavior throughout LLM pretraining.
3 GROKKING (DELAYED GENERALIZATION IN LLM PRETRAINING
While most existing works study grokking of small models on specified algorithmic tasks, investigat- ing similar phenomena in LLM pretraining poses several new unique challenges, which inherently motivate several designs in our experimental settings.
Training-test evaluation gap: most LLMs are trained for the next token(s) prediction while evaluated on diverse benchmark tasks, making the conventional comparison between training and test loss/accuracy invalid. Hence, we need to adapt the definition of grokking to the LLM setting.
Heterogeneous, web-sourced, cross-domain training data: While previous grokking studies focus on clean training data with the same distribution/format, most LLMs are pretrained on het- erogeneous, cross-domain, web-sourced data that contain redundant, contaminated, imbalanced samples. Hence, their loss may converge at different rates and pose distinct effects on downstream tasks. To address these challenges, we carefully filter and group the data in our analysis.
Generalization to diverse downstream tasks: Previous studies of grokking focus on general- ization to highly-specified and limited tasks, while practical LLMs expected generalize to arbitrary tasks or domains via prompting. Hence, LLMs' generalization behaviors may vary drastically across different downstream tasks/benchmarks and difficulty levels during pretraining.
Near single-pass pretraining: Most LLMs' pretraining only takes approximately one epoch, so the model has been trained only once each sample, and its training loss may even converge before visiting later data. However, such "memorization" of unseen data is not caused by overfitting repeated replay. Instead, it reflects the interpolation composition of leamned data, which might be counted as "generalization" in previous works but differs from the emergent generalization of L.LMss on new tasks.
3

---

Published as a conference paper at ICLR 2026
3.1 STUDY TRAINING DYNAMICS OF LLM PRETRAINING
Memorization is measured by the pretraining objective. Next token prediction (NTP) is the most widely adopted pretraining objective for LLMs. Let the pretraining corpus be Dinin it aims to minimize the negative log-likelihood (NLL) of each token Ft, given previous tokens T4,<j in a text sequence z; (Ii,1....,, 51) drawn from Drain:
(,
log po (o (i,<3))
(I)
Since (0) directly measures how the model memorizes each token in 3i, we will track it in the pretraining process and identify the sample with consistently small and converged loss across checkpoints as memorized. Specifically, for a sample 2; with loss L (82) at pretraining step t, we identify it as memorized since step tž if (0,)) S and [L_(0,) (Or) S 5 for all t 2 t7, where a small threshold, enforces stability over time, and T indexes the final pretraining checkpoint.
In Appendix [.3 We verify that varying & and 6 does not change the post-memorization behavior
analyzed in Section 4
Generalization is measured standard benchmarks after instruction tuning. As foundation models with various emergent capabilities, LLMs are expected to generalize to an open set of unseen tasks through simple prompting/instrutionn Hence, their generalization performance is usually measured across standard benchmarks defined on diverse downstream tasks. To equip pretrained LLM & with the instruction instruction-following capability, an instruction tuning algorithm Air further finetunes 0 on dataset Drr. The generalization performance on benchmark-i i is then measured by
Gi = (vval(Orr: D:), O = Ar Drr),
where D; is the dataset in benchmark-i and Eval, defines the corresponding evaluation metric, which varies across benchmarks for different types of tasks. For example, accuracy is usually used in multi-choice QA tasks while BLEU/ROUGE scores used in summarization and generation tasks. For code generation tasks, the generalization performance is usually measured by unit tests, where a test is considered correct if the generated code can pass the unit tests.
Grokking refers to delayed generalization improvement after memorization is completed. Following the same spirit of previous works on grokking, we mainly focus on the phenomenon that Gi starts to surge long after (0) has converged//llateneee memorization as previous works. However, the test loss adopted by previous grokking analysis is no longer an ideal metric for LLM generalization, since it focuses on token truths while downstream tasks focus on semantic-level matching, and some of them do not provide ground truths (e.g., code generation).
We also use the training loss to track
matching to ground
Experimental Setup We equidistantly sample 10 checkpoints from the OLMoE pretraining trajectory. To evaluate memorization, we probe OLMoE on samples drawn directly from its pretraining corpus Drain ensuring fidelity to the model's original distribution. We evaluate generalization on widely used benchmarks spanning four domains: muth, code, commonsense QA, and domain-specific QA (10k pretraining and 10k test samples per domain; see Appendix A). To exclude contaminated benchmark samples from the generalization analysis, we apply the Min-K%++ membership inference method (Zhang et al., 2024 to identify them. Moreover, we mainly focus on the samples whose model predictions become consistently correct before the pretraining ends, and analyze when this generalization emerges and how long it persists. For downstream tasks, we apply lightweight LoRA- based instruction tuning ArT on DIT equip checkpoints with basic instruction-following capability while preserving pretrained capabilities (Appendix B
3.2 ASYNCHRONOUS MEMORIZATION DELAYED GENERALIZATION. & LOCAL GROKKING
To understand when memorization unfolds during pretraining and how affects generalization, Figure I contrasts the number of memorized training samples (using the criterion defined above) at each pretraining checkpoint with benchmark performance for each domain. It shows that (1) training samples are memorized at different pretraining steps; (2) generalization typically follows memorization with lag: benchmark performance starts to achieve stable gains after sufficient data has been memorized. An interesting discovery is that the lag length varies across domains: generalization
4

---

Published as a conference paper at ICLR 2026
Memorized at (Pretraining Steps)
5k
125k
245k
370k
490k
615k
735k
850k
960k
# Memorrizd Pretraining Samples
Math
Commonsense
Domain-Spedffic
2500
Benchmark Perfomance
2000
0.88
(Cumullative
0066
500
0.2
25
370
400
615
785
855
22
266
270
00
855
720
246
615
72
880
1100
Pretraining Step (k)
Figure I: Memorized training samples at different pretraining steps and the generalization per-
formance on standard benchmarks for each domain. It shows asynchronous memorization different data in each domain and delayed generalization leap after memorizing certain amount of data.
leap math and coding tasks requires memorizing many more samples than commonsense and domain-specific QA tasks.
These observations reveal the complex training dynamics of LLMs caused by training data hetero- geneity and their uneven attributions to different benchmark tasks. implies that the widely studied global grokking- synchronous memorization of most training data and delayed contemporary generalization leap on most test samples-care cannot be found in LLM pretraining. That being said, the memorization of certain data is plausible to impose long- tong, delayed effect the generalization to specific downstream tasks. To investigate such local dynamic patterns, we group data in each domain by their memorization steps and particularly focus the early- meroriee groups, they leave more checkpoints after t to study the subsequent dynamics of generalization. Accordingly, we group samples benchmarks by the training steps since when their predictions become consistently correct. To enable an analysis of how memorization drives downstream generalization performance, we pair each training data group with its most relevant benchmark samples via bipartite graph matching (by Hungarian algorithm (Kuhn I955)), with the training-ter data similarity defined in the embedding space of Sentence' Transformer (Reimers Gurevych, 2019). Additional implementation details of the pairing procedure are provided in Appendix B.1
05k Test@
Pretraing
Test@490k
Pretrain@225k
Testg3655 Pretrain@3370k
Test@885k
Pretraining
Objective
0.75
Memorized
3a5k
370k
0.2
Grokking Phase
Delay
Performence
Downstream
0.00
280
30
88
990
1100
128
246
370
.00
998
738
866
980
1100
126
246
300
816
720
980
1100
266
900
680
736
808
980
1100
Pretraining Step (K)
Figure 2: Pretraining objective (blue) and benchmark performance (red) across four paired
training-test data groups from the math domain. Pretrain @p - Test@q denotes paired training-test group, where Pretrain@y @p is the training subset whose samples reach stable memorization step p. and Test@q is the benchmark subset whose accuracy stabilizes step q. These pairs are formed using embedding-basse local grokking each pair: delayed generalization leap on benchmark tasks after the pretraining objective plateaued. From left to right, more difficult data memorized (converged) later associate with longer delay towards grokking.
Hungarian matching, as detailed in Appendix [.]] We observe
Figure 2 visualizes the memorization and generalization on four groups: all training-tess pairs exhibit a delayed generalization leap on benchmark tasks after the pretraining objective stably converged on the training data. Hence, the training objective cannot monitor or predict the generalization performance. Moreover, curlier-converging groups generalize sooner after loss converged, while later-converged groups associate with substantially longer delays. Notably, this delay scales with data difficulty. which affects not only the number of steps taken to memorize the data but also the duration of memorization generalization transition.
5

---

Published as a conference paper at ICLR 2026
Remaining Question This local grokking indicates that the model's internal states, except its pretraining loss, might track generalization or reflect the transition better. This raises a fundamental question for understanding the emergence of capability in LLM: What internal state changes within LLM signify the emergence of generalization? Investigating this problem can provide critical insights to uncover the mechanisms of memorization-tor annooiiaation--gggeeraaizzzoooo transition and monitor the pretraining progress practice. To this end, we probe LLMs' internal state dynamics to address the challenge.
4 MONITOR MEMNORTZATON-TOOGEEERAAAAZATT TRAINING DYNAMICS OF ROUTING PATHWAYS
TRANSITION BY
As shown in Section 3 generalization in LLMs emerges well after memorization, suggesting that internal reorganization underlies this transition. Such dynamics are difficult to track in dense archi- tectures, where all neurons are simultaneously involved. MoEs simplify this process: by organizing computation into experts, each input activates only subset of experts, forming discrete routing path- way across layers (Figure B(a)). These pathways reveal how computation is allocated and reorganized during pretraining, providing mechanistic view of the
transition.
In this section, we analyze the evolution routing pathway during pretraining and introduce two metrics to quantify these changes (Figure B): (i) the similarity of pathways across different inputs (Section 4.1). and (ii) the consistency of a single input's pathway (Section 4.2). Importantly, all metrics computed on pretraining samples that have already been memorized, i.e., those whose training objective has stabilized at low values beyond specific checkpoints, and trace how their internal routing continue evolve from memorization to generalization. We then assess these metries as robust generalization monitors (Section 4.3 and conclude by establishing a spectral framework connecting routing geometry generalization bounds (Section 4.4
(a) Pathway Similarity
(b) Single Pathway Consistency
MoE
Layer
Layer
Layer
Layer
MLP
MILP
Fmheeding
et
Pi+l
Pathway Syi AAC...C
Routing Embedding
Pathway 52t YBB...A
cas Cas CEv1)
(41 s2) = EditDDistancess
L-1
mUNX; cos(e2, e241) +
Figure 3: Pathway Complexity Metrics to Monitor Grokking. (a) Pathway similarity between samples measured by edit distance on their sequences of expert choices across layers. (b) Pathway consistency quantifies the smoothness of expert transitions between subsequent layers for the same sample by cosine similarity of their weighted expert embeddings.
4.1 PATHWAY DISTANCE AND KNOWLEDGE SHARING BETWEEN TRAINING SAMPLES
Building on prior findings that grokking coincides with the emergence of structured internal mecha- nisms (Liu al. 2022at Nanda et al.) 2023 Wang al. 2024), hypothesize that as the model begins to generalize, transitions from highly individualized. input-specific computation toward more shared, structured computation across related inputs. This shift should manifest as increasing pathway similarity the convergence of routing patterns across similar samples-indicttigg that the model is developing common mechanisms for solving related tasks.
We test this by examining how the similarity of expert pathway evolves during pretraining. Each input's pathway is the ordered sequence of selected experts through the L. MoE layers. Specifically, for input T;, we define its pathway 8; as: s; concat(e(). (8) (i) ....... (1) where ez denotes the ordered list of expert indices at layer L, obtained by ranking experts according to the mean routing weights across all tokens for input ø; To determine which experts are included, we first rank experts based on their routing weight and then select top-ranked experts until their cumulative routing weight

---

Published as a conference paper at ICLR 2026
exceeds predefined threshold. The weight threshold selects the experts that actually drive each input's computation, rather than picking a fixed number of experts. In our main analysis, we use cumulative routing- weiigt threshold of 0.7, and Appendix shows that the observed pathway trends remain consistent when varying this threshold. The selected experts are then concatenated as a comma-separated string (e.g.. >,, 13) which later joined across layers with hyphens (e.g.. "3,1, 5-... 1')
We use the Levenshtein edit distance patth (3;,5; Edit-Distance(;; 8;), which counts the minimum insertions, deletions, or substitutions required to align two pathways, to measure pathway similarity. This metric captures both local deviations in expert choice and global shifts in pathway structure, reflecting differences in selection, length, and ordering. We monitor the average pairwise distance across samples throughout pretraining to track the emergence of shared routing.
Overall pattern. Our analysis, shown in Figure 4 reveals a clear trajectory in pathway dynamics. Early in pretraining, most inputs share nearly identical routes, producing low edit distance Dpath- As the model memorizes individual inputs, pathways diverge and pathh rises. Crucially, Dpath decreases after memorization, which indicates that semantically related inputs then converge into similar routing sequences, signaling the emergence of shared pathways that support generalization.
Layerwise pattern. While prior work (Lo et al. 2025) reports static depth depeneee specialization in trained MoEs, our results show that the memorization-to- trriziattooo---gggeraiiiiiooo transition is also dynamically depth-dependent during pretraining (Figure Shallow layers show the steepest decline in Dath after memorization, indicating rapid convergence to shared pathways. Deeper layers exhibit smaller reduc- tions, and the final layer even shows an increase in pathh post-memorization. These patterns suggest a depth-specific reorganization: early layers consolidate universal representations, while later layers retain or enhance routing flexibility, balancing shared computation with task-specific specialization.
Global Edit Disiance
Edit Daterce memerrized & 125k)
Edit Disiance Inmmmrizeed 2 2454) .... Pretraning Oljective Imemerized @ 125k) Filted trend (alter 2454)
Global trand Global
Fiteed trand lafler 125k!
Pretraning Objective (mamerized 245k) Specific
Math
Commonsence
555
330
530
535
6.0
520
550
530-
540
550
365
Pretraining Step
Figure 4: Pathway edit distance and pretraining objective of two data groups (memorized at 125k and 245k steps) during pretraining across four domains. The global panel (leftmost) fits the overall quadratic trend over all domains, while domain panels fit separate trends after the corresponding convergence step. Despite early training objective plateauing, pathway edit distance continues to decline, indicating a declining complexity of internal memorization.
Layer
10
12
Edit Distance
Training Step
Figure 5: Liyer-wise evolution of pathway edit distance during pretraining for the data group memorized at 245k step. Overall distance decreases from early to later layers, indicating higher complexity in early layers. After pretraining objective convergence, early-layer routing among similar samples simplifies quickly, later layers adapt more gradually, and the final layer diversifies after memorization.
7

---

Published as a conference paper at ICLR 2026
4.2 PATHWAY CONSISTENCY AND COMPLEXITY FOR SINGLE SAMPLES
We next analyze how the routing complexity of individual inputs evolves during pretraining. Our hypothesis is that increasingly streamlined and consistent expert selection across layers reflects the reuse of shared pathways. signaling the emergence of generalizable processing strategies. To capture this, we define single-sample pathway consistency, which measures the smoothness of expert transitions across consecutive layers.
In typical MoE, the router at layer é is single-layer MLP-a linear transformation followed by softmax producing scores over K experts, with weight matrix We E IR xd We define the expert embedding for expert l as v == Wrlk [ 1......... K, which serves an anchor vector compute routing scores. For input vi, its routing embedding is the weighted sum:
(3) =k=1 (5.k) .v.
where Dj (( is the routing weight assigned to expert k. Although experts are independently pa- rameterized. the strong correlations between adjacent layers in deep networks (He et al. 2016) and MoEs (Li & Zhou 2025) suggest that {vē } can be comparable across neighboring layers, supporting structural analysis of routing behavior.
To quantify routing smoothness, define pathway consistency C, the normalized average cosine similarity between consecutive layer embeddings for input 24;:
cos(e 0i,(+1)
C. -- L-
max;
0,,++) +
where c = 10- ensures numerical stability. Higher C; indicates more erratic transitions, while lower values reflect smoother, more consistent routing across layers.
Figure 6 tracks the evolution pathway consistency and pretraining objective across four domains. A key finding is that even after the objective stabilizes, pathway consistency continues to emprove-revealinn ongoing refinement the internal routing of the model. This suggests that smoother, more coherent cross-layer transitions emerge well after memorization, positioning pathway consistency as a valuable lens for understanding generalization beyond what the training objective alone captures.
Global Condistency Global
Rited (memortoed 490k1 4D5k)
Consisincc imemorized
Preirsining Objective Jmeme
Commoniserio
Dom ann-an-peefeee
3.44
Falth Corsipenccy
0.00
1/42
0.12
Tranira Oblectvv
0..0
9:2
0.32
0.23
322
0.28
0.20
339
0.20
Pretraining Step
Figure 6: Pathway consistency training objective on four domains during pretraining. The global panel (leftmost) shows that the path consistency consistently improves over all data, while each domain panel shows the same trend each domain after the training objective converges. This reflects a progressively smoother transition of experts between consecutive layers after memorization.
data groups (memorized 490k: and 615k) from
4.3
PATHWAY COMPLEXITY METRICS AS MONITORS OF GENERALIZATION
0.010, Ro-0.9092
MSE=o. 0156, 8900.8909
0156,
MSE-6.0839, R130
MSE=0.0535 30, R2#0.6291
MSE=0.0957, R30033399
1.0
2.5
0.5
0.0
0.30
0.32
680
630
1.60
1.65
1.70
175
1.6
1.7
2.605 Validation Loss
2.700
Path way Distance (palrwise)
Pathway Consistency
Trining Loss
Training Loss [moving avarage)
(sample-wiss)
Figure 7: Correlation of the two pathway complexity metrics with benchmark performance. Com- pared to the training objective and its moving average, both the pathway edit distance and consistency are highly correlated with the test accuracy, though they are also computed on training data.
8

---

Published as a conference paper at ICLR 2026
Table 1: Correlation between pathway complexity metries and generalization (test accuracy) across four domains, with the comparison to training/validation loss, p-value: p
<
0.1
0.05
P 0.01
0.001
0.1 0.5
0.8
Preferred correlation:
positive
negative
Math
Code
Commonsense
Domain Specific
Metric
Pearson
Spearman
Pearson
Speurman
Pearson
Spearman
Pearson
Spearman
Pathway Distance (pairwise)
-0,9471
-0.9487
-0.9290
-0.9276
-0.9821
-0.9747
-0.9254
-0.9487
Pathway Consistency (sample-wing
0.9246
0,9487
0.9786
0.9856
0,9804
0.9747
0.9917
0.9487
Training Loss
0.0435
00.1108
0.0680
-0.2052
0.6427
0.4104
0.7944
0.9487
Training Loss (moving average)
0.4714
0.3162
0.2481
-0,2052
0.7931
0.6669
0.8705
0,9487
Validation Loss
0.4801
40.1054
0.5466
0.8721
-0.5752
00,6677
40.3399
-0,3162
Given the strong link between routing dynamies and generalization, we ask: To what extent do our pathway metrics predict test performance across domains? To answer this, we compute Pearson and Spearman correlations between downstream performance and two metries across pretraining checkpoints. As described in Section 3.1 we apply a lightweight LoRA-based instruction tuning (rank = 32; see Appendix [B] solely to equip pretrained checkpoints with basic cntruuction-olllowiiii ability for benchmark evaluation. The robustness of this setup is further supported by the LoRA rank ablation in Appendix E.2 Our correlation analysis focuses on checkpoints after domain-level accuracy begins its sustained rise (shown in Figure I. thereby isolating the mechanisms that drive generalization beyond the random-accuracy phase.
Table summarizes the correlation results across four tasks. Pathway consistency (sample-wise) shows exceptionally strong positive correlations with test accuracy-often exceeding 0.97 and highly significant-indicating that coherent, stable routing across layers is key marker of generalization. In contrast, pairwise pathway similarity consistently yields strong negative correlations (around -0.93), suggesting that as generalization improves, inputs tend follow increasingly similar expert routes. By comparison, shown in Figure training and validation loss metrics exhibit weaker and less consistent correlations. Notably, while moving-average training loss shows relatively high correlation in certain domains (c.g., Code, loss is minimized during training, we would expect a negative correlation with test accuracy-nor not the positive one observed here. This mismatch further underscores that conventional pretraining metrics fail to reflect the internal transition from memorization to generalization.
its direction is inconsistent with expectation: since
These findings position pathway metrics robust, domain-general indicators of generalization. Importantly, they rely only on training dynamics wwihouut requiring held-out validation data- makeenmgangggg them especially valuable for real-time monitoring in large-scale resource-constrained settings.
wyithics-without
4.4 THEORETICAL CONNECTIONS & EXPLANATIONS
We connect expert routing patterns with model generalization via the complexity of routing, formalized through the effective dimension of the routing kernel. This effective dimension will serve as the key determinant of MoE generalization.
Concretely, H is the Gram matrix of the routing kernel with entries Hi, P (Z,, ,y)), capturing similarity between inputs induced by both routing overlap and expert functions. The effective dimension is defined as Tr(H* (H' ")) measuring the complexity of the function space induced by routing.
Theorem 4.1 (Generalization Bound). Under NTK regime with K experts, with probability 1 over a samples, where C1, O2 are NTK -TK-dependent constants and X is the regularization parameter:
E S X2
H + XT) y +2 σ TTH + 11) a² log(118)
72
Bias
Variance Noise
This provides lens grokking: routing evolves from random to structured, the effective dimension collapses, triggering improvements in generalization. Intuitively, this collapse corresponds to simplification of routing pathways: the kernel spectrum becomes more concentrated, reducing the hypothesis space and mitigating overfitting. The full theoretical setup, assumptions, and proof detailed in Appendix C while Appendix C.4 empirically confirms that effective dimension is

---

Published as a conference paper at ICLR 2026
tightly aligned with routing edit distance, establishing pathway complexity as key driver of MoE generalization.
DISCUSSION
Why studying OLMoE matters. OLMoE is the only publicly available large-scale MoE LLM with full pretraining checkpoints, providing rare opportunity to study pretraining dynamics directly. Its architecture, including top-k routing, load-balancing loss, Adam W, RMSNorm, and RoPE, follows standard MoE practices and thus reflects typical large-scale designs. While additional pretraining checkpoints are not yet available, recent studies on other MoE models such as DeepSeekMoE (Dai al. 2024) and Qwen3-MoE (Yang al., 2025) show that promoting similar routing pathways across related samples improves generalization through post- post-training (Li et al. (2025c) or test-time adaptation (Li et al., 2025a bb). These findings support our conclusion that pathway similarity is general signal of model generalization and that the mechanisms observed in OLMoB extend to other MoE. architectures.
Ruling out global routing effects. To ensure that the observed decrease in pathway distance is not driven by global factors such as load balancing, we conduct two analyses. First, a cross- domain comparison (Appendix E.4) shows that pathway distances between different domains (e.g.. math x code) fluctuate irregularly without a consistent downward trend, in contrast to the observed group- and domain-specific decreases in Figure confirming that pathway reorganization is localized phenomenon driven by memorization rather than a global routing effect. Second, the correlation between routing entropy and pathway edit distance is weak (r 0.24, ~~6 variance explained; Appendix E.3). indicating that balanced or saturated routing alone does not account for our findings. Together, these results demonstrate that pathway reorganization arises from structured, domain-specific computation rather than global changes in routing behavior.
Consistent Findings on another MoE (nanoMoe) pretraining justify the generality of our study. To further assess the generality of our findings, we train separate MoE model from scratch (nanoMoE; Appendix (..6) to obtain fully observable pretraining dynamics. Despite differences in scale, data, and training setup, the model exhibits the same post-memmrization routing behavior, where pathway distance decreases and pathway consistency increases, and both metrics show strong correlations with downstream benchmark performance. This independent experiment confirms that the pathway-genne generalization relationship is robust property of MoE training dynamics rather than an artifact of OLMoE.
CONCLUSION
This paper provides the first empirical evidence that, in large-scale LLM pretraining, generalization can emerge well after memorization has been achieved. Through an analysis of routing dynamics 7B-parameter MoE model, we propose two novel pathway- based metrics that accurately track generalization without requiring finetuning external validation sets. Our findings reveal a transition from memorization to generalization, marked by increased pathway similarity and consistency, and supported by theoretical connections between routing structure and generalization bounds. These insights offer a foundation for more transparent and reliable monitoring of generalization in large- scale models. Future work will extend these pathway-based metrics beyond MoF, by constructing analogous "virtual pathways" in dense models, moving toward a unified framework for tracking generalization in foundation models.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL httph:t///// httppp::/aaxxiv.../ org/
abs 2oss2400..o 00157
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan. Zheng Yuan, Jianwei Zhang. Xingxuan Zhang. Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL pttps://arxxr org/ abs 220.//3322000 11.339.1660 16609
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https
//arxiv. wtg/ abs 14165
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1rXIvv1803.000 1803.05457 2018.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint 2tfi::4001.00666, 2024.
Branton DeMoss, Silvia Sapora, Jakob Foerster, Nick Hawes, and Ingmar Posner. The complexity abs /2412.09112
dynamics of grokking, 2024.
Simin Fan, Razvan Pascanu, and Martin Jaggi. Deep grokking: Would deep neural networks 19454
generalize better?, 2024. URL htt.tttss:// /ttects://arrii..rrg///e/e
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe. Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey, 2023. URL http:::////xxvxor///./ abs hbtp://abb//222111//
Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Unified view of grokking, double descent and emergent abilities: A perspective from circuits competition, 2024. URL
aittps:/aaze //tp:iv///xvvv.// org/ abs/ 2402.15175
Ahmed Imtiaz Humayun, Randall Balestrieno, and Richard Baraniuk. Deep networks always grok and here is why, 2024. URL https:///a:// org/ 15555
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83 1955.
Zhongyang Li, Ziyue Li, and Tianyi Zhou, C3po: Critical-layer, core-expert, collaborative pathway optimization for test-time expert re-mixing. arXiv preprint arXiv:2504.07964 2025a.
11

---

Published as a conference paper at ICLR 2026
Zhongyang Li. Ziyue Li, and Tianyi Zhou. R2-12: Re-routing in test-time for multimodal mixture-of- experts. arXiv preprint arXiv:2502.20395, 2025b.
Zhongyang Li, Ziyue Li, and Tianyi Zhou. Routing manifold alignment improves generalization of mixture-oun mixture-of-expert of-exts llms. arXiv preprint arXiv:2511.04119 2025c.
Ziyue Li and Tianyi Zhou. Sparser mixture-of ofdateeerss with cross-layer generalization. In Pro- ceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Pupers), pp- 4988-4002, 2025.
Ziming Liu, Ouail Kitouni, Niklas S Nolte, Eric Michaud, Max Tegmark, and Mike Williams. Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems, 35:34651-34666 2022a.
Ziming Liu, Eric Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data.
arXiv preprint arXiv:22/00011 2022h.
Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-experts in large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pp. 4427-4447, 2025.
Ang Lv, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Rui Yan. Language models "grok" to copy. 2025. URL
http://bb//44000 02490
William Merrill, Nikolaos Tsilivis, and Aman Shukla. A tale of two eircuits: Grokking as competition of sparse and dense subnetworks, 2023. URL
wttpaxiwwwiwiaaa///oa/aa/aaaaaap abs//abs////////////// 11873
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint 10Xiv:18090027900 2018.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, Yuling Gu, Shane Arora, Akshita Bhagia, Dustin Schwenk, David Wadden, Alexander Wettig, Binyuan Hui, Tim Dettmers, Douwe Kiela, Ali Farhadi, Noah A. Smith, Pang Wei Koh, Amanpreet Singh, and Hannaneh Hajishirzi. Olmoe: Open mixture-of-experts language models, 2024. URL http://axxvvvo/// org
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
Pascal Notsawo Jr, Hattie Zhou, Mohammad Pezeshki, Irina Rish, Guillaume Dumas, et al. Predicting grokking long before it happens: A look into the loss landscape of models which grok. arXiv preprint arXiv:2306. 13253, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- eralization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02117 2022.
Lucas Prieto, Melih Barsbey, Pedro A. M. Mediano, and Tolga Birdal. Grokking at the edge of numerical stability, 2025.
abs / /ales/2501.0 2501.04697
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese nett-net.orks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL htthh:tt/:/// //ap:ii////axvv.////a www.iiiwrrr...oooooo crq/ http//b//19000
10084
Zhiquan Tan and Weiran Huang. Understanding grokking through a robustness viewpoint, 2024.
http:////wwwooooo/////////////// URL htlpstts::2 //wrxivvvvv.//. v. wttpobr//////////// 06597
12

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https: http//aavvvvooo org/ abs/ 2302. 13971
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. URL stttps://axxiv.ort
pdf/1706. 0370337662 pdf
Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic journey to the edge of generalization. arXiv preprint arXiv:2405.J 15071, 2024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompiing elicits reasoning in large language models, 2023.
URL http:p/s//aaaat.oorrraa///2..... /
11903
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.0955 093388 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint
2019.
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%++ Improved baseline for detecting pre-training data from large language models. arXiv preprint urXiv:2404.02953 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A survey of large language models, 2025. URL https : / arxiiv. org /bs/
2303 18223
13

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
We evaluate OLMoE on four domains used during pretraining: math, code, commonsense, and domain-specing (scientific text). For each domain, we construct three sets:
Pretraining sample set: 10,000 examples drawn from OLMoE's pretraining data.
Test sample set: 10,000 examples from a benchmark dataset in the same domain, following OLMoE's 's categorization where available.
Validation set: a disjoint dataset used solely to compute validation loss, enabling correlation analysis between in-domain learning and test generalization.
For the Comainn-specife case, we filter the scientifie pretraining data using the keyword "Organic Chemistry", aligning the training distribution with the "college.chemistry" benchmark in MMLU.
Table summarizes all dataset sources. All pretraining sets are directly drawn from OLMoE's pretraining corpora; test benchmarks are standard evaluation datasets; validation sets are only used for correlation analysis.
Table 2: Datasets used in our domain-level evaluation. Pretraining samples are from OLMoE corpora, test benchmarks used for evaluation, and validation sets are disjoint corpora used only for loss- accuracy correlation. All datasets accessed via HuggingFace.
Domain
Pretraining Source
Test Dataset
Validation Dataset
Math
open-web-math/ open-web-math b bcgce starcoderdate
hkust-clp/
qwedsacf/ competition math
dart maatt-olo-poll--- pooathhpool-aatb math evalplus/
Code
newfacade/
humenevalplus tau/ Lau/commonseese..gg
CodeDaatee
Commonsense
emozilla/ dolma v1.7 books blester125/ mediaviki-dolme
kowndinya23/
flan2021 commonsense
Domain-
cais/nmlu ("college.chemistty" subset)
chemNL2/
Specific
ciemistty-bbookhhlleee
(filtered "Organic Chemistry")
-merged
B EXPERIMENT DETAILS
Compute Resources. All experiments are conducted on NVIDIA A100 GPUs with 80GB memory, using public OLMoR checkpoints.
Instruction Tuning. The raw OLMoE checkpoints exhibit weak direct evaluation of generalization behavior unreliable-partien particularly in text-based domains such as commonsense reasoning or code completion. To address this, we apply light instruction tuning using LoRA (Hu et al., 2022) to make the pretrained follow prompts in readalee way. This ensures we can evaluate generalization while preserving the original pretraining behavior.
ability, making
To avoid overriding pretraining representations, adopt minimal finetuning using small number of epochs and low-rank parameter updates. Specifically:
Math: Finetune with LoRA for 3 epochs on the same math SFT dataset meta tetta-mathh mathh/ MetaMathoA used by OLMoE.
Code: Finetune for 3 epochs on Huggin ing hogggiinnggaceeet/c/cceeeaaaoapp
Commonsense & Domain-Specific: Finetune for epochs yalmaa/ppaaa-cleeeee
Key configurations of LoRA include:
Target modules: q proj, proj, v proj
LoRA rank: 32
LoRA dropout: 0.1
14

---

Published as a conference paper at ICLR 2026
Learning rate: 5e-5
Batch size: 4
Max sequence length: 2048
Applying Min-K %++ for Training- Test Separation. To maintain a rigorous separation between the pretraining and test datasets, we employ the % membership inference attack (Zhang et al.) 2024), state-of-the-arr reference-free frree method designed to detect whether specific data samples are part of model's training corpus.
Specifically, apply the Min-K%++ method to the test dataset to identify samples that the model is most likely to have encountered during training. Determining an optimal threshold for classification is challenging due to variability in model behaviors and data distributions. To address this, adopt a conservative approach by removing the top 10% of test samples with the highest Min-K +%++ scores, thereby minimizing the risk of training data contamination in the test set.
This strategy ensures that our evaluation metrics accurately reflect the model's generalization capabil- ities on truly unseen data, enhancing the validity of our experimental results.
B.1 CONSTRUCTION OF TRAINING-TEST PAIRED GROUPS.
Here we provide additional details how construct the training-test paired groups used in Figure 2 to analyze the temporal relation between memorization and generalization. The overall procedure is introduced in Section 3.2] here, we summarize the specific grouping and matching setup for clarity and reproducibility.
Training groups. For each pretraining sample, we track its token-level loss across checkpoints and mark the earliest step E when the loss remains below threshold and deviates by less than 6 from its final value for all later checkpoints. Samples sharing the same fद form training group denoted as Pretrain @ f. For (e, 8), 8 is fixed at 0.05 across domains, while is domain-specife and determined based on the average loss scale of each domain. We test in increments of 0.1 to ensure that the cumulative proportion of memorized samples before the final checkpoint remains between 20-25% across domains, balancing sufficient data coverage with stable per-domain scaling. Ablation studies on the robustness of (s, 8) is provided in Appendix E.3
Test groups. Each test sample's ' accuracy trajectory is recorded across checkpoints. A test sample is considered to have generalized at the earliest step 1.t where its accuracy remains consistently correct thereafter, with mean accuracy above 0.8 and at most one error in the following checkpoints. Samples sharing the same of form a test group denoted as Tesr@1#
Embedding- pairing. To align semantically related training and test groups, we compute sentence embeddings for all samples using the TraeenoorTannrrmerr model " and measure pairwise cosine similarity between groups within each domain. The average similarity between every pair of groups forms a similarity matrix, and one-to-one mapping between training and test groups is obtained using the Hungarian algorithm implemented in SciPy, which maximizes overall semantic similarity. The notation Pretrain @p -> Text q thus indicates that the training group memorized at step is semantically aligned with a test group whose performance stabilizes at step q.
C GENERALIZATION BOUND FOR MOE WITH ROUTING KERNEL
C.I MODEL SETUP AND ASSUMPTIONS
Consider a MoE model where the router is fixed after pretraining, and the expert networks are trainable. Let K be the total number of experts in the model.
fu(á,, x)) R be the k-th expert's 's output, with trainable parameters dz.
(x(x) [0, 1] be the k-th routing weight, which is fixed and pre-determined for any input
We assume Ek=1 gk (2) (e.g., from fixed softmax layer pre-computed weights).
15

---

Published as a conference paper at ICLR 2026
Full model: F(0, k=1 (z) where Q "oi,...,(x'. ) is the collection of all trainable expert parameters.
Key Assumptions:
1. NTK Regime for Experts: Expert parameters A = (é1,..., are initialized as Ox (0) (0, 7) for each k, and remain c-close to initialization (Il0(t) 0(0)]] O(1/Vwidth)) enabling linearization via Neural Tangent Kernel (NTK). We assume Φx (0) are independent across different experts k.
2. Fixed Routing:
The routing weights gu (a) are fixed for all inputs x and do not contain any trainable parameters. They satisty max; l19x I 1.
3. Label Noise:
Training labels y (a;) + 6, with e; NO0 o²), independent of z;.
C.2 ROUTING KERNEL DEFINITION
Define the NTK for this MoE configuration. Consistent with the general routing kernel definition in the main text, Oroute (2), x') here is specifically instantiated
aF(((0),x)
OF(6(0), }
r (z, x²) Ex(o))
00
00
K
0f (0), 0Ø,
0f, (f (0),
Σ g وو(رg (a') (s))
0o;
[ 01,(0, (0).z) (((0),())=
Let (2, E,,(0) Then, the Routing Kernel in this fixed-routing scenario is:
be the NTK for the j-th expert network.
Bay
8by
motte (I, = Σ
The structure of (x, x.) in this fixed-routing setup provides crucial insights into how routing in- fluences generalization. It is composed of two primary factors: the gating weight product (x)gg (a'), and the individual expert NTKs KJ, x.), The term gj (x)99 (a') signifies the importance of the j-th expert for both inputs x and x,, highlighting that two inputs are considered "similar" by the kernel if they are significantly routed to the same experts. If inputs and x' routed distinct sets of experts, their contribution to the sum for given will be diminished. Simultaneously. KG, (I, x') captures the functional similarity learned by the j-th expert for inputs æ and x'. Thus, the overall routing kernel measures a compounded similarity: inputs are similar not only if they are guided through similar pathways (weighted by g, (x))g (x')))(x')) but also if the specific experts they pass through exhibit similar functional behavior for those inputs (K f, (x,ax)) This decomposition underscores how fixed routing patterns (reflected in yj (a:) set the stage for expert specialization and ultimately shape the generalization properties of the MoE model.
C.3 MAIN THEOREM
Theorem C.1 (Generalization Bound for MoE Experts). Under Assumptions 1-3, with probability 1 8 over training samples (Ii yi) the excess risk satisfiex:
2
Bzd [F[(0", , f (x))
(H + +11
Bias
a O + AI) +
o² log(1/))
+ C2
Variance Noise
where Hi,- = Oroate (x;, IT)) X O is regularization, and C₁ C2 are constants depending on the properties of (x) and [x)x)) The bound decays O (2) when X @(1).
16

---

Published as a conference paper at ICLR 2026
Proof. Under NTK regime, the model can be linearized around initialization:
F(0,z) + (V,F(0(0). 8(0))
F(( +)) (x) A8
Initial Ourput
where ((x) = V)F(O(0)) and A0 0 8(0). The quadratic error term ||0|] is ((1/witth) and thus negligible.
Define centered labels i/ 1/4 F(O(0), ri). The problem reduces to ridge regression:
min
1
Σ
(xx))
A0 -v.)² 2 + 2 |
2 1=1
with closed-form solution:
A (ΦD
A = T 4
where H = <b T is the kernel matrix with H
② route (2i, xz))
The excess risk decomposes as:
= (x)) (E]f(x)
+ {(f()))) to?
Bias?
Bias Term:
X2 Bias² 2 < n TH 2
C1X²
n y (H' + x
where C absorbs constants related to the magnitudes of gn (z) and expert gradients.
Variance Term: The parameter covariance is Cov(A0*) o² T + -11)-100
(Φ0
Al) 1, For prediction variance:
0² n Tr( (H X1)-1)
Variance E2 (xx) (alo))"ooVAA'"(A)))
Confidence Term: By standard concentration inequalities (e.g., for sub-Gaussian noise),
2 0² 21 se- <e-nt"/(20")
o² log(1/1 (1/5)
TP
Collecting all components and adjusting constants C₁ C2 completes the proof.
C.4 CONNECTING EDIT DISTANCE TO EFFECTIVE DIMENSION
The bound shows that simple (ow-dimensional) routing kernels generalize better. We now empirically test how routing pattern complexity (via edit distance) correlates with this effective dimension det = Tr (H* + -))
Our setup involves lightweight MoE architecture with 16 experts and 100-dimensional input space. Input data are synthetic Gaussian clusters, providing controlled environment to analyze pre-defined routing strategies. For each experimental trial, 200 samples are split into training and validation sets. We train a one-layer MoE model with ReLUU activations where only expert networks are trainable; the router's 's weights are initialized once and then frozen. To explore diverse fixed routing patterns, we perform multiple independent runs, each starting with different random initialization of the router.
Following training, we extract the average edit distance of the pre-determined expert routing paths and the effective dimension. As illustrated in Figure 8 we observe a strong positive correlation between these two quantities. This empirically demonstrates that the structural properties of fixed routing, measured edit distance, meaningfully impact the complexity of the function space learned by the experts, as predicted by des. This aligmment supports edit distance as a valuable diagnostic for MoE generalization.
17

---

Published as a conference paper at ICLR 2026
Effective Dimension (normalized
0.9
0.8
0.7
0.6
Original Data
0.5
Fitted curve
std.
0.60
0.65
0.70
0.75 Edit Distance
0.80
0.85
Figure 8: Correlation of Edit Distance with Effective Dimension. Each point represents independent experimental run where the router is initialized with different random seed and then fixed for expert training. This varying initialization leads to diverse fixed routing patterns. We observe a strong positive correlation. This suggests that the structural diversity of pre-defined routing significantly impacts the complexity of the function space learned by experts.
D ROUTING DYNAMICS ANALYSIS
To investigate whether routing evolution arises primarily from router-layer updates or other weight updates, we analyze convergence speed and stability across parameter types.
Results. Table reports convergence speed (magnitude drop-off over time) and stability (con- while expert weights stabilize most gradually,
sistency of updates). Router weights adapt fastest indicating division of labor: routing decisions are steered by fast-updating routers atop slow-evolving expert features. Attention layers lie in between.
Table 3:
Convergence speed and stability across parameter types. Higher values indicate faster convergence or greater stability.
Parameter Type Convergence Speed
Stability t
Router
0.151 (fastest)
5.63
Expert
0.063 (slowest)
7.32
Attention
0.083
6.01
Layerwise Routing Dynamies. To localize routing evolution, we compute the edit distance of routing assignments across layers. Layers 0-3 exhibit the most post-convergence change, suggesting that early layers remain active in refining pathways.
Metric Computation. We periodically snapshot parameter norms during training. For each parameter type, compute the mean absolute norm change between checkpoints, normalized by average scale. Convergence speed is defined as the drop in change magnitude from early to late training, while stability is the inverse standard deviation of these normalized changes. This simple but robust analysis consistently reveals router-exper asymmetries.
E ANALYSIS
E EFFECT OF ROUTING-WEIGHT THRESHOLD ON PATHWAY DYNAMICS
To examine the robustness of our findings with respect to the routing-weight threshold used for pathway construction, repeat the analysis in Fig. 4 under different cumulative routing-weight
18

---

Published as a conference paper at ICLR 2026
thresholds ranging from 0.5 to 0.8 (interval 0.1). This threshold determines how many experts considered active in each layer when forming the pathway sequence: a higher threshold includes more experts, while lower threshold selects only the most dominant ones.
Threshoid
Threshell
Threshold 0.7
Threchold 0.8
465
673
410
330
128 Datert
224
648
415
140
Pretrasting Step 1K
Poetrennng mg Step (4)
Pretremiigg Step 00
Pretr 1ang Step
Ech. Disarce Imareriazd 9123k
Fibrad trand 123
Ebcarce Inarrortzad 24381 Fitual brend Jahur 2430
Figure 9: Effect of routing-weight threshold pathway edit distance domain-specifie QA tasks. Each panel shows the average pathway edit distance for this domain under a different cumulative routing-weight threshold (0.5-0.8, interval 0.1). Higher thresholds include more active experts, leading to longer absolute distances, but all settings exhibit the same qualitative pattern: pathway edit distance decreases consistently after memorization, confirming that the observed dynamics are robust to the threshold choice.
Figure 9 presents the results for the domain-specific QA tasks. Although the absolute values of edit distance increase with larger thresholds thresholds-reflectting the longer sequences formed when more experts are included-tte qualitative trends remain consistent across all settings. For every threshold, the pathway edit distance decreases after memorization (marked by vertical dashed lines), and the decrease becomes more pronounced at later checkpoints.
These results demonstrate that our conclusions are robust to the choice of routing-weight threshold. The observed reduction in pathway complexity after memorization, corresponding to the emergence of shared pathways, persists across all threshold settings.
E.2 EFFECT OF LORA RANK ON PATHWAYY
CORRELATION
To examine whether the correlation between pathway metrics and generalization depends on the configuration of instruction tuning, we conduct an ablation on the LoRA rank used during lightweight finetuning. As described in Appendix [B] the default setup adopts a rank of 32 for all domains. Here, we additionally train LoRA adapter with rank 64 on the commonsense domain, following the same data, training steps, and hyperparameters as in the main experiments.
Table Table reports the correlation between pathway complexity metrics and downstream test accuracy under the two LoRA ranks. The results show that increasing the rank from 32 to 64 yields nearly identical correlations for both Pathway Distance and Pathway Consistency, while training and validation losses remain weakly correlated. This consistency indicates that the observed relationships between pathway metrics and generalization are robust to LoRA configuration and primarily reflect intrinsic pretraining- phase dynamics rather than artifacts of instruction tuning.
E.3 EFFECT OF MEMORIZATION THRESHOLDS E AND 5
We conduct an ablation study on the two hyperparameters that define the memorization criterion in Section 3.1 the loss threshold and the stability threshold 8. Specifically, for a sample x; considered memorized from step tī if (,))) S E and
with loss (.)) at pretraining step t, (e,()) - ((r) < 5 for all t 2 t,. We evaluate how sensitive the pathway trends are to these two parameters using the Nomain--peciffe QA task.
Varying s. Figure [0] results when fixing d = 0.05 and varying E from 1.5 to 2.0. Across all settings, the pathway edit distance after memorization consistently decreases with training, and the
19

---

Published as a conference paper at ICLR 2026
Table 4: Effect of LoRA Rank on the Correlation Between Pathway Metrics and Generalization (Commonsense reasoning domain). The results show that increasing the LoRA rank from 32 64 yields nearly identical correlations, confirming that the observed trends are robust to LoRA configuration. p-value:
p< u.1,
p< 0.05
0.01
p p> 0.1 p U.5 0.8 Preferred
correlation: positive
negaiive.
LoRA Rank 32
LoRA Rank 64
Metric
Pearson
Spearman
Pearson
Spearman
Pathway Distance (pairwise)
-0,9821
-0.9747
-0.9277
-0.9487
Pathway Consistency (sample-wise)
0.9804
0.9747
0.9559
0.9487
Training Loss
0.6427
0.4104
0.6197
0.4617
Training Loss (moving average)
0.7931
0.6669
0.6871
0.2052
Validation Loss
-0.5752
-0.4617
40.6631
-0.4104
overall trend remains stable indicatinn that only shifts the point where samples are considered memorized but does not affect the cost-memorization dynamics.
Varying 8. Figure reports the results when fixing = 2.0 and varying 8 between 0.03 and 0.05. Similarly, the decreasing trend edit distance after memorization persists, and the fitted trajectories show negligible variation across values.
This confirms that the proposed pathway-baan based observations are robust to the choice of and 6.
effene 5:.3.55
(fand 6.0.01)
flaed 6:d.8))
r-1.A rfand 6:0
6.0.09
2- Uffend 5-3
545-
3
Figure 10: Effect of the loss threshold E on pathway dynamics. The stability threshold is fixed at 8 = 0.05. All curves show consistent mestrieemoizntion decrease in edit distance, indicating that the results are robust E.
d-0.03 (faxed c-2.01
5-0.04 (fixed c=2.01
5-0.05 gluxd c-2.01
540
540
552
Ed8 Diewsco
C48
240
122
520
L'ttorne Imiamssizdd 12381
vord 1336
Dulunnc mmarro and 2433)
Flized trand
Figure 11: Effect of the stability threshold i on pathway dynamics. The loss threshold is fixed at E 2.0. The same decreasing trend holds across different values, confirming robustness to this parameter.
E.4 CROSS-DOMAIN ANALYSIS OF PATHWAY DYNAMICS
To verify that the observed changes in pathway edit distance are due to the emergence of shared pathways within specific domains rather than uniform effects across all data, such load balancing
20

---

Published as a conference paper at ICLR 2026
regularization, conduct additional analysis focusing the cross-domain samples. Using the same cumulative routing-weight threshold (0.7) as in the main experiments, we compute average path- way edit distances for samples drawn from two different domains across brcckpiints-spccifiiiii math x code-unde identical configurations.
Crass-Damain Pothway
E:d1 Distance
km
325
312
263
Checapoint (Step)
Figure 12: Cross-domain pathway edit distance (math code). The figure shows the global trend of average pathway edit distance during pretraining when computed across two different domains. The cross-domain distances fluctuate irregularly, exhibiting both increases and decreases but no systematic downward trend. This contrasts with the localized decreases observed within domains (Fig. 4. indicating that pathway reorganization occurs locally among semantically related data rather than globally across domains.
As shown in Fig. the cross-domain edit distance fluctuates throughout pretraining, showing irreg- ular increases and decreases without any consistent downward trend. In contrast, Fig. demonstrates that within-domaii pathway distances decrease only for groups of samples that become memorized at different points in training, with each domain exhibiting its own localized transition rather than global drop. These observations confirm that structural changes we observe are domain- and group-specifie, reflecting the emergence of shared pathways among semantically related samples rather than a global effect of routing behavior.
E.5 RELATIONSHIP BETWEEN ROUTING ENTROPY AND EDIT DISTANCE
We investigate whether changes in routing diversity could trivially explain the observed trends in pathway edit distance. Specifically, we test whether routing entropy-hipe when expert usage is balanced and low when few experts dominate-
drive edit distance dynamics.
For each sample, routing entropy is computed per layer from the full outing--wightt vector (no thresholding selection) and averaged across layers. For each sample pair, take the mean of the two samples' entropies and compute their pathway edit distance using a cumulative outing-weight threshold of 0.8, which provides sufficient expert coverage for analysis.
We sample 141,530 pairs across checkpoints spanning both the math and code domains, including within-domain and cross-domain pairs. Figure [3 shows weak correlation between average routing entropy and edit distance (r = 0.24, explaining about 6% of the variance) with no clear monotonic or directional trend. This indicates that neither more balanced routing (higher entropy) nor more concentrated routing (lower entropy) systematically reduces pathway distance. Therefore, the decrease within-domain pathway distance observed after memorization unlikely to result from overall changes in routing balance; instead, it reflects domain- and sooup-specific reorganization of computation.
E.6 CONSISTENT FINDINGS ANOTHER MOE (NANOMoE)
To further validate the generality of our findings beyond OLMoE, we conduct additional experiments on 55M-parameter MoE model, MonoMoE!! NanoMoE provides a small, fully trainable MoE architecture for which we can obtain complete pretraining checkpoints, making it suitable for analysis of routing dynamics.
https: thub. com/wo w hawpo/..ooooooeeecceeeeee/ennana fecamerce//anoooy
21

---

Published as a conference paper at ICLR 2026
Edit Distance Entropy Pearson 0.244 N 141530
2500
750
2000
Edit Distance
1500
Pair count
650
1000
500
550
3.75
3,80
3.85
3,90
3.95
4.00
4.05
4.10
Average Routing Entropy
Figure 13: Edit distance routing entropy. Each hexagon aggregates sample pairs (N 141,530) drawn across checkpoints and across the math and code domains. Routing entropy is computed from the full autting-weight vectors (no thresholding); pathway edit distance uses pathways built with cumulative cutoff of 0.8. The weak correlation (r 0.24) indicates that routing balance does not systematically determine pathway similarity.
Training setup. We train nanoMoE 25B tokens of the Weeeeee dataset (Gao et al. 2020) for 50k iterations on two NVIDIA H100 GPUs and uniformly sample ten checkpoints for analysis. Open WebTet general knowledge content. To approximate domain setting, we construct commonsense subset by filtering 10,000 randomly sampled training samples using Qwen3-8B (Yang et al. 2025) to select those semantically related to commonsense reasoning, yielding 7,857 samples in total. Following the same methodology as our main experiments, identify memorized samples based token-level loss stability and compute pathway dynamics for this subset across checkpoints.
mixed-domaie web corpus with heavy emphasis commonsense and
Pethway-metric dynamics. Consistent with large-scale results, observe that after mem- orization, pathway edit distance decreases while pathway consistency increases as shown in Figure 14 and 15 These trends mirror the nost-memorization reorganization we identify in OLMoE, demonstrating that even the smaller MoE model reorganizes routing toward more structured, shared computation after the loss has already converged.
Correlation with downstream performance. We further evaluate all ten checkpoints in a zero-shot setting on standard commonsense reasoning benchmarks (ARC-Easy/ Eae--/aaayl/langgee (Clark et al. 2018
HellaSwag (Zellers et al. 2019). and OpenBookQA (Mihaylov et al. 2018)). Zero-shot eva mation justified because models pretrained on Web including GPT-2 (Radford et al.. 2019). have been shown achieve strong zero-shot performance commonsense benchmarks. As in the large-scale setting, both pathway metrics exhibit strong correlations with benchmark accuracy (rr) z 0.8-1.0), reported in Table substantially outperforming training loss its moving average. This confirms that routing dynamics remain tightly coupled with downstream generalization.
The nanoMoE results reinforce
central claim: posm-eimoii routing reorganization is not an artifact of model scale, data size, or specific MoE implementation. Instead, it reflects an inherent property of MoE training dynamies. Combined with the large-scale OLM study, these
22

---

Published as a conference paper at ICLR 2026
Disance
(memoracd @ 30k!
Distance (memarized @ Rited trens (alter 45k1
Pretraining Objective (memo rized
Global brand
Fited trend (after 30k)
Pretraining Objective (memorized gp 454)
Giobal
Open WebTixt
Edit Distance
23
Aa0pe1000 BUUIR
92
85
Pretraining Step
Figure 14: Pathway edit distance and pretraining objective nanoMoo during pretraining. The left panel reports the overall pathway edit-distance trajectory across checkpoints without grouping samples by their memorization steps. The right panel instead visualizes pathway dynamics for two memorized data groups (memorized at 30k and 45k steps). After their respective memorization points, both groups exhibit consistent post-memorization decline in edit distance despite a plateaued pretraining objective, indicating that nanoMoE continues to reorganize routing pathways toward more structured and shared computation. This mirrors the memorization-to- to-eeeianttotgeeerazzzaaiio transition observed in lcage OLMoE
Path Consiitency
Path Consetoncy eemerfted 3010
Condetency Immemorteed 35K)
Pretraining Objective imemoritee Instraiiing Objective (memorized 35k)
trand (after
bund lafter Open/WebToxt Text
Global
Path Consisanncy
032
230
Protralning
Figure 15: Eyolution of pathway consistency in nanoMoE.. The left panel reports the overall progression of pathway consistency across checkpoints. The right panel further analyzes two memorized data groups (memorized at 30k and 35k steps), showing that once memorization is achieved, each group undergoes continued refinement of layer-to-layer routing transitions.
findings establish pathway metries stable, architecture-iintrisii indicators of the memorization-to- generalization transition.
Table 5: Correlation Between Pathway Metrics and Downstream Accuracy on nanoMoE across four commonsense benchmarks -asy, ChC--hallenge, HellaSwag, OpenBookQA) p- value:
p 0.1
0.05
0.01
0.1
0.5
P 0.8 Preferred correlation:
positive
Degative
ARC-Easy
ARC-Challenge
HellaSwag
OpenBookQAA
Metric
Pearson
Spearman
Pearson
Spearman
Pearson
Spearman
Pearson
Spearman
Puthway Distance (pairwise)
-0.8113
-1.0000
-1.0000
-1.0000
-0.9713
.1,0000
.0.8366
-0.9000
Puthway Consistency (sample-wise)
0.9416
1.0000
0.8818
1,0000
0.9234
0,8000
0.9979
1.0000
Training Loss
0.3301
40.1000
0.7799
00.000
40,2182
0.2000
0.5984
0,8000
Training Loss (moving avg.)
0.4664
0,4000
-0.6078
-0.8000
-0.3580
0.2000
-0.8655
-0,4000
LLM USAGE STATEMENT
In preparing this manuscript, used LLM solely writing assistance tool. Specifically, the model
was employed to aid in grammar correction, sentence refinement, and improving clarity of exposition. did not contribute research ideation, experimental design, data analysis, interpretation of results.
23


Reviews:
[{"review_id": "x38RNAtVIz", "reviewer": "Reviewer_Tf6m", "summary": "This paper studies the behavior of grokking in LLMs by looking at pathways among experts in MoEs. Using OLMoE checkpoints, the authors report high correlations between the proposed metrics and post‑hoc downstream scores. They define a pathway edit distance and a per sample pathway consistency and observe that these metrics correlate with jumps on downstream benchmark scores after lightweight LoRA instruction tuning. They also provide a NTK‑style bound for a one‑layer MoE with fixed routing, arguing that decreased effective dimension aligns with reduced pathway complexity.", "questions": "In Fig. 4, Please clarify whether distances are computed token‑wise or sequence‑wise, and whether you average per‑token pathways or per‑sequence pathways.\n\nAll generalization is measured after LoRA instruction tuning on different datasets. Is it possible that the observed jumps may be influenced by finetuning dynamics, not purely grokking during pretraining?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Previous studies of grokking mostly used small models trained on synthetic algorithmic tasks, but this work examines a 7‑billion‑parameter mixture‑of‑experts (MoE) model (OLMoE) and shows that grokking still appears in one‑epoch pretraining.\n\nThe observation that that grokking in LLMs are local and asynchronous broadens our understanding of grokking at scale.\n\nThe metrics they define; pathway edit distance and a per sample pathway consistency, rely only on pretraining data and internal activations, so they can be computed during training without external validation.", "weaknesses": "The conversion of top‑k experts per layer into comma‑separated strings and computing Levenshtein distance is ad‑hoc, since edit distance is sensitive to sequence length and arbitrary thresholding. This distance can also decrease simply due to stronger load‑balancing or saturated routers.\n\nThe bound assumes fixed routing and an NTK regime for a one‑layer MoE, while in practice OLMoE updates routing and experts jointly across many layers for trillions of tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5fFvhCLM2C", "reviewer": "Reviewer_VRkv", "summary": "This paper examines grokking in practical LLM pretraining. It investigates when memorization occurs, when downstream generalization improves, and whether a lag exists between them in a realistic one-epoch, next-token pre-training setup spanning diverse domains and tasks. \n\nFocusing on mixture-of-experts (MoE) LLMs, the paper shows that grokking still emerges, with local data groups entering the grokking phase asynchronously due to distributional heterogeneity. \nTo explain this “local grokking,” the authors analyze training data pathways (expert selections across layers) and find that these pathways evolve from random, instance-specific patterns to more structured and transferable ones, even after the pre-training loss has converged. \n\nThey introduce two zero-cost metrics, (1) pathway similarity between samples and (2) layer-to-layer consistency of aggregated experts for a sample, that reliably track downstream generalization without instruction tuning or expensive benchmark evaluations. \n\nA theoretical analysis of a one-layer MoE indicates that more structured routing tightens the generalization bound. \nEmpirically, an analysis of a 7B-parameter MoE model confirms that generalization can arise well after memorization.", "questions": "* I find it challenging to understand the details of how the training-test paired datasets are constructed as described around lines 234--239, beginning with “Accordingly, we group ...”.\nCould the authors please clearly describe the detailed procedure for constructing these datasets?\nIn addition, please explain how this relates to the notation such as “Pretrain@370k -> Test@855k” and “Pretraining Objective (memorized @ 125k)” in the figures.\nI believe that an exact understanding of how these datasets are constructed is crucial, as it directly affects the results and conclusions.\n\n\n\n\n* Reproducibility of investigation and analysis:  \nWhile the results and findings of this paper depend heavily on the training-test paired datasets.\nI feel that reproducing these datasets is difficult for readers (other researchers).\nWould the authors consider releasing the logs and code used to generate the datasets, as well as the datasets themselves, to improve reliability and to support future follow-up studies?\n\n---\nI am open to revisiting the overall assessment following discussions with the authors, particularly in light of the concerns and questions outlined above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Zero-cost, pathway-based indicators derived from MoE routing dynamics  \nThe paper proposes two metrics, sample-to-sample pathway similarity and across-layer pathway consistency, that can track the rise of downstream generalization without instruction tuning or benchmark evaluations.\nLLM evaluation is expensive. Leveraging internal routing information to estimate generalization progress directly during pretraining is highly cost-effective.\n\n* Discovery of “local grokking” under data heterogeneity  \nThe paper reports asynchronous grokking across local data groups due to distributional heterogeneity and differing attributions to other groups.\nReal-world corpora are heterogeneous; focusing on local dynamics captures behaviors that global averages miss, informing data curricula and domain-wise monitoring at scale.\n\n* Theoretical support via a one-layer MoE analysis  \nThe analysis indicates that more structured pathways improve the generalization bound.\nProviding a theoretical link between pathway structure and generalization strengthens the validity and potential generality of the empirical findings.", "weaknesses": "* Limited empirical scope (single model)  \nI understand there are no other publicly available MoE checkpoints, but the paper's results and discussion are tailored to the specific OLMoE 7B model.\nIdeally, robustness should be assessed across a broad range of choices in optimization, model design, and training data, such as learning rate schedules, model scales, expert capacity and number, and data mixtures.\nOtherwise, the paper's results and findings may be model-specific or biased by the model used.\nFrom this perspective, the paper's findings and conclusions may be unreliable or not yet conclusive.\n\n* Correlation versus causation remains underexplored     \nAlthough strong correlations between pathway structure and generalisation are evident, intervention studies (e.g., deliberately altering routing) are limited.\nOther factors, such as learning rate, load balancing, and regularisation, could confound this link.\nFor example, we usually downscale the learning rate during pre-training, so the weight change essentially stabilises in the later part of the training phase.\n\n* [Minor] Although this paper examines only a single MoE model, the title \"Grokking in LLM Pretraining?\" suggests that it covers all LLM pretraining cases, and I feel there is a large gap between them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1vHs44d6yD", "reviewer": "Reviewer_TFkw", "summary": "This paper investigates whether grokking-delayed generalization after training loss/accuracy has converged-appears in LLM pretraining. Using public checkpoints of a 7B MoE model (OLMoE), the authors (i observed asynchronous (“local”) grokking across domains (math, code, commonsense, domain QA), (ii introduce two routing-pathway metrics for MoE that are cheap to compute during pretraining and (iii) provide a routing-kernel NTK analysis linking decreased “effective dimension” to improved generalization. Empirically, the two pathway metrics are strongly correlated with downstream benchmark gains (after light LoRA instruction tuning), while conventional training/validation losses poorly track those gains. The paper claims these metrics offer near-zero-cost, evaluation-free monitoring of LLM generalization.", "questions": "1. If you change LoRA rank/targets or remove LoRA, do correlations with pathway metrics persist?\n2. Have the authors considered causal interventions--such as randomly dropping some experts or adding entropy penalties--to verify whether enforcing more structured routing actually causes earlier or stronger generalization (as opposed to merely correlating with it)\n2. How sensitive are the results to the memorization thresholds, the routing cumulative-weight cutoff, and router temperature/noise during logging?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author uses realistic setting like One-epoch, heterogeneous web-scale data, public 7B MoE checkpoints, and diverse domains. \n2. The author provides evidence of asynchronous memorization and delayed generalization, including matched training/test groups and domain-dependent lags.\n3. The author uses public checkpoints/datasets and explicit data-contamination filtering which has good reproducibility.", "weaknesses": "1. The paper mentions “virtual pathways” for dense models as future work; however, the core claim (test-free generalization monitoring) would be much stronger with experiment on a small-scale dense model.\n2. Consider replacing “zero-cost” with “near-zero-cost” or “cheap to compute” to avoid overstatement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JQYOE6Zn0r", "reviewer": "Reviewer_YR9f", "summary": "This paper investigates whether grokking the delayed transition from memorization to generalization observed in small synthetic settings also occurs during MoE pretraining.\nUsing the OLMoE model, the authors analyze routing dynamics across checkpoints and introduce two new metrics, Pathway Similarity and Pathway Consistency, which aim to monitor generalization without external validation data.\nThey find that as generalization improves, inputs increasingly follow similar expert routes, and routing transitions between layers become more coherent. The proposed pathway-based measures strongly correlate with downstream performance, suggesting that they can serve as internal indicators of generalization in MoE models. These findings position routing-based metrics as potential tools for real-time monitoring of generalization in large or resource-constrained training regimes, without the need for held-out validation data.", "questions": "1. Given these routing observations, do the authors see potential for improving MoE training efficiency for instance, by designing better routing or load-balancing strategies informed by pathway dynamics?\nIf so, how might these insights translate into practical optimization or architectural adjustments?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall, the paper is clearly written and easy to follow, with well-motivated experiments and clear presentation of figures.\n\n2. It provides an insightful analysis of routing dynamics in MoE model, OLMoE.\nIt convincingly shows that as generalization improves, inputs increasingly follow similar expert routes, suggesting an internal reorganization of computation during training. This analysis of how MoE models acquire generalization ability could potentially inform better training or routing strategies in future work.", "weaknesses": "1. While the paper presents an interesting and clearly written analysis, its empirical scope and generality remain limited. The entire analysis and validation are restricted to the OLMoE model.\nIt remains unclear whether the proposed metrics hold under different MoE configurations for example, varying the number of experts, the top-k routing parameter, or the load-balancing loss.\nRecent works such as [1,2,3] have explored different routing and balancing objectives to prevent expert collapse or to improve hardware utilization.\nUnder these regimes, it is uncertain whether the proposed pathway metrics would behave consistently or still correlate with generalization.\n\n2. Questionable “one-epoch” claim.\nThe authors state that grokking occurs “during one-epoch pretraining.”\nHowever, according to [4], OLMoE was trained for 1.3 epochs (“We shuffle all samples randomly at the beginning of each epoch and train for a total of 5.133T tokens (1.3 epochs following Muennighoff et al. [121]).”).\nFurthermore, the DCLM-Baseline dataset used for training reportedly contains ~80 % duplicates ([5]), meaning that the model effectively saw many samples multiple times.\n\n3. Line 324 discusses that shallow layers show a steep decline in after memorization, while deeper layers show smaller reductions or even increases. Similar depth-dependent specialization effects have already been discussed in [6].\n\n4. While the authors argue that their metrics could remove the need for held-out validation, this claim seems optimistic.\nBecause the proposed metrics are only defined for MoE models and depend on architecture-specific details such as depth, number of experts, and routing their use as a universal proxy for generalization appears limited.\nIn practice, held-out monitoring may still be necessary for comparability across different settings.\n\n[1] https://arxiv.org/abs/2408.15664\n\n[2] https://arxiv.org/abs/2412.19437\n\n[3] https://arxiv.org/abs/2501.11873\n\n[4] https://arxiv.org/abs/2409.02060\n\n[5] https://huggingface.co/datasets/Zyphra/dclm-dedup\n\n[6] https://arxiv.org/abs/2406.18219", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)