How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
Authors: Kairong Luo
Zhenbo Sun
Haodong Wen
Xinyu Shi
Jiarui Cui
Chenyi Dang
Kaifeng Lyu
Wenguang Chen
Venue: ICLR 2026 Oral (2026)
Keywords: LLM pretraining
Curriculum Learning
Model Weight Average
Primary area: foundation or frontier models, including LLMs
TL;DR: Use model weight average to enhance curriculum learning in LLM pretraining.

Abstract: Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even after sophisticated data curation. A natural approach to better leverage high-quality data is curriculum-based pretraining, where the model is trained on data sorted in ascending order of quality as determined by a quality metric. However, prior studies have reported limited improvements from such curriculum-based pretraining strategies. This work identifies a critical factor constraining these methods: the incompatibility between the ascending data quality order and the decaying learning rate (LR) schedule. We find that while curriculum-based training substantially outperforms random shuffling when using a constant LR, its advantage diminishes under standard LR decay schedules. Our experiments show this incompatibility can be mitigated by two simple strategies: (1) employing a more moderate LR decay schedule, where the final LR is only moderately smaller than the peak LR, and (2) replacing LR decay with model averaging, i.e., computing a weighted average of the final few checkpoints. By combining these strategies, we improve the average score on a suite of standard benchmarks by 1.64% over random shuffling, without additional data refinement. Validated on 1.5B-parameter models trained over 30B tokens with various data-quality metrics, our findings call for a re-evaluation of curriculum-based LLM pretraining and underscore the potential of co-designing data curricula with optimization methods.

Published as a conference paper at ICLR 2026
How LEARNING RATE DECAY WASTES YOUR BEST DATA IN CURRICULUM- PARRICULUM-BASED LLM PRETRAINING
Kairong Luo
Zhenbo Sun
Haodong Wen
Xinyu Shi"
Jiarui Cui" Chenyi Dang'
Kaifeng Lyu't
Wenguang Chen
TTinnhua University Peng Cheng Laboratory
ABSTRACT
Due to the scarcity of high-quality data, large language models (LLMs) are of- ten trained on mixtures of data with varying quality levels, even after sophisti- cated data curation. A natural approach to better leverage high-quality data is coriiuuuunbbssd pretraining, where the model is trained on data sorted in as- cending order of quality as determined by a quality metric. However, prior stud- pretraining
ies have reported limited improvements from such curriculum-base strategies. This work identifies critical factor constraining these methods: the incompatibility between the ascending data quality order and the decaying learn- ing rate (LR) schedule. We find that while curriculum based training substantially outperforms random shuffling when using a constant LR, its advantage diminishes under standard LR decay schedules. Our experiments show this incompatibility can be mitigated by two simple strategies: (1) employing a more moderate L.R decay schedule, where the final LR only moderately smaller than the peak LR, and (2) replacing LR decay with model averaging, i.c., computing a weighted av- erage of the final few checkpoints. By combining these strategies, we improve the average score on a suite of standard benchmarks by 1.64% over random shuf- fling, without additional data refinement. Validated on 1 -ar-aaraeeeer models trained over 30B tokens with various data-quality metrics, our findings call for a re-evaluation of curriculum-based LLM pretraining and underscore the potential of co-designing data curricula with optimization methods.
INTRODUCTION
Large language models (LLMs) are typically trained on massive text corpora collected from the Internet (Dubey al. 2024 DeepSeek-AI et al. 2024] Yang et al. 2025 OpenAl 2023), cover- ing a wide range of sources and quality levels. High-quality data plays crucial role in enhancing model capabilities, but it is usually limited in amount. To address this issue, current LLM pre- training pipelines employ sophisticated data curation procedures to filter out low-quality data and increase the proportion of high-quality data, including rule-based (or heuristic- based) filtering, qual- ity scoring (model-basee labeling), and score-based data selection (Su et al., 20025] Li et al. [222]] Penedo et al. 2025 2023 Weber et al. 2024). Despite these advances, relatively little attention has been given to developing training strategies that more effectively utilize the high-quality data during training. rather than only during data curation.
A natural idea to improve the utilization of high-qualit, data is to use curriculum learning [ This motivated by the catastrophic forgetting problem (McCloskey Cohen 1989), which refers to the phenomenon that model may forget the knowledge it has learned before when it is exposed to new data (Dai et al. 2025] Liao et al., 2025). In contrast to random shuffling, this curriculum based approach aims to optimize knowledge acquisition by exposing the model to high-quality data in the latter stages of training.
1uokr244 @ mails tsiihisssinguu..cdu..om
i "Corresponding authors.
'Traditionally, curriculum learning refers to training on progressively harder examples. Here, following prior work (Wettig et al, (224). we generalize the term to denote data-ordering strategies such as progressing from low-quality to high-quality data.

---

Published as a conference paper at ICLR 2026
Leamingg (110
Validation Loss
Uniform Descend Ascend
Validation Loss
Uniform
Descend
Leaming Rate (x10
Ascend
2.64
LR
2.52.
LR
2.36
2.46
2.48
1.35
1.50
1.35
1.50
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Step (x10*)
Step (x10*)
(a) Constant LR schedule.
(b) WSD LR schedule.
Validation Loss
4-
Uniform Descend Ascend
Learning Rate (x10-)
Data Quality Score
1.0
Leamingg Rate (x10-)
2.52
LR
High Quality Dats,
Low Learning Rate
2
2.48
Ascend
2.44
WSD LRS
1.35
1.50
Cosine LRS
0.0
0.0
0.5
1.0
1.5
0.0
0.2
0.4
0.6
0.8
1.0
Step (x10")
Fraction of Data
(c) Cosine LR schedule.
(d) LR schedule vs Data schedule.
Figure/ 1: Data curriculum strategies less effective when combined with learning rate (LR) schedules that decay to low scale near the end. (a-c) Experiments on a 1.5B parameter model trained on 30B tokens compare various data curricula (Uniform, Ascending- Ordder, and Descending- Order by DCLM score (Li et al. 2024) under constant, Saamup-Stable-Deeee While curricula improve validation loss over uniform baseline with a constant LR, this advantage is signif- icantly reduced during low-LR phase following LR decay. (d) In the data curriculum, high-quality data placed in latter phase, which coincides with the LR decaying to relatively low scale.
(WSD) (Hu et al.] 2024 Hägele et al] 2024). and cosine schedules.
One successful curriculum learning strategy is multi-stage pretraining: first training on data mix- ture dominated by massive web data, then in the second stage, referred as mid-training (OLMo
et al. 2025) Abdin et al. 2022bb) shifting the data mixture to one that mainly consists of high-quality data. This strategy has been adopted by many recent LLMs, including OLMo (OLMo al. 2025
Phi-4 (Abdin et al. 2024a), and LongCat-Flash (Team et al. (2025.. This two-phase design is most
common, and also promising extend with more stages (Ywen et 2025 Allal et 2025 or follow with long-comten context extension (Yang et al. (025).
Another line of work explores curriculum learning at the instance level, where data samples are sorted according to quality scores and presented to the model sequentially (Wettig et al. 2024 Dai et al. 2025 Zhang et al., 2025 Kim Lee 2024). We refer to this as the data curriculun However, these studies mainly investigate different quality metrics and find that simple end-to-end sorting yields limited benefits. Consequently, several works propose alternative strategies such as folding curriculum (Detailed in Section 2). which reorders samples within consecutive phases in an interleaved manner (Dai et al., 2025) Zhang et al. (025). Despite showing promise, we find that this interleaved approach is fragile: its advantage does not extend to our larger-scale experiments with the DCLM fastText score (Li et al. 2024), widely used scoring metric (see Section
This raises central question: Why do instance-level curriculum learning strategies often yield limited benefits? This is not simply due to unreliable quality scores: metrics like the QuRating score (proposed and used by WWttig et al. (2024)). the PDS score (proposed by Gu et al. (2025b) and discussed by Dai et al.] (2025)), and the DCLM score (Li et al. (2024) are already informative enough to improve training efficiency by guiding high-quality data selection.
Our Contributions. this paper, we identify a key, yet previously overlooked factor: the incom- patibility between the ascending order of data quality and the decaying schedule of learning rate. As illustrated in Ti if we train an LLM with a constant LR, using a data curriculum that sorts data in ascending order of quality can indeed outperform the baseline that trains the model on data uniform order. However, when we switch to a more standard LR decay schedule, such as cosine
We use data schedule as general term for any strategy that specifies the order of training data. Unless stated otherwise, data curriculum denotes schedule that sorts data samples in ascending descending order (reverse curriculum) with respect particular quality metric.
2

---

Published as a conference paper at ICLR 2026
or Sarmmu--Stabe--eeee (WSD) (Lohhchllov & Hutter 2017 Hu al. 2024 (a schedule with warmup, plateau, and decay phases, see Figure [(b the benefit of the data curriculum diminishes. Moreover, observe that as the LR decay becomes more aggressive (e.g.. having a longer decay phase or a lower ending LR), the benefit of the data curriculum diminishes more.
To resolve the incompatibility between data curriculum and LR decay, firstly, we discuss straight- forward remedy: adopting a moderate LR decay schedule in curriculum learning. In Section [31] we show that tuning the ending LR in WSD schedules trades off the benefits of data curriculum and loss convergence, and setting it to a moderate value (approximately decaying to 1/3 of peak LR in our setting) can make use of high-quality data and outperform uniform data ordering.
Further, we propose a strategy that largely resolves the incompatibility between the data curriculum and the LR schedule by replacing LR decay with weight averaging eLi al. 2025c Izmailov al.) 2018 Tian et al. 2025). Weight averaging computes weighted average of recent checkpoints as the final model. It stabilizes model parameters and reduces noise in the training process, similar to the effect of LR decay. However, it can achieve this without diminishing the magnitude of updates. Therefore, we adopt a constant LR throughout training and pair weight averaging with the data curriculum. This combination allows the model to maintain a high learning rate and fully exploit the high-quality data introduced later in the curriculum. We call this approach Curriculum Model Averaging (CMA). Notably, CMA also extends emerging practices of introducing weight averaging into L.MM training (Tian et al. 2025 Li et al. 2025c, Sanyal et al.) 2024). showing that combining weight averaging with curriculum learning yields greater benefits than applying weight averaging to standard uniform-ordering pretraining. This combination is particularly effective under multi-phase pretraining, where high-quality data is introduced in the mid-training phase. In this setting, our approach achieves an average improvement of 1.2% in accuracy and over 2% on core benchmarks (defined in Section
solely by reordering data samples (Table 2
Building on these explorations, we further demonstrate that combining moderate LR decay, cur- riculum learning, and weight averaging can produce synergistic advantages and reveal previously overlooked high-performing pretraining regime, improving standard benchmarks by 1.64% in aver- age accuracy over random shuffling and standard decay. This finding underlines that hyperparam- eter settings optimized for uniform data ordering are not necessarily optimal for curriculum-based training. Prior work has naturally evaluated curriculum- mized for uniform data ordering. However, this regime is suboptimal for curriculum-base and, as previously reported, leads to only marginal benefits (Wettig et al.] 2024] Kim & Lee| (0024). In contrast, identify previously underexplored and more effective regime in the design space of LLM pretraining by co-designing these components, paving the way for future exploration.
training using regimes ori originally opti- training
We validate hypotheses and proposed strategies through experiments at scale sufficient to support our conclusions, training a 1.5 billion- parameter model on 30 billion tokens. Our findings demonstrate robustness across different quality metrics, LR schedules, and data mixtures, highlight- ing that co-designing data curricula with training dynamics is a powerful, data-aware strategy for improving LLM pretraining efficiency.
billion-parameter
2 LEARNING RATE DECAY COUNTERACTS DATA CURRICULUM
In this section, we analyze the critical yet often overlooked interaction between the learning rate (LR) schedule and the data schedule. We first explain how the learning rate acts as an implicit importance weight for each data sample. We then present empirical results to demonstrate three key points: (1) a data curriculum
yield significant benefits over uniform data order under constant LR schedule; (2) these benefits diminish when a conventional decaying LR schedule is applied, particularly during the final, high-quality data regime; and (3) while adjustments to the data schedule can mitigate this issue, the underlying conflict persists.
Analysis of Coupling between Learning Rate and Data Schedules. key insight is that the learning rate schedule acts as an implicit importance weight for each training sample. The parame- ter update at training step t 0 = 0, 7/:9t. where 7/ is the learning rate. The gradient ge can be decomposed into signal component, Elg.). which points in the direction of steady improvement, and a noise component, 6, A decaying learning rate 7) serves a dual purpose: it reduces the noise e, to stabilize training, but it also shrinks the update step taken in the signal direction E[g.]. While modern optimizers like Adam (Kingma & Bal (013)) use more complex update rules, the learning rate remains a dominant factor in the update magnitude. This dual role of 1)s creates fundamental
3

---

Published as conference paper at ICLR 2026
Uniform-Long
Validation Loss
Leurning (X10
Ascend-Long
Validation Loss Diff
Mid
Long
Uniform-Mid
0.05
2.5
Uniform-Sha
Asoend-Mic
Zero
Short
Ascend- Short
0.00
1.35
1.50
Uniform-zero
Ascend-Zero
LR
0.05
0.0
0.5
1.5
1.3
1.4
1.5
Step (x10")
Step (x10*)
(a) Validation loss for different decay steps.
(b) Luiiform
Lsscen
3
Uniform- 1x10
(xcend-Ix10-9
Validation Loss Diff
1x10
Validation Loss
x
2.5
2.6
2
Leaming Rale &
Uniforn-Ix10
Macend-Ix10--
0.05
2x10
1x10-1
Uniform-2x10
3x10
2.4
Ascemd-2x10
1.35
1.50
Uniform-3x10
0.00
Ascend-3x10
0.0
0.5
1.0
1.5
1.3
1.4
1.5
Step (x10+)
Step (x10*)
(c) Validation loss for different ending LRs.
(d) Lusiform
Lascend
training (Long, Mid, Short, Zero, respectively) and ending LRs (1 x 10 10 x 10 2 x 10 3 x 10 the benefit of data curriculum diminishes with more aggressive IRR decay. For each I.R decay, we train 1.5B-parameter models with uniform and ascending ordering of data based on DCLM scores, and measure the difference in validation loss. As shown (b) and (d), this difference becomes smaller with more decay steps smaller ending LRs.
Figure 2: When varying the decay steps across 37%, 18%, 6%
conflict in quality-based curricula. High-quality samples are intentionally processed at the end of training, but this is precisely when conventional LR schedules reduce тソ to its minimum. Conse- quently, the decaying learning rate diminishes the influence of the most valuable data, counteracting the intended benefit of the curriculum.
Experimental Settings. Our experiments grounded in the DataComps-LMM (DCLM) frame- work (Li et al. 2024) at the 1B-1x scale, ensuring our findings are validated at a substantial scale. We adopt the Qwen2.5-1.51 model architecture (Yang et al.) 2024a) and train models 30B ken subset of the DCLM-Baseline dataset. For the data curriculum, we use DCLM's fasttext scores as our quality metric. We the peak LR x 10-3 and the ending LR to 10 aligning with optimal settings found in prior work for uniform data schedules (Li et al.) 2024 Luo et al. 2025
Li al. 20256). We evaluate performance held out for validation. We also report downstream task scores by the OLMES framework (Gu al., 2025a). Among the standard benchmark suite, we choose MMLU (Hendrycks et al.) (021)) ARC (Clark et al.) (2018), and CSQA (Talmor et al. (019)) Core benchmarks, as recent work suggests they have higher signal-to-noise ratio to distinguish model performance (Heineman al., 2025).
high-quality subset of the DCLM-Bascline dataset
Further experimental details provided in Section C.I
Data Curriculum is Highly Effective with a Constant Learning Rate. To isolate the effect of the data schedule from the LR schedule, we first conducted experiments using a constant learning rate of x 10-3, We compared three data schedules: a uniform random baseline, an ascending- order curriculum, and reverse (descending-orden) curriculum, both curricula sorted by DCLM quality scores. shown Figure [(a) the ascending-orde curriculum significantly outperforms the uniform baseline, achieving much lower validation loss and faster convergence. In contrast, the reverse curriculum's 's validation loss trends upward. likely because the data distribution shifts progressively away from the high-quality validation set. These results clearly demonstrate that quality- curriculum effective when its impact is not confounded by a decaying learning rate. Similar trends observed using PreSelect scores (SHUM al. 2025 (see Appendix Fig- ure 8(a)).
The Curriculum's Advantage Diminishes with a Decaying LR Schedule. In sharp contrast to the constant LR experiments, the advantage of the data curriculum largely disappears when we employ WSD schedule (Figure(I(b). The performance degradation is even more pronounced with a cosine schedule, which decays throughout entire range (Figure [[c]]. To further probe this relationship, varied the aggressiveness of the L.R decay by adjusting two WSD parameters: the

---

Published as a conference paper at ICLR 2026
Unitfom
Descend
Descend- Folddng
Ascend
Ascend-Folding
4.5
1.0
Leariing pxio->)
0.8
4.0
Validaion Loos
4.0
Loaning Rato OC10
Dan Quality
0.6
Sglication
2.48
2.52
3.5
2.64
2.56
0.4
2.44
3.00
1.25
1.50
1.35
1.50
02
2.5
0.0
0.4
Fraction of Data
Stup (x10*)
Skep (x10")
Figure 3: A stage-wise "data folding" curriculum mitigates the negative interaction observed between data ordering and learning rate (LR) decay (detailed in Section[], but data folding can not match end-to-end sorting under constant learning rate. Left: We compare simple ascending curricula (Ascend), sorted by DCLM score, against their "folding" counterparts (Ascend+Folding). The folding method involves partitioning the data into stages (three our implementation) and performing the sort within each stage. The Rescend((+Folding curriculum is designed in reverse order. Middle: Under a standard cosine LR schedule, folding strategies reduce validation loss compared to simple sorting but are outperformed by a uniform data baseline. Right: Conversely, with a constant LR schedule where decay does not weaken the utility of high-quality data, the advantage of folding vanishes, a simple ascending-orde order curriculum becomes the most effective strategy.
number of decay steps and the final learning rate. The results in Figure 2 show clear trend: as the decay phase becomes longer or more aggressive (i.e., a smaller ending LR), the performance benefit of the data curriculum over the uniform baseline shrinks, eventually becoming negligible. This confirms the previously overlooked effect of LR decay on the data curriculum, where LR decay undermines the contribution of high- nigh-quality data.
Exploring an Alternative Curriculum: Data Folding. We also investigated whether a different curriculum design could mitigate the coupling effect. We tested a folding curriculum, inspired by prior work (Dai et al. 2025] Zhang al. 2025), where the dataset is split into several chunks and each chunk sorted internally. This stage- wise design distributes high-quality data more evenly throughout training than the standard data curriculum. As shown in Figure 3 under cosine sched- ule, the ascending with folding strategy performed better than simple end-to-end ascending sort but still underperformed the uniform baseline. Conversely, under a constant LR schedule, the simple ascending-orde curriculum proved to be the most effective strategy, outperforming the folding cur- riculum. These results support our hypothesis: when the LR decays, folding offers a slight benefit over simple sorting by processing some high-quality data earlier, but under a constant LR, where this is not a concern, simple end-to-end curriculum remains superior. Refer to Section E. for more detailed discussion data folding experiments.
UNLOCKING DATA CURRICULA POTENTIAL VIA MODERATE LR DECAY AND MODEL AVERAGING
To resolve the interaction between data curricula and learning rate (LR) schedules, we first utilize more moderate LR decay in place of a standard aggressive decay, which find mitigates the issue. We then turn to a more principled approach: model averaging (Izmailov et al. 2018 Li et al., 2025c Tian et al.) 2025). We investigate replacing LR decay entirely with model averaging, which allows high-quality data to be processed with a constant learning rate. While model averaging alone may not match the performance of L.R decay with uniform data, we find that combining data cur- riculum with model averaging produces comparable or even superior results to standard decaying mid-training setting. This reveals a synergistic relationship between
LR schedule, particularly data curricula and model averaging. Furthermore, we find that combination of model averaging and moderate LR decay can yield even stronger and stable results for curriculum- pretraining. Our results highlight previously unexplored regime for improving LLM pretraining and reveal the potential of co-designing LR schedules, data curricula, and model averaging strategies.
3.1 MITIGATING NEGATIVE INTERACTION WITH MODERATE LEARNING RATE DECAY
A straightforward way to mitigate the negative impact of LR decay on data curricula is to use a moderate LR decay instead of standard aggressive one. As shown in Figure 2 for uniform data, the validation loss decreases with more aggressive LR decay, like increasing decay steps and lower ending LRs in our experimental setting. In comparison, the benefits of data curricula increase with
5

---

Published as a conference paper at ICLR 2026
Table I: Curriculum Model Average (CMA) exhibits advantages over standard LR decay schedule pretraining. much better than the widely used Cosine+ Uniform setting. Our proposed methods are highlighted gray. WA: Weight Averaging technique (Section B). Order: Data ordering. LRS: Learning Rate Schedule (WSD: Garmuu-Staabe-Deeaa
Cos: Cosine, Const: Constant). Core: Average score on the first four, high signal- lo-noise tasks according to prior work (Heineman al.) 2025 (MMLU, ARC-c, ARC-e, CSQA). Both the Core and Avg. scores are annotated with a subscript indicating the performance change relative to the baseline (WSD + Uniform). Performance changes are color-coded: bold green (2 0.5 improvement), light green (>
improvement,, and red (decrease).
WA
Order
LRS
MMLU
ARC-e
ARC-e
CSQA
Core
OBQA
PIQA
SIQA
Wino.
Avg.
Uniform
Cos
30.49
38.13
59,47
49.14
44.31
1.00
42.20
71.87
45.19
56.51
49. 1.45
Ascend
Cos
30.80
39.80
59.12
51.27
45.25 0.96
42.60
71.55
45.65
57.06
49.73
X
Uniform
WSD
30.77
14
61.05
50.86
46.21
45.20
72.42
45.75
56.27
50.56
Ascend
WSD
31.58
38.80
61.05
50.37
45.45-u.70
u.76
45.80
71.82
46.01
57.30
50.34-0.22
WMA
Uniform
Coast
30.87
37.12
58.95
53.24
45. 45.04-1.17
43.40
71.76
46.26
57.38
49.87-o
SMA
Uniform
Const
31.22
36.12
59.82
53.97
45.28 0,93
43.40
71.98
46.42
$7.85
50. 0.45
EMA
Uniform
Const
31.39
36.45
59.82
53,48
45.29-0.92
42.40
72.14
46,32
57.54
49.94-0.62
WMA
Ascend
Const
31.67
39.80
61.40
53.07
46.49
45.00
71.93
45.45
57.14
50.68 (0.12
SMA
Ascend
Const
32.28
40.80
62.11
52.91
47. 17.02.0.81
44.80
71.60
45.80
57.22
50.94 40.55
EMA
Ascend
Const
32.17
40.80
61.75
53.07
46. 46.95,a77
44.80
71.55
45.85
57.62
50.95.0.30
a more moderate LR decay, like fewer decay steps or higher ending LRs. The different preferences of LR decay suggest that the optimal ending LR or number of decay steps can differ for curriculum- based training from uniform data training, also indicated by the validation loss curves in Figure [ Since the optimal ending LR typically close to zero for uniform data ordering. but more decay steps are not always better (Li et al. 2025b] 2024 Hu et al. 2024), it is more convenient to ablate on the ending LRs to see whether the optimal LR schedule changes for curricula.
To investigate this, we run experiments on ending LRs in a fine-grained manner and report the training results for both uniform data ordering and data curriculum. As shown in Figure [[a) we find that the data curriculum (Ascend+ WSD) may only achieve marginal improvement or even fail to match the performance of uniform ordering (Uniform+WSD) when the ending LR is close to zero, like at the scale of 10-5, However, the ending LR increases, the performance of curriculum training improves, but may degrade when the ending LR approaches the peak LR. Note that the performance of the data curriculum can decline while its relative benefit over uniform ordering can still increase as the uniform training performance degrades more sharply when ending LR increases. The optimal ending L.R of the data curriculum is around 1 10-3, much higher than that for uniform data ordering, and the tuned data curriculum training outperforms the optimal uniform data training results. These experiments validate that using a more moderate LR decay-for instance, adjusting the ending LR to approximately 1/3 of the peak LR in our setting can effectively mitigate the negative interaction and unlock the benefits of data curriculum.
3.2 MODEL AVERAGE CAN HELP DATA CURRICULUM
CMA: Replacing Learning Rate Decay with Model Averaging. Adjusting the ending LR serves as trade-off between the benefits of data curriculum and LR decay. To fully utilize the benefits of data curricula, we propose to decouple the data schedule from the side effects of LR annealing by replacing LR decay entirely with model averaging. In this approach, LR schedule with a constant LR and apply model averaging to the final checkpoints of the training process. We call this strategy Curriculum Model Averaging (CMA), detailed in Algorithm [ In our default setting, we use Exponential Moving Average (EMA) with a = 0.2 over the last six checkpoints. The types of weight averaging considered include Simple Moving Average (SMA), EMA, and Weighted Moving Average (WMA), as introduced in Section B For comparison, we use standard LR pretraining schedules, including cosine and WSD (introduced in Section [B. The strongest baseline for our evaluation, denoted WSD + Uniform, is the best pest--erforming combination of a standard LR schedule and data order. Downstream task performances are reported in Table [ and practical implementation details are reported in Section
replace the decaying
The Synergy of Data Curriculum and Model Averaging. The results in T lead to several key observations. First, the combination of a data curriculum and model averaging (e.g., EMA + Ascend) outperforms models trained with a standard LR decay schedule, including WSD Uni- form and WSD + Ascend. It also consistently outperforms model averaging applied to a uniform data order (e.g.. EMA Uniform). Second, this synergy is crucial, as other combinations yield limited improvements. For instance, model averaging with a uniform data order (EMA + Uni- form) under constant learning rate does not fully match standard WSD schedule. Furthermore,

---

Published as a conference paper at ICLR 2026
Uniform Decay
Ascendi Decay
Ascend-EMA
L(w) Plane
Lin
Paaie
Tmjootory
Start
Ead
Valley Plooe
Last to Avg
Model Average
Figure 4: Visualization of our intuition about the interplay hetween data ordering and LR schedules. We assume the gradient update can decomposed as a signal direction and a noise direction. High-quality data can offer less noisy direction and more stable signal direction, while low-quality data can induce a more noisy update. Uniform+ Decayy Asccnd+Decay and Ascend EMA represent different training strategies. As- cend+EMA can make the best of the high-quality data in the curriculum. The right-hand figure shows the projection of the trajectories of the last steps for these cases onto the =noise-[(ww) plane.
combining a standard LR decay with a data curriculum (WSD + Ascend) provides only marginal gains and can even degrade performance, confirming the negative interaction we identified between schedules. These results highlight the necessity of combining both a data curriculum and weight averaging, strategy largely overlooked by prior work that has focused on either weight averag- ing (Tian et al.) 2025 Yang et al.] 2024b) or curriculum design (Dai et al. 2025) in isolation. Third, aligning the checkpoint weights with the data schedule is beneficial: EMA and SMA, which assign non-decreasing weights to later (and higher-quality) checkpoints, outperform WMA under a data curriculum. The differences between these moving average strategies are detailed in Section B
Synergy from Decoupling Schedules. We intuitively interpret the synergy between a data curriculum and model averaging from a loss landscape perspective, emphasizing the interplay between two key factors: the learning rate and data quality. We present a visualization of this concept in Figure 4 The learning rate controls the size of the update steps, while data quality influences the -ignal-to-noise ratio of the gradients. Uniform+Decay progresses with a relatively consistent level of noise compared the signal from the training data, and the final decay reduces the noise but also limits the update rate along the signal direction; Ascend+ Decay starts with high noise, but the step-size decays too fast, so does not utilize the good signal from high-quality data to move faster near the end; When it comes to the model averaging strategy, the training over uniform data may not reduce noise as effectively as a near-zero learning rate, which could explain its slightly lower performance reported in Table However, when using curriculum strategy. labeled as Ascend+ EMA, although the training starts with higher noise, the high-quality data introduced late in training provides a clearer and more reliable gradient. This strategy maintains the update magnitude in the high-quality regime, allowing it to take advantage of the good signal near the end along the signal direction. Using model averaging can also reduce noise along the noise direction, probably achieving a better balance between progress and stability compared to aggressive learning rate decay. We also provide a simplified theoretical model in Section 4 and discuss our perspective in the context of prior work (Wen et al., 2025) in Section E.
Motivation:
3.3 RESULTS ON MID-TRAINING WITH MIXED QUALITY DATA
CMA Benefits are More Pronouneed in Mid- Tra-Train Mid-training is emerging practice in
LLM pretraining where a large corpus of average-qualing data is supplemented by a smaller, high-
quality dataset in later training stage (Yang et al. 2025 OLMo al. 2025 Hu et al. 2024). We conduct mid bra-itraning experiments, with settings detailed in Section C.T As shown in Table CMA exhibits larger benefit in this practical setting compared experiments
uniformly high-quality data. The CMA results (e.g.. EMA + A-7) show a significant advantage over the WSD schedule baseline (WSD U,U), improving the average accuracy by 1.20% and achieving over 2.0% im- provement on average across the core suite of benchmarks. This margin is notable given that no additional complex data filtering is applied. provement is that, when high-quality data is sparse, each sample provides a relatively more valuable signal for parameter updates, amplifying the benefit of CMA.
possible explanation for the more prominent im-
Practical and Simplified Strategy also Performs Well. In practice, sorting an entire data cor- pus globally may not be feasible. As alternative, we tested a strategy where data is sorted in ascending order within each training phase separately (A.A in Table D The results show that the benefits of approach over standard LR decay largely persist. However, applying curriculum
7

---

Published as a conference paper at ICLR 2026
Table 2: The benefit of CMA becomes more prominent in the mid-training setting. Our proposed methods are highlighted in gray. WA: Weight Averaging technique (Section B Order: Data ordering in two phases (U: Uniform, A: Ascend). A-T (All-Together) sorts data samples in both phases as a whole. LRS: Learning Rate Schedule (WSD: Warmup-Stale Stable-Deca schedule, Const: Constant LR). Core: Average score on the first four, high signal-to to-noise tasks (MMLU, ARC-c, ARC-e, CSQA). Both the Core and Avg. scores are annotated with subscript indicating the performance change relative to the baseline (WSD + U,U). Performance changes
color-coded: bold green (2 0.5 improvement), light green (> improvement). and red (decrease).
WA
Order
LRS
MMLU
ARC-c
ARC=e
CSQA
Core
OBQA
PIQA
SIQA
Wino.
Avg.
X
U.U
WSD
29.23
33,78
53.86
49.55
41.61
40.40
71.87
44.78
56.43
47.49
X
U.A
WSD
29.44
34.45
52.63
50.12
41.66.0.05
41.00
71.76
44.42
56.75
47.57_00.0
A.A
WSD
30.22
33.11
56.84
47.34
40.88
39,40
71.55
44.78
56.67
47, 47,49000
X
A-T
WSD
29.93
37.12
54.39
49.47
42.73-1.12
39.00
72.20
45.14
56.83
48.01
EMA
U.U
Const
29.84
32.78
52.28
$1.52
41.60....0
42,00
71.60
44.68
$6.99
.0.21
EMA
U.A
Const
29.75
35.12
51.75
48.57
41.30 0.33
42.20
70.51
44.83
56.91
47.45 0.04
EMA
A.A
Const
30.31
36.45
57.54
50.12
43.61.2.0
41,40
72.14
45.09
56.43
18.69.120
EMA
A-T
Const
30.81
36,29
$7.89
50.29
43.82.2.21
44.50
70.62
44.68
54.46
48. 48.69
SMA
A-T
Const
30.65
36.79
57.37
50.78
43.90.2.29
43,60
70.89
44.73
54.74
48.69.1.29
only to the final, high-quality data phase (EMA + U.A) is not sufficient for optimal results. The su- perior performance of the A.A schedule over U.A suggests that applying a curriculum to the initial, lower-quality data phase is also beneficial. One possible explanation is that reordering data in the luwll-quality phase can exploit the model's 's forgetting mechanism to mitigate the adverse effects of toxic samples that pass through the cleaning pipeline by chance. These results further confirm the synergy between model averaging and a data curriculum.
3.4 OVERLOOKED BENEFIT: CO-DESIGN OF DATA CURRICULUM, LR SCHEDULE, AND WEIGHT AVERAGE
CDMA: Combining Moderate LR Decay with Model Averaging. We have identified the ben- efits of the moderate LR decay and weight averaging in curriculum-based pretraining. A natural question is whether combining a moderate LR decay with model averaging under a data curriculum can yield further improvements. We conducted a series of experiments varying the ending learning rate in WSD schedules, from 1 X 10-5 to 3 x 10-3, and then applied EMA to the final checkpoints. As shown in Figure [5] this combination achieves stable and optimal results with a moderate LR de- cay (i.e.. a higher ending LR than in standard practice). While a data curriculum with moderate LR decay alone also achieves near-optimal results, it does not fully match the combined strategy and may need a more careful tuning of ending LRs. These two curriculum- based strategies both out- perform their -orderrnn training counterparts when the LR decay is properly tuned for both curriculum-base based and uniform-ordering. Furthermore, as shown in Figure 5(b) of the mid-training setting, the best combination can improve by 1.68% in the average benchmark accuracy over the baseline (Uniform+ WSD with ending LR 1 x 10-5, corresponding to the left endpoint of the blue dash line), a prominent improvement without any additional data refinement. This motivates the following guideline for curriculum-basee pretraining: use a moderate LR decay and adopt model averaging. To distinguish this from CMA, we call this strategy Curriculum with LR Decay Model Averaging (CDMA). Specifically, curriculum-base LR decay than the optimal setting for uniform data training; their optimal regimes can differ substan- tially (around x 15 for uniform training versus roughly x 10-3 3 for curriculum-basee training in our setting). A weight averaging strategy can further enhance performance and produce more stable benefits in this moderate LR regime. A more systematic recipe for the strategy combinations can be promising direction for future work.
LLM pretraining should use a more moderate
Discussion: Why is this Combination Uxper-Explored? The CDMA strategy is straightforward and effective, which raises the question of why it has been underexplored. A possible explanation is that prior work has focused on an aggressive LR decay regime, which has obscured the discovery of alternative approaches. Confirmed by prior work (Li et al. 20256), this aggressive decay regime is close to optimal for the standard uniform data scenario, and there is a clear trend favoring a near-zero ending LR, as shown in Figure 5 However, the optimal regime for uniform data is not necessarily optimal for other settings. Prior work focusing on curriculum design mostly adopts cosine annealing schedules, within this aggressive decay regime, and has consequently reported marginal or disap- pointing results (Zhang et al.) (025). The negative results caused by these compounding factors can prevent more progress curriculum-based pretraining. Hence, our proposed optimization space
8

---

Published as a conference paper at ICLR 2026
51
Optimal Regine
Optimel Regime
49
Average Store
Averago Score.
Previous Focus Regimo
48
49
Prviius Focus Regine
Asoand EMA
Uniform EMA
Ascand EMA
Uniform EMA
48
Ascand-
Uniffom- WSD
46
Ascend
Uniform-
10
10
10
10
10
10
Ending LR
Ending
(a) DCLM.
(b) Mid-Training (WebOrganizer DDLMM).
Figure 5: This figure compares various training strategies, identifying a high-performing and previously derexplored Optimal Regime where moderate learning rate (I.R) decay, weight averaging. and curriculum learning produce synergistic advantages. We nun experiments on hoth Uniform (uniformly ordered data) and Ascend (training data arranged by ascending DCLM scores) data schedules. For both schedules. we conduct an ablation the ending learning rates of WSD schedules, ranging from 1 10 to 1 10-3, representing aggressive to moderate LR decay. We denote strategies applying weight averaging as EMA, which compute the final model checkpoint via an EMA of the last six checkpoints, and denote those strategics without weight averaging as WSD. We mcasure performance by the average downstream task score (as in Table D. This newly identified regime contrasts with the Previous Focus Regime, which represents common practices without data curriculum or weight averaging, and with an ending LR between x 10 and 1 10 4. This range is typical in prior work, which often uses an ending LR of one-tenth of a peak LR (on the scale of x10 ") (Dubey et al.
2024 DeepSeek-Al et al. 2024) or fixes the ending LR around 10 (Li et al. 2024 2025b). This observation also holds for mid training settings.
involving the co-design of LR schedules, data curricula, and model averaging strategies is still un- derexplored
Ablation Study. We conduct ablation studies to verify the robustness of our method. Our ap- proach generalizes effectively when evaluated with an alternative quality metric (PreSelect) and a different, unfiltered pretraining dataset (WebOrganizer) (SHUM al., 2025 Wettig et al. 2025) Full experimental details are presented in Section C.3
4 A THEORETICAL DEMONSTRATION
As we reported and discussed above, the benefit of curriculum learning emerges when we apply weight averaging method instead of a learning rate schedule with excessive decay, such as Cosine or WSD schedules, in practical pretraining. In the following, we present a simple theoretical model that recovers the above empirical insight. The main proof of this section can be found in Section F
Problem Setup. We consider a quadratic loss function L(w)
1 w 12 where w (W1, W22 R represents the trainable parameter, and w" denotes the ground truth, which is set to the original point (0, 0). We use Stochastic Gradient Descent (SGD) to optimize this problem. We denote the one-sample loss used to calculate the gradient for the t-th iteration as ((w)) ==
||w - 22 where data point æ; sampled from some given dataset D (a<(1), (2). x (M)). Therefore, we have the SGD update rule for the f-th iteration as the y) denotes the learning rate in the 1-th iteration. We initialize the parameter as wo (L, 0). We
w;-1 Ve (1-1 1). where
denote the learning rate schedule by E = {m, ......, 7MM }. We denote w; = (wx (1) (2)). We de- fine WM,E to be the distribution of w.. The randomness within WM comes from the random draw of the distribution in SGD. We further define the expected loss L(M; E) == Ew- [[(w)].
In the following, we consider the training dataset D, which consists of M different data points with varying data qualities. Specifically, data point (x) (x (.))) satisfy that xx (i 1)d and
Uniform(-4. (-L, L). we further set d L/M. x(1) provides a signal in the first dimension and introduces noise in the second dimension. Next, we consider two sampling strategies for each iteration of SGD: (1) We sample one data point uniformly from Uniform (D); (2) we sample one
data point from D in an ascending order. In other word, in t-th iteration, o = x (M-++1) e D. The
visualization of optimization trajectories in simulation experiment can refer to Figure 6
Uniform Sampling Learning Rate Schedule. SGD acts as an exponential averaging of the current parameter and the sampled data point. Once we uniformly sample data points with no ordering, then on the x-axis, the parameter would approximately oscillate from 0 to (m 1)d with large variance. We can prove that given any data schedule E starting with some 77 I 1, the expected loss for uniformly sampling SGD has lower bound
9

---

Published as a conference paper at ICLR 2026
Uniferm WSD
Ascend
Ascend WSMD
Individual Runs
Model Average
Trajoctory Moan End Point
Opšmum w
Figure 6: Visualization of the simulation experiments of the theoretical example. The mean trajectory is averaged over It 20 runs. The yellow star marks the global optimal, and wi represents a signal direction and we represents noise direction. The data samples are distributed evenly along the signal direction and randomly located along the noise direction. Ascend+ WSMD and Ascend+ win by sufficient progress along signal direction; Uniform+ WSD fails for inconsistent signal and thus large variance along signal direction; Ascend+ WSD fails for carly-decay, resulting in insufficient update along the signal direction.
min = E
(1)
This lower bound is derived from the loss on the x-axis. When we apply the uniform sampling, SGD cannot get enough signal towards the right direction; instead, the SGD optimizer would approach the mean of x1 #1 ,...,, 11
(2)
(M) in expectation.
Ascending Data- Pada-Ordering + Practical WSD Schedule. Next, we sample data in ascend- ing order from (Mf) to (1),, using the following WSD learning rate schedule É such that [0.9M] + 1, and 7/, = T-(M-Tb) for [0.9M] + t M, where To = M - J. In this learning rate schedule, we follow a practical setting, where we decay 10% of the total iterations. We then show that for this learning rate schedule E, the expected loss still cannot break the lower bound
T = 2 for 1 < t =
L(M; E) = ((L)).
Ascending Pata-Ordering + WSMD Schedule. In the above, we show that even using an as- cending data-ordering, the loss lower bound does not improve if we decay too much in the learning rate schedule. Next, we show Warmup- Sarmup-Stable-Me Stable-Moderate Decay (WSMD) schedule with less decay and larger ending leamning rate can better utilize the ascending data-ordering and get a smaller loss. Specifically, do modification of the above WSD schedule, setting To = O (M3). WSMD schedule can break through the above lower bound, denoted as E*
[(M; E*) ( 31)).
Ascending Data-Ordering + Stochastic Weight Averaging (SWA). Despite the failure of the practical WSD learning rate schedule, we demonstrate that with constant learning rate, sample SWA surpasses the aforementioned lower bound. The reason is that: (1) First, along the x-axis, with a constant learning rate, the updated parameter gets larger gradient accumulation towards the ground truth compared with the practical WSD with 10% decay, which is too much to get enough loss reduction. (2) Second, the SWA allows appropriate averaging along the y-axis and results in noise reduction, thus leading smaller loss, as the WSMD schedule does.
Theorem 4.1. Given learning rate dj S 1, the parameter derived by the averaging on the last weights t As = n n- WM-- where n (M+)) such that the expected loss
[[[(wM)] = Ö(M L²),
where Ō(-) hides log factors and constants independent of and M.
CONCLUSION
In this work, we investigate the interaction between data schedules and learning-rate (LR) schedules in language model pretraining and identify a fundamental tension between curriculum learning and conventional LR-decay strategies. We propose to combination of moderate decay and model averaging to solve the mismatch, uncovering a previously underexplored optimization regime that better aligns curriculum-based data ordering with LR decay.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marah I Abdin, Jyoti Aneja, Harkirat S. Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 technical report. CoRR, abs/2412 2024a. doi: 10.48550/ARXIV. 281200
2412.08905. URL
org.
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat S. Behl, Alon Benhaim, Misha Bilenko. Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mi- tra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy. Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp A. Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang. Ziyi Yang. Donghan Yu, Chengruidong Zhang. Cyril Zhang. Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. Phi-3 technical report: A highly capable language model lo- cally your phone. CoRR, abs/2404. 14219, 2024b. doi: 1044550ARXIV.2244.444 httpp:://do org/10. www.g/...4555500 48550/ wrwia.iiiiv.. 2404. 14219
URL
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlícek, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of small language model. URL https: //dci. o
CoRR, abs/2502.00237,, 2025. doi: 10.48550/ARXIV255020000
10. 405444550/a1 2ttpkiiv.5550.... 02737
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA:
reasoning about physical commonsense in natural language. In The Fourth AAAI Conference Artificial Intelligence, AAAI 2020. The Thirty-Secone Innovative Applications of Artificial Intel- ligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artifi-
cial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 7432-7439. AAAI
Press, 2020. doi:
v34105. 6239
Daniel Campos. Curriculum learning for language modeling. CORR, abs/2108.02170, 2021. URL https://arxiv. http://www..rr//b//////00// 02110
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. CoRR,
2018. http:////
/rg/ abs /tt.//103...0 05457
Yalun Dai, Yangyu Huang, Xin Zhang, Wenshan Wu, Chong Li, Wenhui Lu, Shijic Cao, Li Dong, and Scarlett Li. Data efficacy for language model training. CoRR,
2025. doi: 10. 4855088550010 www.a. 2155.
48550/ARXIVV2500211544 URL http:h:t:://// /tt::///oo.../ http::::///oo..r//1/////
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Ji- aqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean
11

---

Published as a conference paper at ICLR 2026
Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang. Mingming Li. Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma. Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan. T. Wang, Tao Yun, Tian Pei. Tianyu Sun, W. L. Xiao, and Wangding Zeng. DeepSeek- V3 technical report. CoRR, abs/2412. 2024. doi: 0.48500ARXIV.2412.10000 https: org..o...o..... C00.0..885550 48550/ a 2412. 11244.1..9437
URL
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez. Austen Gregerson, Ava Spataru, Buptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz. Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Eytimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Llama 3 herd of models. CoRR, abs/2407.21 7883, 2024, doi: 1048500ARXX.2240.22277 URL https: //doi. orooorgg//oooo 48550 arxiv. 2407 21783
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI 2025, Al- buquerque, New Mexico, USA, April 29- MMay 4, 2025, pp. 5005-5033. Association for Com- putational Linguistics, 2025a. doi: 10. 10.18653/V1/2025 1IL.:6//////222SSIIIIISSSSSAE wttpg//.....3/111111 7/doi oww./org////oo 10. 18653/v16vvv/e 2025. 1.n55...iiii findings naacl. 282
URL https:
Yuxian Gu, Li Dong, Hongning Wang. Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang. Data selection via optimal control for language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025b.
URL httph:tt:///ett https////peeeevveeee net// forumrrummmo forum?id=chhL51-1 fy
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ul- rich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.). Advances in Neural In- formation Processing Systems 38: Annual Conference cessing Systems 2024, NewrlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. URL httt:://ppppr: Ba5500aaaaabtaan22222aaaaaaaaaa.
Neural Information Pro-
http:/eeeepepepepepeeeeeeeeeeeee http://ssssseappe//////aaaaaaa /hash/
David Heineman, Valentin Hofmann, Ian Magnusson, Yuling Gu, Noah A. Smith, Hannaneh Ha- jishirzi, Kyle Lo, and Jesse Dodge. Signal and noise: A framework for reducing uncertainty in language model evaluation. CORR, abs/2508 13144, 2025. doi: 10.48550/ARX1V.550...33 URL https::/dooiii www.oiiorgg1.... 48550 2 13144
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Confer- ence on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRe- 3mm
view.net, 2021. URL
12

---

Published as a conference paper at ICLR 2026
Shengding Hu, Yuge Tu, Xu Han, Ganqu Cui, Chaoqun He, Weilin Zhao, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Xinrong Zhang, Zhen Leng Thai, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, da- hai li, Zhiyuan Liu, and Maosong Sun. MiniCPM: Unveiling the potential of small language models with scalable training strategies. In Fürst Conference on Language Modeling, 2024. URL https: https://opente wwt.://veepr.......eee. net forum? ?id 3X2L2TF
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.). Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial In- telligence, UAI 2018, Monterey, Califormia, USA, August 6-10, 2018, pp. 876 885. AUAI Press, 2018. URL BtLpttp::/ /tttp://auai... wwtp:o/w///.aooo//eoo
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https::/gpennzevieww net www.oooooioneeoooooo.eo FCnohuRGAnnn
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomás Mikolov. Bag of tricks for efficient text classification. In Mirella Lapata. Phil Blunsom, and Alexander Koller (eds.). Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 2: Short Papers, pp. 427 431. Association for Computational Linguistics, 2017. doi: (0.1863//V1/E1722000 URL htLhttt::: //od: 10. 10.18653/v1/e17-200
Jean Kaddour. Stop wasting my time! saving days of imagenet and BERT training with latest weight averaging. In Has Trained Yea? NeurlPS 2022 Workshop, 2022. URL https://
openreview. net dotpuummidd-OOAABBB
Jisu Kim and Juhwan Lee. Strategic data ordering: Enhancing large language model performance through curriculum learning. CoRR, abs/2405 2024. doi: 1044855/ARX1.24400077700
URL http:::/toooi. org/ 10. 48550/ (01555//axiiv arxiv. 2405. 07490
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL. http: 7/arxiv. wwx.iv.or www.vorrg/aaaa abs/ 1412.1126 6980
Binghui Li, Fengling Chen, Zixun Huang, Lean Wang, and Lei Wu. Unveiling the role of learning rate schedules via functional scaling laws. arXiv preprint arXiv:2509. 19189, 2025a.
Houyi Li, Wenzhen Zheng, Jingeheng Hu, Qiufeng Wang, Hanshan Zhang, Zili Wang, Shijie Xuyang, Yuantao Fan, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part I optimal hyperparameter scaling law in large language model pretraining. CoRR, abs/2500.04415. 2025b.
doi: 10.48850/ARXIV2550....1 047115
htttts::///oiiioo orq/10.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic. Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave Ludwig Schmidt, and Vaishaal Shankar. DataComp-LM: In search of the next generation of training sets for language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural In- formation Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10
13

---

Published as a conference paper at ICLR 2026
15, 2024, 2024. URL http://ppporss.issscc/ppappp http://////ppprreeeee/aaee//e//a
hash/ 1 httpa/aeaaaaaaeeeec..0ee..eeeaa. Benchmarks Track.htm.
and
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozh- skii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poiries João Monteiro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy V, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Moustafa-Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavit, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferran- dis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! CoRR, abs/2300.00166 2023. doi: URL puttps://dos 40050.....//0////.303.. ar.ia. 2305. 66161
48550/
Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, Jin Ma, Xunhao Lai, Deyi Liu. Yao Luo, Xingyan Bin, Hongbin Ren, Mingji Han, Wenhao Hao, Bairen Yi, LingJun Liu, Bole Ma. Xiaoying Jia, Xun Zhou, Siyuan Qiao, Liang Xiang, and Yonghui Wu. Model merging in pre-training of large language models. CoRR, abs/2505.1208/ 2025c. doi: 10.48550/ARXIV22555.10 12082. URL https: //doi. 48550
2505.
Chonghua Liao, Ruobing Xie, Xingwu Sun, Haowen Sun, and Zhanhui Kang. Exploring forgetting in large language model pre-training. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 2122-2127. Association for Computational Linguistics, 2025. URL https: //aclanthon ogy.
10002.acc--10 acll-lon.. 1057 1057
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htLltttts///::eeeeet
Kairong Luo, Haodong Wen, Shengding Hu, Zhenbo Sun, Zhiyuan Liu, Maosong Sun, Kaifeng Lyu, and Wenguang Chen. A Multi-Power law for loss curve prediction across learning rate schedules. In The Thirteenth International Conference on Learning Reprexentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: /owpnreveiieeveeeooo net forum?id' 9oos9XXxI1K
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor con- duct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsuji (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, oct 2018. Association for Computational Linguistics. doi: 10.18653/v1/Dis.. 1260. URL https: //Aclanchollogy.org//11--
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bha- gia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Py- atkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm,
14

---

Published as a conference paper at ICLR 2026
Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi.
OLMo furious. CoRR, abs/2501.00656, 2025. doi: 1044550/ARXIV250000665 URL
https://eet http:///ooo.oogg./... 00t.........55/.........0...0000
OpenAl. GPT-4 technicall report. CoRR, att../a2333.007.4420000000 2023. doi: 10485500ARRXX122500007777
URL http:/t::/::o...ogg//////// 1et.00885500.......
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The Re- finedWeb dataset for Falcon LLM: outperforming curated corpora with web data only. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz. Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
URL http: httt:::/ppaprraa nips ccpessccc/aaae cwtpccprpppppeeeeeeeeeeeeeee.eee www.ffeeeeeeeeeeee /tlps//pppee/22///// and
Benchmarks. html
Guilherme Penedo, Hynek Kydlícek, Vinko Sabolcec, Bettina Messmer, Negar Foroutan, Amir Hos- sein Kargaran, Colin Raffel, Martin Jaggi, Leandro von Werra, and Thomas Wolf. Fineweb2: One pipeline to scale them all - adapting pre-training data processing to every language. CoRR, 0ass/5506.29200, 2025. doi: 11.48500/ARXI.255062200 20920. URL htttt::/ http::////oo..r//1.... org/10. 48550/arXiv 2506 20920
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An ad- versarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artifi- cial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelli- gence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp- 8732-8740, AAAI Press, 2020. doi: V3419//AALL34111...00.
URL https: //ti:///oiiii org 10, 1.0...66999 /wwp/aaaia... v34105.
6399
Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, and sujay sanghavi. Early weight averaging meets high learning rates for LLM pre-training. In First Conference on Lan- guage Modeling, 2024. URL. https::tt///tttt..tt /ttp:///peeeevveeeeee ew www./forrrmmmm. 21d=IABCWENKK
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xi- aojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNLP)), pp. 4463-4473, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: (0.18653/v1/DI/ 1454. URL httptt::/acaaattooooooo/
org/
0019-1454
KaShun SHUM, Yuzhen Huang, Hongjian Zou, dingqi, YiXuan Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts the data that teaches. In Forty- second International Conference on Machine Learning, 2025. URL petettss://ppeeeee //pp/r/viee...eeeeemeeo
Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings. Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisticy (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 2459-2475, Association for Computational Linguistiss, 2025. URL //aclanthology org. acl-long..........a 123/
https:
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQQA: A ques- tion answering challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long Minneapolis, Minnesota, June 2019. Association for Com-
and Short Papers), pp. putational Linguistics. doi: (0.18653/11/N19. 1421. URL https: www.aaaaannotg........ooo
N1 9- 1421/
15

---

Published as a conference paper at ICLR 2026
Meituan LongCat Team, Bayan, Bei Li, Bingye Lei, Bo Wang. Bolin Rong, Chao Wang. Chao Zhang, Chen Gao, Chen Zhang, Cheng Sun, Chengcheng Han. Chenguang Xi, Chi Zhang, Chong Peng, Chuan Qin, Chuyu Zhang, Cong Chen, Congkui Wang, Dan Ma. Daoru Pan, Defei Bu, Dengchang Zhao, Deyang Kong. Dishan Liu, Feiye Huo, Fengcun Li, Fubao Zhang, Gan Dong, Gang Liu, Gang Xu, Ge Li, Guogiang Tan, Guoyuan Lin, Haihang Jing. Haomin Fu, Haonan Yan, Haoxing Wen, Haozhe Zhao, Hong Liu, Hongmei Shi, Hongyan Hao, Hongyin Tang, Huantian Lv, Hui Su, Jiacheng Li, Jiahao Liu, Jiahuan Li, Jiajun Yang, Jiaming Wang, Jian Yang, Jianchao Tan, Jiaqi Sun, Jiaqi Zhang. Jiawei Fu, Jiawei Yang, Jiaxi Hu, Jiayu Qin, Jingang Wang. Jiyuan He, Jun Kuang, Junhui Mei, Kai Liang, Ke He, Kefeng Zhang, Keheng Wang, Keqing He, Liang Gao. Liang Shi, Lianhui Ma, Lin Qiu, Lingbin Kong, Lingtong Si, Linkun Lyu, Linsen Guo, Liqi Yang. Lizhi Yan, Mai Xia, Man Gao, Manyuan Zhang, Meng Zhou, Mengxia Shen, Mingxiang Tuo, Mingyang Zhu, Peiguang Li, Peng Pei, Peng Zhao, Pengcheng Jia, Pingwei Sun, Qi Gu, Qianyun Li, Qingyuan Li, Qiong Huang, Qiyuan Duan, Ran Meng, Rongxiang Weng, Ruichen Shao, Rumei Li, Shizhe Wu, and Shuai Liang. LongCat- Flash technical report. CoRR, abw/2509.013222 2025. doi: 10.48550/ARXV..5500000
URL
485.0/ hittps://di..oz/ 405.0000455000000.1/.// 2509. 01322
Teknium. OpenHermes 2.5: An open dataset of synthetic data for generalist IIm assistants, 2023.
URL
22.5
Changxin Tian, Jiapeng Wang. Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao. Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. WSM: decay-free learning rate schedule via checkpoint merging for LLM tra-ingg CoRR, abs/2507. 1 17634, 2025. doi: 10445500ARXXV2255..7705
URL httptttts:/:oooooo /ttp:////00....000004455555055/0
arxiv 2507 16007.10 17634
Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing. CORR, abs/2408. 1029,, 2024. doi: 1044550AARXI.2200.11000
URL https:://ooo org/1...
(8550/arXivz 2408. 11029
Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, UIrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NewrlPS 2024, Vancouver. BC, Canada, De- cember 10 15, 2024, 2024. URL http: /ttp:: nips http://appr__ieepppaeeeeeeee
and Benchmarks Saaaaaaarras_rrckkaaae Track. htmll
Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Understanding warmu--ttable-decay learning rates: A river valley loss landscape view. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview. plopenreview.nntt/frrnnn wwtp://eeett/oo.eetttoo.ooo
Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high- quality data for training language models. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. httptt //openzeview. net. http://wwww..eeeooommedeeddddeee /tG.://
Alexander Wettig, Kyle Lo, Sewon Min, Hannanch Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. In Forty-second International Conference on Machine Learning, 2025. URL
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu. Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, a5s/2412.151155.
16

---

Published as a conference paper at ICLR 2026
2024a. doi: 1044550/ARXXV22212155515 httppttssr//oo...rr//00..88333..
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue. Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang. Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abs/2505 09388, 2025. doi: 1044855AAXXV22550.00055 003880 URL ptttps://do..rgg/2200 48550/ 2593300000000888
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. The Twelfth Interna- tional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nel, 2024b. URL.tttss///peerreeee
net forum nid 6NYD
Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu. Data mixing laws: Optimizing data mixtures by predicting language modeling performance. In The Thirteenth International Conference on Learning Representations, 2025. / ptttps:///penneviers net/ n wow.eooiouuuoooooooooeroe /CB23 3TMK3
Hu Yiwen, Huatong Song, Jie Chen, Jia Deng, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Zican Dong. Yang Lu, Xu Miao, Xin Zhao, and Ji-Rong Wen. Yulan-mini: Pushing the limits of open data-efficient language model. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguisties (Volume Long Papers). pp. 0374-5400 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 809--888910 291---211-10 doi: 1..18853/v1/2255 ong 268/
acl-long. 268. Utthhtss:://eeet 2ttp:///....tttooooooog//////0/0 ac
antho
Yijiong Yu, Ziyun Dai, Zekun Wang, Wei Wang, Ran Chen, and Ji Pei. Opencsg chinese corpus: A series of high-quality chinese datasets for LLM training. CoRR, abs/2501.081 2025. doi: 10.
4850/ARX11V2500008197 URL http://0..0...........a/.o//..o/
Yang Zhang, Amr Mohamed, Hadi Abdine, Guokan Shang, and Michalis Vazirgiannis. Be- yond random sampling: Efficient language model pretraining via curriculum learning. CoRR, abs/2506. 1 1300, 2025. doi: 1044855//RRIV.250066 1300. URL httph://p::ooo org/1C. (8550/arXiva 2506 11300
Fan Zhou, Zengzhi Wang, Nikhil Ranjan, Zhoujun Cheng, Liping Tang, Guowei He, Zhengzhong Liu, and Eric P. Xing. Megamath: Pushing the limits of open math corpora. arXiv preprint arXiv:2504.0220 028077 2025. Preprint.
17

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Curriculum Learning in LLM Pretraining. While data quality is known to be critical, lever- aging through fine-grained, instance-level curricula has shown limited success and has not been validated at a substantial scale (Dai et al. 2025 Zhang et al., 2025; WWttig et al. 2024 Campos 2021]. Prior studies have reported only marginal gains, sometimes observing benefits from counter- intuitive data orders without a clear underlying mechanism (Wettig et al. 2024). Other works either lack sufficient validation (Campos) 2021 Kim & Lee, 2024) or, finding simple curricula ineffective, propose more complex strategies like data folding (Zhang et al. 2025 Dai et al. (025). However, we find that the benefits of such complex strategies are often confined to smaller-scale experiments with low learning rates and can degrade at larger scales and under high LR regime (Section [E.I). A common thread in these attempts is the use of standard cosine learning rate schedules. We demon- strate that decayed LR prevents the model from effectively learning from the high-quality data in- troduced late in the training process, showing that a more moderate decay is required to unlock the curriculum's 's full potential. A more detailed discussion can refer to Section E.I
The learning rate (LR) schedule is a crucial hyperparameter in model pretraining. Traditional I.R schedules like cosine decay are the most widely used in LLM pre- training (Loshchilov & Hutter] [017]] Dubey et al.) 2024 Li et al. 2024). More recent schedules (WSD) have demonstrated strong performance and are suitable for mid-
Learning Rate Schedules.
like Saarmu-SStabl-Deceee training resumption paradigms (Hu et al. 2024 Hägele et al. 2024). Loss curve scaling laws suggest that they possess an optimal shape for a given peak LR (Tissue et al. 2024 Luo et al.] 2025 Li et al.] 2025a). A prevailing finding for training with standard uniform data ordering is that optimal or near-optimal LR schedule should decay to a value close to zero (Li al. 2025b OL et al.. [2025] Li et al. 2024). Our work challenges this convention in the context of purriculum-based pretrain- ing. We find that for a data curriculum, such an aggressive decay has negative effects on the data schedule. Instead, a moderate decay that maintains a higher learning rate during the high-quality data phase proves superior, especially when using model averaging. Unless otherwise specified, we discuss the LR schedules with a warmup phase. For example, a constant LR schedule refers a linear warmup followed by a constant LR.
Model Averaging. Model averaging, which combines parameters from multiple checkpoints to produce final, improved model, is well-established technique for improving generalization (Iz- mailov et al. 2018 Jin et al. [023] Yang et al., 2024b). It has been successfully applied in LLM pretraining to accelerate convergence and boost performance, and was reportedly used in training prominent models like Llama 3 (Kaddour 2022 Li et al. 2025c Dubey et al. 2024). Prior work has explored combining model averaging with decay-free LR schedules for training on uniformly ordered data and in the mid-training setting (Li al. 2025c Tian et al. 2025). However, Li et al. (2025c) finds that the performance of weight averaging is merely comparable to standard LR decay, and Tian et al. (2025) attempts to improve the results through a weighting strategy derived from the LR schedule (Discussed in Sections B and C.2). Unlike this prior work, which focused on uniform data ordering, our research is the first to investigate the interaction between model averaging and a quality- data curriculum. We find that these two strategies have a strong synergistic relation- ship, as model averaging provides the necessary training stability to learn from high-quality data with a high, moderately decaying learning rate.
T
B PRELIMINARY
Learning Rate Schedule. We consider two primary types of learning rate (LR) schedules in ad- dition to a constant LR schedule. The commonly used cosine schedule defines the LR at step i as r7(() = 170 ( (Lta + 1- cos (#z) where 7/0 is the peak LR, T is the total number of training steps, and a is the ratio of the final LR to the peak LR. A more recent alternative is the Warmup-Stable- Decay (WSD) schedule, which consists of a linear warmup phase, a stable phase with a constant LR No. and final decay phase. For the decay phase, use the I-sqrt function, where the LR is given by rt() = 17 (1
= 2
representing the progress through the decay period, which begins at step Edecay- Further details on our schedule choices are discussed in Section C.I
vr(t)) + r(t), with r(t) = T T-
t- twway
Model Averaging. Model averaging computes a weighted average of several model checkpoints to produce single, final model. We consider three common strategies. Suppose we have N check-
18

---

Published as a conference paper at ICLR 2026
points, Mı,.... MN, typically the last N checkpoints collected at fixed intervals, corresponding to steps ti,...,ty tv. We focus on the model averaging strategies discussed in Li et al. (202cc) summa- rized as follows. Simple Moving Average (SMA) (Izmailov et al. 2018 applies uniform weight all these checkpoints: Mavg NV EE= M, Exponential Moving Average (EMA) assigns ex-
ponentially decaying weights. EMA defined recursively Magg a.; (11-a)a0 a)M
with the base case Mag = M, The hyperparameter a E (0. 1] controls the decay rate; a larger assigns greater weight to the most recent checkpoint. Weighted Moving Average (WMA) uses predefined set of normalized weights Wi...., WN (where 2 uc 1) to compute the final model as Mave Ei=1 w,M,, Prior work (Tian et al. 2025) derives these weights from the learning rate schedule, where each weight is proportional to the drop in learning rate between checkpoints: wy x 17(44) 1)(4+11 for N, and wN n)(x). The weight computation procedure for WMA detailed Section C.2
N
Data Scoring. Raw web data must pass through processing pipeline before it used for pretrain- ing. The raw data first goes through heuristic filtering rules. Then in the model-based filtering phase, a scorer model assigns a quality score to each data sample. For example, DCLM Baseline dataset uses scores from fasttext model (Joulin et al. 2017) measuring similarity to high-quality sources like OpenHermes 2.5 (Teknium 2023) and top posts from the ELJ5 subreddit. Another approach, PreSelect (SHUM et 2025). scores data based its similarity downstream tasks. Typically, these scores are used to filter the dataset by removing samples below a certain quality threshold. In contrast, work does not use these scores discard data; instead, we use these quality scores to define the data ordering for curriculum learning.
19

---

Published as a conference paper at ICLR 2026
C EXPERIMENTS
C.I EXPERIMENTAL SETTINGS
This section details our experimental setup, which is grounded substantial scale for academic research. We describe our pretraining and evaluation procedures, justify our learning rate schedule choices, and outline practical mid-training configuration.
Pretraining Settings. Our experiments are conducted at a substantial scale for academic valida- tion, grounded in the DataComps-ILM (DCLMM framework (Li et al.) 2024) at the 1B-1x level. We use Qwen2.5-1.5B model (Yang et al. 2024a) for pretraining experiments, which is modern architecture incorporating SwiGLU activation functions and Grouped- -ueery Attention (GQA). The training dataset consists of 30B token subset sample from the first shard of the DCLM-Bascline dataset (Li et al., 2024). For our data curricula, we sort data ascending order based on DCLM fasttext scores; reverse currieulum, sorting in descending order, is also used for comparison. After moderate tuning, we set the peak learning rate to 3 x 10-3, with a sequence length of 4096 and batch size of 512, which For LR decay schedules, we set the final learning rate to 1 x 10 10-5, aligning with optimal settings found in prior work (Li et al. 2024 Luo et al. 2025 Li 2025b). To ensure reproducibility, provide a detailed list of hyperparameters in Table
found provides a good trade-off between throughput and stability.
Table 3: Model and optimizer hyperparameters for our Qwen2.5-1.5B experiments.
Hyperparameter
Value
Model Configuration
Sequence Length
4096
Hidden Size
1536
FFN Intermediate Size
8960
Number of Layers
28
Number of Attention Heads
12
Number of Bey-Value Heads (GQA)
Vocabulary Size
151936
Optimizer Configuration
Optimizer
AdamW
Weight Decay
0.1
Adam 31
0.9
Adam $3
0.95
Adam e
1.0 x 10-8
Gradient Clipping
1.0
Evaluation Settings. To ensure a robust comparison between methods, we track validation loss during training and evaluate final performance on comprehensive suite of downstream tasks. Be- cause the data distribution shifts during a curriculum, we created a dedicated high-quality validation set to provide consistent measure of progress. This set consists of 100,000 of the highest-scoring documents from a partition of the DCLM-Baseline dataset that is disjoint from our training data. For downstream evaluation, we use the OLMES benchmark (Gu al. 2025a), reporting performance on MMLU (Hendrycks et al. 2021). ARC-easy/challenge (Clark et al. 2018), CommonSenseQA (CSQA) (Talmor al. 2019). OpenBookQA (Mihaylov et al.] 2018). PIQA (Bisk et al.) 2020), So- cial IQA Sapp et al. 2019), and WinoGrande (Sakaguchi et al. 2020). covering world knowledge, common sense, and reasoning capabilities. Among these, we designate MMLU, ARC, and CSQA as Core benchmarks, as recent work suggests they have higher signal-to-noise ratio for distinguish- ing model performance (Heineman et al. 2025). Furthermore, as shown in Figure the average downstream scores exhibit a strong correlation with validation loss.
LR Schedule Choice Ablation. The decay phase of the WSD schedule can be implemented with various functions. Following prior work (Hägele al. 2024 Luo al. 2025), compared the
20

---

Published as a conference paper at ICLR 2026
Ascending
Ascending
Descending
46
Descending
Dewnitram lask Score
Downstream Task Soore
Overall Tiend
Overall Trend
49-
44
42
47-
Overall r. -0.912 Overall R²: 0.832
Overall r. -0.908 Overall R² 0.825
2.40
2.45
2,50
2.55 Validation Loss
2.65
2.40
2.45
2,50
2.55
2.60
2.65
Validation Loss
(a) Average downstream
validation loss.
(b) Core downstream
validation loss.
Downstream task scores and validation losses show high correlation according to the Pearson cor- relation coefficient (r) and R-square value (Rt²). Average is the average score of the total downstream Lasks and Core is the average of the first downstream tasks (MMLU, ARC-cfe, CSQA) in Tables Tand
Figure 7:
I-sqrt decay function, rt() 110 ) /TTrr((), and the sqrt-cube function, nt()
Yo (1 - r())).. Both decay functions outperform simpler alternatives like linear decay, and as shown in Table they produce strong and comparable results. We adopt the /-sqrí function for our main experiments due to its wide adoption in recent literature (Hägele et al.] 2024; Tian et al. (025)) Unless otherwise specified, experiments use decay phase ratio of approximately between 15% and 20% of total training steps, consistent with optimal ratios reported prior work (Hägele et al. 2024
Hu al. 2024).
Table 4: Models trained under WSD schedules under /-sqrt and sqrt-cube decay functions produce
similar results.
Dataset
Schedule
MMLU
ARC-c
ARC-e
CSQA
OBQA
PIQA
SIQA
Wino.
Avg.
Random
1-sqrt
26.60
27.42
42.28
42.26
37.20
67.85
42.02
51.30
42.12
Filtered
1-sqrt
26.97
30.10
44.04
44.72
36.20
69.15
42.58
51.78
43.19
Random
sqri-cube
26.62
27.42
42.28
42.42
35.00
68.28
43.65
51.70
42.17
Filtered
sqrt-cube
26.70
31.10
44.04
42.59
36.60
68.44
42.89
52.17
43.07
The Mid-Training Experiment Settings. We model mid-training, an emerging and practical pre-
training paradigm (Ynng et al. 2025 OLMo al. 2025 Hu al. 2024). with two-phase setup. The initial stable-L.R phase uses 29B tokens from the WebOrganizer dataset (Wettig et al. 2025) lower-quality data, while the subsequent decay phase uses 5B tokens from the higher-quality DCLM-Baseline dataset (Li et al. 2024). The WebOrganizer data has not undergone model-based filtering, whereas the DCLM-Baseline data represents the top 10% of its source, which has distri- bution roughly similar to the WebOrganizer data. The LR decays to 10-5 during the second, high-quality phase.
C.2 PRACTICAL IMPLEMENTATION DETAILS
Weight Computation for WMA. The computation of Weighted Moving Average (WMA) fol- lows Tian et al. (025). where weights are derived from hypothetical learning rate schedule. For our experiments, we use WSD schedule with a / /sqrrt decay function and final LR set to 5% of the peak L.R. Given normalized LR values ,/,,..,, V/N at each checkpoint (with = 1), the weights are calculated as the drop in learning rate between steps: w; = 7) 7/+1 for i < N, and wy = 7N- Suppose Mo, M1, ... MN the model parameters of the last N + 1 checkpoints, and assume Mr. Mo + Σ Σラ= Ij. where gj represents the total parameter update between checkpoint and j. This weighting strategy ensures that the averaged model is equivalent to re-weighting each
21

---

Published as a conference paper at ICLR 2026
update:
N-1 Σ (n 1++1)
Mr
Mo + Σ
9i
7/
Mo
9j
Mo +
n:g:
This formulation shows how the weighted average effectively re-weights the parameter updates 9: by their corresponding normalized learning rates 7/i thus simulating the effect of an LR schedule through averaging. Notably, for the /-sqrt decay function, this method results in set of monotoni- cally decreasing weights, as shown in Table 5
Table 5: Model Checkpoint Weights. Index -k corresponds to the last k-th checkpoint.
Checkpoint Index Weight
-6
-5
-4
-3
-2
-1
0.4249
0,1760
0.1350
0.1138
0.1003
0.0500
Practical Implementation of CMA (and CDMA). Our Curriculum Model Averaging (CMA) approach consists of three-stage pipeline. as detailed in Algorithm [ First, an offline data scheduling stage. the entire training dataset is sorted in ascending order based quality score. This large-scale sorting is performed efficiently as a one-time process using Apache Spark. A significant practical advantage is that these quality scores (e.g.. DCLM fasttext (Li et al., 2024) or PreSelect (SHUM et al. 2025)) are often already generated during data preprocessing for filtering purposes, allowing our method to be integrated into existing pipelines without requiring an additional, costly scoring step. Second, during the training stage, we employ simple warmup- constant LR schedule, forgoing any LR decay. Finally, to ensure the stability of the final parameters, we perform model averaging on the last several checkpoints saved during the constant-LR phase. We primarily use Exponential Moving Average (EMA), which assigns higher weights to more recent checkpoints trained on higher-quality data, thereby smoothing parameter variance while emphasiz- ing high-quality signals. For CDMA, the only difference is that instead of entirely forgoing LR decay, we use moderate decay (e.g.. to final LR of 1/5 to 1/2 of the peak LR) and then ap- ply weight averaging. Further exploration of optimal ending LRs and the scaling properties this combined strategy are promising directions for future work.
pre-computed
C.3 ABLATION STUDIES
In this section, validate the robustness of our findings. We conduct ablation studies demon- strate that our approach generalizes across a different quality metric (PreSelect) and a different, unfiltered pretraining dataset (WebOrganizer) (SHUM at. 2025 Wettig al. 2025).
Ablation Quality Metric. To test if our approach generalizes other quality metrics, conducted experiments using PreSelect scores (SHUM et al. 2025) to order the data. The results in Table 6 show that when using an ascending order of PreSelect scores, both CMA and CDMA outperform the standard WSD bascline. However, we note that ordering by PreSelect scores yielded slightly lower average performance than ordering by DCLM scores in our setup. Moreover, as shown in Figure M(a), although the ascending-order training shows an overall faster convergence in the latter phase of the training process, the final validation loss goes above the muiform-ordering data training. We conjecture this may be due to an inconsistency. as our base dataset was originally filtered using DCLM scores before being re-sorted by PreSelect scores, while the targeted validation set consists of samples with the highest DCLM scores.
Ablation on Pretraining Dataset. To verify that our method is applicable to broader data distribu- tions, we conducted experiments on the WebOrganizer dataset (Wettig et al. (025) alone, which has not undergone model-based filtering. As shown in Table 7, combining model averaging with data curriculum again produces superior results compared to a standard LR decay approach. Notably, in this setting, model averaging without any decay showed larger benefit than model averaging with moderate decay, possibly indicating that a high ending learning rate is particularly beneficial when high-quality data is extremely sparse.
22

---

Published as a conference paper at ICLR 2026
Algorithm Curriculum Model Averaging (CMA)
1: Input:
Unsorted dataset D. quality scoring function Q(+), training steps T, warmup steps Tws peak learning rate Tppaak. number of checkpoints to average k, averaging decay hyperparameter
a, checkpointing interval s.
2: Output: Final model parameters Oanal-
3: # Stage 1: Data Scheduling
4: Sort dataset D to create sorted where for any samples T4, i < j. then Q(x;) - Q
5: # Stage 2: Warmup-Constant LR Training
6: Initialize model parameters 0g.
7: for t = to T do
8: if t S Tu then
9:
1/+ < "lpeak"" (1/Tw))
D Linear warmup
else
11:
7) <- Tpeaa
Constant LR
12:
end
13:
Fetch next data batch B, from D sorted-
14:
0: + OptimizerUpdate (o (0,-1, qt B₂)
e.g., Adam
if (T' (k- 1)s, ................................ s, T then
16:
Save checkpoint O,.
end if end for
18:
19: # Stage 3: Model Averaging, e.g., EMA SMA.
20: Let the set of saved checkpoints be (Or-ishse) } k-1
21: EMA: Orinal f- Σi a' 0ia
Et- Gr-
P SMA: Onnal f-
E1 a²
22: return Orinal
Validation Loss
Uniform
Deseond
Learning (x10
Validation Loss
Uniforo Deseend
Learning Rate (x10
2.85
2.56
2.70
2.48
2.55
1.35
1.50
1.35
1.50
0.0
0.5
0.0
0.5
1.0
1.5
Step (x10*)
Step (x10*)
(a) Constant LR schedule.
(b) WSD LR schedule.
Figure 8: benefits a data curriculum using PreSelect scores also diminish. We show the validation loss curves for constant and WSD LR schedules under different data schedules, including uniform, ascending, and descending orders by PreSelect scores. Overall, the ascending curriculum outperforms the uniform baseline under constant schedule, but cannot match under the WSD LR schedule. The final validation loss of the data curriculum is higher than that of the uniform-ordering haseline, likely because the score metrics are perfectly targeted the validation set.
D LARGE-SCALE CONTINUAL PRETRAINING WITH MULTI-DOMAIN CURRICULUM
Practical pretraining scenarios typically involve datasets spanning multiple domains, including nat- ural language, code, and mathematical content (OLMo et al. 2025 Yiwen et al., 2025 Allal et al. 2025). To integrate curriculum learning into this of our CMA/CDMA framework that accommodates heterogeneous data sources. Due to computa- tional constraints and practical deployment considerations, evaluate approach in continual
setting, propose adaptation
23

---

Published as a conference paper at ICLR 2026
Table 6: Downstream performance for experiments with PreSelect score ascending data. Our proposed methods (using WA) are highlighted in gray. WA: Weight Averaging (EMA: Exponential, SMA: Simple). LRS: Learning Rate Schedule (WSD: WSD with decay I x10- Const: Constant LR, WSMD: WSD with moderate decay to 1 x 10-3). Core: Average score on the first four, high signal-to-noise tasks (MMLU, ARC-c, ARC-e, CSQA). Both Core and Avg. scores are annotated with a subscript indicating the performance change relative to the baseline (first row). Subscripts
bold green indicate an improvement of 0.5, light green an improvement of 0. and red
decrease.
WA
Order
LRS
MMLU
ARC-c
ARC-e
CSQA
Core
OBQA
PIQA
SIQA
Wino.
Avg.
X
Ascend
WSD
31.12
35.79
57.89
48.81
43.40
41.00
71.82
46.21
58.41
48.88
EMA
Ascend
Const
31.85
37.46
61.05
49.39
44.94.1.54
38.40
70.51
45.34
55.33
48.67
EMA
Ascend
31.98
39.80
61.93
49.39
45.77-237
39.60
70.78
45.60
55.96
49.38.0.50
SMA
Ascend
WSMD
31.99
39.46
62.11
50.04
45.90.2.0
39.40
71.06
46.06
55.96
49.51.aas
Table 7: Downstream performance for experiments on WebOrganizer dataset (Wettig et al. 2025
Our proposed methods (using WA) are highlighted in gray. WA: Weight Averaging (EMA: Expo- nential, SMA: Simple). LRS: Learning Rate Schedule (WSD: WSD with decay to 1 x 10-5, Const: Constant LR, WSMD: WSD with moderate decay to 1 x 10-3). Core: Average score on the first four, high signal-to-noise tasks (MMLU, ARC-c, ARC-e, CSQA). Both Core and Avg. scores are annotated with a subscript indicating the performance change relative to the baseline (first row). Performance changes are color-coded: bold green (2 0.5 improvement), light green (> 0 improve- ment), and red (decrease).
WA
Order
LRS
MMLU
ARC-c
CSQA
OBQA
PIQA
SIQA
Wino. Avg.
Uniform
WSD
28.92
34.45
47.72
47.83
39.73
36.60
72.03
43.76
56.67
46.00
Ascend
WSD
29.09
32.78
51.58
47.42
40. 40.22..0.9
38.00
72.14
44.73
55.09
46. 45.35
EMA
Uniform
WSMD
28.28
34.11
47.89
48.81
20.7
39,20
71.65
43.76
55.56
1640.16
EMA
Ascend
WSMD
28.56
31.10
50.88
48.89
39.86..0.13
39.60
71.44
44.11
56.75
46.42,00.2
EMA
Uniform
Const
28.03
33.44
47.72
47.42
39.15
40.60
70.78
43.65
$5.72
45.92
EMA
Ascend
55.09
48.89
41.60,1.7
38.40
71.00
55.41
47.06.1..6
pretraining setup, where training resumes from base phase checkpoints and proceeds with our cur- riculum strategy. To validate the generalizability of our method, we scale to 3.2B parameter model and conduct experiments with over 150B tokens during the continual phase, comparing against es- tablished baselines.
D.1 MULTI-DOMAIN CURRICULUM LEARNING FRAMEWORK
Real-world pretraining datasets comprise diverse domains with distinct characteristics, making challenging to define a unified quality metric across all data sources (OLMo et al.] 2025 Yiwen al. 2025 Allal al.) (025). To address this limitation while preserving the benefits of curriculum learning, we introduce a multi-domain extension of our CMA/CDMA methodology.
Our approach employs three- stage pipeline designed to maintain both within-domain eurriculum ordering and stable cross-domain mixing ratios:
1. Within-Domain Ranking: Following data preprocessing and resampling (Wettig et al., 2025| Ye et al. 2025). we independently sort samples within each domain using domain- specific quality metrics. Lower ranks are assigned lower-quality samples, establishing an ascending curriculum within each domain.
2. Rank Rescaling: We transform domain-specific rankings to unified global scale. For sample from domain A with within-domain rank FA, the rescaled global rank is computed as:
N NA
Retnball (I = TA
where NA represents the sample count in domain A, and Nooal denotes the aggregate sam- ple count across all domains. This normalization ensures that the rankings of different domains aligned in the same numerical metric.
24

---

Published as a conference paper at ICLR 2026
Algorithm 2 Multi-Domain Curriculum Construction
Require: Domain datasets D1. ..., D2 with domain quality metrics
Ensure: Multi-domain curriculum dataset
Nuvtal t Ei= [2
Compute total sample count
2: for each domain D; do
3: Sort D, by domain-specific quality metric (ascending)
Within-domain ranking
4: Assign ordinal ranks ri(ar) E [1, D:l] to each sample v E D;
5:
Compute rescaled ranks: R(x)) <- (x) x New for all a E D;
6: end for
7: U V- U
b Combine all domains Global interleaving
8: Sort U by rescaled rank R(x) in ascending order
9: return sorted U
3. Global Interleaving: After rescaling, we merge all domain datasets and sort the combined collection by the computed global ranks in ascending order. This produces globally- ordered curriculum that achieves three key properties:
(1) Preservation of within-domain quality- based ordering
(2) Proportional interleaving of samples across domains according to their mixing ratios
(3) Stable domain mixture ratios maintained throughout training
This methodology ensures that higher-quality samples from all domains prioritized during train- ing while maintaining the intended domain distribution. The process can be efficiently implemented using distributed computing frameworks like Spark, making it practical for large-scale pretraining with heterogeneous data sources. Algorithm 2 formalizes this procedure, providing principled approach for constructing multi-domain curricula that balance quality- musee ordering with distribu- tional requirements.
D.2 EXPERIMENTAL SETUP
Table Base Phase Data Mixture
Dataset
Token Count (B)
Ratio
DCLM
608.5
83.51
Fineweb-C
91.8
12.60%
Starcoder
19.0
2.61%
MegaMath
9.3
1.28%
Total
728.7
100%
Table 9: Continual Phase Data Mixture with Top-k Selection
Dataset
Token Count (B)
Count Ratio
Top-K Ratio
Score Metric
DCLM
83.9
53.77%
0.2
fasttext score
Fineweb-C
23.9
15.34%
0.2
fineweb score
Starcoder
38.9
24.95%
0.2
max stars count
MegaMath
9.3
5.94%
0.4
duplicate count
Total
156.1
100%
To evaluate the effectiveness of our multi-domain curriculum learning approach, we conduct large- scale continual training experiments with 3.2B parameter model and over 150B tokens. Our data spans four domains: deduplicated DCLM Bascline (Li al. 2024). Fineweb-Edd Edu-Chinese- V2.1 Yu et al. 2025). StarCoder (L.i et al. 2023), and MegaMath (Zhou et al. 2025)
25

---

Published as a conference paper at ICLR 2026
1.0
Last Checkpoints (Close
Learning Rate (x10-3)
0.8
(Uniform Data Const LR)
Base Phase
0.38
Continual Phose
0.36
0.6
875B
880B
885B
0.4
Base Phase (Uniform Const LR)
Bascline: Uniform WSD LR
0.2
CDMA: Curriculum + WSMD LR
CDMA Model Average Range (Bold)
Cominnal (729B lokons)
*
CDMA Checkpoints
Total: $84 takens
0.0
0
100B
200B
300B 400B
500B
600B
700B
800B
Training Tokens
Figure 9: Learning rate and data schedules for continual pretraining with a 3.2B model. The base phase (729B tokens) uses uniform ordering, while the continual phase employs either uniform data ordering with decay to 10 (baseline) or our multi-domain curriculum with decay to 3.67 x 10 and EMA averaging of the last six checkpoints (a 0.2). Corresponding benchmark results presented in Table 10
Table 10: Benchmark results comparing baseline and our CDMA approach with multi-domain cur- riculum on 3.2B model after continual pretraining. Subscripts bold green indicate an improve- ment of > 0.5, light green an improvement of 0, and red a decrease.
Method
GSM8K
MBPP
CEval
MMLU
ARC-C
ARC-E
CSQA
BoolQ
Core Avg
Buseline
24.64
38,91
40.83
48.53
51.86
78.13
67.32
71.38
52.70
Ours
0.08
38.13
0.78
42.12,
0.44
55.25,33
77.95
0.18
68.39.1um
74.77.339
53.79,1.0
The base phase comprises approximately 730B tokens with uniform sampling, as detailed in Table For the continual phase, we resume training from the final base phase checkpoint and employ refined data mixture (Table We use Fineweb-C to denote Sineweb-Edu-Chinesse
for brevity.
In the continual phase, we implement ity metrics: text quality scores (DCLM Baseline, (StarCoder), and duplicate frequency (MegaMath).
multi-domain curriculum using domain-specific qual- Saa---duu-hhiee--ea GitHub star counts
We employ the WSD learning rate schedule throughout training with a peak learning rate 1 10-3 and batch size of 2048. The bascline method applies linear decay over the final 84B tokens to 10 while maintaining uniform data ordering. In contrast, our CDMA strategy decays to 3.67 x 10-4 and maintains this learning rate, followed by exponential moving average (EMA) over the last six checkpoints with a = 0.2. Each checkpoint interval corresponds to approximately 1.67B tokens. The learning rate schedule and training phases are visualized in Figure 9
D.3 EXPERIMENTAL RESULTS
As shown in Table 10 our CDMA strategy with multi-domain curriculum consistently outper- forms the baseline in large-scale continual pretraining. We observe significant improvements across multiple capability dimensions: general knowledge (MMLU: +0.4 44), commonsense rea- soning (CSQA: +1. 07), reading comprehension (BoolQ: +3. 39), and complex reasoning (ARC- Challenge: +3. 39). Notably, our approach also demonstrates strong performance on Chinese lan- guage understanding (CEval: +1 29), indicating generalization of our approach across different linguistic domains with appropriate quality metrics.
We use duplicate count proxy for importance, under the assumption that frequently repeated samples may contain valuable content.
26

---

Published as a conference paper at ICLR 2026
Table 11: Performance comparison across learning rate schedules (Constant, Cosine) and data orders (Uniform, Ascend, Descend) on various downstream benchmarks. Results show that descending-
order curricula perform worse than uniform ordering, and the performance gap narrows when LR
decay is applied.
LR Schedule
Order
MMLU
ARC-c
ARC-
CSQA
OBQA
PIQA
SIQA
Wino.
Avg.
Uniform
30.30
30.43
55.61
49.80
44.60
70.29
45.19
56.20
47.80
Constant
Ascend
31.14
39.46
61.23
49.96
43,00
70.51
43.14
56.51
49.37
Descend
29.43
33.11
45.96
45.21
41.00
69.86
44.98
56.75
45.79
Uniform
30.49
38.13
59.47
49.14
42.20
71.87
45.19
56.51
49.13
Cosine
Ascend
30.80
39.80
59.12
S1.27
42.60
71.55
45.65
57.06
49.73
Descend
29.51
34.11
52.98
48.81
42.60
72.42
45.45
55.17
47.63
The overall core average improvement of 1.09% represents a meaningful advancement at this scale. However, we note limited gains in mathematical reasoning (GSM8K: +0 0.8) and a slight regression in coding capability (MBPP: -0. 78). The relatively lower mixing ratios can limit the benefits in these domains. Besides, we hypothesize that these domains may benefit from more sophisticated quality metrics than the preliminary measures (e.g., GitHub star counts) employed in this work. This direction presents an opportunity for future research.
E DISCUSSION
E.1 COMPARISON WITH RELATED WORK
In this section, we situate our findings in the context of prior research curriculum learning and discuss our interpretation compared with previous work on the loss landscape. We discuss how experimental setup differs from related approaches and how findings elucidate the interplay between learning rate (LR) scheduling, data ordering, and data quality.
Comparison with Prior Curriculum Learning Studies. Previous work curriculum learning for large-scale language model (LLM) pretraining has often overlooked the interaction between data ordering and the learning rate schedule, typically adopting cosine LR schedules with low peak values on the order of 10 (Wettig et al. 2024, Dai et al. 2025 Zhang et al. 2025). For example, Wettig et al. (2024) reported a modest 0.6% improvement with a low-to-high quality curriculum but also found counterintuitive 0.5% improvement with reverse (descending-order) curriculum. Two factors may explain this paradox. First, low-peak-L.R regime, LRR decay can prematurely reduce the effective learning rate, narrowing the performance gap between curriculum and reverse- curriculum orders. Second, the data-quality metric itself may lack self-consistency, yielding noisy rankings.
contrast, our experiments demonstrate a consistent performance drop for the reverse curriculum- especially under constant LR schedule-and smaller but still negative effect under schedules with LR decay (Table [T]. These results indicate that our data-quality metric produces more self- consistent orderings and that LR decay indeed diminishes the benefits of curriculum. While direct quantitative comparison is not possible due to differences scale and benchmarks, our best configuration (CA) achieves a relative improvement of over 2.7% compared to compara- ble baseline (Cos+ Uniforr)), demonstrating a substantially stronger effect than previously reported.
Limitations of the Data Folding Strategy. To address the limited gains of vanilla data curricula, recent work introduced the folding strategy, which divides the dataset into several consecutive folds and applies sorting within each stage (Dai et al.) 2025) Zhang al. 2025). Following [Dii et al. (225)), we tested the three-fold configuration they identified as optimal. These experiments are conducted with a 0.5B model. We replicate their observation that folding improves performance under low peak LR (1 x 10-4) but find that this benefit diminishes-ane and even reverses-unde a higher peak LR (3 x 10 10-3) 3), as shown in Table 12 The higher-LR setting shows better overall performance than the low-LR setting in experiments. These findings, well as results of 1.5B models in Section 2 suggest that folding is not robust across different model scales or learning- rate settings. Its apparent gains may reflect compensation for suboptimal L.R schedule rather than
27

---

Published as a conference paper at ICLR 2026
Table 12: Effect of the folding strategy under different peak learning rates. The benefit observed at low LR (1 x 10= diminishes or reverses at higher LR (3 x 10 "), indicating the limited
robustness of folding across scales.
Order
Strategy
MMLU
ARC-c
ARC-e
CSQA
OBQA
PIQA
SIQA
Wino.
Avg.
Uniform
1 10-4
25.70
28.43
37.72
34.32
30.20
61.81
40.99
50.83
38.75
Ascend
Sorting Folding
1x 10-4
26.57
28.76
38.42
35.63
28.80
61.97
41.40
50.36
38.99
Ascend
1x 10
25.69
29.43
38.77
35.22
32.20
61.43
40.99
50.04
39.22
Uniform
3 10-a
28.68
33.78
50.35
45,95
36,60
68.66
43.65
53.35
45.13
Ascend Ascend
Sorting Folding
3 10-a
27.78
37.12
47.89
44.47
37.40
67.85
43.30
55,80
45.20
3 10
28.33
33.11
48.25
43.82
38.80
69.21
43.76
52.88
44.77
fundamental advantage. The limitations of the folding strategy and the strengths of our design underscore the importance of jointly considering curriculum design and LR scheduling.
Interpretation via the Loss Landscape. Our interpretation aligns with the river-valley model of the loss landscape (Wen et al., 2025). which characterizes optimization as progress along two pri- mary directions: the signal (river) direction, where loss decreases gradually, and the noise (valley) direction, where loss oscillates sharply. We extend this framework by positing that data quality in- fluences the gradient's components: high-quality data assumed provide stronger, more stable signal direction and less noise, whereas low-quality data provide a weaker signal and induce more noise. In the context of data curriculum, as data quality increases. the update direction becomes more dominated by the stable signal component. This mechanism facilitates faster convergence, as observed in our experiments (Figures [(a) and [ta)). In the river-valley model, when LR decay applied, the optimizer settles toward the valley floor, reducing noise but also slowing progress along signal direction, thus underusing the signal from high-quality data.
28

---

Published as a conference paper at ICLR 2026
F PROOFS IN SECTION 4
In Section we analyze the bounds of expected loss under four different optimization cases:
Uniform Sampling Learning Rate Schedule.
2. Ascending Data-Ordering Practical WSD Schedule.
3. Ascending Data-Ordering WSMD Schedule.
4. Ascending Data-Ordering Stochastic Weight Averaging (SWA).
In the following, we give the proof of their corresponding theoretical claims we mentioned Sec- tion 4 one by one.
Lemma F.1. Consider the uniform sampling. for any learning rate schedule E such that 0 S 7) S 1, and the parameter initialized at (L. 0), holds that
min [(M; E) = S2(L²).
B
Proof. We first consider the update rule of SGD the optimization process on the x-axis as
wi (1) (1) no (w (1) =(")).
Then, taking the expectation over the randomness in SGD and the data generation gives
Elw; ( (1 n1e 11) (1) + Th 1x (
(M 1)d
(1 n1: n1) 2 7Tt
(M - 1)d
The last inequality holds because [w;-1] out the lower bound for the expected loss
(1)
(M-1)d can be shown by induction. Thus, we write
B([((r:)] E B|] 2 (1)) 2 (E(we(")) (
The above equation completes the proof of Equation
For Case 2 and Case 3, we give a more general lemma, for which the conclusions for Case 2 and Case are direct corollaries.
Lemma F.2. Consider the Ascending data-ordering, and class of WSD learning rate schedules with the following formula
170 for 5 S M -To +1
1)
L-(M-Ta) for M +
where To w M
To = O(M) and no = 1
2. it holds for any learning rate schedule E. with
the above formula that
((M 6 Tjd2
To
Proof. We write out the update rule on the x-axis in the Ascending data- data-ordering case
m (1) (1 ne 5 + m:a (M (M+++1)
Using the above update rule, we can get the expression of w M
(1)
as
M
M
M
(1)
M
= II
(1 - n
M
+
Σ
II (1
n n)
(MS+++1)
t=M +2
i-M
4212-11t1
29

---

Published as a conference paper at ICLR 2026
Then, plugging in the formula of the learning rate schedule gives
M
(1- t
M
M
WM
(1)
II
(M
To)
WM
(1)
To+1
+
Σ
II (1 7y)ma
(M-++1)
t=M-
To++2
i= M
Tu+2 j=++1
To
M
1
(1)
Σ
(M-++1)
W
M-To+1 +
11
k
To
k=2
i= +2
1
(I)
Σ
(M-++1)
To
M
T6
+
To
F
To+2
The above equation uses the following fact
(1 ) (1
... (1
Tu
To
i
i
To To
1
To
To
To
where 2 Siis To. Furthermore, we can find that
M-To+1
M
M-
M- To
W (1)
II
(1
Σ
II
(1
ml M
(M-++1)
T1
[=1
t=1
j=++1
M ++
1
Σ
(M (M-++1)
2
Wi
+ To
M-
141
and then
M
w
Σ
1
M M-i+t) (1)
w M-To+1
To
To
+2
M-T++1
1
(M-i+1)
1
1
(1)
(M
(1-1+1)
To
To
2M
To
100
To
M
i+1
11
+2
To-1
M
1
(1)
1
6
To
2M To+II
W0
To
2 To
1=1
i=Zo
Given that 21 = -8- 1ci-1)m 1)d, and M
To O(M), To ((), we can find that
(1) 1 Md
1 To-1
1
w WM
(i 1)d +
1)d
To
2 TT++1
To
TB
To
i=1
T=
Md
(To
1 - 2)d
1
1
(i
1)d.
MTo4-To+I
2T
T2
2
To
T=
Now we analyze each term:
First term:
T:2 Po+i = o(d) since M To = -((M and 2M7 grows exponentially.
Md
Second term:
(T-1))(o-2%) (TTd)..
27
30

---

Published as a conference paper at ICLR 2026
Third term: Let - Tu. then
M
M-To
d (i - 1)d =
Σ
(To
--)
T T=
21-T0
2
j=0
M M-T
To
d
(To 1)
Σ
1
Σ
T
21
5=
1=0
d
2Tod
d
d
(2 1) +))++
T
T
To
To
(1) Therefore, was (Tod) + θ (
(Thdd) since To w(1)
Thus, the expected loss on the x-axis follows
B( (((Tod))) ((T)da)
Similarly, we write out the expected loss on the y-axis. Note that for w the update rule is:
To
2- 12 + o(d)
M
To
(6) Since 22
Uniform(-LA (LL, L) and are independent, have:
To-1
Σ
Ei(22
)
(1)
1
E((x22 (1)
To
To
22 To)
i=To
TO (To-1)
= θ
3
To
The above equation completes the proof. Specifically, taking To M 10.9 and To O(M3)
gives the results in Equation (2) and Equation
In the end, we show how a simple SWA method can beat the practical WSD schedule, which is stated in Theorem 4.1
Proof of Theorem 4.1
We consider the constant learning rate no S 1 and ascending dada-ord
The SGD update is:
wi (1 wo + 70
(M (+++1
The solution to this recurrence is:
w. = - 10-0o''00 + no Σ (1 (1-n))* --i (MM--1))
We consider the average of the last weights:
WM
WM-k.
k=
Substituting the expression for W.s
M-k
WM
nu)
M R
wD
+ no
(1 (1-m)) M (M---)
k=0
M
)oo)
M
ww
+ 770
(1- M- (M8-
k=0
31

---

Published as a conference paper at ICLR 2026
Changing the order of summation in the second term:
M
(M min(n-1.M--) 1.M-
Σ Σ
Σ
(1 10) k-i x!M-1t1))
k=0 ==1
k=
For i $ M the inner sum is:
1/00 M- (1-10) 1 (1 -770"
17u
k=
For >> M the inner sum is:
M-
M-i 1 -
(1 170)
M 6+1
70)
M-R=i=(1
M
170
k=
Thus, have:
M M-
WM
170)
71
k=
1
M
-
- 170) M- 11 (z- (M--11) -i+1)
+
n
i=1
M
(1 -1- 1o) M ([1 - (1 '70)) M MM++t (M (ot
n
M=
Now we analyze the x-component WM Note that #1 (1) (i 1)d and L/M. The first term
(1)
negligible since (1 -10) M-k decays exponentially. For the second term, when i S M n, (MM2/3), this term is exponentially small. The main contribution comes from the third term where i > M n, i.e. the last data points. In this range,
have (1 7o) M-i < (1 71)" Since
(1 7o ) M is not small, and we have:
M
[1 (1 (1- ( (M-++1)
W M
n
im
(1 - no>paat0
)=1
<
(
1 Σ (j 1)d=
(n- 1)m
d = O(nd) = θ
(nL
2n
M
1=1
j=1
Since n ((M2/3), we have
w
M
(1)
= (LL/M//3) so:
1
E (100))1
M2/3
For the y-component ( note that x2 is:
Uniform - L) are independent. The variance of w M
(2)
n
Var
(2)
Σn
(1 - 7o)'
2
n
j=1
(1) 1
L2
L
L
n
Var(z
n2
n
3
=
θ
= θ
M2/3
j=1
Therefore, the total expected loss is:
E[[(wn)] E|
Ō -0 (M-
This completes the proof.
32


Reviews:
[{"review_id": "GcdfCWqYkg", "reviewer": "Reviewer_vPHj", "summary": "This paper investigates the overlooked interaction between data curriculum strategies and learning rate decay in LLM pretraining. The authors argue that when high-quality data are placed at the end of a curriculum schedule, the model fails to fully learn from these data since the LR has already decayed. To address this issue, the paper proposes Curriculum Model Averaging (CMA), which combines a quality-based curriculum with model averaging, replacing LR decay with CMA to focus the final model on high-quality signals. The authors also proposed CDMA, which combines CMA with LR decay. Experiments on a 1.5B-parameter model trained on 30B tokens show that CMA/CDMA consistently improve both validation loss and downstream task performance, especially on Core benchmarks (max.+2.50%).", "questions": "1. How does CMA/CDMA perform on larger models (e.g., >=7B) or longer training (e.g. 100B+ tokens)? It would be better if you provide at least one larger-scale validation experiment to improve the generality of your claims.\n2. How did you tune the hyperparameters or select their specific values?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies a realistic but previously underexplored issue -- the negative coupling between LR decay and quality-based curricula. The authors proposed a simple yet effective method, using checkpoint averaging to counteract LR over-decay, without additional retraining or architecture changes. The approach can be easily integrated into standard pipelines, and is computationally inexpensive. Theoretical modeling and gradient trajectory visualization illustrate how the proposed method alleviates the loss of learning signal caused by LR decay in curriculum learning.", "weaknesses": "Results are mostly at 1.5B model parameters and 30B tokens. It remains uncertain whether the same trends hold for much larger models (e.g., 7B–70B) or longer training runs. The paper also lacks exploration of hyperparameters for checkpoint averaging (decay factor, checkpointing interval, number of checkpoints) or justification for the selected hyperparameter values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dChOlIR6UE", "reviewer": "Reviewer_qZNE", "summary": "This paper investigates a critical and often-overlooked issue in curriculum-based LLM pretraining: the detrimental coupling between the data curriculum and the learning rate (LR) schedule. The authors diagnose that standard LR decay schedules, which are designed to reduce noise and stabilize training, conflict with the goal of curriculum learning (CL), which places the most valuable, high-quality data at the end of training. They frame this conflict through the lens of the LR's \"dual role,\" acting simultaneously as an update step size and an implicit importance weight for data. By decaying the LR, standard methods effectively \"waste\" the best data by learning from it with minimal update steps.\n\nTo resolve this, the paper proposes to decouple these two roles. Their primary solution, Curriculum Model Averaging (CMA), replaces aggressive LR decay with a constant high LR during training, relying on model averaging (e.g., EMA or SMA) over the final checkpoints to ensure stability. They further propose a co-designed strategy, CDMA, which combines a moderate LR decay with model averaging. Through extensive experiments on a 1.5B parameter LLM, the authors demonstrate that their proposed methods outperform standard pretraining baselines, especially when compared to the widely-used cosine schedule. The work identifies an \"optimal area\" of moderate decay where co-designing the curriculum, LR schedule, and model averaging yields the best results, highlighting a previously underexplored optimization regime.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The paper's primary strength lies in its clear and intuitive diagnosis of a fundamental conflict in modern pretraining. \n\n2.  The authors provide strong empirical validation for their central thesis. \n\n3.  The proposed solutions, CMA and CDMA, are not ad-hoc but are direct, logical consequences of the initial diagnosis.", "weaknesses": "1.  While the paper's framing is compelling, the core ideas—the conflict between CL and LR decay, and the use of model averaging for stabilization—are not entirely new. The provided literature survey highlights several precedents. For instance, Weinshall & Amir (2020) theoretically showed that optimal CL requires non-decaying LRs, and Jiang et al. (2021) empirically demonstrated that LR decay undervalues late-stage data. Similarly, model averaging techniques like SWA (Izmailov et al., 2018) have long been proposed as alternatives to LR decay. The paper could strengthen its contribution by more explicitly positioning its work against these specific precedents.\n\n2.  The experiments are missing a key and highly relevant baseline: training exclusively on a high-quality data subset for the same computational budget (e.g., by filtering out the bottom 80% of data and repeating the top 20%). Without this comparison, it is difficult to disentangle the benefits of the curriculum's data ordering from the benefits of simply focusing compute on high-quality data. If this simple filtering baseline performs comparably to CMA/CDMA, it would challenge the necessity of the curriculum itself.\n\n3.  The paper observes that a descending (high-to-low quality) curriculum performs poorly but misses an opportunity for deeper analysis. This ordering, when paired with a standard LR decay, seems intuitively synergistic (high LR on high-quality data, low LR on low-quality data).\n\n---\n\n[1] Weinshall, D., & Amir, G. (2020). On the Theory of Curriculum Learning. *Advances in Neural Information Processing Systems*.\n\n[2] Jiang, L., et al. (2021). Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. *International Conference on Machine Learning*.\n\n[3] Izmailov, P., et al. (2018). Averaging Weights Leads to Wider Optima and Better Generalization. *Uncertainty in Artificial Intelligence*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xMcRvquPpJ", "reviewer": "Reviewer_QrZN", "summary": "This paper argues that **curriculum-based pretraining** (train low→high quality) quietly clashes with the usual **decaying learning-rate (LR)** schedules: when the best data finally arrives (late in training), the LR is tiny, so the model barely learns from it. Experiments with a **1.5B** model on **30B** tokens show that curricula look strong under a **constant LR**, but their gains **shrink under cosine/WSD decay** (Figure 1–2). The authors propose **Curriculum Model Averaging (CMA)**: keep a high/constant LR during training, then do **model averaging** over the last checkpoints (SMA/EMA/WMA) to stabilize. They further recommend **co-designing** a *moderate* LR decay with model averaging (**CDMA**), which outperforms standard decay with uniform order and also beats “decay + curriculum” alone, with especially clear gains in **mid-training** when only some high-quality data is available (Table 1–2; Figure 5). A simple theory sketch supports why curriculum + averaging can keep strong updates from high-quality data while still reducing noise.", "questions": "Do the CMA/CDMA gains persist for **larger models** (e.g., 7B/13B) and for other **quality metrics** (beyond DCLM/PreSelect)? Please include at least one bigger-model run.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper **clearly diagnoses** an intuitive but under-discussed coupling: high-quality data arrives when LR is tiny, dulling its impact. The **constant-LR** comparisons make this visible. \n- **CMA** is **simple and actionable**: keep LR high, then **EMA/SMA** the last checkpoints; implementation details are explicit.  \n- **CDMA** (moderate decay + averaging) finds an **under-explored optimum** that beats standard decay+uniform and decay+curriculum, especially in **mid-training**.", "weaknesses": "- The study focuses on **one model size (1.5B) and a 30B-token corpus**; it’s unclear whether the same sweet spots hold for much larger models or different corpora/metrics. \n- **Baselines are limited** for some comparisons (e.g., stronger **learned curricula** or **adaptive/variance-aware** data-selection methods aren’t included), so it’s hard to judge competitiveness against the latest dynamic strategies. \n- The **mid-training** setting is promising but still tied to the specific phasing and quality signals here; more domains or public corpora would help establish external validity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)