Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Authors: Kazuki Yano
Shun Kiyono
Sosuke Kobayashi
Sho Takase
Jun Suzuki
Venue: ICLR 2026 Poster (2026)
Keywords: Learning rate schedules
Large language models (LLMs)
Primary area: foundation or frontier models, including LLMs

Abstract: We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT).
Decay-based learning rate schedulers are widely used to minimize pre-training loss.
However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored.
In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay.
Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training.
The result also holds across different regimes with mid-training and over-training.
Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability.
These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability.
Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks.

Published as a conference paper at ICLR 2026
PRE-TRAINING LLM WITHOUT LEARNING RATE DECAY ENHANCES SUPERVISED
Kazuki Yano', Shun Kiyonoł, Sosuke Kobayashi', Sho Takase Jun Suzukit
Tohoku University
SB Intuitions yano. kazuki 8dc tohoku. Larralllb-reearrreppp..ooooo..
ABSTRACT
We investigate the role of learning rate scheduling in the large-scale pre- pre-training of large language models, focusing on its influence on downstream performance af- ter supervised fine-tuning (SFT). Decay-based learning rate schedulers widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this pa- per, we examine Saarmu--Stablc--0
(WSO), which maintains a constant learn- ing rate after warmup without any decay. Through experiments with IB and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based sched- ulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape anal- ysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO hances their adaptability for downstream tasks.
INTRODUCTION
Learning rate (LR) scheduling is arguably one of the most critical yet operationally challenging pects of large language model (LLM) pre-training. Although Cosine decay has been conventionally employed in numerous models (Brown et al., 2020; Le et al., 2022; Touvron et al., 2023a), it has
proven inflexible in recent training paradigms such as continual pre-training, as it requires heuristic tuning of the LR from the decayed value (Hägele al., 2024 Ibrahim et al., (024). To address this inflexibility, recent studies have introduced "Varmup-Stable Decay (WSD), which keeps the LR
constant through most of pre-training and decays it only briefly at the end (Hu et al. 2024 Liu al.,
2024a Wen et al. 2025).
These previous studies, regardless of the details of the design choices, decayed the LRs to optimize the performance of pre-trained models. However, the more critical factor for real applications is the performance after post-training, such supervised fine-tuning (SFT). Drawing the findings of (Sun & Dredze (2025) and (Springer et al. (2025), which show that a strong pre-training model does not necessarlly imply superior performance after SFT, it is questionable to schedule LRs the decayed value based on pre-training performance.
In this study, we provide a comprehensive empirical investigation of LR schedulers during pre- training in terms of performance after SFT. In particular, we examine an underestimated schedul- ing, Warmup-Stable-Ont
(WSO), which removes the decay phase from WSD and maintains con- stant LR to the end. We show that WSO consistently achieves superior performance after SFT compared to decay-based schedulers, through experiments on IB and 8B models (Figure IT. Fur- thermore, we demonstrate that WSO is also effective under modern training paradigms, including mid-traine mid-training (OLMo et al. 2024 Meta 2024c) and over-training (Sardana et al. (2044) Gadre et al, 2025).

---

Published as a conference paper at ICLR 2026
Warmup- Stable Only (WSO)
55.0
Leaming Rate
Docay WSD
53.6
Decay
53.9
Decay Cosine
53.8
Training Steps
Average SFT Benchmark Score
Figure 1: Learning rate schedulers used in pre-training and their impact on performance after su- pervised fine-tune fine-tuning (SFT). Varmup-Stablee (WSO), which removes the decay phase, achieves the highest performance after SFT.
To understand why WSO yields superior SFT performance, we draw on insights from the transfer learning literature (Ju al. 2022 Liu et al. 2023), which suggest that models in flatter regions of the loss landscape tend to exhibit better adaptability. Through an analysis of sharpness values, we show that models trained with WSO reside in flatter regions than those trained with other decay-based LR schedulers, and are therefore more adaptable to post-training tasks.
Our contributions tently outperforms decay-based schedulers on downstream tasks after SFT, with comprehensive evidence across 1B and 8B models and diverse evaluation benchmarks. (2) We show that WSO similarly benefits mid-training and over-training scenarios, achieving superior SFT performance compared conventional decay-based schedulers. (3) We reveal through loss landscape analysis that WSO preserves flatter minima than decay-based schedulers, explaining why models trained with WSO achieve better performance after SFT.
follows: (1) We provide the systematic demonstration that WSO consis-
2
PRELIMINARIES
Recent LLMs are typically built with staged training scheme. The most common and fundamental training pipeline consists of two stages, namely pre-training and post-training. In this section, we de- scribe these training stages and review the LR schedulers commonly employed during pre-training.
Pre-training. Pre-training forms the foundation of LLM development, where models learn general language understanding from massive text corpora by minimizing the next-token prediction loss. Recently, pre-training has sometimes consisted of multiple stages; standard pre- pre-training and mid- training (OLMo et al. 2024). We describe mid-training in detail later (Section 2.2 and conduct experiments with both the standard pre-training and the multi-stage setup.
Post-training. Post-training adapts pre-trained models to target tasks, enabling them to follow human instructions and avoid generating harmful outputs. Post-training includes techniques such as supervised fine-tuning (SFT), preference tuning (e.g., DPO (Rafailov et al., (023)), and RL-based alignment (Ouyang al., 2022). While post-training could be multi-stage process with many design choices still under active exploration. SFT is relatively standardized and serves a core stage. In this paper, we focus SFT the canonical post-training stage and evaluate the performance after SFT"
2.1 TASK DEFINITION
Practically, LLM developers evaluate models multiple stages, selecting the best-performing one as the starting point for the subsequent stage. We define Task,(M) as a function that, for a given LLM M, returns the performance on set of pre-defined tasks used assess the target stage s, where
1 'The computational cost pre- mreitrannin typically much larger than that of other stages, so identifying a better pre-traine pre-training configuration has substantial impact on the efficiency of LLM construction. In this study, we focus on evaluating LR schedulers during large-scale pre-training and characterize the potential of non-decay schedulers based on the performance after SFT. An exploration of complex combinations of LR scheduling spanning multiple post-training stages left to future work.
2

---

Published as a conference paper at ICLR 2026
E (pre, post) denotes the training stage, with pre indicating pre-training and post indicating post- training. respectively. We write [M] to denote the model M2 trained with some configuration and initialization with M, where Mrand indicates model whose weights are randomly initial- ized. Moreover, we introduce Mpre and Mpost to represent the sets models obtained through pre-training and post-training, respectively, with various hyperparameter configurations. typical training pipeline for building LLMs can therefore be expressed as follows:
M pre
arg {Taskpro (M (Mo E.ro
[Mrand])
Mpost.
Task M (Moost Mpree Mrana]))
arg max
Mpost E.Moost
This formulation may lead to a suboptimal solution in terms of the performance of the final model, namely, MMosse since selecting the best- cess--erforming models intermediate stages does not guarantee achieving the best performance in the end. Therefore, conceptually, we would like to consider the following search problem obtain better final model for this training pipeline:
arg maxX Mpost MMpre.Mpart EeMpre..Mpor Taas (Mpost [ [Myand]))
The primary objective of this paper is to empirically examine the search problem by evaluating several LR schedulers during the large-scale training stages that precede post-training.
2.2 FURTHER CONSIDERATIONS
Mid-training. Mid-training has emerged critical intermediate stage in modern language model development, occupying a computational middle ground between large-scale pre-training and task-specific post-training (Meta] [024c] OLMo al., 2024). This stage serves multiple strategic objectives. including domain expansion and long contect extension. For example, OL.M 2 (OLMo) al. 2024) demonstrates performance gains through mid-training on curated high-quality data, es- tablishing this stage as essential component of the modern training pipeline. After introducing mid-training, we can rewrite equation 2 as follows:
M post
arg max
(Taskpost (Mpoat [Muial [Mpre [Mranal])}
(3)
(Mpre: Maia poss (MMM Minie
Over-training. Modern LLMs often trained trillions of tokens, far beyond the Chinchilla compute-optimal regime of roughly 20 tokens per parameter (Hoffmann et al., 2022). This practice trades substantially more training compute for improved inference efficiency deployment. Recent production systems use hundreds to thousands of tokens per parameter (Sardana et al. 2024). While full-scale experiments are costly, Section presents results under such configuration, showing the generality of our main findings.
2.3 CURRENT LR SCHEDULING PRACTICES
In current LLM training practice pre-training uses decay-based LR schedulers with Cosine, Lin- ear, or WSD that reduce LR 0-10% of maximum (Touvron al., 2023a Hu et al.) 2024 Bergsma et al. 2025). Additionally, in mid-training, it is common practice to further decay the LR from the final value reached at the end of the preceding pre-training phase (Meta) 2024c| OLMo et al., 2024). These schedulers are chosen to minimize loss at each respective stage, effectively optimizing Taskpre independently. However, the primary objective should be maximize PaskppostMMoosii the performance after the complete pipeline. Thus, optimizing for Taskpre (asppreeMMeel) may be suboptimal. For instance, recent findings from [Spiinger et al. (2025) and (Sun & Dredze (025) reveal that the better performance after pre-training does not guarantee performance after raise a fundamental question: Is LR decay, which is chosen based on pre-training per- formance, still the best choice when the model will undergo supervised fine-tuning? Our work in- vestigates this question by systematically varying LR schedulers in Mpr and Mmnd to understand their impact on the final objective, i.e., Taskpost MM posst)
3

---

Published as a conference paper at ICLR 2026
2.4 FORMALIZATION OF LEARNING RATE SCHEDULERS
We denote the LR at training step 1j s (t, Apre), where Scheduler specifies the LR scheduler and Opee controls the minimum LR factor in pre-training. For example, the WSD scheduler defined as:
Qmax
Turmpp
S T warmup
1e (t, Orpre)
Tmnax
T Tarrmup
S T stabl
(4)
Tmnax
((1 (1- (Cpre)
Tym
Typ
-antte
Opeee
Taable
1 S TpE
where 7max the maximum LR, Tpe denotes the total number of pre-training steps, T Tarmup the number of warmup steps, and Taable is the step at which the decay phase begins.
To investigate the effectiveness of the LR scheduler without decay, we consider a simple variant of WSD, which we call Varmup-Stablc-Onll (WSO). In this variant, the decay phase is omitted, which corresponds to setting Opr 1.0.
" m (t, pprr
7max
Taampp
< T warmup
(5)
7mmax
T Twarmup
< 1 S Tpe
In experiments, {WSO, WSD, Cosine, Linear]. The detailed formulations for Cosine Cosine (t, apre) and
investigate four LR schedulers:
Scheduler
Linear miniee (t, apree are provided in Appendix
LR Scheduling in Mid-training. We parameterize mid-training schedulers with O'mid (Figure 2 where amidd = 0.0 applics Linear decay to zero while a'iid 1.0 maintains the LR constant throughout mid-training. When combined with Opr 1.0, the configuration of
Leaming Rote
WSD. 18. 1.
Pp- Rai
WSD. 01.00
Laca, 0.1.
Linax, 01.
Clodas, d1.00
Criaoo 011.0
Oyre 1.0 and Qmid 1.0 extends WSO across both pre-training and mid-training stages. The detailed formu- lation for mid-training LR schedulers is provided in Ap- pendix
Timining
Figure 2: Mid-training LR schedulers
with different Orpee and Crdid values.
3 EXPERIMENT 1: TWO-STAGE (PRE- AND POST-TRAINING) SETTING
We investigate whether decaying LRs during pre-training truly benefit downstream SFT perfor- mance.
3.1 EXPERIMENTAL SETUP
Model Architectures. We conduct experiments on two model scales following the Llama 3 archi- tecture family: IB and 8B parameter models (same architecture Llama-3.2-1B (Meta 2024b and Llama-3.1180 (Meta 2024a), respectively). Full details are provided in Appendix
Pre-training Configuration. Models are pre-trained on FineWeb-Edu (Penedo et al. 2024 with
maximum LR 7max 3 10 We investigate three LR schedulers as formalized in Section 2.4 experimenting with WSO (Equation[2.4 WSD (Equation?2 2.4). Cosine, and Linear schedulers (de- tailed in Appendix B For each scheduler, vary the minimum LR factor O're E (0.0, 0.1,
following our notation y SSchedules (t, Opre). Setting O'rre 0.0 corresponds to decay to zero. Recent work by Bergsma et al. (2025) shows that this achieves better pre- pre-training performance. Setting Orgr 0.1 corresponds to decay to 10% of maximum, choice commonly used in practice by Chin- chilla (Hoffmann et al., 2022), Llama 3 (Meta) 2024c) and OLMo 2 (OLMo et al. 2024]. Finally, setting Opre 1.0 corresponds WSO. Further hyperparameter details are provided in Appendix C

---

Published as a conference paper at ICLR 2026
Table I: Relative performance across pre-training (PT) and supervised fine-tuning (SFT). For each model size and each metric. values are differences (A) from the best-performing decayybbsed sched- uler for that metric. Note that WSO could perform poorly after PT but best after SFT. Bold indicates the best performance.
PT Valid
PT Task Avg A
SFT Task Avg A
Model
Scheduler
Opeee
Loss $ A
Warmup-Staale-Onll (WSO)
1.0
+0.071
-1.7
+0.3
0.1
+0.004
-1.5
+0.0
WSD
0.0
+0.000
-1.2
-1.0
0.1
+0.021
-2.0
-0.7
Linear
0.0
+0.016
+0.0
-0.9
0.1
+0.019
-0.1
-0.7
Cosine
0.0
+0.016
-25
-0.7
Warmup-Stable-Only (WSO)
1.0
+0.127
-0.8
+1,1
0.1
+0.019
-0.2
-0.8
WSD
0.0
+0.014
+0.0
-0.3
8B
0.1
+0.013
-1.9
-0.6
Linear
0.0
+0.000
-1.8
+0.0
0.1
+0.009
-2.2
-0.3
Cosine
0.0
+0.008
-2.3
-0.1
SFT Configuration.
We perform SFT using the Tulu-3 SFT mixture We conduct comprehen-
sive LR sweep ranging from 5 x 10-7 to x 10-3 to identify the best hyperparameters for each pre-trained mode
Evaluation. We evaluate models at two stages; after pre-training and after SFT. For pre- trained models, we assess zero-shot performance on standard benchmarks, including question answer- ing (ARC-Easy, ARC-Challenge (Clark et al. (018)) OpenBookQA (Mihaylov et al. 2018), BoolQ (Clark et al., 2019)) and commonsense reasoning (HellaSwag (Zellers al, 2019). PIQA (Bisk et al. 2020). WinoGrande (Sakaguchi et al. 2021)), along with validation loss.
For fine-tuned models, we follow the setup of OLMo (Groeneveld et al. 2024 and evaluate along three key dimensions: instruction-following capability (AlpacaEval (Li et al. 2023)), multi-task language understanding (MMLU (Hendrycks et al. 202I),, and truthfulness (TruthfulQA (Lin et al.
)
To highlight how LR decay affects both pre-training and SFT differently, we present results as relative performance metrics normalized against the best decay-based scheduler for each stage. For pre-training, we report both validation loss and the average accuracy across all zero-shot benchmarks (PT Task Avg). For fine-tuning, we report the average across AlpacaEval, TruthfulQA, and MMLU (SFT Task Avg
3.2 RESULTS
Table shows an inversion in model performance across training stage: For pre-training perfor- mance, decay-based schedulers achieve the best performance with O'eee 0. Specifically, Linear and WSD with Opere = 0 achieve the best PT Task Avg scores for the 1B and 8B models, respec- tively. This result is consistent with existing findings (Bergsma et al., 2025). In contrast, after SFT, WSO achieves the best performance for both model sizes, even though it underperforms decay-based schedulers in pre-tnaining metrics. These results demonstrate that while decay-based schedulers may
ft -oto-oooo 2 xture/
tree/ cree/main
FFull details about SFT are provided in Appendix D
Detailedd evaluation settings are provided in Appendix
Detailed per-task evaluation results for all models provided in Appendix

---

Published as a conference paper at ICLR 2026
Table 2: Relative performance across mid-training (MT) and SFT stages. Values are differences from the best decay-based schedule. WSO throughout both stages yields the best SFT performance.
MT Valid
MT Task
SFT Task
Model
(Pre-training) Scheduler
Opr
Loss 4 A
Avg A
Avg A
Warmup-Stable-On Stable-Only (WSO)
1.0
1.0
+0.062
-0.1
+0,8
1.0
0.0
+0.000
+0.0
+0.0
WSD
0.1
1.0
+0.038
-1.5
-0.5
0.1
0.0
+0.047
-1.7
-1.3
0.1
1.0
+0.053
-2.1
-2.5
Linear
0.1
0.0
+0.058
-3.3
-3.8
0.1
1.0
+0.053
-2,4
-2.9
Cosine
0.1
0.0
+0.059
-3.1
-3.7
Warmup-Stable-Ont (WSO)
1.0
1.0
+0.102
-2.1
+1.1
1.0
0.0
+0.000
+0.0
-1.4
WSD
0.1
1.0
+0.057
-5.0
+0.0
0.1
0.0
+0.081
-5.6
-1.1
8B
0.1
1.0
+0.067
-8.3
-2.2
Linear
0.1
0.0
+0.082
-9.0
-3.7
0.1
1.0
+0.068
-8.0
-3.5
Cosine
0.1
0.0
+0.084
-10.1
-4.1
yield superior performance in terms of pre-training metries, WSO is more effective in the overall training pipeline, including SFT.
4 EXPERIMENT 2: THREE- (PRE-, MID-, AND POST-TRAINING) SETTING
Recent LLM developments (OLMo al. 2024 Meta 2024c add mid-training stage between pre-training and post-training, which makes LR scheduling across stages more complex due to the various combinations of pre-training and mid-training schedulers. We investigate whether using WSO in both pre-training and mid mid-training stages yields better performance after SFT than decay- based schedulers.
4.1 EXPERIMENTAL SETUP
To investigate the effect of LR scheduling during mid-training, we conduct experiments following three-stage training pipeline: pre-training, mid-training, and post-training. We systematically vary the LR schedulers both pre-training and mid-training stages to understand their individual and combined effects on downstream performance. To ensure comparability with recent mid-training work, our setup largely follows OLMo 2 (OLMo al. 2024). representative study of mid-traine mid-training.
Pre-training Stage. We pre-train IB and 8B models using the same architccture and configuration as described in Section We adopt pre- pre-training dataset olmo-min x x-1124 (OLMo et al. 2024) used in OLMo 2. Following standard practice in modern L.MM development (Meta [024c] ML jet al. 2024), we employ four LR schedulers with different minimum LR factors, including WSD, Cosine, and Linear schedulers with Ope 0.1, and additionally WSO.
Mid-training Stage and Learning Rate Schedules. Following OLMo 2 (OLMo et al. (0224, we conduct mid-training on the dolm no-mi x- x-1424 dataset. We investigate the two mid-training strategies shown in Figure 2 with O'idd = 0.0 applying further Linear decay following common practice (Meta 2024c: OLMo al. 2024), and amid 1.0 maintaining a constant LR throughout
mid-training
Further training configurations of mid-training are provided in Appendix

---

Published as a conference paper at ICLR 2026
Table 3: Relative performance after over-training (2T tokens). Values differences (A) from the best-performing decay-based scheduler for each metric. Similar to Section best SFT performance.
WSO achieves the
PT Valid
Task Avg
SFT Task
Model
Scheduler
Olpee
Loss | A
Avg
Warmup-Stable-Only (WSO)
1.0
+0.048
-1.5
+0.7
0.1
+0.004
+0.0
+0.0
WSD
0.0
+0.000
+0.0
-0.3
IB
0.1
+0,02.1
-0.9
-0.5
Linear
0.0
+0.017
-0.4
-0.6
0.1
+0,017
+0.0
-0.4
Cosine
0.0
+0.017
-1.3
-0.3
Table 4: Relative performance after over- wer-training with mid-training (2T 500B tokens). Values are differences (A) from the best-performing decay-based scheduler for each metric. Section 4 WSO yields the best SFT performance.
Similar to
MT Valid
MT Task Avg A
SFT Task Avg A
Model
(Pre-training)) Scheduler
Opr
Olmid
Loss | A
Oarmup-Stahle-Only (WSO)
1.0
1.0
+0.055
-0.3
+1.4
1.0
0.0
+0.000
-1.6
-0.5
WSD
0.1
1.0
+0.033
+0.0
-1.0
0.1
0.0
+0.038
-1.7
-1.2
IB
0.1
1.0
+0.068
-2.2
+0.0
Lincar
0.1
0.0
+0.051
-2.8
-0.6
0.1
1.0
+0,046
-1.8
-0.7
Cosine
0.1
0.0
+0.054
-2.3
-1.2
SFT and Evaluation. For SFT, we follow the configuration described in Section For mid- trained models (before SFT), evaluate standard benchmarks to assess the impact of mid- training LR schedulers, following the evaluation suite used in OLMo (OLMo et al.) 2024)). We select benchmarks that comprehensively assess model capabilities, including reasoning tasks CRC--hallengg (Clark et al.] 2018). HellaSwag (Zellers et al.) 2019), WinoGrande (Sakaguchi et al.) 2021)). reading comprehension (DROP (Dua al., 2019)), and mathematical reasoning (GSM8K (Cobbe et al. 2021)). Following SFT, we assess models using an expanded evaluation suite including AlpacaEval (Liet 2023) for instruction following. TruthfulQA (Lin al. 2022 for factual accuracy, GSM8K (Cobbe et al. 2021) for mathematical reasoning, DROP (Dua et al. 2019) for reading comprehension, AGI Eval (Zhong al.) (2024) for general intelligence capabili- ties, BigBench-Haadd (Suzgun et al. (2233 for challenging reasoning tasks, and MMLU for multitask understanding Similar to Section
we present results as relative improvements compared to the
best decay-base based scheduler.
4.2 RESULTS
Table 2 shows an inversion similar to our pre-training findings R decay-based scheduler with Ope 1.0 and Olmia = 0.0 achieve the best performance. However, SFT performance again shows the opposite trend. WSO achieves the best downstream task performance after SFT, even though underperforms
For mid- mid-traine -trriiiinnn performance, the
best decay-based schedulers mid-training metrics. Additionally, find that introducing decay at any stage reduces SFT performance. Notably, for models pre-trained with decay (aiee = 0.1), avoiding decay during mid-training (amid = 1.0) im- proves both mid-training metrics and SFT performance compared to applying decay.
"The detailed evaluation settings for these henchmarks are deseribed in Appendix Detailed per-task evaluation results for all models provided in Appendix

---

Published as a conference paper at ICLR 2026
WSD (dge 1.0)
Deaay wi WSD (Ap 0.0)
Sharpness(Or (0.)
Decay
($pe 0.1)
Decay wi Linear (Ape 0.00
Decay wi Linear (Ope 0.13
Desay wi Cosine (Opn -0.0)
Decay Cosino (dgn
20000
20000
60000
80000
20000
400000
60000
SIRDO
Pre-training Steps
Tante
Pre-training Steps
Tuzble
Sharpness Pre-training Validation Set
Sharpness on Dataset
Figure 3: Sharpness(O.) during pre-training the model. Vertical line at step Taable indicating where WSD decays LR. Decay-based schedulers (Opr 0 or Ope 0.1) lead to sharper minima, while WSO (Apre* = L.0) maintains flatter landscapes.
These results extend our findings to multi-stage training pipelines, where decay at any stage con- sistently harms SFT performance. WSO, which maintains constant learning rates throughout both pre-training and mid-training, shows the best performance across the overall training pipeline, in- cluding mid-training and SFT.
EXPERIMENT 3: THREE-STAGE SETTING IN THE OVER-TRAINING
To further probe generality, we evaluate third regime with substantially larger training budget. This over-training setting serves test of whether the benefits of WSO persist when training trillions of tokens.
5.1 EXPERIMENTAL SETUP
Pre- and Mid-training, We pre-train models on 2T tokens, which is approximately 100x the Chinchilla-optimal amount of data for this model size, evaluate whether WSO maintains its advantages at this data scale. We use the same datasets as in Sections 3 and + for pre-training and mid-training, respectively. We investigate the same set of LR schedulers additionally conduct mid- mid-ttriiing experiments using 500B tokens, following the same experimental setup as in Section
Section 3 We
Evaluation. We evaluate all LR schedulers using the same methodology as Sections and measuring performance both after pre-traie trainini (or mid-training) and after SFT. Detailed configura- tions are provided in Appendices and D
5.2 RESULTS
Tables and 4 confirm that the inversion observed in Sections 3 and persists even in the over- training regime using 2T tokens. Across all investigated schedulers, WSO (Apre 1.0) consis- tently yields worse intermediate metrics but superior SFT performance compared to decay- schedulers. Similar to earlier findings, decay-basee schedulers achieve better pre-training and mid-training metrics, yet WSO outperforms them after SFT. This pattern holds both for single-stage over-trainin ever-training (Table and when combined with mid-training (Table 4), demonstrating that the ben- efits of WSO are robust across different data scales and training configurations.
UNDERSTANDING ADAPTABILITY THROUGH LOss LANDSCAPE GEOMETRY
To understand why models trained with WSO achieve superior SFT performance, we analyze the loss landscape geometry throughout pre-training phase. As suggested in the transfer learning lit-
8

---

Published as a conference paper at ICLR 2026
erature (Ju et al., 2022 Liu et al. 2023). we focus sharpness key geometric property that characterizes the curvature of the loss landscape around converged parameters.
The relation between lower sharpness and better SFT performance stems from how models respond to parameter updates during fine-tuning. When the parameters of the model lie in a flatter region of the loss landscape, which corresponds to lower sharpness, the model demonstrates superior adapt- ability to downstream tasks (Foret et al. [0021] Li et al.) 2025). The intuition is that the performance of the model remains stable during the parameter updates of SFT. model in flat landscape expe- riences less fluctuation in its loss value when its parameters are updated, which translates to more stable performance. This characteristic is believed confer higher adaptability, as the model can in- corporate new data without compromising its pre-trained capabilities (Andriushchenko et al.) 2023)
There are several ways to quantify sharpness, such as the largest eigenvalue of the Hessian (captur- ing the most curved direction) or the trace of the Hessian (capturing the average curvature) (Dinh) jt al. 2017 Kaur et al. 2023). Following established practice in optimization and generalization studies (Ju et al. [2022] Liu et al.) 2023), we adopt the trace as our sharpness measure, since it provides a scalar summary of curvature across all parameter dimensions.
Definition 6.1 (Sharpness). Let L(;;; D) denote the loss function evaluated on dataset 'D with model parameters 0, E RR At training step ł, the sharpness of the loss landscape at parameters 02 is defined as the trace of the Hessian matrix:
d 8 PL(( D)
Sharpness(O.)) = Tr(H2((O)) =
(6)
00?
where Hc(0,) E [dd is the Hessian matrix of the loss with respect to the parameters at 0ç.
Since computing the full Hessian trace is computationally prohibitive for billion-parameter models, we employ Hutchinson's unbiased estimator (Hutchinson) 1989, Liu et al. 2024b). This method requires only feestann-vector vector products, which can be efficiently computed through automatic differ- entiation. Details of our sampling procedure and computational details are provided in Appendix H
We measure sharpness throughout pre-training on validation sets from both the pre-training dataset and the SFT dataset. Figure shows the sharpness for the 1B model from Section 3 We illustrate vertical line at step Ttable to indicate the point at which WSD decays LR. The figure reveals distinct patterns across schedulers. Specifically, Cosine and Linear schedulers exhibit steadily increasing sharpness as the LR decays, while WSD shows a rise during its decay phase. In contrast, WSO maintains lower sharpness. Across both datasets, models with decaying LRs converge regions with about 2-3x x higher sharpness compared to WSO models. Flatter regions obtained by WSO allow more flexible parameter adaptation during SFT, enabling better downstream performance.
Correlation between sharpness and down- stream adaptability. To provide empirical evidence linking the loss landscape down- stream adaptability, we analyze the correlation between the sharpness of pre-trained models and their subsequent SFT performance. Fig- ure 4 presents the SFT performance plotted against the sharpness measured on the pre- training validation set for the IB model across all investigated learning rate schedulers. The analysis reveals a negative correlation (Pearson r -0.709) between the sharpness of the min- ima and the model's performance after SFT. The WSO scheduler (ape = 1.0) resides in the low-sharpness, segh-performacc region, while decay-based schedulers converge sharper minima with lower SFT scores. While the sample size limited, this pattern is consistent with our hypothesis that preserving flatter minima during pre- training enhances the model's 's adaptability.
$1.5 950
Avg. SPT Store
54.0-
Creac -0.3 Connc/
53.0-
Pramon -0.10
$2.5
8000
20056
30000
40000
30300
Sharpness (fr)
Figure 4: Pre-training sharpness negatively corre- lates with downstream SFT performance.
9

---

Published as a conference paper at ICLR 2026
7 RELATED WORK
Learning Rate Scheduling in LLM Training. LR decay has been considered effective for LLM pre-training, with Cosine decay remaining the de facto standard (Kaplan et al.) 2020 Hoffmann et al. 2022; Touvron et al. 2023b). Recent large-scale studies advocate for even more aggressive decay, showing that Linear decay to zero achieves lower pre-training loss in compute-optimal set- tings (Bergsma et al.] 2025). Warmup-Stabe Stable- -Dccayy (WSD) delays decay until the final phase of training (Hu et al. 2024), while theoretical analysis suggests that decay may confine models to narrow loss valleys (Wen et al., (025)) Some methods attempt to avoid the decay phase through checkpoint averaging (Sanyal et al.] (2024) or model merging (Tian et al. 2025). [ii et al. (2023) in- vestigated learning rate tuning strategies within individual training phases, but did not examine how pre-training LR choices propagate through to post-training performance. While these studies have advanced understanding of LR scheduling within single training phase, they share common limitation: evaluation is restricted to the phase in which the schedule is applied, without considering the downstream consequences for subsequent training stages such SFT.
Relationship to Continual Pre-training and Fine-tuning Studies. Recent work on continual pre-training (CPT) has explored LR scheduling for domain adaptation. Wang et al. (2025) showed that models with higher loss potential achieve lower CPT validation loss and advocated for re- leasing high loss potential versions to facilitate downstream tasks. WWan et al. (2024) proposed a path-switching paradigm for LR scheduling in model version updates, though their experimental setup still applies LR decay before performing SFT. Tissue et al. (2024) introduced a scaling law describing loss dynamiic in relation to learning rate annealing; however, they explicitly note that post-training scenarios involving distribution shift are out of scope. These CPT studies focus on settings where the objective function remains language modeling, leaving the impact SFT unex- plored. Meanwhile, a growing body of work has examined the gap between pre-training quality and downstream performance more broadly. [Sun & Dredze (2025) showed that stronger pre-training per- formance does not necessarily translate to superior fine-tuning outcomes, and (Springer et al. (2025) demonstrated that over-trained models become harder to fine-tune. These findings collectively mo- tivate our investigation, as LR schedulers chosen to optimize potentially unreliable pre- pre-traine training met- rics may not be optimal for the overall training pipeline. Our work identifies LR decay as a specific factor that improves pre-training metrics at the cost of downstream adaptability.
Adaptability and Loss Landscape Geometry. Early work showed that parameters in flatter loss regions generalize better than those in sharp minima (Keskar et al.) 2017). motivating sharpness- aware minimization (Foret et al., 2021) and stochastic weight averaging (Izmailov et al., 2018)) Recent theoretical advances explain WSD through river valley loss landscape perspective (Wen) et al. 2025), where the stable phase explores along the valley floor while the decay phase converges toward the center. Concurrent work confirmed that sharpness increase during decay universal across architectures (Belloni et al. 2025). Flat-minima optimizers work well under distribution shift (Kaddour et al.] (002), a property that extends to the gre-trainigg/fine-tu tunnengtunngg paradigm, where the fine-tuning data distribution differs substantially from pre-training (Ju et al., 2022] Liu et al.) 2023). While prior work focused on understanding sharpness dynamics during pre-training (Bel-) loni et al. 2025) Wenn et al.) 2025), we demonstrate how these dynamics concretely impact SFT performance, showing that WSO preserves flatness and enhances downstream adaptability.
8 CONCLUSION
In this study, we investigated the effectiveness of LR schedulers, which have been widely reported as effective for pre-training, in practical scenarios with a focus post-training performance. In particular, we examine Warmup-Stable-Only (WSO), which removes the decay phase from WSD. Experimental results show that WSO consistently outperforms decay- based schedulers in down- stream tasks after SFT across standard pre-training, mid-training, and over- over-training regimes. Loss landscape analysis further reveals that WSO preserves flatter minima, explaining its superior adapt- ability. WSO is simple to apply and yields improved post-training performance, making it a promis- ing alternative for constructing more portable models. We also recommend releasing LLMs trained with WSO so that practitioners can benefit from their adaptability.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work investigates learning rate scheduling for LLM training to improve downstream adaptabil- ity. While our methods may provide new findings on LR scheduling pre-training, we acknowl- edge the broader implications of advancing LLM capabilities. We encourage responsible deploy- ment with appropriate safety measures during post-training. We exclusively used publicly available datasets for pre- pre-training, supervised fine-tuning, and evaluation. Moreover, we developed the lan- guage models entirely from scratch, avoiding the use of any publicly available models ensure reproducibility.
REPRODUCIBILITY STATEMENT
To ensure reproducibility of our results, we provide comprehensive experimental details throughout the paper and appendices. Model architectures for both 1B and 8B parameter models are specified in Appendix [ including layer configurations and attention mechanisms. All pre-training hyper- parameters, including optimizer settings, batch sizes, and training steps, are detailed in Appendix C The supervised fine-tuning configuration, including the learning rate sweep range and evaluation protocols, is described in Appendix D Our sharpness computation methodology using Hutchin- son's 's estimator is fully specified in Appendix H We use publicly available datasets (FbneWeb-EEde olmo-mix- 124, dolmino-min with detailed evaluation settings provided in Appendix
124, and Tulu-3 SFT mixture) and standard evaluation benchmarks,
E Full numerical results for all experiments reported in Appendix F to facilitate comparison and validation.
ACKNOWLEDGMENTS
This work was partly supported by JST Moonshot R&D Grant Number JPMJMS2011-4 1-35 (funda- mental research).
REFERENCES
Maksym Andriushchenko. Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 840-902. PMLR, 23-29 Jul 2023. URL h
https: pttts:://proceedings.o mlr press wttppeeee//o////dddnuuusueseeaaa http:///nnrissscchnccc23232
Annalisa Belloni, Lorenzo Noci, and Antonio Orvieto. Universal dynamics of warmup stable decay:
understanding WSD beyond transformers. In ICMI. 2025 Workshop on Methods and Opportuni- ties at Small Scale, 2025. URL htt.ttss//ppeeeeeeieew.... wttp::/.eeeffoooeeoeeeee 2ttp://fooouiiiii2oooo22CC
Shane Bergsma, Nolan Simran Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hes- tness. Straight to zero: Why linearly decaying the learning rate to zero works best for LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
7/ /ppeneevieww net http::/www..tttoooooaooooooooao.
Yonatan Bisk, Rowan Zellers, Ronan bras, Jianfeng Gao, and Choi Yejin. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference Artificial Intelligence, 34:7432-7439, 04 2020. doi: 10.1609aaaii 334005.....3100..23...333333 .2239.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry. Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., fearorfflesspppee files/paper/2022.
2020. URL https::/ppooeeeengggss.o. www.:iiinnnnnnuuuuueeeeeeee flle//55ccco0rra555771111666e2e2

---

Published as a conference paper at ICLR 2026
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprssing difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 2924-2936 Minneapolis, Min- nesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v URL https://aclanttoooorry org/N11 1300/
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2378,, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19-11 1246. URL https:
7/ //aclaatholLoyyooz///A www.loooy.../NN11-1111
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awart minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2021. htt.httttss:/:peereeee
wtt.:ewwwwwwweeeeeeeee..ee.e...e forum? did= mpos 1 rM
Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhu- rina, Igor Vasiljevic. Luca Soldaini, Jenia Jitsev, Alex Dimakis, Gabriel Ilharco, Pang Wei Koh, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt. Language models scale reliably with over-training and on downstream tasks. In The Thirleenth International Conference on Learning Representations, 2025. URL https: www.reeie...eee.eemmmoo/p....... net/ /trr//oruue Forum? 1i77mme22dddi//aaaaat ZeQBqJamf
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Au- thur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crys- tal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannaneh Hajishirzi. OLMo: Accelerating the science of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15789-15809, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653vv/l 2024.acl-long.88 841. URL htt.::ttps///mm ac www.clllnnoooooyoooooooooooooooe org/ 2024 acl- 841/
Alex Hägele, Elie Bakouch, Atli Kosson, Leandro Von Werra, Martin Jaggi, et al. Scaling laws and compute-optimal training beyond fixed training durations. Advances in Neural Information Processing Systems, 37:76232-76264, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
d7KB jm 3GmQ
12

---

Published as a conference paper at ICLR 2026
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neu- ral Information Processing Systems, 2022. URL https: htt.ttttttpppeeeeeeeeee http////ppeeeeiieeeeeeeeeeeeeeee net / forum?id' L.BRRULOAPR
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06339 2024.
M.F. Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Simulation and Computation, 1803):0059- 1076, 1989 doi: N0.10800036109189000 12806. URL https: http://000...011111000000 9861091890881280
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies to continually pre-train large language models. Transactions on Machine Learning Research, 2024. ISSN 2835-
8856. URL /ttps:// http://wwww.......eeeeeooaaaaaaa
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, pp. 876-885, 2018.
Hongpeng Jin, Wenqi Wei, Xuyu Wang, Wenbin Zhang, and Yanzhao Wu. Rethinking learning rate tuning in the era of large language models. In 2023 IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI), pp. 112-121. IEEE, 2023.
Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10431 PMLR, 17-23 Jul 2022. URL https::/pxxcceednng: mmr..mmm.ppp..... press// v162 fu22a. html
Jean Kaddour, Linara Adilova Key, Bernhard Schölkopf, and Andrew Gordon Wilson. When do flat minima optimizers work? In Advances in Neural Information Processing Systems, volume 35, pp. 16577-16595, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Javier Antorán, Arno Blaas, Fan Feng, Sahra Ghalebikesabi, Ian Ma- son, Melanie F. Pradier, David Rohde, Francisco J. R. Ruiz, and Aaron Schein (eds.), Proceed- ings on "I Can't Believe It's Not Better! Understanding Deep Learning Through Empirical Falsificution" at NeurlPS 2022 Workshops, volume 187 of Proceedings of Machine Learning Research, pp. 51-65. PMLR, 03 Dec 2023. URL hattts::/ppoceedinnee mlr. pprsss/ v187 wta.//aurraaaaaaa.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017. URL https: /tLisstpss//ooeeevviii http://pppeee....eeeee
mett/forum:: ftt.///oommmmidhlolllll Ygg
Teven Le, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvon, et al. Bloom: A 176b-parameter open- access multilingual language model. arXiv preprint arXiv:2211.05 100, 2022.
13

---

Published as a conference paper at ICLR 2026
Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang. Flat-loRA: Low- rank adaptation over a flat loss landscape. In Forty- second International Conference on Machine Learning. 2025. httpt::tt///peeeee
net. ottpe//eeeeeeeeettroorrrrro.--0.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinn models. https: /ectps:/ggitie wtt.ta/.......aaaaaaaaaaaaaaaaaa _aaopaaa__lll______________a_ala 5 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pp. 3214 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguis- URL https: https////caaaatoooooyoo org/2022.
tics. doi:
acl soll-ooge Long. 229/
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better down- stream: Implicit bias matters for language models. In International Conference on Machine Learning. pp. 22188-22214 PMLR, 2023.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scal- able stochastic second-order optimizer for language model pre-training. In The Twelfth Interna- tional Conference on Learning Representations, 2024b. Btth:tts::/ htttt:::///peneevieee nnt/
Forum??d==xxHeeA@No
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2019. http///ppe//ppeeeeeeeaaaee//eeee
Bkg6RiCqY7
Meta. Meta-Ilama-3.15 3.1-88 model card. hatpas//huggingaaee.c//attttaaa
Met a 1.l.am 2024a. Accessed: 2025-09-21.
Meta. Llama-3.2-1b model card.
2-18 2024b. Accessed: 2025-09-21.
Alat Meta. The llama 3 herd of models, 2024c. URL
21783
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct dataset for open book question answering. In Ellen Riloff, David Chiang. Julia Hockenmaier, and icci Tsujii (eds.), Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18633/1/D18-126... URL
electricity?
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Guilherme Penedo, Hynek Kydlfček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the Web for the finest text data at scale. In The Tiiht Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URLhttps: httpat/ppp//ppeeeeieeeee/ 1d==nSSck22QG
14

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In seieth- Conference Neural Information Processing Systems, 2023. URL https: //openreview. wwt./ htt.:oormmmmmhppppseeoee d=HPus IXJaa9
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Sunny Sanyal, Atula Tejaswi Neerkaje, Jean Kaddour, Abhishek Kumar, and Sujay Sanghavi. Early weight averaging meets high learning rates for LLM pre-training. In First Conference on Lan- guage Modeling, 2024. (RLL/ttps///ppeeeeeaa
net ? I---AarnI-L--LAWEEIII
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal. Ac- counting for inference in language model scaling laws. In Forty-first International Conference on Machine Learning, 2024. httts:/::://ppe /ttp////ppeeeeeeeeeeeeeeeee .et. hott:/ooumm? wewr:rmmmmdddmmmmmmdmmm dbm TDUu
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Malladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune. Proceedings of the International Conference on Machine Learning, 2025.
Kaiser Sun and Mark Dredze. Amuro & char: Analyzing the relationship between pre-training and fine-tuning of large language models. In Vaibhav Adlakha, Alexandra Chronopoulou, Xiang Lor- raine Li, Bodhisattwa Prasad Majumder, Freda Shi, and Giorgos Vernikos (eds.), Proceedings of the 10th Workshop on Representation Learning for NLP (RepLANLP-2025), pp. 131-151, Al- buquerque, NM, May 2025. Association for Computational Linguistics. ISBN 979-8-89176- 245-9. doi: 10. 11866///1/0025.rp11mmlll dep14nlp-1.117
URL httaacclanthooogyyy. org/2025.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguis- htt.://..nn2.......... findings-acl.822
tics. doi: 10. (05118655////// 1/2023. Findings acl.acc..00 8247
hetcttsss//allannhlloogo// org/2033,
Changxin Tian, Jiapeng Wang, Qian Zhao, Kunlong Chen, Jia Liu, Ziqi Liu, Jiaxin Mao, Wayne Xin Zhao, Zhiqiang Zhang, and Jun Zhou. Wsm: Decay-free learning rate schedule via checkpoint merging for Ilm pre-training arXiv preprint arXiv:2507.17644 2025.
Howe Tissue, Venus Wang, and Lu Wang. Scaling law with learning rate annealing. arXiv preprint arXiv:2408. 1029, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1397,,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 092888 2023b.
Xingjin Wang, Howe Tissue, Lu Wang, Linjing Li, and Daniel Dajun Zeng. Learning dynamics in continual pre-training for large language models. In Forty-second International Conference on Machine Learning, 2025. URL htttts:://pperreeeeeeeee
net forum? 1
Zhihao Wang, Shiyu Liu, Jianheng Huang, Wang Zheng, YiXuan Liao, Xiaoxin Chen, Junfeng Yao, and Jinsong Su. A learning rate path switching training paradigm for version updates of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp- 13581-13594 Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653/v 1/20:/1..222222.211111 752.. URL ht httpp::/ http://aaaantttooooooo wwwp:/yy.rggg20000
eemn 1 ma n. 752/
15

---

Published as a conference paper at ICLR 2026
Kaiyue Wen, Zhiyuan Li, Jason S. Wang, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Understanding nurmup-taabe--deayy learning rates: A niver valley loss landscape view. In The Thirteenth International Conference Learning Representations, 2025. URL https:
//openreviess ew. net//forum i d=m51 BgogubP
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? Anna Korhonen, David Traum, and Lluís Märquez (eds.). Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.
1PP991142 URL htt.h:tps://// hattps::/aallanthooooy.... www.pll.rrrppp.. 121///9911144 9-122
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299-2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v findiiiinninnnaaeee naacl. 149. URL
findings naacl. naac1.149/
149/
16

---

Published as a conference paper at ICLR 2026
Table 5: Model configurations for the 1B and 8B models.
Configuration
IB
8B
Hidden dimension
2048
4096
FFN dimension
8192
14336
Number of layers
16
32
Number heads
32
32
Number KV heads
8
Head dimension
64
128
Vocabulary size
128256
128256
RoPE 0
10000
10000
RMS norm E
10-5
10
Activation function
SwiGLU
SwiGLU
A MODEL. ARCHITECTURE
We provide detailed specifications for models used in our experiments. Both the IB and models follow the Llama 3 architecture (Meta 2024c), employing RMSNorm, SwiGLU activation, and Rotary Position Embeddings. We use the Llama 3 tokenizer with a vocabulary size of 128,256 tokens for all models.
B LEARNING RATE SCHEDULER FORMULATIONS
We provide the complete formulations for the WSD, Cosine, and Linear LR schedulers used in our experiments.
WSD Schedule: After warmup, the LR remains constant until Taabee then decays linearly Opr mmax at step T:
9max
Tornup
S T warmup
n psnn (1, Cprre)
Tmmx
T warmup < t S T Ttable
Tmmx
((1 C'pre)
Tym Tym-Tame Tume
Tya-t
Oppec
Tstable
< t < pre
WSO Schedule: Obtained by setting Oppre
1 in WSD. After warmup, the LR stays constant:
MSO (t, Cppce
7max
Tasme
S T warmup
Tmaxx
T Twarmup
< l S Tpre
Cosine Schedule: After warmup, the LR follows Cosine decay to Oppre Mmmax
Mmax
< T warmup
Cosine (t, aprre)
7max
Tormp
(pre
1-Ctym
(I +
COS
Turmpp Tim- Feomeey
(-
T Tarmup
(9)
Linear Schedule: After warmup, the LR decays linearly to Opre Tmaxi
Linear (t, apre)
Mmax
Toamu:
i
t < T warmmup
(10)
Mmax
((1 Cipre)
Tya Tramsp
Tyc + Cipre
T Tarmmu
All the schedulers use the same warmup phase described in Section 2.4 and their decay con-
trolled by the minimum LR factor O're E [0.0, 1.0].
Mid-training LR Scheduling. In the mid-training stage, we extend the pre-training learning rate schedulers. The mid-training learning rate at time step &. is defined as:
17

---

Published as a conference paper at ICLR 2026
Table 6: Pre-training hyperparameters for 1B and 8B models. The WSD stable ratio p 0.75 means the LR remains stable for 75% of training after warmup, with decay occurring in the final 25% when Pre 1.
Hyperparameter
IB
8B
Training Configuration
Total training steps
80,000
80,000
Total tokens
350B
500B
Batch size (tokens)
4,194,304
12,582,912
Sequence length
2,048
2,048
Optimizer (AdamW)
Max LR (Mmax)
x 10
3 10
Weight decay
0.1
0.1
Adam B1
0.9
0.9
Adam $2
0.95
0.95
Adam
x 10
1 x 10
Gradient clipping
1.0
LR Schedule
Warmup steps (Twarmap)
1,000
1,000
WSD stable ratio (p)
0.75
0.75
Min LR factor (Cigre)
{0.0. 0.1, 1.0}
{0.0, 0.1, 1.0)
Other
Precision
66loat16
bfloat16
Table Over-training configuration for the 1B model trained on 2T tokens. All other hyperparam- eters are identical to those in Table 6
Hyperparameter
Value
Training Configuration
Total training steps
120,000
Total tokens
2T
Batch size (tokens)
16,777,216
Scheduler (t, Oprre m'id) = 17 Sccheduler (Tpre, aipre)
(1-
Okmid)
Tpre + Tmid - t + Olmid
(11)
Tiid
for l (Tee + 1, Tpe + Tmia], where Tpre is the total number of pre-training steps and Tmid is the total number of mid-training steps.
C PRE-TRAINING HYPERPARAMETERS
We provide detailed hyperparameters used for pre-training our models in Table 6 All experiments the AdamW optimizer (Loshchilov & Hutter 2019) with mixed precision. For over-training experiments, we modify the training duration as shown in Table | where the 1B model is trained for 120,000 steps to process 2T tokens and set different batch sizes while maintaining the other hyperparameters in T
D SFT CONFIGURATION
We performed supervised fine-tuning for all models using the Tulu-3 SET mixture dataset. Since the official dataset does provide predefined train-validation split, create using 9:1 ratio for training and validation, respectively. We perform full parameter training for all models. Table & presents the hyperparameters used in experiments.
18

---

Published as a conference paper at ICLR 2026
Table SFT hyperparameters used in experiments. We perform sweep the specified LRs and select the best value based on AlpacaEval performance.
Hyperpurameter
Value
LR
5.0 10 7,100 9.500 10 1.0 5, 5.500 10
1.0
10
4.5.0 10
1.00 10
Global Batch size
128
LR scheduler
Cosine with warmup
Minimum
Optimizer
AdamW 0.0
Weight decay
Gradient clipping
1.0
Warmup sleps
100
Epochs
1
Training precision
b1oaa166
E EVALUATION DETAILS
For pre-trained models, all benchmarks are evaluated in zero-shot setting.
For mid-trained models (before SFT), we evaluate on standard benchmarks following the evalu- ation suite used OLMo (OLMo et al. 2024). We assess reasoning capabilities using ARC- Challenge (Clark et al., 2018), HellaSwag (Zelers et al., 2019), and WinoGrande (Sakaguchi al. 2021). Reading comprehension is evaluated with DROP (Dua et al. 2019) using 5-shot prompting, while mathematical reasoning is assessed using GSM8K (Cobbe et al. 2021) with 8- shot chain-of-thought (CoT) prompting.
For SFT models, we use the following evaluation configurations. For AlpacaEval, following Springer et al.] (2025), rather than comparing against GPT-4o, where the win rates would uni- reference model of the same architecture to better distinguish performance differences between LR schedules. Specifically, we use the WSO model with ape 1.0, fine-tuned with the lowest LR from our sweep (5 x 10-7) as our reference, ensuring stable and meaningful comparisons within each model scale. Evaluations are performed by Llama-3-70B-Instee MMLU (5-shot), evaluation covers 57 subjects spanning STEM, humanities, social sciences, and other domains. For TruthfulQA, SFT, we additionally evaluate on GSM8K (1-shot), DROP (5-shot), AGI Eval (Zhong et al.) 2024) (3-shot), and Bighench-Hard (Suzgun et al. 2023
formly low,
For
use the standard evaluation protocol. After mid-training and
(3-shot with CoT).
F FULL EVALUATION RESULTS
This section provides complete per-task evaluation results for all pre-trained and fine-tuned models across different LR schedules. While the main text presents aggregated metrics and relative perfor- mance comparisons, here we report the absolute performance values for each individual benchmark.
F.1 PRE-TRAINING EVALUATION RESULTS
Table 9 presents comprehensive zero-shot evaluation results for all pre-trained models across differ- ent LR schedules.
F.1.1 PRE-TRAINING EVALUATION RESULTS IN OVER-TRAINING
Table 10 shows that, also in the over-training regime with 2T tokens, the Cosine scheduler with decay achieves slightly better zero-shot task performance and lower validation loss compared to WSO.
F.2 SFT EVALUATION RESULTS
We select the best learning rate for each pre- pre-trained model based AlpacaEval performance, the primary objective of SFT is to enhance followiiinn capabilities. Selecting hyperparameters based such validation loss does necessarily yield better downstream task performance,
19

---

Published as a conference paper at ICLR 2026
Pre-training evaluation results. Models with decay (O'pee lower validation loss, but not always better zero-shot task performance.
Table 9:
0) generally achieve
Model
Schoduler
Valid Loss
ARC-c
ARC-c
BoolQ
Hella
OBQA
PIQA
Wino
Avg.
Srannup-Stabll-ee Ouly (WSO)
1.0
2.431
70.8
42.2
62.0
$6.3
45.4
70.8
58.5
58.0
WSD
2.364
40.0
57.4
46.4
58.2
0.0
2.360
72.2
39.7
63.7
57.6
45.6
72.2
58.6
58.5
1B
Linear
0.1
2,380
70.3
42.6
63.2
55.6
45.2
71.6
55.7
57.7
2.376
43.4
47.4
70,9
59.7
2.379
71.1
43.6
66.5
59.9
47.8
71.7
56.3
59.6
Cosine
0.0
2.376
74.6
41.9
50.7
58.5
48.4
71.0
55.4
57.2
Warmup Stable Only (WSO)
1.0
2.119
79.4
52.6
69.1
69.1
52.8
76.3
64.5
66.3
2.011
52.8
72.6
53.2
75.9
66.9
WSD
0.0
2,005
81.0
53.0
67.2
72.9
54,2
76.3
65.0
67.1
8B
0.1
2.004
79.4
53.7
64.1
71.2
50.4
75.0
62.4
63.2
Linear
71.5
74.9
0.1
2,001
76.3
47.6
71.3
71.5
52.4
74.3
60.9
64.9
Cosine
0.0
2.000
74.2
46.8
71.7
71.4
52.6
76.3
60.8
64.8
Table 10: Pre-training evaluation results for over-trained 1B models (2T tokens).
Model
Scheduler
Otec
Valid Loss
ARC.e
ARC-c
BoolQ
Hella
OBQA
PIQA
Avg.
Wamuu-Stable-Onll (WSO)
1.0
2.625
74.4
43.3
59.7
63.5
48.6
73.2
62.0
60.7
WSD
0.1
2.582
75.4
45.8
60.0
66.6
50.0
74.7
62.7
62.2
2.578
75.3
46.8
59.2
66.2
50.4
74.4
63.0
62.2
IB
2.599
45.6
65.4
48.0
73.2
61.3
Linear
0.0
2.595
73.9
44.4
66.6
63.2
49.2
73.6
59.8
61.8
Cosine
0.1
2,595
72.9
44.1
65.7
64.9
52.0
74.0
61.5
62.2
2.595
44.4
45.8
72.7
60.9
which is consistent with our main finding that lower pre-training loss does not guarantee better post- SFT performance. We apply an identical learning rate sweep to all pre-trained models, ensuring that no scheduler receives a selective advantage. Table I1 shows the selected learning rates for each model.
Table 12 shows performance after SFT across different pre-training schedules. Models pre-trained with WSO moderate decay (Apre 0.1) often achieve comparable better downstream perfor- mance than those with aggressive decay (a'pre = 0.0), despite having worse pre-trate pre-training metrics.
F.2.1 SFT EVALUATION RESULTS IN OVER- TRAINING
shows the learning rates selected for each over-trained model. Table 14 demonstrates Table 13
that even after over-training with 2T tokens, WSO achieves superior SFT performance compared to decay-based schedulers.
F.3
MID-TRAINING EVALUATION RESULTS
Table 15
presents evaluation results after the mid-training stage.
F.3.1
MID-TRAINING EVALUATION RESULTS IN OVER- TRAINIGG
Table 16 shows that after over-training and mid-training, WSO achieves superior overall perfor- mance despite having nearly identical validation loss.
F.4 SET EVALUATION RESULTS AFTER MID-TRAINING
Table 17 shows the optimal learning rates selected for each pre-trained model based on AlpacaEval performance.
20

---

Published as a conference paper at ICLR 2026
Table 11: SFT learning rates selected for each pre-trained model based AlpacaEval performance.
Model
Scheduler
Selected SFT LR
Warmup-Stable-Ont (WSO)
1.0
3 10
0.1
1 x 10
WSD
0.0
1 10
IB
0.1
x 10 1 x 10
Linear
0.0
0.1
1 1x 10
Cosine
0.0
1 10
Saamup-Stable-Olly (WSO)
1.0
3 10
0.1
3 10 1 1 10
WSD
0.0
8B
1 10 1 x 10
Linear
0.0
0.1
1 1 10
Cosine
3 10
Table 12:
SFT evaluation results.
Models pre-trained with WSO achieve the best downstream
performance.
Model
Scheduler
O'ee
AlpacaEval
TruthfulQA
MMLU
Avg.
Warmup-Staale (WSO)
1.0
84.0
43.4
35.9
54.4
0.1
83.9
41.9
36.6
54.1
WSD
0.0
82.3
40.2
36.7
53.1
1B
0.1
82.0
42.0
36.3
53.4
Linear
0.0
82.4
41.7
35.6
53.2
0.1
83.6
41.0
35.5
53.4
Cosine
0.0
83.6
41.0
35.6
53.4
Warmup-Stable-Onll (WSO)
1.0
79.7
42.5
42.7
55.0
0.1
77.1
40.8
41.4
53.1
WSD
0.0
77.3
39.9
43.7
53.6
8B
0.1
76.4
41.4
42.1
53.3
Linear
0.0
78.4
40.6
42.8
53.9
0.1
78.6
39,9
42.3
53.6
Cosine
0.0
77.8
40.3
43.3
53.8
Table 18 shows SFT performance after mid-training. WSO during mid-training (Omii 1.0) gener- ally achieves better SFT performance compared to those with decay (a'iid = 0.0).
F.5 SFT EVALUATION RESULTS AFTER OVER-TRAINING WITH MID-TRAINING
Table 19 shows the selected learning rates for each over-trained model, and Table 20 shows that WSO achieves superior SET performance compared to decay-based schedulers.
G MID-TRAINING CONFIGURATION DETAILS
We provide the detailed configuration used for mid-tren mid-training experiments in Table 21 Other hyper- parameters are the same as the configurations of pre- pre-training in Table 6 Mid-training is conducted on the do 1i no-mi ---mmix-11 x-11 24 dataset, which consists of diverse high-quality data sources.
21

---

Published as a conference paper at ICLR 2026
Table 13: SFT learning rates selected for each over-trained 1B model based AlpacaEval perfor- mance.
Model
Scheduler
Otrr
Selected SFT LR
Warmup-Stable-Only (WSO)
1.0
1 x 10
0.1
3 10
WSD
0.0
3 x 10
1B
0.1
3 10"3
Linear
0.0
3 10-5
0.1
1 10-5
Cosine
0.0
1 10
Table 14: SFT evaluation results for over-trained 1B models (pre-trained on 2T tokens).
Model
Scheduler
Opee
AlpacaEval
TruthfulQA
MMLU
Avg.
Warmup-Stable-Onig (WSO)
1.0
78.1
38.7
34.5
50.4
0.1
77.2
38.3
33.6
49.7
WSD
0.0
76.0
38.4
33.7
49.4
1B
0.1
75.6
37.8
34.2
49.2
Linear
0.0
75.5
37.9
33.9
49.1
0.1
76.0
37.9
33.9
49.3
Cosine
0.0
76.4
37.9
33.9
49.4
Additionally, we provide the detailed hyperparameters used for mid mid-training in over-training settings in Section in Table 22
H SHARPNESS COMPUTATION DETAILS
We compute the sharpness (Hessian trace) using Hutchinson's stochastic trace estimator (Hutchin- son 1989). which provides an unbiased estimate through random vector sampling. For Hessian matrix H. the trace is estimated as:
Hz,
(12)
where random vectors sampled from a Rademacher distribution (i.c., each element is #1 with equal probability).
Implementation Details. We compute Hessian-vector products using automatic differentiation, which allows efficient computation without explicitly constructing the full Hessian matrix.
Table 23 shows computation configurations for Hutchinson's estimator. We measure sharpness regular intervals throughout pre-training (every 4,000 steps) on held-out validation sets from both the pre-training dataset and the SFT dataset understand how the loss landscape geometry evolves during training.
22

---

Published as a conference paper at ICLR 2026
Table 15: Mid-training evaluation results in Section 4
Model
Pre training Scheduler
Valid Loss J
ARC-C
HellaaSwa
WinoGrande
DROP
GSMBK
Avg.
Warmup Stable-Onf (WSO)
1.0
2.335
47.0
60.5
58.6
23.9
20.4
42.1
2.273
42.2
WSD
0.1
0.0
2,320
45.0
62.0
60.7
23.8
11.0
40.5
IB
2.352
43.8
61.4
59.5
20.2
10.7
39.1
2.326
44.3
60.7
59.7
21.4
12.8
39.8
2.330
38.9
Linear
0.1
1.0
2,325
43.2
60.3
60.1
23.6
13.3
40.1
Warmup Stable Only (WSO)
1.0
2.009
64.9
75.4
69.4
49.7
52.8
62.4
1.907
64.5
WSD
0.1
0.0
1,988
61.4
80.0
71.1
42.6
39.7
59.0
1.991
54.3
77.0
69.7
35.4
36.0
54.5
1.975
$7.1
77.5
69.1
38.6
40.3
56.5
Linear
77.3
37.7
55.5
0.1
1.0
1.974
56.7
77.5
69.9
36.6
40.3
56.2
Table 16: Mid-training evaluation results for over-trained 1B models (pre-trained on 2T tokens, mid-trained 500B tokens).
Model
Pre-training Scheduler
Valid Loss
ARC-C
HellaSwag
WinoGrande
DROP
GSMBK
Avg.
Varmup-Stable-Ong (WSO)
1.0
1.0
2.254
46.7
61.3
60.4
27.0
23.1
43.7
1.0
2.199
47.1
65.2
62.2
23.7
13.7
42.4
WSD
0.1
0.0
2.257
47.1
65.2
62.2
23.4
13.7
42.3
2.231
47.4
65.7
62.6
25.3
19.1
44.0
Cosine
2.253
0.1
1.0
2.245
43.5
64.7
62.1
25.9
14.8
42.2
Linear
2.250
47.2
63.4
59.4
20.9
15.4
41.3
2.267
45.6
63.3
60.1
21.4
18.7
41.8
Table 17: SFT learning rates selected for each model configuration based on AlpacaEval perfor-
mance.
Model
Scheduler
O'ere
O'mid
Selected SFT LR
Warmup-Stuble-OOll (WSO)
1.0
1.0
x 10-4
1.0
0.0
3 10-5
WSD
0.1
1.0
1 10
0.1
0.0
3 10
0.1
1.0
3 10
Linear
0.1
0.0
x 10
0.1
1.0
x 10
Cosine
0.1
0.0
1 10
Warmup-Stable-Only (WSO)
1.0
1.0
x 10
1.0
0.0
1 10
WSD
0.1
1.0
1 10
8B
0.1
0.0
3 10
0.1
1.0
1 10
Linear
0.1
0.0
1x 10
0.1
1.0
1x 10
Cosine
0.1
0.0
X 10
23

---

Published as a conference paper at ICLR 2026
Table 18: SFT evaluation results after mid miid-trainin.. WSO throughout pre- and mid mid-training gener- ally achieves better SFT performance.
Pre-tmining
Ope
Ous
AlpacaFsal
TruthfulQA
DROP
Avg.
Warmu-SStablo-One (WSO)
1.0
1.0
79.4
41.8
29.0
22.7
21.8
23.1
35.7
36.2
1.0
0.0
79.4
39,9
27.2
22.0
21.5
22.7
35.4
35.4
WSD
0.1
0.0
76.8
41.0
18.9
22.0
22.4
23.8
34.2
34.2
0.1
1.0
78.7
40.0
21.2
23.7
23.1
23.8
34.4
35.0
Cosine
0.1
0.0
72.9
38.1
19.9
17.6
22.1
17.9
33.9
31.8
0.1
1.0
74.3
37,9
22.2
17.1
22.6
19.6
34.0
32.5
Linear
0.1
0.0
73.2
39.1
14.0
16.2
22.1
22.3
34.3
31.6
0.1
1.0
76.3
40.8
17.7
16.3
22.8
21.4
35.1
32.9
Oa (WSO)
1.0
43.4
54.7
36.4
40.2
42.9
44.7
1.0
0.0
68.6
44.8
34,5
32.6
40.0
30.9
44.3
42.2
WSD
28.3
0.1
1.0
69.7
43.9
47.3
29.9
36.3
29.0
49.5
43.6
0.1
0.0
64.7
41.1
41.0
26.9
32.3
43.0
39.6
Cosine
0.1
1.0
63.9
41.9
40.8
28.8
34.6
28.5
42.8
40.2
42.5
36.8
28.3
33.6
44.9
39.9
Lincar
0.1
1.0
63.8
41.3
43.5
30.5
33.0
31.0
46.8
41.4
Table 19: SFT learning rates selected for each over-trained and mid-trained 1B model based on AlpacaEval performance.
Model
Scheduler
Opre
Ommid
Selected SFT LR
Warmup-Stable-Only (WSO)
1.0
1.0
x 10-5
1.0
0.0
x 10-5
WSD
0.1
1.0
3 x 10
IB
0.1
0.0
x 10-6
0.1
1.0
1 10-5 1 10
Linear
0.1
0.0
0.1
1.0
1 10
Cosine
0.1
0.0
1 10
Table 20: SFT evaluation results for over-trained 1B models after mid-training (pre-trained on 2T
tokens, mid-trained on 500B tokens, then supervised fine-tuned).
Model
Pre-traiiigg Scheduler
Oge
Osii
AlpacaBval
TruthfuQQA
GSMSK
DROP
AGI Eval
MMLU
Avg.
Stallp-Stabll-Onll (WSO)
1.0
1.0
66.2
38.1
30.3
19.4
24.1
24.8
36.6
34.2
40.4
20.4
18.4
21.3
35.5
32.3
WSD
0.1
0.0
64.8
39,8
15.6
19.5
21.4
23.7
36.0
31.5
IB
Cosine
0.1
0.0
62.5
41.1
18.7
20.5
23.2
18.8
35.9
31.5
Linear
0,1
0.0
64.7
39.0
20.2
19.6
22.6
24.2
34.8
32.2
35.0
Table 21: Mid-training configuration for and 8B models.
Hyperparameter
1B
8B
Training Configuration
Total training steps
36,000
36,000
Total tokens
150B
225B
Batch size (tokens)
4,194,304
12,582,912
Sequence length
2,048
2,048
24

---

Published as a conference paper at ICLR 2026
Table 22: Mid-training configurations in over-training settings for the 1B model trained on 500BT tokens. All other hyperparameters are identical to those in Table
Hyperparameler
Value
Training Configuration
Total training steps
30,000
Total tokens
500BT
Batch size (tokens)
16,777.216
Table 23: Configuration for sharpness (Hessian trace) computation using Hutchinson's 's estimator.
Hyperparameter
Value
Sequence length
1,024
Batch size
Number of views
Hutchinson samples
50
Maximum batches
4,096
Maximum texts
16.192
25


Reviews:
[{"review_id": "JTQuqM3AJo", "reviewer": "Reviewer_m1D7", "summary": "The manuscript presents extensive experimentation devoted to understanding learning-rate scheduling in LLM training. This work provides empirical insights that directly suggest how learning rate scheduling should be selected during pre-training to better support downstream model adaptability. The study recommends adopting Warmup-Stable-Only (WSO) as an alternative learning-rate strategy and releasing WSO-trained models to encourage wider use and adaptability in future LLM development.", "questions": "Please check the detailed comments for weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper examines the common practice of using learning-rate decay in LLM pre-training. The paper provides empirical evidence that keeping a constant learning rate after warmup improves performance. This approach, called the Warmup-Stable-Only (WSO) scheduler, outperforms conventional decay-based schedulers in supervised fine-tuning. Their finding highlights practical effectiveness for optimizing the entire LLM training pipeline.\n2. The paper demonstrates the inversion effect between pre-training and supervised fine-tuning performance across a wide range of settings. Decay-based learning rate schedulers consistently achieve stronger pre-training metrics, whereas the WSO configuration achieves superior results after SFT. This phenomenon is validated across 1B and 8B model scales, in multi-stage training pipelines.\n3. The paper challenges the standard assumption that stronger pre-training performance leads to a better final model. It shows that decay-based learning rate schedules achieve superior pre-training metrics, yet consistently result in worse performance after supervised fine-tuning. Their evidence suggests a need to rethink optimization goals in LLM development. They also emphasize prioritizing downstream adaptability over pre-training loss.", "weaknesses": "1. The experiments are restricted to 1B and 8B parameters, which are relatively small compared to state-of-the-art deployed LLMs (often 30B~70B+). The absence of results at larger scales limits confidence in whether the observed advantages of WSO would extend to all situations.\n2. The study evaluates WSO against only three decay-based schedulers (Cosine, Linear, and Warmup-Stable-Decay). Other commonly used or recently explored learning rate strategies, such as polynomial decay or cyclic policies, are not explored. This limited comparison makes it unclear whether WSO’s benefits extend to other learning-rate policies. The paper would benefit from a brief stability assessment that checks whether WSO remains reliable under different environments.\n3. The experiments tune SFT hyperparameters separately for each pre-trained model. However, they always use selective learning-rate policies during SFT. This choice gives WSO an inherent advantage in downstream evaluation. The learning-rate policy should instead be maintained consistently across all training phases to enable a fair comparison. The work also lacks theoretical significance, relying mainly on empirical observations.\n4. The paper primarily evaluates instruction-following and general reasoning tasks, without testing multilingual ability, coding, or robustness under distribution shift. This narrow benchmark scope limits confidence in how widely WSO’s performance would translate to real-world deployment scenarios.\n5. Important related studies are missing from the references, such as [1].\n\n[1] Jin, Hongpeng, et al. \"Rethinking learning rate tuning in the era of large language models.\" 2023 IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI). IEEE, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6a64OU6yAH", "reviewer": "Reviewer_jBxS", "summary": "This paper investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Specifically, this paper proposes Warmup-Stable-Only (WSO) learning rate schedule for pertaining, which is found to achieve better downstream tasks. Some experiments and intuitive understandings are presented.", "questions": "See above", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. This paper presents WSO, a very simple and intuitive LR schedule to improve SFT on downstream performance.\n2. This paper provides a practical reference for LLM pre-training community to design LR schedule from a global training perspective.", "weaknesses": "The primary concern with this paper is that the proposed approach—while effective—has been extensively discussed, implemented, and validated in prior work, without introducing significant novelty. Furthermore, the absence of references to these existing studies raises questions about the thoroughness of the literature review.\n\n1. In the original WSD paper [(https://arxiv.org/pdf/2404.06395)](https://arxiv.org/pdf/2404.06395), the authors already demonstrated the benefits of switching to high-quality datasets (including SFT data) during the learning rate decay phase, yielding intuitive and positive outcomes.\n\n2. The paper \"Scaling Law with Learning Rate Annealing\" [(https://arxiv.org/pdf/2408.11029)](https://arxiv.org/pdf/2408.11029) introduces a scaling law describing loss dynamics in relation to learning rates, of which the current work appears to be a specific instance.\n\n3. The paper \"Learning Dynamics in Continual Pre-Training for Large Language Models\" [(https://arxiv.org/pdf/2505.07796)](https://arxiv.org/pdf/2505.07796) provides comprehensive analyses, and the findings here seem to represent only a minor subset of their paper. Notably, their Finding 3 states: \"*PT models with higher loss potential consistently achieve lower D_cpt validation losses. Hence, we advocate that when releasing open-source models, it is beneficial to release a high loss potential version to facilitate downstream tasks.*\"\n\n   I strongly encourage authors to read this paper.\n\n4. The paper \"A Learning Rate Path Switching Training Paradigm for Version Updates of Large Language Models\" [(https://arxiv.org/pdf/2410.04103v1)](https://arxiv.org/pdf/2410.04103v1) applies a similar concept to LLM pre-training.\n\nIn essence, the core idea (**Let LR decay happen in the most important stage**) has already been well-established in the field. This paper just translates this idea into a superficial learning rate schedule.\n\nAdditionally, the paper lacks rigorous theoretical analysis, and the evaluation is insufficient. For example, Table 2 reports only loss variations and average SFT performance. To strengthen the claims, the authors should address deeper questions such as:\n\n1. How do the results vary if the pre-training duration is extended or shortened?\n2. Are certain SFT tasks more or less affected by the proposed schedule? If so, what underlying reasons might explain this?\n3. Given that WSO outperforms WSD, why not slightly increase the learning rate during pre-training to further boost downstream SFT performance, even at the expense of higher pre-training loss?\n4. Have the authors considered quantifying this process more formally, such as by deriving a scaling law?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdeKgAcTfv", "reviewer": "Reviewer_NMCt", "summary": "This paper investigates the impact of learning rate (LR) scheduling during LLM pre-training on downstream supervised fine-tuning (SFT) performance. The authors challenge the standard practice of using decay-based schedulers (like Cosine or WSD) which are optimized for pre-training loss. The paper introduces \"Warmup-Stable-Only\" (WSO), a simple scheduler that maintains a constant LR after warmup without any decay. Through comprehensive experiments on 1B and 8B models, the authors demonstrate a consistent \"inversion\": while decay-based schedulers achieve better pre-training metrics, models trained with WSO consistently achieve superior performance after SFT. This finding is shown to be robust across standard pre-training, mid-training, and over-training regimes. The paper provides a mechanistic explanation, analyzing the loss landscape and showing that WSO guides models to flatter minima, which enhances adaptability, whereas decay-based schedulers converge to sharper minima that may compromise downstream performance.", "questions": "The paper compellingly argues that WSO leads to flatter minima (lower sharpness) and that WSO models perform better on SFT. The link is made by showing these two facts separately. To make the justification more persuasive, have the authors considered plotting a direct correlation between the measured sharpness of the pre-trained checkpoints and their final SFT benchmark scores? This would provide a more direct piece of evidence that sharpness is indeed the key indicator for downstream adaptability as hypothesized.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is exceptionally clear, well-written, and easy to follow.\n\nThe central conclusion—that pre-training without LR decay enhances SFT performance—is simple, impactful, and supported by extensive evidence. The experiments are comprehensive, covering multiple model scales (1B and 8B), different training pipelines (two-stage and three-stage with mid-training), and modern training regimes (over-training).\n\nThis work has significant practical implications for the industry. The WSO scheduler is simple to implement and could provide real economic benefits by producing base models that are more adaptable and performant for downstream tasks.\n\nThe mechanistic explanation provided via loss landscape sharpness is insightful. The analysis linking the constant LR of WSO to flatter minima, and in turn, to better adaptability, offers a compelling hypothesis for *why* WSO outperforms decay-based methods in the post-SFT stage.", "weaknesses": "The primary weakness, though minor, is that the investigation of downstream performance is limited to SFT. The paper does not explore other critical post-training stages, such as preference tuning (e.g., DPO) or reinforcement learning-based alignment. It remains an open question whether the significant benefits of WSO pre-training persist or behave differently in these other alignment scenarios. I don't think this would be an issue as the title also constrains the scope to SFT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7LUAgCCnye", "reviewer": "Reviewer_68iN", "summary": "This paper investigates learning rate scheduling in large language model pre-training, specifically examining how different schedulers affect downstream performance after supervised fine-tuning (SFT). The authors propose Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without decay, and demonstrate through experiments on 1B and 8B parameter models that WSO consistently outperforms decay-based schedulers (WSD, Cosine, Linear) on post-SFT tasks, despite achieving worse pre-training metrics. The authors attribute this to WSO preserving flatter loss landscape minima that support better adaptability.", "questions": "1. You attribute WSO’s superior SFT performance to flatter minima (lower sharpness). Could you quantify how much this flatness contributes to downstream adaptability? For example, is there a measurable correlation coefficient between sharpness values and SFT task scores?\n2. Have you examined whether the flatter minima correspond to wider basins of equivalent loss or simply slower convergence zones? This distinction matters for transfer dynamics.\n3. How sensitive are your conclusions to the warmup length? Since WSO keeps the LR constant after warmup, a longer warmup could emulate partial decay.\n4. The study is based on Llama-like architectures. Do you expect the same effect for mixture-of-experts (MoE) or sparse transformer setups where parameter utilization patterns differ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors rigorously demonstrate that WSO outperforms traditional decays across multiple scales (1B and 8B), stages (pre-, mid-, and post-training), and regimes (standard, mid-training, over-training). The consistency of results is convincing — the performance inversion between pre-training and SFT is robust.\n2. Loss landscape analysis connects WSO’s success to flatter minima — a strong explanatory narrative consistent with sharpness-aware generalization theory (Foret et al., 2021; Wen et al., 2025). Figure 3’s curvature dynamics clearly support this interpretation.\n3. The exposition is systematic, well-cited, and transparent. Appendices include hyperparameters, datasets, and evaluation details.", "weaknesses": "1. The explanation of “flatter minima = better adaptability” is qualitative. The paper would benefit from formalizing how curvature interacts with SFT gradient flow (e.g., via a transferability Jacobian or Hessian spectrum analysis across tasks). Without this, the claim remains an empirical observation.\n2. SFT evaluation focuses on AlpacaEval, TruthfulQA, and MMLU. These are instruction-following benchmarks but do not fully probe reasoning or alignment generalization. \n3. WSO maintains a higher effective learning rate longer — potentially increasing training instability or wasted compute in late phases. The authors should quantify total compute efficiency (e.g., perplexity vs wall-clock time) to assess tradeoffs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)