Scaling Law with Learning Rate Annealing
Authors: Howe Tissue
Venus Wang
Lu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Scaling Laws
Full Loss Curve Prediction
Learning Rate Schedule
LLM Pretraining
Primary area: deep_learning
TL;DR: A new scaling law formula with learning rate annealing that can fit and predict full loss curves.

Abstract: We find that the cross-entropy loss curves of neural language models empirically adhere to a scaling law with learning rate (LR) annealing over training steps: $$L(s) = L_0 + A\cdot S_1^{-\alpha} - C\cdot S_2,$$ where $L(s)$ is the validation loss at step $s$, $S_1$ is the area under the LR curve, $S_2$ is the LR annealing area, and $L_0$, $A$, $C$, $\alpha$ are constant parameters. This formulation accounts for two main effects: (1) power-law scaling over data size, and (2) the additional loss reduction during LR annealing. Unlike previous studies that only fit losses at final steps, our formulation captures the entire training curve, allowing for parameter fitting using losses from any training step. Applying the scaling law with LR annealing and fitting only one or two training curves, we can accurately predict the loss at any given step under any learning rate scheduler (LRS). This approach significantly reduces computational cost in formulating scaling laws while providing more accuracy and expressiveness. Extensive experiments demonstrate that our findings hold across a range of hyper-parameters and model architectures and can extend to scaling effect of model sizes. Moreover, our formulation provides accurate theoretical insights into empirical results observed in numerous previous studies, particularly those focusing on LR schedule and annealing. We believe that this work is promising to enhance the understanding of LLM training dynamics while democratizing scaling laws, and it is helpful to guide both research and industrial participants in refining training strategies for further LLMs.

Scaling Law with Learning Rate Annealing
Howe Tissue x 1-sun200ttsinguuaoorg
Venus Wang
Sangxxing12@gmai.ccom
Lu Wang
vanglullveellzznig@mall.co
Abstract
We find that the cross-entropy loss curves of neural language models empirically adhere to scaling law with learning rate (LR) annealing over training steps: [(s) Lo + A- ST C-S2, where [((s) is the validation loss at step 8, S1 is the area under the LR curve, S2 is the LR annealing area, and Lo. A, C, a are constant parameters. This formulation accounts for two main effects: (1) power-law scaling over data size, and (2) the additional loss reduction during LR annealing. Unlike previous studies that only fit losses at final steps, our formulation captures the entire training curve, allowing for parameter fitting using losses from any training step. Applying the scaling law with LR annealing and fitting only one or two training curves,
accurately predict the loss at any given step under any learning rate scheduler (LRS). This approach significantly reduces computational cost in formulating scaling laws while providing more accuracy and expressiveness. Extensive experiments demonstrate that our findings hold across a range of hyper- parameters and model architectures and extend scaling effect of model sizes. Moreover, our formulation provides accurate insights into empirical results observed in numerous previous studies, particularly those focusing LR schedule and annealing. This work is promising to enhance the understanding of LLM training dynamics while democratizing scaling laws, and it is helpful to guide researchers in refining training strategies for further LLMs.
Introduction
In recent years, large language models (LLMs) have garnered significant academic and industrial
attention (Brown et al. 2020 Touvron et al. 2023). The scaling law suggests that the validation loss of language models follows power-law pattern the size of the model and the data increase (Hestness)
ct al. 2017 Kaplan et al. 2020 Henighan et al. 2020). This law provides powerful framework for forecasting LLM performances before large-scall training by fitting losses smaller scales (OpenAl DeepSeck-Al 2024 Dubey et al. 2024). Numerous studies have explored the formulation to model the scaling effect LLMS in various different settings (Bahri al. 2021] Hernandez et al.
2023
202 1
Caballero et al. 2022 Michaud et al.) 2023 Muennighoff et al. 2023).
However, typical scaling law formulations focus only the final loss at the end of training (Hoffmann et al. 2022). Specifically, previous approaches generally rely on set of training runs and fit the scaling law curve solely the final loss from each run, while ignoring middle losses during training which do not follow traditional scaling laws. This approach underutilizes the training compute and fails to capture the training dynamics within each run. Further, the application of scaling laws in LLM developments is limited since the loss curve through the whole training process is not modeled. An expressive formulation that models full loss curves enables prediction of future training dynamics and also offers insights on the learning process of LLMs.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

1
433
Learning Rate x10-
S2 (Annealing Area)
4-2
A-a
A--
S1 (Forward Area)
24
2
3
4
5
7
10 Step
11
12
13
14
15
16
17
18
19 20
Figure 1:
Visualization of Si and S2 at the 20-th step of cosine LR scheduler. S is the forward area, i.e., sum of red grid areas; S2 the decayed annealing area, i.e., weighted sum of blue grid areas, where lighter shades indicate smaller weights. Both Si and S2 contribute to loss reduction, and balancing their values is crucial for achieving the lowest possible final loss.
In this study, we discover scaling law that models the full loss curve within complete LLM training run. Specifically, we dive deeper into the training dynamics during LR annealing, and incorporate LR annealing factor into the traditional scaling law formula to formulate the process. This design is motivated by the observed correlation between LRS and loss curves, where loss gradually decreases as we consume more training steps (proportional with data amount or number of tokens) and then sharply declines when the LR undergoes significant annealing (Loshchilov & Hutter 2016: Smith al., 2018 [brahim et al. 2024 Hu et al. 2024). We discover that the validation loss L(x) at step s determined by two main factors: the forward area S under the LR curve and the degree of LR annealing S2: ((s) Lo AA.ST S2.
S
= Te
S2 Σ i=1 =1
(M--1 M1) X
where 7), is the learning rate step i, and À is saper-parameter representing the decay factor for LR annealing momentum (see Sec. detail), which typically ranges from 0.99 to 0.999. Ln- A, C, a undetermined positive constants. Si is also known as the summed LR (Kaplan et al. 2020). and S2 represents the LR annealing area. A visualization of S₁ and S₂ provided as Fig.
Eq. [ describes how loss changes for each step in full loss curve during training. In Eq. I the term Lo + A SS represents a rectified scaling law that captures the expected loss decreases as a power-law function of the number of training steps. The
term -C S2 accounts for the further loss drop due to learning rate annealing. Remarkably, this simple formulation accurately describes the validation loss at any training step across various LRS and even allows for unseen LRS. For example, we can fit Eq. [ to the full loss curve of constant and cosine LRS with 20K total steps (Fig. 2. and then predict the full loss curve for various unseen LRS with longer total steps (e.g. 60K) (Fig.
predict the loss curve
We validate our proposed equation through extensive experiments and find that: (1) Our formulation performs consistently well across various aaper-parameters and model architectures; (2) Eq. can be extended incorporate other scaling factors, such model sizes; (3) Our proposed equation accurately fits the loss curves of open-sourced models; (4) Our formulation can be used to verify and explain numerous previous findings on LR annealing and scheduling.
In Sec. 3 we derive the scaling law formulation with LR annealing and elucidate the potential theory underpinning our formulation. Extensive experiments are conducted to validate the formulation. In Sec. we apply our formulation to verify and explain the empirical results from various previous studies. Our approach offers theoretical insights into the crux of loss drop, LR schedule, and LR annealing, enabling I.MM participants to better understand training dynamics of L.MM and select optimal training recipes in advance In Sec. we compare approach to typical scaling law

---

4.5
3.
1.5
C3sTe LBS, Grouns Loss
ca8Te maan
127
144
1.50
124
coanss JtS, Sound ThiSt usa
liting Cure
Corolent Leanrig fute Cute
132
conmbnt (AS, dround Trihh
Ccere Leartigg Fate Cane
141
"tiing curve
Spep
Step
(a) LR curves.
(h) Loss curves.
(c) Zoomed-in view.
Figure 2: Using Eq. [ to fit full loss curves yield by constant and cosine LRS. Total steps 20K,
Tmax
x 10 Imin 0. The fitted equation is L(s) 2.628 0.429 S S50.550
0.411 S2-
Learting Rats
Grant Fuin Less
239
Graand futh Lois Preeiition
Curce
115
237
Pun habcish Bru 1575%
140
2.00
200
(a) Full loss curve prediction of the cosine LRS (60K steps, M/mi 0.1 1/max)
Leaming Batr Cume.
Grann Tur Peededer curve
Graund
Frad ct Curve
336
230
2.30
2.-
547
0.00
12222
2000
1007
1200
A0005
12:00
2026
13:03
60000
15:03
(b) Full loss curve prediction of the multi-step cosime LRS (80% + 10% 10%) DDeepSekk Al 2024
Frediction Curve
200
Prediction
236
Yrah ind Erre 2316
14P
147
1392
2003
1000
1780
4010
M0. WE 2000 2010 40 aPo0 Malo
(c) Full Loss curve prediction of the WSD LRS (20% cosine annealing to 7mii
0) Hu et al. 2024
Ground Tort
Ground Truth
3.0
530
Frad Crore 1.222%
201
3000
Step
(d) Full Loss curve prediction of the Cyclic LRS (mmihh 2017
Figure 3: Using the fitted equation from Fig. 2 to predict full loss curves for unseen LRS with 60K total steps. The left, middle, and right columns present the LR curve, the loss curve, and zoomed-in view of loss curve, respectively. The red rectangle means the zoomed-in zone. Warmup steps (500) are not shown this figure. The fitted equation accurately predicts each loss curve, particularly for capturing the trend of loss changes as the LR varies. Notable, all LRS and loss curves shown here
were unseen during the fitting in Fig. 2 The mean prediction errors across different LRS as low ~ 0.2%.
3

---

formula, such as the Chinchilla scaling law (Hoftmann et al. 2022). We show that our formulation is more general and requires significantly less compute fit, which democratizes the development of L.MSs and scaling laws.
2 Preliminary
Scaling Laws. Cross-entrops loss of language models the validation set is reliable indicator of LLMs' performance on downstream tasks (Caballero et al., 2022 Du et al., (024). Kaplan et al. (2020) empirically discovered power-law relationship between validation loss 7. and three factors: model size N, dataset size D, and training compute. As an application scaling law, Hoffmann et al. (2022) developed Chinchilla, compute-optimal L.I,, by balancing model size and dataset size. They used simplified and intuitive scaling law equation: L(D, .N = Lo + A D + B N where Lo. A, B, a, B are positive constants. Traditional scaling law formulations fit only the loss the final training step, while ignoring losses from other steps. Collecting new loss value of data size requires launching a another training run with the same LRS, which resource-intensive.
Learning Rate Annealing. Learning rate annealing is a widely-used technique in training neural networks, where the learning rate progressively reduced from maximum to minimum value following a pre-defined LRS. Various LRS schemes have been proposed to improve the performance and stability of model training (Loshchilov & Hutter 2016). For example, the popular cosine LRS reduces the LR in a cosine-like pattern over full training steps. WSD LRS (Hu et al., 2024) keeps a constant LR for the majority of training, and applies annealing only in the final (e.g. 10% 20%) steps. In LLM training, it has been widely observed that more pronounced decrease in LR often results in more precipitous drop in the validation loss.
3 Empirical Observations and Experiments
In this section, we elaborate the origin, the intuition, and the experimental basis behind Eq. T We then validate our formula through extensive experiments.
3.1 Similarity between LR, Gradient Norm, and Loss
The first key observation that the shapes of LR curve, gradient norm curve, and validation loss curve are quite similar across various LRS when training LLMs (Fig. 4.. This suggests implicit connection between learning rate and loss, where gradient norm could be the bridge.
constant LRS is a special LRS, where every training step can be viewed as endpoint of the LRS. Notably, the Chinchilla scaling law (Hoffmann al. 2022) exactly fits losses of last steps, i.e., LRS endpoints. Therefore, it is expected that the loss of all steps under a constant LRS adheres to the Chinchilla scaling law, i.e., power-law over training step s.
Scaling Laws for Constant LRS.
Figure 4: The shapes of LR (top), gradient norm (middle), and validation loss (bottom) curves ex- hibit high similarity across various LRS (labeled as different colors).
Extra Loss Changes in LR Annealing. Un- like constant LRS, LR annealing or re- w-armuup brings significant local changes in the loss (see Fig. 4 causing the full loss curve to deviate from the power-law formulation that considers only the training steps s. We hypothesis that such loss changes can be captured by an additional LR (1) related term, i.e.,
L(s) = To + A --
where the first two terms follow traditional scaling laws, while the last term denotes the extra loss change brought by LR annealing. Recall the similarity between LR and loss curves, we can form an initial guess for f() f(n) = C 7, where C' is a positive constant.

---

2.75
3C
Aring 41
Creandhyr anndrmaaing
275
Grandlaam cas anraaling luninghain Locs
2.74
2.75
2.74
L nar Bakko X20
2.73
124 Star
2.75
2.72
2.72
2.73
Greand
24
Ora 2nd an in Mormasie
2.70
Lunving Arrr
0.0
Leaming Pate
Taringg
2.69
39000
43020
41003
42000
43000
46000
45000
40500
47000
47500
48030
48500
49000
49500
Steps
(a) Different delay steps the annealing process asso- ciated with different annealing steps (0.1K, 0.5K, 1K and 2K).
(b) Different delay steps the re-warmup process associated with different re-warmup steps (O.1K, 0.5K, IK and 2K).
Figure 5: The delay phenomenon between the LR and validation loss curves, suggesting that LR annealing (re-warmup) has momentum.
Training Discount in Annealing. The form of Eq. 2 is still imperfect. Note that the gradient norm [ig] decreases almost proportionally with LR during the annealing process (shown in Fig. 4 Thus, the amount of parameter movement (approximately 7) ||g|| per step) in the LR annealing stage declines an almost quadratic rate compared to stages before annealing. As the parameter movement become smaller, the change in loss also slows down accordingly. Therefore, the loss drop brought by the power law term (i.e., the first two terms in Eq. 2 should also diminish during LR annealing. This consideration leads to an improved form:
L(s) Lo + A
S
Σ n
where S₁ is the forward area, i.c., the area under the LR curve (as visualized in Fig. D. which could be approximately interpreted as the total amount of parameter updates.
3.2 LR Annealing Momentum
Another key observation is that LRR annealing has momentum. To refine the formulation of f(n), we design special LRS where the LR decreases linearly from Ymaz to Mmin and then increases. The increasing stage always has a fixed slope, reaching the maximum value in 5K steps, while the slope of the decreasing stage is varied, with durations of 0.1K, 0.5K, 1K, and 2K. Symmetrically, we design another LRS where the LR increases linearly from Tmmn to Mmar and then decreases. Fig. 5 shows the corresponding LR and loss curves. We observe delay phenomenon between the LR and the validation loss. Firstly, the turning point of the validation loss curve consistently lags behind the turning point of the LR curve, indicating that the validation loss continues along its previous trajectory for some steps even after the LR changes direction. Secondly, the steeper the slope of the LR annealing (or re-warmup), the more pronounced the delay phenomenon becomes. Thirdly, given the same L. slope, the left figure (where LR decreases then increases) consistently shows a longer delay compared to the right figure (where LR increases then decreases).
Interestingly, this phenomenon closely resembles the physical experiment of a small ball rolling down slope. The steeper the slope, the faster the ball accelerates. When the ball lands, the accumulated momentum causes the ball to slide further. Inspired by this delay phenomenon, we hypothesize that f()), the loss reduction induced by LR annealing, has cumulative historical formation so that the past change of LR will affect the following loss curve for a few steps. In summary, learning rate annealing exhibits momentum. To capture this, we define f(n) C- S2, where S2 is
mi = A A (T8-1 - -))), S2 Σ m. vi
(4)
where m, is the LR annealing momentum at step i (m² 0), and An 7/-1 1/2 denotes the LR annealing amount at step i. À is the decay factor that signifies how much historical information
retained. We find that À values between 0.99 and 0.999 generally works well. In contrast, X 0
5

---

implies no momentum effect, reducing f(n) to C. T/. which degenerate to the initial form mentioned above. Note that S2 applies not only LR annealing (S² > 0), but also to LR re-warmup (S2 0). This means that our equation is applicable to scenarios like continual pre-training, where LR re- warmup plays an important role improving outcomes. In practical scenarios, we find that is often not necessary to carry the stride term of in S2 because can be very small in most cases. Therefore,
we can simplify to S2 Σ
(Mk=1 m))) X See Appendix 1.3 and 1.4 for more explanations, The simplified form also presents an interesting definition based on visualization. As shown in Fig. S2 could be converted from Sj by LR annealing.
m,
3.3 Final Formulation
We formally present our formulation in this section.
Scaling Law with LR Annealing. Given the same training and validation dataset, the same model size, the same training hsper-parameters such as warmup steps, max learning rate Qman and batch size, the language modeling loss training step 8 empirically follows the equation M(s) = Lo + A ST - C S2, where Sı and S2 are defined in Eq. 1 Lo, A, C, a are positive constants. Note that the equation does not require the given max learning rate to be optimal.
Our formulation describes the loss of each training step across different LRS. It allows fitting based on a simpler LRS with shorter training steps and enables the prediction of validation losses for more complex LRS with longer training steps. Notably, loss curves with different max learning rates have different values of La, A, C, a, and our scaling law does not fit divergent and collapsed loss curves (e.g., overly large LR). We also discuss some possible corner cases (i.e., 17 = 0) in Appendix L.3
Loss Surface as a Slide. To better understand our formulation, we view the loss surface of language
models slide in Fig. 6 The optimization process be seen as sliding down the slide according to the power-law scaling (orange line), while oscillating on the inner wall (blue dashed line). When the LR anneals (red line), the amplitude of the oscillation decreases, resulting in reduction in loss.
Balance between S₁ and S2. Note that in Eq. T aL < 0 and aL < always hold, indicating that 852 increases both S and S2 help to reduce the loss. However, as shown intuitively in Fig. [ there exists delicate balance between S and S2. When LR begins to anneal and S2 starts to increase, the forward area S of subsequent steps starts to diminish instead. Our equation aptly describes this delicate balance. In Sec. we elaborate this topic in detail.
8S.
Loss Surface
3.4 Experiments
Constant
Annealing Area 5; Diraction
Aprealing
LR Warmup. LR warmup is important for training LLMs. During the warmup stage, neural networks are prone to random optimization, resulting in un- predictable outcomes (Hestness et al. 2017). Var- ious studies, along with our own pilot experiments (Appendix B), show that LR warmup significantly accelerates model convergence. High gradient norms usually observed during the LR warmup stage, especially in the initial steps of training (see Fig. 4 This indicates that model parameters undergo sub- stantial updates during this stage. Therefore. in all our experiments, linearly warmup LR reach Mmaz and compute S and S2 assuming constant LR value Mmaz in the warmup stage.
Power law Foward Area 5, Direction
Figure 6: Loss surface of language models as a slide after simplification. Optimization direction could be decomposed into two direc- tions: power-law scaling direction (S1, slid- ing down) and annealing direction (S2, inner height of the slide).
Experimental Setups and Fitting Details. We use standard experimental setups for LLM pre- training. To verify the robustness of our formulation across different experimental settings, we have

---

0.0500
N-B6.90 MMM
38
Groune Sruth LoSs
Loo Drew Droo (arrstlligg VA. No Aoceeiiingg
0.0475
N-261 SM
3.
37-
11aphe 11n 12TH0
Nin
14- 54.954
N--6616660 SM
0.0450
34-
35
PredictN-
0.0425
33
32
3.
0.0375
3.0
0.0350
29
2.8
0.0325
277
2.6
0.0300
10000
15000
20000
25000
30000
35000
40000
10000
20000
30000
40000
50000
60000
Stop
Stops
(a) The loss drop brought LR annealing for different model size Dashed lines represent trends over steps. The loss drop brought by LR annealing scales with data size and model size.
(b) Curve fitting and prediction cosine LRS (60K steps Mnaii 0.1 Mmmaa) of different model sizes using our N--xttnded scaling law. Results for N=1214.4M are predicted.
Figure 7: The loss drop brought by LR annealing (left) and the N-extended full loss curve fitting and prediction (right).
We adopt = 0.999 in all our experiments. We follow the fitting approach of Hoffmann al. (2022 obtain parameters in our equation (see Appendix C
five distinct experimental setups (see Appendix D
Fitting and Prediction Results. We fit Eq, I on the loss curves under constant and cosine LRS with 20K total steps (see Fig. 2 and then predict the full loss curves under several unseen LRS with 60K total steps (see Fig. 3). The results show an almost perfect fit, achieving coefficient of determination (R-) greater than 0.999. This underscores the robust capability of our equation to accurately fit loss curves across diverse LRS using a single parameter tuple.
The prediction results in Fig. iindicate that our formulation is broadly applicable and generalizes robustly across four unseen LRS, with equation can accurately predict losses even for complex LRS that include multiple LR re-warmup stages (Fig. 3d), despite that the loss curves used for fitting do not contain any LR re-warmup stages.
prediction error low 0.2%. Moreover, our
Extensive Experiments on Different Setups. To demonstrate the broad applicability of our proposed equation, we conduct additional fitting and prediction experiments using various setups. (1) We use an alternative set of training on the Mixture of Experts (MoE) architecture (Appendix(F.2)) (3) We apply our equation to predict loss curves for much longer training run involving 1.7B parameter model trained on 1.4T tokens (Appendix
(Appendix E.T): (2) We test our equation
E.3 (4) We fit the loss curves of open-sourced models, including BLOOM- 176B trained on 300B tokens (BigScience) 2022) and OLMo- trained on 2T tokens (Groeneveld et al. 2024) (Appendix E.4). All experiments produce excellent results, indicating that our equation effective across diverse experimental setups, including different training hyper-parameters, architecture, model sizes, and dataset scales. We also present the ablation studies on S₁ and S2 in Appendix E.3 which shows each component in formulation is important and indispensable.
3.5 Extension to Model Size Scaling
Loss Drop During Annealing Scales with Model Size N. We explore the effect of model size N on the loss drop during the annealing stage. Specifically, we compare the final losses obtained with constant LRS and a WSD LRS (10% cosine annealing to Tmmin = 0) to estimate the loss drop due to different total steps and different model sizes. The
LR annealing. We conduct this experiment experimental results are shown Fig. 7a It suggest that the loss drop from LR annealing scales with both annealing steps and model sizes. This implies that the annealing area S2 in our equation should also increase as the model size increases. We suppose there is simple relationship of S2 o N where y is positive constant.
7

---

Model Size Scaling, Building on the experiments and analysis above, we extend our proposed Eq. to incorporate model size scaling, based on traditional scaling laws:
L(s, Lo + A- + + N -
C- S2 -N",,,
(5)
where N is the number of non-embedding model parameters, and B. 8, are constants related to N. We realize S2 x No via a multiplier N to the original annealing term -C S2.
Fitting and Prediction with Model Size. We validate Eq. Eq.5lb by fitting the full loss curves of models with varying sizes. We then apply the obtained equation to predict full loss curve on the unseen largest model size. Results in Fig. sho an almost perfect fit (R²² >> 0.998) and prediction for entire training dynamics of larger-scale models. This indicates the effectiveness and robustness of our proposed N N-extended equation. Additional N N-extended experiments with other setups further confirm the robustness of our formulation (see detail in Appendix E.6).
Application
We apply our proposed formulation to validate and provide an accurate explanation for existing experimental findings on the training dynamics of LLMs. These key insights also help select critical LRS before starting model training. An interesting summary is that
The art of learning rate schedule lies in the delicate balancing act forward area and annealing area.
Determining Parameters in Cosine LRS Many papers have found that in LLM pre-training using
cosine LRS, setting the cosine cycle length T as the total steps S, and setting min LR as nearly (rather than 10% max LR) can lead to the optimal loss (Hoffmann al. 2022 Hu et al. 2024 Hägele et al.) 2024 Parmar et al. 2024). It becomes quite easy to determine parameters via our proposed scaling law. For example, in our experimental settings, the predicted loss curve with and a minimum LR of O indeed achieves the optimal loss in the final step, shown in Fig. 20 in Appendix HH Moreover, our equation gives a quite intuitive explanation: setting T > S leads to incomplete annealing, while T < S leads to small forward area S due to early annealing. Thus, the optimal configuration is to set T equal to S. Also, setting the minimum LR to 0 maximizes the annealing amount, thereby increasing the annealing area S2, which facilitates lower final loss.
Why Is WSD LRS Better than Cosine LRS? Recent studies have shown that WSD LRS (Hu et al., 2024) and multi-step cosine LRS (DeepSeek- All 2024) result in lower loss compared to the traditional cosine LRS. Our experiments also support this finding (refer to
2.85
Stepo
40c Staps
EOC Stegs
2,80
EGC Stape
Staps
the ground-truth loss in Fig. 3a
We validate and elucidate this finding using pro-
3b
3c).
velidstton Loss
2.75
posed equation. Fig. 22 in Appendix H shows
the learning rate curve (left) and the predicted loss drop (right) for different LRS. The results suggest that for WSD and multi-step cosine LRS, the negative Sy-item (-C- -2) is slightly larger than that of the cosine LRS, whereas the S1- item (A ST " is significantly lower. Specif- ically, both the WSD LRS and multi-step co- sine LRS unintentionally employ strategies that marginally reduce S2 but substantially increase Sr. leading to an overall decrease in validation loss.
2.70
0.0
0.2
0.4
0.6
0.8
1.0
Ratio Annealing Staps
Figure 8: Illustration of the predicted loss in rela- tion to the ratio of annealing steps in WSD LRS (cosine annealing), presenting parabola-lide like curves, with distinct optimal loss.
Optimal Annealing Ratio of WSD LRS In
the case of WSD LRS, it is crucial to ascertain the optimal annealing ratio for training steps. Hägele et al. 2224) found that there is an optimal annealing ratio for WSD LRS. This phenomenon can
8

---

be further elucidated through our proposed equation. Fig. X (and Fig. 21 depicts the final loss predicted by our equation for various annealing ratios and total training steps. The predictions form parabola-like curves, and align well with the actual experimental results reported in previous studies. This suggests that moderate annealing ratio, typically around 10% to 20%, is optimal, as it well balances I and 22, thus minimizing the overall validation loss. Moreover, our equation can directly predict the optimal annealing ratio for different total steps without large-scale experiments, which saves a lot of resources.
Many Other Takeaways Moreover, we use our equation to verify and explain more phenomena as follows: (1) Appendix H.I: An empirical reason of loss dropping more sharply when LR an- neals (Loshchilov & Hutter 2016; Ibrahim et al. 2024; DeepSeck-Al 2024). (2) Appendix H.2 the comparison between constant and cosine LRS, aligned with previous works (Hu et al. 2024). (3) Appendix H how to choose the optimal annealing function in WSD LRS, aligned with previous works (Hägele al., 2024). (4) Appendix and H.S how re-warmup (including re-warmup peak LR and steps) in continual pre-training, aligned with previous works (Gupta et al. (2023). Given the instances above, believe that equation can help analyze and select more training recipes in specific scenarios.
5
Comparison with Chinchilla Scaling Law
5.1 Reduction to Chinchilla Scaling Law
Our scaling law can predict the full loss curve across various LRS. In this section, we show that our equation is not contradiction with traditional scaling laws and is a generalized form of the Chinchilla scaling law (Hoffmann et al. 2022). Specifically, all the final loss values for different total training steps following our equation should also follow a power-lay relationship. We prove this by dividing two conditions: (1) constant LRS, and (2) other LRS.
Constant LRS. In the case of a constant LRS, the annealing area S2 is always zero and the forward area S] = ?maz s, where s is the step, and Mmaz is the constant maximal LR. Thus, the whole train loss curve becomes: L(s) Lo + (A Mmaz)) .s Lu s'-s which aligns with the Chinchilla scaling law equation.
Other LRS. For non-constant LRS, we use sta- tistical approach to show that our equation can be reduced the Chinchilla scaling law. Specifically, we verify whether the Chinchilla scaling law ade- quately fits the endpoints of loss curves predicted by our equation. The parameter tuple of our equa- tion is (Lo, A, C, a). We randomly sample differ- ent parameter tuples (detailed in Appendix F.I.
Table 1: Mean and std of R² for parameter fits.
LRS
mean( f
std(
Huber Loss
Cosine WSD
0.972
0.056 0.053
0.00017
0.979
0.00013
Each parameter tuple represents synthetic fitting result corresponding to distinct set of experimental setups (e.g., dataset, model size, etc.). For each sampled parameter tuple, we apply our equation to predict the final loss for different total training steps with both cosine and WSD LRS, and then employ the predicted losses to fit the Chinchilla scaling law. We calculate the mean and standard deviation of I2 values, and huber loss for each fit. The results in Table demonstrate that Chinchilla scaling law fits well on the data predicted by our scaling law equation. Thus, our equation can be considered a generalization that can be reduced to the Chinchilla scaling law.
5.2 Scaling Laws Democratization
Our scaling law equation allows to utilize all loss values from full loss curve during training, while traditional scaling laws can only collect single data point from the full loss curve. This feature allows us to fit scaling laws with much less cost. For direct comparison, we compare the computational efficiency of our approach and the Chinchilla scaling law (Hoffmann et al., 2022). Specifically, we assume to collect 100 data points for parameter fitting, and estimate the computational costs needed to fit the respective scaling law equations under different LRS configurations (see Table More details can be found in Appendix F2 The results indicate that proposed equation uses less than

---

Table 2: The comparison of computational cost for fitting different scaling law equations.
Formulation
LRS
Computational Cost
Applicable to Other LRS?
Chinchilla
Cosine
100%
No
Chinchilla
WSD (20%)
21.6%
No
Chinchilla
WSD (10%)
11.8%
No
Ours
Any
< 1.0%
Yes
1% of the computational cost required by the Chinchilla scaling law. Further, our scaling law with LR annealing, can be universally applied to predict loss curves for unseen LRS, thus conserving even more computational resources. This approach significantly democratizes the study of scaling laws in LLM pre-training, paving the way for a more environmentally friendly and methodology.
6 Discussion
Firstly, we analyze the impact of the decay factor of our equation in Appendix I.1 and it suggests that selecting proper decay factor is important for determining the balance point between S1 and Sy; Secondly, we analyze the root reasons of the delay phenomenon mentioned in Sec. B Appendix 1.2 It suggests that neither the Adam optimizer (Kingma & Bal 2015) nor Si are the root reasons and this can be an important future work; Thirdly, we discuss potential variations of our proposed equation (e.g. 17 = case and L x S variant), and investigate other possible scaling law formats with LR annealing in Appendix 1.3 and L.4 The results validate the superiority of our proposed formula.
As for theoretical justification, we acknowledge that our scaling law derived primarily from empirical observations rather than rigorous mathematical proofs. While theoretical analysis would strengthen our claims, we note that such analysis faces significant challenges real-world L.LM training scenarios. The complexity of modern optimization dynamics Sernaiics-ivvolving adaptive learning rates, momentum, batch normalization, and high-dimensional non-convex landscapes- -makes closed- form theoretical analysis extremely difficult.
is worth noting that nearly all influential scaling laws in the field, including the seminal OpenAI scaling law Kaplan et al.) (2020) and Chinchilla scaling law Hoffmann et al. (2022). are fundamentally empirical in nature. Despite the absence of formal proofs, our proposed scaling law demonstrates remarkable consistency across: (1) Multiple model architectures (125M to 1.2B parameters, dense and MoE models): (2) Various learning rate schedules (constant, cosine, WSD, etc.); (3) Different training configurations (batch sizes, datasets, training horizons, sequence length, etc.); (4) Validation on open-source models (BL.OM and OLMO). Our prediction errors are consistently below 0.35%. This extensive empirical validation, combined with the interpretability of our formulation (where S captures power-law scaling and S₂ models annealing momentum), provides strong evidence for the reliability of our approach. We believe that developing rigorous theoretical foundations for such scaling laws represents an important direction for future research.
Conclusion
In conclusion, we discover that the loss curves of neural language models empirically adhere to a scaling law with LR annealing over training steps s: L(s) Lo A A-S- S S.. This equation can accurately predict full loss curves across unseen LR schedules. We present the underlying intuition and theory for deriving our equation and demonstrate that our approach can be extended to capture the scaling effect of model sizes. Extensive experiments demonstrate that our proposed scaling law has good accuracy, scalability, and holds under various experimental setups. also offers accurate insights to the training dynamics of LLMs, and explains numerous phenomena observed in previous studies. We believe that the scaling law with annealing promising to reshape the understanding of researchers for LLM training and scaling laws.
10

---

References
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., and Sharma, U. Explaining neural scaling laws. Proceedings
of the National Academy of Sciences of the United States of America, 2021. doi: 10. 10.1073/pnas 2311878121.
BigScience. Bloom: A 176b-paramees open-access multilingual language model. arXiv preprint arXiv: 2211.05100, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, TT, Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M.. Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL B www.reiiinnnnnnnnnnoeeeeeeeee neurips..... cttps.cccpppprr #111:/pppp020000//11/44450000611
Paperr pdf
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. International
Conference Learning Representations, 2022. doi: 10.48550/rxXi.2220...22
Computer, Redpajama: an open dataset for training large language models, 2023. URL URL https
com www.o//toooeeeeoooooooooooooeeee (ocoopputer//PPaaa RedPajj h/ggghhbb..cm///gggteeeer.r...
DeepSeek-AL. Deepseek Ilm: Scaling open-source language models with longtermism, arXiv preprint arXiv: 2401.02954, 2024.
Du, 7., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv: 2403.15796 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A.,
Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv: 2407.21783, 2024.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv: 2101.03961, 2021.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney. R., Tafjord, O., Jha, H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A.. Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zetrlemoyer, I.., Dodge, J., Lo, K. Soldaini, L.,, Smith, N. A., and Hajishirzi, H. Olmo: Accelerating the science of language models. arXiv preprint arXiv: 2402.008318 2024.
Gupta, K., Thérien, B., Ibrahim, A., Richter, M. I.., Anthony, Q., Belilovsky, E., Rish, I,, and Lesort, T. Continual pre-training of large language models: How (re)warm your model?, 2023. URL https://arxx: http:///axxvv..raaaaa/o 2308 04014
Henighan, T., Kaplan, J., Katz, M., Chen, M.. Hesse, C.. Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint
arXiv:2010.1470) 2020.
Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 1712.00409, 2017.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K. van den Driessche, G., Damoe, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, W., Vinyals, O., and Sifre,
L. Training compute-optimal large language models. arXiv preprint arXiv: 225561555,, 2022.
11

---

Hu, S., Tu, Y., Han, X., He. C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., Zhang, X., Thai, Z. L., Zhang, K., Wang, C., Yao. Y., Zhao, C., Zhou, J., Cai, Zhai, Z., Ding, N., Jia, C.. Zeng, G., Li, D., Liu, 7., and Sun, M. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv: 2404.06395, 2024.
Huber, P. Robust Estimation Location Parameter. The Annals of Mathematical Statistics, 35 (1):73-101, 1964. doi: 00.1214/amm/111770055 1177703732
https://:: org/oo////// 1tt///1..111////11 aoms/
Hägele, A., Bakouch, E., Kosson, A., Allal, L. B., Werra, L. V., and Jaggi, M. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv: 2405.18392 2024.
Ibrahim, A., Thérien, B., Gupta, K., Richter. M. L., Anthony, Q. G., Belilovsky, E., Lesort, T., and Rish, I. Simple and scalable strategies to continually pre-train large language models. Trans. Mach. Learn. Res., 2024, 2024, URL htthttps.:. http://oppeeeivveeeeeeeeee wwwtteeeurneeeeee dit.rrummidDDimPeeeeeiii
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A. Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv: 2001.08361, 2020.
Kingma, D. P. and J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. Uttt:t::://arat orgg org/abs/g/aaoaaoo ha////bs///111
6980
Kosson, A., Messmer, B., and Jaggi, M. Analyzing & eliminating learning rate warmup in GPT pre-training. In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning, 2024. URL httpps/ /pwpr/pppreeeeeeeeeeeeeeeeeeeeee httpa/////pppeeeeeeeeeeeeeeeeeee Sww.rrrsrnnndddeeeSSSSSSSSS
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. /ttp http//oruumid-rrgz2aaaal
Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts, International Conference on Learning Representations, 2016.
Loshchilov, I. and Hutter, Decoupled weight decay regularization. arXiv preprint arXiv: 1711.05101, 2017.
Michaud, E. J., Liu, Z.,, Girit, U., and Tegmark, M. The quantization model of neural scal- ing. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neu- ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA. USA, Decem- ber 10 16, 2023, 2023. URL http:ht::///peeee nips cc/ 566344005553444c2255556333 htwp:///e22eaaaaa-----rrrrrraeaa Abstract- wow.ernneeeeeeeceecoooo
erence. html
Muennighoff, N., Rush, A. M., Barak, B., Scao, L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL. hutpp:///penreriew.nne wtwp:eewwwwweeeeeouuneuu.... J5BuTrE]35
Nocedal, J. Updating quasi newton matrices with limited storage. Mathemutics of Computation, 35 (151):951-953) July 1980. ISSN 0025-5718. doi: 10. 10.000000025-518--1100 1970-07855-7
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Parmar, J., Satheesh, S., Patwary, M., Shoeybi, M., and Catanzaro, Reuse, don't retrain: recipe for continued pretraining of language models. arXiv preprint arXiv: 2407.07263, 2024.
Penedo, G., Kydlíček, H., allal, I. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale, arXiv preprint arXiv: 2406. 17557, 2024.
12

---

Smith, L. N. Cyclical learning rates for training neural networks, In 2017 IEEE winter conference on applications of computer vision (WACV), pp. 464 472. IEEE, 2017.
Smith, S. L., Kindermans, P., Ying, C., and Le, Q. Don't decay the learning rate, increase the batch size. In 6th International Conference Learning Representutions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::/openreevee. net.
Touvron, H., Martin, L.., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P.. Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.0922:
2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv preprint arXiv: 1706.03762, 2017.
13

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: The abstract and introduction clearly and accurately reflect the paper's contri- butions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
14

---

This paper primarily focuses on empirical contain many theoretical results. We provide the assumptions in Section 3.3, and present how we derive our empirical scaling law in Section 3.1 and Section 3.2. We validate our scaling law via experiments mainly in Section 3.4.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We clearly present our experimental settings Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our experiments conducted under the standard and popular framework, Megatron, which is a fully open-sourced and reproduciable pre-training library. The primary training dataset, Fineweb, is also open-sourced and reproduciable.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc///pp/ httpss/ccccciigggcieeeee///////e htwbliiccgu.ddes/ddduubiiii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We present the experimental settings and details in Table 3 of Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report all the R² of our fitting results and the prediction errors of our prediction results. Please see the scaling law fitting/predittion figures this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We present the experiments compute resources in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:///rrpp...cpppp/b///cccaaea
Answer: [Yes]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the social impact in the Appendix A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not involve such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The methods in this paper do not rely on LLMs; instead, we focus on training language models from scratch and investigating scaling laws.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
A Limitation and Social Impact
One main limitation of our work is that our scaling law is primarily based on empirical analyses and experimental verifications. We acknowledge that there lack of rigorous theoretical analysis and proof because it is difficult to build theoretical deduction in non-toy environment with thousands of LLM training factors. However, based on our extensive experimental verification, our scaling law can reasonably reflect the learning dynamics of the pre-training process, and extrapolate to unseen learning rate schedules and longer training steps with much fewer computational cost. Other limitations lie in that we only conduct experiments on text modality and do not confirm there would be similar laws for other modalities like vision, audio, etc.
This study presents technical contributions to the field of large language models and scaling laws. While acknowledging that most research may have broader societal impacts, this work does not involve specific social impact that warrant particular discussion in this context.
B Impact of Warmup Steps
We conduct experiments on the impact of learning rate warmup steps. As shown in Fig. 9 we find that 500 warmup steps can speed up convergence, and get the lowest validation loss compared 100 or no LR warmup. The finding is aligned with previous works Liu et al. (2020); Kosson et al. (2024).
The experimental results also guide to choose 500 warmup steps in the main experiments this work
C Fitting Details
Given a learning rate scheduler, we can easily compute out S1 and Sy of each step in advance. To estimate A, a.), we adopt a similar fitting method as Chinchilla scaling law (Hoffmann)
et al. 2022). Specifically, we minimize the Huber loss (Huber, 1964) between the predicted and the observed log loss using the L-BFGS algorithm (Nocedal) 1980):
min Ag,A,C,a
Σ
Hubers (log L(i) - log L(i))
(6)
Step
We implement this by the utilization of minimize in scipy library. Huber loss is to enhance to robustness of the fitting results and we set d of Huber loss as 1.0 10 10 We mitigate the potential issue of local minima of fitting by choosing the optimal fit from range of initial conditions. Note that in practice, we can also fit the full loss curves using multiple LRS with a single tuple of (Lo, a). In this situation, we the Huber losses in Eq. of all fitted LRS.
Note LR warmup in training from scratch is different from LR re- re-warmup in continual training, where we do not regard re- re-warmup steps as hlyee-pprameee and will show how to apply our equation to find optimal warmup recipes in Appenidx. H.4 and H.5
20

---

Steps Warmup Ground Truth Loss
100 Steps Warmup Ground Truth Loss
3.8
Steps Warmup
3.6
3.5
Salldston LO3S
3.4
3.3
3.2
3.1
2.9
2.7
2500
5000
7500
10000
12500
17500
Step
Figure 9: The comparison of the true loss curve of different warmup steps. We experiment on cosine LRS with 20K total steps.
D Experimental Setups
In this work, multiple sets of experimental setups, in order to validate that our equation work across different experimental setups. For clarification, we present the experimental setup list as shown in Table In most experiments, we also successfully fit our equation on BLOOM's and OLMo's ' loss curves, and their experimental settings totally different. Refer their papers for the experimental settings (BigScience) 2022 Groeneveld et al. 2024).
the main setting A. Other than the five settings,
We use the standard and popular pre-training library, Megatron, wwt.onnttttttttttttttette com/NVIDIA. Megatron-LM) as our training framework. For compute resource, we use GPU cluster of A100. The experiments primarily conducted using 16 x 8 GPU cards.
E
Our Scaling Law on Extensive Experiments Setups
E.1
Another Set of Training Alype-pparameter
Fig. and Fig. 3 show that equation work very well under main experimental setup. For proving that our scaling law with LR annealing can apply to different (but given) experimental settings, we change the setting from to (refer Table 3) and observe whether our equation can still work or not. The fitting results are shown in Fig. 10 The prediction results are shown in Fig. II The results suggest that our scaling law with LR annealing can still work well across different experimental setups.
E.2
Experiments on Another Architecture: MoE
Fig. and Fig. show that our equation can work very well on the dense Llama-like architec- 2017 Touvron et al. 2023). We prove that our scaling law can also apply to different model architectures and we replace Dense model with Mixture of Experts (MoE) architec- ture. We add widely-used auxiliary loss do load balancing among experts (Fedus al.) (02T). The experimental setting is shown Setting D in Table Moreover, we change the LRS and total steps to 60K WSD with 10K annealing steps in fitting, testing whether our scaling law is effective under various circumstances. The fitting results are shown in Fig. T2 while the prediction results are shown The results suggest that our scaling law can with LR annealing can still work well on MoE
ture Vaswani
Fig. [ architecture.
21

---

Table 3: Experimental settings adopted in this work. Model size denotes the number of non- embedding paramters. Our datasets include Fineweb (Penedo et al. 2024) and RedPajama-CC (Com- puter 2023). denotes pre-training multilingual dataset including mixture of sources such as common
crawis. books, arxiv, code, etc. We use AdamW W Optimizer (Kingma & Bal 2015| Loshchilov &
Hutter 2017), denoted as AO. Most experiments adopt Llama-3's tokenizer (Dubey et al. 2024). Ext
Ilama-2 2's extended from Llama-2's tokenizer (Touvron et al. 2023) by adding vocabulary.
Setups
Setting A (mainly)
Setting B
Setting C
Model Size
594M
293M
multiple
Train Dataset
Fineweb RedPajama-CC
Finweb
Mixture-trail
Val Dataset
RedPajama-
Mixture-valid*
Total Steps
60K
120K
143K
Maximal LR
2 x 10
2 x 10
1.381 x 10-3
Warmup Steps
500
100
500
Batch Size (tokens)
4M
2M
4M
Sequence Length
4096
4096
4096
Tokenizer
Llama-3's
Llama-3's
Ext Llama-2's
3,,82 in AO
0.9, 0.95
0.9, 0.95
0.9, 0.95
Weight Decay
0.1
0.1
0.1
Gradient Clip
1.0
1.0
1.0
Setups
Setting (MoE)
Setting E (1.4T tokens)
Model Size
8 x 106M
1704M
Train Dataset
Fineweb
Mixture-train
Val Dataset
RedPajama-CC
Mixture-valid
Total Steps
60K
350K
Maximal LR
2 10
6 x 10
Warmup Steps
500
1000
Batch Size (tokens)
4M
4M
Sequence Length
4096
8192
Tokenizer
Llama-3's
Llama-3's
31,82 AO
0.9, 0.95
0.9, 0.95
Top-k Experts
2
Auxiliary Loss
0.01
Weight Decay
0.1
0.1
Gradient Clip
1.0
1.0
2,00
35
Constant Grund Futh
ciai Curve
3.4
Constant (BS Grund
Predeciio Curve
Codin [B5 Ground
Couhae Ground Trthh
Prod
Pradiction
3.-
312
35
2 casPSt11 -uuees
250
Constant Leatting Costne Leaming
33
257
25-
2500
20289
Stap
Stop
Step
(a) Learning rate.
(b) Zoomed-out loss fitting.
(c) Zoomed-in loss fitting.
Figure 10: Full loss curve fitting on cosine (30K steps Mmin and constant LRS. The figures omit the warmup in the first 100 steps. After fitting. we get a universal loss equation = 2.761 + 0.517 0.4499 0.458 S2. Refer to setting in Table for setups.
22

---

15
Ground Truh Loa
18
Prediction Ourve
152
250
1.50
1.22
2.3
0.75
2.00
32
34
2.6
2.2
Vigmi Prodccto bno - 2.04%
246
0.2
Learning Rata Curva
23
0.00
25003
(0003
6206
10000
200068 12030
2030
00803
9.000
19300
120.30
2.8
37883
10493
110.0
100000
(a) Full curve prediction of constant LRS.
Ground Futh Loss
fredkction Curve
332
6.75
33
3.-
0.50
053
Moan
0.25
Loanning Rato Curve
6:00
20003
40300
00890
03000
100000
10000 2353
330CE +0008 58000 6900 TR0CK 0680058900
20000
60300
30003
$8900
Step
Step
Step
(b) Full curve prediction WSD LRS (90K total steps; 10 cosine annealing to 7/mnin
Ground Iruh
Ground
Prediction
2.60
Prediciin Curve
3.62
33
3.5
31-
24-
Maas Prodition Ener 0.235%
Learning Rane Curoe
0.00
ac000
4200 06830
2030
66893
13233
18206
133000
Step
Skep
Step
(c) Full curve prediction of WSD LRS (110K total steps; 10 % cosine annealing to Yjmin
Figure 11: Full loss curve prediction (120K steps) by the universal loss curve equation across various LRS, fitted in Fig. 10 The left, the medium, and the right figures in each row are learning rate curve, zoomed-out loss prediction, and zoomed-in loss prediction, respectively. The red rectangle means the zoomed-in zone. The figures omit the warmup in the first 100 steps. Please note that these predictive results, which means that none of the points in this figure (except constant LRS) are involved in the fitting process. The mean prediction errors across various LRS are low to ~ 0.2%. Refer setting B in Table for experimental setups.
2/00
Ground Loss
Predittion Cutra
3.75
Ground Futh Loss
1.40
Predctiin Curve
2.55
1.22
3.4
WSD Ground Less
WSD Predction Cares
2337
2.4
0.75
3.30
32
Cara Lnd
42425 10000-00000--3300 34152
264
145
235
Codia Ground Futh
Cndta Preditioon Curoe
Canelatt Grund bith
Courne Leaming Rana Curve
24
203
Constart Pradiciion Curve
Learning hace Curvs
16-
222
Ground Tuth Loss
W3D Learning Rate Curve
21
242
wSD Predetion Curve
0.00
1300e
2008a
2003
10893
56830
68300
2230
23000
35230
42300
5000
45000
280
4ice sate 4300 So09 5a500 Stak 3250 tas
(a) Learning rate.
(b) Zoomed-out loss fitting.
(c) Zoomed-in loss fitting.
Figure 12: Full loss curve fitting on MoE model. After fitting, we get a universal loss equation 2.801 + 0.424 S-06119 0.347 S2. Refer to setting D in Table for experimental setups.
23

---

Leaming Rate
Ground Futh
255
254
Ground Fith
Fredcciion Curve
Predition Curve
11
142
26
33
15-
5
2.66
Veen Fradiciie tror -or--3a 0.191%
243
Stap
Step
Step
(a) Full curve prediction of multi-step cosine LRS (80% + 10% + 10%)
DeepSeck A 2024
leaming Rate Curse
Futh
2.5
Ground Futh Loss
2.0
Prediction Curve
2.5
2.53
333
25
Mear Proicce Enor 5.134%
252
Step
Step
Step
(b) Full curve prediction of WSD LRS (17% exponential annealing to Mann
0) Hu et al. 2024
Lesming
2.0
33
0.5
15-
22-
2.00
23
3.6
Step
Step
Step
(c) Full curve prediction of Cyclic LRS Smith 2017
Figure 13: Full loss curve prediction on MoE model by the universal loss curve equation across various unseen LRS fitted in Fig. 12 The left, the medium, and the right figures in each row are learning rate curve, zoomed-out loss prediction, zoomed-in loss prediction, respectively. The red rectangle means the zoomed-in in zone. The LR curve figures omit 500 warmup steps. Note that these are all predictive results, and none of the points in the figures are involved in the fitting process. The mean prediction errors across various LRS are low to ~ 0.2%. Refer to setting D in Table 3 for experimental setups.
E.3 Scaling Up: Prediction for Much Longer Steps
Our equation has proven its utility in predicting the validation loss over a significantly large number of total steps. This scalability feature is particularly useful in handling large-scale training scenarios.
To illustrate its effectiveness, we apply our equation predict the loss curve during the annealing stage of the training process. The model we train is a sizable 1.7 billion parameter model, and the training involved a tremendous number of 1,400 billion total training tokens. This is considerable scale that tests the practicality and effectiveness of our equation. The specific experimental setup is Setting E, which can be found in Table
The fitting and prediction results are shown in Figure 14 and Figure 15 respectively. It shows that we successfully get to know the loss curve in the critical annealing stages after 10x longer steps in advance, which is crucial to handle the relationship between training dynamics and training recipes. For example, Llama-3 adopts annealing do pre-training data selection (Dubey al. 2024).
E.4 Open-sourced Full Loss Curves
For further verification for our proposed scaling law, we apply our equation on open-sourced language models and the corresponding full loss curves, including BLOOM-176B (BigScience) 2022) and
24

---

Table 4: Ablation studies on S1 and S₂ in our scaling law formualtion. The prediction errors on different LRS for each form are reported.
Scaling Law Forms
Cosine
Multi-step Cosine
WSD
Cyclic
[(s) Lp A S1 S
0.159 %
0.176 %
0.235%
0.322%
w/o S1: L(s) Lo + AA. - C. S2
-C
0,465%
0,458%
1.162%
1.139%
w/o S2: L(s) = Lo + ST
1.261%
1 1.265%
1.519%
1.203%
OLMo-1B (Groeneveld al., 2024). As shown in Fig. [6 our equation also fits very well on the open-sourced model training curves, even when the model size scales up to 176B (e.g. BLOOM) and token number scales up to 2000B over 740K steps (e.g. OL.o)..
E.5 Ablation Studies S1 and S2
In Sec. 3 we present the strong capability of our proposed scaling law. The formulation of our scaling, [((s) Lo + ST C. S2, contains two key components including S₁ and Sg. this section, we conduct ablation studies on S1 and S2. Specifically, we compare the forms without S₁ or 52 using setting For each format, we re-fit the full loss curves under 20K cosine constant and re-predict the full curves on longer steps under different LRS. The prediction error results are shown as Table The results indicate that the prediction error increases significantly the absence of either S S2 and suggest that each component in our scaling law is important and indispensable.
E.6 Our N-extended Scaling Law on Another Experiments Setups
Fig. 7b show that our N-extended equation can work very well on our main experimental setup. Similarly, for proving that our N-extended scaling law can apply to different (but given) experimental settings, we change the setting from to C (refer to Table 3) and observe whether our equation can still work or not. The fitting results are shown in Fig, 17 The results suggest that our N-extended scaling law with LR annealing can still work well across different experimental setups.
F
Comparison with Chinchilla Scaling Law
F.1
Reduction to Chinchilla Scaling Law
We have proved that scaling law can be reduced to chinchilla scaling law for constant LRS in Sec. 5 For other learning rate schedulers, we adopt a method based on statistics to show that our scaling law function be reduced to the chinchilla scaling law. Specifically, chinchilla scaling law fits well the endpoints loss curves predicted by our scaling law. The parameter tuple of our equation is (Lo, A, C, a). We then randomly sample 1000 sets of parameter tuples in some uniform distributions: Lo (11,3), A ~ (0.3, 0.5). C (0.2, 0.6), a U-00.6, 0.4).
check whether
Each parameter tuple could be seen as the fitting result of distinct set of experimental setups (e.g.
dataset, batch size, model size, etc.). For each generated parameter tuple, we apply our equation to predict the final loss of different total training steps on two LRS including cosine and WSD (10% annealing ratio). range from 5K steps to 60K steps. We conduct the prediction on two LRS including cosine and WSD (10% annealing ratio). The predicted final loss points are used fit the chinchilla equation through minimizing the Huber loss. The fitting examples are shown in Fig. 18
F.2 Scaling Law Computational Cost Comparison
We suppose scenario where it requires 100 fitting points get the parameters of scaling laws. We assume the distance between each point as K steps. We compute the required training steps using different approaches as follows:
It's 's worth noting that some of these sampled parameter tuples might not be reasonable or likely to happen in real-wor hol-world scenarios, but choose to keep them nonetheless.
25

---

225
158
3.5
3.
Frita Gater
125
Lite conarss 111.2211-4400 5J30792
0 10
Leambng
33-
16
Lass
43
556
30
188
Step
Step
Step
(a) Learning rate.
(b) Zoomed-out loss fitting.
(c) Zoomed-in loss fitting.
Figure 14: Full loss curve fitting on 30K Steps. After fitting, we get universal loss equation
L 2.788 0.906 -5-0.116 0.254 S2. Refer to setting E in Table 3 for experimental setups.
Graund Lou
Curve
Prediction Surve
Learting
376
23-
Learring
SE2K JEROK SUOEE JUERU JAOCE JXERS 18830
24
3080
30000
180000
130008
201800
10008 303000
12287
A6ERT
25000
(a) Learning rate.
(b) Zoomed-out loss prediction.
(c) Zoomed-in loss prediction.
Figure 15: Full loss curve prediction (350K steps) by the universal loss curve equation under WSD LRS (10% cosine annealing ratio to Troin 0). We adopt equation and accurately predict the loss curve the annealing stage after the 10x longer steps. This is meaningful to the development for large-scale LLM pre-training. Refer setting E in Table 3 for experimental setups.
Bloom Ground Truth Loss
2.9
Ground Truth Loss
Bloom Fitting
OLMo Fitting Curve
True Validation Loss
2.8
True Validation
3.0
Fitting Curve:
R² 0.99940
1.918 0.688-51" 1832552
2.7
2.6
Fitting Curve:
Lis) = 2.152 1.067551 (-0.555)))) 0.205
00999996
2.0
2.6
2.4
2.4
2.3
2.2
2.1
20000
40000
60000
80000
100000
150000
300000
450000
600000
Steps
Steps
(a) Full loss curve fitting on BLOOM-176B.
(b) Full loss curve fitting on OLMo- IB (2T tokens).
Figure 16: Open-sourced full loss curve fitting using our proposed equation, which shows that equation has strong scalability on model size and token number. We extract the curve of BLOOM from hbatpp/://agggnngaaa....o///giee
wtw.a//////.ooooeeesoorrrrra and we choose the column loss validation as validation loss. We extract the curve BBtnw-mwusrrdirkgr and we choose the
of OLMo from httpp: wttp:/dbb.aaaaiilliaaaaaa column eval/ppile/CrossStttogg/ as validation loss. Both models adopt cosine LRS.
26

---

3.9
Ground-truth Loss
3.8
N=18.68M
3.7
Rtting Curve: Loss 1,422 (.9.7551000778
N .34
+7.562 (1-0.137) 0.00633....099
3.6
N=106.2M
R-
3.5
N=161.9M
3.4
2M293M
y diicata Los
3.3
-u 20000000000000000oooo www.
4=401M
3.2
N=594M
3.1
N=1720M
3.0
2.9
2.8
2.7
2.6
2.5
2.4
2.3
2.2
2.1
0
20000
40000
60000
80000
100000
120000
140000
Steps
Figure 17: Curve fitting cosine LRS (143K steps to Mmin = 0) of many model sizes using our scaling law extended model size N. Refer to setting Table for experimental setups.
Fitted Chinchilla
Fitted
10.0000 1000
Loss 2:230 + 4.90515tep "( (0020077
Loss 2,261 SL420"Stopp(
0,990
0.996
2.00
2.80
2.75
2.70
2.79
2.05
2.85
Stegs
Stops
(a) The predicted loss of different total steps with cosine LRS and the fitted chinchilla curve.
(b) The predicted loss of different total steps with WSD LRS and the fitted chinchilla curve.
Figure 18: Chinchilla scaling law fits well the validation loss endpoints predicted by our formulation, taking cosine LRS (on the left) and WSD LRS (on the right) as examples.
27

---

3.5
1.50
350
3.23
1.3
20% WSD Cosine Armeeling
50% WSD Cosine Anmeeling
023
20% WSD Sqrt Annealing
WSD Annealing
660
19000
20000
10000
S8393
610
1990
20000
30000
40050
58505
(a) LR curve of WSD (20% 1-sqrt/cosine annealing).
(b) LR curve of WSD (50% 1-sqrt/cosine amncaling).
Figure 19: The learning rate curves of 20% (left) and 50% (right) annealing ratio WSD LRS, with cosine and 1-sqrt annealing method.
Adopting Chinchilla scaling law, typical cosine LRS requires total steps of at least 1K 2K 3K + 100K 5050K;
Adopting Chinchilla scaling law, WSD LRS (notating annealing ratio r) requires total steps of at least (1K + 2K + 3K + .... + 100K)r )r + 100K (1 -r) = (100 + 4950r) Adopting our scaling law, all we need is only one training curve with moderate total steps (and the number of fitting points is far more than 100), such as one curve with 50K steps
G
WSD Scheduler and Annealing Functions
Hu al. 2024 proposes warmup-stabbledeeay (WSD) LRS including three learning rate stages, which could help get a lower validation loss compared to the typical cosine LRS. The format is like
Teximp Imaxi T Tarmup
WSD(s)=
Mmaz: Twarmup < T'ttabl
(7)
Imnnn [(s) (Amaz 7mmin) T stable <
S
Tiotal
Where 0 < f() < 1 is typically decreasing function about step and Tmaz is the maximal learning rate, Hägele al. (2024) consolidates the effectiveness WSD scheduler by many empirical experiments. Moreover, Häggle et al. (2024) also finds that using 1-sqrt annealing and moderate annealing ratio (e.g. 20%) can further decrease the final loss. The 1-sqrt annealing defined as:
S Taable
f(x) =
(8)
T hotal Tttable
Hägele et al. (2024) mentions 1-square annealing method as baseline, which is defined as: Also,
Tstable
f(s) =1 Tiotal
T stable
We draw the learning rate curve of WSD (20% and 50% 1-sqrt annealing) in Fig. 19 compared with cosine annealing. Other than 1-sqrt annealing.
H Takeaways: Experimental Findings Verification and Explanation
H.1 It verifies and explains why loss drops more sharply when LR anneals.
We adopt our equation to help researchers understand why loss drops more sharply when LR anneals, which has been widely observed in many previous studies. We substitute the fitted parameters (see
'The empirical rule that more fitting points always achieve better fitting results always holds true. Our equation can also use more points and LRS for fitting, such as 30К constant 70K cosine. Nevertheless, we collect far more fitting points than the typical scaling law with significantly fewer training steps.
28

---

3.0
Predicted T50.65 (min
2.0
Predicted loss curve ofT=1 45 (min [r=0.11 max_r)
Predicted loss
(min Ir=0.1 max.r)
Predicted loss curve of T=5 (min Ir=0)
T-0.65 (min_iiie
2.9
LR T=1:45 (min_Ir=0.1 max.ir)
1.5
Learning Rate x10
Validation Loss
LR T-5 (min Ir-0.1 max
(r=0)
2.8
1.0
0.5
2.7
0.0
2.6
100
200
300
400
500
Step (K)
Figure 20: Predicted loss curves of different cycle length T and min LR in cosine LRS. The results well align with previous studies.
2.85
20k Steps
40k Steps
60k Steps
80k Stops
2.80
100k Steps
Validation Loss
2.75
2.70
2.65
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Forward Area 51
Figure 21: The relationship between predicted final loss and the forward area S of different total steps. Different points denote different annealing ratios.
29

---

2,00
0.25
0.20
Loariig Rat X38-
1.25
0.15
Negative Costne
Si-iem Contre
1,00
Negative
Sivieem Mage Cosirre
Negative 52 iam Mulisstage Cosine
0.50
Mult-slep Costne LRS
0.25
-0.25
-0.36
30000
10000
30000
40000
50000
60000
Stop
(a) Learning rate curves of three types of LRS.
(b) S1 S1-item and negative Sy-item different LRS.
Figure 22: The comparison between Si-item and negative S2-item in different LRS.
Fig. 2 to our equation as an instance. We draw how the S -iittem (A ST and the negative S St-item (-C. S2) impacts the loss along with WSD scheduler. Fig. 23 suggests that starting from annealing stage, negative S2-item has a much more significant impact on the overall loss than Si-ttem, which makes loss drop more sharply compared with the stable LR stage. In conclusion, LR annealing brings out quick increase of the annealing area, resulting a drastic decrease in validation loss.
0.30
2.00
0.25
1.75
0.20
1.50
Loss Drop
0.15
S1-item
1.25
Negative S2-item
0.10
Annealing Start
1.00
Learning Rate Curve
Learning
0.05
0.75
0.50
0.00
0.25
-0.05
0.00
10000
12000
14000
16000
18000
20000
Step
Figure 23: How $1-item and negative Sz-item changes WSD scheduler. Gray area means the amount of loss drop brought by S1 and S2 in annealing stage.
H.2 It verifies and explains the phenomenon, where constant LRS gets lower loss than cosine LRS if setting small total steps, and vice versa.
In the experiments, find that if set small total steps, the final loss of constant LRS could be even lower than cosine LRS, and vice versa. Refer to the ground-truth loss in Fig. 2 (20K steps). To validate this phenomenon, we use our equation draw the prediction loss curve 10K total steps and 100K total steps in Fig. 24 It shows that our proposed equation can verify well that the better LRS changes over the total steps. Moreover, Fig, 24c shows the predicted final loss of different total steps using constant and cosine LRS. It further convincingly suggests that constant LRS indeed gets lower loss if setting small total steps, but the scaling slope smaller than cosine LRS's, s, resulting in higher loss in more steps.
30

---

200
3.00
Casnnets Prodiciion Carsiant
1.75
295
Prodicion lass Gasnne UBS
Predction loss Canstant LKS
1.75
Cosine UR 1103 stepi)
Condont (10K stops)
2.00
Constonn LR [103 ciepo)
1.50
32
1.25
2.55
1.2
100
3.1
2.00
0.75
3.-
2.75
0.50
0.50
29
0.25
2770
0.25
0.00
2.55
2000
8000
10500
20000
40:00
600000
80003
100000
(a) The predicted loss curve of 10K stops.
(b) The predicted loss curve 100K steps
The fnnil losu Covire LBS
The inall loss of Constaat LRS
2.85
2.90
2.85
Valnaaion
2.80
2.70
2.85
50000
Steps
(c) The predicted final loss of differrnn total steps.
Figure 24: Comparison of constant and cosine LRS in different steps.
From a more essential and comprehensive perspective, las; is a power-law decreasing function 8L is stable over training steps. In the early stages, 25; while DSa
is large when S is small, thus increasing Si by maintaining large LR (e.g. constant LRS) in the early stages can greatly help reduce the loss. That is, S1 plays dominant role over S2. In the later stages, | is much smaller when Si becomes large, thus increasing S, in the later stages does not significantly help reduce the loss. That is, S2 plays dominant role over St. At this stage, It is time to start LR annealing to increase S2. Interestingly, this perspective aligns directly with the idea of WSD LRS (Hu et al.! 2024): In the early stages, the neural network is exploring globally and it is suitable time to use a larger LR; In the later stages, the neural network is exploring locally and it is suitable time use smaller LR.
BL
H.3
It verifies and explains that the optimal annealing function in WSD LRS depends on the annealing ratio.
In the context of the WSD LRS, the selection of the annealing method in the annealing stage is also pivotal to optimize the training process. Hägele et al. (2024) conclude that the 1-sqrt annealing (refer to Appendix for 1-sqrt function and curve) yields lower final loss compared the other annealing methods (e.g. cosine). They claim that the conclusion holds true across different annealing ratios.
However, predict using our equation (Fig. 25a the 1-sqrt annealing approach does get lower loss than the cosine annealing approach when using small annealing ratios (e.g. 10%). but it performs much worse than the cosine annealing approach when using 50% annealing ratio.
To verify whether the predictions from our equation are accurate, we conduct experiments by training models using different annealing methods and ratios within a fixed 50K total steps. As illustrated in Fig. [56] at a 10% annealing ratio, the 1-sqrt method outperforms the cosine method, whereas at a 50% annealing ratio, the latter method exhibits lower final loss. The true experimental results align quite well with our prediction, which also overturns some of the conclusions made by previous works. We conclude that the optimal annealing function in WSD LRS depends the annealing ratio.
31

---

Corour 3H6 Maltaaedd
Nor
for
201
200
2.
220
1.20
(a) The predicted loss curve of cosine and 1-sqrt nealing method of different annealing ratio.
(b) The true loss curve of different annealing ratios with cosine and 1-sqrt annealing methods.
The predicted (left) and true loss (right) of cosine and 1-sqrt annealing method at different annealing ratios. Experimental results (right), aligned with our prediction (left), refute the previous finding "the order and results of different annealing hold across settings" (Hägele et al. 2024).
Figure 25:
2.85
Recskiing
Pe-cokiing Marrep Stags 500 Wernep Steps 10000
Conitant 20-3
2:80
CPT Coaine Mat CPT Doalng Max
Wildefnn love
2.75
Lo
2.75
2.70
vestiiingg fial winig
Vestoriing Sul voring mosk
2.45
2.45
2.60
sok JUK aux
2.50
22K 40K Continual Pre-traning Stups
(a) The predicted validation loss with different re- warmup max LR in the continual pre- pre-training process. All the re- we-warmup steps are 500 steps.
(b) The predicted validation with different re- warmup max learning rate in the continual pre- pre-training process.
The predicted validation loss with different re-warmup max learning rate and re- e-warmup steps in the continual pre-training process. The LRS of continual pre-training is cosine (7'=10KK) and the min learning rate
Figure 26:
Our scaling law function provides explanatory framework for these observations. We draw the LR curves of --sqrt and cosine annealing in Appendix G At 10% annealing ratio, although the forward area S of the cosine method slightly larger than that of the 1-sqrt method, the larger annealing area S2 of the 1-sqrt method plays a more critical role in reducing the overall final loss. However, as the annealing ratio increases, the difference of S₁ between two LRS gradually becomes larger and larger, till breaking the delicate balance between S₁ and Sy at 50% annealing ratio, resulting in lower final loss for the cosine method. This relationship underscores the importance of carefully selecting the annealing strategy to optimize model training outcomes within the WSD scheduler. Still, our equation can help predict a better annealing method without experiments, which saves a lot of resources.
H.4 It verifies and explains that in continual pre-training, the higher max learning rate to re-warmup, the higher the initial peak loss will be, and then the more sharply will decrease.
In continual pre-training (CPT), the learning rate scheduler is usually set as re-warmup to new max L.R at the beginning. By many experiments, Gupta et al. (2023) concludes that the higher max learning rate to re-warmup, the higher the initial peak loss will be, and then the more sharply will decrease.
32

---

According to our scaling law function in the re-warmup process, the annealing area S2 will reduce negative value (S² 0) and thus the validation loss increases. The higher max LR in re- re-warmup. the annealing area S2 becomes more negative and thus there would be a higher peak loss. But still, higher max LRR could make the forward area S1 grow faster and the loss decreases more sharply after re-warmup. We use the fitted equation to predict the continual pre-training process with different max LR as shown in Fig. 26a The predicted loss curves reproduce a quite similar phenomenon with previous works (Gupta et al. 2023
There is more profound strategy using our equation CPT. As shown in Fig. 26a after ensuring total steps during CPT, we can apply our equation to predict a better max LR and scheduler to get the lowest final loss without experiments, which saves a lot of resources.
H.5
It verifies and explains that in continual pre-training, the steps of re-warmup have little impact on the final loss.
Meanwhile, how many steps to re- re-warmup is another important issue in the continual pre-training. Gupta et al. (2023) find that the longer re- re-warmup steps could smooth the transition of loss curve but the number of re- warmup steps does not significantly influence the final validation loss. We use the fitted equation to predicted the continual pre-training dynamics with different re-warmup steps. The results, shown in Fig. 26b] present a good alignment with previous works (Gupta et al. 2023).
Based on our theory, given the fixed max LR, when the re- re-warmup steps are longer, the annealing area decreases more slowly and the loss curve rises more smoothly, but both final S1 and S2 are quite stable across different re-warmup steps. First, the annealing area S2 of different re-warmup steps are very close due to the same max LR and the same min LR. Besides, though different re- warmup steps bring in temporary distinct losses, re-warmup only cover small percentage compared with all training steps. Thus, the forward area S1 is also close across different re- re-warmup steps, resulting in the close overall loss across different steps of re-warmup.
I Discussion
I.1 The impact of Decay Factor À
1.00
330
(a) Learning rate curves.
(b) The loss curve of A 0.99. (c) The loss curve of 0.999.
Figure 27: The comparison of fitting effect of different decay factor X.
The decay factor À in our equation plays a crucial role to indicate the information retaining degree in LR annealing. We set À as 0.999 in our all experiments. We explore the difference from another decay factor À = 0.99. We fit and get different equations for different X. We compare them (1) on the predicted loss curves for I-square and 1-sqrt annealing methods, and (2) on the predicted loss curves in different annealing ratios of WSD LRS (cosine annealing).
The results, illustrated in Fig. and 28 reveal several key insights into the impact of decay factor:
Delay Steps. larger decay factor results longer delay steps. Comparing Fig. 27b and Fig. 27c A 0.999 introduces more obvious delay phenomenon, which is consistent across both the 1-square
*Strictly speaking, continual pre-training process often include LR re- re-warmup as well as data distribution shift. Here we primarily research on the condition where there is no distribution shift between two training stages.
The conclusions transfer across most cases because the loss change brought by LR re- warmup is significantly larger than the loss change brought by data distribution shift (Gupta al. 2023 Ibrahim et al. (024).
33

---

2.85
Validation Loss
2.75
0.969 20k Steps
A 0.99 20t Simps
Cosine
Costne
Stops
60k Stops
0.599 300K Steps
0.99 100c Steps
2.65
Ratio of Annealing Steps
Figure 28: The predicted loss in different annealing ratios of WSD LRS for À = 0.99 and À = 0.999.
and 1-sqrt annealing methods. The root reason is simple: larger can retain more LR historical momentum, causing longer delay steps after LR finish annealing.
Optimal Annealing Ratio. larger decay factor tends favor higher annealing ratio. As shown in Fig. 28] The optimal annealing ratio of À = 0.999 is larger than that X = 0.99. Meanwhile, due to the similar reason, A 0.999 favors 1-sqrt annealing method while À 0.99 favors I-square annealing method, as shown in Fig. 27
Balance Point between S and Sz- More essentially, the selection of X decides the balance point of S and S2. For example, = 0.999 means that, LR annealing only retain the information of previous approximately T-x 1000 steps, which can be seen the window size of LR annealing momentum. The window size could be very close to the optimal annealing steps. After reaching window size, S2 increases very slowly, with the cost of large decrease ST.
The analyses above highlights the importance of selecting a decay factor that aligns closely with empirical data to ensure the accuracy of predictions. We recommend that the future developers try different À for their own setups 5
1.2 Possible Root Reasons of Delay Phenomenon Learning Rate Annealing
In Sec. 3. discover the delay phenomenon, which proves that LR annealing has momentum. We discuss possible root reasons of the phenomenon in this section.
Adam Optimizer? No. We notice that Adam optimizer (Kingma & Ba 2015) also has the first- order momentum decay factor 31 and the second-order momentum decay factor 32. which presents the possible connection to the the delay phenomenon.
We keep 31 0.9, and conduct delay experiments on different 8g E {0.95, 0.99, 0.999) (default: 0.95) of Adam W optimizer (Loshchilov & Hutter 2017) observe whether larger Bn causes more longer delay steps. The learning rate and ground-true loss curve are shown in Fig. 29a suggests that the ground-truth loss curves of different 32 almost coincide with each other, and their delay steps also the same. Therefore, we believe that Adam optimizer has little to do with the delay phenomenon, despite its momentum form seeming very related to our experiments. Speaking of which, we even once tried to mimic the form of Adam Optimizer to describe LR annealing momentum, attempting to discover connection between them, but the fitting results were mess.
as AAttullyy can be fitted as parameter, instead of a hyper parameter requiring manual tuning. We regard because X = 0.999 performs well in our all experiments. Besides, fitting with À could bring in additional time complexity due to the recomputation of S2.
34

---

2.75
Constant
Seray Nor
2.
9zm0 95
lanting Foundd
8,=0.99
Ay
Learnnn Bako X22
CoKieen
2.74
2.74
2.75
2:75
La Soay des
2.72
2.72
272
277
Learning Rate
2.70
30000
39000
40003
43000
.2000
43200
46020
39000
42503
45000
Steps
(a) The comparison true loss curve with sctting
(b) The comparison delay steps of different re- warmup steps (and thus different S1).
different 82 of Adam optimizer.
Figure 29: The possible root reason analysis (left: Adam optimizer, right: S1) of delay phenomenon.
Forward Area S Not Really. No matter how LR changes, S1 is always increasing over steps, resulting in consistently reducing the validation loss brought from Sj. Therefore, the forward area, Si would lengthen delay steps in LR annealing then re-warmup, but would shorten delay steps in LR re-warmup then annealing. The delay phenomenon is indeed related to Sj.
But still, S1 is not all the reasons of delay phenomenon. We prove this by Fig. 5b which suggests that even though in LR re-warmup then annealing, the delay phenomenon, while not that pronounced, still exists. Moreover, we conduct delay experiments by adjusting the slope of LR after tuning point of LR. As shown in Fig. [9b] We find that more smooth slope of LR re- warmup, with smaller S1, but still causes longer delay steps. Therefore, we conclude that S indeed influences the specific delay length, but is not the root reason.
Other Possible Reasons? The delay phenomenon could be intuitive in some cases. For example, suppose that learning rate decreases directly from 2e-4 to 2e-5 in one step, and maintains 2e-5. In this case, although the loss would decrease lower value but the parameter changes in one step is too small in neural networks. Given sudden low LR, neural networks still require some steps to gradually optimize to local minimum, incurring delay phenomenon. But still, analysis above still ends with a rough description, and we have not figured out the root reasons and look forward to completing this part in future work.
1.3 Other Possible Scaling Law Formats with LR annealing
Adding a Le-weighted Coefficient to S2 (Original Version)? Imagine that when LR anneals to nearly 0, the neural network's parameters almost do change and the validation loss should not change, either. Therefore, the original version mentioned in Sec. 3 should perform better than the simplified version. Specifically, in Eq. [ S2 still has historical momentum even if LR is nearly making the loss continue to decrease and misalign with observed training dynamics.
The original version can cover this corner case by add we-eghtte coefficient to S2. Specifically, S2 approaches when is close to O, counteracting the original formulation' 's tendency to overestimate loss reduction when 1) 0.
The original equation for the annealing area S₂ in our scaling law function is follows:
m, X m + (Th-1 - 17k)
(1-11 1 - nk) x -
(10)
S₂ Σ m, T
Where the red part is the added w.e-hhett coefficient and e undetermined positive constant." could be very small in practice.
35

---

135
112
L00 2.50 24000
1.35
126
Scretadd Learnng jute Carve Learengfuee Cove
148
Fax Oeried Fill c2m1
2000
1002
20000
(a) LR of cosine and constant.
(b) The fitting curve of L S2
(c) The fitting curve of L SS.
Figure 30: The comparison of fitting effect between L S/ with L Lxx S2.
We have tried this formulation to fit data. We find that the fitting results are quite similar and very close to O, showing little use in practical effect. Hence, we adopt the original format in our experiments. However, we still recommend future developers to try this format if using some uncommon LRS such as random LRS and zero-padded LRS.
L x S rather than L x S²² Actually, all we know is that L and S2 have positive correlation. Thus I x 8% rather than L x S2 might be more reasonable assumption. That is, our equation would be changed L(s) Lo + A ST C- Sy Theoretically, the introduction of < as an additional fitting parameter is expected to provide more nuanced control over how changes in the learning rate annealing affect validation loss, potentially leading improve the accuracy of our equation.
However, the empirical results, depicted in Fig. 30 demonstrate that the fitting improvement with the inclusion of Ç is quite marginal when compared to the version without this parameter. This slight enhancement does not justify the additional complexity introduced by managing negative values of S2. Furthermore, the empirical observation that Ç tends to converge close to 1 (e.g. 1.125 in Fig. (3cc reinforces the notion that the original formulation of the function, without the power term C, is adequately robust. This finding suggests that the direct influence of the learning rate annealing area, as initially modeled, sufficiently captures the essential dynamics without the need for this additional complexity. Another additional complexity lies in that S becomes incalculable when S2 < in LR re- re-warmup.
Studies of scaling laws are mostly empirically driven. ever-parameterizing the scaling law equation essentially leads to more accurate fitting results. However, it will also complicate the final format and hinders us to focus on major factors for the training dynamics. We choose our main format not due to absolute prediction accuracy but to pursue the simplification (i.e., fewest extra parameters) to model the essential training dynamics of LLMs. Notably, in our main format in Eq. [ we only introduce one extra parameter compared to Chinchilla scaling law, i.e., the coefficient C of the S₂ term. As suggested in Sec. our main scaling law format still has a strong and robust capacity across many practical scenarios. We believe and expect that there should be more powerful specific format (maybe with more parameters) after this work.
1.4 Optimizing Learning Rate Schedule
A natural next step of this work would be optimizing LR schedule based our proposed scaling law. From a practical enginecring aspect, is feasible and efficient to select better LRS from many candidates based on the prediction of the scaling law. Specifically, WSD (rather than cosine) LRS should be used to confirm the larger values for both S₁ and 52, as stated in Sec. and Appendix HH.2) WSD LRS annealing ratio can be determined by the method stated in Sec. 4 Annealing function can be selected by the method stated in Appendix H.3] is quite easy to get the (nearly) optimal LR schedule based on the composition of the methods above.
From another perspective, one might adopt our scaling law to directly solve optimal LR schedule mathematically. It could be found that our Eq. [ leads to a collapsed LRS: some zero learning rates last. This problem is related to the issue (1) case) that we discussed above Appendix [.3 Mathematical optimization strongly depends on absolute accuracy, while our scaling law in such
36

---

scenario does not achieve perfectly 100% accuracy (shown in Fig. Bp. In comparison, our mentioned variant forms (e.g., Eq. [TP with extra parameters should be more preferably used solve the optimal LRS, because they cover more corner cases and have higher accuracy. We believe that in the future, there will be stronger and more parameterized specific forms, which are more suitable for directly mathematically solving the optimal LR schedule. At this stage, we believe that the approach based on the practical engineering mentioned above sufficient to obtain (nearly) optimal LR schedule.
37


Reviews:
[{"review_id": "GqGstEgHNe", "reviewer": "Reviewer_ekvs", "summary": "Authors propose a scaling law which is able to predict loss of a training run based on the LR schedule. Its effectiveness is validated by closely fitting pretraining loss curves.", "questions": "See weaknesses", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow. \n2. A common knowledge is that the learning rate is the single most important hyperparameter. Utilizing only the learning rate schedule to accurately predict the loss curve seems natural and well-motivated.\n3. Empirical evidence is strong, as well as the application of this scaling law on analyzing existing learning rate schedule.\n\nWeakness:\n1. I am wondering about direct applications of the proposed scaling law. For example, given everything else the same, would it be possible to derive an optimal learning rate schedule that results in the smallest final loss?\n2. One weakness is that the same set of parameters (L0, A, C, alpha) can only be used to predict exactly same training runs with the same max learning rate. This leads to two questions:\na) How do the four parameters change when everything else is the same but only the max learning rate changes?\nb) Does the change above imply anything about what might the optimal learning rate be for this training run?", "was_revised": true, "final_justification": "A common knowledge is that the learning rate is the single most important hyperparameter. Utilizing only the learning rate schedule to accurately predict the loss curve seems natural and well-motivated. Empirical evidence is strong, as well as the application of this scaling law on analyzing existing learning rate schedule.", "rebuttal": "Thank you for your thoughtful recognition and for sharing such detailed insights! \n\n## W1: Deriving an optimal rate schedule\n\nYes and we discussed this issue in detail in Appendix I.4. Simply speaking, when optimizing LR schedule mathematically, one should use Eq. 4 rather than Eq. 1, which is more precise but quite difficult to solve in math. From another practical engineering aspect, it is feasible and efficient to select better LRS from many candidates based on the prediction of the scaling law. For example,  we show how to choose between WSD and cosine LRS and how to choose the optimal annealing ratio in Section 4.\n\n## W2: How our parameters change\n\nWe answer this question one by one.\n\n(a) Indeed, the parameters (L0, A, C, $\\alpha$) would change under those conditions. And we emphasized this in Section 3.3.\n\n(b) Great question! It is quite possible to research this interesting question by researching how $L_0,A,C,\\alpha$ changes over the pre-set max learning rate. For instance, as the learning rate increases, these parameters may follow a consistent pattern, potentially leading to a parabolic-like trend in final performance, which could help predict the optimal learning rate. While we haven’t conducted this experiment, we believe it offers a promising approach to determining the optimal learning rate.\n\n\n\nThank you again for your valuable input and for sparking such an interesting discussion! Please let us know if there’s anything further we can assist with."}, {"review_id": "0eaKWELiRL", "reviewer": "Reviewer_vZDB", "summary": "This paper investigates the relationship between the loss curve in large language model pre-training and the corresponding learning rate schedule. The authors propose a formula to model the training loss, expressed as:\n$$\nL_0 + C_1 S_1^\\alpha - C_2 S_2\n$$\n\nwhere $L_0$ is the irreducible loss, the second term models the initial rapid decrease in loss as a function of the cumulative learning rate (forward accumulation), and the third term accounts for the annealing phase of the learning rate schedule (backward accumulation).\n\nTo empirically validate their proposed formula, the authors conduct a comprehensive set of experiments. They also demonstrate the robustness of their model by successfully fitting the formula to the training trajectories of existing open-source models, such as DeepSeek.\n\nOverall, this paper offers a novel and promising perspective on scaling law research. The proposed theory has the potential to be highly beneficial for practical pre-training applications, provided it is further substantiated with more rigorous theoretical grounding and extensive reliable empirical verification.", "questions": "* **Figure 3:** Figure 3 looks okay, but it is not entirely convincing given that it relies on an *interpolation* fit rather than an *extrapolation* fit to an unseen scale.\n\n* **Figure 7b:** In the 1.2B parameter run shown in Figure 7b, the fit deviates from the actual learning curve between steps 10,000 and 30,000. The empirical curve is not very smooth in this region, which suggests the theory may not be very robust.\n\n* **Theoretical Grounding of the Annealing Term ($S_2$):** The $\\lambda$ (or $S_2$) term is not well-grounded and seems quite arbitrary. I am concerned that this may be overfitting to the authors' specific experimental setup. It seems that $\\lambda$ likely depends on multiple factors, such as the choice of optimizer (e.g., AdamW vs. Muon vs. others). For AdamW, it could depend on the $\\beta_1$ and/or $\\beta_2$ parameters.\n\n* **Claim on LRS Superiority:** Regarding the section \"Why Is WSD LRS Better than Cosine LRS?\", I am not sure this statement is true.\n\n* **Lack of Empirical Validation:** In the \"Optimal Annealing Ratio of WSD LRS\" section, the authors provide a prediction from their theory but do not validate it with a real experiment.\n\n* **Speculative Applications:** Overall, I believe the applications in Section 4 are mostly speculative insights, if I understand correctly.\n\n* **Section 5:** The same applies to Section 5.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength \n\n- fresh research direction ; novel observation (learning rate schedule vs whole learning curve) \n\n- extensive experiments  (a lot of efforts) \n\n- potential application to pre-training \n\n\nweakness\n\n- the $L_0 + C_1 S_1^\\alpha - C_2 S_2$ is not very grounded. In particular, the term and its definition $S_2$ is not very well-motivated and not very interpretable. \n\n- the hparams $\\lambda$ in $S_2$ is not interpretable. \n\n- fig 2 b is a bit concerned to me; as the constant learning rate and cosine has almost the same learning curve shapes and their final performance is very closed (2.83 vs 2.84); in practice, their shape should be *very different*. for example, see Fig 6 in https://arxiv.org/pdf/2405.18392. this makes me cast doubte on the remaining experiments.", "was_revised": true, "final_justification": "authors address some of my concern;", "rebuttal": "We sincerely appreciate your thorough engagement and the numerous insightful questions you’ve raised, which have greatly enriched our work. \n\n## W1: The $L_0+A\\cdot S_1^{-\\alpha}−C\\cdot S_2$ is not very grounded.\n\nWe present in detail how we derive $S_2$ in Section 3.2. Basically, we observe the delay phenomenon in LR annealing and we find that there exists momentum in LR annealing. Therefore, the final $S_2$ is momentum-like (or EMA-like) form. As noted by reviewer `EhjS` comments, the formulation \"is interpretable, and illustrates the trade‐off between rapid progress at high learning rates and the benefits of annealing\". To further substantiate our findings, we conduct extensive experiments in Section 3 to validate the proposed scaling law.\n\n\n\n## W2: The hyper-param $\\lambda$ in $S_2$ is not interpretable.\n\n1. $\\lambda$ is the momentum coefficient, which is quite common in momentum-like form which means the . We adopt $\\lambda = 0.999$ in our all experiments and it shows that the choice of $\\lambda$ is not that sensitive and $\\lambda=0.999$ is quite robust across many experimental setups.\n2. In Appendix I.1, we have *fully* discussed and investigated the essence and the impact of decay factor $\\lambda$ in detail. We show that $\\lambda$ influences three aspects: (1) delay steps; (2) optimal annealing ratio; and (3) balance point between $S_1$ and $S_2$.\n\n\n\n## W3: Concerns about Figure 2b\n\nThis is an actually great question, and we *fully* discuss this in Appendix H.2. Simply speaking, the loss curves of constant/cosine LR, and how different they are, depend on some important hyper-parameters such as training steps. And our curve is actually not contradictory to the paper you mentioned.\n\nIn Figure 24, we show that the curves of cosine and constant could be quite similar if we train 10K steps (shown in Figure 24a), but become quite different if training for 100K steps (Figure 24b). Moreover, we show that in small training steps, constant LR could lead to better performance than cosine LR. When training for more steps, the final performance of cosine LRS surpasses that of constant LRS, and this gap gradually widens (Figure 24c).\n\nMoreover, the max learning rate can also influence the curves of constant and cosine LR schedules. And larger max learning rate often leads to larger gap between them. We set max learning rate as 2e-4 in Figure 2, and it is not large. Our results are all based on rigorous and standardized experiments.\n\n\n\n## Q1: Figure 3\n\nNo, we did *extrapolation* prediction rather than interpolation. We fit only 20K training steps (Figure 2) and predict the loss for 60K training steps under many **unseen** different LRS (Figure 3).\n\nIn our additional experiments in our Appendix E, we also did *extrapolation* prediction. For example, in Appendix E.1, We adopt another experimental setup and fit only 30K training steps (Figure 10) and predict the loss for over 100K training steps under **unseen** LRS.\n\n\n\n## Q2: Figure 7b\n\nOur proposed scaling law calculates the *expected* loss for each training step. In empirical experiments, there could be more other factors influencing the validation loss. For example, some training samples near specific checkpoints are more similar to the validation set, and then it could make a temporarily lower validation loss. Additionally, larger batch sizes typically result in lower variance and makes the loss curve smoother. Furthermore, prior scaling laws, such as OpenAI and Chinchilla scaling laws, also compute expected losses with varying degrees of variance. Therefore, we believe this does not suggest that our scaling law lacks robustness.\n\n\n\n## Q3: Theoretical grounding of the annealing term\n\nWe *fully* discussed all your mentioned concerns about $\\lambda$ in Appendix I.1 and I.2 of our paper. \n\n**Selection of $\\lambda$:** We adopt $\\lambda=0.999$ in all of our experiments (with many different setups, see Table 3), and the results are all good, which means that $\\lambda=0.999$ is quite robust across different setups. Moreover, we even mention in the footnote of page 27 that one can fit $\\lambda$ as a parameter like  $L_0,A,C,\\alpha$, which is also reasonable and totally feasible.\n\n**How $\\beta_1$ and $\\beta_2$ in Adam Optimizer impacts**: We have discussed this issue in our paper. We conduct an experiment in Figure 29a of Appendix I.2. In conclusion, we find that changing $\\beta_2$ does not nearly influence the delay steps as well the momentum term (including $\\lambda$).\n\n**How $\\lambda$​ impacts**: In Appendix I.1, we discussed and investigated the essence and the impact of decay factor $\\lambda$ in detail. We show that $\\lambda$ influences three aspects: (1) delay steps; (2) optimal annealing ratio; and (3) balance point between $S_1$ and $S_2$.\n\nOverall, we believe that we have sufficiently discussed $\\lambda$ in our paper.\n\n\n\n## Q4 & Q5 & Q6 & Q7\n\nIn this section, we aim to demonstrate that **the experimental conclusions from previous works** can be also validated and explained using our predicted scaling law.\n\n**Claim on LRS Superiority**: We cited the conclusion drawn from [1] and [2], and used our scaling law to verify and explain their conclusions. We provide a perspective about the balance between $S_1$ and $S_2$. We believe there are still cases where WSD underperforms compared to cosine (as I mentioned above regarding the relationship between cosine and constant LRS). In summary, we believe what really matters is that our scaling law can predict the final loss under various circumstances, allowing for case-specific evaluations.\n\n**Lack of Empirical Validation**: Similarly, we cited the conclusion drawn from [3], and used our scaling law to verify and explain their conclusions. Their paper has already shown that 10%~20% annealing ratio in WSD LRS is quite good in most situations, which is consistent with the results predicted by our scaling law.\n\n**Speculative Applications:** Section 4 demonstrates how we apply our scaling law to validate and explain insightful findings, and these results align with the conclusions of prior work. In fact, we have already conducted extensive experiments in Section 3 to verify our scaling law, including the main experiments in Figure 2 and Figure 3, as well as four additional validation experiments documented in Lines 209–219 and Appendix E.1–E.4.\n\n**About Section 5**: Section 5 presents a comparison between our scaling law and the Chinchilla scaling law. First of all, in Table 4, we have validated the performance comparison (in real experiments) between our and Chinchilla’s scaling laws. In Section 5, we focus more on formal reduction and analysis of resource consumption.\n\n[1] [MiniCPM: Unveiling the Potential of Small Language Models with Scalable  Training Strategies](https://arxiv.org/pdf/2404.06395)\n\n[2] [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://arxiv.org/pdf/2401.02954)\n\n[3] [Scaling Laws and Compute-Optimal Training Beyond Fixed Training  Durations](https://arxiv.org/pdf/2405.18392)\n\n\n\nThank you again for your valuable feedback and the many thought-provoking questions that have helped refine our work. We look forward to any further insights you may have!"}, {"review_id": "oWHdFpjWxq", "reviewer": "Reviewer_EhjS", "summary": "The authors identify empirical relationships among per‐step loss, gradient norms, and the learning‐rate schedule, and propose a novel scaling law that directly links the per-step loss to the learning‐rate scheduler. Their formulation is capable of capturing the abrupt loss drop following a learning‐rate decay, making it applicable across a variety of schedules.", "questions": "**Questions**\n\n1. Is it possible to theoretically claim the reasonability of the proposed scaling‐law relationship, analogous to [Theorem 1 in 1]? Such a result would greatly strengthen the paper’s claims.\n2. Under the proposed law, a “constant‐then‐zero” schedule emerges as optimal (cf. [Theorem 2 in 1]), which reproduces the failure of explaining the corner behavior that adding zero‐LR steps always makes improvement. According to the discussion sections, the authors proposed an ad-hoc adjustment to this issue. Based on this modification, can you theoretically derive certain optimality as done in [1]? \n3. The authors mentioned that the proposed scaling law can be extended to accommodate the model-size with empirical validation. Based on this, if we fit a scaling law on small scale experiments, we should be able to use the fitted law to determine the configuration of large scale training schedulers. Is there any experiments related to this? Does the performance beats the default configuration? \n\n---\n\n**Reference**\n\n[1] Luo, K. *et al.* (2025). A multi‐power law for loss‐curve prediction across learning‐rate schedules. *arXiv:*2503.12811.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Pros**\n\n1. Designing optimal learning‐rate schedules is critical for modern deep‐learning training, yet our theoretical understanding remains limited. This work takes a significant step by formalizing the empirically observed effect of the schedule on each training step’s loss.\n2. The paper presents a step‐by‐step derivation that bridges intuitive observations (e.g. the delay in loss reduction) and the final scaling‐law form. Each refinement is well motivated and tied back to real‐world phenomena.\n3. The resulting formula is concise, interpretable, and accurately predicts unseen loss curves, while explicitly illustrating the trade‐off between rapid progress at high learning rates and the benefits of annealing.\n\n**Cons**\n\n1. Most of the derivation rests on visual/empirical observations rather than rigorous theory (cf. Question 1).\n2. The law can compare configurations only within certain families of schedulers; it does not immediately generalize to arbitrary or hybrid schedules without ad‐hoc adjustments (cf. Question 2).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback and the opportunity to address your comments.\n\n## Q1: Empirical observations rather than rigorous theory\n\nWe believe that theoretical analysis can strengthen our claims. However, it’s important to recognize that in real-world LLM training, theoretical analysis is extremely difficult. Almost all scaling law papers are proposed empirically. The paper you mentioned [1] also addresses this issue and simplifies the problem to a highly toy scenario to derive its results: *\"we consider a simple setting where SGD optimizes a quadratic loss function with noisy gradients.\"*\n\n We believe that advancing theoretical analysis is an important research direction, and we also believe that our conclusions are sufficiently supported by the extensive experiments in Section 3 we have conducted.\n\n[1] [A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules](https://arxiv.org/pdf/2503.12811)\n\n\n\n## Q2: Optimizing learning rate schedule\n\n**Optimizing learning rate schedule**: In Appendix I.4 of the paper, we discuss the optimization of the learning rate schedule in detail. Specifically, when optimizing Eq. 10 (as opposed to Eq. 1), the problem becomes quite difficult to solve, as it is no longer convex (the Hessian matrix of this problem is no longer positive definite). From another practical engineering aspect, it is feasible and efficient to select better LRS from many candidates based on the prediction of the scaling law. For example,  we show how to choose between WSD and cosine LRS and how to choose the optimal annealing ratio in Section 4.\n\n**The scope of our scaling law**: Besides, based on your concern \"The law can compare configurations only within certain families of schedulers\". We have the response below:\n\nWhen the LR schedule becomes even stranger and irregular, the loss becomes more difficult to predict, which is quite intuitive and inevitable. In this case, our scaling law will still make accurate enough predictions and try to keep the error within a usable range. \n\nAs the Table 4 shows, when the predicted LR schedule becomes more complicated, the error of our scaling law becomes indeed larger from 0.159% to 0.322%. The errors are all quite small so that we can adopt our scaling law on the schedules.\n\nMore extremely, we also perform an extra experiment on irregular and random LR, where we set LR of each step uniformly sampled from 0~2e-4, and S2 term indeed approaches zero during the whole training run. The mean error in such extreme situation is 0.605%, indeed larger but within a moderate range. Your mentioned \"constant‐then‐zero\" is a quite extreme situation. Overall, our scaling law applies within a specific scope (e.g., consistent maximum learning rate and standard learning rate schedules). However, for commonly used learning rate schedules, it offers high accuracy, simplicity, and strong interpretability.\n\n\n\n## Q3: About $N$-extended scaling law\n\nYes and the experiment in Figure 7b shows that we can derive a model size extended scaling law from smaller model sizes. The fitted equation is $1.504 + 0.440\\cdot S_1^{-0.546} + 3.067\\cdot N^{-0.158} - 0.273\\cdot S_2\\cdot N^{0.052}$. We didn't use this equation to select the better learning rate schedule but we validated this fitted equation by predicting the full loss curve for a larger model. Actually, if we replace $N$ with specific value (e.g. 594.6M), and the equation becomes $2.622 + 0.440\\cdot S_1^{-0.546} - 0.380\\cdot S_2$, which is quite similar with our fitted equation in Figure 2: $L(s) = 2.628 + 0.429 · S_1^{−0.550} − 0.411 · S_2$. This means that our equation can also incorporate certain values of N to make specific predictions for a given model size.\n\nWe hope this clarification addresses your concerns and thank you again for your thoughtful review."}, {"review_id": "avLlGYRwgE", "reviewer": "Reviewer_ssnF", "summary": "This paper proposes to extend existing LLM scaling laws to account for learning rate annealing, which additionally fits the loss at intermediate steps during training. The paper is well written, and provides intuition for the final formulation of the novel scaling law. Extensive experiments are provided to back up the observations.", "questions": "* How feasible would it be to consider generalization bounds under this new scaling law, e.g. [1]?\n\n\n[1] Finzi, Marc, et al. \"Compute-Optimal LLMs Provably Generalize Better with Scale.\" https://arxiv.org/abs/2504.15208.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The paper is well-written and has applications in reducing the compute for training LLMs (Table 2). The observations are backed up by experiments. Further, the authors show that their novel scaling law is a generalization of the Chinchilla scaling law (Section 5).\n\n**Weaknesses**: The scaling law discussed in the paper does not have theoretical justification, but the authors mention this limitation. One minor note in the conclusion (line 344) is that it would be better to remove \"theory\". Additional discussion on related work would be useful to put this work into context.", "was_revised": true, "final_justification": "The authors addressed my concerns, and I think the paper has sufficient novelty for publication in NeurIPS.", "rebuttal": "## W1: Theoretical justification and related works\n\nYes, our scaling law is empirical, as noted in our limitations section. We appreciate your suggestion and will revise certain expressions in the paper accordingly. Additionally, we will incorporate more related work to strengthen the paper. We will make this revision in the next version of the paper. Additionally, we appreciate your suggestion to include a more detailed discussion of related work to better contextualize our findings. We will expand the related works to provide a clearer comparison with prior studies and to highlight how our work fits into the broader research landscape. This will help strengthen the paper’s contribution and provide a more comprehensive perspective.\n\n\n\n## Q1: How feasible would it be to consider generalization bounds under this new scaling law?\n\nThank you for your insightful question regarding the feasibility of deriving generalization bounds under the new scaling law. We believe that exploring generalization bounds is feasible but would require additional theoretical work. We would like to emphasize some strong generalization effects demonstrated in our experiments:\n\n1. In our experiments, the learning rate schedule (LRS) used for prediction was unseen during fitting, and the error was controlled within a very small range (<0.35%).\n2. In our experiments, we performed data fitting using only a short number of training steps (e.g., 20K) and extrapolated generalization to longer training steps (e.g., 60K).\n3. As you also noted, our work is a generalized form of the Chinchilla scaling law. We can achieve significantly more fitting points under the same computational resources, leading to better prediction and generalization performance. Moreover, compared to the Chinchilla scaling law, our model introduces only one additional parameter, and this simplicity contributes to greater robustness.\n\nWe will incorporate the paper you mentioned in the next version of our manuscript. We are committed to addressing these points in the revised manuscript and hope our responses clarify our approach and intentions. Please let us know if you have any further recommendations or require additional clarification. We greatly appreciate your time and expertise in reviewing our work."}]

Author rebuttal:

Meta-review: The paper empirically derives the scaling law for various neural language laws under different learning rate annealing processes. In particular, the paper supposes a scaling law of the form: $L(s) = L_0 + A S_1^{-\alpha} + C S_2$. Extensive experiments are performed under various learning rate conditions (e.g., cosine decay and WSD). 

Overall, the reviewers are in favor of acceptance with some mild concerns about the experimental results relating to the cosine and WSD learning rate schedules. It would be helpful for the authors to incorporate within the text a description of where the proposed scaling law comes from and/or theoretically derive such a scaling law from simple model. I highly suggest that the authors incorporate the comments from the reviewers into the main text.
Decision: Accept (poster)