Title: Scaling Law with Learning Rate Annealing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scaling Laws

Full Loss Curve Prediction

Learning Rate Schedule

LLM Pretraining
Abstract: We find that the cross-entropy loss curves of neural language models empirically adhere to a scaling law with learning rate (LR) annealing over training steps: $$L(s) = L_0 + A\cdot S_1^{-\alpha} - C\cdot S_2,$$ where $L(s)$ is the validation loss at step $s$, $S_1$ is the area under the LR curve, $S_2$ is the LR annealing area, and $L_0$, $A$, $C$, $\alpha$ are constant parameters. This formulation accounts for two main effects: (1) power-law scaling over data size, and (2) the additional loss reduction during LR annealing. Unlike previous studies that only fit losses at final steps, our formulation captures the entire training curve, allowing for parameter fitting using losses from any training step. Applying the scaling law with LR annealing and fitting only one or two training curves, we can accurately predict the loss at any given step under any learning rate scheduler (LRS). This approach significantly reduces computational cost in formulating scaling laws while providing more accuracy and expressiveness. Extensive experiments demonstrate that our findings hold across a range of hyper-parameters and model architectures and can extend to scaling effect of model sizes. Moreover, our formulation provides accurate theoretical insights into empirical results observed in numerous previous studies, particularly those focusing on LR schedule and annealing. We believe that this work is promising to enhance the understanding of LLM training dynamics while democratizing scaling laws, and it is helpful to guide both research and industrial participants in refining training strategies for further LLMs.


Peer reviews:
comments: SUMMARY:
This paper proposes to extend existing LLM scaling laws to account for learning rate annealing, which additionally fits the loss at intermediate steps during training. The paper is well written, and provides intuition for the final formulation of the novel scaling law. Extensive experiments are provided to back up the observations.

STRENGTHS AND WEAKNESSES:
**Strengths**: The paper is well-written and has applications in reducing the compute for training LLMs (Table 2). The observations are backed up by experiments. Further, the authors show that their novel scaling law is a generalization of the Chinchilla scaling law (Section 5).

**Weaknesses**: The scaling law discussed in the paper does not have theoretical justification, but the authors mention this limitation. One minor note in the conclusion (line 344) is that it would be better to remove "theory". Additional discussion on related work would be useful to put this work into context.

QUESTIONS:
* How feasible would it be to consider generalization bounds under this new scaling law, e.g. [1]?


[1] Finzi, Marc, et al. "Compute-Optimal LLMs Provably Generalize Better with Scale." https://arxiv.org/abs/2504.15208.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors identify empirical relationships among per‐step loss, gradient norms, and the learning‐rate schedule, and propose a novel scaling law that directly links the per-step loss to the learning‐rate scheduler. Their formulation is capable of capturing the abrupt loss drop following a learning‐rate decay, making it applicable across a variety of schedules.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros**\n\n1. Designing optimal learning‐rate schedules is critical for modern deep‐learning training, yet our theoretical understanding remains limited. This work takes a significant step by formalizing the empirically observed effect of the schedule on each training step’s loss.\n2. The paper presents a step‐by‐step derivation that bridges intuitive observations (e.g. the delay in loss reduction) and the final scaling‐law form. Each refinement is well motivated and tied back to real‐world phenomena.\n3. The resulting formula is concise, interpretable, and accurately predicts unseen loss curves, while explicitly illustrating the trade‐off between rapid progress at high learning rates and the benefits of annealing.\n\n**Cons**\n\n1. Most of the derivation rests on visual/empirical observations rather than rigorous theory (cf. Question 1).\n2. The law can compare configurations only within certain families of schedulers; it does not immediately generalize to arbitrary or hybrid schedules without ad‐hoc adjustments (cf. Question 2).\n\nQUESTIONS:\n**Questions**\n\n1. Is it possible to theoretically claim the reasonability of the proposed scaling‐law relationship, analogous to [Theorem 1 in 1]? Such a result would greatly strengthen the paper’s claims.\n2. Under the proposed law, a “constant‐then‐zero” schedule emerges as optimal (cf. [Theorem 2 in 1]), which reproduces the failure of explaining the corner behavior that adding zero‐LR steps always makes improvement. According to the discussion sections, the authors proposed an ad-hoc adjustment to this issue. Based on this modification, can you theoretically derive certain optimality as done in [1]? \n3. The authors mentioned that the proposed scaling law can be extended to accommodate the model-size with empirical validation. Based on this, if we fit a scaling law on small scale experiments, we should be able to use the fitted law to determine the configuration of large scale training schedulers. Is there any experiments related to this? Does the performance beats the default configuration? \n\n---\n\n**Reference**\n\n[1] Luo, K. *et al.* (2025). A multi‐power law for loss‐curve prediction across learning‐rate schedules. *arXiv:*2503.12811.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the relationship between the loss curve in large language model pre-training and the corresponding learning rate schedule. The authors propose a formula to model the training loss, expressed as:\n$$\nL_0 + C_1 S_1^\\alpha - C_2 S_2\n$$\n\nwhere $L_0$ is the irreducible loss, the second term models the initial rapid decrease in loss as a function of the cumulative learning rate (forward accumulation), and the third term accounts for the annealing phase of the learning rate schedule (backward accumulation).\n\nTo empirically validate their proposed formula, the authors conduct a comprehensive set of experiments. They also demonstrate the robustness of their model by successfully fitting the formula to the training trajectories of existing open-source models, such as DeepSeek.\n\nOverall, this paper offers a novel and promising perspective on scaling law research. The proposed theory has the potential to be highly beneficial for practical pre-training applications, provided it is further substantiated with more rigorous theoretical grounding and extensive reliable empirical verification.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n\n- fresh research direction ; novel observation (learning rate schedule vs whole learning curve) \n\n- extensive experiments  (a lot of efforts) \n\n- potential application to pre-training \n\n\nweakness\n\n- the $L_0 + C_1 S_1^\\alpha - C_2 S_2$ is not very grounded. In particular, the term and its definition $S_2$ is not very well-motivated and not very interpretable. \n\n- the hparams $\\lambda$ in $S_2$ is not interpretable. \n\n- fig 2 b is a bit concerned to me; as the constant learning rate and cosine has almost the same learning curve shapes and their final performance is very closed (2.83 vs 2.84); in practice, their shape should be *very different*. for example, see Fig 6 in https://arxiv.org/pdf/2405.18392. this makes me cast doubte on the remaining experiments.\n\nQUESTIONS:\n* **Figure 3:** Figure 3 looks okay, but it is not entirely convincing given that it relies on an *interpolation* fit rather than an *extrapolation* fit to an unseen scale.\n\n* **Figure 7b:** In the 1.2B parameter run shown in Figure 7b, the fit deviates from the actual learning curve between steps 10,000 and 30,000. The empirical curve is not very smooth in this region, which suggests the theory may not be very robust.\n\n* **Theoretical Grounding of the Annealing Term ($S_2$):** The $\\lambda$ (or $S_2$) term is not well-grounded and seems quite arbitrary. I am concerned that this may be overfitting to the authors' specific experimental setup. It seems that $\\lambda$ likely depends on multiple factors, such as the choice of optimizer (e.g., AdamW vs. Muon vs. others). For AdamW, it could depend on the $\\beta_1$ and/or $\\beta_2$ parameters.\n\n* **Claim on LRS Superiority:** Regarding the section \"Why Is WSD LRS Better than Cosine LRS?\", I am not sure this statement is true.\n\n* **Lack of Empirical Validation:** In the \"Optimal Annealing Ratio of WSD LRS\" section, the authors provide a prediction from their theory but do not validate it with a real experiment.\n\n* **Speculative Applications:** Overall, I believe the applications in Section 4 are mostly speculative insights, if I understand correctly.\n\n* **Section 5:** The same applies to Section 5.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
Authors propose a scaling law which is able to predict loss of a training run based on the LR schedule. Its effectiveness is validated by closely fitting pretraining loss curves.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper is well written and easy to follow. 
2. A common knowledge is that the learning rate is the single most important hyperparameter. Utilizing only the learning rate schedule to accurately predict the loss curve seems natural and well-motivated.
3. Empirical evidence is strong, as well as the application of this scaling law on analyzing existing learning rate schedule.

Weakness:
1. I am wondering about direct applications of the proposed scaling law. For example, given everything else the same, would it be possible to derive an optimal learning rate schedule that results in the smallest final loss?
2. One weakness is that the same set of parameters (L0, A, C, alpha) can only be used to predict exactly same training runs with the same max learning rate. This leads to two questions:
a) How do the four parameters change when everything else is the same but only the max learning rate changes?
b) Does the change above imply anything about what might the optimal learning rate be for this training run?

QUESTIONS:
See weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02