Title: Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
Venue: NeurIPS 2025
Accepted: yes
Keywords: scaling laws for hyperparameters

weight decay

batch sizing

critical batch size

data parallelism

large language models (LLMs)

pre-training

AdamW optimizer

compute-optimal training
Abstract: Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate η and weight decay λ. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B.  Recent work suggests the AdamW timescale, τ = B/(ηλD), should remain constant across training settings, and we verify the implication that optimal λ scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal τ obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict λopt in advance of large-scale training.  We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast to prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores fundamental scaling laws for hyperparameters during large language model (LLM) pre-training, specifically focusing on two critical aspects: (1) optimal weight decay ($\\lambda_{opt}$) and (2) optimal and critical batch sizes ($B_{opt}$, $B_{crit}$). The authors present extensive empirical studies conducted across multiple model sizes, dataset sizes, and batch sizes, leading to clear, well-supported scaling laws.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Strong Empirical Foundation:**  \n   The authors conducted extensive experimentation (around 400 models trained), covering a wide range of model sizes, batch sizes, and data scales. This experimental rigor strongly supports the reliability and generalizability of their findings.\n\n2. **Novelty and Clarity of Contributions:**  \n   Introducing a clear, practical method for estimating $B_{crit}$—that is both efficient and generalizable—is a significant methodological contribution. The clear articulation of how to scale $\\lambda_{opt}$ through the AdamW timescale $\\tau_{EMA}$) is also valuable and original.\n\n\n5. **Insightful Analysis and Theoretical Connections:**  \n   The authors provide intuitive explanations for observed phenomena, such as the EMA view of AdamW and its implications, and clearly demonstrate consistency with theoretical frameworks (e.g., maximal update parameterization (µP)).\n\n---\n### Weaknesses \n\n1. **Limited Exploration of Optimizers and LR Schedules:**  \n   The main experiments focus exclusively on AdamW with a linear LR schedule. While the authors claim general applicability, additional validation with other optimizers (e.g., Shampoo, LAMB, Muon) and LR schedules (cosine decay, constant LR with averaging) would strengthen generalizability claims.\n\n2. **Small Batch Behavior:**  \n   The observation that smaller batch sizes do not asymptotically approach $D_{min}$ as predicted theoretically is intriguing and warrants deeper exploration. Currently, explanations are only speculative.\n\n3. **Practical Systems Considerations:**  \n   Real-world training is constrained by hardware limitations (e.g., bandwidth, memory, synchronization overheads). The paper does not significantly explore or discuss these practical system-level constraints and their impacts on batch size recommendations.\n\n4. **Dataset and Architecture Limitations:**  \n   Experiments were conducted on just one dataset (SlimPajama) and one specific GPT-style architecture. While the authors provide a good range of sizes and scales, more diverse datasets, architectures, and domains might be needed to fully validate the universal applicability of these scaling laws.\n\nQUESTIONS:\n1. Have you explored or considered how changes in numerical precision (e.g., training in mixed precision or using quantized formats) could affect your proposed scaling laws and recommendations? If so, could you briefly summarize your findings? If not, could you discuss how you expect reduced precision might influence batch size and weight decay scaling?\n\nLIMITATIONS:\n1. **Generalizability of Method:**\n    - *The proposed method for estimating $ B_{crit} $ is claimed to be suitable for arbitrary optimizers and learning rate schedules. Have the authors empirically validated or tested this methodology with optimizers other than AdamW (e.g., LAMB, Shampoo) or with alternative LR schedules (e.g., cosine decay, constant LR)? If not, what would be the expected challenges or limitations?*\n\n2. **Practical Systems Implications:**\n    - *Given that real-world model training is often heavily constrained by hardware limitations and communication overhead, how do the authors suggest practitioners incorporate these practical constraints when applying the proposed scaling laws and recommendations for optimal batch size?*", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper conducts a large-scale empirical study on the interaction between weight decay $\\lambda$, batch size $B$, token budget $D$, model size $N$, and learning rate $\\eta$. At a high-level, the paper conducts hyperparameter scaling law analysis focused on the interaction of $\\lambda$ with the other terms, encapsulated by the term $\\tau_{EMA}$. On a more granular level, the paper's primary contributions are:\n\n* The finding that the optimal value of $\\tau_{EMA}$ obeys a power law decrease (as opposed to holding constant, as suggested by [Wang & Aitchison, 2024]).\n* Methodology for estimating optimal and critical batch sizes, along with observed power law behavior of these parameters.\n* Usage of the hyperparameter rules and scaling laws above to investigate pareto frontiers of FLOPs versus wallclock trade-offs. Notably, under a simple wall-clock model, the authors show that over-trained small models often dominate larger, Chinchilla-optimal models.\n\nIn short, the paper does as the authors claim in the abstract and introduction - empirical scaling law analyses of hyperparameters, especially batch size $B$ and weight decay $\\lambda$.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nI will be quite clear here - I think this is a good paper that significantly advances our understanding of hyperparameters in transformer pre-training. The paper is generally well-written, rigorous, explicit, and produces useful insights and analyses.\n\n### Explicit methodology\n\nPerhaps my favorite part of the paper is how explicit the methodology is. Consider Section 4.2. Rather than just giving the Pareto frontiers with some cursory discussion about their conclusions, the paper goes into detail about how it computed the exact Pareto frontier, (L251-256), with explicit references to what equations they used to fit what parameters. This makes the paper much easier to reproduce in other settings and on other models, and I think sets a great example to the scaling law community.\n\n### Careful methodology\n\nIn addition to being explicit, the authors are clearly careful. This makes me trust their findings so much more. As a useful example (though this trend occurs throughout the paper) consider the estimation of critical batch size in Section 3.2. The authors identify a thorny issue (the fact that the number of steps, integral to the learning rate schedule, is unknown). They propose a specific method to remedy it, detail it explicitly, and discuss in even more depth in Appendix F.2. The authors have clearly thought about every aspect of their methodology, and I think the reader and community benefits from this kind of work being accepted.\n\n### Breadth and utility of results\n\nI think an undeniable strength of the paper is that it has a lot of useful results. Weight decay, batch size, and even what model size to train are all individually important topics. Including them all in the same paper is ambitious (though not without drawbacks, see below) but I think that the authors basically succeed on all fronts. It would be easy for practitioners to immediately incorporate this into their training regiments, and improve training on multiple fronts. The paper is also extremely useful in its breadth of connections to related work - someone relatively new to the area would get a pretty good understanding of the landscape of work just by reading this single paper.\n\n## Weaknesses\n\nI will go into a bit of detail here, because I want to emphasize ways the paper could potentially be improved. I want to be clear though, these are constructive suggestions, and I still wholeheartedly recommend acceptance.\n\n### Breadth versus depth\n\nMy biggest critique of the paper is that the three main topics it covers (weight decay, batch size, pareto frontiers) are related, but not wholly so. This makes the paper a bit less cohesive. While I think the paper should be accepted as is, it makes me wonder if pulling the paper apart would have been at all beneficial.\n\nTo take this further, it means that each individual thrust has some empirical aspects and ablations that could have been expanded upon had it been the singular thrust of a paper. For example, one of the contributions in Section 3 is an estimate of the critical batch size. However, [McCandlish et al., 2018] provide a method for estimating this quantity as well. If Section 3 were the subject of a full paper, I would absolutely ask to see comparisons between the critical batch size estimates in this work, and the critical batch size estimates from [McCandlish et al., 2018]. In this paper, I don't think it's critical (given the breadth) but it's a comparison point that is missing.\n\nSimilarly, had section 4 been the primary focus, I would ask to see how the Pareto frontier changes as the training time model changes. This is done to some extent (Figure 6, left versus right) but there's a wide spectrum of possible models that would be fascinating to explore here. To some degree, this is a good criticism to have - it means that the paper is doing interesting things, and I look forward to follow-up work. But I do want to mention it as a drawback of fitting these three topics under a single paper's umbrella.\n\n### Ablations and their role\n\nRelated to the above is the fact that by bundling these all together, it means that the paper is often combining multiple things into a single experiment, making it so that the findings are predicated upon one another.\n\nAs a concrete example, consider the critical batch size analysis in Section 3. It is unclear whether the optimal setting of $\\lambda$ is actually required for any of the subsequent findings. Would the use of constant $\\tau_{EMA}$ (as opposed to the power law decaying one in equation (4)) qualitatively change the findings in this section? It is essentially for the reader to know. This also rears its head in Section 4, which relies on the critical batch size estimates from this work. Do the pareto frontiers change significantly with different methodology for estimating the critical batch size?\n\nI will note that these are not dealbreakers in any way shape or form. They can absolutely be investigated in follow-up work. But just as above, I want to note that there are trade-offs of the paper's design.\n\n## Minor comments\n\n* $\\tau_{EMA}$ is undefined by the time Figure 1 shows up. More generally, I found Figure 1 to be much less useful as a synopsis of the work than the bullet points on L51-L55.\n* In Figure 2, middle - you state that $\\lambda$ scales linearly with $B$. While this is true for most $B$, this is clearly untrue for larger $B$. Should this be interpreted as $\\lambda$ having a predictable relationship up to something like the critical batch size, or is something more complex going on?\n* While I generally liked the colored text boxes with key takeaways, I found key takeaway 3 (L290) to be out of place and inappropriate for an academic setting. It's just a strange thing to advertise your methodology after you get done describing it. Moreover, it discusses the contributions in a pretty vague way that doesn't actually interact well with any of the specifics preceding it. I would strongly advocate removing it altogether.\n* Figure 5, right, is difficult to match lines with the legend (e.g. I can barely tell which is 100M and which is 1B).\n* Algorithm 1, in the appendix, mixes greek letters with pseudo-code names in a way that's very hard to read (e.g. $\\tau_{EMA}$*_scaling_law_fitting_points.add*).\n* The paper uses inconsistent \\citet, \\citep and the like. For example, L79 and L112 both use \"Wang and Aitchison\" in the sentence, but have different citation styles. This occurs elsewhere in the paper.\n\nQUESTIONS:\n1. It is not immediately clear how the findings in Sections 3 & 4 are predicated on the weight decay analysis in Section 2. Do the authors have any sense (or data to back it up) that the findings on critical/optimal batch size and Pareto frontiers would change qualitatively had $\\lambda$ been set via a simpler rule (such as the $\\tau_{EMA} \\approx c$ heuristic from [Wang & Aitchison, 2024])?\n\n2. Would the Pareto frontiers change qualitatively when using a different mechanism to estimate critical batch size?\n\n3. What's going on in the last few points of Figure 2, middle, where the optimal values of $\\lambda$ and $\\eta$ drop with the largest batch size?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work empirically studies hyper-parameters scaling of large langage models. The first part focuses on how to scale weight-decay and learning rates for AdamW when batch scales. They generalize a formula for a quantity $\\tau = B/(\\eta\\lambda D)$ that was formulated in a multi-epoch setting and shown to stay invariant for optimal training in that case. They show that this quantity remains approximately constant in optimal training in the one epoch training. Additionally, they show that the weight decay $\\lambda$ is \"easier\" to scale than the learning rate, in that tuning $\\lambda$ instead of $\\eta$ brings lower losses and that it scales better with batch. Finally, they show that when accounting for token-per-parameter (TPP) ratio, $\\tau$ is not constant and behaves as a power-law in TPP. The second part focuses on how do the optimal and critical batches scale. They define experimental set-ups to measure $B_{crit}$ and $B_{opt}$ and show that both of these evolve as power-laws in compute. Additionally, different TPP yield different power-laws for $B_{crit}$ and $B_{opt}$. Finally, they discuss implications of their finding on the impact of batch size on the optimal trade-off between compute and training time. In particular, they show that the usual $TPP=20$ tokens per parameters requires not too large batch for efficient training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- This work presents a lot of practical results on scaling of hyper-parameters that could be very useful to practicioners.\n- Experiments are performed at reasonably large scales (up to 3B parameters models).\n- They generalize previous formulas such as from [1] to account for the TPP ratio and show that the optimal $\\tau$ depends on TPP.\n- The paper is well written with clear formulation of results in color boxes.\n\n**Weaknesses**\n- Different results in this work seem unrelated to each other, outside the fact that they all study hyper-parameters scaling. For example, the weight decay and learning rate scaling for AdamW in section 2 seems independent from the batch scaling results in sections 3, 4.\n- The empirical results do not always support the claims: for example, I was not fully convinced by the claim \"Sweeping $\\eta$ shows a trend towards an optimal $\\tau$\" in figure 2 by looking at the right figure.\n- I'm not sure why the authors rounded the exponent values in key takeways 1 and 2 (0.5 and 0.4). The results are slightly different? (line 110, 196) Especially line 196 the authors claim to be very close from a previous number by showing two digits precision but round it in the takeaway.\n\nQUESTIONS:\n- figure 2, right : It seems to me that there does not seem to be any trend in $\\tau$ unless I'm mistaken? Could you comment on that?\n- figure 2, left : the trend in $\\tau$ being constant mainly happens at small batch scale. But aren't we mainly interested in how this variable evolves for large batch scale? As soon as B=8064, there seem to be some large discrepancies (factor 10 variations). On the other hand, line 188, you underline that another measurement fails at small batch scale. Could you comment on which regime is of interest for batch when studying scaling laws and link it to the regimes you study?\n- Did you consider plotting figures 3 and 4 in a log-log axis to better see the power-law scaling as linear curves?\n- figure 5 right: I'm not sure to understant if the blue and green curves are fitted on the experiments and if yes, which data points were used and why did you not add the data points on the figures? Also I was wondering which TPP-ratio was used? It seems an important parameter from figure 5 left.\n\nLIMITATIONS:\nYes, the authors discussed limitations in appendix B.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper study scaling laws for many hyperparameters: formulas for how to scale hyperparameters as we scale model size N, dataset size D, and batch size B.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The author conducted numerous experiments and summarized the scaling laws of parameters.
2. AdamW and these hyperparameters are crucial for the pre-training of LLM.

**Weaknesses**

1. The authors only conducted experiments on dense LLMs. Nowadays, sparse activation LLMs are more popular. The authors can extend their experience to the MoE scenario.
2. The author's experience focuses solely on the AdamW optimizer.
3. The authors only conducted experiments on LLM pre-training tasks, which generally only require one epoch of training. It is currently unknown whether the authors' findings are applicable to other tasks that require multiple epochs of training.

QUESTIONS:
The paper cites in [54] that a "surge" phenomenon has been reported in the Adam optimizer. Could this issue potentially undermine the validity of the conclusions presented in this work?

LIMITATIONS:
See Weakness 2 and 3.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper does a comprehensive empirical study over different hyperparameters (HPs) in AdamW. The authors provide scaling laws for weight decay as a function of parameter count, data count and batch size, using models parameterized in muP. The authors also propose a new method for measuring the critical batch size and the optimal batch size and show the dependence of these quantities on the total data, showing that they do not scale as a power law of the target loss or FLOPs.

STRENGTHS AND WEAKNESSES:
The main strength of this paper is in the thorough empirical evaluation and the obtained scaling laws as a guide for how practitioners should change hyperparameters when scaling up. Parameterizing the models in muP removes the confounding effects on the learning rate when scaling up N and the finding that scaling up weight decay is better than learning rate is very valuable. Highlighting the key findings and takeaways in each section leads to a clear exposition of the ideas.

The methodology for fitting B_opt and B_crit is also interesting. I believe that the limitation explained by the authors in section 3.2 regarding the learning rate scheduler could be mitigated by incorporating weight averaging and using a constant learning rate during training, then decaying at various time steps - similar to [1]. Do you believe this will have a major impact on the results?

I do not find any major weaknesses in this paper.

[1] Wen, Kaiyue, et al. "Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective." arXiv preprint arXiv:2410.05192 (2024).

QUESTIONS:
Is there a similar EMA view that could be applied for general optimizers (i.e. Muon, Shampoo etc) - particularly matrix-based optimizers?

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-06