Power Lines: Scaling laws for weight decay and batch size in LLM pre-training
Authors: Shane Bergsma
Nolan Simran Dey
Gurpreet Gosal
Gavia Gray
Daria Soboleva
Joel Hestness
Venue: NeurIPS 2025 poster ()
Keywords: scaling laws for hyperparameters
weight decay
batch sizing
critical batch size
data parallelism
large language models (LLMs)
pre-training
AdamW optimizer
compute-optimal training
Primary area: deep_learning
TL;DR: We derive scaling laws for optimal weight decay and batch size in LLM pre-training, finding optimal (and critical) batch size scales primarily with dataset size; we discuss implications for optimizing time and compute efficiency.

Abstract: Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate η and weight decay λ. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B.  Recent work suggests the AdamW timescale, τ = B/(ηλD), should remain constant across training settings, and we verify the implication that optimal λ scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal τ obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict λopt in advance of large-scale training.  We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast to prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives.

Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness
Cerebras Systems
{shane. bewgsmarrggeoeoeee joolecerbasssrseerr @cerebras net
Abstract
Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate and weight decay x. We study scaling laws for HPs: formulas for how to scale HPs
scale model size N, dataset size D, and batch size B. Recent work [1] suggests the A timescale, T = B B/ (n))), should remain constant across training settings, and we verify the implication that optimal À scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal 7 obeys a precise power law in the lokens-per-paramett thus provides method to accurately predict Apps in advance of large-scale training. We also study scaling laws for optimal batch size Bog (the B enabling lowest loss at a given N, D) and critical batch size Be (the B beyond which further data parallelism becomes ineffective). In contrast prior work, we find both Bapt and Ben scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and under dual training time and compute objectives.
ratio, D/N. This law
AdamW timesrale decrnates with TPP
Optimal batch size grows with data
Critical hatch size grows with data
10P
10*
1.75. 20T
111M
1.7B
Boot (optimal batch size)
111M
Darit (critical batch 82z)
111M
BOTPP
256M
3.38
256M
176.0
90
266M
610M
6TOM
200
TPP
111M
610M
1.78
1.78
3.38-30TPPPP
111M-310FP
Optimal
10
1.78-30T PP
111M
1.044 PTP4
TPP)
16
Bopt 0.0906.p8.38
Beit 0.0471
10-2
10
10
10
1041
TPP (tokens (tokers-per-paraeeee perts--epprprrraa parameeee))
D) (total tokers)
Diiin (min. tokens reach target loss)
Figure Hyperparameters and their power lines: Optimal obeys power law in tokens-per. parameter (left), while optimal batch size (middle) and critical batch size (right) obey power laws in D. Faded markers indicate points not used fitting; all fits generalize well to larger-scale runs.
Introduction
LLMs predictably improve as model size N and training data size D increase [2-4]. Today, state-of- the-art LLMS are trained at computational scales that leave no scope for hyperparameter (HP) tuning, although is widely accepted that good HPs are critical for effective training [5-7].
Both theoretical and empirical efforts have sought to address this. Theoretically, maximal update parameterization (nP) allows the optimal learning rate Nopt and initial weight variance o remain
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

stable when scaling model width [8, 5], enabling a "tune small and train large" strategy. Empirically, DeepSeek LLM [7] adopted "scaling laws for HPs," where optimal batch size Bept and optimal learning rate Togt are estimated at small scale, and then extrapolated via power law fit in total compute FI.OPs, C. A similar approach was used in Kaplan et al. [4], forecasting Bept and Top from loss I. and model size N.
Relying on a unique predicted Bope is inflexible-it precludes adjusting B for compute/time trade-off's or hardware constraints. It is also unclear whether C, L, N, or D (or a combination) best explains scaling. [7] notes, "for models with the same [C] but different model/data allocations, the optimal parameter space varies slightly." Also, no comparable study has been done for weight decay X.
This paper introduces flexible, unified approach to HPs, using both μP and scaling laws. We fit power laws to losses derived from hundreds of yP-trained models, focusing on combinations of X, B, N, and D. We study both compute-optinnl and overtrained models. The fewest FLOPs achieve loss typically occurs when training at =220 tokens-per per- parameter (TPP= D /N) [3, 9). but overtrained models (>20 TPP) offer more-efficient inference [10]. We study TPPs from 20 to 1280.
To capture X's interaction with other scaling HPs (17, B), we model scaling of the AdamW timescale, = B/ (n))). [1] found optimal 7 stable with varying D, but we show it obeys power law in TPP (Fig. 1, left). This law thus enables accurate estimation of Aapt for any N. D. B.
Leveraging these better HPs as B scales, we also study optimal batch size: the Begt that minimizes loss at a given N and D. While Baps scales as a power law in C when TPP is fixed (Fig. 5, left), our results show this arises from more fundamental power-law dependence on D (Fig. 1, middle).
Importantly, increasing B > Bups can still reduce training time (fewer steps) and improve hardware utilization. This raises the question: how much extra data is needed when using large B Prior work defines the critical batch size Batt as the point where training a target loss requires 2 x Dmins with rapidly-diminishing returns in training speed thereafter [11]. We show Berit also scales with D (Fig. 1, right), not as suggested in [4] (Fig. 5, middle), consistent with recent results from [12].
Finally, amid intense competition to advance LLM performance, a key question is: which N, D, and B yield the best trade-off between training speed and compute cost? Using our fit Bar law, we derive Pareto-optimal solutions to these competing objectives, and show that small, overtrained models can be best-offering both faster steps and greater parallelism via larger D (and thus higher Bea).
Key findings and takeaways are highlighted in the paper. Our main contributions are:
The first large-scale empirical study varying weight decay À across N, D, and in LLMs.
Showing the Adam timescale obeys power law, enabling Aops for any N, D, B (Sec. 2).
A new method for estimating Bet suitable for any LR schedule or optimizer (Sec. 3.2).
Confirmation that both oy and Bait scale power laws in D (Sec. 3).
New methods for selecting N, D, and B to trade-off training time vs. compute (Sec. 4).
2 Scaling of the AdamW timescale T, and optimal weight decay Xopt
2.1 Background: μP, Adam W, and Tepoch
juP μP is increasingly used in LLM training [13-18]. With jP, base HPs are tuned on proxy model and then transferred to wider [5] and deeper [19] models. Given the width of the proxy model, dn. and target, de. nP prescribes scaling factors apply to the LR, initial weight variance, and other base HPs. In particular, the optimal base LR, Nopt is scaled down to Topt (do/d) Mope
While μP enables the same base LR to be used across different N. fogt has empirically been found to vary with B [5, 20, 21, 16]. Moreover, recent work has also observed 7opt decreasing in D. leading to proposals for scaling Nope as a (decreasing) power law in D [16, 22].
The EMA view of AdamW Rather than adjusting 77 as D scales, Wang and Aitchison [1] proposed that, if using the AdamW W optimizer [23] with μP, then the weight decay, X, should instead be adjusted. To see this, note an AdamW W update at each step, t, can be expressed in terms of yy as:
m
0 =1 (1 n1)0011 " Vo, +
(1)
2

---

Here, is the yd-adjusted LR, and mi and 8n are (bias- correcoe))) seaoonenniillywwiighhoen moving averages (EMAs) of gradients and squared gradients [24]. Wang and Aitchison [1] observed Adam W's parameters, 0,, can also be viewed as an EMA- of weight updates. Specifically, the standard EMA form Y/ (1 d- + aæe matches AdamW W when = 0,, a = n,,, and 2₁ - m, The +or+ quantity 1/4 1/1/ provides a measure of the number of iterations (i.e., steps) over which updates are averaged: [1] denotes it as Tier. They show that if the timescale is measured in epochs as Tepoch Tier /urr/M M, where MM is seerations-per-ppeee
then the optimal Tepoch remains stable when or D scale (on image tasks). Le., if M scales up, À should be scaled down to maintain constant Tepoch-
2.2 Methods: The Adam W timescale for LLMs, т, and its scaling
Since LLM pre-training only uses one "epoch" of data, normalize the timescale Thtr/ where S is the total number of optimization steps.' Moreover, since S=D/B,
T
RAD
reflects the fraction of past iterations to include in the final weights. While Wang and Aitchison [1] did not vary B, their work suggests this fraction should remain constant B scales. We hypothesize that when moving from compute-efficient to overtrained LLMs, updates can be integrated over smaller fraction of the data; specifically, that Tops decreases as a power law in TPP D/N:
Togil (TPP) c. TPP"*
where and are parameters to be fit. Appendix Algorithm I summarizes the fitting procedure. Taking the yP-adjusted 7/ as our LR, Appt can computed from Eqs. (2) and (3):
B
B TPPPP-TT
Aept 11 D Topt! (TPP)
(4)
G 19.
2.3 Experimental details
We use a GPT2-like LLM [25], with ALiBi embeddings [26] and SwiGLU [27]. We train on SlimPajama [28] and always evaluate over held-out set of 1.1B tokens. We use AdamW and μP, with yP HPs derived from a smaller proxy model, and linear LR schedule, with a 10% warmup followed by decay-to-7ero [29]. Appendix C has full experimental details.
2.4
Results: and scaling À
Sweeping weight decay.
Weight decay, Leaening rate,
Sweeping learning rate, 1]
10P
f
2.9
63
2.0
63
126
Optimal HP value
126
Validation lass
Validation lass
2.8
504
504
1008
10-1
1008
2016
2016
2.7
4032
2.7
8064
8064
2.6
10
2.6
10
10-1
10
102
10 10
10
10
109
101
AdamW timescale,
AdamW timescale,
Figure 2: (610M 20TPP): For each B, we sweep and find Tops (left). Topt is stable around 0.21 21 for B [63, 2016). meaning Aop scales linearly with B over this range (middle). When sweeping (right), the lower boundary over all curves is a bowl with minimum at 0.21; ; the smallest settings have Topt within 2x of this value, but as B increases, Topt quickly drifts higher.
Because it depends on y, timescale r varies during LR decay. Here we compute 7 at peak LR. Notably, two training runs share the same LR schedule (shape) and r (at peak LR), their final AdamW timescale (EMA contributions over the data) will match even if B differs. Appendix E.4 provides further details.

---

Table 1: (610M, 20TPP) Validation losses comparing tuning À vs. 1) across B (data from Fig. 2).
A
B=
63
126
252
504
1008
2016
4032
8064
1.6e-02
0.1
2.595
2.570
2.563
2.573
2.599
2.649
2.755
2.923
Tuned
0.1
2.583
2.570
2.563
2.571
2.597
2.625
2.754
2.923
1.6e-02
Tuned
2.579
2.565
2.563
2.570
2.592
2.637
2.733
2.891
Finding 1: The optimal T remains stable B scales; Xopt scales linearly with B (Fig. 2).
As B increases and À is tuned, we find that Tops remains roughly constant-i.e., changes in B lead to commensurate changes in App (Fig. 2, middle)-but only up to certain point, after which Topt begins to drift. The drift point corresponds to the critical batch size Beit above which gradient information no longer scales linearly with B and diminishing returns set in (Sec. 3).²
If tuned instead, Top fails to scale with B up to Bert (Fig. 2, middle); we observe instead a certain maximum 17. above which training becomes unstable; above this, loss spikes occur from which training does not recover. Consequently, 1j has less flexibility to scale with B; training stability is more fundamental than timescale.
In general, LLMs typically train faster and utilize hardware better with larger B, but only up to Ber:: beyond this point, much more data (and compute) is needed to obtain the same loss, without meaningfully reducing the total number of sequential training steps (the poor trade-off for B Ben is depicted in Fig. 4). Furthermore, Sec. 3 will show that there is also optimal batch size, Bopt. suffer (because batches are small). In practice.
below which loss worse and utilizatio/pparallliiso LLMs should therefore be trained in the regime Bope < B S Berit- Notably, this is precisely the range where we have shown weight decay to scale predictably with batch size; our findings therefore support the direct optimization of weight decay in the most practically relevant training regimes.
Finding 2: With Adam W, we should adjust X, not nJ, as B changes.
Since tuning at scale is infeasible, we need a recipe for selecting HPs in advance. Unlike 17, optimal follows a predictable relationship with B (Fig. 2, middle), making À the more viable target for real-world adjustment. Moreover, since 1) has less flexibility to maintain optimal timescale, we hypothesize adjusting/ À could also be more effective. We tested this by comparing either tuning n (using a default μuP proxy- -itney--unddddd 17). Tuning À was strictly superior in 6 of 8 cases (Table 1).
practice in LLM pre-training
tuning À (using the [3, 32 32-34])
We also compared adjusting À versus 17 as D changes. For a 111M 200TPP model, default HPs obtain a loss of 2.810, wning 1) achieves 2.80 2.808, and tuning obtains 2.805. While differences are small, the key point is that, when scaling B or D: optimizing À alone is viable and effective.
Finding 3: Topt decreases as a power law in TPP; the law holds at scale (Fig. 1, left).
At each N and D, we calculated Topt over all (A, B) pairs; we then fit Eq. (3) to the results. Full details are in Appendix E.2. A precise power law emerges (R²=0.975), with an optimal around 1.0 at I TPP, decreasing to 0.01 at 1000 TPP. 10th and 90th percentiles of fitted m, over all points are (-0.529, -0.507) (computed as in [3] by bootstrapping: re-fitting on 80% of points, 1000x), indicating a reliable power-law trend. A decreasing t stands in contrast to the prescription of Wang and Aitchison [1] (for multi-epoch training), who advocated keeping constant as D changes.
Fig. 1 (left) includes four (labeled) points not used in fitting, but computed later to evaluate predictive ability. Even though some of these points are interpolated in terms of the fitting range (i.e., TPP range), they nevertheless represent much greater scales; e.g., training a 3.3B-30TPP model requires 1000x the FLOPs compared to training the 11 1 model (whose data point is plotted nearby). In other words, the law generalizes across at least 3 orders of magnitude in compute.
TTriniing with B > Bun can be viewed as training with less effective data, thus decreasing the TPP, and shifting Toge higher (by our power law) and thus Auy lower. A decrease the optimal learning rate nays when B > Bun has been previously observed [30, 31J. These works measure TJoee at very large batch sizes, e.g., 10x or 100x Ban. which was not practical at the scales that we trained. Nevertheless, our observation of a similar "surge" phenomenon with Aup, occurring in AdamW rather than vanilla Adam, motivates further study.

---

Discussion The T scaling law also predicts previously-observe HP scaling in the literature. E.g.. for a fixed N. Eqs. (2) and (3) together imply: 7om B Cap D where Cyp and Myp are parameters. This matches Equation (1) in [16], and our implied Map is close to their fit value (see Appendix E.3.1). ). [22] also scales 1) as a power law in D. They note that fitted power law exponents are similar when B is doubled, although the optimal 1y is "higher". More precisely, we can see from their Figure 13 that the optimal 7) appears to, in fact, also conste--o with the derived Toppt equation above.
Key takeaway 1: With AdamW, you can find for small N, D by tuning X. From there, Topt scales x (P/N)-0.5. At larger N, D, set Aops via Eq. (4) and enjoy well-tuned models.
3 Scaling of optimal batch size oopt and critical batch size Berit
We now develop methodology to estimate Bops and Beit over the dimensions of total training tokens D, total compute FLOPs C. and validation loss L. Results show power-law scaling of both Bug and Ban with D, enabling estimation of Bem at scale via small number of test runs at modest budgets.
3.1 Background: Bopt and Berit
Bopt As noted above, recent work has pursued an optimal B: the B achieving lowest loss given N, D. Hu et al. [17] fit Bo using a power law in (estimated) loss, and use μP to set v,. Joint power laws for optimal 8) and B have also been fit [7, 35, 36]. E.g., [7] estimated Bup = 0. 0.292CO.327T (in tokens); we refer to this fit as Bdeepsook below. Li et al. [37] found Topt to scale in N, D, while Begt primarily scales in D. Qwen2.5 [38] also report studying how Toopt and Bope scale with and D (across dense and mixture-of-expert LLMs), but without further details.
Berit Let D be the number of training tokens required to reach target loss 7 when using batch size of B. S D/B is the corresponding number of optimization steps. Doubling a small B doubles per-step gradient information; L. can be reached half the optimization steps, using the same total D (so-called "perfect scaling" [39, 40)). But as B increases further, step-wise gradient information becomes more and more redundant: eventually, much larger D is required to reach L, and S decreases only marginally. McCandlish et al. [11] show that (D, S) pairs can be well fit by the equation:
S/Smin
(D/ Dmin 1)-1
where Smin and Dmia are parameters to be fit. Intuitively, Dmin < D is the asymptotically minimum number of tokens that can reach f. (achieved with Bop)) and Smin < S' is the asymptotically minimum number of steps that can reach i. (achieved B - oc). Eq. (5) defines a hyperbolic curve, like those in Fig. 4, where B controls the position on the curve and can be set depending on the importance of time (higher B + higher D, lower S) or compute (lower B lower D, higher S).
Definition 3.1. The critical batch size at L is defined from the fit of Eq. (5) as Bemt=Diin! Smin-
From Eq. (5),
derive (Appendix F.1), for given B, the D needed compared to Dmin' D = mia (1 + Beit)
(6)
Eq. (6) implies that when B=Bent we require 2 x Dmin tokens (and x Smin steps) to reach L. Bem is a transition point along the D vs. S curve: for B > Buns much higher D is needed for only small reductions in S (Fig. 4). Kaplan et al. [4] refer to Bu as the optimal compromise between time and compute. They determine Ben at smaller scales and fit power law for Be as a function of î.
Eqs. (5) and (6) also imply Bopt is theoretically equal to 1. In practice, loss degrades below a particular Bogt [17, 7, 35], a finding that "appears to contradict the conventional wisdom" about Bent [35]. With well-tuned X, we find small differences in loss across small B, suggesting Eq. (5) may nevertheless provide a good fit to observed data. Appendix has further discussion.
In recent work, Zhang et al. [12] define Ben as the point where D = 1.2 x Dmin- [12] uses a different training setup, with constant LR, weight averaging, and no weight decay. Notably, they observe little change in Ben as N varies at fixed D, but, for 302M model, find power-law scaling in D as But 22.91 D7.17 (in tokens), consistent with observed scaling across models fixed TPP. See Appendix D.3 for further differences with Zhang et al. [12].
5

---

3.4
610M Increasing B
1.78
3.3
192
1536
3.2
304
3072
6144
1012
2.6
Validation loe
3.1
(total trining
Peis-
3.0
oi M001
100
Peen 3079
Font 124m
2.9
Pen
Muk 2017
288
Best
2.
21 H--ppciliic scaling law points Finted Equation (5)
27
Ocserved data
10
10'
1914 10
1012
10 10"
10
(total training Lokers)
(total number of optimization steps)
Figure 3: (111M): Fitted B- specific power laws, LD). for inferring steps to reach tar- get loss L (arrowed blue line).
Figure Eq. (5) fits observed data well: Increasing B (moving leftward along curves) decreases optimization steps (x-axis), but requires more tokens (y-axis) reach target loss (in color). Ben is the transition point along each fitted curve.
3.2 Methods: estimating Bupt and Berit, and their scaling
Estimating Bupt We use the same experimental settings Sec. 2.3. To ensure good HPs, we sweep by factors of 2 at each B, D, N, except at the largest scales (see Appendix Table 3) where we set À via the projected value from Eq. (4). In all figures, B is reported in units of sequences.
Estimating Beiit Unlike Bop,. measuring Be requires training models with different B the same L. Unfortunately, we do not know a priori how many steps are required to reach L, yet we need this information to configure LR schedule that reaches its minimum value on the final step (the typical setup, shown to be consequential in prior work [3, 4)... Unfortunately. it is not feasible to search for the precise steps needed, i.e., by conducting training runs with different schedules/step budgets.
McCandlish et al. [11] address this issue by performing a single training run at a constant LR, while Zhang et al. [12] also use a constant LR, but use weight averaging to frequently generate higher-quality checkpoints for evaluation (Appendix D.3).
In contrast, we desired a method agnostic to the LR schedule. We achieved this by fitting batch-size- specific power laws that model how loss scales with D. These laws allow us to accurately interpolate the D required to reach L. Fig. 3 depicts, for different and given i, the interpolated D values (intersection points of arrowed line and fitted loss curves). The full process to obtain Bent f. is:
1. For each B, train over different D, and subsequently fit B-specific power law LD(D)= = EN D-8 on the resulting loss values (fitted curves in Fig. 3).
2. Use fitted (DD to infer the DB needed reach as: DB = Lg) (Dan/L-EN)) BN)
3. Fit Eq. (5) on the resulting S=D/B) pairs, and obtain Bunl D mina/ Smin-
This method makes no assumptions about the LR schedule or optimizer. while enabling measurement of Ben arbitrary losses without re-training. Appendix F.2 provides further details, including fits of L.(D) at other model scales (Fig. 9) and summary of the full procedure (Algorithm 2).
Bopt and Bar scaling We collect Bop across different N and D, and fit power law in both data D and compute C (via the standard approximation 6N [4, 3)), For Bert we use the procedure described above to estimate Bent across multiple L, across different N. From each Bert estimate, we obtain a pair (Dmin, Buit). We propose that Bait follows power law in D according to:
Beri (Dmin) = CBe.. D Me
(7)
min
Where CBa and m.n are fit on the (Diin Beat) pairs.

---

3.3
Results: Bopt and Bertt
10a
10
30TPP
N
Roa( 0.0477124
111M
111M
Bey (D) 0.0306174.56
Bopt (optimal batch size)
ST
Bri (critical batch size)
266M
Optinal (oriiicll batch size
10p
610M
50TPP
610M
1.78
S0TPP
1.78
10
TCPPP 20TPP
@0TPP
20 PP
10a
BOTPP 20TPP
P07
20TPP
10a
10e
Bosqpask (C) = 0. 092
i: -N=100M
N=
10a
20TPP
20T PP
NM1B
100B
10
10
1001
10
10
2.25
2,50
2.75
3.00
10
10
10
C (Training FLOPs)
L (target validation loss)
D (tokens)
Figure 5: Prior work suggests Bogt scales in C (left) and Ban in loss (middle), but this only holds at a fixed N/TPP (same data as Fig. middle/right): Fig. shows scaling in D is the fundamental relationship. Plotting Baeepseck (C)) law from [7] (right), but over D (using C' ss 6N D to obtain C for the spurious dependence on N), we see [7] used generally efficient B values (i.e., within the Bogt < B < Bent regime) despite fitting C' rather than D (Bop and Beit lines from Fig. 1).
Finding 4: Eq. (5) provides a decent fit the trade-off between training time and compute.
Across different model scales and loss targets, we consistently find that our (D, S) pairs fit Eq. (5) well (examples in Fig. 4, appendix Fig. 10). Fits are worse at very small B, as noted above: smaller batches are not monotonically more efficient; Appendix B discusses some potential reasons for this.
Finding 5: Baps and Berit obey power laws in and Dmins not in or L.
Bug and Bent datapoints fit power laws quite well (Fig. 1, middle, R²=0.984) and (Fig. 1, right, R² R.00.)40). 10th and 90th percentiles over all points are (0.367, 0.391) for fitted m. and (0.491, 0.526) for m m... (computed as in Sec. 2.4). Note m..... is higher when fitted over all points (as opposed to only small-scale runs), partly reflecting the 111M points trending lower as TPP increases.
Our fitted Be power law exponent is very close to that from [12]: 0.47 vs. 0.462. Given the many differences in approach (including dataset, use of weight decay, LR schedule, etc., Appendix D.3), this agreement suggests the fundamental relationship of Ban with D persists across such differences.
Fig. 5 (left) plots Bapt versus C' and Fig. 5 (middle) gives Buit versus L using the same data in Fig. 1. In each case, a power law does not fit all points (as proposed previously), but points at the same N, same TPP, can roughly be linked by (parallel) lines. This is consequence of power-law scaling in D (see Appendix F.4). That is, scaling in D is the fundamental scaling relationship: Bupi and Beal both scale in D regardless of TPP, model size, or loss it is only when using another (misleading) scaling factor such C or L. that TPP or model size appears important, as in these plots.
Fig. (right) compares the recommended batch sizes from Bacessek to those from Bopt and Beit- Since Baeessok scales in C, it is larger for larger N. Over range of modern model sizes, Bdeeseok values generally fall between our projected Beg and Butt varying in the extent to which they are compute-efficient (close Boy)) lime-efficient (close Bund)
Finding 6: Weight decay affects the accuracy of fitted batch size scaling laws.
Prior work has typically held fixed when fitting batch-size scaling laws [7, 35, 12]. Doing so not only degrades loss (Sec. 2) but also reduces the accuracy and generality of the fitted scaling relationships. We demonstrate this in appendix Table rather than tuning À for each B, we train with several fixed À values across all runs. As Table 5 shows, increasing À systematically raises the estimated Bop This arises because the fundamental scaling variable is the Adam timescale (r(rDD): when À increases, the batch size that minimizes loss must increase proportionally to preserve the optimal r. In Appendix F.5, we show that these effects distort the fitted power-law slope and reduce fit quality (R²), leading to scaling laws that do not generalize to large-scale training- even
7

---

if the same fixed weight decay is used there. When À is tuned to maintain the optimal timescale (final row of Table 5). the resulting aun follows a clean and accurate power law.
Similar distortions occur for Ben when À is fixed (Appendix F.5).
Key takeaway 2: You can estimate Bogs and Bent for a small N by training with different B, D and Aaps and computing loss. From there, Bupt X D and Bent & D At larger N and D, Eq. (6) lets you estimate trade-offs in FLOPs (xr D) vs. training time (x S = D/B) at different B.
4 Training settings for balancing time and compute
4.1 Background: compute-optimal and overtrained models
Given a fixed training FLOPs budget, C, how should we allocate model size N versus number of training tokens D in order minimize loss? Hoffmann et al. [3] propose model loss as:
L(N, D) E Nooma.Na
Devns D
(8)
Nconat: a, Deonst: and B are parameters fit on observed training runs. From Eq. (8), [3] derives functions for loss-optimal (C) and Dem((C (constraining L(N, D) by C Rs GND). Results indicate Nop and Dapt scale roughly equally as C increases, with the optimal D/N ratio relatively constant around 20 TPP. Replication studies have found similar results [9, 35], and 20 TPP has become a rule-of-thumb for compute-optimal training [13, 12].
Overtrained, inference-eefficient models [10, 42, 43] have largely trained with similar batch sizes to those used in compute-optimat training; such efforts should now consider training with much greater data parallelism, leveraging finding that Bops and Bui will be higher given the higher training D.
4.2 Methods: exploring the trade-offs of FLOPSs vs. time
To compare models of different sizes on a common temporal axis, we must map number-of--ppiiizzttonn-ser to common temporal scale. Our initial approximation is Training Time x Total FLOP/BB, which also FLOPs per token times number of steps. E.g., FLOPs & 6N D, Training Time x 6.V D/B 6N S. This aligns well with our measured runtimes: doubling N doubles step time; doubling halves wall-clock time (for the same S).
Now, assume a model of size N can train loss 1 using Dmin tokens (here min denotes using Bopt). Let us refer to N and Dmin as base setting. A variety of N, Dmin pairs can reach in the Bog setting, from small models trained on many tokens, to large models trained fewer tokens. [3] refers to these as iso-loss contours of Eq. (8). Suppose a given base setting requires C(N, Dmin) FLOPs. From this setting, we may increase B decrease training time (fewer steps), but Eq. (6) indicates need for (1 + B/ Beit) extra data order to reach the same L. If FLOPs is linear in D (as in C' = D), we will require the same proportion of extra FLOPs, i.e.,
CN Dmin, B) C(N, Dmin) ) + B/Beit (Dmin))
where C+ Dmin, B) denotes the total FLOPS needed at B > Bopt, and Buni (Dmin) captures that the excess FLOPs depends on Bum which itself scales with D In other words, the base setting dictates Ben. and B/ Bun dictates the excess FLOPs.
Consider target FLOP budget of C, Dmin: B) = C and the goal of reaching L as fast as possible. Since time x Total FLOP//R, time minimized by maximizing However, by construction, B not free variable: it is constrained by Eq. (9) and can be expressed as function of N and Dmin:
B(N, Dmin) = (C(N, Dmin) (CC(x.Dmm
Ô
-1) -1)
Berie(Donle (Dmin)
(10)
Time is therefore minimized by finding N, Dmin that maximize this function (over all the N, Dmin that train to loss i.). The C/C ratio is measure of the excess FL.OPs that can be spent toward increasing B; largest when C(N, Pmin) is smallest, i.e., when N and Dmin is most compute-efficin (i.e., N/ Dmin x 20 TPP). But Eq. (10) a whole captures an elegant tension between compute efficiency
8

---

and Bem:: we can maximize B (and minimize training time) by either (1) minimizing the FLOPs of the base setting (generating more excess FLOPs for increasing B), or (2) maximizing Dmin (overtraining. which increases Ba min)). For a given Č, either (1) or (2) may take precedence.
We use the following procedure to explore the time vs. compute Pareto frontier for a target loss t:
1. Fit Eq. (8) on our copt training runs. Express resulting L(N, Dmin) as Dmin g (N).
2. Using Dmin e, (N), get contour points (N, Dmin) of the given L. Each such pair consumes C(N, min) x 6.N Min FLOPs and takes D time (rightmost points on Fig. 6 curves).
3. Use Eq. (9) to compute C+ Dmin= B) as we scale B (crucially, using the estimate of Beit from fitted Eq. (7)), and generate further points along each curve.
4. The non-dominated points over all curves provide the time vs. compute Pareto frontier.
4.3 Results: balancing time and compute
We carry out this procedure using model sizes of 150M, 210M, 550M, 1.IB, and 2.1B, and loss target of f.=2.6, yielding iso-loss contour points from 150M 600TPP to 2. 2TPP. Our fit of Eq. (8)
yielded a=0.313 x 282, giving an optimal TPP ratio 200.6 at ==2.6.
Finding 7: Overtrained, but not undertrained, models are on the FLOPs vs. time Pareto frontier.
f-266
X10lG
I-26
2.18-2TPP+
610M, 20TPP
2.85
2.18-2TPP+
I. 1R-STPP
2.75
256M 80TPP
1.1TTTPPP
Total Total FLOPs
1021
350M 20TPP+
210M-200TPP
1500-600TPP
FLOP=
Total F
2.50
Pareto front
2,80
2.75
Total FLOPs
J0?
550M 20TPP
150M-600TPP4
210M-200TPP
2:25
Validation
2.70
Increasing 17
Increasing B
2.65
FOOOOCO
2.60
Better
1.25
Pareto front
2.55
Better
10
10a4
10a2
10
10
10
Total FLOPs Jas Training Time]
Total FLOPs [s= Training Time]
Number optimization steps,
Figure 6: (left): Iso-loss curves illustrating time-compute Pareto frontier (f.=2.6). As B increases along curves, more compute (y-axis), but less time (x-axis) is required. Here time x Total FLOPs/ n. (middle): Observed runs where some overtrained models (red line) are frontier: L in color, B labeled. (right): Iso-loss curves, but where time = steps; a very different frontier emerges.
Specifically, when using Training Time x Total FLOPS\ B, we find overtrained models are FLOP-optimal certain time budgets (Fig. 6, left). Compute-efficient 20 TPP are optimal in pure FLOPs (i.e.. ignoring time), as expected, while cempute-efficien and overtrained models dominate undertrained (< 20 TPP) models in time and FLOPs. Indeed, this is expected from Eq. (10): undertraining reduces both the excess FLOPs and Bun terms, and thus is never optimal with this model of time.
Finding 8: When using B > Baps it is Pareto-inefsiciet to train to 20 TPP.
Notice that Fig. 6 adds " "...TPP+" to curve labels. Here the + sign is reminder that as we increase B, require (1 + B/ Berit) extra data to reach the same L: i.c.. points with higher B are trained to higher actual TPP than the base setting. For example, once the 2TPP+ curve in Fig. 6 reaches 10x its minimum FLOPs, it is actually training at 20 TPP. Since starting from undertrained base setting is never Pareto optimal (as just discussed above), it is always suboptimal to train a model with large B to 20 actual TPP. If large-batch training is needed, the configuration should start from 20 TPP+ base setting and scale from there (to >2 TPP).
We can see this finding play out in real training runs. Fig. 6 (middle) demonstrates observed runs where our 266M S0TPP models dominate our 610M 20TPP models (i.e., FLOPPs and time) when both train with large B. (Note in this plot, results are not iso-loss: the frontier is over L, C, and time.)

---

Finding 9: The Pareto-optimal settings depend on the formulation of time/purallelism strategy.
While FLOPs/B is a good model of data parallel training, it does not incorporate the potential for model parallelism [44]. In the extreme we could assume that all 6N FLOPs could be executed concurrently per input token. Under this formulation, training time is proportional only to the number of steps, regardless of model scale. Fig. 6 (right) shows the Pareto frontier that would result from this formulation; if we pay no time cost for larger models, we can train faster by using undertrained large models, although, exactly as with overtrained models, they suffer in FLOOPs
While LLMs cannot be fully parallelized due to the inherent sequential nature of a Transformer's 's layer- by-layer computation, this formulation could be refined by incorporating depth or other architectural features. For example, Inbar and Sernau [45] predict training time via linear regression over total FLOPs and memory-cop operations, fit to real (single-TPO TPU) runs. As formulations improve, different Pareto-optimal configurations will emerge.
Finding 10: Inaccurate Bert scaling leads to inaccurate Pareto-optimal configurations.
Because the Pareto frontier Sec, 4 depends directly on the Berit power-law fit, any error in that fit produces corresponding errors in the predicted trade-offs between training time and compute. Accurate Bei estimation, in turn, depends on effective tuning (Finding 6 and Appendix F.5). When À is fixed in standard practice (and thus the (D) slope is misestimated), Bei will be systematically over- or underpredicted at scale, altering the computed fronticr and the apparent Pareto-optimal configurations. To illustrate, artificially varying the Bait exponent changes which models appear on the frontier: as the exponent increases (and Ban rises), higher-TPP models move to the frontier, when Bem is underestimated, only low-TPP (e.g.. 20 TPP) models appear Pareto-optimal. Hence, an inaccurate Ben scaling law produces misleading frontiers and can lead to unexpectedly- longer training durations and suboptimal compute allocations.
Recent work suggests that other estimators of Beni (such as those based on the gradient noise scale [1I]) can also be systematically biased (Appendix D.2), leading to similar distortions in the Pareto frontier.
Key takeaway To balance time and compute at a target loss, select (N, D from Eq. (8), determine Beni min)) via Eq. (7), and use Eq. (9) to estimate compute for any B. Under Time & FLOPs/D B, the resulting time compueee trade-off off favors higher D (overtraining) (Fig. 6).
Conclusion
We have presented a comprehensive empirical study of hyperparameter scaling laws in LLM pre- training, focusing on weight decay and batch size. Our approach leverages the AdamW timescale (T) to develop robust scaling relationships that predict optimal hyperparameter settings across broad spectrum of model (N), dataset (D), and batch sizes (B). We demonstrated that optimal decreases as a power law with the tokens-per-paramee per-parameter ratio, providing a systematic method to set weight decay optimally across diverse training scenarios.
Furthermore, we introduced novel, practical methodology for estimating critical batch size (Bait). Our findings diverge from influential prior work that tied Ban predominantly to compute or loss, while agreeing with the recent findings of [ [12] that underscore dataset size as the principal scaling factor. Additionally, we showed that contrary to previous studies suggesting optimal batch size (Bop) scales primarily with compute, it also exhibits clear power-law dependence on D.
Also, our analysis of Pareto-optimal configurations reveals an important strategic advantage for smaller, overtrained models in scenarios where rapid training and high parallelism are prioritized.
Appendix B notes limitations and directions for further study suggested by our results. particular, as inference-time scaling comes to the fore, inference time and compute must also be considered as first-class Pareto objectives. Moreover, finer- grained configuration decisions, such as model depth and context length, should be considered along with N, D, and B.
10

---

Acknowledgments and Disclosure of Funding
We thank the NeurIPS reviewers for their helpful feedback. None of the authors received third-party funding or third-party support for this work. None of the authors have financial relationships with outside parties that could potentially be perceived influence this research.
References
[1] Xi Wang and Laurence Aitchison. How set Adam W's weight decay as you scale model and dataset size. arXiv preprint arXiv:2405. 13698, 2024.
[2] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically, 2017.
[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35, 2022.
[4] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
[5] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, 2021.
[6] Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 2.0.122200911322,, 2023.
[7] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. DeepSeek LLM: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.0298 2024.
[8] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:20/1.. 14522, 2020.
[9] Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
[10] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023.
[11] Sam McCandlish, Jared Kaplan, Dario Amodei, et al. An empirical model of large-batch training. arXiv preprint 181.0:882.006622 2018.
[12] Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Difan Zou, Udaya Ghai, Dean Foster, and Sham Kakade. How does critical batch size scale in pre-training? arXiv preprint arXiv:2410.21677
2024.
[13] Nolan Dey, Gurpreet Gosal, Hemant Khachane, William Marshall, Ribhu Pathria. Marvin Tom, and Joel Hestness. Cerebras-GPT: Open compute-optimal language models trained on the Cerebras wafer-scale cluster. arXiv preprint urXiv:2304.03200 2023.
[14] Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, and Joel Hestness. BTLM-3B-8K: 7B parameter performance in 3B parameter model. 2023.

---

[15] Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, al. Jais and Jais-chat: Arabic- centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:23008. /6149, 2023.
[16] Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D Cox, and Rameswar Panda. Power scheduler: A batch size and token number agnostic learning rate scheduler. arXiv preprint arXiv:2408.
2024.
[17] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang. Weilin Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06330 2024.
[18] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprini arXiv:2404. 14219, 2024.
[19] Nolan Dey, Bin Claire Zhang, Lorenzo Noci, Mufan Li, Blake Bordelon, Shane Bergsma, Cengiz Pehlevan, Boris Hanin, and Joel Hestmess. Don't Ion'tin be lazy: CompleteP enables compute-efficient deep transformers. arXiv preprint arXiv:2505.01618, 2025.
[20] Lucas Lingle. A large-scale exploration of p-transfer. arXiv preprint arXiv:2404.05722, 2024.
[21] Lorenzo Noci, Alexandru Meterez, Thomas Hofmann, and Antonio Orvicto. Why do learning rates transfer? Reconciling optimization and scaling limits for deep learning. arXiv preprint arXiv:2402. 17457, 2024.
[22] Johan Bjorck, Alon Benhaim, Vishrav Chaudhary, Furu Wei, and Xia Song. Scaling optimal LR across token horizons. arXiv preprint 1rXiv12400.19911, 2024.
[23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[25] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
[26] Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations, 2022.
[27] Noam Shazeer. GLU variants improve transformer, 2020.
[28] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. Web page, 2023.
[29] Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, and Joel Hestness. Straight to zero: Why linearly decaying the learning rate to zero works best for LLMs. arXiv preprint 1535522502..5988 2025.
[30] Shuaipeng Li, Penghao Zhao, Hailin Zhang, Xingwu Sun, Hao Wu, Dian Jiao, Weiyan Wang, Chengjun Liu, Zheng Fang, Jinbao Xue, et al. Surge phenomenon in optimal learning rate and batch size scaling. arXiv preprint arXiv:2405. 14578, 2024.
[31] Oleg Filatov, Jan Ebert, Jiangtao Wang, and Stefan Kesselheim. Time transfer: On optimal learning rate and batch size in the infinite data limit. arXiv preprint wrXiv:2410.05838, 2024.
[32] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
12

---

[33] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxan- dra Cojocaru, Mérouane Debbah, Etienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The Falcon series of open language models. arXiv preprint 1rXiv:2311.66867 2023.
[34] Aleph Alpha. Introducing 1-LLMM Transparent and compliant. Web page, 2024. Ac- cessed: 2024-09-07.
[35] Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon, Resolving discrepancies in compute-optimal scaling of language models. arXiv preprini ar arXiv:2406. 19/46, 2024.
[36] Quentin Carbonneaux, Gal Cohen, Jonas Gehring, Jacob Kahn, Jannik Kossen, Felix Kreuk, Emily McMilin, Michel Meyer, Yuxiang Wei, David Zhang, et al. CWM: An open- weights 1lm for research on code generation with world models. arXiv preprint arXiv:2510.02.887 2025.
[37] Houyi Li, Wenzhen Zheng, Qiufeng Wang, Hanshan Zhang. Zili Wang, Shijie Xuyang, Yuantao Fan, Zhenyu Ding, Haoying Wang. Ning Ding, Shuigeng Zhou, Xiangyu Zhang, and Daxin Jiang. Predictable scale: Part I, step law optimal hyperparameter scaling law in large language model pretraining. arXiv preprint arXiv:5503.04715 2025.
[38] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprini arXiv:2412. 115,,115500 2024.
[39] Siyuan Ma, Raef Bassily, and Mikhail Belkin. The power of interpolation: Understanding the effectiveness of SGD in modern over- vee-parametrrzed learning. In International Conference on Machine Learning, pages 3325-3334. PMLR, 2018.
[40] Christopher J Shallue, Jachoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(1 (1 12) 2019.
[41] Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405. 18392, 2024.
[42] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: suite for analyzing large language models across training and scaling, 2023.
[43] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arX 2024.
[44] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari. Jared Casper. Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using DeepSpeed and Megatron train Megatron- Turing NLG 530B, large-scale generative language model. arXiv preprint arXiv: 19900, 2022.
[45] Itay Inbar and Luke Sernau. Time matters: Scaling laws for any budget. arXiv preprint arXiv:2406. 1892,, 2024.
[46] David Patterson. Joseph Gonzalez, Quoc Le, Chen Liang, Lluis Alquelliiuel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104. 10350, 2021.
[47] Emily M Bender, Timnit Gebru, Angelina McMillan-Majo, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610-623, 2021.
13

---

[48] Emma Strubell, Ananya Ganesh, and Andrew McCallum, Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.0224. 2019.
[49] Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305. 14342, 2023.
[50] Kimi Team. Kimi K2: Open agentic intelligence. Tech report, 2025.
[51] Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pages 1842-1850. PMLR. 2018.
[52] Nikhil Vyas, Depen Morwani, Rosie Zhao, Mujin Kwun, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham Kakade. SOAP: Improving and stabilizing shampoo using adam. arXiv preprint 14X.112409.11321,, 2024.
[53] Kaiyue Wen, Zhiyuan Li, Jason Wang, David Hall, Percy Liang, and Tengyu Ma. Understanding warmup-stable-decat learning rates: river valley loss landscape perspective. arXiv preprint arXiv:2410.05 192, 2024.
[54] Tanishq Kumar, Zachary Ankner, Benjamin F Spector, Blake Bordelon, Niklas Muennighoff, Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision. arXiv preprint arXiv:2411.04330, 2024.
[55] Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, and Micah Gold- blum. Small batch size training for language models: When vanilla SGD works, and why gradient accumulation is wasteful. arXiv preprint arXiv:2507.0710/ 2025.
[56] Alexandru Meterez, Depen Morwani, Jingfeng Wu, Costin-Andrei Oncescu, Cengiz Pehlevan, and Sham Kakade. Seesaw: Accelerating training by balancing learning rate and batch size scheduling. arXiv preprint arXiv:2510. 147177, 2025.
[57] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher, 2022.
[58) Atli Kosson, Bettina Messmer, and Martin Jaggi. Analyzing & reducing the need for learning rate warmup GPT training. arXiv preprint wwXiv22410.23922 2024.
[59] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint fttf::770.038880 2017.
[60] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. arXiv preprint arXiv: 1rrXiv:/904.000 01044990449222 2019.
[61] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
[62] Noah Golmant, Nikita Vemuri, Zhewei Yao, Vladimir Feinberg, Amir Gholami, Kai Rothauge, Michael W Mahoney, and Joseph Gonzalez. On the computational inefficiency of large batch sizes for stochastic gradient descent. arXiv preprint
2018.
[63] Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. Advances in neural information processing systems, 32, 2019.
[64] William Merrill, Shane Arora, Dirk Groeneveld, and Hannanch Hajishirzi. Critical batch size revisited: A simple empirical approach to large-batch language model training. arXiv preprint wwxiv:2505.23971 2025.
[65] Alex Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv preprint 1rXiv:1404.5997, 2014.
14

---

[66] Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting distributed synchronous SGD, arXiv preprint arXiv: 164.0008,, 2016.
[67] Samuel Smith and Quoe V Le. A Bayesian perspective on generalization and stochastic gradient descent. arXiv preprini arXi:: 2017.
[68] Samuel L.. Smith, Pieter-Jan Kindermans, and Quoe V. Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
[69] Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: Closing the generalization gap in large batch training of neural networks. Advances in Neural Information Processing Systems, 30, 2017.
[70] Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and scaling rules for adaptive gradient algorithms. Advances in Neural Information Processing Systems, 35:7697-7711, 2022.
[71] Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350, 2017.
[72] Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, 31, 2018.
[73] Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. arXiv preprint arXiv: 12281, 2018.
[74] Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[75] Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. arXiv preprint TrXiv:1910.074555 2019.
[76] Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamics: Lear- ing dynamics of neural network with normalization, weight decay, and sgd. arXiv preprint 2rX1v:2006.08419, 2020.
[77] Atli Kosson, Bettina Messmer, and Martin Jaggi. Rotational equilibrium: How weight decay balances learning across neural networks. arXiv preprint arXiv:2:05..12212 2023.
[78] Francesco D' Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammar- ion. Why do we need weight decay modern deep learning? Advances Neural Information Processing Systems, 37:23191-23223, 2024.
[79] Nolan Dey, Quentin Anthony, and Joel Hestness. The practitioner's guide to the maximal update parameterization. Blog post, September 2024.
[80] Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz Odrzygóádž. Piotr Sankowski, Marek Cygan, and Sebastian Jaszczur. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07877, 2024.
[81] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540 2024.
15

---

A Broader impacts
This paper presents methods to train LLMs more efficiently: practitioners can use our methods to reduce the total compute FLOPs used to train models, subject to time constraints. Given the intense pressure to advance LLM capabilities as quickly as possible, our methods can therefore reduce the associated environmental and financial costs of LLM training [46, 47].
Moreover, hyperparameter tuning a key contributor to these costs, and impairs equity Al research, as tuning success depends directly on researcher finances [48]. We hope our exploration of optimal hyperparameter scaling can reduce the burden of hyperparameter tuning scale and thus improve equity in AI.
B Limitations
While our findings corroborate prior work and provide strong evidence for the proposed scaling laws in T, Bepts and Bett, there are several limitations that merit further study.
EMA perspective As the EMA perspective regards parameters y/ as function of updates zy- fails to account for 2, actually depending on earlier values of yu (c.g.. 1/-1)). Yet although this perspective has formal limitations, we nevertheless find it useful conceptual model of training, it predicts behavior that is supported by experiments.
Optimization and training setup Our work focuses AdamW (the standard optimizer for LLM training). While the EMA perspective applies directly to other optimizers that use decoupled weight decay, such as Sophia [49] and MuonClip [50]. it may not apply to approximate second order methods, e.g., Shampoo [51]. However, it can be used when applying Adam W (and related optimizers) in Shampoo's 's eigenbasis, which was shown be effective in SOAP [52].
We present results with a single (standard) learning rate schedule. Our method for obtaining B estimates would be quite efficient with warmup-stable-der (WSD) schedule [17, 53], could perform a single training run with each batch size, but decay at various milestones in order to get points along the scaling law, essentially following the approach in Hägele al. [41], but with separate laws for each batch size.
We used the maximal update parameterization in all experiments, which generates a learning rate adjustment for each model width. Our results suggest this approach enables good models at arbitrary N, D, and B when combined with adjustments to A. This strategy is informed by experiments comparing re-adjusting vs. in Sec. 2.4. However, it is not feasible, at this scale, to verify whether substantially better models could be obtained by sweeping the full cross-produce of r and À values.
Our study specifically focuses on the practically important setting of single-epoch LLM pre-training. Wang and Aitchison [1] indeed noted differences in optimal when using multi-epoch training, possibly due to data repetition. Reconciling these differences by isolating the effects of repetition versus scale is an interesting follow-up direction.
Here we only experimented with single dataset, vocabulary, and context length. We obtained similar Berit scaling law to Zhang et al. [12], but it would be interesting to see if differences in the coefficient of our power laws could be attributed to specific differences approach (c.g., differences in dataset, context length, learning rate schedule, use of weight decay, etc.). Appendix D.3 has further discussion of differences with Zhang et al. [12].
We have also not explored how changes in numerical precision could affect scaling laws. Recent work [54] showed that, in terms of scaling laws, lower precision reduces the model's effective parameter count. This suggests precision would have no impact on scaling of Beg or Ben which do not scale in N. Lower precision, however, could increase the effective TPP (via smaller effective N). thereby altering Tope-
Small batches, large batches, and dynamic batch sizing We consistently find that smaller and smaller batches do not grow asymptotically closer to Dmin. predicted by theory, but eventually degrade in loss. One possibility is that tuning is not sufficient with very small B, and further tuning of other hyperparameters may be needed, such as the Adam 3 parameters (as suggested in recent
16

---

Table 2: Model architectures used in experiments
Model
dmodet
Riagers
dhead
111M
768
10
64
266M
768
32
64
610M
2048
10
64
1.7B
2048
32
64
3.3B
2048
64
64
Table 3: Models, tokens- poxens-per-parametter (TPP) and corresponding dataset sizes (in tokens) used in main experiments. We also list the total number of batch sizes, B, trained at each scale and TPP, as well as the number of B for which we tuned X. For B where was not tuned,
inferred via the
Topt scaling law (Sec. 2). Additional sweeps of 7, were done at each B at 6 610M-20TPP scale for the experiments in Sec. 2.4. Around 400 different LLMs were trained in total across all the experiments.
Model
TPP
D
Number of B Number of B with À tuned
111M
20
2.19B
8
111M
80
8.76B
8
111M
200
21.9B
7
7
111M
320
35.0B
8
8
111M
1280
140.1B
1
266M
20
5.31B
7
7
266M
80
21.2B
7
266M
320
85.0B
6
6
266M
1280
339.8B
0
610M
20
12.1B
8
8
610M
80
48.5B
7
610M
200
121.3B
6
6
610M
320
194.1B
1
1.7B
20
34.3B
7
7
1.7B
80
137.2B
7
1
1.7B
320
548.6B
0
3.3B
20
66.5B
O
3.3B
23
76.5B
3.3B
30
99.8B
2
1
work [35, 12, 55]). Some preliminary tests using the 32 scaling rule from Marek et al. [55] showed loss improvements small B. Since unlikely to train with small batches scale, and using them even with smaller LLMs significantly impairs our ability to train both efficiently and quickly, is unfortunately difficult to justify further exploration in this direction.
Regarding large batches, our methods do not account for the many practical systems-related issues, including bandwidth and communication overheads, memory limits of hardware, synchronization delays, etc. Moreover, as batch sizes increase, techniques such as optimizer sharding may needed, which further complicate performance model [33]. Our scaling laws do, however, explicitly define a practically relevant regime of training batch sizes: Bops < B < Buii- Practitioners can leverage this identified regime alongside system-specifi profiling (e.g., evaluating utilization various batch sizes) to select optimal settings balancing algorithmic and systems constraints.
Exploring optimal dynamic batch sizing is natural future direction for our work. While the potential gains were found to be small in theory by McCandlish al. [11], more recent work has found significant wall clock speedups [56].
C Experimental Details
Table provides details on the model architecture hyperparameters for models used in the experiments. Table 3 provides, for each model scale and TPP, the dataset sizes used in training, the
17

---

Table Tuned hyperparameters for uuP proxy model
Wase
8.67e-02
f)
1.62e-02
Cripput 9.17
Ckoutput 1.095
number of batch sizes tested, and the number of batch sizes for which À was tuned. Around 400 models in total were trained for the main experiments.
All the models in our main experiments were trained on the SlimPajama dataset [28], cleaned and deduplicated version of the RedPajama dataset. We use the GPT-2 [25] vocabulary of size 50257, and context length of 2048 tokens. Following standard practice, we do not apply weight decay or bias LayerNorm layers. AdamW settings are 31 = 0.9, B2 = 0.95, and c 1e-8. Validation loss is always computed over held-out 1.1B tokens, regardless of training TPP. We report cross-entrope loss. By default we parameterize with μP, with hyperparameters set via proxy tuning, as described below.
For given TPP, all models have the exact same warmup phase: linear warmup of the learning rate from O to the maximum value. In all our runs, warmup was 10% of the total steps. Learning rate warmup is standard practice in LLM pre-training [32. 57, 42, 43, 58].
All models in the main experiments were trained on a Cerebras CS-3 system. 610M -arammaeeeeerr 20TPP models take roughly 6 hours each to train single CS-3.
For given model configuration, we find results be very stable across random seeds. To quantify the variance, we repeated 11 Mepraaeeeerr,, 20 TPP training four additional times for six different hyperparameter settings, resulting in 5 total validation loss results for each of the six training runs. Standard deviation of the validation loss was below 0.003 in all cases.
Proxy model hyperparameter tuning To find the optimal jP hyperparameters (HPs), we trained 39M proxy model using width dmodes of 256, with 24 layers and head size of 64. We trained this model on 800M tokens with a batch size of 256 sequences and a context length 2048. We randomly sampled 350 configurations of base learning rates, base initialization standard deviation, and embedding and output logits scaling factors, and used the nop-performing values as our tuned HPs (Table 4).
D Additional related work
D.1 Optimizers for large-batch training
Prior work has explored optimizers designed specifically for large-batch training, including LARS [59] and LAMB [60]. It is instructive to consider these prior findings in light of the scaling laws from our paper. In particular, both original BERT [61] and the LAMB replication were trained on 85.2B tokens. Applying our fitted Be power law over D=85.2B, we obtain an estimated Ber of about 12M tokens. Original BERT was trained for 90% of steps with a batch size of 65K tokens (512 sequences of length 128). LAMB increased the batch size to 4M tokens (32K sequences), justifying their claim, "BERT training can be reduced from 3 days to just 76 minutes" [60]. However, based on the predicted Bait of 12M, batch size 4M is still well within the expected range of efficient batch sizes. Moreover, the LAMB paper later notes, "we did not observe any speedup by increasing the batch size from 65536 to 131072 [sequences, or 16.8M tokens]." In other words, they reach the point of diminishing return exactly where exceeds our predicted Beni-
It is likely that some optimization issues solved by LAMB (to enable stable large-batch training) are solved other ways in modern LLM training setups, via, e.g., gradient clipping, pre-LayerNorm placement, better initialization and stability control through jaP etc. Scaling À rather than 1 with B, as we propose, further supports stable, efficient training. However, gradient redundancy imposes an inherent limit on useful batch sizes, ensuring critical batch size remains relevant.
18

---

D.2 Critical batch size
Observations of critical batch size have previously been related to data complexity [62], loss curva- ture [39, 63], and model architecture [40].
Merrill et al. [64] define Ben the largest B such that loss does not degrade by more than fixed fraction e from the Beg setting. They measure this Bem instantaneously throughout training, by
repeatedly branching from a checkpoint with different B settings and assessing the impact on loss.
Recent work also defines Bent in terms of how 7j scales with B [31, 30]; unlike our work, these recent studies use a constant learning rate schedule and no weight decay.
We follow McCandlish et al. [11]'s definition of Ben (Definition 3.1). Given various theoretical assumptions, McCandlish et al. [11] derived a direct equivalence between Bett and what they call the grudient noise scule (GNS): the variation of the gradients between different training examples. However, they noted that the GNS "accurately predicts the largest usable batch size (at the order of magnitude level)," which is below the level of precision needed for large-scale training. Merrill et al. [64] recently found "the gradient noise scale underestimates the CBS [i.e., Bandl." This lack of precision may be why, in Kaplan et al's original scaling laws paper [4], they note that, "although the critical batch size roughly matches the gradient noise scale, we are using direct [empirical] measurement of Bent." " Our approach to measuring Beri (See. 3.2) similarly provides a direct empirical measurement, but one that can be efficiently computed with any learning rate schedule or optimizer.
D.3 Detailed comparison with Zhang et al. [12]
Here we provide further comparison with the concurrent work by Zhang et al. [12]. The primary point of distinction of our paper is that we conducted a large-scale empirical study into the scaling of AdamW's weight decay hyperparameter (including its scaling with B), ultimately deriving a precise power law for the optimal Adam W timescale in tikkessper-parameetee weight decay. Further, we also explored scaling of Bog in addition decay, further methodological differences in our main experiments include that we used longer context length (2048 vs. 512), a cleaner dataset (SlimPajama vs. C4), the μP parameterization, decaying LR schedule (more on this below). and that we tuned HPs most N, D. B (Table 3). while [12] performed a HP sweep for a 15 model, and re-used optimal values at other scales. We now focus on differences in estimating and measuring the scaling of Ben.
Zhang et al. [12] did not use to Beri-- Beyond use of weight
Estimating Berit for specific target loss Both our work and Zhang et al. [12] require measuring, for different batch sizes, how many training steps it takes to reach a particular target loss. Since the number of steps to reach that loss is not known priori, it is inherently difficult to study Beei when using a LR decay schedule, where you must specify the number of steps in advance. Using a constant LR (as was done in early work on Beit [II] simply does not result in competitive models [29]. Unfortunately, it is not feasible to search for the precise step count needed, i.e., by conducting full training runs with different schedules/step budgets.
Zhang al. [12] creatively solve this issue by conducting single training run at a constant LR, while using weight averaging to generate higher-qualin checkpoints for evaluation. With this approach, they still "need to frequently evaluate the model on a holdout evaluation set" [12].
Given LR decay, as opposed to weight averaging, remains the standard practice for current state- of-the-art LLMs, we independently developed a different approach. This led the novel method described in our paper. In contrast with [12], we do not need to frequently evaluate the model, as we instead fit a B-specific loss power law through a few validation loss values (Sec. 3.2). Indeed, our approach may improve the efficiency of [12]'s method, as it would obviate the cost of continuous validation, which concerned them (see their section "Evaluation data size and frequency").
Estimating the Berüt power law Collecting Ben data across multiple model scales and loss targets is expensive. Zhang et al. [12] establish Bent scaling in D through three targeted experiments:
measuring Bun while scaling N but leaving D fixed to 3.07B
measuring Bea while scaling D but leaving N fixed to 302M
measuring Bea while scaling both N and D proportionally (at 20 TPP)
19

---

Interestingly, Ben was found to only scale weakly in N, but scale similarly whenever D is scaled. They then fit power law to their data points for the -0rammeree model, obtaining the fit Ban 22.91 D0.47 (in tokens).
In comparison, took more brute-force approach, computing Ban across many different N and D values, and ultimately fitting our Beit power law across multiple different model sizes and TPP settings (Fig. right). Also, unlike [12]. we assessed the quality of fit via computation of both Ria values and parameter quantiles via bootstrapping (Sec. 3.3).
Recall also that [12] used different definition of critical batch size. Let denote their quantity Betanng They set Bxhang to be the B such that the data required to reach a loss target is 1.2 x D (i.e., 1.2x the data required with Bopt).
We can use Eq. (6) to align their fitted law with our own. By this equation, we have:
Bzhang
D = Dmin (1 +
Ben
D (1.2
Briting 0.2
Bent
+ Bent 5.ahaagg
Thus, to convert their coefficient to our scale, we multiply it by 5, and, dividing by the number of tokens in our sequences, obtain Bzhang = 0.055990..47 The Bahang coefficient (0.0559) is 19% larger than our own (0.0471), perhaps reflecting differences in training setup or data quality (and worth investigating further in future work). However, the exponents suggesting that both works are independently measuring the same fundamental scaling behavior.
quite similar (0.47 vs. 0.462),
We emphasize that Ban directly reflects the fundamental limit to data parallelism in training neural networks. Given the significant implications of Ber scaling in D rather than C or T, (including those discussed in Sec. 4), we note the scientific and practical value in having different approaches independently observe this same phenomenon.
D.4 Hyperparameter scaling with B
It has long been recognized that the optimal learning rate, Yopp, scales with B. with reports of both linear [65-68] and square-root scaling [69, 60, 70]. Recent work has found Nopt to decrease when B >> Ben [30, 311, which resonates with our own findings (Fig. 2, right). Since it is difficult to predict exactly how 77 will scale with B, studies of Beat have often done full HP sweeps at each B
The only work we are aware of that specifically recommends scaling weight decay with B is Loshchilov and Hutter [23], who suggest À x VB, though this rule is not evaluated systematically. It is also important to note that Loshchilov and Hutter [23] use the independent form of weight decay, where decay is applied independently of the learning rate 7). unlike common implementations such as Adam W in PyTorch [6]. In these more typical dependent implementations, weight decay is scaled by n, so any increase in with B (e.g., x B or BB) already increases the effective weight decay strength accordingly.
D.5 and effective learning rates
The concept of effective learning rates, influenced by weight decay, has been widely discussed [71- 78). In its simplest form, the effective intrinsic LR is simply 17,, but in these prior works, effective LRs typically measure functional updates relative to weight magnitude, which is particularly relevant for normalization-base networks, Comparison of the effects of À vs. 77 adjustments in the context of LR decay schedules was explored in [29].
The behavior of effective LRs (relative update sizes) over the course of training has been studied comprehensively by Kosson et al. [77], including comparing the effects of increasing 7/ vs. increasing X. This work shows that higher 77 values can cause large relative updates early in training, which can destabilize training or require longer warmups [58]. High v and low À can also lead larger weight norms [77, 78], which also has a destabilizing effect, particularly low- precision training. These
20

---

20TPP 80TPP
20TPP
109
100
80TPP
320TPP
320TPP
Optimal weight decay,
Optimal weight decay,
10-1
10-1
10a
10a
10a
Batch size,
Batch size, B
(a) I1 scale
(b) scale
20TPP
20TPP
80TPP
200TPP
Optimal weight decay,
Optimal weight decay.
10-1
10-1
10a
10
10a
10a Batch size, B
Batch size,
(c) 610M scale
(d) 1.7B scale
Figure 7: Optimal weight decay scaling with B: The optimal weight decay increases linearly small batch sizes- cuntll > Bent-
effects may explain why we were able to achieve higher effective LRs 1jÀ by tuning rather than tuning with B (Fig. 2, middle).
ad For given dataset size D, the r and the batch-normalized effective LR B are equivalent, and thus effective LRs and the AdamW timescale be viewed different perspectives the AdamW optimization process.
E
Scaling of r and A: additional details and results
E.1 À scaling with
Fig. 7 shows how optimal changes across B, for all of the model scales and TPP levels where we did hyperparameter sweeps. There is strong linear relationship between À and over the smaller batch sizes B < Bans with optimal À eventually plateauing (or decreasing). Note the standard use of 1= [3, 32-34] is only optimal specific B, and this changes with TPP.
E.2
Additional details on fitting
We now describe how we obtained the optimal т values at specific model scales and TPP ratios. Rather than taking empirical minimum loss, we fit a parabola to the (I, points in log space and took the analytic minimum of the parabola. If we have multiple loss values at the same (e.g., our data for single scale and TPP comprises multiple different batch sizes), we only kept the lowest loss points at each prior to parabola fitting. We used validation loss on the held-out validation set. For
21

---

Algorithm Generating the optimal 7 power law
Input: small batch size B, optimal per-N learning rates Initialize tau wew_scinnngg___aaee_ _ioinss = for N in model_scoee scales do for D in 10N, 80NN, 320N, do
Reset loss_pocinss = for A in lambdda_ hamadd_rruggee range do
Train LLM (N, D, B, A, n). get validation loss L'
B/
loss points [T)
end
arg min, (loss_ _poists)
tav_scalung_lag lattt_llll__fffnngaa potp:/poooottntTTT1. Topi))
end for end for
Fit m for Tape = cTPP" on tau taa_ hou_sscling law Seckiinggpoiiiss
our calculations, we input B in units of tokens (in contrast to the outpur of our reported Bopt and Bai scaling laws, which we report in units sequences, of 2048 tokens). Algorithm sketches the full procedure for generating the power law (Eq. (3)).
E.3
Relationship to prior power laws
E.3.1 Relationship to Nppt scaling laws in dataset size, D
Both Shen al. [16] and Bjorck al. [22] propose scaling laws for the optimal learning rate, Toppt as power law in the amount of data, D:
Too B Ggp D
We now discuss how this power law also follows from the power law Tops in TPP. By Eq. (3), we have:
Top (TPP)
C. TPPm
D
Cr
N
Substituting in definition of (Eq. (2)), and assuming A, B, and are fixed, this implies Nopt will scale as:
B
m
Cr
AD
Nm-
Nm+
+ 1/ppp B
(m++1)
A.
Cap
D
(11)
Nm,
where
Cyp
and
Mmp
(m, + 1)
A C
Eq. (11) is exactly the form of the power law used Shen et al. [16], and explains the results across batch sizes seen in Bjorck et al. [22], as discussed in Sec. 2.4.
X-cr and map = - (m,, + 1), we can use these formulas to compare our fit coefficients to those Shen et al. [16].
Comparison to fit in Power Scheduler [16] Given Cop = n
In Shen et al. [16], they find map - 0.51. In our case, m, - 0.520, and therefore maj -0.48, which is quite similar.
BBoorkk et al. [22] use fixed A=0.1, fixed batch size 0.5M tokens (for most their experiments), and fit scaling laws separately at different model sizes (except in their Section 4). Shen et al. [16] use μPP to adjust the LR for different model scales, so the derivation applies at any particular N; they do not report which optimizer used nor any of its settings.
22

---

Comparing our Cup to their Cyp (= 4.6) is a bit less straightforward. First of all, Shen et al. [16] inputs B sequences (of length 4096). We thus convert to the scale of our coefficient by dividing by their sequence length, obtaining Cyp = 0.0011. Secondly, the power law of Shen et al. [16] is actually for the base μPP learning rate fi while our derivation above assumes the adjusted (final) learning rate 7 (Sec. 2.1).
Let us first compare Cop coefficients at the proxy-model scale, i.e. where 7) = -j. If we were to use a 28M-parameter proxy model, and a default A=0.1 (and using our fit values of c = 1.084 and N Wy = 0.527), then, by Cap Aey our Cop would also equal 0.0011.
Now we consider how our coefficient varies when N scales. To convert our n scaling law into one for the base ij. we can instead use Syo = >-p-e-" where p = P/W, P is the width of the proxy model, and W the width of the target model. The width also affects the number of parameters, N, and hence the term N In Transformers, N scales roughly as N LW2, where L is the model depth and W is the model width. we round the fitted exponent to m, R -0.5, and substitute the value x 1/W into the denominator, we therefore have:
Nm,
Ape, CAD
N- 0.5 x
W
X (T.W2) -0 0.5 W
X L 0.5 (W2)
0.5 W
x L -0.
which is invariant to changes in W-ice., the μP adjustment cancels out the model scaling in width. So, if we only scale W, T scaling would stay in agreement with the Power Scheduler recipe, but we increase depth, scaling would decrease ") proportional to 1/VT. in manner that is not accounted for in Shen et al. [16].
The key point is that the scaling law used by Shen et al. [16] is valid, indeed, has similar fitted exponents, to what would be predicted by the Tope scaling law-but in specific context only (small models, models only scaling in width). Moreover, we have shown it may be less effective to adjust y in order to optimize 7 (as these approaches implicitly do); we obtained superior results by instead adjusting A. By considering n, X, and B holistically, our scaling laws are superset of these laws for Topt. as well as other laws that we discuss further presently.
E.3.2 Relationship to loopt scaling laws in model size, N
See. 2.2 gave our recipe for tuning hyperparameters. for arbitrary N, D, and B setting. Here we advocated sctting peak 7) to the μP yPP-adjusted learning rate (where the base learning rate comes from proxy-tuning). Rather than further adjusting this LR based on the dataset size or batch size, we argued for instead adjusting À so that r is tuned to its optimal value. Based on both theory, and our empirical findings comparing tuning 7) to tuning X, we believe that using μaP to scale Mop with model width is sufficient for well-tuned models. That is, the theoretical scaling law for 7up (in model width), given by jPP, is sufficient for good performance. We discuss this perspective further in this section, specifically how the μP scaling law can explain recent work in empirical 'log power laws.
As noted in Sec. 2.1, when using juP,, a base 1) is tuned on small proxy model, and then scaled depending on the width of the target model. Let W be the width of the target model, and let P be the width of the proxy model. suP prescribes scaling the optimal base learning rate, Noptt down to Topp = poopt where p = P/W. That is, 7opt = Poope 'W. As models grow in size, P and Topt do not change, Nop. will scale x 1/W. Dey et al. 179, Figure 2] show that, indeed, range of LLMs from the GPT, Llama, and DeepMind series are roughly following a scaling law where their chosen learning rate, 17 is following 7) x 1/W. In other words, if one were to build scaling law for Nop based on published LLM settings, it would roughly obey the μP theoretical scaling law.
Furthermore, we can develop a scaling law for Vlppt in model size, N, using jP, and show that it matches a recent empirical scaling law by Porian et al. [35]. The number of model parameters in any Transformer-basee LLM scales roughly in the depth, L, and width, W, N x LW?. If we
23

---

61640
60x06-
66x
66
3.x
10x
70x
20x 61
20x 06
1.x
16 06
-00
200
200
150
230
0
1000
2003
2000
4300
3003
10000
50000
3000
2000
2000
Weight upeane (1)
seepet-peann (1)
(a) 557 steps
(b) 5568 steps
55680 steps
Figure 8: J is invariant to steps, with learning rate decay schedule (11 IM scale, proxy-tuned peak 77 with linear decay-to-zero): Here we adjust weight decay, A, in order maintain 7 at constant value, decreasing À proportional to the increase in S. Regardless of the total number of optimization steps, we see that same 7 corresponds to the same shape of the distribution of weight update coefficients (i.e., the same shape over the data, regardless of how the data is discretized in training). For batch sizing, this means that we use a constant D but increase by K x (decreasing S by Kx),, we will incorporate information across the data similarly similarly-providder provided we use the same
assume that maintain fixed width-to-depe to-thh ratio, i.e., R W/L, L W/R, then have N W W 1/3 Now, since μP prescribe scaling Nopt W-1 then for fixed aspect ratio, Noppt & N-1/3
Taking a very different approach, Porian et al. [35] developed an empirical scaling law Toppe as function of the number of model parameters. At each model scale, they trained with variety of batch sizes and learning rates, and found the optimal settings of these hyperparameters. All models were trained to 20 TPP. They then fit power law through the optimal LR settings, and found that 'lopt 0 N- exactly as would be expected if one simply followed j.P.
As provides principled approach scaling hyperparameters, μP can adapt to scaling when aspect ratio is not fixed. We therefore advocate using uP to set 'lopt, rather than fitting special Toppt power laws. However, with regards our overall approach, it does μP theoretical scaling law or an empirical one. The key point is that these laws can be used to set particular model scale, while the law should further be used to À depending on the B D values.
actually matter whether one uses the
E.4 The EMA perspective and learning rate schedules
To understand how the EMA view applies with dynamic LR schedule, we follow the discussion of Bergsma et al. [29], who extended the formulation Wang and Aitchison [1]. [29] consider EMAs with time- varying smoothing, 0₃ [0, 1]. Letting an = 1 (i.e., 81 21), they express ye in terms of all inputs vx:
= 01
V2 (1 a2)a111 02
V1 Σ 1m
Il (1- 5m6+1
(12)
(IT) -3,) ar. In other words, or reflects the contribution of input x; to output yt at time t, such that ytt = Σl_ Cr Unlike a standard EMA with fixed smoothing parameter, in this extended EMA the coefficients need not decrease exponentially as í decreases. Indeed, any set of coefficients can be generated by some particular smoothing schedule.
The EMA coefficient on each input is denoted Cr where Cr21
terms of learning rate schedules for AdamW training, a: = 7/ becomes the smoothing parameter step / (cf. Sec. 2.1). The EMA itself, yt. is the model parameters. The EMA is over weight updates: a large coefficient or,; means the ith weight update contributes lot to the EMA step The EMA coefficients thus provide more granular view of the contribution timescale than alone.

---

We now study the question, how do the EMA coefficients change as the step count changes? We generated the Ct, coefficients for the linear decay-to-zero LR schedule, and plot these coefficients at the final step (i.e., showing the contribution of weight updates to the final parameters). We use the juP-tuned and adjusted peak 7,, for 111M models. The learning rate increases linearly to the peak for the first 10% of steps, then decreases from the peak to O for the remainder of steps. We simulated three cases: where we take 557 steps, where we take 5568 steps, and where we take 55680 steps (5568 steps would be 20 TPP for I1IM model using B=192). From the perspective of batch sizing, these different steps could be achieved by decreasing the batch size twice by 10x.
We adjusted À for each step count in order to obtain the same three specific values. In Fig. 8, we see that the same implies the same shape of coefficients across the steps, and hence the same contribution over (normalized) time. That is, weight updates from the same portion of the training data contribute equally to the final model parameters."
The key takeaway is that since r is independent of the number of steps, it theoretically provides a B-independent measure of the AdamW timescale over weight updates, regardless of learning rate schedule. However, this equivalence for different / breaks down when B > Berit and weight updates themselves no longer contain linearly Bx the information of single sample.
F Scaling of Bott and : additional details and results
F1
Derivation of extra data" Eq. (6)
Eq. (5) can be written as:
D Dmin
Smin
D
- Smin
(D Dmin) Smin)
D D min Smin
+ DS DD
SDmin
0
Given B D/S, we can solve for D.
substitute in S D/ get equation in single variable, from which
D
DD
B DSmin
= 0
B
2 DBSmin
DD min 0
+ D(D BSSin
Dmin) =
D
Dmin + BSmin
Given Bent = Dmin/ Smin. we can substitute Smin = Dmin/ Beit and obtain:
D Dmnn
D Dmin B
Berit
D = Dmin (1+
B
F.2
Estimating Beri
We first provide some learnings from developing the Buit estimation procedure.
First, regarding the functional form (D()) Fx + D D- we found that including the irreducible loss term Ex was important for obtaining good fits. Fxy conceptually represents the Bayes risk plus the minimum loss obtainable for model of size N (i.e., the first two terms of Eq. (8)). Second, only interpolated points were reliable; we only compute Buit for loss values where all points are between, or very near to, curve fitting points. Third, each power law should have at least 3 points for fitting, in order to capture the concavity of the scaling in D. Finally, we sample our B values logarithmically and, as in McCandlish al. [11], perform our fits to Eq, (5) in log space.
'Note we do not plot the initial coefficients cr. here, but they are equal across the scales for a given 7, unlike the non-initial coefficients, which reduce by 10x as the number of steps increases by 10x. So constant means both the contribution across the data, and the same bias (dependence initial weights).
25

---

3.4
B
3.4
B
3.2
192
1536
3.2
3072
3.0
6144
768
6144
Validation loss
3.0
Validation
2.8
2.8
2.6
2.6
2.4
2.4
2.2
2.2
Observed data
Observed data
2.0
2.0
10P
1040
10t1
10:2
109
10th
10
1012
(total training tokens)
(total training tokens)
(a) I1 scale
(b) 266M seale
3.4
B
3.4
3.2
8064
3.2
1008
2016
2016
Validation los
3.0
Validation los
3.0
2.8
2.8
2.6
2.6
2.4
2.4
2.2
2.2
Observed data
Observed data
2.0
2.0
109
1040
10"1
1012
109
10
10
1012
17 (total training tokens)
(total training tokens)
(c) 610M scale
(d) 1.7B scale
Figure 9: Scaling laws in D for computing Buiti Full set of B-specific power laws, for 111M to 1.7B scales, fitted after training models with different batch sizes, B, and dataset sizes, D, at each scale (empirical data from real training runs indicated by points). From these power laws, compute the amount of data needed to reach any target loss, as illustrated in main paper Fig. 3.
Fig. 9 illustrates all the fit scaling laws for the Ben experiments. Notice that beyond the fitting points, the curves may not predict behavior well. In particular, we would expect all curves to eventually converge as increases. Because loss targets beyond the fitting points are unreliable, we only compute Be at loss targets where all the data sizes can be estimated through interpolation.
Fig. 10 shows the specific fits of Eq. (5) particular loss targets. While Eq. (5) reflects the data trend well over the given B values, we consistently find that points with very small B do not approach Dmia- We discussed this observation further in Appendix B.
Finally, for clarity, we provide Algorithm 2, which gives the detailed approach to generating the Ben power law in procedural form.
F.3 Estimating Bert for the 3.3B model
To obtain an estimate of Ber for the 3.3B model (shown in Fig. right), it was not feasible apply our full Bent fitting procedure at this scale (i.e., fitting B-specific loss power-laws, etc.). Instead, we estimated Ban based on two B, Dmin pairs. That is, (based on an initial estimate Ben) we trained a 3.3B model to 23TPP with 6-2016 and got loss of 2. 261688, and a separate model to 30TPP with B=4032, obtaining a loss of 2. 1695. Given these losses are very close, these two models should have the same Be and thus same Dmin-
26

---

D. total number of examples plee sequences)
D. total number of example pe (sequences)
10a
1)
1)-
Sais
Dmin
Sass
1mm
Smin
Smin
Dmin
Dmán
88
B 1475.2
10a
88
B 2358.0
6164
3072
107
3072
1536
1530
192
768
192
10"
10
10
10a
S, total number of steps
S, total number of steps
(a) 11MM, loss=2.872
(b) 266M, loss=2.600
D. total number of examples (sequences)
D. total number of examples (sequences)
Sman
Smin
D Din
Dmin
83
4258.6
83
D 4111.9
10a
10a
8064
8064
A032
4032
2010
504
1008
10
105
10
105
S, total number of steps
S, total number of steps
610M scale, loss=2.437
1.7B scale, 1oss=2.295
Figure 10: Example fits of trade-off' Eq. (5) plotting Smin and D The empirical data has good fit with Eq. (5) across model scales and loss targets. These estimates of Berit are used in fitting the Bei--caaing power law (Fig. 1 right).
We solve for this Boit follows. Let B3 4032 and B1 2016. By Eq. (6):
D2 D min + B2/ Beit), and D Dmin (1 + Beit)
Letr D2/Di (i.e., 30/23 in this case). If we divide the above equations, and solve for Beris find:
B2 1g-BBi
Buit
r-1
Plugging in our values of r, BI, and B2, we obtain an estimated Be of 4610, corresponding to Dmia of approximately 16 TPP.
F44
Beit scaling in loss
Fig. 5 (middle) shows that Bunr is clearly not a power law in loss, as proposed in prior work [11, 4, 30]. However, if we only consider points with the same TPP, there does appear to be somewhat of power-law relationship. In fact, this is implied by Bui being a power law in D. along with the (standard) assumption that loss scales similarly in N and D.
27

---

Algorithm 2 Generating the Berit power law
Initialize berit coalinglaw_fittinne
= []
for N in model_ scales do
Fit B-specific (and N-specific) scaling laws, L (D): in batch _ssiesssss do
Reset scaling Baainng_www__r_ _ew___itteee
=
for D in 10.N, 20N, 80N, 320.N. do
Train LLM D, B3. get validation loss L' scaling_lawe htt___iitnggpiinnt (D (
end
Fit Ey, Dema. /3 for (D) = Ex + Dooms D on scaling_lave Battl__ittingooooe points
end for
Use Ln(D) estimate Bera various loss values:
for I. in lossT TrasTorgets) N] do
Reset tradeoff fitting points [] for B in batch sizes [V] do
B
Get DB = I.
Doas
T Ey
tradeoff filling points.dddl (Da, S=Pp/B)
end
Fit Dmin: Smn for Eq. (5) on tradeoff
Ben = Puinf Swin
berit_ vorit_scalingllrr low wittiffttnoootttttte (Diin, Beri))
end for
end for
Fit m for Bea = c(Dmin)" on berit_scaling Law fittingg_pints
Specifically, let F D/N be the fixed TPP ratio. Therefore, N D/F. Assuming loss follows Eq. (8), we have:
L(N, = E' + Neonst N ~ + Deonst D
D
= + Neonst
D Donn D
= E + Nomms
far D + D Deoms D-8
Now, if a = 3, as is commonly accepted [3, 9, 80, 35, 81], we have:
L(D) E + f + Doona)
E + Keanst Da
where Keonsi For + Deonst is constant. In other words, at fixed TPP, loss is power law in data. Given Ben is also fundamentally a power law in data, then by the transitivity of power law relationships, Bort also a power law in loss in this context. This relationship can also be derived by expressing the D in Eq. (7) as power law in Be and substituting into E + KD-a
F.5
Weight decay affects B scaling laws
Weight decay affects scaling of Bapt- Table 5 illustrates how fixing at different values alters the fitted Boge power law, Larger À systematically yields larger Bapt and poorer fit quality (lower It²). When À fixed, the batch size that minimizes loss is only a conditional optimum (we denote it as Bopqa) because it compensates for suboptimal timescales rather than representing the globally tuned Begt obtained when is optimized jointly.
The degraded R² values arise because fixed À forces B balance partially competing goals: (1) maintaining good T value, and (2) remaining below Be to avoid gradient redundancy. A simple power law cannot capture this coupled behavior.
From the relation = B/(nDD) and the empirical scaling Topt x (P/N)"" (Eq. (3)), one can derive that, for constant 7, X, and N, the batch size preserving Tops should scale as x Dm+1 (with + 1 sz 0.47 for our data). The exponents in Table 5, however, deviate from 0.47. For small D
28

---

Table 5: Different À settings systematically affect fitted power laws for Bopt and result poorer fit quality (lower R²). Fitted parameters change with À and consequently projected Bopt values (in sequences) for different token budgets D.
Weight decay
Scaling law
R²
D=le10
D=le11
D=1e12
0.4
Begi 0.000670.607
0.706
587
2377
9615
0.2
Bept = 0.0012D0.64.
0.926
323
1128
3937
0.1
Bopr 0.0123DU.sE
0.972
240
644
1729
0.05
B Popt 0.384
0.689
192
358
667
0.025
Bewi 10.37).120
0.161
163
215
284
Tuned
Bopr 0.0306200.883
0.984
207
500
1207
and large A. the B that preserves Tope lies near or above Beit. yielding worse loss due to gradient redundancy. This means o artificially lower for small D values. Since Bope is affected differently for different D, the scaling law slope is distorted (in this case, increased). Analogous issues disrupt x for small X. These distortions reduce R² and impair generalization to large-scale training. When À is tuned, this confound is removed, and the resulting Bopt law aligns cleanly with the expected D scaling.
Weight decay affects scaling of Berit- A similar confound arises for Beri- Fixing À causes deviations from Topt to mingle with true gradient-redundang effects. At 11 111M scale and a target loss of 3.03, e.g.. larger batches perform worse solely because drifts from its optimal value, reducing the fitted Bem from 867 (tuned λ) to 707 (fixed A=0.1). At other loss targets, Bem is less affected. Since estimated Bun affected differently at different scales, the slope of the Ber scaling law again artificially distorted. Ben will appear increase faster than D0.5, and projections to larger scales will be inaccurate. contrast, tuning isolates relation that generalizes across scales.
effects, yielding stable D0.5
29

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims in the abstract and introduction are supported by extensive empirical results, culminating in the scaling law fits in Fig. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are presented in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-ppecification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
30

---

Justification:
While mathematical derivations are included in the appendices in order to support the empirical findings, there are no new theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Ys]]
Justification:
In Appendix C, we disclose the information needed to reproduce the experimental results in the paper. Model validation losses are obtained through changes in configuration files (i.e., adjusting X, B, N and D). Datasets and tokenizers are publicly-availabl and we disclose our standard model architecture settings and optimization hyperparameters in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Model validation losses are all obtained through changes in configuration files (i.e., adjusting A, 17, B, N and D) rather than code. Datasets and tokenizers are publicly- availabee and we disclose our standard model architecture settings and optimization hyperparameters in Appendix C.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: httpp/////pcccpppliiccic guides/CodeSubmi. wwt.esammssssioosseoooiooooeeeee onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: : htppss//nipps /nips. "t/ppllc/guideeeee Seaidde//Cooeuubmisseeeee wtt.s/uwmmssssoooosoioooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix C, we disclose the information needed to reproduce the experimental results the paper. We disclose our standard model architecture and optimization hyperparame- ters in this section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report statistical fit for our scaling laws in Sec. 2.4 and Sec. 3.3: R² values, and fitted parameter percentiles from bootstrapping analysis (re-fitting the power law on 80% of points, 1000x, following Hoffmann al. [3]). Random seed variance of validation losses are also reported in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the enror bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Appendix C reports the hardware used and training time details. Additionally, given the extensive experimental details and standard LLM settings, other practitioners can easily derive compute requirements based on their own specific hardware setting.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (c.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttps:////errppeee http://s///aaiit....///////////a httppp/ccc///tttieeeiielell.aaae
Answer: [Yes]
Justification: This work conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix A discusses the broader impacts of our work.
Guidelines:
The answer NA means that there is no socictal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
No models or datasets are released with our work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite and use all training and evaluation datasets in accordance with their licenses.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from partieular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.co/o//..o. dattsttsssss has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and sereenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
35

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non- standard component of the core methods in this research? Note that the LLM used only for writing, editing, formatring purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration not required.
Answer: [NA]
Justification: The method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: futtps://neur: ips / 2025//022//2/// /LLM) for what should or should not be described.
36


Reviews:
[{"review_id": "PrT4oJZJSO", "reviewer": "Reviewer_rwA5", "summary": "This paper does a comprehensive empirical study over different hyperparameters (HPs) in AdamW. The authors provide scaling laws for weight decay as a function of parameter count, data count and batch size, using models parameterized in muP. The authors also propose a new method for measuring the critical batch size and the optimal batch size and show the dependence of these quantities on the total data, showing that they do not scale as a power law of the target loss or FLOPs.", "questions": "Is there a similar EMA view that could be applied for general optimizers (i.e. Muon, Shampoo etc) - particularly matrix-based optimizers?", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The main strength of this paper is in the thorough empirical evaluation and the obtained scaling laws as a guide for how practitioners should change hyperparameters when scaling up. Parameterizing the models in muP removes the confounding effects on the learning rate when scaling up N and the finding that scaling up weight decay is better than learning rate is very valuable. Highlighting the key findings and takeaways in each section leads to a clear exposition of the ideas.\n\nThe methodology for fitting B_opt and B_crit is also interesting. I believe that the limitation explained by the authors in section 3.2 regarding the learning rate scheduler could be mitigated by incorporating weight averaging and using a constant learning rate during training, then decaying at various time steps - similar to [1]. Do you believe this will have a major impact on the results?\n\nI do not find any major weaknesses in this paper.\n\n[1] Wen, Kaiyue, et al. \"Understanding warmup-stable-decay learning rates: A river valley loss landscape perspective.\" arXiv preprint arXiv:2410.05192 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you so much for your thoughtful review**, and for the positive comments about the paper's comprehensive empirical evaluation, clarity of exposition, and the significant value of the obtained scaling laws for practitioners.\n\n### Regarding: \"incorporating weight averaging and using a constant learning rate during training, then decaying at various time steps\"\n\nYou are absolutely right: our approach applies to any LR schedule, but requires training over different dataset sizes.  If one uses a constant schedule plus weight averaging, one could instead identify the training steps required to reach a target loss by continuously evaluating the averaged model during training.  Alternatively, one could use a warmup-stable-decay (WSD) schedule, with different decay points (arXiv:2404.06395,arXiv:2405.18392).\n\nThe weight averaging approach was previously used in Zhang et al. (arXiv:2410.21676).  As we note on lines 621-626 of the appendix, that approach still has the significant overhead of frequently evaluating the averaged models until one is found that obtains the target loss.  Our approach (fitting a scaling law through a selection of evaluated loss points) offers an orthogonal efficiency benefit, and could in fact be combined with weight-averaging (or WSD+decay) in order to reduce the number of evaluations.\n\nAs for how a different schedule might impact results: this is indeed an empirical question, but we do hypothesize that as long as some effective variance-reduction approach is used in optimization (e.g., weight averaging or LR decay) then Bcrit will maintain its fundamental scaling with dataset size.\n\nWe will add clarification about these points to Section 3.2 of the main paper, integrating and clarifying the points that are now in the appendix.\n\n### Regarding: \"Is there a similar EMA view that could be applied for general optimizers (i.e. Muon, Shampoo etc) - particularly matrix-based optimizers?\"\n\nYes, this is an active area of interest for us.  Many optimizers can indeed be interpreted in terms of an EMA timescale, although the exact derivation depends on the optimizer's structure:\n- optimizers such as Sophia (arxiv:2305.14342, Algorithm 3) use weight decay in a form that can be directly rewritten as an EMA, as was done with AdamW.\n- the standard way weight decay is used with Muon also admits an EMA interpretation.  E.g., in the recent MuonClip optimizer from Kimi (github.com: MoonshotAI/Kimi-K2/blob/main/tech_report.pdf), the update on Line 6 of Algorithm 1 can be directly revised into an EMA form.\n- the EMA perspective should also hold when EMA-compatible optimizers are applied in a different weight basis, e.g., as in SOAP (arxiv:2409.11321), where AdamW is applied in Shampoo’s eigenbasis (arxiv:1802.09568).\n- more generally, it is possible to analytically derive a composite timescale for optimizers that involve multiple momentum terms or weight averaging (as in AdEMAMix, arxiv:2409.03137), following the approach in our Appendix E.4.  The same principles apply but the derivation process necessarily depends on the specific optimizer.\n\nWe fully agree with your suggestion and intend to explicitly expand our discussion of these connections in the final paper. Moreover, experimenting with additional optimizers is an important next step we are actively pursuing.\n\nThank you again for your valuable feedback!"}, {"review_id": "yqqrCpnl1x", "reviewer": "Reviewer_4mAg", "summary": "This paper study scaling laws for many hyperparameters: formulas for how to scale hyperparameters as we scale model size N, dataset size D, and batch size B.", "questions": "The paper cites in [54] that a \"surge\" phenomenon has been reported in the Adam optimizer. Could this issue potentially undermine the validity of the conclusions presented in this work?", "limitations": "See Weakness 2 and 3.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The author conducted numerous experiments and summarized the scaling laws of parameters.\n2. AdamW and these hyperparameters are crucial for the pre-training of LLM.\n\n**Weaknesses**\n\n1. The authors only conducted experiments on dense LLMs. Nowadays, sparse activation LLMs are more popular. The authors can extend their experience to the MoE scenario.\n2. The author's experience focuses solely on the AdamW optimizer.\n3. The authors only conducted experiments on LLM pre-training tasks, which generally only require one epoch of training. It is currently unknown whether the authors' findings are applicable to other tasks that require multiple epochs of training.", "was_revised": false, "final_justification": "", "rebuttal": "**Thank you very much for your helpful comments**, and for generously noting the numerous experiments, and the importance of our work in hyperparameter-tuning for pre-training LLMs.\n\n### Regarding: the \"surge\" phenomenon with the Adam optimizer\n\nWe believe the surge phenomenon observed in Li et al (arxiv:2405.14578) [54] *supports* some of the empirical findings presented in our work.  We definitely should have discussed the connection to [54] further in the main body of the paper; we will address this in the camera-ready version.\n\nWe did note in the appendix: \"Recent work has found ηopt to decrease when B > Bcrit [54, 53], which resonates with our own findings (Figure 2, right).\"  However, it would be better to introduce this point when Figure 2 is first introduced, in the main body of the paper, and to discuss it in more detail.  The fact we observe: (1) a similar surge with optimal weight decay (as opposed to optimal η), and (2) a surge in the optimal η when using AdamW (rather than vanilla Adam as in [54]), suggests the surge phenomenon may be more general than initially presented by Li et al.  Further analysis is definitely warranted here.\n\nThank you for raising this point!\n\n### Regarding: \"The authors only conducted experiments on dense LLMs. Nowadays, sparse activation LLMs are more popular.\"\n\nWe agree MoEs are very important frontier-scale models. However, we intentionally chose dense LLMs for several reasons:\n\n- Dense models have fewer confounding factors (e.g., routing strategy, number of experts), making them simpler and clearer to study, and for other researchers to reproduce and build upon.\n\n- Dense models serve as a critical baseline for extending insights to MoEs in practice.  In our own studies, algorithmic innovations (including hyperparameter scaling strategies) are typically validated on dense models first.\n\n- Dense models remain widely deployed in both research and practical scenarios due to their ease of training, predictable hardware utilization, simpler inference pipelines, and suitability for fine-tuning.\n\nIn fact, in our ongoing experiments with MoEs, we find similar fundamental weight decay and batch-size scaling relationships.  However, achieving these results required additional methodological innovations (e.g., in parameterizations, routing, etc.) that go beyond the scope of the current paper.  Thus, while MoEs represent a valuable direction for future work, studying dense models here was a justified, intentional scope choice rather than a weakness.\n\n### Regarding: \"It is currently unknown whether the authors' findings are applicable to other tasks that require multiple epochs of training.\"\n\nYou're correct: our study specifically focuses on the practically important setting of single-epoch LLM pre-training. Prior work (Wang and Aitchison, ICML 2025) has indeed noted differences when using multi-epoch training, possibly due to data repetition. Reconciling these differences by isolating the effects of repetition versus scale is an interesting follow-up direction. We will explicitly clarify this scope limitation in the revised paper.\n\nThanks again for your insightful feedback!"}, {"review_id": "EGbPU1F4qC", "reviewer": "Reviewer_S6CK", "summary": "This work empirically studies hyper-parameters scaling of large langage models. The first part focuses on how to scale weight-decay and learning rates for AdamW when batch scales. They generalize a formula for a quantity $\\tau = B/(\\eta\\lambda D)$ that was formulated in a multi-epoch setting and shown to stay invariant for optimal training in that case. They show that this quantity remains approximately constant in optimal training in the one epoch training. Additionally, they show that the weight decay $\\lambda$ is \"easier\" to scale than the learning rate, in that tuning $\\lambda$ instead of $\\eta$ brings lower losses and that it scales better with batch. Finally, they show that when accounting for token-per-parameter (TPP) ratio, $\\tau$ is not constant and behaves as a power-law in TPP. The second part focuses on how do the optimal and critical batches scale. They define experimental set-ups to measure $B_{crit}$ and $B_{opt}$ and show that both of these evolve as power-laws in compute. Additionally, different TPP yield different power-laws for $B_{crit}$ and $B_{opt}$. Finally, they discuss implications of their finding on the impact of batch size on the optimal trade-off between compute and training time. In particular, they show that the usual $TPP=20$ tokens per parameters requires not too large batch for efficient training.", "questions": "- figure 2, right : It seems to me that there does not seem to be any trend in $\\tau$ unless I'm mistaken? Could you comment on that?\n- figure 2, left : the trend in $\\tau$ being constant mainly happens at small batch scale. But aren't we mainly interested in how this variable evolves for large batch scale? As soon as B=8064, there seem to be some large discrepancies (factor 10 variations). On the other hand, line 188, you underline that another measurement fails at small batch scale. Could you comment on which regime is of interest for batch when studying scaling laws and link it to the regimes you study?\n- Did you consider plotting figures 3 and 4 in a log-log axis to better see the power-law scaling as linear curves?\n- figure 5 right: I'm not sure to understant if the blue and green curves are fitted on the experiments and if yes, which data points were used and why did you not add the data points on the figures? Also I was wondering which TPP-ratio was used? It seems an important parameter from figure 5 left.", "limitations": "Yes, the authors discussed limitations in appendix B.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- This work presents a lot of practical results on scaling of hyper-parameters that could be very useful to practicioners.\n- Experiments are performed at reasonably large scales (up to 3B parameters models).\n- They generalize previous formulas such as from [1] to account for the TPP ratio and show that the optimal $\\tau$ depends on TPP.\n- The paper is well written with clear formulation of results in color boxes.\n\n**Weaknesses**\n- Different results in this work seem unrelated to each other, outside the fact that they all study hyper-parameters scaling. For example, the weight decay and learning rate scaling for AdamW in section 2 seems independent from the batch scaling results in sections 3, 4.\n- The empirical results do not always support the claims: for example, I was not fully convinced by the claim \"Sweeping $\\eta$ shows a trend towards an optimal $\\tau$\" in figure 2 by looking at the right figure.\n- I'm not sure why the authors rounded the exponent values in key takeways 1 and 2 (0.5 and 0.4). The results are slightly different? (line 110, 196) Especially line 196 the authors claim to be very close from a previous number by showing two digits precision but round it in the takeaway.", "was_revised": true, "final_justification": "I believe this work should be accepted.\n\nThe autors present various interesting empirical scaling results that could be useful to practicionners and contribute to shed some light in the broad landscape of scaling models.\n\nThe main weakness in my opinion was the absence of links between the sections in the paper on batch scaling and weight decay scaling but the authors presented new results in the rebuttal on the links between both.", "rebuttal": "**Thank you very much for the detailed review** and for highlighting the scale of the experiments, the quality of the writing, and the work being very useful for practitioners.\n\n### Regarding: \"the weight decay and learning rate scaling for AdamW in section 2 seems independent from the batch scaling results in sections 3, 4.\"\n\nYou raise a fair point regarding the perceived disconnect between Section 2 and Sections 3-4.  Prompted by your comment (and Reviewer CcAT), we have run additional experiments clearly demonstrating that improper weight-decay tuning significantly distorts batch-size scaling results (see our response to Reviewer CcaT for details).  Specifically, when weight decay isn't properly scaled, it systematically distorts the optimal and critical batch-size power laws, leading to inaccurate Pareto frontiers.  We propose adding a new subsection (Sec. 3.4: \"Weight decay affects batch size scaling accuracy\") to explicitly unify these findings.  This addition clearly ties together the paper's results and highlights the importance of our weight-decay scaling methodology.\n\n### Regarding: \"why the authors rounded the exponent values in key takeways 1 and 2\"\n\nThank you for pointing this out.  Similar to prior work (abs/2406.19146), we rounded the exponent values (e.g., to 0.5 and 0.4) intentionally for simplicity and readability, but we see this introduced confusion given the precision emphasized elsewhere in the paper.  To resolve this clearly, we'll adjust the key takeaways to report precise exponent values consistently throughout, ensuring alignment with the main text.\n\n### Regarding: \"there does not seem to be any trend in η [in Figure 2, right]\"\n\nYou are correct, this wasn't clear.  The plot only shows η implicitly, as we sweep η while holding B fixed in order to get the curves for each batch size.  The actual trend we highlight is the lower boundary of the convex hull over all the points in the plot (over all the curves).  This is a bowl with its minimum around τema=0.21.  The 5 smallest batch sizes achieve their minimum within 2x of this τema.  However, as B increases, the minimum drifts higher.  We attribute this to the fact that the best *timescale* is always around τema=0.21, but to achieve this optimal timescale when using a high B, you would need to use a very high η.  In practice, we can’t push the η this high - we observe a maximum η, above which training becomes unstable; above this value, loss spikes occur from which training does not recover.  We will clarify this in the revised paper.\n\n### Regarding: \"Which regime is of interest for batch size when studying scaling laws?\"\n\nThis is an important question that deserves clearer discussion in the paper.  We propose to add the following discussion under Finding 1 to clarify which batch size regimes are of primary interest when studying scaling laws:\n\n> As batch size increases, we find that τema_opt remains roughly constant up to a certain point, after which it begins to drift. The drift point corresponds to the critical batch size Bcrit, above which gradient information no longer scales linearly with batch size and diminishing returns set in (Section 3).\n\n> In general, LLMs typically train faster and utilize hardware better with larger B, but only up to Bcrit: beyond this point, much more data (and compute) is needed to obtain the same loss, without meaningfully reducing the total number of sequential training steps (the poor tradeoff for B > Bcrit is depicted in Figure 4).  Furthermore, Section 3 will show that there is also an \"optimal\" batch size, Bopt, below which loss is worse *and* utilization/parallelism suffer (because batches are small).  In practice, LLMs should therefore be trained in the regime Bopt ≤ B ≤ Bcrit.  Notably, this is precisely the range where we have shown weight decay to scale predictably with batch size; our findings therefore support the direct optimization of weight decay in the most practically relevant training regimes.\n\nWe also note that what counts as a \"large\" batch can vary by context. For example, optimizers designed for \"large-batch\" training, such as LAMB, are often evaluated at batch sizes still below Bcrit. See Section D.1 for further discussion; we found that analyzing LAMB's results through the through the lens of our scaling framework was very helpful for understanding the relevance of Bcrit in practice.\n\n### Regarding: \"plotting figures 3 and 4 in a log-log axis to better see the power-law scaling\"\n\nYes, thanks for this, but note that because the power laws used in Figures 3 and 4 have \"irreducible\" terms (e.g., the E term in $L(D) = E + D_c D^{-\\beta}$), they actually do not reduce to straight lines on a log-log scale, rather they have asymptotes (at the minimum achievable loss in Figure 3, and the minimum number of steps and tokens in Figure 4).  We'll clarify this briefly in the text.\n\n### Regarding clarity on Figure 5\n\nWe will definitely revise to make Figure 5 more clear in the paper.\n\n- We will clarify that the blue and green curves in Figure 5 (right) are based on the power law fits from Figure 1 (middle and right).  We'll add the exact fitted equations to the plot to make this clear, and we will also mark Bdeepseek = 0.292(C)^0.3271 (the fitted law from [7]).  We did not repeat the fitting points here because this plot serves to compare our correct batch-size scaling (in dataset size D) with the misleading DeepSeek scaling (in compute C, which introduces a false dependence on model size).\n\n- We will also clarify that Figure 5 (left) is exactly the same data as in Figure 1 (middle), while Figure 5 (middle) is the same data as in Figure 1 (right).  Scaling in D (Figure 1) is the fundamental scaling relationship: the optimal and critical batch sizes scale in the dataset size D regardless of TPP, model size, or loss.  It is only when using another (misleading) scaling factor such as C that TPP or model size appears to be important, as in these plots.\n\n- Finally, we will emphasize explicitly that despite relying on a misleading scaling variable (C), DeepSeek’s corresponding choices for batch-size happen to align with our recommended regime (Bopt ≤ B ≤ Bcrit, as noted above).\n\nYour detailed feedback has significantly improved the clarity and coherence of our results. Thank you again for your valuable insights and thorough review!"}, {"review_id": "2Sj9ovJqje", "reviewer": "Reviewer_CcAT", "summary": "This paper conducts a large-scale empirical study on the interaction between weight decay $\\lambda$, batch size $B$, token budget $D$, model size $N$, and learning rate $\\eta$. At a high-level, the paper conducts hyperparameter scaling law analysis focused on the interaction of $\\lambda$ with the other terms, encapsulated by the term $\\tau_{EMA}$. On a more granular level, the paper's primary contributions are:\n\n* The finding that the optimal value of $\\tau_{EMA}$ obeys a power law decrease (as opposed to holding constant, as suggested by [Wang & Aitchison, 2024]).\n* Methodology for estimating optimal and critical batch sizes, along with observed power law behavior of these parameters.\n* Usage of the hyperparameter rules and scaling laws above to investigate pareto frontiers of FLOPs versus wallclock trade-offs. Notably, under a simple wall-clock model, the authors show that over-trained small models often dominate larger, Chinchilla-optimal models.\n\nIn short, the paper does as the authors claim in the abstract and introduction - empirical scaling law analyses of hyperparameters, especially batch size $B$ and weight decay $\\lambda$.", "questions": "1. It is not immediately clear how the findings in Sections 3 & 4 are predicated on the weight decay analysis in Section 2. Do the authors have any sense (or data to back it up) that the findings on critical/optimal batch size and Pareto frontiers would change qualitatively had $\\lambda$ been set via a simpler rule (such as the $\\tau_{EMA} \\approx c$ heuristic from [Wang & Aitchison, 2024])?\n\n2. Would the Pareto frontiers change qualitatively when using a different mechanism to estimate critical batch size?\n\n3. What's going on in the last few points of Figure 2, middle, where the optimal values of $\\lambda$ and $\\eta$ drop with the largest batch size?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\nI will be quite clear here - I think this is a good paper that significantly advances our understanding of hyperparameters in transformer pre-training. The paper is generally well-written, rigorous, explicit, and produces useful insights and analyses.\n\n### Explicit methodology\n\nPerhaps my favorite part of the paper is how explicit the methodology is. Consider Section 4.2. Rather than just giving the Pareto frontiers with some cursory discussion about their conclusions, the paper goes into detail about how it computed the exact Pareto frontier, (L251-256), with explicit references to what equations they used to fit what parameters. This makes the paper much easier to reproduce in other settings and on other models, and I think sets a great example to the scaling law community.\n\n### Careful methodology\n\nIn addition to being explicit, the authors are clearly careful. This makes me trust their findings so much more. As a useful example (though this trend occurs throughout the paper) consider the estimation of critical batch size in Section 3.2. The authors identify a thorny issue (the fact that the number of steps, integral to the learning rate schedule, is unknown). They propose a specific method to remedy it, detail it explicitly, and discuss in even more depth in Appendix F.2. The authors have clearly thought about every aspect of their methodology, and I think the reader and community benefits from this kind of work being accepted.\n\n### Breadth and utility of results\n\nI think an undeniable strength of the paper is that it has a lot of useful results. Weight decay, batch size, and even what model size to train are all individually important topics. Including them all in the same paper is ambitious (though not without drawbacks, see below) but I think that the authors basically succeed on all fronts. It would be easy for practitioners to immediately incorporate this into their training regiments, and improve training on multiple fronts. The paper is also extremely useful in its breadth of connections to related work - someone relatively new to the area would get a pretty good understanding of the landscape of work just by reading this single paper.\n\n## Weaknesses\n\nI will go into a bit of detail here, because I want to emphasize ways the paper could potentially be improved. I want to be clear though, these are constructive suggestions, and I still wholeheartedly recommend acceptance.\n\n### Breadth versus depth\n\nMy biggest critique of the paper is that the three main topics it covers (weight decay, batch size, pareto frontiers) are related, but not wholly so. This makes the paper a bit less cohesive. While I think the paper should be accepted as is, it makes me wonder if pulling the paper apart would have been at all beneficial.\n\nTo take this further, it means that each individual thrust has some empirical aspects and ablations that could have been expanded upon had it been the singular thrust of a paper. For example, one of the contributions in Section 3 is an estimate of the critical batch size. However, [McCandlish et al., 2018] provide a method for estimating this quantity as well. If Section 3 were the subject of a full paper, I would absolutely ask to see comparisons between the critical batch size estimates in this work, and the critical batch size estimates from [McCandlish et al., 2018]. In this paper, I don't think it's critical (given the breadth) but it's a comparison point that is missing.\n\nSimilarly, had section 4 been the primary focus, I would ask to see how the Pareto frontier changes as the training time model changes. This is done to some extent (Figure 6, left versus right) but there's a wide spectrum of possible models that would be fascinating to explore here. To some degree, this is a good criticism to have - it means that the paper is doing interesting things, and I look forward to follow-up work. But I do want to mention it as a drawback of fitting these three topics under a single paper's umbrella.\n\n### Ablations and their role\n\nRelated to the above is the fact that by bundling these all together, it means that the paper is often combining multiple things into a single experiment, making it so that the findings are predicated upon one another.\n\nAs a concrete example, consider the critical batch size analysis in Section 3. It is unclear whether the optimal setting of $\\lambda$ is actually required for any of the subsequent findings. Would the use of constant $\\tau_{EMA}$ (as opposed to the power law decaying one in equation (4)) qualitatively change the findings in this section? It is essentially for the reader to know. This also rears its head in Section 4, which relies on the critical batch size estimates from this work. Do the pareto frontiers change significantly with different methodology for estimating the critical batch size?\n\nI will note that these are not dealbreakers in any way shape or form. They can absolutely be investigated in follow-up work. But just as above, I want to note that there are trade-offs of the paper's design.\n\n## Minor comments\n\n* $\\tau_{EMA}$ is undefined by the time Figure 1 shows up. More generally, I found Figure 1 to be much less useful as a synopsis of the work than the bullet points on L51-L55.\n* In Figure 2, middle - you state that $\\lambda$ scales linearly with $B$. While this is true for most $B$, this is clearly untrue for larger $B$. Should this be interpreted as $\\lambda$ having a predictable relationship up to something like the critical batch size, or is something more complex going on?\n* While I generally liked the colored text boxes with key takeaways, I found key takeaway 3 (L290) to be out of place and inappropriate for an academic setting. It's just a strange thing to advertise your methodology after you get done describing it. Moreover, it discusses the contributions in a pretty vague way that doesn't actually interact well with any of the specifics preceding it. I would strongly advocate removing it altogether.\n* Figure 5, right, is difficult to match lines with the legend (e.g. I can barely tell which is 100M and which is 1B).\n* Algorithm 1, in the appendix, mixes greek letters with pseudo-code names in a way that's very hard to read (e.g. $\\tau_{EMA}$*_scaling_law_fitting_points.add*).\n* The paper uses inconsistent \\citet, \\citep and the like. For example, L79 and L112 both use \"Wang and Aitchison\" in the sentence, but have different citation styles. This occurs elsewhere in the paper.", "was_revised": true, "final_justification": "I stand by every word in my review, and the other reviewer feedback seems to basically mirror my own. The author feedback was basically what I expected, so I have opted to keep my score as is.", "rebuttal": "**Thank you very much for your thoughtful and encouraging review.** We deeply appreciate your recognition of the paper's experimental rigor, writing, usefulness, and breadth of connections to related work, along with your constructive suggestions, which we address in detail below.\n\n### Regarding \"how the findings in Sections 3 & 4 are predicated on the weight decay analysis in Section 2\"\n\nWe did observe challenges in measuring Bopt and Bcrit prior to developing our weight decay scaling methodology, but we did not investigate this systematically until reading your points here.  Analyzing our data now, we find a very compelling story regarding the importance of tuning weight decay.\n\nPrior work uses a fixed weight decay when measuring batch size scaling laws (e.g., λ=0.1 in arxiv:2401.02954, λ=1e-4 in arxiv:2406.19146).  Using a fixed weight decay will not only result in worse loss (Section 2), but will affect the accuracy of the scaling laws.  We propose to add a new Section 3.4 to the paper titled, \"Weight decay affects batch size scaling accuracy.\"  We will demonstrate this finding in two subsections: \"Weight decay affects scaling of Bopt\" and \"Weight decay affects scaling of Bcrit\".\n\n#### \"Weight decay affects scaling of Bopt\"\n\nIn terms of fitting a Bopt power law, rather than tuning λ at each batch size, we tested setting λ to different fixed values.  We find that different weight decay settings *systematically affect the fitted power laws for optimal batch size*.  In the following table, we show the fitted parameters for different weight decay settings, as well as the projected Bopt (in sequences) for different token budgets:\n\n| Weight decay | Scaling law | D=1e10 | D=1e11 | D=1e12 |\n|:----:|:-----------:|:-----:|:-----:|:-----:|\n|  0.4  | Bopt = 0.0006 D^0.607 | 587 | 2377 | 9615 |\n|  0.2  | Bopt = 0.0012 D^0.543 | 323 | 1128 | 3937 |\n|  0.1  | Bopt = 0.0123 D^0.429 | 240 | 644 | 1729 |\n|  0.05 |  Bopt = 0.384 D^0.270 | 192 | 358 | 667 |\n| 0.025 |  Bopt = 10.3 D^0.120  | 163 | 215 | 284 |\n| **Tuned** | **Bopt = 0.0306 D^0.383** | **207** | **500** | **1207** |\n\n\"Optimal\" batch size increases with λ because the EMA timescale $\\frac{B}{\\eta\\lambda D}$ is the fundamental scaling variable: if λ is larger, then B must increase to maintain the optimal ratio.  If the timescale is instead maintained across batch sizes via tuning λ (final row), the optimal batch size follows more accurate scaling.\n- Given the timescale $\\frac{B}{\\eta\\lambda D}$ scales with $\\mathrm{TPP}^m$ (Equations 2 and 3 in paper), it can be shown that when η, λ, and model size are fixed, \"optimal\" B scales as $D^{m+1}$ (where $m+1 \\approx 0.473$ for our data).  I.e., this is how B will scale in order to maintain the timescale.\n- The exponent deviates from this value in the fits above because, for small D and high λ, the \"optimal\" B is near or above Bcrit, and therefore obtain worse loss, so \"Bopt\" is artificially lower for small D values.  Since \"Bopt\" is affected differently for different D, the scaling law slope is distorted (in this case, increased).  Analogous issues disrupt \"Bopt\" for small λ.\n- Measures of scaling law accuracy are consequently worse for fixed-λ fits, especially at larger scales.\n\nIn summary, without tuning λ, the scaling law fit for Bopt is affected by small-scale effects, and will therefore not generalize to large-scale training, *even if the fixed weight decay is maintained at those larger scales.*\n\n#### \"Weight decay affects scaling of Bcrit\"\n\nSimilarly, not tuning weight decay leads to inaccurate estimates of Bcrit.  For example, at 111M scale and a loss target of 3.03, larger batch sizes obtain worse loss purely because they cause τema to deviate from the optimal timescale.  This makes Bcrit appear lower than it actually is (i.e., from true gradient redundancy): from 867 when λ is tuned to 707 when we restrict to λ=0.1. At other loss targets, Bcrit is less affected.  Since the estimated Bcrit is affected differently at different scales, the slope of the Bcrit scaling law is again artificially distorted.  Bcrit will appear to increase faster than $D^{0.5}$, and projections to larger scales will be inaccurate.\n\n### Regarding \"Would the Pareto frontiers change qualitatively when using a different mechanism to estimate critical batch size?\"\n\nAs we discuss below, recent work has found other mechanisms (such as GNS) may underestimate Bcrit.  Similarly, as noted above, if λ is not tuned, the slope of the Bcrit power law will change.  In either case, such changes will affect the Pareto frontier in Section 4.  To illustrate this, we will add an experiment to the paper where we vary the exponent in the Bcrit power laws and plot the changes to the Pareto frontier.  We find that as the exponent increases, and Bcrit increases, models of higher TPP move to the frontier.  E.g., at very low Bcrit sizes, 20TPP models dominate the frontier, while at the highest Bcrit, 200TPP and 600TPP models are Pareto optimal.  In this way, inaccurate estimates of Bcrit lead to incorrect Pareto frontiers; large-scale training decisions based on these frontiers could lead to suboptimal training durations and compute costs.\n\nTogether with the points above, we can now link suboptimal λ tuning (Section 2), to poor batch size scaling laws (Section 3), to incorrect conclusions regarding time/compute Pareto optimality (Section 4).  These additions thus help unify the paper's contributions across sections.  Thanks again for your very helpful contribution here!\n\n### Regarding \"comparisons between the critical batch size estimates in this work, and the critical batch size estimates from [McCandlish et al., 2018]\"\n\nThanks also for raising this; we definitely should have included a discussion of the gradient noise scale (GNS) from McCandlish et al., 2018 (arxiv:1812.06162), and will remedy this in the camera version of the paper.  A recent paper by Merrill et al. (arxiv:2505.23971) compared the GNS estimate of Bcrit to empirical measurements based on sweeping batch size and found \"the gradient noise scale underestimates the CBS.\"  In fact, the original McCandlish et al paper only noted that the GNS \"accurately predicts the largest usable batch size (at the order of magnitude level),\" which is far below the level of precision needed for large-scale training.  This lack of precision may be why, in Kaplan et al's original scaling laws paper (arxiv:2001.08361), they note that, \"although the critical batch size roughly matches the gradient noise scale, we are using a direct [empirical] measurement of Bcrit.\"  Our approach to measuring Bcrit provides an efficient empirical approach that can be used with any learning rate schedule or optimizer.\n\nWe have also now developed instrumentation to measure GNS during training, so direct comparison between GNS estimates and empirical measurements in the context of modern LLMs is something we can also pursue in advance of the camera-ready deadline.\n\n### Regarding Section 4 and other models of training time\n\nExploring more realistic Transformer time models would indeed be valuable. Figure 6 (right) was interesting as the extreme version of model parallelism, but this is not realistic for the reasons mentioned on lines 285-286 (Transformer's inherent sequential nature).  A more realistic model of Transformer training could still assume perfect parallelism within a layer, but account for the \"critical path\" of sequential operations across the forward and backward passes.  Including Pareto frontiers for different depths under such a model would be a great addition to the paper - thanks a lot for the encouragement here!\n\n### Regarding \"what's going on ... where the optimal values of λ and η drop with the largest batch size?\"\n\nYes, we do interpret λopt as having a predictable relationship with B up until the critical batch size, at which point gradient information stops scaling linearly with B (due to redundancy), and the timescale over data breaks down.  Fortunately, the predictable scaling overlaps with the batch size \"regime of interest\" in practice: Bopt ≤ B ≤ Bcrit - please see our response to Reviewer S6CK for more details.\n\nRegarding why λopt and ηopt drop with the largest batch size: this is also a good point, and we should have discussed this more in this section.  As noted in our response to Reviewer 4mAg, prior work by Li et al (arxiv:2405.14578) (for vanilla Adam) showed that a kind of \"surge phenomenon\" happens with the optimal learning rate, where it first increase with batch size up to Bcrit and then decreases when B > Bcrit.  It's definitely interesting that we observed this with ηopt despite using AdamW rather than Adam.\n\nYet our observations of this phenomenon for λopt are a bit mixed: while it definitely levels off around Bcrit, we don't observe consistent drops (see appendix Figure 7).  Note the prior work investigated extremely large batch sizes, even up to 10x or 100x Bcrit, which was not practical at the scales that we trained.  But this could be pursued further in follow-up work.\n\n### Regarding the \"Minor comments\":\n\nRegarding your suggestions for Figure 1, Key takeaway 3, Figure 5, Algorithm 1, and use of `\\citet` vs. `\\citep`:\n- Thanks for these careful observations. We'll make all suggested corrections and clarify figures, text boxes, citations, and algorithms accordingly."}, {"review_id": "Flv4LiZYZq", "reviewer": "Reviewer_f1ZL", "summary": "This paper explores fundamental scaling laws for hyperparameters during large language model (LLM) pre-training, specifically focusing on two critical aspects: (1) optimal weight decay ($\\lambda_{opt}$) and (2) optimal and critical batch sizes ($B_{opt}$, $B_{crit}$). The authors present extensive empirical studies conducted across multiple model sizes, dataset sizes, and batch sizes, leading to clear, well-supported scaling laws.", "questions": "1. Have you explored or considered how changes in numerical precision (e.g., training in mixed precision or using quantized formats) could affect your proposed scaling laws and recommendations? If so, could you briefly summarize your findings? If not, could you discuss how you expect reduced precision might influence batch size and weight decay scaling?", "limitations": "1. **Generalizability of Method:**\n    - *The proposed method for estimating $ B_{crit} $ is claimed to be suitable for arbitrary optimizers and learning rate schedules. Have the authors empirically validated or tested this methodology with optimizers other than AdamW (e.g., LAMB, Shampoo) or with alternative LR schedules (e.g., cosine decay, constant LR)? If not, what would be the expected challenges or limitations?*\n\n2. **Practical Systems Implications:**\n    - *Given that real-world model training is often heavily constrained by hardware limitations and communication overhead, how do the authors suggest practitioners incorporate these practical constraints when applying the proposed scaling laws and recommendations for optimal batch size?*", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Strong Empirical Foundation:**  \n   The authors conducted extensive experimentation (around 400 models trained), covering a wide range of model sizes, batch sizes, and data scales. This experimental rigor strongly supports the reliability and generalizability of their findings.\n\n2. **Novelty and Clarity of Contributions:**  \n   Introducing a clear, practical method for estimating $B_{crit}$—that is both efficient and generalizable—is a significant methodological contribution. The clear articulation of how to scale $\\lambda_{opt}$ through the AdamW timescale $\\tau_{EMA}$) is also valuable and original.\n\n\n5. **Insightful Analysis and Theoretical Connections:**  \n   The authors provide intuitive explanations for observed phenomena, such as the EMA view of AdamW and its implications, and clearly demonstrate consistency with theoretical frameworks (e.g., maximal update parameterization (µP)).\n\n---\n### Weaknesses \n\n1. **Limited Exploration of Optimizers and LR Schedules:**  \n   The main experiments focus exclusively on AdamW with a linear LR schedule. While the authors claim general applicability, additional validation with other optimizers (e.g., Shampoo, LAMB, Muon) and LR schedules (cosine decay, constant LR with averaging) would strengthen generalizability claims.\n\n2. **Small Batch Behavior:**  \n   The observation that smaller batch sizes do not asymptotically approach $D_{min}$ as predicted theoretically is intriguing and warrants deeper exploration. Currently, explanations are only speculative.\n\n3. **Practical Systems Considerations:**  \n   Real-world training is constrained by hardware limitations (e.g., bandwidth, memory, synchronization overheads). The paper does not significantly explore or discuss these practical system-level constraints and their impacts on batch size recommendations.\n\n4. **Dataset and Architecture Limitations:**  \n   Experiments were conducted on just one dataset (SlimPajama) and one specific GPT-style architecture. While the authors provide a good range of sizes and scales, more diverse datasets, architectures, and domains might be needed to fully validate the universal applicability of these scaling laws.", "was_revised": true, "final_justification": "0. Sequence length is also quite important for LM pre-training, did you use a constant seq_len for all settings? how to ablate the impact of seq_len? \n\n1. Does the proposed scaling law generalize to other optimizers like Muon?\n\n2. What is the impact of different learning rate schedules beyond cosine decay with warmup? Have you ablated different warmup durations?\n\n3. How does AdamW's epsilon parameter affect performance across different model sizes? This hyperparameter often requires adjustment as models scale.\n\nI will be happy to increase my rate if the authors can address this.", "rebuttal": "**Thank you for your helpful review.**  Your questions and comments will significantly improve our paper.  We particularly appreciate your recognition of the paper's strong empirical foundation and clarity of exposition, and your kind words about the significance, value, and originality of the methodological contributions.\n\n### Regarding:  \"Have you explored or considered how changes in numerical precision ... could affect your proposed scaling laws and recommendations?\"\n\nThis is a very interesting and timely question: Indeed, recent work (arxiv:2411.04330) showed that, in terms of scaling laws, lower precision reduces the model's \"effective parameter count.\" Extending this idea, we hypothesize that:\n1. There would be no impact on the scaling of Bopt or Bcrit, since these are independent of model size.\n2. Lower precision would increase the \"effective\" tokens-per-parameter ratio (TPP) (via fewer effective parameters), thereby altering the optimal EMA timescale (Figure 1, left) and consequently the scaling of optimal weight decay.\nThis merits explicit discussion in the paper and we will add it in the camera-ready version.\n\n### Regarding: \"The proposed method for estimating Bcrit is claimed to be suitable for arbitrary optimizers and learning rate schedules. Have the authors empirically validated this?\"\n\nWe agree that clarification is important here.  Our method in Section 3.2 is explicitly *designed* to be optimizer/schedule agnostic.\n- The B-specific power laws for *loss* (Figure 3) allow us to estimate Bcrit for a specific target loss without actually training to that loss.\n- The procedure applies directly to arbitrary schedules and optimizers as it fundamentally doesn't depend on multiple validation losses from a single training run (i.e., as when using a constant LR schedule).\n\nThus, by design, the procedure itself does not strictly require empirical validation. However, you're correct that we have not yet applied the procedure beyond AdamW and linear decay. We acknowledge this limitation explicitly (Appendix B), and agree that exploring other optimizers is an important next step for our research.\n\n### Regarding: \"how do the authors suggest practitioners incorporate [hardware limitations and communication overhead] when applying the proposed scaling laws?\"\n\nYou raise a good point about practical system constraints.  While our paper emphasizes algorithmic scaling, we agree a more detailed discussion is beneficial.  In the camera-ready version, we'll clarify that our scaling laws explicitly define a practically relevant regime of training batch sizes (Bopt ≤ B ≤ Bcrit) (see our response to Reviewer SCxz for further details). Practitioners can leverage this identified regime alongside system-specific profiling (e.g., evaluating utilization at various batch sizes) to select optimal settings balancing algorithmic and hardware constraints.\n\nMoreover, we acknowledge that hardware limitations also impact the training time models in Section 4. Following Reviewer CcaT's suggestion, we are now exploring alternative, system-informed models of training time, and we plan to expand our discussion accordingly.\n\n### Regarding the rating of the paper\n\nGiven your many positive comments on the paper's empirical rigor, clarity, and methodological value, and the strong positive consensus among other reviewers, we respectfully ask if you'd reconsider your \"Reject\" rating. If our responses here clarify your concerns and our proposed improvements sufficiently address your highlighted limitations, revisiting your score could significantly impact both paper acceptance and potential spotlight considerations.\n\nWe would be grateful for any further concerns or suggestions you might have. Thank you again for your thoughtful engagement and valuable insights."}]

Author rebuttal:

Meta-review: The paper significantly advances the understanding of hyperparameters in transformer pretraining. It has explicit and careful methodology, particularly in computing Pareto frontiers and estimating critical batch size, both ensuring reproducibility and making it a valuable resource. The results are broad and highly useful for practitioners (and the NeurIPS community more generally), offering practical insights on scaling weight decay, batch size, and model size to improve training. The work also provides a comprehensive overview of related research.

A key achievement is the generalization of existing formulas to incorporate the token-per-parameter (TPP) ratio, demonstrating that optimal values are TPP-dependent. The paper is well-written, with clear explanations and effective highlighting of key findings. Experiments were conducted at reasonably large scales, involving models up to 3 billion parameters, which enhances the credibility of the findings. A particularly valuable insight is the observation that scaling up weight decay is more beneficial than scaling the learning rate. The proposed method for measuring optimal and critical batch sizes is also noteworthy.

The reviewers have highlighted some important minor directions of improvement, and the authors have suggested many concrete changes, including multiple new appendices. I’m sure it goes without saying, but please implement these changes for the camera ready version.
Decision: Accept (poster)