Resolving Discrepancies in Compute-Optimal Scaling of Language Models
Authors: Tomer Porian
Mitchell Wortsman
Jenia Jitsev
Ludwig Schmidt
Yair Carmon
Venue: NeurIPS 2024 spotlight (2024)
Keywords: LLMs
scaling laws
Chinchilla
copmute-optimal
power laws
Primary area: optimization_for_deep_networks
TL;DR: We explain how Kaplan et al. got to 0.88

Abstract: Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $\beta_2$ parameter is essential at lower batch sizes.

Resolving Discrepancies in Compute-Optimal Scaling of Language Models
Tomer Porian" Mitchell Wortsman
Jenia Jitsev! Ludwig Schmidt
Yair Carmon"
Abstract
Kaplan al. [30] and Hoffmann et al. [25] developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan al. scaling law on two datascts "OpenWebTere and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors cor- rected, we obtain excellent agreement with the Hoffimann et al. (i.e.. "Chinchilla") scaling law. Counter to hypothesis implied in Hoffmann et al. [25], find that careful learning rate decay is not essential for the validity of their scaling law. As secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW (3 parameter is essential at lower batch sizes.
Introduction
We consider the problem of compute-optimal language model training: given compute budget C, we wish to predict how to best allocate across model size (in parameters) and dataset sine tokens). With pretraining budgets ever-increasing, compute- compute-optimal scaling a question of paramount importance. In their seminal work, Kaplan et al. [30] proposed a scaling law predicting that the optimal ratio of tokens parameters decays power of C. This scaling law
influential in determining the size of GPT-3 and several subsequent models [12, 51, 43, 32, 47, 62, 52). However, Hoffmann al. [25] challenged its validity, arguing instead that the optimal token-to- -o-rammterr ratio should be approximately independent of C, and that contemporary models had too many parameters relative to their number of training tokens. Based on this prediction, they trained 67B parameters model called Chinchilla and which outperformed larger models with a similar compute budget.
While Hoffmann al. [25] and subsequent work [55, 56, 17, 26, 15] established that following the Hoffmann et al. scaling law leads to better performance than Kaplan et al. scaling, it is still important to understand why the two works arrived different conclusions. Is the difference due to architecture, training setup, pretraining data, results analysis, or perhaps something else entirely? The answer could teach important lessons how correctly predict and perform model scaling.
Hoffmann et [25] hypothesize that the scaling law discrepancy is due to Kaplan et al. [30] not tailoring the learning rate schedule for each token budget separately. While they demonstrate that mismatched learning rate decay results in a higher loss, they do not show it leads to a different compute-optimal scaling law, We further discuss Hoffmann et al. [25]'s hypothesis in Appendix A. To the best of our knowledge, this hypothesis the only explanation offered in the literature so far.
Our contribution. In this work, uncover three factors contributing to the discrepancy, and disprove Hoffman et al.'s hypothesis about the role of learning rate decay; Figure illustrates our main
'Tel Aviv University; correspondence to temerpor@gmail com and ycarmon@tauex. tau. ac, il. t University of Washington.
Jülich Supercomputing Centre (JSC) and LAION.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reproducing Kaplan et al.
Counting last layer FLOPs
Correcting warmup
0.83 10 (.85)
0.706 10.00 0.8.0.79
0.002 (0.30.0000000 0.0
[*(Cc) (IT.41)
[C(Cp) 787B (6100, S1GB)
(Cc) 292B (SemB, st5a)
10a
(-(C)
100
a
C
10P
10
101*
10
10
Compute C (FLOPs)
Optimizer tuning (no decay)
Cosine decay (no tuning)
0.497 (0.c0, 0.50)
0.571 (0.36, 0.08)
N*(Ce) 77B (oon, st0))
(C() 1838 (1001 2401
95% confidence region
Power law
10
(C)
Hoffmann law
Kaplan law
1
Observations
10:
d
1077
2019
1017
10
Compute C [FLOPs]
Compute C (FLOPs)
By analyzing over 900 training runs, we uncover the factors leading to the discrepency between the scaling laws of Kaplan et al. (panel a) and Hoffmann al. (panel e). Each panel shows observations of the optimal model size N* as function of the compute budget C, as well as power law fits of the form (C)) x C Labels show point estimates and 95% confidence intervals for and for the optimal model at Cc = 5.88e23, the compute budget used for training Chinchilla.
Figure I:
results. We begin by reproducing the Kaplan et al. scaling law in Llama-derived pretraining setup using the OpenLM library [21] and the RefinedWeb dataset [41] (Figure la). Our first observation that accounting for the computational cost of the decoding layer (as done in Hoffmann et al. [25] but not Kaplan et al. [30]) shifts compute-optimal scaling toward more constant rotra-to-parrmeter ratio (Figure lb). Second, we note that the constant-length warmup period of Kaplan et al. [30] is too long for smaller models, inflating the optimal number of tokens at lower compute budgets; scaling the warmup period with the model size further shifts the scaling in the Hoffmann et al. direction (Figure Ic). Next, we match the learning rate decay to the token budget of each configuration we test (as Hoffmann al. [25] conjecture to be essential) but observe little effect the compute- optimal scaling law (Figure Id). Finally, we set the learning rate, batch size, and the Adam W 83 parameters individually for each model size, leading to compute-optimal scaling that agrees closely with Hoffmann et al. (Figure le). Notably, the latter configuration uses a constant learning rate schedule, showing that learning rate decay is not essential for the Hoffmann et al. scaling law to emerge. We repeat our experiment on the WebTeee2xxx2 dataset [18], observing similar results despite performing hyperparameter tuning only on RefinedWeb.
We complement our main results with the following analyses:
1. In the last phase of experiments (Figure 1e) we choose different hyperparameters for each model size. To do so, we conduct hyperparameter sweep for small-scale models and use the results fit power laws for the optimal batch size and learning rate as function of model parameters. This approach is inspired by DeepSeek [15], and our hyperparameter scaling laws roughly agree. However, we observe that setting the AdamW Bg parameter be 0.95 suboptimal at smaller batch sizes (128 and below), and increasing it allows establishing clear trends from our small-scale hyperparameter sweep.
We study the scaling of the optimal loss as function of the compute budget. We show that the steps we take to settle the Kaplan et al./Hoffmann et al. discrepancy (namely shortening warmup and scaling learning rate and batch size) significantly decrease this loss at smaller scales, but only marginally improve it at larger scales. In contrast, introducing cosine learning rate decay

---

schedule substantially decreases the loss, with benefits persisting at larger scales. Similar to Hoffmann et al. [25], we observe some curvature on the optimal loss curve. Nevertheless, the optimal loss with tuned hyperparameters is fairly consistent with saturating power law.
We calculate the computational cost each of our experiments and plot how prediction quality improves as we consider larger training runs. We observe that the cost of our hyperparameter sweep is comparable to that of a scaling law fit experiment, but the compute saved by using a constant instead of cosine learning rate schedule roughly makes up for that cost.
Code and data release. To facilitate future research, we share the data and the code necessary to reproduce our analyses and figures at https://githu: com/formll/ resolving--calling-eee wraaling-Law-discree available at https http///gggggggfcccccooo co wo/formll/resolleeeeeeelollleeee http://eoovinnscaaallloaaeeaeeee scaling- lav-disccee sax-discrepanciess
In addition, checkpoints of the models we train
2
Preliminaries and experiment design
2.1 Notation and problem setting
We train language models size on D tokens of data (essentially without repetition). The precise definition of N plays an important role in this paper: Unless mentioned otherwise, N denotes the number of parameters in all the linear layers the model. That is, .V excludes embedding layers, but includes the model's 's head: the final linear layer producing the predicted token logits. (In the models we train there is no tying of the embeddings and the head).
Let FLOPs(N, D) be the ammount of floating point operations (FLOPs) required to train model of size N D tokens. Throughout, employ the approximation
FLOPs(NN D) Rs 6,N D
(I)
In Sections 3.1 and 3.2 we compare our definition of N to the one used in Kaplan et [30]. In Appendix B we also discuss the effect of taking attention FLOPs into account and FLOP estimation approaches in other works.
Let L(N, D) be the log loss (in expectation over the training data distribution and any randomness of the training procedure) obtained by model of size N trained for D tokens. Assuming fixed compute budget C, we aim to predict
(C) == argmin L
argmin N>0 D=FLOPs(N, D)-C
min
L(N,
6N
N>0
i... the model size yielding the smallest loss when trained with compute budget C under the approximation (I).
We also let
C
D²
C
D²
6N* and p'(C)
N*(C))
(C)
denote the optimal number of tokens and the optimal token-to- tor-ttmmterreerr ratio. To predict these quantities, power laws of the form:
N (C) No C DC(C) D3 Co and p" PC(C) pó C",
and fit the exponents a, b, and coefficients where Ný, Dūs Pō from data as described below.
2.2 Training setup
We train decoder-only Transformer language models using OpenLM [21], which integrates many of the architecture and training advances in Llama [55, 56] and subsequent works. We largely base our initial training configuration on the hyperparameter search in Gadre et al. [17]. Our setup does not replicate Kaplan et [30], but we match or closely approximate several key hyperparameters as discussed in Section 3. See Appendix C for detailed description of our setup and chosen hyperparameters.
"This notation abstracts away the fact that there many different models of size N and many different ways to train them for tokens. Ideally, L(N, D) represents loss attained by the optimal architecture size N' trained with the best possible training method that uses 1) tokens. In practice, for any value of (N, D) we consider only single configuration. but this configuration is the result of architecture search and optimizer tuning, performed either directly indirectly by building prior work.

---

Model set. We search for compute-optime models over set consisting of 16 models with sizes ranging from 5M to 901M. We pick model layer numbers and widths such that N increases by multiples of roughly /2 while the aspect ratio d/l stays between 32 and 64 as suggested in Kaplan et al. [30]. The number of attention heads in each configuration is 4, as preliminary experiments showed this is optimal for smaller models, and increasing did not noticeably improve larger models. Table 2 in the appendix specifies all the models in our grid.
Data. We perform our experiments on OpenWebTex12 [18] which contains roughly 30B tokens of data from Reddit and resembles the WebText2 dataset used in Kaplan et al. [30], well RefinedWeb [41] dataset which contains roughly 600B tokens from CommonCrawl [1] and resembles the Mes dataset that formed roughly half of the data mix in Hoffmann et al. [25].
Evaluation and FLOP grid. We evaluate models on 160M tokens held out from the training data. We perform the evaluation whenever the product of 6.V and the number D of training tokens seen so far crosses an element of FLOP grid of the form (1.25e16 2' 2 our data analysis. We also record the average training loss every 20 steps.
Ji=o. This grid plays central role in
2.3
Data analysis
Our technique for estimating the compute-optimal power law is akin to the second (IsoFLOP- approach of Hoffmann et al. [25], but differs in several details. The approach consists of two steps: directly estimating N (C:) for all C; in our FLOPs grid, and fitting power law these estimates. We briefly outline each step below and provide full details in Appendix D.
Estimating (C:). For each value of C, we train several models from our set (Table 2) for C, FLOPs and extract an IsoFLOP curve of loss vs. model size (see Figure 10). For FL.OP values where validation loss is not available (specifically Section 3.1 and Appendix B) we use the smoothed training loss instcad. We estimate (C,) and its uncertainty using woise-and-interpolle based on Gaussian noise with aupirically-calibaate every Ci, this yields "bootstrap sample" population optimal size estimates; we take their median as the point estimate for N (C,). The procedure also yields an estimate the log-scale standard deviation of N (C;) (shown error bars in Figure 1).
procedure
magnitude and Akima interpoaltion [3]. For
Fitting power law. We fit power laws of the form (4) by performing weighted linear regression in log space, with the weights inversely proportional the squared log-space standard deviations computed above (i.e., log-space Gaussian maximum likelihood estimation). To obtain point estimate for the power law parameters we fit the point estimates for each (C,) value. To quantify uncertainty, we fit power laws to bootstrap samples, obtaining population of Ný. a, and (+) samples. We construct confidence intervals from their quantiles.
Main results: settling the scaling law discrepancy
In this section, we describe in detail our main results, visualized in Figure 1, tabulated in Table and plotted in detail in Appendix E. The following subsections address each panel of Figure I in order.
3.1 Reproducing the Kaplan et al. scaling law
To reproduce the Kaplan et al. scaling law, match the setup of [30] in terms of the batch size 1219 tokens) and in terms of the learning rate schedule (warmup for 3000 229 x 1.57B tokens followed by cosine decay to zero at 2.5e5 2 s 131B tokens). Other configurations do not match exactly, but the suite of models we train covers a range of sizes and compute similar to Kaplan et al. [30]. For this reproduction only, we also take the "model size" N to be the number of parameters in all linear layers except the head (last decoding layer). That is, for a model of width d and vocabulary size v, we subtract d from our usual definition of N (see Table last column).
As Figure la shows, with this setting we obtain a compute-optimal exponent a and power law fits close to the power law (C..9(C///.66 1.9.64944999.88 obtained by Kaplan et al. [30, Figure 14, left).

---

Table I: Summary of our main results (described in Section 3).
Experiment
Dataset
a estimate
R2 of a fit
/² range
Hoffmann et al. [25]
Text
0.5
Kaplan et al. [30]
WebTex12
0.88
Adjusted Kaplan et [30]
WebTex12
0.73
Reproducing Kaplan et ($3.1)
OpenWebTex12
0.864 (0.82, 0.90)
0.998
(5, 2617)
RefinedWeb
0.835 (0.82, 0.85)
0.999
(8, 1536)
Counting last layer FLOPs ($3.2)
OpenWebTex12
0.699 (0.66, 0.72)
0.998
(8, 262)
RefinedWeb
0.706 (0.69, 0.72)
0.998
(9, 232)
Correcting warmup ($3.3)
OpenWebText2
0.603 (0.57, 0.63)
0.994
(7, 55)
RefinedWeb
0.602 (0.59, 0.62)
0.993
(7, 50)
Cosine decay ($3.4)
OpenWehTexxt
0.574 (0.54, 0.61)
0.999
(7, 42)
RefinedWeb
0.571 (0.56, 0.59)
0.998
(10, 39)
Optimizer tuning (no decay) ($3.5)
OpenWebTex12
0.518 (0.49, 0.54)
0.998
(11, 22)
RefinedWeb
0.497 (0.49, 0.50)
0.997
(14, 16)
Reprod. adjusted Kaplan et al. (SH)
RefinedWeb
0.717 (0.71,
0.992
(12, 345)
3.2 Counting last layer FLOPs
Kaplan et al. [30] chose to define model size without counting embedding parameters since they found this makes scaling laws in the infinite-compute regime more consistent across network depths [30, Figure 6]. Perhaps because their model head and embeddings had tied weights, this led them to also discount the contribution of the model head the model's FLOPs per token [30, Table 1, last row]. However, as Table reveals, not accounting for the model head leads to under-approximation that grows smoothly model size decreases, from roughly 10% larger models roughly 90% at smaller models. Thus, counting the head FLOPs (i.e., using our definition of N) results in significantly more accurate approximation. As shown in Figure 1b, switching to our model size count also reduces the exponent a by more than 0.1, closer to Hoffmann et al. but not all the way there.
3.3 Correcting learning rate warmup
Next, we address the duration of the learning rate warmup period, which Kaplan [30] set proportionally to their full training duration, designed to reach complete convergence. Figure 2 (left) shows this warmup period too long: for smaller-scale models, the optimal number of tokens as function of compute is less than or close to the number of warmup tokens, and therefore these models are suboptimally trained. The same issue evident in Figure 14 (right) of Kaplan al. [30] which shows that for many compute budgets the optimal number of steps is below or close to the number of warmup steps (fixed 3000). Figure (left) also provides an intuitive explanation for the increased value of a: smaller compute scales, models are 'forced' use more training tokens than would otherwise be optimal in order to "escape" the long warmup period. Having escaped, the warmup Once this warmup period is escaped, the optimal number of tokens grows only slowly, leading to fast rate of increase the optimal model size and hence the large exponent.
With the problem identified, we propose simple heuristic for more appropriately choosing the warmup duration: for each model. we the number of warmup tokens be identical the model size N. We validate our warmup heuristic in Appendix F. The bottom row of Figure 2b illustrates the validity of our new choice of warmup, showing that the optimal number of tokens is always least times greater than the (interpolated) duration of the warmup period corresponding to the model of the appropriate size. As is evident from this figure and from Figure Ic, shortening the warmup shifts the scaling law in the direction of Hoffmann et al. further, yielding exponent of roughly 0.6.
3.4 Learning rate decay has limited impact on compute-optimal allocation
With learning rate warmup corrected, we turn to study learning rate decay, which Hoffmann al. [25] conjecture to be main cause of the difference between their result and Kaplan et al. [30]. We

---

Kaplan et al. warmup
Our warmup
1010
101
95% confidence region
Power law fit
D'(C)
10a
10a
Hoffmann law Kaplan law
10a
10a
Warmup period Observations
107
10'
1016
10
10
101H
1016
1017
10th
10
Compute C [FLOPs]
Compute C [FLOPs]
Figure 2: The optimal number of tokens D as function of the compute budget C. Left: Using the warmup period of Kaplan al. [30], smaller models reach compute-optimality during warmup. Right: Setting the number of warmup tokens to be identical to the model size (visualized using the power law fit) ensures models reach compute-optimanne law closer to Hoffmann et al.. We replicate these plots for all of our experiments in Appendix E.
well after the warmup and yields a scaling
observe that the long 131B tokens decay period in Kaplan et al. [30], which is aimed toward training full convergence, means that their compute-cconstraneed experiments see virtually no learning rate decay: Figure 2 shows that, at our compute scales, it is never optimal to train for more than 10B, which corresponds less than 1.5% decay with a cosine schedule.
To correct this, we follow the second approach of Hoffmann et al. [25] and choose the learning rate schedule for every model and FL.OP budget individually. For each FLOP value in our grid, we pick the 7 models from Table 2 which yield token-to-pare parameter ratios in the range 1 to 100, and train them with cosine learning rate schedule that decays 1% of the maximum learning rate when reaching the target FLOP value. 5 This is roughly twice as expensive as previous experiments, which required only single training run for each model size (see additional discussion in Section 4.2). As Figure ld shows, adding cosine decay results in a slightly cleaner linear trend (A² improves from 0.993 to 0.998) and an exponent slightly closer the Hoffmann al. scaling law (0.57 instead of 0.6), but most of the gap remains. Therefore, even with the FLOP count and warmup issues corrected, adding learning rate decay is not sufficient reproduce the Hoffmann et al. scaling law.
3.5 Correcting batch size, learning rate and B2
A final factor contributing to the Kaplan et al. alooffffmnnn et al. discrepancy is the choice of optimization hyperparameters, particularly the batch size: with a fixed batch size of 219 tokens, compute-optime models at smaller scales train for only a few hundred steps. which is likely too little. Kaplan et al. [30] notice this issue, and attempt to correct for it using post-processing based on an empirical model of large-batch size training [36]; we return to their result at the end of this section.
Here, we take the more direct approach of predicting near-optimal hyperparameters for each model size.' 6 Since changing the batch size often also requires re-tuning the learning rate [19, 36, 49, 61], we sweep over both parameters for models of sizes 5M to 108M, with an additional validation sweep over models of size 220M. Initially, we kept 83 at its previous value of 0.95. However, this led to poor results at smaller batch sizes: as the batch size gets smaller, the squared gradients become noisier, and Adam W requires more smoothing to obtain a correct denominator. Therefore, we added 0.99 and 0.999 to the sweep, obtaining improved performance on small batch sizes. This empirical observation matches the theoretical work by Zhang et al. [63], that showed that when the batch size small, higher values of 32 are crucial for the convergence of Adam. In Appendix G we describe the parameter sweep in full, validate the optimality of our prescribed hyperparameters on the largest model we train, and provide additional discussion about the role of (2--
Figure 3 plots our estimates for the optimal values of batch size and learning rate for each model size. It shows clear trends, to which we fit power laws in the number of parameters N. Observing good
*e set the warmup period to be the minimum of the model size N and 20% of the total token budget. We ablate the final learning rate in Appendix F.
"More specifically, we predict the optimal hyperparameters per model size when trained for 20 tokens per parameter. As we discuss in Section 5.2, this choice of training budget is potentially an issue but further analysis in Appendix G.4 suggests it does not significantly impact results.

---

Batch size
Learning rate
Grld points, 8. 0.95
Grid points, B5 0.99
0.024
Grid points, #: = 00.99
256
Interpolated optimal valuees
0.012
Points used for fit
BS 0.0003744 (R' 0.966)
0.003
LR 3.7M 3.3 (R² (R"-0.9977)
Rounded fit: BS = 1601M108866
0.601.
LA 0.0047(W108e6)--2
DeepSeek
0,00075
BS 00068
15M 220M
10RM
LR - 0.117
N
Figure 3: Fitting scaling laws for the optimal batch size and learning rate as function of the model size N. Markers indicating grid points are shaded by their excess loss compared to all configurations for this parameter, reaching maximum transparency for loss that is suboptimal by 0.03 or more. We also plot interpolation-base based estimates of the optimal parameter values and fit them with power laws.
extrapolation to nearby values of N, we apply these power laws (with slight rounding) to select the batch size and learning for all model sizes and tabulate the results in Table 4. In Appendix G.5 we validate our learning rate and batch size scaling law for a model with 901M parameters.
Our parameter tuning approach is inspired by DeepSeek [15), who predict the optimal batch size and learning rate as a function of compute. Translating compute to model size using the Hoffmann et al. scaling law, we find remarkable agreement in the batch size predictions (a difference of less than 0.05 in exponent and less than 60% in predictions over our models), and somewhat different learning rate predictions (a difference of 0. 11 exponent and a factor of 2-3 in predictions), potentially due to using different weight decay. Both our results appear to contradict the conventional wisdom about the existence of a critical batch size [49, 61, 36] below which every batch size is good, finding instead an optimal batch size below which performance degrades. This suggests further tuning of 33 or other hyperparameters may be warranted. We discuss DeepSeek [15] further in Section 5.1.
With the new hyperparameters, we obtain a close reproduction of the Hoffmann al. scaling law (Figure le) with the scaling exponent matching 0.5 to within 0.6% and the predicted model size at Chinchilla compute within 15% of Chinchilla's s size. Notably, here we use a constant learning rate schedule, demonstrating that careful learning rate decay is not necessary for this scaling law to hold.
Finally, we reproduce the adjusted scaling law N (C) 1.3e9(C CC/8.66e1999..3 which Kaplan et al. [30] obtain by estimating the compute required to reach the same results at a sufficiently low batch size. To do so, we use our tuned hyperparameters previous corrections (head FLOP count and warmup duration). We obtain an exponent of 0.717 and good agreement with their adjusted scaling law; see Figure 18 in Appendix
proxy for suitable batch size and revert our
4
Additional Analysis
4.1 Trends in compute-optimal loss
Figure 4 shows the minimum loss achievable for each compute budget C' in the experiments shown Figure I. We estimate the minimum loss using the same interpolation procedure we use to extract the optimal parameter number N* and token count D*. The figure shows that, at low compute scales, shortening the warmup duration and tuning hyperparameters leads to substantial loss improvements (cach by up to 0.5 nat per token). However, at larger scales these interventions do not significantly improve the loss. contrast, learning rate decay becomes increasingly beneficial as compute grows, and appears to also improve the rate of decrease in the loss. Perhaps coincidentally, the effects of overestimating the optimal loss (due to long warmup and large batch size) seem to closely offset the effect of underestimating computational cost (by discounting the contribution from the model's head): the first and last curves in Figure 4 closely overlap.
Similarly to Hoffmann et al. [25] we observe a curvature in the optimal loss, while Kaplan et al. [30] report a near- pear-perfect power law behavior. This difference is due combination of the difference in FLOP counts discussed in Section 3.2 and the fact that the experiments of Hoffmann et al. [25]
7

---

Estimated optimal loss L (CC)
Reproducing Kaplan
20
95% confidence region
5'100
Counting last byer FLOPs
Point Estimate
Oprrecting warmup
Nominal value
Cosine decay
0.54
RMS relative
4x100
Optimizer tuning (ne decay]
128 932-3391 2.01
0.52
10
of reduced sweep Cost of full sweep
bootsitrag ap errer
Extrapolation
0.50
0.48
0.8
3x000
1017
10:4 10
Compute C (FLOPs)
Compute [[FLOPS]
Figure 4: The minimum loss achievable by models with compute budget C. For the Kaplan et al. scaling law reproduction, we estimate C as in Section 3.1. See expanded version in Figure 19.
Figure 5: Compute optimal exponent prediction, con- fidence, and root-mean-squal relative error as a func- tion of the total scaling experiment budget for the tuned optimizer experiment described in Section 3.5.
Original fit
One more point fit
Two more points for fit
Estimated optimal loss L (C)
turing
Ins decky!
(no 17-11----1 (C,.l.)
Kect a)
10
100
3x
3 10
104
1000
10
100""
1000 1044"
10:3
1044
11000"
1020
10ts
Tour"
10
103
10c0
Compute C (FLOPs]
Compute [FLOPs]
Compute [FLOPS]
Figure 6: The compute-optimal loss curve of Figure extended to compute budget C+, x 8e19 by training a single model with size, learning rate and batch size determined using our scaling laws. Each subplot uses a different number of lower-compute loss measurement fit the loss trend.
extend higher compute budgets where the loss is closer to its irreducible level. Indeed, for the tuned optimizer experiment (Section 3.5) we find that a saturating power law fits the optimal loss and extrapolates well, while extrapolating poorly for other experiments (see Figure 19 in the appendix). This suggests that a predictable trend in LCN*(C), (C))) is an indicator of locally- optimal hyperparameters. The exponent of our saturating power fit is approximately 10.1, twice as large as the exponent found in Kaplan et al. [30].
Finally, we validate our compute-optimal loss scaling law by training and evaluating a model using a larger compute budget. Specifically, we train a 901M parameter model with a compute budget of C_ 8e19 FLOPs, which our scaling law predicts to be compute-optimal, using the batch size and learning prescribed by our hyperparameter scaling laws (see Table 4), reaching a loss value of L4 = 2.943. In Figure 6, we add the point L4) to the red curve in Figure 4 and find that it falls within the predicted trend. Notably, L4 is obtained with a single training run using our predicted optimal configuration, while at lower compute values we estimate the optimal loss by interpolating an IsoFLOP curve.
4.2 Scaling law accuracy as a function of compute
We now estimate the computational cost of our scaling law experiments, quantifying the effect of the learning rate schedule, and plot how our predictions improve and become more confident with increased computation. We find that the training cost of each experiment that utilized fixed learning rate schedule was 1.54c20 FLOPs, while the experiments that used a varying-length cosine learning rate schedule required 2.99c20 FLOPs; essentially double the compute; see Appendix for more details. We also find that the cost of the hyperparameter sweep described in Section 3.5 was 2.04c20 FLOPP-slightly less than the combined cost of two scaling experiments that leveraged it (one on each dataset). Moreover, in hindsight, we could have arrived at similar hyperparameters using only models of size at most 57M and simple heuristic for choosing f2 based on batch size, which would have cost only 1.44e19 FLOPs.
8

---

Figure 5 shows the evolution of the predicted compute-optimal model size exponent a, its confidence interval, and a measure of the prediction accuracy as we modulate the experiment's compute budget by truncating our FI.OP grid. The figure shows that the prediction becomes steadily more accurate and confident as compute increases, We present these results, as well as the results in Section 4.1, on the OpenWeeTexx22 dataset as well (see Appendix I).
Discussion
5.1 Related work
While neural scaling laws precede the advent of large language models [24, 45], breakthroughs in model [57] and data [42, 44] scaling allowed Kaplan et al. [30] to demonstrate the dramatic possibility of unbounded improvement with scale, triggering explosion in the literature the topic. Here focus on the relatively fewer works that tackle optimal resource allocation under a compute constraint.
For language modeling, Hu et al. [26] and DeepSeek [15] repeat subsets of the analyses in Hoffmann et al. [25] and derive compute optimal scaling laws. Employing Approach 3 of Hoffmann al. [25] (see also [9]). Hu et al. [26] find that, for their models, optimal scaling favors larger token-to- parameter ratios than in Hoffmann et al. [25] and in our results. They attribute this difference to modeling improvements since [25] and argue the same holds for Llama [56]. However, our setup incorporates most of the advances in Llama 2 and still produces power laws very close to Hoffmann et al. [25]. Like us, DeepSeek [15] perform hyperparameter tuning and use isoFLOP analysis to determine compute-optimal model sizes on multiple datasets. While they arrive at an exponent on the order of Hoffmann et al. [25] for the main dataset they study, they report higher exponent a 0.578 for OpenWeb'Tea WebText2 (i.e., predicting lower aaken-parameter-rannn superior quality of the dataset. We also study this dataset but arrive much closer to the Hoffmann et al. scaling law. We conjecture the larger exponent might be due to repeating training data, which likely occurred in their experiment given the dataset's limited size and their compute budget. Settling these discrepancies could be source of further valuable lessons on optimal model scaling.
at scale), which they attribute to the
Recent work also studies compute-bounded scaling laws beyond the compute-optimal regime. In- formed by the increasingly common practice of training medium-scale models beyond compute optimality [c.g.. 55, 56, 29]. Sardana and Frankle [46] account for the expected inference cost of the model, showing that it naturally skews optimal settings toward smaller models. Gadre et al. [17] directly predict the loss and downstream performance for models trained past the point of compute optimality, and Muennighoff et al. [37] model joint compute-date bottlenecks. All three works rely on the Hoffmann et al. law as reference point, with [17, 37] baking it to their parametric forms.
Compute-optimal scaling is studied beyond the language domain, particularly in vision. Henighan et al. [23] study autoregressive modeling for variety of tasks and find scaling laws roughly consistent with the Kaplan et al. [30] adjusted scaling law (with exponent a = 0.73). That work shares the methodological issues described in the top row of Figure 1 (FLOP count and long warmup), but performs hyperparameter tuning for smaller scale models; in Appendix H we reach similar results when doing the same. Zhai et al. [59] characterize the compute- compute-efficient frontier of Vision Transformers (ViT's, while Cherti et al. [13] studies compute constrained scaling of CLIP models. However, they do not offer a power law for scaling model size with compute. Alabdulmohsin et al. [4] tackle model design under an inference compute constraint by fitting multi-term parametric forms to obtain predictions for the optimal ViT shape. Goyal al. [20] point out intricate interplay between data filtering and compute constraints. Finally, Bachmann et al. [7] study compute-optimal scaling of MLP's and obtain exponent a 0.35, suggesting that MLP require much more rapid data growth than more sophisticated architecture. Overall, whether and to what extent does Hoffmann et al. scaling hold in the vision domain remains a compelling open problem.
Recent theoretical work study compute-optinal scaling laws in simplified, analytically tractable settings [11, 33, 38, 28]. In particular, Paquette et al. [38] obtain a power law with exponent 0.5 (as in Hoffmann et al. (25]) for a random- linear regression setting [35, 5], conjecturing that this is part of a broader. universal phenomenon. Jeon and Van Roy [28] also establish exponent of 0.5 for data generated by wiitt two-layer ReLU networks, using cooommttonntheoretee arguments.
We also remark on two themes of our paper that draw from prior work. The first is the importance of hyperparameter tuning: several works [30, 27, 17, 15] make the case that smooth, predictable

---

scaling laws emerge when models on all scales are properly tuned. Our work (and particularly Section 4.1) provides another example of this principle and agrees with previous observations that tuning is particularly important at smaller scales. Second, previous studies [59, 8, 15, 26] as well as the concurrent work [22], propose alternative learning rate schedules that address a key shortcoming of cosine decay: the need to commit to a step budget in advance. We consider a constant learning rate that requires no commitment at all. We show this simple choice suffices to reproduce the Hoffmann et al. law and quantify the computational savings compared to a cosine schedule. However, Section 4.1 (and also [25], among others) show that in terms of loss, the constant schedule clearly underperforms the cosine schedule.
Concurrent and independent work. Pearce and Song [40] also study the discrepency between the Kaplan et al. and Hoffmann et al. scaling laws. By re-analyzing data extracted from the Hoffmann et al. [25] experiments by Besiroglu et al. [9], they identify the last layer FLOP count as a cause for the discrepancy. Moreover, they report on a small-scale experimental study (with model sizes up to 5M and training tokens number up to 530M) in which they observe that a non-decaying learning rate schedule is sufficient for reproducing the Hoffmann et al. exponent and that learning rate tuning is necessary. These results independently corroborate part of our observations in Sections 3.2, 3,4 and 3.5. Pearce and Song [40] do not identify the warmup duration issue describe in Section 3.3. As a consequence, when reproducing the Kaplan et al. exponent they reach a value close to 0.73 rather than the "raw' value 0.88 reported in Kaplan al. [30] (see discussion in Section 3.1, Section 3.5, and Appendix H). In addition, our experiments roughly match the Kaplan et al. [30] compute budget, which is about 3 orders of magnitudes larger than budget in Pearce and Song [40], and we perform careful tuning of both the learning rate and the batch size.
5.2 Limitations
Computational scale is notable limitation, as well defining feature, of our results: our ex- periments are roughly on the scale of those in Kaplan et al. [30] but are substantially smaller than those of Hoffmann et al. [25]. Scaling may effectively mitigate each of the issues we identify; with scale, the contribution of the model head becomes negligible, any (fixed) warmup period eventually becomes reasonably long, and hyperparameter sensitivity decreases, as shown in Figure and Fig- ure 15. Nevertheless, we believe that experimental protocols that induce correct scaling behavior at low computational budgets are crucial for developing the empirical science of machine learning. particularly in academic settings.
Due to limited compute budgets, our hyperparameter sweep only targeted the smaller models in our grid, and furthermore trained each model for only 20N steps, i.e., the optimal point according to the Hoffmann al. scaling law. This raises the concern that the hyperparameters we chose unfairly favor models trained for that particular perfect tuning for each model size and each results. We believe this is unlikely: at small scales (where hyperparameter tuning is crucial) our original set of hyperparameters favored higher token-to- t-r-mmmeer ratios because they still had a sufficient number of steps to train for, and therefore choosing hyperparameters specifically for them not likely to result in significant gains. In Appendix G.4 we analyze our existing tuning results to estimate the potential gains from perfect tuning, and find that they are likely to have small impact on our conclusions. Moreover, transferring our hyperparameters to another dataset yields similar results.
ratio, and rerunning our experiment with to-aaaameteerr ratio would have yielded different
Finally, a broader limitation of compute-optimal scaling as defined by Kaplan et al. [30], Hoffmann al. [25] and our work, is that only concerns the pretraining loss rather than more direct measures of a model's 's capabilities. Here again, scale is an issue: most zero-shot and in-context capabilities do not emerge at the scales we consider here, and predicting them from small-scale proxies is an important open problem [48, 17]. Instead, it is possible to study downstream performance via fine-tuning, though this may cause the clean scaling patterns seen pretraining to break down [54], potentially because the fine-tume tie-ttnnng procedure is sensitive to the choice of hyperparameters [27].
10

---

Acknowledgments
We thank Georgios Smyrnis, Samir Yitzhak Gadre, Achal Dave, and Mehdi Cherti for helpful discussion and assistance with OpenLM and with the JSC cluster.
TP and YC acknowledge support from the Isracli Science Foundation (ISF) grant no. 2486/21 and the Adelis Foundation. MW was supported in part by a Google Fellowship. JJ acknowledges funding by the Federal Ministry of Education and Research of Germany under grant no. 011S22094B WestAl AI Service Center West. LW acknowledges funding from Open Philanthropy. We gratefully acknowledge compute budget granted by Gauss Centre for Supercomputing e. V. and by the John von Neumann Institute for Computing (NIC) on the supercomputers JUWELS Booster and JURECA at Jülich Supercomputing Centre (JSC).
References
[1] Common Crawl. https::/ccmmonerari// org.
[2] New scaling laws for large language models, 2023. URL https:///tw...... http::///wwwwwesss..ooeo lesswrong.. com/ posts/mi httpp///ww/////ggge//essaaaaaaaa lennellwss-oooe corr-arge---e Marge-language language lwngaage-models......eooeee..sse Ac- cessed: 2024-08-19 19.
[3] H. Akima. A new method of interpolation and smooth curve fitting based local procedures. Journal of the ACM (JACM), 1970.
[4] I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting ViT in shape: Scaling laws for compute-optimal model design. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[5] A. Atanasov, Bordelon, S. Sainathan, and C. Pehlevan. The onset of behavior for networks in the lazy and rich regimes. In International Conference on Learning Representations (ICLR), 2023.
[6] J. L. Ba. J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:6607.064500 2016.
[7] G. Bachmann, S. Anagnostidis, and T. Hofmann. Scaling mips: tale of inductive bias. Advances in Neural Information Processing Systems (NeurlPS), 2024.
[8] M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable Im 2 1.6 b technical report. arXiv:2402. 17834, 2024.
[9] T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt. ar04v:2404. 10102, 2024.
[10] S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox-20b- An open-source autoregressive language model. arXiv:2204.067745 2022.
[11] B. Bordelon, A. Atanasov, and C. Pand Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning (ICML), 2024.
[12] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 2020.
[13] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, I. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on computer vision and pattern recognition (CVPR), 2023.
[14] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113 2023.

---

[15] DeepSeek. Deepseek LLM: Scaling open-source language models with longtermism. arXiv:2401.0295. 2024.
[16] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, L. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference Machine Learning (ICML.), 2023.
[17] S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang. J. Li. S. Keh, R. Xin, M. Nezhurina, Vasiljevic, Jitsev, A. G. Dimakis, G. Ilharco, S. Song, T. Kollar, Y. Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt. Language models scale reliably with over-training and on downstream tasks. arXiv:2403.08540, 2024.
[18] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.0002) 2020.
[19] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training imagenet hour. arXiv: 02677, 2017.
[20] S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter. Scaling laws for data filtering data curation cannot be compute agnostic. In Conference on computer vision and pattem recognition (CVPR), 2024.
[21] S. Gururangan, M. Wortsman, S. Y. Gadre, A. Dave, M. Kilian, W. Shi, J. Mercat, G. Smyrnis, M. Jordan, R. Heckel, A. Dimakis, A. Farhadi, V. Shankar, and L. Schmidt. OpenLM:
G. Ilharco, a minimal but performative language modeling (Im) repository, 2023, https: //gt::///gttbb com/ mlf mlfound ilfoundation cotptttinnooooon___.. 1mm
[22] A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. Von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405. 18392, 2024.
[23] T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhari- wal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv:2010. 14701, 2020.
[24] J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv:172..00 1712.0049, 2017.
[25] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] S. Hu. Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint 244.::2404.06355, 2024.
[27] M. Ivgi, Y. Carmon, and J. Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022.
[28] H. J. Jeon and B. Van Roy. Information-theerrticc foundations for neural scaling laws. arXiv:240..01456 2024.
[29] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, F. B. Diego de las Casas, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. arXiv:2310.06825, 2023.
[30] Kaplan, S. McCandlish, T. Henighan, B. Brown, B. Chess, R. Child, Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models, arXiv:2001 2001:22001.08311, 2020.
12

---

[31] B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, J. Hu, M. Tintore. S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, Reizenstein, and G. Sizov, xformers: A modular and hackable transformer modelling library. https://g" https://github..... thub com/ facebookressarcc/fffrreees ormerrs, 2022.
[32] O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation, 2021.
[33] L. Lin, J. Wu, S. M. Kakade, P. L. Bartlett, and J. D. Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv:2406.0844 084666666, 2024.
[34] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv: 2017.
[35] A. Maloney, D. A. Roberts, and Sully. solvable model of neural scaling laws. arXiv:2210.16859, 2022.
[36] S. McCandlish, J. Kaplan, D. Amodei, and The OpenAl Dota Team. An empirical model of large-batch training. arXiv: 1812.0881.60600 2018.
[37] N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo. T. Wolf. and
C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems (NeurIPS), 2024.
[38] E. Paquette, C. Paquette, L. Xiao, and J. Pennington. 4+3 phases of compute-optimal neural scaling laws. trXiv:2405.15074 2024.
[39] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. 2019.
[40] T. Pearce and J. Song. Reconciling Kaplan and Chinchilla scaling laws. arXiv:2406. 12907, 2024.
[41] G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Advances Neural Information Processing Systems (NewrIPS), 2023.
[42] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners, 2019.
[43] W. Rae, Borgeaud, T. Cai, Millican, Hoffmann, Song, Aslanides, S. Henderson, R. Ring, S. Young, al. Scaling language models: Methods, analysis & insights from training gopher. 1rXiv:2112.11146. 2021.
[44] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1-67, 2020.
[45] S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. constructive prediction of the 2019.
generalization error across scales. arXiv:
[46] N. Sardana and J. Frankle. Beyond Chinchilla-optimal:: Accounting for inference in language model scaling laws. arXiv:2401.00448, 2023.
[47] L.. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilié, D. Hesslow, Castagné, A. Luccioni, F. Yvon, al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv:2211.05100 2022.
[48] R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems (NeurIPS), 2024.
[49] C. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. F. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research (JMLR), 20 1-): 2019.
13

---

[50] N. Shazeer. Glu variants improve transformer. arXhv:2002.05002, 2020.
[51] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. 19Xx:0900.00053, 2019.
[52] Smithh, M. Patwary, B. Norick, P. LeGrresey, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al. Using DeepSpeed and Megatron to train Neggaron-Tuuring NLG 530B, large-scale generative language model. rxiv:2201.11990 2022.
[53] Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063. 2024.
[54] Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICI.R), 2022.
[55] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models. arX11 2023.
[56] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Mao, X. Martinet, Mihaylov. Mishra, I. Molybog. Y. Nic, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian,
E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307, .00288, 2023.
[57] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and
I. Polosukhin. Attention is all you need. 2017.
[58] M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, Pennington, J. Sohl-Dickstein, K. Xu. J. Lee, J. Gilmer, and S. Korbblith. Small-scale proxies for large-scale transformer training instabilities. In International Conference on Learning Representations (ICLR), 2024.
[59] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Conference on computer vision and pattern recognition (CVPR), 2022.
[60] B. Zhang, Titov, and R. Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[61] G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. In Advances in Neural Information Processing Systems (NeulPS), 2019.
[62] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. OPT: Open pre-trained transformer language models. arXiv:2205.0100 2022.
[63] Y. Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems, 35:28386-28399, 2022.
14

---

A Interpreting Hoffmann et al.'s hypothesis
Hoffmann et al. [25] hypothesize that the scaling law discrepancy due to a difference in learning rate schedules. The term "learning rate schedule" comprises both the warmup and decay components, and it plausible to interpret the "Modeling the scaling behavior" paragraph in Hoffmann et al. [25] $2 as conjecturing that both components contribute the scaling law discrepancy. However, believe that Hoffmann et al. emphasize the decay component of the learning rate schedule in their hypothesis due to two main reasons.
1. Figure Al in Hoffmann et al. In $2, Hoffmann et al. refer to Figure A1 to provide evidence that the learning rate schedule affects compute optimal scaling. However, in this figure they only vary the decay component of the schedule, keeping the warmup fixed. Hence, it appears that-in the scope of their Sopothesis-Hoffmann learning rate decay.
al. equate learning rate schedule with
2. Subsequent literature. In Hu et al. [26] $4.1, Hägele et al. [22] $1 and in [2] the authors refer to the decay component of the learning rate schedule the cause of the discrepancy. This suggests that the community has interpreted Hoffmann et al.'s 's hypothesis as referring to the decay component.
B Estimating FLOPs and accounting for attention
In this section, we compare our definition of model size N to three alternatives and also discuss choices made by related work. Before that, we provide the precise expression for computing N (by our definition) from the model depth I, width d, and vocabulary size e = 50 432. Due to efficient implementation considerations, OpenLM sets the model's feedforward dimension to
Since each SwiGLU feedforward block has 3 di X d parameter matrices, and since each attention block has 4e parameters in linear layers, our total estimate is:
dFF 256
255 8d/3
N = (3dFF + 4d)dl + de.
(5)
We begin by considering, instead of the number of weights in linear layers, the total number of non-embedding learnable weights Nexact (e.g., including also LayerNorm gains). The fourth column of Table 2 shows that the difference between this number and N is negligible. We also note that embedding layers have a negligible contribution to the model FLOP counts, since they do not require matrix-vector products.
Consequently. the only non- negllggiie source of error in the approximation FLOPs(N, D) = 6N D the attention layers. Since OpenLM the attention dimension is identical to the model width, Kaplan et al. [30, Table 1] shows that the attention operation costs an additional 6nd FLOPs per token per layer for a forward and backward pass, where n 2048 the sequence length. Thus, we define an effective model size of
Notf == N + ndl,
we have that 6Nar D captures the cost of training the model for D tokens, including attention.
We now consider the difference between these approximations and its effect on compute-optimal scaling laws. The fifth column of Table 2 compares Not to N. It shows that the ratio Not/N changes smoothly between roughly 1.1 to roughly 1.2 and back to 1.1 as our model sizes grow. We note that had this ratio been completely constant, there would have been working with N and working with Nen since a power law in one would correspond directly to a power law in the other. Since in our model grid this ratio is approximately constant, we expect to see limited differences between the scaling laws resulting from each definition. Figure 7 confirms this expectation, showing quantitatively and qualitatively similar results Figure 1. Consequently, we cannot determine with certainty which definition is more appropriate for predicting compute- optimal model sizes. Nevertheless, we observe our final experiment (with parameter tuning) predicts that the optimal (effective) model size at the Chinchilla parameters larger than the one size predicted using our standard definition. These predictions are directly comparable since model size and effective model size are essentially identical at these scales. If we take the Hoffmann et al. scaling law as ground truth, then the prediction we get using Not is a bit worse.
essentially difference between
compute scale to be about 16 B
15

---

Counting last layer and attention FLOPs
Correcting warmup
0.717
(0.00.7331
0.622 (161,0))
[CCC 954B (TH1B 11)
(CC)c) 413B (STLR 4779)
10#
10
Cosine decay
Optimizer tuning (no decay)
0.577 (0.56, 0.59)
0.505 (0.50 0.6))
[Cc) 228B
(18B. 288
(CC)c) 94B (84B, 1063)
10a
N110
10t
100
1014
101
1037
10th Compute [FLOPS]
Compute [FLOPs
95% confidence region
Power law fit
Hoffmann law
Kaplan law
Observations
Figure 7: A reproduction of Figure where replace standard definition of model size N with the effective model size Nei (defined in eq. (6)) for which the approximation 6.N D also captures the FLOPs cost of attention operations in the model. We observe similar results Figure though our final experiment produces a mildly higher prediction for the optimal model size at the Chinchilla/oopher compute scale.
Finally, touch third measure of model size, which does not count the contribution of the model's head to the FLOP count. That is, we consider
NKaplan N
This is the definition that Kaplan al. [30] use in their experiment, approximating the flop count as 6.N Kaplln D. Their main motivation for this choice is an observation that not counting embedding parameters leads to more predictive scaling laws in the unlimited compute regime. However, as the final column of Table 2 shows, this approximation leads to large, systematic error in FLOPs counts for smaller models. In Sections 3.1 and 3.2 we show that this one of the primary factors behind the Kaplan et alffHoffmann et al. discrepancy.
We conclude this section with overview of the model size definitions used by related works other than Kaplan et al. [30]. Henighan et al. [23] use NKsplan as Kaplan et al. [30] and observe high scaling exponents as result. Hoffmann et al. [25] account for both linear and attention layers their FLOP computation essentially using Non in their first two estimation approaches. However, their third approach appears ignore the attention FLOPs and also count the embeddings parameters, iee, setting N' N + de. DeepSeek [15] compare 3 definitions of model size, including Napllan N. and hybrid of Nor and Napllan that takes attention into account and ignores the model head. They report that the latter option gives the best prediction of the compute-optimat loss at large scales. However, we note that both [25] and [15] claim that attention costs double the FLOPs mentioned in Kaplan et al. [30]; we believe this is likely because they do not account for the fact that the attention causal, meaning it requires only half the FLOPss of unstructured matrix-vedt -vectrr product. Finally Muennighoff et al. [37], Gadre et al. [17]. Hu et al. [26] use N as we do.
16

---

Table 2: Model architectures and different parameter counts, in millions. The column N gives definition of models size, Nexact is the exact number of trainable parameters in the model, Nen is the effective size which also accounts for the cost of attention operatoins, and Naaplan does not count parameters in the model's head. See Appendix B for precise definitions and discussion.
Depth
Width
N
Nexact
Neff
NKaplan
3
96
5.173
5.176 (+0.05%)
5.763 (+11.4%)
0.331 (.935%)
4
128
7.504
7.508 (+0.06%)
8.552 (+13.9%)
1.049 (-86.0%)
5
160
9.810
9.817 (+0.07%)
11.45 (+16.7%)
1.741 (-82.2%)
6
224
15.60
15.61 (+0.07%)
18.35 (+17.6%)
4.301 (-72.4%)
8
288
22.49
22.51 (+0.08%)
27. (+20.9%)
7.963 (-61.5%)
9
320
28.67
28.70 (+0.08%)
34.57 (+20.5%)
12.53 (.56.2%)
10
384
37.06
37.09 (+0.08%)
44.92 (+21.2%%
17.69 (-52.2%)
12
480
57.38
57.43 (+0.08%)
69. 18 (+20.5%)
33.18 (-42.1%)
14
576
84.79
84.85 (+0.08%)
101.3 (+19.4%)
55. (-34.2%)
15
640
108.5
108.5 (+0.07%)
128.1 (+18.1%)
76.19 (-29.7%)
18
704
149.0
149.1 (+0.07%)
175.0 (+17.4%)
113.5 (-23.8%)
21
832
220.9
221 .0 (+0.06%)
256.7 7 (+16.29)
178.9 (-19.0%)
23
1024
347.1
347.3 (+0.05%)
395.3 (+13.9%)
295.4 (-14.8%)
26
1120
455.3
455.5 (+0.05%)
514.9 (+13.1%)
398.8 (-12.4%)
26
1312
612.0
612.2 (+0.05%)
681.8 .8 (+11.4%)
545.8 (-10.8%)
30
1504
901.7
902.1 (+0.04%
994.1 (+10.2%)
825.9 (-8.41%)
C
Additional training setup description
Modeling. We train decoder-only Transformer [57] language models for next-token prediction using OpenI.,, Pytorch [39] for efficient training medium scale language models. We use the library with largely the same configuration as [17), leveraging xFormers [31], bfloat16 automatic mixed precision, (ak)-LayerNNomm [6. 16, 58), SwiGLU [50], depth-scaled initialization [60], and rotary positional embeddings [53]. We use the GPT-NeoX-200 tokenizer [10] whose vocabulary size of 50432 closely matches the vocabulary size of Kaplan et al. [30]. We use a sequence length of 2048 which is twice the sequence length used in Kaplan al. [30]. but attempt to match parameters like batch size and warmup duration in their size in tokens. We do not tie the weights of the embeddings and head layers.
Optimization. Throughout the paper, we use the AdamW optimizer [34] to minimize the standard log loss with an additive z-loss term for stabilization [14] (coefficient 1e-4) as an auxillary loss (for our analysis, we record the log loss without the z-loss term in both train and validation). As advocated for in Wortsman et al. [58], we use independent weight decay [34] with parameter 1e-4, we set the "weight decay" parameter in the standard Py PoTcch AdamW implementation to be 1e-4/n, where 7 is the base learning rate. In Table 3 and Table we describe our choice of hyperparameter in our experiments.
Hardware and computational cost. We train our models cluster with 40GB A A100 GPU's, using between 4-32 GPU's in parallel per training run. We use the OpenL Torch distributed data parallel implementation as well as gradient checkpointing. According to logs, the total compute cost of all the experiments going into this paper is 22.3K GPU hours., and the total FLOP count is 3.03e21 FLOPs.
Data repetition. The datasets we work with are large enough to allow us to perform all of our training runs without any data repetition. However, due to two software issues, some experiments experienced limited data repetition. In particular, data going into our hyperparameter sweep might
17

---

Table Fixed hyperparameters. We use these values in the experiments Sections 3.1 3.4 and later tune part of them as described in Section 3.5 and Table 4 below.
Name
Value
Batch size
256
Learning rate
3c 3
A independent weight decay
1e 4
AdamW Bi
0.9
AdamW B2
0.95
Z-loss weight
le-4
Table 4: Tuned hyperparameters. We choose these parameters according to the scaling law present in Section 3.5. Due to parallelization requirements, we round batch size to be multiple of number of GPUs used in each run. We also round learning rate two significant digits. All other hyperparameters are as in Table 3.
N (millions) Learning rate
Batch size
0.013
20
0.99
7
0.011
28
0.99
9
0.011
32
0.99
15
0.009
44
0.99
22
0.008
56
0.99
28
0.0074
64
0.99
37
0.0068
80
0.99
57
0.0059
104
0.99
84
0.0051
128
0.99
108
0.0047
160
0.99
149
0.0043
192
0.99
220
0.0038
256
0.95
347
0.0032
320
0.95
455
0.003
448
0.95
611
0.0027
512
0.95
901
0.0024
640
0.95
have been repeated to 10 times. Moreover, on OpenWebTex12,, some of our larger-scale training runs might have seen data repeated up times. We believe this had limited to no impact our results, as the hyperparameter sweep involved fairly small models unlikely to be able to memorize, while [37] show that 4 data repetitions have only marginal effect on model performance, In our main experiments on the much larger RefinedWeb dataset we have verified that no data repetition occurred.
D Additional data analysis details
This section provides comprchensive description of our procedure for fitting the power law for (+).). Our procedure for the power law is analogous, using the relationship (3).
Training loss smoothing. We smooth the training loss using variable-length smoothing averaging window. In particular, estimate the loss at step as the average of the losses in steps i [pi] to
18

---

Reproducing Kaplan et all
Counting last layer FLOPs
Correcting warmup
Lo25
250
1pa
1344
12.1
3p4
129
134
124
Compute C [FLOPs]
Optimizer tuning (no decay)
Cosine decay
0.17
6-10
0
501
14.54
7M
103M
561
14:M
222M
347M
903M
30"
Compute [[FLOPS]
Compute [FLLOPs]
Figure 8: Training loss vs. compute for main experiments on the RefinedWeb dataset. The smoothed loss is overlaid on semi- cami-transparent raw loss.
Reproducing Kaplan et
Counting last layer FLOPs
Correcting warmup
LoS5
301
15
401
Compute C [[FLOPs]
Optimizer tuning (no decay)
Cosine decay
5+10
5x10
0
4x17
5M
94M
785
105M
9M
149M
22001
2+17
37M
611M
57M
903M
1plr "Ve Compute [[FLOPS]
Compute [[FLOPs]
Figure 9: Training loss vs. compute for our main experiments on the Open WebTee2 dataset. The smoothed loss is overlaid on semi-transparent raw loss.
+ [pi], for p- = 0.05. We also compensate for the lag introduced by logging the averaged training loss every 1: = 20 by shifting the training loss's index k/2; this compensation is quite important for matching the validation loss early in the optimization. We have verified that the smoothed training loss matches the validation loss (where it is available) roughly the validation loss's sampling error.
Fetching the loss at C, FLOPs from a single run. To estimate the loss of a model of size N trained to compute C, we linearly interpolate the validation/training loss (in log-space) at the two steps closest to C,/(6N B), where B is the batch size in tokens. We also require the nearest step to be within 10% of Cc/(6NB), and do not return the loss if not such step exists. For most of our experiments, we compute the validation loss precisely at step [C, However, for the
19

---

Reproducing Kaplan
Counting last FLOPs
Correcting warmup
3x20
7x:00
5x000
5x
Los
4x30P
4x.00
4x:00
204
137
101
10'
137
30P
307
15
N
Optimizer tuning (no decay)
Cosine decay
7x20
7x.20
Ostinal in interpelated curve interpslates
4x:00
1.25e+16
2
Compute [PLOP4]
3x.00
Figure 10: IsoFLOP curves for our main experiments on the RefinedWeb dataset. The values marked with stars are estimates of (C;) for the depicted C values. See Appendix D for a related discussion.
experiments in Section 3.1 and Appendix B, which consider alternative definitions of N do not have validation loss samples, and we use the training loss instead.
Estimating loss noise. Defining the ideal loss the population log loss expectation over model training, there are two sources of error in estimating it: finite samples in the validation set, and variation between training seeds. We estimate the former directly by storing the validation loss 100 subsamples of the holdout data and find the standard deviation to be in the range 0.001-0.000 across the different experimental settings. To gauge the error due seed variance, we train smaller- models from our grid on 7 seeds using the tuned hyperparameters for 20.V tokens each. We find roughly log-lineat relationship (see Figure 12) between the poost-warmup) smoothed training loss and the inter-seed standard deviation. For RefinedWeb (Table 4), it appears to saturate around the sampling error, and we heuristically assign standard deviation 0.05 samples with loss 7, standard deviation 0.002 to samples with loss < 3, and linearly interpolate the standard deviation in log-space to samples with loss in the range [3, 7). For Open WebText2 we observe significantly more cross-seed variance as well as less stable loss during training (compare the loss curves in Figures 8 and 9), potentially due to a difference in document lengths. Therefore, we set our standard deviation estimate to go from 0.1 at loss 6 to 0.01 loss saturating outside the interval and log- oog-space interpolating inside it.
Estimating N (Cx) and its uncertainty. Given a list of N values and their respective loss samples compute C, (fetched as described above), we estimate the optimal value N*,(,)) using the following bootstrap-Iis like procedure, For each bootstrap sample, we add independent Gaussian noise to each loss, whose standard deviation is determined according to the heuristic formula described above. We then interpolate the curve of loss vs. N using Akima [3] interpolation in log-space and find the value minimizing the interpolant; this forms our population of bootstrap samples for N (C,) estimate. We estimate their standard deviation in log-space, and take the maximum between that value and one-third of the average log-spacing on the grid (roughly 3 log /2). Occasionally, N appears on the edge of the N grid (though we attempt to avoid this our experiment design). If more than half of the bootstrap samples land at the edge of the grid, we omit the value of C; from the subsequent power fit. Otherwise, we keep only the samples outside the grid edge, and blow up the standard deviation estimate by the fraction of omitted samples.
20

---

Reproducing Kaplan
Counting last FLOPs
Correcting warmup
Lozs
4x200
4x20
4x
3x10
3X00
3x.00
204
137
107
10
137
10P
337
15
N
Optimizer tuning (no decay)
Cosine decay
Sx
Ostimal in interpelated curve interpslates
4x:0
1.25e+16
1.
Compute [PLGP4]
3820
3x:10
Figure 11: IsoFLOP curves for our main experiments on the Open Web'ee22 dataset. The values marked with stars are estimates of (C;) for the depicted C₁ values. See Appendix D for a related discussion.
RefinedWeb
OpenWebText2
N SM
13-
20-1
N 7M
Standard devaton of loss
N 28M
57M
Validation loss
Estimated vaidation samping
Bootstrap
Figure 12: Cross-seed loss standard deviation vs. loss mean, overlaid with our heuristic "bootstrap" noise standard deviation (see Appendix D). The dash-dot lines indicate the standard deviation due finite validation set size, estimated on the last validation checkpoint and averaged across seeds.
E Additional plots for main experiments
In Figure 13 we complement Figure by plotting our observation and power law fits for D* p* and N* for all the experiments described in Figure In Figure 14 we reproduce this figure for the OpenWebText2 dataset, showing consistent qualitative and quantitative results.
Ablation of warmup duration and final learning rate value
We preform small scale experiments to verify that our heuristic warmup duration (as discussed in Section 3.3) and the learning rate at the end of cosine decay (as discussed in Section 3.4) have little impact on our results. We train a model with 108M parameters on the RefinedWeb dataset. We use the same hyperparameter models on held-out Refined Web data and calculate the validation loss and standard deviation due to sampling described in Appendix D.
in Table 4, with training duration of ~ 2. tokens. We evaluate the
Warmup duratiom. We vary the warmup tokens from N/4 to 16N. We tabulate the results in Table 5, and find that N tokens for warmup is nearly optimal.
21

---

N
D
10th
-0.03 (-00)
0.83 (052,
0.103
10.15.0.1
(CCc
0.01 juil 001)
Nh(Co) 31 (JT,
Dh((Ce) 32B (280, 2003
Reproducing Kaplan
10P
10a
102
10
10-5
10.2
1040
10.
1010
Counting last layer FLOPs
10a
(-0.42, -.57)
0.91)
0.18 (6.13, 0.35)
("(C 7870 Jo00, 91OR)
(PM(Ge)
1241 1000, Hony
10"
10a
10P
10
109
10
10
10
016
10
1020
16
10
109
-0200 (-021-6-10
1.80)
AY(Go) 2000 poin, MARR)
DM(CC) 337B tama, soam)
Correcting
10a
10a
10P
10
10
10
10
10
2017
10
10
10
10
10
10
Cosine decay (no tuning)
109
0.143 (-018, -012)
0.37 (A56,
0.428 10.41.0444
(1C)
NYCCe 183D 2102,, 248)
DM(Ce) 53B [400, sosny
10P
10a
107
10'
108
10
10
10
10
17
10
10
10
10
1097
10
10
10th
Optimizer tuning (no decay)
109
0.00594 (-an), Duts
(LA07 0.00)000)
à 0.503 (0.00..0
(M(C3)
15.51 (200 mad)
A"(Ce) soa)
XM(C2) (1.
10*
10P
10a
10P
109 (1fo
10:5
10
1018
10
1017
10
10
10lo
10
10
109
Compute C [FLOPs]
Compute C [FLOPs]
Compute C' [FLOPs]
951% confidence region
Hoffmann law Kaplan law
Observations
Power law fit
Warmup period
Observations and power law fits of p², N and D* for our main experiments on the RefinedWeb [41] dataset. Here Cc 5.88e23 denotes the compute budget used to train Chinchilla [25].
Figure 13:
22

---

N
1
1010
0.74 (-08),-00 -006
0.994 (0.5)000)) (100)
0L788
(011,
CCC
IL0G (1.00 and)
AM 4T (2T.
(M(CC) 23B
Reproducing Kaplan
108
10
10P
109
10
10
10
10.2
10
10
10
Counting last layer FLOPs
0.4 (-0,45, 0.22)
0,099 (206, (.72)
0.301
10.28, 0.34)
(C)) = 0.19 (um. 0.22)
("(Co) 36BB (4120) ST]
Dh((Ce) 1398 1ee0, 3arop
10*
10a
10a
109
10
16
1015
10
(-0.0, C.12)
(0.5.06))
0.83)
1,25 (058 1.37)
Ah(Go] 2870 (1T1R) 4N0R)
DM((G))
3008 p2MR, MAyy
Correcting warmup
10
10"
10P
10
10
10
00
10
1070
10
10
0.138 (-0.),
0.40)
(C
3.05 (050, 1065)
XNCCo
195B (1161) S200))
DM(Ce) 50GB (ann, sonn)
10a
Cosine decay
10F
10c
109
107
105
10
10
10
19
20th
2017
10
112.5"
10
1037
100
10
Optimizer tuning (no decay)
0.0008 (-0.08,000)
0.509 (0.00
(000e)
1000 (196, aa)
AT(Ce) (AIR) 1118)
109
10a
10a
10a
107
10.
10
10
10
10
(16
10
10
10
10
10
10
1
Compute C [FLOPs]
Compute C [FLOPs]
Compute C' [FLOPs]
95% confidence region
Hoffmann law Kaplan law
Observations
Power law fit
Warmup period
Figure 14: Observations and power law fits of p*, N and D for our main experiments the Open WebText2 [18] dataset. Here Cc 5.88e23 denotes the compute budget used to train Chinchilla [25].
23

---

Table 5: Ablation of the warmup duration. Durations of up 4N achieve very similar and even slightly better results than the duration N we use throughout the paper. The sampling standard deviation is 0.002 across all experiments.
Warmup tokens
Loss
N/4
3.546
N/2
3.542
N (our strategy)
3.532
2N
3.528
4N
3.536
8N
3.543
16N
3.586
Table 6: Ablation of final learning rate value. The loss is almost not impacted when varying this value from 0.1 1 to 10% of the peak learning rate. The sampling standard deviation is 0.002 across all experiments.
Learning rate at the end of the decay
Loss
0.001 peak learning rate
3.444
0.01 peak learning rate (our strategy)
3.442
0.1 x peak learning rate
3.440
Final learning rate value. We vary the final learning rate value from 0.1% 10% of the peak learning rate. Here we train the models with cosine decay and with hyperparameters as described in Table This is the same setting as described in Section 3.4. We tabulate the results in Table 6, and find that the loss is barely impacted by the final learning rate value.
G Fitting hyperparmaeters
This section provides a detailed description of our hyperparameter tuning procedure.
G.1 Full parameter sweep results
We perform extensive parameter sweep over 6 models from our grid (Table 2) with sizes between 5M and 221M parameters. For each model, we sweep over learning rate and batch sizes, as well as three values of [3.. We train each model of size N for 20N tokens (i.e., following the Hoffmann et al. scaling law) and record the validation loss that the end of training. Overall, our hyperparameter sweep includes 642 training runs, and perform it only single dataset (RefinedWeb). Figure 15 plots all the loss values recorded in the sweep. Compared to analogous plots in Wortsman et al. [58], we observe more sensitivity the choice of learning rate, particularly for smaller models. We conjecture that this is because all the models in [58] train for the same amount of tokens, so the smaller models become fully convergence for wide range of learning rates.
G.2 Estimating the optimal batch size and learning rate via interpolation
To estimate the optimal batch size and learning rate for each model size, we adopt a two-stage interpolation approach. In the first stage, for each model size and batch size, we estimate the optimal learning rate by interpolating (in log-space) the loss as a function of learning rate using Akima [3] interpolation, where for every learning rate we assign the lowest loss obtained from the three values of 3z. We minimize the interpolant and save its minimizing argument and minimum value. In the second stage, repeat this procedure over the sequence of batch size and interpolated loss pairs, finding an optimal batch size for each model size. To extract estimate of the optimal learning rate, we simply interpolate the (batch size, minimizing learning rate) sequence and evaluate it at the optimal batch size.

---

53-
Teal 2.0
3.0
50
30
13657
7.00
Leaming
Batch Fize 255
Batch 200 512
11-
0.99
Fin
-0 0- 57.4H
0 126M
0 221M
10n0
24002
Learning F222
402 m2 220
Figure 15: Full hyperparameter sweep results, plotting validation loss after 20 training steps as function of the learning for different model sizes N, values of (3 and batch sizes. Plot design inspired by Wortsman et al. [58].
Batch size
Learning rate
0.024
Grid points, F3 0.95
512
Interpolated optimal Points used for fit
256
Fit:
BS 0 D0088N (A2 -09900 0.998) LR 0.27N-1228 (R² 0.992)
128
0.006
Rounded BS 160//10866)2n 0.004///18866)11
0.003
DeepSeek 2454
BS 5.5
0.00075
N
Figure 16: reproduction of Figure using only B3 0.95. When only use this value of 82, for small models the optimal batch size saturates, resulting in a less consistent trend. Here the "rounded fit" the one obtained using all values 32, as in Figure
G.3 The necessity of tuning 33
To demonstrate the importance tuning 82, we repeat the analysis described above except while only considering experiments 33 = 0.95. Figure 16 shows the result of this experiment, illustrating that breaks part the clean scaling trend depicted in Figure
G.4 Estimating scaling law with ideal tuning
We determine our learning rate and batch size scaling laws by sweeping over hyperparameters for models of sizes N < 108M with each model trained for 20NN tokens. As discussed in Section 5.2, this is limitation as it potentially "bakes in" preference toward Hoffmann et al. scaling. An ideal tuning strategy would select different hyperparameters for each model size and each compute budget, equivalently each model size and each token-to to-parmetter ratio p.
25

---

Estimated loss with ideal tunling
16-1
10
Estimated L
10-3
SM
4 10
M 15M
57M
10-5
N 108M
220M
5
10
20
107
100 Model size N
Token-to-parameter ratio p
10"
(C(C)
95% confidence region
Kaplan law Observations
Power law fitt 0.483 (0.4 0.50)
Estimated ideal tuning 0.457 0.46)
107
10
10
101s
Compute C [FLOPs]
The estimated excess loss caused by using the hyperparameters in Table instead of the ideal hyperparameters for each model size N and token-to-panee taken-to-parameter ratio p. Light dashed lines show the raw excess loss estimates. To these we apply a median filter of width 2 (in terms of token-top-paraneet
Figure 17: Top-left:
ratio) and plot the results in solid lines. Top-right: The estimated IsoFLOP curves with ideal tuning, obtained by subtracting the excess loss from the actual loss. Bottom: Comparing the compute-optimal model sizes obtained from direct observations (with hyperparameters in Table 4) with our estimate for compute-opting model sizes given ideal tuning per model.
In this section, we use the training loss data from our hyperparameter sweep to approximate such ideal tuning and estimate its effect on the compute-optimal scaling law. We do in three steps.
Estimating suboptimality as function token-to-pann toraaametrrrrrr ratio. We estimate the best hyperparameters for 20 using the same interpolation logic as in Section 3.5 but for the training loss after D = PN tokens. (We do not consider values of below since they are too close the warmup period.) Thus, for every value of N and p. obtain estimate of the loss with optimal hyperparameters, denoted L*. We also use interpolation to estimate the loss under our chosen hyperparameters for each model size (given by Table 4), denoted L. Figure 17 top-left shows the (smoothed) estimated suboptimality of our hyperparmaeters, i.e. L L as a function of p for each value of N in the sweep.
Updating IsoFLOP curves. For all model sizes in Table 4 up to 220M and FLOP values in our grid up to 1.6e18, we estimate the loss attained by an ideal tuning by subtracting the from our observed loss the smoothed sub-optimality as estimated above the corresponding value p. For model sizes below 220M that are not present in the hyperparameter sweep we interpolate smoothed sub-optimality based on neighboring model sizes (while keeping the p fixed). We include all model size/FI..) combinations with token-to-pan parameter ratio in between 2 and 30. To estimate the sub-optimality for token multipliers between 20 and (not present sweep), we extend our smoothed sub-optimality measures symmetrically around p 20. Figure 17 top-right
26

---

Table 7: Validation loss for the 901M parameter model trained on 14B tokens with different learning rates and batch sizes. The prescribed learning rate and batch size are 0.0024 and 640, respectively. The sampling standard deviation is 0.002 across all experiments.
BS
160
320
640
1280
2560
LR
0.0006
2.962
0.0012
2.970
2.947
2.954
0.0024
3.050
2.970
2.943
2.955
3.013
0.0048
2.977
2.964
2.970
0.0096
2.991
shows the original IsoFLOP curves (as in Figure 10) along with our estimated loss with ideal tuning.
Re-fitting the scaling law. To estimate the effect of ideal tuning on our estimate of the compute- optimal exponent we apply our 'bootstrap" fitting procedure described in Appendix D on the updated IsoFLOP curves described above. To the scaling law we only use FL.OP values in (2* 1.25e16)) k= For k > 7 we do not have data to estimate loss under ideal tuning.
Conclusions. The top-left panel of Figure 17 shows that our hyperparameters yield losses within 1e-2 of the optimal value for models sizes above 15M and p values above 10. For smaller models or p values, the potential loss reduction from ideal tuning is greater, as is also evident in the top-right panel of the figure. Nevertheless, the top-right panel also shows that the compute-optime model size (marked by stars the IsoFLOP curves) do not move much due to the loss reduction. The bottom panel further reveals that for most FLOP values the difference between compute-optimal model sizes is within their estimated standard deviations. It also shows that approximating ideal hyperparameter tuning moves the estimate of the compute-optimal exponent by less than 0.032, bringing further away from the Kaplan et al. exponent. Furthermore, since learning rate and batch size sensitivities decrease with model size, we expect ideal tuning to have an even smaller effect at larger compute budgets,
are likely to see even better agreement in if we introduce larger models to ideal tuning estimate (which would require extending the hyperparameter sweep to larger models as well). Indeed, dropping the two smallest FLOP values from the power law fit (that is, using FLOP values (2 1.%.1.25cl 1.25c16) k==)) yields exponent a = 0.489 for the original observations and exponent a = 0.5 for ideal tuning. Overall, we estimate that ideal hyperparameter tuning would produce similar results to our maaling-law-basee hyperparameter choices.
G.5 Validating the hyperparameter scaling laws on a larger compute scale
We extend the training of our largest model (with 901M parameters) to 1<4B RefinedWeb tokens (the compute-optimal value for that model size according to our scaling law), and test whether our predicted batch size and learning rate are optimal at this scale. particular, we compare the learning and batch size prescribed in Table 4 (0.0024 and 640, respectively) to 12 configurations. In 8 of them we very either the learning rate or the batch size, and of them we very both. We evaluate the models on held-out Refined Web data and calculate the validation loss and standard deviation due to sampling as described in Appendix D. The results are shown in Table We find that the prescribed learning rate and batch size are indeed optimal, as they yield the lowest validation loss.
H Reproducing the adjusted Kaplan et al. scaling law
Figure 18 shows that by reintroducing the FLOP count and long warmup issues from Sections 3.1 to 3.3 but training with optimized hyperparameters, we recover the "adjusted" form of the Kaplan et al. [30] compute-optimal scaling law, given by 1.3e9 (C/8.64e19) 0.13, Kaplan et al. [30] derive this scaling law by theoretically compensating for the fact that they used a too-large batch size for smaller models. It is reassuring to observe that when we add back the other issues we have identified but appropriately decrease the batch size via parameter tuning, we obtain very close agreement with this adjusted scaling law.
27

---

0.25 (30)
Ca 020 (0.66,
NYCC) 130B (64SB, SDI)
[M(Ce) 1313 (LU7) 161B)
10p
10"
10
10
10
10
10
10th
10th
16
16
10
10ts
10ah
Compute C' [FLOPs]
Compute C [FLOPs]
Compute C [FLOPs]
95% confidence region Power law fit
Hoffmann law
Observations
Adjusted Kaplan law
Warmup period
The optimal model size N as a function of the compute budget C, for models trained with the tuned hyperparameters of Section 3.5 but with the long warmup and discounting the model head the FLOP counts done in Kaplan al. [30].
Figure 18:
Reproducing Kaplan et al.
Counting last layer FLOPs
Correcting warmup
Estimated optimal loss L (C)
Fit: L= 1.32e3 1-233-C-000 +2.66
Lib 7.7455 - c 2.86
Fit: = 9.63e5-C -0--5-0-
1,353, 3.23
1.00
1
x
4 100
4 10
10
3 10
3 10
10"
10
10.9
10
10
10
10"
10
1
Compute C [FLOPs]
Optimizer tuning (no decay)
Cosine decay
Fit: L= 1.29222. c-030s 2.01
Fit: 5.62e5 C dass 3.08
10
10
- Optimal loss
Saturating power law fit Extrapolation
10
95% confidence region
3 10
x 10"
1017
1017
1048
10tH
Compute [FLOPs]
Compute C [FLOPs]
Figure 19; An expanded version of Figure 4 showing a saturating power law fit for each experiment.
I The compute-optimal loss
Extending Figure 4. In Figure 19 fit saturating power law of the form L(C) F LoC- to each of the compute-optimal loss curves in our experiments. As the figure shows, the fit is predictive only for the experiment where hyperparameters to Hoffmann et al. [25], by minimizing the Huber prediction loss for log L(C') over E, log E) and Log(L.).
tuned. We fit the saturating power law similarly
Results on the Open WebTex22 dataset. We perform the same analysis on the Open WebTeex dataset. In Figure 20 we show the fit of the saturating power law to the compute-optimal loss curves for the OpenWeeTeex
dataset, and in Figure 21 we show the Open WebText2 version of Figure The loss scaling law is predictive for the WeenWeTTex22 dataset as well.
28

---

Reproducing Kaplan et al,
Counting last layer FLOPs
Correcting warmup
Estimated optimal loss L" ((C)
FR: 7.6722 6.5 +212
Fi: 2.06e5 C -26 +2.32
Fit 3.56e5 -0-6555--300 2.75
5 10
x
5 10
x 10
10
10
10"
10
1017
10
10
10
10.8
10
10
10t8
1029
Compute C [FLOPs]
Optimizer tuning (no decay)
Cosine decay (no tuning)
Fit: 1 3.9902 -h- 2.07
Ait - 1.38c5 -1120 2.55
10
5 x 10
Optimal loss
Saturating power law fit Extrapolation
x x000
4 x 10
95% confidence region
x 100
3 10
10
104s
10a7
10
10a
Compute C [FLOPs]
Compute [FLOPS]
Figure 20: Reproduction of Figure 19 on the OpenWebTexts dataset.
100
95% confidence region
0.65
Point estimate
0.60
Nominal value
N' exponent
RMS relative bootstrap error
0.55
10
Cost of reduced sweep
0.50
Cost of full sweep
0.45
0.40
10-2
10
10
20
Compute [FLOPs]
Figure 21: Reproduction of Figure 5 on the Open WebText2 dataset.
29

---

The computational cost of our experiments
We now discuss the calculation of the cost of each of our main experiments, comparing fixed learning rate schedules (constant or fixed-length cosine as in Kaplan et al. [30]) with cosine schedule tailored for each. Each of our experiments consists of directly estimating N (C) for grid of C values of the form C₃ 2 16 FLOPs and going from to K 11. With a cosine learning rate schedule, each value of C' requires distinct training runs, so the cost of the experiment is Ekzo C&Ck, where my is the number of models train for C2 FLOPs between 6 and 7 in our experiments. A constant learning rate schedule offers savings since we can extract performance at different FLOP values from the same run, so the cost of the experiment is Σ m C2 where m/, is the number of models we train for al most Cg FLOPs. At the maximum budget we have m'x = m between 6 and 7, but for all smaller & < K we have my between 0 and 2 (typically 1). Thus, we Lk= myCk, which for doubling grid of C is roughly half the cost. For fair comparison, when empirically summing over the cost of our experiments omit runs where the number of tokens is more than 100 times the model size where loss is more than above the optimal loss for the compute budget since they do not contribute to the analysis and when experimenting with a cosine schedule we were more careful execute them.
K-1 save 2 (my m²)Cx
30

---

NeurIPS Paper Checklist
Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect
paper's contributions and scope?
Answer: [Yes]
We match our claims made with experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Note Section 5.2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
31

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we provide the analysis code and data, as well as the necessary configura- tions to the same experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
32

---

Answer: [Yes]
Justification: Yes, we provide the analysis code and data, with instructions to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https: httppssccpppllccc/ee Seeuuddee/ooeeeeeeeeee
ssi agionolicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, note Appendix for details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we attach error bars and confidence intervals to the results when applicable. We also explain our methods for calculating these (Appendix D).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
33

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide details on the compute resources used in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https : http///eeurpp..eeee httpp///llctthiicccteeeeeeeeeeee
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification: The authors have reviewed the NeurIPS Code of Ethics and believe that the research conducted in the paper conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There are no societal impacts of work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The resources used in the paper are properly credited and the license and terms of use explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We document data (in the form of single pickled dataframe) in readme file included in the supplementary materials.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "v8CXHgR3pQ", "reviewer": "Reviewer_szos", "summary": "- Identifies and eliminates discrepancies between the scaling laws of Kaplan et al and Hoffman et al via three interventions: accounting for un-embed layer FLOPs, correcting warmup hyperparameters, and tuning optimizer hyperparameters.\n- Also derives scaling laws for optimal learning rate and batch size as a function of model size", "questions": "- Why did you opt for the IsoFlop estimation approach (approach 2 in chinchilla) rather than the arguably more direct \"approach 3\" in the chinchilla paper? \n- In Appendix B, you mention that the main non-negligible source of error in approximating FLOPs with 6ND is the attention-related computation. This error can be quite non-trivial for small models with large context lengths, but would be small for large models. So, how would the scaling law exponents change if the FLOP estimator accounts for attention flops?", "limitations": "None", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The discrepancy between the scaling laws of Kaplan et al and Hoffman et al has been a major source of confusion/frustration in scaling law research. This paper conclusively identifies + mitigates this discrepancy through multiple targeted interventions. \n- They also reproduce these results on two datasets, indicating that this discrepancy is not specific to the pretraining dataset. \n- Section 4 provides complementary findings that are helpful for speeding up scaling law experiments: scaling laws for optimal learning rate and batch size, role of AdamW $\\beta_2$, constant learning rate schedule suffices to reproduce Chinchilla scaling laws (albeit with higher loss)", "weaknesses": "The main limitation of this paper (as noted in the discussion) is that the range of models considered is fairly small compared to Hoffman et al. due to compute constraints. Nevertheless, it would be useful to have some sanity checks for checking if the estimated scaling laws for loss, learning rate, and batch size extrapolate reliably to 3B and 7B scale models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful questions and the encouraging comments, recognizing that our work conclusively identifies and mitigates the discrepancy between the two scaling laws, and appreciating the value of findings for speeding up scaling law experiments. Below, we address the questions brought up in the review.\n\n**Question 1: Approach 3 in Hoffmann et al.**  \nIn our opinion, Approach 3 is less direct because it arrives at the scaling law for $N^\\star(C)$ via an assumption about how the loss depends on $N$ and $D$. In contrast, Approach 2 and our approach directly estimate $N^\\star(C)$ and fit a scaling law to it, without requiring additional assumptions on the structure of the loss. Moreover, Besiroglu et al. [7] found that Approach 3 is sensitive to the curve-fitting method which has led to inaccurate conclusions in Hoffmann et al. Therefore we decided not to use Approach 3.\n\n**Question 2: Attention FLOPs**  \nThe scaling law exponents do not change significantly when we account for the attention FLOPS - see Figure 6 in the appendix and discussion in lines 531–551.\n\n**Extending experiments to larger models**  \nTraining a model with $N=3B$ with $\\rho=16.5$ (our compute optimal tokens-to-parameters ratio) would require roughly 9e20 FLOPs, which is ~36 times larger than the largest individual training run in this project. A 7B model would be 200 times more expensive. Since verifying the optimality of hyperparameters requires several such training runs, such experiments are beyond our current resources. Consequently, we conducted a more modest scale-up to ~8e19 FLOPs, which is ~4 times the largest budget in our original FLOP grid. As discussed in our response to Reviewer Kjsg, our predicted learning rate and batch size are near-optimal even at that scale. Furthermore, as Figure 3 in the attached pdf shows, the loss remains fairly predictable as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OB2zyyVaW3", "reviewer": "Reviewer_comx", "summary": "The paper provides an explanation for the discrepancies between the scaling laws of Kaplan et al. and Hoffman et al. The authors start by reproducing the scaling laws of Kaplan at al. They then introduce incremental changes in the methodology: accounting for the last layer computational cost, setting a more reasonable learning rate warmup, and more thorough scale-dependent hyperparameter optimization. After introducing these changes, they recover the scaling laws of Hoffman et al. The paper provides evidence that careful learning rate decay is not necessary to resolve the differences between these two scaling laws. The paper demonstrates the importance of such scale-dependent hyperparameters for obtaining reliable scaling laws with small computational budgets.", "questions": "For Section 3.5, do you warmup the learning rate? How so?\n\nHoffman et al. decay learning rate to 10% of peak, why do you use 1% of peak?", "limitations": "The authors adequately discuss the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "I believe the paper to be novel — I am unaware of prior work aiming to explain the differences between the scaling laws of Kaplan et al. and Hoffman et al., beyond the hypothesis put forth by Hoffman et al. which this paper directly addresses. \n\nThe paper is very clearly written. Its experimental set-up is well-motivated in relation to the prior work of Kaplan et al. and Hoffman et al. The experiments are thorough, and the methodology sound.\n\nBeyond the scientific interest of studying why Kaplan et al. and Hoffman et al. arrived at such different scaling law coefficients, this paper provides valuable insights for researchers studying the scaling of language models, particularly regarding the importance of scale-dependent hyper-parameter optimization for small computational regimes.", "weaknesses": "Hoffman et al. suggests that their fitted scaling law parameters differ from those of Kaplan et al. because Kaplan et al. include very sub-optimally trained models in their analysis, which bias the fitted parameters. That is, for certain (N, D, L) tuples, the loss L is substantially overestimated. They suggest that this is mainly due to the poor learning rate schedule used by Kaplan et al. Contrary to what the paper under review claims at times, I view this paper as supporting the broad hypothesis of Hoffman et al. rather than disproving it. Starting with training choices similar to those of Kaplan et al., the authors are able to reproduce their scaling law. They then show that improving these training choices — via a more reasonable warmup schedule, batch size, learning etc. — yields a scaling law that more closely resembles that of Hoffman et al. That is, the poorer training hyper parameters of Kaplan et al. explain the difference with Hoffman et al. While Hoffman et al. emphasize the learning rate schedule, I don’t read their paper as claiming that learning rate decay is necessary to obtain their scaling law —as the authors claim at times—, but rather that the poor learning rate scheduler used by Kaplan et al. is sufficient to mess up their derived scaling law. To me, the hypothesis of Hoffman et al. is consistent with the existence of different hyper parameter choices for which similar (N, D, L) is attainable with a constant learning rate — they do not claim that is is not possible.\n\nThe authors decouple “warmup” from “decay”, whereas to me the learning rate schedule comprises both. Thus, when Hoffman et al. say that they set “the learning rate schedule to approximately match the number of training tokens”, I imagine that they mean adapting both the warmup and the decay period (e.g., to use heuristics such as warmup for 5% of the total training steps). Unfortunately, Hoffman et al. do not give details on what warmup schedule they used. However, looking at Figure A1, it is clear that the warm-up period that they use is a small fraction of the total number of training steps — unlike Kaplan et al. Broadly, I find it misleading to say “Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential” when Hoffman et al. say “Our work differs from Kaplan et al. (2020) in several important ways. First, the authors use a fixed number of training tokens and learning rate schedule for all models. […] In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results n the best final loss regardless of model size” — notice the difference in language, “learning rate schedule” (which is broadly understood to include also warmup) instead of “learning rate decay”.\n\nI ask the authors to please better scope their claim: “Counter to a hypothesis of Hoffmann et al. [21], we find that careful learning rate decay is not essential for the validity of their scaling law” — Hoffman et al. do not claim necessity, and they refer to “learning rate schedule” rather than only learning rate decay.\n\nLastly, the authors may be underestimating the value of learning rate decay, since they only consider models that are reasonably close to Chinchilla-optimal. When considering, for the same model size, very different number of training tokens, cosine decay may play a more important role, at least if all other hyper parameters are kept fixed. The authors are aware of this and discuss it in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and for finding our paper novel, clearly written, and our experiments well-motivated, sound,  thorough, and providing valuable insights. Below we address the main concern regarding the conjecture in Hoffmann et al., as well as the two additional questions. With these points addressed we hope the reviewer will consider increasing our paper’s score.\n\n**The conjecture in Hoffmann et al.:**  \nWe agree that the term “learning rate schedule” comprises both the warmup and decay components, and that it is plausible to interpret the “Modeling the scaling behavior” paragraph in Hoffmann et al. §2 as conjecturing that both components contribute to the scaling law discrepancy. However, based on Appendix A of Hoffmann et al. as well as the subsequent literature discussing this paper, we believe that Hoffmann et al. emphasize the decay component of the learning rate schedule in their conjecture. \n\n_A closer look at Hoffmann et al. Appendix A:_ In §2, Hoffmann et al. refer to Figure A1 to provide evidence for the claim that “setting the learning rate schedule to approximately match the number of training tokens results in the best final loss regardless of model size.” However, in Figure A1 they only vary the decay component of the learning rate, keeping the warmup. Hence, it appears that — in their conjecture regarding the scaling law discrepancy — Hoffmann et al. equate learning rate schedule with learning rate decay.\n\n_Subsequent literature:_ The following sources treat the conjecture of Hoffmann et al. as pertaining to only learning rate decay, suggesting that the above interpretation of Hoffmann et al.’s conjecture is common in the literature:\n\n* Hu et al. [44, §4.1] write that  \n  > … Hoffmann et al. (2022) make a key observation that setting $T > S$ results in dropped performance while setting $S = T$ results in improved training efficiency, confirming that the learning rate shouldn’t be kept high throughout the training.  \n\n  They do not mention aspects of the learning rate schedule other than the decay period.\n\n* In “Scaling laws in compute-optimal training” (to be cited in the revision) Hägele et al. write that\n  > Importantly, the Chinchilla project (Hoffmann et al., 2022) showed that the cosine schedule achieves optimal loss only when the cycle length matches the training duration, but underestimates the model performance during training.\n\n  The cosine cycle length controls the learning rate decay, not the warmup.\n\n* A LessWrong blog post titled “New Scaling Laws for Large Language Models” discusses the scaling law discrepancy and states that \n\n  > It looks like OpenAI used a single total annealing schedule for all of their runs, even those of different lengths. This shifted the apparent best-possible performance downwards for the networks on a non-ideal annealing schedule. And this lead to a distorted notion of what laws should be.\n\n  Again, no mention of warmup.\n\nWe nevertheless agree that the interpretation of Hoffmann et al.’s conjecture is subtle and requires further justification and clarification, which we are happy to add to the revision. In particular, in line 8 (the abstract) we will write “Counter to a hypothesis implied in Hoffmann et al.” and in line 30 we will remove the word “decay” from “tailoring the learning rate decay schedule”. In addition, we will add an appendix with a detailed discussion about the conjecture of Hoffmann et al., along the lines of our response above.\n\n**Question 1: Warmup in hyperparameter sweep**  \nWe use the same heuristic for choosing warmup duration as discussed in section 3.3, equating the warmup duration (in tokens) with the number of model parameters. We will add this detail to Appendix E.\n\n**Question 2: Cooldown value**  \nWe decayed the learning to 0.01 of its peak value following Gadre et al. [14], which was the source of most of the initial hyperparameters used in our experiments. Following the reviewer’s question, we ran a small-scale experiment to test the potential effect of using a different final learning rate. In particular, we train a model with 108M parameters on the RefinedWeb dataset. We use the same hyperparameter as in Table 3, with a training duration of ~2.1B tokens. We use three different values for the final learning rate at the end of the cosine decay - 0.1%, 1%, and 10% of the peak learning rate. Finally, we evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:\n\n|   Learning rate at the end of the run | Validation loss   |\n|-----------:|:------------------|\n|     0.001 x peak learning rate | 3.444 ± 0.002     |\n|      0.01 x peak learning rate  | 3.442 ± 0.002     |\n|      0.1 x peak learning rate   | 3.440 ± 0.002      |\n\nThis experiment suggests that the precise level of final decay has little impact on the quality of the trained model, with no statistically significant difference between decay to 1% and decay to 10%. Therefore, we do not believe the final decay level affects the scaling laws we obtain.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RW6cePuff8", "reviewer": "Reviewer_Kjsg", "summary": "This paper studies the discrepancy between the scaling laws for Kaplan et al. and Hoffmann et al. and identifies three factors behind it: last taking last layer cost into account, warmup duration, and tuning optimizer hyperparams with model scale. They show the transition from Kaplan et al. scaling laws before correcting for these effect to Hoffmann et al. scaling laws after correcting for these effects.", "questions": "1. Is the sweep described in appendix E done with constant learning rate or cosine decay?\n2. While the papers fixing of warmup to be always less than the model size is certainly a step in the right direction as compared to Kaplan et al. is there. a reason to believe that this is optimal? or that the scaling law would not change with change with the warmup duration?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper studies the important question of scaling laws for language models thoroughly (with the limitation of model scale). This fills an important missing piece in the literature and places the scaling law research on a firmer grounding. While running these experiments they also discover additional interesting observations such as the importance of tuning beta2 at smaller batch sizes.", "weaknesses": "The main limitation of this work as also discussed by the authors is the scale of the models use (< 1B).\n\nA more stronger instance of this limitation is that the paper uses experiments on model of size 5M to 100M to fit scaling laws for learning rate and batch size, it is unclear if these laws would hold up for large models. It would have been useful if the paper did an check of this optimality at a significantly larger scale (~1B, not just 200M). This is less of a concern for the runs in the paper since all of them have model sizes <=1B and since Adam is known to be stable (Wortsman et al. 2023) across a range of learning rates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the useful questions and kind words. We were glad to read that you find our paper studies the important question of scaling laws thoroughly and fills an important missing piece in the literature. Below, we address the weaknesses and questions raised in the review.\n\n**Scale of experiments**  \nFollowing the reviewer’s suggestion, we extend the training of our largest model (with 901M parameters) to ~14B RefinedWeb tokens (the compute-optimal value for that model size according to our scaling law), and test whether our predicted batch size and learning rate are optimal at this scale. In particular, we compare the learning and batch size prescribed in Table 4 (0.0024 and 640, respectively) to 8 configurations, each varying either the learning rate or the batch size. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:\n\n|   Learning Rate | Validation Loss   |\n|----------------:|:------------------|\n|          0.0006 | 2.962 ± 0.002     |\n|          0.0012 | 2.947 ± 0.002     |\n|          0.0024 | 2.943 ± 0.002     |\n|          0.0048 | 2.964 ± 0.002     |\n|          0.0096 | 2.991 ± 0.002     |\n\n|   Batch Size | Validation Loss   |\n|-------------:|:------------------|\n|            160 | N/A    |\n|           320 | N/A     |\n|           640 | 2.943 ± 0.002     |\n|           1280 | 2.955 ± 0.002     |\n|           2560 | 3.013 ± 0.002     |\n\nThese results indicate that we accurately predict the optimal hyperparameters at the ~1B scale. The runs with batch sizes 160 and 320 did not complete by the rebuttal deadline, and we will attach them as they arrive; intermediate loss evaluations suggest they will not outperform our predicted optimal batch size. \n\n**Question 1: learning rate scheduler in sweep**  \nWe use a constant learning rate in our sweep experiments, with a warmup duration of N (number of parameters) tokens.\n\n**Question 2: warmup heuristic**  \nTo address the reviewer’s question, we perform a small-scale experiment where we train a $N$=108M parameter model on the RefinedWeb dataset for $20N=$~2.1B tokens, using the hyperparameters in Table 4 (for the appropriate model size) but varying the warmup duration from $N/4$ to $16N$. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in L583. We tabulate the results below:\n\n| Warmup tokens   | Validation loss   |\n|:---------|:------------------|\n|$N/4$    | 3.546 ± 0.002     |\n| $N/2$     | 3.542 ± 0.002     |\n| $N$        | 3.532 ± 0.002     |\n| $2N$       | 3.528 ± 0.002     |\n| $4N$       | 3.536 ± 0.002     |\n| $8N$       | 3.543 ± 0.002     |\n| $16N$      | 3.586 ± 0.002     |\n\nThe results show that a warmup duration of $N$ is nearly optimal, with durations of up to $4N$ achieving very similar and even slightly better results. We conclude that the training loss is fairly insensitive to the precise warmup duration as long as it stays in a reasonable range. Consequently, we believe that our scaling laws are not strongly dependent on the particular choice of warmup duration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ReseL3bkd0", "reviewer": "Reviewer_CKuz", "summary": "This paper aims to resolve the discrepant conclusion about compute-optimal model size drawn from (Kaplan et al., 2020) and (Hoffmann et al., 2022), two famous scaling-law papers that have guided the exploration of early large language models. The authors concluded that the discrepancies arise from last-layer FLOPS and warmup steps. And contrary to the claims of (Hoffmann et al., 2022), learning rate decay is not essential. As a byproduct, the authors also develop strategies to select hyperparameters, including optimal learning rates and batch sizes, and emphasize the need to use larger beta_2 when training with small batch sizes using Adam.", "questions": "1. One of my concerns on scaling laws is that the need for careful hyperparameter selection may impede its applications. Firstly, it makes fitting a law extremely complex, and making mistakes that lead to unreliable predictions is easy. Also, the need to tune hyperparameters increases the costs to fit a law, indicating that we need accurate extrapolation to orders of larger magnitude to pay off the costs. I would like to learn about the authors' comments on potential strategies to stabilize the process and reduce the costs of hyperparameter tuning.\n\n2. Given that many scaling laws exist, and they are discrepant possibly because of different experimental settings or assumptions, could you summarize the condition for the scaling laws in this paper?", "limitations": "This paper only explores the compute-optimal settings and provides limited extrapolation results based on their findings (only one dataset and only scales up to 3e19 flops).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper targets an interesting and important topic on the discrepancy of scaling laws. Given that scaling laws guide the design decision to train modern large language models, solid results on this topic can confirm appropriate scaling strategies and help consolidate the reliability of scaling laws for applications such as large-scale performance prediction.\n2. This paper presents extensive and detailed experiments to show how to align two scaling laws for compute-optimal model sizes.\n2. The discussion on hyperparameters selection, including learning rates, batch sizes, warmup durations, and beta_2 of Adam provide helpful guidance on reproducing scaling law results.", "weaknesses": "1. Discrepancies in scaling laws indicate concerns about the correctness of scaling laws. Most of the paper discusses how to resolve the discrepancies between two scaling laws but has not fully addressed the concerns about the correctness of the two scaling laws as only limited results (only Fig. 4 with one dataset) show resolving the discrepancies generally leads to accurate prediction in large-scale experiments.\n2. The conclusion of this paper is not clear enough. It would be clearer if the authors could conclude a standard pipeline to reproduce the scaling law experiments and enable reliable predictions with this pipeline.\n3. This paper focuses on L(C), the compute optimal scaling laws, instead of L(N, D), which is a more general scaling laws that consider both model sizes and training data. Given that the latest large language models are overtrained to reduce inference costs, limiting to the compute-optimal setting may be less impactful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed review, finding that our paper presents extensive and detailed experiments and provides helpful guidance on reproducing scaling law results. We address the weaknesses and questions below - the main issue appears to be a lack of clarity regarding which scaling law (Kaplan or Hoffmann) is “correct.” We hope our answers clarify this, and we look forward to engaging in constructive discussion about how to introduce these clarifications to the revision best. \n\n**Unclear which scaling law is correct**  \n Conclusively deciding the correctness of compute-optimal scaling laws is a difficult problem due to computational barriers. Nevertheless, our paper adds to a growing body of evidence that Hoffmann et al. (“Chinchilla”) scaling law is more accurate than the Kaplan et al. scaling. Let us explain this in detail:\n\n* The difficulty of validating scaling laws. Directly testing the prediction of a compute-optimal scaling law at the largest compute budget in which one can afford to train is almost impossible by definition, as such verification would require multiple largest-scale training runs. Indeed, neither Kaplan et al. nor Hoffmann et al. directly verify the compute-optimality of extrapolations of their scaling laws.\n\n* Theoretical evidence in favor of the Hoffmann et al. scaling law. The compute-optimal exponent $\\alpha=0.5$ found by Hoffmann et al. has the elegant interpretation that model size and data size should scale with constant proportion. Recent theoretical work has proven the same scaling holds for simple, analytically tractable settings under fairly broad assumptions. The paper “4+3 Phases of Compute-Optimal Neural Scaling Laws” by Paquette et al. shows this for a random-feature linear regression setting, while the paper “Information-Theoretic Foundations for Neural Scaling Laws” by Jeon and Van Roy shows this for data generated by infinite-width 2 layer ReLU networks using information-theoretic arguments. These works hint at (and, in the former case, explicitly conjecture) the possibility of the optimality of proportional scaling being a broader, universal phenomenon. \n\n* Prior empirical evidence in favor of the Hoffmann et al. scaling law. Hoffmann et al. nevertheless provide strong evidence in favor of their scaling law - the Chinchilla 70B model outperformed the larger Gopher model that was trained using similar compute, architecture, and data, as well as other other larger contemporary models. As we discuss in our related work section, in language modeling, subsequent work [12, 22] has produced scaling laws more similar to Hoffmann et al., and other papers rely on the validity of this scaling as a starting point [14, 29]. Therefore, we believe that - for the specific context of next token prediction with Transformer decoder-only models - it is established that the Hoffmann et al. scaling law is the more accurate one; what was missing prior to our work is an explanation for why this is so. \n\n* New evidence in favor of the Hoffmann et al. scaling law from our work. Our paper attributes the difference between the Kaplan and Hoffman scaling laws to what are arguably methodological errors in Kaplan et al.: Incorrect FLOP counting, overly-long warmup period, and suboptimal optimizer configuration. Furthermore, we show that the Hoffmann scaling law extends to non-decayed learning rate schedules. In addition to providing useful lessons for future scaling law studies, our findings further affirm that the Hoffmann et al. scaling is the “more correct” one.\n\n**Unclear conclusion from the paper**  \nWe hope the discussion above helps position our paper in context and thus clarify its conclusion. Moreover, our final reproduction of the Hoffmann scaling constitutes a well-defined pipeline that enables reliable predictions - the training code is based on the open-source open_lm library, and all of the configurations and analysis code are included in our submission.\n\n**Limited impact of compute-optimal setting**  \nWe agree that studying over-trained language and more broadly the individual effect of scaling $N$ and $D$ are important topics that warrant further research. However, note that the very concept of overtraining is defined relative to compute-optimal training. Thus, a better understanding of the compute-optimal setting is an essential stepping stone for broader studies as well. In particular, our conclusions about FLOP counts, warmup, and hyperparameter tuning should be relevant beyond the strictly compute-constrained setting.\n\n**Limited study of predictive accuracy**  \nRegarding the concern that we study predictive accuracy only in one figure (Figure 4) and one dataset, note that Figure 5 also studies the predictive accuracy as we vary the amount of compute used to fit the scaling laws. In the attached pdf we further study predictive accuracy in two ways:\n\n1. We reproduce Figures 4 and 5 for the OpenWebText2 dataset (this involves simply running the analysis code attached to our submission with different parameters).\n\n2. Following suggestions from Reviewers zsos and CKuz, we trained a model for ~8e19 FLOPs (4x our previous largest budget) under our predicted compute-optimal settings and hyperparameters. We show that the resulting loss agrees fairly well with our extrapolation in Figure 4, with further improvements as we fit it on slightly larger budgets of up to 1.6e18 FLOPs. Moreover, we confirm the optimality of our predicted learning rate and batch size - see the response to reviewer CKuz for details.\n\nThus, even though the focus of our paper is explaining the scaling law discrepancy and not measuring predictive accuracy, we study the latter to a greater extent than the influential papers of Kaplan et al. and Hoffmann et al. We nevertheless agree that the two extensions described above will strengthen our paper and will include them in the revision. \n\n(Due to length constraints we answer the two questions in the review in the comment below.)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We attach a file containing the relevant figures for the rebuttal.

Meta-review: This paper examines the discrepancies between the two most influential scaling law papers in the literature (Kaplan et al. and Hoffmann et al.). The authors do a thorough job and manage to reconcile the conflicts in the prior work. All reviewers support acceptance and the AC agrees. This paper will be a good addition to the literature.
Decision: Accept (spotlight)