Title: Resolving Discrepancies in Compute-Optimal Scaling of Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: LLMs

scaling laws

Chinchilla

copmute-optimal

power laws
Abstract: Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $\beta_2$ parameter is essential at lower batch sizes.

Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Abstract

Kaplan et al. [30] and Hoffmann et al. [25] developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan et al. scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., 'Chinchilla') scaling law. Counter to a hypothesis implied in Hoffmann et al. [25], we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW β 2 parameter is essential at lower batch sizes.

1 Introduction

We consider the problem of compute-optimal language model training: given a compute budget C , we wish to predict how to best allocate it across model size (in parameters) and dataset size(in tokens). With pretraining budgets ever-increasing, compute-optimal scaling is a question of paramount importance. In their seminal work, Kaplan et al. [30] proposed a scaling law predicting that the optimal ratio of tokens to parameters decays as a power of C . This scaling law was influential in determining the size of GPT-3 and several subsequent models [12, 51, 43, 32, 47, 62, 52]. However, Hoffmann et al. [25] challenged its validity, arguing instead that the optimal token-to-parameter ratio should be approximately independent of C , and that contemporary models had too many parameters relative to their number of training tokens. Based on this prediction, they trained a 67B parameters model called Chinchilla and which outperformed larger models with a similar compute budget.

While Hoffmann et al. [25] and subsequent work [55, 56, 17, 26, 15] established that following the Hoffmann et al. scaling law leads to better performance than Kaplan et al. scaling, it is still important to understand why the two works arrived at different conclusions. Is the difference due to architecture, training setup, pretraining data, results analysis, or perhaps something else entirely? The answer could teach us important lessons on how to correctly predict and perform model scaling.

Hoffmann et al. [25] hypothesize that the scaling law discrepancy is due to Kaplan et al. [30] not tailoring the learning rate schedule for each token budget separately. While they demonstrate that mismatched learning rate decay results in a higher loss, they do not show it leads to a different compute-optimal scaling law. We further discuss Hoffmann et al. [25]'s hypothesis in Appendix A. To the best of our knowledge, this hypothesis is the only explanation offered in the literature so far.

Our contribution. In this work, we uncover three factors contributing to the discrepancy, and disprove Hoffman et al.'s hypothesis about the role of learning rate decay; Figure 1 illustrates our main

10 17

10 19

Compute

C

[FLOPs]

10 6

10 7

10 8

N

(

C

)

a

=

0

.

835 (0

.

82

,

0

.

85)

N

/star

(

C

C

)

=

3T

(

3T

,

4T

)

a

Reproducing Kaplan et al.

10 17

10 19

10 7

10 8

a

=

0

.

706 (0

.

69

,

0

.

72)

N

/star

(

C

C

)

=

787B

(

630B

,

916B

)

b

Counting last layer FLOPs

10 17

10 19

10 7

10 8

a

=

0

.

602 (0

.

59

,

0

.

62)

N

/star

(

C

C

)

=

292B

(

249B

,

355B

)

c

Correcting warmup

95% confidence region

Power law fit

Observations

10 17

10 19

Compute

C

[FLOPs]

10 7

10 8

N

(

C

)

a

=

0

.

497 (0

.

49

,

0

.

50)

N

/star

(

C

C

)

=

77B

(

70B

,

86B

)

e

Optimizer tuning (no decay)

10 17

10 19

Compute

C

[FLOPs]

10 7

10 8

a

=

0

.

571 (0

.

56

,

0

.

59)

N

/star

(

C

C

)

=

183B

(

152B

,

240B

)

d

Cosine decay (no tuning)

results. We begin by reproducing the Kaplan et al. scaling law in a Llama-derived pretraining setup using the OpenLM library [21] and the RefinedWeb dataset [41] (Figure 1a). Our first observation is that accounting for the computational cost of the decoding layer (as done in Hoffmann et al. [25] but not in Kaplan et al. [30]) shifts compute-optimal scaling toward a more constant token-to-parameter ratio (Figure 1b). Second, we note that the constant-length warmup period of Kaplan et al. [30] is too long for smaller models, inflating the optimal number of tokens at lower compute budgets; scaling the warmup period with the model size further shifts the scaling in the Hoffmann et al. direction (Figure 1c). Next, we match the learning rate decay to the token budget of each configuration we test (as Hoffmann et al. [25] conjecture to be essential) but observe little effect on the computeoptimal scaling law (Figure 1d). Finally, we set the learning rate, batch size, and the AdamW β 2 parameters individually for each model size, leading to compute-optimal scaling that agrees closely with Hoffmann et al. (Figure 1e). Notably, the latter configuration uses a constant learning rate schedule, showing that learning rate decay is not essential for the Hoffmann et al. scaling law to emerge. We repeat our experiment on the OpenWebText2 dataset [18], observing similar results despite performing hyperparameter tuning only on RefinedWeb.

We complement our main results with the following analyses:

In the last phase of our experiments (Figure 1e) we choose different hyperparameters for each model size. To do so, we conduct a hyperparameter sweep for small-scale models and use the results to fit power laws for the optimal batch size and learning rate as a function of model parameters. This approach is inspired by DeepSeek [15], and our hyperparameter scaling laws roughly agree. However, we observe that setting the AdamW β 2 parameter to be 0 . 95 is suboptimal at smaller batch sizes (128 and below), and increasing it allows establishing clear trends from our small-scale hyperparameter sweep.

We study the scaling of the optimal loss as a function of the compute budget. We show that the steps we take to settle the Kaplan et al./Hoffmann et al. discrepancy (namely shortening warmup and scaling learning rate and batch size) significantly decrease this loss at smaller scales, but only marginally improve it at larger scales. In contrast, introducing a cosine learning rate decay

schedule substantially decreases the loss, with benefits persisting at larger scales. Similar to Hoffmann et al. [25], we observe some curvature on the optimal loss curve. Nevertheless, the optimal loss with tuned hyperparameters is fairly consistent with a saturating power law.

We calculate the computational cost of each of our experiments and plot how prediction quality improves as we consider larger training runs. We observe that the cost of our hyperparameter sweep is comparable to that of a scaling law fit experiment, but the compute saved by using a constant instead of a cosine learning rate schedule roughly makes up for that cost.

Code and data release. To facilitate future research, we share the data and the code necessary to reproduce our analyses and figures at https://github.com/formll/ resolving-scaling-law-discrepancies . In addition, checkpoints of the models we train are available at https://huggingface.co/formll/resolving-scaling-law-discrepancies .

2 Preliminaries and experiment design

2.1 Notation and problem setting

We train language models of size N on D tokens of data (essentially without repetition). The precise definition of N plays an important role in this paper: Unless mentioned otherwise, N denotes the number of parameters in all the linear layers of the model. That is, N excludes embedding layers, but includes the model's head : the final linear layer producing the predicted token logits. (In the models we train there is no tying of the embeddings and the head).

Let FLOPs ( N,D ) be the ammount of floating point operations (FLOPs) required to train a model of size N on D tokens. Throughout, we employ the approximation

In Sections 3.1 and 3.2 we compare our definition of N to the one used in Kaplan et al. [30]. In Appendix B we also discuss the effect of taking attention FLOPs into account and FLOP estimation approaches in other works.

Let L ( N,D ) be the log loss (in expectation over the training data distribution and any randomness of the training procedure) obtained by a model of size N trained for D tokens. 4 Assuming a fixed compute budget C , we aim to predict

i.e., the model size yielding the smallest loss when trained with compute budget C under the approximation (1).

We also let

denote the optimal number of tokens and the optimal token-to-parameter ratio. To predict these quantities, we use power laws of the form:

and fit the exponents a, b, r and coefficients where N ⋆ 0 , D ⋆ 0 , ρ ⋆ 0 from data as described below.

2.2 Training setup

We train decoder-only Transformer language models using OpenLM [21], which integrates many of the architecture and training advances in Llama [55, 56] and subsequent works. We largely base our initial training configuration on the hyperparameter search in Gadre et al. [17]. Our setup does not replicate Kaplan et al. [30], but we match or closely approximate several key hyperparameters as discussed in Section 3. See Appendix C for a detailed description of our setup and chosen hyperparameters.

Model set. We search for compute-optimal models over a set consisting of 16 models with sizes ranging from 5 Mto 901 M. We pick model layer numbers l and widths d such that N increases by multiples of roughly √ 2 while the aspect ratio d/l stays between 32 and 64 as suggested in Kaplan et al. [30]. The number of attention heads in each configuration is 4 , as preliminary experiments showed this is optimal for smaller models, and increasing it did not noticeably improve larger models. Table 2 in the appendix specifies all the models in our grid.

Data. We perform our experiments on OpenWebText2 [18] which contains roughly 30 B tokens of data from Reddit and resembles the WebText2 dataset used in Kaplan et al. [30], as well a RefinedWeb [41] dataset which contains roughly 600 B tokens from CommonCrawl [1] and resembles the MassiveWeb dataset that formed roughly half of the data mix in Hoffmann et al. [25].

Evaluation and FLOP grid. We evaluate models on 160 Mtokens held out from the training data. We perform the evaluation whenever the product of 6 N and the number D of training tokens seen so far crosses an element of a FLOP grid of the form { 1 . 25e16 · 2 i } 11 i =0 . This grid plays a central role in our data analysis. We also record the average training loss every 20 steps.

2.3 Data analysis

Our technique for estimating the compute-optimal power law is akin to the second (IsoFLOP-based) approach of Hoffmann et al. [25], but differs in several details. The approach consists of two steps: directly estimating N ⋆ ( C i ) for all C i in our FLOPs grid, and fitting a power law to these estimates. We briefly outline each step below and provide full details in Appendix D.

Estimating N ⋆ ( C i ) . For each value of C i , we train several models from our set (Table 2) for C i FLOPs and extract an IsoFLOP curve of loss vs. model size (see Figure 10). For FLOP values where validation loss is not available (specifically Section 3.1 and Appendix B) we use the smoothed training loss instead. We estimate N ⋆ ( C i ) and its uncertainty using a noise-and-interpolate procedure based on Gaussian noise with empirically-calibrated magnitude and Akima interpoaltion [3]. For every C i , this yields a 'bootstrap sample' population optimal size estimates; we take their median as the point estimate for N ⋆ ( C i ) . The procedure also yields an estimate of the log-scale standard deviation of N ⋆ ( C i ) (shown as error bars in Figure 1).

Fitting a power law. We fit power laws of the form (4) by performing weighted linear regression in log space, with the weights inversely proportional to the squared log-space standard deviations computed above (i.e., log-space Gaussian maximum likelihood estimation). To obtain a point estimate for the power law parameters we fit the point estimates for each N ⋆ ( C i ) value. To quantify uncertainty, we fit power laws to bootstrap samples, obtaining a population of N ⋆ 0 , a , and N ⋆ ( · ) samples. We construct confidence intervals from their quantiles.

3 Main results: settling the scaling law discrepancy

In this section, we describe in detail our main results, visualized in Figure 1, tabulated in Table 1 and plotted in detail in Appendix E. The following subsections address each panel of Figure 1 in order.

3.1 Reproducing the Kaplan et al. scaling law

To reproduce the Kaplan et al. scaling law, we match the setup of [30] in terms of the batch size ( 2 19 tokens) and in terms of the learning rate schedule (warmup for 3000 · 2 19 ≈ 1 . 57 B tokens followed by cosine decay to zero at 2 . 5e5 · 2 19 ≈ 131 B tokens). Other configurations do not match exactly, but the suite of models we train covers a range of sizes and compute similar to Kaplan et al. [30]. For this reproduction only, we also take the 'model size' N to be the number of parameters in all linear layers except the head (last decoding layer). That is, for a model of width d and vocabulary size v , we subtract d · v from our usual definition of N (see Table 2, last column).

As Figure 1a shows, with this setting we obtain a compute-optimal exponent a and power law fits close to the power law 1 . 6e9( C/ 8 . 64e19) 0 . 88 obtained by Kaplan et al. [30, Figure 14, left].

3.2 Counting last layer FLOPs

Kaplan et al. [30] chose to define model size without counting embedding parameters since they found this makes scaling laws in the infinite-compute regime more consistent across network depths [30, Figure 6]. Perhaps because their model head and embeddings had tied weights, this led them to also discount the contribution of the model head to the model's FLOPs per token [30, Table 1, last row]. However, as Table 2 reveals, not accounting for the model head leads to under-approximation that grows smoothly as model size decreases, from roughly 10% at larger models to roughly 90% at smaller models. Thus, counting the head FLOPs (i.e., using our definition of N ) results in a significantly more accurate approximation. As shown in Figure 1b, switching to our model size count also reduces the exponent a by more than 0 . 1 , closer to Hoffmann et al. but not all the way there.

3.3 Correcting learning rate warmup

Next, we address the duration of the learning rate warmup period, which Kaplan et al. [30] set proportionally to their full training duration, designed to reach complete convergence. Figure 2 (left) shows this warmup period is too long: for smaller-scale models, the optimal number of tokens as a function of compute is less than or close to the number of warmup tokens, and therefore these models are suboptimally trained. The same issue is evident in Figure 14 (right) of Kaplan et al. [30] which shows that for many compute budgets the optimal number of steps is below or close to the number of warmup steps (fixed at 3000). Figure 2 (left) also provides an intuitive explanation for the increased value of a : at smaller compute scales, models are 'forced' to use more training tokens than would otherwise be optimal in order to 'escape' the long warmup period. Having escaped, the warmup Once this warmup period is escaped, the optimal number of tokens grows only slowly, leading to a fast rate of increase in the optimal model size and hence the large exponent.

With the problem identified, we propose a simple heuristic for more appropriately choosing the warmup duration: for each model, we set the number of warmup tokens to be identical to the model size N . We validate our warmup heuristic in Appendix F. The bottom row of Figure 2b illustrates the validity of our new choice of warmup, showing that the optimal number of tokens is always at least 5 times greater than the (interpolated) duration of the warmup period corresponding to the model of the appropriate size. As is evident from this figure and from Figure 1c, shortening the warmup shifts the scaling law in the direction of Hoffmann et al. further, yielding an exponent a of roughly 0.6.

3.4 Learning rate decay has limited impact on compute-optimal allocation

With learning rate warmup corrected, we turn to study learning rate decay, which Hoffmann et al. [25] conjecture to be a main cause of the difference between their result and Kaplan et al. [30]. We

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

10 7

10 8

10 9

10 10

D

(

C

)

Kaplan et al. warmup

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

10 7

10 8

10 9

10 10

Our warmup

95% confidence region

Power law fit

Warmup period

Observations

observe that the long 131 B tokens decay period in Kaplan et al. [30], which is aimed toward training to full convergence, means that their compute-constrained experiments see virtually no learning rate decay: Figure 2 shows that, at our compute scales, it is never optimal to train for more than 10 B, which corresponds to less than 1 . 5% decay with a cosine schedule.

To correct this, we follow the second approach of Hoffmann et al. [25] and choose the learning rate schedule for every model and FLOP budget individually. For each FLOP value in our grid, we pick the 7 models from Table 2 which yield token-to-parameter ratios in the range 1 to 100 , and train them with a cosine learning rate schedule that decays to 1% of the maximum learning rate when reaching the target FLOP value. 5 This is roughly twice as expensive as previous experiments, which required only a single training run for each model size (see additional discussion in Section 4.2). As Figure 1d shows, adding cosine decay results in a slightly cleaner linear trend ( R 2 improves from 0 . 993 to 0 . 998 ) and an exponent slightly closer to the Hoffmann et al. scaling law ( 0 . 57 instead of 0 . 6 ), but most of the gap remains. Therefore, even with the FLOP count and warmup issues corrected, adding learning rate decay is not sufficient to reproduce the Hoffmann et al. scaling law.

3.5 Correcting batch size, learning rate and β 2

Afinal factor contributing to the Kaplan et al./Hoffmann et al. discrepancy is the choice of optimization hyperparameters, particularly the batch size: with a fixed batch size of 2 19 tokens, compute-optimal models at smaller scales train for only a few hundred steps, which is likely too little. Kaplan et al. [30] notice this issue, and attempt to correct for it using post-processing based on an empirical model of large-batch size training [36]; we return to their result at the end of this section.

Here, we take the more direct approach of predicting near-optimal hyperparameters for each model size. 6 Since changing the batch size often also requires re-tuning the learning rate [19, 36, 49, 61], we sweep over both parameters for models of sizes 5 Mto 108 M, with an additional validation sweep over models of size 220 M. Initially, we kept β 2 at its previous value of 0 . 95 . However, this led to poor results at smaller batch sizes: as the batch size gets smaller, the squared gradients become noisier, and AdamW requires more smoothing to obtain a correct denominator. Therefore, we added 0 . 99 and 0 . 999 to the sweep, obtaining improved performance on small batch sizes. This empirical observation matches the theoretical work by Zhang et al. [63], that showed that when the batch size is small, higher values of β 2 are crucial for the convergence of Adam. In Appendix G we describe the parameter sweep in full, validate the optimality of our prescribed hyperparameters on the largest model we train, and provide additional discussion about the role of β 2 .

Figure 3 plots our estimates for the optimal values of batch size and learning rate for each model size. It shows clear trends, to which we fit power laws in the number of parameters N . Observing good

5M7M

15M 28M

57M 108M 220M

901M

N

64

32

16

128

8

256

512

Batch size

5M7M

15M 28M

57M 108M 220M

901M

N

0.024

0.00075

0.0015

0.003

0.006

0.012

0.048

Learning rate

Grid points,

2 =0.95

Grid points,

2 =0.99

Grid points,

2 =0.999

Interpolated optimal values

Points used for fit

Fit:

BS = 0.00037

N

0.703

(

R

2 =0.986)

LR = 3.7

N

0.36

(

R

2 =0.997)

Rounded fit:

BS =  160(

N

/108e6) 2/3

LR =  0.0047(

N

/108e6)

1/3

DeepSeek fit:

BS = 0.00068

N

0.654

LR  = 0.17

N

0.25

extrapolation to nearby values of N , we apply these power laws (with slight rounding) to select the batch size and learning rate for all model sizes and tabulate the results in Table 4. In Appendix G.5 we validate our learning rate and batch size scaling law for a model with 901 Mparameters.

Our parameter tuning approach is inspired by DeepSeek [15], who predict the optimal batch size and learning rate as a function of compute. Translating compute to model size using the Hoffmann et al. scaling law, we find remarkable agreement in the batch size predictions (a difference of less than 0 . 05 in exponent and less than 60% in predictions over our models), and somewhat different learning rate predictions (a difference of 0 . 11 exponent and a factor of 2 -3 in predictions), potentially due to using different weight decay. Both our results appear to contradict the conventional wisdom about the existence of a critical batch size [49, 61, 36] below which every batch size is good, finding instead an optimal batch size below which performance degrades. This suggests further tuning of β 2 or other hyperparameters may be warranted. We discuss DeepSeek [15] further in Section 5.1.

With the new hyperparameters, we obtain a close reproduction of the Hoffmann et al. scaling law (Figure 1e) with the scaling exponent matching 0 . 5 to within 0 . 6 %and the predicted model size at Chinchilla compute within 15 %of Chinchilla's size. Notably, here we use a constant learning rate schedule , demonstrating that careful learning rate decay is not necessary for this scaling law to hold.

Finally, we reproduce the adjusted scaling law N ⋆ ( C ) = 1 . 3e9( C/ 8 . 64e19) 0 . 73 which Kaplan et al. [30] obtain by estimating the compute required to reach the same results at a sufficiently low batch size. To do so, we use our tuned hyperparameters as a proxy for suitable batch size and revert our previous corrections (head FLOP count and warmup duration). We obtain an exponent of 0 . 717 and good agreement with their adjusted scaling law; see Figure 18 in Appendix H.

4 Additional Analysis

4.1 Trends in compute-optimal loss

Figure 4 shows the minimum loss achievable for each compute budget C in the experiments shown in Figure 1. We estimate the minimum loss using the same interpolation procedure we use to extract the optimal parameter number N ⋆ and token count D ⋆ . The figure shows that, at low compute scales, shortening the warmup duration and tuning hyperparameters leads to substantial loss improvements (each by up to 0.5 nat per token). However, at larger scales these interventions do not significantly improve the loss. In contrast, learning rate decay becomes increasingly beneficial as compute grows, and appears to also improve the rate of decrease in the loss. Perhaps coincidentally, the effects of overestimating the optimal loss (due to long warmup and large batch size) seem to closely offset the effect of underestimating computational cost (by discounting the contribution from the model's head): the first and last curves in Figure 4 closely overlap.

Similarly to Hoffmann et al. [25] we observe a curvature in the optimal loss, while Kaplan et al. [30] report a near-perfect power law behavior. This difference is due to a combination of the difference in FLOP counts discussed in Section 3.2 and the fact that the experiments of Hoffmann et al. [25]

)

C

(

L

Estimated optimal loss

5×10 0

4×10 0

3×10 0

10 16

10 17

10 18

Compute

Reproducing Kaplan et al.

Counting last layer FLOPs

Correcting warmup

Cosine decay

Optimizer tuning (no decay)

Fit,

L

=128.93

C

0.106 +2.01

Extrapolation

10 19

C

[FLOPs]

10 18

10 19

10 20

Compute

C

[FLOPs]

0.44

0.46

0.48

0.50

0.52

0.54

0.56

0.58

N

exponent

a

10

3

10

2

10

1

95% confidence region

Point estimate

Nominal value

RMS relative

bootstrap error

Cost of reduced sweep

Cost of full sweep

10 16

10 17

10 18

10 19

10 20

Compute

C

[FLOPs]

3×10 0

4×10 0

Estimated optimal loss

L

(

C

)

Original fit

Optimizer tuning (no decay)

Fit,

L

=128.93

C

0.106 +2.01

Extrapolation

(

C

+,

L

+)

10 16

10 17

10 18

10 19

10 20

Compute

C

[FLOPs]

3×10 0

4×10 0

One more point for fit

Optimizer tuning (no decay)

Fit,

L

=97.27

C

Extrapolation

(

C

+,

L

+)

10 16

10 17

10 18

10 19

10 20

Compute

C

[FLOPs]

3×10 0

4×10 0

Two more points for fit

Optimizer tuning (no decay)

Fit,

L

=87.16

C

0.092 +1.68

Extrapolation

(

C

+,

L

+)

extend to higher compute budgets where the loss is closer to its irreducible level. Indeed, for the tuned optimizer experiment (Section 3.5) we find that a saturating power law fits the optimal loss and extrapolates well, while extrapolating poorly for other experiments (see Figure 19 in the appendix). This suggests that a predictable trend in L ( N ⋆ ( C ) , D ⋆ ( C )) is an indicator of locallyoptimal hyperparameters. The exponent of our saturating power fit is approximately -0 . 1 , twice as large as the exponent found in Kaplan et al. [30].

Finally, we validate our compute-optimal loss scaling law by training and evaluating a model using a larger compute budget. Specifically, we train a 901 Mparameter model with a compute budget of C + ≈ 8e19 FLOPs, which our scaling law predicts to be compute-optimal, using the batch size and learning prescribed by our hyperparameter scaling laws (see Table 4), reaching a loss value of L + = 2 . 943 . In Figure 6, we add the point ( C + , L + ) to the red curve in Figure 4 and find that it falls within the predicted trend. Notably, L + is obtained with a single training run using our predicted optimal configuration, while at lower compute values we estimate the optimal loss by interpolating an IsoFLOP curve.

4.2 Scaling law accuracy as a function of compute

We now estimate the computational cost of our scaling law experiments, quantifying the effect of the learning rate schedule, and plot how our predictions improve and become more confident with increased computation. We find that the training cost of each experiment that utilized a fixed learning rate schedule was 1 . 54e20 FLOPs, while the experiments that used a varying-length cosine learning rate schedule required 2 . 99e20 FLOPs; essentially double the compute; see Appendix J for more details. We also find that the cost of the hyperparameter sweep described in Section 3.5 was 2 . 04e20 FLOPs-slightly less than the combined cost of two scaling experiments that leveraged it (one on each dataset). Moreover, in hindsight, we could have arrived at similar hyperparameters using only models of size at most 57 Mand a simple heuristic for choosing β 2 based on batch size, which would have cost only 1 . 44e19 FLOPs.

Figure 5 shows the evolution of the predicted compute-optimal model size exponent a , its confidence interval, and a measure of the prediction accuracy as we modulate the experiment's compute budget by truncating our FLOP grid. The figure shows that the prediction becomes steadily more accurate and confident as compute increases. We present these results, as well as the results in Section 4.1, on the OpenWebText2 dataset as well (see Appendix I).

5 Discussion

5.1 Related work

While neural scaling laws precede the advent of large language models [24, 45], breakthroughs in model [57] and data [42, 44] scaling allowed Kaplan et al. [30] to demonstrate the dramatic possibility of unbounded improvement with scale, triggering an explosion in the literature on the topic. Here we focus on the relatively fewer works that tackle optimal resource allocation under a compute constraint.

For language modeling, Hu et al. [26] and DeepSeek [15] repeat subsets of the analyses in Hoffmann et al. [25] and derive compute optimal scaling laws. Employing Approach 3 of Hoffmann et al. [25] (see also [9]), Hu et al. [26] find that, for their models, optimal scaling favors larger token-toparameter ratios than in Hoffmann et al. [25] and in our results. They attribute this difference to modeling improvements since [25] and argue the same holds for Llama 2 [56]. However, our setup incorporates most of the advances in Llama 2 and still produces power laws very close to Hoffmann et al. [25]. Like us, DeepSeek [15] perform hyperparameter tuning and use isoFLOP analysis to determine compute-optimal model sizes on multiple datasets. While they arrive at an exponent on the order of Hoffmann et al. [25] for the main dataset they study, they report a higher exponent a = 0 . 578 for OpenWebText2 (i.e., predicting lower token-parameter-ratio at scale), which they attribute to the superior quality of the dataset. We also study this dataset but arrive much closer to the Hoffmann et al. scaling law. We conjecture the larger exponent might be due to repeating training data, which likely occurred in their experiment given the dataset's limited size and their compute budget. Settling these discrepancies could be a source of further valuable lessons on optimal model scaling.

Recent work also studies compute-bounded scaling laws beyond the compute-optimal regime. Informed by the increasingly common practice of training medium-scale models beyond compute optimality [e.g., 55, 56, 29], Sardana and Frankle [46] account for the expected inference cost of the model, showing that it naturally skews optimal settings toward smaller models. Gadre et al. [17] directly predict the loss and downstream performance for models trained past the point of compute optimality, and Muennighoff et al. [37] model joint compute-data bottlenecks. All three works rely on the Hoffmann et al. law as a reference point, with [17, 37] baking it to their parametric forms.

Compute-optimal scaling is studied beyond the language domain, particularly in vision. Henighan et al. [23] study autoregressive modeling for a variety of tasks and find scaling laws roughly consistent with the Kaplan et al. [30] adjusted scaling law (with exponent a = 0 . 73 ). That work shares the methodological issues described in the top row of Figure 1 (FLOP count and long warmup), but performs hyperparameter tuning for smaller scale models; in Appendix H we reach similar results when doing the same. Zhai et al. [59] characterize the compute-efficient frontier of Vision Transformers (ViTs), while Cherti et al. [13] studies compute constrained scaling of CLIP models. However, they do not offer a power law for scaling model size with compute. Alabdulmohsin et al. [4] tackle model design under an inference compute constraint by fitting multi-term parametric forms to obtain predictions for the optimal ViT shape. Goyal et al. [20] point out an intricate interplay between data filtering and compute constraints. Finally, Bachmann et al. [7] study compute-optimal scaling of MLP's and obtain exponent a = 0 . 35 , suggesting that MLP require much more rapid data growth than more sophisticated architecture. Overall, whether and to what extent does Hoffmann et al. scaling hold in the vision domain remains a compelling open problem.

Recent theoretical work study compute-optimal scaling laws in simplified, analytically tractable settings [11, 33, 38, 28]. In particular, Paquette et al. [38] obtain a power law with exponent a = 0 . 5 (as in Hoffmann et al. [25]) for a random-feature linear regression setting [35, 5], conjecturing that this is a part of a broader, universal phenomenon. Jeon and Van Roy [28] also establish an exponent of 0 . 5 for data generated by infinite-width two-layer ReLU networks, using information-theoretic arguments.

We also remark on two themes of our paper that draw from prior work. The first is the importance of hyperparameter tuning: several works [30, 27, 17, 15] make the case that smooth, predictable

scaling laws emerge when models on all scales are properly tuned. Our work (and particularly Section 4.1) provides another example of this principle and agrees with previous observations that tuning is particularly important at smaller scales. Second, previous studies [59, 8, 15, 26] as well as the concurrent work [22], propose alternative learning rate schedules that address a key shortcoming of cosine decay: the need to commit to a step budget in advance. We consider a constant learning rate that requires no commitment at all. We show this simple choice suffices to reproduce the Hoffmann et al. law and quantify the computational savings compared to a cosine schedule. However, Section 4.1 (and also [25], among others) show that in terms of loss, the constant schedule clearly underperforms the cosine schedule.

Concurrent and independent work. Pearce and Song [40] also study the discrepency between the Kaplan et al. and Hoffmann et al. scaling laws. By re-analyzing data extracted from the Hoffmann et al. [25] experiments by Besiroglu et al. [9], they identify the last layer FLOP count as a cause for the discrepancy. Moreover, they report on a small-scale experimental study (with model sizes up to 5 Mand training tokens number up to 530 M) in which they observe that a non-decaying learning rate schedule is sufficient for reproducing the Hoffmann et al. exponent and that learning rate tuning is necessary. These results independently corroborate part of our observations in Sections 3.2, 3.4 and 3.5. Pearce and Song [40] do not identify the warmup duration issue we describe in Section 3.3. As a consequence, when reproducing the Kaplan et al. exponent they reach a value close to 0.73 rather than the 'raw' value 0.88 reported in Kaplan et al. [30] (see discussion in Section 3.1, Section 3.5, and Appendix H). In addition, our experiments roughly match the Kaplan et al. [30] compute budget, which is about 3 orders of magnitudes larger than budget in Pearce and Song [40], and we perform careful tuning of both the learning rate and the batch size.

5.2 Limitations

Computational scale is a notable limitation, as well as a defining feature, of our results: our experiments are roughly on the scale of those in Kaplan et al. [30] but are substantially smaller than those of Hoffmann et al. [25]. Scaling may effectively mitigate each of the issues we identify: with scale, the contribution of the model head becomes negligible, any (fixed) warmup period eventually becomes reasonably long, and hyperparameter sensitivity decreases, as shown in Figure 4 and Figure 15. Nevertheless, we believe that experimental protocols that induce correct scaling behavior at low computational budgets are crucial for developing the empirical science of machine learning, particularly in academic settings.

Due to limited compute budgets, our hyperparameter sweep only targeted the smaller models in our grid, and furthermore trained each model for only 20 N steps, i.e., the optimal point according to the Hoffmann et al. scaling law. This raises the concern that the hyperparameters we chose unfairly favor models trained for that particular token-to-parameter ratio, and rerunning our experiment with perfect tuning for each model size and each token-to-parameter ratio would have yielded different results. We believe this is unlikely: at small scales (where hyperparameter tuning is crucial) our original set of hyperparameters favored higher token-to-parameter ratios because they still had a sufficient number of steps to train for, and therefore choosing hyperparameters specifically for them is not likely to result in significant gains. In Appendix G.4 we analyze our existing tuning results to estimate the potential gains from perfect tuning, and find that they are likely to have small impact on our conclusions. Moreover, transferring our hyperparameters to another dataset yields similar results.

Finally, a broader limitation of compute-optimal scaling as defined by Kaplan et al. [30], Hoffmann et al. [25] and our work, is that it only concerns the pretraining loss rather than more direct measures of a model's capabilities. Here again, scale is an issue: most zero-shot and in-context capabilities do not emerge at the scales we consider here, and predicting them from small-scale proxies is an important open problem [48, 17]. Instead, it is possible to study downstream performance via fine-tuning, though this may cause the clean scaling patterns seen in pretraining to break down [54], potentially because the fine-tuning procedure is sensitive to the choice of hyperparameters [27].

Acknowledgments

We thank Georgios Smyrnis, Samir Yitzhak Gadre, Achal Dave, and Mehdi Cherti for helpful discussion and assistance with OpenLM and with the JSC cluster.

TP and YC acknowledge support from the Israeli Science Foundation (ISF) grant no. 2486/21 and the Adelis Foundation. MW was supported in part by a Google Fellowship. JJ acknowledges funding by the Federal Ministry of Education and Research of Germany under grant no. 01IS22094B WestAI - AI Service Center West. LW acknowledges funding from Open Philanthropy. We gratefully acknowledge compute budget granted by Gauss Centre for Supercomputing e.V. and by the John von Neumann Institute for Computing (NIC) on the supercomputers JUWELS Booster and JURECA at Jülich Supercomputing Centre (JSC).

References

Common Crawl. https://commoncrawl.org .

New scaling laws for large language models, 2023. URL https://www.lesswrong.com/ posts/midXmMb2Xg37F2Kgn/new-scaling-laws-for-large-language-models . Accessed: 2024-08-19.

H. Akima. A new method of interpolation and smooth curve fitting based on local procedures. Journal of the ACM (JACM) , 1970.

I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting ViT in shape: Scaling laws for compute-optimal model design. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

A. Atanasov, B. Bordelon, S. Sainathan, and C. Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In International Conference on Learning Representations (ICLR) , 2023.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv:1607.06450 , 2016.

G. Bachmann, S. Anagnostidis, and T. Hofmann. Scaling mlps: A tale of inductive bias. Advances in Neural Information Processing Systems (NeurIPS) , 2024.

M. Bellagente, J. Tow, D. Mahan, D. Phung, M. Zhuravinskyi, R. Adithyan, J. Baicoianu, B. Brooks, N. Cooper, A. Datta, et al. Stable lm 2 1.6 b technical report. arXiv:2402.17834 , 2024.

T. Besiroglu, E. Erdil, M. Barnett, and J. You. Chinchilla scaling: A replication attempt. arXiv:2404.10102 , 2024.

S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv:2204.06745 , 2022.

B. Bordelon, A. Atanasov, and C. Pand Pehlevan. A dynamical model of neural scaling laws. In International Conference on Machine Learning (ICML) , 2024.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS) , 2020.

M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on computer vision and pattern recognition (CVPR) , 2023.

A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1-113, 2023.

DeepSeek. Deepseek LLM: Scaling open-source language models with longtermism. arXiv:2401.02954 , 2024.

M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML) , 2023.

S. Y. Gadre, G. Smyrnis, V. Shankar, S. Gururangan, M. Wortsman, R. Shao, J. Mercat, A. Fang, J. Li, S. Keh, R. Xin, M. Nezhurina, I. Vasiljevic, J. Jitsev, A. G. Dimakis, G. Ilharco, S. Song, T. Kollar, Y . Carmon, A. Dave, R. Heckel, N. Muennighoff, and L. Schmidt. Language models scale reliably with over-training and on downstream tasks. arXiv:2403.08540 , 2024.

L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv:2101.00027 , 2020.

P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch SGD: Training imagenet in 1 hour. arXiv:1706.02677 , 2017.

S. Goyal, P. Maini, Z. C. Lipton, A. Raghunathan, and J. Z. Kolter. Scaling laws for data filtering-data curation cannot be compute agnostic. In Conference on computer vision and pattern recognition (CVPR) , 2024.

S. Gururangan, M. Wortsman, S. Y. Gadre, A. Dave, M. Kilian, W. Shi, J. Mercat, G. Smyrnis, G. Ilharco, M. Jordan, R. Heckel, A. Dimakis, A. Farhadi, V. Shankar, and L. Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. https://github.com/ mlfoundations/open_lm .

A. Hägele, E. Bakouch, A. Kosson, L. B. Allal, L. Von Werra, and M. Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv:2405.18392 , 2024.

T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv:2010.14701 , 2020.

J. Hestness, S. Narang, N. Ardalani, G. F. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv:1712.00409 , 2017.

J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

S. Hu, Y. Tu, X. Han, C. He, G. Cui, X. Long, Z. Zheng, Y. Fang, Y. Huang, W. Zhao, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395 , 2024.

M. Ivgi, Y. Carmon, and J. Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022.

H. J. Jeon and B. Van Roy. Information-theoretic foundations for neural scaling laws. arXiv:2407.01456 , 2024.

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, F. B. Diego de las Casas, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7B. arXiv:2310.06825 , 2023.

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv:2001.08361 , 2020.

B. Lefaudeux, F. Massa, D. Liskovich, W. Xiong, V. Caggiano, S. Naren, M. Xu, J. Hu, M. Tintore, S. Zhang, P. Labatut, D. Haziza, L. Wehrstedt, J. Reizenstein, and G. Sizov. xformers: A modular and hackable transformer modelling library. https://github.com/ facebookresearch/xformers , 2022.

O. Lieber, O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation, 2021.

L. Lin, J. Wu, S. M. Kakade, P. L. Bartlett, and J. D. Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv:2406.08466 , 2024.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv:1711.05101 , 2017.

A. Maloney, D. A. Roberts, and J. Sully. A solvable model of neural scaling laws. arXiv:2210.16859 , 2022.

S. McCandlish, J. Kaplan, D. Amodei, and The OpenAI Dota Team. An empirical model of large-batch training. arXiv:1812.06162 , 2018.

N. Muennighoff, A. Rush, B. Barak, T. Le Scao, N. Tazi, A. Piktus, S. Pyysalo, T. Wolf, and C. A. Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems (NeurIPS) , 2024.

E. Paquette, C. Paquette, L. Xiao, and J. Pennington. 4+3 phases of compute-optimal neural scaling laws. arXiv:2405.15074 , 2024.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. 2019.

T. Pearce and J. Song. Reconciling Kaplan and Chinchilla scaling laws. arXiv:2406.12907 , 2024.

G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, H. Alobeidli, A. Cappelli, B. Pannier, E. Almazrouei, and J. Launay. The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only. In Advances in Neural Information Processing Systems (NeurIPS) , 2023.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners, 2019.

J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv:2112.11446 , 2021.

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR) , 21(140):1-67, 2020.

J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. arXiv:1909.12673 , 2019.

N. Sardana and J. Frankle. Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv:2401.00448 , 2023.

T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ili´ c, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, et al. BLOOM: A 176b-parameter open-access multilingual language model. arXiv:2211.05100 , 2022.

R. Schaeffer, B. Miranda, and S. Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems (NeurIPS) , 2024.

C. J. Shallue, J. Lee, J. Antognini, J. Sohl-Dickstein, R. Frostig, and G. E. Dahl. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research (JMLR) , 20(112):1-49, 2019.

N. Shazeer. Glu variants improve transformer. arXiv:2002.05202 , 2020.

M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053 , 2019.

S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al. Using DeepSpeed and Megatron to train Megatron-Turing NLG 530B, a large-scale generative language model. rXiv:2201.11990 , 2022.

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing , 568:127063, 2024.

Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICLR) , 2022.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 , 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V . Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288 , 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. 2017.

M. Wortsman, P. J. Liu, L. Xiao, K. E. Everett, A. A. Alemi, B. Adlam, J. D. Co-Reyes, I. Gur, A. Kumar, R. Novak, J. Pennington, J. Sohl-Dickstein, K. Xu, J. Lee, J. Gilmer, and S. Kornblith. Small-scale proxies for large-scale transformer training instabilities. In International Conference on Learning Representations (ICLR) , 2024.

X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. In Conference on computer vision and pattern recognition (CVPR) , 2022.

B. Zhang, I. Titov, and R. Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019.

G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. In Advances in Neural Information Processing Systems (NeuIPS) , 2019.

S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068 , 2022.

Y. Zhang, C. Chen, N. Shi, R. Sun, and Z.-Q. Luo. Adam can converge without any modification on update rules. Advances in neural information processing systems , 35:28386-28399, 2022.

A Interpreting Hoffmann et al.'s hypothesis

Hoffmann et al. [25] hypothesize that the scaling law discrepancy is due to a difference in learning rate schedules. The term 'learning rate schedule' comprises both the warmup and decay components, and it is plausible to interpret the 'Modeling the scaling behavior' paragraph in Hoffmann et al. [25] §2 as conjecturing that both components contribute to the scaling law discrepancy. However, we believe that Hoffmann et al. emphasize the decay component of the learning rate schedule in their hypothesis due to two main reasons.

Figure A1 in Hoffmann et al. In §2, Hoffmann et al. refer to Figure A1 to provide evidence that the learning rate schedule affects compute optimal scaling. However, in this figure they only vary the decay component of the schedule, keeping the warmup fixed. Hence, it appears that-in the scope of their hypothesis-Hoffmann et al. equate learning rate schedule with learning rate decay.

Subsequent literature. In Hu et al. [26] §4.1, Hägele et al. [22] §1 and in [2] the authors refer to the decay component of the learning rate schedule as the cause of the discrepancy. This suggests that the community has interpreted Hoffmann et al.'s hypothesis as referring to the decay component.

B Estimating FLOPs and accounting for attention

In this section, we compare our definition of model size N to three alternatives and also discuss choices made by related work. Before that, we provide the precise expression for computing N (by our definition) from the model depth l , width d , and vocabulary size v = 50432 . Due to efficient implementation considerations, OpenLM sets the model's feedforward dimension to d FF = 256 ⌊ 255+ ⌊ 8 d/ 3 ⌋ 256 ⌋ . Since each SwiGLU feedforward block has 3 d ff × d parameter matrices, 2

and since each attention block has 4 d parameters in linear layers, our total estimate is:

We begin by considering, instead of the number of weights in linear layers, the total number of non-embedding learnable weights N exact (e.g., including also LayerNorm gains). The fourth column of Table 2 shows that the difference between this number and N is negligible. We also note that embedding layers have a negligible contribution to the model FLOP counts, since they do not require matrix-vector products.

Consequently, the only non-negligible source of error in the approximation FLOPs ( N,D ) = 6 ND is the attention layers. Since in OpenLM the attention dimension is identical to the model width, Kaplan et al. [30, Table 1] shows that the attention operation costs an additional 6 nd FLOPs per token per layer for a forward and backward pass, where n = 2048 is the sequence length. Thus, if we define an effective model size of

we have that 6 N eff D captures the cost of training the model for D tokens, including attention.

We now consider the difference between these approximations and its effect on compute-optimal scaling laws. The fifth column of Table 2 compares N eff to N . It shows that the ratio N eff /N changes smoothly between roughly 1 . 1 to roughly 1 . 2 and back to 1 . 1 as our model sizes grow. We note that had this ratio been completely constant, there would have been no essentially no difference between working with N and working with N eff since a power law in one would correspond directly to a power law in the other. Since in our model grid this ratio is approximately constant, we expect to see limited differences between the scaling laws resulting from each definition. Figure 7 confirms this expectation, showing quantitatively and qualitatively similar results to Figure 1. Consequently, we cannot determine with certainty which definition is more appropriate for predicting computeoptimal model sizes. Nevertheless, we observe our final experiment (with parameter tuning) predicts that the optimal (effective) model size at the Chinchilla/Gopher compute scale to be about 16 B parameters larger than the one size predicted using our standard definition. These predictions are directly comparable since model size and effective model size are essentially identical at these scales. If we take the Hoffmann et al. scaling law as ground truth, then the prediction we get using N eff is a bit worse.

10 16

10 17

10 18

10 19

10 7

10 8

N

(

C

)

a

=

0

.

717 (0

.

70

,

0

.

73)

N

/star

(

C

C

)

=

954B

(

781B

,

1T

)

Counting last layer

and attention FLOPs

10 16

10 17

10 18

10 19

10 7

10 8

a

=

0

.

622 (0

.

61

,

0

.

63)

N

/star

(

C

C

)

=

413B

(

331B

,

477B

)

Correcting warmup

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

10 7

10 8

N

(

C

)

a

=

0

.

577 (0

.

56

,

0

.

59)

N

/star

(

C

C

)

=

228B

(

180B

,

288B

)

Cosine decay

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

10 7

10 8

a

=

0

.

505 (0

.

50

,

0

.

51)

N

/star

(

C

C

)

=

94B

(

84B

,

105B

)

Optimizer tuning (no decay)

95% confidence region

Power law fit

Observations

Finally, we touch on a third measure of model size, which does not count the contribution of the model's head to the FLOP count. That is, we consider

This is the definition that Kaplan et al. [30] use in their experiment, approximating the flop count as 6 N Kaplan D . Their main motivation for this choice is an observation that not counting embedding parameters leads to more predictive scaling laws in the unlimited compute regime. However, as the final column of Table 2 shows, this approximation leads to a large, systematic error in FLOPs counts for smaller models. In Sections 3.1 and 3.2 we show that this is one of the primary factors behind the Kaplan et al./Hoffmann et al. discrepancy.

We conclude this section with an overview of the model size definitions used by related works other than Kaplan et al. [30]. Henighan et al. [23] use N Kaplan as Kaplan et al. [30] and observe high scaling exponents as a result. Hoffmann et al. [25] account for both linear and attention layers in their FLOP computation essentially using N eff in their first two estimation approaches. However, their third approach appears to ignore the attention FLOPs and also count the embeddings parameters, i.e., setting N ′ = N + dv . DeepSeek [15] compare 3 definitions of model size, including N Kaplan , N , and a hybrid of N eff and N Kaplan that takes attention into account and ignores the model head. They report that the latter option gives the best prediction of the compute-optimal loss at large scales. However, we note that both [25] and [15] claim that attention costs double the FLOPs mentioned in Kaplan et al. [30]; we believe this is likely because they do not account for the fact that the attention is causal , meaning it requires only half the FLOPs of an unstructured matrix-vector product. Finally Muennighoff et al. [37], Gadre et al. [17], Hu et al. [26] use N as we do.

C Additional training setup description

Modeling. We train decoder-only Transformer [57] language models for next-token prediction using OpenLM, a Pytorch [39] for efficient training of medium scale language models. We use the library with largely the same configuration as [17], leveraging xFormers [31], bfloat16 automatic mixed precision, (qk)-LayerNorm [6, 16, 58], SwiGLU [50], depth-scaled initialization [60], and rotary positional embeddings [53]. We use the GPT-NeoX-20B tokenizer [10] whose vocabulary size of 50432 closely matches the vocabulary size of Kaplan et al. [30]. We use a sequence length of 2048 which is twice the sequence length used in Kaplan et al. [30], but we attempt to match parameters like batch size and warmup duration in their size in tokens. We do not tie the weights of the embeddings and head layers.

Optimization. Throughout the paper, we use the AdamW optimizer [34] to minimize the standard log loss with an additive z-loss term for stabilization [14] (coefficient 1e -4 ) as an auxillary loss (for our analysis, we record the log loss without the z-loss term in both train and validation). As advocated for in Wortsman et al. [58], we use independent weight decay [34] with parameter 1e -4 , i.e., we set the 'weight decay' parameter in the standard PyTorch AdamW implementation to be 1e -4 /η , where η is the base learning rate. In Table 3 and Table 4 we describe our choice of hyperparameter in our experiments.

Hardware and computational cost. We train our models on a cluster with 40GB A100 GPU's, using between 4-32 GPU's in parallel per training run. We use the OpenLM/PyTorch distributed data parallel implementation as well as gradient checkpointing. According to our logs, the total compute cost of all the experiments going into this paper is 22.3K GPU hours., and the total FLOP count is 3 . 03e21 FLOPs.

Data repetition. The datasets we work with are large enough to allow us to perform all of our training runs without any data repetition. However, due to two software issues, some experiments experienced limited data repetition. In particular, data going into our hyperparameter sweep might

have been repeated up to 10 times. Moreover, on OpenWebText2, some of our larger-scale training runs might have seen data repeated up to 4 times. We believe this had limited to no impact on our results, as the hyperparameter sweep involved fairly small models unlikely to be able to memorize, while [37] show that 4 data repetitions have only a marginal effect on model performance. In our main experiments on the much larger RefinedWeb dataset we have verified that no data repetition occurred.

D Additional data analysis details

This section provides a comprehensive description of our procedure for fitting the power law for N ⋆ ( C ) . Our procedure for the D ⋆ power law is analogous, using the relationship (3).

Training loss smoothing. We smooth the training loss using a variable-length smoothing averaging window. In particular, we estimate the loss at step i as the average of the losses in steps i -⌊ pi ⌋ to

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

6×10 0

Loss

Reproducing Kaplan et al.

10 15

10 16

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

6×10 0

Counting last layer FLOPs

10 15

10 16

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

6×10 0

Correcting warmup

N

5M

7M

9M

15M

22M

28M

37M

57M

84M

108M

149M

220M

347M

455M

611M

901M

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

6×10 0

Loss

Optimizer tuning (no decay)

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

6×10 0

Cosine decay

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Loss

Reproducing Kaplan et al.

10 15

10 16

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Counting last layer FLOPs

10 15

10 16

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Correcting warmup

N

5M

7M

9M

15M

22M

28M

37M

57M

84M

108M

149M

220M

347M

455M

611M

901M

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Loss

Optimizer tuning (no decay)

10 15

10 16

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Cosine decay

i + ⌊ pi ⌋ , for p = 0 . 05 . We also compensate for the lag introduced by logging the averaged training loss every k = 20 by shifting the training loss's index k/ 2 ; this compensation is quite important for matching the validation loss early in the optimization. We have verified that the smoothed training loss matches the validation loss (where it is available) roughly to the validation loss's sampling error.

Fetching the loss at C i FLOPs from a single run. To estimate the loss of a model of size N trained to compute C i we linearly interpolate the validation/training loss (in log-space) at the two steps closest to C i / (6 NB ) , where B is the batch size in tokens. We also require the nearest step to be within 10% of C i / (6 NB ) , and do not return the loss if not such step exists. For most of our experiments, we compute the validation loss precisely at step ⌈ C i / (6 NB ) ⌉ . However, for the

10 6

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

6×10 0

7×10 0

Loss

Reproducing Kaplan et al.

10 7

10 8

10 9

3×10 0

4×10 0

5×10 0

6×10 0

7×10 0

Counting last layer FLOPs

10 7

10 8

10 9

3×10 0

4×10 0

5×10 0

6×10 0

7×10 0

Correcting warmup

Optimal

N

in interpolated curve

Interpolated isoflop curve

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

6×10 0

7×10 0

Loss

Optimizer tuning (no decay)

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

6×10 0

7×10 0

Cosine decay

1.25e+16

2.56e+19

Compute

C

[FLOPs]

experiments in Section 3.1 and Appendix B, which consider alternative definitions of N we do not have validation loss samples, and we use the training loss instead.

Estimating loss noise. Defining the ideal loss as the population log loss in expectation over model training, there are two sources of error in estimating it: finite samples in the validation set, and variation between training seeds. We estimate the former directly by storing the validation loss on 100 subsamples of the holdout data and find the standard deviation to be in the range 0 . 001 -0 . 002 across the different experimental settings. To gauge the error due to seed variance, we train smaller-scale models from our grid on 7 seeds using the tuned hyperparameters for 20 N tokens each. We find a roughly log-linear relationship (see Figure 12) between the (post-warmup) smoothed training loss and the inter-seed standard deviation. For RefinedWeb (Table 4), it appears to saturate around the sampling error, and we heuristically assign standard deviation 0 . 05 to samples with loss > 7 , standard deviation 0 . 002 to samples with loss < 3 , and linearly interpolate the standard deviation in log-space to samples with loss in the range [3 , 7] . For OpenWebText2 we observe significantly more cross-seed variance as well as less stable loss during training (compare the loss curves in Figures 8 and 9), potentially due to a difference in document lengths. Therefore, we set our standard deviation estimate to go from 0 . 1 at loss 6 to 0 . 01 at loss 3 , saturating outside the interval and log-space interpolating inside it.

Estimating N ⋆ ( C i ) and its uncertainty. Given a list of N values and their respective loss samples at compute C i (fetched as described above), we estimate the optimal value N ⋆ ( C i ) using the following bootstrap-like procedure. For each bootstrap sample, we add independent Gaussian noise to each loss, whose standard deviation is determined according to the heuristic formula described above. We then interpolate the curve of loss vs. N using Akima [3] interpolation in log-space and find the value minimizing the interpolant; this forms our population of bootstrap samples for N ⋆ ( C i ) estimate. We estimate their standard deviation in log-space, and take the maximum between that value and one-third of the average log-spacing on the grid (roughly 1 3 log √ 2 ). Occasionally, N ⋆ appears on the edge of the N grid (though we attempt to avoid this in our experiment design). If more than half of the bootstrap samples land at the edge of the grid, we omit the value of C i from the subsequent power law fit. Otherwise, we keep only the samples outside the grid edge, and blow up the standard deviation estimate by the fraction of omitted samples.

10 6

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

Loss

Reproducing Kaplan et al.

10 7

10 8

10 9

3×10 0

4×10 0

5×10 0

Counting last layer FLOPs

10 7

10 8

10 9

3×10 0

4×10 0

5×10 0

Correcting warmup

Optimal

N

in interpolated curve

Interpolated isoflop curve

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

Loss

Optimizer tuning (no decay)

10 7

10 8

10 9

N

3×10 0

4×10 0

5×10 0

Cosine decay

1.25e+16

2.56e+19

Compute

C

[FLOPs]

3×10 0

4×10 0

6×10 0

Loss

10

3

10

2

10

1

Standard deviation of loss

RefinedWeb

3×10 0

4×10 0

6×10 0

Loss

10

3

10

2

10

1

OpenWebText2

N

= 5M

N

= 7M

N

= 9M

N

= 15M

N

= 28M

N

= 57M

Train loss

Validation loss

Estimated validation sampling error

Bootstrap noise standard deviation

E Additional plots for main experiments

In Figure 13 we complement Figure 1 by plotting our observation and power law fits for D ⋆ , ρ ⋆ and N ⋆ for all the experiments described in Figure 1. In Figure 14 we reproduce this figure for the OpenWebText2 dataset, showing consistent qualitative and quantitative results.

F Ablation of warmup duration and final learning rate value

We preform small scale experiments to verify that our heuristic warmup duration (as discussed in Section 3.3) and the learning rate at the end of cosine decay (as discussed in Section 3.4) have little impact on our results. We train a model with 108 Mparameters on the RefinedWeb dataset. We use the same hyperparameter as in Table 4, with a training duration of ∼ 2 . 1 B tokens. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in Appendix D.

Warmup duratiom. We vary the warmup tokens from N/ 4 to 16 N . We tabulate the results in Table 5, and find that N tokens for warmup is nearly optimal.

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Reproducing Kaplan et al.

r

=

-

0

.

668 (

-

0

.

69

,

-

0

.

65)

ρ /star

(

CC

)

=

0

.

01

(

0

.

01

,

0

.

01

)

ρ /star

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

835 (0

.

82

,

0

.

85)

N /star

(

CC

)

=

3T

(

3T

,

4T

)

N /star

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

163 (0

.

15

,

0

.

18)

D /star

(

CC

)

=

32B

(

28B

,

39B

)

D /star

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Counting last layer FLOPs

r

=

-

0

.

402 (

-

0

.

42

,

-

0

.

37)

ρ /star

(

CC

)

=

0

.

18

(

0

.

13

,

0

.

28

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

706 (0

.

69

,

0

.

72)

N /star

(

CC

)

=

787B

(

630B

,

916B

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

294 (0

.

28

,

0

.

31)

D /star

(

CC

)

=

124B

(

109B

,

159B

)

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Correcting warmup

r

=

-

0

.

207 (

-

0

.

24

,

-

0

.

17)

ρ /star

(

CC

)

=

1

.

11

(

0

.

67

,

1

.

85

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

602 (0

.

59

,

0

.

62)

N /star

(

CC

)

=

292B

(

249B

,

355B

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

398 (0

.

38

,

0

.

41)

D /star

(

CC

)

=

337B

(

275B

,

393B

)

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Cosine decay (no tuning)

r

=

-

0

.

143 (

-

0

.

18

,

-

0

.

12)

ρ /star

(

CC

)

=

2

.

93

(

1

.

73

,

4

.

24

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

571 (0

.

56

,

0

.

59)

N /star

(

CC

)

=

183B

(

152B

,

240B

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

428 (0

.

41

,

0

.

44)

D /star

(

CC

)

=

535B

(

410B

,

635B

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

1

10

2

10

3

Optimizer tuning (no decay)

r

=

0

.

00594 (

-

0

.

01

,

0

.

02)

ρ /star

(

CC

)

=

16

.

51

(

13

.

03

,

19

.

94

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

6

10

7

10

8

a

=

0

.

497 (0

.

49

,

0

.

50)

N /star

(

CC

)

=

77B

(

70B

,

86B

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

9

10

10

b

=

0

.

503 (0

.

50

,

0

.

51)

D /star

(

CC

)

=

1T

(

1T

,

1T

)

95\% confidence region

Power law fit

Observations

Warmup period

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Reproducing Kaplan et al.

r

=

-

0

.

74 (

-

0

.

81

,

-

0

.

65)

ρ /star

(

CC

)

=

0

.

005

(

0

.

002

,

0

.

02

)

ρ /star

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

864 (0

.

82

,

0

.

90)

N /star

(

CC

)

=

4T

(

2T

,

7T

)

N /star

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

138 (0

.

11

,

0

.

18)

D /star

(

CC

)

=

23B

(

14B

,

45B

)

D /star

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Counting last layer FLOPs

r

=

-

0

.

4 (

-

0

.

45

,

-

0

.

32)

ρ /star

(

CC

)

=

0

.

19

(

0

.

09

,

0

.

62

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

699 (0

.

66

,

0

.

72)

N /star

(

CC

)

=

695B

(

412B

,

1T

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

301 (0

.

28

,

0

.

34)

D /star

(

CC

)

=

139B

(

98B

,

240B

)

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Correcting warmup

r

=

-

0

.

203 (

-

0

.

26

,

-

0

.

12)

ρ /star

(

CC

)

=

1

.

25

(

0

.

58

,

4

.

37

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

603 (0

.

57

,

0

.

63)

N /star

(

CC

)

=

287B

(

171B

,

435B

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

398 (0

.

37

,

0

.

43)

D /star

(

CC

)

=

350B

(

220B

,

567B

)

10

16

10

17

10

18

10

19

10

1

10

2

10

3

Cosine decay

r

=

-

0

.

138 (

-

0

.

21

,

-

0

.

06)

ρ /star

(

CC

)

=

3

.

05

(

0

.

93

,

10

.

66

)

10

16

10

17

10

18

10

19

10

6

10

7

10

8

a

=

0

.

574 (0

.

54

,

0

.

61)

N /star

(

CC

)

=

195B

(

116B

,

326B

)

10

16

10

17

10

18

10

19

10

9

10

10

b

=

0

.

426 (0

.

40

,

0

.

46)

D /star

(

CC

)

=

505B

(

309B

,

823B

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

1

10

2

10

3

Optimizer tuning (no decay)

r

=

-

0

.

0368 (

-

0

.

08

,

0

.

02)

ρ /star

(

CC

)

=

10

.

03

(

4

.

94

,

21

.

39

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

6

10

7

10

8

a

=

0

.

518 (0

.

49

,

0

.

54)

N /star

(

CC

)

=

99B

(

69B

,

141B

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

9

10

10

b

=

0

.

483 (0

.

46

,

0

.

51)

D /star

(

CC

)

=

1T

(

725B

,

1T

)

95% confidence region

Power law fit

Observations

Warmup period

Final learning rate value. We vary the final learning rate value from 0 . 1% to 10% of the peak learning rate. Here we train the models with cosine decay and with hyperparameters as described in Table 3. This is the same setting as described in Section 3.4. We tabulate the results in Table 6, and find that the loss is barely impacted by the final learning rate value.

G Fitting hyperparmaeters

This section provides a detailed description of our hyperparameter tuning procedure.

G.1 Full parameter sweep results

We perform an extensive parameter sweep over 6 models from our grid (Table 2) with sizes between 5 Mand 221 Mparameters. For each model, we sweep over learning rate and batch sizes, as well as three values of β 2 . We train each model of size N for 20 N tokens (i.e., following the Hoffmann et al. scaling law) and record the validation loss that the end of training. Overall, our hyperparameter sweep includes 642 training runs, and we perform it on only a single dataset (RefinedWeb). Figure 15 plots all the loss values recorded in the sweep. Compared to analogous plots in Wortsman et al. [58], we observe more sensitivity to the choice of learning rate, particularly for smaller models. We conjecture that this is because all the models in [58] train for the same amount of tokens, so the smaller models become fully convergence for a wide range of learning rates.

G.2 Estimating the optimal batch size and learning rate via interpolation

To estimate the optimal batch size and learning rate for each model size, we adopt a two-stage interpolation approach. In the first stage, for each model size and batch size, we estimate the optimal learning rate by interpolating (in log-space) the loss as a function of learning rate using Akima [3] interpolation, where for every learning rate we assign the lowest loss obtained from the three values of β 2 . We minimize the interpolant and save its minimizing argument and minimum value. In the second stage, repeat this procedure over the sequence of batch size and interpolated loss pairs, finding an optimal batch size for each model size. To extract an estimate of the optimal learning rate, we simply interpolate the (batch size, minimizing learning rate) sequence and evaluate it at the optimal batch size.

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

3.5

4.0

4.5

5.0

5.5

6.0

Final Loss

Batch size = 8

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

4.8e-02

3.5

4.0

4.5

5.0

5.5

6.0

Batch size = 16

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

4.8e-02

3.5

4.0

4.5

5.0

5.5

6.0

Batch size = 32

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

4.8e-02

3.5

4.0

4.5

5.0

5.5

6.0

Batch size = 64

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

4.8e-02

Learning rate

3.5

4.0

4.5

5.0

5.5

6.0

Batch size = 128

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

Learning rate

3.5

4.0

4.5

5.0

5.5

6.0

Final Loss

Batch size = 256

7.5e-04

1.5e-03

3.0e-03

6.0e-03

1.2e-02

2.4e-02

Learning rate

3.5

4.0

4.5

5.0

5.5

6.0

Batch size = 512

2 = 0.95

2 = 0.99

2 = 0.999

N

= 5.17M

N

= 7.5M

N

= 15.6M

N

= 28.7M

N

= 57.4M

N

= 108M

N

= 221M

5M7M

15M 28M

57M 108M 220M

901M

N

32

8

16

64

128

256

512

Batch size

5M7M

15M 28M

57M 108M 220M

901M

N

0.003

0.0015

0.006

0.012

0.024

0.00075

Learning rate

Grid points,

2 =0.95

Interpolated optimal values

Points used for fit

Fit:

BS = 0.00088

N

0.662

(

R

2 =0.998)

LR = 0.27

N

0.218

(

R

2 =0.992)

Rounded fit:

BS =  160(

N

/108e6) 2/3

LR =  0.0047(

N

/108e6)

1/3

DeepSeek fit:

BS = 0.00068

N

0.654

LR  = 0.17

N

0.25

G.3 The necessity of tuning β 2

To demonstrate the importance of tuning β 2 , we repeat the analysis described above except while only considering experiments β 2 = 0 . 95 . Figure 16 shows the result of this experiment, illustrating that breaks part of the clean scaling trend depicted in Figure 3.

G.4 Estimating scaling law with ideal tuning

We determine our learning rate and batch size scaling laws by sweeping over hyperparameters for models of sizes N ≤ 108 Mwith each model trained for 20 N tokens. As discussed in Section 5.2, this is a limitation as it potentially 'bakes in' a preference toward Hoffmann et al. scaling. An ideal tuning strategy would select different hyperparameters for each model size and each compute budget, or equivalently each model size and each token-to-parameter ratio ρ .

5

10

15

20

Token-to-parameter ratio

10

5

10

4

10

3

10

2

10

1

Estimated

L

L

N

= 5M

N

= 7M

N

= 15M

N

= 28M

N

= 57M

N

= 108M

N

= 220M

10 7

10 8

Model size

N

4×10 0

5×10 0

L

Estimated loss with ideal tuning

10 16

10 17

10 18

Compute

C

[FLOPs]

10 7

10 8

N

(

C

)

95% confidence region

Observations

Power law fit:

a

= 0.483 (0.47, 0.50)

Estimated ideal tuning

Power law fit:

a

= 0.457 (0.44, 0.46)

In this section, we use the training loss data from our hyperparameter sweep to approximate such ideal tuning and estimate its effect on the compute-optimal scaling law. We do so in three steps.

Estimating suboptimality as a function of token-to-parameter ratio. We estimate the best hyperparameters for ρ < 20 using the same interpolation logic as in Section 3.5 but for the training loss after D = ρN tokens. (We do not consider values of ρ below 2 since they are too close to the warmup period.) Thus, for every value of N and ρ , we obtain an estimate of the loss with optimal hyperparameters, denoted L ⋆ . We also use interpolation to estimate the loss under our chosen hyperparameters for each model size (given by Table 4), denoted L . Figure 17 top-left shows the (smoothed) estimated suboptimality of our hyperparmaeters, i.e. L -L ⋆ as a function of ρ for each value of N in the sweep.

Updating IsoFLOP curves. For all model sizes in Table 4 up to 220 Mand FLOP values in our grid up to 1 . 6e18 , we estimate the loss attained by an ideal tuning by subtracting the from our observed loss the smoothed sub-optimality as estimated above at the corresponding value of ρ . For model sizes below 220 Mthat are not present in the hyperparameter sweep we interpolate the smoothed sub-optimality based on neighboring model sizes (while keeping the ρ fixed). We include all model size/FLOP combinations with token-to-parameter ratio in between 2 and 30 . To estimate the sub-optimality for token multipliers between 20 and 30 (not present in sweep), we extend our smoothed sub-optimality measures symmetrically around ρ = 20 . Figure 17 top-right

shows the original IsoFLOP curves (as in Figure 10) along with our estimated loss with ideal tuning.

Re-fitting the scaling law. To estimate the effect of ideal tuning on our estimate of the computeoptimal exponent a , we apply our 'bootstrap' fitting procedure described in Appendix D on the updated IsoFLOP curves described above. To fit the scaling law we only use FLOP values in { 2 k · 1 . 25e16 } 7 k =0 . For k > 7 we do not have data to estimate loss under ideal tuning.

Conclusions. The top-left panel of Figure 17 shows that our hyperparameters yield losses within 1e -2 of the optimal value for models sizes above 15 Mand ρ values above 10 . For smaller models or ρ values, the potential loss reduction from ideal tuning is greater, as is also evident in the top-right panel of the figure. Nevertheless, the top-right panel also shows that the compute-optimal model size (marked by stars on the IsoFLOP curves) do not move much due to the loss reduction. The bottom panel further reveals that for most FLOP values the difference between compute-optimal model sizes is within their estimated standard deviations. It also shows that approximating ideal hyperparameter tuning moves the estimate of the compute-optimal exponent by less than 0 . 032 , bringing further away from the Kaplan et al. exponent. Furthermore, since learning rate and batch size sensitivities decrease with model size, we expect ideal tuning to have an even smaller effect at larger compute budgets, so we are likely to see even better agreement in a if we introduce larger models to our ideal tuning estimate (which would require extending the hyperparameter sweep to larger models as well). Indeed, dropping the two smallest FLOP values from the power law fit (that is, using FLOP values { 2 k · 1 . 25e16 } 7 k =2 ) yields exponent a = 0 . 489 for the original observations and exponent a = 0 . 5 for ideal tuning. Overall, we estimate that ideal hyperparameter tuning would produce similar results to our scaling-law-based hyperparameter choices.

G.5 Validating the hyperparameter scaling laws on a larger compute scale

We extend the training of our largest model (with 901 Mparameters) to ∼ 14 B RefinedWeb tokens (the compute-optimal value for that model size according to our scaling law), and test whether our predicted batch size and learning rate are optimal at this scale. In particular, we compare the learning and batch size prescribed in Table 4 ( 0 . 0024 and 640 , respectively) to 12 configurations. In 8 of them we very either the learning rate or the batch size, and in 4 of them we very both. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in Appendix D. The results are shown in Table 7. We find that the prescribed learning rate and batch size are indeed optimal, as they yield the lowest validation loss.

H Reproducing the adjusted Kaplan et al. scaling law

Figure 18 shows that by reintroducing the FLOP count and long warmup issues from Sections 3.1 to 3.3 but training with optimized hyperparameters, we recover the 'adjusted' form of the Kaplan et al. [30] compute-optimal scaling law, given by 1 . 3 e 9 · ( C/ 8 . 64 e 19) 0 . 73 . Kaplan et al. [30] derive this scaling law by theoretically compensating for the fact that they used a too-large batch size for smaller models. It is reassuring to observe that when we add back the other issues we have identified but appropriately decrease the batch size via parameter tuning, we obtain very close agreement with this adjusted scaling law.

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

2

r

=

-

0

.

416 (

-

0

.

43

,

-

0

.

40)

ρ /star

(

CC

)

=

0

.

20

(

0

.

16

,

0

.

25

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

7

10

8

a

=

0

.

711 (0

.

70

,

0

.

72)

N /star

(

CC

)

=

730B

(

645B

,

839B

)

10

16

10

17

10

18

10

19

Compute

C

[FLOPs]

10

9

b

=

0

.

289 (0

.

28

,

0

.

30)

D /star

(

CC

)

=

134B

(

117B

,

151B

)

95% confidence region

Power law fit

Observations

Warmup period

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Estimated optimal loss

L

(

C

)

Fit:

L

=1.32e3

C

0.175 +2.66

Reproducing Kaplan et al.

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Fit:

L

=7.74e5

C

0.333 +2.86

Counting last layer FLOPs

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Fit:

L

=9.63e5

C

0.353 +3.23

Correcting warmup

Optimal loss

Saturating power law fit

Extrapolation

95% confidence region

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Fit:

L

=1.29e2

C

0.106 +2.01

Optimizer tuning (no decay)

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Fit:

L

=5.62e5

C

0.336 +3.08

Cosine decay

I The compute-optimal loss

Extending Figure 4. In Figure 19 we fit a saturating power law of the form L ( C ) = E + L 0 C -ℓ to each of the compute-optimal loss curves in our experiments. As the figure shows, the fit is predictive only for the experiment where hyperparameters are tuned. We fit the saturating power law similarly to Hoffmann et al. [25], by minimizing the Huber prediction loss for log L ( C ) over ℓ, log( E ) and log( L 0 ) .

Results on the OpenWebText2 dataset. We perform the same analysis on the OpenWebText2 dataset. In Figure 20 we show the fit of the saturating power law to the compute-optimal loss curves for the OpenWebText2 dataset, and in Figure 21 we show the OpenWebText2 version of Figure 5. The loss scaling law is predictive for the OpenWebText2 dataset as well.

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Estimated optimal loss

L

(

C

)

Fit:

L

=7.67e2

C

0.157 +2.12

Reproducing Kaplan et al.

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Fit:

L

=2.06e5

C

0.296 +2.32

Counting last layer FLOPs

10 17

10 18

10 19

3×10 0

4×10 0

5×10 0

Fit:

L

=3.56e5

C

0.323 +2.75

Correcting warmup

Optimal loss

Saturating power law fit

Extrapolation

95% confidence region

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Fit:

L

=3.99e2

C

0.140 +2.07

Optimizer tuning (no decay)

10 17

10 18

10 19

Compute

C

[FLOPs]

3×10 0

4×10 0

5×10 0

Fit:

L

=1.38e5

C

0.295 +2.55

Cosine decay (no tuning)

10 18

10 19

10 20

Compute

C

[FLOPs]

0.40

0.45

0.50

0.55

0.60

0.65

N

exponent

a

10

2

10

1

10 0

95% confidence region

Point estimate

Nominal value

RMS relative

bootstrap error

Cost of reduced sweep

Cost of full sweep

J The computational cost of our experiments

We now discuss the calculation of the cost of each of our main experiments, comparing fixed learning rate schedules (constant or fixed-length cosine as in Kaplan et al. [30]) with a cosine schedule tailored for each. Each of our experiments consists of directly estimating N ⋆ ( C ) for a grid of C values of the form C k = 2 k · 1 . 25 e 16 FLOPs and k going from 0 to K = 11 . With a cosine learning rate schedule, each value of C requires distinct training runs, so the cost of the experiment is ∑ K k =0 m k C k , where m k is the number of models we train for C k FLOPs-between 6 and 7 in our experiments. A constant learning rate schedule offers savings since we can extract performance at different FLOP values from the same run, so the cost of the experiment is ∑ K k =0 m ′ k C k where m ′ k is the number of models we train for at most C k FLOPs. At the maximum budget we have m ′ K = m K between 6 and 7 , but for all smaller k < K we have m ′ k between 0 and 2 (typically 1 ). Thus, we save ∑ K -1 k =0 ( m k -m ′ k ) C k ≈ ∑ K -1 k =0 m k C k , which for our doubling grid of C is roughly half the cost. For a fair comparison, when empirically summing over the cost of our experiments omit runs where the number of tokens is more than 100 times the model size or where loss is more than 1 nat above the optimal loss for the compute budget since they do not contribute to the analysis and when experimenting with a cosine schedule we were more careful not execute them.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

We match our claims made with experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Note Section 5.2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes, we provide the analysis code and data, as well as the necessary configurations to run the same experiments as in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, we provide the analysis code and data, with instructions to reproduce our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, note Appendix C for details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, we attach error bars and confidence intervals to the results when applicable. We also explain our methods for calculating these (Appendix D).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, we provide details on the compute resources used in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and believe that the research conducted in the paper conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There are no societal impacts of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models that have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The resources used in the paper are properly credited and the license and terms of use are explicitly mentioned and respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We document our data (in the form of a single pickled dataframe) in a readme file included in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to resolve the discrepant conclusion about compute-optimal model size drawn from (Kaplan et al., 2020) and (Hoffmann et al., 2022), two famous scaling-law papers that have guided the exploration of early large language models. The authors concluded that the discrepancies arise from last-layer FLOPS and warmup steps. And contrary to the claims of (Hoffmann et al., 2022), learning rate decay is not essential. As a byproduct, the authors also develop strategies to select hyperparameters, including optimal learning rates and batch sizes, and emphasize the need to use larger beta_2 when training with small batch sizes using Adam.\n\nSTRENGTHS:\n1. This paper targets an interesting and important topic on the discrepancy of scaling laws. Given that scaling laws guide the design decision to train modern large language models, solid results on this topic can confirm appropriate scaling strategies and help consolidate the reliability of scaling laws for applications such as large-scale performance prediction.\n2. This paper presents extensive and detailed experiments to show how to align two scaling laws for compute-optimal model sizes.\n2. The discussion on hyperparameters selection, including learning rates, batch sizes, warmup durations, and beta_2 of Adam provide helpful guidance on reproducing scaling law results.\n\nWEAKNESSES:\n1. Discrepancies in scaling laws indicate concerns about the correctness of scaling laws. Most of the paper discusses how to resolve the discrepancies between two scaling laws but has not fully addressed the concerns about the correctness of the two scaling laws as only limited results (only Fig. 4 with one dataset) show resolving the discrepancies generally leads to accurate prediction in large-scale experiments.\n2. The conclusion of this paper is not clear enough. It would be clearer if the authors could conclude a standard pipeline to reproduce the scaling law experiments and enable reliable predictions with this pipeline.\n3. This paper focuses on L(C), the compute optimal scaling laws, instead of L(N, D), which is a more general scaling laws that consider both model sizes and training data. Given that the latest large language models are overtrained to reduce inference costs, limiting to the compute-optimal setting may be less impactful.\n\nQUESTIONS:\n1. One of my concerns on scaling laws is that the need for careful hyperparameter selection may impede its applications. Firstly, it makes fitting a law extremely complex, and making mistakes that lead to unreliable predictions is easy. Also, the need to tune hyperparameters increases the costs to fit a law, indicating that we need accurate extrapolation to orders of larger magnitude to pay off the costs. I would like to learn about the authors' comments on potential strategies to stabilize the process and reduce the costs of hyperparameter tuning.\n\n2. Given that many scaling laws exist, and they are discrepant possibly because of different experimental settings or assumptions, could you summarize the condition for the scaling laws in this paper?\n\nLIMITATIONS:\nThis paper only explores the compute-optimal settings and provides limited extrapolation results based on their findings (only one dataset and only scales up to 3e19 flops).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the discrepancy between the scaling laws for Kaplan et al. and Hoffmann et al. and identifies three factors behind it: last taking last layer cost into account, warmup duration, and tuning optimizer hyperparams with model scale. They show the transition from Kaplan et al. scaling laws before correcting for these effect to Hoffmann et al. scaling laws after correcting for these effects.\n\nSTRENGTHS:\nThe paper studies the important question of scaling laws for language models thoroughly (with the limitation of model scale). This fills an important missing piece in the literature and places the scaling law research on a firmer grounding. While running these experiments they also discover additional interesting observations such as the importance of tuning beta2 at smaller batch sizes.\n\nWEAKNESSES:\nThe main limitation of this work as also discussed by the authors is the scale of the models use (< 1B).\n\nA more stronger instance of this limitation is that the paper uses experiments on model of size 5M to 100M to fit scaling laws for learning rate and batch size, it is unclear if these laws would hold up for large models. It would have been useful if the paper did an check of this optimality at a significantly larger scale (~1B, not just 200M). This is less of a concern for the runs in the paper since all of them have model sizes <=1B and since Adam is known to be stable (Wortsman et al. 2023) across a range of learning rates.\n\nQUESTIONS:\n1. Is the sweep described in appendix E done with constant learning rate or cosine decay?\n2. While the papers fixing of warmup to be always less than the model size is certainly a step in the right direction as compared to Kaplan et al. is there. a reason to believe that this is optimal? or that the scaling law would not change with change with the warmup duration?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides an explanation for the discrepancies between the scaling laws of Kaplan et al. and Hoffman et al. The authors start by reproducing the scaling laws of Kaplan at al. They then introduce incremental changes in the methodology: accounting for the last layer computational cost, setting a more reasonable learning rate warmup, and more thorough scale-dependent hyperparameter optimization. After introducing these changes, they recover the scaling laws of Hoffman et al. The paper provides evidence that careful learning rate decay is not necessary to resolve the differences between these two scaling laws. The paper demonstrates the importance of such scale-dependent hyperparameters for obtaining reliable scaling laws with small computational budgets.\n\nSTRENGTHS:\nI believe the paper to be novel — I am unaware of prior work aiming to explain the differences between the scaling laws of Kaplan et al. and Hoffman et al., beyond the hypothesis put forth by Hoffman et al. which this paper directly addresses. \n\nThe paper is very clearly written. Its experimental set-up is well-motivated in relation to the prior work of Kaplan et al. and Hoffman et al. The experiments are thorough, and the methodology sound.\n\nBeyond the scientific interest of studying why Kaplan et al. and Hoffman et al. arrived at such different scaling law coefficients, this paper provides valuable insights for researchers studying the scaling of language models, particularly regarding the importance of scale-dependent hyper-parameter optimization for small computational regimes.\n\nWEAKNESSES:\nHoffman et al. suggests that their fitted scaling law parameters differ from those of Kaplan et al. because Kaplan et al. include very sub-optimally trained models in their analysis, which bias the fitted parameters. That is, for certain (N, D, L) tuples, the loss L is substantially overestimated. They suggest that this is mainly due to the poor learning rate schedule used by Kaplan et al. Contrary to what the paper under review claims at times, I view this paper as supporting the broad hypothesis of Hoffman et al. rather than disproving it. Starting with training choices similar to those of Kaplan et al., the authors are able to reproduce their scaling law. They then show that improving these training choices — via a more reasonable warmup schedule, batch size, learning etc. — yields a scaling law that more closely resembles that of Hoffman et al. That is, the poorer training hyper parameters of Kaplan et al. explain the difference with Hoffman et al. While Hoffman et al. emphasize the learning rate schedule, I don’t read their paper as claiming that learning rate decay is necessary to obtain their scaling law —as the authors claim at times—, but rather that the poor learning rate scheduler used by Kaplan et al. is sufficient to mess up their derived scaling law. To me, the hypothesis of Hoffman et al. is consistent with the existence of different hyper parameter choices for which similar (N, D, L) is attainable with a constant learning rate — they do not claim that is is not possible.\n\nThe authors decouple “warmup” from “decay”, whereas to me the learning rate schedule comprises both. Thus, when Hoffman et al. say that they set “the learning rate schedule to approximately match the number of training tokens”, I imagine that they mean adapting both the warmup and the decay period (e.g., to use heuristics such as warmup for 5% of the total training steps). Unfortunately, Hoffman et al. do not give details on what warmup schedule they used. However, looking at Figure A1, it is clear that the warm-up period that they use is a small fraction of the total number of training steps — unlike Kaplan et al. Broadly, I find it misleading to say “Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential” when Hoffman et al. say “Our work differs from Kaplan et al. (2020) in several important ways. First, the authors use a fixed number of training tokens and learning rate schedule for all models. […] In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results n the best final loss regardless of model size” — notice the difference in language, “learning rate schedule” (which is broadly understood to include also warmup) instead of “learning rate decay”.\n\nI ask the authors to please better scope their claim: “Counter to a hypothesis of Hoffmann et al. [21], we find that careful learning rate decay is not essential for the validity of their scaling law” — Hoffman et al. do not claim necessity, and they refer to “learning rate schedule” rather than only learning rate decay.\n\nLastly, the authors may be underestimating the value of learning rate decay, since they only consider models that are reasonably close to Chinchilla-optimal. When considering, for the same model size, very different number of training tokens, cosine decay may play a more important role, at least if all other hyper parameters are kept fixed. The authors are aware of this and discuss it in the limitations section.\n\nQUESTIONS:\nFor Section 3.5, do you warmup the learning rate? How so?\n\nHoffman et al. decay learning rate to 10% of peak, why do you use 1% of peak?\n\nLIMITATIONS:\nThe authors adequately discuss the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Identifies and eliminates discrepancies between the scaling laws of Kaplan et al and Hoffman et al via three interventions: accounting for un-embed layer FLOPs, correcting warmup hyperparameters, and tuning optimizer hyperparameters.\n- Also derives scaling laws for optimal learning rate and batch size as a function of model size\n\nSTRENGTHS:\n- The discrepancy between the scaling laws of Kaplan et al and Hoffman et al has been a major source of confusion/frustration in scaling law research. This paper conclusively identifies + mitigates this discrepancy through multiple targeted interventions. \n- They also reproduce these results on two datasets, indicating that this discrepancy is not specific to the pretraining dataset. \n- Section 4 provides complementary findings that are helpful for speeding up scaling law experiments: scaling laws for optimal learning rate and batch size, role of AdamW $\\beta_2$, constant learning rate schedule suffices to reproduce Chinchilla scaling laws (albeit with higher loss)\n\nWEAKNESSES:\nThe main limitation of this paper (as noted in the discussion) is that the range of models considered is fairly small compared to Hoffman et al. due to compute constraints. Nevertheless, it would be useful to have some sanity checks for checking if the estimated scaling laws for loss, learning rate, and batch size extrapolate reliably to 3B and 7B scale models.\n\nQUESTIONS:\n- Why did you opt for the IsoFlop estimation approach (approach 2 in chinchilla) rather than the arguably more direct \"approach 3\" in the chinchilla paper? \n- In Appendix B, you mention that the main non-negligible source of error in approximating FLOPs with 6ND is the attention-related computation. This error can be quite non-trivial for small models with large context lengths, but would be small for large models. So, how would the scaling law exponents change if the FLOP estimator accounts for attention flops?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}