Resolving Discrepancies in Compute-Optimal Scaling of Language Models

Authors: tomer  porian
mitchell  wortsman
jenia  jitsev
ludwig  schmidt
yair  carmon
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:29:50

Abstract: Kaplan et al. and Hoffmann et al. developed influential scaling laws for the optimal model size as a function of the compute budget, but these laws yield substantially different predictions. We explain the discrepancy by reproducing the Kaplan scaling law on two datasets (OpenWebText2 and RefinedWeb) and identifying three factors causing the difference: last layer computational cost, warmup duration, and scale-dependent optimizer tuning. With these factors corrected, we obtain excellent agreement with the Hoffmann et al. (i.e., "Chinchilla") scaling law. Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential for the validity of their scaling law. As a secondary result, we derive scaling laws for the optimal learning rate and batch size, finding that tuning the AdamW $\beta_2$ parameter is essential at lower batch sizes.

Main paper: 
 introduction: 
 We consider the problem of compute-optimal language model training: given a compute budget C, we wish to predict how to best allocate it across model size (in parameters) and dataset size(in tokens). With pretraining budgets ever-increasing, compute-optimal scaling is a question of paramount importance. In their seminal work, Kaplan et al. [30] proposed a scaling law predicting that the optimal ratio of tokens to parameters decays as a power of C. This scaling law was influential in determining the size of GPT-3 and several subsequent models [12, 51, 43, 32, 47, 62, 52] . However, Hoffmann et al. [25] challenged its validity, arguing instead that the optimal token-to-parameter ratio should be approximately independent of C, and that contemporary models had too many parameters relative to their number of training tokens. Based on this prediction, they trained a 67B parameters model called Chinchilla and which outperformed larger models with a similar compute budget.While Hoffmann et al. [25] and subsequent work [55, 56, 17, 26, 15] established that following the Hoffmann et al. scaling law leads to better performance than Kaplan et al. scaling, it is still important to understand why the two works arrived at different conclusions. Is the difference due to architecture, training setup, pretraining data, results analysis, or perhaps something else entirely? The answer could teach us important lessons on how to correctly predict and perform model scaling.Hoffmann et al. [25] hypothesize that the scaling law discrepancy is due to Kaplan et al. [30] not tailoring the learning rate schedule for each token budget separately. While they demonstrate that mismatched learning rate decay results in a higher loss, they do not show it leads to a different compute-optimal scaling law. We further discuss Hoffmann et al. [25] 's hypothesis in Appendix A. To the best of our knowledge, this hypothesis is the only explanation offered in the literature so far.Our contribution. In this work, we uncover three factors contributing to the discrepancy, and disprove Hoffman et al.'s hypothesis about the role of learning rate decay; Figure 1 illustrates our main * Tel Aviv University; correspondence to tomerpor@gmail.com and ycarmon@tauex.tau.ac.il.† University of Washington.‡ Jülich Supercomputing Centre (JSC) and LAION.38th Conference on Neural Information Processing Systems (NeurIPS 2024). Cosine decay (no tuning) results. We begin by reproducing the Kaplan et al. scaling law in a Llama-derived pretraining setup using the OpenLM library [21] and the RefinedWeb dataset [41] (Figure 1a ). Our first observation is that accounting for the computational cost of the decoding layer (as done in Hoffmann et al. [25] but not in Kaplan et al. [30] ) shifts compute-optimal scaling toward a more constant token-to-parameter ratio (Figure 1b ). Second, we note that the constant-length warmup period of Kaplan et al. [30] is too long for smaller models, inflating the optimal number of tokens at lower compute budgets; scaling the warmup period with the model size further shifts the scaling in the Hoffmann et al. direction (Figure 1c ). Next, we match the learning rate decay to the token budget of each configuration we test (as Hoffmann et al. [25] conjecture to be essential) but observe little effect on the computeoptimal scaling law (Figure 1d ). Finally, we set the learning rate, batch size, and the AdamW β 2 parameters individually for each model size, leading to compute-optimal scaling that agrees closely with Hoffmann et al. (Figure 1e ). Notably, the latter configuration uses a constant learning rate schedule, showing that learning rate decay is not essential for the Hoffmann et al. scaling law to emerge. We repeat our experiment on the OpenWebText2 dataset [18] , observing similar results despite performing hyperparameter tuning only on RefinedWeb.We complement our main results with the following analyses:1. In the last phase of our experiments (Figure 1e ) we choose different hyperparameters for each model size. To do so, we conduct a hyperparameter sweep for small-scale models and use the results to fit power laws for the optimal batch size and learning rate as a function of model parameters. This approach is inspired by DeepSeek [15] , and our hyperparameter scaling laws roughly agree. However, we observe that setting the AdamW β 2 parameter to be 0.95 is suboptimal at smaller batch sizes (128 and below), and increasing it allows establishing clear trends from our small-scale hyperparameter sweep.2. We study the scaling of the optimal loss as a function of the compute budget. We show that the steps we take to settle the Kaplan et al./Hoffmann et al. discrepancy (namely shortening warmup and scaling learning rate and batch size) significantly decrease this loss at smaller scales, but only marginally improve it at larger scales. In contrast, introducing a cosine learning rate decay schedule substantially decreases the loss, with benefits persisting at larger scales. Similar to Hoffmann et al. [25] , we observe some curvature on the optimal loss curve. Nevertheless, the optimal loss with tuned hyperparameters is fairly consistent with a saturating power law. 3. We calculate the computational cost of each of our experiments and plot how prediction quality improves as we consider larger training runs. We observe that the cost of our hyperparameter sweep is comparable to that of a scaling law fit experiment, but the compute saved by using a constant instead of a cosine learning rate schedule roughly makes up for that cost.Code and data release. To facilitate future research, we share the data and the code necessary to reproduce our analyses and figures at https://github.com/formll/ resolving-scaling-law-discrepancies. In addition, checkpoints of the models we train are available at https://huggingface.co/formll/resolving-scaling-law-discrepancies.2 Preliminaries and experiment designQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] We match our claims made with experimental results.• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
notation and problem setting: 
 We train language models of size N on D tokens of data (essentially without repetition). The precise definition of N plays an important role in this paper: Unless mentioned otherwise, N denotes the number of parameters in all the linear layers of the model. That is, N excludes embedding layers, but includes the model's head: the final linear layer producing the predicted token logits. (In the models we train there is no tying of the embeddings and the head).Let FLOPs(N, D) be the ammount of floating point operations (FLOPs) required to train a model of size N on D tokens. Throughout, we employ the approximation FLOPs(N, D) ≈ 6N D (1) In Sections 3.1 and 3.2 we compare our definition of N to the one used in Kaplan et al. [30] . In Appendix B we also discuss the effect of taking attention FLOPs into account and FLOP estimation approaches in other works.Let L(N, D) be the log loss (in expectation over the training data distribution and any randomness of the training procedure) obtained by a model of size N trained for D tokens. 4 Assuming a fixed compute budget C, we aim to predictEQUATIONi.e., the model size yielding the smallest loss when trained with compute budget C under the approximation (1) .We also letEQUATIONdenote the optimal number of tokens and the optimal token-to-parameter ratio. To predict these quantities, we use power laws of the form:EQUATIONand fit the exponents a, b, r and coefficients where N ⋆ 0 , D ⋆ 0 , ρ ⋆ 0 from data as described below.We train decoder-only Transformer language models using OpenLM [21] , which integrates many of the architecture and training advances in Llama [55, 56] and subsequent works. We largely base our initial training configuration on the hyperparameter search in Gadre et al. [17] . Our setup does not replicate Kaplan et al. [30] , but we match or closely approximate several key hyperparameters as discussed in Section 3. See Appendix C for a detailed description of our setup and chosen hyperparameters.Model set. We search for compute-optimal models over a set consisting of 16 models with sizes ranging from 5M to 901M. We pick model layer numbers l and widths d such that N increases by multiples of roughly √ 2 while the aspect ratio d/l stays between 32 and 64 as suggested in Kaplan et al. [30] . The number of attention heads in each configuration is 4, as preliminary experiments showed this is optimal for smaller models, and increasing it did not noticeably improve larger models. Table 2 in the appendix specifies all the models in our grid.Data. We perform our experiments on OpenWebText2 [18] which contains roughly 30B tokens of data from Reddit and resembles the WebText2 dataset used in Kaplan et al. [30] , as well a RefinedWeb [41] dataset which contains roughly 600B tokens from CommonCrawl [1] and resembles the MassiveWeb dataset that formed roughly half of the data mix in Hoffmann et al. [25] .Evaluation and FLOP grid. We evaluate models on 160M tokens held out from the training data. We perform the evaluation whenever the product of 6N and the number D of training tokens seen so far crosses an element of a FLOP grid of the form {1.25e16 • 2 i } 11 i=0 . This grid plays a central role in our data analysis. We also record the average training loss every 20 steps.Our technique for estimating the compute-optimal power law is akin to the second (IsoFLOP-based) approach of Hoffmann et al. [25] , but differs in several details. The approach consists of two steps: directly estimating N ⋆ (C i ) for all C i in our FLOPs grid, and fitting a power law to these estimates. We briefly outline each step below and provide full details in Appendix D.Estimating N ⋆ (C i ). For each value of C i , we train several models from our set (Table 2 ) for C i FLOPs and extract an IsoFLOP curve of loss vs. model size (see Figure 10 ). For FLOP values where validation loss is not available (specifically Section 3.1 and Appendix B) we use the smoothed training loss instead. We estimate N ⋆ (C i ) and its uncertainty using a noise-and-interpolate procedure based on Gaussian noise with empirically-calibrated magnitude and Akima interpoaltion [3] . For every C i , this yields a "bootstrap sample" population optimal size estimates; we take their median as the point estimate for N ⋆ (C i ). The procedure also yields an estimate of the log-scale standard deviation of N ⋆ (C i ) (shown as error bars in Figure 1 ).Fitting a power law. We fit power laws of the form (4) by performing weighted linear regression in log space, with the weights inversely proportional to the squared log-space standard deviations computed above (i.e., log-space Gaussian maximum likelihood estimation). To obtain a point estimate for the power law parameters we fit the point estimates for each N ⋆ (C i ) value. To quantify uncertainty, we fit power laws to bootstrap samples, obtaining a population of N ⋆ 0 , a, and N ⋆ (•) samples. We construct confidence intervals from their quantiles.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Note Section 5.2.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
main results: settling the scaling law discrepancy: 
 In this section, we describe in detail our main results, visualized in Figure 1 , tabulated in Table 1 and plotted in detail in Appendix E. The following subsections address each panel of Figure 1 in order.To reproduce the Kaplan et al. scaling law, we match the setup of [30] in terms of the batch size (2 19 tokens) and in terms of the learning rate schedule (warmup for 3000 • 2 19 ≈ 1.57B tokens followed by cosine decay to zero at 2.5e5 • 2 19 ≈ 131B tokens). Other configurations do not match exactly, but the suite of models we train covers a range of sizes and compute similar to Kaplan et al. [30] . For this reproduction only, we also take the "model size" N to be the number of parameters in all linear layers except the head (last decoding layer). That is, for a model of width d and vocabulary size v, we subtract d • v from our usual definition of N (see Table 2 , last column).As Figure 1a shows, with this setting we obtain a compute-optimal exponent a and power law fits close to the power law 1.6e9(C/8.64e19) 0.88 obtained by Kaplan et al. [30, Figure 14, left] . Kaplan et al. [30] chose to define model size without counting embedding parameters since they found this makes scaling laws in the infinite-compute regime more consistent across network depths [30, Figure 6 ]. Perhaps because their model head and embeddings had tied weights, this led them to also discount the contribution of the model head to the model's FLOPs per token [30, Table 1 , last row]. However, as Table 2 reveals, not accounting for the model head leads to under-approximation that grows smoothly as model size decreases, from roughly 10% at larger models to roughly 90% at smaller models. Thus, counting the head FLOPs (i.e., using our definition of N ) results in a significantly more accurate approximation. As shown in Figure 1b , switching to our model size count also reduces the exponent a by more than 0.1, closer to Hoffmann et al. but not all the way there.Next, we address the duration of the learning rate warmup period, which Kaplan et al. [30] set proportionally to their full training duration, designed to reach complete convergence. Figure 2 (left) shows this warmup period is too long: for smaller-scale models, the optimal number of tokens as a function of compute is less than or close to the number of warmup tokens, and therefore these models are suboptimally trained. The same issue is evident in Figure 14 (right) of Kaplan et al. [30] which shows that for many compute budgets the optimal number of steps is below or close to the number of warmup steps (fixed at 3000). Figure 2 (left) also provides an intuitive explanation for the increased value of a: at smaller compute scales, models are 'forced' to use more training tokens than would otherwise be optimal in order to 'escape' the long warmup period. Having escaped, the warmup Once this warmup period is escaped, the optimal number of tokens grows only slowly, leading to a fast rate of increase in the optimal model size and hence the large exponent.With the problem identified, we propose a simple heuristic for more appropriately choosing the warmup duration: for each model, we set the number of warmup tokens to be identical to the model size N . We validate our warmup heuristic in Appendix F. The bottom row of Figure 2b illustrates the validity of our new choice of warmup, showing that the optimal number of tokens is always at least 5 times greater than the (interpolated) duration of the warmup period corresponding to the model of the appropriate size. As is evident from this figure and from Figure 1c , shortening the warmup shifts the scaling law in the direction of Hoffmann et al. further, yielding an exponent a of roughly 0.6.With learning rate warmup corrected, we turn to study learning rate decay, which Hoffmann et al. [25] conjecture to be a main cause of the difference between their result and Kaplan et al. [30] . We The optimal number of tokens D ⋆ as a function of the compute budget C. Left: Using the warmup period of Kaplan et al. [30] , smaller models reach compute-optimality during warmup.Right: Setting the number of warmup tokens to be identical to the model size (visualized using the power law fit) ensures models reach compute-optimality well after the warmup and yields a scaling law closer to Hoffmann et al.. We replicate these plots for all of our experiments in Appendix E.observe that the long 131B tokens decay period in Kaplan et al. [30] , which is aimed toward training to full convergence, means that their compute-constrained experiments see virtually no learning rate decay: Figure 2 shows that, at our compute scales, it is never optimal to train for more than 10B, which corresponds to less than 1.5% decay with a cosine schedule.To correct this, we follow the second approach of Hoffmann et al. [25] and choose the learning rate schedule for every model and FLOP budget individually. For each FLOP value in our grid, we pick the 7 models from Table 2 which yield token-to-parameter ratios in the range 1 to 100, and train them with a cosine learning rate schedule that decays to 1% of the maximum learning rate when reaching the target FLOP value. 5 19 tokens, compute-optimal models at smaller scales train for only a few hundred steps, which is likely too little. Kaplan et al. [30] notice this issue, and attempt to correct for it using post-processing based on an empirical model of large-batch size training [36] ; we return to their result at the end of this section.Here, we take the more direct approach of predicting near-optimal hyperparameters for each model size. 6 Since changing the batch size often also requires re-tuning the learning rate [19, 36, 49, 61] , we sweep over both parameters for models of sizes 5M to 108M, with an additional validation sweep over models of size 220M. Initially, we kept β 2 at its previous value of 0.95. However, this led to poor results at smaller batch sizes: as the batch size gets smaller, the squared gradients become noisier, and AdamW requires more smoothing to obtain a correct denominator. Therefore, we added 0.99 and 0.999 to the sweep, obtaining improved performance on small batch sizes. This empirical observation matches the theoretical work by Zhang et al. [63] , that showed that when the batch size is small, higher values of β 2 are crucial for the convergence of Adam. In Appendix G we describe the parameter sweep in full, validate the optimality of our prescribed hyperparameters on the largest model we train, and provide additional discussion about the role of β 2 .Figure 3 for this parameter, reaching maximum transparency for loss that is suboptimal by 0.03 or more. We also plot interpolation-based estimates of the optimal parameter values and fit them with power laws.extrapolation to nearby values of N , we apply these power laws (with slight rounding) to select the batch size and learning rate for all model sizes and tabulate the results in Table 4 . In Appendix G.5 we validate our learning rate and batch size scaling law for a model with 901M parameters.Our parameter tuning approach is inspired by DeepSeek [15] , who predict the optimal batch size and learning rate as a function of compute. Translating compute to model size using the Hoffmann et al. scaling law, we find remarkable agreement in the batch size predictions (a difference of less than 0.05 in exponent and less than 60% in predictions over our models), and somewhat different learning rate predictions (a difference of 0.11 exponent and a factor of 2-3 in predictions), potentially due to using different weight decay. Both our results appear to contradict the conventional wisdom about the existence of a critical batch size [49, 61, 36] below which every batch size is good, finding instead an optimal batch size below which performance degrades. This suggests further tuning of β 2 or other hyperparameters may be warranted. We discuss DeepSeek [15] further in Section 5.1.With the new hyperparameters, we obtain a close reproduction of the Hoffmann et al. scaling law (Figure 1e ) with the scaling exponent matching 0.5 to within 0.6% and the predicted model size at Chinchilla compute within 15% of Chinchilla's size. Notably, here we use a constant learning rate schedule, demonstrating that careful learning rate decay is not necessary for this scaling law to hold.Finally, we reproduce the adjusted scaling law N ⋆ (C) = 1.3e9(C/8.64e19) 0.73 which Kaplan et al. [30] obtain by estimating the compute required to reach the same results at a sufficiently low batch size. To do so, we use our tuned hyperparameters as a proxy for suitable batch size and revert our previous corrections (head FLOP count and warmup duration). We obtain an exponent of 0.717 and good agreement with their adjusted scaling law; see Figure 18 in Appendix H.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
additional analysis 4.1 trends in compute-optimal loss: 
 Figure 4 shows the minimum loss achievable for each compute budget C in the experiments shown in Figure 1 . We estimate the minimum loss using the same interpolation procedure we use to extract the optimal parameter number N ⋆ and token count D ⋆ . The figure shows that, at low compute scales, shortening the warmup duration and tuning hyperparameters leads to substantial loss improvements (each by up to 0.5 nat per token). However, at larger scales these interventions do not significantly improve the loss. In contrast, learning rate decay becomes increasingly beneficial as compute grows, and appears to also improve the rate of decrease in the loss. Perhaps coincidentally, the effects of overestimating the optimal loss (due to long warmup and large batch size) seem to closely offset the effect of underestimating computational cost (by discounting the contribution from the model's head): the first and last curves in Figure 4 closely overlap.Similarly to Hoffmann et al. [25] we observe a curvature in the optimal loss, while Kaplan et al. [30] report a near-perfect power law behavior. This difference is due to a combination of the difference in FLOP counts discussed in Section 3. Each subplot uses a different number of lower-compute loss measurement to fit the loss trend.extend to higher compute budgets where the loss is closer to its irreducible level. Indeed, for the tuned optimizer experiment (Section 3.5) we find that a saturating power law fits the optimal loss and extrapolates well, while extrapolating poorly for other experiments (see Figure 19 in the appendix). This suggests that a predictable trend in L(N ⋆ (C), D ⋆ (C)) is an indicator of locallyoptimal hyperparameters. The exponent of our saturating power fit is approximately -0.1, twice as large as the exponent found in Kaplan et al. [30] .Finally, we validate our compute-optimal loss scaling law by training and evaluating a model using a larger compute budget. Specifically, we train a 901M parameter model with a compute budget of C + ≈ 8e19 FLOPs, which our scaling law predicts to be compute-optimal, using the batch size and learning prescribed by our hyperparameter scaling laws (see Table 4 ), reaching a loss value of L + = 2.943. In Figure 6 , we add the point (C + , L + ) to the red curve in Figure 4 and find that it falls within the predicted trend. Notably, L + is obtained with a single training run using our predicted optimal configuration, while at lower compute values we estimate the optimal loss by interpolating an IsoFLOP curve.We now estimate the computational cost of our scaling law experiments, quantifying the effect of the learning rate schedule, and plot how our predictions improve and become more confident with increased computation. We find that the training cost of each experiment that utilized a fixed learning rate schedule was 1.54e20 FLOPs, while the experiments that used a varying-length cosine learning rate schedule required 2.99e20 FLOPs; essentially double the compute; see Appendix J for more details. We also find that the cost of the hyperparameter sweep described in Section 3.5 was 2.04e20 FLOPs-slightly less than the combined cost of two scaling experiments that leveraged it (one on each dataset). Moreover, in hindsight, we could have arrived at similar hyperparameters using only models of size at most 57M and a simple heuristic for choosing β 2 based on batch size, which would have cost only 1.44e19 FLOPs.Figure 5 shows the evolution of the predicted compute-optimal model size exponent a, its confidence interval, and a measure of the prediction accuracy as we modulate the experiment's compute budget by truncating our FLOP grid. The figure shows that the prediction becomes steadily more accurate and confident as compute increases. We present these results, as well as the results in Section 4.1, on the OpenWebText2 dataset as well (see Appendix I). 
related work: 
 While neural scaling laws precede the advent of large language models [24, 45] , breakthroughs in model [57] and data [42, 44] scaling allowed Kaplan et al. [30] to demonstrate the dramatic possibility of unbounded improvement with scale, triggering an explosion in the literature on the topic. Here we focus on the relatively fewer works that tackle optimal resource allocation under a compute constraint.For language modeling, Hu et al. [26] and DeepSeek [15] repeat subsets of the analyses in Hoffmann et al. [25] and derive compute optimal scaling laws. Employing Approach 3 of Hoffmann et al. [25] (see also [9] ), Hu et al. [26] find that, for their models, optimal scaling favors larger token-toparameter ratios than in Hoffmann et al. [25] and in our results. They attribute this difference to modeling improvements since [25] and argue the same holds for Llama 2 [56] . However, our setup incorporates most of the advances in Llama 2 and still produces power laws very close to Hoffmann et al. [25] . Like us, DeepSeek [15] perform hyperparameter tuning and use isoFLOP analysis to determine compute-optimal model sizes on multiple datasets. While they arrive at an exponent on the order of Hoffmann et al. [25] for the main dataset they study, they report a higher exponent a = 0.578 for OpenWebText2 (i.e., predicting lower token-parameter-ratio at scale), which they attribute to the superior quality of the dataset. We also study this dataset but arrive much closer to the Hoffmann et al. scaling law. We conjecture the larger exponent might be due to repeating training data, which likely occurred in their experiment given the dataset's limited size and their compute budget. Settling these discrepancies could be a source of further valuable lessons on optimal model scaling.Recent work also studies compute-bounded scaling laws beyond the compute-optimal regime. Informed by the increasingly common practice of training medium-scale models beyond compute optimality [e.g., 55, 56, 29], Sardana and Frankle [46] account for the expected inference cost of the model, showing that it naturally skews optimal settings toward smaller models. Gadre et al. [17] directly predict the loss and downstream performance for models trained past the point of compute optimality, and Muennighoff et al. [37] model joint compute-data bottlenecks. All three works rely on the Hoffmann et al. law as a reference point, with [17, 37] baking it to their parametric forms.Compute-optimal scaling is studied beyond the language domain, particularly in vision. Henighan et al. [23] study autoregressive modeling for a variety of tasks and find scaling laws roughly consistent with the Kaplan et al. [30] adjusted scaling law (with exponent a = 0.73). That work shares the methodological issues described in the top row of Figure 1 (FLOP count and long warmup), but performs hyperparameter tuning for smaller scale models; in Appendix H we reach similar results when doing the same. Zhai et al. [59] characterize the compute-efficient frontier of Vision Transformers (ViTs), while Cherti et al. [13] studies compute constrained scaling of CLIP models. However, they do not offer a power law for scaling model size with compute. Alabdulmohsin et al. [4] tackle model design under an inference compute constraint by fitting multi-term parametric forms to obtain predictions for the optimal ViT shape. Goyal et al. [20] point out an intricate interplay between data filtering and compute constraints. Finally, Bachmann et al. [7] study compute-optimal scaling of MLP's and obtain exponent a = 0. Recent theoretical work study compute-optimal scaling laws in simplified, analytically tractable settings [11, 33, 38, 28] . In particular, Paquette et al. [38] obtain a power law with exponent a = 0.5 (as in Hoffmann et al. [25] ) for a random-feature linear regression setting [35, 5] , conjecturing that this is a part of a broader, universal phenomenon. Jeon and Van Roy [28] also establish an exponent of 0.5 for data generated by infinite-width two-layer ReLU networks, using information-theoretic arguments.We also remark on two themes of our paper that draw from prior work. The first is the importance of hyperparameter tuning: several works [30, 27, 17, 15] make the case that smooth, predictable scaling laws emerge when models on all scales are properly tuned. Our work (and particularly Section 4.1) provides another example of this principle and agrees with previous observations that tuning is particularly important at smaller scales. Second, previous studies [59, 8, 15, 26] as well as the concurrent work [22] , propose alternative learning rate schedules that address a key shortcoming of cosine decay: the need to commit to a step budget in advance. We consider a constant learning rate that requires no commitment at all. We show this simple choice suffices to reproduce the Hoffmann et al. law and quantify the computational savings compared to a cosine schedule. However, Section 4.1 (and also [25] , among others) show that in terms of loss, the constant schedule clearly underperforms the cosine schedule.Concurrent and independent work. Pearce and Song [40] [30] (see discussion in Section 3.1, Section 3.5, and Appendix H). In addition, our experiments roughly match the Kaplan et al. [30] compute budget, which is about 3 orders of magnitudes larger than budget in Pearce and Song [40] , and we perform careful tuning of both the learning rate and the batch size.Computational scale is a notable limitation, as well as a defining feature, of our results: our experiments are roughly on the scale of those in Kaplan et al. [30] but are substantially smaller than those of Hoffmann et al. [25] . Scaling may effectively mitigate each of the issues we identify: with scale, the contribution of the model head becomes negligible, any (fixed) warmup period eventually becomes reasonably long, and hyperparameter sensitivity decreases, as shown in Figure 4 and Figure 15 . Nevertheless, we believe that experimental protocols that induce correct scaling behavior at low computational budgets are crucial for developing the empirical science of machine learning, particularly in academic settings.Due to limited compute budgets, our hyperparameter sweep only targeted the smaller models in our grid, and furthermore trained each model for only 20N steps, i.e., the optimal point according to the Hoffmann et al. scaling law. This raises the concern that the hyperparameters we chose unfairly favor models trained for that particular token-to-parameter ratio, and rerunning our experiment with perfect tuning for each model size and each token-to-parameter ratio would have yielded different results. We believe this is unlikely: at small scales (where hyperparameter tuning is crucial) our original set of hyperparameters favored higher token-to-parameter ratios because they still had a sufficient number of steps to train for, and therefore choosing hyperparameters specifically for them is not likely to result in significant gains. In Appendix G.4 we analyze our existing tuning results to estimate the potential gains from perfect tuning, and find that they are likely to have small impact on our conclusions. Moreover, transferring our hyperparameters to another dataset yields similar results.Finally, a broader limitation of compute-optimal scaling as defined by Kaplan et al. [30] , Hoffmann et al. [25] and our work, is that it only concerns the pretraining loss rather than more direct measures of a model's capabilities. Here again, scale is an issue: most zero-shot and in-context capabilities do not emerge at the scales we consider here, and predicting them from small-scale proxies is an important open problem [48, 17] . Instead, it is possible to study downstream performance via fine-tuning, though this may cause the clean scaling patterns seen in pretraining to break down [54] , potentially because the fine-tuning procedure is sensitive to the choice of hyperparameters [27] .Hoffmann et al. [25] hypothesize that the scaling law discrepancy is due to a difference in learning rate schedules. The term "learning rate schedule" comprises both the warmup and decay components, and it is plausible to interpret the "Modeling the scaling behavior" paragraph in Hoffmann et al. [25] §2 as conjecturing that both components contribute to the scaling law discrepancy. However, we believe that Hoffmann et al. emphasize the decay component of the learning rate schedule in their hypothesis due to two main reasons. EQUATIONWe begin by considering, instead of the number of weights in linear layers, the total number of non-embedding learnable weights N exact (e.g., including also LayerNorm gains). The fourth column of Table 2 shows that the difference between this number and N is negligible. We also note that embedding layers have a negligible contribution to the model FLOP counts, since they do not require matrix-vector products.Consequently, the only non-negligible source of error in the approximation FLOPs(N, D) = 6N D is the attention layers. Since in OpenLM the attention dimension is identical to the model width, Kaplan et al. [30, Table 1] shows that the attention operation costs an additional 6nd FLOPs per token per layer for a forward and backward pass, where n = 2048 is the sequence length. Thus, if we define an effective model size ofEQUATIONwe have that 6N eff D captures the cost of training the model for D tokens, including attention.We now consider the difference between these approximations and its effect on compute-optimal scaling laws. The fifth column of Table 2 compares N eff to N . It shows that the ratio N eff /N changes smoothly between roughly 1.1 to roughly 1.2 and back to 1.1 as our model sizes grow. We note that had this ratio been completely constant, there would have been no essentially no difference between working with N and working with N eff since a power law in one would correspond directly to a power law in the other. Since in our model grid this ratio is approximately constant, we expect to see limited differences between the scaling laws resulting from each definition. Figure 7 confirms this expectation, showing quantitatively and qualitatively similar results to Figure 1 . Consequently, we cannot determine with certainty which definition is more appropriate for predicting computeoptimal model sizes. Nevertheless, we observe our final experiment (with parameter tuning) predicts that the optimal (effective) model size at the Chinchilla/Gopher compute scale to be about 16 B parameters larger than the one size predicted using our standard definition. These predictions are directly comparable since model size and effective model size are essentially identical at these scales. If we take the Hoffmann et al. scaling law as ground truth, then the prediction we get using N eff is a bit worse. 6)) for which the approximation 6N eff D also captures the FLOPs cost of attention operations in the model. We observe similar results to Figure 1 , though our final experiment produces a mildly higher prediction for the optimal model size at the Chinchilla/Gopher compute scale.Finally, we touch on a third measure of model size, which does not count the contribution of the model's head to the FLOP count. That is, we considerEQUATIONThis is the definition that Kaplan et al. [30] use in their experiment, approximating the flop count as 6N Kaplan D. Their main motivation for this choice is an observation that not counting embedding parameters leads to more predictive scaling laws in the unlimited compute regime. However, as the final column of Table 2 shows, this approximation leads to a large, systematic error in FLOPs counts for smaller models. In Sections 3.1 and 3.2 we show that this is one of the primary factors behind the Kaplan et al./Hoffmann et al. discrepancy.We conclude this section with an overview of the model size definitions used by related works other than Kaplan et al. [30] . Henighan et al. [23] use N Kaplan as Kaplan et al. [30] and observe high scaling exponents as a result. Hoffmann et al. [25] account for both linear and attention layers in their FLOP computation essentially using N eff in their first two estimation approaches. However, their third approach appears to ignore the attention FLOPs and also count the embeddings parameters, i.e., setting N ′ = N + dv. DeepSeek [15] compare 3 definitions of model size, including N Kaplan , N , and a hybrid of N eff and N Kaplan that takes attention into account and ignores the model head.They report that the latter option gives the best prediction of the compute-optimal loss at large scales. However, we note that both [25] and [15] claim that attention costs double the FLOPs mentioned in Kaplan et al. [30] ; we believe this is likely because they do not account for the fact that the attention is causal, meaning it requires only half the FLOPs of an unstructured matrix-vector product. Finally Muennighoff et al. [37] , Gadre et al. [17] , Hu et al. [26] use N as we do. Modeling. We train decoder-only Transformer [57] language models for next-token prediction using OpenLM, a Pytorch [39] for efficient training of medium scale language models. We use the library with largely the same configuration as [17] , leveraging xFormers [31] , bfloat16 automatic mixed precision, (qk)-LayerNorm [6, 16, 58] , SwiGLU [50] , depth-scaled initialization [60] , and rotary positional embeddings [53] . We use the GPT-NeoX-20B tokenizer [10] whose vocabulary size of 50432 closely matches the vocabulary size of Kaplan et al. [30] . We use a sequence length of 2048 which is twice the sequence length used in Kaplan et al. [30] , but we attempt to match parameters like batch size and warmup duration in their size in tokens. We do not tie the weights of the embeddings and head layers.Optimization. Throughout the paper, we use the AdamW optimizer [34] to minimize the standard log loss with an additive z-loss term for stabilization [14] (coefficient 1e-4) as an auxillary loss (for our analysis, we record the log loss without the z-loss term in both train and validation). As advocated for in Wortsman et al. [58] , we use independent weight decay [34] with parameter 1e-4, i.e., we set the "weight decay" parameter in the standard PyTorch AdamW implementation to be 1e-4/η, where η is the base learning rate. In Table 3 and Table 4 we describe our choice of hyperparameter in our experiments.Hardware and computational cost. We train our models on a cluster with 40GB A100 GPU's, using between 4-32 GPU's in parallel per training run. We use the OpenLM/PyTorch distributed data parallel implementation as well as gradient checkpointing. According to our logs, the total compute cost of all the experiments going into this paper is 22.3K GPU hours., and the total FLOP count is 3.03e21 FLOPs.Data repetition. The datasets we work with are large enough to allow us to perform all of our training runs without any data repetition. However, due to two software issues, some experiments experienced limited data repetition. In particular, data going into our hyperparameter sweep might have been repeated up to 10 times. Moreover, on OpenWebText2, some of our larger-scale training runs might have seen data repeated up to 4 times. We believe this had limited to no impact on our results, as the hyperparameter sweep involved fairly small models unlikely to be able to memorize, while [37] show that 4 data repetitions have only a marginal effect on model performance. In our main experiments on the much larger RefinedWeb dataset we have verified that no data repetition occurred.This section provides a comprehensive description of our procedure for fitting the power law for N ⋆ (C). Our procedure for the D ⋆ power law is analogous, using the relationship (3) .Training loss smoothing. We smooth the training loss using a variable-length smoothing averaging window. In particular, we estimate the loss at step i as the average of the losses in steps i -⌊pi⌋ to i + ⌊pi⌋, for p = 0.05. We also compensate for the lag introduced by logging the averaged training loss every k = 20 by shifting the training loss's index k/2; this compensation is quite important for matching the validation loss early in the optimization. We have verified that the smoothed training loss matches the validation loss (where it is available) roughly to the validation loss's sampling error.Fetching the loss at C i FLOPs from a single run. To estimate the loss of a model of size N trained to compute C i we linearly interpolate the validation/training loss (in log-space) at the two steps closest to C i /(6N B), where B is the batch size in tokens. We also require the nearest step to be within 10% of C i /(6N B), and do not return the loss if not such step exists. For most of our experiments, we compute the validation loss precisely at step ⌈C i /(6N B)⌉. However, for the Estimating loss noise. Defining the ideal loss as the population log loss in expectation over model training, there are two sources of error in estimating it: finite samples in the validation set, and variation between training seeds. We estimate the former directly by storing the validation loss on 100 subsamples of the holdout data and find the standard deviation to be in the range 0.001-0.002 across the different experimental settings. To gauge the error due to seed variance, we train smaller-scale models from our grid on 7 seeds using the tuned hyperparameters for 20N tokens each. We find a roughly log-linear relationship (see Figure 12 ) between the (post-warmup) smoothed training loss and the inter-seed standard deviation. For RefinedWeb (Table 4 ), it appears to saturate around the sampling error, and we heuristically assign standard deviation 0.05 to samples with loss > 7, standard deviation 0.002 to samples with loss < 3, and linearly interpolate the standard deviation in log-space to samples with loss in the range [3, 7] . For OpenWebText2 we observe significantly more cross-seed variance as well as less stable loss during training (compare the loss curves in Figures 8 and 9 ), potentially due to a difference in document lengths. Therefore, we set our standard deviation estimate to go from 0.1 at loss 6 to 0.01 at loss 3, saturating outside the interval and log-space interpolating inside it.Estimating N ⋆ (C i ) and its uncertainty. Given a list of N values and their respective loss samples at compute C i (fetched as described above), we estimate the optimal value N ⋆ (C i ) using the following bootstrap-like procedure. For each bootstrap sample, we add independent Gaussian noise to each loss, whose standard deviation is determined according to the heuristic formula described above. We then interpolate the curve of loss vs. N using Akima [3] interpolation in log-space and find the value minimizing the interpolant; this forms our population of bootstrap samples for N ⋆ (C i ) estimate. We estimate their standard deviation in log-space, and take the maximum between that value and one-third of the average log-spacing on the grid (roughly 1 3 log √ 2). Occasionally, N ⋆ appears on the edge of the N grid (though we attempt to avoid this in our experiment design). If more than half of the bootstrap samples land at the edge of the grid, we omit the value of C i from the subsequent power law fit. Otherwise, we keep only the samples outside the grid edge, and blow up the standard deviation estimate by the fraction of omitted samples. In Figure 13 we complement Figure 1 by plotting our observation and power law fits for D ⋆ , ρ ⋆ and N ⋆ for all the experiments described in Figure 1 . In Figure 14 we reproduce this figure for the OpenWebText2 dataset, showing consistent qualitative and quantitative results.We preform small scale experiments to verify that our heuristic warmup duration (as discussed in Section 3.3) and the learning rate at the end of cosine decay (as discussed in Section 3.4) have little impact on our results. We train a model with 108M parameters on the RefinedWeb dataset. We use the same hyperparameter as in Table 4 , with a training duration of ∼ 2.1B tokens. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in Appendix D.Warmup duratiom. We vary the warmup tokens from N/4 to 16N . We tabulate the results in Table 5 , and find that N tokens for warmup is nearly optimal. Final learning rate value. We vary the final learning rate value from 0.1% to 10% of the peak learning rate. Here we train the models with cosine decay and with hyperparameters as described in Table 3 . This is the same setting as described in Section 3.4. We tabulate the results in Table 6 , and find that the loss is barely impacted by the final learning rate value.This section provides a detailed description of our hyperparameter tuning procedure.We perform an extensive parameter sweep over 6 models from our grid (Table 2 ) with sizes between 5M and 221M parameters. For each model, we sweep over learning rate and batch sizes, as well as three values of β 2 . We train each model of size N for 20N tokens (i.e., following the Hoffmann et al. scaling law) and record the validation loss that the end of training. Overall, our hyperparameter sweep includes 642 training runs, and we perform it on only a single dataset (RefinedWeb). Figure 15 plots all the loss values recorded in the sweep. Compared to analogous plots in Wortsman et al. [58] , we observe more sensitivity to the choice of learning rate, particularly for smaller models. We conjecture that this is because all the models in [58] train for the same amount of tokens, so the smaller models become fully convergence for a wide range of learning rates.To estimate the optimal batch size and learning rate for each model size, we adopt a two-stage interpolation approach. In the first stage, for each model size and batch size, we estimate the optimal learning rate by interpolating (in log-space) the loss as a function of learning rate using Akima [3] interpolation, where for every learning rate we assign the lowest loss obtained from the three values of β 2 . We minimize the interpolant and save its minimizing argument and minimum value. In the second stage, repeat this procedure over the sequence of batch size and interpolated loss pairs, finding an optimal batch size for each model size. To extract an estimate of the optimal learning rate, we simply interpolate the (batch size, minimizing learning rate) sequence and evaluate it at the optimal batch size. Token-to-parameter ratio The estimated excess loss caused by using the hyperparameters in Table 4 instead of the ideal hyperparameters for each model size N and token-to-parameter ratio ρ. Light dashed lines show the raw excess loss estimates. To these we apply a median filter of width 2 (in terms of token-top-parameter ratio) and plot the results in solid lines. Top-right: The estimated IsoFLOP curves with ideal tuning, obtained by subtracting the excess loss from the actual loss.Bottom: Comparing the compute-optimal model sizes obtained from direct observations (with hyperparameters as in Table 4 ) with our estimate for compute-optimal model sizes given ideal tuning per model.In this section, we use the training loss data from our hyperparameter sweep to approximate such ideal tuning and estimate its effect on the compute-optimal scaling law. We do so in three steps.1. Estimating suboptimality as a function of token-to-parameter ratio. We estimate the best hyperparameters for ρ < 20 using the same interpolation logic as in Section 3.5 but for the training loss after D = ρN tokens. (We do not consider values of ρ below 2 since they are too close to the warmup period.) Thus, for every value of N and ρ, we obtain an estimate of the loss with optimal hyperparameters, denoted L ⋆ . We also use interpolation to estimate the loss under our chosen hyperparameters for each model size (given by Table 4 ), denoted L. shows the original IsoFLOP curves (as in Figure 10 ) along with our estimated loss with ideal tuning.3. Re-fitting the scaling law. To estimate the effect of ideal tuning on our estimate of the computeoptimal exponent a, we apply our 'bootstrap' fitting procedure described in Appendix D on the updated IsoFLOP curves described above. To fit the scaling law we only use FLOP values in {2 k • 1.25e16} 7 k=0 . For k > 7 we do not have data to estimate loss under ideal tuning.Conclusions. The top-left panel of Figure 17 shows that our hyperparameters yield losses within 1e-2 of the optimal value for models sizes above 15M and ρ values above 10. For smaller models or ρ values, the potential loss reduction from ideal tuning is greater, as is also evident in the top-right panel of the figure. Nevertheless, the top-right panel also shows that the compute-optimal model size (marked by stars on the IsoFLOP curves) do not move much due to the loss reduction. The bottom panel further reveals that for most FLOP values the difference between compute-optimal model sizes is within their estimated standard deviations. It also shows that approximating ideal hyperparameter tuning moves the estimate of the compute-optimal exponent by less than 0.032, bringing further away from the Kaplan et al. exponent. Furthermore, since learning rate and batch size sensitivities decrease with model size, we expect ideal tuning to have an even smaller effect at larger compute budgets, so we are likely to see even better agreement in a if we introduce larger models to our ideal tuning estimate (which would require extending the hyperparameter sweep to larger models as well). Indeed, dropping the two smallest FLOP values from the power law fit (that is, using FLOP values {2 k • 1.25e16} 7 k=2 ) yields exponent a = 0.489 for the original observations and exponent a = 0.5 for ideal tuning. Overall, we estimate that ideal hyperparameter tuning would produce similar results to our scaling-law-based hyperparameter choices.We extend the training of our largest model (with 901M parameters) to ∼14B RefinedWeb tokens (the compute-optimal value for that model size according to our scaling law), and test whether our predicted batch size and learning rate are optimal at this scale. In particular, we compare the learning and batch size prescribed in Table 4 (0.0024 and 640, respectively) to 12 configurations. In 8 of them we very either the learning rate or the batch size, and in 4 of them we very both. We evaluate the models on held-out RefinedWeb data and calculate the validation loss and standard deviation due to sampling as described in Appendix D. The results are shown in Table 7 . We find that the prescribed learning rate and batch size are indeed optimal, as they yield the lowest validation loss.Figure 18 shows that by reintroducing the FLOP count and long warmup issues from Sections 3.1 to 3.3 but training with optimized hyperparameters, we recover the "adjusted" form of the Kaplan et al. [30] compute-optimal scaling law, given by 1.3e9 • (C/8.64e19) 0.73 . Kaplan et al. [30] derive this scaling law by theoretically compensating for the fact that they used a too-large batch size for smaller models. It is reassuring to observe that when we add back the other issues we have identified but appropriately decrease the batch size via parameter tuning, we obtain very close agreement with this adjusted scaling law. Figure 19 : An expanded version of Figure 4 showing a saturating power law fit for each experiment.Extending Figure 4 . In Figure 19 we fit a saturating power law of the form L(C) = E + L 0 C -ℓ to each of the compute-optimal loss curves in our experiments. As the figure shows, the fit is predictive only for the experiment where hyperparameters are tuned. We fit the saturating power law similarly to Hoffmann et al. [25] , by minimizing the Huber prediction loss for log L(C) over ℓ, log(E) and log(L 0 ).Results on the OpenWebText2 dataset. We perform the same analysis on the OpenWebText2 dataset. In Figure 20 we show the fit of the saturating power law to the compute-optimal loss curves for the OpenWebText2 dataset, and in Figure 21 we show the OpenWebText2 version of Figure 5 .The loss scaling law is predictive for the OpenWebText2 dataset as well. Cosine decay (no tuning) We now discuss the calculation of the cost of each of our main experiments, comparing fixed learning rate schedules (constant or fixed-length cosine as in Kaplan et al. [30] ) with a cosine schedule tailored for each. Each of our experiments consists of directly estimating N ⋆ (C) for a grid of C values of the form C k = 2 k • 1.25e16 FLOPs and k going from 0 to K = 11. With a cosine learning rate schedule, each value of C requires distinct training runs, so the cost of the experiment is K k=0 m k C k , where m k is the number of models we train for C k FLOPs-between 6 and 7 in our experiments. A constant learning rate schedule offers savings since we can extract performance at different FLOP values from the same run, so the cost of the experiment isK k=0 m ′ k C k where m ′k is the number of models we train for at most C k FLOPs. At the maximum budget we have m ′ K = m K between 6 and 7, but for all smaller k < K we have m ′ k between 0 and 2 (typically 1). Thus, we saveK-1 k=0 (m k -m ′ k )C k ≈ K-1k=0 m k C k , which for our doubling grid of C is roughly half the cost. For a fair comparison, when empirically summing over the cost of our experiments omit runs where the number of tokens is more than 100 times the model size or where loss is more than 1 nat above the optimal loss for the compute budget since they do not contribute to the analysis and when experimenting with a cosine schedule we were more careful not execute them. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: Yes, we attach error bars and confidence intervals to the results when applicable. We also explain our methods for calculating these (Appendix D). Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: Yes, we provide details on the compute resources used in Appendix C.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Justification: There are no societal impacts of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper does not release data or models that have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The resources used in the paper are properly credited and the license and terms of use are explicitly mentioned and respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.This notation abstracts away the fact that there are many different models of size N and many different ways to train them for D tokens. Ideally, L(N, D) represents the loss attained by the optimal architecture of size N trained with the best possible training method that uses D tokens. In practice, for any value of (N, D) we consider only a single configuration, but this configuration is the result of architecture search and optimizer tuning, performed either directly or indirectly by building on prior work.We set the warmup period to be the minimum of the model size N and 20% of the total token budget. We ablate the final learning rate in Appendix F.More specifically, we predict the optimal hyperparameters per model size when trained for 20 tokens per parameter. As we discuss in Section 5.2, this choice of training budget is potentially an issue but further analysis in Appendix G.4 suggests it does not significantly impact our results. 
 
 Appendix:  


Decision: This paper examines the discrepancies between the two most influential scaling law papers in the literature (Kaplan et al. and Hoffmann et al.). The authors do a thorough job and manage to reconcile the conflicts in the prior work. All reviewers support acceptance and the AC agrees. This paper will be a good addition to the literature.

Reviews: review: paper_summary: paper_summary: - Identifies and eliminates discrepancies between the scaling laws of Kaplan et al and Hoffman et al via three interventions: accounting for un-embed layer FLOPs, correcting warmup hyperparameters, and tuning optimizer hyperparameters.
- Also derives scaling laws for optimal learning rate and batch size as a function of model size
strength_weakness: strength_weakness: The main limitation of this paper (as noted in the discussion) is that the range of models considered is fairly small compared to Hoffman et al. due to compute constraints. Nevertheless, it would be useful to have some sanity checks for checking if the estimated scaling laws for loss, learning rate, and batch size extrapolate reliably to 3B and 7B scale models.
questions: questions: - Why did you opt for the IsoFlop estimation approach (approach 2 in chinchilla) rather than the arguably more direct "approach 3" in the chinchilla paper? 
- In Appendix B, you mention that the main non-negligible source of error in approximating FLOPs with 6ND is the attention-related computation. This error can be quite non-trivial for small models with large context lengths, but would be small for large models. So, how would the scaling law exponents change if the FLOP estimator accounts for attention flops?
limitations: limitations: None
paper_summary: paper_summary: The paper provides an explanation for the discrepancies between the scaling laws of Kaplan et al. and Hoffman et al. The authors start by reproducing the scaling laws of Kaplan at al. They then introduce incremental changes in the methodology: accounting for the last layer computational cost, setting a more reasonable learning rate warmup, and more thorough scale-dependent hyperparameter optimization. After introducing these changes, they recover the scaling laws of Hoffman et al. The paper provides evidence that careful learning rate decay is not necessary to resolve the differences between these two scaling laws. The paper demonstrates the importance of such scale-dependent hyperparameters for obtaining reliable scaling laws with small computational budgets.
strength_weakness: strength_weakness: Hoffman et al. suggests that their fitted scaling law parameters differ from those of Kaplan et al. because Kaplan et al. include very sub-optimally trained models in their analysis, which bias the fitted parameters. That is, for certain (N, D, L) tuples, the loss L is substantially overestimated. They suggest that this is mainly due to the poor learning rate schedule used by Kaplan et al. Contrary to what the paper under review claims at times, I view this paper as supporting the broad hypothesis of Hoffman et al. rather than disproving it. Starting with training choices similar to those of Kaplan et al., the authors are able to reproduce their scaling law. They then show that improving these training choices — via a more reasonable warmup schedule, batch size, learning etc. — yields a scaling law that more closely resembles that of Hoffman et al. That is, the poorer training hyper parameters of Kaplan et al. explain the difference with Hoffman et al. While Hoffman et al. emphasize the learning rate schedule, I don’t read their paper as claiming that learning rate decay is necessary to obtain their scaling law —as the authors claim at times—, but rather that the poor learning rate scheduler used by Kaplan et al. is sufficient to mess up their derived scaling law. To me, the hypothesis of Hoffman et al. is consistent with the existence of different hyper parameter choices for which similar (N, D, L) is attainable with a constant learning rate — they do not claim that is is not possible.

The authors decouple “warmup” from “decay”, whereas to me the learning rate schedule comprises both. Thus, when Hoffman et al. say that they set “the learning rate schedule to approximately match the number of training tokens”, I imagine that they mean adapting both the warmup and the decay period (e.g., to use heuristics such as warmup for 5% of the total training steps). Unfortunately, Hoffman et al. do not give details on what warmup schedule they used. However, looking at Figure A1, it is clear that the warm-up period that they use is a small fraction of the total number of training steps — unlike Kaplan et al. Broadly, I find it misleading to say “Counter to a hypothesis of Hoffmann et al., we find that careful learning rate decay is not essential” when Hoffman et al. say “Our work differs from Kaplan et al. (2020) in several important ways. First, the authors use a fixed number of training tokens and learning rate schedule for all models. […] In contrast, we find that setting the learning rate schedule to approximately match the number of training tokens results n the best final loss regardless of model size” — notice the difference in language, “learning rate schedule” (which is broadly understood to include also warmup) instead of “learning rate decay”.

I ask the authors to please better scope their claim: “Counter to a hypothesis of Hoffmann et al. [21], we find that careful learning rate decay is not essential for the validity of their scaling law” — Hoffman et al. do not claim necessity, and they refer to “learning rate schedule” rather than only learning rate decay.

Lastly, the authors may be underestimating the value of learning rate decay, since they only consider models that are reasonably close to Chinchilla-optimal. When considering, for the same model size, very different number of training tokens, cosine decay may play a more important role, at least if all other hyper parameters are kept fixed. The authors are aware of this and discuss it in the limitations section.
questions: questions: For Section 3.5, do you warmup the learning rate? How so?

Hoffman et al. decay learning rate to 10% of peak, why do you use 1% of peak?
limitations: limitations: The authors adequately discuss the limitations of their work.
paper_summary: paper_summary: This paper studies the discrepancy between the scaling laws for Kaplan et al. and Hoffmann et al. and identifies three factors behind it: last taking last layer cost into account, warmup duration, and tuning optimizer hyperparams with model scale. They show the transition from Kaplan et al. scaling laws before correcting for these effect to Hoffmann et al. scaling laws after correcting for these effects.
strength_weakness: strength_weakness: The main limitation of this work as also discussed by the authors is the scale of the models use (< 1B).

A more stronger instance of this limitation is that the paper uses experiments on model of size 5M to 100M to fit scaling laws for learning rate and batch size, it is unclear if these laws would hold up for large models. It would have been useful if the paper did an check of this optimality at a significantly larger scale (~1B, not just 200M). This is less of a concern for the runs in the paper since all of them have model sizes <=1B and since Adam is known to be stable (Wortsman et al. 2023) across a range of learning rates.
questions: questions: 1. Is the sweep described in appendix E done with constant learning rate or cosine decay?
2. While the papers fixing of warmup to be always less than the model size is certainly a step in the right direction as compared to Kaplan et al. is there. a reason to believe that this is optimal? or that the scaling law would not change with change with the warmup duration?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper aims to resolve the discrepant conclusion about compute-optimal model size drawn from (Kaplan et al., 2020) and (Hoffmann et al., 2022), two famous scaling-law papers that have guided the exploration of early large language models. The authors concluded that the discrepancies arise from last-layer FLOPS and warmup steps. And contrary to the claims of (Hoffmann et al., 2022), learning rate decay is not essential. As a byproduct, the authors also develop strategies to select hyperparameters, including optimal learning rates and batch sizes, and emphasize the need to use larger beta_2 when training with small batch sizes using Adam.
strength_weakness: strength_weakness: 1. Discrepancies in scaling laws indicate concerns about the correctness of scaling laws. Most of the paper discusses how to resolve the discrepancies between two scaling laws but has not fully addressed the concerns about the correctness of the two scaling laws as only limited results (only Fig. 4 with one dataset) show resolving the discrepancies generally leads to accurate prediction in large-scale experiments.
2. The conclusion of this paper is not clear enough. It would be clearer if the authors could conclude a standard pipeline to reproduce the scaling law experiments and enable reliable predictions with this pipeline.
3. This paper focuses on L(C), the compute optimal scaling laws, instead of L(N, D), which is a more general scaling laws that consider both model sizes and training data. Given that the latest large language models are overtrained to reduce inference costs, limiting to the compute-optimal setting may be less impactful.
questions: questions: 1. One of my concerns on scaling laws is that the need for careful hyperparameter selection may impede its applications. Firstly, it makes fitting a law extremely complex, and making mistakes that lead to unreliable predictions is easy. Also, the need to tune hyperparameters increases the costs to fit a law, indicating that we need accurate extrapolation to orders of larger magnitude to pay off the costs. I would like to learn about the authors' comments on potential strategies to stabilize the process and reduce the costs of hyperparameter tuning.

2. Given that many scaling laws exist, and they are discrepant possibly because of different experimental settings or assumptions, could you summarize the condition for the scaling laws in this paper?
limitations: limitations: This paper only explores the compute-optimal settings and provides limited extrapolation results based on their findings (only one dataset and only scales up to 3e19 flops).

Comments: title: Response to the authors.
Complete results of batch size sweep
comment: I think that my current score is appropriate. Thank you.
Dear reviewer,

Before the author-reviewer discussion period ends, we would like to know if our responses above fully address your questions and concerns. If so, we kindly ask you to reconsider our paper’s score.
Since our 1B model experiments address the main limitation pointed out in your review, would you consider raising our paper’s score?
We are glad that our discussion of the conjecture in Hoffmann et al. was helpful and we are sure including it in the revision will also help future readers - thank you for bringing it up! The interpretation of Hoffmann et al.’s conjecture was the main weakness pointed out in the review. Now that it is resolved, and the two additional questions are answered, would you be willing to increase your score?
Thank you for your response, and for the detailed discussion of the conjecture in Hoffmann et al. I maintain my positive assessment of the work.
We tabulated the results of our experiments of varying both learning rate and batch size below:
|   LR \ BS | 320           | 640           | 1280          |
|----------:|:--------------|:--------------|:--------------|
|    0.0012 | 2.970 ± 0.002  | 2.947 ± 0.002 | 2.954 ± 0.002 |
|    0.0024 | 2.970 ± 0.002  | 2.943 ± 0.002 | 2.955 ± 0.002 |
|    0.0048 | 2.977 ± 0.002 | 2.964 ± 0.002 | 2.970 ± 0.002  |

Indeed, our predicted hyperparameters are optimal.
Ok, thank you. I'd like to keep my score as is, great work!
We have queued the 4 additional runs requested and hope they will complete before the discussion period is over. However, we note that Figure 10 suggests that, for larger models, the interaction between batch size and learning rate is not very strong.
Would it be possible to show the performance on the grid [320, 640, 1280] x [0.0012, 0.0024, 0.0048]. This is important since batch size and learning rates have strong interactions. Single runs should be sufficient since the standard deviation is low.
We tabulated the full results of the batch size sweep below:
|   Batch Size | Validation Loss   |
|-------------:|:------------------|
|            160 | 3.050 ± 0.002      |
|           320 | 2.970 ± 0.002      |
|           640 | 2.943 ± 0.002     |
|           1280 | 2.955 ± 0.002     |
|           2560 | 3.013 ± 0.002     |

The results show that our predicted batch size is optimal among the values we have tested.
**Question 1: hyperparameter sensitivity**   
As we remark in the limitations section, we believe that scaling up the models reduces optimization hyperparameter sensitivity, which would explain why Hoffmann et al. were able to obtain their scaling law without very careful parameter tuning. Moreover, Figure 10 in our paper shows that scaling up the model size leads to higher robustness to learning rate and batch size. 

Finding ways to make smaller-scale models also require less per-scale tuning is an important problem - maximal update parameterization (𝜇P) is a promising direction still under active research; we will discuss it in the revision.

**Question 2: Condition for scaling laws**   
We focus on decoder-only Transformer models, trained for next token prediction with the log loss. This focus is the same as in Kaplan et al. and Hoffmann et al., the two papers that presented the scaling laws in question. Beyond that, we argue that the discrepancy between the two scaling laws is not due to a difference in assumptions, but rather due to differences that are largely methodological errors in Kaplan et al.