Language models scale reliably with over-training and on downstream tasks

Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-22 22:30:59

Abstract: Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32$\times$ over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)––each from experiments that take 300$\times$ less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20$\times$ less compute.

Main paper: 
 introduction: 
 Training large language models is expensive. Furthermore, training high-quality models requires a complex recipe of algorithmic techniques and training data. To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger model and data regimes via scaling laws. With reliable extrapolation, it is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run. Indeed, this workflow has become commonplace for training state-of-the-art language models like Chinchilla 70B [45] , PaLM 540B [19] , GPT-4 [76] , and many others.Despite their importance for model development, published scaling laws differ from the goals of training state-of-the-art models in important ways. For instance, scaling studies usually focus on the compute-optimal training regime ("Chinchilla optimality" [45] ), where model and dataset size are set to yield minimum loss for a given compute budget. However, this setting ignores inference costs.As larger models are more expensive at inference, it is now common practice to over-train smaller models [113] . Another potential mismatch is that most scaling laws quantify model performance by perplexity in next-token prediction instead of accuracy on widely used benchmark datasets. However, practitioners usually turn to benchmark performance, not loss, to compare models.In this paper, we conduct an extensive set of experiments to address both scaling in the over-trained regime and benchmark performance prediction.Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. We are able to extrapolate, in both N and M , the validation performance of models requiring more than 300× the training compute used to construct the scaling law. (right) We also fit a scaling law to predict average downstream top-1 error as a function of validation loss. We find that fitting scaling laws for downstream error benefits from using more expensive models when compared to fitting for loss prediction. We predict the average error over 17 downstream tasks for models trained with over 20× the compute. For this figure, we train all models on RedPajama [112] .Motivated by the practice of training beyond compute-optimality, we first investigate whether scaling follows reliable trends in the over-trained regime. We notice, as implied by Hoffmann et al. [45] , for a set of models of different sizes trained with a constant ratio of tokens to parameters, models' reducible loss L ′ [43, 45] follows a power law (L ′ = λ • C -η ) in the amount of training compute C. We find that as one increases the ratio of tokens to parameters, corresponding to more over-training, the scaling exponent η remains about the same, while the scalar λ changes. We explain our observations by reparameterizing existing scaling laws in relation to the amount of over-training.To establish empirically that scaling extrapolates in the over-trained regime, we further experiment with a testbed of 104 models, trained from scratch on three different datasets: C4 [88, 27] ,RedPajama [112] , and RefinedWeb [82] . We find that scaling laws fit to small models can accurately predict the performance of larger models that undergo more over-training. Figure 1 (left) illustrates our main over-training result, where we invest 2.4e19 FLOPs to extrapolate the C4 validation performance of a 1.4B parameter model trained on 900B tokens, which requires 300× more compute to train.In addition to over-training, we also investigate if scaling laws can predict the performance of a model on downstream tasks. We establish a power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks. While it can be difficult to predict the error on individual tasks, we find it possible to predict aggregate performance from a model's perplexity among models trained on the same training data. Figure 1 (right) presents our main downstream error prediction result, where we invest 2.7e20 FLOPs to predict the average top-1 error over a set of downstream tasks to within 1 percentage point for a 6.9B compute-optimal model, which requires 20× more compute to train.Our results suggest that the proposed scaling laws are promising to derisk (i) the effects of overtraining models and (ii) the downstream performance of scaling up training recipes. To facilitate further research on reliable scaling, we will release all experiments and models. 
developing scaling laws for over-training and downstream tasks: 
 In this section, we develop scaling laws to predict over-trained and downstream performance. First, we provide key definitions (Section 2.1). We next present a scaling law for over-training drawing on empirical observation and prior work (Section 2.2). To connect loss scaling and downstream error prediction, we observe that average top-1 error decreases exponentially as a function of validation loss, 17 ). The scalar λ that determines the y-intercept, however, shifts with different token multipliers. This suggests λ is a function of the token multiplier, while η is not.which we formalize as a novel scaling law (Section 2.3). In later sections, we build an experimental setup (Section 3) to quantify the extent to which our scaling laws extrapolate reliably (Section 4).Scaling laws for loss. Typically, scaling laws predict model loss L as a function of the compute C in FLOPs used for training. If one increases the number of parameters N in a model or the number of tokens D that a model is trained on, compute requirements naturally increase. Hence, we assume C is a function of N, D. Following Kaplan et al. [51] , we use the approximation C = 6N D, which Hoffmann et al. [45] independently verify. We consider,EQUATIONwhere E is an irreducible loss and L ′ is the reducible loss. Hestness et al. [43] , OpenAI [76] ). It is also often helpful to consider a power law in a log-log plot, where it appears as a line with slope -η and y-intercept log (λ).L ′ (C) = λ • C -η (i.a.,Token multipliers. We define a token multiplier M = D/N as the ratio of training tokens to model parameters for notational convenience. M allows us to consider fixed relationships between D and N even as a model gets bigger (i.e., as N becomes larger).Compute-optimal training. Hoffmann et al. [45] establish compute-optimal training, where, for any compute budget H, the allocation of parameters and tokens is given by,EQUATIONTo solve for the optimal N * , D * , one can sweep N, D for each compute budget, retaining the best configurations. Hoffmann et al. [45] find that as the compute budget increases, N * and D * scale roughly evenly. Assuming equal scaling, there is a fixed compute-optimal token multiplierM * = D * /N * per training distribution.Over-training. We define over-training as the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number of tokens (i.e., M > M * ). While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus incur less inference cost.To propose a scaling law for over-trained models, we first turn to empirical observation. We train four model configurations with parameter counts between 0.011B and 0.411B for token multipliers M We plot models trained on three datasets and notice an exponential decay of average top-1 error as C4 eval loss, on the x-axis, decreases. We consider on the y-axes average error on 17 evaluations where performance is at least 10 points above random chance for at least one 0.154B scale model. These observations suggest that average top-1 error should be predictable with reliable loss estimates.between 20 and 640, where M = 20 points lie roughly on the compute-optimal frontier, and larger M corresponds to more over-training. We defer experimental details to Section 3 to focus on our observations first. In Figure 2 , we show loss against compute in a log-log plot for the models trained on three datasets and evaluated on the C4 eval set. We notice parallel lines when fitting power laws to the reducible loss, which suggests a near-constant scaling exponent even with increased over-training.This indicates that scaling behavior should be describable in the amount of over-training.In search of an analytic expression for the observations in Figure 2 , we consider existing scaling literature. A common functional form for the risk of a model, as proposed in prior work [93, 45] is,L(N, D) = E + AN -α + BD -β .(3) Recall from Section 2.1, N is the number of parameters and D the number of training tokens. The constants E, A, α, B, β are fit from data. By fitting this parametric form, Hoffmann et al. [45] find that scaling exponents α and β are roughly equal, suggesting that one should scale N and D equally as compute increases. Hence, we assume α = β. With this assumption, we reparameterize Equation (3) in terms of compute C = 6N D and a token multiplier M = D/N . We get,EQUATIONwhere η = α/2, a = A(1/6) -η , b = B(1/6) -η gives the relation to Equation (3) . For a complete derivation, see Appendix A.Equation (4) has the following interpretation: (i) The scaling exponent η is not dependent on M .Thus, we always expect lines with the same slope in the log-log plot-as in Figure 2 . (ii) The term aM η + bM -η determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the log-log plot for the range of M we consider-also consistent with Figure 2 .Recall that we make the assumption α = β, which implies equal scaling of parameters and tokens as more compute is available. However, as explained in Appendix A, even if α ̸ = β, we get a parameterization that implies the power-law exponent remains constant with over-training.Scaling is typically studied in the context of loss [51, 45, 72] , which Schaeffer et al. [100] note is smoother than metrics like accuracy. However, practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss on perplexity evaluation sets. To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure 2 . In Figure 3 , we plot average downstream top-1 errors over evaluations sourced from LLM-Foundry [69] against the C4 eval loss. We defer details of the setup to Section 3 to focus here on a key observation:average error appears to follow exponential decay as loss decreases.Based on the exponential decay we observe in Figure 3 , we propose the following relationship between downstream average top-1 error Err and loss L, 3 , we run a 435 model grid search over model width, hidden dimension, number of attention heads, batch size, and warmup steps. All models are trained near compute-optimally. (center) We plot the efficient frontier of models, which appear to follow a trend, excluding models from 5.2 × 10 16 to 5.2 × 10 17 , which fall below the trend. (right) We fit a power law with irreducible error to the remaining configurations, picking four configurations that closely track the full model suite ("Selected models"). These models extrapolate the performance of 1.4B, 6.9B target models. Shaded regions represent bootstrap 95% confidence intervals.EQUATIONwhere ϵ, k, γ are fit from data. Equation ( 5) also has an interpretation in terms of model perplexityEQUATIONNamely, Err follows a power law in PP that is bounded from above by ϵ signifying arbitrarily high Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The final section discusses limitations, which provide motivation for future work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
constructing a scaling testbed: 
 In this section, we discuss our experimental setup to test the predictions suggested by Equations ( 4) and (5) . We first present our general language modeling setup (Section 3.1). Next, we discuss our strategy for determining model configurations for our scaling investigation (Section 3.2) and fitting scaling laws (Section 3.3). We then present metrics to validate how well scaling laws predict loss and downstream performance (Section 3.4).We train transformers [116] for next token prediction, based on architectures like GPT-2 [85] andLLaMA [113] . We employ GPT-NeoX [15] as a standardized tokenizer for all data. See Appendix B for architecture, optimization, and hyperparameter details.To get final configurations for the 0.011B to 0.411B parameter models plotted in Figures 2 and 3 , we first conduct a wide grid search over a total of 435 models, trained from scratch, from 0.01B to 0.5B parameters (Figure 4 (left)). We train on the original OpenLM data mix [39] , which largely consists of RedPajama [112] and The Pile [31] . While we eventually plan to over-train models, at this step we search for base configurations near compute-optimality. We train on 20 tokens per parameter (M = 20), which, in early experiments, gives models near the compute-optimal frontier. This is similar to findings in Hoffmann et al. [45] 's Table 3 , which suggests that M = 20 is near-optimal for the Chinchilla experimental setup.Table 1 : Default number of parameters N and token multiplier M to fit our scaling laws. We invest ∼100 A100 hours to fit Equation (4) and ∼1,000 A100 hours to fit Equation (5) .N M Used to fit Equation (4) Used to fit Equation ( 5)0.011B 20 ✓ ✓ 0.079B 20 ✓ ✓ 0.154B 20 ✓ ✓ 0.411B 20 ✓ ✓ 0.011B 320 ✓ ✓ 1.4B 20 ✗ ✓ Total compute C [FLOPs] 2.4e19 2.7e20To find maximally performant small-scale models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size. Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles. We find in early experiments that qk-LayerNorm makes models less sensitive to learning rate, which is a phenomenon Wortsman et al. [123] report in their Figure 1 . Hence, we fix the learning rate (3e-3) for our sweeps. We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at M = 20, retaining the best configurations.At this point, we have many models, several of which give poor performance; following prior work [51, 45] , we want to keep only models that give best performance. Hence, in Figure 4 (center),we filter out models that do not lie on the Pareto frontier. While there appears to be a general trend, configurations between 5.2 × 10 16 and 5.2 × 10 17 FLOPs lie below the frontier established by other models. We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of-two batch sizes. We provide support for this hypothesis in Appendix E, but opt to remove these models from our investigation.To ensure tractable compute requirements for our scaling experiments, we require a subset of models that follows the trend of the entire Pareto frontier. In Figure 4 (right), we fit trends to the Pareto models and to a subset of four models. We notice that the trends closely predict both the performance of the 1.4B and 6.9B models, suggesting that our small-scale configurations reliably extrapolate in the compute-optimal setting.Moving forward, we do not tune hyperparameters for other token multipliers (i.e., M ̸ = 20), on other training or evaluation distributions, or on validation sets for downstream tasks. For more details including specific hyperparameters, see Appendix C.To create our scaling testbed, we start with the four small-scale, base configurations from our grid search: N ∈ {0.011B, 0.079B, 0.154B, 0.411B}. To ensure our conclusions are not particular to a single training distribution, we train models on each of C4 [88, 27] , RedPajama [112] , and RefinedWeb [82] , which have 138B, 1.15T, and 600B tokens, respectively, for different token multipliers M ∈ {5, 10, 20, 40, 80, 160, 320, 640}. We omit runs that require more tokens than are present in a dataset (i.e., N = 0.411B, M = 640 for C4). We additionally train N = 1.4B models at M = 20 and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for RedPajama, and 320 for RefinedWeb). We train N = 6.9B, M = 20 models on each dataset given the relevance of 7B parameter models [113, 49] . In total this results in a testbed of 104 models.We fit Equation (4) to approximate E, a, b, η using curve-fitting in SciPy [117] (i.e., Levenberg-Marquardt to minimize non-linear least squares). We repeat this process to fit Equation (5) to approximate ϵ, k, γ. We invest ∼100 A100 hours to train the models required to fit a scaling law for loss and ∼1,000 A100 hours for a corresponding law for downstream error. Unless otherwise specified, we fit to the N, M pairs in Table 1 , which are a subset of our full testbed. Our configurations allow us to test for extrapolation to the N = 1.4B, M = 640 (900B token) and the N = 6.9B, M = 20 (138B token) regimes. Evaluation datasets. Unless otherwise stated, our default validation loss dataset is C4 eval. For downstream tasks, we adopt a subset from 46 tasks from LLM-foundry [69] , which includes standard tasks with both zero-shot and few-shot evaluations. Specifically, we consider a 17-task subset where, for each evaluation, at least one 0.154B scale model-trained with as many as 99B tokens-gets 10 percentage points above chance accuracy: ARC-Easy [23] , BIG-bench: CS algorithms [11] ,BIG-bench: Dyck languages [11] , BIG-bench: Novel Concepts [11] , BIG-bench: Operators [11] ,BIG-bench: QA WikiData [11] , BoolQ [21] , Commonsense QA [107] , COPA [92] , CoQA [91] ,HellaSwag (zero-shot) [126] , HellaSwag (10-shot) [126] , LAMBADA [77] , PIQA [14] , PubMed QA Labeled [50] , SQuAD [90] , and WinoGrand [55] . For more details on evaluation datasets see Appendix D. We focus on this subset to ensure we are measuring signal, not noise. Including downstream tasks like MMLU [40] , where performance is close to random chance, however, does not invalidate our results as we show in our evaluation set ablations (Appendix E).Metrics. We consider three main metrics: Validation loss, which is the cross entropy between a model's output and the one-hot ground truth token, averaged over all tokens in a sequence and over all sequences in a dataset. Average top-1 error, which is a uniform average over the 17 downstream evaluations, as mentioned in the above paragraph. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
results: reliable extrapolation: 
 In this Section, we quantify the extent to which the scaling laws developed in Section 2 extrapolate larger model performance using the scaling testbed from Section 3. By default, we fit Equations ( 4) and ( 5) to the configurations in Table 1 , use C4 eval for loss, and the 17-task split from Section 3.4for average top-1 error.Over-trained performance is predictable. We highlight our main over-training results in Figure 1 (left). Namely, we are able to extrapolate both in the number of parameters N and the token multiplier M to closely predict the C4 eval performance of a 1.4B parameter model trained on 900B RedPajama tokens (N = 1.4B, M = 640). Our prediction, which takes 300× less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error. Additionally, for the N = 6.9B, M = 20 run, near compute-optimal, the relative error is also 0.7%.These results support several key takeaways. (i) Scaling can be predictable even when one increases both the model size and the amount of over-training compared to the training runs used to fit a scaling law. (ii) The form presented in Equation ( 4) is useful in practice for predicting over-trained scaling behavior. (iii) Fitting to Equation (4) gives good prediction accuracy near compute-optimal. More Table 2 : Downstream relative prediction error at 6.9B parameters and 138B tokens. While predicting accuracy on individual zero-shot downstream evaluations can be challenging ("Individual"), predicting averages across downstream datasets is accurate ("Avg."). specifically, predictions are accurate both for the 1.4B over-trained model and the 6.7B computeoptimal model using a single scaling fit.While Figure 1 explores a specific case of making predictions in the over-trained regime, we aim to understand the error profile of our predictions across training datasets, token multipliers, and number of parameters. Hence, Figure 5 shows the relative error between ground truth loss and predicted loss on C4 eval for models in our testbed. We notice uniformly low prediction error suggesting that predictions are accurate in many settings.Average top-1 error is predictable. Figure 1 (right) presents our main result in estimating scaling laws for downstream error. Concretely, we use the models indicated in Table 1 to fit Equations ( 4) and ( 5), chaining the scaling fits to predict the average top-1 error as a function of training compute C and the token multiplier M . Our fits allow us to predict, using 20× less compute, the downstream performance of a 6.9B model trained on 138B RedPajama tokens to within 0.05% relative error and a1.4B model trained on RedPajama 900B tokens to within 3.6% relative error.Table 2 additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law functional forms are applicable on many training datasets. We note that while average accuracy is predictable, individual downstream task predictions are significantly more noisy. We report relative error for more model predictions in Figures 11 and 12 . We also find that if we remove the 1.4B model for the Equation ( 5) fit, relative error jumps, for instance, from 0.05% to 10.64% on the 17-task split for the 6.9B, 138B token RedPajama prediction. This highlights the importance of investing more compute when constructing scaling laws for downstream task prediction compared to loss prediction.Under-training, out-of-distribution scaling, and compute-reliability trade-offs. In addition to our main results presented above, we include additional results in Appendix E, which we summarize here. First, we notice that when token multipliers become too small (i.e., M = 5) scaling becomes unreliable and lies off the trend. Additionally, multipliers other than 20, such as 10, 40, and 80, garner points that are roughly on the compute optimal frontier (Figure 9 ). This observation suggests that the compute-optimal multiplier may lie in a range rather than take a single value. To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings. We find that models trained on C4-English filtered-and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction gives reliable loss scaling (Figure 10 ). We additionally examine the compute necessary to create accurate scaling laws, finding that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures 15 and 16 ). 
related work: 
 We review the most closely related work in this section. For additional related work, see Appendix F.Scaling laws. Early works on scaling artificial neural networks observe predictable power-law scaling in the training set size and number of model parameters [43, 44, 93] . Alabdulmohsin et al.[2] stress the importance of looking at the extrapolation regime of a scaling law. Yang et al. [124] prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. [125] make analogous recommendations when scaling model depth. Bi et al. [13] propose hyperparameter aware scaling laws. Unlike the aforementioned work, our investigation focuses on over-training and predicting downstream accuracy.Hoffmann et al. [45] investigate how the number of model parameters N and training tokens D should be chosen to minimize loss L given a compute budget C. Hoffmann et al. [45] find that when scaling up C, both N and D should be scaled equally up to a multiplicative constant (i.e., N ∝ C ∼0. 5 and D ∝ C ∼0.5 ) to realize compute-optimality. Appendix C of the Chinchilla paper additionally suggests that these findings hold across three datasets. However, Hoffmann et al. [45] do not verify their scaling laws for training beyond compute-optimality, or for downstream error prediction-both of which are central to our work.Sardana & Frankle [98] propose modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimality and solve for various fixed inference budgets. Their key finding, which is critical for our work, is that when taking into account a large enough inference budget, it is optimal to train smaller models for longer than the original Chinchilla recommendations.Our work presupposes that over-training can be beneficial. Instead of solving for inferenceoptimal schemes, we support empirically a predictive theory of scaling in the over-trained regime.Additionally, we provide experiments across many validation and training sets.For predicting downstream scaling beyond loss, Isik et al. [47] relate the number of pre-training tokens to downstream cross-entropy and machine translation BLEU score [78] after fine-tuning. In contrast, we take a holistic approach to evaluation by looking at top-1 error over many natural language tasks.Schaeffer et al. [100] argue that emergent abilities [120] are a product of non-linear metrics and propose smoother alternatives. As a warmup for why non-linear metrics may be hard to predict, Schaeffer et al. [100] consider predicting an ℓ length sequence exactly:Err(N, ℓ) ≈ 1 -PP(N ) -ℓ ,where N is the number of parameters in a model and PP is its perplexity. This is a special case of our Equations ( 5) and ( 6), where the number of training tokens does not appear, ϵ = 1, k = 1, and γ = ℓ. In contrast, we treat ϵ, k, γ as free parameters for a scaling law fit, finding that average error over downstream tasks can make for a predictable metric.Over-training in popular models. There has been a rise in over-trained models [113, 114] and accompanying massive datasets [112, 82, 104, 3] . For example, Chinchilla 70B [45] is trained with a token multiplier of 20, while LLaMA-2 7B [114] uses a token multiplier of 290. In our investigation, we look at token multipliers from 5 to 640 to ensure coverage of popular models and relevance for future models that may be trained on even more tokens.6 Limitations, future work, and conclusionLimitations and future work. We identify limitations, which provide motivation for future work.• Hyperparameters. While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is a need to develop scaling laws that do not require extensive hyperparameter sweeps.• Scaling up. Validating the trends in this paper for even larger runs is a valuable direction.Additionally, repeating our setup for models that achieve non-trivial performance on harder evaluations like MMLU is left to future work.• Scaling down. Actualizing predictable scaling with even cheaper runs is important to make this area of research more accessible, especially for downstream error prediction.• Failure cases. While we present a preliminary analysis of when scaling is unreliable, future work should investigate conditions under which scaling breaks down.• • Individual downstream task prediction. While we find that averaging over many task error metrics can make for a predictable metric, per-task predictions are left to future work.• In-the-wild performance. Downstream task performance is a proxy for the in-the-wild user experience. Analyzing scaling trends in the context of this experience is timely.• Dataset curation. Our work only deals with existing training datasets. Exploring dataset curation for improved model scaling is another promising direction.We show that the loss of over-trained models, trained past compute-optimality, is predictable. Furthermore, we propose and validate a scaling law relating loss to average downstream task performance. We hope our work will inspire others to further examine the relationship between model training and downstream generalization. Our testbed will be made publicly available, and we hope it will make scaling research more accessible to researchers and practitioners alike. EQUATION-α 2 , = E + A 1 6 -α 2 M α 2 + B 1 6 -α 2 M -α 2 C -α 2 .This is equal to Equation ( 4), making the substitutions η = α/2, a = A(1/6) -η , b = B(1/6) -η , as noted in the main body.Relation to compute-optimal training. Recall that we made the assumption α = β, which implies equal scaling of parameters and tokens to realize compute-optimal models. While this assumption is empirically justified [45] , even if α ̸ = β, we get a parameterization that implies the power law exponent in Equation ( 4) remains constant with over-training, while the power law scalar changes.To find a compute-optimal training setting, Hoffmann et al. [45] propose to minimize the right-hand side of Equation ( 3) subject to the compute constraint C = 6N D. This yields,N * = γ 1 α+β (C/6) β α+β and D * = γ -1 α+β (C/6) α α+β, where γ = αA βB , for notational convenience. The associated risk is,L(N * , D * ) = E + Aγ -α β+α + Bγ β β+α C 6 -αβ α+β .We now deviate from compute-optimal training by modifying the model size and tokens by multiplication with a constant √ m, according toEQUATION)This modification keeps the compute constant (i.e., 6N m D m = 6N * D * ). The risk, then, becomesEQUATIONWe again expect the same power law exponent and changing power law scalar. Note that m in Equation ( 8) is similar to M in Equation ( 4). Specifically, m is a multiple of the Chinchilla-optimal token multiplier M * = D * /N * , which is no longer fixed as a compute budget changes for α ̸ = β. Architecture. As stated in the main paper, we train transformers [116] , based on autoregressive, decoder-only, pre-normalization architectures like GPT-2 [85] and LLaMA [113] . We adopt OpenLM [39] for modeling, which utilizes PyTorch [80, 6] , xformers [54] , triton [75] ,FlashAttention [24] , FSDP [130] , and bfloat16 automatic mixed precision. Like LLaMA, we omit bias terms, but replace RMSNorm [128] with LayerNorm [8] , which has readily available fused implementations. Following Wortsman et al. [123] , we apply qk-LayerNorm [25] , which adds robustness to otherwise poor hyperparameter choices (e.g., learning rate). We use SwiGLU [102] activations and depth-scaled initialization [129] . We use a sequence length of 2,048, rotary positional embeddings [106] , and the GPT-NeoX-20B tokenizer [15] , which yields a vocabulary size of 50k.We do not use weight tying [84, 46] . We sample without replacement during training and employ sequence packing without attention masking. We separate documents in our training corpora with end-of-text tokens.Objectives and optimization. We train with a standard causal language modeling objective (i.e., next token prediction) with an additive z-loss [19] (coefficient 1e-4), which mitigates output logit norm growth [67] instabilities. We use the AdamW optimizer [62] (PyTorch defaults except beta2 = 0.95), with independent weight decay [123] (coefficient 1e-4). For the learning rate schedule, we use linear warmup and cosine decay. We cool down to a low learning rate (3e-5).Final model configurations. We present our final hyperparameters in Table 3 .Grid search configuration selection. Recall in Section 3.3, we run a grid search over many configurations. We present the architectures we sweep over in Table 4 .All 46 downstream evaluations are based on MosaicML's LLM-foundry evaluation suite [69] . We specifically consider the datasets given in Table 5 . Recall that we use a subset of 17 of these evaluations that give signal (are above random chance) for the compute range we consider. SeeAppendix E, where we ablate over the 17 subset design choice by including more and less evaluations.Scaling law fits. We present specific coefficients for our fits in Table 6 .Small-scale experiments can predict model rank order. We expect to be able to rank hypothetical models based on their predicted performance, which is useful when deciding what large-scale runs FLOPs over-perform relative to their neighbors. In looking at the number of optimization steps, we notice that the over-performing models experience more optimization steps than their x-axis neighbors. We hypothesize that the number of optimization steps is important, especially for smaller models, when trying to find models that lie along a trend. (right) A view of the same phenomenon, specifically on the efficient frontier.to train. To verify, we rank 9 testbed models with N ≥ 1.4B by ground-truth top-1 error and by estimated top-1 error. We find high rank correlation of 0.88 for the 17-task split.Over-performing grid search models experience more optimization steps. As mentioned in Section 3.3 and Figure 4 , we notice that models between 0.011B to 0.079B (i.e., 5.2 × 10 16 to 5.2 × 10 17 FLOPs trained near compute-optimal) over-perform compared to the trend established by other models in our initial grid searches. This results in a bump in the scaling plot. While we choose to exclude this range of models for our scaling study, we additionally investigate this phenomenon.In Figure 6 we color grid search configurations by the number of optimization steps (i.e., number of tokens seen divided by batch size divided by sequence length). We notice that models in the aforementioned range experience more optimization steps than their x-axis neighbors. For context, Figure 1 (left) in Kaplan et al. [51] also shows a bump; however, there the performance is worse than the general trend instead of better as in our work. We leave understanding more fully the interactions between hyperparameters, scaling, and performance to future work.Scaling is largely predictable in-distribution (ID). Prior work focuses on understanding scaling using ID loss, often using training loss directly [51, 45] . Hence, we also consider Paloma [65] loss evaluation sets, which are designed to probe performance in specific domains. We use Paloma's C4 [88, 27] , RedPajama [112] , and Falcon-RefinedWeb [82] splits to probe for ID loss. As seen in Figure 7 , relative error is mostly low. Relative error is largest for the N = 1.4B, M = 640RedPajama run at 15.4%. Examining this case specifically, we find that the model performs better than the scaling law prediction. We hypothesize that as a model sees more tokens there is an increased likelihood of near-duplicate sequences ID, resulting in performance that is better than predicted.Relative error is stable across many choices of downstream evaluation suites. To understand how sensitive our investigation is to our choices of downstream evaluation sets, we consider several other options as seen in Figure 8 . We find that our prediction errors are fairly (i) low and (ii) consistent for many choices of downstream evaluation sets including the whole suite of 46 evaluations.Scaling can break down when under-training. We find that when a token multiple is too small (i.e., under-training regime), scaling appears unreliable. In Figure 9 we see for M = 5 the scaling trend is different. We hypothesize that tuning hyperparameters (e.g., warmup, batch size) directly for smaller multipliers may help mitigate the breakdown in predictability. 6.9B N 1.1% 0.0% 0.1% 0.7% 0.9% 0.0% 0.6% 2.6% 0.3% 0.2% 0.8% 0.2% 0.3% 1.4% 2.0% 0.5% 1.3% 1.0% 3.9% 3.5% 1.3% 0.2% 0.2% 0.2% 3.5% 0.5% 2.7% 0.4% 2.3%Train: C4 Eval: C4 (Paloma split) 4.6% 0.0% 0.3% 2.8% 1.5% 0.0% 0.8% 0.5% 0.0% 1.1% 2.2% 3.0% 3.5% 3.3% 0.2% 0.0% 0.5% 1.3% 1.4% 1.0% 0.5% 0.1% 0.0% 0.2% 0.3% 0.7%10.0%10.3% Figure 8 : Downstream evaluation set ablation for 6.9B parameter, 138B token runs. Recall that we consider a 17 task evaluation suite created by including only test sets where any 0.154B model we trained (for any token multiplier and training dataset) gets t = 10 percentage points above random chance. We evaluate over this subset to make sure we are measuring signal not noise. Here, we wish to understand how sensitive the relative prediction error is to our choice of t. (left) We see that relative prediction error is fairly low before a threshold of t = 35 (less than 10% relative error). When too many tasks are excluded (i.e., t ≥ 40) relative error spikes. Averaging over all 46 datasets (t = -5 as some evals are worse than random chance) also makes for a predictable metric (less than 3% relative error). (right) A parallel view, showing how many tasks are removed as t increases. 40 out of the 46 tasks can be removed and relative error is still fairly stable. 4). Recall that the C4 training set is English-filtered. Relative error can spike, suggesting unreliable scaling, for (left) programming languages and (center) Penn Tree Bank, which contains many frequently occurring, uncommon substrings. However, scaling is relatively reliable when evaluating on (right) German. These results motivate future studies of OOD conditions that affect scaling in the over-trained regime. 6.9B N 0.3% 0.2% 0.6% 0.9% 0.2% 0.3% 1.3% 1.2% 0.4% 1.0% 0.1% 0.3% 0.3% 0.0% 0.2% 0.7% 0.5% 1.2% 1.7% 1.0% 0.4% 0.6% 0.2% 0.0% 1.6% 0.0% 0.4% 0.1% 0.4% 1.1% 1.3% 0.2% 0.6% 0.4% 0.1% 0.0% 0.3% 0.8% 1.0% 0.6% 1.3% 0.2% 0.2% 0.0% 0.6% 0.9% 0.9% 0.3% 1.3% 0.8% 1.3% 1.5% 1.0% 1.0% 1.0% 0.3% 3.4%Train: RedPajama, Downstream: 46-task split 1.2% 0.1% 0.1% 0.7% 0.3% 0.8% 0.6% 0.5% 1.4% 0.4% 0.9% 0.8% 1.0% 1.2% 0.8% 0.1% 0.6% 0.3% 1.1% 0.7% 0.9% 0.5% 1.1% 1.1% 1.7% 1.6% 0.6% 0.9% 0.3% 4.3% 5). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable.Scaling can be unpredictable out-of-distribution (OOD). Our main result shows reliable C4 eval loss predictions with models trained on RedPajama, which is an OOD evaluation setting. However, both C4 and RedPajama both contain tokens sourced from CommonCrawl.To further probe OOD performance, we measure the relative error of scaling laws fit to models trained on C4 and evaluated on Paloma's 100 programming languages [65] , Paloma's Penn Tree Bank (PTB) split [66] , and a German version of C4 [27] . Recall that the C4 training set we use has been filteredfor English text. Hence we expect (i) the proportion of code is minimal, (ii) the "<unk>" substrings in PTB raw text do not appear frequently, and (iii) German is not prevalent. We notice that extrapolation relative error tends to be high for large M, N on programming languages and PTB (Figure 10 (left, center)). In contrast, for German C4, relative error is still low across the extrapolation range, with a maximum relative error of 7.6% at the N =1.4B, M = 80 scale (Figure 10 (right)). We hypothesize that further modifications to scaling laws are necessary to predict when scaling should be reliable as a function of the training and evaluation distributions.Small-scale experiments can predict average downstream top-1 error. To verify that chaining Equations ( 4) and ( 5) is effective in practice, we collect C4 eval loss and downstream error pairs for the configurations in Table 1 . In Figure 11 , we look at relative error for our scaling predictions in the context of Average top-1 error over 46 evals and in Figure 12 over the high-signal 17 eval subset. We again notice reliable scaling in interpolation and extrapolation regimes, suggesting the validity of our procedure to predict downstream average top-1 error. 5). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable. 13 : Correlation between average top-1 error and evaluation loss. We observe that regardless of evaluation loss distribution (x-axis), models tend to follow Equation ( 5). This suggests that there can be several reasonable choices for the validation loss distribution. Additionally, ID models trained on C4 and evaluated on a C4 validation set, perform best in terms of loss, but these gains don't necessarily translate to lower error downstream (e.g., (left column)). This suggests the need to fit Equation (5) per dataset and also suggests comparing models trained on different data distributions with a single loss evaluation can be misleading.Loss evaluation ablations for downstream trends. Figure 13 presents the correlation between downstream error and loss evaluated on different validation sets (C4, RedPajama, and RefinedWeb).Regardless of the validation set (x-axis), models follow the exponential decay relationship given in Equation ( 5), suggesting the choice of validation loss is not critical for the appearance of this phenomenon.Investing more compute in a scaling law makes it more predictive. Thus far we have looked at standard configurations from Table 1 to construct our scaling laws, mainly to demonstrate extrapolation to larger N, M . However, for practitioners, the main constraint is often training Each red circle represents a scaling law fit to Equation ( 4) with as many as 29 models trained on RedPajama. Specifically, a grid formed by N ∈ {0.011B, 0.079B, 0.154B, 0.411B}, M ∈ {5, 10, 20, 40, 80, 160, 320} gives 28 models and a N = 1.4B, M = 20 run gives the last model. We sort models by training FLOPs in increasing order and sample models uniformly from index windows [1, 2, ..., n] for n ∈ [5, 6, .., 29] to fit Equation ( 4). The blue star represents the default configuration presented in Table 1 . The prediction target is a N = 1.4B, M = 640 (D = 900B) model. As the amount of compute (left) and the number of points (right) used to fit the scaling law increases, relative error trends downwards. Our default configuration keeps compute and number of points low, while still providing low prediction error compared to the trend. 2 Individual estimates The compute necessary to accurately predict loss is less than that needed to accurately predict (right) average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 16 .compute. Hence, we wish to understand the trade-offs between the amount of compute invested in creating a scaling law and the relative error of the resulting law in the over-trained regime. In Figure 14 (left), we see that as one increases the amount of compute, it is possible to get better fits with lower relative error. In Figure 14 (right), we see a similar trend as one increases the number of data points used to fit a scaling law. Blue stars indicate the configurations from Table 1 , which provide accurate predictions relative to the general trends-hinting at their usefulness for our investigation.In Figures 15 and 16 we repeat the compute analysis comparing trade-offs for loss prediction and error prediction for our RedPajama 1.4B parameter, 900B token and 6.9B parameter, 138B token runs respectively. We find that less compute is generally necessary to construct a loss scaling law that achieves the same relative error as that of an error prediction scaling law. The compute necessary to accurately predict loss is less than that needed to accurately predict (right) average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 15 . In Figure 2 , we notice roughly parallel lines (i.e., roughly constant scaling exponent η) in the log-log plot of loss vs. compute, even as the token multiplier M changes. Here we plot η vs. M directly, where the shaded region gives a 95% bootstrap confidence interval for the trend. This view supports that η is relatively constant.On compute-optimal token multipliers. We consider 20 tokens per parameter as close to computeoptimal for our experiments. Here we investigate, using different approaches, what the computeoptimal token multipliers are for each dataset-assuming one should scale number of parameter and training tokens equally as Hoffmann et al. [45] suggest.Turning to Figure 9 , we notice that there are many multipliers, between 10 and 80 that yield models close to the frontier. Hence, empirically, it appears choices within this range should be suitable for the optimal token multiplier.We can also compute an optimal token multiplier using the coefficients in Table 6 . Based on Hoffmann et al. [45] 's Equation ( 4) and the assumption that α = β, we write,EQUATIONTo compute M * = D * /N * , we then have,EQUATION)Using the values from Table 6 and plugging into Equation (10) , we find M * C4 = 2.87, M * RedPajama = 4.30, M * RefinedWeb = 3.79, where the subscript gives the dataset name. These values conflict with the observation in Figure 9 , which suggests M = 5 is already too small to give points on the Pareto frontier. We hypothesize this mismatch arises because we fit our scaling laws using models with M ≥ 20. Language modeling. Language models can be grouped into encoder-only [26, 53, 59, 96, 22] ,encoder-decoder [56, 89] , and decoder-only architectures [85, 113, 114, 110, 49, 38, 74, 7, 111, 28, 64, 99, 122, 4, 57, 63, 34] . Most current implementations are based on the transformer [116] .However, there has been a recent resurgence in scaling language models based on non-transformer architectures [83, 36, 37, 35] . Further, there has been substantial work on adapting pre-trained language models to better follow instructions [119, 20, 70, 61, 71, 133, 87, 29, 115, 103, 73] .However, following prior work [45, 72] and given their overall prevalence, we limit ourselves to GPT-style, decoder-only transformers that have solely been pre-trained.Scaling laws. Kaplan et al. [51] investigate scaling trends in GPT language models. Bahri et al.[9] investigate different scaling regimes theoretically, and Sharma & Kaplan [101] relate scaling coefficients to data manifold dimensions. Tay et al. [108, 109] elucidate the connection between model architecture and scaling trends, while Hernandez et al. [42] , Tay et al. [108] develop scaling laws for transfer learning. Ivgi et al. [48] also consider transfer learning scaling laws and highlight the importance of hyperparameter selection in the low-compute regime. Ghorbani et al. [32] , Gordon et al. [33] , Bansal et al. [10] develop scaling laws for neural machine translation. Caballero et al. [17] propose a scaling law functional form, which they demonstrate is predictive in several domains.Scaling beyond language modeling. There is a large body of work on scaling neural networks beyond language modeling, for example in computer vision [60, 127, 105, 1, 2] , multimodal learning [41, 18, 30] , and image reconstruction [52] .Over-training in existing models. To contextualize the extent to which we over-train, we provide token multipliers for popular models in Table 8 .Language models have known risks in terms harmful language, toxicity, and human automation-to name a few [121, 12] . We will include the following for our public release "WARNING: These are base models and not aligned with post-training. They are provided as is and intended as research artifacts only." However, even as research artifacts, we recognize that models can still be misused by malicious actors or can be harmful to benevolent actors. When deciding to release our models and experiments, we considered (i) the benefit to the scientific community and (ii) the benchmark performance relative to other models that have already been released. For (i) we feel that our testbed is of use to others in the community who want to do scaling research, but do not necessarily have the means to train these model artifacts themselves. Hence, we predict (and hope) releasing all models and experiments will be helpful to others wanting to participate in scaling research. For (ii), we note that there are publicly available models [113, 114, 49] , which outperform models from our testbed and that are more likely to be widely adopted. Finally, we recognize that advancing scaling science also has potential for harm. Specifically, while we are concerned with loss and downstream task performance for popular evaluation settings, it is possible that nefarious actors may use scaling laws to help design more harmful models.In terms of licensing, we will release our code, models, and experiments under an MIT licence, which is also attached to our supplementary submission. [132, 131, 118] reading comprehension multiple choice 3 0.25 AGIEval LSAT RC [132, 131, 118] reading comprehension multiple choice 3 0.25 AGIEval SAT English [132] reading comprehension multiple choice 3 0.25 ARC-Challenge [23] world knowledge multiple choice 10 0.25 ARC-Easy [23] world knowledge multiple choice 10 0.25 BBQ [79] safety multiple choice 3 0.50 BIG-bench: CS algorithms [11] symbolic problem solving language modeling 10 0.00 BIG-bench: Conceptual combinations [11] language understanding multiple choice 10 0.25 BIG-bench: Conlang translation [11] language understanding language modeling 0 0.00 BIG-bench: Dyck languages [11] symbolic problem solving language modeling 10 0.00 BIG-bench: Elementary math QA [11] symbolic problem solving multiple choice 10 0.25 BIG-bench: Language identification [11] language understanding multiple choice 10 0.25 BIG-bench: Logical deduction [11] symbolic problem solving multiple choice 10 0.25 BIG-bench: Misconceptions [11] world knowledge multiple choice 10 0.50 BIG-bench: Novel Concepts [11] commonsense reasoning multiple choice 10 0.25 BIG-bench: Operators [11] symbolic problem solving language modeling 10 0.00 BIG-bench: QA WikiData [11] world knowledge language modeling 10 0.00 BIG-bench: Repeat copy logic [11] symbolic problem solving language modeling 10 0.00 BIG-bench: Strange stories [11] commonsense reasoning multiple choice 10 0.50 BIG-bench: Strategy QA [11] commonsense reasoning multiple choice 10 0.50 BIG-bench: Understanding fables [11] reading comprehension multiple choice 10 0.25 BoolQ [21] reading comprehension multiple choice 10 0.50 COPA [92] commonsense reasoning multiple choice 0 0.50 CoQA [91] reading comprehension language modeling 0 0.00 Commonsense QA [107] commonsense reasoning multiple choice 10 0.25 Enterprise PII classification [81] safety multiple choice 10 0.50 HellaSwag (10-shot) [126] language understanding multiple choice 10 0.25 HellaSwag (zero-shot) [126] language understanding multiple choice 0 0.25 Jeopardy [69] world knowledge language modeling 10 0.00 LAMBADA [77] language understanding language modeling 0 0.00 LogiQA [58] symbolic problem solving multiple choice 10 0.25 MMLU (5-shot) [40] world knowledge multiple choice 5 0.25 MMLU (zero-shot) [40] world knowledge multiple choice 0 0.25 MathQA [5] symbolic problem solving multiple choice 10 0.25 OpenBook QA [68] commonsense reasoning multiple choice 0 0.25 PIQA [14] commonsense reasoning multiple choice 10 0.50 PubMed QA Labeled [50] reading comprehension language modeling 10 0.00 SIQA [97] commonsense reasoning multiple choice 10 0.50 SQuAD [90] reading comprehension language modeling 10 0.00 Simple Arithmetic: NoSpaces [69] symbolic problem solving language modeling 10 0.00 Simple Arithmetic: WithSpaces [69] symbolic problem solving language modeling 10 0.00 WinoGender MC: Female [94] safety multiple choice 10 0.50 WinoGender MC: Male [94] safety multiple choice 10 0.50 WinoGrande [95] language understanding schema 0 0.50 WinoGrand [55] language understanding schema 0 0.50 4) and ( 5) using configurations from Table 1 .Training dataset Fit for Equation (4): L(C, M ) = Fit for Equation (5): Guidelines:EQUATION• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We are transparent about how many GPU hours it takes to construct our scaling laws and train our models (e.g., in Table 1 ).Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We have reviewed the code of ethics and feel that our research abides by this code in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]  
 
 Appendix:  


Decision: The paper proposes a new LLM scaling law covering over-training regime rather than a compute-optimal regime which is widely used in latest LLM training. It also covers validation loss instead of training loss, and downstream performance as well. The authors claim this significantly saves compute.

The reviewers commonly appreciate that it is an important problem in the field, and the paper provides an useful solution. ("These results, including equations and practical outcomes, are beneficial for researchers and engineers developing large language models.", "The setting considered by the paper is very relevant at the moment", "I think this is a solid paper that attempts to answer an important question."). Reviewer RC8r also appreciate it's cleared written and well presented.

There was a common question around the novelty of the paper if the existing scaling law could already provide the information. The authors provides a clarification that the existing scaling laws actually show much larger error in the over-training regime. There was also a common question if the scaling is only shown on small models. The authors provide a clarification that the current trend tends to focus more on small language models and it could be more optimal for the inference. Most of the other questions raised by the reviewers are addressed by the authors.

Overall, the paper provides an important information for some of the models developed these days which are over-trained SLMs. However, even with the clarifications from the authors, it couldn't provide information about larger models and the studies on the scaling law itself not unique. For those reasons, the authors couldn't convince the reviewers to make the scores to meet the NeurIPS publication bar.

Reviews: review: paper_summary: paper_summary: This paper investigates the power laws (scaling laws) of neural language models, particularly from the perspective of over-training and the relationship between validation loss (perplexity) and NLP downstream tasks. The authors define over-training as the situation where runs consume large amounts of computational resources, and they introduce a token multiplier, M. It is computed by D / N, where D is the number of training data tokens and N is the number of parameters. Through various model training setups and three different training corpora, the authors demonstrate that the validation loss can be computed and predicted using an equation that includes M. They also introduce another equation that illustrates the relationship between validation loss and downstream task error.
strength_weakness: strength_weakness: The authors mentioned several limitations and future work in the paper. I agree with them, and especially the ‘scaling up’ part is the primary concern of this paper. The model sizes range from 0.011B to 6.9B, but open-source models are larger than these sizes - for instance, Llama 2 starts at 7B, and Llama 3 starts at 8B [1]. Furthermore, model size is crucial for techniques such as CoT [2]. I hope to hear the authors' opinions on this concern.
questions: questions: - M could be related to overfitting if N is large but D is not sufficient. The authors mentioned that when M equals 5 (under-training), the scaling becomes unreliable. I assume this is because the models are not well trained—underfitting. Could you explain and discuss the relationship between over-training and overfitting?

- (Minor) It is hard to distinguish between M=320 and M=640 in the figures. Could you change the colors to make them more distinguishable?
limitations: limitations: Yes
paper_summary: paper_summary: The authors propose a scaling law for the “Chinchilla over-trained” regime where models are trained on many more tokens (in this paper, up to 30x) than Chinchilla-optimal. They motivate a scaling law relating pre-training compute and “over-training” to validation loss. They empirically demonstrate that the proposed scaling law accurately predicts the validation loss of a 1.4B 32x over-trained model and a 6.9B Chinchilla-optimal model. They then study a simple scaling law relating perplexity to downstream benchmark error. They select a subset of 17 benchmarks for which a 154M parameter models performs 10% above random chance accuracy, and show show that average downstream error of the 1.4B and 6.9B models is predictable.
strength_weakness: strength_weakness: My main concerns are twofold: authors do not compare with the standard scaling laws of Kaplan et al. (fitted on their own testbed), and it is unclear how authors choose the models used to fit their scaling laws (Table 1).

Authors do not compare their over-trained scaling law with that proposed by Kaplan et al. The authors could fit this scaling law as in Hoffman et al. , Section 3.3, without any additional model training. Specifically, how well can the standard Kaplan et al. law predict the validation loss of the 1.4B over-trained model, when fitted on the model testbed with N < 1B described in Section 3.2?

Regarding the claim that validation loss (resp accuracy) is predictable with 300x (resp. 20x) less compute, I find this misleading, since authors train and evaluate a reasonably large model testbed, but only report the compute required for the 5 (resp. 6) models that they ultimately choose for the fit.  The authors do not discuss how this “train”/“test” split was chosen. Clearly, the train/test split should be chosen before seeing the evaluation results for any of the models, rather than including models until the fit seems "good enough", or choosing the smallest subset for which the fit is “good enough”. Otherwise, both the claim of 300x/20x compute as well as Figure 5 are misleading. Similarly, Figure 1 would be misleading, and it should include all models with N < 1.4B. 

The authors consider token multiplier M <= 640, however current models are even more overtrained. If am not mistaken, for Llama 3 8B, M ~= 2000. Demonstrating the validity of the proposed scaling laws for the amount of over-training of current models would have been ideal, even at smaller model scales.

Lastly, the proposed scaling laws are validated at substantially lower compute scales than current models with publicly available weights. It would have been ideal to at least see results for over-trained 7B models. I understand that the experiments presented in the paper already require a substantially amount of compute, and more closely matching the compute scales of recent models would be unfeasible for most research labs — therefore I am not taking this point into consideration when scoring the paper.
questions: questions: * How well can the standard Kaplan et al. law predict the validation loss of the 1.4B over-trained model, when fitted on the model testbed with N < 1B described in Section 3.2?
* How were the models in Table 1 chosen? Were they chosen prior to the other models being evaluated? Was it apparent that including only one over-trained model, namely 11M, 320M, would suffice?
* Figure 18 shows that HellaSwag and ARC-Easy scores are very correlated with validation loss. However, Table 2 shows very large individual top-1 error for HellaSwag and ARC-Easy. How can this be?
limitations: limitations: Limitations are adequately discussed.
paper_summary: paper_summary: While existing scaling law studies look at compute-optimal pretraining, this paper considers scaling laws in the context of both pretraining and downstream performance. They perform scaling experiments and find that performance is predictable even in overtraining, and average downstream performance is also predictable.
strength_weakness: strength_weakness: To my mind, the largest weakness of this paper is the lack of novelty. In particular, my understanding is that the most important findings are that (1) performance is predictable in overtraining, and (2) average downstream performance is predictable. I’m not sure why (1) should be surprising – doesn’t a parametric scaling law, such as method 3 from the Chinchilla paper, also give the ability to predict loss when overtraining? I think the authors could improve the motivation for this consideration by providing a back of the envelope calculation: for instance, I plugged in the model size and dataset size for Gopher 280B into the Chinchilla scaling law and got a predicted test ppl of ~7.3 on MassiveText. However, Gopher actually had a (validation) perplexity of ~8.1, this constitutes a relative error of around 10% – substantially larger than the relative errors obtained by the authors of this paper. If the authors can provide an argument of this sort I'd find that helpful. 

I thought (2) was a more interesting claim, but I’ve seen this analyzed in Owen 2024 (https://arxiv.org/pdf/2401.04757), albeit with a different methodology. As such, I felt that the core results of the paper weren’t very novel. However, I’d be happy to update my assessment if the authors can provide evidence that my understanding is incorrect. 

One interesting point that the authors mentioned is that performance on individual tasks is less predictable. But this is only mentioned in passing, and I felt that it could be expanded upon a fair bit. What are the implications of this observation? Are there any patterns for which individual tasks are or aren’t predictable? 

I’m slightly concerned about data leakage being an issue for the downstream tasks, given that the training data (The Pile and RedPajama) covers a wide swath of the internet, and some of the downstream benchmarks have been criticized for data leakage or having label errors. 

Minor comment: The last paragraph of section 5 is a bit confusing: “There has been a rise in over-trained models [113, 114] and accompanying massive datasets [112, 82, 104, 3]. For example, Chinchilla 70B [45] is trained with a token multiplier of 20, while LLaMA-2 7B [114] uses a token multiplier of 290.” This makes it sound a bit like Chinchilla is overtrained, which I don’t think the authors are trying to say, so I’d suggest something like the following instead: “For example, while Chinchilla 70B is trained compute-optimally with a token multiplier of 20, LLaMA-2 7B…”
questions: questions: - Doesn’t a standard parametric scaling law fit, such as method 3 from the Chinchilla paper, also allow one to predict loss when overtraining? 
- How do the results from this paper differ from those in this other paper, which also looks at downstream performance? https://arxiv.org/pdf/2401.04757 
- If I plug in the coefficients from the Chinchilla scaling law into equation 8, with $\alpha = 0.34$ and $\beta = 0.28$, I find that the predicted value for $\eta$ is $\frac{\alpha \beta}{\alpha + \beta} \approx 0.154$. In comparison, the values of $\eta$ in table 6 are generally around 0.25. What’s the cause of this difference, and how do you know? 
- Did the authors consider alternative functional forms for the downstream performance scaling law, besides the negative exponential? 
- Did the authors check for data leakage? How?
limitations: limitations: I felt that the authors did a good job describing some of the limitations of their work.

Comments: comment: We apologize for not addressing your concerns; we unfortunately misunderstood your original messages! This said, we appreciate you attending to the other reviews and still reconsidering your score! We aim to address your outstanding concerns below.

**Comparison to Kaplan et al.** There is no reason we cannot provide this comparison. We unfortunately misunderstood your original comment, but please see the table below for the requested results! Note, Equations (3) and (4) in the paper are the same (i.e., both have four free parameters, with the relation given in L108). Concretely, "we reparameterize Equation (3) in terms of compute $C = 6ND$ and a token multiplier $M = D/N$ [to] get, [Equation (4)]" (L107). We include Equation (4) to provide the reader with intuition about what *should* happen when one over-trains (L110-114). We are also careful not to claim Equation (4) as a novel scaling law. For example, in the introduction: "We explain our observations by reparameterizing existing scaling laws in relation to the amount of over-training" (L42-43). Our main contributions with respect to over-training are empirical (as detailed in our response to reviewer CG6A).

| Scaling Form | Model | Relative Error on C4 eval loss|
|--------------|-------|--------------------------------|
| $L(N,D) = AN^{-\alpha} + BD^{-\beta}$ (Kaplan et al. like) | open_lm_1b (M=640.0) | 16.1281% |
| | open_lm_7b (M=20.0) | 17.2352% |
| $L(C, M) = E + (aM^{\eta} + bM^{-\eta}) C^{-\eta}$ (our reparameterization of Hoffmann et al.) | open_lm_1b (M=640.0) | 0.7103% |
| | open_lm_7b (M=20.0) | 0.7320% |


**Reported compute savings.** Sorry that we misunderstood your comment! We understand your skepticism about our 300x/20x savings claim and ultimately your concern about cherry-picking. We reiterate, using the configurations in Table 1, a practitioner should expect to predict runs similar to our N=1.4B and N=6.9B runs with the claimed compute savings. We will refine the wording to make sure this claim is more precise and clear.

With respect to the entire scaling testbed, we do feel the need to clarify that the results are not cherry-picked. For the sake of transparency, we detail key moments in our development timeline, specifically for the over-training portion of the project. We hope this will provide clarity and help address your very legitimate concerns.
1. Chinchilla Approach 3 hints that scaling should be reliable; however, they do not directly measure (in terms of relative error), how good scaling predictions are emperically. Hence, we started our investigation by trying to reproduce Chinchilla. This resulted in a large grid search, as we did not understand how to set hyperparameters to get reliable scaling trends. The grid search ultimately culminated in Section 3.2 of our writeup. Note, at this stage, we had not touched any of our downstream evaluations or our main training distributions. In fact, at this stage we did not even have a project idea! We were, however, largely able to reproduce Chinchilla Approach 3 and observed reliable scaling for compute-optimal models.
2. At this stage, we were looking for promising research directions. Motivated by many open-weight releases at the time, we thought it would be nice to understand what happens empirically when one over-trains models. We hypothesized that we should see “parallel lines” in the log-log plot based on the functional form of Approach 3. Hence, we chose a subset of our entire grid search configurations (as detailed in Section 3.2), tokenized C4, and trained these configurations for various token multipliers (i.e., number of tokens). This set of training runs eventually contributed to our scaling testbed. We did in fact see parallel lines and then set out to understand if this phenomenon could be reliably extrapolated with some of our smaller-scale runs. 
3. At first, we thought we could predict over-trained behavior with only Chinchilla optimal models. However, after playing around with equations and parameterizing with token multipliers, we realized that to extrapolate in M, we would have to fit to at least 1 data point where $M \neq 20$ (i.e., at least one non-compute-optimal run). We knew that we would have enough compute to over-train our largest run at $M=640, N=1.4$B. From our scaling testbed we chose the smallest config ($N=11$M) with the largest token multiplier ($M=320$) such that we could still probe for extrapolation in $N, M$. The other models in Table 1 are, somewhat arbitrarily, Chinchilla optimal.
4. We constructed Appx. Tables 14-16 to ablate our decisions. Our meta-analysis here is that the combination of our compute constraints and intuition that small models (e.g., over-trained 11M models) could give reasonable signal were reasonable.
Thanks! We appreciate you reconsidering your score post-rebuttal!
Thank you for your response. My two main concerns were not effectively addressed. Having read the other reviewers’ reviews, I’ll increase my score, since I agree that the paper does show that language models scale reliably with over-training.

**Comparison to Kaplan et al.**: What I mean is to consider how well the “standard” functional form used by prior work, which you write in Equation 3, can predict the over-training regime. Without this comparison, it is simply not possible to judge whether the proposed reparametrization in Equation 4 is a valuable contribution in itself or not. Why should practitioners fit Equation 4 and not Equation 3? Note that Equation 3 is the power law of Kaplan (Equation 4.1 in the Kaplan paper) + the irreducible loss term. I included in my review “The authors could fit this scaling law as in Hoffman et al. , Section 3.3, without any additional model training”. Reviewer CG6A echoes a similar weakness “doesn’t a parametric scaling law, such as method 3 from the Chinchilla paper, also give the ability to predict loss when overtraining?”. I fail to understand why you cannot provide this comparison. You do not need to train any additional models, you have the (N, D, L) triplets. It is just a matter of fitting Equation 3.

**Reported compute savings**: I am not referring to the compute spent on the grid search for hyper-parameter tuning. I am referring to the compute required to train the scaling test bed itself, bar the N=1.4B and N=6.9B models which are the ones you ultimately aim to extrapolate to. “Hence, we chose a datapoint M=320 and happened to do so at the smallest parameter scale to save compute”. This is not convincing. A single M=320 datapoint happened to be good enough for the fit. But you did not happen to train a single M=320 model, you trained many models with M > 20. The concern is in choosing the fit/predict split after the models were already trained and evaluated. Otherwise, the stated compute savings and Figure 5 altogether can be misleading. 

> we plot relative error vs. many different potential configurations in Appx. Tables 14-16

These plots precisely illustrate my point. Looking at the C4 eval plots, the blue stars are in the Pareto frontier of relative error vs FLOPs spent. The 300x/20x multipliers are not representative of what one would typically obtain, they represent a “best” case scenario, potentially indicative of cherry-picking regarding the fit/predict split.

**Over-training in Llama3 8B and larger models**: I maintain my initial positive assessment that the experiments are of a large enough scale for a proof of concept.
Thanks for the responses to my questions! I believe this addresses my main concerns. I also think I was being a bit harsh on the novelty point, because even though there's a lot of overlap in results the methodology is quite different and probably more reliable than Owen 2024. As such, I'll increase my score.
Thanks! Given that we've addressed the concerns, we were wondering if you might reconsider our score. We understand this decision is entirely at your discretion and are thankful for your attention regardless!
Thank you for your responses and the revised figures. Your response have addressed my concerns and clarified my questions. Good luck!