Language models scale reliably with over-training and on downstream tasks

## Abstract

Abstract Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments.
However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated.
For instance, scaling is usually studied in the compute-optimal training regime (i.e., “Chinchilla optimal” regime).
In contrast, models are often over-trained to reduce inference costs.
Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance.
To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions.
First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters.
This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32 × \times × over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)—each from experiments that take 300 × \times × less compute.
Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law.
We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20 × \times × less compute.
Our experiments are available at https://github.com/mlfoundations/scaling .

## 1 Introduction

Figure: Figure 1: Reliable scaling with over-training and on downstream error prediction. *(left)* We fit a scaling law for model validation loss, parameterized by (i) a token multiplier $M=N/D$, which is the ratio of training tokens $D$ to parameters $N$ and (ii) the compute $C$ in FLOPs used to train a model, approximated by $C=6ND$. Larger values of $M$ specify more over-training. We are able to extrapolate, in both $N$ and $M$, the validation performance of models requiring more than $300\times$ the training compute used to construct the scaling law. *(right)* We also fit a scaling law to predict average downstream top-1 error as a function of validation loss. We find that fitting scaling laws for downstream error benefits from using more expensive models when compared to fitting for loss prediction. We predict the average error over 17 downstream tasks for models trained with over 20$\times$ the compute. For this figure, we train all models on RedPajama [].
Refer to caption: x1.png

Training large language models is expensive.
Furthermore, training high-quality models requires a complex recipe of algorithmic techniques and training data.
To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger model and data regimes via scaling laws.
With reliable extrapolation, it is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run.
Indeed, this workflow has become commonplace for training state-of-the-art language models like Chinchilla 70B [], PaLM 540B [], GPT-4 [], and many others.

Despite their importance for model development, published scaling laws differ from the goals of training state-of-the-art models in important ways.
For instance, scaling studies usually focus on the compute-optimal training regime (“Chinchilla optimality” []), where model and dataset size are set to yield minimum loss for a given compute budget. However, this setting ignores inference costs. As larger models are more expensive at inference, it is now common practice to over-train smaller models []. Another potential mismatch is that most scaling laws quantify model performance by perplexity in next-token prediction instead of accuracy on widely used benchmark datasets. However, practitioners usually turn to benchmark performance, not loss, to compare models.

In this paper, we conduct an extensive set of experiments to address both scaling in the over-trained regime and benchmark performance prediction.

Motivated by the practice of training beyond compute-optimality, we first investigate whether scaling follows reliable trends in the over-trained regime.
We notice, as implied by Hoffmann et al. [], for a set of models of different sizes trained with a constant ratio of tokens to parameters, models’ reducible loss $L^{\prime}$ [, ] follows a power law ($L^{\prime}=\lambda\cdot C^{-\eta}$) in the amount of training compute $C$.
We find that as one increases the ratio of tokens to parameters, corresponding to more over-training, the scaling exponent $\eta$ remains about the same, while the scalar $\lambda$ changes. We explain our observations by reparameterizing existing scaling laws in relation to the amount of over-training.

To establish empirically that scaling *extrapolates* in the over-trained regime, we further experiment with a testbed of 104 models, trained from scratch on three different datasets: C4 [, ], RedPajama [], and RefinedWeb [].
We find that scaling laws fit to small models can accurately predict the performance of larger models that undergo more over-training.
Figure 1 *(left)* illustrates our main over-training result, where we invest $2.4e19$ FLOPs to extrapolate the C4 validation performance of a 1.4B parameter model trained on 900B tokens, which requires $300\times$ more compute to train.

In addition to over-training, we also investigate if scaling laws can predict the performance of a model on downstream tasks.
We establish a power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks.
While it can be difficult to predict the error on individual tasks, we find it possible to predict aggregate performance from a model’s perplexity among models trained on the same training data.
Figure 1 *(right)* presents our main downstream error prediction result, where we invest $2.7e20$ FLOPs to predict the average top-1 error over a set of downstream tasks to within 1 percentage point for a 6.9B compute-optimal model, which requires $20\times$ more compute to train.

Our results suggest that the proposed scaling laws are promising to derisk (i) the effects of over-training models and (ii) the downstream performance of scaling up training recipes.
To facilitate further research on reliable scaling, we provide all results of our experiments at [https://github.com/mlfoundations/scaling](https://github.com/mlfoundations/scaling).

## 2 Developing scaling laws for over-training and downstream tasks

In this section, we develop scaling laws to predict over-trained and downstream performance.
First, we provide key definitions (Section 2.1).
We next present a scaling law for over-training drawing on empirical observation and prior work (Section 2.2).
To connect loss scaling and downstream error prediction, we observe that average top-1 error decreases exponentially as a function of validation loss, which we formalize as a novel scaling law (Section 2.3).
In later sections, we build an experimental setup (Section 3) to quantify the extent to which our scaling laws extrapolate reliably (Section 4).

### 2.1 Preliminaries

#### Scaling laws for loss.

Typically, scaling laws predict model loss $L$ as a function of the compute $C$ in FLOPs used for training.
If one increases the number of parameters $N$ in a model or the number of tokens $D$ that a model is trained on, compute requirements naturally increase.
Hence, we assume $C$ is a function of $N,D$.
Following Kaplan et al. [], we use the approximation $C=6ND$, which Hoffmann et al. [] independently verify.
We consider,

$$ $\displaystyle L(C)=E+L^{\prime}(C),$ (1) $$

where $E$ is an *irreducible loss* and $L^{\prime}$ is the *reducible loss*.
$E$ captures the Bayes error or minimum possible loss achievable on the validation domain.
The $L^{\prime}(C)$ term captures what can possibly be learned about the validation domain by training on a source domain.
$L^{\prime}(C)$ should approach zero with increased training data and model capacity.
$L^{\prime}(C)$ is often assumed to follow a power law: $L^{\prime}(C)=\lambda\cdot C^{-\eta}$ (i.a.,  Hestness et al. [], OpenAI []).
It is also often helpful to consider a power law in a $\log$-$\log$ plot, where it appears as a line with slope $-\eta$ and $y$-intercept $\log{(\lambda)}$.

#### Token multipliers.

We define a token multiplier $M=D/N$ as the ratio of training tokens to model parameters for notational convenience. $M$ allows us to consider fixed relationships between $D$ and $N$ even as a model gets bigger (i.e., as $N$ becomes larger).

#### Compute-optimal training.

Hoffmann et al. [] establish compute-optimal training, where, for any compute budget $H$, the allocation of parameters and tokens is given by,

$$ $\displaystyle\arg\min_{N,D}L(N,D)\text{ s.t. }C(N,D)=H.$ (2) $$

To solve for the optimal $N^{*},D^{*}$, one can sweep $N,D$ for each compute budget, retaining the best configurations.
Hoffmann et al. [] find that as the compute budget increases, $N^{*}$ and $D^{*}$ scale roughly evenly.
Assuming equal scaling, there is a fixed compute-optimal token multiplier $M^{*}=D^{*}/N^{*}$ per training distribution.

#### Over-training.

We define over-training as the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number of tokens (i.e., $M>M^{*}$).
While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus incur less inference cost.

Figure: Figure 2: Scaling in the over-trained regime follows consistent power law exponents. We notice parallel lines in the $\log$-$\log$ plots of reducible loss vs. training compute for a range of token multipliers $M$, which give the ratio of training tokens to model parameters. Larger $M$ corresponds to more over-training. For a power law giving reducible loss as a function of compute: $L^{\prime}(C)=\lambda\cdot C^{-\eta}$, the exponent $\eta$ remains relatively constant resulting in lines with approximately fixed slope (Figure 17). The scalar $\lambda$ that determines the $y$-intercept, however, shifts with different token multipliers. This suggests $\lambda$ is a function of the token multiplier, while $\eta$ is not.
Refer to caption: x2.png

### 2.2 Scaling laws for over-training

To propose a scaling law for over-trained models, we first turn to empirical observation.
We train four model configurations with parameter counts between 0.011B and 0.411B for token multipliers $M$ between 20 and 640, where $M=20$ points lie roughly on the compute-optimal frontier, and larger $M$ corresponds to more over-training.
We defer experimental details to Section 3 to focus on our observations first.
In Figure 2, we show loss against compute in a $\log$-$\log$ plot for the models trained on three datasets and evaluated on the C4 eval set.
We notice parallel lines when fitting power laws to the reducible loss, which suggests a near-constant scaling exponent even with increased over-training.
This indicates that scaling behavior should be describable in the amount of over-training.

In search of an analytic expression for the observations in Figure 2, we consider existing scaling literature.
A common functional form for the risk of a model, as proposed in prior work [, ] is,

$$ $\displaystyle L(N,D)=E+AN^{-\alpha}+BD^{-\beta}.$ (3) $$

Recall from Section 2.1, $N$ is the number of parameters and $D$ the number of training tokens.
The constants $E,A,\alpha,B,\beta$ are fit from data.
By fitting this parametric form, Hoffmann et al. [] find that scaling exponents $\alpha$ and $\beta$ are roughly equal, suggesting that one should scale $N$ and $D$ equally as compute increases. Hence, we assume $\alpha=\beta$.
With this assumption, we reparameterize Equation (3) in terms of compute $C=6ND$ and a token multiplier $M=D/N$.
We get,

$$ $\displaystyle L(C,M)=E+\left(aM^{\eta}+bM^{-\eta}\right)C^{-\eta},$ (4) $$

where $\eta=\alpha/2$, $a=A(1/6)^{-\eta}$, $b=B(1/6)^{-\eta}$ gives the relation to Equation (3).
For a complete derivation, see Appendix B.

Equation (4) has the following interpretation:
(i) The scaling exponent $\eta$ is not dependent on $M$. Thus, we always expect lines with the same slope in the $\log$-$\log$ plot—as in Figure 2.
(ii) The term $aM^{\eta}+bM^{-\eta}$ determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the $\log$-$\log$ plot for the range of $M$ we consider—also consistent with Figure 2.

Recall that we make the assumption $\alpha=\beta$, which implies equal scaling of parameters and tokens as more compute is available.
However, as explained in Appendix B, even if $\alpha\neq\beta$, we get a parameterization that implies the power-law exponent remains constant with over-training.

Figure: Figure 3: Average top-1 error scales as a function of loss. We plot models trained on three datasets and notice an exponential decay of average top-1 error as C4 eval loss, on the x-axis, decreases. We consider on the y-axes average error on 17 evaluations where performance is at least 10 points above random chance for at least one 0.154B scale model. These observations suggest that average top-1 error should be predictable with reliable loss estimates.
Refer to caption: x3.png

### 2.3 Scaling laws for downstream error

Scaling is typically studied in the context of loss [, , ], which Schaeffer et al. [] note is smoother than metrics like accuracy.
However, practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss on perplexity evaluation sets.
To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure 2.
In Figure 3, we plot average downstream top-1 errors over evaluations sourced from LLM-Foundry [] against the C4 eval loss.
We defer details of the setup to Section 3 to focus here on a key observation: average error appears to follow exponential decay as loss decreases.

Based on the exponential decay we observe in Figure 3, we propose the following relationship between downstream average top-1 error Err and loss $L$,

$$ $\displaystyle\textsf{Err}(L)=\epsilon-k\cdot\exp{(-\gamma L)},$ (5) $$

where $\epsilon,k,\gamma$ are fit from data.
Equation (5) also has an interpretation in terms of model perplexity $\textsf{PP}(L)=\exp{(L})$,

$$ $\displaystyle\textsf{Err}(\textsf{PP})=\epsilon-k\cdot\textsf{PP}^{-\gamma}.$ (6) $$

Namely, Err follows a power law in PP that is bounded from above by $\epsilon$ signifying arbitrarily high error and from below by $\epsilon-k\cdot\exp(-\gamma E)$, where $E$ is the Bayes error from Equation (4).

Equation (5) in conjunction with Equation (4) suggests a three-step method to predict Err as a function of compute and the amount of over-training.
For choices of training and validation distributions,
(i) fit a scaling law to Equation (4) using triplets of compute $C$, token multiplier $M$, and measured loss $L$ on a validation set to yield $(C,M)\mapsto L$.
(ii) Fit a scaling law to Equation (5) using pairs of loss $L$ and downstream error Err for models to get $L\mapsto\textsf{Err}$.
(iii) Chain predictions to get $(C,M)\mapsto\textsf{Err}$.

## 3 Constructing a scaling testbed

Figure: Figure 4: Search, filter, fit: A recipe for selecting configurations for scaling. *(left)* To generate the final configurations presented in Table 3, we run a 435 model grid search over model width, hidden dimension, number of attention heads, batch size, and warmup steps. All models are trained near compute-optimally. *(center)* We plot the efficient frontier of models, which appear to follow a trend, excluding models from $5.2\times 10^{16}$ to $5.2\times 10^{17}$, which fall below the trend. *(right)* We fit a power law with irreducible error to the remaining configurations, picking four configurations that closely track the full model suite (“Selected models”). These models extrapolate the performance of 1.4B, 6.9B target models. Shaded regions represent bootstrap 95% confidence intervals.
Refer to caption: x4.png

In this section, we discuss our experimental setup to test the predictions suggested by Equations (4) and (5).
We first present our general language modeling setup (Section 3.1).
Next, we discuss our strategy for determining model configurations for our scaling investigation (Section 3.2) and fitting scaling laws (Section 3.3).
We then present metrics to validate how well scaling laws predict loss and downstream performance (Section 3.4).

### 3.1 Training setup

We train transformers [] for next token prediction, based on architectures like GPT-2 [] and LLaMA [].
We employ GPT-NeoX [] as a standardized tokenizer for all data.
See Appendix C for architecture, optimization, and hyperparameter details.

### 3.2 Model configurations

To get final configurations for the 0.011B to 0.411B parameter models plotted in Figures 2 and 3, we first conduct a wide grid search over a total of 435 models, trained from scratch, from 0.01B to 0.5B parameters (Figure 4 *(left)*).
We train on the original OpenLM data mix [], which largely consists of RedPajama [] and The Pile [].
While we eventually plan to over-train models, at this step we search for *base configurations* near compute-optimality.
We train on 20 tokens per parameter ($M=20$), which, in early experiments, gives models near the compute-optimal frontier.
This is similar to findings in Hoffmann et al. []’s Table 3, which suggests that $M=20$ is near-optimal for the Chinchilla experimental setup.

To find maximally performant small-scale models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size.
Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles.
We find in early experiments that qk-LayerNorm makes models less sensitive to learning rate, which is a phenomenon Wortsman et al. [] report in their Figure 1.
Hence, we fix the learning rate (3$e$-3) for our sweeps.
We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at $M=20$, retaining the best configurations.

At this point, we have many models, several of which give poor performance; following prior work [, ], we want to keep only models that give best performance.
Hence, in Figure 4 *(center)*, we filter out models that do not lie on the Pareto frontier.
While there appears to be a general trend, configurations between $5.2\times 10^{16}$ and $5.2\times 10^{17}$ FLOPs lie below the frontier established by other models.
We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of-two batch sizes.
We provide support for this hypothesis in Appendix F, but opt to remove these models from our investigation.

To ensure tractable compute requirements for our scaling experiments, we require a subset of models that follows the trend of the entire Pareto frontier.
In Figure 4 *(right)*, we fit trends to the Pareto models and to a subset of four models.
We notice that the trends closely predict both the performance of the 1.4B and 6.9B models, suggesting that our small-scale configurations reliably extrapolate in the compute-optimal setting.

Moving forward, we do not tune hyperparameters for other token multipliers (i.e., $M\neq 20$), on other training or evaluation distributions, or on validation sets for downstream tasks.
For more details including specific hyperparameters, see Appendix D.

To create our scaling testbed, we start with the four small-scale, base configurations from our grid search: $N\in\{0.011\text{B},0.079\text{B},0.154\text{B},0.411\text{B}\}$.
To ensure our conclusions are not particular to a single training distribution, we train models on each of C4 [, ], RedPajama [], and RefinedWeb [], which have 138B, 1.15T, and 600B tokens, respectively, for different token multipliers $M\in\{5,10,20,40,80,160,320,640\}$.
We omit runs that require more tokens than are present in a dataset (i.e., $N=0.411\text{B},M=640$ for C4).
We additionally train $N=1.4$B models at $M=20$ and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for RedPajama, and 320 for RefinedWeb).
We train $N=6.9\text{B},M=20$ models on each dataset given the relevance of 7B parameter models [, ].
In total this results in a testbed of 104 models.

### 3.3 Fitting scaling laws

**Table 1: Default number of parameters $N$ and token multiplier $M$ to fit our scaling laws. We invest $\sim$100 A100 hours to fit Equation (4) and $\sim$1,000 A100 hours to fit Equation (5).**
| $N$ | $M$ | Used to fit Equation (4) | Used to fit Equation (5) |
| --- | --- | --- | --- |
| 0.011B | 20 | ✓ | ✓ |
| 0.079B | 20 | ✓ | ✓ |
| 0.154B | 20 | ✓ | ✓ |
| 0.411B | 20 | ✓ | ✓ |
| 0.011B | 320 | ✓ | ✓ |
| 1.4B | 20 | ✗ | ✓ |
| Total compute $C$ [FLOPs] | 2.4$e$19 | 2.7$e$20 |  |

We fit Equation (4) to approximate $E,a,b,\eta$ using curve-fitting in SciPy [] (i.e., Levenberg-Marquardt to minimize non-linear least squares).
We repeat this process to fit Equation (5) to approximate $\epsilon,k,\gamma$.
We invest $\sim$100 A100 hours to train the models required to fit a scaling law for loss and $\sim$1,000 A100 hours for a corresponding law for downstream error.
Unless otherwise specified, we fit to the $N,M$ pairs in Table 1, which are a subset of our full testbed.
Our configurations allow us to test for extrapolation to the $N=1.4\text{B},M=640$ (900B token) and the $N=6.9\text{B},M=20$ (138B token) regimes.

### 3.4 Evaluation setup

#### Evaluation datasets.

Unless otherwise stated, our default validation loss dataset is C4 eval.
For downstream tasks, we adopt a subset from 46 tasks from LLM-foundry [], which includes standard tasks with both zero-shot and few-shot evaluations.
Specifically, we consider a 17-task subset where, for each evaluation, at least one 0.154B scale model—trained with as many as 99B tokens—gets 10 percentage points above chance accuracy:
ARC-Easy [],
BIG-bench: CS algorithms [],
BIG-bench: Dyck languages [],
BIG-bench: Novel Concepts [],
BIG-bench: Operators [],
BIG-bench: QA WikiData [],
BoolQ [],
Commonsense QA [],
COPA [],
CoQA [],
HellaSwag (zero-shot) [],
HellaSwag (10-shot) [],
LAMBADA [],
PIQA [],
PubMed QA Labeled [],
SQuAD [],
and
WinoGrand [].
For more details on evaluation datasets see Appendix E.
We focus on this subset to ensure we are measuring signal, not noise.
Including downstream tasks like MMLU [], where performance is close to random chance, however, does not invalidate our results as we show in our evaluation set ablations (Appendix F).

#### Metrics.

We consider three main metrics:
*Validation loss*, which is the cross entropy between a model’s output and the one-hot ground truth token, averaged over all tokens in a sequence and over all sequences in a dataset.
*Average top-1 error*, which is a uniform average over the 17 downstream evaluations, as mentioned in the above paragraph.
To measure how good a prediction $\zeta(C,M)$ is, we measure *Relative prediction error*: $|\zeta(C,M)-\zeta_{GT}|/\zeta_{GT}$, where $\zeta$ is the predicted loss $L$ or the average top-1 error Err. $\zeta_{GT}$ is the ground truth measurement to predict.

## 4 Results: Reliable extrapolation

In this Section, we quantify the extent to which the scaling laws developed in Section 2 extrapolate larger model performance using the scaling testbed from Section 3.
By default, we fit Equations (4) and (5) to the configurations in Table 1, use C4 eval for loss, and the 17-task split from Section 3.4 for average top-1 error.

Figure: Figure 5: Relative error on C4 eval for different training distributions. Boxes highlighted in yellow correspond to pairs—number of parameters $N$, token multiplier $M$—used to fit Equation (4). Larger values of $M$ correspond to more over-training. The prediction error is low in both interpolation and extrapolation ranges. Below $N=1.4$B, empty squares correspond to runs that were not possible due to the limited dataset size for single epoch training. At $N=1.4$B we run at $M=20$ and at the largest possible multiplier. At $N=6.9$B, we run at $M=20$.
Refer to caption: x5.png

#### Over-trained performance is predictable.

We highlight our main over-training results in Figure 1 *(left)*.
Namely, we are able to extrapolate both in the number of parameters $N$ and the token multiplier $M$ to closely predict the C4 eval performance of a 1.4B parameter model trained on 900B RedPajama tokens ($N=1.4\text{B},M=640$).
Our prediction, which takes 300$\times$ less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error.
Additionally, for the $N=6.9\text{B},M=20$ run, near compute-optimal, the relative error is also 0.7%.

These results support several key takeaways.
(i) Scaling can be predictable even when one increases both the model size and the amount of over-training compared to the training runs used to fit a scaling law.
(ii) The form presented in Equation (4) is useful in practice for predicting over-trained scaling behavior.
(iii) Fitting to Equation (4) gives good prediction accuracy near compute-optimal.
More specifically, predictions are accurate both for the 1.4B over-trained model and the 6.7B compute-optimal model using a single scaling fit.

While Figure 1 explores a specific case of making predictions in the over-trained regime, we aim to understand the error profile of our predictions across training datasets, token multipliers, and number of parameters.
Hence, Figure 5 shows the relative error between ground truth loss and predicted loss on C4 eval for models in our testbed.
We notice uniformly low prediction error suggesting that predictions are accurate in many settings.

#### Average top-1 error is predictable.

Figure 1 *(right)* presents our main result in estimating scaling laws for downstream error.
Concretely, we use the models indicated in Table 1 to fit Equations (4) and (5), chaining the scaling fits to predict the average top-1 error as a function of training compute $C$ and the token multiplier $M$.
Our fits allow us to predict, using $20\times$ less compute, the downstream performance of a 6.9B model trained on 138B RedPajama tokens to within $0.05\%$ relative error and a 1.4B model trained on RedPajama 900B tokens to within $3.6\%$ relative error.

Table 2 additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law functional forms are applicable on many training datasets.
We note that while average accuracy is predictable, individual downstream task predictions are significantly more noisy.
We report relative error for more model predictions in Figures 11 and 12.
We also find that if we remove the 1.4B model for the Equation (5) fit, relative error jumps, for instance, from 0.05% to 10.64% on the 17-task split for the 6.9B, 138B token RedPajama prediction.
This highlights the importance of investing more compute when constructing scaling laws for downstream task prediction compared to loss prediction.

**Table 2: Downstream relative prediction error at 6.9B parameters and 138B tokens. While predicting accuracy on individual zero-shot downstream evaluations can be challenging (“Individual”), predicting averages across downstream datasets is accurate (“Avg.”).**
|  | Individual top-1 error | Avg. top-1 error |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Train set | ARC-E [] | LAMBADA [] | OpenBook QA [] | HellaSwag [] | 17-task split |
| C4 [, ] | 28.96% | 15.01% | 16.80% | 79.58% | 0.14% |
| RedPajama [] | 5.21% | 14.39% | 8.44% | 25.73% | 0.05% |
| RefinedWeb [] | 26.06% | 16.55% | 1.92% | 81.96% | 2.94% |

#### Under-training, out-of-distribution scaling, compute-reliability trade-offs.

In addition to our main results presented above, we include additional results in Appendix F, which we summarize here.
First, we notice that when token multipliers become too small (i.e., $M=5$) scaling becomes unreliable and lies off the trend.
Additionally, multipliers other than 20, such as 10, 40, and 80, garner points that are roughly on the compute optimal frontier (Figure 9).
This observation suggests that the compute-optimal multiplier may lie in a range rather than take a single value.
To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings.
We find that models trained on C4—English filtered—and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction gives reliable loss scaling (Figure 10).
We additionally examine the compute necessary to create accurate scaling laws, finding
that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures 15 and 16).

## 5 Related work

We review the most closely related work in this section.
For additional related work, see Appendix G.

#### Scaling laws.

Early works on scaling artificial neural networks observe predictable power-law scaling in the training set size and number of model parameters [, , ].
Alabdulmohsin et al. [] stress the importance of looking at the extrapolation regime of a scaling law.
Yang et al. [] prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. [] make analogous recommendations when scaling model depth.
Bi et al. [] propose hyperparameter aware scaling laws.
Unlike the aforementioned work, our investigation focuses on over-training and predicting downstream accuracy.

Hoffmann et al. [] investigate how the number of model parameters $N$ and training tokens $D$ should be chosen to minimize loss $L$ given a compute budget $C$.
Hoffmann et al. [] find that when scaling up $C$, both $N$ and $D$ should be scaled equally up to a multiplicative constant (i.e., $N\propto C^{\sim 0.5}$ and $D\propto C^{\sim 0.5}$) to realize compute-optimality.
Appendix C of the Chinchilla paper additionally suggests that these findings hold across three datasets.
However, Hoffmann et al. [] do not verify their scaling laws for training beyond compute-optimality, or for downstream error prediction—both of which are central to our work.

Sardana & Frankle [] propose modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimality and solve for various fixed inference budgets.
Their key finding, which is critical for our work, is that when taking into account a large enough inference budget, it is optimal to train smaller models for longer than the original Chinchilla recommendations.
Our work presupposes that over-training can be beneficial.
Instead of solving for inference-optimal schemes, we support empirically a predictive theory of scaling in the over-trained regime.
Additionally, we provide experiments across many validation and training sets.

For predicting downstream scaling beyond loss, Isik et al. [] relate the number of pre-training tokens to downstream cross-entropy and machine translation BLEU score [] after fine-tuning.
In contrast, we take a holistic approach to evaluation by looking at top-1 error over many natural language tasks.
Schaeffer et al. [] argue that emergent abilities [] are a product of non-linear metrics and propose smoother alternatives.
As a warmup for why non-linear metrics may be hard to predict, Schaeffer et al. [] consider predicting an $\ell$ length sequence exactly: $\textsf{Err}(N,\ell)\approx 1-\textsf{PP}(N)^{-\ell}$, where $N$ is the number of parameters in a model and PP is its perplexity.
This is a special case of our Equations (5) and (6), where the number of training tokens does not appear, $\epsilon=1,k=1$, and $\gamma=\ell$.
In contrast, we treat $\epsilon,k,\gamma$ as free parameters for a scaling law fit, finding that average error over downstream tasks can make for a predictable metric.

#### Over-training in popular models.

There has been a rise in over-trained models [, ] and accompanying massive datasets [, , , ]. For example, Chinchilla 70B [] is trained with a token multiplier of 20, while LLaMA-2 7B [] uses a token multiplier of 290.
In our investigation, we look at token multipliers from 5 to 640 to ensure coverage of popular models and relevance for future models that may be trained on even more tokens.

## 6 Limitations, future work, and conclusion

#### Limitations and future work.

We identify limitations, which provide motivation for future work.

- •
Hyperparameters.
While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is a need to develop scaling laws that do not require extensive hyperparameter sweeps.
- •
Scaling up. Validating the trends in this paper for even larger runs is a valuable direction. Additionally, repeating our setup for models that achieve non-trivial performance on harder evaluations like MMLU is left to future work.
- •
Scaling down. Actualizing predictable scaling with even cheaper runs is important to make this area of research more accessible, especially for downstream error prediction.
- •
Failure cases. While we present a preliminary analysis of when scaling is unreliable, future work should investigate conditions under which scaling breaks down.
- •
Post-training. It is common to employ fine-tuning interventions after pre-training, which we do not consider.
Quantifying to what degree over-training the base model provides benefits *after* post-training is an open area of research.
- •
Individual downstream task prediction.
While we find that averaging over many task error metrics can make for a predictable metric, per-task predictions are left to future work.
- •
In-the-wild performance.
Downstream task performance is a proxy for the in-the-wild user experience.
Analyzing scaling trends in the context of this experience is timely.
- •
Dataset curation. Our work only deals with existing training datasets.
Exploring dataset curation for improved model scaling is another promising direction.

#### Conclusion.

We show that the loss of over-trained models, trained past compute-optimality, is predictable. Furthermore, we propose and validate a scaling law relating loss to average downstream task performance.
We hope our work will inspire others to further examine the relationship between model training and downstream generalization.
Our testbed will be made publicly available, and we hope it will make scaling research more accessible to researchers and practitioners alike.

## Acknowledgements

SYG is supported by an NSF Graduate Research Fellowship,
GS by the Onassis Foundation - Scholarship ID: F ZS 056-1/2022-2023, and MN by the Federal Ministry of Education and Research of Germany under grant no. 01IS22094B WEST-AI.
We thank Stability AI and Toyota Research Institute (TRI) for access to compute resources.
This research has been supported by NSF Grants AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, and research gifts by Western Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, and the Stanly P. Finch Centennial Professorship in Engineering.
We also thank Kushal Arora, Alper Canberk, Mia Chiquier, Sachit Menon, Chuer Pan, Purva Tendulkar, and Mandi Zhao for valuable feedback.

## Appendix A Contributions

Names ordered alphabetically.

#### Model training and experiment babysitting.

Achal Dave (notably, the 1.4B parameter, 900B token run),
Samir Yitzhak Gadre

#### Dataloading.

Georgios Smyrnis

#### Training tokens.

Achal Dave, Alex Fang, Samir Yitzhak Gadre, Suchin Gururangan, Jeffrey Li, Vaishaal Shankar (lead), Mitchell Wortsman

#### Evaluation tokens.

Achal Dave, Samir Yitzhak Gadre, Reinhard Heckel, Vaishaal Shankar (lead), Rulin Shao

#### Loss/perplexity evaluation.

Samir Yitzhak Gadre

#### Downstream evaluation.

Vaishaal Shankar

#### Project-specific planning, infrastructure, plots, and analysis.

Samir Yitzhak Gadre

#### OpenLM [ 39 ] open-source infrastructure.

Achal Dave (core contributor),
Alex Fang,
Samir Yitzhak Gadre (core contributor),
Suchin Gururangan (core contributor),
Jenia Jitsev,
Sedrick Keh,
Jeffrey Li,
Jean Mercat,
Marianna Nezhurina,
Vaishaal Shankar (core contributor),
Georgios Smyrnis (core contributor),
Igor Vasiljevic,
Mitchell Wortsman (core contributor),
Rui Xin

#### Theory.

Yair Carmon (original idea that “parallel lines” should show up in scaling plots),
Samir Yitzhak Gadre (various derivations, empirical verification, related validation loss to average top-1 error as in Equation (5)),
Reinhard Heckel (derived a scaling form based on Chinchilla Approach 3 [], which appears in Equation (4)),
Niklas Muennighoff (derived a scaling form based on Chinchilla Approach 3, similar to Equation (4)),
Mitchell Wortsman (provided intuition about irreducible loss and why it is critical).

#### Writing.

Yair Carmon,
Achal Dave,
Reinhard Heckel,
Samir Yitzhak Gadre (lead),
Niklas Muennighoff,
Ludwig Schmidt

#### Compute.

Achal Dave, Jenia Jitsev, Thomas Kollar, Ludwig Schmidt, Vaishaal Shankar

#### Advising.

Yair Carmon (co-lead),
Achal Dave (co-lead),
Alexandros G. Dimakis,
Reinhard Heckel (co-lead),
Gabriel Ilharco,
Jenia Jitsev,
Pang Wei Koh,
Thomas Kollar,
Niklas Muennighoff (co-lead),
Ludwig Schmidt (co-lead),
Luca Soldaini,
Shuran Song

## Appendix B Scaling-law derivations

We first show that reparameterizing Equation (3) in terms of the compute $C$ and token multiplier $M$ for $\alpha=\beta$ yields Equation (4).
Combining $C=6ND$ and $M=D/N$ yields $N=\sqrt{C/(6M)}$ and $D=\sqrt{CM/6}$. Inserting these into Equation (3) yields,

$$ $\displaystyle L(C,M)$ $\displaystyle=E+A\left(\frac{C}{6M}\right)^{-\frac{\alpha}{2}}+B\left(\frac{CM }{6}\right)^{-\frac{\alpha}{2}},$ $\displaystyle=E+\left(A\left(\frac{1}{6}\right)^{-\frac{\alpha}{2}}M^{\frac{ \alpha}{2}}+B\left(\frac{1}{6}\right)^{-\frac{\alpha}{2}}M^{-\frac{\alpha}{2}} \right)C^{-\frac{\alpha}{2}}.$ $$

This is equal to Equation (4), making the substitutions $\eta=\alpha/2$, $a=A(1/6)^{-\eta}$, $b=B(1/6)^{-\eta}$, as noted in the main body.

#### Relation to compute-optimal training.

Recall that we made the assumption $\alpha=\beta$, which implies equal scaling of parameters and tokens to realize compute-optimal models.
While this assumption is empirically justified [], even if $\alpha\neq\beta$, we get a parameterization that implies the power law exponent in Equation (4) remains constant with over-training, while the power law scalar changes.

To find a compute-optimal training setting, Hoffmann et al. [] propose to minimize the right-hand side of Equation (3) subject to the compute constraint $C=6ND$. This yields,
$N^{\ast}=\gamma^{\frac{1}{\alpha+\beta}}(C/6)^{\frac{\beta}{\alpha+\beta}}$ and
$D^{\ast}=\gamma^{-\frac{1}{\alpha+\beta}}(C/6)^{\frac{\alpha}{\alpha+\beta}}$, where $\gamma=\frac{\alpha A}{\beta B}$, for notational convenience.
The associated risk is,

$$ $\displaystyle L(N^{\ast},D^{\ast})=E+\left(A\gamma^{\frac{-\alpha}{\beta+ \alpha}}+B\gamma^{\frac{\beta}{\beta+\alpha}}\right)\left(\frac{C}{6}\right)^{ -\frac{\alpha\beta}{\alpha+\beta}}.$ $$

We now deviate from compute-optimal training by modifying the model size and tokens by multiplication with a constant $\sqrt{m}$, according to

$$ $\displaystyle N_{m}=\frac{1}{\sqrt{m}}N^{\ast},\quad D_{m}=\sqrt{m}D^{\ast}.$ (7) $$

This modification keeps the compute constant (i.e., $6N_{m}D_{m}=6N^{\ast}D^{\ast}$). The risk, then, becomes

$$ $\displaystyle L(f_{N_{m},D_{m}})=E+\left(m^{\frac{\alpha}{2}}A\gamma^{\frac{- \alpha}{\beta+\alpha}}+m^{-\frac{\beta}{2}}B\gamma^{\frac{\beta}{\beta+\alpha} }\right)C^{-\frac{\alpha\beta}{\alpha+\beta}}.$ (8) $$

We again expect the same power law exponent and changing power law scalar.
Note that $m$ in Equation (8) is similar to $M$ in Equation (4). Specifically, $m$ is a multiple of the Chinchilla-optimal token multiplier $M^{\ast}=D^{\ast}/N^{\ast}$, which is no longer fixed as a compute budget changes for $\alpha\neq\beta$.

## Appendix C Additional training details

#### Architecture.

As stated in the main paper, we train transformers [], based on auto-regressive, decoder-only, pre-normalization architectures like GPT-2 [] and LLaMA [].
We adopt OpenLM [] for modeling, which utilizes PyTorch [, ], xformers [], triton [], FlashAttention [], FSDP [], and bfloat16 automatic mixed precision.
Like LLaMA, we omit bias terms, but replace RMSNorm [] with LayerNorm [], which has readily available fused implementations.
Following Wortsman et al. [], we apply qk-LayerNorm [], which adds robustness to otherwise poor hyperparameter choices (e.g., learning rate).
We use SwiGLU [] activations and depth-scaled initialization [].
We use a sequence length of 2,048, rotary positional embeddings [], and the GPT-NeoX-20B tokenizer [], which yields a vocabulary size of 50k.
We do not use weight tying [, ].
We sample without replacement during training and employ sequence packing without attention masking.
We separate documents in our training corpora with end-of-text tokens.

#### Objectives and optimization.

We train with a standard causal language modeling objective (i.e., next token prediction) with an additive z-loss [] (coefficient 1$e$-4), which mitigates output logit norm growth [] instabilities.
We use the AdamW optimizer [] (PyTorch defaults except beta2 = 0.95), with independent weight decay [] (coefficient 1$e$-4).
For the learning rate schedule, we use linear warmup and cosine decay.
We cool down to a low learning rate (3$e$-5).

## Appendix D Additional grid search details

#### Final model configurations.

We present our final hyperparameters in Table 3.

**Table 3: Main models and hyperparameters used in our investigation. Models have number of parameters $N$, with number of layers $n_{\text{layers}}$, number of attention heads $n_{\text{heads}}$, model width $d_{\text{model}}$, and width per attention head $d_{\text{head}}$. Batch sizes are global and in units of sequences. Each sequence has 2,048 tokens. A100 GPU hours are at $M=20$, which are near compute-optimal runs. For the 1.4B scale, a batch size of 256 performs slightly better than 512.**
| $N$ | $n_{\text{layers}}$ | $n_{\text{heads}}$ | $d_{\text{model}}$ | $d_{\text{head}}$ | Warmup | Learning rate | Batch size | $M=20$ A100 hours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.011B | 8 | 4 | 96 | 24 | 100 | 3$e$-3 | 64 | 0.3 |
| 0.079B | 8 | 4 | 512 | 128 | 400 | 3$e$-3 | 512 | 5 |
| 0.154B | 24 | 8 | 576 | 72 | 400 | 3$e$-3 | 512 | 12 |
| 0.411B | 24 | 8 | 1,024 | 128 | 2,000 | 3$e$-3 | 512 | 75 |
| 1.4B | 24 | 16 | 2,048 | 128 | 5,000 | 3$e$-3 | 256 | 690 |
| 6.9B | 32 | 32 | 4,096 | 128 | 5,000 | 3$e$-4 | 2,048 | 17,000 |

#### Grid search configuration selection.

Recall in Section 3.3, we run a grid search over many configurations.
We present the architectures we sweep over in Table 4.

**Table 4: Topologies for our grid searches. We consider 130 architectures for our grid search. After sweeping over batch size and warmup, we get a total of 435 configurations. For a complete list of hyperparameter configurations, please see: [https://github.com/mlfoundations/scaling](https://github.com/mlfoundations/scaling)**
| $n_{layers}$ | $n_{heads}$ | $d_{model}$ | Number of |
| --- | --- | --- | --- |
|  |  |  | parameters [B] |
| 4 | 4 | 96 | 0.010 |
| 4 | 12 | 96 | 0.010 |
| 12 | 12 | 96 | 0.011 |
| 12 | 4 | 96 | 0.011 |
| 8 | 4 | 96 | 0.011 |
| 16 | 4 | 96 | 0.011 |
| 16 | 12 | 96 | 0.011 |
| 8 | 12 | 96 | 0.011 |
| 24 | 4 | 96 | 0.012 |
| 24 | 12 | 96 | 0.012 |
| 4 | 4 | 192 | 0.021 |
| 4 | 8 | 192 | 0.021 |
| 4 | 12 | 192 | 0.021 |
| 8 | 8 | 192 | 0.023 |
| 8 | 4 | 192 | 0.023 |
| 8 | 12 | 192 | 0.023 |
| 12 | 4 | 192 | 0.025 |
| 12 | 8 | 192 | 0.025 |
| 12 | 12 | 192 | 0.025 |
| 16 | 4 | 192 | 0.026 |
| 16 | 8 | 192 | 0.026 |
| 16 | 12 | 192 | 0.026 |
| 24 | 8 | 192 | 0.030 |
| 24 | 4 | 192 | 0.030 |
| 24 | 12 | 192 | 0.030 |
| 4 | 12 | 288 | 0.033 |
| 4 | 4 | 288 | 0.033 |
| 8 | 12 | 288 | 0.037 |
| 8 | 4 | 288 | 0.037 |
| 4 | 4 | 320 | 0.038 |
| 4 | 8 | 320 | 0.038 |
| 12 | 12 | 288 | 0.041 |
| 12 | 4 | 288 | 0.041 |
| 8 | 8 | 320 | 0.043 |
| 8 | 4 | 320 | 0.043 |
| 16 | 4 | 288 | 0.045 |
| 16 | 12 | 288 | 0.045 |
| 12 | 4 | 320 | 0.049 |
| 12 | 8 | 320 | 0.049 |
| 24 | 4 | 288 | 0.053 |
| 24 | 12 | 288 | 0.053 |
| 16 | 8 | 320 | 0.055 |
| 16 | 4 | 320 | 0.055 |
| 4 | 12 | 488 | 0.062 |
| 4 | 4 | 512 | 0.065 |
| 4 | 16 | 512 | 0.065 |
| 4 | 8 | 512 | 0.065 |
| 24 | 8 | 320 | 0.066 |
| 24 | 4 | 320 | 0.066 |
| 4 | 4 | 576 | 0.074 |
| 4 | 8 | 576 | 0.074 |
| 4 | 12 | 576 | 0.074 |
| 8 | 12 | 488 | 0.075 |
| 8 | 4 | 512 | 0.079 |
| 8 | 8 | 512 | 0.079 |
| 8 | 16 | 512 | 0.079 |
| 4 | 4 | 640 | 0.085 |
| 4 | 16 | 640 | 0.085 |
| 4 | 8 | 640 | 0.085 |
| 12 | 12 | 488 | 0.087 |
| 8 | 4 | 576 | 0.090 |
| 8 | 12 | 576 | 0.090 |
| 8 | 8 | 576 | 0.090 |
| 12 | 16 | 512 | 0.093 |
| 12 | 8 | 512 | 0.093 |

## Appendix E Evaluation dataset details

All 46 downstream evaluations are based on MosaicML’s LLM-foundry evaluation suite [].
We specifically consider the datasets given in Table 5.

**Table 5: 46 downstream tasks. All downstream tasks considered in this work, evaluated via LLM-foundry []. For more information on each dataset and specifics about the LLM-foundry category and evaluation type, please see: [https://www.mosaicml.com/llm-evaluation](https://www.mosaicml.com/llm-evaluation).**
| Downstream task | LLM-foundry category | Evaluation type | Shots | Samples | Baseline |
| --- | --- | --- | --- | --- | --- |
| AGIEval LSAT AR [, , ] | symbolic problem solving | multiple choice | 3 | 230 | 0.25 |
| AGIEval LSAT LR [, , ] | reading comprehension | multiple choice | 3 | 510 | 0.25 |
| AGIEval LSAT RC [, , ] | reading comprehension | multiple choice | 3 | 268 | 0.25 |
| AGIEval SAT English [] | reading comprehension | multiple choice | 3 | 206 | 0.25 |
| ARC-Challenge [] | world knowledge | multiple choice | 10 | 2376 | 0.25 |
| ARC-Easy [] | world knowledge | multiple choice | 10 | 2376 | 0.25 |
| BBQ [] | safety | multiple choice | 3 | 58492 | 0.50 |
| BIG-bench: CS algorithms [] | symbolic problem solving | language modeling | 10 | 1320 | 0.00 |
| BIG-bench: Conceptual combinations [] | language understanding | multiple choice | 10 | 103 | 0.25 |
| BIG-bench: Conlang translation [] | language understanding | language modeling | 0 | 164 | 0.00 |
| BIG-bench: Dyck languages [] | symbolic problem solving | language modeling | 10 | 1000 | 0.00 |
| BIG-bench: Elementary math QA [] | symbolic problem solving | multiple choice | 10 | 38160 | 0.25 |
| BIG-bench: Language identification [] | language understanding | multiple choice | 10 | 10000 | 0.25 |
| BIG-bench: Logical deduction [] | symbolic problem solving | multiple choice | 10 | 1500 | 0.25 |
| BIG-bench: Misconceptions [] | world knowledge | multiple choice | 10 | 219 | 0.50 |
| BIG-bench: Novel Concepts [] | commonsense reasoning | multiple choice | 10 | 32 | 0.25 |
| BIG-bench: Operators [] | symbolic problem solving | language modeling | 10 | 210 | 0.00 |
| BIG-bench: QA WikiData [] | world knowledge | language modeling | 10 | 20321 | 0.00 |
| BIG-bench: Repeat copy logic [] | symbolic problem solving | language modeling | 10 | 32 | 0.00 |
| BIG-bench: Strange stories [] | commonsense reasoning | multiple choice | 10 | 174 | 0.50 |
| BIG-bench: Strategy QA [] | commonsense reasoning | multiple choice | 10 | 2289 | 0.50 |
| BIG-bench: Understanding fables [] | reading comprehension | multiple choice | 10 | 189 | 0.25 |
| BoolQ [] | reading comprehension | multiple choice | 10 | 3270 | 0.50 |
| COPA [] | commonsense reasoning | multiple choice | 0 | 100 | 0.50 |
| CoQA [] | reading comprehension | language modeling | 0 | 7983 | 0.00 |
| Commonsense QA [] | commonsense reasoning | multiple choice | 10 | 1221 | 0.25 |
| Enterprise PII classification [] | safety | multiple choice | 10 | 3395 | 0.50 |
| HellaSwag (10-shot) [] | language understanding | multiple choice | 10 | 10042 | 0.25 |
| HellaSwag (zero-shot) [] | language understanding | multiple choice | 0 | 10042 | 0.25 |
| Jeopardy [] | world knowledge | language modeling | 10 | 2117 | 0.00 |
| LAMBADA [] | language understanding | language modeling | 0 | 5153 | 0.00 |
| LogiQA [] | symbolic problem solving | multiple choice | 10 | 651 | 0.25 |
| MMLU (5-shot) [] | world knowledge | multiple choice | 5 | 14042 | 0.25 |
| MMLU (zero-shot) [] | world knowledge | multiple choice | 0 | 14042 | 0.25 |
| MathQA [] | symbolic problem solving | multiple choice | 10 | 2983 | 0.25 |
| OpenBook QA [] | commonsense reasoning | multiple choice | 0 | 500 | 0.25 |
| PIQA [] | commonsense reasoning | multiple choice | 10 | 1838 | 0.50 |
| PubMed QA Labeled [] | reading comprehension | language modeling | 10 | 1000 | 0.00 |
| SIQA [] | commonsense reasoning | multiple choice | 10 | 1954 | 0.50 |
| SQuAD [] | reading comprehension | language modeling | 10 | 10570 | 0.00 |
| Simple Arithmetic: NoSpaces [] | symbolic problem solving | language modeling | 10 | 1000 | 0.00 |
| Simple Arithmetic: WithSpaces [] | symbolic problem solving | language modeling | 10 | 1000 | 0.00 |
| WinoGender MC: Female [] | safety | multiple choice | 10 | 60 | 0.50 |
| WinoGender MC: Male [] | safety | multiple choice | 10 | 60 | 0.50 |
| WinoGrande [] | language understanding | schema | 0 | 1267 | 0.50 |
| WinoGrand [] | language understanding | schema | 0 | 273 | 0.50 |

Recall that we use a subset of 17 of these evaluations that give signal (are above random chance) for the compute range we consider.
See Appendix F, where we ablate over the 17 subset design choice by including more and less evaluations.

## Appendix F Additional results

#### Scaling law fits.

We present specific coefficients for our fits in Table 6.

**Table 6: Scaling law fit parameters. Here we present our scaling coefficients fit to Equations (4) and (5) using configurations from Table 1.**
| Training dataset | Fit for Equation (4): $L(C,M)=$ | Fit for Equation (5): $\textsf{Err}(L)=$ |
| --- | --- | --- |
|  | $E+(a\cdot M^{\eta}+b\cdot M^{-\eta})C^{\eta}$ | $\epsilon-k\cdot\exp{(-\gamma L)}$ |
| C4 [, ] | $1.51+\left(141\cdot M^{0.121}+190\cdot M^{-0.121}\right)C^{-0.121}$ | $0.850-2.08\cdot\exp{(-0.756\cdot L)}$ |
| RedPajama [] | $1.84+\left(212\cdot M^{0.136}+367\cdot M^{-0.136}\right)C^{-0.136}$ | $0.857-2.21\cdot\exp{(-0.715\cdot L)}$ |
| RefinedWeb [] | $1.73+\left(157\cdot M^{0.127}+246\cdot M^{-0.127}\right)C^{-0.127}$ | $0.865-2.21\cdot\exp{(-0.707\cdot L)}$ |

#### Small-scale experiments can predict model rank order.

We expect to be able to rank hypothetical models based on their predicted performance, which is useful when deciding what large-scale runs to train.
To verify, we rank 9 testbed models with $N\geq 1.4\text{B}$ by ground-truth top-1 error and by estimated top-1 error.
We find high rank correlation of 0.88 for the 17-task split.

#### Over-performing grid search models experience more optimization steps.

As mentioned in Section 3.3 and Figure 4, we notice that models between 0.011B to 0.079B (i.e., $5.2\times 10^{16}$ to $5.2\times 10^{17}$ FLOPs trained near compute-optimal) over-perform compared to the trend established by other models in our initial grid searches. This results in a bump in the scaling plot.
While we choose to exclude this range of models for our scaling study, we additionally investigate this phenomenon.
In Figure 6 we color grid search configurations by the number of optimization steps (i.e., number of tokens seen divided by batch size divided by sequence length).
We notice that models in the aforementioned range experience more optimization steps than their x-axis neighbors.
For context, Figure 1 *(left)* in Kaplan et al. [] also shows a bump; however, there the performance is worse than the general trend instead of better as in our work.
We leave understanding more fully the interactions between hyperparameters, scaling, and performance to future work.

Figure: Figure 6: Understanding over-performing models in our grid search. *(left)* Models trained with $5.2\times 10^{16}$ to $5.2\times 10^{17}$ FLOPs over-perform relative to their neighbors. In looking at the number of optimization steps, we notice that the over-performing models experience more optimization steps than their x-axis neighbors. We hypothesize that the number of optimization steps is important, especially for smaller models, when trying to find models that lie along a trend. *(right)* A view of the same phenomenon, specifically on the efficient frontier.
Refer to caption: x6.png

#### Scaling is largely predictable in-distribution (ID).

Prior work focuses on understanding scaling using ID loss, often using training loss directly [, ].
Hence, we also consider Paloma [] loss evaluation sets, which are designed to probe performance in specific domains.
We use Paloma’s C4 [, ], RedPajama [], and Falcon-RefinedWeb [] splits to probe for ID loss.
As seen in Figure 7, relative error is mostly low.
Relative error is largest for the $N=1.4\text{B},M=640$ RedPajama run at 15.4%.
Examining this case specifically, we find that the model performs better than the scaling law prediction.
We hypothesize that as a model sees more tokens there is an increased likelihood of near-duplicate sequences ID, resulting in performance that is better than predicted.

Figure: Figure 7: In-distribution (ID) settings. Boxes highlighted in yellow correspond to data points used to fit Equation (4). Relative error is generally low across interpolation and extrapolation regimes. Relative error is largest for the RedPajama $N=1.4\text{B},M=640$ prediction at 15.4%. In this case, we find that our scaling law predicts the model should perform worse than it does in practice.
Refer to caption: x7.png

#### Relative error is stable across many choices of downstream evaluation suites.

To understand how sensitive our investigation is to our choices of downstream evaluation sets, we consider several other options as seen in Figure 8.
We find that our prediction errors are fairly (i) low and (ii) consistent for many choices of downstream evaluation sets including the whole suite of 46 evaluations.

Figure: Figure 8: Downstream evaluation set ablation for 6.9B parameter, 138B token runs. Recall that we consider a 17 task evaluation suite created by including only test sets where any 0.154B model we trained (for any token multiplier and training dataset) gets $t=10$ percentage points above random chance. We evaluate over this subset to make sure we are measuring signal not noise. Here, we wish to understand how sensitive the relative prediction error is to our choice of $t$. *(left)* We see that relative prediction error is fairly low before a threshold of $t=35$ (less than $10\%$ relative error). When too many tasks are excluded (i.e., $t\geq 40$) relative error spikes. Averaging over all 46 datasets ($t=-5$ as some evals are worse than random chance) also makes for a predictable metric (less than $3\%$ relative error). *(right)* A parallel view, showing how many tasks are removed as $t$ increases. 40 out of the 46 tasks can be removed and relative error is still fairly stable.
Refer to caption: x8.png

**Table 7: Downstream relative prediction error at 6.9B, 138B tokens, with and without the 1.4B data point. Recall in Table 1, we introduce a $N=1.4$B, $M=20$ run to get better downstream error predictions. Here we compare, prediction errors with and without this model for fitting the scaling law. Note that without the model (i.e., rows with “w/o 1.4B”) average top-1 predictions, over the 17 tasks. are less accurate.**
| Scaling law fit | Train set | ARC-E | LAMBADA | OpenBook QA | HellaSwag | 17 eval |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | [] | [] | [] | [] |  |
| Table 1 | C4 [, ] | 28.96% | 15.01% | 16.80% | 79.58% | 0.14% |
| Table 1 w/o 1.4B | C4 [, ] | 0.92% | 2.04% | 96.16% | 61.79% | 0.42% |
| Table 1 | RedPajama [] | 5.21% | 14.39% | 8.44% | 25.73% | 0.05% |
| Table 1 w/o 1.4B | RedPajama [] | 8.13% | 11.07% | 7.56% | 30.98% | 10.64% |
| Table 1 | RefinedWeb [] | 26.06% | 16.55% | 1.92% | 81.96% | 2.94% |
| Table 1 w/o 1.4B | RefinedWeb [] | 15.39% | 6.26% | 6.79% | 6.52% | 15.79% |

#### Scaling can break down when under-training.

Figure: Figure 9: Scaling with small token multipliers. For smaller multipliers (e.g., $M=5$ in cyan), scaling does not follow the same trend as that of larger multipliers. Additionally, many token multipliers (e.g., $M\in\{10,20,40,80\}$) garner points close to the compute-optimal frontier.
Refer to caption: x9.png

We find that when a token multiple is too small (i.e., under-training regime), scaling appears unreliable.
In Figure 9 we see for $M=5$ the scaling trend is different.
We hypothesize that tuning hyperparameters (e.g., warmup, batch size) directly for smaller multipliers may help mitigate the breakdown in predictability.

#### Scaling can be unpredictable out-of-distribution (OOD).

Our main result shows reliable C4 eval loss predictions with models trained on RedPajama, which is an OOD evaluation setting.
However, both C4 and RedPajama both contain tokens sourced from CommonCrawl.

To further probe OOD performance, we measure the relative error of scaling laws fit to models trained on C4 and evaluated on Paloma’s 100 programming languages [], Paloma’s Penn Tree Bank (PTB) split [], and a German version of C4 [].
Recall that the C4 training set we use has been filtered for English text.
Hence we expect (i) the proportion of code is minimal, (ii) the “<unk>” substrings in PTB raw text do not appear frequently, and (iii) German is not prevalent.
We notice that extrapolation relative error tends to be high for large $M,N$ on programming languages and PTB (Figure 10 *(left, center)*).
In contrast, for German C4, relative error is still low across the extrapolation range, with a maximum relative error of 7.6% at the $N=$1.4B, $M=80$ scale (Figure 10 *(right)*).
We hypothesize that further modifications to scaling laws are necessary to predict when scaling should be reliable as a function of the training and evaluation distributions.

Figure: Figure 10: Out-of-distribution (OOD) settings. Boxes highlighted in yellow correspond to data points used to fit Equation (4). Recall that the C4 training set is English-filtered. Relative error can spike, suggesting unreliable scaling, for *(left)* programming languages and *(center)* Penn Tree Bank, which contains many frequently occurring, uncommon substrings. However, scaling is relatively reliable when evaluating on *(right)* German. These results motivate future studies of OOD conditions that affect scaling in the over-trained regime.
Refer to caption: x10.png

#### Small-scale experiments can predict average downstream top-1 error.

Figure: Figure 11: Relative error on average top-1 predictions (46 task split). Boxes highlighted in yellow correspond to data points used to fit Equation (5). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable.
Refer to caption: x11.png

To verify that chaining Equations (4) and (5) is effective in practice, we collect C4 eval loss and downstream error pairs for the configurations in Table 1.
In Figure 11, we look at relative error for our scaling predictions in the context of Average top-1 error over 46 evals and in Figure 12 over the high-signal 17 eval subset.
We again notice reliable scaling in interpolation and extrapolation regimes, suggesting the validity of our procedure to predict downstream average top-1 error.

Figure: Figure 12: Relative error on average top-1 predictions (17 task split). Boxes highlighted in yellow correspond to data points used to fit Equation (5). Using our fits, we accurately predict downstream average top-1 error across interpolation and extrapolation regimes. This result supports that (i) chaining a scaling law and our proposed exponential decay function is a valid procedure and (ii) average top-1 error can be highly predictable.
Refer to caption: x12.png

#### Loss evaluation ablations for downstream trends.

Figure 13 presents the correlation between downstream error and loss evaluated on different validation sets (C4, RedPajama, and RefinedWeb). Regardless of the validation set (x-axis), models follow the exponential decay relationship given in Equation (5), suggesting the choice of validation loss is not critical for the appearance of this phenomenon.

Figure: Figure 13: Correlation between average top-1 error and evaluation loss. We observe that regardless of evaluation loss distribution (x-axis), models tend to follow Equation (5). This suggests that there can be several reasonable choices for the validation loss distribution. Additionally, ID models trained on C4 and evaluated on a C4 validation set, perform best in terms of loss, but these gains don’t necessarily translate to lower error downstream (e.g., *(left column)*). This suggests *the need to fit Equation (5) per dataset* and also suggests comparing models trained on different data distributions with a single loss evaluation can be misleading.
Refer to caption: x13.png

#### Investing more compute in a scaling law makes it more predictive.

Figure: Figure 14: Trade-offs between scaling law for loss fitting considerations and reliability. Each red circle represents a scaling law fit to Equation (4) with as many as 29 models trained on RedPajama. Specifically, a grid formed by $N\in\{0.011\text{B},0.079\text{B},0.154\text{B},0.411\text{B}\},M\in\{5,10,20, 40,80,160,320\}$ gives 28 models and a $N=1.4B,M=20$ run gives the last model. We sort models by training FLOPs in increasing order and sample models uniformly from index windows $[1,2,...,n]$ for $n\in[5,6,..,29]$ to fit Equation (4). The blue star represents the default configuration presented in Table 1. The prediction target is a $N=1.4B,M=640\text{ }(D=900\text{B})$ model. As the amount of compute *(left)* and the number of points *(right)* used to fit the scaling law increases, relative error trends downwards. Our default configuration keeps compute and number of points low, while still providing low prediction error compared to the trend.
Refer to caption: x15.png

Figure: Figure 15: Compute vs. relative error for the 1.4B, 900B token RedPajama run. *(left)* The compute necessary to accurately predict loss is less than that needed to accurately predict *(right)* average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 16.
Refer to caption: x16.png

Figure: Figure 16: Compute vs. relative error for the 6.9B, 138B token RedPajama run. *(left)* The compute necessary to accurately predict loss is less than that needed to accurately predict *(right)* average downstream error. This claim is supported by the fact that the slope of the trend for loss is steeper than for top-1 error. These findings corroborate Figure 15.
Refer to caption: x17.png

Thus far we have looked at standard configurations from Table 1 to construct our scaling laws, mainly to demonstrate extrapolation to larger $N,M$.
However, for practitioners, the main constraint is often training compute.
Hence, we wish to understand the trade-offs between the amount of compute invested in creating a scaling law and the relative error of the resulting law in the over-trained regime.
In Figure 14 *(left)*, we see that as one increases the amount of compute, it is possible to get better fits with lower relative error.
In Figure 14 *(right)*, we see a similar trend as one increases the number of data points used to fit a scaling law.
Blue stars indicate the configurations from Table 1, which provide accurate predictions relative to the general trends—hinting at their usefulness for our investigation.
In Figures 15 and 16 we repeat the compute analysis comparing trade-offs for loss prediction and error prediction for our RedPajama 1.4B parameter, 900B token and 6.9B parameter, 138B token runs respectively.
We find that less compute is generally necessary to construct a loss scaling law that achieves the same relative error as that of an error prediction scaling law.

Figure: Figure 17: Scaling exponent vs. token multiplier. In Figure 2, we notice roughly parallel lines (i.e., roughly constant scaling exponent $\eta$) in the $\log$-$\log$ plot of loss vs. compute, even as the token multiplier $M$ changes. Here we plot $\eta$ vs. $M$ directly, where the shaded region gives a 95% bootstrap confidence interval for the trend. This view supports that $\eta$ is relatively constant.
Refer to caption: x18.png

Figure: Figure 18: Downstream top-1 error vs. C4 eval loss for each of the 46 downstream evals. Here we plot models from our testbed for each scatter plot. We see that some individual evaluations, like ARC-Easy, follow exponential decay. Others, like BIG-bench: CS algorithms, show step function behavior. Still others, like MathQA, hover around random chance.
Refer to caption: x19.png

#### On compute-optimal token multipliers.

We consider 20 tokens per parameter as close to compute-optimal for our experiments.
Here we investigate, using different approaches, what the compute-optimal token multipliers are for each dataset—assuming one should scale number of parameter and training tokens equally as Hoffmann et al. [] suggest.

Turning to Figure 9, we notice that there are many multipliers, between 10 and 80 that yield models close to the frontier.
Hence, empirically, it appears choices within this range should be suitable for the optimal token multiplier.

We can also compute an optimal token multiplier using the coefficients in Table 6.
Based on Hoffmann et al. []’s Equation (4) and the assumption that $\alpha=\beta$, we write,

$$ $\displaystyle N^{*}(C)=G\left(\frac{C}{6}\right)^{\frac{1}{2}},D^{*}(C)=G^{-1} \left(\frac{C}{6}\right)^{\frac{1}{2}},G=\left(\frac{a}{b}\right)^{\frac{1}{4 \eta}}.$ (9) $$

To compute $M^{*}=D^{*}/N^{*}$, we then have,

$$ $\displaystyle M^{*}=\left(\frac{b}{a}\right)^{\frac{1}{2\eta}}.$ (10) $$

Using the values from Table 6 and Equation (10), we find $M^{*}_{\text{C4}}=3.36$, $M^{*}_{\text{RedPajama}}=7.42$, $M^{*}_{\text{RefinedWeb}}=5.85$, where the subscript gives the dataset name.
These values conflict with the observation in Figure 9, which suggests $M=5$ is already too small to give points on the Pareto frontier.
We hypothesize this mismatch arises because we fit our scaling laws using models with $M\geq 20$.
Additionally, we hyperparamter-tune at $M=20$.
As previously discussed, it is likely possible to find better hyperparameter configurations at $M=5$ with further hyperparameter tuning at this token multiplier.

## Appendix G Additional related work

#### Language modeling.

Language models can be grouped into encoder-only [, , , , ], encoder-decoder [, ], and decoder-only architectures [, , , , , , , , , , , , , , , , ].
Most current implementations are based on the transformer []. However, there has been a recent resurgence in scaling language models based on non-transformer architectures [, , , ].
Further, there has been substantial work on adapting pre-trained language models to better follow instructions [, , , , , , , , , , ].
However, following prior work [, ] and given their overall prevalence, we limit ourselves to GPT-style, decoder-only transformers that have solely been pre-trained.

#### Scaling laws.

Kaplan et al. [] investigate scaling trends in GPT language models.
Bahri et al. [] investigate different scaling regimes theoretically, and Sharma & Kaplan [] relate scaling coefficients to data manifold dimensions.
Tay et al. [, ] elucidate the connection between model architecture and scaling trends, while Hernandez et al. [], Tay et al. [] develop scaling laws for transfer learning.
Ivgi et al. [] also consider transfer learning scaling laws and highlight the importance of hyperparameter selection in the low-compute regime.
Ghorbani et al. [], Gordon et al. [], Bansal et al. [] develop scaling laws for neural machine translation.
Caballero et al. [] propose a scaling law functional form, which they demonstrate is predictive in several domains.

#### Scaling beyond language modeling.

There is a large body of work on scaling neural networks beyond language modeling, for example in computer vision [, , , , ], multimodal learning [, , ], and image reconstruction [].

#### Over-training in existing models.

**Table 8: Token multipliers of existing models. In our work, we run experiments with token multipliers between 5 and 640 for {GPT-2 [], LLaMA []}-style decoder-only architectures.**
| Model family | Parameters $N$ | Training tokens $D$ | Token multiplier $M$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| T5 [] |  | 11B |  |  | 34B |  |  | 3.1 |
| GPT-3 [] |  | 175B |  |  | 300B |  |  | 1.7 |
| Gopher [] |  | 280B |  |  | 300B |  |  | 1.1 |
| Chinchilla [] |  | 70B |  |  | 1.4T |  |  | 20.0 |
| LLaMA [] |  | 7B |  |  | 1T |  |  | 140.0 |
| LLaMA [] |  | 70B |  |  | 1.4T |  |  | 20.0 |
| LLaMA-2 [] |  | 7B |  |  | 2T |  |  | 290.0 |
| LLaMA-2 [] |  | 70B |  |  | 2T |  |  | 30.0 |
| XGen [] |  | 7B |  |  | 1.5T |  |  | 210.0 |
| MPT [] |  | 7B |  |  | 1T |  |  | 140.0 |

To contextualize the extent to which we over-train, we provide token multipliers for popular models in Table 8.

## Appendix H Broader impact

Language models have known risks in terms of harmful language, toxicity, and human automation—to name a few [, ].
We include the following for our public release “WARNING: These are base models and not aligned with post-training. They are provided as is and intended as research artifacts only.”
However, even as research artifacts, we recognize that models can still be misused by malicious actors or can be harmful to benevolent actors.
When deciding to release our models and experiments, we considered (i) the benefit to the scientific community and (ii) the benchmark performance relative to other models that have already been released.
For (i) we feel that our testbed is of use to others in the community who want to do scaling research, but do not necessarily have the means to train these model artifacts themselves. Hence, we predict (and hope) releasing all models and experiments will be helpful to others wanting to participate in scaling research.
For (ii), we note that there are publicly available models [, , ], which outperform models from our testbed and that are more likely to be widely adopted.
Finally, we recognize that advancing scaling science also has potential for harm. Specifically, while we are concerned with loss and downstream task performance for popular evaluation settings, it is possible that nefarious actors may use scaling laws to help design more harmful models.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: - Line 69-70: "M = N/D" seems to be a typo. It should be "M = D/N."
- Is Figure 1 a log-log plot?
soundness: 4
strengths: - The motivation is clear and the investigation is significant. Exploring scaling laws that are closer to real-world application scenarios provides strong guidance for the experimental setup of pre-training large language models.
- The experimental logic is very clear. The authors first selected certain experimental settings to fit and obtain hyperparameters for the specific scaling law. Then, they validated its accuracy on over-training and downstream performance across additional experimental settings, which is a reasonable approach.
summary: This paper highlights two issues overlooked in current discussions of scaling laws for large language models: (1) To reduce inference costs, smaller models are often trained on more data, and (2) scaling law research commonly uses perplexity to measure model performance, whereas real-world evaluations focus on task completion performance, such as accuracy. In response, this paper investigates and confirms scaling laws in large language models related to over-training and performance on downstream tasks.
weaknesses: - Some experiments are missing. The authors’ initial motivation for studying the scaling law in over-training was to reduce inference costs, but this raises a question: training smaller models on more data achieves performance comparable to larger models under the same compute budget. However, this issue is not well explained—are there relevant experiments or clear prior studies addressing this?
- The experimental validation of the scaling law for downstream performance is not rigorous enough. As shown in Table 2, there is a significant gap between the predicted and actual values of downstream performance based on the scaling law for individual datasets, while the average gap across 17 tasks is relatively smaller. Does this indicate that predictions from the downstream performance scaling law carry substantial risk? More specifically, we do not know for which specific number or types of tasks this scaling law provides accurate predictions.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * Lines 142-144: can you elaborate on this sentence: "While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus incur less inference cost." and why it is "common practice to over-train smaller models" (line 44)?
* The exponential relationship between loss and error is motivated in Figure 3, but have you evaluated larger sets of tasks to validate the relationship?
* Does the paper assume that over-training effects remain uniform across model sizes and datasets?
* Could the authors elaborate on the characteristics of tasks where downstream predictions were less accurate? Are there specific aspects that tend to yield higher prediction errors? 
* As the paper mentions exploring out-of-distribution scaling, can you provide findings on how well the scaling laws hold when tested on domains or tasks different from the training dataset?
soundness: 3
strengths: * The problem of scaling laws under over-training is practical and interesting, addressing gaps in existing literature around compute-optimal training, allowing for the efficient resource utilization in large-scale language modeling.
* The proposed laws appear logical
* The evaluation appears comprehensive, with a diverse and appropriate testbed to test the hypotheses around scaling laws under over-training. As a result, they achieved reasonably good performance: average top-1 error on 17 tasks within a 0.05% relative error and 0.7% relative error on validation loss.
* The paper is clearly presented, with detailed steps for law derivation,  model configuration, and scaling law fitting.
summary: The paper investigates scaling laws for language models  (LM) in the over-trained regime, where models are trained with greater token-to-parameter ratios to lower inference costs.
They furthermore derive power-law scaling laws for downstream task performance (error rates) from the conventional laws using LM loss.
Using a testbed of 104 models (0.011B to 6.9B parameters) with varying token counts across three distinct datasets, they fit power-law scaling laws to predict validation loss and error rates, validating the consistency and reliability of the proposed laws.
The paper is well-motivated and addresses an important problem, with technically significant and practically valuable findings.
weaknesses: - Performance of downstream task: Although average performance aligns with the laws, variability in individual predictions and occasional high errors may suggest that the scaling laws may not universally hold without further model or data refinements. Furthermore, the laws in equation 4 assume $\alpha =\beta$ which can limit the applicability.
- The paper does not address how the scaling laws would hold under different architectural configurations or training optimizations, which are relevant to future model development. 
- The paper explored limitations such as sensitivity in out-of-distribution settings, posing challenges for future refinement of scaling laws in varied training regimes and tasks. 
- The claim of "predictable downstream performance" seems somewhat misleading, as the performance for specific tasks can vary widely. A more precise phrasing could emphasize “predictable average performance” to better align with the actual findings.
- Presentation:
1. In Figure 1, the paper defines M = N/D (line 70) while D=MN (Fig 1 caption) and line 127
2. Figures need better visualization and explanation for understanding. For example, in Figure 1, colors for M=320 and M=640 are very similar.
3. Typos: Line 142: than in

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - How accurate would this method be for predicting the performance of much larger LLMs, such as 30B, 70B, 400B? Are there any special considerations in order to ensure the law stays accurate?
- Would this law also work on "efficient" LM architectures such as mixture-of-experts, approximate attention and attention-free models?
- How might overtraining affect post-training, fine-tuning or inference-time compute scaling methods? Would their benefits scale proportionally the same to the losses and evals?
soundness: 3
strengths: - Extensive experiments across many training datasets (C4, RedPajama and RefinedWeb) and model sizes.
- Comprehensive evaluations using diverse validation datasets.
summary: This paper explores the implications of over-training in LLMs, and tries to fit a scaling law for extrapolating beyond the compute-optimal (Chinchilla optimal) regime. The proposed scaling law is able to predict the final validation loss of a model using 300x less compute than what would have been needed to fully train the model.
weaknesses: - As the field of LLMs is currently agressively expanding into multimodality (images, audio, video and data), the proposed method might soon not hold up anymore. There is no mention of early-fusion multimodal models or token-free language modeling.
- Lack of consideration for post-training, fine-tuning and inference-time compute scaling methods, which are now integral to all production-ready models.

---

confidence: 5
contribution: 4
initial_rating: 5
presentation: 4
questions: There is only one thing that WILL NOT affect my overall rating towards this article. 

1. See Weakness [1], it is better to discuss the differences between the proposed over-train scaling law and [1]. 

I am willing to defend my overall rating in the rebuttal period.

[1] Temporal Scaling Law for Large Language Models
soundness: 4
strengths: 1. Originality: This work firstly investigates and tracks next-token prediction model performance under the over-training regime and on downstream benchmark evaluation tasks.
2. Strong experiment results: This work builds a test-bed with more than 100 models with different sizes and training tokens. Through predicting the scaling law, this work successfully predicts the performance of larger models trained with more tokens.
3. Theoretical relationship with prior work: This work provides a stimulating analysis (Section 2) between scaling laws under the over-training regime and existing Chinchilla/Kaplan scaling laws, providing strong theoretical evidence supporting the derived over-training scaling law.
4. Writing: This paper is well-written with a clear, logical presentation of motivation, derivation process, experiment result analysis, and conclusion.
summary: This article investigates the behaviors of scaling laws in the regime of over-training and on downstream benchmarks. Through derivation and initial observations, the article provide empirical evidence that over-training largely follows a similar, tractable, and constant-slope scaling law. Through further investigations and experiments, the article derives the parameters in the scaling law for over-training. With that conclusions, the article is then able to derive a scaling law relating model size, training tokens, and average downstream task performance, for the first time proving that average performance across a subset of downstream tasks are tractable.
weaknesses: 1. This work suggests that "there remain gaps between current scaling studies and how language models are ultimately trained and evaluated" (Line 13-14), suggesting that over-trained regime is under-investigated. There is an interesting work [1] that investigates the pre-training process of language models. In [1], the authors predict the training outcomes using the gathered data from an initial training period, and they successfully predicts the training outcomes of relatively small models (<0.1B) trained on large number of tokens (400B tokens). In that case, $M>4000$ and falls into the over-training category. It would be interesting to investigate the difference between the proposed over-train scaling law and [1].

[1] Temporal Scaling Law for Large Language Models

Meta-review: This paper fills a gap in the current scaling laws literature related to overtraining of smaller models. It measures downstream task performance, rather than simply aggregated perplexity.

Pros: Comprehensive experiments on a topic of practical interest to the machine learning community.

Cons: The individual task performance seems to be much less predictable than aggregated performance, and might not be described by scaling laws. In this sense, the paper appears to be somewhat overclaiming by claiming scaling is reliable. 

There are also some strong assumptions involved in theory, specifically that some coefficients match.

Decision: Accept (Poster)