Language models scale reliably with over-training and on downstream tasks
Authors: Samir Yitzhak Gadre
Georgios Smyrnis
Vaishaal Shankar
Suchin Gururangan
Mitchell Wortsman
Rulin Shao
Jean Mercat
Alex Fang
Jeffrey Li
Sedrick Keh
Rui Xin
Marianna Nezhurina
Igor Vasiljevic
Luca Soldaini
Jenia Jitsev
Alex Dimakis
Gabriel Ilharco
Pang Wei Koh
Shuran Song
Thomas Kollar
Yair Carmon
Achal Dave
Reinhard Heckel
Niklas Muennighoff
Ludwig Schmidt
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
scaling laws
over-training
task prediction
Primary area: foundation or frontier models, including LLMs
TL;DR: Fit our scaling laws to small-scale runs trained near compute-optimal, predict the downstream error (average top-1) of large over-trained runs.

Abstract: Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32$\times$ over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)––each from experiments that take 300$\times$ less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20$\times$ less compute.

Published as a conference paper at ICLR 2025
LANGUAGE MODELS SCALE RELIABLY WITH OVER- TRAINING AND ON DOWNSTREAM TASKS
Samir Yitzhak Gadre' .2, Georgios Smyrnis", Vaishaal Shankar', Suchin Gururangan' Mitchell Wortsman', Rulin Shao Jean Mercat Alex Fang", Jeffrey Li,, Sedrick Keh², Rui Xin', Maarianna Nezhurina 6.7, Igor Vasiljevie", Jenia Jitsev Luca Soldaini", Alexandros G. Dimakis Gabriel Ilhareo Pang Wei Koh Shuran Song" Thomas Kollar² Achal D Reinhard Heckel 13- Niklas Muennighoff Ludwwig Schmidt
Yair Carmon 2-
ABSTRACT
Scaling laws are useful guides for derisking expensive training runs, they predict performance of large models using cheaper, scall-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, 104 models with 0.01 to 6.9B parameters trained with various numbers of tokens on three data distributions. First, fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables
create a testbed
predict the validation loss of 1.4B parameter, 900B token (i.e., 32x over-trained) and a 6.9B parameter, 138B token run (i.e., a compute- optimal run)-each from experiments that take 300x less compute. Second, we relate the perplexity of language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20x less compute. To facilitate further research on reliable scaling, we provide all results of our experiments. Our experiments are available at https: //github. com/ m1 foundat ions/ scaling.
INTRODUCTION
Training large language models is expensive. Furthermore, training high-quality models requires a complex recipe of algorithmic techniques and training data. To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger model and data regimes via scaling laws. With reliable extrapolation, is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run. Indeed, this workflow has become commonplace for training state-of-the-art language models like Chinchilla 70B (Hoffmann et al., 2022), PaLM 540B (Chowdhery et al., 2022), GPT-4 (OpenAI, 2023), and many others.
Despite their importance for model development, published scaling laws differ from the goals of training oute-of-theeart models in important ways. For instance, scaling studies usually focus on the compute-optimal training regime ("Chinchilla optimality" (Hoffmann et al., 2022)), where model and dataset size are set to yield minimum loss for given compute budget. However, this setting ignores inference costs. As larger models are more expensive at inference, it is now common practice to over-train smaller models (Touvron al., 2023a). Another potential mismatch is that most scaling laws quantify model performance by perplexity in next-token prediction instead of accuracy on
Equal advising, ordered alphabetically. 'Columbia University. Toyota Research Institute, ²U Austin, 4 Apple, University of Washington, "Juelich Supercomputing Center, Research Center Juelich. 'LAION. Allen Institute for AI, 9UC Berkeley, 1"Bespoke Labs, 1taanford University, 12Tel Aviv University. 13 Munich, "Contextual AT

---

Published as a conference paper at ICLR 2025
0.8
0.68
0.64
Average top- emer: 17-tekk spiit
0.7
Raducib oss. Ca enl
0.60
2-
0.56
0.52
10ah
0.50
0.48
0.0.1118
Prediction Interpolation
0.098
0.46
N 1548
Extragolation
0.44
N- 6.98
M- 640
2.6
2.4
10:1
10
10th
10th
10t7
10th
10
5.5
5.0
4.5
4.0
Compute
[FLOPs]
Figure I: Reliable scaling with over-training and on downstream error prediction. (left) We fit a scaling law for model validation loss, parameterized by (i) token multiplier M D/N, which is the ratio of training tokens D to parameters N and (ii) the compute C' in FLOPs used to train model, approximated by to extrapolate, in both N and M, the validation performance of models requiring more than x00x training compute used to construct the scaling law. (right) We also average downstream top-1 error function of validation loss. We find that fitting scaling laws for downstream error benefits from using more expensive models when compared to fitting for loss prediction. We predict the average over 17 downstream tasks for models trained with over x0x the compute. For this figure, we train all models on RedPajama (Together Computer, 2023).
6.N D. Larger values
specify more over-training. We are able
scaling law to predict
widely used benchmark datasets. However, practitioners usually turn to benchmark performance, not loss, to compare models.
In this paper, we conduct an extensive set of experiments to address both scaling in the over-traing regime and benchmark performance prediction.
Motivated by the practice of training beyond compute-optimality we first investigate whether scaling follows reliable trends in the over-trained regime. We notice, as implied by Hoffmann al. (2022), for a set of models of different sizes trained with a constant ratio tokens to parameters, models' reducible loss L' (Hestness et al., 2017; Hoffmann et al., 2022) follows power law (L' X C-") in the amount of training compute C. We find that as one increases the ratio tokens to parameters, corresponding more over- werrttraning, the scaling exponent remains about the same, while the scalar changes. We explain our observations by reparameterizing existing scaling laws in relation to the amount of over-training.
To establish empirically that scaling extrapolates in the over-trained regime, we further experiment with testbed of 104 models, trained from scratch three different datasets: C4 (Raffel et al., 2019; Dodge et al., 2021), RedPajama (Together Computer, 2023), and RefinedWeb (Penedo et al., 2023). We find that scaling laws fit small models can accurately predict the performance of larger models that undergo more over-training Figure (left) illustrates our main over- werrtrrinnng result, where we invest 2.4e19 FLOPs to extrapolate the C4 validation performance on 900B tokens, which requires x00x more compute to train.
1.4B parameter model trained
In addition to over-training, we also investigate if scaling laws can predict the performance model on downstream tasks. We establish power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks. While can be difficult to predict the еттог individual tasks, find it possible to predict aggregate performance from model's 's perplexity among models trained on the same training data. Figure (right) presents our main downstream error over a set of downstream tasks to within percentage point for a 6.9B compute-optimal model, which requires 20 more compute train.
prediction result, where invest 2.7e20 FLOPs to predict the average top-
2

---

Published as a conference paper at ICLR 2025
Our results suggest that the proposed scaling laws are promising derisk (i) the effects of over- training models and (ii) the downstream performance of scaling up training recipes. To facilitate further research on reliable scaling, will provide all results of our experiments.
DEVELOPING SCALING LAWS FOR OVER-TRAINING AND DOWNSTREAM TASKS
In this section, we develop scaling laws to predict over-trained and downstream performance, First, we provide key definitions (Section 2.1). We next present scaling law for over-training drawing on empirical observation and prior work (Section 2.2). To connect loss scaling and downstream error prediction, we observe that average top-1 error decreases exponentially as a function of validation loss, which we formalize as a novel scaling law (Section 2.3). In later sections, we build an experimental setup (Section 3) to quantify the extent to which our scaling laws extrapolate reliably (Section 4).
2.1 PRELIMINARIES
Scaling laws for loss. Typically, scaling laws predict model loss L as a function of the compute C in FLOPs used for training. If one increases the number of parameters N in model or the number of tokens D that a model trained on, compute requirements naturally increase. Hence, we assume C' is a function of N, D. Following Kaplan et al. (2020), we use the approximation C = 6ND, which Hoffmann et (2022) independently verify. We consider,
L(C) =(C)=E+I((C),
(1)
where E is irreducible loss and L' the reducible loss. E captures the Bayes error minimum possible loss achievable on the validation domain. The L'(C) term captures what can possibly be learned about the validation domain by training on source domain. L'(C) should approach zero with increased training data and model capacity. TC(C) is often assumed to follow power law: ['(C)= = X C- (i.a., Hestness al. (2017); OpenAl (2023)). is also often helpful to consider power law in log- log plot, where it appears as a line with slope -1j and y- y-intercept log (A).
Token multipliers. We define a token multiplier M = D/N as the ratio of training tokens to model parameters for notational convenience. M allows consider fixed relationships between D and N even as a model gets bigger (i.e., as N becomes larger).
Compute-optimal training. Hoffmann et al. (2022) establish compute-optimal training, where, for any compute budget H, the allocation of parameters and tokens is given by,
arg min L(N, D) s.t. C(N,)) H. N,D
(2)
To solve for the optimal N D" one can sweep N, D for each compute budget, retaining the best configurations. Hoffmann et al. (2022) find that the compute budget increases, N and D scale roughly evenly. Assuming equal scaling, there is fixed compute-optimal token multiplier M D'/N' /N* per training distribution.
Overning We define over-training the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number tokens (i.c., M > M*). While loss should be higher than in the compute-optimg allocation for given training budget, the resulting models have fewer parameters and thus incur less inference cost.
2.2 SCALING LAWS FOR EVER-TRAINING
To propose a scaling law for over-trained models, we first turn to empirical observation. We train four model configurations with parameter counts between 0.01 and IB for token multipliers M between 20 and 640, where M 20 points lie roughly on the compute-optimal frontier, and larger M corresponds to more over-training. We defer experimental details to Section 3 to focus observations first. In Figure 2, we show loss against compute in a log-log plot for the models trained on three datasets and evaluated on the C4 eval We notice parallel lines when fitting power laws to
3

---

Published as a conference paper at ICLR 2025
Training set: C4
Training set: RedPajama
Training set: RefinedWeb
640
N 0.0118
Raduciale ose ca ova
0.0798
320
N 0.154B
token multipller M
N 0.4110
160
Interpolation
Extrapolation
10
10th
10ah
10
10tH
10ah
10 Compute (6ND, [FLOPs]
10
10at
Compute (6ND. MN) [FLOPs]
Compute (6ND,D MN) [FLOPs]
Figure 2: Scaling in the over-trained regime follows consistent power law exponents. We notice parallel lines in the log-log plots of reducible loss vs. training compute for range of token multipliers M. which give the ratio of training tokens to model parameters. Larger M corresponds to more over-training. For power law giving reducible loss function of compute: ['( À C-", the exponent v remains relatively constant resulting in lines with approximately fixed slope (Figure 17). The scalar À that determines the y-intercept, however, shifts with different token multipliers. This suggests À is function of the token multiplier, while is not.
the reducible loss, which suggests a near-constant scaling exponent even with increased over-training. This indicates that scaling behavior should be describable in the amount of over-training.
In search of analytic expression for the observations in Figure 2, we consider existing scaling literature. A common functional form for the risk of a model, as proposed in prior work (Rosenfeld al., 2020; Hoffmann al., 2022) is,
L(N, D) E F AN + BD-
Recall from Section 2.1, N is the number of parameters and D the number of training tokens. The constants F, A, a, B are fit from data. By fitting this parametric form, Hoffmann al. (2022) find that scaling exponents a and 3 are roughly equal, suggesting that one should scale N and D equally as compute increases. Hence, we assume a 3. With this assumption, we reparameterize Equation (3) in terms of compute C = D and a token multiplier M = D/N. We get.
L(C, M) E + (aM" + bM C-",
where 7 a/2, A(1/)) " derivation, see Appendix A.
(11 -1 gives the relation to Equation (3). For complete
Equation (4) has the following interpretation: (i) The scaling exponent 7j is not dependent on M. Thus, we always expect lines with the same slope in the log-log plot-as in Figure 2. (ii) The term aM" + 6M-- determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the log-log plot for the range of M we consider-alss consistent with Figure 2.
Recall that we make the assumption B, which implies equal scaling of parameters and tokens as more compute available. However, as explained Appendix A, even * 3, we get parameterization that implies the power-law exponent remains constant with over-training.
2.3 SCALING LAWS FOR DOWNSTREAM ERROR
Scaling typically studied in the context of loss (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al., 2023b), which Schaeffer et al. (2023) note is smoother than metrics like accuracy. However, practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss perplexity evaluation sets. To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure 2. In Figure 3, we plot average downstream top-1 errors over evaluations sourced from We defer details of the setup to Section to focus here on a key observation: average error appears to follow exponential decay as loss decreases.
MMo 2023) against the C4 eval loss.
4

---

Published as a conference paper at ICLR 2025
Training set: C4
Training set: RedPajama
Training set: RefinedWeb
Average top3 ereer: 17ask salt
0.80
0.80
0.75
0.75
0.75
0.70
0.70
0.70
0.60
0.60
0.60
Model
Interpolation
Extrapolaton
0.50
0.50
0.50
Loss: eval
Loss: eval
Figure 3: Average top-1 error scales as a function of loss. We plot models trained on three datasets and notice exponential decay of average top-1 error C4 eval loss, the x-axis, decreases. We consider on the y-axes average еπrог on 17 evaluations where performance is at least 10 points above random chance for at least one 0. scale model. These observations suggest that average top-1 error should be predictable with reliable loss estimates.
Based on the exponential decay we observe in Figure 3, we propose the following relationship between downstream average top-1 error Err and loss L,
Err(L) k exp (-yL),
(5)
where k 7 are fit from data. Equation (5) also has an interpretation in terms of model perplexity PP(L) exp (L),
Err(PP) c k PP
(6)
Namely, Err follows power law PP that bounded from above by signifying arbitrarily high error and from below by - h exp(-> F),, where E is the Bayes error from Equation (4).
Equation (5) in conjunction with Equation (4) suggests three-step method to predict Err as function of compute and the amount of over-training. For choices of training and validation distributions, (i) fit scaling law to Equation (4) using triplets compute C, token multiplier M, and measured loss L on a validation set to yield (C, M) + L. (ii) Fit a scaling law to Equation (5) using pairs of loss L and downstream error Err for models to get L + Err. (iii) Chain predictions to get (C, M) + Err.
3 CONSTRUCTING A SCALING TESTBED
In this section, we discuss our experimental setup to test the predictions suggested by Equations (4) and (5). We first present our general language modeling setup (Section 3.1). Next, we discuss strategy for determining model configurations for our scaling investigation (Section 3.2) and fitting scaling laws (Section 3.3). We then present metrics validate how well scaling laws predict loss and downstream performance (Section 3.4).
3.1 TRAINING SETUP
We train transformers (Vaswani et al., 2017) for next token prediction, based on architectures like GPT-2 (Radford et al., 2019) and LLaMA (Touvron al., 2023a). We employ GPT-NeoX (Black et al., 2022) as a standardized tokenizer for all data. See Appendix B for architecture, optimization, and hyperparameter details.
3.2 MODEL CONFIGURATIONS
To get final configurations for the O. to 0.411B parameter models plotted in Figures and 3, first conduct a wide grid search over a total of 435 models, trained from scratch, from 0.01B to 0.5B parameters (Figure 4 (left)). We train the original OpenLM data mix (Gururangan al., 2023), which largely consists of RedPajama (Together Computer, 2023) and The Pile (Gao et al., 2020). While we eventually plan to over-train models, at this step search for base configurations near
5

---

Published as a conference paper at ICLR 2025
Grid search models
Selected models
Loss: Open evall
Target 1.48 model
Target
interpolation
Extrapolation
1.8
10
10 Compute (6ND) (FLOPS)
10
10
10th
10
10
10th
10
Compute (6ND) [FLOPS]
Compute (6ND) (FLON)
Figure 4: Search, filter, fit: A recipe for selecting configurations for scaling, (left) To generate the final configurations presented in Table 3, we run 435 model grid search over model width, hidden dimension, number of attention heads, batch size, and warmup steps. All models are trained near compute-opimmally. (center) We plot the efficient frontier of models, which appear to follow a trend, excluding models from 5.2 x 10 to 5.2 x 10²7, which fall below the trend. (right) We fit a power law with irreducible error to the remaining configurations, picking four configurations that closely track the full model suite ("Selected models"). These models extrapolate the performance 1.4B, 6.9B target models. Shaded regions represent bootstrap 95% confidence intervals.
compute-optimality. We train on 20 tokens per parameter (M = 20), which, in early experiments, gives models near the compute-optimal frontier. This is similar findings Hoffmann al. (2022)'s Table 3, which suggests that M = 20 is near-optimal for the Chinchilla experimental setup.
To find maximally performant small-scale models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size. Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles. We find in early experiments that qk-LaaerNorm makes models less sensitive to learning rate, which a phenomenon Wortsman et al. (2023) report in their Figure 1. Hence, we fix the learning rate (3e-3) for our sweeps. We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at
20, retaining the best configurations.
At this point, we have many models, several of which give poor performance; following prior work (Kaplan et al., 2020; Hoffmann et al., 2022), we want to keep only models that give best performance. Hence, in Figure 4 (center), While there appears to be a general trend, configurations between 5.2 x 10 and 5.2 x 10 FLOPs lie below the frontier established by other models. We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of batch sizes. We provide support for this hypothesis in Appendix E, but opt remove these models from our investigation.
filter out models that do not lic the Pareto frontier.
To ensure tractable compute requirements for our scaling experiments, require subset of models that follows the trend of the entire Pareto frontier. In Figure 4 (right), we fit trends to the Pareto models and to subset of four models. We notice that the trends closely predict both the performance of the 1.4B and 6.9B models, suggesting that our small-scale configurations reliably extrapolate in the compute- compute-oppimal setting.
Moving forward, we do not tune hyperparameters for other token multipliers (i.e.. M * 20), on other training or evaluation distributions, or on validation sets for downstream tasks. For more details including specific hyperparameters, see Appendix C.
To create our scaling testbed, we start with the four small-scale, base configurations from our grid search: N E {0.011B, 0.079B, 0515, 0.411B). To ensure our conclusions are not particular to a single training distribution, we train models on each C4 (Raffel et al., 2019; Dodge et al., 2021), RedPajama (Together Computer, 2023), and RefinedWeb (Penedo et al., 2023), which have 138B. 1.15T, and 600B tokens, respectively, for different token multipliers M (5, 10, 20, 40, 80, 160. 320, 640). We omit runs that require more tokens than are present in a dataset (i.e.. N = 0.411B, M = 640 for C4). We additionally train N 1.4B models at M 20 and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for

---

Published as a conference paper at ICLR 2025
Table 1:
Default number of parameters . and token multiplier M to fit our scaling laws. We
invest ~ 100 A100 hours to fit Equation (4) and ~1,000 A100 hours to fit Equation (5).
N
M
Used to fit Equation (4) Used to Equation (5)
0.011B
0.079B
20
0.154B
20
/
0.411B
20
/
0.011B
320
1.4B
20
x
Total compute [FLOPs]
2.4e19
2.7e20
RedPajama, and 320 for RefinedWeb). We train N 6.99, .M = 20 models on each dataset given the relevance of 7B parameter models (Touvron al., 2023a; Jiang al., 2023). In total this results in testhed of 104 models.
3.3 FITTING SCALING LAWS
We fit Equation (4) to approximate E, a, b, 17 using curve-fitting in SciPy (Virtanen et al., 2020) (i.e., Levenberg- eevenberg-Marquard to minimize non-linear least squares). We repeat this process to fit Equation (5) to approximate c, k, 7/ We invest ~00 A100 hours to train the models required to fit a scaling law for loss and ~1 1,,000 A100 hours for a corresponding law for downstream error. Unless otherwise specified, we fit to the N, M pairs in Table 1, which are a subset of our full testbed. Our configurations allow
test for extrapolation to the N 1.4B, M 640 (900B token) and the N 6.9B, M = 20 (138B token) regimes.
3.4 EVALUATION SETUP
Evaluation datasets. Unless otherwise stated, our default validation loss dataset is C4 eval. For downstream tasks, adopt subset from 46 tasks from LLM-foundry (MosaicML., 2023), which includes standard tasks with both zero-shot and few-shot evaluations. Specifically, we consider a 17-task subset where, for each evaluation, least one 0.154B scale model-trained with as many as 99B tokens-geet gets 10 percentage points above chance accuracy: ARC-Easy (Clark et al., 2018), BIG-bench: CS algorithms (bench authors, 2023), BIG-bench: Dyck languages (bench authors, 2023), BIG-bench: Novel Concepts (bench authors, 2023), BIG-bench: Operators (bench authors, 2023), BIG-bench: QA WikiData (bench authors, 2023). BoolQ (Clark al., 2019), Commonsense QA (Talmor et al., 2019), COPA (Roemmele et al., 201 CoQA (Reddy et al., 2019), HellaSwag (zero-shot) (Zellers al., 2019), HellaSwag (10-shot) (Zellers al., 2019), LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), PubMed QA Labeled (Jin et al., 2019), SQuAD (Rajpurkar et al., 2016), and WinoGrand (Levesque et al., 2012). For more details on evaluation datasets Appendix D. We focus on this subset ensure downstream tasks like MMLU (Hendrycks et al., 2021), where performance is close to random chance, however, does invalidate our results
measuring signal, not noise. Including
show in our evaluation set ablations (Appendix E).
Metrics.
We consider three main metrics: Validation loss, which is the cross entropy between a model's output and the one-hot ground truth token, averaged over all tokens in a sequence and over all sequences in a dataset. Average top-l error, which a uniform average over the 17 downstream evaluations, mentioned the above paragraph. To measure how good prediction <(C, M) is, we measure Relative prediction error: (C(C, M) ferl/Ser, where is the predicted loss L or the average top-1 error Err. ŞGT is the ground truth measurement to predict.
RESULTS: RELIABLE EXTRAPOLATION
In this Section, we quantify the extent to which the scaling laws developed in Section extrapolate larger model performance using the scaling testhed from Section 3. By default, we fit Equations (4)
7

---

Published as a conference paper at ICLR 2025
Train: C4
Train: RedPajama Eval: C4 oval
Train: RefinedWeb Eval:
Eval: eval
20.0%
0.0118
1.1% 0.0%
6.2% 0.7% 0.9% 0.0%
6.6%
0.3% 0. 0.0%
0.3% 1.7% 1.1% 0. 0.0%
0.9%
0.9%
1.9%
1.0%
0.0%
5.15%
0.0798
2.6%
0.3%
2%
0.4%
0.1%
0.7%
8%
2. 2.2%
0
0. 7% 1.4%
3%
2.4%
1%
0%
0.5%
1.2%
0%
0.9%
Relsfive error
1.5%
176
0.3%
0.9%
0.6%
6.0%
0.411B
0.5%
2.8%
0.2%
2.0%
0.2%
0.3%
0.3%
1.4%
1.1%
0.2%%
0.15%
0.8%
0.9%
0.9%
0.3%
4.0%
1.48-
0.8%
1.5%
0.1%
0.7%
0.0%
2.0%
6.98
4.3%
0. 0.7%
666
10
20
80-
160
320
640
320
640
10
20
40
80
260
320
640
0.0%
Figure 5: Relative error on C4 eval for different training distributions. Boxes highlighted in
yellow correspond to pairs-number of parameters N, token multiplier M-used to fit Equation (4). Larger values of M correspond more over-training. The prediction error low in both interpolation and extrapolation ranges. Below N 1.4B, empty squares correspond to runs that were not possible due to the limited dataset size for single epoch training. At N = 1.4B we run at M = and the largest possible multiplier. At N 6.9B, we run at M 20.
and (5) the configurations in Table use C4 eval for loss, and the 17-task split from Section 3.4 for average top-1 error.
Over-trained performance is predictable. We highlight our main over-training results in Figure 1 (left). Namely, we are able to extrapolate both in the number of parameters N and the token multiplier M to closely predict the C4 eval performance of 1.4B parameter model trained on 900B RedPajama tokens (N = 1. 1.4B, M 640). Our prediction, which takes 300x less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error. Additionally, for the N 6.9 .M 20 run, near compute-optimal the relative error is also 0.7%.
These results support several key takeaways. (i) Scaling can predictable even when one increases both the model size and the amount of over-training compared to the training runs used to fit a scaling law. (ii) The form presented in Equation (4) is useful in practice for predicting over-trained scaling behavior. (iii) Fitting to Equation (4) gives good prediction accuracy compute-optimal More specifically, predictions are accurate both for the 1.4B over-trained model and the 6.7B compute- optimal model using single scaling fit.
While Figure explores a specific case of making predictions in the over over-trained regime, we aim to understand the error profile of our predictions across training datasets, token multipliers, and number of parameters. Hence, Figure 5 shows the relative error between ground truth loss and predicted loss C4 eval for models in our testbed. We notice uniformly low prediction error suggesting that predictions are accurate in many settings.
Average top-1 error is predictable. Figure/ (right) presents our main result in estimating scaling laws for downstream error. Concretely, we use the models indicated in Table to fit Equations (4) and (5), chaining the scaling fits to predict the average top-1 error as a function of training compute C' and the token multiplier M. Our fits allow us to predict, using 20x less compute, the downstream performance of 6.9B model trained on 138B RedPajama tokens within 0.05% relative error and 1.4B model trained on RedPajama 900B tokens to within 3.6% relative enror.
Table 2 additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law functional forms are applicable on many training datasets. We note that while average accuracy is predictable, individual downstream task predictions are significantly more noisy. We report relative error for more model predictions in Figures 11 and 12. We also find that we remove the 1.4B model for the Equation (5) fit, relative error jumps, for instance, from 0.05% to 10.64% the 17-task split for the 6.9B, 138B token RedPajama prediction. This highlights the importance of investing more compute when constructing scaling laws for downstream task prediction compared loss prediction.
8

---

Published as a conference paper at ICLR 2025
Table 2: Downstream relative prediction error at 6.9B parameters and 138B tokens. While predicting accuracy on individual zero-shot downstream evaluations can be challenging ("Individual"), predicting averages across downstream datasets is accurate ("'Agg.).
Individual top-1 error
Avg. top-1 error
Train set ARC-E
LAMBADA
OpenBook QA
HellaSwag
17-task split
C4
28.96%
15.01%
16.80% 8.44%
79.58%
0.14%
RedPajama 5.21%
14.39%
25.73%
0.05%
RefinedWeb
26.06%
16.55%
1.92%
81.96%
2.94%
Under-training, out-of-distribution scaling, compute-reliabilig trade-offs. In addition to our main results presented above, we include additional results Appendix E, which we summarize here. First, we notice that when token multipliers become too small (i.e., M 5) scaling becomes unreliable and lies off the trend. Additionally, multipliers other than 20, such as 10, 40, and 80, garner points that roughly the compute optimal frontier (Figure 9). This observation suggests that the compute-optimal multiplier may lie in a range rather than take a single value. To probe the limits of reliable scaling, we attempt to break models trained on C4 --nglishh filtered and evaluated on next token prediction on code domains have high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction gives reliable loss scaling (Figure 10). We additionally examine the compute necessary to create accurate scaling laws, finding that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures 15 and 16).
scaling laws in -itto---iitriuttion settings. We find that
RELATED WORK
We review the most closely related work in this section. For additional related work, see Appendix
Scaling laws. Early works on scaling artificial neural networks observe predictable pawer-law scaling in the training set size and number of model parameters (Hestness et al., 2017; 2019; Rosenfeld al., 2020). Alabdulmohsin al. (2022) stress the importance of looking at the extrapolation regime of scaling law. Yang et al. (2021) prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. (2024) make analogous recommendations when scaling model depth. Bi et al. (2024) propose hyperparameter aware scaling laws. Unlike the aforementioned work, our investigation focuses on over- over-training and predicting downstream accuracy.
Hoffmann et al. (2022) investigate how the number of model parameters N and training tokens D should be chosen to minimize loss L given a compute budget C. Hoffmann et al. (2022) find that when scaling up C, both N and D should be scaled equally up to multiplicative constant (i.e., N x C-0 and D xx CC00.5) to realize compute-optimality. Appendix C of the Chinchilla paper additionally suggests that these findings hold across three datasets. However, Hoffmann et al. (2022) do not verify their scaling laws for training beyond compute-optimalin or for downstream error prediction- both of which are central to our work.
Sardana & Frankle (2023) propose modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimality and solve for various fixed inference budgets. Their key finding, which is critical for our work, is that when taking into account large enough inference budget, is optimal to train smaller models for longer than the original Chinchilla recommendations. Our work presupposes that over-training can be beneficial. Instead of solving for inference-optimal schemes, we support empirically predictive theory of sealing in the over-trained regime. Additionally, we provide experiments across many validation and training sets.
For predicting downstream scaling beyond loss, Isik et al. (2024) relate the number of pre-training tokens to downstream cross-entropy and machine translation BLBU score (Papineni et al., 2002) after fine-tuning. In contrast, we take a holistic approach to evaluation by looking at top-1 error over many natural language tasks. Schaeffer et al. (2023) argue that emergent abilities (Wei et al.. 2022b) are product of non-line non-linear metrics and propose smoother alternatives. As a warmup for why non- non-linear metrics may be hard to predict, Schaeffer et al. (2023) consider predicting an length sequence

---

Published as a conference paper at ICLR 2025
exactly: Err(NN, = 1 PP(N)- where N is the number of parameters in model and PP is its perplexity. This is a special case of our Equations (5) and (6), where the number of training tokens does not appear,
1, k 1, and 7 L. In contrast, law fit, finding that average error over downstream tasks can make for predictable metric. Owen (2024) observe the scaling behavior of many open source models downstream tasks. However, their study does not control for different architectures, training codebases, optimization schemes, and training datasets. We create standardized, open-source setting, which controls these factors.
treat k, Y free parameters for a scaling
Over-training in popular models. There has been rise in over-trained models (Touvron et al., 2023a;b; Llama Team, 2024) and accompanying massive datasets (Together Computer, 2023; Penedo et al., 2023; Soldaini et al., 2024; Albalak et al., 2024). For example, Chinchilla 70B (Hoffmann et al., 2022) is trained with a token multiplier of 20, while Llama-2 7B (Touvron et al., 2023b) uses a token multiplier of 290. In our investigation, we look at token multipliers from 5 to 640 for coverage of popular models. The recent Llama3 8B model is a notable outlier, with token multipliers of 1900. However, it is unclear if, at 15T tokens, Llama3 8B was trained in the single epoch regime considr in this paper. Practically. training a 1.4B parameter model at this multiplier is prohibitive due to I) compute limitations and 2) the 2.8T training token requirement for single-epoch run, which is larger than public datasets at the time of our training runs.
6 LIMITATIONS, FUTURE WORK, AND CONCLUSION
Limitations and future work. We identify limitations, which provide motivation for future work.
Hyperparameters. While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is need to develop scaling laws that do not require extensive hyperparameter sweeps.
Scaling up. Validating the trends in this paper for even larger runs is a valuable direction. Additionally, repeating our setup for models that achieve non-trivial performance on harder evaluations like MMLU is left to future work.
Scaling down. Actualizing predictable scaling with even cheaper runs important to make this area of research more accessible, especially for downstream error prediction.
Failure cases. While we present preliminary analysis of when scaling is unreliable, future work should investigate conditions under which scaling breaks down.
Post-training. It is common to employ fine-tuning interventions after pre-training, which do not consider. Quantifying to what degree over-training the base model provides benefits after post-training an open area of research.
Individual downstream task prediction. Accurate per- task predictions are left future work. In-the-wild performance. Downstream task performance is proxy for the in-the-wild user experience. Analyzing scaling trends in the context of this experience timely.
Dataset curation. Our work only deals with existing training datasets. Exploring dataset curation for improved model scaling is another promising direction.
Conclusion. We show that the loss of over-trained models, trained past compute-optimality, is predictable. Furthermore, we propose and validate a scaling law relating loss to average downstream task performance. We hope our work will inspire others to further examine the relationship between model training and downstream generalization. Our testbed will be made publicly available, and we hope will make scaling research more accessible to researchers and practitioners alike.
ACKNOWLEDGEMENT
SYG is supported by an NSF Graduate Research Fellowship. GS by the Onassis Foundation Scholarship ID: F ZS 056-1/2022-2023, and MN by the Federal Ministry of Education and Research of Germany under grant no. 011S22094B WEST-AL We thank Stability Al and Toyota Research Institute (TRI) for access to compute resources. This research has been supported by NSF Grants AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, and research gifts by Western Digital, Amazon, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco, and the Stanly P. Finch Centennial Professorship in Engineering. We also thank Kushal Arora, Alper Canberk, Mia Chiquier, Sachit Menon, Mariah Oxley, Chuer Pan, Purva Tendulkar, and Mandi Zhao for valuable feedback.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In International Conference on Learning Representations (ICLR), 2022.
https: /ttp/vaaivvv/vv.///iv//../...avl org/obgg//eeooo 2110. 02095.
Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Advances in Neural Information Processing Systems (NeulPS), 2022.
https:// //axxiv org/ /bg/abs//// 2209. 06640.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan. Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint, 2024. https: / /wwpiv..vvvvvv//////////////aa www.vvorrgggaae org/abs//////aaaao.aa//a.......r 24402 6827.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff. Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't t reach for the stars! arXiv preprint. 2023. https: arxiv org/ abs 2301 03988.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Concel-Kedziorsie Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-basee formalisms. In Conference of the North American Chapter of the Association for Computational Linguistics (NACCL), 2019, https: hatps://aalaattin waw.aaaaaaattoooo..oooo...o.ooo. orgg 1245.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, David Berard, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Laurent Kirsch, Michael Lazos, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2024. https: htt.D/tttpppppttttcc..ogggg ort/blog///ooooooooooooooooooooo hwwpllcgppytochhhhhh/oo 2 -apppapp----ee -apper-tutoria.. tutorial.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. ftttss:/acclantholoos emn 1 p--ma nn. 804.
orgg/022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint, 2016. https: arxiv. org/ httgg//b////// 110. 06450.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint, 2021. attps://arxxe wttp:////...././////1/
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang. Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning (ICML), 2022. https: https////cccceeeeeeeeeeee mlr. press/vil 62 bt/.aaanaallbbbb html.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In Transactions on Machine Learning Research (TMLR), 2023. https: // /ppnneeview. net httt://oooooooollmeee
Emily M Bender, Timnit Gebru, Angelina McMillan-Main Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings ACM conference on fairness, accountability, and transparency (FAccT), 2021. https: / /dl.. acm. org/dod)
10. 1e..114/3300 4421 421 8888888888888 012211888844442 3445922.
11

---

Published as a conference paper at ICLR 2025
DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang. Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song. Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang. Tong Wu, Yu Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang. Yu mei You, Shuiping Yu, Xin yuan Yu, Bo Zhang. Haowei Zhang, Lecong Zhang. Liyue Zhang, Mingehuan Zhang, Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek I1m: Scaling open-source language models with longtermism. arXiv preprint, 2024. https: /arpxii//axxvvvv/ wwwoorggbbb...
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, al. Bloom: A 776b-parameter open-access multilingual language model. arXiv preprint, 2022. https:
www.xiv.orggaba /ttp://2211...1110
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence (AAAI), 2020. https: //apivvaaaivva///aa/aaa/aaaaaaaa org/ abs 10los/19:0 191//991111. 11641.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao. Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler. USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. Gpt-neox-2066 20b: An open-source autoregressive language model. BigScience Episode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. https: / aclanthologgy . org/ 2022. bww.cienceeeeeeeee 1.9.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020. https: //axxivvv.../////////aaa//aaaavl org/ abs/ 2005 14165.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In International Conference on Learning Representations (ICLR), 2023. https: /ttess//opeeeeieeeee
net/ pecrummidd=sce jveqlCZ. CZ.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference Computer Vision and Pattern Recognition (CVPR), 2023. https: //tp///aavvvv///////////eeee.aav wwwpxxvvvorr//eeeomooo wrr.or///aaaoo //s///////// 2212.07143.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García. Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstent Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. In Journal of Machine Learning Research (JMLR), 2022. https : /arxivvvv. org/ abs 2204. 02311.
12

---

Published as a conference paper at ICLR 2025
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, et al. Scaling instruction- finetunnd language models. arXiv preprint, 2022. https: httparxxvvvvoo..e/ea org. abs
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang,
Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
https: http/aaaannntoooo...oo.oooo org/ N19 1300.
Kevin Clark, Minh- ThangT Luong, Quoc V. Le, and Christopher D. Manning, ELECTRA:
Pre-training text encoders as discriminators rather than generators. International Conference on Learning Representations (ICLR), 2020. https: /ptp//eeeeeeeeeeeeeeeeeeeeeeeeee net. / ii=rr AME1BtVE.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint, 2018.
org awtaaassss10 05457.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. Advances in Neural Information Processing Systems (NeurIPS), 2022. https:
aativ.iiiivv///ii//aaa/aaaaaavva http://aaaawaaaaaaaa/aass/essaea hwtp:/ab//22000 2ttp://22....1.......
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023. https:/ /ppppoeeedingsssse mlr www.s/eessseeee dehghani 22a html.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. https: // //aclannhologyy.... wtggN19-11/0
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. https://acl ht//accaanhhooggyy. org/ /tgg/22211 wwneemmlll..mmmm ma in. 98.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mfxture-of-experts. In International Conference on Machine Learning (ICML), 2022. https: attps://arxivo: http///xxxvvo///// ortl 2112.0690s
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint, 2024. https: / //////aaa waw.v.ovvvooo////o abs/2402. 01306.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Mitchell Wortsman Ryan Marten, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Mehdi Cherti Richard Vencu, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis. Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https: wawpiiiiii..///.......aaiiaaaa.. org/ abs /2004 14108.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite. Noa Nabeshima. Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint, 2020. https: //arxiv. wwtppiiwoorooa/////op/o
00027.
13

---

Published as a conference paper at ICLR 2025
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint, 2021. httppp///aaxx..org/aaaaaa
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. https: /alp:://ccllee /aaaaaaaaanthooo
org/2021 emnlp manlp-mminn.... main. 478.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint, 2024. https: eetps://arxiae //tp///aavvv////////////////aaa/ org wwg/abs/////aaaaaaaae/.aaataaaar httgg//bb//22//// 2400.. 00838.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint, 2023. https:/ /ttp///axivvvvvv www.vvv.rrggaaa... 2tt///312.0000
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers, In Advances in Neural Information Processing Systems (NeurIPS), 2021. https: httpp:://ppeneee /oppnreven/....eee net/ Eorum? i Ldy3Wd42CWN3...
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR), 2022. https: // //aaxiii.orgg.e.......a........a /ttps.//11..00000.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. Preprint, 2023. https : /www mi microsoft. are-a all need.
com/en-us/ rtw./ereeaaccccecc wwwpaarcccppbbbbiac wpwplicctaintttettboooooooo
Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. OpenLM: a minimal but performative language modeling (Im) repository, 2023. https: /ttp::://ttttiii hhb. mmlmmmle. foundat i ons/ lopen_le..
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021. https: aaaiiiiiiiii//ii www.wwwwww................oowoot abs /olbs/20000 2009. 03300.
T. J. Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. arXiv preprint, 2020.
httts::/arxiv..rrt abos 22000.000022 14701.
Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint, 2021. /ttt::////xx....rr/aaab////// 2202. 01293.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is 1712. 00409.
predictable, empirically, arXiv preprint, 2017. pettps:/aaxivv.oot
Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Computational challenges in deep learning. In Principles and Practice of Parallel Programming (PPoPP), 2019.
pettps://arxte org/ httg//ab////// albs/1909. 01736.
Jordan Hoffmaan,, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. In Advances Neural Information Processing Systems (NeurIPS), 2022. attps://arts htt//axxvvvoo wwtpp/iwwww/.../oooo/oooooooese htarg//aba/20000 /203. 15556.
14

---

Published as a conference paper at ICLR 2025
Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word classifiers: A loss framework for language modeling. In International Conference on Learning Representations (ICLR), 2017. https:/ arxiv. http://ttww...aaa///////aa 2tg/abbs/611..14....
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance large language models, arXiv, 2024. https: /awxivivvvv..////////aa//aaaaav/ org/ abs///ss/22/00 04177.
Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. petps://aclanteen Findings-entr e.m.p.mmlpp........ 4.
org/2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Florian Bressand Diego de las Casas, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint, 2023.
https: arxiv...... org/ abs//bb///// 2310. 06825.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu. William Cohen, and Xinghua Lu. Pubmedqa: dataset for biomedical research question answering. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2019. https:/ //aclaaahoooooooooooo ogy. .org/rrggg. 2 1259.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint, 2020. https: /arxiv.vvvvv. www.riiivooooraaooooooo abs /2011 08361.
Tobit Klug, Dogukan Atik, and Reinhard Heckel. Analyzing the sample complexity of self-supervised image reconstruction methods. arXiv preprint, 2023. https: /ttp//aaxvvv..//.. abbs/////////// abs /2305. 19079.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint, 2019.
11942.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library, 2022. https://githus: xformers.
comm/facebookresearccc
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In International conference on the principles of knowledge representation and reasoning, 2012. https://aaal or.g wwwporrgrg//////////.//../../rs/ pappps/pp////aaaaaa /0/ 4.4 tee-thhewwite www.hhewwwnorrrdeeee schema--heccccccc chaa enge.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising
pre- training for natural language generation, translation, and comprehension. In Anmual Meeting of org/
the Association for Computational Linguistics (ACL), 2020. https: http://aaaaaahhllaoooo 2020. acl-main mmain.........mmmm.mmmmmmmmimm.m 703.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint, 2023, https: wrwia.avvooooom org/ abs 2305. 06161.
Jian Liu, Leyang Cui, Hammeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: challenge dataset for machine reading comprehension with logical reasoning. In International Joint Conference on Artificial Intelligence, 2020. https: /tt///axvvvv.////////ee////aaevm http://wwxxw.ooaas///////s/
08124.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint, 2019. http: / axaiiiiii wwt.lv..oo..////oo 907, 11692.
15

---

Published as a conference paper at ICLR 2025
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
httpps://arae
org/ abs// 22tp.220.... 03545.
AI Meta Llama Team, The llama 3 herd of models. arXiv preprint, 2024. https: org/abs 240..240..00 220.2217782. 21783.
Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint, 2023. https: / arxiv org/ abs 3100. 16787.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint, 2017. pictps://arxt org/ alss /obs/1111 05101.
Anton Lozhkov, Raymond Li. Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He. Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang. Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zehaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder and the stack v2: The next generation. arXiv preprint, 2024. https: httppp///aa..../////.//// .rgg 19179...1....
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna- Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. Fingpt: Large generative models for a small language. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023, https: httss///aaaatthhooooooeo www.hholllo.........o.3.33 org/2023. emr.p-me mo-maiio in. 164.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hannench Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. arXiv preprint, 2023. https: httpppallaaa..amm.ppooooa.aaa..a wwtpppalmaaaa.......maaaa allen.ai.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of English: The Penn Treebank. In Computational Linguisties, 1993. https:
//aclantho www.ooyyyooooooooooooo wt3////////00000
William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. https:
//aclanthology org/ 2021.
emnlp main. 133.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
abs/
1809.02789.
MosaicML. Llm evaluation scores, 2023.
11m- evaluation.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Annual Meeting of the Association hitps: / ////aaaannttt aallanthology orrgyo22..2....
for Computational Linguistics (ACL), 2022.
acl- acl-long. 8911
16

---

Published as a conference paper at ICLR 2025
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint, 2023a. /tttps://axiv..ooo wtwpa///.r.a//aaooaaaao.a..assaa //00 2012230888222...011
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems (NeulPS), 2023b. attps://aaxa org/ abs 2305 16264.
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint, 2024. https: //a wwwaaiivorggaaaaeeeeee. www.//gg//aaaa.ooooooo.o......./ 220.222222220 09906.
Erik Nijkamp. Tian Xie, Hiroaki Hayashi. Bo Pang, Congying Xia, Chen Xing. Jesse Vig. Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojeiech Kryscinski, Lidiya Murakhovs' ka, Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingho Zhou, Shafiq Rayhan Joty, and Caiming Xiong. Long sequence modeling with xgen: A 7b Ilm trained on 8k input sequence length. arXiv preprint, 2023. https://arxxe org/ www.//////0 03450.
OpenAl. Triton, 2021.
OpenAI. Gpt-4 technical report, 2023. httpps://arxiv.org//aaaat
08774.
David Owen. How predictable is language model benchmark performance? arXiv preprint, 2024.
https://arma wrgg/bsg////oooooooooo..ooo...ss hbtpg///bb///.....//// 2401.04757.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring broad discourse context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2016. http://wwe ac web. org/ antho
P16-1144.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), 2002. https: httpps//acaaaathhlloooo woweclannhooooyyooooooooo wrgg/02-110000 1040.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. https: //ac/aaaacccc /aclaanhhooogy....o.... 2002 Findings-acl. 165.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019, ht htsps: aaxii.iiiiv///a.a http://wwwwww.aaaaaa/////o baog//bb//1 ab....1...1013...
Patronus AI EnterprisePII dataset, 2023. https: ptttps://tinynn ctt.:/i.....oommmaa/aaeeeee wwt./r5x//2/t.....ooc
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouci. and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only, arXiv preprint, 2023, https: / htt///axvvvvr..eeea abs / 2000. 01116.
Bo Peng. Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng. Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanislaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2023. ht https: acaalaa meelnnhhooogy /org/ 2023. fi andis i ngs emn 936.
17

---

Published as a conference paper at ICLR 2025
Ofir Press and Lior Wolf. Using the output embedding improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2017. https: /aw.aaaaannnooo......o.o org Brg/EBL /E/1E1122222222
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Preprint, 2019. https: 8/ language_
Language models are unsupervised multitask learners.
///4/mmm //d4mccfpkssywwv..m.mmmm cl wwunfroo..oooooooooooooo http://ettterranguggeeeeeee wttp://iiweeeeeeeeeeeeeeeeeeeeee wearresrseeeeeesreeeeere pdf.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukeuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint, 2021. bttps://arxivv wrg./iiorrrraeoooe abs 221221222.. 11446.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS), 2023. https: http://aaxvvvvvo org/ abs/ /200. 18290.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint, 2019. https: /arxivv....///////////a//////a orgg/abs / 910. 10683.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang. Michael Matena. Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. In The Journal of Machine Learning Research (JMLR), 2020. https:
//arxiv.orgg///o abs/1910 10683.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2016. https: / aalaaclattt anthology. org/ DD/6/166- 1264.
Siva Reddy, Dangi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. In Transactions of the Association for Computational Linguistics (TACL), 2019. https:
//aclanthology wogg//////oooo 2t10001911111 1016.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium, 2011. httpp:://pppppp pittppppeeii..................ee ict .ac.eddu// -gordon/ copa. html.
Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive prediction of the generalization error across scales. In International Conference Learning Representations (ICLR), 2020. https: /ttp///wwwgg/aa//////o//o//tooaa 999. 12673.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. Conference of the North American Chapter of the Association for Computational Linguistics (NAACI)) 2018. https:
wwwg/r////N -0202..
18

---

Published as a conference paper at ICLR 2025
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint, 2019. hrt.v.o..i../.../.......... abs/1907. 10641.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint, 2019, http:///axiiy org/orgg/aoo /abs/ 1910.0110 01108.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Empirical Methods in Natural Language Processing (EMNLP), 2019.
http///llaaatttootoo/ org/ atg/D19/ 1454.
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In NeurlPS Workshop on Efficient Natural Language and Speech Processing (ENLSP), 2023. https: org/ abs
Teven Le Scao, Thomas Wang. Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, al. What language model train if you have one million gpu hours? In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2022. https:/ paaps://alaatttooi 54.
org/2022.22..... findingsnnneneee emnlp.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? In Advances in Neural Information Processing Systems (NeurIPS), 2023.
https: //ap:i//aavvv///e/eeeeeeeeaaevea org/ abs 2304 15004.
Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. In Journal of Machine Learning Research (JMLR), 2022. https: http::/:///xxxxo.or/////a/ 4004. 10802.
Noam Shazeer. Glu variants improve transformer. arXiv preprint, 2020. abs/2002. 05202.
Shivalika Singh. Freddie Vargus. Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint arXiv:2402.06619, 2024. https:// //axiv////aaivva/aa///a/aa org/ als 2402.0020 .0122.0661999
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Sehwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint, 2024. https:
//arxivv.oo org/ abs/ 2402.22000 00159.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https:/ /ttp::////peeeiwwww www..eee//eeee net/forumm forum?
id-UmvSIP-PyVV
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint, 2021. https://arxiv.
abs/2104. 09864.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA:: question answering challenge targeting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguisties (NAACL), 2019, https:
//aclanthooogg www.:/orgg/1gg10
Yi Tay, Mostafa Dehghami, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre- training and fine-tuning transformers. In International Conference on Learning Representations (ICLR), 2022. https:/ http:::/oppeerevieeeeeee net fww.mrrrrmooom (ot.ruummmmddffooooffoo wwt.::wwwweeeeeeeeeteeett
19

---

Published as a conference paper at ICLR 2025
Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In Conference Empirical Methods in Natural Language Processing (EMNLP), 2023. https: / anthology orgg 2223..22..2 f indings-emtn 825.
MosaicMI. NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023, www. mosai wtt. mp.t-b.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos. Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang. Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstenn Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications. arXiv preprint, 2022. ht ptttp:://rrivvt org/ abs 220022212222 08239.
Together Computer. Redpajama: an open dataset for training large language models, 2023. https: /RedP j -aaa-aata
//github hwtpb//cc/oottttteeeeeeeeeeeeeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. ILaMA: Open and Efficient Foundation Language Models. arXiv preprint, 2023a. https: /ttp///axivvvv/vv org/ abs/ 22002. 13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama Open Foundation and Fine-Tuned Chat Models. arXiv preprint, 2023b. https: / /aaxivv..v/////////a//aa///aavvv org/ abs 092888.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong. Wei- Yin Ko. Daniel D' D'suuza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned acen-access multilingual language model. arXiv preprint, 2024. aictps://arxtt
org/abs/ 2402.07877.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. https: http/rivvvvv//o/ org/ aww/aaassao 03762.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, JJ Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SeiPy 1.0: Fundamental Algorithms for Scientifie Computing in Python. Nature Methods, 2020. https: htopps:/rdduu /dcu. be/ 8whh..
20

---

Published as a conference paper at ICLR 2025
Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From Isat: The progress and challenges of complex reasoning. Transactions on Audio, Speech, and Language Processing. 2021. https: /ttp/v/aivvv//..///v//.../...ava www.ooggg/////oo.eoe.ooeeeeeeeee abs//108..221/ 00648.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners, In International Conference on Learning Representations (ICLR), 2022a. https::/openree net/Eorum?i dEr.ooozg...............
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sehastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In Transactions on Machine Learning Research (TMLR), 2022b. https: openreview. net/
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint, 2021. https: /awxiviivvv.//////i////////aaivv org/ /ww.aas/////ooooooooo/.../t..aee 2 112. 04359.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint, 2023. https: //axivv//aivv///// org/abss 2309.
14322.
Yizhe Xiong. Xiansheng Chen, Xin Ye, Hui Chen, Zijia Lin, Haoran Lian, Zhenpeng Su, Jianwei Niu, and Guiguang Ding. Temporal scaling law for large language models. arXiv preprint, 2024. attps:://rxil org/ als////s/////// alos/2404 2404. 17785.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. In Advances Neural Information Processing Systems (NeuPSS) 2021. https: /ttpavaxiivvv.v////i//eee//a.ava wtg. abs/ 2200..203.. 03466.
Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite depth neural networks. In International Conference on Learning Representations (ICLR), 2024. https: / //openreview net/ http///ommmmmd/dtmmmaee
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. https: http:///aaaatthhhh oog.. org/ 9-11--4772
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022, https: ptttps://arxiiee /aaxxvv. org/abs 2216 04560.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeulPS), 2019. http://///a.......t/a/////......
Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Empirical Methods in Natural Language Processing (EMNLP), 2019. http:://aaaaathhooggooe
Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. In Very Large Data Bases Conference (VLDB), 2023. https: //ll.. a.m. org/ doi/10. 107714778/2 111111111//11111111 07788/3611540 3611569..
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. Jec-qa: A legal-domain question answering dataset. Association for the Advancement of Artificial Intelligence (AAAI), 2020. https://arxie orae.org//ssso 120011
21

---

Published as a conference paper at ICLR 2025
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint, 2023. https:/ /ttps/////vv....//aaa///////////
http://aaa//233003333 06364.
Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efficient instruction tuning code large language 00788.
models. arXiv preprint, 2024.
22


Reviews:
[{"review_id": "0ZN8LC1oTi", "reviewer": "Reviewer_5cNe", "summary": "This paper highlights two issues overlooked in current discussions of scaling laws for large language models: (1) To reduce inference costs, smaller models are often trained on more data, and (2) scaling law research commonly uses perplexity to measure model performance, whereas real-world evaluations focus on task completion performance, such as accuracy. In response, this paper investigates and confirms scaling laws in large language models related to over-training and performance on downstream tasks.", "questions": "- Line 69-70: \"M = N/D\" seems to be a typo. It should be \"M = D/N.\"\n- Is Figure 1 a log-log plot?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The motivation is clear and the investigation is significant. Exploring scaling laws that are closer to real-world application scenarios provides strong guidance for the experimental setup of pre-training large language models.\n- The experimental logic is very clear. The authors first selected certain experimental settings to fit and obtain hyperparameters for the specific scaling law. Then, they validated its accuracy on over-training and downstream performance across additional experimental settings, which is a reasonable approach.", "weaknesses": "- Some experiments are missing. The authors’ initial motivation for studying the scaling law in over-training was to reduce inference costs, but this raises a question: training smaller models on more data achieves performance comparable to larger models under the same compute budget. However, this issue is not well explained—are there relevant experiments or clear prior studies addressing this?\n- The experimental validation of the scaling law for downstream performance is not rigorous enough. As shown in Table 2, there is a significant gap between the predicted and actual values of downstream performance based on the scaling law for individual datasets, while the average gap across 17 tasks is relatively smaller. Does this indicate that predictions from the downstream performance scaling law carry substantial risk? More specifically, we do not know for which specific number or types of tasks this scaling law provides accurate predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6ctnUmpPMa", "reviewer": "Reviewer_fbs9", "summary": "The paper investigates scaling laws for language models  (LM) in the over-trained regime, where models are trained with greater token-to-parameter ratios to lower inference costs.\nThey furthermore derive power-law scaling laws for downstream task performance (error rates) from the conventional laws using LM loss.\nUsing a testbed of 104 models (0.011B to 6.9B parameters) with varying token counts across three distinct datasets, they fit power-law scaling laws to predict validation loss and error rates, validating the consistency and reliability of the proposed laws.\nThe paper is well-motivated and addresses an important problem, with technically significant and practically valuable findings.", "questions": "* Lines 142-144: can you elaborate on this sentence: \"While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus incur less inference cost.\" and why it is \"common practice to over-train smaller models\" (line 44)?\n* The exponential relationship between loss and error is motivated in Figure 3, but have you evaluated larger sets of tasks to validate the relationship?\n* Does the paper assume that over-training effects remain uniform across model sizes and datasets?\n* Could the authors elaborate on the characteristics of tasks where downstream predictions were less accurate? Are there specific aspects that tend to yield higher prediction errors? \n* As the paper mentions exploring out-of-distribution scaling, can you provide findings on how well the scaling laws hold when tested on domains or tasks different from the training dataset?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The problem of scaling laws under over-training is practical and interesting, addressing gaps in existing literature around compute-optimal training, allowing for the efficient resource utilization in large-scale language modeling.\n* The proposed laws appear logical\n* The evaluation appears comprehensive, with a diverse and appropriate testbed to test the hypotheses around scaling laws under over-training. As a result, they achieved reasonably good performance: average top-1 error on 17 tasks within a 0.05% relative error and 0.7% relative error on validation loss.\n* The paper is clearly presented, with detailed steps for law derivation,  model configuration, and scaling law fitting.", "weaknesses": "- Performance of downstream task: Although average performance aligns with the laws, variability in individual predictions and occasional high errors may suggest that the scaling laws may not universally hold without further model or data refinements. Furthermore, the laws in equation 4 assume $\\alpha =\\beta$ which can limit the applicability.\n- The paper does not address how the scaling laws would hold under different architectural configurations or training optimizations, which are relevant to future model development. \n- The paper explored limitations such as sensitivity in out-of-distribution settings, posing challenges for future refinement of scaling laws in varied training regimes and tasks. \n- The claim of \"predictable downstream performance\" seems somewhat misleading, as the performance for specific tasks can vary widely. A more precise phrasing could emphasize “predictable average performance” to better align with the actual findings.\n- Presentation:\n1. In Figure 1, the paper defines M = N/D (line 70) while D=MN (Fig 1 caption) and line 127\n2. Figures need better visualization and explanation for understanding. For example, in Figure 1, colors for M=320 and M=640 are very similar.\n3. Typos: Line 142: than in", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xovT9Zxrm5", "reviewer": "Reviewer_tXac", "summary": "This paper explores the implications of over-training in LLMs, and tries to fit a scaling law for extrapolating beyond the compute-optimal (Chinchilla optimal) regime. The proposed scaling law is able to predict the final validation loss of a model using 300x less compute than what would have been needed to fully train the model.", "questions": "- How accurate would this method be for predicting the performance of much larger LLMs, such as 30B, 70B, 400B? Are there any special considerations in order to ensure the law stays accurate?\n- Would this law also work on \"efficient\" LM architectures such as mixture-of-experts, approximate attention and attention-free models?\n- How might overtraining affect post-training, fine-tuning or inference-time compute scaling methods? Would their benefits scale proportionally the same to the losses and evals?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Extensive experiments across many training datasets (C4, RedPajama and RefinedWeb) and model sizes.\n- Comprehensive evaluations using diverse validation datasets.", "weaknesses": "- As the field of LLMs is currently agressively expanding into multimodality (images, audio, video and data), the proposed method might soon not hold up anymore. There is no mention of early-fusion multimodal models or token-free language modeling.\n- Lack of consideration for post-training, fine-tuning and inference-time compute scaling methods, which are now integral to all production-ready models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dUyvihbrbs", "reviewer": "Reviewer_HT5J", "summary": "This article investigates the behaviors of scaling laws in the regime of over-training and on downstream benchmarks. Through derivation and initial observations, the article provide empirical evidence that over-training largely follows a similar, tractable, and constant-slope scaling law. Through further investigations and experiments, the article derives the parameters in the scaling law for over-training. With that conclusions, the article is then able to derive a scaling law relating model size, training tokens, and average downstream task performance, for the first time proving that average performance across a subset of downstream tasks are tractable.", "questions": "There is only one thing that WILL NOT affect my overall rating towards this article. \n\n1. See Weakness [1], it is better to discuss the differences between the proposed over-train scaling law and [1]. \n\nI am willing to defend my overall rating in the rebuttal period.\n\n[1] Temporal Scaling Law for Large Language Models", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Originality: This work firstly investigates and tracks next-token prediction model performance under the over-training regime and on downstream benchmark evaluation tasks.\n2. Strong experiment results: This work builds a test-bed with more than 100 models with different sizes and training tokens. Through predicting the scaling law, this work successfully predicts the performance of larger models trained with more tokens.\n3. Theoretical relationship with prior work: This work provides a stimulating analysis (Section 2) between scaling laws under the over-training regime and existing Chinchilla/Kaplan scaling laws, providing strong theoretical evidence supporting the derived over-training scaling law.\n4. Writing: This paper is well-written with a clear, logical presentation of motivation, derivation process, experiment result analysis, and conclusion.", "weaknesses": "1. This work suggests that \"there remain gaps between current scaling studies and how language models are ultimately trained and evaluated\" (Line 13-14), suggesting that over-trained regime is under-investigated. There is an interesting work [1] that investigates the pre-training process of language models. In [1], the authors predict the training outcomes using the gathered data from an initial training period, and they successfully predicts the training outcomes of relatively small models (<0.1B) trained on large number of tokens (400B tokens). In that case, $M>4000$ and falls into the over-training category. It would be interesting to investigate the difference between the proposed over-train scaling law and [1].\n\n[1] Temporal Scaling Law for Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)