Scaling Data-Constrained Language Models
Authors: Niklas Muennighoff
Alexander M Rush
Boaz Barak
Teven Le Scao
Nouamane Tazi
Aleksandra Piktus
Sampo Pyysalo
Thomas Wolf
Colin Raffel
Venue: NeurIPS 2023 oral (2023)
Keywords: large language models
scaling laws
data engineering
TL;DR: Scaling laws for training LLMs on multiple epochs & code filling + filtering experiments

Abstract: The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.

Scaling Data-Constrained Language Models
Niklas Muennighoff'
Alexander M. Rush
Boaz Barak
Teven Le Scao
Aleksandra Piktus
Nouamane Tazi 1
Sampo Pyysalo 3 Thomas Wolf
Colin Raffel
Huggiin Face
Harvard University
University of Turku
n. mww.migooof@gmml.ccmmmmm
Abstract
The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically,
a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available https:/ //g/ thub.u.... com hww.:hhgggnngngegeeeee com./mgginngaac///aaaaaaaaaaaaaa attpffacedaaaaaaai..oo
Cata-Constrained Scaling Laws
Return on compute when repeating
Allocating compute when repeating
3.2
Final test loss
Parameters
8.678
Loss: 2.376
2.359
Rapid diminishing
apochs.
good
repetitions
repeating worthiess
128
488
1208
4808
1.2T
1788
2428
(1)
(40)
(100)
19.7)
Tokens (Epochs)
Tokens (Epochs)
Models trained
Regime compute (IsoFLOP)
Loss assuming repeated data is worth the same as new deta
Efficient frontier essuming repeated data worth the same new data
predicted data- col-ccrttrained scaling
Efficient frontier prodicted data constrained scaling
Figure I: Return and Allocation when repeating data. (Left): Loss of LLMs (4.2B parameters)
scaled on repeated data decays predictably ($6). (Right): To maximize performance when repeating, our cont-coontr scaling laws and empirical data suggest training smaller models for more epochs in contrast to what assuming Chinchilla scaling laws [42] hold for repeated data would predict ($5).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Recent work on compute-optimal language models [42] shows that many previously trained large language models (LLMs, which we define as having more than one billion parameters) could have attained better performance for a given compute budget by training a smaller model on more data. Notably, the 70-billion parameter Chinchilla model [42] outperforms the 280-billion parameter Gopher model [89] while using a similar compute budget by being trained on four times more data. Extrapolating these laws for compute allocation (hereafter "Chinchilla scaling laws") to a 530 billion parameter model, such as the under-trained MT-NLG model [99], would require training on a massive 11 trillion tokens, corresponding to more than 30 terabytes of text data. For most languages, available data is several orders of magnitude smaller, meaning that LLMs in those languages are already data-constrained. Villalobos et al. [1 12] estimate that even high-qualia exhausted by the year 2024 given the Chinchilla scaling laws and the trend of training ever-larger models. This motivates the question [112, 81]: what should we do when we run out of data?
English language data will be
In this work we investigate scaling large language models in a constrannd regime, and whether training an LLM with multiple epochs of repeated data impacts scaling. Using multiple epochs is, of course, standard machine learning generally; however, most prior large language models have been trained for a single epoch [51, 15] and some work explicitly advocates against reusing data [40]. An exception is the recent Galactica models [108] that were trained for 4.25 epochs and exhibit continually decreasing validation loss and improving downstream performance throughout training. However, the experiments of Galactica do not compare this setup to an alternative non-data constrained model trained for one epoch unique data. Without this comparison. it is difficult to quantify the trade-off between additional compute versus additional data collection.
Our main focus is to quantify the impact of multiple epochs in LLM training such that practitioners can decide how allocate compute when scaling models. Toward this end, we assembled a battery of empirical training runs of varying data and compute constraints. Specifically, we train more than 400 models ranging from 10 million to 9 billion parameters for up to 1500 epochs and record final test loss. We use these results to fit new duta-construined scaling law that generalizes the Chinchilla scaling law [42] to the repeated data regime and yields a better prediction of loss in this setting. Figure I summarizes our main results targeting the value of repeated data (Return) and optimal allocation of resources in that regime (Allocation). We find that, while models trained for single epoch consistently have the best validation loss per compute, differences tend to be insignificant among models trained for up to 4 epochs and do not lead to differences in downstream task performance. Additional epochs continue be beneficial, but returns eventually diminish to zero. We find that, in the data-constrained regime, allocating new compute to both more parameters and epochs is necessary, and that epochs should be scaled slightly faster. These findings suggest simple way to continue scaling total training compute budgets further ahead in the future than the previously anticipated limits.
Finally, given the challenges imposed by data constraints, we consider methods complementary to repeating for improving downstream accuracy without adding new natural language data. Experiments consider incorporating code tokens and relaxing data filtering. For code, English L.Ms,, such as PaLM [19] or Gopher [89], are trained on small amount of code data alongside natural language data, though no benchmarking was reported to justify that decision. We investigate training LLMs on mix of language data and Python data 10 different mixing rates and find that mixing in code is able to provide a 2x increase in effective tokens even when evaluating only natural language tasks. For filtering, we revisit perplexity and deduplication filtering strategies on both noisy and clean datasets and find that data filtering is primarily effective for noisy datasets.
2 Background
Predicting the scaling behavior of large models critical when deciding on training resources.
Specifically, two questions are of interest: (Allocation) What is the optimal balance of resources? (Retrn) What the expected value of additional resources? For scaling I.LMs, the resource is compute (measured in FLOPs), and it can be allocated to training a larger model or training for more

---

steps." The metric used to quantify progress is the model's loss on held-out data, i.c. the ability to predict the underlying data as measured the model's cross-entropy [2, 42]. We aim to minimize the loss (T.) subject to compute resource constraint (C) via optimal allocation to N and D as:
argmin ((N, s.t. FLOPs(N, D) C N,D
(1)
Currently, there are established best practices for scaling LLMs. Return follows a power-law: loss
scales as a power-law with the amount of compute used for training [39, 46, 6. 35, 7. 41]. Allocation is balanced: resources are divided roughly equally between scaling of parameters and data [42]. These scaling laws were established empirically by training I.L.S and carefully extrapolating behavior.
Chinchilla [42] uses three methods for making scaling predictions:
(Fixed Parameters) Train with fixed model size but on varying amounts of data.
(Fixed FLOPss) Train with fixed computation while parameters and training tokens vary.
(Parametric Fif) Derive and fit formula for the loss.
For the parametric fit, the loss (L) is a function of parameters (N) and training tokens (D):
B Na D8
L(N,D)
+
(2)
Where {4, a, B. }} are learned variables using the training runs from the first two ap- proaches [42]. Using these learned variables, they propose calculating the optimal allocation of compute (C) to N and D as follows:
NC) = G(C/6)"
Dopr(O)= = G-
(AA
B
a
where
G=
a a + 3
b
+ 3
These methods lead the conclusion that a Ri 3 and hence N and D should be scaled proportionally for compute-optimal training. As loss can be an imperfect proxy for performance on natural language tasks [123, 97, 105], they also validate their conclusions on various downstream tasks.
Method: Pata-Constrained Scaling Laws
We interested in scaling behavior in the data-constrained regime. Specifically, given limited amount of unique data, what is the best Allocation of and Return for computational resources. Prior work [46, 42] assumes that the necessary data support scaling is unlimited. Our aim is therefore to introduce a modified version of Equation 2 that accounts for data constraints and fit the terms in the modified scaling law data from large body of experiments.
The primary method we consider is repeating data, i.c. allocating FLOPs to multiple epochs on the same data. Given budget of unique data Dc, split the Chinchilla total data term D into two parts: the number unique tokens used, Up. and the number of repetitions, Rp (i.c. epochs 1). Given total training tokens D and data budget De these terms are simply computed as UD
minn&De D} and RD= = (D/UD) - 1. When training for a single epoch like done in prior scaling studies, RD We thus interested in minimizing Equation with the additional constraint of data budget De:
argmin I(N,D) S.E. FLOPS D) = C,UD S De N,D
Symmetrically, for mathematical convenience, split the parameter term N into two parts: the base number of parameters needed to optimally the unique tokens UN, and the number of times
In this work we use [46]'s approximation for the compute cost: FLOPs(N, D) z 6.V D, where N denotes the number of model parameters and D denotes the number of tokens processed.

---

to "repeat" this initial allocation, Ry. We compute UN by first rearranging Equation 3 to find the optimal compute budget for the unique tokens used (U)). We input this value into the Nopt formula of Equation 3 to get UN min { N Nopt: N.. Un thus corresponds to the compute-optina number of parameters for Up or less if N Nopt. Once we have UN. we compute the repeat value as Ry (N/Ux) 1.
To empirically explore the scaling behavior in data-limited setting we train LLMs under these constraints. We consider three different experimental protocols in this work:
(Fixed Unique Data) In $5 we fix the data constraint De and train models varying epochs and parameters. These experiments target Allocation, specifically tradeoff of D and N.
(Fixed FLOPs) In $6 we fix the computation available and vary Dc (and thus also UD and UN). These experiments target Return, i.c. how well does repeating scale compared to having more unique data.
(Parametric Fit) We fit formula introduced in $3.1 all our training runs and evaluate its predictive capability throughout $5 and $6.
Before discussing experimental results we describe the parametric assumptions.
3.1
Parametric Fit
To extrapolate scaling curves, it is necessary to incorporate repetition into the Chinchilla formula (Equation 2). We generalize Equation 2 by replacing D and N with terms corresponding to the effective data (D') and effective model parameters (N').
A L(N = Nia
B DIS
+
Intuitively, D' should be smaller equal D where D is the total number of processed tokens since repeated tokens provide less useful information to the model than new ones. We use an exponential decay formulation, where the value of data token processed loses roughly its value per repetition, where R2 is a learned constant. After some derivations and approximations (see Appendix A), this boils down
1/Rb) fraction of
-Ap D' Un + Un "
(5)
Note that for RD 0 (no repetitions), D' UD For RD < RD. e- Ro/Rz =
R2 and so R
UD UDR' 1+ Rp/Rt)==(>>-- = Up(1 = RD)=
and hence this case, repeated data is worth almost the same as fresh data. (This is also consistent with the predictions the "deep bootstrap" framework [76].) As Rp grows, the value of repeated tokens tends to zero, and the effective data D' becomes much smaller than D. The formula implies that matter how many times we repeat the data, we will not get better loss than could be obtained with single epoch on UD + UDR fresh tokens.
Just processing repeated tokens yields diminishing return, both intuitively and empirically, models with sizes that vastly outstrip the available data also offer diminishing returns per parameter. Hence we use symmetric formula for the number of effective parameters, where again Ry is learned,
By RW
N' = UN UN
The learned constants RD Ry roughly correspond the "half-life" of repeated data and excess parameters. For example, at RD = R the number of effective tokens D' is Un + Rp - e-1) which means that the Up Rp repeated tokens worth on average 1 1/e fraction of fresh ones.
Using methodology similar to [42], Ry and Ri can be fit on empirical measurements, which yields data-driven estimates. See Appendix A for more details the derivations and the fitting procedure.

---

Empirical IsoLoss Contours
Predicted IsoLoss Contours
7096M
4.08
4.65
2129M-
709H
Parameters
212M 146M
3.72
Loss
70M
3.98
4.18
7M
Loss: 8.10
10
30 Epochs
59
100
300
1000
10
30 Epochs
59.100
300
1000
3.95
Compute-ogtima
Chinchilla efficient frontier
Models trained
Lowest loss for 100M tokens
Data- conttroodt scaling laws efficient frontier
Figure 3: IsoLoss contours for 100 million unique tokens. (Left): 93 models trained with varying parameters and epochs on a fixed dataset. Contours show an interpolation of results with the same final test loss. (Right): Comparison with the loss predictions from proposed scaling laws for the same budget of 100 million unique tokens and the predicted efficient frontier. The diminishing returns from training repeated data can be seen in the increase in distance of the contour curves.
Experimental Setup
EPACH
For all experiments, train transformer language models with the GPT-2 architecture and tokenizer [88]. Mod- els have up 8.7 billion parameters and are trained for up to 900 billion total tokens. Following [42] we use cosine learning rate schedules that decay 10x over the course of training for each model (different schedules led different estimates in [46]). Unlike [46], do not use early stopping to also explore the extent of overfitting when repeating. Other hyperparameters
based prior work [89, 42] and detailed in Appendix S. Models are trained on subsets of C4 [90]. The data constraints are carefully defined to ensure maximal overlap as shown in Figure 2. Unlike [40], always repeat the entire avail- able data rather than subsets of it. Data is shuffled after each epoch. As repeating data can result extreme overfitting (see Appendix H), we report loss on a held-out test set unless otherwise specified (see Appendix K). This contrasts training loss used [42], but should not alter our findings as held-out data stems from the same underlying dataset.
Figure 2: Dataset setup. We ensure that runs using less data (more epochs) al- ways
subset of the data used in runs with more data (fewer epochs).
Results: Resource Allocation for Data-Constrained Scaling
Our first experimental setting considers scaling in setting where all models have the same data constraint. For these experiments, the unique training data budget De is fixed at either 100M, 400M 1.5B tokens. For each data budget, train set of language models with increasing amounts of compute that is allocated to either more parameters or more epochs on the unique training data.
Figure 3 (left) shows the main results for scaling with 100M unique tokens² (see Appendix C for 400M and 1.5B tokens). For 100M tokens, the corresponding one-epoch compute-optimal model
Although small, for example, this is the order of magnitude of realistic data constraint reflecting data available after filtering the OSCAR dataset [84] for Basque, Punjabi, or Slovenian.
5

---

2.3B paramaters trained for 558 tokens
4.28 paramaters trained for 84B tokens
8.7B paramaters trained for 1788 tokens
3.4
Valication lass
2.8
2.4
58
158
258
358
458
555
58
258
458
658
858
58
408
1008
1408
1808
Training tokens
Training tokens
Training tokens
Epochs
FLOP budget (C)
Parameters (N)
Training tokens (D)
Data budget (Dc)
9.3 x 1020
2.8B
55B { 55, 28, 18, 14, 11, 9, 4, B.255B 84, 42, 28, 21, 17, 12, 1.9)
2.1 x 1021
4.2B
84B
9.3 X 1021
8.7B
178B
{178, 88, 58, 44, 35, 25, 13, 4)B
Figure 4: Validation Loss for Different Data Constraints (IsoFLOP). Each curve represents the same number of FLOPs spent equal size model. Colors represent different numbers of epochs due to repeating because data constraints. Parameters and training tokens are set to match the single-epoch compute-optimgs configurations for the given FLOPs. Models trained on data that is repeated for multiple epochs have consistently worse loss and diverge if too many epochs are used.
according to scaling laws from [42] has UN of approximately 7M parameters (see Appendix B for the scaling coefficients we use). Results show that more than 50% reduction in loss can be attained by training for several epochs (Rp > 0) and increasing model size beyond what would be compute-opine optpiall for 100M tokens (Rx > 0). We find the best loss to be at around 20-60x more parameters and epochs, which corresponds to spending around 7000 more FLOPs. These results suggest that one-epoch models significantly under-utilize their training data and more signal can be extracted by repeating data and adding parameters at the cost of sub-optimal compute utilization.
Figure 3 (right) shows the predicted contours created by fitting our cata-constrained scaling laws on 182 training runs. In the single-epoch case (RD = 0) with near compute-optimal parameters (Ry 0) our scaling equation ($3.1) reduces to the Chinchilla equation. In this case, both formulas predict the optimal allocation of compute parameters and data to be the same, resulting in overlapping efficient frontiers. As data is repeated for more than single epoch, our fit predicts that excess parameters decay faster in value than repeated data (Ry < RD)). As a result, the data- constrained efficient frontier suggests allocating most additional compute to more epochs rather than more parameters. This contrasts the Chinchilla scaling laws [42], which suggest equally scaling both. However, note that they do not repeat the entire training data and their parametric fit explicitly relies on the assumption that models are trained for a single epoch only. Thus, there is no guarantee that their scaling predictions hold for repeated data.
For all three data budgets, our results suggest that Allocation optimized by scaling epochs faster than additional parameters. We confirm this at scale by training the data contt-cnsttaieed compute-optimal model for 9.3 10 FLOPs and 25 billion unique tokens as suggested by our efficient frontier. Despite having 27% less parameters, this model achieves better loss than the model suggested by the Chinchilla scaling laws (Figure 1, right). Similarly, the 120 billion parameter Galactica model trained on repeated data should have been significantly smaller according to data-constrained scaling laws (Appendix G). An additional benefit of using a smaller model is cheaper inference, though adding parameters make it casier parallelize training across GPUs.

---

Empirica Loss (Fixed training lnggth)
Predicted Loss (Varlable training engthh)
2.8
2.7
trect
Final test
2.08 parameters trained for 358 tokens
Repeating 4 epochs almost as new data
2.5
cc
2.88 parameters
4.28 parameters 84B
2.4
2.3
8.78 parameters trained for 1788 tokens
8.70 parameters
2.2
100%
50%
25%
14%
108
1008
17
10T
100T
Fraction training tokens that unique
Total training tokens
of trained
Loss assuming reprated is the
data Loss predicted by our data-constrained scaling laws
Loss assuming training is stopped when exhausting all unique data
Figure 5: Empirical and Extrapolated loss with constrained data. (Left): Loss function of repeated tokens for three different training budgets each with fixed number of parameters. Loss curves predicted by our canttroontt scaling laws shifted to exactly match the loss at 100% unique data. Return on FL.OPs decays with repeated data in a regular pattern. (Right): Extrapolating from the proposed data-constrained scaling law shows that at small numbers epochs are benign, but at large number of epochs loss stops improving.
Adding parameters and epochs causes the loss to decrease and eventually increase again, suggesting that too much compute can hurt performance. Results from [46] also show that loss can increase when too many parameters are used, even with early stopping. However, we expect that appropriate extreme case) could prevent this
regularization (such as simply removing all excess parameters behavior. Thus, our formula presented in $3 and its predicted isoLoss contours in Figure do not model the possibility that excess epochs parameters could hurt performance.
6 Results: Resource Return for Data-Constrained Scaling
Next, consider the question of Return on scaling. To quantify this value, we run experiments with three FLOP budgets across eight respective data budgets to compare return on FLOPs.
Figure shows the configurations and validation curves for models trained on the same number of total tokens. Conforming to intuition and prior work on deduplication [55], repeated data is worth less, thus models trained less unique data (and, correspondingly, more epochs) have consistently higher loss. However, the loss difference for a few epochs is negligible. For example, the N = 8.7 billion parameter model trained for four epochs (Dc 44 billion unique tokens) finishes training with only 0.5% higher validation loss than the single-epoch model (De 178 billion unique tokens).
In Figure 5 (left), compare the final test loss of each model to predictions from our parametric fit. The data-constrained scaling laws can accurately measure the decay in the value of repeated data as seen by the proximity of empirical results (dots) and parametric fit (lines). We note however that significantly underestimates the final test loss of failing models where loss increases midway through training. such as models trained for 44 epochs (not depicted).
In Figure 5 (right), we extrapolate the three budgets by further scaling compute while keeping the data constraints (De) at 55B, 84B, and 178B tokens, respectively. The parameter R;, introduced in $3 represents roughly the "half-life" of epochs: specifically the point where repeated tokens have lost of their value. Through our fitting in Appendix A, we found R 15, corresponding to 15 repetitions (or 16 epochs). Graphically, this can be seen by the stark diminishing returns in the proximity of the 16-epoch marker and the flattening out soon after.
Overall, the Return when repeating data is relatively good. Meaningful gains from repeating data can be made up around 16 epochs (Ri)) beyond which returns diminish extremely fast.
7

---

Repeal
Reper
DATA SUDGRT
22
Filin
20-
18
Strategy
Deduplictta Parpieittydiiee
age
Repeating data
Fillng missing data Python Perplexity-fiee fliter then repest
DATA BUDGET
Deduplicte then repeat
100%
10%
Papest
Data Burget
Figure 6: Strategies for data-constrained settings and their downstream performance. (Left): Schematic showing alternative data use strategies of code filling and filtering. (Right): N = 4.2 billion parameter models trained for total of D 84 billion tokens with varying budgets De. For repeating and filling with code, five models with different seeds are trained for each dot and the standard deviation is visualized as the shaded area.
7 Results: Complementary Strategies for Obtaining Additional Data
While repeating data is effective, it has diminishing returns. We therefore consider strategies for scaling D targeting improved downstream performance as opposed to directly minimizing loss.
Figure 6 (left) illustrates the strategies: (a) Code augmentation: We use Python code from The Stack [49] to make up for missing natural language data. The combined dataset consisting of code and natural language samples is shuffled randomly.
(b) Adapting filtering: We investigate the performance impact of deduplication and perplexity filtering, two common filtering steps that can severely limit available data. Removing such filtering steps can free up additional training data.
For these experiments, we set maximum data budget (Dc) of 84 billion tokens. For repetition and code filling, only a subset of Dc is available and the rest needs to be compensated for via repeating adding code. For both filtering methods, billion tokens), as is easier to gather noisy data and filter than it is to gather clean data for training. select the top 25% samples with the lowest perplexity according
start out with approximately twice the budget (178
For perplexity filtering, language model trained on Wikipedia. This results in 44 billion tokens that are repeated for close to two epochs reach the full data budget. For deduplication filtering, all samples with 100-char overlap are removed resulting in 21 billion tokens that are repeated for four epochs during training. See Appendix N for more details the filtering procedures.
When comparing across data strategies, loss ceases to good evaluation metric as the models are trained different data distributions. We thus evaluate models on 19 natural language tasks with zero to five in-context few-shot exemplars [15] producing I 14 scores per model. As our evaluation tasks cover different metrics and random basclines, re-scale all scores to be in the same range to better reflect performance ranges before averaging. Details on the evaluation datasets are in Appendix K.
In Figure (right) we compare the downstream performance of all strategies. For repeating data, differences downstream performance are insignificant for up to around epochs (25% budget) and then start dropping, which aligns with our results on test loss in $6. Filling up to 50% of data with code (42 billion tokens) also shows no deterioration. Beyond that, performance decreases quickly on natural language tasks. However, adding more code data may benefit non-natural language tasks, which are not considered the benchmarking. Two of the tasks benchmarked, WebNLG [17,3] 34], a generation task, and bAbl [122, 57], a reasoning task, see jumps in performance soon as code is added, possibly due to code enabling models to learn long-range state-tracking capabilities beneficial for these tasks.
Of the filtering approaches, we find copllexty-filtering to be effective, while deduplication does not help. Prior work found deduplication was able to improve perplexity [55]; however, it did not evaluate on downstream tasks. Deduplication may have value not captured our benchmark, such as reducing memorization [45, 40, 16, 10]. We also investigate filtering different noisier dataset in Appendix O, where we find it to be more effective. Overall, data-constrained regime, we recommend reserving filtering for noisy datasets and using both code augmentation and repeating to
8

---

increase data tokens. For example, first doubling the available data by adding code and then repeating the new dataset for four epochs results in 8 more training tokens that are expected be just as good as having had 8x more unique data from the start.
8 Related Work
Large language models Scaling up transformer language models [111] across parameter count and training data has been shown to result in continuous performance gains [19]. Starting with the 1,4 billion parameter GPT-2 model [88], a variety of scaled-up language models have been trained, commonly referred to as large language models (LLMs). They can be grouped into dense models [15, 47, 58, 89, 20, 13, 132, 109, 103, 108, 130, 95, 56, 65] and sparse models [30, 131, 28, 135] depending on whether each forward pass makes use of all parameters. These models are generally pre- pre-trained to predict the next token in sequence, which makes them applicable to various language tasks directly after pre-training [15, 118, 50, 71, 102] by reformulating said NLP tasks as context continuation tasks (see [67] for carlier proposal on this topic). We focus on the most common scenario, where a dense transformer model is trained to do next-token prediction on large corpus and evaluated directly after pre-training using held-out loss or zero- to few-shot prompting.
Scaling laws Prior work has estimated an optimal allocation of compute for the training of LLMs. Kaplan et al. [46] suggested a 10x increase in compute should be allocated to a 5.5x increase in model size and 1.8x increase in training tokens. This first scaling law has led the creation of very large models trained on relatively little data, such as the 530 billion parameter MT-NLG model trained on 270 billion tokens [99]. More recent work [42], however, showed that model size and training data should rather be scaled in equal proportions. These findings called for a renewed focus on the scaling of pre-training data rather than scaling model size via complex parallclization strategies [98, 91, 9, 78). Up-sampling is often employed when pre-training data is partly limited, such as data from a high-quality domain like Wikipedia or text a rare language for training multilingual LLMs [60, 82]. Hernandez et al. [40] study up-sampling of data subsets and find that repeating only 0.1 0.1% of training data 100 times significantly degrades performance. In contrast, our work focuses on repeating the entire pre-training corpus for multiple epochs rather than up-sampling parts of it.
Alternative data strategies Large pre- pre-training datasets are commonly filtered to remove undesired samples or reduce noise [101]. Perplexity- -based filtering, whereby a trained model is used to filter out samples with high perplexity, has been found beneficial to reduce noise web-crawled datasets [121]. Mixing of data is employed for the pre-training data of multilingual LLMs, where text data from different languages combined [23, 126, 100, 74]. However, both for code and natural language models, mixing different (programming) languages has been reported to under-perform monolingual models [80, 113]. Some work has investigated mixing code and natural language data for prediction tasks, such summarizing code snippets [44] predicting function names [4]. Several pre-training datasets for LLMs include low amounts of code data [31, 89, 95]. However, these past works generally do not provide any ablation on the drawbacks of including code or the benefits for natural language task performance. We perform a detailed benchmarking of mixing Python and natural language in LLM pre- pre-training at 10 different mixing rates.
Conclusion
This work studies data-constrained scaling, focusing on the optimal use of computational resources when unique data is limited. We propose an extension to the Chinchilla scaling laws that takes into account the decay in value of repeated data, and we fit this function using a large set of controlled experiments. We find that despite recommendations of earlier work, training large language models for multiple epochs by repeating data is beneficial and that scaling laws continue to hold in the multi-epoch regime, albeit with diminishing returns. We also consider complementary approaches to continue scaling models, and find that code gives the ability to scale an additional 2x. We believe that our findings will enable further scaling of language models to unlock new capabilities with current data. However, our work also indicates that there are limits on the scaling horizon. In addition to collecting additional data, researchers should explore using current data in more effective manner.
9

---

Acknowledgments and Disclosure of Funding
This work co-funded by the European Union under grant agreement No 101070350. The authors wish to acknowledge -S Center for Science, Finland, for generous computational resources on the LUMI supercomputer:" We are thankful for the immense support from teams LUMI and AMD, especially Samuel Antao. Hugging Face provided storage and additional compute instances. This work was supported by a Simons Investigator Fellowship, NSF grant DMS RMS-21341 2134157, DARPA grant W91 INF22000002,, and DOE grant NEE-SC002219 We are grateful to Harm de Vries, Woojeong Kim, Mengzhou Xia and the EleutherAl community for exceptional feedback. We thank Loubna Ben Allal for help with the Python data and Big Code members for insightful discussions on scaling laws. We thank Thomas Wang, Helen Ngo and TurkuNI.P members for support on early experiments.
References
[1] Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer, 2023, Scaling Laws for Generative Mixed-Modal Language Models. arXiv preprint arXiv:2301.03728
[2] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. 2022. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-22312.
[3] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Car- Ios Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. SantaCoder: don't 't reach for the stars! arXiv preprint arXi::2301.03988
[4] Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th joint meeting on foundations of software engineering, pages 38-49.
[5] Stephen H. Bach, Victor Sanh, Zheng- Xin Yong, Albert Webson, Colin Raffel, Nihal Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Xiangru Tang, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts.
[6] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. 2021. Explain- ing neural scaling laws. arXiv preprint arXiv:2102.00 067.07011
[7] Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. 2022. Data scaling laws in NMT: The effect of noise and architecture. In International Conference Machine Learning, pages 1466-1482. PMLR.
[8] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610-623.
[9] Zhengda Bian, Hongxin Liu, Boxiang Wang, Haichen Huang, Yongbin Li, Chuanrui Wang, Fan Cui, and Yang You. 2021. Colossal-AI: A unified deep learning system for large-scale parallel training. arXiv preprint arXiv:2110.14883
[10] Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. 2023. Emergent and Predictable Memorization in Large Language Models. arXiv preprint 2rXiv:2304.11155
[11] Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint arXiv:2304.01373.
hhttp::/www/ wwmilummuuuu cumms-suprrcmputee.com eu/
10

---

[12] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Rea- soning about Physical Commonsense in Natural Language. In Thirty-Fourth Fourth AAAI Conference Artificial Intelligence.
[13] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, al. 2022. GPT-NeoX-20B: An Open-Source Autoregressive Language Model. arXiv preprint arXiv:2204.06745.
[14] Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large scale autoregressive language modeling with mesh-tensorflow. you this software, please cite using these metadata, 58.
[15] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901.
[16] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint wrXiv:2202.06646.
[17] Thiago Castro Ferreira, Claire Gardent, Nikolai linykh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. 2020, The 2020 Bilingual, Bi-Directional WcbNLG+ Shared Task Overview and Evaluation Results (WebNLG+ 2020). In Proceedings of the 3rd WebNLG Workshop on Natural Language Generation from the Semantic Web (WebNLG+ 2020), pages 55-76, Dublin, Ireland (Virtual). Association for Computational Linguistics.
[18] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. 2020. Generative pretraining from pixels. In International conference on machine learming, pages 1691-1703. PMLR.
[19] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm:
Scaling language modeling with pathways. arXiv preprint arXiv:2204.0231/
[20] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling Instruction instruction-Finntuned Language Models. arXiv preprint arXiv:2210.1144
[21] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In NAACI.
[22] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge.
[23] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoy- anov. 2019. Unsupervised Cross-lingual Representation Learning at Scale. arXiv preprint arXiv:1911.02116.
[24] Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 1-13, 2005, Revised Selected Papers, pages 177-190. Springer.
[25] Marie-Cathering De Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitment- bank: Investigating projection naturally occurring discourse. proceedings of Sinn und Bedeutung, volume 23, pages 107-124.

---

[26] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek. Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et 2023. Scaling vision transformers to 22 billion parameters. arXiv preprini arXiv22302005442.
[27] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals. Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal transformers. arXiv preprint arXi::1807.038/3
[28] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. 2022. Glam: Efficient scaling of language models with -ffturreof--xperts. In International Conference on Machine Learning, pages 5547-5569. PMLR.
[29] Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge. Computer Speech & Language, 196.16656
[30] William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res, 23:1-40.
[31] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint arXiv:2101.00027.
[32] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation.
[33] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. Realtoxieityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint
arXiv:2009.11 1462.
[34] Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. 2021. The gem benchmark: Natural language generation, its evaluation and metrics. arXiv preprint arXiv:2102.01672
[35] Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. 2021. Scaling laws for neural machine translation. arXiv preprint arXh::2109.077400
[36] Himanshu Gupta, Saurabh Arjun Sawant, Swaroop Mishra, Mutsumi Nakamura, Arindam Mitra, Santosh Mashetty, and Chitta Baral. 2023. Instruction Tuned Models are Quick Learners. arXiv preprint arXiv:2306.05539
[37] Kenneth Heafield. 2011. KenLM: Faster and Smaller Language Model Queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland. Association for Computational Linguistics.
[38] Peter Henderson, Mark Krass, Lucia Zheng, Neel Guha, Christopher D Manning, Dan Jurafsky, and Daniel Ho. 2022. Pile of law: Learning responsible data filtering from the law and a 256gb open-source legal dataset. Advances in Neural Information Processing Systems, 35:29217- 29234.
[39] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020. Scaling laws for autoregressive generative modeling. arXiv preprint arXiw:2010.14701.
[40] Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, 7ac Hatfield-Dodds, Tom Henighan, Tristan Hume, et 2022. Scaling Laws and Interpretability of Learning from Repeated Data. arXiv preprint arXiv:2005.10487.
12

---

[41] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. arXiv preprint arXiv:2102.01295
[42] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, et al. 2022. Training Compute-Optina Large Language Models. arXiv preprint arXiv:2203. 255.5
[43] Cheng-Zhi Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, lan Simon, Curtis Hawthorne, Andrew M Dai, Matthew D Hoffman,, Monica Dinculescu, and Douglas Eck. 2018. Music transformer. arXiv preprinl arXiv: 1889.042281
[44] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung. and Luke Zettlemoyer. 2016. Summarizing source code using neural attention model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2073-2083.
[45] Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating Training Data Mitigates Privacy Risks in Language Models.
[46] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models, arXiv preprint arXiv:2001.08361.
[47] Mikhail Khrushchev, Ruslan Vasilev, Alexey Petrov, and Nikolay Zinov. 2022. YaLM 100B.
[48] Diederik P Kingma and Jimmy Ba. 2014, Adam: method for stochastic optimization. arXiv preprint arXiv: 12166690.
[49] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022. The Stack: 3 TB of permissively licensed source code. arXiv preprint arXiv:2211.
[50] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11911
[51] Aran Komatsuzaki. 2019. One epoch is all you need. arXiv preprint 1eXivv1906.0660
[52] Taku Kudo and John Richardson. 2018. SentencePiece: subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
A simple and language independent
[53] Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. gua: A new benchmark dataset for cross-lingual abstractive summarization. arXiv preprint arXiv:2010.03093
WikiLin-
[54] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[55] Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Kellison-Burch and Nicholas Carlini. 2021. Deduplicating Training Data Makes Language Models Better. arXiv preprint arXiv:2 064999999.......
[56] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav
13

---

Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. 2023. StarCoder: may the source be with you! arXiv preprint arXiv:2305.0611
[57] Perey Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang. Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:22 1.09110.
[58] Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021. Jurassic-1; Technical details and evaluation. White Paper. A12/ Labs, 1.
[59] Chin- Yew Lin. 2004, Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
[60] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.1066
[61] Yu-Hsiang Lin, Chian Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, et al. 2019. Choosing transfer languages for cross-lingual learning. arXiv preprint arXiv: 12688.
[62] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. 2023. The Flan Collection: Designing Data and Methods for Effective Instruction Tuning. arXiv preprint arXiv:2301.1368..
[63] Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, Deb Roy, and Sara Hooker. 2023. The Data Provenance Initiative: Large Scale Audit of Dataset Licensing & Altribution in AI.
[64] Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. 2023. A Pretrainer's 's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity.
[65] Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna- Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, Thomas Wang, Nouamane Tazi, Teven Le Scao, Thomas Wolf, Osma Suominen, Samuli Sairanen, Mikko Merioksa, Jyrki Heinonen, Aija Vahtola, Samuel Antao, and Sampo Pyysalo. 2023. FinGPT: Large Generative Models for a Small Language. In The 2023 Conference on Empirical Methods in Natural Language Processing.
[66] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po-Ssu Huang, and Richard Socher. 2020. Progen: Language modeling for protein generation. arXiv preprint arXiv:2004.33997.
[67] Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The Natural Language Decathlon: Multitask Learning as Question Answering. CoRR, abs/1 hts/1886.08730.
[68] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannanch Hajishirzi. 2021. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15944
[69] Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51.
[70] Niklas Muennighoff. 2020. Vilio: State-of-the-art visio-linguistic models applied to hateful memes. arXiv preprint arX.v.2012.07788
14

---

[71] Niklas Muennighoff.. 2022. SGPT: GPT Sentence Embeddings for Semantic Search. arXiv preprint arXiv:2202.08004
[72] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023. OctoPack: Instruction Tuning Code Large Language Models. arXiv preprint arXiv:2308.07124.
[73] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. MTEB; Massive Text Embedding Benchmark. arXiv preprint arXlv:2210.07316.
[74] Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xo Yong. Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.0172
[75] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever.
2021. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021
[76] Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. 2021. The Deep Bootstrap Frame- work: Good Online Learners are Good Offline Generalizers. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
[77] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't 't Give Me the Details, Just the Summary! Topic-Awer Aware Convolutional Neural Networks for Extreme Summarization, In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
[78] Deepak Narayanan, Mohammad Shoeybi. Jared Casper. Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernaue, Bryan Catanzaro, et al. 2021. Efficient large-scale language model training on gpu clusters using megatron-Im, In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-15.
[79] Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLE A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
[80] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474.
[81]
nostalgebraist. 2022. chinchilla's 's wild implications. lesswrong.
[82] Gabriel Orlanski, Kefan Xiao, Xavier Garcia, Jeffrey Hui, Joshua Howland, Jonathan Mal- maud, Jacob Austin, Rishah Singh, and Michele Catasta. 2023. Measuring The Impact Of Programming Language Distribution. arXiv preprint arXiv:2302.01993.
[83] Pedro Javier Ortiz Su' arez, Laurent Romary, and Benoit Sagot. 2020. A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703-1714, Online. Association for Computational Linguistics.
[84] Pedro Javier Ortiz Su' arez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipelines for processing huge corpora on medium low resource infrastructures. Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7) 2019. Cardiff, 22nd July 2019, pages 9 16, Mannheim. SibbbizzInstitue f'ur Deutsche Sprache
[85] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022 02155......
15

---

[86] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cap- pelli, Hamza Alobeidli, Baptiste Pannier. Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv preprint arXiv:2306.01 16.
[87] Shrimai Prabhumoye, Mostofa Patwary. Mohammad Shoeybi, and Bryan Catanzaro. 2023. Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models. arXiv preprini urXiv:2302.07.36
[88] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9.
[89] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song. John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444
[90] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67.
[91] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Duta Mining, pages 3505-3506.
[92] Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pages 90-95.
[93] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-
106.
[94] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. In The Tenth International Conference on Learning Representations.
[95] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Iliè, Daniel Hesslow, Ro- man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.0510
[96] Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. 2022. What Language Model to Train if You Have One Million GPU Hours? arXiv preprins arXiv:2210.1544
[97] Seongjin Shin, Sang- Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, et al. 2022. On the effect of pretraining corpora on in-context learning by a large-scale language model. arXiv preprint 2XXv::2044.13009
[98] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-Im: Training multi-bllion parameter language models using model parallelism. arXiv preprint 19Xiv:199.000000
[99] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deepspeed and megatron to train megatron-turing nig 530b, a large-scale generative language model. arXiv preprint a2X1 11990.
[100] Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna Rumshisky, et al. 2022. Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model. arXiv preprint arXiv:2208.00 0144..
16

---

[101] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. 2022. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206. 14486.
[102] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonse et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 2tti:220060046..5
[103] Hui Su, Xiao Zhou, Houjing Yu, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. 2022. WeLM:
A Well-Read Pre-trained Language Model for Chinese. arXiv preprint arXiv:2209.100 10372.
[104] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. 2022. Scaling Laws Model Architectures: How does Inductive Bias Influence Scaling? arXiv preprint arXiv:2207.10551.
[105] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. 2021. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109, 10686.
[106] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying Language Learning Paradigms. arXiv preprint 2rXiv:2205.05131.
[107] Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Gareia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, al. 2022. Transcending scaling laws with 0.1 % extra compute. arXiv preprint 0rXiv:2210.11323
[108] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.
[109] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprini (tti::22200002:2::3
[110] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. 2023. Llama: Open and efficient foundation language models. arXiv preprint 2302 13971..
[111] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
[112] Pablo Villalobos. Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho. 2022. Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning. arXiv preprint arXiv:2211.04325
[113] Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski. Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv: 1912.0776.
[114] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for Gerpose--urppsee Language Understanding Systems. CoRR, htbs/1995000537
[115) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MalMillan, Noah A Smith, Iz Beltagy, et al. 2023. How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources. arXiv preprint arXiv:2306.0477
17

---

[116] Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv preprini arXiv:2212. 10560.
[117] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652
[118] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903
[119] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:21 12.04359.
[120] Johannes Welbl, Nelson F Liu, and Matt Gardner. 2017. Crowdsourcing multiple choice science questions. arXiv preprint 1XX::1707.00209
[121] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Comneau. Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2019. CCNet: Extracting high quality monolin- gual datasets from web crawl data. arXiv preprint arXiv: 1.00359.
[122] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv: 150.:5500.0055
[123] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Ves Stoyanov. 2022. Training Trajectories of Language Models Across Scales. arXiv preprint arXiv:2212.09803.
[124] Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, and Ahmed Hassan Awadallah. 2021. MetaXL: Meta representation transformation for low- resource cross-lingual learning, arXiv preprint arXiv:2104.07908.
[125] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint Att.:2300.112444
[126] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010. 1194..
[127] Ge Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2021. Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems, 34: 34:17084-17097.
[128] Zheng-Xin Yong. Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, et al. 2022. BLOOM+ 1: Adding Language Support to BLOOM for Zero-Shot Prompting. arXiv preprint 222.v:2212.09535.
[129] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties.
[130] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, al. 2022. GLM-130B: An Open Bilingual Pre-trained Model. arXiv preprint arXiv:22
[131] Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. PanGu-alphaa: Large-scale Autoregres- sive Pretrained Chinese Language Models with Auto-parallel Computation. arXiv preprint arXiv:2104.12369
18

---

[132] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.
[133] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint 2rXiv:2303.18223.
[134] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. urXiv:2305.11206
Lima: Less is more for alignment. arXiv preprint
[135] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906.
19


Reviews:
[{"review_id": "wQfqHRJNPs", "reviewer": "Reviewer_sNww", "summary": "This paper studies the scaling behavior of language models by repeating the training data to multiple epochs. The authors extend the recent successful Chinchilla scaling law to a data-constrained regime by adding exponential decay of data and model terms then fit the empirical observations. The key takeaway is that repeating data up to 4 times yields negligible changes to loss compared to having unique data, consistent with the observation of Galactica. The authors further studies additional methods like adding code data to 50% and find out this does not compromise model performance on text tasks. ", "questions": "The authors reported that WebNLG and BaBI immediately improveSee about discussions as one mix in code data. I am curious about what kind of synergy between text and code data and what are the families of text tasks that would benefit the most from mixing code? ", "limitations": "See above discussion about weakness", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper studies a very important problem (what should one do to further scale up data when data is used up) and offers very practical advice (repeating the data 4 times and using code 50% to the training data). I believe this paper will make a clear impact in the area of language models and general AI. ", "weaknesses": "Although this paper has clear advantages in terms of practical guidance, I am concerned about the following weakness: \n\nThe choice of the specific parametric form in equation 5 and 6 are not clearly explained and may be subject to debate, specifically:\n\n- Why the decay in equation 5 should be exponential (other than, say, polynomial) as when data points are small, exponential may look similar to polynomial (or other functional forms).\n- Why the decay in model scale (Eq. 6) should follow the same form as data (Eq. 5). For the original Chinchilla scaling law, the two terms taking symmetric forms are understandable due to their nature. But for repeating data, model parameter and data scale are not as symmetric as unique data setting, thus considering the same form of decay as data (though I do believe there should be a certain form of decay in model scale) might be less justified\n\nDifferent types of data may need different levels of repetition. \n\n- The Chinchilla model mostly considers webpages v.s. the Galactica model mostly consider academic papers. Intuitively, the level of complexity of webpages might be lower than papers, and one may want to repeat the complex data over simple ones.\n\nThe conclusions hold for smaller models that are less than 10B may not hold for models that are larger than 65B\n\n- This is actually my largest concern with regard to this paper. Empirically, at least for fine-tuning, people have observed that larger models do not require as many fine-tuning epochs as smaller models. In general, models that are smaller than 10B has different behaviors than models larger than 65B\n- An alternative explanation for the repeating 4 times conclusion drawn by this paper could be: smaller models do not have enough capability to fit the data with only one pass, but larger models may be able to do so.\n\nThe conclusion holds for data scale that is less than 100B may not hold for data scale that is larger than 1T. \n\n- Most of the experiments in this paper consider training models with less than 100B data. In the current practice, models (even if they are just 7B) are mostly trained with 1T data (either code or text). This means that in practice, the problem is whether repeating 1T data 4 times, rather than repeating 10B data 4 times.\n- As the data becomes large, a possible model behavior could be: with 10B data, because data is relatively small, repeating is beneficial, yet when data becomes large, repeating may not be as beneficial as the small data regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thought-provoking points. \n\nWeaknesses:\n\nChoice of the parametric form:\n- Decay: We agree that these experiments do not rule out other parameteric forms, such as polynomial. We think exponential was an intuitive starting point out of possible decay formulations and the sharp dropoff in data value (which rules out forms like linear decay). We stuck with it given that it fits our data well with few parameters. \n- Parameters: While we do not have a mechanistic explanation for this term, our hypothesis why this decay is necessary is that if you are trying to fit an over-parameterized model to little data then parameters learn redundant features, and so they end up \"repeating\" in a similar way to the data. \n\nDifferent types of data:\n- We agree that different types of data is an important factor, although it is challenging to change dataset genre, while controlling for scale. We do run experiment with another large-scale web dataset, OSCAR, which has a different data distribution with less filtering. Despite this difference we found the trends to be about the same. While its difficult to define a general-purpose scalar value of complexity, controlled experiments based on this factor is certainly an interesting research direction for future work. (And personally it would be refreshing if quality had a non-linear scaling factor compared to quantity.)\n\nLarger models:\n- We agree that it is important to further validate these findings at even larger scales than our 8.7B models. We already spent a significant amount of compute for the experiments at hand, so we leave this to future work.\n- We investigated 3 model scales in detail (2.7B, 4.2B, 8.7B) and found very similar behavior. However, it could indeed be that there are fundamental changes when going significantly beyond 10B.\n\nMore data:\n- We have swapped the models in the first plot (right) for larger models trained on more data: 8.67 billion parameters for 178 billion tokens and 6.34 billion parameters trained for 242 billion tokens. This has reinforced our findings as we see the same behavior of many epochs still being beneficial (9.7 epochs). We are indeed not hitting the trillion token regimes and it is important for future work (with access to even more compute) to investigate this.\n- This depends on the model size. E.g. with 10B tokens and a 1B model, you will get lots of value from repeating. However, with 1T tokens and a 1B model, then the model will already be close to its maximum capacity at 1T tokens, so repeating is expected to have less value. If we adapt the model size to be bigger in the second case and have more capacity, then our current findings suggest that we can get the same value out of repeating as in the smaller case.\n\n\nQuestions:\n\nOur intuition is that code data benefits tasks that require long-term state tracking. Here is an example input from bAbI: “Sandra travelled to the office. Sandra went to the bathroom. Mary went to the bedroom. Daniel moved to the hallway. John went to the garden. John travelled to the office. Daniel journeyed to the bedroom. Daniel travelled to the hallway. John went to the bedroom. John travelled to the office. Where is Daniel?” The correct answer is “hallway\". Without code data, the models benchmarked are not able to solve this task. However, as soon as code data is added the ability of state tracking appears to emerge and they can solve such tasks. We believe that this could be due to the necessity of keeping track of variable states in order to accurately predict code."}, {"review_id": "ZelQoToDLC", "reviewer": "Reviewer_ATmT", "summary": "This paper investigates scaling properties of languages, in the presence of *repeated data*: plenty of work has studied the scaling properties of LLMs (and ML architectures in general), both with respect to the number of parameters and amount of data, but they generally assume each new datapoint is unique. However, as newer LLMs start to reach the limit of available (internet) data, repeated use of the same datapoints sets will become more common and understanding the \"worth\" of a repeated datapoint is a very important research question.\n\nThe authors conduct extensive experiments, training hundreds of model ranging from 10 million to 9 billion parameters and trained up for up to 1500 (repeated) epochs. They then propose a novel *data-constrained* scaling law by introducing a multiplicative factor the data term that decays with the number of repetitions and (briefly) validate it, showing a better fit than the typical Chinchilla scaling law in the presence of repeated data.\n\nThey then use their proposed scaling to study two questions:\n\n1. *Allocation: What is the optimal balance of (compute/data) resources?*\nTheir results suggest that its better to first scale epochs/number of repetions, and only then parameter count: by repeating data, they are able to train a comparable model (in terms of performance and compute budge) with 27% less parameters.\n2. *Return: What is the expected value of additional resources?*\nThey discover that while there are diminishing returns to adding more compute resources in the presence of the same training dataset, there's a notable value in repeating data: training up to 4 epochs yields almost the same loss as training on unique data, but beyond that the value of adding compute resources to train further effectively drops to zero.\n\nLastly, they explore several methods to address data limitations without the need to produce new natural language data. They found that incorporating code tokens into training data yielded a two-fold increase in effective tokens, even for tasks solely evaluating natural language. They also explored varying data filtering techniques, concluding that data filtering primarily yields substantial benefits on noisy datasets, and not so much on cleaner ones.", "questions": "N/A", "limitations": "See above", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This was very interesting read, and is quite well written. I think this work has the potential to have big impact in the LLM pretraining community: the limits of unique data we can get from the internet are already being reached, but the findings in this paper suggest that won’t necessarily be a problem for a while, and that repeating tokens a few times to obtain an optimal capacity-data compute allocation is probably fine. The experiments on filtering even hint that it might be better to even be more aggressive in current filtering and repeating the cleaner tokens. Their evaluation on Section 7 is also quite strong, using downstream performance, and the findings with code are insighful and also super relevant to the pretraining community, and further confirm “implicit” knowledge in the community.", "weaknesses": "The only slightly more serious flaw I see is that the proposed scaling law is not well validated: from what I see, in the main paper only a single plot showing the fit is shown, and no comparable fitting metrics (like r2), commonly reported in scaling law papers, are given to quantitatively compare to the unique-data scaling laws.\n\nI think the paper could also be improved by making the theoretical discussion more concise and moving some of the (very interesting) findings from the appendix to the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough summary of the work and careful review. \n \n> by repeating data, they are able to train a comparable model (in terms of performance and compute budget) with 27% less parameters.\n\nWe have updated the main plot (right) to larger scales comparing a 8.7 billion parameter model and a 6.3 billion parameter one. The difference is even more significant now with the smaller model performing better. We further show that the difference between these two models is also significant in terms of downstream performance. Thus, we would go beyond calling them comparable but rather that the smaller model is indeed better.\n\n\n> The experiments on filtering even hint that it might be better to even be more aggressive in current filtering and repeating the cleaner tokens.\n\nThis is a great observation. Indeed the models that repeat C4 twice are better than the models trained on OSCAR (Appendix) with just one repeat.\n\n**Weaknesses:**\n\n> ..fitting metrics..\n\nWe have added R^2 and loss for the different versions of our exponential decay formulation as well as applying no decay at the end of Appendix A. Thank you for this suggestion.\n\n\n> I think the paper could also be improved by making the theoretical discussion more concise and moving some of the (very interesting) findings from the appendix to the main text.\n\nWe agree with the sentiment that much of the Appendix could be part of the main text, however, we are unsure how to reduce the theoretical discussion to make space for that without sacrificing content necessary to understand the main text. If you have specific suggestions, we are open to revising the text based on them.\n"}, {"review_id": "HecUREkdFq", "reviewer": "Reviewer_v1HT", "summary": "This paper demonstrates the scaling law both mathematically and experimentally in data-constrained situations. They analyze three experimental protocols: allocation (fixed unique data), return (fixed FLOPs), and parametric fit, and obtain new conclusions that were not found in previous research. For example, they demonstrate that increasing the data repetition (epochs) is more effective than increasing the parameters when the number of unique tokens is fixed, and the performance drop is shown to be negligible when there is a slight decrease in unique data within a predetermined number of FLOPs. They also analyze the effects of code data augmentation and data filtering strategies that can be applied in data-constrained situations.", "questions": "Since many large language models utilize IF (instruction fine-tuning) and RLHF (reinforcement learning from human feedback), do authors have plans to expand research on scaling laws of data-constrained language models in the IF or RLHF scenario, where the data includes instruction or human feedback data?", "limitations": "The authors adequately suggest limitations in various aspects.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The research is well-motivated and necessary.\n- They present scaling laws and propose optimal training strategies in a situation where data is limited.\n- They derive reliable and generalizable conclusions through extensive experiments.", "weaknesses": "- Increasing the maximum parameter size (e.g. >= 10B in Chinchilla [1]) would further enhance comprehension of the scaling law in large language models.\n\n[1] Hoffmann, Jordan, et al. \"An empirical analysis of compute-optimal large language model training.\" Advances in Neural Information Processing Systems 35 (2022): 30016-30030.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We agree that future work confirming these findings at >10B would be very interesting. The Galactica 120B model from prior work does provide preliminary evidence that you can repeat at these larger scales, but more experiments are needed.\n\nQuestions:\n\n>  Data-Constrained IF (instruction fine-tuning) and RLHF\n\n* While we are very interested in IF and RLHF, we think the scaling environment for these problems is different enough to warrant an analysis outside of the Chinchilla regime. For IF, often the datasets are quite small and training looks more like traditional ML training (for example the LIMA dataset from Meta only contains 1000 examples). For RLHF, the reward model is \"data-constrained\" in the sense that annotations are costly. However this model looks more like a classification/regression model, and would likely have different scaling properties than next-token prediction models. Furthermore, published models only scale up to the order of 1M examples which is a different order of magnitude than these examples. Scaling the RL phase of the system is quite interesting, however, this phase also has a different objective and training procedure (PPO).\n"}, {"review_id": "CCgrU4yHbL", "reviewer": "Reviewer_r7xA", "summary": "This paper studies an important empirical question: what is the optimal tradeoff between LLM size and the amount of data available for training it. The authors follow an existing line of work (e.g., Chinchilla), and additionally consider the role of data repetition (via multiple epochs), e.g., what is the effect of training a model for K epochs on N tokens each, compared to training it for one epoch on K*N unique tokens. The authors adjust the power law equation introduced in Chinchilla to take into account the “effective data”, which considers the lower value of repeated tokens, and similarly the “effective model size”, which considers the lower value of parameters trained on such data. They present an overwhelmingly large set of experiments, some VERY expensive (the total amount of compute spent on this work is 4.5 million GPU hours). This makes this study practically irreproducible for 99.999% of the community on the one hand, but very valuable still to those training such huge models. The conclusions reached by the authors are interesting: e.g., models can be trained on data that is repeated for up to 4 times, and reach a similar loss as models trained on a similar amount of unique tokens. They also study the effect of augmenting text corpora with code, and applying deduplication filtering steps.\n", "questions": "- I am not sure I entirely understand the data repetition setup. In particular, the caption of fig2 says “Training runs with different epochs reuse subsets of the same data to ensure different training data is not a confounding factor.” What does subsets of the same data mean? Is the data sampled from the training data in each repetition? Doesn’t that mean that the data is not effectively repeated?\n\nTypos and such:\n- #132-133: I think R_D and U_D in these lines should be R_N and U_N?\n- #147: [65]partir\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- An important empirical question.\n- A potentially more accurate modeling of the params/data scaling law.\n- A very large set of experiments covering a range of models and scenarios.\n- Some interesting and non-trivial observations. \n- Paper is generally well written\n\n", "weaknesses": "- There is something unclear about using the validation or the test loss. The authors say (#42) that they are reporting test loss, but many of the tables and graphs say that they are showing validation loss. Which one is it? Reporting test loss is a problem, as it might lead to overfitting the test data (e.g., by others who want to adapt the proposed recipes).\n- The loss differences are often very small. E.g., in Fig1 (right), a difference of 0.006, or in Fig5 (left). Are these results significant? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. \n\nWeaknesses:\n\n> There is something unclear about using the validation or the test loss. The authors say (#42) that they are reporting test loss, but many of the tables and graphs say that they are showing validation loss. Which one is it? Reporting test loss is a problem, as it might lead to overfitting the test data (e.g., by others who want to adapt the proposed recipes).\n\nWe are sorry for the lack of clarity. We have rephrased the mention of test loss to “we report loss on a held-out test set unless otherwise specified”. In the main text, only Figure 4 reports validation loss throughout training instead. We have also added a detailed explanation of how loss is computed in a new section Appendix K “Evaluation Details”. We are not sure why reporting the test loss would be a problem. We are using a fully held-out dataset and compute the loss over it for each model at the end of training. Apart from loss, we also show that our findings are mirrored in downstream performance in the Appendix. Given that all our models will be open-sourced, future work can also run them and compute loss on their own datasets if desired.\n\n\n> The loss differences are often very small. E.g., in Fig1 (right), a difference of 0.006, or in Fig5 (left). Are these results significant?\n\nWe have updated Fig1 (right) to larger models with 8.67B and 6.34B parameters. The loss difference for that setup is 0.017. At these scales, loss decreases very slowly, for example for the 8.67B model, its last 5,000 steps ($\\approx$10B tokens) correspond to about a 0.017 loss decrease. Thus, we consider it to be significant. We have also added downstream performance of these two models in the Appendix. Across our 19 downstream evaluation tasks, the 6.34B model has an average score of 25.9, while the 8.7B model scores 23.5. This reinforces our findings that data should be scaled faster than parameters in the repeated regime.\n\nQuestions:\n\nWe are sorry for the confusion. We meant to say that if we have e.g. 10B unique tokens, those same 10B unique tokens are also a subset of every setup where we have >10B unique tokens. This ensures that as much as possible of the data is shared across runs and our results are not impacted by simply having trained on a better data subset. We have rephrased the caption of that Figure as “We ensure that runs using less data (more epochs) always use a subset of the data used in runs with more data (fewer epochs).”.\n\n\nTypos:\n- #132-133: The behavior for R_D = R_D* is the same as the behavior for R_N = R_N* due to the symmetry hence either one is fine in these lines.\n- #147: Thank you for noting, we have fixed it.\n"}]

Author rebuttal:
We thank all reviewers for their thorough reviews. We have made the following updates to the paper:

Increased the scale of the models in the first plot (right) showing even more significant results in support of scaling data faster than parameters when repeating (also in terms of downstream performance shown in a table in the Appendix). As we cannot upload a new version of the paper, we’re attaching the plot and table to this message.

Added several new Appendix sections:
- Appendix G: Case study on how Galactica should have allocated compute according to data-constrained scaling laws
- Appendix K: Details on how loss is computed, how scores are normalized and smoothing for the graphs
- Appendix P: Loss curves of complementary strategies

Improved several Appendix sections:
- Appendix A: More details on the calculation of $U_N$ and comparison of different fits and their loss and $R^2$
- Appendix F: Added experiments on an alternative formulation to let excess parameters hurt performance by decaying alpha and beta
- Appendix I: Fixed one loss plot for OSCAR
- Appendix Q: Added hyperparameter sensitivity as a new limitation
- Appendix R: Specified some more hyperparameters

Rephrased several sentences throughout and fixed typos.

We agree with the overall sentiment expressed by the reviewers that there are many exciting related research questions, such as behavior at hundreds of billions of parameters and trillions of tokens, different datasets beyond C4 and OSCAR and data-constrained scaling laws during finetuning. We look forward to answering these together with the broader research community in future work.


Meta-review: This paper presents a valuable investigation into the scaling of language models in data-constrained regimes, offering insights into the optimal use of compute resources and potential approaches for mitigating data scarcity. The authors conducted a large set of experiments and proposed a scaling law for compute optimality, which was empirically validated. The provision of trained models and datasets used in the experiments for public access adds to the value of this paper. Overall, this work is a very solid contribution to the communnity in pretraining of LLMs.
Decision: Accept (oral)