Title: value: Scaling Data-Constrained Language Models
Authors: value: Niklas Muennighoff
Alexander M Rush
Boaz Barak
Teven Le Scao
Nouamane Tazi
Aleksandra Piktus
Sampo Pyysalo
Thomas Wolf
Colin Raffel
Venue: NeurIPS 2023
Keywords: value: large language models
scaling laws
data engineering

Abstract: value: The current trend of scaling language models involves increasing both parameter count and training dataset size. Extrapolating this trend suggests that training dataset size may soon be limited by the amount of text data available on the internet. Motivated by this limit, we investigate scaling language models in data-constrained regimes. Specifically, we run a large set of experiments varying the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameter models. We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data. However, with more repetition, the value of adding compute eventually decays to zero. We propose and empirically validate a scaling law for compute optimality that accounts for the decreasing value of repeated tokens and excess parameters. Finally, we experiment with approaches mitigating data scarcity, including augmenting the training dataset with code data or removing commonly used filters. Models and datasets from our 400 training runs are freely available at https://github.com/huggingface/datablations.


Review discussion:
- An important empirical question.
- A potentially more accurate modeling of the params/data scaling law.
- A very large set of experiments covering a range of models and scenarios.
- Some interesting and non-trivial observations.
- Paper is generally well written
- There is something unclear about using the validation or the test loss. The authors say (#42) that they are reporting test loss, but many of the tables and graphs say that they are showing validation loss. Which one is it? Reporting test loss is a problem, as it might lead to overfitting the test data (e.g., by others who want to adapt the proposed recipes).
- The loss differences are often very small. E.g., in Fig1 (right), a difference of 0.006, or in Fig5 (left). Are these results significant?
- I am not sure I entirely understand the data repetition setup.
In particular, the caption of fig2 says “Training runs with different epochs reuse subsets of the same data to ensure different training data is not a confounding factor.” What does subsets of the same data mean? Is the data sampled from the training data in each repetition? Doesn’t that mean that the data is not effectively repeated?
Typos and such:
- #132-133: I think R_D and U_D in these lines should be R_N and U_N?
- #147: [65]partir
N/A
No ethics review needed.
Thank you for your thorough review.
Weaknesses:
> There is something unclear about using the validation or the test loss. The authors say (#42) that they are reporting test loss, but many of the tables and graphs say that they are showing validation loss. Which one is it? Reporting test loss is a problem, as it might lead to overfitting the test data (e.g., by others who want to adapt the proposed recipes).
We are sorry for the lack of clarity. We have rephrased the mention of test loss to “we report loss on a held-out test set unless otherwise specified”. In the main text, only Figure 4 reports validation loss throughout training instead. We have also added a detailed explanation of how loss is computed in a new section Appendix K “Evaluation Details”. We are not sure why reporting the test loss would be a problem. We are using a fully held-out dataset and compute the loss over it for each model at the end of training.
Apart from loss, we also show that our findings are mirrored in downstream performance in the Appendix. Given that all our models will be open-sourced, future work can also run them and compute loss on their own datasets if desired.
> The loss differences are often very small. E.g., in Fig1 (right), a difference of 0.006, or in Fig5 (left). Are these results significant?
We have updated Fig1 (right) to larger models with 8.67B and 6.34B parameters. The loss difference for that setup is 0.017. At these scales, loss decreases very slowly, for example for the 8.67B model, its last 5,000 steps ($\approx$10B tokens) correspond to about a 0.017 loss decrease.
Thus, we consider it to be significant. We have also added downstream performance of these two models in the Appendix. Across our 19 downstream evaluation tasks, the 6.34B model has an average score of 25.9, while the 8.7B model scores 23.5. This reinforces our findings that data should be scaled faster than parameters in the repeated regime.
Questions:
We are sorry for the confusion. We meant to say that if we have e.g. 10B unique tokens, those same 10B unique tokens are also a subset of every setup where we have >10B unique tokens. This ensures that as much as possible of the data is shared across runs and our results are not impacted by simply having trained on a better data subset. We have rephrased the caption of that Figure as “We ensure that runs using less data (more epochs) always use a subset of the data used in runs with more data (fewer epochs).”.
Typos:
- #132-133: The behavior for R_D = R_D* is the same as the behavior for R_N = R_N* due to the symmetry hence either one is fine in these lines.
- #147: Thank you for noting, we have fixed it.
Thank you for the clarifications, I am happy with the responses.
- The research is well-motivated and necessary.
- They present scaling laws and propose optimal training strategies in a situation where data is limited.
- They derive reliable and generalizable conclusions through extensive experiments.
- Increasing the maximum parameter size (e.g. >= 10B in Chinchilla [1]) would further enhance comprehension of the scaling law in large language models.
[1] Hoffmann, Jordan, et al. "An empirical analysis of compute-optimal large language model training." Advances in Neural Information Processing Systems 35 (2022): 30016-30030.
Since many large language models utilize IF (instruction fine-tuning) and RLHF (reinforcement learning from human feedback), do authors have plans to expand research on scaling laws of data-constrained language models in the IF or RLHF scenario, where the data includes instruction or human feedback data?
The authors adequately suggest limitations in various aspects.
No ethics review needed.
Thank you for your review. We agree that future work confirming these findings at >10B would be very interesting. The Galactica 120B model from prior work does provide preliminary evidence that you can repeat at these larger scales, but more experiments are needed.
Questions:
>  Data-Constrained IF (instruction fine-tuning) and RLHF
* While we are very interested in IF and RLHF, we think the scaling environment for these problems is different enough to warrant an analysis outside of the Chinchilla regime. For IF, often the datasets are quite small and training looks more like traditional ML training (for example the LIMA dataset from Meta only contains 1000 examples). For RLHF, the reward model is "data-constrained" in the sense that annotations are costly.
However this model looks more like a classification/regression model, and would likely have different scaling properties than next-token prediction models.
Furthermore, published models only scale up to the order of 1M examples which is a different order of magnitude than these examples. Scaling the RL phase of the system is quite interesting, however, this phase also has a different objective and training procedure (PPO).
Thank you for updates and answer about the question. I raised my score after reading the rebuttal.
This was very interesting read, and is quite well written. I think this work has the potential to have big impact in the LLM pretraining community: the limits of unique data we can get from the internet are already being reached, but the findings in this paper suggest that won’t necessarily be a problem for a while, and that repeating tokens a few times to obtain an optimal capacity-data compute allocation is probably fine. The experiments on filtering even hint that it might be better to even be more aggressive in current filtering and repeating the cleaner tokens. Their evaluation on Section 7 is also quite strong, using downstream performance, and the findings with code are insighful and also super relevant to the pretraining community, and further confirm “implicit” knowledge in the community.
The only slightly more serious flaw I see is that the proposed scaling law is not well validated: from what I see, in the main paper only a single plot showing the fit is shown, and no comparable fitting metrics (like r2), commonly reported in scaling law papers, are given to quantitatively compare to the unique-data scaling laws.
I think the paper could also be improved by making the theoretical discussion more concise and moving some of the (very interesting) findings from the appendix to the main text.
N/A
See above
No ethics review needed.
Thank you for your thorough summary of the work and careful review.
> by repeating data, they are able to train a comparable model (in terms of performance and compute budget) with 27% less parameters.
We have updated the main plot (right) to larger scales comparing a 8.7 billion parameter model and a 6.3 billion parameter one. The difference is even more significant now with the smaller model performing better. We further show that the difference between these two models is also significant in terms of downstream performance.
Thus, we would go beyond calling them comparable but rather that the smaller model is indeed better.
> The experiments on filtering even hint that it might be better to even be more aggressive in current filtering and repeating the cleaner tokens.
This is a great observation.
Indeed the models that repeat C4 twice are better than the models trained on OSCAR (Appendix) with just one repeat.
**Weaknesses:**
> ..fitting metrics..
We have added R^2 and loss for the different versions of our exponential decay formulation as well as applying no decay at the end of Appendix A. Thank you for this suggestion.
> I think the paper could also be improved by making the theoretical discussion more concise and moving some of the (very interesting) findings from the appendix to the main text.
We agree with the sentiment that much of the Appendix could be part of the main text, however, we are unsure how to reduce the theoretical discussion to make space for that without sacrificing content necessary to understand the main text. If you have specific suggestions, we are open to revising the text based on them.
> If you have specific suggestions, we are open to revising the text based on them.
I guess my (lazy) suggestion is just arxiv'ing a longer (8+) pages :)
This paper studies a very important problem (what should one do to further scale up data when data is used up) and offers very practical advice (repeating the data 4 times and using code 50% to the training data). I believe this paper will make a clear impact in the area of language models and general AI.
Although this paper has clear advantages in terms of practical guidance, I am concerned about the following weakness:
The choice of the specific parametric form in equation 5 and 6 are not clearly explained and may be subject to debate, specifically:
- Why the decay in equation 5 should be exponential (other than, say, polynomial) as when data points are small, exponential may look similar to polynomial (or other functional forms).
- Why the decay in model scale (Eq. 6) should follow the same form as data (Eq. 5). For the original Chinchilla scaling law, the two terms taking symmetric forms are understandable due to their nature.
But for repeating data, model parameter and data scale are not as symmetric as unique data setting, thus considering the same form of decay as data (though I do believe there should be a certain form of decay in model scale) might be less justified
Different types of data may need different levels of repetition.
- The Chinchilla model mostly considers webpages v.s. the Galactica model mostly consider academic papers. Intuitively, the level of complexity of webpages might be lower than papers, and one may want to repeat the complex data over simple ones.
The conclusions hold for smaller models that are less than 10B may not hold for models that are larger than 65B
- This is actually my largest concern with regard to this paper. Empirically, at least for fine-tuning, people have observed that larger models do not require as many fine-tuning epochs as smaller models. In general, models that are smaller than 10B has different behaviors than models larger than 65B
- An alternative explanation for the repeating 4 times conclusion drawn by this paper could be: smaller models do not have enough capability to fit the data with only one pass, but larger models may be able to do so.
The conclusion holds for data scale that is less than 100B may not hold for data scale that is larger than 1T.
- Most of the experiments in this paper consider training models with less than 100B data. In the current practice, models (even if they are just 7B) are mostly trained with 1T data (either code or text). This means that in practice, the problem is whether repeating 1T data 4 times, rather than repeating 10B data 4 times.
- As the data becomes large, a possible model behavior could be: with 10B data, because data is relatively small, repeating is beneficial, yet when data becomes large, repeating may not be as beneficial as the small data regime.
The authors reported that WebNLG and BaBI immediately improveSee about discussions as one mix in code data. I am curious about what kind of synergy between text and code data and what are the families of text tasks that would benefit the most from mixing code?
See above discussion about weakness
No ethics review needed.
Thank you for your detailed review and thought-provoking points.
Weaknesses:
Choice of the parametric form:
- Decay: We agree that these experiments do not rule out other parameteric forms, such as polynomial. We think exponential was an intuitive starting point out of possible decay formulations and the sharp dropoff in data value (which rules out forms like linear decay). We stuck with it given that it fits our data well with few parameters.
- Parameters: While we do not have a mechanistic explanation for this term, our hypothesis why this decay is necessary is that if you are trying to fit an over-parameterized model to little data then parameters learn redundant features, and so they end up "repeating" in a similar way to the data.
Different types of data:
- We agree that different types of data is an important factor, although it is challenging to change dataset genre, while controlling for scale. We do run experiment with another large-scale web dataset, OSCAR, which has a different data distribution with less filtering.
Despite this difference we found the trends to be about the same.
While its difficult to define a general-purpose scalar value of complexity, controlled experiments based on this factor is certainly an interesting research direction for future work. (And personally it would be refreshing if quality had a non-linear scaling factor compared to quantity.)
Larger models:
- We agree that it is important to further validate these findings at even larger scales than our 8.7B models. We already spent a significant amount of compute for the experiments at hand, so we leave this to future work.
- We investigated 3 model scales in detail (2.7B, 4.2B, 8.7B) and found very similar behavior.
However, it could indeed be that there are fundamental changes when going significantly beyond 10B.
More data:
- We have swapped the models in the first plot (right) for larger models trained on more data: 8.67 billion parameters for 178 billion tokens and 6.34 billion parameters trained for 242 billion tokens. This has reinforced our findings as we see the same behavior of many epochs still being beneficial (9.7 epochs). We are indeed not hitting the trillion token regimes and it is important for future work (with access to even more compute) to investigate this.
- This depends on the model size. E.g. with 10B tokens and a 1B model, you will get lots of value from repeating.
However, with 1T tokens and a 1B model, then the model will already be close to its maximum capacity at 1T tokens, so repeating is expected to have less value. If we adapt the model size to be bigger in the second case and have more capacity, then our current findings suggest that we can get the same value out of repeating as in the smaller case.
Questions:
Our intuition is that code data benefits tasks that require long-term state tracking. Here is an example input from bAbI: “Sandra travelled to the office. Sandra went to the bathroom. Mary went to the bedroom. Daniel moved to the hallway. John went to the garden. John travelled to the office. Daniel journeyed to the bedroom. Daniel travelled to the hallway. John went to the bedroom. John travelled to the office. Where is Daniel?” The correct answer is “hallway". Without code data, the models benchmarked are not able to solve this task.
However, as soon as code data is added the ability of state tracking appears to emerge and they can solve such tasks. We believe that this could be due to the necessity of keeping track of variable states in order to accurately predict code.
We thank all reviewers for their thorough reviews. We have made the following updates to the paper:
Increased the scale of the models in the first plot (right) showing even more significant results in support of scaling data faster than parameters when repeating (also in terms of downstream performance shown in a table in the Appendix). As we cannot upload a new version of the paper, we’re attaching the plot and table to this message.
Added several new Appendix sections:
- Appendix G: Case study on how Galactica should have allocated compute according to data-constrained scaling laws
- Appendix K: Details on how loss is computed, how scores are normalized and smoothing for the graphs
- Appendix P: Loss curves of complementary strategies
Improved several Appendix sections:
- Appendix A: More details on the calculation of $U_N$ and comparison of different fits and their loss and $R^2$
- Appendix F: Added experiments on an alternative formulation to let excess parameters hurt performance by decaying alpha and beta
- Appendix I: Fixed one loss plot for OSCAR
- Appendix Q: Added hyperparameter sensitivity as a new limitation
- Appendix R: Specified some more hyperparameters
Rephrased several sentences throughout and fixed typos.
We agree with the overall sentiment expressed by the reviewers that there are many exciting related research questions, such as behavior at hundreds of billions of parameters and trillions of tokens, different datasets beyond C4 and OSCAR and data-constrained scaling laws during finetuning. We look forward to answering these together with the broader research community in future work.

Meta-review:
role: PC
data: comment: value: This paper presents a valuable investigation into the scaling of language models in data-constrained regimes, offering insights into the optimal use of compute resources and potential approaches for mitigating data scarcity. The authors conducted a large set of experiments and proposed a scaling law for compute optimality, which was empirically validated. The provision of trained models and datasets used in the experiments for public access adds to the value of this paper. Overall, this work is a very solid contribution to the communnity in pretraining of LLMs.

Decision: Accept (oral)