Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an empirical study to explore the scaling law for training Transformer-based large language models (LLM). In this paper, the scaling law is the trade-off between model size and training tokens, and it is explored by different fitting methods. To obtain the empirical data for fitting the scaling curves, the authors train various language models by varying the model size and the training FLOP counts. The estimated power-law relationship indicates that the optimal scaling way is to increase the size of training data equally while enlarging the model size. To verify the correctness of this optimal scaling law, the authors train the *Chinchilla* model with larger training tokens but decrease the model size to 70B compared to *Gopher* (280B). According to the evaluation results shown in Appendix H, *Chinchilla* outperforms other *Gopher* on most benchmarks while sharing the same FLOPs cost. 

The contributions of this paper are as follows: 1) refining the existing scaling law explored by *Gopher*; 2) revealing an optimal scaling law regarding the training tokens and model size for training LLMs.

strengths_and_weaknesses: **Strength**:
1. ***Significance***: The scaling law explored in this empirical study is useful to train the LLM. The authors also claim that the current LLMs are under-trained, the conclusion in this paper may show an interesting direction for this community to keep optimizing the LLMs: we need to pay attention to efficiently learning the data instead of enlarging the model size.
2. ***Originality***: Although this paper follows the research methodology of a previous study [1], i.e., an empirical study, this paper eventually shows a new scaling law. 

**Weakness**:
1. ***Soundness***: Since this paper uses the empirical method to explore the optimal scaling law, the theoretical foundation is not very solid. How can we judge whether a model is under-trained in a more sound way? Besides, given that the three modeling approaches in this paper rely on empirical training records, one problem is the random factors of training. As the authors claimed in Checklist 3.(c), the costs of training these models are expensive, they did not perform training with different random seeds.
2. ***Clarity***: This paper is easy to follow and well-written. But there are some typos in the paper:
    - Line 1: a transformer -> a Transformer
    - Line 37: budget, Instead, -> budget, instead
    - Line 71: [24] first -> Kaplan et al. [24] first 
    - Line 74: differs from [24] -> differs from Kaplan et al. [24]
    - The paper title is different from the information shown in this forum 

**Reference**:

[1] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., ... & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361

***Update***: The authors addressed most of concerns mentioned in Weaknesses part.
questions: 1. In Section 3.1 (Line 130), you mentioned that smoothing and interpolation methods were applied to each training loss curve. I found the details of the smoothing method in Appendix D.1. But which interpolation method did you use in Approach 1?
2. Have you examined the overlap between pre-training data and downstream fine-tuning data? (especially for the evaluation of language modeling tasks)
3. You mentioned that current LLMs are under-trained. Is there any metric to evaluate the bottleneck of model capacity instead of empirical methods?

***Update***: The authors have replied to Q1, Q2 with further clarification.
limitations: - Societal Impact: The societal impact is positive. For those researchers who work on developing large-scale language models, the authors provide a useful scaling law to save the computational budget. 
- Limitation: The major limitation still lies in the research methodology, the empirical study may not always be solid as I stated in the Weaknesses part.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores what is the best model size and number or tokens for training Transformer models. They trained over 400 language models with model sizes ranging from 70M to over 16B parameters on 5 to 500 billion tokens, and it suggests the model size and training data should be scaled equally. Based on this finding, they train a 70B parameters model, a predicted compute-optimal model, called Chinchilla, on more training data which outperforms the Gopher 280B model with same compute budget. 
strengths_and_weaknesses: Strengths:

The paper is very interesting and inspiring. 

It gives a good guideline when people want to scale up their LMs. 

Strong results. 


Weaknesses:

It is only tested on autoregressive models. I’m wondering whether this observation is held on BERT-like models.   

questions: I'm wondering how the data quality affects this scale-law. 
limitations: None
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Kaplan et al. (2020) suggested that large models should not be trained to their lowest possible loss to be compute-optimal: instead, model size should grow faster than the size of the training set, given a fixed computational budget increase. 
This paper is a more thorough investigation of the question: what are the optimal model size and number of training tokens for a given training budget, tuning additional hyperparameters ignored by prior work? The authors find that model size and training budget should, in fact and in contrast to prior findings, be scaled equally -- since this isn't typically done, many state-of-the-art models are undertrained. They then use this insight to train a new model (which they call Chinchilla), which uses more data but the same compute as a state-of-the-art model and significantly outperforms the latter on multiple downstream tasks. 
strengths_and_weaknesses: Strengths:
- This paper provides experimental results that contradict previous findings and might be quite consequential for the future of large language models. 
- The experiments are quite extensive. (Except for the fact that there are limited complete pretraining runs, but this makes sense given the computational cost.) 
- The paper is clearly written.

Weaknesses:
- No obvious ones.
questions: None.
limitations: The authors do explicitly address the limitations of their work in Section 5, which is great. I don't see any direct negative societal impact of the word besides, maybe, the deployment of biased language models, but the authors mention this risk.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Current NLP is centered around language modeling. We've observed that their performance can significantly increase when their parameter count is increased and therefore a lot of effort has been focused on scaling that property. 
The authors here note that the community hasn't really explored the importance of the amount of training data, and so that is the focus here. The authors, through many experiments, discover that when the number of parameters is doubled, so should the amount of training data. 
The experiments are thorough and the marquee result is incredibly interesting and impactful- with a 70B param model they are able to outperform GPT-3 and other 170B+ parameter models on many challenging downstream tasks including MMLU and some Big Bench tasks. 
strengths_and_weaknesses: Strengths:
1) Very thorough experiments.
2) Good writing
3) An easy-to-understand conclusion that is very straightforward to implement. 
4) The bottom line here (how much data is needed to train big models) is an incredibly important result which will be useful for years to come.
5) This paper explores a very important but very under-explored topic.

Weaknesses:
1) No code/models are released, which is really bad for reproducibility. 
questions: Are all tokens created equal? I didn't really see a discussion of this in the paper, but when you talk about doubling the training data, is any data going to be OK? Are there specific domains that are more beneficial for downstream tasks? I'm sure 20 papers could be written about this question, I don't think you need to provide all the answers, but it could be beneficial to have a brief discussion of this in the paper.


limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: Four experts reviewed this paper and they all recommended acceptance. The paper finds that current Transformer-based large language models (LLM) are significantly undertrained. This is likely to be of great interest to the AI/NLP community, as it challenges current practices and recommendations from prior work. The paper's main recommendation is that, given a increase of computation budget, model size and number of training tokens should be scaled equally. The claims of the paper are supported with an extensive amount of experimentation, including 400 language models, model sizes ranging from 70M to over 16B parameters, and amounts of data ranging from 5 to 500 billion tokens. Reviewers either listed no weaknesses or had most of their concerns addressed by the authors' responses. The main remaining limitation is that the authors couldn't release any code or data, but the work seems mostly reproducible from the paper (extensive methodological and experimental details are given in the appendix).
confidence: Certain
Decision: Accept