An empirical analysis of compute-optimal large language model training
Authors: Jordan Hoffmann
Sebastian Borgeaud
Arthur Mensch
Elena Buchatskaya
Trevor Cai
Eliza Rutherford
Diego de las Casas
Lisa Anne Hendricks
Johannes Welbl
Aidan Clark
Tom Hennigan
Eric Noland
Katherine Millican
George van den Driessche
Bogdan Damoc
Aurelia Guy
Simon Osindero
Karen Simonyan
Erich Elsen
Oriol Vinyals
Jack William Rae
Laurent Sifre
Venue: NeurIPS 2022 Accept (2022)
Keywords: NLP
Deep Learning
Large Language Models

Abstract: We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4$\times$ more data. Chinchilla uniformly and significantly outperformsGopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, a 7% improvement over Gopher. 

Training Compute-Optimal Large Language Models
Jordan Hoffmann"
Sebastian Borgeaud"
Arthur Mensch
Elena Buchatskaya
Trevor Cai
Eliza Rutherford
Diego de Las Casas
Lisa Anne Hendricks
Johannes Welbl
Aidan Clark
Tom Hennigan
Eric Noland
Katie Millican
George van den Driessche
Bogdan Damoc
Aurelia Guy
Simon Osindero
Karen Simonyan
Erich Elsen
Oriol Vinyals
Jack W. Rae
Laurent Sifre"
Equal contributions
DeepMind
(sborgeaud) amensch sifre) @deepmind. com
Abstract
We investigate the optimal model size and number of tokens for training Trans- former language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4x more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron- Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substan- tially less compute for fine-tuning and inference, greatly facilitating downstream usage. As highlight, Chinchilla reaches state-of-te thefthh-rrtt average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
parameters on 5 to 500 billion tokens, find that for compute-opting
Introduction
series Large Language Models (LLMs) have recently been introduced [6. 30, 38, 48, 52], with the largest dense language models now having over 500 billion parameters. These large autoregressive many tasks using variety of evaluation protocols: zero-shot generalization, few-shot training, and as a basis for fine-tuning. The compute and energy cost for training large language models is substantial [38, 52] and rises with increasing model size. In practice, the allocated training compute budget is often known in advance: practitioners have access certain number of accelerators for given period of time. Since
transformers [53] have demonstrated impressive performance
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

17
Approach 1
100B
Approach
Approach 3
Parameters
Kaplan (2020)
10B
Chinchilla (70B)
Gopher (280B)
1.08
GPT-3 (175B)
Megatron- Turing NLG (530B)
100M
10M
10
10
10 FLOPs
1022 1025
Figure 1: Overlaid predictions. We overlay the predictions from our three different approaches, along with projections from [23]. We find that all three methods predict that current large models should be substantially smaller and therefore trained much longer than is currently done. In Figure A3, we show the results with the predicted optimal tokens plotted against the optimal number of parameters for fixed FLOP budgets. Chinchilla outperforms Gopher and the other large models (see Section 4.2).
Table 1: Current LLMs. We show five of the current largest dense transformer models, their size, and the number of training tokens. Other than LaMDA [52], most models are trained for approximately 300 billion tokens. We introduce Chinchilla, substantially smaller model, trained for much longer than 300B tokens. Table A3 shows our projected optimal relation between model size and tokens.
Model
Size (# Parameters)
Training Tokens
LaMDA [52]
137 Billion
768 Billion
GPT-3 [6]
175 Billion
300 Billion
Jurassic [30]
178 Billion
300 Billion
Gopher [38]
280 Billion
300 Billion
MT-NLG 530B [48]
530 Billion
270 Billion
Chinchilla
70 Billion
1.4 Trillion
typically only feasible to train these large models once, accurately estimating the best model hyperparameters for a given compute budget is critical [51].
Kaplan et al. [23] showed that there is power law relationship between the number of parameters in autoregressive language model (LM) and its performance (measured in evaluation perplexity). One notable conclusion [23] that large models should not trained to their lowest possible loss to be compute optimal; they argue that model size should grow faster than the size of the training set for a given increase of computational budget. As a result, the field has been training larger and larger models while keeping the size of the training set to approximately 300 billion tokens, expecting performance improvements (Table 1). While we find that there is effectively trade-off between model size and training size, estimate that large models should be trained for many more training tokens than recommended by [23]. Specifically, given 10x increase computational budget find that model size and the number of training tokens should be scaled in equal proportions.
this work, we revisit the question: Given a fixed FLOPs budget,' 1 how should one trade-off model size and the number of training tokens? To answer this question, we model the final pre-training loss2 L(N, D) as a function of the number of model parameters N, and the number of training tokens, D. Since the computational budget C' is deterministic function FLOPs(NN D) of the number of seen training tokens and model parameters, we are interested in minimizing under the constraint
For example, knowing the number of accelerators a target training duration. Forr simplicity, we perform our analysis on the smoothed training loss which is unbiased estimate of the test loss, the number of training tokens is less than the number of tokens in the entire corpus.

---

FLOPs(N, D) = C:
Nopt ( C)2, Dopt
L(N,)) N. SL (L,PS(N,D)=
argmin
(I)
The functions Nope(C) and Daps (C) describe the optimal allocation of a computational budget C. We empirically estimate these functions based on the losses of over 400 models, ranging from under 70M to over 16B parameters, and trained on 5B to over 400B tokens- with each model configuration trained for several different training horizons. Our approach leads to considerably different results than that of [23]. We highlight our results in Figure and how our approaches differ in Section 2.
Based on our estimated compute-optimal frontier, we predict that for the compute budget used to train Gopher, an optimal model should be 4 times smaller, while being training 4 times more tokens. We verify this by training a more compute-optinal 70B model, called Chinchilla, on 1.4 trillion tokens. Not only does Chinchilla outperform its much larger counterpart, Gopher, but its reduced model size reduces inference cost considerably and greatly facilitates downstream uses on smaller hardware. The energy cost of large language model is amortized through its usage for inference and fine-tuning. The benefits of a more optimally trained smaller model, therefore, extend beyond the immediate benefits of its improved performance.
2 Related Work
Large language models. A variety of large language models have been introduced in the last few years. These include both dense transformer models [6. 30, 48, 38, 52] and mixture-of (MoE) models [11, 12, 60J. The largest dense transformers have passed 500 billion parameters [48, 8). The drive to train larger and larger models clear-so so far increasing the size of language models has been responsible for improving the state-of-the-ar in many language modelling tasks. Nonetheless, large language models face several challenges, including their overwhelming computational requirements (the cost of training and inference increase with model size) [38, 52] and the need for acquiring more high-quality training data. In fact, in this work we find that larger, high quality datasets will play a key role in any further scaling of language models. Concurrent to our work, 540 billion parameter model trained on 768 billion tokens was released- PaLM [8]. While this model outperforms Chinchilla, uses approximately 5x the compute and is nearly 8x larger, making it more difficult to use.
Modelling the scaling behavior. Understanding the scaling behaviour of language models and their transfer properties has been important in the development of recent large models [23, 18]. Kaplan et al. [23] first showed a predictable relationship between model size and loss over many orders of magnitude. The authors investigate the question of choosing the optimal model size to train for given compute budget. Similar to us, they address this question by training various models. Our work differs from Kaplan et al. [23] in several important ways. First, the authors a fixed number of training tokens and learning rate schedule for all models; this prevents them from modelling the the loss. In contrast, we find that setting the learning rate schedule
impact of these hyperparameters to approximately match the number of training tokens results in the best final loss regardless of model size-see Figure A1. For fixed learning rate cosine schedule to 130B tokens, the intermediate loss estimates (for D' << 130B) are therefore overestimates of the loss of model trained with a schedule length matching D'. Using these intermediate losses results in underestimating the effectiveness of training models on less data than 130B tokens, and eventually contributes to the conclusion that model size should increase faster than training data size as compute budget increases. In contrast, our analysis predicts that both quantities should scale at roughly the same rate. Secondly, we include models with up to 16B parameters, as we observe that there is slight curvature in the FLOP-loss frontier (see Appendix E)-in in fact, the majority of the models used in our analysis have more than 500 million parameters, contrast the majority of runs in [23] are significantly amaller-many being less than 100M parameters. Clark et al. [9] specifically looked in to the scaling properties of Mixture of Expert language models, showing that the scaling with number of experts diminishes as the model size increases- their approach models the loss as function of two variables: the model size and the number of experts. However, the analysis is done with a fixed number of tokens, potentially underestimating the improvements of branching.
Estimating hyperparameters for large models. The model size and the number of training tokens not the only two parameters to chose when selecting language model and a procedure to train
3

---

6.0
5.5
2010
5.0
4.5
2.50
1010
1011
3.5
500M
1:0
3.9
1.66
2.5
12
10
2013
10:
10 FLOPs
10
FLOPS
FLOPS
Figure 2: Training curve envelope. On the left we show all of our different runs. We launched range model sizes going from 70M to 1OB, each for four different cosine cycle lengths. From these curves, we extracted the envelope of minimal loss per FLOP, and we used these points to estimate the optimal model size (center) for a given compute budget and the optimal number of training tokens (right). In green, show projections of optimal model size and training token count based on the number FLOPs used to train Gopher (5.76 x 1023).
it. Other important factors include learning rate, learning rate schedule, batch size, optimiser, and width-to-depth ratio. In this work, focus model size and the number of training steps, and we rely on existing work and provided experimental heuristics to determine the other necessary hyperparameters. Yang al. [57] investigates how to choose variety of these parameters for training an autoregressive transformer, including the learning rate and batch size. McCandlish et al. [33] finds only weak dependence between optimal batch size and model size. Shallue et al. [46], Zhang et al.
[59] suggest that using larger batch-sizes than those we use is possible. Levine et al. [28] investigates the optimal depth-to-width ratio for variety of standard model sizes. We use slightly less deep models than proposed as this translates to better wall-clock performance on our hardware.
Improved model architectures. Recently, various promising alternatives to traditional dense transformers have been proposed. For example, through the use of conditional computation large MoE models like the 1.7 trillion parameter Switch transformer [12], the 1.2 Trillion parameter GLaM model [11], and others [1, 60] are able to provide large effective model size despite using relatively fewer training and inference FLOPPs However, for very large models the computational benefits of routed models seems to diminish [9]. An orthogonal approach to improving language models is to augment transformers with explicit retrieval mechanisms, as done by [4, 15, 29]. This approach effectively increases the number data tokens seen during training (by factor 10 in [4]). This suggests that the performance of language models may be more dependant the size of the training data than previously thought.
3 Estimating the optimal parameter/training tokens allocation
We present three different approaches to answer the question driving our research: Given a fixed FLOPs budget, how should trade-off model size and the number of training tokens? In all three cases we start by training a range of models varying both model size and the number of training tokens and use the resulting training curves to fit an empirical estimator of how they should scale. We assume power-law relationship between compute and model size as done in [9, 23], though future work may want to include potential curvature in this relationship for large model sizes. The resulting predictions are similar for all three methods and suggest that parameter count and number of training tokens should be increased equally with more compute wwith proportions reported in Table This is in clear contrast to previous work on this topic and warrants further investigation.
3.1 Approach Fix model sizes and vary number of training tokens
In our first approach we vary the number of training steps for a fixed family of models (ranging from 70M to over 10B parameters), training each model for different number of training sequences. From these runs, we are able to directly extract an estimate of the minimum loss achieved for a given number of training FLOPs. Training details for this approach can be found in Appendix D.

---

32
30
Takire
1008
2.6
3e39
Se19
600 0000
2.4
3620
1420
5220
22
100M 300M
FLOPs
Figure 3: IsoFLOP curves. For various model sizes, we choose the number training tokens such that the final FLOPs is constant. The cosine cycle length is set to match the target FLOP count. We find a clear valley loss, meaning that for given FLOP budget there an optimal model to train (left). Using the location of these valleys, we project optimal model size and number of tokens for larger models (center and right). In green, we show the estimated number of parameters and tokens for optimal model trained with the compute budget of Gopher.
For each parameter count N train 4 different models: each different horizon (measured in number of training tokens) over which we decay the learning rate by factor of 10x; the range of horizons varies by a factor of 16x for each parameter count. We smooth and linearly interpolate each training loss curve. From this, we obtain continuous mapping from FLOP count to training loss for each run. We then determine which run achieves the lowest loss for each FLOP count. Using these interpolants, we obtain mapping from FLOP count C' to the most efficient choice of model size Nopt and number of training tokens Dopt such that FLOPs(Noptr (Noptt Dopt) C., We apply this mapping onto logarithmically spaced values of C' and obtain many empirical triplets (C,, D D. Finally, we power laws to these empirical data, estimating a and such that Nopt xx Ca and Dopt оx Co. We find that 0.50 and b 0.50 as summarized in Table 2. We perform simple experiment for early validating our analysis: given a budget 10 FL.OPs, we compare the performance training model with a size recommended by our analysis to training model with size suggested by the analysis of
the model size we predict has a clear advantage (Section D.4).
3.2 Approach IsoFLOP profiles
In our second approach we vary the model size for fixed set of 9 different training FLOP counts (ranging from 6 10 x 1021 FLOPs), and consider the final training loss for each point. This differs from the first approach that considers points (N,, D,, Li); along the entire training runs; the data points are here scarcer but more representative of the performance of fully trained model.
For each FLOP budget, we plot the final loss (after smoothing) against the parameter count in Figure (left). In all cases, ensure that we have trained a diverse enough set of model sizes to see clear minimum in the loss. We a parabola to each IsoFLOPs curve to directly estimate at what model size the minimum loss is achieved (Figure 3 (left)) As with the previous approach, we then fit power law between FLOPs and loss-optimal model size and number of training tokens, shown in Figure (center, right). Again, fit exponents of the form Nopt C and Dopt O and we find that a = 0.49 and b = as summarized in Table 2.
3.3 Approach Fitting a parametric loss function
Lastly, we model all final losses from experiments in Approach & 2 parametric function of model parameter count and the number of seen tokens. Following a classical risk decomposition (see Section D.2), we propose the following functional form
A
B
İ(N, (N
Na
D
The first term captures the loss for an ideal generative process on the data distribution, and should correspond to the entropy of natural text. The second term captures the fact that perfectly trained
'Note that all selected points are within the last 15% of training. Thus, when training model over D) tokens, should pick cosine eycle length that decays 10x approximately D tokens see Appendix B.
5

---

IsoLoss contours
IsoFLOPs slices
1008
5,00
408
FLOPs
6c+18
1e+19
Model size
3e+19
Bauoo D
6c+19
3.00
a
1e+20
3c+20
1e+21
3e+21
Efficient frontier
Gopher
100M
Empirical data
2.00
IsoFLOPs
1018
1019
1020
10
10a2
102h budget
Gopher
100M
18
108
408
Training FLOPs
Model size
Figure 4: Parametric fit. We fit a parametric modelling of the loss i(N,, D) and display contour (left) and isoFLOP slices (right). For each isoFLOP slice, we include corresponding dashed line in the left plot. In the left plot, we show the efficient frontier in blue, which is a line in log-log space. Specifically, the curve goes through each iso-loss contour at the point with the fewest FLOPs. We project the optimal model size given the Gopher FLOP budget to 40B parameters.
transformer with N parameters underperforms the ideal generative process. The final term captures the fact that the transformer is not trained to convergence, we only make finite number of optimisation steps, on a sample of the dataset distribution.
Model fitting. To estimate (A, B, E, a, 3). we minimize the Huber loss [19] between the predicted and observed log loss using the L-BFGS algorithm [36]:
min A.B.E.o,8
Σ
Hubers
(log L(N, - log L,
Runs
We account for possible local minima by selecting the best fit from grid of initialisations. The " is robust to outliers, which we find important for good predictive performance over held-out data points. Section D.2 details the fitting procedure and the loss decomposition.
Huber loss (6 = 10 10-3)
Efficient frontier. We approximate the functions Nopt and Dopt by minimizing the parametric loss i under the constraint FLOPs(N, as 6ND [23]. The resulting Nopt and Dopt balance the two terms in Equation (3) that depend on model size and data. By construction, they have power-law form. We show contours of the fitted function Ż in Figure 4 (left), and the closed-form efficient computational frontier in blue. From this approach, we find that a 0.46 and b = 0.54 as summarized in Table 2.
3.4 Optimal model scaling
We find that the three approaches, despite using different fitting methodologies and different trained models, yield comparable predictions for the optimal scaling in parameters and tokens with FLOPSs (shown in Table 2). All three approaches suggest that compute budget increases, model size and the amount training data should be increased in approximately equal proportions. The first and second approaches yield very similar predictions for optimal model sizes, shown in Figure I and Figure A3. The third approach predicts even smaller models being optimal at larger compute budgets. We note that the observed points (L, N. D) for low training FLOPs (C < 1e21) have larger residuals
-L--(N D) than points with higher computational budgets. The fitted model places increased weight on the points with more FLOPs aog-amatmaaicllllyy considering the low-computational budget points as outliers due to the Huber loss. As consequence of the empirically observed negative Nopt (see Appendix E), this results predicting lower Nopt than the
curvature in the frontier two other approaches.

---

Table 2: Estimated parameter and data scaling with increased training compute. The listed values are the exponents, and b, on the relationship Nopt & Ca and Dopt x Ch. Our analysis suggests near equal scaling in parameters and data with increasing compute which is in clear contrast to previous work on the scaling of large models. The 10th and 90th percentiles are estimated via bootstrapping data (80% of the dataset sampled 100 times) and are shown in parenthesis.
Approach
Coeff. where Nopt C
Coeff. b where Dopt x C
1. Minimum over training curves
0.50 (0.488, 0.502)
0.50 (0.501, 0.512)
2. IsoFLOP profiles
0.49 (0.462, 0.534)
0.51 (0.483, 0.529)
3. Parametric modelling of the loss
0.46 (0.454, 0.455)
0.54 (0.542, 0.543) 0.27
Kaplan et al. (2020) [23]
0.73
In Table A3 we show the estimated number of FLOPs and tokens that would ensure that model of given size lies on the compute-optimat frontier. Our findings suggests that the current generation of large language models are considerably over-sized, given their respective compute budgets, as shown in Figure 1. Furthermore, the amount of training data that is projected to be needed is far beyond what is currently used train large models, and underscores the importance of dataset collection in addition to engineering improvements that allow for model scale. While there is significant many orders of magnitude, our analysis clearly suggests that given the training compute budget for many current LLMs. smaller models should have been trained on more tokens to achieve the most performant model. In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38]. In both cases we reach the similar conclusion that model size and number of training tokens should be scaled in equal proportions.
uncertainty extrapolating
4 Chinchilla
Based on our analysis in Section 3. the optimal model size for the Gopher compute budget is somewhere between 40 and 70 billion parameters. We test this hypothesis by training a model on the larger end of this range-70B parameters-for 1.4T tokens, due to both dataset and computational efficiency considerations. In this section we compare this model, which we call Chinchilla, to Gopher and other LLMs. Both Chinchilla and Gopher have been trained for the same number of FLOPs but differ in the size of the model and the number of training tokens. While pre-training a large language model has considerable compute cost, downstream fine-tuning and inference also make up substantial compute usage [38]. Due to being 4x smaller than Gopher, both the memory footprint and inference cost of Chinchilla also smaller.
4.1 Model and training details
The full set of hyperparameters used to train Chinchilla are given in Table 3. Chinchilla uses the same model architecture and training setup Gopher with the exception of the differences listed below.
We train Chinchilla on MassiveText (the same dataset as Gopher) but use a slightly different subset distribution (Table A1) account for the increased number of training tokens.
We use AdamW [32] for Chinchilla rather than Adam [24] as this improves the language modelling loss and the downstream task performance after finetuning."
We train Chinchilla with a slightly modified SentencePiece [25] tokenizer that does not apply NFKC normalisation. The vocabulary is very similar- 15% of tokens are the same as those used for training Gopher. We find that this particularly helps with the representation of mathematics and chemistry, for example.
Whilst the forward and backward pass are computed in bfloat16, we store a float32 copy of the weights in the distributed optimiser state [41]. See Lessons Learned from [38] for additional details.
'A model trained with AdamW only passes the training performance of a model trained with Adam around 80% of the way through the cosine cycle, though the ending performance is notably better- see Figure A7
7

---

In Appendix G we show the impact of the various optimiser related changes between Chinchilla and Gopher. All models in this analysis have been trained on TPUv3/TPUv4 [22] with JAX [5] and Haiku [17]. We include Chinchilla model card [35] in Table A13.
4.2 Results
We perform an extensive evaluation of Chinchilla, comparing against various large language models. We evaluate on a large subset of the tasks presented in [38]. shown in Table A6, As the focus of this work is on optimal model scaling, we included a large representative subset, and introduce few new ovaluations to allow for better comparison to other existing large models. The evaluation details for all tasks are the same as described in [38].
Language modelling. Chinchilla significantly outperforms Gopher on all evaluation subsets of The Pile [13], as shown in Figure A8. Compared to Jurassic-1 (178B) [30], Chinchilla is more performant on all but two subsets- dm_ comm_mthemnin www.mmatttticcc.m. and ubuntu_irce see Table A7 for a raw bits- sits-per-byte comparison. On Wikitext [34], Chinchilla reaches 7.16 perplexity compared to 7.75 for Gopher.
The Massive Multitask Language Understanding (MMLU) benchmark [16] consists of range of exam-like questions on academic subjects. In Table A8, we report Chinchilla's 's average 5- shot performance on MMLU (the full breakdown of results is shown in Table A9). On this benchmark, Chinchilla significantly outperforms Gopher despite being much smaller, with an average accuracy of 67.6% (improving upon Gopher by 7.6%). Remarkably, Chinchilla even outperforms the expert forecast for June 2023 of 63.4% accuracy (see Table A8) [50]. Furthermore, Chinchilla achieves greater than 90% accuracy on different individual tasks- sigh_school_gov _ar__oov_____daa_ee and_poldd__ppppppppppppppdd i tics, International.. _aw,_law sociology, and us cerrein_poolicee To our knowledge, no other model has achieved greater than 90% accuracy on a subset. In Figure A9, we show a comparison to Gopher broken down by task.
MMLU.
Reading comprehension. On the final word prediction dataset LAMBADA [37], Chinchilla achieves 77.4% accuracy, compared to 74.5% accuracy from Gopher and 76.6% from MT-NLG 530B (see Table 4). On RACE-h and RACE-m [27], Chinchilla greatly outperforms Gopher, improving accuracy by more than 10% in both cases-see Table 4.
BIG-bench. We analysed Chinchilla on the same set of BIG-bench tasks [49] reported in [38]. Similar to what we observed in MMLU, Chinchilla outperforms Gopher on the vast majority of tasks (see Figure A10). We find that Chinchilla improves the average performance by 10.7%, reaching an accuracy of 65.1 % versus 54.4% for Gopher. Full accuracy results can be found in Table A10.
Common sense. We evaluate Chinchilla various common sense benchmarks: PIQA [3], SIQA [45], Winogrande [44], HcllaSwag [58], and BoolQ [10]. We find that Chinchilla outperforms both Gopher and GPT-3 on all tasks and outperforms MT-NLG 530B all but one task-sea Table 5. On TruthfulQA [31], Chinchilla reaches 43.6%, 58.5%, and 66.7% accuracy with O-shot, 5-shot, and 10-shot respectively. In comparison, Gopher achieved only 29.5% O-shot and 43.7% 10-shot accuracy. In stark contrast with the findings of [31], the large improvements (14.1% in 0-shot accuracy) achieved by Chinchilla suggest that better modelling of the pre-training data alone can lead to substantial improvements on this benchmark.
Closed-book question answering. Results on closed-book question answering benchmarks are reported in Table A11. On the Natural Questions dataset [26), Chinchilla achieves new closed-book SOTA accuracies: 31.5% 5-shot and 35.5% 64-shot, compared to 21% and 28% respectively, for Gopher. On TriviaQA [21] we show results for both the filtered (previously used in retrieval and open-
Table 3: Chinchilla architecture details. We list the number of layers, the key/value size, the bottleneck activation size dmodet, the maximum learning rate, and the training batch size (# tokens). The feed-forward size is always set to dmodet- Note that we double the batch size midway through training for both Chinchilla and Gopher.
Model
Layers
Number Heads
Key/Value Size
dmadel
Max LR
Batch Size
Gopher 280B
80
128
128
16,384
4 x 10-5
3M + 6M
Chinchilla 70B
80
64
128
8,192
1 1 10
1.5M + 3M
8

---

Table 4: Reading comprehension. On RACE-h and RACE-m [27], Chinchilla considerably improves performance over Gopher. Note that GPT-3 and MT-NLG 530B use a different prompt format than we on RACE-h/m, so results are not comparable to Gopher and Chinchilla. On LAMBADA [37], Chinchilla outperforms both Gopher and MT-NIG 530B.
Chinchilla
Gopher
GPT-3
MT-NLG 530B
LAMBADA Zero-Shot
77.4
74.5
76.2
76.6
RACE-m Few-Shot
86.8
75.1
58.1
RACE-h Few-Shot
82.3
71.6
46.8
47.9
Table 5: Zero-shot comparison on Common Sense benchmarks. We show comparison between Chinchilla, Gopher, and MT-NLG 530B on various Common Sense benchmarks. We see that Chinchilla matches or outperforms Gopher and GPT-3 on all tasks. On all but one Chinchilla outperforms the much larger MT-NLG 530B model.
Chinchilla
Gopher
GPT-3
MT-NLG 530B
Supervised SOTA
HellaSWAG 80.8%
79.2%
78.9%
80.2%
93,9%
PIQA 81.8%
81.8%
81.0%
82.0 %
90.1%
Winogrande
74.9%
70.1%
70.2%
73.0%
91.3%
SIQA 51.3%
50.6%
83.2%
BoolQ 83.7%
79.3%
60.5%
78.2%
91.4%
book work) and unfiltered set (previously used in large language model evaluations). In both cases, Chinchilla substantially out performs Gopher. the filtered version, Chinchilla lags behind the open book SOTA [20] by 7.9%. On the unfiltered set, Chinchilla outperforms GPT-3 (see Table A11).
5 Discussion & Conclusion
The trend so far in large language model training has been to increase the model size, often without increasing the number of training tokens. The largest dense transformer, MT-NLG 530B, is now over 3x larger than GPT-3's 170 billion parameters from just two years ago. However, this model, as well as the majority of existing large models, have all been trained for a comparable number of tokens-around 300 billion. While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that substantially underperforming compared to what could be achieved with the same compute budget.
We propose three predictive approaches towards optimally setting model size and training duration, based on the outcome of over 400 training runs. All three approaches predict that Gopher is substantially over-sized and estimate that for the same compute budget smaller model trained on more data will perform better. We directly test this hypothesis by training Chinchilla, 70B parameter model, and show that it outperforms Gopher and even larger models nearly every measured evaluation task.
Whilst our method allows us to make predictions on how to scale large models when given additional compute, there are several limitations. Due to the cost of training large models, we only have two comparable training runs at large scale (Chinchilla and Gopher), and we do not have additional tests intermediate scales. Furthermore, we assume that the efficient computational frontier can be described by pawer-law relationship between the compute budget, model size, and number of training tokens. However, we observe some concavity log (Nopt) at high compute budgets (see Appendix E). This suggests that we may still be overestimating the optimal size of large models. Finally, the training runs for our analysis have been trained on less than an epoch data; future work may consider the multiple epoch regime. Despite these limitations, the comparison of Chinchilla to Gopher validates our performance predictions, that have thus enabled training better (and more lightweight) model at the same compute budget.

---

Though there has been significant recent work allowing larger and larger models to be trained. our analysis suggests an increased focus on dataset scaling needed. Speculatively, we expect that scaling to larger and larger datasets is only beneficial when the data is high-quality. This calls for responsibly collecting larger datasets with a high focus on dataset quality. Larger datasets will require extra care to ensure train-test set overlap is properly accounted for, both in the language modelling loss but also with downstream tasks. Finally, training for trillions of tokens introduces many ethical and privacy concerns. Large datasets scraped from the web will contain toxic language, biases, and private information. With even larger datasets being used, the quantity (if not the frequency) of such information increases, which makes dataset introspection all the more important. Chinchilla does suffer from bias and toxicity but interestingly it seems less affected than Gopher (see Appendix I).. Better understanding how performance of large language models and toxicity interact is an important future research question.
While we have applied our methodology towards the training of auto-regressive language models, we expect that there a similar trade-off between model size and the amount of data other modalities. As training large models is very expensive, choosing the optimal model size and training steps beforehand is essential. The methods we propose are easy to reproduce in new settings.
Acknowledgments and Disclosure of Funding
We'd like to thank Jean-baptiste Alayrac, Kareem Ayoub, Chris Dyer, Nando de Freitas, Demis Hassabis, Geoffrey Irving, Koray Kavukcuoglu, Nate Kushman and Angeliki Lazaridou for useful comments on the manuscript. We'd like to thank Andy Brock, Irina Higgins, Michela Paganini, Francis Song, and other colleagues at DeepMind for helpful discussions. We are also very grateful to the JAX and XLA team for their support and assistance.
References
[1] M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov. M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasumuru, G. Anantharaman, X. Li, S. Chen, H. Akin, M. Baines, L. Martin, X. Zhou, Koura, B. O'Horo, J. Wang, L. Zettlemoyer, M. Diab, Z. Kozareva, and V. Stoyanov. Efficient Large Scale Language Modeling with Mixtures of Experts. arXiv:21 12. 10684, 2021.
[2] E. M. Bender, Gebru, A. McMillan- Majaar, and Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 2021.
[3] Y. Bisk, R. Zellers, J. Gao, Y. Choi, et al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 2020.
[4] S. Borgeaud, A. Mensch, J. Hoffmann. T. Cai, Rutherford, K. Millican, G. van den Driessche, J.-B. Lespiau, B. Damoc, A. Clark, D. de Las Casas, A. Guy. J. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C. Jones, A. Cassirer, A. Brock, M. Paganini, G. Irving,
Vinyals, S. Osindero, K. Simonyan, J. W. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. Proceedings of the International Conference on Machine Learning, 2021.
[5] J. Bradbury, R. Frostig, P. Hawkins, Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, S. Wanderman-Milne and Q Zhang. JAX: composable transformations of Python+NumPy programs. 2018. httptttp: //gi /gttthb..... com/ com//goolee/ja
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Knueger, Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners, In Advances in Neural Information Processing Systems, 2020.
[7] S. Bubeck. Convex Optimization: Algorithms and Complexity. Foundations and Trends in Machine Learning, 833-4)233-33777 2015.
10

---

[8] A. Chowdhery, S. Narang. J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv:2204.02311, 2022.
[9] A. Clark, D. d. I. Casas, A. Guy, A. Mensch, M. Paganini, J. Hoffmann, B. Damoc, B. Hechtman, T. Cai, S. Borgeaud, G. Driessche, E. Rutherford. T. Hennigan, M. Johnson, K. Millican, A. Cassirer, C. Jones, E. Buchatskaya, D. Budden, L. Sifre, S. Osindero, O. Vinyals, J. Rae, E. Elsen, K. Kavukcuoglu, and K. Simonyan. Unified scaling laws for routed language models. In Proceedings of the International Conference on Machine Learning, 2022.
[10] C. Clark, K. Lee, M.-W. Chang. T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1924-2936, 2019.
[11] N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, Zoph, L. Fedus, M. Bosma, Z. Zhou, T. Wang, Y. E. Wang. K. Webster, M. Pellat, K. Robinson, K. Heiise-el T. Duke, L. Dixon, K. Zhang, Q. V. Le, Y. Wu, Z. Chen, and
C. Cui. GLaM: Efficient scaling of language models with mixture-of-experts 122.9221...06005 2021.
[12] W. Fedus, B. Zoph, and N. Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv:2101.03900 039611, 2021.
[13] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima. S. Presser, and C. Leaby. The Pile: An 800GB dataset of diverse text for language modeling. (XXi::2001.0002. 2020.
[14] S. Gehman, S. Gururangan, M. Sap. Y. Choi, and N. A. Smith. RealToxicityrromps: Evaluating neural toxic degeneration in language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2020.
[15] K. Guu, K. Lee, Z. Tung. P. Pasupat, and M.-W. Chang. REALM: model pre-training In Proceedings of the International Conference on Machine Learning, pages 3929-3938, July 2020.
language
[16] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. w:Xiv:2009.03300 2020.
[17] T. Hennigan, T. Cai, T. Norman, and I. Babuschkin. Haiku: Sonnet for JAX. 2020. URL http://githir . com/ dewpmind/dd//ee/ee/eeeed haiku.
[18] D. Hermandez, J. Kaplan, T. Henighan, and S. McCandlish. Scaling laws for transfer. arXiv:2102.01293, 2021.
[19] Huber. Robust Estimation of Location Parameter. The Annals of Mathematiaal Statistics, 35(1):73-101, 1964.
[20] G. Izacard and E. Grave. Distilling knowledge from reader to retriever for question answering. In International Conference on Learning Representations, 2020.
[21] M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. a7Xiv:1705.0355 00157055..35,,1177 2017.
[22] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia,
N. Boden, A. Borchers, R. Boyle, P.-L. Cantin, C. Chao, C. Clark, Coriell, M. Daley, M. Dau, J. Dean, B. Gelb, T. V. Ghaemmaghami, R. Gottipati, W. Gulland, R. Hagmann, C. R. Ho,
D. Hogberg, J. Hu, R. Hundt, D. Hurt, Ibarz, A. Jaffey, A. Jaworski, A. Kaplan, H. Khaitan,
D. Killebrew, A. Koch, N. Kumar, S. Lacy, J. Laudon, J. Law, D. Le, C. Leary, Z. Liu, K. Lucke, Narayanaswami, R. Ni. K. Nix, T. Norrie, M. Omernick, N. Penukonda, A. Phelps, J. Ross, M. Ross, A. Salek, E. Samadiani, C. Severn, G. Sizikov, M. Snelham, Souter, D. Steinberg, A. Swing, M. Tan, G. Thorson, B. Tian, H. Toma, E. Tuttle, V. Vasudevan, R. Walter, W. Wang, E. Wilcox, and D. H. Yoon. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the Annual International Symposium on Computer Architecture, page 1-12, 2017.
A. Lundin, G. MacKean, A. Maggiore, M. Mahony, K. Miller, R. Nagarajan,
11

---

[23] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, and D. Amodei. Scaling laws for neural language models. Stti:200010886611 2020.
[24] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv: 1412:69012 2014.
[25] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, arXiv: 1808. 06226, 2018.
[26] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, M. Kelcey, J. Devlin, K. Lee, K. N. Toutanova, L. Jones, M.-W. Chang, A. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
[27] G. Lai, Q. Xie, H. Liu, Y. Yang, and Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2017.
[28] Y. Levine, N. Wies, O. Sharir, H. Bata, and A. Shashua. The depth-to sopth-to-width interplay in self-attention. arXiv:2006. 12467, 2020.
[29] P. Lewis. E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih. T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Information Processing Systems, 2020.
[30] O. Lieber. O. Sharir, B. Lenz, and Y. Shoham. Jurassic-1: Technical details and evaluation. White Paper. A121 Labs, 2021.
[31] S. Lin, J. Hilton, and O. Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv:2109.07958 2021.
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[33] S. McCandlish, J. Kaplan, D. Amodei, and O. D. Team. An empirical model of large-batch training. arXi:: 2018.
[34] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. In Interna- tional Conference on Learning Representations, 2017.
[35] M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019.
[36] J. Nocedal. Updating Quasi-Newton Matrices with Limited Storage. Mathematics of Compula- tion, (5(15)):773-782, 1980.
[37] D. Paperno, G. Kruszewski, A. Lazaridou, Q. N. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, and R. Fernández. The LAMBADA dataset: Word prediction requiring broad discourse context. arXiv:1606.0603/ 2016.
[38] J. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, F. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang. A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A. Wu, E. Elsen, S. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre,, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J.-B. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d' Autume, Y. Li, T. Terzi, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, J. Bradbury, M. Johnson, L. Weidinger, Gabriel, W. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training Gopher.
212 2021.
12

---

[39] J. W. Rae, A. Potapenko. S. M. Jayakumar, T. P. Lillicrap, K. Choromanski, V. Likhosherstov, D. Dohan, Song, A. Gane, T. Sarlos, al. Compressive transformers for long-range sequence modelling. Advances in Neural Information Processing Systems, 2020.
[40] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21 21140))1-67, 2020.
[41] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. Zero: Memory optimizations toward training trillion parameter models. In International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
[42] H. Robbins and S. Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 2203):00-407, Sept. 1951.
[43] R. Rudinger, Naradowsky, B. Leonard, and B. Van Durme. Gender bias in coreference resolution. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018.
[44] K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
[45] M. Sap, H. Rashkin, D. Chen, R. LeBras, and Y. Choi. SociallQA: Commonsense reasoning about social interactions. Proceedings of the Conference Empirical Methods in Natural Language Processing, 2019.
[46] C. J. Shallue, J. Lee, J. Antognini, Sohl-Dickstein, R. Frostig, and G. E. Dahl. Measuring the effects of data parallelism on neural network training. /tt1:////003000 2018.
[47] J. W. Siegel and J. Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313-321, 2020.
[48] S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper, Z. Liu, S. Prab- humoye, G. Zerveas, V. Korthikanti, E. Zhang, R. Child, R. Y. Aminabadi, Bernauer, X. Song, M. Shoeybi, Y. He, M. Houston, S. Tiwary, and B. Catanzaro. Using Deepspeed and Megatron to Train Megatron-turing NLG 530b, A Large-Scale Generative Language Model. arXiv:2201.11990 2022.
[49] A. Srivastava et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. mrXiv:2206.06115 2022.
[50] J. Steinhardt. Updates and lessons from Al forecasting, 2021. URL https:// bounded-regret. ghost. wttp:/iioaiiioeeee foreecsito.......g//.......
[51] Y. Tay, M. Dehghani, J. Rao, W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv:2 10686, 2021.
[52] R. Thoppilan, D. D. Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng, A. Jin, T. Bos, L. Baker, Y. Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C.-C. Chang, I. Krivokon, W. Rusch, M. Pickett, K. Meier-Hellstem, M. R. Morris, T. Doshi, R. D. San- tos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. O1- son, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. Chi. and Q. Le. LaMDA: Language models for dialog applications. arXiv:2201 00000.0829,..... 2022.
[53]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
13

---

[54] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane. J. Haas, I.. Rimell, I.. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel. Ethical and social risks of harm from language models, (oXivarrXi::1122044333 2021.
[55] Welbl, A. Glaese, J. Uesato, S. Dathathri, J. Mellor, L. A. Hendricks, K. Anderson, P. Kohli, B. Coppin, and P.-S. Huang. Challenges in detoxifying language models. In Findings of the Association for Computational Linguisties: EMNLP. 2021.
[56] A. Xu, E. Pathak, E. Wallace, Gururangan, M. Sap, and D. Klein. Detoxifying language mod- els risks marginalizing minority voices. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, 2021.
[57] G. Yang, F. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems, 2021.
[58] R. Zellers, A. Holtzman, Y.
Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
[59] G. Zhang, L. Li, Z. Nado, J. Martens, S. Sachdeva, G. Dahl, C. Shallue, and R. B. Grosse. Which algorithmic choices matter at which batch sizes? insights from noisy quadratic model. In Advances in Neural Information Processing Systems, 2019.
[60] B. Zoph, Bello, S. Kumar, N. Du, Y. Huang, J. Dean, N. Shazeer, and W. Fedus. Designing effective sparse expert models. arXiv:2202.08900 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The claims in the abstract describe the work clearly.
(b) Did you describe the limitations of your work? [Yes] We address limitations of our work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have a discussion both in model card and in Appendix I.
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Results are not theoretical.
(b) Did you include complete proofs of all theoretical results? [N/A] Results are not theoretical/no proofs.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as URL)? [No] The code and the data are proprietary. However, for the scaling methodology we provide clear instructions on how to reproduce the results.
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] We provided all training details and hyperparameters.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] Due the cost of training large models, we do have multiple runs. However, the clear & predictable trends suggest the noise is very small. (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] All experiments are ran on TPUv3/TPUv4 and this is stated in the text.
14

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasigg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the preexisting work for all data/models that
(b) Did you mention the license of the assets? [Yes] We use the same data as in Rae et al.
[38] which uses proprietary dataset. We also show results with an open source dataset-
C4
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We do not introduce new assets.
(d) Did you discuss whether and how consent obtained from people whose data you'le ro using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [Yes] We include model card which includes this information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] There
human subjects crowd sourcing in this work. (b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] There was no human subjects or crowd sourcing this work.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] There was human subjects crowd sourcing in this work.
15


Reviews:
[{"review_id": "r6SbeNf2lvt", "reviewer": "Reviewer_QHWQ", "summary": "This paper presents an empirical study to explore the scaling law for training Transformer-based large language models (LLM). In this paper, the scaling law is the trade-off between model size and training tokens, and it is explored by different fitting methods. To obtain the empirical data for fitting the scaling curves, the authors train various language models by varying the model size and the training FLOP counts. The estimated power-law relationship indicates that the optimal scaling way is to increase the size of training data equally while enlarging the model size. To verify the correctness of this optimal scaling law, the authors train the *Chinchilla* model with larger training tokens but decrease the model size to 70B compared to *Gopher* (280B). According to the evaluation results shown in Appendix H, *Chinchilla* outperforms other *Gopher* on most benchmarks while sharing the same FLOPs cost. \n\nThe contributions of this paper are as follows: 1) refining the existing scaling law explored by *Gopher*; 2) revealing an optimal scaling law regarding the training tokens and model size for training LLMs.\n", "questions": "1. In Section 3.1 (Line 130), you mentioned that smoothing and interpolation methods were applied to each training loss curve. I found the details of the smoothing method in Appendix D.1. But which interpolation method did you use in Approach 1?\n2. Have you examined the overlap between pre-training data and downstream fine-tuning data? (especially for the evaluation of language modeling tasks)\n3. You mentioned that current LLMs are under-trained. Is there any metric to evaluate the bottleneck of model capacity instead of empirical methods?\n\n***Update***: The authors have replied to Q1, Q2 with further clarification.", "limitations": "- Societal Impact: The societal impact is positive. For those researchers who work on developing large-scale language models, the authors provide a useful scaling law to save the computational budget. \n- Limitation: The major limitation still lies in the research methodology, the empirical study may not always be solid as I stated in the Weaknesses part.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**:\n1. ***Significance***: The scaling law explored in this empirical study is useful to train the LLM. The authors also claim that the current LLMs are under-trained, the conclusion in this paper may show an interesting direction for this community to keep optimizing the LLMs: we need to pay attention to efficiently learning the data instead of enlarging the model size.\n2. ***Originality***: Although this paper follows the research methodology of a previous study [1], i.e., an empirical study, this paper eventually shows a new scaling law. \n\n**Weakness**:\n1. ***Soundness***: Since this paper uses the empirical method to explore the optimal scaling law, the theoretical foundation is not very solid. How can we judge whether a model is under-trained in a more sound way? Besides, given that the three modeling approaches in this paper rely on empirical training records, one problem is the random factors of training. As the authors claimed in Checklist 3.(c), the costs of training these models are expensive, they did not perform training with different random seeds.\n2. ***Clarity***: This paper is easy to follow and well-written. But there are some typos in the paper:\n    - Line 1: a transformer -> a Transformer\n    - Line 37: budget, Instead, -> budget, instead\n    - Line 71: [24] first -> Kaplan et al. [24] first \n    - Line 74: differs from [24] -> differs from Kaplan et al. [24]\n    - The paper title is different from the information shown in this forum \n\n**Reference**:\n\n[1] Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., ... & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361\n\n***Update***: The authors addressed most of concerns mentioned in Weaknesses part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fIBUHVR_fqc", "reviewer": "Reviewer_x4Sw", "summary": "Kaplan et al. (2020) suggested that large models should not be trained to their lowest possible loss to be compute-optimal: instead, model size should grow faster than the size of the training set, given a fixed computational budget increase. \nThis paper is a more thorough investigation of the question: what are the optimal model size and number of training tokens for a given training budget, tuning additional hyperparameters ignored by prior work? The authors find that model size and training budget should, in fact and in contrast to prior findings, be scaled equally -- since this isn't typically done, many state-of-the-art models are undertrained. They then use this insight to train a new model (which they call Chinchilla), which uses more data but the same compute as a state-of-the-art model and significantly outperforms the latter on multiple downstream tasks. ", "questions": "None.", "limitations": "The authors do explicitly address the limitations of their work in Section 5, which is great. I don't see any direct negative societal impact of the word besides, maybe, the deployment of biased language models, but the authors mention this risk. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper provides experimental results that contradict previous findings and might be quite consequential for the future of large language models. \n- The experiments are quite extensive. (Except for the fact that there are limited complete pretraining runs, but this makes sense given the computational cost.) \n- The paper is clearly written.\n\nWeaknesses:\n- No obvious ones.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XKQ6U26-VNe", "reviewer": "Reviewer_1hyi", "summary": "Current NLP is centered around language modeling. We've observed that their performance can significantly increase when their parameter count is increased and therefore a lot of effort has been focused on scaling that property. \nThe authors here note that the community hasn't really explored the importance of the amount of training data, and so that is the focus here. The authors, through many experiments, discover that when the number of parameters is doubled, so should the amount of training data. \nThe experiments are thorough and the marquee result is incredibly interesting and impactful- with a 70B param model they are able to outperform GPT-3 and other 170B+ parameter models on many challenging downstream tasks including MMLU and some Big Bench tasks. ", "questions": "Are all tokens created equal? I didn't really see a discussion of this in the paper, but when you talk about doubling the training data, is any data going to be OK? Are there specific domains that are more beneficial for downstream tasks? I'm sure 20 papers could be written about this question, I don't think you need to provide all the answers, but it could be beneficial to have a brief discussion of this in the paper.\n\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) Very thorough experiments.\n2) Good writing\n3) An easy-to-understand conclusion that is very straightforward to implement. \n4) The bottom line here (how much data is needed to train big models) is an incredibly important result which will be useful for years to come.\n5) This paper explores a very important but very under-explored topic.\n\nWeaknesses:\n1) No code/models are released, which is really bad for reproducibility. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PmInOr131Z4", "reviewer": "Reviewer_CUYr", "summary": "This paper explores what is the best model size and number or tokens for training Transformer models. They trained over 400 language models with model sizes ranging from 70M to over 16B parameters on 5 to 500 billion tokens, and it suggests the model size and training data should be scaled equally. Based on this finding, they train a 70B parameters model, a predicted compute-optimal model, called Chinchilla, on more training data which outperforms the Gopher 280B model with same compute budget. ", "questions": "I'm wondering how the data quality affects this scale-law. ", "limitations": "None", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is very interesting and inspiring. \n\nIt gives a good guideline when people want to scale up their LMs. \n\nStrong results. \n\n\nWeaknesses:\n\nIt is only tested on autoregressive models. I’m wondering whether this observation is held on BERT-like models.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept