Title: Language models scale reliably with over-training and on downstream tasks
Venue: NeurIPS 2024
Accepted: no
Keywords: large language models

scaling laws

over-training

task prediction
Abstract: Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32$\times$ over-trained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)––each from experiments that take 300$\times$ less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20$\times$ less compute.

Abstract

Scaling laws are useful guides for derisking expensive training runs, as they predict performance of large models using cheaper, small-scale experiments. However, there remain gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., 'Chinchilla optimal' regime). In contrast, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but models are usually compared on downstream task performance. To address both shortcomings, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we fit scaling laws that extrapolate in both the amount of over-training and the number of model parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32 × overtrained) and a 6.9B parameter, 138B token run (i.e., a compute-optimal run)-each from experiments that take 300 × less compute. Second, we relate the perplexity of a language model to its downstream task performance by proposing a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models, using experiments that take 20 × less compute.

1 Introduction

Training large language models is expensive. Furthermore, training high-quality models requires a complex recipe of algorithmic techniques and training data. To reduce the cost of finding successful training recipes, researchers first evaluate ideas with small experiments and then extrapolate their efficacy to larger model and data regimes via scaling laws. With reliable extrapolation, it is possible to quickly iterate at small scale and still pick the method that will perform best for the final large training run. Indeed, this workflow has become commonplace for training state-of-the-art language models like Chinchilla 70B [45], PaLM 540B [19], GPT-4 [76], and many others.

Despite their importance for model development, published scaling laws differ from the goals of 26 training state-of-the-art models in important ways. For instance, scaling studies usually focus on the 27 compute-optimal training regime ('Chinchilla optimality' [45]), where model and dataset size are set 28 to yield minimum loss for a given compute budget. However, this setting ignores inference costs. 29 As larger models are more expensive at inference, it is now common practice to over-train smaller 30 models [113]. Another potential mismatch is that most scaling laws quantify model performance by 31 perplexity in next-token prediction instead of accuracy on widely used benchmark datasets. However, 32 practitioners usually turn to benchmark performance, not loss, to compare models. 33

In this paper, we conduct an extensive set of experiments to address both scaling in the over-trained 34 regime and benchmark performance prediction. 35

10 16

10 17

10 18

10 19

10 20

10 21

10 22

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

Reducible loss: C4 eval

N

=0.011B

N

=0.079B

N

=0.154B

N

=0.411B

N

=1.4B

N

=6.9B

Prediction

Interpolation

Extrapolation

M

=20

M

=320

M

=640

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Loss: C4 eval

0.4

0.5

0.6

0.7

0.8

Average top-1 error: 17-task split

10 22

0.56

0.60

0.64

0.68

0.72

2.4

2.6

0.44

0.46

0.48

0.50

0.52

Motivated by the practice of training beyond compute-optimality, we first investigate whether scaling 36 follows reliable trends in the over-trained regime. We notice, as implied by Hoffmann et al. [45], for a 37 set of models of different sizes trained with a constant ratio of tokens to parameters, models' reducible 38 loss L ′ [43, 45] follows a power law ( L ′ = λ · C -η ) in the amount of training compute C . We 39 find that as one increases the ratio of tokens to parameters, corresponding to more over-training, the 40 scaling exponent η remains about the same, while the scalar λ changes. We explain our observations 41 by reparameterizing existing scaling laws in relation to the amount of over-training. 42

To establish empirically that scaling extrapolates in the over-trained regime, we further experiment 43 with a testbed of 104 models, trained from scratch on three different datasets: C4 [88, 27], 44 RedPajama [112], and RefinedWeb [82]. We find that scaling laws fit to small models can accurately 45 predict the performance of larger models that undergo more over-training. Figure 1 (left) illustrates our 46 main over-training result, where we invest 2 . 4 e 19 FLOPs to extrapolate the C4 validation performance 47 of a 1.4B parameter model trained on 900B tokens, which requires 300 × more compute to train. 48

49

50

51

52

53

54

55

56

In addition to over-training, we also investigate if scaling laws can predict the performance of a model on downstream tasks. We establish a power law relationship between language modeling perplexity and the average top-1 error on a suite of downstream tasks. While it can be difficult to predict the error on individual tasks, we find it possible to predict aggregate performance from a model's perplexity among models trained on the same training data. Figure 1 (right) presents our main downstream error prediction result, where we invest 2 . 7 e 20 FLOPs to predict the average top-1 error over a set of downstream tasks to within 1 percentage point for a 6.9B compute-optimal model, which requires 20 × more compute to train.

57

58

59

60

Our results suggest that the proposed scaling laws are promising to derisk (i) the effects of overtraining models and (ii) the downstream performance of scaling up training recipes. To facilitate further research on reliable scaling, we will release all experiments and models.

2 Developing scaling laws for over-training and downstream tasks

In this section, we develop scaling laws to predict over-trained and downstream performance. First, 61 we provide key definitions (Section 2.1). We next present a scaling law for over-training drawing on 62 empirical observation and prior work (Section 2.2). To connect loss scaling and downstream error 63 prediction, we observe that average top-1 error decreases exponentially as a function of validation loss, 64

10 17

10 19

10 21

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

Reducible loss: C4 eval

Training set: C4

10 17

10 19

10 21

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

Training set: RedPajama

10 17

10 19

10 21

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

Training set: RefinedWeb

N

=0.011B

N

=0.079B

N

=0.154B

N

=0.411B

Interpolation

Extrapolation

10

20

40

80

160

320

640

token multiplier

M

which we formalize as a novel scaling law (Section 2.3). In later sections, we build an experimental 65 setup (Section 3) to quantify the extent to which our scaling laws extrapolate reliably (Section 4). 66

67

68

69

70

71

72

2.1 Preliminaries

Scaling laws for loss. Typically, scaling laws predict model loss L as a function of the compute C in FLOPs used for training. If one increases the number of parameters N in a model or the number of tokens D that a model is trained on, compute requirements naturally increase. Hence, we assume C is a function of N,D . Following Kaplan et al. [51], we use the approximation C = 6 ND , which Hoffmann et al. [45] independently verify. We consider,

where E is an irreducible loss and L ′ is the reducible loss . E captures the Bayes error or minimum 73 possible loss achievable on the validation domain. The L ′ ( C ) term captures what can possibly be 74 learned about the validation domain by training on a source domain. L ′ ( C ) should approach zero 75 with increased training data and model capacity. L ′ ( C ) is often assumed to follow a power law: 76 L ′ ( C ) = λ · C -η (i.a., Hestness et al. [43], OpenAI [76]). It is also often helpful to consider a power 77 law in a log -log plot, where it appears as a line with slope -η and y -intercept log ( λ ) . 78

79

80

81

Token multipliers. We define a token multiplier M = D/N as the ratio of training tokens to model parameters for notational convenience. M allows us to consider fixed relationships between D and N even as a model gets bigger (i.e., as N becomes larger).

82

83

84

85

86

87

88

89

90

91

92

Compute-optimal training. Hoffmann et al. [45] establish compute-optimal training, where, for any compute budget H , the allocation of parameters and tokens is given by,

To solve for the optimal N ∗ , D ∗ , one can sweep N,D for each compute budget, retaining the best configurations. Hoffmann et al. [45] find that as the compute budget increases, N ∗ and D ∗ scale roughly evenly. Assuming equal scaling, there is a fixed compute-optimal token multiplier M ∗ = D ∗ /N ∗ per training distribution.

Over-training. We define over-training as the practice of allocating compute sub-optimally, so smaller models train on a disproportionately large number of tokens (i.e., M > M ∗ ). While loss should be higher than in the compute-optimal allocation for a given training budget, the resulting models have fewer parameters and thus incur less inference cost.

2.2 Scaling laws for over-training

To propose a scaling law for over-trained models, we first turn to empirical observation. We train four 93 model configurations with parameter counts between 0.011B and 0.411B for token multipliers M 94

3

4

5

6

Loss: C4 eval

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Average top-1 error: 17-task split

Training set: C4

3

4

5

6

Loss: C4 eval

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Training set: RedPajama

3

4

5

6

Loss: C4 eval

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Training set: RefinedWeb

Model

Interpolation

Extrapolation

between 20 and 640, where M = 20 points lie roughly on the compute-optimal frontier, and larger 95 M corresponds to more over-training. We defer experimental details to Section 3 to focus on our 96 observations first. In Figure 2, we show loss against compute in a log -log plot for the models trained 97 on three datasets and evaluated on the C4 eval set. We notice parallel lines when fitting power laws to 98 the reducible loss, which suggests a near-constant scaling exponent even with increased over-training. 99 This indicates that scaling behavior should be describable in the amount of over-training. 100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

In search of an analytic expression for the observations in Figure 2, we consider existing scaling literature. A common functional form for the risk of a model, as proposed in prior work [93, 45] is,

Recall from Section 2.1, N is the number of parameters and D the number of training tokens. The constants E,A,α,B,β are fit from data. By fitting this parametric form, Hoffmann et al. [45] find that scaling exponents α and β are roughly equal, suggesting that one should scale N and D equally as compute increases. Hence, we assume α = β . With this assumption, we reparameterize Equation (3) in terms of compute C = 6 ND and a token multiplier M = D/N . We get,

where η = α/ 2 , a = A (1 / 6) -η , b = B (1 / 6) -η gives the relation to Equation (3). For a complete derivation, see Appendix A.

Equation (4) has the following interpretation: (i) The scaling exponent η is not dependent on M . Thus, we always expect lines with the same slope in the log -log plot-as in Figure 2. (ii) The term aM η + bM -η determines the offsets between curves with different token multipliers. Hence, we expect non-overlapping, parallel lines in the log -log plot for the range of M we consider-also consistent with Figure 2.

̸

Recall that we make the assumption α = β , which implies equal scaling of parameters and tokens as more compute is available. However, as explained in Appendix A, even if α = β , we get a parameterization that implies the power-law exponent remains constant with over-training.

2.3 Scaling laws for downstream error

Scaling is typically studied in the context of loss [51, 45, 72], which Schaeffer et al. [100] note is smoother than metrics like accuracy. However, practitioners often use downstream benchmark accuracy as a proxy for model quality and not loss on perplexity evaluation sets. To better connect scaling laws and over-training to task prediction, we revisit the suite of models plotted in Figure 2. In Figure 3, we plot average downstream top-1 errors over evaluations sourced from LLM-Foundry [69] against the C4 eval loss. We defer details of the setup to Section 3 to focus here on a key observation: average error appears to follow exponential decay as loss decreases.

Based on the exponential decay we observe in Figure 3, we propose the following relationship 126 between downstream average top-1 error Err and loss L , 127

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

10 17

10 19

10 21

Compute (6

ND

) [FLOPs]

2

3

4

5

6

Loss: OpenLM eval

Search

10 17

10 19

10 21

Compute (6

ND

) [FLOPs]

Filter

10 17

10 19

10 21

Compute (6

ND

) [FLOPs]

Fit

Grid search models

Selected models

Target 1.4B model

Target 6.9B model

Interpolation

Extrapolation

10 22

1.9

1.8

1.7

where ϵ, k, γ are fit from data. Equation (5) also has an interpretation in terms of model perplexity 128 PP ( L ) = exp ( L ) , 129

Namely, Err follows a power law in PP that is bounded from above by ϵ signifying arbitrarily high error and from below by ϵ -k · exp( -γE ) , where E is the Bayes error from Equation (4).

Equation (5) in conjunction with Equation (4) suggests a three-step method to predict Err as a function of compute and the amount of over-training. For choices of training and validation distributions, (i) fit a scaling law to Equation (4) using triplets of compute C , token multiplier M , and measured loss L on a validation set to yield ( C, M ) ↦→ L . (ii) Fit a scaling law to Equation (5) using pairs of loss L and downstream error Err for models to get L ↦→ Err . (iii) Chain predictions to get ( C, M ) ↦→ Err .

3 Constructing a scaling testbed

In this section, we discuss our experimental setup to test the predictions suggested by Equations (4) and (5). We first present our general language modeling setup (Section 3.1). Next, we discuss our strategy for determining model configurations for our scaling investigation (Section 3.2) and fitting scaling laws (Section 3.3). We then present metrics to validate how well scaling laws predict loss and downstream performance (Section 3.4).

3.1 Training setup

We train transformers [116] for next token prediction, based on architectures like GPT-2 [85] and LLaMA [113]. We employ GPT-NeoX [15] as a standardized tokenizer for all data. See Appendix B for architecture, optimization, and hyperparameter details.

3.2 Model configurations

To get final configurations for the 0.011B to 0.411B parameter models plotted in Figures 2 and 3, we 148 first conduct a wide grid search over a total of 435 models, trained from scratch, from 0.01B to 0.5B 149 parameters (Figure 4 (left) ). We train on the original OpenLM data mix [39], which largely consists 150 of RedPajama [112] and The Pile [31]. While we eventually plan to over-train models, at this step 151 we search for base configurations near compute-optimality. We train on 20 tokens per parameter 152 ( M = 20 ), which, in early experiments, gives models near the compute-optimal frontier. This is 153 similar to findings in Hoffmann et al. [45]'s Table 3, which suggests that M = 20 is near-optimal for 154 the Chinchilla experimental setup. 155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

To find maximally performant small-scale models on validation data, we tune model width, number of layers, number of attention heads, warmup steps, and batch size. Our validation set, OpenLM eval, contains tokens from recent arXiv papers, the OpenLM codebase itself, and news articles. We find in early experiments that qk-LayerNorm makes models less sensitive to learning rate, which is a phenomenon Wortsman et al. [123] report in their Figure 1. Hence, we fix the learning rate (3 e -3) for our sweeps. We also perform smaller grid searches over 1.4B and 6.9B parameter model configurations at M = 20 , retaining the best configurations.

At this point, we have many models, several of which give poor performance; following prior work [51, 45], we want to keep only models that give best performance. Hence, in Figure 4 (center) , we filter out models that do not lie on the Pareto frontier. While there appears to be a general trend, configurations between 5 . 2 × 10 16 and 5 . 2 × 10 17 FLOPs lie below the frontier established by other models. We hypothesize these models over-perform as they are trained for more optimization steps than their neighbors based on our power-of-two batch sizes. We provide support for this hypothesis in Appendix E, but opt to remove these models from our investigation.

To ensure tractable compute requirements for our scaling experiments, we require a subset of models that follows the trend of the entire Pareto frontier. In Figure 4 (right) , we fit trends to the Pareto models and to a subset of four models. We notice that the trends closely predict both the performance of the 1.4B and 6.9B models, suggesting that our small-scale configurations reliably extrapolate in the compute-optimal setting.

̸

Moving forward, we do not tune hyperparameters for other token multipliers (i.e., M = 20 ), on other training or evaluation distributions, or on validation sets for downstream tasks. For more details including specific hyperparameters, see Appendix C.

To create our scaling testbed, we start with the four small-scale, base configurations from our grid search: N ∈ { 0 . 011 B , 0 . 079 B , 0 . 154 B , 0 . 411 B } . To ensure our conclusions are not particular to a single training distribution, we train models on each of C4 [88, 27], RedPajama [112], and RefinedWeb [82], which have 138B, 1.15T, and 600B tokens, respectively, for different token multipliers M ∈ { 5 , 10 , 20 , 40 , 80 , 160 , 320 , 640 } . We omit runs that require more tokens than are present in a dataset (i.e., N = 0 . 411 B , M = 640 for C4). We additionally train N = 1 . 4 B models at M = 20 and at the largest token multiplier possible without repeating tokens (i.e., 80 for C4, 640 for RedPajama, and 320 for RefinedWeb). We train N = 6 . 9 B , M = 20 models on each dataset given the relevance of 7B parameter models [113, 49]. In total this results in a testbed of 104 models.

3.3 Fitting scaling laws

We fit Equation (4) to approximate E,a,b, η using curve-fitting in SciPy [117] (i.e., Levenberg188 Marquardt to minimize non-linear least squares). We repeat this process to fit Equation (5) to 189 approximate ϵ, k, γ . We invest ∼ 100 A100 hours to train the models required to fit a scaling law for 190 loss and ∼ 1,000 A100 hours for a corresponding law for downstream error. Unless otherwise specified, 191 we fit to the N,M pairs in Table 1, which are a subset of our full testbed. Our configurations allow 192 us to test for extrapolation to the N = 1 . 4 B , M = 640 (900B token) and the N = 6 . 9 B , M = 20 193 (138B token) regimes. 194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

10

20

40

80

160

320

640

M

0.011B

0.079B

0.154B

0.411B

1.4B

6.9B

N

1.1% 0.0% 0.2% 0.7% 0.9% 0.0% 0.6%

2.6% 0.3% 0.2% 0.4% 0.1% 0.7% 0.8%

1.5% 0.5% 1.1% 1.1% 3.3% 2.8% 0.6%

0.5% 0.2% 0.0% 2.8% 0.2% 2.0%

0.8%

1.5%

4.3%

Train: C4

Eval: C4 eval

10

20

40

80

160

320

640

M

0.3% 0.0% 0.3% 1.7% 1.1% 0.0% 1.0%

2.2% 0.3% 0.2% 0.7% 1.4% 2.1% 2.3%

0.8% 0.5% 0.6% 0.0% 0.4% 0.4% 0.3%

0.4% 0.2% 0.1% 0.3% 0.3% 1.4% 1.1%

0.1%

0.7%

0.7%

Train: RedPajama

Eval: C4 eval

10

20

40

80

160

320

640

M

0.9% 0.0% 0.9% 1.9% 1.0% 0.0% 1.1%

2.4% 0.1% 0.0% 0.5% 1.2% 2.0% 0.9%

0.9% 0.2% 0.6% 2.8% 2.2% 0.8% 0.9%

0.2% 0.1% 0.5% 0.8% 0.9% 0.9% 0.3%

0.6%

0.0%

1.6%

Train: RefinedWeb

Eval: C4 eval

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

Relative error

3.4 Evaluation setup

Evaluation datasets. Unless otherwise stated, our default validation loss dataset is C4 eval. For downstream tasks, we adopt a subset from 46 tasks from LLM-foundry [69], which includes standard tasks with both zero-shot and few-shot evaluations. Specifically, we consider a 17-task subset where, for each evaluation, at least one 0.154B scale model-trained with as many as 99B tokens-gets 10 percentage points above chance accuracy: ARC-Easy [23], BIG-bench: CS algorithms [11], BIG-bench: Dyck languages [11], BIG-bench: Novel Concepts [11], BIG-bench: Operators [11], BIG-bench: QA WikiData [11], BoolQ [21], Commonsense QA [107], COPA [92], CoQA [91], HellaSwag (zero-shot) [126], HellaSwag (10-shot) [126], LAMBADA [77], PIQA [14], PubMed QA Labeled [50], SQuAD [90], and WinoGrand [55]. For more details on evaluation datasets see Appendix D. We focus on this subset to ensure we are measuring signal, not noise. Including downstream tasks like MMLU [40], where performance is close to random chance, however, does not invalidate our results as we show in our evaluation set ablations (Appendix E).

Metrics. We consider three main metrics: Validation loss , which is the cross entropy between a model's output and the one-hot ground truth token, averaged over all tokens in a sequence and over all sequences in a dataset. Average top-1 error , which is a uniform average over the 17 downstream evaluations, as mentioned in the above paragraph. To measure how good a prediction ζ ( C, M ) is, we measure Relative prediction error : | ζ ( C, M ) -ζ GT | /ζ GT , where ζ is the predicted loss L or the average top-1 error Err . ζ GT is the ground truth measurement to predict.

4 Results: Reliable extrapolation

In this Section, we quantify the extent to which the scaling laws developed in Section 2 extrapolate larger model performance using the scaling testbed from Section 3. By default, we fit Equations (4) and (5) to the configurations in Table 1, use C4 eval for loss, and the 17-task split from Section 3.4 for average top-1 error.

Over-trained performance is predictable. We highlight our main over-training results in Figure 1 (left) . Namely, we are able to extrapolate both in the number of parameters N and the token multiplier M to closely predict the C4 eval performance of a 1.4B parameter model trained on 900B RedPajama tokens ( N = 1 . 4 B , M = 640 ). Our prediction, which takes 300 × less compute to construct than the final 1.4B run, is accurate to within 0.7% relative error. Additionally, for the N = 6 . 9 B , M = 20 run, near compute-optimal, the relative error is also 0.7%.

These results support several key takeaways. (i) Scaling can be predictable even when one increases 225 both the model size and the amount of over-training compared to the training runs used to fit a scaling 226 law. (ii) The form presented in Equation (4) is useful in practice for predicting over-trained scaling 227 behavior. (iii) Fitting to Equation (4) gives good prediction accuracy near compute-optimal. More 228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

specifically, predictions are accurate both for the 1.4B over-trained model and the 6.7B computeoptimal model using a single scaling fit.

While Figure 1 explores a specific case of making predictions in the over-trained regime, we aim to understand the error profile of our predictions across training datasets, token multipliers, and number of parameters. Hence, Figure 5 shows the relative error between ground truth loss and predicted loss on C4 eval for models in our testbed. We notice uniformly low prediction error suggesting that predictions are accurate in many settings.

Average top-1 error is predictable. Figure 1 (right) presents our main result in estimating scaling laws for downstream error. Concretely, we use the models indicated in Table 1 to fit Equations (4) and (5), chaining the scaling fits to predict the average top-1 error as a function of training compute C and the token multiplier M . Our fits allow us to predict, using 20 × less compute, the downstream performance of a 6.9B model trained on 138B RedPajama tokens to within 0 . 05% relative error and a 1.4B model trained on RedPajama 900B tokens to within 3 . 6% relative error.

Table 2 additionally shows the relative error of our downstream performance predictions for models trained on C4, RedPajama, and RefinedWeb, indicating that our scaling law functional forms are applicable on many training datasets. We note that while average accuracy is predictable, individual downstream task predictions are significantly more noisy. We report relative error for more model predictions in Figures 11 and 12. We also find that if we remove the 1.4B model for the Equation (5) fit, relative error jumps, for instance, from 0.05% to 10.64% on the 17-task split for the 6.9B, 138B token RedPajama prediction. This highlights the importance of investing more compute when constructing scaling laws for downstream task prediction compared to loss prediction.

Under-training, out-of-distribution scaling, and compute-reliability trade-offs. In addition to our main results presented above, we include additional results in Appendix E, which we summarize here. First, we notice that when token multipliers become too small (i.e., M = 5 ) scaling becomes unreliable and lies off the trend. Additionally, multipliers other than 20, such as 10, 40, and 80, garner points that are roughly on the compute optimal frontier (Figure 9). This observation suggests that the compute-optimal multiplier may lie in a range rather than take a single value. To probe the limits of reliable scaling, we attempt to break our scaling laws in out-of-distribution settings. We find that models trained on C4-English filtered-and evaluated on next token prediction on code domains have a high relative error in many cases. Perhaps surprisingly, evaluating the same models on German next token prediction gives reliable loss scaling (Figure 10). We additionally examine the compute necessary to create accurate scaling laws, finding that scaling laws can be constructed more cheaply for loss prediction than for downstream error prediction (Figures 15 and 16).

5 Related work

We review the most closely related work in this section. For additional related work, see Appendix F.

Scaling laws. Early works on scaling artificial neural networks observe predictable power-law scaling in the training set size and number of model parameters [43, 44, 93]. Alabdulmohsin et al. [2] stress the importance of looking at the extrapolation regime of a scaling law. Yang et al. [124] prescribe architectural and hyperparameter changes when scaling model width to realize performant models; Yang et al. [125] make analogous recommendations when scaling model depth. Bi et al. [13] propose hyperparameter aware scaling laws. Unlike the aforementioned work, our investigation focuses on over-training and predicting downstream accuracy.

Hoffmann et al. [45] investigate how the number of model parameters N and training tokens D should be chosen to minimize loss L given a compute budget C . Hoffmann et al. [45] find that when scaling up C , both N and D should be scaled equally up to a multiplicative constant (i.e., N ∝ C ∼ 0 . 5

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

and D ∝ C ∼ 0 . 5 ) to realize compute-optimality. Appendix C of the Chinchilla paper additionally suggests that these findings hold across three datasets. However, Hoffmann et al. [45] do not verify their scaling laws for training beyond compute-optimality, or for downstream error prediction-both of which are central to our work.

Sardana & Frankle [98] propose modifications to the Chinchilla formulation to incorporate inference costs into the definition of compute-optimality and solve for various fixed inference budgets. Their key finding, which is critical for our work, is that when taking into account a large enough inference budget, it is optimal to train smaller models for longer than the original Chinchilla recommendations. Our work presupposes that over-training can be beneficial. Instead of solving for inferenceoptimal schemes, we support empirically a predictive theory of scaling in the over-trained regime. Additionally, we provide experiments across many validation and training sets.

For predicting downstream scaling beyond loss, Isik et al. [47] relate the number of pre-training tokens to downstream cross-entropy and machine translation BLEU score [78] after fine-tuning. In contrast, we take a holistic approach to evaluation by looking at top-1 error over many natural language tasks. Schaeffer et al. [100] argue that emergent abilities [120] are a product of non-linear metrics and propose smoother alternatives. As a warmup for why non-linear metrics may be hard to predict, Schaeffer et al. [100] consider predicting an ℓ length sequence exactly: Err ( N,ℓ ) ≈ 1 -PP ( N ) -ℓ , where N is the number of parameters in a model and PP is its perplexity. This is a special case of our Equations (5) and (6), where the number of training tokens does not appear, ϵ = 1 , k = 1 , and γ = ℓ . In contrast, we treat ϵ, k, γ as free parameters for a scaling law fit, finding that average error over downstream tasks can make for a predictable metric.

Over-training in popular models. There has been a rise in over-trained models [113, 114] and accompanying massive datasets [112, 82, 104, 3]. For example, Chinchilla 70B [45] is trained with a token multiplier of 20, while LLaMA-2 7B [114] uses a token multiplier of 290. In our investigation, we look at token multipliers from 5 to 640 to ensure coverage of popular models and relevance for future models that may be trained on even more tokens.

6 Limitations, future work, and conclusion

Limitations and future work. We identify limitations, which provide motivation for future work.

Hyperparameters. While our configurations are surprisingly amenable to reliable scaling across many training and testing distributions without further tuning, there is a need to develop scaling laws that do not require extensive hyperparameter sweeps.

Scaling up. Validating the trends in this paper for even larger runs is a valuable direction. Additionally, repeating our setup for models that achieve non-trivial performance on harder evaluations like MMLU is left to future work.

Scaling down. Actualizing predictable scaling with even cheaper runs is important to make this area of research more accessible, especially for downstream error prediction.

Failure cases. While we present a preliminary analysis of when scaling is unreliable, future work should investigate conditions under which scaling breaks down.

Post-training. It is common to employ fine-tuning interventions after pre-training, which we do not consider. Quantifying to what degree over-training the base model provides benefits after post-training is an open area of research.

Individual downstream task prediction. While we find that averaging over many task error metrics can make for a predictable metric, per-task predictions are left to future work.

In-the-wild performance. Downstream task performance is a proxy for the in-the-wild user experience. Analyzing scaling trends in the context of this experience is timely.

Dataset curation. Our work only deals with existing training datasets. Exploring dataset curation for improved model scaling is another promising direction.

Conclusion. We show that the loss of over-trained models, trained past compute-optimality, is predictable. Furthermore, we propose and validate a scaling law relating loss to average downstream task performance. We hope our work will inspire others to further examine the relationship between model training and downstream generalization. Our testbed will be made publicly available, and we hope it will make scaling research more accessible to researchers and practitioners alike.

References 326

327

328

329

Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. In International Conference on Learning Representations (ICLR) , 2022. https://arxiv.org/abs/2110.02095 .

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Advances in Neural Information Processing Systems (NeuIPS) , 2022. https://arxiv.org/abs/2209.06640 .

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint , 2024. https://arxiv.org/abs/2402. 16827 .

Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! arXiv preprint , 2023. https://arxiv.org/abs/ 2301.03988 .

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Conference of the North American Chapter of the Association for Computational Linguistics (NACCL) , 2019. https://aclanthology.org/N19-1245 .

Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, David Berard, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Laurent Kirsch, Michael Lazos, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Pytorch 2: Faster machine learning through dynamic python bytecode transformation and graph compilation. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) , 2024. https://pytorch.org/blog/pytorch-2-paper-tutorial .

Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficient large scale language modeling with mixtures of experts. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https: //aclanthology.org/2022.emnlp-main.804 .

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint , 2016. https://arxiv.org/abs/1607.06450 .

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint , 2021. https://arxiv.org/abs/2102.06701 .

Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Maxim Krikun, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in nmt: The effect of noise and architecture. In International Conference on Machine Learning (ICML) , 2022. https: //proceedings.mlr.press/v162/bansal22b.html .

BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. In Transactions on Machine Learning Research (TMLR) , 2023. https: //openreview.net/forum?id=uyTL5Bvosj .

Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings ACM conference on fairness, accountability, and transparency (FAccT) , 2021. https://dl.acm. org/doi/10.1145/3442188.3445922 .

DeepSeek-AI Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi 378 Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, 379 Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, 380 Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng 381 Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu 382 Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui 383 Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song, 384 Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu 385 Wang, Yaohui Wang, Yongji Wang, Tong Wu, Yu Wu, Xin Xie, Zhenda Xie, Ziwei Xie, 386 Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang, Yu mei You, Shuiping Yu, 387 Xin yuan Yu, Bo Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, 388 Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, 389 Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek llm: Scaling open-source language 390 models with longtermism. arXiv preprint , 2024. https://arxiv.org/abs/2401.02954 . 391

392

393

394

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Association for the Advancement of Artificial Intelligence (AAAI) , 2020. https://arxiv.org/abs/1911.11641 .

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model. BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models , 2022. https://aclanthology.org/2022.bigscience-1.9 .

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are fewshot learners. In Advances in Neural Information Processing Systems (NeurIPS) , 2020. https://arxiv.org/abs/2005.14165 .

Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In International Conference on Learning Representations (ICLR) , 2023. https://openreview. net/forum?id=sckjveqlCZ .

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. https://arxiv.org/abs/2212.07143 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. In Journal of Machine Learning Research (JMLR) , 2022. https://arxiv.org/abs/2204.02311 .

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint , 2022. https://arxiv.org/abs/2210.11416 .

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https://aclanthology.org/N19-1300 .

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations (ICLR) , 2020. https://openreview.net/pdf? id=r1xMH1BtvB .

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint , 2018. https://arxiv.org/abs/1803.05457 .

Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://arxiv.org/abs/2205.14135 .

Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML) , 2023. https://proceedings.mlr.press/v202/dehghani23a.html .

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https: //aclanthology.org/N19-1423 .

Jesse Dodge, Maarten Sap, Ana Marasovi´ c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.98 .

Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML) , 2022. https://arxiv.org/abs/2112.06905 .

Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint , 2024. https://arxiv. org/abs/2402.01306 .

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Mitchell Wortsman Ryan Marten, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Mehdi Cherti Richard Vencu, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https://arxiv.org/abs/2304.14108 .

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint , 2020. https://arxiv.org/abs/2101.00027 .

Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint , 2021. https://arxiv.org/abs/2109.07740 .

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural 484 machine translation. In Conference on Empirical Methods in Natural Language Processing 485 (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.478 . 486

487

488

489

490

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint , 2024. https://arxiv.org/abs/2402. 00838 .

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint , 2023. https://arxiv.org/abs/2312.00752 .

Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. In Advances in Neural Information Processing Systems (NeurIPS) , 2021. https: //openreview.net/forum?id=yWd42CWN3c .

Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations (ICLR) , 2022. https://arxiv.org/abs/2111.00396 .

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio Cesar, Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. Preprint , 2023. https://www.microsoft.com/en-us/research/publication/ textbooks-are-all-you-need .

Suchin Gururangan, Mitchell Wortsman, Samir Yitzhak Gadre, Achal Dave, Maciej Kilian, Weijia Shi, Jean Mercat, Georgios Smyrnis, Gabriel Ilharco, Matt Jordan, Reinhard Heckel, Alex Dimakis, Ali Farhadi, Vaishaal Shankar, and Ludwig Schmidt. OpenLM: a minimal but performative language modeling (lm) repository, 2023. https://github. com/mlfoundations/open_lm .

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) , 2021. https://arxiv.org/abs/2009. 03300 .

T. J. Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. arXiv preprint , 2020. https://arxiv.org/abs/2010.14701 .

Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint , 2021. https://arxiv.org/abs/2102.01293 .

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Frederick Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint , 2017. https://arxiv.org/abs/1712. 00409 .

Joel Hestness, Newsha Ardalani, and Gregory Diamos. Beyond human-level accuracy: Computational challenges in deep learning. In Principles and Practice of Parallel Programming (PPoPP) , 2019. https://arxiv.org/abs/1909.01736 .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://arxiv.org/abs/2203.15556 .

Hakan Inan, Khashayar Khosravi, and Richard Socher. Tying word vectors and word 533 classifiers: A loss framework for language modeling. In International Conference on Learning 534 Representations (ICLR) , 2017. https://arxiv.org/abs/1611.01462 . 535

536

537

538

Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv , 2024. https://arxiv.org/abs/2402.04177 .

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

Maor Ivgi, Yair Carmon, and Jonathan Berant. Scaling laws under the microscope: Predicting transformer performance from small scale experiments. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https://aclanthology.org/2022. findings-emnlp.544 .

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Florian Bressand Diego de las Casas, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint , 2023. https://arxiv.org/abs/2310.06825 .

Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019. https://aclanthology.org/D19-1259 .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint , 2020. https://arxiv.org/abs/2001.08361 .

Tobit Klug, Dogukan Atik, and Reinhard Heckel. Analyzing the sample complexity of selfsupervised image reconstruction methods. arXiv preprint , 2023. https://arxiv.org/abs/ 2305.19079 .

Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint , 2019. http://arxiv.org/abs/1909.11942 .

Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, and Daniel Haziza. xformers: A modular and hackable transformer modelling library, 2022. https: //github.com/facebookresearch/xformers .

Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In International conference on the principles of knowledge representation and reasoning , 2012. https://aaai.org/papers/59-4492-the-winograd-schema-challenge .

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-tosequence pre-training for natural language generation, translation, and comprehension. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2020. https: //aclanthology.org/2020.acl-main.703 .

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint , 2023. https://arxiv.org/abs/2305.06161 .

Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In International Joint Conference on Artificial Intelligence , 2020. https://arxiv.org/abs/2007.08124 .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint , 2019. http://arxiv.org/abs/1907.11692 .

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. https://arxiv.org/abs/2201.03545 .

Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. arXiv preprint , 2023. https://arxiv.org/abs/2310.16787 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint , 2017. https://arxiv.org/abs/1711.05101 .

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Marc Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder 2 and the stack v2: The next generation. arXiv preprint , 2024. https://arxiv.org/abs/2402.19173 .

Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, HannaMari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. Fingpt: Large generative models for a small language. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/2023. emnlp-main.164 .

Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneneh Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. arXiv preprint , 2023. https://paloma.allen.ai .

Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. In Computational Linguistics , 1993. https://aclanthology.org/J93-2004 .

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021. https://aclanthology.org/2021.emnlp-main.133 .

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. https://arxiv.org/abs/1809. 02789 .

MosaicML. Llm evaluation scores, 2023. https://www.mosaicml.com/llm-evaluation .

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2022. https://aclanthology.org/ 2023.acl-long.891 .

Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue 629 Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: 630 Instruction tuning code large language models. arXiv preprint , 2023. https://arxiv.org/ 631 abs/2308.07124 . 632

Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, 633 Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained 634 language models. In Advances in Neural Information Processing Systems (NeuIPS) , 2023. 635 https://arxiv.org/abs/2305.16264 . 636

637

638

639

Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint , 2024. https://arxiv.org/abs/2402.09906 .

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, Senthil Purushwalkam, Tong Niu, Wojciech Kryscinski, Lidiya Murakhovs'ka, Prafulla Kumar Choubey, Alex Fabbri, Ye Liu, Rui Meng, Lifu Tu, Meghana Bhat, Chien-Sheng Wu, Silvio Savarese, Yingbo Zhou, Shafiq Rayhan Joty, and Caiming Xiong. Long sequence modeling with xgen: A 7b llm trained on 8k input sequence length. arXiv preprint , 2023. https://arxiv.org/abs/2309.03450 .

OpenAI. Triton, 2021. https://github.com/openai/triton .

OpenAI. Gpt-4 technical report, 2023. https://arxiv.org/abs/2303.08774 .

Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2016. http://www.aclweb.org/ anthology/P16-1144 .

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2002. https://aclanthology.org/P02-1040 .

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Annual Meeting of the Association for Computational Linguistics (ACL) , 2022. https://aclanthology.org/2022.findings-acl.165 .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS) , 2019. https://arxiv.org/abs/1912.01703 .

Patronus AI. EnterprisePII dataset, 2023. https://tinyurl.com/2r5x9bst .

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint , 2023. https://arxiv.org/abs/2306.01116 .

Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Wo´ zniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/2023.findings-emnlp.936 .

Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the Conference of the European Chapter of the Association for Computational Linguistics (EACL) , 2017. https://aclanthology.org/E17-2025 .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Preprint , 2019. https://d4mucfpksywv.cloudfront.net/better-language-models/language_ models_are_unsupervised_multitask_learners.pdf .

Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis 684 Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, 685 Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, 686 Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, 687 Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, 688 Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, 689 Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, 690 L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena 691 Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, 692 Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias 693 Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, 694 Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris 695 Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason 696 Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol 697 Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, 698 and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training 699 gopher. arXiv preprint , 2021. https://arxiv.org/abs/2112.11446 . 700

701

702

703

704

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https: //arxiv.org/abs/2305.18290 .

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint , 2019. https://arxiv.org/abs/1910.10683 .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In The Journal of Machine Learning Research (JMLR) , 2020. https: //arxiv.org/abs/1910.10683 .

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2016. https://aclanthology.org/D16-1264 .

Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. In Transactions of the Association for Computational Linguistics (TACL) , 2019. https://aclanthology.org/Q19-1016 .

Melissa Roemmele, Cosmin Adrian Bejan, , and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium , 2011. https://people. ict.usc.edu/~gordon/copa.html .

Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations (ICLR) , 2020. https://arxiv.org/abs/1909.12673 .

Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2018. https://aclanthology.org/N18-2002 .

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. arXiv preprint , 2019. https://arxiv.org/ abs/1907.10641 .

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled 731 version of bert: smaller, faster, cheaper and lighter. arXiv preprint , 2019. http://arxiv. 732 org/abs/1910.01108 . 733

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: 734 Commonsense reasoning about social interactions. In Empirical Methods in Natural Language 735 Processing (EMNLP) , 2019. https://aclanthology.org/D19-1454 . 736

737

738

739

Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. In NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP) , 2023. https://arxiv.org/abs/2401.00448 .

740

741

742

743

744

745

746

747

Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one million gpu hours? In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2022. https://aclanthology.org/2022. findings-emnlp.54 .

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems (NeurIPS) , 2023. https://arxiv.org/abs/2304.15004 .

Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data 748 manifold. In Journal of Machine Learning Research (JMLR) , 2022. https://arxiv.org/ 749 abs/2004.10802 . 750

Noam Shazeer. Glu variants improve transformer. arXiv preprint , 2020. https://arxiv. 751 org/abs/2002.05202 . 752

Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F Karlsson, Abinaya Mahendiran, 753 Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, et al. 754 Aya dataset: An open-access collection for multilingual instruction tuning. arXiv preprint 755 arXiv:2402.06619 , 2024. https://arxiv.org/abs/2402.06619 . 756

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell 757 Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open 758 corpus of three trillion tokens for language model pretraining research. arXiv preprint , 2024. 759 https://arxiv.org/abs/2402.00159 . 760

761

762

763

764

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://openreview.net/forum?id= UmvSlP-PyV .

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint , 2021. https:// arxiv.org/abs/2104.09864 .

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019. https://aclanthology.org/N19-1421 .

Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. In International Conference on Learning Representations (ICLR) , 2022. https://openreview.net/forum? id=f2OYVDyfIB .

Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2023. https://aclanthology.org/ 2023.findings-emnlp.825 .

MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially 782 usable llms, 2023. www.mosaicml.com/blog/mpt-7b . 783

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, 784 Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, 785 Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, 786 Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, 787 Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, 788 Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel 789 Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, 790 Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, 791 Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, 792 Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera793 Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog 794 applications. arXiv preprint , 2022. https://arxiv.org/abs/2201.08239 . 795

796

797

Together Computer. Redpajama: an open dataset for training large language models, 2023. https://github.com/togethercomputer/RedPajama-Data .

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint , 2023. https://arxiv.org/abs/ 2302.13971 .

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv preprint , 2023. https://arxiv.org/abs/2307.09288 .

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint , 2024. https://arxiv.org/abs/2402.07827 .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS) , 2017. https://arxiv.org/abs/1706.03762 .

Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, ˙ Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 2020. https://rdcu.be/b08Wh .

Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From lsat: The progress and challenges of complex reasoning. Transactions on Audio, Speech, and Language Processing , 2021. https://arxiv.org/abs/2108.00648 .

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

International Conference on Learning Representations (ICLR) , 2022. https://openreview. net/forum?id=gEZrGCozdqR .

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In Transactions on Machine Learning Research (TMLR) , 2022. https: //openreview.net/forum?id=yzkSU5zdwD .

Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint , 2021. https://arxiv.org/abs/2112.04359 .

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili´ c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint , 2022. https://arxiv.org/abs/2211.05100 .

Mitchell Wortsman, Peter J Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint , 2023. https://arxiv.org/ abs/2309.14322 .

Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. In Advances in Neural Information Processing Systems (NeuIPS) , 2021. https://arxiv.org/abs/2203.03466 .

Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou. Feature learning in infinite depth neural networks. In International Conference on Learning Representations (ICLR) , 2024. https://openreview.net/forum?id=17pVDnpwwl .

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Annual Meeting of the Association for Computational Linguistics (ACL) , 2019. https://aclanthology.org/P19-1472 .

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. https://arxiv.org/abs/2106.04560 .

Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems (NeuIPS) , 2019. https://arxiv.org/abs/1910.07467 .

Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initialization and merged attention. In Empirical Methods in Natural Language Processing (EMNLP) , 2019. https://aclanthology.org/D19-1083 .

Yanli Zhao, Andrew Gu, Rohan Varma, Liangchen Luo, Chien chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel. In Very Large Data Bases Conference (VLDB) , 2023. https: //dl.acm.org/doi/10.14778/3611540.3611569 .

Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. Jec-qa: A legal-domain question answering dataset. In Association for the Advancement of Artificial Intelligence (AAAI) , 2020. https://arxiv.org/abs/1911.12011 .

Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint , 2023. https://arxiv.org/abs/2304.06364 .

Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, 884 Qian Liu, and Niklas Muennighoff. Astraios: Parameter-efficient instruction tuning code large 885 language models. arXiv preprint , 2024. https://arxiv.org/abs/2401.00788 . 886

Contents 887

909

910

911

912

913

914

915

916

917

918

919

920

921

A Scaling-law derivations

We first show that reparameterizing Equation (3) in terms of the compute C and token multiplier M for α = β yields Equation (4). Combining C = 6 ND and M = D/N yields N = √ C/ (6 M ) and D = √ CM/ 6 . Inserting these into Equation (3) yields,

This is equal to Equation (4), making the substitutions η = α/ 2 , a = A (1 / 6) -η , b = B (1 / 6) -η , as noted in the main body.

̸

Relation to compute-optimal training. Recall that we made the assumption α = β , which implies equal scaling of parameters and tokens to realize compute-optimal models. While this assumption is empirically justified [45], even if α = β , we get a parameterization that implies the power law exponent in Equation (4) remains constant with over-training, while the power law scalar changes.

To find a compute-optimal training setting, Hoffmann et al. [45] propose to minimize the right-hand side of Equation (3) subject to the compute constraint C = 6 ND . This yields, N ∗ = γ 1 α + β ( C/ 6) β α + β and D ∗ = γ -1 α + β ( C/ 6) α α + β , where γ = αA βB , for notational convenience. The associated risk is,

We now deviate from compute-optimal training by modifying the model size and tokens by 922 multiplication with a constant √ m , according to 923

This modification keeps the compute constant (i.e., 6 N m D m = 6 N ∗ D ∗ ). The risk, then, becomes 924

̸

We again expect the same power law exponent and changing power law scalar. Note that m in 925 Equation (8) is similar to M in Equation (4). Specifically, m is a multiple of the Chinchilla-optimal 926 token multiplier M ∗ = D ∗ /N ∗ , which is no longer fixed as a compute budget changes for α = β . 927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

B Additional training details

Architecture. As stated in the main paper, we train transformers [116], based on autoregressive, decoder-only, pre-normalization architectures like GPT-2 [85] and LLaMA [113]. We adopt OpenLM [39] for modeling, which utilizes PyTorch [80, 6], xformers [54], triton [75], FlashAttention [24], FSDP [130], and bfloat16 automatic mixed precision. Like LLaMA, we omit bias terms, but replace RMSNorm [128] with LayerNorm [8], which has readily available fused implementations. Following Wortsman et al. [123], we apply qk-LayerNorm [25], which adds robustness to otherwise poor hyperparameter choices (e.g., learning rate). We use SwiGLU [102] activations and depth-scaled initialization [129]. We use a sequence length of 2,048, rotary positional embeddings [106], and the GPT-NeoX-20B tokenizer [15], which yields a vocabulary size of 50k. We do not use weight tying [84, 46]. We sample without replacement during training and employ sequence packing without attention masking. We separate documents in our training corpora with end-of-text tokens.

Objectives and optimization. We train with a standard causal language modeling objective (i.e., next token prediction) with an additive z-loss [19] (coefficient 1 e -4), which mitigates output logit norm growth [67] instabilities. We use the AdamW optimizer [62] (PyTorch defaults except beta2 = 0.95), with independent weight decay [123] (coefficient 1 e -4). For the learning rate schedule, we use linear warmup and cosine decay. We cool down to a low learning rate (3 e -5).

C Additional grid search details

Final model configurations. We present our final hyperparameters in Table 3.

Grid search configuration selection. Recall in Section 3.3, we run a grid search over many configurations. We present the architectures we sweep over in Table 4.

D Evaluation dataset details

All 46 downstream evaluations are based on MosaicML's LLM-foundry evaluation suite [69]. We specifically consider the datasets given in Table 5. Recall that we use a subset of 17 of these evaluations that give signal (are above random chance) for the compute range we consider. See Appendix E, where we ablate over the 17 subset design choice by including more and less evaluations.

E Additional results

Scaling law fits. We present specific coefficients for our fits in Table 6.

Small-scale experiments can predict model rank order. We expect to be able to rank hypothetical models based on their predicted performance, which is useful when deciding what large-scale runs

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

10 16

10 17

10 18

10 19

Compute (6

ND

) [FLOPs]

3

4

5

Loss: OpenLM eval

10 16

10 17

10 18

10 19

Compute (6

ND

) [FLOPs]

3

4

5

Grid search models

1000

2000

3000

4000

5000

6000

7000

Number of optimization steps

to train. To verify, we rank 9 testbed models with N ≥ 1 . 4 B by ground-truth top-1 error and by estimated top-1 error. We find high rank correlation of 0.88 for the 17-task split.

Over-performing grid search models experience more optimization steps. As mentioned in Section 3.3 and Figure 4, we notice that models between 0.011B to 0.079B (i.e., 5 . 2 × 10 16 to 5 . 2 × 10 17 FLOPs trained near compute-optimal) over-perform compared to the trend established by other models in our initial grid searches. This results in a bump in the scaling plot. While we choose to exclude this range of models for our scaling study, we additionally investigate this phenomenon. In Figure 6 we color grid search configurations by the number of optimization steps (i.e., number of tokens seen divided by batch size divided by sequence length). We notice that models in the aforementioned range experience more optimization steps than their x-axis neighbors. For context, Figure 1 (left) in Kaplan et al. [51] also shows a bump; however, there the performance is worse than the general trend instead of better as in our work. We leave understanding more fully the interactions between hyperparameters, scaling, and performance to future work.

Scaling is largely predictable in-distribution (ID). Prior work focuses on understanding scaling using ID loss, often using training loss directly [51, 45]. Hence, we also consider Paloma [65] loss evaluation sets, which are designed to probe performance in specific domains. We use Paloma's C4 [88, 27], RedPajama [112], and Falcon-RefinedWeb [82] splits to probe for ID loss. As seen in Figure 7, relative error is mostly low. Relative error is largest for the N = 1 . 4 B , M = 640 RedPajama run at 15.4%. Examining this case specifically, we find that the model performs better than the scaling law prediction. We hypothesize that as a model sees more tokens there is an increased likelihood of near-duplicate sequences ID, resulting in performance that is better than predicted.

Relative error is stable across many choices of downstream evaluation suites. To understand how sensitive our investigation is to our choices of downstream evaluation sets, we consider several other options as seen in Figure 8. We find that our prediction errors are fairly (i) low and (ii) consistent for many choices of downstream evaluation sets including the whole suite of 46 evaluations.

Scaling can break down when under-training. We find that when a token multiple is too small (i.e., under-training regime), scaling appears unreliable. In Figure 9 we see for M = 5 the scaling trend is different. We hypothesize that tuning hyperparameters (e.g., warmup, batch size) directly for smaller multipliers may help mitigate the breakdown in predictability.

10

20

40

80

160

320

640

M

0.011B

0.079B

0.154B

0.411B

1.4B

6.9B

N

1.1% 0.0% 0.1% 0.7% 0.9% 0.0% 0.6%

2.6% 0.3% 0.2% 0.8% 0.2% 0.3% 1.4%

2.0% 0.5% 1.3% 1.0% 3.9% 3.5% 1.3%

0.2% 0.2% 0.2% 3.5% 0.5% 2.7%

0.4%

2.3%

3.6%

Train: C4

Eval: C4 (Paloma split)

10

20

40

80

160

320

640

M

4.6% 0.0% 0.3% 2.8% 1.5% 0.0% 0.8%

0.5% 0.0% 1.1% 2.2% 3.0% 3.5% 3.3%

0.2% 0.0% 0.5% 1.3% 1.4% 1.0% 0.5%

0.1% 0.0% 0.2% 0.3% 0.7%10.0% 10.3%

3.0%

15.4%

10.3%

Train: RedPajama

Eval: RedPajama (Paloma split)

10

20

40

80

160

320

640

M

1.1% 0.0% 0.9% 1.6% 0.8% 0.0% 1.1%

2.3% 0.1% 0.4% 1.0% 2.0% 3.2% 2.3%

1.1% 0.2% 0.1% 1.5% 0.8% 2.0% 2.4%

0.7% 0.1% 1.6% 2.3% 2.0% 2.3% 4.3%

1.4%

6.0%

5.6%

Train: RefinedWeb

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

Relative error

0

10

20

30

40

50

Inclusion threshold

t

(i.e., include evals where any model gets

t

percentage points above random chance at 0.154B scales)

10

2

10

1

Relative prediction error

0

10

20

30

40

Number of excluded datasets (out of 46-total)

10

2

10

1

C4

RedPajama

RefinedWeb

10 16

10 18

10 20

Compute (6

ND

,

D

=

MN

) [FLOPs]

2

3

4

5

6

Reducible loss: C4 eval

Training set: C4

10 16

10 18

10 20

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

6

Training set: RedPajama

10 16

10 18

10 20

Compute (6

ND

,

D

=

MN

) [FLOPs]

1

2

3

4

5

Training set: RefinedWeb

N

=0.011B

N

=0.079B

N

=0.154B

N

=0.411B

Interpolation

Extrapolation

5

10

20

40

80

token multiplier

M

988

989

990

991

992

993

994

995

996

997

998

999

1000

10

20

40

80

160

320

640

M

0.011B

0.079B

0.154B

0.411B

1.4B

6.9B

N

15.2%0.0% 8.4% 2.9% 0.2% 0.0% 0.2%

2.1% 0.3% 1.7% 0.2% 1.8% 3.0% 9.7%

3.3% 0.6% 0.7% 1.6% 6.4% 4.3% 2.8%

0.5% 0.3% 0.4% 9.5% 4.1%22.4%

2.5%

4.3%

3.3%

Train: C4

Eval: 100 programming languages

(Paloma split)

10

20

40

80

160

320

640

M

5.7% 0.0% 3.6% 1.1% 1.4% 0.0% 3.5%

0.0% 0.4% 1.6% 2.1% 3.0% 5.1%24.3%

0.9% 0.7% 0.7% 1.6% 1.3% 4.7% 2.0%

0.7% 0.3% 0.1% 0.5% 0.7% 3.9%

9.9%

10.0%

9.8%

Train: C4

Eval: Penn Tree Bank

(Paloma split)

10

20

40

80

160

320

640

M

4.1% 0.0% 2.1% 0.6% 1.5% 0.0% 1.2%

1.5% 0.1% 0.5% 0.2% 0.2% 1.1% 1.7%

2.3% 0.1% 0.6% 0.0% 2.2% 2.2% 0.7%

0.5% 0.0% 0.0% 2.8% 0.2% 3.1%

0.9%

7.6%

3.4%

Train: C4

Eval: C4 German eval

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

Relative error

10

20

40

80

160

320

640

M

0.011B

0.079B

0.154B

0.411B

1.4B

6.9B

N

0.3% 0.2% 0.6% 0.9% 0.2% 0.3% 1.3%

1.2% 0.4% 1.0% 0.1% 0.3% 0.3% 0.0%

0.2% 0.7% 0.5% 1.2% 1.7% 1.0% 0.4%

0.6% 0.2% 0.0% 1.6% 0.0% 0.4%

0.3%

3.1%

0.1%

Train: C4,

Downstream: 46-task split

10

20

40

80

160

320

640

M

0.1% 0.4% 1.1% 1.3% 0.2% 0.6% 0.4%

0.1% 0.0% 0.3% 0.8% 1.0% 0.6% 1.3%

0.2% 0.2% 0.0% 0.6% 0.9% 0.9% 0.3%

1.3% 0.8% 1.3% 1.5% 1.0% 1.0% 1.0%

0.3%

3.4%

2.1%

Train: RedPajama,

Downstream: 46-task split

10

20

40

80

160

320

640

M

1.2% 0.1% 0.1% 0.7% 0.3% 0.8% 0.6%

0.5% 1.4% 0.4% 0.9% 0.8% 1.0% 1.2%

0.8% 0.1% 0.6% 0.3% 1.1% 0.7% 0.9%

0.5% 1.1% 1.1% 1.7% 1.6% 0.6% 0.9%

0.3%

4.3%

2.8%

Train: RefinedWeb,

Downstream: 46-task split

0%

2%

4%

6%

8%

10%

Relative error

Scaling can be unpredictable out-of-distribution (OOD). Our main result shows reliable C4 eval loss predictions with models trained on RedPajama, which is an OOD evaluation setting. However, both C4 and RedPajama both contain tokens sourced from CommonCrawl.

To further probe OOD performance, we measure the relative error of scaling laws fit to models trained on C4 and evaluated on Paloma's 100 programming languages [65], Paloma's Penn Tree Bank (PTB) split [66], and a German version of C4 [27]. Recall that the C4 training set we use has been filtered for English text. Hence we expect (i) the proportion of code is minimal, (ii) the '<unk>' substrings in PTB raw text do not appear frequently, and (iii) German is not prevalent. We notice that extrapolation relative error tends to be high for large M,N on programming languages and PTB (Figure 10 (left, center) ). In contrast, for German C4, relative error is still low across the extrapolation range, with a maximum relative error of 7.6% at the N = 1.4B, M = 80 scale (Figure 10 (right) ). We hypothesize that further modifications to scaling laws are necessary to predict when scaling should be reliable as a function of the training and evaluation distributions.

Small-scale experiments can predict average downstream top-1 error. To verify that chaining 1001 Equations (4) and (5) is effective in practice, we collect C4 eval loss and downstream error pairs for 1002 the configurations in Table 1. In Figure 11, we look at relative error for our scaling predictions in the 1003 context of Average top-1 error over 46 evals and in Figure 12 over the high-signal 17 eval subset. We 1004 again notice reliable scaling in interpolation and extrapolation regimes, suggesting the validity of our 1005 procedure to predict downstream average top-1 error. 1006

10

20

40

80

160

320

640

M

0.011B

0.079B

0.154B

0.411B

1.4B

6.9B

N

1.2% 0.7% 1.2% 2.3% 1.1% 1.1% 3.1%

3.9% 1.2% 2.5% 1.0% 0.5% 0.6% 0.4%

1.3% 2.9% 1.1% 2.2% 3.9% 1.4% 0.2%

2.3% 1.3% 1.7% 2.8% 2.2% 1.8%

0.7%

9.6%

0.1%

Train: C4,

Downstream: 17-task split

10

20

40

80

160

320

640

M

0.6% 0.4% 2.1% 2.5% 0.6% 0.8% 0.8%

0.8% 0.7% 0.2% 1.3% 0.8% 0.5% 1.3%

0.9% 1.7% 0.2% 0.1% 0.2% 0.7% 0.0%

2.6% 2.0% 1.0% 2.6% 1.9% 3.4% 3.4%

0.7%

3.6%

0.0%

Train: RedPajama,

Downstream: 17-task split

10

20

40

80

160

320

640

M

2.5% 0.4% 0.7% 1.5% 1.3% 1.7% 0.9%

1.0% 2.7% 0.4% 0.4% 0.9% 0.3% 2.3%

1.5% 0.1% 0.0% 1.3% 2.1% 0.6% 0.6%

0.8% 2.2% 2.0% 3.6% 3.4% 1.2% 1.6%

0.4%

5.6%

2.9%

Train: RefinedWeb,

Downstream: 17-task split

0%

2%

4%

6%

8%

10%

Relative error

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Loss: C4

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

Average top-1 error: 17-task split

2

3

4

5

6

7

8

Loss: RedPajama

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

3

4

5

6

Loss: RefinedWeb

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

C4

RedPajama

RefinedWeb

Interpolation

Extrapolation

2.5

3.0

3.5

4.0

4.5

5.0

5.5

6.0

Loss: C4

0.66

0.68

0.70

0.72

0.74

0.76

Average top-1 error: 46-task split

2

3

4

5

6

7

8

Loss: RedPajama

0.66

0.68

0.70

0.72

0.74

0.76

3

4

5

6

Loss: RefinedWeb

0.66

0.68

0.70

0.72

0.74

0.76

C4

RedPajama

RefinedWeb

Interpolation

Extrapolation

Loss evaluation ablations for downstream trends. Figure 13 presents the correlation between 1007 downstream error and loss evaluated on different validation sets (C4, RedPajama, and RefinedWeb). 1008 Regardless of the validation set (x-axis), models follow the exponential decay relationship given 1009 in Equation (5), suggesting the choice of validation loss is not critical for the appearance of this 1010 phenomenon. 1011

Investing more compute in a scaling law makes it more predictive. Thus far we have looked 1012 at standard configurations from Table 1 to construct our scaling laws, mainly to demonstrate 1013 extrapolation to larger N,M . However, for practitioners, the main constraint is often training 1014

10 18

10 19

10 20

10 21

Compute [FLOPs] used for the scaling fit

10

4

10

3

10

2

10

1

10 0

Relative error: C4 eval

5

10

15

20

25

30

Number of samples used for the scaling fit

Trend

Individual estimates

Default setting from Table 2

10 18

10 19

10 20

10 21

Compute [FLOPs] used for the scaling fit

10

4

10

3

10

2

10

1

10 0

Relative error: C4 eval

10 18

10 19

10 20

10 21

Compute [FLOPs] used for the scaling fit

Relative error: 17-task split

Trend

Default setting from Table 2

Individual estimates

compute. Hence, we wish to understand the trade-offs between the amount of compute invested 1015 in creating a scaling law and the relative error of the resulting law in the over-trained regime. In 1016 Figure 14 (left) , we see that as one increases the amount of compute, it is possible to get better fits 1017 with lower relative error. In Figure 14 (right) , we see a similar trend as one increases the number of 1018 data points used to fit a scaling law. Blue stars indicate the configurations from Table 1, which provide 1019 accurate predictions relative to the general trends-hinting at their usefulness for our investigation. 1020 In Figures 15 and 16 we repeat the compute analysis comparing trade-offs for loss prediction and 1021 error prediction for our RedPajama 1.4B parameter, 900B token and 6.9B parameter, 138B token 1022 runs respectively. We find that less compute is generally necessary to construct a loss scaling law that 1023 achieves the same relative error as that of an error prediction scaling law. 1024

10 18

10 19

10 20

10 21

Compute [FLOPs] used for the scaling fit

10

4

10

3

10

2

10

1

10 0

Relative error: C4 eval

10 18

10 19

10 20

10 21

Compute [FLOPs] used for the scaling fit

Relative error: 17-task split

Trend

Default setting from Table 2

Individual estimates

20

40

80

160

320

640

Token multiplier

M

0.11

0.12

0.13

0.14

0.15

Scaling exponent

C4

RedPajama

RefinedWeb

Trend

On compute-optimal token multipliers. We consider 20 tokens per parameter as close to compute1025 optimal for our experiments. Here we investigate, using different approaches, what the compute1026 optimal token multipliers are for each dataset-assuming one should scale number of parameter and 1027 training tokens equally as Hoffmann et al. [45] suggest. 1028

1029

1030

1031

Turning to Figure 9, we notice that there are many multipliers, between 10 and 80 that yield models close to the frontier. Hence, empirically, it appears choices within this range should be suitable for the optimal token multiplier.

1032

1033

Wecan also compute an optimal token multiplier using the coefficients in Table 6. Based on Hoffmann et al. [45]'s Equation (4) and the assumption that α = β , we write,

To compute M ∗ = D ∗ /N ∗ , we then have, 1034

Using the values from Table 6 and plugging into Equation (10), we find M ∗ C4 = 2 . 87 , M ∗ RedPajama = 1035 4 . 30 , M ∗ RefinedWeb = 3 . 79 , where the subscript gives the dataset name. These values conflict with the 1036 observation in Figure 9, which suggests M = 5 is already too small to give points on the Pareto 1037 frontier. We hypothesize this mismatch arises because we fit our scaling laws using models with 1038 M ≥ 20 . 1039

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.72

0.74

0.76

0.78

0.80

0.82

0.84

Top-1 error: AGIEval LSAT AR

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.71

0.72

0.73

0.74

0.75

0.76

0.77

0.78

Top-1 error: AGIEval LSAT LR

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.72

0.74

0.76

0.78

Top-1 error: AGIEval LSAT RC

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.70

0.72

0.74

0.76

0.78

0.80

Top-1 error: AGIEval SAT English

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.625

0.650

0.675

0.700

0.725

0.750

0.775

0.800

Top-1 error: ARC-Challenge

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.3

0.4

0.5

0.6

0.7

Top-1 error: ARC-Easy

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.46

0.48

0.50

0.52

0.54

0.56

0.58

0.60

Top-1 error: BBQ

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.675

0.700

0.725

0.750

0.775

0.800

0.825

Top-1 error: BIG-bench: Conceptual combinations

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.96

0.97

0.98

0.99

1.00

Top-1 error: BIG-bench: Conlang translation

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.6

0.7

0.8

0.9

1.0

Top-1 error: BIG-bench: CS algorithms

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Top-1 error: BIG-bench: Dyck languages

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.73

0.74

0.75

0.76

0.77

Top-1 error: BIG-bench: Elementary math QA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.740

0.745

0.750

0.755

0.760

Top-1 error: BIG-bench: Language identification

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.72

0.73

0.74

0.75

0.76

0.77

Top-1 error: BIG-bench: Logical deduction

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.44

0.46

0.48

0.50

0.52

0.54

0.56

Top-1 error: BIG-bench: Misconceptions

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

Top-1 error: BIG-bench: Novel Concepts

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.80

0.85

0.90

0.95

1.00

Top-1 error: BIG-bench: Operators

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Top-1 error: BIG-bench: QA WikiData

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.84

0.86

0.88

0.90

0.92

0.94

0.96

0.98

1.00

Top-1 error: BIG-bench: Repeat copy logic

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.425

0.450

0.475

0.500

0.525

0.550

0.575

0.600

Top-1 error: BIG-bench: Strange stories

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.44

0.46

0.48

0.50

0.52

0.54

Top-1 error: BIG-bench: Strategy QA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.68

0.70

0.72

0.74

0.76

0.78

Top-1 error: BIG-bench: Understanding fables

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.35

0.40

0.45

0.50

0.55

0.60

Top-1 error: BoolQ

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.68

0.70

0.72

0.74

0.76

0.78

0.80

0.82

Top-1 error: Commonsense QA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.2

0.3

0.4

0.5

0.6

Top-1 error: COPA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.60

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Top-1 error: CoQA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.44

0.46

0.48

0.50

0.52

0.54

0.56

0.58

Top-1 error: Enterprise PII classification

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.3

0.4

0.5

0.6

0.7

Top-1 error: HellaSwag (10-shot)

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.3

0.4

0.5

0.6

0.7

Top-1 error: HellaSwag (zero-shot)

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.6

0.7

0.8

0.9

1.0

Top-1 error: Jeopardy

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Top-1 error: LAMBADA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.70

0.72

0.74

0.76

0.78

0.80

Top-1 error: LogiQA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.740

0.745

0.750

0.755

0.760

0.765

Top-1 error: MathQA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.730

0.735

0.740

0.745

0.750

0.755

0.760

0.765

0.770

Top-1 error: MMLU (5-shot)

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.73

0.74

0.75

0.76

0.77

Top-1 error: MMLU (zero-shot)

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.600

0.625

0.650

0.675

0.700

0.725

0.750

0.775

Top-1 error: OpenBook QA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.25

0.30

0.35

0.40

0.45

0.50

Top-1 error: PIQA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.5

0.6

0.7

0.8

0.9

1.0

Top-1 error: PubMed QA Labeled

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.975

0.980

0.985

0.990

0.995

1.000

Top-1 error: Simple Arithmetic: NoSpaces

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.96

0.97

0.98

0.99

1.00

Top-1 error: Simple Arithmetic: WithSpaces

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.48

0.49

0.50

0.51

0.52

Top-1 error: SIQA

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.5

0.6

0.7

0.8

0.9

1.0

Top-1 error: SQuAD

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.45

0.50

0.55

0.60

Top-1 error: WinoGender MC: Female

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.35

0.40

0.45

0.50

0.55

0.60

Top-1 error: WinoGender MC: Male

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

Top-1 error: WinoGrand

2.5

3.0

3.5

4.0

4.5

5.0

5.5

Loss: C4 eval

0.36

0.38

0.40

0.42

0.44

0.46

0.48

0.50

0.52

Top-1 error: WinoGrande

C4

RedPajama

RefinedWeb

Random chance

F Additional related work 1040

Language modeling. Language models can be grouped into encoder-only [26, 53, 59, 96, 22], 1041 encoder-decoder [56, 89], and decoder-only architectures [85, 113, 114, 110, 49, 38, 74, 7, 111, 1042 28, 64, 99, 122, 4, 57, 63, 34]. Most current implementations are based on the transformer [116]. 1043 However, there has been a recent resurgence in scaling language models based on non-transformer 1044 architectures [83, 36, 37, 35]. Further, there has been substantial work on adapting pre-trained 1045 language models to better follow instructions [119, 20, 70, 61, 71, 133, 87, 29, 115, 103, 73]. 1046 However, following prior work [45, 72] and given their overall prevalence, we limit ourselves to 1047 GPT-style, decoder-only transformers that have solely been pre-trained. 1048

Scaling laws. Kaplan et al. [51] investigate scaling trends in GPT language models. Bahri et al. 1049 [9] investigate different scaling regimes theoretically, and Sharma & Kaplan [101] relate scaling 1050 coefficients to data manifold dimensions. Tay et al. [108, 109] elucidate the connection between 1051 model architecture and scaling trends, while Hernandez et al. [42], Tay et al. [108] develop scaling 1052 laws for transfer learning. Ivgi et al. [48] also consider transfer learning scaling laws and highlight 1053 the importance of hyperparameter selection in the low-compute regime. Ghorbani et al. [32], Gordon 1054 et al. [33], Bansal et al. [10] develop scaling laws for neural machine translation. Caballero et al. [17] 1055 propose a scaling law functional form, which they demonstrate is predictive in several domains. 1056

1057

1058

1059

Scaling beyond language modeling. There is a large body of work on scaling neural networks beyond language modeling, for example in computer vision [60, 127, 105, 1, 2], multimodal learning [41, 18, 30], and image reconstruction [52].

1060

1061

1062

Over-training in existing models. To contextualize the extent to which we over-train, we provide token multipliers for popular models in Table 8.

G Broader impact

Language models have known risks in terms harmful language, toxicity, and human automation-to 1063 name a few [121, 12]. We will include the following for our public release 'WARNING: These are 1064 base models and not aligned with post-training. They are provided as is and intended as research 1065 artifacts only.' However, even as research artifacts, we recognize that models can still be misused 1066 by malicious actors or can be harmful to benevolent actors. When deciding to release our models 1067 and experiments, we considered (i) the benefit to the scientific community and (ii) the benchmark 1068 performance relative to other models that have already been released. For (i) we feel that our testbed 1069 is of use to others in the community who want to do scaling research, but do not necessarily have the 1070 means to train these model artifacts themselves. Hence, we predict (and hope) releasing all models 1071 and experiments will be helpful to others wanting to participate in scaling research. For (ii), we note 1072 that there are publicly available models [113, 114, 49], which outperform models from our testbed 1073 and that are more likely to be widely adopted. Finally, we recognize that advancing scaling science 1074 also has potential for harm. Specifically, while we are concerned with loss and downstream task 1075 performance for popular evaluation settings, it is possible that nefarious actors may use scaling laws 1076 to help design more harmful models. 1077

H Licensing 1078

In terms of licensing, we will release our code, models, and experiments under an MIT licence, which 1079 is also attached to our supplementary submission. 1080

1081

1082

1083

1084

1085

1086

1087

1088

1089

1090

1091

1092

1093

1094

1095

1096

1097

1098

1099

1100

1101

1102

1103

1104

1105

1106

1107

1108

1109

1110

1111

1112

1113

1114

1115

1116

1117

1118

1119

1120

1121

1122

1123

1124

1125

1126

1127

1128

1129

1130

1131

1132

1133

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The experiment section justify the claims made in the abstract and introduction, namely that the developed scaling laws for over-training and downstream task prediction are predictive in practice for larger scale runs.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The final section discusses limitations, which provide motivation for future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

1134

1135

1136

1137

1138

1139

1140

1141

1142

1143

1144

1145

1146

1147

1148

1149

1150

1151

1152

1153

1154

1155

1156

1157

1158

1159

1160

1161

1162

1163

1164

1165

1166

1167

1168

1169

1170

1171

1172

1173

1174

1175

1176

1177

1178

1179

1180

1181

1182

1183

1184

1185

1186

1187

Answer: [Yes]

Justification: All assumptions are clearly stated and full proofs/derivations are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We point to all public datasets and open source training infrastructure. We additionally specify all hyperparameters used for training.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

1188

1189

1190

1191

1192

1193

1194

1195

1196

1197

1198

1199

1200

1201

1202

1203

1204

1205

1206

1207

1208

1209

1210

1211

1212

1213

1214

1215

1216

1217

1218

1219

1220

1221

1222

1223

1224

1225

1226

1227

1228

1229

1230

1231

1232

1233

1234

1235

1236

1237

1238

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include code and data needed to reproduce all figures in the paper. Our datasets are sourced from HuggingFace and our training code utilizes OpenLM, which is open-source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We explicitly have sections and appendices that detail our experimental setup (training and evaluation) and title the sections and appendices to indicate this.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: When appropriate we report bootstrap 95% confidence intervals (e.g., in Figure 4 and Figure 17). We do not train models with many seeds, which is prohibitively expensive. Given the large size of the C4 validation set, we observe that bootstrap 95% confidence intervals for loss (computed over either token an sequence sampling) are close to zero.

Guidelines:

The answer NA means that the paper does not include experiments.

1239

1240

1241

1242

1243

1244

1245

1246

1247

1248

1249

1250

1251

1252

1253

1254

1255

1256

1257

1258

1259

1260

1261

1262

1263

1264

1265

1266

1267

1268

1269

1270

1271

1272

1273

1274

1275

1276

1277

1278

1279

1280

1281

1282

1283

1284

1285

1286

1287

1288

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We are transparent about how many GPU hours it takes to construct our scaling laws and train our models (e.g., in Table 1).

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the code of ethics and feel that our research abides by this code in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

1289

1290

1291

1292

1293

1294

1295

1296

1297

1298

1299

1300

1301

1302

1303

1304

1305

1306

1307

1308

1309

1310

1311

1312

1313

1314

1315

1316

1317

1318

1319

1320

1321

1322

1323

1324

1325

1326

1327

1328

1329

1330

1331

1332

1333

1334

1335

1336

1337

1338

1339

1340

1341

1342

1343

Justification: This work is related to predicting the performance of language models, before they are trained. As such, it falls under the category of basic research. However, because we produce generative language model artifacts as part of our paper, we recognize that these pre-trained models can pose risk. We provide a discussion of risks in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We provide discussion of responsible release in Appendix G. Specifically, models in this release are know to be less capable than state-of-the-art, publicly available models [113, 114, 49], and, hence, we feel the risk for misuse is low.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We utilize data-sources publicly available on the HuggingFace platform and abide by the terms of use. For C4: Open Data Commons License Attribution family, for RedPajama: a list of licenses (found here.), for RefinedWeb: Open Data Commons License

1344

1345

1346

1347

1348

1349

1350

1351

1352

1353

1354

1355

1356

1357

1358

1359

1360

1361

1362

1363

1364

1365

1366

1367

1368

1369

1370

1371

1372

1373

1374

1375

1376

1377

1378

1379

1380

1381

1382

1383

1384

1385

1386

1387

1388

1389

1390

1391

1392

1393

Attribution family. We use the OpenLM repo for training and also abide by their MIT license.

We cite all papers and repos in the main text.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Our code release documents all new model assets under the exp_db/ folder and includes a MIT license. This is also specified in Appendix H.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This research does not involve crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWhile existing scaling law studies look at compute-optimal pretraining, this paper considers scaling laws in the context of both pretraining and downstream performance. They perform scaling experiments and find that performance is predictable even in overtraining, and average downstream performance is also predictable.\n\nSTRENGTHS:\nI think this is a solid paper that attempts to answer an important question. While I’m concerned about the lack of novelty (see the weaknesses), I overall lean towards accepting the paper. I think that the fact the paper reproduces findings from other papers using a different methodology is a good sign that the overall results are correct, and is valuable information (e.g. [Owen 2024](https://arxiv.org/pdf/2401.04757) also finds that average downstream performance is more predictable than for individual downstream tasks). To me, this is the primary contribution of the paper, and an important one as such.\n\nWEAKNESSES:\nTo my mind, the largest weakness of this paper is the lack of novelty. In particular, my understanding is that the most important findings are that (1) performance is predictable in overtraining, and (2) average downstream performance is predictable. I’m not sure why (1) should be surprising – doesn’t a parametric scaling law, such as method 3 from the Chinchilla paper, also give the ability to predict loss when overtraining? I think the authors could improve the motivation for this consideration by providing a back of the envelope calculation: for instance, I plugged in the model size and dataset size for Gopher 280B into the Chinchilla scaling law and got a predicted test ppl of ~7.3 on MassiveText. However, Gopher actually had a (validation) perplexity of ~8.1, this constitutes a relative error of around 10% – substantially larger than the relative errors obtained by the authors of this paper. If the authors can provide an argument of this sort I'd find that helpful. \n\nI thought (2) was a more interesting claim, but I’ve seen this analyzed in Owen 2024 (https://arxiv.org/pdf/2401.04757), albeit with a different methodology. As such, I felt that the core results of the paper weren’t very novel. However, I’d be happy to update my assessment if the authors can provide evidence that my understanding is incorrect. \n\nOne interesting point that the authors mentioned is that performance on individual tasks is less predictable. But this is only mentioned in passing, and I felt that it could be expanded upon a fair bit. What are the implications of this observation? Are there any patterns for which individual tasks are or aren’t predictable? \n\nI’m slightly concerned about data leakage being an issue for the downstream tasks, given that the training data (The Pile and RedPajama) covers a wide swath of the internet, and some of the downstream benchmarks have been criticized for data leakage or having label errors. \n\nMinor comment: The last paragraph of section 5 is a bit confusing: “There has been a rise in over-trained models [113, 114] and accompanying massive datasets [112, 82, 104, 3]. For example, Chinchilla 70B [45] is trained with a token multiplier of 20, while LLaMA-2 7B [114] uses a token multiplier of 290.” This makes it sound a bit like Chinchilla is overtrained, which I don’t think the authors are trying to say, so I’d suggest something like the following instead: “For example, while Chinchilla 70B is trained compute-optimally with a token multiplier of 20, LLaMA-2 7B…”\n\nQUESTIONS:\n- Doesn’t a standard parametric scaling law fit, such as method 3 from the Chinchilla paper, also allow one to predict loss when overtraining? \n- How do the results from this paper differ from those in this other paper, which also looks at downstream performance? https://arxiv.org/pdf/2401.04757 \n- If I plug in the coefficients from the Chinchilla scaling law into equation 8, with $\\alpha = 0.34$ and $\\beta = 0.28$, I find that the predicted value for $\\eta$ is $\\frac{\\alpha \\beta}{\\alpha + \\beta} \\approx 0.154$. In comparison, the values of $\\eta$ in table 6 are generally around 0.25. What’s the cause of this difference, and how do you know? \n- Did the authors consider alternative functional forms for the downstream performance scaling law, besides the negative exponential? \n- Did the authors check for data leakage? How?\n\nLIMITATIONS:\nI felt that the authors did a good job describing some of the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a scaling law for the “Chinchilla over-trained” regime where models are trained on many more tokens (in this paper, up to 30x) than Chinchilla-optimal. They motivate a scaling law relating pre-training compute and “over-training” to validation loss. They empirically demonstrate that the proposed scaling law accurately predicts the validation loss of a 1.4B 32x over-trained model and a 6.9B Chinchilla-optimal model. They then study a simple scaling law relating perplexity to downstream benchmark error. They select a subset of 17 benchmarks for which a 154M parameter models performs 10% above random chance accuracy, and show show that average downstream error of the 1.4B and 6.9B models is predictable.\n\nSTRENGTHS:\nThe setting considered by the paper is very relevant at the moment, as major model releases in the past year fall precisely in the “Chinchilla over-trained” regime. The work is also novel, as I am not aware of prior work that proposes and empirically validates scaling laws tailored for the Chinchilla over-trained regime.\n\nThe proposed scaling law in the over-trained regime is well-motivated both by prior scaling laws and by further empirical observations in the over-trained regime (Figure 2). While the models trained (5-8 1e21 FLOPs) are at least two orders of magnitude smaller than the latest open models (e.g., Gemma 2, Llama 3, Qwen 2), the experiments are of a large enough scale for a proof of concept.\n\nThe paper is clear and it gives sufficient details on the experimental set-up.\n\nWEAKNESSES:\nMy main concerns are twofold: authors do not compare with the standard scaling laws of Kaplan et al. (fitted on their own testbed), and it is unclear how authors choose the models used to fit their scaling laws (Table 1).\n\nAuthors do not compare their over-trained scaling law with that proposed by Kaplan et al. The authors could fit this scaling law as in Hoffman et al. , Section 3.3, without any additional model training. Specifically, how well can the standard Kaplan et al. law predict the validation loss of the 1.4B over-trained model, when fitted on the model testbed with N < 1B described in Section 3.2?\n\nRegarding the claim that validation loss (resp accuracy) is predictable with 300x (resp. 20x) less compute, I find this misleading, since authors train and evaluate a reasonably large model testbed, but only report the compute required for the 5 (resp. 6) models that they ultimately choose for the fit.  The authors do not discuss how this “train”/“test” split was chosen. Clearly, the train/test split should be chosen before seeing the evaluation results for any of the models, rather than including models until the fit seems \"good enough\", or choosing the smallest subset for which the fit is “good enough”. Otherwise, both the claim of 300x/20x compute as well as Figure 5 are misleading. Similarly, Figure 1 would be misleading, and it should include all models with N < 1.4B. \n\nThe authors consider token multiplier M <= 640, however current models are even more overtrained. If am not mistaken, for Llama 3 8B, M ~= 2000. Demonstrating the validity of the proposed scaling laws for the amount of over-training of current models would have been ideal, even at smaller model scales.\n\nLastly, the proposed scaling laws are validated at substantially lower compute scales than current models with publicly available weights. It would have been ideal to at least see results for over-trained 7B models. I understand that the experiments presented in the paper already require a substantially amount of compute, and more closely matching the compute scales of recent models would be unfeasible for most research labs — therefore I am not taking this point into consideration when scoring the paper.\n\nQUESTIONS:\n* How well can the standard Kaplan et al. law predict the validation loss of the 1.4B over-trained model, when fitted on the model testbed with N < 1B described in Section 3.2?\n* How were the models in Table 1 chosen? Were they chosen prior to the other models being evaluated? Was it apparent that including only one over-trained model, namely 11M, 320M, would suffice?\n* Figure 18 shows that HellaSwag and ARC-Easy scores are very correlated with validation loss. However, Table 2 shows very large individual top-1 error for HellaSwag and ARC-Easy. How can this be?\n\nLIMITATIONS:\nLimitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the power laws (scaling laws) of neural language models, particularly from the perspective of over-training and the relationship between validation loss (perplexity) and NLP downstream tasks. The authors define over-training as the situation where runs consume large amounts of computational resources, and they introduce a token multiplier, M. It is computed by D / N, where D is the number of training data tokens and N is the number of parameters. Through various model training setups and three different training corpora, the authors demonstrate that the validation loss can be computed and predicted using an equation that includes M. They also introduce another equation that illustrates the relationship between validation loss and downstream task error.\n\nSTRENGTHS:\nThe main contribution of this paper is the exploration of the scaling laws in language models concerning over-training and NLP downstream tasks. These results, including equations and practical outcomes, are beneficial for researchers and engineers developing large language models. As highlighted by the authors, these insights are valuable for researchers in their future work.\n\nWEAKNESSES:\nThe authors mentioned several limitations and future work in the paper. I agree with them, and especially the ‘scaling up’ part is the primary concern of this paper. The model sizes range from 0.011B to 6.9B, but open-source models are larger than these sizes - for instance, Llama 2 starts at 7B, and Llama 3 starts at 8B [1]. Furthermore, model size is crucial for techniques such as CoT [2]. I hope to hear the authors' opinions on this concern.\n\nQUESTIONS:\n- M could be related to overfitting if N is large but D is not sufficient. The authors mentioned that when M equals 5 (under-training), the scaling becomes unreliable. I assume this is because the models are not well trained—underfitting. Could you explain and discuss the relationship between over-training and overfitting?\n\n- (Minor) It is hard to distinguish between M=320 and M=640 in the figures. Could you change the colors to make them more distinguishable?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}