Understanding Emergent Abilities of Language Models from the Loss Perspective
Authors: Zhengxiao Du
Aohan Zeng
Yuxiao Dong
Jie Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: pretrained language model
emergent ability
scaling law
Primary area: natural_language_processing
TL;DR: We demonstrate the existence of emergent abilities in the lens of pre-training loss regardless of the continuity of evaluation metrics

Abstract: Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities on certain tasks—regardless of the continuity of metrics—when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

Understanding Emergent Abilities of Language Models from the Loss Perspective
Zhengxiao Du 1.2 Aohan Zeng Yuxiao Dong", Jie Tang?
'Zhipu A1 TTsinghua University (zx-du20.zah22] thaillstsinghuu.cdu..c
Abstract
Recent studies have put into question the belief that emergent abilities [58] in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose study emergent abilities in the lens of pre- training loss, instead of model size training compute. We demonstrate that the Transformer models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks, with a fixed data corpus, tokenization, and model architecture. We also discover that a model exhibits emergent abilities certain resks-regardless of the continuity metrics-wher its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre- training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.
Introduction
Scaling of language modes (LMs) both model and data sizes has been shown be effective
for improving the performance on a wide range of tasks 442 6 23 widespread adoption of LM applications, e.g., ChatGPT. The success of such scaling is guided by
65
55) [3],, leading to the
scaling laws (2 28 10 23 which study the predictability of pre-training loss given the model and
data sizes.
While scaling laws focus on the pre-training loss, the scaling effect on the performance of downstream tasks has thus far less studied. Emergent abilities [58) defined as abilities that present in larger LMs but not present in smaller one. The existence of such abilities is recently challenged for two reasons. First, small LMs trained on sufficient amount of high-quality data can outperform large models on tasks with claimed emergent abilities 55 56 26]. For example, LLaMA- 13B with less compute (55] outperform GPT-3 (175B) MMLU 21,, due to more training tokens and improved data-filted filt-fiii methods. Second, Schaeffer et al. [46] claim that emergent abilities appear due to the nonlinear or discontinuous metrics selected to evaluate certain datasets, rather than from fundamental change in larger models.
The Chinchilla scaling laws [23] show that different combinations of model sizes and data sizes can lead to different pre-training losses even with the same training compute. Consequently, the pre-training loss can naturally better represent the learning status of LMs than the model or data sizes. However, the relationship between the loss of an LM and its performance on downstream tasks not yet well understood. Existing literature has either focused the transfer learning paradigm 33 or constrained its study to single models, tasks, or prompting methods 44 61].
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this work, we propose to study emergent abilities from the perspective of pre -reiiiinnng loss instead of model size or training compute. To examine the relationship between the pre-training loss of LMs and their performance, we pre-train more than 30 LMs of varied model and data sizes from scratch, using a fixed data corpus, tokenization, and model architecture. Their downstream performance is evaluated on 12 diverse datasets covering different tasks, languages, prompting types, and answer forms. We demonstrate that the pre-training loss of an LM is predictive of its performance on downstream tasks, regardless of its model size data size. The generality of this conclusion is further verified by extracting and observing the performance and loss relationship of the open LLaMA [55] and Pythia B models.
Over the course, we find that performance on certain downstream tasks only improves beyond the level of random guessing when the pre-traine vre-trainig loss falls below specific threshold, i.e., emergent abilities. Interestingly, the loss thresholds for these tasks are the same. When the loss is above this threshold, performance remains at the level of random guessing, even though performance on other tasks continues to improve from the outset. To exclude the impact of discontinuous metrics [46 61 we evaluate the emergent performance increase using continuous metrics and show that the emergent abilities persist across both discontinuous and continuous metrics.
Based on these observations, we define the emergent abilities of LMs from the perspective of pre- training loss: an ability emergent it is not present in language models with higher pre-training loss, but is present in language models with lower pre-training loss. According to the loss scaling laws [22 28). the pre-training loss is a function of model size, data size, and training compute. Therefore, the new emergent abilities can also account for the previously-observed emergent abilities in terms of model size or training compute.
The advantage of the new definition lies in its ability to better capture the tipping points in training trajectories when LMs acquire emergent abilities. Once again (58), the existence of emergent abilities suggests that we cannot predict all the abilities of LMs by simply extrapolating the performance of LMs with higher pre-training loss. Further scaling the model and data size lower the pre-training loss may enable new abilities that were not present in previous LMs.
2 Does Pre-training Loss Predict Task Performance?
Table 1: English and Chinese datasets evaluated in the experiment, and their task types, prompting types, answer forms and metrics. For prompting type, refer to the ohain-of-thoughle prompting (59) as few-shot CoT and the original in-context learning prompting [6] as few-shot.
Dataset
Task
Prompting Type
Answer Form
Metric
English datasets
TriviaQA
27
Closed-book QA
Few-shot
Open-formed
EM
HellaSwar wag 644
Commonsense NIJI
Zero-shot
Mulit-choice
Accuracy
RACE 31
Reading Comprehension
Few-shot
Multi-choice
Accuracy
WinoGrande
24
Coreference Resolution
Zero-shot
Multi-choice
Accuracy
MMLU 21
Examination
Few-shot
Multi-choice
Accuracy
GSM8K 112
Math Word Problem
Few-shot CoT
forn-formed
EM
Chinese datasets
15
Closed-book QA Commonsense NLI
Few-shot
Open-formed Multi-choice
EM
ClozeTT 63
Zero-shot
Accuracy
CLUEWSC 62
Coreference Resolution
Zero-shot
Multi-choice
Accuracy
C3 12
Reading Comprehension
Few-shot
Multi-choice
Accuracy
C-Eval 25
Examination
Few-shot
Multi-choice
Accuracy
Math Word Problem
Few-shot CoT
Open-formed
EM
We study the relationship between the performance of the language models (LMs) on 12 downstream tasks and the pre-training loss. We pre-train LMs different model sizes (300M, 540M, 1B, 1.5B, 3B, 6B, and 32B) on varied numbers of tokens with fixed data corpus, tokenization, and architecture. In addition, we leverage the open LLaMA (55) models (7B, 13B, 33B, and 65B) validate our observations.

---

It is not straightforward that the loss of LMs decides the performance on downstream tasks. Generally the performance is decided by the probability to predict the ground truth y given the prompt æ, i.c. p(y(a)) The probability can be written as function of the cross entropy loss:
p exp(-
where a(y(a) is the cross entropy loss of the LM given the context & and the target While { has the same form the pre-training loss T., they
equal. First, the pre-training loss is average of all the tokens in all the documents pre-trained on, According to our empirical observation, the losses of different documents
not uniform. Second, if x and similar documents do not exist the pre-training corpus, (()(z) is the generalization loss, which is often related to other factors beyond the training loss, such as the model size. For example, in computer vision, highly over-parameterized models often improve over an under- soder-parameterized models in test performance when both models
converge on the training data [14
2.1 Pre-training Setting
All the models are pre-trained on a mixture of English and Chinese corpus. The ratio of English to Chinese 4:1 the pre-training corpus. The model architecture is similar LLaMA (55] with two differences: we use grouped-quee -uery attention [1] to replace the multi-query attention and we apply rotary position embedding on half the dimensions the query and key vectors. More details can be found in Appendix
2.2 Evaluation Tasks
To present a comprehensive demonstration, we evaluate the pre-trained models on 12 datasets across different tasks and prompting types in both English and Chinese. The six task types include:
Closed-book QA: Answering questions about the real world based solely on the pretrained knowledge. We use TriviaQA 31 for English. For Chinese, we build closed-book QA dataset based on NLPCC- KBQA 15 dataset following the TriviaQA format.
Commonsense Natural Language Inference (NLI); Selecting the most likely followup given an event description. We use the HellaSwag dataset for English and the Cloze' dataset in Yao et al. 663 for Chinese.
Reading a given article or paragraph and answering questions about it. We use RACE 31 for English and C3 [52] for Chinese. Both are based on multi-choice questions.
Reading comprehension:
Coreference Resolution: Given a sentence with pronouns, determine which pronoun refers which entity. We use the WinoGrande dataset (44) for English and the CLUEWSC dataset [62 for Chinese.
Examination:
Multiplechhoice questions in examinations. For English, we use MMLU (21, which includes mathematics, US history, computer science, law, and more. For Chinese, we use C-Eval 25 which ranges from humanities to science and engineering.
Solving real-life, situational and relevant problems using mathematical concepts. For English use the GSM8K [12) dataset. For Chinese, translate the questions and answers in GSM8K to Chinese, namely GSM8K-Chinese.
Math Word Problem:
The prompting types cover few-shot [6], zero-shot, and few-shot Mhain-of-thoughee (CoT) [59]. The datasets are summarized in [
2.3 Pre-training Loss vs. Performance
In the first experiment, train three models with 1.5B, 6B, and 32B parameters and observe their behaviors until trained on 3T, 3T, and 2.5T tokens, respectively. The training hyperparameters are shown in Table (Appendix).
We evaluate the performance of intermediate training checkpoints. The checkpoints are saved around every 43B tokens during pre-training. We plot the points of task performance (y-axis) and training

---

TriviQAA
HellaSwag
RACE
WinoGrande
1:58
55
75
Faiggn sgb
formance
50
tandem
49
65
40
60
30
50
2.75
255
2.25
2.30
1.75
2.75
2.50
2.25
2.00
1.75
2.75
2330
2.45
2.00
1.75
2.75
2.50
2.25
2.00
1.75
NLPCC-KBQA
ClozeT
CLUEWSC
80
25
75-
Performance
20
70
50
1
60-
55
50
2.75
2.50
2.25
2,00
1.75
2.75
2.50
2.25
2.00
1.75
2.75
2.50
2.25
2.00
1.75
2.75
2.50
2.25
2:00
1.75
GSMøK-Chinese
60
30-
Peformance
50-
20-
40-
20
20
275
2.50
225
2.00
1.75
2.75
2.50
2.25
2.00
1.75
2.75
2.50
2.25
2.00
1.75
2.75
2.50
2.25
2.00
1.75
Training Loss
Training Loss
Training Loss
Training Loss
Figure 1: The performance-vs-los curves of 1.5B, 6B, and 32B models. Each data point the loss (I-axis) and performance (y-axis) of the intermediate checkpoint of one of the three models. We mark the results of random guess in black dashed lines.
Table 2: Statistical measures of the correlation between task performance and pre-training loss in Figure The spearman correlation coefficient (50) measures the monotonicity of the relationship between the two variables, and the pearson correlation coefficient measures the linearity of the relationship. Both vary between -1 and +1 with 0 implying no correlation. Correlations of or +1 imply an exact monotonic/linear relationship.
Dataset
THQQ
RACE
NQA
Cloze'
GSM
GISMC
Spearman
0..96
-1.99
0.977
-0.788
-0.984
-0.986
-0.988
0.804
-0831
-0,955
-0,948
Pearson
-0.94
-0.994
-0.963
-0.988
-0.982
-0.985
-0.993
-0,972
-0,903
-0,884
-0.874
-0,829
loss (æ-axis) Figure D and provide statistical measures of the two variables in Table From the curves and statistics, 12 downstream tasks.
that the training loss is good predictor of the performance
Generally, the task performance improves as the training loss goes down, regardless of the model sizes. On MMLU, C-Eval, GSM8K, and GSM8K-Chie -Cinese, models of three sizes perform at the random level until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss decreases. Detailed analysis on this is shown Section 3
Importantly, the performance-ve vo-loos data points of different model sizes fall on the same trending curve. That is, by ignoring the color differences (model sizes), the data points of different models are indistinguishable. For example, when the training loss falls around 2.00, the green and orange points TriviaQA are indistinguishable. This indicates that the model performance downstream tasks largely correlates with the pre-training loss, regardless of the model size.
Both spearman and pearson correlation coefficients show that performance is strongly related to pre-training loss on TriviaQA. HellaSwag, RACE, WinoGrande, etc. The pearson correlation coefficients on these tasks specifically show that points from different models lie on the same trending curve. The relationship is weaker on MMLU, CEval, GSM8K, and GSM8K-Chines verifying the emergence of performance which we discuss in Section

---

Interestingly, we find that the overall training loss is good predictor of performance on both English and Chinese tasks, although is computed on mixture of English and Chinese tokens. This implies that the learning dynamics of English and Chinese tokens are likely very similar during multilingual pre-training.
2.4
Training Token Count vs. Performance
TriviaQA
HellaSwag
RACE
WinoGrande
40
45
60
Perforaacee
58
rondom
56
10
30
NLPCC-KBQA
ClazeT
C3
CLUEWSC
17.5
60
70
Porfomannes
12.5
50
55
10.0
60
60
2.5
55-
30
55-
MMLU
C-Evall
GSMBK
GSMBK.Chinese
Porformance aanaan
35.0
32.5
12.5
30.0
30.0
10.0
10.0
27.5
27.5
7.5
7.5
25.0
25.0
5.0
22.5
20.0
Training
Training
Training
Training
Figure 2: The performance-vs-los. curves of smaller models pre-trained with different numbers of training tokens. Each data point the loss (æ-axis) and performance (y-axis) of the final checkpoint of one model, i.e., each point corresponds to one model trained from scratch. We mark results of random guess black dashed lines.
Following the empirical experiments in scaling laws 122 smaller models with different numbers of training tokens. The model sizes range from 300M, to 540M, IB, 1.5B, 3B, and to 6B, while the numbers of pre-training tokens range from 33B to 500B. Varying the number of pre-training tokens is necessary since to achieve optimal performance we need to set the cosine learning rate schedule to reach the minimum at the corresponding token count 28 233 The number of tokens used and hyperparameters for all models are shown in Table (Appendix).
28 23
we further pre-train 28 relatively
each line, each data point represents the performance and pre-training loss of the corresponding model pre-trained completely from scratch with the certain token count (and learning rate schedule). We can see that similar to the observations from Figure I the data points of different models sizes and training tokens largely fall the same trending curves. In other words, the LMs with the same pre-training loss regardless of token count and model size exhibit the same performance on the 12 downstream tasks.
Another similar observation is that the performance curves on MMLU, C-Eval, GSM8K, and GSMSK- Chinese do not yield an uptrend, meaning that the performance of these models on these four tasks are close to random (with fewer than 500B tokens). For simplicity, we only plot the performance of the latest checkpoint in each training in Figure 2 The complete performance curves with intermediate checkpoints of each model, in which we can observe the same trend but larger variance, are shown in Figure 5 (Appendix).
5

---

2.5 LLaMA's Loss vs, Performance
TriviaQA
Hell Swag
Natural Questions NaturalQuestions
70
80
70
50
30
50
20
40
10
2.0
SIQA
WinoGrande
PIOA
50.0
75
Perforance
65.0
70
42.5
03
63
80
LLaMA 138
37.5
55
LLaMA 338
35.0
50
random
2.0
Training Loss
Training Loss
Training Loss
Figure 3: The performance-vs are extracted from the figures in the original LLaMA paper [55] Note that the LLaMA2 paper (56) does not such figures with related information.
of LLaMA. The values of performance and training loss
To validate the generality of our observations, analyze different model series with required information made publicly available, i.e., LLaMA (55) and Pythia [3] Compared to our models, ILaMA uses pre-training corpus that excludes Chinese documents, leverages different pre-training framework [37 and adopts slightly different model architecture. Since the intermediate checkpoints of LLaMA not available, we extract the pre-training loss and corresponding performance on six question answering and commonsense reasoning tasks from the figures in its original paper, and plot the points in Figure
Excitingly, most data points from the LLaMA models with different sizes (7B, 13B, 33B, 65B) fall on the same upwards trend. This observation further confirm conclusion that the model's pre-training loss can predict its performance on downstream tasks, regardless of model size and token count, Note that there is only one exception the early stage of LLaMA-65B. We loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss. Without access its intermediate checkpoints, we unfortunately cannot further analyze the result. One possible explanation is that they use exponential smoothing on either the loss or downstream performance plots. Exponential smoothing would perturb the earlier points more than other points, potentially leading to this effect. Note that the outliers only constitute the initial 10% training tokens. The results for Pythia are shown in Appendix IF) which also support our conclusion.
that when the training
Observed from previous experiments and analysis, we can conclude that the pre-training loss is a good indicator of LMS' performance downstream tasks, independent of model sizes, training tokens, languages, and pre-training frameworks.
3 Analysis of Different Tasks and Metrics
3.1 Performance Trends of Different Tasks
In Figures [ [and we can separate the datasets into two groups: First, on TriviaQA, HellaSwag, RACE, WinoGrande, NLPCC-KBQA, ClozeT, CLUEWSC, and C3, performance improves smoothly with decreased pre-training loss from the very beginning. Second, on MMLU, C-Eval, GSM8K, and SSSM8-Chhiness the performance remains flat when the loss is higher than a certain threshold. Once the pre-training loss is lower than this threshold, the performance starts improve.

---

MMLU (Accuracy)
MMLU (CorrectChoicerrab)
(BrierScore)
1.58
0.35
Wiehh (Y
60
68
0.5
0.30
Parformance
50-
random
0.40
0.7
0.35
0.8
0.30
2.75
2.50
2.25
2.00
1.75
2.75
2.50
2.00
1.75
2.75
2.25
1.75
(Accuracy)
C-Eval (CorrectChoiceProb)
(BrierScore)
0.50
60
0.45
0.55
Performance
50-
0.60
0.40
40
0.35
0.70
0.75
0.30
0.25
0.85
2.75
2.50
2.25
2.00
1.75
2.75
2:50
2.25
2.00
1.75
2.75
2.50
2.25
2.00
1.75
Training Loss
Training Loss
Training Loss
Figure 4: The performance-vs-loss curves of different metrics on MMLU and C-Eval. Accuracy: discontinuous; CorrectChoiceProb and BrierScore: continuous. We mark the result of random guess in black dashed lines.
The correlation coefficients in Table Dalso reveal the difference: coefficients in the first group are close to indicating strong correlations, while correlations in the second group are weaker.
Take MMLU as an example of the second group, when the pre-training loss is higher than 2.2, accuracy remains around 25%. Since each question in MMLU has four options, this means the model prediction no better than random guessing. However, when the pre-training loss drops below 2.2, the accuracy increases the loss decreases, similar to the trend observed in the first group of tasks. The performance trends of C-Eval, GSM8K, and GSM8K- -hhnnese follow a similar pattern. Despite differences in languages, tasks, prompting types, and answer forms among the four datasets different, the thresholds for performance improvement are surprisingly all around 2.2.
RACE in the first group has prompting format similar to MMLU: both consist of multi-choice examination questions with in-context demonstrations, but their performance curves are quite different. We hypothesis that it is the task difficulty that makes the difference. Tasks of the first group of datasets are easier than those of the second group. For example, RACE requires the model to select correct answers for questions about a given article, and HellaSwag lets the model select the possible followup of a situation based on commonsense. In contrast, MMLU and C-Eval consist of questions designed for high school, college. professional examinations, requiring broader range of knowledge. GSM8K and considered as impossible to be solved by pre-trained language models before Chain-of-Thouun prompting.
are math word problems that were previously
The phenomenon can be related to grokking, which describes the improvement of performance from the random chance level to perfect generalization [40]. Power et (40) find that this improvement can occur well past the point of overfitting. In pre-training, the models are usually underfitting instead of overfitting overall. Since the pre-training corpus is mixture of different documents, is possible that the model already fits some patterns patterns-such as numerical addition in the data, while still underfitting the overall corpus.
Certainly, the observations on the second groups of datasets can also be related to emergent abili- ties (58), that is, abilities that only present in large models. According to the scaling law (28,, with the number of training tokens fixed, the pre-traine vee-rainnng lass follows a power law with respect to model sizes. other words, there a monotonic relationship between model size and pre-training loss. For the second group of tasks, there is threshold of model sizes that corresponds to the tipping point in the pre-training loss. When the model size exceeds this threshold, the model can exhibit performance above the random chance level.
7

---

3.2 Influence of Different Metrics
Schaeffer al. [46] propose an alternative explanation emergent abilities of LMs. that is, emergent abilities appear due to the researchers' choice of nonlinear or discontinuous metrics. The accuracy on multi-choice questions (e.g.. MMLU) is discontinuous, since the score on question is either or O. To validate this claim, we examine the intermediate checkpoints on MMLJU and C-Eval with continuous metrics rather than discontinuous accuracy used in the original benchmarks. The first metric is the predicted probability of the correct answer, denoted as CorrectChoicePPohh The second one is the Brier Score [5] used in Schaeffer et al. [46]:
1 N C
BrierScore = Σ
N E
(oos- 0ty)²
where iúy is the predicted probability of sample for class j and Vij is the ground truth probability. The metric measures the prediction error and lower value indicates better performance.
We plot the results measured by different metrics on MMLU and C-Eval Figure All three metrics- accuracy, correct choice probability, and Brier (value increase for the first two and decrease for the third) when the pre- pre-training loss drops below a certain threshold. The Brier Score also decreases when the pre- vre-training loss is above the threshold. However, the decrease of Brier Score does not always represent improvements on the task, since the Brier Score is related to not only the predicted probability of the correct answer but also the predicted probabilities of the incorrect answers. We find that the distribution of the correct answers is uniform in the four options in MMLU and C-Eval. The best Brier Score for a context-fres predictor is achieved by always giving uniform probability to all the options. In this case, the Brier Score is equal to 0.75. Therefore, the performance in terms of Brier Score is no better than random guess before the loss reaches the threshold. This observation further confirms our previous conclusion. We discuss the contrary observations of Schaeffer al. 46 and Xia et al. 61 in Appendix C
emergent performance improvements
We conclude that emergent abilitics of language models occur when the pre-training loss reaches a certain tipping point, and continuous metrics cannot climinate the observed tipping point.
4 Defining Emergent Abilities from the Loss Perspective
In previous sections, show that 1) the pre-training loss is predictive of the performance of language modes on downstream tasks, and 2) some tasks exhibit emergent performance improvements from the random guess level when the pre-training loss drops below certain threshold regardless of model size, token count, and continuity of metrics. Based on these observations, we give a new definition of emergent abilities from the pre-training loss perspective:
Definition. An ability emergent if not present in models with higher pre- pre-training loss but is present in models with lower pre-training loss.
The normalized performance on an emergent ability as function of the pre-training loss L is:
(L) if L <1
otherwise
where f(L) is a monotonically decreasing function of L, is the threshold, and the normalized performance of random guess is O.
Next we will show how the new definition can be related to previously observed emergent abilities [38] In Henighan et al. [22), they give the scaling relation for the loss with model size N when the number of training tokens D is fixed:
No N where Loo is the irreducible loss, and oN is the coefficient. The equation shows that the loss of language models follows power-law plus a constant. Combining Equation (3) and Equation we can get the normalized performance as a function of the model size N
ON
L(N) Loc +
s LLoo + (No) aN)
if N 2 No (1) Loo) oN
(5)
otherwise
8

---

From this equation, we can explain the emergent abilities observed in Wei et al. [58] when model sizes are smaller than No- (1 - the normalized performance is zero. When model sizes exceed No (n- )z) ) the increase in model size leads to a decrease of pre-training loss and an improvement normalized performance.
5 Related Work
Relationship of Pre-training Loss and Task Performance. In the transfer learning setting, i.e. the language model is pre- pre-trained on the general corpus and fine-tuned on supervised data of specific tasks, Tay et al. (54) find that models with the same pre-training loss can have different downstream
performance after finetuning, due to inductive bias model architectures such as Transformers and Switch Transformers. Tay et al. (53) further study the effect of model shapes on downstream fine-tuning. Liu al. [33] also study the effect of inductive bias of model sizes and model algorithms on the relationship of pre- pre-training loss and downstream performance after fine-tuning, but their theory only applics in the saturation regime, where the models are close to minimal possible pre- pre-training loss. Instead, large language models today generally under-trained 23 [55, far from the saturation regime. Overall, these studies focus on the vvetrain-iinetun paradigm, in which inducitive bias helps improve transferability. while study prompted performance of large language models without finetuning [29] 6]. For the prompted performance of large language models, Xia et al. [61 claim that perplexity is a strong predictor of in-context learning performance, but the evidence limited the OPT model [66] and a subset of BIG-Bench [51]. Instead, Shin et al. 44) find that low perplexity does not always imply high in-context learning performance when the pre-training corpus changes. Gadre et al. (T8) fits the relation of perplexity and the top-1 error averaged over many natural language tasks with a power law. Instead, we focus on the different relations of tasks and small part of tasks that show emergency trends.
Emergent abilities. Wei et al. (58) propose the idea of emergent abilities, abilities that only present in large language models. This is similar to the claim of Ganguli et al. [19] that it more difficult to predict the capacities of language models than to predict the pre-training loss. The existence of emergent abilities has been challenged. Hoffmann al. 23 show that smaller language models trained with sufficient data can outperform undertrained larger language models, supported by follow- up models [55] 26 (6). On the other hand, Schaeffer et al. [46] claim that emergent abilities are due to the discontinuous metrics used for evaluation, also found in Xia et al. [6]]. Similarly, Hu et al. [4 propose to predict the performance of emergent abilities with the infinite resolution evaluation metric. In this paper we prove the existence of emergent abilities from the perspective of pre-training loss, even with continuous metrics.
Conclusion
Our paper proposes a new definition of emergent abilities of language models from the perspective of pre-training loss. Empirical results show that the pre-training loss is better metric to represent the scaling effect of language models than model size or training compute. The performance of emergent abilities exhibits emergent increase when the pre-training loss falls below certain threshold, even when evaluated with continuous metrics.
The new definition offers precise characterization of the critical junctures within training trajectories where emergent abilities manifest. It encourages future studies to investigate the shifts language models at these junctures, which facilitate the development of new capabilities.
7 Limitation
We study the relationship of pre-training loss and task performance across model sizes, training tokens, tasks, languages, prompting types, and answer forms. Factors we have not considered are model architectures and training algorithms. We analyze the performance-loss curves of LLaMA and Pythia with slightly different architectures, and find that the relationship holds for all the models. But there are fundamentally different model architectures, such routed Transformers [16], and

---

non- non-Tranfformer architectures [77 39 beyond our consideration. Both our models and LLaMA use AdamW optimizer (35], while there are other optimizers for language model pre-training 48 32
The disadvantage of studying emergent abilities in the lens of pre-training loss that the pre-training loss is affected the tokenizer and the distribution of pre-training corpus. The values of pre-training loss of language models trained on different corpus are not directly comparable. One possible solution evaluate different language models on public validation set with the normalized perplexity (43 to account for the different vocabulary sizes.
The paper should not be considered
push to expand model sizes and data sizes of language models beyond current scales. It is not guaranteed that new tipping points emerge in larger scales. Also, instruction tuning [37] 45 34] can improve the zero-shot performance of language models on unseen tasks, including MMI.U and GSM8K.
Acknowledgments and Disclosure of Funding
This work is supported by the Natural Science Foundation of China NSFC 62425601 and 62276148, research fund from Zhipu, New Cornerstone Science Foundation through the XPLORER PRIZE and Tsinghua University (Department of Computer Science and (achhooooyy--Smmmeee Ltd., China Joint Research Center for Industrial Intelligence and Internet of Things (JCHITT). Corresponding authors: Yuxiao Dong and Jie Tang.
10

---

References
[1] Joshua Ainslic, James Lee- Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: training generalized multi-query transformer models from multi-head checkpoints. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6- 10, 2023, pages 4895 4901. Association for Computational Linguistics, 2023.
BRLLhttps:/// htt::/acllnnii hwwt://ooo.aaaoooooooooooooo wwwploogy.orrg/922 org/ emn wappppmminn.mm 298
[2] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. httpss
/ //hugiigfface. ct/ http://pcceeuuggggeee www.ugggigggggoooo accHa//ppeennccce.e 11mm://mmppmmmm______mmmmmmmm_mp weweeeennmmeeeeeeeeaaaaaaaaeeaaa leaderboard, 2023.
[3]
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 2397-2430. PMLR, 2023.
[4] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secone Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432 7439. 1609/
AAAI Press, 2020. doi: 1.1609/AAAII.33400..00
URL /tttps:/ddo..or
aaai. v34105. 6239.
[5] Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1): 3. 1950. doi: 10.1 10..175/152000001100000000000010
URL https: / journals..sa..s.. awe.soc. org http://wwwww...oooooo///////////
078 0001 vooeitt 2_ 0 co xm1
[6] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell. Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: An- nual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 1447c0d6bfcb496774 8bf 42w.44244bbtrraaaa html
22020/hash/
[7] Yuan Cao and Quanquan Gu. Generalization error bounds of gradient descent for learning over-parameterizze deep relu networks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 3349-3356. AAAI 3330AAAALV3404.....0
Press, 2020. doi:
URL http::: org/ 10. 1609/aaa)
v34104. .5736
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani
11

---

Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1 1-00::11211111000000000010111111 13, 2023. URL http://jmIr http://mtr...oggppprssss/ wttg:pppers//v//221111111
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling cestruction-finetueed language models. CoRR, 0tbss//21..1111... 2022. doi: (0.48550/ARXIV22210 11416. URL httpp:: http://oooorrgg////// org/ 20550//rXXv... 2210. 11416
[10] Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Marc' Aurelio Ranzato, Jack W. Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 4057-4084 PMLR, 2022. URL html
https://proceedingg mlr wtt.....eeeeese////arrrrreeeeeee
[11] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/ 803.05457, 2018. http://///////iiv... http://awxxi..orr/////////////// 0005//1103000577
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14168, 2021. URL https://arxiv wrt.//gg//////// abs/a////1100 14168
[13] Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, April 2023. htthhttps://tthhhi httttp///ggiiuuu..mm///ggtteeree httpp/////gggggeeeeooooo///////a
Data
[14] Yehuda Dar, Vidya Muthukumar, and Richard G. Baraniuk. A farewell to the bias varianee trade- off? an overview of the theory of overparameterized machine learning. CoRR, abs/2109.02335 2021. URL hattps://arxiv.org/aaasss
02355
[15] Nan Duan. Overview of the nipec-icepol 2016 shared task: Open domain chinese question answering. In Natural Language Understanding and Intelligent Applications, pages 942-948. Springer International Publishing, 2016. ISBN 978-3-319-504964
[16] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23: 110:1-19,, 2022. htthtt////mml wwtpor/ppppse//////11. -9928 html
[17] Daniel Y. Fu, Tri Dao, Khaled Kamal Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré, Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: / http:///eeeeeeeeeeeeemee http://wwwwweee/.ereeeeeeeeeee http://www...e////dddodooooo////
[18] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li. Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt, Language models scale reliably with over-training and on downstream tasks. CoRR, abs/2403.0883 08550, 2024.
12

---

[19]
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, 7ac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. Predictability and surprise in large generative models. FAccT '22: 2022 ACM Conference
Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21- - 24, 2022, pages 1747-1764, ACM, 2022. doi: 10.1145/3331144.33352222 URL nttps://doil org 10. 1 3531146.11111 3533229
[20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101 100027, 2021. URL
00027
[21] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[22] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. CoRR, abs/2010.14701, 2020. URL UtLp:tts: otttt::://axxi..rrg/.a. abs///bs///0100 2010. 14701
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203, 15556, 2022. doi: 10.48550/ARXXI.2200.. 15556. URL http::t//oooooooo org/ /0 48550/arXiv. 2203. 15556
[24] Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinite resolution evaluation. arXiv e-prints, pages ar Xri--22310, 2023.
[25] Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang. Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He C-eval: A multi- level multi-discipline chinese evaluation suite for foundation models. CORR, abs/2305.08322 2023. doi: 10.48550/ARRIVV22000 08822.
//doi http://oo.oorg/////// 48550/arXii 2305
08322
[26] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Mlarie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/2310.06825. 2023. doi: 10.48550/ARXIV.23111 10.06825.. URL (tLLhttps:/// h/tp:::///ooi.oo org/ 10. 48550/ wttp../ax.ii...... 06825
[27] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large seale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pages 1601-1611, Association for Computational Linguistics, 2017. doi: 10.18653VV/P17-11477 URL wtwphhtps/// http:://dooooorg///// 1005..11///1111112222
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Sealing laws for neural language models. CoRR, abs/2001.08311. 2020. URL httpt:t/p//axiii / 083661
13

---

[29] Takeshi Kojima. Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Sammi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[30] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 -Nvember 4, 2018, pages 66-71, Association for Computational Linguistics, 2018. doi: 10. (0.18653/V1/DI1--222 URL https::/:///o /
[31] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In Martha Palmer, Rehecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785- 794. Association for Computational Linguistics, 2017. URL https: org/ 100 18653/ v1/d17-1082
[32] Hong Liu, Zhiyuan Li, David Hall, Perey Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. CoRR, abs/2305. 2023. doi: 11.48550ARRXII.230.....2.
JtL/https: //oi. org/10. 48550/arXiv 2305. 14342
[33] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conferencee on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22188-22214. PMLR, 2023. html
URL
mlr. cttres/v202/1iuse
[34] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22631-22648. PMLR, 2023. URL https::/prooeeedingg mlr prwss/ressssss.. w202 2es22lonngpre23.. html
[35] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpinns.........ssseeeee
[36] OpenAI. GPT-4 technical report. CoRR, a/s//aa3..///7//4 2023. doi: 10.48550/ARXIV.3300. 08774.
23033.87747774
[37] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HI.7 2019, Minneapolis, MN, USA, June 2-7, 2019, Demonstrations, pages 48-53, Association for Computational Linguistics, 2019. doi: 10. 1/N19-00/419 www.tttss https::t///// http::://do...o 1atp://...16533/11111111
[38] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016
[39] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional
14

---

language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett. editors, International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaiï, USA, volume 202 of Proceedings of Machine Learning Research, pages 28043-28078. PMLR, 2023. URI. https://proceedings. mlr press/v202/poli2330 html
[40] Alethea Power, Yuri Burda, Harrison Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitiing on small algorithmic datasets. CoRR, abs/2201 .02177, 2022.
hRtp::ttp:///axiiv.co http:///waxxvvoo.aaaa// www.orggg//........ http//bb/22222... 02177
[41] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susamnah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang. Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre,, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/21 14446, 2021. URL https://arxiv wwwpii/..rggabb///////a 2212. 11446
[42] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL http://jmlr htgpppapers/v21/2--74.. html
[43] Jihyeon Roh, Sang-Hoon Oh, and Young Lee. Unigran-normalizeed ieplex as a language model performance measure with different vocabulary sizes. CoRR, abs/201 1.12200, 2020. URL htttps://arxiv..rg//bbbb
13220
[44] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAA! 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. 1609/
AAAI Press, 2020. doi: 10.1609/AAALLV34155.... 63999 URL https://doi.orgg aaai. www.03.4..... 6399..
[45] Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V.
Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao. Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representatiins, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL /tt/ wwwpwewwwwwwweeeeeeeee/e hett/Forumiid--yrrbbeee
[46] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilitics of large language models mirage? CoRR, abs/2304. 15004, 2023, doi: 1044850/ARXIV.2304..55 15004. URL
https:://oi.org ot5. 2304. 15004.
[47] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for
15

---

Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguisties, 2016. doi: 10.18653/V URL
https://don http///to.or////////// org/10 18653/v1/p16-1162
[48] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmssmässan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages PMLR, 2018 httt:://pproceedngg......... mir /ttp://eeeiiias.i....rreee/////s wwwpee////aaaeeeaa html
[49] Seongjin Shin, Sang- Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woo-Myoung Park, Jung- Woo Ha, and Nako Sung. On the effect of pretraining corpora on in-context learning by a large-scale language model. In Marine Carpuat, de Marneffe, and Iván Vladimir Meza Ruíz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 5168-5186. Association for Computational Linguistics, 2022. doi: 18.18653/V 1/2022. NAACL-MAIN URL hhtpps://don wrw.oogg//oooo 10, 11663881/222222 naacl - main. 380
[50] C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1) 1904. ISSN 00029556.
[51] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexan- der W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubara- jan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. CORR, abs/2206.0466 2022. doi: 1044850/ARXXV22206.000 https://doi. org/10 100
URL
0206 04615.
[52] Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Trams. Assoc. Comput. Linguisties, 8:141-155, 2020. doi: 10.1162/TACL A\ 00305. URL
[53] Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung. Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pretraining and finetuning transformers. In The Tenth International Conference on Learning Representations, ICLR 2022, Viriual Event, April 25-29, 2022. OpenReview.net, 2022.
[54] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 12342- 12364. Association for Computational Linguistics, 2023. URL htt.:/tts//ccaaanhol..ggo
orgg/0233 findings-ennlpp 825
[55] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 2023. doi: 10.48550/ARX1122300..5999 URL https://doi. http://oo.orrgg//// 48550/ 48550/arXixv arviv. 2302. 13971.
[56]
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes. Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony
16

---

Hartshomn, Saghar Hosseini. Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Pumit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abllaaa0 2023. doi: 10.48550/ARXIV.2307.00 URL https::////oioioo ortg///////// ww..488858.iiv... 2307 09288
[57] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
net/ wwwp://wwwwwoooooneeeeeemee http://wwww.eeo/oomood/reerrrooo
[58] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus, Emergent abilities of large language models. Trans. Mach. Learn. Res., 2022, 2022. URL https////ppenneeeeeeneeefrororre yzkSU5zdwD
[59] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Shain-of-thhuugee prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022. URL http: /ttpt:::/ppapp....tt wwtp:nipss..pppppppeeeeppppppppe samper_fflla files/ ppppp/appppp////e 9d560961 1111333324441111 htt.e//ffafaatttoaaaaaaaaaaaaatr
html
[60] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@ EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94-106. Association for Computational Linguistics. 2017.
[61] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13711-13753 Association for Computational Linguistics, 2023, doi: 10. 16663//122223AALLLLLO 767. URL https org/1. /0. 18653/v1/2023 acl -lngg 767
[62]
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUF:: A chinese language understanding evaluation benchmark. In Donia Scout, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference (Online), December 8-13, 2020, pages 4762-4772. International Committee on Computational Linguistics, 2020. URL httpt:::///oom //doi. or org/ 100 18653/6633vvv v/222 http:///wm2222.....omaaaaaaaaaa 419
Computationall Linguistics, COLING 2020, Barcelona, Spain
[63] Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Zhengyan Zhang, Chaojun Xiao, Xiaozhi Wang, Fanchao Qi, Junwei Bao, Jinran Nie, Zheni Zeng. Yuxian Gu, Kun Zhou, Xuancheng Huang, Wenhao Li, Shuhuai Ren, Jinliang Lu, Chengqiang Xu, Huadong Wang, Guoyang Zeng. Zile Zhou, Jiajun Zhang, Juanzi Li, Minlie Huang, Rui Yan, Xiaodong He, Xiaojun Wan, Xin Zhao, Xu Sun, Yang Liu, Zhiyuan Liu, Xianpei Han, Erhong Yang, Zhifang Sui, and Maosong Sun. CUGE: A chinese language understanding and generation evaluation benchmark. CoRR, 10, 2021. URL httpppss/aaaav...g//////////////
17

---

[64] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluis Märquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pages Association for Computational Linguistics, 2019. URL https: /ttp://oiiioo org/ 10. 18653/v1 p19-1472
[65] Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130B: an open bilingual pre-trained model. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreviiew hwtpr/eeiiwwwntttd/ffee hetp//ff/d=-Auurrrreee
[66] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig. Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, awtpaa/.../1/00/00 2022. doi: 10448550/ARXI..225..00
URL https: /doi. org/ 10. 48550/ arrvv./axxiv. 2205. 01068
18

---

A Pre-training Settings
A.1 Pre-training Corpus
Source
Ratio
CommonCrawl
80.2%
Code
10.0%
Books
3.8%
Wikipedia
3.8%
Papers
1.6%
StackExchange
0.6%
Table 3: The ratio of different sources in the English corpus.
Our pre-training corpus is a mixture of English and Chinese documents. The ratio of English tokens Chinese tokens in the pre-training corpus is 4:1. Both the English and Chinese corpora consist of webpages, wikipedia, books, and papers. The distribution of different sources in the English corpus is shown in Table The distribution and processing pipeline are similar Redpajama [I3] During vre-training, documents from Wikipedia and Books are trained for multiple epochs, but most documents (93.4% the pre-training corpus) are never repeated.
We tokenize the data with the byte pair encoding (BPE) algorithm The vocabulary size 65k.
47 in the SentencePiece pack-
age 30)
A.2 Hyperparameters
The hyperparameters for training 1.5B, 6B, and 32B models are shown in Table The hyperpa- rameters for training of smaller models are shown in Table 5 The sequence length 2048 and the optimizer AdamW [35 with 31 = 0.9 and 82 = 0.95.
B Evaluation Settings
The evaluated splits and numbers of examples are summarized in Table For English datasets, follow Gopher [1 and Chinchilla (23'ss selection of evaluation splits. For Chinese datasets, we the validation split when the ground labels are always available. For CLUEWSC, the size of the validation set is too small (100), so we combine the train and validation splits. SSM8K-Chinees is translated from GSM8K with machine translation and human proofreading.
C Are Emergent Abilities of Language Models Mirage?
46 claim that emergent abilities proposed in discontinuos metrics. 66] also support the idea.
58 are mainly mirage caused by nonlinear and
[61] use the perplexity of correct options as the metric for BIG-Bench and find that the metric impproves smoothly almost all the tasks of BIG-Bench. We argue that the perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions. The correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options. The perplexity of correct options and incorrect options may decrease simultaneously. In fact. [61] already observe perplexity incorrect options decreasing during pre-traine pre-training and only at
Parameters
Tokens
model
hidden
heads
layers
Batch Size
Max LR
1.5B
3T
2048
6912
16
24
1344
5e-4
6B
3T
4096
13696
32
28
4224
4e-4
32B
2.5T
6656
22272
58
8832
3e-4
Table 4: Hyperparameters of pre-training of 1.5B, 6B, and 32B models.
19

---

Parameters
Tokens
model
hidden
heads
layers
Batch Size
Max LR
300M
67B
1152
3840
9
12
1152
2.8c-3
300M
125B
1152
3840
9
12
1152
2.8c-3
300M
250B
1152
3840
9
12
1152
2.8e-3
300M
500B
1152
3840
9
12
1152
2.8e-3
540M
33B
1536
5120
12
12
1152
2e-3
540M
66B
1536
5120
12
12
1152
2c-3
540M
125B
1536
5120
12
12
1152
2e-3
540M
250B
1536
5120
12
12
1152
20-3
540M
500B
1536
5120
12
12
1152
20-3
IB
33B
2048
6912
16
16
1152
1,5e-3
IB
67B
2048
6912
16
16
1152
1,5e-3
IB
125B
2048
6912
16
16
1152
1,5e-3
IB
250B
2048
6912
16
16
1152
1,5e-3
IB
500B
2048
6912
1152
1.5e-3
1.5B
67B
2048
6912
16
24
1152
le-3
1.5B
100B
2048
6912
24
1152
1e-3
1.5B
125B
2048
6912
16
24
1152
1e-3
1.5B
250B
2048
6912
24
1152
1e-3
1.5B
375B
2048
6912
16
24
1152
1e-3
1,B
500B
2048
6912
24
1152
le-3
3B
67B
3072
10240
24
24
1152
7e-4
3B
125B
3072
10240
24
1152
7e-4
3B
250B
3072
10240
24
24
1152
7e-4
3B
500B
3072
10240
24
24
1152
7e-4
6B
33B
4096
13696
32
28
1152
4e-4
6B
67B
4096
13696
32
28
1152
4e-4
6B
125B
4096
13696
32
28
1152
4e-4
6B
250B
4096
13696
32
28
1152
4e-4
Table 5: Hyperparameters of pre-training of smaller models. Each line represents one model pre- trained completely from scratch with the certain number of tokens and its corresponding learning rate schedule.
Dataset
Evaluated Split
Num. Examples
TriviaQA
validation
11,313
HellaSwag
validation
10,042
RACE
test
4,934
WinoGrande
validation
1,267
MMLU
test test
14,042
GSM8K
1,319
NLPCC-KBQA
validation
10,613
ClozeT
validation
938
CLUEWSC
train & validation validation
508
C3
3,816
C-Eval
validation
1,346
GSM8K-Chinese
test
1,212
Table 6: Statistics of evaluation datasets.
the end of training that the perplexity of correct and incorrect options starts to diverge. This supports the existence of emergent abilities.
46 use Brier Score as the metric for BIG-Bench. argue that increase in Brier Score does not always represent improvement of performance on the multi-choice task, since Brier Score is also related to the allocation of probabilities for incorrect options. For example, questions the MMLJU datasct have four options (A, B, C, and D) and the frequency of the four options as correct is equal.
20

---

Consider two models that give the same probability independent of questions. One model predicts (1,0,00 O, 0) for the four options and the other model predicts (0.25, 0.25, 0.25, 0.25). The Brier Score for the former is 1.5 while the Brier Score for the latter is 0.75. However, both models do not learn the relationship between questions and correct options at all. One can argue that the latter model better fits the distribution of correct options in the dataset, but the improvement is not as large as the different 1.5 and 0.75, We should consider the Brier Score of 0.75 as the performance of the random guess baseline, and any decrease in Brier Score above 0.75 should not be considered as the real improvement on the task.
In Figure 6 of 46], they evaluate 4 tasks in BIG-Bench with the Brier Score metric and find that the emergent abilities disappear. We hypothesis that they normalize the Brier Score with the number of options in each question, otherwise the Brier Score of 0.25 on the swahili english proverbs task is too low for the smallest model. Four tasks have 2, 2, 4, options each question. The values of Brier Score for random guess baselines on the four tasks are 0.25, 0.25, 0. 0875, and 0.16. Only the largest model surpasses the random guess baseline. This also supports the existence of emergent abilities.
D Complete Performance-vs-Loe Curves of Smaller Models
TriviaQA
HellaSwag
RACE
WinGGrande
TOOM
65.0
540M
60
Performance
3.58
40
50.0
50
57.5
55.0
52.5
25
NLPCC-KBOA
ClazeT
C3
CLUEWSC
37.5
15.0
70
Performance
12.5
65
10.0
7.5
60
55
30
0.0
50
50
MMLU
C-Eval
GSMBK
GSMBCChinese
40.5
40.0
20.0
20.0
37.5
37.5
Performance
30.0
27.5
25.0
25.0
5.0
22.5
22.5
2.5
2.5
20.3
2.75
2.50
20.0
2.75
2:50
0.0
2.75
2.50
2.25
2.75
2550
2.25
Training
Training
Training
Figure 5: The complete performance-vs-loss curves of smaller models.
The performance-vs. lo-loss curves for all the intermediate checkpoints are shown in Figure The trend is the same as Figure? 2 but with larger variance.
E Loss vs Compute as an Indicator of Performance
We show the performance-compee curves in Figure Compared with Figure 1 we observe that points from different models do fall on the same curves on most tasks. This proves that pre-training loss is a better indicator of task performance than compute.
21

---

TriviaQA
HellaS
RACE
WinoGrande
1.58
80-
Kdoof
60
13
70-
Perforance
ranidom
45
63-
20
10
10
100
15at
1022
100
102
102
1093
1021
10
159
80
Performanc
60
70
65
30-
30
10
1001
1002
10
10
10
10
1071
102r
1007
MMLU
C-Eval
GSMBK
CSMBKCCinesS.
40
Performance
50-
20-
10
10-
Training Compute
Training Compute
Training Compute
Training Compute
Figure 6: The performance--s--ooppuc curves of 1.5B, 6B, and 32B models.
F Pythia's Loss vs. Performance
To further support our conclusion, we plot the performance-loss curves of Pythia B in Figure 7 Pythia is suite of open language models with intermediate checkpoints released. For downstream tasks, we select SciQ 600, LAMBADA (38) WinoGrande 44 ARC-Easy [11, ARC-Challenge [11, and PIQA 4 with reported performance in the official repository. We compute the cross-entropy loss of intermediate checkpoints on the corpus Pile (20). From the plot, we can observe that the points from different models fall on the same curve all the tasks. This supports our conelusion that pre-training loss is predictive of task performance.
However, neither Pythia nor LLaMA can be used amalyze the emergent abilities. The largest Pythia model fails to achieve performance above random chance on MMLU and GSM8K [2]. Instead, LLaMA has intermediate checkpoints released and performance curves MMLU and GSM8K are not available.
G Loss vs. Performance on BIG-bench
series of diverse tasks designed to evaluate the capacities and limitations of pre-trained language models. Wei et al. [58) find that large language models exhibit emergent abilities on four tasks from BIG-Bench. Among the four tasks, the test set size the figure-of-speech detection task is too small and the variance is too high. We evaluate the other three tasks in the same setting as Section 2.3 and the results are shown in Figure g With pretraining loss decreases along the x-axis, clearly observe the tipping point in the performance curves.
BIG-bench [11
22

---

LAMBADA
WineGrande
0.66
0.95
0.65
0.64
0.94
Peformance
0.93
0.60
0.92
0.58
0.91
0.56
0.54
ARC-Easy
ARC-Challenge
PIOA
0.70
0.36
0.76
Perforance
0.68
0.34
0.74
0.64
0.72
5
0.60
600
0.70
gythia-12b
Pythia 6.90
0.56
0.08
pytha-1 46
1.9
1.8
1.6
Training Loss
Training Loss
Training Loss
Figure 7: The performance-vs-los curves of Pythia. The performance of Pythia is from the official repository and the loss is evaluated with the released checkpoints.
Word unscramble
Mod, arithmetic
IPA transliterate
1.58
60
6B
35
Performance
random
40
25
30
15
2-
10
0
2.00
Training Loss
Training Loss
Training Loss
Figure 8: The performance-vs-lose curves of 1.5B, 6B, and 32B models on tasks in BIG-bench. Each data point is the loss (æ-axis) and performance (y-axis) of the intermediate checkpoint of one of the three models. We mark the results of random guess in black dashed lines.
H Compute Resources
All the models are trained on DGX-A100 GPU (8x80G) servers. The 1.5B. 6B, and 32B models in Section 2.3 take 8 days on 256 A100 GPUs, 8 days on 1024 A100 GPUs, and 20 days on 2048 A100 GPUs respectively. The small models in Section 2.4 take about 20 days on 256 A100 GPUs.
I Broader Impact
This paper finds that pre-training loss is predictive of downstream task performance and some tasks the performance only begins to improve when the pre-training loss falls below a certain threshold. Combined with previous works scaling laws 22 22 23 we predict the amount of compute required to achieve a certain performance. This can be used estimate the cost training large model.
The paper might encourage companies to expand model sizes and data sizes of language models beyond current scales to pursue new emergent abilities, leading to waste of compute resources. We want to emphasize that the analysis of previous performance trends do not necessarily apply to the larger models.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the main claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The model are pre-trained with the open-source framework, Megatron-LM and we provide the training hyperparameters. The datasets can be replaced with open-source counterparts, such RedPajama.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky some cases, in which case authors are welcome to describe the particular way they provide for reproducibilityy the case of closed-source models, may be that access to the model limited in some way (c.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
25

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The training code is Megatron-L.M The pre-training dataset cannot be released since it contains proprietary content.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines tps //nips cttpubcccclbccceccc ht
httgg//ddde//deeuuuieeee / ssi sww.booliisssoooooocccccooo onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of pre-training are described in Appendix and those of evaluation are described in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: is too computationally expensive to train multiple pre-trained language models from scratch.
Guidelines:
The answer NA means that the paper does include experiments.
26

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper does not include human subjects or participants.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks since does not include release of data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators of code, datasets, and models used in the paper are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, ppaper thcode. cww/cooo////oooooo coomdatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "KiytQWTp9F", "reviewer": "Reviewer_RA4k", "summary": "This paper investigates emergent abilities in language models from the perspective of pre-training loss, rather than model size or training compute. The authors challenge recent skepticism about emergent abilities by demonstrating that: (1) Models with the same pre-training loss, regardless of model and data sizes, exhibit similar performance on various downstream tasks. (2) Emergent abilities manifest when a model's pre-training loss falls below a specific threshold, before which performance remains at random guessing levels. \nThe study examines different metrics (including continuous ones) and proposes a new definition of emergent abilities based on pre-training loss. The authors argue that this perspective better represents the learning status of language models and provides a more precise characterization of when emergent abilities appear during training.", "questions": "None.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper offers a fresh approach to understanding emergent abilities by focusing on pre-training loss rather than model size or compute, providing valuable insights into the scaling behavior of language models.\n- The paper builds upon existing scaling laws and provides a mathematical formulation (Equation 5) that explains the relationship between model size, pre-training loss, and emergent abilities.\n- The study directly addresses recent challenges to the concept of emergent abilities, providing a nuanced perspective that reconciles conflicting observations in the field.", "weaknesses": "- The authors acknowledge that they have not considered fundamentally different model architectures (e.g., routed Transformers) or non-Transformer architectures. This limitation may affect the generalizability of their findings.\n- As noted in the limitations section, pre-training loss is affected by tokenizers and pre-training corpus distribution, making direct comparisons between models trained on different corpora challenging. While the authors suggest using normalized perplexity on a public validation set, this solution is not implemented in the current study.\n-  While the paper establishes a correlation between pre-training loss and emergent abilities, it does not provide a causal explanation for why certain abilities emerge at specific loss thresholds. A deeper investigation into the underlying mechanisms could strengthen the paper's contributions.\n- While the authors discuss some contrary observations from previous studies, a more comprehensive comparison with existing literature on emergent abilities and their proposed explanations would enhance the paper's positioning within the field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We admit the limitations of the work in different model architectures, tokenizers, and pre-training corpus distribution. The main reason is that the work analyzes not only the final checkpoints of different models, but also the intermediate checkpoints, which are not publicly available for many open-source models. Training multiple models with different architectures, tokenizers, and pre-training corpus is beyond our available computational resources. We also want to argue that previous works on scaling laws often study a fixed choice of architectures, tokenizers, and pre-training corpus [1][2].\n\nWe also admit the lack of theoretical analysis in the paper. The main question the paper answers is whether the emergent abilities exist and how to track them, which is necessary for further study on emergent abilities. We believe the paper can inspire more theoretical work on the topic.\n\n1. Kaplan et al. Scaling Laws for Neural Language Models.\n2. Hoﬀmann et al. Training Compute-Optimal Large Language Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2VfDPDOtGM", "reviewer": "Reviewer_NWes", "summary": "The paper investigates the link between the pre-training loss of LLMs and their downstream performance on popular benchmarks. The authors train a series of models ranging from 300M to 32B parameters on English-Chinese datasets of different sizes and study how these models perform on TriviaQA, HellaSwag, RACE, WinoGrande, MMLU and GSM8K benchmarks and their Chinese counterparts, as a function of their pre-training loss. They observe a strong link between pre-training loss and benchmark performance that is not affected by model size and dataset size, from which they conclude that pre-training loss is the main indicator of downstream performance. On MMLU and GSM8K (and their Chinese counterparts), models exhibit an emergent behavior wrt. loss, i.e. their performance is close to random before a loss value of about 2.2 and steadily increases for lower loss values. This relationship seems to hold, even under continuous evaluation metrics.", "questions": "1. The paper claims that pre-training (test) loss is a good indicator of downstream performance. What is the composition and size of the test set that the loss is evaluated on? What properties (size, diversity, etc.) should such a test-set have in order to be predictive of downstream performance?\n2. Prior work [1, 2] has found that emergent behaviors can become predictable with \"higher resolution\" benchmarks, i.e. with larger test sets or repeated sampling from the model. This is a dimension that the paper does not touch upon. Do the authors believe, that the studied emergent abilities would still appear only after models cross the particular loss threshold, i.e. be essentially 0 before, even with those higher resolution benchmarks?\n3. In related work (line 254), the authors mention that some prior work [3] has observed a disconnect between pre-training loss and downstream performance, which stands in contrast to the claims made in the paper. It would be great if the authors could comment on more on the reasons for these discrepancies.\n\nReferences\n- [ 1] Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., https://arxiv.org/abs/2304.15004\n- [ 2] Predicting Emergent Abilities with Infinite Resolution Evaluation, Hu et al., https://arxiv.org/abs/2310.03262\n- [ 3] Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models, Liu et al., https://proceedings.mlr.press/v202/liu23ao.html", "limitations": "The paper sufficiently discusses limitations, in my opinion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Formulating downstream (emergent) abilities in LLMs in terms of pre-training loss is a valuable contribution, that can unify several different factors that are so far thought to contribute to emergence, such as pre-training compute, model size, and (test-)perplexity. This perspective also provides a better connection between scaling laws -- which typically describe the relationship between parameter count, dataset size, compute, and pre-training loss -- and emergent abilities, which mostly have been studied through the lens of parameter count and compute so far.\n2. The paper presents an extensive evaluation over a large range of model sizes, dataset sizes, compute budgets, that show a strong connection between pre-training loss and downstream performance. The results are additionally validated with results and models proposed by prior work (Llama-1 and Pythia).\n2. The paper is easy to understand and follow.", "weaknesses": "1. The paper seems to make two main claims: 1) that pre-training loss is the main predictor of downstream performance, and 2) that emergent abilities appear suddenly after models cross a certain loss-threshold. I find the evidence for 1) to be convincing, but I am somewhat less confident about 2). This is mostly because the paper only shows emergent abilities on a subset of the benchmarks they were originally proposed on (MMLU and GSM8K) in [1]. Notably, BigBench is absent. Including results on the tasks in BigBench that were shown to exhibit emergent behavior would be helpful to judge whether pre-training loss predicts the emergence threshold as well as pre-training compute or model size. Including these benchmarks would also help to put the paper into a better perspective wrt. subsequent work questioning the existence of emergent abilities [2], which also studies these benchmarks.\n2. There are some discrepancies and unexplored links between the results reported in the paper and findings in prior work. To develop a better understanding of how the findings here relate to prior work, it would be helpful to include a discussion in the paper. See Questions 2. and 3. below for more details.\n3. I find the paragraph in 3.1, line 197 rather speculative. Grokking refers to an improvement on the test-set, despite stagnation on the training set, whereas the models studied here seem to still be improving on the training set. I am not sure whether there is a link between grokking and the emergent behavior on the specific tasks here.\n4. Minor: There are a number of typos and small writing issues, e.g.\n    - \"has thus far less studied\" (line 23), \", a highly over-parameterized models\" (line 79), \"coefficients how that\" (line 126),...\n    - line 120: \"climbs as the loss increases\" should probably be \"decreases\".\n    - line 141: \"On each line\" seems to be missing a reference to Figure 2.\n\nReferences\n- [ 1] Emergent Abilities of Large Language Models, Wei et al., https://arxiv.org/abs/2206.07682\n- [ 2] Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., https://arxiv.org/abs/2304.15004", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For weakness 1, the results on BIG-Bench are presented in the PDF of global rebuttal. The original emergent ability paper evaluates 4 tasks on BIG-Bench. The test set size of the figure-of-speech detection task is too small and the variance is too high. Therefore we evaluate the other three tasks.\n\nFor question 2: We already study the effect of continuous metrics in section 3.2, in which we evaluate MMLU and C-Eval with two continuous metrics, one of which is the Brier Score used in Schaeffer et al. The results show that continuous metrics cannot eliminate the observed tipping point in the performance curves. Increasing the size of the test set of GSM8K is not straight-forward. We will try repeated sampling from the model, but it takes some time. We think repeated sampling does not change the results since it only decrease the variance of the performance evaluation.\n\nFor question 3: Liu et al. mainly analyze BERT-like models, which are pretrained on masked language modeling and finetuned and evaluated on supervised classification tasks, which we denote as \"transfer learning setting\" in the paper. Implicit bias in model sizes and training algorithms can change the transferability of pretrained knowledge. Our work focuses on GPT-like models, which are pretrained on autoregressive language modeling and evaluated on prompted tasks without finetuning. Since the pretraining and evaluation settings are more consistent, the transferability is less important.\n\nFor weakness 3, pretraining is generally considered as multi-task learning. It is possible that the model already stagnates on some tokens in the pretraining corpus, such as digit calculation, while the overall pre-training loss is still decreasing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a8Mrkk9fcX", "reviewer": "Reviewer_ZLu4", "summary": "They demonstrate that 1) pre-training loss is generally predictive of downstream capabilities in language models, rather than models size or number of tokens used during pre-training; and 2) emergent capabilities can also be clearly described in terms of pre-training loss. They also demonstrate that using continuous metrics, they still observe emergence, countering findings from prior work. They conduct their analysis on a suite of standard English and Chinese evals, using a range of models that they pre-trained themselves. They also further validate their findings using the llama and Pythia series of models.", "questions": "See the weaknesses for most of my questions/concerns.", "limitations": "No major limitations. See the weaknesses section.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* The paper does a good job of correcting an extremely prominent (but largely incorrect) narrative/claim that the phenomenon of emergence in LLMs always completely disappears when a continuous metric is used in place of a discontinuous one. This paper demonstrates a few cases where even in the presence of a continuous metric, emergence is still observed. Correcting this misconception in the scientific discourse is important.\n* The paper is overall well written and the experiments are very thorough, encompassing a range of models and tasks.\n* They even went out of their way to ablate the effect of learning rate schedule on their findings in Section 2.4, this is extremely thorough work, and they should be commended for it.\n\nI have a handful of concerns regarding the writing, as discussed in the weaknesses section, but on balance I think this paper should be accepted, on the condition that my concerns are fixed in the camera ready.", "weaknesses": "* From the intro: \"For example, LLaMA-13B with less compute [53] can outperform GPT-3 (175B) on MMLU [21]\": Training for longer could be one explanation for this, another one could be llama used a higher quality corpus than gpt-3 did. I think the data quality element is a little underrated. In general we should expect different pretraining datasets to potentially have different loss / different emergence points (even if the x-axis loss is on a held out validation set and models have the same vocabulary). There is some discussion of this in the limitations section, but in general I think this point should be highlighted more in the paper. Concretely, the pretraining loss -> emergence point phenomenon is only guarenteed to be consistent for a series of models pretrained on the same data.\n* The discussion of exact match in Section 2 kind of comes out no nowhere, and I'm not sure why it is discussed where it is. It would be great if there were more motivation for this in the paper.\n* I understand why you did the ablation that you did in Section 2.4 (ablate the effect of lr schedule, as in Chinchilla), but in the paper this is not very clearly motivated and readers with less background knowledge might not understand why this ablation is important.\n* “Note that there is one only exception at the early stage of LLaMA-65B. We can see that when the training loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss.”: This could be because they used some exponential smoothing on either their loss or downstream performance plots. Exponential smoothing would perturb the earlier points more than other points (especially for curves that are fast changing or have a rapid change at the beginning), potentially leading to this effect. Moreover, did you smooth out the loss in some way when plotting loss verses downstream in the previous sections? If so, this would at least be good to note in the paper.\n* Section 3.1 could use clearer discussion of explanations for why emergence occurs on the different tasks. I think grokking is a possible explanation, but what exactly grokking means (outside of the context of the simple algorithmic tasks presented in the original grokking paper) is a little vague/imprecise, and I also don't think this is the only explanation. In the case of gsm8k, emergence is, in my opinion, more likely due to the fact that the model has to get a sequence of reasoning steps correct to answer the question, this leads to an exponential (e.g. p(step_correct)^n_steps) for getting the full answer correct (some analysis of this in the paper could be cool). In the case of MMLU and C-Eval there actually is an existing paper which discovers an explanation, using interpretability techniques, for emergence on multiple choice tasks (https://arxiv.org/pdf/2307.09458).\n* There's a handful of related works that aren't discussed in the paper and whose findings contradict those of this paper (I'll give you my reasoning for why your work might not contradict theirs, as a suggestion in parentheses, but you should include some discussion of this in the paper somewhere, either in the related work or elsewhere):\n1) https://proceedings.mlr.press/v202/liu23ao.html (my understanding is that their theory only applies near the global optim, which is generally not the case with real-world language models)\n2) https://arxiv.org/pdf/2109.10686 (see the paragraph \"zooming in verses zooming out\", your paper is more about the zoomed out setting)\n3) Not strictly language modeling, but see Figure 3 of https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf (could be because they are doing probing so hidden-state size increases probe capacity causing differences with model size, but it's unclear if this is actually the case)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We agree with the influence of data quality on performance besides compute. We will add the point in the introduction when making the comparison.\n\nThank the reviewer for appreciating the ablation of learning rate schedule. We will make it clearer in the final version.\n\nThe exponential smoothing is one possible explanation for the outliers on LLaMA-65B. We will add the explanation to the paper. Also, we didn't use any smoothing in all the plots in the paper.\n\nThank the reviewer for providing related works on explanations of emergent abilities. We will add these explanations in the paper. For contradictory conclusions of previous works [1][2], we think they mainly study the pretraining-finetuning paradigm, in which inductive bias helps improve transferability. Instead, we study the pretraining-prompting paradigm without finetuning on specific tasks, which is more common in language model pretraining. We will add more discussion about this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4NBxrwfOkI", "reviewer": "Reviewer_BK9j", "summary": "This paper formulates the concept of emergence as a relationship between language modeling training loss and task performance, rather than in relation to model/data scale.", "questions": "Could you discuss a little further why your findings are so at odds with existing work like Schaeffer et al.? As far are as I can tell, it is because you've decided to define emergence without accounting for straightforward metric thresholding effects.", "limitations": "No obvious unlisted limitations.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "The paper is clear. I appreciate the central message, as descriptions that relate the capabilities strictly to the scale of a language model overlook factors such as data quality and architecture.  If you actually demonstrated that loss was a better predictor than model scale on specific capabilities, the paper would have a lot of value.\n\nThe paper is partly making the point of \"agreement on the line\" https://arxiv.org/abs/2206.13089 that in-distribution loss tends to be strongly correlated with out of distribution or specialized metrics. This point is worth making. The claim that we can identify emergence with respect to loss is more dubious, as the threshold-based measurements used are overly generous to claims of emergence in general.", "weaknesses": "Even when different models have very similar pretraining loss, they may have extreme variation in other qualities, including specific benchmarks. This paper does not engage substantially with the existing literature on the connection or lack thereof between loss and specialized metrics; it nods to them in the literature review on \"Relationship of Pre-training Loss and Task Performance\" but does not clarify why their findings are so different from existing results.\n\n>We argue that the perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions. The correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options. \n\nI looked into this appendix because I was intrigued as to why their results are so strongly at odds with Schaeffer et al. I completely disagree with this claim (which the paper provides no support for); if perplexity is the wrong metric, why measure loss at all? To the contrary, recent work including https://arxiv.org/pdf/2404.02418 and https://arxiv.org/pdf/2305.13264 continue to support the notion that claims of emergence should consider probabilities rather than exact matching, which obscures auxiliary task dependencies such as the capability of answering in multiple choice format. In general, our community now understands how much emergence can be attributed to metric thresholding effects, so you need much stronger evidence to argue against Schaeffer et al. \n\nThe core claim that one can predict performance on any given task just from the loss of a model would be a more useful finding if it applied across different architectures, not just different data and model scales. Although the paper considers both Llama and Pythia, they do not plot these different architectures on the same figures and it is therefore not clear whether loss is similarly predictive across architectures or whether it is only within an architecture, in which case there is no real argument for preferring it over scale as a basis of measurement. There is no excuse for continuing to present metric thresholding effects as evidence of meaningful emergence.\n\n> That is, by ignoring the color differences (model sizes), the data points of different models are indistinguishable. \n\nThis simply doesn’t seem to be true. RACE in particular does seem to have a different slope for green compared to the other colors. For CLUESC orange and blue seem to be nearly separable, although that could be random. Still seems to be true for a bunch of these and it’s an interesting point to make, but I think you need to qualify it by actually comparing the slopes from a linear regression for each color.\n\n> an ability is emergent if it is not present in language models with higher pre-training loss, but is present in language models with lower pre-training loss.\n\nNot clear what it means to claim any ability is not present and then is present. It seems that they mean that below a certain point you have random chance answers. So by definition, if you perform at random chance at any point, then they would say the ability is emergent regardless of whether there is a clear breakthrough. This is the definition of a thresholding effect.\n\nMinor:\n- Please use natbib with \\citet when you have an inline citation.\n- \"emergency trends\" should be emergent\n- > until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss increases.\n    - Do you mean the loss decreases? Or am I missing something? Why would the loss increase after this point?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to argue that the reviewer misunderstands the difference between our work and Schaeffer et al. In fact, Schaeffer et al do not use perplexity as the evaluation metric for multi-choice tasks. Instead, they propose to use Brier Score, which we also evaluate in Figure 4. The main difference is that Schaeffer et al only evaluate the final checkpoints of different models while we also evaluate the intermediate checkpoints. With more data points available, we show that emergency on some tasks exists even with continuous metrics like Brier Score. \n\nWe are against evaluating multi-choice tasks with the perplexity of correct options because we think a good metric should reflect the ability of distinguishing correct options from incorrect options. However, it is not the main argument of the paper and also not the reason why our findings are different from Schaeffer et al.\n\nWe prefer loss over compute/scale since it can better track the learning dynamics of intermediate checkpoints. Loss of intermediate checkpoints is influenced by many factors beyond compute/model scale, such as learning rate scheduling. We show the performance-vs-compute curves in Figure 6 and find that points from different models do not fall on the same curves on most tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WiBDERW6db", "reviewer": "Reviewer_kB8g", "summary": "This paper measures a range of language models' pretraining loss and their downstream performance, arguing that emerging capabilities are better measured by losses as opposed to previously proposed model size or compute (FLOPs). The paper also offers evidence against the argument that emergent abilities are a \"mirage\" when the task metrics are continuous.", "questions": "1. Is EM the same as perplexity/scoring eval?\n1. Why use the training loss as opposed to validation/test loss?", "limitations": "L130\n> we find that the overall training loss is a good predictor of performance on both English and Chinese tasks,\nNaturally, you could compute the per-corpus loss on the x-axis to strength the claim. For the final version, it'd be great to have a per-corpus loss analysis on what kinds of corpora are most predictive of downstream performance.\n\nNot limitations just typos:\nLine 128: verifying the emergency of performance -> emergence\n\nLine 218: ground probability -> did you mean ground truth probability?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Clean and comprehensive study on a range of tasks, different models and different model sizes, with a significant number of intermediate checkpoints.\n1. Insightful results that builds on top of prior literature: (I) Loss > size and FLOPs and (2) It's not just that downstream task metrics needed to be continuous. Line 226 also makes a good observation on a quirk of Brier Score. \n\nLine 122 makes a thought-provoking claim that many researchers should seriously consider, especially when considering the inference cost of these models during architectural design.\n> That is, by ignoring the color differences (model sizes), the data points of different models are indistinguishable… . This indicates that the model performance on downstream tasks largely correlates with the pre-training loss, regardless of the model size.", "weaknesses": "1. The authors define emergence as a discontinuity, although it seems possible to fit an exponential function (especially when the x-axis is FLOPs as in Figure 6), or a linear function for many tasks such as the first group in section 3.1. Although Figure 4 offers stronger evidence for discontinuity, it is not definitive, as only MMLU, GSM8K and C-Eval are measured (but I accept that these are reasonably sufficient for an academic paper). \n1. The authors discuss why the harder nature of MMLU and GSM8k may be causing the discontinuity, but it would be useful to present a qualitative analysis on the \"easier\" questions that models \"first\" get right with just a bit more reduction in pretraining loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For weakness 1, we add results on three tasks on BIG-Bench in the PDF file of global rebuttal.\n\nFor question 1, when the language model directly predicts the answer, EM and perplexity give the same trend. However, with chain-of-thought reasoning, the language model first predicts the intermediate reasoning process then gives the answer. Perplexity evaluation does not work.\n\nFor question 2, most of our pretraining corpus go through fewer than one epoch during pretraining and in the early experiments we find that the training loss is consistent with the validation loss. Therefore we use the training loss for simplicity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The results on BIG-Bench are presented in the PDF file. The original emergent ability paper evaluates 4 tasks on BIG-Bench. The test set size of the figure-of-speech detection task is too small and the variance is too high. Therefore we evaluate the other three tasks. We can observe with pretraining loss as the x-axis, we can clearly observe the tipping point in the performance (compared with Figure 2 in Wei et al.)

Meta-review: This work demonstrates that 1) pre-training loss is generally predictive of downstream capabilities in language models, rather than model size or number of tokens used during pre-training; and 2) emergent capabilities can also be clearly described in terms of pre-training loss. They also demonstrate that using continuous metrics, they still observe emergence, countering findings from prior work. They conduct their analysis on a suite of standard English and Chinese evals, using a range of models that they pre-trained themselves. They also further validate their findings using the Llama and Pythia model families.

I am not going to factor in the review of Reviewer RA4k, as it is too short and the reviewer did not engage further either in discussions with the authors or discussions with the other reviewers.

This work has the potential to be a timely contribution on understanding emergent properties in LLMs, if the results support the conclusions drawn by the authors. All reviewers other than ZLu4 were initially skeptical at least to some degree about the evidence that the authors provide for one of the two major claims in this work -- that a specific pre-training loss threshold is needed to lead to emergent abilities in LMs. The authors had lengthy exchanges with the reviewers and most reviewers were satisfied by the author response. There are still some outstanding concerns from Reviewer BK9j, which I urge the authors to take seriously, specifically regarding whether all modes indeed lie on the same general curve, especially because there is little overlap between the loss ranges for different model size. Despite this remaining concern, there is strong support for this work from the remaining reviewers, and I believe this work will at least serve to deepen the discussion on the emergent properties of LLMs. I suggest that the authors incorporate the reviewers' suggestions in a revised manuscript, particularly emphasizing the role of pretraining data and clarifying the comparison with and discussion of previous work.
Decision: Accept (poster)