Understanding Emergent Abilities of Language Models from the Loss Perspective

Authors: zhengxiao  du
aohan  zeng
yuxiao  dong
jie  tang
zhipu  ai
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:44:58

Abstract: Recent studies have put into question the belief that emergent abilities in language models are exclusive to large models. This skepticism arises from two observations: 1) smaller models can also exhibit high performance on emergent abilities and 2) there is  doubt on the discontinuous metrics used to measure these abilities. In this paper, we propose to study emergent abilities in the lens of pre-training loss, instead of model size or training compute. We demonstrate that the models with the same pre-training loss, but different model and data sizes, generate the same performance on various downstream tasks. We also discover that a model exhibits emergent abilities on certain tasks---regardless of the continuity of metrics---when its pre-training loss falls below a specific threshold. Before reaching this threshold, its performance remains at the level of random guessing. This inspires us to redefine emergent abilities as those that manifest in models with lower pre-training losses, highlighting that these abilities cannot be predicted by merely extrapolating the performance trends of models with higher pre-training losses.

Main paper: 
 introduction: 
 Scaling of language modes (LMs) on both model and data sizes has been shown to be effective for improving the performance on a wide range of tasks [42, 6, 23, 8, 65, 55, 36] , leading to the widespread adoption of LM applications, e.g., ChatGPT. The success of such scaling is guided by scaling laws [22, 28, 10, 23] , which study the predictability of pre-training loss given the model and data sizes.While scaling laws focus on the pre-training loss, the scaling effect on the performance of downstream tasks has thus far less studied. Emergent abilities [58] are defined as abilities that present in larger LMs but not present in smaller one. The existence of such abilities is recently challenged for two reasons. First, small LMs trained on a sufficient amount of high-quality data can outperform large models on tasks with claimed emergent abilities [55, 56, 26] . For example, LLaMA-13B with less compute [55] can outperform GPT-3 (175B) on MMLU [21] , due to more training tokens and improved data-filtering methods. Second, Schaeffer et al. [46] claim that emergent abilities appear due to the nonlinear or discontinuous metrics selected to evaluate certain datasets, rather than from a fundamental change in larger models.The Chinchilla scaling laws [23] show that different combinations of model sizes and data sizes can lead to different pre-training losses even with the same training compute. Consequently, the pre-training loss can naturally better represent the learning status of LMs than the model or data sizes. However, the relationship between the loss of an LM and its performance on downstream tasks is not yet well understood. Existing literature has either focused on the transfer learning paradigm [33, 54] or constrained its study to single models, tasks, or prompting methods [49, 61] .In this work, we propose to study emergent abilities from the perspective of pre-training loss instead of model size or training compute. To examine the relationship between the pre-training loss of LMs and their performance, we pre-train more than 30 LMs of varied model and data sizes from scratch, using a fixed data corpus, tokenization, and model architecture. Their downstream performance is evaluated on 12 diverse datasets covering different tasks, languages, prompting types, and answer forms. We demonstrate that the pre-training loss of an LM is predictive of its performance on downstream tasks, regardless of its model size or data size. The generality of this conclusion is further verified by extracting and observing the performance and loss relationship of the open LLaMA [55] and Pythia [3] models.Over the course, we find that performance on certain downstream tasks only improves beyond the level of random guessing when the pre-training loss falls below a specific threshold, i.e., emergent abilities. Interestingly, the loss thresholds for these tasks are the same. When the loss is above this threshold, performance remains at the level of random guessing, even though performance on other tasks continues to improve from the outset. To exclude the impact of discontinuous metrics [46, 61] , we evaluate the emergent performance increase using continuous metrics and show that the emergent abilities persist across both discontinuous and continuous metrics.Based on these observations, we define the emergent abilities of LMs from the perspective of pretraining loss: an ability is emergent if it is not present in language models with higher pre-training loss, but is present in language models with lower pre-training loss. According to the loss scaling laws [22, 28] , the pre-training loss is a function of model size, data size, and training compute. Therefore, the new emergent abilities can also account for the previously-observed emergent abilities in terms of model size or training compute.The advantage of the new definition lies in its ability to better capture the tipping points in training trajectories when LMs acquire emergent abilities. Once again [58] , the existence of emergent abilities suggests that we cannot predict all the abilities of LMs by simply extrapolating the performance of LMs with higher pre-training loss. Further scaling the model and data size to lower the pre-training loss may enable new abilities that were not present in previous LMs. 
does pre-training loss predict task performance?: 
 Table 1 : English and Chinese datasets evaluated in the experiment, and their task types, prompting types, answer forms and metrics. For prompting type, we refer to the chain-of-thought prompting [59] as few-shot CoT and the original in-context learning prompting [6] as few-shot.It is not straightforward that the loss of LMs decides the performance on downstream tasks. Generally the performance is decided by the probability to predict the ground truth y given the prompt x, i.e. p(y|x). The probability can be written as a function of the cross entropy loss:EQUATIONwhere ℓ(y|x) is the cross entropy loss of the LM given the context x and the target y. While ℓ(y|x) has the same form as the pre-training loss L, they are not equal. First, the pre-training loss is an average of all the tokens in all the documents pre-trained on. According to our empirical observation, the losses of different documents are not uniform. Second, if x and similar documents do not exist in the pre-training corpus, ℓ(y|x) is the generalization loss, which is often related to other factors beyond the training loss, such as the model size. For example, in computer vision, a highly over-parameterized models often improve over an under-parameterized models in test performance when both models converge on the training data [14, 7] .All the models are pre-trained on a mixture of English and Chinese corpus. The ratio of English to Chinese is 4:1 in the pre-training corpus. The model architecture is similar to LLaMA [55] with two differences: we use grouped-query attention [1] to replace the multi-query attention and we apply rotary position embedding on half the dimensions of the query and key vectors. More details can be found in Appendix A.To present a comprehensive demonstration, we evaluate the pre-trained models on 12 datasets across different tasks and prompting types in both English and Chinese. The six task types include:Closed-book QA: Answering questions about the real world based solely on the pretrained knowledge. We use TriviaQA [31] for English. For Chinese, we build a closed-book QA dataset based on NLPCC-KBQA [15] dataset following the TriviaQA format.Commonsense Natural Language Inference (NLI): Selecting the most likely followup given an event description. We use the HellaSwag dataset [64] for English and the ClozeT dataset in Yao et al. [63] for Chinese.Reading comprehension: Reading a given article or paragraph and answering questions about it. We use RACE [31] for English and C3 [52] for Chinese. Both are based on multi-choice questions.Coreference Resolution: Given a sentence with pronouns, determine which pronoun refers to which entity. We use the WinoGrande dataset [44] for English and the CLUEWSC dataset [62] for Chinese.Examination: Multiple-choice questions in examinations. For English, we use MMLU [21] , which includes mathematics, US history, computer science, law, and more. For Chinese, we use C-Eval [25] which ranges from humanities to science and engineering.Math Word Problem: Solving real-life, situational and relevant problems using mathematical concepts. For English we use the GSM8K [12] dataset. For Chinese, we translate the questions and answers in GSM8K to Chinese, namely GSM8K-Chinese.The prompting types cover few-shot [6] , zero-shot, and few-shot chain-of-thought (CoT) [59] . The datasets are summarized in Table 1 .In the first experiment, we train three models with 1.5B, 6B, and 32B parameters and observe their behaviors until trained on 3T, 3T, and 2.5T tokens, respectively. The training hyperparameters are shown in Figure 1 : The performance-vs-loss curves of 1.5B, 6B, and 32B models. Each data point is the loss (x-axis) and performance (y-axis) of the intermediate checkpoint of one of the three models. We mark the results of random guess in black dashed lines. Table 2 : Statistical measures of the correlation between task performance and pre-training loss in Figure 1 . The spearman correlation coefficient [50] measures the monotonicity of the relationship between the two variables, and the pearson correlation coefficient measures the linearity of the relationship. Both vary between -1 and +1 with 0 implying no correlation. Correlations of -1 or +1 imply an exact monotonic/linear relationship. 1 , and provide the statistical measures of the two variables in Table 2 . From the curves and statistics, we can see that the training loss is a good predictor of the performance on 12 downstream tasks.• Generally, the task performance improves as the training loss goes down, regardless of the model sizes. On MMLU, C-Eval, GSM8K, and GSM8K-Chinese, all models of three sizes perform at the random level until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss decreases. Detailed analysis on this is shown in Section 3. • Importantly, the performance-vs-loss data points of different model sizes fall on the same trending curve. That is, by ignoring the color differences (model sizes), the data points of different models are indistinguishable. For example, when the training loss falls around 2.00, the green and orange points on TriviaQA are indistinguishable. This indicates that the model performance on downstream tasks largely correlates with the pre-training loss, regardless of the model size. • Both spearman and pearson correlation coefficients show that performance is strongly related to pre-training loss on TriviaQA, HellaSwag, RACE, WinoGrande, etc. The pearson correlation coefficients on these tasks specifically show that points from different models lie on the same trending curve. The relationship is weaker on MMLU, CEval, GSM8K, and GSM8K-Chinese, verifying the emergence of performance which we discuss in Section 3.• Interestingly, we find that the overall training loss is a good predictor of performance on both English and Chinese tasks, although it is computed on a mixture of English and Chinese tokens. This implies that the learning dynamics of English and Chinese tokens are likely very similar during multilingual pre-training. The performance-vs-loss curves of smaller models pre-trained with different numbers of training tokens. Each data point is the loss (x-axis) and performance (y-axis) of the final checkpoint of one model, i.e., each point corresponds to one model trained from scratch. We mark the results of random guess in black dashed lines.Following the empirical experiments in scaling laws [22, 28, 23] , we further pre-train 28 relatively smaller models with different numbers of training tokens. The model sizes range from 300M, to 540M, 1B, 1.5B, 3B, and to 6B, while the numbers of pre-training tokens range from 33B to 500B. Varying the number of pre-training tokens is necessary since to achieve optimal performance we need to set the cosine learning rate schedule to reach the minimum at the corresponding token count [28, 23] . The number of tokens used and hyperparameters for all models are shown in Table 5 (Appendix).On each line, each data point represents the performance and pre-training loss of the corresponding model pre-trained completely from scratch with the certain token count (and learning rate schedule). We can see that similar to the observations from Figure 1 , the data points of different models sizes and training tokens largely fall on the same trending curves. In other words, the LMs with the same pre-training loss regardless of token count and model size exhibit the same performance on the 12 downstream tasks.Another similar observation is that the performance curves on MMLU, C-Eval, GSM8K, and GSM8K-Chinese do not yield an uptrend, meaning that the performance of these models on these four tasks are close to random (with fewer than 500B tokens). For simplicity, we only plot the performance of the latest checkpoint in each training in Figure 2 . The complete performance curves with intermediate checkpoints of each model, in which we can observe the same trend but larger variance, are shown in Figure 5 (Appendix). [55] . Note that the LLaMA2 paper [56] does not cover such figures with related information.To validate the generality of our observations, we analyze two different model series with required information made publicly available, i.e., LLaMA [55] and Pythia [3] . Compared to our models, LLaMA uses a pre-training corpus that excludes Chinese documents, leverages a different pre-training framework [37] , and adopts a slightly different model architecture. Since the intermediate checkpoints of LLaMA are not available, we extract the pre-training loss and corresponding performance on six question answering and commonsense reasoning tasks from the figures in its original paper, and plot the points in Figure 3 .Excitingly, most data points from the LLaMA models with different sizes (7B, 13B, 33B, 65B) fall on the same upwards trend. This observation further confirm our conclusion that the model's pre-training loss can predict its performance on downstream tasks, regardless of model size and token count. Note that there is only one exception at the early stage of LLaMA-65B. We can see that when the training loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss. Without access to its intermediate checkpoints, we unfortunately cannot further analyze the result. One possible explanation is that they use exponential smoothing on either the loss or downstream performance plots. Exponential smoothing would perturb the earlier points more than other points, potentially leading to this effect. Note that the outliers only constitute the initial 10% training tokens. The results for Pythia are shown in Appendix F, which also support our conclusion.Observed from previous experiments and analysis, we can conclude that the pre-training loss is a good indicator of LMs' performance on downstream tasks, independent of model sizes, training tokens, languages, and pre-training frameworks. 
performance trends of different tasks: 
 In Figures 1 and 2 , we can separate the datasets into two groups: First, on TriviaQA, HellaSwag, RACE, WinoGrande, NLPCC-KBQA, ClozeT, CLUEWSC, and C3, the performance improves smoothly with decreased pre-training loss from the very beginning. Second, on MMLU, C-Eval, GSM8K, and GSM8K-Chinese, the performance remains flat when the loss is higher than a certain threshold. Once the pre-training loss is lower than this threshold, the performance starts to improve. The correlation coefficients in Table 2 also reveal the difference: coefficients in the first group are close to -1, indicating strong correlations, while correlations in the second group are weaker. Take MMLU as an example of the second group, when the pre-training loss is higher than 2.2, the accuracy remains around 25%. Since each question in MMLU has four options, this means the model prediction is no better than random guessing. However, when the pre-training loss drops below 2.2, the accuracy increases as the loss decreases, similar to the trend observed in the first group of tasks. The performance trends of C-Eval, GSM8K, and GSM8K-Chinese follow a similar pattern. Despite differences in languages, tasks, prompting types, and answer forms among the four datasets are different, the thresholds for performance improvement are surprisingly all around 2.2. RACE in the first group has a prompting format similar to MMLU: both consist of multi-choice examination questions with in-context demonstrations, but their performance curves are quite different. We hypothesis that it is the task difficulty that makes the difference. Tasks of the first group of datasets are easier than those of the second group. For example, RACE requires the model to select correct answers for questions about a given article, and HellaSwag lets the model to select the possible followup of a situation based on commonsense. In contrast, MMLU and C-Eval consist of questions designed for high school, college, or professional examinations, requiring a broader range of knowledge. GSM8K and GSM8K-Chinese are math word problems that were previously considered as impossible to be solved by pre-trained language models before Chain-of-Thought prompting.The phenomenon can be related to grokking, which describes the improvement of performance from the random chance level to perfect generalization [40] . Power et al. [40] find that this improvement can occur well past the point of overfitting. In pre-training, the models are usually underfitting instead of overfitting overall. Since the pre-training corpus is a mixture of different documents, it is possible that the model already fits some patterns-such as numerical addition-in the data, while still underfitting the overall corpus.Certainly, the observations on the second groups of datasets can also be related to emergent abilities [58] , that is, abilities that only present in large models. According to the scaling law [28] , with the number of training tokens fixed, the pre-training loss follows a power law with respect to model sizes. In other words, there is a monotonic relationship between model size and pre-training loss. For the second group of tasks, there is a threshold of model sizes that corresponds to the tipping point in the pre-training loss. When the model size exceeds this threshold, the model can exhibit performance above the random chance level.Schaeffer et al. [46] propose an alternative explanation of emergent abilities of LMs, that is, emergent abilities appear due to the researchers' choice of nonlinear or discontinuous metrics. The accuracy on multi-choice questions (e.g., MMLU) is discontinuous, since the score on a question is either 1 or 0. To validate this claim, we examine the intermediate checkpoints on MMLU and C-Eval with continuous metrics rather than discontinuous accuracy used in the original benchmarks. The first metric is the predicted probability of the correct answer, denoted as CorrectChoiceProb. The second one is the Brier Score [5] used in Schaeffer et al. [46] :BrierScore = 1 N N i=1 C j=1 (y ij -ŷij ) 2 (2)where ŷij is the predicted probability of sample i for class j and y ij is the ground truth probability.The metric measures the prediction error and a lower value indicates better performance.We plot the results measured by different metrics on MMLU and C-Eval in Figure 4 . All three metricsaccuracy, correct choice probability, and Brier Score-show emergent performance improvements (value increase for the first two and decrease for the third) when the pre-training loss drops below a certain threshold. The Brier Score also decreases when the pre-training loss is above the threshold. However, the decrease of Brier Score does not always represent improvements on the task, since the Brier Score is related to not only the predicted probability of the correct answer but also the predicted probabilities of the incorrect answers. We find that the distribution of the correct answers is uniform in the four options in MMLU and C-Eval. The best Brier Score for a context-free predictor is achieved by always giving uniform probability to all the options. In this case, the Brier Score is equal to 0.75. Therefore, the performance in terms of Brier Score is no better than random guess before the loss reaches the threshold. This observation further confirms our previous conclusion. We discuss the contrary observations of Schaeffer et al. [46] and Xia et al. [61] in Appendix C.We conclude that emergent abilities of language models occur when the pre-training loss reaches a certain tipping point, and continuous metrics cannot eliminate the observed tipping point. 
defining emergent abilities from the loss perspective: 
 In previous sections, we show that 1) the pre-training loss is predictive of the performance of language modes on downstream tasks, and 2) some tasks exhibit emergent performance improvements from the random guess level when the pre-training loss drops below a certain threshold regardless of model size, token count, and continuity of metrics. Based on these observations, we give a new definition of emergent abilities from the pre-training loss perspective:Definition. An ability is emergent if it is not present in models with higher pre-training loss but is present in models with lower pre-training loss.The normalized performance on an emergent ability as a function of the pre-training loss L is:EQUATIONwhere f (L) is a monotonically decreasing function of L, η is the threshold, and the normalized performance of random guess is 0.Next we will show how the new definition can be related to previously observed emergent abilities [58] .In Henighan et al. [22] , they give the scaling relation for the loss with model size N when the number of training tokens D is fixed:L(N ) = L ∞ + N 0 N α N (4)where L ∞ is the irreducible loss, and α N is the coefficient. The equation shows that the loss of language models follows a power-law plus a constant. Combining Equation (3) and Equation ( 4), we can get the normalized performance as a function of the model sizeEQUATIONFrom this equation, we can explain the emergent abilities observed in Wei et al. [58] : when model sizes are smaller than N 0 • (η -L ∞ ) -1/α N , the normalized performance is zero. When model sizes exceed N 0 • (η -L ∞ ) -1/α N , the increase in model size leads to a decrease of pre-training loss and an improvement in normalized performance. 
related work: 
 Relationship of Pre-training Loss and Task Performance. In the transfer learning setting, i.e. the language model is pre-trained on the general corpus and fine-tuned on supervised data of specific tasks, Tay et al. [54] find that models with the same pre-training loss can have different downstream performance after finetuning, due to inductive bias in model architectures such as Transformers and Switch Transformers. Tay et al. [53] further study the effect of model shapes on downstream fine-tuning. Liu et al. [33] also study the effect of inductive bias of model sizes and model algorithms on the relationship of pre-training loss and downstream performance after fine-tuning, but their theory only applies in the saturation regime, where the models are close to minimal possible pre-training loss. Instead, large language models today are generally under-trained [23, 55] , far from the saturation regime. Overall, these studies focus on the pretrain-finetune paradigm, in which inducitive bias helps improve transferability, while we study prompted performance of large language models without finetuning [29, 6] . For the prompted performance of large language models, Xia et al. [61] claim that perplexity is a strong predictor of in-context learning performance, but the evidence is limited to the OPT model [66] and a subset of BIG-Bench [51] . Instead, Shin et al. [49] find that low perplexity does not always imply high in-context learning performance when the pre-training corpus changes. Gadre et al. [18] fits the relation of perplexity and the top-1 error averaged over many natural language tasks with a power law. Instead, we focus on the different relations of tasks and a small part of tasks that show emergency trends.Emergent abilities. Wei et al. [58] propose the idea of emergent abilities, abilities that only present in large language models. This is similar to the claim of Ganguli et al. [19] that it is more difficult to predict the capacities of language models than to predict the pre-training loss. The existence of emergent abilities has been challenged. Hoffmann et al. [23] show that smaller language models trained with sufficient data can outperform undertrained larger language models, supported by followup models [55, 26, 56] . On the other hand, Schaeffer et al. [46] claim that emergent abilities are due to the discontinuous metrics used for evaluation, also found in Xia et al. [61] . Similarly, Hu et al. [24] propose to predict the performance of emergent abilities with the infinite resolution evaluation metric. In this paper we prove the existence of emergent abilities from the perspective of pre-training loss, even with continuous metrics.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The training code is Megatron-LM2 . The pre-training dataset cannot be released since it contains proprietary content. 
conclusion: 
 Our paper proposes a new definition of emergent abilities of language models from the perspective of pre-training loss. Empirical results show that the pre-training loss is a better metric to represent the scaling effect of language models than model size or training compute. The performance of emergent abilities exhibits emergent increase when the pre-training loss falls below a certain threshold, even when evaluated with continuous metrics.The new definition offers a precise characterization of the critical junctures within training trajectories where emergent abilities manifest. It encourages future studies to investigate the shifts in language models at these junctures, which facilitate the development of new capabilities.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The details of pre-training are described in Appendix A and those of evaluation are described in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
limitation: 
 We study the relationship of pre-training loss and task performance across model sizes, training tokens, tasks, languages, prompting types, and answer forms. Factors we have not considered are model architectures and training algorithms. We analyze the performance-loss curves of LLaMA and Pythia with slightly different architectures, and find that the relationship holds for all the models. But there are fundamentally different model architectures, such as routed Transformers [16] , and non-Transformer architectures [17, 39] beyond our consideration. Both our models and LLaMA use AdamW optimizer [35] , while there are other optimizers for language model pre-training [48, 32] .The disadvantage of studying emergent abilities in the lens of pre-training loss is that the pre-training loss is affected by the tokenizer and the distribution of pre-training corpus. The values of pre-training loss of language models trained on different corpus are not directly comparable. One possible solution is to evaluate different language models on a public validation set with the normalized perplexity [43] to account for the different vocabulary sizes.The paper should not be considered as a push to expand model sizes and data sizes of language models beyond current scales. It is not guaranteed that new tipping points emerge in larger scales. Also, instruction tuning [57, 45, 9, 34] can improve the zero-shot performance of language models on unseen tasks, including MMLU and GSM8K. 3 : The ratio of different sources in the English corpus.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: It is too computationally expensive to train multiple pre-trained language models from scratch.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks since it does not include release of data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the creators of code, datasets, and models used in the paper are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our pre-training corpus is a mixture of English and Chinese documents. The ratio of English tokens to Chinese tokens in the pre-training corpus is 4:1. Both the English and Chinese corpora consist of webpages, wikipedia, books, and papers. The distribution of different sources in the English corpus is shown in Table 3 . The distribution and processing pipeline are similar to Redpajama [13] . During pre-training, documents from Wikipedia and Books are trained for multiple epochs, but most documents (93.4% in the pre-training corpus) are never repeated.We tokenize the data with the byte pair encoding (BPE) algorithm [47] in the SentencePiece package [30] . The vocabulary size is 65k.The hyperparameters for training of 1.5B, 6B, and 32B models are shown in Table 4 . The hyperparameters for training of smaller models are shown in Table 5 . The sequence length is 2048 and the optimizer is AdamW [35] with β 1 = 0.9 and β 2 = 0.95.The evaluated splits and numbers of examples are summarized in Table 6 . For English datasets, we follow Gopher [41] and Chinchilla [23] 's selection of evaluation splits. For Chinese datasets, we use the validation split when the ground labels are always available. For CLUEWSC, the size of the validation set is too small (100), so we combine the train and validation splits. GSM8K-Chinese is translated from GSM8K with machine translation and human proofreading.C Are Emergent Abilities of Language Models a Mirage? [46] claim that emergent abilities proposed in [58] are mainly a mirage caused by nonlinear and discontinuos metrics. [61] also support the idea.[61] use the perplexity of correct options as the metric for BIG-Bench and find that the metric impproves smoothly on almost all the tasks of BIG-Bench. We argue that the perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions. The correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options. The perplexity of correct options and incorrect options may decrease simultaneously. In fact, [61] the end of training that the perplexity of correct and incorrect options starts to diverge. This supports the existence of emergent abilities.[46] use Brier Score [5] as the metric for BIG-Bench. We argue that increase in Brier Score does not always represent improvement of performance on the multi-choice task, since Brier Score is also related to the allocation of probabilities for incorrect options. For example, questions in the MMLU dataset have four options (A, B, C, and D) and the frequency of the four options as correct is equal.Consider two models that give the same probability independent of questions. One model predicts (1, 0, 0, 0) for the four options and the other model predicts (0.25, 0.25, 0.25, 0.25). The Brier Score for the former is 1.5 while the Brier Score for the latter is 0.75. However, both models do not learn the relationship between questions and correct options at all. One can argue that the latter model better fits the distribution of correct options in the dataset, but the improvement is not as large as the different of 1.5 and 0.75. We should consider the Brier Score of 0.75 as the performance of the random guess baseline, and any decrease in Brier Score above 0.75 should not be considered as the real improvement on the task.In Figure 6 of [46] , they evaluate 4 tasks in BIG-Bench with the Brier Score metric and find that the emergent abilities disappear. We hypothesis that they normalize the Brier Score with the number of options in each question, otherwise the Brier Score of 0. The performance-vs-loss curves for all the intermediate checkpoints are shown in Figure 5 . The trend is the same as Figure 2 , but with larger variance.We show the performance-compute curves in Figure 6 . Compared with Figure 1 , we observe that points from different models do not fall on the same curves on most tasks. This proves that pre-training loss is a better indicator of task performance than compute. To further support our conclusion, we plot the performance-loss curves of Pythia [3] in Figure 7 .Pythia is a suite of open language models with intermediate checkpoints released. For downstream tasks, we select SciQ [60] , LAMBADA [38] , WinoGrande [44] , ARC-Easy [11] , ARC-Challenge [11] ,and PIQA [4] with reported performance in the official repository. We compute the cross-entropy loss of intermediate checkpoints on the corpus Pile [20] . From the plot, we can observe that the points from different models fall on the same curve on all the tasks. This supports our conclusion that pre-training loss is predictive of task performance.However, neither Pythia nor LLaMA can be used to analyze the emergent abilities. The largest Pythia model fails to achieve performance above random chance on MMLU and GSM8K [2] . Instead, LLaMA has no intermediate checkpoints released and performance curves on MMLU and GSM8K are not available.BIG-bench [51] is a series of diverse tasks designed to evaluate the capacities and limitations of pre-trained language models. Wei et al. [58] find that large language models exhibit emergent abilities on four tasks from BIG-Bench. Among the four tasks, the test set size of the figure-of-speech detection task is too small and the variance is too high. We evaluate the other three tasks in the same setting as Section 2.3 and the results are shown in Figure 8 . With pretraining loss decreases along the x-axis, we can clearly observe the tipping point in the performance curves. Figure 7 : The performance-vs-loss curves of Pythia. The performance of Pythia is from the official repository and the loss is evaluated with the released checkpoints. Figure 8 : The performance-vs-loss curves of 1.5B, 6B, and 32B models on 3 tasks in BIG-bench. Each data point is the loss (x-axis) and performance (y-axis) of the intermediate checkpoint of one of the three models. We mark the results of random guess in black dashed lines.All the models are trained on DGX-A100 GPU (8x80G) servers. The 1.5B, 6B, and 32B models in Section 2.3 take 8 days on 256 A100 GPUs, 8 days on 1024 A100 GPUs, and 20 days on 2048 A100 GPUs respectively. The small models in Section 2.4 take about 20 days on 256 A100 GPUs.This paper finds that pre-training loss is predictive of downstream task performance and on some tasks the performance only begins to improve when the pre-training loss falls below a certain threshold. Combined with previous works on scaling laws [28, 22, 23] , we can predict the amount of compute required to achieve a certain performance. This can be used to estimate the cost of training a large model.The paper might encourage companies to expand model sizes and data sizes of language models beyond current scales to pursue new emergent abilities, leading to a waste of compute resources. We want to emphasize that the analysis of previous performance trends do not necessarily apply to the larger models.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://github.com/NVIDIA/Megatron-LMhttps://github.com/NVIDIA/Megatron-LM 


Decision: This work demonstrates that 1) pre-training loss is generally predictive of downstream capabilities in language models, rather than model size or number of tokens used during pre-training; and 2) emergent capabilities can also be clearly described in terms of pre-training loss. They also demonstrate that using continuous metrics, they still observe emergence, countering findings from prior work. They conduct their analysis on a suite of standard English and Chinese evals, using a range of models that they pre-trained themselves. They also further validate their findings using the Llama and Pythia model families.

I am not going to factor in the review of Reviewer RA4k, as it is too short and the reviewer did not engage further either in discussions with the authors or discussions with the other reviewers.

This work has the potential to be a timely contribution on understanding emergent properties in LLMs, if the results support the conclusions drawn by the authors. All reviewers other than ZLu4 were initially skeptical at least to some degree about the evidence that the authors provide for one of the two major claims in this work -- that a specific pre-training loss threshold is needed to lead to emergent abilities in LMs. The authors had lengthy exchanges with the reviewers and most reviewers were satisfied by the author response. There are still some outstanding concerns from Reviewer BK9j, which I urge the authors to take seriously, specifically regarding whether all modes indeed lie on the same general curve, especially because there is little overlap between the loss ranges for different model size. Despite this remaining concern, there is strong support for this work from the remaining reviewers, and I believe this work will at least serve to deepen the discussion on the emergent properties of LLMs. I suggest that the authors incorporate the reviewers' suggestions in a revised manuscript, particularly emphasizing the role of pretraining data and clarifying the comparison with and discussion of previous work.

Reviews: review: paper_summary: paper_summary: This paper investigates emergent abilities in language models from the perspective of pre-training loss, rather than model size or training compute. The authors challenge recent skepticism about emergent abilities by demonstrating that: (1) Models with the same pre-training loss, regardless of model and data sizes, exhibit similar performance on various downstream tasks. (2) Emergent abilities manifest when a model's pre-training loss falls below a specific threshold, before which performance remains at random guessing levels. 
The study examines different metrics (including continuous ones) and proposes a new definition of emergent abilities based on pre-training loss. The authors argue that this perspective better represents the learning status of language models and provides a more precise characterization of when emergent abilities appear during training.
strength_weakness: strength_weakness: - The authors acknowledge that they have not considered fundamentally different model architectures (e.g., routed Transformers) or non-Transformer architectures. This limitation may affect the generalizability of their findings.
- As noted in the limitations section, pre-training loss is affected by tokenizers and pre-training corpus distribution, making direct comparisons between models trained on different corpora challenging. While the authors suggest using normalized perplexity on a public validation set, this solution is not implemented in the current study.
-  While the paper establishes a correlation between pre-training loss and emergent abilities, it does not provide a causal explanation for why certain abilities emerge at specific loss thresholds. A deeper investigation into the underlying mechanisms could strengthen the paper's contributions.
- While the authors discuss some contrary observations from previous studies, a more comprehensive comparison with existing literature on emergent abilities and their proposed explanations would enhance the paper's positioning within the field.
questions: questions: None.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper investigates the link between the pre-training loss of LLMs and their downstream performance on popular benchmarks. The authors train a series of models ranging from 300M to 32B parameters on English-Chinese datasets of different sizes and study how these models perform on TriviaQA, HellaSwag, RACE, WinoGrande, MMLU and GSM8K benchmarks and their Chinese counterparts, as a function of their pre-training loss. They observe a strong link between pre-training loss and benchmark performance that is not affected by model size and dataset size, from which they conclude that pre-training loss is the main indicator of downstream performance. On MMLU and GSM8K (and their Chinese counterparts), models exhibit an emergent behavior wrt. loss, i.e. their performance is close to random before a loss value of about 2.2 and steadily increases for lower loss values. This relationship seems to hold, even under continuous evaluation metrics.
strength_weakness: strength_weakness: 1. The paper seems to make two main claims: 1) that pre-training loss is the main predictor of downstream performance, and 2) that emergent abilities appear suddenly after models cross a certain loss-threshold. I find the evidence for 1) to be convincing, but I am somewhat less confident about 2). This is mostly because the paper only shows emergent abilities on a subset of the benchmarks they were originally proposed on (MMLU and GSM8K) in [1]. Notably, BigBench is absent. Including results on the tasks in BigBench that were shown to exhibit emergent behavior would be helpful to judge whether pre-training loss predicts the emergence threshold as well as pre-training compute or model size. Including these benchmarks would also help to put the paper into a better perspective wrt. subsequent work questioning the existence of emergent abilities [2], which also studies these benchmarks.
2. There are some discrepancies and unexplored links between the results reported in the paper and findings in prior work. To develop a better understanding of how the findings here relate to prior work, it would be helpful to include a discussion in the paper. See Questions 2. and 3. below for more details.
3. I find the paragraph in 3.1, line 197 rather speculative. Grokking refers to an improvement on the test-set, despite stagnation on the training set, whereas the models studied here seem to still be improving on the training set. I am not sure whether there is a link between grokking and the emergent behavior on the specific tasks here.
4. Minor: There are a number of typos and small writing issues, e.g.
    - "has thus far less studied" (line 23), ", a highly over-parameterized models" (line 79), "coefficients how that" (line 126),...
    - line 120: "climbs as the loss increases" should probably be "decreases".
    - line 141: "On each line" seems to be missing a reference to Figure 2.

References
- [ 1] Emergent Abilities of Large Language Models, Wei et al., https://arxiv.org/abs/2206.07682
- [ 2] Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., https://arxiv.org/abs/2304.15004
questions: questions: 1. The paper claims that pre-training (test) loss is a good indicator of downstream performance. What is the composition and size of the test set that the loss is evaluated on? What properties (size, diversity, etc.) should such a test-set have in order to be predictive of downstream performance?
2. Prior work [1, 2] has found that emergent behaviors can become predictable with "higher resolution" benchmarks, i.e. with larger test sets or repeated sampling from the model. This is a dimension that the paper does not touch upon. Do the authors believe, that the studied emergent abilities would still appear only after models cross the particular loss threshold, i.e. be essentially 0 before, even with those higher resolution benchmarks?
3. In related work (line 254), the authors mention that some prior work [3] has observed a disconnect between pre-training loss and downstream performance, which stands in contrast to the claims made in the paper. It would be great if the authors could comment on more on the reasons for these discrepancies.

References
- [ 1] Are Emergent Abilities of Large Language Models a Mirage?, Schaeffer et al., https://arxiv.org/abs/2304.15004
- [ 2] Predicting Emergent Abilities with Infinite Resolution Evaluation, Hu et al., https://arxiv.org/abs/2310.03262
- [ 3] Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models, Liu et al., https://proceedings.mlr.press/v202/liu23ao.html
limitations: limitations: The paper sufficiently discusses limitations, in my opinion.
paper_summary: paper_summary: They demonstrate that 1) pre-training loss is generally predictive of downstream capabilities in language models, rather than models size or number of tokens used during pre-training; and 2) emergent capabilities can also be clearly described in terms of pre-training loss. They also demonstrate that using continuous metrics, they still observe emergence, countering findings from prior work. They conduct their analysis on a suite of standard English and Chinese evals, using a range of models that they pre-trained themselves. They also further validate their findings using the llama and Pythia series of models.
strength_weakness: strength_weakness: * From the intro: "For example, LLaMA-13B with less compute [53] can outperform GPT-3 (175B) on MMLU [21]": Training for longer could be one explanation for this, another one could be llama used a higher quality corpus than gpt-3 did. I think the data quality element is a little underrated. In general we should expect different pretraining datasets to potentially have different loss / different emergence points (even if the x-axis loss is on a held out validation set and models have the same vocabulary). There is some discussion of this in the limitations section, but in general I think this point should be highlighted more in the paper. Concretely, the pretraining loss -> emergence point phenomenon is only guarenteed to be consistent for a series of models pretrained on the same data.
* The discussion of exact match in Section 2 kind of comes out no nowhere, and I'm not sure why it is discussed where it is. It would be great if there were more motivation for this in the paper.
* I understand why you did the ablation that you did in Section 2.4 (ablate the effect of lr schedule, as in Chinchilla), but in the paper this is not very clearly motivated and readers with less background knowledge might not understand why this ablation is important.
* “Note that there is one only exception at the early stage of LLaMA-65B. We can see that when the training loss is higher than 1.8, LLaMA-65B performs worse than smaller models with the same training loss.”: This could be because they used some exponential smoothing on either their loss or downstream performance plots. Exponential smoothing would perturb the earlier points more than other points (especially for curves that are fast changing or have a rapid change at the beginning), potentially leading to this effect. Moreover, did you smooth out the loss in some way when plotting loss verses downstream in the previous sections? If so, this would at least be good to note in the paper.
* Section 3.1 could use clearer discussion of explanations for why emergence occurs on the different tasks. I think grokking is a possible explanation, but what exactly grokking means (outside of the context of the simple algorithmic tasks presented in the original grokking paper) is a little vague/imprecise, and I also don't think this is the only explanation. In the case of gsm8k, emergence is, in my opinion, more likely due to the fact that the model has to get a sequence of reasoning steps correct to answer the question, this leads to an exponential (e.g. p(step_correct)^n_steps) for getting the full answer correct (some analysis of this in the paper could be cool). In the case of MMLU and C-Eval there actually is an existing paper which discovers an explanation, using interpretability techniques, for emergence on multiple choice tasks (https://arxiv.org/pdf/2307.09458).
* There's a handful of related works that aren't discussed in the paper and whose findings contradict those of this paper (I'll give you my reasoning for why your work might not contradict theirs, as a suggestion in parentheses, but you should include some discussion of this in the paper somewhere, either in the related work or elsewhere):
1) https://proceedings.mlr.press/v202/liu23ao.html (my understanding is that their theory only applies near the global optim, which is generally not the case with real-world language models)
2) https://arxiv.org/pdf/2109.10686 (see the paragraph "zooming in verses zooming out", your paper is more about the zoomed out setting)
3) Not strictly language modeling, but see Figure 3 of https://cdn.openai.com/papers/Generative_Pretraining_from_Pixels_V2.pdf (could be because they are doing probing so hidden-state size increases probe capacity causing differences with model size, but it's unclear if this is actually the case)
questions: questions: See the weaknesses for most of my questions/concerns.
limitations: limitations: No major limitations. See the weaknesses section.
paper_summary: paper_summary: This paper formulates the concept of emergence as a relationship between language modeling training loss and task performance, rather than in relation to model/data scale.
strength_weakness: strength_weakness: Even when different models have very similar pretraining loss, they may have extreme variation in other qualities, including specific benchmarks. This paper does not engage substantially with the existing literature on the connection or lack thereof between loss and specialized metrics; it nods to them in the literature review on "Relationship of Pre-training Loss and Task Performance" but does not clarify why their findings are so different from existing results.

>We argue that the perplexity of correct options is not the correct metric to evaluate the performance of multi-choice questions. The correct metric of multi-choice questions should reflect the ability of distinguishing correct options from incorrect options. 

I looked into this appendix because I was intrigued as to why their results are so strongly at odds with Schaeffer et al. I completely disagree with this claim (which the paper provides no support for); if perplexity is the wrong metric, why measure loss at all? To the contrary, recent work including https://arxiv.org/pdf/2404.02418 and https://arxiv.org/pdf/2305.13264 continue to support the notion that claims of emergence should consider probabilities rather than exact matching, which obscures auxiliary task dependencies such as the capability of answering in multiple choice format. In general, our community now understands how much emergence can be attributed to metric thresholding effects, so you need much stronger evidence to argue against Schaeffer et al. 

The core claim that one can predict performance on any given task just from the loss of a model would be a more useful finding if it applied across different architectures, not just different data and model scales. Although the paper considers both Llama and Pythia, they do not plot these different architectures on the same figures and it is therefore not clear whether loss is similarly predictive across architectures or whether it is only within an architecture, in which case there is no real argument for preferring it over scale as a basis of measurement. There is no excuse for continuing to present metric thresholding effects as evidence of meaningful emergence.

> That is, by ignoring the color differences (model sizes), the data points of different models are indistinguishable. 

This simply doesn’t seem to be true. RACE in particular does seem to have a different slope for green compared to the other colors. For CLUESC orange and blue seem to be nearly separable, although that could be random. Still seems to be true for a bunch of these and it’s an interesting point to make, but I think you need to qualify it by actually comparing the slopes from a linear regression for each color.

> an ability is emergent if it is not present in language models with higher pre-training loss, but is present in language models with lower pre-training loss.

Not clear what it means to claim any ability is not present and then is present. It seems that they mean that below a certain point you have random chance answers. So by definition, if you perform at random chance at any point, then they would say the ability is emergent regardless of whether there is a clear breakthrough. This is the definition of a thresholding effect.

Minor:
- Please use natbib with \citet when you have an inline citation.
- "emergency trends" should be emergent
- > until the pre-training loss decreases to about 2.2, after which the performance gradually climbs as the loss increases.
    - Do you mean the loss decreases? Or am I missing something? Why would the loss increase after this point?
questions: questions: Could you discuss a little further why your findings are so at odds with existing work like Schaeffer et al.? As far are as I can tell, it is because you've decided to define emergence without accounting for straightforward metric thresholding effects.
limitations: limitations: No obvious unlisted limitations.
paper_summary: paper_summary: This paper measures a range of language models' pretraining loss and their downstream performance, arguing that emerging capabilities are better measured by losses as opposed to previously proposed model size or compute (FLOPs). The paper also offers evidence against the argument that emergent abilities are a "mirage" when the task metrics are continuous.
strength_weakness: strength_weakness: 1. The authors define emergence as a discontinuity, although it seems possible to fit an exponential function (especially when the x-axis is FLOPs as in Figure 6), or a linear function for many tasks such as the first group in section 3.1. Although Figure 4 offers stronger evidence for discontinuity, it is not definitive, as only MMLU, GSM8K and C-Eval are measured (but I accept that these are reasonably sufficient for an academic paper). 
1. The authors discuss why the harder nature of MMLU and GSM8k may be causing the discontinuity, but it would be useful to present a qualitative analysis on the "easier" questions that models "first" get right with just a bit more reduction in pretraining loss.
questions: questions: 1. Is EM the same as perplexity/scoring eval?
1. Why use the training loss as opposed to validation/test loss?
limitations: limitations: L130
> we find that the overall training loss is a good predictor of performance on both English and Chinese tasks,
Naturally, you could compute the per-corpus loss on the x-axis to strength the claim. For the final version, it'd be great to have a per-corpus loss analysis on what kinds of corpora are most predictive of downstream performance.

Not limitations just typos:
Line 128: verifying the emergency of performance -> emergence

Line 218: ground probability -> did you mean ground truth probability?

Comments: title: Differences from Schaeffer
Clarification
comment: I thank the authors for their clarification. I still believe that some of the transitions may potentially be predictable from higher loss models.

I will maintain my score, but increase my confidence level, and would support acceptance of the paper.
Thank you for the valuable feedback. We understand that on some tasks it is difficult to differentiate emergent abilities from exponential performance curves. We want to emphasize that the point of emergent abilities is to identify some tasks on which it is difficult to predict the performance of larger models with the performance of smaller models. On word unscramble and IPA transliterate, before the tipping point, the curve is speciously linear with a very small slope, rather than exponential. Therefore it is very difficult to predict the sudden increase in the performance as the pretraining loss further decreases. This also distinguishes these tasks from those whose performance increases smoothly with an almost constant slope.
Thanks, I appreciate the response. I think this is a good work and will keep my score.
I thank the authors for answering my questions in the rebuttal, and for providing additional results on BIG-Bench. They help in assessing how the findings of this work relate to observations made in prior work and make me more confident about the results.

I continue to believe that the pre-training loss perspective is a valuable contribution for predicting emergent downstream abilities. However, I am still not quite sure that the downstream abilities emerge "suddenly". The results on BIG-Bench, particularly word unscramble and IPA transliterate, might actually be following a continuous trend, e.g. a sigmoid, and for modular arithmetic there seem to be too few data points below the "emergence threshold" to accurately judge whether there is a sudden transition. Some sort of higher-resolution sampling might also show more continuous trends here.
Therefore, I maintain my score.

An interesting suggestion for future work would be to compare checkpoints from different model families on the same test-set (potentially with tokenizer-based normalization) to see whether the same loss values result in the same downstream performance for across families. To do this it would be possible to use open-checkpoint models such as Pythia, LLM360 [1] and OLMo [2].

- [1] LLM360: Towards Fully Transparent Open-Source LLMs, Liu et al., https://arxiv.org/abs/2312.06550
- [2] OLMo: Accelerating the Science of Language Models, Groeneveld et al., https://arxiv.org/abs/2402.00838
> I don't believe that this is true if you look at the Brier score plots provided and note that the slopes are substantially different depending on which model you are considering. 

The slope of the 1.5B model on Brier Score is different from those of 6B and 32B models is because all the checkpoints of the 1.5B model performs no better than the random chance on MMLU and C-Eval, which means these points are below the tipping point of our proposed emergent abilities. For points at the intersection of 6B and 32B models, which are above the tipping point, the slopes are very close. Our point is that on some tasks with emergent abilities (like MMLU and C-Eval), the slopes of loss-performance curves at high-loss regions are different from those at low-loss regions. Therefore it is very difficult to predict the performance of larger models on these tasks with the performance of smaller models.
My understanding reading the paper was that you find different results from Schaeffer et al. by rejecting the argument for continuous metrics. You stated

>  The main difference is that Schaeffer et al only evaluate the final checkpoints of different models while we also evaluate the intermediate checkpoints. With more data points available, we show that emergency on some tasks exists even with continuous metrics like Brier Score.

I don't believe that this is true if you look at the Brier score plots provided and note that the slopes are substantially different depending on which model you are considering. Therefore, you haven't provided evidence that intermediate training time can be treated similarly to other scale effects in terms of the relationship between aggregate loss and specific task capabilities (which makes sense, given that early stage training dynamics are different from later stage). 

The remaining difference from recent findings against emergence is therefore the rejection of continuous metrics, as I stated:

> In general, our community now understands how much emergence can be attributed to metric thresholding effects, so you need much stronger evidence to argue against Schaeffer et al.

While you make a fair point that you don't have access to each training corpus to allow evaluation of train loss and the models are trained on substantially different data, you then do not have the evidence to support an argument that aggregate (train) loss is directly predictive of specific task capabilities.
For the improvement on Brier Score before the emergency threshold, we already analyze this in Line 223 on Page 8. We find that the distribution of the correct answers is uniform in the four options in MMLU and C-Eval. A naive predictor that always gives uniform probability to all the options can achieve a Brier Score of 0.75, while another naive predictor that always gives all the probability to the first option has a Brier Score of 1.5. However, both models do not learn the relationship between questions and correct options at all. Therefore, decrease of Brier Score above 0.75 does not represent improvement on performance. In the plot we mark the Brier Score of random guess (0.75) in black dashed lines and obviously above the threshold the model's Brier Score is worse than random guess. We can't say a model's ability on a task is improving while its performance is worse than random guess.

We can't plot the curves of Llama and Pythia on the same plot because they are trained on different corpora. Pythia is trained on Pile, of which webpages constitutes 18%. Llama is trained on their own corpora, of which webpages 67%. In other words, they do not have a common x-axis. Also, the intermediate checkpoints of Llama are not publicly accessible. Training models with different architectures is beyond available computational resources. However, we already show that models with the same architecture and different sizes show similar trends. We believe it will be helpful for the community since most models adopt Llama architecture with moderate modifications.
Thank you for clarifying, I confused your arguments against Xia et al. (that perplexity is the wrong metric) and against Schaeffer et al. (that Brier score is the wrong metric). I think this would have been easier to keep straight if you used the natbib \citet in your discussion, but I acknowledge that I confused those two different arguments, although you did make both of them.

To be clear then, you claim that the same emergence point that Schaeffer et al. dismiss is actually a real point of emergence because the blue dots (1.5B) are flat whereas the orange dots (6B) have a positive slope. However, looking at each individual color makes it clear in the Brier score plot: for a 6B model, before the supposed phase transition, there is a lot more noise in the distribution of scores but they do not have a clearly insignificantly flatter trendline. In particular, it is entirely obvious that the 1.5B models over perform on the task relative to their loss when we compare them to 6B models with similar loss.

The reason for the emergence artifact in the thresholding functions is clearly a thresholding effect. This fact persists even if you look at loss on the X axis. I appreciate the creativity of plotting different models together with all their checkpoints, but the defense of emergence in this paper does not seem to hold, and that appears to be the framing given. The argument that we can use earlier checkpoints to plot scaling laws for many models simultaneously does not hold unless those laws all show similar slopes for different models, and they obviously don't or this paper would have shown different architectures together on the same plot instead of just different scales of the same architecture, which would make the different slopes more obvious. 

I would raise my score if you could show that Llama and Pythia have similar slopes when plotted together and that they therefore support the idea that the 6B and 1.5B actually have the same slope, which they don't seem to as the 6B simply increases in variance of performance below the threshold.