Predicting Emergent Abilities with Infinite Resolution Evaluation
Authors: Shengding Hu
Xin Liu
Xu Han
Xinrong Zhang
Chaoqun He
Weilin Zhao
Yankai Lin
Ning Ding
Zebin Ou
Guoyang Zeng
Zhiyuan Liu
Maosong Sun
Venue: ICLR 2024 poster (2024)
Keywords: Predictable Scaling; Scaling Law; Large Language Models; Pre-train;
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We perform predictable scaling for task performance and analyze emergent abilities quantitatively.

Abstract: The scientific scale-up of large language models (LLMs) necessitates a comprehensive understanding of their scaling properties. However, the existing literature on the scaling properties only yields an incomplete answer: optimization loss decreases predictably as the model size increases, in line with established scaling law; yet no scaling law for task has been established and the task performances are far from predictable during scaling. Task performances typically show minor gains on small models until they improve dramatically once models exceed a size threshold, exemplifying the ''emergent abilities''. In this study, we discover that small models, although they exhibit minor performance, demonstrate critical and consistent task performance improvements that are not captured by conventional evaluation strategies due to insufficient measurement resolution. To measure such improvements, we introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. With PassUntil, we conduct a quantitative investigation into the scaling law of task performance. The investigation contains two parts. Firstly, a strict task scaling law that is not conventionally known to exist, is identified, enhancing the predictability of task performances. Remarkably, we are able to predict the performance of the 2.4B model on code generation with merely 0.05\% deviation before training starts, which is the first systematic attempt to verify predictable scaling proposed by GPT-4's report. Secondly, underpinned by PassUntil, we are able to study emergent abilities quantitatively. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function and has a increasing speed. We then examine two hypothesis and imply that the ``multiple circuits hypothesis'' might be responsible for the accelerated emergence.

Published as a conference paper at ICLR 2024
PREDICTING EMERGENT ABILITIES WITH INFINITE RESOLUTION EVALUATION
Shengding Hu', Xin Liu", Xu Han 1,3 Xinrong Zhang', Chaaquu He', Weilin Zhao', Yankai Lin", Ning Ding'. Zebin Ou Guoyang Zeng", Zhiyuan Liu" Maosong Sun 1 DDparrmeett of Computer Science and Technology, Tsinghua University BBeijing Language and Culture University.
Shanghai Artificial Intelligence Laboratory "Renmin University China. *Zhihu Inc. hs hsd23@mails wew.lss.sssnnggn tsinghua. edu. on
Modelbest Inc.
ABSTRACT
The scientific scale-up of large language models (LLMs) necessitates a compre- hensive understanding of their scaling properties. However, the existing literature on the scaling properties only yields an incomplete answer: optimization loss de- creases predictably as the model size increases, in line with established scaling law; yet no scaling law for task has been established and the task performances are far from predictable during scaling. Task performances typically show minor gains on small models until they improve dramatically once models exceed size threshold, exemplifying the "emergent abilities". In this study, we discover that small models, although they exhibit minor performance, demonstrate critical and consistent task performance improvements that are not captured by conventional evaluation strategies due to insufficient measurement resolution. To measure such improvements, we introduce PASSUNTIL, an evaluation strategy with theoreti- cally infinite resolution, through massive sampling in the decoding phase. With PASSUNTII, we conduct a quantitative investigation into the scaling law of task performance. The investigation contains two parts. Firstly, a strict lask scaling law that is not conventionally known to cxist, is identified, enhancing the predictability of task performances. Remarkably, we are able predict the performance of the 2.4B model on code generation with merely 0.05% deviation before training starts, which is the first systematic attempt to verify predictable scaling proposed by GPT-4's report (OpenAI, 2023). Secondly, underpinned by PASSUN we are able to study emergent abilities quantitatively. We identify kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function and has a increasing speed. We then examine two hypothesis and imply that the "multiple circuits hypothesis" might be responsible for the accelerated emergence.
"See the world in grain of sand"
INTRODUCTION
Large Language Models (LLMs) (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Chowd- hery et al., 2022) have become a center of interest among AI researchers recently. These models, trained expansive datasets and furnished with enormous number of parameters, have demon- strated unparalleled proficiency across diverse domains, such as text generation (Dubois et al., 2023), code completion (Chen et al., 2021; Rozière et al., 2023), and academic test (Hendrycks et al., 2020).
The impressive success of these L.MSs depends heavily on scaling up the model parameters and pre-training data volume. It has been consistently observed that, when considering continuum of models with nearly identical architectures, larger models coupled with increased pre-training corpora consistently yield diminished training loss. This observation has been mathematically for- malized as the scaling law of loss (Kaplan et al., 2020; Henighan et al., 2020), which states that the reducible loss achieved by the model in the log scale is linear to the model size in the log scale. Scal- ing law has provided guidance for the scientific scaling of LLMs, including determining the balance
Corresponding Authors.

---

Published as a conference paper at ICLR 2024
of the model size and pre-training data size (Hoffmann et al., 2022; Muemnighoff et al., 2023). This has transformed what was once a somewhat blind scaling process into a methodology underpinned by empirical assurance. Nonetheless, such beneficial scaling law yield predictions solely on the loss, not extending to the real task performance encountered in practice. This divergence establishes substantial gap in comprehensive scaling-up methodology (Ganguli et al., 2022).
emoj_moviie movie
emoi_
ernoj movie
Exact arrnn match evelusted by PassUnnt
0.5
Curve
30
BIG- wnto choices O-shot
BIG w/o choices 1-shot
DIG-G choices 2-shot
0.1
25
20
0.3
15
Exac:
100
w/o PASSWIILL
107
100
10P
107 1.0 1.00 30 3014
Model, Size
Model S1.z
Figure 1: We can discriminate subtle performance improvement (left), which is evaluated as all zeros in conventional methods (right). The right figure directly uses Figure 9()) in Sorscher et al. (2022) as a comparison, which the authors utilize to illustrate a "break-through" behavior in task performance. The internal figure inside the left figure shows the performances in log(- loo())) space, which displays strong linearity, supporting the task scaling law (3).
The challenge in extending loss caling law to task performance predominantly stems from the dis- contimuity observed in task performance during scaling. Language models below a certain size yield trivial performance, i.e., random guessing on multiple choices or zero scores on generation tasks. However, when the model size surpasses a certain threshold, a distinct surge in performance ap- pears, which leads to substantially non-trivial performance. This phenomenon is summarized as the "emergent abilities" (Srivastava et al., 2022: Wei et al., 2022a), and is observed across various model families and tasks. It seems that qualitative changes happen inside the model, which makes the model start to manifest unique capabilities. While these emerging phenomenon indicate that LLMS are becoming stronger, they complicate the prediction on task performance.
A pivotal question arises: can we unlock predictable scaling of the task performance, from the apparent discontinuities? We hypothesize that the perceived discontinuity from trivial to excellent performance might stem from limited evaluation resolution By employing more nuanced resolution, one could potentially uncover the scaling law for tasks. The most related work to ours is Schaeffer et al. (2023), which proposes two methodology to make emergent abilities continuous, i.e., "change of metrics" and "increase resolution" " by expanding test set size. Our motivation diverges from the "change of metric" approach of Schaeffer et al. (2023), which posits that employing other continuous metrics can cause emergent abilities to disappear. A limitation of alternative smooth metrics (e.g., distribution distance) is they yield insufficient insights into the target metrics (e.g., exact match) that evaluators intuitively perceive. In contrast, our method extends the "increase resolution" approach in a novel way, which target directly at predicting the performance such as code generation in our experiments.
We introduce an evaluation strategy named PASSUNTIL that, for the first time, enables quantitative exploration of the scaling properties of task performance. PASSUNTIL deploys extensive random sampling in the decoding phase (e.g., 10 sampling times), and evaluates each sampling result until any generation passes the target test. Therefore, this evaluation strategy has infinite measurement resolution as long as computational resources are not bounded. Moreover, it can provide maximum likelihood estimates of target metrics such accuracy and exact match. To refine our evaluation resolution and accuracy, we suggest fitting to instance-level scaling law since different test instances might have different speeds of performance improvement during scaling.
With the proposed evaluation strategy, we delve into the scaling law governing task performance. To begin with, train two series of models ranging from 0.03B to 2.4B. These models strictly adhere to pre- pre-train loss scaling law, providing a solid foundation for analyzing task performance scaling behavior. We mainly disclose two findings in our exploration.
"y "resolution", view evaluation a measurement of the real probability of completing task. And resolution is the smallest prohahility difference that the evaluation strategy can detect.
2

---

Published as a conference paper at ICLR 2024
Firstly, task performances are predictable with PASSUNTIL. We validate the presence of subtle but non-negligible performance in smaller models that can be captured by PASSUNTIL. These performances are on the order of 10-5 and exhibit steady enhancement the model scales up. Subsequently, we derive the mathematical form of task scaling law, experimentally verifying an almost strict linear relationship between log(-log( (og()UU) and log(N), where PU denotes the estimation of target metric given by PASSUNTIL and N is the number of model parameters. This relationship enables us attain highly accurate predictions. For instance, in the code generation task, our predictions exhibit a mere 0.05% deviation from the actual values.
Secondly, we discover a phenomenon of accelerated emergence. To begin with, we discover that the shape of the task scaling curve is not uniform across tasks. Several task manifest scaling func- tions that diverge from the typical task scaling law. In other words, their scaling curve is smooth and incremental but can not be fitted by the typical scaling law function. Their scaling curve of log())) w.r.t. log(N) is concave, which is akin to an acceleration in the performance scal- ing speed. We provide mathematical definition of such phenomenon. With the quantitative defini- tion, we exclude a possible multi-step reasoning explanation (Schaeffer et al., 2023), and propose an alternative hypothesis. This hypothesis is predicated on potential transformer circuits (Nelson et al., 2021) that are used to explain the "grokking" phenomenon (Power et al., 2022; Varma et al., 2023). It is in harmony with the observed scaling function.
Our work represents the first open-source attempt regarding the predictability of task performance. While GPT-4's report (OpenAI, 2023) has initiated this exploration, it has not provided comprehen- sive details, We will open-source all checkpoints to facilitate future research in this direction.
2 RELATED WORK
Predicting task performance before training is an aspirational objective for the development of pre- dictable Al systems, and a multitude of studies approach this aim from various perspectives.
Loss Scaling Law. Scaling phenomena have been observed across a broad spectrum of deep learning architectures. The power-law scaling behavior of loss RNN- models is investigated in Hes- tness et al. (2017). Kaplan et al. (2020) delineate the loss scaling trends for Transformer-basee language models and explores the scaling behavior of optimal hyper-parameters. They formally established the following scaling law
L= cN Lo,
where N is the number of non-embedding parameters of LLM, c, a are positive coefficients, and Lo is the irreducible loss representing the randomness in data. This formulation has catalyzed the proliferation of LLMs. Subsequently, scaling laws are established for various domains and sce- narios, including multi-modality (Henighan et al., 2020; Zhai al., 2022), computation constraint scenario (Hoffmann et al., 2022), data engineering (Muennighoff et al., 2023; Sorscher et al., 2022), and reinforcement learning (Gao et al., 2023). Yao Wang (2023) extend the scaling law into loss scaling methods. The relationship of our work with these
prediction by introducing existing literature is twofold. First, these works concentrate on training and validation loss metrics, which do not reliably predict task performance. Second, our research builds these scaling laws and extends the mathematical form of (1. to the scaling law of task performance.
Scaling Behavior of Task Performance. Despite the predictable decrement in LLM loss, task performance improvements are twisted during scaling. While some tasks, predominantly those relying memorization of knowledge, have shown progressive improvement, numerous tasks exhibit breakthrough behavior as model size increases (Srivastava et al., 2022; Wei et al., 2022a). Wei al. (2022a) illustrate that the concept of "emergence" is also pertinent to prompting techniques such as Chain-of-Thouue 2020), complicating the pursuit of understanding the scaling law of task performance. that the law of loss scaling offers no assurance for task performance, engendering a lack of guidance in pre-training methodology. Fortunately, several studies endeavor to demystify these emergent abilities. GPT-4's technical report (OpenAI, 2023) reports that GPT-4's task performance can be predicted with less than 1/10000 of computation. albeit without disclosing the methodology and acknowledging that certain abilities are still beyond prediction. Subsequent research (Schaeffer et al., 2023) attributes emergence to two reasons. The first one is non-smooth metrics. We disagree with it since the alternative metrics could not explain the sudden increase in target metrics such
(Wei et al., 2022b) and In-context Learning (Brown et al.,
appears
3

---

Published as a conference paper at ICLR 2024
exact match, which of paramount interest We align with their second attribution to improve resolution by adding more test samples. Different from their method, we propose a practical method to improve resolution without the need of adding test samples. Our work is also the first open-source attempt to quantitatively investigate the scaling behavior of task performance, proposing task scaling law and accelerated emergence phenomenon.
3 PILOT EXPERIMENTS ON INCREASING RANDOM SAMPLE NUMBERS
We initiate our exploration by visualizing the effect of improving evaluation resolution on open- sourced models. We choose four small models and evaluate them on subsets of BigBench task (Srivastava et al., 2022): Emoji Movie and Date Understanding (see Appendix D.4.2 and D.4.3 for the subsets). We employ beam search and random sampling (with three sample times: 100, and 10,000) during decoding. If any sampled answer of a test instance is evaluated as correct, then the instance marked as "passed". We present the number passed instances in Figure
Eauii Moric
Daie Unknisolig
007-000
022
00213k
Fradion
Figure 2: BS denotes beam search, RS-K denotes random sampling K times.
We can see that even for such tasks presenting substantial difficulty to small models, most instances are passable with enough random sampling times, which will contribute to the subtle task perfor- mance improvement. Inspired by this observation, propose our evaluation strategy that centered around improving the resolution of evaluation.
4 METHODS
In this section, describe our methods to increase the resolution of evaluation, which empowers the investigation the scaling behavior of task performance. The first an evaluation strategy PASSUNTIL, and the second is an instance-level scaling curve fit. We also derive the task scaling law based on the loss scaling law.
4.1 INFINITE RESOLUTION WITH PASSUNTIL
We view task performance evaluation as the measurement of the probability of a model passing task. Given task instance suppose the probability that model pass it is P(s). job is to estimate E. [Ps)) Randomly sampling fixed time K could estimate P(s). However, define the budget K that is both acceptable in computation and has enough resolution for hard samples that have small P(s). We propose PASSUNTIL which performs an evaluation right after an answer generated and determines whether it is passed before we sample the next generation.
hard
We stop sampling until r (a constant) samples have passed the evaluation and record the sampling number K. We name the estimate of Pss) as the PASSUNTIL score PU, which is defined
PU K
Theoretically, PU has the capability to measure success rates that are infinitesimally small. The PASSUNTHL has the following properties.
"The definition of "pass" does not need to be generating exactly the ground truth answer. For example, suppose predict model's performance AlpacaEval (Li al., 2023b), we define "pass" as the model generation being hetter than GPT-4, judged by GPT-4. Therefore the "pass" has broad application.
4

---

Published as a conference paper at ICLR 2024
Theorem 1. PU is maximum likelihood estimate for P(s).
Proof. The failure time f = K follows the negative binomial distribution with success proba- bility P(s). r/K is known to be an maximum likelihood estimate for P(s).
In practice, we set r to as small as 1 or considering the efficiency of evaluation. We also set the upper bound of K to large number, such as 10", to prevent endless sampling if we encounder an extremely low P(s). Note that many instances stop before reaching this upper-bound. Next we discuss the necessity and limitations of PASSUNTIL.
Necessity. Generally, deriving P(s) theoretically from the token probability on the ground truth solution is not feasible. This is due to two primary facts: firstly, there are likely to be multiple viable solutions; secondly, even though there is only one solution, there exist multiple decoding approaches besides the optimal tokenization to decode the solution'.
Limitations. (1) Currently, our evaluation strategy is designed to be applicable when a random base- line achieves P(s) O. In the context of multiple-choice grade the evaluation metric, evaluations tend to exhibit a biased high score relative to the true performance of the model (e.g., P(s) = 0.25 with random guess for four options). This random noise can overshadow the improvements made by smaller models. The exploration of scaling law for tasks with non-zero random baselines remains a subject for future research. (2) We currently only consider random sampling
viable target de- coding strategy due to its widespread use in LLMs. Using beam search as target decoding strategies and their relationship with random sampling poses an interesting avenue for future exploration and study.
4.2 FROM LOSS-SCALING LAW TO TASK SCALING LAW
Then, we derive the task scaling law that PASSUNTIL will follow. We assume that the test loss of generating the next token decreases according to the scaling law of (q.(1).
lyl II P(y [1 V1--1)
lyl
PU
II exp(-cNN Lus),
(3)
where F||j=| is the input sequence and Vi:jol is the most probable sequence that decodes the correct answer (assuming its dominance compared to other sequences). Assume that the test sample passable given a sufficiently potent LLM, then the irreducible loss for each token Lo: approaches O. And assume the test loss of each token in the answer decreasing with uniform speed when scaling (i.e., 0; a, Vi), we can derive the following function for PU on task performance:
(U a; N) ~ exp( Exp(E= NNN-9)= (sp(-cN--y
(4)
where E, c. The resulting mathematical model is similar that in GPT4 technical port (OpenAI, 2023) and Equation (4) in Schaeffer et al. (2023).
4.3 FITTING STRATEGY
Dataset-level Fit. When fitting the parameters c, a in PU, dataset-level fit is plausible. For the j-th model in the scaling curve, the individual test sample's PU is first averaged over the test set to procure log(- lo followed by linear regression log Nj.
Instance-level Fit. We notice that differences between instances lead to different scaling behaviors, which means a dataset-level might not be accurate when the difficulty in the test set diverse. For example, PU on easy questions get saturated to on small model while the hard questions still receive trivial performance (see Appendix B. for illustration). We propose to fit an individual PASSUNTH score (IPU) for each question and aggregate them into estimate for the whole dataset.
PU((c..a.:::) N)
Σ IPU(C,, N)
(5)
[S]
For example, [4513], 1717,18), and [16,17,18] all decode into string "123" in GPT-4's tokenizer with vocab "cl "00kkba....""
5

---

Published as a conference paper at ICLR 2024
5 PREDICTABLE SCALING EXPERIMENTS
In this section, we demonstrate how the proposed framework works in practice. We first pre-train Iwo series of language models ranging from 0.03B 2.4B using two dataset mixtures. We predict the performance of the 2.4B model based on the performance of the rest of the models in the series.
5.1 SCALING CONFIGURATIONS
Model Configurations. We propose to keep a consistent "shape" of the Transformers while ex- panding their sizes. For the i-th model in the scaling curve, we the number of layers be 4i, the number of attention heads to be [ x(++2] J, and the dimension of head to be 64. This results in the hidden state's dimension dm being d We set the dimension of the feed-forward layer to be
2.5dm. The specific values are listed in the model configurations in Table of Appendix D.I. The architecture is similar to LLaMA (Touvron et al., 2023a) (see Appendix D.1 for details).
Pre-training Corpora. For series I, we use the StarCoder dataset (Li et al., 2023a) as our pre- training data. For series 2, we Leveraging the optimal compute I LLMs (Hoffmann et al., 2022), we set the maximum pre-training tokens for each model size to be the 20,,, where N is the number of non-embedding parameters of the model. The detailed portion within the data mixture can be seen in Appendix D.2.
mixture of StarCoder and Pile (Gao et al., 2020) dataset.
Modd Scrce
Modid Serics 2
Ioan Loo
1f 0.906
0
1.47 1.25
00003
6/0
1000
Ooupute
Compato (EPLON))
Figure 3: Training loss of the two series of models trained on different data mixtures. The internal figure illustrates the end-step reducible loss relative to model size, represented in logarithmic scale.
Hyper-parameters are also of paramount importance in training a series of models that scale successfully. We examine the cosine learning rate scheduler, aligning our approach with that of Hoffmann et al. (2022), and determine the critical batch size in accordance with Kaplan et al. (2020). Nonetheless, due to constraints in space, we move the details to Appendix D.3.
5.2 Loss SCALING LAW VERIFICATION.
We present the training loss curves for models in Figure It is evident that the end-step training losses decrease in line with the scaling law. These empirically observed loss scaling laws lay a foundation for the subsequent approximation of task performance. Note that despite the occurrence of the loss spike in the 1.5B and 2.4B models, convergence to the scaling law is ultimately achieved, exemplifying the robustness of such an empirical law.
5.3 DATASET-LEVEL FIT
We select HumanEval (Chen et al., 2021), Emoji Movie, and Date Understanding (Srivastava al., 2022) as the evaluation tasks. Note that Emoji Movie is conventionally cited as representing "emer- gent abilities" (Srivastava et al., 2022) (see the right figure Figure 1). HumanEval is assessed using a zero-shot learning setting, while Emoji Movie and Date Understanding are evaluated em- ploying 4-shot In-context Learning (Brown et al., 2020). We additionally use Mhain-of-Thoughee Reasoning (Wei et al., 2022b) for Emoji Movie. See Appendix D.4 for the illustration and evalua- tion details of each task. remove the distracting test instances from our evaluation list. For Emoji Movie, remove the movie names that are common words (e.g., "it") identified by NLTK (Bird et al., 2009). These common words make the exact string match susceptible to random guess's correctness See Appendix D.5 for details).
6

---

Published as a conference paper at ICLR 2024
710
Ted Pafaseer
Pefarwe:
128
14
01803
0800
310
0.50
83
0/0
6758
870
02
111
370
Patimaiin
1003
Rad Potcoadia
Feal Petiensio
0291
2
1003
2.10
1001
1000
3/2
300
0000
1U,03 5.300
1.5
E.0L2 110 1St
0.80
130
Data Dedeeeeetade
Figure 4: Task performance scales predictably with model scale. The red points denote the real performance of 2.4B model, which are close to the task scaling laws fitted from 0.03B to 1.5B.
We observe that all three tasks exhibit a strong linear relationship between log(- log (PU)) and log(N), verifying the success of task scaling law given by Eq.(3). The estimation of the scaling law functions utilizes the 0.03b to 1.5B models, which predicts the performance of the 2.4B model with small yet acceptable deviations.
5.4 INSTANCE-LEVEL FIT
According to $ 4.3, we take the difference among test samples into consideration to improve the estimation. We plot how instance-level PASSUNTIL scales in Figure 13 of Appendix E.4. The fitted curves demonstrate that the performances of different instances not only originate from unique starting points but also scale at varying speeds. Nevertheless, they can be fitted by task scaling law individually. Some instances deviate from the scaling law, which needs future investigation.
Method
HvaanEval (1)
HumanEval (2) Date Understanding (2) Emoji Movie (2)
Real Value Dataset-level Fit
0.05990
0.04279
0.00346
0.002608
0.06550
0.05191
0.00377
0.002381
Instance-level Fit
0.05987
0.04402
0.00352
0.003112
Table I: Prediction of our framework compared to the real performance on two series of models. The number after the task denotes the model series used in the evaluation.
Bellame levd F.
Fi1
Vide
2
00.32
Seaxnrd Gow.dimel
10 10
35. 21e 15
35 Mec 280
Figure 5: PU w.r.t. the test loss on Figure We successfully predicted the performance of 2.4B HumanEval of model series 1. model with 0.05% deviation (left) and 1.7% deviation (right).
Estimating PASSUNTIL from Test LOss. Estimating at the instance level presents challenges for hard instances that lack adequate non-zero PU values for fitting. These samples may also contribute to PU as the model size increases. We suggest leveraging test loss on ground truth answers to assist the prediction for such instances (See Appendix A.2 for a detailed discussion of its validity). We leverage the "easy" instances, which have both test loss and non-zero PU to estimate the relation between test loss and PU (Figure 5). Then we predict the test loss of each instance on 2.4B model transform the predicted test loss to predicted PU
based on 0.03B ~ 1.5B models. Finally, Details are presented in Appendix E.2. We provide according to the aforementioned relationship. the final prediction result of 2.4B model in Table 1, and draw the predicted PU curve in Figure 6. We can see that the predictions are accurate, with only 0.05% difference on HumanEyal of series and 1.7% difference on Date Understanding of series

---

Published as a conference paper at ICLR 2024
6 QUANTITATIVE ANALYSIS OF EMERGENCE
Building on the discovery of the predictability of task performance, we proceed with our in- vestigation into a quantitative analysis of scaling behavior of broader range of tasks. We prove that even with the refined resolution brought by PASSUNTIL and predictability of other emergent abilities, there are still certain abilities hard to be predicted. We establish their mathematical definitions, and examine the possible explanations for such scaling behaviors.
We study the scaling curve on the "Unnatural In-context Learning (UICL)" categories in Big- Bench (Srivastava et al., 2022). "Unnatural In- context Learning" is a set of 8 tasks designed to specifically study the in-context learning abil- ity. These tasks involve input-output pairs that have been intentionally altered deviate from the typical training distribution, thereby neces- sitating the model's focus on unconventional in-context patterns. Task details and examples are in Appendix D.4.4. We randomly select 20 questions in the test set from each task and sam- ple 4-shot examples from the remaining ques- tions to serve in-context examples. The eval- uation metric employed is the exact match, and the upper bound sampling time is set to 10a, When fitting the scaling curve, we only utilize the dataset-level PASSUNTIL since these test instances are manually constructed to test one skill of LLM and thus might be devoid of dif- ficulty variation. Since our test set is small, we bootstrap 100 times from the 20 question's test result and use the bootstrapped to calculate the standard error of each PASSUN UTII estimate (shown in the green hue in the Figures).
Suckin Sciling
455
SepeRcaei Lay
Law
02220112
10
BURDALS 025
Figure 7: Scaling curve for task "Dates" and "Identity". Concave functions are observed be- tween log(- log(PU)) and log N. Scaling law fit curves are in grey and super-scaling law fit curves are in green.
Ler
Sab-Seckng Soee
Categorization of Emergence. The evaluation on task "Dates" and "Identity" is shown in Fig- ure 7. Other tasks are shown in Appendix E.3. "Dates" exhibit very smooth and consistent im- provement starting from 0.03B, while the other tasks are bit twisty. Nevertheless, 5/8 of these in-context learning tasks display a strictly concave function between log( log(PU) and log N. The others (3/8) miss 1 or 2 valid estimation points due to their extreme difficulty for 0.03B and OB models, since O PASSUNTIL is obverseved even with 10a sampling time, which we left for future exploration. The 5/8 tasks deviates from the scaling law (Eq.(3)) which requires this function to be linear. This means, unlike those tasks governed by the task scaling law, where "growth speed" is uniform across different model sizes, there exist some tasks that an increase in "growth speed" a as models enlarge. This phenomenon exemplifies an accelerated emergence phe- nomenon. To provide concrete discussion of accelerated emergence, we provide our categorization of task scaling curves first.
Figure 8: Three basic types of scaling curve, cor- responding to convex, linear, and concave fune- tion between log( log(PU)) and log N.
Mathematical Definition of Emergence. Since the loss scaling law of Eq.(1) is the only widely accepted principle during model scaling, we rely on its derived task scaling law of E3.(3) as a separator between emergence and other scaling behavior.
Definition Given a spectrum of models, we let the number of non-embedding parameters be variable N, suppose the PU(N) estimated by PASSUNTH. on a task is a continuous function of N. Define F(N)= = log(- log Jog(PU(N)) then the scaling curve of a task can be categorized into three basic main categories
*if F(N) has both convex and concave parts, then we can call it mixed growth.
8

---

Published as a conference paper at ICLR 2024
1. if F(N) is a linear function log. N, then the task obeys scaling law growth.
2. if F(N) is convex function of log N, then the task obeys sub-scaling law growth.
3. if F(N) is concave function of log N, then the task obeys super-scaling law growth, or " "accelerated emergence".
Figure 8 shows visualizations of three types of growth. Qualitatively, the scaling curves of all three types appear analogous to exponential growth when performance starts to become noticeable. However, they are qualitatively different. Task scaling curves with task scaling law growth sub- scaling law growth are easier to predict and control, whereas accelcrated emergence is not easy to predict, which might go out of control when the model gets larger.
Cause of Shape of Scaling Curve. The above mathematical definition provides us the opportunity to examine the hypothesis regarding the genesis of these scaling behavior. Here, we first study the following hypothesis: Emergent abilities may be induced by multi-step reasoning (Srivastava et al., 2022; Wei et al., 2022a; Schaeffer et al., 2023).
We prove that, surprisingly, "multi-step reasoning" leads to sub-scaling law growth.
Theorem 2. Suppose each reasoning step's 's success rate, measured by PASSUNTIL obeys the scaling law growth, then the mulli-step success rate follows the sub-scaling law growth.
Proof. Suppose the success rate of reasoning step i obeys scaling law growth with coefficient oa and agg then FNN) = log (E, C exp (-a; log N)). Using
Schwarz inequality, we can
prove that F²F a(log Nya 2 O. Therefore, the scaling curve is convex. See Appendix C.1 for more. O
This proof also be understood more intuitively: the growth speed will initially be boosted by the improvement of those easy steps, and eventually be bounded by the most difficult steps, thus showing a decreasing growth speed. Then, we propose alternative hypothesis: suggesting that multiple neural "circuits" (Nelson et al., 2021) may be represented within the LLMs, and that as long one such circuit can successfully solve the test instance, the test instance is deemed passed. This hypothesis is inspired by the explanation of "grokking" phenomenon given by Varma et al. (2023). They propose that there exists a memorization circuit and generalization circuit inside the transformers, and the "grokking" phenomenon is led by the generalization circuit getting more effi- cient than the memorization circuit during training. We will demonstrate that with this hypothesis, the scaling curve exhibits characteristics of emergence.
Theorem 3. Suppose multiple circuits i exist in the LLMs that are responsible for solving the task, and each displays scaling law growth and has PU;- And suppose the success rate of the task is the majority voting of these circuits, i.e., F(N) = log (- log max; PU,). Then, F(N) is a concave function of lo
Proof. F(N) = min,(log e 04 log N). Since the minimum operator keeps concavity, F(N) is concave function of log N. See Appendix C.1 for a more elaborated proof.
We loosely test the hypothesis by fitting the scaling curve for the UICL task. In practice, similar to Varma al. (2023), we adopt soft version of the majority voting. We apply a weighted com- bination between two circuits. And we assume the number of the circuits is 2. Therefore, we fit wi (aj log N - log c1) + w2(a2 log N - log c2) to F(N), where wi and W2 is given by the Soft- max of a; log N log c, The resulting fit curve is demonstrated in the green line in Figure 7 and Appendix E.3. We can see that this hypothesis produces fit curves that align more accurately with the observed performance scaling curve.
CONCLUSION.
Our work introduces a novel evaluation strategy capable of detecting minimal performance improve- ments during model scaling, thus opening avenues for quantitatively measuring the task scaling laws and the emergence abilities. This method has enabled the successful prediction of the task perfor- mance of larger models. Additionally, we have performed a quantitative analysis of emergent abili- ties, providing clearer insight into their nature and origination. This research not only enhances understanding of LLMs' scaling properties but also sets the stage for future explorations in scientific scale-up of L.L.s.

---

Published as a conference paper at ICLR 2024
ETHICAL STATEMENT
In this paper, we demonstrate that although we can predict a set of emergent abilities, the accelerated emergence remains hard be predicted. The hypothesis regarding the cause of accelerated emer- gence implies that we need a better understanding of the working mechanism to produce accurate predictions for such emergent ability. Without understanding of the working mechanism, any fit curve to the early stage of task performance improvement might be governed by another stronger, yet unknown, "generalization" eircuit when the model gets sufficiently large. Thus, this hypothesis calls for deeper research into the mechanism of LLMs to prevent the safety concerns brought by accelerated emergent abilities.
REPRODUCIBILITY STATEMENT
We will open-source and all evaluation scripts for reference.
ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China (No.2022ZD0010550))
REFERENCES
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O' O'Reilly Media, Inc.", 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners. Advances in neural information processing systems, 33:1877-1901, 202. 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained code. arXiv preprint arXiv:2107.0337 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 14387. 2023.
Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference Fairness, Accountability, and Transparency, pp. 1747-1764, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.0 08415, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.085 003000,090000 2020.
10

---

Published as a conference paper at ICLR 2024
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXtv:2010.14711, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani. Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary. Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, 2017.
empirically, arXiv preprint /7
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0886/ 2020.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprini arXiv:2305.06 2023a.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cnetruction-oolowinng models. https:://// //gi thub www/otttsut.oooooooooooooooo.ase laalllabbaaa wwwaaaaaaaaaaa.aaaaaaaaaaaaaaala _eval, 2023b.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305. 16264, 2023.
Elhage Nelson, Nanda Neel, Olsson Catherine, Henighan Tom, Joseph Nicholas, Mann Ben, Askell Amanda, Bai Yuntao, Chen Anna, Conerly Tom, DasSarma Nova, Drain Dawn, Gan- guli Deep, Hatfield-Dodds Zac, Hernandez Danny, Jones Andy, Kernion Jackson, Lovitt Liane, Ndousse Kamal, Amodei Dario, Brown Tom, Clark Jack, Kaplan Jared, McCandlish Sam, and Olah Chris. A mathematical framework for Transformer circuits. 2021. URL https: / //transformes www.nnfooore--rruitttee pub 20/2222///2222 20121/frameet htt//worrmww/////mmeom///aaomm// wwwiiiidxihii. html.
OpenAl. Gpt-4 technical report, 2023.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gen- cralization beyond overfitting on small algorithmic datasets. arXiv preprint 22X11:2201.02177, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485 5551,, 2020.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? arXiv preprint arXiv:2304. 15004, 2023.
Noam Shazeer. GLU variants improve transformer. CoRR, abs/2002.05202, 2020. URL https: wwwpxii.rrg/ab////000/
05202.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
11

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXtv:2302.1397,, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2337.00228. 2023b.
Vikrant Varma, Rohin Shah, Zachary Kenton, János Kramár, and Ramana Kumar. Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390, 2023.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0777 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 248377, 2022b.
Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ry- der, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiw:200.003466 2022.
Yiqun Yao and Yequan Wang. Research without re-search: Maximal update parametrization yields accurate loss prediction across scales. arXiv preprint arX1v:2304.00875, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lueas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022.
12

---

Published as a conference paper at ICLR 2024
Note: clicking each 4 in the appendix will allow you to jump back to the corresponding position the main paper to continue reading.
A DISCUSSION
A.1 LIMITATIONS
Our work has several limitations.
1. Scale Limitation. Firstly, we currently do not extend the prediction of task performance to much larger models (e.g., 10B and more). We will try to scale up the experiment in the future.
2. Scope Limitation. Secondly, we are not claiming that we can accurately predict the task performance on all tasks. For example, only fit the scaling curve for the tasks that display emergence. We still have a long way to go before we can predict these tasks. Even for the tasks that might not display "emergence" we currently do not complete thorough prediction for them. We will add predictions on more of these tasks in the future. That said, predictable scaling, as OpenAl points out (OpenAI, 2023), is still very challenging and aspirational goal for AI researchers. Our work serves the initial attempt it.
3. Explanation Limitation. Thirdly, although we propose a hypothesis regarding the cause validation for the hypothesis is superficial. We satisfactorily
of accelerated emergence, the scaling curve under this hypothesis. However, whether this hypothesis is true from the underlying mechanism remains unknown.
A.2 DISCUSS OF THE USE OF LOSS AN ASSISTANCE METRIC
In our experiments of Individual PASSUNTIL, we use loss on ground truth as an assistance to PAS- SUNTH. which may raise a misunderstanding: why don't you directly use loss to predict the perfor- mance? We provide detailed illustration below.
1. It's important distinguish between "loss is not predictive of task performance" and "loss can help predict task performance." The former suggests that loss is not sufficient statistic for estimating task performance without other measurement, while the latter indicates that loss is one of useful factors in improving prediction accuracy. In our paper, we clearly verify both statements. Without utilizing the PassUntil method, one cannot deduce actual performance (accuracy) solely from loss values. For example, a loss of 1.0 does not directly translate to an accuracy of 0.2 for a task. And actual performance must be empirically measured. Furthermore, shown in Figure 5, the loss of individual sample does not have one-to-one correlation with PassUntil results, much less with discrete accuracy.
2. However, loss does provide useful information. Once measure PassUntil across large sample set, we can establish a statistical relationship between loss and PassUntil (not pos- sible if we only rely on loss data). This relationship can enhance our prediction accuracy.
3. The incorporation of loss for improved predictions is driven by practical considerations, such as limited computational resources, rather than being necessity. Figure demon- strates that even without loss data, we can accurately predict task performance. Imagine scenario where we can measure every sample with sufficient resolution to ensure each is passed at least once; in such case, loss data would not be necessary.
B SUPPLEMENTARY MATERIALS FOR PASSUNTIL
In this section, we provide some additional comments about our evaluation strategy. We present our intuition for instance-level PASSUNTIL.
B. INSTANCE-LEVEL PASSUNTIL INTUITION.
Table 2 delineates the PASSUNTIL for both an easy and challenging instance within Hu-
manEval. It was observed that with increase in model size, the easier instance (index 24) exhib-
13

---

Published as a conference paper at ICLR 2024
ited a higher PU. However, the challenging instance (index 20) continued manifest trivial performance, suggesting a potential variance in their respective scaling curves. Blindly averaging performance instances will make the improvement hard instances vanish compared to the easy ones, leading to an inaccurate prediction after the model gets saturated in the easy instances.
PASSUNTIL
Instance index
0.03B
0.1B
0.2B
0.5B
0.9B
1.5B
20
0.000625
0.001875 0.568125
0.008125 125 0.796875
24
0.00375
0.05125
0.350625
0.3625
Table 2: HumanEval, an easy instance (index 24) gets a much higher PU compared to the hard one (index 20).
C SUPPLEMENTARY MATERIALS ON EMERGENT ABILITIES
THEORETICAL ANALYSIS OF HYPOTHESIS
b We present the proof of two theorems about the cause of emergent abilities in Section 6 briefly. In this section, we provide the elaborated proofs.
Theorem 2. Suppose the success rate of each reasoning step measured by PASSUNTIL, obeys the scaling law growth. Then the multi-step's 's success rale rale follows the sub-scaling law growth.
Proof. Suppose the PU of reasoning step i obeys a scaling law growth with coefficient & and ags The overall success rate is
F(N) log
log II P.
log
-log] II exp (-c exp(exepppee - log N))
(6)
log
Σ c exp (-a; log N)
Then take the second derivative of the F(N) over log N, we can get
F
afa exp(-xppppppo log N) E, exp(-a; log N)
Ölog N)²
( 4 expepppppp -a; log N))2
E aiq exp(-a; log N))²
C 4 expeeppuo log N))2
Let k; e exp(-a;; log N) > 0, the (7)
Σ, af E 12, axkk;)2
(8)
E k2)
Using Cauchy-Schwarz inequallty,, can prove that
8 F
20, Va, 0,0
(9)
Alog NY
Only when a a.V.E/E Constant, the equation holds, i.e., when all the steps in the reasoning chain scale with the same speed. Thus, F(N) is a convex function of log N, and the scaling curve exhibits sub-scaling law growth.
Theorem 3. Suppose multiple circuits exist in the LLMs that responsible for solving the task, each displays sealing law growth, the PASSUNTIL of the task is the majority voting of these circuits, iee. F(N)= = log (- log max; P:) Then, F(N) is concave function of log N.
14

---

Published as a conference paper at ICLR 2024
Proof.
F(N)= log (- log max exp (-q exp( a; log NT))
log min o exp(-a; log log
(10)
mmin(oo c; a; log
Since the minimum operator keeps concavity, F(N) is concave function of log N.
D DETAILS OF EXPERIMENTAL CONFIGURATIONS
In this section, we detail the model configurations, training configurations, and data mixtures used for the two series of models.
D.1 MODEL CONFIGURATION
b Table shows the detailed model configurations and training configuration of the series models in the scaling curve, which aims keep uniform "shape" while expanding the model size. We use a similar architecture to Llama 2 (Touvron et al., 2023b). Some minimal differences include: we use
tied embedding between the input and output embeddings, and we use gated- GeLU (Hendrycks & Gimpel, 2016) instead of gated-SiLU SiLU (Shazeer, 2020).
Name
N (B)
dm
dzf
dn
L
BS (M)
TS
Tokens (B)
dr
2.5dm
64
1(8+-1)
[v
1
0.03B
0,036
512
1280
64
8
12
0.33
2196
0.72
0.1B
0.109
768
1920
64
12
16
0.88
2464
2.18
0.2B
0.241
1024
2560
64
16
20
1.57
3064
4.82
0.5B
6
0.499
1344
3360
64
21
24
2.10
4758
9.99
0.9B
0.892
1664
4160
64
26
28
2.95
6049
17.9
1.5B
8
1.542
2048
5120
64
32
32
4.26
7230
30.8
2.4B
2.45
2432
6080
64
38
36
5.51
8900
49.0
Table 3: Model configurations and training configurations of the models in the scaling curve. N(B)
represents the number of non-embedding parameters the model, measured billions. BS(M) indicates the number of tokens in batch (i.e., batch size) used to train the model, measured in millions. TS denotes the training steps. Tokens(B) refers the total number of tokens used to train the model.
D.2 PRE-TRAINING CORPORA
t We pre-train two series of LLMs using different data mixtures to demonstrate the generality of our experiments. Tables and respectively display the specific data mixture proportions for Series and Series 2 LLMs.
D.3 HYPER-PARAMETER STUDY
b Learning Rate. We use a cosine learning rate scheduler, analogous to those in preceding stud- ies (Touvron et al., 2023a;b; Hoffmann et al., 2022). The maximum learning rate is consistently fixed at 0.01 across varying model scales, with no significant loss explosion at this rate. This sta- bility is potentially attributed to normalization strategies (Yang et al., 2022) and increased batch size across scales. Echoing findings from Hoffmann et al. (2022), we ascertain that for training LLMs up to specific end step, the optimal cycle length of the cosine learning rate scheduler is equivalent to the end step. Deviations from this optimal cycle length, either longer or shorter, result in sub-optimal performance.
Batch Size. estimate the optimal batch size required for model pre-training, we replicate the experiments in alignment with Kaplan al. (2020) to determine the optimal batch size of model
15

---

Published as a conference paper at ICLR 2024
and adjust the real batch size slightly from the optimal batch size maximize GPU utility. The values of batch sizes and train steps are listed in Table 3.
D.4
TEST SET CONFIGURATIONS
In this section, we introduce the test sets and evaluation details in our experiments.
D.4.1 HUMANEVAL
The HumanEval (Chen et al., 2021) dataset released by OpenAl encompasses 164 programming problems. Each problem is composed of a function signature, a docstring, a body, and multiple unit tests. Our assessment of this dataset is conducted utilizing zero-shot approach. The completion of code, as generated by LLMs, is deemed passed only it successfully passes all unit tests. For our evaluations, set the upper bound of sampling times in PASSUNTIL to 10
D.4.2 EMOJI MOVIE
& Emoji Movie is a subtask of BigBench (Srivastava et al., 2022) and requires LLMs to identify well-known movies from their plots described using emojis. Our evaluation methodology incorpo- rates the use of Chain-of-Thoughh (CoT) and 4-shot In-context Learning. We randomly seleet test instances (initially 50 instances, with 9 distracting instances removed, see Appendix D.5) to constitute our test set and arbitrarily designate 4 instances as few-shot contexts. For CoT, we use GPT-4 to generate prompts for each instance in the few-shot context. The model is expected to read the 4-shot in-context examples, generate thought, and then provide the answer. Our evaluation methodology employs extract string match, i.e. where the output of the model contains the target film name. We set the sampling upper bound times set to be 10a.
D.4.3 DATE UNDERSTANDING
t Date Understanding, subset of BigBench (Srivastava et al., 2022), is constructed to evaluate the capability of LLMs in comprehending dates, by posing questions re- lated to the date reasoning. For the evaluation of this task, employ 4-shot In-context Learning. We ran- domly sample 47 instances to form the test set (initially 50 instances, with distracting instances removed, see Appendix D.5). We random sample instances from the remaining dataset to serve as in-context examples. We also use extract string match to measure the output from LLMs and set the sampling upper bound times 10
Corpora
Token Portion
StarCoder Python
0.3
StarCoder,Othear OOthers 0.7
Table 4: Pre-training corpora used for scaling the Code LLMs (model series 1).
Corpora
Token Portion
StarCoderPPythe StarCoder.Other Stack Overflow
0.15
0.12
D.4.4 UNNATURAL IN-CONTEXT LEARNING TASKS
0.03
Arxiv
0.05
& The Unnatural In-context Learning tasks serve as a series of distinctive subtasks within BigBench (Srivas- tava et al., 2022). These subtasks are designed to as- sess the models' ability to perform in-context learning where the context sequences are intentionally altered to be likely outside of the training distribution, necessitating the model's attention to unconventional in-context pat- terns. Some instances of these subtasks are exemplified in Table 6. For each task, 20 instances are randomly sam- pled to compose the test set, utilizing a 4-shot In-context Learning configuration. Four instances are randomly selected from the remaining dataset provide context. We use extract string match to measure the output from LLMs and set the sampling upper bound times to 10
Pile
0.65
Table 5: Pre-training corpora used for scaling the Code-Text LLMs (model ries 2).
16

---

Published as a conference paper at ICLR 2024
Task Name
Example
Dates
Input: 2015-10-22 Target: /10/22/2015!
Dates with Unnatural Form
Input: !08!24:1984! Target: 1984-08-24
Dates with Unnatural Con-
Input: 96980-49655-06744
Target: 149665//0674:96988)
tent
Dates with Unnatural Form and Content
Input: 149665110674199980)
Target: 96980-49665-10674
Identity
Input: a, b, c, d, e Target: b. c, d, e
Reverse Natural Content
Input: I, u, o, b, Target: a, b, o, u,
Reverse to Natural Content 2-digits
Input:
0, Target: 20
Target:
Input:
Table 6: Example Tasks in Unnatural In-context Learning Tasks
D.5 REMOVING DISTRACTING FACTOR IS IMPORTANT WHEN MEASURING TINY PERFORMANCE.
We notice that removing the distracting factor is important when measuring the minor perfor- mance gain during scaling. The distracting factor means that test instance is drastically different from the other test instance in terms of required abilities or evaluation bias. Note that we select the distracting factor based the observation of test instances, which does not lead to information leakage when predicting the 2.4B model.
For Emoji Movie, some of the movie names are common words, enabling even a modestly sized model to "guess" them correctly based on our assessment criteria: the determination of model cor- rectness is contingent upon the presence of movie names within the model's output. Figure 9 shows that there is significant association in the pass rates between models of varied scales. In other words, the scaling law does not have much of an impact on model performance for these prob- lems. Consequently, it becomes essential to exclude such distracting factors from consideration. We remove the movie names that are common words identified by the popular toolkit NLTK 5.
For Date Understanding, we omit the following instance shown in Table 7. These instances only require the model to extract the answer from the context and don't require reasoning about the date.
In GPT-4 report (OpenAI, 2023), they split the HumanEval dataset into separate bins with different difficulties and conducted scaling prediction for each bin, thus removing the distraction of easy examples to hard examples.
Example
Today's meeting is rescheduled am tomorrow, 10/16/1924. What is date tomorrow in MM/DD/YYYY?
Yesterday was 1/199, Today could not be 12/32/1929 because December has only 31 days. What is the date yesterday in MM/DD/YYYY?
Today Jane watching NFL 2003. What is date today in MM/DD/YYYY"
Table 7: Distracting instances in Date Understanding Tasks.
E ADDITIONAL. EXPERIMENTAL RESULTS
In this section, display some additional experimental results, including the additional fit curve of dataset level of PASSUNTIL, and the methods of utilizing test loss to assist the instance-level PASSUNTIL estimates.
hhttps://ww.nlte
17

---

Published as a conference paper at ICLR 2024
Inorgtion
107
ted
Jolaes
mosellghtt
10
0.069
Model Siec (8)
Figure 9: Large and small models have similar PU on these instances (mainly due randomly sample from the vocabulary space), which creates distracting factors in our experiments.
E.1 ADDITIONAL DATASET LEVEL PASSUNTIL RESULT.
The performances of series models on HumanEval are represented in Figure 10. This prediction is less accurate compared to series 1. However, with instance level PASSUNTIL, the prediction precision improves.
2.4
Curon Parfenmne
Py Curer
Redd Prrformunc
2.0
1.8
1.6
0.04
0.02
1.2
1.0
0.125
0.125
0.35
Modal Size (B)
Figure 10: Additional figure on Test Loss Assitted PASSUNTIL Estimate.
18

---

Published as a conference paper at ICLR 2024
E.2 ESTIMATING PASSUNTIL FROM TEST Loss
t As shown in Figure 11, we propose leveraging test loss on ground truth answers to assist the prediction for "hard samples". For model series and HumanEval task, the linear relationship found to be PU ~ 0.22T. For model series 2 and HumanFval task, the linear relationship is found be PU ~ 0.23T. For model series 2 and Date Understanding task, the linear relationship is found to be PU 0.96L. And for model series and Emoji Movie task, the linear relationship is found to be PU ~ 0.43L.
Averge Coraedradi
Zesepol Grensteni
Uralessnnaiingg 2)
Emaapillss 23
Figure 11: Additional figure on the relation between test loss and UNTTL..
E.3 MORE RESULTS OF THE UNNATURAL IN-CONTEXT LEARNING TASKS
In Figure 12, we present the scaling curves for the remaining fix sub-tasks of the Unnatural In-context Learning tasks. Notably, the curves in (a), (b), and (c) demonstrate concave pattern, correlating log( (g(ooo((F((V))) with log N. Specifically. the 2-digits task displays an interesting in- verse scaling trend, indicating further investigation to delineate clearer trend.
Regarding tasks in (d) and (e), we observed that these tasks pose significant challenges for smaller models. Specifically, models with 0.03B and 0. B.IB parameters failed to achieve non-zero pass rates,
19

---

Published as a conference paper at ICLR 2024
rendering the fit analysis less meaningful. Additionally, for the Reverse to Natural Content task, there's 's discernible, albeit slight, sub-scaling law growth trend. This trend may be attributed to the multi-step nature inherent in this task.
(b)
Poscne Salta el Coutesd
Dates Unaziand Form
350
3.50
1:55
000
3.20
3.5
7.40
1.0
055
1.35
13AGE E5 370
BARGHI DO 055
0,000
Separ
Sear Shakin
0.375
Secling Lon
Soiing Lawe Fii
1.335
0.034
0.314
0060
000
13033 155
1U 023
0.80
62
(d)
(e)
Dato Unsutarl Custeal Beza
Uniatard Gustar
Devene Natraal Curnrt
Sepertiilag Lev Soaza Lant Fii
SqpesSulila Lar
2:00
Sodiaa Lave Fr:
222
Scding Lane Pi:
0.00
1.25
1,00
11200
0..1
0.13
1.11
0000
0.04
3.4
0.06
Figure 12: Additional figure unnatural in-context learning. The grey line shows the scaling law fit, while the green line shows the super-scaling law fit.
20

---

Published as a conference paper at ICLR 2024
E.4 RESULT OF INDIVIDUAL PASSUNTIL ON MORE SAMPLES
Figure 13 shows more instances of individual PASSUNTIL scaling curves of model series on Humaneval task.
G-1
964
9-1
10
2-4
102
104
9-4
9-1
2c41
102
9-1
25
6-4
264
1a2
JPU
96.1
0-4
6-.1
9-1
3-11
1n-0
1n4
9ol
100
2-1
10.2
104
9-1
100
18
130
120
6-1
361
16.0
164
0155
01
01
038
010 01
030
940
03
1.0
32
000
01
02
055
Model Size
Figure 13: Result of instance-level scaling law fit. The label the left upper corner of each subplot denotes the index the sample in the test set
'nttps:// thtp///tggtttthoeee thub. wttpub.ccom//aaeee htw.anaaiahmaauuuuuue wtt.ee/awaeeeeeeeeeeeee////
21


Reviews:
[{"review_id": "FDpW0TAjzX", "reviewer": "Reviewer_BHGc", "summary": "The paper aims to understand the scaling properties of large language models (LLMs) and discover a task scaling law to predict task performance. The authors introduce an evaluation strategy called PASSUNTIL, which utilizes massive sampling in the decoding phase and has theoretically infinite resolution. They conduct a quantitative investigation into the scaling law of task performance, leading to two main findings: (1) Task performances are predictable with PASSUNTIL; (2) The emergent abilities do exist and can be characterized by super-scaling law.", "questions": "1. Scaling laws are generally in an average sense over a dataset but not for individual tokens. For example, in a single generation, an error incurred in earlier tokens is likely to cause errors in subsequent tokens. \n\n2. This paper evaluates the \"emoji,\" \"date,\" and \"identity\" tasks. However, the number of evaluation datasets seems relatively small compared to existing works like Wei et al., 2022a. As a result, it is uncertain whether the conclusions drawn in this paper are broadly applicable. Could you please discuss the generalizability of your findings?\n\n3. The scaling laws in Equation (4) are for general tasks, irrespective of whether they involve multi-step reasoning. Ideally, they should be applicable to all tasks. However, Figure 6 demonstrates that it does not work well for multi-step reasoning tasks, and the authors propose another super scaling law to address this issue. It makes this paper a bit ad-hoc. \n\n4. Theorem 2 is based on the premise that emergent abilities are induced by multi-step reasoning. However, Appendix E of Wei et al., 2022a includes numerous non-reasoning tasks that exhibit emergent abilities. \n\n5. In the evaluation of PU, both beam search and random sampling decoding are utilized. Does the temperature setting influence the performance, and if so, how is this factor taken into account?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper presents a pioneering open-source effort to predict task performance in large language models, aiming to significantly contribute to and encourage future research in this area.\n\n2. The authors introduce PASSUNTIL, a novel evaluation strategy that appears to provide a more equitable assessment compared to existing metrics, showcasing their innovative approach to addressing the challenges in the field.\n\n3. The proposed scaling laws demonstrate a strong fit with the data across several datasets, highlighting the effectiveness and potential applicability of the authors' findings in various contexts.", "weaknesses": "1. The derivation of Equation (3) appears to have some discrepancies. (Please refer to Question 1)\n\n2. The number of evaluation datasets used in this study is somewhat limited in comparison to previous works on scaling laws. (Please see Question 2)\n\n3. The task scaling law seems somewhat arbitrary, as some tasks require standard scaling laws while others necessitate super scaling laws. (Please refer to Question 3)\n\nOverall, this paper serves as a valuable starting point, but it lacks a comprehensive investigation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJ7vUiJJAn", "reviewer": "Reviewer_v7YM", "summary": "Previous work pointed out that limited resolution may yield biased estimates of LLM performance. This paper proposes sampling until a non-zero number of success is achieved, then shows doing so allows for better estimation of small quantities.", "questions": "1. Why did you need to train your own models? What's wrong with using existing models e.g. Llama, Mistral, Pythia? Using existing models so seems much cheaper.\n\n2. [Disclaimer: I'm not affiliated with the following paper] What do y’all think of Arora & Goyal et al. 2023’s A Theory for Emergence of Complex Skills in Language Models https://arxiv.org/abs/2307.15936? It seems to disagree with your Theorem 2, if I understand correctly. It'd be good to know what the relationship is between these.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Overall, I think this is a good paper. I think it is well motivated, thorough and insightful. I would be happy to increase my score if a number of modifications are made (or if the authors tell me why I'm mistaken!).", "weaknesses": "Here, I order my feedback in sequential order based on moving through the paper top to bottom.\n\n> We hypothesize that the perceived discontinuity from trivial to excellent performance might stem from limited evaluation resolution. By employing a more nuanced resolution, one could potentially uncover the scaling law for tasks. Our hypothesis diverges significantly from that of Schaeffer et al. (2023),\n\nI think this misstates Schaeffer et al. (2023). Their abstract states, “we provide evidence that alleged emergent abilities evaporate with different metrics or _with better statistics_”. The “better statistics” refers to increased resolution. Specifically, their Figure 4 shows that increasing resolution by sampling significant amounts of additional data improves predictability on the original metric i.e. Accuracy. One of Schaeffer et al. (2023) et al.'s key takeaways is that predicting capabilities with scaling can, in certain cases, demand significantly higher resolution.\n\n>  PASSUNTIL deploys extensive random sampling in the decoding phase (e.g., 105 sampling times), and evaluates each sampling result until any generation passes the target test metric. Therefore, this evaluation strategy has infinite measurement resolution as long as computational resources are not bounded.\n\nAssuming unbounded computational resources is not a trivial assumption to make, and indeed, in the experiments, the authors do not sample as dictated by their procedure. Rather, the authors set a finite upper bound on the number of samples. Consequently, I am minorly concerned with their story and with their results since the Pass Until (PU) score they discuss is **not** the PU score they numerically compute. I think they should use PU as defined or offer a correction if a limited compute budget exists.\n\n> Theorem 1.\n\nThe authors note that the failure time $f = K - r $ follows a negative binomial distribution. I have two comments.\n\n1. Wikipedia seems to suggest that the author's estimator $PU := r/\\mathbb{E}[K]$ is a *biased* estimate (https://en.wikipedia.org/wiki/Negative_binomial_distribution#Maximum_likelihood_estimation), directly contradicting Theorem 1. The citation given is Haldane Biometrika 1945 \"On a method of estimating frequencies\" and a quick Google search discovers many Math.StackExchange & Cross Validate.StackExchange posts backing up Wikipedia. Could the authors please double check? I suspect the step in their derivation of dividing by $\\mathbb{E}[K]$ may be incorrect, but I don't have time to investigate. My guess is that if the estimator is defined as $\\hat{P}(s) := \\frac{r}{K}$, then its expectation $\\mathbb{E}[\\hat{P}(s)] := \\mathbb{E}[\\frac{r}{K}] \\neq \\frac{r}{\\mathbb{E}[K]}$, as the authors claim.\n\n2. When $r$ is known, the minimum variance unbiased estimator (MVUE) for a Negative Binmoial is known to be $\\frac{r-1}{K-1}$ (where K is the total number of trials to get $r$ successes, in the authors' notation; https://en.wikipedia.org/wiki/Negative_binomial_distribution#MVUE_for_p). I am curious to know why the authors chose to not use this estimator? If the authors have a compelling reason, I would be happy to listen. If the authors do not have a compelling reason, I would strongly recommend changing to this MVUE and recomputing their results; doing so should not require resampling from the models. Defining PassUntil as this MVUE would also salvage Theorem 1.\n\n> Table 1\n\nI think this Table could be much more easily understood as a Figure. Assuming you're using Python, I recommend doing the following in Matplotlib & Seaborn: Create two axes, one for Emoji Movie & Date Understanding. Then, the x axis of both matplotlib axes should be number of instances passed (from 0 to 41). The y axis should be the 4 models, and the 4 sampling options (BS, RS-1, RS-100, RS-10000) should be the hues. \n\n> Equation 4\n\nTwo comments here:\n\n1. I believe this equation is incorrect. Equation (3) is a product from $i=1$ to $i=|y|$, and Equation 4 correctly converts that product into a sum, but the right hand side has no dependence on $|y|$. Your Equation 4 states that generating a length 1 output and generating a length $10^9$ output have the same PU scaling. Rather, I believe you should have $\\exp(-c |y| N^{-\\alpha}).\n\n2. Once corrected, your Equation 4 is identical to Schaeffer et al.’s $Accuracy(N)$ equation (Page 4) (modulo the slightly different definitions of the scaling law form). It would be good to acknowledge this.\n\n> Table 2: Prediction of our framework\n\nHere, I also recommend converting the table to a figure. Also, where are your confidence intervals or standard errors?\n\n> Figure 6: Scaling curve for task “Dates” and “Identity”. \n\nAt a granular level, where are the error bars? You should be able to repeat this experiment by bootstrapping which 20 questions are in the test set and which are the in-context examples.\n\nAt a higher level, I'm a bit skeptical about this contribution for a few meta-reasons.\n\n1. Previous emergent papers didn't emphasize this BIG-Bench task (Unnatural In-Context Learning) and to be honest, I haven't heard it mentioned as an emergent ability.\n2. The task specifies 8 subtasks (https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/unnatural_in_context_learning#subtasks) but this manuscript only considers 4 and I don't know why those 4 were chosen.\n3. The curves are fit with no held-out data for testing generalization of the fit.\n\nPut together, I'm unsure whether these specific tasks might have been cherry picked. BIG-Bench has >100 tasks, so I'd guess any pattern can be found if one looks for it. That doesn't mean the pattern is a general phenomenon; rather, by random chance alone, some curves will look concave or convex with respect to log N.\n\nLastly, by limiting the number of samples ($10^5$), I believe the PU estimator is no longer unbiased. The whole promise of PassUntil is that one can keep sampling so long as necessary. Is this correct? If so, how can we correct for the finitely many samples?\n\n> if F (N ) is a concave function of log N , then the task obeys super-scaling law growth, which is defined as “emergent”\n\nI have two thoughts here. On one hand, I feel like \"Emergence Still Exists But Becomes Continuous.\" is an oxymoron based on the definition offered by Wei et al. 2022, and that one shouldn't attempt to overwrite an established definition. But on the other hand, a better definition is a welcome invitation. Perhaps it would be better to more clearly title this section \"An Alternative Definition of Emergence\" or \"A Quantitative Definition of Emergence\" to make clear that the existing definition is being more precisely defined/refined.\n\n> Theorem 2. Suppose each reasoning step’s success rate, measured by PASSUNTIL obeys the scaling law growth, then the multi-step success rate follows the sub-scaling law growth.\n\nDisclaimer: Empirically, this seems wrong to me. Looking at Figure 7 (right), subscaling behavior is _not_ what multi-step success task curves look like.  I think the problem is the assumption: \"Suppose each reasoning step’s success rate, measured by PASSUNTIL obeys the scaling law growth.\" What I've observed is that LLMs exhibit non-trivial conditional dependencies such that if the first few tokens are correct, subsequent tokens are more likely to be correct, and vice versa. I'm not saying the math is incorrect. I'm saying that this assumption isn't a good fit to reality. This is also visible in Schaeffer et al. 2023's Figure 4: the independence assumption results in geometric decay, but LLMs don't decay this quickly.\n\n> Theorem 3. In this scenario, the success rate of the task is the majority voting of these circuits\n\nThis statement is unclear. I'm willing to buy that multiple circuits exist, but why should the decision-making be via majority vote. Is this an assumption? If so, I recommend changing the language to \"In this scenario, *if* the success rate of the task is the majority\nvoting of these circuits.\" But if not, then could the authors please clarify why decision-making is via majority vote?\n\n\n> set the sampling upper bound times to $10^5$\n\nIn my experience, this takes a significant compute budget. Could the authors please quantify how long such evaluations take in compute, time and money?\n\n## Possibly Unacknowledged Limitation\n\nSome LLM evaluation processes don't involve sampling. This includes non-generative tasks e.g., MMLU or greedy decoding aka temperature=0. In such evaluation processes, does PassUntil make sense? And if not, hopefully the authors can add a limitation section.\n\n## Suggested Alternative Title\n\nI won't take this into account for the review, but reading the title \"Unlock Predictable Scaling from Emergent Abilities,\" I don't know if \"from\" is the right preposition. I'd recommend a slightly different title like \"Predicting Emergent Abilities of LLMs with Infinite Resolution Evaluations\". I think that's more clear and also more exciting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o4RvlUQ6TA", "reviewer": "Reviewer_jwFq", "summary": "This paper introduces a novel evaluation strategy, PASSUNTIL, to detect subtle improvements in small models and quantitatively predict the performance. It also presents the task scaling law, which provides insights into the relationship between model size and performance. Additionally, the paper explores emergent abilities through a mathematical analysis of multi-step reasoning and circuit hypotheses. Experimental validation is conducted to support the theoretical analysis.", "questions": "PU vs. test loss in the task scaling law design: This work seems to give a vague definition of \"pass\" in PASSUNTIL. Can PU represent all performance metrics of tasks when required?  In the task scaling law, can this relationship between PU and test loss be applicable to all tasks? In the paper \"Are emergent abilities of large language models a mirage?\" that this work cited, when the task performance metric is changed to Token Edit Distance, the relationship with loss does not align with the formula provided in this work.\n\nThis work challenges the prevailing hypotheses for emergent ability by employing a redefined metric F(N). However, it should be considered that the choice of metrics, including the new definition of F(N), may potentially impact emergence. How does this work explain the emergence observed in previous works, especially for multi-step reasoning? Does this contradict Theorem 2?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Proposed the evaluation strategy \"PASSUNTIL\" with theoretically infinite resolution, enabling the prediction of task performance and the derivation of the task scaling law.\n\n2. Analyzed emergent abilities using a mathematical definition, challenging prevailing hypotheses and introducing an alternative circuit hypothesis based on theoretical derivations.\n\n3. Conducted experiments to validate the theoretical analysis and provided the first open-source attempt to investigate the predictability of task performance.", "weaknesses": "1.\tMotivation vs. actual design of the task scaling law:\n\nThe paper states that “Despite the predictable decrement in LLM loss, task performance improvements are twisted during scaling” and \"First, these works concentrate on training and validation loss metrics, which do not reliably predict task performance.\" This suggests that the loss metrics may not be highly dependable for predicting task performance. However, the task scaling law design still focuses on the correlation between PU and test loss. This may raise questions about its alignment with the initial motivation because the design of the task scaling law does not go beyond the scope of loss.\n\n2.\tDesign of the experiment to predict performance:\n\nIn the experiments for performance prediction, only the method designed in this paper was implemented, without conducting a comparison with other related methods previously proposed.\n\n3.\tThe analysis of emergent ability\n\nThis work challenges the prevailing hypotheses for emergent ability by employing a redefined metric F(N). However, it should be considered that the choice of metrics, including the new definition of F(N), may potentially impact emergence. How does this work explain the emergence observed in previous works, especially for multi-step reasoning? Does this contradict Theorem 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "07sydjGamM", "reviewer": "Reviewer_5dKH", "summary": "This paper proposes a new approach to \"smooth\" the emergent abilities. That is, we can sample the model outputs from the logits many times until we get the correct results. The proposed approach can unlock the predictable scaling from emergent abilities. The paper also provides some insights about why chain of thoughts and rethink the relationship between CoT and emergent ability.", "questions": "1) Can the PassUntil curve be more smooth if we select the \"top-K\" candidates before sampling? Such as setting a value for top_k in huggingface model.generate API", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) It is an important topic to make the emergent abilities predictable.\n2) The insights about the relationship between CoT and emergent ability are interesting. The authors also provide some theoretical evidence about the insights.\n3) The proposed approach is easy to implement.", "weaknesses": "1) The PathUntil seems to be very expensive in the early stage, because of the low probability of sampling the correct answer.\n2) The smoothness of PathUntil highly depends on the output length. For HumanEval it may be okay because the code is simple and short. However, it would be very hard to make it very smooth for the long answers. \n3) It would be helpful to provide a more detailed discussion between \"ppl on task data\" and \"passuntil on the task data\". I can understand these two are different, but this may be helpful to let more readers to understand the insight of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)