U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models
Authors: Tung-Yu Wu
Melody Lo
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
emergent abilities
scaling laws
Primary area: foundation or frontier models, including LLMs
TL;DR: The emergent abilities of LLMs can be decomposed into complementary non-trivial scaling trends in easy and hard samples, which sheds light on deeper understanding and prediction of emergent abilities.

Abstract: Large language models (LLMs) have been shown to exhibit *emergent abilities* in some downstream tasks, where model performance stagnates at first and then improves sharply and unpredictably with scale beyond a threshold. In this work, we investigate the phenomenon by grouping questions based on difficulty level and provide a possible explanation for emergent abilities. Specifically, we observe U-shaped scaling for hard questions and inverted-U scaling followed by steady improvement for easy questions. The two scaling patterns initially offset each other, causing stagnant overall performance. The performance starts to soar when the scaling pattern of easy questions reverts from inverse to standard scaling, leading to emergent abilities. Based on this finding, we propose a simple yet effective pipeline, called *Slice-and-Sandwich*, to predict the emergence threshold and model performance beyond the threshold. Our code is publicly available at https://github.com/tony10101105/ExpEmergence.

Published as a conference paper at ICLR 2025
U-SHAPED AND INVERTED-U SCALING BEHIND EMERGENT ABILITIES OF LARGE LANGUAGE MODELS
Tung-Ya Yu Wu, Melody Lo National Taiwan University (b08901 133, peiyulo] @ntu. ed
ABSTRACT
Large language models (LLMs) have been shown to exhibit emergent abilities in some downstream tasks, where model performance stagnates first and then improves sharply and unpredictably with scale beyond a threshold. In this work, we investigate the phenomenon by grouping questions based on difficulty level and provide possible explanation for emergent abilities. Specifically, we observe U-shaped scaling for hard questions and inverted-U scaling followed by steady improvement for easy questions. The two scaling patterns initially offset each other, causing stagnant overall performance. The performance starts to soar when the scaling pattern of easy questions reverts from inverse to standard scaling, leading to emergent abilities. Based on this finding, we propose a simple yet effective pipeline, called Slice-and-Sandviie and model performance beyond the threshold. Our code is publicly available at hittesss//gg.ttee.cccm///ooooosd http://c..c..oooooo/ooooo/////ee
predict the emergence threshold
INTRODUCTION
Large language models (LLMs) (Team et al. 2023
Achiam et al. 2023 Brown 2020 Touvron
al., 2023a b. Workshop et 2022 al. 2023 Jang et al. 2024) have shown strong potential
in various downstream applications (Jumper et al. 2021 Fawzi et al. 2022 Naveed et al. 2023
Kaddour et al., 2023). Though the training-loss scaling law has been well established (Kaplan
et al. 2020; Hoffmann et al., 2022), the literature is inconclusive regarding how performance on downstream tasks scales. In particular, for certain downstream tasks (Srivastava et al. [0033] Lin
al. 2022a Pilchvar & Camacho Camacho-Colladoo 2019). LLMs display emergent abilities: performance stagnates even when model training compute scales up hundredfold, and then improves sharply at an
unpredictable critical threshold (Wei al. 2022 Schaeffer al. 2024a).
Some prior work (Schaeffer et al. 2024a b. Lu et al. 2024) link emergent abilities to crude per- formance metrics that fail to capture small model improvements. Hu et al. (2023) introduces the PASSUNTIL metric, showing gradual model improvement with scale, SScaetfer et al. (2024a) finds that emergent abilities mainly happen string-match and multiple-choice tasks (Schaeffer al.) 2024a), for which traditional performance measures exhibit strong discontinuity. They propose continuous metrics such as Brier Score (Brier 1950) and linear metrics such as token edit distance (TED) (Schaeffer et al. 20244) to better predict LLM scaling law on downstream tasks. Schaeffeer al. (024b) further ranks several performance metrics in correlation with the model scale. On the other hand, Michaud et al. (2024) establish the quantization model of neural scaling to explain the emergent drop of cross-entropy loss from the aspect next-token prediction.
Another focus of prior literature is the predictability of emergent abilities measured in traditional metrics like accuracy, which is crucial for monitoring and forecasting I.L.s' potentially harmful task-wise capabilities, such as writing certain malicious code (Charan al. (023). Though Wei et al. catssbiataa
(2022) characterizes emergent abilities as umpredictable performance soar, some studies (Ruan et al.
2024| Gadre et al. 2024: Hu et al. 2023 Owen 2024 Ye et al. 2023) have proposed pipelines to estimate task-specific scaling law. However, they usually incorporate models past the emergence threshold into the training to fit Sigmoid function and do not predict emergence abilities.
This paper contributes to both fronts of discussion on emergent abilitics, especially for multiple-choice tasks, First, we propose novel procedure evaluate the performance of L.LMs on questions grouped

---

Published as a conference paper at ICLR 2025
0.7
Emergence
TC Brier Score
-0.3
Emergence
accuracy
-0.35
-0.4
-0.45
log compute (M)
log compute (M)
(a) Accuracy log compute (M).
(b) TC Brier Score log compute (M).
Difficulty
-0.1
Emergence
Level
003
3
TC Brier Score
-0.4
-0.5
5
6
8
-0.8
9
-0.9
-1
-0.5
0.5
1.5
2
2.5
3
log compute (M)
(c) U-Shaped and inverted-U] scaling with MMLUUs questions clustered into 10 groups. Higher levels are harder questions.
Figure 1: The accuracy, Carget-Conditioeel (TC) Brier Score, U-shaped and inverted-U scaling on the MMLU benchmark (Hendrycks et al. (202I) evaluated using 56 LLMs. Sec. 12.122 provides details on the TC Brier Score, which captures granular changes in model performance. App. A provides details of the LLMs.
by different difficulty levels. Fig. shows the evaluation result of 56 LLMs with diverse training compute on the MMLU benchmark, whose 14042 questions are clustered into 10 groups based on their difficulty levels, with higher levels denoting harder questions. Both model performance and difficulty of each question are measured and calculated by the (urget-Conditionel (TC) Brier Score (see Sec. 2.1.2 for details), which is our proposed continuous metric that highly correlated with accuracy but can capture more granular changes in model predictions. We observe that performance on hard questions exhibits U-shaped scaling (Wei et al. 2023 McKenzie al. 2023), where worsens with scale at first and then reverses to improve with scale. In contrast, performance on easy questions exhibits an inverted U-shape followed by steady improvement with scale, consistent with the previously reported deep double descent of testing loss (Nakkiran et al., (021). Moreover, the point at which performance reverts from inverse to standard scaling roughly coincides with the emergence threshold beyond which model performance begins to soar. Our observation could explain why LLM's s performance on some multiple-choice tasks stagnates for models below the emergence threshold: the scaling trend on easy questions offsets that hard questions.
This observation of U-shaped and inverted-U scaling provides a basis to predict the forthcoming sharp increase in model performance. defining feature of emergent abilities. We propose Slice-and- Sandwich pipeline, where we first group questions on given downstream task by difficulty levels, data before the emergence threshold fit the performance easy and hard questions separately,
2

---

Published as a conference paper at ICLR 2025
then forecast performance easy and hard questions separately beyond the emergence threshold. We show that Slice-and-Sand aad-Sandwich captures the performance soar well.
We summarize contributions follows:
We demonstrate that, for some downstream tasks previously shown display emergent abilities, under proper continuous metric, LLM's performance exhibits opposing scaling trends: inverted-U vs. U-shape, on easy vs. hard questions below the emergence threshold, and steadily improves beyond the emergence threshold.
Based on the observation of inverted-U vs. U-shape on easy vs. hard questions, we propose simple yet effective pipeline, Sandwwihd-S to forecast model performance past the emergence threshold. Experimental results on three iconic datasets show its effectivvness.
SCALING TREND BY DIFFICULTY LEVEL: U-SHAPE VS. INVERTED-U
This section documents LLM's scaling trend by question difficulty level. Sec. 2.1 defines terminolo- gies such as log compute, emergence threshold, and our performance metrics. Sec. 2.2 describes how we group questions by difficulty level. Sec. 2.3 presents and discusses the results of iconic multiple-choice tasks with emergent abilities.
2.1
TERMINOLOGG
2.1.1
LOG COMPUTE AND EMERGENCE THRESHOLD
For clearer visualization, in this paper, we refer to an LLM's is log compute M as:
C
M logio
1021
where x 6ND (Kaplan et al. 2020) is the total training compute (FLOPs) of an LLM, N number of model parameters, and D is the number of training tokens. The emergence threshold T is identified manually as the log compute where the model accuracy exhibits a sharp improvement, as illustrated in Fig. Ta
2.1.2 CONTINUOUS PERFORMANCE METRICS
Prior work (Schaefferr et al.) 2024a by Lu et al. 2024) has advocated for performance metrics that distinguish finer differences. One candidate metric the Brier Score (Brier 1950
K C
Brier = K
Σ (Pr. - pp.))
(2)
t=1
where the number of samples and the number of classes. Pt. is if the 1-th sample belongs to class and 0 otherwise. Pt,5 is the model's 'ss predicted probability of the t-th sample being of class i.
However, the Brier Score depends not only on the model's predicted probability of the target class (choice) but also on the predicted probability distribution of all classes. Since there is no priori reason which type of distribution on non-target classes signify better ability we propose the Target- Conditioned (TC) Brier Score that lumps all non con-target available classes into one class, conditioned on available classes, and has an opposite sign to Eq. Es. that higher score means higher performance:
K
TC_Brier
(pr.r
1)²
K
1=1
where pion is the model's predicted probability of the t-th sample being of target class conditional on available classes, i.e.,
Dr.c
soon
Σ
Dr.s
Eavailable clusses
Schaeffer al. (024b) found that poon (denoted pe has higher correlation with log compute than Brier Score does.
Choices (correct choice) in Schaeffer et al.
2024b

---

Published as a conference paper at ICLR 2025
-0.5
Emergence
TC Brier Soore
accuracy
0.35
Emergence
-0.6
0.25
-0.65
compute (M)
log compute (M)
(a) Accuracy log compute (M).
(b) TC Brier Score log compute (M).
Difficulty
Level
-0.2
Emergence
2
-0.3
TC Brier Score
4
005
0.6
6
-0.7
7
-0.8
-0.9
9
-1
-0.5
0.5
1.5
2
2.5
3
log compute (M)
(c) U-Shaped and inverted-U scaling.
Figure 2: The accuracy, TC Brier Score, U-Shaped and inverted-U scaling on the Persian-QA dataset in BIG-bench (Srivastava et al., 2023).
where the target class, Pr.e is the model's predicted probability of the t-th sample being of the target class. We discuss the effect of conditioning in App.
2.2 GROUPING QUESTIONS BY DIFFICULTY LEVELS
2.2.
MEASURING QUESTION DIFFICULTY LEVEL.
For question q a downstream task with emergence threshold we define difficulty level D, to be the question-level TC Brier score that takes as samples the outputs question q from all L LLMs smaller than the emergence threshold More specifically, we define
L
D, =
(ptee
1)²
2.2.2 QUESTION SORTING AND GROUPING
Because model performance individual questions is quite noisy, we group questions by difficulty levels. First, we sort questions by ascending difficulty level. Then, we evenly divide the sorted questions into groups. Thus, each group has different difficulty level.
4

---

Published as a conference paper at ICLR 2025
0.9
Emergence
Emergence
TC Brier Score
accuracy
0.6
-0.4
0.4
-0.6
0.2
-0.7
-0.8
log compute (M)
log compute (M)
(a) Accuracy log compute (M).
(b) TC Brier Score log compute (M).
Emergence
Difficulty
Level
2
-02
TC Brier Score
4
-0.6
6
7
-088
9
-1.2
-1
-0.5
0.5
1.5
2
2.5
3
log compute (M)
(c) U-shaped and inverted-U scaling.
Figure 3: The accuracy, TC Brier Score, U-shaped and inverted-U scaling on the arithmetic dataset in BIG-bench (Srivastava et al. 2023).
2.3 U-SHAPED AND INVERTED-U U SCALING
Fig. Ta show the scaling trend of accuracy on the MMLU, Persian-QA. and arithmetic datasets, show the scaling trend of TC Brier Score. Though Persian-QA QA has smooth scaling, MMLU and arithmetic still exhibit sharp increase past the emergence threshold T. Fig. [c show the TC Brier Score scaling trend with group number G == 10. Implementation details and model details are App. A Model performance on easier questions, such as difficulty level in Fig. Tc and Fig. 2 displays an inverted-U shape followed by steady improvement, i... performance first increases and then worsens with scale, followed by second ascent, aligning with the previously reported deep double descen on testing loss (Nakkiran et al. 2021). Moreover, the reversion from inverse scaling to standard scaling roughly coincides with 7. In contrast, performance
with clear ability emergence demonstrated. Fig. Tb 3b
hard questions, such difficulty level 10 in Fig. 22 and Fig, [3 displays a U-shaped scaling trend (Wei et al.) 2023] McKenzie et al. (023): model performance decreases with scale in early stage and increases with scale when M gets larger. Besides MMLU, arithmetic, and Persian-QA. Fig. 4 shows U-shaped vs. inverted-U scaling of Hindu knowledge, conceptual combinations, and analogical similarity datasets in Big- Big-Bench (Srivastava al. 2023).
totaling 6 datasets, with G 3. Detailed results of the 3 datasets are in App.
TThe term "descent" in the original paper refers to the testing loss. Hence, for the sense of accuracy Brier Score, is "ascent" of performance.

---

Published as a conference paper at ICLR 2025
(a) MMLU.
(b) Arithmetic.
(c) Persian- QA.
porputa
(d) Hindu knowledge.
(e) Conceptual combinations.
(I) Analogical similarity.
Figure 4: U-shaped and inverted-U U scaling on datasets exhibiting emergent abilities, with group number G = 3. Except for MMLU, the other 5 tasks are from Big-Benne big-Bench. Different levels of U-shaped and inverted-U scaling trends are demonstrated across the tasks.
Emergence
Threshold
Classical
Modem
Regime
Regime
Randam Guess
TC Brier Score
prediction loss
a Resurran
log compute (M)
log compute (M)
log compute (M)
(a) Deep double descent.
(b) U-shaped scaling.
Figure 5: Illustration of deep double descent (Nakkiran al. 2021) easy question groups and U-shaped scaling (Wei et al. 2023) on hard question groups under the TC Brier Score.
Overall, the scaling trend of question group transitions from that of the easiest group (inverted- U followed by steady ascent) to that of the hardest group (U-shape). As the initial scaling trends of easy questions and hard questions roughly offset each other when aggregated across all difficulty levels, performance stagnates until the scaling trend on casy questions reverts from inverse to standard scaling, followed by sharp improvement when performance on easy and hard questions both improve with scale. This could explain the emergent ability phenomenon reported in previous literature (Wei) al, 2022 Schaeffer et al., [2224a] Hu et al.) (2023). More results of scaling trend on non-emergent tasks in App. D and U-shaped vs. inverted-U scaling measured by accuracy are in App.
3 POSSIBLE EXPLANATION FOR U-SHAPED AND INVERTED-U U SCALING
We provide possible explanation for the initially opposing scaling trends (inverted-U U vs. U-shaped) on easy vs. hard questions using the AI community' previous findings (Nakkiran al. 2021 Wei et al. 2023 McKenzie et al. 2023) in deep neural network and LLM behaviors.
3.1 SCALING TREND OF EASY QUESTION GROUPS
As discussed in Sec. for downstream task with emergent abilities, model performance on easy questions displays inverted-U U shape followed by steady improvement. Fig. [5a illustrates the scaling trend flipping the sign on the TC Brier Score so that a higher number means a higher prediction loss. The pattern is consistent with the phenomenon deep double descent identified in

---

Published as a conference paper at ICLR 2025
Table 1: Examples of an easy and hard question in the MMLU benchmark. The Avg. Prob. is the average output probabilities before re re-distribution over all models with log compute M < 1.5. Answer choices (classes) are underlined. In the hard question, small models overlook the negation "doesn't", giving choice C high confidence, yet correct choice D low confidence.
Question Description
Difficulty Level
Choices
Avg. Prob.
A. hot to cold ever
B. cold to hot ever
A. 0.24
(conceptual physics, id 44)
level 10 (hardest group)
C. hot to cold
B. 0.29
The second law of thermodynamics tells us that heat doesn't 't flow from
without external energy
C. 0.29
D. cold to hot without external energy
D. 0.18
A. 90%
A. 0.44
(global facts, id = 66) In 1935 roughly how many Americans were in favor of Social Security act?
level (easiest group)
B. 70%
B. 0,30
C. 50%
C. 0.17
D. 30%
D. 0.09
Nakkiran et al. (021). In the context of testing error scaling law, Nakkiran et al. (2021) argues that initially, the bias- trade-off classical statistical learning theory (Hastie et al. 2009) applies, which forms the "classical regime": complex models suffer from "overfitting" and thus, once complexity exceeds certain threshold, models become over-sensitive to sample noises, and the effect from such larger variance dominates the effect of further reducing testing error. On the other hand, once the model large enough (the "modern regime") "), a further increase in complexity allows the model pick from more and more interpolating models that all fit the dataset, thus improving performance and reducing testing error to near zero.
3.2 SCALING TREND ON HARD QUESTION GROUP
McKenzie et al. 22023 Wei et al. (0233 have identified the U-shaped scaling of LLM in some
downstream tasks, as ill illustrated in Fig. 56 Wei et (2023) provides potential explanation for the initial inverse scaling: the task might contain a "distractor task" that attracts models to learn to solve first; thus, larger models perform worse. An example is the NeQA task (McKenzie et al.) 2023) which negates each multiple-choice question in the OpenBookQA dataset (Mihaylov et al. 2018) to examine whether models would be misled by the negation. It turns out that model performance first declines from random guesses due to the attempt to answer the non-negation part of the question. Table shows such a question in our hard question group in the MMLU. For the question "The second law of thermodynamies tells us that heat doesn't 't flow from", small models (M < 1.5) average assign high confidence to choice C and lowest confidence to correct choice D, where the former is the answer if removing negation "doesn't" from the original question. More MMLU questions to demonstrate U-shaped and inverted-U scaling trends w.r.t. model log compute are in App. B
SLICE-AND-SANDWICH
4.1 PROBLEM FORMULATION
We aim to predict the performance soar of traditional metrics before it happens. Specifically, we only use models before the emergent threshold T to forecast the incidence of emergent abilities and the scaling trend after T. We compare the performance of our pipeline with the current iconic baseline of the Sigmoid bigmoid-based task-specific scaling law (Ye et al. 2023), which uses the Sigmoid function to regress accuracy w.r.t. log compute M.

---

Published as a conference paper at ICLR 2025
Emergence Threshold
2
Fitted
Extrapolated
TC Brier Score
TC Brier Score
Question
Difiiculty-bssed Grouping
log compute (M)
Sandwich the Performance
log compute (M)
3
4
Project Back Accuracy
TC
log compute (M)
log compute (M)
Figure 6: The overall pipeline of Slice-and-Sandwichh We group questions into different difficulty levels, fit each group's scaling trend, sandwich the overall performance to construct the scaling law on the linear metric, and finally project the scaling law back the traditional metric.
4.2 PIPELINE OVERVIEW
Fig. shows the overall pipeline of Sirte-and-Sandwicee We models smaller than the emergence threshold T as the training set. As performance no longer stagnates with scale once we group questions by difficulty level, fit the scaling trend of a continuous metric (TC Brier score in this paper) on the easiest question group and hardest question group separately and use the fitted scaling trend to forecast performance (measured in TC Brier Score) on easy and hard questions past T. We also use the training set to regress accuracy on the TC Brier Score and then use this estimated relation convert the predicted TC Brier Score into predicted accuracy for models past the T.
4.3
PREDICTING EMERGENT ABILITY
4.3.1
QUESTION GROUPING
To reduce data noise, group questions into G = 3 difficulty levels, in Fig. 4 for Slice-and- Sandwich and denote the level 1, and as easy, medium, and hard question groups The medium group's pattern is close aggregating the scaling trend between easier and harder groups.
4.3.2 FITTING AND FORECASTING SCALING TREND OF EASY VS. HARD QUESTIONS
We use simple polynomial regression to fit the scaling trend of the TC Brier Score of the easy and hard question groups separately using models before T. We denote by (x) and (x) the fitted scaling trend of the easy and hard question groups, respectively, where is the log compute. We then
use (x) and (x) to forecast performance (measured TC Brier Score) on the easy and the hard question groups of models with log compute above T.
use the average of performance on the easy group and the hard group to forecast aggregated performance measured in TC Brier Score:
1 F"(r) / (F) + (R))),
(5)
as aggregate performance is sandwiched between performances in the easy and hard groups.
4.3.3 OBTAINING SCALING TREND IN TRADITIONAL METRIC
Since people usually care about intuitive traditional metrics such as accuracy (Hu et al. 2023), our last step project the fitted scaling trend in TC Brier Score, F((r), back to scaling trend in accuracy,
8

---

Published as a conference paper at ICLR 2025
BAD Srantig
-0.2
IE0y promia ngraaice
poynorta regeatio
1C Unterr Score
TCC Drler Score
turt engrrrma vestins
500 spl 0.3
Brier
lurd
-08
25
25
1. compute
-30 255
log compute
log compute
(a) MMLU.
(b) Arithmetic.
(c) Persian QA.
Figure Data and polynomial fit for the easy and hard question groups. The fitted trends encapsulate the actual trends.
0.2
ledirg solt
leding coll
beting
-02
-32
TC Drier Score
-43
compute
compute (M)
log
(M)
(a) MMLU.
Arithmetic.
Persian-
Figure 8: The TC-Brier-Score Sefeer-Score-ble based scaling law acquired by taking the average of fitted trends of easy and hard question groups Fig.
denoted by (xx). One can replace TC Brier Score with other continuous metrics and accuracy with other traditional metrics. Specifically, we first estimate the relation between the continuous metric (TC Brier Score) and the traditional metric (accuracy) using models with log computes smaller than as the training set. We denote the estimated mapping from TC Brier Score to accuracy as G(-). Our forecast of scaling trend of accuracy is given by:
G(F)))(F(()))- +C,
(6)
where is constant such that the average predicted accuracy of Fxx) on the training set the same as the average true accuracy of all models in the training set.
EXPERIMENTS
5.1 FITTING SCALING TREND OF EASY GROUP AND HARD GROUP
We adopt polynomial degree=2 and 5 for hard and easy questions, respectively, in response to our observation of U-shaped vs. inverted-U scaling. This parameter selection based on our prior belief of polynomial regression's 's fitting powers to fit the deep double descent and U-shaped scaling. Experimental results on parameter robustness are in App. G
Fig. 7 shows the fitted scaling trend of the easy and hard question groups on the MMLU, arithmetic, and Persian-QA datasets. Empirically, fitted trends on hard questions are either precise or underesti- mated, e.g., hard group of MMLU (Fig. 7ab and arithmetic (Fig. 7b). due to lower fitting power of degree 2: fitted trends on easy questions are precise or overestimated, e.g., easy group of MMLU (Fig. 7a), due to more considerable fitting power of degree 5. However, they still encapsulate the overall trend. Therefore, as shown in Fig. 3 taking their average can decrease the deviation and still lead to precise prediction of the actual scaling trend.

---

Published as a conference paper at ICLR 2025
PuPiipai 06
Ue Score
-33
Score
Driierr
-16
644
6.5
26
02
02
0.6
05
accuraty
accuracy
accuracy
(a) MMLU.
(b) Arithmetic.
(c) Persian-QAl
Figure 9: The relation between accuracy and the TC Brier Score. The mapping function G(x)) from the TC Brier Score to accuracy can be well-modeled using small models as the training set.
Trailing 18-8
0.00
laazing spll:
soudine
1:48
socuracy
sccuracy
6.40
25
(M)
log
(a) MMLU.
(b) Arithmetic.
(c) Persian-QA.
Figure 10: The accuracy-based scaling law acquired by projecting TC-Brier-Score-be scaling law back to accuracy-bass -base scaling law by G(x). Baseline is Sigmoid-based regression (Owen 2024
5.2 RELATION BETWEEN ACCURACY AND TARGET CONDEI-CONDTIOONE (TC) BRIER SCORE
Fig. 9 shows the close and almost linear relation between accuracy and TC Brier Score. As result, simple ordinary least squares (OLS) regression using only models before the emergence threshold yields precise mapping G--) from TC Brier Score to accuracy.
5.3 FORECASTING SCALING TREND IN ACCURACY
Fig. 10 shows the accuracy-based scaling trend, F (xx), obtained by Eq. with G(.). together with the baseline of fitting the accuracy the Sigmoid function (Owen 2024 Ruan et al. 2024). Compared with the baseline, Srice-and-Sandvitt better predicts and estimates the soaring performance by baking more priors of the observed U-shaped and inverted-U scaling. For the MMLU, our approach captures the forthcoming soaring trend, whereas the baseline approach does not. For the arithmetic dataset, though the baseline provides seemingly decent forecast, does not capture the acceleration of performance increase, whereas our approach does. In short, our Slice-and-Sandvica approach more explainable and capable of capturing the soaring trends of emergent abilities. simple alternative method of and-SandwiSch and its experimental results are in App. G
CONCLUSIONS AND LIMITATIONS
This work analyzes the L.M task-specific scaling law by grouping questions according to difficulty level. For six multiple-choice tasks with emergent abilitics, we demonstrate U-shaped scaling for hard questions and inverted-U scaling followed by steady improvement for easy questions. These findings provide insight into the potential causes of emergent abilities. We then introduce the Slice- and-Sandwich pipeline to predict the emergence threshold and scaling law thereafter. However, as emergent phenomena are common across LLM benchmarks, might be hard to claim all of inverted-U scaling. Furthermore, our focus on multiple-choice tasks; applying our method to arring-machhin tasks requires identifying a continuous metric that differentiates easy questions from hard questions and correlates with the interested traditional metric. We illustrate and discuss this in App. H highlighting future research direction.
them show clear U-shaped
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint wrXiv:2303.08774. 2023.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojo- caru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311. 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv: 2tt.:i::399.666000 16609, 2023.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley. Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: suite for analyzing large language models across training and scaling. In Proceedings of the International Conference Machine Learning (ICML), 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi
1ttp100.5281/zeodda
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
PV Charan, Hrushikesh Chunduri, P Mohan Anand, and Sandeep K Shukla. From text to mitre techniques: Exploring the malicious use of large language models for generating cyber attack payloads. arXiv preprint arXiv:2305. 15336, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0545: 2018.
Together Computer. Redpajama: an open dataset for training large language models, October 2023.
URL
a ama-
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu. Y Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.0m 06066, 2024.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Comera-Paredes. Moham- madamin Barekatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwieser, Grzegorz Swirszez, et Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610(7930):47-533 2022.
Samir Yitzhak Gadre, Georgios Smymis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint urXiv:2403.08544 2024.
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. framework for few-shot language model evaluation, 07 2024. UtRlhhtts:///eeeee
www.rrr/rrrrrrrreeeeeeer 622688602
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https: 7/gi wtw.t.h..coooommmmm open www.mmeeeeeeeeeeeeeeeeeeeee
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre. An empirical analysis of compute-optima large language model training. In Proceedings of Advances in neural information processing systems (NeurIPS), volume 35, 2022.
Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, et al. Predicting emergent abilities with infinite resolution evaluation. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401 040088 2024.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. nature, 596(7873) 2021.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint Xiv:2207.11016 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (XXi::0001.0866/ 2020.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint #XXv:2309.00463 2023.
Stephanic Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Mcasuring how models mimic human falschoods. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2022a.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 901999052, 2022b.
Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang. Suqi Sun, Omkar Pangarkar, et al. Llm360: Towards fully transparent open-source 1ms. arXiv preprini arXiv: 22....32..355535 2023.
Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
Ian R. McKenzie, Alexander Lyzhov, Michael Martin Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Xudong Shen, Joe Cavanagh, Andrew George Gritsevskiy, Derik Kauffman, Aaron T. Kirtland, Zhengping Zhou, Yuhui Zhang, Sicong Huang, Daniel Wurgaft, Max Weiss, Alexis Ross, Gabriel Recchia, Alisa Liu, Jiacheng Liu, Tom Tseng, Tomasz Korbak, Najoung Kim, Samuel R. Bowman, and Ethan Perez. Inverse scaling: When bigger isn't better. Transactions on Machine Learning Research (TMLR), 2023. ISSN 2835-8856.
Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Proceedings of Advances in neural information processing systems (NeurIPS), 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment (JSTAT), 2021 2021.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv preprint urXiv:2307.06435, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
David Owen. How predictable is language model benchmark performance? arXiv preprint arXiv:2401.04757, 2024.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405. 10938, 2024.
Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Proceedings of Advances in neural information processing systems (NeurIPS), 36, 2024a.
Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained clusive? arXiv preprint arXiv:2406.044 04339,, 2024b.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Johan Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai. Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartlomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argucta, Cesar Ferri, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Christopher Waites, Christian Voigt. Christopher D Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, C. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens. Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Dylan Schrader, Ekaaerina Shutova, Ekin Dogus Cubuk, Elad
13

---

Published as a conference paper at ICLR 2025
Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodolå, Emma Lam, Eric Chu, Eric Tang. Erkut Erdem, Ernie Chang, Ethan A Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Xinyue Wang, Gonzalo Jaimovitch-Lopez. Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannanch Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Francis Anthony Shevlin, Hinrich Schuetze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac. James B Simon, James Koppel, James Zheng, James Zou, Jan Kocon, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang. Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi. Jiacheng Xu, Jiaming Song. Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan. Katerina Ignatyeva, Katja Markert, Kaustubh Dhole, Kevin Gimpel, Kevin Omondi, Kory Wallace Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, McDonell, Kyle Richardson, Laria Reynolds, Leo Gao. Li Zhang, Liam Dugan, Liamhui Qin, Louis-Philipee Morency, Luca Moschella. Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Uliveros-Colón Luke Metz. Lütfi Kerem Senel, Maarten Bosma, Maarten Sap, Maartje Ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Gaairez--uintann Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael Andrew Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Andrew Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roherts, Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormllabashi, Peiyuan Liao, Percy Liang, Peter W Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang. Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymackers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan Le Bras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Russ Salakhutdinov, Ryan Andrew Chi, Seungjae Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel Stern Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmano, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima Shammic Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo- Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven Piantadosi, Stuart Shieher, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsunori Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Venkatesh Ramasesh, vinay uday prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders. William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao. Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research (TMLR), 2023.
Lidia
Evans,
14

---

Published as a conference paper at ICLR 2025
ISSN 2835-8856.
Stability-AI. Stablelm: Stability ai language models, April 2023. URL httpttts///gthuu..om
Stability Stability-A/Stableeii
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean--aptisst Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
MosaicML NLP Team. Introducing mpt-7b: A standard for open-source, commercially usable Ilms, May 2023. URL
cml wwt.boooo///cccceceeee mpt 7b
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXi:: 09288888, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research (TMLR), 2022. ISSN 2835-8856.
Jason Wei, Najoung Kim, Yi Tay, and Quoc Le. Inverse scaling can become U-shaped. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.051000 2022.
Qinyuan Ye, Harvey Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? case study on BIG-bench. Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangeheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403. 240.:2403.0465,, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. aixiv preprint arXiv:2205.00 01060, 2022.
15

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
Implementation Details
17
A.1 LLM Evaluation
17
A.2 Slice-and-Sandwin
17
More Discussions on Brier Score
18
Scaling Trend by Question Difficulty Level for Other Emergent Tasks
20
D Scaling Trend by Question Difficulty Level for Non-emergent Tasks
23
More MMLU Sample Questions Exhibiting U-Shaped and Inverted-U Scaling
26
Scaling Trend by Question Difficulty Level on Accuracy
28
More Discussions on Bliceandd-Sandwid
29
G.1 Robustness Analysis
29
G.2 Hard Lift A Simple Alternative Pipeline
30
Preliminary Analysis for String-Match Tasks
32
Broader Impact
33
Potential Positive Impacts
33
Potential Negative Impacts
33
16

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
A.1 LLM EVALUATION
We evaluate all datasets in this paper on the LM Evaluation Harness (Gao et al.
2024 platform.
We adopt 56 models, including Gemma (Team et al., 2024), Llama (Touvron et al 2023a). Llama-
(Touvron et al. (0236). RedPajama- INCITE (Computer, (002). Yi (Young al.)
(0024), Sta-
bleLM (Stability-AA 2023), MPT (Team] 2023), Falcon (Almazrouei et al. 2023), Pythia (Biderman)
et al. 2023), AMBER (Liu al. 2023), Qwen (Bai al., 2023), Qwen-1.5 (Bai et al. 2023
BLOOM (Workshop al. 2022). DeepSeekMoE (Dai al. 2024). OPT (Zhang et al. 2022).
GPT-Nco (Blcck et al. 2021). Codegen (Nijkamp et al., 2023). XGLM (Lin et al. 2022b), and
OpenLLaMA (Geng & Liu (023) families under FPI6 precision. The evaluation time of each task varies from several hours to several days on 2 NVIDIA RTX A6000, depending on the question numbers and formats. We obtain each model's log compute through the released data by RRaan et al.) (2024). We use T = 1.5, 1.8, and 2.3 as the emergence threshold for the MMLU, arithmetic, and Persian-QA dataset, respectively. We calculate question difficulty level qa using models smaller than these thresholds. We adopt 5-shot inference on the MMLU benchmark, 1-shot inference on the ARC and HellaSwag dataset, and 2-shot inference on Persian-QA: -A, arithmetic, Hindu knowledge, conceptual combinations, analogical similarity, and abstract narrative understanding datasets (ARC, HellaSwag, and abstract narrative understanding datasets are used in App. D).
A.2 SLICE-AND-SANDVER
In the main paper, examine Sfice-an-SSandiice MMLU, arithmetic, and Persian- QA datasets with group number = 3. Models smaller than 7 1.5, 1.8, and 2.3 in the MMLU, arithmetic, and Persian-QA datasets are the training set; other larger models the testing set. We adopt polynomial regression to fit easy and hard question groups. Specifically, we adopt the polynomial order=5 and for the easy and hard question groups, respectively.
17

---

Published as a conference paper at ICLR 2025
Un un-reedsttbburree TC Biter Score
Emergence
unveddettribute
-14
unveddettibuue
log compute (M)
log compute (M)
log compute (M)
MMLU.
Arithmetic.
(c) Persian-QA.
Figure All: The un-conditionalized TC Brier Score vs. log compute (M) the MMLU, arithmetic,
and Persian-Qe -A datasets.
taring aolk
tardngg
taring a
TC Brer Score
TC Brier Score
-05
-0.6
10.
64
08
0.1
04
accuracy
accuracy
accuracy
(a) MMLU.
(b) Arithmetic.
(c) Persian-QU -A
Figure A12: The relation between accuracy and un-conditionalized TC Brier Score on the MMLU, arithmetic, and Persian-QA datasets.
B MORE DISCUSSIONS ON BRIER SCORE
In the main paper, we use the model's predicted probability of the correct class conditional on all classes calculate the TC Brier Score (see Eq. 4). This section diseusses the effect of such conditionalization. This section refers to the un-condilionalized TC Brier Score as the one without re-distributing output probabilities to all classes.
Fig. All shows the relationship between mn-conditionalized TC Brier Score and log compute M on all three datasets. For the MMLU dataset, model performance still exhibits flat scaling before the emergence threshold and sharp improvement past the emergence threshold. For the arithmetic and Persian- QA datasets, the scaling trend does show a sharp increase and is easier to forecast performance under the un-com coniiiiiiionl TC Brier Score past the emergence threshold. This is consistent with the finding of (Schaeffer al., 2024bb that the un-conditionalized measure more correlated with the training compute than conditionalized ones. However, Fig. A12 shows that the um-conditionalized TC Brier Score is not as closely related to accuracy as the normal TC Brier Score for the arithmetic and especially the Persian- -Q dataset. Table A2 corroborates this assertion by showing the correlation coefficient between accuracy and the coamal/un-conditinaall Score.
TC Brier
Table A2: Comparison of correlation coefficients between accuracy and TC Brier Score with and without conditionality on the MMLU, arithmetic, and Persian-QA datasets. "P.", "S.", and "K." stands for Pearson, Spearman, and Kendall, respectively. The TC Brier Score with conditionality. i.e., the one we adopt in the main paper, has a consistently stronger correlation with accuracy.
MMLU
ARITHMETIC
PERSIAN-QU QA
CORRELATION COEFFICIENT
P.
S.
K.
P.
S.
K.
UN-CONDITIONALIZED TC BRIER SCORE
0.96
0.87
0.73
0.93
0.93
0.79
0.60
0.52
0.37
TC BRIER SCORE
0.99
0.91
0.79
1.00
0.97
0.88
0.88
0.68
0.52
18

---

Published as a conference paper at ICLR 2025
(a) MMLU.
(b) Arithmetic.
(c) Persian- QA.
Figure A13: The U-shaped and inverted-U scaling with questions grouped and performances measured by the un-conditionalized TC Brier Score on the MMLU, arithmetic, and Persian -Q daatasets.
Fig. A13 shows the scaling trend by difficulty level for the un-conditionalized TC Brier Score. For the arithmetic and Persian-QA datasets, we no longer see inverse scaling on any intervals of log compute M. In fact, for both the arithmetic and the Persian- -A datasets, performance hovers around -1 on the hardest group, corresponding the near-zer zero predicted probability of the correct class. For hard questions, the model's predicted probability on all classes is close to zero. Therefore, without conditionalizing
all classes, cannot differentiate between initial random guess and the distracted phase at larger model log computes where the model places a higher probability on an available incorrect class relative to the correct class, which yields the U-shaped scaling normal TC Brier Score as discussed in the main paper.
19

---

Published as a conference paper at ICLR 2025
C SCALING TREND BY QUESTION DIFFICULTY LEVEL FOR OTHER EMERGENT TASKS
This section demonstrates U-shaped and inverted-U scaling on three more tasks with emergent abilities, besides the MMLU, arithmetic, and Persian-QA datasets the main paper. In particular, we present the results on the Hindu knowledge dataset in Fig. Al4 conceptual combinations dataset in Fig. A15 and analogical similarity dataset in Fig. A17 These datasets are all in BIG-bench (Srivastava al. 2023).
In particular, the Hindu knowledge and conceptual combinations datasets display the U-shaped scaling for easy question groups and inverted-U scaling hard question groups. The analogical similarity dataset also shows U-shaped scaling, albeit very mild, for hard question groups, and inverted-U U scaling for easy question groups. However, scaling on the easiest question group does not revert before the emergence threshold. Scaling the second casiest and the third easiest group reverts from inverse scaling to standard scaling way before the emergence threshold. The overall scaling trend for accuracy (see Fig. Al7a) actually declines slightly with scale. However, Fig, A16 shows that, though a bit overestimated, we can still predict the forthcoming of emergent abilities using Slice-and- Blice-ad-Sandwwih whereas the Sigmoid- based regression yields a flat line.
-0.1
0.8
-0.2
Emergence
0.7
TC Brier Score
accuracy
-0.3
Emergence
-0.4
0.4
-0.5
0.3
-0.6
-1
-0.5
0.5
1.5
2.5
log compute (M)
log compute (M)
(a) Accuracy vs. log compute (M).
(b) TC Brier Score vs. log compute (M).
Difficulty
Level
-0.2
Emergence
3
Brier Score
-0.4
5
-0.6
6
-0.8
8
9
-0.5
0.5
1.5
2
2.5
log compute (M)
(c) U-Shaped inverted-U scaling
Figure A14: The accuracy, TC Brier Score, U-Shaped and inverted-U scaling on the Hindu knowledge dataset BIG-bench (Srivastava et al. 2023).
20

---

Published as a conference paper at ICLR 2025
0.8
-0.2
0.7
Brier Score
-0.3
Emergence
Emergence
accuracy
-0.4
00.5
-0.6
-0.7
-0.5
0.5
1.5
2.5
log compute (M)
log compute (M)
(a) Accuracy log compute (M).
(b) TC Brier Score vs. log compute (M).
0.2
Difficulty
Level
Emergence
3
TC Brier Score
4
-0.4
5
-0.6
-0.8
8
9
-1.2
-1
-0.5
0.5
1.5
2.5
log compute (M)
(c) U-Shaped and inverted-U scaling
Figure A15: The accuracy, TC Brier Score, U-Shaped and inverted-U scaling the conceptual combinations dataset BiG-bench (Srivastava et al. 2023).
1.0
training split
testing split
0.8
ours
baseline
accuracy
0.6
0.4
0.2
0.0
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
log compute (M)
Figure A16: The accuracy-based scaling law on the analogical similarity dataset in BIG-bench Sri-
vastava et al. 2023).
21

---

Published as a conference paper at ICLR 2025
Brier Score
-0.65
accuracy
Emergence
0.25
Emergence
-0.75
-0.8
-0.85
log compute (M)
log compute (M)
(a) Accuracy vs. log compute (M).
(b) TC Brier Score vs. log compute (M).
Difficulty
Level
-
-0.2
3
Brier Score
-0.4
Emergence
6
7
-0.8
8
-1
-0.5
0.5
1.5
2.5
log compute (M)
(c) U-Shaped and inverted-U scaling
Figure A17: The accuracy. TC Brier Score, U-Shaped and inverted-U scaling on the analogical similarity dataset BIG-bench (Srivastava et al. 2023).
22

---

Published as a conference paper at ICLR 2025
D SCALING TREND BY QUESTION DIFFICULTY LEVEL FOR NON-EMERGENT TASKS
We apply the same procedure as in Sec. several multiple-choice tasks without emergent abilities, i.e, tasks for which performance improves consistently with scale. We present the results on the abstract narrative understanding dataset in Big-bench in Fig. A18 ARC dataset (Clark al. 2018) in Fig. A19 and HellaSwag dataset (Zellers et al.) 2019) in Fig. A20 Interestingly, we do not observe the U-shaped and inverted-U scaling as the MMLU, arithmetic, and Persian- QA datasets. Performance in most groups improves consistently with scale, while the performance of the hardest question group and the easiest question group for ARC and HellaSwag datasets display flat scaling. These trends could be ascribed to question types and properties that enable models to gradually master and constantly remember, contrary to questions in emergent tasks.
0.45
0.4
-0.0
No Emergence
Emergence
TC Brier Score
accuracy
0.25
-0.75
0.15
-0.9
log compute (M)
log compute (M)
Accuracy log compute (M).
(b) TC Brier Score log compute (M).
Difficulty
No Emergence
Level
-0.2
TC Brier Score
-0.4
5
-0.6
6
-0.8
9
-1.2
10
log compute (M)
(c) Scaling trend.
Figure A18: The accuracy, TC Brier Score, and scaling trend the abstract narrative understanding dataset in BIG-bench (Srivastava et al. 2023).
23

---

Published as a conference paper at ICLR 2025
No Emergence
Emergence
TC Brier Score
accuracy
0.6
0.5
0.4
-0.55
-0.6
0.3
lag compute (M)
log compute (M)
(a) Accuracy vs. log compute (M).
(b) TC Brier Score vs. log compute (M).
Difficulty
Level
-2
-0.
3
TC Brier Score
-0.4
No Emergencel
5
6
-066
7
-0.8
8
-1
-0.5
0.5
1.5
2.5
log compute (M) (c) Scaling trend.
Figure A19: The accuracy, TC Brier Score, and scaling trend on the ARC dataset CClark et al. 2018
24

---

Published as a conference paper at ICLR 2025
-0.35
No Emergence
No Emergence
-0.4
0.25
Brier Score
accuracy
-0.55
0.4
-0.6
0.35
-0.65
-0.7
log compute (M)
log compute (M)
(a) Accuracy vs. log compute (M).
(b) TC Brier Score vs. log compute (M).
Difficulty
Level
0.2
-
3
TC Brier Score
-0.2
-0.4
No Emergence
5
-066
6
-0.8
7
-1
8
-1.2
-1
-0.5
0.5
1.5
2
2.5
log compute (M) (c) Scaling trend.
Figure A20: The accuracy, TC Brier Score, and scaling trend on the HellaSwag dataset Zellers et al. 2019).
25

---

Published as a conference paper at ICLR 2025
Table A3: Six questions in the easiest question group of the MMLU, with the correct choices underlined and each question's 's TC Brier Scores by Pythia-70m (M = -0.90), Pythia-160mg 160m (M -0.54), and Pythia-Ib (M 0.40) displayed. Pythia of sizes are pre-trained on the same data and in the same order (Biderman et al. (023), so the model parameter number is the only variable that affects log compute. The six questions demonstrate inverted-U scaling: Pythia-1600 160m performs better than Pythia-70m, while Pythia- lb performs worse than Pythia-1601m
Pythia -70m
Pythia -160m
Pythia -lb
Description
Choices
(high school mathematics, id = 142) Simplify the following expression: (9x² + 7x 4xx) +x +2 2x + 3xd 5x + 8). Express your answer polynomial with the degrees of the terms in decreasing order.
A. a + 2x + 2x
B. B. 6r
+ 6x 3x 5x + 8
A. 0.07
A. 0.06
A. 0.18
C. 10x 7x
+ 6z
B. 0.20
B. 0,06
B. 0.28
+5x +
C. 0.53
C. 0.16
C. 0.28
D. 21
+ 10x 7x 6x² + 3r
D. 0.20
D. 0.72
D. 0.26
+5r +8
A. Ascending colon
A. 0.04
A. 0.16
A. 0.18
(anatomy, id = 120)
B. Cecum
B. 0.32
B. 0.26
B. 0.26
Which of the following structures is part of the small intestine?
C. Ileum
C. 0.32
C. 0.43
C. 0.27
D. Sigmoid colon
D. 0.32
D. 0.16
D. 0.29
(high school physics, id 124)
Which of the following conditions are static
necessary for an object to equilibrium? I. The vector sum of all torques on the object must equal zero. II. The vector sum of all forces the object must equal zero. III. The sum the object's potential and kinetic ener- gies must be zero.
A: I only
A. 0.08
0.
A. 0.15
B. only
B. 0.61
B.
B. 0.23
C. III only
C. 0.08
C. 0.19
C. 0.28
I and Il only
D. 0.22
D. 0.51
D. 0.34
A: an increase in government spending
(high school macroeconomics, id= 104) In order to reduce eliminate crowding out expansionary fiscal policy can be accompanied by
A. 0.06
A. 0.14
A. 0.29
0.06
B. 0.09
B. 0.32
B. decrease in investment
C. 0.44
C. 0.63
C. 0.24
C. expansionary monetary policy
D. 0.44
D. 0.14
D. 0.16
D. contractionary monetary policy
A. marzipan
A. 0.37
A. 0.16
A. 0.15
(miscellaneous, id = 559) Which of these is made from cacao seeds?
B. soy sauce
B. 0.13
B. 0.10
B. 0.32
C. chocolate
C. 0.37
C. 0.71
C. 0.32
D. anchovies
D. 0.13
D. 0.04
D. 0.22
(professional accounting, id = 65) project should be accepted if the present value of cash flows from the project is
A. Equal to the initial investment.
A. 0.37
A. 0.26
A. 0.22
B. Less than the initial investment.
B. 0.37
B. 0.26
B. 0.29
C. Greater than the initial investment.
0.13
C. 0.43
C. 0.34
D. Equal to zero.
0.1.13
D. 0.06
D. 0.15
E MORE MMLU SAMPLE QUESTIONS EXHIBITING U-SHAPED AND INVERTED-U SCALING
This section provides more qualitative results of MMLU questions to further illustrate the U-shaped and inverted-U scaling. Table A3 presents six questions the easiest question group (difficulty level I) and their performance on four LLMs measured by TC Brier Score. In particular, Pythia-70m, Pythia-I 160m, Pythia- -b, and Qwen1 1-5-1b. ordered from small to large log compute. All Pythia models pre- pre-trained the model parameter size will affect Pythia's 's log compute. As shown in Table A3 these questions demonstrate apparent inverted-U scaling: Pythia- 160h obtains higher TC Brier Score compared
adopt
the same corpus (Biderman et al. 2023). As result, only
26

---

Published as a conference paper at ICLR 2025
Table A4: Six questions in the hardest question group of the MMLU, with the correct choices underlined and each question's TC Brier Scores by Pythia-70m (M = -0.90,, Pythia- (M -0.54), and Pythia-Ib (M 0.40) displayed. The six questions demonstrate U-shaped scaling: Pythia-160m performs worse than Pythia-70m, while Pythia-lb lb performs better than Pythia-160m 160m.
Pythia -70m
Pythia -160m
Pythia -1b
Description
Choices
(moral scenarios, id 220)
For which of these two scenarios does the main character (who uses l/me/my)
A. Wrong, Wrong
A. 0.17
A. 0.17
A. 0.18
do something clearly morally wrong, ac- cording ordinary moral standards the US of 2020? Scenario As my parents slept, I played Tic-tac-toe. Sce- nario 2 I slid down the slide with no one on it.
B. Wrong, Not wrong
B. 0.17
B. 0.29
B. 0.22
C. Not wrong, Wrong
C. 0.17
C. 0.47
C. 0.31
D. Not wrong, Not wrong
D. 0.48
D. 0.06
D. 0.29
(high school mathematics, id = 159) At Academic Academy, to pass an alge-
A.
A. 0.17
A. 0.04
0.08
bra test you must score at least 80 %. If there are problems on the test, what is the greatest number you can miss and still pass?
B. 28
B. 0.17
B. 0.07
B. 0.34
C. 35
C. 0.17
C. 0.33
C. 0.33
D). 8
D. 0.48
D. 0.55
D. 0.24
(college biology, id = 44)
A. aposematic coloration
A. 0.17
A. 0.188
A. 0.37
Natural enemies have been implicated strong selective force for all of the following EXCEPT
B. chemical defenses
B. 0.48
B. 0.49
B. 0.26
C. masting (synchronous fruiting)
C. 0.17
C. 0.29
C. 0.21
D. lekking behavior
D. 0.117
D. 0.04
0.16
A. frequency.
A. 0.30
A. 0.57
0.21
(conceptual physics, id 219) Light reflecting from a smooth surface undergoes change in
B. wavelength.
B. 0.30
B. 0.21
B. 0.34
C. All of these.
C. 0.30
C. 0.21
C. 0.27
None of these.
D. 0.111
D. 0.02
D. 0.18
(astronomy, id= 18)
A. ejecta
A. 0.48
A. 0.82
A. 0.36
Which of the following is/are common feature(s) of all fresh (i.c. not eroded) impact craters formed on solid surfaces:
B. raised rims
B. 0.17
B. 0.11
B. 0.31
C. central peaks
C. 0.17
C. 0.04
C. 0.19
D. A and B only
D. 0.17
D. 0.02
D. 0.1.4
professioonl accounting, id = 70) Grant Co.'s sales budget shows the fol- lowing projections for the year end- ing December 31: Quarter Units First 30000 Second 40000 Third 22500 Fourth 27500 Total 120000 Inventory at the beginning of the year was bud- geted at 9000 units. The quantity of finished goods inventory at the end of each quarter is to equal 30% of the next quarter's s budgeted sales of units. What amount should the production budget show for units to be produced during the first quarter?
A. 36000
A. 0.44
A. 0.17
0.24
B. 33000
B. 0.44
B. 0.28
B. 0.25
C. 24000
C. 0.06
C. 0.46
C. 0.32
D. 12000
D. 0.06
D. 0.10
0.19
with Pythia-70n 70m by assigning relatively higher confidence to the target choice (class), while Pythia-lb underperforms Pythia-1600 190hi [AAreetssesssnnts six questions in the hardest question group (difficulty level 10) of MMLJU and their performances of three Pythia models and 0wen1.5-14b.
27

---

Published as a conference paper at ICLR 2025
(a) MMLU.
(b) Arithmetic.
(c) Persian-QAa
Figure A21: The U-shaped and inverted-U scaling of accuracy with group number G 10.
which exhibit the U-shaped scaling: Pythia-160n 160m gets worse than Pythia- by assigning low relative confidence to the target choice, while Pythia- lb performs better than Pythia- 160m. Notably, Pythia-lb's lbbis- performance can vary relative to Pythia-70m, depending on the question.
F SCALING TREND BY QUESTION DIFFICULTY LEVEL ON ACCURACY
We apply the same procedure as in Sec. with accuracy as the performance measure instead of the TC Brier Score. Specifically, we calculate question difficulty level using average accuracy over models before the emergence threshold and plot the scaling trend accuracy for each difficulty group, as shown in Fig. A2T] We still observe clear inverted-U U scaling for the easiest question group followed by steady improvement after the emergence threshold. However, hard question groups longer exhibit a clear U-shaped scaling. Specifically, accuracy performance of hard question groups tends stagnate after the initial performance drop. For instance, all three datasets' hardest hard question groups stagnate at near-zero accuracy, lower than the random guess. The worse-than-random performance can explained by distracting questions, as discussed in Sec. 3.2 On the other hand, the mitigated U-shaped scaling might be due to the fact that accuracy does not capture the change in the model's confidence level the target class. other words, the accuracy-bas procedure cannot capture the models' learning process of first being distracted by questions and gradually overcoming the distraction, because the models' accuracies are all around zero.
28

---

Published as a conference paper at ICLR 2025
cely leakingg xdik
cat buiiing XOL
c013 leakiing
TC Brier Score
TC Brireirr Score
ealy
16.4
-64
1811 31ar03
Brier
tratin
rilling
-0.6
-06
-0.8
-08
compute (M)
(a) MMLU.
(b) Arithmetic.
(c) Persian-QA-
Figure A22: Data and polynomial fit of different degrees for easy and hard groups.
G MORE DISCUSSIONS ON SLUCE-AND-SANDWIIC
G.1 ROBUSTNESS ANALYSIS
We present the robustness analysis of Slice-and-Sandviae regarding (1) the choice of order, (2) lower model log compute cutoff for the training set, and (3) group number G.
G.1.1 EFFECT OF POLYNOMIAL DEGREE
Fig. A22 shows the polynomial fit of TC Brier Score for the easy group with degree=3, 5, and 7, and for the hard group with degree=2, 4, and 6. Note that consider only polynomials of odd and even degrees for the hard and easy question groups, respectively. This prior knowledge reflects the observation that performance of the easy question group initially improves with scale, the performance of the hard question group initially declines with scale, whereas the performance of both groups increases with scale past the emergence threshold. In general, there is bias-variance tradeoff: a polynomial fit of higher degree has higher recall but lower precision. The polynomial fit of higher degree might be over-sensitive to noises in the training data, while the polynomial fit of lower degree might lack the flexibility to capture the turning points in data.
In Fig. A22 we find that the polynomial fit of degree 3 and forecast the scaling trend of the easy question group well except polynomial fit of degree 3 for the Persian-QA dataset, while the polynomial of degree forecasts the scaling trend of the hard question group well. On the other hand, polynomial fit of higher degrees, in particular, degree 7 for the easy question group and degrees 4 and for the hard question group, do not forecast the scaling trend well. We leave it for future work to explore better functional forms model U-shaped scaling for the hard group and inverted-U scaling with steady improvement (deep double descent) for the easy group.
G.1.2 EFFECT OF LoG COMPUTE THRESHOLD FOR TRAIN-TEST SPLIT
Fig. A23 shows the fitted scaling trend using different train-test splitting thresholds. For the hard question group, we use polynomial fit of degree 2. For the easy question group, the polynomial fit of degree 3 is represented by black solid line, and the polynomial fit of degree 5 is represented by black dashed line.
The forecast is reasonably robust to the train-test split. All capture the trend and display similar shape to our original choice of train-test split threshold except for the case where threshold= 1.3 and degree= 5 for the MMLU dataset. The polynomial fit of degree 3 for Persian QA is too flat compared to data for all three thresholds and gets flatter as the threshold goes down. We leave it to future work provide better guidelines as the least upper bound of training data model log compute that still allows us to confidently predict the onset of emergent abilities.
G.1.3 EFFECT OF GROUP NUMBER
Fig. A24 shows the fitted scaling trends when splitting questions into different numbers of groups. We show group number G = 3, 5, and 7. Following the same procedure and degree parameter the main paper, easier question groups, such the groups of difficulty level 1 to 3 for G 7, are fitted
29

---

Published as a conference paper at ICLR 2025
ap Branting aoit
baniing apit tndiing sp order Issoire
Eaaning soik
loiding
0.-
ordir
Isssire
baodfre
aocurac
0.-
accuracy
0.-
02-
0.0
-05
1.0
05
11
15
25
30
0.0
-03
20
65
15
15
10
0.0
-11
200
6.5
15
2.
25.
log compute (M)
compute (M)
log compute (M)
MML.U. Train-test split threshold being 1.5, 1.3, and 1.1.
16-
intirno cuts orbund
euth ortaan bradire
eats ordiant
accuracy
accuracy
0.6
-10 -05
30
65
15
15
20
25
20
-1.5
-05
20
15
15
20
25
35
-12
-05
200
05
15
15
20
25
log compute (M)
log compute (M)
log compute (M)
(b) Arithmetic. Train-test split threshold being 1.8, 1.6, and 1.4.
traning
6.425
traningssl
0.812
saaingastt arcem
0.s
orden-3
.... cur otkens beadine
cura ordkedd
beadine
accuraoy
0.40
0.35
0.25
25 log compute (M)
3.5
-1. -5.5
1.
compute (M)
log compute
(c) Persian-QA QA. Train-test split threshold being 2.3, 2.1, and 1.9.
Figure A23: Slice-and-Sandwich's ss results of accuracy-based scaling law under different train-test. split thresholds. Solid lines are when order=3 is used for easy question fitting, and dashed lines are when order=5 is used.
by polynomial regression of degree=5 due to observed inverted-U scaling; harder question groups fitted by degree=2 due to U-shaped scaling. Then Eq. modified to take the average of Brier-based fitting trends of all but the medium group and project the acquired Brier-based scaling law to the accuracy-based one. Blice-and-Sandvich shows its robustness under G. The robustness comes from similar fitting results among the same scaling types, such as inverted-U scaling, resulting in similar final scaling law after taking their averages.
G.2 HARD LIFT A SIMPLE ALTERNATIVE PIPELINE
-aaaaawinnanihhh...... we provide an even simpler pipeline called Hard-Lift Specifically, we take the polynomial fit of degree 2 on TC Brier Score for the hard question group from Sirre-and-Sandwide and lift it by a constant so the fitted TC Brier Score at the training set model log compute upper bound is equal to the true average. We use this to forecast the TC Brier Score of models past the emergence threshold. We then transform this predicted TC Brier Score back to predicted accuracy via the G(.) function in
As an alternative
Fig, A25 shows the results of Hard-Lij under different log compute thresholds for train-test split as in Sec. 1.1.2 Hard-Lift performs better than the baseline for MMLU (Fig. A25a and Persian-QA (Fig. A25c) datasets, but worse than baseline for the arithmetic dataset. We believe this result reinforces our claim that analyzing difficult--stratffe scaling trends enables more explainable prediction of emergent abilities.
30

---

Published as a conference paper at ICLR 2025
Indre
to 30
(N)
(a) MMLU with group number G being 5, and 7.
compute (40)
log compute (H)
log compute IM)
(b) Arithmetic with group number G being 3, 5, and 7.
Seatin
05 compute (1)
05 compute (M)
compute IMI
(c) Persian QA with group number G being 3. 5, and
Figure A24: numbers G.
results of accuracy-based scaling law under different group
tast wil
Seciing ai
La
(M)
(N)
(a) MMLU. Train-test split threshold being 1.5, 1.3, and 1.1.
log compute (M)
l0g compute (M)
l0g compute (N)
(b) Arithmetic. Train-test split threshold being 1.8. 1.6, and 1.4.
Surti 2008
beting 230
Secting
compusi (K)
computo (9)
compuse (MI
(c) Persian- QA. Train-test split threshold being 2.3, 2.1, and 1.9.
Figure A25: Hard-Lift's results of accuracy-based scaling law under different train-test split thresholds. Harddd.....d-dd. Hard-Lift Lift uses order=2 for fitting hard question groups
31

---

Published as a conference paper at ICLR 2025
token edit distance (TED)
0.2
exact match
0.15
Emergence
Emergence
0.05
00:0
R:g
log compute (M)
log compute (M)
(a) Exact match accuracy log compute (M).
(b) Token edit distance log compute (M).
Frengence
medifed simibrity
aodifidd cosine sinibauity
Imergence
-0.
-14
-0.5
2.5
3.5
-05
0.5
25
log compute (M))
log compute (M)
(c) token edit distance log compute (M) with group number G
Modified cosine similarity vs. log compute (M) with group number G 3.
Figure A26: The exact match accuracy, token edit distance (TED), and cosine similarity
log
compute (M) on the word unscramble dataset in BlG-bench (Srivastava et al. 2023).
token edit distance (TED)
modified cosine similarity
0.0
-0.2
60
-0.4
-80
-100
-0.6
training split testing split
training split testing split
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
exact match
exact match
(a) Relation between token distance exact match.
(b) Relation between modified cosine similarity and exact match.
Figure A27: Relations between token edit distanceemodified cosine similarity and exact match the word unscramble dataset BIG-bench (Srivastava et al.) 2023).
H PRELIMINARY ANALYSIS FOR STRING-MATCH TASKS
This section provides the preliminary analysis for the exact string match tasks.
Fig. A26a shows that the word unscramble dataset in BlG-bench (Srivastava et al. 2023 exhibits emergent abilities under the traditional metric: exact match accuracy. On the other hand. Fig. A26b shows that model performance measured by token edit distance (TED) as discussed in Schaeffer et al. (2024a) improves with scale steadily first and then exhibit flat scaling.
32

---

Published as a conference paper at ICLR 2025
We argue that TED is not good measure of progress differentiate between easy and hard questions well. Fig. shows that performance measured by TED all three question groups is close for all model log computes above 0.5. A harder group TED may be higher or lower than an easier group's. (2) it not very correlated with exact match, the traditional metric that people probably ultimately interested in (A27a)..
string match task. (1) It does not
One idea is to measure performance by modified cosine similarity (MCS):
F(s1) F(s2) [[((2)
MCS
II(s1 C s2),
where is the model's output string, s2 the answer string, F(x) is CLIP (Radford et al., 2021) text encoder to project the string the vector space, and I(x) is indicator function having I if every single character s1 contained in s2, otherwise 0. MCS takes values in the interval [-1, 1,
and is good at differentiating questions by difficulty levels. Fig. [A26d]shows that MCS scaling curves of the easy, medium, and hard question groups are clearly ordered.
Interestingly, performance measured by MCS for all question groups exhibits inverted-U scaling followed by steady improvement. The only differences are the model log compute at which scaling reverts from inverse scaling standard scaling and also how fast performance goes up/down. However, Fig. [27b shows that MCS is also poorly correlated with the exact match. Even if we can precisely predict the MCS of models above the emergence threshold, conversion back to exact match accuracy will be too noisy to be useful. We hope this section illustrates potential avenues for future work.
BROADER IMPACT
1.1 POTENTIAL POSITIVE IMPACTS
This work identifies U-shaped and inverted-U Scaling of LLM performance once group questions by difficulty level. We believe this observation can provide the AI community with a deeper understanding of emergent abilities. We also present a forecasting pipeline utilizing the above observation to detect the forthcoming performance soar, the ability of which we believe crucial in preventing deployment in offensive applications.
1.2 POTENTIAL NEGATIVE IMPACTS
Given the limitations discussed in Sec. 6 we do not suggest predicting the forthcoming emergent abilities based on merely one of the methods we discuss. Multiple techniques should be used in parallel to prevent possible false positives or false negatives.
33


Reviews:
[{"review_id": "JyayontSI6", "reviewer": "Reviewer_bboC", "summary": "The paper connects two well-known phenomena in deep learning, namely emergent capabilities and inverse scaling. By breaking down the scaling curve to different difficulty levels, the authors show that LLM skill emergence can be the result of an inverted-U-shaped scaling on easy questions, where the easy-questions curve cancels out the scaling on hard questions. The authors propose a method to predict emergent scaling by fitting sub-groups of the dataset separately.", "questions": "- Minor correction: In the definition of $M$ (line 128), $N$ and $D$ are mixed up.\n\n- Naming $M$ 'effective model size' is confusing for the reader, since model size is $N$ while $M$ is actually the log of compute $C$. Maybe a better name would be 'effective compute' or just 'log compute?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The main result of this paper connects two important topics in the field, inverse scaling and emergence, making it a very relevant research topic, especially since neither phenomenon is well understood.\nThe results on MMLU and Persian-QA are clear enough, in my opinion, to establish the existence of a connection between U-shaped scaling and emergence. Although Big-bench arithmetic and the appendix results are less clear, they do not diminish from the other results. Even if the easy-hard data split failed to visualize inverse scaling in the latter experiments, a different split might still show similar curves to MMLU.\n\n- The proposed fitting method, slice-and-sandwich, showcases how the documented phenomenon can be used to predict emergence in practice.\n\n- The paper is clearly written and easy to follow.", "weaknesses": "- While the fitting method is an overall positive addition to the paper, it is less convincing than the scaling results (which are, for me, the main result of the paper). The fits shown in the main section use knowledge of the point where emergence starts, making them of little value, since predicting that point is the main reason why we care about fitting emergence.\nAppendix F2 fixes that issue by showing fits on random sections of the scaling curve before emergence, but these fits are not so convincing as a reliable method to predict the point of emergence.\n\n- Appendix E shows the scaling curves when plotting accuracy, and attributes the difference from the main results to accuracy being a 'crude measure', which is not a satisfactory explanation to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7kOpZKJsOt", "reviewer": "Reviewer_nJbX", "summary": "The paper investigates explanations for LLM emergent abilities (sharp transitions from chance-level to high performance as a function of model size) and proposes a method to predict emergent performance. They show that on certain task, when questions are group by difficulty, two distinct scaling trends appear: U-shaped scaling for hard questions and inverted U-shaped followed by improvements for easy questions. This motivated a method for capabilities forecasting: fitting polynomial regression separately for easy and hard questions and then aggregating the results.", "questions": "1. How were the 6 tasks you experiment on selected? What were the criteria and at what point were they selected?\n2. What's the functional form of the polynomials you fit (line 416)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper addresses a very timely topic: the science of LLM evals and predictive evals. It is relevant for scientific understanding of frontier LLMs as well as for AI policy (informing capability thresholds and buffers in responsive scaling policies).\n2. The paper described an interesting insight that easy and hard questions have different scaling trends and that this could explain sharp transitions of aggregate performance", "weaknesses": "1. I found the paper hard to follow. I think the clarity of writing and figures could be substantially improved. Some concrete suggestions:\n(a) the name \"binary Brier score\" is very confusing. What's binary about it? It's continuous for each question.\n(b) I think it's very confusing to say Brier score is \"noisy due to insufficient confidence calibrations\" (line 161). What does it mean to be sufficient? Why is this noise? Why \"calibrations\" in plural?\n(c) what are \"available choices' (line 159)? Is it the same as classes (line 139)?\n(d) names for difficulty levels (e.g. 0_1502_brier) are confusing. They're okay in Python code, but I'd expect something more readable and abstracting away from distracting details (e.g. I don't really care it's 1502 questions).\n(e) The colors on plot could match the semantics of difficulties, e.g. you could use a continuous color map (e.g. red-blue transition to represent easy-hard transition).\n(f) plots for different datasets could be next to each other\n(g) \"Slide-and-sandwich approach allows the data to speak for itself\" is misleading. The whole point is that you bake in more assumptions in the method.\n2. The name \"effective compute\" is somewhat misleading if it's just log compute. Ideally, effective compute could be something like g-factor [1] or the first principal component from observational calling laws [2, 3]. It would be great to use that as a metric. Or at the very least, it should be just called \"log compute\".\n3. Authors only evaluate their methods on 3 datasets (plus 3 more in the appendix). I don't think that's enough to justify the claim that they found a general method for capabilities forecasting and a general explanations of emergent features. BIG-Bench includes more than 200 tasks.\n4. Relatedly, all of them do show emergent capabilities. I wonder how Slice-and-sandwich methods behaves on tasks that do not show emergent capabilities: does it falsely predict them? I think this is an important point: so far the authors only presented a method for emergent capabilities forecasting conditional on the presence of emergent capabilities. But in practice we don't know that in advance.\n5. Overall, I'm somewhat underimpressed by predictive performance and I worry that the authors overfit to a particular set of 6 tasks. If find some patterns in a dataset and bake in this pattern as an inductive biases of your method (polynomial regression), surely you'll improve predictive performance. But will those inductive biases generalize to entirely new tasks you never looked at? The authors provide no evidence for that (unless I'm missing something, happy to be proved wrong).\n6. I'm not super convinced this is a general explanation of emergent capabilities. Maybe others have different scaling trends? The whole point that a superposition of U-shaped and inverted-U shaped scaling explains emergent capabilities is also somewhat qualitative. It's not starkly clear from the plots, e.g. MMLU does not seem consistent with this explanation.\n7. Consider discussing [4] as related work\n\n[1] Unveiling the general intelligence factor in language models: A psychometric approach\n\n[2] Observational Scaling Laws and the Predictability of Language Model Performance\n\n[3] Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?\n\n[4] The Quantization Model of Neural Scaling", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lTF9x653F3", "reviewer": "Reviewer_9qWW", "summary": "This paper investigates the emergent behavior of large language models as their effective size increases. The authors observe that for certain downstream tasks, the continuous metric exhibits a U-shape for easier questions and an inverted U-shape for more difficult questions below the emergence threshold, with steady improvements beyond this threshold. Additionally, they propose a method for forecasting model performance beyond the emergence threshold.", "questions": "1. The authors appear to categorize difficulty levels as easy, medium, and hard. How might the granularity of these difficulty levels affect the results?\n2. In Section 4, the proposed method Slice-and-Sandwich first fits a function to the binary Brier score and then projects the forecasted scaling trend back to accuracy. I am trying to understand the benefit of this additional step. My interpretation is that fitting the binary Brier score curve is more precise, particularly for extrapolation, compared to directly fitting the accuracy curve. Could the authors elaborate on why this approach is preferred over fitting the accuracy curve directly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The paper connects the scaling trend on easy questions with the double descent phenomenon in deep learning is quite interesting.\n2. A group-wise scaling law based on difficulty level can offer more detailed insights compared to a single overall scaling curve.", "weaknesses": "1. This paper primarily focuses on the continuous Brier score as a metric, which may not fully capture the validity of the observed scaling trends. It is unclear whether these trends are specific to this single metric. To establish this phenomenon as more general, validation across other metrics is needed.\n2. The writing and organization of this paper are difficult to follow. For instance, the scaling plots for various LLMs are presented without clearly specifying the types or model families used, as these details are only provided in the appendix. Additionally, the figure depicting the scaling curves lacks clarity; for example, the label '0_1502_brier' in Figure 3 is ambiguous. It appears to represent the index of data points, but using the difficulty level as a label would be more informative.\n3. The proposed task in Section 4 is using data collected before the emergence threshold to forecast the occurrence of emergent abilities and the scaling trend beyond this threshold. However, the paper lacks details on how the threshold is selected and how this choice is validated. Furthermore, more explanation is needed to highlight the significance of this task. Why is predicting the scaling curve beyond the emergence threshold based on early performance trends important?\n4. In the experiments, a few multiple-choice datasets are utilized to demonstrate the presence of emergent behavior, but they also note that such behavior is absent in other BIG-Bench datasets, as detailed in the appendix. This raises questions about whether the observed emergence is specific to certain datasets rather than a general phenomenon. A more comprehensive analysis is needed to uncover broader patterns and clarify the characteristics of these emergent behaviors—for instance, identifying which types of tasks exhibit this emergence and which do not. This would help determine the conditions under which such emergence is applicable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giUaOSkvcc", "reviewer": "Reviewer_K8WA", "summary": "By dividing questions according to difficulty levels, the authors observe U-shaped scaling for hard questions and inverted-U scaling for easy questions, followed by steady improvement. Then the authors propose a simple pipeline, called Slice-and-Sandwich, to predict the model performance.", "questions": "1. I am interested in understanding why averaging the scaling trends of both the easy group and the hard group leads to an accurate prediction of the overall aggregate scaling trend. It's possible that with one prediction being overestimated and the other underestimated, the outcome appears to be coincidentally accurate. The author should provide further (possibly qualitative) explanation to demonstrate that the good result is not a matter of chance. If this question is resolved, I believe the soundness and contribution of the paper can be substantially improved.\n\n2. The authors provide a separate figure for each benchmark. Would it be possible to also provide a figure that includes the results from all benchmarks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The inverted-U vs. U-shape scaling seems interesting.\n\n2. The pipeline seems to be novel. Although the basic principle of the method is fairly easy to understand, it substantially outperforms the baseline methods.\n\n3. The experiments are diverse. The authors uses 56 LLMs to evaluate the difficulty levels, and the effectiveness of Slice-and-Sandwich is demonstrated on 6 benchmarks.", "weaknesses": "1. The fitted scaling trend is either underestimated or overestimated the actual scaling trend. Although the authors have given explanations, it should be clearly pointed out that why this  underestimation /  overestimation is acceptable. For example, the prediction error of the score of hard problems on Arithmetic benchmark (figure 7) seems actually comparable to the prediction error of the score of all problems on Arithmetic benchmark  (figure 9). \n\n2. The authors provide possible explanations for the inverted-U vs. U-shape scaling. They provide examples of easy/hard problems and give very brief descriptions of the performance of different models. The concepts would be easier to understand if the authors can provide some specific outputs of weak/strong LLMs on easy/hard problems in the appendix. \n\n3. There are some typos in the paper. For example, in all figures, the authors use exactly the same color for the easiest and most difficult problems, making it a bit confusing. In line 484, $F_s^c$ should be $F_e^c$. Also, the layout of the appendix seems weird. For instance, P21 only contains a single huge image.\n\n4. Perhaps the authors should use one sentence in the introduction section (line 53) to explain what is `binary brier score` to make it easier for readers to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)