Observational Scaling Laws and the Predictability of Langauge Model Performance
Authors: Yangjun Ruan
Chris J. Maddison
Tatsunori Hashimoto
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Language Models
Scaling Laws
Emergent Capability
Agents
Post-Training Methods
Primary area: natural_language_processing
TL;DR: We introduce observational scaling laws that unify a large set of public LMs in a shared capability space, enabling a low-cost, high-resolution, and broad-coverage scaling analysis for complex LM phenomena

Abstract: Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement of training models across many different scales has limited their use. We propose an alternative, observational approach that bypasses model training and instead builds scaling laws from ~100 publically available models. Building a single scaling law from multiple model families is challenging due to large variations in their training compute efficiencies and capabilities. However, we show that these variations are consistent with a simple, generalized scaling law where language model performance is a function of a low-dimensional capability space, and model families only vary in their efficiency in converting training compute to capabilities. Using this approach, we show the surprising predictability of complex scaling phenomena: we show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models; we show that the agent performance of models such as GPT-4 can be precisely predicted from simpler non-agentic benchmarks; and we show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.

Observational Scaling Laws and the Predictability of Language Model Performance
Yangjun Ruan ,2,3
Chris J. Maddison 2:3 cmaddis@cs, torrrtoooo...................o.. edu
Tatsunori Hashimoto'
yjruan@cs, toronto. edu
thashim@stanfe ord. edu
Stanford University
University of Toronto
3 Vector Institute
Abstract
Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach building this understanding, but the requirement of training models across many different scales has limited their use. We propose an alternative, observational approach that bypasses model training and instead builds scaling laws from 100 publically available models. Building single scaling law from multiple model families is challenging due to large variations in their training compute efficiencies and capabilities. However, we show that these variations are consistent with a simple, generalized scaling law where language model performance is function of a low-dimensional capability space, and model families only vary their efficiency in converting training compute to capabilities. Using this approach, we show the surprising predictability of complex scaling phenomena: we show that several emergent phenomena follow smooth, sigmoidal behavior and are predictable from small models; we show that the agent performance of models such as GPT-4 be precisely predicted from simpler non-agentic benchmarks; and we show how to predict the impact post-training interventions like Chain-of-Thhe Thought and Seff-Consistency as language model capabilities continue to improve.
Introduction
Language model (LM) scaling plays a central role in discussions of model capabilities and affects everything from the tasks they can perform the effectiveness of post-training techniques such as Chain-of-Thoo -hoo-ghht-hhhtt (99). Due to this importance, understanding and predicting LM behaviors across scales, benchmarks, and algorithmic interventions is major question for many researchers and engineers. Machine learning researchers may wish to understand whether their proposed algorithmic interventions remain effective in the face of future model scaling, while engineers and benchmark builders may wish to understand whether complex capabilities such as agentic abilities will scale predictably in the same way as existing LM benchmarks.
4rl 65 have been powerful tools for understanding the scaling trend of LMs, which have shown that LMs follow precise power-law relationship between compute measures (such training FLOPs) and downstream capabilities ranging from perplexity (37 44 to benchmark performance 34 35 This power-law relationship has been used in a variety of ways including hyperparameter and architecture selection [2) 37 4e! well model capability forecasting
Scaling laws 6 36
25 66 671. Unfortunatcly, scaling analyses remain uncommon in many benchmarking and post- training studies, most researchers do have the compute resources build scaling laws from scratch, and open models are trained at too few scales (3-5) for reliable scaling predictions.
We show that many scaling analyses, such as understanding complex L.M capabilities (e.g., "emergent" be done with a lower-cost, higher-resolution, and
behaviors) and post-training interventions, broader-coverage alternative to the standard approach of training LMs across compute scales.
The starting point of our work the observation that there now exist hundreds of open models spanning a large range of scales and capabilities. While we cannot directly use these models for
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

compute scaling laws (as the training compute efficiency varies widely across model families), we might hope that there exists more general scaling law that holds across model families. In particular, we hypothesize that the downstream performance an LM is a function of a low-dimensional space Compute measures
Liama
PC
Log-linear scaling
Standard compute scaling laws
OPT
Log/FLOPps
Open LLM Leaderboard
Model
Low-rank
PC1
Log-linear scaling
Metiic
decompositinn
PC2
Complex downstream capabilities
Principal capability measures (PC)
Apcuracy
Fitted
Extrapolated
Standardized
benchmark leaderboard
PC1 + wxPC2
Figure I: Observational scaling laws generalize existing compute scaling laws which directly relate training compute to downstream capabilities (dashed line) by hypothesizing the existence of low- rank space of LM capabilities that have a log-linear relationship with compute (center), and can be extracted directly from standardized LM benchmarks (left). This enables us to get low-cost, high-resolution scaling predictions of LMs' complex downstream capabilities from their observable standard benchmark metrics using nearly 100 publicly accessible LMs (left to right).
of capabilities (e.g., natural language understanding, reasoning, and code generation), and that model families vary only in the efficiency by which they convert training compute to these capabilities. such relationship held, it would imply that there is log- -inear relationship from low-dimensional capabilities to downstream capabilities across model families (which would allow us to build scaling laws that leverage all existing models), as well as log-linear relationship between training compute and capabilities within each model family (as in standard compute scaling) (Fig.
Through an analysis of standard LM benchmarks (e.g., Open L.M I Leaderboard
8
we find
few such capability measures that have scaling relationships with compute within model families (R² > 0.9) (Fig. 3/. and with downstream metrics across familics. We call such relationships observational scaling laws they predict complex downstream capabilities from simple observable quantities that we expect to scale with compute (like standardized benchmark performance)
The ability to build scaling laws across large number existing LMs from their standard benehmark metrics has significant advantages in cost, resolution, and coverage: Observational scaling incurs no training cost, while leveraging models spanning a much larger compute range than any single model family. It also significantly increases the resolution of scaling laws by virtue of using more models, which is useful for studying nearly discontinuous phenomena like "emergent" capabilities. Finally, observational scaling can combine model families from heterogeneous sources with very different scaling properties (c.g.. LLaMA
91 vs StarCoder (48) which allows us to study how different scaling strategies impact downstream performance and algorithmic interventions.
Finally, we show that using observational scaling laws is low-cost and straightforwarr, as there are few model families that are sufficiently representative to replicate many of our core findings (Sec. 5). By using these representative families, we find that future works can easily make scaling predictions on benchmarks and post-training interventions by evaluating only 10-20 models.
We demonstrate the utility of observational scaling laws in three different settings that are challenging for compute scaling laws but are accurately predicted by ours: (i) Emergent capabilities (Sec. 4.1): We show that the high resolution of observational scaling laws reveals that the emergent behaviors of LMs (98] follow smooth sigmoid, and can be predicted accurately using sub Llama-2 7B models. (ii) Agentic capabilities (Sec. 4.2h: We show that the more complex capabilities of LMs as agents, as measured by AgentBench [57] and AgentBoard [6]], can be predicted with simple benchmark metrics. Our scaling law precisely predicts the GPT-4 performance using weaker models (sub GPT-3.5) and identifies programming capabilities as driving agent performance. (iii) Post-training interventions (Sec. 4.3p: We show that our scaling laws can reliably predict the gains of post-training techniques, such CoT [99] and Self-Con Self-Consistency [97] scale, even when they are fitted on weak models (sub

---

L 7B). Finally, we show how to select only 10-20 representative models to replicate our core findings, making our scaling analyses more accessible with a low cost (Sec. 5).
2 Related Work
In this section, we briefly review the most relevant related work on downstream scaling laws and benchmark correlations. We include extended related work discussion in Appxx. C
Downstream scaling laws
Scaling laws have been generalized beyond pretraining loss to analyze across various domains. How-
transfer learning I 35 85 and downstream performance 15
30
34
ever, whether the LM downstream performance demonstrates a rapid "emergence" or is predictable
with scaling laws remains debated [23 27
38
39
60
79
83
98
[01
Finnveden (25) and Owen
677 have investigated the use of linear and sigmoidal scaling laws, derived from pretraining loss or computational measures, to extrapolate the benchmark performance. Arora and Goyal [5] derived a theory characterizing how LMs' complex skills can be derived as a composition of base skills. Our work differs in that we build practical higher-resolution scaling laws predict LM downstream performance using multiple model families and their observable standard benchmark metrics.
Correlations between benchmarks Numerous works have studied the correlations between NLP benchmarks in vairous contexts [56 68, 69 [1]. Most relevant to our work, Ilié (40] found that single factor explains 85% of the variation on the Open LLM Leaderboard [8] and GLUE leaderboard 195], while Burnell et al. [14] extracted three factors for LM capabilities that account for 82% of the variation on HELM [ST]. aligning with our observations. Our work also observes such benchmark correlations and low-rank structures but is unique in utilizing these properties for the purpose of scaling predictions that can be used directly for benchmark and algorithm development.
3 Observational Scaling Laws
In this section, we introduce our observational scaling laws that generalize the standard compute scaling laws (Sec. 3.1). The key idea is extract low-dimensional capability measure for LMs from their observable benchmark performance (Sec. 3.2), which we find has a log-linear relationship with compute scale measures (Sec. 3.3) and can thus be used as surrogate "scale" for scaling analysis of complex LM capabilities (Sec. 3.4).
3.1 Generalizing Compute Scaling Laws
Standard compute scaling compute scaling laws, there a hypothesized power-law relationship between models' compute measures Cm (c.g., training FLOPs) and their errors Em (c.g.. perplexity). Specifically, for a model m within a family f (c.g., Llama-2 7B, 13B, and 70B) we hypothesize
log(Em) Rs By log(C.n) +
(1)
and if this linear fit is sufficiently accurate, we draw inferences about the performance of a model at future compute scales C" > C' by extrapolating this relationship. However, fitting such scaling law can be tricky, as each model family f and downstream benchmark has its own scaling coefficients 3, and ay. This means that scaling experiments, especially for post-training analysis, are often fitted on very few (3-5) models sharing the same model family, and any predictions are valid only for a specific scaling strategy used within model family.
Several studies [e.g.. 25 67 have generalized the functional form to analyze the scaling of LMs' downstream performance (where Em is normalized to [0, 1)) with a sigmoidal link function a:
(Em) By log(Cm)) + ayy,
Observational scaling In our work, we hypothesize the existence of a low-dimensional capability measure for LMs that relate compute to more complex LM capabilities and can be extracted from observable standard LM benchmarks, as illustrated in Fig. I Specifically, given T simple benchmarks and Bum the error of a model m on benchmark capability vector Sm R such that,
[T].
we hypothesize that there exists some
- (Em) R3 Sm + Sm ~ 0 (yylo((Cm) + Vs Bim ~ no Sm.
(3)
(4)
(5)
for B,, KKK R, and orthonormal vectors 7/ R
3

---

PCA Explained Variance
0.967
00
Explained variance ratio
PG-
0:45
0.34
0.38
0.24
0.08
0.95
0.21
0.35
-06
PC2
0.330
0.40
04
02
PC3
0.10
0.00
0:63
0.4
-22
FC4
0.99
0.14
0.34
0.14
0.31
0.88
...2
2..09
-04
0.2
-06
PC-S
0.14
0.26
0.06
0.21
0.01
0.12
2.92
0.09
0.0
2
3
4
5
MMMU
ARC-C
HelleGaog
Winegrad
TrotfugAA
GSMeK
XWinegra ad
HumenEval
PC
(a) PCA explained variance
(b) Principal component weights
Figure 2: Just few capability dimensions explain most variability LM benchmarks. We find that (a) the benchmark mondel matrix is low-dimensional with the top PCs explaining 97% of the variance and (b) the PCs are interpretable: PC-1, PC-2, and PC-3 emphasize LMs' general, reasoning, programming capabilities, respectively.
diverse range of standard
can view Eq. (3) and Eq. (4) as a generalization of Eq.. (2) since combining them can recover
the original scaling relationships for single model family. However, when there are multiple
model families, Sm serves as shared, low-dimensional space of model capabilities from which all
downstream metrics (E and B) are derived (as indicated by the absence of f in Eq. (3) and Eq. (5), and model families only vary in their efficiency in converting compute into capabilities Eq. (4) One
usefuul way of interpreting EE. (4)is is that 0 represents the compute efficiency of model family f. and Sm is the capabilities of model m expressed in terms of log-FLOPs for this model family.
Finally, Eq. (5) ensurre that these capabilities are not latent variables to be estimated for each model family, but are instead functions of fully observable properties (B). Since E R TgXx XT is orthonormal,
we can linearly estimate 5m := BBm, which makes our scaling analysis significantly more robust. Importantly, this enables us to apply this to a large number public models from heterogeneous sources, including those proprietary ones without any public information C' such as GPT-4.
3.2 Identifying Low-Dimensional Capability Space (Eq.
We validate the existence of low-dimensional capability measure S that linearly relates standard LM benchmarks B by showing that only few principal components of B capture most of its variation (Eq. (5)). We demonstrate that the benchmark-model matrix / for reasonable, broad set of benchmarks and models is low-rank and that Eq. (5) is reasonable assumption.
Models Since the benchmark-modet matrix B be directly measured for any LM, we include a large number of publicly accessible models for subsequent analysis. We collected a broad set of open LMs covering 21 model families (a collection of models across scales such as LLaMA-2 2 7B, 13B. 70B) and a total of 77 models. These encompass models trained from heterogencous recipes, including standard training recipes like LLaMA (I]. those trained on synthetic data like Phi [50]. and models specifically trained on code data like StarCoder [48). For this analysis, we consider only pretrained base models to avoid the complexities introduced by instruction tuning. We also include an analysis for instruction-tune models that include proprietary ones like GPT-4 [66] in [pppx. E. which demonstrates similar results. See Table for a detailed list collected models.
Benchmarks We collected set of diverse benchmarks that assess various LMs' capabilities. These include popular aggregated benchmarks like MMLU [32] that assess the general knowledge of LMs. For more specialized evaluations, we included ARC-C [19], HellaSwag (108), Winogrande 77 for commonsense reasoning. GSM8K 20 for mathematical reasoning, HumanFval (16) for programming, TruthfulQA 53 for truthfulness, and XWinograd [64 for multilingual capabilities. We carefully collected these metries from standardized evaluation protocols for comparability across L.Ms. In particular, we compiled them from standardized leaderboards, like the Open LLM Leaderboard [8] and EvalPlus (55), when available. Otherwise, we used standardized libraries such as the LM Eval Harness 28 to evaluate the LMs. See[Appx. for for full details of our data collection pipeline.
PCA analysis After obtaining the benchmark metrics for LMs, we addressed potential missing values (less than 1% of all data), which may have occurred due to evaluation failures, by using PCA imputation. Subsequently, we applied PCA to extract the principal components of the evaluation metrics as the "principal capability" (PC) measures S (additional details in Appx. D.3).

---

Uamar2
Uama
Qwe116
Owee
Faloo
Pyth
8 100
- 0.23
IP
0.5
P a to
-I- y
05
00
24
L-PP (1221)
Lng.sFL.PP (1231))
Leg.A (122)))
Leg (10
LnteeP (1121)
GPT-Meel
Dodel ama
Shattder
RP 0.1
0.75
-2
-02
0.9
g -12
24
0.5
10
2.5
20
19 Lokue (1tan))
L0R (122))
Log.d (1221))
Log.0hLors (122))
LPP (1421)
(1g..
Figure 3: The extracted PC measures linearly correlate with log-compute within each model family. The linearity generally holds for various model families, and also for lower-ranked PCs (Fi.. F.2)
PC measures are low-dimensional We observe that the extracted PC measures are predominantly low-rank, with the top 3 PCs explaining 97% of the variance, which supports a low-dimensional representation of benchmarks B (Fig. 2a Surprisingly, we find that the first PC alone explains nearly 80% of the variation LM capabilities. Taking a closer look at these PCs, we find that these capability measures represent interpretable directions in which LMs capabilities may naturally vary as a function of scale Fig. 2b). Specifically, PC-1 represents the "general capability" as a weighted average of all metrics; PC-2 corresponds to the "reasoning capability", emphasizing mathematical and coding benchmarks; and PC-3 primarily reflects the "programming capability". These findings suggest that many simple LM capabilities (as covered in our benchmarks) can be expressed as a linear combination of just a few "principal capabilities" S.
3.3 Principal Capability Measures as Surrogate Scale Measures (Eq. (4)
We now show that the PC measures S scale log-linearly with training FL.OPs within each model family, and can thus be interpreted as cross-family generalization of compute C. We discuss some additional applications of PC measures as a smooth cross fammly evaluation metric in in[Apxx.
Setup We collected all available information about training FL.OPs on each of our models, analyzing papers and other public information to identify model size N and pretraining data size D. For the models where we were able identify this information, we used the simple estimate of C' x 6 to obtain model training FLOPs 44]. See Table D. for our collected compute measures.
PC measures linearly correlate with log-compute measures Fig. 3
illustrates the correlation between the top PC-1 measure with the corresponding training FLOPs for models within each model family. We find that for each model family with controlled training recipes and comparable compute scale measures, the LMs' PC-1 measure linearly correlates with their log-training FLOPs (with R > 0.9). This linear correlation holds across a broad range of model families including those specifically trained on multilingual data like BLOOM (100) or those code like StarCoder [48]
also generally holds for lower-ranked PCs such as PC-2 and PC-3, as shown in Fig. E.2 Together with|Sec. 3.2, these results support the validity of EEquation a4d (5) in which hypothesized
that models share the same capability space and a log-linear relationship determines the efficiency by which each model family converts their compute into these principal capabilities.
3.4 Fitting Observational Scaling Laws (Eq. (3))
Fitting regression with PC measures Given a certain downstream error metric E normalized to [0, 1] that measures certain LM capabilities, we slightly generalize Eq. (3) to
Em (3 Sm a
where 3 E R. and a E R are the regression weights and bias, h E [o. 1] is the sigmoidal scale that accounts for the potential discrepancies in the floor performance. We fit the regression with ordinary least squares and restrict h E [0.8, 1.0], which results in h² 1 in most experiments.
Recall that the core component of our scaling law is the fitted linear transformation Pm. := 3 Sm a that maps the extracted PCs into a scalar capability measure for a target downstream metric. While this is perfectly acceptable for prediction, our scaling analysis would be more interpretable if we expressed capabilities in units of FLOPPs rather than an arbitrary scalar measure. We can achieve this by utilizing the fact that for a single family J. our observational scaling law reduces to compute scaling law (Eq. (3) & Eq. (4) Specifically,
Defining interpretable compute-like measures
5

---

note that when Eq. (4) holds exactly, we have that for model m within a family f.
Pm 3 Sm + a² w log(Cm)) + by
where wy = T, and by = B+ V/ + a². This implies a linear correlation between the scalar capability P' and the compute log(C') for models within specific family on downstream task (see empirical validation in Fig. E. Since 0g and Vf are unknown a priori, we can fit these coefficients wy.b, via linear regression from log(C) to P using models from the specific family f-
In the multi-model family case, we can map all models to a shared, FLOPs-based capability measure of a specific family f. The core idea is to represent each model's capabilities by the following hypothetical: "how many FLOPs f) would it take for model in family f to match model m". We call Cn./ the f-equivalent FLOPs for model m, as it represents the performance of model m relative models in the reference model family f. This measure can be computed fairly easily as
log log(Cm..f = j
(8* Sm + a" - -b7),
(8)
Throughout the remainder of this work, we apply this scalar transformation where we pick Llama-2 [92 as the reference family f. and the x-axis of all of our plots can be interpreted as "model capabilities, as measured in units of Llama-2 FLOPs".
obtained from solving for log( log(Cm) in Eq. (7)
4 Validating Observational Scaling Laws
We evaluate the usefulness of observational scaling laws by showing that they accurately predict the scaling behaviors of LMs over complex, surd-to-predic phenomena (like emergent phenomena and agentic abilities) and help estimate the value of techniques such as Chain-of-Thougee
To ensure that our scaling laws are actually predictive and that we are not simply overfitting through various choices in scaling law construction and hyperparameters, we design our exper- iments to have systematic holdout sets and robustness checks. We have also preregistered our predictions for future models after the initial release of the paper test of whether our scaling laws overfit current models. We release our code including the implementation and collected data at https://gi gi
Details in scaling law fits For extracting PC measures, we fixed the number of PCs K = 3 as it covered 97% of the variation in benchmark performance and it consistently yielded the best performance across most of our experiments, see Appx. E.4 for robustness checks on PC selection. For the capability-equivall equivalent scale transformation, we used the Llama-2 [92 as the reference model family as it is currently the most representative and widely used open model in the community. For better interpretability and visualization, we used the accuracy metric, typically defined as Y for fitting the scaling laws and making the plots.
E,
Holdout validation To validate our observational scaling laws, our primary objective is assess how accurately the scaling laws fit the available data and extrapolate from smaller-scale, less capable models to larger-scale, more powerful models. We validate this through systematic holdouts for the test set, where we split available models into weaker and stronger ones based on both scale or capability (e.g.. FLOPs or accuracy). We used the weaker models fit the scaling law and evaluated the extrapolated predictions on the stronger ones. To prevent any train-test leakage, all preprocessing steps (e.g.. PCA imputation) were fitted on the train set only and then applied to the test set. Unless otherwise stated, we set the cutoff to include all models with training FLOPs less than or equal to that of Llama-2-7B (8.4 1022) as training data, resulting 30 models. We included robustness checks for different holdout strategies in Appx. E.4
training set of 47 models and test set of
As baselines, we compare our scaling predictions to using existing compute-based scale measures like training FLOPs and model size. We used the mean squared error (MSE) on the test set our main evaluation measure, which is comparable as the target range is always normalized (0 to 1).
Preregisteration of predictions In the initial release of our paper (May 2024), we have prereg- istered our scaling predictions for future models (see preregistered functional forms in Appx. E.9) and committed to updating the manuscript on ArXiv with our prediction results after 4 months. We have assessed these predictions on new models released after the initial paper release, collected as of September 1st 2024, including most capable open models to date such as Llama 3. 3-00B (24] and Qwen2-72B [105 (see the full collected model list in Appx. D. resulting in an additional test set of 20 models for robustness checks. The results are included in Fig. 4 and additional results on other tasks and new benchmarks are included in Appx. E.3

---

V9-15
R.00M
DeepSeeces?
GPT-Mecd
XGLN
Word Uiscrambble
Persian GA
3-Digh Suisitection
2-0igit Muliplication
Nomalzed Easi Match
1.0
gino 102.0 7,030
Norma Commalteeae Accuroy
6.6
057 5.40)
gim di 1.07x-4 6.87)
Je Aicee
argggme di 50x-33171
2u ail
2.8-
HSR 1.30-02
Hi 2.3e-22
HSRau 0.20
HBBau 1.20
2.6-
0.4-
0.2
Noma
0.2
0.0-
0.0-
1b
18
5
15
Po (16211)
Lo (16211)
LoqusIRLOPs (1E211)
LoqusIFLOPs (LE211)
(a) Training FLOP based scaling law
Word Unsoramble
Dight Substraction
Dight MuRiplication
0.8- y- simo d/2.00c 6.13) HiSrrw i 1.7e04
Accuracc
0.6-
y- 5 s.mo di2 32x 5.430
Normlttee Arccus Rucuuracy
1.
MiSeaa e 6.5
sigmstiss 50x 9.92
L.
MSEa i 2.8e03
sls.mmstizzze 6.451
0.6.
0.8
Normalized
9.4-
0.3
02
62-
0.0
0.0
0.0
Lnggz(.lmaa (182))
(18213)
Logg PLOPs (1821))
Log.
Legydtllma---inp FLOPs (182
(b) Ohservational scaling laws
Figure 4: "Emergent" capabilities of LMs can be accurately predicted from weaker models to stronger ones with observational scaling laws, and using PC measures as the predictor provides much more accurate predictions than using compute measures like training FLOPS and model size (see 1.1 Our preregistered predictions also accurately extrapolate to new models released after the initial paper release, including 3lama-3.114055 (24). Four tasks from BigBench [22] which are identified as "emergent" in 198), are used for illustration.
4.1 Predictability of "Emergent" Capabilities
Recent works have argued that many LM capabilities are "emergent" and cannot easily be predicted
from scall-scale models [27 98. There have been ongoing debates about whether these capabilities
are truly discontinuous and whether the discontinuity is an artifact of the metric used [2] 39
60 78
or lack of high-resolution data points (38] The debate has been complicated by the fact that existing
scaling analyses (including the original ones in Wei et al. (98) have very few points 38 When there are only 5 models across many orders of magnitudes of scale, phenomena can appear to be discontinuous, even if the underlying phenomenon
smooth but rapidly varying sigmoid.
We show that the higher resolution of observational scaling laws allows us to clearly see smooth sigmoidal curves in phenomena that were identified as emergent in Wei et al. (98), and even more surprisingly, we can often accurately forecast the transition points where models go from near- random to high performance using only models whose performance only slightly above random. Our findings validate the observational approach to scaling laws and provide evidence that higher- resolution scaling laws could help us better understand scaling phenomena for LMs.
Setup We tested on four BigBench [2] tasks that were labeled as "emeegeet" in Wei et al. 198. including two arithmetic tasks (3-digit subtraction and 2-digit multiplication) and two non non-arithmetic tasks (word unscramble and Persian QA). Additional results on more tasks covering Wei et al. 198] are included in Appx. E.5) For the models, we included base pretrained models following the approach of Wei et al. [98] For non-arithmetic tasks, we used the default FLOPPs cutoff. For arithmetic tasks, we found that this cutoff resulted in excess of training data near perfect performance (see results in Fig. 1.13). making the prediction tasks trivial. Consequently, we reduced the cutoff to quarter of the default value and also excluded GSM8K (which may be superset of arithmetic tasks) from our base metrics B to make the tasks more challenging.
Prediction results Fig. 4/shows our prediction results using our PC measures as well as the baseline of predicting performance based on training FLOPs. We find that these capabilities can be accurately predicted using our PC measures, even when only using models that perform poorly. In contrast, using training FLOPs results significantly poorer extrapolation on the test set and fits on the train set, as indicated by the much higher MSE values. This discrepancy is likely due to the incomparability of training FL.OPs across different model families. Additional results of the model size baseline are included in Appx. E.5
7

---

(1 (ii v
AgeniBench
Cools:
Noraaeed C
PM0
Uha troliic
B00
Destoment
AgentBoard
a/o
ozs
3
Imp:llamachhhiiiee (107)))
Iagulllarmeecheetnne FLOPx(1121))
(a) AgentBench
(b) AgentBoard
(c) Weight visualization
Figure 5: (a)-(b) The agentic capabilities of instruction-tuned LMs measured by agent benchmarks can be accurately predicted from weaker models (sub GPT-3.5) to stronger ones (e.g.. GPT-4) by their PC measures. (c) The fitted weights (B (8T7) 7) on both benchmarks demonstrate the importance of programming capabilities (HumanEval) for the agentic capabilities of LMs.
4.2 Predictability of Agentic Capabilities
There is significant interest in building autonomous agents using LMs, with notable examples including AutoGPT (75), Devin [46), and SWE-agent [106). Although the performance of these agents still falls far below human-leved on challenging real-world tasks [43] 62 [10), there is a belief that future models larger scales will significantly enhance these agents' capabilities. However, there is a significant uncertainty about whether existing models that are trained for language and code capabilities will transfer well agentic tasks that require taking actions over many rounds. In this section, we utilize our observational scaling laws to analyze the scaling properties of LMs' agentic capabilities w.r.t their backbone model capabilities and show that agent performance highly predictable from simple benchmark metrics.
Setup We tested on two standardized agent evaluation benchmarks, AgentBench [57] and Agent- Board [61, each is a collection of diverse tasks for evaluating LMs' agentic capabilities. For both benchmarks, we utilized their provided aggregated metrics on all tasks for prediction. Specifically, we used the "Overall Score" (denoted as "OA" there), and the "Average Success Rate" on AgentBoard. We included models that have been evaluated on each benchmark, which encompasses both open instruction-tuned models like LLaMA-2-Chat [92], and proprietary models like GPT-4 [66] and Claude-2 (3), see Table D.2 for a complete list of models. We followed the same procedure to collect standardized benchmark metrics B for instruction-ture measures are not available for proprietary models, only our observational scaling laws apply here and not compute scaling laws. The default FLOPs cutoff does not apply either, and thus we held out the top 10% performing models on each agent benchmark as the test set to simulate weak-to-strong predictions, which included GPT-4 and Claude-2 on AgentBench and GPT-4 on AgentBoard.
AgentBench, which is a weighted average of scores across all tasks
models, see Appx. D.1.2 for details. Notably, since compute scale
Prediction results Fig. 5) illustrates the prediction results with our observational scaling laws using PC measures. We find that on both agent benchmarks, the performance of held-out models (GPT-4/Claude-2) can be accurately predicted from models with much weaker performance (> 10% gap). This indicates that the more complex agentic capabilities of LMs are well-correlated with and predictable from their base model capabilities, suggesting the promising scaling properties of LM-based agent capabilities as backbone LMs continue to scale up.
visualize the weights assigned to the base evaluation metrics on both benchmarks, which are derived from the regression weights fitted on PC measures and applied with leamned PCA transformation, i.e., BTT We observe that the fitted weights assign significant importance to programming capabilities (HumanEval) on both benchmarks, underscoring its significance in defining the agentic capabilities of LMs. The weights also emphasize general knowledge (MMLU) on AgentBench, and reasoning capabilities (GSM8K) on AgentBoard, suggesting that these capabilities may also be important for LMs agentic capabilities.
Interpreting the capability dimensions In Fig. 5c
4.3 Predicting the Impact of Post- West-Training Techniques
When researchers propose a new prompting or post-training technique to improve a pretrained model, how can we know whether these gains will persist across models and scales? Systematic scaling analyses have been rare due to the small number of models within a single model family. Moreover, some recent works have argued that certain interventions, such as Chain-of-Thought [99], behave in an emergent way that is not predictable from smaller models (98]. Using observational scaling laws, we show that it is possible to make relatively accurate predictions on the effectiveness of techniques
8

---

Sendi
y-alpmaid) CaT
550)
bol SD forsies
dyyraay 19e
Hole
2:13
1:21
1:15
6.07
0:51
2:34
Accupwcy
1est
ACcuEscI
Vean
Get
2:00
1:30
0.14
6.75
6.1
Cat
145
02
02
MMRU
ABC
HelkSwaol
Winopied
Awinogad
HemaniBial
1.5
25
Jague Equhr. FLOPX (1521))
LogufLlame Equil. FLOPs (1E21)
(a) Scaling prediction of post-training techniques
(b) Weight visualization
Figure 6: (a) The LM performance with and without techniques like CoT and Self-Consistency can be accurately predicted with observational scaling laws. The fitted scaling curves indicate that CoT has a better scaling behavior than SC. See Fei E.15 for detailed per-method scaling plots and comparison with compute baselines. (b) The fitted weights (B8 7) demonstrate a very different pattern when CoT is applied, emphasizing general knowledge (MMLU) and programming capabilities (Humaneval).
such as Chain-os Thought (CoT) [99 and Self-Consistency (SC) |97] as model scale increases. We focus on these post-training interventions in particular, they sometimes discussed as examples of post-training interventions that require scale to be effective [98 [99].
Our approach to quantifying the scaling properties of post-training is straightforward: we fit one observational scaling law using base model performance on a target benchmark (e.g.. GSM8K few- shot), and then fit another on the performance models with the post-training intervention (e.g.. GSM8K w/ CoT)) Each of these fits produces sigmoidal scaling curve as function of log(Cy) and the relative gaps
function of log(Cr) indicates the scaling efficiency of the intervention.
Setup We tested on GSM8K with CoT and SC as post-training techniques and included additional results on BigBench-Hare Hard [83] with CoT in Appx. F.6 As with our study on emergent phenomena on arithmetic tasks, we excluded GSM8K from the base metrics B to avoid making the prediction tasks trivial. We included all the pretrained base models listed in Table D.I iicluding those specifically trained for code data and applied the default FLOPs cutoff for holdout validation. For CoT, we followed Wei et al. [99] and compared CoT prompting using eight reasoning examples with naive prompting using only few-shot examples in the greedy decoding setting. For SC, we sampled five CoT reasoning paths temperature 0.7 to aggregate the final answers following Wang et al. (97] and compared it with a single sampled CoT answer.
Prediction results Fig. 6a shows the scaling predictions for CoT and SC using observational scaling laws. We find that the performance with (CoT, CoT + SC) and without (Naive) post-training techniques for stronger, larger scale models can be accurately predicted from weaker, smaller scale models. In contrast, predictions based on compute scale measures like model size and training FLOPs are less reliable as seen in Fig. E.15 Notably, the scaling trends between the two techniques differ; CoT shows much more pronounced scaling trend compared to Self- Self-Consistency w/ CoT.
Interpreting the capability dimensions Another advantage of observational scaling laws over scaling laws constructed on single families is that we can visualize the capabilities that are important to the post-training intervention. Fig. 6b) visualizes the fitted regression weights B, mapped to the space of base capability benchmarks B via J We clearly see that when we go from Naive to COT, there are significantly higher weights placed on MMLJU and HumanEval meaning that scaling models in way that enhances general knowledge (MMLU) and code (HumanEval) leads greater gaps between CoT and the baseline, while improving along commonsense, such as Winogrande does not necessarily lead to improvements scale. These analyses can inform how different post-training interventions affect different scaling recipes such as code models vs general-purpose LLMs.
5 Selecting Low-Cost Model Subsets for Practical Scaling Analyses
Although our observational scaling law incurs no training cost, it still requires evaluating our bench- marks and post-training methods on larger number of models. To make observational scaling analyses more broadly accessible, we identify a small set of representative models that maintain high prediction accuracy while significantly reducing the evaluation cost.
Method More specifically, we consider the constrained optimization problem of identifying the optimal set of models to choose for a regression problem, subject to the constraint that we select a model subset M of at most M models from the set of all models M.. To define optimality, we turn

---

CoT
Con
I-Cotimoll by Swactto (Dem)) Rando Selection Moddl
Neve: Credy
Get Surge
**C
0.8
1=ESogree82a dEx
(2 4.20-1
Accuracy
Accuracy
0.4
0.2
0.2
0.0
0.0
1.0
1.5
2.0
25
3.0
3.5
1.0
1.5
2.0
2.5
3.5
3.5
Number of Sullcted Modala
Loguo 2 Equlv. FLOPs (1621)
Loguo larna 2 Equir. (1E22))
(a) Prediction error model counts
(b) Prediction results with only 12 models chosen by V-optimality
Figure 7: (a) Selecting the model subsets with our V-optimality criterion leads to significantly lower errors than random selection, and quickly converges the errors of using the full set of models. (b) Using 12 (out of 47) models selected by our method maintains the overall prediction accuracy.
to the theory of optimal experimental design, which states that for linear regression with a fixed design X and subset M, the expected prediction error from using the subset XM is Tr(XTX X XM) )
This gives a straightforward objective achieving the V-optimality [70]:
Tr(ST S (STSS))
min MEP(Ma) s.t. MiS.
where S' E RM is the model-capability matrix obtained from our PC analysis. We conduct struc- tured, exhaustive search over the 21 model families where we include or exclude entire model families under the budget constraint, as we believe these selected models are more interpretable.
Validation We followed the setup in Sec. 4.3 for validating our selection method, as this represents the most likely application scenario for our observational scaling laws by practitioners. Our objective is to replicate our scaling analysis (using a full set of 47 models) in Fig. 6a uajun a small subset of models selected by our method. In Fig. 7a we compute the geometric average of test MSEs on all prediction tasks (Naive, CoT, CoT SC) as the evaluation metric for different selection methods. We find that our V-optimality selection method significantly outperforms random selection and quickly converges to the prediction performance of using the full set of models. In 7b we show that using only small subset of 12 models selected by our method, the fitted scaling curves already effectively capture the scaling trends of different post-training methods, in contrast to randomly selected models (Fig. E. To facilitate future scaling analyses alow cost, we provide reference list of models selected with our method under different budget constraints in Table E.
6 Conclusion, Limitations, and Future Work
We have presented observational scaling laws that generalize existing compute scaling laws to handle multiple model families using a shared, low-dimensional capability space. Using this approach, we show that we can build low-cost, high-resolution, and broad-coverage scaling laws that allow us to make accurate predictions for many complex scaling phenomena, such as emergent behaviors, agentic capabilities, and the value of post-training interventions. We provide concrete practical prescriptions for practitioners to perform similar scaling analyses the hopes of encouraging more quantitative, scaling-law- based approaches to designing benchmarks and post-training methods.
Limitations and future work Finally, we discuss some limitations of our approach and findings: Firstly, observational scaling laws are primarily applicable post-training scaling analyses and do not directly translate to pretraining scenarios in the same way as standard compute-based scaling laws. Secondly, our study mostly focuses on the scaling behavior of model capabilities measured through few-shot prompting or basic prompting techniques (such as CO',, self-consistency, or simple agent scaffolding). Extending our approach to other post-training setups, including scenarios involving fine-tuning or more intensive inference-time computation [12 81). would be valuable. Thirdly, while we have demonstrated that our observational scaling analyses can provide meaningful insights into improving particular models' complex capabilities, promising direction for future work would be to apply the findings from our approach, such as by deriving surrogate measures for model complex capabilities that can be used to optimize models directly and efficiently. Lastly, our assumptions do not account for potential benchmark contamination (where particular benchmark data leaks into model training) or the heterogeneity within model families (where models within the same family may have varying compute efficiencies and scaling behaviors). Investigating the impact of these assumptions approach would be an interesting avenue for future research.
10

---

Acknowledgements
We thank Zitong Yang for his assistance with an early experiment of the project. We also thank Jimmy Ba, Yann Dubois, Honghua Dong, Pavan Kapanipathi. Lisa Li, Karthik Narasimhan, Ethan Perez, Chenglei Si, Tristan Thrush, Zitong Yang. Shunyu Yao, the Hashimoto Group, and anonymous reviewers for their helpful discussions or feedback on the paper draft. This project not possible without the open-source contributions including HuggingFace, EleutherAl LM Eval Harness [28). Open LLM Leaderboard [8] EvalPlus (55), vLLM 45). LMSys Chatbot Arena Leaderboard [8
and AlpacaEval Leaderboard 40
TH and YR were supported in part by gifts from the Tianqiao and Chrissy Chen Institute, Open Philanthropy, Amazon ARA, Meta, and IBM. Resources used in preparing this research were provided in part by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), RGPIN-2021-03445.
References
[1] Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sedghi. Exploring the limits of large scale pre-training. arXiv preprint wrXiv:21 2021.
[2] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311. 2023.
[3] Anthropic. Claude 2, July 2023. Accessed: 2023-08-31.
[4] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024.
[5] Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint arXiv:2307. 15936, 2023.
[6] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.06701, 2021.
[7] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[8] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open 1lm leaderboard. https
//huggingface..c co/ /ttpacce//augggiaaaaae///////eee http://oope__mmmmmeeeeeee 1ea_eebbbarrrrrrdd htgpgg/n.nngggac/o//oooeooeooarr 2023.
[9] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek IIm: Scaling open-source language models with longtermism. arXiv preprint 1xXi::240100295: 2024.
[10] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[11] Sid Black. Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-200 An open-source autoregressive language model. arXiv preprint arXiv:2204. 06745, 2022.
[12] Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:240..21787 2024.

---

[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry. Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
[14] Ryan Burnell, Han Hao, Andrew RA Conway, and Jose Hernandez Orallo. Revealing the structure of language model capabilities. arXiv preprint arXiv:2306. 10062, 2023.
[15] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint wwxiv:2210.148// 2022.
[16] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021.
[17] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality. https://1msys: org/
2llog2203303--000 1008-30-vicunn v0iinn/ March 2023. Accessed: 2024-05-13.
[18] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
[19] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.804377 2018
[20] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
[21] Databricks. tuned 1lm.
Dolly:
The first open commercially viable instruction-
http::/wwww.aaaarcccccccooob//ba hww.abbrrr..c..mmooooo..oo....e. hotp://ooccccooooo00//////// dolly-fires first-rr-tt-er open- commercia htw.://ww........oooaaaaaaaaaaaa www www.:databbicccccc
https://ww..dtabrrttt
April 2023.
Accessed: 2024-05-13.
[22] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2023.
[23] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796 2024.
[24] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[25] Lukas Finnveden. Extrapolating gpt-n n perfoaaan https: lesswrong, com/posts/ (aal:si.KK.cccaPPPPeetrrppaaaggp hatp:pppollliinngg-------.- perforaaace 2020. Accessed: 2024-05-
07.
[26] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language mod- els scale reliably with over-training and on downstream tasks. arXiv preprint arXi::2403.08544 2024.
[27] Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. Proceedings of the 2022 ACM Conference on Fairness, Accountability and Transparency, pages 1747-1764, 2022.
12

---

[28]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for fhw-shot language model evaluation, 12 2023. https::t:::/eenoo. org/ records/10256836
[29] Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL
//saaa/////
[30] Behrooz Chorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Gareia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. arXiv preprint arXiv:2109.07740. 2021.
[31] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the the rise of code intelligence. arXiv preprint arXiv:2401. 2tt1::2001.14966, 2024.
[32] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint #rXiv:2009.03300 2020.
[33] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.0380 2021.
[34] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Hee- woo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:201O. 14701, 2020.
[35] Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01220 2021.
[36] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: 17t:::011.000999 2017.
[37] Jordan Hoffmmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 155566 2022.
[38] Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In The Twelfth International Conference on Learning Representations, 2024.
/aattps://ppnnreeet
net/ fttt://oommdddddioooooooooo
[39] Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelli- gence linearly. arXiv preprint XXiv:2444.09937 2024.
[40] David Ili.. Unveiling the general intelligence factor in language models: psychometric approach. arXiv preprint arXiv:2310.11616 2023.
[41] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
[42] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint wwxiv:2401.04005 2024.
[43] Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-hench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023.
13

---

[44] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wrXiv:200.08830 2020.
[45] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[46] Cognition Labs. Introducing devin, the first ai software engineer. March 2024. URL https 7/www. wwww.cognition-- labs, www.llaaaa..omiini....e...e...ae com.//nnttrouucing-deeiii Accessed: 2023-05-03.
[47] LAION. Open assistant. httts://prrojetts: laionn htt.aaa......ppe--asssstttttttte 2023. Ac- cessed: 2024-05-13.
[48] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint vrXiv2235506166 2023.
[49] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction- following models. httpp://gii ht/ggtttub..... www/tatttu 1ab/ waw.spacaaaaaappaaaaaaaaaaaaaaee evall 2023.
[50] Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint Xiiv:2309.05463 2023.
[51] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.091100 2022.
[52] Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwand et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403. 198877 2024.
[53] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint vrXiv:2109.07958, 2021.
[54] Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe. Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. Few-shot learning with multilingual language models. arXiv preprint arXiv:21 /2. /0668, 2021.
[55] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URI https
//openreview www.eeewwwwww.eeeeeeeeeeeeeee wwtp/foonnmm/ htte:/formmiidqqgxeee 11x11110001
[56] Nelson Liu, Tony Lee, Robin Jia, and Percy Liang. Do question answering modeling improvements hold across benchmarks? arXiv preprint arXiv:2102.01065, 2021.
[57] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations, 2023.
[58] Frederic Lord. Applications of item response theory to practical testing problems. Routledge. 2012.
[59] Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
[60] Sheng Lu, Irina Bigoulaeva, Rachneet Sachdeva, Harish Tayyar Madabushi, and Iryna Gurevych. Are emergent abilities in large language models just in-context learning? arXiv preprint arXiv:2309.0168 01809, 2023.
14

---

[61] Chang Ma, Junlei Zhang, Zhihao Zhu. Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-turn Ilm agents. arXiv preprint arXiv:2401. 13178, 2024.
[62] Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311. 2023,
[63] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning, pages 7721-7735. PMLR, 2021.
[64] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Cheng-Xin Yong, Hailey Schoelkopf. et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022.
[65] Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alek- sandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024.
[66] OpenAL. Gpt-4 technical report, 2023.
[67] David Owen. How predictable is language model benchmark performance? arXiv preprint TrXiv:2401.04757 2024.
[68] Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer, and Leshem Choshen. Efficient benchmarking (of language models). arXiv preprint arXiv:2308. 1696, 2023.
[69] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating Ilms with fewer examples. arXiv preprint PXXi:2442.14499. 2024.
[70] Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
[71] Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang, Renfen Hu, and Lijiao Yang. Revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. In Chinese Computationnll Linguistics and Natural Language Processing Based on Naturally Annotated Big Data: 17th China National Conference, CCL 2018, and 6th International Symposium, NLP-NABD 2018, Changsha, China, October 19-21, 2018, Proceedings 17, pages 209-221. Springer, 2018
[72] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar- 10? arXiv preprint arXiv:1806.00451, 2018
[73] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[74] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-prood q&a benchmark. arXiv preprint arXiv:2311.12022. 2023.
[75] Toran Bruce Richards. Auto-gpt: Auto:opp::Auuomoooootooo Autonomous artificial intelligence software agent. https 2023. URL http:://tt/tttiiii com/ Initial release: March 30. 2023.
[76] Baptiste Roziere, Jonas Gchring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. htt.::2..../900 2023.
[77] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
1021.
15

---

[78] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances Neural Information Processing Systems, 36, 2023.
[79] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[80] Zhihong Shao, Damai Dai, Daya Guo. Bo Liu (Benjamin Liu), and Zihan Wang. Deepseck-v2: A strong, economical, and efficient mixture-of-expetts language model. ArXiv, abs/2405.04434 2024. URL hwt.thtttp/ssaee www.nti
org/CorpusII 269613809
[81] Charlic Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint arXivv24088.33314, 2024.
[82] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alone Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:22066 5,610,,, 2022.
[83] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09266 2022.
[84] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583- 8583-18599, 2020.
[85] Yi Tay, Mostafa Dehghani, Samira Abnar, Hyung Won Chung, William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Scaling laws vs model architectures: How does inductive bias influence scaling? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[86] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint wrXiv:2403.08290 2024.
[87] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma Improving open language models at a practical size. arXiv preprint arXiv:2408.00) 18, 2024.
[88] Qwen Team. Introducing qwenl.5. httpa:///qgeemmmiittuuboboooo Accessed: 2024-05-13.
2024. 2024
[89] The MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, com- mercially usable Ilms. https://www.attbbr..ck.. 2024-05-13.
www.wrickabbcckcccmm wwwpbloggmpptoooeo.oooooooo 2023. Accessed:
[90] François Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Alle- siardo. On the correlation of word embedding evaluation metries. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4789 4797, 2020.
[91] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2rX2v:2302.11377 2023.
[92] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09228 092888 2023.
16

---

[93] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[94] Pablo Villalobos. Scaling laws literature review, 2023. URL http:://ppccaaa.rrgg//////////// scaling-laws- laws- wwt.eettttteeeeeeieeeeee Accessed: 2024-05-12.
[95] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
[96] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song. and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. The Twelfth International Conference on Learning Representations, 2023.
[97] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[98] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
[99] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. -hain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[100] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A 176b coob-parameter open-access multilingual language model. arXiv preprint arX1v:2211.05100, 2022.
[101] Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer. and Ves Stoyanov. Training trajectories of language models across scales. arXiv preprini arXiv:2212.0988 2tt::22222009800 2022.
[102] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
[103] Yiheng Xu, SU Hongjin, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, et al. Lemur: Harmonizing natural language and code for language agents. The Twelfth International Conference on Learning Representations, 2024.
[104] Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Advances in neural information processing systems, 32, 2019.
[105] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1066 10671, 2024.
[106] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent computer interfaces enable software en- gineering language models, 2024.
[107] Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403..466 2024.
[108] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 100.0099900008000000 2019.
17

---

[109] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[110] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo., Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environ- ment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2023.
18

---

A Algorithm
In Algorithm A.I we include the detailed algorithm for fitting the observational scaling laws as described in Sec. 3
Algorithm A.1: Fitting observational scaling laws
Args: number of models M, number of LM benchmarks T, number of principal components K, reference model family f
Input: base LM benchmark error metrics E IRTxM, .target downstream error metric E E RM, LM compute scales C' R4
Result: functional form fitted scaling law
Extract principal capability measures with applicable metric preprocessing
"/
B PCAlmpute(B))
Fill in missing values with PCA imputation
E < Normalize(E) E)
Normalize metric to [0, 1] for sigmoid non-linearity
S + PCA(B, K)
Fit PCA transformation E IR KxT and extract top S = =yB
1* Fit non-linear regression with weights @ E RRK and bias a E R, and sigmoidal scale h E IR */ 8* x²" h² + Fit (E (3 S +
Obtain optimal parameters
P T
Obtain aggregated capability measures P RM
** Project to the capability-equivalent scale of a reference model family
w" by t Fit(P, w log((Cr) + 6) bo Fit linear projection with models in the reference family b Compute f-equivalent FLOPs for all models
log( Cÿ) - ( /'//""
H Return the fitted scaling law with capability scale transformation return F: B + o' (3* yB + a²) or C + ht a (w" log + b*)
#/
19

---

CoT
0.-
Rig -0.97
1.0
Lamae]
Limaa
0.
Owan
Dheen
FV.
siggno 0.50
Gemma
Accuratt
0.6
Oodel
0.2
sigro
0.4
yn aigmo
Off
JPT-MAU
0.2
0.4
Codilbrra
0.0
-2
0
2
6
Logu (1221))
Deupdiul
Logio(Equiv. FLOPs (1E21))
Figure B.1: PC-1 provides a smooth capability mea- sure with wider dynamic range than specific bench- marks like MMLU (Fig. E.4). In contrast to compute scale measures, it also enables the comparison of mod- els from heterogeneous sources on unified scale.
Figure B.2: By transforming the fitted scal- ing curves f-equivalent scales for different model families, we can compare their scaling properties with CoT and analyze the effect training recipes on the scaling behavior.
B Discussion and Other Applications of Observational Scaling
Our work validates the hypothesis that there is -ww-iimesiional space of LM capabilities that captures their scaling behaviors and can be measured via a low-rank decomposition of existing LM benchmarks which interestingly connects to the item response theory in psychometrics (58) that models humans' test performance by their fundamental abilities such as general intelligence. While the majority of our work focuses on applications to scaling laws and predictions, we also find that the shared, low-dimensional capabilities could potentially be used as an evaluation metric and optimization target for LMs. We discuss some of these possibilities here.
PC-1 as smooth capability measure with high dynamic range Many existing benchmarks suffer from limited dynamic range: they either saturate quickly for large models (e.g., HellaSwag, Winogrande) or have completely random performance for small models (e.g.. MMLU, GSM8K), see Fig. E.4.fo for the behavior of each benchmark. In contrast, we find that PC-1 is a smooth capability measure that be used compare LMs across many (at least 5) orders of magnitude. This allows us to compare models from heterogeneous sources and of extremely different capabilities on a single, unified scale (Fig. B.I. We believe that the high dynamic range of PCI may make it suitable as an optimization target for pretraining, where architecture or data interventions can be benchmarked against PC-1 at small scales and validated large scales.
Training data efficiency measurements using PC-1 Extending these ideas further, since PC-1 serves as a unified measure of capabilities, it may serve as good way to compare compute efficiencies across many model families. In Fig. B.1] we plot PC-1 against log-FL.OPs and find that most models to capabilities tradeoff curve. The Phi family is
fall along clear pattern in the training-copput clear outlier in compute efficiency, though this is likely because we are not accounting for the fact that Phi uses additional inference FLOPs to generate training data that not shown this figure.
analyze the interactions between post-training techniques and model families by projecting the fitted scaling curves in Fig. 6aj feequivalent FLOPs for different families using [q. (8) We can then identify which model families benefit the most from these techniques and the point at which they start to benefit. Fig. B.2 shows an example of comparing the predicted scaling of CoT across model families. We find that LMs benefit similarly from CoT, but that Phi is once again an outlier in its behavior; it benefits from CoT much earlier than other model families, but scales less rapidly. Similarly, models specifically trained on code SeepSeck-Codeee also demonstrate an earlier transition but less rapid scaling compared to models trained with standard protocols. The distinct behavior of Phi/DeepSeek- Coder relative to other models indicates the importance of pretraining data in determining model scaling behaviors. While we did not specifically focus on these types of analysis this work, we hope that our approach enables future works to gain further insights into differences between LM training recipes and their scaling behavior.
Post-training interventions and their interactions with model families Finally,
20

---

C Extended Related Work
Compute scaling laws In standard scaling laws 66 34 44 65]. the "scale" is defined by the compute resources allocated to training LMs, such as the number of training FLOPs C, model param- eters N, and training tokens D. Scaling laws are typically formulated as a power-law relationship between LMs' cross-entropy loss L and their compute scale measures. Common functional forms h
include L(N,D) D) Ne + D + [37 65 or L(C)= T + h [34 44), where C' R 6N D 44 for
the Transformer 13 The parameters {a, B, a, b, ) or {7, C, h are fitted by training LMs across different compute scales, varying N and/or D, and measuring their loss. Our work differs from compute scaling laws in our goals compute scaling aims to understand the scaling properties of pretraining. and thus focuses on a single model family and relates downstream performance to directly controllable quantities such training compute. In contrast, we are interested in scaling laws for downstream, post-training performance, which leads us to consider scaling laws across model families and use more directly observable capability measures than compute.
Downstream scaling laws Scaling laws have been generalized beyond pretraining loss to analyze transfer learning 33 (5) and downstream performance [13] 30 [4] across various domains, see Villalobos [94] for a comprehensive review. In particular, there has been evidence suggesting that the few-shot performance of LMs on downstream benchmarks is closely tied to compute measures like model size [13), but whether this is predictable with scaling laws remains debated. Extensive research has explored the difficulties of predicting benchmark performance due their appearing rapid "emergence" [27 [83 98]. while recent works argued the discontinuity is due to the metrics used 60
or the lack of data points [3 (see Anwar al. for survey on this topic). Finnveden and Owen [67] have investigated the use of linear and sigmoidal scaling laws, derived from pretraining loss or computational measures, extrapolate the benchmark performance. Notably,
79
25
Owen 667 also utilized publicly available LMs from different families to fit their compute scaling laws despite the potential discrepancies in their compute efficiencies. Recent studies have also more extensively investigated the correlations between the pretraining loss and downstream performance of LMs 33 IOI, aiding in the understanding of downstream scaling (26) and emergent capabilities (23 of LMs. On the theory front, Arora and Goyal [5] derived theory characterizing how performance on complex skills of LMs can be derived as a composition of base skills. While our work shares similar goals in that aim to understand the downstream, post-training performance of models, differ in our approach in that we aim to build practical higher-resolution scaling laws using multiple model families and their observable standard benchmark metrics.
Correlations between benchmarks Numerous works have investigated the correlations between different benchmarks across various contexts. Extensive research has explored the relationship between the of-iisiibbtttiinnnnnn performance and in on-distribution performance of machine learning 104. In the realm of NLP and LM benchmarks, Qiu al. [7T], Torregrossa et al. [90 found that different evaluations and metrics for word embeddings are highly correlated, and Liu et [56] observed a strong correlation between question-answering benchmarks. Moreover,
models 63 72
73
84
Perlitz et al. [68 Polo et al. 669 observed strong correlations between samples within various LM benchmarks and utilized this observation to develop more efficient benchmarks. Most relevant to our work. Ili [4) found that single factor explains 85% of the performance on the Open LLM Leaderboard [8] and GLUE leaderboard [95), while Burnell et al. [14] extracted three factors for LM capabilities that account for 82% of the variation the HELM benchmark [5I]. aligning with our observations. Our work also observes such benchmark correlations and low-rank structures but is unique in utilizing these properties for the purpose of scaling predictions that can be used directly for benchmark and algorithm development.
21

---

D Experimental Details
D.1 Model Collection & Evaluation
D.1.1 Pretrained Base Models
Model collection We collected a broad set of representative open LMs covering 21 model families
and total of 77 models. These model families include L [92 Llama 191 Llama-3 [24].
Qwen1.5 [88] Qwen 77]. Mistral [41]. Mixtral (42). Yi [107], Gemma 86]. Falcon 22.. Phi [50].
Pythia [TO] BLOOM [O00, GPT-Neo/J [T] OPT [09]. MPT 89) XGLM [44. CodeLlama [76..
StarCoder [48] StarCoder2 [59]. DeepSeek-Coder 31. For preregistration test, we have collected
an additional set of 20 models covering 8 families released after May 2024 and of September
1st 2024, including Llama-3.1 (24), Qwen2 [T05. DeepSeek V2 80] Gemma-2, [87]. Jamba [52],
Yi-1.5 [107], etc. For each model, collected their available metadata including the number of model parameters N and the amount of pretraining tokens D by analyzing papers and other public information. We then estimated the training FLOPs C. using the simple estimate of C 2 6N D 44 for each model. Note that for models that were continually pretrained on additional data such as CodeLlama, we used the sum of the pretraining tokens and the additional continual pretraining tokens to estimate D. See Table for the collected metadata of these models.
Benchmark collection & evaluation We collected a set of diverse benchmarks that assess various LMs' capabilities, including MMLU [32). ARC-C [19), HellaSwag [108), Winogrande (77), GSM8K [20], TruthfulQA [53]. and XWinogrande [64], HumanEval [16]. For MMLU, ARC-C. HellaSwag, Winogrande, GSM8K, and TruthfulQA, we primarily sourced results from the Open LLM Leader- board g with updates current as of May 6th, 2024. When there were missing benchmark results, we followed the standardized evaluation protocols of the Open LLM Leaderboard and used the LM (28) library to evaluate the LMs. For XWinogrande, we used the LM Eval Harness library to evaluate the models with 5-shot examples. For HumanEval, we primarily used the EvalPlus [55] library and followed their standardized protocols for evaluation, and sourced the results from the EvalPlus leaderboard 2 when available. We used the 'Base Tests' results provided by EvalPlus for all the models. See Table D. for all collected benchmark results.
Eval Harness
D.1.2 custruction-Tuned Models
Model collection We collected the set of instruction-tuned models that have been evaluated on the
AgentBench [57] and AgentBoard [6]] benchmarks. These include models like GPT (66. Claude (3).
Llama-2-Chat [92), Manelllam--Instruce (76).
(41), Vicuna [7].
LLM-Chat
[.. Lemur-Chaa [103, OpenChat [96). WizardLM [T02], Guanaco [22]. Koala 229 Dolly-v2 [1].
OpenAssistant 47. We followed the same procedure in Appx. D.I.I to collect the metadata of open models, while for proprietary models these metadata were not publicly available. Note that we only counted the pretraining tokens (and the continual pretraining tokens when applicable) for D and excluded the data for instruction- -tuning or additional finetuning, as these are typically only a small fraction of the total data and are nuanced to estimate due to the complexities in data curation for instruction-tuning. See Deele D.2 for the collected metadata of these models.
Benchmark collection & evaluation For instruction tunee models, we also included standard LM evaluations such as MMLU (32), ARC-C [19] HellaSwag [18], Winogrande (77), TruthfulQA [53] GSM8K 20) and HvaanEval (16). and we followed the same protocols in Appx. D. for evaluating open models. For proprietary models like GPT and Claude, it is more nuanced to evaluate them with a unified protocol (c.g.. due to the lack of access to likelihood scores). so we collected the official results from their respective papers and documentation for all standard benchmarks (except for HvmanEval, which we were able to evaluate using the EvalPlus library). Additionally, we collected capabilities of these instruction-tune models (as of February 2nd, 2024) for reference, we did not utilize this metric for our downstream predictions. See Table D.2 for all collected benchmark results.
Elo scores from the Chatbot Arena [18] which assess cestruction-follovvii
D.2 Downstream Evaluation
For all downstream tasks of pretrained hase models included in Sec. 4.1 and Sec. 4.3) we used the LM Eval Harness (28) library to evaluate all the models. For the "emergent" capability tasks in Sec. 4.1 we applied likelihood evaluation [3 with 2-shot examples. For the post-training intervention
leaderboard
22

---

Table D.1: Collected metadata and base evaluation metrics for base pretrained models used in Sec. 4.1 Sec. 4.3] and Secc. 3 Model names follow the HuggingFace naming. See data collection details Appx. D.I.I For the most up-to-date results, please refer to https: //github. www..oom/oooo.ee connngy ObaScaling/Bi httgg//loommaaneeeeee http:/eellllreellleeeeeeeeeeeeee ts wwsssusssaass.eeee 11m_ benchmark, eval csv
Mede Friele
Model
Ponan
Datx (T)
(1E25
MMLU
ARC-C
Hellówng
Waegrel
TreihflQA
XWargral
20
20
$6.00
0.4250
0:3201
0.7114
0:5433
0.3938
(.754)
0:1250
2
Liase 2 13b-lf
13.9
2.0
155,00
0.5454
0.5811
0/8291
03365
03417
0.7565
0.1829
70.0
2.0
$42/00
0,5465
0.6732
0.8133
0.3314
04492
088345
0.2068
Ilara-h
67
1.0
40.00
0.590
885091
0.781
02141
0.333
06871
52.5
144
733
0.5655
0.8171
0.803
0.4317
033111
0.7075
8533
14
511.68
0.0005
0.5745
0.9904
0.675
0/4313
017666
0.337
1
Mes- 3-00
Mato-lare-A4a 3-85
70.9
811
15.0
15.00
1000100
723
0.1023
11.640
0.8302
0.879
017731
0.5522
04433
04555
148001
IL3941
113344
Useal.S-1
1.3
2.4
23.92
014677
11.31h
060142
06030
0.3943
0.3829
Q.eel
4.0
2.4
57.60
0.5653
0.1846
0.7155
0.633
04733
0.6855
0.3653
O.cel.5:78
7.0
4.0
148.00
0.609
0.5415
0.785
0.117
0.508
0.534
0.3476
Owell 5:358
32.0
4.0
765.00
0.725
0.6357
0.850
0.8145
0.573
0.4207
5-731 5-738
73.9
30
1796.00 00
0.7730
0.688
0.8999
0.935
0.9961
0.8356
0.4512
Qnex-T8
7.0
2.4
100.91
0.5864
0.137
0.784
0.766
04776
0.3366
0:3171
Quas
3.0
253.00
0.670
0.5835
0.680
0.943
0.3557
Mormi
Marra- 1
75
0.616
015099
07795
0/4155
007900
0.2744
Moual
Mistad--78B-r.11
45.9
0.7188
0.663
0.8546
0.319
04693
080002
0.3354
Y:
Y:-48
Y:45
$6.0
6.1
300
3.0
612.81
(09
01411
0.1635
143555
0.969
077533
08859
0L9739
04696
0.3623
019556
07230
012063
Gressn.
gessre
20
600
72:00
0.177
0.436
0,630
0.338
0.3354
falonsen lb
1.0
0.35
2,10
0.3538
0.3507
D5535
06354
03396
0,5355
Fdoo
txcre-ith
filoas To
40.
7.0
1.5
1.0
280191
65.00
0.566
0.370
04787
02213
0L7228
03426
04132
02176
Phu
phi-1 5 pas-
1.5
3.7
0.15
1.4
22.64
1:17
0.130
0.590
B35500
06000
DU5083
0.7388
03333
0.4246
0.4134
0.3115
0.4019
redea- Youe dodenal
0.3
0.13
0.2534
112106
0.2723
0.4554
04455
05521
1L0000
pathis dolaode
0.15
0.3
0.29
0.3456
0.346
5.3133
0.523
04434
0.3235
0.0000
Pydio
pedea- J-bbdoleepa
sythis ib- (h-doapped
1.4
1.0
0.3
0.3
2.50
1.50
0.3427
03550
0,3510
02965
0.3359
0.530
03954
03855
05941
0.550
0.041
pedia- 1-dddddappd
2.8
0.3
3.04
03525
09209
01821
033566
015455
pellia 222-222-- E2--iaappd
12.0
0.3
21.60
0.3565
0.435
03025
03546
03300
06824
0.1159
Mover 150me
0.55
01341
1.15
03341
05183
04434
05786
65061
Noorrsll
11
03311
3.25
0.570
07831
0.500
04140
000000
siven-7hl
7.1
00311
0.1135
04505
0650
0.380
Niver
176.0
01366
395.9
0.705
035043
05511
00756
0.346
03359
Ap--no- 125m
0,125
0.3
0.22
0.2507
162255
50225
04355
05522
xpk-ao-1
13
0.38
2.00
0.3422
0.3123
0494)
05595
03953
0.0066
spei-5b
6,05
0.402
14.59
0.2678
0.435
06554
C3596
D5911
0.3159
20.0
0.472
50.64
0.2500
0.453
0.7345
0.590
03151
07363
0L1250
qpe 115g
0.125
0.18
0.14
0000
02285
02141
0510
0.227
0.587
0.0000
0.15
0.3585
03451
0.575
0.3871
0.5000
0.000
OPT
opp-3 7h.
opo- 7h
6.7
3.7
0.18
0.18
3.07
7.30
03565
03555
0.3396
03311
06565
06588
06136
0.333
0.357
0.505
65000
op-17h
50.5
0.18
3.80
0.1376
05403
03564
0.5516
00055
op-60h
06.0
0.18
71.7%
0.700
065633
0.505
0.000
0.3513
0.6476
00033
MPT
mpt.ttb
repe7b
30.9
7.1
1.0
1.0
190/00
42.00
0.3807
0.4010
0.5599
1L43TD
07753
0/342
IL7244
00400
0L3355
0L3642
U7344
63453
IL1646
0L3134
XGLM
spins T1
spin-- 58
45
177
01.
015
15.50
5.10
102310
012345
1.344h
112385
037933
0L7123
0L3344
06555
UC000
spim-730 26
7.5
0.5
22.30
01377
113433
00077
03332
05906
68935
0L000
70
0.520
Devil brea
Caddlarn Sdbiif
Coddllass Boolf
34.9
2.52
2.52
514.08
0.502
0.5100
0.467
0.753
0.3566
04375
03911
0.766
0.4756
0.3541
70.9
3,02
1206.00
0.5967
0.5674
0.921
0.532
03975
07356
0.5458
comradattwe-tt
1111
1.0
6.01
035667
1122
0.4774
011100
StaCCddrr
7.0
1.0
42.00
182865
113886
0.4046
013440
stncedertave
15.5
1.0
00.00
013212
113009
047733
IL5550
0.402
155552
013410
sarueder3 36
31
3.3
56.40
0.3866
0.3456
0.462
0.454
044449
06607
0.217
StaCoan22
warrodecd-7b
7.0
3.7
0.421
0.851
0.5315
03351
0.463
desrreck-on conke 1. J.b-buu
133
2.
15.69
0.26GG
0.2571
0.3225
0.5212
04255
0.063
0,2570
Dapdlak-Cokkr
67
20
50.40
0.3830
0.3300
0.334
0L5800
044288
0.4350
docpsock-coae -3al--3--a
33.0
20
395,00
0.4091
0.4249
0.9995
04343
0.990
0.9611
0.5120
tasks in Sec. 4.3, we used the same evaluation protocol as the original papers, as described in the main paper. For agentic capability tasks of instruction-tumm models Sec. 4.2) we directly sourced the results from the AgentBench [57] and AgentBoard [6] leaderboards and scaled the metrics to [0, 1].
23

---

Table D.2: Collected metadata and base evaluation metrics for cottuctiiottuuned models used in Sec. 4.2 Model names follow the HuggingFace naming for open models. See data collection details
Appx. D.1.2
Medel Funll
Madd
Panax (B)
Dete
FLOPS (EZ0
Anam-Bli
XMMLU
ARC-C
Wizegrande Battz
TruthiiQQ
Busneskrd
138-00012
OPT
11965586
03660
0.563
694044
11181122
0.700
(6853)
0,550
0.4323
0.7144
1.
chade-latiasll 1.1
1009/714
6.700
(1857)
0.600
0.5115
(lana--lb-ded
74
19h00
1011 0013
6.505
0.993
0/201
0.613
0 18850
100
94
931.0
1003
06305
0.5/9
0.808
0.933
03111
codetaana-he tammat
Osblera-
1.50
0,389
codiltiru Mb-iaarac
348
2.53
514.88
1043
01942
115427
0,560
07451
0:4441
04873
10064716
03833
Veamu
rasm--17--18
130
74
24
196.33
196.0
10003559
0.903
0.555
0.000
D7283
0706
05995
05967
03111
0.901
Dep
67.0
24
904.09
10017234
07174
06775
0.990
05421
0.533
6.2012
Cle
3.00
472 90
0.099
0.500
Dale
0.564
04911
WisstMM
klaadin 2b-c1
eisadlis- 12b-12
24
591.03
L$6.09
1005/001
0.3907
0.580
0,993
0.335
05221
02100
02751
04321
03314
0.002
1.4
$46.09
06221
01344
0.96-7
04240
03331
Kods
Melk-:3
130
18
75.00
963 7996
0.500
0.329
0.729
0,208
0.503
01228
Saily
21.01
0.302
04453
04050
D.3 PCA Analysis
PCA imputation The PCA imputation starts with simple mean imputation for missing values in the data matrix, and then PCA is applied to transform the data into a lower-dimensional space where the missing values imputed by the PCA reconstruction. The above procedure is repeated until the imputed values converge or reach a maximum of 1000 iterations. By default, we used the first principal component (PC-1) impute the missing values, as found be the most robust in our preliminary experiments. Notably, when there are train and test splits, we first applied the PCA imputation procedure the training set and then applied the same transformation to the test set to prevent any train-test leakage.
PC extraction When applying PCA extracting the capability measures, we extracted the top K = 3 principal components from the model-capability matrix. By default, we mean-centered the data before applying PCA without additional scaling, since most evaluation metrics are already normalized into j0, Ij. Similar to PCA imputation, we only fitted the PCA on the training set and applied the same transformation to the test set to prevent any train-test leakage.
24

---

E Additional Results
E.1 PC Analysis of Instruction- Tuned LMs
In Fig. H.T. we conducted PC analysis for instruction-tuned models (see the model list in ble D.2) following exactly the same procedure as Fig. We find that the extracted PC measures for instruction-tuned LMs follow similar patterns as pretrained models and exhibit an even more significant low-rank structure, with the top 3 PCs explaining about 98. 28.6% of the variance in the benchmark performance.
PCA Explained Variance
1.0
0.986
Explained variance ratio
PD-
0.8
-04
PC-2
-0.27
-0.2
-0
0.6
PC-3
0.29
0.02
0.16
0.18
0.00
-3.67
0.38
0.4
PC-4
0.01
0.49
0.23
-0.13
-0.8
-0.16
0.05
-04
PC-S
0:45
-0.46
0.0
-08
1
2
3
4
MMUU
ARC
Helledwwn
Wine Minogranda
TrunftuDA
GSMM
HumanEvel
PC
(a) PCA explained variance
(b) Principal component weights
Figure F.1: The extracted PC measures for instruction-tunee tuned LMs follow similar low-rank structures and interpretable patterns as pretrained base LMs (see Fig. 2).
E.2 Properties of PC measures
Lower-ranked PCs linearly correlate with log-compute measures In Fig. 3, we showed that the top PC-1 linearly correlates with log-compute scale measures (log-training FLOPs) within each comparable model family. In Fig. E.2 we show that this linear correlation generally holds for lower- ranked PCs, specifically PC-2 and PC-3, though the correlation tends decrease with lower-rank PCs compared to the top PC-1.
Aggregated PCs linearly correlate with log-compute measures When fitting our observational scaling laws, we utilized the (hypothetical) linear relation between the aggregated PC measures Pm = 8* Sm and the log-compute measures log(Cm) within each model family to transform Pm into compute-cquivalee scales (Eg. (8) This linear correlation has been partially validated through the linear correlation of top PCs (ig. & Fig. E.2). Here we more directly validate this linearity by analyzing the aggregated PC measures P... fitted specific tasks. Specifically. in Fig. E.3 visualize the fitted P... on the "emergent" capability tasks (i.e., Fig. 4h) versus the compute measures log(C..) within each comparable model family. We find that the aggregated PC measures generally exhibit a linear correlation with the log-compute measures within each family. Notably, the linear correlation is consistently significant for the Llama-2 family, which have used the default reference family for computing the equivalent scales in our experiments.
Single benchmark metric suffers from limited dynamic range In B.I we have shown that PC-1 can serve a smooth capability measure for LMs that provide meaningful readouts across many orders of scales (about orders of magnitude). In Fig. E.4 we show that using a single benchmark metric as LM capability measures amy suffer from a limited dynamic range. In particular, they may either saturate quickly for large models (e.g., HellaSwag, Winogrande) or provide random readouts for weak models (e.g., MMLU, GSM8K).
25

---

Uamuz
Lime
Owern.5
Oncn
Fulon
Pylhe
044
-2-0
244 -247
E4 67-003
04
94 Rh-
244 -0.00
x+-3 13.16mmm 0.49
1.- 6.94
>-000x 0.14
-000 2114
+-0 tha
12
02
-20
-14
1PP (1121)
Pop:sPLOPx DPx (1221)
Lcgu/t DP (1
OP (10211)
L05to (1
Lspu (1FP1)
BLOCM
OFT
X0LM
Codetlane
Statiddr
10 20
66 2.08
pf 005
80 1,22
2222
y-
- 022-0
02-
-34
Logis/LOPa (121))
Logus (1221)
Lep::/
Logg (1822)
Logul/ (1221))
(a) PC-2
Uant2
Lomee
Oweel:
Owee
Fakoon
Pythe
6º 0.00
8-2 1t 2.00
83 nf 004
02- R 1.0
a3-
2.00
E.-
a.--
-27
LegolTLOPs (1221)
Lep::/L.Ps (1227))
LL (1221))
Lopul/ions (1E21)
BLOCM
Neard
StaCodkr
33
FP 14
-3.95
R²-004 094
03
-2.95
-22
0.5 Long (1211
25 (oStc)LOPR EP1
12
20
(1P2)
Lsew ao..0.p..0 (1w.(182))))
Lopp/PL.Px (1F211
(b) PC-3
Figure E.2: The lower-ranked PC measures also linearly correlate with log-compute measures within each comparable model family, though the correlation decreases with lower-rank PCs.
Word Unscramble
Perrien BA
SPg Sucetaction
2-0 gri Miltoiiation
R 2.98
Re - 0.50
R 0.35
1,07
Riaea--
Lagg or de
Aeproooe
Apopppient
CFT
MTT
Logu//_OPs (1E21))
Loqud (1521))
Log.d/RLPP. (1221))
Figure E.3: The aggregated PC measures exhibit a strong linear correlation with the log-compute measures within each comparable model family, especially for Llama-2 which have used as the default reference family for computing the f-equivalent FLOPSs in our experiments.
26

---

R2g 0.71
Mana
Rag 0.95
Osan
0.8
Dren
Tabor
24
B0001
Ruboru
133
03
For
MAA
MI
Loga P1 (1E31))
Loga (10Px(121))))
(a) MMLU
(b) HellaSwag
2.96
37
18g
Lime
0.85
0.83
Oven
0.6
W0oggs
6.70
24
0.62
Ear
0.55
NDT
X0.02
0.52
NAA
MAA
Logud 2222
LooPs (1E21)
(c) Winogrande
(d) ARC-C
0.03
0.5
Qweet
Ovennt
Mitial
0.50
Musia
8:45
Falcan
Pydkas
0.42
0:63
OFF
0.55
OFT
HGLD
Lng:
Log OP
(e) TruthfulQA
(f) XWinogrande
3.8
0.82
Usna
R2. 0..5
Uama
3.4
2.4
Eivaras
Pyllie
0.2
41-
0.8
27154
33
Logg: (1621))
GSMSK
(h) HumanEval
Figure E.4: Using a single benchmark metric to measure LM capabilities may suffer from a limited dynamic range. They may either saturate quickly for large models (e.g.. HellaSwag, Winogrande) or provide random readouts for weak models (e.g., MMLU, GSMSK).
27

---

E.3 Additional Preregisteration Results
Preregistered predictions on post-training analysis tasks In Fig. E.5 we tested our preregistered predictions the wost-training analysis tasks. We observe reasonable forecasts models, and the predictions using PC measures outperform the ones using compute measures like training FLOPs.
Thir
Liame-2 Lome
Uame 311
Gemman
Pythia BLOGH
OFT
Codeciama
Eeepdenic Soder
lest
Owen1:
Faxor
starchverr
Decpbeee V2
Pruineg
GFThhe))
StarCasard
Naive + Greedy
CoT Greedy
CoT + SC Samples
0.5 y 0.995igmoid(0.22 4,46) #0.01
0. w.99sigmoio( 58x 4.01) 0.01
y 0.98sigmoici0. 68x 4,46) 000
MSE 4.6e-04
MSEr 1.4e-02
MSEromn 1.5e002
0.4
MSE uuxt 8.9
MSF
MSE jux
Accuracy
Accuracy
0.6
Accuracy
0.6
0.4
0.1
0.2
0.2
10
5
10
10
Log:s (1E21))
LFgio (1E21))
Log:s(FLOPs (1E21))
(a) Trainig FLOP based scaling laws
Tiln
Jama
Gammaz
Sysh
Deapdiell DendSeokive
Greedy
Greedy
Samples
y sigmoid(1100g 4.77)
sigmoidi2 5.53)
y sigmoid(2. 5.74)
0.8
0.8
MSE suc 2.6e-03
MSE
Accurary
Accuracy
0.6
Accuracy
0.6
0.3
0.4
0.4
0.1
0.0
600
1
2 Lognn(Uama-2-Equiv FLOPs (1E21))
LognnLLama-2-Equu. FLOPs (1E21))
Logun(Uama-2-Equl. FLOPs (1E21))
(b) Observational scaling laws
Figure E.5: Our preregistered predictions of observational scaling laws using PC measures (with # 3) provides reasonable forecasts for new models that are released after our initial paper release. The predictions naive prompting is bit off, but still align with the general trend and perform better than using compute measures like training FLOPs.
28

---

Preregistered predictions on Open LLM leaderboard v2 benchmarks Besides testing new models on exsiting benchmarks with preregistred predictions, we also tested observational scaling laws on the new, more challenging benchmarks being used in Open L.M Leaderboard v2. In particular, we selected a subset of new tasks where at least some exsiting open models demonstrate non-trivial performance (which will exclude benchmarks like IFEval and MUSR) and where scaling predictions from base benchmarks are non-trivial (which will exclude MMLU Pro), including GPQA [74], MATH [33], and BBH [83]. We fit both observational scaling laws and compute-based scaling laws on these benchmarks and compared their extrapolation performance. Since the tasks are more challenging, it requires larger cutoff threshold to include more data points with non-trivial performance on these tasks. We set the FLOPs cutoff to be 16.8, 25.2, 8.4 X 102 for GPQA, MATH, and BBH, respectively. The results are in Fig. E.6 We find that observational scaling laws provides reasonable forecasts on these new, challenging benchmarks and outperform compute-based scaling laws when extrapolating to larger models.
Tra
Uama 2
Laama 3.1 Qwen1.5
YI-1.5
Faicon
GPT/Neo))
MPT
Laaa
StarCoder2
Uaimar-
Gemman-
BLOOM
GPQA
MATH Lvi
BSH
Normalized Accuracy
0.25
MSEman -4.cc-0 04
sigmoid(0.555 6.06)
Normalized lzed Accuracy
MSEexx
sigmoid(0.777 77x 7.15)
2.5e-04
Normal liede Accuracy
0.5
MSEraa: -27c--3
sigmoid(0.19x 2.74)
MSEus 2.3c 03
0.4-
MSEas! 1.3e.02
MSE 2277.02
0.20
0.10
0.2
0.2
0.05
5.0 LogiolFLOPs (1E21))
Logie(FLOPs (1E21))
Logio (1E21))
(a) Training FLOP based scaling laws
Liama
Liama
Falcon
GPT
Startoder2
Test
Uama
Qwen1.5
Gemma
Phi
OPT
Jamba
3
= sigmoio(1.84)
sigmoid(2.227 9.33)
y sigmoid(1.19)
Normalized Accuracy
0.20
MSE oues 2.70-04
MSF us! 8.30 04
Normalizer aaaalized Accuracy
0.25
MSE
MSEjux
2.20.03
6.66-05
Normalized Accuracy
1.50
0.20
0.4
0.15
0.3
0.30
0.10
0.05
69
0.05
0.00
1.0
1.5
2.0
2.5
3.0
3.5
1.0
1.5
2.0
2.5
3.0
L.g.o(Uaama---Equvv FLOPs (1E21))
LoguulLiaa--2Equik FLOPs (1E21))
Logio(Uama-2-Equk FLOPs (1E21))
(b) Observational scaling laws
Figure E.6: Observational scaling laws also provide reasonable forcasts new, more challenging
benchmarks being used in Open LLM Leaderboard v2 and outperform compute-base based scaling laws.
29

---

E.4 Robustness Checks
Number of PC selection Recall that we defaulted to use 3 PC measures for all of our prediction tasks. Here provide additional analysis the impact of using different numbers of PCs the prediction performance and validate the robustness of our choice. In particular, we compare the fitted curves and prediction performance of using 1-4 PCs all our tasks. The results are in Fig. E.7 Fig. E.8) and Fig. E.9 for post-training analysis, "emergent" capability, and agentic capability tasks, respectively. Our results indicate that using than using compute measures like FLOPs, and using 3 PCs consistently leads to the most robust predictions across all the tasks. These validate our choice of using 3 PCs as the default number of PCs and indicate the robustness of our results to the choice of the number of PCs.
than 2 PCs leads to better prediction performance
Dustl
1-agedal
0.20
2625-
020-
819
033
CK-
Lauullmmergs R.0PU117200
arod- 2.05x (121)))
Logu 2oo FLOee (11)))
Sow. FLORI (152)))
(a) Naive Greedy
F.0Ps
PC
P wiil 5000 Mithou actindo
PC 2
y-nigrasth
+8
Lgg.AAlme Eran Forrs 121
Logn//arm Eapr.. Eapo, ruths (121))
Leguslaarr Caui (LOP
(b) CaT + Greedy
7.0Ps
P
P
yregraal Mill surrd
aggi mee sing
aign diic 2.13 na 1ilend
Millar 1100-0
30
55
1 22
5
29 25 4%
as
10 12
(1221)
Lnou 2ww. (1200))
Log.l.amaattue FLOPUTEEEE
(c) CoT SC 5 Samples
Figure E.7: Comparing the prediction performance of using different numbers of PCs for observational scaling laws on the post-training analysis tasks included in Sec. 4.3) Using PC measures consistently leads to better prediction performance than using compute measures like FLOPss with PCs being the best across different tasks.
30

---

Jama3
Usmeo
X0.-0
Spert
#i Max 2
yinigraad) The (.9)
nire (T1)
Nemokkee Earc Warth
100 sas Sertt
Deach
Mihoou
25 2.5
Log. roors Eany
Lwg.lllee (107)/1120))
Lopyllamm (iatt)
Logu Cque FLOrvITEEE
Word Unscramble
T-OPs
P
P
PC 2-4
>*iignobs Stc (44))
pmaggndtt dia- 700)
/=#ignoo0-.00 Dix 6:44
e- aip.atp-33 (400)
1t 120
Mthoou 2
le-dd
oth 4eds
244
U0t was Awrt
Mihrou 1erd2
8.0
Logus/amma (tzr))
Lopy/lhmm (1at))
Logus Cque FLOPVITEEE
(b) Persian QA
T-OPy
PC ooagrooodiio 190) Se
P
PCB+3
PC A-
#*rignolr1 (NT)
yn Pgg 2211 Und trap
aigaee Sic 380
3aaged call oPo some
25
20 (220715P11221)
Lag. (E2r))
Lopy/lam (eat))
Log. Caue FLOvEEEr
(c) 3-Digit Substraction
F_OPs
PC
PC#+3
PC 2-4
rxiignoe Sic AIT)
pnageoat Sio 120
sigaadd 2lt AAR)
MSSho ed 2e03
NdE (dee))
MSCwee 24w33
Accenayy
-25
Lgg. (t2r))
Log.llameeeonn (1079/11200
Lagu (e2t)
Logul Caue FLOYIEEE
(d) 2-Digit Multiplication
Comparing the prediction performance of using different numbers of PCs for observational scaling laws on different "emergent" capability tasks included in [Sec. 4.1 Using 3 PCs consistently leads to the best prediction performance across different tasks.
Figure E,8:
31

---

fratant
Vieaa
Open
Quanars
Dollyn
201ot
Codellanolteteet
>-agmolee0 5ix, 4:15)
y- EiJtsiggnooooo..o te 542) 6.02
y-0 -haigmall 55) 0.01
y y- 4889
0..
16.000
6.4
0.4
7e-1
MöEr 3.5ed
Nommiiee Rocumey
MiSaau S.P6-3
No Cemmaleed Aocuraay
Millaw 2 derD4
Normalooad Accurast
MSlla 2 222e-04
Normal temt Accumecy
MiSau + 1.0e-3
02-
002
0.1
0.1
Log:y Chad Equix. Equil, (1822(1
Loggetlame 2hat Equie Equix (LOPx(1121) Louma 2 Chat Chuul (102))
LogyulUame Chaal Equii FLOPs
(a) AgentBench
Mitrral Instrcct Miuua
Cosollama tosnus
LETU
Couste2
Dewcsenb- McChal
5- smnggmmiej 6ih-- (35)
0.
y- 97epmead?! nix 700 o
0.5
y- 0770pmoa53 sac- 440) 0
35-
sgegeei 16x-2
MiSSau 2.0e
MSSau 1::000
MSEay 8 6.600
MSEay 1.8e02
02- Actuneyy
2
0.3
02-
02
0.2
22
0.0
0.
0.0
00
Logie 2-c.tt----aaaauaaaaa (1621)
2anc(laam2-Chha-auuu (1E25()
Lagne 2 Chhat-oour
(LOPs((121)
Logoell www.ilmaaaarhhaa 2
(b) AgentBoard
Figure E,9: Comparing the prediction performance of using different numbers of PCs for observational scaling laws the agentic capability tasks included in Sec. 4.2 Using 3 PCs leads the best prediction performance across different tasks.
Holdout cutoff selection The cutoff for selecting the holdout set could have significant impact on the prediction performance of observational scaling laws, as it determines the size of the training that could be crucial when the entire dataset is not large (as in case). Here analyze how the prediction performance changes with different holdout cutoffs for various predictive measures (PCs compute measures) and provide quantitative comparison that characterizes their overall prediction performance under varying cutoffs.
Specifically, we conducted the analysis on the post-training analysis tasks in Sec. 4.3 and the "emergent" capability tasks in Sec. 4.1] where there more data points (compared to the agentic capability tasks in |ecc. 4.2) to provide a more robust analysis. For each task, we vary the FLOPs cutoff control the ratio of the test set from 60% to 5% (linearly spaced), which consequently changes the difficulty of the prediction task from more difficult (less training data with weaker performance) easier (more training data with stronger performance). We then compare the test MSE of using different predictive measures under different cutoffs and quantify the overall prediction performance using the area under the curve (AUE). For "emergent" capability tasks, we additionally include a variant of the cutoff strategy that holds out test data based on the accuracy on the task, which simulates more challenging weak-to-strong prediction scenario and offers an extra robust analyses.
The results are depicted in Fig. an and Fig. 1. We observe that in most of our evaluated setups, using PC measures (especially with 3 PCs) generally leads earlier transition the low prediction error region and much lower AUE compared to using compute scales like training FLOPs and model size. This indicates that PC measures more robust under different cutoffs and more sample-efficient for scaling analysis.
32

---

CoT+ Gredy
FUOPL AED SAI COT
131
9.5
01
02
Figure 1.10: Comparing different scale measures under different holdout cutoffs on post-training analysis tasks in Sec. 4.3 The training/test data size is varied by changing the FLOPs cutoff and the area under the test error curves (AUE) is used to measure the overall prediction errors. PC measures (with # 2 3) consistently lead to earlier transition to low prediction error region and much lower AUE compared to compute measures like training FLOPs and model size.
VAST SPECEIIO
10
(a) Varying FL.OPs cutoff
CA.
D.igh
20iglMM
(b) Varying accuracy cutoff
Figure E.11: Comparing different scale measures under different holdout cutoffs on "emergent" capability tasks in Sec. 4. The training/test data size is varied by changing the FLOPs (a) or accuracy (b) cutoff and the area under the test error curves (AUE) is used to measure the overall prediction errors. In 7 out of setups, PC measures (with # = 3) lead to much lower AUE compared to compute measures like training FLOPs and model size.
33

---

E,5 Emergent Capabilities
Predicting with model sizes In Fig. E. 12 we show the prediction performance of using model size for the "emergent" capabilities of LMs. We find that it leads to significantly worse forecasts compared to using training FLOPs and PC measures and poorly captures the "emergence" trend. This is probably because models from different families were trained with very different data sizes and quality and may use different architectures.
Using default cutoff for arithmetic tasks In Fig. we applied different FLOPs cutoff than the default one on arithmetic tasks to make the prediction tasks more challenging. Here, we present the results of using the default FL.OPs cutoff on arithmetic tasks Fig. 1.1 We find that using the default FLOPs cutoff makes the prediction tasks trivial with too many data points close to perfect performance. Notably, using PC measures still outperforms using compute measures like model size and training FLOPs, indicating its robustness to the choice of the cutoff.
Additional tasks In Fig. E.14 we present the results on additional "emergent" capability tasks included in Wei al. 198]. Similar the main tasks (Fig. 4). used the default FLOPs cutoff for non-arithmetic tasks (IPA Transliterate) and a quarter of the default cutoff for arithmetic tasks (3-Digit Addition, 2-Digit Addition). We find that using PC measures consistently leads the best prediction performance compared to using model size or training FLOPs. While the extrapolation does not exactly match the trend of the ground truth the IPA Transliterate task, possibly due the fact that the specific task capabilities are not well covered by our collected benchmark metrics, it still provides a reasonable forecast the "emergence" behavior.
Lime-]
Gemma
SLOOW
XSLM
Stan
Lbucramble
Mulipiicaton
sigmee de 2.17 14.7%) Adt
sig.ae/2..B--- 2.59)
1.0
6w32
Nomre Accurco
2-
Norma
4.4
0.6
3.0
Figure E.12: Using model sizes gives poor predictions for the "emergent" capabilities of LMs.
34

---

Tal
Jamo2 Uemo
Lame-]
OwRY
Semina
Pd
BLOOW
OPT
XSLM
lesk
Oween 5
Paroon
Pylie
G- Moor
M-11
Staroodee
2.0 gai Muleplaaiian
Siiggi Subetraction
S:0gg Additan
2.0g Additam
Mil ded1
Acpursoy
Milr & d0
Acourooy
Millaw tedi
Accursay
MRl Gabe))
C.A
LoguulModde Size (10)
LogualMode Siow (R))
Loggs Stee (80
Logus/Mdde Stee (9))
(a) Model size based scaling laws
90gt Bubstuucion
20ggt4diioon
y- siginno di 72c 370 MSD T4-22
Narrin Aocurio
MSDuu + 20-00
D-sigoxd2..0 615)
1.0
MDDuu 7-le-0
5- s-sggmstt22ee 5.50)
MCDDu S-.00 4y
5- 113ggmastt Ex 3-75)
2.2
0.6-
000
20
2B-
3.0
Lopg (1121))
Lopy (1221))
Lop (1Ea)))
(b) Training FLOP based scaling laws
2 Munitiinnii
3Dgt Bucctrnttion
Adigt.ddiion
A0igt
Millac 6o-22
MiFic 2 16-f2
Millau 67ed8
Millat 6.0e45
0.
1.4
2.4
Log Dipp. (1
Log 2 FLORs (1921))
Log www.-22-ppp..... (1
Logiv]. Eap. (LCPx(1E2))
(c) Observational scaling laws
Figure E.13: Using the default FLOPs cutoff on arithmetic tasks makes the prediction tasks trivial with too many data points close to perfect performance. Observational scaling laws using PC measures (with # 3) still outperform compute scaling laws using model size and training FLOPss.
35

---

Train
Uame22
Llama3
Qwen
Gemma
Phi
BLOOM
XGLM StarCoder
Test
Liama
Owen1.5
Yi
Falcon
Pyshia
GPT-Nea/J
MPT
3-Digit Additton
2 Digit Addiiton
IPA Transiiterat
ym #ggmoi((0.48x 2.70)
y sigmoid(( 19x 116-1166)) 6.)))
0.6
sigmmiit 4.21)
1.0
Nornlized Accuracy
0.8-
MSSSsan
MSESau
+
Normalized Accuracy
3-ee-01
Normalized BLEU
M5G
0.4
X
0.
0.0
LMgoo(Model Size (8))
LMode Size (8))
Logi>(Model Size (8))
(a) Model based scaling laws
3-Digtt Additton
2-Digit Addiiton
IPA Transliterate
ym sigmoid22..111 61x-6xx---5 4.5)
y =igmmid((217> 121--228881
y= (9.70x
Normalized Accurscy
MSEcan
Normallized Accurasy
MSL 4
MSE
Normelized BLEU
MSGmee
MSEaas
0.6
0.0
Lagiu(FLOPs (1E21)
(FL00 (1E21))
(FLis (1E21)
(b) Training FLOOP based scaling laws
3-Digit Adciiton
2-Digit Additton
IPA Transliterate
y= www.sggmodds.s... 3.41)
y s0g.moo0000..0 6000088x 14,03)
y= sigmoid(3.811 3..1/ 8.78)
Normalized Accuracy
MSErran
MSEeut
Normalized Accuracy
MSE
MSEjue
MSEsae
MSEaar
Normalized
0.6-
0.0
2
2
-2
Logro(Llama-2Equir FLOPs (1E2))
(Llama-2-Equiv. FLOPs (1E21))
Logio(Llam==2Eequv FLOPs
(c) Observational scaling laws
Figure E. Results on additional "emergent" capability tasks included in Wei al. 98 Ob- servational scaling laws using PC measures (with = 3) consistently lead to the best prediction performance compared to compute scaling laws using model size and training FLOPs. Although the extrapolation does not exactly match the trend of the ground truth on the IPA Transliterate task, still provides reasonable forecast of the "emergence" behavior.
36

---

E.6 Post- Teattirging Method Analysis
Prediction results with different scale measures In Fig. E.15 we show prediction perfor- mance of using different scale measures various prediction tasks for the post-training method analysis on GSM8K. Similarly, using PC measures well captures the scaling trend and consistently leads to the best prediction performance across all tasks.
Train
Lamaz22
Owent
Mittral
Faleon
BLOOM
MPT XBLM
StarDoder
Test
Laama
Owen
Vi
PN
OPT-Nes)
Stardeder2
Miltral
Pythis
OPT
Codellamm
Naive Greedy
CoT Greedy
CaT SC 5 Samples
90.igmoodd 18x 0.0rm
y- sig. 09x
sigmaiddjo.mm
MSF 6. 0o-04
MSFaus 7e.02
MSF +1 Re:02
0.3-
MSExux 1.3e-02
MStast 1.9e-01
MSlass! =22e=01
Accuracy
Accuracy
Accuracy
0.2
0.1
0.0
0.0
0.0
Logs:(Model Size (B))
Logse Size (B))
Logia(Model Size (8))
Model size based scaling laws
Naive + Greedy
CoT + Greedy
CoT SC Samplee
y= 0.99oggmodd(0336x 4.445 0.00
3.99sigmoid(00588 4.01) 0.01
= 0.08sigmaicc(0.660 4.46) 0.02
MSEyais 4 3.6o
MSE = 1.
MSE 1.5e-02
MBE = 3.7e00
MSEaee 3.7e-02
MSlleee 33 3.8e-02
Accurany
Accuracy
0.6
Accuracy
0.6
0.2
0.4
0.4
0.1
0.2
0.2
0.0
0.0
00
-2.5
0.0
2.5
7'
10.0
-2.5
0.0
2.5
5.0
7.5
-2.5
0.0
5.0
7.s
LOOPs (1E21))
Loguo (FLOPs (1E21))
Logio(FLOPa (1E21))
(b) Trainig FLOP based scaling laws
+ Greedy
CoT Greedy
CoT SC Samples
0.4
y" sigmaic(1 -.77)
y sigmoid(2..0 04x 5,53)
y sigmoid/2. 5.74)
MSErxs 4.6e-05
0.8
MSEraa
08
MSGrsn
MSEaue
MSF 4.76
MSSaee
Accuracy
Accuracy
0.6
Accuracy
0.6
0.2
0.4
0.4
3
1.0
1.5
2.0
2.5
3.0
3.5
1.0
1.5
2.0
2.5
3.0
3.5
Loggo(Llama-2Equucc FLOPs(1E21)
2-quar FLOPs (1E21))
LlggooLlama---Equurv FLOPs (1E21))
(c) Observational scaling laws
Figure E.15: Predicting the impact of post-training techniques GSM8K with different scale
measures. Observational scaling laws using PC measures (with = 3) consistently lead to the best prediction performance across all tasks.
37

---

Results on BBH We further validated our observational scaling laws for predicting the impact of CoT on the BigBench- Hard Hard tasks [83] following the same setup in Lnc. 4.3 In particular, used the defaulted FLOPS cutoff and the same PC measures (# 3). We normalized the prediction accuracy on each BBH task by their respective random prediction accuracy and aggregated the normalized accuracy across all tasks for predictions. The results are depicted in Fig. E, Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks. Furthermore, using PC measures accurately captures the scaling trends both setups, even when using training FLOPs leads to less tight captures in the "Naive" setup or fails capture the behavior of models trained on synthetic data (Phi).
Test
Uama
Owen Mistral
Y
Ph
GPT-NeoiJ
XSUM
Starboder2 DeesBeek
Ulama
Gemma
Pychia
OPT
CodeLlama
Naive Greedy
CoT Greedy
y sigmoid(0, 17x-3.14 14)
y sigmoid(0 2293)
0.4
MSEras =.2ee-3
MSEras 7.1e-03
MSEsus:
3.7e-02
0.6
MSE 1.1e-01
Accuracy
Accuracy
0.4
0.2-
0.1
0.0
LMoie Size (8))
Logio(Model Size (8))
(a) Model size based scaling laws
Nalve - Greedy
CoT Greedy
#igmold(0.42x 4.11)
sigmoid(O 70x 60)
MSEaam 1.6e-03
0.8
MSE
4.3e-03
6.5e
Accuracy
0.3
Accuracy
0.2
0.4
0.1
0.2-
0.0
-2.5
5.0
7.5
-2.5
5.0
z's
Log1o(FLOPs (1E21))
Log:o(FLOPs (1E21)
(b) Trainig FLOP based scaling laws
Naive Greedy
CoT Greedy
y sigmoid[1.26x -6877
y sigmoid(1 62x 5.03)
MSEram 4e04
MSE 1.2e-03
0.4
MSE 3.0e-08
MSEjee 9.7e-03
0.6
Accuracy
Accuracy
0.4
0.2
0.2
0.0
Logra(Llame-2Equvv FLOPs (1E21))
Logne(Lleme FLOPs (1E21))
(c) Observational scaling laws
Figure E.16: Predicting the impact of CoT on BBH tasks. Both using training FLOPs and PC measures leads to reasonable predictions, while PC measures accurately capture the scaling trends in both setups, even when using training FLOPs leads less tight captures in the "Naive" setup or fails to capture the Phi model (which was trained on synthetic data) as an outlier.
38

---

E.7 Model Subset Selection
Prediction results with different number models selected by V-optimality In Fig. 7a we demonstrated how the prediction enrors change with the number of models selected by method. Here we present a qualitative analysis of the prediction results with different numbers of models selected in Fig. E.17 We find that with more than models, the fitted scaling curves have already converged to accurately capture the scaling trend, indicating the efficiency of our method.
Than
Liara 2 Liara
Owaa
Notre
Falcon
Dodkllare Skar Codeer
SterSodaz DuypBokk
Text
Cwca Madre
Liar
Cemrna
WPFT
Greedy
Greedy
Samples
--0.05xigmoii022220 8.79) 0.05
MBE.. 1.5e00
MBE... 1.2e11
MSE 2.46-22
MSFase 1.1
M42 led
ane Ow
0.3
Anqurary
Accuraay
0.2
0.4
0.4
0.1
0.2
0.2
3.0
Logno(Llema--Eguev FLOPs (1E21))
LoggeeL Equiv. FLOPs (1E21)
Loggo(Ussma---Eqpi (1E21(1)21)
(a) 4 models
Greedy
Cready
Samples
0.4
0.977sgmmott( 12x 6.08) 0.0g
stgmoid( 4.70 MSE
sigimmotd( (3811.81)
MSllswe Be0S
08-
MBEaat 6.8e-04
MSEas:
Accurscy
Aguuracy
Accurscy
0.2
9.4
x
022
0.2
2.
35
2.0
30
Loggo[lamms2Eqqu.... Ftt.:uuu....PPP111..111 (1E21))
Log.dtLlama Equir. FLOPs (1E21))
Logio|Llama Equir. FLOPs (1E221)))
(b) 8 models
Naive Greedy
CoT Greedy
CoT SC Samples
y- 0.977aigmoidf A2x 6.11)+ 0.0g
O 94agmeitt224aa 42x 573) + 0.06
y- 20e0ggmol22.... 46) 0.04
MBErux 3.7605
MBErax
0.8
MSE
0.3
Accurazy
Acquracy
Accurnyy
0.2
0.1
0.2
0.2
0.0
100
1.5
2.5
300
3.5
2.0
3.5
Log:o(Ulema 2-Euui FLOPs(1E21))
Lfg:ofUlama Equir.
(1E21))
Log.of 2.uui--Euuii. FLOPs (1E21))
(c) 12 models
Naive + Gready
CoT Gready
CaT SC Samples
y 0.5 0.999igmotet11 16x 4,99) 0.11
sigmmmidd 50x- (2)
ym eigmoldi2 38x- 5.42)
6.8c-04
DB-
8.1e03
0.8
MEEsa:
Accurscy
0.3
Accuracy
0.6
Accuraca
0.2
0.1
0.2
00
0.0
1.0
1.5
2.0
2.5
3.0
355
1.5
1.5
2.0
25
35
35
Log. FLOPs
Lf
FLOPs (1621))
Logio
FLOPs
(d) 20 models
Figure E.17: Prediction results with different numbers of models selected with our V-optimality criterion. The predictions have accurately captured the scaling trend with more than 8 models.
39

---

Prediction results with randomly selected models We present the prediction results with ran- domly selected models from all available models in Fig. E.18 in comparison the results with models selected by our V-optimality criterion QFig. E. All these results are produced with fixed random seed. We find that using randomly selected models leads to a much worse prediction performance, even with 16 models, demonstrating the critical need to carefully select models for effective scaling analyses.
Naiva e Gready
CoT + Gready
CoT + SC Samples
y SteigmaddD 47x--4444444-- 0..0
Milicas Ae-12 MSEaet =1.0e00
0.8
y 95sigmod( 1dx 4,200 0.00
MS5irnh Ba
0.8
y 94aigmoid 9ix 5.90) d.e
MSE 2.8
0.80
0.7
MBEage 544-02
MSEret 2.0e02
Nocurscy
0.6
Accurazy
0.6
0.5
0.20
0.4
0.10
0.2
02-
0.1
1.0
1.5
2.0
255
3.0
3.5
1.
1.5
2.0
2.5
3.0
35
1egioollmme222 FLOPs (1E21)
2el.llm222em Equil. FLOPs (1E21)
Logio 2-Equiv (LOPx(1E21)
models
Naive + Greedy
CoT + Greedy
CoT SC + Samples
y sis.modddo000 2.62)
1.0
y sigmoidjp.88. 1.47)
1.0
y wwg.medr0 0.30)
MSE..w 1.8c06
MBE 4.7e04
MSE 96-04
MSEse 2 2.9e-0
0.3-
MSE * 1.1e01
MSE + 1e-02
Accuracy
Acsurnyy
0
Antumey
0.2
0.0
5
2
-15
-10
-5
Loggo(Ulmaa@.quuvv (.g.amme22gggi..P((111e)) (1021))
Log Equiv. FLOPM (1E21))
(omu FLOPs
(h) 8 models
Nalva + Gready
nigmoid Gready
ynaigmotd Samples
1.0
y sagmoidd
MSEsue 3 3.605
MSEaw! 1.3e-00
MBEln 6.3
MSE 5.1e02
sig
0.8
MSEas: 5 6.8e-02
Mil = 8e-00
Accurscy
Accuracy
0.6
Accuracy
0.6
0.2
D.
0.2-
0,0
Llg..Llamsz-eeuu. FLOPs (1E21))
Lf0:://aaaaaaaaap FLOPs (1E21))
(LogooLllmae--Ean 2-Euuummm---uu....m F FLOPs (1E21))
(c) 12 models
Greedy
Greedy
1.0
w-0.97slgmmd77.54 52x 1420) 200
1.0
sigmold22.7x 5.96)
MGErux 3.0e00
MSS 2.0e-04
MSErww 7.7600
MSEuu
MSEuue
0.8
MSE
Accuracy
Accuracy
8.6
0.6
0.2
02
0,0
0.0
3.
Log,ofl lama-mm2-22 2-Eqiiv. FLOPs (1E21)
Log.o/Llemee22ee Equiv. FLOPa (1E21)
Logg.(.lima22-gu. (1022(1121))
(d) 20 models
Figure E.18: Prediction results with different numbers of randomly selected models. The prediction performance is much worse than our selection method, even when 20 models are being selected.
40

---

Table E.1: Selected models for scaling analysis of post-training methods under different budgets.
Budget
Selected Models
8 models
Llama-2 [7B, 13B, 70B), Mixtral (8x7B), Phi (1.5B, 2), MPT 30B)
Llama-2 {7B, 13B, 70B), Llama-3 {8B, 70B), DeepSeck- -odde (1.3B, 6.7B, 33B),
12 models
Falcon [IB, 7B, 40B, 180B)
Llama-2 {7B, 13B, 70B). J. Mixtral (8x7B), Qwen (7B, 14B, 72B),
20 models
DeepSeck CepSeck-Coddrr {1.3B, 6.7B, 33B ) CodeLlama (7B, 13B, 34B, 70B),
MPT (7B, 30B), Falcon ( (IB, 7B, 40B, 180B)
Llama-2 [7B), J. Llama (7B). Qwen (7B), DeepSeck-Code Coder (1.3B, 6.7B),
8 models, sub
Phi [1.5, 2). MPT (7B)
Llama-2 [7B), ). Llama (7B). Qwen (7B). DeepSeek Coder (1.3B, 6.7B),
12 models, sub
Phi [1.5, MPT (7B). Gemma (2B, 7B), Falcon (1B, 7B)
Recommended model series for scaling analysis To facilitate future scaling analyses for post- training techniques, provide reference list of models selected with method under different budget constraints in Table E.I These models were chosen from all available ones (see) Table D.I with Llama-2 models always being included (as it is currently the most representative and widely used model family), and are expected to be representative of them. Notably, the selected models cover diverse capability ranges and dimensions to capture potential scaling dimensions. For example, under the 12 model budget constraint, the selected models cover both stronger models (Llama-3) and weaker ones (Falcon), well models with specialized programming capabilities (DeepSeek- Coder). Updating this list with other constraints (e.g.. total inference FLOPs) or new model families straightforward, and provide both implementations and guidelines in our released code.
41

---

E,8 Additional Analysis
We have received valuable feedback from anonymous reviewers and have conducted extrnsive additional analysis address their remaining questions.
Extracting PC measures with non- non-matrix factorization We note that the benchmark coefficients on our principal capability measures are not guaranteed be non-negative, which may hinder the interpretability of the extracted components. Therefore, we conduct an additional analysis with non- negative matrix factorization (NMF) to ensure the non-negativity of the component benchmark coefficients that may provide more interpretable capability dimensions. The results are included in Fig. 1. We observed the NMF components do generally demonstrate interpretable decomposition, as well as positive and smooth scaling with training FLOPPs within each model family (as our PC measures).
While NMF offers enhanced interpretability and positive scaling properties compared to PCA, also has notable limitations. Firstly, unlike PCA, NMF does not enforce orthogonality among its extracted components, evident the observed correlation between Components 3 and Consequently, the coefficients assigned to each model across dimensions may not serve as independent measures of specific capabilities. Secondly, the ordering of NMF components lacks uniqueness and intrinsic physical meaning. This contrasts with PCA components, which are systematically ordered by their explained variances. The PCA approach provides an 'importance' measure for each dimension and allows for controlled trade-offss between representativeness and noise inclusion by adjusting the number of PCs used the analysis.
composest
Own1
Pette 4877
Conpusees
Composeeyy
8.22
Componeneg
988
2:11
2.79
16
2.14
0:20
4.94
90
Eneaiize 2012
3.00M
Ina///cN 09221
Conilima
LecaPtre-pooe
Composet44
11-1220
Ma. el-zze
s.x 4r70
consean
97: 031
8:68
4.77
7:40
6442
6:30
7.53
000
Ko-
K
MAKU
Atnic
Trefhaak
CRUSK
Homeffill
2
(a) NMF component weights
(b) NMF compoenent scaling
Figure E.19: Extracting PC measures with non-matrix factorization: More interpretable principal capability measures can obtained by non-negative matrix factorization (NMF). (a) NMF ensures the non-negativity of the component wise benchmark coefficients and provides interpretable de- composition. For example, we may view component and 4 as reasoning and language understanding capabilities, respectively. (b) The NMF components generally demonstrate smooth, positive scaling with increasing FLOPs. The results also hold across other model families and components.
42

---

Liber
V- demoing 344
ViSau +2.0
Noroa Accuraayy
laf Se:34
Mitiour Moace
Clacert
52
Roure
Placcu
Sruttarra F.0Rs 01211
10bss Chot Equx. FLOPU/1E210)
(b)
Figure E.20: Pushing the limit of cutoff point: The cutoff can be further pushed back each individual task while still providing reasonable predictions. (a) Emergent capability tasks: We include three representative tasks, and the task sseciicc FLOPs cutoff are 25, 84, and 8 x 10 respectively (from left to right), compared to the unified 84 x 10 in our current setup. We also test the newly released I Llama-3.1 405B (FP8) assess the generalization on AgentBench that has more available data points with an 80/20 train/test split. The extrapolations underestimate performance some extent, but still align with the overall observed trend.
larger scale. (b) Agentic tasks: We test
Word Unecombre
Perdan 3A
30gt Subutracion
2. Mutplieztion
0d8
Mix-139)
MiTuue 3.3604
0.00-
6.940
3
0.31
0.23
0.76
0.30
0004
2210
0.23
0:33
(1E21)
Log. (1521)
Logg oPu (1E21))
OP (1E21)
(a) Qwen1.5
(h) OPT
Figure E.21: Scaling predictions with single-family models: For scaling prediction from FLOPs within single family, at least 5 models typically required for accurate extrapolation, but the performance is highly dependent on the specific setup. We test Qwenl.5 on -on-algorithmic and OPT arithmetic tasks. Both model families demonstrate accurate extrapolation task but not the other.
Porsla
Multpicationn
yo wr-sgooorzz.000 D0x 11)
y- wt-.gee2.33----22 B445
>- eimmo d5t 32x-1495 MSD + 1e600
yeogrodl2 Dit 1.5)
Nomma Morc
MSE 1.2604
No Nommatte Accuriig
MSSSuu -1 Ee-04
e aigh
MSEve 2.1003
AccurCT
6.4
komsrre
2.0
15
Log (10P)
Logio (152))
Logio(llassBaake FLOP&
Logu
(182))
(a) Emergent capability
(b) Post-training
Figure E.22: Confidence intervals of scaling predictions: We calculate 95% confidence intervals for predictions from the non-linear regression models each data point, and the observed data points fall within these confidence intervals. When extrapolating from very few data points above the random-guess level (e.g., in Persian QA), the confidence intervals may be wider. We include representative tasks for both emergent capability (left) and post-training analysis (right) setups.
43

---

E.9 Fitted Functional Forms for Preregistration of Predictions
In Table E.2 we included the functional forms fitted scaling laws in our experiments. These functional forms served preregistration of our predictions for future models at the time of the initial paper release, which has been used to test the generalizability our scaling analysis to unseen models.
Table E.2: The functional forms of the fitted scaling laws included in paper, preregistered
for predictions of future models. Each functional form is presented as the logit of the normalized accuracy metric (Y,h) h) oー1 ((Y (1 h)) /h) X that is equivalent to Eq. (6) Each
benchmark metric scaled to be within the range [0, 1].
Setup
Task
Functional Form
(Y, 1.00)
2,00 10 (CLtma))) 6.31
Word Unscramble
1.02MMLU 3.02ARCA: 5.73HelliSwng + 2 2.44Winogmd
1.06IramtiQA 1.2IGSMEK + 2.48XWinograt 0.08HumenEval 12.28
2.32 1o 8.43
Persian QA
2.86PC1 3.18PC2 0.10PC3 6.26
2.08MMLU 1.08ARCCC 1.13HelliSwng 0.53Winograd
"Emergent" capabilities Sec. 4.1
0.361hatsulQA
+ 0.60XWinograd 1.55HummEival
*"V 1.000
10 8.92
3-Digit Substraction
5.98PC1 + 39.55KC3
2.17MMLU 2.32ARCC 3.44HellnSwng 7.96Wiogrrd +
0.65ThubiulQA 34 27XWingraal 20. 20.3991mmanEval 20.99
-Y. 1.00)
2.22 1o (Cimmn) 4.45
2-Digit Multiplication
3.60PCI 4.24PC2 + 8.05PC3 - 2.68
1.62MMLU 9.9ACCC 0.00HelliSag W.noWinograd
0.14InasuulQA 6.suXWinograd 6.52HumanEval
(Y, 4'' 0.99)
l 5.52
AgentBench
2.32PC1 0.79PC2
2.34MMLU 0.82ARCC + 0.32HelliSang + 0.54Winegmade
Agentic capabilities
0.G0TrunduQQ 0.42GSMBK 2.63HaarmEvvll 6.37
See.
(Y,
0.98 (CunmeCCn 6.60
AgentBoard
3.02PC1 2.60PC2 1.17PC3 2.98
1.10MMLI 0.31ARC-C 0.55HellaSeag 0.14Winggrand:
0.561m0001QA + 3.30Humanbval
44

---

Sctup
Functional Form
(Y, 1.00)
1.09 logio (C 4.77
Greedy
2,89PCI + L.55PC1 0.38PC3 3.57
1.53MMLU 1.30ARC-C 1. 2.22%cllaSaag + 0.75Wingrrdd +
0.16TumMiiIQ 0.15XWinggad 1.92HumanEval
(Y,, 1.00)
2.04 logio (Cimm=)) 5.53
GSM CUT Greedy
2.56PCI 4.64PC1 4.21PC3
5.03MMLU 2.04ARC-C 0. 1.Hellawwg + 0.96Wingral +
1.75muhffl0 2 2.39XWinegnad S.S.Humntvvll 4.77
* 1.,0)
Post-training (unalysis Sec. 4.3)
2.19 logio (Ctum)2 5.74
GSM CoT
2.73PC1 4.82PC2 + 4.9GPC3 2.49
B.SSMMLU 2.27ARC-C 0. 0.08Hellawag 1.11 1.11Winggrd +
1.97TamMiIQ 2.78XWinggad 2.45HominEval
1.00)
1.26
4.87
BBH Naive Greedy
2.70PCI 3.06PC1 0.84PC3
1.41MMLU + L.05ARC-C 5.7 + 0.36Wingral +
0.11muhhflAA 0.61XWinogad H.6.Huamat 6.47
6 "Y 1.00)
1.62 logio (Cummm)) (.6loooo
5.03
BBH CoT Greedy
4.20PC1 + 3.81PC2 2.92PC5 - 3.12
0.84MMLUU 1.30ARCCC 1.57HeliaSwag + 0.42Winggrad
0.44TomMMlQQ 1.96XWingedd 6.62HunanEaal
45

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims the abstract and the introduction were carefully drafted to precisely describe the contributions and scope of the paper and checked to ensure that they are consistent with the empirical results in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the major limitations of our work in the conclusion section
(Sec. 6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
46

---

Answer:[NA)
Justification: Our work does not contain any theory. Note that our paper does involve set of assumptions develop our observational scaling laws (Sec. 3.1, which have been empirical validated throughout our paper Sec. & Sec. 4).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have carefully described all the experimental setups in Sec. 4 and included all experimental details in Appx. DJ We have also included complete algorithm for our methd in inAllorii A. We have also released our code to reproduce the results and provide the link in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
47

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released the code and data that we have collected to reproduce our results. We have included the link in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. ec/
public/guides httpi///uuudee/ooouuuuinsssssss www.ssssiiiiiyyyii/o ollicy for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. wwwbbicc/gudes//ooooooooooooooo
setbbbmmmssiiiiii ssi ssionPolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have carefully described all the experimental setups in Sec. and included all experimental details in Appx. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We include our results with error bars in Fig. E.22 We also performed several robustness checks of our method to hyperparameters in Appx. H.4
Guidelines:
The answer NA means that the paper does not include experiments.
48

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Our paper does not involve experiments that require significant computational resources and our results are not sensitive the compute being used, we did not include this information.
Justification:
Guidelines:
The answer means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 't make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httttp:://nuurpppe http:ppppwppppppccpccccccceeeeea
Answer: [Yes]
Justification: Our work has been conducted in accordance with the NeurIPS Code of Ethics. We have carefully considered the ethical implications of our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
49

---

The paper is on the foundational research side and is not tied to particular applications. We do see any direct societal impact the work performed.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve releasing data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited every public library or leaderboard that has been used in our work, see our reference list.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
50

---

The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi wwwode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information
supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects, did not need IRB approval.
Guidelines:
51

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
52


Reviews:
[{"review_id": "OVPjO1CedL", "reviewer": "Reviewer_V1YP", "summary": "The paper proposes observational scaling laws, a method to use benchmark scores of LLMs to infer how their performance would change if the amount of training compute was scaled, without actually having to train additional models. The authors apply PCA to a model-task benchmark matrix in order to obtain latent capabilities, and find that only a small set of such capabilities can explain almost all of the benchmark variation. They use the latent capabilities to infer how efficient different model families are at converting compute into these capabilities, and to infer how models will perform on a new benchmark, based on their capabilities. The authors then examine how these observational scaling laws can be use to predict model performance on emergent, agentic and prompt-induced abilities and find that the predicted scaling performance closely matches the actual performance of held-out models.", "questions": "1. Would there be benefits to using more fine-grained benchmarks to estimate the skill vector $S_m$, for example a detailed breakdown over different BigBench tasks?\n2. Would it be possible to use the observational scaling laws to estimate the effect of finetuning? I.e. suppose we knew for a few models how finetuning them on a particular dataset changed their performance on some target measure. Could we apply the observational scaling laws to infer the performance of finetuning more capable models?\n3. How many models per family would be needed in order for direct extrapolation from compute/model size for that family to produce similarly accurate results as the observational scaling laws here?\n4. The conceptual model of latent capabilities that determine benchmark performance is strongly reminiscent of [item response theory](https://en.m.wikipedia.org/wiki/Item_response_theory) (IRT) from psychometrics, which underpins exams such as GRE. In IRT the underlying skills also determine a sigmoid-shaped probability curve, according to which a student will be able to answer a question with a particular difficulty correctly. It would be quite interesting to draw a connection between IRT and the latent capability model used by the observational scaling laws.", "limitations": "The paper does not explicitly discuss limitations. I believe that the discussion throughout the paper is sufficient, though there may still be some potential limitations that the authors do not touch upon, for intance:\n1. How well do the scaling laws hold up if benchmark data has leaked into training?\n2. All models within the same family are assumed to share the same compute-efficiency. I think that this is a valid assumption, because models in the same family are by and large trained with the same architecture and dataset, but it's conceivable that there may be more heterogeneous families.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed observational scaling laws are a valuable contribution towards anticipating LLM abilities as they scale:\n    - They are helpful in extrapolating various LLM abilities (emergent, agentinc, prompting-based, and possibly other abilities), which can help in clarifying some of the scaling debates around these abilities, e.g. such as for emergent abilities, and which gives practitioners a tool for anticipating the performance gains of different methods, such as of prompting strategies.\n    - They do so purely based on observational benchmark performance, i.e. they alleviate the need for costly (interventional) training of additional models at different scales.\n    - They seem to obtain reliable estimates by doing a joint inference over different model families and benchmarks.\n2. The paper conducts a comprehensive analysis spanning benchmarks for 77 models from 21 families which demonstrate that the proposed observational scaling laws, using latent capability estimates, can predict the performance of more capable models from less capable ones, with low error and better than naive methods that directly estimate performance from compute or model size.\n3. ​The paper offers some interesting insights:\n    - The authors show that emergent LLM abilities may be explainable by a lack of \"sampling resolution\" in terms of number and performance of models, corroborating recent results in that area.\n    - The latent capability model allows for inferring how much the performance on different benchmarks contributes to abilities (e.g. agentic and prompting-induced ones), showing that for instance performance on coding benchmarks strongly contributes to agentic and chain-of-thought abilities.\n4. Overall, the paper is well written.", "weaknesses": "1. The conceptual model proposed in the paper is that the (principal component) PC-dimensions (latent capabilities) obtained based on model-benchmark performances capture basic, supposedly interpretable skills (e.g. general, reasoning, and programming capabilties) that increase the more training compute is applied to the models. However, as shown in Figure E.2 in the appendix, for some models for PC-2 and PC-3 there is a negative or flat relationship between training compute and model skill, as measured by the corresponding PC. Additionally, the performance on some of the benchmarks correlates negatively with some of the PCs, e.g. -0.62 for HellaSwag and PC-2 (which supposedly captures reasoning abilities). Given these discrepancies, I am not sure that the mental model of interpretable skills is really applicable for the PCs (even though they they seem to be useful for the scaling predictions).\n2. In Section 4.2 and Figure 5 the authors show that observational scaling laws can be used to extrapolate the performance of LLMs in agent settings. However, the test sets used here are quite small (2 for AgentBench, 1 for AgentBoard), which makes these conclusions somewhat unreliable. Perhaps a different training-test split (e.g. 80/20 or 70/30) could be used (instead of 90/10) in order to obtain a larger test set. Would the results still hold?\n3. There are some minor clarity issues\n    1. Section 3.1 defines an error metric ($E_m$) which is somewhat unclear, i.e.\n        - Is $E_m$ capturing errors or performance (~= 1 - errors)? Perplexity is mentioned as an example, but shouldn't perplexity decrease the more compute is applied to the model (Eq. 1)?\n        - Is it always normalized? Line 111 mentions that, but I am not sure about the initial definition.\n    2. I found it a bit difficult to understand Figures 5c and 6c from the captions. It would be useful to mention in the caption that they are based on $\\beta^T \\gamma$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We appreciate that you acknowledged our paper offers a “valuable contribution” with a comprehensive analysis and interesting insights. We would like to address your remaining questions and concerns in the following response.\n\n\n\n### Interpretability of the capability dimensions\n\n> *”However, as shown in Figure E.2 in the appendix, for some models for PC-2 and PC-3 there is a negative or flat relationship between training compute and model skill, as measured by the corresponding PC. Additionally, the performance on some of the benchmarks correlates negatively with some of the PCs, e.g. -0.62 for HellaSwag and PC-2 (which supposedly captures reasoning abilities). Given these discrepancies, I am not sure that the mental model of interpretable skills is really applicable for the PCs (even though they they seem to be useful for the scaling predictions).”*\n\nThis is a really good point!\n\nFirst, we would like to note that the negative scaling of PC-2 & PC-3 for some model families is due to their negative coefficients on certain benchmarks for decorrelation with PC-1. We agree that this may undermine the interpretability of these PCs to some extent.\n\nSecond, we would like to note that interpretability is *not a must* for the conceptual foundation of observational scaling laws, especially for making the scaling predictions that we mostly care about (as the reviewer has mentioned). What we need is the existence of low-rank capability measures that robustly correlate with compute measures and downstream capabilities and enable the scaling predictions from small to large scales, which have been validated in the paper.\n\nFinally, if interpretability is really needed in the extracted capability measures, we can apply [non-negative matrix factorization](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization) which enforces the non-negativity of coefficients. We have conducted the analysis and included the results in Figure 2 in the supplementary PDF:\n- In Figure 2(a), we visualize the component-wise benchmark coefficients. We observe that the coefficients are enforced to be non-negative and provide an interpretable decomposition. For example, we may view Component-1 as reasoning (that emphasizes GSM8K, HumanEval, and MMLU) and Component 4 as language understanding capabilities (that emphasizes (X)Winograde, HellaSwag), respectively.\n- In Figure 2(b), we visualize the scaling of NMF components with respect to log FLOPs for each family, using Component 1 as an example. We observe a smooth, positive scaling that generally holds across model families, which also hold for other components (omitted due to space constraints).\n\nWhile NMF offers enhanced interpretability and positive scaling properties compared to PCA, it also has notable limitations. Firstly, unlike PCA, NMF does not enforce orthogonality among its extracted components, as evident in the observed correlation between Components 3 and 4. Consequently, the coefficients assigned to each model across dimensions may not serve as independent measures of specific capabilities. Secondly, the ordering of NMF components lacks uniqueness and intrinsic physical meaning. This contrasts with PCA components, which are systematically ordered by their explained variances. The PCA approach provides an 'importance' measure for each dimension and allows for controlled trade-offs between representativeness and noise inclusion by adjusting the number of PCs used in the analysis.\n\nWe will adjust the claims and include the discussion in our future version of the paper.\n\n\n### Number of test points in agentic setups\n\n> *”However, the test sets used here are quite small (2 for AgentBench, 1 for AgentBoard), which makes these conclusions somewhat unreliable. Perhaps a different training-test split (e.g. 80/20 or 70/30) could be used (instead of 90/10) in order to obtain a larger test set. Would the results still hold?”*\n\nWe would like to clarify that the small number of test points is due to the small number of models evaluated by AgentBench and AgentBoard, and that a sufficient number of data points need to be used in the training set for fitting reliable scaling laws. We look forward to testing our current scaling predictions on more capable models once they are released, similar to what we have done in other setups (Figure 1(a) in the supplementary PDF). We will also adjust the claims about these results in the paper to acknowledge the limited number of test points.\n\nWe have also tested the 80/20 train/test split on AgentBench where there are more available data points. The results are included in Figure 1(b) in the supplementary PDF. We observe although the extrapolation performance tends to underestimate the performance to some extent, it still aligns with the overall observed trend.\n\n\n\n### Scaling extrapolation within a single model family\n\n> *“How many models per family would be needed in order for direct extrapolation from compute/model size for that family to produce similarly accurate results as the observational scaling laws here?”*\n\nThis is an interesting question! It is hard to do a comprehensive apple-to-apple comparison since most accessible model families contain only a few data points (e.g., Llama-2) or without public compute measures (e.g., GPT).  We did preliminary experiments with the Qwen1.5 family (with 7 models) and OPT (with 8 models), and tested them on tasks where the scaling predictions are non-trivial. The results are included in Figure 3 in the supplementary PDF. We find that typically at least 5 models are required for accurate extrapolation but the performance is highly dependent on the speciﬁc setup. For example, using five Qwen1.5 models achieves decent extrapolation on word unscramble but poor extrapolation on Persian QA.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bkhWXmh3WN", "reviewer": "Reviewer_JzvB", "summary": "The paper proposes a generalized class of scaling laws that encompass multiple model families of different sizes. These resulting scaling laws are capable of predicting “emergent” behaviors, complex agentic performance, and inference techniques in an extrapolative manner, as seen with GPT-4. The observed laws yield less error compared to standard FLOP-based laws.", "questions": "1. How does the method apply to Llama 3, where the amount of data is significantly larger than the standard Chinchilla optimal? Can we still achieve accurate predictions for Llama 3 models?\n2. For complex tasks, increasing amounts of compute are dedicated to inference time, especially for agentic benchmarks. Can the scaling law capture the scaling trends for inference time compute?\n3. Are there any insights on how to choose the PC-dimension to balance preventing overfitting with achieving better estimation?", "limitations": "Limitations are discussed. The authors claim they do not foresee any direct societal impact from their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with extensive experiments covering major benchmarks.\n2. A unified scaling law for various families and benchmarks is important to the whole community.\n3. The framework is general and demonstrates strong extrapolation performance.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We appreciate that you acknowledged that our work is “important to the whole community” with extensive experiments and strong extrapolation performance. We would like to address your remaining questions and concerns in the following response.\n\n\n### Applicability to Llama-3\n\n> “*How does the method apply to Llama 3, where the amount of data is significantly larger than the standard Chinchilla optimal? Can we still achieve accurate predictions for Llama 3 models?*”\n\nWe would like to clarify that our main results already included models that are significantly overtrained such as Llama-3 and Qwen, as shown in Figures 4 & 6. We have also further tested the newly released Llama-3.1 405B (FP8) with results included in Figure 1 in the supplementary PDF. We observe fairly accurate predictions on these models from observational scaling laws.\n\n\n### Generalization to inference-time scaling\n\n> *”For complex tasks, increasing amounts of compute are dedicated to inference time, especially for agentic benchmarks. Can the scaling law capture the scaling trends for inference time compute?”*\n\nThat is a good question! Our results on the agentic setups (with > 100 output tokens per step) and on the analysis of CoT (as another type of spending inference for capability) indicate the possibility of predicting model performance on inference-compute-intensive setups. Furthermore, we believe our observational scaling laws can be generalized to account for the inference time compute, e.g., by combining the capabilities measures of pretrained models and inference-time quantities (e.g., inference compute) to study the scaling properties of specific inference-time techniques with respect to certain model capabilities. This is an interesting direction, and we leave it for future work.\n\n\n### PC selection\n\n> *”Are there any insights on how to choose the PC-dimension to balance preventing overfitting with achieving better estimation?”*\n\nWe would like to note that we have conducted an extensive ablation study on the selection of the number of PCs in Appendix C.3. In our results, using three PCs generally leads to the best extrapolation performance across different setups. Furthermore, for a more fine-grained PC selection, we may utilize the total explained variances of the included PCs as a practical unsupervised selection criterion to balance representativeness and noise inclusion. We have also done some preliminary experiments, where we found selecting the smallest number of PCs (for preventing overfitting) with a total variance above 97% (for keeping representativeness) could select the best number in most of our setups.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEfEBzTjbn", "reviewer": "Reviewer_M7jM", "summary": "This paper demonstrates the correlation, known as scaling laws, between training FLOPs and large language models’ (LLMs) downstream task abilities. The authors decompose performance metrics to fit this “Observational Scaling Law” and confirm its validity across emergent capabilities, agentic capabilities, and post-training interventions.", "questions": "1. Why using PCA to decompose the main component ? This may need to explain why using PCA is work.   Do these main components have any corresponding physical significance?\n2.  I believe this paper aims to predict the performance of future large models without the need for training, but it only considers FLOPs. In reality, data quality and model size should also be considered. Is there any analysis of these two factors?", "limitations": "See in Question.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Sufficient validation. Including 21 model families and general, reasoning and agent-related datasets (open LLM  leadboard)\n2. Clear formalization.  Observational Scaling Law in Eq.3, Eq.7.", "weaknesses": "I did not find any specific weaknesses in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We appreciate that you acknowledged our paper provides sufficient validation and clear formalization. We would like to address your remaining questions and concerns in the following responses.\n\n### Choice of using PCA\n\n> *”Why using PCA to decompose the main component ? This may need to explain why using PCA is work. Do these main components have any corresponding physical significance?”*\n\nOur motivating hypothesis is the existence of a low-rank space of capability measures that captures the major model benchmark performance, and PCA is the most standard method for low-rank decomposition. The PC components extract the most significant, orthogonal directions in the capability space that capture the different capability dimensions of LMs (as illustrated in Figure 2b), making it a desirable choice for our purpose. We have also conducted an additional analysis with nonnegative matrix factorization (included in Figure 2 in the supplementary PDF) and a detailed discussion on the tradeoff in [our response to Reviewer V1YP](https://openreview.net/forum?id=On5WIN7xyD&noteId=qBHgXOUGb4).\n\n### Additional factors\n\n> *”I believe this paper aims to predict the performance of future large models without the need for training, but it only considers FLOPs. In reality, data quality and model size should also be considered. Is there any analysis of these two factors?”*\n\nWe would like to clarify that our predictions are based on the low-dimensional capabilities measures extracted from the benchmarks – which project models from different training recipes (e.g., trained on different data) to a unified space. This enables us to utilize a lot of public models without needing to deal with their training heterogeneousness – which is one of the biggest advantages of our observational scaling approaches. In our results, our scaling law predictions have already accommodated models trained from different data (e.g., specifically trained on multilingual data like BLOOM, or those on code like StarCoder) or with different sizes (e.g., Llama-2 7B, 13B, 70B). The model-specific factors like the data quality are incorporated into the “compute/data efficiency” by which each model family converts FLOPs to capabilities, as seen by how different model families behave differently in their scaling properties on the capability dimensions (Figure 3 in the paper).\n\nWe welcome any suggestions for additional critical analyses that you believe would further validate our findings, and we would be glad to conduct them for additional validation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OHk2zE5Spd", "reviewer": "Reviewer_oXqB", "summary": "They propose using PCA decomposition of the performance of a range of models across a number of benchmarks to form an observational scaling law, which effectively predicts downstream performance across several different model families, including predicting post-training interventions like chain-of-thought. They find that 3 principal components explain much of the variance on the LM benchmarks that they test. They can then use these principal components to 1) derive a llama2 effective compute quantity to use as a universal x-axis that works across all model families; and 2) for a new task and model family fit an observational scaling law based on the relevant principal components. They demonstrate that they are able to accurately extrapolate the performance of more capable models using weaker models on a number of standard benchmarks, including emergent tasks from Big-Bench. They also show that they can predict GPT-4's agent capabilities, and the effect of techniques like chain of thought and self-consistency using their method. Finally they demonstrate an approach for selecting a minimal sufficient set of models for deriving the observational scaling law.", "questions": "For a given take, are all models evaluated with the same prompts? It says you got some evals from Open LLM Leaderboard and others from EvalPlus and yet others from OpenLMLeaderboard. Are we sure all of these sources used the same evaluation procedure?", "limitations": "I think they mostly did a good job of acknowledging limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Developing a better understanding of the scaling of downstream capabilities in language models is an important problem and their paper represents very promising progress towards this goal.\n* Their experiments are pretty thorough, encompassing a range of models, tasks, and inference-time techniques.", "weaknesses": "* They show that their method can extrapolate predictions of stronger model capabilities using weaker models; however we don't get to see how far in advance they are able to make predictions. It would bo nice to see an ablation where the cutoff point used for fitting is pulled back so as to understand when extrapolation begins to fail. Moreover, they claim that they can predict emergence capabilities, however their cutoff point is conveniently just a little bit after performance begins to improve beyond chance, enabling their extrapolations to work. In the case where the cutoff point is pulled back further, are such emergent predictions still possible? I feel that demonstrating this would be important for being able to make the claim that prediction of emergence tasks is possible.\n* They don't give any kind of uncertainty measure or confidence interval for their predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We appreciate that you acknowledged that our work represents “very promising progress” towards an important problem with thorough experiments. We would like to address your remaining concerns with the following responses.\n\n\n### Ablation study on the cutoff point\n\n> *”It would bo nice to see an ablation where the cutoff point used for fitting is pulled back so as to understand when extrapolation begins to fail. Moreover, they claim that they can predict emergence capabilities, however their cutoff point is conveniently just a little bit after performance begins to improve beyond chance, enabling their extrapolations to work. In the case where the cutoff point is pulled back further, are such emergent predictions still possible?”*\n\nWe would like to clarify that our cutoff point was carefully selected to unify all the setups as much as possible to avoid the concerns of \"cutoff overfitting\" to each specific setup. We have also made the unified cutoff point as hard as we could by sweeping over the cutoffs (see Figure E.9), where we found some signals from data points above the random-guess regions are needed for reliable scaling predictions, especially for emergent capability setups – it is not possible to fit reliable scaling with only random-guess data points that extrapolate to the effective scaling region. We will explain this more explicitly in the main text.\n\nFor the ablation study on the cutoff point, we have already included a quantitative analysis in Figure E.9 where we measure the test error with varying cutoffs to control the training/test size. In addition, we have also included extra results of pushing back the cutoff point for each specific emergent capability task, included in Figure 1(a) in the supplementary PDF. We presented the results on three representative tasks due to space constraints and tested on the newly released **Llama-3.1 405B (FP8)** to assess the generalization to a larger scale. We observe that with a more aggressive cutoff setup, observational scaling laws can still provide reasonable predictions, even to a larger scale (Llama-3.1 405B) that was not available for testing before. We will include these additional results in our future version of the paper.\n\n\n### Confidence intervals\n\n> *”They don't give any kind of uncertainty measure or confidence interval for their predictions.”*\n\nWe have calculated the 95% confidence intervals for predictions from the non-linear regression models at each data point using parametric bootstrap. The results are included in Figure 4 in the supplementary PDF. We find that the observed data points mostly fall within these confidence intervals, though the intervals may be wide when there are very few effective data points above the random-guess region (e.g., in Persian QA).\n\n\n\n\n\n### Unification of setups\n\n> *”For a given take, are all models evaluated with the same prompts? It says you got some evals from Open LLM Leaderboard and others from EvalPlus and yet others from OpenLMLeaderboard. Are we sure all of these sources used the same evaluation procedure?”*\n\nYes, we have carefully checked the evaluation setups of these results to follow the same evaluation protocols (including prompts, few-shot examples, etc). For example, in the [documentation of Open LLM Leaderboard](https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about#reproducibility), they have detailed how to conduct the evaluation following the same protocol. For the models we have evaluated by ourselves, we have also carefully followed the same procedure.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2VSke3Xon", "reviewer": "Reviewer_DfgG", "summary": "This paper proposes observational scaling laws to align scaling laws of computing from different model families, which are trained on various recipes, by projecting model benchmark performance to surrogate compute. This enables applying scaling law analysis without actually training models. Using the observational scaling laws, the authors predict emergent abilities, including agent performance of models trained with much more compute.", "questions": "1. Eqn. 3 and 4 indicate that model performance scale with log compute in sigmoids. But Eqn. 5 shows linear projection between model performance and surrogates of log compute. This misalignment seems weird.\n2. What are the implications of observational scaling laws on evaluating pretraining data quality? Do different families of models show very different compute efficiency according to your results in aligning them?", "limitations": "The applicability of observational scaling laws seems to depend on training data distribution. The influence has not been discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper presents an elegant approach to resolve the misalignment of scaling laws due to different training recipes (e.g., pretraining data).\n2. This paper also shows the surrogate compute drawn from the proposed observational scaling laws can predict emergent abilities with sigmoid function.", "weaknesses": "1. The predictability of emergent abilities seems to be overclaimed. The provided results in Fig. 4, 6 and 7 support that the sigmoid function can well fit the relationship between downstream performance and (surrogate) compute. However, the feasibility of making reliable predictions on emergent ability is doubtful. The fitting data in Fig. 4  include points that have shown non-trivial performance and some even reached the inflection point.  And extrapolation results in Fig. 5 only scale 4x.\n\n2. Projecting the model benchmark performance to low dimensional surrogates of compute seems to imply the assumption that different model families mainly differ in the speed to obtain abilities and have little difference in the trade-off of different capabilities. Let's say A models are trained with 1% code + 99% text, while B are trained with 99% code + 1% text, do observational scaling laws still hold for these two model families?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and suggestions. We would like to address your remaining questions and concerns with the following responses.\n\n\n### Capability predictability and cutoff selection\n\n\n> *“The predictability of emergent abilities seems to be overclaimed… However, the feasibility of making reliable predictions on emergent ability is doubtful. The fitting data in Fig. 4 include points that have shown non-trivial performance and some even reached the inflection point”*\n\nWe would like to clarify that for the emergent capability setups, some data points above the random-guess region are needed since it is not possible to fit reliable scaling with only random-guess data points that extrapolate to the effective scaling region. Our cutoff point was carefully selected to unify all the setups as much as possible to avoid the concerns of \"cutoff overfitting\" to each specific setup, so there may be more non-trivial data points in some setups than others. We will adjust the claims and explain this point more explicitly in the paper.\n\nIn addition, we have also included extra results of pushing back the cutoff point for each specific emergent capability task, included in Figure 1(a) in the supplementary PDF. We presented the results on three representative tasks due to space constraints and tested on the newly released **Llama-3.1 405B (FP8)** to assess the generalization to a larger scale. We observe that with a more aggressive cutoff setup, observational scaling laws can still provide reasonable predictions, even to a larger scale (Llama-3.1 405B) that was not available for testing before. We will include these additional results in our future version of the paper.\n\n\n\n> *”And extrapolation results in Fig. 5 only scale 4x.”*\n\nWe would like to clarify that the X-axis is on the log-e instead of log-2 scale, so the extrapolation is about ~7.5. We commit to testing our current scaling predictions on more capable models once they are released, similar to what we have done in the emergent capability setups by adding the newly released Llama-3.1 405B (Figure 1(a) in the supplementary PDF), which will enable us to further check how much our predictions can scale.\n\n\n### Applicability to different model families & data distribution\n\n> *”Projecting the model benchmark performance to low dimensional surrogates of compute seems to imply the assumption that different model families mainly differ in the speed to obtain abilities and have little difference in the trade-off of different capabilities. Let's say A models are trained with 1% code + 99% text, while B are trained with 99% code + 1% text, do observational scaling laws still hold for these two model families?”*\n\nThe training data that different models are trained on will mainly determine their speed in converting to training compute to different capabilities (as shown in Figure 3), but not the observational scaling results. In our experiments, we have included models trained on mostly text (e.g., Llama), mostly code (StarCoder), multilingual data (BLOOM), and synthetic data (e.g., Phi), and observed consistent scaling predictions for these models.\n\n\n\n\n> *”What are the implications of observational scaling laws on evaluating pretraining data quality? Do different families of models show very different compute efficiency according to your results in aligning them?”*\n\nAs discussed above, the training data distributions will determine the efficiency in converting the compute to different capabilities. For example, the models trained on pure code data may demonstrate better compute scaling for the PC component corresponding to the programming capabilities than models trained mostly on natural language. In our results, different model families indeed demonstrate different compute efficiencies in converting to different PC components, as shown by the varying slopes of different scaling curves in Figure 3.\n\n\n\n> *”The applicability of observational scaling laws seems to depend on training data distribution. The influence has not been discussed.”*\n\nAs discussed above, we would like to clarify that the scaling predictions from observational scaling laws could accommodate models trained from different data training distributions, instead of “depending on” them, and our results have already accommodated many models trained from very different data distributions. The data distributions determine their compute efficiency in converting FLOPs to capabilities instead of from principal capabilities to more complex downstream performance (observational scaling).\n\n\n\n\n### Formulation misalignment\n\n> *”Eqn. 3 and 4 indicate that model performance scale with log compute in sigmoids. But Eqn. 5 shows linear projection between model performance and surrogates of log compute. This misalignment seems weird.”*\n\nWe could unify the two formulations, e.g., by adding a sigmoid relation in Eqn. 5, and it will probably also work reasonably well if we properly deal with the logit transformations during the floor (~0) and ceiling (~1) region. We choose the current formulation because it simply works well in practice and does not involve dealing with the potential ill-behaved, numerical issues of the logit transformation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their helpful feedback and suggestions. 

We are glad that the reviewers found our work offers a valuable contribution [V1YP] and very promising progress [oXqB] toward an important problem [oXqB, JzvB] with a comprehensive analysis [V1YP], extensive experiments [JzvB, M7jM], and interesting insights [V1YP]. 

We have provided extensive empirical results (included in the supplementary PDF) and responses to address reviewers’ remaining questions. Specifically, we have attempted to address the following major ones:
- [Reviewer oXqB, DfgG, V1YPm] Pushing back the cutoff points on emergent capability [oXqB, DfgG] and agentic tasks [V1YPm, DfgG]: In individual responses, we have clarified the reason for the use of our current cutoff selection. Furthermore, we have also done additional analysis on pushing back the cutoff selection on each setup for robustness check and tested the newly released **Llama-3.1-405B** for scalability check, included in Figure 1 in the PDF.
- [Reviewer V1YP, M7jM] The use of PCA for capability decomposition and its interpretability: We have analyzed an additional low-rank decomposition method (non-matrix factorization) in Figure 2 in the PDF and discussed its tradeoff against PCA in individual responses. 
- [Reviewer V1YPm] Scaling extrapolation within a single model family: We have analyzed the scaling prediction from FLOPs within a single family and included results in Figure 3 in the PDF.
- [Reviewer oXqB] Confidence intervals: We have calculated the 95% confidence intervals for predictions, included in Figure 4 in the PDF.

Please see detailed responses to specific questions below.

Meta-review: The paper introduces Observational Scaling Laws, a method to use benchmark scores of LLMs to infer how their performance would change if the amount of training compute was scaled, without actually having to train additional models. The proposed method applies PCA to the benchmark performances and finds that a small set of PCs (3) can explain much of the benchmark variation. The results show that the latent PCs can infer how models will perform on a new benchmark and how efficient models are at using their compute. 

All 5 reviewers and I are in agreement that the contributions are important and timely, and are supported by extensive experiments on 77 models and 21 model families. The additional results presented on the newly released Llama3.1 model during the rebuttal also strengthens the contribution and shows that this work has promise to predict capabilities of new models.

The authors were responsive to the reviewer's few concerns in their rebuttal and I urge the authors to update their paper with the promised changes, especially the addition of confidence intervals on the predictions and a separate section detailing the possible limitations of the work.
Decision: Accept (spotlight)