Observational Scaling Laws and the Predictability of Langauge Model Performance

Authors: yangjun  ruan
chris j maddison
tatsunori  hashimoto
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:08:52

Abstract: Understanding how language model performance varies with scale is critical to benchmark and algorithm development. Scaling laws are one approach to building this understanding, but the requirement of training models across many different scales has limited their use. We propose an alternative, observational approach that bypasses model training and instead builds scaling laws from ~100 publically available models. Building a single scaling law from multiple model families is challenging due to large variations in their training compute efficiencies and capabilities. However, we show that these variations are consistent with a simple, generalized scaling law where language model performance is a function of a low-dimensional capability space, and model families only vary in their efficiency in converting training compute to capabilities. Using this approach, we show the surprising predictability of complex scaling phenomena: we show that several emergent phenomena follow a smooth, sigmoidal behavior and are predictable from small models; we show that the agent performance of models such as GPT-4 can be precisely predicted from simpler non-agentic benchmarks; and we show how to predict the impact of post-training interventions like Chain-of-Thought and Self-Consistency as language model capabilities continue to improve.

Main paper: 
 introduction: 
 Language model (LM) scaling plays a central role in discussions of model capabilities and affects everything from the tasks they can perform to the effectiveness of post-training techniques such as Chain-of-Thought [99] . Due to this importance, understanding and predicting LM behaviors across scales, benchmarks, and algorithmic interventions is a major question for many researchers and engineers. Machine learning researchers may wish to understand whether their proposed algorithmic interventions remain effective in the face of future model scaling, while engineers and benchmark builders may wish to understand whether complex capabilities such as agentic abilities will scale predictably in the same way as existing LM benchmarks. Scaling laws [6, 36, 37, 44, 65] have been powerful tools for understanding the scaling trend of LMs, which have shown that LMs follow a precise power-law relationship between compute measures (such as training FLOPs) and downstream capabilities ranging from perplexity [37, 44] to benchmark performance [34, 35] . This power-law relationship has been used in a variety of waysincluding hyperparameter and architecture selection [9, 37, 44] as well as model capability forecasting [25, 66, 67] . Unfortunately, scaling analyses remain uncommon in many benchmarking and posttraining studies, as most researchers do not have the compute resources to build scaling laws from scratch, and open models are trained at too few scales (3) (4) (5) for reliable scaling predictions. We show that many scaling analyses, such as understanding complex LM capabilities (e.g., "emergent" behaviors) and post-training interventions, can be done with a lower-cost, higher-resolution, and broader-coverage alternative to the standard approach of training LMs across compute scales. The starting point of our work is the observation that there now exist hundreds of open models spanning a large range of scales and capabilities. While we cannot directly use these models for training compute to downstream capabilities (dashed line) by hypothesizing the existence of a lowrank space of LM capabilities that have a log-linear relationship with compute (center), and can be extracted directly from standardized LM benchmarks (left). This enables us to get low-cost, high-resolution scaling predictions of LMs' complex downstream capabilities from their observable standard benchmark metrics using nearly 100 publicly accessible LMs (left to right).compute scaling laws (as the training compute efficiency varies widely across model families), we might hope that there exists a more general scaling law that holds across model families. In particular, we hypothesize that the downstream performance of an LM is a function of a low-dimensional space of capabilities (e.g., natural language understanding, reasoning, and code generation), and that model families vary only in the efficiency by which they convert training compute to these capabilities. If such a relationship held, it would imply that there is a log-linear relationship from low-dimensional capabilities to downstream capabilities across model families (which would allow us to build scaling laws that leverage all existing models), as well as a log-linear relationship between training compute and capabilities within each model family (as in standard compute scaling) (Fig. 1 ). Through an analysis of standard LM benchmarks (e.g., Open LLM Leaderboard [8] ), we find a few such capability measures that have scaling relationships with compute within model families (R 2 > 0.9) (Fig. 3 ), and with downstream metrics across families. We call such relationships observational scaling laws as they predict complex downstream capabilities from simple observable quantities that we expect to scale with compute (like standardized benchmark performance)The ability to build scaling laws across a large number of existing LMs from their standard benchmark metrics has significant advantages in cost, resolution, and coverage: Observational scaling incurs no training cost, while leveraging models spanning a much larger compute range than any single model family. It also significantly increases the resolution of scaling laws by virtue of using more models, which is useful for studying nearly discontinuous phenomena like "emergent" capabilities. Finally, observational scaling can combine model families from heterogeneous sources with very different scaling properties (e.g., LLaMA [91] vs StarCoder [48] ) which allows us to study how different scaling strategies impact downstream performance and algorithmic interventions.Finally, we show that using observational scaling laws is low-cost and straightforward, as there are a few model families that are sufficiently representative to replicate many of our core findings (Sec. 5). By using these representative families, we find that future works can easily make scaling predictions on benchmarks and post-training interventions by evaluating only 10-20 models.We demonstrate the utility of observational scaling laws in three different settings that are challenging for compute scaling laws but are accurately predicted by ours: (i) Emergent capabilities (Sec. 4.1):We show that the high resolution of observational scaling laws reveals that the emergent behaviors of LMs [98] follow a smooth sigmoid, and can be predicted accurately using sub Llama-2 7B models.(ii) Agentic capabilities (Sec. 4.2): We show that the more complex capabilities of LMs as agents, as measured by AgentBench [57] and AgentBoard [61] , can be predicted with simple benchmark metrics.Our scaling law precisely predicts the GPT-4 performance using weaker models (sub GPT-3.5) and identifies programming capabilities as driving agent performance. (iii) Post-training interventions (Sec. 4.3): We show that our scaling laws can reliably predict the gains of post-training techniques, such as CoT [99] and Self-Consistency [97] at scale, even when they are fitted on weak models (sub Llama-2 7B). Finally, we show how to select only 10-20 representative models to replicate our core findings, making our scaling analyses more accessible with a low cost (Sec. 5). 
related work: 
 In this section, we briefly review the most relevant related work on downstream scaling laws and benchmark correlations. We include an extended related work discussion in Appx. C. Downstream scaling laws Scaling laws have been generalized beyond pretraining loss to analyze transfer learning [1, 35, 85] and downstream performance [15, 30, 34] across various domains. However, whether the LM downstream performance demonstrates a rapid "emergence" or is predictable with scaling laws remains debated [23, 27, 38, 39, 60, 79, 83, 98, 101] . Finnveden [25] and Owen [67] have investigated the use of linear and sigmoidal scaling laws, derived from pretraining loss or computational measures, to extrapolate the benchmark performance. Arora and Goyal [5] derived a theory characterizing how LMs' complex skills can be derived as a composition of base skills.Our work differs in that we build practical higher-resolution scaling laws to predict LM downstream performance using multiple model families and their observable standard benchmark metrics.Correlations between benchmarks Numerous works have studied the correlations between NLP benchmarks in vairous contexts [56, 68, 69, 71] . Most relevant to our work, Ilić [40] found that a single factor explains 85% of the variation on the Open LLM Leaderboard [8] and GLUE leaderboard [95] , while Burnell et al. [14] extracted three factors for LM capabilities that account for 82% of the variation on HELM [51] , aligning with our observations. Our work also observes such benchmark correlations and low-rank structures but is unique in utilizing these properties for the purpose of scaling predictions that can be used directly for benchmark and algorithm development. 
observational scaling laws: 
 In this section, we introduce our observational scaling laws that generalize the standard compute scaling laws (Sec. 3.1) . The key idea is to extract a low-dimensional capability measure for LMs from their observable benchmark performance (Sec. 3.2), which we find has a log-linear relationship with compute scale measures (Sec. 3.3 ) and can thus be used as surrogate "scale" for scaling analysis of complex LM capabilities (Sec. 3.4).Standard compute scaling In compute scaling laws, there is a hypothesized power-law relationship between models' compute measures C m (e.g., training FLOPs) and their errors E m (e.g., perplexity). Specifically, for a model m within a family f (e.g., Llama-2 7B, 13B, and 70B) we hypothesizeEQUATIONand if this linear fit is sufficiently accurate, we draw inferences about the performance of a model at future compute scales C > C by extrapolating this relationship. However, fitting such a scaling law can be tricky, as each model family f and downstream benchmark has its own scaling coefficients β f and α f . This means that scaling experiments, especially for post-training analysis, are often fitted on very few (3) (4) (5) models sharing the same model family, and any predictions are valid only for a specific scaling strategy used within a model family.Several studies [e.g., 25, 67] have generalized the functional form to analyze the scaling of LMs' downstream performance (where E m is normalized to [0, 1]) with a sigmoidal link function σ:EQUATIONObservational scaling In our work, we hypothesize the existence of a low-dimensional capability measure for LMs that relate compute to more complex LM capabilities and can be extracted from observable standard LM benchmarks, as illustrated in Fig. 1 . Specifically, given T simple benchmarks and B i,m the error of a model m on benchmark i ∈ [T ], we hypothesize that there exists some capability vector S m ∈ R K such that,EQUATIONfor θ f , ν f , β ∈ R K , α ∈ R, and orthonormal vectors γ i ∈ R K .)\TPEMRIHZEVMERGIVEXMS 4'%)\TPEMRIH:EVMERGI We can view Eq. ( 3) and Eq. ( 4) as a generalization of Eq. ( 2), since combining them can recover the original scaling relationships for a single model family. However, when there are multiple model families, S m serves as a shared, low-dimensional space of model capabilities from which all downstream metrics (E and B) are derived (as indicated by the absence of f in Eq. ( 3) and Eq. ( 5)), and model families only vary in their efficiency in converting compute into capabilities (Eq. ( 4)). One useful way of interpreting Eq. ( 4) is that θ f represents the compute efficiency of a model family f , and S m is the capabilities of model m expressed in terms of log-FLOPs for this model family. Finally, Eq. ( 5) ensures that these capabilities are not latent variables to be estimated for each model family, but are instead functions of fully observable properties (B). Since γ ∈ R K×T is orthonormal, we can linearly estimate Ŝm := γB m , which makes our scaling analysis significantly more robust. Importantly, this enables us to apply this to a large number of public models from heterogeneous sources, including those proprietary ones without any public information on C such as GPT-4.3.2 Identifying a Low-Dimensional Capability Space (Eq. ( 5))We validate the existence of a low-dimensional capability measure S that linearly relates to standard LM benchmarks B by showing that only a few principal components of B capture most of its variation (Eq. ( 5)). We demonstrate that the benchmark-model matrix B for a reasonable, broad set of benchmarks and models is low-rank and that Eq. ( 5) is a reasonable assumption.Models Since the benchmark-model matrix B can be directly measured for any LM, we include a large number of publicly accessible models for subsequent analysis. We collected a broad set of open LMs covering 21 model families (a collection of models across scales such as LLaMA-2 7B, 13B, 70B) and a total of 77 models. These encompass models trained from heterogeneous recipes, including standard training recipes like LLaMA [91] , those trained on synthetic data like Phi [50] , and models specifically trained on code data like StarCoder [48] . For this analysis, we consider only pretrained base models to avoid the complexities introduced by instruction tuning. We also include an analysis for instruction-tuned models that include proprietary ones like GPT-4 [66] in Appx. E.1, which demonstrates similar results. See Table D .1 for a detailed list of collected models.We collected a set of diverse benchmarks that assess various LMs' capabilities. These include popular aggregated benchmarks like MMLU [32] that assess the general knowledge of LMs. For more specialized evaluations, we included ARC-C [19] , HellaSwag [108] , Winogrande [77] for commonsense reasoning, GSM8K [20] for mathematical reasoning, HumanEval [16] for programming, TruthfulQA [53] for truthfulness, and XWinograd [64] for multilingual capabilities. We carefully collected these metrics from standardized evaluation protocols for comparability across LMs.In particular, we compiled them from standardized leaderboards, like the Open LLM Leaderboard [8] and EvalPlus [55] , when available. Otherwise, we used standardized libraries such as the LM Eval Harness [28] to evaluate the LMs. See Appx. D.1 for full details of our data collection pipeline.PCA analysis After obtaining the benchmark metrics for the LMs, we addressed potential missing values (less than 1% of all data), which may have occurred due to evaluation failures, by using PCA imputation. Subsequently, we applied PCA to extract the principal components of the evaluation metrics as the "principal capability" (PC) measures S (additional details in Appx. D.3). PC measures are low-dimensional We observe that the extracted PC measures are predominantly low-rank, with the top 3 PCs explaining ∼ 97% of the variance, which supports a low-dimensional representation of benchmarks B (Fig. 2a ). Surprisingly, we find that the first PC alone explains nearly 80% of the variation in LM capabilities. Taking a closer look at these PCs, we find that these capability measures represent interpretable directions in which LMs capabilities may naturally vary as a function of scale (Fig. 2b ). Specifically, PC-1 represents the "general capability" as a weighted average of all metrics; PC-2 corresponds to the "reasoning capability", emphasizing mathematical and coding benchmarks; and PC-3 primarily reflects the "programming capability". These findings suggest that many simple LM capabilities (as covered in our benchmarks) can be expressed as a linear combination of just a few "principal capabilities" S.0SK10*034W) 4' 6 2 ! ]!\ 0PEQE 0SK10*034W) 4' 6 2 ! ]!\ 0PEQE 0SK10*034W) 4' 6 2 ! ]!\ 5[IR 0SK10*034W) 4' 6 2 ! ]!\ 5[IR 0SK10*034W) 4' 6 2 ! ]!\ *EPGSR 0SK10*034W) 4' 6 2 ! ]!\ 4]XLME 0SK10*034W) 4' 6 2 ! ]!\ &0331 0SK10*034W) 4' 6 2 ! ]!\ +482IS. 0SK10*034W) 4' 6 2 ! ]!\ 348 0SK10*034W) 4' 6 2 ! ]!\ <+01 0SK10*034W) 4' 6 2 ! ]!\ 'SHI0PEQE 0SK10*034W) 4' 6 2 ! ]!\ 7XEV'SHIVMeasures as Surrogate Scale Measures (Eq. ( 4))We now show that the PC measures S scale log-linearly with training FLOPs within each model family, and can thus be interpreted as a cross-family generalization of compute C. We discuss some additional applications of PC measures as a smooth cross-family evaluation metric in Appx. B.Setup We collected all available information about training FLOPs on each of our models, analyzing papers and other public information to identify model size N and pretraining data size D. For the models where we were able to identify this information, we used the simple estimate of C ≈ 6N D to obtain model training FLOPs [44] . See Table D .1 for our collected compute measures. PC measures linearly correlate with log-compute measures Fig. 3 illustrates the correlation between the top PC-1 measure with the corresponding training FLOPs for models within each model family. We find that for each model family with controlled training recipes and comparable compute scale measures, the LMs' PC-1 measure linearly correlates with their log-training FLOPs (with R 2 > 0.9). This linear correlation holds across a broad range of model families including those specifically trained on multilingual data like BLOOM [100] or those on code like StarCoder [48] . It also generally holds for lower-ranked PCs such as PC-2 and PC- 4) and (5) , in which we hypothesized that models share the same capability space and a log-linear relationship determines the efficiency by which each model family converts their compute into these principal capabilities.3.4 Fitting Observational Scaling Laws (Eq. ( 3)) Fitting regression with PC measures Given a certain downstream error metric E normalized to [0, 1] that measures certain LM capabilities, we slightly generalize Eq. ( 3) toEQUATIONwhere β ∈ R K and α ∈ R are the regression weights and bias, h ∈ [0, 1] is the sigmoidal scale that accounts for the potential discrepancies in the floor performance. We fit the regression with ordinary least squares and restrict h ∈ [0.8, 1.0], which results in h * = 1 in most experiments.Defining interpretable compute-like measures Recall that the core component of our scaling law is the fitted linear transformation P m := β * S m + α * that maps the extracted PCs into a scalar capability measure for a target downstream metric. While this is perfectly acceptable for prediction, our scaling analysis would be more interpretable if we expressed capabilities in units of FLOPs rather than an arbitrary scalar measure. We can achieve this by utilizing the fact that for a single family f , our observational scaling law reduces to a compute scaling law (Eq. ( 3) & Eq. ( 4)). Specifically, we note that when Eq. ( 4) holds exactly, we have that for a model m within a family f ,EQUATIONwhere w f = β * θ f and b f = β * ν f + α * . This implies a linear correlation between the scalar capability P m and the compute log(C) for models within a specific family on a downstream task (see empirical validation in Fig. E .3). Since θ f and ν f are unknown a priori, we can fit these coefficients w f , b f via linear regression from log(C) to P using models from the specific family f .In the multi-model family case, we can map all models to a shared, FLOPs-based capability measure of a specific family f . The core idea is to represent each model's capabilities by the following hypothetical: "how many FLOPs ( Cm,f ) would it take for a model in a family f to match a model m". We call Cm,f the f -equivalent FLOPs for model m, as it represents the performance of model m relative to models in the reference model family f . This measure can be computed fairly easily asEQUATIONobtained from solving for log(C m ) in Eq. (7) . Throughout the remainder of this work, we apply this scalar transformation where we pick Llama-2 [92] as the reference family f , and so the x-axis of all of our plots can be interpreted as "model capabilities, as measured in units of Llama-2 FLOPs". 
validating observational scaling laws: 
 We evaluate the usefulness of observational scaling laws by showing that they accurately predict the scaling behaviors of LMs over complex, hard-to-predict phenomena (like emergent phenomena and agentic abilities) and help estimate the value of techniques such as Chain-of-Thought.To ensure that our scaling laws are actually predictive and that we are not simply overfitting through various choices in scaling law construction and hyperparameters, we design our experiments to have systematic holdout sets and robustness checks. We have also preregistered our predictions for future models after the initial release of the paper as a test of whether our scaling laws overfit current models. We release our code including the implementation and collected data at https://github.com/ryoungj/ObsScaling. Details in scaling law fits For extracting PC measures, we fixed the number of PCs K = 3 as it covered ∼ 97% of the variation in benchmark performance and it consistently yielded the best performance across most of our experiments, see Appx. E.4 for robustness checks on PC selection.For the capability-equivalent scale transformation, we used the Llama-2 [92] as the reference model family as it is currently the most representative and widely used open model in the community. For better interpretability and visualization, we used the accuracy metric, typically defined as Y = 1 -E, for fitting the scaling laws and making the plots. Holdout validation To validate our observational scaling laws, our primary objective is to assess how accurately the scaling laws fit the available data and extrapolate from smaller-scale, less capable models to larger-scale, more powerful models. We validate this through systematic holdouts for the test set, where we split available models into weaker and stronger ones based on both scale or capability (e.g., FLOPs or accuracy). We used the weaker models to fit the scaling law and evaluated the extrapolated predictions on the stronger ones. As baselines, we compare our scaling predictions to using existing compute-based scale measures like training FLOPs and model size. We used the mean squared error (MSE) on the test set as our main evaluation measure, which is comparable as the target range is always normalized (0 to 1). Preregisteration of predictions In the initial release of our paper (May 2024), we have preregistered our scaling predictions for future models (see preregistered functional forms in Appx. E.9) and committed to updating the manuscript on ArXiv with our prediction results after 4 months. We have assessed these predictions on new models released after the initial paper release, collected as of September 1st 2024, including most capable open models to date such as Llama 3.1-405B [24] and Qwen2-72B [105] (see the full collected model list in Appx. D.1.1), resulting in an additional test set of 20 models for robustness checks. The results are included in Fig. 4 , and additional results on other tasks and new benchmarks are included in Appx. E.3. Our preregistered predictions also accurately extrapolate to new models released after the initial paper release, including Llama-3.1-405B [24] . Four tasks from BigBench [82] , which are identified as "emergent" in [98] , are used for illustration.Recent works have argued that many LM capabilities are "emergent" and cannot easily be predicted from small-scale models [27, 98] . There have been ongoing debates about whether these capabilities are truly discontinuous and whether the discontinuity is an artifact of the metric used [23, 39, 60, 78] or lack of high-resolution data points [38] . The debate has been complicated by the fact that existing scaling analyses (including the original ones in Wei et al. [98] ) have very few points [38] . When there are only 5 models across many orders of magnitudes of scale, phenomena can appear to be discontinuous, even if the underlying phenomenon is a smooth but rapidly varying sigmoid. We show that the higher resolution of observational scaling laws allows us to clearly see smooth sigmoidal curves in phenomena that were identified as emergent in Wei et al. [98] , and even more surprisingly, we can often accurately forecast the transition points where models go from nearrandom to high performance using only models whose performance is only slightly above random. Our findings validate the observational approach to scaling laws and provide evidence that higherresolution scaling laws could help us better understand scaling phenomena for LMs.Setup We tested on four BigBench [82] tasks that were labeled as "emergent" in Wei et al. [98] , including two arithmetic tasks (3-digit subtraction and 2-digit multiplication) and two non-arithmetic tasks (word unscramble and Persian QA). Additional results on more tasks covering Wei et al. [98] are included in Appx. E.5. For the models, we included base pretrained models following the approach of Wei et al. [98] . For non-arithmetic tasks, we used the default FLOPs cutoff. For arithmetic tasks, we found that this cutoff resulted in an excess of training data near perfect performance (see results in Fig. E .13), making the prediction tasks trivial. Consequently, we reduced the cutoff to a quarter of the default value and also excluded GSM8K (which may be a superset of arithmetic tasks) from our base metrics B to make the tasks more challenging.Prediction results Fig. 4 shows our prediction results using our PC measures as well as the baseline of predicting performance based on training FLOPs. We find that these capabilities can be accurately predicted using our PC measures, even when only using models that perform poorly. In contrast, using training FLOPs results in significantly poorer extrapolation on the test set and fits on the train set, as indicated by the much higher MSE values. There is significant interest in building autonomous agents using LMs, with notable examples including AutoGPT [75] , Devin [46] , and SWE-agent [106] . Although the performance of these agents still falls far below human-level on challenging real-world tasks [43, 62, 110] , there is a belief that future models at larger scales will significantly enhance these agents' capabilities. However, there is a significant uncertainty about whether existing models that are trained for language and code capabilities will transfer well to agentic tasks that require taking actions over many rounds.In this section, we utilize our observational scaling laws to analyze the scaling properties of LMs' agentic capabilities w.r.t. their backbone model capabilities and show that agent performance is highly predictable from simple benchmark metrics.Setup We tested on two standardized agent evaluation benchmarks, AgentBench [57] and Agent-Board [61] , each is a collection of diverse tasks for evaluating LMs' agentic capabilities. For both benchmarks, we utilized their provided aggregated metrics on all tasks for prediction. Specifically, we used the "Overall Score" on AgentBench, which is a weighted average of scores across all tasks (denoted as "OA" there), and the "Average Success Rate" on AgentBoard. We included models that have been evaluated on each benchmark, which encompasses both open instruction-tuned models like LLaMA-2-Chat [92] , and proprietary models like GPT-4 [66] and Claude-2 [3] , see Table D .2 for a complete list of models. We followed the same procedure to collect standardized benchmark metrics B for instruction-tuned models, see Appx. D.1.2 for details. Notably, since compute scale measures are not available for proprietary models, only our observational scaling laws apply here and not compute scaling laws. The default FLOPs cutoff does not apply either, and thus we held out the top 10% performing models on each agent benchmark as the test set to simulate weak-to-strong predictions, which included GPT-4 and Claude-2 on AgentBench and GPT-4 on AgentBoard. Prediction results Fig. 5 illustrates the prediction results with our observational scaling laws using PC measures. We find that on both agent benchmarks, the performance of held-out models (GPT-4/Claude-2) can be accurately predicted from models with much weaker performance (> 10% gap). This indicates that the more complex agentic capabilities of LMs are well-correlated with and predictable from their base model capabilities, suggesting the promising scaling properties of LM-based agent capabilities as backbone LMs continue to scale up.Interpreting the capability dimensions In Fig. 5c , we visualize the weights assigned to the base evaluation metrics on both benchmarks, which are derived from the regression weights fitted on PC measures and applied with learned PCA transformation, i.e., β γ. We observe that the fitted weights assign significant importance to programming capabilities (HumanEval) on both benchmarks, underscoring its significance in defining the agentic capabilities of LMs. The weights also emphasize general knowledge (MMLU) on AgentBench, and reasoning capabilities (GSM8K) on AgentBoard, suggesting that these capabilities may also be important for LMs' agentic capabilities.When researchers propose a new prompting or post-training technique to improve a pretrained model, how can we know whether these gains will persist across models and scales? Systematic scaling analyses have been rare due to the small number of models within a single model family. Moreover, some recent works have argued that certain interventions, such as Chain-of-Thought [99] , behave in an emergent way that is not predictable from smaller models [98] . Using observational scaling laws, we show that it is possible to make relatively accurate predictions on the effectiveness of techniques such as Chain-of-Thought (CoT) [99] and Self-Consistency (SC) [97] as model scale increases. We focus on these post-training interventions in particular, as they are sometimes discussed as examples of post-training interventions that require scale to be effective [98, 99] .EQUATIONOur approach to quantifying the scaling properties of post-training is straightforward: we fit one observational scaling law using base model performance on a target benchmark (e.g., GSM8K fewshot), and then fit another on the performance of models with the post-training intervention (e.g., GSM8K w/ CoT). Each of these fits produces a sigmoidal scaling curve as a function of log( Cf ), and the relative gaps as a function of log( Cf ) indicates the scaling efficiency of the intervention.Setup We tested on GSM8K with CoT and SC as post-training techniques and included additional results on BigBench-Hard [83] with CoT in Appx. E.6. As with our study on emergent phenomena on arithmetic tasks, we excluded GSM8K from the base metrics B to avoid making the prediction tasks trivial. We included all the pretrained base models listed in Table D .1 including those specifically trained for code data and applied the default FLOPs cutoff for holdout validation. For CoT, we followed Wei et al. [99] and compared CoT prompting using eight reasoning examples with naive prompting using only few-shot examples in the greedy decoding setting. For SC, we sampled five CoT reasoning paths at temperature 0.7 to aggregate the final answers following Wang et al. [97] and compared it with a single sampled CoT answer. Prediction results Fig. 6a shows the scaling predictions for CoT and SC using observational scaling laws. We find that the performance with (CoT, CoT + SC) and without (Naive) post-training techniques for stronger, larger scale models can be accurately predicted from weaker, smaller scale models. In contrast, predictions based on compute scale measures like model size and training FLOPs are less reliable as seen in Fig. E.15. Notably, the scaling trends between the two techniques differ; CoT shows a much more pronounced scaling trend compared to Self-Consistency w/ CoT. Interpreting the capability dimensions Another advantage of observational scaling laws over scaling laws constructed on single families is that we can visualize the capabilities that are important to the post-training intervention. Fig. 6b visualizes the fitted regression weights β, mapped to the space of base capability benchmarks B via β γ. We clearly see that when we go from Naive to CoT, there are significantly higher weights placed on MMLU and HumanEval -meaning that scaling models in a way that enhances general knowledge (MMLU) and code (HumanEval) leads to greater gaps between CoT and the baseline, while improving along commonsense, such as Winogrande does not necessarily lead to improvements at scale. These analyses can inform how different post-training interventions affect different scaling recipes -such as code models vs general-purpose LLMs. 
selecting low-cost model subsets for practical scaling analyses: 
 Although our observational scaling law incurs no training cost, it still requires evaluating our benchmarks and post-training methods on a larger number of models. To make observational scaling analyses more broadly accessible, we identify a small set of representative models that maintain high prediction accuracy while significantly reducing the evaluation cost.Method More specifically, we consider the constrained optimization problem of identifying the optimal set of models to choose for a regression problem, subject to the constraint that we select a model subset M of at most M max models from the set of all models M a . To define optimality, we turn 2YQFIVSJ7IPIGXIH1SHIPW 3ZIVEPP17):3TXMQEPMX]7IPIGXMSR3YVW 6ERHSQ7IPIGXMSR %PP1SHIPW(a) Prediction error vs model counts0SK100PEQE)UYMZ*034W) %GGYVEG] 'S8 2EMZI+VIIH] ]!WMKQSMH\ 'S8+VIIH] ]!WMKQSMH\ 8VEMR 8IWX 0SK100PEQE)UYMZ*034W) %GGYVEG] 7IPJ'SRWMWXIRG] 'S87EQTPI ]!WMKQSMH\ 'S87'7EQTPIW ]!WMKQSMH\(b) Prediction results with only 12 models chosen by V-optimality Figure 7 : (a) Selecting the model subsets with our V-optimality criterion leads to significantly lower errors than random selection, and quickly converges to the errors of using the full set of models. (b) Using 12 (out of 47) models selected by our method maintains the overall prediction accuracy.to the theory of optimal experimental design, which states that for linear regression with a fixed design X and subset M, the expected prediction error from using the subset X M is Tr(X X X M X M -1 ).This gives a straightforward objective achieving the V-optimality [70] :EQUATIONwhere S ∈ R M ×K is the model-capability matrix obtained from our PC analysis. We conduct a structured, exhaustive search over the 21 model families where we include or exclude entire model families under the budget constraint, as we believe these selected models are more interpretable.Validation We followed the setup in Sec. 4.3 for validating our selection method, as this represents the most likely application scenario for our observational scaling laws by practitioners. Our objective is to replicate our scaling analysis (using a full set of 47 models) in Fig. 6a using a small subset of models selected by our method. In Fig. 7a , we compute the geometric average of test MSEs on all prediction tasks (Naive, CoT, CoT + SC) as the evaluation metric for different selection methods. We find that our V-optimality selection method significantly outperforms random selection and quickly converges to the prediction performance of using the full set of models. In Fig. 7b , we show that using only a small subset of 12 models selected by our method, the fitted scaling curves already effectively capture the scaling trends of different post-training methods, in contrast to randomly selected models (Fig. E .18). To facilitate future scaling analyses at alow cost, we provide a reference list of models selected with our method under different budget constraints in Table E .1. 
conclusion, limitations, and future work: 
 We have presented observational scaling laws that generalize existing compute scaling laws to handle multiple model families using a shared, low-dimensional capability space. Using this approach, we show that we can build low-cost, high-resolution, and broad-coverage scaling laws that allow us to make accurate predictions for many complex scaling phenomena, such as emergent behaviors, agentic capabilities, and the value of post-training interventions. We provide concrete practical prescriptions for practitioners to perform similar scaling analyses in the hopes of encouraging more quantitative, scaling-law-based approaches to designing benchmarks and post-training methods.Limitations and future work Finally, we discuss some limitations of our approach and findings: Firstly, observational scaling laws are primarily applicable to post-training scaling analyses and do not directly translate to pretraining scenarios in the same way as standard compute-based scaling laws. Secondly, our study mostly focuses on the scaling behavior of model capabilities measured through few-shot prompting or basic prompting techniques (such as CoT, self-consistency, or simple agent scaffolding). Extending our approach to other post-training setups, including scenarios involving fine-tuning or more intensive inference-time computation [12, 81] , would be valuable. Thirdly, while we have demonstrated that our observational scaling analyses can provide meaningful insights into improving particular models' complex capabilities, a promising direction for future work would be to apply the findings from our approach, such as by deriving surrogate measures for model complex capabilities that can be used to optimize models directly and efficiently. Lastly, our assumptions do not account for potential benchmark contamination (where particular benchmark data leaks into model training) or the heterogeneity within model families (where models within the same family may have varying compute efficiencies and scaling behaviors). Investigating the impact of these assumptions on our approach would be an interesting avenue for future research. 
 
 Appendix: In Algorithm A.1, we include the detailed algorithm for fitting the observational scaling laws as described in Sec. 3. ). In contrast to compute scale measures, it also enables the comparison of models from heterogeneous sources on a unified scale. F : B → h * σ β * γB + α * or Cf → h * σ w * log( Cf ) + b * 2% 2% 0SK10*034W) 4' 6 2 avg ! 0PEQE 0PEQE 0PEQE 5[IR 5[IR 1MWXVEP 1M\XVEP =M +IQQE *EPGSR 4LM 4]XLME &0331 +482IS. 348 148 <+01 'SHI0PEQE 7XEV'SHIV 7XEV'SHIV (IIT7IIO'SHIV0SK 10 )UYMZ*034W) %GGYVEG] 'S8 0PEQE ]!WMKQSMH\ 4LM ]!WMKQSMH\ +IQQE ]!WMKQSMH\ 'SHI0PEQE ]!WMKQSMH\ (IIT7IIO'SHIV ]!WMKQSMH\Our work validates the hypothesis that there is a low-dimensional space of LM capabilities that captures their scaling behaviors and can be measured via a low-rank decomposition of existing LM benchmarks -which interestingly connects to the item response theory in psychometrics [58] that models humans' test performance by their fundamental abilities such as general intelligence. While the majority of our work focuses on applications to scaling laws and predictions, we also find that the shared, low-dimensional capabilities could potentially be used as an evaluation metric and optimization target for LMs. We discuss some of these possibilities here. PC-1 as a smooth capability measure with high dynamic range Many existing benchmarks suffer from a limited dynamic range: they either saturate quickly for large models (e.g., HellaSwag, Winogrande) or have completely random performance for small models (e.g., MMLU, GSM8K), see Fig. E .4 for the behavior of each benchmark. In contrast, we find that PC-1 is a smooth capability measure that can be used to compare LMs across many (at least 5) orders of magnitude. This allows us to compare models from heterogeneous sources and of extremely different capabilities on a single, unified scale (Fig. B .1). We believe that the high dynamic range of PC1 may make it suitable as an optimization target for pretraining, where architecture or data interventions can be benchmarked against PC-1 at small scales and validated at large scales. Training data efficiency measurements using PC-1 Extending these ideas further, since PC-1 serves as a unified measure of capabilities, it may serve as a good way to compare compute efficiencies across many model families. In We find that LMs benefit similarly from CoT, but that Phi is once again an outlier in its behavior: it benefits from CoT much earlier than other model families, but scales less rapidly. Similarly, models specifically trained on code (DeepSeek-Coder), also demonstrate an earlier transition but less rapid scaling compared to models trained with standard protocols. The distinct behavior of Phi/DeepSeek-Coder relative to other models indicates the importance of pretraining data in determining model scaling behaviors. While we did not specifically focus on these types of analysis in this work, we hope that our approach enables future works to gain further insights into differences between LM training recipes and their scaling behavior.Compute scaling laws In standard scaling laws [6, 34-37, 44, 65] , the "scale" is defined by the compute resources allocated to training LMs, such as the number of training FLOPs C, model parameters N , and training tokens D. Scaling laws are typically formulated as a power-law relationship between LMs' cross-entropy loss L and their compute scale measures. Common functional forms include L(N, D) = a N α + b D β + e [37, 65] or 34, 44] , where C ≈ 6N D [44] for the Transformer [93] . The parameters {α, β, a, b, e} or {γ, c, h} are fitted by training LMs across different compute scales, varying N and/or D, and measuring their loss. Our work differs from compute scaling laws in our goals -compute scaling aims to understand the scaling properties of pretraining, and thus focuses on a single model family and relates downstream performance to directly controllable quantities such as training compute. In contrast, we are interested in scaling laws for downstream, post-training performance, which leads us to consider scaling laws across model families and use more directly observable capability measures than compute. Downstream scaling laws Scaling laws have been generalized beyond pretraining loss to analyze transfer learning [1, 35, 85] and downstream performance [15, 30, 34] across various domains, see Villalobos [94] for a comprehensive review. In particular, there has been evidence suggesting that the few-shot performance of LMs on downstream benchmarks is closely tied to compute measures like model size [13] , but whether this is predictable with scaling laws remains debated. Extensive research has explored the difficulties of predicting benchmark performance due to their appearing rapid "emergence" [27, 83, 98] , while recent works argued the discontinuity is due to the metrics used [60, 79] or the lack of data points [38] (see Anwar et al. [4] for a survey on this topic). Finnveden [25] and Owen [67] have investigated the use of linear and sigmoidal scaling laws, derived from pretraining loss or computational measures, to extrapolate the benchmark performance. Notably, Owen [67] also utilized publicly available LMs from different families to fit their compute scaling laws despite the potential discrepancies in their compute efficiencies. Recent studies have also more extensively investigated the correlations between the pretraining loss and downstream performance of LMs [39, 101] , aiding in the understanding of downstream scaling [26] and emergent capabilities [23] of LMs. On the theory front, Arora and Goyal [5] derived a theory characterizing how performance on complex skills of LMs can be derived as a composition of base skills. While our work shares similar goals in that we aim to understand the downstream, post-training performance of models, we differ in our approach in that we aim to build practical higher-resolution scaling laws using multiple model families and their observable standard benchmark metrics. Correlations between benchmarks Numerous works have investigated the correlations between different benchmarks across various contexts. Extensive research has explored the relationship between the out-of-distribution performance and in-distribution performance of machine learning models [63, 72, 73, 84, 104] . In the realm of NLP and LM benchmarks, Qiu et al. [71] , Torregrossa et al. [90] found that different evaluations and metrics for word embeddings are highly correlated, and Liu et al. [56] observed a strong correlation between question-answering benchmarks. Moreover, Perlitz et al. [68] , Polo et al. [69] observed strong correlations between samples within various LM benchmarks and utilized this observation to develop more efficient benchmarks. Most relevant to our work, Ilić [40] found that a single factor explains 85% of the performance on the Open LLM Leaderboard [8] and GLUE leaderboard [95] , while Burnell et al. [14] extracted three factors for LM capabilities that account for 82% of the variation on the HELM benchmark [51] , aligning with our observations. Our work also observes such benchmark correlations and low-rank structures but is unique in utilizing these properties for the purpose of scaling predictions that can be used directly for benchmark and algorithm development. [92] , Llama [91] , Llama-3 [24] , Qwen1.5 [88] , Qwen [7] , Mistral [41] , Mixtral [42] , Yi [107] , Gemma [86] , Falcon [2] , Phi [50] , Pythia [10] , BLOOM [100] , GPT-Neo/J [11] , OPT [109] , MPT [89] , XGLM [54] , CodeLlama [76] , StarCoder [48] , StarCoder2 [59] , DeepSeek-Coder [31] . For preregistration test, we have collected an additional set of 20 models covering 8 families released after May 2024 and as of September 1st 2024, including Llama-3.1 [24] , Qwen2 [105] , DeepSeek V2 [80] , Gemma-2, [87] , Jamba [52] , Yi-1.5 [107] , etc. For each model, we collected their available metadata including the number of model parameters N and the amount of pretraining tokens D by analyzing papers and other public information. We then estimated the training FLOPs C using the simple estimate of C ≈ 6N D [44] for each model. Note that for models that were continually pretrained on additional data such as CodeLlama, we used the sum of the pretraining tokens and the additional continual pretraining tokens to estimate D. See Table D .1 for the collected metadata of these models.L(C) = c C γ + h [We collected a set of diverse benchmarks that assess various LMs' capabilities, including MMLU [32] , ARC-C [19] , HellaSwag [108] , Winogrande [77] , GSM8K [20] , TruthfulQA [53] , and XWinogrande [64] , HumanEval [16] . For MMLU, ARC-C, HellaSwag, Winogrande, GSM8K, and TruthfulQA, we primarily sourced results from the Open LLM Leaderboard1 [8] , with updates current as of May 6th, 2024. When there were missing benchmark results, we followed the standardized evaluation protocols of the Open LLM Leaderboard and used the LM Eval Harness [28] library to evaluate the LMs. For XWinogrande, we used the LM Eval Harness library to evaluate the models with 5-shot examples. For HumanEval, we primarily used the EvalPlus [55] library and followed their standardized protocols for evaluation, and sourced the results from the EvalPlus leaderboard2 when available. We used the 'Base Tests' results provided by EvalPlus for all the models. See Table D .1 for all collected benchmark results.We collected the set of instruction-tuned models that have been evaluated on the AgentBench [57] and AgentBoard [61] benchmarks. These include models like GPT [66] , Claude [3] , Llama-2-Chat [92] , Codellama-Instruct [76] , Mistral-Instruct [41] , Vicuna [17] , Deepseek-LLM-Chat [9] , Lemur-Chat [103] , OpenChat [96] , WizardLM [102] , Guanaco [22] , Koala [29] , Dolly-v2 [21] , OpenAssistant [47] . We followed the same procedure in Appx. D.1.1 to collect the metadata of open models, while for proprietary models these metadata were not publicly available. Note that we only counted the pretraining tokens (and the continual pretraining tokens when applicable) for D and excluded the data for instruction-tuning or additional finetuning, as these are typically only a small fraction of the total data and are nuanced to estimate due to the complexities in data curation for instruction-tuning. See Table D .2 for the collected metadata of these models. Benchmark collection & evaluation For instruction-tuned models, we also included standard LM evaluations such as MMLU [32] , ARC-C [19] , HellaSwag [108] , Winogrande [77] , TruthfulQA [53] , GSM8K [20] , and HumanEval [16] , and we followed the same protocols in Appx. D.1.1 for evaluating open models. For proprietary models like GPT and Claude, it is more nuanced to evaluate them with a unified protocol (e.g., due to the lack of access to likelihood scores), so we collected the official results from their respective papers and documentation for all standard benchmarks (except for HumanEval, which we were able to evaluate using the EvalPlus library). Additionally, we collected Elo scores from the Chatbot Arena3 [18] which assess instruction-following capabilities of these instruction-tuned models (as of February 2nd, 2024) for reference, we did not utilize this metric for our downstream predictions. See Table D .2 for all collected benchmark results.For all downstream tasks of pretrained base models included in Sec. 4.1 and Sec. 4.3, we used the LM Eval Harness [28] library to evaluate all the models. For the "emergent" capability tasks in Sec. 4.1, we applied likelihood-based evaluation [13] tasks in Sec. 4.3, we used the same evaluation protocol as the original papers, as described in the main paper. For agentic capability tasks of instruction-tuned models in Sec. 4.2, we directly sourced the results from the AgentBench [57] and AgentBoard [61] leaderboards and scaled the metrics to [0, 1]. In Fig. E .1, we conducted a PC analysis for instruction-tuned models (see the model list in Table D.2) following exactly the same procedure as Fig. 2 . We find that the extracted PC measures for instruction-tuned LMs follow similar patterns as pretrained models and exhibit an even more significant low-rank structure, with the top 3 PCs explaining about 98.6% of the variance in the benchmark performance. , we show that using a single benchmark metric as LM capability measures amy suffer from a limited dynamic range. In particular, they may either saturate quickly for large models (e.g., HellaSwag, Winogrande) or provide random readouts for weak models (e.g., MMLU, GSM8K). EQUATIONRecall that we defaulted to use 3 PC measures for all of our prediction tasks. Here we provide additional analysis on the impact of using different numbers of PCs on the prediction performance and validate the robustness of our choice. In particular, we compare the fitted curves and prediction performance of using 1-4 PCs on all our tasks. The results are in .9 for post-training analysis, "emergent" capability, and agentic capability tasks, respectively. Our results indicate that using more than 2 PCs leads to better prediction performance than using compute measures like FLOPs, and using 3 PCs consistently leads to the most robust predictions across all the tasks. These validate our choice of using 3 PCs as the default number of PCs and indicate the robustness of our results to the choice of the number of PCs. Using PC measures consistently leads to better prediction performance than using compute measures like FLOPs with 3 PCs being the best across different tasks. (a) AgentBench0SK10*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I *034W 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 8VEMR 8IWX 0PEQE 0PEQE 0PEQE 5[IR 5[IR =M +IQQE *EPGSR 4LM 4]XLME &0331 +482IS. 348 148 <+01 'SHI0PEQE 7XEV'SHIV 7XEV'SHIV (IIT7IIO'SHIV 1MWXVEP 1M\XVEP (a) Naive + Greedy 0SK10*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I *034W 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 8VEMR 8IWX 0PEQE 0PEQE 0PEQE 5[IR 5[IR =M +IQQE *EPGSR 4LM 4]XLME &0331 +482IS. 348 148 <+01 'SHI0PEQE 7XEV'SHIV 7XEV'SHIV (IIT7IIO'SHIV 1MWXVEP 1M\XVEP (b) CoT + Greedy 0SK10*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I *034W 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\ 17)train!I 17)test!I 4'! 0SK100PEQE)UYMZ*034W) %GGYVEG] ]!WMKQSMH\EQUATIONEQUATION.9: Comparing the prediction performance of using different numbers of PCs for observational scaling laws on the agentic capability tasks included in Sec. 4.2. Using 2 or 3 PCs leads to the best prediction performance across different tasks.The cutoff for selecting the holdout set could have a significant impact on the prediction performance of observational scaling laws, as it determines the size of the training set that could be crucial when the entire dataset is not large (as in our case). Here we analyze how the prediction performance changes with different holdout cutoffs for various predictive measures (PCs vs compute measures) and provide a quantitative comparison that characterizes their overall prediction performance under varying cutoffs. Specifically, we conducted the analysis on the post-training analysis tasks in Sec. 4.3 and the "emergent" capability tasks in Sec. 4.1, where there are more data points (compared to the agentic capability tasks in Sec. 4.2) to provide a more robust analysis. For each task, we vary the FLOPs cutoff to control the ratio of the test set from 60% to 5% (linearly spaced), which consequently changes the difficulty of the prediction task from more difficult (less training data with weaker performance) to easier (more training data with stronger performance). We can then compare the test MSE of using different predictive measures under different cutoffs and quantify the overall prediction performance using the area under the error curve (AUE). For "emergent" capability tasks, we additionally include a variant of the cutoff strategy that holds out test data based on the accuracy on the task, which simulates a more challenging weak-to-strong prediction scenario and offers an extra robust analyses.The results are depicted in Predicting with model sizes In Fig. E .12, we show the prediction performance of using model size for the "emergent" capabilities of LMs. We find that it leads to significantly worse forecasts compared to using training FLOPs and PC measures and poorly captures the "emergence" trend. This is probably because models from different families were trained with very different data sizes and quality and may use different architectures.Using default cutoff for arithmetic tasks In Fig. 4 , we applied a different FLOPs cutoff than the default one on arithmetic tasks to make the prediction tasks more challenging. Here, we present the results of using the default FLOPs cutoff on arithmetic tasks in Fig. E . 13 . We find that using the default FLOPs cutoff makes the prediction tasks trivial with too many data points close to perfect performance. Notably, using PC measures still outperforms using compute measures like model size and training FLOPs, indicating its robustness to the choice of the cutoff.Additional tasks In Fig. E .14, we present the results on additional "emergent" capability tasks included in Wei et al. [98] . Similar to the main tasks (Fig. 4 ), we used the default FLOPs cutoff for non-arithmetic tasks (IPA Transliterate) and a quarter of the default cutoff for arithmetic tasks (3-Digit Addition, 2-Digit Addition). We find that using PC measures consistently leads to the best prediction performance compared to using model size or training FLOPs. While the extrapolation does not exactly match the trend of the ground truth on the IPA Transliterate task, possibly due to the fact that the specific task capabilities are not well covered by our collected benchmark metrics, it still provides a reasonable forecast of the "emergence" behavior. Results on BBH We further validated our observational scaling laws for predicting the impact of CoT on the BigBench-Hard tasks [83] following the same setup in Sec. 4.3. In particular, we used the defaulted FLOPs cutoff and the same PC measures (# = 3). We normalized the prediction accuracy on each BBH task by their respective random prediction accuracy and aggregated the normalized accuracy across all tasks for predictions. The results are depicted in Fig. E . 16 . Surprisingly, we observe that using training FLOPs leads to reasonable predictions of LM performance with and without CoT on BBH tasks, possibly due to the denoising effect of aggregation over all tasks. Furthermore, using PC measures accurately captures the scaling trends in both setups, even when using training FLOPs leads to less tight captures in the "Naive" setup or fails to capture the behavior of models trained on synthetic data (Phi). 0SKPrediction results with different number of models selected by V-optimality In Fig. 7a , we demonstrated how the prediction errors change with the number of models selected by our method.Here we present a qualitative analysis of the prediction results with different numbers of models selected in Fig. E .17. We find that with more than 8 models, the fitted scaling curves have already converged to accurately capture the scaling trend, indicating the efficiency of our method. Prediction results with randomly selected models We present the prediction results with randomly selected models from all available models in Fig. E .18, in comparison to the results with models selected by our V-optimality criterion (Fig. E.17). All these results are produced with a fixed random seed. We find that using randomly selected models leads to a much worse prediction performance, even with 16 models, demonstrating the critical need to carefully select models for effective scaling analyses. (d) 20 models E.18: Prediction results with different numbers of randomly selected models. The prediction performance is much worse than our selection method, even when 20 models are being selected.We have received valuable feedback from anonymous reviewers and have conducted extrnsive additional analysis to address their remaining questions. Extracting PC measures with non-matrix factorization We note that the benchmark coefficients on our principal capability measures are not guaranteed to be non-negative, which may hinder the interpretability of the extracted components. Therefore, we conduct an additional analysis with nonnegative matrix factorization (NMF) to ensure the non-negativity of the component-wise benchmark coefficients that may provide more interpretable capability dimensions. The results are included in Fig. E.19. We observed the NMF components do generally demonstrate a interpretable decomposition, as well as a positive and smooth scaling with training FLOPs within each model family (as our PC measures). While NMF offers enhanced interpretability and positive scaling properties compared to PCA, it also has notable limitations. Firstly, unlike PCA, NMF does not enforce orthogonality among its extracted components, as evident in the observed correlation between Components 3 and 4. Consequently, the coefficients assigned to each model across dimensions may not serve as independent measures of specific capabilities. Secondly, the ordering of NMF components lacks uniqueness and intrinsic physical meaning. This contrasts with PCA components, which are systematically ordered by their explained variances. The PCA approach provides an 'importance' measure for each dimension and allows for controlled trade-offs between representativeness and noise inclusion by adjusting the number of PCs used in the analysis. We calculate 95% confidence intervals for predictions from the non-linear regression models at each data point, and the observed data points fall within these confidence intervals. When extrapolating from very few data points above the random-guess level (e.g., in Persian QA), the confidence intervals may be wider. We include representative tasks for both emergent capability (left) and post-training analysis (right) setups.EQUATIONIn Table E .2, we included the functional forms of fitted scaling laws in our experiments. These functional forms served as a preregistration of our predictions for future models at the time of the initial paper release, which has been used to test the generalizability of our scaling analysis to unseen models.Table E .2: The functional forms of the fitted scaling laws included in our paper, are preregistered for predictions of future models. Each functional form is presented as the logit of the normalized accuracy metric φ -1 (Y, h) = σ -1 ((Y -(1 -h)) /h) = X that is equivalent to Eq. ( 6). Each benchmark metric is scaled to be within the range [0, 1]. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboardhttps://evalplus.github.io/leaderboard.htmlhttps://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 


Decision: The paper introduces Observational Scaling Laws, a method to use benchmark scores of LLMs to infer how their performance would change if the amount of training compute was scaled, without actually having to train additional models. The proposed method applies PCA to the benchmark performances and finds that a small set of PCs (3) can explain much of the benchmark variation. The results show that the latent PCs can infer how models will perform on a new benchmark and how efficient models are at using their compute. 

All 5 reviewers and I are in agreement that the contributions are important and timely, and are supported by extensive experiments on 77 models and 21 model families. The additional results presented on the newly released Llama3.1 model during the rebuttal also strengthens the contribution and shows that this work has promise to predict capabilities of new models.

The authors were responsive to the reviewer's few concerns in their rebuttal and I urge the authors to update their paper with the promised changes, especially the addition of confidence intervals on the predictions and a separate section detailing the possible limitations of the work.

Reviews: review: paper_summary: paper_summary: The paper proposes observational scaling laws, a method to use benchmark scores of LLMs to infer how their performance would change if the amount of training compute was scaled, without actually having to train additional models. The authors apply PCA to a model-task benchmark matrix in order to obtain latent capabilities, and find that only a small set of such capabilities can explain almost all of the benchmark variation. They use the latent capabilities to infer how efficient different model families are at converting compute into these capabilities, and to infer how models will perform on a new benchmark, based on their capabilities. The authors then examine how these observational scaling laws can be use to predict model performance on emergent, agentic and prompt-induced abilities and find that the predicted scaling performance closely matches the actual performance of held-out models.
strength_weakness: strength_weakness: 1. The conceptual model proposed in the paper is that the (principal component) PC-dimensions (latent capabilities) obtained based on model-benchmark performances capture basic, supposedly interpretable skills (e.g. general, reasoning, and programming capabilties) that increase the more training compute is applied to the models. However, as shown in Figure E.2 in the appendix, for some models for PC-2 and PC-3 there is a negative or flat relationship between training compute and model skill, as measured by the corresponding PC. Additionally, the performance on some of the benchmarks correlates negatively with some of the PCs, e.g. -0.62 for HellaSwag and PC-2 (which supposedly captures reasoning abilities). Given these discrepancies, I am not sure that the mental model of interpretable skills is really applicable for the PCs (even though they they seem to be useful for the scaling predictions).
2. In Section 4.2 and Figure 5 the authors show that observational scaling laws can be used to extrapolate the performance of LLMs in agent settings. However, the test sets used here are quite small (2 for AgentBench, 1 for AgentBoard), which makes these conclusions somewhat unreliable. Perhaps a different training-test split (e.g. 80/20 or 70/30) could be used (instead of 90/10) in order to obtain a larger test set. Would the results still hold?
3. There are some minor clarity issues
    1. Section 3.1 defines an error metric ($E_m$) which is somewhat unclear, i.e.
        - Is $E_m$ capturing errors or performance (~= 1 - errors)? Perplexity is mentioned as an example, but shouldn't perplexity decrease the more compute is applied to the model (Eq. 1)?
        - Is it always normalized? Line 111 mentions that, but I am not sure about the initial definition.
    2. I found it a bit difficult to understand Figures 5c and 6c from the captions. It would be useful to mention in the caption that they are based on $\beta^T \gamma$.
questions: questions: 1. Would there be benefits to using more fine-grained benchmarks to estimate the skill vector $S_m$, for example a detailed breakdown over different BigBench tasks?
2. Would it be possible to use the observational scaling laws to estimate the effect of finetuning? I.e. suppose we knew for a few models how finetuning them on a particular dataset changed their performance on some target measure. Could we apply the observational scaling laws to infer the performance of finetuning more capable models?
3. How many models per family would be needed in order for direct extrapolation from compute/model size for that family to produce similarly accurate results as the observational scaling laws here?
4. The conceptual model of latent capabilities that determine benchmark performance is strongly reminiscent of [item response theory](https://en.m.wikipedia.org/wiki/Item_response_theory) (IRT) from psychometrics, which underpins exams such as GRE. In IRT the underlying skills also determine a sigmoid-shaped probability curve, according to which a student will be able to answer a question with a particular difficulty correctly. It would be quite interesting to draw a connection between IRT and the latent capability model used by the observational scaling laws.
limitations: limitations: The paper does not explicitly discuss limitations. I believe that the discussion throughout the paper is sufficient, though there may still be some potential limitations that the authors do not touch upon, for intance:
1. How well do the scaling laws hold up if benchmark data has leaked into training?
2. All models within the same family are assumed to share the same compute-efficiency. I think that this is a valid assumption, because models in the same family are by and large trained with the same architecture and dataset, but it's conceivable that there may be more heterogeneous families.
paper_summary: paper_summary: The paper proposes a generalized class of scaling laws that encompass multiple model families of different sizes. These resulting scaling laws are capable of predicting “emergent” behaviors, complex agentic performance, and inference techniques in an extrapolative manner, as seen with GPT-4. The observed laws yield less error compared to standard FLOP-based laws.
strength_weakness: strength_weakness: See questions.
questions: questions: 1. How does the method apply to Llama 3, where the amount of data is significantly larger than the standard Chinchilla optimal? Can we still achieve accurate predictions for Llama 3 models?
2. For complex tasks, increasing amounts of compute are dedicated to inference time, especially for agentic benchmarks. Can the scaling law capture the scaling trends for inference time compute?
3. Are there any insights on how to choose the PC-dimension to balance preventing overfitting with achieving better estimation?
limitations: limitations: Limitations are discussed. The authors claim they do not foresee any direct societal impact from their work.
paper_summary: paper_summary: This paper demonstrates the correlation, known as scaling laws, between training FLOPs and large language models’ (LLMs) downstream task abilities. The authors decompose performance metrics to fit this “Observational Scaling Law” and confirm its validity across emergent capabilities, agentic capabilities, and post-training interventions.
strength_weakness: strength_weakness: I did not find any specific weaknesses in this paper.
questions: questions: 1. Why using PCA to decompose the main component ? This may need to explain why using PCA is work.   Do these main components have any corresponding physical significance?
2.  I believe this paper aims to predict the performance of future large models without the need for training, but it only considers FLOPs. In reality, data quality and model size should also be considered. Is there any analysis of these two factors?
limitations: limitations: See in Question.
paper_summary: paper_summary: They propose using PCA decomposition of the performance of a range of models across a number of benchmarks to form an observational scaling law, which effectively predicts downstream performance across several different model families, including predicting post-training interventions like chain-of-thought. They find that 3 principal components explain much of the variance on the LM benchmarks that they test. They can then use these principal components to 1) derive a llama2 effective compute quantity to use as a universal x-axis that works across all model families; and 2) for a new task and model family fit an observational scaling law based on the relevant principal components. They demonstrate that they are able to accurately extrapolate the performance of more capable models using weaker models on a number of standard benchmarks, including emergent tasks from Big-Bench. They also show that they can predict GPT-4's agent capabilities, and the effect of techniques like chain of thought and self-consistency using their method. Finally they demonstrate an approach for selecting a minimal sufficient set of models for deriving the observational scaling law.
strength_weakness: strength_weakness: * They show that their method can extrapolate predictions of stronger model capabilities using weaker models; however we don't get to see how far in advance they are able to make predictions. It would bo nice to see an ablation where the cutoff point used for fitting is pulled back so as to understand when extrapolation begins to fail. Moreover, they claim that they can predict emergence capabilities, however their cutoff point is conveniently just a little bit after performance begins to improve beyond chance, enabling their extrapolations to work. In the case where the cutoff point is pulled back further, are such emergent predictions still possible? I feel that demonstrating this would be important for being able to make the claim that prediction of emergence tasks is possible.
* They don't give any kind of uncertainty measure or confidence interval for their predictions.
questions: questions: For a given take, are all models evaluated with the same prompts? It says you got some evals from Open LLM Leaderboard and others from EvalPlus and yet others from OpenLMLeaderboard. Are we sure all of these sources used the same evaluation procedure?
limitations: limitations: I think they mostly did a good job of acknowledging limitations.
paper_summary: paper_summary: This paper proposes observational scaling laws to align scaling laws of computing from different model families, which are trained on various recipes, by projecting model benchmark performance to surrogate compute. This enables applying scaling law analysis without actually training models. Using the observational scaling laws, the authors predict emergent abilities, including agent performance of models trained with much more compute.
strength_weakness: strength_weakness: 1. The predictability of emergent abilities seems to be overclaimed. The provided results in Fig. 4, 6 and 7 support that the sigmoid function can well fit the relationship between downstream performance and (surrogate) compute. However, the feasibility of making reliable predictions on emergent ability is doubtful. The fitting data in Fig. 4  include points that have shown non-trivial performance and some even reached the inflection point.  And extrapolation results in Fig. 5 only scale 4x.

2. Projecting the model benchmark performance to low dimensional surrogates of compute seems to imply the assumption that different model families mainly differ in the speed to obtain abilities and have little difference in the trade-off of different capabilities. Let's say A models are trained with 1% code + 99% text, while B are trained with 99% code + 1% text, do observational scaling laws still hold for these two model families?
questions: questions: 1. Eqn. 3 and 4 indicate that model performance scale with log compute in sigmoids. But Eqn. 5 shows linear projection between model performance and surrogates of log compute. This misalignment seems weird.
2. What are the implications of observational scaling laws on evaluating pretraining data quality? Do different families of models show very different compute efficiency according to your results in aligning them?
limitations: limitations: The applicability of observational scaling laws seems to depend on training data distribution. The influence has not been discussed.

Comments: title: Additional Response
comment: I thank the authors for their thorough response to my questions!

The additional results on NMF capabilities are interesting and this method might be helpful in settings where interpretability of the capability dimensions is needed. However, I think that it is easy to read too much meaning into the capabilities, and it is important to make sure that any interpretations of the capabilities are consistent with their actual task-relationship. The authors and I seem to be in agreement that ultimately the most important function of the capabilities is to provide accurate estimates of benchmark performances, which they seem to be achieving.

It is great to see the additional results with the 80/20 split on AgentBench, as well as the addition of the Llama-3.1-405B model.

I also think the comparison with non-observational scaling laws Qwen and OPT models shows that the observational approach seems to be able to more accurately forecast the performance of scaled models, while requiring less models per family.

I maintain my positive score and support the acceptance of the paper.
Thank you, I have looked at the pdf now and the results presented look satisfactory to me.
Thank you very much for your reply. The supplementary pdf is included in the [global response](https://openreview.net/forum?id=On5WIN7xyD&noteId=trfqKNaK2D), and the download link can be found at the bottom of that page. We'd be happy to address any questions after you've reviewed the additional results and appreciate your willingness to reevaluate based on this information.
Thank you for your response. I appreciate the work to include comparisons of the cutoff point and also confidence intervals (although I don't seem to see this supplementary pdf on openreview that you are referring to). I'm willing to raise my score 1 point to a 6.
Thank you so much for the detailed reply. 

It is impressive to see the scaling trend holds for LLaMa 3.1 405B although I think the difference in training data has not been sufficiently resolved. The sigmoid curve of different model families do not overlap.

Thank you so much for your inspiring work. Since the original rating is high, I will maintain my score.
### Clarity

> *”Is $𝐸_𝑚$ capturing errors or performance (~= 1 - errors)? Perplexity is mentioned as an example, but shouldn't perplexity decrease the more compute is applied to the model (Eq. 1)?”*

In our formulation, $𝐸_𝑚$ refers to the model error (~= 1 - accuracy). For the case of perplexity, it does decrease with more compute, which corresponds to the case where $\beta_f < 0$ (we did not constrain $\beta_f$ to be nonnegative). Due to the potential confusion it may cause, we will adjust the formulation with a negative sign included for $\beta_f$.


> *”Is it always normalized? Line 111 mentions that, but I am not sure about the initial definition.”*

It does not have to be normalized in the general case (e.g., perplexity). For our specific case in Eq 2 & 6 with the use of a sigmoid nonlinearity for downstream predictions, normalization is required.

> *”I found it a bit difficult to understand Figures 5c and 6c from the captions. It would be useful to mention in the caption that they are based on $\beta^{T}\gamma$”*


Thanks for your suggestions. We will update the captions with the suggested clarifications in our future version of the paper.


### Other questions & suggestions

> *”Would there be benefits to using more fine-grained benchmarks to estimate the skill vector 𝑆𝑚, for example a detailed breakdown over different BigBench tasks?”*

Fine-grained benchmarks could help in cases where the available benchmarks may not be able to form a capability space that sufficiently captures the dynamics of downstream capabilities. For example, if the downstream task is related to "a scientific research agent", then the more fine-grained performance on the "Professional" split of MMLU could potentially offer additional predictive gains.

In our case, the included benchmarks seem to well capture the downstream tasks that we have tested.



> *”Would it be possible to use the observational scaling laws to estimate the effect of finetuning?... Could we apply the observational scaling laws to infer the performance of finetuning more capable models?”*

This a good idea! We believe it is possible to establish scaling laws for fine-tuning, by combining the capabilities measures of pretrained models and finetuning-related quantities (e.g., fine-tuning data size) to study the scaling properties of specific finetuning techniques with respect to certain model capabilities. This is an interesting direction, and we leave it for future work.



> *”The conceptual model of latent capabilities that determine benchmark performance is strongly reminiscent of item response theory (IRT) from psychometrics,”*

Thanks for the suggestions! We will include a discussion of the connection to IRT in our future version of the paper.



### Limitation discussion

> *”I believe that the discussion throughout the paper is sufficient, though there may still be some potential limitations that the authors do not touch upon, for instance, …”*

Thanks for the suggested limitation discussion! We acknowledge that they are indeed parts of the limitations of our work and will more extensively discuss them in a separate section in the future version of the paper.