Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
Authors: Rylan Schaeffer
Hailey Schoelkopf
Brando Miranda
Gabriel Mukobi
Varun Madan
Adam Ibrahim
Herbie Bradley
Stella Biderman
Sanmi Koyejo
Venue: ICML 2025 poster (2025)
Keywords: evaluations
benchmarks
scaling laws
emergent abilities
capabilities
frontier models
foundation models
Primary area: deep_learning->large_language_models
TL;DR: What makes predicting downstream capabilities of frontier AI models with scale difficult?

Abstract: Predictable behavior from scaling advanced AI systems is an extremely desirable property for engineers, companies, economists and governments alike, and while a well-established literature exists on how pretraining performance scales, predictable scaling behavior on downstream capabilities remains elusive. While many factors are certainly responsible, this paper shines a light on a significant factor that makes predicting scaling behavior on widely used multiple-choice question answering benchmarks challenging and illuminates a path towards making such downstream evaluations predictable with scale. Using five model families and twelve well-established multiple-choice benchmarks, we show that downstream performance is computed from negative log likelihoods via a sequence of transformations that progressively degrades the statistical relationship between performance and scale. We then reveal the mechanism causing this degradation: downstream metrics require comparing the correct choice against a small number of specific incorrect choices, meaning accurately predicting downstream capabilities requires predicting not just how probability mass concentrates on the correct choice with scale, but also how probability mass fluctuates on specific incorrect choices with scale. We empirically study how probability mass on the correct choice co-varies with probability mass on incorrect choices with increasing compute, suggesting that scaling laws for \textit{incorrect} choices might be achievable. Our work also explains why pretraining scaling laws are commonly regarded as more predictable than downstream capabilities and contributes towards establishing scaling-predictable evaluations of frontier AI models.

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
Rylan Schaeffer
Hailey Schoelkopf² Herbie Bradley
Brando Miranda
Gabriel Mukobi! Varun Madan Adam Ibrahim² 3
Stella Biderman
Sanmi Koyejo"
Abstract
tributes towards establishing evaluations of frontier Al models.
Predicting changes from scaling advanced Al systems is desirable property for engineers, economists, governments and industry alike, and, while well-established literature exists on how pretraining performance scales, predictable seal- ing behavior on downstream capabilities remains elusive. While many factors are certainly re- sponsible, this paper identifies a significant factor that makes predicting scaling behavior widely used multiple-choice question answering bench- marks challenging and illuminates a path towards making such downstream evaluations predictable with scale. Using five model families and twelve well-established multiple-choice benchmarks, we demonstrate that downstream performance is com- puted from negative log likelihoods via a se- quence of transformations that progressively de- grades the statistical relationship between perfor- mance and scale. We then pinpoint the mecha- nism causing this degradation: downstream met- rics require comparing the correct choice against a small number specific incorrect choices, mean- ing accurately predicting downstream capabilities requires predicting not just how probability mass concentrates on the correct choice with scale, but also how probability mass fluctuates on the alter- native incorrect choices with scale. We empiri- cally study how probability mass on the correct choice co-varies with probability mass on incor- rect choices with increasing compute, suggesting that scaling laws for incorrect choices might be achievable. Our work also explains why pretrain- ing scaling laws are commonly regarded as more predictable than downstream capabilities and con-
1. Introduction
Predictable scaling behavior of frontier Al systems such as Gemini (Team et al., 2023; Reid et al., 2024) and GPT (Ope- nAI, 2024; OpenAI et al., 2024) is crucial for anticipating capabilities and informing key decisions regarding model development and deployment (Anthropic, 2023; OpenAI, 2023; Dragan et al., 2024). Predictable scaling behaviors enable engineers to make informed decisions about opti- mal model design choices and de-risk investment in ex- ceedingly expensive pretraining runs by determining the payoff from scaling up compute; for example, OpenAl pub- licly noted that "A large focus of the GPT-4 project was building deep learning stack that scales predictably" and that "[OpenAl] developed infrastructure and optimization methods that have very predictable behavior across multiple scales" (Achiam et al., 2023); OpenAl noted that this ide- ally goes beyond predicting loss values, and that "Having sense of the capabilities of model before training can improve decisions around alignment, safety, and deploy- ment". Additionally, predicting capabilities is of interest beyond Al practitioners: economists and governments also have a significant interest in predicting the capabilities of current and future frontier Al systems for better decision- making (regulation, taxation, and safety) and forecasting of economic impacts (Council of Economic Advisers, 2024). Downstream capabilities are especially of interest because quantities like pretraining loss are difficult to translate into quantities more meaningful to society, such as the impact on economic labor or societal harms.
However, while scaling laws for pretraining loss are well- established (Hestness al., 2017: Rosenfeld et al., 2019; Henighan et al., 2020; Kaplan et al., 2020; Gordon et al., 2021; Heraanez et al., 2021; Jones, 2021; Zhai et al., 2022; Hoffmann et al., 2022; Clark et al., 2022; Neumann & Gros,
'Stanford Computer Science EEleutherAI 'MILA University of Cambridge. Correspondence to: Ry-
lan Schacffer wataaaaf@@ssssaffrr...o.o. Sanmi Koyejo <sanmi @ es. stanford.eduz
2022; Hernandez et al., 2022; Maloney et al., 2022; Sardana & Frankle, 2023; Muennighoff et al., 2024; Besiroglu et al., 2024), the literature is less conclusive regarding predicting specific downstream capabilities with scale. Prior work has
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
dub hu
on Benchnark Sanple Le
chchiicss (ccreecesccorrect (correct choice) = 0.4
A) negate and exponentiate
() incorperate choices
B) mask
C) argmax
P-vocab(correct choice) 0.15
Accuracy 1.0
Training Compute (FLOP)
P"choices(tncorrect choice #1) 0.3
ccchoicesinncorect choice #2) 0.2
("chorces(incooreet choice #3) 0.1
Figure 1. Multiple-choice benchmark accuracy is computed from negative log likelihoods via sequence of transformations that degrades predictability. Computing Accuracy begins with computing the negative log likelihoods of each choice, then negating and exponentiating each to obtain the probability of cach choice (A), Choices are then restricted to a set of available choices by masking invalid continuations, and renormalizing obtain relative probability mass on each choice (B). Lastly, the model's choice is defined as arg max, {p (Aeeeeess (Available Choice,)); ) and Accuracy is 1 and only if the model's choice is the correct choice (C).
observed that performance on standard natural language pro- cessing (NLP) benchmarks can exhibit emergent abilities (Brown et al., 2020; Ganguli et al., 2022; Srivastava et al., 2022; Wei et al., 2022) where performance changes unpre- dictably with scale, but further work demonstrated that such unpredictable changes can be artifacts of researchers' analy- ses, i..., choices of metrics and lack of sufficient resolution from too few samples (Srivastava et al., 2022; Schaeffer et al., 2023; Hu et al., 2024). More recently, Du et al. (2024) claim that downstream capabilities can be predicted, but only after the pretraining cross-entropy loss falls below a certain threshold, and Gadre al. (2024) claim that while performance on individual tasks can be difficult predict, aggregating results across dozens of diverse benchmarks yields clearer scaling trends. In this work, we ask: in con- trast with groongl-preddctaal pretraining losses, why has predicting specific downstream capabilities with scale re- mained elusive?
difficult to predict, such as Accuracy or Brier Score (See Fig. I and Sec. 4 for further details):
log Po
Vocab (Correct Choice)
Ssaling-Predictabl
Nocab (Correct Choice)
+ Pa
Chaices (Correct Choice)
+ Pa
Accuracy
Scaling- Unpredictable
This paper demonstrates the following findings:
Calculating downstream metrics requires a se- quence of transformations applied to the original quantities. These transforma- tions progressively deteriorate the statistical rela- tionship between those metrics and the scaling pa- rameters (parameters, data, and compute). This formalizes an intuition that "more complex" metrics might be less easily predictable.
2. Contribution: Explaining Why Predicting Downstream Capabilities With Scale Has Remained Elusive
2. Accurately predicting downstream multiple-choice performance requires modeling not only the proba- bility mass assigned to the correct choice with scale, but also the probability mass assigned to the incor- rect alternatives. This explains the cause of compar- ative unpredictability of multiple-choice benchmarks; it also suggests potential path forward for success- ful predictive models of downstream performance in the area of multiple-choice question answering, and in general the need to model external information not related to scaling-predicttll predictable log likelihoods needed for downstream metric computation.
Our goal is to understand what breaks down between the predictability of pretraining losses and the unpredictability of downstream evaluations. To do this, we investigated the comparative predictability of different evaluation method- ologies and setups, focusing on popular and comparatively simple (yet still highly difficult to predict) multiple-chote choice question answering benchmarks. We began with scaling- predictable pretraining log likelihoods and tracked how these log likelihoods are transformed in the process of cal- culating downstream evaluation metrics that are notoriously
2

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
3. Continuous metries such as Brier Score are in- sufficient for recovering predictability. We observe that, contrary to prior work showing that metrics such as Brier Score can hide emergent behavior at times, Brier Score is insufficient to improve the statistical relationship degraded by incorporating incor- rect choices' probability mass.
5. LLM360 (Liu et al., 2023): LLM360 includes two 7B parameter LLMMs trained on 1.3T and 1.4T tokens. We selected 13 checkpoints of Amber spaced approxi- mately logarithmically.
NLP Benchmarks We evaluated the above model fam- ilies on widely-used multiple-choice benchmarks for as- sessing comprehension, reasoning, and world knowledge: A12 Reasoning Challenge (ARC) Easy and Hard (Clark et al., 2018), HellaSwag (Zellers et al., 2019), MathQA (Amini et al., 2019), MCTACO (Zhou et al., 2019), MMLU (Hendrycks et al., 2020), OpenbookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), RACE (Lai al., 2017), SeiQ (Welbl et al., 2017), SIQA (Sap al., 2019a), Wino- Grande (Keisuke et al., 2019) and XWinoGrad En (Muen- nighoff et al., 2023). For MMLU, we analyzed each of the 57 subjects (e.g.. Abstract Algebra) independently. For each benchmark, we used default evaluation settings from the LM Evaluation Harness (Gao et al., 2023).
3. Methodology: Data for Studying Scaling of Downstream Capabilities
To study how downstream capabilities on specific tasks change with scale for different model families, we gener- ated per-sample scores from large number of model fam- ilies and multiple-choice NLP benchmarks. To ensure the computed scores were consistent with prior work, we used EleutherAl's Language Model Evaluation Harness (Gao et al., 2023).
Model Families Because our goal is to explore the scaling behavior of evaluations with increasing compute, we chose to evaluate model families with dense combinations of pa- rameter counts and token counts. The following families were evaluated (additional details in App. E):
Performance Metries We used three common multiple- choice metrics (Srivastava et al., 2022; Schaeffer et al., 2023; Du et al., 2024): Accuracy, Brier Score (Brier, 1950), and probability mass on the correct choice relative to the available choices pg
Choices (Correct Choice).
1. Pythia (Biderman et al., 2023b): The Pythia family contains 8 models from 70M to 12B parameters trained on the Pile (Gao et al., 2020) for 300B tokens. We used 8 checkpoints per size of the non-deduplicated variants.
Compute Budget Calculations Following prior work (Kaplan et al., 2020), we approximated the pretraining compute C (in terms of training FL.OP) of given model checkpoint as a function of the parameter count (excluding embeddings) N and the amount of training data seen in tokens D: C C(N, D.
2. Cerebras-GPP (Dey et al., 2023): The Cerebras-GPP family contains 7 models ranging from 111M to 13B parameters. The models were trained on the Pile (Gao et al., 2020) for different durations as part of scaling study with a ratio of ~ 20x tokens to parameters in "Chinchilla" manner (Hoffmann et al., 2022).
4. What Makes Predicting Downstream Performance Difficult?
Performance on multiple choice benchmarks is commonly published as Accuracy, Brier Score, or probability mass on the correct choice out of the available choices pg (Corret Choice). These quantities are computed via a sequence of transformations that begins with the neg- ative log likelihood of the correct choice on this particular benchmark sample as some function ,(; -) of compute:
3. OLMo (Groeneveld et al., 2024): The OLMo family contains IB parameter model trained for 3T tokens and two 7B parameter models trained for 2T-2.5T to- kens. We selected 7 checkpoints for IB (spanning 84B' to 3T tokens) and 7 checkpoints for 7B (spanning 4B to 2.4T tokens).
4. INCITE (AI, 2023): The INCITE family contains 3B and 7B parameter models, trained on 0.8T and IT tokens of RedPajama-v model has only a single checkpoint, so we excluded it. We found this family to be a slight outlier from other families, which we speculate because its pretraining data were contaminated by benchmarks (Elazar al., 2023).
Cyncab (Correct Choice) = (aaeee)=(CCCooma [(Comuute, Benchmark Datum)
2023). The 3B
Two details are critical. Firstly, this negative log likelihood is not computed in expectation over a corpus; it is specific to this particular singular datum in the benchmark. All the
"This approximation neglects FLOP costs associated with atten- tion calculations over sequence length; however, such operations are negligible so long as dmodei >> > Retx/12 and this approxi- mation is therefore standard in most language model scaling law analyses.
MMLMo 1B checkpoints below 84B tokens were unfortunately accidentally lost by their creators.
3

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
1.8l
1.84
UncorreLated Correlated Anticorre lated Uniform Mixed
of Samples
Complementary CDF
1 CDF
de
8.0
1.0
Correlations
Correlations
Renchmarkk: ARC-Challenge
Parformance Metric. "eg.""""""Cnne Choice)
Correazion Metric: Spearman
Cedtra (Pexn: Daca Scaling
INCITE 28 Pron. (Don Seslnn)
0.8
LL34300 Ariarr TD Tokan (Paan Scalin))
OLMc 78 Pauus. (Data Scalng)
Pynki Pron (boo Sea
2
Pyrni
0.4
-.4
2
02
0.0
-1.00
-0.75
-1.00
-0.55
0.00
13.05
03.50
0.75
1.00
-1.00
-0.5
-0.50
-0.35
0.00
0.35
0.30
8.50
1.00
Condacioo Botweon FLOPs and Soards (Per Sample)
Conclation Bebecce FLDP and Stores (Po Sampph)
Figure 2. Distributions of score-compute correlations and their corresponding complementary cumulative distribution functions. Left: For each benchmark, model family, performance metric, and correlation metric, computed how scores correlate with compute. This yields distribution (over samples) of score-compute conrelations. Note: the uniform distribution is small but non-zero everywhere. Right: To casily extract what fraction samples in benchmark has score-compute correlations above any given threshold, we converted the probability distributions to complementary cumularive distribution functions, i.c., 1 minus the empirical cumulative distribution function (CDF). Top: Idealized distributions. Bottom: Actual data on ARC Challenge.
scores we discuss are per-datum. Secondly, this negative log likelihood is computed over the vocabulary of the model. One can then compute the probability mass of the correct choice, again with respect to the vocabulary:
and Brier score:
Brier Scores def
Σ (AAvail. Choice, =
2
Yocab
(Correct Choice) = exp (-Cy (-Cyycab(Correee (Correct Choice))
Correct Choice) pb
Choices (Avail. Choice,)
Pa
Next, probabilities are restricted to the set of available choices by masking invalid continuations and normalizing again with respect to this set:
where 1(:) is an indicator variable. To quantify how this sequence of transformations affects predictability of perfor- mance, we measured how per-sample scores correlate with pretraining compute, and then studied how the distribution (over samples) of correlation values shifts from log likeli- hoods to p to accuracy or Brier Score. Specifically, for each combination of model family, benchmark, performance met- ric, correlation metric, we computed a correlation value for each sample in the benchmark between pretraining compute and scores. This yielded distribution (over samples) of correlation values for the combination (Fig, 2 Left). Visu- alizing the distribution correlations for the combination told us what fraction of samples in the benchmark yielded scores that are correlated, uncorrelated or anticorrelated
Po Vocab (Correct Choice)
Choices Choices (Correct Choice) def
Pg
Vocab (Available Choice,) Σ: Pa
Vocab (Correct Choice) to Choices (Correct Choice)
We distinguish the support over the token space of the model versus over the set of available choices in the benchmark's 's question because, as we will show, the support crucially affects predictability. Finally, the choices-normalized proba- bility masses become standard downstream metries:
Accuracyo def
1 (Correct Choice = arg max {
po Choices/ Choices(Avail. (Avail. Choicc,)))
4

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Distributions of Scare- Compute Correlations by Metric Benchmark: ARC- MRRCCCalleree Challenge
Metric: kuroj)oonCoonest Choice)
Metric: (Coeet Coorect Choice)
Model Family
Cembrax (Parom. and Data Scaling)
INCITE Param.
Scaling)
LLM300 Arrer 78 Tokers (Paan Scaling)
OLMo 78 Parm (Dam Scalin)
Pythis Param. (Tats Sealing)
Pythia 2008 Tokars (Pwam. Scaling)
Metric: Orier Soore
Metric Accuracy
1.0
0.8
0.0
0.4
0.2
0.0
-3.00
-0.73
-0.50
-0.25
0.00
0.55
0.50
0.75
1.00
-1.00
-0.75
-0.50
-0.55
0.00
0.20
0.50
0.75
1.00
Cormlation Between Fl OPs and Score (Per Sample)
Correation Retween OPs and Scores (Per Sample)
Figure 3. Multiple-choice metrics like Accuracy and Brier Score computed via sequence of transformations that de- grades correlations between performance scores and pretraining compute. (A) Initially, scores under log
pa Vocab (Correct Choice) Choice) has effect for rank Choices (Correct Choice) decorrelates scores from compute. (C)
and compute highly correlated. Transforming log (Correct Choice) pe
Vocab Yoeah (Correct Choice) - Po Cheiees (Correct Choice) - Brier Score minorly decreases score-compute correlations. (D) Transforming Choices (Correct Choice) -> Accuracy more substantially decorrelates scores from compute. Correlation: Spearman. Results are consistent across benehmarks and all three correlation metries (App. H).
correlations. (B) Transforming Pa
Transforming Po
with compute (Fig. Right). We found consistent results using all three standard correlation metrics: Pearson (1895), Kendall (1938) and Spearman (1961).
returns the fraction the benchmark's 's samples with score- compute correlations greater than the threshold e (Fig. 3A), Beginning with log likeliboods, approximately 90% of sam- ples exhibit score-compute correlations > 0.75, regardless of the model family (Fig. 3A). Transforming negative log likelihoods into probability masses Yocab (Correct Choice) does not affect the distribution of score-compute corre- lations for Spearman and Kendall. However, transform- ing Pe creases the distribution of score-compute correlations (Fig. 3B), with only 40% of samples having score-coue ccore-compute cor- relations 0.75. Transforming Pe Chuiicss (Correct Choice) into Brier Score has little-to-no effect (Fig. 3C), but transforming into Accuracy (Fig. 3D) furthers decreases score-compute correlations. To quantitatively test whether these transformations indeed decrease the correlation be- tween scores and compute, we measured four statistics of these score-compute correlation distributions: (1) the mean,
We demonstrate how the sequence of transformations af- fects the distribution of score-compute correlations using ARC Challenge (Clark et al., 2018) as an illustrative bench- mark; we note that all other benchmarks exhibited similar patterns as well (App. H). We visualized the distributions via their complementary (empirical) cumulative distribution functions (complementary CDFs) (App. C):
Vocab (Correct Choice) into Pe Choices (Correct Choice) de-
((c) Σ [(C.>0)
where S is the number of samples in the benchmark and C, is the correlation (over the models in the model family) between compute and scores on the s-th sample in the bench- mark. For given threshold c, the complementary CDF S(c)
5

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Cormiation Complemation Canuarins Dxtrrbution Frection Conseatios
AdmonCCree Driaa"
Dra Sooe
Faelyy
Leatin Hlea ieterraed
Ldloo Anber Tikex (Phi
OLdo T0 Paoer (Doa Solt 1p (Din
60
12
Figure 4. All four statistics of score-compute correlation distributions demonstrate that transforming log Yocab (Correct Choice) into Accuracy causes score-compute correlations to deteriorate. We find consistent trend that the sequence of transforma- Yorah tions degrades score-compute correlations, shown by the right-to-left log Pe P (Correct Choice)-to-pi Vocah (Correct Choice)-to- Cholces (Correct
Po
Score-to-Accuracy vertical stripes. This trend holds across benchmarks and model families for three correlation metries (Spearman, Pearson and Kendall) and for four statistics correlation distributions (mean, median, the area under survival function, and negative Wasserstein distance from perfect correlation or perfect cotrrreerlli....... See App. Figs. 7, 8, for other correlation metrics and other score-compute correlation distribution statistics.
(2) the median, (3) the area under the complementary CDF and (4) the negative² of the minimum of two Wasserstein distances: between the empirical correlation distribution and an ideal distribution of all correlations = 1, and between the empirical distribution and an ideal distribution of all correlations = -1. Across the four summary statistics, for most benchmarks and for most model famillis, discov- ered consistent ordering of metrics of the score-compute correlation distributions (Fig. 4):
follow this ordering, computed what fraction of (bench- mark, correlation metric, model family, correlation distri- bution statistic) tuples obey the ordering. To be maximally conservative, we checked for strict inequalities only. We found that across benchmarks, model families, and the correlation distribution statistics, the claimed ordering of metries held at least 82.4% of the time for Pearson, 85.6% for Spearman and 90.4% for Kendall.
Vocab (Correct Choice)) Corr( Compute, log Doeap
5. Probability Mass on Incorrect Choices Causes Unpredictability
Corr (Compute, Pa Vocah (Correct Choice))
Corr (Compute, Pa
Choices (Correct Choice))
What is the mechanism that degrades how scores correlate with compute? All three metrics with degraded correla- tions Pø
Corr (Compute, Brier Score)
Choices (Correct Choice), Accuracy, and Brier Score depend not just on how the model's probabil- ity mass Pe rect choice as compute increases, but also depend on how the model's probability mass fluctuates on incorrect available choices {pr Vocah (Incorrect Choice) Incoreett Choices compute increases. As an example, suppose
Corr (CrmppCueue,, Accuracy)
Vocah (Correct Choice) concentrates on the cor-
To quantitatively confirm that the correlation scores indeed
We chose the negative Wasserstein distance for consistency with the other statistics: higher values correspond higher corre- lations between scores compute.
6

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
1441
Mayle Rieih
feler
IMOTE Prusr (Bin
Ca Powr (Dea Sidre))
112
Pyele 300 Tele thove Staing
Pytla Tuurr my.
023
0.3
(oroo)) Concepracl Physes
1.
Made Fami
Telem Plare Solii)
HMOTE 18 Prus [Duo
04
de Poen (hea (rder))
PyPia 300 toice 2hrem toling)
Pyhh frarr ng!
Ever
02
Figure Predictability deteriorates because probability mass fluctuating on alternative incorrect choices with scale. Left: Transitioning from Nicaab(C (Correct Choice) Chiice (Correct Choice) demonstrates that Pa (Correct Choice) contains little information about po
Choices (Correct Choice) and vice versa; loosely speaking, any value one can map to any value of the other. Center: While
Choices (Correct Choice) 0.5 must yield Accuracy 1, for any (Correett Choice) 0.5, knowing Pa Choosss (Correct Choice) De
contains little information about Accuracy and vice versa. Right: Brier Score is more predictable from py Chhicess (Correct Choice) than Accuracy, but still quite variable. Two benchmarks shown: MatiQA & MMLU Conceptual Physics.
Choices
Vocab (Correct Choice) 0.4 on 4-way multiple-choice question; what is the accuracy? Spreading the remain- ing mass uniformly on the incorrect choices will make Accuracy = 1, whereas concentrating mass on single incorrect choice will make Accuracy 0. O.
Pg
pute from an observed metric. We can see that Brier Score, metric meant to produce more continuous scores (Schaeffer al., 2023), is less variable than Accuracy, provided known Pe Choiocs (Correct Choice), but it cannot re- cover information about p.ycab (Correct Choice) that is lost when shifting to Po that this is because of the additional information regarding the underdetermined values of pg Choices (Incorrect Choice) for each incorrect choice.
Choices (Correct Choice). We next show
To demonstrate how drastically the probability mass placed on incorrect choices can alter performance, we visual- ized the relationships between pairs of metrics immedi- ately preceding and following a given transformation (Fig. 5). For negative log likelihood of the correct choice and V Vooeb (Correct Choice) (not pictured), we observed a clean correspondence between performance under the metric and compute: one can reliably map given value of these met- rics to compute, and vice versa. In contrast, once perfor- mance is evaluated using a metric that a function of the incorrect choices pp Choices (Correct Choice), Accuracy or Brier Score nearly any value of score under one metric can map to any value of Doyocab (Correct Choice) or Pe
6. Scaling Behavior of Probability Mass on Incorrect Choices
Pe
In general, aggregate performance over a distribution is of- ten of interest. Such focus aggregate performance leads to an important insight: in MCQA, probability mass fluc- tuations on incorrect choices do not "average out". Unlike estimating the mean of random variable, where positive and negative deviations cancel, the nonlinear nature of met- rics like Accuracy and Brier Score means that probability
Choices (Correct Choice) respectively (Fig. 5), thereby breaking the chain along which one can cleanly infer com-

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
HellaSwag
Compute
10
Model famdy Centire (Paoo,, and Uata bcaling)
Model Famby Pydio 120 Parer (Daca bosing)
Model Family Pybta 32201 Tokees (Ptram Scalnn)
20:20
1n-9 In
In
10
10
10
30
"Q
10-14
10
10- 10-
15-90 12-
32
D3
53-
50
Correc Chilces
Corroct Chiice)
Comoat Choce!
PIQA
Compute
Model Farily Conderas (Para.. and Data Soalin)
Medil Farily: Pythis 125 Param (Dats Sollag)
Midel Fars by Pythis 3015 Tokers (Parem. Scalnn)
2e1:0
1e
gengen C1000
ter
10
30 10
10
10
10" 30-7 10 50 10
13 D0
30-20
D0-14 10"
10:7
10-30 10 33
10 32 10
10-2
(Coneee Choine)
[Come Chaloe)
Comrer Chainn)
Figure 6. Probability on the correct choices and the incorrect choices correlated, but fluctuate substantially. Probability mass on correct choices and incorrect choices positively covaries and typically increases with compute. However, the spread is large: for any given value pa (Correct Choice). the mass on incorrect choices can vary by many orders magnitude.
mass shifts between incorrect options affect scores in com- plex ways that persist under averaging. For example, if probability mass shifts from incorrect option A incor- rect option B, the impact on accuracy depends on whether either option had enough mass compete with the cor- rect answer there's no natural cancellation. This perhaps counter-intuitive behavior partially explains why predicting aggregate performance via averaging has remained elusive.
ing trends for each incorrect choice might be possible; doing so might enable better predicting changepoints metrics like Accuracy or Brier Score. However, whether per- benchmark per-sample per-choice scaling trends can be fit and accurately extrapolated is unclear since the spread varies by several orders of magnitude. We leave this challenge to future work.
This analysis suggests that modeling probability mass flue- tuations on incorrect choices could improve predictions of metrics like Accuracy and Brier Score, though the magni- tude of improvement remains an open question. For metrics like Accuracy, such predictions should be made for each sample because knowing the average mass (across many data) placed on incorrect choices says little about how much mass is placed on any single incorrect choice for single sample. We conclude by providing preliminary evidence that achieving such feat might be possible. Specifically, test how probability masses on correct choices and prob- ability masses on incorrect choices covary with increasing compute (Fig. 6). Multiple benchmarks display strong posi- tive relationships between mass on correct choices and mass on incorrect choices, suggesting that fitting per-sample seal-
7. Discussion
This work identifies why multiple-choice assessments of frontier AI models are unpredictable, well the under- lying mechanism: probability mass on incorrect choices. Our results have implications for the design of future eval- uations of frontier Al models that are reliably predictable with scaling. We hope that our work will be extended to further the science of scaling-predictatl evaluation of AI systems, especially for complex and important model capa- bilities. We note several future directions for extension of our work, and we hope that the community also adopts framing to further improve scaling- predictable evaluations. For Related Work, please see Appendix A. For Future Directions, please see Appendix B.
8

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Acknowledgements
RS acknowledges support from Stanford Data Science and the OpenAI Superalignment Fast Grant. SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, ARPA-H, the MacArthur Foundation, Schmidt Sciences, OpenAl, and Stanford HAL.
Impact Statement
This work contributes to the crucial task of making Al sys- tem capabilities more predictable as they scale. Our research has several important implications for the development and deployment of frontier Al systems:
First, by identifying why downstream capability prediction has remained challenging, our work can help AI develop- ers and organizations make more informed decisions about model development and resource allocation. This could lead to more efficient and targeted development of AI systems, potentially reducing the environmental and computational costs associated with training large models.
Second, our findings have implications for Al governance and safety. The ability to better predict model capabilities is essential for proactive policy-making and risk assessment. Our work highlights fundamental challenges in capability prediction that should inform discussions about AI devel- opment timelines and safety measures. This is particularly relevant policymakers and organizations work to establish frameworks for responsible AI development.
Third, our research advances the scientific understanding of how capabilities emerge and scale. This improved understanding could help bridge the gap between theoretical scaling laws and practical applications, potentially enabling more reliable forecasting of AI progress.
However, we acknowledge that improving capability predic- tion could potentially be misused to optimize for harmful applications or accelerate unsafe development practices. To mitigate these risks, we have focused our research on un- derstanding fundamental mechanisms rather than providing specific optimization techniques, and we emphasize the importance of using these insights within framework of responsible AI development.
We encourage future work in this area to carefully consider both the benefits of improved predictability and the potential risks of capability optimization, ensuring that advances in understanding AI scaling behavior contribute to the safe and beneficial development of artificial intelligence.

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
References
J. Z., Foster, C., Jaiswal, M., Lee, W. Y., Li, H., Lovering, C., Muennighoff, N., Pavlick, E., Phang. J., Skowron, A., Tan, S., Tang, X., Wang. K. A., Winata, G. I., Yvon, F,, and Zou, A. Lessons from the trenches on reproducible evaluation of language models. arXiv preprint, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Bisk, Y., Zellers, R., Bras, R. L.., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourd AAAI Conference on Artificial Intelligence, 2020.
AI. T. Releasing 3b and 7b redpajama-ineite fam- ily of models including base, instruction-tuned & chat models. https: wwwwwwttttoo together.e...... ai /blog/ redpajama-nun cooma--moelss -v.,, 2023, Accessed: 2024-05- 19.
Bowman, S. R. Eight things know about large language models, 2023.
Alabdulmohsin, I., Neyshabur, B., and Zhai, X. Revisiting neural scaling laws in language and vision, 2022. URL https: arxiv. org/ abs/ 2209.220..02 06640.
Brier, W. Verification of forecasts expressed in terms of probability. Monthly weather review, 18-1) 1950.
Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorsii, R., Choi, Y., and Hajishirzi, H. MathQA: Towards interpretable math word problem solving with operation-basel for- malisms. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pp. 2357-2367, Minneapo- lis, Minnesota, June 2019. Association for Computa- tional Linguistics. doi: 10. 10.18653/1//111-12222 URL htaps://aclantttl11e www.atttoooooo..oo111112222 org/NN9-11499
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances neural information processing systems, 33: 1877-1901, 2020.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws, 2023.
Clark, A., De Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., et al. Unified scaling laws for routed lan- guage models. In International Conference on Machine Learning, pp. 4057-4086. PMLR, 2022.
Anthropic. Anthropic's responsible scaling pol- icy. https: hotp:://wwww hwwp httpss//ww..nttbroroc..ooom news/
anthropi reeeiics-eeepeoee www.rressppsss..eeo wesiibblessccc wwl--scaalnnnppoloe ing-policy,
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457 2018.
2023. Accessed: 2024-05-19.
Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert, N. Rajani, N., Sanseviero, O., Tun- stall, L., and Wolf, T. Open Ilm leader- board. https:/ hassss//hugginggaeee co/ spaces/ HuggingFaceti open 11m leaderboard, 2023.
Computer, T. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL
RedPajane aatajjaaaDDDa..
Besiroglu, T., Erdil, E., Barnett, M., and You, Chinchilla scaling: A replication attempt, 2024.
contributors, W. Survival function, 2023. URL
petps://ent www.wwiiiiiiioo.. worg/wkkikk wwwrivvallfnccioon [Online; accessed 22- May-2024).
Biderman, S., Prashanth, U. S.. Sutawika, L., Schoelkopf, H., Anthony, Q. Purohit, S., and Raff, E. Emergent and predictable memorization large language models, 2023a.
Council of Economic Advisers. The 2024 economic report of the president, 03 2024. Accessed on 09/29/2024.
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H.,
DeepSeek-AL, ;, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z.,, He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J.,, Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma. S., Nie, X., Pei, T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z.., Ruan, C., Sha, Z.,, Shao, Z.,, Song, J., Su,
O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and van der Wal, O. Pythia: suite for analyzing large language models across training and scaling, 2023b.
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J.. Abbasi, B., Aji, A. F., Ammanamanchi, S., Black, S., Clive, J., DiPofi, A., Etxaniz, J., Fattori, B., Forde,
10

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang, P., Wang,
A. A framework for few-shot language model evaluation,
S., Wang, Y., Wang. Y., Wu, T., Wu, Y., Xie, X., Xie, Z Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D.,
12 2023. URL https::/p///.. pttt:p:://eendd..o zenodo record/ 10256836.
You, Y., Yu, S.. Yu, X., Zhang. B., Zhang, H.,, Zhang, L., Zhang, I.,, Zhang, M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q., and Zou, Y. Deepseek I1m: Scaling open-source language models with longtermism, 2024.
Gordon, M. A., Duh, K., and Kaplan, J. Data and parameter scaling laws for neural machine translation. In Proceed- ings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 3915-5922. 2021.
Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpa Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K. R., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W. Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichan- der, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Sol- daini, L., Smith, N. A., and Hajishirzi, H. Olmo: Accel- erating the science of language models, 2024.
Dragan, A., King, H., and Dafoe, A. Introduc- ing the frontier safety framework. https: / //deepmind. http://iii..ogge//dddeoe. google//issoove//llog int roducing the frontier sattrerrraattttaraeeeeeee 2024. Accessed: 2024-05-19.
Du, 7., Zeng, A., Dong, Y., and Tang, J. Understanding emergent abilities of language models from the loss per- spective, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXh::2009.33000 2020.
Elazar, Y., Bhagia, A., Magnusson, I. H., Ravichander, A., Schwenk. D., Suhr, A., Walsh, E. P., Groeneveld, D., Soldaini, L.., Singh, S., et al. What's in my big data? In The Twelfth International Conference on Learning Representations, 2023.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardr, Measuring math- ematical problem solving with the math dataset. 2021.
Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M. Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., Xin, R., Nezhurina, M., Vasiljevic, I., Jitsev, J., Dimakis, A. G., Ilharco, G., Song, S., Kollar, T., Carmon, Y., Dave, A., Heckel, R., Muennighoff, N., and Schmidt, L. Language models scale reliably with over-training and on downstream tasks, 2024.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Hernandez, D., Kaplan, J., Henighan, T., and McCan- dlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021.
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Account- ability, and Transparency, pp. 1747-1764, 2022.
Hernandez, D., Brown, T., Conerly, T., DasSarma, N., Drain, D., El-Showk, S., Elhage, N., Haffield-Dodds, Z., Henighan, T., Hume, T., et al. Scaling laws and inter- pretability of learning from repeated data. arXiv preprint arXiv:2205. 10487, 2022.
Gao, L., Biderman, S., Black, S., Golding, L.. Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M., Ali, M., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint 1712:1111..0400 2017.
Gao, L., Schulman, J., and Hilton, Scaling laws for reward model overoptimization, 2022.
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXhv:2203.15556 2022.
A., Foster, C., Golding, L., Hsu, J., Le No A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
J., Reynolds, L., Schoelkopf, H., Skowron, A., Sutawika,
L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou,

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Hu, S., Liu, X., Han, X., Zhang, X., He, C., Zhao, W., Lin, Y., Ding, N., Ou, Z., Zeng, G., Liu, Z., and Sun, M. Predicting emergent abilities with infinite resolution evaluation, 2024.
Lyu, C., Wu, M., and Aji, A. F. Beyond probabilities: Unveiling the misalignment in evaluating large language models. arXiv preprint 14Xiv:2402.13888 2024.
Maloney, A., Roberts, D. A., and Sully, J. solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022.
Huang, Y. Zhang, J., Shan, Z., and He, J. Compression represents intelligence linearly, 2024.
McCandlish, S., Kaplan, J., Amodei, D., and Team, O. D. An empirical model of large-batch training, 2018.
Jones, A. L. Scaling scaling laws with board games. arXiv preprint arXiv:2 (ee::2204433110 031440333,, 13, 2021.
McKenzie, I., Lyzhov, A., Parrish, A., Prabhu, A., Mueller, A., Kim, N., Bowman, S., and Perez, E. The inverse scaling prize, 2022. URL https: /tttps://githb. com/ inverse- sca hpggrrizee.
Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. triv- iaqa: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv e-prints, art. arXiv:1705.cm 705.03551, 2017.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.086// 2020.
Muckatira, S., Deshpande, V., Lialin, V., and Rumshisky, A. Emergent abilities in reduced-scale generative language models, 2024.
Keisuke, S., Ronan, L.,, Chandra, B., and Yejin, C. Wino- grande: An adversarial winograd schema challenge at scale. Communications of the ACM, 2019.
Muennighoff, N., Wang. T., Sutawika, L., Roberts, A., Bi- derman, S., Scao, T. I.., Bari, M. S., Shen, S., Yong, Z.-.., Schoelkopf, H., Tang, X., Radev, D., Aji, A. F,, Al- mubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff, E., and Raffel, C. Crosslingual generalization through multitask finetuning. 2023.
Kendall, M. G. A new measure of rank correlation. Biometrika, 30(1/2):81 93, 1938.
Kleinbaum, D. G. and Klein, M. Survival Analysis: A Self-Learning Text. Springer, edition, 2012. ISBN 978-1441966452. doi: 10. 11100/997-1-4441-666
Muennighoff, N., Rush, A., Barak, B.. Le Scao, T., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and Raffel, C. A. Scaling data contrronst language models. Advances in Neural Information Processing Systems, 36, 2024.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kel- cey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019.
Neumann, O. and Gros, C. Scaling laws for multi- agent reinforcement learning model. arXiv preprint arXiv:2210.00849, 2022.
OpenAL.
Openai's approach frontier risk. https://openai..com httpa//oooccooooooaaaaaaaaeeeeee our our-approach-to-fteee www.:accccttt--tttteee------eee-
Lai, G., Xie, Q., Liu, H., Yang. Y., and Hovy, E. RACE: Large-scale Re ReAding comprehension dataset from ex- aminations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, September 2017. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/ D17-1082. URL https: acl aclaoooologyy....ooooooooooooooe org/ D17-1082.
2023.
Accessed: 2024-05-19.
OpenAL.
Hello gpt-4o. https:// con/ hndex/hello-gptt440/ 2024. Accessed: 2024-05-
16.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, L., Almeida, D.. Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapino, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd. M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F.
Liu, Z., Qiao, A., Neiswanger, W., Wang. H., Tan, B., Tao, T., Li, J., Wang. Y., Sun, S., Pangarkar, O., Fan, R., Gu, Y., Miller, V., Zhuang, Y., He, G., Li, H., Koto, F., Tang,
L., Ranjan, N., Shen, Z., Ren, X., Iriondo, R., Mu, C.,
Hu, Z., Schulze, M., Nakov, P., Baldwin, T., and Xing,
E. P. Llm360: Towards fully transparent open- source Ilms, 2023.
12

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Fleti, A., Floundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L.. Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontio-Lopes, R., Gordon, J., Grafstein. M., Gray, S., Greene, R., Gross, J., Gu, S., Guo, Y., Hallacy, C.,, Han, J. Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K. Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Łukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W. Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight. M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A.. Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy. D., Li, C. C., Lim, R., Lin. M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning,
performance?, 2024.
Pearson, K. Vii. note regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-3322:240-245 1895.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., I Lazaridou, A., Firat, O., Schritwwiesr, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint vrXiv:2403.055300 2024.
Rosenfeld, J. S., Rosenfeld, A.. Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. In International Conference on Learning Representations, 2019.
Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance, 2024.
Sap, M., Rashkin, H., Chen, D., Le Bras, R., and Choi, Y. Social iqu: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C.,, McMillan, P., McNeil, J., Medina, D., Mchta, A., Menick, J. Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O' Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E... Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P. Michael, Pokorny, Pokrass, M., Pong, V. H.. Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F. Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D.. Schulman, J., Sel- sam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J. Slama, K. Sohl, I., Sokolowsky, B., Song, Y., Stau- dacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, F. F.C. Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder. P., Weng, J., Weng, I.,, Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024.
2019a.
Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interac- tions, 2019b.
Sardana, N. and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws, 2023.
Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Pro- cessing Systems, volume 36, pp. Associates, Inc., 2023.
Curran
Spearman, C. The proof and measurement of association between two things. The American Journal of Psychology, 1961.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Carriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.0 2022,,
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac J.-B., Yu,
J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Owen, D. How predictable is language model benchmark
13

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
Wei, J. Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 2022.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing multiple choice science questions. In Derezynski, I.., Xu, W., Ritter, A., and Baldwin, T. (eds.), Proceedings of the 3rd Workshop Noisy User-generated Text, pp. 94-106, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v 1/W17-411. org/ /
URL https:ttp//.m httpt::s//aaaaathhlogogoooo
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 1905:v:1900078.. 2019.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 121111 2022.
Zhang, B., Liu, Z., Cherry, C., and Firat, O. When scaling meets Ilm finetuning: The effect of data, model and fine- tuning method, 2024. URL https: /awxivxxvvooogg org/ 0ass/240.11111
Zhou, B., Khashabi, D., Ning, Q., and Roth, D. "going a vacation" takes longer than "going for walk"; A study of temporal commonsense understanding. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Pro- ceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pp. 3363 (36633369,,4 Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.186533/1//D11-1333 URL https: // //aclannthee hac.aanhhooogg..org/11----1
14

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
A. Related Work
Language Model Evaluation The capabilities of Al models are typically evaluated using constructed datasets to assess performance on a specific task, acting as a proxy for some real-world usage scenario. However, performing robust and reliable evaluations is challenge, with many potential pitfalls and unsolved problems (Biderman et al., 2024). For example, might prefer to ask models open-ended questions and evaluate their answers in natural language, but it then often becomes difficult to robustly score the resulting model outputs, especially for partial correctness. For this reason, it is common practice for evaluation benchmarks to simplify their scoring via approximations, such as extracting a sub-string from free-form outputs heuristically (Joshi et al., 2017: 7; Kwiatkowski et al., 2019; Hendrycks et al., 2021) and checking that it matches a specific gold target string, or casting a task to a mulriple-chore incorrect answers is known, and the model's answer is determined by selecting the most likely option among these strings. For more details on the precise procedures typically used for multiple choice elsewhere in the literature, see Biderman et al. (2024). We believe that the multiple-choiee choice format is valuable, due to its flexibility, popularity and relevance (Brown et al., 2020; Beeching et al., 2023; Biderman et al., 2024), but we discuss its limitations in Section
format. in which a closed set of correct and
Scaling Laws Many neural networks exhibit power-law scaling of the pretraining loss as a function of the amount of compute, data, or parameters used for training (Hestness et al., 2017; Brown et al., 2020; Hoffmann et al., 2022). These neural scaling laws demonstrate that the pretraining loss can be highly predictable as a function of these fundamental inputs, which has a number of practical applications: Scaling laws fit to smaller training runs can be used to predict the pretraining loss of a much larger training run, and can be used to determine effective hyperparameters (McCandlish et al., 2018; DeepSeek-At et al., 2024), or the optimal allocation of dataset and model size for a given compute budget (Hoffmann et al., 2022; Muennighoff et al., 2024; Dey et al., 2023; Sardana & Frankle, 2023; Besiroglu et al., 2024). In some cases, such laws can be used predict performance of a larger model in a particular domain, such as coding (Achiam et al., 2023). The existence of scaling laws turns deep learning into predictable science the macro level by providing a simple recipe for improving model quality and de-risking returns increasing investment into scale (Ganguli et al., 2022; Bowman, 2023).
Emergent Abilities Language models have been observed to exhibit apparent emergent abilities behaviors down- stream task performance that cannot be predicted from smaller scales (Wei et al., 2022: Srivastava et al., 2022). Emergence appears not to be simply product of training compute or model size, but is also dependent on other factors such as dataset composition (Muckatira et al., 2024; Wei et al., 2022). Schaeffer et al. (2023) find that some emergent phenomena can be a "mirage" arising due to choices made by researchers such as the use of discontinuous metrics and insufficient resolution. However, Du et al. (2024) note that for many tasks, emergence remains despite the use of continuous metrics. Additionally, discontinuous metrics have been argued to often be the most reflective of real- world usefulness, so emergence in these hard metrics is important. Hu et al. (2024) found that for generative evaluations, infinite resolution can be achieved but requires significant compute and that generated answer be verifiable.
Predicting Downstream Task Performance Although predicting macroscopic pretraining loss is useful, a far more useful goal is to predict the scaling of model performance on particular downstream tasks domains. If this was possible, then model developers could tune their datasets and training procedures in a more fine-grained way before launching computationally intensive training runs. Model performance on a particular downstream task is typically correlated with compute, albeit with a few exceptions (McKenzie et al., 2022; Huang et al., 2024). However, despite attempts to fit scaling laws to values other than loss, including benchmark scores (Gadre et al., 2024; Zhang et al., 2024), model memorization (Biderman et al., 2023a), or reward (Gao et al., 2022), these downstream performance metrics are usually more noisy or require more compute to fit accurately. Owen (2024) and Gadre et al. (2024) both find that while aggregate benchmark performance with more compute can be predicted, the scaling behaviour of individual tasks can be noisy. Additionally, Owen (2024), Du al. (2024) and Gadre et al. (2024) claim that predicting scaling behavior on task without access to models exhibiting beater-than-random performance (i.e., "before emergence occurs"") cannot be done reliably. Concurrently to our work, Ruan et al. (2024) propose Observational Scaling Laws by mapping model capabilities from compute to a shared lowedimensional space of capabilities across model families before predicting performance on novel tasks. Our goal in this work is to investigate the comparative unpredictability of individual downstream performance scores, and advise how to create more sealing-predictalle evaluations that are closely coupled with wal-world use-cases.
15

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
B. Future Directions
Direction Beyond Multiple Choice Benchmarks Our study is restricted to benchmarks evaluated via log likelihood- based multiple-choice formats. While we believe this is inherently valuable due the usefulness and prevalence of such tasks, this limits the application of our findings. We hope that our discoveries and proposed mechanisms may be used inform the study of predictable and reliable evaluation writ large, and that future work should explore the extent which our findings can be generalized to more complex capabilities. Our findings corroborate those of Lyu et al. (2024), who find that multiple-choice answer scores often diverge from generative evaluations. Consequently, particularly important direction for further study is to investigate generative evaluations, which may contain similar transformations distancing performance from the observed loss.
Direction 2: Predicting Benchmark Performance A Priori Our work provides an explanation why multiple-choier
benchmark performance is not easily predictable for metrics such as Accuracy and Brier Score, as observed in the literature (Du et al., 2024). However, our analyses assume access to entire model families' scores across several orders of magnitude of pretraining FLOPs, and not employ backtesting, as sensibly recommended by Alabdulmohsin et al. (2022) and Owen (2024). predictive model should be able identify change points well in advance on standard metries like Accuracy or Brier Score.
C. Definition of Survival Function
The survival function Sx (z) also known as the reliability function, the tail distribution, or the complementary cumulative distribution function - gives the probability that a random variable exceeds a certain value æ (Kleinbaum & Klein, 2012; contributors, 2023):
Sx(x) def Pr 2| x]
fx(x') = = -1-Fx(x) S( -
(2)
where Fx(x) Pr[X S 2] is the cumulative distribution function (CDF) and fx(x) is the probability density function (pdf) or probability mass function (pmf) of the random variable X. The CDF Fx (x) gives the probability that the random variable X is at most while the survival function Sx (1) gives the probability that X exceeds x.
When the true distribution of X is unknown, we can use the empirical CDF (ECDF) (x) and the empirical survival function (ESF) Sx (z):
(x)x) del
1(x > 2) = = 1
(3)
where n is the number of observations, 2n is the realized value of the random variable for observation i, and 1[x; > x is the indicator function. The empirical survival function (x) specifies the fraction of observations for which the sampled random variable X exceeds æ.
D. Compute Resources for Experiments
Experiments were done across wide family of model families and sizes. The GPUs we used for medium-sizee models (7B parameters and above) used a single A A100s with 80GB of vRAM. For smaller models ( (<8B) we used A100s with 80GB of vRAM, Quadro RTX 8000 with 48GB of vRAM, or RTX A4000 with 16GB vRAM. For 70B parameter models, we used least 2 A A100 GPUs with 80GB of vRAM.
E. Additional Model Family Details
Here we provide further experimental details regarding our selection of model families.
1. Pythia (Biderman et al., 2023b): We consider two "families" for Pythia in our experiments. Pythia (Parameter Scaling) refers to the use of fully-trained checkpoints from 9 different model sizes (all model sizes documented in Biderman et al. (2023), as well as a 14M parameter model trained later by the authors). Pythia-12B (Data Scaling) refers to the use of 8 checkpoints across training for the Pythia- 1rthia-12B 12B model, namely having seen 2M, 64M, 2B, 6B, 20B, 60B, 200B, and 300B tokens in training.
16

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
2. Cerebras-GPT (Dey et al., 2023): Cerebras (Parameter and Data Scaling) refers use of checkpoint per model in the Cerebras-GPT family, each fully trained for differing quantities of data documented by the model creators, for 7 checkpoints in total.
3. OLMo (Groeneveld et al., 2024): OLMo (7B Data Scaling) refers to the use checkpoints for OLMo-7B across training, namely, checkpoints having seen 4B, 44B, 133B, 442B, 885B, 1.5T, and 2.4T tokens.
4. INCITE (AI, 2023): INCITE-7B (Data Scaling) considers 6 checkpoints over training for the 7B parameter model, having seen 240B, 280B, 400B, 500B, 700B, and IT tokens.
5. LLM360 (Liu al., 2023): LLM360 Amber (Data Scaling) considers 13 checkpoints of the Amber model, having seen OB, 3,5B, 7B, 10.5B, 17.5B, 31,5B, 49B, 87.5B, 147B, 252B, 430B, 738B, and 1.26T tokens.
F. Broader Impact
This paper contributes to better understanding of the predictability of large language models (LLMs), which can have both positive and negative societal impacts. On the positive side, by making LLM benchmarks more predictable, this research can help society anticipate and plan for potential challenges associated with their development and deployment. This increased predictability can facilitate proactive measures mitigate risks and ensure the responsible use of Al technologies.
However, the increased predictability of LLMs could theoretically be exploited by malicious actors to accelerate the development of Al systems designed for malicious purposes. We also stress the importance of proactive risk assessment and the implementation safeguards to prevent the misuse of Al technologies.
17

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
G. Score-Compute Correlation Distributions' Statisties
G.I. Pearson Correlations
timm
340
Figure Statistics empirical distributions of correlations between scores compute for benchmarks and model fam-
ilies. These correlation values were computed with Pearson correlation and are consistent with the main text's results computed with Spearman correlation (Fig. 4): The sequence of transformations from log Pe Vocah (Correct Choice) - Pb Norsh Yursh (Correct Choice) - Choices (Correct Choice) Accuracy degrades predictability.
Do
18

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
G.2. Spearman Correlations
Figure 8. Statistics for empirical distributions of correlations between scores and compute for all benchmarks and model families. These correlation values computed with Spearman correlation. The sequence of transformations from log Xucah (Correct Choice) Vocab [Correct Choice) + Choices (Correct Choice) + Accuracy degrades predictability.
19

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
G.3. Kendall Correlations
Figure Statistics for empirical distributions of correlations between and compute for all benchmarks and model fam- ilies. These conelaatinn values were computed with Kendall correlation and are consistent with the main text's results computed Yocah Youth (Correct Choice)
with Spearman correlation (Fig. The sequence of transformations from log / (Correct Choice) Po
Choices (Correct Choice) + Accuracy degrades predictahility.
H. Per-Benchmark Score-Compute Correlation Distributions
H.1. NLP Benchmark: ARC Challenge (Clark et al., 2018)
20

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
Benchmark: ARC-Challenge
Performance Metric: boggs Choice)
Correlation Metric: Spearman
Model Family
Cembra (Param. and Data Staling
ICTT 78 Panm (Pata Scaling)
0.8
LLMS5 Amber 18 Takers (Poram Sealing)
OLMs IB Param. (Data Scalng)
Pytlia Param. (Data Stalin)
Extimated Dunsity
Pythia 3008 Tolens (Param Scaling)
0.6
L-Eppiiiall
0.4
2
0.2
0.0
0.25
0.00
0.25
0.25
1.00
0.50
0.55
0.00
0.50
1.00
Correlation Oztweon FLOPs and Scores (Per Sample)
Conrection Between FLOPs and Scores (Per Sample)
12u Connet Choic)
Mheii Fandly
Parx Sulrd
JMD0 Aertwe Tomes Paan Solle))
Farms (Dira Sallel
Tylua 2014 baans (houm salay!
675
1.0
8.5
875
COONT
Useil
Place
ludrnd
Cintedoc speil
1M000 Town Sollec (DaaSSuing)
Sylla 2014 baae (heve. indiig
pau Serpl
Contiiun
ABC Chilings
Senkai Paai Diu Seling)
La Lanbiii Q4000)
Cantitos
54% 18 (heaSo
Yours
Gevelcia Bilees F10P Sare Samplr
Scats (Por Sarpp))
Cantizio Brtate 5.00% Stare (1he Sie
Figure 10. ARC Challenge: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
21

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.2. NLP Benchmark: ARC Easy (Clark et al., 2018)
ABC he Cornet Orike)
Vaeid Pandyy
MCTT favm (Dea Soleel
Canértox ponmn
Cindatox
Canlddo webil
tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.78
830
0.0
6.5
1.0
0.50
8.1
075
8.75
0.20
0.50
4.55
goac Coone Cheins)
Maeid Pam
Place Data lulro)
Candedoo peare
Cantadoo welt
Tames Per iodee)
Tylla 2014 lown (heam. Indiig
025
025
Ornaada Drver Sampl)
Contitin Stown 7.50% Scoee (Te Sampa)
Candisio Detuets 7.00% Score (Th Camrlo)
Day
Seokai PPar Diu Sulind
JAM50 Tames
(Thas Seiing)
sen Stars (Tewe Srdda)
Cemiasse Bower T10h and Sorn Phe Sengll
Connerdon Strwen
Scaee (Te Smrpt)
Contisdo Behees 7.0% and Scare (Th Suralo
ARCBay
(herm auellrw fedind
(Sin toabe)
(The Sering
Fram (Biu Salle]
Crnelate Reteee FLOP) Serms
Somes (Ph Strpic)
Contistio Brheer
Seam (Ph Sorrto
Figure 11. ARC Easy: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
22

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.3. NLP Benchmark: HellaSwag (Zellers al., 2019)
He kdeag be Cornet Onike)
Vaeid Pandyy
MCTT fasm (Dea Soleed
Canériox ponnn
Jid0o taves tooee)
Farm (Dra Soliel
Sylua 2014 baan theae suily!
0.78
8.0
0.30
6.5
1.0
830
0.50
8.10
075
8.75
0.20
0.50
3.55
(hr
goah Coone Chain))
Maeid Paerly
Place Data lulro)
Candedio peare
Contadoo weei
JADD Aer Tomes Per iodle)
Tylla 201 loww (heam. Indiig
025
025
Ornaada Dover Sore the Sampl)
Contitin Stown 1LD0% Stree (Te Samph)
Candidio Detuets 7.00% Score (Th Camral)
Seakai PPar Diu Sulind
JA450 Tames
(Tha Seilng)
sen Stars (Peve Stade))
Comiacse Bower TL00 and Soro the Senpll
Comterdon Strwen Scoee (Te Sirpp)
Contirdoo Behees 7.00 Score (Ph Surrdt)
Ho lo5ea
Conten (herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Orvelate Releee F10P)
Somes (Ph Strpin
Contistio Brheer
Figure 12. HellaSwag: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
23

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.4. NLP Benchmark: MathQA (Amini et al., 2019)
Maniga bn Connet Cholee
Madil Pardyy
10111 favn (iin budee!
fevm Bine tudee!
7ydua 2013 Tou's (Taue. Solle!
+1.0
-025
-8.0
-12
633
100
-100
-05
-0.30
6.5
025
-1.0
5.00
02)
6.30
Mar126 1goi ConeeCContt Choloe)
Vadia Family
(Parn Sulrd
1A400 Aartee Towes PPere Solke)
Pytka With Toars (Trse. Sollet
-05
6.5
025
025
Gevaacte Drver Socie
Dontitin Stower
7.D0% Stoee (Te Sampa)
Candidio Detuets 7.00% Scare (Th Camrlo)
Vodit
Place anditee Salred
Cardasos ganns
Candidoo parte
Candisoos seail
(4000 harves Teler Paace (Daz-Sulng)
sinn Stans (Teve Stulay)
FLOR SAre Sampll
Scate
(1h
Mar14
Conden (harm anellew fudind
HOTEE harm (Doa Sules]
Ca didiii (p0rn
24 18 Parrs (DeaS))
Tou
Gevelcia Bicees ELOP and Sore Phe Sample
Consoraon Siieen 2L.0% Scone (Par Siript)
Canatatio Batutes 5.00% and Stari (Phe Saerao
Figure 13. HellaSwag: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
24

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.S. NLP Benchmark: MC TACO (Zhou et al., 2019)
MC TADO in (Comec Cholee)
Vadil Pardyy
10111 favn (iin budee!
Ayle fevm Bine tudee!
7ydu 201 Tou's (Taue. Solly!
+1.0
-025
-8.0
-12
633
-55
-55
6.5
025
-00
-1.0
E.0
02)
6.30
1goi Comect Choloe)
Vadia Fanily
(Parn Sulrd
Lartaiic QUARIA
1A400 Aartve Towes PPere Solke)
Pytka With Toars (Prse. Solret
-05
925
Genaaaae Drver Sore the Sampl)
Dontitin Stown
Stoee (Tr Simph)
Candisio Deruess 7.00% and Stoue (Ph Camrlo
MC-TACO
Vodi
place anditee Salred
Cardtsos Qanns
Candidoo parte
Candidoo seeil
(4000 harves Teler
(Daz-Sulng)
sinn Stans (Teve Stula))
FLOP Scres Sampl]
Condizio Behwirn R.B0% (Ph
MC-TACO
Conden (harm anellew fedind
HOTE harm (Doa Soles]
Ca didii (part)
p001
2446 18 Paris (DeaS))
You
Gevelcaa Diceet ELOF Sore Samgll
Conterion Siieen 2L.0% Scone (Pa Strppl)
Canatatio Baiutes 5.00% and Stari (Phe Sterako
Figure 14. MC TACO: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
25

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.6. NLP Benchmark: MMLU Abstract Algebra (Hendrycks et al., 2020)
MMLU Altreet Agrino keg Cornect Onike))
Vaeid Pandyy
MCTT fasm (Dea Soleed
Candrtox spnmn
Jid0o feron tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
6.5
1.00
0.50
075
8.5
0.20
0.50
(hr
1ois Coone
Maeid Pae
Sendess Place Data lulro)
Cantedox sare
Contedoo SAMEL
JAnDo harow Tomes Per iodle)
Tylla 2014 lour (haam. Indiig
025
Oenaada Drver Sorn
Consition Stower 7L.0% Scoee (Te Simpa)
Canditio Deruets 7.00% Store (Ph Camrll)
KMIU Aberom Algicc
Senkas (Par Diu Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sen Stars (Peve Stade))
Comiacse Bower TLOh and Sorn the Sengll
Contedion Strwer 1L.0% Scoee (Pe Sirpt)
Contirdio Behees 7.0% and Scare (Ph Saerlo
KMLU Abstruc: Alyebs
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reieee F10P Serws phe Serpic)
DP% Scmes (Phe Strper)
Contistio Boheer (P% Seaen (Phe Soerit
Figure 15. MMLU Abstract Algebra: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
26

---

Why Has Predicting Downsiream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.7. NLP Benchmark: MMLU Anatomy (Hendrycks et al., 2020)
MMLU Anatory keg Cooeect Coreet Oroke))
Vaeid Pandyy
MCTT favm (Dea Soleed
Canértox ponnn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
0.00
67
100
0.50
075
8.55
0.00
Cooney Chein)
Maeid Paerly
Sendes Place Data lulro)
Candedox BRRrO
Condadoon Sewil
JAnDo her Tomes Per iodle)
Tylla 2014 lown (heam. Indiig
025
Oenaada Dover Sore
Dontitinn Stewer Stoee (Te Sampa)
Candidio Setuets 7.00% Store (Phe Camral)
1000 foctury
Seokai PPar and Sulind
Dastsbe
JMM50 Tames
(Tha Seilng)
sen Stars (Peve Stade))
Comiacse Bower TLOT and Sort the Sengll
Comnendon Strwen 1L.0% Scoee (Pe Sirpi)
Contirdoo Behees 7.0% and Stare (Ph Seerlo
MILUL Anatort
(herm purl fedind
Castsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reieee F1OP) Seres
Scnes (Ph Strper)
Contistoo Brheer
Seae
Figure 16. MMLU Anatomy: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
27

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.8. NLP Benchmark: MMLU Astronomy (Hendrycks et al., 2020)
ke Mores Cornect Onike)
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrdix spannn
Jid00 tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7 830
67
0.50
8.55
Ho Cooney (hein)
Maeid Pae
Sendes Place Data lulro)
Candedex
Cantedox
Contedoo
JAnDo herow Tomes Per iodle)
Tylla 2014 low (heam. Indiig
925
Grnaada Drver Sorne
Contitinn Srower 7LD0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Ph Camrll)
KMLU Amoonns
Seokai PPar and Sulind
JAM50 Tames
(Thas Seiing)
Sylea sene Stars (Tewe Srdda)
Comiasse Bower T10h and Sorn the Sengil
Comnerion Strwen 7L.0% Sone (Te Smrpt)
Contirdoo Behees 7.0% and Scare (Ph Surrlt
KNLU Astunsery
(herm auellrw fedind
Eastske
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Serws phe Sempin)
Sones (Ph Strpin)
Contistio Brheer
Seaen (Phe
Figure 17. MMLU Astronomy: Downstream performance is computed via sequence of transformations that deteriorate correla- tions between scores and pretraining compute.
28

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.9. NLP Benchmark: MMLU Business Ethics (Hendryeks et al., 2020)
MMLU Businss Biless ke CCoeect Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtox sonnn
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
0.50 8.10
075
8.75
(hr
glat Cooney Chaine)
Maeid Paemly
Sendes Place Data lulro)
Centedox
Cantedox sers
Contadoo
JApDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
Dontitiin Srower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camral
MMLU Tihics
(
Seokai PPar and Sulind
JAM50 Tames
(Thas Seiing) Sodae)
sen Stars (Peve Srddy)
Cemiaase Bower T10h and Sorn the Sengil
Contedinn Strwen 7L.0% Soee (Pe Smipt)
Contisdoe Behees 7.00 and Scare (Ph Seerlt
MMLU Busrnu Bivo
(herm auel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serw phe
Scnes (Ph Strppn)
Contistio Boheer
Figure 18. MMLU Business Ethics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
29

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.10. NLP Benchmark: MMLU Clinical Knowledge (Hendryeks et al., 2020)
MMLU Choial Knowleige 1e Coreet Onike))
Vaeid Pandyy
MCTT fasm (Dea Soleed
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
8.0
0.00
6.50
8.1
075
8.55
0.00
0.50
(hr
goan Coone Cheins)
Maeid Paerly
Sendes Place Data lulro)
Cantedoo perrsw
Contedoo
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
Oenaadte Drver
Consirinn Stower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (The Samrdl)
MMJU Ciiica Hanuledge 17
Senkai PPar and Sulind
Dastsbe
JAM50 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower T10h and Sorn the Sengil
Contedinn Strwen 7L.0% Sonee (Pe Siipt)
Contirdoo Behees 7.0% and Scare (Ph Serrlo
MMLU Clncal Kocwleter
(herm puel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee FL0P Serws phe
DP% Scmes (Ph Strppnn
Contistion Boheer
Figure 19. MMLU Clinical Knowledge: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
30

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
H.I1. Benchmark: MMLU College Biology (Hendryeks et al., 2020)
MMLU Colgee 1e Cornet Ohike))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrdiv sontn
tooee)
Farm (Dra Sollel
Sylua 2014 boans theae sally!
0.7
0.50
075
8.75
(hr
goach Coonnt Cheiks)
Maeid Pae
Place Data lulro)
Cantedoo peare
JADDo Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
Dontitiin Srewer Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Ph Carraa
MMLLI College Bollgg
Seokai PPar and Sulind
JA450 Tames
(Thas Seilng)
sene Stars (Teve Srdda)
Comiacse Bower T10h and Sorn the Sengil
Connerdon Strwen Scoee (Pe Smrpt)
Contisdoe Behees 7.00 and Scare (Ph Suralo
MMLUI Colgee Bobar
(herm auel fedind
(Sin toabe)
(The Sering
Fram (Biu Salleg
Crvelate Beteee F10P phe
Scnes (Ph Strpin)
Contistio Brhere
Figure 20. MMLU College Biology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
31

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.12. NLP Benchmark: MMLU College Chemistry (Hendrycks et al., 2020)
MMLU Colleee Overittr 1e Cornct Onike)
Vaeid Pandyy
MCTT favm (Dea Soleel
Canditoo ponnn
Caneasoo perce
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
0.50
8.5
(hr
College goan Chain))
Maeid Paerly
Sendes Place Data lulro)
Centedox pears
Contedoo
JApDo herow Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
025
Dontitinn Srower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Ph Camraa)
MMIU Collegn Cheekitry
Seokai PPar and Sulind
Dastsbe
JAM50 Tames
(Thas Seiing) Sodae)
sene Stars (Tewe Srddy)
Cemiacse Bower T10h and Sorn the Sengil
Contidin Strwen 7L.6% Sonee (Pe Smrpt)
Contisdoo Behees 7.0%% and Scare (Ph Suealo
MMLU Collge Oueriitry
(herm peel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee FL0P phe
DP% Scmes (Ph Strppn)
Contistion Beheer
Figure 21. MMLU College Chemistry: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
32

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.13. NLP Benchmark: MMLU College Computer Science (Hendryeks et al., 2020)
MNLU Collge k4 Corect Ohike))
Vieid Pandyy
MCTT favm (Dea Soleed
Candrtox ponen
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
0.7 830
1.00
67
6.50
075
8.75
0.20
Collng 1oai Chein)
Maeid Paemly
Sendes Place Data lulro)
Cantedoo perre
Cantedoo Soail
JAnDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
925
Grnaadae Drver Soree
Dontition Srowen 7L.0% Stoee (Te Sampa)
Candidio Setuets 7.00% Store (Ph Camrla
KILL Collgg Comp
Seokai PPar and Sulind
JAM50 Tames
(Thas Seiing) Sodae)
sen Stars (Teve Srdda)
Comiacse Bower TL0h and Sorn the Sengll
Conterdon Strwen 7L.0% Sone (Te Smipi)
Contisdoo Behees 7.0% and Scare (Ph Surado
MNLU Collgg Corp Sel
(herm auellrw fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP Serw phe
Somes (Ph Strpei)
Convistoo Brheer
Figure 22. MMLU College Computer Science: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
33

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.14. NLP Benchmark: MMLU College Mathematics (Hendryeks et al., 2020)
MMLU Collage Malks k Coonect Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrtox wonen
Caneladoe
tooee)
Farm (Dra Sollel
Sylu 2014 baans theae suily!
32
0.7
65
0.50
8.75
0.50
4.5
goath Coonn Chaio)
Maeid Pae
Sendes Place Data lulro)
Contetor
Candedox
Contedoo
JADDo herow Tames Per iodle)
Tylla 2014 lowr (heam. Indiig
025
Ornaadae Drver
Dontitin Srower Scoee (Te Sampa)
Candisio Betuets 7.00% Scare (Th Camrko
NMLU U Eollagy Matk
Seokai PPar Diu Sulind
JAM50 Tames
(Thas Seiing)
sene Stars (Teve Srdda)
Cemiasse Bower T10h and Sorn the Sengil
Comnendon Strwen 7L.0% Sooee (Pe Smipt)
Contisdoe Behees 7.00 and Scare (Ph Sarall
NMLUI Collag Matho
(herm auellow fedind
Castsker
(Sin toabe)
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serws phe
DP% Somes (Phe Strpei)
Contistio Beheer
Figure 23. MMLU College Mathematics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
34

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.15. NLP Benchmark: MMLU College Medicine (Hendrycks et al., 2020)
MMLU Medicie 10155m Corect Ohike)
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtex ponen
perce
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
830
0.00
67
6.50
8.1
075
8.75
0.20
6.75
(hr
plesh CooneY Chains)
Maeid Paerly
Place Data lulro)
Cortedoo
Cantedox perrue
Cantedoon SAMEE
JAnDo herow Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
025
Dontitinn Srowen 7L.0% Scoee (Te Sampa)
Cantidio Detuets 7.00% Store (The Samril)
KMLU Cologg
Senkai PPar Diu Sulind
Dastsbe
JAM50 Tames
(Tha Seiing)
sen Stars (Tewe Stade))
Comiacse Bower TL0h and Sorns the Sengll
Contedinn Strwen 7L.0% Sone (Pe Siipi)
Contirdoo Behees 7.0% and Scart (Ph Suerlo
KNLU Collaz Medicie
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee F10P the
DP% Scnes (Phe Strppn)
Contistio Boheer
Figure 24. MMLU College Medicine: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
35

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.16. NLP Benchmark: MMLU College Physics (Hendrycks al., 2020)
MMLU Collge Plylis k Cornect Ohike))
Vaeid Panlyy
MCTT favm (Dea Soleed
Candrtox ponnn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7 830
1.00
67
0.50
8.10
8.5
0.70
0.50
(hr
glec Coone Chiin))
Maeid Pae
Sendes Place Data lulro)
Cantedox
Contedoon
JAnDo herow Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
Oenaadae Drver Soree
Dontitin Srowen 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Stare (The Camraa)
MMLU Collega Plydix
Senkai iPar and Sulind
Dastsbe
JAM50 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacte Bower TLOP and Soros the Senpll
Contedinn Strwen 7L.0% Sonee (Te Smrpt)
Contirdoo Behees 7.0%% and Stare (Ph Seerlo
MMLU Colgge Phylic
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee F10P Serws the Serpin)
1P% Scmes (Phe Strppn)
Contistion Boheer (P% Seaen (Phe
Figure 25. MMLU College Physics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
36

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.17. NLP Benchmark: MMLU Computer Security (Hendryeks et al., 2020)
MML Corp Searty he 1 Corndt Ohinn)
Vaeid Pandly
MCTT favm (Dea Soleed
Candrtox ponen
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae sally!
32
0.7
0.50
070
8.75
(hr
goach Coonnt Cheins)
Maeid Paemly
Place Data lulro)
Candedoo
Contedoon
JADDo her Tames Per iodle)
Tylla 2014 low (heam. Indiig
925
Dontitinn Srower Scoee (Te Sampa)
Cantidio Detuets 7.00% Store (The Camraa
MME Corpp
(
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing) Sodae)
sene Stars (Teve Srdda)
Comiacse Bower T10h and Sorn the Sengi)
Comninion Strwen 7L.0% Sonee (Pe Sirpt)
Contisdoo Behees 7.00 and Scare (Ph Saealt
MMLL Cove Searly
(herm auellew fedind
(Sin toabe)
(The Sering
Fram (Biu Salleg
Crvelate Beteee F1OP phe
Scnes (Ph Strpin)
Contistio Brheer
Sear
Figure 26. MMLU Computer Security: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
37

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.18. NLP Benchmark: MMLU Conceptual Physics (Hendrycks et al., 2020)
MMUU Coooppt Plylks 14 Coreet Ohinn)
Vaeid Pandyy
MCTT fasm (Dea Soleed
Candrdic pointry
Jid0o feroon tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
077 830
6.5
6.50
8.10
075
875
0.00
0.50
(hr
Plylics
gooth Coone Chain)
Maeid Pae
Place Data lulro)
Centedix sooen
Cantedox perse
Cantadoo WOIL
JADDo herow Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
925
Oenaadae Drver
Consiriin Stower 71.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camralo
NMEUI Coccepts Plysict
Senkai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sen Stars (Peve Stade))
Comiacte Bower TLOh and Sorn the Sengi)
Comtedin Strwen 7L.0% Sonee (Pe Smipt)
Contisdoo Behees 7.0%% and Stare (Ph Saerlt
MNIUU Goncpptal Hopsis
Comten (herm auel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee F10P Sere phe Sempin)
Sones (Ph Strppi)
Contistio Behere (P% Seaen (Ph
Figure 27. MMLU Conceptual Physics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
38

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.19. NLP Benchmark: MMLU Econometrics (Hendrycks et al., 2020)
NMUU Euroeetrrcs ke Cornect Onike)
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrtox ponen
Conelsooo
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
830
6.
6.50
8.5
Cooner
Maeid Pae
Sendes Place Data lulro)
Ornesdoo perve
Contedoo SAMEE
JApDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
Contitinn Stower 7L.0% Scoee (Te Sampa)
Candidio Setuets 7.00% Store (Th Samrlo
UI Frocomettios
Seokai PPar and Sulind
Dastsbe Seall
JA450 Tames
(Tha Seiing) Sodae)
sene Stars (Tewe Srddy)
Comiatse Bower T10h and Sorn the Sengil
Comnerion Strwen 7L.6% Soee (Pe Smrpt)
Contisdoe Behees 7.0%% and Scare (Ph Saralo
NMLUI Eurnendrio
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP Serw phe
Scmes (Phe Strppin
Contistio Brheer
Figure 28. MMLU Econometries: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
39

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.20. NLP Benchmark: MMLU Electrical Engineering (Hendryeks et al., 2020)
WMUU Extrical H Oniie))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrdor pomen
Conelatoo
tooee)
Farm (Dra Sollel
Sylua 2014 baans theae sally!
32
0.7
67
0.50
8.10
070
875
(hr
Hooi Coonet Fagenereg
Maeid Paemly
Place Data lulro)
Cantedox pers
Contedoo SEMER
JADDo Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
025
Grnaatae Drvee Soree
Consitiin Stower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (Th Samrla)
MMEU sericca 4g 15 (Connn
Seokai PPar and Sulind
Dastsbe
JA450 Tames
(Thas Seilng)
sene Stars (Teve Srdda)
Comiacse Bower T10h and Sorn the Sengi)
Connerion Strwen 7L.0% Sonee (Te Smrpt)
Contisdoe Behees 7.0% and Scare (Ph Saerlo
MMUU Eectred Enginarig eg
(herm auellew fedind
(Sin toabe)
(The Sering
Fram (Biu Salle]
Crnelaie Beteee FLOP Serws phe
DP% Scmes (Ph Strpine
Contistio Brheer
Sear
Figure 29. MMLU Electrical Engineering: Downstream performance is computed via a sequence of transformations that deterio- rate correlations between scores and pretraining compute.
40

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.21 NLP Benchmark: MMLU Elementary Mathematics (Hendrycks et al., 2020)
MVLU Bovestar Mare 14 bcrss Cooeect Ohoke))
Vieid Panlyy
MCTT favm (Dea Soleel
Candrtox sonnn
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
830
65
100
6.50
8.1
075
8.5
0.00
(hr
Eoei Coonet (heins)
Maeid Pae
Sendes Place Data lulro)
Cantedoo perrs
Contedoo
JAnDo herow Tomes Per iodee)
Tylla 2014 lour (heam. Indiig
025
025
Consitinn Stower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (Th Camrla)
MMB UI Elwennzzyy Matte 1 Conrne
Senkai PPar Diu Sulind
JAM50 Tames
(Tha Seiing)
sene Stars (Peve Srddy)
Comiacse Bower T10h and Sorn the Sengil
Contedion Strwen 7L.0% Sone (Pe Smrpt)
Contisdoe Behees 7.0% and Scari (Ph Sardio
MYLU Borostary Mato
(herm puel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP phe
1P% Scnes (Ph Strpen)
Convistio Brheer
Figure 30. MMLU Elementary Mathematics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
41

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
H.22. NLP Benchmark: MMLU Formal Logic (Hendryeks et al., 2020)
MULU Forell ke Corect Ohike)
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrtox permn
Jid00 tames tooee)
Farm (Dra Sollel
5ylu 2014 boans theae sally!
32
0.7
1.00
0.50
075
8.5
0.20
Fored goat Coone (hein)
Maeid Paemly
Place Data lulro)
Candedoo serre
Candedoo SeMiE
JApDo her Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
Dontitinn Srower 7..0% Stoee (Te Sampa)
Candidio Detuets 7.00% Store (Phe Stmral
MM Foree Loge
Seokai PPar and Sulind
Dastsbe
JA450 Tames
(Thas Seilng) Sodae)
sen Stars (Teve Srddy)
Cemaasse Bower T10h and Sorn the Sengil
Contedion Strwen Sone (Pe Smrpt)
Contisdoo Behees 7.00% and Scare (Ph Saealo
MVLU Foreal Logs
(herm auellrw fedind
(Sin toabe)
(The Sering
Fram (Biu Salle]
Crnelate Beteee F1OP phe
Scnes (Phe Strper)
Contistoo Brheer
Sear
Figure 31. MMLU Formal Logic: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
42

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.23. NLP Benchmark: MMLU Global Facts (Hendryeks et al., 2020)
MIVU Giokal keg Cornett Onike))
Vaeid Panlyy
MCTT favm (Dea Soleel
Candrtix sonen
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
0.50
075
8.75
0.00
0.50
(hr
Hooh Cooney Chrins)
Maeid Pae
Sendes Place Data lulro)
Cantedox
Cantedoo
JANDo herow Tomes Per iodle)
Tylla 2014 loua (heam. Indiig
925
Dontitin Srower 7L.0% Stoee (Te Sampa)
Candidio Detuets 7.00% Store (Phe Camralo
Seakai PPar and Sulind
JAM50 Tames
(Tha Seiing)
sen Stars (Teve Stade))
Comiacse Bower TLOh and Sorn the Sengil
Comnenion Strwen 7L.0% Sooee (Te Smrpt)
Contirdoo Behees 7.0% and Scare (Ph Suraoo
MILU Gobal Facts
(herm auel fedind
(Sin toadey
(The Sering
5ylu Fram (Biu Salle]
Crnelaie Reteee F10P Serws the
1P% Scmes (Phe Strpen)
Contistion Beheer
Figure 32. MMLU Global Facts: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
43

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.24. NLP Benchmark: MMLU High School Biology (Hendryeks et al., 2020)
MM h Cornet Ohike)
Vaeid Pandyy
MCTT favm (Dea Soleed
Caneedox pormn
Caneadoo Heroe
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
8.0
0.0
67
100
0.50
075
8.75
Hoat CooNN Chein)
Maeid Pae
Place Data lulro)
Candedoo p0rre
Contedoo SAMEl
JAnDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
Ornaadae Drver
Dontitinn Srower Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (The Camral)
Seokai PPar and Sulind
Dastsbe
JAM50 Tames
(Thas Seiing) Sodae)
sen Stars (Tewe Srddy)
Cemiasse Bower T10h and Sorn the Sengil
Connerdon Strwen Scoee (Pe Sirpp)
Contisdoe Behees 7.0%% and Stare (Ph Suealo
MMJU HS Beloge
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beteee FLOP the
Scmes (Phe Strppi)
Contistio Beheer
Seaen (Phe
Figure 33. MMLU High School Biology: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
44

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.25. NLP Benchmark: MMLU High School Chemistry (Hendryeks et al., 2020)
MMLUU HS keg Corect Ohoke)
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrtie wonmn
Caneisoo
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7 8.0
6.5
6.50
8.75
0.00
(hr
gaath Coonn Chain))
Maeid Pae
Place Data lulro)
Cantetoc worm
Dantedoo serve
Contedoo Seaid
JMDDo her Tomes Per iodee)
Tylla 2014 lowr (Peam. Indiig
025
025
Oenaadae Drver
Dontitinn Scoee (Te Simpa)
Candisio Detuets 7.00% Score (Ph Camrlo
MSG Carviry
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing) Sodae)
sene Stars (Tewe Srddy)
Cemiacse Bower T10h and Sorn the Sengil
Connerion Strwen Sooee (Pe Sirpt)
Contisdoe Behees 7.0% and Scare (Ph Suralo
MMLLU HS Cveritty
(herm auel fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beteee F1OP the
Scmes (Ph Strpei)
Contistoo Boheer
Figure 34. MMLU High School Chemistry: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
45

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.26. NLP Benchmark: MMLU High School Computer Science (Hendryeks et al., 2020)
NMLU Comp ke Cooeet Ohoke)
Viei Pandyy
MCTT favm (Dea Soleel
Candrtox pontn
Jid00 have lodee)
Farm (Dra Sollel
Sylu 2014 baans theae sally!
32
0.7
67
6.50
075
8.75
goath CooNN Chains)
Maeid Paemly
Place Data lulro)
Cantedox
Cantedoo
JADDo Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
Grnaada Drver
Dontition
Scoee (Te Samph)
Cantidio Detuets 7.00% Score (Th Carrlo
Seokai PPar and Sulind
Dastsbe Seall
JAM50 Tames
(Thas Seiing)
sene Stars (Teve Srdda)
Cemiacse Bower T100 and Sorn the Sengi)
Comtendon Strwen 7L.0% Sone (Te Smrpt)
Contisdoo Behees 7.00 and Store (Ph Surado
HNLU HS Corp Sei
(herm peel foaind
(Sin toabe)
(The Sering
Fram (Biu Salleg
Crnelate Beteee FLOP Serws phe
Scnes (Phr Strpen)
Figure 35. MMLU High School Computer Science: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
46

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.27. NLP Benchmark: MMLU High School Chemistry (Hendryeks et al., 2020)
MMLUU HS keg Corect Ohoke)
Vaeid Pandyy
MCTT favm (Dea Soleel
Cardrtie sonmn
Caneisoo
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7 8.0
6.5
6.50
8.75
0.00
(hr
gaath Coonn Chain))
Maeid Pae
Place Data lulro)
Cantetoc worm
Dantedoo serve
Contedoo Seaid
JMDDo Tomes Per iodee)
Tylla 2014 lowr (Peam. Indiig
025
025
Oenaadae Drver
Dontition Scoee (Te Simph)
Candisio Detuets 7.00% Score (Ph Camrlo
MSG Carviry
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing) Sodae)
sene Stars (Tewe Srddy)
Cemiacse Bower T10h and Sorn the Sengil
Connerion Strwen Sooee (Pe Smrpt)
Contisdoe Behees 7.0% and Scare (Ph Suralo
MMLLU HS Cveritty
(herm auel fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP phe
Scmes (Ph Strpei)
Convistoo Boheer
Figure 36. MMLU High School Chemistry: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
47

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.28. NLP Benchmark: MMLU High School European History (Hendryeks et al., 2020)
MMLU Euro Horeey ke (Coeet Coment Choki))
Vsei Panlyy
MCTT fasm (Dea Soleel
Canérfox sonen
Lid00 tames tooee)
Farm (Dra Soliel
Sylu 2014 boans theae suily!
32
0.7
830
675
0.50
8.10
075
8.75
0.20
0.50
3.55
Foas Coone Chein)
Maeid Paerly
Sendes Place Data lulro)
Cantedox
Contedoo
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
Dontition Srower Scoee (Te Sampa)
Candidio Detuets 7.00% Stare (Th Camrla)
MMEUT 11 Hiowy
Senkai PPar and Sulind
JA450 Tames
(Tha Seiing)
sen Stars (Tewe Stade))
Comiacse Bower TLOh and Sorn the Sengil
Comnerdon Strwen 7L.0% Sone (Pe Siipi)
Contirdoo Behees 7.00% and Scare (Ph Sarato
MMLU HS Eao Hoory
Comten (herm auel fedind
(Sin toadey
Castsker
(The Sering
Fram (Biu Salle]
Crnelate Beteee FLOP) Seres phe
1P% Scmes (Phe Strppnn
Contistio Boheer
Figure 37. MMLU High School European History: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
48

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.29. NLP Benchmark: MMLU High School Geography (Hendryeks et al., 2020)
MMLU 45 ke CCoeet Ohike))
Vieid Pandyy
MCTT favm (Dea Soleed
Candrtox poren
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7 830
1.00
6.5
0.50
075
8.75
0.20
(hr
gooth Coone Chein)
Maeid Pae
Sendes Place Data lulro)
Cantedox gerre
Cantedoo
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
025
Grnaadae Drver Sore
Contitiin Srowe Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Th Camrla
MMLU IS Geggapty (
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing)
sen Stars (Tewe Stade))
Comiacse Bower T10h and Sorn the Sengil
Comtedion Strwen 7L.0% Sone (Pe Smrpt)
Contirdoo Behees 7.0% and Scare (Ph Surrdo
MKLU H5 Goagessyy
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP Serws phe
Scmes (Ph Strppi)
Convistio Boheer
Figure 38. MMLU High School Geography: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
49

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.30. NLP Benchmark: MMLU High School Government & Politics (Hendryeks et al., 2020)
MEKUJ Gwt Puttic CCoeect Soree Ohike))
Viei Parlyy
MCTT fasm (Dea Soleel
Candrrox ponnn
Jid0o feron tame lodee)
Farm (Dra Soliel
Sylua 2014 baan theae sully!
32
0.7
830
0.50 8.10
075
8.75
0.20
0.50
(hr
glanh Coonn Chain)
Maeid Paerly
Sendes Place Data lulro)
Candedoo perrie
Contedoon
JANDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
025
Oenaadte Drver Soree Ph
Consition Stower 7LD0% Scoee (Te Sampa)
Candidio Setuets 7.00% Store (The Camral)
MIVLU Polrice
Senkai PPar and Sulind
Destsbe
JA450 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower TLOh and Sore the Sengil
Comnerion Strwen 1L.0% Scoee (Pe Smrpt)
Contisdoo Behees 7.00% and Stare (Ph Surato
MYILU Gvt Poltic
Comten (herm puel fedind
Dastske
(Sin toadey
(The Sering
5ydu Fram (Biu Salle]
Crnelaie Beteee F1OP Seres phe Serpin)
Scnes (Ph Strpen)
Convistion Behere
Figure 39. MMLU High School Government & Politics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
50

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.31. NLP Benchmark: MMLU High School Macrocconomies (Hendrycks et al., 2020)
MMLU HS Mecoeorcnics k4:15 1c11 (Conect Ohiin))
Vseid Panlyy
MCTT fasm (Dea Soleel
Candrtic ponmn
Candiatoo Hante
Jid0o feron tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
67
6.50
075
8.75
0.70
0.50
(hr
goos Coonn
Maeid Pae
Sendes Place Data lulro)
Candedoo perre
Contedoon
JAnDo herow Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
025
Consitiin Stower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camral)
MMLUI Mccoeeooonds
Senkai PPar Diu Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sen Stars (Teve Stade))
Comiacse Bower TLOh and Sorn the Sengil
Connerdon Strwen 7L.0% and Sone (Pe Siipi)
Contisdoo Behees 7.0% and Scare (Ph Surrdt
HMLU HS Mesweorcnia
Comten (herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Retwee F1OP Seres phe
Scnes (Ph Strppi)
Convistion Beheer
Figure 40. MMLU High School Macrocconomics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
51

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.32. NLP Benchmark: MMLU High School Mathematics (Hendrycks et al., 2020)
MMLU Cornett Coreet Choke))
Vieid Panlyy
MCTT favm (Dea Soleed
Candrtic parmn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
0.0
0.50
075
8.75
Maeid Pae
Sendes Place Data lulro)
Cartetice worms
Cantedoon SAMEE
JAnDo herow Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
Ornaadae Drver
Consitiin Srewer Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (Th Camrlo
Seokai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seiing) Sodae)
sen Stars (Peve Srddy)
Comiacse Bower T10h and Sorn the Sengil
Conterdon Strwen SoNe (Te Sirpi)
Contirdoo Behees 7.0% and Scare (Ph Surddo
MKLU HE Math
(herm auel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beteee F1OP Sere the
Somes (Ph Strppn)
Convistio Beheer
Figure 41. MMLU High School Mathematics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
52

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.33. NLP Benchmark: MMLU High School Microcconomies (Henddycks et al., 2020)
MMLU Micooponoee k (Coeect Onike))
Vieid Panlyy
MCTT favm (Dea Soleel
Candrdie pontn
Jid0o tames tooee)
Farm (Dra Soliel
Sylu 2014 boans theae suily!
32
0.7
830
65
0.50
8.1
075
8.75
0.20
0.50
goas Cooney Chain))
Maeid Paerly
Sendes Place Data lulro)
Candedoo perve
Contedoo
JinDo herow Tomes Per iodle)
Tylla 2014 lowr (heam. Indiig
025
Ornaadae Sore
Consitiin Srowen 71.0% Stoee (Te Sampa)
Candidio Detuets 7.00% Scare (The Camrla)
MOELL Miconccooo
Senkai iPar and Sulind
JAM50 Tames
(Tha Seiing)
sen Stars (Teve Stade))
Comiacse Bower T10h and Sorn the Sengil
Contecion Stree 7L.0% Sone (Pe Siipt)
Contirdoo Behees 7.0% and Scare (Ph Suralo
MMLU HS Massocoor
(herm purl fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Serws phe
Scmes (Ph Strppi)
Figure 42. MMLU High School Microcconomics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
53

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.34. NLP Benchmark: MMLU High School Physics (Hendrycks et al., 2020)
MMIU HS Flyics keg 1 Cornet Oninn)
Vaeid Pandyy
MCTT favm (Dea Soleel
Canditao poren
lodee)
Farm (Dra Sollel
Sylua 2014 baans theam sudly!
0.7
65
0.50
8.75
Ho Coone Chains)
Maeid Pae
Place Data lulro)
Contetoo
Cantedox perre
Cantadoo SAMEE
JANDo Tames Per iodle)
Tylla 2014 low (heam. Indiig
925
Ornaada Drver
Dontitinn Srower Scoee (Te Sampa)
Cantidio Detuets 7.00% Store (Ph
Plysie
Senkai iPar and Sulind
JAM50 Tames
(Thas Seiing)
sene Stars (Teve Srada)
Comiacse Bower T10h and Sorn the Sengi)
Comnidion Strwen 7L.0% Scoee (Te Smrpt)
Contisdo Behees 7.00 and Scare (Ph Surado
MMLU HS Phylo
(herm auellrw foaind
Castsker
(Sin toabe)
(The Sering
Fram (Biu Salleg
Orvelate Beteee FLOP) Serws phe
Scnes (Ph Strpin)
Contistio Boheer
Figure 43. MMLU High School Physics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
54

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.35. NLP Benchmark: MMLU High School Psychology (Hendryeks et al., 2020)
HS Paycedaay ke Cortct Ohike)
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtox ponen
Candidoo
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
675
1.00
0.50
075
8.75
0.20
gooth Covnet Chains)
Maeid Paerly
Sendes Place Data lulro)
Cantrdoo perre
Contedoon SIMEE
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
Oenaadte Drver Sorne
Contitiin Srowe Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Ctmral)
MREU US Phychology
Senkai PPar and Sulind
Destsbe
JM450 Tames
(Tha Seiing)
sen Stars (Tewe Stade))
Comiasse Bower T10h and Sorn the Sengi)
Connirion Strwen 7L.0% Scoee (Pe Sirpt)
Contisdo Behees 7.0% and Stare (Ph Surado
MUHUL HS Paycidaay
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Retwee F1OP Seres phe
DP% Scmes (Ph Strppi)
Contistio Beheer
Sear (Ph
Figure 44. MMLU High School Psychology: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
55

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.36. NLP Benchmark: MMLU High School Statistics (Hendryeks et al., 2020)
MMLU Suiiiics ke Coreet Ohike)) Caneitooo H0re
Vaeid Pandyy
MCTT favm (Dea Soleel
Canerdex poren
Jid00 tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae sally!
32
0.7
65
6.50
8.75
0.20
0.50
(hr
gaan Coone Chain))
Maeid Paemly
Sendes Place Data lulro)
Dantedoo
Contedoo
JADDo Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
Grnaada Drver Soree
Consition Srower Scoee (Te Sampa)
Cantidio Detuets 7.00% Store (Ph Camrlo
MM Statetie
Seokai PPar and Sulind
Corsidio
Dastsbe
JA450 Tames
(Thas Seiing)
sene Stars (Tewe Srdda)
Cemiasse Bower T10h and Sorn the Sengil
Conterion Stewe (L.0% Sone (Te Smipt)
Contisdoo Behees 7.0% and Scar (Ph Saraoo
MMLU Sudoiia
(herm aeel fedind
(Sin toabe)
(The Sering
Fram (Biu Salleg
Crnelate Beteee FLOP
Scmes (Ph Strpin)
Convistion Boheer
Figure 45. MMLU High School Statistics: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
56

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.37. NLP Benchmark: MMLU High School US History (Hendryeks et al., 2020)
HS US heg Cornct Ohoie))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtox sonnn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
0.50
8.1
075
8.75
0.20
0.50
goath Coonn Chains)
Maeid Pae
Sendes Place Data lulro)
Cantedox
Contedoo
JAnDo herow Tomes Per iodee)
Tylla 2014 lowr (heam. Indiig
025
Ornaadae Drver Soree
Contitiin Srowe Scoee (Te Sampa)
Candidio Betuets 7.00% Score (Phe Camral)
MOB UE
Seokai PPar and Sulind
Destsbe
JA450 Tames
(Thas Seiing) Sodae)
sen Stars (Tewe Srddy)
Comiacse Bower TL0h and Sorn the Sengi)
Connendon Strwen (L.0% Scoee (Pe Sirpt)
Contisdo Behees 7.00 and Stare (Ph Surado
MMLLL HS US Holvey
(herm purl fedind
Castske
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reteee F10P Serws phe
1P% Scmes (Phr Strpen)
Contistoo Boheer
Figure 46. MMLU High School US History: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
57

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.38. NLP Benchmark: MMLU High School World History (Hendryeks et al., 2020)
MMLU HS Word (415 Coreett Ohoki))
Vieid Panlyy
MCTT favm (Dea Soleed
Candrtox sporen
Jid00 tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
0.50
8.10
075
8.75
0.00
0.50
1o Coone Chein))
Maeid Pae
Sendes Place Data lulro)
Candedox parr
Cantedoon SAMLE
JAnDo her Tomes Per iodle)
Tylla 2014 lour (Faam. Indiig
Consition Srower Scoee (Te Sampa)
Candidio Detuets 7.00% Score (Phe Camral)
KNEE IS Worl Baary
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing)
sen Stars (Teve Srddy)
Comiatse Bower TLOh and Sorn the Sengil
Contedinn Strwen Sone (Pe Sirpt)
Contirdoo Behees 7.00% and Scart (Ph Surrlt
MNLU HE Word Hittrry
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beieee F1OP phe
1P% Scnes (Phr Strppn)
Convistio Boheer
Figure 47. MMLU High School World History: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
58

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
H.39. NLP Benchmark: MMLU Human Aging (Hendrycks et al., 2020)
MMLL heg Cornett Choie))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrdox ponnn
Coneaatoo
lodee)
Farm (Dra Sollel
Sylua 2014 baans theae sally!
32
0.7
0.50
075
8.75
goot Cooney Cheins)
Maeid Paemly
Place Data lulro)
Centrior
Cantrdoo serr
Cantedoc
JADDo her Tomes Per iodle)
Tylla 2014 low (heam. Indiig
Consitinn 7..0% Scoee (Te Sampa)
Candisio Detuets 7.00% Store (Th Samrll)
MSE Agiz (
Seokai (Par Diu Sulind
JA450 Tames
(Thas Seiing)
sene Stars (Teve Srdda)
Comiacse Bower TL0h and Sorn the Sengil
Connerion Strwen Scoee (Pe Smipt)
Contisdo Behees 7.0% and Scari (Ph Suralt
MMLU Humnn Aghnt
(herm auellrw foaind
(Sin toabe)
(The Sering
Fram (Biu Salleg
Crvelate Beteee F10P Serms phe
Scnes (Ph Strppn)
Contistoe Brhere
Sear
Figure 48. MMLU Human Aging: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
59

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.40. NLP Benchmark: MMLU Human Sexuality (Hendryeks et al., 2020)
MMLU Seuaiity 1e Corect Ohoie))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrdic spantn
Coneasoo perce
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7 830
0.00
65
6.5
075
8.75
0.00
(hr
goai Cooney Chein)
Maeid Paemly
Sendes Place Data lulro)
Candedor perree
Contedoo
JAnDo her Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
025
Grnaadae Drver Sorel
Contirion Stower 7..0% Stoee (Te Sampa)
Candidio Detuets 7.00% Store (Th Camrla)
KMLU Seauiity
Seokai PPar and Sulind
Destsbe
JA450 Tames
(Thas Seiing)
sen Stars (Tewe Srddy)
Comiacse Bower T10h and Sorn the Sengil
Contedin Strwen 7L.0% Sone (Pe Smrpt)
Contisdo Behees 7.00% and Scare (Ph Saralo
KMLU Haran Seually
(herm auel fedind
Eastske
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Seres the Sempin)
DP% Scmes (Phe Strppi)
Contistio Boheer (P% Seaen (Phe
Figure 49. MMLU Human Sexuality: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
60

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.41. NLP Benchmark: MMLU International Law (Hendryeks et al., 2020)
MMBU Inceratonal 10 Cortett Oniie)
Vaeid Pandyy
MCTT fasm (Dea Soleed
Candisoo penen
Jid0o feron tames todee)
Farm (Dra Soliel
Sylu 2014 baans theae suily!
32
0.7
830
65
6.50
8.1
075
8.5
0.00
0.50
(hr
goac Cooner Chein))
Maeid Paerly
Sendes Place Data lulro)
Cantedox serse
Contedoo
JADDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
025
Dontitinn Stower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Th Camrla)
Senkai iPar and Sulind
JAM50 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower T10h and Sorn the Senpl)
Connerion Strwen 7L.0% SoNe (Te Smrpp)
Contisdoo Behees 7.0% and Stare (Ph Surdlo
MHLU Insorational Lay
Comten (herm puel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee F10P Serws phe Sempin)
DP% Scmes (Ph Strppn)
Contistion Beheer
(P% Seaee (Phe
Figure 50. MMLU International Law: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
61

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.42. NLP Benchmark: MMLU Jurisprudence (Hendrycks et al., 2020)
MMLU keg CConect Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleel
Canentox pontn
Canelttoo perce
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae sally!
32
0.7
6.50
075
8.5
Soech Coonet Chain))
Maeid Paemly
Place Data lulro)
Contetoo WORMY
Candesoo perre
Contedoo
JADDo Tomes Per iodee)
Tylla 2014 low (heam. Indiig
Consitiin Srowen 7L.0% Scoee (Te Sampa)
Candisio Detuets 7.00% Store (Th Camrll)
MMLU Aeiguudeees
Seokai PPar and Sulind
Corsidio
Destsbe
JA450 Tames
(Thas Seiing) Sodae)
sen Stars (Tewe Srddy)
Cemiacse Bower T10h and Sorn the Sengil
Connedion Stewen 7L.0% Sone (Pe Sirpp)
Contirdoo Behees 7.00% and Store (Ph Sarrdo
MMLU Ariordtoxs
(herm auellew fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beteee F1OP
Scnes (Phr Strpen)
Contistio Beheer
Figure 51. MMLU Jurisprudence: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
62

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.43. NLP Benchmark: MMLU Logical Fallacies (Hendryeks et al., 2020)
MMLU ke Cornect Ohike))
Vaeid Pandyy
MCTT fasm (Dea Soleed
Cardrtex sonmn
Conelatoo perce
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7 830
675
6.50
8.1
075
8.5
0.00
0.50
(hr
Togical gloc Cheins)
Maeid Paerly
Sendes Place Data lulro)
Devirdor voren
Contedoo SAMEE
JAnDo herow Tomes Per iodle)
Tylla 2014 loww (heae. Indiig
025
Oenaadte Drver
Dontirinn Srower Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (Ph Camral)
MMI Engical
Senkai PPar and Sulind
JM450 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower T10h and Sorn the Sengl)
Contedin Strwen 1L.0% Sone (Pe Smipt)
Contisdoe Behees 7.0% and Score (Ph Suerlo
MMLU Logioal Falacia
(herm puel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reteee F10P Serws phe
1P% Sones (Ph Strpen)
Contistio Boheer
Sear Soeriy
Figure 52. MMLU Logical Fallacies: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
63

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.44. NLP Benchmark: MMLU Machine Learning (Hendryeks et al., 2020)
KMUU Mackes Learing keg Cornett Ohoket) Caneasoo
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtio pontn
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae sally!
0.7
6.50
075
8.5
goai Coone Chein))
Maeid Paemly
Sendes Place Data lulro)
Cantedox
Cantedoo
JApDo her Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
Consitinn Stowen 71.0% Stoee (Te Sampa)
Candidio Deruets 7.00% Store (Phe Camrlo
KMLU Leoving
Seokai PPar and Sulind
Dastsbe
JAM50 Tames
(Thas Seiing) Sodae)
sene Stars (Teve Srddy)
Comiacse Bower T10h and Sorn the Sengil
Contedinn Strwen 7L.0% SoNe (Pe Smrpt)
Contisdoo Behees 7.0% and Scare (Ph Suralt
KNLU Macked Learing
(herm peel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beteee F10P Serws phe
Scmes (Phe Strper)
Contistio Brheer
Seam
Figure 53. MMLU Machine Learning: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
64

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.45. NLP Benchmark: MMLU Management (Hendrycks et al., 2020)
MMLU 1 Coreet Onoke))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtox poannn
Jid0o tames tooee)
Farm (Dra Sollel
Sylla 2014 boans theae suily!
32
0.7
67
0.50
075
8.5
(hr
Masagreee
gaat Coonet
Maeid Paerly
Sendes Place Data lulro)
Contetor
Cantedox
Cantedoo
JinDo herow Tomes Per iodle)
Tylla 2014 lowr (haam. Indiig
025
Grnaadte Dever Sore
Dontitin Srowen 7L.0% Stoee (Te Sampa)
Candidoo Detuess 7.00% Store (The Camral)
MM. Masagrost
Senkai PPar and Sulind
JAM50 Tames
(Thas Seiing)
Sylea sen Stars (Tewe Stade))
Comiacte Bower TL0h and Soros the Senpl)
Connicion Strwen (L.0% Sooe (Pe Smrpt)
Contirdoo Behees 7.0%% and Scare (Ph Suerlo
MMLU Manggeee
(herm auellrw fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reteee F10P Seres phe
Scmes (Phe Strppnn
Contistio Beheer (P% Seae (Phe
Figure 54. MMLU Management: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
65

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.46. NLP Benchmark: MMLU Marketing (Hendrycks et al., 2020)
MMLU Markting Cornect Caneaado Jerve
Coment Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrdoo pontn
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
830
0.50
075
8.75
goat Chein))
Maeid Pae
Place Data lulro)
Candesoo parrs
Contedoo
JADDo herow Tomes Per iodee)
Tylla 2014 low (heam. Indiig
Consition Srower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camral)
MNTU Morteting (
Seokai PPar and Sulind
JAM50 Tames
(Thas Seiing)
sen Stars (Teve Srddy)
Comiacse Bower T10h and Sorn the Sengil
Connerdon Strwen SoNe (Pe Smrpi)
Contirdoo Behees 7.0% and Scart (Ph Suralo
MKLU Makding
(herm puel fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F1OP Serws phe
Sones (Ph Strpen)
Contistoo Brhere (P% Seaen
Figure 55. MMLU Marketing: Downstream performance is computed via sequence of transformations that deteriorate correla- tions between scores and pretraining compute.
66

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.47. NLP Benchmark: MMLU Medical Genetics (Hendrycks et al., 2020)
NMLU Metical Geodics ke Cooeett Ohike)) Canelatoo
Vaeid Pandyy
MCTT favm (Dea Soleed
Candntox ponnn
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
830
100
0.50
075
8.75
0.00
(hr
go Coone Cheins)
Maeid Paemly
Place Data lulro)
Certetox
Cantedox serre
Cantedoo
JADDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
Contition Stower Scoee (Te Sampa)
Candisio Detuets 7.00% Score (Ph Camral)
KMIU Geavtis
Seokai PPar and Sulind
Dastsbe
JA450 Tames
(Thas Seiing) Sodae)
sene Stars (Teve Srddy)
Cemiasse Bower T10h and Sorn the Sengll
Contedion Strwen 7L.0% Sone (Pe Smipt)
Contisdoe Behees 7.0% and Scare (Ph Surrlo
KMLU Metical Gendica
(herm auel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Seres phe
Scmes (Ph Strppn)
Contistio Brheer (P% Seaen (Phe Soerio
Figure 56. MMLU Medical Geneties: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
67

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
H.48. NLP Benchmark: MMLU Miscellaneous (Hendryeks et al., 2020)
MMLU Misce far ke Cornect Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candrdox pormn
Conelatio H0roe
Jid00 tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
830
0.50
8.75
0.20
Coone Cheins)
Maeid Pae
Place Data lulro)
Candedox
Candedoo perre
Contadoo SAMEE
JADDo her Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
Ornaada Drver
Contitinn Stewer 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Th Ctmral)
MMLU MRoc
Seokai PPar Diu Sulind
JA450 Tames
(Thas Seilng)
sene Stars (Teve Srdda)
Cemiasse Bower TL0h and Sorn the Sengll
Connerion Strwen 7L.0% Scoee (Pe Sirpt)
Contisdoe Behees 7.00 and Scart (Ph Surdio
MMLU Mecc lar DAD
(herm auellew fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P phe
Sones (Ph Strpin)
Contistio Beheer
Sear
Figure 57. MMLU Miscellaneous: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
68

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.49. NLP Benchmark: MMLU Moral Disputes (Hendryeks et al., 2020)
MKLU Meul Deoutes ke Corect Ohoie))
Vaeid Pandyy
MCTT favm (Dea Soleed
Canériox sonnn
Jid0o tames todee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.78
8.0
0.00
67
1.0
0.50
8.10
075
8.75
0.00
0.50
(hr
Foath CooNey Chaio))
Maeid Paerly
Sendes Place Data lulro)
Candedoo serr
Contedoon SOMEE
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
Oenaadte Drver Soree
Dontitinn Srewen 7L.0% Scoee (Te Sampa)
Candidio Setuets 7.00% Scare (Ph Camral)
Senkai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sen Stars (Tewe Srddy)
Comiacse Bower TL0h and Sorn the Sengil
Contedion Strwen 7L.0% Sone (Te Siipt)
Contisdoo Behees 7.00 and Scare (Ph Suralo
HILLI Morel Dexto
(herm purl fedind
Dastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serws phe
DP% Scmes (Phe Strpen)
Contictoo Boheer
Figure 58. MMLU Moral Disputes: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
69

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.50. NLP Benchmark: MMLU Moral Scenarios (Hendrycks et al., 2020)
MILU Maril Scmaress ke Cornett Ohike))
Vaei Parlyy
MCTT fasm (Dea Soleel
Candrrio Sporma
Candrdoo
Ji000 feron tames todee)
Farm (Dia Soliel
Sylu 2014 boans theae sally!
027 830
1.00
0.50
8.10
8.5
0.30
0.50
glesh Coone Chaine)
Mieid Fuerly
Sendes Placm Dat Sulrn)
Cartetox WorM
Cantedox penee
Contador
JA400 Aarow Tomes Per iodle)
Tylla 2018 lour (haae. Indiag
Genaatte Drvee Sore Sampl)
Consition Stowen 7LD0% Stoee (Te Simpl)
Candidio Detuets 7.00% Stare (The Camrak)
Senkai (Par and Suling
JM450 Tames
(Tha Seilng)
sen Stars (Peve Srada))
Ceniacse Bower TLOP and Soro the Sengil
Comnerdon Strwen 1L.0% Sonee (Pe Smrpi)
Contirdoo Behees 7.00% and Store (Ph Sterdt
MIILL Maral Scoueits
Comten (herm purl fndind
(Sin tradey
(The Sering
Fram (Diu Salle]
Crnelaiee Reiwee 110P Sere Serpic)
1P% Scmes PPt
Contistio Boheer
Staee
Figure 59. MMLU Moral Scenarios: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
70

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.51. NLP Benchmark: MMLU Nutrition (Hendryeks et al., 2020)
MMLU Nersioo keg Cooeect Coment Onoke))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candertie soannn
Jid00 tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
830
0.50 8.1
075
8.75
6.75
Ho Coonn Chaios)
Maeid Paerly
Place Data lulro)
Candedoo pears
Contedon
JADDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
025
Dontitinn Stower Scoee (Te Sampa)
Cantidio Detuets 7.00% Scare (The Camrla)
1000 Narition
Seokai PPar and Sulind
Corsidio
Dastsbe Seald
JAM50 Tames
(Thas Seiing) Sodae)
sen Stars (Tewe Srddy)
Cemiasse Bower T10h and Sorn the Sengil
Comnerdon Strwen 7L.0% Sonee (Te Smipi)
Contisdoo Behees 7.0% and Scare (Ph Suralo
MILLL Nartiion
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Reteee F10P
DP% Scmes (Ph Strpen)
Contistio Brheer
Figure 60. MMLU Nutrition: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
71

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.52. NLP Benchmark: MMLU Philosophy (Hendrycks al., 2020)
MVLU Phicoodty Cornet Corett Ohike) k4.5
Vaeid Pandyy
MCTT fasm (Dea Soleel
Canériox ponnn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
830
0.0
0.50 8.1
075
8.75
0.00
0.50
(hr
Ho Coonet Chein))
Maeid Paerly
Sendes Place Data lulro)
Candedoo S0ITO
Contedoo SIMLE
JADDo herow Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
Oenaadte Drver
Contitinn Srower Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (The Camral)
Senkai PPar and Sulind
JA450 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower TLOh and Sorn the Sengil
Contedion Strwen 1L.0% Sonee (Te Siipt)
Contirdoo Behees 7.00 and Scare (Ph Surrdt
NILL Phicuetoy
(herm purl fedind
(Sin toadey
Castsker
(The Sering
Fram (Diu Salle]
Crnelaie Reiwee F10P Seres phe Serpic)
Scmes (Ph Strppin
Contistio Beherr
Seaen (Phe
Figure 61. MMLU Philosophy: Downstream performance is computed via sequence of transformations that deteriorate correla- tions between scores and pretraining compute.
72

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.53. NLP Benchmark: MMLU Prehistory (Hendrycks et al., 2020)
MMLU Cornect Corent Choke))
k Canelsoo
Vaeid Pandyy
MCTT fasm (Dea Soleed
Candrtix soamn
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7 830
675
0.50
8.1
075
8.75
0.20
3.70
goot Coonn Chein))
Maeid Paerly
Sendess Place Data lulro)
Centrtox woree
Candedoo perre
Contedoo SAMEE
JANDo herow Tomes Per iodle)
Tylla 2014 lown (heam. Indiig
025
Oenaadte Drver
Contitinn Srower 7LD0% Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (The Camral)
Senkai PPar and Sulind
JA450 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower TLOT and Sorn the Sengll
Comnecion Strwen 7L.0% Sonee (Te Siipt)
Contirdoo Behees 7.0% and Scare (Ph Sarrlt
MNLU Prefutory
(herm puel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Serm phe
Sones (Ph Strppn)
Contistoo Brhere
Seaen (Phe Sterit
Figure 62. MMLU Prehistory: Downstream performance is computed via sequence of transformations that deteriorate correla- tions between scores and pretraining compute.
73

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.54. NLP Benchmark: MMLU Professional Accounting (Hendryeks et al., 2020)
MMLU Profesional Accouning ke 115 Coreett Ohike))
Vieid Panlyy
MCTT favm (Dea Soleed
Canerdie poren
Jid0o tames tooee)
Farm (Dra Soliel
Sylua 2014 boans theae suily!
32
0.7
1.00
6.5
0.50 8.1
075
8.5
0.00
0.50
(hr
Aveuering
Ho Coonn Cheins)
Maeid Pae
Place Data lulro)
Cartetoo
Dentetox paarse
Contedoo
JADDo her Tomes Per iodle)
Tylla 2014 lour (heam. Indiig
025
Consiriin Stower 7.50% Stoee (Te Sampa)
Candidio Detuets 7.00% Score (The Camral)
MM Profensionll Accuantng (
Senkai PPar and Sulind
Dastsbe leall
JAM50 Tames
(Tha Seilng)
sen Stars (Tewe Stade))
Comiacse Bower TL0h and Sorn the Sengil
Connerdon Strwen 7L.0% Scoee (Pe Smipi)
Contisdoo Behees 7.0% and Scare (Ph Seerlt
MMLU Profoskoal Accoarting
(herm auel fedind
Castsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serms phe
Scnes (Ph Strpen)
Contistoo Beheer
Figure 63. MMLU Professional Accounting: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
74

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.55. NLP Benchmark: MMLU Professional Law (Hendrycks et al., 2020)
NILL Profes and 101 Cornett Ohike))
Vaeid Pandyy
MCTT favm (Dea Soleel
Candific ponnn
Conelaooo
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
830
0.50
8.1
075
8.75
0.20
(hr
plec Chains)
Maeid Paerly
Place Data lulro)
Cantedoo perre
Contadoo SAMEE
JADDo herow Tomes Per iodee)
Tylla 2014 low (Peam. Indiig
025
Consiriin Stowen 7L.0% Scoee (Te Sampa)
Cantidio Detuets 7.00% Store (Ph Camrle)
MRILL Profect
Seokai PPar and Sulind
Dastsbe
JA450 Tames
(Thas Seiing) Sodae)
sen Stars (Tewe Srddy)
Cemiacse Bower TL0h and Sorn the Sengil
Connerdon Strwen 7L.0% Sone (Te Sirpt)
Contisdo Behees 7.00% and Stare (Ph Saerlt
MYLL Profao and Law
(herm peel fedind
Castsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serws phe
Scnes (Phe Strpen)
Contistoo Boheer
Sear (Phe
Figure 64. MMLU Professional Law: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
75

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.56 NLP Benchmark: MMLU Professional Medicine (Hendryeks et al., 2020)
MMLU Pufeesionl Modicie k Coonect Ohoke))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrdox ponmn
Jid0o feronn tames tooee)
Farm (Dra Soliel
Sylla 2014 boans theae suily!
0.7 830
1.00
65
0.50
8.10
075
8.75
0.00
Hoon Coone Cheins)
Maeid Pae
Sendes Place Data lulro)
Certedox
Centrdox perrs
Contedoo
JinDo her Tomes Per iodle)
Tylla 2014 loww (heam. Indiig
025
025
Oenaadte Drver Soree
Consitiin Stower 7L.0% Scoee (Te Sampa)
Candidio Setuets 7.00% Stare (Th Camrla)
MME P of
Senkai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seiing)
sen Stars (Teve Stade))
Comiacse Bower TLOh and Sorn the Sengil
Conterion Strwen 7L.0% Scoee (Pe Sirpt)
Contirdoo Behees 7.0% and Stare (Ph Saerlo
MMUU Profesional Maticie
(herm puel fedind
(Sin toadey
(The Sering
5ylu Fram (Biu Salle]
Crnelaie Beieee FL0P) Seres phe
Scmes (Phe Strppin
Contistion Beheee
Figure 65. MMLU Professional Medicine: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
76

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.57. NLP Benchmark: MMLU Professional Psychology (Hendryeks et al., 2020)
MMLU Profosional Fuckeiogr ke 1i Coreet Ohiie))
Vieid Pandyy
MCTT favm (Dea Soleed
Canértox ponen
Jid0o tames tooee)
Farm (Dra Soliel
Sylu 2014 boans theae suily!
32
0.7
830
67
1.0
0.50
8.1
075
8.5
0.20
0.50
Ho Chein))
Maeid Pae
Sendes Place Data lulro)
Cantedox perre
Cantedoo
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
Consiriin Stower 7L.0% Stoee (Te Simpa)
Candidio Detuets 7.00% Store (Ph Camrll)
MMIU Professioaal Poy ( Coornct
Senkai iPar Diu Sulind
Dastsbe
JMM50 Tames
(Thas Seiing)
sen Stars (Tewe Srddy)
Comiacse Bower TLOh and Sorn the Sengll
Connerdin Strwen 7L.0% Scoee (Pe Smipt)
Contisdoe Behees 7.0% and Scare (Ph Surddo
MMLU Profossonal Puychooog
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serms phe
Scnes (Ph Strpen)
Convistio Boheer
Figure 66. MMLU Professional Psychology: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
77

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.58. NLP Benchmark: MMLU Public Relations (Hendrycks et al., 2020)
MMLU Puelic Relaioes ke (Coeec Ohike))
Vaeid Pandyy
MCTT favm (Dea Soleed
Canditoo ponnn
Jid00 tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
0.50
075
8.75
(oact Cooney Chain))
Maeid Paemly
Place Data lulro)
Cantedox
Cantedoo
JADDo her Tomes Per iodee)
Tylla 2014 low (heam. Indiig
925
Contirion Srower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (Ph Camral)
MMLU PuRl: Rebrions
Seokai PPar Diu Sulind
JAM50 Tames
(Thas Seiing)
sene Stars (Teve Srddy)
Cemiasse Bower T10h and Sorn the Sengil
Contedin Strwen 7L.0% Sonee (Pe Smipi)
Contisdo Behees 7.0% and Scare (Ph Seeato
MNLU Public Rebatora
(herm auel fedind
Castske
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beieee F10P Serws phe
Sones (Ph Strpei)
Contistion Boheer (P% Seaen
Figure 67. MMLU Public Relations: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
78

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.59. NLP Benchmark: MMLU Security Studies (Hendrycks et al., 2020)
MNLU Securiy Studis k Cornet Ohoie)) Caneasoox
Vaeid Pandyy
MCTT favm (Dea Soleed
Canditic spanmen
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.7
830
67
100
0.50
8.10
070
8.75
0.20
(hr
(lech Coone Chains)
Maeid Pae
Sendes Place Data lulro)
Centrdix woren
Candedoo perre
Cantadoo
JAnDo herow Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
Dontitinn Srowen 71.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camral)
KMLU Seculty Studie
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing)
sen Stars (Tewe Srddy)
Comiacse Bower TL0h and Sorn the Sengll
Contedinn Strwen 7L.0% Sonee (Pe Siipt)
Contisdoe Behees 7.0% and Score (Ph Surddo
MNLU Socrrty Staiio
(herm auel fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Reteee F10P Serws phe
Scmes (Ph Strppin
Contistio Beheer
Seaen (Phe
Figure 68. MMLU Security Studies: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
79

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.60. NLP Benchmark: MMLU Sociology (Hendrycks et al., 2020)
MMLU Socilogy b Cornett Croke)
Vaeid Panlyy
MCTT fasm (Dea Soleed
Canditiox soonen
Lin0o faroon tames tooee)
Farm (Dra Soliel
Sylla 2014 boans theae saily!
32
077
830
0.00
6.5
100
0.50
8.1
075
8.75
0.20
0.50
Ho Chiino)
Maeid Paerly
Sendes Place Data lulro)
Cartedox worms
Candedox
Contadoon SAMER
JADD0 herow Tomes Per iodle)
Tylla 2014 lowa (haae. Indiig
025
Oenaadae Drver Sore
Contitinn Srowe
Stoee (Te Sampa)
Candidio Detuets 7.00% Store (Th Camrla)
Sociiagg
Senkai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sen Stars (Peve Stade))
Comiacse Bower TLOT and Soro the Sengl)
Comnendon Strwen 1L.0% Sone (Te Smrpt)
Contirdoo Behees 7.0% and Scart (Ph Suralo
MILL Socilagy
Conten (herm puel fedind
Dastsker
(Sin toadey
(The Sering
Fram (Diu Salle]
Crnelaie Reiwee F10P Seres phe Sempin)
DP% Scnes (Ph Strpin)
Contistio Brherr (P% Seaen (Phe
Figure 69. MMLU Sociology: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
80

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.61. NLP Benchmark: MMLU US Foreign Policy (Hendryeks et al., 2020)
NMLU US Fueigs Policy ke Corect Onike)
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtix ponen
Jid0o tames tooee)
Farm (Dra Sollel
Sylua 2014 boans theae suily!
32
0.78
0.00
6.5
1.00
0.50
075
8.75
0.20
0.50
(hr
goan Crone Cheins)
Maeid Paerly
Sendes Place Data lulro)
Cantrtox
Cantedox
Cantedoo SEMER
JAnDo herow Tomes Per iodle)
Tylla 2014 lowa (heam. Indiig
025
025
Grnaadae Drver Soree
Contitinn Srower 7L.0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camraa)
KMLU US Forign Pallyy
Senkai PPar and Sulind
JA450 Tames
(Tha Seilng)
sen Stars (Teve Stade))
Comiacse Bower TL0h and Sorn the Sengil
Contedin Strwen 1L.0% Sonee (Te Smrpt)
Contirdoo Behees 7.0% and Scare (Ph Sarrit
KMLU US Feeign Policy
(herm purl fedind
Dastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Retwee F1OP Serws phe
1P% Scnes (Phe Strppi)
Contistio Boheer
Figure 70. MMLU US Foreign Policy: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
81

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.62. NLP Benchmark: MMLU Virology (Hendrycks et al., 2020)
HMLD k Coreet Comes Onoke)
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrdix ponmn
Jid00 tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
32
0.7
830
67
0.50
075
8.75
0.00
Ho Coonny (hain))
Maeid Pae
Place Data lulro)
Candedoo peare
Contedon
JADDo her Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
Contitinn Stewer Scoee (Te Sampa)
Candisio Detuets 7.00% Score (Ph Camrll)
NNIU Vinlayy
Seokai iPar Diu Sulind
JAM50 Tames
(Thas Seiing)
sene Stars (Teve Srddy)
Cemiaase Bower T10h and Sorn Phe Sengil
Connerdon Strwen Scoee (Te Smrpi)
Contisdoo Behees 7.0% and Scare (Ph Suralo
NNLU Viddag
(herm peel fedind
Eastsker
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelate Beieee FLOP Serws
Scnee (Phe Strpin)
Contistoo Brheee
Seaen (Phe
Figure 71. MMLU Virology: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
82

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.63. NLP Benchmark: MMLU World Religions (Hendrycks et al., 2020)
UMUU Wurid Reggion 10 Cornct Ohike))
Vaeid Pandyy
MCTT favm (Dea Soleed
Candrtoo soaren
Jid0o tames tooee)
Farm (Dra Sollel
Sylu 2014 boans theae suily!
0.7
830
6.50
8.1
075
8.75
0.00
0.50
(hr
gloch Bolgiow
Maeid Paemly
Place Data lulro)
Cantedox perree
Condedoo
JADDo herow Tomes Per iodee)
Tylla 2014 lowa (heam. Indiig
025
025
Dontirinn Stewen 7..0% Scoee (Te Sampa)
Candidio Detuets 7.00% Store (The Camral)
MIELL Wooid Bolgioo
Seokai PPar and Sulind
JA450 Tames
(Thas Seiing) Sodae)
sen Stars (Peve Srddy)
Comiacse Bower T10h and Sorn the Sengll
Connecdon Strwen 7L.0% Sonee (Pe Sirpt)
Contisdo Behees 7.00 and Scari (Ph Sarado
MMLU Worid Rekgorn
(herm purl fedind
(Sin toadey
(The Sering
Fram (Biu Salle]
Crnelaie Beteee F10P Serw phe
1P% Scmes (Phe Strppi)
Contistoo Brheer
Sear
Figure 72. MMLU World Religions: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
83

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.64. NLP Benchmark: OpenBookQA (Mihaylov et al., 2018)
CperBookg4 10.5 10 Cornet Orike)
Vieil Pandyy
MCTT favm (Dea Soleel
Candrtox pontn
Cindatox
Canerdoo
lodke)
Farm (Dra Sollel
Sylu 2014 baans theam sudly!
32
0.7 8.0
675
0.50
075
8.75
(hr
Gpeiieaag go Chaios)
Maeid Pam
Place Data lulro)
Candetoo
Contedor
Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
025
Ornaaia Drvea Socre Sampll
Contitinn Stower 7.50% Scoee (Te Sampa)
Candidio Detuets 7.00% Score (Th Caralo
Senkai iPar and Sulind
JAM50 Tames
(Thas Seilng)
sen Stars (Teve Sraday)
Cemiacse Bower T10h and Sorn the Serpil
Comnidion Strwen 7L.0% Sooee (Te Smrpt)
Contisto Behees 7.00 and Score (Ph Saral)
OperBuekg4
(herm anellrw feaind
(Sin toabe)
(The
Fram (Bis Salleg
Ornelete Reteee P1OP) Serms
1P% Scmes (Ph Streen)
Seam (Ph Sornt)
Figure 73. OpenBookQA: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
84

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.65. NLP Benchmark: PIQA (Bisk et al., 2020)
M1AA 1ng Commec Chokee
Madi Pardyy
10.11 favn (iin budee!
fevm (ine tudee!
Tydu 2013 Teu's (Tave. Solly!
+1.0
-025
-8.0
-12
635
140
-55
-550
6.5
025
-00
-1.0
-035
-138
5.00
02)
6.30
Again Connct Choloe)
Vadia Family
(Parn Sulrd
LI400 Aarie Towes Phere Solke)
Tydka With Tours (Prse. Sollet
025
025
400
Genaaiae Drver Sorie the Sampl)
Contitin Stown
7LD0% Stree (Te Simpa)
Cantidio Detuets 7.00% Store (Th Camral)
PIAA
Vodit
Place anditee Salred
Candusok QAnMA
Candidon parte
Caneidoo seall
(5000 harves Teler Paace
(Daz-Sulng)
sinn Stans (Teve Stulay)
Sare Sampll
(1h
RGA
Conden (harm anellew fudind
HOTEE harm (Doa Soles]
24 18 Paris (DeaS))
Tou
Gevelcil Dicees ELOP) and Sore Phe Samgle
Contoraon S1ieenn 22.0% Scone (Pa Sirppe)
Canatatio Batutes 5.00% and Stare (1he Sierao
Figure 74, PIQA: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
85

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.66. NLP Benchmark: RACE (Lai et al., 2017)
RACE 1ng Commet Choka)
Madil Parlyy
14111 favn (iin budee!
fevm Bine tudee!
7ydu 2013 Tou's (Taue. Solle!
+100
-025
-8.0
-52
635
100
-85
6.5
025
5.00
02)
6.30
Comect Cholo))
Vodia Fanily
(Parn Sulrdd
LA000 Aartve Towes PPere Solke)
Pytka with Toars (Tase. Sollet
025
025
400
Oenaaiae Drver Socie the Sampl)
Contitin Stower
7LD0% Stoee (Te Sampa)
Candisio Deruets 7.00% Score (Ph Camral)
RACE
place Salled
Cardtsok QuanA
Candison JAst0
Candidoo seal
(5000 Teler Paace
(Daz-Sulngg
sinn Stans (Teve Stulay)
Serpll
(1h
RACE
(harm andllew falind
Ca didiii (p00
Candisoon
HOTEE (Doa Sobes]
24 18 Parrs (DeaSo)ing
Tou
Gevelcia Dicees ELOF) and Sore Phe Samgle
Contoraon S1ieen 2L.0% Scone (Pa Sirpp)
Canatatio Bntutes 5.00% and Stari (Ph Saerao
Figure 75. RACE: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
86

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.67. NLP Benchmark: SciQ (Welbl et al., 2017)
5d2 p Commct Choka)
Madil Pardyy
10111 favn (iin budee!
Ayler Kevm Bine tudee!
7ydua 2013 Tou's (Taue. Solly!
+1.0
-025
-8.0
-52
635
-A5
-0.30
-550
6.5
025
-00
-1.6
-035
-138
5.00
02)
6.30
Agaih Comnet Choloe)
Vadia Family
PParn Solrdd
Caniitio
Cantioon
LA400 Aartee Toues PPere Solke)
Pytka With Toars (Prse. Sollet
025
025
Ornaadae Drver Sorn Sampl)
Dontiiin Stown 7LD0% Scoee (Te Sampa)
Candisio Deruets 7.00% Score (Th Camrll)
5d2 Corret
Vodit
Place anditee Salred
Candison
Cantidoo seall
(A000 harves Teler Paace
(Daz-Sulng) sinn Stans (Teve Stalay)
Sampl)
(1h
5:0
Conden (harm anellew fudind
HOTEE harm (Doa Soles]
2446 18 Paris (DeaS))
Tou
Gevelcia Bicees ELOP) and Sore Phe Sangle)
Contoraon S1ieen 22.0%% Scone (Pa Sargpa)
Canatatis Batutes 5.00% and Stami (1he Sieraoo
Figure 76. SciQ: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
87

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.68. NLP Benchmark: Social IQA (Sap et al., 2019b)
( Coreet Oncke)
Vaeid Panlyy
MCTT fasm (Dea Soleel
Cardrfox pontn
Creidio uaiil
Jid00 feroon tames todee)
Farm (Dia Soliel
Sylua 2014 boans theae sudly!
32
027
830
0.50
8.1
075
8.5
0.70
(hr
Eoesh Cooney Chein)
Maeid Puerly
Sendes Place Data lulrn)
Cantrror
Candetoo perve
Contadoo WAME
JMND0 harow Tomes Per iodle)
Tylla 2018 loww (haae. Indiag
925
Grnaada Orver Sorn
Dontitin Stowen 71.0% Stoee (Te Simph)
Candiddo Detuets 7.00% Store (The Camral
Senkai PPar and Sulind
Dastsbe
JA450 Tames
(Tha Seilng)
sin Stars (Peve Stade))
Comiaase Bower TLOF and Soros the Sempll
Connerdon Strwen 7L.0% Sooee (Te Smipt)
Contirdoo Behees 7.00% and Score (Ph Sterdo
Sccial Interecion QA
Comten (herm purl fedind
(Sin toabey
term (The Sering
Fram (Diu Salle]
Crnelate Releee F10P) Seres phu
1P% Scmes (Phe Strppn)
Convistio Beherr (P% Staee (Phe
Figure 77. Social IQA: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
88

---

Why Has Predicting Downstream Capabilities of Frontier AT Models with Scale Remained Elusive?
H.69. NLP Benchmark: Winogrande (Keisuke et al., 2019)
Wenggedd 1 Cornet Orike)
Vieil Pandyy
MCTT favm (Dea Soleel
Caneadox percee
lodke)
Farm (Dra Sollel
Sylu 2014 baans theam sudly!
0.7
830
675
100
6.50
075
8.75
0.20
Mesggaade goac Chaio))
Maeid Pam
Sendes Place Data lulro)
Cantedoo
Contedor
JADDo Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
400
Ornaaite Drvea Socie
Dontitin Stowen 7..0% Scoee (Te Sampa)
Candidio Detuets 7.00% Scare (Th Caralo
Winggaad
Seakai iPar and Sulind
JA450 Tames
(Tha Seiing)
sene Stars (Teve Sraday)
Cemaacse Bower T10h and Strn the Sengil
Comnicion Strwen
7L.0% Scaee (Pe Smrpt)
Contisio Behees 7.0% and Scar (Ph Sural)
Wenggadde
(herm auellrw feaind
(Sin toabe)
(The Sering
Fram (Bis Salleg
Orvelate Beteee FLOP Serws Sempin)
Scmes (Ph Streen)
Contistoe Brheee
Seam (Ph Sordt)
Figure 78. Social IQA: Downstream performance is computed via sequence of transformations that deteriorate correlations between scores and pretraining compute.
89

---

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
H.70. NLP Benchmark: XWinograd English (Muennighoff et al., 2023)
Miloogedd k4 1 Coreett Onike)
Vaei Pandyy
MCTT favm (Dea Soleel
Candrfox pontn
Condatoo perce
lodke)
Farm (Dra Sollel
Sylu 2014 baans theam sudly!
32
0.7
830
0.0
67
1.0
0.50
075
8.75
0.20
goah Chaios)
Maeid Pae
Place Data lulro)
Contadoo leeil
Tomes Per iodle)
Tylla 2014 low (heam. Indiig
025
025
Ornaaia Socre Sampl)
Dontitin Stower 7.50% Scoee (Te Sampo)
Candisii Detuets 7.00% Score (Th Carrlo
KWkaggal
Senkai iPar Diu Sulind
JAM50 Tames
(Thas Seilng)
sen Stars (Teve Srada)
Cemiacse Bower T10h and Strn the Sengil
Comnindon Strwen 71.0%% Scoee (Te Smrph)
Contisdoo Behees 7.00 and Score (Ph Suralo
XWWongrad-B EN
(herm auellrw feaind
(Sin toabe)
(The
Fram (Bis Salleg)
Orveline Reteee FLOP Serw phe
Somes (Ph Strein
Contistoo Brheer
Figure 79. XWinograd English: Downstream performance is computed via a sequence of transformations that deteriorate correlations between scores and pretraining compute.
90


Reviews:
[{"review_id": "F59070TgHN", "reviewer": "Reviewer_AAE5", "summary": "The paper studies the relationship between model scale and downstream task performance, to understand why it has been hard to formulate a \"scaling law\" to describe the relationship unlike known results for pretraining performance.\nThe paper conducts extensive empirical analyses on many tasks, benchmark datasets, model families and compute scales, to derive a comprehensive picture about their relationship to task performance.\nThe paper proposes an interesting explanation when tasks are specialized to multiple choice questions (MCQ) and back it with their analyses, that there are transformations going from pretraining loss to downstream MCQ performance that systematically degrade the statistical relationship between the two.\nThe paper also conjectures that measuring and accounting for the log likelihoods of incorrect answers in MCQs may yield a more stable relationship between model scale and task performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "There are 3 main claims in the paper.\n1. Downstream task performance does not have a predictable relationship with the scale of the model, whereas pretraining loss does. The claim can be scoped better by emphasizing that the downstream tasks are of multiple-choice question type. The evidence for the claim can be strengthened by including a plot showing task performance vs model scale (and contrasting it with the plots in the related literature for pretraining scaling laws). \n\n2. As we go from the pretraining loss to the MCQ task objective, each transformation step weakens the statistical correlations with model scale. The evidence contributed by the paper for this claim is very comprehensive!\n\n3. The variations in probability of incorrect choices do not \"cancel out\" when averaging across datapoints in a dataset. And modeling those probabilities (at a per sample granularity) is key to better prediction of downstream performance metrics as a function of model scale. The evidence for this claim is indirect at best. More direct lines of evidence may be to control for the probability of incorrect choices, and check whether there is a more predictable relationship between task performance and model scale after controlling for confounding.", "methods_and_evaluation_criteria": "The evaluation methods are sound.", "theoretical_claims": "There are no theoretical claims in the paper. There is a broad claim about how aggregating the incorrect choice probabilities across independent realizations does not lead to a less noisy estimate (unlike in Monte Carlo estimation); this may be good to justify theoretically.", "experimental_designs_or_analyses": "The experiment design is sound, and quite an accomplishment in how thorough the empirical results are reported.", "relation_to_broader_scientific_literature": "The related work section in Appendix is good; I wondered if some abridged version of it could be moved to the main paper (perhaps streamlining Sections 1 and 2 in the process).", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "Fig 5 center: The comment about \"0 < P_choices(correct choice) < 0.5 contains little information about accuracy\" is a little inscrutable. The plot shows not much dispersion at all for \"Fraction of Accuracy=1\", unlike Fig 5 left. I can see two explanations: one is that to understand the claim we need to look at the composition of Fig 5 left and Fig 5 center, and realize that a large dispersion in Fig 5 left has large downstream effect. The other explanation is that Accuracy is a binary variable (when measured per datum); there are only 2 values for accuracy and each accuracy value maps to a large spread of P_choices(correct choice) values.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HhWUwIaJvn", "reviewer": "Reviewer_M3MH", "summary": "This paper explores why predicting the downstream performance of advanced AI systems, especially on multiple-choice question-answering benchmarks, is difficult despite well-understood scaling laws during pre-training. The key finding is that downstream performance degrades as it involves comparing the correct choice to a small set of incorrect choices, making it harder to predict how probability mass shifts with scale. They highlight that to predict downstream capabilities accurately, it’s essential to account for how probability mass fluctuates between correct and incorrect choices as models scale. The study suggests that scaling laws for incorrect choices may be achievable.\n\n\n## update after rebuttal\nMy main concern still stands. I believe the study of \"unpredictable\" scaling laws is narrow scoped towards MCQ benchmarks. And even there the finding is rather superficial: probability mass on incorrect samples increases unpredictably. It is unclear why this happens. The authors agree, that eventually if we scale enough then this should not be a concern since pre-training loss is a consistent objective. If so, then is that point (where we enough data to predictably scale performance on MCQ) itself predictable? Many questions like these remain unanswered. Because of this, I am leaning towards a weak reject, **though after reading other reviews, I am willing to raise my score by 0.5 to 2.5, and would not be opposed to accepting this paper if other reviewers are willing to champion it**.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thorough and thoughtful review. We greatly appreciate that you found our analysis \"interesting,\" \"quite comprehensive,\" and that it \"gets at the main cause\" of unpredictability in downstream capabilities. We're particularly pleased that you found our story \"presented in a clean way\" and that the \"paper is easy to read.\" Thank you also for highlighting Figure 3 as effectively demonstrating the degradation of predictive power on the ARC benchmark.\n\n### On Specificity to Multiple Choice Question Answering (MCQA) Benchmarks\n\nWe agree that our focus on MCQA benchmarks represents a scope limitation. We are upfront about this but made this decision for several reasons:\n\n1. MCQA benchmarks remain a widely used evaluation format for frontier models due to their objective scoring and standardization. They're employed by major benchmarking efforts (e.g., MMLU, HELM) and by leading labs to track progress.\n2. The MCQA format is particularly valuable for investigating scaling predictability because it allows us to isolate specific transformation steps in a controlled manner. This methodological clarity helped us identify the core mechanism behind unpredictability.\n3. We view our manuscript as contributing to the science of scaling-predictable evaluations. While any given task or type of task will have its quirks, our paper shows how one can dive into the mechanisms of apparent unpredictability For example, new work on studying predictable scaling behavior of inference-time compute (https://arxiv.org/abs/2502.17578) also applied a per-sample analysis, albeit in a different context. We will add a citation to this work and clarify how analyses like ours can be applied to generative evaluations.\n\n### Fundamental Limitation or Temporary Obstacle\n\n> It mentions that the problem mainly stems from unpredictable movement of probability mass on the incorrect options, but does not discuss any ways in which this can be fixed, or is this a fundamental limitation?\n\nThis is certainly not a fundamental limitation and we have subsequent work showing how to overcome this limitation by predicting how probability mass changes on incorrect choices. However, demonstrating and dissecting the problem thoroughly is itself already quite long (as you can tell by our manuscript). We will move our Future Directions section into the main text if accepted.\n\n### Clarification of How Mass Changes on Incorrect Choices\n\n> Intuitively, one might imagine that the probability mass on all incorrect samples should reduce monotonically with scale...\n\nThis was our belief initially, but probability mass on incorrect choices almost always increases with compute, at least in the models we were able to study. For example, suppose we have a question about what pet a child has. As models train, (1) they place more mass on syntactically valid words and phrases, so if an incorrect option is \"airplane\", its mass will still increase relative to a randomly initialized model, even though airplane isn't a valid pet ; (2) they place more mass on semantically plausible alternatives e.g., \"cat\" can be a likely pet, even if it isn't the correct option for this question; (3) they learn the nature of MCQA tasks and place more mass on the available options regardless of content.\n\nAt some point, probability mass on incorrect choices _must_ decrease, but it seems the models we studied are not close to this tipping point.\n\n### Responses to Specific Questions\n\n> In Figure 6, do the trends change depending on the number of incorrect options present for each question?\n\n> For True/False questions, is the scaling more predictable?\n\nThis is an excellent question. Most widely-used MCQA NLP benchmarks have 4 options. MMLU-Pro (https://github.com/TIGER-AI-Lab/MMLU-Pro) but offers 4-10 choices per question, but was publicly released after our data were collected. We expect that if there are more incorrect options, scaling is less predictable and if there are fewer options (e.g., True/False), scaling would be more predictable. We will add this to our Discussion.\n\n### Visual Presentation Improvements\n\nWe appreciate your feedback about plot label sizes. We will increase font sizes in all figures.\n\n### Conclusion\n\nWe believe addressing these concerns will substantially strengthen the paper and hope they address your \"weak reject\" assessment. Thank you again for your constructive feedback, which has genuinely helped us improve our work.", "claims_and_evidence": "- There are some scaling-predictable quantities (from parameters, data, compute), like log-prob over vocabulary. But, these under go transformations, that make the scaling predictable quantities less predictable.\n- Downstream performance depends a lot on the probability mass assigned to the incorrect choice, with scale. Suggests that we need to model external information different from scale-predictable quantities like log prob.\n- Continuous metrics like Brier score are insufficient to predict downstream performance.", "methods_and_evaluation_criteria": "Yes, the paper uses some common benchmarks and model families (though it does not cover Llama or Gemma model families which are common open source models).\n\n- The paper conducts scaling analysis on a comprehensive set of model families with multiple models (scaling data, parameters and compute), including Pythia, Cerebras-GPT, OLMo, INCITE, LLM360.\n- The work only evaluates downstream performance on NLP benchmarks, for which it uses: ARC, HellaSwag, MathQA, MCTACO, MMLU, OpenbookQA, PIQA, RACE, SciQ, SIQA, WinoGrande and XWinoGrad En. \n- The performance is measured in terms of accuracy, Brier score, and probability mass on the correct choice.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "Their experimental protocol is stated cleanly -- I really like their Figure 3, which presents the degradation of predictive power on the ARC benchmark. In general, they follow the following protocol.\n\n- The authors compute the correlation between score and compute, as we scale compute in a given model family. \n- The score metric is one of the following: p(correct choice | vocab), p(correct choice | given other choices), accuracy, brier score.\n- Then, they plot the distribution of this correlation across samples. \n- Ideally, this distribution should be concentrated at values close to 1.0.\n- But, they find that everytime some transformation is applied on the log-probs predicted by the model, the correlation degrades, and the distribution of this correlation shifts away from 1.0.\n\nThey find that the main reason for the above degradation in correlation is the probability mass on incorrect choices.", "relation_to_broader_scientific_literature": "The paper talks about predictable downstream performance, as a function of scaling data, parameters and compute for pre-training. Existing analyses in this space has been centered on predicting loss (NLL) on test-data, either in data rich or data constrained settings. From that perspective, this analyses is novel, but the concern is that it is too narrow (only focused on MCQ), and only presents some concerns, as opposed to also presenting practically implementable fixes to make the downstream performance more predictable.", "essential_references_not_discussed": "- Language Models (Mostly) Know What They Know (Kadavath et. al.), also conducts an analysis of performance and calibration on multiple choice questions benchmarks, albeit they don't focus on predictable scaling.", "other_strengths_and_weaknesses": "Strengths:\n- The analyses of downstream prediction capabilities on MCQ benchmarks is quite comprehensive, and it gets at the main cause: the probability mass on incorrect outputs varies in unpredictable ways as we scale compute.\n- The story is presented in a clean way, and the paper is easy to read. \n\nWeaknesses:\n- The analyses and findings are very heavily focussed on MCQ benchmarks. It is unclear why or how the predictability scales on open-ended QA benchmarks, or reasoning problems.\n- It mentions that the problem mainly stems from unpredictable movement of probability mass on the incorrect options, but does not discuss any ways in which this can be fixed, or is this a fundamental limitation? \n- Intuitively, one might imagine that the probability mass on all incorrect samples should reduce monotonically with scale, so why does it increase on some, and decrease on others? The paper does not explain this. An investigation of this underlying cause seems needed to really understand if the lack of predictability is fundamental, even for MCQ benchmarks.\n\nIn general, I feel that the analysis is interesting. It mechanistically and systematically explains why the correlation degrades but somehow leaves the reader wanting by the end of Section 6, at which point it still remains unclear why the probability on incorrect samples has a high and un-predictable sensitivity with scale. Also, it is unclear how to transfer the analysis or insights to non MCQ benchmarks (I did look at the discussion in Appendix B which mainly defers discussion to future work). Because of these two points, I am leaning towards a weak reject, but if the authors can add additional discussion on these points, I would be happy to re-consider my score.", "other_comments_or_suggestions": "- The plot labels (result plots) are very small and hard to read.", "questions_for_authors": "- In Figure 6, do the trends change depending on the number of incorrect options present for each question. For example, if there are more incorrect options, then is the join probability distribution less correlated?\n- How is the analysis affected by the relationship between correct and incorrect options? For example, if the incorrect answers are close negatives, vs. being very different negatives.\n- For True/False questions, is the scaling more predictable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IaZBAiauB9", "reviewer": "Reviewer_Zmym", "summary": "The paper investigated why we can not accurately transform the scaling laws from the general negative token likelihood to the downstream accuracy of multi-choice QA. \nThe main reasons are accountable for the phenomenon are 1. there is a sequence of transformation from NLL to accuracy; 2. the positive correlation of scaling up compute and wrong choice probabilities.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Zmym for their thoughtful assessment of our work, particularly noting that our paper is \"pretty clear\" and addresses an \"important\" question with \"good experimental settings.\"\n\n### Quantifying Score-Compute Decorrelation Per Transformation\n\n> Since we know that the trasnformations bring the detoriation to the correlation, I would like to how how much each step contribute to this.\n\nThis is an incisive question. While the answer depends on the model family and the benchmark, Figure 4 suggests the answer, but  we can be much more quantitative. We will create a new figure with 4 subfigures, one for each of the statistics of the score-compute correlation distributions (mean, median, AUC of CDF, and negative Wasserstein distance). The x axis will be the sequence of metrics, the y axis will be the statistic’s value, and the hue will be the model family. This will show exactly how much “predictability” is lost under each transformation.\n\nIf the paper is accepted, we will include this detailed quantitative breakdown in the final version, including a new table summarizing these contributions across benchmarks.\n\n### On Visualizing Distributions\n\n> using a pdf is way more straightforward than using a cdf, people can well distinguish where there is of high density. \n\nWe appreciate the suggestion about using PDFs instead of CDFs. We actually explored both approaches during our analysis. While PDFs can be more intuitive in some contexts, we found that CDFs (specifically complementary CDFs) offered three advantages for our particular analysis: (1) they avoid bandwidth parameter tuning artifacts that emerged with KDE-based PDF estimation; (2) they directly visualize the tail behavior critical for our analysis; and (3) they more clearly quantify \"what fraction of samples have correlations above X\" - the key relationship we wanted to highlight. We'd be happy to include a comparison of both visualization approaches in an appendix if you think this would benefit readers.\n\n### Terminology\n\nThank you for suggesting clearer wording. We used 'fraction of accuracy=1' to precisely distinguish between two concepts: (1) the proportion of individual samples where the model answered correctly (binary per-sample outcome), versus (2) the overall accuracy score across the dataset (continuous value between 0-1). While mathematically equivalent, maintaining this precision helps readers follow our sample-specific correlation analysis. We'll revise this wording to improve clarity while preserving this distinction.\n\n### Request for Additional Constructive Criticism\n\nWe value your assessment and would appreciate any additional specific suggestions that might strengthen the paper. In particular, are there specific analyses or clarifications about the transformation process that would make our findings more compelling or accessible to the broader ML community?\n\nWe appreciate your 'weak accept' assessment and are committed to strengthening the paper to earn a more enthusiastic endorsement. We believe the quantitative breakdown of correlation degradation across transformations will substantially enhance the paper's contributions and clarity.\n\nThank you again for your constructive engagement with our work.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes, the author runs through the popular LLMs with middle pertaining checkpoints and popular QA benchmarks. \nIt is a good experimental setting for the question they would like to explore.", "relation_to_broader_scientific_literature": "I think it is highly related to the literature on how to evaluate the LLM during training.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Paper is pretty clear and the question they tried to investigate is important.", "other_comments_or_suggestions": "The paper is pretty refined. But I do think the authors complicates things in several ways. \nFor example, using a pdf is way more straightforward than using a cdf, people can well distinguish where there is of high density. \nAnd also, the using of wording \"the fraction of accuracy=1\" I believe \"accuracy\" would be enough.", "questions_for_authors": "Since we know that the trasnformations bring the detoriation to the correlation, I would like to how how much each step contribute to this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper focuses on Multiple Choice Question-Answering benchmarks, and explores why performance on those benchmarks do not predictably improve with scale. The reviewers agreed that the paper studies a well-motivated problem, and also that the paper overclaims its scope ("downstream tasks" as too broad compared to the MCQ tasks actually studied). All of the reviewers also appreciated the thorough empirical analysis. There were lingering questions around the "why" that the authors identified (that probability mass on incorrect options varies unpredictably as the models' scale increases); addressing those questions will substantially strengthen the paper.

(Regarding reviewer scores, one of the reviewers indicated that their "weak reject" score was actually an on-the-fence rating and that they were willing to accept the paper if the other reviewers reached that consensus. Taking that into account, there is broad consensus among the reviewers.)
Decision: Accept (poster)