Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
Authors: David Heineman
Valentin Hofmann
Ian Magnusson
Yuling Gu
Noah A. Smith
Hannaneh Hajishirzi
Kyle Lo
Jesse Dodge
Venue: NeurIPS 2025 spotlight ()
Keywords: language models
benchmarks
evaluation
Primary area: evaluation
TL;DR: Measuring and improving the signal-to-noise ratio in language model benchmarks.

Abstract: Developing large language models is expensive and often involves making decisions with small experiments, typically by evaluating on large, multi-task evaluation suites. In this work, we analyze specific properties which make a benchmark more reliable and useful for such decisions, and interventions to design higher-quality evaluation benchmarks. We introduce two key metrics that show differences in current benchmarks: signal, a benchmark’s ability to separate better models from worse models, and noise, a benchmark’s sensitivity to random variability between training steps. We demonstrate that benchmarks with a better signal-to-noise ratio are more reliable when making decisions at small scale, and those with less noise have lower scaling law prediction error.  These results suggest that improving signal or noise will lead to more useful benchmarks, so we introduce four interventions designed to directly affect signal or noise.  For example, we propose that switching to a  metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and scaling law error. We also find that filtering noisy benchmarks such that they have better signal-to-noise ratio leads to more reliable evaluations. We also find that averaging the output of a model's checkpoints to reduce noise leads to consistent improvements. We conclude by recommending that those creating new benchmarks, or selecting which existing benchmarks to use, aim for high signal and low noise.  We use 30 benchmarks for these experiments, and 465 open-weight language models from 60M to 32B parameters, resulting in a new, publicly available dataset of 50K evaluation benchmark results, totaling 200M instances.

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation
David Heineman" Valentin Hofmann" Ian Magnusson Yuling Gu" Noah A. Smith" Hannaneh Hajishirzi" Kyle Lo" Jesse Dodge
Allen Institute for Artificial Intelligence
"Paul G. Allen School of Computer Science & Engineering, University of Washington contact: davidheallenai.
Abstract
Developing large language models is expensive and involves making decisions with small experiments, typically by evaluating on large, multi-task evaluation suites. In this work, we analyze specific properties which make benchmark more reliable for such decisions, and interventions to design higher-quality evaluation bench- marks. We introduce two key metries that show differences in current benchmarks: signal, a benchmark's 's ability to separate better models from worse models. and noise, benchmark's sensitivity to random variability between training steps. We demonstrate that benchmarks with better signal-to-noise ratio are more reliable when making decisions at small scale, and those with less noise have lower scaling law prediction error. These results suggest that improving signal or noi se will lead to more useful benchmarks, so we introduce three interventions designed to directly affect signal or noise. For example, we propose that switching a metric that has better signal and noise (e.g., perplexity rather than accuracy) leads to better reliability and improved scaling law error. We also find that filtering noisy subtasks, to improve an aggregate signal-to-nois ratio, Icads to more reliable multi-task evaluations. We also find that averaging the output of model's 's intermediate checkpoints to reduce noise leads to consistent improvements. We conclude by recommending that those creating new benchmarks, selecting which existing benchmarks to use, aim for high signal and low noise. We use 30 bench- marks for these experiments, and 375 open-weight language models from 60M 32B parameters, resulting in a new, publicly available dataset of 900K evaluation benchmark results, totaling 200M instances.
all allenai/siinall-ndd-ns
dataaset//alleeaa/siignaaaaad
Introduction
Language model development is expensive. During the development process, researchers need to make decisions such as what architecture to use, what training methods to employ, and what data to train on. These decisions rely on measuring phenomena smaller, more economical scales, then hoping the trends measured hold for large scale models. This paradigm exists across the research community; many papers experiment with small baselines then scale up the perforrrronn model [7 38) inter alia], and there has been extensive research on using scaling laws to predict the performance of larger models 9 inter alia]. While there large and ever-growing number of benchmarks, prior work has shown these scaling procedures only works for some benchmarks and not others [66] 15 50 This poses significant challenge because, as we develop more general-purpose language models, developers need to be evaluating on even more diverse benchmarks, some of which may not be well-suited for this critical approach. We need a deeper understanding
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

SellaSwag (low noise, low signal)
ARC lenge thigh noise, high signal)
MMLU noise, high signal)
0.65
125 exporaa
(25 copoitt
125 oxpin)
0.62
Fru chudlpo 18
Fru dhacopon
That Stecstect
0.3-
Ngh
0.34
gral
Accureyy
0.52
pecuurtty
0:40
Accurs
0:32
0.4-
0:335
0.30
0.43-
Theawe
0.25
cane
low
0.33
5,575
0.25
0.35
0.20
Compute
Compute
Compute
Figure 1: Training curves for the 25 pretraining corpora in DataDecide 38 on three development benchmarks across different model sizes the ordering of different model pre-training corpora, shown by different colors, at a small scale (e.g.. 150M) should agree with ordering at larger scale (IB), implying better decision accuracy. We hypothesize that one indicator of decision accuracy is the ratio between the si signal (main plot) and the noise of scores within single training run (inset axis). In this work, we quantify the signal-to-noise ratio at different compute scales, and in later sections, show that it is predictive of large scale phenomena like decision-making error.
of what intrinsic properties can measure to tell if benchmark provides useful information, if it needs to be reformulated, or if it is best discarded altogether.
To formalize this setup, we study two common experimental settings for language model development: (i) train pair of small models (e.g.. on different pretraining corpora) and use their ranking to predict the ranking of two large models (38) and (ii) fit scaling law on set of small models and predict the performance of a large model 19 3.. We hypothesize that the ability to predict both settings are related to a measure which is cheaper to compute and casier to improve: signal and noise. Signal measures how spread out scores are for different models on single benchmark and noise measures the variability of benchmark score during training.
To illustrate the connection from signal and noise to an experimental setting, consider an example of comparing models trained using different pretraining corpora (illustrated in Figure []: the tasks where scores are either too close (HellaSwag, left) or too noisy (ARC Challenge, center) are the benchmarks where we would be less confident that a ranking of models at a small scale would hold at a large scale. Following this observation, we show in Section that the s signal-to-ne to- ratio (SNR) is highly correlated with the likelihood that ranking of models at small scale will hold at large scale, and then show that noise is highly correlated with the prediction error of a scaling law fit.
Based on these observations, in Section we propose a set of interventions designed to reduce noise or increase signal, and then we measure their impact on our experimental setups of decision accuracy and scaling law error. For example, we show that by averaging out the checkpoint-to checkpoint noise for a model, we improve our ability to predict performance of large models from small models. We also show that it is possible to find subsets of existing benchmarks that have higher signal-to-ne ggal-to-noise se ratios than the full evaluation sets, and that even though those subsets can have fewer than half as many instances, they improve both experimental setups. Finally, we show that SNR can be used to improve metric construction, where choosing a metric that has better SNR leads to consistent improvements on wide variety of benchmarks.
Our core contributions are as follows: (i) we introduce definitions for signal, noise, and signal- to-noise ratio in the setting of evaluating language models, and show this framework is useful for measuring the utility of benchmarks, and (ii) we demonstrate interventions based on this framework which improve both prediction settings. Our core results evaluate 465 language models on 30 benchmarks across model sizes. We release our data, evaluation results, and trained models.
2 Predicting Large Model Phenomena with Small Models
Using small scale experiments to make predictions about large model behavior is ubiquitous in language model development 27 finding a good mix of data from multiple sources to train on typically involves evaluation of small models to calculate an optimal weighting of datasets, then training large model on the optimized
60
31 42 This process can take many forms. For example,

---

mix [35 66]. In Blakeney et al. [5]. mid-training runs on a sample of candidate pretraining datasets are used to estimate the quality of training from-scratch. Dubey al. [77] predicted the downstream task using scaling laws to compare candidate data mixes. Hyperparameter transfer methods, such as maximal update parametrization (iPP), also rely on small scale experiments [68] However, the results from small scale experiments are not always reliable. Work on so-called emergent capabilities [65] shows that for some benchmarks, language model performance only rises above random chance for models trained at large compute budgets. Later work has further explored emergence behavior in particular tasks, such as MCQA tasks [67] or generative math and code tasks [57] or by observing the capabilities of open-weigh models [51].
While these different experimental setups are all important, we focus on two straightforward and common setups in making data decisions for language model development: decision accuracy and scaling law prediction error. this section, we present the motivation for both experimental settings, and in Section 3 we show how the signal-to-noise ratio is an effective framework for predicting how useful a benchmark in these scenarios.
2.1 Decision Accuracy and Scaling Law Prediction Error
Decision Accuracy. Consider a scenario where practitioner intends to train large model, and needs to decide between training on Dataset a or Dataset 6 to get the best performance on some downstream task, represented by a scalar B(.). A simple and intuitive approach is to train small model 8a on Dataset a and another, Sb, on Dataset b, then choose the dataset that led to the best downstream task performance for training the large model. We evaluate this procedure by training two large models, m, and my. one on each of the datasets, and see if the ranking of the two small models, Sn and 8b, the benchmark is the same for the large models [1 In the scenario where we are deciding between more than two choices, we consider pairwise rankings between all pairs P. Following Magnusson et al. (38] we refer to this small-to-large agreement as "decision accuracy":
Decision Accuracy =
Σ
I [siign - B(ss)) = sign( ((m.)) - B(ms))] (1)
(a.D)EP
We use models of 7 sizes (from 60M parameters up to IB parameters) trained on 25 different pretraining corpora from Magnusson al. (38]. Our prediction task to use set of small models (e.g., 60M parameter models) to predict the ranking of the 1B models on a given benchmark (e.g., MMLU). High decision accuracy means the ranking of the small models accurately predicts the ranking of the large models on that benchmark; this is an indication that the benchmark is useful for this process of using small models to make decisions about which dataset to train on. We illustrate an example of this in Figure L which shows training curves for 25 data recipes on 3 model sizes. We hypothesize that if model scores are very close together, or the evaluations are very noisy, it is more likely that the ranking from small to large models will change. leading to worse decision accuracy; we formalize and test this hypothesis in the following sections 2
Scaling Law Prediction Error. Scaling laws 227 24 inter alia] have been used extensively to predict the validation loss of large model using set of smaller "scaling law" models. Recent work has also used scaling laws to predict downstream task performance [19 3 by first predicting task loss then using the predicted loss to predict task performance (c.g.. accuracy); this is the setup we use this work. The prediction error for the scaling law fit is defined as the relative error between the predicted
and true performance of the large model: Prediction Error
|Measured Value True Valuel
[Tre Valuel
Calculating prediction error requires training a set of scaling law models on the same corpus with varying tokens/sizes (c.g., 190M to IB params), training a large model (c.g.. 13B), and fitting a scaling law to the smaller models predict the larger model performance We describe the scaling law functional form and fitting details in App. A. following the setup in Bhagia et al. B
TTriniing multiple large models is too expensive for most development scenarios, but is necessary evaluate how accurate this process is.
We observe similar findings on other rank agreement metrics, like Spearman rank correlation (Table
Decision accuracy. in particular, is equivalent to Kendall's tau modulo a scale and shift (App. (Ap..A.2))
*Scaling law predictions can be used to make development decisions (e.g., about which training dataset is best) by training set of models and fitting a scaling law for each option being considered [17] but in this work we just evaluate scaling law error directly.
3

---

2.2 Evaluation Dataset
We perform our analysis using existing development benchmarks and models:
Models. Our set of models includes: (i) a suite of scaling law models from 190M to 3.2B, with a corresponding target at 7B and 13B 3 (ii) suite of 25 models each trained with different pre- training corpora from 60M to 1.3B [38, (iii) the final 30 checkpoints for OL.o 2 1B, 7B, 13B and 32B 42. and (iv) 73 open-weight base models. Additionally, in our comparison between sources of modeling noise in $3.1, we train and release 20 IB models, with 10 models trained varying the data order initialization and 10 varying the random seed initialization, along with evaluation on 3.2K intermediate checkpoints.
Benchmarks. We evaluate 30 development tasks which we categorize knowledge QA, math, and code. We use the OLMES [22] standard where applicable, and reproduce the OLMo 2 evaluation setup 42 for all other benchmarks. Following Gadre et al. [19], we also include mulli-task averages for each group, and for the OLMES core tasks. For our test of subset selection in $5.1
we include a
synthetically generated benchmark, generated using AutoBencher [32
We include full details on the sets of models and benchmarks in App. A.5
3 Quantifying Signal and Noise
To illustrate the impact of noise on a decision-making setup, Figure Ishows training curves for 25 IB models trained with different data recipes and, in inset plots, the training curve for a single IB model on three tasks. Some tasks (left, HellaSwag) exhibit low noise between training checkpoints but low signal between models, and others (center, ARC-Challenge) exhibit high noise and high si signal. In this section we define signal and noise, and define two simple metrics to estimate the signal-to-noiss ratio that can be calculated from a set of model evaluations on given benchmark.
3.1 Measuring Noise
There are numerous sources of noise in the language model development pipeline. Previous work has shown multiple training runs under the same configuration can lead to different performance as a result of different initialization or data order 14 13.. In addition, as illustrated Figure [ performance can even vary significantly from one checkpoint to the next: within the final 30 checkpoints of training for 1B models on ARC Challenge, we observe a range of 1.7% accuracy. With these motivations, we consider four potential noise measurements, each calculated on using evaluation on a single benchmark: (i) training multiple models and varying only the random initialization, (ii) training models and varying the training data order, (iii) measuring the total checkpoint-to-cheekeee noise across a full, single training run, and (iv) measuring the chackpoint-to-cheekpoin noise of the final n checkpoints of a single training run. We formalize these definitions in App. A.3
To get estimates for four potential sources of noise, we train 10 different IB-5xC models varying the initialization and data orders, and evaluate all intermediate checkpoints. We find that the initialization noise, data order noise, and checkpoint- weeeckponntto-chcckppeee to-checkpoint noise across the whole training run all correlate highly with the relative standard deviation of the final checkpoints (R² of 0.82, 0.86, and 0.95, respectively, see Figure and see the training curves in Figure(1) These results lead us to define noise se as the relative standard deviation of the final checkpoints, as this requires no additional training cost and only uses the final n checkpoints rather than the full training curve. We define
noise as: Rel. (td.(m)
En (m, - m)² /m.
3.2 Measuring Signal
A benchmark is most useful during language model development if it can detect true difference between good model and a poor model, assuming a true difference exists between the models the ability that the benchmark aims to measure. This statistical power is what enables us to use small models for development decisions like training dataset to use. To formalize this idea, we consider a benchmark to have high signal when models evaluated on it have wide and evenly distributed range of scores. We measure signal using a metric from the numerical integration literature: dispersion, calculated as the maximum difference between the scores of any two models, divided by the mean

---

Signal
Noise
Signal
1.0
1.0
1.2
-.0.591
AP 0.350
MMLU
MCE
= 0.034
AICE WALLL
RF 0.626
Decition Accuracy
Dectston Acouracy
Dectsionn Acourscyy
ARC
0.7
984
F.A
KODA
COOL
07
2004
0604
14con
RGA Brce
0.7
Wired
Pepe "Ou CRDA
eeoc assa Seuta
oaga
prvn
2.6
Model Sine
Sed0k
Hau Seaak
5020
6.02
0.04
9.06
0.08
0.10
0.32
0.14
0.03
0.02
0.01
0.006
Rel. Dispersionttial checkpolntss
Rel. Std.ifinal nn train steps)
SNR Rel Dispersion Ral. Std.
Figure 2: Signal, noise, and signal-to-noise ratio (æ-axis) vs. decision accuracy (y-axis), (see Section for for definitions). The signal alone (left) and noise alone (center) have low correlation with decision accuracy, while the signal-to-noi
ratio (right) is correlated with decision accuracy. The Lignal-to-noile ratio gives us information about wether a benchmark is useful during development, as high decision accuracy (and signal signal-to-noiss -o- ratio) means development decisions made a small scale generalize to large scale models.
score of all models to account for different scales. This metric is designed specifically to measure how well a set of points cover space; that is, how spread out the points are from each other. We also considered 20 different measures of spread, including variance, mean pairwise distance, Gini coefficient, etc., in Appendix [A.4
In the following section we introduce si signal-to-noise ratio, and find that this definition of signal leads to signal-to-noise ratio with the highest correlation with decision accuracy. We define signal as Rel. Dispersion(M) (M) m m, myl/m, the normalized maximum difference between any pair of models j. k.
3.3 Measuring Signal-to-noise Ratio
Using our measures of signal ($3.2) and noise (3.1). we propose measuring the signal-to-noie ratio. For both measures, we first divide by the average to be independent of particular units (e.g., to compare accuracy to unbounded task perplexity). We define the signal-to-noise ratio:
Rel. Dispersion final train checkpoint)
Signal-to-Noise Ratio =
(2)
Rel. Std. (final n train checkpoints)
where signal (Rel. Dispersion) is measured over population of models trained using similar compute budget, and noise (Rel. Std.) is measured over the final n intermediate training checkpoints of a single model. We emphasize that, while this is one particular instantiation of the si signal-to- noise ratio, our framework is designed to be independent of a particular metric: we find many other measures of signal produce similar results in Appendix A.4 and measures of noise have high correlation in Appendix A.3
4 Signal and Noise Correlate with Better Predictions
In this section, we show that the signal-to-noise ratio correlates with decision accuracy for small scale experiments, and that the noise of the target model correlates with scaling law prediction. These findings motivate our use of SNR to improve benchmarks' statistical properties in Section 5
4.1 Higher signal-to-noise ratio indicates higher decision accuracy
Setup. We hypothesize that a higher signal-to-nois ratio makes it easier to distinguish between models. To test this, we measure decision accuracy using the ranking of the small DataDecide models (60M to 750M) to predict the ranking of the large DataDecide model (IB). To calculate signal we use the final checkpoint of each of the 25 small models, and to calculate noise, we use the standard deviation around the final checkpoints of the small-scale models. Since we have a measure of noise for each model, we use the average of the noise across the small models.
5

---

Noise
Example fit: SocialIQA
0.65
100%
Scaling Law Prediction Error
Minerva MATH
0.60
2.4%
nolue
Math Taske
MesMCQA
10%
Cide Tasks
RC Accuracy
0.55
MNLU
MBPP-
Knguledge Tabks IriyiaQA
SociialiA
0.50
1%
All Tasks
Jeapardy
Scaling Lan Rtdda Prediited Mode
0.1%
R²
0.653 + 0.068
0.426
Hellaswag
0.40
Reul Mooke
0.1
0.01
10th 10
100P
104t
10
10ss
Rel, (td.(final n train checkpoints)
Compute
Figure 3: Left: Correlation between the noise and scaling law prediction eπror (see Section 2 for definitions). We observe benchmarks with a lower noise around the scaling law target (I-axis) also exhibit lower error (y-axis). Right: Example of scaling law for one benchmark (SocialIQA), with examples on all benchmarks in Figure [3] We conjecture that the noise of the target model (see inset axis) acts as bound on the true minimum scaling law error; if the observed scaling law error below this noi se, then the error only possible by random chance. Therefore, when benchmarks exhibit a similar scaling law error but different noise (e.g., MBPP+, SocialIQA and TriviaQA; see Figure 15) we argue that those with the lowest noise are better.
Signal-to-noise predictive of decision accuracy, Figure shows the signal, noise and signal- to-noise ratio plotted against the decision accuracy across the OLMES benchmarks. While the signal or noise alone do not correlate with decision accuracy, we find a strong correlation between SNR and decision accuracy (R. 0.791, R 0.626). We conclude that benchmarks which have higher SNR at small scales exhibit higher decision accuracy, and are more likely for their results to hold at larger scale. In Appendix [..] observe benehmarks with higher SNR also exhibit lower variance when calculating decision accuracy using different checkpoints around the end of training.
4.2 Tasks with higher noise also have higher scaling law error
Setup. We fit scaling laws to predict the performance of OLMo 2 13B using final checkpoint of the set of scaling models trained by Bhagia al. (3 We calculate the scaling law prediction error as the relative error of the predicted and final 13B checkpoint. To estimate the noise, we calculate the relative standard deviation of the final 30 checkpoints of the 13B training run, each spaced 1000 training steps until the end of training 4 We hypothesize that the range of the final k checkpoints of the prediction target (the large, 13B model) acts as an bounr-bound on the true minimum scaling law prediction error. An example of the prediction error and noise around the prediction target is illustraed using SocialIQA with lower standard deviation of the prediction target to also have lower prediction error.
(rightt),,,, Assuming scaling law with no bias, we expect tasks
Noise measures the reliability of scaling law prediction errors. In Figure 3 (left), we show the scaling law error and standard deviation for predicting the 13B model performance over 30 tasks. We observe a correlation between the standard deviation of the prediction target and the prediction error across tasks (R- 0.653, R²² 0.426), however the fit not perfect. For example, we observe four tasks (MBPP+, SocialIQA, MMLU and TriviaQA) which exhibit similar error (around 2-3%), but exhibit different amounts of noise around the prediction target. For these benchmarks with similar eтror but lower noise, we can be confident that the error we observe from the single scaling law fit is the result of the true error of the scaling law fit rather than random chance. In practice, we recommend practitioners prefer making decisions based on scaling law predictions using tasks with low error and low noise.
Previous work has fit multi-task averages to predict scaling laws. In particular, Gadre al. 19 find that the error from the individual tasks in their work to be too difficult to predict accurately. In Figure
WW found 30 checkpoints to be an adequate off between sample size and compute cost. We provide guidance on selecting when calculating noise, and its impact experimental results, in Appendix A.3.2

---

In this section, we introduce three interventions designed to improve the signal, noise, or SNR: filtering subtasks by SNR (.5.)) averaging checkpoint scores during a training run ($5.2), mcasuring language modeling loss over the test set using bits-per-byte ($5.3). In each setup, we show using Decision Accuracy
Sggaboootoieee Paati (1R)
Scaling Law Target
20.0
95%
Rai. Ste (138)
.005-
Histee
6.0
10.0
25 NULU sudt
5.0
Deddson
83/
Schoook rortec SUR
Sudoada sortee rancenty
80%
.02
50
MNLU
Subtask
MMLU Sustask
bypelonntttee tuelo 1181
AutoBen-cher SNR
AutoBencher Decision Accuracy
AutoBencher Naks at Scaling Law Target
20.0-
"aptencalorall volue talls
92%
Fal Sus (138)
15.0
90%
10.0-
Suctamis serted 5'81
10
1.5
20
30
1.0
1.5
200
25
5
10
155
20
25
30
Ind Ladeldddddooooo www.ottteeeeeee... nwceerrrrrrrrr kmck
1nc Lutted Autottencher Surblenk an
Ind ludlededdddd AutooBencter Subtesk
Figure 4: Evaluating an intervention designed to increase si signal-to-noise ratio (SNR): selecting subsets of a benchmark (Top: MMLU; Bottom: AutoBencher) that have higher SNR dramatically improves decision accuracy and the noise of the scaling law prediction target. MMLU and Auto- Bencher are made of different subtasks; for each benchmark sort its subtasks by their SNR, then greedily add subtasks to our subset in order of decreasing SNR (left to right). Despite the subsets made in this way having fewer test instances, of AutoBencher (e.g., with 6 subtasks) that have higher SNR than the full sets, and also have better decision accuracy and noise around the scaling law target. Named subtasks in Figure 16 in Appendix.
find subsets of MMLU (c.g., with 16 subtasks) and
3 we also plot results for multi-task averages for each task group ( ("Knowledge', 'Math', 'Code') and average across 'All Tasks'. We find that some individual tasks easier to predict than multi-task averages, and have lower noise around the prediction target. In particular, generative tasks like TriviaQA or Jeopardy which evaluate the exact match of short-form generation exhibit lower error than the multi-task averages, and exhibit lower noise around the prediction target. For practitioners, we argue using individual tasks may be better decision in some cases than the multi-task average, that task better represents the ability than a multi-task average.
Our core results report SNR at the scales of our experimental settings for decision accuracy and prediction error. However, SNR can be calculated at any model size, show how the signal-to- noise ratio changes for tasks at larger IB, 7B, 13B and 32B scales in Appendix B.3
Improving Predictions by Improving SNR
signal-to-noise ratio to intervene on the task improved the resulting error in both prediction settings.
5.1 Filtering noisy sub-tasks improves signal-to-noise ratio
Setup. Many tasks are a mucro-average of subtasks. We hypothesize that some subset of subtasks is usually higher quality than the rest of the set, and that the signal-to-noise ratio may be an indicator of high quality subtasks. To test this, we first calculate the signal-to-noisst ratio of each subtask, ratio and greedily add the highest SNR subtasks. As a
then rank the subtasks by signal-to-nois baseline, randomly shuffle the subtasks, and report the average of 10 calculations of each metric, with the shading indicating 11 standard deviation.
Results. We show results in Figure 4 For MMLU, using only 16 subtasks had a higher signal-to- noise ratio than using the full test set. For AutoBencher, we observe the same but with only 6 tasks. The lower signal-to-nois for AutoBencher by using the high SNR subset compared to the full benchmark. We hypothesize that the quality of a task subset may influce that task's si signal-to-non data collected from MMLU Redux, which identified MMLU subtasks with high labeling error [21]. We find that out of the 20 MMLU subtasks which contain errors in least 5% of instances, half of these subtasks (10 of 20) are also in the lowest 20 tasks sorted by their hignal-to-noiss ratio. This presents
ratio also led higher decision accuracy: +2.6% for MMLU and +5%
ratio. To test this, we use the
7

---

Table I: Evaluating intervention designed to average noise: for given model one benchmark, we calculate its score as the average of the scores of its final &o checkpoints (evaluated using bits-per-byte task formulation). Left: On small models used make predictions ('Avg. Pred.'), or to the large target models ('Avg. Target"), or both ('Avg. Both'), ) decision accuracy improves. indicates the decision accuracy is the same across columns. Right: On small models used fit scaling laws ('Avg. Train'), ) scaling law error improves. We show results on a subset of benchmarks,
and report all benchmarks and the primary metric (aceuracy, exact match, pass @ 1) in Tables 5 and
Decision Accuracy (00M-5XC to 1B-3xC). %
Prediction Enor (1 (138-51), Abs. %
Task
Final
Arg. Pred.
Avg. Target
Avg. Both
Task
Final Clip
Ave. Train
Knowledge QA Tasks
Knowledge QA Tinks
ARC Clullenge
94.5
94.3
HellsSwag
HellaSoag
92.4
93.1
93.1
94.0
ConsooniosseQA
0.59
ARC Easy
92.1
92.2
91.9
92.0
Jespunly
0.57
0.54
MMLU
91.5
91.6
91.6
91.6
SociaiIQQA
0.50
0.59
AunoBencher
88.5
98.9
PIQA
1.01
MMLU Pro
90.0
89.
90.0
89.3
MMLU
1.68
1.74
ACT Eval
$6.3
96-7
86.5
87.0
MMLII Pro
1.76
1.75
McdMCQA
86.6
9G.6
86.6
86.6
AGE Eval
1.80
1.98
Jcopurdy
THiviQQ
83.5
84.3
83.8
84.6
TrivinQA
2.62
OpenBoooQA
81.1
81.7
81.6
82.0
SQuAD
2.80
2.79
SocialTQA
79.9
79.5
79.
79.0
OpenBookQA
4.02
3.38
PIQA
72.5
AutoRencher
CummoneensQAA
65.8
68.
65.
65.6
ARC Easy
5.13
5.13
Buoli)
63.7
64..2
63.5
64.0
MexMOQQ
7.72
7.98
SQUAD
60.8
60.4
62.0
61.6
ARC Chailengs
8.44
8.43
Knowledge 19-Task Avg.
71.3
71.77
71.7
Knowledge 19-Taek Avg.
Code Tasks
Code Tasks
HumanEval*
95.6
95.6
2.57
MBPP
95.3
95.3
95.3
90.3
HumanEval
7.71
8.85
Code 4-Task Avg."
96.77
96.7
96.7
96.7
Code 4 4-Task Avg.
3.15
2.75
Math Tasks
Math Tasks
Minerva MATIE*
90.0
90.0
90.0
90.0
Minerva MATH GSMBK Math 6 Task Aug.
1.08
0,98
GSSMKK
Math 6 Task Avg."
88.3
88.3
88.3
88.3
11.33
2.30
Avg.
68.9
70.7
69.5
71.8
30-Task Arg.
1.03
evidence that low SNR may indicate low quality tasks, and we believe this is good opportunity for future work evaluation development.
Intuitively, benchmark developer may increase the statistical power of a comparison between models: by sampling more data by the original process used to construct the benchmark, in order to make a benchmark larger [64], or collect larger number of tasks in an evaluation suite [58). Our evidence in Figure suggests that larger benchmarks may not necessarily be better for comparing models. We further explore this phenomenon in App. [B.2 sub-sampling instances of benchmarks, finding some benchmarks can exhibit a higher SNR despite having 10 times fewer instances.
5.2 Averaging checkpoinn-fo--checkpill noise leads to better predictions
Setup. Typically, models are only compared using the evaluation of the final checkpoint. In the previous sections, we argued that noise is a good indicator of whether we can use a benchmark to predict a large scale phenomenon. In this section, we want measure the effect of averaging this particular source of step-to-step noise, as a way of improving our ability to make prediction. In the decision accuracy setting, we can average the results of the small model, the large model (in this case, the IB model), or both. In the prediction error setting, averaging the small models will help in fitting the scaling law, but averaging the target model will just make the result more reliable, so we average the target model in both settings and only change whether average the models used to fit the scaling law. Finally, we introduce an additional way to average step-to-step noise during a training run, by evaluating whether the ranking of the IB models during training agrees with the ranking at the end of training. Note, as our measure of noise is between intermediate training checkpoints, we only reducing one of many sources of modeling noise.
Results on Final Checkpoints. In Table I we observe averaging the noise improved both measures of error. Averaging noise improved decision accuracy by +2.4% for the 30-task average, this procedure improved decision accuracy all but two tasks. For reducing the scaling law prediction error, averaging the training checkpoints improved prediction error for 20 of 30 tasks.
8

---

HellaSwaag 5xC
Challenge
1.00
1.00
Dlcorior AcCUrayy
Dectsion hocuray
Smoothing
EMA (N-2)
0.95
Docbion
0.95
0.95
(N-5)
(N-20)
Single spoitt
20K Training Stop
20K
40K Training Stop
60K
20K
6.0K
6.0K
Training Stop
Figure 5: When stopping training run early, averaging the checkpoint--o-checkpiii noise improves the decision accuracy between an intermediate and the final training step. Shown are decision accuracy from carly-stopping for HellaSwag, ARC-C and MMLU by using both single checkpoint and the exponential moving average (EMA), with all tasks included Figure
Minerva MATH Exact Match
Experiment Setting
SNR (T)
Rel. Emur (L), % Primary
Decision Ace CT) Primzry
Primary
5NR=1.92
Knowledge Tasks
Exocct Match
TrivQQ
61.8
2.5
SQUAD
ARC Fasy
23.8
21.0
29. 0
64.6
7.8
5.3
27.8
0.8
59.7
93.0
61.7 77
93.0
2.01
Jeopardy
22.6
5.01
AutoRencher
31.3
0.2
HelloSwng
11.8
14.9
1.4
1.0
74.3
96.3
MMLU
Challenge
9.8
36.9
4.3
0.4
89.0
92.00 0
23C
Training Step
40K
636
80C
SacialIQA
5.5
48.0
0.4
Minerva MATH BPB
PIQA
4:2
8.8
0.5
1.3
73.3
72:7
5.80
5NR-87.30
Knowledge 19-Task
AGI Real
2.5
19.5
13.7
3.4
58.7
88.0
Bit--perayye
9.80
Moth Tasks
3.20
GSMBK
1:2
7.0
38.6
5.9
46.0
76.7
3.40
Math 6 G-Task Arg.
1.8
22.6
46.0
D.0
42.3
88.3
1.60
Code Tasks
HumanEval
6.1
26.
9.2
7.9
74.3
96.7
23C
Training Step
40K
62K
80
4-Thok Avg
5.5
42.0
9.7
80.3
98.7
(1008) training
Avg.
18 (1008) DataDecide final checkpoints
Figure 6: Impact of changing benchmark metric to bits-per-byte (BPB) from the primary score (e.g.. accuracy, pass@1, etc.). Left. Columns are (i) SNR of IB models trained to 100B tokens; (ii) scaling law prediction error of 1B (and smaller) models used to predict 13B model performance; (iii) decision accuracy for using 150M model to predict 1B model ranking. For almost all tasks at the scales explored here, bits-per-byte shows higher SNR, and lower scaling law prediction error, and higher decision accuracy than the primary score. Full results across 30 benchmarks and model scales in Table 17 Right. Example of primary metric and BPB on single IB (100B tokens) training curve (blue curve) and the final checkpoint of 25 models for Minerva MATH (green 'x's). Visually, the BPB training curve is smoother, corresponding higher SNR and lower enror in the prediction settings reported in the table, with all tasks Figure
Results on Early Stopping, Another prediction setting is to determine whether the ranking of two partially trained models will exhibit the same order the end of training. We hypothesize that averaging the step-to-step noise will similarly improve this setting. In Figure we report the decision accuracy for early stopping by using single checkpoint (red), compared to exponential moving average of the training curve (blue). We find for almost any training step, applying smoothing led to higher decision accuracy when comparing models during training. In both settings, reducing the checkpoint-to-checpppin noise allowed a more accurate extrapolation.
5.3 Measuring bits-per-byte improves benchmark signal-to-note ratio
Setup. Recent work has begun to evaluate by using the test set as a perplexity set, with the intuition that the discontinuous metrics like accuracy or exact match erode the relationship between the language modeling perplexity and the downstream metric [54 25 We aim to measure whether the intervention to use a continuous metric improves the signal-to-nois ratio and corresponding error. We calculate the bits-per-byte (BPB) using the correct continuations of each test set bits-per-byte byte is the negative log likelihood of the correct answer divided by the number of UTF-8

---

bytes in the answer string 20 37. We compare BPB to the 'primary' task metric (accuracy, exact match, pass@ @ I, etc.) on the signal-to -ooaiottt-oisee ratio, and whether it improves decision- making using decision accuracy from 150M to IB and reduces the scaling law prediction error at 13B.
Results. In Figure 6 we report the signal-to-noise ratio, scaling law error and decision accuracy for benchmarks using BPB instead of the primary metric, along with an example training curves for Minerva. Most benchmarks have higher signal-to-noise ratio when using the BPB, particularly generative math and code benchmarks like GSM8K (1.2 to 7.0) and MBPP (2.0 to 41.8). To verify this improvement in signal-to-nois we observe an improvement in decision accuracy at the small scale for 90.0% of all benchmarks and a lower scaling law prediction error for 73.3% of all benchmarks. We see BPB results in dramatic improvement for tasks that small scale models are not able to accomplish at all. primarily generative tasks. Our results confirm that BPB is a useful metric is both higher quality development benchmark, particularly for challenging tasks at small scales that do not show above random-chance si signal.
ratio corresponds to an improvement in our docision-makingg setups,
6 Related Work and Discussion
Predicting model behavior at large scales is crucial aspect to language model development, as discussed in the beginning of 2 Noise within evaluation benchmarks is frequently studied as the
intrinsic noise of the dataset 2
40 [] rather than the noise as a result of differences in the model during training. Closest to our work is Madaan et al. [36], which report a measure of SNR using the benchmark score of a single model and noise using 10 seed models, rather than population of models. We find that the noise of single model alone, while a useful measure of modeling noise, is not sufficient as a measure of correlation to decision accuracy ($4), and show the step-to-step noise is a cheap alternative to seed noise. Similarly, Kydlíček et al. [29] focus on identifying high quality translations of tasks, but do not focus on decision making. Finally, EvalArena [63] also reports a measure of SNR using the final checkpoints of a small/large model pair (e.g.. Llama 7B vs. 70B). While statistical measures based on intrinsic noise rather than modeling noise are important indicators of dataset noise, we find that many benchmarks may have low statistical variability but high checkpoint- toohhikttt--hhekoonntt noise (such as BoolQ, as observed in Figure 9). which can only be captured with a measure of modeling noise.
Interventions to improve evaluation have been well explored, such as constructing higher quality benchmarks by identifying errors 662 2T expanding test sets [64] selecting high quality instances from benchmarks [45]. or generating entirely new synthetic benchmarks from a model [32]. These works typically justifyy their decisions using inter-annotator agreement, or high correlation with the original benchmark. We believe this body of work can benefit from verifying their methods using SNR, rather than noise or reconstruction error alone, to indicate whether the benchmark serves as a useful development tool.
Notably, this scope of our connection between the signal-to-noise ratio and predicting large scale phenomena is limited to the two decision accuracy and prediction error settings, and only studies the noise of the model during training. Future work may explore how signal-to-noise ratio indicates other small-to-large phenomena [65 57). and the effects of additional sources of noise on the ability
to extrapolate from scall-scale experiments, such as from the evaluation configuration [55] 22
In this work, we identify signal and noise as a cheap way of estimating whether a benchmark is useful in predicting large-scale phenomena with small scale experiments. We conclude that new benchmark development should use these measures of modeling noise as a guide for building evaluation tools for model developers, and practitioners adopt interventions, such as those introduced in this work, that improve their ability to compare models.
Acknowledgments and Disclosure of Funding
We would like to thank Pang Wei Koh for feedback on the manuscript; and Dany Haddad, Dirk Groeneveld, Luca Soldaini, Matt Jordan, Oyvind Tafjord, Ronan Le Bras and Saumya Malik for insightful discussions. This material based upon work supported by the U.S. National Science Foundation under Grant No. 2313998. Any opinions, findings, and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation. IM is supported by the NSF CSGrad4US Fellowship.
10

---

References
[1] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.0771
2021.
[2] Taylor Berg-Kirkpateer Kirkpatrick, David Burkett, and Dan Klein. An empirical investigation statistical significance in NLP. In Jun' Tsujii, James Henderson, and Marius Paşca, editors, Proceed- ings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 995-1005, Jeju Island, Korea, July 2012. Association for Computational Linguistics. /ttp///ts///lll
[3] Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A Smith, Dirk Groeneveld, Pang Wei Koh, et al. Estab- lishing task scaling laws via compute-cfficient model ladders. arXiv preprint urXiv:2412.04403, 2024.
[4] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7432-7439, 2020.
[5] Cody Blakeney, Mansheej Paul, Brett W. Larsen, Sean Owen, and Jonathan Frankle. Does your data spark joy? performance gains from domain upsampling at the end of training, 2024. URL ftttss://arxiv.org//bta
03476
[6] Sam Bowyer, Laurence Aitchison, and Desi R Ivanova. Position: Don't use the clt in 11m evals with fewer than a few hundred datapoints. arXiv preprint arXiv:2503.01747. 2025.
[7] Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9263-9274, Online, November 2020. Association for Computational Linguistics. doi: aclanthology org. 2020. emnlp-main. 7457]
URL https://
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[9] Leshem Choshen, Yang Zhang, and Jacob Andreas. A hitchhiker's guide to scaling law estimation. arXiv preprint arXiv:24100 1840, 2024.
[10] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2924-2936, 2019.
[11] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning 2018.
challenge. arXiv preprint
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
[13) Alexander Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Fisenstein, Matthew D Hoffman, et al. Un- derspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(22)):1-61, 2022.
[14] Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv: arXiv:2002.06305, 2020.

---

[15] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15796 2024.
[16] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2368-2378, 2019.
[17] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint urXiv:2407.2178: 2024.
[18] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open IIm leaderboard v2. Inttps://hugginggaace co/ -o/sppaces/oppnnnnnmmm leaderboard/ open 11m leaderbobd 2024.
[19] Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, el. Language models scale reliably with over-training and on downstream tasks. arXiv preprint 1rXxiv2403.085444 2024.
[20] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv:2101.0002) 2020.
[21] Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, al. Are done with mmlu? arXiv preprint 240.:::20060412277 2024.
[22] Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint arXiv:2406.0844 2024.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 0rXiv:2009.03300 2021.
[24] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optiman large language models. arXiv preprint arXiv:2203. 2022.
[25] Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. arXiv preprint arXiv:2404. 09937, 2024.
[26] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1601-1611, 2017.
[27] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
[28] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7,452-466, 2019.
[29] Hynek Kydlíček, Guilherme Penedo, Clémentine Fourier, Nathan Habib, and Thomas Wolf. Finetasks: Finding signal in a haystack of 200+ multilingual tasks, 2024. URL https:/7 tasks
[30] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint 2.X.:22206.1888888 2022.
12

---

[31] Jeffrey Li, Alex Fang, Georgias Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37:1 4200-14282, 2024.
[32] Xiang Lisa Li, Evan Zheran Liu, Percy Liang, and Tatsunori Hashimoto. Autobencher: Creating salient, novel, difficult datasets for language models. arXiv preprint (oXii:2407708335.1 2024.
[33] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga. Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint 2.00:::2211091//11 10, 2022.
[34] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chaigpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558-21553 2023.
(35] Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng. Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training. arXiv preprint arXiv:2407.01492 2024.
[36] Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp. Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation bench- marks. arXiv preprint arXiv:2406. 10229, 2024.
[37] Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, et al. Paloma: benchmark for evaluating language model fit. arXiv preprint arXiv:2312. 10523, 2024.
[38] Ian Magnusson, Tai Nguyen, David Heineman, Jena D. Hwang. Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, Ben Bogin, and Jesse Dodge. Datadecide: How to predict best pretraining data with small experiments. under submission, 2025.
[39] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381 2381-2391, 2018.
[40] Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv preprint 1.X06:2411.00640, 2024.
[41] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprini arXiv:24
2024.
[42] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501. 00656, 2024.
[43] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning (CHIL., pages 248-260, 2022.
[44] Tim Pearce and Jinyeop Song. Reconciling kaplan and chinchilla scaling laws. arXiv preprint arXiv:2406. /2007, 2024.
[45] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. arXiv preprint arXiv:2402.1 14992, 2024.
[46] Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. Varbench: Robust language model benchmarking through dynamic variable perturbation. arXiv preprint 2rXiv:2406.176811 2024.
13

---

[47] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, 2016.
[48] Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266
2019.
[49] David Rein, Betty Li Hou, Asa C. Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman, Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint A.Ni:2331.122222 2023.
[50] Nicholas Roberts, Niladri Chatterji, Sharan Narang, Mike Lewis, and Dieuwke Hupkes. Com- pute optimal scaling of skills: Knowledge vs reasoning. arXiv preprint arXiv:2503. 10061, 2025.
[51] Yangjun Ruan, Chris Maddison, and Tatsunori B Hashimoto. Observational scaling laws and the predictability of langauge model performance. Advances in Neural Information Processing Systems, 37:15841-15892 2025.
[52] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, pages 9732-8740, 2020.
[53] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 4463 4473,44733--1 2019.
[54] Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained elusive? arXiv preprint wrXiv:2406.0439, 2024.
[55] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying about prompt formatting, 2024. URL httpt:t///rxii.orrrrrrrr// http://wwwrrr///////////// 11324
[56] Kashun Shum, Yuzhen Huang, Hongjian Zou, Ding Qi, Yixuan I. Liao, Xiaoxin Chen, Qian Liu, and Junxian He. Predictive data selection: The data that predicts the data that teaches. arXiv preprint arXiv:2503.00880 2025.
[57] Charlic Snell, Eric Wallace, Dan Klein, and Sergey Levine, Predicting emergent capabilities by finetuning. 2024. httpps::/aaai..oo/////aaae 26035
[58] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities language models. arXiv preprint arXiv:2206. 04615, 2022.
[59] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 4149-4158, 2019.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta,
14

---

Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Ilyaan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL httph:ttp///tt wttp.///...rg/aa////////// 09288
[61] (Kaggle Datasets) Tunguz. 200,000+ jeopardy! questions. httts://www.kkggg com/ 2019.
[62] Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry. Do large language model benchmarks test reliability? arXiv preprint arXiv:2502.03461, 2025.
[63] Sida I. Wang, Alex Gu, Lovish Madaan, Dieuwke Hupkes, Jiawei Liu, Yuxiang Wei, Naman Jain, Yuhang Lai, Sten Sootla, Ofir Press, Baptiste Rozière, and Gabriel Synnaeve. Eval-Arena: noise and errors on Ilm evaluations. https://gi
com/ crux- eval/eval-arena 2024.
[64] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[65] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint wrxiv:2206.0765. 2022.
[66] Alexander Wettig, Kyle Lo. Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation. arXiv preprint arXiv:2502.10341 10341, 2025.
[67] Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabhar- wal. Answer, assemble, ace: Understanding how transformers answer multiple choice questions. arXiv preprint arXiv:2407. 15018, 2024.
[68] Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer, 2022. URL htttts or org/abs/
2203. 03466
[69] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association Computational Linguisties, pages 4791-4800, 2019.
[70] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, arXiv preprint 2r04 063664 2023.
15

---

A Methodology Details
A.1 Scaling Law Details
Hoffmann et al. 24 models the improvement for larger model training budgets power function, proportional the model parameters N and training tokens D, with the exact functional form and prediction setup varying between work 44.. Recent work has begun using the downstream task the prediction target [17 19], in this work we follow Bhagia et al. [3] by fitting a scaling law function to the language modeling Ioss over the correct continuation, then from the task loss to the downstream evaluation. We use the following functional form:
A Na
B D
L(N,
U(L) 1 T +6
D3
We follow the same methodology as Bhagia et al. B and use the Huber loss to fit ( D) and use a non-linear least squares optimizer to fit (L). The prediction enror is defined the relative error of the scaling law fit:
Measured Value True Valuel
Prediction Error
True Value
A.2 Decision Accuracy Details
Decision accuracy is one of many rank agreement metrics we could use to show that models trained across pre-training corpora agree small scale and large scale. We present two alternatives here:
Kendall's т. Here, rather than report Kendall's т, we show it is proportional to decision accuracy. Kendall's is defined as the difference between the concordant pairs C' and discordant pairs D, divided by the total pairs of models: (C (M). We can then rewrite decision accuracy
defined only by the number of concordant pairs C: decision accuracy = C2)
Since we do not allow ties, C' and D make up the total number of pairs (2) C + D, we can rewrite decision accuracy follows:
(7)
2C
2
(
2 (decision accuracy)
Therefore, the decision accuracy measure in Magnusson et al. 38 is equivalent to Kendall's modulo a scale and shift.
Spearman's 's Rank Correlation. Kendall's 7 is not sensitive to outliers, and instead we can incorpo-
6 d2 This statistic will be rate the strength of the difference in rank with Spearman's pt p nin -1) more sensitive to large differences model ranking.
We use decision accuracy in this work for consistency, and to provide a more interpretable metric of rank agreement (for instance, decision accuracy 80% indicates that 80% of the pairs mixes agree between the small scale and large scale). To show that both additional measures of agreement produce similar conclusions, we include correlation with these additional measures of agreement in Table 3
A.3 Measures of Modeling Noise
Seed Noise. To measure the noise introduced from changing the random seed initialization between training runs, we can compute the standard deviation of the final checkpoint from multiple training runs with different random seeds. To estimate seed noise, we train M models using the same configuration, and average the scores over the final checkpoints of T total training checkpoints to smooth the pheccppoin-tt-cchecpppoe noise, then compute the standard deviation:
Seed Noise(M) (MM) M, DI U(t,)
(5)
16

---

Data Order Noise. This is noise introduced from changing the order of sampled documents from the training data. We estimate the data order noise using the same method as seed noise.
Total Variation. To measure the checkpoint-to--cecckpoii noise throughout an entire training run, measure the total variation of the intermediate training checkpoints on the downstream benchmark. We measure total variation as the average change in metric score across '" training checkpoints minus an improvement term:
Total Variation = -00- U())
Checkpoint-to-cheekppoii Noise. Calculating the above sources of noise are either too expensive estimate large scales (e.g., training LLMs by varying the random seed) difficult to run (e.g.. evaluating every checkpoint on an LLM training curve). Instead, we propose an estimate measuring only the noise of the final training checkpoints of training:
Checkpoint-to--checppoin Noise (UUG)) -+1
A.3.1 Correlation between Sources of noise
To measure the relationship between each source of noise, we train 10 1B-5xC models varying the random seed initializations and 10 models varying the data order. In Figure 7 we measure the correlation between the seed noi se, data order noise and total variation against the step-to-step noise. Each source of noise is highly correlated with the step-to-step noise (R > 0.9 for all measures). While would be ideal to calculate and reduce all sources of noise, seed noise se and data order noise are too expensive to measure (e.g., for large model runs as in Madaan et al. [36] ), so only calculating step-to-step noise is a reasonable estimate for the modeling noise. Thus, we use step-to-step noise in our estimate of the modeling noise.
A.3.2 Selecting the Number of Checkpoints in Noise
The noise calculation introduced in Section 3.1 requires selecting some n intermediate checkpoints to estimate the checkpoint-tor to-kppiittt-cchccooiit noise. In this section, we provide guidance selecting and discuss its impact on our findings. Increasing the number of intermediate checkpoints n will lead to less biased estimate of noise. Thus, we can calculate the minimum number of n intermediate checkpoint samples such that the sample noise sn is a reasonable estimate of the population noise σ..
We first assume the checkpoint to checkpoint scores are independent and normally distributed (which we observe when computing decision accuracy on intermediate checkpoints in Figure 7). Under this assumption, the ratio between the sample variance and the population variance follows a scaled chi
squared distribution: (n-1)s² 2a-1
Therefore we would like to calculate the probability that the sample standard deviation is within one standard deviation of the population standard deviation a: |sn ol <
We can rewrite this inequality:
1
<1
<2
And then, can substitute the chi-squared distribution to compute the likelihood w.r.t. n:
8n
X
X < 2 = ( < 4(n 1))
P
V
n-1
We then solve the inequality for the smallest value of for particular threshold
P P 4(n 1)) a
Solving this inequality numerically with = 0.95 for increasing values of n, we find that n provides the smallest sample size such that the probability that the sample standard deviation (the
17

---

HellaSwag
HellaSwag
HellaSwag
Accuracy
0.5
0.5
botal sortotton
0.4
0.625
0.4
6.625
data
0.3
0.600
nolse
nolse
0.3
60x
ARC Challenge
ARC Challenge
ARC Chal
0.40
Rununtaryrg Joue!
tuaytig
Pun burving
0.35
0.95
Accuracy
0.35
0.30
pata
0.25
onder
0.25
0.36
0.20
0K
20K
40K
60K
80K
20K
40K
66k
05
20K
40K
60K
80K
MMLU
MMLU
0.36
tuaying onter)
Seiying api
0.34
Accuracy
0.32
0.32
0:30
Feed
0.30
0.35
Joneer
data
0.30
0.34
0.26
0.26
0.26
<0K
20K
20K
60x
Training Step
Training Step
Training Step
Seed Intialization Noise
0.0200
0.90
ARC eoccc
0.0200
AII Chal lango
0.0200
XAC charngs
0.0175
Witogrante
0.0175
SodiiigA
Total Variatton
0.0175
0.0150
0.01.50
0.0150
0.0125
0.0125
Wregrande
0.0125
CommoneenseaA *aducia
Wisgranee
0.0100
0.01.00
0.0100
0.0075
JBC
0.0075
HellSwoog
InllaSonnn
0.0050
MHLU
1038 Runs
0.0050
+10A
1008 Runs
10-8 Rurs
0.00500. 0ww000000000.....00 01250.0150
0.005
0.010
0.015
0.0050 0.0075 01000 01250.0150
Checkpaint-S5 Checkpaint Noise
Checkpoint Checkpaint Noise
Checkpeint Checkpaint Noise
Figure 7: Top: 10 different training runs (IB-5xC scale) varying random seed initialization and data order, plotting ARC-C accuracy smoothed across a window of 20 checkpoints. Bottom: Total variation the relative standard deviation (STD normalized by average performance; $3) of scores from different seeds, data after averaging the last 20 training checkpoints vs. the Rel. Std. over the last 20 training checkpoints. Benchmarks with high checkpoint- toppoinn-tt-checkpiinn noise also exhibit high noise due to random seed initialization, data order and noise along the full training curve. Noise for all tasks reported in Figure T9
observed noise) is within one standard deviation of the population standard deviation (the true noise) with 95% confidence. In addition, we can specify stricter bound by defining the sample standard deviation to be within k- of the population standard deviation: |s.. al < k.
We then verify this empirically using our estimate for noise the 7B scale (from $5.2 If assume the 30 intermediate checkpoints provide a reasonable estimate of the population standard deviation, we then compute the sample standard deviation s, for 30. We re-compute S, 1000 times for different subsets to calculate the likelihood that the sampled standard deviation is within k of the population standard deviation a. In the below table, we report this likelihood with tolerances k E (0.2, 1.0} for subsets E {5, 10, 20} and bold all results with likelihood above 0.95.
In practice, find that for large bound (#1 std. dev.) be satisfied for almost all benchmarks with n = 5 intermediate checkpoints, but for smaller bounds, (20% +1 std. dev.), using n = 20 gives adequate estimate for 34 of 39 benchmarks we considered in work.
For our experiment on the IB-5xC checkpoints, we estimate noise using the average noise of the last 5 checkpoints for all 25 models, so estimate of noise considers 5 25 125 scores.
18

---

Table 2: Ablating the term in noise: Likelihood that the sample standard deviation for inter- mediate checkpoints is a reasonable estimate for the population standard deviation on OLMo 7B, calculated using 30 intermediate checkpoints (Values for 0.95 in bold). We find that for low tolerance (within 0.2o), 20 intermediate checkpoints provides an adequate estimate of noise.
threshold &
f:
0.2
# Ckpts in Not (m))
5
10
20
5
10
20
AGI Eval
0.42
0.61
0.95
1.00
1.00
1.00
ARC Challenge
0.44
1.00
1.00
ARC Easy
0.38
0.65
0.97
1.00
1.00
1.00
AutoBencher
0.47
0.71
0.97
1.00
1.00
1.00
BBH
1,00
1.00
BoolQ
0.16
0.45
0.88
1.00
1.00
1.00
HumanEval
HumanEvale
0.47
0.76
0.99
1.00
1.00
1.00
CommonsensecA
0.96
1,00
1.00
1.00
DROP
0.48
0.76
0,99
1,00
1.00
1,00
GSM8K
0.49
0.77
0.99
1,00
1.00
1.00
GSM Symbolic
0.37
0.64
0.96
1,00
1.00
1.00
GSM Symbolic
1,00
GSM Symbolic P2
0.32
0.57
0.94
1,00
1.00
1.00
HellaSwag
0.39
0,97
1,00
1.00
1.00
Jeopurdy
0.42
0.69
0.98
1,00
1.00
1.00
MBPP
0.43
0.63
0.96
1,00
1.00
1.00
MBPP-
MedMCQA
0,50
0.79
0.99
1,00
1.00
1.00
Minerva MATH
1.00
Minerva MATH 500
0.28
0.53
0.92
1,00
1.00
1.00
MMLU
0.00
0,00
0.54
0.83
1.00
1.00
MMLU Pro
0.51
0.99
1,00
1.00
1.00
All Tasks
0.00
0.00
0.08
0.83
1.00
1.00
Code Tasks
0.99
L,00
1.00
Knowledge Tasks
0.00
0.00
0.15
0.83
1.00
1.00
Math
1.00
OLMES Core
0.31
0.49
0.92
1.00
1.00
1.00
OLMES Gen
0.48
0.74
0.98
1,00
1.00
1.00
OpenBookQA
0.42
0.98
1,00
1.00
1.00
PIQA
0.43
0.69
0.98
1.00
1.00
1.00
SocialIQA
1.00
SQuAD
0.48
0.72
0.99
L.00
1.00
1.00
TriviaQA
0.48
0.99
1,00
1.00
WinoGrande
0.42
0.67
0,97
1,00
1.00
1.00
A.4 Measures of Signal
When designing an measure of signal, we want to incorporate the uniformity of benchmark scores and the overall range of scores. Given the final checkpoints of training under similar compute spend China, we evaluate multiple approaches to measuring signal:
Measurements.
Variance measures average squared distance from the mean: Var(Cnnal) = E ||a-ā||
Mean distance measures average pairwise distance between points: Mean Dist (Cinat) m(n-1) Eiey |ca cy
Relative standard deviation, or the coefficient of variation, measures the standard deviation
divided by the mean: Rel. Std. (tn.al Meant
(araa)
Star Discrepancy measures the largest difference between any point and the uniform distribution: Discrepancy( (Ciina) = supte]o,1] A. E 1fc S t} -t
Dispersion measures the largest difference between any two points, or the largest unfilled space in the range of performance: Dispersion
max ||c, -cyl--
Note, include metries that sensitive and sensitive outliers, and find our results hold when measuring both types of spread (Table 3).. We also include variants of these terms, such using a min-max normalization or scaling the mean.
Choosing the a signal measurement. Table we calculate the correlation between signal- to-noise ratio and decision accuracy when using each of the signal variants. We see that many
19

---

Table 3: Correlation of signal-to-noise ratio decision accuracy, using different measures of signal. We use the measure which is most predictive of decision accuracy as our measure signal. We include alternative methods for calculating decision accuracy (Pearson correlation and Spearman's rank correlation coefficient), as detailed in Appendix A.2
Fits are illustrated in Figure T0
SNR Decision Acc R
SNR Pearson R²
SNR Spearman R
Meusure of Signal
Rel. Dispersion
,
oz e3//2
0,5687
0.4052
0,49002
Rel. Std. Dev.
0/K
0,5657
0,3850
0.4771
Rel. Mean Pairwise Distance
Σi, la 1//0
0.3624
Interquartile Range
Q3
Qi
0.4836
0.2866
Distance Standard Deviation
Σ, (c,
0,4745
0,3667
0,3950
RMS Deviation
[, - 2)2
0.4633
0.3435
0,3812
Meun Pairwise Distance
22 Eig la - eyl
0.4589
0,3325
0,3758
Range
max(c) - min(e)
0.4574
0,3604
0,3865
Dispersion
max,, le G
0,3604
Quartile Deviation
(Qa Q1)/2
0.4528
0,2896
0,3655
Average Absolute Deviation
E le
0.4507
0.3186
0,3672
Median Absulute Deviation
mediam median(e))
0.4168
0,2663
0,3346
Rel. Mean Squared Pairwise Distance
E. (4- ,
0.2908
0,1627
0,2324
Mean Squared Pairwise Distance
E. e,²
0.2480
0,1457
0,1953
Gini Coeflicient
Lig la syl
0.0944
0.0978
0.0829
Discrepancy SSale)
su (L) F (1) shifting
0,0768
0.0454
Star Rel. Discrepancy
suplo.d cl (.) F())/F(()
0.0379
0.0587
0.0420
Dispersion Scalet+Scale)
lo; with shifting
0.0374
0.0679
0.0382
Halfspace Depth
min
(F.
(2,(2))
0.0395
Discrepancy
(e) F(e)
0.0340
0,0754
0.0401
Projection Depth
[1+ +
1x mslle))
(EATCC)
0.0331
0.0392
0.0353
Star Discrepancy
suP[o.e] (e) F(O1
0.0319
0,0665
0,0356
straight forward measures have similarly high correlations. We use relative dispersion, the highest correlated among them, as our measure of signal.
A.5 Dataset Details
A.5.1
Models
We evaluate 465 models which represent stages of the Docision-making process during pre-training.
Unlike existing collections of model evaluations [88 33 our set is targeted at development models:
Scaling Law Models. 25 ladder models from Bhagia al. 3] (190M, 370M, 760M, 1.3B, 3.2B) x {0.5xC, 1xC, 2xC, 5xC, 10xC) trained on OLMoE mix, and 7B-4T / 13B-5T as prediction targets.
Decision Accuracy Models. 225 models from Magnusson al. 38 trained on 25 data recepies for (4M, 20M, 60M, 90M, 150M, 300M, 530M, 750M, 1.3B) trained to 5x Chinchilla optimal.
Random Seed & Data Order Models. 20 models 1B-5xC models trained on the OLMoE mix, 10 models trained with different random seed initializations and 10 models trained with different data order seeds.
Final Checkpoints. 120 models representing the 30 final checkpoints before the end of training for OLMo IB, 7B, 13B and 32B 42,, with checkpoints spaced by 1000 training checkpoints.
External Models. 73 ween-weight base models from the DCLM, DeepSeck, Gemma, Llama, Orca, Phi, Pythia, Qwen, SmolLM, StableLM and Yi model families. We estimate the training FLOPs using the reported token count.
We perform all evaluation using up to 2 H100s for a particular model, and use 94K H100 hours total for all evaluation. For training our randomly initialized seed and data order models, we use 23K GPU hours, using a cluster of 2x8 H100s for each training run.
A.5.2 Benchmarks
We intentionally select benchmarks that are widely adopted pre-training evaluation. We use the OLMES [22] standard when applicable, and for other benchmarks, reproduce the evaluation setup
20

---

Easy
MMLU
ARC Challenge
HellaSwag
5MA - -2..5
3Mitt 34440
6000
SMA -8. suroee
5000
5tam
8000
6000
>uumpp
4000
Sumper
4000
4000
2000
2000
1000
2000
5.90
0.25 Bamples from ain Cepial
[Sampled from Final 5 Capis)
(Sampled from hha 5 (kota)
(Sampleed from Riial Ckptt)
BoolQ
CommonsenseOA
OpenBookQA
SocialIQA
2500
SMR 5:45
ShR - -5.43 00900e
3000
SMR - 2.12000
SKR - 4,444
3000
2000
Sapple
Sampae
2000
2000
1000
2000
1000
0:45 0:50 D.55 0.50 0.55
0.55
0.70
5.75 D.62 n.ss
0.60
0.65
0.20 0.75
0.85
70 0.75 3.00
Dedition Accuraoy ot 300m
Decsion Accurtcy
300M
Decttion Accurscy at 3003
Dec slon Accuracy 300M
(Sampled 3 Capts)
(Samoled Olots)
Hampled Captsl
(Samaled Ogits)
PIQA
WinoGrande
2000
Sineles
2000
15.00
1500
1000
500
500
5.00
0.65
5.70
0.75 0.20
0.50
0.55 0.60 0.65
Deridin ArCurary 100M
Decaien Anourac 100M
Final
Figure 8: As the benchmark's signal-to-nois se ratio increases (across histograms), decision accuracy (from 300M to IB scale) not only increases but becomes more consistent. We test this by resampling decision accuracy for combinations among last checkpoints the small and large models, respec- tively, since noise in the results of either size can change rankings. Note how CSQA and MMLU have similar signal (Rel. Dispersion = 0.056 vs 0.054) but different noise (Rel. Std. = 0.01 vs. 0.004).
from OLMo 2 [42]. Notably, all tasks use few-shot examples and we evaluate MCQA benchmarks in both the rank choice (RC) and multiple choice (MC) setting, since our small (< parameter) models show random-chance performance on MCQA benchmarks.
Knowledge QA. MMLU 23 ARC 11 BoolQ [10], CSQA [59, OBQA 39 PiQA 4 SocialIQA
[53]. HellaSwag 669, WinoGrande 122.
DROP [6),
CoQA 48 Jeopardy 66],
NaturalQs (28
SQuAD 47 TriviaQA (26). MedMCQA
4
MMLU Pro 64, AGI Eval 700, GPQA 49
Math. GSM
12 GSM Plus 46.. GSM Symbolic
4
Minerva
30
Code. HvmanEval
8
HumanEval+ 34.
MBPP
MBPP+
34
Using strong LLMs have become tool for augmenting existing benchmarks with more difficult
questions or answer choices (64] and re-evaluating benchmark quality (2). and may provide cheap
method for improving signal. To test this, we add an additional synthetic benchmark:
Autobencher. To test whether fully generated benchmarks
act adequate development
benchmark, we generate a dataset of 30K MCQA questions using Autobencher (32]. Autobencher iteratively mines for Wikipedia articles and uses strong LM to generate and prune questions based on saliency, novelty and difficulty constraints.
B
Full Results
B.1 Noise measures the reliability of decision accuracy.
As discussed in 331 the pheckpoint-to-checkpee noise can change the ranking of models, which may effect the decision accuracy we observe by only evaluating the final DataDecide model. To measure the impact of beeckpoonn-to-checkpoiii
noise on decision accuracy, we can estimate the distribution of possible decision accuracies given the step to step noise. To do this, we sample one of the final 5 checkpoints for both the small and large model, and repeatedly sample to estimate the
21

---

Signal-te at 18-5xC
Decision Accuracy (150M 1B-5xC)
Scaling Dev. 138
Micter
PNU
90%
Std Dev. of Prodt Emer
Sinaaits Nota
Deciaion Accurag)
8.%
VNU felurwon
5.0%
10-
700%
4.0%
5.0%
bo -Rox
60%
8.0%
50%
1000
10000
1000
LO0G0
1000
10000
Instances
Instancess
Instancess
Figure 9: sogmal-tonniisse ratio, decision accuracy, and scaling law prediction error for randomly sampled subsets of instances for 6 development benchmarks. large sample size alone does not improve signal-to-noise se ratio. For example, a 1000 question subset of ARC Easy has higher decision accuracy than MMLU despite having 90% fewer instances.
distribution. A wider distribution would indicate that should be less confident in the decision accuracy.
We show the distribution of decision accuracies for 10K random samples in Figure g a higher signal-to-nois ratio, the sampled decision accuracy distribution has a higher mean and lower variance. Additionally, we find that tasks with similar signal, but different noise (e.g., CSQA and MMLU, where CSQA has higher noise), the tasks with lower noise also have a lower variance of sampled decision accuracy distribution.
For tasks with
B.2 Increasing benchmark size has diminishing returns
Setup. One intuitive way to reduce modeling noise is to increase the size of the benchmark, while this is expensive in practice, recent work has given LLMs access privileged information to generate distractor options or full benchmarks [32 (4). To test the impact of sample size on modeling noise, the existing of benchmarks, select random sample of instances and recalculate SNR, decision accuracy and scaling law eтгor. To test the limits of synthetic benchmarks, we use our version of AutoBencher, which has 33K instances, or 2x more test instances than the next largest benchmark in our dataset (MMLU).
Results. Figure 9 shows how each metric improves the number of instances increases. Initially, all benchmarks benefit from more samples (up until ~1K samples) as expected. However, we find dimishing returns for some benchmarks after only 1K instances, in particular the signal-to- noise ratio for AutoBencher shows an inflection point at around 2K instances. This is due to the AutoBencher having high noise, as shown by the scaling law standard deviation (right figure) noise.
despite having the largest sample size, AutoBencher has the highest tneckpoint-to-checkppe In fact, the 300 instance subset of ARC-Easy has lower noise than the full 30K instance AutoBench. As using LLMs as part of benchmark construction has become more popular method of constructing benchmarks, high quality, small benchmark can actually show a less noisy signal.
B.3 Sional-to-Noise Ratio Large (>32B) Scales
Setup. For models larger than the DataDecide scale (1B-100B), we can rely on the signal-to-noise ratio directly to indicate development benchmarks which may not be useful. We estimate the signal- to-noise ratio at the compute scales used to train the OLMo 2 models: 1.5B-4T, 7B-4T, 13B-5T and 32B-6T. For noise, we use the final 30 intermediate checkpoints, one checkpoint for every 1000 training steps until the end of training. For signal, we do not have access to different data recepies trained on the same model, so instead we use a population of geinh base models trained to similar compute budget the OLMo 2 models. We use models trained using 100% of the estimated FLOPs, which results in a population of at least models for each size.
Results. Table reports the SNR for each compute budget, sorted by SNR at the 1.5B-4T model scale. SNR can indicate when benchmarks saturated, for example ARC Easy and SocialIQA have high SNR at 1.5B-4T, but low SNR at 32B-6T: 7.89 to 5.10 and 8.73 to 1.95 respectively. For these
22

---

Table 4: Signal-to-noise ratio for language model development benchmarks for the compute For benchmarks measuring a similar ability, we recommend
budgets of the OLMo family 422 using benchmarks with higher si signal-to-noise ratio ratio for particular model scale. Performance on all models is shown in Figure 12
Model Size
L-B-4T 2 10 FLOPS
7B-4T 1 L6 10 FLOPS
13B-ST 3.9. 16 FLOPS
32B-6T 1.2: 10 F1.0Ps SNRsigot/Soo
Compute -
Benchmark
SNRaigra/NWee
SNRaggra/Na
SNRagut/Nwe
Knowledge QA Tasks
HellaSwag
193.7T.180/0.000
23.94 23.944.om /0./0.008
1T.81q.054/0.0m.
8.200.00 028//0.05
TriviaQA
28.15g 411/0.015
17.03.
13..000105/0...0.. 135/0.003
60.3To.148 11..004000000
27. 19g 399.064/0.002
Jeopardy
23. 23..66.37740...
14. (4.38g.082 28.062200.00
18.499.084/0.00
8.00 8.00g.082/0. 000/00000004
OLMES Gen
19.3/4g 247/0.01
32. 32.58g.120 120/0.0000
1.199.002/ 002/0.02/..... 02a
1,000 048/0.046
OLMES Core
19. 19.11g.118/0.00
9.6 9.61o.039/0 1..03//0.000
7.13p.000/0
8. 16 06..02//..... 003
AutoBencher
17.62g. 77.62g.264/0.00 264/0.015
11.42g. 11.426.102/0 102/010///0000
8. 8.23.. 105/0.
3.730 050/9.034
MMLUU Pro
264/00024
17. 17.44..166/ 160///0000000 /0.0010
9.34.. 0344.099//.00. /o.010
15.04g.13e 10/000009
MMLU
14.52g.139/0.0
3. 3.390.078/0
7.5 7.510.044/0
19..001/0..02
PIQA
058/0.044
5.31o.025/0. 025/0.004
3.522.025/0 2220022//..00
4. 977..115/..00
WinoGrande
1d.12o.119/0.00
7. 7.350.062/0.0 008
7 08.00700/..000 /000
6. 6.600, 040/0.007
CommonscusoQA
12.17 12.177.120/0.00
5. 66
2 69.9
00000.30/0.oooooo 00..033/0... 7. 7.00p
DROP
10.79g.a57/0.08
20.79 202//92//00000 201/0.018
19 226/0.019
1..10.143/0.02
ARC Challenge
9.4 163/00000
5.85p.oe 01.014
2.32o 033/0. 01
4.74o. 00oo0004/..00 014
SocialIQA
8.730.119/0..
5. 0.155.045/0.00
1. 020.020/0.01
1.95g 955.0226/.0033
MedMOQA
8.59p. 1o/o..00//000 106/0.012
5.790.05 /o, 00a
7. 7.7Uo.0n0/o.0 o0../..//.... pos
4.000 041/0.010
ARC Easy
7.89 102/0.01
5.770.00s
3.940.018/0.
5.10 0..00.018/0.000 oos
SQuAD
6.110,080/0.00
9.766.
10. 10.45g 044/0.004
3.92o. 220/00277.000
AGI Eval
5.31o.106 /0.020
4. 1.230.a06/0.. 018
2. 2.740.080/0.03
6.400.00 000.0622/.002
BoolQ
4.87 1.87o.116/0.0
2.990. 2.990.045/0
1. 1.180.016/0.0
2.670. 01700000/..0
OpenBookQA
p.82p.1 145/0 030
2. 253//. 025
2. 2.420.045/0.0 0200
3,05p 063/0.00000. 021
Math Tasks
GSM+
13.07a. 200/0...0.......
8.55 2..50.299/0... 200/0.033
8.422. 198/0.9
GSM Symbolic Pl
7818p.a01 10.116
4.850.877/ 377/00144
6.540.400/0 4b0/0.09
5.3 5.31o.277/0.00
GSMBK
830.687/0.11
8. 8.21o.484// 484/0.066
6 265.0
6.61o. 1.6.oo160/0.000 160/0.024
GSM Symbolic P2
62.2...n/ /0.222
2.98o.re9/0. 268
3 39...800/..16s
4.67o. 1.670.4e8/0.11 468/0.100
GSM Symbolic
3. USg 20.217
8.94 8.940.c2r
6. 1.6.o.228. 203/o/./...oo
4. 4.29,. 1.240.184/0 180/18840008.
Minerva MATH
28 50. 250
9. 9.320.643. 009
7. 7.48p.5077 507/0 076
10. 10.19 400/0.000
Minerva MATH 500
0.91o.49 0..11.49//00 589
4. 4.450.745/0. 168
4. 4.440. 4.440.647/0.1 647/0..66
4.30o, 383/0.888.0080 080
Code Tasks
HumanEsale
3.70p.e 13.
7.18 7.18u. 80..////..00 /6.060
8. 8.470. /0.045
3.34,. 10140.111/.0000
Humanbival
3,64,, 64.444442/00 48420.......0 124
6. 6,250, 396
5. 5.188. 18..884/0..00
3.199.117/0.06
MBPP+
207 235
3. 00.0000///.00 064
4. 2.720.205/o. 262.0205/0008
137/0.477
MBPP
0. 0.886 261
5. 39..0.0
4 1.520.265/0.d0
3.57 1677//0070
Multi-task Averages
Knowledge Tasks
17.70g 17.700.148/0. 140/0/..../000000
1.61 1.610.000/0.o
9. 9.82g.048/0.08
1.03 10300088/..00
OLMES- Gen
17. 17.35g 1 1.43/0.00 ooe
2. 074407./0.00 0ns
9. 52t..4t//o... nes
0.930 www./////..... 056
All Tasks
13.92 152/0.011 011
3.68p. 100.122//0.0000 128/0.000 0..
9.266.055/o.
2.940.075 028
Math Tasks
5.78, 78.78.66/00.1
11.72,.6500 6.0/0.0000000
5. 36.0..5400....
7.870.258/ 258/0588
Code Tasks
3.280.335/0.102
3.20o.371/0.. /o.00
8.87o. 3087..00/0.oo /o.
126/0.022
benchmarks, they have less powerful comparisons at larger sizes. SNR also indicates when particular benchmarks become useful. For example, Minerva MATH 500 has the lowest SNR of all tasks at 1.5B-4T (SNR = 0.91) but much higher SNR already at 7B-4T (SNR = 4.45).
Additionally, some individual tasks show better SNR than mutli-task averages. For the OLMES Core average, HellaSwag has higher SNR at all model sizes. For OLMES Gen, TriviaQA has higher SNR at all model sizes. In cases where the SNR of the mutli-task average is low, like the OLMES Average, we recommend comparing models based on individual, high SNR tasks.
C Additional Results
We include for our core experiments across all benchmarks we study:
23

---

Diseosion
Dioursion
M--a7ad CLSA ane
o 2.188
Co0 Sa Accurscd
Do0 34
Doc Docsiun
0.5
0.3
1000
10.0 SKR Dota Rd. Std Stop Rol SL.
3AR Dota Rd. Dispersion Step ld. Std
5MR Doto Rd. Dipersion Stop Rd Stl
5MB Dota Idl. Sid/ Stop foll S12
Noan Furwiss stance
wartiatttiee Range
Disaacee Standard Der
Desiation
A- 6.73 3800
3-3.455 GOLD
3--3533 GO0B
0-0050 6,00 TLaa
Docs Accurscy
300
10 1060
SNR Dur No. HID /Stpp Rel 51d
Sti - Dusa ICR/ Stop RLL 54d
SKI Duta Dttt S0d Stop Id.. Sid
5 - DUtL R95 DOV/ Stop Poll 30
1.0
3- GOT 3850
Painess Diatance
B- & GOLD
Diuperaion
Ranga
1-0 6603
Quartle Ceclation
Dods on AlCuuNCy
DoGs on Accuracy
Doda on
8.5
35
35
20
1e 4. 1 8.0
20
2.5 B
26
casies
50R - Data WTD / Step fdd. Sid
SNR Data Diapardion / Siep tal. Sid
510 Data lange Stop Ro. Std
5VR - Dala Quartie Der SSop Rel 3td
Quarti Devistorn
Auarage Atsclate Dev
Rebust Bange
Robur Sange
1.0
R- G4T2 3550
10
R- 3,675 GOLD
A- 0-550 6600
Dicsiion AcCunNyy
Ducaionn Aucuracy
8.5
35-
41 3353
10 T.80
23
46 100
100
5U0 - Duta Qua de Dav/ Step Rei Std
SMR Data ADD Step Ral 3td
31R Data Robual Range Step Rei Std
5MR Data Robust Bange Step Tal. Sid
Abunlie Dextatoon
Medaan Aburuza Deciation
Nean Squared talte
Hean Squared make Dietance
1.0
4 SELL
R- 2.46 GOLL
18
3.52
Dicsion
25-
050 656.6.885
40.333
403333
336613381 020384 6
6303 (0391) 120385008
3NR Data NAD Step Rel. Std
5N3 Daca NAD/ Step Fel. Std
SMR Data tel NSFO 5See Re: Std
SMR
Data HSPO/ Step Rec 31
1.0
Mi Eut0 5317
GHi Coeffcert
Star Decrepnnyy Scal
Star Rel Diceesancy
4-uttte BAGR
Dispersion Shift Scale)
DecEAM Accossy
Decosar Accoracy
30 9316
23 28
28
SNR Duto Gini Step Rel Std
SKR Duta Star DEseNsancy Stup ALL Sti Shit Cato Star Not Disressncy/ Stop Rd. Sid
SMK Daca Dipur on Stop RiLL Skd
Ma Iapacce Duptis
Diccepancy
Projection Depth
Star D'ccnpancy
Dock Accurscy
Docs Accurscy
Docs Accurscc
J8h6 you
5KS Dala Hulispoc Desth
5 Duta Dicresancy Stop Ret Std
5V5 Dela Projuction Desth /So0 Std
54h Data Star Discresancy Stop Pot
Correlation between decision accuracy and variants of signal-to-noise ratio, using different measures of signal. To pick the measure of signal, we use the metric which is most predictive of decision accuracy.
Figure 10:
24

---

Noise
100%
GSM Symbolic P1
DROP BBH
Minerva MATH
HomanEval
MedMCQA
Math Tasks
Scaling Law Prediction Error
MBPP
Jasks
ARC Challenge
WinoGrande
10%
GSMBK
MMLU Pro
AgI Evat
AutoBencher
ARC Easy
OLMES Gen
MMLU
OLMES Core 9
MBPP
OpenBookQA
SQUAD
TriviaQA
HumanEval+
SocialIQA
wledge Tasks
All Tasks
PIQA
1%
CommansenseQA
Jeopardy
0.1%
R 0.653
0.068
R² 0.426
HellaSwag
0.1
0.01
Rel. (fdnafinal n train checkpoints)
Figure 11: Scaled-up up version of the Figure in $4.2
with labels on each task.
25

---

ABC Challengs UOSIIEE
ABC Easy
Bhorg
Commo MARTRA
07
13-
04
R.0%
Oomout 12V PLOR!
1.0%
OperBeekOA
PIDA
SoCiARDA
Rill
0..
64
Oomoute (2A) PLON!
1CH rLOh
Compite F.00
DROP
CONEK
Irepanty
1Dn rLOh)
Compone 1.0%)
SQuAO
THAANQA
MBFP
HumanEval
Human Evall
Autodencher
GSM
Pymayy sooe
22
2SM Symbotc
CSM Symbolc
MesMICAA
Mieena MATH 5.00
2211
Figure 12: Performance language models from 60M parameters 32B parameters, which use to measure spread at different training budgets in Table For our core experiments, we use the DataDecide models measures spread, and large scales, external models trained similar compute budgets.
26

---

TrivinDA
PIDA
OLMES Core 9
HellaSweeg ag
Pinury Store
9.
0.3
3,502
-0.002
0.003
Ménury Store
0.84
0.32
70.000 004
10004
Preunn Soore
3.75
-0,003
A.0000
4,00
Prnuun Store
3.003
0.20
5.70
0.6
0.5
5.015
0.75
Ww
Man00
2.65
0.70
0005
10K
20C
23K
13K
20K
10x
36K
S0K: Vairingg Steps
Ria Traiing Stess
Finul Trinning Staps
min Staps
ARC Fasy
Jopardy
CommonsensigA
Boolg
026
0.85
005
Primary Score
0.85
0.50
0.00
9.75
Vams
0.024
10K
20C
23K
33k
20k
30k
30K
30K Pairing Steps
Trailing Stess
33K Training Steps
ming Stops
SQUAD
WinoGrandd
MMLU Pro
MedMCQA
00.007 007
p.009
0.425
0.05
Pamary Some
0.25
0,011
Pomany Srom
Pinnry
0.00
0.75
5.20
70.05
0.15
0.325
0.70
009
011
0.300
10K
20C
10C
10C
26K
33K
10K
20K
10x
10K
20K
30K
32K
Fairing Steps
Ahu Traiiin Stees
Final Triniing Staps
Thu ming Stops
AGI Eval
MMLU
8BH
SocialiQA
0.650
0.40
Penury Sigre
0.55
Prinuyy Score
Primary Soore
0.012
Prosay
6.35
0.50
D0G
A01s
0.600
0.45
0.4
0.575
8,020
0.40
0.3
10K
20C
25k
13k
20k
10ic
3kk Staps
35R Vairing Staps
Training Stess
Triniig Staps
AutoBencher
ARC Challenge
DROP
OpenBookQA
0.75
"a.01
0.6
016
0.70
0.50
(009
Pinary 500
0.60
A.91A
Prnur Score
0.5
Pamuny Soare
0.50
0:55
A003
0.50
0.00
0.55
0.35
0,021
0.50
10K
20C
30C
0K
10K
2:K
33K
10K
20K
30K
26
10K
20K
30K
Final 32K Braining Siepa
Fit 10K Trassig Stess
Fiall Thitiing Skaps
ming Stops
GSM+
GSMIK
GSM Symbolic
HumannEval
AU000024
3.4
Primay Soae
0.1
Pronury Sorm
Penwyy
6.2
0.1
Googgrd 126
0.0
10K
20C
10C
22k
20K
10x
32K Baining Steps
Aio Training Stees
Final Fultsing Staps
ming Stops
HumanEval+
Minerva MATH
OLMES Gen
MBPP
0.4
0.125
0001
3,046
Prou Scor
0.3
0.2
645
0.300
0.075
07777
Prnury Score
0.6
004
Pernary Soors
0.5
254
0K
10K
20C
30C
0K
10K
20k
30X
06
12K
20K
30K
26
10K
20K
B0K
Fru 32K Braining Sieps
(ho 10K Training Steps
Fmall Thatiing Skeps
Drai min Steps
MBPP+
GSM Symbolic P1
Minerva MATH 500
0.35
Jaass
Pinay Sone
0.3
Panee Sooe
3.10
18
0.000
138
328
3.00
0.545
10K
20C
30K
10K
20K
30x
20K
30x
Final 3OK Baining Stese
Nh 30K Trainin Steps
Final 30K Thatiing Stops
Figure 13: Final 30 checkpoints, each spaced 1000 training steps, for OLMo 2 1B, 7B, 13B and 32B along with the Rel. Std. Dev., which is used to estimate noise.
27

---

TriviQA
SQuAD
OLMES Gen
ARC Easy
6/6
0.70
Shilw22.5
5Mf==2.57
Mene
Primary Masti
Prtmaa Meent
636
0.55
6.0
0.00
0.5
OX
Joopandy
AutoBencher
HollaSwag
DROP
5MR=15..5
6.00
55
0.23
S1R--11 57
SNRw11:22 26
Drmmary Keeee
0.55
756
216
256
Thatting Step
Thehing Step
Th maninn Step
Teitiing Step
MMLU Pro
MMLU
ARC Challlenee
HumanEvvll
542 -10.77
0/40
SMR-2.04
Shit-1/43
0.3
5MR-5.42
PrinuyyNorrs
0.85
Pommey Meae
0:32
0.45
0.30
0.30
2:1m
Trarsing
Stop
Step
CommoreenseeAA
SociallQA
HummenEval=
OLMES Core
5:43
5MR-5/43
0.30
6.2
Minaay aitie
6.0
0.00
0.30
0:54
0.50
6:14
35
206
<36
6X
o0k
206
<bc
20K
+3K
66R
43K
OX
ook
Talning
Step
Stop
Toiting Stop
WinoGrande
PIDA
MedMCQA
0.64
5188-4.32
6.70
SNit-4:23
Shit-3.55
6.22
SMR-3345
Warte
0.62
Minnayy MoMiz
thmaay Meltit
0.4-
0.22
6.70
6.2
Training
Tranning Step
Step
Tratning Slop
AGI Eval
OpenBookQA
MBPP
Minerva MATH
6.28
ShR=2.25
Stitw3.04
ShR=197
Prinucy Nag
0.52
0.20
Primary Meae
6.22
0.27
0.28
0.30
6:0
Trsining
Step
Symbolic P1
0.02 55R m1.77
Sk 166
0.02
5AR=1:43
Himmry Mes10e
0.32
Premary Mente
01311
0:00-
0.31
206
50K
3C
20K
<0C
CéC
20k
43K
EX
00K
Thatting Step
Thanning Shep
mip Shep
Thithing Step
Minerrv MATH 500
GSM Symbolic
GSMBK
GSM Symbolic 92
0.00
Peitary Mobii
100
Petracy Mette
2.00
Am
0.00
600
5.50
Figure 14: IB-5xC training curves and final checkpoints for DataDecide models across tasks, sorted by the signal-to-noise ratio.
28

---

Ninerva NATH
NILLL
MNLU Pro
AGI Eval
Prr Gren
Score
Pr Scate
Jane
udt
1297
2p1
cplt Compure
Compuue
Comautte
Compute
ARC Challenge
ARC Easy
Boslq
CommonsenseDA
Fitraay Soore
Pitraay Sooee
E2
103 10th 319
10
3pd 13 Compane
10ah
12
Carpuree
Corratto
Compute
Hells Swrag
Open
POA
SocarIQA
Pripay Scae
Pe0a y Scoee
P-iay' Sqate
Powwy Scow
10 10t
10t
10"
Compute
Compute
Compute
Crandee
CSMEK
Jespardy
SQuAD
Beraay Doae
Ponnay Store
Prinary Store
0.00
0.5
0.
TrivieGA
MBPP
MBPP
Humannivll
0.0
335
Pemary 02
Har saaee
HumanEva
AutoBencher
GSM+
GSM Symbolic
0.30
Precay Score
25
13"
Compute
GSM Symbolic P1
GSM Symbolic
MedMCQA
Minerva MATH
3..
0.33
Pritaay Scote
Stote
Mitrary Stade
3.34
66
230 calt
Compute
Corraute
Compute
pute
Figure 15: Scaling law fits for all tasks using the OLMo 2 13B-5T prediction target.
29

---

KMLLI SMR
MMLL Diciion Accuracy
MKLU Noise at Scaling Law Target
50
16.0
13.0
160
50%
60%
65%
95%
615
005
AutoBenncher SNR
AutoBencher Deciion Acouracy
AuteBennher Noise Scalin Law larget
MMC
1781
-0/25
13,0
.012
.00
Figure 16: Larger version of Figure showing the names of each subtask, sorted by SNR from bottom (highest SNR) to top (lowest SNR).
30

---

Table 5: Scaling law error for BPB and primary score for all tasks with averaging the final checkpoints in the ladder train models.
Predicting Bits Sits-per-byte Error,
Predicting Primary Score
Abs.
Rel. Error, %
Abs. Error, %
Rel. Error, %
Task (1)
Final Only
Avg. Train
Final Only
Avg. Train
Final
Avg. Train
Final
Avg.
Only
Only
Train
Knowledge QA Tasks
HellaSwag
0.76
0.80
1.16
1.22
0.31
0.16
0.37
0.20
CommonscnseQA
6.24
5.32
8.75
7.46
0.59
0.46
0.75
0.58
Jcopardy
5.08
5.14
18.51
18.73
0.57
0.54
0.69
0.66
SocialIQA
0.66
0.41
0.74
0.46
0.50
0.59
0.80
0.95
PIQA
1.23
1.39
1.40
0.89
1.08
1.22
MMLU
0.56
0.49
0.75
0.66
1.68
1.74
3.28
3.39
MMLU Pro
0.78
0.71
0.73
0.67
1.76
1.75
7.51
7.45
AGI Eval
2.79
2.66
3.33
3.18
1.89
1.98
5.43
5.70
OLMES Gen
4.66
3.92
4.19
2.16
6.22
3.20
BoolQ
1.49
1.76
8.54
10.11
4.13
2.48
4.91
2.96
OLMES Core 9
0.47
0.25
0.62
0.33
2.47
2.62
3.23
3.42
TriviaQA
1.56
2.05
2.27
2.98
2.33
2.62
2.89
3.25
SQuAD
4.96
4.96
32.35
32.37
2.80
2.79
3.23
3.21
OpenBookQA
3.18
2.80
4.02
6.22
5.22
AutoBencher
2.92
2.78
4.70
4.49
3.86
3.69
7.47
7.14
ARC Easy
1.36
1.37
2.89
2.90
5.13
5.13
5.87
5.87
MedMCQA
5.07
5.38
3.35
3.67
7.72
7.98
19.72
20.41
ARC Challenge
2.08
3.15
3.14
8.44
8.43
13.02
13.01
WinoGrande
1.01
1.38
0.83
1.12
10.01
10.82
12.47
13.49
BBH
61.84
65.01
12.81
13.47
33.09
33.08
66.61
66.59
DROP
47.51
48.19
10.75
10.91
35.17
35.20
68.77
68.82
Knowledge 19-Task Avg.
1.18
0.87
1.32
0.98
1.43
1.20
2.22
1.85
Math Tasks
Minerva MATH
0.73
0.66
1.50
1.36
1.08
0.98
15.28
13.93
Minerva MATH 500
0.34
0.14
0.71
0.29
17.35
1.78
306.18
31.36
GSM Symbolic P2
2.57
2.83
5.23
5.75
7.46
3.50
164.53
77.13
GSM8K
2.43
2.48
5.90
6.01
7.46
3.85
20.55
10.61
GSM+
2.02
1.95
4.54
4.40
29.14
28.54
130.01
127.36
GSM Symbolic
1.87
1.71
4.64
4.25
39.88
38.88
132.62
129.30
GSM Symbolic
2.31
2.35
5.04
5.11
27.15
83.62
178.46
549.63
Math 6-Task Avg.
2.05
2.01
4.52
4.42
11.33
2.30
65.52
13.28
Code Tasks
HumanEval+
1.92
2.21
3.57
4.10
1.05
0.04
3.91
0.16
MBPP
0.30
0.32
0.46
0.48
2.57
1.79
11.63
8.10
MBPP+
6.49
6.62
12.56
12.81
9.08
8.79
33.14
32.11
HumanEval
1.59
2.01
3.85
4.87
7.71
8.85
24.00
27.55
Code 4-Task Avg.
3.23
3.33
6.07
6.25
3.15
2.75
11.61
10.15
All 30-Task Avg.
0.47
0.15
0.62
0.20
1.03
0.86
2.10
1.76
31

---

Table 6: Decision accuracy averaging the final checkpoints for bits-per-byte and the primary metric (aceuracy, exact match, pass@1 I).
Bits-per-byte, Avg. Pred
Primary Metric, % Avg. Pred
Task (1)
Final Ckpt
Avg. Target
Avg. Both
Final Ckpt
Avg. Target
Avg.
Both
Knowledge QA Tasks
ARC Challenge
94.56
94.88
94.38
94.67
82.91
82.27
82.91
82.00
HellaSwag
92.42
93.19
93.21
94.00
71.05
71.26
72.37
72.33
ARC Easy
92. 92.23
92.15
91.96
92.00
93.96
93.99
94.05
94.00
MMLU
91.53
91.64
91.63
91.67
89.08
88.84
89.60
89.00
AutoBencher
88.55
88.95
89.19
89.67
88.80
89.05
88.81
89.00
MMLU Pro
90.00
89.40
90.04
89.33
83.34
83.77
84.20
84.67
AGI Eval
86.38
86.75
86.54
87.00
57.38
58.60
56.45
57.67
MedMCQA
86.67
86.67
86.67
86.67
61.33
61.33
61.33
60.33
Jeopardy
84.12
84.46
81.88
85.00
83.01
82.60
83.74
83.33
TriviaQA
83.55
84.29
83.86
84.67
69.10
69.54
69.09
69.33
OpenBookQA
81.53
81.75
81.68
82.00
66,82
66.98
68.05
68.33
OLMES Core
79.05
79.32
80.33
74.67
73.92
74.24
73.67
SocialIQA
79.92
79.57
79.45
79.00
55.58
55.58
56.09
56.67
WinoGrande
73.20
74.29
72.83
74.00
50.52
50.27
49.81
49.00
PIQA
72.60
72.91
71.93
72.00
72.78
72.66
73.09
72.33
CommonsenseQA
65.86
66.25
65.42
65.67
68.74
69.05
70.61
71.00
BoolQ
63.72
63.51
64.00
50.38
48.90
50.66
49.33
SQuAD
60.93
60.59
62.02
61.67
58.69
58.35
59.72
59.33
OLMES Gen
61.16
55.44
55.11
58.86
62.06
54.87
53.42
50.12
DROP
56.67
56.48
57.46
57.33
57.77
59.06
57.80
59.33
BBH
57.48
57.25
57.66
57.33
59.15
59.88
60.85
61.33
Knowledge 19-Task Avg.
71.39
71.49
71.62
71.67
70.70
75.82
72.65
78.00
Math Tasks
Minerva MATH 500
90.33
90.33
90.33
90.33
51.00
51.00
51.00
51.00
Minerva MATH
90.00
90.00
90.00
90.00
51.00
51.00
51.00
51.00
GSM Symbolic P1
81.33
81.33
81.33
81.33
41.67
41.67
41.67
41.67
GSM Symbolic P2
79.67
79.67
79.67
79.67
40.33
40.33
40.33
40.33
GSM+
79.00
79.00
79.00
79.00
59.67
59.67
59.67
59.67
GSM Symbolic
78.33
78.33
78.33
78.33
51.67
51.67
51.67
51.67
GSM8K
76.67
76.67
76.67
76.67
46.33
46.33
46.33
46.33
Math 6-Task Avg.
88.33
88.33
88.33
88.33
42.67
42.67
42.67
42.67
Code Tasks
Hvaln
96.33
96.33
96.33
96.33
71.33
71.33
71.33
71.33
HumanEval
95.67
95.67
95.67
95.67
80.00
80.00
80.00
$0.00
MBPP
95.33
95.33
95.33
95.33
76.00
76.00
76.00
76.00
MBPP+
93.00
93.00
93.00
93.00
70.67
70.67
70.67
70.67
Code 4-Task Avg.
96.67
96.67
96.67
96.67
85.67
85.67
85.67
85.67
All 30-Task Avg.
68.57
70.63
69.78
71.33
62.15
68.88
67.29
77.33
32

---

Figure 17: Bits-per-byte vs. primary metric the full suite of tasks shown in Figure
Experiment Setting
SNR (t)
Rel. Error (L). %
Decision Acc (1). % Primary
Metric
Primary
BPB
Primary
BPB
BPB
Knowledge QA Tasks
TriviaQA
27.9
61.8
2.5
0.5
68.3
85.3 3
SQuAD
23.8
27.8
61.7
OLMES Gen
23.1
20.6
0.9
2.6
63.3
67.3
ARC Easy
21.0
64.6
5.3
0.8
93.0
93.0
Jeopardy
20.2
22.6
3.5
18.6
82.0
83.0
AutoBencher
15.9
31.3
0.2
4.5
89.3
89.3
HellaSwag
11.8
14.9
1.4
1.0
74.3
95.3
DROP
11.5
9.9
59.0
11.3
57.3
58.7
OLMES Gen
11.2
40.0
2.1
0.4
89.0
89.0
MMLU Pro
11.0
27.6
2.7
1.3
83.0
89.0
MMLU
35.9
0.4
92.0
ARC Challenge
6.6
44.8
9.7
2.1
83.3
95.0
CommonsenseQA
5.5
41.9
3.6
5.9
68.7
65.7
SocialIQA
5.5
48.0
0.4
1.9
55.0
80.0
OLMES Core 9
5.4
73.2
0.2
73.3
79.3
WinoGrande
4.6
3.6
10.3
0.9
49.7
75.0
PIQA
4.2
8.8
0.5
1.3
73.3
72.7
BBH
3.6
2.5
67.1
12.9
64.7
55.0
MedMCQA
3.5
29.5
8.8
4.6
60.3
86.7
AGI Eval
13.7
3.4
58.7
88.0
OpenBookQA
2.1
24.2
7.7
3.3
65.7
82.7
BoolQ
1.5
64.8
5.1
6.6
47.7
62.3
Knowledge 19-Task Avg.
13.7
44.3
0.8
1.0
79.0
80.0
Math Tasks
Minerva MATH
1.9
88.6
11.9
1.9
51.0
90.0
GSM+
1.8
7.3
20.0
4.8
59.7
79.0
GSM Symb.
1.3
83.0
5.1
51.0
78.3
GSM8K
1.2
7.0
38.6
5.9
46.0
76.7
Math 6- 6-Task Avg.
1.8
22.6
46.0
5.0
42.3
88.3
Code Tasks
HumanEval
6.1
25.1
9.2
7.9
74.3
95.7
HumanEval+
5.5
27.4
29.7
7.1
66.0
96.3
MBPP
23.6
1.0
68.3
95.3
MBPP+
1.7
30.8
39.5
8.9
62.7
93.0
GSM Symb. P1
1.6
6.6
538.6
5.2
41.3
81.3
Minerva MATH 500
1.4
90.5
52.5
0.9
50.7
90.3
GSM Symb. P2
74.8
5.1
40.3
79.7
Code 4-Task Avg.
5.5
42.0
29.5
9.7
80.3
96.7
All 30- 30-Task Avg.
10.0
0.4
77.0
83.7
33

---

Challenge
Easy
BoolQ
CommonsenseQAA
11
Dcttion Accursyy
Decision lucuray
Deciston Accuney
Decttoo Rccuracy
0.95
0.8
0.7
0.7
0.90
0.90
0.6
0.6
Training Stop
Training Stop
Tralning Stop
Triiiing Stop
HellaaSwag
OpenBookQA
PIQA
SocialiQA
Octtion Accuras)
1.00
Dectiin Accuracy
1.0
Decivion Arruewyy
1.0
1.0
0.9
0.9
0.9
0.95
0.8
0.8
0.8
Deddton
0.7
0.7
0.50
6K
20K
40K
60K
00
205
40K
60K
0K
20K
40K
60K
20K
40K
60K
Training
Step
Towining Step
Avg.
Dectioon Accuracy
Deciaton Accuray
Deciiton Acturayy
Smoothing (Ni
EMA (N-55)
0.7
0.7
EMA (Ni-20)
Checkpeint
0.90
0.6
Thaining Step
Training Step
Thaining Step
Figure 18: When stopping training early, averaging the checkpoint-to-checpoin noise improves the decision accuracy between an intermediate and the final training step. Shown are decision accuracy from early-stopping for the core OLMES tasks by using both single checkpoint and the exponential moving average (EMA)
34

---

Easy
Easy
0.7
Accuracy
0.6
0.5
0.700
0.675
0.700
0/0er
0.4
0.3
20K
10k
23K
40C
23K
Commonsenss@AA
CommonsenseQA
CommonsenseQA
beeviingeed
fite
berye mter gell
AcCus-))
0.5
DA vo SO00
0.625
seed
0.4
ondor
0.355
0K
23K
40%
50k
B3K
0%
23K
406
b0k
0K
23K
42K
60k
80C
SocialIQA
SocialICA
SocaalQA
23 Ran Ivryingg recc
fur very dut order
10 10 harvrr sns dea arter)
0.50
0.50
Accurecy
0:45
2.52
Food
order
0.40
0.40
60K
23K
ADIC
tOK
23K
60K
breving
0.6
ondert
hur baryer
Accuncy
0.5
LOcal MartsSoon
0.6
0.600
deta
mobe
0.3
Challenge
Challenge
Challenge
Accurscy
0.35
0.35
0.35
2.38
0.25
seed
0.55
0.25
TMCJ
0.20
0.20
0K
20K
40%
60K
B5K
0K
20K
40K
50k
0K
235
60C
30C
Winogrande
Winogrande
Winogrande
baayligg
Ewary
hayeng san waar
0.575
0.575
0.550
0.50
0.550
0.525
seed
0.575
order
0.525
0.58
0.500
0.58
0.775
0K
20K
40K
60x
8:K
0K
20K
40C
60K
2.5
4.KK
60K
80C
PIDA
PIQA
Delyynng seed
0.75
0.75
ACcUraCy
0.70
0.70
0.65
0.60
0.750
seed
0.60
0.750
order hoise
0.50
0.725
S0K
20K
46<
2:6
60K
0.36
0.34
0.34
0.34
Accuracy
0.32
0.30
35
0.30
0.28
3.54
order
0.28
0.28
0.26
23K
23K
606
Training Step
Training Step
Training Step
Figure 19: Visualization for the seed noise, data order noise and total variation for all OLMES tasks.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract accurately summarizes the main contribution and scope of the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss the limitations of our work as part of $
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
36

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We specify the open models and datasets used to perform the core evaluation of the work in 2.2 and describe the exact evaluation setup in Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
37

---

Answer: [Yes]
Justification: We will provide the code used for launching and analyzing evaluation, and will release the full dataset to reproduce the tables and figures part of the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/ htte://ddde//eeeuuuieeeeee ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https Cttlpb.cc/pbllli http:///iiieouuieee//eeeeeeeeeee www.iislsoosoooooioooooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify these details for collecting evaluation results 2.2 and describe the exact evaluation setup in Appendix A.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For our main results on the correlation between SNR and decision accuracy, and correlation between SNR and prediction error in $4 we report confidence interval.
For our experiment randomizing the sub-task in $5.1 we report error bars for the standard deviation using different random selections.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
38

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative еттоr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In our description of the models used to run experiments AA.5.1 we describe both the compute used run evaluation and the compute used train the IB seed and data order models discussed in 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https////eerrrss.cccplollccccce. https/////lll//tticiiiiieeeeeee.
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method performs fundamental research on language model evaluation, and is not tied to particular application. We do not see any direct societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
39

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper cites the original authors for all of the benchmarks and models used in this work, as listed $A.3
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
40

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com com/daaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper will release the accompanying code and assets part of the supplimentary material, and will provide documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not perform crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not perform crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
41

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for implementing the core methodology the work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httt:p///eerrrppssccc//ofrreeeee what should or should not be described.
for
42


Reviews:
[{"review_id": "KytxgTtKZK", "reviewer": "Reviewer_RGQH", "summary": "This work tackles a “meta-evaluation” problem: how to evaluate the reliability of benchmark suites when we must make model-selection or scaling decisions from small-scale experiments. The authors design two complementary properties of a benchmark—signal (its power to separate stronger from weaker models) and noise (its sensitivity to stochastic variation such as checkpoint choice)—and show that the signal-to-noise ratio (SNR) strongly predicts decision accuracy and scaling-law extrapolation error across 30 diverse benchmarks and 465 open-weight language models (60 M – 32 B parameters). They then introduce four practical interventions—metric replacement (e.g. perplexity over accuracy), noisy-benchmark filtering, checkpoint averaging, and other dataset refinements—that each raise SNR and yield more robust conclusions.", "questions": "Could you provide examples of the datapoints you filtered out (e.g. from MBPP) and clarify whether they were incorrect, ambiguous, or otherwise problematic?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Rather than analyzing individual models or tasks, the paper tackles a \"meta-evaluation\" problem: it studies how we reason from small models to large ones using an evaluation benchmark, a rarely explored but important area.\n2. Using pairwise ranking accuracy to quantify decision reliability avoids strong distributional assumptions and is easy to interpret.\n3. Section 3.1 convincingly shows that averaging the final n checkpoints is a stable and unbiased proxy for intrinsic variance under some assumptions (same/similar learning rate schedules).\n\nWeaknesses:\n\n1. The Decision Accuracy experiment is very close to the DataDecide paper, and the Scaling-Law Prediction Error setup mirrors prior work. More discussion/comparison is required to acknowledge the contributions in prior work and discuss novel contributions in this paper.\n2. The custom “Decision Accuracy” formula feels ad-hoc and non-standard; a rank-based statistic such as Spearman’s Correlation Coefficient (https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient) is similar, and might be more familiar.\n3. Missing comparison to EvalArena. EvalArena (https://github.com/crux-eval/eval-arena) also analyzes benchmark signal-to-noise ratio but is not cited or compared.", "was_revised": true, "final_justification": "Thank you for the detailed response. It clarifies the contribution of the paper compared to prior work and EvalArena, and addressed some of my concerns, including those related to spearman Correlation. I also appreciate the case study and I like the point of using the SNR to identify potentially problematic test cases in benchmarks. Therefore, I raised my score.", "rebuttal": "Thank you for your review and noting that our work addresses “a rarely explored but important area”! We address your feedback and questions below:\n\n**Concerns:**\n\n1. **Novelty w.r.t. Decision Accuracy and Scaling Law Error**\n\nOur core contribution is the signal to noise (SNR) framework, and the interventions to improve SNR. You are correct that decision accuracy and scaling law prediction error are from previous work, and this is intentional as our goal is to show that SNR is useful for language model development practices in existing literature. Our work does not compete with or aim to replace previous work on scaling laws or decision-making processes, instead we aim to build on that work by introducing a tool, SNR, which can be used to understand which benchmarks are appropriate (or not) for scaling laws or approaches like decision accuracy.\n\nMoreover, scaling laws and decision accuracy are two instantiations of more general steps that play a key role in building ML models. For example, scaling laws are an example of an attempt to directly estimate model performance, while decision accuracy is an example of ranking training data candidates. We believe our contribution, SNR, captures information that would be useful to other approaches beyond these specific metrics, though we leave that to future work. In our final draft, we will clarify that Sec. 2.1 is presenting background work prior to our discussing our core method in Sec. 3. Additionally, we will give more discussion at the end of Sec. 1 to further highlight our contributions mentioned in the abstract and discussed later in the paper.\n\n2. **Choice of Decision Accuracy over other Rank Statistics**\n\nWe use the decision accuracy metric for two reasons: first, to mirror its use in the DataDecide paper, and second because a particular value for decision accuracy is more interpretable than a correlation coefficient. For example, decision accuracy of 80% means that 80% of pairwise comparisons between different data mixes correctly predict which is better at large scale, but 0.8 correlation doesn’t have a similarly grounded interpretation. That said, decision accuracy is a rank-based statistic, in fact it is proportional to Kendall’s Tau modulo a scale and shift: $\\tau = 2 \\cdot (\\text{decision accuracy}) - 1$, which we briefly show below and will add to the final revision of our work: \n\nKendall’s Tau is defined using the number of concordant pairs $C$ and discordant pairs $D$ between two rankings. \n\n$$\\tau = \\frac{C - D}{\\binom{N}{2}}$$\n\nDecision accuracy is defined as only the number of concordant pairs $C$:\n\n$$\\text{decision accuracy} = \\frac{C}{\\binom{N}{2}}$$\n\nSince we do not allow ties, $C$ and $D$ make up the total number of pairs $\\binom{N}{2} = C + D$, we can rewrite decision accuracy as follows:\n\n$$\\tau = \\frac{C - (\\binom{N}{2} - C)}{\\binom{N}{2}} = \\frac{2C - \\binom{N}{2}}{\\binom{N}{2}} = 2 \\cdot \\frac{C}{\\binom{N}{2}} - 1$$\n$$\\tau = 2 \\cdot (\\text{decision accuracy}) - 1$$\n\nAdditionally, we reproduce the experiment in Figure 2, which calculates the correlation with SNR and the decision accuracy, by using SNR to predict the Spearman rank correlation of the 25 DataDecide mixes instead:\n\n| Signal Measure   |   SNR R² (Spearman) |   SNR R² (Kendall) |   SNR R² (Decision Acc) |\n|:-----------------|--------------------:|-------------------:|------------------------:|\n| Rel. Dispersion  |              0.4902 |             0.5687 |                  0.5687 |\n\nWe find the $R^2$ value to show that SNR is predictive of both the original Decision Accuracy setting and when using the Spearman correlation. We will include this in the final revision of our work.\n\nAs our central contribution was a framework to improve existing methods of evaluating data decisions, we did not change the existing rank statistic used in DataDecide. However, we agree that other rank-correlation statistics could be used to evaluate the interventions we propose for improving signal-to-noise.\n\n\n3. **Comparison to EvalArena**\n\nThank you for highlighting this work! We will include a discussion in our final draft.\n\nWhile EvalArea defines a term “signal-to-noise ratio”, their definition is significantly different from ours. Their term is calculated using the scores of final checkpoints for two models within the same model family (e.g., 7B and 70B variants of the same model). Our work instead defines noise as the checkpoint-to-checkpoint variance of one model evaluated on an entire benchmark during training and signal as the spread of a population of models trained at the same compute budget. \n\nConceptually, our measure uses modeling noise, in the form of checkpoint-to-checkpoint noise, while EvalArea’s measure of noise is based on the statistical variability of the dataset. A benchmark may have low statistical variability but high checkpoint-to-checkpoint noise (e.g., a benchmark with large sample size but also a high inter-instance correlation, which we observe with BoolQ). Also, our measure of SNR is calculated at a particular model scale (1B, 7B, 13B, etc.), instead of using the difference of scores between model scales to assign a single score to a benchmark. This leads to distinct values for SNR at each model size, as we find in Table 3, which allows us to state which benchmarks are useful at a particular scale.\n\nEmpirically, we can briefly validate that the EvalArea formulation of SNR will result in a different ranking of benchmarks by using the experiment in Figure 2 (right). In this setting, we calculate signal-to-noise using the implementation in the EvalArea setup for each small DataDecide model {60M, 90M, 150M, 300M, 530M, 750M} to the 1B model. We then calculate the correlation between the SNR term proposed in our work, and discussed in EvalArena, with decision accuracy:\n\n| Measure of SNR                                 | $R^2$ with Decision Accuracy at 1B |\n|:-----------------------------------------------|----------------------------------:|\n| Signal-to-Noise in our work                    | 0.569                             |\n| EvalArena Signal-to-Noise                      | 0.181                             |\n\nWhile we believe the statistical measures discussed in EvalArena are important indicators of dataset noise, we find in the above table that our SNR approach, based on measuring the checkpoint-to-checkpoint modeling noise, is more predictive in the decision accuracy setting.\n\nFinally, our work goes beyond measuring the noise of existing benchmarks by applying our measure of SNR by designing interventions to improve the evaluation procedure (in Section 5). We believe EvalArena’s work and other related work discussed in our Sec. 6, such as estimating dataset noise using bootstrap confidence intervals, are complimentary methods to the signal-to-noise framework we propose in our work. \n\n**Questions:**\n\n> Could you provide examples of the datapoints you filtered out (e.g. from MBPP) and clarify whether they were incorrect, ambiguous, or otherwise problematic?\n\nThank you for this question. We will include examples of instances in our final draft. We note that our filtering experiment in Section 5.1 excludes subtasks of MMLU and AutoBencher rather than individual instances; to address this question more fully in the paper we will include the following analysis.\n\nIn Figure 15 (Appendix D), we include the exact subtasks filtered in the experiment ranked by their signal-to-noise ratio, which correspond to the subsets that were included in Figure 4.\n\nWe can use prior work that has explored the quality of MMLU subtasks to indicate whether the filtered subtasks may have been problematic. If we compare the list of 57 MMLU subtasks ranked by SNR (Figure 15, top left) to the 57 MMLU subtasks listed by the number of errors annotated as part of MMLU Redux (Table 5 of [1]), we find that out of the 20 MMLU subtasks which contain errors in least 5% of instances, 10/20 of these subtasks are also in the lowest 20 tasks sorted by their signal-to-noise ratio. This presents evidence that low SNR may indicate low quality tasks, and we believe this is a good opportunity for future work in evaluation development.\n\nComplimentary to the experiment in Sec. 5.1, we believe that SNR can be helpful for identifying problematic evaluation setups beyond filtering subtasks. For example, in Figure 8 in Appendix C.1 we find that BoolQ has a particularly low SNR despite having a large sample size. We believe this is due to the benchmark formatting only including ‘True’ and ‘False’ continuations where small models can perform particularly well by only guessing the majority class. \n\n[1] Are We Done with MMLU? (Gema et al., NAACL 2025)"}, {"review_id": "tnfkb1FKoL", "reviewer": "Reviewer_bZgC", "summary": "The paper analyzes specific properties which make a benchmark more reliable and useful for assessing LLMs, and interventions to design higher-quality evaluation benchmarks. \nThey discuss two key metrics of a benchmark : signal and noise. Signal measures the ability of the benchmark to differentiate between better and worse models, while noise measures the variability of different training steps of a given model on the benchmark. They show that benchmarks with high SNR are more reliable to extrapolate decisions made at smaller models to larger models, and less noise leads to lower scaling law prediction error. To use this insight to design better benchmarks, they try some interventions like using metrics which have higher SNR, and averaging the model outputs from multiple checkpoints. They also identify smaller subsets of existing benchmarks, which have a higher SNR, and improve the performance on both the tasks, despite having less than half the number of samples of the full benchmark", "questions": "- By definition, computing the “signal” of a benchmark assumes that the models being evaluated actually differ in their quality/abilities. How does one determine if this is truly the case for a given set of models? It seems like a recursive problem, to select the right set of models to evaluate the benchmarks, and the right benchmarks to evaluate the models?\n\n- For the experiments on averaging checkpoint-to-checkpoint noise, the text in section 5.2 says that you average the target model in both settings, and only change whether you avg the smaller models used to fit the scaling law. However, in Table 1, right table, you use the term “Avg Train”, which is not clear. Does it mean averaging both the small and target models?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The problem discussed in this paper is highly relevant in current time, when new LLMs and benchmarks are released at a rapid rate, making it hard to determine which benchmarks are reliable for comparing different LLMs. It is even more useful for someone developing LLMs : training smaller language models and evaluating them on benchmarks to extrapolate the performance of the bigger models is a very common practice, since it’s infeasible to experiment with training the full model directly. \n- There are extensive investigations done to tease out the effect of signal and noise on the decision accuracy and scaling law prediction error. The figures related to the investigations are comprehensive. \n- The paper is very well-written, with clear descriptions of the key terms like signal, noise, decision accuracy, scaling law prediction error. I like that the authors explored different notions of noise, instead of just choosing one. \n- The hypotheses are well investigated through extensive experiments.", "was_revised": true, "final_justification": "I stand by my initial rating of 5 (Accept), after considering the discussions with the authors in the rebuttal phase.", "rebuttal": "Thank you for your review and noting that “[t]he paper is very well-written” and that “[t]he hypotheses are well investigated through extensive experiments”! We address your questions below:\n\n**Questions:**\n\n> By definition, computing the “signal” of a benchmark assumes that the models being evaluated actually differ in their quality/abilities. How does one determine if this is truly the case for a given set of models?\n\nThis is a good question, and gets at an important idea: specific measurements of signal and noise are really only valid for the set of models and benchmarks used to make those measurements. They might not generalize to other models, especially if those other models are significantly different than the ones used to calculate signal and noise (e.g., they are significantly larger or smaller, or have significantly different capabilities). That said, SNR can be used to find generalizable claims; for example, if a set of models reasonably should be able to do a given task but have low SNR on a benchmark, a researcher might find problems with the format or annotations on the benchmark which they could then fix (and could have been problematic for any model evaluated on that benchmark). We do not need to assume that models differ in a particular skill to compute the signal of a benchmark. We only need to assume these models are drawn from a population of models one wants to study.\n\nFor example, one may want to compute SNR for a benchmark using models trained on  a set of data recipes at the 1B scale. Assume these particular recipes only vary sources of web data, and do not contain code. Therefore, the resulting models may show very similar scores on HumanEval, near random chance. This may not indicate that HumanEval is a problematic benchmark, but that it cannot distinguish between those particular models.\n\nWhen we compute SNR at larger scales (Table 3), the external models we used were selected with the broad assumption that the models may have been trained with different data, architecture, learning rate schedules, etc. This allows benchmarks to have a higher spread if scores are sensitive to any of those modeling decisions.\n\n> Does [“Avg Train” in Table 1, right] mean averaging both the small and target models?\n\nThat is correct, we average the final checkpoints of the target model for the scaling law experiments in Section 5.2. In Table 1, right, both columns average the “train” models (the models used to fit the scaling law prediction). We will explicitly note this in the writing."}, {"review_id": "D14pJINXs0", "reviewer": "Reviewer_CAzp", "summary": "This paper introduces a framework for analyzing language model evaluation through signal-to-noise ratio (SNR) to reduce uncertainty in benchmark-based decisions. The authors define \"signal\" as a benchmark's ability to distinguish between models of different quality (measured by relative dispersion of scores) and \"noise\" as sensitivity to training variability (measured by relative standard deviation across final checkpoints). The paper proposes three interventions: filtering noisy subtasks, averaging checkpoint scores, and using bits-per-byte metrics instead of discrete accuracy measures.", "questions": "How sensitive are the results to the specific choice of relative dispersion for measuring signal? Can you test alternative signal metrics that might be less sensitive to outliers?\n\nHow well does the SNR framework generalize to other critical decisions in LLM development, such as architecture selection, training procedure optimization, or safety evaluation? (Even theoretical arguments would be nice)\n\nIt was a bit unclear to me why relative dispersion and checkpoint-level noise specifically should predict decision accuracy and scaling law reliability? \n\nCan the authors provide a systematic analysis of when these interventions are cost-effective given their computational overhead and the magnitude of improvements achieved?", "limitations": "The authors provide a reasonable discussion of limitations, acknowledging the focus on specific experimental settings and the need for future work on other sources of noise. \n\nThe evaluation focuses primarily on two specific use cases (dataset selection and scaling law prediction) but doesn't address other critical decision-making scenarios in LLM development, such as architecture choices, training procedure selection, or safety evaluation trade-offs. As well as mentioning about the potential brittleness of the signal metric to outliers and distribution assumptions and theoretical understanding of why these specific definitions work.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\nThe experimental evaluation is comprehensive, encompassing 465 open-weight models from 60M to 32B parameters across 30 diverse benchmarks, resulting in 50K evaluation results. The systematic correlation analysis between SNR and both decision accuracy and scaling law prediction error provides strong empirical validation of the framework's utility.\nThe three proposed interventions show consistent improvements across multiple evaluation settings. The bits-per-byte intervention is particularly compelling. The checkpoint averaging intervention provides a simple but effective noise reduction strategy.\n\nWeaknesses\nThe choice of relative dispersion (max difference divided by mean) as the signal metric I believe to be quite problematic. This measure is highly sensitive to outliers and may not accurately capture a benchmark's discriminative power. More robust measures like interquartile range or statistical depth-based metrics would provide better characterization.\nMany of the reported improvements, while consistent, are relatively small. For example, the filtering intervention improves MMLU decision accuracy by only 2.6%, and averaging noise improves the 30-task average by 2.4%. Given the computational costs of implementing these interventions, the practical significance may be limited.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback, and for noting the “strong empirical validation of the framework's utility”. We address your feedback and questions below:\n\n**Concerns:**\n\n1. **Choice of Relative Dispersion**\n\nWe would like to emphasize that our work does not argue that one particular metric is the correct way to measure the spread of datapoints, rather, that our contribution is that signal, noise and SNR are useful. When selecting the signal metric, we considered 20 alternatives including some measures that are not sensitive to outliers, with results in Table 2 in Appendix A. Below, we extend these results with interquartile range and its variants, along with 1-dimensional versions of depth-based metrics (Halfspace Depth and Projection Depth). \n\nWe calculate the correlation between decision accuracy and SNR calculated using the additional metrics as our measure of signal, and plan to include these additional results in the paper:\n\n| Measure of Signal         |                                             |   SNR $R^2$  |\n|:--------------------------|:--------------------------------------------|-------------:|\n| Rel. Dispersion           | $\\max_{i,j} \\|c_i - c_j\\|/\\bar{c}$            |   0.569      |\n| Rel. Std. Dev.            | $\\sigma/\\mu$                                |   0.566      |\n| Interquartile Range       | $Q_3 - Q_1$                                 |   0.464      |\n| Robust Range              | $P_{95} - P_5$                              |   0.444      |\n| Quartile Deviation        | $(Q_3 - Q_1)/2$                             |   0.453      |\n| Median Absolute Deviation | $\\text{median}(\\|c_i - \\text{median}(c)\\|)$   |   0.417      |\n| Halfspace Depth           | $\\min\\left( F_n(x),\\ 1 - F_n(x) \\right)$ where $F_n(x) = \\frac{1}{n} \\sum_{i=1}^n \\mathbb{I}[c_i \\leq x]$ |   0.036  |\n| Projection Depth          | $\\left( 1 + \\frac{\\|x - \\text{med}(c)\\|}{\\text{MAD}(c)} \\right)^{-1}$                                       |   0.033  |\n\nFor measures that are not sensitive to outliers such as quartile deviation $(Q_3 - Q_1)/2$ and median absolute deviation $\\text{median}(|c_i - \\text{median}(c)|)$, we found that using these as measures of signal resulted in a slightly lower, but similar, correlation with decision accuracy as relative dispersion ($R^2 = 0.4528$ and $R^2 = 0.4168$ respectively). \n\nOur data in Appendix A.2.1 and the table above show that different metrics lead to similar conclusions, so we argue our work is not particularly sensitive to the choice of metric. Finally, we would like to reiterate that our contribution is the framework itself to capture these complementary notions of signal and noise, and that our framework is general enough to support plug-and-play with different signal metrics depending on the use case.\n\n2. **Magnitude of Improvement**\n\nThe +2.6% improvement of decision accuracy on MMLU from 91.5% to 94.1% shows that SNR can further improve the sensitivity of a widely used benchmark with an already high decision accuracy, which is a notable 31% error reduction, $0.31=\\frac{(1-0.915)-(1-0.941)}{(1-0.915)}$. We believe reducing this likelihood of reporting a false ranking of models at a small scale is useful, and demonstrates the utility of SNR.\n\nAdditionally, we find that SNR is a powerful tool for detecting poorly performing task setups along with improving already high quality benchmarks. For example, we show the decision accuracy on the 30-task average improves from 77% to 83.7% (+6.7%) using the bits-per-byte task format. For individual tasks like Minerva MATH, we show up to +49% improvements in decision accuracy (51% to 90%). \n\n3. **Computational Cost**\n\nFirst, we remark that calculating SNR is a one-time cost, not something that needs to be recalculated with every new model developed. To clarify, we do not propose SNR as a way to improve model performance on a given benchmark. Rather, ML practitioners building new models can simply use the conclusions from previously-calculated SNR values. \n\nFor example, if a benchmark has high SNR for a set of 1B models, a practitioner developing a new 1B model could reasonably decide to use that benchmark and compare against other 1B models. On the other hand, if a benchmark has low SNR, the ML practitioner could choose not to evaluate on it. A researcher might also examine a benchmark that has low SNR to find some problem that they can fix (e.g., incorrect annotations or otherwise problematic data); once any problems are fixed and the benchmark has higher SNR, that fixed benchmark can be used in perpetuity without incurring any additional cost.\n\nCompared to other methods of measuring the quality of a benchmark, we believe SNR requires far less computation. For example, performing a DataDecide-style experiment at the 7B scale would require training multiple 7B models on different training datasets, which is too expensive at that scale to be a practical method for measuring evaluation quality. Instead, one can compute SNR only for the evaluation cost of the checkpoints used for signal and noise. In practice, our cost for computing SNR at the 7B scale (42 models) on the full 30-task evaluation suite required 789 NVIDIA H100 hours. Compared to the development costs for other 7B models such as OLMo 2, which reported 269K H100 hours in total [1], our SNR analysis on all benchmarks requires 0.29% of the total development cost. Given the enormous costs of model training, we believe the compute cost associated with improving evaluation procedure is well worth the improvements we show in our work.\n\nAdditionally, in Figure 6 (Appendix B) we show that other estimates of modeling noise, such as seed noise and data order noise, highly correlate with the step-to-step noise of only the final checkpoints. This one-time cost allows us to further reduce the cost as one can get an estimate of noise without training models across random seeds.\n\nWhile the evaluation cost is non-trivial, it allows estimating the quality of benchmarks at much larger scales without training a population of models from-scratch.\n\n[1] Holistically Evaluating the Environmental Impact of Creating Language Models (Morrison et al., ICLR 2025)\n\n**Questions:**\n\n> How sensitive are the results to the specific choice of relative dispersion for measuring signal? Can you test alternative signal metrics that might be less sensitive to outliers?\n\nPlease see our response to concern 1 above. We extensively evaluated 20 different metrics for our signal term, which we discuss in Appendix A, and include additional results on alternative signal metrics in the above table.\n\n> How well does the SNR framework generalize to other critical decisions in LLM development, such as architecture selection, training procedure optimization, or safety evaluation?\n\nWe extend SNR to a large set of models in Table 3 as part of Appendix C.3, with the results on external models across compute scales up to $10^{24}$ estimated FLOPs in Figure 11. In this setting, we use results across model families trained with similar compute budgets. This provides an estimate of SNR across different training procedures, although we believe explicitly extending this across architectures (e.g., MoEs or state-space models) is a good opportunity for future work.\n\nHowever, our work primarily uses SNR in the context of evaluating data decisions, as this family of decisions is particularly sensitive to downstream evaluation. This follows existing work that shows that held-out language modeling loss is a good predictor of downstream performance for architecture and training decisions, but that the mapping from validation loss to downstream performance varies between training datasets [1]. Therefore, we anticipate on these modeling and architectural decisions, there is less need to rely on downstream evaluation for decision-making compared to validation loss.\n\n[1] LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (Mayilvahanan et al., ICML 2025)\n\n> … why relative dispersion and checkpoint-level noise specifically should predict decision accuracy and scaling law reliability?\n\nTo clarify our contribution, our framework proposes that, in general, some measures of signal and noise are predictive of decision accuracy and scaling law error, rather than only the two particular measures we use to quantify signal and noise (the relative dispersion and checkpoint-level noise).\n\nIn Table 2, we show that many different measures of signal exhibit similar correlation with decision accuracy when used as the signal term in SNR. And in Figure 6, we find that different sources of noise (e.g., the seed noise, data order noise, and checkpoint-to-checkpoint noise) all have high correlation. So, many different measures could be used to capture signal and noise. \n\n> Can the authors provide a systematic analysis of when these interventions are cost-effective given their computational overhead and the magnitude of improvements achieved?\n\nPlease see our response to concern 2, on improvement, and concern 3, on computation cost. Additionally, we emphasize that calculating SNR is a one-time cost to evaluate the quality of a benchmark. For instance, when a researcher is developing a new benchmark, they likely want to evaluate a suite of models to showcase that their benchmark is useful. In this case, calculating SNR only requires the cost of the evaluation, and can indicate how well the benchmark can distinguish between model decisions. The researcher might find that small (e.g., 1B) models evaluated on their benchmark have low SNR, while large (e.g., 32B) models have high SNR, in which case they could describe their new benchmark as useful for evaluating models trained with large compute budgets. On the other hand, a researcher might find low SNR across model scales, which may indicate a need to investigate annotation errors, formatting or other problems in their benchmark."}, {"review_id": "uB2k5QU0MZ", "reviewer": "Reviewer_Nqe6", "summary": "The paper identifies two metrics, signal and noise, to evaluate the reliability and usefulness of benchmarks in decision making during LLM deployment, e.g., model architecture, training approach and training dataset. Signal measures the ability of the benchmark to distinguish different models, while noise measures the sensitivity to randomness during training. The paper discovers that signal to noise ratio (SNR) is highly correlated with decision accuracy. Finally, the paper discusses ways to improve SNR.", "questions": "1. How to choose $n$ in Equation (2)? A practical guidance on choosing $n$ greatly improves the significance of the work.\n2. SNR is dependent on the models. If a benchmark has high SNR on a set of models $\\mathcal M$, will it be reliable and useful for decision making on a new model $m$ that is not in $\\mathcal M$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength in Quality**: The paper conducts abundant experiments to support the claim. In Section 4, the experiment is conducted on 25 models and three tasks, so the results are convincing.\n\n**Strength in Significance**: The paper contributes to benchmark design by introducing a practical metric. SNR is easy to compute as it only involves small models. Moreover, it is highly correlated to decision accuracy.\n\n**Weakness in Clarity**: The notations are not clearly defined. For instance, in Equation (1), the range of $B(\\cdot)$ is not given (i.e., does it output a scalar or a vector or a model?), and $m_a$, $m_b$ are not defined (Though $m$ is defined as a model, what does the subscript mean here?) Also, in line 146, and 160, both use the subscript $i$, but the former refers to training step, while the latter refers to model. \n\n**Weakness in Significance**: SNR computation (Equation (2)) relies on a hyperparameter $n$, which is set differently in experiments (5 in Section 4.1, 30 in Section 4.2). In practice, it is hard to determine the best choice of $n$.", "was_revised": true, "final_justification": "The authors perfectly resolved the issue of the selection of $n$, and promised to resolve the notation problems in the final version. Therefore, I agree to raise the score to 5.", "rebuttal": "Thank you for your review and highlighting the “abundant experiments” and that “the results are convincing.” \n\nYour feedback brought up three points: practical guidance on selecting $n$ in the noise term of SNR, clarifying aspects of our notation, and the impact of the particular model set $\\mathcal{M}$ on SNR. We address these points, in that order, below:\n\n**Concerns:**\n\n1. **Selecting $n$ in the noise term**\n\nThank you for raising this question. Below, we provide guidance on the selection of $n$ and will include this discussion in the final draft of our work.\n\nIn the noise calculation (Eq. 2), increasing the number of intermediate checkpoints $n$ will lead to a less biased estimate of noise. Thus, we can calculate the minimum number of $n$ intermediate checkpoint samples such that the sample noise $s_n$ is a reasonable estimate of the population noise $\\sigma$:\n\nWe first assume the checkpoint to checkpoint scores are independent and normally distributed (which we observe when computing decision accuracy on intermediate checkpoints in Figure 7). Under this assumption, the ratio between the sample variance and the population variance follows a scaled chi squared distribution:\n\n$\\frac{(n-1)s_n^2}{\\sigma^2} \\sim \\chi^2_{n-1}$\n\nTherefore we would like to calculate the probability that the sample standard deviation $s_n$ is within one standard deviation of the population standard deviation $\\sigma$:\n\n$|s_n - \\sigma| < \\sigma$\n\nWe can rewrite this inequality:\n\n$\\left| \\frac{s_n}{\\sigma} - 1 \\right| < 1 \\Rightarrow 0 < \\frac{s_n}{\\sigma} < 2$\n\nAnd then, can substitute the chi-squared distribution to compute the likelihood w.r.t. $n$:\n\n$\\frac{s_n}{\\sigma} \\sim \\sqrt{\\frac{\\chi^2_{n-1}}{n-1}}$\n\n$P\\left( \\sqrt{\\frac{\\chi^2_{n-1}}{n-1}} < 2 \\right)$\n\n$P\\left( \\chi^2_{n-1} < 4(n-1) \\right)$\n\nWe can then solve the inequality for the smallest value of $n$ for a particular threshold $\\alpha$:\n\n$P\\left( \\chi^2_{n-1} < 4(n-1) \\right)>\\alpha$\n\nSolving this inequality numerically with $\\alpha=0.95$ for increasing values of $n$, we find that $n=9$ provides the smallest sample size such that the probability that the sample standard deviation (the observed noise) is within one standard deviation of the population standard deviation (the true noise) with 95% confidence. \n\nIn addition, we can specify a stricter bound by defining the sample standard deviation to be within $k\\cdot \\sigma$ of the population standard deviation:\n\n$|s_n - \\sigma| < k\\cdot \\sigma$\n\nWe then verify this empirically using our estimate for noise at the 7B scale. If we assume the 30 intermediate checkpoints provide a reasonable estimate of the population standard deviation, we then compute the sample standard deviation $s_n$ for $n < 30$. We re-compute $s_n$ 1000 times for different subsets to calculate the likelihood that the sampled standard deviation is within $k\\cdot \\sigma$ of the population standard deviation $\\sigma$. In the below table, we report this likelihood with tolerances $k\\in \\{0.2, 1.0\\}$ for subsets $n\\in \\{5, 10, 20\\}$ and highlight all results with a likelihood above $0.95$:\n\n| $k$ |       0.2  |        0.2  | 0.2         | 1.0        | 1.0         | 1.0         |\n|:-----------------|-----------:|------------:|:------------|:-----------|:------------|:------------|\n| $n$ (# checkpoints)    |       5    |       10    | 20          | 5          | 10          | 20          |\n| AGI Eval         |       0.42 |        0.61 | **0.95**    | **1.00**   | **1.00**    | **1.00**    |\n| ARC Challenge    |       0.44 |        0.7  | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| ARC Easy         |       0.38 |        0.65 | **0.97**    | **1.00**   | **1.00**    | **1.00**    |\n| AutoBencher      |       0.47 |        0.71 | **0.97**    | **1.00**   | **1.00**    | **1.00**    |\n| BBH              |       0.42 |        0.6  | 0.95        | **1.00**   | **1.00**    | **1.00**    |\n| BoolQ            |       0.16 |        0.45 | 0.88        | **1.00**   | **1.00**    | **1.00**    |\n| HumanEval        |       0.52 |        0.79 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| HumanEval+       |       0.47 |        0.76 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| CommonsenseQA    |       0.39 |        0.64 | **0.96**    | **1.00**   | **1.00**    | **1.00**    |\n| DROP             |       0.48 |        0.76 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| GSM8K            |       0.49 |        0.77 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| GSM+             |       0.5  |        0.79 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| GSM Symbolic     |       0.37 |        0.64 | **0.96**    | **1.00**   | **1.00**    | **1.00**    |\n| GSM Symbolic P1  |       0.47 |        0.69 | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| GSM Symbolic P2  |       0.32 |        0.57 | 0.94        | **1.00**   | **1.00**    | **1.00**    |\n| HellaSwag        |       0.39 |        0.65 | **0.97**    | **1.00**   | **1.00**    | **1.00**    |\n| Jeopardy         |       0.42 |        0.69 | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| MBPP             |       0.43 |        0.63 | **0.96**    | **1.00**   | **1.00**    | **1.00**    |\n| MBPP+            |       0.41 |        0.63 | **0.96**    | **1.00**   | **1.00**    | **1.00**    |\n| MedMCQA          |       0.5  |        0.79 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| Minerva MATH     |       0.38 |        0.53 | 0.93        | **1.00**   | **1.00**    | **1.00**    |\n| Minerva MATH 500 |       0.28 |        0.53 | 0.92        | **1.00**   | **1.00**    | **1.00**    |\n| MMLU             |       0    |        0    | 0.54        | 0.83       | **1.00**    | **1.00**    |\n| MMLU Pro         |       0.51 |        0.78 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| All Tasks        |       0    |        0    | 0.08        | 0.83       | **1.00**    | **1.00**    |\n| Code Tasks       |       0.49 |        0.78 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| Knowledge Tasks  |       0    |        0    | 0.15        | 0.83       | **1.00**    | **1.00**    |\n| Math Tasks       |       0.55 |        0.83 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| OLMES Core 9     |       0.31 |        0.49 | 0.92        | **1.00**   | **1.00**    | **1.00**    |\n| OLMES Gen        |       0.48 |        0.74 | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| OpenBookQA       |       0.42 |        0.73 | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| PIQA             |       0.43 |        0.69 | **0.98**    | **1.00**   | **1.00**    | **1.00**    |\n| SocialIQA        |       0.3  |        0.44 | 0.88        | **0.99**   | **1.00**    | **1.00**    |\n| SQuAD            |       0.48 |        0.72 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| TriviaQA         |       0.48 |        0.76 | **0.99**    | **1.00**   | **1.00**    | **1.00**    |\n| WinoGrande       |       0.42 |        0.67 | **0.97**    | **1.00**   | **1.00**    | **1.00**    |\n\nIn practice, we find that for a large bound ($\\pm 1$ std. dev.) can be satisfied for almost all benchmarks with $n=5$ intermediate checkpoints, but for smaller bounds, (20% of $\\pm 1$ std. dev.), using $n=20$ gives an adequate estimate for 34 of 39 benchmarks we considered in our work.\n\nFor our experiment on the 1B-5xC checkpoints, we estimate noise using the average noise of the last 5 checkpoints for all 25 models, so our estimate of noise considers $5\\cdot 25=125$ scores, rather than only 5 scores as the estimate of noise.\n\nWe will add these details in our final draft to provide guidance on selecting the minimum number of intermediate checkpoints to estimate the step-to-step noise.\n\n2. **Clarity of notation**\n\nThank you for your feedback on our notation, we will review the terms in Section 2 and 3 to ensure clarity. To answer the specific points: the benchmark score $B(\\cdot)$ is a scalar score; the subscripts for model $m$ denoted as $m_a$, $m_b$ refer to models trained on two datasets $a$ and $b$; we will use a different subscripts on L160 to denote each pair of models, which is separate from the subscript $i$ used on L146 to denote the intermediate checkpoint.\n\n**Questions:**\n\n> How to choose $n$ in Equation (2)?\n\nPlease see our response to concern 1 above. We plan to include this finding in the final draft of our work.\n\n> If a benchmark has high SNR on a set of models $\\mathcal{M}$, will it be reliable and useful for decision making on a new model $m$ that is not in $\\mathcal{M}$?\n\nThe application of SNR to new models outside of the set $\\mathcal{M}$ depends on the similarity of $m$ to the models studied. The broadest definition we use is the set of “external models” we study in Appendix C.3. In this set, we calculate SNR using models from different families, but with similar architectures. In this case, we expect the SNR findings to hold to new dense transformer-based models trained to similar compute budgets, but perhaps different data, slight architectural changes (for instance, MLA or GQA) or a different learning rate schedule, as these were varied in the set used to calculate signal.\n\nHowever, we believe extending this across architectures (e.g., MoEs or state-space models) is a good opportunity for future work."}]

Author rebuttal:

Meta-review: This paper introduces a framework for evaluating benchmarks through the lens of signal (a benchmark’s ability to separate stronger from weaker models) and noise (its sensitivity to random training variability). The authors thus propose the signal-to-noise ratio (SNR) as a predictor of benchmark reliability. With a wide range of benchmarks and open-weight models, the authors demonstrate that SNR correlates strongly with decision accuracy and scaling law prediction error, and propose practical interventions.

This paper is quite timely, as (fair) LLM evaluation is a pressing issue. Multiple reviewers appreciated its novelty as a “meta-evaluation” study and its practical utility for practitioners. Weaknesses noted include limited novelty in some experimental setups (which mirror prior work), sensitivity of the chosen “signal” metric to outliers, modest improvements in some settings, and some clarity issues in definitions and notation. During the rebuttal, the authors clarified their contributions relative to prior work, and provided both theoretical and empirical explanations, which successfully addressed reviewer concerns.

Overall this paper makes a technically solid and impactful contribution to the methodology of benchmark design and evaluation. Considering the reviewers' comments and the timely topic, I recommend accept with spotlight.
Decision: Accept (spotlight)