Quantifying Variance in Evaluation Benchmarks

## Abstract

Abstract Evaluation benchmarks are the cornerstone of measuring capabilities of large language models (LLMs), as well as driving progress in said capabilities.
Originally designed to make claims about capabilities (or lack thereof) in fully pretrained models, evaluation benchmarks are now also extensively used to decide between various training choices. Despite this widespread usage, we rarely quantify the variance in our evaluation benchmarks, which dictates whether differences in performance are meaningful.
Here, we define and measure a range of metrics geared towards measuring variance in evaluation benchmarks, including seed variance across initialisations, and monotonicity during training.
By studying a large number of models – both openly available and pretrained from scratch – we provide empirical estimates for a variety of variance metrics, with considerations and recommendations for practitioners.
We also evaluate the utility and tradeoffs of continuous versus discrete performance measures and explore options for better understanding and reducing this variance.
We find that simple changes, such as framing choice tasks (like MMLU) as completion tasks, can often reduce variance for smaller scale ( ∼ similar-to \sim ∼ 7B) models, while more involved methods inspired from human testing literature (such as item analysis and item response theory) struggle to meaningfully reduce variance.
Overall, our work provides insights into variance in evaluation benchmarks, suggests LM-specific techniques to reduce variance, and more generally encourages practitioners to carefully factor in variance when comparing models.

## 1 Introduction

Benchmark evaluation datasets are the cornerstone of establishing and defining progress with large language models (LLMs).
Virtually any new model release is accompanied by a range of scores on common evaluation benchmarks, illustrating how the model tallies up against previous releases (Mesnard et al., ; AI@Meta, ; Achiam et al., ; Reid et al., ).
As such, evaluation datasets play an important role in claiming progress and the title of state-of-the-art.
Consequently, choices in model development are often based on how they impact performance on benchmarks considered important by the field, giving benchmarks a prominent role in model iteration as well.
Yet, despite their importance, benchmark scores are often regarded as a one-dimensional number, and it is rare that they are given a more detailed consideration.
While it is well known that benchmarks scores can be heavily influenced by the choice of prompt (Sclar et al., ), the distributions of labels in the provided few-shots (Weber et al., ) or even the symbols that are used for the different options in a multiple choice setup (Zheng et al., ; Alzahrani et al., ), papers rarely report more than a single number per benchmark, or specifics on how each number was computed.
Furthermore, statistical significance values are scarcely reported on major release papers or leaderboards, or even in papers that study how scores vary across various dimensions.
These issues muddy the power of evaluation datasets, both during development and evaluation: if we cannot ‘trust’ our evaluation results or do not understand what improvements are statistically significant, we cannot make sound comparisons, thus making it more challenging to reliably use benchmarks during model development.

To address this, we present a deep dive into variance in benchmark scores, at much larger scale than any previous work.
Across all our experiments, we consider 13 different popular benchmarks and compute their performance over 280 different models, including fully trained public models as well as a set of 7B models and their intermediate checkpoints that we trained from scratch, differing only in their initialisation random seed.

With this, our contributions are three-fold:

- •
We provide a comprehensive reference guide for what magnitudes of variance are expected for what benchmarks across various circumstances.
- •
We make suggestions of how variance can be reduced for smaller scale models on choice tasks of important value (MMLU).
- •
We caution against the use of methods from human standardised testing (item analysis, item response theory) as a means of reducing variance, finding them to be ineffective.

Our work brings to light the often overlooked problem of variance in evaluation benchmarks, quantifies its effects, and provides a set of positive and negative results on how to mitigate it.

## 2 Models and Benchmarks

We run our analysis by comparing benchmark results across a large number of models trained across various setups.
In this section, we describe these models and list the benchmarks we investigate.

### 2.1 Models

In our analysis, we use over 280 models for our analysis, including intermediate checkpoints.
First, we train ten Llama-2-7B-architecture models from scratch on our own pre-training data mixture inspired by Touvron et al. () (See Appendix A).
These 10 runs are identical, except for the model initialisation seed.
The model hyper-parameters, the pre-training data mixture, and the data-loading mechanism is consistent across all these ten runs.
We train these models for 210 billion tokens and store 21 checkpoints for each model, leaving us with 10 sets of 21 model snapshots. We refer to these 210 checkpoints as the “seed models.”
In addition, we use 41 intermediate and fully-trained models based on the Llama-1 and Llama-2 architecture pre-trained on the same data mixture used for training the seed models.

Finally, we use 32 publicly available models from Huggingface (Wolf et al., ): Meta-Llama-3 {8, 70}B (AI@Meta, ), Gemma {2, 7}B (Mesnard et al., ), DBRX-Base (Databricks, ), Mistral 7B (Jiang et al., ), Mixtral 8x{7, 22}B (Jiang et al., ), Qwen-1.5 {0.5, 1.8, 4, 7, 14, 32, 72, 110}B (Bai et al., ), Pythia {1, 1.4, 2.8, 6.9, 12}B (Biderman et al., ), Falcon {7, 40}B (Almazrouei et al., ), DeepSeek {7, 67}B (Bi et al., ), DeepSeek-MoE 16B (Bi et al., ), DeepSeek V2 (DeepSeek-AI, ), StableLM {1.6, 3, 7}B (StabilityAI, ), and MPT {7, 30}B (MosaicML NLP Team, ).

The set of models used for the analysis are diverse across architectures, data mixtures, and sizes ranging from 0.5B to 236B total parameters. Details of all models are presented in Table 6.

### 2.2 Benchmarks

We do a comprehensive analysis using 13 large-scale well-established NLP benchmarks: AGIEval (Zhong et al., ), AI2 Reasoning Challenge (ARC-C) (Clark et al., ), BIG Bench (Hard) (Srivastava et al., ; Suzgun et al., ), COPA (Roemmele et al., ), GSM8k (Cobbe et al., ), Hellaswag (Zellers et al., ), HumanEval (Chen et al., ), MATH (Hendrycks et al., ), MMLU (Hendrycks et al., ), Natural Questions (Kwiatkowski et al., ), PIQA (Bisk et al., ), SIQA (Sap et al., ), and TriviaQA (Joshi et al., ).

These benchmarks are a mix of choice- and generation-based benchmarks, that span various capabilities ranging from general knowledge to coding.

## 3 How much variance do we observe?

We first investigate how much variance there is across different models and datasets.
We define a range of metrics for quantifying different kinds of variance.

First, using the 7B models we trained ourselves, we consider variance due to changes in seed, across otherwise identical setups.
This *seed variance* gives us a metric useful for performing data ablations – to conclude that pretraining dataset or hyperparameter set B is better than pretraining dataset or hyperparameter set A, we would want the performance increase to be larger than that due to random seed variance across different models trained in setup A.
To this end, we also compute a dataset’s *monotonicity*, quantifying how stably it develops during training.

To ground the seed variance numbers, we compare them with bootstrapped 95% confidence intervals on individual models, as well as observed variance across different setups.
In all our experiments, we consider both the (discrete) metric preferred for the benchmark and a more continuous representations for the same task.

### 3.1 Analysis Methodology

For our initial variance analysis, we use both benchmark-level scores (to compute variance and monotonicity) and sample level scores (to estimate 95% confidence intervals).
Here, we provide a brief description of the metrics we compute.

#### Seed Mean ( μ ⁢ ( 𝒮 , 𝕄 ) 𝜇 𝒮 𝕄 \mu(\mathcal{S},\mathbb{M}) italic_μ ( caligraphic_S , blackboard_M ) )

We compute the performance using metric $\mathcal{S}$ of the final checkpoint (at 210B tokens) of each of the 10 “fully trained” models in ${\mathbb{M}}$ (one for each seed).

#### Seed variance ( 𝔼 ⁢ ( 𝒮 , 𝕄 ) 𝔼 𝒮 𝕄 \mathbb{E}(\mathcal{S},\mathbb{M}) blackboard_E ( caligraphic_S , blackboard_M ) )

Given a benchmark, a preferred metric $\mathcal{S}$, and a set of models $\mathbb{M}=\{\mathrm{M}_{1},\mathrm{M}_{2},\dots\mathrm{M}_{n}\}$, we define the benchmark seed variance $\mathbb{E}(\mathcal{S},\mathbb{M})$ as the standard deviation of the metric $\mathcal{S}$ scores $\{\mathbb{S}_{\mathbb{M}}=\mathcal{S}_{M_{1}},\mathcal{S}_{\mathrm{M}_{2}}
\dots\mathcal{S}_{\mathrm{M}_{n}}\}$ for each of the models in $\mathbb{M}$.

To estimate the variance expected due only to random seed changes, we take the average of this metric over all checkpoint timesteps $\mathbb{E}(\mathcal{S},\mathbb{M})=\frac{1}{21}\sum_{time=\{10..210B\}}\mathbb
{E}(\mathcal{S},\mathbb{M}^{(time)})$, where for example $\mathbb{E}(\mathcal{S},\mathbb{M}^{(time)})$ corresponds to the standard deviation of performance of the 10 model checkpoints (across seeds) after 200B tokens of training. For each benchmark, we consider both a discrete and a continuous metric.(^1^11With the exception of the datasets Big Bench (Hard), MATH, Natural Questions, and TriviaQA.)
The benchmark details are provided in Table 5 of Appendix A.

#### Confidence intervals ( 95 % percent 95 95\% 95 % CI )

We use the bootstrapped library(^2^22https://github.com/facebookarchive/bootstrapped) to compute 95% bootstrapped confidence interval (CI) values for each of the benchmarks on all 210 checkpoints from our 10 random seeded pretraining runs.
Since bootstrapping is expensive, we also compute analytic interval (for discrete metrics) using the formula:

$$ $CI_{\texttt{analytic}}(\mathrm{M})=1.96*\sqrt{\frac{\mathcal{S}_{\mathrm{M}} \times(1-\mathcal{S}_{\mathrm{M}})}{N}},$ $$

where $\mathcal{S}_{\mathrm{M}}$ is the obtained preferred metric score for model $\mathrm{M}$ on a given benchmark and $N$ is the number of test instances present in that benchmark.
Empirically, we observe that, for the distributions we consider, bootstrapped and Analytic CIs converge when the number of bootstrap samples is large.

#### Monotonicity values ( mon disc subscript mon disc \text{mon}_{\text{disc}} mon start_POSTSUBSCRIPT disc end_POSTSUBSCRIPT / mon cont subscript mon cont \text{mon}_{\text{cont}} mon start_POSTSUBSCRIPT cont end_POSTSUBSCRIPT )

We compute the extent to which the scores for a benchmark develop monotonically during training.
We define monotonicity for seed $i$ as the Kendall Rank correlation between the list of scores $[\mathcal{S}_{\mathrm{M}^{10B}_{i}},\mathcal{S}_{\mathrm{M}^{20B}_{i}},\ldots,
\mathcal{S}_{\mathrm{M}^{210B}_{i}}]$ and a monotonically increasing or decreasing array of the same length, for discrete and continuous metrics, respectively.

### 3.2 Results

In this section, we present our comprehensive analysis for two scenarios.

#### Seed variance

In Table 1, we report the observed variance across our 7B seed models in which the training setup is same across all init seeds, including a deterministic data ordering.
We contextualise these numbers with the per-model 95% confidence interval, reported in the form of an average of 210 (one for each model) confidence interval sizes. The latter is easily computable from a single training run, whereas the former requires multiple (expensive) training runs with different seeds.

For some benchmarks (e.g. AGIEval, MMLU), scores are around chance accuracy ($\sim 25\%$) even after training for $210\text{B}$ tokens. Benchmarks with few test examples (like COPA and HumanEval) exhibit high variance (both seed variance and 95% CIs). Generally, the 7B seed variance is well below the 95% CI for the same benchmark, though the ratio of the two is quite variable. Having access to the former value, which is smaller but closer to what would be needed to, for instance, compare two data mixes, may allow practitioners to make more fine-grained decisions during model development.

Motivated by prior work which suggests a move to continuous metrics (Srivastava et al., ; Schaeffer et al., ; Du et al., ; Schaeffer et al., ), we show a comparison of discrete and continuous metrics along with their signal to noise ratios (SNR) in Table 2.
To maintain consistency, we used probability mass of the predicted answer for all choice-based benchmarks and NLL of the correct answer for generation-based benchmarks; more details are provided in Appendix A.
We observe that the SNR is considerably higher for continuous metrics for all benchmarks, suggesting that they may be better when comparing models in the sense that they are less confounded by noise.
These results may thus help in building better scaling laws for downstream evaluation tasks (Achiam et al., ), along with accurate comparisons between two models that have performances lying within the confidence interval for the discrete metric.

**Table 1: Variance values on 7B seed models. Benchmarks are listed in alphabetical order. We report means - $\mu(\mathcal{S},\mathbb{M})$, standard deviations - $\mathbb{E}(\mathcal{S},\mathbb{M})$, confidence intervals - $95\%$ CI, and monotonicities - $\text{mon}_{\text{disc}}$, $\text{mon}_{\text{cont}}$. We also report size and chance level performance for reference—note all generative tasks have a chance level performance of 0. $\mathbb{E}(\mathcal{S},\mathbb{M})$ is generally lower than $95\%$ CI. We also observe that $\text{mon}_{\text{cont}}>\text{mon}_{\text{disc}}$ for all benchmarks.**
| Benchmark | Size | Chance | $\mu(\mathcal{S},\mathbb{M})$ | $\mathbb{E}(\mathcal{S},\mathbb{M})$ | $95\%$ CI | $\text{mon}_{\text{disc}}$ | $\text{mon}_{\text{cont}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AGIEval | 2546 | 20 | 23.44 | 0.77 | 1.63 | 0.37 | 0.29 |
| ARC-C(^3^33We exclude 7 problems from ARC-C as 4 of them have only 3 answer choices, and 3 of them have 5 answer choices.) | 1165 | 25 | 39.71 | 0.80 | 2.74 | 0.88 | 0.91 |
| Big Bench (Hard) | 6511 | 0 | 29.10 | 0.87 | 1.07 | 0.77 | - |
| COPA | 100 | 50 | 78.80 | 2.15 | 8.30 | 0.56 | 0.90 |
| GSM8k | 1319 | 0 | 4.10 | 0.41 | 0.87 | 0.74 | 0.30 |
| Hellaswag | 10042 | 25 | 70.08 | 0.21 | 0.93 | 0.99 | 0.99 |
| HumanEval | 164 | 0 | 11.89 | 1.11 | 3.98 | 0.79 | 0.98 |
| MATH | 5000 | 0 | 1.52 | 0.23 | 0.28 | 0.52 | - |
| MMLU | 14042 | 25 | 25.86 | 0.57 | 0.72 | 0.09 | 0.15 |
| MMLU-Cloze | 14042 | 25 | 37.47 | 0.22 | 0.79 | 0.95 | 0.96 |
| Natural Questions | 3610 | 0 | 16.43 | 0.60 | 1.04 | 0.91 | - |
| PIQA | 1838 | 50 | 76.93 | 0.41 | 1.99 | 0.87 | 0.93 |
| SIQA | 1954 | 33 | 46.69 | 0.55 | 2.21 | 0.66 | 0.81 |
| TriviaQA | 11313 | 0 | 42.69 | 0.45 | 0.83 | 0.99 | - |

**Table 2: 7B seed models. Comparison between discrete (Disc) and continuous (Cont) metrics along with the signal to noise ratio (SNR). The means - $\mu(\mathcal{S}=\text{Disc},\mathbb{M})$, $\mu(\mathcal{S}=\text{Cont},\mathbb{M})$ and standard deviations (Disc Std, Cont Std) reported here (and used to calculate SNR) are computed across the final checkpoints across the 10 seeds.**
| Benchmark | $\mu(\mathcal{S}=\text{Disc},\mathbb{M})$ | Disc Std | Disc SNR | $\mu(\mathcal{S}=\text{Cont},\mathbb{M})$ | Cont Std | Cont SNR |
| --- | --- | --- | --- | --- | --- | --- |
| AGIEval | 23.44 | 0.93 | 25.20 | 0.2267 | 0.0009 | 254.93 |
| ARC-C | 39.71 | 0.87 | 45.89 | 0.2684 | 0.0007 | 381.64 |
| COPA | 78.80 | 2.04 | 38.63 | 0.5376 | 0.0008 | 662.41 |
| GSM8k | 4.10 | 0.52 | 7.88 | 0.9948 | 0.0653 | 15.24 |
| Hellaswag | 70.08 | 0.12 | 608.23 | 0.2833 | 0.0001 | 1921.15 |
| HumanEval | 11.89 | 1.75 | 6.79 | 0.2186 | 0.0018 | 124.08 |
| MMLU | 25.86 | 0.49 | 52.45 | 0.2511 | 0.0007 | 347.57 |
| MMLU-Cloze | 37.47 | 0.12 | 302.73 | 0.2698 | 0.0004 | 678.42 |
| PIQA | 76.93 | 0.39 | 198.98 | 0.5168 | 0.0003 | 1641.14 |
| SIQA | 46.69 | 0.51 | 91.87 | 0.3656 | 0.0009 | 387.11 |

#### Monotonicity

In Table 1, we list the monotonicity values for each of the continuous and discrete metrics listed in Table 5.
Higher monotonicity values are indicative of evaluations that more stably represent model improvement.
In almost all cases, the mononicity is better for the continuous metrics than for the discrete metrics, mirroring our findings with SNR above.
However, for some benchmarks, such as HellaSwag and TriviaQA, the difference is minimal, likely since these benchmarks saturate earlier in training. Likewise, for benchmarks where performance remains at chance level we observe very low monotonicities.

In Figure 1, we visualise the development of discrete and continuous metrics and their seed variance during training, for ARC-C, GSM8k, and HumanEval.
Generally (with the exception of GSM8k), continuous metrics have better predictive scaling compared to the discrete metrics because they have higher monotonicity and SNR.
Interestingly, we see that the variance remains relatively constant as performance increases, suggesting that the estimates may extrapolate well to models trained for longer.
Overall, these results suggests that monitoring continuous metrics could be more fruitful during model development than tracking discrete metrics.

Figure: Figure 1: Development of model performance over time. Boxplots for both discrete and continous metrics depicting the model improvement over time for ARC-C, GSM8k, and HumanEval. Top row depicts discrete metrics for each of the benchmarks, and the bottom row is composed of the continuous metrics.
Refer to caption: extracted/5666202/figures/benchmark_performance_seed_comparison.png

### 3.3 The curious case of MMLU

Motivated by prior work considering the inconsistency of multiple choice benchmarks (Wang et al., ; Alzahrani et al., ), we examined two formulations of MMLU: (Standard) MMLU and MMLU-Cloze.

Standard MMLU refers to the prompting format where the choices along with the choice texts are present for the few-shot examples as well as the question in the prompt text.
To evaluate the sample, we append the choice letters (“A”, “B”, “C”, or “D”) at the end of the prompt text, and pick the choice that has the lowest negative log-likelihood (NLL).
For MMLU-Cloze, just the correct choice’s text is present for the few-shots, and we pick the choice that gives the lowest NLL after appending the choice texts at the end of the prompt.
The complete prompts used for the two cases are detailed in Appendix B.

In Figure 2, we plot performance over training and see that standard MMLU is at chance performance even after training on $210\text{B}$ tokens.
The cloze formulation performs better, and importantly has lower seed variance and much higher monotonicity (0.95 instead of 0.09, see Table 1).
This result seems surprising, given that the cloze format is not standard.
Further investigation yields that fully-trained large models tend to have better performance on standard MMLU compared to MMLU-cloze (e.g. 78.7% on standard MMLU vs. 60.6% for MMLU-Cloze for LLaMa 3 70B).
Despite this difference in absolute performance, we find the performance on standard and cloze formats is highly correlated for fully trained large models (Pearson correlation of 0.92 on the 70 models listed in § 2.1).

To understand this dichotomy better, we train a Llama-2-$13$B-like model from scratch on our pre-training mix.
We observe a sudden jump in performance at around $800\text{B}$ tokens (for both discrete and continuous metrics), after which standard MMLU performs better than MMLU-cloze (see Figure 2).

Given these results, we encourage researchers to use cloze formulations when doing pretraining ablations, as they are less confounded by noise during early stages of training, but still seem predictive of final performance on the standard MMLU format.

Figure: (a)
Refer to caption: extracted/5666202/figures/mmlu_seed_comparison_small_fig.png

## 4 Understanding variance through the lens of item analysis

In the previous section, we computed the empirically occurring variances for commonly used evaluation benchmarks, considering benchmark-level scores, and we showed how looking at continuous metrics or cloze formulations of tasks can boost SNR.

As another avenue of possibly reducing variance, and to better understand it, we take inspiration from item analysis, a common method used to assess the usefulness of individual test questions on standardised tests administered to humans (Livingston, ; University of Washington, ).
Item analysis focuses on metrics of individual samples (e.g. difficulty) to understand the types of questions on tests in terms of how individuals (in our case, models) perform on them.

### 4.1 Method

In applying item analysis to benchmarks, we consider two metrics.
Item difficulty refers to the average score on an item across models;
Item discrimination refers to the correlation between models’ performances on a single data point and models’ overall performances. Intuitively, items with either high or low difficulty will have low discrimination (as all models will be wrong or right, respectively).

As we wish to make recommendations about evaluation datasets that extend to future models, we split our 70 models into train and test sets. We consider two splits: “random” and “difficulty”.
As the name suggests, in the random split, we split models randomly; In the difficulty split, we hold out the best performing 14 models.
The full lists of models in each split can be found in § D.1.
We then calculate item analysis metrics on individual data points for the train and test sets. As is often done with human testing, we also consider the use of removing data points with low item discrimination, and observe the effects this has on evaluation metrics such as mean, standard error of the mean (std. err.),(^4^44Note that the confidence intervals of § 3.1 are 1.96 times the standard error.) and monotonicity.

### 4.2 Results

Figure: Figure 3: Item analysis results on GSM8k and ARC-C. Results on additional benchmarks provided in § D.2. First column shows a scatter plot of item difficulty (x-axis) vs item discrimination (y-axis). Second column shows a scatter plot of item discrimination calculated over models from the train or test set of the difficulty split. Third column is the same as the second, except on the random split. As expected (since train and test splits come from the same distribution), discrimination on train models for this split is positively correlated to discrimination on test models. Fourth, fifth, and sixth columns show the effects of iteratively removing up to 20% of items (based on discrimination) on the mean (fourth column), standard error (fifth column) of model performance on the test set from the difficulty split by looking at the delta. Error bars indicate 95% confidence intervals in the delta. Monotonicity (sixth column) is calculated over the 10 runs from § 2.1. Orange curves show effects from randomly removing points, as a baseline.
Refer to caption: x1.png

In Figure 3, we show results for two illustrative benchmarks: ARC-C and GSM8k.
Full results across other benchmarks can be found in § D.2.
Overall, we find that item discrimination scores may not provide much useful signal for the field of language model evaluations (unlike their widespread usage in human standardised testing).
This is especially true given that state-of-the-art models perform better and better, and we would like tests to stay informative when models improve.
To illustrate this, we show how high discrimination on train (weaker) models often does not correspond to high discrimination on test (stronger) models (Figure 3, second column).
Striping around $x=0$ corresponds to items that train set models always get wrong (yielding 0 discrimination) but are informative on test set models.
Similarly, striping around $y=0$ corresponds to items that test models always get right (yielding 0 discrimination) but are informative on the train set.
If we instead consider item discriminations on a random split of models (Figure 3, third column), we see stronger correlations, indicating that the low correlation is in fact due to the difference in item discrimination on weaker and stronger models.

In § D.3, we qualitatively inspect examples with negative item discrimination (which are thus anti-correlated with overall model performance), but are not able to discern any clear patterns for most benchmarks (a notable exception being Hellaswag, see Figure 8).
While these negative results suggest item discriminations may not be the most informative means of understanding (or reducing) variance on stronger models, we consider further application to explore the causal effect.

Specifically, we consider pruning data points with low item discrimination, with the hopes that this will reduce variance or improve monotonicity.
More precisely, we prune data points with low item discrimination on the train set of models from the difficulty split and we visualise metrics calculated using the pruned subset on the test set of models from the difficulty split.
Results are presented in the three rightmost columns of Figure 3.
Overall, while we find modest improvements in both standard error (a decrease) and monotonicity (an increase), the drift in the estimated accuracy is mildly concerning.
It may be acceptable for the purpose of comparing models, but may also provide an overestimate of capabilities if considering the absolute score.
One hypothesis for this discrepancy with human testing could be that item discrimination for human tests typically does not consider out-of-distribution splits – it takes into account the entire spectrum of scores.
However, even beyond the difficulty split, we similarly find little-to-no benefits on the random split (see Figure 7).
As a result, we overall would not suggest the use of item analysis-based methods for understanding variance in language model evaluations, though the underlying cause for this mismatch remains an open question for future work.

## 5 The false promise of item response theory for LLMs

In a similar category to item analysis, *item response theory* (Cai et al., ; van der Linden, ; Brzezińska, ; Lord and Novick, ) describes a set of statistical models used to analyse human abilities on standardised test data.
In the recent past, the method has become popular as a means of understanding model performance on a set of evaluation samples (Lalor et al., ; Vania et al., ; Rodriguez et al., ).
Most recently, Polo et al. () used IRT to cluster evaluation points with the aims of reducing eval benchmark size (and thus, cost of running).

Following our mixed findings applying item analysis, we apply the IRT method from Polo et al. () to our models and the overlapping set of evaluation benchmarks.
For a brief summary of the IRT method, we refer to § E.1.
Specifically, we go beyond the comparisons drawn in prior work and consider how our defined variance metrics (§ 3) change under this model.
We believe the application to evaluating intermediate checkpoints during pretraining is especially relevant, as that’s the application where smaller evaluation datasets could have the most efficiency gains (as opposed to one-time evaluations of larger models).

**Table 3: Variance values for Tiny Benchmark (across seeds). Full represents the full benchmark, and IRT/IRT++ use the 100 examples proposed in Polo et al. (). $\mathbb{E}(\mathcal{S},\mathbb{M})$ is the seed variance defined in § 3.1, which is represented as $\mathbb{E}$ in this table.**
| Benchmark | Full $\mu$ | IRT $\mu$ | IRT++ $\mu$ | Full $\mathbb{E}$ | IRT $\mathbb{E}$ | IRT++ $\mathbb{E}$ |
| --- | --- | --- | --- | --- | --- | --- |
| ARC-C | 39.71 | 46.21 | 42.32 | 0.80 | 1.80 | 1.86 |
| GSM8k | 4.10 | 3.21 | 4.62 | 0.41 | 1.16 | 1.49 |
| Hellaswag | 70.08 | 71.80 | 68.81 | 0.21 | 2.06 | 2.42 |

**Table 4: Monotonicity values for Tiny Benchmark. We list the monotonicity values for both discrete ($\text{mon}_{\text{disc}}$) and continuous ($\text{mon}_{\text{cont}}$) metrics for the 7B seed models from § 3.2. Full represents the full benchmark, and IRT/IRT++ use the 100 examples proposed in Polo et al. ().**
| Benchmark | $\text{mon}_{\text{disc}}$ (Full/IRT/IRT++) | $\text{mon}_{\text{cont}}$ (Full/IRT/IRT++) |
| --- | --- | --- |
| ARC-C | 0.88 / 0.64 / 0.63 | 0.91 / 0.78 / 0.82 |
| GSM8k | 0.74 / 0.32 / 0.30 | 0.30 / 0.24 / 0.24 |
| Hellaswag | 0.99 / 0.84 / 0.80 | 0.99 / 0.93 / 0.94 |

In Tables 3 and 4, we report various metrics on the discrete performance measure for GSM8k, Hellaswag, and ARC-C.
We find that simply using the performance on the 100 datapoints selected by Polo et al. () for each benchmark can lead to quite large deviations in the mean (an overestimation by 7% for ARC-C).
The full IRT++ method obtains less deviation, replicating prior findings (Polo et al., ).
However, both methods suffer from greatly increased seed variance (final two columns, Table 3), indicating that the tiny-benchmarks method may have limited use during pretraining ablations as it makes model comparisons more likely to be confounded by randomness from the initialisation and data ordering seed.
This increased variance is also reflected in the monotonicity metrics – we see a decrease in monotonicity in Table 4, indicating that performance oscillates more during training (see Figure 9).

Figure: Figure 4: Tiny Benchmarks Means and Standard Errors of the mean (proportional to 95% CI).
Refer to caption: extracted/5666202/figures/updated_meta_hf_tiny_all_sorted.png

Beyond the smaller scale models, we also considered the use of tiny-benchmarks for evaluating larger models, like the ones used for item analysis in Section 4.
In Figure 4, we find that IRT-based methods generalise relatively well when it comes to the average performance metric (with the IRT++ estimator performing better), but have much larger standard error of the mean.
This increased error cautions against the use of IRT-based subsets for model evaluations that will be used to compare different models.
To quantify how this increased standard error of the mean may affect model rankings, we also compute the Kendall rank correlation on our 70 models using the performance estimate obtained from using the full dataset, as well as the IRT and IRT++ methods.
In Table 7, we find that the correlation can drop as low as 0.76, corresponding to 12% of model pairwise comparisons giving the opposite result when using the IRT or IRT++ method (versus the full dataset mean estimate).
Furthermore, we find that the number of flips is relatively higher on models that perform better, suggesting that IRT-based methods may not scale well (similar to item analysis).
These findings reinforce the promise of IRT-based methods for a point estimate of the mean (relatively low error, Figure 4), but caution against the use of IRT-based methods when comparing models due to the increased variance of the estimate.

## 6 Related work

While a significant body of work exists proposing natural language processing (NLP) benchmarks to evaluate the capabilities of models, there is comparatively less work studying the benchmarks themselves.
Before the era of chat large language models, Marie et al. () conducted a large scale meta-analysis of 769 research papers published from 2010 to 2020 and identified troubling trends,
including one that partially motivates our work: models are frequently declared superior to competitors based on small differences in performance scores, without proper hypothesis testing that takes into account natural fluctuations in benchmark scores. Spiritually similar claims were made by Dehghani et al. () in their provocatively titled paper “The Benchmark Lottery”.
Kocmi et al. () further leveraged large-scale human experiments to evaluate benchmarks with automated metrics and concluded that commonly used metrics such as BLEU score had led to poor deployment decisions.
Their conclusion was echoed by a meta-analysis of 3500 NLP benchmark scores published on Papers with Code (Blagec et al., ).

More recently, with accelerating progress in NLP, researchers have begun to study benchmarks in earnest to understand their properties and limitations (Gehrmann et al., ).
Von Werra et al. () proposed a framework to evaluate benchmarks themselves and provided a mechanism for researchers to share their benchmarking analyses.
Certain papers have studied specific aspects of benchmarks, focusing on the sensitivity of language models to various factors. Sclar et al. () tested how sensitive language models are to differently formatted prompts, while Wang et al. () and Alzahrani et al. () find that models are inconsistent across changes in the format of Multiple-Choice Question Answering (MCQA) benchmarks. Our work builds on these works by focusing on the inherent variance in benchmarks (e.g. due to model seed) that pracitioners should consider when making decisions, and suggesting minor modifications (e.g. in how a task is scored or formulated) that can reduce this variance.

With the aims of improving efficiency in model development cycles, recent work proposes reducing the size of evaluation benchmarks by picking representative samples (Vivek et al., ; Polo et al., ). Polo et al. () show that methods from human standardised testing (specifically, item response theory; Lord and Novick, ) can be combined with clustering to subselect evaluation benchmarks without incurring too much deviation from the mean.
However, they do not consider the increased variance from their method nor how small deviations in means can compound when comparing multiple models.
We go beyond their work by considering the use of additional methods from human standardised testing literature (item analysis; Livingston, ), as well as showing that such methods generally do not meaningfully reduce variance.

Perhaps most similar to ours is the work of Xiang et al. (), who study different sources of variance in NLP benchmarks and offers cautionary advice about when one should (not) be confident in benchmark scores.
Their approach is limited to the machine translation setting; here we quantify and study variance in 13 different NLP benchmarks (covering general knowledge, reasoning, coding, and math) across 280 models, including many frontier LLMs.

## 7 Conclusion

As language models become more and more prevalent, it has become increasingly important to get a sense of their capabilities.
One of the primary ways to assess these capabilities is through the use of evaluation benchmarks, where a model is scored on a series of examples.
These scores are often directly compared, without consideration of the variance.
This obscures the interpretation of evaluation results, in assessing final models as well as making decisions during model development.
In this work, we aimed to quantify evaluation benchmark variance across a range of settings (from pretraining intermediate checkpoints, to the largest frontier LLMs) using a diverse set of metrics (seed variance, confidence intervals, and monotonicity).
Beyond quantifying variance, we also experimented with various techniques used in human standardised testing (item analysis; University of Washington (), item response theory; Cai et al. ()), but generally found these methods to be ineffective on the models and benchmarks we considered, in terms of reducing variance.
Future work could explore such avenues further, and it is possible that as models reach closer and closer to human-level performance these methods will provide more useful insights.
On the other hand, in line with recent work advocating for a teleological approach to measuring capabilities (McCoy et al., ), we demonstrated LLM-specific techniques (e.g. the use of continuous metrics or cloze-formatted tasks) can improve the signal-to-noise ratio in our evals.
Such techniques are not available when assessing humans, but provide a unique opporutnity for LLM evaluations, especially when performing pretraining ablations.
We hope our work spurs future work in this direction of reducing variance, in addition to serving as an empirical guide for model practitioners to use when comparing models and assessing performance.

## Appendix A Models and Benchmarks Details

For pre-training the 7B Llama-2 like checkpoints, we use a pre-training mix of publicly available data.
We apply filtering to remove documents containing a high amount of personal information.
We use a learning rate of $3.0\times 10^{4}$, sequence length of 4096, and a batch size of $4.1M$ tokens to train the 7B models for $50000$ steps. We use 256 80GiB A100 GPUs for a single pre-training run for 50k steps on our internal cluster. We do 10 such runs with different seeds. Each step takes 4.3 seconds.

For running the evaluations, we use 8 GPUs for each evaluation job comprising multiple evaluation datasets in a single job. A single evaluation job takes on average takes 3.5 hours for 13 benchmarks.

In Table 5, we provide the discrete metric (preferred), the continuous metric, and the number of samples for each of the benchmarks we consider.
We can choose any continuous metric like character NLL, raw NLL, probability mass, log of probabilities, etc. for the benchmarks, but to maintain consistency, we choose probability mass of the predicted answer for choice-based tasks and negative log likelihood (NLL) of the target answer for generation-based benchmarks.
Choice-based benchmarks are evaluated by appending the possible option choice letters or choice texts and then choosing the option with the lowest NLL.
Generation-based benchmarks involve free-form generation, where the answer is extracted from the model’s response using various post-processing techniques.

**Table 5: Benchmark Details Details of all benchmarks used in the paper alphabetically. Exact Match (EM) is computed for 1 generation (maj@1). Prob Mass is the probability mass of the predicted answer and Target NLL represents the NLL of the target answer. CoT represents chain of thought prompting.**
| Benchmark | License | # samples | # few-shot | Disc Metric | Cont Metric |
| --- | --- | --- | --- | --- | --- |
| AGIEval<br>(Zhong et al., ) | MIT | 2546 | 3-5 | Acc | Prob Mass |
| ARC-C<br>(Clark et al., ) | Apache 2.0 | 1165 | 0 | Acc | Prob Mass |
| Big Bench Hard<br>(Srivastava et al., ) | Apache 2.0 | 6511 | 3 (CoT) | EM | - |
| COPA<br>(Roemmele et al., ) | BSD 2-Clause | 100 | 0 | Acc | Prob Mass |
| GSM8k<br>(Cobbe et al., ) | MIT | 1319 | 8 (CoT) | EM | Target NLL |
| Hellaswag<br>Zellers et al. () | MIT | 10042 | 0 | Acc | Prob Mass |
| HumanEval<br>(Chen et al., ) | MIT | 164 | 0 | Pass@1 | Target NLL |
| MATH<br>(Hendrycks et al., ) | MIT | 5000 | 4 (CoT) | EM | - |
| MMLU<br>(Hendrycks et al., ) | MIT | 14042 | 5 | Acc | Prob Mass |
| Natural Questions<br>(Kwiatkowski et al., ) | MIT | 3610 | 5 | EM | - |
| PIQA<br>(Bisk et al., ) | Academic Free | 1838 | 0 | Acc | Prob Mass |
| SIQA<br>Sap et al. () | - | 1954 | 0 | Acc | Prob Mass |
| TriviaQA<br>Joshi et al. () | Apache 2.0 | 11313 | 5 | EM | - |

**Table 6: Model Details Details of all models in the paper categorized by model family along with the number of parameters.**
| Model Family | Models | Model Sizes ($\#$ params) |
| --- | --- | --- |
| Meta-Llama<br>(AI@Meta, ; Touvron et al., , ) | Llama-1, Llama-2,<br>Llama-3 | 7-70B |
| Google<br>(Mesnard et al., ) | Gemma | 2-7B |
| Databricks<br>(Databricks, ) | DBRX-Base | 132B |
| Mistral<br>(Jiang et al., , ) | Mistral, Mixtral | 7-141B |
| Qwen<br>(Bai et al., ) | Qwen-1.5 | 0.5-110B |
| EleutherAI<br>(Biderman et al., ) | Pythia | 1-12B |
| TII-UAE<br>(Almazrouei et al., ) | Falcon | 7-40B |
| DeepSeek<br>(Bi et al., ; DeepSeek-AI, ) | DeepSeek, DeepSeek-MoE,<br>DeepSeek-V2 | 7-236B |
| StabilityAI<br>(StabilityAI, ) | StableLM | 1.6-7B |
| MosaicML<br>(MosaicML NLP Team, ) | MPT | 7-30B |

## Appendix B MMLU prompt formats

We use the following prompt variations for the standard and cloze versions of MMLU.
We list down the preamble and the shot formatting for both cases.
The final question is formatted like the few shot examples without the gold choice letter or text.

### B.1 MMLU

### B.2 MMLU-cloze

## Appendix C Variance Analysis Additional Results

In this section, we present additional results on model performance development for the remaining benchmarks - COPA, Hellaswag, PIQA, and SIQA (see Figure 5).
This supplements the results presented in Figure 1 and Figure 2.
We observe similar trends except for SIQA.
The error bars for both discrete and continuous metrics are similar, however, the continuous metric plot has less number of outliers.

Figure: Figure 5: Development of model performance over time. Boxplots for both discrete and continous metrics depicting the model improvement over time for COPA, Hellaswag, PIQA, and SIQA. Top row depicts discrete metrics for each of the benchmarks, and the bottom row is composed of the continuous metrics.
Refer to caption: extracted/5666202/figures/benchmark_performance_seed_comparison_appendix.png

## Appendix D Item analysis additional results

### D.1 Splits

We used 70 base models for the item analysis results. We provide the splits used below.

Difficulty split (train): LLaMa 3 8B, Mistral 7B, Qwen {0.5, 1.8, 4}B, LLaMa 2 7B, LLaMa 2 13B, LLaMa 2 70B, DeepSeek 7B, DeepSeek MoE 16B, Falcon 7B, Falcon 40B, Gemma 2B, Gemma 7B, LLaMa 1 {7, 13, 33, 65} B, MPT 30B, Pythia {1, 1.4, 2.8, 6.9, 12}B, StableLM {3, 7}B. In addition to these open source models, we use 30 internal checkpoints from LLaMa-architecture models we pre-trained on our interal data mix.

Difficulty split (test): LLaMa 3 70B, Mixtral 8x{7,22}B, Qwen 1.5 {7, 13, 32, 72, 110}B, DBRX, DeepSeek 67B, and 4 internal held out models.

Random split (train): LLaMa 3 {8, 70}B, Mistral 7B, Mixtral 8x{7,22}B, Qwen 1.5 {0.5, 1.8, 4, 7, 13, 32, 72}B, LLaMa 2 7B, LLaMa 2 13B, LLaMa 2 70B, DBRX, DeepSeek MoE 16B, DeepSeek 67B, Falcon 40B, Gemma 2B, Gemma 7B, LLaMa 1 {7, 33, 65} B, MPT 30B, Pythia {1, 1.4, 2.8, 6.9, 12}B, StableLM 3B. In addition to these open source models, we use 25 internal checkpoints from LLaMa-architecture models we pre-trained on our interal data mix.

Random split (test): DeepSeek 7B, Falcon 7B, Qwen 1.5 110B, LLaMa 1 13B, StableLM 7B, and 9 internal checkpoints.

### D.2 Additional results

We present results on additional benchmarks, in a similar format to Figure 3, in Figure 6. Furthermore, we provide extended results on the random split of models in Figure 7.

Figure: Figure 6: Item analysis results on six additional benchmarks, in the same format as Figure 3.
Refer to caption: x2.png

Figure: Figure 7: Results on 8 benchmarks when removing points based on item discrimination on the random split. These plots are similar to the final 3 columns in Figure 3 and Figure 6. Specifically, we show the effects of iteratively removing up to 20% of items (based on discrimination) on the mean (first column), standard error (second column) of model performance on the test set from the random split by looking at the delta. Error bars indicate 95% confidence intervals in the delta. Monotonicity (sixth column) is calculated over the 10 runs from Section 2.1. Orange curves show effects from randomly removing points, as a baseline. As we can see, these plots look qualitatively similar to Figure 3 and Figure 6 indicating that the observed lack of benefit from pruning based on item discrimination is not simply due to using the difficulty split of models.
Refer to caption: x3.png

### D.3 Inspection of samples with low item discrimination

We provide the 3 items from GSM8k, ARC-C and Hellaswag with the lowest item discrimination.

For GSM8k:

For ARC-challenge (correct answer is italicized):

For Hellaswag:

Figure: Figure 8: Scatter plots of two features correlated with item discrimination (calculated on the train set of models from the difficulty split). Low item discrimination tends to correspond to short prompts that do not contain ‘[header]’ tags.
Refer to caption: x4.png

Note that for Hellaswag, we did find some correlations to item discrimination in terms of features of the problems. Specifically, as shown in Figure 8, we found that items with low discrimination tend to feature shorter prompts and do not contain tags such as ‘[header]’ in the prompt.

## Appendix E Item response theory additional information

### E.1 A brief primer on IRT

While IRT can refer to a variety of methods, here we focus on the two-parameter multidimensional IRT model used by Polo et al. () to make tiny-benchmarks. Specifically, we define a matrix of model scores on a set of evaluation examples, $Y$, such that $Y_{ms}$ is the score of model $m$ on evaluation example $s$. As this model is mostly applied to discrete metrics in our cases (e.g., accuracy), we focus our exposition on the case where $Y_{ms}\in[0,1]$ (see Polo et al. () for details on extending to continuous metrics). The IRT model then learns vector embeddings for each model, $\theta_{m}$, vector embeddings for each example $\alpha_{s}$ as well as a scalar bias for each example $\beta_{s}$ to maximize the likelihood of the observations:

$$ $\displaystyle P(Y_{ms}=1|\theta_{m},\alpha_{s},\beta_{s})=\frac{1}{-\alpha_{s} ^{\top}\theta_{m}+\beta_{s}}$ $$

Polo et al. () then learn values of $\theta_{m},\alpha_{s},\beta_{s}$ for a set of train models across a range of benchmarks. Then, they perform clustering on the evaluation samples where the embedding of each sample is given by $(\alpha_{s},\beta_{s})$. Finally, they subselect 100 data points that are the most representative and assign weights equal to the size of their clusters.

For a new model, they propose two methods for evaluation. In the first, which is termed “IRT” (to match their paper), we simply use the weighted performance of a model on the 100 data points they identify. In the second, which is termed “IRT++”, we consider a weighted combination of “IRT” and an adjusted estimate (which is achieved by 1. learning a $\theta_{m}$ for the new model on the 100 evaluated data points, using fixed $\alpha_{s},\beta_{s}$, then 2. using the learned $\theta_{m}$ with the fixed $\alpha_{s},\beta_{s}$ for all data points to estimate model performance). Polo et al. () find IRT++ to outperform the IRT estimator, which we reproduce (see Figure 4).

For a full description of the method, we refer the reader to Polo et al. ()—we include this primer here for completeness.

### E.2 Additional results using TinyBenchmarks

See Table 7 and Figure 9.

**Table 7: Change in model ranking when using IRT-based methods We compute and list the Kendall rank correlation $\tau$ between model ordering when using IRT-based estimates for each benchmark. To contextualize these, we also compute the percentage of pairwise comparisons which would be flipped (denoted %). We also show results limited to the 14 best performing models (the test set of the difficulty split—see § D.1) in the last two columns.**
| Benchmark | IRT $\tau$ | IRT++ $\tau$ | IRT % | IRT++ % | IRT % (diff.) | IRT++ % (diff.) |
| --- | --- | --- | --- | --- | --- | --- |
| ARC-C | 0.759 | 0.798 | 12.17 | 10.09 | 4.40 | 5.49 |
| GSM8k | 0.913 | 0.912 | 4.51 | 4.51 | 10.99 | 10.99 |
| Hellaswag | 0.881 | 0.794 | 5.96 | 10.35 | 15.38 | 30.77 |

Figure: Figure 9: Increased variance when using IRT or IRT++ based estimation of benchmark means during pretraining. While Table 4 shows the decreased monotonicity when estimating with IRT-based methods, here we show performance curves through training for each of the 10 pretraining runs from § 2.1. Curves are visibly noisier (and less monotonic), showing the increased difficulty pracitioners may have in interpreting results if using IRT-based methods.
Refer to caption: extracted/5666202/figures/appx_full_tiny_seed_comparison.png

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: 1. What is the most striking example with which you can demonstrate the potential impact of this work?
2. There are many challenges to evaluating foundation models and it is clearly also a matter of how much money (time, compute) to invest in which aspect to arrive at a conclusive result. So evaluation is inherently a trade-off and it is important to understand and acknowledge this trade-off. In my opinion there seem to be much more critical aspects that need to be addressed than the variance studied in this paper. For example, the paper "EVALUATING LLMS’ MATHEMATICAL AND CODING COMPETENCY THROUGH ONTOLOGY-GUIDED INTERVENTIONS" by Pengfei Hong et al seems to be a good route towards useful evaluations. I would rather invest time and compute into that direction and not bother about the methods proposed in this submission. Are you advocating for the opposite strategy?
soundness: 3
strengths: The paper is clearly written and easy to understand.
summary: This work aims to quantify evaluation benchmark variance across a range of settings (from pretraining intermediate checkpoints, to the largest frontier LLMs) using a diverse set of metrics (seed variance, confidence intervals, and monotonicity). Beyond quantifying variance, the paper also explores item response theory.
weaknesses: The key problem for me is that I do not get the value proposition of this work. It's difficult to see how this work could develop relevance/impact for the evaluation of foundation models. Since that's why I am hesitant to support the paper I focus my review and questions below fully on this point.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - Are there any other kinds of variance that could have been used for this study instead of initialization seed?
- Only training-based sources of variance are discussed, what about inference-based?
- Section 3.3 outlines a very interesting case of differences in evaluation results after a reformulation of the setup, could this be shown for other benchmarks? perhaps a similar pair?
soundness: 2
strengths: - The paper explores how to make evaluations more precise by reporting variance.
- Provides estimates for the expected variance across several benchmarks and models.
- An important finding is made on the unreliability of IRT-based methods for evaluation comparisons across models. This is very relevant for evaluation reporting.
summary: This paper discusses the problem in current evaluation and reporting practices where performance may vary across different development choices. This variance is scoped for training initialization seed and monotonicity. The paper also discusses ways to reduce this variance, in particular for choice based benchmarks.
weaknesses: - The framing of ‘variance’ in the paper seems too broad. There are other possible kinds of variance worth exploring or mentioning.
- The title of the paper and general framing suggests a general focus on variance in evaluations, but the paper currently fails to contextualize two very distinct types of variance: training and inference. For example, the (training) seed variance discussed falls within training. Other possible sources of variance for each should be mentioned where possible. A basic example of the inference type can be found in prompt sampling.
- While studying the training seed variance is useful, this is really only feasible for smaller models, as it would be too expensive for larger models. This may reduce the utility of the results in large model comparisons.

The paper could mention previous work such as Picard 2021 (https://arxiv.org/abs/2109.08203) on the impact of training initialization seed variance, or fine tuning seed variance [Dodge 2020 (https://arxiv.org/abs/2002.06305)]. And also extended discussions on how obscuring or not disclosing these variances can be harmful to the evaluation process (e.g. [Leech 2024 (https://arxiv.org/abs/2407.12220)]).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: I have quite a few questions that I think would improve the quality of the paper, and for my own clarifications:
 
- Why are the Disc Std in tab 2 and std in tab 1 different? Aren’t they computing the same metric over the same set of models? The means are exactly the same for the discrete metrics so I presume the stds should be too?

- In tab. 2 what decoding strategy do you use for the generative tasks? The difference between GSM8k (0.99) and HumanEval (0.21) seems quite high for a difference in log-likelihoods. Are these token-length normalised likelihoods or unnormalised likelihoods?

- Comment: Typo in fig 3 caption, should be item difficulty (y-axis) and item discriminability (x-axis).

- For the analysis in fig 3, how does the correlation between item discrimination b/w train and test look like for another randomly selected set of train and test models? The key question is what the variance in the correlation obtained for a random split of random train-test models would be? This would make the conclusion (that the low correlation between train and test item discriminability is due to the differences in model capability) more strong and robust.

- The “best" and “worst” models used for creating the train and test splits do not share the same training data mixtures right? So I would expect that the takeaways from fig 3, especially the ones involving the difficulty split are also confounded by different levels of data contamination with respect to the test sets? 

- For the points in section 5, the main claims revolve around using the IRT benchmarks themselves. How much of that can be explained by the increased variance from just having a smaller number of test samples? i.e. how would the results look like if instead of the IRT test set, a random set of evaluation points of the same size as the IRT set were used for the analyses?
soundness: 2
strengths: - The paper presents a timely and important topic of variance in evaluation benchmarks, that should be widely considered while reporting benchmark performance. 
- The paper is cogently written and presents convincing demonstrations of the importance of considering variance in evaluations while doing model comparisons.
- The paper showcases results and cautions against using sample efficient benchmarking methods while doing model pretraining, since they are likely to provide a higher-variance signal during training.
summary: This paper aims to quantify the amount of variance prevalent in popular LLM evaluation benchmarks, mainly by varying the random seeds for model training. The paper makes an important point about how we currently consider benchmark scores to only be point estimates rather than considering several other different factors of stochasticity while doing model comparisons. The paper conducts several experiments to quantify this seed variance across different benchmarks, and provides practical guidance on what metrics and evaluations to use during pretraining to provide most signal.
weaknesses: - The provided variance numbers in Tab. 1, while being important as a reference, cannot be used directly for making comparisons across different model scales or training durations since it is not clear how those numbers would change with those factors, and whether we’d expect larger or smaller deviations in performance.

- Some important empirical details are missing, for example, could you provide more details on how you compute the SNRs for both discrete and continuous? This is important since you are making the claim that Cont SNRs being larger than Disc SNRs suggests that we should shift towards continuous metrics for making model comparison decisions, and this claim can only be validated if the precise method of computing these SNRs is justified.

- Another important question that is critical for the takeaways of the paper: Is the unnormalised monotonocity the best metric here to capture the “stableness” of a benchmark? Shouldn’t that monotonicity be weighted by the monotonicity you would expect by chance? So something like a cohen’s kappa coefficient here seems more appropriate rather than just the direct unnormalised monotonicity. For eg, see the analysis in Geirhos et al, Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. I appreciate that this might be hard to formalise since its unclear what the “monotonoicty expected by chance is” but I believe this is worth at least a discussion point, and would like to hear the author’s thoughts on this. Also worth mentioning here that monotonicity was also explored for doing benchmark selection as good validation datasets for the fine-web training recipe. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale.

- Suggestion on formatting the paper: the results sections in 3 and 4 have some very important insights / takeaways for practitioners to adopt for pretraining and actual evaluation. I think it would majorly improve the paper if these could be highlighted at the end of each subsection in bold, or even better, a small box signifying the key takeaway.

- There are other kinds of variance inducing factors that haven't been investigated. For example, how does task type affect this variance?
How does number of shots and choice of shots affect variance? How does model size impact variance? Does it increase as we scale up the model? There should be a discussion added about the other variance inducing factors which haven't been considered in the current work, and it must be made clear in the paper that "seed variance" is the only type of variance being investigated in this work.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Considering that the author trained different models from scratch, such as Hellaswag, which consistently outperformed random choice across a wide range of checkpoints, it would be beneficial if the author could use intermediate checkpoints to demonstrate the benchmark variance at different stages of training and performance. Furthermore, showing how benchmark variance changes with training progress could be helpful for models of various sizes and training data volumes.
- When calculating Seed Variance, would it be more reasonable to exclude checkpoints that are clearly still within the random result range from the statistical analysis?
- When models of different sizes achieve the same performance after being trained with different numbers of tokens (e.g., a 7B model trained with 120B tokens and a 1.5B model trained with 400B tokens), do they exhibit significant differences in benchmark variance?
soundness: 3
strengths: - By training different models from scratch, the paper provides the most direct results for evaluating the variance brought by randomness. Based on these results, it offers valuable suggestions for assessing smaller-scale models.
- The paper introduces methods from human testing literature and explains the reasons for their limited success, providing insights for future work.
summary: This paper investigates the variance present in current evaluation benchmarks by repeating the training of models with different random seeds. It demonstrates the variance caused by random factors in benchmarks, providing valuable references for assessing model evaluation results, particularly for the MMLU evaluation of smaller-scale models. The paper attempts to reduce evaluation variance using methods inspired by human testing literature, such as item analysis and item response theory, but finds that these methods have limited effectiveness.
weaknesses: - The paper focuses only on the variance caused by model random seeds and does not compare or combine this with existing work on the analysis of evaluation result variance. For example:
  1. It does not compare the variance caused by random seeds with the impact of other random factors during evaluation (such as option order, prompts, etc.).
  2. It does not explore whether the variance is further amplified when models with different seeds encounter situations like randomized options.
- The paper primarily showcases the overall variance of 210 7B model checkpoints. Given that models of different sizes exhibit significant differences in performance when trained with varying numbers of tokens, the overall variance statistics may have limited reference value for models trained with fewer or more data.

---

confidence: 2
contribution: 3
initial_rating: 3
presentation: 2
questions: See weaknesses above please. 

Overall, I think we should definitely report variance in benchmark estimates to compare significance of improvement, however I believe retraining $k$ times to obtain seed variance might not be the critical factor in total variance, and hugely expensive. I do think the recommendation of using continuous metrics makes a lot of sense. 

I think there are important shortcomings in my view, although I might likely be wrong. If weakness 1 is adequately alleviated, I would upgrade my score. Note: Weakness 3 and 4 are minor and have little effect on rating.
soundness: 1
strengths: Key takeaways I appreciated:

- Continuous metrics show higher monotonicity and lower variance than discrete metrics, making them preferable. I really like this point!
- Variance (likely from benchmarking, not seed variance) could obscure the effectiveness of pretraining by making it harder to track performance improvements.

Benchmarking variance is a promising and underexplored area, especially given the recent focus on reasoning and the close performance of current models on benchmarks-- and seed variance can have a lot of effect.
summary: This paper quantifies and highlights the important issue of seed variance, amongst different sources of variance, in evaluating language model performance *during pretraining*, emphasizing how insufficient variance quantification in benchmarks can obscure statistically significant performance differences within the same pretraining run.

**Methodology**: It retrains 10 LLaMA 7B models from scratch, and then studies the seed variance along with different metrics throughout the pretraining runs, such as-- 95% confidence intervals, and monotonicity across diverse benchmarks, offering a reference for understanding performance variation across setups.

**Claim 1**: Continuous metrics exhibit higher monotonicity and lower variance than discrete metrics and should be used more widely.

*Application*: Continuous metrics and cloze formats show lower variance and higher signal-to-noise ratios compared to traditional discrete measures, especially in smaller LLaMA 7B models. Simple modifications, such as reframing choice tasks as completion tasks, appear promising for reducing variance.


**Claim 2**: Efficient benchmarking methods can inadvertently increase variance, and it’s essential to verify if methods are distinguishable before using them.

*Application*: Techniques from standardized testing, such as item analysis and item response theory, are found ineffective in meaningfully reducing variance for large language models.
weaknesses: See Weaknesses in the order of importance. If there is lack of time, please prioritize the earlier questions:

**P1: Seed Variance and Its Impact on Benchmark Scores**

> We provide a comprehensive reference guide for what magnitudes of variance are expected for what benchmarks across various circumstances

I worry the paper title/abstract/contribution1 is overclaiming as the evidence does not support the above claim-- It seems to me that paper focuses only on a single source of variance: seed variance. Given that studying variance is a common practice (the most popular benchmark, Chatbot Arena provides variance along with scores: https://lmsys.org/blog/2024-04-19-arena-hard/) and previous research on variance sources such as benchmarking details [1] and prompt design [2], the question studied here becomes: Does seed variance contribute overwhelmingly to the overall variance which makes it a critical factor to study, beyond [1,2,Arena]? 

I would appreciate:

a) A comparative analysis showing the relative impact of seed variance versus other variance sources on benchmark results.

b) A better distinction in the work for why studying seed variance specifically is important, given the existing literature on other variance sources.

Since the authors claim:
>  if we cannot ‘trust’ our evaluation results or do not understand what improvements are statistically significant, we cannot make sound comparisons, thus making it more challenging to reliably use benchmarks during model development.

a) I could not see the results of the 32 models apart from Llama-2-7B seed runs anywhere. Would be great to know whether the variance obtained Llama-7B seed runs can demonstrate that improvements reported in those models being deemed not-significant on some benchmarks.
b) To what degree are these variance estimates transferable beyond the Llama-2-7B model?.

**Overall:** Currently, it seems to me that, variances from [1] and [2] are critical but easy-to-obtain, while seed variance is compute-heavy to obtain and potentially not as significant source given different prompts and small benchmark details can cause far larger performance shifts. I request authors to present evidence or arguments suggesting this is not the case, and I am open to changing my mind.

[1] When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards

[2] Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting.

**P2: Efficient Benchmarking in High-Variance Contexts**

(a) The rationale here is unclear. 
Q1. If variance preservation is a priority, why subsample to an extreme degree of 100 samples? What are the trade-offs between efficiency and variance preservation in benchmarking? 

Q2. Conversely, does aggregating several 100 sample estimates reduce variance while still providing efficiency gains compared to evaluating on the whole dataset? Or in most concrete scenarios, efficient benchmarking just might be limited? 

(b) The paper says:
> While these negative results suggest item discriminations may not be the most informative means of understanding (or reducing) variance on stronger models 

> we overall would not suggest the use of item analysis-based methods for understanding variance in language model evaluations 

I doubt those papers claim their methods are intended to reduce or understand variance. Could the authors provide a citation for this claim? If no direct claims about variance reduction exist in the cited works, could the authors discuss in the work why they believe these methods should be evaluated in terms of variance reduction.

(c) While I agree with the authors that generally, subsampling might lead to critical issues in benchmarking – the results presented seem underwhelming given I fully buy the claim of using continuous metrics to reduce variance.

Specifically, the results in Table 7 indicate that Kendall’s tau still remains quite high and relatively stable. Do the authors believe this is damning evidence against efficient benchmarking methods? (I agree in principle there will be cases where this is damning, but the shown results do not seem to be those cases.) Similarly, Table3 and Table 4 show variance and monotonicity increases but the increase in variance/decrease in monotonicity remains quite small post-subsampling compared to the relative gains in benchmarking efficiency (except in the case of GSM-8k discrete metric).

**P3: MMLU Evaluation and Metric Selection**

I fully agree that continuous metrics may indeed have lower variance than discrete metrics (Table 2 demonstrates convincing gains in PSNR), the emphasis on MMLU in this context feels misplaced as one metric is near random. 

Metrics that perform near random chance are unreliable indicators of progress, while those consistently above random are more dependable. However, for smaller models, this reliability may simply stem from selecting a metric that surpasses random performance earlier in training, rather than addressing the broader variance concerns noted above. I don't know why this specific example was picked.

P4. **Poor Writing** 

The writing lacks clarity and precision, with ambiguous and poorly articulated claims scattered throughout the paper. This made the review difficult. In the summary, I’ve tried my best to interpret the paper’s main claims — please let me know if my summary underclaims compared to the intended or the claims differ from those presented in the work.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Can authors show similar evidence for instruction tuned models? It seems to me that it is not expensive to instruction tune base models authors already have obtained, or at least use already available instruction tuned models. Alternatively, can authors present more benchs that do not heavily rely on instruction like templates?

Can authors present evidence that using continuous metrics they can do better on predicting either end model performance from early training phases or larger scale model performance from smaller scale pretraining?
soundness: 3
strengths: Question of which metrics used for benchmarking model capabilities are properly reflecting model performance and allow differentiation of models already in early training stage or on smaller scales in general, where performance signal is weak, is very important, as it also leads to better methods for scaling law derivation and proper model comparison via derived scaling laws.
summary: Authors address the question of measuring variance in standardized evaluation benchmarks. They do so by constructing various ways to assess variance including variance due to different init seeds and looking at evaluation monotonicity at checkpoints in course of pre-training. Authors study a number of already pre-trained models and also train Llama like models from scratch on 7B scales, varying their init seed and obtaining intermediate checkpoints to use for variance estimation. Authors show evidence for continuous metrics delivering better signal-to-noise ratio across all benchmarks, also showing better monotonicity for continuous metrics than for discrete ones. Authors also look into techniques used in human performance testing like item analysis and item response theory and find those not useful for assessing model performance and improving SNR.
weaknesses: - Claims put forward by authors are made by experimenting with base models only. Especially when taking benchmarks that have certain question formatting at their core - MMLU, ARC, HellaSwag - it seems to me a strong confound when attempting to make a statement about ability of a benchmark to measure model capabilities. Base models do not handle instruction format well, in general struggling with handling question-answer interaction. This is usually installed via instruction tuning, after which models adhere to various instruction like interactions. I think thus to make proper statements about effect of variance or in general about how to render benchmark signal useful for predictions about model quality, benchmarks that do not rely on certain instruction template are not a good choice when dealing with base models. In my opinion, when working with such benchmarks, authors should have instruction-tuned base models, conducting measurements after that. Benefit of continuous metrics might be thus due to base models not handling instruction like benchs and not due to actual benchmark content related to complexity of problems posed in it. Same might hold for the "curious MMLU" case presented by authors, where cloze variant shows smaller variance and better SNR than multiple choice format standard MMLU form - this might be again just due to base models not being able to handle problem formulation template.

- Further weakness in my opinion is that many of the claims are based on pre-training from scratch done only on one model scale of 7B. It might be more insightful to see trends across scales, even smaller ones. Experiments on larger scale are though expensive, even if going bit further to 13B.

- It is also not quite clear how the claim of better suitability of continuous metrics is backed up. It seems to me there is no clear evidence presented by authors that using those metrics eg indeed allows better prediction from earlier to later training stages or from smaller to larger scale pretraining.

Meta-review: This work investigates the variance in LLM evaluation benchmarks, primarily looking at sources of variance that manifest during training rather than at test time. To this end, a set of 7B Llama-style models are trained and experiments are conducted to determine how the conclusions drawn from benchmarks can vary throughout the training process. Various factors are explored in the experiments, including whether discrete or continuous metrics are used, how choice tasks are framed, and whether techniques from human testing can reduce variance.

The reviewers mostly had a negative opinion of this paper, with the bulk of the criticism centred on the perceived significance of variance due to the choice of random seed/initialisation. Several reviewers (i.e., tZvi, i8dM, 7Zuc, and VsPx) suggested that it would be more worthwhile to investigate other sources of variance that manifest at inference time. I have a dissenting opinion here: the focus of this submission is on how variance in benchmarking *during pre-training* can impact subsequent modelling choices. This is in contrast to much of the other work in this area and the suggestions of the reviewers, who tend to focus on evaluating models that have already finished training. With that said, I still am not prepared to override the majority decision of the reviewers. As pointed out by reviewer w3M8 and V3wm, the submission in its current form does not allow for inferences to be made across model scales, which would be important for deriving scaling laws that are robust to evaluation variance during training.

Decision: Reject