How Reliable is Language Model Micro-Benchmarking?
Authors: Gregory Yauney
Shahzaib Saqib Warraich
Swabha Swayamdipta
Venue: ICLR 2026 Oral (2026)
Keywords: efficient evaluation
meta-evaluation
language models
Primary area: foundation or frontier models, including LLMs

Abstract: Micro-benchmarking offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing benchmarks. Can these micro-benchmarks, however, rank models as consistently as the full benchmarks they replace? And can they rank models more consistently than selecting a random subset of data points? In many scenarios, we find that the answer is no. We introduce a meta-evaluation measure for micro-benchmarking which investigates how well a micro-benchmark can rank two models as a function of their performance difference on the full benchmark. This approach can determine which model pairs can be ranked correctly by a micro-benchmark, allowing for a finer-grained analysis of the trade-off between micro-benchmark size and reliability.
Prior work has suggested selecting as few as 10 examples; we find that no micro-benchmarking method can consistently rank model pairs 3.5 points of accuracy apart on MMLU-Pro or 4 points apart on BIG-bench Hard. In order to consistently rank model pairs with relatively similar performances, we show that often as many as 250 examples must be selected, at which point random sampling is competitive with existing micro-benchmarking methods. When comparing only 8B instruction-tuned models on MMLU-Pro micro-benchmarks with 25 examples, we find that more than half of pairwise comparisons are not likely to be preserved. Our work provides actionable guidance for both micro-benchmark users and developers in navigating the trade-off between evaluation efficiency and reliability.

Published as a conference paper at ICLR 2026
How RELIABLE IS LANGUAGE MODEL MICRO-BENCHMARKING
Gregory Yauney Shahzaib Saqib Warraich
Swabha Swayamdipta
Thomas Lord Department of Computer Science University of Southern California Los Angeles, CA, USA Eyauney@gmail.com (warraich, swabhas edu
ABSTRACT
flicrobbncchmarknng offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing bench- marks. Can these micro- mhirrobhenchmark,, however, rank models as consistently the full benchmarks they replace? And can they rank models more consistently than selecting random subset of data points? In many scenarios, find that the answer is no. We introduce meta-evaluation measure for micro-benchmarking rank two models function
which investigates how well a micro-benchhart their performance difference on the full benchmark. This approach can deter- mine which model pairs can be ranked correctly by micro-benchmark, allowing for a finer-grained analysis of the trade-off between micro-benchmark size and reliability. Prior work has suggested selecting as few as 10 examples: we find that no micro-bencheee benchmarking method can consistently rank model pairs 3.5 points of accuracy apart on MMI.U-rr or 4 points apart on BIG-bench Hard. In order consistently rank model pairs with relatively similar performances, we show that often as many as 250 examples must be selected, which point random sampling is competitive with existing beccobbenchmariing methods. When comparing only 8B instruction-tunee models on MMLU-PP micro-benchmarks with 25 examples, we find that more than half of pairwise comparisons not likely to be preserved. Our work provides actionable guidance for both berro-benchmark users and developers in navigating the trade-off between evaluation efficiency and reliability. Code and data are at: titb.
INTRODUCTION
Miero-benchmarking methods reduce the evaluation time of language models by predicting perfor- mance on full benchmark from performance on small subset (Vivek al., 2024: Polo et al., 2024; Perlitz et al., 2024; Ye et al., 2023; Gupta et al., 2025). Current bencro-bbnchmrrkngg determines these subsets based different criteria ($2). For instance, Anchor Points (Vivek et al., 2024) selects the centroids of test example clusters in the space of model predictions. BinyBenncharks (Polo et al., 2024) selects examples close to the centroid of clusters obtained using Item Response Theory (Cai et al., 2016). But there is a general trade-off between efficiency and the reliability of the judgments drawn from any evaluation set: smaller eval are more cost-effective, but they do not always accurately reflect which model will perform best in practice "Shalev-Shwarr & Ben- Ren-D,vid, 2014; Dror al.. 2018). Do we ask: How can we measure the extent to which micro-benchmarks reflect the model performance judgments of full benchmarks?
suffer from this general trade-off, as well? Specifically,
Prior work has evaluated reconstruct the accuracy of any single model on the full evaluation set (Polo et al., 2024), and (ii) how well they preserve the aggregate rankings of set of models (Vivek et al., 2024; Perlitz et al., 2024). We evaluate ditero-benchmate instead on their ability to predict pairwise model rankings on the full benchmark: Given that model M1 outperforms model M2 on the full benchmark, what the probability that model M, outperforms M, on the biich-bbnnchark? Inspired by statistical power analysis (Card et al., 2020), measure the minimum performance difference between models
using two meta-evaluation approaches: (i) how well they

---

Published as a conference paper at ICLR 2026
M1 and M2 on the full benchmark that still consistently yields a correct pairwise ranking of the two models on the micro-benchmark We introduce a meta-evaluation measure, the Minimum Detectable Ability Difference (MDAD) that offers fine-grained view of which performance judgments preserved by micro-benchmark ($3). A direct consequence of MDAD is an understanding of how micro-benchmark size affects its reliability across model pairs being compared ($5).. In addressing pairwise estimates, MDAD offers complementary benefits to existing meta-evaluation measures, which deal with pointwise estimates or aggregate rankings ($5.1).
Unlike micro-benchmark selection, selecting examples uniformly at random has the advantages of speed and simplicity: it does not require evaluating models to learn prediction correlations, nor does it train auxiliary models of instance difficulty. However, existing meta-evaluation has not characterized when micrr-benchhark selection outperforms random sampling (Vivek al., 2024; Polo et al., 2024). Our meta-evaluation measure, MDAD, reveals that the intuitive baseline of random sampling competitive with existing nicro-benchmarr selection under all but the most extreme dataset reductions ($5.2). We also use MDAD show the limits of mhicro-benchmarking in the common setting of comparing same-size models, which often have similar performances on a task. When selecting 25 examples from MMLU-Pro, 51% of pairwise comparisons among set of 3B-parameter instruction-tune tunne models are not likely to be preserved ($5.3).
Figure 1 compares evaluation with others on revealing the limits of micro-henchmarking. When comparing nearly 100 models MMLU- Pro (Wang et al., 2024), Kendall's tau rank correlation between the full benchmark and
Aggregate correlation with full benchmark examples (8.3%)
10 examples (0.2%)
Kendall's tau
1.0
1.0
0.24
on4
0.5
0.5
0.52
0.50
micro-benchhark shows that Anchor Points is better correlated with the full benchmark than random sampling is at extremely small dataset sizes (Figure 1, top left). However, a correla- tion of, say, 0.74, does not identify which model comparisons remain challenging for a micro- benchmark. Our measure, MDAD, considers the probability that a micro-benchmark agrees with the full benchmark accuracy difference between a model pair (Fig- 1, bottom panels). For two models that differ by points of accuracy on the full benchmark (dashed lines in Figure 1, bottom), we show that when only 10 examples are selected, no micro- benchmark can distinguish these models more than 65% of the time (Figure 1, bottom left). In contrast, when 500 examples are selected (Fig- ure I, bottom right), many mncro-benchmarke can distinguish the same models more than 90% of the time, including even random sampling, which Kendall's tau rank correlation corrobo- rates (Figure 1, top right). Thus, if practitioner wants to distinguish models 2 points of accuracy apart, then they could simply use random sam- pling to select at least 500 examples. If instead they wanted only to distinguish models that dif- fer by more then 4 points, then only 10 examples selected by Anchor Points would suffice.
bicch-bbenchmarking method
Micre-benchmarking method
Fine-grained probability of correct ranking
10 examples (0.2%)
500 examples (8.3%)
1.0
1.0
Agreement
0.6
0.4
0.4
function of the
12
16
20
8
32
16
Accuracy difference between models on full benchmark
Micro-benchmarking methods
Random (uniform)
Anchor Points
Stratified sampling (confidence)
tiny
Figure Existing meta-evaluation metries (c.g. Kendall's tau rank correlation) summarize micro- benchmark performance for MMLU-Pro in the aggregate (top). At extreme dataset reductions, miicro-benchmars can yield high aggregate rank correlation with full benchmarks (top left), but no mnicro-benchmmn
has a high probability of agree- ing with the full benchmark when ranking model pairs that differ by fewer than 4 points of accuracy (bottom left, gray background). Once enough ex- amples are selected to distinguish such model pairs (bottom right, gray background), random sampling is competitive. See $3 for details and Figure 4 for comparisons to another existing measure.
Overall. our meta-evaluation provides guidance for navigating the trade-off between evaluation efficiency and reliability. Very small miero-benchharks have value, but it vital to know that they will often only be able to distinguish models with very different performances. For the more pertinent task of distinguishing models with similar performances, larger micro-benchmarks are necessary, at which point random sampling is often enough for reliable, simple, and efficient evaluation.
2

---

Published as a conference paper at ICLR 2026
2 HICRO--ENCHMARKING PRELIMINARIES
We give a formal description of berrhbbenchmarning using the terminology from Vivek et al. (2024). Given a large evaluation dataset Dial, the goal is to select a miero-benchmark Dmiero C Diall where ID microl < Duull- Micro-benche biico-bennchmar selection typically assumes access to set of source models U that have been evaluated on the full dataset Diunt A method selects a becco-bbnnchmrrk D Pamicro& of a fixed size m = n with common high-level goal: for new set of target models T. performance on the micro-benchmark Dincco should be similar to performance on the full benchmark. This goal is realized in various ways by different methods.
bhich-bb selection. We consider four this paper. Anchor Points (Vivek et al., 2024) first calculates correlations between example pairs using source model confidences and then selects the centroids of test example clusters in the resulting embedding space, in order to obtain a high correlation between model rankings on the full benchmark and the selected micro-benchmark. The tinyBenchmart IRT method (Polo et al., 2024) instead aims to select micro-benchmart that minimizes the error when predicting accuracy for individual source models. does so by training an Item Response Theory (IRT) model, which results in example and source model embeddings. It then selects the closest example to centroids obtained from clustering these embeddings. Stratified random sampling (Fogliato et al., 2024b) randomly selects examples from clusters obtained based on model confidence examples. We also consider diversity- bbased method that samples examples spread evenly in the space of source model correlations used by Anchor Points, enabled by sampler that can select negatively-dependent samples (Bardenet et al., 2024). Appendix B gives detailed descriptions of these methods.
selection methods throughout
Existing meta-evaluations for micro-benchmarks. Prior work has measured the degree to which micro-benchmarks preserve target model performance in two ways: (i) for individual models using mean estimation error and (ii) in the aggregate for a whole set of target models using rank correlation. Mean estimation error measures the difference between model performance on the miero-benchmark and the full evaluation set Dial a set of target models T (Polo et al., 2024). On the other hand, Kendall's 's tau rank correlation (Nelsen, 2001) between all target model rankings on Dmiero and Dtuit measures becch-benchmark fidelity an entire set of target models (Vivek et al., 2024). A pair of models M1, M2 E T is said to be a discordant pair if the models are ranked differently on the full benchmark and the niero-benchmark, Let pertp be the performance of model M an evaluation set D, and let C' be the set of these discordant pairs. The metrics are calculated as:
err
(T)
1
Σ MET
pe) - perf (M)
Kendall's
= ===
2 |C"
(171)
(I)
Danm Dia
Dun
Driem
Random sampling. Uniform random sampling selects a fixed-size subset of examples indepen- dently and uniformly at random, without any model dependence:
Dmicco Unif ((R S Daa | = n))
We also consider another variant of stratified random sampling that takes into account a benchmark's t pre-defined subtasks {D: where Dio = U, D,, where an equal number of examples are selected from each subtask (Polo et al., 2024; Perlitz et al., 2024):
D Paniroo Ü Rg where R, ~ Unif (f S D. D.|[RR| = 1%/1))
Micro-benchmark settings. Many choices go into building nicro-benchmarks: which source models to use, which target models to evaluate, and even which examples to select from in the first place. Prior work typically averages evaluation metrics over many partitions of fixed set of models into source and target models at random (Vivek et al., 2024; Polo et al., 2024), by model family (Vivek et al., 2024), or by model release date (Polo et al., 2024). The examples selected for a micro-benchmark also vary over optimization hyperparameters, such random seeds.
Polo et al. (2024) propose method variants that also incorporate IRT model predictions. We focus on their core method, our initial experiments showed similar results for variants.
3

---

Published as a conference paper at ICLR 2026
3 MDAD: A META-EVALUATION FOR MICRO-BENCHMARK RELIABILITY
We present a meta-evaluation for micro-benchmarks based on how consistently pairwise model rankings on them agree with those obtained on the full benchmark. Given that model Mj outperforms model M2 on the full dataset, how likely is it that model M1 also outperforms model M2 on the micro-benchmark? We consider this pairwise ranking agreement probability as a function of the performance difference between the model pair, aggregated over all target model pairs.
We first calculate the probability agreement between hhrrr-bbenchmakk and the full dataset at a given difference in performance between two models on the full dataset. Let the performance differ- ence between models M1 and M2 on eval dataset D be (M, M22 = perf (M1) perff (M2).. We fix a set of ordered buckets of pairwise performance differences B and define agreement for each bucket B B. Assuming without loss of generality that the model with higher performance on the full benchmark is called M1, then for D selected from Druli:
agreement ( micro: Druli B)
Pr Mc.M
D (M1, M2) 0 A (M1, M2) M2)
(4)
In practice, compute agreement using all pairs of target model performances and calculate the frequency of target model comparisons in each bucket that match on both Diicro and D
The agreement function be summarized using single value that call the Minimum Detectable Ability Difference (MDAD): what is the lowest performance difference on the full benchmark at which pairwise model rankings goal here is to adapt the idea from statistical power analysis of estimating the minimum difference in model performance that can be consistently detected by dataset (Card al., 2020; Cohen, 1962). Following conventions from statistical power analysis, we consider judgment consistent if it is correct least 80% of the time:?
secro-bbencmark
consistent with those the full benchmark? Our
MDAD(Drp miccro Diai) arg min agreement (aaimennttmmcooo Dmicro Diun, B]} s.t. Pr 2 0.8
(5)
centroid (3) BEB
In practice, we report the centroid of the bucket corresponding the lowest performance difference. MDAD captures how well micro-benchmark preserves pairwise model rankings, unlike mean estimation error, which only considers single models, or rank correlation, which considers all model
Probability micro-benchmark agrees with full benchmark
MDAD
examples (0.2%)
50 examples (0.8%)
examples (16.6%)
0.9
0.9
Agrtemont
155
0.8
0.8
0.5
MDAD
7.07
0.6
0.5
0.5
0.5
A
9.4
1.0
13255 n. esci Ensit 7501 200 35.Che 0N)
Accuracy difference between models on full benchmark
Number examples micro-benchmarre
Random (uniform)
Anchor Points
Figure 2: Agreement and MDAD measures on MMLU-Pro for uniform random sampling and Anchor Points with 300 source models. The three left panels show the probability that pairwise ranking of models on a micro-benchmark. agrees with the full benchmark's ranking, as a function of the accuracy difference between those models the full benchmark. The rightmost panel summarizes all these agreement curves by showing the minimum detectable accuracy difference between models each miero-benchmark size, i.e. the accuracy difference which each curve in the first three panels crosses the 0.8 probability of agreement threshold. Points A-F show how each agreement curve summarized by MDAD: each point marks the minimum difference in accuracy where an agreement curve surpasses 0.8 probability of agreement. For MDAD, lower values are better. Error bars represent 95% bootstrap confidence intervals over 50 trials.
Appendix C. shows qualitatively similar results for different thresholds.
4

---

Published as a conference paper at ICLR 2026
Table 1: Summary of differences between MDAD and existing meta-evaltution measures.
Meta-evaluation measure
What does it measure?
Unit of comparison
Aggregation
Mean estimation error
Raw performance
Individual model
Across all models
Kendall's ' tau rank correlatinn MDAD
Model rankings Model rankings
Model pair
Across all model pairs
Model puir
Model puirs split by performance difference
rankings in the aggregate. Lower MDAD is better because then even small performance differences across model pairs can be reliably measured under D If two models' performances differ by less than mhicro-benchmarks MDAD, then that micro-benchmark is not likely to be able to consistently distinguish them. For example, suppose a micro-benchmark at given size results an MDAD of 10. Then at least 80% of the time, that micro-benchmarrk on the full benchmark by at least 10 performance points. If we instantiate agreement (Equation 4) using accuracy the performance measure, Equation 5 then measures the Minimum Detectable Accuracy Difference. Figure 2 provides an illustrative example of the relationship between agreement various accuracy differences and MDAD, which summarizes the agreement curve. For the rest of the paper, MDAD will refer to this accuracy-based instantiation. Table gives a conceptual overview of how MDAD differs from existing metl-evaluation measures.
correctly rank only model pairs that differ
4 EXPERIMENTAL DESIGN
Our goal is to measure the reliability of cocro-bennchmrks as function of which pairwise model rankings on a hiero-benchmmrk predict pairwise model rankings on a) the full eval set and b) fresh draw of equal size from the same task distribution.
For all experiments, we simulate different draws from a benchmark by splitting each benchmark in half (each subtask is divided in half as well): the train half is used to select the micro-benchhark, and the held-out half used to measure generalization. We also uniformly at random partition set of models into source models for selecting hicro-benchharks and set of target models whose accuracy we are predicting, as in Vivek et al. (2024) and Polo et al. (2024). We account for variance in niiro-bbnnchark construction by averaging over random samples of datasets and source models, following Card et al. (2020) and Perlitz et al. (2024). Most of our experiments select micro- benchmark from entire benchmark (as in Polo et al. (2024)), though in $5.4 we also evaluate selecting nhirr-benchharks per subtask (as in Vivek et al. (2024)) and report averages across subtasks. We compute meta-evaluation measures with the same data split used for micro-benchmark selection for most experiments. following prior work. In $5.4, we do so using the held-out set. We use 50 trials, each with a partition of a) data points into set for selection and a set for measuring generalization and b) models into source and target sets. Details are in Appendix D. Appendix E analyzes MDAD estimates for up 100 trials; we find MDADs have stabilized by 50 trials.
MDAD implementation details. Following prior work, we consider benhmmarinni methods specifically designed for classification tasks. We report accuracy as a percent from 0 to 100 and measure agreement using accuracy difference buckets at a resolution 0.5 points of accuracy, i.c. B {[0, 0.25), J0.25, 0.75), [0. 7575,, 1.25), reporting MDAD as the bucket centroid. 3 In our experiments, MDAD takes on average 2.40 seconds to compute (with a standard deviation of 0.24 s).
Benchmarks and miicrr-snschmars.. We consider 47 subtasks of MMLU (10,631 examples, Hendrycks et al., 2021). all 24 subtasks of BIG-Bench Hard (BBH, 5,761 examples, Suzgun et al., 2023), all 14 subtasks of MMLU-Pro (12,032 examples, Wang et al., 2024), and GPQA (448 examples, Rein et al., 2024). We investigate four mniro-bbncchmnn Points, BinyBenchmarks, stratified sampling by confidence, and diversity. We also compare to uniform random sampling and subtask-stratified random sampling.
selection methods discussed in $2: Anchor
Size of mnirro-benchmarde by selecting A E {10, 25, 50, 100, 250, 500, 1000} examples. For GPQA, a smaller bench- In Appendix G. we also evaluate
We evaluate constructing micro-benchmarks various small sizes
mark. we select k {10, 25, 50, 100, 200} examples.
AAppendix F shows that bucket resolutions of 0.25, 0.5, and all yield similar MDAD values.
5

---

Published as a conference paper at ICLR 2026
10 examples
25 examples
50 examples
1000 examples
250 examples
MDAD
MMLU-Pro
9.8
2.6
0.4
Agreamane
02
12
20.0
20
33
20
10
13
0
20
1.
20.0
20
20
10 3te 253
Accuracy difference between models fulli benchmark
Number of exemples In micro-bendhhark
Random (uniform)
Stratified sampling (confidence)
Anchor Points
Random (subtask stratifled, equal)
Diversity
tiryBenchmarks
Figure 3: Comparing six micro-benchhmrking approaches on two benchmarks. y-axis shows agree- ment (Equation 4), the probability that -errhhbbne agrees with the full benchmark when comparing two models, as a function of how much those models differ on the full benchmark (z-axis). The rightmost column summarizes agreement curves using MDAD (Equation 5). For small micro- benchmarks, all methods struggle to compare models that differ by fewer than 4 points of accuracy on the full benchmark. Anchor Points does best, followed by tiny Benchhehsarks. Error bars show 95% bootstrap confidence intervals over 50 trials. Figure (Appendix G) shows all benchmarks.
constructing micro- beero-benchharks at various proportions of the original benchmark by selecting (2%, 4%, 8%, 16%, 32%, 40%} examples, finding qualitatively similar results.
Models. For BBH, MMLU-Pro, and GPQA, we use the results of 470 models tagged as official on the Open LLM Leaderboard v2 (Fourrier et al., 2024), as evaluated with the LM Eval Harness (Gao et al., 2024). For MMLU, we use the results of the 366 models from the Open LLM Leader- board (Fourrier et al., 2024), as in Polo et al. (2024). Model accuracy spans large ranges for all benchmarks, ranging from 25 to 75 on BBH, 27 to 76 on MMLU, to 59 on MMLUUPro, and 21 to 45 GPQA. Unless otherwise stated, we randomly partition models into source models and target models as in (Vivek et al., 2024: Polo et al., 2024). We train hicro-benchmarks with {10, 50, 100, 150, 200, 250, 300} source models in order to determine whether increasing the number sarro-benchmarks Prior work has typically used a fixed number of source models for most experiments, either 10 source models (Vivek et al., 2024) or nearly 300 source models (Polo et al., 2024). Figures report results using 300 source models unless otherwise stated. Our approach of freshly computing beccobbbencmarks using publicly available cached model predictions follows prior work (Polo et al., 2024).
of source models substantially improves
Difference from standard evaluations. Our experiments not attempt to find the "best" examples for evaluation-raan rather, they designed to assess the reliability of existing miicrobbnncmarke We also specifically seek to understand the conditions under which random sampling is an effective alternative existing nhicro-benchmarking For this reason, do release specific subsets of benchmarks like MinL (Polo et al., 2024) or Flash-HELM (Perlitz et al., 2024).
5 RESULTS
We consider four miicro-benchmark selection methods and two random sampling baselines across the four benchmark suites, using agreement (Eq, 4) and MDAD (Eq, 5) from $3. MDAD reveals limitations of all the micro-benchharrkng approaches regime and provides a finer- finer-grained analysis than existing meta-evaluation measures ($5.1, $5.3). We show that random sampling competitive with other niccrobbnnchark selection if sampling least 250 examples ($5.2). $5.4 analyzes miiro-benchmark generalization to new draws the task. The complete results per benchmark for parameter settings are Appendix G. Appendix H shows that our mialysis also holds for a micro-benchmarkee method that selects whole subtasks.
consider in the extreme dataset reduction
Larger mhicrobenchmarke afford lower MDAD. Figure 3 examines how agreement varies with micro-benchmark size across methods and benchmarks. Each agreement curve is summarized by

---

Published as a conference paper at ICLR 2026
MDAD value (Figure 3, rightmost column); lower MDAD values correspond to higher reliability. As a first case study, consider BBH becro-benchmarks (Figure 3, bottom row). As more examples are selected, model pairwise rankings the full eval set. For example, the agreement curves in the 100-example panel have shifted to the left of their positions in the 10-example panel. Just as agreement steadily increases more examples selected, MDAD steadily decreases for all methods as more examples are selected. As the number of examples increases from 10 to 100 to 1000, MDAD for tinyBenchmarks drops from 16 to 1.
becco-bbnchmarks are likely to be more predictive of those on
All evaluated micro-benchmarks have limits at extremely small sizes. The leftmost column of Figure 3 shows that BBH nicro-benchmark of size 10 cannot reliably rank model pairs unless they differ by almost 15 points of accuracy! The only exception to this is Anchor Points, which has higher agreement at smaller
sizes. When 10 examples are selected from BBH, Anchor Points achieves an MDAD of 6. If model pair differs by more than 6 points on the full BBH, then this micro-benchmark is likely to correctly rank these models. If a model pair has an accuracy difference on the full BBH of less than the MDAD, e.g. 2 points of accuracy, then this mnero-benchmark is unlikely to correctly rank these models. Overall all micro- beroo-emeneaainngggggg. methods are limited when selecting only 10 examples. No method can consistently distinguish models that differ on the full benchmark by fewer than 3 points of accuracy on MMLU, 3.5 points of accuracy on MMLU-Pro, points of accuracy on BBH, or 6.5 points of accuracy on GPQA.
Anchor Points has the lowest MDAD at the smallest sizes but stagnates. The rightmost column of Figure 3 shows that Anchor Points has lower MDAD than other methods when selecting extremely few examples. We suspect this is because its method of selecting examples based on correlations in source model confidence is more closely aligned with what MDAD measures. On the other hand, Anchor Points has the highest MDADs when selecting 1,000 examples. We hypothesize this is due to very imbalanced cluster sizes in the underlying clustering that it performs. Anchor Points works by first clustering all of the full benchmark's 's examples using k-medoids and then selecting one example from each cluster. When selecting 10 examples from MMLU-Pro, cluster sizes are relatively even. But when selecting 1,000 examples, there is an extreme size imbalance between the 1,000 clusters of the benchmark examples: 47% of clusters are singletons, and the largest 10% of clusters together contain half of all the examples. Contrast this with tiny Benchmarks, which also performs clustering but uses k-means rather than k-medoids and uses a different embedding space. In the same setting of selecting 1,000 examples from MMLU with tinyBenchmarks, only 5% of clusters are singletons, and the largest 10% of clusters only contain 21% % of examples. Each selected example from the large clusters in Anchor Points must stand in for many more data points.
5.1 MDAD AFFORDS FINER-GRAINED ANALYSIS THAN EXISTING MEASURES
Figure 4 compares MDAD to mean estimation error and Kendall's tau rank correlation using the experimental design from $4. MDAD provides complementary- not to these existing measures. We also report the correlation between MDAD and other meta-evaluation measures, using Kendall's 's tau rank correlation.* Across all settings, MDAD has T = 0.701 (p < 0.05) Kendall's tau rank correlation with mean estimation error. MDAD also has a T = -0.787 (p < 0.05) correlation with the Kendall's tau rank correlation evaluation measure. Despite the high correlation and similar trends at high level, e.g. Anchor Points stands out on both MDAD and rank correlation, there are still fine- grained differences between the measures. MDAD does not map one-to-one to other measures.
MDAD provides more granular information than rank correlation. Even when micro- benchmarks have similar rank correlations in the aggregate, MDAD can identify fine-grained differ- ences between methods. For example, tinyBenchmarks and uniform random sampling both have identical rank correlations when selecting 10 examples from MMLU-Pro (Points A and B, Fig. 4). But uniform random sampling has much higher MDAD than tiny Benchhhhasrks in this setting (C and
*Note here that we are using Kendall's tau rank correlation a new way. to compare MDAD to existing meta- evaluation measures. One of these measures is itself the Kendall's It's tau rank correlation between micro- berco-bbecc and full benchmark model rankings that we have been considering throughout. Each setting with combination of henchmark, micro-bbnnchmakk size, and values. We calculate the correlation between the rankings of settings induced by those values.
method yields set of meta-evaluation measure
7

---

Published as a conference paper at ICLR 2026
MMLU
MMLU-Pro
BBH
GPQA
Rank carseaatio Eatimation arror
2.5
g.10
30
10
10
20
100
500
1000
10
100
250
500
(1N
10
15
50
100
500
03224
27 3g)
200
200
1.0
1.0
1.0
0.6
0.6
0.4
0.0
325
100
210
350
1202
0.0
10
120
310
2000
0.0
(5351
100
250
300
1000
0.0
50
200
200
MDAD
4.15
<..
29%
2.. 2
83%
Number examples micro-benchmark
Random (uniform)
Stratified sampling (confidence) Diversity
Points tinyBenchmarks
Random stretiiiettetfffed equal)
Figure 4: MDAD gives more granular information than mean estimation error and Kendall's tau rank correlation. Anchor Points is the only method that consistently outperforms random sampling at small dataset sizes across all metrics. Top row: Mean estimation error. Middle row: Kendall's 's tau rank correlation. Bottom row: Minimum Detectable Accuracy Difference (MDAD, ours, Equation 5). MDAD panels are the same in Figure 3, shown here for of comparison. Points A-H labeled for ease of reference in $5.1. Error bars represent 95% bootstrap confidence intervals over 50 trials.
D. Fig. 4). Comparing across datasets shows that different rank correlations can map to the same MDAD value. When selecting 10 examples, Anchor Points has a much higher rank correlation of 0.73 BBH but rank correlation of 0.43 on GPQA (E and F, Fig. 4). In both cases, MDAD is (G and H, Fig. 4). Even though the rank correlations are very different, both mnterr-benchmarde afford consistently accurate model comparisons when the models differ by at least 6 points of ac- curacy. Considering model comparisons only in the aggregate, as rank correlation does, obscures this finer-grained analysis. Additionally, MDAD values have a concrete interpretation- minimum model performance difference that an evaluation dataset can distinguish at least 80% of the üimlewhlee Kendall's s tau values are harder to interpret on their own.
the
MDAD accounts for consistent errors across models, unlike mean estimation error. Mean estimation error is defined for single model; it does not take into account whether maicro-benchmate consistently overestimates or consistently underestimates model accuracy for different models. For a simple illustrative example, if mean estimation error is 5 points, then the micro-bence beitro-bnncmmark s accuracy for a single model will be off by 5 points on average. when comparing two models, the micro- benchmark overestimates both of their accuracies by 5 points, the mhicro-bnnchmat can still yield the correct pairwise model ranking. MDAD accounts for this by directly measuring whether pairs of models are correctly ranked. For example, when 100 examples are selected from MMLU-Pro, Anchor Points has higher estimation eтror than random sampling but lower MDAD. Mean estimation error does not directly capture when models will be ranked correctly, but MDAD does.
5.2 RANDOM SAMPLING IS COMPETITIVE WITH OTHER METHODS
Figure 4 shows that all methods improve according to all metrics as more examples are selected. For each benchmark, there is a hicro-benchhark size at which both random sampling baselines become competitive with the other this occurs around 250 examples, and for GPQA (a much smaller dataset to begin with), this occurs at 200 examples. When selecting this many examples, all MDADs are 2 less. But it is worth noting that when selecting fewer examples, MDADs for all methods are higher. Even when the other methods outperform random sampling at smaller dataset sizes, their MDAD values show that they do not always consistently rank models that differ by few points of accuracy. Consider. for example, selecting 10 examples from MMLU: tiny Benchmaass (MDAD of 12.5) outperforms random sampling
selection. For MMLU, MMLU-Pro, and BBH,
8

---

Published as a conference paper at ICLR 2026
(MDAD of 20) by having a lower MDAD, but this means it cannot consistently distinguish models that differ by fewer than 12.5 points of accuracy on the full MMLU. When tinyBenchmart achieves an MDAD of 2 by selecting 500 examples from MMLU, random sampling also has an MDAD of 2.
Case study: MDAD better distinguishes If we were to only use Kendall's tau rank correlation, we would have trouble differentiating most methods from random sampling when selecting between 10 and 100 examples. Consider performance on MMLU for 1100 examples (Figure 4, left column). At these extreme dataset reductions, methods like tiny Benchmarks and stratified sampling by confidence have similar Kendall's tau rank correlations to random sampling, indicating that they all rank models equally well in the aggregate. But these methods have lower MDADs than random sampling, showing they can more reliably distinguish models that differ by fewer points of accuracy. This observation also holds for the other benchmarks.
from random at < 100 examples.
5.3 MDAD CAN INTERPRET WHICH MODEL COMPARISONS WILL BE PRESERVED
We have so far established that at small sizes, most
Cumulative distribution of 8B accuracy differences
20
Random (umflom) Stratitedd sampling
can distinguish only those models whose performance differ greatly on the full benchmark. This holds when comparing target models across various sizes and training regimes. But what about when comparing specific set of models that we ex- pect might have more similar perfor- mances? We consider a case study of 32 instruction-tuned 8B-parameter models on MMLU-Pro.
comparttons
100%
80%
MDAD
15
Andhor Poinss iny Benchhmarrs
60%
51%
40%
21%.
20%
10
25
250 500 1000 (0.5)
50
100
(0.25)
(.455)
1.75 15.2% (26.0%
Number of examples In micro-benchmark
Accuracy difference
When comparing 8B- SB-parameter instruction models on MMLU-Pro: model accuracies in narrow range, so nearly half of pairwise accuracy differences are less than 5 points (left), which is less than the MDAD for miero-benchmarks at small dataset sizes (right).
Figure 5:
Most models have accuracies between 27 and 40 on the full benchmark, yielding very low pairwise accuracy differences (Figure 5, left). MDAD computed from these 8B-parameter models (Figure 5, right) helps us understand when model comparisons are preserved on niicr-bbnnchmars of various sizes. All mnicro-benchmaree have an MDAD of 5 or more when selecting 10 or 25 examples. These micro-benchharks are not likely to reproduce the full benchmark's ' ranking on the 51% of model comparisons that differ by at most 5 points of accuracy. When 1,000 examples are selected, most micro-benchmarks have an MDAD of 2. These can consistently rank more models, though they will still not be able to consistently rank the 21% of model pairs that differ by no more than 2 points of accuracy.
MDAD explains why ranks stabilize when comparing models. Perlitz et al. (2024) observe that micro-bence seco-bbenchmarss can often consistently predict the top-ranked target models, even when selecting few examples. MDAD offers a mechanism by which this occurs: top-ranked models often have high pairwise accuracy differences from many models, and micro-benchmarke often agree with the full benchmark when comparing very different models. That is, top-ranked models often differ from many models by more than micro- Siccro-benchmark3 's MDAD. While all micro- betrro-bencmarkss have high agreement with the full benchmark for the top cop-eerfomming models once 25 or more examples are selected, they are less likely to agree with the full benchmark when comparing the models in the middle of the distribution that are closer in accuracy to each other (Figure 14, Appendix L). Figure (right) shows that when selecting 25 examples, all methods achieve an MDAD of 5 or more. For the top-ranked model, two-thirds of all model comparisons are above this MDAD. For a model in the middle of the distribution, only one third of all pairwise comparisons are above this MDAD. Appendix L shows similar results when restricting comparisons to various other sets of models, like instruction-tunee -unee 70B -ar-marmettrr models.
5.4
BECRO-BBNCCMARKS GENERALIZE TO NEW EVALUATION SETS
One potential advantage of miiro-benchhaar is that they may be able to exploit correlations between model predictions in order to predict how well models will do on fresh draws of the task. Or they could instead overfit their predictions the specific examples they were selected from. We test.

---

Published as a conference paper at ICLR 2026
Random (uniform)
Stratified sampling
Diversity
Anchor Points
tinyBenchmarks
10
30
MMLU-Pro
MDAD
20
20
20
20
10
10
10
12
10
25153 30% 20%
12%
40%
16%
24%
79050s 15%
24% 32% 10%
10% 20% 32%
Percent of subtask selected for micro-benchmark
Micro-benchmark on full dataset used to construct
Micro-benchmark ri on new draw of the task
Figure 6: MDAD is modestly higher on MMLU-Pro when predicting relative model performance on held-out draw of the task (dashed lines) than when predicting relative performance on the full dataset used to select the nicro-benchharks (solid lines). See Appendix J for results on other datasets.
whether either is the case by evaluating how well model comparisons micro-benchmare predict model comparisons on held-out unseen set that was not used to construct the micro-henchmark, We find almost difference in MDADs when predicting model performance on held-out set using berro-bbnncharks that select from entire benchmarks, indicating that these ciiro-benchmarkiie methods effectively generalize new draws of dataset (full results in Appendix I). However, when selecting secro-benccharks subtasks individually, we do find that micro- benchmarks are slightly less able predict model performance fresh draws of the subtask, as evidenced by higher MDADs (Figure 6). Anchor Points experiences the least increase in MDAD when moving from the full benchmark fresh draw of the benchmark. Diversity and Binyhenchmarks experience larger increases. In all cases, MDAD is high. See Appendix for full details and results.
DISCUSSION AND CONCLUSION
We have investigated how well various benehobbencm methods reproduce model performance judgments from full benchmarks using the meta-evaluation measures of agreement and Minimum De- tectable Ability Difference (MDAD). We find that when micro beero-bbnccharks produce model rankings with high aggregate correlation with the rankings from full benchmarks, they cannot always consis- tently distinguish model pairs with similar performance. Once enough examples selected allow for distinguishing model pairs with similar performance, random sampling is competitive with other methods. Our meta-evaluation measures can guide micro-benchharking method designers in building reliable and efficient model comparisons. While our experiments primarily focus on accuracy-basee evaluation, there exist straightforward extensions of MDAD other metrics, including ones used in open-ended generation. In future work, we plan to explore using MDAD to directly guide data selection strategies.
We hope our meta-evaluation measures can help practitioners select the right micro-benchmark size for the job, rather than going with a one-size-fits-al recommendation from prior work. If the goal is to produce ranking of models that may differ by, say, five points of accuracy to track finer gradations in model performance over the course of training, evaluation sets should be large enough to afford low MDAD. This is particularly relevant when seeking identify if model exceeds the current state-of-the-art, since each new model has historically achieved only incremental improvements in NLP (Card et al., 2020). However, if the goal is just get general sense of model performance, then even 10 examples chosen by Anchor Points or tiny Benchmanssrss from datasets like MMLU-Pro and BIG-bench Hard can suffice.
remain valuable tools for efficiently understanding model performance, but it is important to know the limits of that understanding.
REPRODUCIBILITY STATEMENT
We release source code for reproducing all experiments, including all implementation details for datasets and nhicro-bnncmmrkeee methods, on GitHub. Appendix E describes the micro- benchmarking methods in detail, and Appendix D describes experimental setup and infrastructure in greater detail. The GitHub repository also includes all intermediate micro-benchmartee results as well as the final results reported in figures throughout this paper.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported in part by funding from the Allen Institute for AI, an Intel Rising Star Award and a USC Women in Science and Engineering (WiSE) Gabilan Fellowship. A portion of this work was done while Swayamdipta was a visitor the Simons Institute for the Theory of Computing. We thank all members of the USC DILL Lab for constructive feedback, especially Matthew Finlayson and Sayan Ghosh. We also thank Kate Donahue, Varsha Kishore, Oliver Richardson, and Tejas Srinivasan for useful feedback.
REFERENCES
Rémi Bardenet, Subhroshekhar Ghosh, Hugo Simon-Onfroy,, and Hoang Son Tran. Small coresets via negative dependence: DPPs, linear statistics, and concentration. NeurlPS, 2024.
Meriem Boubdir, Edward Kim, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. Which prompts make the difference? Data prioritization for efficient human LLM evaluation. arXiv preprini arXiv:2310. 14424, 2023.
Li Cai, Kilchan Choi, Mark Hansen, and Lauren Harrell. Item response theory. Annual Review of Statistics and Its Application, 3(1)2977331, 2016.
Dallas Card, Peter Henderson, Urvashi Khandelwal, Robin Jia, Kyle Mahowald, and Dan Jurafsky. With little power comes great responsibility. In EMNLP, 2020.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot Arena: An open platform for evaluating Ilms by human preference. In ICML, 2024.
Jacob Cohen. The statistical power of abnormal-socia psychological research: review. The Journal of Abnormal and Social Psychology, 65(3):145, 1962.
Rohan Deb, Kiran Koshy Thekumparampil. Kousha Kalantari, Gaurush Hiranandani, Shoham Sabach, and Branislav Kveton. FisherSFT: Data-cfficient supervised fine-tuning of language models using information gain. In ICMI., 2025.
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In ACL, 2018.
Logan Engstrom, Andrew Ilyas, Benjamin Chen, Axel Feldmann, William Moses, and Aleksander Madry. Optimizing ML training with metagradient descent. arXiv preprint arXiv:2503.1375) 2025.
Kawin Ethayarajh and Dan Jurafsky. Utility is in the eye of the user: A critique of NLP leaderboards. In EMNLP, 2020.
Dante Everaert and Christopher Potts. GIO: Gradient information optimization for training dataset selection. In ICLR, 2024.
Riccardo Fogliato, Pratik Patil, Nil-Jana Akpinar, and Mathew Monfort. Precise model benchmarking with only few observations. In EMNLP, 2024a.
Riccardo Fogliato, Pratik Patil, Mathew Monfort, and Pietro Perona. A framework for efficient model evaluation through stratification, sampling, and estimation. In ECCV, 2024b.
Clementine Fourrier, Nathan Habib, Alina Lozovskay,, Konrad Szafer, and Thomas Wolf. Open LLM Leaderboard v2, 2024. URL https://hugginngface
spppccsopppnnn 11m--1mmmmm- leaderboard/
blog.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron. Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang. and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL http::://zzenddoorr/reeeeeee 12608602.
11

---

Published as a conference paper at ICLR 2026
Sayan Ghosh, Tejas Srinivasan, and Swabha Swayamdipta. Compare without despair: Reliable preference evaluation with generation separability. In EMNLP Findings, 2024.
Vipul Gupta, Candace Ross, David Pantoja, Rebecca J. Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using SMART filtering of benchmark datasets. NAACL, 2025.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández Marchant, Kevin Sheppard, Tyler Reddy,
del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, September 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
D. G. Horvitz and D. Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47(260);663 1952. doi: 10.1080/01621459, 1952.111534444 URL httph: www.wtonlnnnonnnnnommmmm wwtp://cccoccc/o///eeeee 01621459. 1 1952. 10483446.
http:://www...ttaooooooee/eeeoo/ 1080/
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in NLP. In NAACL, 2021.
Bernard Koch, Emily Denton, Alex Hanna, and Jacob Gates Foster. Reduced, reused and recycled: The life of dataset in machine learning research. In NeurlPS: Datasets and Benchmarks Track, 2021.
Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In ICML, 2021.
John Patrick Lalor and Pedro Rodriguez. py-irt: scalable item response theory library for python. INFORMS Journal Computing, 35(1):5-13, 2023.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. TMLR, 2023.
Nelson F Liu, Tony Lee, Robin Jia, and Percy Liang. Do question answering modeling improvements hold across benchmarks? ACL, 2023.
Lovish Madaan, Aaditya K Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. arXiv preprint 2tt.::2006.1022299 2024.
R.B. Nelsen.
Encyclopedia of Mathematics EMS Press, 2001.
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Scheuer and Leshem Choshen. Efficient benchmarking (of language models). In NAACL, 2024.
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: Evaluating LLMs with fewer examples. In JCML, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-pre -proof Q&A benchmark. In COLM, 2024.
Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P Lalor, Robin Jia, and Jordan Boyd-Graber. Evaluation examples are not equally informative: How should that change NLP leaderboards? In ACL., 2021.
12

---

Published as a conference paper at ICLR 2026
Michael Saxon, Ari Holtzman, Peter West, William Yang Wang, and Naomi Saphra. Benchmarks as microscopes: A call for model metrology. In COLM, 2024.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. TMLR, 2023.
Mirac Suzgun, Nathan Scales, Nathanacl Schärli, Scbastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V Le, Ed H Chi, Denny Zhou, al. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of ACL,, 2023.
Shir Ashury Tahan, Ariel Gera, Benjamin Sznajder, Leshem Choshen, Liat Ein Dor, and Eyal Shnarch. Label-efficient model selection for text generation. ACL, 2024.
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, and Samuel Bowman. Comparing test sets with item response theory. In ACI, 2021.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor Points: Benchmarking models with much fewer examples. In EACL, 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, et al. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. NeurlPS Datasets and Benchmarks Track, 2024.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS:
Selecting influential data for targeted instruction tuning. In ICML, 2024.
Gregory Yauney and David Mimno. Stronger random baselines for in-context learning. In COLM, 2024.
Qinyuan Ye, Harvey Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? A case study on BIG-bench. Findings of EMNLP. 2023.
Guanhua Zhang, Florian E Dorner, and Moritz Hardt. How benchmark prediction from fewer data misses the mark. In NeurlPS, 2025.
Hongyu Zhao, Ming Li, Lichao Sun, and Tianyi Zhou. BenTo: Benchmark task reduction with in-context transferability. ICLR, 2025.
Vilém Zouhar, Peng Cui, and Mrinmaya Sachan. How to select datapoints for efficient human evaluation of NLG models? TACL, 2025.
13

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
Efficient evaluation. Benchmark datasets have been used to evaluate machine learning models for decades (Koch et al., 2021). Though benchmarks are not sufficient to fully map capabilities (Ethayarajh & Jurafsky, 2020; Kiela et al., 2021; Chiang et al., 2024), they remain an important tool for comparing models (Saxon et al., 2024). NLP benchmarks have grown ever-larger as LM capabilities have expanded (Srivastava et al., 2023; Liang et al., 2023). small subset of a benchmark for evaluation with the goal of estimating a model's performance on the full benchmark. This is done by exploiting correlations in model's predictions across examples (Fogliato et al., 2024b) and correlations across multiple models' predictions (Vivek et al., 2024; Ye al., 2023; Liu et al., 2023), by training instance difficulty models using Item Response Theory (Polo et al., 2024; Vania et al.. 2021; Rodriguez et al., 2021), or even by deduplication (Gupta et al., 2025). We study the reliability of mairro-bnnchaate that summarize performance across an entire dataset, though other work focuses on estimating performance across subtasks (Fogliato et al., 2024a). In contemporaneous work, Zhang et al. (2025) give complementary results on the efficacy of random sampling for predicting individual model accuracy. They find that a regression based extension of random sampling has competitive mean estimation error with other mhicro-benchmarinng methods when selecting 50 data points. They also study extrapolating from lower-accuracy source models to high-accuracy target models, finding that random sampling and a strategy based on augmented inverse propensity weighting achieve lower mean estimation error than other methods.
setro-bbnccmarking methods select
Farther afield, active testing instead selects test instances in an online per-model manner (Kossen et al., 2021). Still other works use model- proxies to select unlabeled examples for annotation (Tahan et al., 2024; Zouhar et al., 2025). Subsets of training data can be selected using gradients (Everaert & Potts, 2024; Xia et al., 2024; Engstrom et al., 2025) information gain (Deb al., 2025), though such methods have not yet been applied to evaluation data selection.
Evaluation reliability. For classification, larger evaluation datasets yield more reliable model comparisons (Shalev-Shward) & Ben-David, 2014; Dror et al., 2018; Card et al., 2020) and are more robust to some forms of dataset reuse (Yauney Mimno, 2024). Card et al. (2020) use statistical power to estimate the minimum detectable effeet size afforded by benchmark. though this approach requires assumptions about independence of examples, generalization error, and expected per- per-example agreement between model predictions. In contrast, our approach directly estimates the probability that pairwise model comparisons on a target models. Perlitz et al. (2024) also frame reliability consistency over random choices in order to study the reliability of individual model error rate and model rankings over evaluation choices, but they do not identify which model comparisons are preserved by smaller datasets. Madaan al. (2024) measure the variance in performance across random seeds and find that some micro-bor ceerro-benchmarkingg methods increase variance. Our approach offers an interpretation of when variance impacts model comparisons. Other work considers the reliability afforded by LM generations for human evaluation (Ghosh et al., 2024; Boubdir et al., 2023).
agree with the full benchmark for a set of
B DETAILS OF MICRO-BENCHMARKEN METHODS
Anchor Points. We evaluate the "Anchor Points Weighted" method from Vivek et al. (2024). First, correlation matrix C' between examples is constructed, where entry Ci, is function of the correlation between examples i and j across all source models. k-medoids is used to select examples that maximize the correlation between the selected and the remaining examples. To estimate a target model's 's performance using these examples, the method averages the correct class probability of each example, weighted by the size of each example's corresponding cluster.
tiny Benchmarks. We evaluate the "IRT" method from Polo et al. (2024), which trains an Item Response Theory model using py-irt (Lalor & Rodriguez, 2023) from source model predictions to produce embeddings for examples that are then clustered. After ppper-paraette sweeps, use 10 dimensions for the IRT embeddings, and for IRT model training we use a learning rate of 0.1 and 2000 epochs.
14

---

Published as a conference paper at ICLR 2026
Stratified sampling by confidence. We implement variant of the stratified random sampling proposed by Fogliato et al. (2024b). We adapt the algorithm to work with multiple source models by taking the mean of model confidence across all source models. We perform k-means clustering into 10 strata based on these mean confidences. From each cluster, we uniformly at random sample a number of examples proportional to the size of that cluster. From model performance this chosen subset of examples, we use the Horvitz- Horvitz-Thompson (HT) estimator (Horvitz & Thompson, 1952), as in Fogliato al. (2024b), arrive an estimate model performance on the full benchmark.
Diversity. We implement method that selects diverse examples. The setup is similar to Anchor Points: each example has an embedding where each coordinate source model's confidence the correct class. Rather than using k-medoids to select examples, we use the sampler from Bardenet al. (2024) to select diverse set examples this embedding space. Surprisingly, dimensionality reduction of embeddings as few as 4 dimensions does not degrade performance.
C DIFFERENT AGREEMENT THRESHOLDS FOR MDAD
Figure shows comparison of 0.7, 0.8, 0.9, and 0.95 as the threshold of agreement in Equation 5. All our other experiments use a threshold of 0.8. Agreement between a full benchmark is higher for larger differences in model performance As the agreement threshold for MDAD increases, the main effect is that MDAD also increases. All methods tend to experience this MDAD increase, the overall results for different thresholds qualitatively similar.
and the
MMLU
MMLU-Pro
BBH
GPQA
190
10 Wirerkoeii
200 11991 1981
1400
100
200
000
352
(3201 1181 17700
300
1000
100 2008
255
11:01
500
110
0380 11.71
122
2.0
Number of examples selected for mncrr-bbenhhmake
Random (uniform)
Stratified sampling (confidence)
Anchor Points
Random (subtask stratified,
Diversity
tinyBenchmarks
Figure 7: MDADs when using different thresholds for agreement qualitatively similar. The second row of MDAD panels with a 0.8 threshold are the same as in Figure 4. Error bars represent 95% bootstrap confidence intervals over 50 trials.
15

---

Published as a conference paper at ICLR 2026
D FULL EXPERIMENTAL SETUP AND COMPUTING INFRASTRUCTURE
For most experiments, we construct micro-benchharkke using 300 source models and evaluate them using 50 target models. We average all evaluation metrics over 50 runs of random partitions into source and target models, For the experiments in Section 5.2 and Appendix L, we choose 300 random source models and evaluate with fixed set of von-overlappingg target models. Experiments were implemented using NumPy (Harris et al., 2020). We report calculated MDADs to the nearest 0.5.
Models. For MMLU, we use the results of the 366 models from the Open LLM Leaderboard (Fourrier et al., 2024), in Polo et al. (2024). For the other benchmarks, we use cached model predictions from the 470 models tagged as official on the Open LLM Leaderboard v2 (Fourrier et al., 2024). For each of these benchmarks, subtasks, yielding 447 models for MMLU-Pro, 409 models for BBH, and 420 models for GPQA. Our evaluations include 101 models with 0.5-3B parameters and 39 models with 70B+ parameters. The largest model included our evaluations is 141B.
include models that had full per-example evaluations
all
Computing infrastructure. Micro-benchmarking methods and analysis were run on cluster node with GeForce GTX 1080 Ti GPUs and Macbook Pro with an Apple M3 Pro processor and 18GB of RAM.
Runtime. Table 2 gives the average time (in seconds) for each method on each benchmark to perform one trial with 300 source models. Our full ex- periments are 50 trials across 7 dif- ferent settings for number of source models, for total of 91.05 hours.
Micro-beachmarring method
MMLU
MMLU-Pto
BBH
GPQA
Random (uniform) Random Stratified sampling (confidence)
3.1
3.5
3.8
0.8
stratified, equal)
3.0
2.8
2.9
0.5
Ancher Points liny Benchmarks Diversity
4.7
1.1
150.9
164.5
58.9
7.5
315.0
266.2
207.0
16.8
Table 2: Average time (seconds) for completion of one trial.
E QUANTITATIVE ERROR ANALYSIS OF MDAD
Our results throughout the paper estimate MDAD across 50 trials of our meta-evaluation experiments. Table shows estimated MDADs and 95% confidence intervals for up to 100 trials for uniform random sampling, Anchor Points, and Binhhennsmarks when selecting 50 and 100 examples from MMLU-Pro. Estimated MDADs have stabilized by 50 trials.
Table 3: MDADs with 95% confidence intervals for up to 100 trials for uniform random sampling, Anchor Points, and tiny Benhharssssrks when selecting 50 and 100 examples from MMLU-Pro.
examples
100 examples
Number of trials
Random (uniform)
Anchor Points
tiny Benchmarks
Random (uniform)
Anchor Points
tiny Benchmenssrks
10
5.6 i: 34
2.90
2.1
3.6 1 3.1
4.6 2.7
2.8 2,3
3.6 1.7
30
6.1 + 1.
3.50
1.2
5.2 + 1.7
4.5 + 1.6
3.0 1.6
3.6
1.4
6,2
3,50
5,4 1.2
4.4 1.3
3.4
4.0
50
6.3
4.10
5.4 1.8
4.4 1.3
3.6 1.1
3,8
6.5 1.0
4.10
54 1.2
4
3.7
3,8
6.6
400
54+
4.6
3.1
0.7
80
6.6
4,00 1.1
5.3 1.0
4.7
3.6
3.70.7
6.6
5.2
4.
100
6,6 0.9
4,10 1.2
1.2.088
4.4
3.7 1..1.99
3,7 0.6
16

---

Published as a conference paper at ICLR 2026
F DIFFERENT BUCKET RESOLUTIONS FOR MDAD
Figure shows a comparison of 0.25, 0.5, and 1.0 as bucket resolutions when calculating MDAD (Equation 5). All of our other experiments use resolution of 0.5. These different resolutions result in very similar MDADs in most cases. The one exception is when selecting 10 examples from GPQA with random sampling: agreement values in the larger buckets are low that bucket has an agreement value greater than 0.8.
MMLU
MMLU-Pro
BBH
GPQA
639 p300
(5391
0.712
0130
2008
390 399 0.54
14.40
(1.00)
8800
10
01.00
Number of examples selected for micro-benchhmrkk
Random tuniform]
Stratiried sampling (confidencc)
Anchor Points tinyBenchmanks
tsubtask equall
Diversity
Figure 8: MDADs when using different bucket resolutions for agreement are qualitatively similar. The middle row of MDAD panels are the same as in Figure 4. Error bars represent 95% bootstrap confidence intervals over 50 trials.
17

---

Published as a conference paper at ICLR 2026
G FULL RESULTS
Figure is an expanded version of Figure with full correctness curves for all datasets. Figure 10 gives results when selecting a fixed percentage of examples from each subtask in a benchmark.
Figure 16 (at the end of the appendix) gives full results for all benchmarks when selecting a fixed number of examples from each benchmark. Figure 17 (also at the end of the appendix) gives full results for all benchmarks when selecting fixed percentage of examples from each benchmark.
MMLU agreermnn
D
13.322
(N) eacenplee (820)
-4 ricepples (6.45)
-N exampptd (8.8%]
M] -1-100 acamples (1.2%)
[M 202 masmpple (8.7%)
MMLU-PPO Agreement
15 exompkes (8.34)
examples (0.99)
15 oxamples 11.750
11-100 commlles 13.%
coamptes (8.7%)
Agremment
E S.S.
MOLD
GPQA Agreerment
11:31
1211
Accurscy difference between models on full benchmark
Diversity
Figure 9: Expanded version of Figure with all datasets. Comparing six micro-benchmarking approaches four evaluation benchmarks. y-axis reports agreement, the probability that micro- benchmark agrees with the full benchmark when comparing two models, as function of how much those models differ on the full benchmark (r-axiss). The rightmost column summarizes agreement curves using MDAD. Error bars represent 95% bootstrap confidence intervals over 50 trials.
MMLU
MMLU-
GPQA
790
HOAD
2505
200%
125
35%
Fraction selected milro-benchmarrk subtask
Random (unifor)
sampling (confidence)
Anchor
Random (sublask stratified, equal)
Diversity
tinyBenchmarrks
Figure 10: Results when selecting fixed percentage of examples from each subtask of benchmark. Top row: Mean estimation error. Middle row: Kendall's tau rank correlation. Bottom row: Minimum Detectable Accuracy Difference (MDAD, ours). Results are averaged over all subtasks. Error bars represent 95% bootstrap confidence intervals over 10 trials.
18

---

Published as a conference paper at ICLR 2026
H MDAD CAN ANALYZE METHODS THAT SELECT ENTIRE SUBTASKS
Throughout this paper, we evaluate nicro-benchmarking methods that select examples without regard to which subtasks of the original benchmark they are from. But our methods and proposed meta-evaluation measures generalize to other kinds of micro-benchmarking methods as well. For example, BenTo is paicro-benchmarkiigg method that selects whole subtasks for smaller evaluation sets by estimating task transferability (Zhao et al., 2025). When selecting micro benchmark from MMLU, BenTo selects 801 examples three subtasks of MMLU for vicro-benchmarking. We calculate MDAD and other meta-evaluation measures for this selected subset of examples, as well as uniform random sampling for selecting 801 examples (Table 4). Both BenTT and Random achieve very similar results. Table 4: Results for the BenTo nicro-benchmarking method.
Method
BenTo Mean estimation error
Kendall's ' tau rank correlation
MDAD
1.34 + 0.39
0.9332 0.0153
1,5 0.5
Random (uniform)
1.30 = 0.38
0.9166 0.0175
1,5 + 0.5
I GENERALIZING TO NEW TASK DRAWS WHEN SELECTING FROM ENTIRE BENCHMARKS
Figure 11 gives results for meta-evaluating berrharbnn with respect to held-out set when selecting from the benchmark as a whole. MDADs can increase by up to 0.5 points in this setting.
Kamdbes Jaaturrl
Rynatee
Mostihee (uethireed)
Swenity
Rester
NKLU
MINLL
Picre www.leeaaaaacccccmc en coonE ncele cremrtt
Figure 11: For all methods MMLU, MMLU-Pro, and BBH, MDAD is nearly the same when predicting relative model performance on a held-out draw of a task (dotted lines) as when predicting relative performance
the full of examples used to select the hiero-benncharks (solid lines). For GPQA, a much smaller dataset, there is a larger gap performance. Error bars represent 95% bootstrap confidence intervals 50 trials.
19

---

Published as a conference paper at ICLR 2026
J GENERALIZING TO NEW TASK DRAWS WHEN SELECTING FROM SUBTASKS SEPARATELY
Figure 12 is expanded version of Figure with results for comparing micro-henchmarking methods on a held-out set when selecting from subtasks individually.
banchmark on datasee used lo corstructit
Nicro Senchhark on new draw of the task
Figure 12: For methods, MDAD is modestly higher when predicting relative model performance on a held-out draw of task (dotted lines) than when predicting relative performance on the full set of examples used select the herro-benchmarks (solid lines). The MMLU-Pro row the same as in Figure 6. Error bars represent 95% bootstrap confidence intervals over 10 trials.
Table gives changes in other meta-evaluation measures in this setting for MML.UProo. When moving from the original draw of the task new draw of the task, mean estimation error increases by less than point, and Kendall's tau rank correlation decreases by less than 0.05 for all evaluated micro-benchmmrkinn methods.
Table 5: Changes in mean estimation error and Kendall's s tau rank correlation for MMLU-Pro when generalizing to new draws of the task, averaged across all selected micro-benchmark sizes (corresponding to the MDADs Figure 6, which are included here for reference).
Method
Mean incresse in MDAD
Mean increase in mean estimation enor
Mean decrease in Kendall's is tau rank conrelation
Random (uniform)
1.18
D.65
0.039
Stratified sampling (confidence) Divensily Anchor Points
0.41
0.041
0.37
0.51
0.014
RnyRenchmarks
1.07
0.74
0.038
We also find that comparisons between models at the same scale are not always preserved, but becro-bbnnchmrks can still consistently distinguish the performance differences between smaller (7B) models and larger (70B) models. In Section 5.4, we find that the MDAD of beich-bbenchmrkinn methods can increase on a new draw of the dataset by up to 1.2 points when selecting examples from individual subtasks (Figure 6). When selecting 100 examples from MMLU-PE all micro- benchmarking methods have an MDAD of at most when generalizing to a new draw of the task. When comparing models in the 6B-8B range each other, 33.3% of comparisons will not be preserved by the miero-benchmarks because they involve accuracy differences below the MDAD. When comparing models in the 68B-72B range each other. 35.7% of comparisons will not be preserved. But the miiro-benncharks can still consistently distinguish between small and large models because only 6.9% of those comparisons have accuracy differences less than 5.
20

---

Published as a conference paper at ICLR 2026
K INCREASING NUMBER OF SOURCE MODELS HAS MODEST EFFECT
So far we have been evaluating how MDAD decreases as the number selected examples increases. For micro-benchmmrking method designers, key parameter is how many source models use when selecting the micro-benchmark. Whereas previously we have examined performance with 300 source models, Figure 13 shows aggregate results for all benchmarks for many different numbers of source models used to select the micro-benchmate selected, increasing the number of source models provides only modest improvement in model distinguishability for any of the methods. The effect of more source models is most pronounced for Anchor Points on BBH and GPQA when moving from 10 50 source models, though more source models do not yield further improvements. For all methods, increasing the number of source models is not as effective as evaluating on more examples.
For nearly all datasets and numbers of examples
335 16.7%1
01-300 16.99) 31-3200 13U.5
Si-Deerrerr3 231
51-191 aarole (1.9%)
20- cooppti (1.5)
-38 scarpla (33m5)
(unifem)
Sratiled camsling (oneiiencc)
Styfeechmanc
Figure 13: Allowing micro-benchmarking methods access to increasing numbers source models with full predictions does not improve MDAD as much as evaluating even slightly more examples, as indicated by horizontal lines in nearly all panels. Random sampling provided as a baseline, as does not rely any source models. Note that the r-axis is number of source models, not number of examples selected, Error bars represent 95% bootstrap confidence intervals over 50 trials.
21

---

Published as a conference paper at ICLR 2026
L
COMPARING SPECIFIC SIZES OF MODELS
Probsbility micro-benchmark agresing with benchmark when comparing models
Randim (uniform))
Stratifled sampling (cendi dence) Anchor points (welghtedd)
tinBBencharks
08
10
10
Piu INLARR
07
06
-03
274 0R4 AR, 276 4TG
PPV 47% 4%A 3% 201
Number of examples selecttdd for melcr--bernt carre--bcchmaakkkkkkkkk
Figure 14: When comparing 8B 8B-parameter instraction-tuned models on MMLU-Pro: per-model agreement with the full benchmark is lower for the models in the middle of the accuracy distribution that have more similar accuracies to many models.
10.55
Meddl accerastes
14.00
Acceroes defomeces
15
Acourddy
Acaray diferance
soacted recro-aaa
(a) MMLU-Pro, 70B-parameter instruct models.
HOND
20
125
15%
15%
00%
028
50
selvcted reore sondieoss
Mamber sanmoiss
(b) BIG-bench Hard, 7B parameter instruct models.
20073
22.0%
Medel accaracis
22.39
Accuracy drfemaces
20
Tiratie earying
10.0%
11.0%
199
Aicesa
(c) BIG-bench Hard, 70B-parameter instruct models.
Figure 15: Pairwise model comparisons are often between models with similar accuracies when comparing specific classes of models.
22

---

Published as a conference paper at ICLR 2026
MMLU-Pro
J1.2.00
Veater crampte reoosanddees
Sancen
BBH
MAurE readets
SOARCE realds
208 searce redes
1.0 wouce reddel
200 source modlds
smarce modh
380
18.
Rastrve hareek mtd.el. Cepa
Demrly
Avche Furt
tegfeviinns
Je
Figure 16: Full results when selecting fixed number of examples from across entire benchmarks. The column with 300 source models is the the results presented in Figure Top row: Mean estimation error. Middle row: Kendall's tau rank correlation. Bottom row: Minimum Detectable Accuracy Difference (MDAD, ours). Error bars represent 95% bootstrap confidence intervals over 50 trials.
23

---

Published as a conference paper at ICLR 2026
MMLU
2130
Romber
MMLU-Pro
SoUrC roddes
50 SOUIE roddo
150 SPAICA Todelt
120 source modil
EOUICH TOORS
250 EORIC Tod48
320 rodes
205
300
0010
Tascen 1n scat
2caefe com Sandderon
Antreooor
fasky (antom
Baw.. scat
fonfderon
Antreroord
GPQA
Figure 17: Full results when selecting a fixed percentage examples from across entire benchmarks. Top row: Mean estimation error. Middle row: Kendall's tau rank correlation. Bottom row: Minimum Detectable Accuracy Difference (MDAD, ours). Error bars represent 95% bootstrap confidence intervals over 50 trials.
24


Reviews:
[{"review_id": "cUEVwv6pax", "reviewer": "Reviewer_3saE", "summary": "The paper introduces an evaluation framework for micro-benchmarking. Given a full benchmark $D_{full}$ and a selection strategy *S*, the proposed metric evaluate how faithfully the ranking obtained on the subset $D = S(D_{full}) \\subset D_{full}$ reflects the ranking on the full benchmark $D_{full}$. They define the *Minimum Detectable Accuracy Difference* (MDAD), which is the smallest performance gap on the full benchmark for which pairwise model rankings on a micro-benchmark remain consistent with those on the full benchmark $D_{full}$. Their findings are multiple. For example, with a budget of only 10 examples,  none of the selection strategies they consider can correctly rank all pair of models which differs by 3.5 points on MMLU-Pro with at least \"80% confidence\". Moreover, as the size of the micro-benchmark increases , random selection increasingly becomes competitive with more sophisticated methods.", "questions": "- Why is the **conclusion** merged with the **discussion**?\n- Why is **Anchor points** more performing than the other strategies when selecting only few examples ($|D| \\leq 100$) and more generally why is it that it better \"tells apart\" models with a small performance difference better than the other (this advantage disappears as the number of selected examples |$D$| increases)?\n- L235: `We use 50 trials, each with a partition of data points and models.` Does it mean you consider 50 seeds and for each seed you partition your data into two sets (one for selection and the other for generalization)?\n- Figure 3 (right)/Figure 4: Why does **Anchor points** stagnates (in terms of MDAD) and performs worse than the other strategies when the number of selected samples reaches a certain point (typically $\\geq 500$)?\n- Line 458-459: What could explain the fact that selecting micro-benchmarks of subtasks independently results in some sort of slight overfitting?\n- What is the impact of varying the confidence of the ranking in the MDAD from 80% to say 95%? \n- How expensive/time-consuming is it to compute the MDAD?\n- Do you think it is possible to adapt the MDAD to buckets of relative/absolute performance difference in terms of scores given as percentage ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a theoretical approach for evaluating micro-benchmarking. It is important in practice, since assessing large models across multiple full-scale benchmarks can quickly become prohibitively expensive. The proposed approach is valuable in that it provides a way to determine whether given a selection strategy and a budge of $n examples, one can reliably distinguish between models whose performance differs by a specified range `[a, b]`.\n- The experiments cover multiple selection strategies, benchmarks and setups which help broadly assessing how their metric behaves.", "weaknesses": "- The authors appear to derive MDAD by applying statistical testing principles to micro-benchmarking (somewhat analogous to subset selection) but this is never made explicit. The paper does not clearly define hypotheses (e.g., $H_0$,  $H_1$ ) or discuss error types, leaving the methodological novelty of their approach uncertain. While the formulation itself may be original, the overall idea seems less so.\n- Despite the discussion in **Section 5.1**, the benefits of MDAD over metrics such as *mean estimation error* and *rank correlation* remain unclear. Each strategy has its own strengths and weaknesses, but the provided examples fail to convey the broader implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yaqO52QJnZ", "reviewer": "Reviewer_h3U2", "summary": "The paper presents a comprehensive study on the existing techniques for language model micro-benchmarking, which aims to improve the evaluation efficiency by only selecting a representative subset of the whole benchmark for evaluation. Specifically, this paper introduces a meta-evaluation measure for micro-benchmarking which investigates how well a micro-benchmark can rank two models as a function of their performance difference on the full benchmark. Evaluation shows that no micro-benchmarking method can consistently rank model pairs when the performance difference is small and, when the performance difference becomes larger, random sampling has a comparable performance with other existing micro-benchmarking techniques.", "questions": "None beyond the above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper explores an important and interesting research direction.\n- The paper is well-written and easy to follow.\n- This paper proposes a novel metric, namely MDAD, to evaluate the effectiveness of micro-benchmarking, which is more robust and reliable than existing metrics such as mean\nestimation error and rank correlation.\n- The evaluation has involved a large number of comprehensive experiments to study the performance and impact of existing micro-benchmarking techniques.", "weaknesses": "- While the paper has conducted a very comprehensive evaluation, there’s a lack of concrete case studies in the paper to provide a more intuitive discussion of the results. The readability of the paper can be further improved with more case studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1K4XOuHh3l", "reviewer": "Reviewer_1D3U", "summary": "The paper presents Minimum Detectable Ability Difference (MDAD), a meta-evaluation measure. It uses MDAD to assess different micro-benchmarking methods, compare MDAD to other meta-evaluation measures, and analyze micro-benchmarking efficiency-reliability tradeoff.", "questions": "The results echo a broader message of \"the bitter lesson\". Specified methods work well in small-scale settings (model sizes, number of examples, etc.) but do not generalize to larger scales. What is your perspective from that context? What is its relation to the micro-benchmarking methods?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written and structured.\n2. The proposed measure is well motivated and intuitive.\n3. The experiments are well designed, including a few benchmarks, a large number of models, across different subset sizes.\n4. The figures are effective in showcasing the relevant results.\n5. Sections 5.1 and 5.3 are doing a good job and demonstrating why MDAD adds value above other methods.", "weaknesses": "1. Figures are hard to read.\n2. The conclusions themselves are relatively known and expected. This is reasonable. The new measure and quantifiable results are new and interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPcVxmFUmG", "reviewer": "Reviewer_E4xW", "summary": "This work investigates the reliability of micro-benchmarks, small subsets of larger evaluation datasets, in faithfully ranking model performance. The authors introduce a new meta-evaluation metric, MDAD (Model Disagreement Agreement Distance), which quantifies how consistently a micro-benchmark preserves the pairwise ranking of models relative to the full benchmark. Unlike other meta-benchmarking strategies, the authors claim that MDAD provides a finer-grained view of evaluation stability, revealing when and why model rankings diverge under extreme data reduction. By analyzing MDAD across various subset selection strategies, the study highlights the trade-offs between sampling efficiency and ranking fidelity, offering practical insights into how different sampling techniques perform as dataset size decreases.", "questions": "* What potential applications beyond evaluation do you foresee for the proposed metric? Could MDAD, or a variant of it, be extended to support data selection strategies for supervised fine-tuning (SFT) or other training workflows?\n* Which experimental assets will be publicly released, for example, code, sampled subsets, or analysis scripts? Please provide specific details regarding the planned scope and accessibility of these materials.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The experiments demonstrate MDAD’s potential to offer deeper insights into selecting the most reliable micro-benchmarking approach (Fig. 3).\n* The experiments further show that MDAD provides more fine-grained and informative analysis than existing meta-evaluation metrics for micro-benchmarks (Fig. 4).\n* The authors appropriately situate their work within the broader literature, clearly articulating how their contributions extend prior research.", "weaknesses": "* Some figure titles and labels are difficult to read; improving their legibility would make the results clearer and more accessible.\n* Given that meta-evaluations are conceptually dense and rely heavily on prior work, the paper would benefit from a thorough proofreading and clarity-focused revision to improve readability.\n* Including comparative tables or summary diagrams that explicitly contrast this meta-evaluation method with existing approaches would help highlight its unique advantages.\n* The paper’s contribution is summative and complementary, it extends and provides additional insights useful to decide on existing micro-benchmarking techniques rather than proposing an entirely new framework.\n* The authors provide source code in the supplementary materials, but it remains unclear whether these materials will be publicly released to support transparency and reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)