DISCO: Diversifying Sample Condensation for Efficient Model Evaluation
Authors: Alexander Rubinstein
Benjamin Raible
Martin Gubri
Seong Joon Oh
Venue: ICLR 2026 Poster (2026)
Keywords: Efficient evaluation
Large Language Models
anchor point
fingerprint
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Evaluate your LLMs on benchmarks like MMLU at 1% cost

Abstract: Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eval and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. To address the growing cost of standard evaluation, new methods focused on efficient evaluation have started to appear. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples is not essential; what matters is to select samples that maximise diversity in model responses. Our method, **Diversifying Sample Condensation (DISCO)**, selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoretical view, inter-model disagreement provides an information-theoretically optimal rule for such greedy selection. **DISCO** shows empirical gains over prior methods, achieving state-of-the-art results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC.

Published as a conference paper at ICLR 2026
DISCO: DIVERSIFYING SAMPLE CONDENSATION FOR EFFICIENT MODEL EVALUATION
Alexander Rubinstein
Benjamin Raible
Martin Gubri²
Scong Joon Oh
Tübingen Al Center, University of Tübingen Parameter Lab
Project Page
DISCO Codebase
ABSTRACT
Evaluating modern machine learning models has become prohibitively expensive. Benchmarks such as LMMs-Eva and HELM demand thousands of GPU hours per model. Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact. To address the growing cost of standard evaluation, methods focused on efficient evaluation have started appear. The typical approach follows two steps. First, select an anchor subset of data. Second, train a mapping from the accuracy on this subset to the final test result. The drawback is that anchor selection depends on clustering, which can be complex and sensitive to design choices. We argue that promoting diversity among samples not essential; what matters is to select samples that maximise diversity in model responses. Our method, Diversifying Sample Condensation (DISCO), selects the top-k samples with the greatest model disagreements. This uses greedy, sample-wise statistics rather than global clustering. The approach is conceptually simpler. From a theoret- ical view, inter-mode, disagreement provides rule for such greedy selection. DISCO shows empirical gains over prior meth- ods, achieving state-of-the-a results in performance prediction across MMLU, Hellaswag, Winogrande, and ARC.
cooomationn-hhoretiiee
optimal
INTRODUCTION
Test set samples
Model evaluation is becoming increasingly costly. Models have grown in size, which makes each inference expensive. Recent scaling of test- time computation has further raised the cost per task. End-user requirements have also broad- ened, covering both the content of the output and its style (Wang et al., 2018; 8; Liang al., 2022; Kim et al., 2023; Zhang et al., 2024). As result, evaluation on modern benchmarks often requires hundreds to thousands of GPU hours. For instance, LMMs-Eval can take between 30 and 1400 hours on x A100 GPUs (Zhang et al., 2024). HELM requires more than 4000 GPU hours (Liang et al., 2022).
Models
Uniform model outputs
Diverse outputs
Low information content
High info content
More eval budget $$$
Less eval budget $
Figure 1: Imbalance. More evaluation budget is spent on less informative samples in test sets.
Several efficient evaluation approaches have emerged. A common framework works two parts: subset selection and performance prediction. The first part selects a static subset of anchor points from the evaluation dataset. The second part predicts full benchmark performance by extrapolating from accuracy on this subset. To select anchor points, existing methods often rely clustering, Samples are grouped by the similarity of responses they induce a set of reference models (Vivek et al., 2023; Polo et al., 2024). Variants of this framework include dynamic anchor selection (Hofmann et al., 2025), modified prediction models (Kipnis et al., 2024), and new benchmarks for method comparison (Zhang al., 2025).

---

Published as a conference paper at ICLR 2026
We seek to improve both parts of this framework. For subset selection, we argue that diversity among samples is not essential. What matters is diversity in model responses. We prove that inter-model disagreement is the most informative signal for estimating benchmark performance perffrrmce when the goal is to differentiate and rank models (Proposition 1). Evaluation should therefore focus on samples that elicit varied responses (Figure 1). For performance prediction, argue that existing methods add unnecessary complexity by estimating hidden model parameters before predicting test performance (Polo et al., 2024; Kipnis al., 2024). We instead propose a direct route. Model signatures, defined as the concatenation of outputs on the selected subset, serve as inputs to simple predictors of benchmark performance. This framework is simpler, yet matches and surpasses more complex alternatives.
We validate these ideas through Diversifying Sample Condensation (DISCO). DISCO selects a small, informative subset of evaluation samples by focusing on model disagreement. Disagreement is measured by predictive diversity scoring (PDS, Rubinstein et al. (2024)), originally proposed for -uttr---iittibutionn detection. A simple metamodel then predicts benchmark performance directly from the model signatures this subset. We evaluate DISCO in both language and vision domains. On MMLU, for example, DISCO reduces evaluation cost by 99.3% (see Appendix B.4) with only 1.07 percentage points of error. Compared with prior methods such as Anchor Points (Vivek et al., 2023), SinyBenchmark, (Polo et al., 2024), and Metabench (Kipnis et al., 2024), DISCO achieves a stronger efffciency-preeeeee trade-off.
2 RELATED WORK
We review prior work relevant to our approach. We first highlight the escalating cost of evaluation for contemporary large models and motivate the need for efficiency. We then survey prior attempts at efficient benchmarking, covering instance and task reduction techniques. Finally, we describe our novelty and contributions.
Cost of evaluation. The evaluation of modern large models is currently driven by increasingly sophisticated benchmarks assessing a wide array of capabilities, from the foundational GLUE (Wang al., 2018) and the comprehensive HELM (Liang et al., 2022) LMMs-Eval for multimodal models (Zhang et al., 2024), the diverse BIG-bench (Srivastava et al., 2022), Prometheus for measuring diverse LLM capabilities (Kim al., 2023), and GAIA for general Al assistants (Mialon et al., 2023). This progress comes at an escalating cost: models have grown significantly in size, making each inference step more eesourcc-intensive while the scaling of test-time computations has dramatically increased the per-task evaluation costs. Furthermore, end-user requirements have diversified to encompass not only output content but also style and manner. Consequently, a single evaluation on modern benchmarks can demand hundreds to thousands of GPU hours. For example, LMMs-Eva can require between 30 and 1400 hours on 8 x A A10 GPUs per model (Zhang et al., 2024; Polo et al., 2024), and HELM evaluations can exceed 4000 GPU hours per model (Liang al., 2022; Polo et al., 2024).
Label-efficient evaluation. In the LLe context, labelling test set used to be cost bottleneck for evaluation. In this context, the concept of "active testing" has been explored, where labelling budget is maximally assigned to information-rich samples (Majumdar& & Niksic, 2017; Ji et al., 2021; Deng& & Zheng, 3021; Kossen et al., 2021; Hu et al., 2023; Kossen et al., 2022; Huang et al., 2024; Fogliato et al., 2024). In our case, we are concerned with the inference costs of evaluation. As such, active testing approaches are not directly applicable, as they require a full inference over the test set to identify informative samples label.
Efficient benchmarking, In the LLM era. benchmarks have diversified to measure multiple capa- bilities and styles of model behaviours. Researchers have proposed strategies to build an efficient benchmark in the first place (Perlitz et al., 2023; Rädsch et al., 2025). There were attempts to compress multiple benchmarks, measuring an array of capabilities of LLMss into a single one by eliminating redundancies (Kipnis et al., 2024; Zhao et al., 2024; Yuan et al., 2025). Others have focused on selection of small, informative subsets, also known as "Anchor point" approaches (Vivek et al., 2023; Polo et al., 2024; Li et al., 2025; Gupta et al., 2025). Given an entire dataset, they compute a small subset of data points according to the representativeness criterion, determined through the correctness patterns of a large number of source models. Subsequently, target model performance is estimated based on weighted accuracy computed on the selected subset. In particular, tiny Benchmacssssss (Polo al., 2024) have adopted Item Response Theory (IRT) (Lord & Novick, 2008)
2

---

Published as a conference paper at ICLR 2026
Evalustion
script
Evaluation dataset
Outputs Standard evaluation (13 hours)
performance
Selection algorithm
Performance
predictor
Selected subset
Outputs (model signature) Efficient evaluation (6 minutes)
Eslimated
performance
Figure 2: Problem overview. We aim selecting much smaller evaluation dataset than the original evaluation dataset, while keeping the estimated performances as close as possible. Figure 3 details the selection algorithm and the performance predictor.
to estimate model performance in principled manner. Hofmann al. (2025) proposed an IRT-based approach L.LM evaluation that selects anchor points dynamically for each model, guided by its predictions previously chosen anchors. To address the growing number of methods for efficient LLM evaluation, Zhang et al. (2025) recently introduced a large-scale benchmark. In this work, we adopt approaches from the black-box model analysis techniques, explained below.
Our novelty and contribution. We differentiate our approach, Diversifying Sample Condensation (DISCO), from previous work in two aspects. (1) model disagreement (Rubinstein et al., 2024) is simpler and more effective proxy for sample informativeness than representativeness (Vivek et al., 2023; Polo et al., 2024). (2) The application of metamodels model signatures is simpler and more effective approach than direct accuracy evaluation approaches (Vivek et al., 2023) or prior approaches that require estimating latent model parameters (Polo al., 2024; Kipnis al., 2024).
3 PROBLEM
Our task is the estimation of model performance a benchmark. Let x y be predictive model over a dataset D == {(21, vi)..... (zN.9N)] sampled iid from some distribution. We are interested in estimating the model performance the dataset Sy-- An example metric for model performance is accuracy: for probabilistic classifier X - [0, 1J accuracy is defined as 1 Σ, 1 (arg max. (x,) = y.
N
We interested in estimating S/P in cost-effective way. We seek ways sample subset of size K N from the original set to estimate 5%/ The overall problem described in Figure 2. integral ingredient for both prior work and ours is the set of source models F (f......, fM, held-out set of models whose ground-truth performances are known. We define the targel models F (f (fl....,f Junn } as the models whose performances we aim to estimate.
SOLUTION
This section presents DISCO, solution to the problem of efficient performance evaluation. DISCO composed of two steps: (i) the dataset selection, where given an original dataset and an held- out set of source models, identify much smaller subset of samples; and (ii) the performance prediction, where given the model outputs on our DISCO selected evaluation set, we estimate the model performance the original set.

---

Published as a conference paper at ICLR 2026
Dataset selection (94.1)
Performance prediction (54.2)
modelts
Outputs
Indices anchor
Anchor samples
Ourputs
Estimated
top-k
77.3%
Ok
Scoring
Oocputs
PDS
Diversity-
Outputs
signature
Estimated performance
induring sample
63
0.3
77.3%
03
27
Figure 3: DISCO overview. First, we select a subset of an evaluation dataset with the most informative samples. Second, we predict the performance unseen models from their outputs on the selected samples.
4.1 DATASET SELECTION
At this stage, we require score that quantifies each sample's informativeness for predicting perfor- mance on the full dataset. Using this score, we rank the samples and select a top-k subset that best preserves the dataset's information content.
4.1.1 PRIOR SELECTION METHODS
We first review existing approaches for selecting representative data points in the evaluation set, referred to as anchor points.
Anchor-conf Vivek al. (2023) choose K anchors = {ax}t (xx)
N that minimise the sum of distances between each data point and the closest anchor: min Σ i (e(2;), e(a))), where the
e(x: abbreviates the concatenated model likelihoods E(x, [f (z))).................... g....... (x)yg for the input and ground-truth label (x, y) for the source models F f M).
Anchor-corr (Polo al., 2024) is nearly identical to Anchor-conf, except that the embedding uses correctness scores instead of likelihoods: e(x, y) := { .)....
, AM (æ, v)), where 8 ",, y)
[arg max. fa (x:) y) encodes correctness of model fT on sample xr
Anchor-IRT (Polo et al., 2024) uses the Res-RRespose Theory (IRT) to define a parametric model Pr (s." 0m a4 (3)) sigmoid(- [ss] 8th 8.). It predicts the correctness of model fr sample with parameters 8 Rd ag E Rd and 3, IR. Using observations of the sample- wise correctness of source models (Ir, Vi, 8 the parameters are inferred with an Expectation- Maximisation algorithm. Now, they continue the anchor selection based on the sample-wise embed- dings e(x;) = (a,, B.).
Best validation Kipnis al. (2024) finds anchor set A through an iterative search. The algorithm first generates a large number candidate anchor sets, (A Ap), ). by uniformly sampling from the full dataset D. For each candidate set Ap. simple scalaarto-scalar regression model, 9p- is trained on the source models F. This model learns to map the performance on the subset, S to the known
ground-truth performance on the full dataset, 5tf Fach trained regressor 9p subsequently evaluated on held-out validation set of models. The final anchor set A is selected the candidate Ap whose corresponding regressor Ip yields the lowest prediction error (e.g.. RMSE) on this validation set.
sample-wis wise statistics to determine samples with maximal information content. This greatly simplifies the sampling procedure. We exploit the model diversity, not model confidence correctness. A set of models can be highly confident and diverse at the same time. We argue that inputs that induce model diversity are more useful for performance prediction.
How DISCO differs. Unlike clustering,
4

---

Published as a conference paper at ICLR 2026
4.1.2 DISCO SELECTION
We now present our selection method. In this part, we explain how we identify such samples in the test dataset. Our sample selection strategies are illustrated in Figure 3. The main approach in Diversifying Sample Condensation (DISCO) is to select a subset DiISO of the original evaluation set D by sampling the top-k samples based on disagreement score, such as PDS. This follows the intuition shown Figure 1.
We start with an information-theorrii observation below.
(Al) be the index of uniformly chosen model. Let (xx) [o, 1] be the predictive probability for class c of model f input Tú. We write D.th for the categorical random variable following ( (x))) (z.)..... (C (I,)). De- fine ensemble mean prediction to be fe(x;) == Lf (x=) each class e and define corresponding prediction random variable as Tt. following Cat fi (zz),. (((z).)...((..))) Let S(m) = S(f ,Tm DD denote function of model and dataset DD such model accuracy, that is injective with respect to m. Assume that the only randomness in 8d comes from m (A2). Then,
Proposition 1. Lei 'D = {(ri, W:) be test set and m ~
fc
Mlm.ū (5) = H())) - Em [H (0 JSD (I)...., )
where H() is entropy, MT(.) is mutual information, and JSD(.) is generalised Jensen-Shannon Divergence for multiple distributions (Fuglede & Topsoe, 2004).
See proof in Appendix G.
We conclude that the sample i conveying the greatest level of information for the prediction of S(m) (e.g., model accuracy) the one with the greatest JSD (i) ).a)) This generalised Jensen- Shannon divergence translates the diversity of distributions (Fuglede & Topsoe, 2004). Based on
the insight that model diversity matters for performance prediction, we also consider an alternative measure that measures the model diversity: predictive diversity score (PDS) (Rubinstein et al., 2024). is more interpretable, as is a continuous generalisation of the number of unique argmax category predictions among M source models:
PDS (o).... ........MMM.. = C Σ max f(" (x:).
PDS is related to JSD through the enveloping inequalities below: Proposition 2. Denoting PDS, PDS (T!....,0?) ... JSD; JSD (0) ....,... (.,") for each sam- ple i, we have
2
M
2 (PDS; 1)2 & JSD; & M-1
log M (PDS; 1).
See proof Appendix H13.
In the experiments, we consider both JSD and PDS criteria for sample selection.
4.2 PERFORMANCE PREDICTION
Once a subset of dataset samples A is selected, we use the responses of the target model f on A to estimate the true performance.
4.2.1 PRIOR PREDICTION METHODS
We first review existing approaches for estimating the true performance using predictions anchor points A {aI....,
Weighted sum Vivek et al. (2023) estimates the true performance by directly computing the accuracy on the anchor set: WS(f, A) == (1/K) Σx Wi sh where wy is the number original training samples z; assigned the anchor aę in the Anchor-Corr method.
P-RRT (Polo et al., 2024): makes adjustments to the vanilla accuracy on the anchor set by adding correction term derived from the IRT in Anchor-IRT in: p-IRT(J, A) = (1/K) A 8k +
5

---

Published as a conference paper at ICLR 2026
1/(N- - K) EKea Pi, where P. is the IRT estimation computed based on the parameters obtained in Anchor-IRT.
8P-IRT (Polo et al., 2024) a mixture of the two approaches above: gp- X- WS(S A) (1 -1-X)---RR((F-.. A) where X E [0, 1l.
ability-IRT Kipnis et al. (2024) is a two-stage method that uses the anchor set as a diagnostic tool rather than just a miniature test. First, it uses pre-calibrated IRT model to estimate latent "ability" score, 0f, from the target model's pattern of correct and incorrect responses A. Second, pre-trained regressor, g. predicts the final performance S/ using both the simple anchor set accuracy A and this more informative ability score 0/ as input features. The final prediction is given by S/ O 0,), leveraging deeper measure of the model's capability to improve the estimate.
8
How DISCO differs. Previous prediction methods rely on scalar summaries of performance, such as the (weighted corrected) accuracy the anchor set. In contrast, approach leverages much richer signal: the model signature, defined as the concatenation of the model's raw outputs on the selected samples. By learning direct mapping from the -igenniiensi signature the final performance, we bypass the complexities of psychometric modeling and demonstrate that a simpler, more direct approach can be more effective.
4.2.2 DISCO PREDICTION
Given a smaller set of test dataset DiSCO, we estimate the performance of model f closely as possible to the true full test performance S/. We deliberately opt for simple approaches here, in order to make a point that simple is best; we also compare against rather complex prior work and show that our simple method outperforms it. Our performance prediction framework is depicted in Figure 3.
Model signatures. We hypothesise that models with similar output patterns on Drisco will exhibit similar performance. To capture this pattern, define model signature the concatenation of the model's outputs on DDiSCo: f(DDisco) :=
Such function signature may have high dimensionality. it is the product of model output dimensionality (e.g., 1000 for ImageNet) and the number of selected samples [Dpiscol (e.g., can go up to 50k for ImageNet validation set). To reduce the storage burden and improve generalizability, we consider applying a dimensionality reduction technique based on principal component analysis (PCA): Q f (DDisco)
KNN prediction. Built on the hypothesis that the similarities in function signature imply performance similarity, consider the kNN predictor based held-out set of models F. Given function to evaluate, we identify the K most similar models in F using the Euclidean distance between their signatures after dimensionality reduction. We estimate f's performance by averaging the performances of the K most similar models.
Parametric mapping. We also consider parametric prediction variant. A single parametric mapping R is trained for the prediction of model performance. As the training set, we use M model signatures Q (DDisco), ... Q J J (DDisco) for F as the training set for the regression problem of training a mapping R(-) to let R o Q fm (Dpisco) approximate S/-- The predictor R can be implemented using a neural network, linear regression, Random Forest, for example.
EXPERIMENTS
In this section, we introduce the evaluation protocol ($5.1) and the experimental setup ($5.2), present the main results of Diversifying Sample Condensation (DISCO) in language domain ($5.3), analyse contributing factors ($5.4), and demonstrate that the method is domain domain-agnostic and can also be successfully applied the vision domain ($5.5).

---

Published as a conference paper at ICLR 2026
5.1 EVALUATION PROTOCOL
To ensure a fair comparison, all methods follow an identical evaluation protocol, i.e., they use the ingredients and perform the sequence of steps during the training and testing stages.
Training. Following $ 4, select anchor datapoints and train the performance predictor.
Input: source models F (f',..., fM), full test dataset D (Dxr, Dy) with questions D2 and ground truth answers yy, parameter K.
Output: set of anchor datapoints Ak, predictor R.
1. Evaluate source models F on D₂ and obtain model outputs F(D,) = {f(x) f
2. Calculate their full test performance (e.g., accuracy) S 15/ e based on F(Dx)
and Dy.
3. Use F(D,) and optionally Dy select set AK E D2 of K anchor datapoints with respective selection method (e.g., PDS/ISD, IRT, etc.) explained in $ 4.1.
4. Train predictor R (e.g., Random Forest, gp-IRT,, ability-IRTT, etc.) explained in $ 4.2 to predict full test performance from model's outputs anchor datapoints
f(Ak)
{f(z) x Ak), such that S/ $ ((f(AA)) vf
Testing. Test the performance predictor that is trained as explained above.
fMame ). set of anchor points Ak, predictor R, ground truth performances of target models S 18/ fe F) computed the same way as S
Input: target models ff {f² ,..,, fMaas))
Output:
performance of the efficient evaluation method.
1. Evaluate target models F on anchor points Ax and obtain their outputs F(Ax) = (f(x) : Ak, F).
2. Use predictor estimate the ground truth performances of the target models $ = (R(f(Ak)) f F).
3. Calculate performance (e.g., MAE, Spearman rank correlation, etc.) of the efficient evalua- tion method by comparing S and Sf.
5.2
SETUP
We describe our experimental setup: the datasets, metrics, models, and model splits.
Datasets. We evaluate DISCO on four widely used language modeling benchmarks: MMLU (Hendrycks al., 2021), HellaSwag (Zellers al., 2019), Winogrande (Sakaguchi et al., 2021), and ARC (Clark et al., 2018). Details on the benchmarks can be seen in Appendix A.
Metrics. We evaluate DISCO and baseline approaches using two complementary metrics. First, the Mean Absolute Error (MAE) of the model accuracies, reported as percentage points (%p), captures the absolute error of accuracy prediction. Second, to assess the consistency of the relative ordering of models, we report the Spearman rank correlation (Rank) in model ranking between the true and estimated model performances.
Models. Building on the TinyBenchmarks framework (Polo et al., 2024), we evaluate 424 large language models (L..Ms) from Hugging Face's Open L.M Leaderboard (Fourrier al., 2024). The models cover GPT- (Radford et al., 2019), LLaMA- (Touvron et al., 2023), DeepSeek- (DeepSeck-Al et al., 2025), and BERT-style (Devlin et al., 2019) architectures, with model sizes ranging from 1.3 billion to 72 billion parameters.
Model split. DISCO based on a meta-model approach where a predictor is constructed based on the model signatures of pool of source models F and tested on disjoint set of target models. This approach has traditionally been criticised for its dependency on the set of existing models: the predictor may fail retain performance with unforeseen changes in future models. To address

---

Published as a conference paper at ICLR 2026
Approach
Selection
Prediction
MMLU (14k)
HS (10k)
WG (1.3k)
ARC (1.2k)
$4.1
$4.2
MAE]
Rank1
MAEJ
Rank?
MAE
Rankt
MAE
Rank
Baseline
Random
Direct eval.
3.45
0.916
2.85
0.839
3.60
0.827
2.61
0.898
Random
gP-IRT
2.79
0.922
1.96
0.819
1.64
0.928
2.22
0.921
tiny tinyBenchmarks
Anchor-IRT
8P-IRT
3.25
0.922
2.19
0.830
2.24
0.850
4.55
0.708
Anchor-corr
gp-IRT
2.08
0.927
1.27
0.937
1.95
0.918
2.18
0.948
Metabench
Best val. ability-IRR 2.08
0.904f
0.80
0.974
1.23f
0.947f
1.14f
0.971
Sig. kNN
1.82
0.912
1.49
0.899
1.58
0.920
2.30
0.905
Model signature
Random
Sig. + RF
1.81
0.933
1.36
0.938
1.29
0.926
1.72
0.938
Sig. kNN
1.31
0.972
1.32
0.956
1.19
0.951
1.96
0.937
High PDS
Sig. + RF
1.07
0.987
1.01
0.984
1.00
0.967
1.47
0.971
DISCO (ours)
Sig. KNN
1.14
0.975
1.50
0,944
1.26
0.955
2.11
0.939
High JSD
Sig. + RF
1.30
0.987
0.86
0.972
1.09
0.973
1.75
0.938
Table 1: DISCO achieves ttate- test-set compression by using model signatures combined with PDS for accurate performance prediction. Compression of MMLU, HellaSwag (HS). Winogrande (WG), and ARC datasets by DISCO (ours), BinyBenchmarks, Metabench, and other baselines. each dataset, we reduce the test set to 100 data points (except for Metabench, see helow), achieving inference cost reduction of 99.3% and 99.0%, on MMLU and HS, respectively. Sig. + RF/KNN stands for model signature with Random Forest/kNN prediction 4.2.2). Mean absolute error (MAE) is the %p difference in accuracy, and Rank is the Spearman rank correlation between the true model ranking and the estimated model ranking.
i Results for Metabench are not directly comparable, as requires more examples to converge: 150 datapoints for MMLU and ARC (+50%), 450 for HS (+350%), and 200 for WG (+ 100%). Confidence intervals in App. D.
this concern, introduce the chronological split, where the source models F consist of models published before January 13, 2024, and the meta test set consists of models after the cutoff date. The train-test ratio is 9:1.
5.3
MAIN RESULTS
Table shows the main results. Uniform random sampling, together with direct evaluation with the corresponding annotated labels, yields 3.45%p MAE and ,916 rank correlation 100 samples. The approaches introduced in tinyBenchmarks Polo et al. (2024) improve over this baseline, confirming their findings.
We measure the efficacy of DISCO in two steps: adopt a model-signature approach on top of uniform random sample selections first, and then consider sampling accord- ing to predictive diversity scoring (PDS). Even without PDS, uniform random samples, model signatures are achieving 1.81%p %p MAE and .933 rank correlation with Random Forest (RF), reaching the state-of-the-art perfor- mance with simple and practical ingredients. When PDS further considered for sample selection, to diversify the model outputs, we achieve 1.07%p MAE and .987 rank correlation (see Appendix C for qualitative comparison of predicted ranks for DISCO vs direct evaluation), demon- strating a significant leap over the prior state of art from tnnBBenchmarss Polo et al. (2024) from ICML 2024.
0.8
High PDS / RiF Spearman=0.99 Pearson=0.982
Estimated accuracy
0.6
0.4
0.3
0.4
0.5
0.6
0.7
0.8
True accuracy
Figure True and estimated performance on MMLU, Seatter plot of the performances To provide an understanding of the distributional compari- of 40 models.
son of the true model performances and the estimated performances, show scatter plot in Figure 4, As signified by the high Spearman's 's correlation coefficient .97,, the estimated performances closely follow the true performances.
Figure shows the performance against varying degrees of the test set reduction. We observe that the ranking of estimated evaluation methodologies does not change much across wide range of degrees of reduction. In particular, our DISCO is consistently the best method across all ranges of the number of samples involved. For the extreme rates of compression, 10 samples, the non-
8

---

Published as a conference paper at ICLR 2026
29.0
Randem: Birect ssataiin
102
Anchoncarr Sp-RT (hyBerrhhak)
0.35
Moan Abzolute Ereor (NAS)
-4- Random/Sigmature-
Sperrnnn Rank Correlation
*dn Highttt//ignattree
(NN
Sigh/OO//ignature RF(DISCO)
3.0-
0.79
Random Birect evaiation
Archeecor (ppiR jnylenchmaret
0.63
Bandom/Sigrature KNM(DISCO)
1.0
Hight Siggnttue -AM.ouure--KMMIOOOOO
Highhrss/iggaaurr RF(DISCO)
0.2
Samples
Samples
Figure 5: MMLU performance estimation vs. compression rates. Mean absolute error (MAE), measured in &p difference in accuracy, and the Spearman rank correlation between the true model ranking and the estimated model ranking are shown. 100 samples, the results are identical Table Main observations: DISCO hits a better efficiency-precision trade-off across entire range compression rates. For an extreme compression rate, kNN a better choice than random forest (RF).
parametric performance predictor of kNN yields better performance than the parametric Random Forest, suggesting that non-parametric approaches may be more suitable extreme compression.
5.4 FACTOR ANALYSIS
We analyse the impact of several design choices involved in our DISCO on the MMLU dataset. See Table 2 for an overview.
Split
Rank
Method
Dims
Rank
Chrono.
0.987
None
3100
0.918
Uniform 0.986
UMAP
64
0.982
Modelsplitt In a recent benchmark for efficient LLM eval- uation Zhang al. (2025), the authors observed that pre- diction performance drops sharply when test models out- perform training models. We extend this idea by replacing performance-batt based splits with chronological splits, train- ing on older models and testing on newer ones. This bet- ter reflects real-world usage, whereas performance-based splits create an artificial stress test.
(a) Model split
UMAP
128
0.970
UMAP
256
0.970
PCA
0.983
PCA
0.987
Strat.
Rank
PCA
128
0.987
Yes
0.978
PCA
256
0.987
No
0.987
PCA
300
0.985
(b) Stratification
(d) Dim reduction
For this purpose, we introduced the chronological split in $5.2. We examine the impact of this model splitting on the result. We observe that our DISCO is robust to the choice of splitting strategy. Chronological splitting yields a rank correlation of .987, which is nearly identical to the .986 obtained with uniform splitting (Table 2 (a)).
Method
Rank
kNN
0.971
[
Rank
Rand. Forest
0.987
0.652
2-layer MLP
0.959
10
0.787
3-layer MLP
0.962
30
0.844
Linear reg.
0.888
0.969
Ridge reg.
0.918
Stratification. We measure the efficacy of the stratifica- tion strategy in (Polo et al., 2024), where equal numbers of anchor points are selected from each of 57 tasks in the MMLU dataset (Table (b)). We find that stratifica- tion (.978) is not effective when data points are sampled according to PDS (.987).
0.975
Lasso
0.908
0.987
Grad boosting
0.980
(c) Source models
(e) Prediction model
Table 2: Factor analysis for DISCO on MMLUU Highlighted in bold are the default design choices for DISCO. All comparisons are based on 100 selected samples.
Number of source models. We analyse the sensitivity of DISCO to the number of source models [F] (Table (c)).
With only 100 models (.969 rank correlation), already outperforms TinyBenchmarks, which uses all 382 available source models (.927 in Table 1). As the number of source models increases, rank correlation steadily improves, reaching a maximum of .987 for F 382.
Dimensionality reduction. We compare PCA with different target dimensions Uniform Mani- fold Approximation and Projection (UMAP) (McInnes et al., 2020) for dimensionality reduction (Table 2 (d)). We notice that dimensionality reduction helps reduce potential overfitting: without (using all 3100 dimensions), the correlation is .918, while with PCA at 256 dimensions, it improves to .977. Overall, PCA outperforms UMAP and remains robust across wide range of dimensions.
Prediction model. We consider wide range of prediction models (Table 2 (c)). Random Forest achieves the highest rank correlation of 987, outperforming all other methods.

---

Published as a conference paper at ICLR 2026
5.5 RESULTS FOR VISION DOMAIN
In this section, we give a quick overview of the DISCO applied to the vision domain. For detailed results, see Appendix
Approach
Selection $4.1
Prediction $4.2
val (50k)
|MAE, Rank
Baseline
Random Uniform correctness
Direct eval.
3.03
0.652
Setup. We ImageNet-1k (Russakovsky et al., 2015) with 1.28M images and 400 pre-
Weighted sum Weighted sum
Lifelong Bench.
2.06
0.838
trained models from timm (Wightman, 2019), spanning convolutional (Krizhevsky et al., 2012) and transformer (Dosovitskiy al., 2021) archi- tectures (0.3M-300M parameters). Following the language domain, we adopt a chronologi- cal split with a cutoff of 5 April 2023 (88: 12 train-test). Performance is evaluated using mean absolute error (MAE) and Spearman's 's rank correlation. The details on the baselines for the vision domain are in Appendix J.2.
Uniform confidence
SSEPY
3.05
0.762
Sig. kNN Sig. + RF
1.72
0.808
Model signature
Random
0.86
0.944
Sig. kNN Sig. + RF
1.68
0.819
DISCO (ours) High PDS
0.63
0.969
Table 3: DISCO compression of ImageNet validation dataset. We evaluate the generalisation of our DISCO the computer vision domain. We reduce the set to 100 anchor points. The main metrics are mean absolute егror (MAE), measured in &p difference in accuracy. and the Spearman rank correlation (Rank) between the model ranking and the estimated model ranking. Main observations: (1) Same as for language experi- ments, model signature is an effective strategy for per- formance estimation. (2) Using PDS on top improves performance more.
Results. Our DISCO approach significantly compresses the ImageNet validation set by re- ducing it to just 100 data points, achieving an inference cost reduction of 99.8%. DISCO with uniform random sampling and random forest prediction model signatures achieves 0.86%p MAE and .944 rank correlation, surpassing the baseline. Using a predictive diversity score (PDS) for data selection and Random Forest for prediction, our method achieves a 0.63%p MAE and .969 rank correlation, substantially outperforming the baseline (Table 3). The results demonstrate that DISCO is effective in both language and vision domains.
DISCO (.969/0.63) outperforms Lifelong Bench. (Prabhu et al., 2024) (.838/2.06) and SSEPY (Fogliato et al., 2024) (.762/3.05 in both rank correlation and MAE. The conclusion from language experiments holds: instead of selecting anchor points with wide coverage of sample difficulty, one should focus selecting the points on which models typically disagree.
6 CONCLUSION
Evaluating ML models is increasingly expensive due to larger models, datasets, and benchmarks. It especially true for general-purpose LLMS requiring broad evaluation.
We propose DISCO, which selects a small informative subset of the evaluation data and estimates model performance from predictions on it. DISCO cuts evaluation costs by over 99% with minimal error and consistently outperforms prior methods.
This enables practical use: efficient evaluation on limited compute, frequent performance tracking during training, and cheap end-user checks of deployed models.
Limitations. The main limitation DISCO robustness to distribution shifts in the model population. Shifts can arise from new architectures, training methods, or objectives, which introduce patterns unseen during training and reduce estimator accuracy. Future work could address this with adaptive sample selection or periodic retraining newer models (see details in Appendix F).
We also discuss unsuitable tasks for DISCO. The main constraint is that DISCO requires predictive probabilities for several predefined answer choices for each question. These answer choices corre- spond to the classes in Proposition I in the original submission. That makes DISCO not suitable for open-ended generation tasks such translation summarisation. Applying DISCO to such tasks would first require defining sets of correct and incorrect outputs. We leave such experiments for future work.
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
Benjamin, Joon, and Alexander conceived the project. Alexander led the language experiments, Benjamin led the vision experiments. Joon and Martin helped design the experiments. Alexander, Martin, and Joon led the writing of the paper. Martin and Joon provided helpful feedback throughout the project.
ACKNOWLEDGMENTS
This work was supported by the Tübingen AI Center. AR thanks the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for support. This research utilised compute resources the Tübingen Machine Learning Cloud, DFG FKZINNTT 37/1057-1 FUGG.
REFERENCES
Zhiqiang Shen Aidar Myrzakhan, Sondos Mahmoud Bsharat. Open-IIm-lcaderboare From multi- choice to open-style questions for Ilms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545, 2024. 25
Peter Clark. Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0803.55557 2018. 7, 16
DeepSeek-AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li. H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu. Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https:ttps:.coc / /ttp//iivvvo////e org/ abs 2500 12948.
Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15069-15078, 2021. 2
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran,
11

---

Published as a conference paper at ICLR 2026
and Thamar Solorio (eds.), Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 1/619911/11--423. URL https: /ccl antho logy.
the
org/N1 9- 423/
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2021. URL https::///.. http//axiiv...////e//eee/e/eee.a org abs 2010 11929. 10,
Riccardo Fogliato, Pratik Patil, Mathew Monfort, and Pietro Perona. A framework for efficient model evaluation through stratification, sampling, and estimation. In European Conference on Computer Vision, pp. 140-158. Springer, 2024. 2, 10, 26, 27
Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. https: //ugg ngface. col spaces open- 1annnn-llee 11m- leaderboaad open wea.errboaarrrrd..roo 2024. 7
B. Fuglede and F. Topsoe. Jensen-shannon divergence and hilbert space embedding. In International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings, pp. 31-, 2004. doi: 10.1109/ TTT:2000.1365061
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang. Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. URL https: /ttp://z..oo.////eo////oooo.. org/ records 1261866662 25
Vipul Gupta, Candace Ross, David Pantoja, Rebecca Passonneau, Megan Ung, and Adina Williams. Improving model evaluation using SMART filtering of benchmark datasets. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 4595-4615, Albuquerque, New Mexico, April 2025. Association doi: N0.18833/1/2022. haa.0/5-aal--ong2250
for Computational Linguistics. ISBN 979-8-89176-188-6
URL https: /aa/aaaaannnttoooeoeee org. 2025 nwwcclaaannooooaa 235/. 2
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICIRR), 2021. 7, 16
Valentin Hofmann, David Heineman, Ian Magnusson, Kyle Lo, Jesse Dodge, Maarten Sap, Pang Wei Koh, Chun Wang, Hannanch Hajishirzi, and Noah A. Smith. Fluid language model benchmarking. In Second Conference on Language Modeling, 2025. URL https: www.reviee.eeeeeeeeeeeeee. net/ forum? Ld=mxccg3YRgj... 1,3
Zhengyu Hu, Jieyu Zhang, Yue Yu, Yuchen Zhuang, and Hui Xiong. How many validation labels do you need? exploring the design space of label-efficient model ranking, ArXiv, abs/2312.01619, 2023. URL https:://// /////aaa/ api semant ccscoolar Serg//ooouusddee 265610019.
Yuheng Huang, Jiayang Song, Qiang Hu, Felix Juefei-Xun Xu, and Lei Ma. Active testing of large language model via multi- stage sampling, arXiv preprint arXiv:2408.03555 2024. 2
Disi Ji, Robert L Logan, Padhraic Smyth, and Mark Steyvers. Active bayesian assessment of black- box classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7935-7944, 2021. 2
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Scongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine- aiee-grinedd eval- uation capability in language models. In The Twelfth International Conference on Learning Representations, 2023. 1,2
12

---

Published as a conference paper at ICLR 2026
Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench
sparse benchmark of reasoning and knowledge in large language models. In unknown, 2024. URL org/
httpss:/aage semant
1, 3,
Jannik Kossen, Sebastian Farquhar, Y. Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In International Conference Machine Learning, 2021. URL https: // //arxiv....////////// wwt. pttpppdf////// 05331. pdf. 2
Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Thomas Rainforth. Active surrogate estimators: active learning approach to label-efficiett model evaluation. Advances Neural Information Processing Systems, 35:24557-245775 24570, 2022. 2
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, I. Bottou, and K.Q. Wein- berger (eds.), Advances Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: /ptpeeediocceeeeesooooooo... neurips. wwtppppppafiiieeeeeeeeeeeeeeeeee wwl.ffilsssslleeeeeeeee wasesspppprr/ pdf.. pdf. 10, 26
Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. Active evaluation ac- http:/ewwwwww..eefr.reeeeeee
quisition for efficient LLM benchmarking, 2025. URL
id-tKnPtyDt 6H.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.091 2022. 1,2
Frederic M Lord and Melvin R Novick. Statistical theories of mental test scores. IAP, 2008. 2
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https:
/tt.xxivorgg//aaaae.a.o//. abs/ 15110 210.2 26
Rupak Majumdar and Filip Niksic. Why is random testing effective for partition tolerance bugs? Proceedings of the ACM on Programming Languages, 2(P0PL))1-24, 2017.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018. 26
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. URL
wrg. http:/aabs//.....0 03426. http://awwwww...abb//////a//
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023. 2
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark. Pradcep Dasigi. Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo Furious, 2024. URL htwps://///// /taxxiivvvv/////////aaa////aaava org/ / abs/ /200 00656. 17
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011. 26
Yotam Perlitz, Elron Bandel, Ariel Gera, Ofir Arviv, Liat Ein-Dor, Eyal Shnarch, Noam Slonim, Michal Shmueli-Secie Scheuer, and Leshem Choshen. Efficient benchmarking of language models. arXiv preprint ar0iv:2308.1 /696, 2023. 2
13

---

Published as a conference paper at ICLR 2026
Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating LLMs with fewer examples. In forty-first International Conference on Machine Learning, 2024. URL https: httLtttt:/pperrreeeeeee http:p/:////ppeeeiie/e// net wwwee/ooouniddggmmmme.a.. 2, 3, 4, 5, 6, 7, 8,9
1,
Ameya Prabhu, Vishaal Udandarao, Philip Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. Efficient lifelong model evaluation in an era of rapid progress. In The Tiirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httpthtts///peeeeeeeeeee //tee///DDDD.Attttttt/////eerr net/ Forum? d=A7 CTKY1. 10, 26, 27
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Tim Rädsch, Leon Mayer, Simon Pavicic, Emre Kavur, Marcel Knopp. Banş Öztürk, Klaus Maier-Hein, Paul F Jaeger, Fabian Isensee, Annika Reinke, et al. Bridging vision language model (ylm) evaluation gaps with a framework for scalable and cost-effective benchmark generation. arXiv preprint Xwx::x22.2255666 2025. 2
Alexander Rubinstein, Luca Scimeca, Damien Teney, and Seong Joon Oh. Scalable ensemble diversification for ood generalization and detection. arXiv preprint arXiv:2409.16797, 2024. 2, 3,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. URL https: http://axxvvvor/o/ee org. /bss/ 1409. 0575. 10, 26
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106 2021.
16
Igal Sason. On reverse pinsker inequalities. arXiv preprint 1rXiv:1503.07111 2015. 23
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso et Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04611 04611, 2022. 2
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie- Lachaux, Timothéc Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL https / //aaxiivv.... org/ abs 220023222222 13971.
Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. arXiv preprint arXiv:2309.086338 2023. 1, 3. 4,
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1rXiv:1804.0740 2018. 1,
Ross Wightman. Pytorch image models. metzochhimmge-mer
https://giitin com ccom/wwigee
2019. 10, 25, 26
Peiwen Yuan, Yueqi Zhang, Shaoxiong Feng, Yiwei Li, Xinglin Wang. Jiayi Shi, Chuyi Tan, Boyuan Pan, Yao Hu, and Kan Li. Beyond one-size-fits-all: Tailored benchmarks for efficient evaluation. arXiv preprint 2r02 2025. 2
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. 7, 16
Guanhua Zhang, Florian E Dornrr, and Moritz Hardt. How benchmark prediction from fewer data misses the mark. arXiv preprint urXiv:2506.07677 2025. 1, 9, 20
14

---

Published as a conference paper at ICLR 2026
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https: http:///aawww...aaaaeeee 12772. 1,2
Hongyu Zhao, Ming Li, Lichao Sun, and Tianyi Zhou. Bento: Benchmark task reduction with in-context transferability. arXiv preprint arXiv:2410.13804 2024. 2
15

---

Published as a conference paper at ICLR 2026
DISCLAIMER FOR USE OF LLMS
We primarily used LLMs in coding co-pilot applications to facilitate experimentation and help with plotting code for result presentation. LLMs were also used as writing tools to assist in refining the paper. However, the final version was carefully reviewed and finalized by the authors. No LLMs were used ideation and experimental design.
A EXTENDED EXPERIMENTAL SETUP
Datasets. We evaluate DISCO on four widely used language modelling benchmarks:
Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2021) question- answering dataset that covers 57 tasks about world knowledge and problem-sol ability.
HellaSwag (Zellers et al., 2019) dataset that focuses on commonsense natural language inference.
Winogrande (Sakaguchi et al., 2021): dataset of 273 expert-crafted pronoun resolution problems originally designed to be unsolvable for statistical models.
A12 Reasoning Challenge (ARC) (Clark et al., 2018); question-answering dataset that contains only natural, grade-school science questions (authored for human tests) and requires knowledge and reasoning.
B COMPUTATIONAL COSTS
We report the space-time complexity for the main stages of DISCO, well the of direct evaluation of a target model. The numbers correspond to a single H100 GPU and are extrapolated from evaluations of five diverse 32B LLMs MMLU (standard deviation across runs).
B.1 DISCO PIPELINE OVERVIEW
The DISCO pipeline consists of two stages: an offline stage (run once) and online stage (run for each new target model).
Offline Stage
Evaluate M source models on the full test dataset (M = 385 in this experiment)
Store source model outputs
Select 100 anchor points that maximise PDS/JSD
Concatenate outputs anchor points to form model signatures
Train predictor to estimate model performance the full test dataset from these signatures
Online Stage
Evaluate one target model on the 100 anchor points
Store target model outputs
Concatenate to obtain the target model signature
Run the predictor estimate performance the full test dataset
For every target model, the anchor points and predictor trained offline reused.
B.2 COST METRICS
The majority of the compute is required by the offline stage (3284 GPU-hours).
16

---

Published as a conference paper at ICLR 2026
DISCO: offline stage DISCO: online stage
3284.05 + 592.90 GPU-hours
0.07 + 0.01 GPU-hours
Direct evaluation
8.53 + 1.54 GPU-hours
Table DISCO computation metrics (single H100 GPU, MMLU, runs) compared to direct evaluation cost. Computation savings are computed as the difference between the direct evaluation cost and online computation cost, i.e., 8.53 0.07 8.46 GPU-hours (mean). This yields 8.46 1.54 GPU-hours saved per evaluated model.
Source outputs (offline)
86.54 MB
Source signatures (offline)
400
Target outputs (online) Target signature (online)
224.78 KB
1 KB
Table 5: DISCO storage requirements (offline stage for 400 source models and online stage for one target model).
B.3 BREAK-EVEN ANALYSIS: How MANY EVALUATIONS JUSTIFY DISCO SETUP?
DISCO breaks even at 389 evaluations. Since each DISCO evaluation saves 8.46 GPU-hours per model (vs. 8.53 GPU-hours direct evaluation, minus 0.07 GPU-hours online DISCO cost), the break-even point is:
3284
389 = 8.46
In practice, hundreds of checkpoint evaluations naturally oceur during model development. For example, single OLMo-2-32B (OLMo et al., 2024) training run includes 753 checkpoints on Hugging Face, already exceeding break-even.
In some cases, there is need to evaluate source models at all if offline predictions are downloaded from platforms such as open-llm- laa--llmm--addbbbrrddd
B.4 COMPARISON TO ALTERNATIVE APPROACHES
We briefly remind the pipelines of the compared methods:
Selection:
select set anchor points, i.e., a subset full test dataset based on different signals (random, IRT, model disagreement, etc.).
Prediction: estimate model performance on the full test dataset from outputs on anchor points.
That is why we use "Selection" and "Prediction" columns explain the difference between methods. See $ 4.1 for details on selection methods, and $ 4.2 for details on prediction methods.
Method
Selection
Prediction
Offline (GPU-h)
Online (GPU-s)
Baseline
(use all samples)
Direct eval
30739 + 5514
Baseline
Random
Direct eval
218 39
Benchmaaas
Random
gp-IRT
3284 + 592
219 +-2 39
tiny Benchmakss
Anchor-corr
gP-IRT
3284 +
219 39
Benchhaaass
Anchor-IRT
gP-IRT
3284 + 592
219 + 39
DISCO
High-PDS
RF
3284 + 592
218 39
DISCO
High-PDS
KNN
3284 + 592
218 + 39
Table 6: Comparison to alternative approaches: offline (GPU-h) and online (GPU-s) costs, and computation savings (GPU-h). DISCO dominates all other methods.
17

---

Published as a conference paper at ICLR 2026
The differences in online cost across methods negligible (e.g., 219 vs. 218 GPU-seconds). Offline costs are equal up rounding. Efficient evaluation methods allow for saving (30739-22 -218) 100% 99.3% of evaluation cost in comparison to full evaluation.
30739
C QUALITATIVE MEANING OF RANK CORRELATION IMPROVEMENTS
To justify the additional computation required for DISCO relative direct evaluation, we illustrate how the increase in rank correlation from 91.6 (direct evaluation) to 98.7 (DISCO) in Table translates into qualitative improvements in model ranking.
Figure 6 includes scatter plots of true vs. predicted ranks (see Figure 6). The direct-evaluation predictor demonstrates noticeable spread around the diagonal, while DISCO's predictions align almost perfectly with it, indicating substantially more reliable ranking.
Ground-truth vs Predicted Rank
Prediction by direct evaluation
Prediction by DISCO
40
Ps
0.92
P. 0.99
R Pg
Predicted rank
30
20
10
0
10
20
30
40
0
10
20
30
40
Ground-truth rank
Ground-truth rank
Figure 6: True vs. predicted rank comparison: direct evaluation vs. DISCO. Ps means Spearman rank correlation.
D ADDITIONAL EVALUATION RESULTS
D.1 REPORT CONFIDENCE INTERVALS
We report the standard deviation for the previously reported results on MMLU from Table 3, evaluated over one fixed chronological split and independent runs.
We briefly remind the pipelines of the compared methods:
Selection:
select set anchor points, i.e., a subset the full test dataset based on different
signals (random, IRT, model disagreement, etc.).
Prediction:
estimate model performance on the full test dataset from outputs on anchor
points.
That is why "Selection" and "Prediction" columns to explain the difference between methods. See $ 4.1 for details on selection methods, and $ 4.2 for details on prediction methods. MAE is mean absolute error; Rank is Spearman's rank correlation.
DISCO results are more stable than those of IRT and random sampling. This is because the only random component is Random Forest initialisation. In contrast, IRT is trained using variational inference, where stochastic gradient optimisation introduces additional randomness beyond model parameter initialisation.
18

---

Published as a conference paper at ICLR 2026
Method
Selection
Prediction
MAE
Rank
Baseline
Random
Direct evaluation
3.45 + 0.67
91.6 + 2.6
tiny Benchmarsss
Random
gp-IRT
2.79 0.20
92.2 + 2.3
ünnBenchharks
Anchor-corr
gP-IRT
2.08 + 0.20
92.7 2.1
tiny Benchmarks
Anchor-IRT
gp- gp-IRT
3.25 0.49
92.2
DISCO
High JSD
KNN
1.14 0.00
97.5 0.0
DISCO
High JSD
RF
1.30 0.02
98.7 0.1
DISCO
High PDS
KNN
1.31 + 0.00
97.2 + 0.0
DISCO
High PDS
RF
1.07 + 0.04
98.7 = 0.2
Table 7: MMLU, single chronological split: MAE (%p) and Spearman rank (%), mean t std over runs. DISCO (PDS/SD + RF/KNN) has lower variance than baselines.
D.2 MULTIPLE TRAIN/TEST SPLIT FOR CHRONOLOGICAL EVALUATION
To expand the number chronological splits, we bootstrap 5 different train/test chronological splits using the following protocol: for each run, split models into 385 old and 40 new based on timestamps, then bootstrap 346 source and 36 test models from these sets. Details on the chronological
split can be seen in 5.2. Results for the new splits be seen below.
Method
Selection
Prediction
MAE
Rank
Baseline
Random
Direct evaluation
2.85 + 0.85
93.3 + 3.0
tiny Binchhaasssrks
Random
gp-IRT
2.42 0.43
93.6 2.5
tiny Benhhmaasss
Anchor-corr
gR-IRT
1.93 + 0.31
92.9 + 3.0
tiny Benchmaksss
Anchor-IRT
gp-IRT
3.13 + 0.33
90.2 - 4.5
DISCO
High PDS
KNN
1.23 + 0.09
97.0 + 1.1
DISCO
High PDS
RF
1.25 + 0.14
98.0 + 0.6
Table 8: MMLU, bootstrapped chronological train/test splits (5 runs): MAE (%p) and Spearman rank correlation (%). mean + std. DISCO remains best across methods.
Bootstrapped chronological splits slightly change the mean values (e.g., rank correlation from 98.6 to 98.0 0 and MAE from 1.06 to 1.25 for DISCO), but they do not alter the superiority of DISCO over other baselines.
E SENSITIVITY OF DISCO TO MODEL CALIBRATION
To evaluate the sensitivity of DISCO model calibration, compared the Expected Calibration Error (ECE) of target models with the Mean Absolute Error (MAE) between their true performance and DDSCO-predictee performance MMLU. We observe Pearson correlation of 0.49 between MAE and ECE, indicating that better-calibrated models lead to more accurate performance (lower MAE) predictions by DISCO.
This phenomenon is explained by the information relationship between confidence and correctness. For perfectly calibrated model, the mapping between prediction confidence and correctness is deter- ministic and monotonic, resulting in high mutual information. In contrast, for a highly miscalibrated model (c.g., random guessing or uniformly confident but incorrect), prediction confidence becomes statistically independent of correctness, leading to low mutual information. Consequently, the more calibrated a model is, the more predictive its confidence patterns are of its true performance, and therefore the more informative its signature is for DISCO performance prediction.
The corresponding scatter plot shown in Figure 7.
During this analysis, observed that two factors confounded in calibration metrics: (1) overall confidence level, and (2) how well predictive uncertainty is reflected confidence. To isolate the effect of overall confidence, compared MAE with mean prediction confidence separately. We find
19

---

Published as a conference paper at ICLR 2026
MAE vs ECE
Models
0.015
LS fit
MAE
0.010
0.005
P 0.49
slope 0.04
intercept 0.002
0.000
0.05
0.10
0.15
0.20
0.25
ECE
Figure 7: Correlation between DISCO prediction enror (MAE) and Expected Calibration Error (ECE).
Pearson correlation of -0.47 between MAE and mean confidence, suggesting that overall model confidence is the dominant component of ECE that influences DISCO performance.
Figure 8 presents the corresponding scatter plot.
MAE vs Conf.
Models
0.015
LS fit
MAE
0.010
0.005
P -0.47
slope 0.02
intercept 0.019
0.000
0.3
0.4
0.5
0.6
0.7
Conf.
Figure 8: Correlation between DISCO prediction error (MAE) and mean model confidence.
F PERFORMANCE GAP EXPERIMENTS
In addition to source/target model splits discussed in $ 5.4, added experiments with wider performance gap between source and target models to identify potential failure modes for DISCO. Inspired by (Zhang et al., 2025), introduce a performance split with varying gaps. We sort all models by their average performance and take the top- 10% or top-30% (40 or 128 models) as target models, while using the bottom-90%, accuracy gap between the weakest target model and the strongest source model 0.07%p or 8. 8.18.
nottom-50% (385 213 models) as source models. The
All model splits summarised in Table 9.
Table 10 reports Spearman's 's rank correlation.
For source/target split with performance gap, the difference between DISCO and direct evaluation 1.8%p. which is lower than for the IID split (6.5%p), the chronological split (7.1%p), or the performance split without gap (8.7%p)
20

---

Published as a conference paper at ICLR 2026
IID
Chron.
Performance w/o gap
Perf. w/ gap
Prelim. model sorting Target models Source models
By timestamp
By performance
By performance Top-30% Bottom-50%
Every 10th model Everything else
10%
Top-10% Bottom-90%
Bottom-90%
Table 9: Source/target models splits.
Chron.
Perf. w/o gap
Perf. w/ gap
Direct eval on random subset DISCO
92.1 + 1.3
91.6 + 2.6
89.8 5.9
87.4 + 5.7
98.6 + 0.3
98.7 + 0.2
98.1 0.2
89.2 + 1.0
Mcan difference (DISCO direct eval)
+6.5
+7.1
+8.7
+1.8
Table 10: DISCO benefit direct evaluation random subset across various source/target models splits.
While this scenario can be seen as a failure mode for DISCO, we believe that such a source/target performance gap does not happen in practice. Instead, the accuracies of source and target models are often mixed. There are two main reasons for that. First, when practitioners develop new models, their early versions often worse than the best previously evaluated models. Second, it takes time before new models consistently outperform older ones. Infrequent extra evaluations can allow practitioners to always keep the performance gap low. That makes the performance split with gap less realistic than other splits. We thus conclude that DISCO does not break when the source and target model distributions differ, but only when the difference is unrealistically substantial.
G MUTUAL INFORMATION AND JENSEN-SHANNON DIVERGENCE
In this section, we show that Mutual Information is equivalent JSD in our setting. We present the setup and assumptions, then prove the proposition.
G.O.1 SETUP.
Let m ~ Unif(1,.... M) be the index of a uniformly chosen model. Given m, the prediction on datapoint has categorical law Py mm Define the ensemble mean distribution
M
py:
E [uiff
Pram
Σ
Prim
M
m-
Let S(m) denote any statistic that is a deterministic function of m computed on (e.g. accuracy on D).
G.0.2 ASSUMPTIONS.
Assumption A1 (Uniform prior). The model index is uniformly distributed: m Unif (1, M).
We note that Assumption Al does not assume a uniform prior over the source models f fM It only assumes that the model index is drawn uniformly, m Unif1..... ., M. the prior over models non-uniform,
replicate models proportionally to their sampling probabilities. In this case, the index distribution can be made uniform without changing the resulting model sampling outcomes.
Assumption A2 (Deterministic predictions). Conditional on m, each prediction Y; is fully determined by (or more generally, any residual randomness independent across and independent of m).
G.0.3 PROPOSITION.
Proposition 3. Under Assumptions A2-AI, if S(m) is injective, then i
MI m. (5(m); Pi) Hy. (Px.) - Em-- Hy, (PrAm)
21

---

Published as a conference paper at ICLR 2026
Proof. By Assumption A2 and since S(m) is deterministic function of m, have the Markov chain
V If S is injective, then is recoverable from S(m), hence
S(m)-
(SSmm); P.) [(m; P.).
By the definition of mutual information,
[(m 8,) = Hgg (Px.) Em Hyy (Pt)))]
Marginal distribution (using Assumption Al):
M
P3 Σ Pr(m) Prim M Σ Pr
Thus Hz.(Ps) is the entropy of the mixture distribution. Conditional entropy (using Assumption Al):
Em [Hg (Pr:1m)
Σ Hy. (Prgmm)
Combine:
M M7
I(m; (mm; = 11g (Px)
Σ Mg (Py.m)) ): JI (imm)mi))
m=1
We note that JSD and, as a consequence, DISCO predictor directly depend on the heterogeneity of source models. This heterogeneity is captured by how distinguishable the model-conditional predictive distributions Py.mm are from their mixture, as measured by their average KL divergence to
the ensemble mean. The larger this KL divergence, the higher the JSD. According to Proposition 3, larger JSD equivalent to higher mutual information between outputs and benchmark accuracy, which leads to better performance of the DISCO predictor. Conversely, if this KI. divergence is small, the JSD also small. particular, if we have many copies of the same model, then ISD as well as mutual information become zero, leading to poor predictor performance.
H BOUNDS FOR WONSEN-SHANOO DIVERGENCE (JSD) VIA PREDICTIVE DIVERSITY SCORE (PDS)
In this section, we show that JSD is bounded quadratically below and linearly above by PDS. We first relate JSD to total variation ($ H.1), then show total variation is monotone in PDS ($ H.2), and then combine these results in $ H.3.
H.1 BOUNDS FOR JSD VIA TOTAL VARIATION (TV)
We begin by showing that JSD is bounded quadratically below and linearly above by total variation. We first introduce the setup with required definitions ($ H.1.1), then prove the proposition ($ H.1.2).
H.1.1 SETUP.
Let (P P be distributions on K classes. Define the mixture
1
P
=
M
Σ
Pr:
pm
T1 m
22

---

Published as a conference paper at ICLR 2026
Definition (Jensen-Shee Shannon divergence).
M
M Σ
JSp((PR = Σ DKL (R H(P)
H(P H(Pr.lm)
M
Definition (Total variation). For distributions P, Q on the same support,
TV(P.Q) P Q11
H.1.2 PROPOSITION.
Now, we show that JSD is bounded quadratically below and linearly above by total variation. Proposition 4 (JSD-TV sandwich bounds). For any M 3 2 distributions (P IM with mixture
P.
2
M Σ
M
/ P)3 & JSD( [SD( Vm/m m-
MM
x
log M
Σ m==
TV(P;; y P).
In2
M
M -
M
m=1
Proof. Lower bound. By Pinsker's inequality (e.g. Equation (Sason, 2015)),
2
(PIQ))))
TV(..Q)
In2
Substituting Q P and averaging over m yields the lower bound.
Upper bound. Fix m. Write
P 7.lm + (1 - a)s, a = M -
1 Σ Pylle
M-1
Define t(i) ((1/PP./m (i)
when Prijml (i) (set t(i) +00 if Prip (i) = 0,5(() 0). Then
1 and
(KL Pe/m PP) = Epp m |- log (a + (1- (1-a)t)]-
Let f(u) log(a + (1 a)u), >/2 0. Then f is convex, decreasing, with f(1) 0, f(0)
log(1/ (//aa) = log M. By convexity,
f(u) < (1 - (l=w)f(0) = (1 - u) log M.
Thus.
DKL (kI(PP, & log M Epg [11 [1]] x log M Er [1 +]
Now
Epp.i [[1 - +t+1 Y m ((1)/pp (1)) ) E (P (()))+
By the balance-of-devvattios identity ($ 1),
(p (1) -()))==TV(C TV (P C)-
Finally, since P ===a aPy; p/lm + (1 (1- one has
a)s,
TV(P () M-1 TV(P P).
Combining yields
DKL P. (Y.. <
M M
log M TV(P /m P).
Averaging over m gives the upper bound.
Remark 1. The lower bound is quadratic in total variation, the upper bound linear. Thus, JSD interpolates between quadratic growth near equality and linear growth in worst-case separation.
23

---

Published as a conference paper at ICLR 2026
H.2 BOUNDS FOR TOTAL VARIATION VIA PREDICTIVE DIVERSITY SCORE
We next show that total variation is monotone in PDS. We introduce the setup with definitions and lemmas H. then prove the proposition H.2.2).
H.2.1 SETUP.
Fix a class c. Let Xm Pr (c) and P(c). Define:
Definition 3 (Envelope and spread, per class).
1
E max X M4
U
Σ
Xm pl)
2M
m=1
Definition (Predictive Diversity Score).
K
(DS (m(m))
Σ max Pe: m (c).
Lemma 1 (Balance-of-deviation identity). For any a1..... am with Σ... am = 0, writing a+ max(0,a),
Σ
(mm)= +
Σ
(am)= = 1 Σ aml
m-1
m m-1
m-1
Proof. Decompose a = a+ a_,, |a| = a+ + a_ Summing and using Em Om = 0 gives
Σ am,+ Σ am,
Σ am, Σ am,-
Then
Σ
Jaml
Σ (amtt + am,
= -
> Om
m
Applying Lemma with am Xm 11 yields
U. M
Σ (Xm - p). m:Xm>>
H.2.2 PROPOSITION.
Now, we show that total variation monotone in PDS. Proposition (Spread-envelope bounds). Use notation from Appendix H.2.1. For each class c, if at most models satisfy X. μ4, then
&
i Ue
&
Fe.
M
M
Aggregating over classes.
1 M
E
&
U
<
E,
M
where
K
K
M Σ
E..
U=
Σ U
(V(Ps)m P).
M
Proof. If F. 0, then X..
l for all m so U- 0. Otherwise, let m² = arg max, Xm. Then
U
Σ (Xm 1)) 2 M (X,= 14) E
For the upper bound, each positive term at most Ec, and there are at most such terms, hence
Ue & 1 Fe.
Summing classes gives the aggregated bound.
24

---

Published as a conference paper at ICLR 2026
H.3 FINAL SANDWICH INEQUALITY
Finally, we combine results from $ H.1.1 and H.2 to show that JSD is bounded quadratically below and linearly above by PDS.
Proposition 6 (JSD-PDS sandwich). Use notation from Proposition 1.
2 M (PDS((P))m))) mm) - 1)² & [S &
M
log M (PDS((PP.))) 1). M M-1
Proof. From Theorem 4,
JSD >
U
JSD
&
M M
log M U.
In2
Define
K
K
E =
Ees
U: : Σ Ue-
By the definitions,
K
M
M
M
U =
P7 Pram(() (c) - P(c) =
2M Σ
| P/lm mm
P|1 =
Σ
TV(Py.|m P),
2M
M
where p Pr:
-m=1
Dd Al and
Pilm
E
max m
Pp/m (c) -P(c)) -PDS- 1.
From Proposition 5,
1 (PDS- 11 <U< (PDS
M
Combining and noticing that = < M yields the quadratic lower bound and linear upper bound in (PDS 1).
I IMPLEMENTATION DETAILS
The results of all experiments were averaged over five runs. For the language tasks, we either used pre-computed LLM outputs downloaded from the -eeedelr-la (Aidar Myrzakhan, 2024) on Hugging Face or computed the outputs using in-evaluation-hammess (Gao et al., 2024). LLM evaluation performed on single NVIDIA H100 GPU, while parametric prediction methods (described in 4.2.2) were trained on a single NVIDIA GTX 2080 Ti GPU. For the vision tasks, we used models from the timm library (Wightman, 2019), also evaluated single NVIDIA GTX 2080 Ti GPU.
Training parametric model takes less than one minute for both vision and language domains. Details on LLM evaluation time can be seen in Table 4.
For the MMLU dataset, we observed that computing disagreement scores (as described in 4.1.2) using all available source models led to worse DISCO performance than using only a subset of them. This can be explained by the fact that including additional, highly similar or redundant models may dilute the effective heterogeneity of the ensemble, which is crucial for DISCO as discussed in G.O.3. Consequently, we select subsets of source models when computing disagreement scores. These subsets are obtained by randomly sampling M models from the source models, where M is treated as a hyperparameter and tuned jointly with other hyperparameters. For MMLU, the selected value was M 100. We chose random sampling here it provides simple and unbiased way to control ensemble size without introducing additional selection criteria.
Details for prediction methods:
KNN: We used k 1 in all experiments unless stated otherwise.
25

---

Published as a conference paper at ICLR 2026
Random Forest (RF): Implemented using scikit-learn (Pedregosa al., 2011) with default
parameters.
2-Layer MLP: Trained for 200 epochs using the AdamW optimiser (Loshchilov Hutter, 2019) with default settings and learning rate of 0.001. Hidden dimension: [128].
3-Layer MLP: Trained for 700 epochs using the AdamW optimiser (Loshchilov & Hutter,
2019) with default settings and learning rate of 0.001. Hidden dimensions: [128, 128).
Linear Regression:
Implemented using scikit-learn (Pedregosa al., 2011) with default
parameters.
Implemented using scikit-learn (Pedregosa et al., 2011). with default parameters and regularisation weight
Ridge Regression:
Lasso Regression: Implemented using scikit-learn (Pedregosa al., 2011), with default parameters and regularisation weight À 0.0001.
Gradient Boosting (GB): Implemented using scikit-learn (Pedregosa et al., 201 1), with default parameters and 200 base estimators.
Details for dimensionality reduction methods:
PCA:
We used the scikit-learn implementation (Pedregosa et al., 2011) and varied the number of principal components as shown in Table 2.
UMAP: We used the umap-learn library (McInnes et al., 2018) and varied the number of components as specified in Table 2.
VISION RESULTS
We introduce the setup (SJ.1), describe baselines ($J.2), and present results ($J.3).
J.1 SETUP
Dataset. We ImageNet- I (Russakovsky et al., 2015) with 1.28 million images. Models. We consider 400 models from t timm (Wightman, 2019) that are pretrained on ImageNet. The models cover convolutional (Krizhevsky al., 2012) and transformer (Dosovitskiy al., 2021) architectures. Model sizes range from 0.3M to 300M parameters.
Model Split. As in the language domain 5.2). we use the chronological split. The cutoff date is April 2023. The train-test ratio of models is 88:12.
Metrics. We use mean absolute error (MAE) and Spearman's 's rank correlation between the and predicted performances.
Evaluation. Evaluation protocol follows the one in $ 5.1
J.2 ABOUT BASELINES FOR VISION DOMAIN
Our work is not the first to propose efficient evaluation in the vision domain. The two closest methods are Lifelong Benchmark (Prabhu et al., 2024) (NeurIPS 2024) and SSEPY (Fogliato et al., 2024) (ECCV 2024). They propose efficient evaluation methods for visual models, using similar two-stage framework for efficient evaluation in the language domain (see $ 4 Figure of our submission):
1. Select "impportantreeprssnttttii"""" anchor points.
2. Estimate model performance based on model outputs on the anchor points.
In (Prabhu et al., 2024), mean correctness scores across source models are used to measure sample difficulty, and anchor points are selected by sampling every k-th datapoint after sorting them by difficulty (where scores predicted for each test datapoint. Predicted correctness scores are binary values indicating relative position (after sorting by difficulty) to the hardest anchor point the target model got right. In
# anchor points ), Final performance is predicted as weighted sum of correctness
#all datapoints
26

---

Published as a conference paper at ICLR 2026
kardom DDreett evaliatio
Random/ Signatura- RF
MMan Abrolute Eror (MAK)
20.0
HighrDS,Signan -NN((DSCC)
HighPSS /5ggnnatuure -R((DSc
Spearman Rank Correcti
0.00
5.0-
0.70
0.40
2.0-
0.59
fandom Diect eustiat
1.0
Random/Signaurn -
High#OSsSignature kNN(DIS-Co)
HighPDSSSignaturee AFIDISCO)
18
Samples
Samples
Figure 10: ImageNet performance estimation compression rates. Mean absolute error (MAE), measured in %p difference in accuracy, and the Spearman rank correlation between the true model ranking and the estimated model ranking are shown. At 100 samples, the results are identical to Table 3. Main observations: Same as for language experiments DISCO hits a better efficiency- precision trade-off across the entire range of compression rates.
(Fogliato et al., 2024), confidence scores of the target model are used to measure sample difficulty. Then samples are clustered by difficulty with K-Means, and anchor points are selected as the centroids of the clusters. Final performance is predicted weighted sum of anchor correctness scores. Weights for the weighted sum are determined based on the corresponding cluster sizes using the Horvitz- Thompson estimator.
The main message of our paper that selecting anchor points for efficient evaluation, better to select diversity-nnuucing data points (DISCO) than to make good coverage of sample difficulty (prior work). In 5.3, we have shown that our approach beats prior approaches in the language domain.
Likewise, in the vision domain, the existing approaches (Prabhu et al., 2024; Fogliato et al., 2024) focus good coverage of sample difficulty rather than maximizing per-sample information by seeking
data points. We test empirically Table whether the same conclusion holds in the vision domain by comparing DISCO to (Prabhu et al., 2024; Fogliato al., 2024).
We computed the results for these baselines ourselves, as the papers do not contain results on the same setup described in J.1.
ImageNet. For fair comparison,
J.3
MAIN RESULTS
Table 3 shows the main results. See $ 5.5 for their overview.
We evaluate the effectiveness of DISCO two stages. First, we apply the model-signature approach using uniform random sampling. Then, we enhance it by selecting samples based on predictive diversity score (PDS). The results follow a similar trend to the language domain. With uniform random sampling. model signatures combined with Random Forest achieve 0.86%p MAE and rank correlation 944, significantly outperforming the naive baseline. Incorporating PDS further improves performance, reaching 0.63%p MAE and rank correlation of .969.
To illustrate how well the estimated performances align with the true values, present scatter plot in Figure 9, The high Pearson correlation coefficient of .970 indicates strong agreement between the two.
0.90
High Signature-RF Spearman=0
Estimated accuracy
Pearson
0.85
Figure 10 shows performance across varying levels test reduction. The relative ranking of evaluation methods remains largely stable, except for the kNN predictor, which degrades the number of anchor points increases. No- tably, DISCO consistently outperforms all baselines, even under extreme compression with few 10 samples.
0.75
0.70
0.70
0.75
0.80
0.85
0.90
True accuracy
Figure 9: True and estimated accuracy ImageNet for 50 models.
27


Reviews:
[{"review_id": "vxs6q2YXMR", "reviewer": "Reviewer_W8ah", "summary": "This paper sheds light on a important problem of efficiently estimating a model's performance of large contemporary benchmarks $D$ by evaluating the model performance on a much smaller selected subset $|A| = K \\ll D$. The core idea, DISCO, puts forward the notion that samples should be selected where models disagree the most (to maximize inter-model response diversity), not necessarily samples that are \"representative\" in input latent space. This is done using either Jensen-Shannon divergence (JSD) or a computationally simpler proxy, Predictive Diversity Score (PDS). \n\nAfter selecting $A$ each model is represented by its \"signature\" which is then mapped using a lightweight predictor (k-NN/Random Forest/Regression) to estimate actual dataset performance. The paper also provides theoretical justification for using PDS by sandwiching bounds with JSD and connecting it with mutual information. \n\nThe empirical performance is strong across language datasets like MMLU, HellaSwag, ARC, Winogrande and also in Vision (Imagenet) while dramatically reducing inference cost with $K = 100$.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The insight of shifting focus from representativeness of samples to disagreement of models on these samples is a novel and fresh perspective.\n2. PDS is computationally cheap to calculate from logits and the final predictor from the model signature is straightforward to implement in existing benchmark evaluation pipelines.\n3. The authors provide reasonable theoretical bounds to use PDS as a proxy for JSD for the subset dataset selection.\n4. The contribution includes strong empirical results and generality across domains, which adds to the fact that this method is model agnostic, thus is more likely to be used in the real world.", "weaknesses": "1. In proposition 1, the authors implicitly use the assumption - uniform prior over source models. This is not examined fully in practice because real model pools are heterogeneous and non-uniform. How sensitive are the PDS/JSD selection and signature predictors to such non-uniformities?\n2. Computing PDS/JSD requires running $M$ source models across the entire benchmark dataset $D$ which is likely significant for large datasets. The authors should report absolute compute (GPU-hours), memory/storage for signatures and break-even points (how many target models or evals justify the offline cost) for the method to be more practically used.\n3. Minor typo in section 4.2.2. \"we also compare against a rather complex prior work\nand show that our simple method wins against the stolen\" - The word stolen is out of context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bKtbySUpNa", "reviewer": "Reviewer_dg58", "summary": "This paper introduces DISCO (Diversifying Sample Condensation), an efficient evaluation approach that improves both steps common to prior frameworks: subset selection and performance prediction. Instead of clustering or relying solely on model confidence/correctness, DISCO selects samples that induce maximal inter-model disagreement, and then learns a direct mapping from a model signature—the concatenated raw outputs of a model on these selected samples—to predict full benchmark performance. Evaluated on 424 LLMs across MMLU, HellaSwag, Winogrande, and ARC, and 400 vision models on ImageNet-1k, DISCO achieves low MAE and high ranking fidelity with very small subsets (e.g., 100 samples), showing strong efficiency and competitive performance relative to prior methods.", "questions": "Please see the weaknesses section for a more detailed explanation of my concerns, but here are a few specific questions:\n\n- Regarding the vision experiments: could you clarify whether this is the first application of subset-based evaluation techniques to vision models (e.g., ImageNet-1k), or if there are existing baselines in that domain that you compared against or were inspired by?\n\n- About calibration: since the proposed method relies on model output probabilities both for computing disagreement (PDS) and constructing model signatures, how sensitive is it to differences in model calibration? \n\n- On ranking fidelity: how should we interpret the magnitude of change in rank correlation you report? For instance, is improving from 0.916 (baseline) to 0.987 (your method on MMLU) a substantial and practically meaningful difference in ranking quality, or does it mainly indicate incremental improvement at the high end of the correlation scale?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper identifies a clear and timely gap in the literature on efficient evaluation and proposes a simple, well-motivated solution that is both conceptually clear and practically effective (specially in vision domain and for low budget regimes).\n\n- The mathematical framing provides valuable intuition for why the proposed approach works and strengthens the overall narrative without adding unnecessary complexity.\n\n- The experimental evaluation is thorough and well executed, assessing the performance of the proposed method across multiple benchmarks and model families, and comparing it against several baselines.\n\n- Table 2 (factor analysis) offers additional insight into how specific design choices affect performance.\n\n- The paper is well written and structured, making the methodology and results easy to follow.", "weaknesses": "While the paper is clearly executed and well motivated, I have some reservations about the practical efficiency of the proposed approach, particularly for LLM evaluation.\n\n- This first point is more of a critique of the overall line of work rather than one targeted specifically at DISCO. From Figure 5, it appears that randomly selecting slightly over 1000 samples with direct evaluation achieves on-par or even slightly better performance on both MAE and rank correlation compared to more sophisticated sample-selection techniques. This raises the question of whether the added complexity of sample-selection mechanisms is justified. To identify the most informative samples, one must evaluate a large pool of models, store their outputs, and compute disagreement statistics between their predictions for each sample. This process seems computationally and memory intensive, potentially offsetting the efficiency gains claimed from evaluating fewer samples. In contrast, simply randomly selecting a slightly larger subset (e.g., 2 000 samples) and then combining it with the proposed model-signature representation and simple predictors (e.g., linear or random forest) might yield even better results with significantly lower overhead.\n\n- Relatedly, Table 1 shows that while the proposed method consistently improves over baselines across datasets, the magnitude of improvement in MAE appears modest—though I am open to being proven wrong, as this may depend on implementation details or evaluation scale. The improvement in rank fidelity is clear but numerically small given the additional computation required.\n\n- Moreover, the paper does not include a quantitative analysis of computation or memory cost across methods. While this may not yet be standard practice in this research line, such an analysis would substantially strengthen the claim of “efficiency” and help contextualize the trade-off between computational expense and predictive accuracy.\n\n- Finally, while the vision results (Table 3) are particularly strong and show a clear improvement in ranking fidelity, it remains unclear whether this work is the first to apply subset-based evaluation techniques to vision or if comparable baselines exist. Clarifying this point would help position the contribution more clearly within the broader literature.\n\nI would like to note that I am open to revising my assessment if the authors can address these concerns. I appreciate the quality of the work and would genuinely welcome clarification that changes my current view.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ghQJyQYqQ9", "reviewer": "Reviewer_zKXy", "summary": "The paper proposes DISCO, an efficient evaluation framework that (i) selects a tiny subset of test items by ranking samples via model disagreement (measured with JSD or the Predictive Diversity Score, PDS), and (ii) predicts a target model’s full-benchmark performance from its model signature (concatenated outputs on the selected subset) using simple predictors (kNN or Random Forest). The key thesis is that we should focus on diversifying model responses, rather than on the representativeness of samples, because disagreement maximizes the information about model performance. A formal proposition links mutual information about a model’s score to the Jensen–Shannon divergence across model predictions, motivating the selection rule; DISCO then shows strong empirical results on MMLU, HellaSwag, Winogrande, ARC, and also on ImageNet, often compressing datasets to 100 samples while preserving rank/accuracy with low error. This allows model evaluation with ~99% less inference costs.", "questions": "- Missing sentence in the abstract? Before \"The typical approach follows two steps.\" There must be something about efficient sampling (or in that sentence itself) otherwise it's unclear what we're talking about.\n- The two assumptions for Proposition 1 should be added in the main text so that result is complete, I believe there is enough space in the paper for this.\n- Typo?: line 288 \"the stolen\"\n- Table 2: The PCA results are missing (supposed to be in subplot (d)) while the Prediction model ablation is said to be in (e) while actually being in (d)\n- In section 5.3 it is stated that \"performance-based splits create an artificial stress test\" and that the chronological split better reflects real-world usage. While I generally agree with this statement, I still think a performance-based split should be included for comparison since this is a known failure mode of efficient LLM evaluation methods.\n- Why are there no other baselines besides random selection with direct eval. in Table 3 for vision models? Presumably there is nothing preventing past benchmarks to being adapted to this setting.\n\n# Summary\nI think this is a strong paper, well written and well justified both theoretically and relative to past works and with adequate empirical evaluations. I would be happy to increase my score if the authors address the identified weaknesses and points in the questions section. Most of these represent relatively small, although important, additions / modifications.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear practical value: Achieving 99%+ cost reduction while maintaining prediction accuracy addresses a real problem in modern ML evaluation where benchmarks require thousands of GPU hours.\n- Elegant simplicity: The approach is conceptually simpler than prior work—avoiding complex clustering algorithms and IRT parameter estimation in favor of greedy sample selection and direct prediction from model signatures.\n- Solid empirical validation: Comprehensive experiments across multiple benchmarks (MMLU, HellaSwag, Winogrande, ARC, ImageNet) with hundreds of models, demonstrating consistent improvements over baselines.\n- Theoretical motivation: Proposition 1 provides an information-theoretic justification linking mutual information to JSD, showing why model disagreement is informative for performance prediction.\n- Domain agnostic: Demonstrates effectiveness in both language (LLMs) and vision (ImageNet) domains.\n- Thorough ablations: Table 2 systematically analyzes design choices (model split, stratification, number of source models, dimensionality reduction, prediction model).\n- Clarity: The paper is well written, the claims are clear and the experiments support these claims adequately.", "weaknesses": "- The empirical results sections are somewhat opaque. I find it hard to understand exactly how the experiments were conducted from the text alone. The experimental set-up seems valid, but perhaps a reworking of the text or an additional, more detailed, description of the empirical set-up could be added in the appendix.\n- Statistical rigor: No confidence intervals or significance tests on reported metrics. Single train/test split for chronological evaluation. Unclear if the improvements are statistically significant.\n- Limited failure analysis: Insufficient discussion of when/why DISCO might fail. No analysis of which types of benchmarks or tasks are unsuitable. The acknowledged limitation about distribution shift in model populations lacks empirical investigation (see questions, the performance split would be useful)\n- Missing cost analysis: No discussion of computational cost for computing PDS scores over full dataset. Cost of training metamodel not reported. How does total cost (sample selection + prediction) compare to just running models on more samples?\n- From my understanding the current DISCO pipeline relies on the signature being of the same size (i.e. same size of model outputs and same number of samples), the predictor is then trained on this fixed size representation (that is reduced with PCA). Currently, if one wants to change the number of samples used it would require the entire re-training of the predictor, the main reason seems to be the concatenation of the predictions which treats each scalar prediction *of each sample* as a different variable. I would be very interested in seeing a discussion on how this could be changed so that the pipeline is adaptable with respect to the number of samples (i.e. being able to use a predictor trained on N samples with less or more samples).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)