How Benchmark Prediction from Fewer Data Misses the Mark
Authors: Guanhua Zhang
Florian E. Dorner
Moritz Hardt
Venue: NeurIPS 2025 poster ()
Keywords: Evaluation
Large language model
Primary area: evaluation

Abstract: Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.

How Benchmark Prediction from Fewer Data Misses the Mark
Guanhua Zhang 1,2 Florian E. Dorner .2.3 Moritz Hardt
Max Planck Institute for Intelligent Systems, Tübingen Tübingen Al Center ETT Zurich
Abstract
Large language model (L.M) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select small subset of evaluation points and predict overall benchmark performance from that subset. this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly com- petitive baseline: Take random sample and fit regression model the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilitie
Introduction
Increasingly, computational cost is major bottleneck in the evaluation recent generative mod- els. Growing model size and benchmark task difficulty, well as the sheer number of available benchmarks all escalate the problem. For example, evaluating a single 176B parameter model on the HELM multi-task benchmark required 4,200 GPU hours [35]: ; even major companies noted the
significant computational burden of evaluation on the BigBench multi-task benchmark [77.
The problem has prompted much recent work on more efficient LLM evaluation. The typical approach find subset of data points to evaluate on, and predict benchmark performance from these few evaluations. The simplest method is the random sample mean: Take random sample of n evaluation points, and compute the mean of the benchmark metric on the sample. For a metric, like accuracy, with values in the interval [0, 1), the sample mean gives an additive approximation up to error ((1/ /n). More sophisticated methods try to improve over this baseline by following a common strategy: cleverly choose small core set of evaluation points, evaluate multiple known models on these points, then fit a model predict overall benchmark performance from these evaluations.
"Corresponding author: euwahha..hhangggeuuiiingg....e. Code available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

16
MANDOM-SAMPLING
MAXDOM--AMPPLIG- LEARN AIPW
AIPW
Estimation Gap (%)
Source models
Estimation Gap (%)
Source models
2
60
70
Real Accuracy (%)
Real Accuracy (5)
Figure 1: The r-axis denotes the real accuracy in ImageNet while the y-axis denotes the estimation gap (equation I for each target model. The gray stands for the aceuracy range of source models. Left: source models are randomly sampled across all models. Right: source models are randomly sampled from models with accuracy lower than 80%.
We group existing efforts following this strategy under the term benchmark prediction. Previous research has proposed several hypotheses for why benchmark prediction can work: Core sets identify the most informative data points [4]] they exploit the dependence between model performance different data points [60], and they can capture the unobserved abilities of models [43].
The goal of our work is to systematically examine the strengths and weaknesses of benchmark prediction solution concept for efficient LLM evaluation.
1.1 Our Contributions
We conduct a large-scale, systematic evaluation of 11 state-of-the-ar benchmark prediction methods across 19 diverse benchmarks. For each benchmark, collect detailed performance results for at least 83 models on all data points. We split all models into two groups: source models and target models. For the source models, performance data is available for all data points. In contrast, for the target models, performance information is available only for up to 50 data points. Each method must adhere this constraint of selecting more than 50 data points, and the objective is estimate each target model's mean performance across the full benchmark. To evaluate method effectiveness, compute the average estimation gap-the absolute difference between the and estimated full-benchmark performances across all target models.
Many methods work well on similar models, but simple baseline works best. We first study the interpolation regime where the source and target models are random drawn from the same set of all models. Our empirical findings first confirm that in this regime it is possible to reduce the average estimation gap relative to RANDOM SANDOM-SAMPLING SAMPLING, which simply reports the mean performance on randomly selected core set. All evaluated methods outperform RANDOM- SAMPLING in over half of the benchmarks. Given that all methods operate on the same number of core-set evaluations, their computational costs are comparable. Thus, a lower average estimation gap indicates superior data more informative use of the evaluation budget.
cerciency-fffettiie
Surprisingly, what works best in the interpolation regime is remarkably simple: after randomly sampling a core set, rather than computing its mean, fit regression model to predict the true mean performance. This method, RANDOM- LANDOM-SAMPLINGE SAMPLING-LEARY
consistently outperforms most other methods and reduces the average estimation gap by an average of 37% compared to RANDOM- S SAMPLING. This suggests that the manner of core-set selection is relatively unimportant; rather, the key to success modeling the correlation between core-set and full- hill-enchmark performances.
Methods fail the evaluation frontier. However, our analysis reveals a major limitation:
effectiveness of benchmark prediction methods drops sharply when source and target models are not drawn from the same distribution. We call this the extrapolation regime. To explore this regime, conduct experiment in which select the lop-performing models (according to the full

---

benchmark) target models and use the poorer perfor-erf ones as source models. This setup reflects the typical use of benchmarking the evaluation frontier where new models are being released that are likely better than existing ones.
In the extrapolation regime, we show that most benchmark predictions methods fail outperform the naive RANDOM-SAN SAMPLING bascline. This is illustrated in Figure [] When source models cover the more than halves the estimation error.
full range of performances (left), When source models are restricted to the lower-than-80% accuracy (right), RANDOM-S AMPLING- LEARN still outperforms RANDOM-SAMPLING for target models similar to the source distribution, but its predictions substantially degrade for better- seater-performing targets outside the source range.
AIPW an overlooked exception to the rule. One notable exception a method inspired by augmented inverse propensity weighting (AIPW)ussee in other statistical applications- we introduce in the context benchmark prediction. AIPW reliably outperforms RANDOM-SAMPLING both under interpolation and extrapolation. Although it sometimes performs worse than RANDOM- SAMPLING-LEARN when targets resemble sources, it consistently maintains an advantage when they do not, thanks to being a consistent estimator. However, illustrated in Figure (right), even AIPW sees diminishing improvements as target models' accuracies exceed those of the sources.
Benchmark prediction relies on model similarity. To more systematically examine the general- ization of benchmark prediction methods, we calculate the model similarity [38], quantifying how closely the predictions of each target model match those of the source models used in training. We observe a strong negative correlation between model similarity and estimation gap: methods that beat -SAPLLNGG tend to do so primarily for targets similar to sources, while accuracy on disimilar models deteriorates. In contrast, RANDOM AMPLIIG exhibits neutral correlation, providing consistent (albeit less accurate) estimates regardless of similarity.
Main takeaway. Our findings suggest that while benchmark prediction techniques can be useful in specific scenarios, their reliance on similarity between source and target models poses a risk of misestimating the performance of new models. This underscores the importance of applying these methods with caution, especially for evaluating models that significantly deviate from previous ones.
2 Related Work
Evaluating large language models (LLMs) has become increasingly costly as these models grow in size and capabilities 33 16
65, 68). These costs manifest in several ways. First, the collection and annotation of evaluation data can require significant resources [66]. To mitigate these costs, researchers have turned to methods such as using LLMs-as-judges 21
23 or employing active labeling 32 31 10 12 70 to generate evaluation data and labels. However, these savings come with drawbacks. For instance, LLM-as-a-judge does not produce reliable evaluation outcomes, as judge models tend to prefer models similar to them, and have other biases
Another significant cost in LLM benchmarking arises from the model inference itself. Generating responses with LLMs can be time-consuming (35] 68 53), and common inference time scaling techniques [57
24
54
33]
may exacerbate this issue. The success of scaling laws (29 50 in
predicting model performance has fueled interest in the development of benchmark prediction
techniques 60 43 40
41, which aim to estimate benchmark performance by evaluating LLMs
on a limited set of data
The key idea underpinning benchmark prediction is that not all evaluation examples carry the same amount of information [49] It is hypothesized that smaller core set of examples can represent the entire test set, allowing for accurate estimation of overall benchmark performance [60]. This is similar to efficient model training approaches, which aim identify a subset of training data that enable performance comparable to training on the full dataset [52] [69]. Indeed, popular benchmark prediction method, k-medoids clustering, is a classical approach core-set selection for training [5]. However, it is important to recognize that the objectives of training and evaluation differ significantly. While training focuses on minimizing empirical risk and enhancing model performance,
which focuses on identifying the best model from a pool, benchmark prediction is more challenging seeks to forecast overall benchmark performance for any new model.
Unlike bandit literature 68 53
3

---

evaluation seeks to provide unbiased estimation of model's performance to facilitate fair model comparison 40). Our work challenges the assumption that core-set selection is the key to the success of benchmark prediction by introducing competitive methods that do not rely on core-ser selection.
Many existing approaches treat benchmark prediction as learning problem, aiming predict model's s overall performance based on its performance on subset of data [60] 43 34 30 45). Despite
promising results, previous work has highlighted limitations terms of estimation variance [36]. Going further, we highlight that most benchmark prediction methods rely on model similarity, with estimation performance deteriorating when target models deviate from familiar source models.
3
What is Benchmark Prediction?
3.1 Problem Formulation
We define benchmark as triplet (D,F, Here F refers to the set of models to be evaluated on the benchmark and 8 represents the evaluation metric. Lastly, 'D represents the benchmark data with [D] = N data points. A data point is referred to D, where (x, y), refers to the query and refers to the ground truth answer.
=ff,2) refers to the performance of any f F any data point E D. For example, s(f.
1[f(x) = y] if the benchmark uses standard accuracy as the metric.
$(f,D') |D'| E.ED s(f, represents the average performance of f E on any D.
Dff = {s(ff=))> represents the vectorized performance of f F on all data points in D, and s(F", 2) {=(S, =)) represents the vectorized performances of all models in F' F on data point E D.
S(F'D') = {s(f. D')) = (s(F),2) the performance matrix of all models in F C F on all data points in D' C D.
We refer to F(+) Uh..... (MM F as the set of source models, whose performances on every data point of the benchmark S(F()) D) are known. The rest of the models are the target models F(()) F F((), which are only be evaluated on << N data points save computational costs.
Benchmark prediction with fewer data aims to estimate {(ʃ, D) for every E EF)) with only n data points. In practice, benehmark prediction often involves two steps: identifying representative core-set C C 'D with |C| n data points, and @ learning a performance estimator h to estimate the average performance on the full benchmark based on the core-set. Formally, the goal of benchmark prediction is to find C and to minimize the estimation gap over target models,
estimation gap:
|5(1,D) - (D)]] JEF(()
Σ
FF
For simplicity, in the remainder of the paper, we will denote the estimated performance of target model FF() as h(f). instead explicitly writing ((,(F.C),S(FT))) P)).
3.2 Benchmark Prediction Methods
Previous methods. In this paper, we examine five widely-used benchmark prediction methods,
RANDOM-SAMPLING randomly samples a subset as C and directly returns the mean performances as J. J.RANDOM-SAMPLING (f).
ANCHOR-POINTE POONTTO-WIINTTEEEEE 60 uses k-medoids clustering to identify C and returns weighted sum based on the density of each cluster as SAANCHOR-POINEES Beaters--WGGGeE WEIGHTED (f).
(60) extends ANCHOR-POINTS, WEIGHTED. Instead of directly returning the weighted sum, a linear regression model is learned as PANCHOR-POINTS-PEEDOO (f)-
SITHOORPPOITSS-PREDDDE
P-IRT 43 extends ANCHOR-PPINTS POINTS-PREDICTE Item Response Theory (IRT) model as h² IRT IRT().. (f).
by replacing the regression model with an
GP-IRT further generalizes P-IRT by combining its estimation with WEIGHTED weighted sum, and use it as h GP-IRT (f).

---

New methods. We introduce six methods that have not yet been applied to benchmark prediction.
randomly samples a subset as and learns Ridge regression model which predict ((f,D) based sff.C), as A R SAMPLING LEARN (f).
SOMOOM--MMPLIGG
performs SANDOM-SAMPLING-LIEOE the run based on cross-validation.
RANDOM-SFARCH--LEE
for 10,000 times and selects
LASSO trains a Lasso regression model to predict {(f, , D) based on s D) with sparsity con- straints on number of non-zero weights lower than n. The learned model is then used as h (f).
OPPIMMIZE employs gradient descent to optimize both subset selection vector, which models C, and a linear regression model, / DOUBLLEOOPIMIZS
27
Principal Component Analysis (PCA) treats benchmark prediction as matrix completion problem by assuming the performance matrix S(F, D) is of low rank. By randomly sampling a subset as C, this methods conducts PCA to impute the missing values for target models [59] 6
Augmented inverse propensity weighting (AIPW) 48]: Inspired by the application of prediction powered inference [ to the LLM-as-a-judee setting [4), we apply a more general AIPW estimator to benchmark prediction. We train Ridge regression model for every target model which predicts the point-wise performance s(f, 2) based on s(F(t), ,)) Formally,
= arg min g'
[g'is( 2)] s(f.
(2)
ZEC
The idea behind the AIPW estimator is to use the predicted performance $(f,2) g[s([((), as proxy score to estimate $(ʃ D) and "debias" that estimator as follows
(N- -
hAIPW (f) = s(f,c)+ 1 N-n
$(f,2) 1 ((((222) N - 2ED-C
Σ
(3)
SEC
Unlike the other Saaming-based baselines, AIPW is consistent estimator for s D) Compared to SUDDOM-SAMPIIGG it reduces estimator variance by factor of up to 1 ((((f. sf. 2)) 14, where p the Pearson correlation coefficient.
More details of each method are listed in Appendix
4 Experiments
4.1 Experiment Setup
We select a diverse range of benchmarks from the following sources
HELM-Lite benchmarks [35]:
OpenbookQA [39
GSMBK 9
LegalBench
22 Math 26
MedQA (28). and MMLU [25 We obtain the per-data point performances of F 83 models
from the official leaderboard.
GLUE benchmarks [6]] MRPC
13 RTE
SST-2 2 [55], MNLI 64 and QNLI
We
use the per-data performances of [ 87 models provided by AnchorPoini [60.
OpenLLM benchmarks [6:: IFEval 677, Math 226. MMLU-Pro [62]
Arc- Arc-Challenge
g
BBH 58 GPQA [47] and MUSR (56). We use F
448 models provided by Huggingface and
collect their performance scores.
ImageNet
5T:
We collect [F| 110 models from Pytorch HubBand evaluate them ImageNet.
A summary of benchmark statistics is provided in Appendix B
'Since P-IRT and GP-IRT requires s(f, 2) to be binary, we only use benchmarks with accuracy as metric.
"The provided score file for QQP is broken so exclude it.
http:/aaac...pppppeeeeeee/eeepee 11m- Im--eeddeebboord/penn 11m leaderboardif 1tt.eeddeeeerdrdraaa

---

IP Openhonigy
(0:0)
344
[-:33]
31
(-10.0
2
(-15.4)
2.9
(-1322)
3,0
[-0.4)
34
[-19.1)
2
(1:2
37
(bba))
4,0
(3:4
35
(-13.2)
3.0
77
37
3-
6.3
40
(0.))
(10.5)
(13.9)
(31.5)
(30.9)
(37.2)
(9.5)
BSN LaghhBeach
(0.4)
5.7
(-36.6)
(-247)
4:3
(:33.0)
[-42.2)
93
(3119)
(:20.5)
34
(:7.3)
5-3
(-42.8
37
(-37.4)
(:38.8)
3.5
Hash
(0:0)
4.1
(4.44)
3.9
(:10,)
3.7
[.0,5]
3,8
3.8
[46.1)
6.0
[7:4]
3.8
(43.6)
5.6
6.9
(4:2)
4.3
(4.7)
3.9
(0:2)
(0.0)
(1.4)
(:37.8)
4.9
3.3
3.4
3.5
3.3
2.9
3.2
4.3
3.5
3.7
3.6
(000
[-31
(:30,6)
(-28-4)
(-31.5)
(-40.6)
(:34.3)
1:14.00
(-21.2)
(-25.7)
(-2b-1)
Estimationn Eap % (Moductin over RANDOW SANNDOS-SMMIIIA Www.--SSLLOO
52
3.0
2.7
3.0
2.6
3:7
3.3
3.4
3.5
3.2
3.0
(0.0)
(42.6)
(447.0)
(42.5)
( (49.1)
(2.-8)
1 (35.5)
(34.3)
(35.7)
(:8.4)
(-43.3
OLTE-STE
(0:0)
5/2
(-54.0)
[-54 9)
2.3
(-50.6)
(:58.3)
2.2
6:536)
(-46.4)
2.8
(-57.)
(-50.4
[-55-7]
(-57.5)
2.2
MUM 807--
(0:0
4.5
(-41.6)
2.7
(-40.5
2.3
(-11.5
2.7
(499.0)
2.3
(6.9]
4.5
[-1.33
2.7
[-31.3)
3.1
[41.3)
2.7
(-27.8)
3.3
(-30.0)
2.9
5.5
2-3
9.4
2.8
7.6
31
27
2.3
3.2
3.2
(0.0)
(-49.0
170.1
(-40.5
(-52.5)
(-7-2)
(-44.0)
(-50.3)
(-58.1)
(-4144
(-42.0)
5.3
3.7
2.4
3.2
2.2
2.8
(000
(-4M0)
(:29.)
(-4b-4
[:34.1)
(-40.8)
[-40.0)
[:3366
(-99,)
[-40.1]
(-40.3)
1P
(0.0)
4.6
(-28.6)
3.3
(-21.1)
3,6
(-20.3)
3.6
(-33.2)
3.0
(11.8)
5.1
(:29.6)
3.3
(59.4)
7:3
(27.8)
5,5
(-17:5)
3,5
(-21.1)
3.6
22
1.0
1.5
1.3
2,0
2.1
(0:0)
(-18-)
(-12-2)
(-16.)
(-99.2
(-37-4
(-41.2)
(7:5)
(204)
(198)
(-4.7)
4.9
7.
2.4
2,6
3.
4.2
(0.0)
(-41.5
(45.4)
(-27.5)
(:50.8)
(:3.8)
[-4G.)
(21.1)
(-23,8)
(-13.2)
(-14.5)
522
3.1
3.3
3.3
2.7
3.2
2.7
6.8
3.8
3.5
3,4
(0.0
(-41.2
(-37 4)
(-37.5)
(-47.8
(-39.5)
(-48.1) 1)
(242)
[-78:7]
3)
(-34.3) 3)
GpeaLLK-38H
(0.2)
54
(-16.7)
4.5
(:52.7)
2.6
(45.1)
3.0
(:37.5)
3.4
(-33.1)
3.5
4.5
1.3
2.1
4.2
1.6
1.4
3.5
9.3
2.2
4,2
3.8
(0:0)
(-63.5
(-57.4)
(:14.1)
(-68.3)
(-71.3)
(:20.))
(:12.3)
(-35.0)
(-16.2)
(-23.6)
(00)
5.4
(-64.) 1)
1.9
(-56.8
2.3
(-168)
3.3
(-775)
1.7
(-676)
1.7
[-52 5)
2.5
(41 5)
5.6
(-574)
2,3
[-46 7)
2.9
(-47.7
2.8
-3
TragwTiet
(0:0)
(:54.7)
(-50.1)
(-41.)
(558.)
(217.0)
(-11.7)
(-50.3)
2.4
Averaga
(0:0)
4.8
(-37.2)
2.9
(-314)
37
(-30.)
3.3
(-2.1)
27
(-22,0)
3.8
(-36 l)
3.0
(c:4)
4.8
(-21.1)
3.6
[-25 0)
3.5
(-79.)
3.3
RANDOM-SAMPPRN
TANDON-SAMPtONN-- -RAAR
PCA
HANDOMS-S SRA.SSSARRRRRAA ELARS
AIPW
DOUBLR- Ormaaz
LASSO
ANczon- Poouum-oosss----- WicomED
ASCHOn: Popers Puctecton
P-IIFF
GP -RT
The estimation gaps (1) for target models (equation T under the interpolation split, where source and target models identically distributed. Each target is evaluated 50 data points. The estimation gap reduction (4) over RANDOM-S AAMPLING is shown in parentheses. A negative reduction means that the method achieves a lower gap than RANDOM-SAMPLING. % is omitted.
Figure 2:
4.2 Estimation Gap Reduction under Interpolation
As done in previous work 60
we examine the effectiveness of benchmark prediction methods under the interpolation model split where source models are identically distributed with target models.
45
Interpolation model split. For each benchmark, we randomly select 75% of models as source models F((), for which performance scores across all data points S(F(s), D) are available. The remaining 25% of models serve as target models F=()) for assessment of benchmark prediction methods. Each target model is evaluated on only n = 50 data points unless specified otherwise. Benchmark prediction methods are used estimate the full benchmark average performance 5(f, D) of each target model / (r(() and evaluated based the estimation gap from equation 1 Each experiment is repeated over 100 random trials, and we report the average estimation gap across all target models in these trials to ensure robustness. See standard errors in Appendix C
The results are presented in Figure 2 Compared to LANDOM-SAMPLIA benchmark methods effectively reduce the estimation gap in over half of the evaluated benchmarks. Notably, nine out of ten methods reduce the estimation gap by more than 20% on average across all benchmarks, as indicated in the last row. This verifies the effectiveness of benchmark prediction methods in the interpolation setting, where source and target models are identically distributed. Interestingly, the top-performing method the simple baseline, RANDOM-SEARCH--EE. achieves a 42. 5% reduction compared to RANDOM-S AMPLING averaged accross all benchmarks. In comparison to the previous sfae--of-the-an GP-IRT, which leads to a 29.9% reduction on average, RANDOM-SEARCH LEARN results in a lower estimation gap in nearly all benchmarks.
Results.
all other
which

---

ITI Opanhonage
(2.0)
23
(227.3)
166
(104.5)
4.8
(-2.6)
23
(216.3)
74
1257:91
3.
(156.1)
0.6
(-31.8
1.6
(367.]
10U
(99.3)
4.b
(22.6)
2.
SEUK-G8MXX
(9.0)
(203.5)
125
(907 3:
(-4.5)
(164 9:
92
[S32
(126.1)
8/0
(22.1)
4.2
690
(59.0)
3.3
(18.1)
22.
Tagelllench
(3.0)
(41.6)
T6
[-23.2)
(22.9)
0.1
(-15.8)
(-32.3)
(-32.0)
37
(-34.2)
HELH-Hath
(3.0)
3.7
[241.8)
1200
[440 72
20.2
(0:1)
3.7
(180:5) 5t
30.5
(513.5)
22,9
(157.71
9.6
(40.3)
5.3
(432.00
13:8
(17.3)
4.4
(-2.4)
3.6
20.0
BELI-YAOQDA
(0.0)
4.7
[707 91
18.1
(167 9
13.6
(:11.0)
4.2
(266. 4]
37.3
(797.6
18.8
(67-4)
8.8
(-32.6)
3.2
(778 5]
17.9
(10:8)
5.2
(10.1)
5.2
HELU-
(332:1
41
9)
(52.9)
2,4
(4.4)
4.6
(900)
17.5
Essination Gip S (Reduction BANDOO SAMPLID 15
QUIC MAPC
(3.0)
4-5
(450.5)
75.
1428.17 73
22.7
(-28.3]
(388.5)
21.0
(320.7)
27.5
(304.2)
17-4
(:329)
(402.3)
(2.9)
44
[-17.)
4.9
12.9
11.2
4.3
12.1
12.3
6.6
4.6
13.1
6.3
5.6
SE ITC XTR
(0.0)
(163.1)
(127-4) 4]
[-13.3)
(146.7)
(150.9)
(34.5)
(-5.9)
(166.3)
(28.1)
(14.5)
OLTE-687-2
(2.0)
7.9
(405.3)
14.8
(232 0g
10.3
(255)
24
(396.0)
9
(737-3)
21.9
(72.)
5.0
(-0.5)
2.4
(404.3)
17.4
(134.4)
6.3
(104.5)
6.0
15.0
QUIE-HSLI
(3.0)
(462.2)
28.5
29.4
(-10.3)
(455.7)
(4.4)
(-7.8)
(-1.2)
3.0
(0.5)
5.5
13.0
13.7
5.1
23.8
14.1
5.4
5.7
13.0
0,4
6.6
(3.0)
(153.3)
(146 3:
(-6-6)
(150.8) 8:
[156 9)
(-1.5)
(4:5)
(151.3)
(17.0)
(19.2)
12.5
5.1
23.0
25.6
4.8
22.7
33.2
24.8
7.0
24.1
5.2
5.6
(9.0)
(16659)
(400
(-6.4)
(950
(305 91
(10.8)
(392.00
[0.0]
(9:5)
OpanLLN-Nath.
(388.3)
12:1
(5:2)
(308.11
10:0
(295.8)
15.7
(11.3)
4,4
(:13.5)
15.2
.6.5
8.5
4.7
10.0
(3.0)
(182.0)
(308
[-14:1)
(186.8)
(240.5)
(42.0)
(48.0)
(217.6)
(-18.3)
(-17.6)
DpenLLK-ABD
(0.0)
5.2
(9
10.3
(96.9)
10.2
[:19:3)
4.2
(30.9)
9.8
(150.5)
13.5
(32.1)
7.8
(10.2)
4.7
(354.5)
13.2
(37.7)
4.3
(1773)
UpenLtZ: am
(2.0)
5.5
(197.7)
16.3
(278:3)
20-7
(:4.5)
6.2
(182.8)
15.8
(231.1)
18:1
(06.9)
<1
(53.4)
8:4
(223.9)
12.6
(BL.9)
10.0
(64.7)
40
Opant2
(3.0)
(32.0)
6.
(:2.0)
(25.7)
0.5
(-7.3)
(-8.8)
(-13.9)
[-10.7]
Ope0LEM-M18R
(3.0)
5.5
(10.0)
6.1
(10:1)
6.1
[:26.2]
4.0
(0.7)
5.8
(2.5)
5.7
(:347)
4.2
(-22.5)
4.3
(26.0)
7.0
(-31.7)
1.8
[-35.3]
3.6
ImageTet
(0.0)
4.4
(24-7)
5.5
(1:00:3
9.7
[-35
2.8
(1944
5.2
(bi 7)
7.4
(4:3)
4.8
(155.5)
11.2
(61 4)
7.1
(-95)
4.0
(-19)
4.0
Average
(0.0)
(200.3) 3)
13.0
(225 3)
(-12.6)
4.0
(165.5)
32.8
(250 8)
(86.0) 0)
8.2
(22.4)
(211.2)
(16.1)
5.1
(5.1)
Basoose BANDOM--AAMMLLE SAMMARR
RANDoRt- SAMIINN LEARS
PCA
RASDOSS- St uRcns LeARS
AIPW
DOCIBE Uprraaze
LAGOO
Ascion- Potoyss W Wesuarro
Ancront- Port rn- Patsourrut
P-IRT
GP-RR
The estimation gaps (L) for target models (equation I under extrapolation split, where source models are the lowfst-penoomming 50% and target models are the top 30%. Each target model is evaluated on n = 50 data points. We also report the estimation gap reduction (J) over RANDOM- AMMLING in parentheses. A negative reduction implies that the method achieves a lower estimation gap than RANDOM-SAMPLING % is omitted.
Figure 3:
On the other hand, the selection of the core-set does not significantly enhance the effectiveness of benchmark prediction. For example, the second best-performing method, RANDOM-S SANDOM-SAMPLINGG AMPLING- LEARN, also consistently outperforms RANDOM-SA -SMOLL-S across all benchmarks, despite the sole difference being the use of a Ridge regression model rather than directly averaging across the core-set. With 37.2% reduction in estimation gap, it performs comparably to RANDOM-SEARCH LEARN, despite the latter conducting 10,000 iterations of RANDOM CAMMPLING-EARRE to identify the best subset. Moreover, it surpasses methods like DOUBLE- -OOTIMIEE and GP-IRT, which select subsets through optimization or clustering. Another benchmark prediction method, A AIPW, which also utilizes randomly sampled core-set, consistently achieves a lower estimation gap across all benchmarks, yielding results comparable to the state-of-the-ar GP-IRT. These findings challenge the prevailing notion that the core of benchmark prediction lies in identifying the most informative or representative subset. Instead, our results suggest that the primary driver of benchmark prediction success is learning predict the mean, with core-set selection playing relatively minor role.
4.3 Estimation Gap Increase under Extrapolation
We examine the effectiveness of benehmark prediction methods under the extrapolation model split where target models all perform better than source models.
Extrapolation model split. Different from the random source-target model split last subsection, we begin by ranking all models for a given benchmark based on their average performance on the full benchmark ((f.D). The lowest-pe 50% of these models are designated as source models, while the top 30% serve as target models for evaluating benchmark prediction methods. This strategy
7

---

reflects real-world model development scenarios, where developers debug and assess improved models based on existing less effective models. The estimation gap as defined in equation I iis used for measuring the effectiveness of benchmark prediction. We again repeat each experiment 100 times.
Results. The results are shown in Figure The average estimation gap for HANDOM-SAMPLING (4.6%) is largely comparable with the interpolation setting (4.8%) as it doesn't 't rely on source models. However, for all other methods, the estimation gap increases when compared to the inter- polation setting. Nearly all benchmark prediction methods that outperform in the interpolation scenario now show diminished performance. Notably, the previous best method now results in a 185.1 % increase in estimation gap than RANDOM- SAMPLING, and performs worse than RANDOM- SANN-SAMRRA SAMPLING across all benchmarks. The only method that still outperforms RANDOM- AMPLING on average is AIPW, beating RANDOM-SAMPLING in 18 out of 19 benchmarks. This is because AIPW, like RANDOM- RANDOM-SAMPLING is a consistent estimator, but has lower variance than random RANDOM- WENDDO--SMPLLNN SAMPLING when its predictor is effective. However, the estimation gap reduction (-12.6%) of AIPW over SANDOM-SAMPLING in the extrapolation setting is also less pronounced than in the interpolation setting (-30.4%).
PADOM-SAMPLIAN
LANDOM-SEACCH-LE
This stark contrast between interpolation and extrapolation settings underscores the heavy reliance of most benchmark prediction methods on the similarity between source and target models. This is unsurprising, given that many methods approach benchmark prediction problem, which often struggles in out-of-domain scenarios. However, unlike traditional machine learning, which primarily emphasizes in-domain performance, key objective of benchmarking is to assess and identify new superior models. Therefore, extrapolation is a more prevalent and pertinent setting than interpolation in the context of benchmarking, and the decline in the estimation gap of benchmark prediction methods in this setting calls for more caution.
machine learning
4.4 Reliance on Model Similarity
In this subsection, we investigate the extent to which benchmark prediction methods rely on the similarity between target and source models.
Model similarity. We follow previous works 38 20 and define the model similarity of target
model f to all source models Fr() as follows,
S(fF(() D) =
M Σ (EFF 1 Ceap
Cobs Ceup
Here, Carp 5 D) D) + (1 (() (() measures the chance agreement rate, s(f, is independent of s s(f',,) In contrast, the observed agreement rate. For simplicity, we use S(f)
i.e., the expected probability of (s(f,2)= =
Cobs - + 2 =[s(f= = sff to denote F() D) in the remainder of the paper. S(f) quantifies how similar the performance pattern of the target model f is to all source models Fr(), with a higher value indicating greater similarity 20
We aim to examine the correlation between model similarity and estimation gap. However, we note that the estimation depends on the standard deviation of s(f. 2). Since we use accuracy as the metric in our experiment, s(ʃ,)) =) is Bernoulli with parameter p = 8(J, DD and standard deviation of = (Pr(1 ps). By randomly sampling n data points as C, Chebyshev's inequality ensures that (5ffC) - s(f, D) < /
(5) with probability at least (1 a). In other words, the performance of target models with lower oy is easier to estimate with the same amount of data. Thus, the standard deviation of the basic estimation gap could potentially confound the observed correlation between model similarity and estimation gap. Consider the method R RAND RANDOM-SAL SMMLING whose estimation does not depend on source models. If all target models with low o coincidentally have high S(f), while those with high a have low S(f), then spurious correlation between estimation gap and model similarity to target models could appear even for RANDOM- SAMPLING To prevent this, we define the normalized estimation gap as
of normalized estimation gap for f: E(f) 8(f== ((f. ((f h(f)l-
(6)
Then we measure the Pearson correlation between model similarity in equation 4 and the normalized estimation gap in equation
8

---

4.75
4.37
0:13
4:28
4.20
-0.00
0.28
0.23
0.17
4.21
0.38
4.17
0.24
0.3
-0.1
-01
40337
40.22
4.33
-0.19
-0.1
-0.35
4.13
16LA-19C
0.02
0.37
022
0.18
4.35
0.77
2.
0.23
0.58
0.35
0.17
Smialo
GLME KTE
4.61
-0.44
.0.53
-4.51
.0.57
-2.42
-2.6
-0.91
-0.54
4.55
0138-537-3
40:0
-0.41
-0.43
10.00
-0.25
-0.57
-043
4.41
-0.44
0.47
0:23
.0.51
4.445
0.25
.04
.0.53
.0.4
-0.7
0.47
OpaciAy: IFEra
-0.16
40.5
-4.12
-0.1
10.15
4.11
0:08
40.09
+0.08
0.16
-0.08
08
-0:37
0:19
-0.51
-0.15
4.33
SpeziLit
Spezi221 201
0:34
4.044
0:35
0:32
0.20
0.04
SpeaL 27Q1
40.00
-0.44
40.7
401
0..1
-0.1
00.35
0.02
-0.08
4.11
Iougelett
0000
410
HANDAR SAStPsAN
Bainot SAMPLII LEAARE
PCA
RANDOR SEARCR LGART CAAR
APP
Asoiic Poers- H laaang
Pomers-Fasscro PRNDACTiO
187
ANCHOS- Miche-Poners-PPe Ponerrs
Figure 4: The Pearson correlation between normalized per-model estimation gap (equation 6 and model similarity (equation 4. Negative correlation indicates that target models that are dissimilar to source models tend have larger estimation gap, and vice versa.
Results. The results are shown in Figure 4 A clear negative correlation between model similarity and estimation gap emerges for almost all benchmark prediction methods except for RANDOM- SAMPLING. In particular, the best- best-performing method under the interpolation model split, RANDOM- SAMPLING-LEARN. exhibits negative correlation below -0.2 in 13/19 benchmarks. Despite its asymptotic unbiasedness, we also find negative correlations for AIPW. This is perhaps unsurprising: While AIPW is consistent independent of how well its regression model g[s =]] predicts s(f; its variance depends precisely on that prediction quality. If the predictions are good, AIPW improves substantially over RANDOM-S AMPLINN, while there is no improvement when predictions are fully uninformative. But intuitively, predicting s(f, 2) harder when f is very different from the models F(s) used for training the predictor ((s(())),,
4.5 Ablation on Core-set Size
We conduct ablation study the size of the core-set n. We experiment with (10, 20, 50, 100, 2003. and the summarized results are shown in Table I (detailed results can be found in Appendix C). As expected, the estimation gap generally decreases as n increases for most methods. Our previous conclusions remain valid across both settings. With larger core-set sizes, most methods continue perform better than SANDOM-SAMPLAN so in the extrapolation model split. Interestingly, we also find that R RANDOM-S AMPLING outperforms all other methods when given twice as much data, even in the interpolation model split.
in the interpolation split but fail to do
AIPW remains effective in both settings. However, its advantage over RANDOM-SANE SAMLLING diminishes increases. While AIPW reduces the estimation gap by -30.4% in interpolation and 12.6% extrapolation for n 50, these advantages shrink to -12.4% in interpolation and a mere 2..3% in extrapolation for 200. This is because the estimator variance reduction factor of AIPW is up to p ( sff, (f.=)))))))))) 2. On the other hand, the advantage of AIPW remains significant when the dataset is large and thus Tv small. Figure 5 compares compprsss AIPW with ri 50 to RANDOM SAMPLING with 100 data points using ImageNet. AIPW achieves a lower average normalized estimation gap compared R RANDOM-SAMPPLNNG normalized estimation gap for AIPW is biased with respect to model similarity. In contrast, the
despite using only half the data. However, the

---

Table 1: Ablation study on the core-set size n. We report the estimation gap averaged over all benchmarks. is neglected for each metric. The lowest estimation gap each column is highlighted in bold. See detailed results in Appendix
Interpolation 50
Extrapolation 50
10
20
100
2001
10
20
100
200
LANDOM-SAMPLING
11.0
7.7
4.8
3.3
2.1
10.7
7.4
4,6
3.1
2.0
RANDOM-SAMPA SAMPLING-LLALAR
4.2
2.9
1.5
17.6
15.8
12.1
11.1
PCA
6.6
5.2
3.7
2.8
2.1
19.9
17.6
14.9
12.2
9.3
3.3
SANDDO--SEARC-LEEE SEARRCH-LEARN
4.5
3.7
2,7
2.0
1.4
16.0
14.4
12.8
11.8
11.1
LASSO
3.6
2.2
22.0
15.3
6.6
4.8
3.0
2.3
1.9
11.3
9.0
8.2
8.0
7.0
ANCHOR POINTS WEIGHTED
6.9
4.9
4.0
3.2
10.4
6.7
5.6
4.7
34
ANCHOR- POINTS- PREDICTOR
4.7
4.1
3.6
3.4
4.1
16.2
14.8
13.4
12.4
11.2
P.IRT
7.3
5.9
3.5
2.1
1.3
9.8
8.2
5.1
3.7
3.0
3.3
7.8
x on Ieapeltet
RANDOM-SAMPINN
Tangelie
0.10
0.30
Fitted
Average
Avar Gap
Estmation
0.08
Neerage Hormalize Eatimttion
0.08
3.04
0.02
0.002
0.00
3.00
0.45
0.55 Similerity
Model Similarity
Figure 5: Average normalized estimation gap relative model similarity for AIPW (n=50) and RANDOM-SAMPAN (n=100) on ImageNet. Each bar represents the target models whose similarity source models falls within the corresponding range. The normalized estimation gap defined as shown in equation 6 On average, AIPW outperforms SAANDO-SAMPLINC even with half the data. However, SANDOM-SAMPLIN shows better performance when model similarity is low.
normalized estimation gap under RANDOM-S RANDOM-SAMPLINC remains largely neutral regarding model similarity. Consequently, while AIPW reduces the average, it produces higher gap for models with low similarity compared to RANDOM-SAMPLING SAMPLING with twice the data.
5 Conclusion
In this paper, we study the problem of benchmark prediction from fewer data and examine benchmark prediction methods. Our findings call into question the necessity of meticulous core-set selection and reveal that these methods are most proficient at interpolating scores among similar models. However, except RANDOM-S P-POM-SMMPLINGG and AIPW, all methods face significant difficulties when predicting target models that differ substantially from those they have encountered before.
We caution against the indiscriminate use of benchmark prediction techniques, as their dependence on model similarity causes most of them to fail precisely when most needed: the evaluation frontier, where the aim is to assess new models with unknown capabilities. Even in the context of interpolation, no method outperforms RANDOM- AMPLING, when that simple baseline is given access to twice as consistent estimator with lower variance,
much data. Thus, while recommend to use AIPW this suggests that simply raising the sampling budget for RANDOM-S AMPLING can competitive, especially in settings where predictions of other models for fitting AIPW costly to obtain.
We thank Yatong Chen and Jiduan Wu for helpful discussions and feedback on draft versions of this work. Florian Domnr is grateful for financial support from the Max Planck ETH Center for Learning Systems (CLS).
Acknowledgement.
10

---

References
[1] Anastasios Nikolas Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction- inference. Science, 382:669 674, 2023.
[2] Anastasios Nikolas Angelopoulos, John C. Duchi, and Tijana Zrni.. Ppi++: Efficient prediction- powered inference. ArXiv, abs/2311.01452 2023.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. ArXiv, abs/1308.3443 2013.
[4] Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth PASCAL recognizing textual entailment challenge. 2009.
[5] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael Jordan. Au- toeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008 2024.
[6] Jian SannFFeg Cai, Emmanuel Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal optimization, 20(4): 2010.
[7] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans Iims as the judge? a study on judgement biases. arXiv preprint arXiv:2402. 10669, 2024.
[8] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: /83.055503. 2018.
[9] Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.. 14168, 2021.
[10] Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without a testing set. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2677-2685, 2020.
[11] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL PASCAL recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177-190, Springer, 2006.
[12] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1 15069-15078. 2021.
[13] William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the International Workshop on Paraphrasing. 2005.
[14] Florian Dorner, Vivian Yvonne Nastl, and Moritz Hardt. Limits to scalable evaluation the frontier: Lim as judge won't i beat twice the data. In The Thirteenth International Conference on Learning Representations, 2025.
[15] Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies. 2009.
[16] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard https::/// https://huggingfaccs co/ wwwpppacs//ooooeeeceocccooooeeoo 11m- o 11m [mm_leaderbbord 2024.
[17] Deep Ganguli, Nicholas Schiefer, Marina Favaro, and Jack Clark. Challenges in evaluating AI systems, 2023.
[18] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9. Association for Computational I. Linguisties,
2007.

---

[19] Adam N Glynn and Kevin M Quinn. An introduction to the augmented inverse propensity weighted estimator. Political analysis, 18(1):36-56, 2010.
[20] Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K. Chandra, Ponnurangam Kumaraguru, Douwe Kicla, Ameya Prabhu, Matthias Bethge, and Jonas Geiping, Great models think alike and this undermines ai oversight. ArXiv, abs/2502.04313, 2025.
[21] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuchao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. ArXiv, abs/2411 1. 15594,, 2024.
[22] Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, Jessica Wu, Joe Nudell, Joel Niklaus, John Nay, Jonathan H. Choi, Kevin Tobia, Margaret Hagan, Megan Ma, Michael Livermore, Nikon Rasumov-Rahe, Nils Holzenberger, Noam Kolt, Peter Henderson, Sean Rehaag, Sharad Goel, Shang Gao, Spencer Williams, Sunny Gandhi, Tom Zur, Varun Iyer, and Zehua Li, Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, 2023.
[23] Veronika Hackl, Alexandra Elena Müller, Michael Granitzer, and Maximilian Sailer. Is gpt-4 reliable rater? evaluating consistency in gpt-4 text ratings. ArXiv, abs/2308.025 02575, 2023.
[24] Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv preprint Ar0sv:3305. 18466, 2023.
[25] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, abs/2009.03340 2020.
[26] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.0087.. 2021.
[27] Eric Jang, Shixiang Shane Gu, and Ben Poole. Categorical reparameterizzation with gumbel- softmax. ArXiv, abs/1611 1..1144, 2016.
[28] Di Jin, Eilcen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang. and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009. 2020.
[29] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2rXiv:2001.0836/ 2020.
[30] Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench sparse benchmark of reasoning and knowledge in large language models. 2024.
[31] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample- efficient model evaluation. In International Conference on Machine Learning, 2021.
[32] Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active surrogate estimators: An active learning approach to label-efficient model evaluation. ArXiv, abs/2202.0688 2022.
[33] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensing nlp tasks. Advances in neural information processing systems, 33:9459-9474, 2020.
[34] Yang Li, Jie Ma, Miguel Ballesteros, Yassine Benajiba, and Graham Horwood. Active evaluation acquisition for efficient I1m benchmarking, ArXiv, abs/2410.05952, 2024.
12

---

[35] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga. Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher R e,e Diana Acosta-Navas, Drew A. Hudson, E. Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan S. Kim, Neel Guha, Niladri S. Chatterji. O. Khattab, Peter Henderson, Qian Huang. Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas F. Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Annals of the New York Academy of Sciences, 1525:140 - 146, 2023.
[36] Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Oluwasanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. Quantifying variance in evaluation benchmarks. ArXiv, abs/2406. 10229, 2024.
[37] Pranav Mani, Peng Xu, Zachary Chase Lipton, and Michael Oberst. No free lunch: Non- asymptotic analysis of prediction-powered inference. ArXiv, abs/2505.201 2025.
[38] Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. ArXiv, abs/1905. 10t//995.12500. 12580, 2019.
[39] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2399 Brussels, Belgium, October-November 2018. Association for Computational Linguisties.
[40] David Owen. How predictable is language model benchmark performance? fass/2401.047571 2024.
ArXiv,
[41] Lorenzo Pacchiardi, Konstantinos Voudouris, Ben Slater, Fernando Mart Plumed, Jos's e Hern' Ordezz-Oallo., Lexin Zhou, and Wout Schellaert. Predictaboard: Benchmarking Ilm score predictability. ArXiv, abs/2502. 14445, 2025.
[42] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. ArXiv, 130/2444..1066 2024.
[43] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. ArXiv, abs/2402. 2024.
[44] Felipe Maia Polo, Ronald Xu, Lucas Weber, Mírian Silva, Onkar Bhardwaj. Leshem Choshen, Allysson Flavio Melo de Oliveira, Yuekai Sun, and Mikhail Yurochkin. Efficient multi-prompt evaluation of lims. arXiv preprint arXiv:2405. /5002, 2024.
[45] Ameya Prabhu, Vishaal Udandarao. Philip H. S. Torr, Matthias Bethge, Adel Bibi, and Samuel Albanie. Efficient lifelong model evaluation in an era of rapid progress. In Neural Information Processing Systems, 2024.
[46] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383-2392. Association for Computational Linguistics, 2016.
[47] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. ArXiv, abs/2311. 1.0022 2023.
[48] James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 1995.
[49] Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan L.. Boyd-Graber. Evaluation examples are not equally informative: How should that change nip leaderboards? In Annual Meeting of the Association for Computational Linguistics, 2021.
13

---

[50] Yangjun Ruan, Chris J. Maddison, and Tatsunori B. Hashimoto. Observational scaling laws and the predictability of language model performance. ArXiv, abx/2405.10938,, 2024.
[51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei- Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115:211 - 252, 2014.
[52] Noveen Sachdeva and Julian McAuley, Data distillation: A survey. Trans. Mach. Learn, Res., 2023, 2023.
[53] Chengshuai Shi, Kun Yang, Jing Yang, and Cong Shen. Best arm identification for prompt learning under a limited budget. arXiv preprint arXiv:2402.09723, 2024.
[54] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute opti- mally can be more effective than scaling model parameters. arXiv preprint vrXiv22408.03314 2024.
[55] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of EMNLP, pages 1631-1642, 2013.
[56] Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. ArXiv, abs/2310. 1b6/2910.16049, 2023.
[57] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference machine learning, pages
PMLR, 2020.
[58] Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether chain-of-thoughh can solve them. In Anmual Meeting of the Association for Computational Linguistics, 2022.
[59] Roman Vershynin. Four lectures on probabilistic methods for data science. ArXiv, abs/1612.06661, 2016.
[60] Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. Anchor points: Benchmarking models with much fewer examples. ArXiv, abs/2309.0866 2023.
[61] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@ EMNLP, 2018.
[62] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max W.F. Ku, Kai Wang, Alex Zhuang, Rongqi "Richard" Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark. ArXiv, abs/2406.015744, 2024.
[63] Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge. ArXiv, abs/2410.21819, 2024.
[64] Adina Williams, Nikita Nangia, and Samuel R. Bowman, A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACI-HILT, 2018.
[65] Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: a flexible and expandable family of evaluations for ai models. ArXiv, abs/2310. 17567, 2023.
[66] Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models. In Conference on Empirical Methods Natural Language Processing, 2023.
14

---

[67] Jeffrey Zhou, Tianjian Lu. Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followin evaluation for large language models. ArXiv, abs/2311 abs/2311.07911, 2023.
[68] Jin Peng Zhou, Christian K. Belardi, Ruihan Wu, Travis Zhang, Carla P. Gomes, Wen Sun, and Kilian Q. Weinberger. On speeding up language model evaluation. ArXiv, abs/2407.06172, 2024.
[69] Xiao Zhou, Renjie Pi, Weizhong Zhang, Yong Lin, Zonghao Chen, and T. Zhang. Probabilistic bilevel coreset selection. ArXiv, abs/2301.09880, .09880, 2023.
[70] Vilém Zouhar, Peng Cui, and Mrinmaya Sachan. How to select datapoints for efficient human evaluation of nlg models?, 2025.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

This paper is mainly an empirical work and doesn't provide many new theoretical results.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Appendix B and the code.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
We release our codes in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuia wwwiii/gguie///eeeeeeoecc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://eeur: http://s.....pbbic/cchcccccheaee
Answer: [Yes]
Justification: Authors have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper doesn't release any new data model.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used models and datasets are well cited in Section 4
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppsippteeeee www.rreswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [NA]
Justification:
This paper doesn't tt provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper doesn't involve crowd-sourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper doesn't involve crowd-sourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
21

---

Justification:
The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy hettap///aaarriio.cccc//ootrroep for what should or should not be described.
22

---

A Details of Benchmark Prediction Methods
A.1 Problem Formulation
We repeat the notation and the problem formulation here for the reader's convenience.
A benchmark is represented as a triplet FD,F, 8).
D represents the benchmark data with [D] N data points. data point is referred D, where z = (z, y), refers to the query and refers to the ground truth answer.
F refers all potential models that can be evaluated on the benchmark.
represents the metric of the benchmark.
s(f.2) refers to the performance any f E FF on any data point E D. For example, s(f.2) [[f(z) y) if the benchmark uses standard accuracy the metric.
s(f,D') = D" EEED' sf,,)) 2) represents the average performance of E F on any D' D. s') [{s(f=)) DED represents the vectorized performance of f E F on all data points in D' C CD and s(F'
represents the vectorized performances of all models F C on data point E D. (s(F",=)) as the performance matrix of all models in F' C on all data points D' D.
S(F', D') = ((ff."))reF'
refers set of source models, whose performances on every data F(() {fi.... .f...... ........ fm]
point of the benchmark S(F(+), D) are known.
The rest of the models are referred to as target models F()) F \\   F(+), which can only be evaluated on at most n & N data points to save computational costs.
Benchmark prediction with fewer data aims estimate s(f, D) for every f (F() with only n data points. In practice, benchmark prediction often involves two steps: @ identifying representative core-set 'D with |C| n data points, and @ learning a performance estimator to estimate the average performance on the full benchmark based on the core-set. Formally, the goal of benchmark prediction find and to minimize the estimation gap over target models,
estimation gap:
[FOO
Σ - [D]]] SEPO
-
For simplicity, in the remainder of the paper, we will denote the estimated performance of target model f F() as h(f), instead explicitly writing CIs(f.C),
A.2 Benchmark Prediction Methods
Previous methods In this paper, we examine five widely-used benchmark prediction methods,
RANDOM-SAMPLING randomly samples a subset as C and directly returns the mean performance,
ARANDOM-SAMPLIN (ʃ) = *(f.)).
(8)
If the benchmark metric s is standard accuracy, the gap with high probability based on Hoeffding's 's inequality.
((f is bounded by O O 1(/1/)
ANCHOR-POINTS- DON--III--WWWEHHHDD 660) treats benchmark prediction k-medoids clustering prob- lem. The selected medoids are used as C, and a weight vector 0 E IR" is calculated as the normalized cluster size of each medoid. The final estimate for any target model f e F(() is
ANCHOR-PoIse PANNS-r- (ʃ) s(f
ANCHOOR-POINTS PRINITS-PREDICTOOR 660 extends ANCHOR-POINTS-WEIIII
Instead of directly
returning the weighted sum, a linear regression model g(*(f.C)) is learned to predict s(ʃ, D C).
PANCHOR-POINTS- (f) = g(s(f.C))
(10)
1 Σ
where g -arg min
||s(], D -
(11)
M
FEF())
where note that g(s(f.C) is (N nn) dimensional vector and g(s(f.C) its mean.
23

---

P-IRT 43 extends ANCHOR-POINTS- PREDICTOR by replacing the regression model in equation with an Item Response Theory (IRT) model. Following the notation for ANCHOR- POINTS-PREDICTOR we estimate performance for any f F(() follows:
N - g[s(S,C)) N SU.CC).
ht P-
(
(12)
N
43 further generalizes P-IRT by combining its estimation with ANCHOR-POINTS- WEIGHTED as weighted sum,
GP-IRT
h GP-IRRT (ʃ) AA AAANIOOKPONNTSSWWEWIIDDD (f) + PIRR- (f),
(13)
where À is chosen heuristically control the error of P-IRT.
New methods We introduce six methods that have not yet been applied to benchmark prediction.
RANDOM-SAMPLING LEARN randomly samples subset as C and adopts Ridge regression model g for estimation as follows,
h RAADOM--AMMPR SAMPLING LERR (ʃ) - g[s([[C)]
(14)
where 9 = arg min Σ
-
(15)
M
SEFO)
LANDOM-SEARCH-LERS performs RANDOM- SAAMMM-AAMPLIGLL-AAAAAA for 10,000 times and selects the pest-performing subset C' based on cross- voossvalliation. A Ridge regression model g is then trained and used in the same way as
LASSO trains a Lasso regression model with weights A E IR as follows,
h LASSOO (f) = s(f.))'
(16)
where arg min
(s - 8(J, Di)2 X|10111
(17)
T
SEC
where À is selected so that only n dimensions of 0 are non-zero and Pg is the non-zero slice of PROUBEE-OPTMIZA optimizes both a subset selection vector π R and a linear regression model with weights f IN with gradient descent as follows,
h DOUBLE OPTIMMIE (f) [s(f. D) (T.)))
(18)
where n, 0
arg min {[s(f. D) Top))s ((F
(19)
x'.0'
where refers to the bitwise multiplication between two vectors, and TopMask( replaces the top largest values of n' with Is and the rest with Os. We directly pass the gradient TopMask(T's n) to T' during optimization following the Straight-Through technique [27
Principal Component Analysis (PCA) treats benchmark prediction as matrix completion problem.
This method assumes the performance matrix S(F, D) is of low rank. By randomly sampling subset as C, this methods conducts PCA to impute the missing values for target models [59] 6). As more intuitive view,
could also take the acquired principal components model capability indicators [50], i.e., the (M x k) erA-trransfommed scores indicate the k-capabilities of each model, while the (I x N) principal components represent the capability requirements for each data point. We select among (2, 5, 10, 20) through cross-validation. The Pseudo codes are in Algorithm|]
Augmented inverse propensity weighting (AIPW) 48): Inspired by the application of prediction powered inference | to the aua--a--- setting 14, we apply a more general AIPW estimator to benchmark prediction. We train Ridge regression model for every target model which predicts the point-wise performance s s(ʃ, ,
based on s(FF(e), ;). Formally,
g = arg min
Σ [g'is([ ((((())))))) 2)] s(f. =)).
(20)
g'
n
zeC

---

The idea behind the AIPW estimator is to use the predicted performance $(f,2) g[s( as proxy score estimate s(f. D) and "debias" that estimator follows
hAIP (f)= = ((J.C) 1 N-n
1
E SED-C
8(f.2) - 1 ((1))) n SEC
(21)
Unlike the other learning-basee baselines, AIPW is consistent estimator for s(f. D) Compared to RANDOM-SAMPLE
reduces estimator variance by factor of up to I+ I+ p(S f, 2,2), 2))² (14), where p the Pearson correlation coefficient. Recent research [37 shows that AIPW estimator will outperform random sampling and only the correlation between ((f,z) z) and s(,, z) is above certain level that depends
Algorithm PCA Impute Process
1: Input: Data matrix with missing values
2: Parameters: number of components max iteration max_ _ixtr stopping threshold tol
3: Output: Imputed data matrix
4: Step I: Initialization
5: Compute initial values for missing entries using column means
6: Step 2: Iterative Imputation
7: for iteration f- 1 to max_iter do
8: PCA Decomposition:
9: Perform PCA retaining & components
10:
Transform data to the lower-dimensional space
11:
Reconstruct the data from the lower-dimensional space
12:
Evaluate Convergence:
13:
Compute the norm of differences between imputed and original values at missing entries if norm < tol then
14:
Break the loop
16:
end if
Update Imputed Values:
18:
Replace missing values with reconstructed values
end for
20:
return Fully imputed data matrix
25

---

B
Additional Experiment Setup
We select a diverse range of benchmarks from the following sources
HELM-Lite benchmarks 35
OpenbookQA (39 N = 500 data points.
GSM8K 19] N 1000 data points.
LegalBench 22 NN- 2047 data points.
Math [26]: N 437 data points.
MedQA N 1000 data points.
MMLU 25 N 567 data points.
We obtain the per-data point performances of |F] = 83 models from the official leaderboard. Note that Helm-Lite often only uses a subset of the original testing set for each benchmark to save compute.
GLUE benchmarks 161
MRPC 13: N 408 data points.
RTE 11
18
N 277 data points.
S
872 data points.
MNLI 64
N 9815 data points.
QNLI 46 N 5463 data points.
We use the per-data performances of [ = 87 models provided by AnchorPoint 660
OpenLLM benchmarks [16]
IFEval 67 N 541 data points.
Math 26 N 894 data points. Only level 5 MATH questions are used in OpenLLM.
MMLU-Pro N 12032 data points.
Arc-Challenge
g = 1172 data points.
BBH [58] : N = 5761 data points.
GPQA 47 N 1192 data points.
MUSR 56 N 756 data points.
We use |F
and ImageNet 51 We collect | 110 models from Pytorch Hub 10 and evaluate them on ImageNet with 50, 000 data points.
448 models provided by Huggingface and collect their performance scores.
For simplicity, we report the overall average accuracy directly for MMLU, MMLU-Pro, and BBH, rather than the weighted average accuracy computed across sub-tasks. Alternatively, one could apply benchmark predictions separately to each sub-task and then calculate the weighted average accuracy.
"Since P-IRT and GP-IRT requires a 2) to be binary, we only use benchmarks with accuracy as metric.
'The provided score file for QQP is broken so exclude it.
htttpe:/huggingfaceee http://gaa.a.eeee/ppp/////appeee 11m -ne.eeeebeedrrbrrrrrpeee... 11mm leaderboardif
26

---

Table 2: Training and inference time of each method on ImageNet with N 50000 data points and [F] = 110 models. Training based on source models, and inference is on 27 target models.
Training Time (s)
Inference Time (s)
RANDOM-SAMPLE
0.00
SANDOM-SAMPLING LEARN
0.02
0.00
PCA
0.59
19.20
ATPW
0..0
0.27
RANDOM-
RANDOM-SEARCH-L SEARCH-LEARD
81.02
0,00
LASSO
105.58
0.01
DOUBLE- DOUBLE-OPTIMIZA
4.88
0.00
ANCHOR POIIOR--OONTS---
WEIGHTED
84.26
0.00
CERIOR-POINTS-PREDEEE
197.71
0.26
P-IRT
585.72
0,90
GP-IRT
1750.20
0.89
Table Average estimation gap between the predicted rankings based the coreset and the actual rankings based on the full benchmark, measured by Kendall's 7 (f). The results are averaged over all benchmarks.
Interpolation 50
Extrapolation 50
F1 10
20
100
1
200
10
20
100
200
RANDOM- CANDOM-SAMPLIIG
SAMPLING
0.52
0.61
0.70
0.78
0.84
0.36
0.43
0.53
0.63
0.73
SANDOM-SAMPLINGLAAN
0.57
0.66
0.75
0.86
0.07
0.12
0.18
0.27
0.36
PCA
0.55
0.63
0.72
0.78
0.83
0.04
0.10
0.21
0.40
0.57
AIPW
0.52
0.62
0.72
0.79
0.84
0.33
0.40
0.51
0.61
0.70
RANDOM-SEAREE SEAMM-SSERRAAAAAREREAE
0.66
0.70
0.76
0.82
0.86
0.13
0.13
0,20
0.29
0.38
LASSO
0.68
0.71
0.77
0.81
0.82
0.05
0.06
0.12
0.19
0.22
DOUBLE-OP
0.66
0.36
ANCHOR POINTS WEIGHTED
0.65
0.70
0.76
0.81
0.85
0.37
0.41
0.50
0.60
0.69
P-RTT
0.52
0.58
0.71
0.80
0.87
0.28
0.31
0.42
0.56
0.69
GP-IRT
0.53
0.59
0.72
0.80
0.86
0.28
0.33
0.45
0.59
0.71
C Additinoal Experiment Results
C.1 Detailed Results
In this paper, we experiment with n e (10, 20, 50, 100, 200} under both interpolation and extrapola- tion settings. The detailed results, along with standard errors, are reported in Figures 7 g
33
and
C.2 Running time
While some opted to run all methods without them, they sufficiently fast standard hardware. Table 2 presents the training and inference times for each method on ImageNet. Among the models, GP- IRT is the slowest during training because it involves fitting large Item Response Theory (IRT) model. During inference, PCA is the slowest, as it requires multiple imputations of the entire matrix. Although AIPW needs training separate regressor for each target model during inference, the regressor is small, making the inference process remain efficient.
benchmark prediction methods could potentially benefit from use GPUs, we
C.3 Ranking Preservation
We further compare the predicted rankings of target models with the actual rankings based on the full benchmark using Kendall's r. Specifically, we calculate Kendall's r each random trial and average the results over 100 trials. Our conclusions mostly remain unchanged, with almost all benchmark prediction methods outperforming Random Sampling under interpolation, while none can surpass RANDOM SAROOM--APPLLLE SAMPLING under extrapolation.
27

---

5:7
5C
6.5
87
IEIK Opanhonete
30.35
=0.12
20:32
#3.13
20.13
10.24
=0.12
20.32
2:0.11
20.14
#:.14
(0.0)
[:31.]]
(-29.4)
(-15 2)
(-40.7)
[-1.4]
[-29
(-18.1)
(:37 0)
(-19.3)
(-20. 5)
30.3
822
9.9
9.1
T/8
14.5
5.7
7.9
1022
10.3
11
AEUE GEMMA
10:38
=0:15
10.30
1:
40.14
45.27
20.15
20.35
2:0.13
20:18
10:38
[6.0]
[-19.6)
[-1.0)
(:10.6]
6)
(43.8)
(-17.1)
[-444]
(-21.8)
(0.7)
(-0.1)
12.7
5.4
67
10.3
5.0
67
6.6
100
8.6
8.6
BELK-LopolBecck
(:33.)
(-47.0)
8-
(.47.1)
(.233)
(-32.5)
(:32.4)
8.5
1133
101
HELN-HLLB
10.18
10:15
10.15 15
1:.16
15.15
#3323
10.15
10.0 in
+:.14
:6.18
1:.18
(0.0]
[:15.2]
(331.3)
(.222)
(1200) 0)
(57.7)
(:14.3
(23.7)
(11.21
(1.7)
(1111
11.3
7.5
8.0
2.7
6.7
9.6
8.3
9.3
6.4
8.7
8.7
-0.
=0.13
=0.14 14
#:.17
=0.13
#:5.10
=0.15
20,00
45.12
20.15
+:0
(0.0)
[-37-4)
[-S1--1 3)
(-18 M)
(-43%)
6:19 S)
[-30
(-21 7)
(-46 11
(-26.7)
(221 51
11.7
5.6
0.2
B.T
4.8
5,7
T.1
0.5
4.7
B.c
7.0
HIEN MHED
20.30
+0.11
=00.11
+0.09
#5.15
10.1
24.14
=0.2
#:014
[-51.7)
8)
1)
(-89.4)
(-92.2)
11
=0:13
=10.33
30
=0.13
=0.14
6.7
20.3
+0:33
Eximation % (Roduction over RANDOU SAAIPLzN %5
(0.0)
(-48.5)
(-47.4)
(-32.3)
(-54.6) 6)
(-2.6)
(-33.3)
(-41.2
(:54.1)
(-50
(-50.1)
12.3
3.9
4.7
%
3.
6.4
4.3
5
44
4.4
CLUB-ATE
10.10 14
10.1
+0.11 11
+-3.15
+-.11 11
+..4
+0.
1015
(0.0]
[465.0]
(1.4)
(...0))
(-734)
(.61))
[-47.5)
(45.3)
(3.22
( (.240)
(.64.)
10.5
5.3
7.2
5.0
4.5
11
6.1
5.0
4.0
5.5
5.5
12
OLTE-E8T-2
20.11
80.11
1...1
20.13
20.1 13
+:0.3
10.0]
[-40.2]
(-31.6)
(-33.7)
1-57 1)
(13.5) 51
[-417]
(-44.0) D3
(-G1-7)
4)
(-47.6)
6.0
7.7
4.6
5.6
5.C
5.0
(-48.1)
(-57.5)
(:50.9)
(55.5)
4.2
3.3
4.5
GUBE QUL.
10.22
+0.09
-0.
1:3.8
-0.09
+0000 09
+0.15
15
1...0
10.13
+0.13
[0.0]
[0777)
(0.9)
(-40.2)
(.74.3)
08.891 5)
48.33
(6..7)
(:75.4
(82.3)
(02.3)
10.7
6.7
7.3
D.1
577
1232
7.5
115
6.8
0.3
OpeniLN- 1PEvai
10.08
=0.05
10.25
+.0.07
=0.05
1:3 50
00.05
10.00 57.37
1..05
20.06
+5.06
(0.M)
[:37.7]
[-31.9) 9)
6-14
[-46 7)
(14 2)
[-10
(7:5)
(337.0) 0)
(-10a)
(11722)
-10
4.6
3.2
3.4
3.7
2.3
3.6
2.9
5.0
4.4
4.3
OpunLLN-Hust
#0.30
+0.03
+0.04
#:304
=0.03
20.03
20.05
+3.05
[330:1]
8)
()49.)
9)
(28.4)
2)
[-622]
10.5
3.9
+4.04
=0.05
50.00
+0.00
[0.0]
(-45.0)
(-3.6)
(:18.0)
(-64.2) 2)
(-18.4)
[-36 9]
(2.8)
(6.0)
(-23
(-23.1)
13.3
6.0
8.
4.4
5.
13.
4
8.5
0pwnt17--R8 12C-1 FRC
+0.30 14
-0.06
+0 07
10. 1
10.05
4-000 08
007
+0.09
(8.0]
(...5)
(..08)
(34.6)
(-26.6) 6)
(3.44
56.5)
[0.8]
66.3)
(:354)
(336.2
12.3
5.7
8.3
10.5
4.1
5.5
7.
7
4.3
4.3
OpentLy 2011
=0.04
$3.08
-0.03
1005
+3.03
10.07
1:0.07
-8
[0.0]
5)
(14.3)
(:00.9
3)
(.4.9)
(3.0)
(6677
(2243)
(22.9)
11.7
2.5
10.8
2.0
8.7
OpenLL
0.006
67.007
[-78.5)
4)
(-8-4)
(+52.8
9)
[-30
(-16.4)
()
(-21
2.7
8.9
2.3
9.2
5.3
3.0
=0.0
=0.02
#0.02
+0.07
=0.02
#0.02
=0.04
20,37
+3.02
=0.04
+0.04
[0.0]
[-78.1] 1)
(-78.9) 9)
(22.5))
(-51 7)
(-78.5) 5)
[-64
(-26.4)
(-90. 1]
[-52
(-52.5)
10.2
3.4
4.5
5.U
2.8
3.b
3.7
31.5
2.3
5.6
5.b
Imagstet
+0:36
-0.06
+0.07
1:
40.05
+0000 07
+0.05
#0,
+3.04
40.09
+0.09
[6.0]
(166.1) 1)
(-56.2) 2)
(-33.2)
(:72.0)
(-64.7)
(-63.4)
(12.5)
(-77.33
(-44
(.45.3)
11:0
5.4
6
8
4.5
8
86
7-3
7.
Average
(-48.2)
(-5.8)
RANDOM HANDOM-SAMPGARG SAMPUNNG
RANDOSS-SASPLAAE LEAAS
PCA
RANDOM: Sp. ABCTI LeARS
AIPW
Docmie: Oprontze
Lasso
ANCOUR- POINER W WitaarEe
AscHOn- Posttss Possts- Poennceoo
P-HIT
GP-IRT
Figure 6: The estimation gaps (L) for target models (calculated as equation I under interpolation model split, where source models are identically distributed with target models. Each target model can only be evaluated on = 10 data points. We also report + the standard error of the mean and the estimation gap reduction (4) over RANDOM-SAMPLAN implies that the method achieves lower estimation gap than RANDOM- AADDM--AMPPLIGG... % is omitted. Best viewed in color.
parentheses. A negative reduction
28

---

477
4.1
IEIK Opanhonete
10.11
=0.1
=011
$3.00
+0.1
20.2
=0.09
20.38
1:200
20.11
20.1
(0.0)
[-16.6]
(-18.9)
(-14.0)
(-16,6
(14.3)
[-22
[-9.2]
(:22.6)
(22.9
(-9.7)
0.0
6.3
0.4
8.1
5.7
30.5
6.U
7.0
5.9
7.7
7.3
AELK GEMMA
10.32
10:13
-011
1:2:1
=0.13
45.2 22
=0011
4:111
-1014
#0:13
[8.0]
(:8.9)
(-7.5)
(-11.7)
(-17.5)
(53.7) 7)
[-13.4]
(10.8)
(-0.1)
(11.5)
(8:4)
9:5
5.1
3.6
5.5
4.1
5.6
7.2
0.5
5.6
BELK-LogolBecck
(-46.5
(-39.4)
(-55.4)
(:35.0)
(:37.4)
5.0
8.4
HELH-HLUD
10. 12
10.1
:0.11
+:3.11
-0.11
+3.11
12.12
+:5.12
#0.14
13.13
[0.0]
(1:-)
(-36.3)
(:11.5)
(9.5)
(w0.3) 9)
(226)
(24 4)
(17.0)
(24-3)
(12.0)
222
5.0
5.8
5.6
5.5
7.7
6.3
6.E
5G
7.3
8.0
10.34
1.0.1
=00
4:2.12
2.0.1
4.O.U 16
=0.12
=0.11
+.0.1
20.13
#5.12
[0.0]
1-27.61 6)
[-29 7)
6-19 11
[-35 0)
(-5-0 6)
(225
(-17.0)
(:29) 6]
(-10.4)
(:15.4)
8.1
4.5
5.0
5.0
4.2
4.5
5.2
0.4
4.3
6.3
3.0
HIEN WHED
10.34
+0.09
=00.00
#-0.11
00.08
+:0.13
=01
+3.00
=0:11
#0.1
(0.0)
[-48.8]
8)
(-47.1)
[-22
50:34
+9009
00.08
=011
20.11
+0.3
Eximation Gop S (Roduction over RANDOL Wwwooo--SA.im SimPLaa %)
(0.0)
(-46.5)
(:50.4)
(-39.5)
(-52.6) 6)
(-13.2)
(-37
(-44.1)
(-48.4)
(-42.5)
(-45.5)
7.9
5.0
3.3
5.7
2.5
5.6
4.7
3.4
35
3.5
CLUB-ATE
10.13
0..08
10 d4
+0.1
10.09
+0.
101
001
0
+0.1
(0.0]
[.61,6]
(8.65
(.5.7)
(.33.) 4)
53.9)
1 (-40.)
(55.0)
63.5)
( (5.44
(56.2)
2.0
3-8
4:1
4.4
3.5
4.4
4.5
5.4
4.5
4.6
OLTE-E8T-2
10.13
=0.08
09
10.08
#5.11
00.09
-01
5.007
Li
10.1
10.01
[-44.8)
4)
(-36.0)
(:50.2) 2)
B)
1-37.0)
(-38.00 03
(-518)
1:30
(-33.3)
4.4
3.8
3.8
(-46.1)
(20.4)
(-55.3)
(.48.3)
3.5
3.0
3.5
3.5
GUBE QULI
10.14
+0.09
-0.1
$3.11
-0.08
+0.00
+0..1
-0.1
+.007
-0.
10.1
[6.0]
[.0.0]
(8.3)
(:47.0)
(.5.6) 6)
(0.3)
(4449)
(50.7)
(0.22)
(:562)
(56.8)
7.7
5.0
5.3
611
4.5
DE,
5.2
3.00
08
OpeniLy 1PEvai
10.06
=0.04
+0.04 24
+..04
=0.03
#3.00 07
20.04
20.26
2.3.04
20.05
+5.05
(0.0)
[:35% 11
[:30 7)
6000 01
(-41 4)
(286))
[-31
(10.0)
(17.5) 91
(43:2)
(:0.b))
3.4
2.6
2.8
2.0
2.5
2.5
2.0
4.1
2.8
3/8
3.4
OpunLLN-Hush
$0.00
+0.03
+0.003
#3.03
=0.02
20.2
+0.04
[-
()40.8
(50
(6.3)
[-0.2]
3.3
7.8
=3:03
+0.04 04
=0.04
t0
[0.0]
(-42.4)
(20.6)
(-22.8)
(-56.6) 6)
(-4.0)
(-37
(L.B)
(-43.5)
(-7.5)
(-6.6)
8
4.7
5.5
3.7
4.3
9.6
6.0
1pwnt17--m FRC
+0.33 04
0.05
+0.00 06
+5.00
+0.04
4-000 07
10.00
10
+6.00
(6.0)
[ (.468) 8)
(-43.8)
(:37.5)
[:58.1) 1)
(-43.))
50.6))
(9.5]
53.6)
1. 1.30.7
(:31.0)
8.4
4-4
7
5.9
3.5
5
4.9
100
4
6.8
6.6
OpentLy 2011
10.00
00.03
$3.05
10.63
10.04
10.05
[0.0]
[.47.)
3)
(8800)
(:30
(332-7)
(
(30.5)
52.4)
(.3.9)
(18.5)
2.2
6.4
OpenLL
10.00
$0.05
=0.01
20000
[-12.2]
(-N8-7) 7)
(111
(-2744
8)
(226
(-20
(-08.0) 0]
(151)
(3:3)
2.4
5.5
2.1
6.8
4.6
QUaaLLH-HH2BR
#0.06
=0.02
#0.02
$0.04
=0.02
#0.02
=0.03
#9.00
#2.02
=0.03
#0.03
[0.0]
(-71.5)
(-57 4)
(-35.00 0)
(-7544
(-72.80 5)
[-56
(-19.0) 0)
(-71.8)
(-45.5)
(-48.6)
7.b
2.8
4.4
4.4
2.4
2.8
310
25.3
2.1
5:3
5:3
Imagsiet-
+0.32
+0.05
=0.07
+0.00
40.04
+.00 00
=0.05
-40
4..04
40.00
+:
[6.0]
[-02.7]
(-42.7)
(-42.5)
[-68 8)
3)
[-60.5)
(-72.7)
1:30
(-30.6)
4.7
3.
37
4
5/6
5.
Average
(-43.0)
(:2..)
(-503)
(:3,3)
RANDOM RANDOM-SAMPGARG SAMPUNNG
RASDOSS-SASPLACE LEAARS
PCA
RANDOM: Sp ABCTt LeARS
AIPW
Docmue: Oprontze
Lasso
ANCOUR- POINER W Winamre
AscHOn- Postts- Possts- Pronnecoo
P-HIT
GP-IRC GP-IRT
The estimation gaps (L) for target models (calculated as equation I under interpolation model split, where source models are identically distributed with target models. Each target model can only be evaluated on = 20 data points. We also report + the standard error of the mean and the estimation gap reduction (4) over RANDOM-SAMPLAN implies that the method achieves lower estimation gap than RANDOM- AADDMM-SMPPLING... % is omitted. Best viewed in color.
Figure 7:
parentheses. A negative reduction
29

---

3.4
311
2,3
2.9
3.0
3.4
288
3.7
4.5
3.5
3.0
-8.06
+3.07
-0.06
-0.06
+0.07
-0:31
+5005
00.08
40.06
(0.0)
(-9.3)
(-18.0)
(:15:4
(-12.2)
(-0.4)
(-19.1)
[7.2]
[30.8]
(3.4)
(-12.2)
77
38
6.5
56
6.1
4.0
BELK-CENEK
-0.08
(0:0)
(10.5)
(1339)
(111.5)
(33-7)
(1109)
(29.7)
(37
1:9.51
57
388
3.0
3Y2
3.5 +0.00
REIX- Lagg-Lagg
10.1
+:0.07
=0.1
-0.07
+-O. 06
=0008
=:0.0 05
+0.00
=0.07
+:0 06
(0.0)
(:36,6)
(:24.7)
(:33.0) 03
(-42.3)
(:31.))
(-30.5)
(-7.0)
(-2.88
(-37.4)
(:38.8
4.1
3.0
3.7
5.8
5.9
4%
1.6 10.07
HELK-Huth
-0
10.07
-0.07
0.
8-0.0 D2
-00 2.
10.00
=0 11
+:0 07
(0.0)
(444)
(.3.5)
(600)
(.5))
(44.
(-4))
(43.8)
[081)
(4.2)
[.77
5.1
4.0
3.8
4.0
4.1
4.5
3.9
4.7
522
4.2
4.0
=0.00
1:5.07
=0.07
11
1.0.07
(0:0]
(:2 11
03
6:21 M)
(220 01
6-31
(:24 21
4:
11 4)
(:17.00
(:21.5)
3.5
2.9
3.
3.6
=0.08
+5.00
+0.00
(.1..)
(30.8)
40.61
(34.8)
OUTE-MRPC
=0.09
5-2
+-0.08
:0.06
=0
3.0
+00000 06
20.30 0..05
+00 05
+0.07
=0 06
3.3
+:0 07
+0.06
3.0
Estimatonn Cap S (Moduction over MA.DOOM SAMPLING N)
(0.0)
(-92.0)
[:47.00
[-42.5 5)
(6456 31
(1278))
(:35.5) 9)
(334.)
(:35.7)
(:35.
(-41.3
522
2.3
2.3
2.6
2.2
2.4
2.8
2.2
2.5
2.3
2.2
MUIK- RITR
-0.00
+5007
=0,07
06
4.00 06
=0.07
200 06
15,07
=0
4.0 05
10,06
(0.0)
(-54 R1
[-54 9)
(-50 6)
(-58 5]
(-536)
(-46 4]
(-57 21
(-50 4)
(-55 71
(-57.5)
4.5
2.7
2.5
2.7
2.3
4.3
2.7
3.1
2.7
3.3
2.8
2LU5-89T-2
=0.08
+5.05
=0.05
=0.07
05
07
10.06
(0.0)
(41.0)
(-4..5)
01
(0.9)
(.1..)
31
(-11.5)
(-30.0
2.8
3.
2.3
3.2
=0.00
+0,000
-0. 00
+:000
(0.0)
(-09 01
(20-1)
(-495)
(-57
(-7.7)
(-44 0)
(-M0
(-58-1)
(-47 4)
(-47
5.3
2.7
2.5
2.2
2.5
GUEE ONLI
=0.09
+0.07
=01
=0.07
+0.00
=0.00
+0 07
10.07
=0
=0.00 07
+0.07
(0.0)
(49.0)
(93.)
[44b4] 4)
(4.31
(40.)
(4.00
(53.6)
(5915)
(6.11
( 68.3)
4.6
3.2
3.5
3.6
3.0
3.2
7.3
36
36
3.6
OpenLLM 1P2YaL
=0.03
+5.00
=0.30
=0.03
+.0000 02
-0 25
200 02
10,05
-0.04
+.0
10.03
(0:0)
(:28 6]
[:21
(:20.5)
(:31 21
01 83
(229 b)
(W
(27
(:21.1)
2.2
1.8
L-9
1.8
1.5
L4
13
2.5
2.5
2.6
2.1
OpentiM
=0.02
+:0.02
=0.00
=0.02
+3.02
=0.00
10.02
(0.9)
(-18.2)
8)16.8
2]
(-574)
?)
5)
(20.9)
(15.6)
[-4:2]
3.5
OpeniL -PL--P
=0.04
=0.03
10.00
04
1:0004
10.03
(0
(-1..)
(:27
(-10.0)
(31.)
(-4b
(21.1)
(-250)
(-13.2
(14.5
5-2
3.1
3.3
3.2
277
3.8
3.4
OponLLY-ABC
=0.05
#0.03
20.03
=0.04
+0.03
=0.04
20.03
+0.00
-0.04
10.04
+0.04
(0.0)
(-41.2)
()
(-37 5)
(-41 9]
(-59.5
(-46
(24.2)
(-76-)
(-37 5)
(-54.5)
5.4
3.0
4.5
4.2
2.0
3.5
3.0
6.8
3.4
3.7
3.b +0.00
OpeniLyy mm
-0.04
+00 22.02
=4,00
-3.03
+0.0 OZ
-0.03
#0 02
$0.00
=0 00.03
1:0 03
(0:0)
(-44.0 0)
6:10 73
(:2522 2)
(
(333.2
(:
(25.5
(:375)
(:124)
(:31.1
4.9
1.8
2.1
4.2
1.6
L44
39
4.3
2.2
4.1
3.8
OpenLLM-OPPA
=0.04
+:3.01
=0.00
=0.03
=0.01
+0.03
10.03
10,03
[0:0)
(161.5)
(-71.3)
[-12-5]
(1580)
(-216)
3.3
1.7
2.3
OpenL1M
=0.04
+3.01
+0.01
10,04
+0,02
(0.0)
(-64.1)
(-56 8)
(-38.8) 8)
(-67.55 5]
(-57.6)
(-525)
(-6.8)
(-57.4)
(-46 7]
(-47.2)
4.5
2.4
1.9
2.0
1.9
2.4
InageHes
=0.07
+00.04
=0.07
=0.04
40.03
=0.05
+0.03
10.21
=0.03
10.00 04
10,04
(0.0)
(-54.1) 71
(:1.9)) 91
(-501)
(-61 91
(-51
(-06.8)
(117
(-51 7)
(-49 6)
(-16
4.8
2.0
377
33
2.7
3.
3.0
4.9
3.8
3.5
3.3
Averge =0.02
+5.01
-0.00 02
=0.01
40.0 01
=0.02
+0.0
100002
=0.01
4.0.0
40.01
(0.0)
(-37.2) 21
4)
(-304) 4)
1]
(-22.0)
(-36.1)
[4.4]
(-211)
0)
(-28
RANdoss-SAMMMA BAMPLING
RASDOM SAMPUUSO La AtAM
PCA
SANDOOH-SLAROR-- Laaoo---AAMM
AIPW
DovBLe- OpynutR
LAsSo
ANCHOR- AACHOR-PONYTS- PoRyss-W W TIOHTTO
Ascrun- Potsire Powsts- PastecTon
P-IKF GP-IRT
Figure& 8: The estimation gaps (4) for target models (calculated as equation D under interpolation model split, where source models are identically distributed with target models. Each target model can only be evaluated on n 50 data points. We also report +'t the standard error of the mean and the estimation gap reduction (4) over BANDOM-SAMPLIN in parentheses. A negative reduction implies that the method achieves lower estimation gap than Best viewed in color.
% is omitted.
30

---

BEIK-Opanbosiq
=0.05
#0.05
#5.08
=0.00
=0.1
20.04
(0.0)
[-4.0)
(-16.5)
[-13.]
[-8 5)
(3:7)
[-14.7]
(9.3)
(92
(-21.6)
(-28.2)
277
277
4
4.2
2.7
HELM GERAE
20.05
10.00 00
+0.04
+0.05
340
1:0.t
+::0s
-40C7
440
=0.05
(0.C)
(11.3)
(:15.4)
[-14.2]
(-10.4)
[96. (56.0)
[-10.5)
(39.0)
(90.3)
(-13.3)
(-0,))
BFIK Taghtteeeee
10.07
3-4
10:06
2.6
3.8
-0.05
2-6
10.
2-6
9.4
-3.05
2.8
10 06
4-2
2.3
00004
2.
4.5
(0.2)
(.2.3)
(:3100)
(:11.4)
(:332)
(10.3)
23,
(-40.7)
(-4.0)
2-7
2-7
4.2
0.5
00.05
(-73)
(117.9)
[-8.3)
(44.3)
(1M.1)
(47:3)
(124 M)
(:1..))
(-7.5)
3.0
2.
3.0
2.7
BELN-HAdQA
10.06
.-0 OB
+0.05
+ d6
+-0 DHa
+0
+40 C4)
00.05
(0.C)
10.21
2-13.2)
[-17.3)
[-3.3)
(7:7)
(-12.22
(6.9)
(47.5)
(-B8.1)
3:3
2.6
2.5
24
2-6
2-Y
3.0
1
2.3
HELN-W01
10.06
30.00
04.04
0.0.4
10:05
-0.04
(0.0)
(7.33)
(.20.))
1:24.11
1..991
(3122)
1:22.5)
(.43)
(30.6)
(:29:2)
(.27.7)
2.0
L-8
24
A.0.04
2-7
3.5
1.
Edimation Gup (Raductoon over RAAooo-EAMRRR MINIINGG 1o)
E:30
(-41.3
(-40
(-100))
(-92.9)
(-45.0)
2.6
2-7
GE
20.00
=0005
+0.05
=0.04
20004
#0.05
2..04
=0.05
20,00
20.04
=0.04
(0.0)
[-50 5)
[-47 6)
[-16.5]
(:54.4)
[-48 5)
[-40.4]
[-46 0)
(-12.4)
(-56. 7)
(-56.4)
3/5
1.5
1.6
1.8
1.7
2.9
2.1
2.5
2.4
1.7
GLE S5T
-0.05
+00000
+3.00
20.04
=4.04
#0
20.04
=0.00
2:0
2..04
=0.04
33.5
(0.0)
(-39.7)
[-44.8) B)
[-40:2]
[-44.L]
(:3.3)
(-30.5)
(-154)
(-19.4)
(-42.7)
[-43.8)
1.9
7.6
1.5
41
2.1
1.8
2.2
(.50.)
110L3)
(-46.1)
QLIE-OIL:
30.06
10.06
13.00
:0:05
10.05
10.00
:0.05
10:05
10.06
1.00
10.5
(0.6)
( 50.3)
(.1.5)
[.4.0)
( (5..5)
(5255)
1:3.71
(.52.)
(55.0)
(84.5
[ (9.1)
2.9
2.3
2.5
2.5
2.2
3.0
2.3
0.0
2.3
24
Open IEEraL
=0.02
30.00
+5.02
=0.02
30.02
+:0 03
2.0.02
.0..44
2.0
=0.02
[0:0]
(:20.2)
(:13.6)
(:14.0)
(:25
(2:7)
(:20.7)
(30% 1)
E70
(223.)
(:16.7)
1-3
Gpun
(.11)
3.5
20.03
+0.00
23.03
=0.02
10.01
03
+0.02
20.04
1:0
50.000
=0.02
(0.0)
( (40.6)
(38.3)
(:28.5)
[:40
C:20.37
[:99.7]
64K. 9)
[-0.7]
(:2433
(2253)
3/8
2.2
2.4
2.3
2.0
1.9
2.1
5.8
3.7
2.2
2.1
Dewnt.11-AR-
=0.04
(0:0)
[-42
#0.00
(:37 1)
#5.02
[-39 1)
=0.03
(-47.0)
#0.02
(-90 5)
#0.00
1-45 1)
20002
(54 7)
=0005
(-3.6)
(-47 6)
02
(-45
=0.02
-25
4.5
2,2
2.9
2.0
2.2
244
5.2
3.0
2.4
2.3
OpeELLK-EEE
=0002
#2.02
#0.02
#0.00
20:04
=0.02
[-48
(31.4)
(-99.7)
3.5
2.1
Open
40.02
+4.01
+0
10:02
60.02
=0.02
(0.0]
( 59.33
(-49.6)
1:11
(-50.5)
(-02.2)
[:11.0
(:7.1)
(-30.5)
(:25.
(-27.7)
3/2
1.6
2.0
1.5
1.5
3.5
3.2
1.8
1.4
Open11N WIST
10.05
+0.01
+0.01
10.02
10.01
+0.00
0000
10.05
+0.00 00
0.01
(0.0)
( 57.5)
(.46.1
(3333)
[ (.00.)
( 63.3)
(2.99)
(4.0)
L.30.3 9)
51.6)
48.8)
3.3
1-7
1-7
1-6
1.5
11.2
166
166
Imagetet
28.038
10.03
10:22
10.08
(0:0]
[15.3)
(-47.))
(-52.)
(50.b)
(243.00
(49.3)
(33322)
(111.)
Average
-0.07
+0.00
61.01
40.01
+.
+10 62
4:3.0
+0.01
(:30
(11..)
(-7.5)
(3164)
(-1844) 4)
1-28
12
(10.5)
(:37 9)
(-15.)
HARDOM HANDOM-SAMPSIRO SAMPSING
RASDOMM-SAMPE SAMPLAA-A -LEAA
PCA
RANDOSS-S Saanns---AAAA LpAAN
AIPW
Docaue- Docme-Orrait Orraurz
Lasio
ANcsooi- POINVE WRIImE
AscHoR- Pu Puserr- Puarrs-Paeticnt Pretactar
P-IRT
GP GP-RRT
The estimation gaps (L) for target models (calculated as equation T under interpolation model split, where source models are identically distributed with target models. Each target model can only be evaluated on n = 100 data points. We also report + the standard error of the mean and the estimation gap reduction (L) over RANDOM-SAMPI INNG in parentheses. A negative reduction implies that the method achieves a lower estimation gap than RANDOM-S SANDOM-SAPPLIGG % is omitted. Best viewed in color.
Figure 9:
31

---

1:5
122
1-4
25
1-6
1.5
74
L1
11
::302
-0.00
:0.03
10..3
(0.91
(:15:1
(:21)
178:21
[14-4]
(12.7]
(-22.)
(-30,0
1.5
2.1
=:200
=0.03
+0.04
(-9.0)
(-12.5)
(-13.5)
(-11-4)
(100.0)
12:4)
[4R.5)
(-10.8)
[-h
2.6
2.0
5-3
1.8
1
3.4
2-
3.4
2-3
1-6
1.5
BELI-CogalBBnnc
1:0.0
10.04
3:3.0
1:000
04004
0.007
0.04
0.000
10
10.05
10.05
(0.0)
(:2.7)
(23.7)
(:3244]
(:299)
128:3]
1:27.31
[31.1]
(-32.))
(:39.2
(40.4)
1.7
1-6
1.5
188
1.6
3.9
1.9
2.5
7.3
L-4
L-7
2:.03
2:0.03
5:0.03
-..00
(:13.4)
(7:5)
(127-6)
T9:00
(-171)
11:71
2.3
2275
FM.M
10:04
-..04
-3.04
0.09
00.68
10.05
10.05
(3.0)
(392)
(-B.5)
2:15.2)
(-9.6)
(98.4)
(-0.3)
(25.7)
(114.5)
(-23.))
(-165)
2.2
1.8
177
1.8
1.8
2.7
2.1
2.3
8.5
1.5
1.5
1..04
8:0.0 04
5.0.0
8:0.00
5:0.0
0Z
04
0.
-0.
:00.0
10.05
(0.01
(3:33)1
2/13:71
F22.51
(.6.0)
(
(290.00 01
(3324)
(:27.)
2.0
1/2
(-42.3)
#:002
12
(0.5)
2.0
-0.05
244
(:20.0)
1
1-7
8.2
(-48.0)
=0.02
L.5
(-44.0)
L.1
-2.00
Excmution Gup (Rodustion over Romoo SAMPLNNG MOTLING
QLIE-RTE
+0.04
+0.00
0.004
0.05
10.3%
10.3
10.06
10.02
10.00
(0.0)
(-25.5)
(-42.6)
(124.3)
(:23.0)
(8.6)
(15.0)
[-35.0]
(213.3)
(:5..)
(55.2)
10
1.1
1.1
1.1
2,4
1.7
3.3
1.1
L.2
OUTB-BTT-2
$3.08
+5300 02
2.0.0
5:0.3
2.0.02
=0.33
0.004
=0.0
=0.03
10.03
(0.01
6:39 ()
3:39-9
(-33.b)
(-42.5)
(110-4)
0:3.00
E26.0)
(-42.9)
(-30.2)
2.6
1-3
1.3
333
2.3
1.7
1
1.4
GLUG-NNLI
(153.0)
(-4..6)
[-10.9)
(-45.)
(-46.1)
CLUE-ONLI
$0.04
:0.04
13.00
8:0.03
0.004
-7007
04
=0.33
00.05
20.55
10.05
(0.9)
(448,0)
(22.8)
(.48.))
(.5.9)
11:23
(..8)
[40.0]
(36.0)
(440.)
(50.8)
1.9
3.5
1.7
2.0
3.5
300
2.0
4.0
4.4
1.4
2,000
Open LPRAA
+0.00
+:0.00
20.00
-1
-3
-0.02
03
=0 03
-0.01
+0.01
(0.0]
(:20.4)
2-13.21
(2.62
(:22.6)
[-14.3)
14:23
(109.9)
(128.8)
(-26.1)
(-13.7)
0.4
1.0
0.7
0.5
(-12.5)
(6
(:17.5)
2.5
18
1.3
OpenLLM MILU Pro
+0.00
=0.01
=3.02
-0.01
=0.01
=0.03
-0.01
=0.3
02
=0001
+0.01
1.75
[0.0]
(-35.3)
(7.7)
(224.0)
(-37.0)
(-37.9)
[-2122]
(84 71
[41 R3
(-26.6)
(-25.6)
2.5
3.5
1.7
1.7
1.5
3.4
1.8
4.7
3.7
1.4
1.5 +0.022
Opent2M-ARC
+0.00
+0
43.0
02
-2.02
12
02
D4
04
-10
(3.0)
(-30.3)
3:2.21
2:3377
(-42.5)
[-46.1]
1-29-]
6:
[47.8 8:
(-42.))
(-40.9)
1.6
2.0
1-
23
1-7
1.5
(0.0)
(-40
(20700
(26.2)
(-43.9)
[-49
(-15.5)
[44-6]
[2:81
(-37.2)
(-40.2)
16
2.7
1.7
Upen.Lit GPGA
+0.00
+t.00
+0.01
=0.0
=0.01
=0003
0Z
02
=0
=10.0
+0.01
[0.0]
(-46.2)
(:39.5)
(-3.7)
(-52.1)
[-61.8)
[19.0)
2:2.9)
(-16.0)
(-26.6)
(-25.6)
2-4
1.2
1.6
188
1.2
0.9
2.0
3.1
2.2
1.l
1.4
+0.01
+0.01
+0.00
-0.01
-0
+0.01
+1.01
(:31.6)
(-51.5)
[-63.6)
[-111)
Y1
(-47.)
31
13
13.6
1.22
ImageMes
=0.00
=0.02
+0.02
(3.0)
(-40.6)
(37.6)
(-45.1)
[-44.0)
[-50.00
[-42.8)
(50..0) 0)
[-34 5)
(-53.8)
(-SA.1)
1.5
21
2.1
3.2
1.3
Average
+0.00
+-0.11 1t
+0.01
-0.01
01
01
002
-00 02
+0..01
+0.01
[0.0)
(-28.2)
(4.9)
(4244)
(-32.4)
(7.8)
(.6.8)
153.21 21
(113.5)
(:35.9)
(-315)
RANDOsi-S SAMPPLSS SAMPULNN
RAamoss-S SAAinss--AAuuuuee SAsPupnn- LEAAR
PCA
RANDOM--SEAAe SEASOO.-SMMMH--AAA LAARN
AIPW
DOVBLE Of maze
LAMSO
ASCROR-Pure Poorss W Vetorreet
ARCHOR- Potern PREDICTTR
P-RRY"
SRT
Figure 10: The estimation gaps (L) for target models (caleulated as equation under interpolation model split, where source models are identically distributed with target models. Each target model can only be evaluated on n 200 data points. We also report + the standard error of the mean and the estimation gap reduction (4) over R RANDOM-SAMA APPLING in parentheses. A negative reduction implies that the method achieves lower estimation gap than CANDOM-SAMPLL Best viewed in color.
% is omitted.
32

---

5.
13
123
53
105
20.0
6.9
2.5
11.
6.1
5.6
40.00
0007
+5305
-0.06
40.00
.0.1
+0.00
1:30
-6.11
-0.
(6.0)
(10L2)
(110:4)
(:9.)
(80.1)
1242.90
(18.4)
(:52.)
(124.3)
(5.3)
(-4.0)
7.
152
275
151
33.7
12.6
6.5
SELK-CEMEX
+0.12
20.13
(200,81
(.0.0
(302.5)
19:0
(3300)
1190.21
(20.9)
(9.0)
14.2
10.5
REIK-LagalRecch
10.18
20.11
=0,37
#:.17
10.11
4.3.07
=0.14
20.16
1:207
=0.15
#3.16
(0.0)
[15.4)
(524))
(-18.
(:34 03
(2n 1)
[-31
(-9.5)
(-40 6)
(1.6)
(5:1)
9.3
20.2
20.9
9.3
L34
21.2
11.7
8.0
2280
10.5
10.3
HILM Mach
20.34
20.18
-0..77
#:0.4
20
#3.11
=0.17
20.00
$3.09
20.16
#0.15
(0.0)
(117:0)
(222:21
(-0-1)
(102.5) 51
13
(25.5) 5)
(-14.1)
(179
(1266)
(8.5)
10
23.0
2413
8.9
23.5
25.5
13.0
0.0
19.9
12.0
12.4
BELI Plccl
10:15
(116.3)
(128.8)
(-6.2)
(102.5)
122.3)
(-43.5)
(87.9)
(LB.7)
(17.3)
HELN-INED
16
10.1
14
0.11
+0.0
10:55
+0.15
(0:0)
[44.5]
(46.00
6-14
(30.3)
(46.5)
(-10.)
(-54.4)
6)
[-11
(-7.4)
9.7
32.9
34.3
7.2
2X
54.7
15.5
6.6
2
7.6
7.2
QUIE-15PO
10. 15.15
10.2
20.22
+:0.13
#:0.2
=..23
+0.12
2.3.2
20.14
+3.13
(0.0)
(237.8)
(263 41
(-25.5)
1308 77
(92.3)
(-32.5)
(188.5)
(-2L5)
(-26.2)
Estmmtionn Gup (Reduction over RANDON SAMPLNN
11.5
13.3
23.4
ADL3
12.G
13.3
12.5
21.4
L3.0
9.6
8.0
GLUB-ATE
40 17
10.18
16
1:016
+0
14
+0 18
16.15
+-.14
(6.0)
(14.8]
(20 3)
(-12.8)
(0.3)
(12.8)
(8.2)
(:1.8)
(13.0)
(-22.0)
(:23.2)
21.0
5.9
193
S2
7.6
6.0
7.3
2.2
23
(239.0)
(223.71 71
(300-11
(.33.3)
(5.1)
23.1
29.7
28.5
20.0
9.5
OUTE-MOTT
20.17
+0.36
20.36
#0.16
-0.36
2.5. 36
20.19
20.14
1:0.33
20.16
#0.15
(6.0)
(153.0)
(1ON M1
(-7.0)
(7A8 41
1152 W3
(2:1)
(-17-3)
(:25
(22.6)
[-4.0)
12.2
13.9
23.9
11.7
13.9
14.2
12.6
20.3
13.0
9.7
9.5
BUDE QUT.
20.10
50.13
-.0.33
4.0.11
=:0.1
20.16
1::13
20.13
+0.13
(0.0)
(13.9)
(15.2))
(-5.9)
(.5.8)
5)
(2.8)
(-10.4)
(13.9)
(22100
(:21 9)
32.0
31.3
33.1
12.0
30.5
60.8
22.6
2915
30.4
13:2
13.2
OpenILN-TFENol
1:0.1
#:011
200
(150.0) 0
(153.5)
0)
(10.3)
OpenlEM-Mech
40.3
05
+-3.05
00
0..05
0
+.3.0
10.09
+0.00
12
(6.0)
(42.8)
(53.6)
(-6.9)
(39.0)
(43.9)
(21.8)
(88
(15.5)
(36.1)
(26.5)
13.1
21.0
36.3
12:3
184
24.55
13.2
13.3
21.0
10.3
10.1
Opant12 SAUE Pro
10.00 08
+0.05
+-3.00 03
-0.05
06
00.07
10 06
05
-0.07
+0.07
[0.0]
(00.3)
(3000 51
(..5)
(46.0)
(97.3 5)
(1.00
[1.9]
[00, B)
(1.33
(2277)
13.1
L4.4
27.3
10.5
12.
19.5
12.0
J5
L5.D
10.8
10.5
OpeniLL--00 ABC
20.05
00.05
10.07
+:000
(9:7)
22)
5)
(1511)
5)
(-19.9) 9)
20.7
21.2
20.5
20.0
20.07
20.07
(0.0)
(62.4)
(84
3)
(36.2)
(-6.0)
(24
())
[-14
(-13.9)
74
'3.5
OpenLL G/AAA
20.30
+0.02
=0.02
8:0.U 1B
+00
+0.00
0b
+40
+:303
=0.00
+0.00
(C.0)
(:37.0)
(:36.5)
(-0.8)
[-42.2)
(-39.9)
[-16.6)
(-20 3)
(-37.0)
(-21.)
(-24.3)
19.0
Y-4
7-5
10.3
6.5
7.5
7.0
5.2
5.9
4.4
4.5
07
-0.07
+1.04
03.00
[0.0]
[-43.0]
(-44.0)
(:22.0)
(:30.0] 0)
(-43.8)
[-46.3]
41
[-47.2)
()363)
(46.2)
8.0
0
7.3
6.5
11:4
IrageTot
[1..0)
(223.4)
[:3
(-b.3)
(9.44)
(18
(18.6)
17.0
.60
20.5
9.7
Averae
10.03
20.03
=0.03
#3.03
G3
+3.00
20.03
20.
4:3 02
20.03
+3.03
(0.90
(72.7)
(5e 1)
(.45)
(16.9)
1122 7)
(8.2)
[-8.6)
(6) b)
(5.4)
(-7.0)
HANDOM--55 SAMPLING
RANNeIS-SASILLeN-- LEARS
PCA
RANDOR SEARCRR- LEARS
AIPW
LASRO Dorma- Oy mont
AncHion PoisTs SonnssWWaotte WZIGHrEn
AScIto- PRemecTO
P-IRE
GPP UP-IRR
The estimation gaps ()) for target models (calculated as equation I under extrapolation model split, where source models are the lowest- powest-performing 50%, and target models the top 30% based on average performance over the full benchmark. Each target model can only be evaluated on 10 data points. We also report + the standard error of the mean and the estimation gap reduction (L) over RANDOM-SA SAMLLING in parentheses. A negative reduction implics that the method achieves lower estimation gap than RANDOM-S PRROO-SAMPLLAA SAMLLING. % is omitted. Best viewed in color.
Figure II:
33

---

3.8
98
8.7
57
8.5
13.3
59
1,
17.1
6.1
4.8
40.00
+:305
-0.06
+007
+0.00
1:00
-01
+0.00
(6.0)
(158.7)
(128:4)
(-0.9)
(136,6)
31
(58.5)
(:55.0)
1220.3)
(60.6)
(27.8)
3.5
14:1
211
124
20-3
82
81
SELK-CEMEX
+3.09
06.36
(150.4)
(294.7)
(-2.0
(384.8)
(38.0)
(60:1)
(48.0)
-25
11.3
18
8.3
6.6
REIK-TagalRecch
10.14
20.11
+:.11
#:0.1
a.3.07
#0..1
10.30
1:2077
20.12
#5.12
(0.0)
(79-4)
(TON 01
6-21 71
(+4.2)
(69 4)
(-28
(-28 6)
(:14 S)
(+1.1)
(-10.5)
6.3
17.4
24.5
5.1
29.7
8.6
5.5
220
102
8.5
HILM Mach
=0.1
$0.15
=.0.1
+.0.1
13.13
2..11
=0.12
20.08
$3.08
20.14
#:.12
[0.0)
(178.3)
(293 10)
(-2.7)
1127771
3)
0)
(-11.3) 5)
(2004)
(62 9)
(35.7)
7.5
20.8
2913
7.0
19.3
22.9
U.u
4.2
18.9
13.1
125
SELL PlcciA
10.14
(174.3)
(-7.9)
(156.2)
[17.1)
(-44.4)
(72.9)
(06.3)
HELN-INED
+0:11
1011
101.1
+0
+3.06
H0112
10.17
(0.0)
(85.0)
(52.9)
6-14
(25. 23
(90
(-5
(-26.7)
(P9.6)
(11:2)
(5.9)
7.1
2
29.A
5.0
24.3
22.0
4.1
X
6.7
5.1
-29
QUIE-15PO
10.11
10.21
+0.23
+:0.08
:0.0
50.10
+0.07
1.324
20.12
+3.12
(0.0)
(368.5)
(3LL.))
(-30.6)
(238-3) 31
1922 37
FU80.91
(-43.1)
1267 S1
(-6.4)
(-14.5)
Extimtiion Gup & (Reduction RANDON SAMPLING
7.0
13.2
22.9
7.1
$2.3
12.3
0.2
6.3
L2.5
8.1
7.0
GLUB-STE
10
1-1.11
-0
14
10114
10
1813
10.13
(6.0)
(65.7)
(63-4)
(:10.8)
(36.1)
(50.2)
(18.6)
(:19.7)
(58.9)
(2.9)
(0.1)
15.0
16
4.6
15.4
56
4-5
1.
(292.0)
(202.71
1297.31
(..3)
(90.3)
23.0
20.6
28.2
20.0
7.0
8.1
OUTE-MOTT
20.12
+0.36
3.0.36
#5.11
36
#.5. 36
=0.13
:0.1
10.1
20.12
+0.12
(0:0)
(257.0) 01
(295 71
(-10.00 (1
22R8 41
12H6 23
(4:1)
(-13.8)
(-16.2
(0.0)
(-0%)
8.4
13.9
23.9
8.L
L3.
14.1
8.5
8.5
8.4
8.3
BUDE QUT.1
#0.13
20.13
=.0.13
#:0.12
50 13
4:5.13
=0.13
20.33
13:13
20.12
#0.12
15
[0.0)
[65.8]
(65.2)
(-5.P)
(65.4)
5)
(1.3)
()
(63.5)
(0.0)
(-0.6)
8.5
25.9
27.5
8.1
25.0
38.7
13.5
9.3
25.7
8.8
8.9
OpenILN-TPENol
(217.2)
(224.2)
(-4.5)
(202.3)
(129.7)
(5.1)
OpenlEM-Heth
40.50 04
05
+0.044
0.00
+.3.4
1000
+0.00
(G.0)
(104.2)
(110.8)
(-3.5)
(87.6)
(02.3) B)
(63.5)
(34.9) ()
(50.7)
(86.9)
(42.3)
87
18.6
25.6
7.0
1.8.L
23%
9.6
7.7
13.
6.6
6.9
Opant12 MAUE Pro
10.0 06
=0.05
0.0.2 26
+.3.0
-0.05
+-3. 06
00.05
+0
33.04
10.04
+0.04
(0.9)
(333.5)
(193.00 01
(.9.5)
(307 b1
1173 73
(13:3)
(.1.2)
(100.9)
( 8)
(20.7)
8.5
12.2
24.3
6.7
10.8
16.3
7.8
6.0
15.0
8.7
8.7
OpeniLL-M00 ABC
20.05
4:3.05
+0.05
06
+0.06
(41.6)
(67-9)
(:00.17
(2
20.3 71
(-29.0)
(28)
(2.0)
18,9
12.7
8.6
7.0
20.0 15.06
20,00
#:205
#0.05
(0.0)
(125.) 1)
(172.0) 01
(-5.5)
(103 11
1142 0/
[27 3)
(2.8)
(115
(-17
(-15.9)
8.4
74
1.2
8.4
5.4
OpenLL G/AA
10.00 06
+0.02
=0.00
$0.05
+0.62
+0.00
=0.05
140
+3.03
#0.04
+0.04
(0.0)
(-13.8)
(-10.2)
(-3.0)
[-18.6] 8)
(-16.7)
[-14.7]
[-0.3)
[-10.8)
[ 0)
(-23.4)
Y.0
6.6
5-6
6.2
6.15
5.5
6.6
5.7
37
Y.Y
06
07.07
10.07
+1.04
+0.
4-3
00.0
+0.00
(6.0)
[:1.9)
(-21.))
(:25.2)
6)
(-21.7)
[:37.1]
(-24.3)
(3.33
(33.9)
(.7.5)
6.6
6.1
IrageTot
+3.007
80.05
(.7.)
(:30.3)
(:2.0)
0)
(4.9)
(21.5)
15.6
6.5
9.0
6.7
Averae
10.00
+0.03
00
#3.02
40
+3.00
20.02
2:0
42.02
=0.02
+0.00
6
(0.9)
(129. 9)
(147 11
6:10 9)
(107 6]
1128 9)
(25 9)
[+
(115.3)
(17.1)
(10.1)
HANDOM--55 SAMPLING
SANDRSS-SASIPLNNN--A. LEARSS
PCA
RANDOR SEARCR-- LEARG LEARS
AIPW
Dorma- Oy monl
LANRO
AncHinn PoisTs SonnssWWaotte WZiGHrEn
Ascito- Premecoo
P-IRE
GPP (P-IRRT
The estimation gaps ()) for target models (calculated as equation I under extrapolation model split, where source models are the lowest- porfst-performing 50%, and target models the top 30% based on average performance over the full benchmark. Each target model can only be evaluated on 20 data points. We also report + the standard error of the mean and the estimation gap reduction (L) over RANDOM-SA SAMPLING in parentheses. A negative reduction implics that the method achieves lower estimation gap than RANDOM-S PRROM-SAMPLLA SAMPLIN.. % is omitted. Best viewed in color.
Figure 12:
34

---

6.
Openbackgg
=0.05
=3.05
=0.06
+0.04
(0.0)
[227.3)
[104.5)
(-2.6)
(216.3)
(267
(159.1)
[-31.8]
(362
(90 1:
(22.6)
33
80
4:2
16.7
4.3
-2..0
BELM S2NC
+0:00
1:3 Dd
30.3
+0:05
=0:09
0B
=0.09
:0.0
05
0B
+4.00
(0.0)
(203.6)
[307 3)
(-45)
(164.5)
(S92.5) 5)
(129.] J)
[22.1]
0)
(52.0)
(1B.1)
7.6
16.5
4.1
6-1
90
6.5
38
1-
3.7
3.5
BEIM-Tage Bench
13.07
0.008
0.06
06
0..06
(111.0)
(-912)
(-15
5)
(332.0)
(-54.2)
20.2
3.5
00
20.1
=0007
20.0
(3.0)
(241.3)
(440.7)
(0.1)
(180.5)
(313.5)
(157.7)
[40.8)
(432.0)
(17 3:
1-2.4)
13..5
4.2
0.
32
5.2
5.2
HELM-NNRQA
4-000 07
1:0 13
3:3.12
+7007
L0144
11
-0
-0.08
+0.08
+40
(0.0)
(282.9)
(L67.9)
(-11.6)
(266.4)
(200.8)
[87.4]
[-32.6]
(278.5)
(10.8)
(10
4.6
11.5
12.3
3.8
13.1
12.0
6.2
2.4
12.0
4.5
4.4
2:.00
=0.00
+0000
-0.44
=0.07
(0.0)
()
(1:2-1)
(-215)
(150.5)
(29.9)
()50.9)
[4.4)
[-4.0]
17.5
21.
22.7
27.5
3.5
1919
-0.05
2:0 08
(3.0)
1450
(429.7)
(-28.3)
(368.5)
(S39.7)
(304.2)
(-30.9)
(403.J)
12.0)
(-17.9)
Edtimttion Gup (Reducion RANDOL SAMPLING
4.9
12.0
11.2
4.3
12.1
12.3
6.6
4.6
13.1
6,3
5.6
GLTE RITT
$0.00
#5.18
23.16
=:007
20013
=0.00
00.0
202
=0.11
2.0.00
(0.0)
(168
41
(-13:9)
(148 2)
(190
E34
(-5
(16
(76 1)
[14 9)
GUIR SET
2.5
14.5
#0.04
10.3
2.9
14.5
=0.0
23.5
5.0
2.5
27
=0.08
6.3
6.0
15.0
(406
(252.0)
(-2.5)
(217.5)
(72.2]
(104.6)
5.0
OLHS-WNLT
$0.08
23.36
23.99
=0.07
5.036
=.033
:0.03
=0.07
2.007 07
=0.07
+.0.77
(0.0)
F462
(4000 9)
(-10.3)
(405.7)
(471.))
(4:4)
(-7.8)
(-15.4)
(-1.2)
(0:5)
5.5
13.0
13.7
5.1
13.8
14.1
5.4
5.7
13:G
6.4
6.5
GLIT ONLT
#0.00
(0.0)
3)
#2.13
(149.3)
$0.13
=:008
(-6.6)
(150.))
=:0.33
(156.))
=:0133
[:1 53
=0.00
+0.00
14.5)
=0. 13
(1
=0.00
[1
+0.000
12.5
23.5
25.6
35.2
24.5
24
5.6
DpeaILN-TFEWal
10.05
(350.3)
(400.2)
(.0.4)
(950.3)
(385.3
(0.3)
19.5)
12:1
10.0
L5.7
3.4
Satt
#3.03
=3.00
20.07
=0.03
20.05
=.0.5
=0.05
=0.1
2.0 04
20.03
+0.03
(0.0)
(1)8.3)
(700.0)
(-5.2)
(166 ")
(1667 99)
(155) 5)
(246 K)
[16. 0)
(11 3)
(-15
5.7
18.2
23.4
4.0
13.5
20:1
8.1
8.5
18.2
4.7
4.7
10.0
DpeaLL--RDD-PLO
#0.04
=:.00
$3.00
-3003
20.05
05
=:
=0
20.0
=0.03
+0.00
[0.9]
(152.0)
3)
(:14.31
(186.8)
(240.5)
[42.0]
(48.9)
(217.6)
(.19.3)
(:37.6)
10:3
10.2
13.5
7.9
15.2
4.3
OpenL2M-33C
10.04
(-19:3)
(88.0)
(168.8)
(62:11
(-10
(154
(37.7)
(-17.3)
20.7
D.1
0.0
OpeaLLM-EEN
#0.04
#3.00
=3.0
=0.03
20.05
=0.05
=:
2:0 04
20.03
=0.05
10.05
(0.0)
[1:977
(788.3)
(-4-8)
(169 5)
(3:5) 3)
F66
[55
(12% 91
(61 4:
(14.1)
$.1
5.5
5.0
5%5
6.5
+
4.7
6/
4.
0pa
2:0
0.00
1:0.03
-0.03
10.03
:0.03
10.08
(0.0)
(32:0)
(41.4)
(:2.0)
[25-7]
[25.1]
(.7.1)
(.8.8)
[16.8] 8:
(.33.)
(:10.)
6.1
4.2
02
=0.00
10.00
(0.0)
4-4
(10.0) 0)
5.5
[10:1]
(-26.)
2.8
(0:7)
5.7
12.5)
744
[-24.1]
4.6
[-22 5)
11.7
[26-0)
[-31
4.0
(-35.3)
4.5
5,0
ImageHez
1-..00 06
+ 04
65/00
+0.04
0.004
01
-..04
10.44
10.04
10.04
[0.0]
(24.7)
(120.3) 3)
(:35.5)
119.4]
165.1 71
(4:3)
(155. 9)
(61. 0y
(.9.5)
1.3.9)
4.6
13.6
14.0
4.0
12.8
15.6
8.2
5.6
15.4
5.1
4.7
Averge
+.3.01
2:0,00 00
23.03
+:.00
:0.01
10.01
(0.0)
()
(12..))
(200.3)
01
(22.4)
(215.2)
[16:1)
(6.3)
HANDOnS- BAAN001-SAMMMMNN SAMMMNNG
Harmont AAPPENNN LeARe
PCA
RANDOW- Si Annil LEARR
AIPW
Deveuep OF muus
LASSO
Ascmonn Pousrs Wies--aurrre Viturrra
Ascunee Powers Platuurrot
P-1RT
OP-1RT
The estimation gaps (L) for target models (calculated as equation 1 under extrapolation model split, where source models are the lowest- perees-peroormng 50%, and target models are the top 30% based on average performance over the full benchmark. Each target model can only be evaluated on n = 50 data points. We also report the standard error of the mean and the estimation gap reduction (L) over RANDOM-SAMPR MPLLING in parentheses. A negative reduction implies that the method achieves a lower estimation gap than RANDOM-
Figure 13:
% is omitted. Best viewed in color.
35

---

Openbackgg
(0.0)
[260.3)
=0.04
=0.04
(62.5)
(-5.2)
(272.5) 3)
645
(228 3)
=0:05
[-21.2
(366 31
=0.03
18.6)
(-39.2)
+0.02
33.0
2/2
84
30
2.4
2.1
BELM S2NCK
+9.04
1:0 07
+0.07
=0
00.07
-0
=0
-0.04
2: 04
+0.05
(0.0)
(2
(324.9)
(-5.0)
(24L.0)
(775.0)
(262. 5)
[28.7]
(563.8)
13.7)
[-3.5)
1.6
5.5
15.3
28
52
91
6.4
6.
5.1
2.5
2-4
BEIM-Tagg Bench
07
13.00
10.05
1005
04.04
(322.8)
(-22.6)
(22.6)
(-29.8)
(-54.8)
10.5
1:.5
03
2.3
2.5
17.5
+0.04
(3.0)
(298.3)
(364.4)
(-2.4)
(27L.)
(770.4)
(266.1)
(54.7)
(586
(-8.1)
10.1)
2.4
1.3
2.
3.7
3.5
HELN-NNAQA
06
12
10.1
10017
-1
-0.13
-0.3
07
+0.05
+0.05
(0.0)
(4
(230.2)
(-7.8)
(400.3)
(404.5)
(268.0) (3)
[-168]
(403.8)
(10.7]
(17
10.9
0.2
2.6
12.6
12.0
6.0
2.2
21.0
4.1
3.3
8:.008
=0.07
=:0.44
-0.03
=5.05
=0.05
(+13.6)
(242.0)
(793.5)
(-28.0)
(35
10.2
2.0
#0.00
(3.0)
(632.8)
69:18
[480.3] 3)
(-23.2)
(80L4)
15 (364
(533.0)
10
[-41.]]
(813.4)
(-7.0)
=0.04
(-29.6)
Esimmtioon Cup (Reducion over RANDOU BANTOUAAUPEEEO BANPENN
3.0
12.7
8.3
2.8
LJBB
12.3
6.7
4.4
13.5
4.8
4.2
GUTE RITT
#0.06
20.18
20.12
5.0.4
2.018
=:0.1
=0.00
07
=0. 1D
=0.08
1.0 1.006
(0
()
(194.00
(-8-4)
(794) 4)
(106
(119 9)
11
(S45
(59 St
(35
2.1
13.4
6.0
2.0
13.6
19.2
3.0
2.3
16.1
2.9
2.7
GUIR SET
20.05
#0.04
+3.03
03
04.04
=0.03
=0.04
(-2.1)
(824.2)
28.3
4.6
3.2
3.3
OLHS-WNLT
#0.05
23.36
23.35
=0.05
2..35
=.0.3
=.0077
10.05
=.000 06
=0.05
1.0.5
(0.0)
[7:5
(7M0) 5)
(10.6)
(721 4)
(753144
[16-11
(4.9)
19.5)
(-5.1)
[-3.5]
3.0
13.8
13.5
3.6
13.8
14.1
3.8
4.5
130
5.8
5.8
GUIT ONLT
#2.06
#5.13
$0.13
=0.05
20.13
=:0133
=0.05
07
=0.3 13
=0.08
1.0
(0.0)
8)
(46-3)
(-6.6)
(75635)
(763.5)
[: 13
(16.8)
(256
(50
(50 6)
OpeaILN-TFEHal
22.6
19.6
27.4
22-2
81
223
20:05
4.6
10.0
(93811)
(0:3)
(718.0)
(712.8)
(340.9)
(500.
(40
(38.2)
10.8
LO.B
9.7
2.4
Satt
$0.02
=3.06
20.07
=0002
=0.05
=00.5
=0.05
=0.1
20.03 03
=0.02
+0.022
[0
()00
(326 9)
(1:1)
E76 ()
(111
(126 1)
(2881 Y)
(205 4]
(15:8)
[-3.6]
41
14.7
20.2
3.7
15.0
17.5
8.0
688
16.9
3.3
3.3
#0.03
=2.00
$3.03
=:0022
20.05
=5005
=0
=D
=0 04
=0.02
+0002
[0.9]
2)
(390.7)
(9.4)
(203.2)
(353.7)
195.11
(310.0)
(:19.)
(.19.6)
8.7
7.5
10.3
8.3
80
12.9
3.0
3.1
SpenL2M-33C
[1:00
TAG
(-23.3)
(12977
(170.00
(117 3)
C10S 9)
(:210)
(-10.4)
8.5
10'5
OpeaLLM-EEH
#3.03
#5.05
35.05
=0.02
20.05
=0005
=0.04
=0.0 03
20.03
=0.05
+0.34
(0:0)
()
(368.6)
(-6.1)
(741.0)
(791
(108 5)
[17 4)
(166 4]
(187 b)
(101 4]
5.6
6.5
5.7
5.6
33
3.9
6
+15
0pa
2:0.00
0.00
0.002
:0.02
:0:02
10.00
(0.9)
(T8. 4)
(92.8)
(3.0)
[72.5]
[67.6]
(.5.))
0)
(7
[4800)
1.0.7)
550
4.1
4.3
=0.00 02
=0.02
02
02
10002
(0.0)
5)
[47.9]
(-25.4)
5)
71
4)
[-11
(14
(-12.7)
1.0
4.8
9.3
1.4
4.4
5.9
4.8
11.00
6.%
2.3
2.3
ImageHez
10.00 04
+:0 14
65/10
+..00
0.03
0.00
04
1015
10
10.05
10.05
[0.0)
(63.2)
(232.5)
(:3.8)
[47.0) 0)
197.3 21
(60.3)
(27L.4) 4)
(331.00
( (239))
(.24.)
12:1
12.2
2.8
L1.8
15.5
6.0
4.7
12.4
3.7
3.4
Avergg
+...01
00
2:.03
+0.00
0..03
02
:0.01
10.01
(0.0]
F200.0
(418.9)
(763 4)
[16.6]
(6.00
HANDOns HAAN0NN-SAMMMMNN SAMMMNNG
Harmont AAPPENNN LeARes
PCA
RANDOW Si Annil LeAAR
AIPW
Diveuep OF muusE
LASSO
Ancmon- Pourrs Wie---auuree Viturrr
Ascunne Powtrs Paatuurron
P-1RT
GP-1RT
The estimation gaps (L) for target models (calculated as equation 1 under extrapolation model split, where source models are the lowest- perves-prrooming 50%, and target models are the top 30% based on average performance over the full benchmark. Each target model can only be evaluated on n = 100 data points. We also report + the standard error of the mean and the estimation gap reduction (L) over RANDOM-SAMPA achieves a lower estimation gap than RANDOM- SAMPLING is omitted. Best viewed in color.
Figure 14:
in parentheses. A negative reduction implies that the method
36

---

10
5-8
1.5
10
5.4
39
6.5
15
0.6
0.6
40.0 01
+003
40.08
+3.07
-003
4:.04
+0.04
+0.01
+:307
-6.61
+..00
(8.0)
(456.3)
(100.2)
(1.8)
(460.8
1824
(302.2)
(-46.3)
(51.)
(-36.0)
(:32.3)
SELK-CEMEX
16
7
15
20.2
=0:09
2:1
1.4
-1
(357.3)
(39)
(359.71
[1398.00
(418.2)
(34.3)
(-22.)
(..0)
4.
1.9
5C
3.0
1-7
REIK-IagolBerch
10.04
20.05
+0.05
#3.03
=0.05
+.3007
20.05
10.0
1:3003
20.03
+3.00
(0.0)
(98.0] 0)
(430 9g
(-20
E107 01
1225 4)
(011 b)
(22
(T8.2 2)
[-31
(-32.7)
1.5
EX
1.6
012
22.3
6.7
2.5
13:
14
1.8
HIEM Mach
20.0 02
20.05
20.04
#30003
20.05
#2.00
2.0.1
20, 33.03
$7.07
20.02
+5.00
16
(0.0)
(503-3)
(160. 71
(9.2)
1552 81
[1418.0) 0g
(364 11
(71.0)
(967 3)
(4..5
(21.0)
2.1
15.0
1.0
15.7
18.5
7-4
27
14.4
3.3
2.9
JELL PleciA
10
29:09
(6.0)
(852.3)
(-11.0)
(665.71
3)
(254.5)
(27.7)
(593.5)
(60.2)
(30.8)
HELN-INLD
+0.00
+0.07
06
+0.003
O7
0.007
+0.0 00.00
+5.05
10.04
+0.00
(0.0)
(404.9)
(218-4) 4]
(-6-3)
6417 91
1906
(151
(-59 4)
(213.0)
(54
(-3.)
1.7
15.0
4.00
1.7
100
27.3
14.6
1.4
25.0
1.5
1.3
OUIE-15PO
10.00 00.00
(0.0)
(1001.0)
10.16
(425 21
(-1.6)
+:3.03
(AC10.7)
-0.16
(1902.5) 5)
4.0. 16
(752.0)
500000
(-20.33 5)
+0.00
[1414 6)
2.221
(-10.9)
00.02
(-22.9)
+0.00
Estmmaiion Gup & (Roduction over RANDON SARDON-SSMMAEEE SAMPLNG
1.4
12.4
3.5
2.3
11.8
12.3
9.8
2.7
D
2.5
1.8
SLUB-STE
40.00 30
10
00.06
+:
.10
14
11
on
8-.14
-4.04
+0.00
(6.0)
(782.5)
(137:0)
(61.3)
(734.2)
1775 4)
(396.2)
(91.7)
(580,0)
(75.5)
(30.7)
13.2
1-5
1.555
17
28
1-7
1.5
(355.4)
(94.0)
(.6.2)
6881.
(1392.3)
(104.5)
(9.0)
(23:2)
(6.8)
25.2
27.8
2.0
2.4
QUTE-MOTT
10.04
+0.35
-0.36
$5.03
35
133.36
=0.07
10.04
1:3055
20.04
+0.04
(6.0)
(1096-2)
11:12.51
(-11LM
(107 2)
(1119 4)
(122.9)
(23
(27.4)
(1.8)
(-0.))
2.5
13.8
23.1
2.4
L3.8
14.1
3.8
3.7
5.8
5.5
BUBE QUT.1
20.04
20.13
20.12
#0.04
50.3
#3.113
=0005
+0.06
13.13
20.08
+5.07
(0.0)
(426.5)
(397
(-7.2)
(424.00 91
9)
(45.0 0)
(42
(424.0]
(123.7) 7]
(1368)
2.2
15.0
20-7
2.4
16.2
20.0
22:1
3.7
22.3
4.8
3.5
OpenIEN-TFENol
40.00
+:0.07
13
03.03
(816.9)
(AB.1)
(634.6)
(60.2)
(118.4)
(50.3)
OpenlEM-Mech
+0.01
00.05
+-.001
05
05
+3.04
00.02
+0.0
(G.0)
(414.3)
(417.1)
(-0.9)
(388,8)
1928 91
(289.7)
(430.5)
(407.81
(66.4)
(7.2)
2.7
13.2
2.6
1.5
16.5
6.7
6.2
16.0
2.3
2.5
Opanti2 MAUE Pro
+0.0
=0.05
24
+:3.00
-0.05
+-300 06
-0.03
10 08
+3.04
-0.02
+0.01
(0.9]
[387.3)
(5077 11
(.5.0)
(400.22 21
1922 0)
(147.8)
(127 0)
[459.2 2)
(.248)
(17.0)
2.5
7.5
5.2
2.1
7.3
10.2
8.9
3.7
12.0
2.5
2.6
OpeniLL-MRC ABC
20.03
4:0.22
-:0.5
20.02
+:0.00
(1541-9
6-17.66
CTMA M
(75677)
(4)
(-1.9)
(2:3)
12.6
12.5
13.7
2.3
7.5
20.00
10.04
#2.02
=0.03
20.04
(0.0)
(357.4)
(456 71
(-6.4)
(388.5) 5)
1396
(108.3)
(-17.00
(268 3)
(291.2)
(168.)
244
2.7
50
2.9
3.2
211
OpenLi G/A4A
10.00 32
+0.02
10.000 oz
$0.02
=0.02
+5.00
=0.02
+40
+5.02
20:01
+0.00
(0.0)
[142.5)
(148.2) 2]
(14.3)
(135.6) 6)
1133 S)
(22.6)
(32.L)
(117.0)
(4.99)
(2.7)
2.5
4.2
4.5
2.2
6.c
3.5
4.g
3.5
5.
411
YYY
QpaaLLH-HKBB
0.007
+:.001
-0.07
o1
00:07
8::00
00.02
01
(6.0)
(65.1)
(70.4)
(:13.7)
(58.3)
(42.0)
(58:4)
(40.1)
1151.11
(63.2)
(5.8)
4.2
1.5
4.6
IrageTot
00.8
80.03
(104.5)
(314.0)
(:28.22 21
(95.4)
(120.22
(306 8)
[:29
(2255)
11:1
7.0
3.4
3.0
2.5
Averaee
10.01
0.03
=0.00
#3.01
+0
+:300
00.02
+10 01
#3.02
20.01
+0.01
(0.0)
(469 2)
(S0P 4]
(-2.)
(465.2 21
1677 7)
(766 1)
(16.6)
(453 13
(40.2)
(16 (16.5)
HANDOM--55 SAMPLING
RANDeSS-SASULLNNt-- LEARS
PCA
RANDORL SEARCRR- LEARS
AIPW
Dorma: Oy mont
LAN5O
AncHonn PoisTs Konnts-Waontee WaiGHrEn
ASCIOO- PRemecTO
P-IRE
GPP PRR
Figure 15:
The estimation gaps ()) for target models (calculated as equation I under extrapolation 50%, and target models are the top 30%
model split, where source models are the based on average performance over the full benchmark. Each target model can only be evaluated on 200 data points. We also report + the standard error of the mean and the estimation gap reduction (L) over
in parentheses. A negative reduction implics that the method achieves a lower estimation gap than RANDOM-S SANDOM-SAMPPING % is omitted. Best viewed in color.
37

---

C.4 Case Studies
We further investigate two additional experimental settings that deviate from the primary setting in the main paper.
Fewer source models under interpolation. Different from the previous interpolation setting that utilized 75% of models as source models, we now use only 10 models as source models for each benchmark and use the rest as target models. All other settings remain unchanged. This setting allows us to assess the effectiveness of benchmark prediction when "training data" from source models is more limited. Results are shown in Figure I6 Consistent with the findings in the paper, most methods still outperform RANDOM-SAMPLIN RANDOM-SAMPLING-LEANN remain to be the best-performing methods.
while
SEEARCH---AARN and
Near extrapolation. We modify the previous extrapolation setting, which used the lowest- performing 50% of models as source models and the top 30% as target models. this new setting, we designate the top 25% of models as target models and utilize all remaining models source models. All other settings remain unchanged. This setup enables us to examine whether benchmark prediction methods demonstrate improved performance when the distribution gap between source and target models is reduced. Results are shown in Figure 17 Consistent with the findings in the paper, most methods fail to consistently outperform
SAMPLING, except for AIPW.
38

---

33
29
32
2.0
BELI- SLLP--pezbookk
=0.00
10.1
20.38
=0.08
13.08
=0.15
+0.07
15.00
=0.11
1:000
+0,000
(0.0)
(-7.))
[-155)
(-152)
(-18 6]
(s:11
(-164)
(13.4)
[22:2]
(-1.6)
(-13-7)
4.5
3.0
3.5
3X
10
02
37
5.9
6.2
<81
IEIX QOMM
=0.11
#5.03
20,00
00.09
15,00
=0.2
10:12
=0.14
=:011
=0.1
(0.0)
(-14.0)
(-19.7)
(-14-1)
(:15.2)
(3
(-16.6)
(31.5)
(38.3)
(-3.1)
[+7.8]
5.7
3.5
37
32
4.0
3.2
5.3
34
TK 10.00
BELTY Legaliench
-0.14
+:3.00
=0.13
-0.11
10.11
-0.00
[0:0]
[-14-2
(:29.1)
(4
21
(+92.6]
(:39 5)
(-40.6)
7.0
3.0
1.0.1
=0.12
15.11
(0.9]
(-6-8)
[-11.0 0)
(-6
(-11 5]
(34
(-5.4)
(42.0)
(56-5)
(0:8)
[-7 61
5-2
3.0
4.0
4.5
52
3.5
HELPS NedtA
=0.12
10.3
=0.1
#-0.1
440.3
=-0 15
+:0 09
10.11
=0.33
10.11
+40000
(0.0)
(:26.31 11
(-20-4)
[-233 m)
(.26.1) 11
(3334)
(:29 21
(-11.21
70:3)
(:24.3)
(:20.7)
4.5
3.3
3.4
3.5
3.3
2.5
3.2
4,3
3.8
3.5
3.3
HRLM MASLTT
=0.12
+55.08
20000
=0.09
4.0. 00
2-0
=0.1
=0
1.0 00
10.00
(0:0]
(-331-5) 5]
(-30
[-98 1)
(-331 0)
(-45.1)
(-35 8)
(-12 11
(-924)
4)
(-S1
GLUE
=0.13
5.2
+:3.08
2.9
2.5
3.6
2.6
3.7
3.2
3.5
3.4
3.2
2.8 $0.00
Estimacion Gap % (Roductionn over RANDOO SASPLING 5D
(0.0)
(-44.0)
(-42.4)
(-29
(-38.6)
(-34.7)
(-96.8)
(-45.5)
2.0
2.5
-0.13
10,00
(0:0)
(-54.2) 21
(-47
9)
(-568)
(-45%) 9)
(-531
(-46.9)
7)
(-16.0)
2.6
3.0
2.7
3.0
GLUX SET 2
=0.11
+0007
20.07
=0.08
+0.00 08
=0.11
=0.07
$0,00
=0
#0.11
+0,00
(0.0)
(-38.5) 5]
(:47.4)
(-40.4 4)
(-47.55 5]
(9:5)
(-37 6)
(-51.7)
(-38.4)
(-22.7) 71
(-30.9)
5.3
2.0
522
2.7
2.5
b.b
31
2.7
2.3
2.9
2.9
TMLL
-0133
+-0
-0.33 31
09.09
IN
-0 14
+:00 09
aU.
=0.07
1:0
10.08
(0:0)
(-99 11
(66
()00 4)
(-54 11
11 3)
(-44 4)
(-51
(-575)
(-47 4)
(-47.7)
5.2
2.6
2.0
2.4
2.5
3.2
2.1
2.0
2.6
2.8
GUER QNLT
=0:13
#0.1
18.1
-01
#01
$0,00
10.00
(0.0)
(-46.6)
5]
1)
(-61.3)
(-50.4)
OpenLin
=0:1
+:0.00
05
+0.00
=0.16
10.30
13
14000
(0.0)
(:30.4)
(-22.4)
(-20.8) 8)
(-33.5)
(18.0)
(:20.0)
(G4.4)
(20
(-18,8)
(-20.6)
21
1.5
1.8
1.3
2.5
7.0
OptaLLN-Huth
=0.07
+00.0
20.56
=0.0
+0.00 05
=0.05
20.04
$0.15
=0.07
1:0.8 08
10,00
(0.0)
(-17.11
[-19 5)
(-145) 5)
(-96 6]
()376)
(-46 1)
(46.3)
(21.5)
(25. 0)
[-3.0]
5.0
2.9
7.3
3.0
2.3
5.1
26
0.0
3.3
6.1
4.0
=0.13
+0.07
=00
-9.09
4.00 00
-0.34
20.00
10.30
-0.09
+0.1
-0.3
(0.0)
(-42.5)
(454)
(:38 13
( 6]
11:3)
(-46.8
(19.3)
(-23.2)
(-19:1)
(-2.1)
5-3
2.4
3.3
322
3.2
27
5.7
3.5
3.4
OpeniLN-AaCC
(0.0)
(:44.8
(38.7)
(8.77)
(.6..)
(-284)
366
3.66 11,00
Opon1LY-E38
=0.13
+70077
10.11
+0.00
(0.0)
(-46.2)
4)164)
(;
(-55.8)
(-32.)
(-46.6)
(20.0)
(-36.6)
(-3..8)
(-32.6)
b.t
4.4
2.2
3.8
OpaaLLM-GPOA
12112
+-0.05
+0.06
-0.13
40.04
-0 04
40.
-0.06
10.
+0
(0.0)
(-64.6)
(-57.5)
(-14.3
(-70.6)
(-72.3)
(25.3) 3)
(:17.5)
(-57.9)
(:2.0)
(-28.4
5.4
1.9
9.3
$1
1.8
1.9
25
5.5
2.4
24
>11
OpeaLLX-HUER
-0:18
+0.05
-0.06
10.004
-0
10 13
-0.06
07.07
10.07
(0.0)
(-65.2) 21
(-57.3)
[-42.7 7)
6]
(-55.])
(-55.) 4]
[+2.2]
(-56.0)
4)
(-47.5)
4.8
2.1
4.0
233
1.U
1.8
2.1
30.8
1.8
2.5
2.5
TrageHes
=0:12
10,00
(0.0)
(:55.7)
(:50.)
(.61.4)
(-56.7)
(122.4)
(-51.5)
(-47.6)
(-48.0)
Averoga
0.03
+-000
10.05
0.07
10.00
00 03
0000
10.03
07
10000 07
10.00
(0.4)
(:38.)
(2.9)
(30.5
(44.2)
( (2.9)
(:3.0)
(5,8)
(.266)
(22.5)
(3L2)
RANDON SASIPLNN
Rammoos SASAPLING. LEARN
PCA
RASDOMS SEARCH LEARN
AIPW
Dumate Donace-Orraazz Orrmniz
LASIO
ASCHOR- Putsta- WetGinED
Assminn Pontee- PatinCTtut
P-IRT
GP GP-IRT
Figure 16: Experiment with fewer source models (randomly selected 10 models as source models) under the interpolation model split. We report the estimation gaps (4) for target models (calculated as equation l]. We also report + the standard eиror of the mean and the estimation gap reduction (L) over RANDOM-SA AMMPINGG in parentheses. A negative reduction implies that the method achieves lower estimation gap than RANDOM- SAMPLING % is omitted. Best viewed in color.
39

---

20
32
2.1
IEIK Opanhonety
20.34
=0.04
10.34
#3.03
20.04
10.03
=0.05
20.52
+:301
20.06
=0.04
(0.0)
(76.1)
(6.7)
(-7.4)
(44.8)
(06 4)
(52.6)
[-5.3)
(222:2]
(64.9)
[2.7)
3.2
5:3
5:3
2.9
0.2
13.3
4.7
5.4
14.1
4.4
3.6
aEUK GEMMA
10.00
=0.07
=0.07
25.05
07
43.00
=0:0
10
43.07
-0.07
10.00
(0.0)
(82.1)
(59.0) 9)
(-8.4)
(31.))
73
(45.7)
(68
(339.8)
(37.6)
(10.3)
-14
5.8
3.9
4.7
5.1
35
4.6
4.
34
BELK-LopolBecch
(-7.7)
(-21.7)
(14.3)
(.360)
(-10.9)
(:37.0)
4.0
3.1
HELH-HLLD
10.06
-0.07
#0.06
$:3.06
07
133000
-0.05
10.06
13306
=0.07
+3.06
(0.9)
(78.8)
(4.2)
(...)
(37.0)
1295 13
(45. 7)
(22.5)
(325-9)
(7:3)
(12.5)
4.7
L1.
EIL
3.9
LO.8
12.5
2,0
L5.
4.5
4.7
10.00
1...1
2-0 20
#:3.07
20.0
20..1
00.00
20,00
43.08
=0.08
+5.00
(0.9)
(154 7)
(24.0) 5)
(-16
$736 19
1169
(48 M)
(:31 9)
(736))
(b.5]
(-0.3)
4.8
6.9
4.5
3.3
6.3
7.4
4.8
2.0
3.3
3.3
3.2
-12
HIEN MHLD
20.00
20.05
4:0.08
07
20.23
#2.00
20.00
+5.05
(0.0)
[44.7)
(32.6)
(1:5)
(-32.5)
-0:15
8.3
#:3.00
14.5
07007
20.00
+0.00
Eximation Gop S (Roduction over RANDOM StammLaa ()
(0.0)
(316.9)
(L05.00 01
(-0.1)
(265.0)
13E1 6))
(12.6)
(16.7)
(456:3)
(46.0)
(5.2)
4.9
13.5
8
4.4
4
13.
6.4
5.7
6.4
5.5
CLUB-ATE
40.00 06
10.
+0 16
+:5.00
+0. 1
N-O.1 14
0.00
+3.29
10:11
+0.09
(C.D)
(377.4)
(142 33
(.9.5)
(268.2)
1181 B)
1-10.3)
(27.6)
(194.6)
(31.7)
(12.3)
2.6
5:1
2.5
2.5
6.2
12.7
2.6
2.5
6/6
4.1
3.7
10
OLTE-E9TH2
06
20.04
-0.07
+:006
(8.0)
(90.8)
(4.9)
(-11.2)
(50.5)
1353:21
(-6.7)
(-1.6)
(L33.5)
4)
(32.4)
2750
2330
27.5
4.5
9.6
(449.3)
(381.2)
13.0
4.5
5.1
GUBE QUL1
10.04
+0.13
=0.18 14
+:3.04
-0.12
H-0 12
=0.08
:0.00
:3.12
10.00
+0.08
(6.0)
(138,5)
(363..)
(.3.4)
(114-3)
7126 0)
(900)
(30.3)
(162.4)
(3.8)
(..8)
5.1
9.7
0.0
4.5
LO.4
24.3
12.5
11.
20.0
4.1
4.1
OpeniL 1PEvad
20.04
+0.05
1-0 26
#:3.03
10.08
=0..1
-.0.07
#1.055
4.3.07
20.03
+5.03
(0.0)
[92-3]
(9% 1))
6-11
170% 11
1924 9)
(143.8)
(185
(S1A 2)
(-17.))
(-19-2)
4.1
7.6
20.1
3.0
7
8-1
7.5
0.1
B.G
3.3
3.2
OpunLLN-Hushh
20.05
200.07
#:2.03
C5
2.005
20.03
+0.00
(84-5)
(-9-4)
(20.5)
8)
(137.11
(-19.3)
(-23.0)
9.9
4.0
04
05:05
+:0.03
04
23.04
20.03
+0.00
(0.0)
(69.2)
(240:2)
(-21.0)
(72.2)
11G0 7)
(12.9)
(-30.5) 5)
(155.5)
(-22.9)
(-23.3)
1.1
5.7
3.6
5.4
4.5
41
4.1
0pwnt17--RR
40.00 06:06
+0.04
+ 04
+..04
04.04
4-0 04
04.04
00.04
+0.044
-6
(C.D)
[12.3)
(-7.7)
(:23.4)
(4.0)
(66.5)
[-12.7)
(39.2)
(69.2)
2)
(.20.6)
5
12.0
14.
4.8
4.6
13.2
6.2
4
11.5
6.7
5.2
OpenlLy 2011
10.05
10.00
1.004
00.04
1:.04
(84.4)
(175.4)
(-12.2)
(2713)
(13.0)
(91.0)
(131.91
(23.7)
(14.3)
4.6
47
OpanLL
=0.03
=0.03
=0.03
#3.03
(0.00
(4:1)
(-2.5)
(-6.5) 5)
(-4.4)
(-1..0)
[-9
(2:1)
(-18.0)
(-16.5)
3.5
4.2
3.7
4.5
2.7
2.7
QUaaLLH-HU3BR
#0.34
=0.02
=0.02
#0.03
=0
#0.00
=0.02
24.04
#0.02
=0.02
#0.02
(0.0)
[-24.0) 9)
(-12.9)
(-35.00
[-35 8)
(-21. 8)
[-45.3)
(-12.5)
(5.0)
(-51
(-51.7)
4.3
3.5
2.4
3.6
4.9
2.9
30g
4.9
3.3
3.3
Izagsiet-
+4. 00
+4.04
=00 02
+0.04
=4.04
1:0 oz
=0.04
20.37
+:003
40.04
+0.04
[8.0)
[-19.9)
(75.5)
[-43.7)
[:30 7)
(13.2) 2)
(-33.1)
1153.11
(-23.0)
(-23.1)
4.5
9111
9.5
5
8.1
5.6
4.0
Average
(200.5)
(64.3)
(15.8)
RANDOM RANDOM-SAMPGRR SAMPUNN
RASDOOE-SASPPANE LEARS
PCA
RANDOM: SR ABCTI- LeARS
AIPW
Dormie: Oprontze
Lasso
ANCHUR- POINER W Winaarre
AscHon- Poitts- Phomnccon
P-HIT
GP-1 GP-IRT
Experiment with the near extrapolation model split by using the top 25% of available models as target models and the remaining bottom 75% models as source models. We report the estimation gaps (4) for target models (calculated as equation [T.. We also report + the standard error of the mean and the estimation gap reduction (4) over LANDOM-SAMPLE reduction implies that the method achieves a lower estimation gap than SADDOM-SAMPLIGG... % is omitted. Best viewed in color.
Figure 17:
in parentheses. A negative
40

---

D Broarder Impacts and Limitations
This paper addresses the benchmark prediction problem in scenarios with limited data. One potential limitation of our study is the relatively small number of models examined. For both the HELM-Lite and GLUE benchmarks, we have collected full benchmark results for fewer than 100 models. Despite conducting 100 random trials for each experiment, including additional and more diverse models could further strengthen the comprehensiveness and robustness of our analysis.
We do not anticipate any direct societal impacts from this work, such as potential malicious or unintended uses, nor do we foresee any significant concerns involving fairness, privacy, or security considerations, Additionally, we have not identified potential harms resulting from the application of this technology.
41


Reviews:
[{"review_id": "5XskvREHQZ", "reviewer": "Reviewer_iudt", "summary": "This paper is a critique of a popular research thread on efficient benchmarking or benchmark prediction (subsampling instances in a benchmark such that evaluating on that subset with a particular method gives a similar result to evaluating on all instances). The paper presents experiments showing that simple methods, such as taking the mean of randomly sampled instances as the prediction or that simply using a linear regression of full performance (y) onto subsample performance (x) performs as well if not better than many more sophisticated techniques, such as those using IRT. They consider two scenarios: Interpolation in which the set of models represented in subset & full are the same, and extrapolation in which the target models have higher ability than the models used for benchmark prediction. In interpolation they find that instance selection isn’t needed; simple baselines like the linear regression work great. In extrapolation they find that most methods perform poorly except for a new method AIPW. The key insight is that benchmark prediction quality depends on the target model (which we only evaluate a subset of instances) being similar to the observed population in terms of its item response pattern.", "questions": "Figures 2 and 3: The figure provides very detailed information but there isn’t really a clear take away. It seems like there are two views here: a viz to answer what benchmarks are easiest to estimate (sort rows by sum and just visualize estimation gap), and a viz to answer what estimation methods are the best (sort columns by sum and use reduction over random-sampling). Also having the takeaways written into the caption would be great\n\nParagraph 1 in related work (line 96) seems a bit tangential to the paper since subsampling items doesn’t really do anything about annotation cost or LLM-as-judge reliability etc.\n\nline 184-192: the source model set for different benchmarks is not the same. This makes it hard to compare which benchmarks are easy to predict because this will depend a lot on the source model set. Can you characterize the population of models? What is the range of compute scales? What is the range of performances? How is this different for the different model populations you use.\n\nLine 202: how is information from the 100 random trials used? are the estimated gaps in figure 2 etc actually mean estimated gap over 100 trials? If so it would be great to see a measure of variance over the trials as well.\n\nOne additional thing you could look at is for the methods that do select the set of core items, how similar is the set of items they select between the different methods? How does that set differ between the interpolation or extrapolation settings? To be clear the paper is already good without this, but I just think it would be an interesting additional line of inquiry.\n\nline 261: the explanation here could be given a succinct intuition before getting into the details of the notation. Something like, c_obs is the observed agreement and c_exp is the chance agreement rate. \n\nI would love to see some of the figures from 4.4 in the body of the paper if there’s space in a camera ready. \n\nSince the subsample size (n) is always in {50, 100, 200} this means the ratio of n to N (full benchmark size) can be very different for different benchmarks. This makes it rather difficult to compare which benchmarks are harder or easier to estimate. This paper is more focused on which prediction methods are better or worse across benchmarks. But if there are any benchmark specific dynamics, do you think it might be better to size the subset as a constant ratio of the full benchmark size so that each benchmark contributes similarly to the estimation of the prediction method’s success?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper's core findings (weakness of myriad methods relative to random baselines) is compelling and experiments demonstrating this are convincing.\n\n2. The insight identifying these two settings (interpolation and extrapolation) are very useful, showing where these methods work & where they break. It's a shift in thinking away from \"must identify high quality items that characterize a benchmark\" to more thinking about the set of models used to perform the prediction. Will change how people think in this area of benchmark prediction.\n\n**Weaknesses**\n\nHonestly, the paper is quite good, I don't see a reason to reject it. If I had to pick something, it would be some issues with framing:\n\nThe techniques in benchmark performance (e.g. IRT) also have another motivation besides strictly worrying about efficient benchmarking. Methods that learn the relationship of item level results to full benchmark results can help us better understand which items are difficult or noisy (so we might correct them if they are mislabeled) or to produce a (weighted) aggregate score that is actually more reliable than the full benchmark score (not predicting the benchmark but making it even better). It’s definitely fine if the paper wants to consider this out of scope, but I think it should mention that there are other ways in which the critiqued methods are valuable beyond the scope considered in this work.", "was_revised": true, "final_justification": "I enjoyed this work, and most of my questions were informational in nature, which were mostly answered. None of the clarifying answers at least would cause me to think any experiments (and thus findings) were invalid, and such I don't feel the need to revise my rating.", "rebuttal": "Dear reviewer iudt,\n\nThank you for your thoughtful and constructive review. We are encouraged that you found our core findings compelling, experiments convincing, and insights useful. We are glad that you recognized our potential impact on changing how people think about benchmark prediction. Below, we address your specific questions and suggestions.\n\n**Q1: Value of IRT methods.**\nThank you for highlighting the broader utility of IRT-based methods beyond efficient benchmarking. We agree that these methods can provide valuable insights, and we will acknowledge these additional motivations in the camera-ready version.\n\n**Q2: Presentation of Figures 2 and 3.**\nThank you for your suggestions regarding figure clarity. Our primary focus is to compare different benchmark prediction methods in both interpolation and extrapolation settings. We will clarify this in the camera-ready version and add explicit takeaways to the captions when we have more space.\n\nWe did not intend to compare the estimation difficulty of different benchmarks in these two figures, as this difficulty is largely determined by the score distribution. Consider the Random-Sampling method; the estimation difficulty for any target model is bounded by its standard deviation and the coreset size, as stated in line 271 of the paper. Consequently, the estimation difficulty for any benchmark largely depends on its model score distribution, since the standard deviation of Bernoulli is $\\sqrt{p(1-p)}$. If most models have very high or very low accuracy, the benchmark is easier to estimate; conversely, if accuracies are clustered around 50 %, the benchmark is harder to estimate. We appreciate the suggestion and will incorporate this discussion into the revised version of the paper.\n\n**Q3: Writing inaccuracy in paragraph 1 of the related work.**\nWe appreciate the comment and will revise this in the camera-ready.\n\n**Q4: Source model set not the same for different benchmarks.**\nWe mainly collect model scores from existing benchmarks, so it’s hard to use the same model list for all benchmarks.  Furthermore, our work extends beyond LLM benchmarks; we also include ImageNet. We appreciate your suggestion and will include the model list and characteristics in the revised version. Please refer to the discussion about the comparison of benchmark estimation difficulties in response to Q2.\n\n**Q5: How is the information from 100 trials used?**\nWe report the average estimation gap over all target models in 100 trials, and the standard error is reported in Figures 6-11 in the Appendix. We thank you for the question and will be more explicit about this in the revised version.\n\n**Q6: Coresets selection similarity.**\nWe agree this is an interesting direction. However, we find it challenging to quantify the coreset similarity using a specific metric. This difficulty arises because we conducted 100 different random trials for each method, and each trial often involved a different selected coreset. For instance, on ImageNet, we examined the coreset selected by AnchorPoint under random seeds 0 and 1, and we found that only three data points overlapped out of 50.\n\n**Q7: Intuition of Notation.**\nWe appreciate your feedback on notation clarity, and we will provide more intuition in the revised version.\n\n**Q8: Placement of ablation results.**\nWe will move the results to the main body in the camera-ready version. Thanks for the suggestion.\n\n**Q9: Coreset size being a static number instead of a ratio.**\nWe appreciate your insightful question. In our experiments, the dataset sizes of the benchmarks vary significantly, ranging from 277 for RTE to 50,000 for ImageNet. Consequently, it is challenging to find a ratio that applies universally to all benchmarks. Additionally, as mentioned in the response to Q2, the difficulty of estimating the benchmark is partly influenced by the coreset size rather than the ratio. We believe it is more effective to compare benchmark estimation difficulties by keeping the coreset size consistent across all benchmarks, while this is beyond our primary focus.\n\nThank you again for your valuable feedback and suggestions. We hope these clarifications and planned revisions address your concerns, and we look forward to further discussion."}, {"review_id": "jlDfAx31zL", "reviewer": "Reviewer_V6AP", "summary": "This paper discusses the problem of benchmark prediction: using model performance on fewer test samples from a benchmark to estimate performance on the full benchmark. The paper motivates this problem as arising due to the high costs of evaluating models on large-scale benchmarks. It then discusses several recent methods that have been proposed to tackle the benchmark prediction problem. Then, a systematic comparison of 11 different benchmark prediction methods (both previously proposed and some new methods) is conducted under two regimes: interpolation and extrapolation. The main finding of the paper is that most benchmark prediction methods underperform a simple random selection + regression baseline under the extrapolation regime. This undercuts claims from recent work suggesting that sophisticated coreset selection methods for finding optimal test samples to evaluate models, are required for strong benchmark prediction performance. Finally, the paper discusses how most benchmark prediction methods exploit model similarity, showcasing that benchmark prediction might not truly yield the promise it holds---evaluating stronger models efficiently from a database of weaker model evaluations.", "questions": "I've noted down all my concerns in the previous section.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Overall, I think this paper makes an extremely important point regarding the recent literature around the benchmark prediction problem. I appreciate the authors conducting a thorough empirical analysis of recent methods under the two different settings of interpolation and extrapolation. Overall, I think the paper is quite strong. I however have some important concerns regarding some of the experimental results and overall structure of the paper, which I note down below.\n\n- *Citing prior relevant work.* There are two important prior works that have been missed in the paper references. [Prabhu et al, NeurIPS'24](https://arxiv.org/pdf/2402.19472) proposed a similar method called Sort & Search, that leverages prior evaluations to regress newer sample level predictions. Another relevant work is [MetaBench, ICLR'25](https://arxiv.org/pdf/2407.12844) that employs IRT based methods to tackle the benchmark prediction problem.\n\n- *Similarity of key takeaways to prior work.* One of the key takeaways mentioned in the paper in lines 62-63 and 221-225 is also one of the suggested takeaways in Prabhu et al (see fig 5 and its caption): “This suggests generalizing beyond a single rank ordering, not better sampling strategies, should be the focus of subsequent research efforts.” I would appreciate if the authors can provide a discussion on the key similarities and differences between the main takeaways.\n\n- *Validity of aggregate accuracy metric for the benchmark prediction problem.* In Prabhu et al, they suggested that aggregate metrics like accuracy are not great for the benchmark prediction task since they can easily be manipulated (see section 2.1 \"Why Adopt Sample-wise Prediction Metrics instead of Overall Accuracy Prediction?\"). Sample-wise metrics are more important as they can enable precise characterization of the models on a per-sample basis. Again, a key discussion around this point would be appreciated.\n\n- *Regarding performance on \"Near-Extrapolation\" settings.* How does the extrapolation performance look like, if you look at “near-extrapolation” instead of on-average extrapolation. Essentially, the motivation would be that in the real-world, in most cases, we want to efficiently evaluate newer models which are a class-above but not significantly ahead of the current generation of models. Hence, in most cases, we care about predicting “near-extrapolation” (for examples performance of target models in the bin 70%-80% from source models of accuracy 20%-70%, rather than target models with accuracy in the bins 90%-95%). Hence, I would suggest employing a bin-windowed-evaluation strategy where you bin the performance of all models into K bins, and report the “near extrapolation” benchmark prediction performance using bins 1 to k to predict the k+1th bin as target models. If this indeed results in performance that is similar to interpolation, I would temper down the claims in the paper regarding the usefulness of the benchmark prediction task and methods.", "was_revised": true, "final_justification": "I am recommending acceptance. The authors have sufficiently addressed all my concerns, in my eyes making the paper much stronger. Note that the original submission was quite strong to begin with. Additionally, I think the topic explored in the paper is of extreme importance for many researchers working on foundation model evaluations, hence I believe that the paper will have sufficient audience at the conference.", "rebuttal": "Dear Reviewer V6AP,\n\nThank you for your detailed and thoughtful review. We are encouraged that you found the problem we addressed important, the empirical analysis thorough, and the paper overall strong. Below, we will address your specific concerns and suggestions.\n\n**Q1: Citing Prior Relevant Work.**\nThank you for highlighting the missing citations. Among them, MetaBench closely resembles the P-IRT method we discussed in our paper. There are two main differences: First, MetaBench employs lower-dimensional IRT models and selects the coreset based on information filtering. Second, metabench jointly fits its models on multiple benchmarks, making a fair comparison to other methods difficult.  Prabhu et al. propose a method called Sort & Search for efficient lifelong model evaluation. Rather than predicting benchmark performance, their method focuses on predicting the performance on individual data points. We further address our similarities and differences to this work in Q2 and Q3.\n\nWe will include the appropriate citations and discussions for both papers in the revised manuscript, and we appreciate your suggestion.\n\n**Q2: Similarity of Key Takeaways to Prior Work.**\nWe appreciate your valuable question. Prabhu et al. select the coreset by first sorting all data points by difficulty and then uniformly sampling from various difficulty levels. They assert that this method is more effective than random sampling for *sample-wise* performance estimation, as suggested in their Figure 4(d). \n\nHowever, in Figure 5, they demonstrate that the difference between sampling methods is smaller than the so-called “aleatoric sampling error,” which arises from their single rank order design, as defined in their equation 1. Therefore, they conclude that generalizing beyond a single rank ordering, rather than using better sampling strategies, is more effective in reducing overall error. However, this conclusion only applies to the framework defined by their equation 1 and does not generalize to other benchmark prediction methods (which do not use a “single rank ordering”).\n\nIn contrast, we have conducted an extensive empirical study on a set of benchmark prediction methods and found that coreset selection plays a minor role compared to prediction *across methods*. We appreciate your suggestion and will acknowledge Prabhu et al. for this relatedness in our revised version.\n\n**Q3: Validity of Aggregate Accuracy Metric.**\nWe appreciate your insightful question. The goal of benchmarking is to obtain an aggregated score, such as accuracy, from the testing data. According to Hoeffding's inequality, this score estimates the model's average performance on the data distribution and generalizes to unseen IID data. Therefore, the aggregated score serves as an effective indicator for comparing models and estimating performance. Benchmark prediction aims to estimate benchmark performance using a few data points to reduce computational burden, so it’s valid to use the aggregated metric. \n\nTheorem 2.1 in Prabhu et al. essentially states that it is possible to predict aggregate performance without correctly predicting performance for individual samples. This is indeed true (most of the methods tested in our work do not even attempt to make performance predictions for individual samples). In our view, Section 2.1 seems to argue that using sample-wise prediction metrics is preferable, *if you care about sample-wise prediction quality* rather than directly motivating the focus on sample-wise quality. While sample-wise quality metrics can be useful, machine learning *for better or worse* mostly relies on aggregate benchmark scores, further motivating our focus on aggregate metrics. \n\nWe appreciate the comment and will add this discussion to the camera-ready version.\n\n**Q4: Performance on \"Near-Extrapolation\" Settings.**\nWe appreciate your insightful suggestion. In the rolling bin setting, without modifying the methods, we must evaluate all models in the previous bin using the full dataset for the next stage to proceed. However, the main purpose of benchmark prediction is to begin with the already evaluated models, rather than assessing new models on the entire dataset.\n\nTherefore, we conduct an experiment on the HELM-GSM8K benchmark under the single-bin near-extrapolation setting. Different from the paper setup (lowest 50% as source models and top 30% as target models), we now use the lowest-performing 75% of models as source models and the remaining 25% as target models. The results are shown below. Consistent with the findings in the paper, most methods fail to outperform Random Sampling, except for AIPW. We appreciate the valuable question and will include these results in the revised version.\n\n| Method |Average estimation gap in HELM-GSM8K under near extrapolation split |\n| ---- | ----|\n| Random-Sampling         | 3.2  |\n| Random-Sampling-Learn   | 5.9  |\n| PCA                     | 5.1  |\n| AIPW                    | 2.9  |\n| Random-Search-Learn     | 6.2  |\n| Lasso                   | 13.3 |\n| Double-Optimize         | 4.7  |\n| Anchor-Points-Weighted  | 5.4  |\n| Anchor-Points-Predictor | 14.1 |\n| P-IRT                   | 4.4  |\n| GP-IRT                  | 3.5  |\n\nThank you again for your valuable feedback and for highlighting these important points. We hope these clarifications and additions address your concerns, and we look forward to further discussion."}, {"review_id": "IySyRJZYkA", "reviewer": "Reviewer_39bg", "summary": "This paper studies a specific paradigm for efficient model evaluation, which they refer to as benchmark prediction. In this setting a predictor h is trained to estimate the performance of a model across an entire test dataset given only the performance of the model on a select subset of data points in the test set. They study prediction performance both with and without train test distribution shift. They find in the easier setting (no distribution shift) all methods perform well, while in the harder setting (with distribution shift) most methods fail to generalize. Two methods do generalize to some degree, the first is the random-sampling baseline where a random subset of the test set is selected and the mean performance across the subset is used as the estimate, while the second is the Augmenting Inverse Propensity Weighting method.", "questions": "My understanding of the literature is that in any setting you might do this subset selection benchmark prediction you can also use bandit algorithms which have been shown to be much more effective than the random sampling baseline in your work. The literature contradicts your recommendation that the random sampling baseline should be used and without a comparison to bandit algorithms it is impossible to draw a conclusion about AIPW. Please correct me if I have missed something here, otherwise we need a comparison to the state of the art to understand the contribution of your analysis -- I would be happy to raise my score in either case.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n* Thorough experimental results for subset selection approaches to efficient evaluation on a diverse set of benchmarks and models\n\nWeaknesses\n* The random-sampling baseline from this work is a common baseline studied in other efficient evaluation works. Both [1] and [2] show this baseline (referred to as “uniform” and “filled subset”, respectively) is quite bad compared to bandit algorithms. The subset selection paradigm for efficient evaluation is certainly not the state of the art and a comparison to the state of the art bandit algorithms is certainly necessary to make a statement regarding efficient evaluation techniques.\n* Should include a list of the models for your analysis somewhere in the main paper or appendix.\n\n[1] Shi, Chengshuai, et al. \"Best arm identification for prompt learning under a limited budget.\" arXiv preprint arXiv:2402.09723 (2024).\n\n[2] Zhou, Jin Peng, et al. \"On Speeding Up Language Model Evaluation.\" arXiv preprint arXiv:2407.06172 (2024).", "was_revised": true, "final_justification": "This work provides important evidence of the misleading nature of current benchmark prediction literature, which is a valuable service to the field.", "rebuttal": "Dear Reviewer 39bg.\n\nWe appreciate your insightful comments. We appreciate your recognition of the thoroughness of our experimental results, and we address your concerns and suggestions below.\n\n**Q1: Comparison to Bandit Algorithms.**\nWe thank you for pointing out the related literature [1] [2] on bandit algorithms. We agree that a comparison is important for clarity. However, we would like to clarify a key distinction between the research objectives.\n\nBenchmark prediction aims to select a small subset of evaluation points to predict overall benchmark performance for any new model. This is useful when one wishes to obtain an accurate estimate for all models, not just the best one. In contrast, the bandit literature assumes a pool of target models and focuses on identifying the best model, rather than accurately estimating all models’ performance. By design, bandit algorithms will focus almost entirely on the best few models, which leads to bad performance estimation for all other models.\n\nThis paper focuses on the benchmark prediction task and explores when and how benchmark prediction is effective. All three other reviewers acknowledge the importance of addressing the problem. In the benchmark prediction literature, the state-of-the-art method, as far as we know, is the GP-IRT method, which we examined in our experiment.\n\nTo further address your suggestion, we experiment with the code from [2] out of the box within the benchmark prediction context. Specifically, we apply both the UCBE method and the UCBE-LRF method to the HELM-GSM8K benchmark under the interpolation model split. The results are shown below. Both bandit methods yield significantly larger estimation gaps compared to methods tailored for benchmark prediction. This supports our claim that while bandit algorithms excel at identifying the best model, they are less effective for accurate performance estimation across all models.\n\nWe appreciate your valuable suggestion and will include the citations and discuss the differences in the related work.\n\n\n| Method | Average estimation gap under interpolation split on HELM-GSM8K |\n| --- | --- |\n| Random-Sampling           |   4.4 |\n| Random-Sampling-Learn |   4.0 |\n| PCA                        |   3.7 |\n| AIPW                        |   3.8 |\n| Random-Search-Learn    |   3.9 |\n| Lasso                      |   6.8 |\n| Double-Optimize            |   3.9 |\n| AnchorPointsWeighted     |   5.6 |\n| AnchorPointsPredictor    |   6.1 |\n| P-IRT                       |   4.3 |\n| GP-IRT                      |   4.0 |\n| UCBE                       |  17.2 |\n| UCBE-LRF                   |  21.6 |\n\n\n**Q2: List of Models Used.**\nWe appreciate your suggestion and will include the model list and characteristics in the revised version.\n\nThank you again for your valuable feedback. We hope these clarifications address your concerns, and we look forward to further discussion.\n\n\n[1] Shi, Chengshuai, et al. \"Best arm identification for prompt learning under a limited budget.\" arXiv preprint arXiv:2402.09723 (2024).\n\n[2] Zhou, Jin Peng, et al. \"On Speeding Up Language Model Evaluation.\" arXiv preprint arXiv:2407.06172 (2024)."}, {"review_id": "oUIXWuZs7L", "reviewer": "Reviewer_k5n7", "summary": "The paper conducts a large‐scale evaluation of 11 micro-benchmarking methods for predicting full benchmark performance from a small subset of data across 19 diverse tasks. The paper shows that a simple baseline—randomly sampling 50 examples and fitting a regression model (Random-Sampling-Learn)—outperforms most sophisticated core-set selection techniques when source and target models are drawn from the same distribution. Critically, they find that nearly all methods break down when predicting the performance of stronger, unseen models, as their accuracy hinges on similarity to the source models. The only approach that remains consistently advantageous in this frontier setting is an augmented inverse propensity weighting (AIPW) estimator. \n\nOverall, the manuscript delivers a rigorous, timely reality-check with clear metrics.", "questions": "The authors do not appear to report the variance of each method across multiple repeated experiments; could they please provide these variance estimates?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper conducts extensive experiments across a diverse set of benchmarks and methods, yielding a comprehensive and robust empirical evaluation.\n\n2. The paper introduces an AIPW-based micro-benchmarking approach that maintains at least the performance of random sampling even in the “new models are strictly better” (extrapolation) regime.\n\n3. The paper offers a thorough quantitative analysis of the extent to which benchmark-prediction methods depend on similarity between target and source models, clearly elucidating why methods fail when that similarity is low.\n\nWeaknesses:\n\n1. The manuscript does not report how well each method preserves the relative ranking of target models (e.g. via Kendall’s τ). Adding these experiments—at least in an appendix—would strengthen the evaluation without incurring excessive additional effort.\n\n2. There is no analysis of method performance when the number of source models or the size of the selected coreset is small. In particular, IRT-based approaches might be unstable with few source models; highlighting whether AIPW or RANDOM-SAMPLING-LEARN become even more advantageous in this low-data regime would be valuable.", "was_revised": true, "final_justification": "The authors' response has resolved my issue. Since my score is already positive, I have decided not to make any further changes to it.", "rebuttal": "Dear reviewer k5n7.\n\nWe appreciate your insightful comments. We are encouraged that you found our experiments extensive and robust, our quantitative analysis thorough, and the manuscript to provide a rigorous, timely reality check. Below, we address your specific concerns and suggestions.\n\n**Q1: Ranking preservation.**\nWe appreciate your valuable suggestion. We follow your recommendation and calculate Kendall's tau between the real ranking and the predicted ranking of target models. The results are averaged across 100 trials for each benchmark. We further average the results across all benchmarks, and the results are shown below. Our conclusions mostly remain unchanged, with all benchmark prediction methods outperforming Random Sampling under interpolation, while none can surpass Random Sampling under extrapolation. We will include the detailed results in the camera-ready version.\n\n| Method | Average ranking difference under interpolation split (corresponding to Figure 2, last row) |\n| --- | --- |\n| Random-Sampling         | 0.705 |\n| Random-Sampling-Learn   | 0.748 |\n| PCA                     | 0.725 |\n| AIPW                    | 0.721 |\n| Random-Search-Learn     | 0.765 |\n| Lasso                   | 0.774 |\n| Double-Optimize         | 0.758 |\n| Anchor-Points-Weighted  | 0.763 |\n| Anchor-Points-Predictor | 0.768 |\n| P-IRT                   | 0.711 |\n| GP-IRT                  | 0.72  |\n\n\n| Method |Average ranking difference under extrapolation split (corresponding to Figure 3, last row) |\n| --- | --- |\n| Random-Sampling         | 0.534 |\n| Random-Sampling-Learn   | 0.185 |\n| PCA                     | 0.206 |\n| AIPW                    | 0.511 |\n| Random-Search-Learn     | 0.198 |\n| Lasso                   | 0.121 |\n| Double-Optimize         | 0.437 |\n| Anchor-Points-Weighted  | 0.504 |\n| Anchor-Points-Predictor | 0.318 |\n| P-IRT                   | 0.419 |\n| GP-IRT                  | 0.454 |\n\n\n**Q2: Analysis when the number of source models is small / when the coreset is small.**\nWe appreciate your suggestion. We conduct an experiment on the HELM-GSM8K benchmark under the interpolation setting. Following your recommendation, we use only 25% of all models as source models, resulting in 21 source models and 62 target models in each random trial. The results are presented below. Interestingly, most benchmark methods perform worse than Random-Sampling. The exceptions include AIPW and Double-Optimize.\n\n| Method | Average estimation gap in HELM-GSM8K under interpolation split with fewer source models |\n| --- | --- |\n| Random-Sampling         | 4.4 |\n| Random-Sampling-Learn   | 4.5 |\n| PCA                     | 6.7 |\n| AIPW                    | 3.8 |\n| Random-Search-Learn     | 4.4 |\n| Lasso                   | 7.9 |\n| Double-Optimize         | 4.2 |\n| Anchor-Points-Weighted  | 4.9 |\n| Anchor-Points-Predictor | 4.9 |\n| P-IRT                   | 4.8 |\n| GP-IRT                  | 4.4 |\n\nWe also conduct another experiment in which the only change to Figure 2 is reducing the coreset size from 50 to 10. The results are presented below. We demonstrate that when the coreset is small, Random-Search-Learn remains the best-performing method, while the IRT method degrades to perform worse than Random-Sampling.\n\n| Method | Average estimation gap in HELM-GSM8K under interpolation split with coreset size n=10 |\n| --- | --- |\n| Random-Sampling         | 10.1 |\n| Random-Sampling-Learn   | 8.2  |\n| PCA                     | 9.9  |\n| AIPW                    | 9.1  |\n| Random-Search-Learn     | 7.6  |\n| Lasso                   | 14.6 |\n| Double-Optimize         | 8.4  |\n| Anchor-Points-Weighted  | 9.7  |\n| Anchor-Points-Predictor | 7.9  |\n| P-IRT                   | 10.2 |\n| GP-IRT                  | 10.1 |\n\nThese results further highlight the potential limitations of existing benchmark prediction methods. We appreciate your insightful suggestion and will include it in the camera-ready version.\n\n**Q3: The variance of each method.**\nWe report the standard errors in Figures 6-11 in the Appendix. We thank you for the question and will be more explicit about this in the revised version.\n\nThank you again for your valuable feedback. We hope these clarifications address your concerns, and we look forward to further discussion."}]

Author rebuttal:

Meta-review: # Summary
The paper evaluates existing benchmark prediction methods for efficient model evaluation, comparing simple techniques like random sampling with more complex methods across scenarios of interpolation and extrapolation. The findings suggest that basic approaches, such as linear regression on random samples, often outperform sophisticated methods like IRT-based ones, highlighting the importance of target model similarity in benchmark prediction quality (interpolation). The study underscores the limitations of current strategies in predicting the performance of stronger, unseen models, with AIPW emerging as a consistently advantageous method in this challenging extrapolation scenario.

# Strengths
- The observation that random sampling + linear regression can outperform sophisticated approaches for benchmark prediction is important and novel.
- The insights that benchmark prediction performance can vary in interpolation and extrapolation further allocate the key challenges in the studied problem. 
- It finds the advantages of AIPW for extrapolation scenarios. 
- Extensive experiments across a diverse set of benchmarks and methods.

# Weaknesses
- Evaluation metrics do not include rank correlation of the benchmark evaluations.
- Missing analysis and comparison of benchmark prediction methods in more scenarios, e.g., when the number of source models or the size of the selected coreset is small.
- There are additional related works with similar observations and baselines (e.g., multi-arm bandit) that need to be discussed or compared with. 
- More justification of the evaluation metrics and settings can strengthen the paper.
- There are several places in the paper where the presentation can be improved.

# Reasons to Accept
- The paper presents an important discovery and a novel, comprehensive evaluation scheme of benchmark prediction. 
- The paper also points out strategies that may overcome the weaknesses of existing approaches. 
- Extensive experiments.
- All reviewers support acceptance. They are satisfied with the rebuttal. 

# Discussion Summary
- In the rebuttal, the authors provided more experimental results to answer the questions from the reviewers. Extensive discussions also help reviewers to better understand the details and potential impact of the work.
- All reviewers responded with positive feedback to the paper and the rebuttal.
Decision: Accept (poster)