Title: How Benchmark Prediction from Fewer Data Misses the Mark
Venue: NeurIPS 2025
Accepted: yes
Keywords: Evaluation

Large language model
Abstract: Large language model (LLM) evaluation is increasingly costly, prompting interest in methods that speed up evaluation by shrinking benchmark datasets. Benchmark prediction (also called efficient LLM evaluation) aims to select a small subset of evaluation points and predict overall benchmark performance from that subset. In this paper, we systematically assess the strengths and limitations of 11 benchmark prediction methods across 19 diverse benchmarks. First, we identify a highly competitive baseline: Take a random sample and fit a regression model on the sample to predict missing entries. Outperforming most existing methods, this baseline challenges the assumption that careful subset selection is necessary for benchmark prediction. Second, we discover that all existing methods crucially depend on model similarity. They work best when interpolating scores among similar models. The effectiveness of benchmark prediction sharply declines when new models have higher accuracy than previously seen models. In this setting of extrapolation, none of the previous methods consistently beat a simple average over random samples. To improve over the sample average, we introduce a new method inspired by augmented inverse propensity weighting. This method consistently outperforms the random sample average even for extrapolation. However, its performance still relies on model similarity and the gains are modest in general. This shows that benchmark prediction fails just when it is most needed: at the evaluation frontier, where the goal is to evaluate new models of unknown capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper conducts a large‐scale evaluation of 11 micro-benchmarking methods for predicting full benchmark performance from a small subset of data across 19 diverse tasks. The paper shows that a simple baseline—randomly sampling 50 examples and fitting a regression model (Random-Sampling-Learn)—outperforms most sophisticated core-set selection techniques when source and target models are drawn from the same distribution. Critically, they find that nearly all methods break down when predicting the performance of stronger, unseen models, as their accuracy hinges on similarity to the source models. The only approach that remains consistently advantageous in this frontier setting is an augmented inverse propensity weighting (AIPW) estimator. \n\nOverall, the manuscript delivers a rigorous, timely reality-check with clear metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper conducts extensive experiments across a diverse set of benchmarks and methods, yielding a comprehensive and robust empirical evaluation.\n\n2. The paper introduces an AIPW-based micro-benchmarking approach that maintains at least the performance of random sampling even in the “new models are strictly better” (extrapolation) regime.\n\n3. The paper offers a thorough quantitative analysis of the extent to which benchmark-prediction methods depend on similarity between target and source models, clearly elucidating why methods fail when that similarity is low.\n\nWeaknesses:\n\n1. The manuscript does not report how well each method preserves the relative ranking of target models (e.g. via Kendall’s τ). Adding these experiments—at least in an appendix—would strengthen the evaluation without incurring excessive additional effort.\n\n2. There is no analysis of method performance when the number of source models or the size of the selected coreset is small. In particular, IRT-based approaches might be unstable with few source models; highlighting whether AIPW or RANDOM-SAMPLING-LEARN become even more advantageous in this low-data regime would be valuable.\n\nQUESTIONS:\nThe authors do not appear to report the variance of each method across multiple repeated experiments; could they please provide these variance estimates?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a specific paradigm for efficient model evaluation, which they refer to as benchmark prediction. In this setting a predictor h is trained to estimate the performance of a model across an entire test dataset given only the performance of the model on a select subset of data points in the test set. They study prediction performance both with and without train test distribution shift. They find in the easier setting (no distribution shift) all methods perform well, while in the harder setting (with distribution shift) most methods fail to generalize. Two methods do generalize to some degree, the first is the random-sampling baseline where a random subset of the test set is selected and the mean performance across the subset is used as the estimate, while the second is the Augmenting Inverse Propensity Weighting method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* Thorough experimental results for subset selection approaches to efficient evaluation on a diverse set of benchmarks and models\n\nWeaknesses\n* The random-sampling baseline from this work is a common baseline studied in other efficient evaluation works. Both [1] and [2] show this baseline (referred to as “uniform” and “filled subset”, respectively) is quite bad compared to bandit algorithms. The subset selection paradigm for efficient evaluation is certainly not the state of the art and a comparison to the state of the art bandit algorithms is certainly necessary to make a statement regarding efficient evaluation techniques.\n* Should include a list of the models for your analysis somewhere in the main paper or appendix.\n\n[1] Shi, Chengshuai, et al. \"Best arm identification for prompt learning under a limited budget.\" arXiv preprint arXiv:2402.09723 (2024).\n\n[2] Zhou, Jin Peng, et al. \"On Speeding Up Language Model Evaluation.\" arXiv preprint arXiv:2407.06172 (2024).\n\nQUESTIONS:\nMy understanding of the literature is that in any setting you might do this subset selection benchmark prediction you can also use bandit algorithms which have been shown to be much more effective than the random sampling baseline in your work. The literature contradicts your recommendation that the random sampling baseline should be used and without a comparison to bandit algorithms it is impossible to draw a conclusion about AIPW. Please correct me if I have missed something here, otherwise we need a comparison to the state of the art to understand the contribution of your analysis -- I would be happy to raise my score in either case.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the problem of benchmark prediction: using model performance on fewer test samples from a benchmark to estimate performance on the full benchmark. The paper motivates this problem as arising due to the high costs of evaluating models on large-scale benchmarks. It then discusses several recent methods that have been proposed to tackle the benchmark prediction problem. Then, a systematic comparison of 11 different benchmark prediction methods (both previously proposed and some new methods) is conducted under two regimes: interpolation and extrapolation. The main finding of the paper is that most benchmark prediction methods underperform a simple random selection + regression baseline under the extrapolation regime. This undercuts claims from recent work suggesting that sophisticated coreset selection methods for finding optimal test samples to evaluate models, are required for strong benchmark prediction performance. Finally, the paper discusses how most benchmark prediction methods exploit model similarity, showcasing that benchmark prediction might not truly yield the promise it holds---evaluating stronger models efficiently from a database of weaker model evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, I think this paper makes an extremely important point regarding the recent literature around the benchmark prediction problem. I appreciate the authors conducting a thorough empirical analysis of recent methods under the two different settings of interpolation and extrapolation. Overall, I think the paper is quite strong. I however have some important concerns regarding some of the experimental results and overall structure of the paper, which I note down below.\n\n- *Citing prior relevant work.* There are two important prior works that have been missed in the paper references. [Prabhu et al, NeurIPS'24](https://arxiv.org/pdf/2402.19472) proposed a similar method called Sort & Search, that leverages prior evaluations to regress newer sample level predictions. Another relevant work is [MetaBench, ICLR'25](https://arxiv.org/pdf/2407.12844) that employs IRT based methods to tackle the benchmark prediction problem.\n\n- *Similarity of key takeaways to prior work.* One of the key takeaways mentioned in the paper in lines 62-63 and 221-225 is also one of the suggested takeaways in Prabhu et al (see fig 5 and its caption): “This suggests generalizing beyond a single rank ordering, not better sampling strategies, should be the focus of subsequent research efforts.” I would appreciate if the authors can provide a discussion on the key similarities and differences between the main takeaways.\n\n- *Validity of aggregate accuracy metric for the benchmark prediction problem.* In Prabhu et al, they suggested that aggregate metrics like accuracy are not great for the benchmark prediction task since they can easily be manipulated (see section 2.1 \"Why Adopt Sample-wise Prediction Metrics instead of Overall Accuracy Prediction?\"). Sample-wise metrics are more important as they can enable precise characterization of the models on a per-sample basis. Again, a key discussion around this point would be appreciated.\n\n- *Regarding performance on \"Near-Extrapolation\" settings.* How does the extrapolation performance look like, if you look at “near-extrapolation” instead of on-average extrapolation. Essentially, the motivation would be that in the real-world, in most cases, we want to efficiently evaluate newer models which are a class-above but not significantly ahead of the current generation of models. Hence, in most cases, we care about predicting “near-extrapolation” (for examples performance of target models in the bin 70%-80% from source models of accuracy 20%-70%, rather than target models with accuracy in the bins 90%-95%). Hence, I would suggest employing a bin-windowed-evaluation strategy where you bin the performance of all models into K bins, and report the “near extrapolation” benchmark prediction performance using bins 1 to k to predict the k+1th bin as target models. If this indeed results in performance that is similar to interpolation, I would temper down the claims in the paper regarding the usefulness of the benchmark prediction task and methods.\n\nQUESTIONS:\nI've noted down all my concerns in the previous section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a critique of a popular research thread on efficient benchmarking or benchmark prediction (subsampling instances in a benchmark such that evaluating on that subset with a particular method gives a similar result to evaluating on all instances). The paper presents experiments showing that simple methods, such as taking the mean of randomly sampled instances as the prediction or that simply using a linear regression of full performance (y) onto subsample performance (x) performs as well if not better than many more sophisticated techniques, such as those using IRT. They consider two scenarios: Interpolation in which the set of models represented in subset & full are the same, and extrapolation in which the target models have higher ability than the models used for benchmark prediction. In interpolation they find that instance selection isn’t needed; simple baselines like the linear regression work great. In extrapolation they find that most methods perform poorly except for a new method AIPW. The key insight is that benchmark prediction quality depends on the target model (which we only evaluate a subset of instances) being similar to the observed population in terms of its item response pattern.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper's core findings (weakness of myriad methods relative to random baselines) is compelling and experiments demonstrating this are convincing.\n\n2. The insight identifying these two settings (interpolation and extrapolation) are very useful, showing where these methods work & where they break. It's a shift in thinking away from \"must identify high quality items that characterize a benchmark\" to more thinking about the set of models used to perform the prediction. Will change how people think in this area of benchmark prediction.\n\n**Weaknesses**\n\nHonestly, the paper is quite good, I don't see a reason to reject it. If I had to pick something, it would be some issues with framing:\n\nThe techniques in benchmark performance (e.g. IRT) also have another motivation besides strictly worrying about efficient benchmarking. Methods that learn the relationship of item level results to full benchmark results can help us better understand which items are difficult or noisy (so we might correct them if they are mislabeled) or to produce a (weighted) aggregate score that is actually more reliable than the full benchmark score (not predicting the benchmark but making it even better). It’s definitely fine if the paper wants to consider this out of scope, but I think it should mention that there are other ways in which the critiqued methods are valuable beyond the scope considered in this work.\n\nQUESTIONS:\nFigures 2 and 3: The figure provides very detailed information but there isn’t really a clear take away. It seems like there are two views here: a viz to answer what benchmarks are easiest to estimate (sort rows by sum and just visualize estimation gap), and a viz to answer what estimation methods are the best (sort columns by sum and use reduction over random-sampling). Also having the takeaways written into the caption would be great\n\nParagraph 1 in related work (line 96) seems a bit tangential to the paper since subsampling items doesn’t really do anything about annotation cost or LLM-as-judge reliability etc.\n\nline 184-192: the source model set for different benchmarks is not the same. This makes it hard to compare which benchmarks are easy to predict because this will depend a lot on the source model set. Can you characterize the population of models? What is the range of compute scales? What is the range of performances? How is this different for the different model populations you use.\n\nLine 202: how is information from the 100 random trials used? are the estimated gaps in figure 2 etc actually mean estimated gap over 100 trials? If so it would be great to see a measure of variance over the trials as well.\n\nOne additional thing you could look at is for the methods that do select the set of core items, how similar is the set of items they select between the different methods? How does that set differ between the interpolation or extrapolation settings? To be clear the paper is already good without this, but I just think it would be an interesting additional line of inquiry.\n\nline 261: the explanation here could be given a succinct intuition before getting into the details of the notation. Something like, c_obs is the observed agreement and c_exp is the chance agreement rate. \n\nI would love to see some of the figures from 4.4 in the body of the paper if there’s space in a camera ready. \n\nSince the subsample size (n) is always in {50, 100, 200} this means the ratio of n to N (full benchmark size) can be very different for different benchmarks. This makes it rather difficult to compare which benchmarks are harder or easier to estimate. This paper is more focused on which prediction methods are better or worse across benchmarks. But if there are any benchmark specific dynamics, do you think it might be better to size the subset as a constant ratio of the full benchmark size so that each benchmark contributes similarly to the estimation of the prediction method’s success?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}