Title: Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: AutoEval

LLM judge

game-theoretic statistics

hypothesis testing

semi-supervised inference

prediction-powered inference
Abstract: Selecting  artificial intelligence (AI) models, such as large language models (LLMs), from multiple candidates requires accurate performance estimation. This is ideally achieved through empirical evaluations involving abundant real-world data. However, such evaluations are costly and impractical at scale. To address this challenge, autoevaluation methods leverage synthetic data produced by automated evaluators, such as LLMs-as-judges, reducing variance but potentially introducing bias. Recent approaches have employed semi-supervised prediction-powered inference ($\texttt{PPI}$) to correct for the bias of autoevaluators. However, the use of autoevaluators may lead in practice to a degradation in sample efficiency compared to conventional  methods using only real-world data. In this paper, we propose $\texttt{R-AutoEval+}$, a novel framework that provides finite-sample reliability guarantees on the model evaluation, while also ensuring an enhanced (or at least no worse) sample efficiency  compared to conventional methods. The key innovation of $\texttt{R-AutoEval+}$ is an  adaptive construction of the model evaluation variable, which dynamically tunes its reliance on synthetic data, reverting to conventional methods when the autoevaluator is insufficiently accurate. Experiments on the use of LLMs-as-judges for the optimization of quantization settings for the weights of an LLM,  for prompt design in LLMs, and for test-time reasoning budget allocation in LLMs confirm the reliability and efficiency of $\texttt{R-AutoEval+}$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method called R-AutoEval+ that uses no more samples than the previous autoeval methods and has reliability guarantees. It evaluates a method or model using both human labeled data and auto evaluators (e.g. LLMs-as-judges). The coefficient $\\rho$ determining reliance on the auto evaluator is iteratively computed, with one iteration involving one human labeled and many autoeval labeled samples. The theorems have learning theory flavour, mostly focusing on sample efficiency bounds under which the method is reliable i.e. doesn’t overestimate the risk with probability at least $1-\\delta$. The main methods used are testing by betting and e-values and PPI++.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper very nicely and gradually introduces the prior methods, which makes it so much easier to follow. The notation is clear and examples were given. The theorems are clearly stated. The method is tested on two different real-world evaluation problems, evaluating the best (smallest model) “good enough” quantization, and the shortest reliable prompt search.\n\nWeaknesses:\nNot sure if significance is well justified. For example, in Figure 9a), it seems like R-AutoEval and R-Eval are very similar. It is hard to know whether this will be a widely used evaluation method in the future.\n\nSeems like the authors renamed the method in [17] to R-AutoEval without mentioning the original name.\n\nQUESTIONS:\n1. In practice how do you deal with the probably correct result i.e. giving the correct answer with probability greater than $1-\\delta$? What happens if the prediction is wrong?\n2. Have authors compared the introduced method with some other evaluation methods (apart from R-Eval, R-AutoEval and others discussed in the paper)?\n3. Line 168, should it be R-Eval instead of AutoEval?\n\nLIMITATIONS:\nDoesn’t discuss what happens in practice when the method returns an incorrect prediction (with probability at most $\\delta$).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The authors provide a framework for automatic evaluation for AI models, where selecting from multiple candidate responses is necessary. They introduce a framework that provides guarantees on the evaluation performance, with enhanced sample efficiency over conventional methods. Empirically the show the reliability and efficiency of their framework.

STRENGTHS AND WEAKNESSES:
Strengths
- Theoretically motivated framework with better sample complexity than conventional approaches
- The guarentee provided by this approach is not asymptotic unlike prior work, yielding narrower confidence intervals
- Adaptive weighting of synthetic and real data based on the confidence of the autoevaluator is a novel contribution and leads to better empirical performance

Weaknesses
- Empirically validate their approach on toyish LLM tasks: Quantization and Prompt Shortening tasks. Could be useful to perform a broader evaluation on evaluation of an LLM on a broad range of topics (e.g math, coding, general QA)
- Evaluation limited to a single family of models and judge/autoevaluator (Llama family of models). Could benefit from a broader evaluation over different model families and sizes to see the effect of performance of the strength of the autoevaluator.

QUESTIONS:
1. Have you done a study of your approach over model size as the reliability of LLM as judge may change with respect to model parameters?
2. Have you looked at scaling properties over risk with respect to the amount of human and synthetic data that is used?
3. Would it be possible to evaluate this framwork in a larger scale setting as the ones decribed above?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript introduces \"R-AutoEval+\", an enhanced prediction-powered AutoEval method that builds upon previous R-AutoEval and Prediction-Powered Inference (PPI) work. R-AutoEval+ incorporates a dynamic parameter on top of PPI, which adjusts the weighting between human and automated evaluations. This adjustment is achieved through a \"testing-by-betting\" strategy, where weights are modified based on observations. This adaptation addresses R-AutoEval's limitations when the AutoRater's accuracy is insufficient.\n\nThe manuscript illustrates the method's application with two real-data examples:\n(1) Selecting the smallest quantized LLMs with a guaranteed performance drop of less than 10%.\n(2) Selecting the shortest prompt template with a guaranteed execution error of less than 0.5. In both cases, R-AutoEval+ demonstrated superior performance compared to R-Eval (pure human evaluation) and R-AutoEval (PPI-based AutoEval).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The manuscript is very well written. The methodology and results are clearly presented. Even though the content is quite dense and deep, the authors were able to present it well with preliminaries, making it self-contained and easier for the reader to understand.\n\n- The two real-data examples are cleverly chosen to demonstrate the method's use cases.\n\n- Solid mathematical justification.\n\nWeakness:\n- The details of real data experiments could have been presented more thoroughly.\n\nQUESTIONS:\n- Since the weights are learned dynamically by the observations up to $i$, does the order of the data to the algorithm affect the results? How robust is the algorithm against the data ordering?\n\n- Regarding the real-data experiments, how was the unlabeled data constructed for TriviaQA? How similar does the synthetic data need to be with the human labeled data? How were the hyperparameters, such as $n$, $N$, $S$, initial weight, synthetic-to-real, etc., determined?\n\n- Similarly, for the instruction-induction task, how was the unlabeled data constructed? How many human evaluation and synthetic samples were used?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces R-AutoEval+, a novel framework for evaluating AI models, particularly LLMs. It addresses the challenge of balancing between costly real-world data evaluations and potentially biased automated evaluations. R-AutoEval+ provides finite-sample reliability guarantees while ensuring improved or at least equivalent sample efficiency compared to conventional methods. The key innovation is an adaptive construction of the model evaluation variable that dynamically adjusts its reliance on synthetic data based on the autoevaluator's accuracy. The framework leverages game-theoretic testing-by-betting and an enhanced version of prediction-powered inference (PPI++). The authors demonstrate the reliability and efficiency of R-AutoEval+ through experiments on LLM weight quantization optimization and prompt design.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n- Provides finite-sample reliability guarantees\n- Adaptively balances the use of synthetic and real-world data, potentially improving sample efficiency\n- Offers provable sample efficiency improvements over existing methods\n- Combines multiple advanced techniques (game-theoretic testing-by-betting, PPI++)\n- Demonstrates practical applicability through experiments on relevant LLM tasks\n- Addresses a crucial problem in AI model evaluation and selection\n\nWeaknesses:  \n- Uses a fixed, discrete set of candidate factors for determining reliance on synthetic data, potentially limiting flexibility\n- Sample efficiency guarantee only holds for sufficiently high target reliability levels\n- Limited experimental validation (only tested on LLM quantization and prompt design)\n\nQUESTIONS:\n- What's the computational overhead of the algorithm compared to other methods?\n- How sensitive is the performance on the selection of hyperparameters? \n\nThe paper goes into mathematical details quite quickly. It would help if a more high-level problem statement was given in the introduction, before going more into depth.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-22"}