Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
Authors: Sangwoo Park
Matteo Zecchin
Osvaldo Simeone
Venue: NeurIPS 2025 spotlight ()
Keywords: AutoEval
LLM judge
game-theoretic statistics
hypothesis testing
semi-supervised inference
prediction-powered inference
Primary area: theory
TL;DR: a new autoevaluation method that is provably better (at least no worse) than evaluation irrespective of the quality of the autoevaluator

Abstract: Selecting  artificial intelligence (AI) models, such as large language models (LLMs), from multiple candidates requires accurate performance estimation. This is ideally achieved through empirical evaluations involving abundant real-world data. However, such evaluations are costly and impractical at scale. To address this challenge, autoevaluation methods leverage synthetic data produced by automated evaluators, such as LLMs-as-judges, reducing variance but potentially introducing bias. Recent approaches have employed semi-supervised prediction-powered inference ($\texttt{PPI}$) to correct for the bias of autoevaluators. However, the use of autoevaluators may lead in practice to a degradation in sample efficiency compared to conventional  methods using only real-world data. In this paper, we propose $\texttt{R-AutoEval+}$, a novel framework that provides finite-sample reliability guarantees on the model evaluation, while also ensuring an enhanced (or at least no worse) sample efficiency  compared to conventional methods. The key innovation of $\texttt{R-AutoEval+}$ is an  adaptive construction of the model evaluation variable, which dynamically tunes its reliance on synthetic data, reverting to conventional methods when the autoevaluator is insufficiently accurate. Experiments on the use of LLMs-as-judges for the optimization of quantization settings for the weights of an LLM,  for prompt design in LLMs, and for test-time reasoning budget allocation in LLMs confirm the reliability and efficiency of $\texttt{R-AutoEval+}$.

Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
Sangwoo Park Matteo Zecchin Osvaldo Simeone
Department of Engineering
King's College London
London, United Kingdom
{sangwoo, park,, matteo. zecchinn siwaldo.simeone)ee..
Abstract
Selecting artificial intelligence (AI) models, such large language models (LLMs), from multiple candidates requires accurate performance estimation. This is ideally achieved through empirical evaluations involving abundant real-world data. How- ever, such evaluations are costly and impractical at scale. To address this challenge, autoevaluation methods leverage synthetic data produced by automated evaluators, such as LLMssaa-judges, reducing variance but potentially introducing bias. Recent inference (PPI)
approaches have employed semi-supervised prediction-poweeee correct for the bias of autoevaluators. However, the use of autocvaluators may lead in practice to a degradation sample efficiency compared to conventional methods using only real-world data. In this paper, we propose R-AutoEval+, novel framework that provides finite-sample reliability guarantees on the model evaluation, while also ensuring enhanced (or at least no worse) sample effi- ciency compared to conventional methods. The key innovation R-AutoEval is adaptive construction of the model evaluation variable, which dynamically tunes its reliance on synthetic data, reverting to conventional methods when the autoevaluator is insufficiently accurate. Experiments the of LLass-asjuugees for the optimization of quantization settings for the weights of an LLM, for prompt design LLMs. and for test-time reasoning budget allocation LLMs confirm the reliability and efficiency of R- R-AutoEval+
Introduction
1.1
Context and Motivation
Selecting an artificial intelligence (AI) model among multiple candidates necessitates accurately estimating each model's performance. Typically, performance assessment involves actively employing each model to gather relevant empirical evidence or data. To mitigate the substantial cost and practical burden of real-world testing, autoevaluation leverages automated tools to evaluate model performance without direct human intervention [35, 31, 12, 8, 52, 5].
Standard evaluation based on human judgment referred to as Eval and autoevaluation AutoEval each present distinct advantages and drawbacks. Eval provides unbiased estimates model's performance but requires costly annotation. In contrast, AutoEval is cheaper, as it can rely on abundant synthetic data, but it may introduce estimation bias [35, 12]. Consequently, neither approach, in isolation, ensures reliable model evaluation, potentially leading erroncous evaluation outcomes.
Reliability in evaluation methods can be established via confidence intervals that accurately capture the true expected performance at a specified coverage level, or through testing strategies that detect whether target performance levels are met at specified false detection probabilities. For Eval, both
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

LLM judge
human judge
Llama3.3-BF
Llama3,3-MX6
Llama3.3-M
20
drop [K]]
AAAAKA ARA
16
10
pefortmere drop 199
200
7.5
modee sbze (GB)
0-
1.1 GB
1.1 GB
ooserage bitwidth
6.3
6.0
Eval
R-Eval
AutoBval
R-AutoBval
R. AutaEE
AutoEvvll
AutoBBval
B. A-AuoEEval+
AutoEval
R AutoBvall
R-AutoBivll+
Figure 1: How to select the lightest quantized Llama-3. 1-iaa-3.--BB-Intttree model [24] (in the MX quantiza- tion format [41]) that guarantees up to 10% performance drop as compared to the unquantized version (BF16) (for the TriviaQA task {28])? (left) Ground-truth risk /t for different MX quantization settings, requiring massive human-labeled data. (right) Performance drop and corresponding model size for the models chosen via Eval, AutoEval [35, 31, 52], R-Eval [47], R-AutoEval [20], and the proposed R- A-AAuoEvva++ We adopt 1.ama-3.33-0B- 70B-Instruct [24] BF 1116/MX6/MX4 as the autoevaluators, and set target risk in (1) to 0.1 and target reliability in (2) to 1 8 0.9. Maximum values reported within the 1.5 interquartile range (IQR) range [34] across 500 independent experiments (see Sec. for details).
approaches can be implemented using standard statistical methods [26, 33, 47), which we refer to as R-Eval.
Achicving reliability in AutoEval more challenging. Recent works, including [10. 22, 21, 42], have successfully applied semi-supervised inference framework known prediction-poweree inference (PPI) [3, 4] to correct for the inherent bias AutoEval by leveraging a small amount of human-labeled, real-world data. These methodologies, referred to here as R-AutoEval, either guarantee reliability only asymptotically - as the sizes of both synthetic and real-world data sets grow indefinitely [10, 22, 21] or lack explicit sample efficiency guarantees i.e., they do not provably yield narrower confidence intervals or higher test powers compared to R-Eval [20].
In this paper, we introduce R-AutoEval+, a novel autoevaluation framework that provides finite- sample (non-asymptotic) reliability guarantces while also ensuring improved (or at least no worse) sample efficiency compared to R- A-AuoEEval. The primary innovation of R- R--AutEEva++ is its sequen- tial construction of the model evaluation variable, enabling it to adaptively adjust its reliance on synthetic data based evolving assessments of the autoevaluator's quality. R-AutoBval+ seam- lessly reverts to R-Eval when synthetic data are deemed to be of insufficient quality, while otherwise employing weighted variant of R- Autoovval to enhance efficiency.
At technical level, R-AutoEval+ + leverages two primary methodologies. The first the game- theoretic testing-by-belting approach [43, 38] to mean estimation introduced by [47]. The second is PPI++ [4], an enhanced variant of PPI that incorporates regularization coefficient to control the estimator's dependence autoevaluator-generate data [4].

---

1.2 Overview of the Main Results
To clarify the main concepts, consider the problem of selecting a large language model (LLM) from multiple candidate models characterized by different sizes. These models are derived from single base LLM through quantization with varying average bitwidths using the MX format [41]. As illustrated in left panel of Fig. 1, our goal is to identify models that maintain performance comparable to the full-precision baseline, tolerating at most a predefined performance degradation threshold a (e.g., = 10% in the figure).
Formally, consider a bounded loss function ((X E [0, 1] that measures the performance of given candidate model. In the context of Fig. I, this loss quantifies the performance gap between the candidate quantized LLM and its full-precision counterpart. Our primary objective verify whether or not the expected loss, or risk, defined as R = E[[(X, Y)], exceeds target level a, i.e. whether the following risk cist-oontoolling condition can be satisfied
R S a.
(1)
The challenge is that evaluating the risk R necessitates access to a large number of human-generated, i.., real-world, responses Y corresponding queries X.
1.2.1 Eval and AutoEval
Given pairs of real-world data D, {(X,, YO) with (X,, Yi) ~ Pxy = Pr for == I,,,, n, the standard Eval approach estimates the risk R for any given candidate model via the empirical average Rm = (1/n) Ei=1 ((),,,)).
In contrast, AutoEval does not assume access to real data Da, relying instead the availability of pre-trained autoevaluator f X + D, where and y denote the respective domains of input and output X, Y. In the example of Fig. 1, the autoevaluator is a larger L.L,, serving as an IIM judge [52]. Specifically, given an unlabeled data Diy' = { with X3 i.d. Px, AutoEval estimates the risk ft'y (1/N) E(X,, ((X.)), thus using the outputs from the LLM judge labels [35, 31, 52].
Lacking formal uncertainty quantification, the risk-controlling condition (1) cannot be guaranteed via the risk estimates R, and Rt provided by Eval and AutoEval. That is, even if the evaluation outcome for a candidate model satisfies the inequality R. S a, or R < a. its actual performance may still fail to meet the condition (1). In the example shown in Fig.1, LLMs selected via Eval and AutoEval exhibit performance degradations that exceed the target threshold of 10%. Furthermore, for AutoEval, the performance degradation becomes larger as the quality of the LLM judge deteriorates.
1.2.2 R-Eval and R-AutoEval
Reliable Eval (R-Eval) [47] and Reliable AutoEval (R-AutoEval) [20] endow Eval and AutoEval, respectively, with reliability guarantees by formulating model evaluation as binary hypothesis test problem. Accordingly, these evaluation protocols test the null hypothesis Ho : R > a that the model's 's risk exceeds the target a against the alternative hypothesis H4 R < that the risk-controlling condition (1) is satisfied.
Define as T, E {0, 1) the test output, with T, 1 indicating the decision for the alternative hypothesis Hi : R < a. Then, an evaluation procedure is said to be reliable at level 1 - the probability of incorrectly concluding that a candidate model satisfies the requirement (1) does not exceed the level e (0, i.. i.c.
Pr [T 1|R a] 8
While Eval constructs the test decision based only real data D,.. R- R-AutoBval incorporates also synthetic data by leveraging PPI [3] as we detail in Sec. 2.
As illustrated in Fig. 1, selecting the smallest LLM among the candidates deemed risk cist-controlling by R-Eval and R-AutoEval, i.e., the models with respective testing results being T, = 1, indeed results in performance drop that remains within the tolerated risk level of 10%. However, as
also shown in Fig. 1, R- R-AutoEval may select a model with a larger size than R-Eval, highlighting the potential inefficiency due to the use of an LLM judge.
3

---

1.2.3 R-AutoEval+
This paper proposes R-AutoEval+, a novel reliable autoevaluation method that adaptively tunes its reliance on synthetic data based on an evolving reliability assessments of the autoevaluator, This approach balances synthetic and real data, reverting conventional methods when the accuracy of the autoevaluator insufficient, while maintaining rigorous statistical guarantees.
R- AutoEval+ + is not only reliable in the sense of satisfying the condition (2), but it also provides a guaranteed improvement (possibly not strict) in terms of sample efficiency. To formalize this property, define the sample complexity of an evaluation method producing test variable Tn as the smallest average size of the real-world data set D, necessary to conclude that a model satisfies the requirement (1) under the reliability constraint (2), i.e., [48]
mimi (6) = E[min(n: T, = 1H < aa)-
An evaluation scheme with smaller sample complexity Renin (6) can generally identify more efficient candidate models with the same amount of real data. The main result is informally outlined as follows.
Theorem 1 (Informal), Under mild regularity assumptions, for sufficiently low tolerated unreliability level 6, R-AutoEval+ is provably more sample efficient than both R-Eval [47] and R- AutoEval [20], i.e.
3-AutoEval* Tl min
(8)
min
(mmi
A-Eval B- A-AutoEval (8) (8)}
Furthermore, this inequality strict when the autoevaluator is sufficiently accurate.
Preliminaries: R-Eval and R-AutoEval
In this section, first review the eesting-by-hetting framework [43, 37, 47, 48, 38), and then review R-Eval [47] and R-AutoEval [20]. Further connections to the state-of-the-art can be found in Appendix A.
2.1 Pesting-By-Bettinn
Consider the problem of testing the null hypothesis Hy R a against the alternative hypothesis Hi R < based on the observations of n i.i.d. bounded random variables {q:]2_ with qi E [m, M] providing unbiased estimates of the risk, i.e., [[q] = R. An e-value E, is a nonnegative statistic of the observations {q: la whose expectation under the null hypothesis does not exceed i.c., E| It > a] < [45]. An e-value E, can be interpreted as providing evidence in favor of the alternative hypothesis H1 i.e., of the risk-controlling condition (1) [43]. With an e-value En, the test
Ta IFE,, 2 1/8)
meets the reliability condition (2) due to Markov's inequality for a fixed n.
The testing-by beeinn--y--ere by-betting approach constructs an e-value E,, sequentially by processing the observations one by one over index i = 1,..., n. Specifically, an e-value can be obtained via the product of the contributions of each observation qi as [47, 6]
F II X - a));
(6)
where Eg = and X E 10M a)) is an arbitrary function the past observations {q, 1i-1 To
verify that the quantify (6) is an e-value, one can use the independence of the observations {4) and the unbiasedness assumption E[q] as E R a] = III"_
(B a] a)) S 1.
The e-value has an interpretation in terms sequential betting game, which supports the design of the sequence of bets X; using online convex optimization [47]. In this game, at each round i, based on the observations {4, [=)) gambler bets amount of her wealth measured by variable X; on the outcome di S a that the next observation qi does not exceed the target a. Accordingly, the e-value E, in (6) represents the wealth accumulated after rounds of betting with initial wealth Eg 1 [47].

---

A more general form of e-value has been also proposed that allows the gambler to have S 2 1 different betting strategies {A,,e)==1+ with each strategy As,i being responsible for fraction Ws,; of the wealth. To elaborate, fix probability vector = [[1,,,- ..., Ws,,), which, like the betting strategies {xa I may depend on the past observations {0p [g)))=1 The resulting e-value is defined as the convex combination [47, B.8]
E
Wa (1 -s, aa))
1=1 ==1
2.2 A-Eval and R-AutoEval
Both R-Eval and R-AutoEval follow the essing-by-bettingg approach presented in the previous subsection. Specifically, R-Eval computes the e-value E in (6) using directly the observations % 4, of the losses l ((X,, Yi) E [m 0, MM 1), obtaining the decision 7R-Eval in (5).
R- A-AuuoEval [20] also computes the e-value E ER-AutoEval in (6), but with effective observations = 1 that incorporate both real data and synthetic data. In particular, each effective observation d: ef uses the real data point (X,, Yt) DD, together with [N/n] autoevaluated samples {(x, (X,)))=(( (i-1)) with X; E Dx. Note that this divides the of unlabeled samples D equally across the effective observations.
Specifically, R-AutoEval uses effective observations obtained via PPI [3] by correcting the bias of the empirical autoevaluated risk using the real data sample as
ri
Σ
((x))( ((X,, YJ (Xx, f(Xi))
(8)
d'=r-(1- 1)+1
bias correction
antoevaluator data
One can readily check that this is an unbiased estimate of the risk, i.c., B[e/] R, with support interval Ef E [m = 1, .M = 2]. However, owing to the possible lower quality the autoevaluator's labels f(X,), using the effective observations (8) may result in larger sample complexity compared R-Eval (see Fig. 1).
3 R-AutEEval+; AutoEval with Reliability and Efficiency Guarantees
In this section, we present the proposed autoevaluation method, R-AutoEval+. R-AutoEval+ aims at balancing the importance assigned to synthetic and real-world data in the effective observations (8). Through the proposed mechanism, R- R-AutoEval+ reverts using the effective observations = l, of R-Eval when the autoevaluator is inaccurate, while leveraging the effective observations of R A-tuoEvaa, y; !! when the autoevaluator is very accurate. The approach is based an adaptive construction the effective observations used in the e-value statistic (7), dynamically tuning the reliance synthetic data based evolving reliability assessments of the autoevaluator.
The key idea is weight the contribution of synthetic data the e-value with factor p [0, 1]. so that setting م O recovers AutoEval, while setting P = yields R- A-toEvvval. In order to automatically identify the best value of the factor p. B-AutoEval+ processes the real-world data samples sequentially across index i = I,..., tracking the performance of S possible candidate values {p. ) with
< <P =1
99)
Specifically, R-AutoEval+ maintains adaptive weights [w.,.... )-5 across index = 1,..., n, with weight Wa, associated candidate value Pa. The resulting effective observations are used in (7).
After providing a more detailed description of R-AutoEvalt, this section demonstrates that R- A-tuoEvvl++ can provably enhance the sample efficiency of both R- R-Eval and B-AuoEvval.
5

---

3.1 Adaptive Effective Observations and E- E-Values
At each round i, R- A-tuuEEva++ computes S effective observations ( with each effective ob- servation e weighting the contribution of synthetic data via the factor ps- In particular, generalizing (8) via PPI++ [4], each s-th effective observation is given by
Ps
f(x,)) ((X,, Y)) PS ((,,, f(x)).
(10)
(i-+1
corraction
sutoevaluator data
where the factor Ps multiplies the contributions of the autoevaluator. The observation (10) is bounded unbiased estimate of the risk R. i... Ele!, R. with E E [m = -Pa M 1 pal [4].
Using the effective observations {9..; = ! J R- A-AutoEval+ constructs variant of the e-value En in (7) in which a different bounded unbiased estimate of the risk, namely é is used for each s-th term, i.e.,
En
Wa,i (1 (e, - a)).
(11)
1= ==1
Note that this is indeed valid e-value, i.c., E R al S
1.
&
1.00
In the e-value (11), the weights w,, associated with each factor P. are updated over index = 1,... n depending on the
reliance factors
Ws.i
0.00
1.00
(a)
0.04
evidence accumulated up to round 1. To do this, define
0.00
0.02
(b)
E. [I (1- a))
(12)
1.00
0.00
the e-value (6) computed using only the effective observations C, up round i. Intuitively, larger value of the quantity E..; indicates that the factor Ps yields large evidence in favor of the alternative (risk-controlling) hypothesis (1).
(c)
5000
rounds
Figure 2: Heatmap of the evolution of the weights { 1000 assigned to the factors Ps as a function of the processing round i for Example 1. The autoevaluator reports the correct loss with probability 7 0.99 (top), 0.9 (middle), and 0.7 (bot- tom). R-AutoEval+ assigns larger weights to synthetic data, i.c., to larger values of Ps. when the autoe- valuator is of higher quality.
Following this logic, the weight w.,i is updated as
Ws.0 E
Wo.i Ex=1 W0'00' Esa-1
(13)
for all s 1. 1....,,, S. with initial strictly positive weights {we.o}s J.511. where ΣS- 0g = and E..0 1. By (13), values of the factor associated with larger evidence F. in favor of the risk-controlling condition (1) assigned proportionally larger weight in the e-value (7).
The update (13) aims at identifying the most informative ef- fective observations é!= for 1,..., by sequentially processing the available data. The following simple example demonstrates its operation.
Example 1. Consider setting where the autoevaluator yields the same oulput the human judge with probability 7, Specifically, assume a binary loss E(X, Y) E {0, 13 with mean equal to the risk
R = 0.1, and autoevaluated loss f(X,f(X) [(X)) given by ((X, f(X)) = ((X, Y) (1 (1-6) :) +
E(X, Y)) E, where e {0, has mean 1 Fix the target risk a = 0.12 and the synthetic-to-real data ratio 10.
100 Fig. 2 provides an heatmap of the evolution of the weights (wr,) over index i for uniformly spaced candidate factors in the range [0, 1). We set initial weights WsDD = 1/S for 0.99, the middle figure to 7 0.9 and the bottom
1,.., S. The top figure corresponds to figure to = 0.7, so that the quality of the autoevaluator decreases in going from the top panel to the bottom panel. The figure shows that, as the autoevaluator becomes less reliable, the update (13) correctly decreases the reliance of the e-value (11) on the effective observations that leverage synthetic data. Specifically, is seen from the figure that the weights {wa,)s== around the values 0.9, 0.5, and 0 ffror 0.99, 0.9, and 0.7, respectively.
S tend to concentrate

---

The overall procedure of R-AutoEval++ + is summarized in Algorithm 1 (Appendix B).
3.2 Sample Efficiency Guarantees
In this section, analyze the sample efficiency of R-AutoEval+. We start by reviewing existing results on R-Eval and R-AutoBval, and then we introduce formal version of Theorem (see Sec. 1) on the sample efficiency of R- R--AutEvaa++
3.2.1 Sample complexity of R-Eval and R-AutoEval
Reference [48] showed that the sample complexity of the e-value (6) can be analyzed by considering its maximum expected logarithmic per-round increment under the alternative hypothesis, i.e.,
[[[oo(1 X. (g o:)) R S a),
(14)
where X, is the optimal constant betting variable X, = arg 1o. ())) E[log(1 A(q: a))] R S a). Specifically, defining F (A.) = IT== (1 X. {- -a)) for the e-value (6) with constant
X X, for all rounds L.... nt, we have the following result.
Theorem 2 (Sample complexity of by-bettin-g (6) [48, Theorem 3.3]). Assume that: (AI) the betting strategy X; admits sublinear regret with respect to the optimal constant X., i.e. log (A.) log En o(n) (see Appendix for example of such betting strategy), and (A2) the instantaneous logarithmic increment of the e-value (x.).) has finite a-th central moment for some a > 2, i.e. Then, the sample complexity (3) obtained by the test
E| log(1 X (g. a:) 9-1 R S al variable T, in (5) with the e-value En in (6) admits the limit
Temin (8) 1
lim
(15)
log(1/6) g+ 8-+0+
To interpret this result, consider the second-order Taylor approximation log(1 y) - ////2. which yields (see also [48, Sec. 4.2))
~ 2 2
Var(q; R S a) (a - R)2
(16)
g.
According to this approximation, the sample complexity grows with the variance Var(q: a) Bllg; R S al of the observation under the alternative hypothesis.
The result (15) can readily applied to obtain the sample complexities of R-Eval and R- AutuEEall by setting 4i l and qi = E respectively, in the e-value (6). the presence of a low-quality autoevaluator, it is known that PPI increase the variance of the risk estimates [3, 4, 22, 10]. Therefore, based on the approximation (16), R-AutoEval may have larger sample complexity than R-Eval when the autoevaluation is not sufficiently accurate.
3.2.2 Sample complexity of R- A-AuutEEval+
R-AutoEval+ optimizes not only the S betting strategies { - but also the weights (we,). associated to the factors {p.]s- determining the reliance of the effective observations (10) on the autoevaluated data. Since the weight update (13) has the form of exponential weights [23], the following property follows from existing results on online convex optimization [18, Lemma 1].
Lemma (Sublinear regret for the weights (13)). For any set of betting strategies (A and for any positive initial weights {wa,o)mm the weight update strategy (13) satisfies
s
log (1 - Xs.i(es a))
log Σ
Wes (1- Ar. -a) S max log s=.....S
max $=1....
Wg.D
log Ey
log F.,
(17)
Intuitively, this result implies that the weight update (13) can identify the best factor Ps the set {p,5 determining optimal level of reliance autoevaluated data. In fact, the first term in
7

---

(17) represents the maximum, i.e., most informative, e-value among all the e-values {E corresponding to the S candidate factors {p=} This result, in turn, suggests that R A--uutEvaa+ may be able outperform both R-Eval and R- AutooEval, reducing to the former when autoevaluated data is of poor quality and to the latter when the autoevaluated data is sufficiently accurate. The following result formalizes this intuition.
Theorem 3 (Sample complexity of R R-AuttEEva+)) For every 8 = =,...,, S, suppose that the betting strategy ' satisfies assumptions (Al) and (A2) in Theorem with Es.n in lieu of En and q, E Then, the sample complexity of R A-AutoEval satisfies the following limit
2,,5
mimin A.AuuoRuaa+(())
(6)
lim 5-0
S
min L=.... Ia.*
(18)
(18(1/8)
Proof. See Appendix F.2.
The limit (18) implies the main result (4) in Theorem 1. In fact, by (15), the ratio 1/gs,+ in (18) corresponds to the scaling of the sample complexity of R-Eval and R- R-AutoEval by setting 1 and S, respectively. The next example shows that the inequality in (4)
can be strict.
Example Consider again the example setting in Example 1. The sample complexity Timin (6) is plotted in the top part of Fig. 3 as a function of log(110g) for a 0.12,
1.0
x10
(
x10
Re AutoBaaa
x 10
10, r 10, and
for (a) y = 0.99, (b) 7 0.9, and (c) Y = 0.7. The results are averaged over 100 independent experiments and we use the
R-Eval
Mamin
0.5
universal portfolio betting strategy (see Appendix C) [15, 16, 483.
Re AuboRvalt
The figure confirms the linear trend of the sample complexity with respect to the term log(1/5) Furthermore, it shows that the inequality (4) can indeed be strict, with R-AutoEval+ outper- forming both R R-Eval and R-AutoEval.
5
log(1 x10
log(1/6)
log(1/6) x10
x10
1.0
The bottom part of the figure plots the maximum expected log- arithmic increment of the e-value, I*.*. in (18) as a function of
a 0.5
the factor Ps for (a) 7 0.99, (b) 7 0.9, (c) 7 0.7. As the autoevaluator becomes less (more) reliable, i.e. as y de- creases (increases), the maximum value of the expected increment Is.+ is obtained for values of P. closer to zero (one), making R-AutoEval+ behave as R-Eual (R-AutoEval).
Ps
Ps
Ps
Figure 3: Sample complexity as a function of log(1/8) (top) and maximum expected increment of the log-e-value Is,+ as a function of pa (bottom) for Example 2.
More generally, using the approximation (16), one can conclude that the sample complexity of R- AutoEval+ is strictly smaller than for R-Eval and R A-AuvlEval as long as the variance of the effective observation tL, in (10) for some index different from = and = S is strictly smaller than for the effective observations & and e in (8). As shown in [4, Example 6.1]. this condition is satisfied when the autoevaluator sufficiently accurate.
4 Experimental Results
For experimental validation, we consider three model selection applications: 1) selecting the lightest quantized LLM with guaranteed performance drop as compared to the baseline model on the TriviaQA data set [28] (see Fig. 1); 2) selecting the shortest prompt template for an LLM with guaranteed accuracy on the Instruct-Induction task [27]; and 3) test-time reasoning budget allocation with guaranteed performance enhancement on the GSM8K data set [13]. We set S = 10 with Ps being uniformly spaced in the range [0, 1] and choose initial weights as w,,0 1/S. We refer Appendix E for results with different choices of such hyperparameters. All the results in this section are reported after averaging over 100 independent experiments, and H100 GPUs are used for LLM executions.
1) Selecting quantized LLMs: As shown in Fig. 1, for this task the candidate set of models consists of LLMs obtained from the baseline -lama-3.1-87 I-It-tnstruct by applying quantization in
'Code is available at https: htttp:://gttieii httggithu..cm//k.llloa AutoEval
8

---

(a) LLM quantization
(b) LLM prompting
Llama3, 3.ta.333..11
1Jama3.3-MXX
Ilama3 3-MX4
smallest model size [GB]
2-9
F-- Evall
prompt length
70
R- R-Bval
7.0
shortest
R Eval
R- Bval
6.5
AutoB
60
R-Auto Eval+
100
300
100
300 100
300
1- e-xcention accuracy
4
7
performance drop [%]
0.5
0.1
0.2
R-EvaL, A-Avoa
R-AutoBval+
0.0
Rvall R AutaBEval AutooEval-
1
4
7
100
200
300
autoevaluator prompt size
human labeled data size (n)
costrolnn model selection using R-Eva [47, 2]. R-AutoEval [20], and the proposed R- B-AuuEEval+ for the problems of (a) selecting the lightest quantized LLM with guaranteed performance drop on the TriviaQA data set [28], and (b) selecting the shortest prompt template with guaranteed execution accuracy on the instruction-nnductin smallest selected model versus the number n of real-world data points, and (a, bottom) corresponding performance drop. (b, top) Length of the shortest selected prompt template versus the number of in-context samples used by the autoevaluator, and (b, bottom) corresponding complement of the execution accuracy.
Figure 4:
task [27]. (a, top) Size of the
the MX format [40]. The candidate formats have different configurations specified by parameters [k1, lig, d1, d2, M] E No where k1, ky are the first, second block granularity levels, d1 d2 are the first, second scale bit-width levels. and M is the mantissa bit-width [40, Table IIJ. We set k (16, 64), d1 = 8, d2 = 1, M E {3, 4, .... 10} with k //2 = {2, 4, 8}. Model selection is carried out based on fixed sequence testing (FST) [7], visiting the candidates order of decreasing average bitwidth. FST guarantees the family-wise error rate (FWER) with error probability no larger than 5.
For the autoevaluator, we adopt larger Llama- 3. 1..1-70B-Inntree Instruct [24), whose quality is controlled by adjusting the weight precision of the autoevaluator from full precisscn BF16 to MX6/MX4, with average bitwidth decreasing from 16 6/4 [41]. We set 8 0.1, a 0.1, n 150, and r 5 for Fig. 1 while vary n from 100 to 300 with r = 3 for Fig. 4. Fig. 1 reports maximum values within the 1.5 interquartile range (IQR) range [34] across 500 independent experiments (the corresponding full box plot can be found in Appendix E).
Fig, 4(a) plots the size of the smallest selected model versus the number of real-world data points (top) and the corresponding performance drop against the full-precision model (bottom). The figure confirms that all schemes do not exceed the target maximum performance drop a 0.1. Furthermore, R-AutoEval+ outperforms both R-Eval and R-AutoEval when the autoevaluator is sufficiently accurate (Llama3.3-P)) Bataaa..--FFF/6MMM (Lla thus confirming the efficiency guarantee of R- A-AAutEEva++ + in Theorem 3.
while recovering R-Eval when the autoevaluator is of low quality
2) Selecting the shortest prompt template: For this second task, the candidate set consists of 25 prompt templates designed to enhance the zero-shot performance of Llama-3. 1..1--B--nnstru using the larger Llama-3.11700 Instructt [24] via the forward mode generation of automatic prompt engineering (APE) [53]. Model selection is carried out via the Bonferroni correction [9], thus applying the test (5) with 8 6/25 in lieu of 6.
For the autoevaluator, we adopt in-context learning [11] on the Llama-3.1 --.....0---Isttteeee with prompt examples randomly chosen from the same held-out data set used for APE. The accuracy of the autoevaluator controlled by varying the number prompt examples from to 7. We set 8 = 0.1, n 200, r = 9, with chosen as the minimum value in the set (0.05, 0.1, 0.95] for which R- A-auloEval [20] finds at least one reliable prompt template with the strongest autoevaluator. We select the longest prompt template if the model selection algorithm does not select any template.

---

Table 1: Selecting the smallest reasoning budget for Qwen3-1.7B that ensures at least 3% accuracy enhancement as compared to non-reasoning mode, evaluated on GSM8K data set [13]: average number of generated tokens with standard deviation shown within parentheses.
autoevaluator (aceuracy)
R-Eval
R- R-AutoEval
R-AutoEval+
BitNet b1.58 (35%)
950.27 (152.84)
942.47 (135.65)
Llama-3.2 3.m---..--B--- Instrrcctct 3.3----ntttt (66%)
900.58 (122.70)
892.86 (112.10)
Qwen3-32B (82%)
1007.45 (150.48)
941.20 (129.61)
983.34 (137.87)
DeepSeck-RI-De RI-Distill-Qwen-222 (89%)
893.39 (105.13)
866.22 (80.58)
Llama-3.3-70B In-ttInntttt (89%)
854.42 (90.26)
847.05 (69.21)
GPT-4.1 (93%)
883.99 (103.00)
856.13 (70.93)
Fig. 4(b) plots the length of the shortest selected prompt template versus the number of in-context samples used for autoevaluator (top) and the corresponding risk defined as the complement of the execution accuracy as defined in [27, A. The figure confirms that all schemes reach execution accuracy no smaller than 1 - a. Moreover, R- B-AuutEvaa+ consistently outperforms both R- E-Eall and R-AutoEval irrespective of the number of in-context samples used for the autoevaluator.
3) Test-time reasoning budget allocation: For the last task, the candidate set consists of computation budgets for the reasoning mode of the Qwen3-1.77 [51] base model, varying between 128 and 1280 tokens. Model selection is carried out via FST, visiting the candidates in order of decreasing reasoning budget. For the autoevaluator, we adopt different kinds of pre-trained LLMs, ranging from large-scale models such as GPT-4.1 [1] to light-weight models such as BitNet b .58 [32]. We set 8 = 0.1, 1000, 4, and 0.03 (i.e., reasoning should improve accuracy by least 3%).
Table 1 confirms again the efficiency gain of R- AutoEval+ as compared to R-Eval and R-AutoEval, saving up to 127 tokens over R- B-Eval and up 66 tokens over R- A-toEEval average. Choosing the autoevaluator from the same family of the model is seen to substantially reduce the gain of autoeval-based approaches. For instance, Llama-3.2-33 3.2-3B Instruct autoevaluator achieves much lower accuracy than Qwen3-32B autoevaluator (66% vs. 82%) but significantly helps reducing the number of reasoning tokens: R-AutoEval+ saves 90 tokens over R-Eval when using Jlama-3.2-3p Instruct autoevaluator, while it saves 42 tokens over R-Eval when using Qwen3-32B autoevaluator; R-AutoEval saves 83 tokens over R-Eval when using JJama-3.2-3B-Instree
autoevaluator, while requires 24 tokens more than R- R-Eval when using Qwen3-32B autoevaluator. Such behavior can be understood as a consequence of positive feedback of LLM judges within the same family [52], also known as preference leakage [30], which makes the bias correction (8) more challenging.
We refer Appendix E for further details and additional experiments.
5 Conclusion and Further Discussions
This work introduced R-AutoEval+, a novel autoevaluation method that can provably enhance the efficiency of the state-of-the-arr evaluation method while maintaining strict finite-sample reliability guarantees. The theoretical properties of R- AAutovaa+++ were confirmed by experimental results on LLM quantization and LLM prompting with LLM judges as autoevaluators.
Some limitations of this work are as follows: (i) R- AutoEval++ requires access to real-world unlabeled data; (ii) the discrete set of candidate factors determining reliance on synthetic data are fixed priori; and lastly, (ii) the sample efficiency guarantee in Theorem only holds for sufficiently high target reliability levels 1 8. Addressing these limitations may leverage the tools in [3, 16, 36), and leave these directions to future work.
Another interesting direction for future research includes combining the benefits of R- AutoEval+ in adaptively weighting synthetic data with the complementary advantages of methods that actively select real data [49, 46, 54, 17, 50].
10

---

Acknowledgments
This work was supported by the European Union's Horizon Europe project CENTRIC (101096379), The work of Osvaldo Simeone was also supported by the Open Fellowships of the EPSRC (EP/W024101/1) and by the EPSRC project (EP/X01 1852/1). Thanks also the Advanced Research and Invention Agency (ARIA) for supporting broader foundational work in this space. The authors also thank Bipin Rajendran for his advice on LLM quantization.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Anastasios N Angelopoulos, Stephen Bates, Emmanuel J Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint wrXiv22100.0005: 2021.
[3] Anastasios N Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zrnic. Prediction-poweret inference. Science, 382(6671):666--67) 2023.
[4] Anastasios N Angelopoulos, John C Duchi, and Tijana Zmni. Ppi++: Efficient prediction-
powered inference. arXiv preprint arXiv:231100155; 2023.
[5] Sher Badshah and Hassan Sajjad. Reference-guided verdict: Lims-as-judges in automatic evaluation of free-form text. arXiv preprint arXiv:2408.0922 2024.
[6] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribation-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1-34, 2021.
[7] Peter Bauer. Multiple testing in clinical trials. Statistics in medicine, 10(6):871-89) 1991.
[8] Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, et al. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint 1rXiv:2212.080037 2022.
[9] Carlo Bonferroni. Teoria statistica delle classi calcolo delle probabilita. Pubblicazioni del R istituto superiore scienze economiche commericiali di firenze, 8:3-62, 1936.
[10] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan, Au- toeval done right: Using synthetic data for model evaluation. arXiv preprint S.Ii:22403.00000 2024.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. The price of debiasing automatic metrics in natural language evaluation. arXiv preprint 1rXiv:1807.02200 2018.
[13] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2/ /4/0/////8, 2021.
[14] David Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4: 1:129-145, 1996.
[15] Thomas M Cover. Universal portfolios. Mathematical finance, 1(1):1-29, 1991.
[16] Thomas M Cover and Erik Ordentlich. Universal portfolios with side information. IEEE Transactions on Information Theory, 42(2):348-366 2002.
[17] Daniel Csillag, Claudio José Struchiner, and Guilherme Tegoni Goedert. Prediction-powered e-values. arXiv preprint arXiv:2502. 04294, 2025.
[18] Steven De Rooij, Tim Van Erven, Peter D Grünwald, and Wouter M Koolen. Follow the leader you can, hedge you must. The Journal of Machine Learning Research, 1581)11281-1316, 2014.
11

---

[19] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[20] Bat-Sheva Einbinder, Liran Ringel, and Yaniv Romano. Semi-supervised risk control via prediction-powered inference. arXiv preprint vrxiv:24/2.11174, 2024.
[21] Benjamin Eyre and David Madras. Auto-evaluation with few labels through post-hoc regression. arXiv preprint 1xXi::2411.16666 2024.
[22] Adam Fisch, Joshua Maynez, Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powe powered inference for hybrid language model evaluation. arXiv preprint arXiv:2406.0429) 04291, 2024.
[23] Yoav Freund and Robert E Schapire. A theooeiii generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):1 1900)1999199,,, 139, 1997.
[24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407:21783, 2024.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. arXiv preprint arXiv:2501.12944, 2025.
[26] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding. pages 409-426, 1994.
[27] Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.10778) 2022.
[28] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.0355/1 2017.
[29] Adam Kalai and Santosh Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research, 3(Nov):423-440, 2002.
[30] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang. Wei Wang, and Huan Liu. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.011 553.011344 2025.
[31] Chia-Wei Liu, Ryan Lowe, Julian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint 2003./6600.0002. 2016.
[32] Shuming Ma, Hongyu Wang, Shaohan Huang, Xingxing Zhang, Ying Hu, Ting Song, Yan Xia, and Furu Wei. Bitnet bl. 58 2b4t technical report. arXiv preprint arXiv:2504. 12285, 2025.
[33] Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint arXiv:0907.3740, 2009.
[34] Robert McGill, John W Tukey, and Wayne Larsen. Variations of box plots. The american statistician, 32(1):22- 1978.
[35] Jekaterina Novikova, Ondřej Dušck, Amanda Cercas Curry, and Verena Rieser. Why need new evaluation metrics for nlg. arXiv preprint arXiv:1707.0687. 2017.
[36] Francesco Orabona and Kwang-Sung Jun. Tight concentrations and confidence sequences from the regret of universal portfolio. IEEE Transactions on Information Theory, 2023.
[37] Aaditya Ramdas and Ruodu Wang. Hypothesis testing with e-values. arXiv preprint arXiv:2410.23614 2024.
[38] Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576-601, 2023.
[39] Siva Reddy, Danqi Chen, and Christopher D Manning. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266, 2019.
12

---

[40] Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mesmakhosroshahi, Ankit More, Levi Melnick, Maximilian Golub, Girish Varatkar, et al. With shared microexponents little shifting goes long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1-13, 2023.
[41] Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, et al. Microscaling data formats for deep learning. arXiv preprint arXiv:2310.10533 2023.
[42]
Lennart Schneider, Martin Wistuba, Aaron Klein, Jacek Golcbiowski, Giovanni Zappella, and Felice Antonio Merra. Hyperband-base bayesian optimization for black-box prompt selection. arXiv preprint arXiv:2412.078200 2024.
[43] Glenn Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2)407-4311, 2021.
[44] Jean Ville. Etude critique de notion de collectif. Nouthicr-Villaan Paris, 1939.
[45] Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The
Annals of Statistics, 49(3): 736-1754, 2021.
[46] Ganghua Wang, Zhaorun Chen, Li, and Haifeng Xu. Cer-eval: Certifiable and cost-efficient evaluation framework for llms. arXiv preprint arXiv:2505.08 038/4, 2025.
[47] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1-27, 2024.
[48] Ian Waudby-Smith, Ricardo Sandoval, and Michael I Jordan. Universal log-optimality for general classes of e-processes and sequential hypothesis tests. arXiv preprint 0228:2204.002888 2025.
[49] Ziyu Xu, Nikos Karampatziakis, and Paul Mineiro. Active, anytime-valit risk controlling prediction sets. arXiv preprint arXiv:2406. 10490, 2024.
[50] Ziyu Xu, Catherine Wang, Larry Wasserman, Kathryn Roeder, and Aaditya Ramdas. Active multiple testing with proxy p-values and e-values. arXiv preprint arXiv:2502.057/5: 2025.
[51] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprins arXiv:2505.09382 2025.
[52] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-asaa-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4466 2023.
[53] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
[54] Tijana Zrnic and Emmanuel Candès. Active statistical inference.
arXiv preprint
arXiv:2403.03208, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claim of the paper is: the proposed autoevaluation method is reliable while ensuring better (at least no worse) efficiency compared to existing benchmarks. We have clarified this in the abstract and introduction, and theoretically proved this claim Sec. 3, followed by the empirical justifications in Sec. 4.
Justification:
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have clarified the two assumptions required to formally guarantee the main claim the paper, which are the same assumptions imposed in prior art. Nonetheless, we have shown experimentally that in practical scenarios (Sec. 3), irrespective of the formal checking of the assumptions, the main claim of the paper holds. In Appendix D, we further clarified this point by considering the setting that does not necessarily hold the assumptions, for which the main claim still holds empirically. Lastly, in Sec. 5, we summarized the limitations of this work and provided some possible directions mitigate the issues.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The theoretical results that we have in this paper are: Thm. I (informal, Sec. 1), Thm. (Sec. 3), and Lemma (Sec. 3). The informal Thm. I is proved in Thm. (clarified after Thm. 3) while Thm. 3 is proved in Appendix E.2 that contains both proof sketch and the full proof. Lemma is also proved in Appendix although we have pointed out classical references that contains the proof.
Justification:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
We have provided a new framework for autoevaluation which is fully summa- rized in Algorithm (Appendix B). We have carried out four experiments: (i) toy case; (ii) LLM quantization; (iii) LLM prompting; (iv) LLM reasoning, and the corresponding details including the specification of the LLMS are provided in Sec. 4. We also provide the full code (Supplementary Material) that reproduces all the results in the paper.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide full code in the Supplementary Material to reproduce all the results in the paper. As the experiment involves LLM executions (LLM quantiza- hunnproompiingrrcsonnnngg we have also provided the corresponding raw data from LLMs facilitate the reproducibility that does necessitate actual LLM runnings.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
14

---

Answer: [Yes]
Justification: The proposed framework, fully summarized in Algorithm 1, has only two hyperparameters, and have provided the default choice for them with which all the experimental results are obtained.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: This paper itself is about achieving reliable and efficient error bars. Nonethe- less, all the experiments are carried out by averaging least 100 independent experiments which are clarified in Sec. 4.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have reported in Sec. 4 the type/number of GPUs used to obtain the experimental results, although such GPUs are only required for data acquisition and the actual algorithm has negligible complexity.
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https:///neuripe wwtpuurps..cpppulllpe pptpblic/thhcsscddeell..oo
Answer: [Yes]
Justification: This work is about obtaining statistical error bar and we have used standard data set that does not violate any of the Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is about foundational research on statistics and does not involve any societal impacts.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work provide a theoretical framework that does not contain any data or models hence poses no such risks.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have used pretrained LLMSs and data set commonly used in research community. We have cited the corresponding original sources in Sec. 4.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
15

---

Justification: This work is about theoretical framework for autoevaluation hence does not contain any assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used for the development in this research.
A Further Related Work
Active evaluation. While standard Eval leverages a fixed real-world data set, active evaluation aims at reducing the labeling cost by adaptively choosing data for human labeling [14, 46). Active evaluation has also been extended AutoEval [54, 49, 17]. This line of work is orthogonal the solution proposed in our work, which focuses on adaptively weighting synthetic data rather than adaptively selecting real-world data.
Game-theoretic statistics. Game-theoretic statistics builds on the notion of e-value. E-values are measures of evidence that offer several advantages over conventional p-values, including the support of optional continuation [38. 37]. The interpretation of an e-value as the wealth associated with a betting strategy makes possible optimize statisties using tools from online convex optimization, leading to state-of-the-- estimation and testing mechanisms [45, 47, 6J. Moreover, active e-values [49] support the implementation of active evaluation [17, 50).
B Algorithm
Algorithm summarizes the overall procedure R-AutoEval+, which reduces to R-Eval [47, 6] by setting P3 = 0 for all s 1, ..... S, and to R-AutoEval [20] by setting Ps = 1 for all S...,
In Algorithm 1, we consider the test
T, --=1( (1-1(mmm max E, 2
(19)
(E(1.....))
which meets the reliability condition (2) thanks to Ville's inequality [44]. The testing design (19) is standard eestingby-bbttingg frameworks [47, 6) due its higher power, as it returns T, 1 any time the test (5) does.
16

---

Algorithm 1: R-AAuoEval+
Input: human-labeled data D,, {Xx, Y) unlabeled data Di = { autoevaluator f, target risk level reliability level - 8, reliance factors (ps | satisfying (9), initial
positive weights {Wa,0)5-1 satisfying Σ Wa,0 betting algorithm Alg (see Sec. C)
Output: test output T.
initialize Eg + 1, Es,0 + 1; A = Alg(E,,2 denoting as f11 1e, 11 Wa,1 Wa,0
for s 1,.... 5
for i
I,.... do
for L,..., S do
get effective observation t/
gf [N/n] E/ (o f(x,, f(xx)) + (X,, Ps (x,,
update Fgg
Es,i + Eg,s-1 (1-Aax) i (C., a))
4 update betting variable 20,441 Asiit1 Alg((e (,11))
end
update e-value Eg
E; + 1i-1- E.= Wa Ca,i
update weights Wsii+1 W,,0-E,,
Ws.i+1 = Σe B
for all 1.......
end
return 1 ( (max)E(l.....)) E, 1/8)
C Betting Strategy
In this section, we summarize commonly used betting strategies [47, 48], namely Waudby-- Rmith-Ramdas (WSR) [47] and universal portfolio (UP) [15]. While both schemes show excellent performance empirically, we start with UP, which satisfies assumption A Theorem 2.
In this section, we denote as Alg a betting algorithm that maps the available past observations into the next betting variable, i.e., at round i, given previous observations 91:-1 = {qr}}=1 with gi E [m, M). a betting algorithm outputs X; = =Alg(q(ss Al)((1.-1))
C.1 Universal Portfolio (UP)
Recall the definition of (A) = II (1 - X(g: - a)), which is the e-value (6) with X; = À for all
i 1, ..., n. UP [15] defines the betting variable X; round i
- Alg(gna-1) M So 1
1
(20)
which satisfies the assumption A1 in Theorem (see also [48, 36]). The integrals in (20) are computed by discretizing the continuous domain [0, 1] of À into a uniformly spaced grid of size 10000 manner similar to [15, Sec. 8].
17

---

Table 2: Computational complexity comparison of the algorithms considered in this work.
Eval
AutoEval
R-Eval
R-AutoEval
R- R-AutoEval+
(nn) O(N)
O(n + nG) O(nr + nG) O(Snr + SnG)
C.2
Waudhy--Smith-Randee (WSR)
WSR defines the betting variable A; at round as [47, 6, 20]
(1gg( nož_1 f
Alg(nu-11 = min
(21)
M - a²
where the empirical standard deviation and empirical mean round i - 1 and at round are respectively defined as
0. 00 + 1== (g) - -3) and j. pm + 1k
(22)
j i+1
with initial guesses Po 1/2 and 0 = 1/4 [47, 20J. We set c = 3/4 following [47].
D Computational Complexity
Here compare the computational complexities of the algorithms considered in this work. Given both the human-labeled (size n) and autoevaluated data (size N). mean-based approaches (Eval and AutoEval, which do not come with reliability guarantees), require (nz) (for Eval) and O(N) (for AutoEval) operations for computing the respective average.
The reliable approaches (R- (R-Eval and R-AutoEval) need additional computation steps for updating the e-values well as the betting strategies (Sec. 2). While updating the e-values each round requires single multiplication, the betting strategy may be more complex. For example, especially the universal portfolio (UP) strategy with sublinear regret (Assumption Al) has a complexity of order O(nG)) where G' is the size of the grid used to approximate the integral operation of UP (Sec. C.I), although this can be decreased by adopting sampling-bas based approaches [29].
Lastly, R- A-AuutEval++ requires S times more computation than R R-AutoEval due to its consideration of S candidate factors. It worth emphasizing that, the costs described above are generally negligible with respect to running the autoevaluator (e.g., an LLM judge). The summary of the computation complexities can be found in Table
E
Additional Experimental Results
While UP betting provably achieves sublinear regret bound (Al in Theorem 2), WSR betting shows excellent empirical result with significantly lower computational overhead. Furthermore, existing works on R-Eval [47, 6] and R-AutoEval [20] adopted WSR as their betting strategy. Accordingly, here investigate the impact of the choice of betting strategy between WSR and UP (see Sec. C for the summary of WSR and UP). Note that we have used WSR for Fig. to maintain consistency with prior art, while for all the other figures, we have used UP, which satisfies assumption 1l in Theorem 2. In this section, we provide all the missing figures with the alternative betting strategy choice. After showing all the counterparts of the figures in the main text, provide additional experiments on LLM quantization for CoQA data set [39] and on constructing two-sided confidence interval for the setting studied in Examples 1 and 2. Lastly, we provide an extended version of Table I for LLM reasoning task by accounting a wider range of autoevaluators. We conclude this section by investigating the impact of R- AutoEval+'s hyperparameters such as S, {w.,o) 1=1 and {p.] as well as of the ordering of data, on the amount of reasoning tokens that can be saved while ensuring the target 3% accuracy gain.
E.1 Toy Example
In Fig. 5, we provide the counterpart to Figs. and 3, considering a WSR betting strategy instead of the UP one. is observed that WSR betting makes the weights of A-AtoEEva++ less concentrated around
18

---

x10
X
10
(a)
(P)
reliance factors p
1.00
(6)
Tt B-rar
0.00
1.00
0.02
B Ausohvval
0.00
0.01
5
1.00
log(1/6)
log(1/8)
(1g(1
0.00
(c)
1.0
<10-2
x10=2
x10-
5000
0.5
rounds (f)
0
Pa
Pe
Pe.
Waudhy-Smith Ramdas (WSR) betting
Figure 5: Same setting with Figs. 2 and but with WSR betting instead of UP betting.
LLM judga
LLM judgo
human julge
human
judge
PC
Llam3 Hamx3.3-B00
L2ma3 3-8F16
Lbmm33 3-MX6
ped mauue deo (8)
perferunce deep
Tecch Jalm [al))
mode
Biall
Rutrnal
Autoitval
Astoshall
R-Atntationl
At Asttaaiall
Autoltvall
St. Antto
Bival
Ratvvll
Asdeavvll
Rt AutesSval
Autoftaa
R-dntoBival
R.Antelvall
Auteiivaal
RotnteBrrll
Waudhy.Smith Smith
(WSR)
pontolio (UP)
Figure 6: (a) Box plot for Fig. 1; (b) same setting as (a) but with UP betting strategy (see Sec. C.1).
the optimal reliance factor, although it still outperforms the other two schemes with substantial margin.
E.2 LLM Quantization
Fig. 6(a) shows the box plot for Fig. 1. Fig. reports maximum values within the 1.5 IQR range [34]. Fig. 6(b) plots the same box plot but with UP betting instead of WSR betting. Fig. 7(a) is the counterpart of Fig. 4(a) which uses WSR betting in lieu of UP betting. is observed that even under the WSR betting strategy, R-AutoEval+ consistently outperforms R-Eval and R-AutoEval, returning a smaller quantized model.
E.3
LLM Prompting
Fig. 7(b) is the counterpart of Fig. 4(b) which uses WSR betting lieu of UP betting. Fig. 8 then shows per-task results of Instruction-Induee We show tasks among 24 tasks that have more than 2000 examples [27]. Even under the different choice of betting strategy, R- A-AutoEvaa+ outperforms both R-Eval and R- A--uoEvvll
task [27] with (a) UP betting and (b) WSR betting.
19

---

LLM quantization
LLM prompting
3-8F16
LJama3.3-M00
Liuma3. 5.M44
smallest model size [GB)
7.0
B-AutoBrvll
prouip length
B B-AutoBval
shor Soooorti
R- Eval
R-Evall
6.5
60
AutoEval+
AutoBBaar
performance drop [%)
100
300100
300 100
300
exocution
accurary
0.5
0.1
0.2
0.0
Roal, AutoRval, AutoRvall
100
200
300
antoevaluator
human labeled data size (n)
prompt size
Waudby-Smitt Saibyy-Smith----e Ramdas (WSR) betting
Figure 7: Same setting as Fig. 4 but with WSR betting strategy (see Sec. C.2).
shortest prompe length
1-execution Rocurney
universal portfolio (UP)
(b) Waudby-Smith Ramdas (WSR)
Figure 8: (a) Per-task result for Fig. 4(b): (b) same setting as (a) but with WSR betting strategy.
E,4 LLM Quantization: CoQA Data Set
We extend the experimental validation R-AutoEval+ by considering different data set for LLM quantization task. Fig. 9 is the counterpart of Fig. 4(a) but with CoQA data set [39] under (a) WSR betting and (b) UP betting. The same trend is observed, confirming the improved efficiency of R-AutoEval+.
E.5
Confidence Interval
Under the same setting as Examples I and 2, we consider the construction of two-sided confidence interval for the expected risk R using R-Ev A-val, A-auulEval, and R-AutoEval+. We follow the hedged capital process [47, Sec. 4.4] to first construct (1 5 c)-upper confidence bound and 8 (1 =loee confidence bound separately to yield their intersection as the two-sided confidence interval. An upper confidence bound can be obtained by finding the largest a for which respective testing T, (5) yields a lower confidence bound can obtained similarly by considering the observation g. adopt uniform grid of size 10000 to approximately find the largest a with T, = 0. We employ the WSR betting strategy, and set the upper bound of X, in (21) to 1/ m) to ensure the monotonicity
+ M ga in lieu of the observation q:- We set e 0.5 and
20

---

CoQA data
Limed LMM
Limre7 2-0A4
staaless model sime (GB)
model size (CB)
A-Aittalival
6.0
B-veral
AUROEVAII
R- Ball 300100
100
300100
300 100
300
100
300
100
300
performanc drop [38]
0.1
divo (96)
0.1
0.0
R- 2.BrL 2 AutoEval & AutoRrior
0.0
7-Brel To TeAaroftool BrAuttoPrncs
100
200 Iuman labeled duta size (x)
300
100
200 Inmnn labeled dataa size (x)
WaaddhySSmith-Rmdll (WSR)
(b) universal portfolio (UP)
Figure 9: LLM quantization task on CoQA data set [39]. All the other settings are the same with
Fig.
0.20
confidence interval
R-Eval
AuAutBEval+
0.15
Re AutoEval
0.05
0.00
10F
10a
10
10P
10
10"
10"
10
10'
0.20
(8)
(b)
(0)
R A-AuutBBval
confidence interval
0.15
0.10
R- R-AutoBval
R-Eval
0.05
0.00
102
10
10
10
10
10a
10
102
10"
human-labelld
human-labeled
labeled
data size (n)
data sine (n)
data
Figure 10: 99.9% two-sided confidence interval for the same setting in Example 2.
of E-Ea''s and R-AutoEval's 's e-values [20, 6]. Fig. 10 shows the corresponding (top) confidence interval and (bottom) its width versus the number of real-world data n. Recall from Example and 2 that the quality of synthetic data decreases in the order of (a), (b), and (c). Across all scenarios, R-AutoEval+ returns confidence intervals that have the smallest size.
E.6
LLM Reasoning: Wider range of autoevaluators
Table extends Table in the main text by considering additional autoevaluators: BitNet [32], Llama-3 [19]. DeepSeek-RI [25], Qwen3 [51], and GPT4 [1]. The table confirms again the efficiency gain of R- A-tttEEva++++ compared to R-Eval and R AutoEval; also demonstrating again the benefit of choosing autoevaluator from the different family of the base model to achieve better efficiency for autoeval-based approaches.
E.7
LLM Reasoning: Sensitivity of testing-by-betting--yy algorithms with respect to data ordering
We first recall that the algorithms that builds upon the festing-by-bettingg by-betting framework [47] need to impose some arbitrary ordering the data to sequentially run the algorithm. Here we investigate the impact of such ordering on the final decision of the algorithms. To this end, we fix the calibration-test data split and only change the ordering of data randomly for 100 times to compute the normalized deviation (standard deviation / mean) of the generated tokens dictated by each algorithm. We consider
21

---

Table 3: Selecting the smallest reasoning budget for Qwen3-1.7B that ensures at least 3% accuracy enhancement as compared to non-reasoning mode, evaluated on GSM8K data set [13]: average number of generated tokens with standard deviation shown within parentheses.
autoevaluator (aceuracy)
R R-Eval
R-AutoEval
R-AutoEval+
BitNet b1.58 (35%)
950.27 (152.84)
942.47 (135,65)
Llama-3.2- 3.2-aB In-ntrucctt (26%)
964.91 (136.55)
954.63 (145.60)
DeepSeck-FE 1----RR--Diiii --Distill- 1.5B (33%)
943.98 (136.38)
935.58 (130.83)
Llama-3.2-3B -nn-trrrrutt (66%)
900.58 (122.70)
892.86 (112.10)
(54%)
941.66 (126.32)
926.85 (118.08)
DeepSeek-RR--isttllll
(67%)
916.34 (129.83)
900.82 (119.73)
1arp3seee-R--52----eee (32%)
983.34 (137.87)
986.53 (140.77)
961.94 (138.98)
Qwen3-32B (82%)
1007.45 (150.48)
941.20 (129.61)
DeepSSckkKI--Distil----e---- (89%)
893.39 (105.13)
866.22 (80.58)
Llama-3.3-70B-IIsstruu (89%)
854.42 (90.26)
847.05 (69.21)
DeepSeck-RI--Dstti-Lllaa---- (88%)
863.72 (71.60)
858.02 (69.59)
GPT-4.1 nano (90%)
879.45 (93.95)
865.28 (72.34)
GPT-4.1 (93%)
883.99 (103.00)
856.13 (70.93)
Table 4: Sensitivity of algorithms with respect to data ordering. Normalized deviation (standard deviation mean) of the generated tokens yielded by each algorithm for fixed calibration-test data split is shown for each algorithm.
autoevaluator
R-Eval
R-AutoEval
R-AutoEval+
GPT4.1
8.62%
3.90%
6,88%
DeepSeek-RI-On -I-S5eek-R1-5288 -0122-Q
8.62%
6.32%
6.00%
GPT-4.1 and DeepSeek- R1 -0520-02--00 Qwen3 8B as the autoevaluators for the experimental results. As can be seen from Table 4, R- AutoEval and R-AutoEval+ + are more robust to the data ordering than R-Eval, which may be possibly attributed to its reduced variability of averaged autoevaluated data (at each round we average number of autoevaluated examples), while R- A-AuulEval+ + tends to be slightly less robust than A-AuoEEval for adaptive nature that incorporates the data ordering.
E.8
LLM Reasoning: Impact of the number of candidate factors S
We now start investigating the impact of hyperparameters required by R-AutoEval+. We first investigate the impact on the number of candidates, S, with candidate factors equally spaced within the interval [o, 1]; and with equal initial weights. We report the average number of generated tokens with corresponding standard deviation in manner similar to Table and 3.
As can be observed in Table 5, as long the candidate factors includes p 0 (R-Eval) and p = (R- Ao-AutoEvval, i.e., as long as S 2 2, R- AutoEval+ consistently outperforms both R-Eval and R- AutoEval, with its gain being maximized at S and saturates after S 10.
E.9 LLM Reasoning: Impact of the values of candidate factors ((p))=)))
Next, we investigate the impact of the values of each candidate factors by going beyond the equally spaced grid. To this end, we consider Dirichlet distribution with concentration parameter 0.1, 1, 10 to allocate the values within the [0, 1] interval. Specifically, the cumulative probabilities the Dirichlet distribution used the candidate factors and we keep the first and the last factors to be and to
Table 5: Impact of the number of candidate factors (S) for R-AutoEval+ The other settings are the same as Table and
autoevaluator
S 2
S = 5
S = 10
= 20
GPT4.1
871.67 (16.24)
854.99 (14.02)
856. 13 (13.90)
856.13 (13.90)
DeepSeck-RI R1-0528 1-558-Qween--BB
964.41 (27.17)
961.94 (27.24)
961.94 (27.24)
961.94 (27.24)
22

---

Table Impact of the values of candidate factors (fpa). 1) for R-AutoEval+, The other settings are the same as in Table I and 3.
autoevaluator
equal grid
Dir( 10)
Dir(1,... ..... 1)
Dir(0.1, ..., 0.1)
GPT4.1
856.13 (13,90)
856.13 (13.90)
860.91 (15.63)
873.27 (15.10)
SeapSeek-R1-0525 -1-0552--0w2-38-BB
961.94 (27.24)
964.22 (26.77)
960.76 (27.09)
966.72 (27.
Table 7: Impact of the initial weights (fw,,o)s-) for R-AutoBval+ The other settings are the same as in Table and 3.
autoevaluator
equal weight
Dir(10, .... 10)
Dir(I, ... 1)
Dir(O.1, 0.1)
GPT4.1
856.13 (13,90)
856.13 (13,90)
856.13 (13.90)
856.68 (12.31)
961.94 (27 (27.24)
959.21 (27.04)
962.41 (26.52)
969.50 (25.59)
ensure the efficiency guarantee as per our Theorem 3. An increased concentration parameter makes samples from the Dirichlet distribution closer to the uniform distribution, and hence the grid becomes closer to the equally spaced grid. Table shows the preference for setting the grid with equal spacing.
E.10 LLM Reasoning: Impact of the weights associated with the candidate factors ((w,,olsm))
The last remaining degree of freedom given by the choice of the weights associated with each candidate factor. To this end, go beyond the uniform distribution and consider again the Dirichlet distribution to allocate the weights for the candidate factors. Table 7 confirms again the preference for equal weighting.
F
Proofs
Here we provide proofs of Lemma and Theorem 3.
F1 Proof of Lemma
This proof follows as in [16, Sec. III] (see also [18, Sec. A] for a more general version).
Proof. We first rewrite the e-value E.,, in (12) in an exponential form
Essi = exp Σ log (11 Aasslles - a))
(23)
where 9x.j is the increment of log-e-value log Ex.j round j. We will further simplify the notation by writing Is,1% as the summation of logarithmic increments up to round i, i.e., Is,1s = Ej=1 Js.j- The update rule (13) for the weight Ws,i can then be rewritten as
We.u exp (ge,1:i-)
(24)
Ee= Wo. exp (9e(.::1-1)
Now, by unfolding the product term in (11) as
>2 W exp 1:1,1:1)
S 2s Wa.0 exp (ga,
E,
(25)
Dam Wx.0
Σ. | Wg.0 exp (9),1mm-1)
have
E,, Σ
Wso0 exp (g.. =
Σ Wa,0E
(26)
am
am
23

---

For any positive weights {ws,o)=-1- it follows that
En 2 Ds,n for all I,..,,,
(27)
from which we conclude
max log En,n - log En S
max log
(28)
SE (E....)))
Ws,0
L
F.2 Proof of Theorem 3
We start this subsection with proof sketch of Theorem 3 that builds upon [48, B.4]. This is followed by a series of useful properties of the R- R-AutoEval+3s e-value (11), which are instrumental for proving Theorem 3. The full proof then follows the proof technique in [48, B.4].
F.2.1 Proof sketch
Waudby-Smith et al. [48] showed that the sample complexity of testing (5) given some e-value E, of the form (6) be studied by investigating the simpler e-value with optimal constant betting strategy A Recall that in Sec. 3,2.1, we have already defined such simpler e-value, and we repeat its definition here again for convenience
E, (A.)
II (1 2. a)).
(29)
The rationale for this simplification is follows: (i) The simpler e-value cannot be too large as compared to the actual e-value obtained using betting strategy that satisfies assumption A1 in Theorem 2 (sublinear regret); (ii) the actual e-value also cannot be large compared to the simpler e-value (so called numeraire property, see below Lemma 2), hence one can instead study the sample complexity of such simpler e-value in order to characterize the actual sample complexity of R-Eval.
That is say. as long as we can also find simpler e-value with bounded deviation from R- AutoEval+'s e-value E, in (11), we can then study the corresponding sample complexity instead. As can be anticipated from Theorem such simpler e-value is described as
Ean
(1. -
(30)
i=1
where is the index that maximizes the expected increment of the s-th log-e-value, i.e.,
arg max Is.t
(31)
a S
While intuitively R- A-AuutEval+'s e-value has a bounded deviation from Es,n (Ag..) ). one subtlety here is that, the best index & might not be equal with the one that is chosen at hindsight, i.e., the following event can happen
arg max Is.+ + arg max Es,.. s=....,
(32)
8= L
with non-neglig probability. To overcome this issue, we foeus on the upper bound the sample complexity by considering a deterministically no larger e-value E,; as compared to R- AuttoEval+'s e-value, defined as follows
s
E Σ wso min Es,,y Ex, S Σ
Wa,0 Ea, (26)
En.
(33)
a=1
Accordingly, we denote the sample complexity of E as
minn (0) E[min(n: E 2
a)
(34)
which is no smaller than n, R-AutoEval+ due to (33). We then show the upper bound of ni (8). which min
will prove Theorem 3.

---

F.2.2 Useful properties
We first summarize the useful properties of E;, which bounds its amount of deviation with respect to the simpler e-value Es,n (As,,). We first discuss the regret, which dictates the lower bound deterministically, then discuss the suboptimality ratio which captures its upper bound probabilistically:
Regrets: The regret of E as compared to the one that chooses the constant betting at hindsight with index chosen as can be formally defined as
k
max AEt0.1((tPp--a)
log (s, log Ej
max log Es,((A) log Ez, + log Es - log Eg
XE AE(U.1((+Prraa)
log (2) log Es,k + log Es, log Σ w. min (Eg m E
xc(0.1/11Px-a))
=
2wx
I <regret 3,k + log Fa, log( Wx.0 Fa,s) = regret,,k + (og(1/ws.p)
(35)
where (a) is due to the definition of Ex in (33), and we have denoted as regret the regret betting associated with the s-th e-value Es in (12) round n, which is sublinear under the choice of betting strategy as per assumption Al in Theorem 2, c.g., universal portfolio strategy (see Appendix C). Accordingly, under assumption A1 in Theorem 2, we conclude that ri is sublinear, i.e.,
rk on).
(36)
The probabilistic upper bound of e-values can be captured by the expected ratio with respect to the simpler e-values. We first summarize the known results of R-Eval [48].
Suboptimality ratio:
Lemma (R-Eval's suboptimality ratio [48, Lemma 5.3]) Consider an e-value of the form (6) with some arbitrary betting strategy Xi Under the alternative hypothesis Hi R S the following
property holds at any round n
E
E
1.
(37)
En (A.)
We then extend the above property to E in (33).
Lemma 3 (E 's suboptimality ratio). Consider the e-value E (33) with some arbitrary betting strategy {Aa,) Under the alternative hypothesis Hi R S. o, the following property holds at any round n
E
1.
(38)
Eg,n (dg,a)
Proof. We first recall Ex = E 0, minf Es,,y Ex,tt from (33). We then have
Ex, Es,n (Aa,,) where the last inequality is due to Lemma 2 by noting that s is fixed variable.
Es Ws.0 Es,n ()s,))
E
S 1.
(39)
E
F.2.3 Full proof
We now show the upper bound of R-AutoEval+. Most of the steps are identical with the proof technique of [48, B.4] that regret and suboptimality ratio to get upper bound of the sample complexity. We just need to take into account for the regret and suboptimality ratio that are tailored for R-AutoEval+ defined in the previous subsection.
Upper bounding n min (34)).
AA-AutoEvalt (8). As discussed earlier, we focus on the upper bound of ntt (8) (see
25

---

Waudby-Smitt et al. [48, B.4] started the proof by first showing that the expected stopping time E[r] given the stopping time of random process (W defined in E N Wn 2 1/8). can be simplified as
Elr] S m 1
+
Pr[lk log Wa - gl 2 cl
(40)
k=
for some constant 9, where the variables e and m are respectively defined as
1 3 and [10{
B
(41)
9
given some 3 E (0, 1).
We now start our proof by taking E3 and -s,+ in lieu of W2 and g from (40), i.c.,
Th min (8) S m + 1+
Pr
k-
Is,* 2 c, log Ei
(42)
k=m
(0)
with the variables and m being redefined
1+ g5,+ and m 1og(1/8)
3
c
(43)
1
9a..
Following the next step of Waudby-Smin et al. [48, B.4], we upper bound (D) as
k: log Ex k- log (x,(()s 2 c/2
m
(1)
Pr kk log (as,()s,))) - gs,Nl 2 c/2)
(44)
(tt)
Note that each term in (44) measures the tail probability associated with the deviation between (t) Et 's logue-value and the one with optimal constant betting/weighting (tt) average log-e-valus and the expected log-e-value.
We first bound (t). The k-th summand of (f) can be bounded as follows denoting as A --1 looo /F// Fzz (Ag,,)): Pr[|Ak| 2 c/2) s Pr[As 2 c/2) Pr(-A, 2 c/2)
= Pr log Es, (Aa,.)) 2 kc/2]
+ Pr(k log Es () - -ー log Exx 2 c/2]
- exp(-ke/2) 1 (k: -r 2 c/2),
(45)
where the first part of the inequality is due to Lemma 3; the second part of the inequality is due to the sublinear regret (36).
The rest is identical with [48, B.4], although we proceed the proof to make the paper self-contained. The term (f) can be further bounded by noting Emm cxp(-ke/2) s 2(1 + //2 /3g... as
2(1 + 8)88/2
(i) 51
1 c/2)
(46)
8ga+
k=m
where the third term is finite if and only rrt is sublinear.
The remaining term, (ff), can be bounded using concentration incquality, in particular Chebyshev- Nemirovski inequality [48, Lemma B.
2a Vo.
(tt) < 1 +
(47)
/ la 1)*
26

---

where we denote the tth central moment Vo,, El| log(1 Xe, - aa) - 1s,,1" R S al.
which is fimite as per the assumption A2 made in Theorem 3.
Putting everything together, we get the upper bound for any / E (0, 1) as
4+ (1 + 3) log(1/6)
2(1 + 8)88/2
min (0) 44
98.*
Bga,*
Vo,
(1+
+ B 1)
+ log(1/8)
+
1 1 2 c/2).
(48)
98.*
Rm
make the factor (1 (++8) B) arbitrarily close to in a manner similar to [48], which concludes the proof of Theorem 3. 0
In the limit 8 - 0 and when considering the ratio min (6) log(1/6),
27


Reviews:
[{"review_id": "rk6gUxBVek", "reviewer": "Reviewer_3kvC", "summary": "This paper introduces R-AutoEval+, a novel framework for evaluating AI models, particularly LLMs. It addresses the challenge of balancing between costly real-world data evaluations and potentially biased automated evaluations. R-AutoEval+ provides finite-sample reliability guarantees while ensuring improved or at least equivalent sample efficiency compared to conventional methods. The key innovation is an adaptive construction of the model evaluation variable that dynamically adjusts its reliance on synthetic data based on the autoevaluator's accuracy. The framework leverages game-theoretic testing-by-betting and an enhanced version of prediction-powered inference (PPI++). The authors demonstrate the reliability and efficiency of R-AutoEval+ through experiments on LLM weight quantization optimization and prompt design.", "questions": "- What's the computational overhead of the algorithm compared to other methods?\n- How sensitive is the performance on the selection of hyperparameters? \n\nThe paper goes into mathematical details quite quickly. It would help if a more high-level problem statement was given in the introduction, before going more into depth.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:  \n- Provides finite-sample reliability guarantees\n- Adaptively balances the use of synthetic and real-world data, potentially improving sample efficiency\n- Offers provable sample efficiency improvements over existing methods\n- Combines multiple advanced techniques (game-theoretic testing-by-betting, PPI++)\n- Demonstrates practical applicability through experiments on relevant LLM tasks\n- Addresses a crucial problem in AI model evaluation and selection\n\nWeaknesses:  \n- Uses a fixed, discrete set of candidate factors for determining reliance on synthetic data, potentially limiting flexibility\n- Sample efficiency guarantee only holds for sufficiently high target reliability levels\n- Limited experimental validation (only tested on LLM quantization and prompt design)", "was_revised": true, "final_justification": "The authors response has clarified the main points and provided the requested evidence. I maintain my score of 4.", "rebuttal": "Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments. \n\n1) _Limited experimental validation (only tested on LLM quantization and prompt design)_ \n\n> During the rebuttal, we have carried out additional experiments on more challenging LLM tasks. In light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 number of tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set \\$n=1000, N=4000, \\delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework. \n\n| autoevaluator spec (model name) | autoevaluator spec (model size) | autoevalator spec (acc) | test-time computation gain of R-AutoEval+ over R-Eval (num. tokens)  | test-time computation gain of R-AutoEval+ over R-AutoEval (num. tokens) |\n| :---: | :---: | :---: | :---: | :---: |\n| BitNet b1.58 | 0.85 B | 0.35 | 40.87 | 7.80 |\n| Llama-3.2-3B-Instruct  | 3.21 B | 0.66 | 90.48 | 7.72 |\n| Qwen3-32B | 32.8 B | 0.82 | 42.14 | 66.25 |\n| DeepSeek-R1-Distill-Qwen-32B | 32.8 B | 0.89 | 117.12 | 27.17 |\n| GPT-4.1 | not available | 0.9258 | 127.21 | 27.86 |\n\n**Table 1: LLM test-time scaling experiment for math problem (GSM8K)**\n\n> In the following, we will use the new setting to address the reviewer’s comments. \n\n2) _Uses a fixed, discrete set [...], potentially limiting flexibility_\n\n> This is indeed a limitation of the scheme as we briefly described in Sec. 5 (Conclusion and Further Discussions). To further explore the impact of the selection of candidate factors, we have carried out an experiment in which we vary the number of candidates, S, which are equally spaced within the interval [0,1]. In what follows, we will adopt GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators. \n\n|  | R-Eval | R-AutoEval | R-AutoEval+(S=2) | R-AutoEval+(S=5) | R-AutoEval+(S=10) | R-AutoEval+(S=20) | R-AutoEval+(S=50) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| test-time computation in tokens (autoeval = GPT4.1) | 983.34 | 883.99 | 871.67 | 854.99 | 856.13 | 856.13 | 856.13 |\n| test-time computation  in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 964.41 | 961.94 | 961.94 | 961.94 | 961.94 |\n\n**Table 2: Impact of the number of candidate factors (\\$S$)**\n\n> As can be observed in the table above, as long as the candidate factors includes \\$\\rho=0$ (R-Eval) and \\$\\rho=1$ (R-AutoEval), i.e., as long as \\$S \\geq 2$, R-AutoEval+ consistently outperforms both R-Eval and R-AutoEval, with its gain being maximized at $S=5$, and saturates after $S=10$. \n\n> Next, we investigate the impact of the values of each candidate factors by going beyond the equally spaced grid. To this end, we consider Dirichlet distribution with concentration parameter 0.1, 1, 10 to allocate the $S-2$ values within the [0,1] interval (we keep the first and the last factors to be 0 and 1 to ensure the efficiency guarantee as per our Theorem 3). The cumulative probabilities of the Dirichlet distribution are used as the candidate factors. An increased concentration parameter makes samples from the Dirichlet distribution closer to the uniform distribution, and hence the grid becomes closer to the equally spaced grid. The table below shows that equal grid performs the best.\n\n|  | R-Eval | R-AutoEval | R-AutoEval+ (equal grid) | R-AutoEval+  (Dir(10,10,...,10)) | R-AutoEval+  (Dir(1,1,...,1)) | R-AutoEval+  (Dir(0.1,0.1,...,0.1)) |\n| --- | --- | --- | --- | --- | --- | --- |\n|  test-time computation in tokens (autoeval = GPT4.1)  | 983.34 | 883.99 | 856.13 | 856.13 | 860.91 | 873.27 |\n|  test-time computation in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 961.94 | 964.22 | 960.76 | 966.72 |\n\n**Table 3: Impact of the value of candidate factors (\\$\\\\{\\rho_s\\\\}_{s=1}^S$)**\n\n> The last remaining degree of freedom is given by the choice of the initial weights associated with each candidate factor. To this end, we go beyond the uniform distribution and consider again the Dirichlet distribution to allocate the weights for the \\$S$ candidate factors. The table below confirms again the preference for equal weighting. \n\n|  | R-Eval | R-AutoEval | R-AutoEval+ (equal weight) | R-AutoEval+  (Dir(10,10,...,10)) | R-AutoEval+  (Dir(1,1,...,1)) | R-AutoEval+  (Dir(0.1,0.1,...,0.1)) |\n| --- | --- | --- | --- | --- | --- | --- |\n|  test-time computation in tokens (autoeval = GPT4.1) | 983.34 | 883.99 | 856.13 | 856.13 | 856.13 | 856.68 |\n|  test-time computation in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 961.94 | 959.21 | 962.41 | 969.50 |\n\n**Table 4: Impact of the initial weights associated with the candidate factors (\\$\\\\{w_{s,0}\\\\}_{s=1}^S$)**\n\n3) _Sample efficiency [...] high target reliability levels_\n\n> This is indeed a limitation of the study, as described in Sec. 5 (Conclusion and Further Discussions). That said, in practice, autoeval methods may be particularly relevant for safety-critical risk-averse scenarios [Giambona et al., 2018], in which high reliability levels are a requirement. Furthermore, we have observed empirically that, even under moderately low reliability levels \\$1-\\delta$ (with larger \\$\\delta$ indicating a less strict reliability requirement), R-AutoEval+ consistently outperforms R-Eval and R-AutoEval. For the ongoing example, this is shown in the following table. \n\n|  | \\$\\delta = 0.05$ | \\$\\delta = 0.1$  | \\$\\delta = 0.15$  | \\$\\delta = 0.2$  | \\$\\delta = 0.25$  | \\$\\delta = 0.3$  |\n| --- | --- | --- | --- | --- | --- | --- |\n| test-time computation gain over R-Eval | 133.72 | 127.21 | 110.34 | 102.88 | 99.29 | 87.65 |\n| test-time computation gain over R-AutoEval | 34.20 | 27.86 | 14.41 | 4.40 | 1.24 | 1.47 |\n\n**Table 5: Efficiency gain for a range of target reliability levels \\$\\delta$**\n\n4) _What's the computational overhead of the algorithm compared to other methods?_\n\n> Given both the human-labeled (size \\$n$) and autoevaluated data (size \\$N$), mean-based approaches (Eval and AutoEval, which do not come with reliability guarantees) require \\$O(n)$ (for Eval) and \\$O(N)$ (for AutoEval) operations for computing the respective average. The reliable approaches (R-Eval and R-AutoEval) need additional computation steps for updating the e-values as well as the betting strategies (Sec. 2). While updating the e-values at each round requires a single multiplication, the betting strategy may be more complex. For example, especially the universal portfolio (UP) strategy with sublinear regret (Assumption A1) has a complexity of order \\$O(nG)$ where $G$ is the size of the grid used to approximate the integral operation of UP (Sec. C.1), although this can be decreased by adopting sampling-based approaches [Kalai and Vempala, 2002]. Lastly, R-AutoEval+ requires $S$ times more computation than R-AutoEval due to its consideration of $S$ candidate factors. It is worth emphasizing that, the costs described above are generally negligible with respect to running the autoevaluator (e.g., an LLM judge). The summary of the computation complexities can be found in the below table. \n\n|  | Eval | AutoEval | R-Eval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- | --- | --- |\n| computation complexity | \\$O(n)$ | \\$O(N)$ | \\$O(n + nG)$ | \\$O(n r + nG)$ | \\$O(S n r + SnG)$ |\n\n**Table 6: Computational complexities of the algorithms considered in this work**\n\n5) _How sensitive [...] hyperparameters?_\n\n> The hyperparameters in the proposed R-AutoEval+ encompass the number of candidate factors, the values of the candidate factors,  and the weights associated with the candidate factors, whose sensitivity has been analyzed in the reply  above.\n\n6) _The paper goes into mathematical details quite [...] depth._\n\n>  We have tried to provide a high-level discussion through the introduction section and Fig. 1, but the page limitations at the time of submission made it difficult to provide further general descriptions. We will certainly add further discussions on the high-level problem statement using the additional content page in case the work is accepted. \n\n#### **References**\n#### [Cobbe et al., 2021] Karl Cobbe, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n\n#### [Muennighoff et al., 2025] Niklas Muennighoff,et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.\n\n#### [Snell et al., 2024] Charlie Snell, et al. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.\n\n#### [Yang et al., 2025] An Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.\n\n#### [Giambona et al., 2018] Erasmo Giambona, et al. The theory and practice of corporate risk management: Evidence from the field. Financial Management, 47(4):783–832, 2018.\n\n#### [Kalai and Vempala, 2002] Adam Kalai and Santosh Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research, 3(Nov):423–440, 2002."}, {"review_id": "LlDsbWoOOF", "reviewer": "Reviewer_nziA", "summary": "This manuscript introduces \"R-AutoEval+\", an enhanced prediction-powered AutoEval method that builds upon previous R-AutoEval and Prediction-Powered Inference (PPI) work. R-AutoEval+ incorporates a dynamic parameter on top of PPI, which adjusts the weighting between human and automated evaluations. This adjustment is achieved through a \"testing-by-betting\" strategy, where weights are modified based on observations. This adaptation addresses R-AutoEval's limitations when the AutoRater's accuracy is insufficient.\n\nThe manuscript illustrates the method's application with two real-data examples:\n(1) Selecting the smallest quantized LLMs with a guaranteed performance drop of less than 10%.\n(2) Selecting the shortest prompt template with a guaranteed execution error of less than 0.5. In both cases, R-AutoEval+ demonstrated superior performance compared to R-Eval (pure human evaluation) and R-AutoEval (PPI-based AutoEval).", "questions": "- Since the weights are learned dynamically by the observations up to $i$, does the order of the data to the algorithm affect the results? How robust is the algorithm against the data ordering?\n\n- Regarding the real-data experiments, how was the unlabeled data constructed for TriviaQA? How similar does the synthetic data need to be with the human labeled data? How were the hyperparameters, such as $n$, $N$, $S$, initial weight, synthetic-to-real, etc., determined?\n\n- Similarly, for the instruction-induction task, how was the unlabeled data constructed? How many human evaluation and synthetic samples were used?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The manuscript is very well written. The methodology and results are clearly presented. Even though the content is quite dense and deep, the authors were able to present it well with preliminaries, making it self-contained and easier for the reader to understand.\n\n- The two real-data examples are cleverly chosen to demonstrate the method's use cases.\n\n- Solid mathematical justification.\n\nWeakness:\n- The details of real data experiments could have been presented more thoroughly.", "was_revised": true, "final_justification": "With the new results, I have a better understanding of the paper. I believe this is a solid work with good impact. I increases my confidence from 3 to 4. \n\nPlease add the new results and the relevant details to the revised paper. Thank you.", "rebuttal": "Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.\n\n1. _The details of real data experiments could have been presented more thoroughly._\n\n> If this work is accepted, we will make sure to add all the relevant details for the real-data experiments, including the method used for constructing the unlabeled data set and the measure of similarity between the human labeled data and synthetic data. \n\n2. _Since the weights are learned dynamically by the observations up to , does the order of the data to the algorithm affect the results? How robust is the algorithm against the data ordering?_\n\n> Yes, the order of the data can affect the final result. To clarify this point, we have carried out additional experiments. Specifically, in light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set \\$n=1000, N=4000, \\delta=0.1$, and report the averaged output from 100 independent runs with different calibration-test data splits. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework. \n \n\n| autoevaluator spec (model name) | autoevaluator spec (model size) | autoevalator spec (acc) | test-time computation gain of R-AutoEval+ over R-Eval (num. tokens)  | test-time computation gain of R-AutoEval+ over R-AutoEval (num. tokens) |\n| :---: | :---: | :---: | :---: | :---: |\n| BitNet b1.58 | 0.85 B | 0.35 | 40.87 | 7.80 |\n| Llama-3.2-3B-Instruct  | 3.21 B | 0.66 | 90.48 | 7.72 |\n| DeepSeek-R1-Distill-Qwen-7B | 7.62 B | 0.54 | 56.49 | 14.81 |\n DeepSeek-R1-0528-Qwen3-8B | 8.19 B | 0.32 | 21.40 | 24.59 |\n| Qwen3-32B | 32.8 B | 0.82 | 42.14 | 66.25 |\n| DeepSeek-R1-Distill-Qwen-32B | 32.8 B | 0.89 | 117.12 | 27.17 |\n| GPT-4.1 | not available | 0.9258 | 127.21 | 27.86 |\n\n**Table 1: LLM test-time scaling experiment for math problem (GSM8K)**\n\n>  To focus on the robustness of the algorithm to the ordering of the data, we fix the calibration-test data split and only change the ordering of data to compute the normalized deviation (std / mean) of  the test-time computation required by each algorithm. We will consider GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators for the experimental results henceforth. As can be seen from the table below, R-AutoEval and R-AutoEval+ are more robust to the data ordering than R-Eval, possibly attributed to its reduced variability of averaged autoevaluated data (at each round we average \\$r=N/n$ autoevaluated data) while R-AutoEval+ tends to be slightly less robust than R-AutoEval for its adaptive nature that incorporates the data ordering. \n\n|  | R-Eval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- |\n| normalized deviation (std/mean) with autoeval = GPT4.1 | 8.62% | 3.90%| 6.88% |\n| normalized deviation (std/mean) with autoeval = DeepSeek-R1-0528-Qwen3-8B | 8.62% | 6.32%| 6.00% |\n\n**Table 2: Sensitivity of algorithms with respect to data ordering**\n\n3. _Regarding the real-data experiments, how was the unlabeled data constructed for TriviaQA? How similar does the synthetic data need to be with the human labeled data? How were the hyperparameters, such as $n, N, S$, initial weight, synthetic-to-real, etc., determined?_\n\n> As TriviaQA does not explicitly contain unlabeled data, we have split the data set and ignored the presence of label information in one split to construct the unlabeled data set. The synthetic data indeed need to be similar enough with the human labeled data to ensure improved efficiency of R-AutoEval+. To clarify this similarity, we have measured the accuracy of the autoevaluator in the first table of this reply, which essentially captures the similarity of autoevaluated data with human labeled data.  The hyperparameter \\$n$ was searched within the interval 100 to 300 as shown in Fig. 4(a), while we have chosen the hyperparameter $N$ based on the amount available unlabeled data (from the above construction).  In our experiments, we have chosen the hyperparameters associated with the candidate factors by setting $S=10$ with equal grid and equal weighting, since such setting gave us a good result in toy experiment. Nonetheless, we have further carried out careful experimental analysis on the impact of hyperparameter choice during the rebuttal.  \n\n> We first investigate the impact on the number of candidates, S, with candidate factors equally spaced within the interval [0,1]; and with equal initial weights.  In what follows, we will adopt GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators. \n\n|  | R-Eval | R-AutoEval | R-AutoEval+(S=2) | R-AutoEval+(S=5) | R-AutoEval+(S=10) | R-AutoEval+(S=20) | R-AutoEval+(S=50) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| test-time computation in tokens (autoeval = GPT4.1) | 983.34 | 883.99 | 871.67 | 854.99 | 856.13 | 856.13 | 856.13 |\n| test-time computation  in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 964.41 | 961.94 | 961.94 | 961.94 | 961.94 |\n\n**Table 2: Impact of the number of candidate factors (\\$S$)**\n\n> As can be observed in the table above, as long as the candidate factors includes \\$\\rho=0$ (R-Eval) and \\$\\rho=1$ (R-AutoEval), i.e., as long as \\$S \\geq 2$, R-AutoEval+ consistently outperforms both R-Eval and R-AutoEval, with its gain being maximized at $S=5$, and saturates after $S=10$. \n\n> Next, we investigate the impact of the values of each candidate factors by going beyond the equally spaced grid. To this end, we consider Dirichlet distribution with concentration parameter 0.1, 1, 10 to allocate the $S-2$ values within the [0,1] interval (we keep the first and the last factors to be 0 and 1 to ensure the efficiency guarantee as per our Theorem 3). The cumulative probabilities of the Dirichlet distribution are used as the candidate factors. An increased concentration parameter makes samples from the Dirichlet distribution closer to the uniform distribution, and hence the grid becomes closer to the equally spaced grid. The table below shows that equal grid performs the best.\n\n|  | R-Eval | R-AutoEval | R-AutoEval+ (equal grid) | R-AutoEval+  (Dir(10,10,...,10)) | R-AutoEval+  (Dir(1,1,...,1)) | R-AutoEval+  (Dir(0.1,0.1,...,0.1)) |\n| --- | --- | --- | --- | --- | --- | --- |\n|  test-time computation in tokens (autoeval = GPT4.1)  | 983.34 | 883.99 | 856.13 | 856.13 | 860.91 | 873.27 |\n|  test-time computation in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 961.94 | 964.22 | 960.76 | 966.72 |\n\n**Table 3: Impact of the value of candidate factors (\\$\\\\{\\rho_s\\\\}_{s=1}^S$)**\n\n> The last remaining degree of freedom is given by the choice of the weights associated with each candidate factor. To this end, we go beyond the uniform distribution and consider again the Dirichlet distribution to allocate the weights for the \\$S$ candidate factors. The table below confirms again the preference for equal weighting. \n\n|  | R-Eval | R-AutoEval | R-AutoEval+ (equal weight) | R-AutoEval+  (Dir(10,10,...,10)) | R-AutoEval+  (Dir(1,1,...,1)) | R-AutoEval+  (Dir(0.1,0.1,...,0.1)) |\n| --- | --- | --- | --- | --- | --- | --- |\n|  test-time computation in tokens (autoeval = GPT4.1) | 983.34 | 883.99 | 856.13 | 856.13 | 856.13 | 856.68 |\n|  test-time computation in tokens (autoeval = DeepSeek-R1-0528-Qwen3-8B) | 983.34 | 986.53 | 961.94 | 959.21 | 962.41 | 969.50 |\n\n**Table 4: Impact of the weights associated with the candidate factors (\\$\\\\{w_{s,0}\\\\}_{s=1}^S$)**\n\n\n4. _Similarly, for the instruction-induction task, how was the unlabeled data constructed? How many human evaluation and synthetic samples were used?_ \n\n> We followed the same approach as TriviaQA, and we have considered 200 human evaluation and 1800 synthetic samples. \n\n#### **References**\n#### [Cobbe et al., 2021] Karl Cobbe, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.\n#### [Muennighoff et al., 2025] Niklas Muennighoff,et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.\n#### [Snell et al., 2024] Charlie Snell, et al. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024."}, {"review_id": "KFZEDPbNoU", "reviewer": "Reviewer_UVVV", "summary": "The authors provide a framework for automatic evaluation for AI models, where selecting from multiple candidate responses is necessary. They introduce a framework that provides guarantees on the evaluation performance, with enhanced sample efficiency over conventional methods. Empirically the show the reliability and efficiency of their framework.", "questions": "1. Have you done a study of your approach over model size as the reliability of LLM as judge may change with respect to model parameters?\n2. Have you looked at scaling properties over risk with respect to the amount of human and synthetic data that is used?\n3. Would it be possible to evaluate this framwork in a larger scale setting as the ones decribed above?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Theoretically motivated framework with better sample complexity than conventional approaches\n- The guarentee provided by this approach is not asymptotic unlike prior work, yielding narrower confidence intervals\n- Adaptive weighting of synthetic and real data based on the confidence of the autoevaluator is a novel contribution and leads to better empirical performance\n\nWeaknesses\n- Empirically validate their approach on toyish LLM tasks: Quantization and Prompt Shortening tasks. Could be useful to perform a broader evaluation on evaluation of an LLM on a broad range of topics (e.g math, coding, general QA)\n- Evaluation limited to a single family of models and judge/autoevaluator (Llama family of models). Could benefit from a broader evaluation over different model families and sizes to see the effect of performance of the strength of the autoevaluator.", "was_revised": true, "final_justification": "I do provide a score of 4 for this evaluation but due to this paper being out of domain from my expertise, I will keep my score conservative towards acceptance. The authors satisfied my empirical concerns in the rebuttal.", "rebuttal": "Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.\n\n1) _Empirically validate their approach on toyish LLM tasks: Quantization and Prompt Shortening tasks. Could be useful to perform a broader evaluation on evaluation of an LLM on a broad range of topics (e.g math, coding, general QA)_\n\n> During the rebuttal, we have carried out additional experiments on more challenging LLM tasks. In light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set \\$n=1000, N=4000, \\delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework. \n\n| autoevaluator spec (model name) | autoevaluator spec (model size) | autoevalator spec (acc) | test-time computation gain of R-AutoEval+ over R-Eval (num. tokens)  | test-time computation gain of R-AutoEval+ over R-AutoEval (num. tokens) |\n| :---: | :---: | :---: | :---: | :---: |\n| BitNet b1.58 | 0.85 B | 0.35 | 40.87 | 7.80 |\n| GPT-4.1 | not available | 0.9258 | 127.21 | 27.86 |\n\n**Table 1: LLM test-time scaling experiment for math problem (GSM8K)**\n\n2) _Evaluation limited to a single family of models and judge/autoevaluator (Llama family of models). Could benefit from a broader evaluation over different model families and sizes to see the effect of performance of the strength of the autoevaluator._\n\n> To address the reviewer’s point, we have considered autoevaluators with different sizes, accuracies on GSM8K data set, whose results and the corresponding test-time computation that can be saved by R-AutoEval+ as compared to R-Eval and R-AutoEval are shown in the table below. \n\n| autoevaluator spec (model name) | autoevaluator spec (model size) | autoevalator spec (acc) | test-time computation gain of R-AutoEval+ over R-Eval (num. tokens)  | test-time computation gain of R-AutoEval+ over R-AutoEval (num. tokens) |\n| :---: | :---: | :---: | :---: | :---: |\n| BitNet b1.58 | 0.85 B | 0.35 | 40.87 | 7.80 |\n| Llama-3.2-1B-Instruct  | 1.24 B | 0.26 | 28.71 | 10.28 |\n| DeepSeek-R1-Distill-Qwen-1.5B | 1.78 B | 0.33 | 47.76 | 8.4 |\n| Llama-3.2-3B-Instruct  | 3.21 B | 0.66 | 90.48 | 7.72 |\n| DeepSeek-R1-Distill-Qwen-7B | 7.62 B | 0.54 | 56.49 | 14.81 |\n| DeepSeek-R1-Distill-Llama-8B | 8.03 B | 0.67 | 82.52 | 15.52 |\n| DeepSeek-R1-0528-Qwen3-8B | 8.19 B | 0.32 | 21.40 | 24.59 |\n| Qwen3-32B | 32.8 B | 0.82 | 42.14 | 66.25 |\n| DeepSeek-R1-Distill-Qwen-32B | 32.8 B | 0.89 | 117.12 | 27.17 |\n| Yi-1.5-34B-Chat | 34.4 B | 0.57 | 74.05 | 5.95 |\n| Llama-3.3-70B-Instruct | 70.6 B | 0.89 | 136.29 | 7.37 |\n| DeepSeek-R1-Distill-Llama-70B | 70.6 B | 0.88 | 125.32 | 5.70 |\n| GPT-4.1 nano | unknown | 0.90 | 118.06 | 14.17 |\n| GPT-4.1 | unknown | 0.93 | 127.21 | 27.86 |\n\n**Table 2: Broad choice of the autoevaluators**\n\n> It is generally observed that choosing the autoevaluator from the same family of the model substantially reduces the gain of autoeval-based approaches. For instance, Llama-3.2-3B-Instruct autoevaluator achieves much lower accuracy – 0.16 – than Qwen3-32B autoevaluator but it can save much more reasoning budget --  48.34 tokens. This demonstrates the importance of the model choice as the reviewer correctly anticipated. We believe that such behavior is related to the positive feedback of LLM judges within the same family [Zheng et al., 2023] also known as preference leakage [Li et al., 2025], which makes the bias correction (8) a more challenging task. We will summarize this behavior in the updated draft if this work gets accepted.\n\n3) _Have you done a study of your approach over model size as the reliability of LLM as judge may change with respect to model parameters?_\n\n> In Fig. 1, Fig. 4(a), Fig. 6, and Fig. 9, we have considered three different LLM as judges by adjusting their size from 70.6 B to 26.7 B and 17.9 B. Furthermore, as described in the reply  above, in the new experiment on test-time scaling for GSM8K data set, we have investigated the crucial impact of the choice of LLM as judges. \n\n4) _Have you looked at scaling properties over risk with respect to the amount of human and synthetic data that is used?_\n\n> In Fig. 4(a) and Fig. 7(a), we have varied the amount of human data from 100 to 300 ($n: 100 \\rightarrow 120 \\rightarrow \\cdots \\rightarrow  300$) and observed its scaling trend. To further understand the scaling trend with the synthetic data size, we have carried out additional experiments that vary the size of synthetic data from 1000 to 6000 (\\$N: 1000 \\rightarrow 2000 \\rightarrow \\cdots \\rightarrow 6000$) for a fixed amount of human data for the new math experiment. We adopt GPT-4.1 as the autoevaluator and the results are summarized in the following table. It is observed that all the schemes ensure the target risk (accuracy gain no less than 3%), and that autoeval-based approaches can increasingly save the reasoning budget with increased number of synthetic data. Interestingly, R-AutoEval+ shows a more robust behavior than R-AutoEval, demonstrating again the power of the proposed adaptive design.  \n\n|  | R-Eval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- |\n| test-time computation (num. tokens) | 983.34 | 995.84→917.33→899.53→883.99→867.36→858.94 | 889.53→866.89→865.96→856.13→852.47→849.31 |\n| accuracy gain | 7.00% | 6.98% →6.55% →6.43%→6.25%→6.16%→6.05% | 6.38% →6.22%→6.19%→6.11%→6.03%→6.03 % |\n\n**Table 3: Impact of the number of synthetic data \\$N: 1000 \\rightarrow 2000 \\rightarrow3000 \\rightarrow4000\\rightarrow5000\\rightarrow6000$**\n\n5) _Would it be possible to evaluate this framework in a larger scale setting as the ones described above?_\n\n> In the new experiments on the GSM8K data set, we have considered a wide range of autoevaluators from model size from 0.85 B to 70.6 B, as well as for the GPT4.1 models which are generally assumed to be in a much larger scale setting. The results in Table 2 confirm the general applicability of the proposed scheme. \n\n#### **References**\n#### [Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.\n#### [Li et al., 2025] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.01534, 2025."}, {"review_id": "7iHcqHKa8G", "reviewer": "Reviewer_dCNW", "summary": "The paper introduces a new method called R-AutoEval+ that uses no more samples than the previous autoeval methods and has reliability guarantees. It evaluates a method or model using both human labeled data and auto evaluators (e.g. LLMs-as-judges). The coefficient $\\rho$ determining reliance on the auto evaluator is iteratively computed, with one iteration involving one human labeled and many autoeval labeled samples. The theorems have learning theory flavour, mostly focusing on sample efficiency bounds under which the method is reliable i.e. doesn’t overestimate the risk with probability at least $1-\\delta$. The main methods used are testing by betting and e-values and PPI++.", "questions": "1. In practice how do you deal with the probably correct result i.e. giving the correct answer with probability greater than $1-\\delta$? What happens if the prediction is wrong?\n2. Have authors compared the introduced method with some other evaluation methods (apart from R-Eval, R-AutoEval and others discussed in the paper)?\n3. Line 168, should it be R-Eval instead of AutoEval?", "limitations": "Doesn’t discuss what happens in practice when the method returns an incorrect prediction (with probability at most $\\delta$).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThe paper very nicely and gradually introduces the prior methods, which makes it so much easier to follow. The notation is clear and examples were given. The theorems are clearly stated. The method is tested on two different real-world evaluation problems, evaluating the best (smallest model) “good enough” quantization, and the shortest reliable prompt search.\n\nWeaknesses:\nNot sure if significance is well justified. For example, in Figure 9a), it seems like R-AutoEval and R-Eval are very similar. It is hard to know whether this will be a widely used evaluation method in the future.\n\nSeems like the authors renamed the method in [17] to R-AutoEval without mentioning the original name.", "was_revised": true, "final_justification": "Authors addressed most of my follow-up questions, so I stuck with my original score of 4.", "rebuttal": "Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.\n\n1. _Not sure if significance is well justified. For example, in Figure 9a), it seems like R-AutoEval and R-Eval are very similar. It is hard to know whether this will be a widely used evaluation method in the future._\n\n> The fact that in some cases using auto evaluator (e.g., LLMs-as-judges) may bring no gain as compared to a vanilla approach is a fundamental limitation of the autoevaluation framework. When the autoevaluator is of insufficient quality and/or the problem at hand is easy enough to be sufficiently analyzed by the human-labeled data, existing approaches such as [17] suffer from performance degradation by utilizing autoevaluators. This is exactly the behavior observed in Figure 9a, where R-AutoEval [17] (blue line) shows worse performance than R-Eval (green). \n\n> However, the benefits are evident when the autoevaluator is sufficiently accurate. For instance, the results for TriviaQA data set (Fig. 4(a), Fig. 7(a)) show that unless the autoevaluator is heavily quantized (i.e., BF16 and MX6), autoevaluator can help finding smaller quantized models. Furthermore,  the results on Instruction-Induction task results in Fig. 4(b), Fig. 7(b) show that unless autoevaluator is operated on low quality prompts (few number of in-context samples), autoevaluator can help finding shorter prompts for the base models. \n\n\n> To further validate the significance of the proposed method, we have carried out additional experiments on more challenging LLM tasks that solves math problems. Specifically, in light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set \\$n=1000, N=4000, \\delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework. \n\n| autoevaluator spec (model name) | autoevaluator spec (model size) | autoevalator spec (acc) | test-time computation gain of R-AutoEval+ over R-Eval (num. tokens)  | test-time computation gain of R-AutoEval+ over R-AutoEval (num. tokens) |\n| :---: | :---: | :---: | :---: | :---: |\n| BitNet b1.58 | 0.85 B | 0.35 | 40.87 | 7.80 |\n| Llama-3.2-3B-Instruct  | 3.21 B | 0.66 | 90.48 | 7.72 |\n| Qwen3-32B | 32.8 B | 0.82 | 42.14 | 66.25 |\n| DeepSeek-R1-Distill-Qwen-32B | 32.8 B | 0.89 | 117.12 | 27.17 |\n| GPT-4.1 | not available | 0.9258 | 127.21 | 27.86 |\n\n**Table 1: LLM test-time scaling experiment for math problem (GSM8K)**\n\n\n2. _Seems like the authors renamed the method in [17] to R-AutoEval without mentioning the original name._\n\n>  Thank you for pointing this out. The original method in [17] was dubbed “SS-RCPS”, and we have introduced a new name for the consistency. We will revise the manuscript by mentioning the original name when we first introduce [17].\n\n3. _In practice how do you deal with the probably correct result i.e. giving the correct answer with probability greater than \\$1-\\delta$? What happens if the prediction is wrong?_\n\n> Wrong predictions are inevitable due to the randomness associated with the realization of the calibration data (see standard PAC-type guarantees). That said,   even in such worst-case settings, R-AutoEval+ can still yield close-to-safe decisions with a minimal violation rate. This can be seen from Fig. 1, where we have reported the worst case within 1.5 interquartile range (IQR) range. Furthermore we have provided the full box plots in Fig. 6 (Appendix D). There, we can observe the attained worst-case performance. The tables below (from high-quality autoevaluator to low-quality autoevaluator) confirm that R-AutoEval and R-AutoEval achieves worst risk lower than 0.125 which corresponds to \\$1.25\\alpha$ unlike R-Eval that suffers from worst risk near 0.14 (\\$1.4\\alpha$). Furthermore, the actual violation ratios of R-AutoEval and R-AutoEval+ are lower than R-Eval up to 1% (while all of them being much lower than the target \\$\\delta=10$%), demonstrating that autoevaluation can provide further benefit in terms of managing such extreme cases. \n\n|  | Eval | R-Eval | AutoEval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- | --- | --- |\n| violation rate | 51.6% | 2.6% |7.4% | 0.2% |1.4% |\n| worst risk | 0.170 | 0.143 | 0.118 | 0.112 | 0.118 |\n\n**Table 2: Worst-case analysis for LLM quantization, autoeval=Llama3.3-BF16 (Fig. 6(a))**\n\n|  | Eval | R-Eval | AutoEval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- | --- | --- |\n| violation rate | 51.6% | 2.6% |14.6% | 0.4% |1% |\n| worst risk | 0.170 | 0.143 | 0.120 | 0.106 | 0.112 |\n\n**Table 3: Worst-case analysis for LLM quantization, autoeval=Llama3.3-MX6 (Fig. 6(a))**\n\n|  | Eval | R-Eval | AutoEval | R-AutoEval | R-AutoEval+ |\n| --- | --- | --- | --- | --- | --- |\n| violation rate | 51.6% | 2.6% |100% | 0.8% |1.6% |\n| worst risk | 0.170 | 0.143 | 0.722 | 0.115 | 0.125 |\n\n**Table 4: Worst-case analysis for LLM quantization, autoeval=Llama3.3-MX4 (Fig. 6(a))**\n\n4. _Have authors compared the introduced method with some other evaluation methods (apart from R-Eval, R-AutoEval and others discussed in the paper)?_\n\n> In the manuscript, we have focused on comparing our methods with Eval and AutoEval (mean-based approach, no reliability guarantees) and R-Eval and R-AutoEval (non-asymptotic reliability guarantees), since they are the state-of-the-art evaluation methods offering statistical validity. For reference, we have now also considered a method that only provides asymptotic reliability guarantees [Boyeau et al., 2024, Fisch et al., 2024] based on the central limit theorem. For the GSM8K data set, we have observed that its violation ratio is 0.12, which is higher than the target level 0.1.  If the work gets accepted, we will further compare our scheme with other  asymptotic benchmarks.\n\n5. _Line 168, should it be R-Eval instead of AutoEval?_ \n\n> You are correct! We will fix the typo, thank you.\n\n#### **References**\n#### [Boyeau et al., 2024] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008, 2024.\n#### [Fisch et al., 2024] Adam Fisch, Joshua Maynez, R Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powered inference for effective hybrid evaluation of language models. Advances in Neural Information Processing Systems, 37:111489–111514, 2024."}]

Author rebuttal:

Meta-review: 1. **Summary**
   The paper introduces R-AutoEval+, a new framework for autoevaluation that adaptively combines real human-labeled data and synthetic evaluations from LLMs-as-judges. The central contribution is a method that ensures finite-sample reliability guarantees while providing sample efficiency improvements (or at least never worse performance) compared to both R-Eval and R-AutoEval. The method builds on testing-by-betting with e-values and an extension of prediction-powered inference (PPI++), adaptively weighting reliance on autoevaluators. Empirical demonstrations include (i) selecting quantized LLMs under accuracy guarantees, (ii) prompt selection with execution risk guarantees, and (iii) rebuttal-period experiments on test-time compute scaling for reasoning LLMs.

2. **Strengths**

   1. Strong theoretical foundation, with rigorous proofs of finite-sample guarantees and efficiency results.
   2. Novel adaptive design that seamlessly reverts to human-only evaluation when autoevaluators are poor, mitigating known risks of prior methods.
   3. Solid empirical validation across several tasks (LLM quantization, prompt design, test-time scaling), demonstrating both reliability and efficiency.
   4. Clear exposition of the methodological innovations (testing-by-betting, adaptive weighting, PPI++).
   5. Active engagement by the authors during rebuttal, with new experiments and ablations that addressed reviewer concerns.

3. **Weaknesses**

   1. Experimental coverage, while improved during rebuttal, remains narrower than ideal; the tasks are limited to LLM quantization, prompting, and one test-time scaling example. A broader set of application domains would better substantiate generality.
   2. Efficiency guarantees hold only under sufficiently high reliability levels. Although empirically the method still performs well at moderate levels, the theoretical coverage is narrower.
   3. Dependence on a fixed discrete set of candidate factors for weighting synthetic data may limit flexibility, though rebuttal experiments suggest performance is stable under different configurations.
   4. Computational overhead, while negligible compared to running LLM evaluators, is higher than for R-AutoEval due to the adaptive weighting.

4. **Decision Rationale**

   * The paper makes a clear and rigorous theoretical advance, addressing a critical limitation of prior autoevaluation frameworks (the trade-off between bias correction and efficiency).
   * The empirical evidence, although somewhat narrow in scope, convincingly demonstrates the utility of the method and its robustness across evaluators and tasks.
   * The rebuttal period was particularly constructive: the authors added experiments on test-time scaling, evaluated across multiple model families, and explored robustness to hyperparameter choices, addressing key reviewer concerns.
   * Despite some remaining limitations, the work is technically solid, timely, and relevant for the community given the centrality of evaluation in current LLM research and deserve wider dissemination.

   For these reasons, I recommend **acceptance at the spotlight level**. 

5. **Discussion and Reviewer Exchanges**

   * **Reviewer 3kvC**: Raised concerns about limited experiments, discrete factors, and high-level clarity. Authors responded with additional experiments on test-time scaling and ablation studies on factor discretization and weights. Reviewer maintained a borderline accept score.
   * **Reviewer nziA**: Initially positive, with questions on robustness to data ordering and experiment details. Authors clarified experimental construction, robustness properties, and provided new sensitivity results. Reviewer increased confidence and remained supportive of acceptance.
   * **Reviewer UVVV**: Expressed concerns about toyish experiments and limited evaluator families. Authors responded with extended experiments on multiple autoevaluators across families and scaling of synthetic data. Reviewer maintained a borderline accept with recommendation to accept.
   * **Reviewer dCNW**: Appreciated the clarity and theory but questioned significance and practical impact. Authors responded with additional significance experiments and clarifications. Reviewer maintained borderline accept.

   Overall, the discussion improved the paper significantly. While some reviewers remained conservative due to limited experimental scope, the theoretical contributions and new rebuttal results consolidated the consensus beyond acceptance.
Decision: Accept (spotlight)