Multiple-Prediction-Powered Inference
Authors: Charlie Cowen-Breen
Alekh Agarwal
Stephen Bates
William W. Cohen
Jacob Eisenstein
Amir Globerson
Adam Fisch
Venue: ICLR 2026 Poster (2026)
Keywords: Prediction-powered inference
Statistical estimation
Model evaluation
LLM as judge
PPI
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: This paper introduces MultiPPI, an optimal procedure for mean estimation by combining expensive, high-quality data with cheap, lower-quality proxies.

Abstract: Statistical estimation often involves tradeoffs between expensive, high-quality measurements and a variety of lower-quality proxies. We introduce Multiple-Prediction-Powered Inference (MultiPPI): a general framework for constructing statistically efficient estimates by optimally allocating resources across these diverse data sources. This work provides theoretical guarantees about the minimax optimality, finite-sample performance, and asymptotic normality of the MultiPPI estimator, and through experiments across three diverse large language model (LLM) evaluation scenarios, we show that MultiPPI consistently achieves lower estimation error than existing baselines. This advantage stems from its budget-adaptive allocation strategy, which strategically combines subsets of models by learning their complex cost and correlation structures.

Published as a conference paper at ICLR 2026
MULTPPL-PPEEDCTIOO-POOEEE INFERENCE
Charlie Cowen-Breen'
Alekh Agarwal
Stephen Bates'
William W. Cohen
Jacob Eisenstein
Amir Globerson
Adam Fisch
Massachusetts Institute of Technology. *Google Research, *Google DeepMind
Work done during internship at Google DeepMind. ecbreen eddu, fischégoogle, com.
ABSTRACT
Statistical estimation often involves tradeoff's between expensive, high-quality measurements and variety of lower-quality proxies. We introduce Multiple- Prediction-Poweree Inference (MultiPPI): general framework for constructing statistically efficient estimates by optimally allocating resources across these diverse data sources. This work provides theoretical guarantees about the minimax optimality, finite-sample performance, and asymptotic normality of the MultiPPI estimator. Through experiments across three diverse large language model (LLM) evaluation scenarios, we show that MultiPPI consistently achieves lower estimation error than existing baselines. This advantage stems from its budget-adaptive allocation strategy, which strategically combines subsets of models by learning their complex cost and correlation structures.
INTRODUCTION
Efficiently estimating expectations of random variables under a fixed budget a fundamental prob- lem in many scientific settings. This paper addresses the general problem of optimally estimating such expectations when each random variable has an associated sampling cost, subject budget constraint. We are specifically motivated by the challenge presented by Al model evaluation, which is a critical, but often resource-intensive, step model development and maintenance.
fixed total
More concretely, in the Al model evaluation setting, a variable X1 might represent a high-quality but expensive metric computed for every model response to an input query, such as score from human annotator powerful proprictary model used as "autorater". The remaining variables, X2....,Xu Xx. might represent cheaper evaluation options (e.g., scores from smaller autoraters or rase- systems), which be viewed as covariates proxies for the true score. Given the op- tion to obtain samples of XI,-..,X X (either jointly or independently), the primary objective is often to then estimate the mean of the high-quality score, E[X1]. In other cases, may be interested in the mean difference between two scores, say, E[X; X2). The core difficulty in each case is in determining which of these variables to query, how many times query them, and then finally how to combine them together to produce a statistically efficient, consistent estimate of the ground truth.
To formalize this, let X (X1,..., Xx) be of random variables with finite variance. We then consider the general problem of efficiently estimating any linear function of the mean of X subject a total observation budget B. That is, for some E R. we want to estimate 6* while spending
[ more than a total budget B on collecting subsets of joint random variables x, (Xt)er at cost er for index subsets 7 C (1....,k) More precisely, if n, is the number of times the subset X, is observed, we require that the n, satisfy a system of lincar budget constraints of the form Σ, cynj < B, where the sum over all such collected subsets
Estimating linear functions of [[ allows for flexibility in how 0* is defined. Given the Al evalua- tion setting above, for example, measuring E[X,] corresponds to = (1,0...., 0), while measuring E[X; Xz] corresponds to (1, - 11, O, .... 0). The flexibility observe subsets of X also intro- duces a key trade-off that unique with respect to previous related approaches to estimation. As we will show, observing variables jointly can be advantageous by reducing overall estimation variance.

---

Published as a conference paper at ICLR 2026
This benefit, however, must be weighed against the data acquisition costs, ez. We make assump- tions about the structure of these costs (e.g., they may be non-additive over the components f). For instance, in AI evaluation setting, obtaining predictions from multiple autoraters often be parallelized, so the cost of multiple predictions (in lateney) is not significantly more than that of the single slowest one. This is always true; in medical diagnostics, for example, ordering many tests may become too taxing for a patient, and therefore undesireable or impossible to do jointly
To solve this cost-optimal, multi-variate estimation problem, introduce the Kultiple-Predictinn Powered Inference (MultiPP)) estimator, which is a cost-aware generalization of the Efficient DeddictionPPwereed Inference (PPI++) estimator of Angelopoulos optimally leverage multiple types of predictions to power inference. The MultiPPI estimator constructs low-variance, consistent estimate of 0" by combining observations from judiciously chosen subsets of X. The core of our method is optimization procedure that jointly determines the number of samples n, to draw from each subset / and the corresponding linear weights X used to form the final estimate. We demonstrate that this allocation problem can be formulated as a second-order cone program (SOCP) for a single budget constraint, and a semidefinite program (SDP) for multiple budget constraints, and thus solved efficiently using standard techniques.
al. (2023b), and extends it to
Theoretically, we show that the MultiPPI estimator is minimax optimal when the joint covariance matrix, Σ Cov(X). is known. For the typical case where it is unknown, however, provide a framework for integrating an initial estimation phase where an approximation of the required covariance matrix, Σ, can be derived from either a small "burn-in" sample or pre-existing labeled "transfer" dataset common scenario in applied settings) and provide finite- sample bounds on the performance degradation that is incurred by substituting Σ for Σ. Finally, we empirically demon- strate the effectiveness of this approach across three diverse LLM evaluation settings, including choosing between autoraters of different sizes, autoraters with different test-time reasoning con- figurations, and complex multi-autorater-dbbat scenarios. In all cases, method achieves lower mquars error and tighter confidence intervals for a given annotation budget than existing base- lines. We demonstrate that MultiPPI achieves this by automatically tailoring its strategy the avail- able budget B: that is, it learns to rely primarily on the cheaper autoraters when the budget is small, and naturally begins incorporate more expensive, better autoraters as the budget increases. Taken together, our work provides a principled and computationally tractable framework for cost-effective, midel-aided statistical inference, in settings with complex cost-versus- perrrusspeeo tradeoffs.
In summary, our main contributions are as follows:
We introduce the MultiPPI estimator and frame the problem of finding the optimal subset sam- pling strategy and estimator weights as efficient second-order cone program (SOCP).
We prove that the MultiPPI estimator is minimax optimal when the covariance matrix E of XI....,A X2 is known, and provide finite-sample performance guarantees for the practical setting where the covariance matrix must first be estimated part of the overall inference problem.
We demonstrate MultiPPI's applicability across multiple LLM evaluation settings, and show how it can effectively combine signals from different model sizes, reasoning configurations, and multi-agent debates to achieve lower error and tighter confidence intervals for a given budget.
2 RELATED WORK
Our work builds upon Cistiction-Poweree Inference (PPI; Angelopoulos et al., 2023a), statistical framework for efficiently estimating population-level quantities by augmenting small set of labeled data with predictions from a machine learning (ML) model. We specifically build on PP1++, the efficient extension of PPI introduced in Angelopoulos et al. (2023b), which also further improves variance by optimally reweighting these predictions. We describe PPI in greater depth in Section 3.
PPI is part of a broader class of statistical methods that leverage ML predictions for estimation. Its principles connect to classical control variates and difference estimators (Ripley, 1987; Särndal et al., 1992: Chaganty et al., 2018), which reduce variance by subtracting a correlated random variable with known mean; the correlated variable in PPI is the ML prediction. whose mean can be (cheaply) estimated on unlabeled data. This approach also shares theoretical foundations with modern semi- parametric inference, particularly methods from the causal inference literature like Augmented In-
2

---

Published as a conference paper at ICLR 2026
verse Propensity Weighting (AIPW; Robins & Rotnitzky, 1995), Targeted Maximum Likelihood Es- timation (TMLE; van der Laan & Rubin, 2006), and double machine learning (DML; Chernozhukov et al., 2018). Recently, PPI has been applied to Generative AI evaluation, where human annotations (or more generally, annotations from some trusted source) are combined with cheaper "autorater" outputs for efficient, unbiased estimates of model performance (Boyeau al., 2024; Chatzi al., 2024; Fisch et al., 2024; Angelopoulos et al., 2025; Saad-Falcon et al., 2024; Demirel et al., 2024).
Existing PPI frameworks, however, assume either single predictor (Angelopoulos et al., 2023a;b) or a fixed set of predictors queried together (Miao et al., 2024). We address the common scenario where multiple predictors (e.g.. autoraters) with different post-performance profiles are available. This introduces complex budget allocation problem: determining which predictors to query (in- dividually, jointly, or in any joint subset), how often to query them, and how to combine the mea- surements they provide for a minimum-variance estimate under fixed budget. Our work partially generalizes Angelopoulos et al. (2025), which optimizes sampling policy for single predictor. Unlike that work, however, which focuses on input-conditional policies and expected budget con- straints, we find a fixed allocation policy that always satisfies a hard budget constraint for every run.
Our allocation problem is also related to budgeted regression with partially observed features (Cesa- Bianchi et al., 2011; Hazan & Koren. 2012) and active learning or testing (Settles, 2009; Kossen et al., 2021: Zhang & Chaudhuri, 2015). We emphasize, however, that our goal is estimation of linear function of a population mean (i.e., a [[X)), and not regression (e.g.. predicting X₁ from Xg) While related, standard approaches to regression, including with partial observations, optimize for sample-wise predictive accuracy rather than for predictive accuracy of a population- level quantity. Our problem also connects multi-armed bandit allocation for adaptive Monte Carlo estimation (Neufeld et al., 2014). A key difference is that these frameworks often use sequential, input-dependent policies minimize regret, making difficult to derive valid confidence intervals (Cls). Our framework, in contrast, computes a fixed allocation policy over predictive models (not individual inputs as in active learning or testing) and guarantees unbiased estimates with valid Cls. Even more broadly, our work shares similar high-level goals with transfer learning and domain adaptation (Pan & Yang. 2010; Ben-David et al., 2010, inter alia)-i.e.. leveraging signals of varying quality and potential bias though the statistical techniques are distinct.
3 PRELIMINARIES
In the following section, we introduce the general estimation problem of interest and summarize existing approaches. Suppose that we are interested in the mean a random variable X,, which is dependent upon another random variable X2 (corresponding to estimating a [[[X] for as described in $1). For example, in the Al model evaluation setting, X₂ may be an autorater's score for model output to user's query, and X₃ may be the ground truth quality of the response as measured by an expert human annotator. Suppose we have access to a small number (n) of i.d. samples that contain labels from both the target rater (X1) and autorater (X2), and large number (N) of i.i.d. samples that contain only the autorater predictions (X2). A naïve approach to estimat- ing the mean is to simply take the sample average of X, and ignore X₂ entirely, which we denote by delnssic 1 Σド x When the prediction X₂ is correlated with X; and easy to query, however, it is natural to consider the "prediction-poweree" PPI estimator (Angelopoulos et al., 2023a:b):
(1,
N
Oppi
1 Σ (x(1) -x(')
Σ
(1)
N
2
1=1
When we can afford to take N to be very large, it is clear that the variance of Oppr is much smaller than that of denssii provided that our model predictions X₂ are close to X1 in mean-squared error. When that fails, Angelopoulos et al. (2023b) propose adding a linear fit of the form:
8pri++ n
-
1
-x()))x3")
I N
Σ XX
The parameter À may be chosen to minimize the variance of 0ppl++ based on the observed labeled data. This strategy yields estimator which asymptotically improves on detasic and Opei in the
3

---

Published as a conference paper at ICLR 2026
limit that - and N > Toward the setting where n and N may be comparable in size, if one is able to choose to or not to request a label x_") for every observed unlabeled point x.(") modification of Oppi.+ allows one to do so in a cost-optimal way (Angelopoulos et al., 2025).
3.1 MULTIPLE PREDICTIVE MODELS
How should one adapt the preceding setting when one has access to many predictions, rather than just X2? One option is to stack all predictions into vector X2.; == (X2..... Xx) and choose À E Rk-1 to be vector in pprl++: this is the estimator proposed by Miao et al. (2024), and can be written
Oppies vector - Σ (x
X21x)
1 N
Σ
X
X
( 2:
(3)
But this approach is suboptimal when (as is becoming standard) the best models may be available only for the highest prices: if any of X2..... is expensive to obtain, our ability to sample X2.k will be limited. This yields suboptimal results, we show in $6. One may instead decide to perform PPI with just one model X,, for whichever * 1 has the best cost/accuracy tradcoff- not clear a priori which one this is, how much worse it may be compared to some combination of a cost-effective subset of X. Alternatively, perhaps it is possible for cheaper models be used recursively estimate the means of more expensive models, thus creating PPI++ cascade: for instance, if k = 3 and (X1, X2, X3) are in decreasing order of cost, we might consider
M
Oppi++ cascade
Σ (x + N
Σ (AXX" - xXX") 3=1
+
1 M
Σ XXXO
(4)
Each of these strategies can be realized as possible instances of the MultiPPI estimator we propose in the next section. Rather than coarsely limiting ourselves to sampling X2.2 (X2..... Xx) together, we allow the flexibility of sampling X2 for generic index subsets I S (I.....,kk ....
4 MULTPLLEPREDICTIONPPPEEEE INFERENCE (MULTIPPI)
As Section 3.1 highlights, it is not obvious how to best allocate a budget across a diverse suite of predictive models, where each model has its cost and performance tradeoffs. We begin by defining the class of permissible estimators: We require that the number of times, nt, that X2 is sampled satisfies linear budget constraint, specified by a set of non-negative costs ci 2 and total budget B 2 O, for each index subset 1 S {I....,k, kk).'
Definition 1. An estimator Ô is budget-satisfying if it measurable function of ni i.i.d. samples of X,, for each C {I,...,k,,, kj, such that E, πeer S. B.
To develop a principled search for the best budget- Bodgee--satisyingg estimator, we begin by asking simple question under idealized conditions:
Question 1. If the covariance matrix, Σ = Cov(X), is exactly known, what is the minimax optimal,
mudget-satisfying estimator of = p with respect to the mean- aeean-squared error, E[(Ō - 33?
The answer to Question will provide us with set of allocations n, and a corresponding budget- satisying estimator MuluPPI which we will evaluate on the 1ı samples of X,, for each I. Once we have addressed this question, address the case of unknown E by describing strategies depending on the empirical covariance matrix E, which may be estimated from data.
It turns out, perhaps surprisingly, that Question reduces to the following tractable alternative: Question 2. If the covariance matrix, Σ Cov(X), is exactly known, what the minimum vari- ance, linear, unbiased budget-satisfying estimator of 0- = a. u?
We demonstrate the equivalence of Question and Question 2 in Theorem 2. For now, the "oracle" assumption knowing the covariance matrix E allows to isolate the resource allocation problem
In Scction B, extend the methodology multiple hudget constraints.
4

---

Published as a conference paper at ICLR 2026
from the separate challenge of estimating how closely related (X1,.... Xx) are to begin with, and to analyze what good procedure for leveraging multiple predictive models under cost constraints should look like in theory. All proofs of our theoretical results are deferred to Appendix F.
4.1 MULTIPPI()): A MINIMAX OPTIMAL ALGORITHM
Recalling notation from Section 1, let X R denote random vector of finite second moment with distribution IP. Let I S 2(1...,)) denote collection of index subsets which may be queried, and for any T E I, let X = (Xr be the corresponding subset of X. Next, let = (nr)reI, ny E N be an allocation of sample sizes, where nı i..d. samples are drawn for each subset 1, and let À {Arhee, À E Rl' define corresponding set of weighting vectors for each subset I. Finally, let (nn A) denote the weighted sum of sample means from each non-emply subset, i.e.,
((1.2)
Σ 1:
The MultiPPI estimator, MumuprI then defined as the optimal estimator in this class that minimizes the MSE subject to our unbiasedness (U) and budget (B) constraints:
MattPPr argmin E (0(n -0) 0 ((n,A)
s.t. Undd B hold,
(6)
where the constraints U and B are
U > ElO(n. A) 0" for all P of finite second moment
and B B
E nier S B.
It can be shown that U reduces to a linear constraint A. which makes optimization convenient.
As previously discussed, the estimators of Equation (3) and Equation (4) can be viewed as special cases of this setup. For instance, Equation (3) corresponds to imposing the additional restriction
that Å 0 for all / E 2(1....k) except for 1 (1,..., kj and = (2..... k); Equation (4) corresponds to the additional restriction that \ for all except for {1,2]. [2,3] and (3}.
4.1.1 OPTIMIZATION
Solving Equation (6) is, in general, non-trivial. Since (n, A) linear in X, it can shown that the optimal (n, A) depend only on the covariance matrix E of X, and so we will denote by MMTHiPP)()) the solution to Equation (6) given any distribution such that Σ = Cov(X) Then, can be further shown (this follows from Theorem 2, presented next) that the MSE of MMtHPPP((;)) is
VB=
a S(n(j,, [U
min
S(n)
Σ n VET
where E, denotes the principle submatrix of E on I, embedded back into Reck k and i denotes the Poorr-Penrooee pseudo-inverse 2 The minimizing n of the above expression then also determines the optimal Y to be the restriction of n, 2 (())a to the coordinates I. If the integrality constraints on ny are relaxed, we show in the appendix that this reduces to a second-order cone problem in the case of single budget constraint, and semi-definite program in the case of multiple budget constraints. This allows for Equation (7) to solved efficiently using standard techniques (Section G).
4.1.2 MINIMAX OPTIMALITY
The minimal MSE VA shown in Equation (7) has more fundamental characterization. Here we show that it is in fact the minimax optimal MSE achievable by any budget-satisfying estimator, taken
More formally, Pt R denotes the orthogonal projection onto span(/) C R* we define E, P.E.T and so 5 (P.E.TT)*.
5

---

Published as a conference paper at ICLR 2026
over the set of distributions P of covariance Σ. Consequently, the estimator defined by (MuniPp(())) minimax optimal over the set of distributions
Px: {distribution P on R* Cov(X) = E for X P}.
Specifically, given costs z and a budget B, let Gg denote the set of budget-satisfyle estimators 0 per Theorem 1. We emphasize that we make no restriction θ to include only linear or unbiased estimators. Then the following result holds:
Theorem (Minimax optimality of MultiPPI for known 2)) For all 2 > 0, we have
inf sup E b.an PEPs
( 0-)]]
Var (fMtutιrс1())) VB,
where the variance is with respect to any distribution P E Ps.
4.2 MULTIPPI(S): A PRACTICAL ALGORITHM
In practice, E is rarely known and must be approximated by estimated covariance matrix E. In general, there are many methods for constructing estimate of E, and many of the theoretical properties of MultiPPI are agnostic to the particular choice made. The following theorem shows that, for any Σ which converges in probability to Σ as our budget tends to infinity, the MultiPPI estimator is asymptotically normal and achieves the optimal variance of Theorem 2. For this result, we need technical condition which amounts to Equation (6) having unique minimizer as B oC; we state it formally in Section F.2.
Theorem 3. Suppose X E R has finite second moment, and suppose that E = Cov(X) satisfies condition 14. Suppose that E 2 Σ in the operator norm - oc. Then for 0 MutriPp((5) arbitrarity dependent on any potential samples used to estimate Σ., we have
BB Mularri(() -0+) -00 1 VO,,V")
as - oc, where V* Bim BVB, and Va defined in Equation (7).
is important to note that the estimator Ô MuluPPI(S) continues to enjoy unbiasedness, budget satis- faction, and asymptotic normality regardless of mis-specification in E.
natural question concerns the level of suboptimality of é MuttiPPT(5) as a function of the degree of mis-specificatioon of E in finite samples. Below, we present meta-result which serves to quantify the sensitivity of our procedure to eπrors in the specification of E.
Theorem 4 (Stability of MultiPPI), Let P be a distribution of covariance E. and suppose that E has minimum eigenvalue Thuin. Let d'lassicaa denote the least MSE of any budget-satisfying satisfying sample mean of 0r, Let E denote any non-random symmetric positive-definte matrix. Then we have
E
(BAMLUPPHS))
-8+)
S VB +
40 classical
1 E|
Vmmi
whenever [ E S Ymin/2 where || || denotes the Frobenius norm.
In general, there are many methods for constructing an estimate E of Σ, and Theorem 4 is agnostic to the particular choice made. In Section E.I, we show how to apply the meta-result above to derive a family of finite-sample bounds E. In practice, we estimate Σ from data, and find the Ledoit- Wolf estimator E. of the covariance matrix Σ perform best in our experiments. This consistent with the fact that the Ledoit-Wolf' estimate is designed to minimize E S ElEF, and Theorem shows that the error of ê MulrPPr(S)
variety of distributional settings and for variety of estimates
controlled by S5 [r.. p. Theorem 9, we apply Theorem 4 to provide finite-sample performance guarantees MultiPI when the Ledoit-Wai Wolf estimator is used to estimate covariance.
In our experiments, evaluate 0 the same samples used to estimate E. A similar approach was taken by Angelopoulos et al. (2023b) for PPI++ when tuning the value X, and we find that easy to implement and yields strong empirical results in practice.

---

Published as a conference paper at ICLR 2026
Note that while the data reuse introduces bias in finite samples- due in part the additional dependency of A on X in Equation (5)-it it preserves consistency and asymptotic normality in the limit as our budget B and the number of (reused) burn-in samples tend to infinity. For analysis of the bias introduced in finite samples, see Section E.5.
4.2.1 PROCEDURE
We now specify an easy-to-implement procedure that makes use of burn-in of N fully labeled samples to estimate 5., and then also reuses the N samples when estimating 6 Mumppr(s. Specifically, we target the practical setting where we are given N fully-labeled samples a priori, and have no ability to obtain more. This is typical of weal-world settings in which we are given, or have already collected, a fixed dataset of "gold" labels that we are then trying to augment with PPI related techniques- and may be encapsulated by the budget constraint "!!.....k) I N. While we may not be able to obtain more fully-labeled samples, we may afforded separate computational budget for querying model predictions that then augment the N fully-labeled samples; taken together, this setting is represented by a system of budget constraints." In summary, we propose the following:
1. Estimate the covariance matrix E on the N fully-labeled samples, which we will reuse in Step 3.
2. Solve for the n,, Ar which minimize Equation (6) given Σ. We refer to this as MultiAlllccte/ E)
3. Sample the nts, 4/ E I additional data points accordingly, and return O MuNPPP) evaluated on both the n, additional data points for each I E I, and the N (reused) initial samples.
EXPERIMENTAL SETUP
In each experiment, our goal is estimate the mean 0- = E[X]] of some random variable X1 to be specified, which we will refer to as the target. This corresponds to the choice a = (1,0,..... .) in our notation. We will also specify model family (X,,..., Xx), together with a cost structure (CI)IEI. In each experiment,
given some number of samples for which the entire vector X = (.,,..., Xx) is visible; we refer to such samples as fully-labeled. Given these samples, we perform the procedure outlined in Section 4.2.1: we estimate Σ using these samples, sample from the auxiliary models (X2,..., xy) according to the allocation specified by MultiAllocate (E). and return O MMMPPII() evaluated on both the N fully-labeled samples and the additional auxiliary data.
In each experiment, we compare several baselines. First, we compare to classical sampling. Second, we compare to PPI++ with each model included in the family (specified in Equation (2)), and to vector PPI++ with every model in the family (specified in Equation (3)).
Baselines:
Experiment 1: Estimating Arena win-rates by autorater ensembles. We focus on the Chatbot Arena dataset (Chiang et al., 2024), where of interest is the win-rate between a pair of models, which is the probability that a given user prefers the response of one model to that of the other. The randomness taken over the prompt, the user, and the model responses. Here, we aim to estimate the win-rate between Claude-2.1 and GPT-4- 106-Previeww this is our target. Our model family consists of autoraters built on Gemini 2.5 Pro (without thinking) and Gemini 2.5 Flash. In our notation, we have (X,, Xx) (human label, Gemini 2.5 Pro label, Gemini 2.5 Flash label). We draw model costs from the Gemini developer API pricing guide (Gemini APD, see Section I. In this case, the cost of querying both models is simply the sum of the costs of querying each model independently.
Experiment 2: Optimal test-time autorater sealing on ProcessBench. In this experiment, we aim to estimate the fraction of correct solutions in the ProcessBench dataset (Zheng et al., 2024), given a small number of labeled examples. The task is simplified from its original form to a binary classification problem: determining whether a given math proof solution contains a process error, without identifying the specific step. We employ Gemini 2.5 Pro with a variable thinking budget as our autorater. Its accuracy correlates with the number of words expended in the thought, with performance gains saturating after approximately 500 words (see Figure 12 in the appendix). We create a family of four autoraters by checkpointing the model's thought process at 125, 250, 375,
We explain how to solve the optimization problem posed by such systems in Appendix B.
7

---

Published as a conference paper at ICLR 2026
and 500 words. A key aspect of this setup is the non-additive, cascading cost structure. Generating response from a model with larger thinking budget makes the outputs of all smaller- budget models available at marginal cost. Consequently, the total cost for subset of models S is modeled with two components: an input cost proportional to the sum of the word budgets in S. and an output cost proportional to the maximum word budget in S. Explicitly, for 5 _ {125, 250, 375, 500), we set
cs = oatput_cost__pe- wooddwordd--o - max S +
Σ
(8)
For concision, in our results we refer to the model assessments after 125, 250, 375, and 500 words by "tiny," "small," "medium," and "large." " respectively.
Experiment 3: Hybrid factuality evaluation through multi-autorater debate. Following Du al. (2023), we evaluate the factual consistency of biographies for 524 computer scientists gener- ated by Gemini 2.5 Pro. For each person p E P. we compare their feemin--generate biography D against a set of known grounding facts F ..., Sm,,
about the person. Our target metric is the proportion of factually consistent pairs (b. f) within the total set S = (bb.f)) FP). }. Concretely, we target the proportion f) E S (6, f) is factually consistent) / /|S).
E P. E
Ground-truth consistency of a pair (b, f) is established by majority voting over five independent judgments from Gemini 2.5 Pro with thinking, method validated by Du et al. (2023) to have over 95% agreement with human annotators. Our experiment, illustrated in Figure 13, assesses the performance of more cost-effective model, Gemini 2.0 Flash Lite, as an autorater. To elicit better autoratings from queries to Gemini 2.0 Flash Lite, we bootstrap performance via multi-round debate. For fixed number of agents A E {1, 2, and fixed number of maximum rounds R E {1, we perform the following procedure: In each round, A instances of Flash Lite are independently prompted to provide reasoned judgment on the consistency of a pair (b, f) E E.S. A "pooler" instance of Flash Lite then consolidates these responses into single yes, no, or uncertain output. A definitive yes or no concludes the process. If the pooler outputs uncertain, and the number of maximum rounds It has not yet been reached, the A agents review all prior responses and continue their debate in a new round. If the output remains uncertain after the final round, either yes or is reported with equal probability- since the dataset is balanced, this outcome is fair insofar as it is as good as random guessing. We impose the maximum round restriction to encapsulate our budget constraint. For a given (A, R), the cost is A R; for collections, the cost follows Equation (8).
6 EMPIRICAL RESULTS
We plot MultiPPI, and the baselines described in Section for a budgets between 0 and 2k: units of cost. We normalize model costs so that one unit of cost always represents exactly one query to our most expensive model. For each fixed budget and each method, we estimate the target, and construct asymptotic 95% confidence intervals C based on Theorem 3. We plot
(i) coverage, P(00 E C); (ii) confidence interval width, |C]: and (iii) mean-squared error E[(@ )². We report both the confidence interval width and the mean-squared error as a fraction of what classical sampling achieves (lower is better). In each case, the target is 0" B[X1), and TP and E. are computed with respect to the empirical distribution over the dataset observed (we perform 500k: random trials with 250 given labels). Note that these 250 labeled points are evidently enough for all estimators considered to achieve good coverage (in Section D.2 we also include results with 1k labeled points). We implement the optimization scheme in cvxpy, and use CVXOPT as our choice of optimizer.
Chatbot Arena. Results are shown in Figure 1 (top). Observe that different base- lines dominate in different budget regimes. In the low-budget regime, scalar PPI++ with Gemini 2.5 Flash is the best baseline, while in the large-budget regime, vector PPI++ with both Gemini 2.5 Pro and Gemini 2.5 Flash is the best baseline. However, we see that MultiPPI improves on all baselines in all regimes. In the appendix, Figure 5 and Figure 2 plot the \ and n, values learned by MultiPPI across budget regimes. Note that the learned values tend to the specifications for PPI++ with Gemini 2.5 Flash in the low-budget regime, and to the specifications for vector PPI++ in the large-budget regime, a finding that we rigorously prove happens in broader generality in Section E.2. Lastly, note that PPI++ with Gemini 2.5 Pro is suboptimal in all regimes. In other words, PPI++ with Gemini 2.5 Pro is not included in the Pareto frontier. This is because, for this task, its correlation with the label is no more than that of PPI++ with Gemini 2.5 Flash, yet it is strictly more expensive.
Experiment 1:
8

---

Published as a conference paper at ICLR 2026
Experiment 2: ProcessBench. Results shown in Figure (middle). Again, that each baseline has a range of budgets for which outperforms all other baselines. In particular, the cheaper models yield better performance when used in PPI++ in the smaller-budget regimes, while the more-expensive models yield better performance in the higher-budget regimes. In partieular, vector PPI++, which uses all & models, steadily improves as the budget increases, but only outperforms the other baselines at the highest budgets. This behavior is explained by Figure 12 in the appendix, which shows that predictive performance improves for larger thinking budgets. Thus the more expensive models yield higher correlation with the label and thus yield low-variance rec- tifiers; on the other hand, their high cost means that this decrease in rectifier variance outweighed by our inability to draw an adequate number of samples from them in the low-budget regimes.
Of note is the fact that the models which think for longer are not general less biased. This phenomenon is shown in Figure 15, which shows that thinking for longer is not enough to resolve the systematic bias present the autorater. However, the figure also shows that simple debiasing schemes like PPI resolve this issue. Note that this trend is not reflected in the correlations between these models and the label, because correlation is invariant to addition of constants.
Finally, MultiPPI improves on all baselines methods in all regimes. Interestingly, Figure 3 and Figure show that the parameters a and nı learned by MultiPPI transition from emulating PPI++ with the tiny model (which is the best baseline in the low-budget regime) to emulating cascaded version of PPI (see Equation (4)), in which the medium model is used to debias the larger model.
Experiment 3: Biography factuality evaluation. Results shown in Figure/ (bottom). Once again, each baseline is dominant over the others certain regimes; MultiPPI improves on all base- lines in all regimes. Of note, however, is the fact that the coverage of all estimators considered, but MultiPPI and vector PPI++ in particular, degrades slightly in the large-budget regime (i.e., the 95% CI under-covers by 1%). We discuss this interesting phenomenon in Section E.5, and find that does not occur when the number of labeled samples grows in constant proportion with the budget (see, for example, our additional results with N 1000 fully-labeled samples in Section D.2).
In terms of the performance-vs-cos profile that MultiPPI leverages: Figure 13 shows that predictive performance increases, across many metrics, as the number of agents and number of rounds in- creases. Note, however, that a marginal increase in number of agents yields greater increase in ac- curacy than marginal increase number of rounds (this is largely because the pooler is more likely to report "uncertain" after the end of the first round than after the end of the second; see Figure 14).
7 CONCLUSION
In this work, we introduce ciiiee-rrriciiio---oeee ciently estimating expectations under budget constraints by optimally leveraging multiple informa- tion sources of varying costs. MultiPPI formulates the optimal allocation of queries across subsets of variables as a second-order cone program in the case of a single budget constraints, or a semi-definite program in the case of multiple- both can be efficiency solved using off-the-shelff tools. We provide theoretical guarantees, including minimax optimality when covariances are known, and demonstrate empirically across diverse LLM evaluation tasks that MultiPPI outperforms existing methods. By adaptively balancing cost and information, MultiPPI achieves lower error for a given budget, au- tomatically shifting its strategy from cheaper proxies to more expensive, accurate predictors as the budget increases, thus offering principled and practical solution for cost-effective inference.
Inference (MultiPPI), framework for effi

---

Published as a conference paper at ICLR 2026
Experiment I: Chathot Arema
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Eraor (frac, of classical)
961
988
Sader (Pho
Solerr
96%
9.%
92%
9216
901
2,000
Bulget
Badget
Balget
Experiment ProcesBench
Coverage
Mcan-squared CI width (frac. of classical)
Mean-squared Ervor (frac. of classical)
100%
989
Scala Stall
Scala
Mutritv
901
PPiie Vocter
801
9216
5016
50%
90%
2,000
4,000
2,000
4,000
Dodget
Budget
Budget
Experiment 3: Factuality
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Error (frac. of classical)
100%
9099
Pffe- Verne
98%
Pite- Soaler mood
Pitee Soaler
80%
PPI1) Scates sacab, seand
90%
PPT Seater saent moodd
PPT Staler Zegens Joand
80%
Pftee Stater Sagens, Soands
80%
928
75%
75%
90%
2,000
Budget
Budget
Budget
Figure 1: Results by budget for the experiments on Chatbot Arena (a), ProcessBench (b), and Fac- tuality (c). For each estimator (all baselines and MultiPPI, the left column plots the empirical cov- erage of the 95% CI, the middle column plots the width of the 95% CI, and the right column plots the empirical meanes error the point estimate. The fulle-labeled sample size is 250.
ACKNOWLEDGEMENTS
We are very grateful for insightful comments and suggestions by the anonymous reviewers. We are also grateful for helpful discussions with Jonathan Berant, Michael Beukman, Frederic Jørgensen, Petros Karypis, Chris Dyer, and Martin Wainwright.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide a detailed specification of the algorithm in Section C. We also include implementation details in Section I, and address computational considerations in Section G. Finally, all experiments shown in $6 were averaged over 500k trials.
REFERENCES
Anastasios Angelopoulos, Stephen Bates, Clara Fannjiang, Michael Jordan, and Tijana Zmic. Preweetion-poweredd inference. Science, 382((6772)6 (66666616666677444 2023a.
10

---

Published as a conference paper at ICLR 2026
Anastasios N Angelopoulos, John C Duchi, and Tijana Zmic. PPI++: Efficient prediction-powertt inference. arXiv preprint arXiv:2311.0145. 2023b.
Anastasios N. Angelopoulos, Jacob Eisenstein, Jonathan Berant, Alekh Agarwal, and Adam Fisch. Cost-optimal active ai model evaluation. arXiv preprint arXiv 2506.07944 2025. URL https: / //arxiv. orq abs/ 220025066.0 060066600 07949.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. theory of learning from different domains. Mach. Learn., 79(1-2):551-175, May 2010.
doi: 10.1007/s109994009--51111
URL https://aci. 1007/
s1
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Au- toEval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008 2024.
Nicoló Cesa-Bianchi, Shai Shalev-Shwartz, and Ohad Shamir. Efficient learning with partially observed attributes. Journal of Machine Learning Research, 12(87)22857-288 2011. URL html.
ptpppaapeaa//////ep / www.-bianncceeeeeeeeeeeeeeeeeeee wwwp:ptw.g.p.aa..eeeeee http: / jmlr httpl//lrr..ra/......e..eeeooeee www.orrgg.oo.............
Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 643-653, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/P16 1/P88 1060. URL https: www.lanthology..oooo.o
1060/.
Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, and Manuel Gomez Rodriguez. Prediction- powered ranking of large language models. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp- 113096-113133, Curran Associates, Inc., 2024. URL https: attps://pproeeeddgggseeeele nwwrips cctppppppppppeeeeeeeeeeeeeee..ee wiiess/ppprr//000 pdf.
file
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural pa- rameters. The Econometrics Journal, 21(1):C1-C68 01 2018. URL 1111/ectj. 120977.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning. 2024.
I1ker Demirel, Ahmed Alaa, Anthony Philippakis, and David Sontag. Prediction-powered general- ization of causal inferences. arXiv preprint arXiv:2406.0288 2024.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
Adam Fisch, Joshua Maynez, Alex Hofer, Bhuwan Dhingra, Amir Globerson, and William W. Cohen. Stratified prediction-powered inference for effective hybrid evaluation of language mod- els. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. net URL https::/ http://wwww...eetttaaaaaaaaaa--- www.:oiddddBaedddaeee.
Gemini API. Gemini Developer API Pricing Gemini API. URL htttttt///i.ggoon dev/
gemini- careini-api//dee httpa/ppcc/c//riciin..ggsese/eee
Elad Hazan and Tomer Koren. Linear regression with limited observation. In ICML, 2012.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.

---

Published as a conference paper at ICLR 2026
Jannik Kossen, Sebastian Farquhar, Yarin Gal, and Tom Rainforth. Active testing: Sample-efficient model evaluation. In /CML, 2021. URL http: icm12021 html FKossenGGR1..
uni-trier. trrie de /t/ddd//onft detp/////d////////eeee/////cocca /
Jiacheng Miao, Xinran Miao, Yixuan Wu, Jiwei Zhao, and Qiongshi Lu. Assumption-lean and data-adaptive post-prediction inference. arXiv preprint arXiv 23/1.14220, 2024. URL https: 14220.
/
James Neufeld, Andras Gyorgy, Csaba Szepesvari, and Dale Schuurmans. Adaptive monte carlo via bandit allocation. In Bric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume of Proceedings of Machine Leurning Reseurch, pp- 1942-1952, Bejing, China, 22-24 Jun 2014. PMLR. URL hitps: ptstps://procceeinnge mmlr. https////2/neeeeeelleeee htt./nnuffeedeeeeeee 4. html.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22:1345-1359, 2010. URL https: /tpp//app...a/ http:/p..emmatiiiicscc.seemooeae org/CorpusID: 740063.
B. D. Ripley. Stochastic simulation. John Wiley & Sons, Inc., New York, NY, USA, 1987. ISBN 0-471-8188444.
James M. Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression mod- els with missing data. Journal of the American Statistical Association, 90(429); 1995. ISSN 01621459. URL http: /ww. jstor. org/ s st ataeelleeeeeeee 2291135.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated eval- uation framework for retrieval-augmennedd generation systems. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers). pp. 338-354, Mexico City, Mexico, June 2024. Association for Computational Linguis- URL https: / /acc/acaaana anthologh org/2024.
tics. doi: 10. (0.18653/v11220 h/a.l2.aaaccllloonneee naacl-long 20/-
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, Uni- versity of Wisconsin Madconsnn-Madison.. 2009. URL http::/ cs wy.. boy..edu/mmartinee (assses/778/Ppeeeee settelssseeeesseeeeeeeeessseesee leessacttveee wetteeeeernnng.... pdf.
Carl-Erik Särndal, Bengt Swensson, and Jan Wretman. Model assisted survey sampling. Springer Series in Statistics, 1992.
Mark J. van der Laan and Daniel Rubin. Targeted maximum likelihood learning. The Inter-
national Journal of Biostatistics, 2(1), 2006. doi: 10.2202/1557-4479. URL https: / //www. www.dwgerrrrr.eeeo com www.o////wo..o////oooccoooo /0ccmmeent//oo//002220//5555
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Martin J Wainwright. High- digenssimnsional statistics: A non-asymptotic viewpoint, volume 48. Cam- bridge university press, 2019.
Chicheng Zhang and Kamalika Chaudhuri. Active learning from weak and strong labelers. In Advances in Neural Information Processing Systems, 2015.
Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Processbench: Identifying process errors in mathematical rea- soning, arXiv preprint arXiv:2412.06550 2024.
12

---

Published as a conference paper at ICLR 2026
CONTENTS
A Ethics statement
15
B Generalization to multiple budget inequalities
15
C Detailed specification of the algorithm
15
C.I The case of a single budget inequality, known
15
C.2 The case of multiple budget inequalities, known E.
16
C,3 The case of unknown Σ
17
Additional experiments
18
D.1 Learned allocations and linear parameters
18
D.2 MultiPPI by varying number of labeled samples
D.3 The impact of shrinkage covariance estimation
19
D.4 Scalability and computational tractability of the estimator
19
D.5 Autorater accuracy scaling
19
E
Additional theoretical results
27
E.1
Finite-sample bounds
27
E.2
Behavior of the estimator in the limiting regimes
28
E.3
Rounding the large budget regime
28
E,4
The continuous problem and its properties
29
E,5 Decay of coverage in the large budget regime
30
Proofs
F.1
Proof of Theorem 2
31
F2 Proof of Theorem
32
F3 Proof of Theorem 4
34
F.4
Proofs for Section E.1
34
F.4.1
Proof of Theorem
34
F4.2
Proof of Corollary
35
F.4.3
Proof of Corollary 7
35
R4.4
Proof of Corollary
35
E.45
Proof of Theorem 9
35
F5
Proofs for Section E.2
36
F.5.1
Proof of Theorem 10
36
F6
Proofs for Section E.4
38
F.6.1
Proof of Lemma 17
40
G Computational considerations
41
13

---

Published as a conference paper at ICLR 2026
H
The dual problem
42
Experimental details
43
L1
Generating model predictions
43
1.11
Experiment I: Chatbot Arena
43
1.1.2
Experiment 2: Processbench
43
1.1.3
Experiment 3: Biography factuality
44
I.2
Constructing the MultiPPI estimator
45
14

---

Published as a conference paper at ICLR 2026
A ETHICS STATEMENT
This paper describes fundamental research on techniques for constructing statistically efficient esti- mates of target metric by optimally allocating resources across multiple types of proxy measure- ments. The primary intended use case which is analyzed in this work is the evaluation of generative AI systems, for which reliable evaluation is mates of model performance can help make Al systems easier to build, deploy, and monitor. We do speculate about broader impacts that may follow from this technical contribution. Gemini was used for light copy-editing during the writing of this work.
technical challenge. Efficient and precise esti-
B GENERALIZATION TO MULTIPLE BUDGET INEQUALITIES
We recall some notation. Fix a set I of index subsets S [k:]. For each
let cj
(o) ,..,,.............. ( E Rg00 denote the vector-valued cost of querying the collection of models indexed by I. Similarly, for each I E I, we let ny 2 an integer denoting the number of times that the collection of models indexed by I is queried. We let n = (Nr)rez refer to the associated allocation.
For vector valued budget B (nn, B), if
R we say that the allocation n satisfies the budget B, and write
(1) Emdef S
B
Enc (m) & B(m).
IET
ICI
or more succinctly,
Σ
ije S B.
IeI
Similarly, for each 7 E I, we let Xr E Rl1 and denote by A = (Ar)sez their collection. Let
R1
A
Σ
A7
x(3))
nI
TET:,>>0
j=1
where (1.) denote independent copies X for every unbiased condition holds for n, X, and write U, if EQn, = TEX for every distribution finite second-moment X.
and 1 S S ny. We say that the
Note that the variance of An. depends only upon = Cov(X) Thus we let
MuNPPPR() 0 where n, are chosen so that the resulting estimator has minimal variance under V such that B and U hold.
C DETAILED SPECIFICATION OF THE ALGORITHM
In this section, we outline the procedure used in all experiments in greater detail. First, we describe the algorithm for the case single budget inequality, for which a more-efficient procedure exists; second, we describe the general case, in which the procedure reduces to a semi-definite program (SDP). We first suppose that > is known, and later explain the procedure the case that must be estimated from data.
C.1 THE CASE OF A SINGLE BUDGET INEQUALITY, KNOWN Σ
We suppose that there is a random vector X E g with known covariance Σ, and our goal is to estimate 0*
IE for some fixed k There is some fixed collection I of index subsets C (1....,, [1....,kk }}} such that we may sample X, == (Xi)eeI- We may sample X, a maximum of ny times, subject to the constraint that Eiez Cyn, B for some ej 2 and B 0.
Step 1: Solve the SOCP
AuT S 4 IEI
sup y
ye
and obtain the solution yj and the multipliers of 2 0 for each / I.
15

---

Published as a conference paper at ICLR 2026
Step 2: Set
X; 2aj '7
n; ((e)
(B)
(xr(2)) EM ESEI v(r(A)) E,dy
for each I I.
Step 3: For each E I, independently sample X nj times, and compute the sample mean
X7 X₁. Return
MultPPI(E) Σ
17 X1
IEI
with a confidence intervals given by
C MulriP((EE) + 21-0/2
Σ
X3.X1
n
1E1
where o² xx, X, denotes the sample variance of X7. 7X,, and 2p denotes the p quantile of the standard normal distribution.
C.2 THE CASE OF MULTIPLE BUDGET INEQUALITIES, KNOWN
We again suppose that there is random vector X E R with known covariance Σ, and goal to estimate f" = a EX for some fixed E RR.. We may now sample X, a maximum of ny (2) times, subject to the constraints that ΣlEI Ci 1n I B(0) for some oj S l < m.
(e) 0 and B 0, with
Step 1:
Solve the SDP
Pi
supt tER
s.t.
= 0,
1n 2
VI I VE v m
Σ
CJ - (()
IET
for real valued ny, and obtain solutions i,
Step 2: Set
n² =|nj, |nji,rael itrcc
X; niEj'PP Pt
EntPT
leI
for all I E I.
Step 3: As in the previous section, for each I I. independently sample X, n7 times, and compute the sample mean X; X,. Return
@MulPPP(E))
Σ X1 X₁
IEI
with (1 a confidence intervals given by
1 Σ
C MuNIPP((E)) + 21 2a/2
27.X1
nj
IEI
where 7² X,, denotes the sample variance of X7 Xy, and žp denotes the nh quantile of the standard normal distribution.
16

---

Published as a conference paper at ICLR 2026
C.3 THE CASE OF UNKNOWN E
In general, the approach is to construct an estimate E of Σ from data, and use this estimate for Σ in the steps outlined above. In principle, it is possible to recycle the data used to construct in step 3 of the above procedures; this preserves asymptotic normality as a consequence of Theorem 3. Below, we detail one approach to doing this the approach used in our experiments, and the approach outlined in Section 4.2.
Suppose that (1, o..... 0), and we have some hard limit N on the number of samples available from X, This typically represents a "gold" label which invaluable in some sense. We also sup- pose that these labeled samples are fully labeled thtt is, that the entire vector X = (X1, Xx) visible in each case-or alternatively, that N is small enough that they are relatively inexpensive to obtain model predictions for.
Step Construct the empirical covariance matrix E from the N fully-labeled samples.
Step 2:
Take I to be all subsets of mooels--hhat is, all subsets of { k: together with the set of all indices (1....,k) Formally, I = (1,....,k) ((2......)
.k..
Step 3: Run Section C.2 with any existing budget constraints, together with the constraint that "II..... < N, and obtain allocations nř, X7-
Step
Sample accordingly, with the guarantee that the number of fully labeled samples ((1....k) queried won't exceed the number available, N. These samples from step may be reused for this.
Step 5: Return the resulting estimator, described in Section C.2.
17

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENTS
D.1 LEARNED ALLOCATIONS AND LINEAR PARAMETERS
Propartion of budget al ocated to different models
80%
Joint
Gemini 2.5 Pro
propertion of budget
60%
20%
1000
1500
2000
3000
Budget (5)
Figure 2: Proportion of budget allocated to different models in Experiment 1: ChatBot Arena. Gem- ini 2.5 Flash, the cheapest model, is most sampled in the low-budget regime, while the proportion of budget allocated to the joint (both models combined) increases monotonically with budget.
Proportion budget allocated different models
100%
Sampling
Sampling large
Propertion of budget
80%
Samoling
Sampling small
Sampling
60%
20%
096
0
500
1000
1500
2500 Budget
2000
3000
3500
Figure 3: Proportion of budget allocated to different models in Experiment 2: ProcessBench. Tiny (125 word thinking budget) most sampled the low-budget regime, while the proportion of budget allocated to the joint (all models combined) increases monotonically with budget.
Loat tod sumbing paramette On recfffr serr
.eareet kunii patamster Dnpttt hert
Leareet 3. poramssee inciidul ture!
320
3.9
332
-05
12.0
2.00
2.00 2500
1607
2002
Figure 4: Linear parameters A learned across budget regimes in Experiment 2: ProcessBench. While only the tiny model (125 word thinking budget) has a nonzero linear parameter in the low- budget regime, cascading behavior is learned in the large-budget regime: the cheaper models prescribed the opposite sign from the more-expensive models in the joint term.
18

---

Published as a conference paper at ICLR 2026
Lareeloonaggae Jored leve
Learre 21288
-020
6.4
2.13
0.13
-123
688
3000
Backert
300
2783
Figure 5: Linear parameters A learned across budget regimes in Experiment 1: ChatBot Arena. While only Gemini 2.5 Pro has a nonzero linear parameter the low-budget regime, cascading behavior is learned in the large-budget regime: the cheaper model (Gemini 2.5 Flash) is prescribed the opposite sign from the more-expensive model (Gemini 2.5 Pro) in the joint term.
D.2 MULTIPPI BY VARYING NUMBER OF LABELED SAMPLES
In this section, compare results of MultiPPI for variable number of fully-labeled samples. MultiPPI continues to achieve smaller MSE than all baselines in all settings considered. In Figure 8, plot the performance of MultiPPI with number of fully-labeled sample varying between N = 10 and N = 200. The extreme case N = 10 is shown in Figure 6, while N 1000 is shown in Figure 7. Even for N 10, we find that MultiPPI improves on all basclines in MSE. It is important to note that, for all methods, including the baselines, the coverage is significantly below 95% due to the small sample size. Nevertheless, even in this extreme setting, MultiPPI performs best in MSE.
D.3 THE IMPACT OF SHRINKAGE COVARIANCE ESTIMATION
In this section, discuss the impact of shrinkage covariance estimation MultiPPI. We provide finite-sample bounds on the induced performance, and empirical results.
In Figure 9, compare performance of MultiPPI with covariance estimation via (a) the empirical covariance matrix, and (b) the Ledoit-Woo estimated covariance matrix.
For finite-samen sample bound on our estimator with shrinkage estimation, see Theorem 9. For more general results on sensitivity to mis-specification, please refer to Theorem 4.
D.4 SCALABILITY AND COMPUTATIONAL TRACTABILITY OF THE ESTIMATOR
SOCPs and SDPs are known run in polynomial time in the number of contraints, which is, in our formulation, [Z]. In the preceding sections we have made the choice I (2.....k) but we show in this section that we may recover much of the same performance with a choice of C (2.....)) which grows only linearly in k. Specifically, we take I {[l,..., k). (2....,. k), (2)..... { which corresponds including terms for each model individually, as well for their joint. We label the version of MultiPPI induced by this choice "MultiPPI (Restricted)." Figure 10 shows that the results of this method are very comparable to those of standard MultiPPI, which we take I to be the collection of all subsets of (1...., k}. For more information the computational tractability of the procedure, please see Section G. While our restricted formulation empirically recovers the performance of the full exponential search space by growing only lincarly in k, a formal theoretical guarantee for this heuristic remains an open question. Proving that the restricted estimator incurs small variance penalty in comparison the full O(2*) formulation would validate its use for larger autorater ensembles.
D.5 AUTORATEER ACCURACY SCALING
19

---

Published as a conference paper at ICLR 2026
Experiment I: Chatbot Arena
Coverage
Mean-squared CI width (frac. of classical)
Meun-squeer Rleun-squared Error (frac. of classical)
Malravt
90%
0.8%
90%
83%
Budget
Budger
Budgot
Experiment 2: ProcessBench
Coverage
Mean-squared CI width (frac. of classical)
Mcanssqareed Error (frac. of classical)
809
Pffiee Vecker
.00
839
40%
80%
Budget
Bodger
Budget
Experiment 3: Factuality
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Error (frac. of classical)
95%
80%
agrat
90%
90%
PRt
70%
Pitee Soaler
Pitee Stalor
8.%
60%
80%
70%
30%
Budga
Budge
Rudlet
Figure 6: Results given only N = 10 labeled examples. Results are shown by budget for the experiments on Chatbot Arena (top), ProcessBench (middle), and Factuality (bottom). For each estimator (all baselines and MultiPPI, the left column plots the empirical coverage of the 95% Cl. the middle column plots the width of the 95% CI, and the right column plots the empirical mean-squared error of the point estimate.
20

---

Published as a conference paper at ICLR 2026
Experiment I: Chathot Arena
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Error (frac, of classical)
98%
96R
90%
94%6
948
92%
90%
Bodget
-10
Bedget
-10
Braggtt
Experiment 2: ProcessBench
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Error (frac, of classical)
Mediss
98%
90%
80%
00%
Fffine Victer
70%
70%
80%
60%
92
5016
50%
009
Dodget
Dodget
-10
Brdget
Experiment 3: Factuality
Coverage
Mean-squared CI width (frac. of classical)
Mean-squared Error (frac. of classical)
91016
889
Jagers
96%
905
Fffi-e Scalor
Scala
Fffi- Scalir
Fft-- Srabir
8d9
80%
73%
00%
Bodget
-10
Budget
Budget
Figure 7: Results given N 1000 labeled examples. Results shown by budget for the experiments on Chatbot Arena (top), ProcessBench (middle), and Factuality (bottom). For each estimator (all baselines and MultiPPI, the left column plots the empirical coverage of the 95% CI, the middle column plots the width of the 95% CI, and the right column plots the empirical mean-squared error of the point estimate.
21

---

Published as a conference paper at ICLR 2026
Experiment 1: Chatbot Arena
Hutmm 1300
Ie40-- www.-aaaavvv.... um 1924 CECANEE
25%
40%
20%
50%
50%
3230
2303
3000
12.00
5002
12000
2002
+003
2.00
1009
2000
2000
R900
Experiment ProcessBench
25%
Poutces 2310 5040
Conergge
401
Pauttinn 1100 aodi
32.0
2300
3000
12000
5003
22000
2003
+003
2000
1009
2000
3000
6000
S000
Experiment 3: Factuality
Hution mnat
30%
42%
PaLtton 1570 5240
Conergg
10%9
72%
Mutoon 1200 andi
35%
32.0
23000
3000
43300
5003
-200
2003
4003
3000
1009
2000
3000
4000
300
Figure MultiPPI with varying number of fully labeled examples.
22

---

Published as a conference paper at ICLR 2026
Experiment 1: ChatBot Arena
Coverage
Mean Mean-squared CI width (frac. of classical)
Mcan squared Erro (frac. of classical)
MuluPP (Labe Woll
(Empere))
89%
90%
00%
80%
0336
83%
889
200
400
600
800
1,000
200
400
6000
1,000
200
400
600
800
1,000
Budget
Budget
Experiment 2: ProcessBench
Coverage
Mean-squared CI width (frac. of classical)
Mean-squee Beun-squared Error (frac. of classical)
70%
(Ladon Wal)
(Imppest)
99%
60%
00%
50%
1,000
1,000
800
1,000
Budget
Experiment 3: Factuality
Coverage
Mean squared CI width (frae. of classical)
Mean squared Eroor (frac. of clussical)
705%
80%
(Eaaiczl
999
722%
787
70%
600
1,000
200
400
800
1,000
200
800
1,000
Budge:
Budges
Budget
Figure 9: Comparison of results with different techniques for covariance estimation, for N 50. We find that Ledoit- Wolf shrinkage covariance estimation yields best performance in all regimes.
23

---

Published as a conference paper at ICLR 2026
Experiment
Coverage
Mean squared CI width (frac, of classical)
Mean squared Error (frac. of classical)
100%
90%
MahiP9i
ManiPy (Roericad)
91191
S1408
80%
933
835
400
600
800
400
200
800
1,000
Budga
Bodge
Rudge
Experiment 2: ProcessBench
Coverage
Mcan Mean-squared CI width (frac. of classical)
Mcan squared Error (frac. of classical)
MabiP91
70%
MabiP01 (Recriced)
89%
60%
.0%
50%
50%
200
400
600
800
1,000
200
400
600
1,000
200
400
600
800
1,000
Experiment Factuality
Coverage
Mean-squared CI width (frac, of classical)
Mean-squared Frror (frac. of classical)
1000
76%
80%
MahiP9 (Reviica))
74%
78%
70%
68%
70%
639
400
800
1,000
Budge
Budge
Budget
Figure 10: Comparison of MultiPPI for I 22......k) (default settings) with MultiPPI (Restricted), as defined in Section D.4.
Patamaanc seuured Cheder miikaa
6.00
8:mm
210
Figure 11: Performance at determination of process error word budget. This is calculated via the procedure described in Appendix I. The majority of the improvement observed due to thinking occurs once 500 words of thought is reached, and plateaus around 1,000 words of thought.
24

---

Published as a conference paper at ICLR 2026
Performance determination process word budget
94%
Ascuracy
92%
90%
500
1000
1500
2000
2500
3000
Word budget until demanded
Figure 12: Performance at determination of process error vs. word budget. This is calculated via the procedure described Appendix I. The majority of the improvement observed due to thinking occurs once 500 words of thought is reached, and plateaus around 1,000 words of thought.
Accuracy
Reard
Fourd: 2uae
Baar)
Rour
85%
75%
11%%%
80%
74%
10.4
37%
72%
3.56
Mumber agents
Kumber agents
Mumber agents
Figure 13: Performance at factuality evaluation with increasing number of agents and rounds of debate. Soft accuracy awards half point reporting an uncertain answer, while hard accuracy awards nothing.
Proportion of uncertain predictions
Round 1
18%
Round
15%
12%
10%
Number of agents
Figure 14: Proportion uncertain predictions by number of agents and rounds of debate. An creased number of agents leads to fewer uncertain predictions, and almost all predictions are certain by the end of the second round of debate.
25

---

Published as a conference paper at ICLR 2026
95% confidence intervals
Multi-PPI
Other detiading beneins
0.4 0.6
0.8
0.2 0.4
0.5
1.0
0.2
0.6
0.8
1.0
Figure 15: Different schemes for evaluation with autoraters on the ProcessBench dataset. Gray: clas- sical sampling-Be autoraters. Orange: pure autoraters, in decreasing order of thinking budget- note that the bias is increasingly pronounced with thinking budget. Green: various schemes for debiasing autoraters, including MultiPPI (top).
26

---

Published as a conference paper at ICLR 2026
E ADDITIONAL THEORETICAL RESULTS
In this section, we provide additional theoretical guarantees on the MultiPPI estimator. First, we establish finite-sample bounds on the estimation risk, under bounded, sub-Gaussian, and auto- regressive data distributions, and extend these results to the Ledoit-Wolf shrinkage estimator. Next, we characterize the behavior of the estimator the boundaries of the budget spectrum. We prove that in the extreme low-budget limit, the allocation deterministically isolates the single model with the best correlation-to- -t-cos ratio; conversely, in the large-budge regime, we show that the contin- relaxation of our discrete allocation problem is asymptotically optimal. Finally, formally explain the empirically observed phenomenon of coverage decay, which occurs when the unlabeled budget grows while the labeled sample size remains fixed.
E.1 FINITE-SAMPLE BOUNDS
We consider the setting of Appendix B, in which we may have several budget constraints. For the time being, we fix (1, O.... 0) in all experiments. Let I" I contain 1. A procedure which is similar to classical sampling is the following: Consider the choice n² A defined such that n = if * 1", and let n.o be the maximal choice afforded by the budget (i.c. n). =
maxigesm ( /0/0 ). Then setting X9 = 0 + 19, and Alo to be a restricted to 19, we
recover the classical estimator
x (
which has MSE o// where of E11. We let a := /n. denote this quantity.
We will compare OManpPr to this finite samples. Let EN denote the empirical covariance matrix constructed from N i.i.d. samples from P. and let ñ. Å denote the solution to MultiAllocate(Ew)) iee. the minimizer of
RN(n,
Σ
EXTENAI
n1
IEI: >0
such that U and hold. On the other hand, let n2", A denote the solution to MultiAllocate (E), i.. the minimizer of
1 AEA
R(n A
Σ
n1
IEIm)
such that U and B hold. In this section, we bound
R( 2 R(n", X*).
Theorem 5. Let Twin denote the minimal eigenvalue of E, and 5 < Ymiu/ 2,,
E ENllop Then for all
R(N 2) < R(n", .2) +
o Jelussical
Ymmn
Corollary 6. Suppose that X, [0 almost surely. Then with high probability,
1/2 Mmax
k l
k N
R (n) t
o classical
Ymin
Ymin
for a universal constant c, and so
1/2
ER 2 < R(n a2,
Timax
1
Claassical
Yuni
N
Tinin
N
for another constant é', where the expectation is taken over the N labeled samples used to construct E.
27

---

Published as a conference paper at ICLR 2026
Corollary 7. Suppose that X is subgaussian with variance proxy K. Then
(V k N
ER A) S A*)
a
eelusiccll
N
In the AR(1) model, and with bounded observations, choosing N > in the limit N + is enough that ER(0. A) + R(m", A.).. This follows as a special case of the following result. Corollary 8. Suppose, in addition to the conditions of T'hhorem 5, that X1, X is stochastic process such that Var Xt > for all t, and Corr(Xt; X,) S (1- p)pl l-s for some < c, < 1. Then we have
ER(G, 2) = R(n, 2) + o(1)
whenever = o(1)
Lastly, below we include finite-sample bound for the performance of the estimator using Ledoit- Wolf shrinkage.
Theorem 9 (Fipite-sample bounds specialized Ledoit-Wolf shrinkage). Let SLW denote the Ledoit Wolf shrinkage estimator of Σ based N samples. Let Ymin denote the minimum eigen- value of Σ. and suppose that X R sub- Gabb-Gaassinn with proxy K. Lastly, suppose that Σ is not multiple of the identity. Then for absolute constants c₁ C2 have
E [((AttuAPP))))) 00)
4 classical
1
S VB
max k2 + C2K "max /
7min
VN
where a² E- trE))
For proofs, see Section F.4.
E.2 BEHAVIOR OF THE ESTIMATOR IN THE LIMITING REGIMES
In this section, we explain a certain limiting behavior of the estimator in the regime of very low budget. Let X = (X (X,,..., X₂)) be a random vector of bounded second moment. We take (1,0,.... 0), so that our target is R[X1]. We consider the setting (as is the case in all experiments) in which (1....,k) U moddlss where for each 7 mmodels we have 1 g T.
As in the experiments, we consider the budget model in which we have fixed number of
For I Imodeess Pr denote multiple correlation coefficient of X, with XI; that is, let Pr Cov, Σ, -C-v, where we define Cov, == (Cov(X, X1))ser The following result shows that, in the low-budget regime, MultiAllocatee Σ) returns such that the only T E Tmmodets for which n, * the one which minimizes the correlation/cost ratio pI/c1.
Theorem 10. Fix B > O and consider the limit as [2(k] + c0. For each I E I, let aj == PI/CI. Suppose that T" uniquely minimizes over E Tmodels- Then the solution to MultiAllocate(S) + satisfies
B
[1 1-
n1
CI
0 1+1
E.3 ROUNDING IN THE LARGE BUDGET REGIME
In this section, consider the suboptimality of the rounding scheme in the large budget regime. We consider the general setup in which we optimize
VB(n) aa
s.t. nı 2 0, Σ cmni S B, supp(a) C U ny 0
We let Dine denote the solution to this problem over all n R [z] Rg0 and ni denote the solution over
all 2/4
Let Uround denote the component-wise floor of Ri Here we show that
Vn(
lim B-000
VB (n())n)
28

---

Published as a conference paper at ICLR 2026
This follows from the fact that
V I (D(m S (Brund)))
and the limit VB ) 3/V Tre monnd
(Bruna) - 1, to be proven next. Consider the difference vector
[0, 1] Now observe that there is some D² ER such that
( = Vi (k*)
for all B, and equality holds if we take Lir By" In particular, since must have U(7
I > 0} 2 supp(a), we may take the same to hold for v² We therefore have
BVB (Pruns)) = Ba¹
B
BEUIPT
MJ P1+ 1ipp7 P1
=a a
[uIPTETIPI+
Now since U(7 vi > 0} 2 supp(a), we may apply continuity of the inverse to conclude that
lim BVB (Broudd) T EuppTE P1 a Vi(2)) B
and the limit is proven.
E,4 THE CONTINUOUS PROBLEM AND ITS PROPERTIES
In this section, study several properties of the the following continuous version of Equation (7), stated below:
min 220 Ej SB S(y), [
S(y)
Σ
E
(9)
VEI
where y may vary over I 2 Here, as is the case throughout, the inequalities y 2 0, E, aiG S B
should be interpreted in the vector-valued sense (see Section B).
We are interested in the (set-valued) minimizers of Equation (9). More specifically, we are interested the continuity of these minimizers Σ. and how this relates to the discrete optimization problem of Equation (7). To address these topics, we introduce the notation, following Boyd & Vandenberghe (2004), that S denotes the of symmetric positive definite i matrices. We are interested in the set-valued solution to the minimization problem
(EuAAt) a [
2e (A)
argmin u420 Σ, Sic supp(a)
(10)
for A E + More formally, recalling the set of feasible allocations K = (2 12:02 0, E, Ujes S
B} C RIZ we have
w² : ++ = K
(11)
+ argmin Flg
where
T EErIA a a range (EA)
Fly. A)
(12)
ao
otherwise.
We now connect the continuous problem to the discrete problem. Below we recall the discrete problem: define
a Σ mmat
n (A)
argmin me Σ, Stor supp ( [U
(13)
Then have the following:
29

---

Published as a conference paper at ICLR 2026
Lemma 11. Suppose that y² (A*) {2²} is singleton and that A(N) A". Then for every é > O, there is some N1 so that
nt
vo <c
whenever E n2 (A(N)) and N, 2,,224 Ni.
For the proofs, see Section F.
E.5 DECAY OF COVERAGE IN THE LARGE BUDGET REGIME
In this section, we discuss the phenomenon of decaying coverage - co. Note that this is not unique to MultiPPI: it can be seen occuring to all baselines we compare to, and is especially pronounced for PPI++ vector. After discussing the phenomenon, we describe one way avoid
Since, to the best of our knowledge, this phenomenon has not been observed in other works con- cerning PPI++, we focus our discussion on the PPI++ estimator and explain why it happens in that setting. Recall from Equation (2) the PPI++ estimator
Öppier Σ
Y: Y Xx
Σ XX
N
where {(x,, Yi)ssn are i..d. according to some joint distribution P, and (X, Keh are i..d Px- Angelopoulos et al. (2023b) (as well as many works before, in the context of control variates) pro- pose a choice of Ā which depends on {(X,, Yi)isn namely, they let
Cov(Ximn Yim)
n + N Var(Xi:))
the relevant empirical covariance and variance computed from {(,,, Y,Jiinn This choice introduces bias in finite samples, and MultiPPI exhibits a similar behavior, discussed in Section 4. In the limit theorems provided this work, c.f. Theorem and in Angelopoulos et al. (2023b), it is assumed that the number of labeled samples (here, denoted n) tends infinity. But this is not the situation presented in our experimental results.
where Cov(X (1:s) Y1:n) and Var(X
Here we consider the bias of pppu+ for fixed n as N + co. This bias is exactly
Cov (xi, Cov(XI:ns Y1s) Var(X1.0)
(ias(()))...) B[0ppr...] E[y] B - X()I
N n N
X1,
by independence of N and X, Now for fixed and N + co, the right-hand side converges upward precisely to the covariance of X1 with the sample regression slope of onto X, which not in general zero. Therefore, the bias will increase but stay bounded N - observed.
Note that this analysis does not apply to the setting in which the ratio N/n bounded. We find, accordingly, that this decay is unobserved in our experiments in which the number of labeled samples constant proportion with the budget.
30

---

Published as a conference paper at ICLR 2026
F PROOFS
Unless explicitly stated otherwise, we prove results for the generalized setup outlined in Section B.
F.1 PROOF OF THEOREM 2
For E. R symmetric positive-definit, let Ps: denote the set of distributions on TR* with covari- ance Σ. For fixed collection of index subsets I with associated costs let θB denote the set of budget satisfying estimators 8, i.e. the estimators which are measurable functions of nu indepen- dent copies of X, (X,))ez for each I, such that (nn) holds. Note that here we allow for multiple budget inequalities, as described in Section B, and so (n) denotes the proposition that all budget constraints are satisfied simultaneously. We emphasize that we make no explicit restriction to linear or unbiased estimators.
Theorem 12 (Minimax optimality for general budget constraints). We have
inf sup E (Ó 0²)3| = Var (OMMuii-alooaacee = VB
h PEPs
where the variance is with respect to any distribution P E Pst
Proof. We first reduce to the case of known and fixed n.
Lemma 13. Let ((m) denote the set of measurable functions
funcioon i which are functions of n, independent copies of X1, for each I I. Then supp(a) _ U(T
0},
inf sup deo(a PEPs
E (0- [ 0-)
min
Σ
1 -AJE
(0- (0- 0-)3 unbounded for all 0 E QB.
otherwise, suppePs
We now reduce the conjecture this lemma. Observe that
OB = (n)
n B(z)
and so the left hand-side of the conjecture is equal to
inf inf sup E (0-00) n B()) (ee() PEPs
inf B
min Ul
Σ
L -xTE,Ar =: Var Var(OMiutiaallccae((())
1:n
since U(n A) is feasible for if and only if supp(a) CUU ny > 0}.
It now suffices to prove the lemma.
Proof of Lemma 13. The claim that supreps (0 80)2] is unbounded for all E θ₂
supp(a) & U(I : nr > 0} follows from observation that if E supp(a) UUI ny > 0}. there exist distributions P E Ps such that 0: = E X,] may be made arbitrary large, while 0 cannot depend such X,.
Therefore, in what follows, we assume sapp(a) S U(I: : nı > 0}. The upper bound is clear from that fact that
(0nA: Unn C (
ie., the set of unbiased linear estimators depending on n samples is a subset of the set of all esti- mators depending on n samples; and from the fact that Var(f = EI=,,o ny. AJ Σ, for every P E Pr. hence the minimal MSE of such estimators is precisely the right-hand side.
We prove the lower bound. Since the Bayes risk for any prior μ lower bounds the minimax risk, it suffices to construct sequence of priors μ for which the risk of the Bayes estimator tends upward to claimed lower bound. Let choose the distribution X ~ NM,, E), and supply the
31

---

Theorem 14 (Generalized asymptotic normality). Suppose X E Ra has finite second moment, and suppose that E Cov(X) satisfies Equation (14). Suppose that E 2g E. in the operator norm as Published as a conference paper at ICLR 2026
prior μ1 (O IIda) for 7 0 arbitrary; we will later take - co. Note that we then have X = (P
By construction, any estimator & E (()) depends the independent set UJeztXy hsiss each x is distributed according to (u1, Er). The posterior* is then
hissgn where
Ulxi (m,,,
IEI
S, = -(peld: + Σ
En
TEI
m, = St E Σj
The Bayes risk of estimating 0 = a. is then Sta. Letting minimax risk least
we have shown that the
a Sa, Sa, 5= ΣmPPTE
TEZ
It remains show that this risk is achievable by the And for choice of À satisfying U(n, 2)- We quickly verify this below:
Putting
A En) Sa
we see that indeed U(n, A) holds. Moreover, we calculate
Var(On.)= Σ nia T S P,Sa = aTs
Enjol
Σ
n1PT Mj '
Pi
Sa TTSa Sa
I=n;30
as desired. This concludes the proof.
F.2 PROOF OF THEOREM 3
We prove generalization of Theorem in which we allow for multiple budget inequalities.
Fix vector By E R7 We consider the limit in which our budget is Bo and let oo.
Suppose that Σ 444 Σ in the operator norm, potentially dependent on the variables sampled X,.
We assume the following condition: Suppose that the following problem has unique minimizer 2::
= (EMPPEE
n" arrmmnn, V(v)
(14)
220, Eveer S Bo. supp(a) C U(T:D>>0 0)
'Morally, done this point: the posterior mean is linear in (X,), and the Multi-PPI estimator is the best such linear estimator. However, this does not yet directly imply the result. See next page for calculation of the posterior.
the assumption that supp(a) Σ, n. PjT Σ, Pt.
Heee
UtI nu 0). and thus lies in the range of
"To find this choice organically, one may solve infimal convolution with Lagrange multipliers.
32

---

Published as a conference paper at ICLR 2026
Let MutuiPrus) denote the MultiPPI estimator with budget tBu. Then for 0 MultiPrr(S) arbitrarily dependent any potential samples used estimale E, we have
Vi (0 (MMAPPP(()) -0') 4 V")
as -> oo, where V limB-ro BVB, and VB defined in Equation (7)-
While MuNUPP)(E) is minimax optimal in the setting of fixed and known covariance Σ, in general not efficient, and the variance V can in general be improved by slowly concatenating onto X nonlinear functions of its components.
Proof of Theorem 14. For fixed i, we let allocation and weights Ả be chosen to minimize the variance of = 6 MumPPr(()) under Σ subject to the budget B = tBo, as in the MultiPPI procedure. Since & Σ, Lemma 11 ensures that Ω/t 4 2z" defined in Equation (14).
We can write vi(0 0*) = ErEr \7 V Rr Wr,n,, where Wí,,, 1 vhr Ee (x((((( 11). For
indices with vs > O, we have ñ, & co. Define nj = [tv/], and let
Wi
(1 - -11)
2=1
It is now enough to show that Wi,,, W7 2 0, and this will follow from Kolmogorov's inequality.
To simplify notation, let us focus on a single subset 1, and define Y, = -x(13) ur. Let us also define Sm اド= Yj. We must show that
S4
Sn 40
Vn
where we have dropped dependence on for convenience. We decompose
Sa
Sn
Sa
Sn Sa
Vn (1 /n/n"
Vn
Vn"
Vn"
A
B
Fix 8 We work on the event Et(t) = {/á n²1 S 8t), which holds with high probability.
We first control A. On Es(t), Vn*|A| is sum of at most ōt iidd. copies of Y,. Kolmogorov's inequality then yields
8t + 2nn I 4
(AA
because n² = [tv"] Taking 8 - 0 yields that A 2 0.
We next control Working again on (t),,, we have
SA
Sn.
S4 Sn
V1
n"
O1(
Recognizing the second term A 2 0. and the first term as tight by the central limit theorem, conclude that Sa/Vī is tight. Now we conclude that B E, 0 because îinn" 2, 1.
Having proven W7,a Wi Fg O. we conclude that
Σ Σ (Ai)
vi - 0*)
(x(1) pr)
- (1)
1=770 1=1
But this is precisely the desired result, since this the solution to the continuous optimization problem, and we are done.
33

---

Published as a conference paper at ICLR 2026
F.3 PROOF OF THEOREM 4
Proof.
The proof is immediate from Theorem 5, proven in Section F.4.
F.4 PROOFS FOR SECTION E.1
F.4.1 PROOF OF THEOREM 5
Proof. We have
R(2, A) R(nn,'n))) = RRs, 2) - RN (fz, 2)
+ RN 2) (nvnn,,1'
(15)
50
+ RN'n" :2) 12(m",77
and so suffices to bound [R@ 2) - RN(ū, (ū, 2) and [NN(t' (n", ,') R(n", Ar), Define
A Nv (n. (n, A) = |R(n A) (x A)
Σ
1 ( ENN)
(16)
IET:n
< Σ EN
Σ n
1
IEII,>0
Now since n² A satisfies U and B, we have
R(n" X²) s 21"
(n,)) Rn(n",,
from which it follows that
/
Σ
= (At) E(A)) 2 Tiin (2)
Σ
||1i11
(17)
nl
10Ini>0
IEI:n >0
and similarly
27pnpp 2
Σ
1 A 2 Ymin (EN)
Σ
1
1/r
TEI
IEZ:
where Tmmi (A) denotes the minimum eigenvalue of the matrix A. We deduce that
Σ
|
< En
n2
1o Ymin
(
IEZ 30
EN
En 5 (7min (E) - 6)
Σ
1
10 (7min (E) 8)
0
JEZ:A,>0
by Weyl's inequality, where we let 5 =11E= Ev Coupled with Equation (16), we have
Σn
AN(n', 22) < 6
no. 7min (E)
Ax i. < 8 Σn +
n (Tmin (Σ) -
Taken together with Equation (15) and the definition of A we conclude that
8 o
R(, 2 R(n', ,2) + 4
Mmin n
for all 8 < 7min /2)
34

---

Published as a conference paper at ICLR 2026
F.4.2
PROOF OF COROLLARY
Proof.
This follows immediately from Theorem 5 and Corollary 6.20 of Wainwright (2019).
F.4.3 PROOF OF COROLLARY 7
Proof. This follows immediately from Theorem 5 and Theorem 4.7.1 of Vershynin (2018).
F.4.4 PROOF OF COROLLARY
Proof.
This follows immediately from the Gershgorin circle theorem, as Etys Cov(X,, Xs)
<
()) Var(X.) < and so Amin (E) is bounded below for all k. On the other hand, Amax (
bounded above account of the same argument and the assumption that X; are bounded.
F.4.5 PROOF OF THEOREM 9
Proof. The result follows immediate from Theorem 5 after the following lemma.
Lemma 15. Suppose that Σ is not multiple of the identity, and that X E Be is sub- Gaussian with proxy K. Let "/max denote the maximum eigenvalue of E. Then the Ledoit- Wolf shrinkage estimator SW satisfies the bound
E
N
1l
S
VN VoKK Pmakk2 + Ky k3 //2
where a² = R
Σ I
Proof. Let Eny denote the empirical covariance matrix. Recall that by definition
SLW (1 - 8)Ê + 8TLT
(1-
where i tr(En)/k: k. and 8 = b2/d2; we have b² EI El and d2 a² + b and and d are such that + b and d > quartic mean. Our strategy will be to employ the observation that
WS
Σ1
(1 θ) Σ) = ml)ll
< (11- f1ī Σ|| + +1111 ml (F)
< <61 - E1 + 4831/ mm113
using the coarse bounds that [1 bound E Σ El and ES²
81 s 1, 81 < and (u w)² < 2u + 2n?. It therefore suffices to
Since X is
with proxy K, ΣN satisfies
K4
Σ - E1E S
7max (k2 + k)
N
by Wainwright (2019). This provides a bound on 62: the estimator is (after truncation) a average of N i.i.d. quartic functionals of X of the form ||XXT 1 each of which have finite second-moment bounded by cK 8 mmac h2 by the sub-Gaussian assumption.
We conclude that we may bound
K4 Ymaxk.
E S
N
We proceed by cases bound E88 On the event (dt2 a²/2), we have < 202/a², /a², so will suffices to bound the probability that dd < a²/2). Since d² is again an average N i.i.d. quartics in X, each of which have second moment bounded by ck &
" k2, we have
K8
Bid - -2-22
T
4
N
35

---

Published as a conference paper at ICLR 2026
We conclude that by Chebyshev's inequality, we have
P(d S 22 z c" K
a maxp?
Lastly, since 0 S <
(since is truncated by d), we conclude that in all cases
26
82 ãōs a² + (d8
and so
E8 < a
RB2 BB
2
1 P(d S /2 - N
(
c K
/ h.2 mmax a4
Ymax a
Taken together, we have shown that
13 miaxk" + +cck S
BIJE
Eli S N
a²
desired.
F.5 PROOFS FOR SECTION E.2
F.5.1 PROOF OF THEOREM 10
Note: For the purpose of this proof only, we slightly change notation, letting denote the number of laheled samples rather than n. This just has the purpose of clarifying the potential conflict with the notation 7.s-
Proof. Let us introduce the notation that P1 is the orthogonal projection onto coordinates I, and thus PJ \5 shares its values with \ on coordinates I, and is elsewhere. As a result, note that we have required
Σ P Tm
Now we aim to minimize
1
(o) 24 Cov +u T Ep)
Σ
m
Imi
n1
or, expanding.
1 V(n, A) =
of 2 Σ Ao Covi+ +
Σ ATENND + Σ LATEEA
+E 1m1 ns
We are interested in minimizing (nn, X) over all À (by which we mean (At)rez) and n satisfying the budget constraint Σ, cini S C. We will first minimize over À for fixed n: define ((n) mInA (nn, A). But
(11 1 En, 1n120
En m >0
Cov Ing 20
+ o3
A)
A-2AT
S1 1n mn 20
E,, 1a,0
Covi, 1 >0/
m
a quadratic form X, where we define E,, = Eii)iir = P.EPJ.
This is of the form
V(n, = A (3 S1 + S2 A - 2-2 TT +d
where
1 E 1
s = (Enutmmmpoo)
S2 = blocklookdar
ny20
T =Cov 1Ry>0)
a1
16I
36

---

Published as a conference paper at ICLR 2026
and d is constant in A. is known that the minimum value of such quadratic form is
( m
T (n) min ((n, 2) m
S2 T.
This is because T lies in the range of 1 S1 + S2. To see this, let us introduce the notation that I+ = (T E I: ny > 0} and let I be its complement. Reorder I necessary so that I+ strictly precedes 2 Then S1 S2 takes the block form
(EIJ)r.JeT+ m 0
0)+
diag(Ei
!)
Now, both (,Ir),,Jer) and block (Siii__iaa__i///a /nt)rezi are symmetric positive-definite hence invertible, on the coordinates It, and T' has support in the span of the coordinates Z+
Given the block form shown above, we see that
(1 S S2 (( + block
(boca)) (lock__daa((x//mmrrr)) 0)
again in the coordinates in which I+ Continuity of the inverse is now enough to conclude that
precedes I
lim m mm)(n) block diag (m1 rEz T= - Cove Σj Cov, L(n) m-+0
TET
But now this is linear function L(n) in n. Consider minimizing this in n, subject to the (simplex) budget constraint nı 2 0, E, Ci S C. The minimum is achieved vertex of the simplex, and the minimizer is unique except in the unlikely situation that
Cov ET
Cov, constant in
assuming that Cov, * 0 for some 1.
Now we claim that m² > L(n) uniformly in n subject to the budget constraint. For this, suffices to show that
(Bu)), + bbockk_ooaaeeee (ooocck_daa((((/nneer
+ block block_ddig(S(//n)(
in the operator norm, uniformly in n. The Woodbury matrix identity implies that the difference exactly
block_diag(n1 (I mwlloc_ddagggee /ni)zi (EIJ)I.JeZ)) -
Now, we have < ny I C/a/ for all I E I+ by the constraint. The operator norm sub- multiplicative, and the first factor is bounded in norm by constant multiple of 1/ min, ey. Similarly, we have
mC [+mblock_diag([;]] (Xx))))) I+- (oblock_diag(]jpp (Erz)
mins Cy
The operator norm of the right-hand side goes to oc uniformly in n, so the operator norm of its inverse goes 0 uniformly as well. In conclusion, we have uniform convergence. Therefore, we have
n (m) = argmin, min V V(n, A)
n
m=+200
37

---

Published as a conference paper at ICLR 2026
F.6 PROOFS FOR SECTION E.4
In this section, we prove Lemma 11. The following suffices for most of the proof:
Lemma 16. The map F K x S ++ - RU } is continuous.
From this, we have the following:
Lemma 17. The correspondence v² is upper hemi-continuous. Consequently, (f*) = {r²} is
a singleton and A(N) + A", then for every e >> there is some No so that || - 20 < whennver g. E 1 (A(H) and Z No-
Proof of Lemma 16. We need three warm-up lemmas.
Lemma 18. Suppose that, for I E M1 k is a symmetric PSD matrix which is SPD on the subspace R,, and zero off of the coordinates. Then range(Ere) M1) = RUJ.
Proof. It is clear that E, M, is SPD on UI: since it is symmetric, it is invertible on Uz.
Lemma 19. For all small enough, range vT(4))) range E, vAA))
Proof. Let us take e small enough that V7 2 12/2 > O whenever vf > O. Let us also take small enough that A by Weyl's inequality. Then each A} and (Af)t is symmetric PSD and SPD the subspace R and zero off the coordinates.
now follows from Lemma 18 that range(E,, = span(e: : Unouso !) S span(e; i Unerzo 1) = range 4)4)).
follows from the assumption and this lemma that a rangol Σ, v/-A)).
We have two final lemmas:
Lemma 20. Suppose that M is symmetric PSD and that range(M) = span(J). Then have
T TM =(
where I(J) denotes the restriction of the coordinates J, and M(j) denotes the restriction of M to the coordinates x J.
Proof. Since My E span(J) for y R we have (My); whenever E J Thus every row of M indexed by an element of J is zero. By symmetry, the same is true for every column. Thus M takes the block form
(Mor
!)
M
in the ordered coordinates such that J precedes SS, Now the fact that M is symmetric means that invertible on its range, and so M(J) is invertible, and we are done.
Our last lemma is this:
Lemma 21. Assume that M,E E Rixt are symmetric PSD, and suppose that rango RJ Assume that || < Amiu (M))))) and M + E is invertible. Then if x E range(M), we have
TTM(M=+ (M+ =T,, T
On the other hand, x (range(M) we have
1
| (M + +3)
1
2
38

---

Published as a conference paper at ICLR 2026
Proof. In block form, again in ordered coordinates such that J precedes JG, we have
x 2M +3) x= (26)) ( Er,
M (u.)) E
(M
E(, B
7 ( [Ma) + S] ai()
=
where S (e, denotes the Schur complement. The norm of the Schur complement is bounded ||||| S |=|, as result of the forthcoming lemma. By submultiplicativity, we have
[MM)) + S)-1 Moill -
-
M IS| (M + +
Now the minimum eigenvalue of M(J) + is at least Amia (M(r)) - ||||| 2 Amia (M()) - -11 2 Amin (MinMM(n)) (M)/ = (| 1/2 by Weyl's inequality, and so we conclude that
| +S- 10---M,3 - s 211M))1
as desired, and we are done.
On the other hand, by standard properties of the Schur complement, we have
(Ma.D + Eu.n)
-1
x T (M+ ((JS)
2 (0'))(5)) x(
EJeJJ
where
= (M + /M + (Js,,)) = ((J,,9)
E(rnn)
denotes the Schur complement. We proceed to bound ||S'||
S S ||||| + | + (u..)) - |=| (1+131I 211 Il) S 2|
since we assume the desired result.
|=| S Amin (M.))/2. We conclude that Amin ((5") N 12 establishing
Below, we summarize the properties of the Schur complement used above:
Lemma 22 (Properties of Schur complement). Let N be symmetric SPD matrix, and S denote one of its Schur complements. Then we have
x
Sx
I
1/
y
and, if N is invertüble,
x
x S-1
1
N
y
for all x, y. whence ||S||| & |||]]
Proof. The proof is immediate from equation 7.7.5. of Horn & Johnson (2012), using the fact that the invertibility N implies that S by Schur's formula.
Now we conclude the proof. Let 1 denote Ulw9 I and I+ denote UL I. By Lemma 18 and Lemma 19 have range range(E, o) R C GRL range E By assumption, we have a E I
Applying Lemma 20 to F. we have
EviA) a= ((ZvrA) vna) (11))
F(y, F(v,A)= a
a(rt)
39

---

Published as a conference paper at ICLR 2026
Now write
-E
+3
whence
(I
Σ
+ +
(I+)
(1+)
with |E || < |=||. Now, the first term is symmetric PSD with range R since I S It. Thus by Lemma we conclude
(Eo(An))
((E,,))=(u(,,00 52
|
(1t)
and we are done.
F.6.1 PROOF OF LEMMA 17
Proof. The result follows immediately from Berge's theorem after an appropriate compactification of the domain. Let Flv, A) represent the composition homeomorphism Rzo - R20 U {oo)
with F, e.g. Flv,
exp( - (F(,,, We may then alternatively write
(²(A) = argmin,ex Flv. A)
where, importantly, Fly. A) is now real-valued.
It remains to show that f is jointly continuous in (y, A); Berge's theorem will then imply the desired result. This is equivalent to showing that F is continuous, and so Lemma 16 suffices.
Finally, prove the required lemma regarding convergence of the discrete problem to the continu- ous one.
Proof of Lemma II. Suppose to the contrary that there is some sequence Na, la of arbitrarily large N, E so that
n 0
2
ta
for some E ni, (A()))) for each a. By the preceding lemma, we know that there is some No so that we may assume ||2- - c/2 whenever N > No. for all n² (A(N)).
For large enough therefore, we have ||2 - 20ll < c/2 for all v n² (A(N)))) By the reverse triangle inequality, then, we must have
n
2 e/2
ta
for all large for each n E mi_ (A(Na) and V² (A(Na)).
Since every such n/to contained in the compact set K, we may find a limit point y E K so that for all 5 > 0, there some a so that |nnta - e'll < 8 for some n E ni, (A(Na)). For this x' we must then also have
| ml 2 e/2
for each v E (A(Na)).
We now argue that in fact // (A*), which will serve contradict the assumption that v²(AA (AA" singleton. Recall that every n ni. (A(Na)) is optimal, and therefore minimizes F(n, A (NN))
over the set K, Z where
Kg = D:v 2 0, Σ Vici & tBp
40

---

Published as a conference paper at ICLR 2026
Since have F(ry, A) 1.2 F(2, A) for all 0, it is not hard to see that if is a minimizer of Fl., A) over K₁, then v/tt is a minimizer of A) over K. Let us define
Bi round (A) round (tu"())
in the set-valued sense. It follows that round (A) S Ke n 2
for all A, and
F(n, A(Na)) < F (Iround: A (N.))
for all n E ni, (A(N)) and Bround E Ti roumd (A(N.)). By definition, it follows that
m round (A (A(N))) vn
in the set valued sense, and so by continuity of F it follows that
to F [Broand: A ANaa) (Bemmnd/ tas A (Na) - F(ux,.44 F(z,A*)) A²).
We conclude that and so taking the limit along the sequence of a previously specified we conclude that
La F(, ((Ma)) = F(n/ta, A (Na)) S F(L",
F(v. A") s F(v",A")
by continuity of F again. Thus / E (A*), yet v/ is separated from / by at least contradiinn the assumption that (A*) is a singleton. This concludes the proof.
G COMPUTATIONAL CONSIDERATIONS
Here show that the Multi-Allocate procedure reduces SOCP in the case of single budget constraint, and to an SDP in the general case. The proof of Proposition 23 shows that the minimiza- tion problem over π. may be reduced one only over via the Cauchy-Schwartz inequality. This minimization over À is the dual of an SOCP, as shown by Proposition 24, and the KKT conditions hold. This
sup T
where the supremum is taken over all E R& such that x Er1r S o7 for all E I. This SOCP simple to implement in the Python package cvxpy.
In the general case, Theorem 2 shows that the optimal choice of is
argmin Bin)
T
Σ SIPP
IEI
Let us denote
M(n) = Σ ni P. Pt
TEI
so that our goal is to solve
mint
subject the constraints that
a M((n)aa 2
and B (z),, which denotes set of linear constraints n. But this is equivalent to the SDP
mint
subject to the constraint that
MM a
and linear constraints n. Once again, this is straightforward to implement in cvxpy.
41

---

Published as a conference paper at ICLR 2026
H THE DUAL PROBLEM
We briefly recall the setup. Let Σ E g be SPD, let I denote a collection of index subsets S {l,..., k), and let 6₁ a positive scalar defined for every E I. It will be convenient to define, for every E I, a vector X E R We denote the concatenation such vectors by the orthogonal projection onto the coordinates indexed by I, and set Σn PYEPT We define the norm |||||x V Er "ru R this induces the seminorms l/lle, |ıylll on RR, and | ||ll, A. Lastly, employ
E = Iez IR! fl. We further recall that Pj I - !Rll
A A(A EPT
IEI
enforce the linear (unbiasedness) constraint A(2) a, for some fixed + 0 T
Our first step will to show how to alleviate the budget constraint. To do so, we first briefly recall this constraint. To describe the budget, recall that we define
(n1)Iez - and employ budget constraint of the form Eier RJCr S B for a fixed B > O. Denoting c (cI)rez R budget constraint may be written g n < B. With all of this said, recall that our original problem of interest is
z|
V(a)= min 2.A
Σ
1 AEE n1
Σ PT- e
(18)
IEIm)>0
IEZmI
We begin by deriving tractable methods to solve Equation (18). Let us assume for the moment that IR 21 will later construct the final budget allocation by rounding. Our first step remove the dependence we show that the above problem is equivalent to the following:
(a)) min Σ
VerlArlle,
A
(19)
YAA
IEI
We next show that this is equivalent to the dual problem
U(a) sup a y yER*
s.t.
A IET
(IvlE,
s c1
(20)
Finally, this is a second order cone program, and can solved with off-the-shelf tools. After we have shown these things, describe how to convert solutions to Equation (20) into solutions to Equation (18).
Proposition 23. The problems described in Equation (18) and Equation (19) yield the same opti- mum V U
Proposition 24. The problems described in Equation (19) and Equation (20) yield the same opti- mum U.
Proof of proposition 23. We now begin the proof.
(2) S (3): Let AA a. Define n by?
/er /erldrrlEs
ny =
It is clear that g
B by construction, and we have
BV(a) < Σ > Σ Vell EvellAllle, = ( Velllll,,
Tmr20 n1
F:A
VEZ
Thii defined long A # for some this fails then À and AA 0 yields contradiction.
42

---

Published as a conference paper at ICLR 2026
Model collection
Cost
Gemini 2.5 Pro
$1.25
Gemini 2.5 Flash
$0.30
Both
$1.55
Table 1: Cost structure for experiment 1.
(3) $ (2): Let n, À satisfy the constraints Equation (18). Consider the vectors g¹/2 @ n 1/22
(Verar)ez and 1n IEI the product of their squared norms is
IR I The Cauchy-Schward inequality yields that
Σ Eami)
)
)
(Σ
1
Σ T:ny20
Vei Av |E,
Tini30 nn
Now let us define À by d1 A n
O, and Xi
0 otherwise. Then have
AÄ PT'= Σ
PT
1mr20
by assumption, and
2
)
2 S (a)
U(a)² Σ
Voillrrlls
E
Σ VeilArllEs
S5n
and we are done.
Remark 25. Note that in general, many ny will be zero.
0 a
Proof of proposition 24. Let (aa) denote the indicator b
00 x
Then Equation (19) is alternatively written
V(a)= = min g(A) + (A2 AEA
where s(A) = (())())) and gr (At) = /cr /rllE,, We now apply the Fenchel duality theorem.
Note that 'iay (y) = a y. and g" (A y) Ef E, (1llvill,, =A,, wwrll Sey
I EXPERIMENTAL DETAILS
Here we detail the experimental setup used. We do so in two parts: first, we explain the details for generating the model predictions (X2...., Xx) in each experiment; second, we explain the details for constructing the proposed estimator, MUIIPPT and the baselines from such predictions.
1.1 GENERATING MODEL PREDICTIONS
1.1.1
EXPERIMENT I: CHATBOT ARENA
We follow the implementation Angelopoulos al. (2025) to request autoratings from Gemini 2.5 Pro and Gemini 2.5 Flash. See section E of Angelopoulos et al. (2025) for implementation details.
I.1.2 EXPERIMENT 2: PROCESSBENCH
We evaluate our method on 500 samples from the OlympiadBench subset of the ProcessBench dataset (Zheng et al., 2024). Binary labels are determined according to whether or not a process occurred in the given (problem, attempted solution) pair.
43

---

Published as a conference paper at ICLR 2026
In following, you will see math problem and an attempted solution. There may or may not be an error the attempted solution. Your task is to review the attempted solution and decide whether not it is correct. Report your answer "correct" "incorrect" in lboxed
Problem:
Find smallest number SnS that there exist polynomials (1] [_[2) Idots, f (n]S with rational coefficients satisfying
$$
x (2)+ (1w_L00000002220000200020000
S$
Attempted solution:
To find the smallest number VC N),, start by considering the given equation: f= 1(x)2 f (_2 + Viddoss 2_x W). Notice that V x ()) is always greater than or equal to for any real value of VV W). [...] Therefore, the smallest number VQ 1)) is W(looxed (4)1).
Now decide whether or not the attempted solution correct. Be sure to report your answer as "correct" or "incorrect" in 'boxed {). For example, if you believe that the attempted solution is correct, then you should respond (boxee/ccoreet) "boxed fincorrect]" ". You must respond in exactly this format and include no other text in your response. If you include any additional text in your response, you will be disqualified.
if you believe that the attempted solution is incorrect, then you should respond
Gemini 2.5 Pro: [Thinking...]
after B words of thought have been produced
Gemini 2.5 Pro: So, the answer is: \boxed (correct).
Figure 16: Prompt used to generate autoratings for Experiment 2.
To generate autoratings, we use Gemini 2.5 Pro and truncate its reasoning process at various check- points. Specifically, using the prompt shown in Figure 16, we instruct the model think for up 3,000 tokens but interrupt it and demand an answer after words of thought have been produced, for B {125, 250, 375, 500). as described in $5. To elicit a definite judgement at each checkpoint, we provide "So, the answer is:" as the assistant and attempt to extract an answer from the subsequent 20 tokens of output with our template.
1.1.3 EXPERIMENT 3: BIOGRAPHY FACTUALITY
We consider evaluating the factuality of a set of biographies generated by Gemini 2.5 Pro. repli- cate the setting of Du et al. (2023): Gemini 2.5 Pro is asked to generate biographies for 524 computer scientists, and we evaluate the factual consistency of such biographies with a set of grounding facts collected by Du et al. (2023).
More specifically, for every person p E P, we associate a senini-gennrraed biography bo and a set of collected grounding facts FP (ff...., ff.,)) about the person. Following Du al. (2023), estimate the proportion of factually consistent pairs (b", f!") of generated biographies 52 with each of the collected grounding facts f.P Concretely, given the set of all pairs
S = 1 PP) P.
S P
FP)
we target the proportion of factually consistent pairs
#(bb,)) E S: (b. is factually consistent)
#S
44

---

Published as a conference paper at ICLR 2026
Biography Richard Hamming:
Worked on Manhattan Project, contributing to computations on early computing devices.
Joined Bell Labs in 1945, working relay calculators and carly digital computers like the IBM 650.
Developet Hamming codes, fundamental set of error detection and correction codes for digital data.
Introduced the concept of Hamming distance, metric for comparing two binary strings.
Fact: Richard Hamming was mathematician who made contributions in computer enginecring and commurications Gemini 2.5 Pro judgement: Factually consistent.
Fact: Hamminn worked on the Manhatian Projec before joining Bell Telephone Laboratories in 1946 Gemini 2.5 Pro judgement: Facuually inconsistent.
Figure 17: Depiction of biography-fane pairs (b, [) as in Experiment 3. Judgements about factual consistency of (b, /) are made by a language model.
We determine the factual consistency, or lack thereof, of pair (b, f) by majority voting over independent judgments from Gemini 2.5 Pro with thinking. Du et al. (2023) found that judgments by ChatGPT achieved over 95% agreement with human labelers on set of 100 samples. This level of agreement is evidently not achieved by certain cheaper models, as we proceed to demonstrate experimentally. In Figure 13, we explore using Gemini 2.0 Flash Lite as an autorater for evaluating the factuality consistency of pairs (b, E S.
To elicit better autoratings from queries Gemini 2.0 Flash Lite, we bootstrap performance via multi-round debate. For a fixed number of agents A E (1,...,55) 5), and fixed number of maximum rounds (1, 2), we perform the following procedure:
1. A instances of Flash Lite are independently prompted to consider the factual consistency of pairs f) E S, and provide an explanation for their reasoning.
2. A "pooler" instance of Flash Lite is then asked to review the pair (b, f) and the responses generated by each of the A other instances, and output judgment in the form of a single word: yes, no, or uncertain.
(a) the pooler outputs "yes" or "no," the judgment is final.
(b) If the pooler outputs "uncertain" and the number of maximum rounds R has not yet been reached, the A instances of Flash Lite are independently shown their prior sponses, and the prior responses of each other, and prompted to continue reasoning given this additional information. This procedure continues until either the pooler no longer reports "uncertain," or the maximum number of rounds / has been reached. (c) If the pooler outputs "uncertain" and the maximum number of rounds R has been reached, a fair coin is flipped and "yes" or "no" are reported with equal probability.
Since the dataset is balanced, the outcome described in (c) is fair insofar as is good as random guessing. We impose the maximum round restriction to encapsulate our budget constraint. To reduce randomness, we generate all autoratings twice, so that the resulting dataset has an effective size of 1048.
Target:
Proportion of factually-consistent pairs, ((b,)) E S: (b, is factually consistent) Model family: (The output of the above procedure given (A, R) A E {1,..., 5}. R E {1,2}}
Cost structure: For a given (A, R), the cost is A R. For collections of models, the cost additive.
I.2 CONSTRUCTING THE MULTIPPI ESTIMATOR
For the results shown in $6, draw 250 fully-labeled samples from each dataset above. We then follow the procedure described SC.3 for = 250, using the empirical distribution over each dataset our source of randomness. In $ D.2, we replicate the study over a range of values of N.
45


Reviews:
[{"review_id": "MXJjvNdfrr", "reviewer": "Reviewer_bTBc", "summary": "This paper proposes Multiple-Prediction-Powered Inference (MultiPPI), an extension of Prediction-Powered Inference (PPI++) to settings with multiple predictive models of varying cost and accuracy. The authors frame the problem of estimating a population mean under budget constraints as an optimization over which predictors to sample and how to weight them. They derive a minimax-optimal solution assuming known covariance, propose a practical version using an estimated covariance matrix, and provide asymptotic and finite-sample guarantees. Experiments on LLM-evaluation tasks (Chatbot Arena, ProcessBench, factuality benchmarks) suggest improved efficiency over baseline PPI methods.", "questions": "1. How sensitive are the theoretical results to covariance misspecification?\n2. How sensitive is MultiPPI’s performance to errors in the covariance estimate? Could shrinkage or robust covariance estimation improve stability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The formulation is clean and the mathematics appear correct.\n2. Good Application. The use of realistic LLM cost scenarios is interesting for the LLM evaluation tasks.\n3. The optimization framing (SOCP/SDP) is good and potentially generalizable.", "weaknesses": "I don't see much weakness of the paper. The main contribution is clear: MultiPPI is a straightforward generalization of PPI++: it replaces a single predictor with a vector and adds a *cost-weighted* sampling constraint. The estimator remains linear, the theory is a direct extension of standard control variates, and the minimax-optimality proof follows textbook arguments once the covariance structure is fixed. I am not sure if this contributions reaches the bar of ICLR though. I would also think that Cost-aware PPI would be a better name of the method, since the main contributions is not from aggregating multiple ML predictions, but  aggregate them in a cost-aware way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DpIBzO8EuU", "reviewer": "Reviewer_YZU1", "summary": "This paper addresses a critical challenge in modern AI model development: balancing the cost and quality of evaluation metrics. It proposes MultiPPI, a statistical framework that optimally allocates resources across diverse data sources (high-cost, high-quality \"gold\" metrics and low-cost, low-quality proxies) to estimate linear functions of population means (e.g., LLM win-rates, factuality accuracy) under a fixed budget. The framework provides theoretical guarantees (minimax optimality, finite-sample bounds, asymptotic normality) and validates performance across three LLM evaluation tasks, consistently outperforming baselines like classical sampling, PPI++, and vector PPI++.", "questions": "1. Are the golden samples, (i.e., the fully-labeled samples that contain both the high-quality metric $\\(X_1\\)$ and all low-cost proxy metrics $\\(X_2, ..., X_k\\)$), prepared to obtain an accurate covariance matrix sufficient for well estimating the mean of the high-quality score, $\\(E[X_1]\\)$,  thereby avoiding the need to compute the low-cost proxy metrics??  \n2. Evaluation metrics are often used in the model training phase. If the model is modified, can the empirical covariance matrix $\\(\\hat{\\Sigma}\\)$ obtained under the old model still be applied to the new model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The authors provide a complete and well-structured theoretical framework, which is rare in applied AI evaluation work:  \n- Minimax Optimality: When the covariance matrix of data sources is known, MultiPPI achieves the minimax risk lower bound (Theorem 4.2), ensuring it is statistically optimal among all budget-feasible estimators. This anchors the method in fundamental statistical principles, rather than heuristic design.  \n- Practical Guarantees: For real-world scenarios where covariance matrices are unknown, the paper derives finite-sample bounds (Theorem 4.3) that quantify performance degradation when using empirical covariance matrices. It also proves asymptotic normality (Theorem 4.4), enabling valid confidence interval construction—critical for practical decision-making (e.g., comparing model versions).  \n- Optimization Tractability: The authors skillfully transform the resource allocation problem into solvable programs: second-order cone programming (SOCP) for single budgets and semi-definite programming (SDP) for multiple budgets. This ensures MultiPPI can be implemented with off-the-shelf tools (e.g., cvxpy), bridging theory and practice.", "weaknesses": "The Method Rely on Accurate Covariance Estimation\n\nIn practice, MultiPPI requires a set of \"fully labeled samples\" (containing both gold metric$X_1$ and all proxies $\\(X_2,...,X_k\\)$ to estimate the empirical covariance matrix $\\(\\hat{\\Sigma}\\)$. The paper uses N=250 or 1000 such samples, but :  \n- Fully labeled samples may be scarce: Gold metrics like human annotations are often extremely costly—collecting 250 fully labeled samples could be prohibitive for small teams or niche tasks (e.g., low-resource language LLM evaluation).  \n- Cost-effectiveness of estimation: The paper does not sufficiently explore whether an accurate covariance matrix can be robustly estimated with a smaller, more cost-effective sample.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxqyRtFe9G", "reviewer": "Reviewer_LyCP", "summary": "This work proposes a method to estimate the resource allocations for evaluation, e.g., assessing the qualities of model predictions in a task using large language models, under the budget constraint. Basic idea is to formulate the task setting as a gold human label with \"auorators\" by language models, and to trade the number of queries to the models and the expected qualities with the maximum budget constraint. It is now treated as an optimization problem assuming the accurate covariance of labeled samples are obtained.", "questions": "See the comments regarding weaknesses.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This work introduce a task setting to maximize the expected evaluation qualities under the resource budget so that the number of queries to language models could be minimized. It is an interesting yet practical setting especially when language models are employed for evaluation, i.e., LLM-as-a-judge.", "weaknesses": "- Clarity is an issue. The task setting assumes $n$ samples with gold labels by human and $n$ samples without gold and thus estimated by autorators only, i.e., language models, in Equation 3. Further discussion introduces the cascade modeling in Equation 4. However, their relation to the proposed approach is not discussed in section 4. As a result, the contribution of this work is not clear.\n- Given the task setting assumes incomplete labels in term of the lack of human ratings, it is not clear how that is reflected in the experimental settings. It is also not clear what optimizer was used in the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qWbhq0IKiY", "reviewer": "Reviewer_pGiG", "summary": "This paper tackles cost-efficient estimation when a high-quality but expensive signal coexists with cheaper proxies. The method formulates variance-minimizing estimators with constraints. With known covariance \\(\\Sigma\\) this becomes a tractable SOCP/SDP and is minimax-optimal. With \\(\\hat{\\Sigma}\\) from a small burn-in, it retains finite-sample bounds and asymptotic normality. Then, the authors show in experiments that, on LLM evaluation (arena wins, reasoning budgets, factuality), their proposed method achieves lower MSE and tighter CIs, adapting from cheap proxies at low budgets to accurate raters as budgets grow, in multiple settings.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.  A major strength is that the paper frames the complex allocation problem in a manner that makes it solvable using standard optimization tools.\n2.  The paper provides finite-sample bounds on the estimator's performance for the practical scenario where the covariance $\\Sigma$ must be estimated from data.\n3.  The effectiveness of MultiPPI is demonstrated across diverse and relevant large LLM evaluation scenarios.", "weaknesses": "1.  The practical algorithm relies heavily on an a priori estimate of the covariance matrix $\\Sigma$, which is derived from an initial \"burn-in\" set of $N$ fully-labeled samples. The performance may degrade if the size $N$ is improperly chosen or if $\\Sigma$ is ill-conditioned.\n2.  The MultiPPI framework allows sampling from any subset $I$ of $k$ variables. However, enumerating or considering many such subsets and solving the corresponding SOCP/SDP problems can become computationally expensive as $k$ increases. The authors should include a discussion on the scalability of the MultiPPI estimator.\n3.  The finite-sample bounds in Theorem 4.3 assume that $|X_i| \\leq 1$, which is a restrictive assumption. Some evaluation signals (e.g., scores/logits) may violate this bound in other application scenarios.\n4.  MultiPPI is designed to estimate a linear function of the mean. This may limit its applicability for estimating a broader scope of non-linear functions of the mean. (This is noted as a potential limitation, perhaps minor given the paper's focus.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)