Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models
Authors: Adam Fisch
Joshua Maynez
R. Alex Hofer
Bhuwan Dhingra
Amir Globerson
William W. Cohen
Venue: NeurIPS 2024 poster (2024)
Keywords: prediction powered inference
auto-raters
Primary area: learning_theory
TL;DR: We provide an improved extension to prediction powered inference that is based on stratified sampling.

Abstract: Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.  PPI achieves this by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate---but potentially biased---automatic system, in a way that results in tighter confidence intervals for certain parameters of interest (e.g., the mean performance of a language model). In this paper, we propose a method called Stratified Prediction-Powered Inference (StratPPI), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system or data distribution, we derive an algorithm for computing provably valid confidence intervals for parameters of any dimensionality that is based on stratified sampling. In particular, we show both theoretically and empirically that, with appropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than  unstratified  approaches. Specifically, StratPPI  is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.

Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation
Adam Fischt**
Joshua Maynez
R. Alex Hofert
Bhuwan Dhingra
Amir Globerson
William W. Cohen
TGoogle DeepMind
Google Research
joscuhjjoohhah,roofoee bahingra, amirg, attp:/aaaaiiiinnncceeaoo scohen)yggoolle.com
Abstract
reddiction-powered inference (PPI) a method that improves statistical estimates based limited human-labeled data. PPI achieves this by combining small amounts of human-labeles data with larger amounts data labeled by a reasonably system, in way that results in tighter
accurate- but potentially biased-autonee confidence intervals for certain parameters of interest (e.g., the mean performance of language model). In this paper, propose method called Stratified Prediction-Powered Inference (StratPPI), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system data distribution, we derive an algorithm for computing provably valid confidence intervals for population parameters (such as averages) that is based on stratified sampling. In particular, we show both theoretically and empirically that, with uppropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than unstratified approaches. Specifically, StratPPI is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.
Introduction
Evaluating machine learning models requires evaluation data. In particular, iteratively improve on method during development, to reliably report an improvement, one needs to confidently and quantitatively assess the performance of the method. This is especially challenging for large language models (LLMs), where gathering high-quality annotations for generations can be difficult and time- consuming-aae and can ultimately become quite costly if gathering more than a few hundred examples.
One oroen-proposed approach to avoiding the evaluation bottleneck is to use a secondary LLM-based system judge the output of the primary one. For instance, if the primary task is developing an LLM-based question- answering (QA) system, one can use second LLM-based system that rates question/answer pairs as acceptable or not [6; 17]. However, automated raters (i.e., autoraters)) may be biased relative to the human raters they are intended to model, as others have noted [1; 9; 18: 23]. This can become substantially worse when models are tailored hill climb on the autorater metrics, and eventually cease to become good metric at all-a phenomenon commonly referred to Good- hart's law, reward hacking in the context of reinforcement learning from human feedback [13; 24).
We thus have two signals for assessing model performance. The first is human labels, which are typ- ically accurate, but expensive collect. As result, usually only small sample size is available, and
Equul contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

an estimate based on these samples alone will have high variance. The second is autorater predictions, which are easy to collect for large sample sizes. but may also be systematically biased. The above may suggest that one must make a choice between either (i) a high-variance, but unbiased, estimate from a small human sample, or (ii) a lower-variance, but biased, autorater-based estimate. However, turns out there are also statistically valid ways of combining the auto-rater and human data for hybrid eval- uations. Following [1; 3; 7] we call such methods prediction poweerd inference (PPI) methods. At a high level, PPI-based methods operate by using a small sample of examples labeled by both humans and autoraters to estimate the bias of the autorater. This bias is then used as a rectifier for the autorater estimate. The resulting estimate can then be shown to provide improved (i.e., tighter) confidence intervals for properties of interest for the system being evaluated, such as its true mean accuracy.
A weakness of standard PPI, however, is that it does not take heterogeneity into account. For example, in our QA setting, an autorater may have one accuracy when predicting if a model answer is correct, and a different accuracy when predicting if it is incorrect. This is especially true in cases where a correct answer easy to verify (e.g., it also present in Wikipedia), but harder to refute (e.g.. no relevant external search results can be retrieved to either support contradict it). In these settings, it may make sense to apply a different PPI strategy within each subdomain, depending on the local quality of the autorater. Moreover, we claim that such heterogenous settings are to be quite expected in practical applications. Inspired by the rich prior literature on stratified sampling and survey design [12; 19; 21], we therefore propose stratiffed approach to PPI, and show that it can be very advantageous when performance varies across subdomains, for either the autorater, or the model being evaluated.
On a technical level, it is not immediately clear how to apply stratification to PPI, it involves two types of samples: one sample that labeled by both humans and an autorater, and another (typically much larger) sample that is only labeled by the autorater. Extending the analysis of [3], in this work we show how confidence intervals based on the asymptotic normality of weighted M-estimators [32] can in fact be derived for the stratified PPI setting. The next challenge we address is how to determine the sample sizes used for stratification (i.e., the sample size of each stratum). Similar to recent work for active statistical inference [37], we further derive optimal rates that depend on certain moments of the underlying distribution that are generally unknown- and provide an approach for effectively approximating these. Finally, we provide extensive empirical evidence showing that our stratified approach (StratPPI) leads to considerably improved confidence intervals over both classical inference methods that use only human labels, and the baseline (unstratified) PPI approach.
2 Related Work
Prediction Powered Inference (PPI) was introduced in [1] as method for obtaining tighter confidence intervals for parameters learned in supervised machine learning (e.g., coefficients in logistic regression) by also leveraging other model-based predictions on additional, unlabeled data. Related ideas were explored by [30], but with a focus on bootstrapping as a way for obtaining confidence intervals. PPI was then extended in several directions. For example [36] showed how the labeled data can be used for both estimating the parameters and the autorater model. PPI++ [3] showed how to obtain confidence intervals that are easy to compute efficiently, and introduced a parameter for weighting the predictions of the autorater such that the overall statistical efficiency can be improved. As noted in previous works on PPI. these approaches are closely related to other statistical methods for introducing control variates based on autoevaluators [9], as well as augmented inverse propensity weighting [27] (see discussion in [1: 31). Like this paper, prior work has also focused using PPI/PPI++ for evaluating machine learning systems with autoraters, including for ranking chatbots in the Chatbot Arena [7] and evaluating retrieval-aus -uggeentd generation systems [28].
Most relevant to our setting, the recent work of [37] focuses on active sampling of examples to label during PPI, where the total number of queried labels is random, but less than the sampling budget in expectation. Specifically, it proposes to label examples for which the autorater is less confident in its predictions, and corrects for the sampling bias with a variant of inverse propensity weighting. In contrast, our somewhat simplified approach takes inspiration from stratified sampling where a coarse stratification is defined in advance, and the total number of labeled samples is constant. Like [37], we derive an analogous optimal allocation strategy for a given budget, though we only apply it at the stratum level, and not for individual examples. Furthermore, while the variable allocation helps reduce variance in heterogenous settings, our stratified treatment also allows for a stratum-specific extension of [3]'s estimated tuning parameters that further improves performance in complementary ways.
2

---

In terms of stratification specifically, concurrent work [14] also proposed Bayesian variants of several PPI methods, including stratified approaches. In contrast, the methods here do not require introducing priors, do not require running expensive Bayesian inference, and give more conventional, frequentist, guarantees of performance. The credible intervals produced by Bayesian methods are related to, but different from, the confidence intervals produced here, and by other prior PPI approaches [1; 3; 7].
Finally, there is conceptual similarity between the PPI rectifiers and post-hoe calibration of classi- fier. There is a rich literature on calibrating classifiers (e.g. [22; 25]), but there is a clear difference in goals between calibration and PPI: the former is aimed at modifying learned autorater to make better probabilistic predictions, and the latter is aimed at obtaining better confidence intervals by making use of an existing autorater. That said, clearly one approach improving PPI to use setter-calibrate classifier, perhaps by taking some of the labeled data and using it for re-calibration. This is similar in motivation to the cross-PPI approach of [36]. We leave such approaches as future work, but we do experimentally explore PPI approaches on both well-calibrated autoraters and poorly calibrated ones.
3 Preliminaries
We begin by briefly reviewing PPI [1], and specifically the efficient PPI++ variant of [3]. Here, and in the rest of the paper, upper-case letters (X) denote random variables; lower-case letters (x) denote constants, and script letters (XY) denote sets, unless specified. All proofs are deferred to Appendix A.
Following [l: 3; 7], we assume empirical sample S, {(X,, Yi),... ..... (Xn. Yx) of n i.i.d. examples drawn from some unknown, but fixed, distribution P. where X; E X' an input, and Y; E y the target output. We also assume a larger sample S. = ((X)))(X))) N >> n, for which the target outputs f: X + y which provides an approximation of Y given the observed input (we use )" to denote the output space of the autorater as it is possible that y * y", e.g., if f(X) E[y X)).
(XN. N, [(x))) where not available, but we have access to autorater function
As an example, assume X a (question, model answer) tuple where the answer produced by an LLM-based QA system we wish to evaluate, Y is a 0/1 "gold" human rating of correctness of the answer, i.e., 1(human rates model answer as correct), and fX), the autorater, imperfectly predicts this human rating Y. One value of interest is then the average accuracy of the QA system, E[Y). More generally, given any convex loss function (x y) satisfying certain regularity conditions (see Definition A. we are interested in estimating the d-dimensional parameter 0*,
0 argmin E [[,,(X,Y)) l
(1)
In the statistics literature, this is broadly referred to as M-estimation (c.g., see [29]). Here, we focus on mean estimation, for which the loss Eg(#, y) _/2 lyy 0 has the optimum 0* E However, our method generalizes to other typical losses such as the squared loss for linear regression, where the pa- rameters 0* are the regression coefficients, or, similarly, the log loss for other generalized linear mod- els. Of course, the true 8* is generally not known, because we cannot exactly calculate the expectation in (1). The goal of PPI, which we share in this work, is to use both the labeled and unlabeled data S, and S to obtain an asymptotically valid confidence interval Ca., for 07 that for any j E [d] satisfies
lim P(0) E Co.a) 2 a. +000
The simplest confidence interval can be obtained using standard techniques by using only the labeled sample Sn, and ignoring the autorater data Sn. However, PPI employs a clever trick which allows for also using the unlabeled sample Syy to get tighter confidence intervals, as described next.
3.1 A rectified prediction-powered loss
The key idea of PPI-based methods is to use autorater predictions to derive low variance, but unbiased, estimate of the objective in (I). Consider the following "rectified" prediction powweed loss:
N
1 N
L LP (0)
la (x,,))((1))
la Y.) to ((x)))
autorater data loss
loss rectifier
In d-dimensions, obtain confidence interval for each coordinate. See [3] for other possible choices.
3

---

The rectifier term on the right hand side of (3) removes the bias of the autorater data so that P
(0)
satisfies (0)]= E (X, Y) However, when f(X) is correlated with Y, the loss LPP (0) will have lower variance, Unfortunately, f(X) may not be good predictor of Y in all cases. In fact, it is even possible for T,PP to be higher variance than the classical estimate (e.g.. f (X) is anti-correlated with Y). Thus, to adapt to the quality of f(X), PPI++ also introduces a tuning parameter À E R.²
N fo,X, (X.)) + 1 E lo (o(X,, xeo ((x.))
(0)
(4)
Clearly, - still remains unbiased for any value of X. For A 0. the framework reduces to classical inference. In most cases, however, proper choice of À & 0 will result in lower variance. PPI++ also suggests how to automatically choose data-dependent
that converges to an optimal value.
3.2 A prediction-powerst confidence interval
We next describe the PPI++ method for deriving confidence intervals based on the rectified loss. Let
P argmin (O))
BEB
be the prediction- -oweeedd estimate. Standard analysis for M-estimators can then be extended to show that the scaled difference of the estimate 0PP and the true 0* is asymptotically normally distributed. PPI++ then leverages this result to compute a confidence interval for 6* that is asymptotically valid.
Theorem (PPI++, [3]) Assume that x L X and NV 2 0. Let Ha- B[ and
V8-- == X2Cov ((8)), Va.e- == Cov (Vlg - AV (X, f(X)),
(6)
where À E R is hyper-parameter Then under the regularity conditions of Definition A.I, we have that vn(0RT 0*) 4 ((0,E'A), where ) := =Hap'(r (r V2 + V22 )
Corollary (PPI++ CI, [3]). Les Ed be the plug-in estimate for Σd using Sn and Sy. Define
C.P +
# 21 A
where 23 denotes the 3-quantile of the standard normal distribution. Then for any j E [d] it holds that lim P (07 E (CE)) 2 1 o.
As mentioned above, [3] further showed that for appropriate choice of X (that can be analytically derived and estimated via Å), the trace of the covariance matrix, Tr(2*),, is at most that of the covari- ance matrix derived without using autorater data, implying that the PPI++-based confidence set C.P can always be at least as tight as that of the classical M-estimator (and often much tighter in practice).
4 Stratified prediction-powered inference
We now present our approach for improving PPI++ estimates via stratification. In particular, show how optimizing a rectified loss computed via stratified sampling can lead to a consistent, but lower variance, estimate of the optimal parameter 6*, and correspondingly tighter confidence intervals. Consider the QA example from the previous section. For most autoraters, is reasonable to assume that the strength of their performance instance, an autorater might be accurate at predicting whether an answer to an unambiguous question (e.g.. "What is the capital of France?") is correct, but relatively poor inferring if an answer to an open-ended question (e.g., "What is the best way to cook a hamburger?") is acceptable or not. Splitting the problem space into different domains allows to derive more specialized form of the prediction powered loss that can better adapt to this autorater heterogeneity via stratified sampling.
vary, depending the type of input being presented. For
Formally, assume that the input space X is partitioned in advance into K non-empty, mutually exclusive, and exhaustive strata A = (Ai, A Ak), ), where K a finite integer. For each stratum,
2 As noted in [3], for some losses la we must have À E [0, 1] to guarantee convexity of LPP The main focus of this paper. however, is mean estimation with (0 , y) 1 0l1 which is convex for any À E R.

---

we further assume that we can estimate wa = P(X E As)) arbitrarily well using large amounts of unlabeled data or prior knowledge; we treat them as known constants here for simplicity. To collect the labeled and unlabeled datasets S, and SN. we then follow a standard stratified sampling procedure in which we draw two i.i.d. sets of samples of fixed size and Ng, respectively, from each stratum k, where _k=1 ng and Σ k= Nr- N. The relative sizes ng/n n and / are free parameters; they can simply be the natural rates, ng/n = N R Wk, or systematically decided (see $4.3). Note that this is fundamentally different sampling model from standard PPI: here examples i.i.d. within each strata, and independent (but not necessarily identically distributed) across each strata.
Next, we define the stratified loss via the weighted sum, L (0) = = W.L (0), where wy is the stratum weight, À (Λi,.... A&) IR* are now trratum--peciftt tuning parameters, and (0) the conditional prediction-poweree loss computed within each stratum, i.c.,
AE
No
A. (0)
=
Co (K
((K.x) +
Σ (o Vik)
(8)
NO
As before, each L is an unbiased estimate of the conditional loss. Like PPI++ also allow for data-dependent parameters À that we show how to automatically tune for best performance in $4.2.
4.1 A stratified prediction-powered confidence interval
We now present our main result, which is a confidence interval for 0 based on the stratified loss. More precisely, the result states that, in PPI++ the minimizer of the stratified loss,
X
argmin SSP (
(9)
#
has an asymptotically normal distribution with mean 0" See Algorithm for pseudocode. Theorem Assume that Å 2 Av. N + r for any 2 ue for any Đv > O. Let Ha.0- E Eo. (X,Y) x Ax), and
+ pr for any pv > O, and N Pk
kf == Co (Vl. (X, f(X) X Ax), a K- = Cov(V (X,Y) d Vg- (X, [(X)) | X E At),
(10)
(11)
where d E is a hyper- Then, under the regularity conditions of Definition A.I,
va(OspP - 0²) 4 NO En),
(12)
where Et = A BBA'' and:
Aw Σ we H
BA
w
pk k.
r Vd
V K.
(13)
Pk
To obtain the result, we combine unstratified PPI++ with the asymptotic properties of weighted M- estimators [32]. The resulting confidence interval for 6* is then derived analogously Corollary 1. Corollary 2. Let 52) be the plug-in estimate for Ed using Sn and Šx. Define
CSPP a..
=
E GSPP + 21 V
(14)
where 23 denotes the B-quantile of the standard normal distribution. Then for any j [d],
lim P P00
CCSP)) l
(15)
n,,N +00
The form of the stratified prediction powered confidence interval is similar to that of PPI++ except that it is based off of the weighted stratum-conditin covariance matrices. The effect of this change, however, significant. In fact, the case of mean estimation, we show that the asymptotic variance of StratPPI I is at most that of PPI++ (even without any additional tuning of X and pt).
5

---

Algorithm Stratified prediction-powered inference for general M-estimaaooss (StratPPI)
Definitions: I is the autorater. Inputs sx. J(xxxxxx)))))) ) include the autorater predictions on sampled unlabeled data for each partition Ag, k E [K]. Inputs {(Xik, Yike f(Xx)) 1 include the autorater predictions on sampled labeled data for each partition Ak, k E [K]. wy is the partition weight for Ag. & E [K]] is the confidence interval coverage error tolerance.
1: Pick weighting parameters, see $4.2.
2: Select λ == (A,...., xx)
3: # Solve for the minimizer of the stratified, prediction-powered empirical loss.
4: @&PP= argming T. (0)
5: # Estimate the Hessian of the true expected loss 0*
6: A. = E
we
D Velger (Xik: Yu.)
Pig
7: # Build the estimated covariance matrix from each stratified component of the loss.
8: ΣΔ Oaxd
9: for k = 1.2..... K do
10:
V, = >2Covv (Vlggrr (xs=J(XX ((xx)))
11:
PA = -Covnu (Vegerr
( Yik) + ds V Vager (X (Xx)))
12:
SA SA + w Am
V
+
VA
A=
Nx
13: # Return coordinate- wisee confidence intervals for 0*
14: CSPP @SPP + 21
SA "i e [d]
Proposition 1. Let A E R be any constant for all strala, and pv and pa to their natural rates wi. Then for Co(x, v) = -llyy 0122 and any stratification (AI,...,Aks we have Tr(E S Tr where Σd is the asymptotic covariance matrix of PPI++, Furthermore, we have that equality holds and only if both E[Y X Au] and (X) X As) are the same for all strata.
More generally, although our results will hold for arbitrary stratifications, it is best if they are heteroge- neous, and chosen such that the individual stratum- contttiinon variances minimized. For example, an autorater systematically over-estimates the model's performance on one subdomain, but sys- tematically under-estimatee the model's performance another, then splitting these subdomains into different strata can result a lower variance LSPP, Similarly, if an autorater has much higher noise some subdomains than others, it can be beneficial stratify those subdomains and then either lower Ae. allocate a higher proportion of samples n and N₂/N, or both for the noisier stratas. In $5 empirically demonstrate that the stratified estimator indeed lead to considerably tighter confidence intervals in practice, especially with additional tuning of A and Pk. as discussed next.
4.2 Optimal weighting of the autorater predictions
In [3] it was shown that the optimal value of À for PPI++ (i.e., the one minimizing the variances of the estimator and the corresponding confidence interval) could found in closed form. We now present a simple extension of this result to the stratified case. For notational convenience, we use the shorthand Vh Vlo(X, Y' X E As and Vll Vl f(X)) X e A
Proposition 2. Assume that pu and pt are fixed. Then the tuning parameters (M...., ()). where
Tr A (Cov(Ves,or, Vll Cov Cov ))x
(16)
2
Re No
A Cov (V R.0- (V
minimize the cumulative asymptotic variance, Tr(E.)).
Note that Tr(2A) is proportional to the total size of the confidence interval CSPP in (14). Furthermore, as in [3], we can use plug-in estimates for the terms (16) to compute dg, where d 2 Aj. From (16), can see that X is closely related to the correlation coefficient of the (curvature-scaled)

---

gradients for minimizing the loss on an autorater label versus true label. Intuitively, the more correlated these terms are, the more we can rely on the autorater labels for finding the true minimizer of (X, (o,o(XX)) Y) For 1-d mean estimation in particular, we can see that X takes on a simple form:
Example 1 (Ag for mean estimation). Consider the 1-d mean loss: lo(x. v) = 2 0) Then:
Cov(Y, f(X))
Cov(Y.J(X)) for large No:
X = (1+ N2. Var(
(17)
Var(f(X))
which is equivalent to the optimal linear regression coefficient, minx, EJjy Auf(X)) X E Acl-
4.3 Optimal allocation of the sampling budget
Our stratification approach has additional hyperparameter that can also be tuned to reduce variance. Recall that py determines the ratio between the labeled data size ny for stratum is and the overall data size (i.e.. Σx ni.- = n). 2k: is similarly defined for the unlabeled data. Any strictly positive values of Pk and A are valid to be used, though not all values will improve performance. turns out that the optimal Pk values can be exactly calculated, as the following proposition shows.
Proposition 3. Assume that b is fixed. Then the sampling rates (pi..... .Pt) and ..), where
Wk / (TTAA
K.A. 0- A
wk V /Tr(A Vil,
k. Aw.))
Pk Σk=n WW Tr(A VV V
and pk E WK V Tr(A V. Awo))
(18)
minimize the cumulative asymptotic variance, Tr(Ea))
Although the solution for P is informative, it is not necessarily practical, as it depends on knowing A, K. 8- A this in turn depends on 0* and P(X, Y), which are both unknown.² 3 In the special case of mean estimation, however, it turns out there is no dependence on l". To address the remaining dependence on IPX,, Y), ; we propose to use autorater confidence scores, assuming they are available. Specifically, assume ) is discrete, and let c(y x) be the confidence of the autorater in label given input x, where c(y) I)) approximates IP(Y = X = z). This will result in the estimate for Tr(A= w K, -
Aw below, which can then plugged into the expression for PE in Proposition 3.
Example 2 (pī for mean estimation). Consider the 1-d mean loss: Co v) = 1 (v- 0)2. Then:
Tr(A,. V. A Var(Y Xxf(X) X As).
(19)
(19) can then be estimated using the observed, but unlabeled, samples scored by the autorater for each stratum k Xik Tike = 1...., No. as Var(Y - \xf(X) X E As) = 0z. where
No
1
oly Txx) (y (x - p=x)² and
(20)
N
VEY
N
1
A
cly Fia) (y- (xt())
(21)
NO
VEY
For d-dimensional data, the result is similar, but with a sum of d variances (one for each dimension).
We also provide simplified expression for of in Appendix B when f(ā) == Evey c(y ()) y. Importantly, as are free use any Pk: >> using this estimate still preserves the asymptotic coverage guarantees in (14), regardless of if the confidence estimate c(y x)) is calibrated or not. Empirically, we show that using this heuristic can indeed lead to substantial improvements.
5 Experimental results
We compare our stratified estimator, StratPPI, to two baselines: (i) the classical estimate, which uses only the labeled data, S,; and (ii) PPI++ which uses both S,, and S,. All of our experiments focus on
SSiiilarly, the optimal solution to tu. also depends on the unknown 0* in general, though this term is less important optimize if assume N to be large. In practice, always keep pa fixed to the natural rate, ws.
7

---

Different
1.00
1.00
1.00
Coversee
0.90
0.85
0.85
0.63
Cassical
0.80
0.50
0.80
StratPPI (prop.)
StratPPI topt.
0.70
200
400
800
600
1000
200
800
1000
0.40
0.40
0:40
Cassical
0.35
0:35
0.33
PPI
0.30
(prop.)
StratPPI (opt.)
0.20
0.20
0.15
0.15
0.15
0.10
0.10
0.10
0.05
800
600
1000
200
830
0.10
0.10
Cassical
PP14-
0.08
0.00
StratFPI (prop.)
StratFPI topt.)
0.64
0.04
0.02
0.02
200
400
500
800
1000
1000
Figure 1: Mean estimation simulation study with K and 0.1. The top row plots coverage (i.e., the fraction of the cases where the CI contained the true parameter value 0*). The middle row plots the mean CI width (L is better). Shaded areas plot the 16/84 quantiles across 5k trials. The bottom row plots the RMSE of fSPP computed across the 5k trials, which shares the same trend with the mean CI width, as the estimator is unbiased. The left column shows setting where strata are homogeneous, and StratPPI provides the no benefits over standard PPI++ (but is not worse). The middle and right columns show heterogeneous settings where the autorater has either a different bias (p) variance (a) per stratum, in which case StratPPI helps substantially. As strata variances known, we only report proportional and optimal sample allocation results for StratPPI.
1-d mean estimation. We explore three different allocation strategies for StratPPI: the first to set Pk Wh. to be data proportional (StratPPI Prop. the second is to set Pk optimally via the oracle Pk = PE (StratPPI Opt.). and the third to use the approximation, Ps x wKyr, k. in Example 2 for A 1 when confidence scores are available (StratPPI Heur We use X-tuning for both PPI++
and StratPPI, as outlined in $4.2. Additional experimental results are given in Appendix C.
5.1 Simulation studies
We start with a simple synthetic experiment that is an analogue of $7.7.1 in [3]. Our goal is to estimate the mean outcome E where Y A00 1). We assume that the input space X' is partitioned into K = 2 strata, A2), of equal mass P(X E Ai) = P(X E A2) 0.5. We then assume that predictions are formed f(Xix) Yi μ4 + ORESR, where εj AN 1). In other words, the predictions do not depend on the covariates Xçk. other than to reflect stratum-specific noise a and bias pig. We three different scenarios: (i) where the two strata are homogeneous with μ1 fl2 and σ1 = 02; (ii) where the two strata have different prediction biases, P1 * μ2; and (iii) where the two strata have differrnt prediction noise levels, σ1 + 02. For each experiment, we sample N 10,000 total predictions fXX) using `1 ý2 0.5, i.c., proportional to masses of the two hypothetical, -eught strata. We then vary the total number n of labeled examples Y. where the allocation is chosen according to p (which differs depending Opt.). We show results in Figure 1 for the mean confidence interval (CI) size and coverage (i.e., the fraction of the cases where the CI contained the true parameter value 0*) of each method, averaged over 1k trials. As the plots in Figure illustrate, when the underlying strata are homogeneous (left col- umn), StratPPI behaves similar to PPI++ However, when the strata are heterogeneous (middle and right columns), StratPPI outperforms both baselines srafficantly-hheeeee + becomes barely
if are using StratPPI Prop. or StratPPI

---

Effective sample
Sermeery Adtibuteble
Clessical
PP
P9+4
StraiPW (prap.
800
33
(heur)
Seaborse
0.20
StreePPt (heur)
600
0.13
20
400
Classical
0.10
15
(prop.)
StratPPI theur.
0.03
200
420
800
500
200
400
600
930
200
400
600
800
Beat berieve-- lenueedd
0:25
Classicll
PPU
500
(prap.)
Attributed GA
PMt (prop)
StrwiPPI (heur))
400
0.20
StreePot (heur)
300
0.15
Clessical
200
0.50
StratPPI (prop.)
100
StratPPI thour)
100
500
400
200
330
400
Classical
50
Strat (prop.)
3000
StrattPtt (heur)
Ge
2000
30
PRl++
PRte-
0.05
20
StratPPI (prop.)
1000
StratPPI (prop.)
StratPPI theur)
1000
Figure 2: Mean estimation on real data with K 10 and a = 0.05. The x axis plots the number of human-laheled examples n; the y axis plots CI width, percent reduction in CI width against the classical estimate and the effective sample size (the amount of human labels necessary to match the confidence interval via classical inference). Shaded areas plot the 16/ /84 quantiles across 1k trials. All StratPPI methods improve over classical inference and PPI++.
more powerful than the classical estimator. Additionally, we see that when the variance differs stratum (right column), optimal allocation of the sampling budget indeed provides additional benefit.
5.2 Real data studies
now demonstrate how our method performs on real datasets, where the underlying structure of the autorater is unknown. To partition X, choose to focus stratifications that are based on the autorater's ' predictions, f(X), based on the intuition that autorater performance can often differ across the type of predictions that it makes. Concretely, if the output space y" of is discrete, then we define A = y"; otherwise we define based on the equal-mass quantiles of ) (which can be estimated by sampling large set of unlabeled X and applying the autorater). We set K 10. For all experiments, we plot performance as a function of n. where n. is the number of human ratings our system is allowed to observe from each dataset. The remainder of the dataset (including any data points that unlabeled, labeled but with the labels removed) is used for the autorater sample Sx-
Seahorse. The Seahorse dataset [11] focuses on multilingual summarization. The authors consid- ered generative models that output summaries for document, and collected labels for many systems that cover serveral dimensions of summary quality. We focus on one quality dimension aerenssionwwhhher whether the summary is fully attributable to the source document-aa and on summarization system-4 fine- tuned 13B parameter mT5 model [33]. The autorater models for each dimension are also m finetuned models, which output probability scores. The data contains 2727 examples for these two tasks, all of which have both human ratings as well as autorater scores.
AttributedQA. In attributed question answering [6], the goal of the QA system is to output both an answer, and retrieved document that provides support for that answer. The system only considered to be correct if the answer is both correct, and indeed supported by the linked document. We evaluate the highest-scoring "retrieve-and-renn QA system from this dataset,and define our autorater be IIB parameter T5 model [26] fine-tuned on collection of natural language

---

entailment tasks [15]. Like Seahorse, the model predicts probability for whether or not the QA system gave an attributable answer. This dataset has 1000 human labels and 3000 autorater labels.
Galaxy, To demonstrate the generality of StratPPI beyond LLM-based settings, we also consider the Galaxy dataset [31], where the task is to estimate the fraction of spiral galaxies in the local universe. The autorater used here is ResNet classifier applied to images from the Sloan Digital Sky Survey (SDSS) [34]. The classifier estimates the probability that the galaxy in question is spiral. We use 16,743 observations from the dataset which contain both the human and autorater labels.
function of n. We report results on the following estimates: Classical inference, PPI++ StratPPI Prop. and StratPPI Heur. We do not report StratPPI Opt. since it is unknown for real data. For each value of n. we sample n of the cases with human labels at an allocation rate Pk (this rate is determined differently in StratPPI Prop. and StratPPI Heur As noted above, we construct the unlabeled dataset Sx by joining the remaining labeled data (without utilizing the true labels) with any unlabeled data available for that dataset. We repeat this over 1000 trials, and for each trial obtain the CI width. We report the mean of these widths in Figure as well as the percent
Methodology. We follow procedure similar to [1; 3] to study CI estimates
100%. We also report reduction in width over the classical inference baseline, [ -1 [Cmethod
(C [Colnssionll
the "effective sample size", which we define as the number of samples required to obtain a CI of the same width as the method at sample size n when using the classical inference baseline instead.
Results. Figure 2 shows large improvement for StratPPI methods over both PPI++ and classical inference for most datasets. Specifically, though all CIs decrease substantially in absolute size with expected, we see that improvements in the percent reduction in CI width over + can be as large as 0.10 - 0.20, 0.20 + 0.30, and 0.25 + 0.35 points in Seahorse, AttributedQA, and Galaxy, respectively. Furthermore, we can observe that many of the datasets exhibit heterogeneous characteristics, for which heuristic allocation helps considerably. In Galaxy, this accounts for a + +10 percent reduction in CI width. In Seahorse and AttributedQA, the improvements are less strong but still clearly apparent. The practical implication of these results is that when limited by a human rating budget, StratPPI is able to produce an estimate of the mean with fewer human ratings via stratification and sampling allocation. For example, for the Seahorse dataset, we can see from the right column in Figure 2 that StratPPI Heur. with only 300 human ratings will be approximately as confident about the mean as the classical estimate that utilizes 600 human ratings, a factor of 2
6 Conclusion
As systems built on top of large language models continue to become more and more advanced, it becomes increasingly challenging to evaluate their performance using automatic tools. Manual labeling, on the other hand, is slow and expensive. Methods which therefore save on annotation degrading. cost critical for reliably evaluating models, and knowing when they are improving ceddiction-poweee inference (PPI) is a promising class of such hybrid evaluation methods, since it leveraged to provably produce statistically valid confidence intervals, while also effectively reducing the number of human labels needed to obtain intervals of certain width. Our results demonstrate that we can push PPI even further by introducing method for performing even lower variance M-estimation by employing stratified sampling. In particular, we find that stratifications based on the predictions of the autoraters themselves proves to be an powerful stratification technique.
Limitations. While the confidence intervals produced by StratPPI (and PPI) enjoy coverage guar- antees, these guarantees asymptotic. When unite-sample performance is of particular importance, techniques that afford stronger guarantees might be preferred [2; 4; 5]. We also note some non- trivial aspects of the stratified sampling setup: (i) the number of strata has to be fixed; if K scales with n a more careful treatment is required; and (ii) the assumed observation model is different from traditional i.i.d. settings-we must be able to sample fixed sized samples from each partition; and (iii) performance may not be improved if the selected stratification is not statistically useful. Furthermore, in practice, the human data might have already been collected, in which case the studied stratified sam- pling setup does not directly apply. We leave the study of such post-stratified estimators to future work.
Broader impacts. This paper introduces more powerful statistical method for evaluating LLMs by merging human evaluations with autoraters in a way that is aware of subdomain differences. Our goal is to help power more reliable evaluations with lower annotation effort, both in terms of cost and time.
10

---

Acknowledgements. We thank Jacob Eisenstein, Jonathan Berant, Anastasios Angelopoulos, Taylan Cemgil, Amoud Doucet, and Chris Dyer for helpful comments and discussions.
References
[1] Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana
Zmic. Prediction inference. Science, 38266671):669 674, 2023. doi: 10.1126/science. adi6000. URL https: /wwww science org/ http://ii/abs/////a/ 11t.11...1.i11ne//e// adi6000.
[2] Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Confor- mal risk control. arXiv preprint: 2208.02814, 2023.
[3] Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zmic. PPI++: Efficient prediction- powered inference. arXiv preprint: 2311.01453, 2023.
[4] Anastasios Nikolas Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms achieve risk control. arXiv preprint: 2110.01052, 2021.
[5] Stephen Bates, Anastasios Nikolas Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution free, risk controlling prediction sets. arXiv preprint: 2101.02700 2020.
[6] Bernd Bohnet, Vinh Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui. Tom Kwiatkowski, Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint: 2212.08031 2023.
[7] Pierre Boyeau, Anastasios Angelopoulos, Nir Yosef, Jitendra Malik, and Michael Jordan. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint: 2403.07008, 2024.
[8] Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang, editors, Proceedings the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291-305, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 18.18653/v1/ 2022.cmmlp- main.20. URL http://://aaaaaahlaaaa//ee/ org/2022. 2022. emnle- maipp--mai.2.. 20.
[9] Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting the Association for Computational Linguistics (Volume I: Long Papers), pages 643-653, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v 1/P18-
URL https: hatl/////acaaaaoooooooo www.aaanttooooooooooooroeooe 0ttg/P8--1055
[10] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating 1lms by human preference. arXiv preprint: 0400.04132, 2024.
[11] Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roce Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das. and Ankur Parikh. SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1397-9413 13, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 22.18653/v1l1 http:///111222...mmmmmmmma. aclanthon aclanthology. org/2023 emnlp-main hamlpmmain.584
URL https://
[12] William G. Cochran. Sampling Techniques, 3rd Edition. John Wiley, 1977. ISBN 0-471-16240- X.

---

[13] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835-10866 PMLR, 2023.
[14] R. Alex Hofer, Joshua Maynez, Bhuwan Dhingra, Adam Fisch, Amir Globerson, and William W. Cohen. Bayesian prediction peediction-powered powered inference. arXive preprint: 2405.06033 2024.
[15] Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Re- evaluating factual consistency evaluation. arXiv preprint: 2294, 2022.
[16] Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust escalate: Llm judges with provable guarantees for human agreement. arXiv preprint: 2407. 18370, 2024.
[17] Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. Evaluating open meiin question answering in the era of large language models. In Anna Rogers, Jordan Graber,, and Naoaki Okazaki, editors, Proceedings of the 6t Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers). pages 5591-5606, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 10. 121866633112222..comm acl- lal.o...37... URL atttss://ccla althoooloo org/ 2023/2223..22 acl- pacl-lome 1oa1lloon..077
[18] Chia-Wel Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metries for dialogue response generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2132, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/DD 1230. URL httpths://ccaaaatllaaaeee www.eeo.rgg.-1--- D213/D0--130.
[19] S.L. Lohr. Sampling: Design and Analysis. Chapman & Hall/CRC Texts in Statistical Science. CRC Press, 2021. ISBN 9781000478235. URL https::/tt/ooooo /boks/ google. bom/bookssidd DahGEAAAQBAJ.
[20] Whitney K. Newey and Daniel McFadden. Chapter 36 large sample estimation and hypothesis testing. volume 4 of Handbook of Econometrics, pages 211 1-2245. Elsevier, 1994. doi: http://o....01000000116111144112
URL URL https: /wwww/wwwwwoo//..oo..../ www.nceeieeeet.......... com/
[21] Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97(4):558-67) 625, 1934. ISSN 09528385. URL. htap:://wwwjjsoor www.:jjsstrr.rr/ttbbbbb.o.o.o org/stable/ 2342192.
[22] Alexandru Niculescu-Mizil and Rich Caruana. Obtaining calibrated probabilities from boosting. In UAI, volume 5, pages 413-20, 2005.
[23] Jekaterina Novikova, Ondicj Dušck, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for NLG. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1241-2251 Copenhagen, Denmark, September 2017. Association for Computational 1/D15///1/11112233 URL https: / acl anthhoothhggy cwtechology.org/DEE.. org/D17-1233
Linguistics. doi:
[24] Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Vlume 1: Long Papers), 2023.
[25] John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
[26] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 2020. URL html. http: / /mjrr. org/papers, papers/v21/20- hmm.l......m..
12

---

[27] James M. Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429): 1995. ISSN 01621459, URI. http::/ /ttp:://wwwww. http:////ww.j..orrroo/.o beststrrrrrrt.oo org/sttable/2291136
[28] Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evaluation framework for retrieval-augmenter generation systems. arXiv preprint: 231 1.0947,, 2024.
[29] A. W. van der Vaart. Asymptotic Statistics, Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.
[30] Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 117748):30266-30275 2020.
[31] Kyle W. Willett Chris Lintott, Steven Bamford, Karen L. Masters, Brooke D. Simmons, Kevin R. V. Casteels, Edward M. Edmondson, Lucy F. Fortson, Sugata Kaviraj, William C. Keel, Thomas Melvin, Robert C. Nichol, M. Jordan Raddick, Kevin Schawinski, Robert J. Simpson, Ramin A. Skibba, Arfon M. Smith, and Daniel Thomas. Galaxy Zoo 2: detailed morphological classifications for 304 122 galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 435(44) 09 2013. ISSN 0035-8711. doi: 10. 10.1093/mmrassstt11 458. URL httttttts//oo..oo org/ 10t.1//////////st11/e.
[32] Jeffrey M. Wooldridge. Asymptotic properties of weighted m-estimators for standard stratified samples. Econometric Theory, 17:451 - 4700, 2001. URL http:: http:/:///....aaaiieeeeemmm www.eeeesccoootooeeeeoeoeeeeoeee org/CorpusID: :1: 123148497.
[33] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text trans- former. arXiv preprint arXiv:2010.1 1934, 2020.
[34]
Donald G. York, J. Adelman, Jr. John E. Anderson, Scott F. Anderson, James Annis, Neta A. Bahcall, A. Bakken, Robert Barkhouser, Steven Bastian, Eileen Berman, William Boroski, Steve Bracker, Charlie Briegel, John W. Briggs, J. Brinkmann, Robert Brunner, Scott Burles, Larry Carey, Michael A. Carr, Francisco J. Castander, Bing Chen, Patrick L. Colestock, A. J. Connolly, J. H. Crocker, István Csabai, Paul C. Czarapata, John Eric Davis, Mamoru Doi, Tom Dombeck, Daniel Eisenstein, Nancy Ellman, Brian R. Elms, Michael L. Evans, Xiaohui Fan, Glenn R. Federwitz, Larry Fiscelli, Scott Friedman, Joshua A. Frieman, Masataka Fukugita, Bruce Gillespie, James E. Gunn, Vijay K. Gurbani, Ernst de Haas, Merle Haldeman, Frederick H. Harris, J. Hayes. Timothy M. Heckman, G. S. Hennessy, Robert B. Hindsley, Scott Holm, Donald J. Holmgren, Chi hao Huang. Charles Hull, Don Husby, Shin-Ichi Ichikawa, Takashi Ichikawa, Zeljko Ivezić. Stephen Kent, Rita S. J. Kim, E. Kinney, Mark Klaene, A. N. Kleinman, S. Kleinman, G. R. Knapp, John Korienek, Richard G. Kron, Peter Z. Kunszt, D. Q. Lamb, B. Lee, R. French Leger, Siriluk Limmongkol, Carl Lindenmeyer, Daniel C. Long, Craig Loomis, Jon Loveday, Rich Lucinio, Robert H. Lupton, Bryan MacKinnon, Edward J. Mannery, P. M. Mantsch, Bruce Margon, Peregrine McGehee, Timothy A. McKay, Avery Meiksin, Aronne Merelli, David G. Monet, Jeffrey A. Munn, Vijay K. Narayanan, Thomas Nash, Eric Neilsen, Rich Neswold, Heidi Jo Newberg, R. C. Nichol, Tom Nicinski, Mario Nonino, Norio Okada, Sadanori Okamura, Jeremiah P. Ostriker, Russell Owen, A. George Pauls, John Peoples, R. L. Peterson, Donald Petravick, Jeffrey R. Pier, Adrian Pope, Ruth Pordes, Angela Prosapio, Ron Rechenmacher, Thomas R. Quinn, Gordon Richards, Michael W. Richmond, Claudio H. Rivetta, Constance M. Rockosi, Kurt Ruthmansdorfer. Dale Sandford, David J. Schlegel, Donald Schneider, Maki Sekiguchi, Gary Sergey, Kazuhiro Shimasaku, Walter A. Siegmund, Stephen Smee, J. Allyn Smith, S. Snedden, R. Stone, Chris Stoughton, Michael A. Strauss, Christopher Stubbs, Mark SubbaRao, Alexander S. Szalay, Istvan Szapudi, Gyula Szokoly, Anirudda R. Thakar, Christy Tremonti, Douglas L. Tucker, Alan Uomoto, Dan Vanden Berk, Michael Vogeley, Patrick Waddell, Shu Wang, Masaru Watanabe, David H. Weinberg, Brian Yanny, and Naoki Yasuda. The sloan digital sky survey: Technical summary. The Astronomical Journal, 120(3): sep 2000. doi: 10. 1036/333111.. URI. httph::tp///... doi. org org/10.10 1006/086000 /01513
13

---

[35] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging L.LM-as-a-judge with MT-hench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL. .ttt://orrum.dduccccioo.....
[36] Tijana Zmic and Emmanuel J. Candès. Cross-priiiction poweeeed inference. Proceedings of the National Academy of Sciences, 121((55)c232208312,, 2024. doi: 10. 22233pn31..22228122111 URL https: : //ww://wwww/...// pnas org http//ooaaab////eeee/ee/e 10. 2073//mas...////////// 2322083121.
[37] Tijana Zrnic and Emmanuel Candès, Active statistical inference. arXiv preprint: 2403.03208. 2024.
14

---

A Proofs
We begin by defining the basic regularity conditions that we assume Eg satisfies:
Definition A.1 (Regularity conditions of tg). Assume that
(i) a compact subset of :
(i) The minimizer 8" int(@) is unique with Eg (X,
(iii) For all (x. v) E X V, (0(2, v) is twice continuously differentiable int(@);
(iv) For all O Lo(X))), (talX.))
and (x
have finite expectations;
80,
80,00,
(v) (XV"(((X,Y) Y)] is non-singular
These regularity conditions are fairly mild for example, it is straightforward to verify that the loss of the mean estimator, ] lly 0|1?, satisfies Definition A.1. We note, however, that PPI++ be applied to merely "smooth enough" losses, which also include non-contimgn non-continuously differentiable functions losses like the quantile loss. As we primarily focus mean estimation in this work, for simplicity of analysis we only consider (the still broad class) of losses satisfying Definition A.I, though our results can be expected to readily extend to the more general case following similar treatment as in [3; 29].
To support our theoretical results, we also provide the following two lemmas.
Lemma A.1 (Slutsky's Theorem, general form). Let X., 4 X and Yn 4 where X X, and Yn are random vectors, and c is constant vector. Then for any continuous function g. 9 9(xn: Yn) 4 g(X, c).
Proof. By Theorem 2.7 of [29] we have (Xn, Yn) 3 (X,c). The continuous mapping theorem then implies that g(X,, Y,) $ c).
Lemma A.2. Under the regularity conditions of Definition A.I, we have that 0SPP 2, 0"
Proof. Under the regularity conditions of Definition A. and the fact that d 2. A where d is constant, the uniform weak law of large numbers can be applied to each term in L S (0) so that
(0) - El(o(X 2 0.
sup d
As 0 is unique, O is compact, and is continuous follows from Theorem 2.1 of [20] that
@SPP P 0"
A.1 Proof of Theorem
Proof. For ease of notation, we will define
N
1
L (0)
la ( to f(xx)
NR
1
Lng (0)
La (Xix, Yik)
Tık
mg (0)
lo (x f(Xx)).
Tkk
We will also use the following shorthand for the conditional gradients:
Vla Vlo(X,Y) X E Ar and
veLa = vlo(X, X A
15

---

As samples are i.d. within each stratum, the CLT gives
VNa
Vi (0*) - E[VeL 4 N (0.
Cov Cov (veh
(22)
and
V Lan (0*) - EIVe 0- VI (0*) - E
$ N (8) Cov (Ver.o))
T
(23)
Since samples Sn and SN are independent from each other, we also have that (22) and (23) converge jointly. Applying Lemma A.I for the following continuous function of Ås 2 X then gives
Vr İo. C..NNVLL (0*) E(VEL
1 -
(e.) - E .-1 10 E
(0.
0, x 4 N
Cov(VV (
+ +N
Cov Cov A vehhor))
(24)
Nk
1 Ae
N 0,
K.
(25)
Pk:
pk
Since H* satisfies [[V O, by the law of total expectation
- =0 0.
(26)
Using this fact, we can write
VL SPP (0)
Σ wk (a (deI (0) + Lmn (0) - (()))-BBV(ea))) -EV(,,0))
(27)
Since samples across the K (where here K is constant) stratas are independent, combining the results of (25) with (27) yields
VVV 1. (.SPP(())))) (0*) S NO B
(28)
Applying the mean value theorem, we have
VL SPP (ŌSPP) (LSPP((0))+ + V2 Vo) (0) 0*)
(29)
for some between gsPp and 0* Let At represent the pseudoinverse of A. Then
h 0- (e (V (6SPT) -I
(30)
As @&PP 2 0t per Lemma A.2, we have Ø & 0r Under the regularity conditions of Definition A.1 and the fact that Å E, Aς. an application of the uniform weak law of large numbers and the continuous mapping theorem then gives
LLLSPP E. V'B(ea]]"'
(31)
which is a constant term. Furthermore, by the law of total expectation,
K
E 26.
Σ
wv Hu
A-1
(32)
Finally, the fact that @SPP 2g 0 and E in)( together with the fact that QSPP is a minimum of ISP implies that VL ((SPP) 3 Combining these results with (28) via Lemma A.I and the fact that A- is symmetric for twice continuously differentiable ta gives
Vn(0 0*) S NO, A BZ (A=)) =V(, A B Aー1)
(33)
16

---

A.2 Proof of Corollary 2
Proof. The regularity conditions of Definition A.1 allow to apply Lemma 4.3 of [20] to show that each the plug-in estimates for each stratum term satisfies
B Hgpr 2 Hx, and
A k.j.
A R.J.0*
and
AAA k.A.
AA R.
which implies that the weighted plug-in in estimate for ΣA satisfies ET 3 Σd (14) is thus equivalent to taking the _ 9) quantiles of the asymptotic normal distribution of @SPP, impplyng (15). M
A.3 Proof of Proposition
Proof. In the special case of the square loss, the Hessian is the identity matrix. Therefore, simplifying and applying the linearity of the trace, we have
k
Tr(E,))
=
Wk Tr
A
K.
(34)
Pk
Pk
k=
wk Tr
(r.
Vide
(35)
k.j.o-
K
K
wik Tr (vdya
+
Σ
wi
(v-yy
(36)
k=1
A=
K
wi Ve
K. .ia
+
wy
Ae R.
(37)
==
where we include the subscript jj to index the diagonal variance terms of both covariance matrices. For ease notation, denote the conditional variances as
VfS Z = k : X (X, X E A and VA.j
7 = k = Vlo Veo - AV (X)) X A
Then we can write (37) as
K
d
wk Av
wk V d
Σ
rrE(Var(Vy,, [2]] E[Var(Vs,) Z)],
K. 51
k.A.e
(38)
and apply the law of total variance to get
Tr(Ex))
ri[Var(Vis [2) + E[Var(Va. 2)]
(39)
(Var(V,,)) Var(E[V,s Z)) + (Var(Vs.)) Var(TE[Va, Z)))
(40)
/Var(V Var(Vass)
(41)
j=1
= Tr(EAA)
(42)
Finally, (41) holds with equality iff both Var(B[VV, Z)) = and Var(E 2) 0. which, combined with the assumption that P(Z k) we > for all k; only satisfied when both E[V5, Z] and EV[,, Z2] are constants.
17

---

A.4 Proof of Proposition 2
Proof. By linearity B we can rewrite n = A B:r A as
K
2 A
T
-.V.
Vd
+ 1
v.ve
k.S.
Aw
(43)
Pk.
Pk.
k=1
Wk
Aw
rpk
Vi
Vr
Aw
(44)
Pk
Pk.
K
Wh
1-1
18k
Vd
k.s.0- + V.yy
A_1
(45)
Pk
w
Nk
By linearity of the trace, we then also have
Tr
wh Tr
A
Ve. + V ViD A
)
(46)
NN
Pk
As each term in the sum is independent, term, i.e.,
minimize the by minimizing each individual
x = argmin Tr
A
N2
T&k + V K.
k,000-
A
(47)
for k = 1, 1,.... K. We can then directly apply Proposition of [3] to get
Tr A (Cov(Vé Vel.0-) + Cov k.0+ V1x,o-))A
(48)
2(1 + rx] AA Cov Cov(V )-mm)
where 7% = N
A.5
Proof of Proposition 3
Proof. Following the same derivation as Proposition 2, we can rewrite
Tr(E#t
Σ
wk
Tr
A
k.
V
A
(49)
Pk
r
K
w
Tr
A
k.
A
K
2
Tr
(A
V.dee
A.
(50)
Pk
Pk
and therefore can optimize गu and independently.
We start with Px- Let 2 Tr Av problem
We then solve the constrained optimization
" 2k
K = 1, Pk: 2 0.
minimize
Σ
s.t.
(51)
Turning this into Lagrangian with additional slack variables SA 0, have
w 2k + p
J(p
Σ Pk
m (pk 22
(52)
Pk
18

---

Setting V.(.. n) = 0 and solving for Pt: then gives:
0J
W
Ppc
Ph
2
2k
+
fe
+
17k- = 0
(53)
0J
0p
-
- Pk:
=
(54)
0J
Pk
2
(55)
ŌT
0J 8k = 0
(56)
Assume that the inequality constraint is inactive, and 17
O. Solving for Pic
w
K
w
P
|
=1
(57)
1
H
K
(58)
1=1
Wkk /
Pk
(59)
K =x== wk /2
We can now verify that the constraint is inactive, with P 2 0, since Σ k= w/ 1, w 2 0 by definition of a valid probability distribution, and we have 2/ 2 since is a sum of non-negative variance terms.
The same analysis can then be applied to Pk. but for zk Tr ( K.
V. Ant).
B A simplified estimate of the sample allocation for mean estimation
In the setting of Example 2 (i.e., 1-d mean estimation), assume that cly x) P(Y = y confidence estimate for label E V, where Y is discrete. Then, if we define the autorater as
f(x)
Σ Edyl== y E X =),
(60)
NEV
the term Var(Y - {xx(() X E As) further simplifies to x Var(Y X E Ax) for any value of dg. Applying the law of total variance, can conveniently write Var(Y X E As)
Var(Y X E Ax) = E\Var(Y X) X E A+1 + Var(E(Y X]] X E As),
(61)
which can be empirically estimated as on on unlabeled autorater data, X.t i = 1, I,..., No. via
0% = =ENe
Σd cly Xx)) - s(Xx))) (f(x.) DEY
(62)
where EN, and Var denote the empirical mean and variance over all Xú, respectively. Lastly, when y binary (which is the case in all of our experiments in $5.2), this becomes
EN [(( (((((())) + Varn, (f(X,)))
(63)
which can be easily calculated in Python as np mean (yhat * (1 - yhat)) + mp. varvar (yhat).
C Additional experimental results
We also evaluate StratPPI on the Chatbot Arena dataset [10], in which we evaluate the win-rate of gpt-4-1106-previer over claude-2. 1.4 See Figure 3. Specifically, we use the standard Chatbot
http:://huuggggggoccco////ttttts arenaa cuman-preferenceeeee
19

---

C width
Percent Reduction wicth
Effective sample size
30
Classical
Chatbot Areno
0.20
StratPP: (prap.)
StratPWI (heur)
(prop.)
600
Str StPP (heur)
0.15
0.10
Ppl
200
Strrf991 (prop))
0.05
200
Figure
Win-rate experiment on Chathot Arena for gpt-4-11066previit vs. claude-2.1. 1.
Scores are based on the average label ('better' = vs. 'worse" = 0) over 10 samples from Gemini Ultra acting as a LLM judge. Interestingly, as these confidence scores are not calibrated, our heuristic becomes overly aggressive at higher Future work can explore how to best incorporate additional regularization into the estimated optimal sampling ratios p.
Arena auto-eval prompt and map [[A>B]], [[A>B] to 1, [[A=B]] to 0.5, and [[B>A]], [[B>A]] to O. We then ceff-consistency sampling and take the average of 10 samples from Gemini Ultra. This is used as both our final autorater estimate and confidence. We find that stratification also helps this setting. However, in line with prior and contemporary work [e.g.. 35; 16] we found confidence scores from the LLM-as-a-judge be fairly uncalibrated, which makes our heuristic allocation less effective at larger (though still effective at small labeled sample sizes ni). Investigating robust heuristics in the face of miscalibration (e.g., via regularization recalibration) would likely help, and is a direction for future work.
yaml
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification:
All claims are supported in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We address this in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification:
Proofs are included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Results are based on publically available data, and simulations, and all proce- dures are described.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification: The data is publicly available, and the methodology is simple to implement. Pseudocode is given. Code may be made available at future date.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. ec/
icyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ httpc//ggiiess/oodaooe
ssisssoiiccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Detailed information is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are provided for the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Compute resources required are very light, as no model training is performed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics :ttt:p:///eeripett www.urriis.cpppulll/e
Answer: [Yes]
Justification: No issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the advantages of the method for more effective develop- ment large language models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "XppCr4MJZ6", "reviewer": "Reviewer_5dNx", "summary": "The paper extends the predictive power inference methods to leverage data stratification strategies, and demonstrates that stratification can be used to obtain unbiased statistical estimates, while reducing variance (with worst case being as bad as prior approaches PPI++, and in practice considerably better). Concretely, the authors\n\n1. Demonstrate that for the stratified prediction-powered loss, the minimizer has an asymptotically normal distribution with the true mean, thereby allowing for construction of confidence intervals leveraging stratum conditioned covariance matrices.\n2. Provide an method for choosing the optimal weighting between the human annotations and autograder annotations for the stratified setting.\n3. Provide the closed form solution for optimal allocation for stratum weights for the labeled and unlabeled data. Furthermore, for the special case of mean estimation, the authors also provide a heuristic way of estimating these weights based on the autograders' confidence for the scores.\n4. Demonstrate for simulations that for cases where the strata are homogeneous, the estimators are at worse as bad as PPI++, while in cases where the strata are heterogeneous, StratPPI outperforms PPI++.\n5. Finally, the authors demonstrate the utility of the proposed approach on a real-world multilingual summarization benchmark, an attributed QA dataset as well as the galaxy dataset, demonstrating that StratPPI obtains tighter confidence intervals compared to other approaches.", "questions": "For Figure 1, column 3, what is the intuition about why is the coverage of StratPPI (prop) so low compared to all other counterparts when n is small ?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method demonstrably produces tighter confidence interval, leveraging data stratification. Such stratification is quite natural to occur in generally available data, in my opinion. So the proposed extension should be very widely applicable in a real world setting.\n2. The paper is relatively easy to follow even for someone who is not very familiar with this subdomain, and is quite self contained.", "weaknesses": "Given the focus of the paper on effective evaluation of language models, it would have been good for the paper to have a stronger focus on evaluation of the numerous publicly available (large) language models. Concretely, imo, the proposed method should be extendible to assessing pairwise relative performance of language models (similar to Section 3.1 of [1]). It would be good to compare confidence intervals for the BT coefficients obtained from PPI++ vs StratPPI for (a subset of) different Language Models on the Chatbot arena.\n\n[1] Boyeau, Pierre, et al. \"AutoEval Done Right: Using Synthetic Data for Model Evaluation.\" arXiv preprint arXiv:2403.07008 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review and thoughtful comments. We provide answers to specific questions and remarks (quoted) below.\n\n> It would have been good for the paper to have a stronger focus on evaluation of the numerous publicly available (large) language models.\n\nAs suggested, we have included an additional experiment in Figure S.1 of the supplement to this rebuttal for estimating win-rates between gpt-4-1106-preview and claude-2.1 on the Chatbot Arena dataset. Specifically, we use the standard Chatbot Arena auto-eval prompt (see https://github.com/lm-sys/arena-hard-auto/blob/main/config/judge_config.yaml), and map `[[A>>B]]`, `[[A>B]` to 1, `[[A=B]]` to 0.5, and `[[B>>A]]`, `[[B>A]]` to 0. We then do self-consistency sampling and take the average of 10 samples from Gemini Ultra. This is used as both our final autorater estimate and confidence. We find that stratification also helps in this setting. As an interesting side note, in line with previous work we found confidence scores from the LLM-as-a-judge to be fairly uncalibrated, which makes our heuristic allocation less effective at larger n (though still effective at small n). Investigating more robust heuristics in the face of miscalibration (e.g., via regularization) would likely help. Proportional allocation is effective at all values of n. We will also include results on the larger multiple model setting when using Bradley-Terry model coefficients for the final version of the paper.\n\n\n> What is the intuition about why is the coverage of StratPPI (prop) so low compared to all other counterparts when n is small?\n\nThe difference in coverage reported is small, and is an artifact of a smaller number of trials run for the coverage estimation. When increasing the number of simulations for estimating coverage to 5000, we observe a more stable coverage result. This is included in Figure S.1 of the rebuttal supplement. (Note that we also fixed column 1 from the submitted manuscript, which had been run at the wrong settings.)\n\nPlease let us know if this has addressed your concerns. We look forward to engaging in the response period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1r8vXh7Ua", "reviewer": "Reviewer_BR9i", "summary": "This paper introduces the stratified PPI method, designed to reduce evaluation bias from autoraters by aligning them with a few human annotations. The key novelty lies in identifying that the autorater's performance varies across different example groups, with biases being smaller for some examples and larger for others. Consequently, they propose a straightforward method to group examples into strata and calculate stratum-specific rectification parameters. Experiments demonstrate that stratified PPI consistently achieves tighter confidence intervals compared to baselines.", "questions": "See weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 1, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper introduces a novel method, stratified PPI, which effectively addresses the issue of evaluation bias in autoraters by considering the varying performance across different groups of examples. The proposed method of grouping examples into strata and calculating stratum-specific rectification parameters is both simple and effective.", "weaknesses": "The evaluations are conducted on simple evaluation problems where the evaluation is binary, while current LLM autoraters use a more fine-grained scales (e.g., 1-5). Additionally, real human evaluations may be multinomial, where people's opinion will diverge on the same instance. I'm not sure how this proposed method will perform in that scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review and thoughtful comments. We provide answers to specific questions and remarks (quoted) below.\n\n> The evaluations are conducted on simple evaluation problems where the evaluation is binary.\n\nOur Stratified PPI method works for general M-estimation problems, which also includes non-binary mean estimates. Our supplement to this rebuttal also includes results for estimating Bradley-Terry coefficients, another M-estimation problem. The autoraters can also output scores (e.g., 1 to 5) that are different from the human labels (see line 110). In fact, the autoraters we used in our experiments output continuous scores in (0, 1).\n\n> Additionally, real human evaluations may be multinomial, where people's opinion will diverge on the same instance.\n\nOnce again, while we focus on simple mean estimation in our experiments, our Stratified PPI method applies to a broad class of general M-estimation problems. This includes multiclass logistic regression, which could serve as a good model for the proposed setting. For details, see [1].\n\nAnother interesting situation (not covered here—but important for future work!) are problems in which the human annotators are not just variable, but actually systematically biased with respect to the ground truth (e.g., when the annotators are crowd-workers vs. experts or the target users).\n\nPlease let us know if this has addressed your concerns. We look forward to engaging in the response period. \n\n[1] Angelopoulos et. al. PPI++: Efficient Prediction-Powered Inference. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5c28fB83Kh", "reviewer": "Reviewer_GJBL", "summary": "The paper proposes an extension of prediction-powered inference (PPI) - a method for calculating confidence intervals with narrower confidence bands. The data is separated into subsets and the bias rectification term in the confidence calculation is weighted based on the similarity to the particular subsets. Evaluation is performed on one synthetic dataset and 4 real datasets, showing narrower confidence bands if the data is heterogeneous.", "questions": "Please see above", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written. Even the necessary background, regarding existing work on PPI, is explained well enough to understand.\nThe adjustment of the calculation based on data similarity makes sense.\nEmpirical evaluation on several datasets is good to see.", "weaknesses": "Line 107 defines Yi as the target output, which in the case of QA would normally be a textual answer. Line 112 then defines Y as the binary gold rating of correctness of the answer. This is contradictory and causes confusion when following the other equations. \n\nThe chosen evaluation datasets don't seem to be particularly widely used. Evaluation on some more common QA datasets would have been more impactful.\n\nIt is unclear whether this method actually helps in calculating more accurate means, or whether it only produces narrower confidence bands. And if it is the latter, then can you give more details on the practical benefits? It's a different method of calculating the confidence bands so the bands would not be directly comparable to other methods and couldn't really say that the new ones are \"better\" than the previous ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review and thoughtful comments. We provide answers to specific questions and remarks (quoted) below.\n\n\n> Line 107 defines $Y_i$ as the target output, which in the case of QA would normally be a textual answer. Line 112 then defines $Y$ as the binary gold rating of correctness of the answer.\n\nThanks for pointing out this confusion. The task here is _evaluating_ the correctness of a QA system, and not inferring the answer itself. As such, $Y_i \\in {0, 1}$ is the target output of the _evaluation_ task (i.e., incorrect vs. correct). The input $X_i$ to the autorater for this evaluation task is the pair (QA question, model answer). We will make this clearer in the text.\n\n\n> The chosen evaluation datasets don't seem to be particularly widely used.\n\nWe chose to evaluate on high-quality, well-cited datasets that also have pre-existing autorater scores. In the supplement to this rebuttal, we have also included results on the Chatbot Arena dataset.\n\n> It is unclear whether this method actually helps in calculating more accurate means, or whether it only produces narrower confidence bands.\n\nOur stratified PPI estimate reduces variance. This reduces the size of the confidence band, which is the main focus of the paper. Narrower confidence bands allow practitioners to get more precise error estimates with fewer annotated samples. This is very important to do given the high costs (in terms of both time and money) of running human evaluations. Additionally, since the PPI estimate is unbiased, this also has the effect of reducing the mean squared error of the estimate, which can be seen from the usual bias-variance decomposition of the error. To illustrate this, we have also plotted the RMSE of our estimator on the task in Figure 1 of the paper in the rebuttal supplement (see Figure S.1). The trends of the confidence interval width and the RMSE are identical.\n\nPlease let us know if this has addressed your concerns. We look forward to engaging in the response period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHF85p9FzA", "reviewer": "Reviewer_deoY", "summary": "Taking inspiration from stratified sampling, this paper proposes a method named Stratified Prediction-Powered Inference (StratePPI). \nWith appropriate choices of stratification and sample allocation, StratePPI can provide substantially tighter confidence intervals than unstratified approaches.  The experimental results on simulated and real data show the effectiveness of StratePPI.", "questions": "In practical applications, how should we stratify data to ensure that StratPPI is better than PPI or PPI++?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces data stratification strategies to improve prediction-powered inference (PPI) for the first time.\n2. The experimental results show that StratPPI is more effective than PPI++.", "weaknesses": "The stratification is very important to StratPPI, and there is little introduction on how to stratify data effectively in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful review and thoughtful comments. We provide answers to specific questions and remarks (quoted) below.\n\n> In practical applications, how should we stratify data to ensure that StratPPI is better than PPI or PPI++?\n\nIn general we should stratify data so that examples with similar expected labels are grouped together (e.g., the data should be relatively homogeneous within each stratum). This reduces the variance of the stratified estimate. This can then be expected to help when the data pre-stratification is not homogenous. In the experiments presented here, we found that equal mass partitioning based on confidence scores reliably satisfied these properties (and, at least at the extreme confidence levels it makes sense to expect that labels will have low variance for any reasonably well-calibrated classifier).\n\nPlease let us know if this has addressed your concerns. We look forward to engaging in the response period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for taking the time to read and comment on our work. We were pleased to receive several good suggestions, and have taken this feedback into account. Please see our uploaded pdf for supplemental results. 

Specifically, we have added another very realistic experiment on the Chatbot Arena dataset, in which we measure the win-rate between `gpt-4-1106-preview` and `claude-2.1`. Details are included in the figure caption. We have also included additional results on our illustrative synthetic task that shows how the MSE of the prediction powered point estimate and the size of the confidence intervals follow similar trends. Stratified PPI improves on both CI size and MSE.

We have responded to the remainder of the comments raised by reviewers individually. Please let us know if any questions or concerns remain. We look forward to additional discussion.

Meta-review: This paper presents the stratified PPI method, aimed at minimizing evaluation bias from autoraters by calibrating them against a limited number of human annotations. One particular innovation is the recognition that an autorater's accuracy differs across various groups of examples, with some groups exhibiting smaller biases and others showing larger ones. Based on this insight, the authors propose a simple approach to categorize examples into distinct strata and compute adjustment parameters for each stratum. Experimental results indicate that the stratified PPI method consistently produces more precise confidence intervals than existing baseline methods.

While the reviewers see the merit in the paper, a repeated weakness has to do with the evaluation: the number of LLMs evaluated the nature of the evaluation task and data etc. While this is something to be addressed, and the authors recognize that and have even added an experiment, it seems that the reviewers are willing to accept the paper in its current form and I respect that.
Decision: Accept (poster)