Title: Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: prediction powered inference

auto-raters
Abstract: Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.  PPI achieves this by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate---but potentially biased---automatic system, in a way that results in tighter confidence intervals for certain parameters of interest (e.g., the mean performance of a language model). In this paper, we propose a method called Stratified Prediction-Powered Inference (StratPPI), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system or data distribution, we derive an algorithm for computing provably valid confidence intervals for parameters of any dimensionality that is based on stratified sampling. In particular, we show both theoretically and empirically that, with appropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than  unstratified  approaches. Specifically, StratPPI  is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.

Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation

Abstract

Prediction-powered inference ( PPI ) is a method that improves statistical estimates based on limited human-labeled data. PPI achieves this by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate-but potentially biased-automatic system, in a way that results in tighter confidence intervals for certain parameters of interest (e.g., the mean performance of a language model). In this paper, we propose a method called Stratified Prediction-Powered Inference ( StratPPI ), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system or data distribution, we derive an algorithm for computing provably valid confidence intervals for population parameters (such as averages) that is based on stratified sampling. In particular, we show both theoretically and empirically that, with appropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than unstratified approaches. Specifically, StratPPI is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.

1 Introduction

Evaluating machine learning models requires evaluation data. In particular, to iteratively improve on a method during development, or to reliably report an improvement, one needs to confidently and quantitatively assess the performance of the method. This is especially challenging for large language models (LLMs), where gathering high-quality annotations for generations can be difficult and timeconsuming-and can ultimately become quite costly if gathering more than a few hundred examples.

One often-proposed approach to avoiding the evaluation bottleneck is to use a secondary LLM-based system to judge the output of the primary one. For instance, if the primary task is developing an LLM-based question-answering (QA) system, one can use a second LLM-based system that rates question/answer pairs as acceptable or not [6; 8; 17]. However, automated raters (i.e., autoraters ) may be biased relative to the human raters they are intended to model, as others have noted [1; 9; 18; 23]. This can become substantially worse when models are tailored to hill climb on the autorater metrics, and eventually cease to become a good metric at all-a phenomenon commonly referred to as Goodhart's law, or reward hacking in the context of reinforcement learning from human feedback [13; 24].

We thus have two signals for assessing model performance. The first is human labels, which are typically accurate, but expensive to collect. As a result, usually only a small sample size is available, and

an estimate based on these samples alone will have high variance. The second is autorater predictions, which are easy to collect for large sample sizes, but may also be systematically biased. The above may suggest that one must make a choice between either (i) a high-variance, but unbiased, estimate from a small human sample, or (ii) a lower-variance, but biased, autorater-based estimate. However, it turns out there are also statistically valid ways of combining the auto-rater and human data for hybrid evaluations. Following [1; 3; 7] we call such methods prediction-powered inference ( PPI ) methods. At a high level, PPI -based methods operate by using a small sample of examples labeled by both humans and autoraters to estimate the bias of the autorater. This bias is then used as a rectifier for the autorater estimate. The resulting estimate can then be shown to provide improved (i.e., tighter) confidence intervals for properties of interest for the system being evaluated, such as its true mean accuracy.

Aweakness of standard PPI , however, is that it does not take heterogeneity into account. For example, in our QA setting, an autorater may have one accuracy when predicting if a model answer is correct, and a different accuracy when predicting if it is incorrect. This is especially true in cases where a correct answer is easy to verify (e.g., it is also present in Wikipedia), but harder to refute (e.g., no relevant external search results can be retrieved to either support or contradict it). In these settings, it may make sense to apply a different PPI strategy within each subdomain, depending on the local quality of the autorater. Moreover, we claim that such heterogenous settings are to be quite expected in practical applications. Inspired by the rich prior literature on stratified sampling and survey design [12; 19; 21], we therefore propose a stratified approach to PPI , and show that it can be very advantageous when performance varies across subdomains, for either the autorater, or the model being evaluated.

On a technical level, it is not immediately clear how to apply stratification to PPI , as it involves two types of samples: one sample that is labeled by both humans and an autorater, and another (typically much larger) sample that is only labeled by the autorater. Extending the analysis of [3], in this work we show how confidence intervals based on the asymptotic normality of weighted M-estimators [32] can in fact be derived for the stratified PPI setting. The next challenge we address is how to determine the sample sizes used for stratification (i.e., the sample size of each stratum). Similar to recent work for active statistical inference [37], we further derive optimal rates that depend on certain moments of the underlying distribution that are generally unknown-and provide an approach for effectively approximating these. Finally, we provide extensive empirical evidence showing that our stratified approach ( StratPPI ) leads to considerably improved confidence intervals over both classical inference methods that use only human labels, and the baseline (unstratified) PPI approach.

2 Related Work

Prediction Powered Inference ( PPI ) was introduced in [1] as a method for obtaining tighter confidence intervals for parameters learned in supervised machine learning (e.g., coefficients in logistic regression) by also leveraging other model-based predictions on additional, unlabeled data. Related ideas were explored by [30], but with a focus on bootstrapping as a way for obtaining confidence intervals. PPI was then extended in several directions. For example [36] showed how the labeled data can be used for both estimating the parameters and the autorater model. PPI++ [3] showed how to obtain confidence intervals that are easy to compute efficiently, and introduced a parameter for weighting the predictions of the autorater such that the overall statistical efficiency can be improved. As noted in previous works on PPI , these approaches are closely related to other statistical methods for introducing control variates based on autoevaluators [9], as well as augmented inverse propensity weighting [27] (see discussion in [1; 3]). Like this paper, prior work has also focused on using PPI / PPI++ for evaluating machine learning systems with autoraters, including for ranking chatbots in the Chatbot Arena [7] and evaluating retrieval-augmented generation systems [28].

Most relevant to our setting, the recent work of [37] focuses on active sampling of examples to label during PPI , where the total number of queried labels is random, but less than the sampling budget n in expectation. Specifically, it proposes to label examples for which the autorater is less confident in its predictions, and corrects for the sampling bias with a variant of inverse propensity weighting. In contrast, our somewhat simplified approach takes inspiration from stratified sampling where a coarse stratification is defined in advance, and the total number of labeled samples is constant. Like [37], we derive an analogous optimal allocation strategy for a given budget, though we only apply it at the stratum level, and not for individual examples. Furthermore, while the variable allocation helps reduce variance in heterogenous settings, our stratified treatment also allows for a stratum-specific extension of [3]'s estimated tuning parameters that further improves performance in complementary ways.

In terms of stratification specifically, concurrent work [14] also proposed Bayesian variants of several PPI methods, including stratified approaches. In contrast, the methods here do not require introducing priors, do not require running expensive Bayesian inference, and give more conventional, frequentist, guarantees of performance. The credible intervals produced by Bayesian methods are related to, but different from, the confidence intervals produced here, and by other prior PPI approaches [1; 3; 7].

Finally, there is a conceptual similarity between the PPI rectifiers and post-hoc calibration of a classifier. There is a rich literature on calibrating classifiers (e.g., [22; 25]), but there is a clear difference in goals between calibration and PPI : the former is aimed at modifying a learned autorater to make better probabilistic predictions, and the latter is aimed at obtaining better confidence intervals by making use of an existing autorater. That said, clearly one approach to improving PPI is to use a better-calibrated classifier, perhaps by taking some of the labeled data and using it for re-calibration. This is similar in motivation to the crossPPI approach of [36]. We leave such approaches as future work, but we do experimentally explore PPI approaches on both well-calibrated autoraters and poorly calibrated ones.

3 Preliminaries

We begin by briefly reviewing PPI [1], and specifically the efficient PPI++ variant of [3]. Here, and in the rest of the paper, upper-case letters ( X ) denote random variables; lower-case letters ( x ) denote constants, and script letters ( X ) denote sets, unless specified. All proofs are deferred to Appendix A.

Following [1; 3; 7], we assume an empirical sample S n = { ( X 1 , Y 1 ) , . . . , ( X n , Y n ) } of n i.i.d. examples drawn from some unknown, but fixed, distribution P , where X i ∈ X is an input, and Y i ∈ Y is the target output. We also assume a larger sample ˜ S N = { ( ˜ X 1 , f ( ˜ X 1 )) , . . . , ( ˜ X N , f ( ˜ X N )) } where N ≫ n , for which the target outputs are not available, but we have access to an autorater function f : X → Y ′ which provides an approximation of Y given the observed input (we use Y ′ to denote the output space of the autorater as it is possible that Y ̸ = Y ′ , e.g., if f ( X ) = E [ Y | X ] ).

As an example, assume X is a (question, model answer) tuple where the answer is produced by an LLM-based QA system we wish to evaluate, Y is a 0 / 1 'gold' human rating of correctness of the answer, i.e., 1 { human rates model answer as correct } , and f ( X ) , the autorater, imperfectly predicts this human rating Y . One value of interest is then the average accuracy of the QA system, E [ Y ] . More generally, given any convex loss function ℓ θ ( x, y ) satisfying certain regularity conditions (see Definition A.1), we are interested in estimating the d -dimensional parameter θ ∗ ,

In the statistics literature, this is broadly referred to as M-estimation (e.g., see [29]). Here, we focus on mean estimation, for which the loss ℓ θ ( x, y ) = 1 2 ∥ y -θ ∥ 2 has the optimum θ ∗ = E [ Y ] . However, our method generalizes to other typical losses such as the squared loss for linear regression, where the parameters θ ∗ are the regression coefficients, or, similarly, the log loss for other generalized linear models. Of course, the true θ ∗ is generally not known, because we cannot exactly calculate the expectation in (1). The goal of PPI , which we share in this work, is to use both the labeled and unlabeled data S n and ˜ S N to obtain an asymptotically valid confidence interval C α,j for θ ∗ j that for any j ∈ [ d ] satisfies 1

The simplest confidence interval can be obtained using standard techniques by using only the labeled sample S n , and ignoring the autorater data ˜ S n . However, PPI employs a clever trick which allows for also using the unlabeled sample ˜ S N to get tighter confidence intervals, as described next.

3.1 A rectified prediction-powered loss

The key idea of PPI -based methods is to use autorater predictions to derive a low variance, but unbiased, estimate of the objective in (1). Consider the following 'rectified' prediction-powered loss:

The rectifier term on the right hand side of (3) removes the bias of the autorater data so that L PP ( θ ) satisfies E [ L PP ( θ )] = E [ ℓ θ ( X,Y )] . However, when f ( X ) is correlated with Y , the loss L PP ( θ ) will have lower variance. Unfortunately, f ( X ) may not be a good predictor of Y in all cases. In fact, it is even possible for L PP to be higher variance than the classical estimate (e.g., if f ( X ) is anti-correlated with Y ). Thus, to adapt to the quality of f ( X ) , PPI++ also introduces a tuning parameter λ ∈ R : 2

̸

Clearly, L PP λ still remains unbiased for any value of λ . For λ = 0 , the framework reduces to classical inference. In most cases, however, a proper choice of λ = 0 will result in lower variance. PPI++ also suggests how to automatically choose a data-dependent ˆ λ that converges to an optimal value.

3.2 A prediction-powered confidence interval

We next describe the PPI++ method for deriving confidence intervals based on the rectified loss. Let

be the prediction-powered estimate. Standard analysis for M-estimators can then be extended to show that the scaled difference of the estimate ˆ θ PP ˆ λ and the true θ ∗ is asymptotically normally distributed. PPI++ then leverages this result to compute a confidence interval for θ ∗ that is asymptotically valid.

where λ ∈ R is a hyper-parameter. Then under the regularity conditions of Definition A.1, we have that √ n ( ˆ θ PP ˆ λ -θ ∗ ) d →N (0 , Σ λ ) , where Σ λ := H -1 θ ∗ ( r · V λ f,θ ∗ + V λ ∆ ,θ ∗ ) H -1 θ ∗ .

Corollary 1 ( PPI++ CI , [3]) . Let ̂ Σ ˆ λ be the plug-in estimate for Σ λ using S n and ˜ S N . Define

where z β denotes the β -quantile of the standard normal distribution. Then for any j ∈ [ d ] it holds that lim n,N →∞ P ( θ ∗ j ∈ C PP α,j ) ≥ 1 -α .

As mentioned above, [3] further showed that for an appropriate choice of λ (that can be analytically derived and estimated via a ˆ λ ), the trace of the covariance matrix, Tr(Σ λ ) , is at most that of the covariance matrix derived without using autorater data, implying that the PPI++ -based confidence set C PP α can always be at least as tight as that of the classical M-estimator (and often much tighter in practice).

4 Stratified prediction-powered inference

We now present our approach for improving PPI++ estimates via stratification . In particular, we show how optimizing a rectified loss computed via stratified sampling can lead to a consistent, but lower variance, estimate of the optimal parameter θ ∗ , and correspondingly tighter confidence intervals. Consider the QA example from the previous section. For most autoraters, it is reasonable to assume that the strength of their performance can vary, depending on the type of input being presented. For instance, an autorater might be accurate at predicting whether an answer to an unambiguous question (e.g., 'What is the capital of France?' ) is correct, but relatively poor at inferring if an answer to an open-ended question (e.g., 'What is the best way to cook a hamburger?' ) is acceptable or not. Splitting the problem space into different domains allows us to derive a more specialized form of the prediction powered loss that can better adapt to this autorater heterogeneity via stratified sampling.

Formally, assume that the input space X is partitioned in advance into K non-empty, mutually exclusive, and exhaustive strata A = ( A 1 , . . . , A K ) , where K is a finite integer. For each stratum,

we further assume that we can estimate w k = P ( X ∈ A k ) arbitrarily well using large amounts of unlabeled data or prior knowledge; we treat them as known constants here for simplicity. To collect the labeled and unlabeled datasets S n and ˜ S N , we then follow a standard stratified sampling procedure in which we draw two i.i.d. sets of samples of fixed size n k and N k , respectively, from each stratum k , where ∑ K k =1 n k = n and ∑ K k =1 N k = N . The relative sizes n k /n and N k /N are free parameters; they can simply be the natural rates, n k /n ≈ N k /N ≈ w k , or systematically decided (see §4.3). Note that this is a fundamentally different sampling model from standard PPI : here examples are i.i.d. within each strata, and independent (but not necessarily identically distributed) across each strata.

Next, we define the stratified loss via the weighted sum, L SPP λ ( θ ) = ∑ K k =1 w k L PP k,λ k ( θ ) , where w k is the stratum weight, λ = ( λ 1 , . . . , λ k ) ∈ R k are now stratum-specific tuning parameters, and L PP k ( θ ) is the conditional prediction-powered loss computed within each stratum, i.e.,

As before, each L PP k,λ k is an unbiased estimate of the conditional loss. Like PPI++ , we also allow for data-dependent parameters ˆ λ k that we show how to automatically tune for best performance in §4.2.

4.1 A stratified prediction-powered confidence interval

We now present our main result, which is a confidence interval for θ ∗ based on the stratified loss. More precisely, the result states that, as in PPI++ , the minimizer of the stratified loss,

has an asymptotically normal distribution with mean θ ∗ . See Algorithm 1 for pseudocode.

Theorem 2. Assume that ˆ λ k p → λ k , n N → r for any r ≥ 0 , n k n → ρ k for any ρ k > 0 , and N k N → ˜ ρ k for any ˜ ρ k > 0 . Let H k,θ ∗ := E [ ∇ 2 ℓ θ ∗ ( X,Y ) | X ∈ A k ] , and

where λ k ∈ R is a hyper-parameter. Then, under the regularity conditions of Definition A.1,

where Σ λ w := A -1 w B λ w A -1 w and:

To obtain the result, we combine unstratified PPI++ with the asymptotic properties of weighted Mestimators [32]. The resulting confidence interval for θ ∗ is then derived analogously to Corollary 1.

Corollary 2. Let ̂ Σ ˆ λ w be the plug-in estimate for Σ λ w using S n and ˜ S N . Define

where z β denotes the β -quantile of the standard normal distribution. Then for any j ∈ [ d ] ,

The form of the stratified prediction-powered confidence interval is similar to that of PPI++ , except that it is based off of the weighted stratum-conditional covariance matrices. The effect of this change, however, is significant. In fact, in the case of mean estimation, we show that the asymptotic variance of StratPPI is at most that of PPI++ (even without any additional tuning of λ k and ρ k ).

Algorithm 1 Stratified prediction-powered inference for general M-estimators ( StratPPI )

Definitions: f is the autorater. Inputs { ( ˜ X ik , f ( ˜ X ik )) } N k i =1 include the autorater predictions on sampled unlabeled data for each partition A k , k ∈ [ K ] . Inputs { ( X ik , Y ik , f ( X ik )) } n k i =1 include the autorater predictions on sampled labeled data for each partition A k , k ∈ [ K ] . w k is the partition weight for A k , k ∈ [ K ] . α is the confidence interval coverage error tolerance.

1: # Pick weighting parameters, see §4.2.

2: Select ˆ λ = ( ˆ λ 1 , . . . , ˆ λ K )

3: # Solve for the minimizer of the stratified, prediction-powered empirical loss.

4: ˆ θ SPP ˆ λ = argmin θ L SPP ˆ λ ( θ )

5: # Estimate the Hessian of the true expected loss at θ ∗ .

6: ˆ A w = ∑ K k =1 w k n k ∑ n k i =1 ∇ 2 ℓ ˆ θ SPP ˆ λ ( X ik , Y ik )

7: # Build the estimated covariance matrix from each stratified component of the loss.

8: ̂ Σ ˆ λ w = 0 d × d

9: for k = 1 , 2 , . . . , K do

13: # Return coordinate-wise confidence intervals for θ ∗ .

Proposition 1. Let λ k ∈ R be any constant for all strata, and fix ρ k and ˜ ρ k to their natural rates w k . Then for ℓ θ ( x, y ) = 1 2 ∥ y -θ ∥ 2 and any stratification ( A 1 , . . . , A K ) , we have Tr(Σ λ w ) ≤ Tr(Σ λ ) , where Σ λ is the asymptotic covariance matrix of PPI++ . Furthermore, we have that equality holds if and only if both E [ Y | X ∈ A k ] and E [ f ( X ) | X ∈ A k ] are the same for all strata.

More generally, although our results will hold for arbitrary stratifications, it is best if they are heterogeneous, and chosen such that the individual stratum-conditional variances are minimized. For example, if an autorater systematically over-estimates the model's performance on one subdomain, but systematically under-estimates the model's performance on another, then splitting these subdomains into different strata can result in a lower variance L SPP λ . Similarly, if an autorater has much higher noise on some subdomains than others, it can be beneficial to stratify on those subdomains-and then either lower λ k , allocate a higher proportion of samples n k /n and N k /N , or both for the noisier stratas. In §5 we empirically demonstrate that the stratified estimator can indeed lead to considerably tighter confidence intervals in practice, especially with additional tuning of λ k and ρ k , as discussed next.

4.2 Optimal weighting of the autorater predictions

In [3] it was shown that the optimal value of λ for PPI++ (i.e., the one minimizing the variances of the estimator and the corresponding confidence interval) could be found in closed form. We now present a simple extension of this result to the stratified case. For notational convenience, we use the shorthand ∇ ℓ k,θ := ∇ ℓ θ ( X,Y ) | X ∈ A k and ∇ ℓ f k,θ := ∇ ℓ θ ( X,f ( X )) | X ∈ A k .

Proposition 2. Assume that ˜ ρ k and ρ k are fixed. Then the tuning parameters ( λ ∗ 1 , . . . , λ ∗ k ) , where

minimize the cumulative asymptotic variance, Tr(Σ λ w ) .

Note that Tr(Σ λ w ) is proportional to the total size of the confidence interval C SPP α in (14). Furthermore, as in [3], we can use plug-in estimates for the terms in (16) to compute a ˆ λ k , where ˆ λ k p → λ ∗ k . From (16), we can see that λ ∗ k is closely related to the correlation coefficient of the (curvature-scaled)

gradients for minimizing the loss on an autorater label versus a true label. Intuitively, the more correlated these terms are, the more we can rely on the autorater labels for finding the true minimizer of E [ ℓ θ ( X,Y )] . For 1 -d mean estimation in particular, we can see that λ ∗ k takes on a simple form:

Example 1 ( λ ∗ k for mean estimation) . Consider the 1 -d mean loss: ℓ θ ( x, y ) = 1 2 ( y -θ ) 2 . Then:

which is equivalent to the optimal linear regression coefficient, min λ k E [ | Y -λ k f ( X ) | 2 | X ∈ A k ] .

4.3 Optimal allocation of the sampling budget

Our stratification approach has an additional hyperparameter ρ that can also be tuned to reduce variance. Recall that ρ k determines the ratio between the labeled data size n k for stratum k and the overall data size n (i.e., ∑ k n k = n ). ˜ ρ k is similarly defined for the unlabeled data. Any strictly positive values of ρ k and ˜ ρ k are valid to be used, though not all values will improve performance. It turns out that the optimal ρ k values can be exactly calculated, as the following proposition shows.

Proposition 3. Assume that λ k is fixed. Then the sampling rates ( ρ ∗ 1 , . . . , ρ ∗ k ) and (˜ ρ ∗ 1 , . . . , ˜ ρ ∗ k ) , where

minimize the cumulative asymptotic variance, Tr(Σ λ w ) .

Although the solution for ρ ∗ k is informative, it is not necessarily practical, as it depends on knowing A -1 w V λ k k, ∆ ,θ ∗ A -1 w ; this in turn depends on θ ∗ and P ( X,Y ) , which are both unknown. 3 In the special case of mean estimation, however, it turns out there is no dependence on θ ∗ . To address the remaining dependence on P ( X,Y ) , we propose to use autorater confidence scores, assuming they are available. Specifically, assume Y is discrete, and let c ( y | x ) be the confidence of the autorater in label y given input x , where c ( y | x ) approximates P ( Y = y | X = x ) . This will result in the estimate for Tr( A -1 w V λ k k, ∆ ,θ ∗ A -1 w ) below, which can then be plugged into the expression for ρ ∗ k in Proposition 3.

Example 2 ( ρ ∗ k for mean estimation) . Consider the 1 -d mean loss: ℓ θ ( x, y ) = 1 2 ( y -θ ) 2 . Then:

(19) can then be estimated using the observed, but unlabeled, samples scored by the autorater for each stratum k , ˜ X ik = ˜ x ik , i = 1 , . . . , N k , as Var( Y -λ k f ( X ) | X ∈ A k ) ≈ ˆ σ 2 k , where

For d -dimensional data, the result is similar, but with a sum of d variances (one for each dimension). We also provide a simplified expression for ˆ σ 2 k in Appendix B when f (˜ x ) := ∑ y ∈Y c ( y | ˜ x ) · y . Importantly, as we are free to use any ρ k > 0 , using this estimate still preserves the asymptotic coverage guarantees in (14), regardless of if the confidence estimate c ( y | x ) is calibrated or not. Empirically, we show that using this heuristic can indeed lead to substantial improvements.

5 Experimental results

Wecompare our stratified estimator, StratPPI , to two baselines: (i) the classical estimate, which uses only the labeled data, S n ; and (ii) PPI++ , which uses both S n and ˜ S n . All of our experiments focus on

Homogenous

Different bias

Different variance

μ=[0, 0], α=[1, 1]

μ=[-2, 2], 0=[1, 1]

μ=[0, 0], α = [0.25, 4]

1.00

1.00

1.00

0.95

0.95

0.95

coverag

0.90

0.90

0.90

0.85

0.85

0.85

Classical

F08:0

F080

080

++ldd

0.75

0.75

0.75

StratPPI (prop.)

StratPPI (opt.)

0.70

0.70

0.70

200

400

600

800

1000

200

400

600

800

1000

200

400

600

800

1000

0.40

0.40

0.40

Classical

0.35

FSE'0

0.35

PPI++

FOE'O

FOE'O

StratPPI (prop.)

width

0.25

0.25

0.25

StratPPI (opt.)

0.20

0.20

0.20

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

200

400

600

800

1000

200

400

600

800

1000

200

400

600

800

1000

0.10

FOTO

0.10-

Classical

PPI++

mse

0.08

F80°0

800

StratPPI (prop.)

StratPPI (opt.)

root

0.06

F90°0

0.06

0.04

0.04

0.04

0.02

0.02

0.02

200

400

600

800

1000

200

400

600

800

1000

200

400

600

800

1000

n

n

n

1 -d mean estimation. We explore three different allocation strategies for StratPPI : the first is to set ρ k = w k to be data proportional ( StratPPI Prop. ), the second is to set ρ k optimally via the oracle ρ k = ρ ∗ k ( StratPPI Opt. ), and the third is to use the approximation, ρ k ∝ w k ˆ σ k , in Example 2 for λ k = 1 when confidence scores are available ( StratPPI Heur. ). We use λ -tuning for both PPI++ and StratPPI , as outlined in §4.2. Additional experimental results are given in Appendix C.

5.1 Simulation studies

We start with a simple synthetic experiment that is an analogue of §7.7.1 in [3]. Our goal is to estimate the mean outcome E [ Y ] , where Y ∼ N (0 , 1) . We assume that the input space X is partitioned into K = 2 strata, ( A 1 , A 2 ) , of equal mass P ( X ∈ A 1 ) = P ( X ∈ A 2 ) = 0 . 5 . We then assume that predictions are formed as f ( X ik ) = Y ik + µ k + σ k ϵ ik , where ϵ ik ∼ N (0 , 1) . In other words, the predictions do not depend on the covariates X ik , other than to reflect a stratum-specific noise σ k and bias µ k . We test three different scenarios: (i) where the two strata are homogeneous with µ 1 = µ 2 and σ 1 = σ 2 ; (ii) where the two strata have different prediction biases, µ 1 = µ 2 ; and (iii) where the two strata have different prediction noise levels, σ 1 = σ 2 . For each experiment, we sample N = 10 , 000 total predictions f ( ˜ X ) using ˜ ρ 1 = ˜ ρ 2 = 0 . 5 , i.e., proportional to masses of the two hypothetical, equal-weight strata. We then vary the total number n of labeled examples Y , where the allocation is chosen according to ρ (which differs depending on if we are using StratPPI Prop. or StratPPI Opt. ). We show results in Figure 1 for the mean confidence interval (CI) size and coverage (i.e., the fraction of the cases where the CI contained the true parameter value θ ∗ ) of each method, averaged over 1 k trials. As the plots in Figure 1 illustrate, when the underlying strata are homogeneous (left column), StratPPI behaves similar to PPI++ . However, when the strata are heterogeneous (middle and right columns), StratPPI outperforms both baselines significantly-whereas PPI++ becomes barely

̸

̸

CI width

Percent Reduction in CI width

Effective sample size

0.25

一

Classical

PPI++

35

++ldd —-

StratPPI(prop.)

F008

StratPPI (prop.)

30-

StratPPI (heur.)

0.20

StratPPI (heur.)

25

F009

0.15-

Summary

20

400

Classical

0.10

15

PPI++

500

10

200

StratPPI (heur.)

200

400

600

800

200

400

600

800

200

400

600

800

hen-Read

0.25-

Classical

50

PPI++

F00S

PPI++

StratPPI (prop.)

StratPPI (prop.)

40-

StratPPI (heur.)

400

Attributed

tieve-the

0.20 →

StratPPI (heur.)

0.15 -

FO

Retri

Classical

++ldd

Best

0.10

20

StratPPI (prop.)

0.05

100

StratPPI (heur.)

100

200

300

400

500

100

200

300

400

500

100

200

300

400

500

50

Classical

0.15

一

StratPPI (prop.)

PPI++

40

F0000

Galaxy

StratPPI (heur.)

0.10-

2000

Classical

+ +ldd

PPI++

0.05

20

StratPPI (prop.)

1000-

StratPPI (prop.)

StratPPI (heur.)

StratPPI (heur.)

1000

2000

3000

2000

3000

1000

2000

3000

n

n

n

more powerful than the classical estimator. Additionally, we see that when the variance differs per stratum (right column), optimal allocation of the sampling budget indeed provides additional benefit.

5.2 Real data studies

We now demonstrate how our method performs on real datasets, where the underlying structure of the autorater is unknown. To partition X , we choose to focus on stratifications that are based on the autorater's predictions, f ( X ) , based on the intuition that autorater performance can often differ across the type of predictions that it makes. Concretely, if the output space Y ′ of f is discrete, then we define A = Y ′ ; otherwise we define A based on the equal-mass quantiles of Y ′ (which can be estimated by sampling a large set of unlabeled X and applying the autorater). We set K = 10 . For all experiments, we plot performance as a function of n , where n is the number of human ratings our system is allowed to observe from each dataset. The remainder of the dataset (including any data points that are unlabeled, or labeled but with the labels removed) is used for the autorater sample ˜ S N .

Seahorse. The Seahorse dataset [11] focuses on multilingual summarization. The authors considered generative models that output summaries for a document, and collected labels for many systems that cover serveral dimensions of summary quality. We focus on one quality dimension-whether the summary is fully attributable to the source document-and on one summarization system-a finetuned 13B parameter mT5 model [33]. The autorater models for each dimension are also mT5-XXL finetuned models, which output probability scores. The data contains 2727 examples for these two tasks, all of which have both human ratings as well as autorater scores.

AttributedQA. In attributed question answering [6], the goal of the QA system is to output both an answer, and a retrieved document that provides support for that answer. The system is only considered to be correct if the answer is both correct, and indeed supported by the linked document. We evaluate the highest-scoring 'retrieve-and-read' QA system from this dataset,and define our autorater to be an 11B parameter T5 model [26] fine-tuned on a collection of natural language

entailment tasks [15]. Like Seahorse , the model predicts a probability for whether or not the QA system gave an attributable answer. This dataset has 1000 human labels and 3000 autorater labels.

Galaxy. To demonstrate the generality of StratPPI beyond LLM-based settings, we also consider the Galaxy dataset [31], where the task is to estimate the fraction of spiral galaxies in the local universe. The autorater used here is a ResNet classifier applied to images from the Sloan Digital Sky Survey (SDSS) [34]. The classifier estimates the probability that the galaxy in question is spiral. We use 16 , 743 observations from the dataset which contain both the human and autorater labels.

Methodology. We follow a procedure similar to [1; 3] to study CI estimates as a function of n . We report results on the following estimates: Classical inference, PPI++ , StratPPI Prop. and StratPPI Heur . We do not report StratPPI Opt. since it is unknown for real data. For each value of n , we sample n of the cases with human labels at an allocation rate ρ k (this rate is determined differently in StratPPI Prop. and StratPPI Heur. ). As noted above, we construct the unlabeled dataset ˜ S N by joining the remaining labeled data (without utilizing the true labels) with any unlabeled data available for that dataset. We repeat this over 1000 trials, and for each trial we obtain the CI width. We report the mean of these widths in Figure 2, as well as the percent reduction in width over the classical inference baseline, |C classical α |-|C method α | |C classical α | × 100% . We also report the 'effective sample size', which we define as the number of samples required to obtain a CI of the same width as the method at sample size n when using the classical inference baseline instead.

Results. Figure 2 shows a large improvement for StratPPI methods over both PPI++ and classical inference for most datasets. Specifically, though all CIs decrease substantially in absolute size with n as expected, we see that improvements in the percent reduction in CI width over PPI++ can be as large as 0 . 10 → 0 . 20 , 0 . 20 → 0 . 30 , and 0 . 25 → 0 . 35 points in Seahorse , AttributedQA , and Galaxy , respectively. Furthermore, we can observe that many of the datasets exhibit heterogeneous characteristics, for which heuristic allocation helps considerably. In Galaxy , this accounts for a +10 percent reduction in CI width. In Seahorse and AttributedQA , the improvements are less strong but still clearly apparent. The practical implication of these results is that when limited by a human rating budget, StratPPI is able to produce an estimate of the mean with fewer human ratings via stratification and sampling allocation. For example, for the Seahorse dataset, we can see from the right column in Figure 2 that StratPPI Heur. with only 300 human ratings will be approximately as confident about the mean as the classical estimate that utilizes 600 human ratings, a factor of 2 × .

6 Conclusion

As systems built on top of large language models continue to become more and more advanced, it becomes increasingly challenging to evaluate their performance using automatic tools. Manual labeling, on the other hand, is slow and expensive. Methods which therefore save on annotation cost are critical for reliably evaluating models, and knowing when they are improving-or degrading. Prediction-powered inference ( PPI ) is a promising class of such hybrid evaluation methods, since it can leveraged to provably produce statistically valid confidence intervals, while also effectively reducing the number of human labels needed to obtain intervals of certain width. Our results demonstrate that we can push PPI even further by introducing a method for performing even lower variance M-estimation by employing stratified sampling. In particular, we find that stratifications based on the predictions of the autoraters themselves proves to be an powerful stratification technique.

Limitations. While the confidence intervals produced by StratPPI (and PPI ) enjoy coverage guarantees, these guarantees are asymptotic. When finite-sample performance is of particular importance, techniques that afford stronger guarantees might be preferred [2; 4; 5]. We also note some non-trivial aspects of the stratified sampling setup: (i) the number of strata has to be fixed; if K scales with n a more careful treatment is required; and (ii) the assumed observation model is different from traditional i.i.d. settings-we must be able to sample fixed sized samples from each partition; and (iii) performance may not be improved if the selected stratification is not statistically useful. Furthermore, in practice, the human data might have already been collected, in which case the studied stratified sampling setup does not directly apply. We leave the study of such post-stratified estimators to future work.

Broader impacts. This paper introduces a more powerful statistical method for evaluating LLMs, by merging human evaluations with autoraters in a way that is aware of subdomain differences. Our goal is to help power more reliable evaluations with lower annotation effort, both in terms of cost and time.

Acknowledgements. We thank Jacob Eisenstein, Jonathan Berant, Anastasios Angelopoulos, Taylan Cemgil, Arnoud Doucet, and Chris Dyer for helpful comments and discussions.

References

Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science , 382(6671):669-674, 2023. doi: 10.1126/science. adi6000. URL https://www.science.org/doi/abs/10.1126/science.adi6000 .

Anastasios N. Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint: 2208.02814 , 2023.

Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zrnic. PPI++: Efficient predictionpowered inference. arXiv preprint: 2311.01453 , 2023.

Anastasios Nikolas Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint: 2110.01052 , 2021.

Stephen Bates, Anastasios Nikolas Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution free, risk controlling prediction sets. arXiv preprint: 2101.02703 , 2020.

Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models. arXiv preprint: 2212.08037 , 2023.

Pierre Boyeau, Anastasios N. Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I. Jordan. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint: 2403.07008 , 2024.

Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schuster. Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 291-305, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022.emnlp-main.20. URL https://aclanthology.org/2022.emnlp-main.20 .

Arun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in natural language evalaution. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 643-653, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1060. URL https://aclanthology.org/P18-1060 .

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint: 2403.04132 , 2024.

Elizabeth Clark, Shruti Rijhwani, Sebastian Gehrmann, Joshua Maynez, Roee Aharoni, Vitaly Nikolaev, Thibault Sellam, Aditya Siddhant, Dipanjan Das, and Ankur Parikh. SEAHORSE: A multilingual, multifaceted dataset for summarization evaluation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 9397-9413, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.584. URL https:// aclanthology.org/2023.emnlp-main.584 .

William G. Cochran. Sampling Techniques, 3rd Edition. John Wiley, 1977. ISBN 0-471-16240X.

Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning , pages 10835-10866. PMLR, 2023.

R. Alex Hofer, Joshua Maynez, Bhuwan Dhingra, Adam Fisch, Amir Globerson, and William W. Cohen. Bayesian prediction-powered inference. arXive preprint: 2405.06034 , 2024.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. True: Reevaluating factual consistency evaluation. arXiv preprint: 2204.04991 , 2022.

Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: Llm judges with provable guarantees for human agreement. arXiv preprint: 2407.18370 , 2024.

Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. Evaluating open-domain question answering in the era of large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5591-5606, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.307. URL https://aclanthology.org/2023.acl-long.307 .

Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2122-2132, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1230. URL https://aclanthology.org/D16-1230 .

S.L. Lohr. Sampling: Design and Analysis . Chapman & Hall/CRC Texts in Statistical Science. CRC Press, 2021. ISBN 9781000478235. URL https://books.google.com/books?id= DahGEAAAQBAJ .

Whitney K. Newey and Daniel McFadden. Chapter 36 large sample estimation and hypothesis testing. volume 4 of Handbook of Econometrics , pages 2111-2245. Elsevier, 1994. doi: https://doi.org/10.1016/S1573-4412(05)80005-4. URL https://www.sciencedirect.com/ science/article/pii/S1573441205800054 .

Jerzy Neyman. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society , 97(4):558-625, 1934. ISSN 09528385. URL http://www.jstor.org/stable/ 2342192 .

Alexandru Niculescu-Mizil and Rich Caruana. Obtaining calibrated probabilities from boosting. In UAI , volume 5, pages 413-20, 2005.

Jekaterina Novikova, Ondˇ rej Dušek, Amanda Cercas Curry, and Verena Rieser. Why we need new evaluation metrics for NLG. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 2241-2252, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1238. URL https://aclanthology.org/D17-1238 .

Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vlume 1: Long Papers) , 2023.

John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1-67, 2020. URL http://jmlr.org/papers/v21/20-074.html .

James M. Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122-129, 1995. ISSN 01621459. URL http://www.jstor.org/stable/2291135 .

Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint: 2311.09476 , 2024.

A. W. van der Vaart. Asymptotic Statistics . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 1998.

Siruo Wang, Tyler H McCormick, and Jeffrey T Leek. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences , 117(48):30266-30275, 2020.

Kyle W. Willett, Chris J. Lintott, Steven P. Bamford, Karen L. Masters, Brooke D. Simmons, Kevin R. V. Casteels, Edward M. Edmondson, Lucy F. Fortson, Sugata Kaviraj, William C. Keel, Thomas Melvin, Robert C. Nichol, M. Jordan Raddick, Kevin Schawinski, Robert J. Simpson, Ramin A. Skibba, Arfon M. Smith, and Daniel Thomas. Galaxy Zoo 2: detailed morphological classifications for 304 122 galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society , 435(4):2835-2860, 09 2013. ISSN 0035-8711. doi: 10.1093/mnras/stt1458. URL https://doi.org/10.1093/mnras/stt1458 .

Jeffrey M. Wooldridge. Asymptotic properties of weighted m-estimators for standard stratified samples. Econometric Theory , 17:451 - 470, 2001. URL https://api.semanticscholar. org/CorpusID:123148497 .

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934 , 2020.

Donald G. York, J. Adelman, Jr. John E. Anderson, Scott F. Anderson, James Annis, Neta A. Bahcall, J. A. Bakken, Robert Barkhouser, Steven Bastian, Eileen Berman, William N. Boroski, Steve Bracker, Charlie Briegel, John W. Briggs, J. Brinkmann, Robert Brunner, Scott Burles, Larry Carey, Michael A. Carr, Francisco J. Castander, Bing Chen, Patrick L. Colestock, A. J. Connolly, J. H. Crocker, István Csabai, Paul C. Czarapata, John Eric Davis, Mamoru Doi, Tom Dombeck, Daniel Eisenstein, Nancy Ellman, Brian R. Elms, Michael L. Evans, Xiaohui Fan, Glenn R. Federwitz, Larry Fiscelli, Scott Friedman, Joshua A. Frieman, Masataka Fukugita, Bruce Gillespie, James E. Gunn, Vijay K. Gurbani, Ernst de Haas, Merle Haldeman, Frederick H. Harris, J. Hayes, Timothy M. Heckman, G. S. Hennessy, Robert B. Hindsley, Scott Holm, Donald J. Holmgren, Chi hao Huang, Charles Hull, Don Husby, Shin-Ichi Ichikawa, Takashi Ichikawa, Željko Ivezi´ c, Stephen Kent, Rita S. J. Kim, E. Kinney, Mark Klaene, A. N. Kleinman, S. Kleinman, G. R. Knapp, John Korienek, Richard G. Kron, Peter Z. Kunszt, D. Q. Lamb, B. Lee, R. French Leger, Siriluk Limmongkol, Carl Lindenmeyer, Daniel C. Long, Craig Loomis, Jon Loveday, Rich Lucinio, Robert H. Lupton, Bryan MacKinnon, Edward J. Mannery, P. M. Mantsch, Bruce Margon, Peregrine McGehee, Timothy A. McKay, Avery Meiksin, Aronne Merelli, David G. Monet, Jeffrey A. Munn, Vijay K. Narayanan, Thomas Nash, Eric Neilsen, Rich Neswold, Heidi Jo Newberg, R. C. Nichol, Tom Nicinski, Mario Nonino, Norio Okada, Sadanori Okamura, Jeremiah P. Ostriker, Russell Owen, A. George Pauls, John Peoples, R. L. Peterson, Donald Petravick, Jeffrey R. Pier, Adrian Pope, Ruth Pordes, Angela Prosapio, Ron Rechenmacher, Thomas R. Quinn, Gordon T. Richards, Michael W. Richmond, Claudio H. Rivetta, Constance M. Rockosi, Kurt Ruthmansdorfer, Dale Sandford, David J. Schlegel, Donald P. Schneider, Maki Sekiguchi, Gary Sergey, Kazuhiro Shimasaku, Walter A. Siegmund, Stephen Smee, J. Allyn Smith, S. Snedden, R. Stone, Chris Stoughton, Michael A. Strauss, Christopher Stubbs, Mark SubbaRao, Alexander S. Szalay, Istvan Szapudi, Gyula P. Szokoly, Anirudda R. Thakar, Christy Tremonti, Douglas L. Tucker, Alan Uomoto, Dan Vanden Berk, Michael S. Vogeley, Patrick Waddell, Shu i Wang, Masaru Watanabe, David H. Weinberg, Brian Yanny, and Naoki Yasuda. The sloan digital sky survey: Technical summary. The Astronomical Journal , 120(3):1579, sep 2000. doi: 10.1086/301513. URL https://dx.doi. org/10.1086/301513 .

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2023. URL https://openreview.net/forum?id=uccHPGDlao .

Tijana Zrnic and Emmanuel J. Candès. Cross-prediction-powered inference. Proceedings of the National Academy of Sciences , 121(15):e2322083121, 2024. doi: 10.1073/pnas.2322083121. URL https://www.pnas.org/doi/abs/10.1073/pnas.2322083121 .

Tijana Zrnic and Emmanuel J. Candès. Active statistical inference. arXiv preprint: 2403.03208 , 2024.

A Proofs

We begin by defining the basic regularity conditions that we assume ℓ θ satisfies: Definition A.1 (Regularity conditions of ℓ θ ) . Assume that

(i) Θ is a compact subset of R d ;

(ii) The minimizer θ ∗ ∈ int(Θ) is unique with E [ ∇ ℓ θ ∗ ( X,Y )] = 0 ;

(iii) For all ( x, y ) ∈ X × Y , ℓ θ ( x, y ) is twice continuously differentiable on int(Θ) ;

(iv) For all θ ∈ Θ , ℓ θ ( X,Y ) , ∂ℓ θ ( X,Y ) ∂θ i , and ∂ 2 ℓ θ ( X,Y ) ∂θ i ∂θ j have finite expectations;

(v) E [ ∇ 2 ℓ θ ( X,Y )] is non-singular.

These regularity conditions are fairly mild-for example, it is straightforward to verify that the loss of the mean estimator, 1 2 ∥ y -θ ∥ 2 , satisfies Definition A.1. We note, however, that PPI++ can be applied to merely 'smooth enough' losses, which also include non-continuously differentiable functions losses like the quantile loss. As we primarily focus on mean estimation in this work, for simplicity of analysis we only consider (the still broad class) of losses satisfying Definition A.1, though our results can be expected to readily extend to the more general case following a similar treatment as in [3; 29].

To support our theoretical results, we also provide the following two lemmas.

LemmaA.1 (Slutsky's Theorem, general form) . Let X n d → X and Y n p → c , where X n , X , and Y n are random vectors, and c is a constant vector. Then for any continuous function g , g ( X n , Y n ) d → g ( X,c ) .

Proof. By Theorem 2.7 of [29] we have ( X n , Y n ) d → ( X,c ) . The continuous mapping theorem then implies that g ( X n , Y n ) d → g ( X,c ) .

Lemma A.2. Under the regularity conditions of Definition A.1, we have that ˆ θ SPP ˆ λ p → θ ∗ .

Proof. Under the regularity conditions of Definition A.1 and the fact that ˆ λ k p → λ k where λ k is constant, the uniform weak law of large numbers can be applied to each term in L SPP ˆ λ ( θ ) so that

As θ ∗ is unique, Θ is compact, and ℓ θ is continuous it follows from Theorem 2.1 of [20] that

A.1 Proof of Theorem 2

Proof. For ease of notation, we will define

We will also use the following shorthand for the conditional gradients:

As samples are i.i.d. within each stratum, the CLT gives

and

Since samples S n and ˜ S N are independent from each other, we also have that (22) and (23) converge jointly. Applying Lemma A.1 for the following continuous function of ˆ λ k p - → λ k then gives

Since θ ∗ satisfies E [ ∇ ℓ θ ∗ ] = 0 , by the law of total expectation

Using this fact, we can write

Since samples across the K (where here K is a constant) stratas are independent, combining the results of (25) with (27) yields

Applying the mean value theorem, we have

for some ˜ θ between ˆ θ SPP ˆ λ and θ ∗ . Let A † represent the pseudoinverse of A . Then

As ˆ θ SPP ˆ λ p → θ ∗ per Lemma A.2, we have ˜ θ p → θ ∗ . Under the regularity conditions of Definition A.1 and the fact that ˆ λ k p - → λ k , an application of the uniform weak law of large numbers and the continuous mapping theorem then gives

which is a constant term. Furthermore, by the law of total expectation,

Finally, the fact that ˆ θ SPP ˆ λ p → θ ∗ and θ ∗ ∈ int(Θ) , together with the fact that ˆ θ SPP ˆ λ is a minimum of L SPP ˆ λ , implies that ∇ L SPP ˆ λ ( ˆ θ SPP ˆ λ ) p → 0 . Combining these results with (28) via Lemma A.1 and the fact that A -1 w is symmetric for twice continuously differentiable ℓ θ gives

A.2 Proof of Corollary 2

Proof. The regularity conditions of Definition A.1 allow us to apply Lemma 4.3 of [20] to show that each of the plug-in estimates for each stratum term satisfies

which implies that the weighted plug-in estimate for ˆ Σ ˆ λ w satisfies ˆ Σ ˆ λ w p → Σ λ w . (14) is thus equivalent to taking the ( α 2 , 1 -α 2 ) quantiles of the asymptotic normal distribution of ˆ θ SPP , implying (15).

A.3 Proof of Proposition 1

Proof. In the special case of the square loss, the Hessian is the identity matrix. Therefore, simplifying and applying the linearity of the trace, we have

where we include the subscript jj to index the diagonal variance terms of both covariance matrices. For ease of notation, denote the conditional variances as

Then we can write (37) as

and apply the law of total variance to get

Finally, (41) holds with equality iff both Var( E [ V f,j | Z ]) = 0 and Var( E [ V ∆ ,j | Z ]) = 0 , which, combined with the assumption that P ( Z = k ) = w k > 0 for all k , is only satisfied when both E [ V f,j | Z ] and E [ V ∆ ,j | Z ] are constants.

A.4 Proof of Proposition 2

Proof. By linearity of B λ w , we can rewrite Σ λ w = A -1 w B λ w A -1 w as

By linearity of the trace, we then also have

As each term in the sum is independent, we can minimize the sum by minimizing each individual term, i.e.,

for k = 1 , . . . , K . We can then directly apply Proposition 2 of [3] to get

where r k = n k /N k .

A.5 Proof of Proposition 3

Proof. Following the same derivation as Proposition 2, we can rewrite

and therefore can optimize ˜ ρ k and ρ k independently.

We start with ρ k . Let z k = Tr ( A -1 w V λ k k, ∆ ,θ ∗ A -1 w ) . We then solve the constrained optimization problem

Turning this into a Lagrangian with additional slack variables s k ≥ 0 , we have

Setting ∇J ( ρ, µ, η ) = 0 and solving for ρ k then gives:

Assume that the inequality constraint is inactive, and η k = 0 . Solving for ρ ∗ k ,

We can now verify that the constraint is inactive, with ρ ∗ k ≥ 0 , since ∑ K k =1 w k = 1 , w k ≥ 0 by definition of a valid probability distribution, and we have z k ≥ 0 since it is a sum of non-negative variance terms.

The same analysis can then be applied to ˜ ρ k , but for z k = Tr ( A -1 w V λ k k,f,θ ∗ A -1 w ) .

B A simplified estimate of the sample allocation for mean estimation

In the setting of Example 2 (i.e., 1 -d mean estimation), assume that c ( y | x ) ≈ P ( Y = y | X = x ) is a confidence estimate for label y ∈ Y , where Y is discrete. Then, if we define the autorater as

the term Var( Y -λ k f ( X ) | X ∈ A k ) further simplifies to ≈ Var( Y | X ∈ A k ) for any value of λ k . Applying the law of total variance, we can conveniently write Var( Y | X ∈ A k ) as

which can be empirically estimated as ˆ σ k on unlabeled autorater data, ˜ X ik , i = 1 , . . . , N k , via

where ̂ E N k and ̂ Var N k denote the empirical mean and variance over all ˜ X ik , respectively. Lastly, when Y is binary (which is the case in all of our experiments in §5.2), this becomes

which can be easily calculated in Python as np.mean(yhat * (1 -yhat)) + np.var(yhat) .

C Additional experimental results

We also evaluate StratPPI on the Chatbot Arena dataset [10], in which we evaluate the win-rate of gpt-4-1106-preview over claude-2.1 . 4 See Figure 3. Specifically, we use the standard Chatbot

CI width

PercentReductioninClwidth

Effectivesamplesize

Classical

30

一

+ +ldd

Arena

0.20

+ +ldd

StratPPI (prop.)

20

StratPPI(prop.)

StratPPI (heur.)

600-

StratPPI (heur.)

0.15

400

10

0.10

++ldd

Classical

200

StratPPI (prop.)

0

StratPPI (heur.)

0.05

200

400

600

200

400

600

200

400

600

Arena auto-eval prompt 5 and map [[A»B]] , [[A>B] to 1, [[A=B]] to 0.5, and [[B»A]] , [[B>A]] to 0. We then do self-consistency sampling and take the average of 10 samples from Gemini Ultra. This is used as both our final autorater estimate and confidence. We find that stratification also helps in this setting. However, in line with prior and contemporary work [e.g., 35; 16] we found confidence scores from the LLM-as-a-judge to be fairly uncalibrated, which makes our heuristic allocation less effective at larger n (though still effective at small labeled sample sizes n ). Investigating robust heuristics in the face of miscalibration (e.g., via regularization or recalibration) would likely help, and is a direction for future work.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims are supported in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We address this in the conclusion.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proofs are included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Results are based on publically available data, and simulations, and all procedures are described.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The data is publicly available, and the methodology is simple to implement. Pseudocode is given. Code may be made available at a future date.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Detailed information is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are provided for the main experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources required are very light, as no model training is performed.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: No issues.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the advantages of the method for more effective development of large language models.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: NA

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
Taking inspiration from stratified sampling, this paper proposes a method named Stratified Prediction-Powered Inference (StratePPI). 
With appropriate choices of stratification and sample allocation, StratePPI can provide substantially tighter confidence intervals than unstratified approaches.  The experimental results on simulated and real data show the effectiveness of StratePPI.

STRENGTHS:
1. This paper introduces data stratification strategies to improve prediction-powered inference (PPI) for the first time.
2. The experimental results show that StratPPI is more effective than PPI++.

WEAKNESSES:
The stratification is very important to StratPPI, and there is little introduction on how to stratify data effectively in this paper.

QUESTIONS:
In practical applications, how should we stratify data to ensure that StratPPI is better than PPI or PPI++?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
The paper proposes an extension of prediction-powered inference (PPI) - a method for calculating confidence intervals with narrower confidence bands. The data is separated into subsets and the bias rectification term in the confidence calculation is weighted based on the similarity to the particular subsets. Evaluation is performed on one synthetic dataset and 4 real datasets, showing narrower confidence bands if the data is heterogeneous.

STRENGTHS:
The paper is well written. Even the necessary background, regarding existing work on PPI, is explained well enough to understand.
The adjustment of the calculation based on data similarity makes sense.
Empirical evaluation on several datasets is good to see.

WEAKNESSES:
Line 107 defines Yi as the target output, which in the case of QA would normally be a textual answer. Line 112 then defines Y as the binary gold rating of correctness of the answer. This is contradictory and causes confusion when following the other equations. 

The chosen evaluation datasets don't seem to be particularly widely used. Evaluation on some more common QA datasets would have been more impactful.

It is unclear whether this method actually helps in calculating more accurate means, or whether it only produces narrower confidence bands. And if it is the latter, then can you give more details on the practical benefits? It's a different method of calculating the confidence bands so the bands would not be directly comparable to other methods and couldn't really say that the new ones are "better" than the previous ones.

QUESTIONS:
Please see above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14

comments: SUMMARY:
This paper introduces the stratified PPI method, designed to reduce evaluation bias from autoraters by aligning them with a few human annotations. The key novelty lies in identifying that the autorater's performance varies across different example groups, with biases being smaller for some examples and larger for others. Consequently, they propose a straightforward method to group examples into strata and calculate stratum-specific rectification parameters. Experiments demonstrate that stratified PPI consistently achieves tighter confidence intervals compared to baselines.

STRENGTHS:
The paper introduces a novel method, stratified PPI, which effectively addresses the issue of evaluation bias in autoraters by considering the varying performance across different groups of examples. The proposed method of grouping examples into strata and calculating stratum-specific rectification parameters is both simple and effective.

WEAKNESSES:
The evaluations are conducted on simple evaluation problems where the evaluation is binary, while current LLM autoraters use a more fine-grained scales (e.g., 1-5). Additionally, real human evaluations may be multinomial, where people's opinion will diverge on the same instance. I'm not sure how this proposed method will perform in that scenario.

QUESTIONS:
See weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the predictive power inference methods to leverage data stratification strategies, and demonstrates that stratification can be used to obtain unbiased statistical estimates, while reducing variance (with worst case being as bad as prior approaches PPI++, and in practice considerably better). Concretely, the authors\n\n1. Demonstrate that for the stratified prediction-powered loss, the minimizer has an asymptotically normal distribution with the true mean, thereby allowing for construction of confidence intervals leveraging stratum conditioned covariance matrices.\n2. Provide an method for choosing the optimal weighting between the human annotations and autograder annotations for the stratified setting.\n3. Provide the closed form solution for optimal allocation for stratum weights for the labeled and unlabeled data. Furthermore, for the special case of mean estimation, the authors also provide a heuristic way of estimating these weights based on the autograders' confidence for the scores.\n4. Demonstrate for simulations that for cases where the strata are homogeneous, the estimators are at worse as bad as PPI++, while in cases where the strata are heterogeneous, StratPPI outperforms PPI++.\n5. Finally, the authors demonstrate the utility of the proposed approach on a real-world multilingual summarization benchmark, an attributed QA dataset as well as the galaxy dataset, demonstrating that StratPPI obtains tighter confidence intervals compared to other approaches.\n\nSTRENGTHS:\n1. The proposed method demonstrably produces tighter confidence interval, leveraging data stratification. Such stratification is quite natural to occur in generally available data, in my opinion. So the proposed extension should be very widely applicable in a real world setting.\n2. The paper is relatively easy to follow even for someone who is not very familiar with this subdomain, and is quite self contained.\n\nWEAKNESSES:\nGiven the focus of the paper on effective evaluation of language models, it would have been good for the paper to have a stronger focus on evaluation of the numerous publicly available (large) language models. Concretely, imo, the proposed method should be extendible to assessing pairwise relative performance of language models (similar to Section 3.1 of [1]). It would be good to compare confidence intervals for the BT coefficients obtained from PPI++ vs StratPPI for (a subset of) different Language Models on the Chatbot arena.\n\n[1] Boyeau, Pierre, et al. \"AutoEval Done Right: Using Synthetic Data for Model Evaluation.\" arXiv preprint arXiv:2403.07008 (2024).\n\nQUESTIONS:\nFor Figure 1, column 3, what is the intuition about why is the coverage of StratPPI (prop) so low compared to all other counterparts when n is small ?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}