Stratified Prediction-Powered Inference for Effective Hybrid Evaluation of Language Models

Authors: adam  fisch
joshua  maynez
r alex hofer
bhuwan  dhingra
amir  globerson
william w cohen
google  deepmind
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 23:33:11

Abstract: Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data.  PPI achieves this by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate---but potentially biased---automatic system, in a way that results in tighter confidence intervals for certain parameters of interest (e.g., the mean performance of a language model). In this paper, we propose a method called Stratified Prediction-Powered Inference (StratPPI), in which we show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies. Without making any assumptions on the underlying automatic labeling system or data distribution, we derive an algorithm for computing provably valid confidence intervals for parameters of any dimensionality that is based on stratified sampling. In particular, we show both theoretically and empirically that, with appropriate choices of stratification and sample allocation, our approach can provide substantially tighter confidence intervals than  unstratified  approaches. Specifically, StratPPI  is expected to improve in cases where the performance of the autorater varies across different conditional distributions of the target data.

Main paper: 
 introduction: 
 Evaluating machine learning models requires evaluation data. In particular, to iteratively improve on a method during development, or to reliably report an improvement, one needs to confidently and quantitatively assess the performance of the method. This is especially challenging for large language models (LLMs), where gathering high-quality annotations for generations can be difficult and timeconsuming-and can ultimately become quite costly if gathering more than a few hundred examples.One often-proposed approach to avoiding the evaluation bottleneck is to use a secondary LLM-based system to judge the output of the primary one. For instance, if the primary task is developing an LLM-based question-answering (QA) system, one can use a second LLM-based system that rates question/answer pairs as acceptable or not [6; 8; 17] . However, automated raters (i.e., autoraters) may be biased relative to the human raters they are intended to model, as others have noted [1; 9; 18; 23] . This can become substantially worse when models are tailored to hill climb on the autorater metrics, and eventually cease to become a good metric at all-a phenomenon commonly referred to as Goodhart's law, or reward hacking in the context of reinforcement learning from human feedback [13; 24] . We thus have two signals for assessing model performance. The first is human labels, which are typically accurate, but expensive to collect. As a result, usually only a small sample size is available, and an estimate based on these samples alone will have high variance. The second is autorater predictions, which are easy to collect for large sample sizes, but may also be systematically biased. The above may suggest that one must make a choice between either (i) a high-variance, but unbiased, estimate from a small human sample, or (ii) a lower-variance, but biased, autorater-based estimate. However, it turns out there are also statistically valid ways of combining the auto-rater and human data for hybrid evaluations. Following [1; 3; 7] we call such methods prediction-powered inference (PPI) methods. At a high level, PPI-based methods operate by using a small sample of examples labeled by both humans and autoraters to estimate the bias of the autorater. This bias is then used as a rectifier for the autorater estimate. The resulting estimate can then be shown to provide improved (i.e., tighter) confidence intervals for properties of interest for the system being evaluated, such as its true mean accuracy.A weakness of standard PPI, however, is that it does not take heterogeneity into account. For example, in our QA setting, an autorater may have one accuracy when predicting if a model answer is correct, and a different accuracy when predicting if it is incorrect. This is especially true in cases where a correct answer is easy to verify (e.g., it is also present in Wikipedia), but harder to refute (e.g., no relevant external search results can be retrieved to either support or contradict it). In these settings, it may make sense to apply a different PPI strategy within each subdomain, depending on the local quality of the autorater. Moreover, we claim that such heterogenous settings are to be quite expected in practical applications. Inspired by the rich prior literature on stratified sampling and survey design [12; 19; 21] , we therefore propose a stratified approach to PPI, and show that it can be very advantageous when performance varies across subdomains, for either the autorater, or the model being evaluated.On a technical level, it is not immediately clear how to apply stratification to PPI, as it involves two types of samples: one sample that is labeled by both humans and an autorater, and another (typically much larger) sample that is only labeled by the autorater. Extending the analysis of [3] , in this work we show how confidence intervals based on the asymptotic normality of weighted M-estimators [32] can in fact be derived for the stratified PPI setting. The next challenge we address is how to determine the sample sizes used for stratification (i.e., the sample size of each stratum). Similar to recent work for active statistical inference [37] , we further derive optimal rates that depend on certain moments of the underlying distribution that are generally unknown-and provide an approach for effectively approximating these. Finally, we provide extensive empirical evidence showing that our stratified approach (StratPPI) leads to considerably improved confidence intervals over both classical inference methods that use only human labels, and the baseline (unstratified) PPI approach. 
related work: 
 Prediction Powered Inference (PPI) was introduced in [1] as a method for obtaining tighter confidence intervals for parameters learned in supervised machine learning (e.g., coefficients in logistic regression) by also leveraging other model-based predictions on additional, unlabeled data. Related ideas were explored by [30] , but with a focus on bootstrapping as a way for obtaining confidence intervals. PPI was then extended in several directions. For example [36] showed how the labeled data can be used for both estimating the parameters and the autorater model. PPI++ [3] showed how to obtain confidence intervals that are easy to compute efficiently, and introduced a parameter for weighting the predictions of the autorater such that the overall statistical efficiency can be improved. As noted in previous works on PPI, these approaches are closely related to other statistical methods for introducing control variates based on autoevaluators [9] , as well as augmented inverse propensity weighting [27] (see discussion in [1; 3] ). Like this paper, prior work has also focused on using PPI/PPI++ for evaluating machine learning systems with autoraters, including for ranking chatbots in the Chatbot Arena [7] and evaluating retrieval-augmented generation systems [28] .Most relevant to our setting, the recent work of [37] focuses on active sampling of examples to label during PPI, where the total number of queried labels is random, but less than the sampling budget n in expectation. Specifically, it proposes to label examples for which the autorater is less confident in its predictions, and corrects for the sampling bias with a variant of inverse propensity weighting. In contrast, our somewhat simplified approach takes inspiration from stratified sampling where a coarse stratification is defined in advance, and the total number of labeled samples is constant. Like [37] , we derive an analogous optimal allocation strategy for a given budget, though we only apply it at the stratum level, and not for individual examples. Furthermore, while the variable allocation helps reduce variance in heterogenous settings, our stratified treatment also allows for a stratum-specific extension of [3] 's estimated tuning parameters that further improves performance in complementary ways.In terms of stratification specifically, concurrent work [14] also proposed Bayesian variants of several PPI methods, including stratified approaches. In contrast, the methods here do not require introducing priors, do not require running expensive Bayesian inference, and give more conventional, frequentist, guarantees of performance. The credible intervals produced by Bayesian methods are related to, but different from, the confidence intervals produced here, and by other prior PPI approaches [1; 3; 7] .Finally, there is a conceptual similarity between the PPI rectifiers and post-hoc calibration of a classifier. There is a rich literature on calibrating classifiers (e.g., [22; 25] ), but there is a clear difference in goals between calibration and PPI: the former is aimed at modifying a learned autorater to make better probabilistic predictions, and the latter is aimed at obtaining better confidence intervals by making use of an existing autorater. That said, clearly one approach to improving PPI is to use a better-calibrated classifier, perhaps by taking some of the labeled data and using it for re-calibration. This is similar in motivation to the cross-PPI approach of [36] . We leave such approaches as future work, but we do experimentally explore PPI approaches on both well-calibrated autoraters and poorly calibrated ones. 
preliminaries: 
 We begin by briefly reviewing PPI [1] , and specifically the efficient PPI++ variant of [3] . Here, and in the rest of the paper, upper-case letters (X) denote random variables; lower-case letters (x) denote constants, and script letters (X ) denote sets, unless specified. All proofs are deferred to Appendix A.Following [1; 3; 7] , we assume an empirical sampleS n = {(X 1 , Y 1 ), . . . , (X n , Y n )} of n i.i.d.examples drawn from some unknown, but fixed, distribution P, where X i ∈ X is an input, and Y i ∈ Y is the target output. We also assume a larger sample SN = {( X1 , f ( X1 )), . . . , ( XN , f ( XN ))} where N ≫ n, for which the target outputs are not available, but we have access to an autorater function f : X → Y ′ which provides an approximation of Y given the observed input (we use Y ′ to denote the output space of the autorater as it is possible thatY ̸ = Y ′ , e.g., if f (X) = E[Y | X]).As an example, assume X is a (question, model answer) tuple where the answer is produced by an LLM-based QA system we wish to evaluate, Y is a 0/1 "gold" human rating of correctness of the answer, i.e., 1{human rates model answer as correct}, and f (X), the autorater, imperfectly predicts this human rating Y . One value of interest is then the average accuracy of the QA system, E[Y ].More generally, given any convex loss function ℓ θ (x, y) satisfying certain regularity conditions (see Definition A.1), we are interested in estimating the d-dimensional parameter θ * ,EQUATIONIn the statistics literature, this is broadly referred to as M-estimation (e.g., see [29] ). Here, we focus on mean estimation, for which the loss ℓ θ (x, y) =1 2 ∥y -θ∥ 2 has the optimum θ * = E[Y ]. However, our method generalizes to other typical losses such as the squared loss for linear regression, where the parameters θ * are the regression coefficients, or, similarly, the log loss for other generalized linear models. Of course, the true θ * is generally not known, because we cannot exactly calculate the expectation in (1) . The goal of PPI, which we share in this work, is to use both the labeled and unlabeled data S n and SN to obtain an asymptotically valid confidence interval C α,j for θ * j that for any j ∈EQUATIONThe simplest confidence interval can be obtained using standard techniques by using only the labeled sample S n , and ignoring the autorater data Sn . However, PPI employs a clever trick which allows for also using the unlabeled sample SN to get tighter confidence intervals, as described next.The key idea of PPI-based methods is to use autorater predictions to derive a low variance, but unbiased, estimate of the objective in (1) . Consider the following "rectified" prediction-powered loss:L PP (θ) = 1 N N i=1 ℓ θ ( Xi , f ( Xi ))autorater data lossEQUATIONThe rectifier term on the right hand side of (3) removes the bias of the autorater data so thatL PP (θ) satisfies E[L PP (θ)] = E [ℓ θ (X, Y )]. However, when f (X) is correlated with Y , the loss L PP (θ) will have lower variance. Unfortunately, f (X) may not be a good predictor of Y in all cases. In fact, it is even possible for L PP to be higher variance than the classical estimate (e.g., if f (X) is anti-correlated with Y ). Thus, to adapt to the quality of f (X), PPI++ also introduces a tuning parameter λ ∈ R:2 EQUATIONClearly, L PP λ still remains unbiased for any value of λ. For λ = 0, the framework reduces to classical inference. In most cases, however, a proper choice of λ ̸ = 0 will result in lower variance. PPI++ also suggests how to automatically choose a data-dependent λ that converges to an optimal value.We next describe the PPI++ method for deriving confidence intervals based on the rectified loss. LetEQUATIONbe the prediction-powered estimate. Standard analysis for M-estimators can then be extended to show that the scaled difference of the estimate θPP λ and the true θ * is asymptotically normally distributed. PPI++ then leverages this result to compute a confidence interval for θ * that is asymptotically valid.Theorem 1 (PPI++, [3] ). Assume thatEQUATIONwhere λ ∈ R is a hyper-parameter. Then under the regularity conditions of Definition A.1, we have that [3] ). Let Σ λ be the plug-in estimate for Σ λ using S n and SN . Define√ n( θPP λ -θ * ) d → N (0, Σ λ ), where Σ λ := H -1 θ * (r • V λ f,θ * + V λ ∆,θ * )H -1 θ * . Corollary 1 (PPI++ CI,EQUATIONwhere z β denotes the β-quantile of the standard normal distribution. Then for any j ∈ [d] it holds that lim n,N →∞ P θ * j ∈ C PP α,j ≥ 1 -α.As mentioned above, [3] further showed that for an appropriate choice of λ (that can be analytically derived and estimated via a λ), the trace of the covariance matrix, Tr(Σ λ ), is at most that of the covariance matrix derived without using autorater data, implying that the PPI++-based confidence set C PP α can always be at least as tight as that of the classical M-estimator (and often much tighter in practice). 
stratified prediction-powered inference: 
 We now present our approach for improving PPI++ estimates via stratification. In particular, we show how optimizing a rectified loss computed via stratified sampling can lead to a consistent, but lower variance, estimate of the optimal parameter θ * , and correspondingly tighter confidence intervals. Consider the QA example from the previous section. For most autoraters, it is reasonable to assume that the strength of their performance can vary, depending on the type of input being presented. For instance, an autorater might be accurate at predicting whether an answer to an unambiguous question (e.g., "What is the capital of France?") is correct, but relatively poor at inferring if an answer to an open-ended question (e.g., "What is the best way to cook a hamburger?") is acceptable or not.Splitting the problem space into different domains allows us to derive a more specialized form of the prediction powered loss that can better adapt to this autorater heterogeneity via stratified sampling.Formally, assume that the input space X is partitioned in advance into K non-empty, mutually exclusive, and exhaustive strata A = (A 1 , . . . , A K ), where K is a finite integer. For each stratum, we further assume that we can estimate w k = P(X ∈ A k ) arbitrarily well using large amounts of unlabeled data or prior knowledge; we treat them as known constants here for simplicity. To collect the labeled and unlabeled datasets S n and SN , we then follow a standard stratified sampling procedure in which we draw two i.i.d. sets of samples of fixed size n k and N k , respectively, from each stratum k, whereK k=1 n k = n and K k=1 N k = N .The relative sizes n k /n and N k /N are free parameters; they can simply be the natural rates, n k /n ≈ N k /N ≈ w k , or systematically decided (see §4.3). Note that this is a fundamentally different sampling model from standard PPI: here examples are i.i.d. within each strata, and independent (but not necessarily identically distributed) across each strata.Next, we define the stratified loss via the weighted sum,L SPP λ (θ) = K k=1 w k L PP k,λ k (θ), where w k is the stratum weight, λ = (λ 1 , . . . , λ k ) ∈ R k are now stratum-specific tuning parameters, and L PP k (θ) is the conditional prediction-powered loss computed within each stratum, i.e.,EQUATIONAs before, each L PP k,λ k is an unbiased estimate of the conditional loss. Like PPI++, we also allow for data-dependent parameters λk that we show how to automatically tune for best performance in §4.2.We now present our main result, which is a confidence interval for θ * based on the stratified loss. More precisely, the result states that, as in PPI++, the minimizer of the stratified loss,EQUATIONhas an asymptotically normal distribution with mean θ * . See Algorithm 1 for pseudocode.Theorem 2. Assume that λk p → λ k , n N → r for any r ≥ 0, n k n → ρ k for any ρ k > 0, and N k N → ρk for any ρk > 0. Let H k,θ * := E[∇ 2 ℓ θ * (X, Y ) | X ∈ A k ],andEQUATIONEQUATIONwhere λ k ∈ R is a hyper-parameter. Then, under the regularity conditions of Definition A.1,EQUATIONwhere Σ λ w := A -1 w B λ w A -1 w and:EQUATIONTo obtain the result, we combine unstratified PPI++ with the asymptotic properties of weighted Mestimators [32] . The resulting confidence interval for θ * is then derived analogously to Corollary 1.Corollary 2. Let Σ λ w be the plug-in estimate for Σ λ w using S n and SN . DefineEQUATIONwhere z β denotes the β-quantile of the standard normal distribution. Then for any j ∈EQUATION)The form of the stratified prediction-powered confidence interval is similar to that of PPI++, except that it is based off of the weighted stratum-conditional covariance matrices. The effect of this change, however, is significant. In fact, in the case of mean estimation, we show that the asymptotic variance of StratPPI is at most that of PPI++ (even without any additional tuning of λ k and ρ k ).Definitions: f is the autorater. Inputs {( Xik , f ( Xik ))} N k i=1 include the autorater predictions on sampled unlabeled data for each partitionA k , k ∈ [K]. Inputs {(X ik , Y ik , f (X ik ))} n ki=1 include the autorater predictions on sampled labeled data for each partitionA k , k ∈ [K]. w k is the partition weight for A k , k ∈ [K]. α is the confidence interval coverage error tolerance.1: # Pick weighting parameters, see §4. 6: Âw = K k=1 w k n k n k i=1 ∇ 2 ℓ θSPP λ (X ik , Y ik )7: # Build the estimated covariance matrix from each stratified component of the loss. 8: Σ λ w = 0 d×d 9: for k = 1, 2, . . . , K do 10:V f = λ2 k Cov N k ∇ℓ θSPP λ ( Xik , f ( Xik ))11:V ∆ = Cov n k ∇ℓ θSPP λ (X ik , Y ik ) + λk ∇ℓ θSPP λ (X ik , f (X ik ))12:Σ λ w = Σ λ w + w 2 k Â-1 w V f N k + V∆ n k Â-1 w 13: # Return coordinate-wise confidence intervals for θ * . 14: C SPP α = θSPP j ± z 1-α 2 Σ λ w,jj : j ∈ [d]Proposition 1. Let λ k ∈ R be any constant for all strata, and fix ρ k and ρk to their natural rates w k . Then for ℓ θ (x, y) = 1 2 ∥y -θ∥ 2 and any stratification (A 1 , . . . , A K ), we have Tr(Σ λ w ) ≤ Tr(Σ λ ), where Σ λ is the asymptotic covariance matrix of PPI++. Furthermore, we have that equality holds if and only if bothE[Y | X ∈ A k ] and E[f (X) | X ∈ A k ] are the same for all strata.More generally, although our results will hold for arbitrary stratifications, it is best if they are heterogeneous, and chosen such that the individual stratum-conditional variances are minimized. For example, if an autorater systematically over-estimates the model's performance on one subdomain, but systematically under-estimates the model's performance on another, then splitting these subdomains into different strata can result in a lower variance L SPP λ . Similarly, if an autorater has much higher noise on some subdomains than others, it can be beneficial to stratify on those subdomains-and then either lower λ k , allocate a higher proportion of samples n k /n and N k /N , or both for the noisier stratas. In §5 we empirically demonstrate that the stratified estimator can indeed lead to considerably tighter confidence intervals in practice, especially with additional tuning of λ k and ρ k , as discussed next.In [3] it was shown that the optimal value of λ for PPI++ (i.e., the one minimizing the variances of the estimator and the corresponding confidence interval) could be found in closed form. We now present a simple extension of this result to the stratified case. For notational convenience, we use the shorthand∇ℓ k,θ := ∇ℓ θ (X, Y ) | X ∈ A k and ∇ℓ f k,θ := ∇ℓ θ (X, f (X)) | X ∈ A k . Proposition 2.Assume that ρk and ρ k are fixed. Then the tuning parameters (λ * 1 , . . . , λ * k ), whereEQUATIONminimize the cumulative asymptotic variance, Tr(Σ λ w ).Note that Tr(Σ λ w ) is proportional to the total size of the confidence interval C SPP α in (14) . Furthermore, as in [3] , we can use plug-in estimates for the terms in (16) to compute a λk , where λk p → λ * k . From ( 16), we can see that λ * k is closely related to the correlation coefficient of the (curvature-scaled)gradients for minimizing the loss on an autorater label versus a true label. Intuitively, the more correlated these terms are, the more we can rely on the autorater labels for finding the true minimizer of E[ℓ θ (X, Y )]. For 1-d mean estimation in particular, we can see that λ * k takes on a simple form: Example 1 (λ * k for mean estimation). Consider the 1-d mean loss: ℓ θ (x, y) = 1 2 (y -θ) 2 . Then:EQUATIONwhich is equivalent to the optimal linear regression coefficient, minλ k E[|Y -λ k f (X)| 2 | X ∈ A k ].Our stratification approach has an additional hyperparameter ρ that can also be tuned to reduce variance. Recall that ρ k determines the ratio between the labeled data size n k for stratum k and the overall data size n (i.e., k n k = n). ρk is similarly defined for the unlabeled data. Any strictly positive values of ρ k and ρk are valid to be used, though not all values will improve performance. It turns out that the optimal ρ k values can be exactly calculated, as the following proposition shows. Proposition 3. Assume that λ k is fixed. Then the sampling rates (ρ * 1 , . . . , ρ * k ) and (ρ * 1 , . . . , ρ * k ), whereEQUATIONminimize the cumulative asymptotic variance, Tr(Σ λ w ).Although the solution for ρ * k is informative, it is not necessarily practical, as it depends on knowingA -1 w V λ k k,∆,θ * A -1w ; this in turn depends on θ * and P(X, Y ), which are both unknown. 3 In the special case of mean estimation, however, it turns out there is no dependence on θ * . To address the remaining dependence on P(X, Y ), we propose to use autorater confidence scores, assuming they are available. Specifically, assume Y is discrete, and let c(y | x) be the confidence of the autorater in label y given input x, where c(y | x) approximates P(Y = y | X = x). This will result in the estimate forTr(A -1 w V λ k k,∆,θ * A -1w ) below, which can then be plugged into the expression for ρ * k in Proposition 3. Example 2 (ρ * k for mean estimation). Consider the 1-d mean loss: ℓ θ (x, y) = 1 2 (y -θ) 2 . Then:EQUATION(19) can then be estimated using the observed, but unlabeled, samples scored by the autorater for each stratum k,EQUATIONEQUATIONFor d-dimensional data, the result is similar, but with a sum of d variances (one for each dimension).We also provide a simplified expression for σ2 k in Appendix B when f (x) := y∈Y c(y | x) • y. Importantly, as we are free to use any ρ k > 0, using this estimate still preserves the asymptotic coverage guarantees in (14) , regardless of if the confidence estimate c(y | x) is calibrated or not. Empirically, we show that using this heuristic can indeed lead to substantial improvements. 
experimental results: 
 We compare our stratified estimator, StratPPI, to two baselines: (i) the classical estimate, which uses only the labeled data, S n ; and (ii) PPI++, which uses both S n and Sn . All of our experiments focus on 1-d mean estimation. We explore three different allocation strategies for StratPPI: the first is to set ρ k = w k to be data proportional (StratPPI Prop.), the second is to set ρ k optimally via the oracle ρ k = ρ * k (StratPPI Opt.), and the third is to use the approximation, ρ k ∝ w k σk , in Example 2 for λ k = 1 when confidence scores are available (StratPPI Heur.). We use λ-tuning for both PPI++ and StratPPI, as outlined in §4.2. Additional experimental results are given in Appendix C.We start with a simple synthetic experiment that is an analogue of §7.7.1 in [3] . Our goal is to estimate the mean outcome E[Y ], where Y ∼ N (0, 1). We assume that the input space X is partitioned into K = 2 strata, (A 1 , A 2 ), of equal mass P(X ∈ A 1 ) = P(X ∈ A 2 ) = 0.5. We then assume that predictions are formed as f (X ik ) = Y ik + µ k + σ k ϵ ik , where ϵ ik ∼ N (0, 1). In other words, the predictions do not depend on the covariates X ik , other than to reflect a stratum-specific noise σ k and bias µ k . We test three different scenarios: (i) where the two strata are homogeneous with µ 1 = µ 2 and σ 1 = σ 2 ; (ii) where the two strata have different prediction biases, µ 1 ̸ = µ 2 ; and (iii) where the two strata have different prediction noise levels, σ 1 ̸ = σ 2 . For each experiment, we sample N = 10,000 total predictions f ( X) using ρ1 = ρ2 = 0.5, i.e., proportional to masses of the two hypothetical, equal-weight strata. We then vary the total number n of labeled examples Y , where the allocation is chosen according to ρ (which differs depending on if we are using StratPPI Prop. or StratPPI Opt.). We show results in Figure 1 for the mean confidence interval (CI) size and coverage (i.e., the fraction of the cases where the CI contained the true parameter value θ * ) of each method, averaged over 1k trials. As the plots in Figure 1 illustrate, when the underlying strata are homogeneous (left column), StratPPI behaves similar to PPI++. However, when the strata are heterogeneous (middle and right columns), StratPPI outperforms both baselines significantly-whereas PPI++ becomes barely more powerful than the classical estimator. Additionally, we see that when the variance differs per stratum (right column), optimal allocation of the sampling budget indeed provides additional benefit.We now demonstrate how our method performs on real datasets, where the underlying structure of the autorater is unknown. To partition X , we choose to focus on stratifications that are based on the autorater's predictions, f (X), based on the intuition that autorater performance can often differ across the type of predictions that it makes. Concretely, if the output space Y ′ of f is discrete, then we define A = Y ′ ; otherwise we define A based on the equal-mass quantiles of Y ′ (which can be estimated by sampling a large set of unlabeled X and applying the autorater). We set K = 10. For all experiments, we plot performance as a function of n, where n is the number of human ratings our system is allowed to observe from each dataset. The remainder of the dataset (including any data points that are unlabeled, or labeled but with the labels removed) is used for the autorater sample SN .Seahorse. The Seahorse dataset [11] focuses on multilingual summarization. The authors considered generative models that output summaries for a document, and collected labels for many systems that cover serveral dimensions of summary quality. We focus on one quality dimension-whether the summary is fully attributable to the source document-and on one summarization system-a finetuned 13B parameter mT5 model [33] . The autorater models for each dimension are also mT5-XXL finetuned models, which output probability scores. The data contains 2727 examples for these two tasks, all of which have both human ratings as well as autorater scores.AttributedQA. In attributed question answering [6] , the goal of the QA system is to output both an answer, and a retrieved document that provides support for that answer. The system is only considered to be correct if the answer is both correct, and indeed supported by the linked document. We evaluate the highest-scoring "retrieve-and-read" QA system from this dataset,and define our autorater to be an 11B parameter T5 model [26] fine-tuned on a collection of natural language entailment tasks [15] . Like Seahorse, the model predicts a probability for whether or not the QA system gave an attributable answer. This dataset has 1000 human labels and 3000 autorater labels. Galaxy. To demonstrate the generality of StratPPI beyond LLM-based settings, we also consider the Galaxy dataset [31] , where the task is to estimate the fraction of spiral galaxies in the local universe. The autorater used here is a ResNet classifier applied to images from the Sloan Digital Sky Survey (SDSS) [34] . The classifier estimates the probability that the galaxy in question is spiral. We use 16,743 observations from the dataset which contain both the human and autorater labels.Methodology. We follow a procedure similar to [1; 3] to study CI estimates as a function of n. We report results on the following estimates: Classical inference, PPI++, StratPPI Prop. and StratPPI Heur. We do not report StratPPI Opt. since it is unknown for real data. For each value of n, we sample n of the cases with human labels at an allocation rate ρ k (this rate is determined differently in StratPPI Prop. and StratPPI Heur.). As noted above, we construct the unlabeled dataset SN by joining the remaining labeled data (without utilizing the true labels) with any unlabeled data available for that dataset. We repeat this over 1000 trials, and for each trial we obtain the CI width. We report the mean of these widths in Figure 2 , as well as the percent reduction in width over the classical inference baseline,|C classical α |-|C method α | |C classical α | × 100%.We also report the "effective sample size", which we define as the number of samples required to obtain a CI of the same width as the method at sample size n when using the classical inference baseline instead.Results. Figure 2 shows a large improvement for StratPPI methods over both PPI++ and classical inference for most datasets. Specifically, though all CIs decrease substantially in absolute size with n as expected, we see that improvements in the percent reduction in CI width over PPI++ can be as large as 0.10 → 0.20, 0.20 → 0.30, and 0.25 → 0.35 points in Seahorse, AttributedQA, and Galaxy, respectively. Furthermore, we can observe that many of the datasets exhibit heterogeneous characteristics, for which heuristic allocation helps considerably. In Galaxy, this accounts for a +10 percent reduction in CI width. In Seahorse and AttributedQA, the improvements are less strong but still clearly apparent. The practical implication of these results is that when limited by a human rating budget, StratPPI is able to produce an estimate of the mean with fewer human ratings via stratification and sampling allocation. For example, for the Seahorse dataset, we can see from the right column in Figure 2 that StratPPI Heur. with only 300 human ratings will be approximately as confident about the mean as the classical estimate that utilizes 600 human ratings, a factor of 2×. 
conclusion: 
 As systems built on top of large language models continue to become more and more advanced, it becomes increasingly challenging to evaluate their performance using automatic tools. Manual labeling, on the other hand, is slow and expensive. Methods which therefore save on annotation cost are critical for reliably evaluating models, and knowing when they are improving-or degrading. Prediction-powered inference (PPI) is a promising class of such hybrid evaluation methods, since it can leveraged to provably produce statistically valid confidence intervals, while also effectively reducing the number of human labels needed to obtain intervals of certain width. Our results demonstrate that we can push PPI even further by introducing a method for performing even lower variance M-estimation by employing stratified sampling. In particular, we find that stratifications based on the predictions of the autoraters themselves proves to be an powerful stratification technique.Limitations. While the confidence intervals produced by StratPPI (and PPI) enjoy coverage guarantees, these guarantees are asymptotic. When finite-sample performance is of particular importance, techniques that afford stronger guarantees might be preferred [2; 4; 5] . We also note some non-trivial aspects of the stratified sampling setup: (i) the number of strata has to be fixed; if K scales with n a more careful treatment is required; and (ii) the assumed observation model is different from traditional i.i.d. settings-we must be able to sample fixed sized samples from each partition; and (iii) performance may not be improved if the selected stratification is not statistically useful. Furthermore, in practice, the human data might have already been collected, in which case the studied stratified sampling setup does not directly apply. We leave the study of such post-stratified estimators to future work.Broader impacts. This paper introduces a more powerful statistical method for evaluating LLMs, by merging human evaluations with autoraters in a way that is aware of subdomain differences. Our goal is to help power more reliable evaluations with lower annotation effort, both in terms of cost and time.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Detailed information is provided.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Error bars are provided for the main experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discusses the advantages of the method for more effective development of large language models.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: NA Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA] Justification: NA Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We begin by defining the basic regularity conditions that we assume ℓ θ satisfies: Definition A.1 (Regularity conditions of ℓ θ ). Assume that(i) Θ is a compact subset of R d ; (ii) The minimizer θ * ∈ int(Θ) is unique with E[∇ℓ θ * (X, Y )] = 0;(iii) For all (x, y) ∈ X × Y, ℓ θ (x, y) is twice continuously differentiable on int(Θ);(iv) For all θ ∈ Θ, ℓ θ (X, Y ), ∂ℓ θ (X,Y ) ∂θi , and ∂ 2 ℓ θ (X,Y ) ∂θi∂θj have finite expectations; (v) E[∇ 2 ℓ θ (X, Y )] is non-singular.These regularity conditions are fairly mild-for example, it is straightforward to verify that the loss of the mean estimator, 1 2 ∥y -θ∥ 2 , satisfies Definition A.1. We note, however, that PPI++ can be applied to merely "smooth enough" losses, which also include non-continuously differentiable functions losses like the quantile loss. As we primarily focus on mean estimation in this work, for simplicity of analysis we only consider (the still broad class) of losses satisfying Definition A.1, though our results can be expected to readily extend to the more general case following a similar treatment as in [3; 29] .To support our theoretical results, we also provide the following two lemmas. Proof. By Theorem 2.7 of [29] we have (X n , Y n ) d → (X, c). The continuous mapping theorem then implies that g(X n , Y n ) d → g(X, c).sup θ∈Θ L SPP λ (θ) -E[ℓ θ (X, Y )] p → 0.As θ * is unique, Θ is compact, and ℓ θ is continuous it follows from Theorem 2.1 of [20] thatθSPP λ p → θ * .Proof. For ease of notation, we will defineLf N k (θ) = 1 N k N k i=1 ℓ θ ( Xik , f ( Xik ) L n k (θ) = 1 n k n k i=1 ℓ θ (X ik , Y ik ) L f n k (θ) = 1 n k n k i=1 ℓ θ (X ik , f (X ik )).We will also use the following shorthand for the conditional gradients:∇ℓ k,θ := ∇ℓ θ (X, Y ) | X ∈ A k and ∇ℓ f k,θ := ∇ℓ θ (X, f (X)) | X ∈ A k .As samples are i.i.d. within each stratum, the CLT givesEQUATIONEQUATION)Since samples S n and SN are independent from each other, we also have that ( 22) and ( 23) converge jointly. Applying Lemma A.1 for the following continuous function of λk p -→ λ k then givesEQUATIONEQUATIONSince θ * satisfies E[∇ℓ θ * ] = 0, by the law of total expectationEQUATIONUsing this fact, we can writeEQUATIONSince samples across the K (where here K is a constant) stratas are independent, combining the results of ( 25) with (27) yieldsEQUATIONApplying the mean value theorem, we haveEQUATIONfor some θ between θSPP λ and θ * . Let A † represent the pseudoinverse of A. ThenEQUATIONAs θSPP λ p → θ * per Lemma A.2, we have θ p → θ * . Under the regularity conditions of Definition A.1 and the fact that λk p -→ λ k , an application of the uniform weak law of large numbers and the continuous mapping theorem then givesEQUATIONwhich is a constant term. Furthermore, by the law of total expectation,EQUATIONFinally, the fact that θSPP w is symmetric for twice continuously differentiableEQUATIONA.2 Proof of Corollary 2Proof. The regularity conditions of Definition A.1 allow us to apply Lemma 4.3 of [20] to show that each of the plug-in estimates for each stratum term satisfiesH k, θSPP p → H k,θ * and V λ k k,f, θSPP p → V λ k k,f,θ * and V λ k k,∆, θSPP p → V λ k k,∆,θ *which implies that the weighted plug-in estimate for Σλ w satisfies Σλ w p → Σ λ w . ( 14) is thus equivalent to taking the α 2 , 1 -α 2 quantiles of the asymptotic normal distribution of θSPP , implying (15) .Proof. In the special case of the square loss, the Hessian is the identity matrix. Therefore, simplifying and applying the linearity of the trace, we haveEQUATIONEQUATION= rEQUATIONwhere we include the subscript jj to index the diagonal variance terms of both covariance matrices. For ease of notation, denote the conditional variances asV f,j | Z = k := λ 2 ∇ℓ θ * j (X, f (X)) | X ∈ A k and V ∆,j | Z = k := ∇ℓ θ * j (X, Y ) -λ∇ℓ θ * j (X, f (X)) | X ∈ A k .Then we can write (37) asEQUATIONand apply the law of total variance to getEQUATIONEQUATIONEQUATION= Tr(Σ λ ).Finally, (41) holds with equality iff both Var(E[V f,j | Z]) = 0 and Var(E[V ∆,j | Z]) = 0, which, combined with the assumption that P(Z = k) = w k > 0 for all k, is only satisfied when bothE[V f,j | Z] and E[V ∆,j | Z] are constants.Proof. By linearity of B λ w , we can rewriteEQUATIONBy linearity of the trace, we then also haveEQUATION)As each term in the sum is independent, we can minimize the sum by minimizing each individual term, i.e.,EQUATION)for k = 1, . . . , K. We can then directly apply Proposition 2 of [3] to getEQUATIONwhere r k = n k /N k .Proof. Following the same derivation as Proposition 2, we can rewriteEQUATIONand therefore can optimize ρk and ρ k independently.We start with ρ k . Letz k = Tr A -1 w V λ k k,∆,θ * A -1 w. We then solve the constrained optimization problemEQUATIONTurning this into a Lagrangian with additional slack variables s k ≥ 0, we have Scores are based on the average label ('better' = 1 vs. 'worse' = 0) over 10 samples from Gemini Ultra acting as a LLM judge. Interestingly, as these confidence scores are not calibrated, our heuristic becomes overly aggressive at higher n. Future work can explore how to best incorporate additional regularization into the estimated optimal sampling ratios ρ.EQUATIONArena auto-eval prompt 5 to 0. We then do self-consistency sampling and take the average of 10 samples from Gemini Ultra. This is used as both our final autorater estimate and confidence. We find that stratification also helps in this setting. However, in line with prior and contemporary work [e.g., 35 ; 16] we found confidence scores from the LLM-as-a-judge to be fairly uncalibrated, which makes our heuristic allocation less effective at larger n (though still effective small labeled sample sizes n). Investigating robust heuristics in the face of miscalibration (e.g., via regularization or recalibration) would likely help, and is a direction for future work.Answer: [Yes] Justification: The data is publicly available, and the methodology is simple to implement. Pseudocode is given. Code may be made available at a future date.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.In d-dimensions, we obtain a confidence interval for each coordinate. See[3] for other possible choices.As noted in[3], for some losses ℓ θ we must have λ ∈ [0, 1] to guarantee convexity of L PP λ . The main focus of this paper, however, is on mean estimation with ℓ θ (x, y) = 1 2 ∥y -θ∥ 2 , which is convex for any λ ∈ R.Similarly, the optimal solution to ρk also depends on the unknown θ * in general, though this term is less important to optimize if we assume N to be large. In practice, we always keep ρk fixed to the natural rate, w k .https://huggingface.co/datasets/lmsys/lmsys-arena-human-preference-55khttps://github.com/lm-sys/arena-hard-auto/blob/main/config/judge_config.yaml 


Decision: This paper presents the stratified PPI method, aimed at minimizing evaluation bias from autoraters by calibrating them against a limited number of human annotations. One particular innovation is the recognition that an autorater's accuracy differs across various groups of examples, with some groups exhibiting smaller biases and others showing larger ones. Based on this insight, the authors propose a simple approach to categorize examples into distinct strata and compute adjustment parameters for each stratum. Experimental results indicate that the stratified PPI method consistently produces more precise confidence intervals than existing baseline methods.

While the reviewers see the merit in the paper, a repeated weakness has to do with the evaluation: the number of LLMs evaluated the nature of the evaluation task and data etc. While this is something to be addressed, and the authors recognize that and have even added an experiment, it seems that the reviewers are willing to accept the paper in its current form and I respect that.

Reviews: review: paper_summary: paper_summary: The paper extends the predictive power inference methods to leverage data stratification strategies, and demonstrates that stratification can be used to obtain unbiased statistical estimates, while reducing variance (with worst case being as bad as prior approaches PPI++, and in practice considerably better). Concretely, the authors

1. Demonstrate that for the stratified prediction-powered loss, the minimizer has an asymptotically normal distribution with the true mean, thereby allowing for construction of confidence intervals leveraging stratum conditioned covariance matrices.
2. Provide an method for choosing the optimal weighting between the human annotations and autograder annotations for the stratified setting.
3. Provide the closed form solution for optimal allocation for stratum weights for the labeled and unlabeled data. Furthermore, for the special case of mean estimation, the authors also provide a heuristic way of estimating these weights based on the autograders' confidence for the scores.
4. Demonstrate for simulations that for cases where the strata are homogeneous, the estimators are at worse as bad as PPI++, while in cases where the strata are heterogeneous, StratPPI outperforms PPI++.
5. Finally, the authors demonstrate the utility of the proposed approach on a real-world multilingual summarization benchmark, an attributed QA dataset as well as the galaxy dataset, demonstrating that StratPPI obtains tighter confidence intervals compared to other approaches.
strength_weakness: strength_weakness: Given the focus of the paper on effective evaluation of language models, it would have been good for the paper to have a stronger focus on evaluation of the numerous publicly available (large) language models. Concretely, imo, the proposed method should be extendible to assessing pairwise relative performance of language models (similar to Section 3.1 of [1]). It would be good to compare confidence intervals for the BT coefficients obtained from PPI++ vs StratPPI for (a subset of) different Language Models on the Chatbot arena.

[1] Boyeau, Pierre, et al. "AutoEval Done Right: Using Synthetic Data for Model Evaluation." arXiv preprint arXiv:2403.07008 (2024).
questions: questions: For Figure 1, column 3, what is the intuition about why is the coverage of StratPPI (prop) so low compared to all other counterparts when n is small ?
limitations: limitations: NA
paper_summary: paper_summary: This paper introduces the stratified PPI method, designed to reduce evaluation bias from autoraters by aligning them with a few human annotations. The key novelty lies in identifying that the autorater's performance varies across different example groups, with biases being smaller for some examples and larger for others. Consequently, they propose a straightforward method to group examples into strata and calculate stratum-specific rectification parameters. Experiments demonstrate that stratified PPI consistently achieves tighter confidence intervals compared to baselines.
strength_weakness: strength_weakness: The evaluations are conducted on simple evaluation problems where the evaluation is binary, while current LLM autoraters use a more fine-grained scales (e.g., 1-5). Additionally, real human evaluations may be multinomial, where people's opinion will diverge on the same instance. I'm not sure how this proposed method will perform in that scenario.
questions: questions: See weaknesses.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: The paper proposes an extension of prediction-powered inference (PPI) - a method for calculating confidence intervals with narrower confidence bands. The data is separated into subsets and the bias rectification term in the confidence calculation is weighted based on the similarity to the particular subsets. Evaluation is performed on one synthetic dataset and 4 real datasets, showing narrower confidence bands if the data is heterogeneous.
strength_weakness: strength_weakness: Line 107 defines Yi as the target output, which in the case of QA would normally be a textual answer. Line 112 then defines Y as the binary gold rating of correctness of the answer. This is contradictory and causes confusion when following the other equations. 

The chosen evaluation datasets don't seem to be particularly widely used. Evaluation on some more common QA datasets would have been more impactful.

It is unclear whether this method actually helps in calculating more accurate means, or whether it only produces narrower confidence bands. And if it is the latter, then can you give more details on the practical benefits? It's a different method of calculating the confidence bands so the bands would not be directly comparable to other methods and couldn't really say that the new ones are "better" than the previous ones.
questions: questions: Please see above
limitations: limitations: Yes
paper_summary: paper_summary: Taking inspiration from stratified sampling, this paper proposes a method named Stratified Prediction-Powered Inference (StratePPI). 
With appropriate choices of stratification and sample allocation, StratePPI can provide substantially tighter confidence intervals than unstratified approaches.  The experimental results on simulated and real data show the effectiveness of StratePPI.
strength_weakness: strength_weakness: The stratification is very important to StratPPI, and there is little introduction on how to stratify data effectively in this paper.
questions: questions: In practical applications, how should we stratify data to ensure that StratPPI is better than PPI or PPI++?
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Clarifying a misunderstanding
Response to rebuttal
Thanks
comment: Glad to hear your explanation; it is indeed a commendable research effort.
Thank you for the review and the response to our rebuttal! We agree that the LM evaluation results should be included in the main paper, and will make sure to put them there.
We thank the reviewer for their reply to our rebuttal, and appreciate the opportunity to engage.


> You can't really use the same symbol for very different things 10 lines apart without redefining it first, so I would suggest changing that.


To restate our response from earlier: the definition of Y_i does not change. Y_i is always the target output for the autorater. The target output for the autorater is always a measure of quality for the input X_i (e.g., if it is correct, helpful, toxic, etc). Y_i never refers to the  target output of the LLM-based QA system, this is always part of X (see  L111). The autorater represents a separate prediction problem from the QA system it is evaluating. We do appreciate the reviewer’s confusion, and will definitely make the problem statement clearer and the distinction explicit.


> Additional results are good, although you could have included a citation or source to show exactly which version of the dataset you are using.


Our apologies for the oversight in the rebuttal! Thanks for pointing this out. We used the dataset available at https://huggingface.co/datasets/lmsys/lmsys-arena-human-preference-55k. The citation for which is:
```
@misc{chiang2024chatbot,
    title={Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference},
    author={Wei-Lin Chiang and Lianmin Zheng and Ying Sheng and Anastasios Nikolas Angelopoulos and Tianle Li and Dacheng Li and Hao Zhang and Banghua Zhu and Michael Jordan and Joseph E. Gonzalez and Ion Stoica},
    year={2024},
    eprint={2403.04132},
    archivePrefix={arXiv},
    primaryClass={cs.AI}
}
```


Of course, this will be properly specified and cited in our final revision.


> I am still unsure about the practical benefits. The provided justification sounds analogous to getting non-significant results using a statistical significance calculation, then switching to a different significance calculation method that indicates a significant result. Even though the measurement changes, the actual data and the real result stay the same. So the benefit should be somewhere else.


There is a substantial misunderstanding here.


"the measurement changes, the actual data... stay the same"


This is false. The point of stratified PPI is that the data collection strategy has changed. There is no repeated testing of any kind: it is simply an estimator computed on a strategically partitioned and collected dataset. You can think of Stratified PPI as analogous to experimental design in statistics: deciding which parts of X-space to oversample/undersample in order to get maximum-precision estimates. To make it possible to aggregate the samples taken from each strata, Stratified PPI also benefits from being able to effectively use additional unlabeled data not only for making predictions, but also for estimating the propensity weights for the partitions of this X-space.


To keep it simple, the reason for improvements is simple: the estimator computed on this data is __lower-variance__ than the standard PPI estimator. The resulting practical benefit is then also simple: we can get better point estimates (lower MSE) with higher confidence (smaller confidence intervals). This allows for better evaluations, and decision-making based on those evaluations, at less cost.
The additional experiments address my concerns. If possible, it would be good for the results for the LM evaluations to be included in the main paper instead of being relegated to the appendix. 
I have increased the confidence for my review. In my opinion, this work is pretty solid.
Thanks for clarifying. You can't really use the same symbol for very different things 10 lines apart without redefining it first, so I would suggest changing that.

Additional results are good, although you could have included a citation or source to show exactly which version of the dataset you are using.

I am still unsure about the practical benefits. The provided justification sounds analogous to getting non-significant results using a statistical significance calculation, then switching to a different significance calculation method that indicates a significant result. Even though the measurement changes, the actual data and the real result stay the same. So the benefit should be somewhere else.

This is somewhat outside of my field of expertise, which I have also indicated with the low confidence score, and I'm not comfortable raising my score higher than it currently is.