Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression
Authors: Benjamin Eyre
David Madras
Venue: ICML 2025 poster (2025)
Keywords: auto-evaluation
prediction-powered-inference
regression
variance-reduction
Primary area: general_machine_learning->evaluation

Abstract: The availability of machine learning systems that can effectively perform arbitrary tasks has led to synthetic labels from these systems being used in applications of statistical inference, such as data analysis or model evaluation. The Prediction Powered Inference (PPI) framework provides a way of leveraging both a large pool of pseudo-labelled data and a small sample with real, high-quality labels to produce a low-variance, unbiased estimate of the quantity being evaluated for. Most work on PPI considers a relatively sizable set of labelled samples, which can be resource intensive to obtain.  However, we find that when labelled data is scarce, the PPI++ method can perform even worse than classical inference. We analyze this phenomenon by relating PPI++ to ordinary least squares regression, which also experiences high variance with small sample sizes, and use this regression framework to better understand the efficacy of PPI. Motivated by this, we present two new PPI-based techniques that leverage robust regressors to produce even lower variance estimators in the few-label regime

Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression
Benjamin Eyre
David Madras
Abstract
systematic enrors (DeGrave et al., 2021). Accurate evalua- tion in the era of large language models (LLMs) presents scaling challenge. Traditional evaluation schemes involve collecting several annotated samples from human labellers, and with ML systems quickly changing (Tu et al., 2023), the repeated collection of this data can quickly exhaust time and resources.
The availability of machine learning systems that can effectively perform arbitrary tasks has led to synthetic labels from these systems being used in applications of statistical inference, such as data analysis or model evaluation. The Prediction Pow- ered Inference (PPI) framework provides way of leveraging both large pool of pseudo-labelled data and a small sample with real, high-qu hugh-quality labels to produce low-variance, unbiased esti- mate of the quantity being evaluated for. Most work on PPI considers a relatively sizable set of labelled samples, which can be resource intensive to obtain. However, we find that when labelled data is scarce, the PPI++ method can perform even worse than classical inference. We analyze this phenomenon by relating PPI++ to ordinary least squares regression, which also experiences high variance with small sample sizes, and use this regression framework to better understand the efficacy of PPI. Motivated by this, we present PPI-based techniques that leverage ro- bust regressors to produce even lower variance estimators in the few-label regime.
Recent advances in the development of LLMs have resulted in the widespread availability of reasonably strong predic- tive models for arbitrary tasks. The proliferation these models has made the practice of replacing human-annotated labels with the outputs of an LLM for the sake of evaluation, or automatic evaluation, more feasible. Inherent bias in these LLM predictions leads to potentially inaccurate evalu- ations, even in the case where many examples are available (Angelopoulos et al., 2023a). Proposed frameworks such as Prediction Powered Inference (PPI) (Angelopoulos et al., 2023a) offer a way to remove the statistical bias these predictions with a small pool of labelled data.
While previous works studying the PPI framework have focussed on circumstances where 50 or more (usually 200+) labelled examples are available (Boyeau et al., 2024; An- gelopoulos et al., 2023b; Zrnic & Candès, 2024b;a), the applicability of PPI in the few-label regime has not been thoroughly investigated - in this work, we find that empiri- cally, PPI performs poorly (sometimes worse than classical inference) when very few labelled examples are available. We argue this is an important case, many developers of ML systems will rarely have access to large evaluation set for every criteria they wish to evaluate their system for, e.g. when developing generative models, important evaluations are often highly qualitative and potentially time-consuming. In these cases, developers may wish to use a small pool of hand labelledelled examples as a way to guide design decisions.
1. Introduction
The deployment of machine learning (ML) systems into sev- eral areas of modern life has the potential lead to several positive outcomes, such as faster and more precise medical care (Rajpurkar et al., 2017), improved educational tools (Jaiswal & Arun, 2021; Lee, 2024), and wide variety of professional assistive technologies (Dehaerne et al., 2022). However, the ever increasing integration of these systems has resulted in a more drastic need for effective evaluation techniques for determining when these systems suffer from
With this in mind, ensuring evaluation with few labels is as efficient and accurate as possible is important for the development of reliable ML systems. Specifically in the case of generative model evaluation, the PPI framework is a natural candidate, as the large set of unlabelled data PPI requires can be directly generated by the model itself. Effi- cient inference with few labels is also relevant to research in the sciences, where collecting more data may be costly but
Work partially completed while this author was Google DeepMind. 'Columbia University "Google DeepMind. Correspon- dence to: Benjamin Eyre ebencyre@ess columbia.edu>.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Regression for the Mcan: Auto-Evaluation with Few Labels through Post-hoc Regression
annotating unlabelled data may be less resource intensive. In this paper, we work towards the goal of improving few- label auto-evaluation by proposing modifications to the PPI framework which can produce lower variance estimates in the low-label regime. Specifically, we do this through em- phasizing and re-interpreting the critical role of the tuning parameter Our contributions are:
can consider another binary function, S(X) E {0, 13 that is meant to approximate h h(X). We can estimate H4.r by instead taking the sample mean of finite pseudolabelled sample DN = {(X". (x,))) Seeing as unlabelled data X is often plentiful and f(X) can be generated without human intervention, one can efficiently reach a value of N where the variance of the sample mean is minimized. However, fX) may be biased. More rigorously:
1. Providing a theoretical analysis of the PPI++ method in the context of mean estimation, and how it relates to univariate regression.
P = [,,==E(((((),, MM..--M,P > O.
(2)
2. Positing two new extensions for PPI inspired by uni- variate regression techniques that improve upon the base method.
If this is true, regardless of how large our sample of N examples is, estimating the mean using pseudolabels will have an irreducible amount of error.
3. Applying these techniques to the task of feature gener- ation rate estimation in research and LLM evaluation.
In response to the shortcomings of this automatic evalua- tion approach, Angelopoulos et al. (2023a) leverage a small pool of in-distribution labelled examples along with a larger sample of pseudolabelled examples to create a low vari- ance unbiased estimator in the Prediction Powered Inference (PPI) framework. The PPI estimate takes as input both the labelled dataset D, and the pscudolabelled dataset DN as well as a tuning parameter À E IR, and estimates P as:
2. Background
In this section, we provide details on the estimation prob- lems we approach, their associated data generative process, and its relationship with contemporary automatic evaluation schemes.
2.1. Estimating Feature Generation Rate
1 N N
1 n + h(X,) (f(x)), (3)
APPI ==
xf(x))
We are interested in the properties of some outputs X gen- erated from some distribution P, X ~ P. These outputs can consist of any modality (e.g. text, images, video) and the distribution can take any form (e.g. a complex generative model). Consider the binary function h(X) E {0, }} that outputs if X presents certain feature of interest, and 0 if X does not present that feature. (X) can represent quanti- tative features such as the presence of a word in text or the value of a certain pixel in an image, or more subjective fea- tures such as toxicity in text. This notion can be extended to non-binary functions, but this work will focus on the binary case. We wish to estimate the quantity EE [E [h(X)]. The best known estimator is the classical estimator, which is simply the sample mean from finite sample of n example, label pairs Dn, = {(,, P(
This approach, as well as the methods that have succeeded it (Angelopoulos et al., 2023b; Zrnic & Candès, 2024a; Boyeau et al., 2024), makes use of both the strong statis- tical power that traditional automatic evaluation schemes promise, well as the asymptotic benefits of being unbi- ased provided by traditional evaluation.
3. Related Work
Since its inception, the PPI framework has been the subject of much investigation, both in terms of application and improving the method. Zrnic & Candès (2024a) propose technique for determining which samples within a batch of unlabelled data to collect labels for to achieve the best possible performance with PPI. Zrnic & Candès (2024b) show how one can use cross-fitting to be able to use PPI even when pre-trained predictive model is unavailable. Fisch et al. (2024) demonstrate that using subgroup information can provably improve the estimates from PPI. While these works focus on how to apply the PPI framework under different assumptions, we instead focus improving the efficacy of the method in its original setting. Boyeau et al. (2024) were the first to propose how the PPI framework can be used to improve contemporary automatic evaluation schemes. We build upon this work by investigating new use cases and improving performance in the low-label regime.
Ph.P: = Ep(((X)), pn
Σ M(X.)
(I)
This estimate is unbiased, and its variance will decay in order J.. However, in the regime where labels are scarce or expensive to procure the variance on the sample mcan will be high, leading to low-fidelity estimates of the mean.
2.2. Prediction Powered Inference for Mean Estimation
We next examine how a strong predictive model could be used to complete this estimation task. In our set up, we
2

---

Regression for the Mcan: Auto-Evaluation with Few Labels through Post-hoc Regression
Several techniques for supplementing model's training data with synthetic data have been proposed. This includes both data augmentation techniques such as Mixup (Zhang, 2017) and those used in reinforcement learning (Pitis et al., 2022; Osiáski et al., 2020). Improved generative models have provided new avenue for researchers and practitioners alike to create new datasets for both training and evaluation (Neuhausen al., 2020). We contribute this branch of research by building upon a method for evaluating generative model with synthetic data generated by the model itself.
Note that we are able to distribute the variance operator this way as the two samples are i.i.d. Given that we assume that << N (e.g, we can generate or collect large amount of unlabeled data cheaply), Var [D]]]]] will be negligible and we may turn our focus to Var [P.]. We can further decompose this variance as:
[ar
= Σ
Varlh(X,) - Af(X,))
n² i=1
n
(5)
Varlb(X)] + X²
The use of an additional, correlated variable as a way to re- duce the variance of an estimator is a well studied technique in the fields of statistics and machine learning. Previous works have specifically investigated how one can reduce the variance in mean estimation using control variate con- structed with unlabelled data in a regime known as semi- supervised learning (Zhang et al., 2019; Zhang& & Bradic, 2022). Furthermore, the use of different regression methods and control variates has also been explored (South et al., 2023; Blanchet et al., 2024). In this work, we use this ex- isting literature to better understand and improve the PPI++ method.
- ox [()],
The first term in this sum equal to the variance of the classi- cal estimate (the labelled sample mean) for ja. P. Therefore, ùPPI will have lower variance than the sample mean esti- mate whenever X² (x)] 2XCov(1(XX) [(X] < 0.
One can find a simple expression for the optimal À given Dn, by simply taking the derivative of Equation 5 with respect to X and setting it to zero, which yields:
AOpt Coolh(X), Var[f(X]]
(6)
4. Using Regression for Improving PPI
In this section, we discuss our approach to improving the PPI estimator in the few-label regime. Our approach pri- oritizes reducing variance and focuses on the role of À in PPI, re-interpreting the task of choosing À post-hoc re- gression. We first draw the connection between optimizing A and linear regression, and then discuss how choosing À correctly can be challenging in the low-label regime, result- ing in high variance estimates. We then highlight two tools from the regression literature which can be used to improve these estimates, and demonstrate how they fit into the PPI framework.
Notably, Angelopoulos et al. (2023b) choose À to minimize both V Var[D, [Pn) and Var[DNx and arrive at similar expres- sion'. The quantity expressed in Equation 6 is known as the regression coefficient (Kenney & Keeping, 1962), and is the solution to univariate ordinary least squares regression.
We emphasize this insight into the role of À post-hoc re- gressor. Previous works have encouraged an interpretation where À is intended to interpolate between two estimators: classical estimation at X = 0 and "real PPI" at X = 1. In- deed, Boyeau et al. (2024) describe À as a "tuning parameter in [0.1] ... When the synthetic data is very good, we can set À 1; when it bad, setting À 0 will throw it away entirely." However, we suggest that À may more accurately be interpreted as a post-hoc transformation of J to make it "closer to" h;; that is, minimizing PPI variance through equiv- alently minimizing the mean-squared error of the post-hoc regression problem, which has the same form as Equation 5:
4.1. Optimal and the Regression Coefficient
To understand how selecting À appropriately can reduce the variance of RiPII, we first provide an expression for the variance of (aPPI, and decompose it into Var and V [r[]::
1
(M(X,) - Af(X.) 6)² = Varlh(X)) + +ar(Af(X))
Var[ApPI] = Varl 1 N {f(x;)
72
2=
N 1=1
2Coo
[er[Dy]
(4)
We note that when n is small, this expression is unlikely to have converged in probability to constant. Consequently, fitting A the labelled data may result frrr being biased estimator in this small n regime.
+ Varl
h(X.) - X((XI)
[P.]
3

---

Regression for the Mcan: Auto-Evaluation with Few Labels through Post-hoc Regression
and admits the same minimizer (derivation details in Ap- pendix B). Here, b in - Puy is the optimal ordinary least squares intercept coefficient, which is the difference between the sample means of the target h(X) and trans- formed input XX). We note that this interpretation better aligns with the fact that PPI is still valid for À outside the [0, 1] interval, and indeed in some situations may achieve its minimal variance one of these values of X².
function class of sigmoidal regressors for post-hoc regres- sion of f, well as an associated PPI estimate:
9 ((((x)) = 1 1 arp(-af((X +3)
(9)
4.2. Variance Reduction through Regularized Regression
Thinking of the X-selection problem simply as univariate regression allows for new intuitions around the challenges of few-label PPI. It is known that ordinary least squares (OLS) is an unbiased estimator, but can suffer from high variance when few examples are available. In regression, a standard way to overcome issues with variance is to use ridge regression. This estimator imposes an L2 penalty in addition to the MSE to reduce the variance of the estimate (at the price of adding small amount of bias). We pro- pose using ridge regression to robustly estimate À in the case where n is small. We specifically propose using the estimator:
1 h(X,) - -(((x)))) Arpl, = N E ((((x)) +
1
(10)
where a, 3 are learned parameters. By using a function class which is better-suited to our problem domain, we hope to find useful transformations which are not possible with a linear regressor and thereby achieve greater variance reduction.
4.4. Discussion Various Other Approaches in Existing Open Source Implementations
Angelopoulos et al. (2023b) demonstrate that when account- ing for the added unlabelled term in Equation 3, the point estimate for the optimal lambda is a scaling of the regression coefficient dependant on the ratio of the size of the labelled and unlabelled datasets. We follow this methodology and scale both the PPI++ estimate AOpt and the Ridge-PPI esti- mate Aa by (1 + )
(Ox(L(X),{(X)) An = [xr]f +
(8)
This value yields the minimum mean squared error for the ridge regression problem with E RR as the ridge coefficient a penalizes regression solutions with large L2-norm, hav- ing the effect of reducing the magnitude of the estimated X. This estimated À is in turn used to calculate pIP
We additionally note that a pre-existing open source im- plementation of the PPI++ algorithm', two additional heuris- tic inductive biases are used when calculating X. First, biased estimate of Coe (M(X)) (x)) is used, where the sam- ple covariance is divided by rather than n 1. We use the unbiased estimate of the covariance in our implementations. Further, the open source implementation clips the estimated À to be within the interval [o, 1]. We do not clip our esti- mates of X, as in principle the optimal regression coefficient may exist outside of that interval. We note that both of these heuristics may be helpful for improving the performance of PPI on some distributions as they both frequently produce smaller estimates of X - this is a similar reason to why we believe Ridge-PPI is helpful, which provides a principled approach to preventing overestimation of A.
4.3. Variance Reduction through Non-linear Regression
Understanding the role as a regression coefficient can also lead us to investigate other hypothesis classes for regres- sion. One may consider an arbitrary transform g of f(X), and it is clear to see that substituting (X)) for Xf in Equation 5 can be done with no difficulties. Similarly, if
one uses the intercept term b- = [[[x(x)] [X)) and substitutes (X)) for Xf(X) in Equation 7, one finds that minimizing the mean squared error for an arbitrary function g(X) is equivalent to minimizing the variance of jrrı-
What might be useful g to consider? In the rate estimation case (i.c. binary h). note that it's somewhat unnatural to perform linear regression of f E [o, 1] onto h E (0, 1). Instead, we propose using the simple but more well-suited
5. Experiments
In this section, we empirically analyze the performance of our proposed approaches in the few-label setting. We provide a description of the data that we analyze, the exact implementation of the methods that we use, and the results of our experiments.
2 Angeloopoulss et al. (2023b) note explicitly that in the mean estimation case (unlike more general cases), À need not he clipped [0, l].
4

---

Regression for the Mean: Auto- Evoluavvlltti with Few Labels through Post-hoc Regression
aphafol
balwes
corois, ncafthcre
Wiae Signals
2.0
0.5
0.
Lxbd
Ginical
duaccll
1.0
Cauaca
1.4
Sprad
2.5
3.4
2.
2.5
23
15
18
17
11
13
19
Figure I: Estimation performance research datasets. Average MAE of each estimate type is normalized by the average MAE of classical estimation at that batch size.
5.1. Datasets
prompts in the dataset, one cannot make conclusions regard- ing a model's sensitivity to any one specific subject matter using the bulk refusal rate, nor is one rate of refusal qualita- tively better than any other. Instead, this dataset intended to provide relevant setting where qualitative annotations are required to evaluate complex model behavior.
5.1.1. RESEARCH DATA
We first apply our method to statistical inference for sev- eral real-world research questions. This suite of datasets, aggregated by Angelopoulos et al. (2023a), presents many instances where pseudolabelled data can be plentiful while gold standard labels are scarce. Some examples include using images of galaxies to estimate the occurrence rate of spiral galaxies and estimating the mean deforestation level in the Amazon over a specified time period. We defer to Angelopoulos et al. (2023a) for a full description of each of the datasets.
Within our previously described framework, examples X; are each tuple containing a prompt and an answer from an LLM. Here, the labelling function h(X) is equal to one when X is an instance of refusal, and zero otherwise.
5.2. Experiments and Estimation Methods
To benchmark the efficacy of different estimation methods for the refusal rate, we conduct an experiment where we randomly sample both D, and Diy from the larger pool of samples. We ablate over several different values of to compare each method's performance with different amounts of labelled data. We keep the amount of unlabelled data fixed at N = 1000. We measure performance of each method as the mean absolute error between the estimate and the true refusal rate over the entire dataset. For each setting of n, we sample Dn, and D 350 times and compute the MAE over all trials.
5.1.2. LLM REFUSAL DATA
For this section, we datatset of prompts and LLM outputs, along with human- made annotations indicating whether the output was an instance of refusal. L.LMs can refuse to answer questions for a variety of reasons, such as when the model is prompted to complete a task outside its capabilities (Xu et al., 2024), asked to provide private information (Liu et al., 2024), or asked an inappropriate or unsafe question (Yuan et al., 2024). The dataset consists of over 50,000 prompt-ansen answer pairs. Prompts consist of wide variety of requests concerning several different topics, and the answers are sourced from multiple different publicly available LLMs. For our purposes, we will only use this data to understand how often a given LLM refuses to answer its prompt, rather than investigating the kinds of prompts the model refuses answer. Given the high diversity of
In our experiments with the LLM Refusal Dataset, we exper- iment with four different methods for estimating the refusal rate:
Classical: Classical estimation using the sample mean from the labelled data D,,-
hhtps//githhhccommmagggpoooopppp
PPI++ Estimating the refusal rate using both D, and
5

---

Regression for the Mean: Aulo-Evaluation with Few Labels through Post-hoc Regression
STD of Rote Estimate
0.20
1.0
0.25
Normalized MSE of Rate Estimate
0.9
0.10
15
17
Labelled Dataset Size
MSE of Refusal Rate Estimate (350 Traks)
MSE Rata Estimato
Syred mile
0.04
my 1w1000
0.03
0.02
0.01
0.5
7
13
15
17
19
20
21
11
13
1.5
17
19
20
21
Labelled Dataset Size
Labelled Dataset Size
(a) Standard deviation of estimates (top) and average MAE of esti-(b) Average MAE of different estimates normalized by the average mates (bottom) different estimation methods.
MAE of classical estimation at that batch size.
Figure 2: Estimation performance on the LLM Refusal Dataset.
Dy through pPppy. Here, the sample estimate of AOpt based off Equation 6.
5.3.2. LLM REFUSAL DATA
While each of the PPI based methods able achieve superior MAE over classical estimation, we find that both Ridge-PPI and Sigmoid-PPI are able to improve upon the performance of PPI++ for all small values of n (Figure 2, left). We also observe that the standard deviation of these estimates smaller, indicating that these techniques have indeed reduced the variance. This effect can most prominently be seen for smaller labelled sets (Figure 2, right), where we see that our improved PPI methods can reduce the MAE of classical estimation by over quarter.
Ridge-PPI (Ours): Similar PPI++, but we instead in Equation 8. Here,
fit À using ridge regression perform cross validation over the labelled set to determine the ridge parameter a.
Sigmoid-PPI (Ours): We follow the procedure de- scribed in Section 4.3 to fit sigmoidal function on our labelled data from ((X h(X) and then use that make PPI estimate. Here we also use L2 regulariza- tion and choose the regularization parameter based cross validation, similar to Ridge-PPI.
5.4. Distributional Influence on PPI's Efficacy
The underlying data distribution has the potential to effect the efficacy of each estimation method. To study how the variance of the target variable /(X),, the predictions f(XX), and the covariance between these variables effect perfor- mance, we perform the experiments described in Section 5.2 for each individual LLM LLM represented in the LLM Refusal Dataset.
5.3. Results
5.3.1. RESEARCH DATA
We find that both Ridge-PPI and Sigmoid-PPI are able to either match or exceed the performance of both classical estimation and PPI++ across each of the settings (Fig. 1). This includes the plankton, alphafold, and forest datasets, where traditional PPI++ fails to outperform classical for several settings of while our methods consistently match or exceed this performance. While across several of the datasets Ridge-PPI and Sigmoid-PPI are able to cut the enror of the leading baseline by significant fraction, other datasets such as ballots, forest, and census healthcare show a more modest improvement. In sections 5.4 and 6. 1, we further investigate this phenomenon and present hypotheses on the conditions that lead it.
We find that the efficacy of PPI++ in comparison to classi- cal estimation can vary highly across different distributions of samples. In some distributions, PPI++ cuts the error of classical estimation by roughly 30%, with our methods producing similar gains (Figure 3, right). Under other dis- tributions, PPI++ performs worse than elassical estimation; PPI++ can accrue 20% more than classical estimation when applied to certain distributions (Figure 3, left). How- ever, in these circumstances. we find that both Ridge-PPi and Sigmoid-PPI perform 10% better than classical estima- tion, demonstrating a circumstance in which our methods
6

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
1.8
1.0
1.6
0.9
Normalized MSE of Rate Estimate
Normalized MSE of Rate Estimate
1.4
0.8
2.2
0.7
0.6
1.0
0.8
5
11
13
15
17
19
20
21
11
13
1.5
17
19
20
21
Labelled Dataset Size
Labelled Dataset Size
Figure 3: Estimation performance on two different subsets of the LLM refusal dataset normalized by the average MAE of classical estimation at that batch size. We note that Var[f(X)) is twice as large on the right (where PPI++ succeeds) as on the left (where it fails).
are able overcome the shortcomings of PPI++.
not concern in the low-label regime, further investigation necessary to determine the theoretical and empirical efficacy of Sigmoid-PPI when is large (Appendix A). An inter- esting path for further work would be determining which statistics of the distribution determine the efficacy of each of these PPI-based approaches.
We suspect that the poor performance of PPI++ distribu- tions like this one may be related to low variance of f(X). Since Var[J(X) appears in the denominator of Equation 6, potential errors from overestimating (o) are amplified when [fr(f(X) is small. This is evidenced by the fact that despite the fact that the two distributions depicted in Figure 3 have similar optimal regression coef- ficients, [f [f()) is times larger for the distribution depicted the right. This would help explain the improved performance of ridge-regression, which biases the estimate towards smaller A. We explore this hypothesis analytically in Section 6.1.
6. Discussion
We now our attention additional analyses help interpret results. In particular, analyze the attributes of the distributions which PPI++ succeeds, and how our estimation methods may help alleviate problems in scenar- ios where PPI++ performs poorly. To carry this out, we extend our previous analysis to consider the fact that the parameter is a random variable; we had previously taken it be constant. Our new analysis highlights the role of V Tfrlf(X] as important, but previously unconsidered, factor in the success of PPI++
We next investigate whether these trends are persistent in the research datasets by calculating the ratio of the MSE for PPI++ the MSE at for Ridge-PPI for each dataset and taking its correlation with V [x)] for that dataset. We find that there is strong anticorrelation between these values (Pearson = -0.69), further supporting our hypothesis that small [ar[f [X) hinders the efficacy of PPI++. While we do not have enough datapoints to make this correlation statistically significant, it serves motivation for the more thorough analysis we present in Section 6.1.
6.1. Stochastic À
In this section, propose expression for the variance of PPI++ using the first and second moments of stochastic X parameter. To simplify these expressions, we assume that X is fit to an independent but identically distributed pool of data. Techniques like cross-fitting have been proposed as a data efficient way to conduct this procedure (Zmic Candès, 2024b).
We note that asymptotically, PPI++ is guaranteed to perform at least as well classical estimation (Angelopoulos et al., 2023b). The failure of PPI++ shown on the left of Fig. 3 can be explained since the sample size is clearly not large enough to reach this asymptotic regime, and so PPI++ is not guaranteed to improve performance. We additionally note that the proof of the PPI++ estimator being asymptotically unbiased presented by Angelopoulos et al. (2023b) does not directly map on our Sigmoid-PPI approach. While this is
We arrive at the following expression for the excess variance

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
of PPI, with full derivation supplied in Appendix C:
Normaltnd Erco of 1
0 can
<0 -12
250
Dhor
Var[äppr] [ar(ān]
= =E[A]?( 1 + 1 (a)r((((X))
Sample Sizen
150
72
+ +Var[[(2E1((X)]) + + + (ar(((()))
2E[M] Cor(h((X), n
50
(11)
9.5
2.15
Var[(0)
This is similar to the expression for the variance of PPI pre- sented in Equation 5 (which assumed a constant A), with the notable exception that V "rr] (>f(X) has been decomposed into two terms depending on the expectation and variance of (stochastic) X, respectively. This reveals an important property of PPI++: using higher variance estimator for À can reduce the efficacy of PPI.
Figure 4:
The normalized variance of PPI++, Ver[Apez] for several values of Var[](X)] and n. Varlayl
V (X)))) h(X)] is set to a value of 0.3 with an additional 1 nz decay multiplier.
While it is not surprising that the error of a mean estimate will depend on the variance of a variable being used to construct the estimate, this formalism remains important as previous expressions for the variance of PPI neglected X-estimation variance as source of error. This also theo- retically justifies the benefits we from using Ridge-PPI: ridge regression has lower variance than OLS regression, and this expression demonstrates how that may lead to a decrease in estimation error.
reveals the importance of understanding À as stochastic quantity.
To end this discussion, we evaluate number of possible variance values for pipp,, normalized by Var[Ax], for a distribution from one of our previous experiments. Taking the values Corr(h( (h( J( (X)), Var(h( (r((())) (X) and (X)( from the Ballots dataset, we plot the variance reduction of PPI++ for various settings of V Varlf(X) and n. We take static value for Var(Cou[J[X]) (X),,()),, but we note that the exact relationship between this quantity and Varlf will likely depend on the data generative procedure which is opaque in most circumstances. Figure demonstrates how a larger Var[s(X)] leads to variance reduction with most smaller values of n. Furthermore, demonstrates that depending on the distributions of (XX and h I(X), specif- ically if V Vfrlf(X) is too small, PPI++ will not reduce error in expectation if the labelled sample size is too scarce. This underpins the importance of techniques that reduce the variance of the estimate, such as Ridge-PPI.
6.2. Efficacy of PPI++
Further insight into PPI++ specifically can be made if we consider the estimator sopa, where
Coo(f(X), ((X)) 14 + (1+ 3) Var[[(X]]
X
B[Aom
(12)
Furthermore, under certain convergence conditions (see Ap- pendices C and D), Varidopll Var(Cou[f(())M)) ulti- mately giving us:
Var[f(X]]=
6.3. Risk Reduction from Ridge-PPI
To theoretically justify the improvement of Ridge-PPI over PPI++ we can create a similar expression for excess risk as the one presented in Section 6.2. In this section, we refer to the estimator using Xa (Equation 8) to caleu- late åupp as ÅAPPI. Specifically, we will reason about when Ridge-PPI can improve upon PPI++. Using the same assumptions as Equation 13, well as the short- hand V : V Var (xr(Coe ((X)]((E(((( + (1 + ()r(((()))) we show:
Var[srrı] Var [An]
Vor|Cou]((X), 2
( ))
[X]]
Var[f(X)]
(orl((x))Corr(h(x)) f(x))² (1 22N2) m
(13)
Counterintuitively, this demonstrates the variance of PPI++ actually scales inversely to Var (X)). This is supported by our empirical findings in Section 5.4. This runs con- trary to what one would expect from Equation and further
Varlärrı,] - Varlárrı]
(14)
8

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
7. Conclusion and Future Work
1 (Varlf(X)) + a)² Var[f(X)]3
(15)
In this work, we expanded on the PPI framework to perform mean estimation when very few labelled examples are avail- able. By relating the optimal À setting for PPI++ in mean estimation to the regression coefficient, retical motivation for two regression-inspired approaches to mean estimation. Both of these approaches use insights from univariate regression to reduce the variance of the estimate when few labels are available. Through exper- iments estimating feature generation rates in the context of data analysis and LLM evaluation, we demonstrate that our approaches produce lower variance estimates than both classical estimation well PPI in the low-label regime. Through additional analyses and experiments, we further elaborate on when our methods can have the most impact.
provide theo-
(orr( n(1+ [VVaa( +a]²
(16)
A full proof is presented in Appendix E. This expression demonstrates how a small [ar(f(X)) contributes to Ridge- PPI improving upon PPI. The first term in Expression 15 presents a variance reduction term, as V is non-negative and the difference is non-positive. When V [X)] is particularly small, ([arlf(())++a negatively dominated by the latter term in the difference. Importantly, the larger the ridge coefficient a, the more negative this difference is, indicating larger reduction in variance.
[xrlf(x]aa will be
The use of predictive models to aid statistical inference is an exciting research frontier. Future work should investigate efficient methods for performing PPI when the labelled and unlabelled samples come from different distributions. Furthermore, the potential impact of the predictive model having varying levels of performance on different subgroups of the distribution and the potential fairesss related concerns that come with it is an important future research direction.
Mcanwhile, the second term, Expression 16, represents increase in loss attributable to the fact that Ridge does not yield the optimal A. This non non-negative term grows as the correlation between ( and (xX) increases. This makes intuitive sense, as stronger correlation between f(X) and f(X) leads to lower variance estimator when using PPI++ Both these terms in tandem demonstrate how Ridge-PPI can be deployed in circumstances where PPI++ struggles in order to improve performance, namely the circumstance where V of [X)] is small.
Impact Statement
Our work investigates the shortcomings of the Prediction Powered Inference framework and proposes new methods to help overcome these problems. PPI has been applied to several fields of great impact to society, such as ecological, social science, and biological research, as well as the evalu- ation of widely used machine learning systems like LLMs. These improvements to PPI have the potential to lead to expedited progress in these application areas by reducing errors in statistical inference. However, while we may have uncovered some cases where PPI does not perform as ex- pected, this is certainly not to be exhaustive. Future work should investigate additional shortcomings of PPI, such as whether bias the labelled or unlabelled data can lead to faimess concerns.
6.4. Optimal Ridge Coefficient
We next look at the optimal selection for the parameter in Ridge-PPI and use it to further investigate how the statistics of the data relate to PPI's success. To do this, we will apply simple first-order optimization techniques to Expression 14. Note that since this expression is not convex a, it is not guaranteed that this critical point in a minima. Furthermore, depending the statisties of the data Ridge may not im- prove upon standard PPI++. However, when the minima is not given by zero or infinity, it will be given by:
Acknowledgements
n1 + V a' Cool (())
(17)
BE is supported by the funds provided by the National Science Foundation and by DoD OUSD (R&E) under Co- operative Agreement PHY-2229929 (The NSF AI Institute for Artificial and Natural Intelligence).
This quantity is difficult to estimate in practice, but supplies more insight into the dynamics of PPI. Notably, this opti- mum is smaller in the case of greater covariance, and greater in the case of large V. Sceing as V is large positive term in the risk, this expression balances between the potential variance and the potential variance reduction permitted by (oo(((X), {X))).
9

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
References
robotics and aulomation (ICRA), pp. 6411-6418. IEEE, 2020.
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zmic, T. Prediction Prediction-powered inference. Science, 382 (6671):669-6744 2023a.
Pitis, S., Creager, E., Manddekar, A.. and Garg. A. Mocoda: Model-based counterfactual data augmentation. Advances in Neural Information Processing Systems, 35:18143- 18156, 2022.
Angelopoulos, A. N., Duchi, J. C., and Zrnic, T. Ppi++: Efficient prediction- inference. arXiv preprint arXiv:2311.0145,, 2023b.
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al. Chexnet: Radiologist-evel pneumonia detec- tion chest x-rays with deep learning. arXivv preprint arXiv:1711.05255, 2017.
Blanchet, J., Chen, H., Lu, Y., and Ying, L. When can regression-adjusted control variate help? events, sobolev embedding and minimax optimality. Advances in Neural Information Processing Systems, 36, 2024.
South, L. F., Oates, C. J., Mira, A., and Drovandi, C. Regu- larized variance control variates. Bayesian Analysis, 18(3):865-888, 2023.
Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., and
Jordan, M. I. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint 3xXx:22403.0708 2024.
Tu, S., Li, C., Yu, J., Wang, X., Hou, L.,, and Li, J. Chat log: Recording and analyzing chatgpt across time, arXiv preprint arXiv:2304. 14106, 2023.
DeGrave, A. J., Janizek, J. D., and Lee, S.-I. Ai for radio- graphic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-619) 2021.
Xu, H., Zhu, Z., Ma, D., Zhang, S., Fan, S.. Chen, L., and Yu, K. Rejection improves reliability: Training llms to refuse unknown questions using from knowledge feedback. arXiv preprint arXiv:2403. 2024.
Dehaerne, E., Dey, B., Halder, S., De Gendt, S., and Meert, W. Code generation using machine learning: A systematic review. leee Access, 10:82434 82455, 2022.
Yuan, Y., Jiao, W. Wang, W., Huang, J.-t., Xu, J., Liang, T., He, P., and Tu, Z. Refuse whenever you feel unsafe: Improving safety in 1lms via decoupled refusal training. arXiv preprint arXiv:2407.0m 09121,, 2024.
Fisch, A., Maynez, J., Hofer, R. A., Dhingra, B., Glober- son, A., and Cohen, W. W. Stratified reedcction-ooeetty inference for hybrid language model evaluation. arXiv preprint ar Xr0: 042911, 2024.
Zhang, A., Brown, L. D., and Cai, T. T. Semi-supervised inference: General theory and estimation of means. 2019.
Jaiswal, A. and Arun, C. J. Potential of artificial intelli- gence for transformation of the education system in india. International Journal Education and Development us- ing Information and Communication Technology, 17(1): 142-158, 2021.
Zhang, H. mixup: Beyond empirical risk minimization.
arXiv preprint
2017.
Zhang, Y. and Bradic, J. High-dimensional semi-supervised learning: in search of optimal inference of the mean. Biometrika, 109(2):387-44)) 2022.
Kenney, and Keeping, E. Linear regression and correla- tion. Mathematics of statistics, 1:252-285, 1962.
Lee, H. The rise of chatgpt: Exploring its potential in medical education. Anatomical sciences education, (5):926-931, 2024.
Zrnic, T. and Candès, J. Active statistical inference. arXiv preprint arXiv:2403.03208, 2024a.
Zmic, T. and Candès, E. J. Cross-prediction-poweeee infer- ence. Proceedings of the National Academy of Sciences, 121(155)e2322083111, 2024b.
Liu, 7., Zhu, T., Tan, C., and Chen, W. Learning to refuse: Towards mitigating privacy risks in lms.. arXiv preprint arXiv:2407. 10058, 2024.
Neuhausen, M., Herbers, P., and König, M. Using synthetic data to improve and evaluate the tracking performance of construction workers on site. Applied Sciences, 10(14): 4948, 2020.
Osiński, B., Jakubowski, A., Zięcina, P., Mitoš, P., Galias, C., Homoceanu, S., and Michalewski, H. Simulation- based reinforcement learning for world autonomous driving. In 2020 IEEE international conference on
10

---

Regression for the Mean: Aulo-Evaluation with Few Labels through Post-hoc Regression
Normalized MSE of Refusal Rate Estimate (350 Trials)
1.0
Normalized MSE of Rate Estimate
0.9
0.8
0.7
0.6
Siamold-PPI N-1000
Ridge PPI N=1000 N000000
Ph++ N-1000
0.5
5
10
20
50
75
100
200
500
1000
Labelled Dataset Size
Figure 5: Estimation performance the LLM refusal dataset with larger labelled batch sizes normalized by the average MAE of classical estimation at that batch size.
A. Larger Labelled Batch Size Experiments
Though our methods were intended to be used for smaller we also experiment with the case where both the labelled and unlabelled datasets are large (Figure 5). We find that our methods are best suited for smaller batch sizes, with the greatest improvements over classical estimation (in relative terms) occurring around labelled batch size this point, we find that the performance Ridge-PPI converges the performance of PPI++ This could potentially be explained by the larger pools of labelled data resulting in smaller being selected in cross- vross-validation, simply because the (1 + N term in the denominator of our PPI++ estimate AOpt and the Ridge-PPI PPll estimate ÅS (Section 4.4) increases with causing the estimated X to be closer to 0 and the methods to behave more similarly to classical estimation.
20. Beyond
We observe that Sigmoid-PPI's performance decays as the size of the labelled dataset approaches the size of the unlabelled dataset. This could be caused by the lack of a scaling mechanism, such as those described in Section 4.4, that allow the method rely on classical estimation as n approaches or exceeds the magnitude of or the presence of potential asymptotic bias. To counteract this, we propose an adjusted variant of Sigmoid-PPI that also uses scaling mechanism. Specifically, this estimator has the same form as the presented in Equation 10, except it uses the following function:
1 o(f(x)) = 1 n 1 exp(- af exp(-af(X)
(18)
Results for this new estimator are presented in Figure 6. We observe that the estimator significantly benefits from this adjustment factor at large labelled dataset sizes: the adjusted estimator continues to make gains over the classical estimate, while the unadjusted estimator may diverge towards excess error.
We recommend using Sigmoid-PPi PPI in cases where the labelled dataset is small, while Ridge-PPI seems to be flexible to different labelled dataset sizes.

---

Regression for the Mean: Anto-Evaluation with Few Labels through Post-hoc Regression
plankton
ballots
1.2
Normalizes MSE Rate Estimate
Normalized MSE of Rate Estimate
1.0
0.8
Classical
Clessical
Sigmoii-PPI
0.6
Sigmoiid-PI
Sigmoid-PPi (adjusted)
Siginoid-P (adjusted)
0.6
0.4
50
75
100
200
500
75
100
200
500
Dataset
Dataset Size
Figure 6: Comparison in performance on the Plankton and Ballots datasets for Sigmoid-PPi PPI and an adjusted Sigmoid-PPI that is scaled by I+ We see that in some circumstances, the Sigmoid-PP''s normalized emor diverges. Meanwhile, the adjustment leads to steady performance gains over the classical baseline.
B. Proof of Equation? 7
In this section, we prove the following expression:
Σ (1(,,) ((()) b)²
-
(19)
where b An Px^ is the optimal ordinary least squares intercept coefficient, which is the difference between the sample means of the target h(X) and transformed input AX(X). We start by expanding the squared loss within the sum.
n n Σ -(xx))A(CK) - iun + Pxx))
-((x) 6)2
(20)
17
((((X)) -m)- - -n)) (X((x,) - Axx)))
(21)
n
((((.)) (AF(X)) - Pxx)² - 2(Af(X.) (xx)) - (.))).
(22)
Distributing the sum and division by across each of the terms, Var[h(X)], Var[> and Cou(h(X), Af()) respectively. Therefore, it's clear that the left hand side of Equation 7 is proportional to the right hand side, and that this expression admits the same minimizing À as the variance depicted in Equation 5.
that each of these terms biased estimates of
C. Stochastic A
In this section, work through the complete decomposition of the variance of PPI assuming stochastic but independent estimator for X.
We analyze the expected mean squared enror of PPI++
N 1 N Σ Xf(X)) +
B(
Σ h(X,) - (f(x)))) E[(mh, - (a + Ab Ac))
(23)
12

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
To simplify expressions, take the following shorthands:
a -
b(X), b
Σ f(X)).
Σ f(X:) n
(24)
7
We know that the risk of an MSE loss can be written as a bias-variance decomposition. Since we assume À is independent, this bias is zero. Of course a dependent X, which is often the case in practice, would introduce bias.
The variance has the following form:
Varla Xb Xc]
(25)
= Var[a] + Var[Ab] + Var[-Ac] + 2Cov(a, Ab) + 2Coo(a, -Ac) + 2Cov(Ab,
(26)
Since we assume is independent of our data, we are left with:
Varla Ab Ac] Var[a] + Var[Ab] + Varl- Ac] 2Cou(a, -Ac)
(27)
We will further decompose this expression by analyzing each of its summands:
Var(a]: This is already fully simplified, and is the variance the classical estimate.
Var [Ab]: Var[Ab] (Vrr + Var(b)E[A] Vor(())
Var[-Ac]: Var[-Ac] = - Var(-c)B(A
2C, -Ac): Covla, -c) = [[- - -a - = E[]]Cou(a, -c)
Ultimately this gives us:
Varla Ab xc| Var[a]
(28)
+V Var(c)
(29)
+Var(A)(B()})++ + E + Var(b) + Var(c))
(30)
2oolaa)
(31)
We can therefore understand the error incurred by stochastic À as being determined by its variance and expectation. More importantly, this gives us an expression for the excess risk of a stochastic À vs. classical inference:
Varla + A6 Ac] Var[a]
(32)
(Var)() Var(c)) (Er()) E[d] Var(b) + Var(c)) 2E(A]Cov(a,c)
(33)
= =E[N N + 1 [)var(f)+
+ + + 1 -=)var(f)) N n
2E Cov(h,
(34)
13

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
Furthermore, if we assume that are using PPI++ and our À estimator is OLS, we can the properties that Aopt where
A
(ov(I(X), (1+ [arl(f(()
B[AOpt
(35)
1 N
Furthermore, under certain convergence conditions (see Appendix D), Var far[)opt]
Var (X
ultimately giving
[[x]
us:
Varla + 2b Ac| Varla]
(36)
(1+ +1+ ())))(((X) (( (o)(I(X),MM))) (1+ Varl(((X N + 1 Var(() Cou(h, (]) Var()) + +( + = V)r(()))
(37)
We will focus the second factor in the first product:
1 (1+ N V [f()]
Co)(f(X),N() ((X))
+ 1 W - 2 Cou(h,
(38)
N n
+ 1) (1 N
2 Coo([(X)),
+ +1) - xt)
= Cou([(X),M(X)
(39)
(1 N )n
(1 N 2(1 + )
M(X)) 1-2 (oo(((X),
-Co(f((X),(x))
(40)
(1 + 2 n
11
Substituting that back into our original expression, we have:
Varla + Ac] Var[a]
(41)
()) (1+ V)rrr(f (x)) )( (oo(F(X), (ar()) + +1 + v
(
(42)
(x)()()),((x))2
1 ))r((()) n
n(1 [)rl((((X)) (2 + (
(43)
(2E + N + )))((())) (or)(())))) (1 ))
(44)
Where in the final line we have used the definition of covariance as Cou(f(X), (XXX arrive at the expression from Section 6.2:
=
Finally, if we substitute in Var[Ropl = Var|Cou( Var(f(X))2
we
((X(Coo(((X))M( (2E[S())] + ( + var(s(())) Cerl [x)]
(45)
(1 + 2 ))
14

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
D. Convergence of OLS Estimate
In Section 6.2, we propose that under certain conditions, we can take Var[Aopt] VarlCool (( In this section, [[r]](X]]]
we set forth the conditions under which that may be the case. Throughout this section we assume we are using the estimator Cov(((X),A(X)) where Var[f(X)] and h(xx)) are the sample estimates for the variance of f(X) (1+ [Var((())]
and covariance of f fX) and h(X), respectively.
In section 6,1, we assumed that the estimator for À used finite sample separate from the data used for creating the PPI estimate. We will make the additional assumption that we have two pools of data for fitting A: one set of n examples for fitiig Co)(f(X), I())) and one pool of N examples for fitting Var (X)), where n << N. This has been a standard assumption throughout this work, as we assume that we have access to a very large number pseudolabels f(X). With that in mind, we rewrite our estimates for these two quantities as Cov,, (f (X). h(X)) and [f to denote the random variables being used to generate them, and rewrite our estimator for À Cov,, (f(X),
(I- Varxx[((X)]
We will now use a set of standard convergence in probability results for the sample variance. Namely, as N grows, we have the following:
n [arx(f(X)] >pp Var[s(X)), (1+ N Varl(s(X)) -tp Var[s(X)]-
(46)
Since this variable converges in probability in a constant, we can use Slutsky's theorem for the following convergence in distribution of our estimator for
(o(,( (1 + ((Fx
M()) Var[f(X)
(47)
Therefore, in this asymptotic regime, we have:
Var[Cov,[f((X) (f(X),
Varl Var[f(X)
(48)
Vxr
Similarly for Ridge-PPI, where An := Corn (XX)_ asymptotically, we have:
(1+ (V+)) we can use an identical line of reasoning and know that,
Var[d.] =Varl (fX),,((
Var[Cou,[f(XX..r + +a) (Var [[ar]](X)] + +0²
(49)
E. Proof of Excess Risk of Ridge
Here we prove our expression for the difference of risks between Ridge-PPI and the classical estimate:
VarlūppIL] Var[An]
(50)
Using the decomposition given by Equation 36 in Section C, as well as the fact that E[Åa] further decompose this expression as:
Coel (XXX))()))))))))))))))))())))))) [1+ [I+ (V ar [(x))aa}
we can
1 (xo(s(X),M(x)) ){ (1 (VarV + a) N ar (0[Varf((X)) +
Roe(s(X),A(X))
+ 1 var(()
- 2 -2cou(h, f))
(51)
(1+
1 ( +
War(()))
(2 + N n
(52)
15

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
We'll start by decomposing the first term in the sum. We'll focus on the second factor in the product.
Coulf (1+ [Var + a) (N + n
2 n Var(f)
Coo(h, f))
(53)
-Co(f((X) ( + 1)Var [[() 2(1 + (varr( + +a), Co)(F(X),N(() Varl (ar[f(()) 20
n11 2 x) + (x)
Var [[r[[(()] +
(54)
Coe(f(X),,(())) (- var(s(X)+ (X)++c mar(s())+a'
a
(55)
n
Substituting this back into the original product, we have:
Co)(f(X), 1 (1+ ()(Varlf()) + (1+ [Var +a)" ( + n W - Cou(h, f))
(ov(f(X),n(x))
(56)
'N
(1+ (V + ) (x)({(X),)(X)) (C)x(((X),M(X)), (Var[f(X) + a
CoulS(X),
(57)
n
Cou(f(X), N(X))
(C))
(58)
n(1 + ()(Var n(1 [Xxx
(59)
m(1+ [V
We now again use the definition of Covariance as Cou (,0 well the fact that (V (Var[f(X)) + a²² Var[s(X)] + a² +
2V 2Var[f
(60)
n2 +1 (Var((() + a
We'll now add and subtract (ork) from the numerator:
(61)
+ aa²
(x) n(1+ 2 (Var(f()))) + ax)²
(orr(/(X),,(x)))VVrrll)
(62)
n + /
We now substitute this expression back into the expression for the difference of risks:
VarlūppI.] Var(ān]
(63)
) + +( - Var(())) (64)
= m(1+ (V or f(X)] +a)
n21 +
16

---

Regression for the Mean: Auto-Evaluation with Few Labels through Post-hoc Regression
We simplify this expression if make the same simplifying convergence assumptions described in Appendices C and D, well the shorthand V = (CaCCC/alf(N)),AXX]]EEEEEEEE})1 ( Varr(((X)))
V (Varlf(X) a)2
(65)
n11 )V ar [(X)] + +a)²
nn1 %)
Finally, we can also calculate the risk reduction from using Ridge-PPI vs PPI++ by subtracting Equation 13 from Equation
65:
VarläppI.] Varlūn] (Varláppi) Varlān))) = Varläppr.] Varlūpprl
(66)
V
(orr(((X),N(x)))1 [x2)
(67)
(Varls(X)) + aa)² Varls(X)J=
n(1+ (V +a)²
-V (Var (Var[f(X))
Varls(X)p
(68)
n11 & (V [f xa)²
F. Optimal Ridge Coefficient
The expression for the risk reduction from using Ridge-PPI vs PPI++ given in Equation 68 is not convex in a, and depending on the statistics of the data Ridge may not improve upon standard PPI++. However, if there does exist optimal setting for a, we select it using first order optimization. If we take the derivative of Equation 68 with respect to we have:
d - V or oorrrjn [úppr.] Varlăppr]
n11 + 3)V
(69)
de
n(1 N [VVvr[[(X)]
Setting this to zero we have:
n21 + V)
VarlūppI. VarluppI] = = a
(70)
da
(ox(f(X),h(X))
17


Reviews:
[{"review_id": "6IkctF3gja", "reviewer": "Reviewer_V1XE", "summary": "The paper provides a method to improve over standard prediction-powered inference by weighting the correction factor in the same way that is done in control variates. The methods are simple and improve over existing baselines, and the authors provide an illuminating discussion.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First of all, we thank you for your constructive feedback and positive reception of the work. We look forward to improving the manuscript based on your review. We will be sure to improve readability of the figures for the camera ready draft. Below, we respond to individual points you raised.\n\n**Optimal Lambda Risk Expression**\n\nWhile Expression 5 in the paper does describe the risk of PPI for a static lambda, we substitute the optimal lambda presented in Expression 6 below.\n\n\\begin{equation}\n    =  \\frac{1}{n}(Var[ h(X) ] + \\frac{1}{Var[f(X)]} -  2 Var[h(X)]Corr[h(X), f(X)]^2).\n\\end{equation}\n\n**Error Bars on MSE Plots**\n\nThank you for your excellent point about the inclusion of error bars in our plots. For the camera ready version of the paper, we will re-run the experiments and include appropriate error bars. Here is a small set of 95\\% confidence intervals for the normalized MSE (see figure 1) of different methods across 350 trials. Here the confidence is represented with a tuple (lower bound, upper bound):\n\n|Dataset | PPI (n=5) | Ridge-PPI (n=5)| Sigmoid-PPI (n=5)|\n|--------|------------------|------------------------|--------|\n|Alphafold | (1.12, 1.65) | (0.49, 0.72) | (0.55, 0.82)|\n|Ballots | (0.27, 0.47) | (0.24, 0.44) | (0.34, 0.6)|\n|Census Healthcare | (0.71, 0.95) | (0.68, 0.92) | (0.58, 0.8)|\n|Forest | (1.92, 3.36) | (0.71, 1.22) | (0.82, 1.09)|\n|Galaxies | (0.84, 1.14) | (0.61, 0.9) | (0.71, 0.97)|\n|Plankton | (1.23, 1.89) | (0.56, 0.76) | (0.54, 0.68)|", "claims_and_evidence": "Claims were clear and method is clear.\n\nMy only concern is that of the variance of the estimated lambda that occurs due to estimation on data. For example, it would have been very useful to see the derivation with the variance with the optimal lambda_hat plugged in. Something like this was attempted in 6.2, but more the estimated covariance term still exists in there, which makes the math less illuminating than seems possible.", "methods_and_evaluation_criteria": "Methods are clear, evaluation criteria are sound.\n\nThe main idea for the method is to see that lambda can be seen as a scalar \"corrector\" for f(X) to approximate h(X), whose mean one wants to estimate. This whole idea can more generally be thought of as a special case of variance reduction by averaging a conditional mean, and the conditional mean is estimated by regressing h(X) on f(X) over a. simple function class that can be fit on few samples. The papers uses linear and logistic regression.\n\nThe evaluation of the method is done on estimating means from few samples in problems ranging from protein function to segmentation. \nThe evaluation showed that on all the small sample sizes considered, the proposed modifications to PPI improves over vanilla mean estimation, and also other modifications in the literature.\n\nI like the refusal rate example.\n\nAlso, the authors should provide confidence intervals for the experiments.", "theoretical_claims": "Theory is correct.", "experimental_designs_or_analyses": "Experiments are valid and support the new methods clearly.", "relation_to_broader_scientific_literature": "The method, as it stands, improves on the PPI methodology and the related work discusses relavant work about PPI modifications.", "essential_references_not_discussed": "None here.", "other_strengths_and_weaknesses": "Simple method, clear new insight about how to interpret PPI, and it works.", "other_comments_or_suggestions": "Please improve the quality of the figures, with larger font sizes and HD figures.", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JdMhknrHTp", "reviewer": "Reviewer_NpF2", "summary": "The paper focuses on the specific scenario where PPI is used for mean estimation. The authors note an equivalence between the tuning parameter estimation for PPI++ and linear regression, and examine several modifications that can be derived. Results are validated through empirical evidence on several benchmark data sets.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our appreciation for your thoughtful review and constructive critique. We look forward to making the writing more clear following your points. Specifically, we will amend the abstract and make the description of the LLM refusal dataset more clear. We respond to individual points of yours below.\n\n**Appendix A**\n\nSigmoid PPI is intended for use with small sets of labelled examples. Within this regime, we see Sigmoid PPI regularly match the original PPI++ or make significant improvements. \n\nHowever, one potential fix for the results seen in Appendix A would be using the following alternative formula:\n\n\\begin{equation}\n    g'(f(X)) := \\frac{1}{1 + \\frac{n}{N}}\\frac{1}{1 + exp(-\\alpha f(X) + \\beta)},\n\\end{equation}\n\nThis would have the estimator behave more similarly to the classical estimator as n -> N and the classical estimator becomes stronger. We are currently experimenting with this modification and hope to include them in the camera ready version of the paper. Below we provide promising results of average MSE at the largest possible $n$ for the research datasets, demonstrating how this method (Sigmoid-Div) improves asymptotic performance:\n\n|Dataset | PPI | Sigmoid-PPI | Sigmoid-Div|\n|--------|------------------|------------------------|--------|\n|Alphafold | 8.94e-05 | 0.0001 | 7.03e-05|\n|Ballots | 1.6e-05 | 0.0002 | 1.53e-05|\n|Forest | 0.00013 | 0.00018 | 0.00015 |\n|Galaxies | 0.00024 | 0.00023 | 0.0002|\n|Plankton | 0.00014 | 0.00013 | 0.00012|\n\n\n**Expanding Section 6, Further Theoretical Justification for Ridge-PPI**\n\nThis was a shared concern for reviewer aoGS as well. Due to the lack of a general rebuttal as well as space restrictions on individual rebuttals, we have included the response to this in the rebuttal to reviewer aoGS. Please read it to see our additional analysis for this issue.\n\n**Bias of PPI++**\n\nFootnote 1 in the paper indicates that for small $n$, PPI++ will be a biased estimator. \nPrior work only considered the asymptotic unbiasedness of $\\lambda$ and there is no existing expression for this bias with small samples.\nHere, we produce such an expression and are happy to include it in the paper if the reviewer finds  it helpful.\nThe expression for this bias is the following (revealed by taking the expectation of PPI assuming $\\lambda$ is not independent from the training data):\n\n\\begin{equation}\n    E[\\hat{\\mu}_{PPI}] - \\mu^* = -Cov[\\hat{\\lambda}, \\hat{\\mu}_f]\n\\end{equation}\n\nWe note that this expression can be applied to any estimator of $\\lambda$, such as Ridge-PPI. Using the definition of covariance can make this expression more interpretable:\n\n\\begin{equation}\n    E[\\hat{\\mu}_{PPI}] - \\mu^* = \\sqrt{Var[\\hat{\\lambda}]Var[\\hat{\\mu}_f]}Corr[\\hat{\\lambda}, \\hat{\\mu}_f]\n\\end{equation}\n\nWe note that we generally do not include this bias term as it can be ignored in cases where we assume that the $\\lambda$ parameter can be fit independently, such as section 6.1. Furthermore, since our methods focus on variance reduction (in line with prior literature) and were effective at reducing overall MSE/MAE, we consider a complete treatment of this bias to be out of scope for this work.\n\n**Other M-Estimators**\n\nWe chose to focus our work on specific insights into the dynamics of PPI when used with mean estimation. \nWhile other M-estimators are interesting as well, we find them out of scope for our work, which is specifically about the small sample behavior of PPI for mean estimation. We believe the small sample focus of our work validates the contribution as novel and useful, despite not considering a wide range of M-estimators.\n\n**Creating Confidence Intervals**\n\nWe note that throughout this work we did not consider the efficacy of constructing confidence intervals for any regime, since the central limit theorem (which past methods rely on) tends not to hold in small data regimes. Since improving the efficacy of PPI in the small data regime was our primary concern, those intervals would not be valid for any of the methods that we compared with.\n\nHowever, the basic machinery from Angelopoulos et al. [2023] will work to create confidence intervals for Ridge-PPI, as constructing the interval is not impacted by how $\\lambda$ is selected.\nAdditionally, for Sigmoid-PPI, assuming that the parameters of the transformation are fit independently as in section 6.1, the confidence interval can be created using a very similar formula to Angelopoulos et al. [2023], with the exception that the transformed predictions are used in place of the original predictions:\n\n\\begin{equation}\n    \\hat{\\mu}_{PPI_g} \\pm 1.96*\\sqrt{\\frac{\\hat{Var}[g(f(X_i)) - h(X_i)]}{n} + \\frac{\\hat{Var}[g(f(X_i^u))]}{N}}\n\\end{equation}", "claims_and_evidence": "The paper claims to provide a theoretical analysis of the link between PPI++ and linear regression, as well as building extentions for PPI derived from this link. Both of those claims are well supported.\n\nWhile the paper clearly states it is focused on mean estimation (it's in the title of the paper!), the abstract makes no reference to this and could be misleading. I would recommend the abstract is amended to be consistent.", "methods_and_evaluation_criteria": "While the link between PPI++ and linear regression is explored theoretically, the extensions from sections 4.2 and 4.3 are only evaluated empirically. Given the concerning empirical evaluation in Appendix A, more theoretical evaluation would be preferable.", "theoretical_claims": "Proofs and derivations appeared correct.", "experimental_designs_or_analyses": "Experimental design was sound.\n\nRegarding section 5.4, I do not clearly understand what this evaluation is doing. I understand the input dataset comes from multiple LLMs, and my understanding is the dataset is being partitioned by LLM for this section. Given the following Section 6 is an extended discussion of these results, more framing may be appropriate. In particular, make it clear up front that the different LLMs have different variances, and perhaps discuss why this is true, if the reason is known.", "relation_to_broader_scientific_literature": "This paper discusses potential extensions for PPI, which is a general procedure for statistical inference that could impact many fields. That said, the extension is focused on one particular use case (mean estimation), which arguably is the most important use case, but nonetheless could limit the impact of the results. The discussion in Section 6 could be critical for the PPI literature as a whole, and deserves further study.", "essential_references_not_discussed": "All necessary references appeared to be included", "other_strengths_and_weaknesses": "The paper proposes several straightforward extensions to PPI for mean estimation that will likely improve results in practice. The exposition is generally very clear and easy to follow. The discussion in Section 6 is quite intriguing.\n\nHowever, while the paper presents several intriguing ideas, it ultimately feels somewhat incomplete. For example:\n- The paper only focuses on mean estimation, and makes no attempt to cover more general M-estimation that the original PPI papers cover.\n- There is no discussion or empirical results for inference procedures such as confidence intervals.\n- Footnote 1 mentions the introduction of bias when $n$ is small, but there is no further theoretical or empirical evaluation of this bias.\n- The results of Appendix A are concerning, and potentially undermine the extension in Section 4.3.\n- The discussion in Section 6 is very interesting, but feels incomplete.", "other_comments_or_suggestions": "- Should equation 9 be $g(f(X)) := \\frac{1}{1+exp(-af(X) + b)}$?\n- The abstract contains the word 'worse' repeated: \"can perform even worse worse than classical inference\"\n- I'm not sure this paper is an appropriate place to call out a biased implementation of software.", "questions_for_authors": "1. Is this method relevant for more general M-estimation? The equivalence to OLS is obvious for mean estimation, but perhaps something analogous exists for other estimates of interest?\n2. Do the proposed extensions impact confidence intervals while retaining coverage, at least empirically? \n3. Following footnote 1, have you performed an empirical evaluation of whether the bias exists in the empirical results?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUgZAuB9IO", "reviewer": "Reviewer_aoGS", "summary": "This paper makes the observation that the variance-minimizing parameter $\\lambda$ in PPI++ can be seen (in the scenario where $N$ is large) as a linear regression coefficient when regressing $Y$ on $f(X)$.  Motivated by this observation, this paper proposes an analog of ridge regression, where $\\lambda$ is regularized towards zero, and logistic regression, where instead of a linear transformation, a non-linear transformation is used.  Further motivation is given later in the paper, when it is observed that the variance of the PPI++ estimate depends inversely to the variance of $f(X)$.  Experiments on some standard benchmark datasets, as well as a (proprietary?) dataset of LLM refusals demonstrate improved estimation performance (as measured by MAE) compared to PPI++.\n\n**Update after rebuttal**:  As discussed below, I have increased my score from 2 to 3", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review. Your questions and comments have helped to make the draft much stronger and clearer. In the camera ready version of the paper, we will improve the related works section and better describe the LLM refusal dataset. We address your individual concerns below.\n\n**Use of MAE instead of MSE**\n\nEvaluating our method in terms of MSE is an excellent suggestion. MAE was initially used as it is more interpretable. In the camera-ready version of the manuscript, we will be sure to use MSE instead of MAE.\n\nThis change produces nearly identical results to that which we presented in the paper. We have included a small set of results to demonstrate this.\n\n\n|Dataset | MSE of PPI (n=5) | MSE of Ridge-PPI (n=5)|\n|--|---|---|\n|Alphafold | 0.042 | 0.019 |\n|Ballots | 0.391 | 0.307 |\n|Census Healthcare | 0.445 | 0.356 |\n|Forest | 0.034 | 0.015 |\n|Galaxies | 0.075 | 0.041 |\n|Plankton | 0.03 | 0.012 |\n\n\n\n**Expanding Section 6, Further Theoretical Justification for Ridge-PPI**\n\n\nSeveral reviewers requested further investigation of the theoretical analyses on where PPI++ succeeds and fails in section 6, as well as an extended theoretical justification of Ridge-PPI. \n\nTowards addressing these concerns, we below produce an expression for the excess risk of Ridge-PPI - an analogous expression to Expression 13 in the paper that makes the same convergence and independence assumptions. We use the shorthand $\\hat{\\mu}_{PPI_a}$ for the PPI estimate (Expression 3 in the paper) using the Ridge-PPI lambda (Expression 8), as well as $V:= Var\\[\\hat{Cov}[f(X), h(X)]\\](2\\mathbb{E}[f(X)]^2 + (\\frac{1}{N} + \\frac{1}{n})Var[f(X)])$:\n\n\n\\begin{equation}\n    Var[\\hat{\\mu}_{PPI_a}] -Var[\\hat{\\mu}_h]\n    = \\frac{V}{(Var[f(X)] + \\alpha)^2} \n\\end{equation}\n\n\\begin{equation}\n    - \\frac{Var[h(X)]Corr(h(X),f(X))^2}{(1+\\frac{n}{N})n} + \\frac{Var[h(X)]Corr(h(X),f(X))^2\\alpha^2}{(1+\\frac{n}{N})n(Var[f(X)] + \\alpha)^2} \n\\end{equation}\n\nTaking the derivative of this expression and setting it to zero yields the solution:\n\n\\begin{equation}\n    \\alpha^* = \\frac{(1+\\frac{n}{N})nV}{Cov[f(X), h(X)]^2}\n\\end{equation}\n\nThis quantity is difficult to estimate in practice, but supplies more insight into the dynamics of PPI. Notably, this optimal $\\alpha$ is smaller in the case of greater covariance, and greater in the case of large V. Seeing as V is a large positive term in the risk, this expression balances between the potential variance and the potential variance reduction permitted by $Cov[f(X), h(X)]^2$.\n\nWe can also now calculate the expected risk reduction from using Ridge-PPI vs PPI++:\n\n\\begin{equation}\n   Var\\[\\hat{\\mu}_{PPI_a}\\] - Var\\[\\hat{\\mu}PPI\\]\n\\end{equation}\n\n\n\\begin{equation}\n    = \\frac{\\alpha}{(Var[f(X)] + \\alpha)^2}(\\frac{Var[h(X)]Corr(h(X),f(X))^2\\alpha}{(1+\\frac{n}{N})n} - V\\frac{2Var[f(X)] + \\alpha}{Var[f(X)]^2})\n\\end{equation}\n\nThis expression further demonstrates the potential harms of small $Var[f(X)]$: the smaller $Var[f(X)]$ is, the greater the improvement from using Ridge-PPI over PPI++.\n\n**Cross Validating $\\lambda$**\n\nWe performed some initial experiments using cross validation to fit a scalar regression parameter on a synthetic regression task. We found that while it can perform well in some circumstances, it requires two key assumptions: 1) that one knows a priori the possible range of regressor values, and 2) that one can use a dense grid of values to validate over (e.g. a grid of 50 options in the [0, 1] interval works reasoanbly well, a grid of 10 does not).\n\nWhen these are both true, this cross-validated regression strategy is a constrained function class that can reduce variance. However, our use of cross validation for selecting an $\\alpha$ parameter did not have these assumptions, as the $\\lambda$ values we were effectively searching over were exclusively ridge regressors. We appreciate this comment, however, as it helped us to improve our understanding of the scalar regression problem and exemplified the computational efficiency of Ridge-PPI.\n\n**Creating Confidence Intervals**\n\nReviewer NpF2 shared this concern. Due to the lack of a general rebuttal as well as space restrictions on individual rebuttals, we have included the response to this in the rebuttal to reviewer NpF2. Please read it to see our additional explanation for this issue.\n\n**Research Datasets and Var[f(X)]**\n\nWe investigate whether the trends observed in Section 5.4 are persistent in the research datasets by calculating the ratio of the MSE at $n=5$ for PPI++ to the MSE at $n=5$ for Ridge-PPI for each dataset and taking its correlation with $Var[f(X)]$ for that dataset. We find that there is a strong anticorrelation between these values (Pearson r = -0.69), further supporting our hypothesis that small $Var[f(X)]$ hinders the efficacy of PPI++. While we do not have enough datapoints to make this correlation statistically significant, it serves as motivation for the more thorough analysis we present in section 6.", "claims_and_evidence": "From a theoretical / methodological perspective, the claims in the paper are somewhat limited and informal, and mainly focused on conceptual goals and intuitions.  From an empirical perspective, the claim of improved estimation seems reasonably well-supported, though I have some questions regarding the setup (see below).", "methods_and_evaluation_criteria": "Note: As I will return to some concerns in different sections of the review form, I will number them globally as (W1), (W2), etc, not necessarily in order of importance.\n\n(W1) One of the main weaknesses of this paper, in my view, is the choice of evaluation criteria, along two lines.  First, one of the main goals of PPI++ is providing *valid confidence intervals*, not just more accurate point-estimates as measured by mean-squared error (though these are related concepts), and there is no discussion of that tension here that I could see, nor any discussion on how one would construct confidence intervals from the proposed approach.  \n\n(W2) Second, even accepting that the goal of this paper is estimation (rather than inference), the empirical evaluation focuses entirely on mean absolute error, which seems like an odd choice as compared to the more typical mean squared error, especially since the \"bias-variance trade-off\" being considered is mainly a trade-off seen with squared errors (e.g., the \"variance\" of an unbiased estimator is exactly the MSE, not the MAE). As a result, I'm not entirely sure what to make of the results in Section 5.", "theoretical_claims": "There isn't much in terms of theoretical claims in this paper, but the claims as written appear to be correct.  There are some opportunities to improve the clarity and precision with which the theoretical claims are made, which I address under \"Other Strengths and Weaknesses\"", "experimental_designs_or_analyses": "For making the main point, which is that empirical performance (at estimation) improves using the proposed methods. \nThe experimental design for both the \"Research Datasets\" (those considered in the original PPI++ paper) and the \"LLM Refusal Dataset\" seems reasonable enough. \n\nThat said, this main analysis could be improved:\n* (W1) First, as mentioned in \"Methods and Evaluation Criteria\", I found the choice of MAE to be somewhat confusing, since the focus otherwise is on reducing variance (and hence, MSE).  \n* (W3) Second, it would be helpful to include some kind of uncertainty quantification in the lines presented (i.e., the average performance is shown, but not e.g., confidence bounds based on bootstrapping).\n\n(W4) For the secondary analyses, I thought that the empirical investigation could be made a bit more rigorous.  In particular, there is a hypothesis that $\\text{var}(f)$ drives performance, and the main evidence provided is somewhat anecdotal (Figure 3).  I would have expected an analysis over the \"research datasets\" as well, asking e.g., whether it always true that when var(f) is smaller (for some definition of \"small\"), that the performance of PPI++ is worse relative to classical.  Figure 4 is nice as a simulated example, but it's less clear what the take-away should be in practice.", "relation_to_broader_scientific_literature": "Given the popularity of PPI++ and related methods, the goal of this paper is very worthwhile, and a strength in my view: Investigating failure modes of the method (particularly for smaller sample sizes) and proposing corrections.  \n\n(W5) However, the observation that the $\\lambda$ parameter in PPI++ is related to a regression coefficient is a relevant observation, but it's not entirely novel or surprising, as I discuss under \"Essential References\".  The findings of this paper are primarily empirical: While there is some connection to ridge regression, there isn't a very rigorous theoretical argument presented as to why \"shrinkage\" should be beneficial here, other than the loose intuition that it might reduces the variance of $\\lambda$.", "essential_references_not_discussed": "(W6) I don't think these missing references are necessarily \"essential\", but I think they would provide important context as to the novelty of the core observation that $\\lambda$ can be interpreted as a regression parameter. While this observation may be a \"novel\" observation in the context of PPI++, it's hardly novel in the broader context of similar estimators in the statistics community.  [Cheng and Cai 2021], for instance, consider similar linear combinations of estimators in causal inference, and make this connection explicit, see e.g., Equation (9) of [Cheng and Cai 2021] and discussion below it, in particular the discussion of how \"the MSE decomposition in (9) has the form of a ridge regression objective\". See [Oberst et al. 2022] for a (somewhat out-dated) review of similar estimators that deal with a slightly different setting (since it involves combining biased and unbiased estimators, while in this setting both the classical and PPI++ estimators are unbiased). While the setting is a bit different, those papers consider a problem very closely related to Equation (7) of the present paper, where you want to minimize the MSE of a linear combination of a biased (here, $f(X)$) and unbiased estimator (here, $h(X)$).\n\n[Cheng and Cai 2021] D. Cheng and T. Cai. Adaptive combination of randomized and observational data. arXiv preprint (2111.15012v1), 11 2021. URL: http://arxiv.org/abs/2111.15012v1, arXiv:2111.15012v1.\n[Oberst et al. 2022] M. Oberst, A. D' Amour, M. Chen, Y. Wang, D. Sontag, and S. Yadlowsky. Understanding the risks and rewards of combining unbiased and possibly biased estimators, with applications to causal inference. arXiv preprint (2205.10467v2), May 2022. URL: http://arxiv.org/abs/2205.10467v2, arXiv:2205.10467v2.", "other_strengths_and_weaknesses": "Let's start with strengths: First, as I mention in \"relation to the broader scientific literature\", I think the goal of this paper is very worthwhile (in particular, giving a critique of PPI++), and I was excited to read it.  The observations, while very conceptual and somewhat heuristic, are nonetheless interesting:  Given that $\\lambda$ can be viewed as a regression parameter, why not regularize it, or more explicitly perform regression?\n\nSecond, I appreciated the extra intuition provided in Section 6:  If $\\lambda$ is independent (easily accomplished via cross-fitting), then its variance plays a role in the excess variance (Equation 11), which gives some intuition and motivation for seeking to minimize it's variance.\n\nGiven these strengths, I was a little disappointed not to see these insights get more fully developed, leaving the current paper a bit \"thin\":\n* (W5, see above) On the theoretical side, I would have expected a slightly more rigorous treatment of the impact and motivation for $\\alpha$, e.g., what is the impact of $\\alpha$ when you push it through the other calculations for e.g., the MSE of the resulting estimator?  The \"ridge regression\" intuition is a bit hand-wavy, since we typically regularize when there are more parameters than data-points, which is not the case here.\n* (W7) On the empirical side, the fact that $\\alpha$ is chosen by cross-validation seems to present some additional questions not explored here.  For instance, how does this compare to just directly selecting $\\lambda$ using cross-validation?  How robust are these conclusions to the choice of $\\alpha$?  What kind of $\\alpha$ values end up being selected here?\n* (W8) I found the experimental setup to be lacking in some other important details, e.g., on the nature of the LLM refusal dataset (see \"Questions for Authors\")", "other_comments_or_suggestions": "Typo in the abstract: \"PPI++ method can perform even **worse worse** than classical inference\"", "questions_for_authors": "These questions are directly related to the \"weaknesses\" I mentioned above, and are given in the order of importance for changing my evaluation.\n\nThese questions are most important:\n1. (W2) What do the empirical results look like if you use MSE instead of MAE?  Is there a reason to prefer MAE over MSE?\n2. (W7) How do these results compare to just directly selecting $\\lambda$ using cross-validation?  How robust are these conclusions to the choice of $\\alpha$?  What kind of $\\alpha$ values end up being selected using the present approach (very large, very small, etc)?\n3. (W5) Can you precisely characterize the impact of $\\alpha$ on the performance of the estimator, and the theoretically optimal choice of $\\alpha$?\n4. (W4) What is the association between $\\text{var}(f)$ and the performance of PPI++ in the research datasets?  Do the same trends hold?\n\nThese questions are less relevant for changing my score, but I'm including them for completeness:\n* (W8) Could you say more about the source of the LLM Refusal Dataset, even in anonymous terms?  E.g., is it a proprietary dataset, or is it a public dataset?  How many labelled examples exist to estimate the ground truth refusal rate?  What is the overall refusal rate (e.g., is it very low, or fairly high), etc? \n\nIt is not necessary to respond to my weaknesses (W1) and (W6) which I would assume to be handled by adding some language around e.g., broader literature and how this work differs.  (W3) is resolved by just adding some confidence intervals, which I assume is an easy fix that doesn't need to be commented on.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xhrxF4vz8F", "reviewer": "Reviewer_UvWk", "summary": "This paper works on statistical inference and its applications in model evaluation. The paper analyzes the pitfalls of a previous method, Prediction Powered Inference (PPI), and proposes two variants to improve performance. The first proposed method is to use ridge regression and add an additional hyperparameter $\\alpha$ to the denominator. The other is to use the sigmoid function to improve the expressiveness. Experiments validate the superiority of the proposed method over two baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your review and your suggestions for improving the work. We especially appreciate your thoroughness in reading the paper, and will be sure to fix any typos currently in the draft for the camera ready version of the paper. Below we respond to your individual points.\n\n**Model Evaluation vs. Statistical Inference**\n\nThroughout this work, we empirically and theoretically consider the problem of mean estimation. Statistical inference in the context of science and model evaluation are two interesting applications of this problem, and so we consider them both. However, since these problems are so related, we believe generalizing both of these as mean estimation better represents our methods' broad applicability.\n\n**Error Bars on MSE Plots**\n\nThank you for your excellent point about the inclusion of error bars in our plots. For the camera ready version of the paper, we will re-run the experiments and include appropriate error bars. Here is a small set of 95\\% confidence intervals for the normalized MSE (see figure 1) of different methods across 350 trials. Here the confidence is represented with a tuple (lower bound, upper bound):\n\n|Dataset | PPI (n=5) | Ridge-PPI (n=5)| Sigmoid-PPI (n=5)|\n|--------|------------------|------------------------|--------|\n|Alphafold | (1.12, 1.65) | (0.49, 0.72) | (0.55, 0.82)|\n|Ballots | (0.27, 0.47) | (0.24, 0.44) | (0.34, 0.6)|\n|Census Healthcare | (0.71, 0.95) | (0.68, 0.92) | (0.58, 0.8)|\n|Forest | (1.92, 3.36) | (0.71, 1.22) | (0.82, 1.09)|\n|Galaxies | (0.84, 1.14) | (0.61, 0.9) | (0.71, 0.97)|\n|Plankton | (1.23, 1.89) | (0.56, 0.76) | (0.54, 0.68)|\n\n\n**Sigmoid Function Improving Performance**\n\nUsing the sigmoid function in Sigmoid-PPI can improve performance when the mapping from $f(X)$ to $h(X)$ is not linear. For instance, since $h(X)$ is a binary value in the refusal rate dataset, mapping $f(X)$ to (0,1) with a sigmoid function could lead to a lower MSE in the regression problem described in expression 7, which we showed to be equivalent to the variance of PPI.", "claims_and_evidence": "- The claim that PPI has a high variance when the number of data is limited is true. The proposed method also works well and makes sense. Theoretical and empirical analyses confirm the effectiveness of the proposal.\n\n- I have a concern about what the focus of the paper is. It seems that the proposed method is a more reliable statistical inference method with lower variance. However, it seems that it partially deviates from the model evaluation. The first experiment of benchmark data sets is not related to model evaluation. Therefore, it is more appropriate to say that the paper mainly works on statistical inference rather than model evaluation.", "methods_and_evaluation_criteria": "The proposed method works well for the problem set. The effectiveness is demonstrated by extensive analysis.", "theoretical_claims": "The theoretical claims are right.", "experimental_designs_or_analyses": "- The experiments consist of two parts. The first part is on benchmark datasets used in the related work. The second part is on large language models (LLMs), and the goal is to estimate the true rejection rate. The experiments are extensive and validate the effectiveness in statistical inference. However, since the main purpose of this paper is model evaluation, I think more experiments on automatic evaluation should be included.\n\n- It is appreciated that multiple seeds are run for each experiment. For example, 350 different seeds are considered in the paper. However, only the average is shown in the figures. It is advantageous to consider statistical tests and to show the variances in the figures.", "relation_to_broader_scientific_literature": "The proposed method is useful for statistical inference and shows promise for application in broader scientific areas.", "essential_references_not_discussed": "There are no essential references that need to be discussed further.", "other_strengths_and_weaknesses": "Strengths:\n\n- The problem studied in the paper, i.e. more accurate statistical inference, is important and interesting.\n\n- The proposed method is simple but effective. \n\n- Extensive experiments confirm the effectiveness of the proposed method.\n\n- The analysis is good and easy to understand.\n\nWeaknesses:\n\n- It seems that variance reduction is not so rare in the machine learning literature. I wonder if there is related work that also proposes similar ridge regression techniques for variance reduction.\n\n- Since the focus is on model auto-evaluation, the paper should spend more space discussing this and provide more related experiments.\n\n- More detailed descriptions of the proposed method should be included. First, how is the ridge regression problem constructed and how is Eq. (8) derived? Second, why can the introduction of the sigmoid function improve the performance?", "other_comments_or_suggestions": "There are minor typos in the paper. The paper should be checked carefully. For example, there are two \"is\" in the first sentence of the last paragraph on page 1.", "questions_for_authors": "Please see \"Weaknesses\".\n\n## update after rebuttal\nThanks for the rebuttal. I will keep my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple extension of the PPI methodology via an interpretation as post-processing a biased estimate. While straightforward, the method seems to work well. The reviewers appreciated the contributions; especially the strength of the empirical results. However, they also raised a number of concerns (on clarity, details for the alpha hyperparameter, MSE instead of MAE, confidence intervals,  etc),  which should be addressed for the final version.

Some math/style issues: use \widehat instead of \hat for better style (e.g., \widehad Cov).
Decision: Accept (poster)