Prediction-Powered Adaptive Shrinkage Estimation
Authors: Sida Li
Nikolaos Ignatiadis
Venue: ICML 2025 poster (2025)
Keywords: prediction powered inference
shrinkage estimation
Primary area: general_machine_learning
TL;DR: We introduce Prediction-Powered Adaptive Shrinkage (PAS), a novel method to estimate multiple means using few labeled data points and black-box model predictions.

Abstract: Prediction-Powered Inference (PPI) is a powerful framework for enhancing statistical estimates by combining limited gold-standard data with machine learning (ML) predictions. While prior work has demonstrated PPI’s benefits for individual statistical problems, modern applications require answering numerous parallel statistical questions. We introduce Prediction-Powered Adaptive Shrinkage ($\texttt{PAS}$), a method that bridges PPI with empirical Bayes shrinkage to improve estimation of multiple means. $\texttt{PAS}$ debiases noisy ML predictions $\textit{within}$ each task and then borrows strength $\textit{across}$ tasks by using those same predictions as a reference point for shrinkage. The amount of shrinkage is determined by minimizing an unbiased estimate of risk, and we prove that this tuning strategy is asymptotically optimal. Experiments on both synthetic and real-world datasets show that $\texttt{PAS}$ adapts to the reliability of the ML predictions and outperforms traditional and modern baselines in large-scale applications.

Prediction-Powered Adaptive Shrinkage Estimation
Sida Li
Nikolaos Ignatiadis 12
Abstract
Prediction- Inference (PPI) (Angelopoulos et al., 2023) addresses the scarcity issue by providing frame- work for valid statistical analysis using predictions from black-box ML models. By combining ML-predicted and gold-standard outcomes, PPI and its variants (Angelopoulos et al., 2024; Zrnic & Candès, 2024; Zrnic, 2025) use the abundance of predictions to reduce variance while relying on the accuracy of labeled' data to control bias.
Cerdiction-Poweree Inference (PPI) is powerful framework for enhancing statistical estimates by combining limited gold-standard data with ma- chine learning (ML) predictions. While prior work has demonstrated PPI's benefits for individ- ual statistical problems, modern applications re- quire answering numerous parallel statistical ques- tions. We introduce Prediction- Powered Adaptive Shrinkage (PAS), a method that bridges PPI with empirical Bayes shrinkage to improve estimation of multiple means. PAS debiases noisy ML pre- dictions within each problem and then borrows strength across problems by using those same pre- dictions reference point for shrinkage. The amount of shrinkage is determined by minimiz- ing an unbiased estimate of risk, and we prove that this tuiing strategy is asymptotically optimal. Experiments on both synthetic and real-world datasets show that PAS adapts to the reliability of the ML predictions and outperforms traditional and modern baselines in large-scale applications.
In this work, we adapt PPI to the estimation of multiple outcome means in compound estimation settings. Many applications of PPI naturally involve parallel statistical prob- lems that can be solved simultaneously. For instance, several PPI methods (Angelopoulos et al., 2024; Fisch et al., 2024) have shown improvements in estimating the fraction of spi- ral galaxies using predictions on images from the Galaxy Zoo 2 dataset (Willett et al., 2013). While these methods focus on estimating single overall fraction, richer analy- sis emerges from partitioning galaxies based on metadata (such as celestial coordinates or pre-defined bins) and esti- mating the fraction of galaxies within each partition. This compound estimation approach enables more granular sci- entific inquiries that account for heterogeneity across galaxy clusters and spatial locations (Nair & Abraham, 2010).
We demonstrate, both theoretically and empirically, the ben- efits of solving multiple mean estimation problems simulta- neously. Our approach builds on the empirical Bayes (EB) principle of sharing information across problems (Robbins, 1956; Efron, 2010) as exemplified by lames-Stein shrink- age (James & Stein, 1961; Xie al., 2012). The connection between modern and classical statistical ideas allows to perform within problem PPI estimation in the first place, fol- lowed by a shrinkage step reusing the ML predictions in an adaptive way, which becomes possible through borrowing information across problems. Our contributions are:
1. Introduction
A major obstacle in answering modern scientific questions is the scarcity of sold-standard data (Miao al., 2024b). While advancements in data collection, such as large-scale astronomical surveys (York et al., 2000) and web crawl- ing (Penedo et al., 2024), have led to an abundance of covariates (or features), scientific conclusions often rely on outcomes (or labels), which are often expensive and labor-intensive to obtain. The rapid development of ma- chine learning (ML) algorithms has offered a path forward, with ML predictions increasingly used to supplement gold- standard outcomes and increase the statistical efficiency of subsequent analyses (Liang et al., 2007; Wang et al., 2020).
1. We propose Prediction- Poweeeed Adaptive Shrinkage (PAS) for compound mean estimation. PAS inherits the flexibility of PPI in working with any black-box predic- tive model and makes minimal distributional assumptions about the data. Its two-stage estimation process makes efficient use of the MI predictions both variance- reduction device and shrinkage target.
Data Science Institute, The University of Chicago 'Department of Statistics, The University Chicago. Correspondence Sida Li <listar2000 @isttr2200@uuccieooo..... uchicago.edu>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Thoouhhout the paper, use the terms "laheled" and "gold- standard" interchangeably.

---

Prediction-Powered Adaptive Shrinkage Estimation
2. We develop a Correlation-Aware Unbiased Risk Estimate (CURE) for tuning the PAS estimator, establish asymp- totic optimality of this tuning strategy, and derive an in- terpretation in terms of Bayes oracle risk upper bound.
Writing jren N (x,)) (Y we find that [[ [ = 0 and
~V Li=1
1 Var Var[)((X)]+ N
1 Var ((xi), (3) n
Var [0Pl)]
3. We conduct extensive experiments on both synthetic and real-world datasets. Our experiments demonstrate PAS's applicability to large-scale problems with deep learning models, showing improved estimation accuracy com- pared to other classical and modern baselines.
that is, gPPt is unbiased for 0 and its variance becomes smaller when the model predicts the true outcomes well. The mean squared error (MSE) of (P is equal to Varl Although we motivated @PPT in (2) as implementing a cor- rection step on two possible baseline estimators (Y and 2 @PPI may have MSE for estimating @ that is arbitrarily worse than either of these baselines.
2. Preliminaries and Notation
2.1. Prediction-Powert Inference (PPI)
Comparison to classical estimator Y. The classical esti- mator Y which only uses labeled data is unbiased for @ and has variance (and MSE) equal to 1 Var[M]]
The PPI framework considers a setting where we have ac- cess to small number of labeled data points (X,, Y))= E (X x 3 and a large number of unlabeled covariates (X))) E (X)N, where X' and Y' represent the covariate and outcome space, respectively. The data points are drawn iid from a joint distribution Pxy.² We are also given a black-box predictive model f X' + y that is independent of the datasets (e.g., pre-trained on similar but unseen data). For mean estimation with y C R, the goal is to leverage the predicted outcomes f f(X;) to improve the estimation of 0 := E[Y]]. Some simple estimators take the form of the following aggregated (summary) statistics
Power-Tuned PPI (PPI++). To overcome the above limi- tation, Angelopoulos et al. (2024) introduce a power-tuning parameter and define
@PTL :Tt = 4 P -21)
21
(4)
which recovers the classical estimator when À = and the vanilla PPI estimator when À = 1. For all values of X, 6pPl is unbiased, so if we select the À that minimizes Var( we can improve our estimator over both the classical esti- mator and vanilla PPI. Such an estimator is defined as the Power-Tuned (PT) PPI estimator 0 := 6PPL where we pick X that minimizes the variance (and thus the MSE) of @PPL. We will revisit PT as one of the building blocks of our proposed PAS estimator in Section 4.
8
Y
8
(1)
1 N f(X.).
2 f(X),
2'= = N 1=1
Comparison to 21. Consider the ideal scenario for PPI with N = oo (that is, the unlabeled dataset is much larger than the labeled dataset) so that Ž' = (X,)). Even then, the MSE of @PPl in (3) is always lower bounded by Var [Y; Xx]] and the lower bound attained by the n
Above, Y is the classical estimator, 1 23,2 2 are the predic- tion means the labeled and unlabeled data, and Y (grayed out) is unobserved. The vanilla PPI estimator is defined as,
1 !
@ePl
Y
[ 17--21 2
Y - ZT)
(2)
perfect ML predictor f(-) E[Y; X; = -]. In words, if Y, is not perfectly predictable from X,, then PPI applied to labeled dataset of fixed size must have non-negligible MSE. By contrast, for N = oc, the prediction mean of unlabeled data 2/ has zero variance and MSE equal to the squared bias (E[f(X,)] - 0,)2. Thus if the predictor satisfies a calibration-type property that [Xf( = E[Y] (which is implied by, but much weaker than the requirement f(X.) Y), then the MSE of 21 could be nearly 0. By contrast, PPI (and PPI++) can only partially capitalize on such predictor f(-).
Baseline
Variance Reduction
Baseline
Debiasing
Both definitions represent @PPI in the form of a baseline estimator plus a correction term In the first rep- resentation, the baseline estimator is the unbiased classical estimator Y. while the correction term has expectation O and attempts to reduce the variance of Y. In the second representation, the baseline estimator is the prediction mean on unlabeled data Z (which in general may be biased for 0), while the correction term removes the bias of 21 by es- timating the bias of the ML model ff on the labeled dataset.
While PPI and PPI++ are constrained by their reliance on unbiased estimators, we show that the compound estima-
To be concrete: (X,, H1 Pxy and (X,, Vi) 2 Pxx inde- pendently, but Y. is unohserved.
"We use the term "PPI++" for the broader framework, while "PT" refers to the specifie estimator.
FFrmm on, we will use the term "classical estimator" to refer to the sample average of the labeled outcomes.
"The same lower bound also applies to power-tuned PPI @"T.
2

---

Powered Adaptive Shrinkage Estimation
tion setting (Section 2.2) enables a different approach. By carefully navigating the bias- varianeee tradeoff through in- formation sharing across parallel estimation problems, we can provably match the performance of both Y and 2'.
ny= labeled points
10 problems
0.5
2.77
210
0.0
2.2. The Compound Mean Estimation Setting
-0.5
-1.0
1.0
X
In this section, we introduce the problem setting that PAS is designed to address seddress-sttimtting estimating the mean of m > 1 parallel problems with single black-box predictive model f. For the j-th problem, where j E [m]] := (I...., m). we ob- serve a labeled dataset (Xi,, Y1)) with n, E N samples and an unlabeled dataset (X ) with N, E N samples. We start with modeling heterogeneity across problems.
N1= 20 unlabeled points projected onto y=|k)
1.00
2.42.
2
0.75
0.50
2.16
0.25
0.00
-1.0
-0.5
0.0
0.5 1.0
1.5
8
Assumption 2.1 (Prior). There exist problem-specific un- observed latent variables Ti with
Figure I. We instantiate the model described in Example 2.3 with
m = 10 problems, each has ny = 10 labeled and N, = 20
unlabeled data (we use different colors for all 10 problems). (Top)
iid Th 2 F,,, j E [m], and 17 : (111 .. Tm
(5)
Labeled data (Xej. Y.y) y with the classical estimator Y, shown for each problem. (Bottom) We apply flawed predictor f(G)
where P., is an unknown probability measure. The latent variable fy fully specifies the distribution of the j-th labeled and unlabeled dataset. We use the notation E,, [-] (resp. E to denote the expectation conditional on nij (resp. m), while Ee,, [-] denotes an expectation also integrating out P,,,-
|a|] to the unlabeled covariates and visualize (Xaj. f(X,,)) as
well as the prediction mean 2
iid, independent of the labeled dataset and identically dis- tributed with f(Xig). In the notation of (7), F3 represents an unspecified distribution satisfying the moment constraints in (6) and
We do not place any restriction over the unknown prior P., Assumption 2.1 posits exchangeability across problems, which enables information sharing, without restricting het- erogeneity (Ignatiadis et al., 2023). In our setting, we are specifically interested in the means
E,, [f(x)i - Mj; Corr,, (f(it)), Y,yl = Pjr Var,, [1y] = oo.
Var [f(Xi;]]= TJ
We further denote 7i-- == Cov (f(x))) Yol = P.T
8j= = E,, Myy, j [m], and 0 = (01, ..., 6m.
(6)
Similar to Eq. (I), we define the aggregated statistics Yj, 2/ 2 for each j E [m]. To facilitate exposition, fol- lowing prior work," we treat the second moments 7j. oj. Y)
Our next assumption specifies that we only model the first two moments of the joint distribution between the outcomes and the predictions. The upshots of such modeling that the exact form of the observation distribution is neither as- sumed nor required in our arguments, and that our approach will be directly applicable to settings where the covariate space X' is high-dimensional or structured.
as known until the end of Section 5. In Section 6, extend our methods to allow for unknown rj. oj. and 7/-
We next introduce a synthetic model that will serve both as a running example and as part of our numerical study.
Example 2.3 (Synthetic model). For each problem j. let Mi ui-1, 1]. We think of n6 as both indexing the problems and generating heterogeneity across problems. The j-th dataset is generated via (with constants set to 0.05, v = 0.1),
Assumption 2.2 (Sampling). For each problem j E [m]] we assume that the joint distribution of (f))))). Yτ²) has finite second moments conditional on 77 and for i E [ny]
f(x+)) Yi
nid ny ~
[1) 03
Tf PGTTO
Drgg0 (7)
F,
(8) In Figure 1, we visualize realizations from this model with problems, n, = 10 labeled observations, and Nj 20 unlabeled observations for each problem. We apply a flawed predictor f(x) = |æ]. The classical esti- mator Y, and the prediction mean 7 deviate from each
Xij ad N v²),, Yoy ind N -13..). c).
where F3, Mj.0,, B,,0,,,0000 o3, +2 are functions of mÿ. Conditional on Nj, the unlabeled predictions f(Xxz), i' E [N5], are also
m
"Our proposal also accommodates using separate predictors {fz 1 for each problem. To streamline exposition, we focus on the practical scenario where single (large) model (e.g., a large language vision model) can handle multiple tasks simultane- ously (Radford et al., 2019; He et al., 2022).
FFor EB, examples include Xie et al. (2012); Soloff et al. (2024); for PPI, see recent works like Fisch et al, (2024).
3

---

Prcdiction-Poweree Adaptive Shrinkage Estimation
other. Nevertheless, 2 contains information that can help us improve upon Yj as an estimator of 0g by learning from within problem (PPI, PPI++, this work) and across problem (this work) structure. We emphasize that, specified in (7), our approach only requires modeling the first and second moments of the joint distribution of (f(X;)), X.y). For in- stance, in this synthetic model, 0j = vı and of = 4n while M, TJ and "/ also admit closed-form expressions in terms of Mi when the predictor takes the form f(x) = |x| or (z) = x² (see Appendix 1.
Table 1. Estimator comparison in the stylized model of Section
Estimator
MSE
VR
CP
gel
of + +
X
X
g -
of
V
X
X
E [-1
(o]+0
X
V
X
(o2 (
[(0 0e) []]
X
V
B 0-1 - E]
v
x
0?+00
To conclude this section, we define the compound risk (Rob- bins, 1951; Jiang & Zhang, 2009) for any estimator 0 = (fI.... )T as the expected squared error loss averaged over problems,
E 0-1 - E, ø]
v
02+-0 dle
Variance Reduction, P. Prior Information, CP: Contextual Prior Information.
which has MSE o and remains unbiased for 0. This idea lies at the heart of approaches such as control variates in simulation (Lavenberg & Welch, 1981; Hickernell al., 2005), variance reduction in randomized controlled experi- ments via covariate adjustment (Lin, 2013) and by utilizing pre-experiment data (Deng et al., 2013, CUPED), well as -asisted estimation in survey sampling (Cochran, 1977; Breidt & Opsomer, 2017). It is also the idea powering PPI and related methods: the unlabeled dataset and the pre- dictive model are used to intercept some of the noise the classical statistical estimator del & Y; compare to Eq. (2) with & - 2 - 7 We refer to Ji al. (2025) and Grons- bell et al. (2025) for informative discussions of how PPI relates to traditional ideas in semi paraametie inference as in c.g., Robins et al. (1994).
R. (0,00 = En fm (0. 07]
(9)
with f (0.0) == 1 m
0, 0,)²
(10)
The Bayes risk, which we also refer simply as mean squared error (MSE), further integrates over randomness in the unknown prior P., in (5),
B 0 = Ep, Rm (00(0))) 0.
(II)
3. Statistical Guiding Principles & Prior Work
In this section, we illustrate both the statistical guiding principles of our approach and some connections prior work through the following stylized Gaussian model:
Prior information (P) via empirical Bayes (EB). In the Bayesian approach we seek to improve upon J by using the prior information that NN oz). The Bayes estimator,
Sampling: gel = - (5 + +), -~ NNO, e NN,, oz)
Prior: A ~ (0 Φ ~ N(O, oz), Corrio, øl p.
In our stylized model, we assume that (0, o, E, are jointly normal and that all their pairwise correlations are zero with the exception of Corr d] p O. We write obi := Var[0 Φ] = (1 p²) og.
E[O [-l
o 0 0
o
reduces variance by shrinking jeh toward O (at the cost of introducing some bias). When of is small, the MSE of E [el) can be substantially smaller than that of gel
We think of jel as the baseline classical statistical es- timator of a quantity 0 that we seek to estimate with small MSE. In our stylized Gaussian model, jel is un- biased for and has noise contribution & + E, so that E[(00)-0 0) [0+) of + o² We describe three high-level strategies used improve the MSE of gel. These strategies are not tied in any way to the stylized model; nevertheless, the stylized model enables us to give precise expressions for the risk reductions possible, see Table 1.
Now suppose that the variance of the prior, o is unknown but we observe data from multiple related problems gen- erated from the same model and indexed by E [m)] say, 0, id NO, oz) and det ind NN(0 o2 +z2). Then an EB anal- ysis can mimic the MSE of the oracle Bayesian that has full knowledge of the prior. To wit, we can estimate of as
03 - Σ 1600 (02 + o?).
Variance reduction (VR). An important statistical idea to improve gel via obtaining further information inter- cept some of its noise, say &, and replacing jel by jel boll- E
and then consider plug-in approximation of the Bayes rule, 0. = Ei0, 0c] := (6)/(0² + o² + + The result- ing estimator is the celebrated Same--Stein estimator (James
*We provide further connections in Appendix A.I.
4

---

Predietion-Powered Adaptive Shrinkage Estimation
& Stein, 1961; Efron & Morris, 1973), whose risk is very close the Bayes risk under the hierarchical model for large m (Efron, 2010, equation (1.25)). The James-Stein es- timator also always dominates the classical estimator under a frequentist evaluation of compound risk in (9) under the assumption that Oc) ind NV(0, of + o²) and m 2 3:
4. Prediction-Powered Adaptive Shrinkage
On a high level, PAS aims to provide lightweight approach that outperforms both baselines in (2) and PPI/PPI++ in terms of MSE when estimating multiple means, PAS also aims at minimal modeling requirements and assumptions.
The stylized example from Section 3 serves as a guiding analogy. We seek to benefit from ML predictions in two ways: first by variance reduction (acting akin to & in the stylized example), and second by increasing prior relevance (acting as a proxy for Φ). We implement both steps to adapt to the unknown data-generating process in an assumption- lean way using within-probless information for the first step (Section 4.1) 1) and across-problen information for the second step (Section 4.2), drawing on ideas from the EB literature.
Rm 5mm(0's Ō 0) < Rm (Ō)) 0) for all 0 E R.".
Contextual prior information (CP) via EB. Instead of using the same prior for each problem, we may try to sharpen the prior and increase its relevance (Efron, 20 by using further contextual information db In the stylized ex- ample, as seen in Table such an approach reduces the vari- ance of the prior from 0g to di < og with corresponding MSE reduction of the Bayes estimator E[ 0t1. ..). With multiple related problems, such strategy can be instan- tiated via EB shrinkage toward an informative but biased predictor (Fay III & Herriot, 1979; Green & Strawderman, 1991; Mukhopadhyay & Maiti, 2004; Kou & Yang. 2017: Rosenman et al., 2023). The strategy of this form that is closest to our proposal is the covariate-powered EB ap- proach of Ignatiadis & Wager (2019), recently applied to large language model evaluation by Fogliato et al. (2024). Therein (following the notation of Section 2.2), the analyst has access to classical estimators Vj. E [m)] and problem- specific covariates W, and seeks to shrink Y, toward ML models that predict Y, from W. By contrast, in our setting we have observation-level covariates Xij and the MI. model operates these covariates. In principle one could simulta- neously use both types of covariates: problem-specific and isservation-speiff.
4.1. The Within Problem Power- Stage
Extending the notation from (4) to each problem provides us with a class of unbiased estimators J.A ance gives
@PM == Y, + A(2 2J). x E R. Calculating the vari-
=:8,(A)
Varo,
OPPI
2 n N nj n
X2 1.
2 n)
Note that the classical estimator has risk o²/ /nj and gets out- performed whenever (A) < O. We can further analytically
solve for the optimal X, which yields
N n4 + N₂
A == arg 5,()) A
(13)
77
and the Power-Tuned (PT) estimator G.T
@rP! with j,25
Combine variance reduction (VR) and prior information (P). One can shrink the variance reduced estimator ا²l & toward 0 via E [ 00ق - <] = {ag/(o² + (0)) - E). In the context of PPI, variance reduction and prior information (with a more heavy-tailed prior) are used by Cortinovis & Caron (2025) within the Frequentist-Assisted by Bayes (FAB) framework of Yu & Hoff (2018). Cortinovis & Caron (2025) only consider single problem and do not pursue an empirical Bayes approach.
03 = Varns TOT
2 nj
N o
(14)
n (n(ng N₂) 7
The formulation of the above PT estimators is well under- stood in the single problem setting (Angelopoulos al., 2024; Miao et al., 2024a). In PAS, we execute this stage separately for each problem, as the optimal power-tuning parameter is problem-dependent and varies case by case.
Combine P, CP, and VR together. Finally, in our stylized example, we can get the smallest MSE (last row of Table 1) by using both variance reduction, shrinkage, and a contex- tual prior. In that case, the Bayes estimator R(O dell E, Φ) takes the form,
4.2. The Across Problem Adaptive Shrinkage Stage
The PT estimator derived in Section 4.1 already possesses many appealing properties: it is unbiased and has lower variance than both the classical estimator and vanilla PPI. However, as our setting involves working with many par- allel problems together, there the opportunity of fur- ther MSE reduction by introducing bias in targeted way. 9 Concretely, based on the PT estimator obtained in
2 o3 + o 0/
0f
(gel
5) +
E
6).
(12)
o3 + 00
EB ideas can be used to mimic the estimator above and provide the starting point for the proposal we describe next.
"See Appendix A.2.1 for some explanations about why we can improve MSE by borrowing information across problems.
5

---

Prediction-Powered Adaptive Shrinkage Estimation
Section 4.1, consider class of shrinkage estimators
1 teni
CO Apprigeeto
(3) Pover Tadlgg (ottle enbkir)
Alastiv (saress prodan)
0 &AS = (6P.s ..... (RAS)) where for any w 0,
F2
0 = w (1 (1-ww)?? 2/
Z
(15)
with wi = wy = w + 02
i
2/
gPas
The motivation is to formally match the form of the Bayes estimator with variance reduction and contextual prior infor- mation in (12) with the following (approximate) analogies: 10
Figure 2. A flowchart illustration of the PAS method. See Algo- rithm for pseudo-code implementation.
gdl c レ T 0ee +++ oj
z/.
Algorithm 1 Prediction Powered Adaptive Shrinkage
(16)
Require: (Xij, Yt)) (8.))= T5, oJ for j E [m], pre-
0 w
dictive model f
1: for j = - to m do
The highlighted is global shrinkage parameter that acts as follows:
2:
> Step 1: Apply predictor (Eq. (1))
3:
,, 2 . = Nottmeans(((A) ((X,,, Y (1) (2.)
4:
Step 2: Power tuning (Eq. (13)
(i) Fixing w, any problem whose PT estimator has higher variance possesses smaller wy and shrinks more toward 2/== a smaller variance increases Wj and makes the
5:
Aj get pett.ptppaam(( (5,, 7J, Rý, N2)
6:
0.T Y, (2 2
7: o² = get pt..ptt..
(Eq. (14))
final estimator closer to 0FTr.
8: end for
9: Step 3: Adaptive shrinkage (Eg. (18)
10: a = get shrink param ( ))= (2 )5)) (2)))=1, ()))))))))
(ii) Fixing all the problems, increasing has an overall effect of recovering 0FT for all j (full recovery when
11: for j = to m do
12: Wj = @/( o5)
w 20), and setting w = 0 recovers Z/.
13: 0PAS = -5055 + (1 (1)) 25)
14: end for
Points (i) and (ii) establish the conceptual importance of w. If we could choose in optimal way, that is,
15: return (0PAS pm 1;-1
w" E arg min { RRm (BMAS, . })}
Theorem 4.1. Under Assumption 2.2. CURE is an unbi-
220
ased estimator of the compound risk defined in (9), that is, for all w 2 0 and all n,
then the resulting estimator _- would satisfy all our desiderata. While this construction is not feasible since the compound risk function in (9) depends on the unknown '7, 0, we can make progress by pursuing a classical statistical idea: we can develop an unbiased estimate of the compound risk (Mallows, 1973; Stein, 1981; Efron, 2004) and then use it surrogate for tuning
PAS
En [CCCURE
0 (
(OEAS .0)
Rm
See Appendices B and F.I for the proof and motivation. With Theorem 4.1 in hand, we now have systematic strategy of picking w by minimizing CURE, following the paradigm of tuning parameter selection via minimization of an unbiased risk estimate (as advocated by, e.g. Li (1985); Donoho & Johnstone (1995); Xie et al. (2012); Candès et al. (2013); Ignatiadis & Wager (2019); Ghosh et al. (2025)):!!
To this end, we define the Correlation aware Unbiased Risk Estimate (CURE),
CURE (0 PAS
m
(2wj - 1)33 + +2(1-w)) w -
m i=1
PAS Ws E arg min CURE w20
+ (1 w1- (QTT =z)a|.
(18)
Both the formula and our nomenclature ("correlation- aware") highlight the fact that we must account for the potentially non-zero covariance between shrinkage source 0uT and target 2/. which can be explicitly written down as
Even though à does not admit closed-form expression, the one-dimensional minimization can be efficiently carried out numerically (e.g., grid search). The final PAS estimator is:
Il "The connection to EB is the following. Xie et al. (2012) and Tibshirani & Rosset (2019) explain that James-Stein-type estimators may he derived by tuning az (in Section 3) via mini- mization of Stein's 's (1981) unbiased risk estimate (SURE).
33 = Cov,, 1OTT . = X Var, [2/]
7 n2 + N,
(17)
We comment more on these analogies in Appendix A.2.2.
6

---

Coddiction-Powerte Adaptive Shrinkage Estimation
Power-Tuning Parameters
0pAS = 0pas w + 3 e + 2
33
0PT
0.8 000
0.4
Figure 2 visualizes the full method for constructing the PAS estimator- from applying the predictor and obtaining aggre- gated statistics to going through the two stages described in Section 4. and this section. A pseudo-code implementation also presented in Algorithm
fx00 x
600) = [x]
Adaptive Shrinkage Parameters
1.0
Ww wWw
0.8
To illustrate the flexibility and adaptivity of PAS, we briefly revisit the synthetic model in Example 2.3, whose special structure allows us to visualize how the power-tuned and adaptive shrinkage parameters vary across problems and different predictors. In Figure 3, we consider m = 200 problems and two predictors: a good predictor (1r = x2 and flawed predictor f2(x) |arl. The model setup in (8) is such that the magnitude of Cov,, [X,, Yý] relative to Varny [Yg] is much larger for problems with my closer to the origin. Therefore, for both predictors, we see dip in >j near the middle (top panel), which shows that PAS adapts to the level of difficulty of each problem when deciding how much power-tuning to apply. On the other hand (bottom panel), the overall shrinkage effect is much stronger (smaller Wj for all j) with fi than with f2. which demonstrates PAS'S ability to adapt to the predictor's quality across problems- while still allowing each problem to have its own shrinkage level. Numerical results are postponed to Section 7.
0.6
fXX) x²
0.2
1000 |X]]
-1.00
-0.75
-0.50 Problems indexed by n: E[X,]]
-0.25
c.00
0.25
0:50
0.75
1.00
Figure 3. The power-tuned and adaptive shrinkage parameters, Aō and Wü across m 200 problems in Example 2.3. On the z-axis, we identify the problem by its nu so the trend is more visible.
so BTT
PAS 2
S min { [ (Z'). BTd (0 PT }}}+ o1).
Our next proposition connects Theorem 5.2 with the lowest possible MSE in the stylized Gaussian example of Section (last row of Table 1).
Proposition 5.3. In addition to the assumptions of Proposi-
tion 5.1, further assume that N, = oc and that there exist E N, 5² 0 such that nj n. and o² = = for all j almost surely. Let 3 : Er, [[] - 8 (which does not depend on j as we are integrating over P.,). Then,
5. Asymptotic Optimality
In (18), we proposed selecting @ by optimizing an unbiased surrogate of true risk. In this section, we justify this proce- dure theoretically. Our first result establishes that CURE approximates the true loss (whose expectation is the com- pound risk in (9)) uniformly in w as we consider more and more problems.
B
0 PAS 10
02 + 32 + o(1)
0²8²
as m + co.
00.
To interpret the result, is instructive to compare the asymp- totic upper bound on the MSE of PAS with the MSE the last line of Table 1, i.e., with (0 a)/ + o Ob- serve that d² plays the role of o² (as already anticipated in (16)) which is smaller than the variance of the classical estimator (due to power tuning). Mcanwhile, 8 plays the role of o If the baseline 2í (that is, the mean of the ML predictions on the unlabeled datasets) is doing a good job of predicting 0,, then 32 will be small, and so PAS may have MSE substantially smaller than that of PT. On the other hand, even if 32 is large (that is, even if the ML model is very biased), PAS asymptotically still has MSE less than or equal to Ø², the MSE of PT. We emphasize that the role of Proposition 5.3 is to provide intuition at the expense of strong assumptions. 12 By contrast, the state- ment of Theorem 5.2 does not restrict heterogeneity (e.g., heteroscedasticity) across problems and allows for varying, finite unlabeled and labeled sample sizes.
Proposition 5.1. Suppose the datasets are generated ac- cording to Assumptions 2.1 and 2.2 and further assume that Er, Y bo, Er, [f,[[(X (Xx,)) < co. Then,
sup CURE (PPA)) E20
-m (OPAS . - o(1), =o(1,
Ep,
where 1) denotes a term that converges to 0 as m + co.
A principal consequence of Proposition 5.1 is that PAS
with the data-driven choice of w in (18) has asymptotically smaller Bayes MSE (defined in (11)) than any of the esti- mators in (15), i.e., it has smaller MSE than both baselines in (2) as well as the PPI and PT estimators.
Theorem 5.2. Under the assumptions of Proposition 5.1,
B 10
PAS
inf w00
(0
PAS)
}
o(1) as - oo.
1WW further elaborate on this proposition in Appendix A.2.3.
7

---

Ceeliction-Powered Adaptive Shrinkage Estimation
6. PAS with Unknown Second Moments
Estimator
MSE fi (x10
MSE f2 (x10
So far we have assumed that the second moments of: 73.77 75 in (7) are known. In practice, e-g., in our numerical ex- periments with real-world datasets, we use sample-based estimates o 23 and b instead (Appendix C.I). This ap- proach works well empirically, even for small ni, but is not covered by the theory above. To address this shortcoming, we develop variants UniPT (Appendix C.2) and UniPAS (Appendix C.3) that provide asymptotic guarantees (as m + (c) without requiring known or consistently estimable second moments (n,, .V, remain bounded). Briefly, UniPr targets optimal single power-tuning parameter À across all problems, and does consistently m - 3c. UniPAS then builds upon Un iP, applying adaptive shrinkage simi- lar to PAS, but with mechanisms to handle the unknown mo- ments. These extensions are justified by theoretical results akin to those for PAS (ef. Theorem 5.2). In our experiments below, UniP PAS is competitive with PAS.
Classical Prediction Avg
3,142 + 0.033
3.142 + 0.033
0.273 + 0.004
34.335 + 0,147
PPI PT
2.689 + 0.027
2.756 + 0.027
2.642 0.027
2.659 + 0.026
Shrink Classical
0.273 0.003
3.817 1 0.042
Shrink Avg PAS (ours)
2.486 + 0.026
2.575 1 0.026
0.272 + 0.003
2.496 1 0.025
Table 2. MSE (#: standard error) of different estimators under the synthetic model with predictors fi()) = x² and fa(x) = |a]]-
m プ (0, 0,)² across K 200 Monte Carlo repli- cates. In the synthetic model, we sample fi (and thus 8₂) from the known prior P... For the real- world datasets, since P. is unknown, we follow the standard evaluation strategy in the PPI literature: we start with large labeled dataset and use it to compute a pscudo-groum truth for each mean f Then in each Monte Carlo replicate, we randomly split the data points of each problem into labeled/unlabeled partitions (where choose 20/80 split ratio). 13
7. Experiments
For real-world datasets, we introduce second metric to assess whether improvements in MSE are driven by a few difficult problems rather than consistent performance gains: the percentage of problems improved relative to the classical estimator (abbreviated as "Improved % 1²)). This metric is defined as 1 (100%). Larger values of this metric preferable.
We apply the proposed PAS estimator and conduct exten- sive experiments in both the synthetic model proposed in Example 2.3 and two real-world datasets.
m Σ 1[(0, 0,)² < (0Classicall 0,) x
All Estimators. We compare the PAS estimator against both classical and modern baseline estimators:
7.1. Synthetic Model
(i.) the classical estimator;
This is the synthetic model from Example 2.3, where we choose m 200, ny 20, and N, 80 for all j. Since we have already visualized the model and the parameters of the PAS estimator in previous sections, we simply report the numerical results (for the good predictor S1 and the flawed predictor f2) in Section 7.1.
(ii.) the prediction mean on unlabeled data;
(ii..) the vanilla PPI estimator (Angelopoulos al., 2023);
(iv.) the PT estimator (Angelopoulos et al., 2024, PPI++))
(v.) the "shrink-classical estimator that directly shrinks the classical estimator toward the prediction mean;
For both predictors, we see that PAS outperforms all the baselines. With a good predictor fi, both the prediction mean and the shrinkage estimator closely track PAS; in con- trast, the PPI and PT estimators fail to fully leverage the accurate predictions, as their design enforces unbiasedness. The situation reverses for the less reliable predictor fz: the prediction mean and the shrinkage estimator have high MSE, while estimators with built-in debiasing mechanisms demon- strate greater resilience. PAS adapts effectively across these extremes, making it handy choice for a wide range of problems and predictors.
(vi.) the "shrink-average" estimator that shrinks 0.T toward the PT group mean across all problems, i.c., toward
gPT m 1 GPT.
(vii.) the UniPT and UniPAS estimators introduced in Sec- tion and detailed in Appendix C.
We include the estimators in (vii.) only for the real-world experiments, as they are specifically designed for settings where the second moments are unknown. See Appendix D for detailed formulations and implementations for (v.) & (vi.) (the latter of which is inspired by the SURE-grand mean estimator of Xie et al. (2012)). The comparisons with (iv.) and (v.) also directly serve as ablation studies of the two stages in constructing the PAS estimator.
7.2. Real-World Datasets
We next evaluate PAS two large-scale real-world datasets, highlighting its ability to leverage state-of-the-arr deep learn-
Metrics. We report the mean squared error (MSE) (+= standard error) of each estimator Ô by averaging
In Appendix E, we vary this ratio from 1% 40%, and pro- vide more details on the benchmarking procedure for MSE
8

---

Predertion-Poweredd Adaptive Shrinkage Estimation
Table 3. Results aggregated over K 200 replicates on three real-world datasets: Amazon review ratings with PeRt-base and BERT-tuned predictors, and spiral galaxy fractions with ResNet 50 predictor. The bottom two rows are the UniPT and UniPAS estimators detailed in Appendix C as variants of PT and PAS. Metrics are reported with standard error.
Amazon (base /)
Amazon (tuned f)
Galaxy
Estimator
MSE (x10
% Improved f
MSE (x10-
% Improved t
MSE (x 10
% Improved t
Classical
24.305 + 0.189
baseline
24.305 + 0.189
baseline
2.073 + 0.028
baseline
Prediction Avg
41.332 + 0.050
30.7 + 0.2
3,945 + 0.011
75.4 + 0.2
7.195 + 0.008
17.0 0.2
PPI
11.063 0.085
62.4 0.2
7.565 + 0.066
70.4 + 0.2
1.149 + 0.017
59.4 + 0.3
PT
10.633 0.089
70.3
6,289 + 0.050
76.0
1.026 0.015
67.7 0.3
Shrink Classical
15,995 = 0.121
0.3
3,828 + 0.039
78,9 0.2
1.522 0.016
48.8 - 0.4
Shrink Avg
9.276 = 0.078
70.4
6,280 +:000:0000 0.058
77.1 0.2
0.976 +
68.9 + 0,3
PAS (ours)
8.517 + 0.071
71,4 0.2
3,287 + 0.024
80.8 + 0.2
0.893+0.00
67.3 + 0.4
UniPT (ours)
10.272 + 0.084
70.0 + 0.2
6,489 + 0.053
76.2 + 0.2
1.117 1: 0.015
67.7 + 0.3
UniPAS (ours)
8,879 + 0.073
69.5 + 0.2
3.356 + 0.031
77.6 + 0.2
0.909 1 0.011
66.7 1 0.3
ing models in different settings." 14
is based on the metadata attribute WVT_BTN, Second, we estimate the fraction of spiral galaxies in all of the galaxy subgroups simultaneously. The predictor is a ResNet 50 network trained on a held-out set with around 50k images.
Amazon Review Ratings (SNAP, 2014). Many commer- cial and scientific studies involve collecting a large corpus of text and estimating an average score (rating, polarity, etc.) from it. A practitioner would often combine limited expert annotations with massive automatic evaluations from ML models (Baly et al.. 2020; Egami et al., 2023; Fan et al., 2024; Mozer & Miratrix, 2025). To emulate this setup, we consider mean rating estimation problems using the Ama- zon Fine Food Review dataset from Kaggle, where we arti- ficially hide the labels in random subset of the full data to serve as the unlabeled partition. Concretely, we estimate the average rating for the top m = 200 products with the most reviews (from 200 to 900).. For the i-th review of the j-th product, the covariate Xij consists of the review's title and text concatenated, while the outcome Yı is the star rat- ing in (1,...,5) 5). We employ two black-box predictors: (1) BERT BERT-base, base, language model without fine-tuning (De-
For both datasets, we randomly split the data for each prob- lem (a food product galaxy subgroup) into labeled and unlabeled partition with a 20/80 ratio. We repeat the random splitting K = 200 times and report metrics averaged over all splits. Here we summarize the results in Table 3:
Amazon Review: similar to the trend in the synthetic model, the more accurate BERT-tured model enables stronger shrinkage for PAS while the biased BERT-base predic- tions necessitate less shrinkage. Our PAS estimator adapts to both predictors and outperforms other baselines. PAS has the lowest MSE and highest % Improved f.
Galaxy Zoo 2: the predictions from ResNet50 are subop- timal, so the variance-reduction from power tuning domi- nates any benefit from shrinkage. PAS achieves the lowest MSE among all estimators, and improves individual esti- mates at a level on par with the shrink-average estimator.
vlin al., 2019) and (2) BERT- which is the same model but fine-tuned on a held-out set of reviews from other products. Neither of them are trained on the reviews of the 200 products we evaluate.
Spiral Galaxy Fractions (Willett et al., 2013). The Galaxy Zoo 2 project contains the classification results of galaxy images from the the Sloan Digital Sky Survey (York et al., 2000, SDSS). We are interested in estimating the frac- tion of galaxies that are classified as "spiral," i... have at least one spiral arm. The covariates Xıy in this applications are images (we provide some examples in Figure 5 of the appendix). Existing PPI papers have focused on estimat- ing the overall fraction; we demonstrate how this dataset's metadata structure enables compound estimation of spiral fractions across distinct galaxy subgroups. We first use pre-defined partition of the galaxies into 100 subgroups that
8. Conclusion
This paper introduces PAS, a novel method for compound mean estimation that effectively combines PPT and EB prin- ciples. We motivate the problem through the lens of variance reduction and contextual prior information- then demon- strate how PAS achieves both goals, in theory and in prac- tice. Our paper differs from many other PPI-related works in its focus on estimation, so a natural next step is to de- velop average coverage controlling intervals for the means centered around PAS. To this end, may be fruitful to build the robust empirical Bayes confidence intervals of Arm- strong et al. (2022). Modern scientific inquiries increasingly demand the simultaneous analysis of multiple related prob- lems. The framework developed in this paper represents a promising direction for such settings.
!We include only the essential setup below and defer additional data and model details (e.g., hyper-parametert preprocessing) to Appendix E.

---

Ceddiction-Powered Adaptive Shrinkage Estimation
Acknowledgments
Cochran, W. G. Sampling Techniques. Wiley Series in Probability and Mathematical Statistics. John Wiley & Sons, New York, 3d ed edition, 1977.
We thank Dan Kluger and Qingqi Zhang for thoughtful feed- back on an earlier version of our manuscript. We also thank Claire Donnat, Yichen Ji, Lihua Lei, Aaron Schein, and Tijana Zrnic for helpful discussions. We are grateful to Ben- nett Hunter and the compute cluster at the Data Science Insti- tute for supporting our computational resources. We thank all four reviewers for their constructive feedback. N.I. grate- fully acknowledges support from NSF (DMS 2443410).
Cortinovis, S. and Caron, F. FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powertt Inference. arXiv preprint, arXiv:2502.02363, 2025.
Deng, A., Xu, Y., Kohavi, R., and Walker, T. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the Sixth ACM In- ternational Conference on Web Search and Data Mining, pp. 132, Rome Italy, 2013. ACM.
Impact Statement
By developing method to improve the efficiency and ac- curacy of mean estimation using machine learning predic- tions, this research has the potential to enhance data analysis across various domains where labeled data is limited but predictive models are available.
Deng, J., Dong. W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 248 1188-255,40 2009.
Devlin, J., Chang. M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding.
We acknowledge that advancements in machine learning can have broader societal consequences. However, the ethical considerations directly arising from this methodological contribution are those typically associated with the general advancement of statistical methods and machine learning. We do not foresee specific negative ethical impacts unique to this work that require further detailed discussion.
Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, volume 1, pp. 1171-1866, 2019.
Donoho, D. L. and Johnstone, M. Adapting to unknown smoothness via wavelet shrinkage. Journal of the Ameri- can Statistical Association, 90(432):1200-122 1995,
References
Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, Prediction poweerd inference. Science, 382 (6671):669-674, 2023.
Durrett, R. Probability: Theory and Examples, volume 49 of Cambridge Series in Statistical and Probabilistic Math- ematics. Cambridge University Press, 5th edition, 2019.
Angelopoulos, A. N., Duchi, J. C., and Zrnie, T. PPI++: Efficient Preweedi Inference. arXiv preprint, (xXiv2311.00155. 2024.
Efron, B. The estimation of prediction error: Covariance penalties and cross-validation. Journal of the American Statistical Association, 99(467):619 63299,636222000 2004.
Efron, B. Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction. Institute of Math- ematical Statistics Monographs. Cambridge University Press, Cambridge, 2010
Arnstrong, T. B., Kolesár, M., and Plaghorg- M. Ro- bust empirical Bayes confidence intervals. Econometrica, 90(6):2567-2602 2022.
Baly, R., Da San Martino, G., Glass, J., and Nakov, We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4982-4991. Association for Computational Linguistics, 2020.
Efron, B. Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496): 1602- 1614, 2011.
Efron, B. and Morris, C. Stein's estimation rule and its competitors- an empirical Bayes approach. Journal of the American Statistical Association, 68(341):117-130, 1973.
Breidt, F. J. and Opsomer, J. D. Model-assisted survey estimation with modern prediction techniques. Statistical Science, 32(2). 2017.
Egami, N., Hinck, M., Stewart, B., and Wei, H. Using imperfect surrogates for downstream inference: Design- based supervised learning for social science applications of large language models. In Advances in Neural Informa- tion Processing Systems, volume 36, pp. 98589-686011 2023.
Candès, J., Sing-Long, C. A., and Trzasko, J. D. Unbiased risk estimates for singular value thresholding and spectral estimators. IEEE Transactions on Signal Processing, 61 (19):46434465), 2013.
10

---

Prediction-Powered Adaptive Shrinkage Estimation
Fan, S., Visokay, A., Hoffman, K., Salerno, S., Liu, L., Leek, J. T., and McCormick, T. H. From narratives to numbers: Valid inference using language model pre- dictions from verbal autopsy narratives. arXiv preprint, MMXiv:240.024366 2024.
Ignatiadis, N. and Wager, S. Covariate powweed empiri- cal Bayes estimation. In Wallach, H., Larochelle, H., Beygelzimer, A., D' D'Alcé-Buee F., Fox, E., and Gamett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Fay III, R. E. and Herriot, R. A. Estimates of income for small places: An application of James-Stein procedures to census data. Journal of the American Statistical Asso- ciation, 74(366a):269-27) 277, 1979.
Ignatiadis, N., Saha, S., Sun, D. L., and Muralidharan, O. Empirical Bayes mean estimation with nonparametric errors via order statistic regression on replicated data. Journal of the American Statistical Association, 118(542): 987-999, 2023.
Fisch, A., Maynez, J., Hofer, R., Dhingra, B., Globerson, A., and Cohen, W. Stratified predictiin-powerse infer- ence for effective hybrid evaluation of language models. In Advances in Neural Information Processing Systems, volume 37, 2024.
James, W. and Stein, C. Estimation with quadratic loss. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp. 361-379, 1961.
Ji, W. Lei, L., and Zrni, T. Predictions as surrogates: Revisiting surrogate outcomes in the age of AI. arXiv preprint, Xiv:22200 .09731, 2025.
Fogliato, R., Patil, P., Akpinar, N.-J., and Monfort, M. Pre- cise model benchmarking with only a few observations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9563-9575. Association for Computational Linguistics, 2024.
Jiang, W. and Zhang, C.-H. General maximum likelihood empirical Bayes estimation of normal means. The Annals of Statistics, 37(4):1647-168: 2009.
Ghosh, S., Ignatiadis, N., Koehler, F., and Lee, A. Stein's unbiased risk estimate and Hyvärinen's 's score matching. arXiv preprint, arXiv:2502.20123, 2025.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Green, E. J. and Strawderman, W. E. James- Stein type estimator for combining unbiased and possibly biased es- timators. Journal of the American Statistical Association, 86(416):1001-1006, 1991.
Kou, C. and Yang. in heteroscedastic hierarchical linear models. In Ahmed, S. E. (ed.), Big and Complex Data Analysis, pp. 249-284. Springer International Publishing, Cham, 2017
Optimal shrinkage estimation
Gronsbell, J., Gao. J., Shi, Y., McCaw, Z. R., and Cheng, D. Another look at inference after prediction. arXiv preprint, arXiv:2411.19908 2025.
Lavenberg. S. S. and Welch, P. D. A perspective on the of control variables to increase the efficiency of Monte Carlo simulations. Management Science, 27(3):322 -3333, 1981.
Hart, R. E., Bamford, S. P., Willett, K. W., Masters, K. L., Cardamone, C., Lintott, C. J., Mackay, R. J., Nichol, R. C., Rosslowe, C. K., Simmons, B. D., et al. Galaxy zo0: comparing the demographics of spiral arm number and a new method for correcting redshift bias. Monthly Notices of the Royal Astronomical Society, 461(4)33663 3682, 2016.
Li, K.C. From Stein's Unbiased Risk Estimates to the method of Generalized Cross Validation. The Annals of Statistics, 13(4):1352-131) 1985.
Liang, F., Mukherjee, S., and West, M. The use of unlabeled data in predictive modeling. Statistical Science, 22(2): 189-205, 2007.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Lin, J. Y.-Y,, Liao, S.-M., Huang, H.-J., Kuo, W.-T., and Ou, O. H.-M. Galaxy morphological classification with efficient vision transformer. In Machine Learning and the Physical Sciences Workshop at the 35th Conference on Neural Information Processing Systems, 2021.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Lin, W. Agnostic notes on regression adjustments to ex- perimental data: Reexamining Freedman's 's critique. The Annals of Applied Statistics, 7(1):295-318. 2013.
Hickernell, F. J., Lemieux, C., and Owen, A. B. Control variates for Quasi-Monte Carlo. Statistical Science, 20 (1):1-31, 2005.
Mallows, C. L. Some comments on Cp. Technometrics, 15 (4):661, 1973.

---

Prediction-Poweet Adaptive Shrinkage Estimation
Miao, J., Miao, X., Wu, Y., Zhao, and Lu, Q. Assumption- lean and data-adaptive cos-prediction inference. arXiv preprint, arXiv:2311.14220 2024a.
httges//www..eglllccm///ttttssst /srap/ feazzon-fine-fooo-eeeeeeess 2024-01-01.
Accessed:
Miao, J., Wu, Y., Sun, Z.., Miao, X., Lu, T., Zhao, J., and Lu, Q. Valid inference for machine learning-assee assisted genome-wide association studies. Nature Genetics, pp- 1-9, 2024b.
Soloff, J. A., Guntuboyina, A., and Sen, B. Multivariate, heteroscedastic empirical Bayes via nonparametric maxi- mum likelihood. Journal of the Royal Statistical Society Series B: Statistical Methodology, pp. qkae040, 2024.
Mozer, and Miratrix, More power to you: Using machine learning to augment human coding for more efficient inference in text-based randomized trials. The Annals of Applied Statistics, 19(1):440-465) 464, 2025.
Stein, C. M. Estimation of the mean of multivariate normal distribution. The Annals of Statistics, pp. 1135-1151, 1981.
Tibshirani, R. J. and Rosset, S. Excess optimism: How biased is the apparent error of an estimator tuned by SURE? Journal of the American Statistical Association, 1146526)6677-12,, 2019.
Mukhopadhyay, P. and Maiti, T. Two stage non-parametric approach for small area estimation. Proceedings of ASA Section on Survey Research Methods, hal, pp. 4058-4065, 2004.
Town, N. haaaeemullllinggua-uuccccee URL
2023. wtwppa.........tt..t....o /wtpttw//
Nair, P. B. and Abraham, R. G. On the fraction of barred spiral galaxies. The Astrophysical Journal Letters, 714 (2):L260, 2010.
sentiment
Wang, S., McCormick, T. H., and Leek, J. T. Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences, 117(48)330266- 30275, 2020.
Penedo, G., Kydlfček, H., allal, I.. B., Lozhkov, A., Mitchell, M., Raffel, C., Werra, L. V., and Wolf, T. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2024.
Willett, K. W., Lintott, C. J., Bamford, S. P., Masters, K. L., Simmons, B. D., Casteels, K. R., Edmondson, E. M., Fortson, L. F., Kaviraj, S., Keel, W. C., al. Galaxy Zoo 2: detailed morphological classifications for 304 122 galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 435(4))2835- 2860, 2013.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
Robbins, H. Asymptotically subminimax solutions of com- pound statistical decision problems. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics
Wolf, T. Huggingface's transformers: State-of-the- art natural language processing. arXiv preprint, 2019.
and Probability, volume 2, pp. 131-149. University of California Press, 1951.
Xie, X., Kou, S., and Brown, L. D. SURE estimates for het- croscedastic hierarchical model. Journal of the American Statistical Association, 14655-0));:65114777 2012.
Robbins, H. An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Math- ematical Statisties and Probability, Volume 1: Contri- butions to the Theory of Statistics, pp. 157-163. The Regents of the University of California, 1956.
York, D. G., Adelman, J., Anderson Jr, J. E., Anderson, S. F., Annis, J., Bahcall, N. A., Bakken, J., Barkhouser, R., Bastian, S., Berman, E.,, et al. The sloan digital sky survey: Technical summary. The Astronomical Journal, 120(3):1579, 2000.
Robins, M., Rotnitzky, A., and Zhao, L. P. Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association, (9(447):846886. 1994.
Yu, C. and Hoff, P. D. Adaptive multigroup confidence intervals with constant coverage. Biometrika, 105(2): 319-335, 2018.
Rosenman, T., Basse, G., Owen, B., and Baiocchi, M. Combining observational and experimental datasets using shrinkage estimators. Biometrics, pp. biom.13821. 2023.
Zmic, T. A note on the predietion-oowered bootstrap. arXiv preprint, 1xXiv:2405.18379 2025.
Zmic, T. and Candès, E. cross-prediction-pee infer- ence. Proceedings of the National Academy of Sciences, 121(15))e232208311, 2024.
SNAP. Amazon Fine Food Reviews (Stanford net- work analysis project). Kaggle Dataset, 2014. URL
12

---

Prowirtion-Poweree Adaptive Shrinkage Estimation
Appendix: Table of Contents
A. Further Connections and Intuitions Behind PAS
Page 13
B. The Correlation-Aware Unbiased Risk Estimate
Page 16
C. Details on PAS with Unknown Second Moments
Page 17
D. Baseline Shrinkage Estimators
Page 21
E. Experiment Details
Page 23
F. Proofs of Theoretical Results
Page 27
A. Further Connections and Intuitions Behind PAS
A.1. Further Connections Related Works
In this part, we elaborate on two important connections existing work.
Stratified PPI (StratPPI; Fisch al. (2024)). Fisch et al. (2024) also consider a setting with stratification of observations into subproblems. However, their overall aim is still estimate single parameter, rather than multiple parameters, do. Thus, in solving individual problems, for instance, they still adhere to the requirement of unbiasedness. As an example, let us revisit the Galaxy Zoo 2 application. Therein, we mentioned the following distinction between our work and previous methods in the PPI literature:
Estimate the fraction 9 of spiral galaxies across the whole universe (Angelopoulos et al., 2023).
(Our work) Estimate the fraction 0g of spiral galaxies within the j-th cluster of galaxies for [m]]
Now suppose that galaxies in the whole universe can be partitioned into the m clusters above. Then the goal of StratPPI is:
Estimate the fraction 8 of spiral galaxies across the whole universe by proceeding with an intermediate step that involves estimating . 8m.
We continue to make this connection explicit in mean estimation setting. Suppose (dropping subscripts for convenience) that we start with covariate-outcome pairs (X, Y) X' x y distributed as
(X, (X,Y)=B iid
P.
Moreover suppose that there exist pairwise disjoint strata A AI....; Am whose union is the full covariate space X, that is, X' Aj. Then define,
wy IP[X E As).
0g = P(Y x XEA,)
j E [m].
and observe the key equality for = [Y()]
m Σ m
StratPPI assumes that the probabilities wy are known exactly and then samples n, labeled as well as N, unlabeled observations from the conditional distribution P/| X E A,).15 These samples are then used alongside PPI++ to estimate 0,
Part of the contribution of StratPPI includes strategy for allocating resources and choosing n, and N₂ for different problems with view toward minimizing the variance the final estimator 0.
13

---

Prediction-Powerve Adaptive Shrinkage Estimation
by TFT, almost verbatim the approach described in Section 4.1. Finally, StratPPI estimates via:
@SPPI
w PT
To summarize, the settings of StratPPI and our paper are similar, but the goals and methods differ. StratPPI seeks an unbiased estimate of single parameter 0, while we seek to estimate parameter vector (01..... :.......mm). 9m) as well as possible in mean squared error, while allowing for the possibility of bias. Moreover, the asymptotic regimes these two papers are different: StratPPI keeps m fixed and takes n, .N + oc, while we assume that the number of problems grows (m + x)) while ny. Ny are allowed to remain bounded.
PPI++ (Angelopoulos et al., 2024) with multivariate mean estimand. Angelopoulos al. (2024) (and other papers in the PPI literature) consider statistical problems going beyond the estimation of a univariate mean. For instance, the results of Angelopoulos al. (2024) also accommodate multivariate mean estimands. Here we explain how one frame compound mean estimation setting into the estimation of single but multivariate mean estimand, which then fits in the more generalized formulation for power tuning considered in PPI++.
If we assume nj = n and N = N for all j. we can collate the responses and predictions from all problems into (J(X,), and ()( iE where
Y. = "',)",,,Y.,,,,, [(x,)= = (J(x))
((x.) (..)) [(xmmmm) (T).
By independence across problems, we have the covariance structures Var [ar( diag( (1)))=)))) and (ox = diag(())))"1) where diag(-) constructs a diagonal matrix with its arguments on the diagonal. Angelopoulos et al. (2024) then consider a class of estimators indexed by a single weighting parameter that is applied to the entire vector:
0prr
1 n n
Σ
Y.
Σ f(x,)
Σ (X.))
n
The optimal X is chosen by minimizing the trace of an asymptotic covariance Σ For the mean estimation problem, Σ simplifies to the covariance of gPPl and can be exactly calculated in finite-sample setting (without relying on n, N + co). Minimizing A over thus admits a simple closed-from solution:
71 TT"TT(CCov (.))). Tr( Va(Varr[ ((x.))
n
m j= 75 n. Σド= 7j
A
(19)
N
Going back our original formulation with multiple problems, what we are doing here is to perform power tuning across problems minimize the of MSEs-wil equals the trace of the covariance matrix [[[Vr[[(X, I]) in the PPI++ formulation over single (univariate) A. Finally, if n2 and N, vary across the above PPI++ formulation fails to hold, but the idea to power tune all problems together carries over to our multiple problem setting. The univariate optimal tuning parameter takes the same form as Equation (19):
X ビ my E N,
In fact, this choice of X" leads to a new estimator, denoted as Un UniPT, PT, that will be further explored in Appendix C.
A.2. Further Intuitions Behind the Design of PAS
1. WIYY DOES SHARING INFORMATION ACROSS PROBLEMS HELP?
Here we provide an explanation of the fundamental statistical difference between estimating a single mean versus estimating a lot means (m + 5o) with an eye toward providing intuition of how we can share information across problems. We emphasize that this is only meant for intuition; the empirical Bayes literature, starting with James & Stein (1961), provides a more nuanced picture
14

---

Prediction-Poweree Adaptive Shrinkage Estimation
Imagine a simplified setting with only single problem (m 1, in the standard PPI setting). We further assume that N3- oo, that 7 =/=E, Bn [f())) and we also assume that of is known. Moreover, suppose want to choose between the PT estimator 0f and the prediction mean 2f by comparing their MSEs data-driven way. We know that the PT estimator is unbiased and its MSE is oi, but we cannot estimate [[ _ accurately since we only have single 81 (a single problem). At best, can use (0"T ) oî By [(0PT - ) of = E, [(0 )...
unbiased estimate of this quantity since
Now suppose we have multiple problems, then we can more precisely learn how good the prediction mean 2 is for estimating 0j by repeating the above estimation procedure for each problem and averaging the results. This average turns out be not only unbiased but also consistent. To wit, as m + oc, by the law of large numbers,
m
(0T- - 21)2 67) +. RE,,,,10 - -2)7,,
(20)
m7
where we emphasize that the mean squared on the right-hand side also integrates with respect to the meta-distribution P,, that determines the distribution of the 0g (cf. Assumption 2.1). This illustrates a mechanism for sharing information across problems: in first step, we can consistently estimate the average squared difference between the true parameters f, and the prediction means 2/ (i.e., how good the ML predictor is on average) by aggregating information from many parallel problems. Then, based on the first step, we can choose whether to use z or 6"" according to their MSE.
Our actual implementation is more involved but embodies the same fundamental principle: we construct a consistent estimate for the MSE a family of shrinkage estimators. While other factors such as heterogeneity in sample sizes (ri,, N2) introduce further complexities, the core idea of sharing information here carries over to the overall design and justification of PAS.
A.2.2. ANALOGY BETWEEN 2/ AND B[0 +]
In Equation (16), we informally draw an analogy between the prediction mcan Z in the compound PPI problem and the posterior mean EE {b] in the stylized Gaussian model. Our goal here is to provide heuristic motivation for the one-dimensional parameterized family of weights wj in Equation (15). We emphasize that the ultimate success of this parameterization choice is judged by the empirical results.
Our heuristic motivation building on the stylized example of Section 3 is as follows. We seek estimate 0 by taking a convex combination gr - and shrinkage target
w(Orl {) + (1 w)s,
for w E [0, 1]. Our main point is that for several choices of shrinkage target 8, the best weight w can be written the form
3
(21)
for some w 2 0616
The Bayes estimator with contextual prior in Equation (12) uses 8 = E[0 Φ) and weight w as in (21) with
w El0 d E [ []] o
* The Bayes estimator without contextual prior uses s = [[O] = 0 and weight as in (21) with
= E (0 - E (0) = Var(0) = og.
A6 this stage, there is 1-1 mapping between 2 0 and [0, 1). However, for PAS, of corresponds to the variance of the j-th power tuning estimator and will be different from problem problem. Then it will he convenient to parameterize all the problems by the same parameter w 2
15

---

Prewiction-Poweree Adaptive Shrinkage Estimation
Suppose now that we ask for the best convex combination (not necessarily Bayes predictor) between O & and h(()) where h(()) is some fixed function. Then, the weight minimizing MSE can be shown take the form in (21) with
w = = E =
Φi] + [((h()) - E +l)
The above expression is interesting as it forces us to inflate "w", i..., shrink less toward h()) in way that depends how close h()) is to H/0 ø). See Ignatiadis & Wager (2019, last paragraph of Section 3) and Appendix A.2.3 below for further discussion of this point.
The takeaway is that for lots of possible predictors, the optimal weights have the same parameterized form up to the single parameter w that varies according to the quality of the predictor. This motivates our one-dimensional family of weights. Once this family has been motivated, we learn w in Equation (15) in way that does not depend on the above analogy all by minimizing CURE.
A.2.3. ELABORATION ON PROPOSITION 5.3
Suppose (as in Proposition 5.3) that nj is the same across all problems, N, = co, and that second moments Pj. Tj. are identical across all problems. Then we could ask: what is the best convex combination between f77 and Z3 in the following sense:
w/ E arg min Be, [10, - (W, + (1-w))2
w,20
By direct calculation (note that the right-hand side is a convex quadratic in wa) we find that:
Eg, Eg, - -
Ep, [0,, - -2 52
This implies the following intuitive result: the larger the MSE (E,, 2 )²) the less weight should assign 2/. If evaluate the MSE this optimal wis, recover precisely the upper bound of Proposition 5.3.
B. The Correlation-Aware Unbiased Risk Estimate
Theorem B.1. Let X, Y be two random variables satisfying [x[X]=
=y, = 0, Varo| = o², CovolX, x|= 7, and the second
moment of Y exists.". 17 Consider estimating with the shrinkage estimator O. = cX + (1 (1- (1- with [0, 1]. Assuming
that o² and 7 are known, the following estimator
CURE(Ô.) = =(2c (2c-1)o² 2(1 (10--1==-01--07++00((00---------
(22)
defined as the Correlation-aware Unbiased Risk Estimate, is an unbiased estimator for the risk of 0. under quadratic loss. That is, letting R(0,,0 0) == -Eo 2) it holds that:
Es CURF(()] R(θ...) 0).
Proof. First, expand the risk:
R(Oc, 0) = Eol(d. - + + co 0)21 -0
VarolcX (1 c)
(1- c)Y] 0)
=200 + (1 -1-c0 c) Y + 2c(1 - c)> + (Eo 0))
WeWe redefine certain variables for generality of this result beyond the setting in Assumption 2.2. In this theorem, & plays the role of y the main text, i.c. all the other parameters are deterministic given
16

---

Prediction-Poweree Adaptive Shrinkage Estimation
Then, taking the expectation of CURE((,):
Ea (
= -2cc- (2c-1)o² + -(001-1000000000000000000000-011
(23)
where the last term is
Be (x
= c)² [[F (X y1) Varo X V]]
= (1 - c)² [ - 0)² + + Varo [ - 27]
[(1 (Bo - 0))² + +(1
(1- - c0) + Varo Y -27)
With a little algebra, we observe
I+ (2c 1)0² 2 + (1-c)) + Varg|Y] -yy)
=c0 + (1 -1- (1-c) Var + c)y.
Thus, a term-by-term matching confirms E, [CURB((..] - O).
Remark B.2 (Connection to SURE). Stein's 's Unbiased Risk Estimate (SURE) was proposed in Charles Stein's seminal work (1981) to study the quadratic risk in Gaussian sequence models. As simple special case of SURE, let Z ~ N o²) and let h : R + R be an absolutely continuous function and EE (2)] < oo, then SURE is defined as
SURE(h) = (h(2) - 2) + 20 2z) -
o²,
with the property that [SURE(()) R(h(2),)) [h( - 0)²]. A proof of this argument relies on Stein's lemma, an identity specific to Gaussian random variables (Stein, 1981). Now consider the specific linear shrinkage estimator ho(Z)= := cZ + (1 Y)Y, with E [0, 1] and Y E H being fixed (that is, Y is constant, or Y is independent of Z and we condition on Y). Then SURE takes the following form:
SURE(h = =hhoZZ) - 2)² + +202 - o2
= (cZ + (1 c)y - 2) + 2002 0² [(1-c)(Y 2)}² + (2c 1)0²
CURE(h. (Z)),
where in (+) we used the fact that in this case (with Y fixed Y independent of Z), holds that y = 0 so that the definition of CURE in (22) simplifies. This explains how CURE defined in B. is connected SURE.
We make one last remark: The derivation of SURE itself requires Gaussianity. However, for linear shrinkage rules as ho SURE only depends on the first two moments of the distribution of Z and thus is an unbiased estimator of quadratic risk under substantial generality as long as Eg [ = 0 and Varæ [Z] = o². This remark has been made by previous authors, e.g., Kou & Yang (2017); Ignatiadis & Wager (2019) and is important for the assumption- validity of PAS.
C. Details on PAS with Unknown Second Moments
In this appendix we explain how apply PAS when second moments are unknown. In Appendix C.I we describe sample- based estimators of the second moments. We also develop Un iPT (Appendix C.2) and Un iPA (Appendix C.3), two new estimators that extend our framework to scenarios where second moments are unknown and must be estimated from data. We derive these methods and their theoretical guarantees within the PAS asymptotic regime, where the number of problems m - oo while individual sample sizes ny. NY remain bounded. Specifically, UniPT introduces global power-tuning strategy, and UniPAS builds upon it to perform adaptive shrinkage, both without requiring knowledge of true second-moment parameters.
Notations. Throughout this section and the proof details in Appendix F, we use P, for convergence in probability and 47, for IP convergence of random variables. We use the "Little-o" notation o(1) for any term that vanishes to zero as m + oc. Similarly, for a sequence of random variables Xm, we use X,, = 0p(1) 1) if Xm, converges to zero in probability, and X, = (r(1) if Xm bounded in probability. All stochastic order relations are understood hold as m -> co.
17

---

Prediction-Powereed Adaptive Shrinkage Estimation
C.I. Sample-based Estimators for oj. 7j.75:7
We first write down the expressions of the unbiased sample-base based estimators for of. r2 and 7/. assuming that ny 2
3g
03 =
(Y - -Y,),
== + N2
Σ
(f(Xit) - 20 22 Σ (s(Xis) - -2 2
i=1
1ly Σ (Y. ())((())))) - k,, where 7g NN ny + N₂
N Σ ((x)) Σ)(X))
These sample-bases estimators serve dual role. Firstly, they are utilized in the practical implementation of the PT and PAS estimators for our numerical experiments on real-world datasets, where true second moments are unavailable. Secondly, they form the basis for defining the UniPT and UniPAS estimators, which we introduce subsequently.
C.2. UniPT: Power-tuning Across Problems with Estimated Second Moments
Under this new setup, the first step is to derive a new variant of the PT estimator without the known second moments. turss out that performing power-tuning across problems using the same for all problems (which we show in Appendix A. to be closely related to the multivariate estimation problem in PPI++) leads to a promising alternative.
Definition C.1 (Univariate Power Tuning (UniPT)). We consider a family estimators for each problem j E [m]]
0 Y + AZz 2 -
where all problems are controlled by a single, global lover-tuning parameter À E R. Our goal is to find the À that minimizes the sum of variances across all problems, Σy=1 Var,, 10,x,100.x). The variance for each problem j is:
of + X2 R2
1 n
2A nij
Var,, 10
7f
N2
where of Var,, (Yy], +3
Var,, [X(x;)) and 7/j Cov,, nz (Y f(xa))) Minimizing Σラ= Var,, with respect yields the theoretically optimal global parameter for the given set of m problems:
Σ -
E 1
2j=1
m
TJ
Σ
B n,N, 7
In practice, since 7 and T3 are assumed unknown, replace them with their sample-based unbiased estimators î and 23 (computed in Appendix C.I) to obtain the estimated global power-tuning parameter:
Å プラ= n A3
n2 Mn-No N } Σラ
We clip Å to the interval [0, 1]. Let this be Setip c [0, 1)). Similarly, we denote the clipped version of X as Nalin. = e [0, 1)). The UniPT estimator for problem j is then:
OUPT= = Setip 2 2/).
The UniPT I estimator, by using a single data-driven Å, offers a practical way to perform power tuning when per-problem moments are unknown. This approach is justified by the following theoretical results, which hold as m - co. Proposition C.2 (Asymptotic Consistency of Clipped Global Tuning Parameter). Assume that:
1. Sample sizes nj, Nj bounded (2 S A'min I nj Tmax <0,, 1 S. Nmin N₂ S Nmax 00).
2. Er, (Var,, [55]] oo and Er, [Var,, [+]]] co.
18

---

Prewirtion-Powered Adaptive Shrinkage Estimation
3. Ep,115] cc, Ep, Bp,((17)) < xc, Er, (e,,((3))) co.
4. (Denominator bounded away from 0) there exists some & > such that
R2 + Nj Jj
lim
P
> = =1
Then, the clipped estimated global tuning parameter delip converges in L to the clipped theoretical optimal global tuning parameter me
Xelip
Vaip.
La 20
as m + oc.
The proof is deferred to Appendix F.5.
This consistency ensures that dclip effectively targets the best single power-tuning parameter for the collection of m problems. Building on this, we can state a result regarding the asymptotic variance of the UniPT estimator.
Theorem C.3 (Asymptotic Variance Optimality of UniPT), Under the assumptions of Proposition C.2, the sum of variances of the UniPT estimators, د Var,, [ asymptotically achieves the minimum possible sum of variances within the class of estimators C ((0)A) 0AA- - MZz -z)),, A [0, 1]}, the sense that:
1 m
m
Var,, 10mm [0(PT]]
min [E[U,]] m
Σ
Var [0 P
as m oc,
m
j=1
where the left-hand side is still random variable with randomness from drawing πy idd F.. The proof is deferred to Appendix F.6.
C.3. UniPAS: CURE and Adaptive Shrinkage with Estimated Second Moments
Once we obtain the UniPT estimator, which is asymptotically optimal within class of unbiased estimators, our next goal is to imitate the steps in Section 4.2 to apply shrinkage across problems. To do so, we must first revisit the formulation of CURE and see how it depends on the now unknown second-moment parameters. By definition:
CURE
PAS
1
(2 W
o3 + 2(1
W
+ (1
W
OT
z/)²
Wi
w
(24)
So there are three places where CURE makes use 33. 7g. which then depend on second moments of the data generating process: (1) in the definition of PT estimator, (2) in the definition of CURE itself and (3) in determining the localized shrinkage level wy. We thus make the following modifications to CURE using the sample-based estimators.
1. We first replace the PT estimator (shrinkage source) to the UniPT estimator. Additionally, 33 is now the variance of 0 8.p2.p.. (as defined in Definition C.I) and ř, its covariance with Z/. We can explicitly write them down
of N₂ + n5 n2
(Aaip..)) ) nj
Ai 5j 5= Velip.m N,
Nyny
2. For d3 and Ny in the definition of CURE, replace them directly with the sample-base based estimators
07
82
N₁ +
12 +2 Aelpp nj
Aen
#2 t3 = ddtip
n2
Nyng
N,
where 03.43, f3 are defined in Appendix C.I.
Note that this condition implied by the finite fourth-moment assumption in Proposition 5.1.
19

---

Prewiction-Powered Adaptive Shrinkage Estimation
3. For 0j in the definition of wy
we replace with the following averaging estimators:
09
σ² Nj +
+
ny Seii
72
2 Aclip Y:
where
(25)
nj
Nyng
n2
m
²²
72
51
7
=
m
711
m
j=1
j=1
1=1
are the average sample co-(variances) across all m problems. 19
With these modifications, we define a new class of shrinkage estimators based on 0 For any global shrinkage parameter z 0, the Sabbei--specifi shrinkage weight is now defined as:
3g = (وا(w)
w + oj
The corresponding family of shrinkage estimators for problem! j is:
QUPAS j:u = 0 + - ررفة 2.)
We then define the modified CURE, denoted CURE, by taking into account all the changes above.
CURE
Ow
UPAS
(20r 1)03 + 2(1 2y)lo + (1 -0) 2,)a.
(26)
Finally, the UniPAS estimator is obtained by selecting the w that minimizes this CURE: Definition C.4 (Univariate Ceddiction-Poeeete Adaptive Shrinkage (UniPAS)). The UniPAS estimator for problem j is
gyPAS @UPAS where
j/
arg min CURE (0_
This UniPAS estimator is fully data-driven and does not rely on knowledge of the true second-moment parameters. The pseudo-code for the full UniPAS algorithm is given below.
The fully data-driven construction of UniPAS is supported by the following theoretical guarantee:
Proposition C.5 (Asymptotic Consistency of CURE for UniPAS). On top of the assumptions in Proposition C.2 and As- sumption 2.2, if we further require that inf Jem, 62 2 8 > 0 for some fixed 6, where
0
22 j.m
Moa
+ N₂+ nj (Netip.m) )2
X ml n,
2
(27)
n2
Nyng
UPAS is asymptoticully consistent estimator
with Poa = Bp,, 1o3), 1,p = Es, 1T}1, ; My = RS, byj. Then, CURE
for the true loss e. 1
(
UTAS
.0) of the UniPAS estimator (which uses weights dag w/(w + +z))) in the sense that:
Ep, sup CURE UPAS) Lm (0
-PAA
00]
m m=+000 0.
w20
We defer the proof Appendix F.7.
This consistency result ensures that minimizing CURE is asymptotically equivalent to minimizing the true MSE of the UniPAS estimator. This leads to the following optimality guarantee:
The rationale is as follows: In defining ر we pretend that of. 7J and 7/y are the same across all problems and so can he estimated consistently. We emphasize that our thcoretical results do not require that these second moments be identical (i.c., it is only working modcling assumption).
20We use the hat notation for CURE make it explicit that CURE is unbiased estimator of risk finite samples. However, we will show a consistent estimator asymptotically as m + oo.
20

---

Prediction-Powered Adaptive Shrinkage Estimation
Algorithm 2 UniPAS
Require: (Xij, Yy)== (Xi)) for j E [m]] predictive model f
for to m do
2: Step Apply predictor (Eq. (1)) to get aggregated statistics and sample-based estimators for second moments
3:
Yj. 2 2
get.means ((Xy, M1 M1)) (Xx))=1::))
4:
oj, fj, 22
get. wtt.ssamle..arrinnee
((x), Yi)) (X.))
5:
b Step Univariate power tuning (Appendix C.2)
6:
delpp clip
E
N 4, ?
nj 4
[0.1]
7:
0Ur =Y, + Aecipp
b Step 3: Construct sample- based estimators of Var(d) and Cov (0)PT, 2
8:
9:
j3
o2 + Nyn, 7] ny
2 Achpp
+j N;
10: 9)
11: end for
12: D Step 4: Adaptive shrinkage
13: d²
1 m - 07. 72 - 7 1
プド= 94
for = m do
Calculate the averaging variance estimator in Equation (25)
16:
a3 =
Notri Poip 2 7clpp?
N;n;
"5
17: end for
18: & Now for each choice of w 2 O, the shrinkage coefficient is determined by w+#
19: The CURE calculation in Equation (26), which still depends the sample-based estimators o3. 0g
20: @ = -eeeess. bet_shrink.o param ( プ= (2 M (0)) (z)) (5))
21: for j 1 m do
22: @UPAS UUPT + (1 2
end for
24: return (OUPAS 1-
Theorem C.6 (Asymptotic Bayes Risk Optimality of UniPAS). Under the assumptions of Theorem C.5, the UniPAS
estimator -PAA
UPAS
0, satisfies:
0 UPAS)
1h (O o() 0C,
B
I inf
and so
0 UPAS
P. (2²) (27) BP.. (0UPT) +0(1) o(1) as co.
B
S min
The proof of Theorem C.6, which follows directly from Proposition C.5, mirrors the argument in in Appendix F.3 used to derive Theorem 5.2 from Proposition 5.1.
We note that the conclusion here is slightly weaker than that of Theorem 5.2 for PAS. Theorem 5.2 guarantees that PAS asymptotically has risk less or equal to that of PT with optimal per-problem choice of the power tuning parameter. By contrast, Theorem C.6 along with Theorem C.3 shows that UniPAS always has risk less equal to that of power tuning that uses the same power tuning parameter for all problems. The main upshot of Theorem C.6 is that UniPAS AS does not require knowledge of second moments.
D. Other Baseline Shrinkage Estimators
D.I. "Shrink-classical" (Shrinkage) Baseline
The "shhink-classicee estimator applics shrinkage directly to the classical estimator Y,. using the prediction mean 2 as a shrinkage target without first applying power-tuned PPI. We include this baseline to isolate the benefits of power tuning
21

---

Prediction-Powereed Adaptive Shrinkage Estimation
from the PAS estimator as ablation study.
Formulation. The "shrink-classical" estimator for problem j takes the form:
@Shrink == w, + (1 wj) jiw
where
W/ w/ +3)), 8] = Varn, [Yg] 03 /nj.
Here 2 0 a global shrinkage parameter analogous Section 4.2. The key difference from PAS is that we shrink the classical estimator Y3 (which is independent of 2/) rather than the power-tuned estimator 05T (which is correlated with Zf).
Optimizing w via CURE. Since Yy and 75 are independent, Theorem 4.1 simplifies. Let 93 Cov [V, 271 0 and o3 = of) /nj. CURE simplifies as:
CURE(6) = (2w, 1)03 + [11 ((1-w))(F) - -2,)²
This follows from Theorem 4.1 by setting Fg = 0. The global shrinkage parameter w is selected by minimizing CURE across all m problems.
@Shrink Wj 25)
3j = w/(6 89),
1
where
W E arg min
CURE (Strinks
(28)
w20
The optimal does not admit closed-form expression, but we can compute numerically by grid search. Below we provide the pseudo-code for implementing the "shrink-classical" estimator.
Algorithm 3 "Shrink-classical" Estimator
Require: {(x,, 1, ) for j [m], variance parameters {o)) 1 predictive model f
1: for j = get..meass(XX)) ( 2 (X 22 M..
do
Y, 2/ get.meemm
3: o3 t oj/n, > variance y)
4: end for
5: s = get strrnk param (fg)ā= (2/) (Gf),=) D use Eq. (28)
6: for j 1 to do
7: Wj a/ + oj)
8: @Strink U, (1 2z) 2/
9: end
10: return (0ptrin))3. 1
D.2. "Shrink-average" Baseline
The "shrink- avveraee estimator represents an alternative, perhaps more classical, shrinkage approach that attempts to further improve upon the unbiased PT estimators. While PAS reuses the prediction means on unlabeled data as shrinkage targets, here we consider shrinking the PT estimators across all problems a shared location, namely their group mean
@PT
Formulation.
The "shrink-average" estimator for problem j takes the form:
jAvz = w + (1 - w/ 0 3,00
where W/ w/:= w/w ,3) 07 = 10rr
22

---

Prewiction-Powereed Adaptive Shrinkage Estimation
Algorithm 4 "Shrink-average" Estimator
Require: (Xiy, Y1z). (X T5: o, for j E [m]] predictive model
1: for j = 1 to mı do
2: p Step I: Apply predictor (Eq. (1))
3: 23 2 gettmeenss (XXi, Y1y ( (Xi))
4:
b Stepp 2: Power tuning (Eq. (13))
5:
Aj -get pt param TT, Ty: myy Nz)
6: 0 =Y, Y, + +2) 2
7: 0j get pt var(O)")) (Eq. (14))
8: end for
9: 0TT
و 05T
10: Step 3: Adaptive shrinkage toward group mean (Eq. (29))
11: get shrink param( (0)")) 0 (03))=1)
for to m
13: W 0/(6 + ).)
14: @Ave 6,0 + (1 - @z)
15: end for
16: return 3T
Optimizing via SURE. Xie et al. (2012) proposed the following unbiased risk estimate to optimize for this estimator. Note that even though the group mean is also correlated with each PT estimator, we still denote the following SURE instead of CURE following the nomenclature in Xie et al. (2012).
w min Σ SURE
(Shrink) 3.
(29)
w20 m
SURE @hrink
(1 (O BPT)]² +1 ( (12m-1166) 1)03).
We refer to Algorithm 4 for full pseudo-code implementation of "shrink-average" estimator.
E. Experiment Details
E.1. Synthetic Model
Motivation. In Example 2.3, we described the following data generation process (copied from Eq. (8))
nj U1-1,11, j i= Xij ~ N(n,,, YuIX.,, ~ X22n c), i 1,...,
and the same for (Xej. Y. ub and two hyperparameters that we chose to be 0.1 and 0.05, respectively. The (marginal) mean and variance of Yij are
8, E,, Yıyl = v, of Var,, [Yy] = 4n} 2p + c.
To understand the motivation behind this setup, we can further inspect the covariance between Xij and Yı,, which can be verified to be Cov,, [X,y YıJ = 2? Therefore, if we consider the ratio between the absolute covariance and the variance (of Yiz) as characterization of the "inherent predictability"
problem, we see that
|Cov,, Xxjj. Yúll Var [Yg)]
2 n 2
4v3 2 +
which has its minimum when n, = 0 and increases monotonically in |yyl for |yyl E [0, given our specific choices of tb and e (see Figure 4). In other words, problems with 77 close to the origin have lower "predictability." whereas when 7] moves away from zero, the problems become easier to solve. This quantitatively reflects the pattern we see in Figure 3, where we display the power-tuning parameters function of n
23

---

Prediction-Powered Adaptive Shrinkage Estimation
Ratio between nC Xyg Yy]l and Var [Yj]
0.15
Ratio
0.10
0.05
4nt
0.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
0
Figure The ratio between [Cov,; [Xay Yyl and Vary, [ss] as function fl. The constants are set to and 0.05.
Expressions for oj Hij o. 73 Ji when (xx |ar|. When work with the synthetic model using the flawed predictor f(x) = |z], we can match the form of our dataset with the general setting in Assumption 2.2 by identifying closed-form expressions for the model parameters fg. Mg: 7]
8g n
of
4nj 2h² + +,
7/ 2n, 2
2 12/(222)
14 V 1 exp
f2j
$
7
2//²
2
33 = n + 2b²
20/2 [
exp
n
+ T7
2Φ
14
- -1)
2x/²
where ((-) denotes the standard normal distribution function.
Expressions for Mg 0.. 73 To when f(x) 2 Similar closed-form expressions can be derived when we use the other predictor ((x) 2². Note that 0g and of remain the same.
8 nī. of = 4 +
4nj v² μ4 カ
-0 + 44,
In experiments involving the synthetic model with both predictors, we are able leverage these closed-form expressions and supplement the ground-truth parameters our datasets.
Interprefation of MSE. In the synthetic experiments, since have access to the true prior for ny (therefore for 0,) and resample them for each problem across K trials, the MSE we obtained in Section 7.1 is unbiased estimate of the Bayes Risk defined in Eq. (11).
E.2. Amazon Review Ratings Dataset
Dataset & Preprocessing. The Amazon Fine Food Reviews dataset, provided by the Stanford Network Analysis Project (SNAP; SNAP (2014)) on Kaggle,2 21 comes in a clean format. We group reviews by their Product ID. For each review, we concatenate the title and body text to form the covariate, while the response is the reviewer's ''s score/rating (1 to stars). Here's sample review:
Score: 4
Product: BBQ Pop Chips
Title: Delicious!
Text: BBQ Pop Chips delicious tasting healthier chip than many the market, They are light and full of flavor. The oz bags are great size have. would recommend them anyone.
24

---

Poweeed Adaptive Shrinkage Estimation
We focus on the top m = 200 products with the most reviews for the compound mean estimation of average ratings. This approach mitigates extreme heteroscedasticity across estimators for different problems, which could unduly favor shrinkage-bast methods when considering unweighted compound risk. There are a total of 74,913 reviews for all 200 products.
Fine-tuning BERT. The Bidirectional Encoder Representations from Transformers (BERT) model is a widely adopted language model for many NLP tasks including text classification (Devlin et al., 2019). However, pre- training BERT from scratch is time-consuming and requires large amounts of data. We therefore use the -uncasedd-dddee-ee sentrd--eetmmttt model from Town (2023) as the base model, denoted as BERT-base. BERT-base is pre-trained general product reviews (not exclusive Amazon) in six languages. achieves 67.5% prediction accuracy on validation set of 100 products (~466k reviews).
Then, we further fine-tune it on the held-out review data, that is, reviews outside the top 200 products, for 2 full epochs. The fine-tuning is done using Hugging Face's rans f formers library (Wolf, 2019). After fine-tuning, the BERT-tuned model achieves 78.8% accuracy on the same validation set.
E.3. Spiral Galaxy Fractions (Galaxy Zoo 2)
Dataset & Preprocessing. The Galaxy Zoo (G7.2) project contains large collection of human-annotated classification results for galaxy images from SDSS. However, instead having a single dataframe, GZ2 has many different eacch for subsets of the SDSS raw data. We begin with a particular subset of 239,696 images with metadata drawn from Hart et al. (2016). Our data cleaning pipeline is inspired by Lin al. (2021), which removes missing data and relabels the class name of each galaxy image to
readable format:
Class Names: Round Elliptical, In-hetween Elliptical, Cigar-shaped Elliptical, Edge-on Spiral, Barred Spiral, Unbarred Spiral, Irregular, Merger
In the downstream estimation problems, we consider a galaxy "spiral" if is classified as one the three classes ending with "Spiral"" otherwise "non-spiral". Below we display a few examples of galaxy images. Each image has dimensions of 424 x 424 3, where the third dimension represents the three filter channels: g (green), r (red), and (infrared). The cleaned dataset has 155,951 images in total.
Sample Images from Galaxy Zoo 2
Spiral
Non-spiral
Figure 5. Example of spiral & non-spiral galaxy images from Galaxy Zoo 2. hun.ssssnuttiiiignnnnnnseeeeeee -eettmeeedssetteettttt
hhtps://daaa, ttttps//dattaggllxxyooooogg
25

---

Prediction-Poweree Adaptive Shrinkage Estimation
The additional SDSS metadata for GZ2 contains valuable information that directly partitions the galaxies based certain attributes, e.g.: REDSHIFT STMPLE BIN based on galaxy redshift measurements, and WVT BIN calculated by weighted Voronoi tessellation. These partitions naturally motivate fine-grained compound mean estimation on this dataset.
After partitioning the images based on WVT 25 we consider only the top m = 100 partitions based on the cutoff that each problem should have 2 150 images (many partitions have very few galaxy images in them), for the same reason as in the Amazon Review dataset. Finally, we have a total of ~100 images as covariates (either Xi, or Xxj) for our problem.
Training the Predictor. We employ the ResNet50 architecture (He et al., 2016), utilizing the pre-trained model from torchvision initially trained on ImageNet (Deng al., 2009). To tailor the model to our task, we fine-tune on ~50k images excluded from the top m problems. The model is trained to classify galaxies into eight categories, later condensed into a binary spiral/nonnspiral classification for prediction. We use batch size of 256 and Adam optimizer (Kingma & Ba, 2015) with a learning rate of le-3. After 20 epochs, the model achieves 87% training accuracy and 83% test accuracy. Despite these promising results, Table 3 indicates that the predictions still require debiasing for accurate estimation.
E.4. Benchmarking in real-world datasets
In this appendix we describe the steps to obtain the MSEs and their standard errors for real-world datasets shown in Table 3.
Let K be the number of experiment trials, T, be the total number of data points for problem j, i.c {Xiy, Y, 122 represents the "raw data" we have, and n,, N, be the desired number of labeled/unlabeled data to simulate, usually calculated through a hyper-parameter splitting ratio (e.g. N₂ r T. n2 = T N, for = 0.8 in our case).
1. Following evaluation methodology in existing PPI literature, e.g., (Angelopoulos et al., 2023), we first calculate the mean of all responses for each problem and treat it as the pseudo ground-truth, i.e., 0z == т, Σ, Yig.
2. For each trial k E [K]. we create a random permutation for the raw data, with indices permuted by K : N - N, and obtain the labeled and unlabeled datasets for problem j as
(xyy 12 = (Xx( Y
{8 No = - (X
3. We proceed with using these datasets to obtain the baseline and PAS estimators. Let 0z be an estimator for the j-th problem at trial k, then our final reported MSE and standard error is calculated as
K
m
MSEx (0)
00
0,)
)
SEK (0)
(
02)
-MSE (0)
K
m
VK
K
m
k=1
Note that the standard error only accounts for uncertainty due to the random splits into labeled and unlabeled datasets.
E.5. Additional Experiment with Varying Labeled/Total Data Ratio
In Table 3, we report our experiment results on different tasks and predictors, but fixing the ratio between labeled and total amounts of data nn/(n, + N2 0.2 for all problems. To verify the broader applicability of our method, we repeat our experiments across much wider range of ratios-from 1% 40%-and report the results in Figure 6. For each ratio, we follow exactly the same data-splitinng and benchmarking procedures specified in Appendix E.4.
E.6. Computational Resources
All the experiments were conducted on a compute cluster with Intel Xeon Silver 4514Y (16 cores) CPU, Nvidia A100 (80GB) GPU, and 64GB of memory. Fine-tuning the BERT-tuned model took 2 hours, and training the 50
The column names and their meanings are available https: / dat /t////taaaaaaaaa/ aat.aaaaaayoo.oorrro.oooooo /d///aaaaaaa http://wwwy.coooo.././/////////a
In the Galaxy Zoo 2 dataset, WVT.BIM denotes Voronoi bins constructed based on each galaxy's intrinsic size and absolute magnitude. The motivation and implementation of this binning strategy are detailed in Hart et al. (2016), who justify such caatiooinn-aliineee with our compound mean estimation setup-by noting that spiral morphology exhibits systematic dependencies on stellar mass and related intrinsic properties.
26

---

PowweetionPPowered Adaptive Shrinkage Estimation
MSE Amazon Reviow
MSE vs Split Fadio on Amazon Revlew Ituned
5,540
Prodic
riaikia Frodition Arg
0.000
6.34
Un FAS
0.33
0
6:32
6.22
0.005
Vain-iss SCER Rato (5)
Tanilct Rudio
MSE Spilt Ratio Gallayy Zon
2.22
Proditie
Snrittge
3.000
2.804
3.000
Figure 6. Average MSFs for the three real-world datasets when the laheled/umlabeled split ratio varies from 1% to 40%. Uni PT and UniPAS the two newly added variants of PT and PAS estimators, respectively.
model took 1 hour. All the inferences (predictions) can be done within 10 minutes. The nature of our research problem requires running the prediction only once per dataset, making it fast to benchmark all estimators for K 200 trials using existing predictions.
E.7. Code Availability
The code for reproducing the experiments is available at http:://gtthubb.mm/sssat220000 predi ion powered-adapticn saaprredacaatttee--siii...... shrinkage.
F. Proofs of Theoretical Results
F.I. Proof of Theorem. 4.1
For each problem j E [m]. we are shrinking the PT estimator 6 obtained from the first stage toward 2J. the prediction mean on the unlabeled data. Conditioning on Mī5, we denote
/3 = Var,,,
[oPT]] = Var
OPPI 1,2g
@..T _/] = Var, n [2/]
93 = Cov ny
where all the first and second moments of 0.T and 2/ exist under the conditions of Assumption 2.2. For each global w 2 0. the shrinkage parameter for the j-th problem defined as wi == w/ (w + o3). Then, following the result in Theorem B. CURE for (PAS- w,0 w + (1 - w
j:m;
CURE (PAA) =(ww - 1)03 + 22(11 wi) + 11) -z71²
27

---

Powerrd Adaptive Shrinkage Estimation
an unbiased estimator of the risk, i.e.,
Eng CURE
(0PA))
R(OPAS 0z)- jicy.
Finally, the CURE for the collection of estimators is P
m,a )
=
1,00
CURE
00
PAS
1 m
CURE
G J:
m
which is an unbiased estimator of the compound risk Rm (
PAS
1) by linearity of the expectation.
F.2. Formal conditions and proof of Proposition 5.1
We aim to prove that CURE converges uniformly to the true squared-error loss e. (O PAS goal is to establish
0) as m - oc. Specifically, our
sup CURE( Lm (0 .0) O. w20
For this proposition, all the expectation and variance terms without subscript are conditioning on 17. We keep using the
notations 8, IE [0 M, E, 03 Var [o and fy Cov (ov|0 Z... For this proposition, additional assumptions placed on the data generating process (integrated over P,,,) We first show how they translate moment conditions the estimators 0. and 2
Lemma F.1. Under the assumptions of Proposition 5.1, and specifically Ep, [f(X1)) < co and Ep, [Y < oo, it holds that:
sup Ee, 121
(0) < oc,
sup Er,
(2/)g] < <0,
121
Er, [09] < <0,
Es, [nt]] oc.
Proof. By Minkowski's inequality, have
Er, [OPT)(] Er, (V, + 13(2 2/1)*
Ep, [2/]1//4 ,ÿEr, [( AzEE, {x, (1))))(())"""
so it suffices to bound the fourth moments of Y,, 2z Zz. We proceed with Y, first. Again, by Minkowski's inequality
47 1/4
121 Enj Y
EF [
1/4 Er,
Er, (n=²y²) 171/1
1/1
n2
EEp [x1]] 1/4
m
i=1
[Ya]] 1/4
Ep,
oo.
The given assumption on the data generating process is used in the last line. Although the left-hand side of the inequality may depend on j (via the deterministic n,)), the right-hand side does not depend on (since we are integrating over P,)) and we may take supremum over all j on the left-hand side. The arguments for Z, and Z, based on the finiteness of
28

---

Prewiction-Powered Adaptive Shrinkage Estimation
Ep, [[ are analogous. Next, by Jensen's inequality we have
03 E,, [00T] - E,,, [0f9))]
Ee, [00] - Er, (8 < <00,
and similarly we can also obtain Er, |ug] < 00.
With Lemma F.1, we will prove Proposition 5.1 via the following steps.
Step 1: Decompose the difference.
We first decompose both CURE and the loss separately as
PAS CURE
0
m
(20; - 1)53 + [ wo -85-222) =2,]²² + +2(1 - -3)5)) )
m j=1
(2w0 - 1)03 (1 - ( - -11)))
/=1
I(w)
1
22(1 - w,ys + 2(1 -w,) 2 2/ -1y) + (1 (1-ww) (1) 21)))
171 j=1
[(w)
lm m000
PAS
0)
m
(w, + (1 (1-ww)) w/)2z -,,²
m
j=1
m
(w + (1 - w,)mls - 02)
m j=1
(w)
1
2(1 - w/) (2) -1) - w,) - 2)) - m1 (1 (1) - 27)²).
(w)
and we interested in bounding
PAS. C w - lm 10 -0)l S sup [[(]] - 1I" [I(w) + sup [Iw)) III III'(())) (w))
sup w20
suup =20
(30)
w20
Step 2: Bounding the first difference A (w) == I(w) I (w). The proof in this step is directly adapted from Theorem 5.1 in Xie et al. (2012) and generalizes to non-Gaussian data. With some algebraic manipulation, we can further decompose
1 m Σ5
Aw(w)
(2w, - 1)03 + (1 1 - 13)²))
m j=1
m
(w + (1 11-wjj -wy)uy-
- 0
m
= CURE( ) - Lm 100000 0) - m j=1
2
Σ
M5 (1 - w)) - -0;)
where C = Σ
(2w, 1)03 + (1 - (
en (0"
0) (0",0)
Σ j=1
(w - -0,)2
j=1
29

---

Prediction-Powered Adaptive Shrinkage Estimation
corresponds to CURE and the loss of the "shrink-towwrr-zerr"" estimator 09 w, WFT We thus have
sup w
A (w)) I
sup w20
fm (
0)
sup 20
Σ
Mg w5y - -0;)
(31)
j=1
Now, rearrangements of terms gives that
m
( lm @ 0)
Σ
(071)²
03 88 2wjj (0
0,10, (0t)
sup w20
sup
w20
m
m
(0TT)² - 03 03
+ sup =20
Σ 2wlz
(0TT)² 07 02 -,3)
m
j=1
(+)
(aa)
For the first term (*),
2
Ep
En
2
33
03
Bp
Var
[
S
1 sup Varr, (0)T) 21
m²
m ;
j=1
j=1
Thus by Jensen's inequality and iterated expectation:
E2,
m
03 TTT) 2
09
S { sup Varp,, [(]7))] 2] m
1/2
(32)
m
For the second term (**), we start by arguing conditionally on 17. which implies in particular that we may treat all the dj as fixed. It is thus without loss of generality to assume that o ... S 6 (by first sorting problems according to the value of a3). Then, since wi is monotonic function of 03 for any fixed w 2 O, we have 1 2 wi ... 2 wm 2 0. The following inequality follows:
sup 220
Σ 2w, j=l
(0,T)² 05 -6:)
max
2
Σg
- o
(33)
12a.2..ccm
20
The following lemma would help us for handling the RHS of (33) (the same structural form of it will appear repeatedly in subsequent parts of the proof).
Lemma F.2. Let Ai,..., An be real numbers. Then
&
max 12e12..20.
G
A
=
max 1skS
A
20
i=1
Proof. Define S₁ Σ A; for I...., n. and let cly..., Cn be real numbers satisfying 2 c 2 ... 2 cn 2 0. Set
Cn+1= O. Then we can rewrite
Σ
A
Σ ca c1+1
A = Σ
(ck cx+1) Sk.
1=1
A=1
k=1
Since C₁ Ck-1, each ag Cj Ck+1 is nonnegative, and
ar C1 Cut1 $ 1.
1=1
Hence,
Σ
GA
Σ
S <
ae SSal
max
[Sal)
(
as
< Sk
max
[S
sksn
1=1
kml
Jm
Rm
30

---

Powerrd Adaptive Shrinkage Estimation
This shows
7 A
max 12c 2t
Σ
G.
S
max 1
A
20
i=1
To see that this upper bound can be attained, consider for each is the choice
C2 Ck =
Ck+1
Ck+2
Cn = 0.
Since 1 2 c1 2 ... 2 Gn 2 O, we have that
Σ
cA
Σ
A
=
|S
i=1
i=1
Taking the maximum over all such & E [I,....n]
n matches maxiskŠa S
Thus,
max 12e 12c2 20
Σ
GA
= max
A
ISkS
1=1
as claimed.
With Lemma F.2 in hand, we have
2 m71
- 05T0; 53)
2 Σ j=1
(OTT)
0,T
max 12c 22. 20
Cj
max ISk5m
m
j=1
Let M₁ = EE= ((0T)) OCTO, 77), it is easy to see that {M&) forms a martingale conditional on 17. Therefore, by
a standard 1,2 maximal inequality (ref. Theorem 4.4.6 in Durrett (2019)), we have
En
max Isksm
M2
Varys < 4E [MR = 4
- -
(34)
which then implics
Ep,
sup w20
2w/j
(107T)² -005000;
03 - -03)
4 Ep
max
M
m
Isksm
j=1
16
Varn, (0 -
- 0,
EP
m
j=1
16
sup Varp, (0)7)) -BP103]
S m
1 Σ 2woj (07T))
2-20000p OCTO, -o?) |
16
sup Varp, 2
- - ))
1/2
Ep,
sup w20
(35)
m
Next, we bound the last expression in (31): 2 sup E (1 wi)r (0 - 0 Note that (1 - ww is also monotonic
in o3. and if we define M₁ := セ My (0 0,), then {MB 1m forms another martingale conditioning on 17. Therefore. following the same argument as (33) gives
(1 W9 14, (OT - 0,))
max M 1 Isks
sup w
< Ep, m
m²
j=1
16 Er, m²
MM' 27
16
sup Er, Var [ u²
m
Fr,
2
sup
m
- ) (O9T 03)
S
16
sup Er,
Varny [ppT] (r]
1/2
(36)
m
w20
m
j
31

---

Prewiction-Powered Adaptive Shrinkage Estimation
The upper bounds derived in (32), (35) and (36) establish control
4 Vm
sup Varz, [0TT)] 1/2 + ssu Varp,, (0 -0g00] + sup Er, Varn, ( m1)
pp x
sup w22
| (w))
since each term on the right-hand side can be controlled by the fourth-moment conditions established in Lemma F.I, we know that A (w) converges uniformly to zero.
Step 3: Bounding the second difference A2(w) (T()) IT" (w). We next cancel out identical terms in the second
difference in (30) and get
2 m
22(w)
m Σ (1-ww)
w5) (2) -1) - 0g)]
(37)
By the same proof logic that has been applied twice above, we now have a function (1 wj) monotonic in 03. and a martingale Qx := Σ}= i/ (2)- 1y) 0,) for k= = I,.... m (recall that 3y Cov,, 105T, ZJDD. The steps from (33)-(34) follows, and we have
w5) - -22 - -my)(05 - -0)1)
16 sup Er, Varn, (2) (2/-11))03 -mg)( - -1:]p
Er,
sup
Σ (1 j=1
S m
m
w20
and so,
2 sup m w20
m Σ (1 (1-w;) (1-w;)]55 - (2/- - 10) -,
16 sup Er, Varns [2] 1775-1177 -m;)(0pT -1)0f/--))]]))))))))) 1/2
Ex n
<
Again, the (oourth-)mmment conditions from Lemma F.l suffice to ensure that
sup Ep, Var [2] -1) 0 < <0.
and to establish control of
Ep, sup w=2
(w))
Step 4: Concluding the argument.
Finally, based on Steps 1-3, we have that
C PAS,
PAS w
Ep
sup L220
Lm (0
< Ep,
sup
+ Ep,
sup
اا(سممه
w20
Lw20
and both terms on the right hand side converge to zero by our preceding bounds and the moment assumptions in the statement of the theorem.
F.3. Proof of Theorem 5.2
We apply a standard argument used to prove consistency of M-estimators.
Let We be the oracle choice of w 2 0 that minimizes the Bayes risk B (0 26 Notice that by definition of w as the minimizer of CURE,
CURE(6 PAS,
< CURE(O PAS.
Then:
km (0PAS 0) Lm
PAS O
0) < 2 sup
CURE PAS
PAS
em (
PAS 00
w
To streamline the proof, assume that the infimum is attained by value w.. If the infimum is not attained, the proof still goes through using approximate minimizers.
32

---

Poweerd Adaptive Shrinkage Estimation
Taking expectations.
T (0%
PAS,
PAS (0x,) wi
S 2 EE,
PAS, CURE(O lm (0 sup
PAS 000
Noting that the right hand side converges to 0 as co, and recalling the definition of wi, we prove the desired result
PAS B (0
S inf BT (0 PAS, w w00
+ ((1)
F.4. Proof of Proposition 5.3
We start with the result in Theorem 5.2
PAS (0
B (6 pAs +0)+((1)..
0
I inf
w20
Now, since we are integrating over 1j P,, for all problems
(0
PAS
1 0)
Σ Ee, (0P.N5 - -,)²
j=1
= =EP, [0RA5 - 0,)
by definition 0PAS = w, + - w where wi = w/(w + g2). Therefore
Er, n.w - -0,²9 = Er, (w: - 0,) + (1 - (2) - 03))
++
(w + jaye Li,,, [OPT - -0,)² + ( B2z,,
- (Z)-0,)²
2 82E (w + 02)2 Er, (0 - 8,) (2) - 0,)
By our assumption, second moment terms like 2² and f are now fixed, so we have (by iterated expectation)
Er (05T - -0,² =²?
Noting that 55 = since N, = co, we have
226²
/z
Ee, (0PAS - -0,)²] = (w- - の2)²
(w + ورحق Do Ez, (2) - 0,)²|
Plugging in w- Ex, [2]] -0))² 0] gives
w28² (w + 0202 +
Ø1
g2 Bp, [12 - -,,
Er, (z] - -0,)²]
+ + 02)2
σ2 + Ep, [2] - -0,)a]
We finally have
82 12/ 0,)2]
BP (d PAS
0² EP, ( - 0 o())
33

---

Prediction-Powered Adaptive Shrinkage Estimation
F.5. Proof of Lemma C.2
We aim to prove that lE,, [Ahtii Malip.m) )² parameter for m problems, and be its unclipped sample-bases based estimator:
+ cc. Let Am be the unclipped theoretical optimal global
and Á Σj= n 93 m j=1
Th notNo 2j=1 N
ny ai+N; + ng
The clipped versions are Xalp.. clip(Am. [0, 1)) and detip = clip(A, [0. 1)).
Step 1: Convergence in probability of the numerator and the denominator.
Let:
1
n + Nj
No
7
D
Ng-N "j
(38)
==1
j=1
1
n2 + N
No.
Dm
22 Tj
(39)
11
nl
j=1
So A = =N N/.DD/ Dm and Nm/ Nm/ We denote AN, Nm N = m プラ= Uj where U,= n - 7g)) Note
that EEz,, [Un] 0, and by assumption and we know that Varp,, [U,] Er, [ng Var,, [isl is bounded (say Vu 30), then have
Ez 1A, 0, Varz,|AN.. +
So AN,, E which implies AN.. f 0. Thus, Nn N S An identical proof will also give Dm D L for the denominator terms.
Step 2: Convergence in probability of unclipped ratio. By assumption 5, we have D (and Dm as well since Dm = D + op()) bounded away from zero in probability. Thus, by standard argument using the Continuous Mapping Theorem, we have
No, + op() N P 0. A Xin
D op(1)
D
for the unclipped ratio.
Step 2: 2 convergence for the clipped ratio. The clipping function g(z) = clip(ar, [0, 1)) is again continuous. Re-applying the Continuous Mapping Theorem gives
Xelip.m clip(,, [0, 1]) clip(A,,. [0, 1]) P, 0. Selip
Finally, the clipping also makes sure that 'Seli- - Notip.ml s 1. This then leads to the stronger convergence result
denp Xolip.m L2
and completes the proof.
F6. Proof of Theorem C.3
Denote VA(A) Var,, 05,Nl=
n,N, 73. Let Sm Ex VY(A). The average sum of variances is Vm (A) 1 Sm (A). We can express Vm (A) quadratic function of A: Vm (A) Cm 2Bm Amd2 where
2y N
ny
m
n2
N₂ Tj N2N
Am
(=D", from Equation (38)
m
j=1
Σ Yi
Bm
(= No from Equation (38))
n2j
j=1
1 of
Cm
n2
1=1
34

---

Prewirtion-Poweree Adaptive Shrinkage Estimation
The minimizer of V. (A) over À E is A Am (assuming Am which holds if not all r3 0). Since Vn (A) is quadratic function in À with positive leading coefficient Am (i.e., upward-opening parabola), its minimum over the closed interval [0, 1] is achieved at A clip(>as [0, 1J) = clip(Bm/Amp)) [0, 1]).
We want to show that Vm (deup) Vm (d((iip..)) F O. The derivative of (m is V 2AmA 2B By the Mean Value Theorem, for some Em between delip and Aetip.mi
Vn (Aelii)p) - Vm (Nali.m) = Vn Vn (m((mm) Velip.me
= =22Am5 - 2B - Nelip.m
Since ddti and Xelip.n are both in [0, IJ. Em is also in [0, 1]. The terms Am D and Bm = No are averages of m independent terms with uniformly bounded variances (under Assumption 4 Lemma C.2). Thus, by Chebyshev's 's inequality, Am (1()) and B. = Or(1) (i.e., they are bounded probability). Since S.. E [0, 1], the term (2Am5m 22 also (1))). Let AAm detip Nelip.m From Lemma C.2, have A - 0. Therefore,
Vm (Aati)) - Vm (Nelpp. = (2AmEm - 2B 22m))Am L The L' convergence then follows from the fact that This establishes the asymptotic variance optimality of the UniPT estimator.
E.7 Proof of Proposition C.5
Organization of the proof. We provide a high-level sketch of the proof in Figure 7. Our main goal is to establish that
CURE - fm - 0 uniformly in as co. To achieve this, we introduce two intermediate estimators-CURRE and CURE" thht serve as bridges between CURE and l.- The proof proceeds by showing that each consecutive pair of estimators is asymptotically close, which allows conclude the overall result via repeated applications of the triangle inequality.
CURE Eq. (26)
Lemma F.7
CURE' Eq. (40)
Lemma F.5
CURE" Eq. (41)
Theorem F.4
lm Eq. (9)
Figure 7. visual sketch of the proof. Each node represents a variant of the original CURE (i.e. a risk estimator conditioned on w); each arrow then represents asymptotic (m oo) closeness result between the estimators.
We first define two intermediate forms of CURE, denoted CURE' and CURE" respectively, between the original CURE (Eq. 24) that requires full knowledge about second moments and the sample estimate-based CURE defined Equation (26):
CURE' 0
UPAS
(2wg 1)03 + 2(1 w9) w z/) w/
(40)
83
with 03 being the variance target defined in Equation (2 In other words, we can treat 03 as fixed (but unknown) plug-in
value w is non-random for all w Next define CURE" by replacing the 03 and řý terms in CURE' which in turn depend on the estimated UniPT parameter delip with variants that depend on Xntip, instead:
CURE" (Ø
UPAS
(2wg - 1)83 + 2(1 w)) + 2 -2/)².
(41)
N + n2j
2
77
where
(Xelip.m)
X Xelip.m
98 81 Aelip. m N
nij
Nyng
CURE" possesses the following desirable properties:
We omit the dependency on m by using the shorthand 63 = 83.m introduced (27).
35

---

Prediction-Poweree Adaptive Shrinkage Estimation
UPAS,
is an unbiased estimator of Rm R0mO (0 conditioning on NT--
Lemma F.3 CURE" w
Proof. This is a direct result of Theorem B.1 and j. fg being unbiased estimators by construction.
Theorem F.4 (Asymptotic Consistency of CURE"). Under the assumption of Proposition C.5,
B2, sup CURE" (0w (6 w20
001
m=+00 0.
UPAS
Proof. For simplicity, hereafter we drop the notations' dependencies on 6..
By triangle inequality,
EF, sup |CURE"
-mml S Er, sup CURE L+20
1ml + Ep,
sup |CURE" - -CUREI L220
(42)
2220
and we remark here that the CURE referred in Equation (42) differs slightly from the CURE for PAS in Theorem 4.1 as we replace #3 with 03 in the definition of its wy (same as wf). 2 But since of is fixed plug-in value, we can still use Theorem 4.1 to bound the first term on the RHS of (42), so the only task here to bound the second term as well.
Let A₂ 33 o3 and Bj 71 Рg. By assumption, we have By, [Aj] 0, E,, [By] 0, Ey [A]] Vo < oc and E, [B}] S V, < oc. Let (w 2w0 and d,(w) 2(1 w,), we have
1 m m Σ (CURE" CURE,)
CURE" CURE-
m E (o(W))) + B,())
Now we can apply triangle inequality again. Since both ey (w) and dj (w) are monotone functions of w (and supported on [0, I)) we can use the same strategies (constructing martingale and applying maximal inequality) as in the proof of Proposition 5.1 Let Mj Σك Aj. we have
Σ
(w)
Σ (cx(w) (x+1 + cm (W)
(1 Icm (w)) max M <
max M 1skmm
(43)
15ksm
Now taking expectation over P,,, and use the maximal inequality gives
( max ISksm
27
m
Ex
|Mj]
Ep, [A7] S 4mV.
Finally, by Jensen's 's inequality, we have
1
m
Ep a
sup
Ag
S Er,
sup
Σ C
220
m
m wz j=1
4VVo mV, 222
S
Ep
max
JM
Er
max 15ksm
|M
S m
2
vm -
m
15k5m
m
Similar to Equation (43), we can show that
m Σ du(w)
2 max Isksm
IN|I
In other parts of this CURE, however, we keep using 53 = Var of 03 and H5 as m + cp.
GUPT Ali..
and 5/ = Cov
(RUTT SiA
2./..
which are also the limits
36

---

Prwwerd Adaptive Shrinkage Estimation
where == Eÿ= Bj. Again by applying maximal inequality, =
21 INEI) S 4mVy 1gksm
Ep,,
max
Finally
Ep
sup #20
1 Σ 3=1
B3
S Ep,
max 15k5m
[Ng]
_ mV
2
4VV mm - 0
as 772 - OC. Combining everything together gives
EF
sup w20
JCURE" CURE
S Ep,
sup L.20
-
Σ S.A)
+ +Bp,
sup L420
dB
O(I/vm)+ + 0 +
CURE" retains the nice asymptotic properties of CURE, but it consists of 83 and %j that cannot evaluate from data (due to the unknown m. Our next lemma derives the asymptotic closeness between CURE" and CURE', where the definition of the latter quantity one step closer to the fully data-driven CURE.
Lemma F.5 (Closeness between CURE' and CURE"). Under the assumption of Proposition C.5, we have
Ep, sup CURE' (0 (O CURE" (O, UPAS
)
m-000 O
2220
Proof. By construction, CURE' and CURE" only differ in their use of 03 versus 03 (similarly for ýJ v.8.5g).. We can thus decompose their difference
sup CURE' w>0
CURE"
sup w>0
1
(2w3 1)(0} - 83) + 2(1 - wi) - -
m
sup
(2wg 1)(67 - x3)1 + m
Σ
sup 1=1 w20
|2(1 - wg)(fs - -5))
171
/=1 4>0
03
831
+ Σ
hig - 355
(44)
m j=1
j=1
since sup,, [20] - 1/ < and sup.. |2(1 w < 2. Since we have dolii - Xetipmn (and Xaap (aalip. 33039 in L and by our fourth-moment assumptions have Ep, [o]]. Es, [+71. and Ep, [fy] all finite, by construction we also have
sup Ep, [00 - Ø
and sup Ez, [(5 - -,
0.
These uniform 2 convergence conditions sufficient make sure the expectation of Equation (44) converges 0, and thus prove our lemma.
At this point, we note that the only intractable piece in CURE' is the unknown variance target 83, which is used for constructing the weights up. CURE then operationalize CURE' by replacing 53 with the sample-based o². Our next lemma shows that 03 + o3 uniformly in T as m - oc, which then leads to our final closeness result between CURE and CURE'.
We defer proof of this convergence result Lemma F.6.
37

---

Powweed Adaptive Shrinkage Estimation
Lemma F.6 (Uniform 1,² convergence of o3). Under the assumption of Proposition C.5, have
sup Er, [(0] - -339 m 0
Proof. We can decompose the difference
1 n2
10² --22))++ + N₂ + my Setip (72 112)) Ny
2 7cdip( 14))
03 83
n,
Nj + n2
2 - (dylp Xalp.m)
24 (Nelip (Aclpp.
Ngng
n,
(.0)
Bounding (+): Using the inequality (a + bbb+ c)² s 3(a² b² +
E,, [())]] 3 (cTEE,,(0' 1002 (b=) + Cτ [1F2 )1=) +2
P.,,,,CC [(5
(45)
where C1 C22 C3 are some bounded terms involving n,; Nj. Satip (note that deti is clipped and bounded). Now, by our assumptions (esp. the finiteness nouth-moments) and the Weak Law of Large Numbers, as m + 0o we have
0²
[o Mot: ,² Er, =: 142, 7 L Er, hi] pay
then by our assumption (again by finite fourth moments in data generating process),
1022 po=)a] Vare, 1amr,,10' Varz, [,,,,,o]1111 = 0(1/m))
m
similarly, we know that the other two terms in the RHS of Equation (45) also O(1/m). Moreover, since all the problems are iid, these rates remain valid even if we take the supremum over j E [m]] We thus have
sup E 0 = o(1).
Bounding (*+): Similarly, we use the inequality (a + b)2 < 2a² + 26² to get
Ex [[ [**)*]] < 2 (L/EE, ((Raiip - (Adli,. ) + Br (Aeliip - \dlip.m))))
(46)
where Rr, hy some bounded terms involving nj, N5. 142,, May. By Proposition C.2, already know that dctip xeiip.m in 2 Further, since both dctip and Nelip.m are bounded within [0, 1). thus
((Rei (Aeiip. )²² = (detii Xeip. )²(Aetip + X²tp..)
< 2(Aolip - \atip.m)²
so we know that Aap (Nelip.m) in L as well. We thus have sup, Ep, [(*+) o().
Putting (*) and (**) together.
Since (0} - - s 2(x)² + 2(*x)) the above results show us that
sup E. [00 }}) < sup Re, [(+) + 2sup Er, [+**))===(11 = o(1)
and we obtain the uniform convergence in L
With Lemma F.6, we can derive an asymptotic closeness result between CURE' and the new CURE that we can actually caleulate from data.
38

---

Prowerd Adaptive Shrinkage Estimation
Lemma E.7 (Closeness between CURE and CURE'). Under the assumption of Proposition C.5, we have
pp sup w20
CC 2
CURE' (O
UPAS
m=-000 0
Proof. Denote Dy(w)= CURE; CURE, as the difference the j-th problem. We can further decompose
(m(w) = 20wjo 2Awnis Awj (2 w - w]) - 2/)²
Qo(w)
R₂(#)
(p(w)
w(o} 03)
with
Awy = w wj = (w + 03) + 03)
Going term by term, for (w(w), we have that for all w > 0
[Q;(w)) = 2 (w (w+ + o7) 83
w(03 - 83
- 100 -3]10331
2
w since w+og S 1 and w + 83 2 282 0 (by our assumption in Proposition C.5). Therefore,
EP
sup #20
Σ (m(w)
S EE
sup [w(Ww]
j=1
m
w0
1 m 2
m S md j=1
S
Er
|03
1311831
Er, [(0] - -3))0 Er, [(}3))]
(47)
5
m
/=1
where the last inequality follows from Cauchy-Schwarz, Schwarz. Handling Ry(w) similarly, we have
1 (gWW)
2
E n
sup w20
Eeg [63 -8)2 Er,, 1531.
(48)
m7
md
Finally, we have
w w183 591
|s/(w)|
12 :5 - wg1(0, - -2/22
(w + 83)(W + 89)
Using w+07
S 1, w+03 S 3. and |2 - W6-- w/1l 2,
2 103 - -
sup [Sس(w)] -
which gives
1 17
2 m mo
EE,,
sup w20
Σ (ww
<
Ez [00] 83121 Er, E,, 2z.4
(49)
Now, by the uniform 2 convergence of 83 given by Lemma F.6, as well as the fourth-moment assumptions that guarantee EP, [(=3)]] < 0o, Ep, (:3l < oo and E,, [(0 - 2 < oo, we immediately see that (47, 48, 49) are all o(1) terms. Finally
I m
UPAS CURE' ( ) < Er, , CURE
Ep, sup w20
sup w20
(w(
Ep,
sup
R (w)
m
w20
1 Σ (Ww
+ Ep,
sup .20
m-+
0.
m7
39

---

Prewertion-Powered Adaptive Shrinkage Estimation
Finally, combining the results from Theorem F.4, Lemma F.5 and Lemma F.7 via triangle inequality, we obtain the final result through
( 1m(0 10 0)}
UPAS
CURE" |CURE"(@ ( mm e
UPAS
Ep sup W20
CURD(O
- Br,,
sup w20
00)
Ep,
sup w20
|CURE' (0, (0,, CURE" (O.
UPAS
UPAS
)
UPAS
CURE' UPAS
+ Ee,
CURE(O_
w20
m=-000 0.
40


Reviews:
[{"review_id": "lBYfROOLZt", "reviewer": "Reviewer_Tcxk", "summary": "Paper proposes prediction-powered adaptive shrinkage (PAS), an extension of prediction-powered inference (PPI) that uses empirical Bayes ideas to further reduce estimates when multiple related estimation problems are solved together. The paper is well written and well thought out, with good theoretical and empirical results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the careful assessment. The thoughtful questions have helped us improve our work. For better exposition, we slightly reordered our responses to the questions.\n\n1. **Finite $m$ results.** Our current theoretical analysis permits for finite sample bounds. We will track these results more explicitly in the revision. For instance, the $o(1)$ term in Prop 5.1 is actually $O(1/\\sqrt{m})$. \n\n2. **How are variance parameters estimated in practice?** For real-world datasets, we use sample-based estimates for the variance parameters. We estimate $\\sigma^2_j$ and $\\gamma_j$ with the standard unbiased estimators\n\n    $$\n    \\hat{\\sigma}^2_j = \\frac{1}{n_j-1} \\sum_{i=1}^{n_j} (Y_{ij} - \\bar Y_j)^2, \\quad \\hat\\gamma_j = \\frac{1}{n_j-1}\\sum_{i=1}^{n_j}(Y_{ij}-\\bar Y_j)(f(X_{ij}) -\\bar{Z}_j^f)\n    $$\n\n    using labeled data. For the prediction variance $\\tau^2_j$, we use predictions in both labeled/unlabeled data:\n    $$\n    \\hat\\tau_j^2 = \\frac{1}{n_j + N_j -1}\\bigg(\\sum_{i=1}^{n_j}(f(X_{ij}) - \\hat Z_j^f)^2 + \\sum_{i=1}^{N_j}(f(\\tilde X_{ij}) - \\hat{Z}_j^f)^2 \\bigg)\n    $$\n\n    where $\\hat Z_j^f = \\frac{1}{n_j + N_j -1}\\left(\\sum_{i=1}^{n_j}f(X_{ij}) + \\sum_{i=1}^{N_j}f(\\tilde X_{ij})\\right)$.\n\n3. **Impact of estimating variance parameters (with Point 2).** \n    We agree with the reviewer that our current theory does not account for estimation errors due to plugging in sample-based estimates of the variance parameters. One possible remedy would be to consider asymptotics in which all of $n_j, N_j, m \\to \\infty$. However, we prefer asymptotics with $n_j, N_j$ fixed and $m \\to \\infty$ to represent the regime in which we have a lot of very noisy/difficult individual problems and also to distinguish results from standard setup in the PPI literature that keeps $m$ fixed and takes $n_j, N_j \\to \\infty$. We note PAS with plug-in variance estimates empirically works well even for very small $n_j$, see point 4 below.\n\n    Motivated by the reviewer's comment, we now consider two further methods: UniPT (that is, power-turning with the same $\\lambda$ for all problems, see response to reviewer qLFS) and UniPAS, which is similar to PAS but uses the same power-tuning parameter across problems. We can prove that UniPAS has asymptotic ($m \\to \\infty$, $n_j,N_j$ fixed) risk less or equal to that of UniPT, PPI, the classical estimator, and the prediction mean. **The result for UniPAS accounts for the data-based estimation of the variance parameters**. Here is a sketch of UniPAS:\n\n    - We start with UniPT by estimating a single power-tuning parameter $\\hat{\\lambda}$ (as in our response to reviewer qLFS) that has the property that $\\hat{\\lambda} - \\lambda^* \\to 0$ in $L^2$ as $m \\to \\infty$ (with $n_j,N_j$ fixed) where $\\lambda^*$ is the optimal single tuning parameter. Call the resulting estimator $\\hat{\\theta}_j^{UniPT}$.\n  \n    - We come up with stable **working** estimates of $Var[\\hat{\\theta}_j^{UniPT}]$ by **pretending** $\\sigma_j^2, \\tau_j^2, \\gamma_j$ are the same across all $j$ (but sample sizes can vary). The common values can be estimated accurately by averaging the estimates in Point 2 over all $m$. Plugging these in, we derive working estimates $\\bar{\\sigma}_j^2$ of $Var[\\hat{\\theta}_j^{UniPT}]$. Then we consider the one-dimensional parameterized family of weights\n        $$ \\bar{\\omega}_j(\\omega) = \\frac{\\omega}{\\omega + \\bar{\\sigma}_j^2}.$$\n        This family retains the property that for $\\omega=0$ it recovers prediction mean and for $\\omega=\\infty$ it recovers UniPT.\n  \n    - Pretend momentarily that $\\bar{\\sigma}_j$ and $\\hat{\\lambda}$ are deterministic. (This is not actually needed; by steps 1 and 2 above asymptotically these converge to a deterministic limit.) Suppose we consider the family of estimators \n  \n        $$\n        \\bar{\\omega}_j(\\omega)\\hat{\\theta}_j^{UniPT} + (1-\\bar{\\omega}_j(\\omega))\\tilde Z_j^f.\n        $$\n\n        An unbiased estimate of risk is given by CURE,\n        $$\\frac{1}{m}\\sum_{j=1}^m (2\\bar{\\omega}_j(\\omega) - 1)Var[\\hat{\\theta}_j^{UniPT}] + 2(1 - \\bar{\\omega}_j(\\omega)) Cov[\\hat{\\theta}_j^{UniPT}, \\tilde Z_j^f] +(1 - \\bar{\\omega}_j(\\omega))^2\\big(\\hat{\\theta}_j^{UniPT} - \\tilde Z_j^f\\big)^2,$$\n        which is analogous to the expression of CURE for PAS. Now here comes the punchline. If above we replace $Var[\\hat{\\theta}_j^{UniPT}]$ and $Cov[\\hat{\\theta}_j^{UniPT}, \\tilde Z_j^f]$ by unbiased estimates (analogous to Point 2), then we still retain an unbiased estimate of risk. Moreover, since we are averaging over $m$ with $m \\to \\infty$, we can establish asymptotic uniform consistency of our objective to the true risk, and thus we can establish asymptotic optimality for UniPAS.\n\n4. **Different $n_j/N_j$ and split ratios.**  Following the reviewer's suggestion we reran our real data analyses with the new methods and different labeled/unlabeled splits, going from 1%-40%. [[Link to the plots]](https://doi.org/10.6084/m9.figshare.28694657.v1)", "claims_and_evidence": "The claims are well supported by both theory and empirical results. The paper is well written and explains the ideas clearly with good intuition building examples.", "methods_and_evaluation_criteria": "Yes. Evaluation on both synthetic and real world examples demonstrate the efficacy of the method.", "theoretical_claims": "Theoretical claims are asymptotic in nature, holding as the number of problems m->infty.\n\nThey also assume the variance parameters tau_j, sigma_j, rho_j are known. While paper refers to this as of secondary concern, and referred to existing papers that took the same approach, this would seem to me to be quite a significant assumption and would be good to work out implications of having to estimate these from data on the theoretical developments.", "experimental_designs_or_analyses": "The experimental design looks sensible, with clear separation of datasets used to estimate the different quantities needed (e.g. the Bert fine-tuning). It is nice to see the method working across both textual and image domains with different predictors etc.", "relation_to_broader_scientific_literature": "The relation to broader literature is clearly set out through the paper.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. What assumptions or methods might be required to strengthen the asymptotic results to a finite m one?\n2. It would be good to more explicitly work out what happens if the variance parameters tau_j, sigma_j, rho_j are unknown and need to be estimated, and the impact of this additional estimation step on the resulting method and theoretical results. \n3. Please elaborate on how these variance parameters are estimated from data in the empirical results (particularly the Amazon and Galaxy Zoo datasets).\n4. It would be nice to look at performance of the method and baselines for smaller n_j/N_j, and also for a wider range of the label/unlabeled split beyond 20-80. The method should be able to work with a wider range.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXOdHFnGSK", "reviewer": "Reviewer_JtAw", "summary": "The paper proposes the Prediction-Powered Adaptive Shrinkage (PAS) method to enhance estimation accuracy for multiple means. PAS integrates Prediction-Powered Inference (PPI) with empirical Bayes shrinkage, first debiasing noisy machine learning (ML) predictions within each task and then leveraging information across tasks for improved estimation. Theoretically, the authors establish the asymptotic optimality of the tuning strategy and prove that PAS achieves a lower asymptotic mean squared error (MSE) than any baseline estimator. Experimental results on synthetic and real-world datasets demonstrate that PAS consistently outperforms existing methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty and contribution in our paper, as well as the many constructive comments. We hope our response below addresses all the concerns directly.\n\n### **Background on PPI**\n\nThe core idea of Prediction-Powered Inference (PPI) is as follows. Given existing black-box ML predictors $f$, how can we use them to enhance classical statistical procedures and improve their efficiency? The focus in this literature is on employing safeguards so that if the ML predictor is good, then statistical gains can be large, while if the ML predictor is bad, then one still retains some form of statistical guarantees (such as consistency). The focus on this literature is not on how to train the best possible ML models (most results are agnostic to that), but rather how to wrap around an existing arbitrary $f$ to improve statistical procedures. In this paper, we follow this established perspective in the PPI literature.\n\n\nHere, *arbitrary* typically means that $f$ can be of any functional form or a total black-box (e.g., API calls to LLMs), and should thus be considered given and fixed. While adapting to the quality of $f$ is important, an “arbitrarily bad” $f$ is rarely considered in practice, since prediction-powered methods are generally deployed only when $f$ is expected to provide at least some signal about the response.\nNevertheless, PAS **does** include mechanisms to adapt to both poor and high-quality predictors. To demonstrate this, we revisit the synthetic model and consider the following predictors:\n1. a (newly added) very poor predictor $f_r$, which outputs a $Unif[0, 2]$ value for all inputs, \n2. the near-optimal predictor $f_1(x) = x^2$ from the paper (in fact, $f_1$ closely approximates $E[Y_{ij} | X_{ij} = x]$).\n\n**Table 1: MSE (± se) $\\times 10^{-3}$ in synthetic model ($K=200$ replicates)**\n\n||Classical|Prediction Avg|PPI|PAS|\n|-|-|-|-|-|\n|$f_r$|3.14 ± 0.03|549.17 ± 2.57|24.05 ± 0.22|3.14 ± 0.03| \n|$f_1$|3.14 ± 0.03|0.27 ± 0.00|2.69 ± 0.03|0.27 ± 0.00| \n\nIn the extreme case where $f_r$ is very biased and provides no useful information, PAS  effectively defaults to the classical estimator, showing robustness against poor predictions. On the other hand, with a very good $f_1$, PAS tracks the predictive mean closely and perform best. Importantly, no over-shrinkage is observed. In words, PAS learns how good the predictor is in a data-driven way.\n\n### **Bounded moment assumptions**\nWe appreciate the reviewer's question regarding the finite moment assumptions. We first remark that these assumptions are satisfied when the response and prediction are bounded (as in the Amazon and Galaxy datasets), or when their joint distribution is reasonably well-behaved (as in our synthetic setting).\n\nWe note that *some* moment assumptions are standard across the PPI literature. Specifically, finite **second moments** of the joint model $(Y_{ij}, f(X_{ij}))$ are generally needed for basic procedures like power tuning. These are typically viewed as mild assumptions, although we admit that they preclude heavy-tailed distributions.\n\nOur work extends PPI to compound mean estimation using shrinkage principles, particularly risk minimization via an unbiased estimate (CURE). To prove the asymptotic optimality of PAS, we require finite fourth moments. This is a technical condition used to control the variance of the risk estimate itself.\n\nFinally, addressing the reviewer's question about violations: if the fundamental second moment assumption fails, the variance-reduction premise of PPI itself becomes ill-defined. If only the fourth moment assumption is violated, PAS may lose its formal asymptotic guarantee to outperform power tuning and the predictive mean. However, CURE remains an unbiased risk estimate provided that second moments exist.\n\n### **A light-weight approach**\n\nA key strength of PAS is its low computational overhead compared to both classical and PPI estimators. Although PAS uses both power tuning and adaptive shrinkage, the first stage yields a closed-form expression for the optimal tuning parameter $\\lambda_j^*$; the second stage involves optimizing CURE over a one-dimensional parameter space, and each evaluation is inexpensive since CURE has an analytic form as well. This makes the optimization very fast in practice.  \n\nWe first precompute all ML predictions. After this step, all estimators (including PAS) are very fast to compute. Below we benchmark the runtimes for three estimators on the Amazon Review dataset with the precomputed ML predictions. In each trial, the task is to estimate the mean product ratings for $m = 200$ products. We then record the time to construct each estimator by taking the average of 100 repeated constructions. The table reports the mean and max time (in milliseconds) to construct each estimator across 10 such trials.\n\n|Estimator|Mean Time (ms)|Max Time (ms)|\n|-|-|-|\n|Classical|7.4|8.4|\n|PT|21.2|22.0|\n|PAS|34.5|35.4|", "claims_and_evidence": "Please see my comments for each category below.", "methods_and_evaluation_criteria": "The proposed approach leverages Prediction-Powered Inference (PPI) and empirical Bayes methods to enhance statistical estimation, which I find to be a novel contribution. The theoretical guarantee of asymptotic optimality and the strong performance in numerical experiments further support its effectiveness.\n\nHowever, the proposed approach's performance is expected to depend heavily on the quality of the predictor used. If I understand the mechanism correctly, PAS does not have a built-in mechanism to correct or exclude misleading predictors, potentially impacting its effectiveness. Additionally, the computational complexity may be high due to the power tuning and adaptive shrinkage steps, which require optimization and could introduce additional overhead.", "theoretical_claims": "Although the asymptotic optimality of PAS provides a strong theoretical guarantee, one of my main concerns is the assumption of finite moments. The authors state that “PAS inherits the flexibility of PPI in working with any black-box predictive model and makes minimal distributional assumptions about the data.” However, the boundedness of moments may not always hold.", "experimental_designs_or_analyses": "The numerical experiments demonstrate the superior performance of the proposed method compared to the baselines. However, given that PAS's performance critically depends on the quality of the predictor, it would be interesting to analyze its sensitivity to heavily biased predictors—for example, by intentionally using a biased predictor to observe its impact. Additionally, since PAS involves power tuning and adaptive shrinkage, the computational cost is expected to be high. A detailed comparison of computational efficiency, such as running time benchmarks, would be helpful to assess the trade-off between accuracy improvement and computational expense.", "relation_to_broader_scientific_literature": "This paper could contribute to estimation, particularly for parallel ML tasks, which look for accurate estimation.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "How does the violation of the assumption of boundness of finite moment affect the optimality?\n\nHow does PAS perform when the ML model is biased, thereby the estimation is incorrect? If there is a way to screen and exclude those poor estimations?\n\nIf one of the estimations is nearly optimal (close to the true value), is the shrinkage method over-shrink it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UqiXcNDv6s", "reviewer": "Reviewer_qLFS", "summary": "This paper proposes a method for adaptively combining ML predictions with gold-standard labels, to estimate a multivariate parameter (e.g., the mean across several partitions of the data) with small mean-squared error.  The paper builds upon the PPI++ estimator, while proposing to additionally perform global shrinkage using an adaptively chosen shrinkage parameter.  Experimental results show that the approach approves over the classical estimator in MSE more often than a series of baseline methods.\n\n**Update after rebuttal**: I have reviewed the response and will maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a very helpful report, which has helped us improve on this work.\n\n* We agree that we should expand on connections to related work, which will be added as a new section in the appendix after our revision. Briefly:\n    - Fisch et al. 2024 (StratPPI): The starting point is similar to ours with multiple parameters $\\theta_1,...,\\theta_m$. However, in StratPPI, these parameters are not of interest per-say. Instead there exists known weights $w_1,...,w_m$ such that the parameter of true interest is $\\theta := \\sum_{j} w_j \\theta_j$. The ultimate goal is to come up with an unbiased and low variance estimator of $\\theta$.\n    - Rosenman et al (2023): When their stratum weights are the same, their estimator corresponds to our shrinkage baseline up to differences in the one-dimensional family of shrinkage weights (our Eq. (15)). \n    - PPI++: Suppose $n_j=n, N_j=N$ for all problems, then indeed we can cast our setting into PPI++ with a parameter vector. Moreover, PT is then using the same $\\lambda$ for all problems (more of which below).\n* Previously, we were comparing to PPI++ applied separately to each problem (with its own $\\lambda_j$). PPI++ itself requires asymptotics with $n_j,N_j\\to\\infty$. It would be possible to extend to asymptotics with $m,n\\to\\infty$; however, based on the reviewer comments we now have a more compelling methodological alternative: if we only seek to compare ourselves to PPI++ with a single $\\lambda$, then we can learn the optimal $\\lambda$ in asymptotics with $n$ fixed, $m\\to\\infty$ (a result complementary to the PPI++ paper which takes $m$ fixed and $n\\to\\infty$). In closed form, we have\n    $$\\lambda^*:=\\frac{\\sum_{j=1}^mn_j^{-1}\\gamma_j}{\\sum_{j=1}^m\\frac{n_j+N_j}{n_jN_j}\\tau_j^2}$$ \n    which coincides with the optimal weight selected by PPI++. We further clip $\\lambda^*$ to be within $[0, 1]$.\n\n    Without knowing $\\tau_j, \\sigma_j, \\gamma_j$, we can still plug in their sample-based estimates (see Point 2 for Reviewer Tcxk) and obtain $\\hat\\lambda$ (after clipping it as well), which has the property that $\\hat\\lambda - \\lambda^* \\overset{L^2}{\\to} 0$ as $m\\to\\infty$ (this is different from the per-problem case when $n_j \\to \\infty$ is needed). We denote this method as UniPT. \n* Taking UniPT as a starting point, we develop UniPAS. This has an asymptotic guarantee ($n_j, N_j$ fixed, $m\\to\\infty$) analogous to PAS replacing PT with UniPT (that is, we try to be asymptotically at least as good as PPI++ with a single $\\lambda$, while PAS was trying to at least as good as PPI++ with a separate-per problem $\\lambda$). The upshot is that UniPAS **does not** require knowledge  of $\\tau_j, \\sigma_j, \\gamma_j$. In empirical results UniPAS is competitive with PAS (although PAS is slightly better).\n* In addition to including two new methods (UniPT, UniPAS), we have reran our real data examples with labeled/unlabeled split ratios ranging from 1% to 40%. [[Link to plots]](https://doi.org/10.6084/m9.figshare.28694657.v1)\n* On the analogy of $Z_j^f$ and $\\mathbb E[\\theta\\mid\\psi]$: We agree that the analogy is somewhat loose and we will provide more details in the revision. Our goal is to provide a heuristic motivation for the one-dimensional parameterized family of weights $\\omega_j(\\cdot)$ (whose ultimate success is judged by the empirical results).  To elaborate on this: in Eq. (12) of Sec 3, we find that the best weights take the form:\n    $$\\frac{\\omega}{\\omega+\\sigma_{\\varepsilon}^2},$$ \n    for $\\omega = \\mathbb E[(\\theta - \\mathbb E[\\theta \\mid \\phi])^2] = \\mathbb E[\\mathrm{Var}[\\theta \\mid \\phi]]$. If we instead take the best convex combination of $\\hat{\\theta}^{cl}-\\xi$ and $\\mathbb E[\\theta]$ (instead of $\\mathbb E[\\theta\\mid\\phi]$), the optimal weights again take the form above, now with $\\omega=\\mathbb E[(\\theta-\\mathbb E[\\theta])^2] = \\mathrm{Var}(\\theta)$. Now suppose that we ask for the best convex combination (not necessarily a Bayes predictor) between $\\hat{\\theta}^{cl}-\\xi$ and $h(\\phi)$ where $h(\\cdot)$ is some fixed function. Then we can show that the optimal convex combination is given by the form above with\n    $$\n    \\omega:=\\mathbb E[(\\theta-h(\\phi))^2] =  \\mathbb E[\\mathrm{Var}[\\theta\\mid\\phi]] + \\mathbb E[(h(\\phi)-\\mathbb E[\\theta\\mid\\phi])^2].\n    $$\n    (The above expression is interesting as it forces us to inflate \"$\\omega$\", i.e., to shrink less toward $h(\\phi)$ in a way that depends on how close $h(\\phi)$ is to $\\mathbb E[\\theta\\mid\\phi]$.) The takeaway is that for many possible predictors, the optimal weights have the same parameterized form up to the single parameter $\\omega$ that varies according to the quality of the predictor. This motivates our one-dimensional family of weights. Once this family has been motivated, we learn $\\omega$ in (15) in a way that does not depend on the above analogy at all by minimizing CURE.", "claims_and_evidence": "The claimed contributions in the introduction are (in my opinion) well-scoped and well-supported, clearly distinguishing asymptotic claims from finite-sample claims, and justifying the benefits of the method with empirical evidence.", "methods_and_evaluation_criteria": "The method certainly makes sense, and has a clear intuitive basis.  Regarding the evaluation criteria, the theoretical development is mainly (in my view) for intuition, given some limitations (e.g., knowledge of certain parameters, see \"other strengths and weaknesses\").  So I view the empirical evaluation as the main \"evaluation\" component, where the evaluation approach appears sound to me---I particularly appreciated the use of the \"percentage of problems improved\" metric as a thoughtful counter to typical concerns with shrinkage-type estimators.", "theoretical_claims": "Unfortunately, I did not check the proofs in the supplement due to a lack of time.", "experimental_designs_or_analyses": "As stated in \"methods and evaluation criteria\", the evaluation appears sound to me, mainly focusing on the real data analysis which I believe presents the most robust empirical evidence for the method.  Given that the focus of this paper is on estimation (and not inference / uncertainty quantification), it is reasonable to measure performance by MSE, and I appreciated the extra inclusion of the \"% improved\" metric, since (as the authors note) MSE across an entire vector can be improved while sacrificing performance on some dimensions.", "relation_to_broader_scientific_literature": "As noted in part by the authors, there is a long line of work on combining imputed labels (which may be biased) with gold-standard labels. PPI++ is one such idea, where PPI constructs an unbiased estimator using imputed and gold-standard labels, while \"power tuning\" is added in PPI++ to improve efficiency by estimating the optimal degree of reliance on imputed labels.  Shrinkage is another idea, which can be shown to provably improve estimation (in a particular sense, namely MSE across the entire vector of parameters) when there are 3 or more parameters to estimate, an idea going back to the James-Stein estimator.  This shrinkage idea has also been applied in (other) areas of combining \"potentially biased data\" with \"unbiased data\", such as in causal inference, where the \"biased\" data is observational and the \"unbiased\" data experimental (see Rosenman et al. 2023, cited among other shrinkage-style estimators in this work).\n\nIn my understanding, this paper brings together these two lines of work with perhaps an additional twist (e.g., adopting a particular Empirical Bayes perspective on shrinkage, which I am less familiar with), and demonstrates that they work well together empirically, while giving some theoretical intuition.", "essential_references_not_discussed": "I believe much of the relevant related work is cited, but I would appreciate more discussion of how the proposed approach relates to some of the (cited) alternatives, e.g., Fisch et al. 2024 (who similarly considered stratified PPI++ across different subpopulations) and Rosenman et al. 2023 (who similarly consider shrinkage towards a biased predictor in a related causal inference problem).  For instance, does the \"Shrinkage\" baseline correspond to the method of Rosenman et al. 2023 or some variant of that method?", "other_strengths_and_weaknesses": "The paper is very clearly written, the synthetic experiments are a nice intuition pump, and the real-data experiments were compelling in my view.\n\nHowever, a few points of clarifications / \"weaknesses\":\n1. How is the PPI++ baseline implemented in the experiments?  As currently presented, I imagine that PPI++ is implemented (separately?) for each dimension of the target parameter to estimate.  However, PPI++ has more general formulations than mean estimation, e.g., one could view the present problem as a more general M-estimation problem, or even the task of learning a generalized linear model with a fixed coefficient for each stratum, both of which are covered in Sections 4 and 3 respectively of [1].  I'm not sure that actually results in a different estimator from the formulation presented here, but it would be helpful to clarify.\n2. As noted in Section 2.2., all of the theoretical work starts from the assumption that $\\tau_j, \\sigma_j, \\rho_j$ are known.  However, a major challenge in this area of research is that these must be estimated from the (same) limited data that we have for estimating $E[Y_{ij}]$.  Typically this deficit is handled somewhat crudely by appealing to asymptotics (i.e., convergence of similar terms in probability to their true values), but I don't see any of that here.  Of course, the ultimate validation is empirical, but I'm curious how the authors would address this weakness in the theoretical development:  For instance, do Proposition 5.1 and Theorem 5.2 still hold as both $m$ (number of tasks) and $n$ (number of samples) tend to infinity?\n3. I didn't follow the analog between the main method derived in Section 4 and the exposition in Section 3.  For instance, it seems in Section 3 that $\\phi$ and $\\theta$ are known to be correlated with the same mean, and in any case we should expect that $E(\\theta \\mid \\psi)$ is a better predictor of $\\theta$ than $E(\\theta)$.  Moving from Equation (12) to Equation (16), we use $\\tilde{Z}_j^f$ in place of $E[\\theta \\mid \\psi]$ by \"analogy\", but I don't understand the conceptual connection, given that e.g., $\\tilde{Z}_j^f$ could be arbitrarily biased, while $E(\\theta \\mid \\psi)$ is not.\n\n[1] A. N. Angelopoulos, J. C. Duchi, and T. Zrnic. Ppi++: efficient prediction-powered inference. arXiv preprint (2311.01453v2), 11 2023. URL: http://arxiv.org/abs/2311.01453v2, arXiv:2311.01453v2.", "other_comments_or_suggestions": "As a minor note, I found the use of distinct notation in Section 3 and Assumption 2.1 to be confusing, and it was a little hard for me to see the connections between the two.", "questions_for_authors": "My questions are essentially derivative from my comments above, but to recap (in priority order):\n1. How is the PPI++ baseline implemented in the experiments, and in particular, how does the \"multivariate\" form of PPI++ compare to the PT baseline shown here?  Is the multivariate form of PPI++ essentially the same, but with only a single fixed parameter lambda instead of a per-coordinate parameter?\n2. Where do existing shrinkage methods fit into the baselines shown in the experiments?  For instance, does the \"Shrinkage\" baseline correspond to the method of Rosenman et al. 2023 or some variant of that method?\n3. As noted in Section 2.2., all of the theoretical work starts from the assumption that $\\tau_j, \\sigma_j, \\rho_j$ are known. For instance, do Proposition 5.1 and Theorem 5.2 still hold as both $m$ (number of tasks) and $n$ (number of samples) tend to infinity, i.e., is there at least an asymptotic argument that this does not matter?\n4. Could you explain the conceptual reason why the analog in Equation (16) of $\\tilde{Z}_j^f$ to $E(\\theta \\mid \\psi)$ makes sense?  I think I get the intuition at a very rough level (it's additional \"prior\" information), but see my comments above in \"other strengths and weaknesses\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FtVvfkMzB1", "reviewer": "Reviewer_RfuV", "summary": "This paper proposes a \"shrinkage estimator\" for estimating multiple problems at once using PPI (prediction powered inference). They discuss the various methods by which one can reduce variance in such an estimator and how their method takes advantage of each, demonstrating theoretically that they can get better estimators by optimizing the CURE (correlation based risk estimate) which they introduce. Empirically they show on synthetic and real datasets that this improves on multi-estimation over both classical and PPI baselines.\n\n#########\nUPDATE AFTER REBUTTAL:\nI remain positive about this paper - keeping my score at 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for considering our idea convincing and thoughtful comments.\n\n* **Sharing information across problems:** We agree with the reviewer that further motivation of why and how it is possible to share information across problems will improve our exposition. In our revision we will add further motivation. Below is an alternative attempt to further explain and elaborate on Prop 5.3. Suppose as in that proposition that $n_j$ is the same across all problems, $N_j=\\infty$, and that second moments $\\rho_j,\\tau_j,\\sigma_j$ are identical across all problems. We make these assumptions throughout the remainder of our response. Then we could ask: What is the best convex combination of $\\hat{\\theta}_j^{PT}$ and $\\tilde{Z}_j^f$ in the following sense:\n\n    $$\\omega_j^*\\in\\mathrm{argmin}_{\\omega_j}\\mathbb E[(\\theta_j-(\\omega_j\\hat{\\theta}_j^{PT}+(1-\\omega_j)\\tilde{Z}_j^f)^2].$$\n\n    By direct calculation (since the RHS is a convex quadratic in $\\omega_j$) we find that:\n    $$\\omega_j^*=\\frac{\\mathbb E[(\\theta_j-\\tilde{Z}_j^f)^2]}{\\mathbb E[(\\theta_j-\\tilde Z_j^f)^2]+\\tilde\\sigma^2}.$$\n    This implies the following intuitive result: the larger the MSE $\\mathbb E[(\\theta_j-\\tilde{Z}_j^f)^2]$, the less weight we should assign to $\\tilde{Z}_j^f$. Note that if we have a single problem $m=1$, then if $n_j$ is sufficiently large, we could estimate $\\tilde{\\sigma}^2$ accurately. However, we cannot estimate $\\mathbb E[(\\theta_1-\\tilde{Z}_1^f)^2]$ accurately since we only have a single $\\theta_1$ (a single problem). At best, we can compute an unbiased estimate of this quantity since:\n    $\\mathbb E[(\\hat{\\theta}_1^{PT}-\\tilde{Z}_1^f)^2]-\\tilde{\\sigma}^2=\\mathbb E[(\\theta_1-\\tilde{Z}_1^f)^2].$\n\n    Now suppose we have multiple problems, then we can now learn how good the ML predictor is for estimating the $\\theta_j$ by sharing information across problems. To wit, as $m \\to \\infty$\n\n    $$\\frac{1}{m}\\sum_{j=1}^m(\\hat{\\theta}_j^{PT}-\\tilde{Z}_j^f)^2-\\tilde\\sigma^2\\stackrel{\\mathbb P}{\\to}\\mathbb E[(\\theta_j-\\tilde Z_j^f)^2],$$\n\n    where we emphasize that the mean squared on the RHS also integrates with respect to the **meta-distribution** $\\mathbb P_{\\eta}$ that models the distribution of the $\\theta_j$. Thus we can also estimate the optimal $\\omega_j^*$. In words: **sharing information across problems allows us to learn how good the ML predictor is** and to then decide how much to shrink toward it. Our implementation shares information in a similar way but is more involved due to the heteroscedasticity across problems (not all second moments are identical) and also for heterogeneity in $n_j$, $N_j$. \n\n* **Role of meta-distribution:** See above. We note that we could state our result in a frequentist setting wherein all parameters are deterministic, similar to the classical result of [James-Stein](https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator). We preferred to state results in terms of a meta-distribution, since we thought this would be a more familiar setup for audience at ICML. \n\n*   **Assumption on known second moments:** \n    We agree that our assumption on second-moment parameters $(\\sigma^2_j, \\gamma_j)$ are not easily satisfied in practice. In Sec 2 of paper, such treatment is more of a theoretical convenience, and sample-based estimates are used for real-world datasets. Therefore, it is true that an implicit requirement for our theory of PAS to work well in many practical settings is that the sample-based estimates are good enough. While our regime ($m\\to\\infty$, $n_j,N_j$ finite) cannot yield asymptotic result on these sample-based estimates, we make the following remarks:\n\n    1. In practice, PAS works well even when $n_j$ is very small (and so the estimates are noisy). To highlight this point further and motivated by your suggestion, we reran our real data examples with different labeled/unlabeled splits, going from 1%-40%. PAS still dominates other baselines even when only 1% of the data is labeled (e.g. $n_j < 10$ for Amazon Review). [[Link to the plots]](https://doi.org/10.6084/m9.figshare.28694657.v1)\n\n    2. In our response to Reviewer Tcxk, we propose a new variant of PAS called UniPAS, whose asymptotic guarantee does not require knowledge of second moments. UniPAS also has competitive empirical performance with PAS.\n\n*   **Other comments:**\n    *   L204: The underline is intentional to denote the abbreviation \"cl\" for classical estimator. We can clarify this.\n    *   L170: Thanks for catching the typo; we will fix it.\n    *   Covariate Shift: We interpret this as potential differences in $P_{\\eta_j}(X_{ij})$ across problems $j$. In the synthetic model, the mean of $X_{ij}$ varies with $\\eta_j$, but that is not an assumption that PAS makes. What makes \"information sharing\" more effective, as mentioned above, is when $(\\theta_j-\\tilde{Z}_j^f)^2$ is small on average across problems. If the question refers to train/test covariate shift within problems, this violates the PPI assumption.", "claims_and_evidence": "I find the claims + evidence shown in this paper to be convincing. The authors provide both theoretical and empirical evidence for their claim, which is that PAS is an improved estimator over PPI in the multi-prediction setup. I find the background and intuition given in sections 2-3 to be quite helpful.", "methods_and_evaluation_criteria": "Yes, the evaluation setup is fairly standard for PPI and adapted to their specific setting (multi-prediction).\n\nIt would be nice to see experiments shown on different sizes of labelled data to demonstrate at what value of n we start seeing improvements from PAS.", "theoretical_claims": "I did not check closely but I did look at the proofs in Appendix A.", "experimental_designs_or_analyses": "fairly standard setup and showing improved risk on two metrics", "relation_to_broader_scientific_literature": "connection to existing literature is well laid out in this paper - discusses very clearly in Section 3 exactly how various methods from the PPI and Bayesian space connect to this work", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Main critique: I would love a little more intuition about what “parallel problems” are supposed to be, in particular early on in exposition and then tied to the examples in 2.2 and methods in 4.2. I was a little unclear on how \"borrowing information across problems\" is done (as stated on line 42). I can see there is shrinkage going on but that is calculated on what seems to be a mostly problem-by-problem level as in (15). So a little more explanation here would help my understanding of what's going on - for instance I don't know see why we need the machinery from Eq 5 of defining the meta-distribution over problem parameters, it doesn't seem there's any assumption made on how problems are connected or distributed.\n\nQuick note on the assumption of correlation/covariance statistics being known - Var(Y) and corr(Y, f) can be tough to estimate is labelled data is small. Would be good to discuss how this is implicitly an assumption that labelled data is big enough, and/or what regime of n you expect this estimator to work well in.", "other_comments_or_suggestions": "L204: “cl” in classical is underlined?\n\nL170 right: typo on “unlabeled” \n\nRelatedly to the \"sharing information\" point, I would be curious to know if there's any type of covariate shift type assumption that you think is helping here- for instance, it is roughly (not exactly) satisfied in the synthetic example Ex 2.2.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an extension of a recent statistical approach called Prediction-Powered Inference.
The extension enables to non-trivially exploit this approach in the context of several parallel and possibly related tasks.

All reviewers were very positive regarding the contribution, novelty and importance of the problem.
So I am happy to accept this paper.

* The reviewers have made several useful suggestions for improving the presentation and providing intuition, please take these into consideration
Decision: Accept (poster)