FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference
Authors: Stefano Cortinovis
Francois Caron
Venue: ICML 2025 poster (2025)
Keywords: Confidence intervals
Bayesian methods
heavy-tailed priors
horseshoe
mean estimation
statistical inference
semi-supervised inference.
Primary area: probabilistic_methods->bayesian_models_and_methods

Abstract: Prediction-powered inference (PPI) enables valid statistical inference by combining experimental data with machine learning predictions. When a sufficient number of high-quality predictions is available, PPI results in more accurate estimates and tighter confidence intervals than traditional methods. In this paper, we propose to inform the PPI framework with prior knowledge on the quality of the predictions. The resulting method, which we call frequentist, assisted by Bayes, PPI (FAB-PPI), improves over PPI when the observed prediction quality is likely under the prior, while maintaining its frequentist guarantees. Furthermore, when using heavy-tailed priors, FAB-PPI adaptively reverts to standard PPI in low prior probability regions. We demonstrate the benefits of FAB-PPI in real and synthetic examples.

FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference
Stefano Cortinovis
François Caron'
Abstract
are good, PPI results in more accurate estimates and shorter CIs than traditional methods that rely solety on labelled data.
Prediction-poweett inference (PPI) enables valid statistical inference by combining experimental data with machine learning predictions. When a sufficient number of high-quality predictions is available, PPI results in more accurate estimates and tighter confidence intervals than traditional methods. In this paper, we propose to inform the PPI framework with prior knowledge on the quality of the predictions. The resulting method, which we call frequentist, assisted by Bayes, PPI (FAB-PPI), improves over PPI when the observed prediction quality is likely under the prior, while maintaining its frequentist guarantees. Further- more, when using heavy-tailes priors, FAB-PPI adaptively reverts to standard PPI in low prior probability regions. We demonstrate the benefits of FAB-PPI in real and synthetic examples.
More formally, for an input/output pair (X, Y) P = Px x Pyjx and a convex loss function (o 3), where 0 E Rd, we wish to estimate
= arg min (X) BER
(I)
For instance, if La Lo v) = w)- is the squared
loss, then 6* E[]]. We assume that we have n labelled observations {(X,, Yi) iid from IP and N unlabelled observations {X.} iid from Px, which are also
independent of the labelled data. The number of unlabelled observations typically much larger than the number of labelled ones, N > n. Additionally, we provided with machine learning prediction rule f, that can be used to predict an output f fxx at any input x. PPI aims to obtain estimator 9 and (1 a) confidence interval CPP for 0r which take advantage of f. Under mild assumptions, 8* can be expressed as the solution to
1. Introduction
go- == E Ei (X, = O,
Statistical inference crucially relies on the availability of high-quality labelled data draw actionable conclusions. As the scale of machine learning models keeps growing, their increasingly accurate predictions become a tempting alternative to labelled data in fields where the latter are tra- ditionally scarce, such as proteomics (Jumper et al., 2021) I). However, blindly using potentially biased predictions as a surrogate for labelled data voids the statistical validity of
where Co is subgradient of La with respect to A. It is easy to see that the quantity above can be decomposed as ge = mg + Ag, where
ma ((C Ao = E(C,(X,)) (X,
(3)
(4)
In this setting, me represents measure of fit of the predictor, whereas Ag. called the rectifier, accounts for the discrepancy between the predicted outputs f(X) and the true outputs Y, effectively quantifying prediction quality, For example, under the squared loss, As = E[f(X) - Y] and a good predictor f is one such that A is close to zero, i.e. f(x ~ E|Y X = x². Note that, while in this case As does not depend on 0, this not true in general.
the conelusions drawn. To address this, rrediction-powere inference (Angelopoulos et al., 2023a) provides general framework for statistical inference in the presence of a large number of black-box predictions by combining them with smaller number of labelled observations, which are used to correct for the discrepancy between the predictions and the true labels. The estimators and confidence intervals (CIs) resulting from PPI are statistically valid regardless of the ma- chine learning model used. Moreover, when the predictions
By estimating the two quantities me and Ag. Angelopoulos et al. (2023b) derive estimator and CI for 8-, which use both labelled and unlabelled data. The resulting CI is shorter than the classical confidence interval based solely on the labelled data when N > n and f is accurate because, in this case, my can be estimated with low variance using the unlabelled data, while Ag is close to zero.
Deparrmeent of Statisties, University of Oxford. Correspon- dence to: Stefano Cortinovis <cortinovis stats. ox. .x.ac.un au.>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

FAB-PPI
Standard PPI employs off-the-shelf estimation and CI pro- cedures for Ao. which do not take advantage of any prior knowledge on the quality of the machine learning model f. However, in many applications, we expect the latter's predic- tions to be (i) usually very good, but (ii) sometimes prone to large errors and hallucinations. We propose to encode such an inductive bias with a horseshoe prior Ty on AS (Carvalho et al., 2010), which accommodates the aforementioned prop- erties by exhibiting (i) an infinitely tall spike at the origin, and (ii) Cauchy-like tails at infinity. In order to construct valid confidence regions for Do using the horseshoe prior, we resort to the trequentist-assistee by Bayes (FAB) frame- work (Pratt, 1961; 1963; Yu & Hoff, 2018). This approach provides confidence regions such that their expected length lower for rectifiers Ag that have high probability under Te, and larger otherwise. While the resulting confidence regions have exact coverage for any prior Te, the horseshoe prior is particularly well-suited for PPI. Being concentrated around the origin, it produces shorter confidence regions when the predictions are good. i.e. | = O. At the same time, its heavy tails ensure robustness when the predictions are poor. Indeed, as shown by Cortinovis & Caron (2024), O, the FAB procedure with the horseshoe prior reverts to the traditional CI based on the sample mean.
mulation of PPI, leading more computationally efficient procedure, along with an additional power tuning parameter to enhance PPI's performance. Stratified PPI (Fisch et al., 2024) improves upon PPI by employing a data stratification strategy. Cross PPI (Zrnic & Candès, 2024b) demonstrates how the training of f can be included in the PPI pipeline. Ac- tive statistical inference (Zrnic & Candès, 2024a) applies an active learning approach to select which inputs from the un- labelled set should be labelled. Closer to our work, Bayesian PPI (Hofer et al., 2024) considers an alternative PPI estima- tor motivated by Bayesian ideas. However, their approach provides Bayesian credible intervals, which do not offer fre- quentist guarantees. Additionally, their approach achieves similar experimental performance to PPI, while we demon- strate that FAB-PPI may significantly improve upon PPI.
As discussed in Angelopoulos et al. (2023a;b), PPI has close ties with control variates for variance reduction (Glasserman, 2003, $4.1). In the case of mean estimation, the form of the PPI estimator is similar to the one proposed by Zhang et al. (2019). PPI is also related to work in semiparametric inference with missing data (Robins Rotnitzky, 1995).
if ||
The concept of Bayes-optimal confidence regions originates from the work of Pratt (1961; 1963). Pratt's approach, which has been given the name FAB by Yu & Hoff (2018), 8), has since been extended in multiple directions (Brown et al., 1995; Farchione & Kabaila, 2008; Kabaila & Giri, 2013; Kabaila & Farchione, 2022; Yu & Hoff, 2018; Hoff & Yu, 2019; Hoff, 2023). In particular, Cortinovis & Caron (2024) show that, when combined with priors with power-law tails, FAB provides robust confidence regions that revert to clas- sical ones in the presence of outliers. Hoff (2023) applied FAB in predictive supervised context, showing that it can lead to more accurate predictions than standard methods.
In this work, we introduce FAB-PPI, Bayes-assisted ap- proach for PPI that encodes prior information on the quality of the machine learning predictions by specifying a prior for the rectifier Ag. FAB-PPI is:
Statistically valid, as its confidence regions have cor- rect coverage for any choice of prior;
Efficient, its confidence regions have smaller ex- pected length when the predictions are good;
Robust, as it reverts to standard PPI when the predic- tions are poor, if the horseshoe prior is used;
3. Background
Modular, as it can be used in conjunction with power tuning (Angelopoulos et al., 2023b).
3.1. Control Variates
The method of control variates is a standard variance- reduction technique in Monte Carlo approximation (Glasser- man, 2003, $4.1). For simplicity, we present the method in the scalar case, but extensions to the multivariate setting are available. Let (Z, Y) be pair of real-valued random variables, and assume we are interested in estimating E[Y] based on an iid sample ((Z,, Y:) Assuming μ = E E is known, one defines the control- estimator (CVE)
The remainder of the paper is organised as follows. Sec- tion 2 reviews related work. Section 3 provides background on control variates, PPI, and FAB confidence regions. Sec- tion 4 describes our novel approach for PPI, called FAB-PPI. Section 5 demonstrates the benefits of FAB-PPI on synthetic and real data. Finally, Section 6 discusses limitations and further extensions of our approach.
2. Related Work
Y = Y ----C A(Z 14)
Σ (Y ---- x(Z--4)), (5)
PPI (Angelopoulos et al., 2023a) was introduced to obtain shorter CIs for the parameters of interest by leveraging ma- chine learning predictions in semi-supervised settings. PPI has since been extended in multiple directions. PPI++ (An- gelopoulos et al., 2023b) proposes different, loss-based for-
where A E R is a tuning coefficient and Z and Y are the sample means of (Z,) and (Y,), respectively. The centred random variable Z, - a serves as a control variate to estimate Ey.. The CVE is a consistent and
2

---

FAB-PPI
unbiased estimator of B|y] with var(Pry) (var(Y) Y) X2 vaar(Z)) while var(Y) /ar(Y)/n. Therefore, the CVE has smaller variance than Y whenever A E (min(0, }, max )), where the optimal coefficient is X* = cov /v)) In this case, var(P (1 -1-10 ) var(()) where pz, is the corre- lation between Z and Y. The more correlated Z and Y, the larger the variance reduction. By plugging the estimator
as an estimator for Ag and the associated classical Cls construct R.s. For the squared loss, the estimator 6PP solving
me + A 0 takes the control variate form
1 N
@e =
((X.)
Σ [(x))) N
(I)
1m1
with control variate s(xi) - N Σラ S(X,) and = 1.
-= D Di=((2 (Z₁ ZY, -Y)
1 2)²
(6)
PPI++ Angelopoulos et al. (2023b) extend standard PPI by introducing an additional control-variate parameter which they call power tuning parameter. The chosen mg is still the sample mean (9), while A now takes the control variate form
for À in Equation (5), one has
v - ]| 10) 8/Vn
1
A
DD 0
(C6 - -(xx,
(12)
as n -> -,00, where is the sample standard deviation of {(Y, - -Z,))==...... Hence, Y = 21 2//3/ v is an
1=1
( --11 1)
E -ma
asympiotically valid (1 111a) a) CI for E[Y], whose asymptotic
-
width is 2 P2,y var(Y)/vn.
where À is estimated from the data. In this case, the centred control variate is f ma. which depends only on the machine learning predictions. For the squared loss, we obtain
3.2. Prediction-Powered Inference
PPI (Angelopoulos et al., 2023a) defines an estimator ① and a CI CTP for a parameter of interest 8" satisfying Equation (2). In particular, let ma and Ag be some estimators of my and Ag. Using Equation (2), the estimator 0 is defined as the solution, in 0, to the equation
n 1 N
@eP =
Σ i=1
f(X.) - 1 1 ((X)))
(13)
11
N
5=1
mg + A. =0. O.
with plug-in estimator
Cp
Similarly, let Rs and Ta-5 be 1 8 and - (a- - 5) CIs for Ag and ma, respectively. Then, the PPI confidence interval CPP is defined as
(14)
(1 + N) Un+N'
where on is the sample covariance of (Y,, f(X)); and
Un-N is the sample variance of ((f(X,)) (f(X,)) >).
Cº = 00|0 0 Rs + +a-8),
(8)
The estimator (13) is closely related (though slightly differ- ent) to the one introduced by Zhang et al. (2019) for mean estimation in semi- cumi-supervieed inference.
where + denotes the Minkowski sum. Typical choices for my and Ta-6 are the sample mean of the unlabelled data,
N mg Σ Co ((.)),
CLT-based Cls. While the definition of the PPI confi- dence interval (8) allows for merging any CIs R, and Ta-5 for Ag and mø, in practice these often chosen to be CLT-based CIs that, once combined into CPP give exact asymptotic coverage,
and classical Cls for sample means, respectively. Different choices for A have been proposed in the literature, leading to different PPI estimators.
lim inf Pr(O" n, 700
CPP) 1-aa
Standard PPI. Angelopoulos et al. (2023a) propose to use the sample mean
Such CL.T-hased CIs rely on the following standard assump- tion on the estimators mh and As.
Assumption 3.1 (CLT assumption for PPI and PPI++), Let mg be the sample mean (9) and consider some estimator (2 of var(mo). with (0 /var((m))) -> almost surely.
1 n1
- -x,,,
(10)
3

---

FAB-PPI
Let 7g be either the PPI estimator (10) or the +PI++ esti- mator (12) and consider some estimator oo of var (A) with
4. FAB-PPI
Our approach, which we call FAB-PPI, combines the PPI framework with the FAB construction of confidence regions by specifying prior on the rectifier Dg. To ease the pre- sentation, here we describe the method for Y, E RR The general multivariate case is discussed in Section $4.
/ 10/var(A) (Do) + 1 a.s. Assume that, as min(n, .N - 00,
(ma- ma)/o/ - 1(),
(15)
(A. 00) + (0,
(16)
3.3. Bayes-Optimal Confidence Regions
As in PPI, we use the sample mean (9) as the estimator of Mg. For Ag, we start by considering a consistent estimator Âg, such as the sample mean (10) used in PPI, or the con- trol variate estimator (12) used in +PI++ Throughout this section, we assume that Assumption 3.1 is satisfied. That is, CLT holds for m. and De with respect to some estimators ) and 0g of var(mo) and var (Ao), respectively. In this setting, let To To Tn)) be a prior on Do with scale parame- ter Ta. which may depend on the labelled data through og. Denote by E(w; T) the log- marginal likelihood, evaluated at w. of Gaussian likelihood model with mean A and variance o² under the prior mo(A;
The FAB framework (Pratt, 1961; 1963; Yu & Hoff, 2018) aims to construct valid confidence regions with smaller ex- pected volume. Let W | B ~ ( o²) with some prior (3) and denote by r(w)= = f y(w (8) (o (8)d3 the cor- responding marginal likelihood. For a E (0, 1). let Ca (w) be an exact (1 a) confidence region for B based on the data w. That is, for any fixed So.
Pr(B E (W) 1 B 30) = 1 a.
(17)
Let vol(C.. = (e)) (w) d33' be the volume of (w),
and consider its expected value under the marginal likeli- hood n
l(w;o, = log / N(w,, A, mo)o)))
[(vol(C. = vol(Ca (w))
(18)
4.1. Bayes-Assisted PPI Estimators
Definition 3.2. For E (0, 1), > 0 and a prior no (8)) the FAB confidence region C. for the mean parameter 3 of
Consider the Bayes- estimator
the normal model Y B ~ (V, )²), is the minimiser of the (Bayesian) expected volume
A = - + Gz (Ao:ão.T))
(20)
for the rectifier Ag. By Tweedie's 's formula (Efron, 201 1), the above estimator is the posterior mean of the mean parameter of a Gaussian likelihood model under the prior To. Note however that we do not assume here that Ag normally distributed for fixed n.
Ca = arg min [[vo
(19)
C.
subject to the (frequentist) coverage constraint (17). We write (w) FAB-CR(w; (w; To, a))..
The solution to Equation (19), which exists and is unique if To(3) is not degenerate (Cortinovis & Caron, 2024, Theo- rem 3.3), may be found numerically as long as the marginal likelihood T (w) can be evaluated pointwise. Additional details are provided in Section S1.2. Intuitively, the FAB confidence region (w) constructed through Equation (19) will be smaller for values of that are likely under the marginal likelihood, and larger otherwise. As a result of this, while FAB guarantees the right coverage for any prior, one that assigns high probability to the value of 39 that gener- ated the data is required to achieve smaller expected volume compared to the standard CI (w + o2)-a/2), whose width does not depend on w.
The FAB-PPI estimator of 0*, denoted by @RBEPP, is then obtained as the solution, in 0, to the equation
in A FARPE 0.
4.2. FAB-PPI Confidence Regions
As in PPI, let Ta-s(mo) denote a standard 1 (a 6) confidence interval for mg. For Ae, we apply the FAB framework with the prior To to obtain a 1 - ö confidence re- gion REASP? (Ao) = FAB-CR 9; nol: TT), og 8). Then, the FAB-PPI confidence region C CEABPE is obtained as
CEABPE = {0 10EER PABPP (Bo) + Ta Ta-o((ma))) (21)
C
Bayes-Assisted Estimator. A natural estimator to use alongside the FAB confidence region C, (w) is the posterior mean ((W)= Ei3 W]. As shown by (Cortinovis & Caron, 2024, Theorem 3.3), it is always contained within the confidence region: (w) E (w( for any w E R and any a E 10. We refer to ((W as the Bayes-assisted estimator.
Algorithm summarises the steps of the FAB-PPI approach in a general convex estimation problem.
4.3. Choosing the Prior
FAB-PPI is motivated by applications in which the PPI pre- dictor f is expected to be generally accurate, as measured
4

---

FAB-PPI
Algorithm FAB-PPI for convex estimation
motivated by Piironen & Vehtari (2017, $3.3). Furthermore, the horseshoe prior has power-law tails, making it particu- larly robust choice for FAB-PPI, as discussed in Section 4.4, Crucially, for both priors AN and THs. the marginal likeli- hood under a Gaussian model with standard deviation ōg can be expressed terms of standard functions (see Sec- tion S1.1 for the horseshoe), enabling us to compute AABee and R (Aa) in Algorithm 1.
Input:
unlabelled (X,) predictor S, prior mol: :T,) error levels
labelled {(X,, Yi)li
a,ó
Set Ā (FAB-PPI) or estimate Ā from data (FAB-
-PI+-) in Angelopoulos et al. (2023b).
for 0 E O gnd do
mg - & Ei=
C/ (,,,,,,(X.)
E Ei= (x,(,,, Yi) - (X,,
4.4. Theoretical Properties
Ao E (1-1)mg
As shown by the following result, proved in Section S3.1, the FAB-PPI CR has exact asymptotic coverage.
1 Σ (E6(X,, ((x))
- -mo)
m - N-1
) 08 I E (Co(X,, Yi) - (C,(x,,((x,)) -3)
-
Theorem 4.1 (Asymptotic coverage). For a E (0,1), let Co CEARER be the FAB-PPI confidence region (21) under the Gaussian prior (22) the horseshoe prior (23). Then, under Assumption 3.
0 5.6 + (1-1)--2 m
Ta- (m)) f mg +
3 VN
21 (a-
R (oo) FAB-CR( Do; Tol: (: Fn), :n), 86,
lim Pr(O" E CFARPP) (in(n.N)++9)
C 2 21-1
P A + -o öze (2e:ã0, Tr)
end for
The proof of Theorem 4.1 crucially relies on showing exact asymptotic coverage of the FAB CR R ( (Ao). While the latter holds for both priors introduced in the previous sec- tions, the two limits behave very differently. In particular, as discussed in Remark S3.4, the volume of RE (Ae) van- ishes asymptotically under THs, while it does not under TN.
Outputs: estimator @RABER arg mine mo A Agabee
and CR Ca
CEABPP = 0 0 R2y REABEE (oo) + Too)
by the rectifier Ag. Such a property may be encoded in ro(Ao:Tn) by choosing a prior that concentrates around zero. As mentioned in Section 3.3, the FAB construction of REABEE (Ao) will exhibit smaller volume compared to the classical CI, and hence result in downstream efficiency gains over standard PPI, if the true rectifier Ae is likely under To. In particular, the prior scale Tn controls the size of the potential efficiency gains and losses of FAB-PPI over PPI: the smaller Tne the more the resulting CR will shrink (resp. grow) when As = 0 (resp. |مم| >> 0). Experimen- tally, we find that the choice Tn = 0g results in parameter- free approach that strikes good compromise. More general choices of Tn are briefly mentioned in Section 6.
The behaviour of Rg READ2P (Ån) under the two priors also differs for large values of observed Ag. In case of increas- ing disagreement between the prior and the data, Gaussian FAB confidence regions are known to become arbitrarily large (Yu & Hoff, 2018). On the other hand, thanks to its power-law tails, the horseshoe results in confidence regions that revert to the corresponding standard CI (Cortinovis & Caron, 2024). Here, we state the implication of this prop- erty on FAB-PPI informally, and provide a formal proof in Section $3.2.
Proposition 4.2 (Robustness under the horseshoe, informal). For a E (0, 1). let CFARF and C. denote, respectively, the FAB-PPI confidence region (21) under the horseshoe prior (23) and the standard CLT-based PPI CI for 0, both viewed as functions of Dg. If Aol > 0, then
A seemingly natural proposal for To that meets the require- ments above is the Gaussian prior
TN TN (oo ō0) = ~\ 0,
(22)
CFABPD a
CFP
However, as we will discuss in Section 4.4, के exhibits undesirable properties for FAB-PPI, Instead, we propose to use the horseshoe prior (Carvalho et 2010)
In practice, this means that, in the presence of heavily biased predictors, FAB-PP with the horseshoe prior reverts to standard PPI. In sense, this represents form of robustness to prior misspecification of FAB-PPI under the horseshoe.
THI 00: /
(v; L)dv, (23)
where Cv 0, 1) denotes the pdf of the half-Cauchy dis- tribution with location parameter and scale parameter 1. In the case of mus, the choice of scaling 7, = oo is further
Overall, Remark $3.4 and Proposition 4.2 provide strong support for preferring THs over TT within the FAB-PP framework.
5

---

FAB-PPI
4.5. FA P-P-PPI for Mean Estimation
eliminating concerns about the loss of tightness in the CI on 6' due to the Minkowski sum in Equation (21). In all experi- ments, we check empirically that N is large enough to make this assumption by monitoring the coverage of the resulting intervals against both the nominal level 1 - a and the cover- age of PPI intervals that also consider the uncertainty in mg (denoted with PPI (full) and PPI++ (full) in Section S6).
To provide a concrete example, a specialised version of Al- gorithm I under the squared loss is derived in Section S2.1. Here, we briefly discuss the differences between the FAB-PPI mean estimator and its standard PPI counterpart, as well as the asymptotic behaviour of the former. Under the squared loss, the rectifier A := Ag does not depend on 0 and the FAB-PPI estimator @FA corresponding to the chosen estimator A (PPI or +PI++) is given by
5.1. Synthetic Data
The simulated experiments below have a common struc- ture. We sample two datasets, n labelled observations {(,,, Y)) iid from P and N unlabelled observations (X.), iid from Px. We use a prediction rule f to obtain predictions (J(X,)) and (J(X:)) We apply the dif- ferent procedures to obtain estimates and 1 - a confidence regions for the mean 0 = E[y]. For all experiments, we set a = 0.1 and report the average mean squared error (MSE), interval volume, and coverage over 1000 repetitions.
@RADPP_ = 0 - 0² (A;ō, Ta)
(24)
N
= - n i=1
f(x,)
Σ 1=1
((X,))
N
- 02 (A;35T mm
where 82 is an estimator of var vār 0 is the PPI estimator corresponding to A, and X is set either to one (PPI) or (14) +PPI++), In both cases, the estimator @EABEP takes the form
sid Biased Predictions. We sample X; ~ N( 1) and Y; sēd X; +e; with E; ~ NN(,, 1), so that 0+ = E[y]= = 0. The pre- diction rule is defined as fX,, = X, where 7 E R. For this choice, the bias of ʃ is controlled by 7, since MSE(/) = +1. For this experiment, we assume that N is infinite, set n = 200, and vary 7 between 1.5 and 1.5. Figure shows the average interval volume as function of y for classical inference, PPI++, and FAB-PPI-+ with both horseshoe and a Gaussian prior. Results for the non power-tuned meth-
Classic Estimator+ PPI correction + Bayes correction,
where the last component depends on the chosen prior. The following proposition, proved in Section S3.3, further differentiates between the priors presented in Section 4.3 in favour of the horseshoe.
Proposition 4.3 (Consistency of FAB-PPI mean estimators). Let 0rASPP and @ZARPP be the FAB-PPI estimators (24) under the horseshoe (23) and Gaussian (22) priors, respectively. If the PPI estimator 9 is consistent estimator of then 0R2E22 is consistent estimator of 0*, while OgABEP is not. HS
0.5
0.4
Intuitively, this is due to the fact that the influence of THS vanishes asymptotically, while for og it does not.
volume
0.3
5. Experiments
0.2
We compare FAB-PPI and power-tuned FAB-PPI (FAB- PPI++) to classical inference, PPI and power-tuned PPI (PPI++) on both synthetic and real estimation problems. For FAB-PPI, we use (HS) and (N) to indicate the use of the horseshoe and Gaussian priors defined in Section 4.3. As already mentioned, PPI is motivated by settings in which labelled data are scarce, while unlabelled data are abundant. Moreover, the application of FAB to PPI specifically targets the estimation of the rectifier Ag. For these reasons, we choose focus on cases where N > n is large enough to rule out any uncertainty in the measure of fit ma. which we estimate using the sample mean Ma (9). As a result of this, given a 1 - ô confidence interval (FAB or not) R. for As. the corresponding 1 - a CI for 8* is obtained simply by setting& = a and shifting Rs by mg. This simplification al- lows us to evaluate the direct effect of FAB on the procedure,
0
1
classical
FAB-PPI++ (HS)
PP1++
PAB- (N)
Figure 1. Biased predictions study. The panel shows the average CI volume as the bias level y varies.
ods, as well as MSE and coverage plots, are reported in Fig- ure S6. Except for the version with the Gaussian prior, all the PPI procedures outperform classical inference for every bias level 7, but the behaviour exhibited by PPI++ deserves attention, as its CI volume is approximately constant across values of y. This is due to the fact that, since N is taken to be infinite and n is fairly large, À ~ cov(Y, = 1 and the rectifier is accurately estimated with similar variance
6

---

FAB-PPI
across all values of y. On the other hand, the CI volume for the FAB-PPI methods varies greatly with 7. When the bias is small (7 = 0), the observed rectifier has a value close to O, leading to smaller Cls. As the bias increases, the volume of the confidence intervals grows, until it surpasses that of the PPI intervals. At this point, the two FAB-PPI procedures be- have differently: the volume of the Gaussian intervals grows without bound, whereas the horseshoe intervals eventually revert to the PPI ones. This example clearly shows that FAB- PPI with horseshoe prior allows to obtain smaller CIs when the predictions are good, while ensuring robustness as the quality of the predictions decreases (Proposition 4.2).
As a result of this, regardless of the value of À used, any additional shrinkage performed on the rectifier by FAB-PPI is beneficial. This example shows that FAB-PPI++ retains the benefits of power tuning, while also taking advantage of the adaptive shrinkage provided by the FAB procedure.
5.2. Real Data
We consider several estimation experiments using the datasets presented in Angelopoulos et al. (2023a) and briefly described in Section S5.1. Each dataset comes with covari- ate/label/prediicionn triples {X,, Y., f(x)) which we randomly split into two subsets with n labelled and N n unlabelled observations, for varying values of For all experiments and methods, we report the average estimation MSE, CI volume and coverage across multiple repetitions.
Noisy Predictions. We consider the mean estimation ex-
ample of Angelopoulos et al. (2023b, $7.1.1), which does not involve any covariate X. We sample Y; kid (O 1), so that 0* = [[[ = 0. The prediction rule is defined as sid
We begin with four experiments, where the machine learning predictions provided are of high quality, and whose goals are as follows. Two of them are mean estimation tasks per- formed on the GALAXIES and FOREST datasets. The third one, performed on the ALPHAFOLD dataset, is an odds ratio estimation task, for which the construction of confidence in- tervals also indirectly involves mean estimation as detailed in Section S5.1. The fourth one, involving the HEALTII- CARE dataset, is a logistic regression task. Figure 3 shows the results for classical inference, +PI++ and FAB-PPI++ applied the datasets involving mean estimation.
((x,) = Y, oye,, where E; ~ N(00 1) and σy is suc- cessively set to 0.1, 1, and 2. For this experiment, we set N = = and vary n from 100 to 1000. Figure 2 shows the average interval volume as function of n for the different methods as the noise level ay varies, while similar plots for the MSE and coverage are reported in Figure S7. In this
Qy = 0.1
Ty = 1.0
ay = 2.0
The mean estimation results for the non power-tuned meth- ods are reported in Figure S8, whereas the ones for the logistic regression experiment are reported in Figure $11. In all cases, TABB-PP/FAA--PII- outperform classical in- ference and the corresponding PPI methods, both in terms of MSE and CI volume, while achieving comparable coverage. These examples suggest that the quality of the predictions of existing machine learning models on several real datasets may fall into the regime where the adaptive shrinkage pro- vided by the FAB framework leads to a further improvement over standard PPI. In these settings, as the predictions are good, FAB-PPI under the horseshoe and Gaussian priors exhibit similar gains, as already seen in Figure 1.
volume
0.4
0.2
0.0
500
1000
500
1000
500
1000
classical
PPI
PPI
PAR-PPI (HS)
FAR-PPI++ (HS)
Figure 2.
Noisy predictions study. The left, middle and right panels show the average CI volume for noise levels dy = 0.1, 1,2. 2.
case, the effect of power tuning matches the observations of Angelopoulos et al. (2023b): as the noise level increases, À decreases and less weight is given to the predicted labels. When the noise is small, all PPI procedures perform simi- larly, and much better than classical inference. When the noise is large, the power-tuned procedures perform similarly to or better than classical inference, whereas the non-tuned alternatives lose ground. At the intermediate noise level, the power-tuned methods clearly outperform the other base- lines. Crucially, FAB-PPI outperforms the PPI counterpart at all noise levels, with FAB- + being the best performer overall. This is because, in this setting, while predictions exhibit increasing variance with ay, they remain unbiased.
However, the same is not true in the presence of bad pre- dictions. For instance, Figure S12 shows the results of quantile estimation experiment on the GENES dataset, where predictions are heavily biased. In this case, the behaviour of the FAB-PPI methods under the horseshoe and Gaussian priors differs significantly: the former matches the perfor- mance of the PPI methods, which outperform classical in- ference, whereas the latter leads to much larger MSE and Cls. As previously discussed, such desirable behaviour of FAB-PPI under the horseshoe prior is due to its robustness against large bias levels (Proposition 4.2). Similarly. Fig- ure S13 reports the results of a linear regression experiment on the CENSUS dataset. For one of the two parameters con-
7

---

FAB-PPI
AL.PHAFOLDD (N 10802)
FOREST (N 1596)
GALAXIES (N 16743)
0.004
0.002
mn8e
0.002
0.001
0.000
0.000
10
0.15
volume
0.10
5
0.1
0.05
1.0
1.0
1.0
coverage
0.8
0.8
0.8
0.6
0.6
0.6
500
1000
1500
200
400
500
1000
classical
PPT++
FAB- (HS)
FAB-PPI++ (N)
Figure Real data mean estimation study. The left, middle, and right panels correspond to ALPHAFOLD, GALAXIES. and FOREST datasets. The top, middle, and bottom rows show average MSF, CI volume, and CI coverage over 1000 repetitions for 0.1.
sidered (panel (a)), FAB-PPI underperforms the alternatives under both priors for small n. However, as increases, the performance under the horseshoe prior improves and even- tually matches that of the PPI methods, while the Gaussian prior does not. This example shows another facet of the horseshoe's robustness: even for moderate bias levels, as the available labelled sample size grows, disagreements be- tween the prior and the data become apparent (i.e. var( (Ag) decreases), eventually leading Proposition 4.2 to take ef- fect.
sians models share these properties. For example, the family of priors with beta prime (aka inverted beta) prior over the variance (Polson & Scott, 2012), which includes the horse- shoe, ceammaallxxonnniil-eee -aammaa (Griffin & Brown, 2011) and other robust priors (Berger, 1980; Strawderman, 1971) as special cases, shares the same three properties. other hand, some other standard priors such as the Laplace prior (Park & Casella, 2008) normal-gamma prior (Caron & Doucet, 2008; Griffin & Brown, 2010) do not have power- law tails and therefore do not offer the same robustness guarantees. Other priors, such as the Student-t, lack an an- alytical expression for (g), therefore requiring additional numerical approximation be applied to FAB-PPI.
6. Discussion and Extensions
We proposed FAB-PPI as a Bayes-informed method sig- nificantly improve the performance of PPI in the presence of high high-quaitty predictions. In doing so, showed that the horseshoc represents sensible default prior for -PII, contrary to the seemingly natural choice of a Gaussian prior. However, several options may be worth exploring.
Furthermore, we used the scale of the noise in the genera- tive model the scale for both the horseshoe and Gaussian priors, this allows us obtain simple, parameter-fiee approach, which generally performs well. Alternatively, one could consider prior scale of na, where is a hyperparameter to be tuned using a validation set. However, in the case of the horseshoe, this renders the marginal likelihood intractable. While using rescaled horseshoe prior for FAB-PPI remains feasible through numerical integration, as shown in Figure S10, this increases the
In particular, the horseshoe prior was chosen due to its pop- ularity and key properties: (i) its spike at zero (ii) plwer-law tails and (iii) the closed-form expression for the marginal density Tty). However, many other ecale-mixture of Gaus-
8

---

FAB-PPI
computational cost of the method. By contrast, rescaled Gaussian prior would encounter this issue. Furthermore, we conjecture that choosing scale that does not depend on may resolve the inconsistency of the estimator based on a Gaussian prior, which was discussed in Section 4.5.
modifications a proteome-wide scale. PLoS biology, 2065)ee000666, 2022.
Brown, L. D., Casella, G., and G. Hwang. J. Optimal con- fidence sets, bioequivalence, and the limacon of Pascal. Journal of the American Statistical Association, 90(431); 880-889, 1995.
As a potential drawback, FAB-PPI shares the computational limitations of the PPI approach (Angelopoulos al., 2023a), which are discussed in Angelopoulos et al. (2023b). In particular, except for special cases such mean estimation and linear regression, the method requires evaluating mg + AS over grid of values of 0. This can be computationally expensive, especially in high-dimensional settings.
Bullock, E. L., Woodcock, C. E., Souza Jr, C., and Olofs- son, P. Satellite- based estimates reveal widespread forest degradation in the Amazon. Global Change Biology, 26 (5):956--296,, 2020.
Caron, F. and Doucet, A. Sparse Bayesian nonparamet- ric regression. In Proceedings of the 25th International Conference on Machine Learning, pp. 88-95, 2008
Supplementary Material and Code. The supplementary material contains additional background, proofs, and exper- iments. All sections, figures, and equations in the supple- mentary material are prefixed with 'S' for clarity. Code for reproducing the experiments is available at ht https: //github.
Carvalho, C. M., Polson, N. G., and Scott, G. The horse- shoe estimator for sparse signals. Biometrika, 97(2): 465-480 2010.
ffbb-ppi..
Cortinovis, S. and Caron, F. Bayes-assisted confidence regions: Focal point estimator and -nnfuee-nfflunce priors. arXiv preprint arXiv:2410.2016% 2024.
Acknowledgements
Stefano Cortinovis is supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Ma- chine Learning (EP/S023151/I).
Efron, Tweedie's formula and selection bias. Journal of the American Statistical Association, 106(496): 1602- 1614, 2011.
Impact Statement
Farchione, D. and Kabaila, P. Confidence intervals for the normal mean utilizing prior information. Statistics & Probability Lettess 78(9):009--1100 2008
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fisch, A., Maynez, J., Hofer, R. A., Dhingra, B., Globerson, A., and Cohen, W. W. Stratified destiction-poweree infer- ence for hybrid language model evaluation. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024.
References
Andrews, D. and Mallows, C. Scale mixtures of normal distributions. Journal of the Royal Statistical Society: Series (Methodological), (6(1)999-102, 1974.
Ghosh, J. K. On the relation among shortest confidence in- tervals of different types. Calcutta Statistical Association Bulletin, 10(4): 147-152, 1961.
Angelopoulos, A. N., Bates, S., Fannjiang. C., Jordan, M. I., and Zmic, T. Prediction Prediction-powered inference. Science, 382 (6671):669-677) 2023a.
Glasserman, P. Monte Carlo Methods in Financial Engi- neering. Springer, 2003.
Griffin, J. E. and Brown, J. Inference with cormal-gamme prior distributions in regression problems. Bayesian Anal- ysis, 5(1): 2010.
Angelopoulos, A. N., Duchi, J. C., and Zmic, T. PPI++ Efficient pepdictiionpowereed inference. arXiv preprint arXiv:2311.01455 2023b.
Griffin, J. E. and Brown, P. Bayesian hyper-lassos with non-convex penalization. Australian & New Zealand Journal of Statistics, 53(4):423 442, 2011.
Berger, J. A robust generalized Bayes estimator and confi- dence region for a multivariate normal mean. The Annals of Statistics, pp. 716-761, 1980.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bludau, I., Willems, S., Zeng. W.-F., Strauss, M. T., Hansen, F. M., Tanzer, M. C., Karayel, O., Schulman, B. A., and Mann, M. The structural context of posttranslational

---

FAB-PPI
Hofer, R., Maynez, J., Dhingra, B., Fisch, A., Globerson, and Cohen, W. Bayesian rediction-poweree inference. arXiv preprint arXiv:2405.06034, 2024.
Strawderman, W. E. Proper Bayes minimax estimators of the multivariate normal mean. The Annals Mathematical Statistics, 42(1):385-388, 1971.
Hoff, P. Bayes-optimal prediction with frequentist coverage control. Bernoulli, 29(2):901-923 2023.
Vaishnav, E. D., de Boer, G., Molinet, J., Yassour, M., Fan, L., Adiconis, X., Thompson, A., Levin, J. Z., Cubillos, F. A., and Regev, A. The evolution, evolvability and engineering of gene regulatory DNA. Nature, 603 (7901))4455 463, 2022.
Hoff, P. and Yu, C. Exact adaptive confidence intervals for linear regression coefficients. Electronic Journal of Statistics, 13:94-119, 2019.
Willett, K. W., Lintou, C. J., Bamford, S. Masters, K. L., Simmons, B. D., Casteels, K. R., Edmondson, E. M., Fortson, L. F., Kaviraj, S., Keel, W. C.. et al. Galaxy Zoo 2: detailed morphological classifications for 304 122 galaxies from the Sloan Digital Sky Survey. Monthly Notices of the Royal Astronomical Society, 435(())2835- 2860, 2013.
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Zidek, A., Potapenko, A., et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):883-88, 2021.
Kabaila, P. and Farchione, D. Confidence intervals that utilize sparsity. Stat, 11(1), 2022.
Yu, C. and Hoff, P. D. Adaptive multigroup confidence intervals with constant coverage. Biometrika, 105(2): 319-335, 2018.
Kabaila, and Giri, K. Further properties of frequentist confidence intervals in regression that utilize uncertain prior information. Australian & New Zealand Journal of Statistics, 5503)2259-270, 2013.
Zhang, A., Brown, L. D., and Cai, T. T. Semi-supervised inference: general theory and estimation of means. The Annals of Statistics, 47(5):2538-2566 2019.
Park, T. and Casella, G. The Bayesian lasso. Journal of the American Statistical Association, 103(482)6611-66, 2008.
Zrnic and Candès, E. J. Active statistical inference. In Proceedings of the 4 International Conference
Machine Learning, pp. 62993-63010, 2024a.
Piironen, J. and Vehtari, A. On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pp. 905-99999 913, 2017.
Zmic, and Candès, ence. Proceedings of the National Academy of Sciences, 121(15))e2322288111,, 2024b.
wwwp-ppediiiioopoeeeeeeee infer-
Polson, N. G. and Scott, On the half-Cauche prior for global scale parameter. Bayesian Analysis, 7(4):887-902, 2012.
Pratt, J. W. Length of confidence intervals. Journal of the American Statistical Association, 56(295):549-577, 1961.
Pratt, J. W. Shorter confidence intervals for the mean normal distribution with known variance. The Annals of Mathematical Statisties, pp. 574-586, 1963.
Puza, B. and O'Neill, T. Interval estimation via tail func- tions. Canadian Journal of Statistics, 34(2):299-310 2006.
Robins, J. M. and Rotnitzky, A. Semiparametric efficiency in multivariate regression models with missing data. Jour- nal of the American Statistical Association, 90(429):122- 129, 1995.
Slater, J. Conftuent funcecoooerricccnnciioo Cambridge University Press, 1960.
10

---

FAB-PPI
S1. Additional Background Material
S1.1. Horseshoe Prior
Consider the Gaussian likelihood model
Y Y 3 N(8,
with standard deviation > 0 and mean parameter 3 E IR. The horseshoe prior (Carvalho et al., 2010) with density whs can be represented as a scale mixture of normals (Andrews & Mallows, 1974)
3 2 1(0,1/0000 リ~ C+ (0,1).
(S25)
(S26)
where 1) 0 and C+ (0, 1) is the half-Cauchy distribution with location parameter O and scale parameter 1. Throughout this section and the main text, assume 77 = 1. The rationale for this choice, along with discussion of the general case v & 1, is provided at the end of this section.
The marginal likelihood is given by
/
(1))
N(y
3,0²)
3,0²)THS(3)d
1
1
1 p(v)dv
2TO
v1 1 va
2
2000
22)3/2 dv.
TV2T0²
Using the change of variable u 1452, we obtain
2v2
(y)
20
(1
u)
1/2du
TT TV
y²
1
TV 2
2'
20
where 1 F1 is (Kummer's) s) confluent hypergeometric function of the first kind, with integral representation
(a, = [(a)T(b a) S.
r(6)
c0t1a2(11 1)5-a-1dd
Alternatively, the marginal can be expressed in function of the imaginary error function (erfi) or Dawson function (aka Dawson integral) as
erfi (eli(ly/// /llll///ee/ /222)
Tイy)
e
(20=)
TV202
1ul/(v20*
2
D
3/2
v 2/0
where Dawson's ''s function is defined
D(z) = =2 f(dd d dt.
The marginal likelihood exhibits power-law tails
Tly) C yy
as yl + x
for some constant C > 0. Let f(y) = log (v) denote the log-marginal likelihood. Kummer's function has the derivative
d
- 11
(a,

---

FAB-PPI
follows that
('(y) 7'(1)
Tty)
2 y F
2, 2 20²
y²2
0
iFi
(,,2
1, 3 yf2
2n²
Applying Tweedie's formula (Efron, 2011), we obtaín the posterior mean
B v] y + o² e o) (1 (1-x(y))y,
(S27)
where the shrinkage function k(y) (0,1) is given by
2 Fi
2, 2.
5
x²
2
<(y)
F
1, 3. y²
2n
Using the asymptotic expansion (Slater, 1960, Chapter 4, Eq. (4.1.))
Fia b, -2) = [(b)
T(b - a)
as + we find
20
<(V)
y²
20
E B v] yl = -
lyl
|y| -
The horseshoe prior THS has two key properties: infinite spike zero, inducing strong shrinkage near y 0. and Cauchy-like tails, ensuring that strong signals remain largely unshrunk (s(y) - 0 and EE This is illustrated in Figure $4.
y] yl + 0 as |y] - oc).
1.00
0.75
0.50
0.25
0.00
10
0
10
9
Figure S4. Shrinkage function x(y) for the horseshoe prior when o² 0.1.
Remark S1.1 (Parameterisation) In this section and in the main text, focused on the specific parameterisation 1. For general 77, similar expressions can be derived for the marginal likelihood and posterior mean, replacing Kummer's F function with the more general degenerate hypergeometric function of two variables, Φ (see (Carvalho et al., 2010, Equations (4) in the main text and (AI) in the appendix)). While Kummer's Fi function is implemented in many standard scientific libraries, such as SciPy, 4, not. Consequently, computing the marginal likelihood when 17 * 1 requires numerical integration. Since the evaluation of the marginal likelihood is crucial to our approach, it is therefore reasonable to set 7 here.
12

---

FAB-PPI
S1.2. FAB Framework
In this section, we provide additional background on the FAB framework (Pratt, 1961; 1963; Yu & Hoff, 2018).
Let Y B ((8, o²) with some prior (B). Denote by a()) SR pvy B) ( the corresponding marginal likelihood. For (0, 1), let Ca be the confidence procedure that solves the constrained optimisation problem
C. = arg min ((v)
C.
under the constraints Pr(3 C. (Y) 18 8) 1 - for all fixed B²
where vol(Ca (v)) Sprec.(w) d3' is the volume of (v) and
E[voI(C. (X)) vol(C, (y)((()))))
(S28)
JR
the expected volume under the marginal distribution (1). By the Ghosh-Pratt identity (Ghosh, 1961; Pratt, 1961),
E[vol(Ca / vol(C.
/ J.J. (1)'
Pr(B's E Ca(()))
That is, minimising ((vo((C.())) is equivalent to minimising Pr(B' E CY)) for each 3r R. Define the acceptance region
Aa(3') (v 18 E Ca())-
The constrained optimisation problem above then reduces to solving, for each B",
(8 = max Pr(Y & (Q)))
An
such that Pr(Y $ A. = 3¹) a.
The term Pr(Y & (B')) may be interpreted as the power of a size-a test
Ho 3 = B' vs H1 3
where Y B N(B, By the Neyman -Parsson lemma, the most powerful test is of the form
Aa(3') {
T(y)
S ka ka(8))
p(y (²)
where ki, (3") is such that Pr(Y E (3) 8 = 8¹ = a. The acceptance region is interval (A,(8'), (3))
(Cortinovis & Caron, 2024, Theorem 3.3). Defining
Wa (8') Φ A. (3) 3'
the confidence region is given by
C. (v) = ( AA)3) - - -21--aa 122 (9') S y S B +031-a(1-w. Aa()))
(S29)
The function w., (8') E [0, 1] is called the spending functtinn or tail function (Puza & O'Neill, 2006; Yu & Hoff, 2018), and
represents the proportion of the rejection budget allocated to the left tail of the acceptance interval [A, (B'), (5)).
The spending function w, satisfies several key properties, which will be useful for asymptotic analysis. Most of these originate from Cortinovis & Caron (2024), Under mild assumptions the prior, satisfied for the models considered in this paper, (8) is continuous in B. If the prior ma is symmetric around zero, we have
W.C (-3') 1 1-w,(.. w.. (8').
(S30)
13

---

FAB-PPI
6
0.75
10
10x
Co
volume
0.25
-10
3
0.00
B
91
classical
N 1)
HS(1)
Figure S5. Comparison of the FAB procedures under a Gaussian (7² 1) and a horseshoe (1) 1) priors when o² and a = 0.1.
Additionally, if the prior mB( := xu(3; o) admits o as scale parameter, writing (a; for the corresponding tail function, we have
Wa (B:0) wa (
(S31)
We now describe other properties of the spending function in the case of Gaussian prior and of prior with power-law tails, such the horseshoe.
Proposition S1.2 (FAB with Gaussian prior (Pratt, 1963; Yu & Hoff, 2018)). If the prior no(3) 0, is Gaussian, the spending function is given by w. (3) Ia
28 07
where gn. (0, 1) lK is the one-to-one function
ga(w) x- -- (aw) ④ (a(1 w)).
(S32)
Wa strictly increasing and
lim wa( 1. $
Proposition S1.3 (FAB with prior with pawer-law tails (Cortinovis & Caron, 2024, Lemma S1.1)). Let To(3: be symmetric prior on 3 such that the marginal density rly) has power-law tails, i.e.
Tty)
Colul - as yy + 00
for some constant Co and some exponent 8 > 1. Then, wa (3; o) is bounded away from and 1, and
lim wa( lim (a B1000
---000
2
The difference between the spending functions of the two priors greatly affects the resulting FAB confidence regions. In particular, while both priors lead to confidence regions that are shorter than the classical one when the observed y is close to zero, their behaviour differs as the disagreement between the prior and the data increases. particular, the FAB confidence regions under the Gaussian prior become unbounded as |y] grows, while horseshoe prior leads to confidence regions that eventually revert to the classical confidence interval. This illustrated in Figure $5.
S2. Derivations
S2.1. FAB-PPI for Mean Estimation
Here, we outline the steps to derive the FAB-PPI mean estimator presented in Equation (24), as well as the corresponding FAB-PPI confidence region.
The convex loss function that corresponds to estimating 8" = EE{] is the squared loss Co y) = 1 (0 - -y) In this case, the subgradient of Lo with respect to 0 is given by (x y) y. As result of this, the measure of fit my and the
14

---

FAB-PPI
rectifier Ag take the form
me [((](X, = -I/(X))
A (X - (xx) -
-Y.
In particular, under the squared loss, the rectifier Ag does not depend on 0, and we indicate this by dropping the subscript 0 and writing A := Ag.
In order to apply FAB-PPI to this setting, we follow the steps outlined in Section 4. In particular, we use the sample mean of unlabelled data (9) as the estimator fgg of ma.
N N f(X)),
ma
E Cx = N
1
and either the sample mean (10) the control variate estimator (12) as the estimator Â of A, in PPI and PPI++ + respectively. To avoid repetitions, in this section we write Á as the following general control variate estimator with tuning parameter À E R,
Â - Cxx, -a (1 Σ Co (o1,,,
-mo
(Cb(2,)))
Ax, + (A - 1)ma
1 N N - + N EXx))
= + 1 -(x))~~
The sample mean estimator (10) and the control variate estimator (12) under the squared loss are recovered by setting À to 1 and Ā as in Equation (14), respectively. From this, the standard PPI mean estimators (11) and (13) obtained by solving the equation ma + Â 0 for 0. Instead, we first define the Bayes-assisted estimator (20) under the chosen prior xo To(A; Tx),
A FABPF = A + 8 ( (A:
where 822 is an estimator of (A) and (2; o, 7) is the derivative of the log-marginal likelihood of Gaussian likelihood model with mean and variance x² under the prior To(A, r). Then, the FAB-PPI mean estimator @RADPF under Tg is given by the solution to the equation
mg + A =0
in 6, that takes the form
1 N N
f
ARPP
Y
-
1
f(X.)
Σ f(X)) - -62 (20(Ai005T))
1=1
which matches the expression in Equation (24). Furthermore, by recognising that the first two terms in the above expression match (13), we can alternatively write the FAB-PPI mean estimator as
FARPP -- (A:O,TT)
where 0 the corresponding standard PPI mean estimator.
Given E (0, 1), construct the FAB-PP -PI confidence region CEARPP for the mean described in Section 4.2. In particular, let T (ma) denote a standard 1 - (a 5) confidence interval for mø,
Ta-s(ma) = im G - 21- 6)/2] @ Ør + 2 21 8)/2
15

---

FAB-PPI
where 01 = + Σ ((;; for conciseness, and (2 is estimator of var( Then, we apply the FAB framework under the prior (A; Tn) obtain confidence region for A,
R A FAB- TAB-CR(A; nol: :7,),,
where, again, の² is an estimator of var Á). Finally, to avoid making assumptions on the specific form of R3 (Å), we use jinf Rg sup (R. (Raaeee)
}] 2 R (A) in the definition of CR (21) to obtain the FAB-PPI interval
CEAEPE
010E E
0 - 0f + 5. 21 (1- 51/2 +
), supp(R)
01I0E E
0 - 0f  21 (a-51/2 + inf ) 01 + 5 (a (a-8)/2 61/2 + sup(Re
}}}
Φ - of 21-(0 (a- 62 sup (FASP?) ) lr + + (1- (R5 ]]
Algorithm summarises the FAB-PPI approach under the squared loss, where & defined for notational convenience and the corresponding sample variances are used as (a+)² ) and 0².
Algorithm 2 FAB-PPI for mean estimation
Input: labelled {(X,, Yo) unlabelled (X,) predictor f' prior :o(: Tn), error levels a, 5
Set λ~ (FAB-Pi) PPD) or estimate X from data (FAB-PPI++) using Equation (14)
01 - 5/2 Σラ= f(2))
1. Ei=, (xx( -Y)
Â f -2 1)05
(0 < N(N-1) E) 8r)2
6 = - Ei= (TS(X.) - -Y, Y, -y²
σ2 102 + (05)2
a
R READEP + FAB-CR (; (T;n),0,)))
Outputs: estimator JEaSPE Ø1 7 (2(((A0,7.)) and CR CEaEEe 101 10--2 -6 21 21-(2-8)) 6)/2 sup(R REABEE) 01 +
al 21- 51/2
iR
S3. Proofs
Some of the results discussed in this section, such as Lemma $3.2 and Corollary S3.5, are concerned with the convergence of closed sets with respect to the Hausdorff distance, which we recall here for completeness. Given two closed subsets C₁ and C2 of R. their Hausdorff distance du defined as
(C, C2) max sup inf jx - vl. sup inf 1x - u1 } ( yEC₂
yECa xEC1
Then, for a collection of closed subsets (Ci(y)) and a closed subset C2 of R. (Ci converges in Hausdorff distance to C2 if limy-+x (H)) C2) = 0. particular, if C2 = [a, bj is a closed interval for some a b, limy- du (c),, C2) and only for all E (0, bad)) ), there exists Yo such that (a+ b-c Ci(v) S ba-c, for y > Vo- In the sequel, we write lim C₁ (1) C2 for limy-+oc du(C (C)()), C2) 0.
S3.1. Theorem 4.1 Asymptotic Coverage FAB-PPI under the Gaussian and Horseshoe Priors
Under the prior No (:; Go), the FAB confidence region for the rectifier Ag
R
AEPP (B:; Ōo) (Av Ao - (7921-6 S Eg < As + -=:1-611-w((0080))
(S33)
where ws(:; )x) is the FAB spending function. The proof of asymptotic coverage is organised follows. First, show that READPP (Ao; 3o) is asymptotically a 5 confidence interval. This established via Lemma $3.1 1 for the Gaussian prior,
16

---

FAB-PPI
and via Lemma S3.2 for the horseshoe prior. This result is then combined with the asymptotic coverage of the standard sample mean estimator for mo conclude the asymptotic coverage of the FAB-PPI estimator of 0*.
We first prove the following lemma for the Gaussian prior, demonstrating that the rectifier has the correct asymptotic coverage.
Lemma S3.1. Let Ag be consistent estimator of Ag such that a CLT holds for Do i.e.
-
+ 1))
0o
as min(n, N) + corresponding 5 FAB confidence region
where var(Ag)
3 almost surely. Let no(;; Go) be the Gaussian prior (22) for Do and consider the
R RADEP (ee;00) = FAB-CR (
Then
lim inf min(n, Pr(Ao E (88Epe(A0: Ao) 2 1 5.
(S34)
Proof. Using Proposition $1.2, for any >> 0, 0,
2r
98 (g5 (2x/o))
go (w's (wss((;0)) ();
(12 21 -8ws(2;0)
and
0 021-6(1-ws((ro)) = 022 1021-8(1-15 021--((1
= 2x + 021-áw,(4/0;;1)
The FAB confidence region (S33) can therefore be written
REABEP (es [s: (A > 0 As 3821-ow, (A - -o S 3Ag + T0gi Bw, (Am())))
(A. 0 3A, -02=--(1-
S Ao S Do +
U {0] A < 0621-6/2)
Consider first A9 0. By the CLT, Pr(O E R REABPP (Ag: As 0) Pr( S 5821-8/2
Do =0) + 8 as
min(n, N) + Additionally, for any 0,
2 31-bwa(2;00) - 21
wal 21 21-6(1wel(-z20) = + 21-5
almost surely as min(n,
It follows from Equation (S33) that, for any E (0, 21-)), there exist No such that for all n, N with min(n, N) 2 No. the FAB confidence region R REASPE (A);e)) contains the set
Sx( { Ao > 0 0 60(21-0 So 33 +0.
u u <0 30g - 60(21-8- - e) s As s A 60(21-8- e) -)
u u)
[ool S 60011-122)
For any fixed Ag 0,
AS Ao
24,
Pr(As E Ss((A: Go)) Pr ( -
-c) -)
<
+
21-5 -c)
(S35)
ō9
ō
17

---

FAB-PPI
Noting that oy 21-6 é + a.s. min(n, N) + oo, obtain that
lim inf (n, Pr(Ag E RgAEE (oo: σ8) A1) 2 lim inf min(, Pr(Ao E Ss(De: Ao) 2 1-6. (S36)
The proof proceeds similarly for Ao < O.
Lemma S3.2. Let Ag be consistent estimator of Ag such that a CLT holds for De i.e.
Ao Ao
N(
as min(n, + oc, where 2a/var(Ao) 1 almost surely. Let the prior Tu(-; :e) on Dg be the horseshoe prior (23) with scale parameter oe and consider the corresponding FAB confidence region
R5 R (Bo;ão) = FAB-CR (Aoimo (:)00);;;)),0001
where wg( (0s(A);ū)) is the associated weight function. Then, for Ag F O, the confidence region R. RABBEP (Ag;0)) reverts to the classical z-interval for Dg, i.e. almost surely.
R
REARPP
(o; Ao 11-21- 21 218/21
lim min(n,N) -000
where the convergence with respect to the Hausdorff distance on closed subsets of R. Moreover, for any Ag E R.
limmin(r.N))ta Pr(As E RgAEee (As: ōØ) As) = 1
(S37)
Proof. Consider the case Ae * As described in Section S1.2, the spending function wy is continuous and satisfies, for any R and ø > 0,
ws (2:0)= we :1)
(S38)
Moreover, by Proposition S1.3, wx(s5) 1) takes values in (0, 1) and satisfies
lim 1s(2;1) lim wr(2; 1) wi (0: 1)
Define A(p) = - 8(1 p)) and B(p) = t 1- - ōp), where (--) is the CDF of the standard normal distribution.
Then, from Equation (S33), we have that
REAREF (Agūãa) Bo
{ VERI A(us(Gab -o: Go) < B(ws(onb + Ag; -o)3)))
ko))
00
f1 E A(ws(bb -o 1)) < + B(wa(v) 20/ 1))
Cn.N,
where the second equality follows from Equation (S38).
Assume that Ae 0, which ensures Âg/ōu almost surely min(n, N) cc. The case As 0 follows similarly.
First, we show that there exists M independent of n such that, for all n, N > 1, if a Cn,N, then + M. By the boundedness of ws(:; 1), there exists (0, )) such that ws(x; 1) E [R, 1 - π]] for all x E R. Since A(.) is decreasing,
(1 Le 1)) Z A(1 x) = -
for all v N, Pick M < c. For all a < M we have, for all n, N 2 1, き < M <cs Hence, 1b Cr,N. As result, C, [M, x0) CA,N-
A(ws(v + Ae/ō8: 1)).
18

---

FAB-PPI
Second, we show that, almost surely, wa(v A Tg 1) converges 1/2 uniformly on (M,xo) min(n, N) oo. Given
that (s;;; 1) is bounded and converges pointwise 1/2, we have that
1 2
(()) sup V22
ws(y;
++ 0
as t + oc, where g(-) is continuous and nonincreasing. As a result of this, and since Âo/ōg + oo almost surely as min(n, N) + oo, we have that
1 w5b + 2o/50; 1) - 2 -9 -ao00/o + M) - 0
sup #2M
(S39)
almost surely as min(n, N) 0o.
Lastly, we combine the previous two steps to show the almost sure Hausdorff convergence of Cn,N- The functions A(-) and B-) are continuous with A(1/2) = 882 and (1122) = 21-8/2. Hence, for every é E (0, 21 21-//2), there exists >> 0 such that
S 1 A(p) + 21 8/2 - and [(p) 21-6/2| Sc
2
By Equation (S39), there exists N1 such that, for all min(n, N) N1.
1
wall + So
sup #2M M
2
Then, uniformly for vb 2 M,
A(ws (vb + /o/ 1))
-, - - [
Blws(+) + As/ 1)) /21-6/- - 6,216 E
and, for min(n, N) N1,
S [-21-6/22 --2 + 6, 216/2 -e] S Cn,N n [ - 6, 1- + e],
which, combined with the first step, gives the desired result. Moreover, Equation (S37) then follows directly, which completes the proof.
On the other hand, in the case Ag 0, asymptotic coverage follows like in the proof of Lemma S3.1. In particular, from Equation (S33), have that
Pr(O& E R. REABPE (Ao; = Pr([aol S 6021-6/2)
thanks to the fact was (0; 1/2, and the result follows from the CLT as min(n, N) - co. While this is not necessary to show asymptotic coverage, is interesting to note that Cr.N does not converge to a deterministic limit almost surely when Ap 0. Instead, by showing continuity of the mapping Ap/ō, + Cn (An (Ān/ and again exploiting the CLT, one may prove that C,,N converges in distribution to the random FAB confidence region C(Y) Y. where C(y) is defined in Equation (S29), for a unit scale and a horseshoe prior, and ~ (O,
With the above two lemmas, we
prove Theorem 4.1, which restate here in extended form.
Theorem S3.3. Consider a convex estimation problem whose solution can be expressed as Equation (2). For all 0 E R. define Dn and ma as Section 4 and let
R3 REASPE (A0;0)) FAB-CR (Do no (-;ōa) 6),
Ta Ta-s Gg.)=
|m + of 2
19

---

FAB-PPI
22 var defined as
where
1 and
(0 var()
1 almost surely min(n, N) - Then, the FAB-PPI confidence region CE
CEADEE = ( -
( (A); + +Ta-s(me;;;))
has correct asymptotic coverage, i.e. it satisfies
lim inf Pr(O" C min
lim min(n,N)~>o
Pr(O R (a: + -Ta-o(mar;; og.))
Proof. By Lemma S3.1 (Gaussian prior) and Lemma S3.2 (horseshoe prior), REARPE (Ao:; :g. is an asymptotically valid 1 - confidence region for Aor, that is
Inf Pr(Ag- R (ao-; (g0-;0.))) 2 6. (n)
Similarly, the CLT for mg implies that
lim inf min(n.N)++30
Pr(mo- E :Ta-s(mo-:3g.)) :b.1) 2 (a 6).
Consider the event
= (Do E REABBE (Aor; {mg- E Ta- oz.)}.
By Boole's 's inequality,
lim inf Pr(E) 2 n,N-
limsup Pr([Ag- & R RAARP [g::30- }} {mg- & Ta-o(foor: ob.))) min(n,N))+ox
lim sup Pr([Ag- $ RR (Ao:;oo)))) limsup Pr((ma- & Ta-s((mos ()))) min(n.N)
N-500
8 - (a 8)
Furthermore, on the event E, we have that
0= A + mg- R (Ao+;āo.) + Ta Tamo(fo--;
where the first equality follows from Equation (2), As result of this,
lim inf Pr(O E R (Āo:;āo)) + Ta ; ;b/))
Mi
as desired.
Remark S3.4. With both the Gaussian and horseshoe priors, we obtain asymptotic coverage. However, the asymptotic confi- dence regions differ significantly. In the Gaussian case, the volume of the confidence region does not vanish asymptotically. Instead, the confidence region converges to A Ae), with volume of 3 Ael. In contrast, when using the horseshoe prior (23), we revert to the usual CLT-based confidence intervals, and the volume of the confidence region converges to zero almost surely.
S3.2. Proposition 4.2 Robustness of FAB-PPI under the Horseshoe Prior
Let Mo be the horseshoe prior (23), and consider the FAB confidence region REABPP (Ap; for Ag, as defined in Equation (S33).
We first state a corollary of Cortinovis & Caron (2024, Theorem 3.4), which follows from the power-law tails of the marginal likelihood under the horseshoe prior (see Section S1.1). The corollary states that, if Agl is very large, then the standard bas- confidence interval, (A - is recovered.
20

---

FAB-PPI
Corollary S3.5. (Cortinovis Caron (2024, Theorem 3.4)) For any a
lim
A+ix
where the convergence is with respect to the Hausdorff distance on closed subsets of E.
Define
Sa, ( Go. mg (6) = R.ADee (oo: + Ta-o(moi 06),
(S40)
where
Ta- (m=: 66) mg o 2- 2) my o 21 (1 8//2
the standard CLT-based confidence interval for mg. From Corollary S3.5, for any fixed o > 0, m E R, of > 0,
lim Sa,s(A, a, m, o (A + m) [- of (1-(0-0)/2 {= + a 21
where, again, the convergence with respect to the Hausdorff distance on closed subsets of R. Therefore, if |مم > the confidence region Sa.5 Da, oø, maz / reverts to the standard interval
A + mo + (00=1-8/2 T. (1-((-))/22)
follows that, if infore Aor| > 0, the confidence region for 0", defined as
CEASPE
{
0 E R REAREP (A:: + Ta-
reverts to the standard, CLT-based PPI confidence region
pPP
0 E
IR
-0021-612 // 17-=1-(0-6/2 S Ao + mo S 5021-6/22 + 3 2/211(0-0012
S3.3. Proposition 4.3 Consistency of FAB-PPI Mean Estimators
Below, BPP and BPP+ to distinguish between the estimators @EAPee 0, Å and & in the cases of FAB-PPI and FAB-PPI++. Then, the FAB-PPI and FAB-PP! mean estimators are given by
@ePP @P - 26 (A PP :0 _aP))
@ops TePt - (GPP1)28 (A .0P3)
where we recall that E(y; r) = log Sg. N(y: A, 0²) )T) A: T)dA, where т is scale parameter of the prior By assumption, both PPI estimators, 0er and 0²0- are strongly consistent estimators of dr, It remains to prove that
2 FF geF gep +
(S41)
(6 26 geut xer)
+0
(S42)
almost surely, as min(n, oc. For any O, we have ('y (y: (y/o0/;
Under the horseshoe prior (23), (s: 1, 1) is bounded. Therefore, (S41) and (S42) hold almost surely by sandwiching.
Under the Gaussian prior (22),
((;0,0)= = y 20
Hence, since AF A and AFF+ A almost surely, where we recall that E[f Y], we obtain
2 A ( G gap)
+
2
(GPe+)2 A 2p : gpet gep)
A
2
almost surely as N) + co. which implies that the FAB-PPI mean estimators under the Gaussian prior (22) are not consistent.
21

---

FAB-PPI
S4. Multivariate FAB-PPI
Here we extend FAB-PPI to the multivariate case, where 0, me, Ag E R4. While most methodology remains the same as in the univariate case, we now need specify a multivariate prior for 4,, for which we consider independent horseshoe priors on each dimension.
S4.1. Multivariate Bayesian PPI Estimators
As in the univariate case, we use the sample mean my as the estimator of mg. Similarly, we consider some consistent estimator Âg of Ae. such the sample mean (10), in PPI, or the control variate estimator (12). as in + Crucially, assume that multivariate CLT holds for this estimator, that is
Eg 1/12 (oo - 20) - (0,I)
as min .V co, where Eg is an estimator of cov( (o)) Again, this holds for both the PPI and PPI++ estimators (An- gelopoulos et al., 2023a;b). We consider independent priors, To ( 0o,k) for l: = 1.....d, d, on the components of is the k-th diagonal element of Eg. The multivariate FAB-PPI estimator ĀςAROP is formed by stacking the individual estimators
Ae, where 02 k
A
PAPPP 0,k
+ + (Ao.i:ō0 60.h))
for each dimension = I....,d. Importantly, note that the k-th dimension of A CASPE only depends on the k-th dimension of the observed (X, Y.) ) (X,, ((X)))) that are used to estimate Ag.k. The FAB-PPI estimator of 0* then becomes the solution, 0, to the equation
FABPE 0 RRd. mo + A
S4.2. Multivariate FAB-PPI Confidence Regions
As in the univariate case, let (fog) denote standard 1- (a- - 8) confidence interval for me. For Ae, we apply the FAB framework with independent horseshoe priors each dimension An, k and use union bound obtain ô confidence region for Ag. In particular, let REAAFE (Do,e; k..x) = FAB-CR ao mol: To.x), To,k. 8/d) be 8/d FAB confidence region for Ag, under the horseshoe prior To(- a. Then,
R5 (Āo, = {oo Do. E REARPP (Do, Go,), k 1, = I.....d Se.))
where Ào
(ō ,g,,)) is a a multivariate FAB confidence region for Ag by union bound. With this, the multivariate FAB-PPI confidence region CRABPE is given by
Gg (A......,Aeel)
C
0 0 E REABPS (An, + Ta (mo
exactly as in the univariate case. Moreover, also multivariate FAB-PPI enjoys asymptotic coverage as min(n, N) - oc. In particular, Theorem 4.1 1 can be easily extended to the multivariate case by applying union bound over the dimensions of Ag.
S5. Experimental Details
S5.1. Datasets
Here provide brief description of each dataset used for the real data experiments in Section 5.2. For additional details, the reader may refer to Angelopoulos et al. (2023a). All of the datasets were downloaded from the examples provided as part of the ppi- upi--y Py package (Angelopoulos et al., 2023b).
AlphaFold. The ALPHAFOLD dataset contains the following features for N = 10 802 protein residues analysed by Bludau et al. (2022): whether the residue is phosphorylated (Z, E {0, 13). whether the residue is part of an intrinsically disordered region (IDR, Y; (0, 1}), and the prediction of the AlphaFold model (Jumper al., 2021) for the probability Y; being
22

---

FAB-PPI
equal one (/(x)) E [0, ID. The goal is estimate the odds ratio of protein being phosphorylated and being part of an IDR, i.e.
H1/ vil
M /0/(1 -
where = Pr(YY = = 1) and p4 = Pr(Y = 1 2 = 0). Following Angelopoulos et al. (2023a), given a E (0, 1),
we construct a/2 confidence intervals Co [o, un] and C1 = [1,,u ui) for nn and μr. respectively. Then, by union bound, the interval
C
C1
co Co
cp Co. a C. E
Co
[
40
u1
ug
11
1
has coverage at least 1 - a. Note that the union bound above may result conservative confidence interval, leading to coverage significantly larger than 1 in practice, as in the left panel of Figure 3.
Forest. The FOREST dataset contains the following features for N 1 596 parcels of land in the Amazon rainforest
examined during field visits (Bullock et al., 2020): whether the parcel has been subject to deforestation (Y; E {0, }}) and the prediction of gradient-boosted tree model for the probability of Y, being equal to one (f(x;) E 1)). The goal is to estimate the fraction of Amazon rainforest lost to deforestation, i.c. l* = E(Y].
Galaxies. The GALAXIES dataset contains the following features for N = 16 743 images from the Galaxy Zoo 2 initiative (Willett et al., 2013): whether the galaxy has spiral arms (Y; E {0. 11) and the prediction of a ResNet50 model (He et al., 2016) for the probability of Y, being equal to one (f(X;) E [0, 1)). The goal is estimate the fraction of galaxies with spiral arms, i.e. EY]
Genes. The GENES dataset contains the following features for N 61 150 gene promoter sequences: the expression level of the gene induced by the promoter and the prediction of transformer model for the same quantity (Vaishnav et al., 2022). The goal is to estimate the median expression level across genes.
Census. The CENSUS dataset contains the following features for N = 091 individuals from the 2019 California census: the individual's 's age, sex, and yearly income, as well as the prediction of a gradient-booster tree model trained on the previous year's raw data for the individual's income. The goal is to estimate the ordinary least squares (OL.S) regression coefficients when regressing income on age and sex.
The HEALTHCARE dataset contains the following features for N 318215 individuals from the 2019 California census: the individual's yearly income and whether they have health insurance (Y; {0, 1}), as well as the prediction of gradient- gradient-boosted tree model trained on the previous year's 's raw data for the probability of Y, being equal to one (f(X,) E [0, 1)). The goal is to estimate the logistic regression coefficient when regressing health insurance status on income.
Healthcare.
S5.2. Implementation
Code implementing the FAB-PPI method is written in Python and made available at https:///ithui: com/ steefanocrttinon novis s //fab--pppp Comparisons with standard PPI are performed using the ppi-py package (An- gelopoulos et al., 2023b). All the experiments presented here were run locally on an Intel Core 17-11850H CPU.
S6. Additional Results
16.1. Experiments with Synthetic Data
The complete results for the experiments discussed in Section 5 are presented here. The legend names for the figures are in Section 5.
S6. I. BIASED PREDICTIONS SIMULATION STUDY
Figure S6 shows the average MSE, CI volume, and CI coverage as a function of the bias level y for the biased predictions study in Section 5.1. Compared Figure I, we include results for the non power-tuned methods, as well as for the ones that
23

---

FAB-PPI
0.03
1.0
0.4
0.02
Volume
coverage
11380
0.8
0.01
0.2
0.6
0.00
classical
PPI
PPI (full)
PPI++
PPI++ (full)
FAB-PPI (N)
FAB-PPI (HS)
FAB-PPI++ (N)
FAB-PPI+ (HS)
Figure $6. Full results for the biased predictions study. The left, middle, and right panels show the average MSE, CI volume, and CI coverage as the bias level y varies.
take into account the uncertainty in the measure of fit ma (i.e. PPI (full) and PPI++ (full)). In this example, power tuning does not play a significant role and the same conclusions as in Section 5.1 hold. In particular, standard PPI induces shorter Cls than classical inference with constant volume across bias levels. On the other hand, FAB methods induce shorter Cls when the predictions are good. As the prediction bias increases, the volume of the FAB CIs with Gaussian prior grows without bound, while the horseshoe prior eventually reverts the PPI intervals. Furthermore, the coverage plot shows that the methods tested achieve similar coverage to the nominal level and to PPI (full) and PPI++ (full).
$6.1.2. NOISY PREDICTIONS SIMULATION STUDY
Figure S7 shows the average MSE, CI volume, and CI coverage as a function of n for the values of ay considered in the noisy predictions study of Section 5.1. Compared to Figure 2, we include results for the methods that use the Gaussian prior (FAB-PPI (N) and FAB-PPI++ (N)) and those that take into account the uncertainty in the measure fit my (i.e. PPI (full) and PPI++ (full)). Like the CI volume plots the main text, the MSE plots clearly show the benefits of both power tuning and adaptive shrinkage through the horseshoe prior: as ay increases, the power-tuned methods clearly outperform the standard alternatives, while shrinkage always helps compared to standard PPI because the predictions remain unbiased. In this case, the Gaussian prior performs similarly to the horseshoe as the prediction rule is unbiased. The coverage plots confirm that all methods achieve comparable coverage across noise levels.
S6.2. Experiments with Real Data
$6.2.1. 1. MEAN ESTIMATION
Full Comparison. Figure S8 shows the average MSE, CI volume, and CI coverage function of for the three datasets considered in Section 5.2. Compared to Figure 3, we include results for the non power-tuned methods, as well as for the ones that take into account the uncertainty in the measure fit mg (i.e. PPI (full) and PPI++ (full)). The results are consistent with those presented in Section 5.2. In particular, FAB methods outperform the standard PPI alternatives and classical inference, while achieving comparable coverage. For the datasets and the values of considered, power-tuned methods perform similarly to the non-tuned ones. Among the FAB methods, the horseshoe and Gaussian priors achieve similar performance.
Example Intervals. Figure S9 shows 10 randomly chosen intervals for the classical, PPI++, +, and FAB-PPI+ methods for the three datasets considered Section 5.2 and different choices of the number of labelled observations n.
24

---

FAB-PPI
dy = 0.1
Oy = 1.0
σy = 2.0
0.010
0.04
0.010
11.00
0.005
0.005
0.02
0.000
0.000
0.00
0.3
0.3
0.6
volume
0.2
0.2
0.4
0.1
0.1
0.0
1.0
1.0
1.0
coverage
0.8
0.8
0.8
0.6
0.6
250
500
750
1000
250
500
750
1000
250
500
750
1000
classical
PPI
PPI (full)
PPI++ FAB-PPI+ (N)
PPI+ (full)
FAB-PPI (HS)
FAB-PPI (N)
FAB-PPI++ (HS)
Figure S7. Full results for the noisy predictions study. The left, middle, and right panels correspond to noise levels ay = 0.1,,,2, respectively. The top, middle, and bottom rows show average MSE, CI volume, and coverage, respectively.
Varying the Prior Scale. We repeat the mean estimation experiment the FOREST dataset while varying the scale of the horseshoe prior used for FAB-PPI++ in Section $6.2.1. In addition to the scale 7 used in the main text, we consider the sample-independent scale 1/v7i and the data-independent scale 1. As already mentioned, the computation of the FAB-PPI confidence regions under a horseshoe prior with scale other than ō involves numerical integration to compute the corresponding marginal likelihood. Figure S10 shows the average MSE, CI volume, and CI coverage for cach of these choices, as well for classical inference and PPI++ While the scale ă achieves the best performance, the other seales also provide shorter CIs than elassical inference and +PI++ In particular, the sample independent scale 1/vū results in good performance across all metrics without requiring the estimation of T.
S6.2.2. LOGISTIC REGRESSION
Figure S11 shows the average MSE, CI volume, and CI coverage as function of n for the logistic regression experiment on the HEALTHCARE dataset mentioned in Section 5.2. As mentioned in the main text, FAB methods outperform the standard PPI alternatives and classical inference, while achieving comparable coverage. Among the FAB methods, the horseshoe and Gaussian priors achieve similar performance.
25

---

FAB-PPI
ALPHAFOLD (N 10802)
FOREST (N = 1596)
GALAXIES (N = 16743)
0.004
1.5
0.002
0.003
1.0
0.002
0.001
0.5
0.001
0.0
0.000
0.000
0.20
10.0
0.15
0.15
volume
7.5
0.10
5.0
0.10
2.5
0.05
1.0
1.0
1.0
coverage
0.8
0.8
0.8
500
1000
1500
200
400
500
1000
classical FAB-PPI (HS)
PPI
PPI (full)
PPI+ FAB-PPI+ (N)
PPI++ (full)
FAB-PPI (N)
FAB-PPI++ (HS)
Figure $8. Full results for mean estimation experiment on real data. The left, middle, and right panels correspond to the ALPHAFOLD, GALAXIES, and FOREST datasets, respectively. The top, middle, and bottom show average MSF, CI volume, and CI coverage, respectively, over 1 (000 repetitions for 0.1.
$6.2.3. QUANTILE ESTIMATION
Figure S12 shows the average MSE, CI volume, and CI coverage as function of for the quantile estimation experiment on the GENES dataset mentioned in Section 5.2. The predictions contained in this dataset are highly biased, and this is reflected in the performance of the FAB-PPI methods. In particular, the Gaussian prior underperforms both classical inference and standard PPI, while the horseshoe prior achieves similar performance to standard PPI thanks to its robustness against large bias levels.
S6.2.4. LINEAR REGRESSION
Figure S13 shows the average MSE, CI volume, and CI coverage as a function of n for the linear regression experiment on the CENSUS dataset mentioned Section 5.2. More specifically, panels (a) and (b) correspond the OLS parameters associated with the age and sex covariates, respectively. On the hand, FAB- seems to perform well for the sex covariate, with similar performance between the Gaussian and horseshoe priors, and slightly improved MSE and CI volume compared to classical inference and standard PPI. On the other hand, the performance of FAB-PPI for the age covariate seems to be affected by bias in the dataset predictions. In particular, FAB-PPI under the Gaussian prior underperforms the alternatives for all n. On the other hand, while the horseshoe prior achieves worse performance than the other methods for small n, its performance improves as n grows, and it eventually matches standard PPI. This suggests that, as n increases and
26

---

FAB-PPI
ALPHAFOLD (N 10802)
FOREST (N 1596)
GALAXIES (N 16743)
72 200
50
50
777
71 250
n1 = 472
72 1500
12 500
n 1000
0.0
0.1
0.2
0.3
0.1
0.2
0.3
A^
B*
classical
PPI++
FAR-PPI++ (HS)
Figure S9. Each subfigure includes 10 randomly chosen intervals for the classical, PPI++ and FAB-PPI++ methods. The left, middle, and right panels refer the ALPHAFOLD, GALAXIES, and FOREST datasets, respectively. The top, middle, and bottom rows correspond different values of n.
var( (A)) decreases, the observed value of the rectifier increasingly considered as extreme, causing the influence from the horseshoe prior eventually vanish thanks to its robustness extreme bias levels.
27

---

FAB-PPI
FOREST (N 1596)
0.0025
0.150
0.0020
0.9
0.125
volume
cuveriage
0.0015
0.8
0.100
0.0010
0.7
0.075
0.0005
0.6
0.050
0.0000
0.5
200
400
200
400
200
400
I
classical
PP1++
FAB-PPI+ (0)
FAB-PPI+ (1/v1)
FAB-PP+++ (1)
Figure $10. Mean estimation experiment on the FOREST dataset with varying horseshoe prior scale. The left, middle, and right panels show average MSE, CI volume, and CI coverage over 100 repetitions for 0.1.
HEALTHCARE (N 318215)
x10-11
x10-5
1.0
1.0
1.0
volume
0.8
coverage
0.8
0.5
0.6
0.4
0.6
2000
4000
2000
4000
2000
4000
classical
PPI
PPI (full)
FAB-PPI (HS)
FAB-PPI
Figure S11. Logistic regression experiment on the HEALTHCARE dataset. The left, middle, and right panels show average MSE, CI volume, and CI coverage 1000 repetitions for 0.1.
28

---

FAB-PPI
GENES (N = 61150)
30
2.5
1.0
0.9
2.0
20
coverage
volumee
10.00
0.8
1.5
10
0.7
1.0
0.6
0.5
0.5
1000
2000
1000
2000
1000
2000
1
classical
PPI
PPI (full)
FAB-PI EPB-PII (HS)
FAB-PPI (N)
Figure $12. Quantile estimation experiment the GENES dataset. The left, middle, and right panels show average MSE, CI volume, and CI coverage over 100 repetitions for a 0.1.
CENSUS (N 380091)
1.0
80000
0.9
60000
volume
600
coverage
(a)
0.8
40000
400
0.7
20000
0.6
200
0.5
x 10
1.0
3
15000
0.9
coverage
volume
mnse
0.8
10000
0.7
5000
0.6
0.5
2000
4000
0
2000
4000
0
2000
4000
n
n
classical
PPI
PPI (full)
PPI++
PPI++ (full)
PAB-PPI (HS)
FAB-PPI (N)
FAB-PPI++ (HS)
FAB-PPI++ (N)
Figure S13. Linear regression experiment CENSUS dataset. The (a) (b) panels correspond the two covariates in the dataset. The left, middle, and right panels show average MSE, CI volume, and CI coverage over 1000 repetitions for a = 0.1.
29


Reviews:
[{"review_id": "oYFILVMCnK", "reviewer": "Reviewer_nWdf", "summary": "This paper proposes a Bayesian adaptation of Prediction-powered inference (PPI) problem. PPI is a method which provides confidence interval and estimators in the presence of predictions of (black-box) machine learning models and small amount of labels. This work allows for the possibility of a prior distribution over predictions indicating their quality. The authors demonstrate that this method improves over a pure frequentist PPI, and defaults to a standard PPI when the prior over predictions is a horshoe prior.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time in evaluating our submission and for providing positive and valuable feedback. We are glad that the reviewer found the paper well written and easy to follow.\n\n> Although usage of prior is interesting, the question I have is if the prior is not accurate (i.e. the estimate of the quality of the predictions is not accurate), then how would the performance of the proposed FAB-PPI get affected?\n\nAs stated in the paper, FAB-PPI is motivated by applications in which the black-box predictor $f$ is expected to be of high quality for the given estimation problem, as measured by the rectifier $\\Delta_\\theta$. When this is the case, FAB-PPI results in smaller confidence regions and more accurate estimates than standard PPI. To achieve this, FAB-PPI specifies a prior distribution centered at zero for $\\Delta_\\theta$. In this setting, the chosen prior is not accurate if the true $\\Delta_\\theta$ is not actually close to zero. In the `Biased prediction` experiment in `Section 5.1`, we investigate the behaviour of FAB-PPI as we vary the accuracy of the prior, which is controlled by $\\gamma$ (the higher $|\\gamma|$, the farther away $\\Delta_\\theta$ from zero, and, hence, the less accurate the prior). In this case, the performance of FAB-PPI depends on the specific prior chosen, as well as on the accuracy level $\\gamma$. Under the Gaussian prior, as $|\\gamma|$ increases, the resulting FAB-PPI confidence regions grow unbounded, eventually becoming larger than both the classical and PPI confidence intervals. Conversely, under the horseshoe prior, as $|\\gamma|$ increases, the resulting confidence regions also become larger initially, but they remain bounded, eventually reverting to the standard PPI confidence intervals. In this sense, this property of FAB-PPI under the horseshoe prior, which actually holds for any prior with power-law tails, may be seen as a form of robustness to prior mispecification. The behaviour of FAB-PPI under the Gaussian and horseshoe priors as $\\gamma$ varies is illustrated in `Figure 1` of the paper. \n\nAdditionally, note that, while in the paper we use the scale $\\sigma$ of the noise of the generative model as the prior scale $\\tau_n$ to obtain a parameter-free approach, in practice one could partially control the behaviour of FAB-PPI in the presence of prior inaccuracy through the choice of the prior scale $\\tau_n$. In particular, let $|\\bar\\gamma|$ be the accuracy level in `Figure 1` at which the FAB-PPI confidence region under the chosen prior becomes larger than the standard PPI confidence interval. Then, a larger $\\tau_n$ would imply that the prior gives relatively more weight to moderately high values of $\\Delta_\\theta$, thereby increasing $|\\bar\\gamma|$ (i.e. leading to an improvement of FAB-PPI over PPI across a wider range of accuracy levels). However, the cost of increasing $\\tau_n$ is that the potential gains when $\\Delta_\\theta \\approx 0$ become smaller (i.e. the size of the improvement of FAB-PPI over PPI for $\\gamma \\approx 0$ decreases), as the prior gives relatively less weight to very small values of $\\Delta_\\theta$.", "claims_and_evidence": "The claims made in the introduction and methodolgy seems to be well supported by theory. The paper is well written and easy to follow.", "methods_and_evaluation_criteria": "The proposed method seems to be plausible and definitely interesting for the problem. The consistency of the resulting estimators with different priors like Horseshoe and Gaussian prior are discussed. The evaluation criteria is sound with multiple real world datasets. Overall I find the presented approach and experiments very pertinent to the problem.", "theoretical_claims": "I have not checked for proofs in detail.", "experimental_designs_or_analyses": "The experiments are conducted on both synthetic and real data. The baselines considered makes sense (a PPI without prior, classical estimator) and the authors demonstrate the superior performance of the approach, especially when the number of samples is low, where use of prior is more justified.", "relation_to_broader_scientific_literature": "I am not familiar too much with PPI, so it is hard to assert the significance of the proposed method with respect to wider literature. However, from the context of PPI as a method, prior augmented PPI looks like a welcome contribution that is useful in its own right.", "essential_references_not_discussed": "From my knowledge of this area, essential references seems to be discussed well.", "other_strengths_and_weaknesses": "The paper is well written and easy to follow. The idea of using prior over predictions is novel and interesting. The authors also show the consistency of the estimators under various priors, which is valuable.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Although usage of prior is interesting, the question I have is if the prior is not accurate (i.e. the estimate of the quality of the predictions is not accurate), then how would the performance of the proposed FAB-PPI get affected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FzXNhhunoB", "reviewer": "Reviewer_QSTy", "summary": "The authors propose a new scheme for combining experimental data with model predictions, effectively for inferring valid confidence intervals for estimators where some of the samples are noisier due to being model predictions. The work extends recent prediction-power inference methods by encoding prior knowledge on model errors, in practice a horseshoe prior, and provides detailed theoretical derivation and analysis of the estimator.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time in evaluating our submission and for providing positive and valuable feedback. We are glad that the reviewer appreciates the potential impact of our work.\n\n> The Supplement S6 seems to provide the missing quantification in terms of coverage -- maybe this could be already in the main paper?\n\nWe thank the reviewer for the suggestion. We will make sure to include coverage plots for the synthetic data experiments in the main text of the revised manuscript.\n\n> Could you open up more the somewhat negative result for Alphafold, where you have clearly too high coverage for all PPI variants. Why does it happen, what are the implications, and could we somehow recognise this failure more in practical use?\n\nThe task performed on the Alphafold dataset involves the estimation of an odds ratio, defined as\n$$\n    \\theta^\\ast = \\frac{\\mu_1/(1 - \\mu_1)}{\\mu_0/(1 - \\mu_0)},\n$$\nwhere $\\mu_0$ and $\\mu_1$ are the means of two groups in the dataset. FAB-PPI applies to estimands that may be expressed as minimisers of a convex loss, as defined in `Eq. (1)`. However, this is not the case for $\\theta^\\ast$ above, preventing the standard application of FAB-PPI to obtain confidence regions for $\\theta^\\ast$. To overcome this, we construct $1-\\alpha/2$ confidence regions for $\\mu_0$ and $\\mu_1$ separately and merge them into a confidence interval for $\\theta^\\ast$ through a union bound. The resulting interval is guaranteed to have coverage at least $1 - \\alpha$, but in practice it is often too conservative. That is, the high coverage observed in `Figure 3` is not due to a failure of FAB-PPI (or of classical inference and standard PPI, for that matter), but rather is a limitation of the union bound construction of the CI for $\\theta^\\ast$. Nonetheless, since the same union bound strategy is used for all the methods (including classical inference and PPI) presented in `Figure 3`, comparing the volume of the resulting CIs is still informative.\n\nThis procedure to construct valid CIs for $\\theta^\\ast$ was used in [1, Section 3.1 of the arXiv version] and is summarised in `Appendix S5.1` of our paper. However, in `Section 5.2` we state that the experiment on the Alphafold dataset involves mean estimation, which is imprecise and may understandably cause confusion when interpreting the coverage in `Figure 3`. We thank the reviewer for raising this point, which we will clarify in the revised manuscript.\n\n[1] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. (2023). Prediction-Powered Inference. arXiv preprint arXiv:2301.09633.", "claims_and_evidence": "There are clearly itemised claims that are backed by both theoretical and empirical evidence. The claimed improvements (validity, robustness and efficiency) are relevant and have value for the community, and especially for people applying PPI in scientific research in other fields.", "methods_and_evaluation_criteria": "The proposed method is sound and it is evaluated according to valid criteria. Adding prior information on model errors makes sense and the horseshoe prior encodes a reasonable inductive bias. The choice of the specific prior is discussed in Section 4.3 and the choice made in this work is justified sufficiently well, e.g. by explaining why some simpler alternatives would not be ideal.", "theoretical_claims": "I have somewhat limited expertise on the kind of theoretical analysis done here and did not manage to evaluate the proofs in detail, but overall the analysis aligns with the claims and I did not observe any major gaps or errors. The theoretical analysis could be considered as the main contribution of the work, given the somewhat obvious initial idea, and hence the overall merits of the paper somewhat depend on how significant these contributions are in isolation. That said, even if the development was completely straightforward the paper would have value.", "experimental_designs_or_analyses": "There is clear synthetic data experiment that helps understanding how the approach works, also illustrating how the CIs would behave under a worse prior choice (Gaussian). The other experiments are reasonable, but somewhat harder to interpret. FAB-PPI is shown to have smaller CI, but there is no direct empirical evaluation of whether the CI is correct. The Supplement S6 seems to provide the missing quantification in terms of coverage -- maybe this could be already in the main paper?\n\nOne weakness is that the artificial experiment setups are extremely simplified (which is good for understanding the idea, but they do not really help in understanding how well it works for higher-dimensional problems etc), but the real-world examples in Sec 5.2 complement them well. The results are again primarily from the perspective of CI volume reduction, but now coverage is explicitly evaluated as well. For Alphafold the coverage does not appear to be correct but instead remains roughly around 1 for all $n$, which contradicts the main claims to some extent. There is some discussion on this and the comparison method PPI++ also fails so the issue seems to be more related to the overall family of PPI methods, but the discussion is not sufficiently deep.", "relation_to_broader_scientific_literature": "The paper builds very extensively on Angelopoulos et al. (2023a,2023b), for instance only citing these two sources (in addition to one quick remark on applications) during the first page of the paper. Other related work is properly described later, but nevertheless the work feels somewhat incremental improvement over what Angelopoulos et al. already provided -- the scientific idea of adding a prior is fairly obvious and the theoretical results to an extent follow from previous PPI works. This is not a major limitation as the PPI idea is already getting traction (100+ citations since 2023) and the approach has clear uses across broad range of sciences, but still a minor weakness.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The work has high potential for impact in other fields. PPI is recent high-profile technique that many application areas are likely already considering, and theoretically justified improvements for PPI are valuable for them. There is a possibility this becomes a de-facto solution for the specific task at hand.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. Could you open up more the somewhat negative result for Alphafold, where you have clearly too high coverage for all PPI variants. Why does it happen, what are the implications, and could we somehow recognise this failure more in practical use?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hm9cfJhFHa", "reviewer": "Reviewer_KbSV", "summary": "Prediction-powered inference (PPI) improves statistical inference by leveraging machine learning predictions (on unlabeled data) alongside labeled data, resulting in more precise estimates and tighter confidence intervals. The proposed semi-supervised approach, frequentist-assisted-by-Bayes PPI (FAB-PPI), improves upon PPI by incorporating prior knowledge about prediction quality. A horseshoe prior is used to adaptively adjust confidence regions which appears to work well empirically. The approach maintains statistical validity, improves efficiency when predictions are \"good\", and reverts to standard PPI when predictions are poor/biased, demonstrating in a few real and synthetic data examples its robustness and practical benefits.  The other referenced Bayes-PPI approach gives credible intervals whereas the exposition provides frequentist-type guarantees.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for evaluating our submission and providing positive and valuable feedback. We are pleased that they recognise our approach’s novelty and clarity.\n\n> The introduction could be further strengthened by stating up front what characteristics of the predictor f will lead it to be considered \"good\".\n\nIn the context of PPI, a \"good\" predictor $f$ is one that is associated with a value of the rectifier $\\Delta_\\theta$ close to zero. The rectifier $\\Delta_\\theta=\\mathbb E[\\mathcal L_\\theta'(X,Y)-\\mathcal L_\\theta'(X,f(X))]$ is a problem-specific measure of the quality of the predictions of $f$, associated with the specified loss $\\mathcal L_\\theta$. For example, under the square loss $\\mathcal L_\\theta(x,y)=\\frac{1}{2}(y-\\theta)^2$, a \"good\" predictor $f$ is one such that $f(x)\\simeq \\mathbb E[Y|X=x]$.\n\nWe thank the reviewer for the suggestion, and we will make sure to clarify this in the revised manuscript right after we define $\\Delta_\\theta$. \n\n> Fig S11. MSE, or deviance?\n\nIn `Figure S11`, like elsewhere in the paper, we use the MSE as the measure of estimation error to ensure consistency across all experiments. We will make sure to state this clearly in the revised manuscript.\n\n> As above, what comments do you have regarding the coverage level of ~1 for all methods for the first real data set Alphafold (or other coverage levels away from 0.9, some with trend with $n$).\n\nThe task performed on the Alphafold dataset involves the estimation of an odds ratio, defined as\n$$\n    \\theta^\\ast = \\frac{\\mu_1/(1 - \\mu_1)}{\\mu_0/(1 - \\mu_0)},\n$$\nwhere $\\mu_0$ and $\\mu_1$ are the means of two groups in the dataset. FAB-PPI applies to estimands that may be expressed as minimisers of a convex loss, as defined in `Eq. (1)`. However, this is not the case for $\\theta^\\ast$ above, preventing the standard application of FAB-PPI to obtain confidence regions for $\\theta^\\ast$. To overcome this, we construct $1-\\alpha/2$ confidence regions for $\\mu_0$ and $\\mu_1$ separately and merge them into a confidence interval for $\\theta^\\ast$ through a union bound. The resulting interval is guaranteed to have coverage at least $1 - \\alpha$, but in practice it is often too conservative. That is, the high coverage observed in `Figure 3` is not due to a failure of FAB-PPI (or of classical inference and standard PPI, for that matter), but rather is a limitation of the union bound construction of the CI for $\\theta^\\ast$. Nonetheless, since the same union bound strategy is used for all the methods (including classical inference and PPI) presented in `Figure 3`, comparing the volume of the resulting CIs is still informative.\n\nThis procedure to construct valid CIs for $\\theta^\\ast$ was used in [1, Section 3.1 of the arXiv version] and is summarised in `Appendix S5.1`. However, in `Section 5.2` we state that the experiment on the Alphafold dataset involves mean estimation, which is imprecise and may understandably cause confusion when interpreting the coverage in `Figure 3`. We thank the reviewer for raising this point, which we will clarify in the revised manuscript.\n\nOther fluctuations in empirical coverage away from the nominal level include uncercoverage for small $n$ in some experiments and overcoverage as $n$ increases for the Forest dataset. The first phenomenon is due to the fact that all the confidence regions compared (classical, PPI, and FAB-PPI versions) are based on the CLT, as detailed in `Assumption 3.1`. In practice, even when the number of unlabelled samples $N$ is sufficiently large as in our experiments, all methods require the number of labelled samples $n$ to be large enough for the CLT on $\\widehat\\Delta_\\theta$ to kick in and ensure the right coverage level, while small $n$ may lead to fluctuations in the coverage level for all the methods considered. To mitigate this, one could alternatively employ (non-uniform) Berry-Esseen bounds (see e.g. [2] and references within) to obtain more conservative asymptotically valid CIs for both PPI and FAB-PPI. \n\nOn the other hand, the seemingly increasing trend in coverage for the Forest dataset when $n > 200$ is due to the size of the dataset, which is the smallest considered ($N = 1596$). In particular, the \"ground truth\" against which the empirical coverage is computed is the sample mean of $Y$ across the labels of the entire dataset. As the size $n$ of the labelled subset used for PPI grows, there is a non-negligible overlap between the latter and the full dataset used to compute the ground truth (e.g. overlap $>30\\\\%$ when $n = 500$), explaining the overcoverage. Once again, this issue applies to all methods considered, and the comparison of the volume of the resulting confidence regions remains informative.\n\n[1] Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., and Zrnic, T. (2023). Prediction-Powered Inference. arXiv preprint arXiv:2301.09633.\n\n[2] M. Austern and L. Mackey (2022).  Efficient concentration with Gaussian approximation. arXiv:2208.09922.", "claims_and_evidence": "The methods are stated clearly, and the contributions are novel to the best of my knowledge.  The choice of the horseshoe prior is appropriate, and there is discussion (but not experimentation) of other priors that would meet the given desiderata (zero spike, power law tails, analytic).", "methods_and_evaluation_criteria": "The tests are on synthetic data and four low dimensional datasets.  The evaluations appear appropriate (volume levels on synthetic data, mse, volume, and coverage on real data).", "theoretical_claims": "The theoretical claims appear sound.  Technical details are in S2 and S3.", "experimental_designs_or_analyses": "The designs seem appropriate and are both synthetic and real world.  The dimensionality of all datasets is low as the method appears to suffer from CoD (mentioned only at the last sentence of the discussion).", "relation_to_broader_scientific_literature": "This manuscript provides an approach for frequentist confidence levels in the semi-supervised setting, which is a novel contribution to the best of my knowledge.  Previous relatives included PPI PPI++ and Bayes PPI for credible intervals.  No discussion of how to scale to multivariate analyses limits the approach, but the exposition may provide instruction for future approaches in the SSL framework.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The introduction could be further strengthened by stating up front what characteristics of the predictor f will lead it to be considered \"good\".", "other_comments_or_suggestions": "Fig S11. MSE, or deviance?", "questions_for_authors": "1. As above, what comments do you have regarding the coverage level of ~1 for all methods for the first real data set Alphafold (or other coverage levels away from 0.9, some with trend with $n$).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose an adaptation of the recent PPI technique which incorporates a Bayesian-like prior on the accuracy of the input prediction function f, resulting in confidence regions that remain valid but are tighter for settings in which f is accurate (which correspond to most application settings of interest).   The paper is very clearly written; the authors analyze the consequences of choosing either a Gaussian or horseshoe prior, and nicely illustrate their methods with both synthetic experiments and several real data sets studied in the prior work.  Authors were unanimous in recommending acceptance.
Decision: Accept (poster)