Anytime-valid, Bayes-assisted, Prediction-Powered Inference
Authors: Valentin Kilian
Stefano Cortinovis
Francois Caron
Venue: NeurIPS 2025 poster ()
Keywords: Confidence sequences
Ville's inequality
method of mixtures
mean estimation
statistical inference
semi-supervised inference.
Primary area: general_machine_learning

Abstract: Given a large pool of unlabelled data and a smaller amount of labels, prediction-powered inference (PPI) leverages machine learning predictions to increase the statistical efficiency of confidence interval procedures based solely on labelled data, while preserving fixed-time validity. In this paper, we extend the PPI framework to the sequential setting, where labelled and unlabelled datasets grow over time. Exploiting Ville's inequality and the method of mixtures, we propose prediction-powered confidence sequence procedures that are asymptotically valid uniformly over time and naturally accommodate prior knowledge on the quality of the predictions to further boost efficiency. We carefully illustrate the design choices behind our method and demonstrate its effectiveness in real and synthetic examples.

Anytime-valid, Bayes-assisted, Prediction-Powered Inference
Valentin Kilian" Department of Statistics, University Oxford
Stefano Cortinovis" Department of Statistics, University of Oxford cort cortinovis@stats. ox ac. uk
François Caron Department of Statistics, University Oxford caron@stats ox ac. uk
ki kilian@stats. ox. ac uk
Abstract
Given large pool of unlabelled data and smaller amount of labels, prediction- powered inference (PPI) leverages machine learning predictions to increase the statistical efficiency of confidence interval procedures based solely on labelled data, while preserving fixed-time validity. this paper, we extend the framework to the sequential setting, where labelled and unlabelled datasets grow over time. Exploiting Ville's ' incquality and the method of mixtures, we propose prediction- powered confidence sequence procedures that are asymptotically valid uniformly over time and naturally accommodate prior knowledge on the quality the predic- tions to further boost efficiency. We carefully illustrate the design choices behind our method and demonstrate its effectiveness in real and synthetic examples.
Introduction
Increasing the sample size of experiment is arguably the single simplest way improve the preci- sion of the statistical conclusions drawn from it. However, in many fields such as healthcare, finance, and social sciences obtaining labelled data is often costly and timecconumingg In these settings, using machine learning (ML) models to impute additional labels represents a tempting alternative to expensive data collection, albeit the risk of introducing bias. Prediction-pon poweered inference (PPI) M is a recently introduced framework for valid statistical inference in the presence of small labelled dataset and large number of unlabelled examples paired with predictions from black-box model.
Formally, given an input/output pair (X, Y) ~ P = Fx x Py|x, consider the goal of estimating
0" arg min El(o(X,Y)) VER
(I)
where fo(x, y) a convex loss function parameterised by 0 R. As an example, the mean 0 EY] is the estimand induced by the squared loss (0 y) (0 - y) For t = 2...... ., we observe
sequence of independent random variables Z,, either drawn from IP (labelled sample) from Fx (unlabelled sample), and we are provided with black-box prediction rule f that maps any input x to a prediction ((x).
Let (X,, Yi)s1 and (X,)221 denote the subsequences of labelled and unlabelled samples, respectively. For = 1,2, ... let N,, denote the number of unlabelled samples observed before the nth labelled one, and assume that N₂, 2 n, with N,, > n in typical settings. PPI constructs an (asymptotic)
confidence interval (CI) CPP for 6*, that exploits the auxiliary information encoded in f. To this end, under mild assumptions, 8" can be expressed as the solution to
go- = E[G (X, =0
(2)
Equual contribution. Order decided by coin toss.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

where fo is a subgradient of lo with respect to A. The quantity go in Equation 2 can be decomposed as go mg Do where
mg == (X, (X))] and A; E[G(,)) (X,
(3)
where my represents a measure of fit of the predictor, while Ag. the rectifier, accounts for the discrepancy between the predicted outputs f(X) and the true labels Y. If Ca a,0,a is a (1 a) confidence interval for go, then the PPI confidence interval Cap defined as
C a,n = { -0100 Co )2,0m))
also achieves the desired coverage, i.e., Pr(0" E C 2 - a. Constructing Ca C0,0.. relies on estimating 9o. for which PPI defines an estimator leveraging both the unlabelled data and the prediction rule f. The resulting method outperforms standard CI procedures based on the labelled data alone when I' is sufficiently accurate and N, >> n. Intuitively, this is because, in this case, A close to zero, while mg can be estimated with low variance from the unlabelled data.
Crucially, coverage of the PPI CI 4 is guaranteed only at a fixed time, i.c., for labelled sample size fixed in advance. This is undesirable many practical settings such as online learning, real-time sime monitoring, or sequential decision-making - where it is essential to continuously draw conclusions as new data arrive. In this work, we address this by proposing an anytime-valid extension of the PPI CI (4) That is, we define confidence sequence (CayDD) J221, satisfying asymptotically the stronger coverage guarantee
Pr(#* E CaRP for all 2 1) 1-a.
while still taking advantage of the prediction rule Analogously to standard PPI, we construct confidence sequence (C", (...0.n 2m for and define Carpp through Equation (4) for n 2 1. While our approach is agnostic to the specific form of the confidence sequence (C
mainly focus asymptotic confidence sequences 2 as they provide versatile time-uniform analogue of standard CLT-based CIs that applies to the PPI framework above full generality. Moreover, being based on the method of mixtures 3 5 they can readily accommodate prior information the quality of the prediction model f. In particular, by means of zero-centred prior on the rectifier Ag, we obtain tighter confidence sequences when the predictions are good, extending the fixed-time Bayes-assisted approach of Cortinovis and Caron 6
The remainder of the paper is organised as follows. Section 2 reviews related work. Section provides background on (asymptotic) confidence sequences and discusses how prior information may be incorporated into their construction. Section [ estimators, whose asymptotic properties are crucial for our approach to anytime- valid, Bayes-assisted PPI, which is described in Section 55 Section demonstrates the benefits of our method on synthetic and real data, Finally, Section discusses limitations of our approach and further extensions. Proofs and additional experiments are provided in the Supplementary Material.
presents PPI in the context of control-variat
2 Related Work
PPI was introduced by Angelopoulos al. 1 as a general framework for valid statistical inference with black-box ML predictors,, and later extended in Angelopoulos et al. 77] Closely related ideas appear in the literature on superiuper inference, missing-dan -ata methods, survey sampling, and
double machine learning &8 9 10 I1 [2]. More recently, Cortinovis and Caron (6] proposed
Bayes-assisted variant of PPI. All of these contributions target fixed-time confidence intervals.
Confidence sequences (CS) were introduced by Darling and Robbins 13] and further developed by Robbins and Siegmund 4 and Lai (5). building on earlier work by Ville (3] and Wald (14).
Interest has surged again in recent years [1] 21, motivated by applications such as A/B testing.
The notion is closely linked e-values [6 17. Building on the e-value framework and on earlier
work by Zrnic and Candès [8] and Waudby- Smith and Ramdas [15. Csillag et al. [9 proposed an exact, time-uniform PPI method that yields CS under stronger conditions (e.g., existence of bounded e-values) but does not leverage prior knowledge about the quality of the ML predictions. Furthermore, applying their method requires an active-learning setup in which, at each time I, the observation Z, can be labelled with strictly positive probability. In particular, it is not applicable to deterministic sequences of observations, such as those describing large initial pool of unlabelled data followed by stream of labelled data, which are the main focus of our experiments.

---

In the setting of double machine learning and semiparametric inference, Dalal et al. [20] and Waudby- Smith al. 2 derive asymptotic confidence sequences for target parameters in the presence of high-dimensional nuisance components.
3 Asymptotic (Bayes-assistedd confidence sequences
In this section, first review background on (asymptotic) confidence sequences (CS), and then show how prior information can be incorporated into asymptotic CS procedures, leading to asymptotic Bayes-assisted confidence sequences.
3.1 Background
We start by defining an exact confidence sequence 13 time-uniform analogue of classical Cls. Definition 1 (Confidence sequence). Let (Ca,))21 be sequence of random subsets of IR. For (0, 1), (Ca,)121 is a 1 - a confidence sequence for fixed parameter μ E IR
Pr(ju Ca, for all 2 1) 2 a.
(5)
We now introduce the notion of asymptotic confidence sequence (AsympCS) 2 20
Definition 2 (Asymptotic confidence sequence). Let a E (0,1) and (at)121 be a real sequence such that lime-roo at 0. Let (ī4)221 be consistent sequence estimators μ. The sequence of random intervals (Ca.))21, with Ca.: = [T, Tu i + Uz] and I, > 0, U, > O, is said to be an asymptotic confidence sequence with (little-o) approximation rate aę if there exists a (usually unknown) confidence sequence (Ca,)i21, with Ca, = fite Li Pt + UJ]. such that
Pr(u for all 1) 1 1-a
and, almost surely as t + oo, max - L,, U U.} o(a:).
Thus, an asymptotic CS may be regarded as an approximation of an exact CS that becomes arbitrarily accurate in the limit. It is worth noting that, while classical sample asymptotic Cls rely on convergence in distribution of scaled estimators, asymptotic confidence sequences rely on almost sure convergence at a given rate of the centred lower and upper bounds relative to those of an underlying exact CS. The following is an example of an asymptotic CS that applies to i.i.d. data.
Theorem Let (Y1)z2 be sequence of i.i.d. random variables with mean μ and such that E|V 2+6 oo for some 5 > O. For any Z let Y, be the sample mean, and oe be the sample vari- ance based on the first observations. For any parameter p > O, the sequence of intervals defined as
Ōe viV
(1
tpp
+
CuA Ca 1, G; = Y,
log
tp
forms (1 -1-a0 AAsympCS with approximation rate 1/ 1/VE.leee Vogott &. for μ.
For the sequel, it is useful highlight some aspects of the proof of this theorem. First, if the random variables (Yr):21 were Gaussian with variance o², then (&,,(Y: p) would form exact CS. This follows from combining the method of mixtures for nonnegative martingales with Ville's inequality B 4 5 21). Second, the proof relies KMT strong coupling 22 23]: there exists i.idd.
Gaussian random variables (Wi)221 with mean f and variance var (Y') such that
Y.
W: +
a.s. as
+
VVlogt
Such coupling plays central role in constructing asymptotic confidence sequences, serving as a substitute for the CLT assumption underlying classical fixed-sample Cls. The construction in Theorem
I extends beyond the i..dd case, provided similar coupling exists.
Theorem 2. Let (ft) be a consistent sequence of estimators of μ. Assume that there exists a sequence of i...d. Gaussian random variables (W:)s21, with mean μ and variance o², such that
Vllot (11
W, +
a.s. as oo.
3

---

Let (o))12 be consistent sequence of estimators of o² with (ō, - ol any parameter p > O, the sequence of intervals (Ca, (Pe. oū p)) forms (1 a) AsympCS with approximation rate 1/vt log t for μ.
logt a.s. Then, for
The asymptotic CS includes tuning parameter p. which can be chosen so to minimise the width of the interval at a specified time t; see [2] Appendix B.2]. However, this method does not allow the incorporation of prior information about the parameter of interest to yield tighter intervals when the data align with such assumptions: the width of Equation (6) is independent of Y,.
3.2 Asymptotic Bsyes-assisted confidence sequences
To address this, we introduce Bayes-assisted analogue of Theorem 1
Theorem 3 (Baycs-assisted AsympCS ii.d. case). Let (Xt):21 a sequence of i.i.d. random variables with unknown mean y and unknown variance o², and such that [E|Y, < oc for some > For any t 2 1, let Y, be the sample mean, and of be the sample variance based on the first t observations. Let T): IR + (0, vt/(2x)) be defined as
[ (2: (())G
(e(2)
(8)
where vt is continuous and proper prior density on RR strictly positive in a neighbourhood of p/o. Then
Tr ViV 2 (Y, ,2,
Ca (Y (1)
Y.
log
forms - a1) -AsympCS with approximation rate 1/ Vt log t for
the density π encodes prior beliefs about the ratio /u/ Under this prior, ?/ represents the marginal density of the standardised mean Y t/ar that would arise if the observations (Yi) were normally distributed. In contrast to the non-assisted AsympCS (6. the width of the Bayes-assisted AsympCS (
In Theorem 3
varies with ,, when the data align with the prior, ne (Y c/ōz) ) is large and the interval narrows; when they conflict, (Y. (Y./o)) even when the prior is strongly misspecified, the Bayes-assisted AsympCS (g) remains valid. In the case of a Gaussian prior π centred at μl with variance 7², we obtain the following AsympCS
) is small and the interval widens. It is worth emphasising that,
tr (1 + a²
Y (Y/O,, - M2)
CBA Ce
Y + log
(10)
1/t
Setting = allows a direct comparison between (10) and its non-assisted counterpart 6 When the data agree with the prior - i.e., Y,/ō. Mo ~ O the Bayes-assisted interval is narrower than the non-assisted one. Conversely, if the data conflict with the prior, (Y./ō, TT/0--=) is large and the Bayes-assisted AsympCS becomes wider than (6). The proof of Theorem is similar to that of (2] Theorem 2.2]. First, note that C.A, (Y,, var(Y): would be exact CS if the observations were normally distributed. This follows from an application of the method of mixtures for nonnegative martingales, using the prior mixing density, together with Ville's inequality. Second, use KMT strong coupling approximate in almost sure sense Y, by sample average of i.i.d. Gaussian random variables. As the non-assisted case, Theorem 3 can be extended to the non-i.i.d. setting, as long as
find such strong coupling.
Theorem 4 (Asymptotic Bayes-assisted CS non-i.i.d. case). Consider the same notation and assumptions as Theorem Let be continuous and proper prior density IR strictly positive in a neighbourhood of u/o, and let be the density (8) for any 2 1. Then, the sequence of intervals (CBA, (Pc, ō.; 12) forms
a) AsympPC with approximation rate 1/Vtlog & for p.
3.3 Asymptotic Type-1 error control
The asymptotic confidence sequences defined above satisfy an asymptotic version of time-uniform Type-I error control (in the sense of [2] $2.5]; see also 24]

---

Theorem 5 (Asymptotic Type-I enror control). Assume the hypotheses of one of Theorems to and let (C) be the corresponding (1 Asy-ssy for p. Then
lim inf Pr (14 E Ca.t for all t 2 m) 2 1 11-a a.
(11)
Control variates and PPI: background and strong coupling
Prediction-powered inference (PPI) closely relates to control variates, standard variance-reduuciion method in Monte Carlo estimation [25] $4.1J. In fact, each PPI estimator can be expressed as control-variate estimator. We begin with review of control variates and derive KM'T-type strong-coupling result for these estimators, before providing additional background on PPI.
4.1 Control variates: definitions and KMT strong coupling
Let V) be real-valued random variables with finite variance, and consider the goal of estimating y = E[V] from an i.i.d. sample (U,, Vi) If μ E[U] is known, the control-variate estimator
(CVE) of y is defined
1 1 " M (V. A(U -)))
7 = V X(U - n))
(12)
where U and V denote the empirical means of (U:)i=1 and (Vi),= respectively, E R is tunable coefficient, and the term U3- μ acts as a control variate. The estimator 7X is unbiased, consistent, and has variance var (7x - (var(V) 2Acov V) + X²var(U))/nn. Compared the standard sample mean estimator V. which attains variance var(V) var(V) using şiev yields variance reduction when À < 2cov(U, V) /v)r)((()) The minimum variance is achieved at the optimal coefficient A = cov(U, V))rr((U), for which var(? = (1 pê:. var(V), where PU,V is the correlation between U and V. That is, stronger correlation leads to greater variance reduction.
In practice, and X are typically unknown, When this the case, given an additional i.i.d. sample (U,) (0,))²m No. independent of (U,, Vi where U, has the same distribution as U, one can estimate by Ā N. 2==10) U, and plug it into Equation (2). For fixed X, this gives
X V A(U - p
(V. A(U, - P)).
(13)
Similarly, X may be estimated from data X (OU, /arr)) where var(-) and (.v denote the sample variance and covariance, respectively. Plugging À into (L3) defines goo++ = 7/ which is similar the semi-supervised least squares estimator of Zhang al. [I]
Eq. (2.15)]. As discussed in Section 3, deriving an AsympCS requires a strong coupling between the estimator and a sequence of i...d. Gaussian random variables. We now establish this coupling, key ingredient for constructing AsympCS for CVEs (and, in particular, for PPI estimators).
Proposition 1 (Asymptotics for CVEs), Assume [2+ and E V 224 < 0o for some 0 < 8 < 1. Then, almost surely as nI + 0C,
gert 7
=V - x- 12) + Vnlogn
(14)
loon
Proposition (KMT coupling for CVEs), Assume E|U and V < 20 for some Additionally, assume that n N_ - = O(1/n--9 with < < 2/ + 8), for some r E [0, 1]. Then,
1.
there exist i.i.d. Gaussian random variables (W )221 with mean and variance
xxxh = var(V XU) + rvar(AU)) var(V) 22cov(U,V) + X² v)var(U)
such that, almost surely as n + 20.
Wey +
(15)
Vnllon
1=1
5

---

Likewise, there exist i..dd Gaussian random variables (WT*)) ")22 with mean "y and variance vev x = var(V) [1 (1-r))t; such that, almost surely as n + 00.
gpvt W +
(16)
Vn
The estimators
a
nda
N,
D ((U), W.,)(,,)
Σ
V. V A(U; - UU) + (N,,, - 1)
(U3 - 2))
n n-
i=1
j=1
(17)
n/n n n1
Σ -V-V-X(U - n/N, 11
n Σ
got ( ((U,))))
V) (18)
1=1
are consistent estimators of V and vM+, respectively, where çi Nn ΣN Uz.
4.2 PPI estimators: definitions and asymptotic properties
Owing to Equation (2 the PPI estimator 8n is the value that solves the equation To.n = 0. where Ie.n ma + Aa. is estimator of go. Here, me.. and Σθ, estimators of me and Ap. respectively. typical choice for ma.n is the sample mean of the unlabelled data,
ma.m
Σ ((X,),
(19)
Different choices for A0, have been proposed in the literature, leading different PPI estimators.
Standard PPI.
Angelopoulos et al. | use the sample mean
A
PP 0,n
(C,),, [X,,
(20)
estimator for Ag. Combining Equation 20 and Equation 19
1 Σ T1 1=1
( (o(xx ma.m
PP f0.n
mA,n + PP
(X,, Y.)
(21)
is CVE with control variate Eg (X,, ((x)) mo,, and control-varias parameter À 1. For the squared loss, the estimator TRD solving Do, = 0 also takes the control-varie form
N. Nn
Y,
1
f(x))
(22)
n
i=1
with control variate f(X.) N, Σym f(X,) and 1. 1.
PPI++ Angelopoulos et al. 7 extend the standard PPI estimator 21 by allowing the control- variate parameter A, which they call power-tuning parameter, to take values other than 1. The resulting estimator is
PP+ 0,n
APP =APP - Noon 1) 8.n
Σ t (X (X,, ((xx.((x)))
me.
(23)
where Rm is the estimator Ao,n = Cov (X, Yi), ((,, ((x))) var
In this case, Appt is CVE with centred control variate Eg (X,, f(X.)) mo which depends only on the black-box predictions. As a result,
Σ (X,, Yi) - So.n (
BP+ = men + -0
Pp+ = A,n
(x(xx,
mo.1
(24)
n

---

is also a CVE. Under the squared loss, we obtain
! [ f(.))
N₂ Nn f(())
Σ ==1
Y. Ao,
(25)
where in this case A0 = Ao for all 0. Standard asymptotic confidence intervals for PPI and PPI++ rely CLTs for the estimators n0 mo and Σa... In contrast, constructing asymptotic confidence sequences requires almost sure approximations by averages of i.i.d. Gaussian variables. Since the estimators for 9ø, ma and As are all CVEs, the asymptotic results of Proposition I and the KMT coupling of Proposition both apply.
Anytime-valid, Bayes-assisted, prediction-poweree inference
In this section we combine the results of Sections 3 and within the PPI framework to obtain
AsympCS for Go- For any E IR and i 2 1, let Uo.i (X,, i, f(X;)), Uo = f/ (X,,, f(X;) and
Di= Ua In the following, assume E U, 22+5 E| (2+5 and Vo, 12+6 < Do for some < and that x
Vo,; (X,, Y.). Define V
+/2 E Va.5 and Unn
N. -
O(1/n with 0 < 22 + 6) for some E [0.
5.1 Anytime-valid PPI
We first derive AsympCS that do not incorporate prior information about the black-box predictor's accuracy. The following result follows directly from Proposition and Theorem 2 owing to the control-variate form of the PPI estimator PP
21 and of the PPI++ estimator Pp+
Proposition 3. Let 9e,n be either the PPI
21 the PPI++ (4) estimator. For PPI, let (0) )²
((Uo, Vo (Uo (see (7p.. J. For PPI++ let (2 Dex+ ((U9.5. Vo.i (see 18)
Then, for any > 0, the sequence of intervals defined as C9 a.0.n CHA, 0, P) forms
(1 - AsympCS with approximation rate 1/v/n log n for ge and asymptotic Type-I ervor control.
5.2 Anytime-valid, Bayes-assisted, PPI
In many modern applications extremely accurate black-box predictors are available (e.g.. 126 27 28 When this is the case, we can leverage this prior information to obtain tighter AsympCS for ge via a zero-mean prior on Ag. Following the decomposition in Equation (3) combine AsympCS for my (Proposition 4 with a Bayes-assisted AsympCS for Ag (Proposition
Proposition (AsympCS for ma). Let ma and (o be the sample mean and sample variance of ((x)),, Let E (0, 1). For any 0, Rā,0, C,n. (mo.:: Tb P) forms (1 (1-8)-As with approximation rate // log for ma and asymptotic Type-I error control.
Proposition (Bayes-assisted AsympCS for Ag). For PPI, let Bon and (o be the sample mean and sample variance of (V9, Ug For PPI++, ++, let Dg,n. be the control-variate estimator
20
23 and (G v ((Ua, Vo,; 1n (see (88). Let K. E 1). For any continuous
proper prior π, the sequence of Bayes-assisted intervals Tw.9.n = -CBA, (Ao, o TT) forms (1 (1-k) (1=x))AsympCS with approximation rate 1/ /n logn Ag and asymptotic Type-I error control,
Finally, for both PPI and PPI++ the confidence sequences Ra,0,n and Ta-á,0, are combined via Minkowski sum to obtain a (1 a) AAsyppSS for go. with approximation rate 1 /n log n and asymptotic Type-I error control, of the form
n2(TTx2)
1 + Nnp²
NNp² +
C9 cg
I0.n +
8mn
log
log
Tm (A..
102
VN,
Nnp²
82
(26)

---

where Io.n is either the PPI estimator (21 or the PPI++ estimator 24) Solving Equation the confidence region for 0". In the case of the squared loss, Cayp is an interval, given by
gives
Cavpp
On +
log
71
o
Nmp?
log
Nap² +
2TT2 nn So.n
VN
NuP²
82
0,1
(27)
where 8n is either the PPI estimator 22
the PPI++ estimator
25
6 Experiments
We compare the PPI and PPI++ AsympCS procedures introduced in Section 5 with and without Bayes assistance to the AsympCS relying solely labelled data (obtained from Theorem and referred to as "classical") on several estimation problems. Bayes-assisted methods are annotated with (G), (L.), or (T) indicate Gaussian, Laplace, Student-t priors with mean zero and scale depending on the task and reported in the Supplementary Material. For the Student-t prior, we set the degrees of freedom to 2 in all experiments. Since PPI is motivated by settings with scarce labelled data and abundant unlabelled data, we consider the following experimental setting: labelled data arrive sequentially, i.c., n
2, .... while large unlabelled dataset is available from the start, i.c. Nn = N for all n. with N >> n large enough to exclude any uncertainty on the measure of fit ma. As discussed by Cortinovis and Caron (6,, this simplifies the comparison between non-assisted and Bayes- assisted PPI, as it rules out any potential loss of efficiency due to the Minkowski sum (26), thereby isolating the effect of the Bayes correction on the CS procedure. For synthetic data, we set N oo to guarantee the simplification holds. For real data, we empirically verify that N is large enough to justify this assumption by confirming that anytime validity preserved specifically, that the cumulative miscoverage rate remains below the chosen threshold = 0.1 for all n. As with CI.T-based Cls, the n which one starts counting the cumulative miscoverage rate of an asymptotic CS is inherently arbitrary; unless otherwise stated, we choose n 40, empirically find this to be a reasonably small labelled sample size at which the KMT coupling generally provides a good approximation.
6.1 Synthetic data
The synthetic experiments follow general structure: we start with N = oo unlabelled samples iid
{X lid Fx and successively sample n labelled observations (X,, Y.) ~ P with the goal of estimating the mean 0" [E
Noisy predictions. This experiment demonstrates that our method can adapt to varying correlation
levels between predictions and true labels by using the PPI++ estimator 23 We sample Y; iid (0, 1), so that 0+ E|y O. The prediction rule is defined as f(X;)
Yi eg where X; is
only used for indexing and e, lid (N )²), with the noise level ay E {0.1, 3). In this case, the (X))
optimal control-variate parameter is given by Xガ = X which decreases with σy. Figure compares the interval volume achieved by classical and non- assisted CS procedures as a function of n. while results under informative priors are reported in Section S7.1 For small noise levels, PPI and PPI++ achieve similar performance, and greatly outperform classical inference. As the noise level grows, the machine learning predictions become less informative and standard PPI loses ground to the classical CS. By contrast, PPI++ adapts the noise level and always performs similarly to, or better than, the other baselines.
= (1 + -
Biased predictions. This experiment illustrates the potential benefits of incorporating prior lid information into our method. We sample X; ~ NN 1) and Y; =x; G, where G;
lid far(0. 1). so that 8- Y O. The prediction rule defined as f(X,) = X; v. where its bias level. For all v. A = 1, so PPI and PPI++ coincide. We vary between - 1.2 and 1.2, and df (5, 10, )o) to study the impact of bias level and noise distribution on the AsympCS procedures. Figure compares comparees the average interval volumes at = 100 as function of v for each value of df. Classical inference and non-assisted PPT volumes remain essentially constant across bias levels, reflecting their lack of prior information, and with the latter consistently outperforming the former
R controls
8

---

.o 0.1, X 0.99
OY 0.8, X 0.61
or 3.0, X 0.10
volume
1
1
0
500
1000
0
500
1000
0
500
1000
classical
PPI
PPI++
Figure 1: Noisy predictions study. The left, middle and right panels show average interval volume over 1000 repetitions function of the labelled sample size n for noise levels σy {0.11, 0.8, 3.0}.
df = 5
df = 10
df
1.5
100
volume
1.0
1
0.5
1
0
classical
PPT
PPI (G)
PPI (L.
PPI (T)
Figure 2: Biased predictions study. The left, middle and right panels show average interval volume over 100 repetitions as a function of the bias level v for df = 5, 10, co.
by leveraging imputed predictions. On the other hand, the volume of the Bayes-assisted procedures varies widely with the bias level v. it is reduced for small v. but grows with |e] as the priors become increasingly misspecified. Notably, the volume under the Gaussian prior inflates the fastest with w//, while heavier-tailed Laplace and Student-t priors offer comparatively greater robustness. These conclusions hold for all values of df, which controls the accuracy of the KMT coupling approximation for a given n. Coverage results Section [S.T show that, while smaller values of df lead to slightly worse coverage, the approximation quality is overall satisfactory in this example.
6.2 Real data
We evaluate our method on several real-world datasets, which are described in Section S6.2 While N+m each dataset is, in principle, static (providing label/prediction pairs (Y,, (Xx)) N**), ), simulate
an online setting akin to Section 6.1 by randomly splitting the data into a labelled set of size mi serving as a labelled data stream, and an unlabelled set of size N.
Figure 3 compares classical and PPI++ AsympCS procedures on the FLIGHTS, FOREST, and GALAX- IES datasets, where the goal is mean estimation. By taking advantage of the unlabelled data, PPI methods consistently yield smaller regions than the classical counterpart, while maintaining reliable coverage. Moreover, Paaess-assiste approaches further improve efficiency for moderate labelled sample sizes, as the quality the predictions generally high in these datasets.
Figure 58 reports results for three additional estimation tasks: linear regression (CENSUS), logistic regression (HFALTHCARE), and quantile estimation (GRNES). For first two tasks, the same conclusions as for mean estimation hold: PPI methods consistently outperform classical inference.

---

FLIGHTS (N 93333)
FOREST (N 1096)
GALAXIES (N 15743)
1500
volume
1000
0.5
500
0.0
0.2
0.2
mnisovvrrago
0.1
0.1
0.0
0.0
5000
10000
200
400
500
1000
classical
PPI++
PPI++ (G)
PPI++ (L)
PPI++ (T)
Figure 3: Mean estimation. The top and bottom rows show the average interval volume and cumulative miscoverage rate over 1000 repetitions for the FLIGITTS FOREST, and GALAXIES datasets.
with Bayes-assisted approaches providing an additional efficiency boost. For the quantile estimation task, non-assisted PPI still improves classical inference by leveraging the machine learning predictions; however, the Bayes-assisted methods yield larger regions than the other approaches, reflecting lower prediction quality in this dataset.
7 Discussion
We extended the PPI framework to the sequential setting via asymptotic confidence sequences, which allow for the seamless integration of prior information about the quality of the auxiliary predictions. However, several directions merit further investigation. The results developed here are for scalar parameter values A. Extensions multivariate settings are discussed in Section [S4] building on earlier work by Waudby-Smith et al. [2] $B.10]. In the non-assisted case, we focused on asymptotic confidence sequences of the form
but other options are possible. In particular, as discussed in Section S8 the farameter-free CS proposed by Wang and Ramdas (29), which is based an improper prior, may used as an exact reference CS place Equation (6).
The AsympCS derived in this paper asymptotically valid for i.i.d. data under mild, nonparametric assumptions. Promising directions include extensions non-i.i.d. observations, as well as the development of nonasymptotic, nonparametric Bayes-assisted confidence sequences under stricter assumptions (e.g., bounded means), building on the work of Waudby- -Siih and Ramdas [15]. In the non-assisted case, the parameter was assumed to be fixed. Waudby-Smith et al. 22 $2.5] considered delayed-start sequences Ca,((m) that may depend on the start time m; this includes allowing the tuning parameter p to depend on m. Their asymptotic Type-I error control result, derived under
assumptions similar to those used here, also applies our setting. Another interesting direction would be to adapt similar ideas to the Bayes-assisted construction.
PPI AsympCS procedures share the computational considerations of their fixed-time counterparts. Beyond mean estimation (e.g.. Figure S8). they typically require constructing grid over 6. When the marginal density 7): not available in closed form (e.g., for the Student-t prior), the Bayes-assisted ver- sion requires numerical integration. If computation is concern, the Laplace prior offers good com- promise: has heavier tails than the Gaussian while still admitting a closed-form expression for т)--
10

---

Acknowledgments and Disclosure of Funding
Valentin Kilian is supported by the Clarendon Funds Scholarship. Stefano Cortinovis supported by the EPSRC Centre for Doctoral Training in Modern Statistics and Statistical Machine Learning (EP/S023151/1) The authors thank the reviewers for their time and valuable feedback, especially the suggestion to incorporate discussion on Type-I error control.
References
[1] A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zmnc. ceddiction-poweree inference. Science, 382(6671))666) 674, 2023.
[2] I. Waudby-Smith, D. Arbour, R. Sinha, E. Kennedy, and A. Ramdas. Time-unifore central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6):2613-28) 2-4000 2024.
[3] J. Ville. Enude critique de la notion de collectif. Gauthier-Viiee
Paris, 1939.
[4] H. Robbins and D. Siegmund. Boundary crossing probabilities for the Wiener process and sample sums. The Annals of Mathematical Statistics, pages 1410-1429, 1970.
[5] L. Lai. confidence sequences. The Annals of Statistics, pages 265-280, 1976.
[6] S. Cortinovis and F. Caron. FAB-PPI: Frequentist, assisted by Bayes, prediction poweered inference. In International Conference on Machine Learning (025),002)), 2025.
[7] A. Angelopoulos, J. Duchi, and T. Zrnic. PPI++: Efficient predeetion-poweree inference. arXiv preprint 231 1.01453, 2023.
[8] M. Robins and A. Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 10(429)):22-1299 1995.
[9] C.-E. Särndal, B. Swensson, and J. Wretman. Model assisted survey sampling. Springer Science Business Media, 2003.
[10] V. Chernozhukov. Chetverikov, M. Demirer, E. Duflo, Hansen, W. Newey, and J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-C68, 2018.
[11] A. Zhang, L. D. Brown, and T. Cai. Semi-supervised inference: general theory and estimation of means. The Annals of Statistics, 47(5):2538-2566, 2019.
[12] Y. Zhang and J. Bradic. High-dimensional semi-supervised learning: in search of optimal inference of the mean. Biometrika, 109(2):387-403, 2022.
[13] D. A. Darling and H. Robbins. Confidence sequences for mean, variance, and median. Proceed- ings of the National Academy of Sciences, 58(1):66-68, 1967.
[14] A. Wald. Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16 (2):1 1945.
[15] I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1-27, 2024.
[16] A. Ramdas, Grünwald, V. Vovk, and G. Shafer. Came-theoretic statistics and safe anytime- valid inference. Statistical Science. 38(4):576-66) 2023.
[17] A. Ramdas and R. Wang. Hypothesis testing with e-values. Foundations and Trends@ in Statistics, (1-2):1-38 2025. ISSN 2978-4212. doi: 10.1561/36000000002.
[18] Zrnic and E. Candès. Active statistical inference. Proceedings of the 41st International Conference Machine Learning. ICML'24. JMLR.org, 2024.
[19] D. Csillag, C. Jose Struchiner, and G. Tegoni Goedert. Prediction-powered e-values. In Proceedings of the 42nd International Conference on Machine Learning. 2025.
[20] A. Dalal, P. Blöbaum, S. Kasiviswanathan, and A. Ramdas. Anytime-Vall Valid Inference for Double/Debiased Machine Leamingg of Causal Parameters. 2408022408.09598, 2024. doi: 10.485550/rxxv.24488 095998.
[21] Howard, A. Ramdas, J. McAuliffe, and J. Sekhon. Time-uniform, nonparametric, nonasymp- totic confidence sequences. The Annals of Statistics, 49(2), 2021.

---

[22] J. Komlós, P. Major, and G. Tusnády. An approximation of partial sums of independent RV' -s. and the sample DF. I. Zeitschrift für Vatrschhenniccheesstheeeee und Verwandte Gebiete. 32(1): 111-131, March 1975. ISSN 1432-2064.. doi: 10. 10.1007/BF005330933
[23] P. Major. The approximation of partial sums of independent RV's. Zeitschrift für Wahrschein- lichkeitstheorie und Verwandte Gebiele, 35(3):213-223 September 1976. ISSN 1432-2064.
doi: 10.1007/BF00532673.
[24] A. Bibaut, N. Kallus, and M. Lindon. Near-optimal non non-parametric sequential tests and confidence sequences with possibly dependent observations. arXiv preprint arXiv:2212.14411, 2022.
[25] Glasserman, Monte Carlo Methods in Financial Engineering. Springer, 2003.
[26] A. Dal Pozzolo, O. Caelen, R. Johnson, and G. Bontempi. Calibrating probability with undersampling for unbalanced classification. In 2015 IEEE symposium series on computational intelligence, pages 159-166. IEEE, 2015.
[27] R. Lam, A. Sanchez-Gonzalez M. Willson, P. Wirnsberger, M. Fortunato. F. Alet, S. Ravuri, Ewalds, Z. Faton-Rosen, W. Hu, A. Merose, S. Hoyer, G. Holland, O. Vinyals, Stott, A. Pritzel, S. Mohamed, and P. Battaglia. Learning skillful medium-range global weather forecasting. Science, 382(6677)::1166 1421, 2023. doi: 10.1 /..11///ieence.dd223336
[28] M. Jumper. R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool,
R. Bates, A. Zidek, A. Potapenko, A. Bridgland, Meyer, S. A A Kohl, Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals,
A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021.
[29] H. Wang and A. Ramdas. The extended Ville's inequality for nonintegrable nonnegative supermartingales. arXiv preprint mxXiv2234.001663 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the paper's actual contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:The limitations are discussed in our Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification: Proofs of all results are included in full detail in the supplementary material, except for some very classical results for which we provide only references. When relevant, we also provide a sketch of the proof in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: As stated in the supplementary material, the code used to perform our experi- ments made available online under a permissive licence.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
14

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As stated in the supplementary material, the code used to perform our experi- ments made available online under permissive licence.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes]
Justification: details necessary to understand the results are provided in the paper. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For all experiments, results averaged many repetitions (100 1000 repetitions, depending on the experiment). Variations from the mean are negligible. Statisti- cal guarantees (i.e., asymptotic time-uniform coverage) are checked empirically computing the average cumulative miscoverage rate for all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments were run locally on Apple Silicon M4 Pro CPU with 24GB of memory, and implementation details are provided in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. hwtp:////cc/tcccccttutuieeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper is mainly thcoretical and uses only publicly available datasets, which do not contain any sensitive information.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work performed mainly theoretical, and we do not foresee any socictal impact.
Guidelines:
16

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work performed is mainly theoretical and doesn't ' pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets have permissive licenses and properly credited in the supple- mentary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
17

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: As stated in the supplementary material, the code used to perform our experi- ments is made available online under permissive licence.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p//eearrpppcc///ooeeeeeeee/ //t/////ooeeeeeccc////////..1 for what should or should not be described.
19


Reviews:
[{"review_id": "4XbufZlekb", "reviewer": "Reviewer_YGZJ", "summary": "This paper extends the Prediction-Powered Inference (PPI) framework, which uses machine learning predictions to improve inference efficiency with limited labeled data, to the sequential (anytime-valid) setting. The authors develop new procedures for constructing confidence sequences that remain valid uniformly over time, as opposed to traditional fixed-time intervals. Their approach incorporates Bayesian prior information about prediction quality to further tighten intervals when appropriate, leveraging Ville’s inequality and the method of mixtures. Experiments on both synthetic and real datasets demonstrate the methods.", "questions": "1. **Figure 3 Interpretation**: In Figure 3, all methods show a miscoverage level below 0.1. Could the authors clarify why this is the case? Does this suggest that the proposed and baseline methods are conservative in practice?\n\n2. **Predictor Performance in Real Data**: What is the predictive accuracy of the machine learning model used in the real data experiments? Providing this information would help contextualize the observed efficiency gains of the proposed methods.\n\n3. **Wording on Predictor Accuracy (Line 215)**: The statement, “In many modern applications, extremely accurate black-box predictors are available,” seems overly strong and may not hold in many settings. I recommend revising this claim and/or providing appropriate references to support it.", "limitations": "See weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: : \n1. The paper tackles a relevant open problem by moving PPI from fixed-time confidence intervals to anytime-valid confidence sequences, for online, streaming, and sequential learning setting.\n2. The theory is solid.\n\nWeaknesses:\n1. **Motivation and Practical Relevance**: The motivation for this work is not fully convincing. While the authors position their method for online and sequential settings, the real data experiments do not actually involve true online data; rather, the authors simulate an online scenario from static datasets. This leaves open the question of whether the proposed method addresses genuine challenges encountered in real-world online applications. (or just a combination of PPI and any-time valid inference) The connection between practical needs and the proposed methodology could be better articulated and supported with more realistic examples or use cases.\n2. **Limited Scope of Empirical Demonstration**: The PPI method is applicable to general M-estimation problems. But the paper only demonstrates their methods (extension of PPI) for mean estimation. This is a significant limitation, as it does not provide evidence of the method’s utility in more complex or widely used models, such as linear or logistic regression. Expanding the empirical evaluation to additional settings would strengthen the impact and applicability of the work.\n3. **Dependence on Predictor Quality**: Although prior information can mitigate some issues, the practical efficiency gains over standard approaches still depend heavily on the auxiliary predictor’s accuracy. In practice, guidance for diagnosing or calibrating this prior could be more detailed.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns regarding the paper. However, I did not assign a higher score because the work appears to be primarily theoretical, with limited practical applicability. For instance, in many scientific domains, any-time valid inference is not commonly used, and the examples presented in the paper are synthetic rather than drawn from real-world studies. The absence of citations to scientific literature that employs any-time valid inference further limits the perceived relevance. Additionally, applying the method requires making several nontrivial choices, such as selecting appropriate priors, which may present challenges for practitioners.", "rebuttal": "We thank the reviewer YGZJ for their thorough review and constructive feedback. We are encouraged that they found our work to be theoretically solid and that it addresses a relevant open problem.\n\nBelow, we address the weaknesses and questions raised by the reviewer.\n\n### On Motivation and Practical Relevance\n\nWe thank the reviewer for raising this important point regarding the practical motivation and experimental design. We fully agree that demonstrating real-world utility is essential.\n\nWhile we use static datasets, these were not chosen arbitrarily. Each one reflects an inherently sequential process: nightly astronomical observations for GALAXY, daily price updates for FLIGHT, and regularly refreshed satellite imagery for FOREST.\n\nBecause our method is anytime-valid, it enjoy several desirable properties from inference based on e-values, such as optional stopping. Our experiments focus on evaluating baselines and PPI methods in simulated sequential settings, to allow controlled and transparent performance comparisons. However, anytime-valid methods like ours offer important practical benefits. For instance, they enable practitioners to stop costly data collection as soon as a desired region width is achieved. As clearly shown in our experimental figures, our method can often dramatically lower the stopping time.\n\nOur simulation of sequential data arrival thus faithfully reflects how such data is generated and used in real-world scenarios. This design enables us to rigorously evaluate the core contribution of our work: the construction of anytime-valid confidence sequences that support ongoing monitoring and decision-making as data accumulates. We will ensure that practical use cases are discussed in more detail in the revised manuscript.\n\n### On the Scope of Empirical Demonstration\n\nWe thank the reviewer for the suggestion on how to strengthen the impact of our work. Indeed, our method, like standard PPI, can be applied to any non-degenerate convex estimation task. To show this, as well as to illustrate the performance of our method when the predictor provided is less accurate, we have experimented with the following additional tasks using datasets discussed in [1]:\n- HEALTHCARE ($N = 316215$), whose task is to estimate the parameters of a **logistic regression model**, where the outcome is a health insurance indicator and the covariate is income, and for which the predictor is good.\n- CENSUS ($N = 378091$), whose task is to estimate the parameters of a **linear regression model**, where the outcome is income and the covariate is age, and for which the predictor is poor.\n- GENES ($N = 59150$), whose task is to estimate the **median expression level** of a population of genes, and for which the predictor is poor.\n\nThe experimental procedure follows the same design as in Section 6.2. The table below compares anytime-valid classical inference, unassisted PPI, and Bayes-assisted PPI in terms of the width of the resulting confidence regions at $n = 500, 1000, 2000$ labelled samples, averaged over $100$ repetitions. \n\n| $n = 500/1000/2000$ | HEALTHCARE ($\\times 10^{-5}$) | CENSUS ($\\times 10^2$) | GENES ($\\times 10^{0}$) |\n|---|---|---|---|\n| classical | 2.3/1.5/1.0 | 4.89/3.31/2.29 | 2.02/1.33/0.92 |\n| PPI | 2.0/1.3/0.8 | 4.09/2.79/1.92 | 1.64/0.99/0.67 |\n| PPI (G) | 1.5/1.1/0.7 | 5.79/5.17/4.28 | 3.18/3.05/2.89 |\n| PPI (L) | 1.5/1.1/0.7 | 5.28/4.17/3.05 | 2.90/2.50/1.68 |\n| PPI (T) | 1.5/1.1/0.8 | 4.90/3.75/2.71 | 2.30/1.57/1.09 |\n\nIn particular, unassisted PPI outperforms classical inference on all datasets, showing the potential benefit of PPI in the presence of correlation between predictions and true labels. On the other hand, Bayes-assisted PPI outperforms unassisted PPI on the HEALTHCARE dataset, where the predictor is accurate, but not on the CENSUS and GENES datasets, for which the zero-centered priors are mispecified. In these cases, the behaviour of the three priors considered (Gaussian, Laplace, and Student-t) differs significantly. In particular, as expected, heavier-tailed priors suffer less from the mispecification, achieving a lower width than the Gaussian prior, which is the most sensitive to the predictor's accuracy.\n\nWhile this is not currently possible for this rebuttal, we will include these results in the form of figures similar to that in Section 6.2 to our revised supplementary material. We believe that these results clarify the broad applicability of our method and shed light on the practical implications of prediction accuracy and prior choice.\n\n### On Predictor Dependence and Prior Guidance\n\nThe reviewer raises a key practical point about the predictor's influence and the choice of prior.\n\nThe fundamental benefit of our method holds as long as the predictor's output is even moderately correlated with the true labels. When a highly accurate predictor is available, our Bayes-Assisted Confidence Sequence offers superior performance. In settings where prior information about the predictor's quality is weak or unavailable, the Non-Assisted version of our method still provides performance gains over standard approaches and serves as a robust default.\n\nFor practical guidance on prior selection, we recommend using priors with heavy tails, as they are more robust to mispecification.\nIn particular, the width of the confidence sets induced by polynomial-tailed priors, such as the Student-t, grow logarithmically with the error in the prior mean (see Eq. 8).\nThis is in contrast to the Gaussian and Laplace priors, whose confidence sets grow linearly and with the square root, respectively.\nHowever, the Student-t prior's greater robustness comes at a computational cost, as computing the associated $\\eta_t$ requires numerical integration.\nIn settings where this is a concern, the Laplace prior might represent a compelling alternative, as it exhibits both closed-form $\\eta_t$ and slower confidence set growth rate than the Gaussian prior.\n\n### Responses to Specific Questions\n\n- **Q1: Figure 3 Interpretation:** The reviewer correctly observes that the miscoverage for all methods is below the nominal 0.1 level. This is an expected and desired consequence of guaranteeing anytime-validity. An anytime-valid confidence sequence must ensure coverage for all t. This is a strictly stronger guarantee than fixed-time coverage. To satisfy the \"for all time\" condition, the intervals at any specific time t must be slightly conservative. We will clarify this in the figure caption.\n\n- **Q2: Predictor Performance in Real Data:** The table below provides the following metrics for all the experiments performed (including the new ones above): normalised RMSE (NRMSE) for the regression models (FLIGHT, CENSUS, GENES) and cross-entropy (CE) for the binary classification models (GALAXY, FOREST, HEALTHCARE).\n\n| FLIGHT | CENSUS | GENES | GALAXY | FOREST | HEALTHCARE |\n|---|---|---|---|---|---|\n| 0.20 | 1.45 | 0.29 | 0.29 | 0.31 | 0.35 |\n\n- **Q3: Wording on Predictor Accuracy (Line 215):** We appreciate the feedback on this statement, we will substantiate the claim by citing prominent examples where highly accurate predictors are available \n\n    - AlphaFold for protein folding: Jumper, John M. et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596 (2021): 583 - 589.\n\n    - GraphCast for weather forecasting: Lam, Remi et al. “Learning skillful medium-range global weather forecasting.” Science (2023): 1416-1421.\n\n    - Large-scale models for fraud detection: Pozzolo, A. D., Caelen, O., et al. (2015). \"Calibrating Probability with Undersampling for Unbalanced Classification\". 2015 IEEE Symposium Series on Computational Intelligence and Data Mining (SSCI)\n\n\nWe will also re-emphasize that our framework is still beneficial even with moderately accurate predictors and includes a non assisted method for when predictor quality is unknown.\n\n\nWe believe these changes will address the reviewer's concerns and substantially improve the paper's clarity and impact. We thank the reviewer again for their valuable guidance.\n\n[1] Anastasios N. Angelopoulos and Stephen Bates and Clara Fannjiang and Michael I. Jordan and Tijana Zrnic. Prediction-Powered Inference. \tarXiv:2301.09633, 2023."}, {"review_id": "mToqTiNUu0", "reviewer": "Reviewer_qqUX", "summary": "This study extends the framework of Prediction-Powered Inference (PPI) to a sequential setting, introducing methods for constructing anytime-valid confidence sequences that incorporate predictions from black-box models. The approach builds on control variates and leverages the method of mixtures and Ville’s inequality. The authors also develop Bayes-assisted variants that incorporate prior beliefs about prediction quality, aiming to improve the efficiency of inference when predictions are accurate. The methods are evaluated on both synthetic and real data.", "questions": "Here are a few points that addressed would help the reader to better evaluate the scope and applicability of the proposed methods:\n\n- The method involves using prior information about the predictions to improve inference. Could the authors give some more intuition or examples for how to choose a good prior in practice? It is not immediately clear how one would decide between different types, and some guidance would be appreciated.\n\n- When the prior is misspecified, the experiments suggest that heavier-tailed priors help. Could this be formalised, or is there a general recommendation for robustness?\n\n- The multivariate setting is discussed in the supplement, but there are no experiments illustrating it. Including one or two practical examples would strengthen the paper.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The study addresses the relevant problem of performing valid statistical inference in settings where data arrives over time and predictions from machine learning models are available. The methodological framework is well-motivated and appears theoretically solid, though I did not check the proofs in detail. The extension from fixed-time to time-uniform (anytime-valid) inference is significant, and the use of prior information to tighten intervals seems both novel and potentially impactful.\n\nThe presentation is generally clear. However, the technical depth may make the paper challenging to follow for readers not familiar with the literature on confidence sequences or prediction-powered inference. A few parts, particularly around the practical aspects of prior selection and computational cost, could be expanded.\n\nEmpirical results support the theoretical claims. That said, the experiments focus on scalar parameters, and while multivariate extensions are discussed in the supplement, no empirical results are shown. Including such results would be important to demonstrate the generality of the approach.", "was_revised": true, "final_justification": "I believe the authors have appropriately addressed my points, and maintain that this is a strong contribution. I thus keep my recommendation for acceptance.", "rebuttal": "We sincerely thank the reviewer for their positive feedback and for recognizing the relevance of our work. \n\n### About the technical depth of the paper\n\nAs mentioned by the reviewer, our presentation is \"generally clear\", but we acknowledge that our paper is technically deep and that reading may be challenging for a reader not completely familiar to the subject. To address this issue we will add additional background on PPI and Confidence Sequences that should make this paper accessible to a wider audience. \n\n### About the prior selection\n\nAs mentioned in our response to YGZJ, for practical guidance on prior selection, we recommend using priors with heavy tails, as they are more robust to mispecification.\nIn particular, the width of the confidence sets induced by polynomial-tailed priors, such as the Student-t, grow logarithmically with the error in the prior mean (see Eq. 8).\nThis is in contrast to the Gaussian and Laplace priors, whose confidence sets grow linearly and with the square root, respectively.\nHowever, the Student-t prior's greater robustness comes at a computational cost, as computing the associated $\\eta_t$ requires numerical integration.\nIn settings where this is a concern, the Laplace prior might represent a compelling alternative, as it exhibits both closed-form $\\eta_t$ and slower confidence set growth rate than the Gaussian prior\n\n### About the computational cost\n\nAs mentioned in Section 7, the main drivers of computational cost for our method are the iteration over a grid of values for $\\theta$, which is required for some estimation problems, like for standard PPI, and the computation of $\\eta_t$, which is required for the Bayes-assisted methods. In particular, when gridding is required, the numbers of evaluations of $\\eta_t$ scales linearly with both the grid size and the number of labelled samples $n$. For heavy-tailed priors, such as the Student-t, where $\\eta_t$ is computed via numerical integration, this can become quite expensive. We will make sure to clarify this in the revised manuscript.\n\n### About our multivariate extension\n\nTo illustrate the multivariate extension of our method, we study a multivariate version of the mean estimation task with biased predictions in Section 6.1. In particular, we sample $5$-dimensional $X_i \\overset{iid}{\\sim} \\mathcal{MVN}(\\mathbf{0}, \\Sigma)$, where $\\Sigma \\in \\mathbb{R}^{5 \\times 5}$ is a Toeplitz covariance matrix with entries $\\Sigma_{ij} = 0.5^{|i-j|}$, and define $Y_i = X_i + \\epsilon_i$, where $\\epsilon_i$ is sampled i.i.d.~from a standard normal distribution, so that $\\theta^* = \\mathbb{E}[Y] = \\mathbf{0}$. Then, we proceed as in Section 6.1 and define biased predictions $f(X_i) = X_i + \\nu$, where $\\nu \\in \\mathbb{R}$ controls the bias level of the predictor.\n\nThe table below compares anytime-valid multivariate classical inference, unassisted PPI, and Bayes-assisted PPI under a Gaussian prior in terms of the volume of the resulting confidence ball at $n = 100$ and $N = \\infty$ when varying $\\nu$ between $0$ and $5.0$, averaged over $1000$ repetitions.\n\n| $\\nu$ | 0.0 | 1.0 | 2.0 | 3.0 | 4.0 | 5.0 |\n|---|---|---|---|---|---|---|\n| classical | 252.6 | 256.4 | 251.3 | 254.3 | 252.7 | 252.3 |\n| PPI | 24.4 | 24.2\t| 24.4 | 24.0 | 24.3 | 24.1 |\n| PPI (G) | 0.06 | 0.36 | 3.8 | 21.6 | 85.9 | 241.4 |\n\nThe behaviour of the methods considered follows the same pattern as in the scalar case. In particular, unassisted PPI always outperforms classical inference, while maintaining roughly the same volume across all values of $\\nu$. On the other hand, Bayes-assisted PPI outperforms unassisted PPI when the predictor is accurate ($\\nu = 0,\\dots,3$), but eventually suffers from the mispecification of the zero-centered prior when the predictor is poor ($\\nu = 4,5$).\n\nWhile this is not currently possible for the rebuttals, we will include this result in the form of a figure similar to Figure 2 in Section 6.2 to our revised supplementary material.\n\nWe believe these changes will address the reviewer's concerns and substantially improve the paper's clarity and impact. We thank the reviewer again for their valuable guidance."}, {"review_id": "sxfFRBn9pZ", "reviewer": "Reviewer_1TNz", "summary": "The authors propose asymptotic anytime-valid confidence sequences that leverage (i) user-specified priors for the signal-to-noise ratio (defined by mean / standard deviation) and (ii) black-box predictors for variance reduction. The work first provides closed-form asymptotic confidence sequences for a user-specified prior with similar techniques to existing works in the literature. Building on top of these results, the authors apply their prior-based confidence sequences to PPI-based estimation, and provide theoretical results on the validity of their approach. Using both synthetic and real-world examples, the authors demonstrate when their approach outperforms existing methods in the literature, highlighting both the benefits and drawbacks of using the PPI-based framework and user-specified priors.", "questions": "* A large benefit of anytime-valid and asymptotic anytime-valid methods are that they remain valid beyond i.i.d. data collection (i.e. data collected under time-varying sampling policies such as contextual bandits). In such settings, do the proposed approaches maintain their anytime-validity guarantees?\n* As with traditional mixture-martingale methods, the user-specified prior on the signal-to-noise ratio must be specified in advance by the analyst. Are there potential extensions that update the prior in an online fashion, while still maintaining the desired anytime-validity guarantees?", "limitations": "As pointed out in weaknesses, the authors use a definition of asymptotic anytime validity that does not explicitly provide any type I error guarantees - only that the sequence converges to some unknown anytime-valid confidence sequence. Generally, asymptotic type I error guarantees require stronger conditions than the goals of Defn 2. I would be willing to raise my score if the authors could elaborate on the additional assumptions needed to provide error guarantees across their approaches.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* The authors provide novel results for constructing asymptotic confidence sequences with user-specified priors and methods for leveraging black-box predictors for variance reduction. In settings where a plethora of data and/or models are available before experimentation, the approaches provided in this work enable reductions in confidence sequence width.\n* The guarantees of this approach are valid even under poor-quality/noisy predictors, making them applicable across many settings. In particular, while performance depends on the quality of the black-box predictors, the validity of the confidence sequences do not, allowing them to be a potentially conservative option regardless of setting.\n\n**Weaknesses**\n* While the authors demonstrate that their approaches are asymptotic confidence sequences (as defined in Defn. 2), they do not provide any type I error guarantees regarding the performance of their approach. Note that to provide error guarantees (such as anytime-valid type I error protection beyond a burn-in time diverging to infinity), one generally requires stronger assumptions than achieving the goals of Definition 2, as discussed in [1] and Assumption 3-$\\eta$ in [2]. \n* The title may be slightly misleading: this work is only concerned with asymptotic anytime-valid inference, rather than standard anytime-valid methods that protect type I error across the entire experiment horizon. \n\n[1] Aurelien Bibaut, Nathan Kallus, and Michael Lindon. Near-optimal non-parametric sequential tests and confidence sequences with possibly dependent observations, 2024.\n[2] I. Waudby-Smith, D. Arbour, R. Sinha, E. H. Kennedy, and A. Ramdas. Time-uniform central limit theory and asymptotic confidence sequences. The Annals of Statistics, 52(6):2613–2640, 2024.", "was_revised": true, "final_justification": "For the recommended score, the authors provided a detailed response regarding conditions for type I error control that were not present in the original manuscript. These additions significantly strengthen its theoretical contributions, and therefore, I would recommend acceptance.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Their comments are insightful and will help us improve our manuscript. We address each point below.\n\n\n\n### On Asymptotic Type I Error Guarantees\n\nWe thank the reviewer for highlighting the question of Type I error control. After further investigation, we confirm that one can have type-I error control protection beyond a burn-in time diverging to infinity. As the reviewer anticipated, and as in Bibaut et al. (2024) and Waudby-Smith et al. (2024), this result requires slightly stronger assumptions than those stated in the main text. Fortunately, these stronger conditions already appear in Section S4 of the supplementary material, where they underpin confidence sequences with sharper convergence rates. \n\nThe key assumption (see Section S4.4) is to assume that the random variables $U_{\\theta,i}=\\ell_\\theta'(X_i,f(X_i))$ and $V_{\\theta,i}=\\ell'_\\theta(X_i,Y_i)$ possess absolute moments of order $2+\\delta$, for some $\\delta>0$. Under this condition, we obtain the following theorem for the anytime-valid PPI confidence sequence presented in Section 5.1; a sketch of the proof is provided below. An analogous result holds for the Bayesian procedures of Section 5.2 under the same assumptions, in addition to the already stated assumptions on the prior.  We will incorporate these theorems into Section S4 of the supplement. We hope this addresses the reviewer's concern.  \n\n__Theorem__ [Type-1 control for non-assisted PPI confidence sequence]. Assume $\\mathbb E[|U_{\\theta,i}|^{2+\\delta}]<\\infty$ and $\\mathbb E[|V_{\\theta,i}|^{2+\\delta}]<\\infty$ for some $\\delta>0$. Assume that $|n/N_n-r|=O(1/n^{1-a})$ with $0<a<2/(2+\\delta)$ for some $r\\in[0,1]$. Let $\\widehat g_{\\theta,n}$ and $\\widehat \\sigma_{\\theta,n}^g$ be as defined in Proposition 3 of our paper. Then, for any $\\rho>0$, \n\n$$\n\\liminf_{m\\to\\infty} \\Pr(\\forall n\\geq m~~g_\\theta\\in \\mathcal C_{\\alpha,\\theta,n}^{\\text{NA}}(\\widehat g_{\\theta,n},\\widehat\\sigma_{\\theta,n}^g,\\rho))\\geq 1-\\alpha.\n$$\n\nFurthermore, for $\\rho_m = \\sqrt{\\frac{-2\\log\\alpha+\\log(-2\\log\\alpha)+1}{m\\log(m\\vee e)}}$ choosen as in Section 2.5 of Waudby-Smith et al. (2024), we have:\n\n$$\n\\liminf_{m\\to\\infty} \\Pr(\\forall n\\geq m,~~g_\\theta\\in \\mathcal C_{\\alpha,\\theta,n}^{\\text{NA}}(\\widehat g_{\\theta,n},\\widehat\\sigma_{\\theta,n}^g,\\rho_m))\\geq 1-\\alpha.\n$$\n\n__Sketch of proof__. By the Marcinkiewicz-Zygmund strong law of large numbers, we have $\\widehat \\sigma_{\\theta,n}^g-\\sigma_{\\theta}^g=o(1/\\log n)$ for some $\\sigma_\\theta^g$. Additionally, by the results of Section S4 in the supplementary material, the stated assumptions imply that the control-variate estimator $\\widehat g_{\\theta,n}$ satisfies the tighter strong coupling in Equation (S18)\n\n$$\n\\widehat g_{\\theta,n}=g_\\theta +  \\frac{\\sigma_{\\theta}^g}{n}\\sum_{i=1}^n W_i +o(\\frac{1}{\\sqrt{n\\log n}})\n$$\n\nfor some i.i.d. zero-mean, unit-variance, Gaussian random variables $(W_i)$. The proof then follows similarly from that in Appendix A7 of Waudby-Smith et al. (2024). Writing $\\widehat g_{\\theta,t}'$=$\\widehat g_{\\theta,\\lfloor t\\rfloor}$ and $\\widehat \\sigma_{\\theta,t}^{g}{}'$=$\\widehat \\sigma_{\\theta,\\lfloor t\\rfloor}^{g}$ for $t\\in\\mathbb R,t\\geq 1$, we have the continuous-time coupling\n\n$$\n\\widehat g_{\\theta,t}'=g_\\theta +  \\frac{\\sigma_{\\theta}^g}{t}B_t +o(\\frac{1}{\\sqrt{t\\log t}})\n$$\n\nwhere $B_t$ is a standard Brownian motion. We have\n\n$$\n\\Pr(\\forall n\\geq m,~g_\\theta\\in \\mathcal C_{\\alpha,\\theta,n}^{\\text{NA}}(\\widehat g_{\\theta,n},\\widehat\\sigma_{\\theta,n}^g,\\rho))\n$$\n\n$$\n=\\Pr(\\forall t\\geq \\mathbb R^{\\geq m},~|\\widehat g_{\\theta,t}'-g_\\theta|\\leq \\frac{\\widehat\\sigma_{\\theta,t}^g{}'}{\\sqrt{t}}\\sqrt{(1+\\frac{1}{t\\rho^2})\\log(\\frac{t\\rho^2+1}{\\alpha^2})} )\n$$\n\n$$\n=\\Pr(\\forall t\\in \\mathbb R^{\\geq m},~|\\frac{\\sigma_{\\theta}^g}{t}B_t +o(\\frac{1}{\\sqrt{t\\log t}})|\\leq \\frac{\\sigma_{\\theta}^g}{\\sqrt{t}}\\sqrt{(1+\\frac{1}{t\\rho^2})\\log(\\frac{t\\rho^2+1}{\\alpha^2})} + o(\\frac{1}{\\sqrt{t\\log t}} ))\n$$\n\n$$\n=\\Pr(\\forall t\\in \\mathbb R^{\\geq m},~|\\frac{\\sigma_{\\theta}^g}{t}B_t +o(\\frac{1}{\\sqrt{t\\log t}})|\\leq \\frac{\\sigma_{\\theta}^g}{\\sqrt{t}}\\sqrt{(1+\\frac{1}{t\\rho^2})\\log(\\frac{t\\rho^2+1}{\\alpha^2})} + o(\\frac{1}{\\sqrt{t\\log t}} ))\n$$\n\n$$\n=\\Pr(\\forall t\\in \\mathbb R^{\\geq m},~|B_t +o(\\frac{\\sqrt{t}}{\\sqrt{\\log t}})|\\leq \\sqrt{(t+\\frac{1}{\\rho^2})\\log(\\frac{t\\rho^2+1}{\\alpha^2})} + o(\\frac{\\sqrt{t}}{\\sqrt{\\log t}} ))\n$$\n\nLetting $t=s d_m$ where $d_m=m\\log m$, the above probability can be written as \n\n$$\n\\Pr(\\forall s\\geq 1/\\log m,~|B_s +o(\\sqrt{\\frac{s}{\\log s d_m}})|\\leq \\sqrt{(s+\\frac{1}{d_m \\rho^2})\\log(\\frac{s d_m \\rho^2+1}{\\alpha^2})} + o(\\sqrt{\\frac{s}{\\log s d_m}}) )\n$$\n\nFor a fixed $\\rho$, the right-handside bound goes to infinity as $m\\to\\infty$. Hence the probability goes to 1 and satisfies the type-I coverage. If $\\rho=\\rho_m$, we obtain \n\n$$\n\\Pr(\\forall s\\geq 1/\\log m,~|B_s +o(\\sqrt{\\frac{s}{\\log s d_m}})|\\leq \\sqrt{(s+\\frac{1}{c_\\alpha})\\log(\\frac{s c_\\alpha+1}{\\alpha^2})} + o(\\sqrt{\\frac{s}{\\log s d_m}}) )\n$$\n\nwhere $c_\\alpha=-2\\log\\alpha+\\log(-2\\log\\alpha)+1$. Using standard results on Brownian motion (Robbins and Siegmund):\n\n$$\n\\Pr(\\forall s\\geq 0,~|B_s|\\leq \\sqrt{(s+\\frac{1}{c_\\alpha})\\log(\\frac{s c_\\alpha+1}{\\alpha^2})})=1-\\alpha,\n$$\n\nwe obtain the required lower bound. \n\n\n\n\n\n\n\n### On the Title\n\nWe agree that it is important to state the *asymptotic* nature of our guarantees explicitly. To make this clear from the outset, we will revise the abstract as follows.\n\n**Revised Abstract:**\n\n>Given a large pool of unlabelled data and a smaller amount of labels, prediction-powered inference (PPI) leverages machine learning predictions to increase the statistical efficiency of standard confidence interval procedures based solely on labelled data, while preserving their fixed-time validity.\nIn this paper, we extend the PPI framework to the sequential setting, where labelled and unlabelled datasets grow over time.\nExploiting Ville's inequality and the method of mixtures, we propose prediction-powered confidence sequence procedures that are **asymptotically** valid uniformly over time and naturally accommodate prior knowledge on the quality of the predictions to further boost efficiency.\nWe carefully illustrate the design choices behind our method and demonstrate its effectiveness in real and synthetic examples.\n\nWe believe this change clarifies the scope of our contribution. If the reviewers and the chair feel it is still necessary, We are happy to adjust the title to ``**Asymptotic** Anytime-valid, Bayes-assisted, Prediction-Powered Inference'', although we note that this may become somewhat lengthy.\n\n### On the Questions for Future Work\n\n__Extension beyond iid data__\n\nOur assumptions mirror those in the standard PPI framework, in which the data are assumed to be i.i.d. The strong coupling results (e.g., Proposition 2) are derived under this assumption. Extending the analysis beyond the i.i.d. case, as suggested by the reviewer, is an interesting research direction that would require careful investigation and an adaptation of the coupling results.\n\n__Update the prior in an online fashion__\n\nIn the current method the prior is fixed. Designing an online-adaptive prior that preserves both the tractability of Equation (8) and the asymptotic guarantees appears non-trivial; we regard this as promising future work."}, {"review_id": "d2QeuQ3EDQ", "reviewer": "Reviewer_BzQ5", "summary": "This paper introduces a framework for performing statistically valid inference based on machine learning prediction and valid at any point in time. This is part of the Prediction-Powered Inference (PPI) framework, where the general idea is to leverage large unlabeled dataset to improve stastistical efficiency. The main limitation of the existing approach is that it works at \"fixed-time\".\n\nThe main contribution is the development of anytime-valid PP confidence sequences for the sequential setting. Importantly, these sequence keep their statistical coverage guarantees across all sample sizes. This is achieved by connecting the PPI framework to the theory of control variates and with modern tools for constructing asymptotic confidence sequences, which is established in multiple propositions and derivations throughout the paper.\n\nAnother contribution is the Bayes-assisted extension of the method. This uses the \"method of mixtures\" to incorporate Bayesian priors about the accuracy of the prediction. Crucially, when the prior is veridical, confidence intervals can become narrower; but the CI remain valid even if the prior is misspecified.\n\nFinally, the paper validates their method with some experiments on synthetic an real data, showing the efficacy of their approach.", "questions": "As mentioned above, the main issue with the paper is its density and it would be unfair to penalize the submission for that, although I do believe that this paper would fare much better in a longer journal format. So I guess this is not very actionable.\n\nWhat is more actionable is strengthening the empirical validation, for example presenting results beyond the mean estimator, and discussing whether and how these results could be extended in the future to a non-scalar quantity (this is only briefly mentioned in passing in the Discussion).", "limitations": "The authors might highlight more (unless they address it) some of the limitations of the paper, such as the focus on scalar quantities.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This is a strong, extremely dense theoretical paper tackling an interesting problem and combining state-of-the-art results of interest at the intersection of statistics and machine learning. This is a high-quality contribution, significant and original, to my knowledge and understanding -- admittedly quite limited in this field.\n\nThe only major issue I found is that the paper is clarity, in that the paper is quite dense and possibly not very well suited for the conference format, whereas it would likely shine more in a journal format with adequate space to breathe. The lack of space is a major constraint and, at least for a non-expert, the reading is quite painful with a non-stop barrage of definitions and propositions, with little explanation or justification. This is not the authors' fault, whose writing is otherwise quite crisp and it's clear they could do better given more space.\n\nThe other issue I found is somewhat limited experimental validation. The theory is general, but the empirical validation focuses on scalar mean estimation problems. It seems that extending beyond the mean would require additional approximations (e.g., constructing a grid of values), but these are just mentioned in the Discussion. Also note that another point worth mentioning more clearly is that the entire theory focuses on a *scalar* estimate.\n\n### Minor comments\n\n- Shouldn't the $\\eta_n$ term in Eqs (22) and (23) be squared (following from Eq. 8)?", "was_revised": true, "final_justification": "The work is theoretically sound and interesting, and the authors have provided additional evidence and results that addressed concerns.", "rebuttal": "We sincerely thank the reviewer for their positive and constructive feedback, and for recognizing our work as a ``high-quality contribution, significant and original.\" We appreciate the insightful comments, which will help us improve the paper. We address the specific points below.\n\n### About the technical depth of the paper:\n\nWe agree with the reviewer that the paper is dense due to the nature of the topic and the space constraints of the conference format. We appreciate the reviewer's understanding on this front. To improve the paper's accessibility for readers less familiar with this area, we will incorporate additional background material on Prediction-Powered Inference (PPI) and Confidence Sequences (CS) to make the paper more self-contained.\n\n### Expanding the Empirical Validation\nWe thank the reviewer for the suggestion on how to strengthen the impact of our work. Indeed, our method, like standard PPI, can be applied to any non-degenerate convex estimation task. To show this, as well as to illustrate the performance of our method when the predictor provided is less accurate, we have experimented with the following additional tasks using datasets discussed in [1]:\n- HEALTHCARE ($N = 316215$), whose task is to estimate the parameters of a **logistic regression model**, where the outcome is a health insurance indicator and the covariate is income, and for which the predictor is good.\n- CENSUS ($N = 378091$), whose task is to estimate the parameters of a **linear regression model**, where the outcome is income and the covariate is age, and for which the predictor is poor.\n- GENES ($N = 59150$), whose task is to estimate the **median expression level** of a population of genes, and for which the predictor is poor.\n\nThe experimental procedure follows the same design as in Section 6.2. The table below compares anytime-valid classical inference, unassisted PPI, and Bayes-assisted PPI in terms of the width of the resulting confidence regions at $n = 500, 1000, 2000$ labelled samples, averaged over $100$ repetitions. \n\n| $n = 500/1000/2000$ | HEALTHCARE ($\\times 10^{-5}$) | CENSUS ($\\times 10^2$) | GENES ($\\times 10^{0}$) |\n|---|---|---|---|\n| classical | 2.3/1.5/1.0 | 4.89/3.31/2.29 | 2.02/1.33/0.92 |\n| PPI | 2.0/1.3/0.8 | 4.09/2.79/1.92 | 1.64/0.99/0.67 |\n| PPI (G) | 1.5/1.1/0.7 | 5.79/5.17/4.28 | 3.18/3.05/2.89 |\n| PPI (L) | 1.5/1.1/0.7 | 5.28/4.17/3.05 | 2.90/2.50/1.68 |\n| PPI (T) | 1.5/1.1/0.8 | 4.90/3.75/2.71 | 2.30/1.57/1.09 |\n\nIn particular, unassisted PPI outperforms classical inference on all datasets, showing the potential benefit of PPI in the presence of correlation between predictions and true labels. On the other hand, Bayes-assisted PPI outperforms unassisted PPI on the HEALTHCARE dataset, where the predictor is accurate, but not on the CENSUS and GENES datasets, for which the zero-centered priors are mispecified. In these cases, the behaviour of the three priors considered (Gaussian, Laplace, and Student-t) differs significantly. In particular, as expected, heavier-tailed priors suffer less from the mispecification, achieving a lower width than the Gaussian prior, which is the most sensitive to the predictor's accuracy.\n\nWhile this is not currently possible for this rebuttal, we will include these results in the form of figures similar to Figure 3 in Section 6.2 to our revised supplementary material. We believe that these results clarify the broad applicability of our method and shed light on the practical implications of prediction accuracy and prior choice.\n\n### Clarification on the Multivariate Extension\nWe thank the reviewer for raising this important point and apologize that our presentation made this seem like a limitation. Our framework is not limited to scalar quantities.\n\nA multivariate generalization of our theory is indeed available, but, given its greater technical depth, and for the sake of presentation clarity, is provided in Section S5 of the supplementary material. We recognize this was not sufficiently highlighted in the main text. In the revision, we will update the main paper, particularly the discussion section, to make the existence and location of these important multivariate results much more prominent.\n\nAdditionally, to illustrate the multivariate extension of our method in practice, we study a multivariate version of the mean estimation task with biased predictions in Section 6.1. In particular, we sample $5$-dimensional $X_i \\overset{iid}{\\sim} \\mathcal{MVN}(\\mathbf{0}, \\Sigma)$, where $\\Sigma \\in \\mathbb{R}^{5 \\times 5}$ is a Toeplitz covariance matrix with entries $\\Sigma_{ij} = 0.5^{|i-j|}$, and define $Y_i = X_i + \\epsilon_i$, where $\\epsilon_i$ is sampled i.i.d.~from a standard normal distribution, so that $\\theta^* = \\mathbb{E}[Y] = \\mathbf{0}$. Then, we proceed as in Section 6.1 and define biased predictions $f(X_i) = X_i + \\nu$, where $\\nu \\in \\mathbb{R}$ controls the bias level of the predictor.\n\nThe table below compares anytime-valid multivariate classical inference, unassisted PPI, and Bayes-assisted PPI under a Gaussian prior in terms of the volume of the resulting confidence ball at $n = 100$ and $N = \\infty$ when varying $\\nu$ between $0$ and $5.0$, averaged over $1000$ repetitions.\n\n| $\\nu$ | 0.0 | 1.0 | 2.0 | 3.0 | 4.0 | 5.0 |\n|---|---|---|---|---|---|---|\n| classical | 252.6 | 256.4 | 251.3 | 254.3 | 252.7 | 252.3 |\n| PPI | 24.4 | 24.2\t| 24.4 | 24.0 | 24.3 | 24.1 |\n| PPI (G) | 0.06 | 0.36 | 3.8 | 21.6 | 85.9 | 241.4 |\n\nThe behaviour of the methods considered follows the same pattern as in the scalar case. In particular, unassisted PPI always outperforms classical inference, while maintaining roughly the same volume across all values of $\\nu$. On the other hand, Bayes-assisted PPI outperforms unassisted PPI when the predictor is accurate ($\\nu = 0,\\dots,3$), but eventually suffers from the mispecification of the zero-centered prior when the predictor is poor ($\\nu = 4,5$).\n\nWhile this is not currently possible for the rebuttals, we will include this result in the form of a figure similar to Figure 2 in Section 6.2 to our revised supplementary material.\n\n### Correction of Typo in Equations\n\nThe reviewer is absolutely correct. The term $\\eta_t$ in Equations (22) and (23) should indeed be squared, as it follows from the variance definition in Equation (8). We are very grateful for this careful catch and will correct the typo in the revision.\n\nWe believe these changes will address the reviewer's concerns and improve the paper's clarity and impact. We thank the reviewer again for their valuable guidance.\n\n[1] Anastasios N. Angelopoulos and Stephen Bates and Clara Fannjiang and Michael I. Jordan and Tijana Zrnic. Prediction-Powered Inference. arXiv:2301.09633, 2023."}]

Author rebuttal:

Meta-review: This work extends the Prediction-Powered Inference (PPI) framework to the sequential (anytime-valid) setting. The approach incorporates prior information about prediction quality to adjust interval sizes and the approach is validated on both real and synthetic data sets.

Strengths:  The reviewers felt that theory was robust and noted that the theoretical guarantees are valid even under poor-quality/noisy predictors, which increases applicability.

Weaknesses:  The theoretical results can stand on their own, but the experiments are quite limited.  A better set of experiments would bolster the more practical side of this work and demonstrate its claimed wide applicability.  Some reviewers, outside of this subarea, felt that the contributions could be better motivated for the ML community at large.

Overall:  While the experimental results are limited, the reviewers felt that the theoretical contributions were sufficiently novel and widely applicable. The authors provided asymptotic Type I error guarantees in their rebuttal, which further strengthen their contribution.
Decision: Accept (poster)