Prediction-Powered E-Values
Authors: Daniel Csillag
Claudio Jose Struchiner
Guilherme Tegoni Goedert
Venue: ICML 2025 poster (2025)
Keywords: prediction-powered inference
e-values
statistical inference
distribution-free methods
Primary area: probabilistic_methods->everything_else
TL;DR: We extend prediction-powered inference to e-values, vastly expanding the set of inference tasks achievable in a prediction-powered manner while also benefiting from the usual virtues of e-values.

Abstract: Quality statistical inference requires a sufficient amount of data, which can be missing or hard to obtain. To this end, prediction-powered inference has risen as a promising methodology, but existing approaches are largely limited to Z-estimation problems such as inference of means and quantiles. In this paper, we apply ideas of prediction-powered inference to e-values. By doing so, we inherit all the usual benefits of e-values -- such as anytime-validity, post-hoc validity and versatile sequential inference -- as well as greatly expand the set of inferences achievable in a prediction-powered manner. In particular, we show that every inference procedure that can be framed in terms of e-values has a prediction-powered counterpart, given by our method. We showcase the effectiveness of our framework across a wide range of inference tasks, from simple hypothesis testing and confidence intervals to more involved procedures for change-point detection and causal discovery, which were out of reach of previous techniques. Our approach is modular and easily integrable into existing algorithms, making it a compelling choice for practical applications.

Prediction-Powered E-Values
Daniel Csillag"
Claudio José Struchiner Guilherme Tegoni Goedert'
Abstract
to "compensate" for the missing expensive data. Classical works in this direction include single imputation and mul- tiple imputation methods (Little & Rubin, 2019), but they generally lack any strong guarantee of correctness. More recently, (Angelopoulos et al., 2023a) proposed prediction- powered inference, which allow for versatile procedures that benefit from strong correctness guarantees, notably includ- ing unbiasedness and type-I error control under very light assumptions.
Quality statistical inference requires a sufficient amount of data, which can be missing or hard to obtain. To this end, prediction-pon powlered inference has risen promising methodology, but exist- ing approaches are largely limited Z-estimation problems such as inference of means and quan- tiles. In this paper, we apply ideas of prediction- powered inference to e-values. By doing so, we inherit all the usual benefits e-values - such as anytime-validity, post-hoc hoc validity and versatile sequential inference- as well as greatly expand the set of inferences achievable in a prediction- powered manner. In particular, we show that every inference procedure that can be framed in terms of e-values has prediction-powered counterpart, given by our method. We showcase the effective- ness of our framework across wide range of in- ference tasks, from simple hypothesis testing and confidence intervals more involved procedures for change-point detection and causal discovery, which were out of reach of previous techniques. Our approach is modular and easily integrable into existing algorithms, making it a compelling choice for practical applications.
At its heart, the idea of perdiction-poweett inference is simple: we leverage predictive model (which can be arbi- trarily complex, e.g., large neural networks) to predict the expensive data from the cheap data. We can then use our whole dataset to perform our inference by imputing missing expensive data with predictions from our model, while lever- aging the available expensive data to quantify our model's inaccuracies, debiasing our inference.
Prediction trediction-powered inference has already inspired large amount of literature, both methodology-wise (c.g.. (Zric & Candes, 2024; Angelopoulos et al., 2023b; Zrnic & Candès, 2023; Gu & Xia, 2024)), as well as applications such as language model evaluations (Chatzi et al., 2024; Boyeau et al., 2024), genome-wide association studies (Miao et al., 2024) and more. However, throughout, the inference tasks considered are fairly limited; previous works are essentially restricted to problems that can be framed in terms of Z- estimation,' which includes many common tasks such as inference means, quantiles and regression coefficients, but not much more. In this paper, we significantly expand this frontier by applying prediction-powered inference to e-values.
1. Introduction
Statistical inference is ubiquitous in many critical areas of application, such as medicine and economics. Central to their use is the availability of moderate amounts of data to empower our inferences. However, such data can be expensive to obtain, which complicates matters.
E-values are recent enticing alternative to p-values. For- mally, an e-value for a null hypothesis Ho is a nonnega- tive real random variable E such that, if Ho holds, then E[ < 1; by Markov's inequality, is then unlikely that the e-value E is high under the null, and thus high e- value (* 1) provides evidence against the null hypoth- esis. Though simple, this is very powerful notion: values allow for powerful procedures under very lax assump- tions (e.g., not even i.i.d., nonparametric and nonasymp-
common strategy is to simply collect a smaller amount of data, in order to minimize costs. Unfortunately, this gener- ally leads to more uncertain inferences. Alternatively, there are methods that leverage auxiliary cheap-to-obtain data
'Equal contribution 'School of Applied Mathematics, Getulio Vargas Foundation, Rio de Janeiro, Brazil. Correspondence to: Danicl Csillag adaniel.csillage fgybrr..
A Z-estimation problem is one in which we seek to infer parameter 0 E O such that Ez(y((Z 0") )] 0. for some known function tb.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Prediction-Powered E- V-Values
totic) (Howard et al., 2018), naturally handle sequential and anytime-valit inference (Ramdas et al., 2022), naturally fit into multiple testing and post-selective inference (Wang & Ramdas, 2020: Xu et al., 2022) and allows for significance levels to be chosen a posteriori (Koning, 2023; Grünwald, 2022) 1000pp-- properties that are notoriously challenging to obtain with the more standard p-values, if not outright impossible, especially in conjunction. Furthermore, e-values are rather universal: any e-value can be converted to p-value by sim- ply taking its reciprocal, and any p-value can be converted to an e-value by a process termed calibration (Vovk& & Wang, 2019), albeit at slight loss of power.
volved procedures, first considering confidence inter- vals/sequences and then general algorithms based on e-values. In particular, we show that simply substi- tuting the base e-values by our prediction-powe poweeed e-values yields valid prediction-powere procedures that are statistically powerful, leading to a modular and widely applicable technique.
3.
We showcase our method in four case studies ranging from simple mean estimation and hypothesis testing to saaange-poin detection and causal discovery. This highlights the wide applicability of our approach, and we consistently note its much improved performance compared to baselines in spite of substantial (often 100x-200x) reductions in data acquisition costs.
By working atop e-values, our procedure gains a great amount of versatility. We show that any inference proce- dure that operates in terms of e-values has a prediction- powered counterpart, given by our method. Moreover, our procedure naturally inherits all of the usual virtues of e-values, in particular including syytime-validity and post- hoc validity. In fact, the sequential nature of our proce- dure further empowers prediction powwered inference meth- ods, allowing us to arbitrarily improve our predictive model and data collection policy over the course of the inference, whereas previous methods require us to fix it priori, or learn it from a separate data split.
Related work Our method, much like most of the ceediction-pweered inference literature, is fundamentally connected to the literature on semiparametric inference. In particular, our prediction-powered e-values can be seen as a use of the AIPW estimator (Robins al., 1994), but atop the e-values rather than the data. Specifically connecting to e- e-values, prior work by (Xu et al., 2024) has explored using an AIPW estimator atop e-values for the mean in order to construct risk-controlling prediction sets (RCPSs); their wariance-redueee method turns out to be a special case of our approach.
We illustrate our procedure in four case studies. First, we use it for a simple problem of estimating prevalence of diabetes on a population from readily available survey data. Secondly, we apply our method for a problem of anytime- valid testing of the hypothesis that a deployed model's s risk does not exceed certain safety level, for the purpose of continuous risk monitoring. We then turn to more involved inference tasks. On the same context of continuous risk monitoring, we apply our method for detection of change- points, in which we seek to identify points in time where some aspect of the time series has changed. Finally, we consider how our method enables powerful procedures for causal discovery under missing (costly) data.
2. A General Method
We will first present how we can transform standard e- value into a prediction-poweret one in the context of hy- pothesis testing. This mechanism can then be leveraged to transform more complex procedures powered by e-values into crediction-powerts ones; we first thoroughly instantiate this for confidence sequences, and then more generally in the context of general e-value-powered algorithms. Throughout, we consider an active data collection setting.
2.1. Hypothesis testing
Our contributions
Our goal is to test some null hypothesis Ho, and for this purpose have a stream of data (X,, Y.) The X. corre- spond to 'cheap" data that we will always have access to, while the Y. correspond to data that is expensive to obtain, and as such we have little access to - but, ultimately, the hypothesis we want test over the distribution of the Y..s
1. We present a new method for prediction Prcdiction-powered in- ference based on e-values. Besides being applicable to a much more general setting than the ones previously considered in the literature, it inherits all the usual ben- efits of e-values, including sequential inference that is valid under arbitrary optional stopping and post-hoe va- lidity. Moreover, it allows for the underlying predictive model to be updated over the course of the inference, yielding much better data efficiency compared to prior work (which require the model to be fit on a separate data split);
Data acquisition costs aside, sound approach to perform such hypothesis test is to leverage an e-value E,, - i.e., a nonnegative random variable that is function of the first n data points, such that under the null Ho it holds that E[E,] < 1. In particular, we consider e-values of the form
2. We show how the base method can be extended from simple hypothesis testing with e-values to more in-
E, : II e (X))
(1)
2

---

Prediction-Powereed E- V-Vuuues
where (et) is predictable sequence of the 'components' of the e-value, i.e., each 6 be arbitrarily dependent
=Elees(a((()))
&i + R. [cs(V))) =x(()(X,))] TT(X,)
on the samples before time i (but nothing else). We will further require that the e-value's components be predictably bounded: for all i, ez(-) E [az, by] for some predictable sequences (a:).== and (6,) and with a; > 0 for all i.
16 P: = 1]
+ E. [c:03) cx(()
&
16, P: = 0]
Tx()) = E.(es((u (e) e:(u((x))) = E.ee: (Y:)
Most e-values in the literature are already of this form (e.g.. (Waudby-Sminn & Ramdas, 2020; Podkopaev & Ramdas, 2023a;b; Waudby-Smith al., 2022; Bar et al., 2024)), or can factored into it. The boundedness assumption can be enforced by simple rescaling and clipping, albeit at slight loss of power.
Furthermore, the boundedness of the e-values' components and on the π ensure that the quantity is always nonnegative. Using these facts along with backward induction argument, one can prove:
Theorem 2.1. EPPi is a valid e-value for the null Ho- Additionally:
Should we have access to perfect models 14: : X + IR, i.e., such that μ (X2) Y, almost surely, then we could instead only use the predictions atop the cheaper data, μ (Xt), to construct the e-value by its components:
(i) If (Eu, E1....) form test supermartingale i.e., nonnegative supermartingale with E[Eo] I 1 under the null Ho - then so is (EPri EpPi ,E!ri...)):
pimputed
= II (X.)) 1
(i) More generally, if (Eo, Ei....) form an e-process i.e., a nonnegative stochastic process such that for all stopping times т, the null Hg implies that [E[F,+] S then so is (Eg"i EEPr,...) all finite stopping times.
However, in the much more realistic scenario that the model is not perfect, gimputed will not be a valid value.
We can, however, debias Ejmputed as per prediction- powered inference (Angelopoulos al., 2023a) and ac- tive statistical inference (Zrnic & Candes, 2024). First, endow the data stream with additional random variables & ~ Bern(5, (X,)) denoting whether we should collect (and thus have access) to the more expensive data Y,, where Ti, T2....:
Besides having valid e-values which assures of type- error control one should check whether they effi- cient/powerful. We can check that, under mild assumptions, our e-process has good power in terms of the expected growth rate (Kelly, 1956) as long as the models μ4 match the true data Y; sufficiently well:
: X + [1- ba/b, 1] is a predictable (i.e., possi- bly arbitrarily dependent on data prior to i, but independent of all from onwards) sequence of functions that produce the probability of data collection.
Theorem 2.2. Suppose that the e;(-) are each L--Lipschitz and that n. (X,) 2 1 - -a//b; + c for some &, > 0, for all Then there exists some constant e > independent of such that
With this augmented data stream (Xi, Yi, Ti, & we can form a new crndiction-poweree sequence of e-values, with form similar to that of the active prediction-powene estima- tors (Zmic& & Candes, 2024):
n
E
log Eppi n
2 log E,
c n
Σ
E ([]lp((X) Y.Il-
n
& ex(p((X))) + [e:(Y,) - ex())(X.))] Tx,),)'
Ppi o
More general and precise statements are also possible, but less compact; see Theorems A.6 in the appendix.
EPPi_
II ppi
(Es ~ (ern(T;(X.))).)
The sequential nature of the prediction-powered e-values - which holds regardless of whether the original e-values were of sequential nature - allows for an extremely versa- tile procedure. For instance, in contrast to most existing prediction-pon powered inference procedures, we are able to update both our underlying prediction model and our data collection rule over the course of our inference process, with restrictions other than not using future information and having to satisfy the boundedness assumptions.
This construction is motivated by the fact that, conditional on all data prior to the time point i, the prediction-powered e-value components eppi match the prrdicciion-oweree ones e, in expectation:
&
E, (((l)(()))) + [e,(Y. - (x.))
(X,)
The resulting algorithm for hypothesis testing is remarkably simple to implement, given its generality. The pseudocode can be found in Algorithm 1.
=E(e((((())) [x)()) eX,))( &
T (X,)
3

---

Powerrd E-Values
Algorithm 1 Prediction-Powent E- Values
exists some e for which
Input: base e-value components (.).)), (().)))))).... c2())...)
Output: prediction-powered e-values (E EP ....) Ep
log En (0)
dr(o)
E
[
- log n
1 E
drl))
71
Initialize μu: X' + y and π. XX + [1 a1 [n/b111] for each i = 1.2...... do
(())c
n
YillJ-
n
Get 'cheap' data X;
Sample & ~ (err((((X))) if f then
These results may be mapped to the actual measure of the confidence interval, but this is nontrivial; see the Appendix.
Collect "expensive" data Y, E
- ea(3) 11-1(xxx- x (x.)))
+(x,)
2.3. General e-value- algorithms
else
EP E ex((x.))
Beyond simple hypothesis testing and confidence sequences, e-values also be used components of more elaborate inference procedures, for example in causal discovery (e.g. (Peters et al., 2017)), change point detection (Shin et al., 2022; Shekhar & Ramdas, 2023a;b) and test-time adapta- tion (Bar et al., 2024). Our prediction-powered e-values can also be seamlessly integrated into such procedures.
end if
Optionally update ι and μ end for
2.2. From hypothesis testing to confidence intervals
With prediction-powest e-values in hand, we can easily produce prediction-poweer confidence intervals/sequences by considering a family of e-values indexed by the parameter in question.
Consider that we have family of e-values E!!) for re- spective nulls H(*), indexed by y E I, and our overall algorithm is of the form A((Fl )xer). A((E))))r)) Moreover, our al algorithm comes endowed with some 'validity" property, and such that this validity depends only on the inputted e- e-values being valid:
Suppose we want produce a confidence interval/sequence for a parameter 0* E θ of the data generating process, and consider the family of nulls H1 lr = 6. indexed by 0. For each such null, we can construct a corresponding prediction -oowered e-value EpDi-(") (#) and then consider the
Assumption 2.5. If E(?)) is a valid e-value for the null Hr(1) for every y E D, then A((E))) )eer)) is valid.
set
Capi-(a) = (0 E e: (0) < 1/a]. n
It is then easy to show that by simply replacing the input e- values by their prediction-powee -powered counterparts, the validity property is maintained:
By the standard duality between hypothesis tests and confi- dence sets, it then holds that:
Proposition 2.6. Under Assumption 2.5, it holds that A((ERT (1) (e)))) is also valid. If the undertying e-values are e-processes, then it further holds that A((EPri- (1) >err) is valid for any finite stopping time T.
Proposition 2.3. (o) is valid confidence interval
i.e. Pio E C. (ap 2 1 - a. Moreover:
(i) If the underlying e-values form a nonnegative super- martingale, then the prediction-powere intervals are anytime-valid (also known as confidence sequences): Plyn E N, 0* E C.P.i (ppp l-
It is still also of interest to consider some notion of 'power' or 'efficiency' of the resulting prediction-powereed proce- dure. However, such an analysis needs to consider more of the particular algorithm in principle, and so should be done on case-by-case basis. Similarly, the appropriate notion of syytime-validity (which would be implied by the underlying e-values forming test supermartingales) depends on the particular definition of validity for the algorithm in question and so should be considered in a case-by-case basis. Nevertheloss, the case of an e-process still holds generally.
(ii) More generally, if the underlying e-values form e- processes, then the prediction-powerel intervals are valid at arbitrary stopping times: Pio* E (ppi-(a))
1 - or for any stopping time r.
Again, we are also interested in how efficient these confi- dence sequences are. Just like hefore, as long as our predic- tive models are good, we get more concentrated intervals, as measured by the area under the log-p llnddsaaseee
2.4. The Asymptotic Case
Though typically considered in the context of nonasymptotic statistics, e-values also have asymptotic analogues (Waudby- Smith et al., 2021; Ramdas & Wang, 2024). We focus
Proposition 2.4. Under the assumptions of Theorem 2.2, let be a measure over the parameter space O Then there
4

---

Prowerd E- Valuus
on the main text only on nonasymptotic e-values, but our ideas directly map the asymptotic setting just as well; see Appendix B.1.
10*
Active Predictionpowered (ours)
10a
Prediction- powwerd (ours)
Only labelled samples
Imputed
Vanilla prediction-poweied
3. Experiments and Case Studies
10a
[Angelopaulos al., 2023a]
Vanilla active prediction powered [ZmickCandes. 2024]
10-1
In this section we present four case studies where we use our method, highlighting the modifications made to the base methods in the process of predietion-empoowement
10:
0.10
0.12
0.14
0.16
0.18
0.20
3.1. Estimation of a Mean: Prevalence of Diabetes from Survey Data
Figure I. Prediction-povered confidence sequences. The plot shows p-landscape (i.e.. parameter on the x-axis, reciprocal the e-value on the y-axis) for the confidence sequence generated by our method (green), along with those for inference using only labelled samples (purple) and by using imputation approach. The 95% confidence intervals for each p-landscape (i.c., region where the p-landscape is above 0.05) is shaded. Our method provides the tightest valid nonasymptotic intervals, comparable to the active asymptotic method of (Zrnic & Candes, 2024); using only the labelled samples or vanilla PPT (Angelopoulas et al., 2023a) yields weaker inferences, and using imputation fails cover the true mean.
In this first case study we seek to estimate the prevalence of diabetes cohort, upon which we work atop the dataset of (CDC, 2015). This estimation is key to the scaling of resources in health systems, as this medical condition can be very common and very costly to treat in many populations.
Actually assessing the presence of diabetes can be somewhat costly, requiring thorough analysis of individual medical records. On the other hand, we have readily available data in the form of short survey responses, consisting of simple questions such as "do you have high blood pressure?", "do you have high cholesterol?", "have you smoked at least 100 cigarettes in your entire life?", and so on (see Appendix C for the full list). Considering that these questions capture health indicators that are fairly predictive of diabetes, it is appealing to leverage them in prediction-powertt manner.
These e-values already in our required form of Equa- tion (1), but additional care needs to be taken with regards to the bounds of the e-values' components. As-is, the compo- nents are bounded just in [0, max /(1 ()1--),.. 0), (1-0)/0}). This means that we would require the data collection prob- abilities n. (X.) to be bounded in [1, 1] i.e., we would always need to collect data; this is clearly insufficient for our purposes.
More formally, we have data stream (X,, 2 where the X; correspond to the responses to our survey questions, and the Y, correspond to a binary indicator of whether the individual is diabetic. For the sake of evaluation, our dataset includes all Y,, but in a real-world setting would be ex- pected that they would be largely missing; we will simulate this missingness. Our goal is to infer the mean
Fortunately, we have direct way of controlling these bounds by the means of the bets (At). If, instead of re- quiring them to be bounded in (- T (- 4), we require them to be bounded in (-1 .) for some 0- < c S 1, then we have that the components are bounded in [1 - c, + cmax(0/( - 0), (1 - /)/0]), which now leads to nontriv- ial bounds on the n. (X.). In particular, for any desired lower bound Tin for T. (X,), we can now solve for some for which
prevalence of diabetes = E| = diabetic]]-
This is the mean of a random variable bounded in [0, 1], and so we can use the e-value- based method for inference of bounded means of (Waudby-Smith & Ramdas, 2020). Our confidence interval/sequence thus given by the set
1-c 1 + (0/(1 0), (1 0)/0] S Tif (2)
c(aa= {8 E [0, 1] EE!" < <//a..
satisfying our requirements; we use Tinf 1%.
for
We then have the following methods for doing inference with a fixed labelling budget Winf!
E n
= II i=1
TV
1 + de = diabetic|
ddibeeiabeeti]] -
Only labelled samples: collect [Taf n. labelled samples, and use the standard, pwppiiiiioopooeeeeeeee e-values of (Waudby-Smitt, & Ramdas, 2020) to estimate the mean. For the bets A, we use the aGRAPA method proposed by (Waudby-Smint & Ramdas, 2020), bounded to (-120:b); b);
where (1 is a predictable sequence of bets bounded in (-1-g. 1)) In particular, each E!(") is test supermartingale and thus a sequence of e-values for corresponding null ( prevalence of diabetes = 0 (Waudby-Smine Smith & Ramdas, 2020).
5

---

Prediction-Powereed E- B-Values
null hypothesis is true (e-values should be small)
10'
Only labelled samples
Prediction-powertt (ours): use our prediction- powered e-values method atop the e-value with bets truncated as per Equation (2). The predictive model is updated over the course of the inference,
10a
Predition-powered (ours)
Active prediction powered fours)
10
Imputed
10a
whenever we get new data label. For the collection probabilities Ti we always yield Tinf, the lowest possible value, an effort to minimize data collection costs. The predictive model is updated over the course of the procedure by retraining on the augmented dataset
0.0
0.2
0.4
0.6
0.8
1.0
null hypothesis is false (e-values should be large)
1014
16:0
Active prediction-poweree (ours): same as the previous predditioon-poweeed method, but with different choice of collection probabilities т₃. This time, rather than opting for constant, always as low as possible, probabilitics, we follow an approximately optimal choice which takes into consideration the X,, as delineated in Appendix B.2. This gives an 'active inference' /' acactive learning' flavor to method.
108 0.0
0.2
0.4
0.6
0.8
1.0
Figure 2. Prediction-powered anytime-valid hypothesis testing. The plot shows the c-values over time for testing two null hy- potheses one on the bottom, which should be rejected, and one top, which should not be rejected. Our prediction-poweet e-values provide the strongest valid signal for rejection (E 2 20 for significance level of 95%, marked by the dashed lines), as the imputation approach rejects before the null is actually violated; non-rejection (E 20), all the methods appear valid, but ours still attains the highest e-value.
simply learn predictive model to predict the missing Y; from the available X,, and impute the missing Y; with it without any care to use some prediction powweeed inference method. This will often yield invalid inferences, but is very common in practice and thus a relevant baseline.
Imputation:
provides tighter intervals in spite of its nonasymptotic na- ture, likely due to its ability to train the predictive model without a data split; indeed, when compared to the basedline of (Zmic & Candes, 2024), which also updates the models, we a similar performance obtained by our method, but now with the stronger guarantees of e-values.
Vanilla prediction-powerede for the sake of compar- ison to prior work, also consider the method of (Angelopoulos al., 2023a). This method requires the prediction model to be fixed priori, so we first split the collected labels in a training set to train it, and use the remaining labels for their prediction prediction-powered inference method. For confidence intervals, we use CLT-based ones as proposed by the authors.
3.2. Testing the Online Risk: Online Monitoring of Deployed Model for Forest Cover Prediction
For our second experiment, we consider the task of moni- toring the risk of predictive model for forest cover types online. Forest cover prediction is of wide use in remote sensing tasks and particularly for tracking of deforestation and land use, which is, in turn, very useful for climate research. Moreover, online risk monitoring is ubiquitous and applicable any setting where predictive model is involved.
Vanilla active ceediction-poweredd we also compare to the method of (Zrnic & Candes, 2024), which does crddiction-powered inference in an active setting, with a similar AIPW approach. We use CLT-based CIs for their estimator, as they propose, and update the underlying predictive model as we do for our method.
Again we have data stream (X,, YJ) where X; indi- cate input variables to our predictive model - in this case, corresponding to data from satellite images - and Y, are the labels denoting the corresponding cover type (which is a categorical variable). Naturally, Y; is generally miss- ing after all, if it weren't, then we would have need to predict it. In our experiment, we work on the dataset of (Blackard, 1998). For the sake of evaluation, we have ac- cess to all Y, but will simulate the missingness. The notion of risk which we are interested is given by the 0-1 loss: Rfsk;(f) = E [Xx,) * Y.]]. We have already trained
Figure shows the result of our experiment. Our prediction- powered methods provides valid confidence intervals that are tighter and more concentrated around the true mean in comparison to only using labelled samples, while the impu- tation approach is strongly concentrated away from the true mean, and would lead to invalid conclusions. In comparison to the method of (Angelopoulos et al., 2023a), our method
6

---

Pordiction-Poweette E- Values
0.7
0.6
0.5
0.4
Exponentialyy-weighee moving average of loss Labelled data
0.3
x
(No mange--pontt detected with only labelled data) Prediction-powered powered change-point detected (ours)
0.2
Prediction-
Active prediction-power change-poin detected (ours)
0.1
1000
2000
3000
4000
5000
Figure Prediction-powere change-poine detection via e-values. The plot shows exponential moving average a time series (in bluc), with the few collected lahels denoted by the scattered Xs. Our prcdiction-poweets methods detect the change-point accurately, while the base method that only considers the labelled data points does not detect any change-point
the predictive model f independent of our data stream (in the case of our experiment, a separate training split) and have similarly evaluated it on separate validation set, also independent our data stream, upon which we obtained validation 0-1 loss of ValRisk. For continuous risk monitor- ing, we want to test the null hypothesis that
The methods we consider for our experiment are akin to those of Section 3.1:
Only labelled samples at every data point i, we sample & Bern(Tinf). If & = 1, then we collect that data point and update the non con-preddition-poweerr e-value Equation (3). Since the data collection is sampled independently of all else, this is a valid e-value, and forms only about s Tnf samples will be collected. However, only data points where & = 1 are used for inference.
Ho: (f) < ValRisk Cool: for all 1,2,
for some tolerance level Gol for example equal to 0.05. In particular, we would like for this hypothesis test to anytime-valide so that any point we can reach safe conclu- sions from it.
supermartingale; moreover,
Inspired by the work of (Podkopaev & Ramdas, 2021), we consider the following e-value:
Prediction-poweree (ours): prediction-powered e-value atop the base-evalue in Equation (3) tweaked to satisfy the boundedness conditions as per Equation (4). We then have two predictive models: one which is the predictive model whose risk we want to monitor l and another which is used for prediction-poweedd inference, which receives X; and predicts the 0-1 loss for that point, 1 [X,, * Y.]. The first model is held static over the course of the inference, while the one for prediction-powent
compute the
E, = II + Ae (1(((.) # Yi] (ValRisk + (1a1))
(3) where (A:) is predictable sequence of bets bounded in 1.Vll(sl + ftal)). This forms test supermartingale for the null Ho.
Much like in the example inference the prevalence of diabetes in Section 1, the e-values are already of the desired form, but additional care must be taken with regard the limits of the components. As-is, they are bounded in [0, + max(1/ (aaxx(1/(ValRiai (ValRisk + <to) 1, 0}], meaning that our collection probabilities would have to be within [1, 1]. Simi- larly to what we did in Section 3.1, we tweak the bounds for the bets X to make them bounded within [0, c/ (ValRisk + (ea)) for some 0 < 1, leading to components bounded in [1 1 + emax (ValRiskk + Etot) - 1, 0)}. We can then solve for the that satisfies
inference is updated whenever we collect a new label. Collection probabilities ग. (X,) are held constant at Tine, leading label collection matching the baseline of only using labelled samples.
Active prediction-powered (ours): the same as our 'non-active' prediction-poweret method, but label collection probabilities are given by the ap- proximately optimal choice presented in Appendix B.2.
+ cmaa + Ecol) 1,03 < Tinfs
(4)
for a desired labelling budget Tint, which we take to equal 0.5%.
Imputation: as a final baseline, we consider simply imputing the 0-1 loss at points where we have not

---

Prediction-Poweree E- E-Values
collected the true label, with no regard to prediction- powreed inference. This is invalid in general, but com- monly used in practice.
Only labelled data
True DAG
Note that standard prediction-powered inference methods (e.g., from (Angelopoulos et al., 2023a)) are no longer di- rectly applicable due to the requirement of anytime-validity, as well as the fact that our hypothesis test does not come from two-sided test for mean (which would then be instance of simple Z-estimation).
Best possible inference
Prediction-powered (ours)
To fully assess the hypothesis test, we consider two settings here. In the first setting, there is no change distribution: the data stream for the inference follows the exact same distribution as training and validation, and thus the null hypothesis should hold. For the second setting, we increas- ingly poison the labels over the course of time to simulate distribution drift.
The results can be seen in Figure 2. Without data poisoning none of the methods reject the hypothesis, which is appro- priate; though it is interesting to note that our prediction- powered methods were the ones with the highest e-values, managing stay at around 1. Under data poisoning, both of our rndiction-poweree methods detect the distribution drift much quicker than the method that only uses labelled samples, despite both having access to the same labelled samples. The active prediction-pon poweeed method seems to reject the hypothesis tiny bit earlier and yields larger e- values (i.e., with more evidence towards rejection), at the cost of just a tiny bit more data. The imputation method seemingly detects the shift even earlier, but does so before the null hypothesis is actually falsified; thus, it produces false alarm with extremely high confidence.
Figure 4. Prediction-povered causal discovery with e-values. We compare ceediction-poweree causal discovery method with one that uses only labelled data. The lighter nodes correspond to the costly variables, while the darker nodes correspond to cheaper readily-available ones. The standard base method does not detect any edges in the causal graph (denoted by the dashed edges), while ours detects as many edges as the 'best possible' method, which uses all the data points regardless of data acquisition costs.
does have access to. Our prediction-powered method de- tects the change-point accurately while retaining the strong guarantees of (Shekhar & Ramdas, 2023a), whereas the non non-prediction- prediction-powered baseline that uses only available labelled data fails to detect any change-point.
3.3. Change-Poine Point Detection: Detecting Changes in the Quality of a Deployed Model
3.4. Causal Discovery: Constraint-Based Structure Learning with Costly Covariates
Still in the context of testing the cover prediction model of Section 3.2, we now consider not just detecting when the risk goes below a certain level, but detecting any change. E-values have seen good use in the change-point detection literature (Shin et al., 2022; Shekhar & Ramdas, 2023a;b); we opt here for the method proposed in (Shekhar & Ramdas, 2023a), where change-poing -pintt detection is reduced to simple algorithm atop confidence sequences initialized at each time step. For the underlying confidence sequences, we use the same ones as Section 1. Compared to Section 3.2, the only change we make to the data is the introduction of crisp Shange-pointt for better visualization.
Causal inference is of essence to any area where one plans interventions, but the usual methods require knowledge of a DAG describing (a simplified version of) the data generating process. Causal discovery (a.k.a. causal structure learning) methods seek learn this from data. Some particularly common methods for causal learning include the PC (Burr, 2003) and FCI (Spirtes et al., 1995) algorithms; all of these belong to the class of so called 'constraint-baser structure learning, where the DAG is inferred by the means of many hypothesis tests for conditional independencies. In spite of potential multiple comparison concerns, these algorithms are generally said to be valid as long the underlying hypothesis tests are valid (i.e., control type-I error).
Figure 3 displays a high-frequency exponentially moving average of our data (to give a notion of the underlying data stream) that uses data at all points, regardless of whether they are accessible to the analyst; scattered throughout are the few data points that were labelled and that the analyst
In this section we consider the problem of causal discov- ery with the PC algorithm (Burr, 2003) with some costly covariates, which will be generally missing. As is usual in the causal discovery literature, we evaluate on synthetic
8

---

Prediction-Poweree E- E-Values
data generated with a randomly generated DAG, in order to have access the true DAG. Our DAG features 6 variables, of which 3 are considered costly. Overall, our cheap data x; consists of the 3 always-available variables, whereas the costly data 1ú consists of the 6 full variables. For constraint- based causal learning, we need to be able to test hypotheses of the form
A Acknowledgements
This research is funded by Canada's International Develop- ment Research Centre (IDRC) (Grant No. 109981) and UK International Development. CJS acknowledges financial support from CNPq and FAPERJ.
HO
(AA.B.C) A 11LL B B
Impact Statement
where A, B and C' consist of subsets of our 6 variables, possibly empty.
This paper presents work whose goal is to advance the field of Machine Learning and Statistics. There are many poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.
There do exist sequential e-value tests for conditional inde- pendence (Shaer al., 2022; Grünwald al., 2022), but they work under the Model-X framework, which requires knowledge of conditionals that are typically inaccessible in the context of causal discovery. We thus opt instead for Fisher's z- t-transformation of partial correlation test, which is commonly used in causal discovery implementations (e.g., (Markus Kalisch et al., 2012: Zheng et al., 2024)). But it is based on p-values, is not of sequential nature, is asymptotic, and works atop rather heavy normality assumptions.
References
Angelopoulos, A. N., Bates, S., Fannjiang. C., Jordan, M. I,, and Zmic, T. Prediction-powere inference. Science, 382:669 674, 2023a. URL https:
www.atiic.ccoooooooooooooo... 256105365.
We first need to adapt it to our required form, follow- ing Equation (I), To do so, we first rearrange our data stream (X,, Y)) to arrive in batches of B samples, (Xbatch, , ybacch these batches will be the unit of data for our prediction-powered procedure. We can then com- pute the test's p-value for each batch, and calibrate this p-value into an value by the means of the following PToE calibrator (Vovk& & Wang, 2019):
Angelopoulos, A. N., Duchi, J. C., and Zmnc, T. Ppi++: Efficient prediction-powent inference. ArXiv, abs/2311.01453, 2023b. URL https:
/ //api. www.enticccooaa..rrggooooo 264935590
Bar, Y., Shaer, S., and Romano, Y. Protected test- time adaptation via online entropy matching: A betting approach. ArXiv, abs/2408.07511, 2024. URL
PToE(p) = 1 p + plogp
org/
p log
CorpusID:221865588
To ensure that our e-values' components are appropriately bounded, we first clip the p-values (prior to calibration) to lie within (10-7, 1] (so that they are bounded at all; this clipping preserves the validity of the p-values), and then rescale the calibrated e-values by the means of rescaling function
Blackard, J. Covertype. UCI Machine Learning Repository, 1998. DOI: http://do..or//10.24433/////0/
Boyeau, P., Angelopoulos, A. N., Yosef, N., Malik, J., and Jordan, M. Autoeval done right: Using syn- thetic data for model evaluation. ArXiv, abs/2403.07008, 2024. URL ht / //ppi. semanticscholar. org/ / Corp/orrrsss 2683633449
rescale,(e) == 7 1)
with 7) chosen so to satisfy labelling budget of Tua 10% (as in the previous sections). Because the p-values are only valid asymptotically, the batch size B cannot be too small; we use B- = 100.
Burr, T. L. Causation, prediction, and search. Techno- metrics, 45:272 273, 2003. URL https://api:
semanticscholas wtt.aaaooor/ooooooo.eo0eeee.e. 10:.1127000.5.666..
The results can be seen in Figure 4. When using only la- belled data according to our data collection budget, the causal discovery method identifies no edges at all. By using our prediction-poweret e-values, edges, matching the best possible scenario (i.e., what would happen if we had access to the whole dataset). In terms of the average structural Hamming distance over various sam- pled graphs, using only labelled data we obtain an average 12.5; our method halves this to 6.7, and the best possible scenario would obtain that of 6.4.
CDC. Cdc - 2014 briss survey data and documenta- tion, 2015. URL htttt: //t::///wwwww www. edc. gov govvbrfss/ annual data/ data/annual annual 2014.html. Last ac- cessed 30 January 2025.
detect over half of the
Chatzi, I., Straitouri, E., Thejaswi, S., and Ro- driguez, M. G. Prediction ranking of large language models. ArXiv, abs/2402.118266 2024. URL
org/CoorpusT:226041120
9

---

Powwerd E-Values
Grünwall, P. Beyond neyman-pearson: E-values able hypothesis testing with data-driven alpha. Pro- ceedings of the National Academy of Sciences of the United States of America, 121 39x2232298121, 2022. URL https: wtt.ppp.....iiiss.s..ee.. org/CorpusID: 248496494
Podkopaev, A. and Ramdas, A. Tracking the risk of deployed model and detecting harm- ful distribution shifts. ArXiv, abs/2110.06177, 2021. URL https: /tttps::/appt /api. semanticscholas org/CorpusTD:2338 2339643222:226342110
Podkopaex, A. and Ramdas, A. Sequential predictive two- sample and independence testing. ArXiv, 0bbs/205.000143 2023a. URL https://ap. wtt.aap.sssaannccchhlll... org http//orpuusDDD. 258426601..
Grünwald, P., Henzi, A., and Lardy, T. Anytime-valid tests of conditional independence under model-x. Journal of the American Statistical Association, 119hiinnnn111911 119:1554 1565, 2022. URL https: http:/p....aaaacccoo.oooooo org/CoopuSTD::222555
Podkopaev, A. and Ramdas, A. Sequential predictive two- sample and independence testing. ArXiv, abs/2305.00143,, 2023b. URL https: /api. semant iccholarr org/ /ctp//ooaa::D2422550000
Gu, Y. and Xia, D. Local prediction-powered infer- ence. ArXiv, abs/2409.18321 2024. URL https:
cemanticscholar.. wtt.ccca....r/r/orrrpsss..... crg/CoppssI: 272968866.
Ramdas, A. Proof of ville's inequality, 2018.
Ramdas, A. and Wang, R. Hypothesis testing with e-values. 2024. URL https: //api.apppp/pp semant semanticscholes. org prg//CoppsIID:222 2737076511
Howard, S. R., Ramdas, A., McAuliffe, J. D., and Sekhon, J. S. Time-uniform, nonparametric, nonasymp- totic confidence sequences. The Annals of Statistics, 2018. URL
wwwppp...atiiichneeeeeeoooooeeee org//co-pusTD:2197777
Ramdas, A., Grünwald, P. D., Vovk, V., and Shafer, G. Game-theoretic statistics and safe anytime-valid inference. ArXiv, abs/2210.01948,, 2022. URL https:
Kelly, J. L. A new interpretation of informa- tion rate. IRE Trans. Inf. Theory, 2:185-189, 1956. URL http:s///pp..emmntcccchhllle orr//Co:pusD::1114333
/ www.sseemanncccoa......oo orrCpppusII 252715629.
Robins, J. M., Rotnitzky, A., and Zhao. L. P. Es- timation of regression coefficients when re- gressors not always observed. Journal of the American Statistical Association, 89:846-866. 1994. URL https: /api..p.../ arg/CorrputD::22066993
Koning, N. W. Post-hoe hypothesis testing and the post-hoc p-value. 2023. URL https:
//api. www.nteetnnttem www.ccccc......ooorroe.eeeo 91659
Shaer, S., Maman, G., and Romano, Y. Model-x sequential testing for conditional independence via testing by betting, In International Conference Artificial Intelligence and Statistics, 2022. URL https:
Little, R. J. A. and Rubin, D. B. Statistical analysis with missing data, third edition. Wiley Series in Prob- ability and Statistics, 2019. URL https://api. semanticscholes wwwpaar.rrrr//eeeee htt.:aarr...reoooorraa:::001011
/ www.mtiicccollr..orrggooooooo/ 252683086.
Markus Kalisch, Martin Mächler, Diego Colombo, Marloes H. Maathuis, and Peter Bühlmann. Causal inference using graphical models with the R package pcalg. Journal of Statistical Software, 47(11):111122 2012. doi: 10.18637/ jss. v04..i1..
Shekhar, S. and Ramdas, Reducing sequential change de-
tection to sequential estimation. ArXiv, abs/2309.09 2023a. URL https://app. http://:///pppsssanniniccc..eeee semanticscholar org http/CorpusTDD.. :t:pPus3:::2227777777 262043770.
Miao, J., Wu, Y., Sun, Z., Miao, X., Lu, T., Zhao, J., and Lu, Q. Valid inference for machine learning-assisted gede association studies. Nature genetics, 2024. URL
Shekhar, S. and Ramdas, A. Sequential change detection via backward confidence sequences. ArXiv, abs/232..024444 2023b. URL https: //t:: /ppi. semanticscholar. org Corp/orruuaa 2566113355663000
http:///pppssennniscccccoo.eeeee org/CozpusID: 2222:99I:::2299144-4
Shin, J., Ramdas, A., and Rinaldo, A. E-detectors: A non-
Peters. J. Janzing. D., and Schölkopf, B. Elements of causal inference: Foundations and learning algorithms.
parametric framework for sequential change detection. The New England Jourmal of Statistics in Data Science,
2017. URL http://ppp..mmaaicccc.ooooo
/ arr//orrpusID:8533000
2022. URL https::///// /ppi. semant iesnhols..
htg//oorpus1D:28426777
10

---

Prediction-Powereed E- V-Values
Spirtes, P., Meek, C., and Richardson, T. S. Causal infer- ence in the presence of latent variables and selection bias. In Conference on Uncertainty in Artificial Intelligence, 1995. URL https: /app: / www.sstssattttoooiooooiiiiicccte org/CorpusSD:1119777
Zmic, T. and Candès, E. J. Cross-prediction-pwweee inference. Proceedings of the National Academy of Sciences of the United States of America, 121, 2023. URL https: www.eesisaattss.........e wrg/CorpusTD:223311
Ville, J.-L. Étude critique de la notion de collectif. 1939. URL
Zmic, T. and Candes, E. J. Active statistical infer- ence. ArXiv, abs/2403.03208, 2024. URL https: / //api. //api. 268248530.
ott//CorpuSIDD:12427777
Vovk, V. and Wang. R. E-values: Calibration, combi- nation, and applications. Political Methods: Quan- titative Methods eJournal, 2019. URL https:
//api. wwwanticschollatt........ org/CorpusID: 221834569.
Wang, and Ramdas, False discovery rate control with e-values. Journal the Royal Statistical Soci- ery: Series B (Statistical Methodology), 84:822 852, 2020. URL https:/ aapi. semant icccholar....... ott//oorpuSID:2211111155
Waudby-Smith, I. and Ramdas, A. Estimating means of bounded random variables by betting. Jour- nal of the Royal Statistical Society Series B: Statistical Methodology, 2020. URL https: //api. wac.tet..........orrrrr..s...... 240070804.
Waudby-Smith, I., Arbour, T., Sinha, R., Kennedy, E. H., and Ramdas, limit theory and asymptotic confidence sequences. The Annals of Statistics, 2021. URL https: // //api. commanticscholas org/CorpusID; 257901246.
Time-uniform central
Waudby-Smith, I., Wu, L., Ramdas, A., Karampatziakis, N., and Mineiro, P. Anytime-valid off-polic infer- ence for contextual bandits. ArXiv, 0ds/2210.1066. http:///ap..eemnnnccccl........ otr/CorrpusD::25299533
2022. URL
Xu, Z. Wang, R. and Ramdas, A. Post-selection inference for e-value based confidence intervals. Electronic Journal of Statisties, 2022. URL https: //api. www.ttttccccooo.oooooooouoo. 247619 247619119.
Xu, Z., Karampatziakis, N., and Mineiro, P. Ac- tive, anytime-valid risk controlling prediction sets. ArXiv, abs/2406.10490. 2024. URL https: //api.
ttt. www.nticscholeee.....oeeei org/CorpusID: 270559841.
Zheng, Y., Huang, B., Chen, W., Ramsey, J., Gong, M., Cai, R., Shimizu, S., Spirtes, P., and Zhang, K. Causal-leam: Causal discovery in python. Journal of Machine Learning Research, 25(60):1-8, 2024.

---

Prediction-Powereed E- Valuus
A. Proofs
Throughout, we denote by Fi the i-th element of the underlying data filtration.
Theorem A.1 (Theorem 2.1 the main text), E is a valid e-value for the null Ho- Additionally:
(1) If (Eo, ......) form a test supermartingale i.e., nonnegative supermartingale with E[Eol S 1 under the null Ho then so is (FO" Epp :..);
(ii More generally, if (Fo, (t....)orrm e-process - i.e., a nonnegative stochastic process such that for all stopping times T, the null Hu implies that [[E,] I 1-then is (E EPPi ;...) for all finite stopping times.
Proof. First, note that Eppi is always nonnegative for all n E by induction, holds for n = 0 (where Epp = EP 1),
and, for the inductive step,
EPri (Eot1(MaHt (( (Xn+1)) + (en+i((n+)) - 6at1 (X
En+
n+1
Tn+1
(Xn+1)
2 0
En+1 2 0;
- ( (Xn+1)))(Xx+))))))))))))))) [en+1 (Yn+1) - en+1 (Na+1 (Hat)(Xnt))))
(x+1((n+2)
If Ea+1= 0, then the left-hand-side equals en+ (Mn+ (Xn+1)) 2 ant1
Otherwise, it equals
ex+1 (Yn+1))4)))))))))) - -11 - Ønt1 (Xa+1))e (pa+t1 (Xu+1)) 2 ant1 - (1 (x+t((Xnt1))l 2 0 Un+1(Xnnt
1n+1(Xnre
= an+1 - (1 Tnt1 (Xn+1(Xn+)))mpp )oxt1 20 = an- 2 (1 (x+1(Xn2)))bne
1 Rn41/bay S Tn+ Tn+1 (Xn+1),
which holds by construction.
So all that remains is to show that its properties under the null hold. Hence, from here out, assume that the null Hn true.
We will first show that, for any n E N, E < 1. To do we will first prove the following lemma by backward induction:
Lemma A.2. Let n N and denote an event. Then, for any S k S n, it holds that II epll Ax BII (Y.) Fx-r-
E
Proof. The base case is when is n. Then
E
II ppi
A, Fc-1
i=
ennuu (Xn)) + En (en(Yn) (x A,
E leppi
A, Jr Fn-1]
EE
Tn(X,)
En
E fen (M (x,))
+E
T (X,) (ex (Xn))) Snn 1,A, PP 1 AA
En
+
(en
en (un (Xx))) 1 = =6,,AAF,nn-
Plsn 0 A,Jn-1]
Jin (X))
E (X(n
A, E
(X,, (Yn) (nn AA nn (Xn)
=Ejen(u (X,))
- E (Yn)) en (Xx)) A,Jn-1l
-E(en(Y)) A, Fa-1]=
II exYY)) A,Fx-1
12

---

Prediction-Powereed E- V-Values
For the induction step, given that the hypothesis holds for k + n, we want to show that it holds for k. It follows, using the law of total expectation:
E
ppl
A,
F
E
ppi
Ppi
[,
E
Ppi
Ppi
A. Fk
k
i=++1
i=++1
ppi
E
Ppi
1 Fk-1
E
ck
ppi
E
II ex(Y))
AA
1 Fk-1
k+1
Link+1
(xx) + (Xx) (ea(YN) - ex(un(X))))
Ek
E
II ex(Y) A,
1A,Fx-1
Li=++1
E
(H(( E
II (xa)) AAFF
AAF
Li=++1
++
EK
(ex(Yx)) (xx (Xx))) E
II e:(Y)) A,Fx
E 1,
P(Lx A Fa-1
AA (Xx)
Li=k+1
EK
+1
EE
(ex(YR) (Xx())))) E
II ex() A Li=k+1
1 = =0A.FF
PiEx = 00 00., Fx-1
TAA (X+)
ea (Xx))) E
II (Y{)) ArFa
A, ArJx-1
i=k+1
+
(ex(Yr) (Xx)))) E
II cx())) A,FF
AA.F-11 Tx(Xt)
Lim++
E
(Xn)) E II eY(Y)) Fr
|A.
i=++1
n II
+
(ex(V))) (uu (x+)))
AA
Li=k+1
ex (Y.) E II (v))) | A
A
= E
(Y
II
cvv) A,F
A
Li=k+1
i=k+1
E E
(Y.) FA
|A,Fn-1
E
II (Y(Y) A, A,
as we desired.
By picking &o and A to be trivial event in Lemma A.2, we conclude that [[Pp]] B (Y.) Fol = E[E.] < 1, and so Eppi is a valid e-value.
ppi
Fol
Now let us show that, if the underlying e-values form a test supermartingale, then so is the iiddiction-poweree process. By definition Eppi Fg = and all we need do show that TE pppl '++ Fx.] I Eppi, It follows:
13

---

Prediction-Powereed E- V-Vuus
Fn) = =B F.ppi
Fi.] = Blen+t1 Fn] FPpi
En+1
en+ (pn+1 (Xn+1))
Tn+ (Xm+1) (en+1(Xn+1) - ex+ (4n++ F Eppi
E Cnn (aa (Xn++))
Fal
E
Eat1
+ E
(Xn+1) (en+1 (ont1(Xn+1) - ex+1 ( 1n++ = F,
Pl = Fx] E
Tn
Ent1
+ E
(en+1((n+1) - ex+1 (M |n++1 = 0,
Pl =0 Fx] EPPi
Tn- (Xn+1)
E [en (Ma+1 (Xn+1)) )) Fn. EpPi
+ E
Tn- 1 (Xx+1) (en+i(Yn+1) en+1 (n+1(Matt((N En+1 Fn
Tn+1 1x+1(Xx+1) Eppi
E Ien+1 (Un+1 (Xn+1)) Frl EPPi + E (e en en+1(Mn+1 En+1 1,] E.P
E 1 (Y
Fal Eppi
EPPi
= [en+1 Ful En
En
Eppi Fn
< Ey
Eppi '2
Finally, we assume that the underlying e-values form e-process for finite stopping times. We want to show that, for any finite stopping time E[EPPi] < 1. Well,
E E.Ppi = [pppi r]];
When T n for each n E N, by Lemma A.2 with & and A = {T n), holds that E[Eppi T n)
E|[ cppi 1 T== n, Ful = EI (YG r = n, Fol = E[E,, t = n²]
Thus
[[PPPi] EE []] EE [ ETE, 1,
and EPPi is an e-process.
ofauge--f-measure To prove the main theorem about power of our prediction-powent e-values, will use the following change measure lemma based on the Wasserstein distance:
Lemma A.3. For any distributions P and Q over some space 2 and any suncitiiitt 2 TR,
[p[Ø] Eol011 < (PIIQ),
where Q(Pl)Q) the Wasserstein distance between and Q.
Proof. The proof follows immediately from the representation of the Wasserstein distance as an IPM. The Wasserstein distance, written as IPM, is
W(PlQQ) sup =1
Ep[f] [g[f]]
If Φ is L-Lipschitz, then o/ L I-Lipschitz, and so
E - Eo[ø]| = IE / [| - [E = L|B
Egld/ S sup
Er[f]
Eglf]l L LP(QQ).
||
14

---

Prediction-Powereed E-Values
We will also a simple upper bound the Wasserstein distance, showing that it is upper bounded by the MAE. Lemma A.4. For any distributions P and Q over some normed space (2, || 1l),
W(P.Q) Q) S E0 @l1 Zqll1.
Proof. By the dual representation of the Wasserstein distance,
QP
= sup E2p- - -Eza-l = sup [8 P. [.p= - ((to)))))
| 1tap
[[f]|up=
S sup
22 - f(Za)l] sup
2z Zp - -
||
[|
P.
a1| Zp
Zq1l I|]-
Theorem A.5 (Theorem 2.2 in the main text). Suppose that the ea(-) are each Li-isscchitz, and that T. rx;)); 2 1 aa/b; +
for some > 0, for all i. Then there exists some constant > 0 independent of such that
E
log E.ppi
2 E
log Fy,
B Blllpa(X3 -VII]..
Proof. First, note that
E
log Eppi
E
1
log
ppi
1
E
log ppi
n2
1=1
& + (X,) (e:( - ==(m(()))))
log
(x,)x((,)))
1=1
12
E E
log (x((((())))
(x(),) (e(()))
- c((u(())))
72
The inner expectation in the last line is random only over (X.)). Moreover, thanks to our assumptions, the value we are taking the expectation over is Lipschitz as a function of (x())): because of the lower bound on the T; with positive margins e,, the value within the log is bounded away from zero, and so the log becomes Lipschitz with some constant > 0.
|oog (z(.)) +
& (X,)) (e) -(01 -
& s su <u(eat) + (x,), (e,(V)) c0O)
Lip
Lip
If E = then this equals u (1
Li Otherwise, this equals
&
(Yi) (1 - (x.))))
u le(()) + ni (X:)
les (Y.) - el (x)
=
(x()x)
Lip
Lip
(X,)) (lea(V.) - -11- - (x.(((x)))ez
(X,) |1- - TTe (x)(x)))
(1 (1-10 mi((x)) (les())llpp Lt (1 11mm(X;))) (X()))
(X,))
In either case, this Lipschitz constant is upper bounded by c:= u - Ls max (X-m(Xx)) 13 (which does not depend on nt).
15

---

Prediction-Powereed V-Values
Hence, by Lemma A.3,
1
& (x(u((xi))
[ei(X)) e((u(x)))) (x)
]]
E
E
log
7
(x(Xi)
(ei(Y;) -((()))) (x, -c -x3(x(X))1X3) [ e (V) y Er T. (x,), Fi-1 (V
E
E
log
e:(Yi)
Tz((x)
1=1
1
[loge es (Y:)
B (l,
n1
- M> E[(((())))M)))
E
log Fm
Apply Lemma A.4 and conclude.
The following is a more precise statement about the growth rate of our prediction-poweree e-values, albeit less directly interpretable:
Theorem A.6. It holds that
Σ n (Xx)) log (c((u(()))) + ex(x) - -x((x(()))) )
log Eppi
E
Σ 1=1
(1 -
(o
log (x((()))
+ E
n
n
n2
(X,))
Proof.
1
log Eppi
E
log
ppi
E
log ppi
1=1
1=1
&
log
(((u((X)) + [es(Y.) ex(u(()))]
71
E
E
log (((()1()))) + [G(()) - -x,(((x)))]
|
(x,)))
E
E
log
((((((())) + [es(Y)) (xx)(
t: (x(
16=1,
P(s: = 1 1|Fi11
E
log
(e(()(X))) [e(()) (x(((()))]
&i (X;)
16 0,
P(s. 01 [i
log
(x,(((,))) + +[e(())) cx(4((,,)]]
| (X,)
(X,)
7=
+ E flog es (Mi(X))) Fe-1) (1 (x(()
Tx(X,) log (x,()(x,)) (e(()) cx((((,))) x,(x,)) + log
72
n. (X,) log (e((((X))) c;(Y,) - (x;
E
(xx
+E
T (X,))
To prove the next result we will make use of Ville's inequality:
16

---

Prediction-Powereed E- E-Values
Theorem A.7 (Ville's inequality (Ville, 1939; Ramdas, 2018)). For any nonnegalive supermartingale (L+) and any
define the (possibly infinite) stopping time N inft 2 1 L 2 x: and denote the expected overshoot when La surpasses as
LN
R
N
< 00
21. 1.
Then,
E Lo
E
P
L
2 2
oz
Proposition A.8 (Proposition 2.3 in the main text). Cppi-(a) valid confidence interval - i.e, Plo E (p)i 2 1
Moreover:
(i) the underlying e-values form nonnegative supermartingale, then the wrediction-powert intervals anytime-valid (also known as confudence sequences): Plyn N, Ch 2 ox;
(ii) More generally, if the underlying e-values form e-processes, then the Pradictiin-powerre intervals are valid at arbitrary stopping times: P E (ap) 2 1 a for any stopping time т.
Proof. By construction, Pie (a) IPEEPPi-(07 1/a] that the null HJ holds and using Theorem 2.1,
((EpPii((")) > 1/a]. By Markov, considering
P(Eppi-(0') 1/a] 21
E [[EPpi-(0')
1
1
1/a
1/a
If the underlying e-values form supermartingale, then by Theorem 2.1 do the prediction-powered e-values; then, using Ville's inequality,
P|vn E N, 0* E CPPi- (a)] P(vn E N, Oppi-(0") < 1/a] = PP(sup Eppi-(0")
< 1/aa
E Sipppii-0 (0")
1
- P|sup Eppi-(0") (0") > 1/a|
1 a.
1/a
1/a
Finally, if the underlying e-values form an e-process, thenby Theorem 2.1 so do the prediction-powered e-values (for finite stopping times), and so, by Markov,
PO E Oppi- (P(pppii(0") (0") < 1/a]
PlE (0")
1/a]
E. EPPPPi-(07 (00)
1/a
1/a
Proposition A.9 (Proposition 2.4 in the main text). Under the assumptions of Theorem 2.2, let v be a measure over the parameter space O Then there exists some e for which
[
[ log
log (0) du(0) S
dv(0)
Blll,(X,,) Y.ll..
n
(
n
Proof. By Fubini,
[ log 1/ (0
E
E
log 1 /Eppi-(0) du(0)
17

---

Prediction-Powereed E- V-Vaus
And we apply Theorem 2.2:
E
1
log 1/E
(0)
dv((0)
B
1
log Eppl (0)
dv(() s- - 1(1
log 5
[I(((()))X33] dr(o)
n2
n
/
E
log
1/E
Σ (1 du(0)
dv(() + (
E
log 1/ 1/E
(H
log 1/ 1/
(/EE)/((00)))) du(01]
+ (()c
B[W (w, (X.)
n1
1
(())c
E
log 1/ / do(0)
(X)) YiIl,
n
where the last step holds by Lemma A.4.
Considering that the object of interest is confidence interval, it is desirable to further bound the measure of the interval. We were unable to prove any sufficiently general result that was (i) nonvacuous, and (ii) decayed reasonably fast as n increased, and imagine that heavy assumptions are necessary; this may be best done on a case-by-case basis. Nevertheless, here is one possible somewhat straightforward result.
Proposition A.10. Under the same conditions of Proposition 2.4, suppose that the prediction-powered e-values are bounded from above by Mppi (i.e., for all E O, Ex (p) < MPPi almost surely), and similarly for the non-prediction powered e-values by M (i.e., for all 0 E O E2(0) < M almost surely). Then: Then
[PP(C E[ʃ log 1 E/Ppiii(0) (v) + ((O)MPM
Bi log U()
B[u(C)) S
log a + log Mppi
log a + log M
Proof. Consider the measure i(AA) = (0)))(((A/(())) it is probability measure. Then:
((CPP))= = Paws (p) < 1/a] = Pe (toeeLL/PPi--0 > al Paas llogellog 1/EPPi-(0) > log a);
We want to apply Markov. To do that, we need the left-hand side to be nonnegative; to do so, we add log M to both sides, which yields
Pawe (log 1/ (0)) > log ax); = Pa-o olo 1/EPpi-(0) + loog MPPi > log a + log PP
Egga [log 1/ Ep)i- + log MPDi
log a log Mp
J log 1 (0) + log MPpi
log a + log Mppi
[()) log E/E dv(() log MpPi
log + log Mppi
So, multiplying everything by v(O), we get that ((CPP))<(()) = v(CPPi) 1 v(0). (v()))-''a 1/E + log MPpi
log 1/ (") + v(O) log Mppi log a + log Mppi
log log Ppi
Finally, taking the expectation both sides, get that
f log 1/ (Q) + v(O) log MPpl log log M/pn
E log 1 1 (0) + v(O) log MPPi log log Mpni
[[P(Ce))]] < E
as we desired.
The same can be done for the pna-prrdiciionpowweeed e-values, replacing Eppl with E. and Mppi with M.
18

---

Prediction-Poweree E- B-Values
Most terms in the inequality depend on n, it's a bit hard intuit. But, if the dependence the in the expectation of the log is good enough, then this should be nonvacuous, least.
Proposition A.11 (Proposition 2.6 in the main text). Under Assumption 2.5, it holds that A((ER the underlying e-values are e-processes, then it further holds that A((EPii(()) (er) is valid for any finite stopping time T.
Mer)) is also valid. If
Proof. To prove that A((ERPi(())))) "err) is valid, by Assumption 2.5, it suffices to show that EPPi-(1) is valid for every E P; and by Theorem 2.1, this is indeed the case.
(1)
Now suppose that the underlying e-values (El"))er form e-processes; then so do the peediction-poweree e-values [E [Eppi-(M) (x) ,er for finite stopping times, by Theorem 2.1. Then, to prove that A((EPri-(n) (er)) is valid for any finite her
stopping time T, again by Assumption 2.5 it suffices to show that E2pi--() is valid, which is indeed the case since they form e-processes for finite stopping times.
B. Additional Results
B.1. The Asymptotic Setting
E-values, though usually defined in non-asymptotic terms, have asymptotic analogues. In particular, (sequential) asymptotic e-value is defined as (sequence of) nonnegative random variable(s) E, such that, under the null Ho. it holds that lim sup, E[E,] S (Ramdas & Wang. 2024). We briefly show here that the core points of the theory we build in the main text can be directly applied here. Most results whose analogues we do not prove still hold, and are just omitted for conciseness.
Proposition B.1. If En is an asymptotic e-value, then so its prediction-powered analogue EPPi
Proof. We want to prove that Epri is an asymptotic e-value. It follows, by Theorem 2.1:
lim sup [[ lim sup B[EA] 1.
n-+00
Proposition B.2. If (0)) is an asymptotic e-value for each OinO, then (ppi-(a) == 10 E θ E pP)- (0)
< 1/a} is an
asymptotic confidence interval, iee, lim sup, Plo & C (a) S
Proof. It holds that
lim P & Cppi-(al) lim sup Suppp(pppp-iee Epppi > 1/a]: 21//] n-000
n=+00
By Markov,
E[EPPi- (0*)
sup P[Eppi-(0") (00) 1/a] < limsup 0-000
a lim lim E[Eppi-(00) (0') <
1/a
n-+00
n-000
The results related to power (e.g., Theorem 2.2) apply to asymptotic e-values without any modification necessary.
B.2, An approximately optimal choice for m
Our rrediction-oweeeee e-values have, at their core, the customizeable choice of data collection probabilities (X,).. While selecting a constant n. (x,)) Tinf, where Tnf the lowest possible value possible (so as to minimize data collection costs) is reasonable approach, it ignores the versatility that the probability can take into account the 'cheap" data X,, which could significantly improve statistical power when used correctly. In an effort to seek better strategy, we try to identify an 'approximately optimal' choice of n,.
The optimality in the sense that, at point in time, the data collection probability function (-) should be chosen so as to maximize the expected log of the e-value, as per (Kelly, 1956); this also similar, e.g., to the GRAPA and aGRAPA strategies of (Waudby-Smin Ramdas, 2020). However, the also have additional constraints:
19

---

Prediction-Powereed E-Values
Its image must be bounded: n. X' a4/b, 1]. Lee. for all X, az/bs Tx(x) S
iii) It must respect some particular maximal data collection budget: E [TTT((X) (X:)) S Budget.
So we seek to solve the following constrained functional optimization problem:
argmax Ellog E Fi-1] = argmin BB|- log en [ii-1] FEL2
TEL
subject
0n/bi S (I) S for (almost) all X R B (X,) Fi-1] < Budget,
where assume that the domain of is bounded (so that there are functions that satisfy the first domain, since π is always positive).
Our approximate solution to this is as follows: the functional gradient of our (unconstrained) loss given by
(e((Y)) (x)) ex(u((X))) (X.)
log 11
where h(t) 1/t llg log 1/t
1/t + logt. The h function is a bit inconvenient for solving this problem in closed form, so, inspired by (Waudby-Saa -onth & Ramdas, 2020), we do a Taylor approximation around some point a (which turns out later combine with the parameter control the budget constraint); this leads to the following approximate functional gradient:
ex(Y)
Ba 1 Tinf
0ie BinT
Fi.
(x,))(X))
Tint
where o,, log 2/a - and 3a (a 1)/a?
The uncontsrained solution is then given by
x T"Xi) ~ E
(Yi))
|x, -1) (aal Ba 1),
(e(us((,)))
and KKT conditions give that:
If the unconstrained optimum above satisfies the boundedness constraint, then that is the optimal choice;
If Qa + (IE
e.(Y)
Fi, /Ta (1 //nt)/ Ant) 0, then x =*(,)) Tinfi
Otherwise, x (X.) 1.
C. Datasets
C.1. For Section 3.1
We use the dataset of (CDC, 2015). It tabular dataset, where each row corresponds to individual; the targets Y; in the original dataset denote whether the individual was (i) diabetic, (ii) pre-diabetic, or (iii) neither. For the purposes of our experiment, we only look for whether they were diabetic or not. The covariates are effectfully responses to the following simple survey questions:
"do you have high blood pressure?"
"oo you have high cholesterol?"
"how long has it been since the last time you have checked your cholesterol levels?"
what is your body mass index (BMI)?"
20

---

Pecdiction-Poweree E- V-Vuus
have you smoked least 100 cigarettes in your entire life?"
has you ever been told you had stroke?"
haae you been diagnosed with coronary heart disease (CHD) or myocardial infarction (Ml)?"
"how much physical activity have you done in the past 30 days (excluding job)?"
"how often do you consume fruit?"
"how often do you consume vegetables?"
"how often do you consume alcohol?"
do you have health care coverage, including health insurance, prepaid plans such as HMO, etc.?"
"Was there a time in the past 12 months when you needed to see doctor but could not because of cost?"
"Would you say that in general your health is: [excellent very good good / fair / poor]"
"Now thinking about your mental health, which includes stress, depression, and problems with emotions, for how many days during the past 30 days was your mental health not good?"
"Now thinking about your physical health, which includes physical illness and injury, for how many days during the pat 30 days your physical health not good?"
"Do you have serious difficulty walking or climbing stairs?"
"What is your age?"
"What is your highest level of education?"
"What is your level of income?"
C.2. For Sections 3.2 and 3.3
We the dataset of (Blackard, 1998). Upon this dataset, in training split, train simple random forest classification model. We also separate validation split to compute the validation loss in Section 3.2. At evaluation time:
For the tnon-poisoned data stream in Section 3.2, where the null should not be rejected, we just use the data remaining after the training and validation splits.
For the 'poisoned" data stream in Section 3.2, switch the label with probability of
clamp(o.1) ((15)) 2
0.5
for time E [0, 1).
For the data stream in Section 3.3, we switch the label with probability of
((t+)+ 0.35
2
1 2 0.23] clamp
for time E [0, 1]. The indicator causes a visible change in the time series, good for visualization. The remaining bit done differently from the previous section so that the change in the distribution is not too drastic.
C.3. For Section 3.4
We generate a random DAG with nodes using the Erdös-Renyi procedure, and mark the last three of these nodes as 'costly'. Relations between the nodes are given by linear functions, whose weights and biases are sampled randomly, with additional independence gaussian noise with a standard deviation of 0.4.
21

---

Prediction-Powereed E- V-Vuuees
D. Code
The source code to reproduce the experiments the paper, as well as additional experiments (e.g. varying seed, varying sampling budget, underpowered settings [i.e., with worse predictive models]). available at https: htt.ggtth.....ooo. com/ decsi aegillllaaaaa/ exxprrimeeeeeeeeeeeee s -prediction wew.littoooooweedeeeeeeeo pwwered-evaluue
22


Reviews:
[{"review_id": "VmtHH15vV4", "reviewer": "Reviewer_3dJo", "summary": "This paper extends the ideas of prediction-powered inference to e-values. The contribution is focused but nice, since as the authors note, prediction-powered e-values allow for a broader set of possible inference techniques and guarantees (anytime validity in particular). The core methodology involves leveraging the same inverse propensity weighting tricks used in AIPW and Active Statistical inference to the sequence of components of an e-value (with additional constraints on $\\pi(X_i)$ to ensure that $E_n^{ppi}$ is non-negative for all $n \\in \\mathbb{N}$), and then theoretically showing that the ppi e-value is a valid e-value. The remainder of the empirical section gives several compelling experiments that show the versatility and effectiveness of prediction-powered e-values.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive remarks, addressed as follows:\n\n**Sequential updating and [Zrnic & Candès, 2024]:** The reviewer raised a good point. We will add comparisons in Section 3.1 to the method of [Zrnic & Candès, 2024]. We've run a preliminary experiment on this and indeed, their method (the sequential variant) performs much more similarly to ours (their martingale-CLT-based interval is slightly smaller than our nonasymptotic e-value-based one: with fixed label sampling probability, ours is [0.127, 0.155], while theirs is [0.128, 0.153]; for reference, vanilla PPI is [0.129, 0.172]).\n\n**Reductions in acquisition costs:** In all the experiments, we stipulate some particular labelling budget, in the form of $\\pi_{inf}$. For example, in Section 3.1 we have $\\pi_{inf} = 1\\%$ (right under equation (2)) and in Section 3.2 we have $\\pi_{inf} = 0.5\\%$ (right after equation (4)). These quantities denote the percentage of samples which we will label, and it is from these numbers that we obtained the 100x-1000x statistic: $1 / 1\\\\\\% = 100$, and $1 / 0.5\\\\\\% = 200$. As for the 1000x number, it corresponded to a previous version of Section 3.2 that we ended up not including in the paper, in which we had $\\pi_{inf} = 0.1\\%$. The results were very much satisfactory, but we ended up increasing $\\pi_{inf}$ so that it was a bit less arbitrary and more in-line with the other values of $\\pi_{inf}$ in the paper. After this change, it seems we forgot to update the introduction. We will fix this for the camera-ready version. We will also clearly present these numbers in the experiments section, as well as compute the exact amount of labels used in practice (though the actual number was generally very close to the stipulated budget).\n\n**As for the remaining questions:**\n- $m$ on page 5: that is supposed to be $\\theta$, we'll fix that.\n- $n$ in the experiments in Fig. 1: the full dataset has 253680 samples; for the only-labelled baseline, we use 253680*1% = 2537 of that, for our method we follow the sampling (which samples about 1% of labels as well), and for vanilla PPI we collect also 1% of labels, and split them 50%-50% between training the model and the statistical inference, since we both want to get a good model (so want a large training split) but also don't want to suffer much in the statistical inference (so want a large inference split). We will also include experiments varying this split's proportion, though it does not seem to improve vanilla PPI significantly.\n- The x axis in Fig. 2 is time, i.e., the course of the inference procedure. The first samples of the experiment are to the left, the later samples are to the right. We will make this more clear in the camera-ready version.\n- We will also add an explanation of the change-point detection algorithm of Shekhar&Ramdas in the Appendix, thank you for the suggestion.", "claims_and_evidence": "The theoretical claims are all clear and well-presented. I appreciate the focus on crisp statements in the main text, with additional generalizations in the Appendix.\n\nThere are, however, two claims that I believe need more support:\n- Contribution 3 cites \"massive (often 100x-1000x) reductions in data acquisition costs\", but from the empirical results it is not clear where this is supported. Please also see the \"Questions for Authors\" section of this review about the labeling and descriptions of the results figures.\n- It is repeatedly mentioned that a strength of sequential prediction-powered e-values is the ability to update the underlying prediction model and data collection rule over the course of the inference process. However, for at least the experiment in 3.1, this is also possible in Active Statistical Inference (Zrnic & Candes, 2024), which relies only on having martingale increments (and uses the martingale CLT). It seems the fairest comparison would be to use the same setup used in \"Active prediction-powered (ours)\", but with the estimator in Zrnic & Candes. Similarly, it is unclear if the vanilla prediction-powered approach uses lambda-tuning or not?", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Core results (Theorem 2.1 & Proposition 2.3) were checked, and auxiliary results (Theorem 2.2, Proposition 2.3/2.4/2.6) appear to be correct.", "experimental_designs_or_analyses": "The experimental designs in all of the four case studies appear sound for demonstrating the applicability of the proposed method. As noted earlier, a comparison to Active Statistical Inference (Zrnic & Candes, 2024), which allows for updates to the predictive model and adaptive sampling, is missing from the experimental section.", "relation_to_broader_scientific_literature": "The paper builds upon the existing literature on prediction-powered inference, particularly the work of Angelopoulos et al. (2023) and Zrnic & Candes (2024). It expands this area by applying the concept to e-values, which have gained popularity as a flexible alternative to p-values --- with impactful applications to sequential and post-hoc inference. The paper also connects to the literature on e-values in various applications like hypothesis testing, confidence sequences (Waudby-Smith & Ramdas, 2020), change-point detection (Shin et al., 2022; Shekhar & Ramdas, 2023), and causal discovery. The key contribution is in bridging these two active areas of research.", "essential_references_not_discussed": "The related work appears to be discussed appropriately throughout the introduction and main text.", "other_strengths_and_weaknesses": "The main issue with statistical inference using e-values is typically a loss in power. Therefore I am surprised by the improvement over PPI in Fig. 1, though the authors claim this is due to data splitting vs. online model updates. The paper should be comparing on fairer terms to Active Statistical Inference (Zrnic & Candes, 2024) in Fig. 1.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "While the presentation is generally very good, a couple of things were unclear:\n\n1) What is $m$ on page 5 for the definition of $E_n^{(\\theta)}$?\n\n2) What is $n$ in the experiments in Fig. 1? I understand that the number of labelled examples is 1% of that. Similarly, what $n$ is used for vanilla PPI, as that seemed to be different following data splitting.\n\n3) What is the x axis in Fig. 2?\n\n4) Can you provide an explanation of the change point detection algorithm in Shekhar & Ramdas in the Appendix.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yN4GJRKtvp", "reviewer": "Reviewer_L9yP", "summary": "The paper proposes a methodology for converting any e-value based inference procedure into a prediction-powered counterpart. More concretely, they consider the setting where we have a data stream $(X_i, Y_i, \\pi_i, \\xi_i)_{i=1}^\\infty$ where $X_i$ is cheap data, $Y_i$ is expensive labelled data, and $\\xi_i \\sim Bern(\\pi_i(X_i))$ indicates whether we have access to $X_i$ or not. Additionally, we have access to a predictive model $\\mu_i$ at each step.\n\nIn this model, they key contribution is that given any specification of a sequence of e-values on the expensive data, $E_n \\doteq \\prod_{i=1}^n e_i(Y_i)$, they show one can construct a corresponding prediction-powered sequence $E_n^{ppi} \\doteq \\prod_{i=1}^n e_i^{ppi}(Y_i)$ by defining $e_i^{ppi} \\doteq e_i (\\mu_i(X_i)) + [e_i(Y_i) - e_i(\\mu_i(X_i))] \\cdot \\frac{\\xi_i}{\\pi_i(X_i)}$.\n\nThe authors supplement this contribution with a lower bound on the expected growth rate that is in terms of that using the true data minus the expected Wasserstein distance between the predictions of the model and the true values $Y_i$. They extend their results to building anytime-valid prediction-powered confidence sequences, as well as to general e-value based algorithms. \n\nFinally, they supplement their theoretical contributions by a series of experiments showing the empirical gains achievable (faster null rejection, smaller interval widths, etc.).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the reviewer for their comments.\n\n**Regarding modelling missingness through $\\pi_i(X_i)$:** Our construction is focused on an active/experimental setting in which we can actually sample $\\xi_i \\sim Bern(\\pi_i(X_i))$ and collect $Y_i$ when $\\xi_i=1$. Extending it to a \"semi-supervised\" setting is non-trivial. Nevertheless, here is one possible way of doing it (though probably suboptimal): at each step, choose $\\pi_i$ to approximate the probability of whether you have access to the corresponding label (this is akin to e.g. a propensity score). Sample $\\xi_i\\sim Bern(\\pi_i(X_i))$. Then you have a couple of cases:\n\n1. If $\\xi_i = 0$ (i.e., don't want to use the label), we continue as normal.\n2. If $\\xi_i = 1$ (i.e., want to use the label) and we have access to the label, we continue as normal.\n3. If $\\xi_i = 1$ (i.e., want to use the label) but we _don't_ have access to the label, then rather than contributing $e^{ppi}_i$ to the e-value, we contribute $e^{ppi-missing}_i:=\\frac{a_i-(1-\\pi_i(X_i))b_i}{\\pi_i(X_i)}$; this works because, whichever the value of $Y_i$, $e^{ppi-missing}_i \\leq e^{ppi}_i$.\n\nWe believe that a more in-depth exploration of \"semi-supervised\" (i.e., that rely solely on observational data) procedures such as this one is best suited to its own paper (e.g., how do we retain good performance even when case (3.) is somewhat frequent?), and so we leave it to future work.\n\nAlso, thank you for pointing out the typos, we will correct them!", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I skimmed through the proofs in the appendix and they appear correct.", "experimental_designs_or_analyses": "I found the experimental analyses to be thorough and well-done. I appreciated the inclusion of different types of inference tasks, the baseline considered and the presentation.", "relation_to_broader_scientific_literature": "This paper is very fundamentally connected to the broader scientific literatures of sequential hypothesis testing via betting and the more recent line of work in prediction-powered inference. They provide extensive references and even apply their method to more niche/particular developments such as using e-values for change-point detection. In my opinion, this is a situation where combining two lines of work really enriches both respective lines.", "essential_references_not_discussed": "Not to my knowledge.", "other_strengths_and_weaknesses": "Both prediction-powered inference and inference with e-values are relatively recent developments in statistical inference that are particularly appealing and relevant — proposing an elegant way to bring them together is in my opinion a very useful contribution. \n\nBeyond allowing for prediction-powered inference with e-values, this methodology allows for continuous updating of the predictive model used during inference, which I find to be particularly neat and practical (as batch predicition-powered inference required a predictive model fixed a priori).\n\nOne potential weakness is that the modeling of missingness as $Bern(\\pi_i(X_i))$ is a bit weird and a bit hard to map to all practical scenarios where one may want to employ prediction-powered inference. For example, sampling $\\xi \\sim Bern(\\pi_i(X_i))$ may not be possible and/or unclear if we could have access to $\\pi_i(X_i)$ in order to construct our e-value in many applications.", "other_comments_or_suggestions": "List of typos:\n- on line 57, left column, i think it should be “allow” rather than “allows”\n- on line 175, right column, change “inputted” to “imputed”\n- on line 225, left column, it should be “thorough” not “through”\n- on line 357, right column, “extremely quicker” is not grammatically correct", "questions_for_authors": "Could you comment more on modeling missingness through $\\pi_i(X_i)$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vnsYdzW67N", "reviewer": "Reviewer_t47N", "summary": "This paper proposes an extension of \"prediction powered inference\" to e-values, a general class of statistics that are used in many methods outside of the estimation. The authors show theoretical results justifying the validity of their prediction powered e-value, and its validity under active sampling, as well results for its growth rate. They also provide a comprehensive set of experiments showing the applications of prediction powered e-values to change point detection, continuous monitoring, and causal discovery.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the reviewer for their insightful comments.\n\n**Growth rate as a function of the MAE of the predictor:** Yes, we can; in particular, we can show that the Wasserstein distance in Theorem 2.2 is upper bounded by the MAE of the predictor:\n$$\\begin{aligned}\n    W(\\mu\\_i(X\\_i) || Y\\_i)\n    &= \\sup_{\\\\|\\phi\\\\|\\_{Lip} \\leq 1} | \\mathbb{E}[\\phi(\\mu\\_i(X\\_i))] - \\mathbb{E}[\\phi(Y\\_i)] |\n    \\\\\\\\ &= \\sup_{\\\\|\\phi\\\\|\\_{Lip} \\leq 1} | \\mathbb{E}[\\phi(\\mu_i(X\\_i)) - \\phi(Y\\_i)] |\n    \\\\\\\\ &\\leq \\sup_{\\\\|\\phi\\\\|\\_{Lip} \\leq 1} \\mathbb{E}[| \\phi(\\mu\\_i(X\\_i)) - \\phi(Y\\_i) |]\n    \\\\\\\\ &\\leq \\sup_{\\\\|\\phi\\\\|\\_{Lip} \\leq 1} \\mathbb{E}[| \\mu\\_i(X\\_i) - Y\\_i |]\n    \\\\\\\\ &= \\mathbb{E}[| \\mu\\_i(X\\_i) - Y\\_i |].\n\\end{aligned}$$\nThis is an interesting result, and we will add it to the camera-ready version of the paper. Thank you for the suggestion!\n\n**Fallback to non-PPI e-value in case of lack of power:** There probably does not exist a procedure which allows for the practitioner to 'peek' at the final prediction-powered e-value $E^{ppi}_n$ and non-prediction-powered one $E_n$, and choose which to use. However, if at any point over the course of the inference the practitioner believes that the prediction-powered e-values are under-powered (e.g., because so far not using it would have been more efficient), then can start selecting $\\pi_i(X_i) = 1$, which would reduce the prediction-powered e-value to the non-prediction-powered one.\nAlternatively, since we are working with e-values, any convex combination (parametrised by some scalar $\\eta_i \\in [0, 1]$) of the e-value components $\\eta_i e^{ppi}_i + (1 - \\eta_i) e_i(Y_i)$ yield a valid e-value, as long as the $\\eta_i$ are predictable (and thus independent of $e^{ppi}_i$ and $e_i(Y_i)$). So rather than increasing the probability of sampling the label, one could also decrease such a $\\eta_i$.\n\n**Semi-supervised setting:** Indeed, our construction is focused on an active setting, and extending it to a \"semi-supervised\" setting is non-trivial. Nevertheless, here is one possible way of doing it (though probably suboptimal): at each step, choose $\\pi_i$ to approximate the probability of whether you have access to the corresponding label. Sample $\\xi_i\\sim Bern(\\pi_i(X_i))$. Then you have a couple of cases:\n\n1. If $\\xi_i = 0$ (i.e., don't want to use the label), we continue as normal.\n2. If $\\xi_i = 1$ (i.e., want to use the label) and we have access to the label, we continue as normal.\n3. If $\\xi_i = 1$ (i.e., want to use the label) but we _don't_ have access to the label, then rather than contributing $e^{ppi}_i$ to the e-value, we contribute $e^{ppi-missing}_i:=\\frac{a_i-(1-\\pi_i(X_i))b_i}{\\pi_i(X_i)}$; this works because, whichever the value of $Y_i$, $e^{ppi-missing}_i \\leq e^{ppi}_i$.\n\nWe believe that a more in-depth exploration of \"semi-supervised\" procedures such as this one is best suited to its own paper (e.g., how do we retain good performance even when case (3.) is somewhat frequent?), and so we leave it to future work.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes --- specifically for Theorem 2.1 --- seems correct.", "experimental_designs_or_analyses": "Yes, all 4 experiments. No particular issues", "relation_to_broader_scientific_literature": "This paper extends active prediction powered inference methods to work with e-values. There has been similar work (mathematically) in the e-value literature in the past for off policy estimation/doubly robust estimation/active risk control, but never applied in a black-box fashion to e-values directly.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The experiments are quite substantial and illustrate a suit of real world use cases, which is nice.", "other_comments_or_suggestions": "- Is it possible to tie the growth rate result in Theorem 2.2 to a property of the predictor, e.g., MSE?\n- One potential drawback is that if one's prediction are quite off, the resulting PPI e-value can be much smaller than the original true e-value --- is there a way of mitigating this and falling back to sampling every true e-value if the predictor is bad?\n- A nitpick, but the original prediction powered inference reduced variance of the estimator based on the accuracy of the predictor, even without active sampling. For prediction powered e-values, it seems like it's advantage lies solely in the active setting, and does not offer any benefits in the \"semi-supervised\" setting where a fixed batch of true e-values is collected. Is there any way to also use prediction powered e-values in the \"semi-supervised\" setting?", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4J4p7dt3Fn", "reviewer": "Reviewer_gWwp", "summary": "This paper introduces prediction-powered e-values. Its primary contribution is extending prediction-powered inference (PPI) beyond Z-estimation problems (e.g. inference of means) to the broader class of inference problems solvable via e-values. The authors show that their method retains key benefits of classic e-values, such as anytime-validity. Moreover, unlike classic PPI, their method allows updating the predictive model throughout the inference process. Empirical validation demonstrates its effectiveness in diverse scenarios, consistently yielding more precise inference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We'd like to thank the reviewer for their review. Please refer to our responses below.\n\n**Labeling budgets:** We agree. The choices of 1% and 0.5% are indeed a bit arbitrary, and were made mainly so as to be a reasonably large reduction in the number of collected labels, while keeping the underlying number of samples reasonably large. We will add experiments varying the budget in the camera-ready version. That said, the currently presented results aren't particularly sensitive to these sampling budgets, though if the number of labelled samples (not the proportion, the absolute number) goes under a certain threshold, then our method does struggle a bit (most likely because the predictive model deteriorates).\n\n**Evaluation of causal discovery setting:** The reviewer raises a good point. We will add comparisons of the average structural Hamming distance over several sampled random graphs to the camera-ready version. We have already run some first simulations, with favorable results: over 40 sampled graphs, the baseline of only labelled graphs obtains a mean structural Hamming distance of 12.15, whereas our method obtains that of 6.7; the best possible one (i.e., if we used all data available, as in Figure 4) is of 6.4.\n\n**Connections to the semiparametric inference and missing data literature:** Indeed, our prediction-powered e-values are essentially an AIPW-like estimator applied to the e-values. This construction was inspired from [Zrnic & Candès, 2024], which were in turn inspired by the usual doubly-robust estimators literature (cf. right after their equation (1)). We will make this connection more explicit in our paper. That said, perhaps the key difference between our method and usual semiparametric inference is that, rather than applying the AIPW-like estimator to the data, we apply them to the e-values -- i.e., directly to our measure of evidence. By doing so, we avoid asymptotics, and inherit all the properties of the original e-values, since all we require is that the e-value be at most one in expectation. The ability to update the predictor sequentially comes for free and is fairly easy to prove, with no significant restrictions (e.g. regularity conditions) necessary.\nIt is not clear whether our method can be seen as a more classic doubly-robust estimation for general e-values (as [Xu et al.] and [De Bartolomeis et al.] do for some previous PPI procedures). In the particular case of the e-value for the mean [Waudby-Smith and Ramdas, 2021], one can look at the resulting PPI e-value as the e-value for the mean applied on the AIPW-like estimator of [Zrnic & Candès]. But this observation does not seem to generalize to more complex e-values (e.g., those of https://arxiv.org/abs/2305.00143, or ones arising from p-to-e calibration).\n\n**Details on the use of the algorithm in the experiments:** In the experiments, the sequential updating of the predictive model was done by simply appending new available data and retraining on all (labelled) data prior, every 100 collected samples. We will add this to the experiments section.\n\n**Discussion on cases where PPI e-values would be underpowered:** Thank you for the suggestion, we will add this.", "claims_and_evidence": "The theoretical claims are convincingly supported. \n\nThe empirical evidence could be improved; see Methods And Evaluation Criteria.", "methods_and_evaluation_criteria": "Some aspects of the experimental setup could be improved:\n\n- Choice of labeling proportion (Sections 3.1 and 3.2):\n  The labeling budgets chosen (1% and 0.5%) are overly restrictive and somewhat arbitrary. An ablation study varying these proportions would improve the understanding of the method's effectiveness across different budget constraints.\n\n- Qualitative nature of causal discovery evaluation (Section 3.4): \n  The causal graph experiments currently rely on a single visualized random graph, making conclusions overly qualitative. To properly validate their method, the authors could sample multiple random graphs systematically and provide quantitative metrics (e.g. structural Hamming distance) averaged across these graphs.", "theoretical_claims": "I did not check the correctness of any proof.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "See: Essential References Not Discussed", "essential_references_not_discussed": "I believe there are essential related works missing from the current submission. In particular, the literature on semiparametric inference and missing data is highly relevant but currently omitted. \n\nThe prediction-powered e-value construction (lines 148-156) strongly resembles doubly robust estimators, introduced by:\n\n- J. M. Robins, A. Rotnitzky, and L. P. Zhao, *“Estimation of regression coefficients when some regressors are not always observed,”* Journal of the American Statistical Association, 89(427), pp. 846–866, 1994.\n- J. M. Robins and A. Rotnitzky, *“Semiparametric efficiency in multivariate regression models with missing data,”* Journal of the American Statistical Association, 90(429), pp. 122–129, 1995.\n\nThe main conceptual difference in the \"modern\" PPI framework is that the imputation model is trained externally on large independent datasets (see e.g. Section 5 in [1] and Appendix A.2 in [2] for a discussion). However, in this submission, the authors train the imputation model on the available labeled data, making their approach essentially an extension of classic semiparametric inference into a sequential setting with e-values. \n\nExplicitly discussing these connections would clarify the paper's position in relation to the established literature.\n\n[1] Xu, Zichun, Daniela Witten, and Ali Shojaie. \"A Unified Framework for Semiparametrically Efficient Semi-Supervised Learning.\" arXiv preprint arXiv:2502.17741 (2025).\n\n[2] De Bartolomeis, P., Abad, J., Wang, G., Donhauser, K., Duch, R. M., Yang, F., & Dahabreh, I. J. (2025). Efficient Randomized Experiments Using Foundation Models. arXiv preprint arXiv:2502.04262.", "other_strengths_and_weaknesses": "- I believe the algorithm is not described in sufficient detail. Specifically, the procedure for estimating and sequentially updating the predictive model is not clearly discussed, despite being central to the performance of the method. Providing details on how these models are initially trained and then updated (e.g., online vs. batch updates, frequency of updates) would be essential for understanding and replicating the approach.\n- An important limitation not explicitly discussed by the authors is that if the predictive model yields very inaccurate predictions (e.g. misspecified model), their proposed method could actually result in worse statistical efficiency or lower power compared to simply using the labeled data alone. A brief discussion or exploration of this potential issue—either theoretically or empirically—would strengthen the paper.", "other_comments_or_suggestions": "- typo line 87: ... over the distribution of the Y∗.s", "questions_for_authors": "- Your prediction-powered e-value appears structurally very similar to doubly robust estimators. Can you explicitly clarify how your approach differs from (or relates to) these classic doubly robust/AIPW methods, particularly since you train the predictive (imputation) models online using labeled samples rather than externally obtained data?\n\n- You chose labeling budgets of 1% (diabetes prevalence estimation) and 0.5% (risk monitoring). What motivated these particular choices, and how sensitive are your empirical conclusions to these specific labeling budgets? Could you provide some evidence of robustness across different budget values?\n\n- You currently provide causal discovery results on a single random DAG. Could you run your method across multiple random graphs and provide quantitative performance metrics (e.g., structural Hamming distance)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93PvHZ15fI", "reviewer": "Reviewer_piUd", "summary": "The authors propose to combine e-values to prediction-powered inference. The result is an e-value that combines in its definition both observed data as well as prediction of an auxiliary model. This approach is shown to improve the power of the testing procedure as long as the model is good enough. The author then provide three example applications encompassing mean estimation, change point detection and casual discovery. In all these cases, the PPI e-values are have better statistical power.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for pointing out the paper of [Xu et al.]. It is definitely a work we need to cite, and we will do so. That said, our contribution differs from theirs.\n\n1. Our construction is significantly more general than theirs. In equation (8) they construct what would be our prediction-powered e-process in the specific case of the mean-estimation e-process of [Waudby-Smith and Ramdas, 2021]; this particular construction can alternatively be seen as the standard mean-estimation e-process applied to the expectation constructed in [Zrnic & Candès, 2024]. Our construction, in contrast, is valid for virtually any e-value, which is much broader. For example, it is not at all clear how their construction could be leveraged for causal discovery, whereas ours is applicable in a straight-forward manner.\n2. The context is fairly different. In their paper, the primary concern is the construction of predictive sets that are endowed with a strong notion of validity. In our paper, we are concerned with statistical inference.\n3. Beyond constructing the e-values, we also investigate the statistical power of our procedure (growth rate, etc.) in the general case, which is something that [Xu et al.] do only in their restricted setting and for a lower bound on the growth rate (not the exact growth rate, as we do). Also note that our analyses on the power are not exactly trivial, as evidenced by the fact that stronger (but less compact) statements are left to the supplementary material.\n\nOverall, given these differences, we believe that our work is a significant contribution, and we hope the reviewer agrees.", "claims_and_evidence": "The claims follows straightforwardly from the properties of e-value and PPI, and they are correct and clearly stated. The authors also do a good job of at explaining the implications of the theoretical results.", "methods_and_evaluation_criteria": "Yes. All the considered evaluation criteria are well-suited to measure the statistical power of the testing procedure.", "theoretical_claims": "Yes, I have checked the theoretical claim about the growth rate and size of confidence intervals. They seem all correct.", "experimental_designs_or_analyses": "Yes and they seem all correct.", "relation_to_broader_scientific_literature": "I think the paper misses a critical reference Active, anytime-valid risk controlling prediction sets from Ziyu Xu and Nikos Karampatziakis and Paul Mineiro in NeurIPS 2024. In this paper the authors \" describe how to use predictors (i.e., the machine learning model for which we provide risk control guarantees) to further improve the utility of our RCPSes by estimating the expected risk conditioned on the covariates\" which to me appear to very similar to the approach presented in the paper.", "essential_references_not_discussed": "Active, anytime-valid risk controlling prediction sets from Ziyu Xu, Nikos Karampatziakis and Paul Mineiro in NeurIPS 2024.", "other_strengths_and_weaknesses": "I think the paper is very well-written, and the author does a great job of providing theoretical claims along with their practical implications. The idea of combining e-values and PPI is valid, and I appreciate the analysis of the growth rate of the proposed scheme. The experiments are well-designed, significant, and effectively support the paper’s claims.\n\nFrom my perspective, the main weakness of the paper is that the same idea has already been presented in Active, Anytime-Valid Risk-Controlling Prediction Sets by Ziyu Xu, Nikos Karampatziakis, and Paul Mineiro. See the section on \"Variance reduction through prediction\" in which the same e-processes based on PPI is proposed. This work was originally posted on ArXiv in June 2024, so I am unsure whether it qualifies as concurrent work. At the very least, it should be mentioned, and the differences between the two should be discussed.", "other_comments_or_suggestions": "My main suggestion is to include and clarify the similarities and difference from prior art.  \n\n##update after rebuttal: I decided to keep my score considering the overlap with existing work \"Active, Anytime-Valid Risk-Controlling Prediction Sets\".", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers appreciated the contribution of the paper that is a general framework that marries two timely topics: using prediction-power inference to make e-value calculations more sample-efficient/yielding higher power. The experiments are convincing and support the relevance of the work.

Please incorporate reviewer's comments in your final draft, e.g. add the connection to AIPW estimators in your positioning and further, make clear that rather than semi-supervised setting, you consider the active/experimental setting where you can obtain labels according to $\pi_i X(_i)$ rather than trying to model and learn $\pi$. However, similar to doubly-robust AIPW, my hunch would be that you can readily use those arguments to make it work (asymptotically) for the semi-supervised case.
Decision: Accept (poster)