Title: Active, anytime-valid risk controlling prediction sets
Venue: NeurIPS 2024
Accepted: yes
Keywords: distribution free

conformal prediction

e-process

confidence sequence
Abstract: Rigorously establishing the safety of black-box machine learning models with respect to critical risk measures is important for providing guarantees about the behavior of the model.
Recently, a notion of a risk controlling prediction set (RCPS) has been introduced by Bates et. al. (JACM '24) for producing prediction sets that are statistically guaranteed to have low risk from machine learning models.
Our method extends this notion to the sequential setting, where we provide guarantees even when the data is collected adaptively, and ensures the risk guarantee is anytime-valid, i.e., simultaneously holds at all time steps. Further, we propose a framework for constructing RCPSes for active labeling, i.e., allowing one to use a labeling policy that chooses whether to query the true label for each received data point, and ensures the expected proportion data points whose labels are queried are below a predetermined label budget. We also describe how to use predictors (e.g., the machine learning model we are providing risk control guarantees for) to further improve the utility of our RCPSes by estimating the expected risk conditioned on the covariates.
We characterize the optimal choices of label policy under a fixed label budget, and predictor, and show a regret result that relates the estimation error of the optimal labeling policy and predictor to the wealth process that underlies our RCPSes.
Lastly, we present practical ways of formulating label policies and we empirically show that our label policies use fewer labels to reach higher utility than naive baseline labeling strategies on both simulations and real data.

Active, anytime-valid risk controlling prediction sets

Abstract

Rigorously establishing the safety of black-box machine learning models concerning critical risk measures is important for providing guarantees about model behavior. Recently, Bates et. al. (JACM '24) introduced the notion of a risk controlling prediction set (RCPS) for producing prediction sets that are statistically guaranteed low risk from machine learning models. Our method extends this notion to the sequential setting, where we provide guarantees even when the data is collected adaptively, and ensures that the risk guarantee is anytime-valid, i.e., simultaneously holds at all time steps. Further, we propose a framework for constructing RCPSes for active labeling, i.e., allowing one to use a labeling policy that chooses whether to query the true label for each received data point and ensures that the expected proportion of data points whose labels are queried are below a predetermined label budget. We also describe how to use predictors (i.e., the machine learning model for which we provide risk control guarantees) to further improve the utility of our RCPSes by estimating the expected risk conditioned on the covariates. We characterize the optimal choices of label policy and predictor under a fixed label budget and show a regret result that relates the estimation error of the optimal labeling policy and predictor to the wealth process that underlies our RCPSes. Lastly, we present practical ways of formulating label policies and empirically show that our label policies use fewer labels to reach higher utility than naive baseline labeling strategies on both simulations and real data.

1 Introduction

One of the core problems of modern deep learning systems is the lack of rigorous statistical guarantees one can ensure about the performance of a model in practice. In particular, we are interested in ensuring the safety of a deep learning system so that it does not incur undue risk while optimizing for an objective of interest. This type of guarantee arises in many applications. For example, a deep learning based medical imaging segmentation system that detects lesions [36, 10, 29] should guarantee that it does not miss most of the lesion tissue while remaining precise and minimizing the total amount of tissue that is highlighted. Hence, it is crucial to provide a statistical guarantee about the 'safety' of any machine learning system to be deployed. Bates et al. [4] introduced the notion of a risk controlling prediction set as a method to derive such guarantees on top of the outputs of a wide range of black-box models. They consider the setting where all the calibration data for verifying statistical safety guarantees is available before deployment and the model only needs to be

Nikos Karampatziakis

calibrated once, i.e., the batch setting. However, this is often unrealistic in a production setup when we have no data concerning the performance of a model on the distribution of interest before we deploy it, and we wish to update our calibration each time a new data point (or group of data points) arrives. Consequently, it is natural to calibrate the machine learning model in an online fashion while receiving new data sequentially. Unfortunately, methods for obtaining statistical guarantees in the batch setting of Bates et al. [4] do not ensure risk control guarantees in the sequential regime. Further, when one uses data from production, the raw data is unlabeled, and one must expend resources (either paying experts or utilizing a more powerful model) to create gold labels for these data points. Hence, we generalize the sequential setup to an active setting in which we see the covariate X (e.g., an image, a natural language query from the user, etc.) and choose whether to query the true label Y . Concretely, consider the following scenarios where an active and sequential method is relevant.

Reduce query cost in medical imaging. A medical imaging system that outputs scores for for each pixel of image that determines whether there is a lesion or not would want to utilize labels given by medical experts for unlabeled images from new patients. Since the cost of asking experts to label these images is quite high, one would want to query experts efficiently, and only on data that would be most helpful for reducing the number of highlighted pixels.

Domain adaptation for behavior prediction. Often, the post-deployment distribution is different from that which has been seen before. For example, during a navigation task for a robot, we may want to predict the actions of other agents and avoid colliding into them when travelling between two points [20]. Since agents may behave differently in every environment, it makes sense to collect the behavior data in the test environment and update the behavior prediction in an online fashion to get accurate predictions calibrated for that specific environment.

Safe outputs for large language models (LLMs). One of the goals with large language models is to ensure their responses are not harmful in some fashion (e.g., factually wrong, toxic, etc.). One can view this as producing a prediction set for the binary label set of Y ∈ { harmful , not harmful } . Many pipelines for modern LLMs include some form of a safety classifier, which scores the risk level of an output, and determines whether it should be output to the user or not [21, 15], or a default backup response should be used instead. One would want to label production data acquired from user interaction with the LLM and used to calibrate cutoff for the scores that are considered low enough for the response to be allowed through.

Example: image classification Let us assume we wish to classify an image X ∈ X , and we have access to a probabilistic classifier s : X → ∆ Y where ∆ Y is the probability simplex over distributions over all possible classes, Y . Let s y ( x ) denote the probability of class y in the distribution s ( x ) . Based on the probabilities from s ( X ) , we can define C ( X,β ) to have the labels with the largest probabilities that sum to β ∈ [0 , 1] in the following fashion:

Now, we can define the miscoverage error of our label set C ( X,β ) as follows:

Now, assume that ( X,Y ) ∼ P ∗ , i.e., the images and class are jointly drawn from a fixed distribution. We want to find a choice of β such that ρ ( β ) := E [ r ( X,Y,β )] is guaranteed to be at most θ ∈ [0 , 1] , i.e., the expected miscoverage over the population of images and labels is at most θ .

In the above image classification example, we do not simply wish to find any β that ensures ρ ( β ) ≤ θ -setting β = 1 would trivially ensure this guarantee for any θ ∈ [0 , 1] . We also want to minimize the size of our uncertainty set C ( X,β ) . To present this formulation in more general terms, we are interested in solving the following problem for a fixed level of risk control θ ∈ [0 , 1] :

where g is the utility of our choice. We make the following natural assumption about r , ρ , and g . Assumption 1 . g and ρ are monotonically decreasing w.r.t. β and we assume ρ (1) = 0 . In addition, ρ is right-continuous.

Our image classification example has an expected risk and utility that satisfy the respective monotonicity assumptions, and such risk measures arise in many applications such as natural language question answering [26], image segmentation [1], and behavior control for robotics [20, 16]. Assumption 1 implies that maximizing g ( β ) is equivalent to minimizing β , as g is decreasing in β , and the right-continuity of ρ allows us to define the notion of an optimal calibration parameter that is the solution to (2):

Our goal in this paper is to derive a sequence of upper bounds on β ∗ that quickly approach the true β ∗ but provide 'anytime-valid risk control' in the sense that they are always greater the smallest 'safe' parameter β ∗ and induce risk under θ , i.e., β ≥ β ∗ implies that ρ ( β ) ≤ ρ ( β ∗ ) ≤ θ . Since we are guaranteed by Assumption 1 that ρ (1) = 0 ≤ θ , we always have a safe option of β = 1 to start with as our upper bound.

Our contributions. The primary contributions of this paper are as follows.

Extensions of RCPS to anytime-valid and active settings. We extend the notion of RCPS in two ways: (1) to enable anytime-valid RCPS which allows one to refine the set as one receives more samples in a stream while maintaining risk control throughout the entire stream, and (2) to define an RCPS that is valid under active learning, i.e., enable us to decide whether to label each example based on the covariates. We also define a way for incorporating risk predictions from the machine learning model to decrease variance further and reduce the number of labels needed to estimate β ∗ . We formulate this betting framework in Section 2.

Deriving powerful labeling policies and predictors. We show in Section 3 that our active, anytimevalid RCPS methods are practically powerful and converge to β ∗ in a label efficient manner by also deriving formulations for the optimal labeling policy and predictors under the standard log-optimality criterion that is used for evaluating anytime-valid methods [13, 34, 19]. We derive explicit regret bounds w.r.t. a lower bound on the growth rate on the wealth processes that underlie our RCPS methods. These bounds characterize how the deviation of any labeling policy and predictor from the log-optimal policy and predictor affect the growth rate of our wealth processes (and hence the earliest time at which a candidate β is removed from consideration as β ∗ ). In Section 4 we also show that machine learning model based estimators of the optimal policy and predictors are label efficient in practice through experiments.

Related work. Most relevant to this paper is the recent work from Zrnic and Cand` es [37] that provides a rigorous framework for statistical inference with active labeling policies, and leverages machine learning predictions through prediction-powered inference [2]. However, their focus is on M-estimation and deriving asymptotic, martingale central-limit theorem based results for a parameter. On the other hand, we provide finite sample anytime-valid results that are also valid at adaptive stopping times that directly utilize e-process [28] construction of sequential tests. Further, our goal is to provide a time-uniform statistical guarantee in the RCPS framework rather than directly estimating a parameter with adaptively collected data. We discuss additional related work in depth in Section 5.

2 Anytime-valid risk control through betting

We use ( X t ) t ∈ I to denote a sequence that is indexed by t with index set I . If the index set or indexing variable is apparent in context, we drop it for brevity. In our setup, we assume that our data points arrive in a stream ( X 1 , Y 1 ) , ( X 2 , Y 2 ) , . . . that proceeds indefinitely. Let ( F t ) be the canonical filtration on the data, i.e., F t := σ ( { ( X i , Y i ) } i ≤ [ t ] ) is the sigma-algebra over the first t points. Recall that we assumed ( X t , Y t ) ∼ P ∗ are i.i.d. draws for each t ∈ N := { 1 , 2 , 3 , . . . } , and want to control the risk ρ ( β ) = E ( X,Y ) ∼P ∗ [ r ( X,Y,β )] where the expectation is take only over ( X,Y ) . We illustrate an overview of our methodology (which we describe in the sequel) in Figure 1.

We desire to output a sequence of calibration parameters, ( ̂ β t ) , such that every β t is 'safe', i.e., ensures that the resulting risk of the output is provably controlled under a fixed level.

Definition 1 . A sequence of calibration parameters ( ̂ β t ) is said to have ( θ, α ) -anytime-valid risk control if it possesses the following property:

̂

̂

Features

(e.g., user prompt

+ LLM response)

Gold label

(e.g., toxicity of response)

Yt

Risk estimate

for each

r

t

(

,

β

)

β

∈ [0,1]

Incurred risk

for each

r

(

,

Yt

,

β

)

β

∈ [0,1]

Flip coin

w/

Lt

q

t

(

)

Black-box

model

Update threshold

β

t

-1

→ ̂

β

t

Deploy

with risk control

β

t

do query

label

(

)

Lt

= 1

don't query

label

(

)

Lt

= 0

label oracle

(e.g., human labeler, powerful LLM)

Treat risk as 0

Ensures

for all

with prob.

ρ

(

β

i

) = /u1D53C[

r

(

X

,

Y

,

β

i

)] ≤

θ

i

∈ 1,2,…

1 -

α

optional (decreases risk variance)

̂

We name this as 'anytime-valid' since the risk control condition (i.e., ρ ( ̂ β t ) ≤ θ ) is guaranteed to hold simultaneously at all t ∈ N . Hence, this allows for the user to process a continuous stream of data and control the probability that a ̂ β t is chosen at any time t that is 'unsafe', i.e., ρ ( ̂ β t ) > θ . We build on recent work that develops a framework for hypothesis testing and parameter estimation with sequential data collection based on martingales and gambling with virtual wealth known as testing by betting [30]. In this framework, the goal is to design an e-process , ( E t ) , w.r.t. a null hypothesis H 0 , which satisfies the following properties when true:

Definition 2 . An e-process , ( E t ) t ∈ N 0 , w.r.t. a hypothesis H 0 , is a nonnegative process for which there exists another nonnegative process, ( M t ) t ∈ N 0 s.t. the following is true when H 0 is true: (1) E [ M 0 ] ≤ 1 , (2) M t ≥ E t for all t ∈ N almost surely and (3) E [ M t | F t -1 ] ≤ M t -1 for all t ∈ N ,, i.e., ( M t ) is a supermartingale.

E-processes will be the main tool we use to construct ( ̂ β t ) . We leverage the probabilistic bound on e-processes provided by Ville's inequality to prove our anytime-valid risk control guarantee.

Fact 1 (Ville's inequality [33]) . For any e-process, ( E t ) , with initial expectation bounded by 1, i.e., E [ E 0 ] ≤ 1 , we have that

In this paper, all our e-processes will also be nonnegative supermartingales, so we denote them as ( M t ) . Now, we will specify the null hypotheses in our risk control setting. For each β ∈ [0 , 1] , we test the null hypothesis H β 0 : ρ ( β ) ≥ θ for a fixed risk control level θ ∈ [0 , 1] . Note that we include equality to θ in the null hypothesis since we do not wish to reject H β ∗ 0 . Let { ( M t ( β )) } β ∈ [0 , 1] be a family of e-processes where ( M t ( β )) is an e-process for H β 0 . Then, we can derive ̂ β t for each t ∈ N as follows:

Theorem 1. The sequence of estimates ( ̂ β t ) in (4) satisfies the anytime-valid risk control guarantee (3) , i.e., P ( ρ ( ̂ β t ) ≤ θ for all t ∈ N ) ≥ 1 -α .

Proof. First, we note that

̂

̂

Since H β ∗ 0 is always true by definition of β ∗ , we get that ( M t ( β ∗ )) is an e-process by Proposition 1. Thus, by applying Ville's inequality, we get that:

Now, we will present a concrete example of an e-process. Denote R t ( β ) := r ( X t , Y t , β ) . We test H β 0 using the betting e-process from Waudby-Smith and Ramdas [34]:

where ( λ t ) is predictable w.r.t. ( F t ) , i.e., λ t can be determined by F t -1 for each t ∈ N , and λ t ∈ [0 , (1 -θ ) -1 ] .

Proposition 1. ( M t ( β )) in (5) is an e-process for all β where H β 0 is true, i.e., where ρ ( β ) ≥ θ .

Proof. We note that ( M t ( β )) is nonnegative by the support of λ t being limited, i.e.,

Now, we will also show that ( M t ( β )) is a supermartingale when H β 0 is true.

The second equality is because λ t is measurable w.r.t. F t -1 and the last inequality is by R t ( β ) being independent of F t -1 and E [ R t ( β )] ≤ θ being true under H β 0 . Thus, we have our desired result.

Now, we have a concrete way to derive ( ̂ β t ) that ensures the risk ρ ( ̂ β t ) , is controlled at every time step t ∈ N . However, this requires one to label every example that arrives, i.e., it requires access to entire stream of labels ( Y t ) . We will now derive a more label efficient way for constructing ( ̂ β t ) .

Remark 1 . Ramdas et al. [27] show that e-processes of the form in (5) characterize the set of admissible e-processes (and hence anytime-valid sequential tests) for testing the mean of bounded random variables. Hence, it is an optimal choice of e-process for our setting, and have been shown to perform better both theoretically and empirically than other sequential tests for bounded random variables (e.g., Hoeffding and empirical-Bernstein based tests [34]).

2.1 Active sampling for risk control

Now, we describe active learning for risk control, where an algorithm sees X t decides whether a label, for the current point, Y t , should be queried or not. At each step t ∈ N , the algorithm produces a label policy q t : X → [ q min t , 1] based on the observed data (i.e., F t -1 ). It then queries the label, Y t , with probability q t ( X t ) that is lower bounded by a constant, q min t . Let L t be the indicator random variable for whether the t th label is queried, i.e., L t ∼ Bern ( q t ( X t )) .

To produce a label efficient method, one would hope to label the most 'impactful' data points that result in the largest growth of M t ( β ) for choices of β ∈ (0 , ̂ β t ) , i.e. that are still in consideration for the next ̂ β t +1 . For the labeling policies we consider in this paper, we let q min t ∈ [0 , 1] be a lower bound on the labeling probability, i.e., q t ( X t ) ≥ q min t almost surely. Thus, we can derive the following e-process for any sequence of labeling policies ( q t ) .

Proposition 2. Let ( q t ) be a sequence of labeling policies, and ( λ t ) be a sequence of betting parameters, and let both sequences be predictable w.r.t. ( F t ) (i.e., q t and λ t are measurable w.r.t. F t -1 for each t ∈ N ). Then, ( M t ( β )) in (6) is an e-process for all β where H β 0 is true.

Proof. The proof of this is similar to that of inverse propensity-weighted e-processes derived in Waudby-Smith et al. [35]. We know that ( M t ( β )) is nonnegative by the support of λ t being limited:

where the first inequality is by R t ( β ) ≤ 1 and q t ( X t ) ≥ q min t , and the second inequality is by λ t ≤ (( q min t ) -1 -θ ) -1 . Now, we will also show that ( M t ( β )) is a supermartingale. We first show the following upper bound:

The first equality is by further conditioning on X t , and the second equality is since L t is defined to be a Bernoulli random variable with parameter q t ( X t ) that is independent of all other randomness when conditioned on X t and F t -1 . The last inequality is by H β 0 being true. Now, we have that

where the inequality is by (7). Hence, we have shown that ( M t ( β )) is a nonnegative supermartingale, and hence also an e-process, under H β 0 .

Theorem 2. ( ̂ β t ) defined w.r.t. (6) satisfies the anytime-valid risk control guarantee (3) .

This is a result of Proposition 2 and Ville's inequality, similar to the proof of Theorem 1. Theorem 2 essentially shows that we can still design e-processes by allowing for a probabilistic label policy.

2.2 Variance reduction through prediction

Often, we also have an estimate of the risk we incur, e.g., in the example given for classification, we have an estimated probability distribution over possible outcomes. As a result, we also have an empirical estimate of E [ r ( X,Y,β ) | X = x ] for each β ∈ [0 , 1] that we can use to reduce the variance of our estimate. This is similar to the usage of control variates for improving Monte Carlo estimation [3, § V.2], and of predictors in the recently formulated prediction-powered inference framework [2]. Let ̂ r t : X × [0 , 1] → [0 , 1] be an estimator of the risk incurred by parameter β conditional on x ∈ X for each time step t ∈ N . ( ̂ r t ) is predictable w.r.t. ( F t ) .

Where does ̂ r come from? We note that often machine learning models have some estimate ˆ P ( X ) of the conditional distribution of Y | X (e.g, class probabilities, conditional diffusion models, LLMs, etc.). Thus, for any realized covariate x , we can derive use E Y ∼ ˆ P ( x ) [ r ( X,Y,β ) | X = x ] from the machine learning model as our choice of ̂ r ( x, β ) . This expectation can either be calculated analytically (as we do in or classification examples in our experiments) or derived using Monte Carlo approximation (for generative models such as LLMs, one can sample from the conditional distribution). In essence, we can obtain a predictor from the very model we are calibrating. We may also update our predictor using new ( X t , Y t ) pairs we receive for calibrating ( ̂ β t ) .

Now, we define our e-process that utilizes our predictor as follows:

and we restrict λ t ∈ [0 , (( q min t ) -1 -θ ) -1 ] (or in other words, q min t ≥ λ t / (1 + λ t θ ) ). Note that this e-process recovers the active e-process defined in (6) if we set ̂ r t ( · , β ) = 0 for all t ∈ N .

Proposition 3. ( M t ( β )) as defined in (8) is an e-process for H β 0 .

Proof. Since the restriction on ( λ t ) ensures M t ( β ) is nonnegative, to show that ( M t ( β )) is an e-process, it is sufficient to show:

The 3rd equality is by definition of L t , the last equality by the definition of ¯ R t , and the last inequality is due to H β 0 being true.

The role of ̂ r t ( X t , β ) is to accurately predict R t ( β ) . Bad predictions can increase the variance of ¯ R r ( β ) and lead to slower growth of M t ( β ) , but do not compromise the risk control guarantee. On the other hand, accurate predictions, which come from pretrained models, decrease variance and improve the growth of M t ( β ) . We characterize the optimal predictor (Proposition 4) and relate the accuracy of a predictor to its effect on the e-process (Theorem 3) in the next section.

3 Optimal labeling policies

Since the goal of having an active labeling policy is to label fewer data points, one reasonable way of doing this is to maximize the growth rate of our e-process ( M t ( β )) defined in (8). Define the following function, for some β ∈ [0 , 1] , of a labeling policy q , predictor ̂ r , and betting parameter λ where we let L | X ∼ Bern ( q ( X )) and ( X,Y ) ∼ P ∗ :

Define the growth rate at the t th step of ( M t ( β )) as G β t := E [ G β t ( q t , ̂ r t , λ t )] , where we let G β t be identical to G t but with X and Y replaced with X t and Y t , respectively. It is a standard notion of power or sample efficiency for e-processes. Typically, our goal when designing an e-process based test is to maximize such a metric, i.e., we want our e-process to be log-optimal [13, 34, 19]. Log-optimality is also called the Kelly criterion in finance [18] and it is known that maximizing the growth rate of a process is equivalent to minimizing the expected time for the process to exceed a threshold, i.e., for our sequential test to reject a value of β , in the limit as the threshold approaches infinity [5]. Thus, in an asymptotic sense, maximizing the growth rate is equivalent to minimizing the expected time for rejection. Our goal is to maximize the growth rate while having a constraint on the number of labels we can produce.

Let B ∈ [0 , 1] be the constraint on our labeling budget, i.e., we label, in expectation, a B fraction of all data points that we receive. To achieve both of these goals, we wish to choose q t , ̂ r t , and λ t that are the solutions to the following optimization problem:

Since solving the above optimization problem is analytically difficult, one can instead maximize a lower bound on the expected growth [32, 25, 24]:

which holds when λ ∈ [0 , (2( q min ) -1 -2 θ ) -1 ] , where q min := inf x ∈X q ( x ) . We can further simplify We can use the lower bound in (9) to formulate the following optimization problem.

Let ( q ∗ , r ∗ , λ ∗ ) be the tuple that is the solution to (10). We can analytically show what r ∗ is.

Proposition 4. The optimal predictor r ∗ in the solution to (10) is r ∗ ( x, β ) = E [ r ( X,Y,β ) | X = x ] for each x ∈ X .

We defer the proof to Appendix A.1. The optimal choice of q ∗ has the following formulation.

Proposition 5. If we fix ̂ r and λ , the solution to the optimization problem in (10) is given by q ∗ β where q ∗ β ( x ) ∝ √ E [ ¯ R ( β ) 2 | X = x ] for each x ∈ X if such a q ∗ β exists.

We defer the proof to Appendix A.2. Let σ β ( x ) := √ V [ r ( X,Y,β ) | X = x ] be the conditional standard deviation of r ( X,Y,β ) . Now, we can argue that the solution to the optimization problem on the growth rate lower bound in (10) has the following characterization.

Corollary 1 . The optimal choice of q ∗ β and λ ∗ that solves (10) is

if q ∗ β ( x ) ∈ [0 , 1] for all x ∈ X , and λ ∗ ≤ (2(inf x ∈X q ∗ β ( x )) -1 -2 θ ) -1 . The resulting growth rate has the following lower bound:

We can show this is true as a consequence of Proposition 4, Proposition 5, and solving the quadratic equation that arises for the growth rate to derive the optimal choice of λ ∗ . Further, we note that we can define regret of a sequence ( λ t ) compared to λ ∗ on ̂ G β t as follows.

Definition 3 . The ̂ G β -regret at the t th step of a sequence of betting parameters ( λ t ) for a risk upper bound θ ∈ [0 , 1] , and a sequence of labeling policies ( q t ) and predictors ( ̂ r t ) where q t ( x ) ≥ ε > 0 for all x ∈ X and t ∈ N almost surely is defined as follows:

Since ̂ G β t ( q, ̂ r, λ ) is exp-concave in λ , existing online learning algorithms such as Online Newton Step (ONS) [8] can get o ( T ) regret guarantees, which means that the growth rate of ( λ t ) averaged over time will approach (or exceed) the optimal growth rate under λ ∗ . For simplicity of analysis, we make the following assumption about the labeling probability of the optimal policy, q ∗ β .

Assumption 2 . Let ε > 0 be a positive constant. Assume that q ∗ β ( x ) ≥ ε for each x ∈ X .

The lower bound in the above assumption is an analog of the propensity score lower bound on optimal policies that is needed for proving valid inference in adaptive experimentation [17, 7]. Further, we do not need this assumption to hold on every β , since we are not necessarily interested in log-optimality w.r.t. fringe β that are quite far away from β ∗ -in practice having this assumption hold for values of β near β ∗ suffices to develop an estimator ̂ β that shrinks toward β ∗ quickly. Now, we describe how much the growth rates deviates based on on how well q ∗ and r ∗ are estimated.

Theorem 3. Let ( λ t ) be a sequence with ̂ G β -regret ( Reg t ) and ( q t ) and ( ̂ r t ) are sequences of labeling policies and predictors that are all predictable w.r.t. ( F t ) . For a positive constant ε > 0 , let q t ( x ) ≥ ε > 0 for each t ∈ N and x ∈ X almost surely. Under Assumption 2 for the same ε , the following bound holds:

We defer the proof to Appendix A.3. The proof idea follows a similar idea that of the regret bound in Kato et al. [17] for deriving an estimator that is close to the optimal estimator for the average treatment effect in an adaptive experimentation setup. Theorem 3 relates the estimation error of q ∗ β ( x ) and r ∗ ( x, β ) to how quickly β will be deemed 'safe'. Hence, if we have good estimates of those quantities, then we can produce an estimates ( ̂ β t ) that are small and close to β ∗ while remaining safe. We will now describe some practical methods for calculating q t and ̂ r t , and demonstrate their empirical performance in some experiments.

4 Experiments

We use PyTorch to model our ( q t ) and ( ̂ r t ) , and we consider the following formulations. 2

Baseline labeling policies. We have baseline labeling policies of labeling all data that arrives, and a policy that just randomly samples B proportion of samples to label - these are denoted respectively as 'all' and 'oblivious'.

Pretrain : We derive an estimate of r ∗ from a pretrained machine learning model, ̂ r pretr , to be our choice of predictor for all time steps. We also derive an estimate of σ ( x, β ) , ̂ σ pretr ( x, β ) , from the pretrained model. We also learn a sequence of normalizing constants ( C t ) s.t. the budget is satisfied. Our labeling policy in this case is q t ( x ) = ̂ σ ( x, ̂ β t -1 ) /C t , where we want to optimize our policy for the previous best bound on β ∗ , ̂ β t -1 . We denote this method as 'pretrain'.

Estimating q ∗ β and r ∗ : We learn sequences of models ( ̂ σ plugin t ) and ( ̂ r plugin t ) using the labeled data points. We preprocess the outputs from ̂ σ pretr and ̂ r pretr to use as the input features to these models, respectively. Each of these sequences of models are then updated at every step. We also similarly learn a sequence of normalization constants ( C t ) for deriving the final labeling policy ( q t ) .

We provide more details on how are methods are formulated in Appendix B. We run all our experiments on a 48-core CPU on the Azure platform, after using a GPU to precompute the predictions made by neural network models. We set θ = 0 . 1 , α = 0 . 05 , and B = 0 . 3 for all our experiments.

4.1 Numerical simulations

We have a simple data generating process of sampling P t ∼ Uniform [0 , 1] and let Y t | X t ∼ Bern ( X t ) . This simulates the setting we have with our real data where we have an accurate pretrained classifier that have a probability estimate of Y t of being 0 or 1. We let our covariates X t = P t . As a result, our risk function is the false positive rate r FPR ( X,Y,β ) := 1 { X ≥ β, Y = 0 } . We run 100 trials where each trial runs until 2500 labels are queried. We compare our methods based on their label efficiency, i.e., how close is ̂ β t to β ∗ = 1 -√ 2 α after a set number of queried labels. In Figure 2, we plot the average ̂ β t reached after a given number of labels queried across trials. The shaded areas denotes pointwise 95% confidence intervals on the uncertainty of the average estimate. We can see that the 'pretrain' and 'learned' methods outperform both the 'all' and 'oblivious' strategies uniformly numbers across labels queried. In Figure 2a, we show the average rate of safety violations, i.e., the average proportion of trials that ̂ β t was unsafe and ρ ( ̂ β t ) > θ at any time step. We can see that all methods control the desired safety violation rate at the predetermined level α .

0.100

0.075

0.050

0.025

0.000

oblivious

司

pretrain learned

Method

(a) Average rate of safety violations ̂ β t .

0.58

( 0.57

0.56

oblivious pretrain

learned

Method

(b) Average final value of ̂ β t (lower is better).

1.0

Method

0.9

all

(β

0.8

learned

oblivious

pretrain

0.7

0.6

1000

2000

2

Labels queried

(c) Average ̂ β t vs. labels queried (lower is better).

4.2 Imagenet

We also evaluate our methods on the Imagenet dataset [9], and we used the pretrained neural network classifiers from Bates et al. [4] to provide estimates of the class probabilities.

Since Imagenet is a classification task with label support on Y = [1000] , our goal is to ensure that the miscoverage rate of the true class is controlled. We follow the same setup as descibed in the introduction, i.e., with our risk measure r specified according to (1). For Imagenet, we reshuffle our dataset for each trial, and run each method till we have queried 3000 labels.

rate

E

rror

0.06

rate

rror

0.04

0.02

0.00

oblivious

pretrain

learned

Method

(a) Average rate of safety violations ̂ β t .

0.88

0.87

(β

0.86

0.85

0.84

oblivious

司

pretrain

learned

Method

(b) Average final value of ̂ β t (lower is better).

1.00

Method

all

0.95

learned

(β

0.90

pretrain

oblivious

0.85

0

2000

Labels queried

(c) Average ̂ β t vs. labels queried (lower is better).

Figure 3: Experimental results for different methods on Imagenet. Again, we see that 'pretrain' and 'learned' are the best performing, and they have very similar performance and hence overlap in Figure 3c. Here, β ∗ = 0 . 8349 , and is delineated by the dotted line in Figures 3b and 3c. Again, each method also has low safety violation rate, i.e., is below the dotted line of α = 0 . 05 in Figure 3a.

In Figure 3, we plot the average ̂ β t across trials. Once again, we can see that the 'pretrain' and 'learned' methods outperform both the 'all' and 'oblivious' strategies here as well. On Imagenet the average safety violation rate is also controlled as well under the predetermined level of α = 0 . 05 .

5 Additional related work

Casgrain et al. [6] provide anytime-valid sequential tests for identifiable functions, which result in similar hypotheses being tested as this paper albeit with equality instead of equality. They, in addition to other recent work [25, 24, 31], using regret bounds for betting-based e-processes to show either derivations for the growth rate of a betting strategy w.r.t. to the optimal growth rate. However, none of these settings incorporate the ability to perform adaptive sampling or inverse propensity weights. Prior work in anytime-valid inference have included inverse propensity weights have been for off policy evaluation [35], adaptive experimentation [7], or estimating the weighted mean of a finite population [32]. However, none of these works explicitly characterize deviation in the sampling policy away from the optimal sampling policy ultimately affects the growth rate as we do in Theorem 3.

Our analysis of power and regret for our algorithm is quite similar to methods in adaptive experimentation for average treatment effect estimation [14, 17] that attempt to derive a no regret treatment policy and outcome regressor that produces an estimator with a variance that approaches the variance of the optimal estimator. Unlike the adaptive experimentation setting, however, we have an additional label budget constraint on our formulation that results in a different optimal policy.

6 Conclusion, limitations, and future work

We have shown that we can extend the RCPS formulation to be anytime-valid, and retain validity and increase label efficiency in an active learning setting. We use the theory of betting and e-processes to develop this framework and show it is verifiably safe, and we verified this with our experimental results. We have primarily considered the i.i.d. setting here for anytime-valid calibration, and one key area in which one can extend this line of work is to account for distribution shift during test time. The empirical Bernstein supermartingales in Waudby-Smith et al. [35] can likely be used to extend our framework control risk in an average sense, but stronger guarantees could be made about the provided risk control if more realistic assumptions are made about the nature of the distribution (e.g., covariate shift, label shift, etc). It may also be possible extend a notion of adaptive conformal inference (ACI) [11, 12] to anytime-valid risk control. Another limitation of this work is the bounded label policy assumption (i.e., Assumption 2) and existence assumption in Proposition 5. We believe that more careful analysis can get rid of these assumptions in future work.

References

A. N. Angelopoulos, A. P. Kohli, S. Bates, M. I. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano. Image-to-Image Regression with Distribution-Free Uncertainty Quantification and Applications in Imaging. In International Conference on Machine Learning , 2022.

A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic. Prediction-powered inference. Science , 382(6671):669-674, 2023.

S. Asmussen and P. W. Glynn. Stochastic Simulation: Algorithms and Analysis . Stochastic Modelling and Applied Probability. Springer, New York, 1 edition, 2007.

S. Bates, A. Angelopoulos, L. Lei, J. Malik, and M. Jordan. Distribution-free, Risk-controlling Prediction Sets. Journal of the ACM , 68(6):43:1-43:34, 2024.

L. Breiman. Optimal Gambling Systems for Favorable Games. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics , pages 65-79, 1961.

P. Casgrain, M. Larsson, and J. Ziegel. Sequential testing for elicitable functionals via supermartingales. Bernoulli , 30(2):1347-1374, 2024.

T. Cook, A. Mishler, and A. Ramdas. Semiparametric Efficient Inference in Adaptive Experiments. In Conference on Causal Learning and Reasoning , 2024.

A. Cutkosky and F. Orabona. Black-Box Reductions for Parameter-free Online Learning in Banach Spaces. In Conference On Learning Theory , 2018.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition , 2009.

A. M. Flores, F. Demsas, N. J. Leeper, and E. G. Ross. Leveraging Machine Learning and Artificial Intelligence to Improve Peripheral Artery Disease Detection, Treatment, and Outcomes. Circulation Research , 128(12):1833-1850, 2021.

I. Gibbs and E. Candes. Adaptive Conformal Inference Under Distribution Shift. In Neural Information Processing Systems , 2021.

I. Gibbs and E. Cand` es. Conformal Inference for Online Prediction with Arbitrary Distribution Shifts. Journal of Machine Learning Research , 25(162):1-36, 2024.

P. Gr¨ unwald, R. de Heide, and W. Koolen. Safe Testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 2024.

J. Hahn, K. Hirano, and D. Karlan. Adaptive Experimental Design Using the Propensity Score. Journal of Business & Economic Statistics , 29(1):96-108, 2011.

L. Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.

H. Huang, S. Sharma, A. Loquercio, A. Angelopoulos, K. Goldberg, and J. Malik. Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts. In IEEE International Conference on Robotics and Automation , 2024.

M. Kato, T. Ishihara, J. Honda, and Y. Narita. Efficient Adaptive Experimental Design for Average Treatment Effect Estimation. arXiv:2002.05308, 2021.

J. L. Kelly. A New Interpretation of Information Rate. The Bell System Technical Journal , page 10, 1956.

M. Larsson, A. Ramdas, and J. Ruf. The numeraire e-variable and reverse information projection. arXiv:2402.18810, 2024.

J. Lekeufack, A. N. Angelopoulos, A. Bajcsy, M. I. Jordan, and J. Malik. Conformal Decision Theory: Safe Autonomous Decisions from Imperfect Predictions. In IEEE International Conference on Robotics and Automation , 2024.

T. Markov, C. Zhang, S. Agarwal, F. E. Nekoul, T. Lee, S. Adler, A. Jiang, and L. Weng. A Holistic Approach to Undesired Content Detection in the Real World. AAAI Conference on Artificial Intelligence , 2023.

F. Orabona and T. Tommasi. Training Deep Networks without Learning Rates Through Coin Betting. In Neural Information Processing Systems , 2017.

A. B. Owen. Monte Carlo theory, methods and examples . https://artowen.su.domains/ mc/ , 2013.

A. Podkopaev and A. Ramdas. Sequential Predictive Two-Sample and Independence Testing. In Neural Information Processing Systems , 2023.

A. Podkopaev, P. Bl¨ obaum, S. Kasiviswanathan, and A. Ramdas. Sequential Kernelized Independence Testing. In International Conference on Machine Learning , 2023.

V. Quach, A. Fisch, T. Schuster, A. Yala, J. H. Sohn, T. S. Jaakkola, and R. Barzilay. Conformal Language Modeling. In International Conference on Learning Representations , 2024.

A. Ramdas, J. Ruf, M. Larsson, and W. Koolen. Admissible anytime-valid sequential inference must rely on nonnegative martingales. arXiv:2009.03167, 2022.

A. Ramdas, P. Gr¨ unwald, V. Vovk, and G. Shafer. Game-Theoretic Statistics and Safe AnytimeValid Inference. Statistical Science , 38(4):576-601, 2023.

L. Saba, S. S. Sanagala, S. K. Gupta, V. K. Koppula, A. M. Johri, N. N. Khanna, S. Mavrogeni, J. R. Laird, G. Pareek, M. Miner, P. P. Sfikakis, A. Protogerou, D. P. Misra, V. Agarwal, A. M. Sharma, V. Viswanathan, V. S. Rathore, M. Turk, R. Kolluri, K. Viskovic, E. Cuadrado-Godia, G. D. Kitas, N. Sharma, A. Nicolaides, and J. S. Suri. Multimodality carotid plaque tissue characterization and classification in the artificial intelligence paradigm: A narrative review for stroke application. Annals of Translational Medicine , 9(14):1206, 2021.

G. Shafer. Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 184(2):407-431, 2021.

S. Shekhar and A. Ramdas. Nonparametric Two-Sample Testing by Betting. IEEE Transactions on Information Theory , 70(2):1178-1203, 2024.

S. Shekhar, Z. Xu, Z. Lipton, P. Liang, and A. Ramdas. Risk-limiting financial audits via weighted sampling without replacement. In Conference on Uncertainty in Artificial Intelligence , 2023.

J. Ville. ´ Etude Critique de la Notion de Collectif . PhD thesis, University of Paris, Paris, 1939.

I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B (Statistical Methodology) , 2023.

I. Waudby-Smith, L. Wu, A. Ramdas, N. Karampatziakis, and P. Mineiro. Anytime-valid off-policy inference for contextual bandits. ACM / IMS Journal of Data Science , 2024.

J. Wu, J. Xin, X. Yang, J. Sun, D. Xu, N. Zheng, and C. Yuan. Deep morphology aided diagnosis network for segmentation of carotid artery vessel wall and diagnosis of carotid atherosclerosis on black-blood vessel wall MRI. Medical Physics , 46(12):5544-5561, 2019.

T. Zrnic and E. J. Cand` es. Active Statistical Inference. In International Conference on Machine Learning , 2024.

A Omitted proofs

Proofs that have been deferred from the main body of the paper are contained here.

A.1 Proof of Proposition 4

We can rewrite the objective in the following way:

Maximizing (11) is the same as minimizing the following equivalent expressions:

where we derive the 1st equality from the law of total variance, and the 2nd equality from the fact that ̂ r ( X,β ) is fixed given X .

Since the only term that ̂ r affects is the integral term in (12), we can choose each ̂ r ( x, β ) for each x ∈ X to minimize the following:

If we remove the constants (i.e., terms unaffected by ̂ r ( x, β )) , and note that q ( x ) -1 -1 > 0 , we get that minimizing (13) is equivalent to minimizing

This is equivalent to minimizng the squared error, i.e., ( E [ r ( X,Y,β ) | X = x ] -̂ r ( x, β )) 2 , which means that r ∗ ( x, β ) = E [ r ( X,Y,β ) | X = x ] , which gets us our desired result.

A.2 Proof of Proposition 5

Since we have shown that maximizing (10) is equivalent to minimizing (12), we can isolate the terms that change wiht q and see that we are looking for the solution to the following optimization problem:

We can define φ ( x ) := p ( x ) q ( x ) rewrite this as

Assume we can define a valid q ∗ β where q ∗ β ( x ) ∈ [0 , 1] for each x ∈ X that satisfies the following conditions:

Explicitly, we define q ∗ β as follows:

One can show its optimality by considering some other labeling policy q ′ where E [ q ′ ( X )] = B (we use a similar proof technique from importance sampling Owen [23, § 9.1]). Now, let φ ∗ ( x ) := p ( x ) q ∗ β ( x ) and φ ′ ( x ) = p ( x ) q ′ ( x )

where the last line is by Cauchy-Schwarz, since φ ′ ( x ) /B is a valid p.d.f. Hence, we have shown our desired result.

A.3 Proof of Theorem 3

By definition of Reg t , we know that

by taking an expectation over F i -1 for each term in the summation Hence, what remains to be shown is the following:

Let ¯ R t ( β ) := r ( X,Y,β ) -̂ r t ( X,β ) and ¯ R ∗ ( β ) := r ( X,Y,β ) -r ∗ ( X,β ) . Wefirst note the following identity using (12):

Now, we make the following derivations for the difference between the variance ( V ) terms:

The 1st equality is by substituting in the identity from (12). The 1st inequality is a result of V [ r ( X,Y,β ) | X = x ] ≤ 1 4 , since r ( X,Y,β ) ∈ [0 , 1] , and q ∗ β ( x ) , q t ( x ) ∈ [ ε, 1] almost surely. The 2nd inequality is by upper bounding q ∗ β ( x ) -q t ( x ) by its absolute value.

Now, if we plug (15) into (14), take the expectation over F t -1 , and take the summation over t , we get our desired result.

B Experiment details

In this section, we discuss additional details about how we implement our methods described in Section 4.

B.1 Formulation of the labeling policy

For the 'pretrain' policy, we use an estimate of the conditional mean and variance derived from s .

These estimates may not be accurate, but might still represent a reasonable partitioning of the feature space where σ ( x, β ) and r ∗ ( x, β ) are similar. Hence, for ̂ σ plugin t and ̂ r plugin t , we model them as linear regresssion models where inputs are a binning of ̂ r pretr ( x, β ) and ̂ σ pretr ( x, β ) , respectively. We then learn the regression model parameters on training data.

B.2 Optimization to maintain the budget constraint

For any predictor ̂ σ , we optimize the Lagrangian corresponding to (10), which is defined as follows for a fixed λ t .

Since we know the optimal form of ̂ r t , we optimize it separately by taking an optimization step with the loss of squared error, ( r ( X,Y,β ) -̂ r t ( X,β )) 2 , for each labeled example for a grid of β values.

To derive the solution, we simplify playing the minimax game with the above Lagrangian to the following objective:

In the case of both 'pretrain' and 'learned' methods, we parameterize our q t in the following fashion:

where our normalization constant C t = exp( c t ) for some value c t ∈ R to ensure it is nonnegative.

̂ σ t is updated separately. For 'pretrain', it is fixed from the beginning, and for 'learned', we take an optimization step to minimize the squared loss against the squared residual, i.e., we update to minimize (( r ( X,Y,β ) -̂ r t ( X,β )) 2 -̂ σ t ( X,β ) 2 ) 2 .

Hence, the only thing that remains to optimize c t , which now simply means we need to solve the following problem (where we treat ̂ β t -1 as fixed):

The actual game payoff we play is the stochastic approximation of the Lagrangian in the following form:

At each step, we take an optimization step on c t towards maximizing the above loss, and determine ν t by playing either best response or a windowed best response that takes an average of best responses over recent rounds. We use the COCOB optimizer [22] for all of learning and optimization which requires no hyperparameter or learning rate selection.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While '[Yes] ' is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] ' provided a proper justification is given (e.g., 'error bars are not reported because it would be too computationally expensive' or 'we were unable to find the license for the dataset we used'). In general, answering '[No] ' or '[NA] ' is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist' ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide exactly what we describe in the abstract/introduction, i.e., a method for constructing active, anytime-valid risk controlling prediction sets along with theoretical guarantees and experiments demonstrating its efficacy.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We elaborate on the limitations of the paper in Section 6 Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide proofs and assumptions for each result (clearly delineated) in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the code for reproducing the experiments in a supplement.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide all code for reproducing the experiments in the paper in a supplement (see above).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide experimental overview in Section 4, and we provide additional details in both the code and the appendix (Appendix B).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We show error bars (from 95% normal CIs) for all our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify our compute resources in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I have reviewed the ethics guidelines and this paper conforms with it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is statistical methodology/theory for enabling active labeling for risk control - it may allow users of machine learning models more cost-effectively calibrate their models to control measures of harmful risk when using them in practice, but there are no direct societal impacts as far as we can discern.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper does not release new data/models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we properly cite the papers for which we use models, code, and data from (e.g, Imagenet, RCPS, etc.).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide our experimental code in the supplement and provide details about it in Appendix B.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We don't perform human research for this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We don't perform human research in this paper.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a probabilistic strategy for stream-based active learning which allows for construction of anytime valid prediction sets. The strategy stems from maximizing the variance process of an e-process, which is also used in designing the prediction sets. Relevant theoretical guarantees (validity and regret) and proof-of-concept experiments are presented.\n\nSTRENGTHS:\nFor me, the most interesting contribution of the paper, is that it brings the perspective of safe betting to active learning.\nThe results are not written in the most flashy way, but the overall thought process is straightforward to follow and proofs are clear.\n\nWEAKNESSES:\n- My assessment is that Section 2 lacks novelty, as in, very similar results have been proposed in earlier work under slightly different problem setting. Would be nice to make section 3 the main contribution of the paper, and present section 2 as a background/recipe section which leads to results of Section 3.\n  - The proposed e-processes of Eq (5) and (6) are both from results of Waudby-Smith [2023 & 2024]. If i understand correctly, both of these processes are proposed for a more general frame-work, and the current paper applies it to its problem setting.\n  - Proposition 1 and Theorem 1 are immediate corollaries by definition of the betting e-processes. \n    - It is best that this is merged together and presented as a corollary for the specific choice of $M_t$. \n    - Theorem 1 holds true for any $M_t$ is an e-process under the null. In the proof however, Proposition 1 is invoked. Would be good if the statement is updated and the assumption on $M_t$ is stated in it.\n\n- I am not widely aware of the literature. I just wanted to raise this flag that the related works section might be inadequate. I was expecting more work to have been done in this area. After reading the paper, I do not have a clear image of short history of results leading to this work, and I think this speaks of an incomplete literature review. There's no related work from active learning literature, while they aim to solve a relatively close problem (in cases identical).\n\n\n- The contributions section, particularly the second paragraph, is too technical given what it is already introduced. I would generally shorten the introduction and present a more thorough literature review in the main text.\n- I think parts of the problem setting are confusing, if not incorrect and lines 44-52 should be re-written. \n  - Line 47 says $f$ outputs and action $A$ within a set $\\mathcal A$. But later in Line 50, it is written that $f$ is a prediction set for $Y$.\n  - Connection between $\\mathcal A$ and $\\mathcal Y$ is missing. I'm assuming $f$ output subsets of $\\mathcal Y$ and I don't understand the auxiliary confusing notation of $A$ and $\\mathcal A$.\n  - If $f$ outputs a prediction set, then the notation $A$ and calling it an action is utterly confusing. Within active learning literature, $X$ is typically the action.\n  - It is not evident in the notation of the risk that $\\rho$ depends on $f$.\n\nQUESTIONS:\n1) Why is assumption 1 in terms of $\\rho$ and not just $r$? Is this needed? If possible, stating the assumption in terms of $r$ is cleaner in my opinion, since this way $\\mathcal X$ and $\\mathcal Y$ become assumption free.\n2) The way section 2.2 is stated, what I collect from it is that the $M_t$ process is robust to adding a $\\mathcal F_t$-adapted and bounded bias term. I can imagine that adding this bias may help with rapid increase of variance process of $M_t$. But I did not understand the motivation of this variance reduction, within the context of section 2. Could you please elaborate?\n3) The proposed optimal labeling policy in Section 3 somewhat resembles a variance maximization techniques which are standard practice in stream-based active learning. Have you looked into the connections? How do these relate?\n4) Doesn't the problem setting allow for comparison to standard active learning policies? Adding these would make the paper relevant to a considerable larger community.\n\nLIMITATIONS:\nLimitations and assumptions are adequately stated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors, in their work, extend the framework of Risk Controlling Prediction Sets (RCPS) to a sequential setting where data is collected adaptively, providing anytime-valid risk guarantees. Additionally, it proposes a framework for active labeling, which allows for selective querying of true labels within a predefined budget, enhancing the utility of RCPS by leveraging predictors to estimate expected risk based on covariates. Next, the authors extend the setting further and develop an active learning setting and optimal labeling policy and under a fixed label budget.\n\nSTRENGTHS:\n- Extended setting of a risk-controlling prediction set setting.\n- Possibly rigorous in theoretical analysis.\n\nWEAKNESSES:\n- The paper is notation-heavy, considers complex settings, and uses concepts that are not so commonly known. Unfortunately, the authors are doing nothing to help readers understand their work.\n- Many symbols (and there are many) are only introduced once.\n- The goal of the method is easy to miss in the body of the text.\n- The experimental section is hard to understand without reading the theory sections, and there are no meaningful conclusions.\n- There are no summaries, conclusions, schemas, pseudo-codes, or additional intuitions to help the reader understand the paper.\n- Experiments feel very limited.\n- I have doubts about the practicality of the introduced setting, as the paper needs better application examples.\n\nDue to very limited time, I was not able to read related works that were necessary to fully understand this paper, verify the theory, and possibly appreciate it fully. I would like to believe that all the theories presented in the paper are right, but even assuming that. I think the paper is below acceptance due to quite a poor presentation.\n\nQUESTIONS:\n- What are other examples of applications for the proposed framework?\n- How in practise one should select $\\theta$ and $\\alpha$?\n\nLIMITATIONS:\nI see no potential negative social impact of this work. Discussion on limitations is limited.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe setting extends the model of Bates et al.---which provide confidence-bound type guarantees on the performance of a trained ``black-box\" predictor which are parametrised, and nested with respect to a monotonic parameter $\\beta$---to the online setting. The goal of the original setting is to provide risk-controlling prediction sets (RCPS) mapping a feature $X$ to a set $\\mathcal{T}(X)$ in the label space, which is judged on the basis of a predetermined safety criterion. The online extension is then well-justified by the observation that the calibration data itself is often limited without deploying the model in practice, particularly if training occurs in an online fashion. \\\\\nThe contributions of the paper are:\n\n1. An extension of the RCPS notion to the online setting, and a derivation of guarantees that are anytime-valid. In other words, confidence sets are refined as data is accrued, nevertheless maintaining risk control over the entire stream.\n2. An extension of the RCPS which is valid under active learning, where a learner may choose to obtain a label based on the covariates, and a fixed total query budget.\n\nIn addition, the authors provide guarantees in the regret sense on the performance of derived methods in terms of the log-optimality criterion common for evaluating anytime-valid methods, which decouples into the regret of an exp-concave sequence dependent only on a quantity called the betting fraction, which due to log-concavity may be estimated sub-linearly in the number of rounds $T$ (due to exp-concavity of the log-optimality with respect to the betting fraction, this should be $O(\\log(T))$) plus two concentration-type terms dependent on the convergence of two additional parameters derived from the risk, which may be estimated from the data. The convergence of such estimates will depend on the variance of the classifier's risk, and is determined on a case-by case basis. Experiments are included for verification of theoretical guarantees. \n\n\nConclusion:\nWhile there are some weaknesses with regard to the communicability of the results for the intended audience, I believe on balance that the ideas in this paper will turn out to be of broad interest to the experimental community, and have clear practical relevance. From the theoretical perspective, although many of the ideas are directly generated from the theory of e-statistics, this is an encouraging example of tailoring theory to a clear practical need.\n\nSTRENGTHS:\n1. The formalism is well developed, and demonstrates a high-degree of understanding of the use of e-variables as diagnostic tools of testing processes, and the derived confidence methods are a natural extension of the theory to a very practical setting.\n2. The extension to label-efficient active learning further pushes the abstract theory into the realms of practicability, and the authors provide an example of a set of estimators for the various relevant quantities in the experiments. This gives an (somewhat implicit) recipe for practitioners looking to deploy the tools developed.\n3. The regret bound derived gives an interpretable decomposition of the optimality of the regret of the growth-function (used to bound the log-optimality criterion), which decouples statistical quantities from the betting fraction, which is obtainable from standard online learning methods. \n4. The experiments (although arguably non-extensive) provide minimal necessary examples and concrete estimators for the relevant quantities present in the regret bounds, and demonstrate empirically the convergence of the sum total of relevant components to their optimal values.\n\nWEAKNESSES:\n1. A few more examples of the method being used in practice, along with the guarantees from the regret bound explicated, conditional upon the estimators idiosyncratic to the exact settings treated would be much more helpful than the abstract bound. Although for a theorist these results make sense, and keeping the general form due to the presence of the risk estimator variance and other empirical quantities does find some justification due to the variability of these quantities (which may alter the order of the regret bound), a few examples would really help illustrate the process + guarantees for a practitioner, which I guess would be the intended audience. I think this paper highlights the difficulty of the communicability of theoretical results in a digestible fashion, but does not constitute a weakness of the results themselves. \n2. The experiments are not particularly extensive, and only encompass two examples; a contrived example of uniformly drawn features with Bernoulli labels (which is still a demonstrative diagnostic for the methods described, but have very well-behaved associated estimates), and a more realistic example based on the Imagenet dataset. Furthermore, as mentioned above, the comparison of online performance with respect to the regret bound would be even more helpful.\n3. It would be nice to see some experiments illustrated with harder examples, for example in cases where the convergence of estimates would have a different than $O(\\sqrt{T})$ rate.\n\nQUESTIONS:\n1. To my understanding, the convergence rate of the estimation terms largely dominate the $O(\\log(T))$ rate from the $G^{\\beta}$-contribution. Are there any non-trivial examples where this isn't the case?\n2. Is there any hope of obtaining an adaptive rate in the growth function---perhaps by means of a tailored variance-reduced estimator---which could yield an instance-dependent result, if the true risk-variance is low? The estimation of $q$ and $r$ seems like it might be particularly troublesome if one tries to derive something for a general case, but this might be a misunderstanding on my part.\n\nLIMITATIONS:\nI have discussed limitations of the work above, and it seems that the authors have laid out the limitations of the degree of practicability due to the i.i.d. assumption (i.e., if training is coupled to the observed data stream). Nevertheless, I still think this is a solid contribution. With regard to the societal impact, the development of a robust theory of confidence in machine learning is undoubtedly a good thing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-29"}