Title: Real-Time Selection Under General Constraints via Predictive Inference
Venue: NeurIPS 2024
Accepted: yes
Keywords: Online multiple testing; Predictive inference; False selection rate; Individual and interactive constraints; Local false discovery rate.
Abstract: Real-time decision-making gets more attention in the big data era. Here, we consider the problem of sample selection in the online setting, where one encounters a possibly infinite sequence of individuals collected over time with covariate information available. The goal is to select samples of interest that are characterized by their unobserved responses until the user-specified stopping time. We derive a new decision rule that enables us to find more preferable samples that meet practical requirements by simultaneously controlling two types of general constraints: individual and interactive constraints, which include the widely utilized False Selection Rate (FSR), cost limitations, and diversity of selected samples. The key elements of our approach involve quantifying the uncertainty of response predictions via predictive inference and addressing individual and interactive constraints in a sequential manner. Theoretical and numerical results demonstrate the effectiveness of the proposed method in controlling both individual and interactive constraints.

Real-Time Selection Under General Constraints via Predictive Inference

Abstract

Real-time decision-making gets more attention in the big data era. Here, we consider the problem of sample selection in the online setting, where one encounters a possibly infinite sequence of individuals collected over time with covariate information available. The goal is to select samples of interest that are characterized by their unobserved responses until the user-specified stopping time. We derive a new decision rule that enables us to find more preferable samples that meet practical requirements by simultaneously controlling two types of general constraints: individual and interactive constraints, which include the widely utilized False Selection Rate (FSR), cost limitations, diversity of selected samples, etc. The key elements of our approach involve quantifying the uncertainty of response predictions via predictive inference and addressing individual and interactive constraints in a sequential manner. Theoretical and numerical results demonstrate the effectiveness of the proposed method in controlling both individual and interactive constraints.

1 Introduction

In recent times, the field of real-time decision has flourished significantly, primarily driven by the exponential growth of available data in both the tech industry and computer science. We consider here a typical application of real-time decision, the problem of online sample selection [2, 5]. For instance, online recruitment systems utilize machine learning algorithms to sequentially choose qualified candidates rather than waiting for all (future) candidates' information to be collected [14]. Additionally, recommendation systems have now become commonplace in providing real-time suggestions for content (e.g., news articles, short videos) with potential high click-through rates to users [24]. Common situations also can be found in real-time precision marketing [43].

We describe the online sample selection problem as follows: samples (individuals) characterized by covariates X t ∈ R d arrive sequentially while their responses Y t ∈ R remain unobserved throughout the process. The data pairs ( X , Y ) of each time are i.i.d. random vectors. At each time point, the analyst is faced with the task of deciding whether to select the current observation based on certain predetermined criteria related to Y t , and this selection process continues until a specific stopping rule is triggered. For example, Y t is the score that measures how one candidate fits a given job position in the recruitment system and the human resource agencies aim to prioritize candidates with higher Y t , such as Y t ≥ b . Or Y t is a binary variable where Y t = 1 / 0 means accepting or rejecting the offer, and the companies wish to find those individuals with Y t = 1 based on the X t .

Our Goal and Motivation. Our goal is to sequentially select samples whose unobserved responses Y t 's are in the specified target region. A natural idea is to make decisions based on the prediction value of Y t from models associated with ( X , Y ) built on some historical data. However, neglecting the uncertainty in predictions could result in numerous false decisions, i.e., selecting those samples whose true responses are beyond the specified region. To measure the selection uncertainty, some existing works reformulate sample selection as a hypothesis testing problem and focus on controlling the online false discovery rate (FDR) [16, 33]; see more discussions and literature in Section 1.2. However, in addition to quantifying statistical uncertainty, online selections often need to take into account various constraints to find informative samples in practice, for example, cost limitations, the impacts of some covariates or the diversity of candidates in the online recruitment [39]. Hence, it's necessary to explore the covariate space to satisfy these requirements. This motivates us to investigate how to efficiently implement online sample selection with statistical guarantees under various constraints.

To address this issue, we summarize common constraints into two types, i.e., individual constraints and interactive constraints . The former one is relevant to the cost of selected samples, and one typical example is the fundamental and crucial criterion, false selection rate (FSR), which quantifies the proportion of falsely selected samples and is equivalent to the well-adopted FDR. The latter constraint captures the interactive influence among selected samples and is regarded as some kind of quadratic constraint on some pairwise functions, such as the similarity or diversity among selected samples.

While the individual constraint associated with online FDR control has gained some attentions [16, 18], it alone fails to capture the nuanced pairwise relationships among different samples. To bridge this gap, we introduce interactive constraints, which are pivotal within our framework. Building upon individual constraints, the interactive criteria significantly expand the range of constraints our method can control. Integrating two distinct types of constraints into a unified framework makes it easier to create practical algorithms and ensures theoretical guarantees.

A motivating example: candidate screening. As an example, in recruitment, screening from the resumes arriving sequentially to determine viable candidates who can get into interview processes is an important problem in human resource management [13, 35]. In this case, one may be interested in: (1) controlling the online FSR to enhance resource efficiency [21], and (2) maintaining a desired level of candidate diversity during the screening process, thereby reducing bias [23, 44]. The individual and interactive constraints and the novel real-time sample selection procedure we propose can solve this problem precisely. See Section 4.2 for the details and more real data examples.

1.1 Our Contributions

In this paper, we design a novel and flexible online selection rule to effectively ensure the above two types of constraints are under control at pre-specified levels simultaneously, named as 'Individual and Interactive Constrained Online Selection" (II-COS). The main idea stems from an oracle model based on the local false discovery rate (lFDR) [12, 38], which is involved in offering valid evidence on whether Y t is our interest at each time point. With some appropriately chosen evaluating functions, the II-COS procedure entails validating whether the estimates of constraints are controlled. Simulated and real-data examples clearly demonstrate the superiority of the II-COS in terms of both online individual and interactive criteria control.

To the best of our knowledge, this is the first work to systematically bridge the predictive inference and online selection procedure with various constraints. Our contributions are summarized as follows:

Under a unified framework, the II-COS addresses how to implement online predictive selection sufficiently in consideration of both individual and interactive constraints. It is flexible to characterize the selective uncertainty and trade-off the sampling efficiency and practical limitations in the covariate space.

Under mild conditions, we establish the theoretical guarantee that the II-COS is able to control both individual and interactive constraints simultaneously and asymptotically under one given stopping rule.

The II-COS is model-agnostic in the sense that its implementation is applicable to any (appropriate) learning algorithms. Extensive numerical experiments indicate that the II-COS can significantly outperform existing ones while yielding effective constraints control.

1.2 Related Works

Our proposed method is built upon two fundamental pillars: (1) quantifying the uncertainty of response predictions using predictive inference; and (2) systematically addressing individual and interactive constraints in a sequential manner. Our work is intricately connected to the fields of predictive inference and online multiple testing. Here we briefly review literature on these two topics.

Predictive Inference. One key ingredient of our proposed method is predictive/conformal inference. Conformal inference [42, 34] provides a powerful and flexible tool to achieve algorithm-agnostic uncertainty quantification of predictions. Conventionally, conformal inference aims to build the prediction intervals and enjoys valid and distribution-free properties by leveraging data exchangeability [25, 3]. Taking a different but related perspective from multiple-testing, Bates et al. [7] pioneered a method to construct conformal p -values to detect outliers with finite-sample FDR control. Building upon this, recent advancements have improved detection power by involving more information or performing model selection [50, 26, 27, 51, 46]. The most related works are Jin and Candès [21] and Wu et al. [45], which considered a similar scenario that one would like to select some individuals of interest by controlling FDR or maximizing the diversity of selected samples in an off-line setting. Their methods are based on the conformal p -values or lFDR constructed with the predicted response values, respectively. Besides the fundamental difference between online and offline paradigms, our framework for characterizing various constraints poses additional challenges in how to select samples sequentially since we have multiple goals to achieve.

Online Multiple Testing. When only considering individual constraint as FSR control, the online sample selection can be reformulated as online multiple testing problem. Methods for online multiple testing have received much recent attention and were pioneered by Foster and Stine [16] who proposed the so-called α -investing strategy, which was later built upon and generalized [1, 29, 30, 18, 19]. The key idea in α -investing and its generalizations is to compare p -values with dynamic thresholds and gain some extra α -wealth for each rejection. We refer to Robertson et al. [33] for a thorough overview. Those rules suffer from the 'alpha-death" issue to some extent [29], which means a permanent end to decision-making when the decision threshold is too small, i.e., the online procedure stops early. This phenomenon occurs in many existing online multiple testing algorithms, as discussed in [29]. Along a different direction, Gang et al. [17] developed a new class of structure-adaptive sequential testing (SAST) rules built on the lFDR to avoid the alpha-death issue. The SAST serves as a building block for developing our II-COS procedure and can be essentially seen as a special case of ours. Later on, Ao et al. [4] reformulated online multiple testing procedure into an online knapsack problem, providing novel policies with near-optimal regret guarantees. Additionally, Xu and Ramdas [48] proposed to use e-values [41] for online multiple testing to address the dependence. However, those existing works do not take predictive inference into account and are concerned only with online error rate control without exploration of the covariate space, which may greatly hamper their applicability.

2 Individual and Interactive Constrained Online Selection Procedure

2.1 Problem Formulation

Assume there exists a historical labeled dataset as D = { ˜ X i , ˜ Y i } n i =1 , where ( ˜ X i , ˜ Y i ) 's are independent and identically distributed (i.i.d.) from ( X , Y ) . A sequence of unlabeled data X 1 , X 2 , · · · ∼ X = ( X 1 , · · · , X d ) ⊤ arrives in a stream with unknown responses Y 1 , Y 2 , · · · . At each time t , one must make a real-time decision about whether or not to select the t -th individual, which is determined by some pre-specified requirement on Y t . Denote A as the target region of Y t , which differs depending on users' specifications. For example, in a regression setting, the requirement could be of the form Y t ∈ [ a, b ] , ( -∞ , a ) or Y t ≥ b .

Let θ t = I { Y t ∈ A} describe the true state of Y t . Denote a decision rule as δ t ∈ { 0 , 1 } , where δ t = 1 indicates that the X t is selected and δ t = 0 otherwise. A false selection is made if δ t = 1 but θ t = 0 . Denote δ t = { δ i : i ≤ t } as the decision rule and T as the time that the procedure stops. Our goal is to build a decision rule δ t to select samples with { Y t ∈ A} up to stopping time T such that the following two general types of constraints hold simultaneously.

Individual Constraint. In practice, one main concern is to control the cost of selecting samples of interest. For example, in online recruitment, companies need to control the proportion of selected

unqualified candidates or the average loss when hiring someone who rejects the offer. In such cases, we can assign each selected sample a cost associated with some pre-specified function of the covariate X and control the expected cost associated with time T at the target level. We refer to this requirement as individual constraint and write it as:

̸

where a ∨ b = max { a, b } and G 0 ( X ) ≥ 0 and G 1 ( X ) ≥ 0 with G 0 = G 1 are the costs corresponding to θ = 0 and θ = 1 , respectively. Here, we take expectation due to the randomness of θ 1 , · · · , θ t .

For example, when we simply choose G 0 ( X ) = 1 and G 1 ( X ) = 0 , the individual constraint is the popular false selection rate (FSR), i.e.

The FSR is essentially equivalent to the well-adopted FDR in multiple testing literature, which is a useful tool to maintain the ability to reliably select samples of interest without excessively false selections [9]. Some works on online FDR control have been well studied. [16, 1].

The individual constraints alone cannot capture the pairwise relationship among different samples. We address this by introducing interactive constraints below.

Interactive Constraint Another common concern is the interactive constraint , which involves choosing more preferable samples. For example, companies would like to retain candidates with a diverse range of backgrounds and experiences in online recruitment, or real-time suggested contents are required to avoid homogeneity in recommendation systems. Here, we introduce a bi-variate weight function g ( X , X ′ ) to evaluate the interaction between selected samples. Denote PC( δ t ) = ∑∑ 1 ≤ i < j ≤ t g( X i , X j ) θ i θ j δ i δ j , PS( δ t ) = ∑∑ 1 ≤ i < j ≤ t θ i θ j δ i δ j . We define the interactive constraint as

Here, since only the correctly selected samples are of interest, the constraint is concerned with the average mutual effects between the correctly selected ones rather than all selected ones. When choosing the function g as some similarities, controlling ˜ C 2 ( δ t ) at a specified constant K , i.e., ˜ C 2 ( δ t ) ≤ K , is controlling the expected similarity (ES). It is equivalent to requiring that correctly selected samples exhibit certain diversity and rich information in the covariate space of interest.

̸

Typically, one useful choice for g ( X , X ′ ) is the weighted RBF kernel g ( X , X ′ ) = exp { -1 σ 2 ∑ d k =1 w k ( X k -X ′ k ) 2 } with parameter σ > 0 to measure the similarity between two independent X and X ′ . The RBF kernel is a common and widely embraced choice in machine learning [49, 28]. Here, { w 1 , . . . , w d } are some given weights per users' needs. For instance, if one is just interested in the effects of the k -th feature, then simply w k = 1 and w j = 0 for j = k . Specifically, the case that w k = 1 for all k = 1 , . . . , d is chosen in Section 4. We also consider other similarity choices of g ( X , X ′ ) , such as the cosine similarity g ( X , X ′ ) = X ⊤ X ′ / ( ∥ X ∥ 2 ∥ X ′ ∥ 2 ) [52].

Due to the randomness in the denominator, it turns out controlling (3) directly is not easy. Instead, we employ a modified interactive constraint,

The constraint (4) aims to control a ratio of expectations, which is still a reasonable interactive measure. In numerical studies, we see that ˜ C 2 ( δ t ) in (3) and C 2 ( δ t ) in (4) yield almost identical patterns. An illustrative example can be found in Appendix D.1.

In sum, the goal is to select samples of interest by a decision rule δ T controlling both the individual and interactive constraints until stopping time T , i.e., C 1 ( δ T ) ≤ α and C 2 ( δ T ) ≤ K. We emphasize that C 1 ( δ T ) and C 2 ( δ T ) as well as their pre-specified levels α and K can be chosen up to the practical applications.

2.2 Oracle Selection Procedure

To design a general rule that is valid for any arbitrary stopping time T , we consider controlling the constraints at each time t in an online fashion, such that sup t ∈ N C 1 ( δ t ) ≤ α and sup t ∈ N C 2 ( δ t ) ≤ K.

Since Y t is unavailable, we consider utilizing predictive inference to measure the suspicious patterns. Let µ ( x ) := Y | X = x be the regression or classification model associated with ( X t , Y t ) , and one reliable estimate as ̂ µ ( · ) , being estimated on the labeled data D with some machine learning algorithm. Denote W t = ̂ µ ( X t ) as a predicted value of Y t and assume that ̂ µ ( · ) is a bijection almost surely. The bijection assumption is considerably mild and widely adopted for the identyification of each X t in the predictive inference framework [45]. The θ t = I ( Y t ∈ A ) is Bernoulli( π ) distributed with π = Pr( Y t ∈ A ) , and W t can be viewed as generated from one two-group model

where f 0 and f 1 denote the probability distribution functions of W t conditional on Y t / ∈ A (i.e., θ t = 0 ) and Y t ∈ A , respectively. Then, the conditional probability of Y t / ∈ A is

where f = (1 -π ) f 0 + πf 1 . The L t coincides with the local FDR in multiple testing literature [12, 17]. With the two-group model (5), we have E [ θ t | X t ] = 1 -L t and further notice that the individual constraint C 1 ( δ t ) in (1) can be exactly satisfied if

holds. Here, we denote V t = ∑ i ≤ t { L i G 0 ( X i ) + (1 -L i ) G 1 ( X i ) δ i } and the number of selected ones as R t = ∑ i ≤ t δ i ∨ 1 for notational convenience. Especially, when G 0 ( X ) = 1 , G 1 ( X ) = 0 , then FSR( δ t ) in (2) can be exactly controlled.

Accordingly, the interactive constraint C 2 ( δ t ) ≤ K in (4) can be achieved if

where the expected total mutual effects conditional on { X i } i ≤ t and the expected number are denoted as TS t and NS t , respectively.

Therefore, if L t is known, when a new sample X t arrives at time point t , we can perform the decision rule as follows. Note that there is no need to consider interactive effects before the first selection. When t comes before the first selection (i.e, R t -1 = 0 ), the decision rule is δ t = 1 if

holds; otherwise, δ t = 0 which means X t is not selected. When X t arrives with R t -1 ≥ 1 , then δ t = 1 if (6) and

hold simultaneously; otherwise, δ t = 0 . Note that if we set C 1 ( δ t ) as FSR( δ t ) and choose K = + ∞ , then our method essentially reduces to the same manner as the controlling step of the SAST in Gang et al. [17]. Our proposed method can be seen as a much more generalized and flexible framework for controlling both the individual and the interactive constraints simultaneously in an online fashion.

We call this method the oracle II-COS (Individual and Interactive Constrained Online Selection). The workflow in Figure 1 shows the procedure of the oracle II-COS. The following result shows that it can exactly achieve our goal.

Proposition 2.1. Assume L t values are known. Then the oracle II-COS selection rule controls both constraints at any stopping time T , i.e., C 1 ( δ T ) ≤ α and C 2 ( δ T ) ≤ K.

Oracle II-COS

𝑋𝑡

Input:

𝑊𝑡, 𝐿 𝑡

,

𝑔(𝑋𝑖 , 𝑋

𝑡

)

𝐶2(𝑡)

Yes

𝛿𝑡

= 0

Yes

𝛿𝑡

= 1

Update:

𝑉𝑡

, 𝑅

𝑡

,

𝑇𝑆𝑡 , 𝑁𝑆

𝑡

No

No

𝑋𝑡

2.3 Data-driven II-COS Procedure

As L t is unknown in practice, we propose a data-driven II-COS procedure, which uses a reliable estimation ̂ L t for implementation. We resort to a data-splitting strategy: randomly split historical data D into two parts, the training set D tr and the calibration one D cal of sizes n 0 and n 1 respectively, where D tr is used for training a predictive model and D cal is to estimate those unknown parameters. Specifically, we first fit a regression or classification model ̂ µ ( · ) on D tr , and then obtain predicted values on D cal , i.e. { ̂ µ ( ˜ X i ) : ( ˜ X i , ˜ Y i ) ∈ D cal } . Note that conditional on D tr , { ̂ µ ( ˜ X i ) : ( ˜ X i , ˜ Y i ) ∈ D cal } are i.i.d. random variables and have the same distribution as W t = ̂ µ ( X t ) , so that it can be utilized to estimate (5).

Therefore, the estimators of f 0 and f , ̂ f 0 and ̂ f , can be obtained by applying the kernel density estimation method to the data { ̂ µ ( ˜ X i ) : ( ˜ X i , ˜ Y i ) ∈ D cal , ˜ Y i / ∈ A} and { ̂ µ ( ˜ X i ) : ( ˜ X i , ˜ Y i ) ∈ D cal } , respectively. And the probability π = Pr( Y t ∈ A ) can be approximated by ̂ π = n -1 1 ∑ ( ˜ X i , ˜ Y i ) ∈D cal I ( ˜ Y i ∈ A ) . Further the L t in (5) can be estimated by

The data-driven II-COS procedure is summarized in Algorithm 1, and it indeed consists of two phases: offline estimation and online decision. The running time of offline estimation is not critical. At each time t , the computational complexity is a linear function of the currently selected number R t . More implementation details can be found in Section 4 and Appendix B.1.

In fact, the proposed II-COS is flexible to trade off the individual and interactive constraints by adjusting the thresholds α and K . If one is concerned only with individual cost control, then we can set K = + ∞ , with which the interactive constraint is out of work. Similarly, only the interactive effect is of interest when α = 1 . Appendix D.2 provides a toy example to illustrate this.

Before further pursuing, we would discuss the stopping time in practice. It's worth noting that the specific choice of stopping rule (and thus stopping time) is completely up to the user. For example, when m ≥ 2 is the desired number of selections, one can set T = inf t { t : ∑ t i =1 δ i = m } . Or when s is the total wages for recruitment, one can set T = inf t { t : ∑ t i =1 δ i s i = s } where s i is the payroll for each selected candidate. Or T is just chosen as one given deadline. With the use of II-COS, practitioners have the flexibility to design diverse stopping strategies that can adapt seamlessly to their specific applications. In brief, our method is flexible and is appropriate for various goals based on the user's requirements by choosing different G 0 ( X ) , G 1 ( X ) , g ( X i , X j ) and varied target levels ( α, K ) and a user-specified stopping time.

Algorithm 1 The data-driven II-COS procedure

Input: Target levels α and K , pairwise function g , cost G 0 ( X ) and G 1 ( X ) , stopping time T , interested region A , labeled data D , prediction algorithm H .

Initialization: t = 0 , V t = R t = 0 , TS t = NS t = 0 ; Decision rule δ t = ∅

Estimation: Randomly split D into training set D tr and calibration set D cal . On D tr , fit ̂ µ ( x ) with H . Obtain ̂ π , ̂ f 0 and ̂ f from D cal .

Online decisions: while t ≤ T do

Set t = t +1 . Compute W t = ̂ µ ( X t ) , ̂ L t by (8) and the similarities g ( X i , X t ) for those δ i = 1 .

end

Output: Selection set { X i : δ i = 1 , δ i ∈ δ T } .

Extension to varying proportion case. In practice, the distribution of ( X t , Y t ) may vary smoothly over time. In Appendix C, we consider the probability of Y t ∈ A (i.e. the proportion of samples in the specified region) varying over time and extend the proposed II-COS to learn π t = Pr( Y t ∈ A ) continuously over time and we also construct the corresponding theoretical guarantees.

3 Statistical Performance Guarantees

In this section, we provide statistical guarantees for the data-driven II-COS procedure. The main difficulties lie in the quantification of data-driven estimation error of L t and we utilize the classical kernel density estimation theory along with the structure of our online procedure to effectively characterize it. For simplicity, we consider that the training data set is given such that the estimated model ̂ µ is fixed. Before presenting our theoretical results, we state the following regularity conditions.

Assumption 3.1 (Density functions and kernel) . The density functions and kernel function satisfy

The f 1 ( · ) and f 0 ( · ) are upper bounded by M > 0 , and the f ( · ) is lower bounded by ℓ > 0 .

The f 0 and f 1 are Hölder-continuous, i.e. | f 0 ( w ) -f 0 ( w ′ ) | ≤ c β | w -w ′ | β for any w,w ′ ∈ R , and the same for f 1 with some fixed 0 < β ≤ 1 and constant c β .

Kernel K ( · ) is a bounded symmetric function and enjoys exponential decay.

̸

Assumption 3.2 (Weight functions) . There exists constants c G > 0 and c g > 0 such that 0 < G 0 ( X ) ≤ c G , 0 < G 1 ( X ) ≤ c G for any X and 0 < g ( X , X ′ ) ≤ c g for any X = X ′ .

Assumption 3.1 is considerably mild and widely adopted in the uniform convergence of kernel density estimation [36]. If f 0 and f 1 have bounded first-order derivatives, the Hölder-continuous assumption would hold with β = 1 . The lower bound of f is to ensure the uniform convergence of the estimated lFDR . Assumption 3.2 is mild since the weight functions are required only to be positive and bounded. It can be satisfied by a large category of G 0 , G 1 and g . For example, we can take G j ( X ) = a j ∥ X ∥ 2 2 for j = 0 or 1 and c G exists when X is bounded. And we can set g as the RBF and orthogonal similarities with c g = 1 .

With those regularity conditions, we establish the validity of the II-COS procedure for the individual constraint control.

Theorem 3.3 (Bound for individual constraint) . Suppose Assumptions 3.1 and 3.2 hold and take the bandwidths for estimating f and f 0 in the order of n -1 / (2 β +1) 1 . Then for any given time t , the individual constraint of the II-COS procedure (Algorithm 1) satisfies C 1 ( δ t ) ≤ α + ∆ n 1 , where ∆ n 1 = Dn -β 2 β +1 1 √ log n 1 and D is a constant depending on M , ℓ , c β , β , π , c G and K ( · ) .

Although the C 1 ( · ) of II-COS might be slightly larger than the target level in finite samples, this gap converges to 0 asymptotically as n 1 increases. In the numerical studies, we find that a small calibration size of around 200 is enough to control the C 1 in a reasonable range. Taking the FSR as an individual constraint, Theorem 3.3 indicates that our method can provide asymptotic online FSR control similar to an online FDR control procedure [16].

The next theorem examines the performance of the II-COS in terms of interactive constraint.

Theorem 3.4 (Bound for interactive constraint) . Suppose Assumptions 3.1-3.2 hold and take the bandwidths for estimating f and f 0 in the order of n -1 / (2 β +1) 1 . Let T s = inf { t : ∑ t i =1 δ i = s } for s > 2 and assume there exists a constant α ′ ∈ (0 , 1) such that ∑ i ≤ t ̂ L i δ i / (1 ∨ R t ) ≤ α ′ . Then for any given time t ≥ T m , the interactive constraint of the II-COS satisfies

The term mα ′ / ( m -1) is used to characterize the lower bound of the denominator term of the interactive constraint. Specifically, when we choose FSR as the individual constraint, we have α ′ = α , which demonstrates the interdependence between controlling individual and interactive constraints. Furthermore, under arbitrary stopping strategies with a stopping time T , we can have the asymptotic guarantee.

Corollary 3.5. Suppose the conditions in Theorem 3.4 hold, the stopping moment T ≥ T m and α ′ < (1 -1 /m ) / 2 . Then the II-COS procedure controls the individual and interactive constraints asymptotically at T , i.e. lim n 1 →∞ C 1 ( δ T ) ≤ α and lim n 1 →∞ C 2 ( δ T ) ≤ K.

4 Experiments and Evaluation

We illustrate the breadth of applicability of the II-COS procedure by experiments on simulated data and real-data applications. As an example, we set the stopping rule as selecting total m = 100 samples, i.e., T = T m = inf t { t : ∑ t i =1 δ i = m } . Additional experiments including the extended IICOS in Appendix C are shown in Appendix D.9. Code for implementing II-COS and reproducing the experiments and figures in our paper is available at https://github.com/lulin2023/II-COS .

Implementation of II-COS . To our best knowledge, online selection with uncertainty qualification has only been studied in the field of online multiple testing, which aims to control online FDR. Hence, we focus on using FSR as the individual criterion and modified ES as the interactive criterion. As X may be measured on scales with widely differing ranges in different dimensions, we assume that X 's have been properly scaled in each dimension before computing g . We choose g as the weighted RBF kernel with σ = 1 , w k = 1 here. Other choices for individual and interactive constraints are considered in Appendix D.5.

Benchmarks . We compare the II-COS procedure with four benchmarks from online multiple testing. The first one is a structure-adaptive sequential testing rule, the SAST [17], which is implemented with ̂ L t . It can achieve the FSR control but ignore the interactive constraint. As mentioned earlier in Section 2, SAST can also be considered as a special case of our II-COS with K = + ∞ . Its details are deferred to Appendix B.2. The other competitors are three well-known online FDR control algorithms LOND [18], SAFFRON [30] and ADDIS [40] implemented with the conformal p -values suggested by Bates et al. [7]. Refer more information in Appendix B.3. All the benchmarks can only control FDR, which demonstrates the flexibility of our method for different constraints.

Performance Measures . The empirical FSR, ES and stopping time ( T m ) are evaluated using the average values of the false selection proportions, the similarity and the stopping time from 500 replications, respectively, where T m serves as a criterion for assessing selection efficiency.

4.1 Results on Synthetic Data

Data Description. We consider a classification model: X | Y = 0 ∼ N 4 ( µ 1 , I 4 ) , and X | Y = 1 ∼ N 4 ( µ 2 , I 4 ) , where µ 1 = (5 , 0 , 0 , 0) ⊤ , µ 2 = (0 , 0 , -3 , -2) ⊤ . We set Pr( Y = 1) = 0 . 2 . The information set is A = { 1 } . The predictor H is taken as random forest with defaulted parameters. We also consider a regression setting and conduct additional experiments in Appendix D.

Firstly, we observe that methods relying on conformal p -values, such as LOND, SAFFRON, and ADDIS, encounter the alpha-death (stop early) issue [29]. These methods struggle to select an adequate number of samples, especially in small calibration sets. In contrast, II-COS ensures the control of both individual and interactive constraints even with a small n cal (e.g., 200). See more details and results in Appendix D.3. Hence, to make a fair comparison, we consider a relatively large size of the calibration set, n cal = 4 , 000 . We fix training data size n tr = 1 , 000 .

0.00

0.05

0.10

0.15

0.20

100

200

300

Time

C 1

(

FSR

)

0.025

0.050

0.075

100

200

300

Time

C

~

2

(

ES

)

Method

II-COS

SAST

LOND

SAFFRON

ADDIS

0.0

0.1

0.2

0.3

II-COS

SAST

LOND

SAFFRON

ADDIS

C 1

(

FSR

)

0.00

0.05

0.10

0.15

II-COS

SAST

LOND

SAFFRON

ADDIS

C

~

2

(

ES

)

0

250

500

750

1000

II-COS

SAST

LOND

SAFFRON

ADDIS

Tm

Results. Figure 2 presents the online FSR (false selection rate) and ES (expected similarity) values of five methods against time t , which are individual and interactive constraints, respectively. The FSR levels of II-COS and SAST are closer to the nominal level than the other three methods. As expected, only the empirical ES levels of II-COS are controlled under the pre-specified level K over time t . The LOND and SAFFRON lead to slightly conservative FSR values, while the FSR levels of ADDIS are inflated compared to the target level. Figure 3 further displays the boxplots of empirical FSR, ES at stopping time T m . We observe that only II-COS achieves satisfactory ES values compared to the nominal level. Moreover, the II-COS has a relatively larger value of T m compared to those of other benchmarks. This is consistent with the fact that the II-COS spends more time exploring the structure information inside the covariate space due to the requirement of interactive constraint. Similar conclusions for the regression example in Appendix D.4 can be drawn.

Regarding efficiency, we also conducted an experiment to compare the effectiveness of II-COS with an oracle method possessing knowledge of true state θ t . The T m of II-COS is very close to the oracle. This close proximity indicates the high efficiency of II-COS. See Appendix D.8 for the details.

4.2 Results on Real Data

We next demonstrate the performance of the II-COS in two real-world applications. Since those online multiple testing methods based on conformal p -values yield few selected individuals, we focus

on comparing the II-COS with the SAST. For comparison, we also include the offline method using conformal p -values, where δ t = 1 if ̂ p t ≤ α . We denote it as CP.

Datasets. We consider the recruitment dataset from Kaggle [22] that contains 45,372 candidates after removing the missing data and records a binary response indicating whether the candidate passes the job interview, and other 11 attributes including education status, handicapped or not, and gender. The other problem is to use 1994 Census Bureau dataset [8] to select a subset of individuals who may have high incomes in precision marketing. This census dataset records 32,561 individuals with their 14 attributes, including gender, race, marriage, education length and so on.

For each dataset, we randomly partition the data into three parts: n tr = 1 , 000 training data, n cal = 1 , 000 calibration data and the rest which are used as the online observations. The categorical attributes are converted into one-hot codes and then are treated as numerical attributes for computing similarity measures. The prediction algorithm H is random forest with defaulted parameters.

Results. Table 1 reports the results among 500 repetitions. Both the II-COS and SAST enjoy valid FSR control, but CP yields an inflated FSR level in income investigation. The II-COS performs well in terms of similarities. To further compare the diversities, we present the proportions of different education status in in Figure 4. It can be seen that the proposed II-COS demonstrates its superior diversity in the specific attributes. See Appendix D.6 for more results for the real data. In summary, the proposed II-COS works well for selecting individuals of interest to achieve various constraints in practical applications.

(a) Candidate dataset [22]

(b) Income dataset [8]

0.2

0.21 0.21

0.49

0.44

0.48

0.02

0.07

0.02

0.3

0.26

0.3

0.01

0.03

0.01

0.0

0.1

0.2

0.3

0.4

0.5

No Qual

High SchoolMatriculationBachelors

Master

Education Status

Propotion

Method

II-COS

SAST

CP

0.35

0.45

0.0

0.1

0.2

0.3

0.4

0.5

4

8

12

16

Education Length

Propotion

Method

II-COS

SAST

5 Concluding Remarks

Broader Impacts. This work focuses on creating reliable machine learning tools for making real-time decisions. One key achievement is a new algorithm called II-COS, designed to select informative samples in real-time while meeting two types of general constraints. II-COS allows for both individual and interactive control, validated through theoretical analysis and numerical tests. Our method is model-agnostic and easily applicable to many real-world cases such as producing diversified results while controlling FSR for online recruitment. One potential negative impact of our work is that researchers will apply the algorithm without sufficient scrutiny. We emphasize that it's important to use caution when applying this method to complex real-world scenarios to prevent misuse.

Limitations. Firstly, we mainly consider binary functions as the interactive constraint. How to adapt the II-COS to other popular constraints, such as the Gini index, deserves further study. Secondly, in certain practical scenarios, it is possible to obtain feedback after decisions. Incorporating the feedback information into our method to enhance its performance warrants future research.

Acknowledgments and Disclosure of Funding

We thank anonymous area chair and reviewers for their helpful comments. Zou was supported by the National Key R&D Program of China (Grant Nos. 2022YFA1003703, 2022YFA1003800), the National Natural Science Foundation of China (Grant Nos. 11925106, 12231011, 11931001, 12226007, 12326325). Ren was supported by the National Natural Science Foundation of China (Grant Nos. 12101398, 12471262), and Young Elite Scientists Sponsorship Program by CAST.

References

Ehud Aharoni and Saharon Rosset. Generalized α -investing: definitions, optimality results and application to public databases. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 76(4):771-794, 2014.

Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems , 32, 2019.

Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends® in Machine Learning , 16(4):494-591, 2023.

Ruicheng Ao, Hongyu Chen, David Simchi-Levi, and Feng Zhu. Online local false discovery rate control: A resource allocation approach. arXiv preprint arXiv:2402.11425 , 2024.

Eric Bach, Shuchi Chawla, and Seeun Umboh. Threshold rules for online sample selection. Discrete Mathematics, Algorithms and Applications , 2(4):625-642, 2010.

Peter L. Bartlett. Learning with a slowly changing distribution. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory , page 243-252, 1992.

Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. Annals of Statistics , 51(1):149-178, 2023.

Barry Becker and Ronny Kohavi. Adult income investigation. UCI Machine Learning Repository https://archive.ics.uci.edu/dataset/2/adult , 1996.

Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 57(1):289-300, 1995.

Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics , 29(4):1165-1188, 2001.

Rick Durrett. Probability: theory and examples , volume 49. Cambridge University Press, 2019.

Bradley Efron, Robert Tibshirani, John D Storey, and Virginia Tusher. Empirical bayes analysis of a microarray experiment. Journal of the American Statistical Association , 96(456):11511160, 2001.

Evanthia Faliagka, Kostas Ramantas, Athanasios Tsakalidis, and Giannis Tzimas. Application of machine learning algorithms to an online recruitment system. In Proc. International Conference on Internet and Web Applications and Services , pages 215-220, 2012.

Evanthia Faliagka, Lazaros Iliadis, Ioannis Karydis, Maria Rigou, Spyros Sioutas, Athanasios Tsakalidis, and Giannis Tzimas. On-line consistent ranking on e-recruitment: seeking the truth behind a well-formed cv. Artificial Intelligence Review , 42(3):515-528, 2014.

Kai-Tai Fang, Runze Li, and Agus Sudjianto. Design and modeling for computer experiments . Chapman and Hall/CRC, 2005.

Dean P Foster and Robert A Stine. α -investing: a procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 70(2):429-444, 2008.

Bowen Gang, Wenguang Sun, and Weinan Wang. Structure-adaptive sequential testing for online false discovery rate control. Journal of the American Statistical Association , 118(541): 732-745, 2023.

Adel Javanmard and Andrea Montanari. On online control of false discovery rate. arXiv preprint arXiv:1502.06197 , 2015.

Adel Javanmard and Andrea Montanari. Online rules for control of false discovery rate and false discovery exceedance. The Annals of Statistics , 46(2):526-554, 2018.

Heinrich Jiang. Uniform convergence rates for kernel density estimation. In International Conference on Machine Learning , pages 1694-1703. PMLR, 2017.

Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. Journal of Machine Learning Research , 24(244):1-41, 2023.

Kaggle. Candidate selection dataset. https://www.kaggle.com/datasets/ tarunchilkur/client , 2020.

G Kanagavalli, R Seethalakshmi, and T Sowdamini. A systematic review of literature on recruitment and selection process. Humanities & Social Sciences Reviews , 7(2):01-09, 2019.

Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. News recommender systems-survey and roads ahead. Information Processing & Management , 54(6):1203-1227, 2018.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for out-ofdistribution testing with labelled outliers. Journal of the Royal Statistical Society Series B: Statistical Methodology , 86(3):671-693, 01 2024.

Ariane Marandon, Lihua Lei, David Mary, and Etienne Roquain. Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics , 52(1):157-183, 2024.

Houwen Peng, Bing Li, Haibin Ling, Weiming Hu, Weihua Xiong, and Stephen J. Maybank. Salient object detection via structured matrix decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 39(4):818-832, 2017.

Aaditya Ramdas, Fanny Yang, Martin J Wainwright, and Michael I Jordan. Online control of the false discovery rate with decaying memory. Advances in Neural Information Processing Systems , 30:5655-5664, 2017.

Aaditya Ramdas, Tijana Zrnic, Martin Wainwright, and Michael Jordan. Saffron: an adaptive algorithm for online control of the false discovery rate. In International Conference on Machine Learning , pages 4286-4294. PMLR, 2018.

Bradley Rava, Wenguang Sun, Gareth M James, and Xin Tong. A burden shared is a burden halved: A fairness-adjusted approach to classification. arXiv preprint arXiv:2110.05720 , 2021.

David S Robertson, Jan Wildenhain, Adel Javanmard, and Natasha A Karp. onlineFDR: an R package to control the false discovery rate for growing data repositories. Bioinformatics , 35 (20):4196-4199, 2019.

David S Robertson, James MS Wason, and Aaditya Ramdas. Online multiple hypothesis testing. Statistical Science , 38(4):557, 2023.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3):371-421, 2008.

Muhammad Ahmad Shehu and Faisal Saeed. An adaptive personnel selection model for recruitment using domain-driven data mining. Journal of Theoretical and Applied Information Technology , 91(1):117, 2016.

Bernard W. Silverman. Weak and strong uniform consistency of the kernel estimate of a density and its derivatives. The Annals of Statistics , 6(1):177-184, 1978.

John D Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 64(3):479-498, 2002.

Wenguang Sun and T. Tony Cai. Oracle and adaptive compound decision rules for false discovery rate control. Journal of the American Statistical Association , 102(479):901-912, 2007.

Stephen Taylor. Resourcing and talent management . Kogan Page Publishers, 2018.

Jinjin Tian and Aaditya Ramdas. Addis: an adaptive discarding algorithm for online fdr control with conservative nulls. Advances in Neural Information Processing Systems , 32:9388-9396, 2019.

Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The Annals of Statistics , 49(3):1736-1754, 2021.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . New York: Springer, 2005.

Michel Wedel and PK Kannan. Marketing analytics for data-rich environments. Journal of Marketing , 80(6):97-121, 2016.

Sang Eun Woo, James M LeBreton, Melissa G Keith, and Louis Tay. Bias, fairness, and validity in graduate-school admissions: A psychometric perspective. Perspectives on Psychological Science , 18(1):3-31, 2023.

Xiaoyang Wu, Yuyang Huo, Haojie Ren, and Changliang Zou. Optimal subsampling via predictive inference. Journal of the American Statistical Association , pages 1-29, 2023.

Xiaoyang Wu, Lin Lu, Zhaojun Wang, and Changliang Zou. Conditional testing based on localized conformal p-values. arXiv preprint arXiv:2409.16829 , 2024.

Pengtao Xie, Yuntian Deng, and Eric Xing. Latent variable modeling with diversity-inducing mutual angular regularization. arXiv preprint arXiv:1512.07336 , 2015.

Ziyu Xu and Aaditya Ramdas. Online multiple testing with e-values. In International Conference on Artificial Intelligence and Statistics , pages 3997-4005. PMLR, 2024.

Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision , 113(2):113-127, 2015.

Yifan Zhang, Haiyan Jiang, Haojie Ren, Changliang Zou, and Dejing Dou. Automs: automatic model selection for novelty detection with error rate control. Advances in Neural Information Processing Systems , 35:19917-19929, 2022.

Zinan Zhao and Wenguang Sun. False discovery rate control for structured multiple testing: Asymmetric rules and conformal q-values. Journal of the American Statistical Association , pages 1-24, 2024.

Ping Zhong, Zhiqiang Gong, Shutao Li, and Carola-Bibiane Schönlieb. Learning to diversify deep belief networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing , 55(6):3516-3530, 2017.

Supplementary Material for 'Real-Time Selection Under General Constraints via Predictive Inference'

This supplementary material contains:

Preliminary terms for self-containment (Appendix A);

Implementation details (Appendix B);

An extension algorithm to varying proportion case (Appendix C);

Additional experiments (Appendix D);

The proofs of all the theoretical results. (Appendix E).

A Preliminary Terms for Self-Containment

Here, we list the preliminary terms we use in the paper for the sake of clarity and self-containment.

FDR [9], false discovery rate, a widely-adopted error rate notion in the field of multiple testing, is defined as the expected proportion of incorrectly rejected null hypotheses as follows:

where H 0 is the unknown set of true null hypotheses, R ( t ) represents the set of rejected null hypotheses until time t and then H 0 ∩ R ( t ) is the set of false discoveries.

FSR , false selection rate, defined as the expected proportion of individuals being not of interest among the selected subset of individuals. It is in fact equivalent to the definition of FDR. In our framework in this paper, we describe it equivalently as:

B Implementation Details of Algorithms

𝒟

𝒟𝑡𝑟

𝒟𝑐𝑎𝑙

ℋ

Ƹ 𝜇(𝑋)

ො

𝜋, ෡

𝑓 0

, ෡

𝑓 1

Offline Estimation

Online Decision

𝑋𝑡

Compute:

𝐿𝑡 ,

𝑔(𝑋𝑖 , 𝑋

𝑡

)

𝐶2(𝑡)

Yes

𝛿𝑡

= 0

Yes

𝛿𝑡

= 1

Update:

𝑉𝑡

, 𝑅

𝑡

,

𝑇𝑆𝑡 , 𝑁𝑆

𝑡

No

No

𝑋𝑡

B.1 Implementation Details of II-COS

Figure 5 shows the implementation flowchart of our proposed data-driven II-COS procedure, including the offline estimation step and online decision step. Here we introduce more implementation details to reproduce the results.

About the classifier. For the classification problem, we use the predicted conditional probability ̂ Pr( Y t = 1 | X t = x ) as ̂ µ ( x ) in the procedure of II-COS. Many commonly used algorithms such as random forest and neural networks can provide such probability estimators. In fact, the choice of ̂ µ is not restricted to probability. For example, the support vector machine outputs the distance to the separating hyperplane for classification, and such distance can be chosen as ̂ µ .

However, for probability estimators, one potential problem is that when the classifier is quite accurate, most of W i 's are concentrated at 0 or 1 and few of them take values between (0 , 1) . In such a situation, it is difficult to accurately and stably estimate lFDR L i 's since the density functions f 0 ( w ) and f ( w ) are not smooth enough and are not lower bounded. This yields that our assumptions are violated, but the II-COS can still perform satisfactorily with some corrections on the estimation ̂ L i 's. Notice that the larger W i is, the more likely Y i ∈ A and hence the smaller L i is. That is ̂ L i should be monotonically decreasing as W i increases. Observing this, we can make a monotonization correction on ̂ L i 's. To be specific, we rearrange ̂ L i by the decreasing order of W i . If ̂ L ( W ( i -1) ) < ̂ L ( W ( i ) ) which violates the monotonicity, we revise it by ̂ L ( W ( i -1) ) = ̂ L ( W ( i ) ) , where W ( i ) is the i th smallest among W . This monotonization correction enables us to avoid obvious errors due to unstable estimation and improve the performances of all the methods utilizing ̂ L i 's.

Choice of K . A useful interactive constraint needs an appropriate specification of K . For any two i.i.d. observations X and X ′ with corresponding θ and θ ′ respectively, the expected C 2 of the individuals of interest is given as C 2 := E [ g ( X , X ′ ) | θ = 1 , θ ′ = 1] , which can be estimated by ̂ C 2 = ∑∑ i<j ; i,j ∈L g ( ˜ X i , ˜ X j ) / {|L| ( |L| 1) } , where L = { i : ˜ Y i ∈ A} . It is reasonable to set K = a ̂ C 2 , where a > 0 is user-specific to control the interactive constraint level. Our numerical evidence reveals that a ∈ (0 . 1 , 0 . 5) works generally well.

B.2 Implementation Details of SAST

Gang et al. [17] proposed a structure-adaptive sequential testing (SAST) rule for online false discovery rate control. In their work, the rejecting rule is as follows: If L t < γ t and {|R t -1 | + 1 } -1 ( ∑ i ∈R t -1 L i + L t ) ≤ α , then δ t = 1 . Otherwise δ t = 0 , where R t -1 = { i ≤ t -1 : δ i = 1 } and γ t is a barrier estimated from an "offline" procedure.

The implementation details of SAST for comparisons in our simulations in Section 4 are different from the original one. Firstly, the original SAST assumes the null density function f 0 is already known while in our setting f 0 remains unknown. Secondly, in our predictive inference setting, the density functions and the null proportion are directly estimated via calibration set as the offline estimation procedure in Algorithm 1, not from current rejection sets. Besides, considering the time-varying structures of the data stream in their setting, Gang et al. [17] incorporated the barrier strategy in their method, which is not necessary to be adopted here.

B.3 Implementation Details of Conformal p -values

The notion of conformal p -values was originally proposed by Vovk et al. [42] to construct prediction interval. Recently, there exist some works to apply conformal p -values to implement sample selection from a multiple-testing perspective, such as Bates et al. [7], Rava et al. [31] and Jin and Candès [21]. In the sample selection problem, the hypothesis has the following form for each t ,

There are two types of conformal p -values and we adopt the one in Bates et al. [7] which utilizes the same class calibration. Recall that for W t , its conformal p -value p t is defined as

The nonconformity score function Q ( W t ) is used to indicate the possibility of θ t = 0 . For example, in regression settings, if A = [ b, + ∞ ) , we can use Q ( W j ) = b -W j . If A = ( -∞ , a ] ∪ [ b, + ∞ ) , then we can choose Q ( W t ) = max { W t -a, b -W t } . And in binary classification settings, if A = 1 and W t indicates the probability of Y t = 1 , we set Q ( W t ) = 1 -W t .

Even though conformal p -values are correlated, using conformal p -values to conduct multiple testing can control the FDR level with finite sample guarantee since they are positive regression dependent on a subset (PRDS) [10]. However, the conformal p -values are lower bounded by 1 / ( |D cal | +1) , which leads to unsatisfactory performance for online multiple testing methods based on p -values. Since these methods require sufficiently small p -values to make rejections.

In our simulations, we implement LOND, SAFFRON and ADDIS for online sample selection by R package OnlineFDR [32] with α = 0 . 1 . Other parameters are defaulted. Here we introduce the details about these online FDR control methods. Ramdas et al. [29] proposed a 'statistical perspective" to control FDR in online setting, which is to keep an estimate of the FDP less than α similar to the offline setting. Specifically, for offline FDR, let the rejection set R ( s ) = { i | p i ≤ s } . An oracle estimate for FDP is given by FDP ∗ ( s ) := |H 0 |· s |R ( s ) |∨ 1 . For online FDR, an oracle estimate of

FDP ∗ ( t ) is ∑ j ≤ t,j ∈H 0 α j R ( t ) ∨ 1 . Table 2 lists a comparison of estimating FDP in classical offline methods and online methods for FDR control in multiple testing. For the online methods, denote the decision rule as δ t = { p t ≤ α t } , where p t is the corresponding conformal p -value at time t for our problem. The test levels { α t } for LOND [18], SAFFRON [30] and ADDIS [40] are listed as follows:

LOND: α t = γ t ( R ( t -1)+1) , where { γ t } ∞ t =1 is a given infinite non-increasing sequence of positive constants that sums to α and R ( n ) = ∑ n t =1 R t denotes the number of discoveries in the first n hypotheses tested.

SAFFRON: At each time t , define C j + = C j + ( t ) = ∑ t -1 i = τ j +1 , where C t = I { p t ≤ λ } . For t = 1 , α 1 = min { γ 1 W 0 , λ } ; For t = 2 , 3 , . . . , α t := min { λ, ˜ α t } , where

ADDIS: The testing levels for ADDIS are given by α t = min { λ, ˆ α t } , where

B.4 Experiments Compute Resources

All the experiments were conducted on 3.11 GHz Intel Gen i5-11300H processors with 16 Gb memory at a Lenovo personal computer and the R platform with version 4.2.1. The time of execution for each of the individual experimental runs is about 6.686 seconds. And the total compute time for the synthetic classification example in Section 4 for 500 replications is about 63.877 minutes.

B.5 A Toy Example for Illustration in Section 2.3

We illustrate the idea of the II-COS procedure via a binary classification example. We aim to select m = 50 data points of a specific class from unlabeled data arriving sequentially. We choose FSR as the individual constraint and mES as the interactive constraint.

The data is generated as follows. The 4-dimensional covariates X = ( X 1 , X 2 , X 3 , X 4 ) ⊤ are generated from a mixture of multivariate normal distributions with mean (0 , 0 , 0 , 0) if Y = 0 and mean ( -3 , -3 , 0 , 0) if Y = 1 . The proportions of Y = 0 and Y = 1 are 80% and 20% respectively. In this illustrative example, we set the historical data size n = 1 , 000 with calibration size n 1 = 500 and would like to select m = 50 data points from the interest region A = { 1 } with the target FSR level α = 0 . 1 . The mES threshold K is set at 0 . 01 for II-COS.

Figure 6 depicts the scatterplot of the first two dimensions of the covariates X , with green dots and red triangles denoting correctly selected points and falsely selected ones, respectively. The SAST method proposed by Gang et al. [17] is taken as one benchmark, which considers only the FSR control. We observe that the selected points of II-COS enjoy significant diversity among the covariate space and only a few false selections are contained. In contrast, the SAST is inclined to choose similar samples concentrated at the center of the concerned group and stops too early to fully explore the covariate space with sequentially arriving samples.

Correctly-selected

Falsely-selected

-5.0

-2.5

0.0

-6

-4

-2

0

X 1

X 2

II-COS

-5.0

-2.5

0.0

-6

-4

-2

0

X 1

X 2

SAST

C Extension to Varying Proportion Case

In practice, the distribution of ( X t , Y t ) may vary smoothly over time. Due to the unknown Y t , it is unrealistic to re-estimate parameters (i.e., Estimation Step in Algorithm 1) on both labeled data and the most recent data. To mitigate this problem, we consider the probability of Y t ∈ A (i.e. the proportion of samples in the specified region) varying over time and extend the proposed II-COS to learn π t = Pr( Y t ∈ A ) continuously over time.

Let Q ( W t ) be one score function, which is high when the possibility of θ t = 0 is large and otherwise low. For example, in regression settings, if A = [ a, + ∞ ) , we can take Q ( W t ) = a -W t . And in binary classification, if A = 1 and W t indicates the probability of Y t = 1 , we can set Q ( W t ) = 1 -W t . One valid conformal p -value for Q ( W t ) can be obtained as Bates et al. [7],

Inspired by the techniques for null proportion estimation in multiple testing literature [37], we note that Pr( ̂ p t ≥ λ ) ≈ Pr( ̂ p t ≥ λ, θ t = 0) ≈ π t (1 -λ ) for large λ ∈ (0 , 1) (i.e., λ = 0 . 5 ).

Thus, we consider using some recent ̂ p t to estimate a reliable π t . Denote q as the size of a neighborhood { t -q, . . . , t -1 } and fix λ . We employ an exponential weighted scheme to estimate π t where

the more recent samples will contribute more to the estimation:

where κ b ( s ) = exp {-| s | /b } and b is the bandwidth parameter. Then, we can compute the distribution of W t as ̂ f t = ̂ f 0 ( w )(1 -̂ π t ) + ̂ f 1 ( w ) ̂ π t and estimate L t by ̂ L λ t = (1 -̂ π λ t ) ̂ f 0 ( W t ) ̂ f t ( W t ) ∧ 1 .

The extended II-COS is to substitute ̂ L λ t for ̂ L t in Algorithm 1. The following theorem establishes the guarantee by assuming the slow change of distribution [6].

Theorem C.1. Assume π t satisfies | π t +1 -π t | ≤ η for any t , and the bandwidth parameter b satisfies b ζ ≤ q for some ζ > 1 . Denote ∆ ′ = c 1 n -β/ (2 β +1) 1 √ log n 1 + c 2 max { b -1 / 3 , (log n 1 /n 1 ) 1 / 6 , ( bη ) 2 / 3 } with constants c 1 , c 2 . Suppose p t and λ satisfy Pr( p t > λ | θ t = 1) = 0 . Under Assumption 3.1-3.2, we have:

(a) For any given time t , the individual constraint of extended II-COS satisfies C 1 ( δ t ) ≤ α +∆ ′ ;

(b) Furthermore, if the conditions in Theorem 3.4 hold, then for any given time t > T m , the interactive constraint of the extended II-COS satisfies C 2 ( δ t ) ≤ K + ( K + c g )∆ ′ 0 . 5 -mα ′ m -1 -∆ ′ .

Besides the part similar to ∆ n 1 in Theorem 3.3, the bound ∆ ′ owns an additional term that can be decomposed into three parts. The b -1 / 3 indicates how many valid samples we use to estimate π t . The second part comes from the approximation error of p -values. The last one characterizes the effects of distribution shift. If we properly choose b such that bη = o (1) , this term is negligible. Thus when b and n 1 both tend to infinity, ∆ ′ converges to 0 and the individual and interactive constraints will be controlled asymptotically.

D Additional Experiments

D.1 Illustration of the Similar Patterns between ˜ C 2 and C 2

We calculate the empirical ˜ C 2 and C 2 during the online sample selection procedure under regression setting in Section D.4 with II-COS from 500 replications. The results are summarized in Figure 7.

0.008

0.010

0.012

0.014

200

400

600

Time

Value

D.2 An Illustration Example of the Flexibility for Choices of α and K .

There exists some trade-off of stopping time and two criteria. In fact, II-COS could result in a short stopping time when only one criterion is considered. Typically, one could choose K = + ∞ for the case the interactive constraint is out of work and only individual criterion control is considered, and meanwhile one can set α = 1 with which the interactive constraint is the only concern. The results in Table 3 evaluated this conclusion where C 1 is the FSR and ˜ C 2 is the ES.

D.3 Effects of the Calibration Size

We perform some additional simulations under the classification setting, to study the stopping early issue for those methods based on the conformal p -values.

Take α = 0 . 1 and K = 0 . 045 for FSR and mES, respectively. In Table 4, we first fix n tr = 1 , 000 and m = 100 and vary n cal from 500 to 2,500 to compare the average numbers of selected samples of these three methods with II-COS until stopping. It is clear that all the three benchmarks are unable to select enough samples across all the settings, especially with a small calibration set. As the calibration size n 1 increases, their selected numbers tend to be close to the target. This can be understood because a larger calibration size would generally yield more accurate detection of the individual of interest and thus alleviate the alpha-death issue to some extent. In contrast, the performances of II-COS, in terms of the number of selected samples until stopping, would be much less influenced by the size of calibration data. The II-COS only stops when m samples are obtained under all the scenarios.

Furthermore, to verify that II-COS can guarantee both FSR( δ T m ) and ES( δ T m ) control with a relatively small calibration size n cal (such as 200), we apply II-COS in synthetic data. The details of the data generation process can be found in Section 4 and D.4.

We fix n tr = 1000 , m = 100 and n cal varies from 200 to 800. The average of FSR( δ T m ) and ES( δ T m ) with II-COS for both scenarios are calculated from 500 replications. The results are summarized as Table 5. It's obvious that for different n cal, both FSR( δ T m ) and ES( δ T m ) are controlled under the pre-specified constant α and K respectively under all scenarios with our II-COS method.

D.4 Results on Synthetic Data for Regression Setting

The following regression setting is considered: Y = -7 X 2 1 +5exp X 2 +10( X 3 + X 4 ) 2 + ε, with X ∼ N 4 ( 0 , I 4 ) and ε ∼ N (0 , 1) . The informative set is A = ( c, ∞ ) , where c is the 80% quantile of Y . The FSR and mES are considered as C 1 and C 2 , respectively. The prediction algorithm H is taken as neural network, with a single hidden layer and 10 hidden neurons, implemented by R package nnet , and K is chosen as 0 . 015 .

The simulation results are summarized in Figure 8 and 9 from 500 replications. The results are similar to those for classification setting. It's further verified that our proposed II-COS outperforms all the benchmarks under both classification and regression scenarios.

0.00

0.05

0.10

0.15

0.20

100

200

300

Time

C 1

(

FSR

)

0.01

0.02

0.03

0.04

0.05

100

200

300

Time

C

~

2

(

ES

)

Method

II-COS

SAST

LOND

SAFFRON

ADDIS

0.00

0.05

0.10

0.15

0.20

II-COS

SAST

LOND

SAFFRON

ADDIS

C1

(

FSR

)

0.00

0.03

0.06

0.09

0.12

II-COS

SAST

LOND

SAFFRON

ADDIS

C

~

2

(

ES

)

0

500

1000

1500

II-COS

SAST

LOND

SAFFRON

ADDIS

Tm

D.5 Experiments for Other Individual and Interactive Constrains.

Other individual constraints To better illustrate the performance of our proposed method, we also conduct an experiment for the general individual constraint. We choose G 0 ( X ) = | ∑ d i =1 X i | , G 1 ( X ) = | ∑ d i =1 X i | / 2 for EC (expected cost), and choose G 0 ( X ) = 1 , G 1 ( X ) = 0 for FSR. Other settings are the same as the classification model. The results are shown in Figure 10. We can see that only the proposed II-COD can guarantee all EC (expected cost), FSR and ES control, while all the benchmarks are out of control for EC or ES.

Other pairwise function g . Besides the RBF kernel, popular choices include the cosine similarity [47, 52] with adjustment g ( X , X ′ ) = X ⊤ X ′ / ( ∥ X ∥ 2 ∥ X ′ ∥ 2 ) + 1 , and the absolute value of cosine similarity, i.e., g ( X , X ′ ) = | X ⊤ X ′ | / ( ∥ X ∥ 2 ∥ X ′ ∥ 2 ) which characterizes the orthogonality between

0.0

0.5

1.0

1.5

II-COS

SAST

LOND

SAFFRON

ADDIS

C 1

(

EC

)

0.0

0.2

0.4

II-COS

SAST

LOND

SAFFRON

ADDIS

C 1

(

FSR

)

0.00

0.05

0.10

0.15

II-COS

SAST

LOND

SAFFRON

ADDIS

C

~

2

(

ES

)

X and X ′ and is often considered in the field of design of experiments [15]. The similarity functions mentioned above all satisfy Assumption 3.2. As a supplement to the experiments in the main text, we choose the cosine similarity with adjustment g ( X , X ′ ) = X ⊤ X ′ / ( ∥ X ∥ 2 ∥ X ′ ∥ 2 )+1 for classification setting. Notice that the original cosine similarity X ⊤ X ′ / ( ∥ X ∥ 2 ∥ X ′ ∥ 2 ) can be negative sometimes. Hence we add a constant 1 which guarantees g ( X , X ′ ) ≥ 0 and does not change the final results. The simulation results are summarized in Figure 11 from 500 replications. As can be seen, the simulation results are similar to those when choosing the RBF kernel in Section 4.1.

0.0

0.1

0.2

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

FSR

0.95

1.00

1.05

1.10

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

ES

0

200

400

600

800

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

Tm

D.6 Additional Results for Real Data Analysis

In Table 6, the average proportions of handicapped and female in the selected candidates are in the last two columns of (a). The average proportions of minority and female among the correctly selected individuals are in the last two columns of (b).

D.7 Additional Experiments for Different Learning Algorithms

Except for the random forest algorithm, we further apply two different learning algorithms H for classification setting to estimate the model ˆ µ ( X ) : Support vector machine (SVM) and Neural network (NN) with a single hidden layer and 10 hidden neurons, which are implemented by R packages kernlab and nnet , respectively. For NN algorithm, the entropy fitting is used for classification setting. The empirical FSR( T m ) and ES( T m ) levels are estimated by the average of the false selection proportion and the expected similarity respectively from 500 replications. The results are summarized in Figure 12. As can be seen, the simulation results are similar as those when applying random forest in Section 4.1.

(a) Candidate dataset [22]

(b) Income dataset [8]

NN

SVM

II-COS

SAST

LOND

SAFFRON

ADDIS

II-COS

SAST

LOND

SAFFRON

ADDIS

0.00

0.10

0.20

0.30

0.40

0.50

Method

FSR

NN

SVM

II-COS

SAST

LOND

SAFFRON

ADDIS

II-COS

SAST

LOND

SAFFRON

ADDIS

0.00

0.03

0.06

0.09

Method

ES

D.8 Experiments for the comparison with oracle procedure

Regarding efficiency, we conducted an experiment to compare the effectiveness of II-COS with an oracle method possessing knowledge of true state θ t . This can be succinctly formulated as follows. At time t , δ t = 1 if θ t = 1 and PC( δ T m ) / PS( δ T m ) ≤ K ,the process halts when ∑ i ≤ t δ i = m . To ensure a fair comparison, we fix α = 0 . 01 for II-COS and choose K = 0 . 045 for both methods. The results are shown in Table 7. The FSR of the oracle procedure is 0 as expected. The T m of II-COS is very close to the oracle. This close proximity indicates the high efficiency of II-COS. Both FSR and ES are effectively controlled by II-COS.

D.9 Experiments for the Extended II-COS under Varying Proportion Case

Let N be a prespecified large integer denoting the number of samples arriving sequentially. Denote n as the size of available labeled history dataset D . We fix N = 5000 , n = 5000 . Consider three different varying patterns for π t :

Table 7: FSR( δ T m ) , ES( δ T m ) ( × 10 -2 ) and T m of the II-COS and the oracle procedure under classification model.

Linear pattern: Vary π t linearly from 0 to 0.5, π t = 0 . 5 t/N .

Sine pattern: π t = { sin(8 πt/N ) + 1 } / 4 , π t ranges between 0 and 0.5.

The other settings are the same as those in classification setting. For the historical dataset, we fix π t = 0 . 2 .

In simulation we generate 500 data points prior to t = 1 to form an initial proportion estimate. The varying proportion estimates are updated every 200 time points. The bandwidth parameter b for estimating π λ t are chosen based on normal reference rule. We set b = 38 . The window size q are chosen by a rule of thumb in practice. Let q = Cb , where C is a fixed constant between 10 and 20. In our simulations, we fix q = 500 and λ = 0 . 5 . As for the density estimation, we first estimate f 0 and f 1 on the calibration data, then compute ̂ f t = ̂ f 0 ( w )(1 -̂ π t ) + ̂ f 1 ( w ) ̂ π t . Thus accordingly, we estimate L t by ̂ L λ t = (1 -̂ π λ t ) ̂ f 0 ( W t ) ̂ f t ( W t ) ∧ 1 . As for those methods based on conformal p -values, they do not need modification since they only require the exchangeability of data in the null hypothesis. The varying proportion case does not violate such a condition.

The simulation results are shown in Figures 13-15, from which we can observe that the extended version of II-COS performs better than the benchmarks under all three different varying settings for both C 1 (FSR) and C 2 (ES) control.

0.00

0.05

0.10

0.15

0.20

0.25

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

FSR

0.00

0.05

0.10

0.15

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

ES

0

1000

2000

3000

4000

5000

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

Tm

0.000

0.025

0.050

0.075

0.100

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

FSR

0.00

0.05

0.10

0.15

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

ES

0

1000

2000

3000

4000

5000

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

Tm

0.00

0.05

0.10

0.15

0.20

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

FSR

0.00

0.05

0.10

0.15

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

ES

0

1000

2000

3000

4000

5000

II-COS

SAST

LOND

SAFFRON

ADDIS

Method

Tm

E Proofs

E.1 Auxiliary Lemmas

The following results include a standard uniform bound for kernel density estimator [36, 20] and a simple corollary from the central limit theorem [11].

Lemma E.1. If Assumption 3.1 hold and we take the bandwidth h = n -1 / (2 β +1) 1 , then with probability at least 1 -1 /n 1

where D 1 = D 1 ( M,c β , β, K ) is a positive constant depending on M and β, c β of Hölder continuity and the kernel K ( · ) .

Lemma E.2. The estimation ̂ π satisfies

with probability 1 -n -2 γ 1 for any constant 0 < γ < 1 / 2 .

Next lemma characterizes the uniform convergence of ̂ L ( w ) .

Lemma E.3 (Uniform convergence of ̂ L ( w ) ) . Suppose Assumption 3.1 holds. Taking the bandwidth of kernel density estimator as h = n -1 / (2 β +1) 1 , then we have

for some positive constant D 2 = D 2 ( M,D 1 , ℓ, π ) with probability 1 -2 /n 1 -1 /n 1 / 3 1 .

Proof .

Note that ̂ L ( w ) = (1 -̂ π ) ̂ f 0 ( w ) ̂ f ( w ) ∧ 1 and

where D ∗ 1 = D ∗ 1 ( D 1 , π ) is a positive constant depending on D 1 and π because ̂ f 0 is estimated by about n 1 (1 -π ) samples. The ( i ) follows the results directly in Lemma E.1 and Lemma E.2 and the bounds of f ( w ) and f 0 ( w ) in Assumption 3.1-(1) with probability 1 -2 /n 1 -1 /n 2 γ 1 . The ( ii ) holds since

for sufficiently large n 1 . Due to the fact β ≤ 1 and taking γ = 1 / 6 in Lemma E.2, the ( iii ) holds with D 2 = 2 { D ∗ 1 + M √ π (1 -π ) + D 1 } /ℓ . Hence we verify the conclusion. □

The next lemma shows the property of data-driven II-COS procedure which is useful to give the bound of the mES gap.

Lemma E.4. Suppose δ t is the decision result at time t ≥ T s by the data-driven II-COS procedure with ∑ i ≤ t ̂ L i δ i / 1 ∨ ∑ i ≤ t δ i ≤ α ′ , where T s = inf { t : ∑ t i =1 δ i = s } and s ≥ 2 . Then we have

Proof . It suffices to show

By the selection procedure, we have

Squaring both sides of (9) and making some decomposition, we have

Notice that

We have and

□

Lemma E.6. If b ζ ≤ q for some ζ > 1 and b is sufficiently large, the exponential weight κ b ( s ) = exp {-| s | /b } satisfies

where C b > 0 is constant determined by b .

Proof . Take Z b = exp {-1 /b } . Notice that

□

The next two lemmas will be used in the analysis of the extended II-COS for varying proportion case.

Lemma E.5. Assume that random variables Y 1 , Y 2 , have density functions bounded by a constant c > 0 , and other two random variables Y 3 , and Y 4 satisfy P ( | Y 1 -Y 3 | ≤ ε ) ≥ 1 -ζ and P ( | Y 2 -Y 4 | ≤ ϵ ) ≥ 1 -ς , where ε > 0 , ϵ > 0 , ζ > 0 and ς > 0 . Then for all t > 0 ,

Proof . Define the events E 1 = {| Y 1 -Y 3 | ≤ ε } , E 2 = {| Y 2 -Y 4 | ≤ ϵ } , and E = E 1 ⋃ E 2 . Note that

and we have

the last inequality holds because the density function is bounded.

Thus for a constant C b > 0 . we have

where ( i ) holds since b is sufficiently large such that Z q b = exp {-q/b } ≤ exp {-b ζ -1 } can be eliminated and Z b < 1 . The last inequality ( ii ) holds by exp { x } ≥ x +1 .

By the same discussion, we have

The last inequality holds since exp { x } ≤ 1 / (1 -x ) for x < 1 . For simplicity, we use the same notation C b to denote the constants. □

E.2 Proof of Proposition 2.1

Proof . For the part of individual constraint control, note that L i is defined as

Define W ∗ = σ ( W 1 , · · · ) . The stopping time T is measurable respect to W ∗ . The individual constraint at time T satisfies

The ( i ) holds since X i 's are independent of each other. By construction of the selection rule, we have that at the stopping time T , (( R T ∨ 1) -1 ∑ i ≤ T { L i W 0 ( X i ) + (1 -L i ) W 1 ( X i ) } δ i ≤ α . It follows that C 1 ( δ T ) ≤ α at a random time T . By construction of the selection rule, we have for stopping time T ,

Taking expectations on both sides of inequality (10) and by double expectation theorem, we finally obtain

It follows that for every time t ≥ T 2 ,

E.3 Proof of Theorem 3.3

Proof . In the data-driven II-COS procedure, δ i is determined by the estimated lFDR ̂ L i and we have

Note that

It suffices to bound the absolute value of the second term. We have

The last inequality follows from Lemma E.3 and max w | ̂ L ( w ) -L ( w ) | ≤ 1 . Notice that n -1 3 1 ≤ n -β 2 β +1 1 by β ≤ 1 . We have where D = 2 c G D 2 +3 .

E.4 Proof of Theorem 3.4

Proof . Notice that

The ( i ) holds by Assumption 3.2 and L ( w ) , ̂ L ( w ) ≤ 1 even when t is random, as the uniform convergence of ̂ L ( w ) . And ( ii ) follows from Lemma E.3. By the similar arguments we have

Denote ∆ n 1 = Dn -β 2 β +1 1 √ log n 1 . Combining (11) and (12), it follows that for every time t ≥ T s ,

The ( i ) follows from (11) , and ( ii ) comes from the operation of our algorithm, where

The ( iii ) and ( v ) are directly from Lemma E.4. The ( iv ) holds due to (12). Thus we have

E.5 Proof of Corollary 3.5

Proof . Define the C 1 constraint conditional on W as

By the proofs of Theorem 3.3, we have for any t

Hence at stopping time T m , we have

By the proof of Theorem 3.4, we can also drop off the expectation and replace ∆ n 1 with sup w ∈R | ̂ L ( w ) -L ( w ) | . That is

Hence at stopping time t = T m , (13) still holds. Take expectations at both sides and we will get

Notice that ∆ n 1 = Dn -β 2 β +1 1 √ log n 1 converges to 0 as n 1 → ∞ and α ′ < (1 -1 /m ) / 2 by the condition. It follows that

E.6 Proof of Theorem C.1

Proof . We first denote some notations. For notational simplicity, we use c to denote constants in the following proofs.

Recall the conformal score function Q ( X t ) . The conformal p -value is

where n 2 = |L| .

Denote the population p -value as p t = F ( Q ( W t )) , where F ( · ) is the distribution of Q ( W t ) conditional on θ t = 0 . Denote π t = Pr( θ t = 1) and

The corresponding local FDR is defined as

It follows the definition of C λ 1 ( δ t ) and C λ 2 ( δ t ) .

and

The estimated proportion is defined as

And denote the ideal estimated proportion via population p-values as

where κ b ( s ) = exp {-| s | /b } and q is the size of a neighborhood for estimation before time t . We first claim the following proposition and the proof is deferred in Section E.7.

Proposition E.7. Suppose the assumptions in Theorem 3 hold. Then

with probability 1 -max { 1 /b, √ log n 1 /n 1 , ( bη ) 2 } 2 γ , where c > 0 is a constant and 0 < γ < 1 / 2 .

Notice that

The last inequality holds since ̂ π t ∈ [0 , 1] and f 0 ( w ) and f 1 ( w ) are upper bounded by M .

Thus by Lemma E.1 and take γ in Proposition E.7 at 1 / 6 , with probability 1 -c max { 1 /b, √ log n 1 /n 1 , ( bη ) 2 } 1 / 3 we have

By the procedure of Lemma E.3 directly, with probability 1 -c max { 1 /b, √ log n 1 /n 1 , ( bη ) 2 } 1 / 3 we have

It follows that

Denote ∆ ′ n 1 = 2 cc G n -β 2 β +1 1 √ log n 1 + c max { b -1 3 , ( log n 1 n 1 ) 1 6 , ( bη ) 2 3 } . By our assumption, bη = o (1) . So

By the additional assumption that Pr( p t > λ | θ t = 1) = 0 , we have π λ t = π t . Thus Hence the first part of Theorem 3 is completed.

At last, we can use the same procedure to prove that

Since π λ t = π t , C λ 2 ( δ t ) = C 2 ( δ t ) and the results directly follow.

E.7 Proof of Proposition E.7

Proof. Notice that

We discuss for the two parts respectively. For the first term of the above inequality, it suffices to show the upper bound of the second moment of | ̂ π λ t -˜ π λ t | .

For convenience, denote U j = I ( p j > λ ) , and V j = I ( ̂ p j > λ ) . It follows

̸

Now we check the upper bound of | p j -̂ p j | . We can rewrite them as p j = F ( Q ( W j )) and ̂ p j = ̂ F n 2 ( Q ( W j )) . Even though n 2 is a random variable, due to the two-group model, ̂ F n 2 ( · ) is still an empirical distribution function composed by i.i.d. samples conditional on { ˜ θ i } i ∈C where C is the index set of calibration set. Thus by DKW inequality, for any ε 1 > 0

The last inequality holds since n 2 ≥ (1 -π ) n 1 / 2 with probability 1 -4 π/ { n 1 (1 -π ) } .

Thus by Lemma E.5 and the fact that p j has a bounded density function, we obtain

Take ε 1 = √ log n 1 / { n 1 (1 -π ) } , for sufficient large n 1 such that √ n 1 log n 1 ≥ 6 + 6 π we have

Again by Lemma E.5 and the same ε 1 , for any i, j ∈ { t -q, . . . , t -1 } := N q ( t ) , we have

Therefore by Lemma E.6, we have

So by definition and Markov' inequality, we have

For the second part, we have

The ( i ) holds since Var { I ( p j > λ ) } ≤ 1 / 4 . As for ( ii ), consider A λ = { w : F ( Q ( w )) > λ } . Then by the two-group model we have Pr( p t > λ ) = ∫ A λ { f 0 ( w )(1 -π t ) + f 1 ( w ) π t } dw. Hence

by our assumptions. And ( iii ) is directly from the second part of Lemma E.6.

Above all, we finally conclude that for all ε > 0 and sufficiently large n 1 , we have

for some constant c > 0 .

Thus take ε = max { 1 /b, √ log n 1 /n 1 / 2 1 , ( bη ) 2 } 1 / 2 -γ √ c and we obtain

with probability 1 -max { 1 /b, √ log n 1 /n 1 , ( bη ) 2 } 2 γ , where 0 < γ < 1 / 2 . Hence we complete the proof.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See the end of Sect. 1, we summarize our main contributions. The experimental results in Sect. 4 and Appendix D validate the theoretical results in Sect. 3 and Appendix C. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Sect. 5, we discuss the limitations of our II-COS algorithm. In Sect. 3, we have explained why our assumptions are weak piece by piece. And see Subsect. 2.2 for the details about our computational complexity.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For the theory assumptions, see Assumption 3.1-Assumption 3.2 in Sect. 3. For the proofs of each theoretical result, see Appendix E in the Supplementary material.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appendix B for the implementation details to reproduce our experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the data and code in the supplemental materials, including sufficient instructions in the zip file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Sect.4 and Appendix B, we provide the experimental setting/details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We calculate the standard error across 500 replications and show the error bars in experimental results, which is calculated by mean + / -se.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix B.4 for the compute resources and more details about the execution time.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the authors have reviewed the NeurIPS Code of Ethics guidelines and ensured that our paper conforms to them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impacts of our work in Section 5.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have cited and provided the URL of the dataset we use in the paper, see Sect. 4 and the references.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper proposes a method for online sample selection. The authors introduce the concepts of _individual_ and _interactive constraints_, and demonstrate theoretically and empirically that their method satisfies both.

STRENGTHS:
The problem seems important and the formulation and approach novel. The authors provide both theoretical guarantees and empirical evidence, in both synthetic and real-world applications, of the effectiveness of their approach. The mathematical formulation seems sound, and the assumptions and theoretical results are clearly stated.

WEAKNESSES:
I am not familiar with the FDR control literature, and had to read parts of the paper (specifically sections 2.2, 2.3 and 4) multiple times to get a gist for the logic of the method and its empirical perfomance. This is reflected in my confidence score. If the paper is accepted, I highly recommend the authors revise the paper to make it easier to follow. A flowchart to illustrate the steps of the algorithm and/or to illustrate the differences between the Oracle and Data-driven selection procedures may be helpful; a toy example could also help. I also suggest including a longer description and/or table of the benchmark methods against which the empirical performance of II-COS was compared.

QUESTIONS:
- Interactive constraint: Why is this defined only with respect to the correctly selected samples? In the case of, e.g., the diversity of selected samples, is the constraint not intended to represent the diversity of all selections?
- Line 159: Should $R_t = \left( \sum_{i \leq t} \delta_i \right) \lor 1$?

LIMITATIONS:
The authors discuss limitations of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-06

comments: SUMMARY:
In this paper, the authors quantify the uncertainty of response predictions using predictive inference; and systematically addressing individual and interactive constraints in a sequential manner.  An online selection rule is developed to ensure the above two types of constraints are under control at pre-specified levels simultaneously.  Simulated and real-data examples  are used to evaluate and illustrate their approach in terms of both online individual and interactive criteria control.

STRENGTHS:
This is a nicely, clearly written paper that develops an online selection rule that is simple yet effective.  The simplicity of the online selection rule will enhance the potential for  this rule to be used in real life.   The authors’ claims are well supported both via theory, simulations and application to real data. The paper along with the appendix provides detailed information that allow for replicability.  Very nice!  I particularly appreciate the comparison with the approaches based on conformal p-values.

WEAKNESSES:
See questions below.

QUESTIONS:
Section 4.1 would be good to tell reader how many replications are used.

LIMITATIONS:
Comments are needed on the practicality of \hat\mu being a bijection.  How critical is this assumption to the method and the theoretical analysis.  If critical then this is a limitation.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies online sample selection with individual and interaction \nconstraints simultaneously. Specifically, the goal is to control (variants) of \nthe false selection rate (FDR) and the expected similarity (ES) under the \nempirical bayes framework. Under distributional conditions, the proposed method \ncontrols the target quantities asymptotically. The method is evaluated on synthetic \nand real data.\n\nSTRENGTHS:\n1. The paper is well-presented and easy to follow.\n2. The problem under consideration is of interest and relevant.\n\nWEAKNESSES:\nPlease see my comments in the questions section.\n\nQUESTIONS:\n1. **The model.** In the motivating examples such as candidate selection, it appears to me \nthat we get to observe the ground truth, i.e., $Y_t$, after time step $t$.\nThis is briefly mentioned in the discussion section, but I think it is reasonable \nto use the observed $Y_t$'s to update the estimation.\n\n2. **The choice of interaction constraints.** It is not well-motivated why \nthe changing ES to (4) is reasonable. As illustrated in the simulation, \nalthough these two quantities seem to coincide as the time step goes \nto infinity, they do differ quite a lot with smaller time steps (this could happen \nwhen $m$ is small).\n\n3. What is the principle for choosing K in general? In particular, for the real data example, \nwhy is K taken to be $1\\times 10^{-3}$ in the fist example and $6\\times 10^{-3}$ in the second?\n\n4. In the numerical experiments, a fairer comparison with offline CP would be with [1], i.e., \nthresholding the p-values with BH-adjusted p-values as opposed to the fixed threshold. \n\n[1] Jin, Ying, and Emmanuel J. Candès. \"Selection by prediction with conformal p-values.\" Journal of Machine Learning Research 24.244 (2023): 1-41.\n\nLIMITATIONS:\nThe authors have partically addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework to perform online sample selection such that the unseen outcomes are in specific target range while also optimizing for constraints like diversity that are dependent on the input covariates. The additional constraint involving input covariates can help ensure properties like the diversity of candidates when selecting individuals for interviews while also guaranteeing that most of the interviewed individuals accept the offer. The paper proposes a data-driven procedure to select the subset of candidates in an online fashion by implementing the proposed algorithm. Under reasonably weak assumptions, the paper provides theoretical guarantees on satisfying both the above constraints in online sample selection. The experiments confirm that this framework ensures low false selection rates (i.e. unseen outcomes are in a specific target range) while optimizing for the additional covariate-dependent constraints like diversity on synthetic and real data.\n\nSTRENGTHS:\nThe paper proposes an intuitive way to incorporate covariate-dependent constraints like the diversity of candidates when performing online sample selection to optimize for metrics like false selection rates. This paper solves an important problem in online sample selection and demonstrates that the proposed method improves the covariate-dependent objective while maintaining comparable performance on false selection rates.\n\nWEAKNESSES:\nIt would be interesting to understand the gaps between an ideal diversity profile and the profile obtained by the proposed method in Fig 3. Analysing the gap w.r.t changing g(X_i, X_j) function choice could be helpful. Would it be helpful to increase the weight of the g(X_i, X_j) term to reduce this gap and understand its implications on the satisfaction of individual constraints?\n\nIt is evident that the SAST baseline outperforms the proposed method in terms of FSR sometimes, which is understandable given there; 's a tradeoff with the interactive constraints (Table 2b, Fig 1).  It would be helpful to learn if we can reduce the gap between SAST and the proposed method by balancing the tradeoff (perhaps using a tunable hyperparameter that balances the two constraints?).\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes, limitations and broader impacts are discussed in the last section of paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}