Real-Time Selection Under General Constraints via Predictive Inference
Authors: Yuyang Huo
Lin Lu
Haojie Ren
Changliang Zou
Venue: NeurIPS 2024 poster (2024)
Keywords: Online multiple testing; Predictive inference; False selection rate; Individual and interactive constraints; Local false discovery rate.
Primary area: other
TL;DR: We address online sample selection, introducing II-COS, a decision rule that efficiently identifies preferable samples meeting practical requirements by managing individual and interactive constraints.

Abstract: Real-time decision-making gets more attention in the big data era. Here, we consider the problem of sample selection in the online setting, where one encounters a possibly infinite sequence of individuals collected over time with covariate information available. The goal is to select samples of interest that are characterized by their unobserved responses until the user-specified stopping time. We derive a new decision rule that enables us to find more preferable samples that meet practical requirements by simultaneously controlling two types of general constraints: individual and interactive constraints, which include the widely utilized False Selection Rate (FSR), cost limitations, and diversity of selected samples. The key elements of our approach involve quantifying the uncertainty of response predictions via predictive inference and addressing individual and interactive constraints in a sequential manner. Theoretical and numerical results demonstrate the effectiveness of the proposed method in controlling both individual and interactive constraints.

Real-Time Selection Under General Constraints via Predictive Inference
Yuyang Huo Lin Lu Haojie Ren
Changliang Zou
Schooo of Statistics and Data Sciences, LPMC, KLMDASR and LEBPS,
Nankai University, Tianji, China
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, China huoyynk@gmail com, 1inlu1020990gmai1. com haojierendsjtuu edu. en, moucl@nankai... edu.
Abstract
Real-time decision-making gets more attention in the big data era. Here, we con- sider the problem of sample selection in the online setting, where one encounters possibly infinite sequence of individuals collected over time with covariate infor- mation available. The goal is to select samples of interest that are characterized by their unobserved responses until the user-specified stopping time. We derive a new decision rule that enables to find more preferable samples that meet practical requirements by simultaneously controlling two types of general constraints: indi- vidual and interactive constraints, which include the widely utilized False Selection Rate (FSR), cost limitations, diversity of selected samples, etc. The key elements of our approach involve quantifying the uncertainty of response predictions via predictive inference and addressing individual and interactive constraints in a se- quential manner. Theoretical and numerical results demonstrate the effectiveness of the proposed method in controlling both individual and interactive constraints.
1 Introduction
In recent times, the field of real-time decision has flourished significantly, primarily driven by the exponential growth of available data in both the tech industry and computer science. We consider here typical application of real-time decision, the problem of online sample selection 5). For instance, online recruitment systems utilize machine learning algorithms to sequentially choose qualified candidates rather than waiting for all (future) candidates' information to be collected [14. Additionally, recommendation systems have now become commonplace in providing real-time suggestions for content (e.g.. news articles, short videos) with potential high click-through rates to Common situations also can be found in real-time precision marketing 43
users 24
We describe the online sample selection problem follows: samples (individuals) characterized by covariates X, E I arrive sequentially while their responses Y, E R remain unobserved throughout the process. The data pairs (X, Y) of each time are i.i.d. random vectors. At each time point, the analyst faced with the task of deciding whether to select the current observation based on certain predetermined criteria related to Y,, and this selection process continues until specific stopping rule triggered. For example, Y, is the score that measures how one candidate fits a given job position in the recruitment system and the human resource agencies aim prioritize candidates with higher Y;, such as Y, 2 b. Or Y, is a binary variable where Y, 1/0 means accepting or rejecting the offer, and the companies wish find those individuals with Y, based the X,-
Equal contribution, and the first authors are listed in alphabetical order. "Correspondence to: Haojie Ren <haojieren @ sjtu.edu.cn>, Changliang 7ou <zouel @ @na.kai.cdu..cn>>>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Our Goal and Motivation. Our goal is to sequentially select samples whose unobserved responses Y,'s are in the specified target region. A natural idea is make decisions based on the prediction value of Y from models associated with (X,Y) Y) built on some historical data. However, neglecting the uncertainty in predictions could result in numerous false decisions, i.e., selecting those samples whose true responses are beyond the specified region. To measure the selection uncertainty, some existing works reformulate sample selection as a hypothesis testing problem and focus on controlling the online false discovery rate (FDR) [16 33 see more discussions and literature in Section 1.2 However, in addition to quantifying statistical uncertainty, online selections often need to take into account various constraints find informative samples in practice, for example, cost limitations, the impacts of some covariates or the diversity of candidates in the online recruitment [39]. Hence, it's necessary to explore the covariate space to satisfy these requirements. This motivates us to investigate how to efficiently implement online sample selection with statistical guarantees under various constraints.
To address this issue, we summarize common constraints into two types, i.e., individual constraints and interactive constraints. The former one is relevant to the cost of selected samples, and one typical example is the fundamental and crucial criterion, false selection rate (FSR), which quantifies the proportion of falsely selected samples and is equivalent to the well-adopted FDR. The latter constraint captures the interactive influence among selected samples and is regarded as some kind of quadratic constraint some pairwise functions, such as the similarity or diversity among selected samples.
While the individual constraint associated with online FDR control has gained some attentions [16 18) it alone fails to capture the nuanced pairwise relationships among different samples. To bridge this gap, we introduce interactive constraints, which are pivotal within our framework, Building upon individual constraints, the interactive criteria significantly expand the range of constraints our method can control. Integrating two distinct types constraints into a unified framework makes easier to create practical algorithms and ensures theoretical guarantees.
A motivating example: candidate screening. As an example, in recruitment, screening from the resumes arriving sequentially to determine viable candidates who can get into interview processes is an important problem in human resource management [13 35). In this case, one may be interested in: (1) controlling the online FSR to enhance resource efficiency [2]. and (2) maintaining a desired level of candidate diversity during the screening process, thereby reducing bias [23 44]. The individual
and interactive constraints and the novel real-time sample selection procedure we propose can solve
this problem precisely. See Section 4.2 for the details and more real data examples.
1.1 Our Contributions
In this paper, we design novel and flexible online selection rule to effectively ensure the above two types of constraints are under control at pre-specified levels simultaneously, named as "Individual and Interactive Constrained Online Selection" (II-COS). S. The main idea stems from an oracle model based on the local false discovery rate (IFDR) 12 38 which is involved in offering valid evidence on whether Y, is our interest at each time point. With some appropriately chosen evaluating functions, the II-COS procedure entails validating whether the estimates of constraints are controlled. Simulated and real-data examples clearly demonstrate the superiority of the I1-COS in terms of both online individual and interactive criteria control.
To the best of our knowledge, this is the first work to systematically bridge the predictive inference and online selection procedure with various constraints. Our contributions are summarized follows:
Under a unified framework, the II-COS addresses how to implement online predictive selection sufficiently in consideration of both individual and interactive constraints. It is flexible to characterize the selective uncertainty and trade-off the sampling efficiency and practical limitations in the covariate space.
Under mild conditions, we establish the theoretical guarantee that the II-COS is able to control both individual and interactive constraints simultaneously and asymptotically under one given stopping rule.
The II-COS is model-agnostic in the sense that its implementation is applicable to any (appropriate) learning algorithms. Extensive numerical experiments indicate that the II-COS can significantly outperform existing ones while yielding effective constraints control.
2

---

1.2 Related Works
Our proposed method is built upon two fundamental pillars: (1) quantifying the uncertainty of response predictions using predictive inference; and (2) systematically addressing individual and interactive constraints in a sequential manner. Our work is intricately connected to the fields of predictive inference and online multiple testing. Here we briefly review literature these two topics.
Predictive Inference. One key ingredient of our proposed method is predictive/conformal inference. Conformal inference 42 34] provides a powerful and flexible tool to achieve algorithm-agnosie uncertainty quantification of predictions. Conventionally, conformal inference aims to build the pre- diction intervals and enjoys valid and distribution- free properties by leveraging data exchangeability Taking different but related perspective from muuliple-testing. Bates et al. 77 pioneered a method to construct conformal p-values to detect outliers with finite-sample FDR control. Building upon this, recent advancements have improved detection power by involving more information or performing model selection 50 26
125 3).
51 46 The most related works are Jin and Candès [2] and Wu et al. [45], which considered a similar scenario that one would like to select some individuals of interest by controlling FDR or maximizing the diversity of selected samples in off-line setting. Their methods are based on the conformal p-values or IFDR constructed with the predicted response values, respectively. Besides the fundamental difference between online and offline paradigms, our framework for characterizing various constraints poses additional challenges in how to select samples sequentially since we have multiple goals to achieve.
27
Online Multiple Testing. When only considering individual constraint as FSR control, the online sample selection can be reformulated as online multiple testing problem. Methods for online multiple testing have received much recent attention and were pioneered by Foster and Stine [66 who proposed
the so-called o-investing strategy, which was later built upon and generalized I 29 30 18 19]. The
key idea in a-investing and its generalizations is to compare p-values with dynamic thresholds and gain some extra a-wealth for each rejection. We refer to Robertson et al. (33] for thorough overview. Those rules suffer from the "alpha-death" " issue to some extent [29]. which means a permanent end to decision-making making when the decision threshold is too small, i..., the online procedure stops early. This phenomenon occurs in many existing online multiple testing algorithms, as discussed in (29]. Along different direction, Gang et al. [17] developed new class of structure-adaptive sequential testing (SAST) rules built on the IFDR to avoid the alpha-death issue. The SAST serves as building block for developing our II-OOS procedure and can be essentially seen special case of ours. Later on, Ao et al. 4 reformulated online multiple testing procedure into an online knapsack problem, providing novel policies with near-optimal regret guarantees. Additionally, Xu and Ramdas [48) proposed to use e-values [41) for online multiple testing to address the dependence. However, those existing works do not take predictive inference into account and are concerned only with online error rate control without exploration of the covariate space, which may greatly hamper their applicability.
2 Individual and Interactive Constrained Online Selection Procedure
2.1 Problem Formulation
Assume there exists historical labeled dataset as D (X,, Po) where (X,, Y.)'s are independent and identically distributed (i.i.d) from (X,Y) A sequence of unlabeled data X1, X2, ~ X "X)) arrives in stream with unknown responses Y1, Yy..... At each time t, one must make a real-time decision about whether or not to select the t-th individual, which is determined by some pre-specified requirement on Y. Denote A as the target region of Y,, which differs depending on users' specifications. For example, in regression setting, the requirement could be of the form Yi E Ja, b). (-o0, a) or Y; 2 b.
Let 0, I(Y, E A) describe the true state of Yr. Denote a decision rule as d, [0, 1}. where &, 1 indicates that the X, is selected and o, = 0 otherwise. A false selection is made if à, = 1 but 02 0. Denote 8' = {6, i < I} as the decision rule and T as the time that the procedure stops. Our goal is to build decision rule 8¹ to select samples with (Y, E A) up stopping time T such that the following two general types of constraints hold simultaneously.
Individual Constraint. In practice, one main concern to control the cost of selecting samples of interest. For example, in online recruitment, companies need to control the proportion of selected
3

---

unqualified candidates or the average loss when hiring someone who rejects the offer. In such cases, we can assign each selected sample a cost associated with some pre-specified function of the covariate X and control the expected cost associated with time 7 at the target level. We refer to this requirement as individual constraint and write as:
Ei 10 - Ox)))(X,) + 0,G, (X,)()
(6 =
(I)
6)) 6.) V1
where aVb = m o. b} and Go(X) 2 and ( 2 0 with Go o G1 are the costs corresponding to and / = 1, respectively. Here, we take expectation due to the randomness of i..... 00..
For example, when we simply choose Go(X) = 1 and (X) O, the individual constraint the popular false selection rate (FSR), i.e.
Σ E (1 0.)6 [ is 8,) V
( = FS) =
(2)
The FSR is essentially equivalent the well-adopted FDR in multiple testing literature, which is a useful tool to maintain the ability to reliably select samples of interest without excessively false selections ( Some works on online FDR control have been well studied. [16
The individual constraints alone cannot capture the pairwise relationship among different samples. We address this by introducing interactive constraints below.
Interactive Constraint Another common concern the interactive constraint, which involves choosing more preferable samples. For example, companies would like to retain candidates with a diverse range of backgrounds and experiences in online recruitment, or real-time suggested contents are required to avoid homogeneity in recommendation systems. Here, we introduce a bi-variate weight function gXX, X') to evaluate the interaction between selected samples. Denote PC((") = ΣΣ g(X,, 00,000 PS(6") ΣΣ 0,0,6,6,. 1si<jst
We define the interactive constraint as
1skj
C2 [PC(8'] E PS(85)
Here, since only the correctly selected samples are of interest, the constraint concerned with the average mutual effects between the correctly selected ones rather than all selected ones. When choosing the function as some similarities, controlling C2(5') at specified constant K, i.e., C2 < K, is controlling the expected similarity (ES). It is equivalent to requiring that correctly selected samples exhibit certain diversity and rich information in the covariate space of interest.
Typically, one useful choice for 9(X,X') X') is the weighted RBF kernel g(X.'')
exp
1 0z Ei= We (Xx - -x()²} with parameter a > 0 to measure the similarity between two independent X and X'. The RBF kernel is a common and widely embraced choice in machine learning 49 28 Here, {WI.... Wa] are some given weights per users' needs. For instance, if
one is just interested in the effects of the k-th feature, then simply w/k = and w, = 0 for j * k.
Specifically, the case that Wp similarity choices of g( (X, X'), such as the cosine similarity (') X X' X/|[|X/||||X
for all k 1, ... d is chosen in Section 4 We also consider other
[52)
Due to the randomness in the denominator, it turns out controlling we employ a modified interactive constraint,
directly is not easy. Instead,
E [PC(8')
C₂ (8') EE [PS(8*]
(4)
The constraint 4 aims to control ratio of expectations, which is still reasonable interactive measure. In numerical studies, we see that C2(5") in (3) and C2(5") in 4 yield almost identical patterns. An illustrative example can be found in Appendix D.I
In sum, the goal is to select samples of interest by a decision nule 8" controlling both the individual and interactive constraints until stopping time T, i.e., (8 < a and C2(8") S K. We emphasize that Ci (5 and C2(8")) as well as their pre-specified levels a and K can be chosen up to the practical applications.

---

2.2 Oracle Selection Procedure
To design general rule that is valid for any arbitrary stopping time T, we consider controlling the constraints at each time t in an online fashion, such that suPten C1(6") S and suPteN C2(6") S K.
Since Y, is unavailable, we consider utilizing predictive inference to measure the suspicious patterns. Let fx(x) == y X x be the regression or classification model associated with V,), and one reliable estimate 77(+), being estimated on the labeled data 'D with some machine learning algorithm. Denote W = (T(;;) as predicted value of Y; and assume that 7i(-) is bijection almost surely. The bijection assumption considerably mild and widely adopted for the identyification of each X, in the predictive inference framework [45]. The 0, L[Y; E AA is Bernoulli(r) distributed with ग Pr(Y, E A), and W, can be viewed as generated from one two-group model
We 0g ~ (1 (1- fo + 0.fi.
where fo and fi denote the probability distribution functions of W; conditional on Yf & A (i.e., 0. 0) and Y A, respectively. Then, the conditional probability of Y & A
L. = Pr(O, = W,) (1 foW2))
(5)
S(W,)
where (1 T) Tf.. The L, coincides with the local FDR in multiple testing literature
12 17 With the two-group model (5). we have E[0; Xx] = 1 - La and further notice that the
individual constraint Ci ( in ① can be exactly satisfied if
V/ Eist Looo + (1- L.))( S.
=
R2
holds. Here, we denote VÚ2 = Eise (L, + (1-
(X,)6:} and the number of selected
ones as Rs = Eis 6, V 1 for notational convenience. Especially, when Go(X) = 1. G1(X) = 0.
then FSR(5') 2 can be exactly controlled.
Accordingly, the interactive constraint C2(0") < K in 4 can be achieved if
ΣΣ g(,, x,) L1)(1 L,)6,65 ISKjS
TS,
ΣΣ (1- L,)(1 - L5)0,0d S K, IgisiSt
NS:
where the expected total mutual effects conditional on {Xi and the expected number are denoted TS, and NS,, respectively.
Therefore, L, is known, when new sample X, arrives time point t, we can perform the decision rule as follows. Note that there is no need to consider interactive effects before the first selection. When comes before the first selection (i.e, R = 0), the decision rule is 6, if
V + GoGo(X,))) + G.
Sa.
(6)
R
holds; otherwise, 6, 0 which means X, is not selected. When X, arrives with R,- 2 then 6, = 1 (6) and
TS
Σ 9X, X,)(1 L:)o;
(1- Lt)
15t-1
S K
NS.-1 +
Σ (1 iS1-1
L.)6g (1-L))
(1-1.)
hold simultaneously; otherwise, 8, = Note that if we set C, (6') as FSR(6') and choose ooc then our method essentially reduces to the same manner as the controlling step of the SAST in Gang al. 17.. Our proposed method can be seen as much more generalized and flexible framework for controlling both the individual and the interactive constraints simultaneously in an online fashion.
We call this method the oracle II-COS (Individual and Interactive Constrained Online Selection), The workflow in Figure show the procedure of the oracle II-CO.. The following result shows that it can exactly achieve our goal.
Proposition 2.1. Assume L, values are known. Then the oracle II-COS selection rule controls both constraints at any stopping time T, i.e., (T S a and Cz( s K.
5

---

X;
Orade II-COS
Impu::
Wo,dy
(e)
Yes
No
C2(1)
Update:
Yes
1
7Sg, NS,
Figure 1: The implementation flowchart of the oracle II-COS procedure.
2.3 Data-driven II-COS Procedure
As L, is unknown in practice, we propose a data-driven II-COS procedure, which uses a reliable estimation T,, for implementation. We resort to a data-splitting strategy: randomly split historical data D into two parts, the training set Dr and the calibration one Deal of sizes no and n1 respectively, where Dir is used for training predictive model and Deal is to estimate those unknown parameters. Specifically, we first fit a regression or classification model īi(-) on Di. and then obtain predicted values on Deat. i.e. (f(X)): (X,, Y1) E Deal). Note that conditional on Die. (ū(X,) : (X,, Yj) E Deal) are i.i.d. random variables and have the same distribution W. ((X+). ), so that it can be utilized to estimate (5)
Therefore, the estimators of fo and f. fo and F. can be obtained by applying the kernel density estimation method to the data (m(X,) (X,, Y)) E Deali Y, A} and (ā(X,) (X,, Y,) E Deal), respectively. And the probability ni E.E. I(Y; A). Further the L, in can be estimated by
Pr(Y, E A) can be approximated by क
L, (1 1-- foW.) A1
(8)
f(W,)
The data-driven II-COS procedure is summarized in Algorithm I and it indeed consists of two phases: offline estimation and online decision. The running time of offline estimation is not critical. At each time t, the computational complexity is linear function of the currently selected number R,. More implementation details can be found in Section and Appendix B.I
In fact, the proposed IICOS is flexible to trade off the individual and interactive constraints by adjusting the thresholds and K. If one is concerned only with individual cost control, then can set = +00, with which the interactive constraint is out of work. Similarly, only the interactive effect of interest when a Appendix D22 provides toy example illustrate this.
Before further pursuing, we would discuss the stopping time in practice. It's worth noting that the specific choice of stopping rule (and thus stopping time) is completely up to the user. For example, when > 2 is the desired number of selections, one set T inf, Ei= 8, m}. Or when is the total wages for recruitment, one can set T inf, {t: E 5,s; = s} where s; is the payroll for each selected candidate. Or T is just chosen as one given deadline. With the use of II-COS, practitioners have the flexibility to design diverse stopping strategies that can adapt seamlessly to their specific applications. In brief, our method is flexible and is appropriate for various goals based on the user's requirements by choosing different Gx(X), G, (X), g( (X,, X,) and varied target levels (a, K) and a user-specified stopping time.

---

Algorithm The data-driven II-COS procedure
Target levels and K, pairwise function 9, cost Go(X) and Gi (X), stopping time T, interested region A, labeled data D, prediction algorithm H.
Input:
Initialization: 0, Vi R = 0, TS; NS, O: Decision rule 8' 0
Estimation: Randomly split D into training set Dur and calibration set Deal. On Dir, fit (x(x with H. Obtain 7, fo and from Deal-
Online decisions: while S T do Set f = + 1. Compute W, (xx),,, Le by 8 and the similarities g(X,, Xx) for those ó; = 1.
Replace L, by L, in
6 and 7
- < 1 and (6 holds then & = 1;
else 6, 0:
R 2 66 and hold then òe 1;
else 6, 0:
Update: 8' 8'- 1--U((60 R. Re-1 de: = Vi-1 (L.Go(Xi) (1 L.)))(XX))o::
TS, = TS
,(x Li)os (1 L.)oi; NS. -NS.1 Σ 1.)0
(1 - L.)0is
end
Output: Selection set (X, 6, J.
Extension to varying proportion case. In practice, the distribution of (X,, Yt) may vary smoothly over time. In Appendix C we consider the probability of Y; E A (i.e. the proportion of samples in the specified region) varying time and extend the proposed II-COS learn मं = Pr(Y E A) continuously over time and we also construct the corresponding theoretical guarantees.
Statistical Performance Guarantees
In this section, we provide statistical guarantees for the data-driven II-COS procedure. The main difficulties lie in the quantification of data-driven estimation error of L, and we utilize the classical kernel density estimation theory along with the structure of our online procedure to effectively characterize it. For simplicity, we consider that the training data is given such that the estimated model ji is fixed. Before presenting our theoretical results, we state the following regularity conditions.
Assumption 3.1 (Density functions and kernel). The density functions and kernel function satisfy
(1) The f-(+) and fo(-) are upper bounded by M > 0, and the f(-) is lower bounded by > 0.
(2) The fo and fi are földer-continuour i.c. (f(()) - fo(w")l S calw u w/18 for any w, w e IR, and the same for fi with some fixed 0 < 3 and constant cp.
(3) Kernel K(-) is bounded symmetric function and enjoys exponential decay.
Assumption 3.2 (Weight functions). There exists constants cG > 0 and Co > 0 such that < Gx(X) < cc. 0 < (X) cg for any X and 0 < x'X 1 Cy for any X o X'.
Assumption 3.1 considerably mild and widely adopted in the uniform convergence of kernel density estimation [36]. If fo and fi have bounded first-order derivatives, the Hölder-continuous assumption would hold with 3 1. The lower bound of f is to ensure the uniform convergence of the estimated IFDR. Assumption 3. mild since the weight functions are required only to be positive and bounded. It can be satisfied by a large category of Gg. G1 and g. For example, we (X)
take G = a, X|| for j = 0 or and ca exists when is bounded. And we can set g as the RBF and orthogonal similarities with c = 1.
With those regularity conditions, we establish the validity of the II-COS procedure for the individual constraint control.
7

---

Theorem 3.3 (Bound for individual constraint). Suppose Assumptions 3.1 and 3.2 hold and take the bandwidths for estimating and fo in the order of n - -1/(2++1) Then for any given time the individual constraint of the IICOS procedure (Algorithm satisfies Ci (8') S + Ang where Vlg ny and D is a constant depending on M, l, CB. B3 n, CG and K())
-B
Am, = 1
201I
Although the C1(+) of II-COS might be slightly larger than the target level in finite samples, this gap converges asymptotically ny increases. In the numerical studies, we find that small calibration size of around 200 is enough to control the C1 in a reasonable range. Taking the FSR as an individual constraint, Theorem 333 indicates that method can provide asymptotic online FSR control similar to an online FDR control procedure [[6].
The next theorem examines the performance of the II-COS in terms of interactive constraint.
Theorem 3.4 (Bound for interactive constraint). Suppose Assumptions 3.7 3.2 hold and take the bandwidths for estimating f and fo in the order nf 11(22++1), Let Let inf inf Eim1 6; 8) for > 2 and assume there exists constant a' E (0, 1) such that Eise L,,,/(15 R.) s Then for any given time 2 Tm, the interactive constraint of the I1-COS satisfies
(K + Cg Ans
C2(6') S
0.5
may
Anj
m-1
The term ma'/(m 1) used to characterize the lower bound of the denominator term of the interactive constraint. Specifically, when we choose FSR as the individual constraint, we have a' = a, which demonstrates the interdependence between controlling individual and interactive constraints. Furthermore, under arbitrary stopping strategies with stopping time T,, we can have the asymptotic guarantee.
Corollary 3.5. Suppose the conditions in Theorem 3.4 hold, the stopping moment T 2 Tm and o' < (1-1111m 1/m)/ Then the II-OOS procedure controls the individual and interactive constraints asymptotically T, i.e. lim Cj (6") S and limm C2(0") K.
4 Experiments and Evaluation
We illustrate the breadth of applicability of the II-COS procedure by experiments on simulated data and real-data applications. As an example, we set the stopping rule as selecting total m = 100 samples, i.e., T' Tm = inf,(t: : Σ'= 6, = m².. Additional experiments including the extended II- COS in Appendix experiments and figures in our paper is available at httttss:///tthhh
Caree are shown in Appendix D.9 Code for implementing II-COS and reproducing the
com/ homm/uull wwllllnn22///// III-
Implementation of II-COS. To our best knowledge, online selection with uncertainty qualification has only been studied the field of online multiple testing, which aims to control online FDR. Hence, we focus using FSR as the individual criterion and modified ES as the interactive criterion. As X may be measured on scales with widely differing ranges different dimensions, we assume that X's have been properly scaled in each dimension before computing 9. We choose g the weighted RBF kernel with a 1, Wp = 1 here. Other choices for individual and interactive constraints are considered in Appendix D.5
Benchmarks. We compare the II-COS procedure with four benchmarks from online multiple testing. The first one is structure- structue-addattvve sequential testing rule, the SAST (17), which is implemented with L, It can achieve the FSR control but ignore the interactive constraint. As mentioned earlier in Section 2 SAST can also be considered as a special case of our II-COS with K too. Its details are deferred Appendix B.2 The other competitors
three well-known online FDR control algorithms LOND 18). SAFFRON 30 and ADDIS 40 implemented with the conformal p-values suggested by Bates et al. 77. Refer more information in Appendix B.3] All the benchmarks can only control FDR, which demonstrates the flexibility of our method for different constraints.
Performance Measures. The empirical FSR, ES and stopping time (Tm) are evaluated using the average values of the false selection proportions, the similarity and the stopping time from 500 replications, respectively, where Tm, serves criterion for assessing selection efficiency.

---

4.1 Results on Synthetic Data
Data Description. We consider classification model: X
Na (11. and X
1 ~ Na (1g I4), where μ (5, 0, 0)T M42 (0, 0, -3 -2) We set Pr(Y 1) 0.2. The information set is A = {1). The predictor H is taken as random forest with defaulted parameters. We also consider regression setting and conduct additional experiments in Appendix D
Firstly, we observe that methods relying on conformal p-values, such as LOND, SAFFRON, and ADDIS, encounter the alpha-death (stop early) issue [2]]. These methods struggle to select an adequate number of samples, especially in small calibration sets. In contrast, II-COS ensures the control of both individual and interactive constraints even with a small neal (e.g., 200). See more details and results in Appendix D.3 Hence, to make a fair comparison, we consider a relatively large size of the calibration set, Paai 4, 000. We fix training data size fhr 1, 000.
0.20
0.15
0.073
((FRR)
0.050
0.05
0.025
0.00
100
200
300
100
200
300
Time
Time
Method
11-C0s
SAST
LOND
SAFFRON 1
ADDIS
Figure 2: The values of FSR(8*) and ES(8") over time for II-COS, SAST. LOND, SAFFRON and ADDIS. The black dashed lines denote the FSR level 0.1 and the ES level 0.045. Shading represents error bars of one standard error above and below.
0.15
c(ess)
Cafss)
0.304
0.1
0.05
250
00.
0.00
I1-009
5407
LoND
ADES
B-4200
Stor
Lon0
ADES
B-ctm
LoND
ADRS
ES 8 and stopping time Tin II-COS, SAST, LOND. SAFFRON and ADDIS. The black dashed lines indicate the corresponding nominal levels.
Figure 3: Boxplots of FSR(
Results. Figure 2 presents the online FSR (false selection rate) and ES (expected similarity) values of five methods against time t, which are individual and interactive constraints, respectively. The FSR levels of II-COS and SAST are closer to the nominal level than the other three methods. As expected, only the empirical ES levels of II-COS are controlled under the pre-specified level K over time t. The LOND and SAFFRON lead to slightly conservative FSR values, while the FSR levels of ADDIS are inflated compared to the target level. Figure 3 further displays the boxplots of empirical FSR, ES at stopping time Tm. We observe that only II-COS achieves satisfactory ES values compared to the nominal level. Moreover, the II-COS has a relatively larger value of T.,. compared to those of other benchmarks. This is consistent with the fact that the II-COS spends more time exploring the structure information inside the covariate space due to the requirement of interactive constraint. Similar conclusions for the regression example in Appendix D.4 can be drawn.
Regarding efficiency, we also conducted an experiment to compare the effectiveness of II-COS with an oracle method possessing knowledge of true state O,. The Tm, of II-COS is very close to the oracle. This close proximity indicates the high efficiency of II-COS. See Appendix D.8 the details.
4.2 Results on Real Data
We next demonstrate the performance of the II-COS in two real-world applications. Since those online multiple testing methods based on conformal p-values yield few selected individuals, we focus

---

on comparing the II-COS with the SAST. For comparison, we also include the offline method using conformal p-values, where 6, if p. I a. We denote it as CP.
Datasets. We consider the recruitment dataset from Kaggle (22 that contains 45,372 candidates after removing the missing data and records binary response indicating whether the candidate passes the job interview, and other 1I attributes including education status, handicapped or not, and gender. The other problem is to use 1994 Census Bureau dataset
select a subset of individuals who may have high incomes in precision marketing. This census dataset records 32,561 individuals with their 14 attributes, including gender, race, marriage, education length and so on.
For each dataset, we randomly partition the data into three parts: Ru = 1, 000 training data, Neaal = 1, 000 calibration data and the rest which are used as the online observations. The categorical attributes are converted into one-hot codes and then are treated as numerical attributes for computing similarity measures. The prediction algorithm H is random forest with defaulted parameters.
Results. Table I reports the results among 500 repetitions. Both the II-COS and SAST enjoy valid FSR control, but CP yields an inflated FSR level in income investigation. The II-COS performs well in terms of similarities. To further compare the diversities, we present the proportions of different education status in in Figure 4 It can be seen that the proposed II-COS demonstrates its superior diversity in the specific attributes. See Appendix D for more results for the real data. In summary, the proposed II-COS works well for selecting individuals of interest to achieve various constraints in practical applications.
Table 1: Average values with candidate dataset and income dataset: FSR(6" ES(5"T)) (x10- and stopping time Tm. The target FSR level is a = 0.2 for both. For the candidate data, the target ES level K = 1 x 10 ; For the income data, K = 6 10 3 The bracket contains the standard error.
(a) Candidate dataset (2
(b) Income dataset
Method
FSR
ES
Tm
Method
FSR
T,
II-COS
0.19 (0.002)
0.98 (0:005)
2227 (91.6)
II-COS
0.16 (0.007)
5.56 (0.128)
2760 (227.0) 0)
SAST
0.19 (0.002)
8.73 (0.269)
310 (35.2)
SAST
0.19 (0.008)
30.90 (1.078)
1200 (202.3)
CP
0.16 (0.002)
10.34 (0.084)
277 (1.49)
CP
0.42 (0.006)
18.84 (0.640)
283 (3.883)
Method
I-CO5
SAST
Mathod
II-COS
0.5
044
Preerton
0.1
0.
No Qual
Scheel Matiic Welenn Rachebes
15
Educofon S20S
Education Lengh
Figure 4: Left: Education status composition of the correctly selected samples (II-COS, SAST and CP) in candidate dataset; Right: Education length (year) composition of the correctly selected samples (II-COS, SAST) income dataset. The plots have error bar to show the variation across the 500 runs.
5 Concluding Remarks
Broader Impacts. This work focuses on creating reliable machine learning tools for making real-time decisions. One key achievement is new algorithm called II-COS, designed to select informative samples in real-time while meeting two types of general constraints. II-COS allows for both individual and interactive control, validated through theoretical analysis and numerical tests. Our method is model-agnostic and easily applicable to many real-world cases such as producing diversified results while controlling FSR for online recruitment. One potential negative impact of our work is that researchers will apply the algorithm without sufficient scrutiny. We emphasize that it's i's important to use caution when applying this method to complex real-world scenarios to prevent misuse.
Limitations. Firstly, we mainly consider binary functions as the interactive constraint. How to adapt the II-COS to other popular constraints, such as the Gini index, deserves further study. Secondly, in certain practical scenarios, is possible to obtain feedback after decisions. Incorporating the feedback information into our method to enhance its performance warrants future research.
10

---

Acknowledgments and Disclosure of Funding
We thank anonymous area chair and reviewers for their helpful comments. Zou was supported by the National Key R&D Program of China (Grant Nos. 2022YFA 1003703, 2022YFA1003800) the National Natural Science Foundation of China (Grant Nos. 11925 106, 12231011, 11931001, 12226007, 12326325). Ren was supported by the National Natural Science Foundation of China (Grant Nos. 12101398, 12471262), and Young Elite Scientists Sponsorship Program by CAST.
References
[1] Ehud Aharoni and Saharon Rosset. Generalized c-investing: definitions, optimality results and application to public databases. Journal of the Royal Statistical Sociery: Series B (Staristical Methodology). 76(4):771-79) 2014.
[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[3] Anastasios N Angelopoulos and Stephen Bates. Conformal prediction: A gentle introduction. Foundations and Trends Machine Leurning, 16(4))494-591 2023.
[4] Ruicheng Ao, Hongyu Chen, David Simchi-Levi, and Feng Zhu. Online local false discovery rate control: A resource allocation approach. arXiv preprint arXiv:2402.11425 2024.
[5] Eric Bach, Shuchi Chawla, and Seeun Umboh. Threshold rules for online sample selection. Discrete Mathematics, Algorithms and Applications, 2(4))625 2010.
[6] Peter L. Bartlett. Learning with a slowly changing distribution. In Proceedings of the Fifth Annual Workshop Computational Learning Theory, page 243-252, 1992.
[7] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. Annals of Statistics, 51(1):149-178, 2023.
[8] Barry Becker and Ronny Kohavi. Adult income investigation. UCI Machine Learning Repository https://archive. ics uci www.eddddaaaae Salp/daataaset//aduutt 1996.
[9] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 57(1):289-300, 1995.
[10] Yoav Benjamini and Daniel Yekuticli. The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29(4): 165-1 188, 2001.
[11] Rick Durrett. Probability: theory and examples, volume 49. Cambridge University Press, 2019.
[12] Bradley Efron, Robert Tibshirani, John D Storey, and Virginia Tusher. Empirical bayes analysis of microarray experiment. Journal of the American Statistical Association, 96(456):1151- 1 1160, 2001.
[13] Evanthia Faliagka, Kostas Ramantas, Athanasios Tsakalidis, and Giannis Tzimas. Application of machine learning algorithms to an online recruitment system. In Proc. International Conference on Internet and Web Applications and Services, pages 215-220, 2012.
[14] Evanthia Faliagka, I Lazaros Iliadis, loannis Karydis, Maria Rigou, Spyros Sioutas, Athanasios Tsakalidis, and Giannis Tzimas. On-line consistent ranking e-recruitment: seeking the truth behind a well-formed cv. Artificial Intelligence Review, 42(3):515-522 2014.
[15] Kai-Tai Fang. Runze Li, and Agus Sudjianto. Design and modeling for computer experiments. Chapman and Hall/CRC, 2005.
[16] Dean P Foster and Robert A Stine. a-investing: procedure for sequential control of expected false discoveries. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 70(2):429 444, 2008.
11

---

[17] Bowen Gang, Wenguang Sun, and Weinan Wang. Structure- adaptive sequential testing for online false discovery rate control. Journal of the American Statistical Association, 118(541): 732-745, 2023.
[18] Adel Javanmard and Andrea Montanari. On online control of false discovery rate. arXiv preprint 1rXiv:1502.06117. 2015.
[19] Adel Javanmard and Andrea Montanari. Online rules for control of false discovery rate and false discovery exceedance. The Annals of Statistics, 46(2):526-554, 2018.
[20] Heinrich Jiang. Uniform convergence rates for kernel density estimation. In International Conference Machine Learning, pages 1694-1703, PMLR, 2017.
[21] Ying Jin and Emmanuel Candès. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1-4) 2023.
[22] Kaggle.
Candidate selection dataset.
Aarunchilkurcliieei 2020.
[23] Kanagavalli, R Seethalakshmi, and T Sowdamini. A systematic review of literature on recruitment and selection process. Humanities & Social Sciences Reviews, 7(2):01-09, 2019.
[24] Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. News recommender systems-survey and roads ahead. Information Processing & Management, 54(6):1203-1227, 2018.
[25] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-Fre free predictive inference for regression, Journal of the American Statistical Associ- ation, 113(523); 1094-1111, 2018.
[26] Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for out-of- distribution testing with labelled outliers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(3):671 1663,, 01 2024.
[27] Ariane Marandon, Lihua Lei. David Mary, and Etienne Roquain. Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics, 52(1):157-183, 2024.
[28] Houwen Peng, Bing Li, Haibin Ling. Weiming Hu, Weihua Xiong, and Stephen J. Maybank. Salient object detection via structured matrix decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 011:818-832221 2017.
[29] Aaditya Ramdas, Fanny Yang, Martin J Wainwright, and Michael I Jordan. Online control of the false discovery rate with decaying memory. Advances in Neural Information Processing Systems, 30:5655-5664. 2017.
[30] Aaditya Ramdas, Tijana Zrnic, Martin Wainwright, and Michael Jordan. Saffron: an adaptive algorithm for online control of the false discovery rate. In International Conference on Machine Learning, pages 4286-4294. PMLR, 2018
[31] Bradley Rava, Wenguang Sun, Gareth M James, and Xin Tong. A burden shared is a burden halved: A fairness-adjusted approach to classification. arXiv preprint I0.05500 2021.
[32] David S Robertson, Jan Wildenhain, Adel Javanmard, and Natasha A Karp. onlineFDR: an R package to control the false discovery rate for growing data repositories. Bioinformatics, 35 (20):416 4199, 2019.
[33] David Robertson, James MS Wason, and Aaditya Ramdas. Online multiple hypothesis testing. Statistical Science, 38(4):557, 2023.
[34] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3):371-4211 2008
[35] Muhammad Ahmad Shehu and Faisal Saeed. An adaptive personnel selection model for recruitment using domain-driver driven data mining. Journal of Theoretical and Applied Information Technology, 91(1):117, 2016.
12

---

[36] Bernard W. Silverman. Weak and strong uniform consisteney of the kernel estimate of a density and its derivatives. The Annals of Statisties, 6(1):177-184, 1978.
[37] John D Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(3):479-498, 2002.
[38] Wenguang Sun and T. Tony Cai. Oracle and adaptive compound decision rules for false discovery rate control. Journal of the American Statistical Association, 102(479):001-912, 2007.
[39] Stephen Taylor. Resourcing and talent management. Kogan Page Publishers, 2018.
[40] Jinjin Tian and Aaditya Ramdas. Addis: an adaptive discarding algorithm for online fdr control with conservative nulls. Advances in Neural Information Processing Systems, 32:9388-9999 9396, 2019.
[41] Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications. The Annals of Statistics, (099)1736-154,4 2021.
[42] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world. New York: Springer, 2005.
[43] Michel Wedel and PK Kannan. Marketing analytics for data-rich environments. Journal of Marketing, 80(6))97-121, 2016.
[44] Sang Eun Woo, James M LeBreton, Melissa G Keith, and Louis Tay. Bias, fairness, and validity in graduate-school admissions: A psychometric perspective. Perspectives on Psychological Science, 18(1):3-31, 2023.
[45] Xiaoyang Wu, Yuyang Huo, Haojie Ren, and Changliang Zou. Optimal subsampling via predictive inference. Journal of the American Statistical Association, pages 1-29, 2023.
[46] Xiaoyang Wu, Lin Lu, Zhaojun Wang, and Changliang Zou, Conditional testing based localized conformal p-values. arXiv preprint arXiv:2409. 168299 2024.
[47] Pengtao Xie, Yuntian Deng, and Eric Xing. Latent variable modeling with diversity-inducing mutual angular regularization. arXiv preprint 1r12 2015.
[48] Ziyu Xu and Aaditya Ramdas. Online multiple testing with e-vauuss. In International Conference on Artificial Intelligence and Statisties, pages 3997-4005. PMLR, 2024.
[49] Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113(2):113-12) 2015.
[50] Yifan Zhang, Haiyan Jiang, Haojie Ren, Changliang Zou, and Dejing Dou. Automs: automatic model selection for novelty detection with error rate control. Advances in Neural Information Processing Systems, 35:19917-19929, 2022.
[51] Zinan Zhao and Wenguang Sun. False discovery rate control for structured multiple testing: Asymmetric rules and conformal q-values. Journal of the American Statistical Association, pages 1-24. 2024.
[52] Ping Zhong, Zhiqiang Gong, Shutao Li, and Carola-Bibiant Schönlieb. Learning to diversify deep belief networks for hyperspectral image classification. IEEE Transactions Geoscience and Remote Sensing, 55(6):3516-3500 2017.
13

---

Supplementary Material for "Real-Time Selection Under General Constraints via Predictive Inference"
This supplementary material contains:
Preliminary terms for self-containment (Appendix A
Implementation details (Appendix B
An extension algorithm to varying proportion case (Appendix
Additional experiments (Appendix D):
The proofs of all the theoretical results. (Appendix
A Preliminary Terms for Self-Containment
Here, we list the preliminary terms use in the paper for the sake of clarity and self-containment.
FDR 9.. false discovery rate, widely-adopted error rate notion in the field of multiple testing, is defined as the expected proportion of incorrectly rejected null hypotheses as follows:
Ho n R(t)
FDR (t)
(R(t) v
where Ho is the unknown set of true null hypotheses, R(t) represents the set of rejected null hypotheses until time f and then Ho 7 (t)) is the set false discoveries.
FSR, false selection rate, defined the expected proportion of individuals being of interest among the selected subset of individuals. It is in fact equivalent to the definition of FDR. In our framework this paper, we describe it equivalently as:
Eisr( (1 0,)
FSR(t) E
(Eis 61) V
B Implementation Details of Algorithms
Offline Estimation
X
Online Decision
Compute:
W.
8
g(X, X.)
D+r
cal
No
x
(2(t)
pXX)
h.
Yes
Update: R, NS,
8
X
Figure 5: The implementation flowchart of the data-drive II-COS procedure.
14

---

B.1 Implementation Details of II-COS
Figure Ishows the implementation flowchart of our proposed data-driven I1-COS procedure, including the offline estimation step and online decision step. Here we introduce more implementation details to reproduce the results.
About the classifier. For the classification problem, we use the predicted conditional probability Pr(Y, = | X = x) as (x() in the procedure of II-COS. Many commonly used algorithms such as random forest and neural networks can provide such probability estimators. In fact, the choice of jr is not restricted to probability. For example, the support vector machine outputs the distance to the separating hyperplane for classification, and such distance can be chosen as fú.
However, for probability estimators, one potential problem is that when the classifier is quite accurate, most of Wi's are concentrated situation, it is difficult to accurately and stably estimate IFDR L 's since the density functions fo(w) and f (w) are not smooth enough and are not lower bounded. This yields that our assumptions are violated, but the II-COS can still perform satisfactorily with some corrections on the estimation L Notice that the larger W; is, the more likely Y; E A and hence the smaller L; is. That is L; should be monotonically decreasing as W; increases. Observing this, can make monotonization correction on T,'s. s. To be specific, we rearrange T. by the decreasing order of W, If (-1)) < (W) revise it by (())))))
1 and few of them take values between (0, 1). In such
which violates the monotonicity, smallest among W. This monotonization correction enables us to avoid obvious errors due to unstable estimation and improve the performances of all the methods utilizing L,'s.
L((()), (a)) where W(x) is the ith
Choice of K. A useful interactive constraint needs appropriate specification of K. For any two i..dd observations X and X' with corresponding and 8' respectively, the expected C2 of the individuals of interest is given C2 [Xg(X, X') 0 1, 1], which be estimated by 1)}, where f {i Y, E A). It is reasonable set K = aC2, where a > is user-specific to control the interactive constraint level. Our numerical evidence reveals that a E (0.1, 0.5) works generally well.
C2 E- g (X, /2)/(1(((
B.2 Implementation Details of SAST
Gang al. [17 proposed a adaptive sequential testing (SAST) rule for online false dis- covery rate control. In their work, the rejecting rule is as follows: If L, < and R1
1} (EERR_, L, - L1 S a, then ó; = 1. Otherwise ó, = 0, where R₂ = {i S t 8, = =13 and 7t barrier estimated from an "offline" procedure.
The implementation details SAST for comparisons our simulations in Section are different from the original one. Firstly, the original SAST assumes the null density function So is already known while in our setting fo remains unknown. Secondly, in our predictive inference setting, the density functions and the null proportion estimation procedure in Algorithm ! not from current rejection sets. Besides, considering the time-varying structures of the data stream in their setting, Gang al. [7 incorporated the barrier strategy in their method, which is not necessary to be adopted here.
directly estimated via calibration set the offline
B.3 Implementation Details of Conformal p-values
The notion of conformal p-values was originally proposed by Vovk et al. 42 to construct prediction
interval. Recently, there exist some works to apply conformal p-values to implement sample selection from a multiple-testing perspective, such Bates et al. [7], Rava et al. [31 and Jin and Candès [21]. In the sample selection problem, the hypothesis has the following form for each f,
Hor Y, A
H Y1 YYEA A.
There are two types of conformal vavuue and we adopt the one in Bates et al. 7 which utilizes the same class calibration. Recall that for W,, its conformal p-vauue pa is defined as
1 ΣIED_ (QQ()) < Q(W))
p.
1 iDeal
15

---

The nonconformity score function Q(W) is used to indicate the possibility of 82 0. For example,
in regression settings, if A [b. + we can use Q(W,) = W, If A (-00, a] [b, +00). then we can choose Q(W,) = max W a, b- W.). And in binary classification settings, if A = and W, indicates the probability of Y2 = 1, we set Q(W,) = W..
Even though conformal p-values are correlated, using conformal p-values to conduct multiple testing can control the FDR level with finite sample guarantee since they are positive regression dependent on a subset (PRDS) [T]. However, the conformal p-values are lower bounded by 1/([Paal + 1), which leads to unsatisfactory performance for online multiple testing methods based on p-values. Since these methods require sufficiently small p-values to make rejections.
In our simulations, implement LOND, SAFFRON and ADDIS for online sample selection by R package OnlineFDR [32) with @ = 0.1. Other parameters are defaulted. Here we introduce the details about these online FDR control methods. Ramdas al. [29] proposed "statistical
perspective" to control FDR in online setting, which is to keep an estimate of the FDP less than a similar the offline setting. Specifically, for offline FDR, let the rejection set R(s) = {ilpp S s}.
For online FDR, an oracle estimate of An oracle estimate for FDP is given by FDP (s) = [(()]VI
|'ol-s
FDP (t) is List.se %2
H(OVI Table lists a comparison of estimating FDP in classical offline methods and online methods for FDR control in multiple testing. For the online methods, denote the decision rule 6, (pr S ), where pa is the corresponding conformal p-value at time / for our problem. The test levels { for LOND 18) SAFFRON (30) and ADDIS [40] are listed as follows:
Table 2: A comparison of FDP in offline methods v.s. online methods for FDR control.
Offline
FDP
FDP(t)
Online
Σ Ejer aj R(DV
BH 9
10:6 R
LOND 18
VI
Storey-BH 37
m-a--o R VI
En 1(p.2A)
Σ j2t 0g (1-A,)
May>A;1
SAFFRON 30
=11=A)
RevV
Σ 0g HAgEM1ST2) T2-A3
ADDIS [40
Revvi
LOND: a, % (R(t 1) 1)+ 1), where (7t) is given infinite non-increasing sequence of positive constants that sums to a and R(n) = >i Rs denotes the number of discoveries in the first hypotheses tested.
. SAFFRON: At each time define Cj+ (1) = Eier++1 where C. = T(p, S A).
For t 1, 01 min (71 Wo, A}: For 2. 04 == min(2. āz), where
ã = Won-Coo Cot + ((1 A)o - Wo) Ctt + (1 - A)a Σ 7t-
J22
* ADDIS: The testing levels for ADDIS are given by a, = minff ( d where
â (1 2)( (a wn) Ci+ Σ
7st 7s--
122
and St = Eicr Hpp S n). TJ Eisr, Hp, S 5n)
B.4 Experiments Compute Resources
All the experiments were conducted on 3.11 GHz Intel Gen i5-11300H processors with 16 Gb memory at Lenovo personal computer and the R platform with version 4.2.1. The time of execution for each of the individual experimental runs is about 6.686 seconds. And the total compute time for the synthetic classification example in Section 4 for 500 replications is about 63.877 minutes.
16

---

B.5 A Toy Example for Illustration in Section 2.3
We illustrate the idea of the II-COS procedure via a binary classification example. We aim to select m 50 data points of specific class from unlabeled data arriving sequentially. We choose FSR as the individual constraint and mES as the interactive constraint.
The data generated as follows. The 4-dimensional covariates X = (X,, X2. X1. X4) are generated from mixture of multivariate normal distributions with mean (0, 0, O, 0) if Y 0 and mean ( (-3, -3. 0, 0) if Y = 1. The proportions of = 0 and In this illustrative example, we the historical data size n 1, 000 with calibration size 1 500 and would like to select m = 50 data points from the interest region A = {1} with the target FSR level a 0.1. The mES threshold K
are 80% and 20% respectively.
at 0.01 for II-COS.
Figure F depicts the scatterplot of the first two dimensions of the covariates X, with green dots and red triangles denoting correctly selected points and falsely selected ones, respectively. The SAST method proposed by Gang et al. [T7] is taken as one benchmark, which considers only the FSR control. We observe that the selected points of II-COS enjoy significant diversity among the covariate space and only a few false selections are contained. In contrast, the SAST is inclined to choose similar samples concentrated at the center of the concerned group and stops too early to fully explore the covariate space with sequentially arriving samples.
Correctly-seeccted
Feleely-selected
I1-COS
SAST
0.0
0.0-
x -2.5
x -2.5
-5.0-
-5.0
-6
X3
X1
Figure 6: Scatter plots of selected points of the II-COS and SAST. It stops when selecting 50 samples Left: the selection results of the II-COS; Right: the results of SAST. Green dots and red triangles indicate correct and false selections, respectively.
C Extension to Varying Proportion Case
In practice, the distribution of (X,; Yr) may vary smoothly over time. Due to the unknown Y,, is unrealistic to re-estimate parameters (i.e., Estimation Step in Algorithm 1) on both labeled data and the most recent data. To mitigate this problem, consider the probability of Y, E A (i.e. the proportion of samples in the specified region) varying over time and extend the proposed II-COS to learn Te Pr(Y, A) continuously over time.
Let Q(W) be one score function, which high when the possibility 0, is large and otherwise W. And in
low. For example, in regression settings, if A = (a, +oo), we can take Q(W,) binary classification, if A 1 and W, indicates the probability of Y, = 1, we can set QWW, 1-W2 W,- One valid conformal v-vulue for Q(W,) can be obtained as Bates et al. (7]
1 + ΣIED_ IQQ(W) S Q(W))
ps
1+ |Deal
Inspired by the techniques for null proportion estimation in multiple testing literature [37], we note that Pr(p. A) s Pr(ø, > X, 0) ne >) for large À E (0, 1) (i.e., À 0.5).
Thus, we consider using some recent P. to estimate reliable m. Denote g as the size of a neighbor- hood {t- ..... 1} and fix A. We employ an exponential weighted scheme estimate where
17

---

the more recent samples will contribute more to the estimation:
-ト - (1 (1- kbll -
Kolj 1)1111331 > }}
where ks((s) expl -ss/bb and b is the bandwidth parameter. Then, can compute the distribution of W as f fo( n.) f and estimate L, by L
(1-7))).W.)) A 1.
P(W)
The extended II-COS is to substitute L for Le in Algorithm the guarantee by assuming the slow change of distribution 66
The following theorem establishes
Theorem C.1. Assume Te satisfies | Wt. S rj for any t, and the bandwidth pa- rameter b satisfies Us S q .for some C
> 1. Denote A cin, (1128/12811) Vlog n
02 max(b-1/a 1//3, (log n1/n1)1/6, (by) (b3) with constants C1, Suppose and X satisfy Pr(p: > X
1) Under Assumption 3.7.mmpptoon...33........ 3.2 we have:
(a) For any given time I, the individual constraint of extended II-COS satisfies C1 S A;; (b) Furthermore, if the conditions Theorem 3.4 hold, then for any given time Tmu the interactive constraint of the extended IICOS satisfies C2(0") < K 0.5
(K+,))
Besides the part similar to An, in Theorem 3.3 the bound A' owns an additional term that can be decomposed into three parts. The b-1/3 indicates how many valid samples we use to estimate The second part comes from the approximation error of p-values. The last one characterizes the effects of distribution shift. If we properly choose such that byj = o(1), this term is negligible. Thus when and bn both tend infinity, A' converges 0 and the individual and interactive constraints will be controlled asymptotically.
D
Additional Experiments
D.1 Illustration of the Similar Patterns between C2 and C2
We calculate the empirical Ö2 and C2 during the online sample selection procedure under regression setting in Section D.4 with II-COS from 500 replications. The results are summarized in Figure
0.034
Velue
0.832
0.010
0.008
Figure 7: Illustration of the similar patterns between C2 and Cz. Line charts of (2 (red line) and C2(8") (blue line) over time the regression example. Experiment details are Appendix The black dashed line is K = 0.015. The two measures yield almost identical patterns.
D.2 An Illustration Example of the Flexibility for Choices of a and K.
There exists some trade-off of stopping time and two criteria. fact, II-COS could result short stopping time when only one criterion is considered. Typically, one could choose K +oo for the case the interactive constraint is out work and only individual criterion control considered, and meanwhile one can set = 1 with which the interactive constraint is the only concern. The results in Table evaluated this conclusion where C₁ is the FSR and C2 is the ES.
18

---

Table 3: Results flexible choice of a and the classification example. Average values among 500 repetitions: "SR(6"""). ES(8"m) and stopping time Tm-
a, K
(i(SR)
C2(ES)
Tm
0.1, +.00
0.101
0,073
446.51
0.1.K 0.045
0.102
0.044
628.82
1, = 0.045
0.849
0.038
107.79
Table 4: Average number of selected samples when stopping for II-COS, L.ND,, SAFFRON and ADDIS.
Classification
Regression
Real
II-COS
LOND
SAFFRON
ADDIS
II-COS
LOND
SAFFRON
ADDIS
500
100
0.20
0.70
100
100
0.20
21.82
24.05
1000
100
0.72
10.40
100
100
0.87
56.72
59.20
1500
100
11.40
36.33
100
100
12.28
74.70
80.05
2000
100
44.20
50.79
100
100
40.99
80.03
82.02
2500
100
68.33
60.95
100
100
59.72
92.01
91.04
D.3
Effects of the Calibration Size
We perform some additional simulations under the classification setting, study the stopping early issue for those methods based on the conformal p-values.
Take 0.1 and K 0.045 for FSR and mES, respectively. In Table and m = 100 and vary Riaal from 500 to 2,500 to compare the average numbers of selected samples of these three methods with II-COS until stopping. It is clear that all the three benchmarks are unable to select enough samples across all the settings, especially with a small calibration set. As the calibration size n1 increases, their selected numbers tend to be close to the target. This can be understood because larger calibration size would generally yield more accurate detection of the individual of interest and thus alleviate the alpha-death issue some extent. In contrast, the performances of II-COS, in terms the number of selected samples until stopping, would be much less influenced by the size of calibration data. The II-COS only stops when samples obtained under all the scenarios.
first fix fhr 1, 000
Furthermore, to verify that II-COS guarantee both FSR(OTT) and ES(OTT)) control with relatively small calibration size neat (such as 200), we apply II-COS in synthetic data. The details of the data generation process be found in Section and D.4
We fix fhr = 1000, m = 100 and Aeat varies from 200 to 800. The average of FO and ES( T with II-COS for both scenarios are calculated from 500 replications. The results are summarized as Table 5 It's obvious that for different Real. both FSR(6"T) and ES( controlled under the pre-specified constant a and K respectively under all scenarios with our II-COS method.
Table 5: Average values of FSR(67m)(%)) and (xx00=) (with standard errors in parentheses) for II-COS with different nea under classification setting (a 10%, K 4.50 10-2) and regression setting (a 10%, 1.50 x 10 3)
Classification
Regression
Pall
FSR (Tm)
ES (T,m)
FSR (T...
ES (T,.)
200
9.22(0.73)
4.41(0.08)
5.74(0.18)
1.49(0.03)
400
9.22(0.69)
4.42(0.07)
4.84(0.19)
1.49(0.04)
600
9.47(0.80)
(.39().09)
5.18(0.19)
1.49(0.04)
800
9.36(0.69)
4.40(0.08)
5.79(0.19)
1.49(0.04)
19

---

D.4 Results on Synthetic Data for Regression Setting
The following regression setting considered: - 5exp X2 10(X; xx)² + with X ~ (0, I,) and (V 1). The informative set is A (c, o0), where is the 80% quantile of Y. The FSR and mES are considered as C1 and C22 respectively. The prediction algorithm H is taken as neural network, with single hidden layer and 10 hidden neurons, implemented by R package nnet, and K chosen as 0.015.
The simulation results are summarized in Figure TigureBaandg a from 500 replications. The results are similar those for classification setting. It's further verified that our proposed II-COS outperforms all the benchmarks under both classification and regression scenarios.
0.20
0.04
0.15
(FSR)
0.03
0.10
0.02
0.05
0.01
0.00
100
200
300
100
200
300
Time
Time
Method
II-COS
SAST
LOND
SAFFRON
ADDIS
Figure 8: Simulation results for regression setting. Line charts of FSR((") (Left) and ES(8") (Right) for II-OS,, SAST, LOND, SAFFRON and ADDIS over time t. The black dashed lines are the corresponding FSR level a 0.1 and the ES level K 0.015. Shading represents error bars of one standard error above and below.
0.12
0.20
1500
0.09
0.15
(FSR)
1000
0.10
0.08
500
0.06
0.00
0.00
0.
11-C08
SAST
LOND
CAPPRAA
ADD9S
11-C08
SAST
LOND
SAMAON
ADDIS
11-006
SAST
LOND
ADDA
Figure Simulation results for regression setting. Boxplots FSR(6 (Left), ES(6"M) (Middle)
and Tm (Right) for II-COS, SAST, LOND, SAFFRON and ADDIS. The black dashed lines are the corresponding FSR level a 0.1 and ES level K 0.015.
D.5 Experiments for Other Individual and Interactive Constrains.
Other individual constraints To better illustrate the performance of our proposed method, we also conduct experiment for the general individual constraint. We choose Gx E Xzl. G (X) = ا / for EC (expected cost), and choose Go(X) = 1, G (X) 0 for FSR. Other settings the only the proposed II-COD can guarantee all EC (expected cost), FSR and ES control, while all the benchmarks out of control for EC or ES.
the classification model. The results shown in Figure TO We
that
Other pairwise function g. Besides the RBF kernel, popular choices include the cosine similarity 47 (52) with adjustment g(X, X') = X X' [[|X] XX/X[[[]X]
2 X X''l2) + 1, and the absolute value of cosine similarity, i.c., (X, X') X / X (2) which characterizes the orthogonality between
20

---

0.15
15+
0.30-
(Fc)
1.0-
(
0.5
0.24
0.05:
0.0
0.0
R-C05
SA57
LOND
SAPPRON
ADDIS
RR-C0S
SAST
LOND
SAFFRON
ADDIS
II-C0S
SAST
LOND
SA ADDIS
Figure 10: Boxplots BC(8"m) mm) FS (m) and (S for II-COS, SAST, LOND, SAFFRON and ADDIS under classification model. The black dashed lines indicate corresponding nominal levels an 0.5, a2 = 0.2, K 0.045.
X and X' and is often considered in the field of design of experiments 15 The similarity functions mentioned above all satisfy Assumption 3.2 As supplement the experiments the main text, we choose the cosine similarity with adjustment g(X, X') X X' setting. Notice that the original cosine similarity
/ [ | + for classification can be negative sometimes. and does not change the final results. The simulation results are summarized in Figure 11 from 500 replications. As can be seen, the simulation
X
|X|| |XX
Hence we add constant which guarantees g(X, (X,X')
results similar those when choosing the RBF kernel in Section
4.1
1.10
800
1.05
400-
200-
I&COS
SAST
LOND
SARRNN
ADDIS
SAS7
LOND
SAFPRON
ADDIS
11-00
SAST
LOND
SA SAPRON
ADDIS
Mothod
Method
Method
Figure 11: Boxplots of FSR(Tm)) FS(Tm) and Tm with random forest algorithms under classification setting for II-OS,, SAST, LOND, SAFFRON and ADDIS. The similarity function g chosen as the cosine similarity with adjustment (XX X') X xX|[[|]|||||] | + 1. The black dashed lines are the corresponding FSR
level a 0.1 and the ES level K 1.0.
D.6 Additional Results for Real Data Analysis
In Table the average proportions of handicapped and female in the selected candidates are in the last two columns of (a). The average proportions of minority and female among the correctly selected individuals are in the last two columns of (b).
D.7 Additional Experiments for Different Learning Algorithms
Except for the random forest algorithm, we further apply two different learning algorithms for classification setting to estimate the model f(X):: Support vector machine (SVM) and Neural network (NN) with a single hidden layer and 10 hidden neurons, which are implemented by R packages kernlab and nnet, respectively. For NN algorithm, the entropy fitting is used for classification setting. The empirical FSR(T.) and ES(T..) levels are estimated by the average the false selection proportion and the expected similarity respectively from 500 replications. The results are summarized in Figure T2 random forest in Section 4.1
can be seen, the simulation results are similar as those when applying
21

---

Table 6: Average values with candidate dataset and income dataset: FSR(6"m), (S(("m (x10-9) and stopping time Tm- The average proportions of handicapped and female in the selected candidates are in the last two columns of (a). The average proportions of minority and female among the correctly selected individuals in the last columns of (b). The target FSR level is 0.2 for both. For the candidate data, the target ES level
x 10-3; 3; For the income data,
10-3
(a) Candidate dataset
22
(b) Income dataset
g
Method
FSR
ES
T'm
Handicap
Female
Method
FSR
ES
Tm
Minority
Female
TFCOS
0.19
0.98
2227
0.15
0.48
TECOS
0.16
5.56
2760
0.05
0.09
SAST
0.19
8.73
310
0.05
0.47
SAST
0.19
30.90
1200
0.03
0.05
CP
0.16
10.34
277
0.04
0.46
CP
0.42
18.84
283
0.06
0.13
NN
SVM
0.50
0.40
0.20
0.10
10-005
SAST
LOND
SAPRRON
ADDIS
B-COs
SAST
LOND
SAFFRON
ADDIS
Mothod
0.06-
0.00
B-Cn5
SAST
LOND
SAPPRON
ADDIS
p-cos
BAST
LOND
SAPROON
ADDIS
Method
Figure 12: Boxplots of FSR(Tm) and ES(T.) with different learning algorithms under classification setting IICOS,, SAST, LOND, SAFFRON and ADDIS. The black dashed lines are the corresponding FSR level a 0.1 and the ES level K = 0.045.
D.8 Experiments for the comparison with oracle procedure
Regarding efficiency, we conducted an experiment to compare the effectiveness of IICOS with an oracle method possessing knowledge of true state 0,. This can be succinctly formulated as follows. At time I, 6, if 0, 1 and PC(6 Tm //PS(8 (T < K,the process halts when E 6; m. To ensure fair comparison. we fix a 0.01 for II-COS and choose K = 0.045 for both methods. The results are shown Table The FSR of the oracle procedure is as expected. The T., of II-COS very close to the oracle. This close proximity indicates the high efficiency of II-COS. Both FSR and ES are effectively controlled by II-COS.
D.9 Experiments for the Extended II-COS under Varying Proportion Case
Let N be a prespecified large integer denoting the number of samples arriving sequentially. Denote the size of available labeled history dataset 'D. We fix N = 5000, 5000. Consider three different varying patterns for n.
Table 7: FSR(8"m), ES(8TT) 10 10 and Tm of the II-COS the oracle proce- under classification model.
1. Blocks patterm: π. 0.3, for E [1, 100
[501, 600] U [1001, 1100] U [000
CCcC
Method
FSR
ES
Tm
22001, 2100]
U [2501, 2600] U [3001,3100]
I1-COS
0.0097
4.41
686.245
Oracle
4.41
681.665
33501, 4000];
m [1101, 1500)
0.2 for / [101,
(601, 1000]
[2101, 2500]
26601, 3000]
U [3101.3500] U 5000).
22

---

2. Linear pattern: Vary Ti linearly from 0 to 0.5,
Hy 0.5t/N.
3. Sine pattern: Te {sin(8rl/N) 13/4, m
ranges between and 0.5.
The other settings are the same as those in classification setting. For the historical dataset, we fix Te = 0.2.
In simulation generate 500 data points prior to = 1 to form initial proportion estimate. The varying proportion estimates are updated every 200 time points. The bandwidth parameter b for estimating " chosen based normal reference rule. We set b 38. The window size q chosen by a rule of thumb in practice. Let Cbb, where C' is a fixed constant between 10 and 20. In our simulations, we fix q 500 and À 0.5. As for the density estimation, we first estimate fo and fi on the calibration data, then compute estimate T, by T (lo3)) A 1. As for those methods based on conformal p-values, they do f((W,)
[o(w)( #.) fi (w) Thus accordingly, we
not need modification since they only require the exchangeability of data in the null hypothesis. The
varying proportion case does not violate such a condition.
The simulation results are shown in Figures 15 from which we can observe that the extended version of II-COS performs better than the benchmarks under all three different varying settings for both C (FSR) and (ES) control.
0.25
5000
0.15
A15
3000
2
0.10
0.10
2000
0.05
0.05
1000
800
B-COS
S457
LOND
SASPOON
ADDIS
8-008
S457
LOND
SAPPOOS
Anni
liC0s
LOND
PPRON
ABDIS
St
Metbod
Mathod
Method
Figure 13: Boxplots of ('SR(8 E and Tim for II-COS, SAST, LOND, SAFFRON and ADDIS (Blocks pattern.. The black dashed lines indicate the corresponding nominal levels.
0.15
4000
0.775
2
2
0.10
3000
3.25
1000
84005
SAST
LOND
SAPRRON
ADDI5
ILCOS
SA57
LOND
SAPRRON
ADDIS
B4C08
S457
LOND
SAPROON
ADDIS
Method
Mathod
Mathod
Figure 14: Boxplots of FS
E and T., for II-COS, SAST, LOND, SAFFRON and ADDIS (Linear pattem). The black dashed lines indicate the corresponding nominal levels.
23

---

220-
011
1000
015-
3000
0...............................
2000
305
000
1000
8-C08
SAET
LOND
SAPPRON
ADDIS
B400
SAST
LOND
SAPPRON
Annis
B-CO8
LOND
SAPRRON
ADDIS
Method
Mathod
Mathod
Figure 15: Boxplots of F Tm
and Tm for II-COS. SAST, LOND, SAFFRON and ADDIS (Sine pattern). The black dashed lines indicate the corresponding nominal levels.
ES Fm
E
Proofs
E.1 Auxiliary Lemmas
The following results include a standard uniform bound for kernel density estimator 36 20 and simple corollary from the central limit theorem
Lemma E.1. If Assumption 3.1 hold and we take the bandwidth h Th1 -1/(28+1) then with probabil- ity at least - 1/n1
sup ff(w) - f(w)] s Ding T8t Vlggn n1:
WER
where D1 = (M, Ca, K)) is a positive constant depending on M and B, og of Hölder continuity and the kernel K K-))
Lemma E.2. The estimation satisfies
n|
(1 T)
with probability ni for any constant < < 1/2
Next lemma characterizes the uniform convergence of i.))).
Lemma E.3 (Uniform convergence of L(w)). Suppose Assumption 3.1 holds. Taking the bandwidth of kernel density estimator as h n1 -2241)) then we have
sup
[(w) L(w) < D2n
log nig
wER
for some positive constant D2 D2(M, Di,e,m) with probability 1 -2/m 1/n1/3 1/3
Proof.
24

---

Note that T(w)- =1 -1- folm) [(w]
(
and
(1 7) Fo(w) L(w))
(1- (1=))f((w)(())) - (1
(1- (1-
sup [(w] L(w)) WER
sup
sup wER
wER
f(w)
f(w)f(w)
ssup WER f(w)((w) {11 (1- (l--))((w)llll(o - fo(w]] + +1- (1-
f(w))
[Folw) So(w)] + So(w)) T -
[(w) f(w)
< sup wER
sup WEB
f(w)
f(w)
1
[Dim T
Vlogn1 + My MT - -T)n 1+
T + D2 n1
SUFT
/og m1
WE f(w)
(
- 7 (Din 23-1
Vlogn1 + M W 7 + Dr n
उनकт }og
(158)
S D2nj
2F41
/log n11
where Dj = (D,, is a positive constant depending on Dj and π because To is estimated by
about n2 (1 π) samples. The (i) follows the results directly in Lemma | and Lemma
E.2 and and
the bounds of f (w) and fo(w) in Assumption (1) with probability 1 2/m1 1/n 27 The (ii) holds since
f(w)= 2 f(w)- Dini Vlog 2 l - ( 3641 Vlogn1 > e/2
for sufficiently large n1. Due to the fact 3 < 1 and taking 7 1/6 in Lemma E.2 the (iii) holds with D2 = 2 + (1 -x) x) + E.. Hence we verify the conclusion.
The next lemma shows the property of data-driven II-COS procedure which is useful to give the bound of the mES gap.
Lemma F.4. Suppose 5' is the decision result at time t. Tg by the data-driven I1-COS procedure with Eise L,5,11 Eisr d, S a', where T3 inf(t: Ei= = and Then we have
E(
ΣΣ 6,6,)
15iCjst
5 (1
2s a')
E ΣΣ 8,5,(1 L.)( -,,} ISKISS
Proof. It suffices to show
ΣΣ
-
2s
a') ΣΣ 8,8g.
5 5-1 ISiciSt
ISKjS
By the selection procedure, we have
Σ
8
(9)
Squaring both sides of (9) and making some decomposition, we have
ΣΣ 8,6, - - Ż,) 21 a') ΣΣ ISisyS!
8rd; + 2
Σ (1-- -1
(1 L.)2,6.
ISiRJS!
Notice that
8
ΣΣ 8.0
1=1
ISiSJS
25

---

We have
(1 (1-d')² ΣΣ
6:dg +
2 - (1-- - (1-Ta,)²0 To:)330s
15K3/50
21-ad ΣΣd
616 +
1
[(1 2²)² 136;
ISiEiS!
a a) ΣΣ 6,6g
2(-dd) ΣΣ
2
ISiSJS!
1SicySA
2(1 8-1
2s
a') ΣΣ 8,65
Isiejst
The next two lemmas will be used in the analysis of the extended II-COS for varying proportion case.
Lemma E.5. Assume that random variables Y1, Y2, have density functions bounded by constant
c > O, and other two random variables Ys. and Ya satisfy P( - Ysl I E) 2 1 Ç and
P(|22 Y2l S 2 1 where 0, 0, < 0 and 0. Then for all t 0.
P( > 1,
P.Y t. Y. - 20(e +) + + s).
Proof. Define the events & = [[V - Y3 & (1=2 Y4 S (). and & U Note that
PP + P/S -,8 P(Yi - - t,
and
P(Y2+e>t,6 < PPY (2 5 t,,2).
we have
-
s 1PV>>t?>>])PP(-->>>> - -(--=>>>).);;>+tt.
< "ffl=:/;..---..=>>>,"">>>>>>> (,,p::="))-....-t.tt-oottt-{----
+ -
< 2c) +3(++s))
the last inequality holds because the density function is bounded.
Lemma E.6. 1 S q. for some Ç > 1 and b is sufficiently large, the exponential weight ks(s) exp(-1sl/b) satisfies
E (s)(s) [ - (3)
I Cb
and
sko(s) Kb(())
Cyb,
where C. > 0 is constant determined by b.
Proof. Take Z,, = expf-1/b3 Notice that
Σ ko(s) expl exp(- (1/-b)) expl- 1)/6)
2 z1+1
1 - exp(- 1/bb
1 Z,
26

---

Thus for a constant C. > 0. we have
Σx (s(s)
- Z )( ( 1 Z3 -24b
(Z6 + +Z
1 {b())}
(4)
Cs(1 Zb)
(0)
Cob'
where (i) holds since b is sufficiently large such that 2 = expl-q/0) S exp(-b5-1) eliminated and Z₁ < The last inequality (it) holds by exp(x) 2 x
can be
By the same discussion, we have
33 5=1 (56(8)
00
- 2
S
87
(8)(s)
1 %b
11
Zb
23 - Z0
(1 200
Zb
1
I 5b Zb
< Cb(b ((b++)). 1).
The last inequality holds since exp(x}
< 1/(1 x) for < 1. For simplicity, use the same
notation C. to denote the constants.
E.2 Proof of Proposition 2.1
Proof. For the part of individual constraint control, note that L: is defined as
L = Pr(0, = 0 W;) = E.' [(1 0,) W.].
Define W (W1,... ...) )
The stopping time T measurable respect to W" The individual
constraint at time T satisfies
0.)Go +; (0g(((
C((8) E
(Eisr 8,) V1
E
E ( V Σ (00-0)))0))) [
i
E
(Re 1) (1--))) (I] (((
E( (L.G((X) -1- L.)
The (i) holds since X,'s 's are independent of each other. By construction of the selection rule, we have that at the stopping time T, ((R V 1)-' ( WX(,) (1 L, (X,))&, I It follows that (10h < a at random time T. By construction of the selection rule, we have for stopping time T,
ΣΣ X (1 L3) < K KIKIS
ΣΣ 5,6, - L,) Lÿ). (10)
ISKGET
Taking expectations on both sides of inequality (10) and by double expectation theorem, we finally obtain
E
ΣΣ 9(X, 9(X, (,0,,,00,,
< K E
ΣΣ 0,0,0,0,3.
ISKSST
ISKSST
27

---

It follows that for every time t 2 T2,
Ef ΣΣ 9(x;, X,),,60000 1sisiST
C2(6") = E (ii
K
E.3
Proof of Theorem 3.3
Proof. In the data-driven II-COS procedure, ó; is determined by the estimated IFDR I, and we have
Σ ((.G))(X,) + (1- (X,,( aa
R
Note that
C
= = B
1
Rr
= E (X,,))( + (1 (1x,)))))))) + + R.
E((I, L)),((X,) (L. (X.))8)))
R
sa+B R.
-ILL:-1GooLX + (L. (X.)6o))
iS!
suffices to bound the absolute value of the second term. We have
E
Σ((4 ((L
L.)))(X)) + (L. L.) (1 - Tx + Ph - [.I[G
R
ist
sup L1 - [xl]Go(Xx)) + sup [h. - L.llGi wER
R
WER
<2e D2n1
log n- 2m n
The last inequality follows from Lemma E.3 and max [ī(w) L(w)) I 1. Notice that ni <
-
n1
by 3 < 1. We have
(666' Sa Dn 5 flomı,
where D = 2eg D2 3.
E.4
Proof of Theorem 3.4
Proof. Notice that
E
ΣΣ xX,,, (,),,, (.)(1 - -Σ 9(X, X1) L.)(1 (1)(1 -1)1))))}}
IsisjS!
15i<j5t
<
Σ
9X,,, X,, 15,,5,50(1t - Lel+ +1,112 - -,₁1 + E. L,1 + L.g 1.ll] [L,l)
1siKitt
(5)
5 2eg
ΣΣ 8,6 (12, - Lal + -1,
ISijjE
(8)
< 20g Dn,
V log n1 E ΣΣ 815,).
(11)
28

---

The (i) holds by Assumption 3.2 and L(w), [(w) < 1 even when t is random, as the uniform
convergence of [ww). And (ii) follows from Lemma
E.3 By the similar arguments we have
E
Σ 1SiciSt
61 - (,)(1 E,)
5,6, - L.)(1 1
12Dm
log ni E
Σ 6,6,).
(12)
Denote An, = Dm1 TF
Vlog n1. Combining 11 and 12 it follows that for every time t 2 T.
E ΣΣ
(X,, X,),,,1 - L.)(1 - Li)}
C2(8") =
ISiSsSt
E
ΣΣ 81 (1)(1-))))
Isijist
E ΣΣ
g(x, (1 - -2)(1 (1)1111)) + Ac, An, E( ΣΣ 8,6,)
1515/ST
1515JSt
E
- L,)(1 [1)) ΣΣ -16 1sis/St
E(
ΣΣ 5,5,)
E
ΣΣ 6105(1 T.)( - -T,}}
(8)
ISisiSI
K + 20g Ans
X
E
ΣΣ
6,0,(1 - L1)1 - -b,)}
E
ΣΣ 1gioSt
6,0,(1 - (1
- [,,
ISKJJII
E
ΣΣ 5,05(1 (.)(1-
- {.)}}
(816)
2Cg An R1
K
+
2mn
E
ΣΣ 8505(1
(2)(1 - 15)]
B( ΣΣ 8,6,) 15is351
(iv)
K +
20g Any 2ma 1
1
x
1 24 m
8,6,(1 -L) (1) ISiラン
ΣΣ
20g Anj
2 m3
2ma'
1
2ma? m-1
The (i) follows from
(11) and (ii) comes from the operation of our algorithm, where
E EE 9X,, (1-)),,(1111111111--1 (1)1-1)))
K K.E. ΣΣ
6,0,(1 - (1)
151
The (vii) and (v) are directly from Lemma E.4 The (iv) holds due to 12 Thus we have
(K + cg) An.
C2( < K +
0.5
E.5 Proof of Corollary 3.5
Proof. Define the C₁ constraint conditional on W as
1
Σ ((L
L.) + (i. - Lx)G(
R. ist
By the proofs of Theorem
3.3 we have for any f
(88) S a
2eg sup [ī(ww) L(w))
weR
29

---

Hence at stopping time Tm, we have
(6(Tm) E Elsup Ci a + ssp [[(w] L(w)l] s a + An.
wER
By the proof of Theorem 3.4
can also drop off the expectation and replace Any with
SUPEER [[(w)- - L(w)). That
{1 2suPwER [Z(w) T(W)l
ΣΣ
9(X,
Lg))
2ma
m-1
Isicist
2Gg sup WE (T(ww) - T(w)l
k
ΣΣ ISKSE
6,0 - -,))
(13)
2mn
m-1
Hence at stopping time t = Tm.
13
still holds. Take expectations at both sides and we will get
(K + Cg An:
C2 K +
0.5
mn.
Notice that A,, Dn Vlogni converges to n2 + 0o and a < (1 1/m)/2 by the
condition. It follows that
lim Ci(5t)) a 81-000
and
lim C2(6"m) S K.
11-+00
E.6
Proof of Theorem
C.1
Proof. We first denote some notations. For notational simplicity, to denote constants in the following proofs.
Recall the conformal score function Q(X.). The conformal p-value is
1 1+ 1+E T(Q(W)) s Q(W)) Fnnz
pr
1+ [4]
where |4].
Denote the population p-value pt F(Q(W))). where F(.) is the distribution of Q(W,) condi- tional on 02 0. Denote Pr(O, = 1) and
Pr(p: A)
The corresponding local FDR is defined as
T. Sa(Ws)
L
f((We)
follows the definition of (5*) and (8)
C²(8") f R
Σ +1
ist
and
E{ ΣΣ (X, -2)0 IsicJSt
(8'
E ΣΣ 9Xx 8,)8,;
The estimated proportion is defined as
ンj Kolj - 1jj
30

---

And denote the ideal estimated proportion via population p-values
no ko - 1 t)1[p; > }}
मे = (1- [1 Hb(
where (s) = exp(-1s1/b) and is the size of a neighborhood for estimation before time t.
We first claim the following proposition and the proof is deferred in Section E.7
Proposition E.7. Suppose the assumptions in Theorem 3 hold. Then
{1 Vlog n
(bn)2] 1/2-7 nl
| و²1 S eemax
(14)
18'
27 with probability max { 1/b, Vlog ui / {
where e is a constant and 0 < < 1/2.
Notice that
sup (w) f(w)]
sup [o(w)( म.) + (w)). fo(w)( T() - (w()TT
sup ffo(w) fo(w)) + sup [w) f((w] - 2M ml.
The last inequality holds since म, [0, 1] and (w) and fi (w) are upper bounded by M.
Thus by Lemma E.1 and take in Proposition E.7 at 1/6, with probability 1
1/3
max {1/b, /logn1/n (bn)²²
we have
sup ( -f'(0) en
गo Vlogm ni emax {
b-b log (lo
n2 ) (bb))
By the procedure of Lemma E.3 directly, with probability 1 cmax 1/b,, vlog ni / (bn)) 1/3
we have
sup [[((w) (w)) s enn उ Vlogn1 + emax {o ( log nogni )² (bn)
w
71
It follows that
C8 S a en क. Vlog n + 200G max { (b- log ni
_/ (bn) 3
n1
Denote A, = 2
J841
Vog ni + emax fb= ( (logny
(bn) }. By our assumption, by
o(1). So
lim A = 0.
b +00
By the additional assumption that Pr(pe > 1 1) 0, we have n.. nt Thus Hence the first
part of Theorem 3 is completed.
At last, we can use the same procedure to prove that
(K Ag
CA S K +
0.5 mar
A
Since T². TT. C8' C2(8") and the results directly follow.
31

---

E.7
Proof of Proposition E.7
Proof. Notice that
Pr( - n. E)
S Pr (
#
+ Pr ( x
2
2
We discuss for the two parts respectively. For the first term of the above inequality, it suffices to show the upper bound of the second moment of ( क.. |.
For convenience, denote Uj T(p; > A). and V3 Tm X). It follows
B - नマ/
10j=in mo65 - 11) 2)2 EF
Σ l---Q
Holi ()((U -V3)
- - - V3)88
Ei,JEN, kell - t) - t)Bi(U,- - VOUU, -V))
()) +
- Kbll
Kobl
(2j Ko - +1) -1)2
Σ
k t) Pr(p; > A) + Pr(pj >>) - Pr(pp > X. > > 21}
EEj= 2-1
Hibli - t)) - 22 j-t-q
Σ kolt t)ma(( - t) { Pr(p. > X.p > A) - PPr > X. > A) + Pr(p > X. > {111 L-g
+2
Now we check the upper bound of [p, - pyl- We can rewrite them as Pj = F()(W;)) and Pj = Fng (Q(W)). Even though n2 is a random variable, due to the two-group model, Fng (-) is still an empirical distribution function composed by i.i.d samples conditional on {0. where C is the index set of calibration set. Thus by DKW inequality, for any E > 0
Pr (lp p,1 S E1) = EE Pr [D3 - ر0 S E1 (0:)
sup F(:) Fnz (2) Se (0shec) ]
< E
Pr
< E 2 (
5 1 2 expp( ((1 - m)m (1-1)m11533
- 4T/ (ni })
The last inequality holds since n22 2 (1 - x /2 with probability 4 4 (1 - m)}-
Thus by Lemma E.5 and the fact that P; has bounded density function, we obtain
[Pr(p > A) Pr(p, > A) < 2s + 6exp(- -(1 - =)m + 12T / (ni -1-T))
Take 5 Vlog / [ni }. for sufficient large n1 such that /n1 log 2 6 GT we have
2 2ylog ni
6 6 12 - 1))
3ylog mj
[Pr(p > A) - Pr > A) <
+
< (1 π 1/2
(1 π 1/2
71
Again by Lemma E.5 and the same E₁, for any i,j E (t q.....t t 1) = =N,(t), we have
n1 T) < 6vlog n1
24m
[Pi(p > A,p; A) Pr(o, X, x)k - 4s +2 (1-
(1 T m) 1/2"
Therefore by Lemma E.6 we have
9C 9Cov/log 11
12 12 n1
E(#) - #/1) <
bm
1/2 (1 (11+1--111-1 - X>2
"1 1/2 (10111=)11111 -1
32

---

So by definition and Markov' inequality, we have
4E( - =;
Pr
E²
36Cb V00bvloogn
48 Aaollogo0 n
bm 1/2(1 (1 a1 E)
1 n/ (1 r) E)
For the second part, we have
Pr [
2)
Pr
Pr(p: > 1)
Rib -ol t) > }}
1-1 x
(Ej=ing Koli t)) -)
2
4E
Ej koli - (Pr(p > A) - Hpp > A)}
2
(E/ Ko - +))222(1-111 - A)2
1z ng(j 1) Var [Tp > 1)} + (Pr(p) > ^) Pr(p: > 2))2
{> j 1 ko - ()))23()1111))
Co
1Σ j= R2 - (r(pp) > - -Pr(pe > A)}2
6Ea(1 X)2
(Ej=ig k - 22) A)2
C
16n2² -) (t) t)
6Ea(1 X)K
1Σ kg - 2E) A)2
1
Co
16(C)bn)2
(15)
bE - X)2
(Σj ~0 - 1))22221 X)2
The (i) holds since Var[I(p, > })} < 1/4. As for (it). consider A₂ = {w: F(((w)) > A}. Then by the two-group model we have Pr(pt > A) AA (.fo(w)(1 x fi(w) Te dd.. Hence
Pr(p, > A) - Pr(pt > 1)y = SA, {fo(w) F. - Tel + fi(ww) ग. TzI jdw s 2] - t/n
by our assumptions. And (iit) is directly from the second part of Lemma E.6 Above all, finally conclude that for all 0 and sufficiently large n1, we have
C
36Cb Vlog n
48 /8gg n1
16(C,,n)²
Pr([ म.
be - 2)2
/2
T) 2)282 ni
1 T) (1 2)2-2
e²(1 - X)2
11 Vlogn (bn))
2] c
< max
1/2
g2
n
for some constant c > 0.
Thus take & = max { 1/b, vog n /n///// (bn)) }
1/2-y
Ve and and we obtain
1 b''
Vlog n
(bm) 1/2-7
b I cmax
1/22
27
with probability 1 - max (1/b, Vlog (bn)²] the proof.
where 0 < 7 1/2. Hence complete
33

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See the end of Sect. [ we summarize our main contributions. The experimental
results in Sect. and Appendix D validate the theoretical results in Sect. 3 and Appendix C
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Sect. 5 we discuss the limitations of our II-COS algorithm. In Sect. 3 we have explained why our assumptions are weak piece by piece. And see Subsect, 2.2 for details about computational complexity.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: For the theory assumptions, see Assumption 3.1 Assumption 3.2 in Sect. For the proofs of each theoretical result, Appendix Elnn the Supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
See Appendix B for the implementation details to reproduce our experimental
results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
35

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the data and code the supplemental materials, including sufficient instructions in the zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/ccpppbllcgggudeeeeeeeeeeee Coaee///ddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Sent Appendix provide the experimental setting/details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We calculate the standard error across 500 replications and show the error bars in experimental results, which is calculated by mean +/
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
36

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix B.4 for the compute resources and more details about the execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp//www.p....poopcccccccccaaa
Answer: [Yes]
Justification: All the authors have reviewed the NeurIPS Code of Ethics guidelines and ensured that our paper conforms to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the broader impacts of our work in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
37

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification; We have cited and provided the URL of the dataset we use in the paper, see Sect. and the references.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
38

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
39


Reviews:
[{"review_id": "Tm08iV5Net", "reviewer": "Reviewer_78XN", "summary": "This paper introduces a framework to perform online sample selection such that the unseen outcomes are in specific target range while also optimizing for constraints like diversity that are dependent on the input covariates. The additional constraint involving input covariates can help ensure properties like the diversity of candidates when selecting individuals for interviews while also guaranteeing that most of the interviewed individuals accept the offer. The paper proposes a data-driven procedure to select the subset of candidates in an online fashion by implementing the proposed algorithm. Under reasonably weak assumptions, the paper provides theoretical guarantees on satisfying both the above constraints in online sample selection. The experiments confirm that this framework ensures low false selection rates (i.e. unseen outcomes are in a specific target range) while optimizing for the additional covariate-dependent constraints like diversity on synthetic and real data.", "questions": "See weaknesses", "limitations": "Yes, limitations and broader impacts are discussed in the last section of paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an intuitive way to incorporate covariate-dependent constraints like the diversity of candidates when performing online sample selection to optimize for metrics like false selection rates. This paper solves an important problem in online sample selection and demonstrates that the proposed method improves the covariate-dependent objective while maintaining comparable performance on false selection rates.", "weaknesses": "It would be interesting to understand the gaps between an ideal diversity profile and the profile obtained by the proposed method in Fig 3. Analysing the gap w.r.t changing g(X_i, X_j) function choice could be helpful. Would it be helpful to increase the weight of the g(X_i, X_j) term to reduce this gap and understand its implications on the satisfaction of individual constraints?\n\nIt is evident that the SAST baseline outperforms the proposed method in terms of FSR sometimes, which is understandable given there; 's a tradeoff with the interactive constraints (Table 2b, Fig 1).  It would be helpful to learn if we can reduce the gap between SAST and the proposed method by balancing the tradeoff (perhaps using a tunable hyperparameter that balances the two constraints?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: It would be interesting to understand the gaps between an ideal diversity profile and the profile obtained by the proposed method in Fig 3. Analysing the gap w.r.t changing g(X_i, X_j) function choice could be helpful. Would it be helpful to increase the weight of the g(X_i, X_j) term to reduce this gap and understand its implications on the satisfaction of individual constraints?\n\n\n**To W1**: Thank you for your thoughtful suggesstions! We'd like to make some explanations as follows:\n\n- First, your suggestion is very reasonable, especially regarding our focus on specific groups such as Bachelor's degrees. As the categorical variable is transformed into one-hot coding to numerically compute the diversity, we can assign greater weights to these specific groups (one-hot variables) of interest. This adjustment allows us to enhance their contribution to achieving diversity.\n- Second, defining an ideal diversity profile precisely is challenging. Since changing g affects how we define our target diversity criterion. From our perspective, the ideal diversity profile can be defined as achieving maximal diversity in the offline setting while selecting the same number of units and controlling the FSR. A related work is [1].  In this aspect, the gap primarily arises due to differences in the online procedure and cannot be easily analyzed for its relationship with g.\n- Third, we conducted an experiment on candidate data to demonstrate the diversity performance using our II-COS method, the offline oracle, and your innovative weighting approach, respectively. Here we assign more weight on Bachelor degree (five times than other education variable). We fix $\\alpha=0.2, K=0.001$. The education status composition of the correctly selected samples and FSR for three methods are shown in the table below. We can observe that this weighting scheme effectively increased the proportion of Bachelor's degrees, aligning closer to the distribution observed in the offline oracle. However, it also resulted in a decrease proportion in Master's degrees, which is less desirable. Therefore, determining an optimal weighting scheme to achieve the ideal diversity profile remains an interesting question for future investigation.\n- As for the implications on individual constraints, we find no significant influence empirically when adjusting the weight of g, since the FSR control of our strategy is quite tight. \n\n Hope these explainations are acceptable for you!\n\n|  | FSR | No Qual | High School |  Matriculation   |  Bachelor  | Master | \n| --- | --- | --- | --- | --- | --- | --- |\n| Oracle |  0.19 | 0.03 | 0.34 | 0.26 | 0.27 | 0.08 |\n| II-COS | 0.19 | 0.03 | 0.44 | 0.26 | 0.21 | 0.07 | \n| More weight on Bachelor | 0.19 | 0.02 | 0.39 | 0.29 | 0.28 | 0.02 | \n\n\nReference:\n[1] Wu, X., Huo, Y., Ren, H., and Zou, C. Optimal subsampling via predictive inference. JASA, 2023.\n\n\n> W2: It is evident that the SAST baseline outperforms the proposed method in terms of FSR sometimes, which is understandable given there 's a tradeoff with the interactive constraints (Table 2b, Fig 1). It would be helpful to learn if we can reduce the gap between SAST and the proposed method by balancing the tradeoff (perhaps using a tunable hyperparameter that balances the two constraints?).\n\n\n**To W2**:  Thank you for your questions! Your observation is very insightful, and in fact, our proposed method already can achieve the trade-off you mentioned.\n\n- First, our method itself provides a sufficiently flexible framework that allows for balancing this trade-off by adjusting the parameters $\\alpha$ and K as needed. Typically, one could choose $K=+\\infty$ for the case the interactive constraint is out of work and only individual constraint control is considered, and meanwhile one can set $\\alpha=1$ with which the interactive constraint is the only concern. The results in Table 2 (Appendix E.2) evaluated this conclusion.\n- Second, as mentioned in Section 2.2 (lines 169-171) of the paper, SAST can be regarded as a special case of our proposed method. When the individual constraint is set to FSR and K is set to $+\\infty$ , our method reduces to the same manner as the controlling step of the SAST.\n- Therefore, additional tuning parameters are not specifically required to achieve this.\n\nWe hope that this interpretation can ease your doubts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JNANAXMwOF", "reviewer": "Reviewer_MMvB", "summary": "The paper studies online sample selection with individual and interaction \nconstraints simultaneously. Specifically, the goal is to control (variants) of \nthe false selection rate (FDR) and the expected similarity (ES) under the \nempirical bayes framework. Under distributional conditions, the proposed method \ncontrols the target quantities asymptotically. The method is evaluated on synthetic \nand real data.", "questions": "1. **The model.** In the motivating examples such as candidate selection, it appears to me \nthat we get to observe the ground truth, i.e., $Y_t$, after time step $t$.\nThis is briefly mentioned in the discussion section, but I think it is reasonable \nto use the observed $Y_t$'s to update the estimation.\n\n2. **The choice of interaction constraints.** It is not well-motivated why \nthe changing ES to (4) is reasonable. As illustrated in the simulation, \nalthough these two quantities seem to coincide as the time step goes \nto infinity, they do differ quite a lot with smaller time steps (this could happen \nwhen $m$ is small).\n\n3. What is the principle for choosing K in general? In particular, for the real data example, \nwhy is K taken to be $1\\times 10^{-3}$ in the fist example and $6\\times 10^{-3}$ in the second?\n\n4. In the numerical experiments, a fairer comparison with offline CP would be with [1], i.e., \nthresholding the p-values with BH-adjusted p-values as opposed to the fixed threshold. \n\n[1] Jin, Ying, and Emmanuel J. Candès. \"Selection by prediction with conformal p-values.\" Journal of Machine Learning Research 24.244 (2023): 1-41.", "limitations": "The authors have partically addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-presented and easy to follow.\n2. The problem under consideration is of interest and relevant.", "weaknesses": "Please see my comments in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: In motivating examples such as candidate selection, we get to observe the ground truth after time t. ... it is reasonable to use the observed $Y_t$'s to update the estimation.\n\n**To Q1**: Thank you for the constructive suggestion. We'd like to clarify a few points:\n\n1. **Feedback and Model Updates**: As you mentioned, it's of great interest to explore online sample selection with feedback (past ground truth Y), which is part of our ongoing work. However, using new observations can improve the accuracy of estimated models, but updating these models at every time t can be time-consuming and inefficient for some machine learning algorithms. Additionally, ensuring selection error rate control with feedback requires stability criteria on the model or learning algorithm. This falls into a different regime compared to our current setting. Our method, II-COS, is model-agnostic and compatible with commonly-used learning algorithms. This paper focuses on a general framework without feedback as the first effort in online selection considering general interactive and individual constraints control.\n2. **Unavailable Responses in Online Settings**: In many cases, the response is not available in online settings. For instance, in online anomaly detection [1], we cannot access the outlier label throughout the procedure. Responses can also be delayed, effectively making ground truth unavailable within the restricted time of the online process. Candidate selection is a special case of this, as identification occurs after resumes have been passed for a period. Hence, our model is suitable for these scenarios.\n\nWe hope this clarification addresses your concerns about our model.\n\nRef: [1] Gang, B., Sun, W., Wang, W. Structure-adaptive sequential testing for online false discovery rate control. JASA, 2023.  \n\n> Q2: It is not well-motivated why the changing ES to (4) is reasonable....\n\n**To Q2**: Thanks for your question. We would like to make some explanations.\n\n- First,  the motivation for using mES instead of ES is the technical difficulty of directly dealing with the expectation of a ratio. The mES is a ratio of expectations, which is easier to control and also serves as a reasonable measure of similarity.\n- In fact, mES and ES are inspired by the mFDR (modified false discovery rate)  and FDR in the field of online multiple testing, where mFDR is a ratio of expectations and FDR is the expectations of a ratio. The mFDR is usually employed as a replacement for FDR and is shown asymptotically equivalent to FDR [1]. Similar techniques also can be applied for the asymptotic equivalence for mES and ES.\n- In numerical studies, we see that empirically the two similarity measures yield almost identical patterns when there are sufficient samples. An illustrative example can be found in Figure 6. Besides, in Sec. 4.1, we have shown the online ES values agains time t in Figure 1(right), from which we can see that our II-COS can guarantee valid online ES control even for small values of t.  Therefore, when m is small, our method is also empirically valid for controlling ES.\n\nRef:  [1] Sun W, Cai T T. , Oracle and adaptive compound decision rules for false discovery rate control, JASA, 2007.\n\n> Q3: What is the principle for choosing K ? \n\n**To Q3**: In Appen C.1 (line 512), we provided implementation guidelines for choosing K. To add more clarity, we'll explain it here.\n\n- For any two i.i.d. observations $X$ and $X'$with corresponding $\\theta$ and , the expected $C_2$ of the individuals of interest is given by  ${C_2}=E[g(X,X')\\mid\\theta=1,\\theta'=1]$, which can be estimated by $\\widehat{C_2}=\\sum\\sum_{i<j; i,j\\in\\mathcal{L}} g({X}_i,{X}_j)/\\{|\\mathcal{L}|(|\\mathcal{L}|-1)\\}$, $\\mathcal{L}=\\\\{i:{Y}_i\\in\\mathcal{A}\\\\}$.\n- We then set $K=a\\widehat{C_2}$, where $a>0$ is user-specific to control interactive constraint level. Our numerical evidence shows that $a\\in (0.1,0.5)$ works generally well. And we set a=0.4 in our experiments.\n- For real data examples, since $\\widehat{C}_2$ differs between the datasets, we set the same parameter $a=0.4$, leading to different K values for each application. Both choices are reasonable.\n\n> Q4: In experiments, a fairer comparison with offline CP would be with [1], i.e., thresholding the p-values with BH-adjusted p-values as opposed to fixed threshold. \n\n**To Q4**: Thank you for your suggestion. We considered multiple factors when selecting the baselines and chose several that we believe are reasonable. Here are some clarifications:\n\n1. **Conformal-p-values-based baselines**: We have compared several online FDR control methods using conformal p-values in [1], including LOND, SAFFRON, and ADDIS (see Figures 1-2). Notably, LOND is exactly the online version of the BH procedure, and SAFFRON is the online version of the Storey-BH procedure. More details can be found in [2] and [3]. As mentioned in line 298, for real-data experiments, online multiple testing methods based on conformal p-values resulted in few selected individuals. Therefore, we focused on comparing II-COS with SAST, and omitted results for conformal-p-value-based methods.\n2. **Online case consideration**: The BH procedure itself cannot be applied in online scenarios as it requires all p-values to be sorted before making a decision, which is not possible when future p-values are unknown. Thus, we compared our method with the fixed threshold method in real-data examples (see Table 1 and Figure 3), as it can be executed in online setting and serves as one of our baselines.\n\nWe hope these clarifications address your concerns. We would appreciate it if you could re-evaluate our work.\n\nRefs:\n[1] Jin, Y. and Candes, E. J. Selection by prediction with conformal p-values. JMLR, 2023.\n\n[2] Ramdas, A., Jordan, M. I. et al., Online control of the false discovery rate with decaying memory. NeurIPS, 2017.\n\n[3] Ramdas, A.,  Jordan, M. I. et al., SAFFRON: an adaptive algorithm for online control of the false discovery rate. ICML, 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8rr4Y83nIq", "reviewer": "Reviewer_vGU3", "summary": "In this paper, the authors quantify the uncertainty of response predictions using predictive inference; and systematically addressing individual and interactive constraints in a sequential manner.  An online selection rule is developed to ensure the above two types of constraints are under control at pre-specified levels simultaneously.  Simulated and real-data examples  are used to evaluate and illustrate their approach in terms of both online individual and interactive criteria control.", "questions": "Section 4.1 would be good to tell reader how many replications are used.", "limitations": "Comments are needed on the practicality of \\hat\\mu being a bijection.  How critical is this assumption to the method and the theoretical analysis.  If critical then this is a limitation.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a nicely, clearly written paper that develops an online selection rule that is simple yet effective.  The simplicity of the online selection rule will enhance the potential for  this rule to be used in real life.   The authors’ claims are well supported both via theory, simulations and application to real data. The paper along with the appendix provides detailed information that allow for replicability.  Very nice!  I particularly appreciate the comparison with the approaches based on conformal p-values.", "weaknesses": "See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q: Section 4.1 would be good to tell reader how many replications are used.\n\n**To Q**: We have mentioned '500 replications' in the begining of Section 4 (line 270) when introducing our evaluation measures. To avoid repetition, we did not restate this in Section 4.1.\n\n> L: Comments are needed on the practicality of \\hat\\mu being a bijection. How critical is this assumption to the method and the theoretical analysis. If critical then this is a limitation.\n\n\n**To L**: Thank you for your nice advice.  \n\n- The assumption is considerably mild and widely adopted for the identyification of each $X_t$ in the predictive inference framework. Please refer to [1] for similar assumptions. Per your suggestion, we will add a comment about this assumption in the final version. \n- Besides, even if this assumption fails, we can view the local FDR as a special weight to reflect the correctness of each selection. And our strategy can also provide desirable cost control and diversity exhibition empirically.  \n\nReference:\n[1] Wu, X., Huo, Y., Ren, H., and Zou, C. Optimal subsampling via predictive inference. JASA, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUZevzZyh1", "reviewer": "Reviewer_vWkU", "summary": "The paper proposes a method for online sample selection. The authors introduce the concepts of _individual_ and _interactive constraints_, and demonstrate theoretically and empirically that their method satisfies both.", "questions": "- Interactive constraint: Why is this defined only with respect to the correctly selected samples? In the case of, e.g., the diversity of selected samples, is the constraint not intended to represent the diversity of all selections?\n- Line 159: Should $R_t = \\left( \\sum_{i \\leq t} \\delta_i \\right) \\lor 1$?", "limitations": "The authors discuss limitations of their work.", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The problem seems important and the formulation and approach novel. The authors provide both theoretical guarantees and empirical evidence, in both synthetic and real-world applications, of the effectiveness of their approach. The mathematical formulation seems sound, and the assumptions and theoretical results are clearly stated.", "weaknesses": "I am not familiar with the FDR control literature, and had to read parts of the paper (specifically sections 2.2, 2.3 and 4) multiple times to get a gist for the logic of the method and its empirical perfomance. This is reflected in my confidence score. If the paper is accepted, I highly recommend the authors revise the paper to make it easier to follow. A flowchart to illustrate the steps of the algorithm and/or to illustrate the differences between the Oracle and Data-driven selection procedures may be helpful; a toy example could also help. I also suggest including a longer description and/or table of the benchmark methods against which the empirical performance of II-COS was compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: I am not familiar with the FDR control literature, and had to read parts of the paper (specifically sections 2.2, 2.3 and 4) multiple times to get a gist for the logic of the method and its empirical perfomance....I highly recommend the authors revise the paper to make it easier to follow. \n\n\n**To W1**: Thank you for your thoughtful suggestions, and we apologize for any confusion caused to readers unfamiliar with the related fields. \n\n- First, we indeed have a flowchart of the data-driven II-COS procedure in Figure 4 of Appendix C. Based on your suggestions, we will add an illustration of the oracle version to the flowchart and move it to the main text in a future version. We are also willing to provide a more detailed description of the benchmarks in the main text, include a toy example of our method, and revise the paper to make it easier to follow according to your helpful advice.\n- Second, due to the limit of main text pages , we have provided more details and related works about our proposed method in supplementary materials. Additionally, in Appendix B, we discuss related works on online FDR control and predictive inference, offering an overview of the literature on online multiple testing and related topics, which can be helpful for readers unfamiliar with these fields. \n\nAgain we greatly appreciate your valuable suggestions for improving the quality of our paper.\n\n> Q1: Interactive constraint: Why is this defined only with respect to the correctly selected samples? In the case of, e.g., the diversity of selected samples, is the constraint not intended to represent the diversity of all selections?\n\n\n**To Q1**: Nice question! We'd like to make the following explanations:\n\n- First, in our online selection problem, we are only concerned with the characteristics of the correctly selected samples of interest. In fact, the individual constraint (e.g., FSR) needs to be prioritized because we should select the samples that are of interest (i.e., $Y_t\\in\\mathcal{A}$). If the interactive constraint is applied to all selected samples, rather than just the correctly selected ones,  we may select many uninteresting samples far from the center of  $Y_t\\in\\mathcal{A}$ to achieve a low similarity. This is not desirable in practice. For example, hiring a obviously unsuitable candidate due to its diversity only.\n- Second, if in other problems and scenarios where constraints need to be applied to all selected samples (rather than just the correctly selected samples), our method can be easily modified to meet this requirement by replacing $1-\\widehat{L}$  in Equation (7) with 1 in the decision rule of II-COS.\n\nWe hope the above clarification will resolve your concerns. Thank you!\n\n\n> Q2: Line 159: Shluld $R_t=(\\sum_{i\\leq t} \\delta_i)\\vee 1$?\n\n\n**To Q2**: Yes, thank you for your meticulous attention to details. We will incorporate this modification into the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores an online sample selection problem with both individual constraints (e.g., false selection rate) and interactive constraints (e.g., diversity). The authors propose a simple yet effective algorithm, supported by strong theoretical analysis and numerical experiments. The reviews were generally positive, and the rebuttals effectively addressed the reviewers' questions. Therefore, I recommend acceptance.

For the camera-ready version, I encourage the authors to incorporate their discussions with the reviewers and enhance the presentation as suggested by reviewer vWkU.
Decision: Accept (poster)