Conformalized Multiple Testing after Data-dependent Selection

Authors: xiaoning  wang
yuyang  huo
liuhua  peng
changliang  zou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:24:13

Abstract: The task of distinguishing individuals of interest from a vast pool of candidates using predictive models has garnered significant attention in recent years. This task can be framed as a *conformalized multiple testing* procedure, which aims at quantifying prediction uncertainty by controlling the false discovery rate (FDR) via conformal inference. In this paper, we tackle the challenge of conformalized multiple testing after data-dependent selection procedures. To guarantee the construction of valid test statistics that accurately capture the distorted distribution resulting from the selection process, we leverage a holdout labeled set to closely emulate the selective distribution. Our approach involves adaptively picking labeled data to create a calibration set based on the stability of the selection rule. This strategy ensures that the calibration data and the selected test unit are exchangeable, allowing us to develop valid conformal p-values. Implementing with the famous Benjamini-Hochberg (BH) procedure, it effectively controls the FDR over the selected subset. To handle the randomness of the selected subset and the dependence among the constructed p-values, we establish a unified theoretical framework. This framework extends the application of conformalized multiple testing to complex selective settings. Furthermore, we conduct numerical studies to showcase the effectiveness and validity of our procedures across various scenarios.

Main paper: 
 introduction: 
 In recent years, there has been a notable focus on the use of predictive models to distinguish specific individuals from a pool of candidates. For instance, in the field of financial investment [17, 2] , machine learning models can be used to predict profits for different investment opportunities. Candidates with high predicted profits can then be given more preference and considered as potential investment options. Similarly, in disease diagnostics [29, 45] , researchers can utilize relevant information and corresponding predictions from machine learning models to identify potential patients.In a typical scenario, we are presented with a labeled/holdout data set D c = {Z i = (X i , Y i )} n i=1 , where X i ∈ R d is the observed covariate and Y i ∈ R is the response, and an unlabelled/test set D u = {X i } n+m i=n+1 . In practice, we only observe the covariates in the test set D u and the responses Y are unknown. Our goal is to distinguish individuals in D u whose undisclosed responses fall within a predetermined region A. Region A can take various forms, such as (b, ∞), [a, b] and (-∞, a) per user's requirements. To estimate the value of Y for the identification, we employ a predictive model μ : R d → R. However, directly using the black-box prediction μ(X i ) as a substitute for Y i leads to inherent uncertainty. In order to quantify this uncertainty, we reformulate our problem as multiple hypothesis testing [10] : for j = {n + 1, • • • , n + m}, H 0,j : Y j ̸ ∈ A v.s. H 1,j : Y j ∈ A.Under this framework, we wish to make rejections as much as possible with the false discovery rate (FDR) controlled at a pre-given level α. Denote the index set of test data as U = {n + 1, • • • , n + m}. The FDR is defined as the expectation of false positive proportion (FDP) over the test units in U, i.e.FDR(U) = E[FDP(U)], FDP(U) = j∈U 1{j ∈ R, Y j ̸ ∈ A} 1 ∨ |R| .where we denote a ∨ b = max{a, b} for any a, b ∈ R, |S| as the cardinality of a set S and R as the rejection set. To derive the testing rule, one feasible method is to construct the conformal p-value [8] by ranking the nonconformity score associated with the test unit's prediction among the scores in the holdout set. Then we can apply the well-known Benjamini-Hochberg (BH) procedure [10] on these conformal p-values to obtain a rejection set with controlled FDR. Here we term this procedure as "conformalized multiple testing".In practice, researchers may be interested in specific subsets rather than analyzing the entire dataset. For example, they might aim to determine the presence or absence of lung cancer among heavy smokers. By establishing data-driven criteria or thresholds based on factors such as the daily number of cigarettes smoked and smoking history, researchers can create a filtered subset consisting of heavy smokers. This allows researchers to gain insights into the patterns of the medical condition within this particular group. Here we address that the group partition may not be predetermined and could instead be learned from data, through methods like clustering [25] or thresholding. Denote the selected subset from unlabelled data as Ŝu ⊂ U. In our paper, we aim to find a rejection set Ru ⊂ Ŝu with the following FDR criterion controlled at α, i.e.FDR( Ŝu ) = E j∈ Ŝu 1{j ∈ Ru , Y j ̸ ∈ A} 1 ∨ | Ru | ≤ α.For simplicity, we use FDR to denote FDR( Ŝu ) only. The selection procedure would distort the distribution of test statistics, invalidating the p-values and leading to the failure of FDR control. This falls into the category of selective inference, which has been addressed in both statistics and machine learning fields [53, 16] . To tackle the selective issue, the use of labeled data becomes crucial. By leveraging labeled data, it is possible to obtain the conditional distribution of the selected data, which in turn allows for the construction of valid p-values.Nevertheless, even with valid p-values, controlling the FDR proves to be a challenging task. This difficulty arises from the inherent randomness of the selected subset Ŝu . Even a minor disturbance in Ŝu can lead to significant changes in the final rejection set Ru . Consequently, in order to address this issue, we focus on several commonly used selection rules with certain selection stability. We employ an adaptive strategy to carefully choose labeled data, thereby creating a calibration set that takes into account the selection stability.In this paper, we construct the selective conformal p-value for each selected individual, built on the marginal conformal p-value [8] . To address the selection effects, we adaptively pick a calibration set from the labeled data according to the selection rule, to ensure the exchangeability between the test data and labeled data. The selective conformal p-values are then constructed using the picked calibration set. By combining the selective conformal p-values with the well-known BH procedure, we achieve FDR control after data-driven selection, as verified through our comprehensive analysis.The main contributions of our paper can be summarized as follows.• Firstly, we frame the problem of multiple testing after data-dependent selection in the predictive setting and propose a viable solution utilizing the labeled data. • Secondly, the proposed method achieves exact FDR control for selection rules with strong stability, including joint-exchangeable rules and the top-K selection. And we further extend our method to handle more general cases where the selection rules satisfy a weaker stability condition such as sample mean selection.• Thirdly, the theoretical advancement extends the scope of classic multiple testing into the selective setting, providing a unified analytical technique for handling the randomness arising from data-driven selection.• Finally, through extensive experiments, we evaluate the reliability of our method in delivering the desired FDR control, and emphasize its easy integration with various algorithms.Multiple testing Ever since the seminal work of Benjamini and Hochberg [10] , the framework of multiple testing has been well developed by many researchers [49, 5, 66, 20, 44] . Our paper connects to the area of two-stage testing [62] , which firstly selects a subset of hypotheses and subsequently applies a multiple testing procedure to the selected set. To maintain the validity of test statistics after the selection, Bourgon et al. [14] recommended using an independent statistic specifically for the purpose of selection, but it is unavailable in our predictive setting. Instead of assuming independence between the test statistic and the selection statistic, Du and Zhang [19] introduced the concept of a "single-index" p-value for the joint modeling of both statistics and provided FDR control under symmetry assumption. Besides, Efron [21] considers applying multiple testing procedures over pre-given groups to guarantee group-wise FDR control, while our work considers the FDR control over data-driven subgroups, which is more challenging.Conformal inference Conformal inference [57, 40, 56, 55] has garnered significant attention in recent years, which leverages data exchangeability to construct model-agnostic prediction intervals. We present some recent developments therein [39, 6, 59, 61, 50, 18, 15, 1] . Within the conformal inference framework, several studies have focused on controlling the FDR in predictive setting, i.e. the conformalized multiple testing [8, 26] . These studies involve constructing a valid testing procedure using a holdout set. Such procedures include the BH procedure based on conformal pvalues [27, 35, 34, 24] , thresholding via an FDP estimator [63, 41] and the e-BH procedure applied to generalized e-values [7, 64] . Different from them, we focus on a selective scenario in conformalized multiple testing, addressing new challenges arising from selective randomness.Selective inference Selective inference concerns the inference problem after data-dependent processing. Previous works have mainly focused on the inference of parameters [60, 31] . Recently, Bao et al. [3] extended selective inference to the realm of conformal inference. They proposed a method to construct selection conditional prediction intervals with controlled false coverage-statement rate (FCR) [12] after data selection. Building upon this work, selective conformal inference with FCR control has been further extended to accommodate more general selection rules [23, 28] or the online setting [4] . In particular, under a certain class of selection rules, Gazin et al. [23] involves a procedure for FDR control after selection which closely aligns with our method. However, their approach focuses on selecting an informative set with FCR control under specific selection assumptions, limiting its applicability in more general scenarios such as those with data-dependent selection. The problem we tackle presents additional complexities, due to the intricate dependence on the selection procedure and final decision procedure, requiring a more intricate and delicate analysis. Besides, Sarkar and Kuchibhotla [47] proposed a post-selection framework to guarantee simultaneous inference [13] for all coverage levels, which differs greatly from our scenarios. 
recap: conformalized multiple testing: 
 We first introduce how to make multiple testing in the predictive setting. Denote the index sets for the labelled data D c as C = {1, • • • , n}. Suppose μ(•) is a predictor via a machine learning algorithm that is pre-given or can be trained on extra labeled data. Thus we can treat μ(x) as fixed. To construct a valid test statistic based on μ(X j ), Bates et al. [8] considered to use conformal p-values built upon the conformal inference framework [56] . Consider a monotone transformation V such that the larger value of V (μ(X j )) indicates the bigger likelihood of Y ̸ ∈ A. For example, if A = (b, ∞), we can use V (y) = b -y. The marginal conformal p-value p M j for X j is defined asEQUATIONwhere we denote V j = V (μ(X j )) as the nonconformity score for j-th sample and C 0 = {i ∈ C : Y i ̸ ∈ A} as the index set of labeled set containing only null samples.The properties of marginal conformal p-value constructed by i.i.d.∼labeled and test data have been investigated by Bates et al. [8] and we present them in the following proposition. Proposition 2.1 (i) guarantees that marginal conformal p-value is superuniform, thus it is a valid p-value. As the conformal p-values have a nice dependence structure, the rejection set obtained by the famous BH procedure [10] enjoys valid FDR control Proposition 2.1 (ii) indicates. For a set of p-values {p i } m i=1 , the BH procedure finds k = max{j : p (j) ≤ jα/m} where p (j) denotes the j-th smallest p-value in {p j } m j=1 and obtain the rejection set as Ru = {j : p j ≤ kα/m}. Proposition 2.1 (Properties of the conformal p-value [8] ). Suppose the labeled data and test data are i.i.d.. For simplicity, we assumeU 0 = {j ∈ U : Y j ̸ ∈ A} = {n + 1, • • • , n + m 0 } for m 0 ≤ m.The conformal p-values in (1) satisfy:(i) The p M j is a marginally superuniform p-value, i.e. for any t,Pr(p M j ≤ t | j ∈ U 0 ) ≤ t.(ii) Furthermore, the BH procedure applied at level α on the conformal p-values {p M j } j∈U controls the FDR level at πα, where π is the null proportion of test samples.The property of the conformal p-value is obtained by the exchangeability between the labeled data and test data. Through this, we can approximate the distribution of V j , where j ∈ U 0 , using V i from i ∈ C 0 . Therefore, when the labeled data and test data have different distributions, maintaining the exchangeability becomes crucial in constructing valid conformal p-values.As Storey [48] suggested, we can further estimate the null proportion and incorporate it in the BH procedure to further increase detection power. With the aid of labeled data, the null proportion can be directly estimated by the corresponding proportion in the labeled set, i.e. π = |C 0 |/|C|.A selection procedure could possibly be employed to the test samples. In this case, the focus lies primarily on the selected subgroup rather than the entire dataset, and decisions are made solely based on this subset. Define the selection rule as S Dc,Du , which is a function of labeled set D c and test set D u . For simplicity, we may omit this subscript. The selection rule maps an individual point X into a selection decision {0, 1}. And the selected subset can be written as Ŝu = {j ∈ U : S(X j ) = 1}.There are many examples for data-dependent S. The S can be the clustering algorithm which automatically determines the subgroup. Alternatively, S is associated with the selection score T j , which is derived from certain components of X j , and a selection threshold τ , such as the sample mean value in {T j } j∈U . In this case, we can express {S(X j ) = 1} = {T j < 1 m k∈U T k }.After the selection procedure, we would make multiple testing on the selected subset Ŝu . However, the distribution of the selected conformal p-values in (1) would be distinct from the original ones, due to the selection effects. Consequently, directly running the BH procedure on the marginal conformal p-values (1) in selected set Ŝu has no guarantee, which may lead to an inflated FDR level or poor power. Addressing this issue raises two important considerations:• How to characterize the selection conditional distribution of the selected individuals to construct valid p-values?• How to take account of the dependence structure among the valid p-values and the stochastic nature of the selection event to design a trustful multiple testing procedure?The first issue is widely considered in post-selection inference. Previous literature heavily relies on the normality assumption to derive the conditional distribution of test statistics [32, 52] . By the spirit of conformal inference, we consider constructing the selective conformal p-value by picking up the calibration set via the same selection rule, thereby guaranteeing the exchangeability between the selected test unit and picked calibration data.To be specific, we employ the selective algorithm S on the labeled set to derive the picked calibration set Ŝc = {i ∈ C : S(X i ) = 1}. If S involves a selection threshold τ , we will choose the calibration set as Ŝc = {i ∈ C : T i ≤ τ }. We hope {V i : i ∈ Ŝc } ∪ V j for j ∈ Ŝu exhibits a certain level of exchangeability, enabling us to capture the distribution of V j .With the aid of Ŝc , the selective conformal p-value can be accordingly constructed as:EQUATIONAfter obtaining valid p-values, ensuring the guarantee of the BH procedure is not straightforward due to the dependence arising from the use of the same calibration set in computing conformal p-values and the randomness from the data-dependent selection. Therefore, the second concern needs to be carefully addressed. In this article, we examine the BH procedure applied to the selective conformal p-values can enjoy finite sample FDR control for several commonly used selection rules. We outline our procedure in Algorithm 1 and refer to our method as Selective Conformal P-Value (SCPV).Algorithm 1 Selective conformal p-value with BH procedure (SCPV) Input: Labeled set D c , test set D u , selection procedure S Dc,Du , prediction model μ(•), target FDR level α ∈ (0, 1).Step 1 (Selection) Apply the selective procedure S Dc,Du to obtain the selected subsets Ŝu and Ŝc .Step 2 (Calibration) Compute {V i : i ∈ Ŝc ∩ C 0 }, {V j : j ∈ Ŝu }.Step 3 (Construction) Construct selective conformal p-value for each j ∈ Ŝu as (2)Step 4 (BH procedure) Compute k * = max{k : j∈ Ŝu 1(p j ≤ αk/m) ≥ k}Output: Rejection set Ru = {j ∈ Ŝu : p j ≤ αk * /m}.We introduce some auxiliary lemmas. The first one is the quantile inflation lemmas which is common in conformal inference literature [46, 3] and we omit its proof.Lemma F.1. Let x (⌈nt⌉) is the ⌈nt⌉-smallest value in {x i ∈ R : i ∈ [n]}.Then for any t ∈ (0, 1), it holds that1 n n i=1 1(x i ≤ x (⌈nt⌉) ) ≤ t.i : i ∈ [n]} are distinct, it also holds that 1 n n i=1 1(x i ≤ x (⌈nt⌉) ) ≥ t - 1 n ,The next lemma is the key of our theoretical results, which characterizes the properties of the constructed p-values. The proof is deferred to Appendix F.4. Lemma F.2. If the data are i.i.d. and the selection rule S Dc,Du is strongly stable, we have that (a) R (j→0) defined in ( 9) is measurable with respect to Φ j defined in 7.(b) For any j ∈ U,it holds that for any random variable t ∈ R that is measurable with respect to the unordered set Φ j , we haveP p j ≤ t | j ∈ Ŝu , Y j ̸ ∈ A, Φ j ≤ t.F.3 Proof of Theorem 3.2Proof. We begin by establishing the validity of the selective conformal p-values. Consider the quantityEQUATIONwhich consists of two components: D U \{j} , the test data with the j-th sample excluded, andD * C∪{j} := [Z i ; i ∈ C ∪ {j}],whose elements are taking values on {Z i } i∈C ∪ {Z j } but without their indexes. According to Lemma F.2(b), we haveP p j ≤ t | j ∈ Ŝu , Y j ̸ ∈ A = E E 1{p j ≤ t} | j ∈ Ŝu , Y j ̸ ∈ A, Φ j ≤ tNext, we proceed to verify the control of FDR. This involves examining the previously defined defined p-valuesp j = 1 + i∈ Ŝc∩C0 1 {V i < V j } 1 + | Ŝc ∩ C 0 | , for j ∈ Ŝu .For any j ∈ Ŝu , define a set of slightly modified p-valuesEQUATIONThese p-values are only used in our analysis. Also define R {a j : j ∈ Ŝu } ⊆ Ŝu as the rejection (indices) set obtained by the BH procedure, from p-values taking on the values in {a j : j ∈ Ŝu }.In the sequel, we will compare R to R {p(j) l : ℓ ̸ = j, ℓ ∈ Ŝu } ∪ {p j }on the event {Y j ̸ ∈ A, j ∈ R}. For the remaining p-values, since the scores have no ties, we consider two cases:(i) If V j ≤ V ℓ , then p (j) ℓ = 1 + i∈ Ŝc∩C0 1 {V i < V ℓ } | Ŝc ∩ C 0 | + 1 = p ℓ .(ii) If V j > V ℓ , then p ℓ ≤ p j . Since j ∈ R, the BH procedure implies ℓ ∈ R. By definition, we havep (j) ℓ ≤ 1 + i∈ Ŝc∩C0 1 {V i < V ℓ } 1 + | Ŝc ∩ C 0 | ≤ 1 + i∈ Ŝc∩C0 1 {V i < V ℓ } 1 + | Ŝc ∩ C 0 | = p j .To summarize, suppose we are to replace p ℓ by p (j) ℓ for all ℓ ̸ = j, ℓ ∈ Ŝu . Then on the event {Y j ̸ ∈ A, j ∈ R}, such a replacement does not change any of those p ℓ ≥ p j ; also, all those p ℓ ≤ p j including p j itself (they are rejected in R ) are still no greater than p j after the replacement. Thus, by the step-up nature of the BH procedure, such a replacement does not change the rejection set, meaning thatR = R {p j : j ∈ Ŝu } = R {p (j) l : ℓ ̸ = j, ℓ ∈ Ŝu } ∪ {p j } =: R (j)on the event {Y j ̸ ∈ A, j ∈ R}. Let R j = 1{j ∈ R}, then a leave-one-out analysis of the FDR impliesFDR = E j∈ Ŝu 1 {Y j ̸ ∈ A} R j 1 ∨ j∈ Ŝu R j (i) = j∈U E   | Ŝu| k=1 1 k 1{|R| = k}1 {Y j ̸ ∈ A} 1 p j ≤ αk/| Ŝu | 1{j ∈ Ŝu }   (ii) = j∈U E   | Ŝu| k=1 1 k 1 R (j) = k 1 {Y j ̸ ∈ A} 1 j ∈ R (j) 1{j ∈ Ŝu }   .The (i) use the property of the BH procedure, and (ii) comes from the facts stated just above. By the step-up nature of the BH procedure, we know that on the event j ∈ R (j) , sending p j to zero does not change the rejection set, i.e., we haveEQUATIONThusEQUATIONEQUATIONBy Lemma F.2(a), it holds thatE   1 p j ≤ α R (j→0) /| Ŝu | 1 {Y j ̸ ∈ A} 1{j ∈ Ŝu } 1 ∨ R (j→0) | Φ j   = 1 1 ∨ R (j→0) E 1 p j ≤ α R (j→0) /| Ŝu | 1 {Y j ̸ ∈ A} 1{j ∈ Ŝu } | Φ j = 1 1 ∨ R (j→0) P p j ≤ α R (j→0) /| Ŝu | | j ∈ Ŝu , Y j ̸ ∈ A, {V i } i∈C∪{j} P j ∈ Ŝu , Y j ̸ ∈ A | Φ jand R (j→0) /| Ŝu | is measurable with respect to the unordered set Φ j . Then use Lemma F.2(b), we haveP p j ≤ α R (j→0) /| Ŝu | | j ∈ Ŝu , Y j ̸ ∈ A, Φ j ≤ α R (j→0) /| Ŝu |.Through summing over j ∈ U, together with tower's rule, this givesFDR ≤ j∈U αE 1{j ∈ Ŝu , Y j ̸ ∈ A} | Ŝu | = αE | Ŝu ∩ U 0 | | Ŝu | ≤ αwhich concludes the proof.F.4 Proof of Lemma F.2Proof. We note that the following proof is conditioned on the event {Y j ̸ ∈ A}. Define For any j satisfying Y j ̸ ∈ A, define the eventEQUATIONDenote the corresponding unordered conformal scores by [v 1 , • • • , v n+1 ] under A C∪{j} (z). Since S Dc,Du is strongly stable, then given D U \j and under A C∪{j} (z) we haveS Dc,Du (x i ) = S Dc∪{Zj },Du\{Zj } (x i ) = S [z1,••• ,zn,zn+1],Du\{Zj } (x i ) for i = 1, • • • , n + 1.It means that the following unordered set{V i } i∈ Ŝc 0 ,+j | A C∪{j} (z) = {v i : S z,Du\{Zj } (x i ) = 1, y i ̸ ∈ A} i=1,••• ,n+1is known and only depend on z and the data D U \j . Besides,| Ŝc0,+j | | A C∪{j} (z) = | i ∈ {1, • • • , n + 1} : S z,Du\{Zj } (x i ) = 1, y i ̸ ∈ A |is also known. Note that by definition (8) , p(j) ℓ ℓ̸ =j is invariant after permuting {V i } i∈ Ŝc∩C0 ∪ {V j }.We know that the modified p-value p (j)ℓ | A C∪{j} (z) = i∈ Ŝc 0 ,+j 1{v i < V ℓ } | Ŝc0,+j | , ∀ℓ ̸ = j, ℓ ∈ Ŝu .is fixed condition on A C∪{j} (z). Also note that R (j→0) only depends on p(ℓ) j ℓ̸ =j, and this implies that R (j→0) is known under A C∪{j} (z). Through marginalizing over A C∪{j} (z), we obtain that p (j) ℓ is measurable with respect to Φ j . Since R (j→0) is only depend on pℓ for ℓ ̸ = j, ℓ ∈ Ŝu , the first part of Lemma F.2 can be readily demonstrated.For the second part, we know it holds that{p j ≤ t} = {V j ≤ V Ŝc∩C0 (⌈t(| Ŝc∩C0|+1)⌉) } = {V j ≤ V Ŝc∩C0∪{j} (⌈t(| Ŝc∩C0|+1)⌉) } = {V j ≤ V | Ŝc 0 ,+j (τ )|(⌈t| Ŝc 0 ,+j (τ )|⌉) } by the construction of p j and Lemma F.1. And then we haveEQUATIONFor ease of presentation, we write | Ŝc0,+j | | A C∪{j} (z) = S c0,+j (z; D U \j ),V Ŝc 0 ,+j (⌈t(| Ŝc 0 ,+j |)⌉) | A C∪{j} (z) = V(z; D U \j ).For any unordered set z, we define the following unordered setΩ(z) = (i 1 , i 2 ) ⊆ [n + 1] : v i1 ≤ V(z; D U \j ), S z,Du\{Zj } (x i1 ) = 1, y i1 ̸ ∈ A, S z,Du\{Zj } (x i2 ) = 1, y i2 ̸ ∈ Awhich is σ(D U \j )-measurable and independent of A C∪{j} (z). Using the exchangeability of (Z i ) i∈C∪U , we can guaranteeE S c0,+j (z; D U \j ) | Ŝc0,+j | 1 V j ≤ V Ŝc 0 ,+j (⌈t(| Ŝc 0 ,+j |)⌉) , S Dc,Du (X j ) = 1, Y j ̸ ∈ A, S Dc,Du (X k ) = 1, Y k ̸ ∈ A | A C∪{j} (z) =E 1 V j ≤ V(z; D U \j ), S z,Du\{Zj } (X j ) = 1, Y j ̸ ∈ A, S z,Du\{Zj } (X k ) = 1, Y k ̸ ∈ A | A C∪{j} (z) = (i1,i2)⊆Ω(z) P Z j = z i1 , Z k = z i2 | A C∪{j} (z) = (i1,i2)⊆Ω(z) P Z j = z i2 , Z k = z i1 | A C∪{j} (z) =E 1 V k ≤ V(z; D U \j ), S z,Du\{Zj } (X k ) = 1, Y k ̸ ∈ A, S z,Du\{Zj } (X j ) = 1, Y j ̸ ∈ A | A C∪{j} (z) Through marginalizing over A C∪{j} (z), it follows that E 1 | Ŝc0,+j )| 1 V j ≤ V Ŝc 0 ,+j (⌈t(| Ŝc 0 ,+j |)⌉) , S z,Du\{Zj } (X j ) = 1, Y j ̸ ∈ A, S z,Du\{Zj } (X k ) = 1, Y k ̸ ∈ A | Φ j =E 1 | Ŝc0,+j | 1 V k ≤ V Ŝc 0 ,+j (⌈t(| Ŝc 0 ,+j |)⌉) , S z,Du\{Zj } (X j ) = 1, Y j ̸ ∈ A, S z,Du\{Zj } (X k ) = 1, Y k ̸ ∈ A | Φ jPlug into (13), we can verify the second part of Lemma F.2 immediately. X i ) = S D k ,D l (X i ) = S Dc∪{Zj },Du\{Zj } (X i ), for any i ∈ C ∪ U and j ∈ U. Thus the proof is completed. F.6 Proof of Proposition 3.5 Proof. Let {T (r) : r ∈ [m]} be order statistics of {T n+1 , • • • , T j-1 , T j , T j+1 , • • • , T n+m } and {T j→-∞ (r) : r ∈ [m]} be order statistics of {T n+1 , • • • , T j-1 , -∞, T j+1 , • • • , T n+m }. By definition, τ topK (T n+1 , • • • , T j-1 , T j , T j+1 , • • • , T n+m ) = T (K+1) and τ topK (T n+1 , • • • , T j-1 , -∞, T j+1 , • • • , T n+m ) = T j→-∞ (K+1) .Note that for j ∈ Ŝu , we haveT (K+1) > T j . Because T (r) = T j→-∞ (r)for all order statistics with T (r) ≥ T j , we have T (K+1) = T j→-∞ (K+1) as well.Therefore, we replace Z j = z for any j ∈ Ŝu , where z is a fixed value such that the corresponding selection score T j (determined by the covariates X j ) is -∞. Here we address that the top-K selection does not the specific scale of the selection scores. Hence the we can scale the selection scores into the range of (0, 1), and denote z as the value such that T j = 0.As the threshold τ topK keeps unchanged after replacing Z j as z, we have S Dc,Du (X i ) = S Dc,Du\{Zj }∪{z} (X i )for any j ∈ Ŝu and i ∈ C ∪ U.F.7 Proof of Proposition 3.7Proof. For simplicity, we suppose that the selection rule produces a selection threshold τ ({T k } k∈U ) which is dependent on test data only. The selected test set is denoted as Ŝu = {j ∈ U : T j ≤ τ ({T k } k∈U )}. And the calibration set is picked up the byŜc (j) = {i ∈ C : T i ≤ τ ({T k } k∈U \{j}} )} for j ∈ Ŝu .We note that the proof is similar for all type of weakly stable selection rule. Given D u \{j}, we know that τ ({T k } k∈U \{j}} ) is fixed. Therefore, for j ∈ Ŝu , i.e. T j ≤ τ ({T k } k∈U \{j}} ) , it holds that {Z i ∈ D c : T i ≤ τ ({T k } k∈U \{j}} )} = {Z i : i ∈ Ŝc (j)} and Z j are exchangeable. Consequently, for j ∈ U 0 and j ∈ Ŝu , it follows that {Z i : i ∈ Ŝc (j) ∩ C 0 } and Z j are exchangeable, which implies {V i : i ∈ Ŝc (j) ∩ C 0 } and V j are also exchangeable.By the definition of p adapt j and the similar procedure for proving Lemma F.2 in Appendix F.4, it is direct that Pr(p adapt Proof. Our proof follows the same strategy as [22, 34] . Recall the final rejection set by conditional calibration can be formulated byj ≤ t | j ∈ Ŝu , j ∈ U 0 ) ≤ t.R = {j ∈ R + : ϵ j ≤ R Rj },where R = |R|, Rj is the rejection number by BH procedure applied to {p adapt,(j) l: ℓ ̸ = j, ℓ ∈ Ŝu } ∪ {0} and R + = {j ∈ Ŝu : p adapt j ≤ α Rj | Ŝu | }.Define ϵ -j as all ϵ i variables for i ∈ R + \ {j} and R * = R(ϵ j ← 0) denote the hypothetical total number of rejections obtained by fixing ϵ j = 0 prior to applying conditional calibration procedure. Then, the FDR can be written asFDR = j∈U E   1{j ∈ R + }1{ϵ j ≤ R Rj }1{Y j ̸ ∈ A} 1 ∨ R   (i) = j∈U E   1{j ∈ R + }1{ϵ j ≤ R * Rj }1{Y j ̸ ∈ A} 1 ∨ R *   = j∈U E   E   1{j ∈ R + }1{ϵ j ≤ R * Rj }1{Y j ̸ ∈ A} 1 ∨ R *   | ϵ -j , D c ∪ D u   (ii) = j∈U E 1{j ∈ R + }1{Y j ̸ ∈ A} 1 ∨ Rj = j∈U E   1{j ∈ Ŝu }1{p adapt j ≤ α Rj | Ŝu| }1{Y j ̸ ∈ A} 1 ∨ Rj   (iii) = j∈U E   E 1{j ∈ Ŝu }1{p adapt j ≤ α Rj | Ŝu| }1{Y j ̸ ∈ A} | Φ j 1 ∨ Rj   (vi) ≤ j∈U E α Rj | Ŝu | 1{j ∈ Ŝu }1{Y j ̸ ∈ A} 1 ∨ Rj = α j∈U E 1{j ∈ Ŝu }1{Y j ̸ ∈ A} | Ŝu | = αE | Ŝu ∩ U 0 | | Ŝu | .Equality (i) holds since R = R * for j ∈ Ŝu , as the pruning procedure can be seen as a special BH procedure, which is not influence by replacing a rejected p-value with 0. Equality (ii) is true because ϵ j is independent of ϵ -j given all the data D c ∪ D u , and R * is measurable to ϵ -j , D c ∪ D u , where ϵ j has no influence on R * by the assignment (ϵ j ← 0). Equality (iii) holds as Rj is measurable with respect to Φ j by the design of p adapt,(j) l. And inequality (iv) comes from the Proposition 3.7, since the weak stability implies | Ŝu | = |{i ∈ U : T i < τ ({T k } k∈U \{j}} )}| is measurable with respect to Φ j and j ∈ Ŝu . Thus the proof is completed.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the data and code in the supplemental materials, including instructions in the zip file. Guidelines:• The answer NA means that paper does not include experiments requiring code. • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
theoretical guarantee: 
 In this section, we aim to verify the FDR guarantee of Algorithm 1 for several commonly used selection rules. Our focus here is to tackle the technical challenges associated with the selective multiple testing problem. Unlike the conventional approach where a fixed number m of test units is considered, we encounter a challenge due to the involvement of a random number of test units | Ŝu |. This randomness makes the analysis considerably intricate unless we impose certain restrictions on the selection rule. To deal with the selection set Ŝu , we introduce the concept of strong stability. • (Leave out) S Dc,Du (X i ) = S Dc∪{Zj },Du\{Zj } (X i ); • (Replace) S Dc,Du (X i ) = S Dc,Du\{Zj }∪{z} (X i ) for a fixed point z.Here we define the strong stability of selection rule in two common ways: leaving one point out or replacing one point with a fixed value. Many popular selection rules are strongly stable, such as joint-exchangeable rule and top-K selection. Detailed discussions are provided in next subsections.The strong stability plays a crucial role in our analysis, as it enables us to fix the randomness of the selected set Ŝu . With the strongly stable property, we can perform a delicate analysis for the rejection set from Algorithm 1 to obtain the theoretical guarantee. ≤ αE[| Ŝu ∩ U 0 |/| Ŝu |] ≤ α.We present the insight of our proof for Theorem 3.2. As a common operation in analyzing FDR [22, 35] , we decompose the FDR into the FDR contribution for each j ∈ U asFDR = E j∈U 1{j ∈ Ru , j ∈ U 0 } 1 ∨ | Ru | = j∈U E   1{p j ≤ α | Ru| | Ŝu| , j ∈ Ŝu , j ∈ U 0 } 1 ∨ | Ru |   .By the stability of selection rule, we can replace | Ru | with a decoupled version | R(j) u | which removes the influence of p j . If given some quantity Φ j that blocks most of the nuisance parameters, the p-value p j has a uniform distribution and | R(j) u |, | Ŝu | are fixed. Then the FDR control is byFDR = j∈U E   E[1{p j ≤ α | R(j) u | | Ŝu| , j ∈ Ŝu , j ∈ U 0 } | Φ j ] 1 ∨ | R(j) u |   ≤ j∈U E α| R(j) u | | Ŝu | 1 1 ∨ | R(j) u | 1{j ∈ Ŝu , j ∈ U 0 } = αE | Ŝu ∩ U 0 | | Ŝu |By the construction of the conformal p-value, we analyze each selected unit j ∈ Ŝu by conditioning on a carefully constructed quantity Φ j = (D * C∪{j} , D U \{j} ). It comprises two components: D U \{j} , the test data with the j-th sample excluded, andD * C∪{j} := [Z i ; i ∈ C ∪ {j}],the unordered set of labeled data along with the j-th sample. The unordered set provides the order statistics but not the specific ordering, which is a common convention in conformal inference literature [35, 34] .Through this approach, we are able to decouple the dependence that arises from the data-dependent selection and the construction of p-values that share the same calibration data. If the selection rule is strongly stable, then | Ŝu | is fixed given Φ j for j ∈ Ŝu and the selective conformal p-value in (2) is valid. By performing a careful analysis of the rejection set Ru , i.e. replacing it with a pseudo rejection set R(j) u that remains fixed given Φ j and j ∈ Ru , we obtain the finite sample FDR guarantee.Firstly, we consider the joint-exchangeable selection procedure. : i ∈ C ∪ U} if S Dc,Du (X i ) = S D k ,D l (X i ) for any i ∈ C ∪ U and D k , D l that are arbitrary partitions of D c ∪ D u .If the selection procedure is independent of both the labeled and test data, it is joint-exchangeable. In the case of the selection with a threshold, the joint-exchangeable selection is equivalent toτ (X 1 , • • • , X |C∪U | ) = τ (X π(1) , • • • , X π(|C∪U |) ).where τ (D) denotes that τ is computed using the dataset D. Therefore, the joint-exchangeable selection includes selection using constant thresholds or thresholds computed by {T i } i∈C∪U exchangeably. We can verify joint-exchangeable selection is strongly stable through the leaving out condition. Proposition 3.4. The joint exchangeable selection procedure S Dc,Du is strongly stable.According to Theorem 3.2, our procedure ensures FDR control for any joint-exchangeable selection rule, which makes the choice of selection rule quite flexible. For example, we can perform a clustering algorithm on the {X i } i∈C∪U to divide the data into different groups. As a special case, our approach aligns with the InfoSCOP proposed by Gazin et al. [23] under the joint-exchangeable selection rule. They proposed a novel procedure for selecting an informative set and also provided FDR control guarantee as an extension of their FCR control results.While the joint-exchangeable rule contains various selection strategies, it is worth noting that many cases involve selection that depends solely on the test data. In the following subsections, we investigate several commonly used selection rules that are determined only by the test data. And the assumption in InfoSCOP [23] is not satisfied under these cases.Next, we consider the top-K or quantile selection rule, which relies solely on the test data D u . This type of rule is extensively studied in the literature [42, 3, 28] and is commonly used in practice.Let τ topK denote the top-K selection threshold, which is defined as the (K + 1)-th smallest value in {T j : j ∈ U }. The top-K rule is equivalent to the quantile selection rule since τ topK corresponds to the (K + 1)/m-quantile of the test data, and the threshold for the q-quantile is the ⌈mq⌉-th smallest value. The selected test set and the chosen calibration set under the top-K rule are defined as:EQUATIONWe can verify that the top-K selection rule is strongly stable by the following proposition. With the support of Theorem 3.2, we can ensure FDR control when employing the top-K selection rule.Proposition 3.5. For top-K selection rule S with threshold τ topK (U), if j ∈ Ŝu , thenτ topK (T n+1 , • • • , T j-1 , T j , T j+1 , • • • , T n+m ) = τ topK (T n+1 , • • • , T j-1 , -∞, T j+1 , • • • , T n+m ).Thus top-K selection rule is strongly stable by the replacing condition.In this subsection, we consider weakening the strongly stable condition. For example, the mean thresholding rule does not satisfy the strong stability. By the insight of our proof, the key requirement is the property of Ŝu such that we can handle the randomness of the selection event. Hence we define the weakly stable selection rule as follows:Definition 3.6 (Weak stability). Given selection set Ŝu = {j ∈ U : S Dc,Du (X j ) = 1}. We call the selection rule S Dc,Du is weakly stable if S Dc,Du (X i ) = S Dc∪{Zj },Du\{Zj } (X i ) for any j ∈ Ŝu and any i ∈ U.The weak stability does not require the S Dc,Du (X i ) = S Dc∪{Zj },Du\{Zj } (X i ) hold for i ∈ C. Except for the selection rules previously discussed, the commonly used mean selection rule by test data only is also weakly stable, i.e. {j ∈ U :T j < 1 m i∈U T i } = {j ∈ U : T j < 1 m-1 i∈U \{j} T i }.As the weakly stable rule fails to guarantee the exchangeability of the selected calibration set and test set, it motivates us to explore a new construction method for selective p-values. The selection set is denoted as Ŝu = {j ∈ U : S Dc,Du (X j ) = 1}. By the definition of weakly stable selection, we know that Ŝu = {j ∈ U : S Dc∪{Zj },Du\{Zj } (X i ) = 1} is also true. Specially, if the selection rule is determined only by the test data, it holds that S Dc∪{Zj },Du\{Zj } (•) = S Du\{Zj } (•).Therefore, we adaptively pick data from the labeled set using the same "leaving out" rule as selecting Z j . For any i ∈ C and j ∈ U, {S Du\{Zj } (X i ) = 1} and {S Du\{Zj } (X j ) = 1} are symmetric to X i and X j . Leveraging this, we can pick up the calibration set byŜc (j) = {i ∈ C : S Du\{Zj } (X j ) = 1} for j ∈ Ŝuand construct the adaptive selective conformal p-value asEQUATION)For example, the mean selection rule picks up calibration set by Ŝc (j) = {i ∈ C :T i ≤ 1 m-1 k∈U \{j} T k }.Our adaptive strategy shares the same goal as the swapping strategy [? 28, 4] in terms of constructing valid p-value after selection. However, our approach is different from the others in core motivations since ours is directly related to weak stability, leading to a faster computation and a more intuitive explanation here. We can verify that p adapt j is valid since {Z k } k∈ Ŝc(j)∪{j} are exchangeable.Proposition 3.7. The adaptive selective conformal p-value p adapt j for weakly stable selection which is determined only by the test data satisfies Pr(p adaptj ≤ t | j ∈ Ŝu , j ∈ U 0 ) ≤ t.However, the p-value for each selected test point p adapt j is based on a different calibration set Ŝc (j), making the dependence structure intricate. Consequently, the BH procedure has no safe guarantee to control the FDR. But we find that BH is robust and can produce satisfactory results empirically.Although this observation is acceptable, it would be desirable to design a new procedure to guarantee the finite sample FDR control. To remedy this, we employ the conditional calibration framework [22] to achieve finite sample FDR control. The overall procedure can be re-framed as the e-BH framework [58] and a recent novel approach for boosting the power of e-BH procedure [33] can be employed in our setting. The details are displayed in Appendix B.Under the weakly stable selection rule, our method differs fundamentally from InfoSCOP [23] in both methodology and theory. Since our approach and InfoSCOP are designed for different goals, resulting in different analytical frameworks. Ours is specifically designed to address the multiple testing problem across various selection rules. From the perspective of conditional calibration, our method is unified, where the BH procedure for strongly stable selection can be seen as a special case. As a comparison, InfoSCOP stands out as a remarkable work for selecting an informative set with FCR control, but it is not primarily designed for our problem, which limits their method's applicability to more general selection rules. 
numerical studies: 
 To demonstrate the wide applicability of the proposed method in Algorithm 1, we conduct comprehensive numerical studies. For regression setting, the region for the hypothesis is A = {y : y > c 0 }, where c 0 is a fixed constant. For classification, we set A = {1}, i.e. the class 1 as the target region, and we denote the prediction μ(X) as the predicted probability of Y = 1. The nonconformity score we use to construct conformal p-value for both settings is V (μ(X j )) = -μ(X j ).Benchmarks: Since selective multiple testing has not been investigated before, we consider several intuitive methods as comparing benchmarks.• SCPV: Our procedure in Algorithm 1. Specially, for mean selection rule, we use the adaptive p-value in (4) along with the BH procedure. The results for using conditional calibration can be found in Appendix B;• OMT: Ordinary multiple testing which constructs the conformal p-value directly as in equation ( 1) for each selected sample based on the entire labeled set.• AMT (BH/BY): An intuitive procedure by multiplying (1) with the selection proportion of null samples. As the adjusted p-values have intractable dependence, making the validity of BH procedure suspicious, we also utilize the Benjamini-Yekutieli (BY) [11] procedure to control the FDR. More details are provided in Appendix A.1.• SCOP: Directly invert the selective prediction interval constructed by Bao et al. [3] into a test and make decision by whether the c 0 is contained in the interval. It is designed for regression setting and does not have FDR guarantee. See more detail in Appendix A.2.We also use the Storey's method [49] to increase power. See more information in Appendix C.1.Selection rule: In the numerical studies, we choose the selection statistic T i based on a specific component of X. The selected subset is Ŝu = {i ∈ U : T i < τ }, where τ is the threshold. Three different choices of selection thresholds are considered.• Exchangeable (Exch): 70%-quantile of the first component of X in both labeled set and test set, that is τ is the 70%-quantile of {T i : i ∈ C ∪ U}.• Quantile (Quan): 70%-quantile of the first component of X in the test set, that is τ is the 70%-quantile of {T i : i ∈ U}.• Mean: the sample mean of the first component of X in the test set, that is τ = 1 m j∈U T j .Evaluation metrics: We empirically evaluate the FDR by averaging the FDP based on selected samples and the power by averaging the proportion of correct selections among all selected alternative test samples, i.e. Power : In synthetic studies, we generate i.i.d. 10-dimensional covariates from X i ∼ Unif([-1, 1]) 10 . The corresponding regression responses are generated as Y i = µ(X i ) + ϵ i , where ϵ i denotes independent random noise. The following data-generating scenarios are considered:= |i ∈ Ŝu : i ∈ R, Y i > c 0 |/|i ∈ Ŝu : Y i > c 0 | over 100 independent runs.• Case A: The data generating model is µ(X) = 4(X (1) + 1)|X (1) |1{X (2) > -0.4} + 4(X (1) -1)1{X (2) ≤ -0.4}. The noise is ϵ i ∼ N (0, σ 2 ), independent of X. And c 0 = 2.• Case B: µ(X) = 1{X T β > 1.5}, where β = (1, -1, 2, -2, 0, 0, 0, 0, 0, 0). The noise is ϵ i ∼ N (0, 0.1σ 2 ), independent of X. And c 0 = 0.12.We fix the labeled data size n = 1, 200 and the unlabeled data size m = 1, 200. We fit the regression models μ(•) on an additional labeled set with size 1, 200 using the random forest algorithm, implemented by R package randomForest with default parameters. Specifically, for both scenarios, we select the first component of X as the selection statistic, i.e., T i = Xi . Figure 1 displays the FDR (left) and power (right) through varying noise strength. Across both settings, SCPV can deliver valid FDR control. As expected, the OMT fails to control FDR. This can be understood since the OMT constructs conformal p-values without consideration of the selection procedure, leading to smaller p-values possibly. Moreover, our method demonstrates greater statistical power compared to AMT. This is because AMT does not make full use of information from the selection procedure. Meanwhile, SCOP fails to control FDR in case B. And even if SCOP can control FDR in case A, the accompanying loss of power is substantial. This is because the SCOP is not designed for multiple testing and can not deliver valid FDR results.  
limitations and discussions: 
 Here we point out the current limitations of our paper and discuss the potential directions. First, our work relies on the i.i.d. assumption. Exploring the selective multiple testing problem in scenarios where the labeled set and test set exhibit different distributions would be interesting. Second, we require the selection rule to be stable for theoretical guarantee. It would be attractive to consider complex selection procedures that lack stability, such as clustering based on test data only.A.1 The adjusted p-value (AMT)Here, we provide an overview of the statistical properties associated with the adjusted p-values.We draw inspiration from the work of Benjamini and Yekutieli [12] , which constructs adjusted confidence intervals for selected parameters for FCR control. They multiplies the confidence level α by a quantity related to the proportion of selected candidates over all candidates. In most cases, the quantity is approximately the selection proportion. And they proved such simple adjustment on the level can provide FCR control.Analogously, we can adjust the p-value after selection by multiplying the selection proportion:EQUATION)where θ represents an estimator that estimates the selected proportion under the null hypotheses.Since the response of the test data is not directly observable, we estimate this proportion by employing the same selection procedure on the labeled data, i.e. θ = |C0| |C0∩ Ŝc|. We can verify that the adjusted p-value is super-uniform for joint-exchangeable selection rule. Proposition A.1. The adjusted p-value is super-uniform i.e.P(p * j ≤ α | j ∈ U 0 ∩ Ŝu ) ≤ αwhere p * j is defined in (8) .Proof. The case p * = 1 is trivial, so we only consider the caseπ 1+|{i∈C0:Vi≤Vj }| 1+|C0| ≤ 1.In this case, we haveP p * j ≤ α | j ∈ U 0 ∩ Ŝu = P |C 0 | |C 0 ∩ Ŝc | 1 + |{i ∈ C 0 : V i ≤ V j }| 1 + |C 0 | ≤ α | j ∈ U 0 ∩ Ŝu ≤ P 1 + |{i ∈ C 0 : V i ≤ V j }| 1 + |C 0 ∩ Ŝc | ≤ α | j ∈ U 0 ∩ Ŝu ≤ P 1 + |{i ∈ C 0 ∩ Ŝc : V i ≤ V j }| 1 + |C 0 ∩ Ŝc | ≤ α | j ∈ U 0 ∩ Ŝu (i) ≤ αwhere (i) use the property that the variables {V j : j ∈ U 0 ∩ Ŝu } and {V i : i ∈ C 0 ∩ Ŝc } are exchangeable when the selection procedure is exchangeable of the test and labeled data.The correlation of the adjusted p-values can be complex when using an arbitrary selection procedure.In order to address this issue, we propose utilizing the Benjamini-Yekutieli (BY) [11] procedure to effectively control the FDR. It replaces the original level α with α/L | Ŝu| , whereL | Ŝu| = | Ŝu| i=1 1 i = log | Ŝu | + O(1).The BY method can handle the dependence between p-values, but deliver a more conservative result.The selection conditional conformal prediction (SCOP) proposed by Bao et al. [3] is a method for constructing valid prediction intervals after selection. The prediction interval is reported only when it is selected. The SCOP aims to control the false coverage-statement rate (FCR) [12] , which is the expected ratio of the number of selected prediction intervals failing to cover their respective true outcomes to the total number of selected prediction interval, i.e.FCR := E j∈U I{j ∈ Ŝu , Y j ̸ ∈ PI(X j )} 1 ∨ | Ŝu | ,where PI(X j ) is the prediction interval. To ensure FCR control, the SCOP involves a similar procedure to pick up a calibration set from the labeled data using the same selection rule. And then the prediction interval for selected individual is constructed via the residuals in picked calibration set.As a natural idea, we can simply invert the prediction interval into a hypothesis testing. If the hypothesis is H 0,j : Y j ≤ c 0 v.s. H 0,j : Y j > c 0 , reject the single hypothesis with type I error at α is equivalent to that the one sided prediction interval covers c 0 . We formulate this idea as an intuitive benchmark as Algorithm 2.Algorithm 2 SCOP for selective multiple testing Input: Labeled set D c , test set D u , selection procedure S Dc,Du , prediction model μ(•), FCR level α ∈ (0, 1).Step 1 Apply the selective procedure S to obtain the selected subsets Ŝu and Ŝc .Step 2 Compute residuals{R i = Y i -μ(X i ) : i ∈ Ŝc }.Step 3 Construct selective conformal prediction intervals for each j ∈ Ŝu byPI(X j ) = (-∞, μ(X j ) + Q α ({R i } i∈ Ŝc )],whereQ α ({R i } i∈ Ŝc ) denotes the ⌈(1 -α)(| Ŝc | + 1)⌉-th smallest value in {R i } i∈ Ŝc . Step 4 Reject sample j if c 0 ∈ PI(X j ) Output: Rejection set RSCOP = {j ∈ Ŝu : c 0 ∈ PI(X j )}.Under the null, we have P(c 0 / ∈ PI(X j )) ≤ P(Y j / ∈ PI(X j )) ≤ α/P(Y j ≤ c 0 ). This implies marginal coverage, as for a single hypothesis H 0j , we can control the type I error. It is important to note that the scaling of the inequality P(c 0 / ∈ PI(X j )) ≤ P(Y j / ∈ PI(X j )) is often too conservative.However, when it comes to simultaneous testing, the SCOP procedure fails to control the FDR. This is because the FDR is built up on the rejection set R ⊂ Ŝu , where the rejection decisions are intricately linked to the entire selection set and the inherent randomness within the selection set further complicates the distribution of the final rejection set. Hence our work address this challenge by carefully analyzing the randomness from selection and rejection set.Here we introduce the frame work of conditional calibration [22] .The first idea of conditional calibration is to control a conditional expectation, given some conditioning statistic Φ j that blocks most or all of the nuisance parameters from influencing the conditional analysis. And we require that p adapt j is conditionally superuniform given Φ j :Pr(p adapt j ≤ t | j ∈ Ŝu , j ∈ U 0 , Φ j ) ≤ tSecondly, the number of rejections should should be bound from below by a known function of Φ j . If all constraints are satisfied, that set of rejections is guaranteed to control the FDR bellow α.Step 1: Calibration For each of the m test points, i ∈ D = {n + 1, ..., n + m}, let Φ j = (D * C∪{j} , D U \{j} ). For each test point j ∈ Ŝu , compute the adaptive p-valueEQUATIONNext, let Ri indicate the number of rejections obtained by applying BH at level α, for some fixed α ∈ (0, 1), to the approximate p-values {p adapt,(j) l: ℓ ̸ = j, ℓ ∈ Ŝu } ∪ {0}.Step 2: Preliminary rejection. Define the preliminary rejection set R + as:R + = i ∈ Ŝu : p adapt i ≤ α Ri | Ŝu | R+ = |R + |.If R+ ≥ Ri for all i ∈ R + , then return the final rejection set R = R + . Otherwise, proceed to the next step.Step 3: Pruning. (a) Deterministic pruning: Define R as:R dtm = max r : i ∈ R + : Ri ≤ r ≥ r .The pruned rejection set R is that containing the indices with i ∈ R + and Ri < R.(b) Randomized pruning: Generate independent standard uniform random variables ϵ i for each i ∈ R + , and define R as:R rdm = max r : i ∈ R + : ϵ i ≤ r/ Ri ≥ r .The pruned rejection set R is the set containing the indices i ∈ R + such that ϵ i < R/ Ri .The conventional conditional calibration offers a flexible framework to decouple the dependence between p-values. But in our selective setting, the number of test units is | Ŝu |, which can be complicatedly dependent with both p j and Rj . And when analyzing the FDR, the event that j-th sample is selected is also involved. So our primary focus is on ensuring E1{pj≤ α Ri | Ŝu | ,j∈ Ŝu}| Ŝu| | Rj | | Φ j ≤ α.The conditional calibration framework primarily focuses on the correlation of p-values. However, a significant challenge arises because FDR control in a selective setting involves not only individual p-values but also the selection procedure itself. Consequently, the selective effects are unavoidable when implementing conditional calibration. To address this, we leverage the stability property of the selection rule, which allows us to effectively conduct analysis over the selected subset effectively and rigorously.We can prove that the conditional calibration applied to the adaptive selective conformal p-values can control the FDR at α. The technical proofs are deferred in Appendix F.8. Theorem B.1. Assume the data are i.i.d. and the selection rule is weakly stable. Then, the FDR output by the above three-step procedure applied to p adapt is smaller than αE| Ŝu∩U | 0 | Ŝu| .It is easy to observe that R dtm ⊆ R rdm , indicating that randomized pruning results in larger rejection sets. Therefore, we employ randomized pruning in practice to enhance power. By our empirical investigations, we find the BH procedure applied to p adapt j can ideally control the FDR, and the conditional calibration approach with random pruning also has a close performance in power. Figure 2 displays the FDR (left) and power (right) through varying noise strength employing conditional calibration and BH procedure.The deterministic pruning process would lose certain power. Although randomized pruning can improve this situation, the external randomness can potentially hinder the reproducibility of the results (the procedure can be quite sensitive to the realization of the ϵ's, leading to different selections across various algorithm runs). A recent method [33] can enhance the power of the pruning process without introducing additional randomness. Specifically, let R(e) represent the rejection set yielded by the e-BH procedure on e at level α ∈ (0, 1). For each j ∈ [m], define R j (e) := R(e) ∪ {j} and subsequently define the function where e = ( e 1 , . . . , e m ) follows the conditional distribution e | S j . With the associated critical value c j := sup {c : ϕ j (c; S j ) ≤ 0}, the boosted e-values are constructed as:ϕ j (c; S j ) := E     m α • 1 c e j ≥ me b j =        m α| Rj (e)| • 1 c j e j ≥ m α| Rj (e)| if ϕ j ( c j ; S j ) ≤ 0 m α| Rj (e)| • 1 c j e j > m α| Rj (e)| if ϕ j ( c j ; S j ) > 0They prove that the boosted e-values are generalized e-values and R(e) ⊆ R(e b ).Our procedure can be viewed as a generalization for a selective scenario of their approach. The conditional calibration approach with deterministic pruning is equivalent to the e-BH procedure applied to {e j : j ∈ Ŝu }, wheree j = | Ŝu|1{pj ≤ α Rj (p) | Ŝu | } α Rj (p)by referencing Jin and Candès [27] and p = {p j } j∈ Ŝu . Another form of e-value existing in conformal inference is based on a specific stopping time [43, 7] . But we find it is not directly applicable in our setting as our calibration sets are different for each test data. Under our stability assumption, we can confirm that e j is a valid e-value in a manner similar to Lemma E.2 in our paper. With this equivalence property, the boosting method can be directly applied to our deterministic pruning approach by constructing the new boosted e-value with m = | Ŝu | and S j = (D * C∪{j} , D U \{j} ).The Storey's method [49] aims to estimate the null proportion π to increase the detection power.In our setting, the null proportion can be directly estimated by the corresponding proportion in the labeled set, i.e. πOMT = πAMT = |C 0 |/|C| and πSCPV = | Ŝc ∩ C 0 |/| Ŝc |. And when applying BH procedure, we will use a level of α/π instead of α, such that the FDR can be controlled at α exactly.• Abalone [37] : contains easily obtainable measurements of abalone. The task is to predict the age of abalone from physical measurements. We use the shell weight as the selection score. The c 0 we used for this task is taken by 12. • Census [9] : contains census data extracted from 1994 Census Bureau database. We focus on people from America and regard the income attribute as the response of interest, which is a binary variable indicating whether one's income exceeds $50K per year. The feature of age is used as selection score T . • Credit [30] : contains transactions made by credit cards over the course of two days, some being frauds. The task is to identify the frauds and we use the specific feature, amount, as the selection score. Since it contains only 492 samples of class 1, we set the null proportion at 0.9 instead. • Promotion [36] : contains employee's past and current performance and the final promotions.The task is to predict whether a potential promotee at checkpoint in the test set will be promoted or not after the evaluation process. We use the specific feature, average score in current training evaluations, as the selection score.We discuss two additional comparing methods which are nicely suggested by the reviewers.Self-consistent/compliant adjustment (SCA) Using the marginal p-values in 1, one can directly achieve FDR control under any data-dependent selection simply by taking the largest self-consistent rejection set, i.e. the largest subset R s.t. p i ≤ α ′ R/K for each i ∈ R ⊆ Ŝu , where α ′ is the largest value that satisfies π 0 α ′ (1 + log (1/ (π 0 α ′ ))) ≤ α where π 0 = |C 0 | /|C| is the null proportion. This is a direct consequence of Theorem 3 of [51] and the PRDS property of conformal p-values from [8] .We analyze the comparison between our approach and the baseline method from two perspectives.From the theoretical point of view, we have observed that the power loss associated with utilizing a selective conformal p-value is usually less than that incurred by the FDR-Linking method. To illustrate this, assume π 0 = 0.7 and α = 0.1 as in the simulation setting of quantile selection, then we derive α ′ ≈ 0.025. The AMT method adjusts the marginal p-value after selection by multiplying the selection proportion θ = 1/0.7. This is equivalent to employing the BH procedure on the marginal p-value with α = 0.07, which evidently yields greater power than SCA. Additionally, AMT does not make full use of the information from the selection procedure. In contrast, our proposed method uses a smaller p-value than AMT, which suggests more power increase.In terms of empirical performance, as demonstrated in both cases from our paper, the SCA method suffers from a power loss, confirming our theoretical analysis. InfoSCOP Gazin et al. [23] propose a novel method named InfoSCOP, which is closely related to our approach under the joint-exchangeable selection rule. Below, we provide a detailed discussion of their method. The primary objective of InfoSCOP is to select informative prediction sets with false coverage rate (FCR) control, although it is not specifically designed for multiple testing.Their focus is on an informative selective prediction set procedure, denoted as Ŝinfo u ⊂ U, where each prediction set C j is I-informative for every j ∈ Ŝinfo u ⊂ U. If a prediction set C α j is I-informative, then all the prediction sets it contains are also I-informative, and it is right-continuous for the coverage level. By leveraging the property of I-informative selection, they link the FCR control problem with BH procedure, and verify that the FCR control can also imply the following FDR control:E j∈ Ŝinfo u 1{Y j ̸ ∈ ∪ C∈I C} 1 ∨ Ŝinfo u .InfoSCOP's procedure to achieve FCR control consists of two main parts. The first part transforms the informative selection procedure into a specific BH procedure. Let p = {p j } j∈U denote the set of p-values over test set. These p-values are inverted into I-adjusted p-values byq j = min{α ∈ (0, 1] : C α j (p)}.The BH procedure is then applied to the q = {q j } j∈U to obtain a selection set BH(q). The second part involves using the adjusted approach from Benjamini and Yekutieli [12] to construct prediction sets for each selected individual, at the level of α|BH(q)|/m, thereby providing FCR guarantee. To mitigate power loss from this adjustment process, they employ the method from Bao et al. [3] to select an initial subset S 0 ⊂ U which reduces the number of units and allows for a larger adjusted level of α|BH|/|S 0 | to construct prediction sets. To maintain their theoretical guarantee, they require the initial selection to be joint-exchangeable, which connects to our setting.The FCR guarantee of InfoSCOP can directly imply FDR control on a data-dependent selection set by ensuring that the "informative prediction set" is informative with respect to the null hypothesis being tested. In this way, InfoSCOP implements an FDR control procedure after selection by applying the BH procedure to selective conformal p-values, which closely aligns with the core approach of our work.For strongly stable selections, our method can be simplified and degenerate into a form similar to InfoSCOP. But the assumption in InfoSCOP is not satisfied by the quantile selection rule based solely on test data. Thus, their theoretical results are not applicable in such cases, while our framework bridges this theoretical gap.And our approach covers a wider range of selection rules. For instance, when dealing with weakly stable rules, we employ conditional calibration on adaptive p-values to ensure rigorous FDR guarantees. The table below compares the performance of our approach with InfoSCOP under mean selection rule. The InfoSCOP shows reasonable empirical performance, which is similar to ours. Therefore, it is possible that InfoSCOP may still work under mean selection, making it an interesting topic for theoretical investigation, which remains unexplored in InfoSCOP. In contrast, we provide FDR control guarantee under a variety of selection scenarios. In conclusion, our approach and InfoSCOP are designed for different goals, resulting in different analytical frameworks. Ours is specifically designed to address the multiple testing problem across various selection rules. From the perspective of conditional calibration, our method is unified, where the BH procedure for strongly stable selection can be seen as a special case. As a comparison, InfoSCOP is an excellent work for selecting an informative set with FCR control, but it is not primarily designed for multiple testing after data-dependent selection. Their FDR guarantee is an extension of FCR control, which limits their method's applicability to different selection rules. In Table 5 , we present the results of FDR and power under three machine learning methods. The data is generated based on the settings specified in case A. The neural network (Neuralnet) with a single hidden layer and 5 hidden neurons is implemented by using the R package neuralnet. And the linear output units are used. Here, we fix the noise strength σ = 0.5. It can be seen that our method controls FDR at the expected level and it also provides satisfactory testing power. In contrast, the FDR values obtained from AMT tend to be overly conservative, leading to a notable deflation of its power. In the first two settings, the OMT methods are inadequate in effectively controlling FDR. While they can successfully control FDR in the last setting, they often suffer from a loss of statistical power.Diabetes is a chronic disease that affects a large and growing number of people worldwide [65] .As such, identifying potential diabetes patients using risk factors and machine learning tools is an attractive approach for early intervention and preventive measures. To this end, we applied our method to the Diabetes Health Indicators Dataset [38] provided by the Behavioral Risk Factor Surveillance System (BRFSS) in the United States. Through this analysis, we are able to effectively identify high-risk individuals while also providing uncertainty quantification measures.In the dataset, the response variable is denoted as Y j which takes the value of 0 or 1, indicating whether the j-th person suffers from diabetes. The dataset also includes patient-related information consisting of 21 features, such as BMI (Body Mass Index), cholesterol level, and other health risk indicators. These covariates provide additional information about each individual that can be used to analyze and predict the likelihood of diabetes. Our goal is identifying those diabetes patients with controlled FDR α = 20%, i.e.H 0,j :Y j = 0 v.s. H 1,j : Y j = 1.The data is processed as follows: a total of n = 2, 000 labeled data points and m = 2, 000 test data points are randomly sampled from the dataset. The prediction model μ is constructed by random forest using another 2, 000 i.i.d. training data. Based on prior knowledge, we understand that individuals with obesity are at a higher risk of suffering from type II diabetes [54] . Therefore, our focus is directed towards making inferences specifically on individuals with a high BMI. Denote the selected subset Ŝu = {j ∈ U : T j > τ }. Several selection rules are considered. Constant: T i is the BMI of i-th individual. τ = 30. Exch: T i is the BMI of i-th individual and the τ is the 70%-quantile of {T j : j ∈ C ∪ U}. Quan: T i is the BMI of i-th individual. τ is the 70%-quantile of {T j : j ∈ U}.Table 6 depicts the results of our proposed SCPV and other compared benchmarks using the thresholds mentioned above. Our method and adjusted methods successfully achieve valid FDR control using all of these selection rules. However, adjusted methods select fewer individuals, leading to powerless results. Meanwhile, the OMT fails to control FDR for most settings. Besides, the dataset is potentially composed of different groups, and it is important to identify individuals while controlling the FDR for each group. This allows us to make more accurate assessments and informed decisions specific to each group. To address this, we employ a clustering algorithm, specifically K-means, to divide the dataset (which uses both labeled and test data but lacks response information) into two distinct groups. Our primary objective is to draw inferences within each individual group. The clustering process results in two groups that exhibit significant disparities in terms of the "MentalHealth" covariate. Consequently, we refer to the group with a lower "MentalHealth" index as Group A, while the other group is denoted as Group B.Regarding the results for the two clustered groups in Table 7 , we observe that our method exhibits stringent FDR control. The OMT lacks power for Group A and yields an inflated FDR level for Group B. As for these adjusted multiple testing methods, they deliver more conservative rejection results. To conclude, our method is powerful to provide subgroup FDR control for adaptively chosen groups. Breast cancer is the most common form of cancer in women, with infiltrating ductal carcinoma (IDC) being the most common form of breast cancer. Accurately identifying and classifying subtypes of breast cancer is an important clinical task, and utilizing deep learning methods for identification can effectively save time and reduce errors. Our dataset consists of complete whole slide images of breast cancer (BCa) specimens scanned at 40 times magnification. Our method can effectively identify individuals who may be at risk of breast cancer, while also measuring the uncertainty of the deep learning model.In this dataset, the label is denoted as Y, with Y taking values of 0 and 1, representing whether the j-th image is a slice from a breast cancer specimen. Our goal is identifying those breast cancer patients with controlled FDR α = 10%, i.e. H 0,j :Y j = 0 v.s. H 1,j : Y j = 1.The data is processed as follows: a total of n = 800 labeled data points and m = 800 test data points are randomly sampled from the dataset. The prediction model μ is constructed by a convolutional Neural Network with 10 layers using another 2000 i.i.d. training data. This network consists of a total of 10 layers, including 4 convolutional layers and 3 max pooling layers.T i represents a score used to assess the risk of breast cancer, such as the probability of developing breast cancer predicted by a model. Our goal is to identify individuals in the high-risk group who are more likely to develop the disease. Denote the selected subset Ŝu = {j ∈ U : T j > τ }. Several selection rules are considered. Constant: T i represents the predicted probability of the i-th individual obtained using model μ. τ = 0.2. Exch: T i represents the predicted probability of the i-th individual obtained using model μ and the τ is the 30%-quantile of {T j : j ∈ C ∪ U }. Quan: T i represents the predicted probability of the i-th individual obtained using model μ. τ is the 30%-quantile of {T j : j ∈ U}.Table 8 displays the outcomes obtained by our proposed SCPV method and other benchmark approaches when employing the specified thresholds. Our method and adjusted methods successfully achieve valid FDR control under these criteria. However, adjusted methods select fewer individuals, resulting in powerless results. Meanwhile, the OMT proves inadequate in controlling FDR for most settings. C 0 = {i ∈ C : Y i ̸ ∈ A} and U 0 = {j ∈ U : Y j ̸ ∈ A}. Equivalently, {Y i } F. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The datasets used in our paper are all correctly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The reviews are strong and agreement is high. The reviewers clearly appreciated 
the author rebuttal and adjusted their scores accordingly. The material provided
during the rebuttal phase clearly improves the manuscript. With that in mind,
I would strongly encourage the authors to incorporate these points and suggestions into their
revision.

Reviews: review: paper_summary: paper_summary: The authors study the validity of Benjamini-Hochberg like procedure on conformal p-values computed on data selected with a particular rule. Assumptions on the rules (and examples verifying them) are mentioned and guarantees demonstrated in those cases. Experiments on synthetic and classical real data and also conducted.
strength_weakness: strength_weakness: I'm surprised by the lack of references to Vovk's work in particular, having studied conformal p-values and conformal testing for a long time.
Theoretically, the weakness is due mainly to the weak rule setting, although it is not a significant issue.
However, I think the experimental part is the most limiting here, being limited to a very classical testing setting in a conference where applications related to deep learning would be of interest.
questions: questions: Could you compare or clarify the contribution with regards to the conditional calibration scheme mentioned in the article ?
limitations: limitations: The approach taken in the weak selection rule has a lower power.
paper_summary: paper_summary: This paper considers the problem of sample selection among a pre-specified group. The authors formulate this problem as a multiple testing problem and develop a procedure based on conformal inference, in which special treatment is adopted to find a specific calibration set that is exchangeable to the selected test unit. The proposed method achieves FDR control when the pre-selection rule has some strong exchangeability property.
strength_weakness: strength_weakness: **Technical contribution.** As introduced in the paper, the technical difficulty of sample 
selection among the selected units with FDR control lies in (1) constructing valid p-values in the 
presence of selection and (2) dealing with the dependency between p-values for FDR control. 
The current paper mainly addresses the first problem, while the second problem is only parially solved 
for quite restrictive selection rules. In fact, finding the exchangeable group/constructing 
valid conformal p-values for selected units has already been quite extensively discussed in [1] and [2]. 
It would be helpful to clarify the technical contribution given this context.  

1. Jin, Ying, and Zhimei Ren. "Confidence on the focal: Conformal prediction with selection-conditional coverage." arXiv preprint arXiv:2403.03868 (2024).
2. Bao, Yajie, et al. "CAS: A General Algorithm for Online Selective Conformal Prediction with FCR Control." arXiv preprint arXiv:2403.07728 (2024).
questions: questions: See the "Weaknesses" section.
limitations: limitations: The paper has partially discussed its limitations.
paper_summary: paper_summary: This paper proposes a method for multiple testing in the conformal setting that outputs the largest possible rejection set with FDR control contained with a data-dependent selection. The proposed method involves two steps: 
(1) constructing selective conformal p-values (i.e., p-values solely use points in the calibration set that are chosen by the selection rule),
(2) applying the Benjamini-Hochberg (BH) procedure to the selective conformal p-values, but using a total hypothesis size of the selected set $|\widehat{\mathcal{S}}_u|$.

The method ensures FDR control only when the selection rule satisfies a form of "stability", which the authors provide several definitions for. Further, they provide several examples of practical selection rules which satisfy the stability condition (e.g., top-K, quantile), which either depend solely on the covariates in the test set, $\mathcal{D}_u$ or along with the covariates of the calibration set $\mathcal{D}_c$.
strength_weakness: strength_weakness: I think the main weaknesses of the paper is that it lacks comparison to two key baselines/prior art.

1) Self-consistent/compliant adjustment: Using the marginal p-values in (1), one can directly achieve FDR control under any data-dependent selection simply by taking the largest self-consistent rejection set, i.e., the largest subset $\mathcal{R}$ s.t. $p_i \leq \alpha' \mathcal{R} / K$ for each $i \in \mathcal{R} \subseteq \mathcal{S}$, where $\alpha'$ is the largest value that satisfies $\pi_0 \alpha' (1 + \log(1 / (\pi_0\alpha'))) \leq \alpha$ --- here $\pi_0 = |\mathcal{C}_0| / |\mathcal{C}|$ is the null proportion. This is a direct consequence of Theorem 3 of [5] and the PRDS property of conformal p-values from [2] referred to in your paper.

Note that this is *not* the same as your AMT baselines --- those work with the selective conformal p-values (which I presume are less powerful than the marginal conformal p-value in (1)). I think understanding the performance of your procedure to this procedure would be key to seeing how the tradeoff between the gain in power from using a smaller BH threshold vs. loss in power from using a selective conformal p-value in your method compares to a method that purely uses marginal p-values.

Post-rebuttal update: the authors ran these experiments and their method performs well. I have changed my score to an accept.

2) InfoSCOP [3]: Although InfoSCOP is cited in the paper, it is not compared against --- the FCR guarantee of InfoSCOP directly implies FDR control on a data-dependent selection set simply by making its "informative prediction set" be informative against the null hypothesis being tested (the types of sets it is informative against are precisely the types of null hypotheses you are interested in testing against). In this vein, I think a more detailed comparison of your method and the InfoSCOP method is needed (i.e., what type of selection rules are allowed for each method, and how the power differs, etc.) for this paper to be comprehensive.

I think the paper should include comprehensive comparisons to these rather significant baselines/prior art to merit publication.
questions: questions: The assumption of having access to a calibration set and still trying to perform selective inference in a conformal setup is a bit strange. One can directly estimate a population level threshold for selecting units based (with false positive error control) on the calibration set directly, since every unit (including the test units) is drawn from the same population --- this is what allows the conformal inference to succeed in this setup. This is quite different from the outlier detection application introduced in [2], where there are units/hypotheses where the distribution is not the same as the calibration setup, and hence one cannot estimate a population level threshold, since there is no singular notion of population. 

Could you elaborate on why the conformal setup makes sense here, instead of trying to directly estimate a rejection cutoff for $T(X_i)$ so the false positive rate is controlled at the population level?


- [1] Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou. Selective conformal inference with false coverage-statement rate control. Biometrika, 2024.
- [2] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 2023.
- [3] Ulysse Gazin, Ruth Heller, Ariane Marandon, and Etienne Roquain. Selecting informative conformal prediction sets with false coverage 
rate control. arXiv:2403.12295, 2024.
- [4] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.
- [5] Weijie Su. The FDR-Linking Theorem. arXiv:1812.08965, 2018
limitations: limitations: The limitations are well addressed.
paper_summary: paper_summary: This paper addresses the problem of conformalized multiple testing following data-dependent selection procedures. To manage the distorted distribution resulting from the selection process, the authors propose adapting the calibration set according to the selection rule. Under the assumption that the selection procedure is stable or weakly stable, the authors prove that the guarantee of the Benjamini-Hochberg procedure is still maintained. Experiments on synthetic and real data have shown the effectiveness and efficiency of the method.
strength_weakness: strength_weakness: 1. The methodological novelty is limited: the idea of adapting the calibration set with the selection strategy to retain exchangeability is not very novel. Although dealing with conformal prediction, [1] also constructs the reference set based on the selection rule. The investigated selection methods are also similar. Overall, the main contribution of this method may be the rigorous proof (which, due to the length, I was unable to verify in every detail within the time constraints).

2. Minor errors:

    a. Line 28, $D_u = \{Z_i\}$ should be $D_u = \{X_i\}$.

    b. Line 153, 'label set' should be 'labeled set'.

[1] Ying Jin and Zhimei Ren. Confidence on the Focal: Conformal Prediction with Selection Conditional Coverage.
questions: questions: May I ask if the idea of adapting the calibration set is similar to constructing the reference set in [1]?
limitations: limitations: The authors have adequately addressed the limitations of the i.i.d. setting and noted that the stability assumptions of the selection procedures are relatively limited.

Comments: comment: We greatly appreciate your efforts in reviewing our work and increasing your confidence. If there are any additional insights or suggestions you would like to share, we are eager to hear them. Thank you once again for your support.
Thank you for your detailed response and addressing my concerns. I have a clearer understanding of conditional calibration as compared to your work. I moreover appreciate the additional experiments. I maintained my rating but increased my confidence in it.
Thank you so much for taking the time to review our work and raising your score. If there's anything else you would like to discuss or inquire about, please don't hesitate to reach out. Once again, we appreciate your support.
Many thanks for the review and raising your score. If you have any other questions, concerns, and comments, please let us know. We would like to provide our responses and address them in the future revision. Thank You!
> Comment: Describe the relationship of your conditional calibration approach with the application of boosting with conditional calibration in Section 6 [1] as applied to conformal multiple testing. Can your procedure be seen as a generalization or specific case of their derandomization (or even randomized) approach to getting FDR control? 

**To C**: Thank you once again for updating the score and providing your insightful comments.

- Indeed, our procedure can be viewed as a generalization for a selective scenario of their approach [1]. The conditional calibration approach with random pruning is equivalent to the e-BH procedure applied to $\{e_j/\epsilon_j:j\in\hat{\mathcal{S}}_u\}$, where $e_j=\frac{\hat{\mathcal{S}}_u\mathbb{I}(p_j\leq\frac{\alpha\hat{R}_j(\mathbf{p})}{\hat{\mathcal{S}}_u})}{\alpha\hat{R}_j(\mathbf{p})}$ and $\epsilon_j$ are independent standard uniform random variables. Additionally, our approach with deterministic pruning is equivalent to the e-BH procedure applied to $\{e_j:j\in\hat{\mathcal{S}}_u\}$. 
- Under our stability assumption, we can confirm that $e_j$ is a valid e-value in a manner similar to Lemma E.2 in our paper. However, exploring the property for general selection rules remains a task, as $\hat{\mathcal{S}}_u$ is a random variable correlated to $p_j$. 
- With this equivalence property, the boosting method [1] can be directly applied to our deterministic pruning approach by constructing the new boosting e-value to enhance power. This boosting method enhances the power of e-BH without sacrificing its FDR control or introducing additional randomness. This excellent work can significantly improve the reproducibility of results from our conditional calibration approach. We will add this discussion to the article in future versions.

[1] J. Lee and Z. Ren. Boosting e-BH via conditional calibration. arXiv, 2024.
Thanks to the authors for addressing my questions. I have raised my score.
Thank you for clarifying your contribution! I will raise my score to 6.
Thank you for running experiments comparing the methods and thoroughly explaining the differences with existing work. --- I have updated my score to accept.

As a final comment, I think it would be helpful to describe the relationship of your conditional calibration approach with the application of boosting with conditional calibration in Section 6 [1] as applied to conformal multiple testing --- can your procedure be seen as a generalization or specific case of their derandomization (or even randomized) approach to getting FDR control?

[1] J. Lee and Z. Ren. Boosting e-BH via conditional calibration. arXiv, 2024.