Conformalized Multiple Testing after Data-dependent Selection
Authors: Xiaoning Wang
Yuyang Huo
Liuhua Peng
Changliang Zou
Venue: NeurIPS 2024 poster (2024)
Keywords: multiple testing
conformal p-value
conformal inference
selective inference
distribution-free
Primary area: safety_in_machine_learning
TL;DR: We propose to construct the Selective Conformal p-value, which takes into account the selection effects to guarantee the false discovery rate in the predictive setting

Abstract: The task of distinguishing individuals of interest from a vast pool of candidates using predictive models has garnered significant attention in recent years. This task can be framed as a *conformalized multiple testing* procedure, which aims at quantifying prediction uncertainty by controlling the false discovery rate (FDR) via conformal inference. In this paper, we tackle the challenge of conformalized multiple testing after data-dependent selection procedures. To guarantee the construction of valid test statistics that accurately capture the distorted distribution resulting from the selection process, we leverage a holdout labeled set to closely emulate the selective distribution. Our approach involves adaptively picking labeled data to create a calibration set based on the stability of the selection rule. This strategy ensures that the calibration data and the selected test unit are exchangeable, allowing us to develop valid conformal p-values. Implementing with the famous Benjamini-Hochberg (BH) procedure, it effectively controls the FDR over the selected subset. To handle the randomness of the selected subset and the dependence among the constructed p-values, we establish a unified theoretical framework. This framework extends the application of conformalized multiple testing to complex selective settings. Furthermore, we conduct numerical studies to showcase the effectiveness and validity of our procedures across various scenarios.

Conformalized Multiple Testing after Data-dependent Selection
Xiaoning Wang"* Yuyang Huo Liuhua Peng
Changliang Zou" Schooo of Statistics and Data Sciences, LPMC, KLMDASR and LEBPS,
Nankai University, Tianji,, China
School of Mathematics and Statistics, The University of Melbourne, Melbourne, Australia
1 1120220048@mail nankai edu. on.hooyynk@gail.. com wuuuuua.penguuimmell.... edu. au, wwwclenankeaaae edu.
Abstract
The task of distinguishing individuals of interest from a vast pool of candidates using predictive models has garnered significant attention recent years. This task can be framed as conformalized multiple testing procedure, which aims at quantifying prediction uncertainty by controlling the false discovery rate (FDR) via conformal inference. In this paper, we tackle the challenge of conformalized multiple testing after data-dependent selection procedures. To guarantee the con- struction of valid test statistics that accurately capture the distorted distribution resulting from the selection process, we leverage a holdout labeled set to closely emulate the selective distribution. Our approach involves adaptively picking labeled data to create calibration set based on the stability the selection rule. This strategy ensures that the calibration data and the selected test unit exchangeable, allowing us to develop valid conformal p-values. Implementing with the famous Benjamini-Hochherrg (BH) procedure, it effectively controls the FDR the lected subset. To handle the randomness of the selected subset and the dependence among the constructed p-values, establish unified theoretical framework. This framework extends the application of conformalized multiple testing to complex selective settings. Furthermore, we conduct numerical studies to showcase the effectiveness and validity of our procedures across various scenarios.
Introduction
In recent years, there has been a notable focus on the use of predictive models to distinguish specific individuals from pool of candidates. For instance, in the field of financial investment [17 2) machine learning models can be used to predict profits for different investment opportunities. Candidates with high predicted profits can then be given more preference and considered potential investment options. Similarly, in disease diagnostics (29 (45, researchers can utilize relevant information and corresponding predictions from machine learning models to identify potential patients.
In a typical scenario, we are presented with labeled/holdout data set D. (Z: = (X,, Y where X; R is the observed covariate and Y; IR is the response, and an unlabelled/test set -=n+1 [xt In practice, we only observe the covariates in the test set D,, and the responses Y are unknown. Our goal is to distinguish individuals in D., whose undisclosed responses fall within predetermined region A. Region A can take various forms, such as (b, xc). [a, bj and (-oo, a) per user's requirements. To estimate the value Y for the identification, we employ predictive model i d R. However, directly using the black-box prediction ja( substitute for Y; leads to
D.
"qqual contribution. Correspondence to: Liuhua Peng prng Changliang Zou <zoucl ndn..i.edu.cc..
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

inherent uncertainty. In order to quantify this uncertainty, we reformulate our problem as multiple hypothesis testing [0]: for j = {n 1, n + m))
Ho. Y3 A v.s. Hi,; E.
Under this framework, we wish to make rejections as much as possible with the false discovery rate (FDR) controlled at pre-given level a. Denote the index set of test data as U = {n+ 1, n + m). The FDR is defined as the expectation false positive proportion (FDP) over the test units in U, i.e.
FDR(U) = E[FD(())) FDP(U) = -Ejeu 10j E R. AA}
1V [V
where we denote a b= max(a, b} for any a, b, E R. [S] as the cardinality of a set S and R as the rejection set. To derive the testing rule, one feasible method is to construct the conformal p-value [ by ranking the nonconformity score associated with the test unit's prediction among the scores in the holdout set. Then we can apply the well-known conformal p-values to obtain a rejection set with controlled FDR. Here we term this procedure as "conformalized multiple testing".
(BH) procedure [10] on these
In practice, researchers may be interested in specific subsets rather than analyzing the entire dataset. For example, they might aim to determine the presence or absence of lung cancer among heavy smokers. By establishing data-driven criteria or thresholds based on factors such as the daily number of cigarettes smoked and smoking history, researchers can create a filtered subset consisting of heavy smokers. This allows researchers to gain insights into the patterns of the medical condition within this particular group. Here we address that the group partition may not be predetermined and could instead be learned from data, through methods like clustering (25) or thresholding. Denote the selected subset from unlabelled data as S,, C U. our paper, we aim to find rejection set R, C S,. with the following FDR criterion controlled at at, i.e.
FDR = Eies, 1 Ru, Y3 & A}
S aa.
1 Rinl
For simplicity, we use FDR to denote FDR(S.) only. The selection procedure would distort the distribution of test statistics, invalidating the p-values and leading to the failure of FDR control. This falls into the category of selective inference, which has been addressed in both statistics and machine learning fields [53 [6] To tackle the selective issue, the use of labeled data becomes crucial. By leveraging labeled data, it is possible to obtain the conditional distribution of the selected data, which in turn allows for the construction of valid p-values.
Nevertheless, even with valid p-values, controlling the FDR proves to be a challenging task. This difficulty arises from the inherent randomness of the selected subset S., Even a minor disturbance in S, can lead to significant changes in the final rejection set Ry,. Consequently, in order to address this issue, we focus on several commonly used selection rules with certain selection stability. We employ an adaptive strategy to carefully choose labeled data, thereby creating calibration set that takes into account the selection stability.
1.1 Our contributions
In this paper, we construct the selective conformal p-value for each selected individual, built on the marginal conformal p-value []. To address the selection effects, we adaptively pick a calibration set from the labeled data according to the selection rule, to ensure the exchangeability between the test data and labeled data. The selective conformal p-values are then constructed using the picked calibration set. By combining the selective conformal p-values with the well-known BH procedure, we achieve FDR control after data-driven selection, as verified through our comprehensive analysis.
The main contributions of our paper can be summarized as follows.
Firstly, frame the problem of multiple testing after data-dependent selection in the predictive setting and propose a viable solution utilizing the labeled data.
Secondly, the proposed method achieves exact FDR control for selection rules with strong stability, including exchangeaceee rules and the top-K K selection. And we further extend

---

our method to handle more general cases where the selection rules satisfy a weaker stability condition such as sample mean selection.
Thirdly, the theoretical advancement extends the scope of classic multiple testing into the selective setting, providing a unified analytical technique for handling the randomness arising from data-driven selection.
Finally, through extensive experiments, we evaluate the reliability of our method in delivering the desired FDR control, and emphasize its easy integration with various algorithms.
1.2 Connections to existing works
Multiple testing Ever since the seminal work of Benjamini and Hochberg [10] the framework of
multiple testing has been well developed by many researchers 44 5
66
20 44 Our paper connects
to the area of two-stage testing [62), which firstly selects a subset of hypotheses and subsequently applies a multiple testing procedure to the selected set. To maintain the validity of test statistics after the selection, Bourgon et al. [14] recommended using an independent statistic specifically for the purpose of selection, but it is unavailable in predictive setting. Instead of assuming independence between the test statistic and the selection statistic, Du and Zhang [19] introduced the concept of "single-index" p-value for the joint modeling of both statisties and provided FDR control under symmetry assumption. Besides, Efron [21] considers applying multiple testing procedures over pre-given groups guarantee group-wise FDR control, while our work considers the FDR control over data-driven subgroups, which is more challenging.
has garnered significant attention in recent years, which leverages data exchangeability to construct modellaggosste prediction intervals.
Conformal inference Conformal inference |5
40
56
55
We present recent developments therein 39
15 [. Within the conformal inference framework, several studies have focused on controlling the FDR in predictive setting, i.e. the conformalized multiple lesting 8 26 These studies involve constructing valid testing procedure using a holdout set. Such procedures include the BH procedure based on conformal p- values [27 35 34] 24). thresholding via an FDP estimator 68 41 and the e-BH procedure applied to generalized e-values 77 [64). Different from them, we focus on a selective scenario in conformalized multiple testing, addressing new challenges arising from selective randomness.
6
59
61
50
[8
Selective inference Selective inference concerns the inference problem after data- detaddependent process- ing. Previous works have mainly focused on the inference of parameters [60 31. Recently, Bao et al. 3 extended selective inference to the realm of conformal inference. They proposed a method to con-
struct selection conditional prediction intervals with controlled false coverage-statement rate (FCR)
12 after data selection. Building upon this work, selective conformal inference with FCR control the online setting 4. In particular, under a certain class of selection rules, Gazin et al. [23 involves a procedure for
has been further extended to accommodate more general selection rules 23
28
FDR control after selection which closely aligns with our method. However, their approach focuses on selecting an informative set with FCR control under specific selection assumptions, limiting its applicability in more general scenarios such as those with data-dependent selection. The problem we tackle presents additional complexities, due to the intricate dependence on the selection procedure and final decision procedure, requiring a more intricate and delicate analysis. Besides, Sarkar and Kuchibhotla [47) proposed post-selection framework to guarantee simultaneous inference [3] for all coverage levels, which differs greatly from our scenarios.
2 Methodology
2.1 Recap: conformalized multiple testing
We first introduce how to make multiple testing in the predictive setting. Denote the index sets for the labelled data D. as C = {1, .I....... n2.. Suppose fi(-) is predictor via machine learning algorithm that is pre-given or can trained on extra labeled data. Thus we can treat f(xx)) as fixed. To construct a valid test statistic based on j(X;). Bates et al. [ considered to use conformal p-values built upon the conformal inference framework [56]. Consider monotone transformation V such that the larger value of V (i(X))) indicates the bigger likelihood of Y & A. For example, if A (b, oo), we can
3

---

use V(v) - y. The marginal conformal p-value p. for Xj is defined as
+
(i E Co
S ViH
Pj
j jel;
(I)
1 +
Col
where we denote V (()(x;)) the nonconformity score for j-th sample and Co {i : Y; E A} as the index set of labeled set containing only null samples.
The properties of marginal conformal p-value constructed by l.i.d--labliee and test data have been investigated by Bates al. (8] and we present them in the following proposition. Proposition 2.] (i) guarantees that marginal conformal p-value is superuniform, thus it is valid p-value. As the conformal p-values have a nice dependence structure, the rejection set obtained by the famous BH procedure I0| enjoys valid FDR control Proposition 2.1 (ii) indicates. For a set of p-values {pr Jm the BH procedure finds & max
P() - ja/m/ where P()) denotes the j-th smallest p-value in {pj ) and obtain the rejection set as R. = {j Pj I ka/m).
Proposition 2.1 (Properties of the conformal p-value 8 Suppose the labeled data and test data are i.i.d. For simplicity, we assume Uy = [j 4 E A) {n +1, 1. ,
mo for my S m.
The conformal p-values in ! satisfy:
(i) The Pj a marginally superuniform p-value, i.e. for any t, Pr(p, st|jEu)) tt
(ii) Furthermore, the BH procedure applied level a on the conformal p-values (o)) Jieu controls the FDR level at wo, where is the null proportion of test samples.
The property of the conformal p-value is obtained by the exchangeability between the labeled data and test data. Through this, can approximate the distribution of V,, where j My using V; from i Co. Therefore, when the labeled data and test data have different distributions, maintaining the exchangeability becomes crucial in constructing valid conformal p-values.
As Storey 48 suggested, we can further estimate the null proportion and incorporate it in the BH procedure to further increase detection power. With the aid of labeled data, the null proportion can be directly estimated by the corresponding proportion in the labeled set, i.e. Ω = |Col/ICl.
2.2 Selective conformal p-value
selection procedure could possibly be employed to the test samples. this case, the focus lies primarily the selected subgroup rather than the entire dataset, and decisions are made solely based on this subset. Define the selection rule as SD..D,, which is a function of labeled set DD and test set Du. For simplicity, may omit this subscript. The selection rule maps individual point X into selection decision (0, 1). And the selected subset can be written S. {j 4 S(X,) 1).
There are many examples for data-dependent S. The S can be the clustering algorithm which automatically determines the subgroup. Alternatively, S is associated with the selection score T₁, which derived from certain components of Xj. and a selection threshold т. such as the sample mean value in (Ti)jeu- In this case, we can express (S(X,) = = (T, < 1 ΣkEU Tt).
After the selection procedure, we would make multiple testing on the selected subset S.,- However, the distribution of the selected conformal p-values (I) would be distinct from the original ones, due to the selection effects. Consequently, directly running the BH procedure on the marginal conformal p-values ( in selected set S, has no guarantee, which may lead to an inflated FDR level or poor power. Addressing this issue raises two important considerations:
How to characterize the selection conditional distribution of the selected individuals to construct valid p-values?
How to take account of the dependence structure among the valid p-values and the stochastic nature of the selection event design trustful multiple testing procedure?
The first issue is widely considered in post-selection inference. Previous literature heavily relies on the normality assumption derive the conditional distribution of test statisties [2] 52]. By the spirit of conformal inference, consider constructing the selective conformal p-value by picking up the

---

calibration set via the same selection rule, thereby guaranteeing the exchangeability between the selected test unit and picked calibration data.
To be specific, we employ the selective algorithm on the labeled set to derive the picked calibration set Se = ii E C S(X,) = 1}. If S involves a selection threshold т, we will choose the calibration set as S. (i E C S r). We hope (V; : S.) U V, for j S.. exhibits a certain level of exchangeability, enabling us to capture the distribution of Vj-
With the aid of S,, the selective conformal p-value can be accordingly constructed as:
Pj = + |i (. Vi S S.HH 1 1+ 1+1 ncol
for j E S..
After obtaining valid p-values, ensuring the guarantee of the BH procedure is not straightforward due to the dependence arising from the use of the same calibration set in computing conformal p-values and the randomness from the data-dependent selection. Therefore, the second concern needs to be carefully addressed. In this article, we examine the BH procedure applied to the selective conformal p-values can enjoy finite sample FDR control for several commonly used selection rules. We outline our procedure in Algorithm and refer to our method as Selective Conformal P- P-Vulue (SCPV).
Algorithm 1 Selective conformal p-value with BH procedure (SCPV)
Input: Labeled set De, test set Du, selection procedure Sp.,Pu, prediction model pc-). target FDR level (0, 1).
Step (Selection) Apply the selective procedure SD,,D. obtain the selected subsets S,, and S... Step (Calibration) Compute (V; : S, Co). (V, : $.
Step 3 (Construction) Construct selective conformal p-value for each j E S, as 2
Step (BH procedure) Compute ks* = max(k: Ejes, 1p; S ak/m) 2 k}
Output: Rejection set Rr {j S. Pj S ak" /m].
3 Theoretical guarantee
In this section, we aim verify the FDR guarantee of Algorithm I for several commonly used selection rules. Our focus here is to tackle the technical challenges associated with the selective multiple testing problem. Unlike the conventional approach where fixed number m of test units is considered, we encounter a challenge due to the involvement of a random number of test units [Sa]- This randomness makes the analysis considerably intricate unless we impose certain restrictions on the selection rule. To deal with the selection set S.,, we introduce the concept of strong stability.
Definition 3.1 (Strong stability). Given selection set S, = (j E u S...D. = 1). The selection rule Sp..p. is strongly stable if either of the conditions holds: for any i E CUU and j E S.,
* (Leave out) SD..D. (X,) =
(Replace) D...D. (X.) 1o2.D.M2,J,E4 (X:) for a fixed point z.
Here we define the strong stability of selection rule in two common ways: leaving one point out or replacing one point with fixed value. Many popular selection rules are strongly stable, such as joint-exchangeable rule and top-K selection. Detailed discussions are provided in next subsections.
The strong stability plays a crucial role in our analysis, as it enables us to fix the randomness of the selected set S,. With the strongly stable property, we can perform a delicate analysis for the rejection set from Algorithm I to obtain the theoretical guarantee.
Theorem 3.2. Suppose the data are ii.d. and the selection rule SD..D. is strongly stable. Then the selective conformal p-values defined in (2) satisfies Pr(p, S tli E Sw..j E Uo) S t, and the output R., of Algorithm I satisfies FDR S aB n (oll/ll//... S
5

---

We present the insight of our proof for Theorem 3.2
As a common operation in analyzing FDR we decompose the FDR into the FDR contribution for each j l as
22
35,
1p S a [R .j Su.j E uo}
Ejeu Lij E Ro.j Mo3
FDR = E
Σ E jEU
|..
1V RRul
1 VRRu
By the stability of selection rule, replace (Ru. with decoupled version (R which removes the influence of Py. If given some quantity Φ that blocks most of the nuisance parameters, the p-value Pj has uniform distribution and [T |Sa.l are fixed. Then the FDR control is by
S
(869 |Sal
J E Suij E Uo} Φ5)
FDR
Σ
E
R
Jeu
Σ
E
a|R
Hi E Su. E Lo aE V (R
1
Su n Mo! |Sul
|Sa
jel
By the construction of the conformal p-value, we analyze each selected unit j E S, by conditioning on carefully constructed quantity 1 = (Pout)) D.nDt). It comprises two components: Du the test data with the j-th sample excluded, and Dou( == [Z;; C [i)], the unordered set of labeled data along with the j-th sample. The unordered set provides the order statistics but not the specific ordering, which is a common convention in conformal inference literature 335 34).
Through this approach, we are able to decouple the dependence that arises from the data-dependent selection and the construction of p-values that share the same calibration data. If the selection rule is strongly stable, then |Sa| is fixed given Φ for j E S,, and the selective conformal p-value in (2) is valid. By performing careful analysis of the rejection set R.,, i.e. replacing it with pseudo rejection set (3) that remains fixed given o2 and j E R., we obtain the finite sample FDR guarantee.
3.1 Joint-exchangeable selection
Firstly, we consider the joint-ecchangeable selection procedure. The joint-exchangeable selection procedure applied {X, e C U with exchangeability, i.c. the selection results remain unchanged after any permutation of data in the merged set C. UM, as Definition 3.3 indicates. Definition 3.3 (Joint-exchangee exchannacha selection). The selection procedure S is joint- exchangeaae with respective to the (X; : C UU) if
Sp.,P, X, Sp. (X;) for any i CUM and Dr. Dr that are arbitrary partitions of D. U Du.-
the selection procedure independent of both the laheled and test data, is joint-exchangeable. In the case of the selection with threshold, the joint-xxchangeable selection equivalent
7X1, ((cum)) ((X+1))) X
where T r(D) denotes that is computed using the dataset D. Therefore, the joint-exchangeable selec- tion includes selection using constant thresholds thresholds computed by (T, hecuur exchangeably. We can verify joint-exchangeable selection is strongly stable through the leaving out condition. Proposition 3.4. The joint exchangeable selection procedure SD.,P. D., is strongly stable.
our procedure ensures FDR control for any joint-exchangcable selection rule, which makes the choice of selection rule quite flexible. For example, can perform clustering algorithm on the {X, Jecuer to divide the data into different groups. As special case, our approach aligns with the InfoSCOP proposed by Gazin et al. 23 under the joint-exchangeable selection rule. They proposed a novel procedure for selecting an informative set and also provided FDR control guarantee as extension of their FCR control results.
According to Theorem 3.2
While the joint-exchangeable rule contains various selection strategies, worth noting that many cases involve selection that depends solely on the test data. In the following subsections, we investigate several commonly used selection rules that are determined only by the test data. And the assumption in InfoSCOP 24 is not satisfied under these cases.

---

3.2 Top-K/Quantile selection
Next, we consider the top-K or quantile selection rule, which relies solely on the test data Du- This type of rule is extensively studied in the literature 442 3 28 and is commonly used in practice.
Let Kropk denote the top-K selection threshold, which is defined the (K 11) smallest value in (T j E 44 The top-K rule is equivalent to the quantile selection rule since TtopK corresponds to the (K 1)/m-quantile of the test data, and the threshold for the q-quantile is the [mq]- -th smallest value. The selected test set and the chosen calibration set under the top-K rule are defined as:
Sn = {j E U: :T, < Tiopk): S. =i E C :.T, < Ttopk)
(3)
We can verify that the top-K selection rule is strongly stable by the following proposition. With the support of Theorem 3.2, can ensure FDR control when employing the top-K selection rule.
Proposition 3.5. For top-K selection rule S with threshold Tropk (4), if E S., then
T-+ Tropk (Tu+1. "i "Tat)""""TTTmmm = Tropk (Tn41, ... T -0; Titb-... Tn+m).
Thus top-K selection rule is strongly stable by the replacing condition.
3.3 General extension to weakly stable selection
In this subsection, we consider weakening the strongly stable condition. For example, the mean thresholding rule does not satisfy the strong stability. By the insight of our proof, the key requirement is the property of S, such that we can handle the randomness of the selection event. Hence we define the weakly stable selection rule as follows:
Definition 3.6 (Weak stability). Given selection set S. = {j E u SD..D.(g)) = 1}. We call the selection rule SD.,D. is weakly stable if
Dp..p. D. (X.) Sp.ulz))... for any j E S., and any i E.
The weak stability does not require the Sp..D. (X,) (z, Mz, (X,) hold for E C. Except for the selection rules previously discussed, the commonly used mean selection rule by test data only is also weakly stable, i.c. fj E 24 Tí < Eeeu Ti} = (j E 4 T, < - Eie(()) l.}.
As the weakly stable rule fails to guarantee the exchangeability of the selected calibration set and construction method for selective p- p-values. The selection set is denoted as S., {j Eu SD..D. (X,) 1). By the definition of weakly stable selection, know that S,, = [j E Eu: SD.U(2))..MM2. (X,) = 1} is also true. Specially, if the selection rule determined only by the test data, it holds that Sp.u(z)).D. (-) = Sp. (-),
test set, it motivates to explore
Therefore, we adaptively pick data from the labeled set using the same "leaving out" rule as selecting Zj. For any i C and j E U. (Sp, \2, (X,) 1} and (S,MM (X,) 1} are symmetric X; and X,. Leveraging this, we can pick up the calibration set by
S.U) = (i E C: Sp. MZ, (X,) 1} for j S.
and construct the adaptive selective conformal p-value
1+ ii = 8.6) V VSH
Pj
adapt
(4)
1 (+(.)
For example, the mean selection rule picks up calibration set by S.(j) =
<
m Eken,) T. Our adaptive strategy shares the same goal as the swapping strategy terms of constructing valid p-value after selection. However, in core motivations since ours is directly related to weak stability, leading to a faster computation and more intuitive explanation here. We can verify that p) pdapp is valid since { { (T4ERE..0U)0)) are exchangeable
28 4 in
approach is different from the others
Proposition 3.7.
The adaptive selective conformal p-value pj ddapt for weakly stable selection which determined only by the test data satisfies Pr(p, adapt t j E S..j Ut) S t.
7

---

adapt is based on different calibration set Se()). However, the p-value for each selected test point p; making the dependence structure intricate. Consequently, the BH procedure has no safe guarantee to control the FDR. But we find that BH is robust and can produce satisfactory results empirically.
Although this observation is acceptable, it would be desirable to design a new procedure to guarantee the finite sample FDR control. To remedy this, we employ the conditional calibration framework [22] achieve finite sample FDR control. The overall procedure can be re-framed the e-BH framework [58] and a recent novel approach for boosting the power of e-BH procedure 33
can be employed in
our setting. The details are displayed in Appendix B
Under the weakly stable selection rule, our method differs fundamentally from InfoSCOP (23 in both methodology and theory. Since approach and InfoSCOP are designed for different goals, resulting in different analytical frameworks. Ours is specifically designed to address the multiple testing problem across various selection rules. From the perspective of conditional calibration, our method is unified, where the BH procedure for strongly stable selection can be seen as a special case. As a comparison, InfoSCOP stands out as remarkable work for selecting an informative set with FCR control, but it is not primarily designed for our problem, which limits their method's applicability more general selection rules.
Numerical studies
To demonstrate the wide applicability of the proposed method in Algorithm [ conduct compre- hensive numerical studies. For regression setting, the region for the hypothesis is A = {y y > co where co is fixed constant. For classification, we set A = (1). i.e. the class the target region, and we denote the prediction (i(X) as the predicted probability of Y = 1. The nonconformity score we use to construct conformal p-value for both settings is (p(X))) -X(x).
Since selective multiple testing has not been investigated before, we consider several intuitive methods as comparing benchmarks.
Benchmarks:
SCPV: Our procedure in Algorithm [ Specially, for mean selection rule, we the adaptive p-value in (4) along with the BH procedure. The results for using conditional calibration can be found in Appendix B
OMT: Ordinary multiple testing which constructs the conformal p- v-luue directly as in equation ( for each selected sample based on the entire labeled set.
AMT (BH/BY): An intuitive procedure by multiplying ( with the selection proportion of null samples. As the adjusted p-values have intractable dependence, making the validity of BH procedure suspicious, we also utilize the Benjamini- Yekutieli (BY) [[]] procedure to control the FDR. More details provided in Appendix A.
SCOP: Directly invert the selective prediction interval constructed by Bao et al. S into a test and make decision by whether the Co is contained in the interval. It is designed for regression setting and does not have FDR guarantee. See more detail in Appendix A.2
We also use the Storey's method 49 to increase power. See more information in Appendix C.I
Selection rule: In the numerical studies, we choose the selection statistic T; based specific component of X. The selected subset is S., = ii E U T. f). where ? is the threshold. Three different choices of selection thresholds are considered.
Exchangeable (Exch): 70%-quantile of the first component of X in both labeled set and test set, that is + is the 70%-quantile of (T: i C CUZ).
Quantile (Quan): 70% 70%-quantile of the first component of X in the test set, that is f is the 70%-quantile of (T; : E ().
Mean: the sample mean of the first component of in the test set, that
1 m EyEu" Tj.
Evaluation metries: We empirically evaluate the FDR by averaging the FDP based on selected samples and the power by averaging the proportion of correct selections among all selected alternative test samples, i.c. Power
li e S. : E R, col/li e S,, Y, col over 100 independent runs.
8

---

4.1
Results on synthetic data
Method
SCPV
OMT
AMT(BH)
AMT(BY)
SCOP
Exch
Quan
Mean
Exch
Quan
Mean
20
100
Case
80
Case
10
5
60
FDR(%)
Power(%)
0-
90
20
15
Case
80
Case
5
0
0.2
0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
0.2 0.5
0.8
0.2
0.5
0.8
0.2
0.5
0.8
Noise S Strength
Noise Strength
Figure 1: Empirical FDR (left) and Power (right) of five methods under different scenarios and selection rules. The Noise Strength varies from 0.1 to 1. The black dashed line in the left plot denotes the target FDR level a 10%.
In synthetic studies, we generate i.i.d. 10-dimensional covariates from X, Unif([-1, 1])10, The corresponding regression responses are generated as Y; ((X)) + c,, where denotes independent random noise. The following data-generating scenarios are considered:
Case A: The data generating model is μ(X) 4(x()) + (LXX() > -0.4] - (() - x)l{x(2) (2) < -0.4). The noise is NN.0 independent of X. And co = 2. Case B: p(X) = 1(x > 1.5), where 3 = (1,-1,2, -2,0,0,0,0,00 0, 0). The noise is 6, ~ N(000.1lo) independent of X. And co 0.12.
We fix the labeled data size regression models (0-) on an additional labeled set with size 1, 200 using the random forest algorithm, implemented by R package randomFoo Forest with default parameters. Specifically, for both scenarios. we select the first component of X as the selection statistic, i.e., T: X!")
1, 200 and the unlabeled data size m 1, 200. We fit the
Figure D displays the FDR (left) and power (right) through varying noise strength. Across both settings, SCPV can deliver valid FDR control. As expected, the OMT fails to control FDR. This can be understood since the OMT constructs conformal p-values without consideration of the selection procedure, leading to smaller p-values possibly. Moreover, our method demonstrates greater statistical power compared to AMT. This is because AMT does not make full of information from the selection procedure. Meanwhile, SCOP fails to control FDR case B. And even if SCOP can control FDR in case A, the accompanying loss of power is substantial. This is because the SCOP is not designed for multiple testing and can not deliver valid FDR results.
4.2 Results on real data
We consider several real data experi- ments including both regression (Reg) and classification (Cla) settings. We sum- marize the datasets in [I The test samples and labeled samples are con- structed by subsampling the dataset with = 1000 and m 2000, and the null proportion is fixed by = 0.8. We sam-
Table Summary of real-world datasets for conformal- ized multiple testing
Abalone 377
Census
Credit 30)
Promotion 36
#Features #Instances
8
14
30
12
4.177
48,842
284,808
54,809
Task
Reg
Cla
Cla
Cla

---

ple another 1000 samples to train ran- dom forest model for classification and regression. See more details in Appendix C.2 The results are reported in Table 2 The AMT(BY) outputs null rejection set most cases, hence we omit it. As expected, SCPV achieves highest power among methods controlling the FDR, verifying its effectiveness and validity.
Table 2: Empirical FDR (%) and Power (%) with target FDR 10%. The bracket contains the standard error (%). The highest power among methods controlling the FDR bolded.
EXCH
QUAN
MEAN
DATASET
METHOD
FDR
POWER
FDR
POWER
FDR
POWER
SCPV
6.68(0.75)
11.3(0.13)
6.61(0.75)
11.2(1.3)
6.37(0.73) 7.92(0.96)
ABALONE
OMT
24.8(0.51)
51.7(0.86)
24.9(0.51)
51.8(0.88)
20.1(0.65) 30.9(1.2)
AMT(BH)
5.12(0.65)
8.30(1.0)
5.13(0.65)
8.30(1.1)
6.09(0.71)
7.50(0.90)
SCPV
7.12(0.80)
15.3(1.5)
7.04(0.75)
15.2(1.5)
7.20(0.69)
15.7(1.3)
CENSUS
OMT
13.9(0.68)
30.3(1.2)
14.0(0.68)
30.4(1.2)
14.7(0.57)
32.6(1.1)
AMT(BII)
2.55(0.49) 5.48(0.86)
2.48(0.49)
5.26(0.85)
6.63(0.65)
12.4(1.1)
SCPV 8.85(0.54) 85.9(0.33)
8.85(0.54) 85.9(0.33)
9.15(0.72) 84.5(0.35)
CREDIT
OMT 12.6(0.62) 86.7(0.33)
12.7(0.61) 86.7(0.33)
14.9(0.70)
85.9(0.36)
AMT(BH)
3,46(0.31) 84.9(0.22)
3.46(0.31) 84.9(0.32)
2.77(0.28) 83.0(0.38)
SCPV 7.44(0.61) 19.6(1.1)
7.67(0.62) 19.7(1.1)
7.55(0.64) 14.3(0.90)
PROMOTION
OMT 19.7(0.68) 35.7(0.65)
19.7(0.69)
35.6(0.66)
18.3(0.68) 25.4(0.58)
AMT(BH) 5.33(0.52) 16.5(1.0)
5.45(0.52) 16.8(1.0)
6.07(0.57) 13.1(0.85)
Limitations and discussions
Here we point out the current limitations of our paper and discuss the potential directions. First, our work relies the i.i.d. assumption. Exploring the selective multiple testing problem in scenarios where the labeled set and test set exhibit different distributions would be interesting. Second, we require the selection rule be stable for theoretical guarantee. It would be attractive to consider complex selection procedures that lack stability, such as clustering based on test data only.
Acknowledgments and Disclosure of Funding
We thank anonymous area chair and reviewers for their helpful comments. Zou was supported by the National Key R&D Program of China (Grant Nos. 1022FA1000303. 2022YFA100380), the National Natural Science Foundation of China (Grant Nos. 11925106, 12231011, 11931001, 12226007, 12326325).
References
[1] Jiahao Ai and Zhimei Ren. Not all distributional shifts are equal: Fine-grained robust conformal inference. Proceedings of the 41 International Conference on Machine Learning, volume
235 of Proceedings of Machine Learning Research, pages 641-665. PMLR, 21-27 Jul 2024.
[2] Arvind Ashta and Heinz Herrmann. Artificial intelligence and fintech: An overview of opportu- nities and risks for banking, investments, and microfinance. Strategic Change, 30(3):111-222, 2021.
[3] Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou. Selective conformal inference with false stterage-statementt rate control. Biometrika, 11 1 2024.
10

---

[4] Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou. Cap: A general algorithm for online selective conformal prediction with fer control. arXiv preprint arXiv:2403.07723, 2024.
[5] Rina Foygel Barber and Emmanuel J Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055-2088 2015.
[6] Rina Foygel Barber, Emmanuel J Candès, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51(2):816-84) 2023.
[7] Meshi Bashari, Amir Epstein, Yaniv Romano, and Matteo Sesia. Derandomized novelty detection with fdr control via conformal e-values. Advances Neural Information Processing Systems, volume 36, pages 65585-65599 Curran Associates, Inc., 2023.
[8] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 51(1):149-175 2023.
[9] Barry Becker and Ronny Kohavi. Adult income investigation. UCI Machine Learning Repository https://archive ics. uci aclu/aataast/2/aduutt 1996.
[10] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series (Statistical Methodology), 57(1):289-300 1995.
[11] Yoav Benjamini and Daniel Yekutieli. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29(2):1165-1188 2001.
[12] Yoav Benjamini and Daniel Yekuticli. False discovery raje-adjusted multiple confidence intervals for selected parameters. Journal of the American Statistical Association, 100(469): 71-81, 2005.
[13] Richard Berk, Lawrence Brown, Andreas Buja, Kai Zhang, and Linda Zhao. Valid post-selection inference. The Annals of Statistics, 41 837, 2013.
[14] Richard Bourgon, Robert Gentleman, and Wolfgang Huber. Independent filtering increases detection power for high-throughput experiments. Proceedings of the National Academy of Sciences, 107621):546-9551, 2010.
[15] Emmanuel Candès, Lihua Lei, and Zhimei Ren. Conformalized survival analysis. Journal of the Royal Statistical Society Series Statistical Methodology, 85(1):24 45: 2023.
[16] Yiqun T Chen and Daniela M Witten. Selective inference for k-means clustering. Journal of Machine Learning Research, 24(152))1-41, 2023.
[17] Robert Culkin and Sanjiv R Das. Machine learning in finance: the case of deep learning for option pricing. Journal of Investment Management, ((4))92-100, 2017.
[18] Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Class-conditional conformal prediction with many classes. In Advances in Neural Information Processing Systems, volume 36, pages 64555-64576. Curran Associates, Inc., 2023.
[19] Lilun Du and Chunming Zhang. Single-index modulated multiple testing. Annals of Statistics, 42(4):1262-11) 1311, 2014.
[20] Lilun Du, Xu Guo, Wenguang Sun, and Changliang Zou. False discovery rate control under general dependence by symmetrized data aggregation. Journal of the American Statistical Association, 1 118(54))6077-61,, 2023.
[21] Bradley Efron. Simultaneous inference: When should hypothesis testing problems combined? The Annals of Applied Statistics, pages 197-223, 2008.
[22] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. The Annals of Statistics, 309)331,,-,, 2022.

---

[23] Ulysse Gazin, Ruth Heller, Ariane Marandon, and Etienne Roquain. Selecting informative conformal prediction sets with false coverage rate control. arXiv preprint arXiv:2403.1229. 2024.
[24] Yu Gui, Ying Jin, and Zhimei Ren. Conformal alignment: Knowing when to trust foundation models with guarantees. arXiv preprint arXiv:2405. 10301, 2024.
[25] Anil K Jain, M Narasimha Murty, and Patrick Flynn. Data clustering: a review. ACM computing surveys (CSUR), 31(3):264-323, 1999.
[26] Ying Jin and Emmanuel Candès. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244):1-41, 2023.
[27] Ying Jin and Emmanuel J. Candès. Model-free selective inference under covariate shift via weighted conformal p-values. arXiv preprint arXiv:2307.09291, 2023.
[28] Ying Jin and Zhimei Ren. Confidence the focal: Conformal prediction with selection- conditional coverage. arXiv preprint arXiv:2403.03888 2024.
[29] Igor Kononenko. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, 23(1):89-109) 2001.
[30] Yann-Aël Le Borgne and Gianluca Bontempi. Reproducible Machine Learning for Credit Card Fraud Detection Practical Handbook. 05 2021.
[31] Vo Nguyen Le Duy and Ichiro Takeuchi. More powerful conditional selective inference for generalized lasso by parametric programming. Journal of Machine Learning Research, 23(300); 1-37, 2022.
[32] Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference, with application to the lasso. The Annals of Statisties, 44(3):907-927, 2016
[33] Junu Lee and Zhimei Ren. Boosting e-bh via conditional calibration. arXiv preprint arXiv:2404. /7562, 2024.
[34] Ziyi Liang, Matteo Sesia, and Wenguang Sun. Integrative conformal p-values for out-of- distribution testing with labelled outliers. Journal of the Royal Statistical Society Series B: Statistical Methodology, page qkad138, 01 2024. ISSN 1369-7412.
[35] Ariane Marandon, Lihua Lei, David Mary, and Etienne Roquain. Adaptive novelty detection with false discovery rate guarantee. The Annals of Statistics, 52(1):157 183, 2024.
[36] Möbius. Hr analytics: Employee promotion data. atttps:/wwar http::///ww..kggll...o www..aallll..aaaaaae..a com/ catoonddtttett///// arashnic/hr-ane 2021.
[37] Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. DOI:
[38] Jullien Nazreen. Diabetes health indicators dataset. https: stttps://www... http::/wwwwwwwwllccoo/
hallaaz/diabbttee-heabbbbb-- wtw.aaatttttiiiitttrrrrdaaaaeeea 2023.
[39] Eugene Ndiaye. Stable conformal prediction sets. In International Conference on Machine Learning, pages 16462-16479. PMLR, 2022.
[40] Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pages 345-356. Springer, 2002.
[41] Bradley Rava, Wenguang Sun, Gareth James, and Xin Tong. burden shared is a burden halved: A faimess-adjuster approach to classification. arXiv preprint arXiv:2/ 10. 10.05720, 2021.
[42] Stephen Reid, Jonathan Taylor, and Robert Tibshirani. Post-selection point and interval esti- mation of signal sizes in gaussian samples. Canadian Journal of Statistics, 45(2):128-144 2017.
12

---

[43] Zhimei Ren and Rina Foygel Barber. Derandomised knockoffs: leveraging e-values for false discovery rate control, Journal of the Royal Statistical Society Series B: Statistical Methodology. 86(1): 2024.
[44] Zhimei Ren, Yuting Wei, and Emmanuel Candès. Derandomizing knockoffs. Journal of the American Statistical Association, 1 118(542):948-35 958, 2023.
[45] Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. Improving the accuracy of medical diagnosis with causal machine learning. Nature Communications, 1 11(1):923,, 2020.
[46] Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. Advances in Neural Information Processing Systems, 52:3543-3553 2019.
[47] Siddhaarth Sarkar and Arun Kumar Kuchibhotla. Post-selection inference for conformal prediction: Trading off coverage for precision. arXiv preprint arXiv:2304.06158 2023.
[48] John D Storey. A direct approach to false discovery rates. Journal of the Royal Statistical Society: Series (Statistical Methodology). 64(3):479 498, 2002.
[49] John D Storey, Jonathan E Taylor, and David Siegmund. Strong control, conservative point esti- mation and simultaneous conservative consistency of false discovery rates: a unified approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66(1):187-205, 2004.
[50] Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In International Conference on Machine Learning, pages 32633-32653. PMLR, 2023.
[51] Weijie J. Su. The fdr-linking theorem. arXiv preprint arXiv:1812.08965. 2018.
[52] Shinya Suzumura, Kazuya Nakagawa, Yuta Umezu, Koji Tsuda, and Ichiro Takeuchi. Selective inference for sparse high-order interaction models. In International Conference on Machine Learning, pages 3338-3347. PMLR, 2017.
[53] Jonathan Taylor and Robert J Tibshirani. Statistical learning and selective inference. Proceedings of the National Academy of Sciences, 112(25):7629 7634, 2015.
[54] Shalini Verma and M Ejaz Hussain. Obesity and diabetes: an update. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 11(1):73-79, 2017.
[55] Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning, pages 475-490. PMLR, 2012.
[56) Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.
[57] Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-leaminn applications of algorithmic randomness. In International Conference on Machine Learning, pages 444-453. PMLR, 1999.
[58] Ruodu Wang and Aaditya Ramdas. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822-852 2022.
[59] Chen Xu and Yao Xie. Conformal prediction for time series. IEEE Transactions on Pattern Analysis & Machine Intelligence, 45(10): 1575-115777 2023.
[60] Fan Yang, Rina Foygel Barber, Prateck Jain, and John Lafferty. Selective inference for group- sparse linear models. Advances in Neural Information Processing Systems, 29:2477-2485. 2016.
[61] Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. Proceedings of the 40th International Conference on Machine Learning, pages 10578-40604 PMLR, 2023.
13

---

[62] Sonja Zehetmayer, Peter Bauer, and Martin Posch. Optimized multi-stage designs controlling the false discovery the family-wise error rate. Statistics in Medicine, 27(21):4145-416 2008.
[63] Yifan Zhang, Haiyan Jiang, Haojie Ren, Changliang Zou, and Dejing Dou. Automs: Automatic model selection for novelty detection with error rate control. Advances in Neural Information Processing Systems, 35:19917-19929, 2022.
[64] Zinan Zhao and Wenguang Sun. False discovery rate control for structured multiple testing: Asymmetric rules and conformal q-values. Journal of the American Statistical Association, pages 1-24, 2024.
[65] Paul Zimmet, Dianna Magliano, William H Herman, and Jonathan Shaw. Diabetes: 21st century challenge. The lancet Diabetes & endocrinology, 2(1):56- 64, 2014.
[66] Tijana Zrnic, Aaditya Ramdas, and Michael I Jordan. Asynchronous online testing of multiple hypotheses. Journal of Machine Learning Research, 22((): 2021.
14

---

A Details of the comparing benchmarks
A.1 The adjusted p-value (AMT)
Here, we provide an overview of the statistical properties associated with the adjusted p-values.
We draw inspiration from the work of Benjamini and Yekutieli [12], which constructs adjusted confidence intervals for selected parameters for FCR control. They multiplies the confidence level a by quantity related to the proportion of selected candidates over all candidates. In most cases, the quantity is approximately the selection proportion. And they proved such simple adjustment on the level can provide FCR control.
Analogously, we can adjust the p-value after selection by multiplying the selection proportion:
pj =min+ 0 E
Co V2 V.H
1]. .1]. jES. S..
+ Co
where 0 represents estimator that estimates the selected proportion under the null hypotheses. Since the response of the test data not directly observable, we estimate this proportion by employing the same selection procedure on the labeled data, i.e. iCal
We can verify that the adjusted C.ns.
p-value is super-uniform for jochneexchangeable selection rule.
Proposition A.1. The adjusted p-value is super-uniform i.e.
P(p S I E Uo n Sw) S
where p is defined in 8
Proof. The case is trivial so we only consider the case # +iieeee.V we have
< 1. In this case,
1+|co]
P p0; saljeuonnn) =
|Col
Hi E V, V
soljEuons.
Co S.l
1 Col
1+ Ki Co : V; s V <a Salj Caljjunng Uo ns.
$ P
1 1 n&
1 Ef nsa V V
I P
<ali E E4 ns,
1+ n Sal
(0)
S
where (i) use the property that the variables {V, : j E Ug n Su) and (V: : C. n S.J are exchangeable when the selection procedure is exchangeable of the test and labeled data.
O
The correlation of the adjusted p-values can be complex when using an arbitrary selection procedure. In order to address this issue, we propose utilizing the Benjamini- Yekutieli (BY) [1T] procedure to effectively control the FDR. It replaces the original level a with a L where
-eajamini-Yekutieliii
IS..1 Σ = log Sal + O(1).
L T
The BY method can handle the dependence between p-values, but deliver a more conservative result.
A.2 Selection conditional conformal prediction (SCOP)
The selection conditional conformal prediction (SCOP) proposed by Bao et al. 3 is method for constructing valid prediction intervals after selection. The prediction interval reported only when is selected. The SCOP aims control the false coverage-statement rate (FCR) (2). which is the
15

---

expected ratio of the number of selected prediction intervals failing to cover their respective true outcomes to the total number of selected prediction interval, i.e.
FCR == E Ejeu Mij E Eu,, Y, & PI(X,)) 1v
where PI(X)) is the prediction interval. To ensure FCR control, the SCOP involves similar procedure to pick calibration set from the labeled data using the same selection rule. And then the prediction interval for selected individual is constructed via the residuals in picked calibration set.
As a natural idea, we can simply invert the prediction interval into hypothesis testing. If the hypothesis is Ho.j Y, I Co v.s. Ho.; YY > Co, reject the single hypothesis with type I error is equivalent to that the one sided prediction interval covers cp. We formulate this idea as an intuitive benchmark as Algorithm 2
Algorithm 2 SCOP for selective multiple testing
Input: Labeled set ee, test set Du, selection procedure Sp.,p.,, prediction model (i-), FCR level a E (0, 1).
Step 1 Apply the selective procedure S to obtain the selected subsets S,, and S..
Step 2 Compute residuals {R, = Y; - ((X,): i S.
Step Construct selective conformal prediction intervals for each j S, by
PI(X,) = (- (ox,, Qu(((R))ss...
where Qa (fRi),e denotes the [[1 - (1S. + 1)]-th smallest value in {Ri hes.
Step 4 Reject sample if cu E PI(())
Output: Rejection set RSCOP E S., :0 E PI(X))).
Under the null, we have P(co & PI(X,)) S P(Y, & PI(X))) s a/P(Y, I co). This implies marginal coverage, as for single hypothesis Hoj. we can control the type error. It is important to note that the scaling of the inequality P(co & PI(X,)) S F(Y, & PI(X;)) is often too conservative.
However, when it comes to simultaneous testing, the SCOP procedure fails to control the FDR. This is because the FDR built up on the rejection set R C S., where the rejection decisions are intricately linked to the entire selection set and the inherent randomness within the selection set further complicates the distribution of the final rejection set. Hence our work address this challenge by carefully analyzing the randomness from selection and rejection set.
B Conditional calibration
Here we introduce the frame work of conditional calibration
The first idea of conditional calibration to control a conditional expectation, given some conditioning statistic D that blocks most all of the nuisance parameters from influencing the conditional analysis. And we require that p.
adapt is conditionally superuniform given Øy:
Pr(p)
adapt
<tlj S..j j Uo 2 <t
Secondly, the number of rejections should should be bound from below by known function of 4y. If all constraints are satisfied, that set of rejections is guaranteed to control the FDR bellow
Step 1: Calibration For each of the test points, i D {n 1,..., + m), let by (Dougy For each test point E S.,, compute the adaptive p-value
(i) Eies.nc, (V < Vo) + (V, < VV) Pt
ve
L S..
(6)
1 + |S. Col
Next, let Rg indicate the number of rejections obtained by applying BH at level a, for some fixed adapt.(f)
(0, 1), to the approximate p-values {pr
; + j, E S.) u (0).
16

---

Step 2: Preliminary rejection. Define the preliminary rejection set R+ as:
R+ {
aR;
E Su adapt
SSa
R4 RR+| If R 2 R4 for all i E R,+, then return the final rejection set R. = RR+- Otherwise, proceed to the next step.
Step 3: Pruning, (a) Deterministic pruning: Define R as:
Ratm max {r= i E R+ R S (5rr2)) 2r]
The pruned rejection set R. is that containing the indices with i E R.+ and Rg < R
(b) Randomized pruning: Generate independent standard uniform random variables e; for each
i E R4,, and define Rt as:
max {r: iE R4 :4 Sr/Ri 2rj.
Rydm
The pruned rejection set R is the set containing the indices E R... such that E; < RR/R.
The conventional conditional calibration offers flexible framework to decouple the dependence between p-values. But in selective setting, the number of test units is |Sul, which can be compli- catedly dependent with both pj and Rj. And when analyzing the FDR, the event that j-th sample is
{ps 18.1 18.1 JES.
selected is also involved. So our primary focus is ensuring E
Φ a.
|R,1
The conditional calibration framework primarily focuses on the correlation of p-values. However, a significant challenge arises because FDR control in a selective setting involves not only individual p-values but also the selection procedure itself. Consequently, the selective effects are unavoidable when implementing conditional calibration. To address this, leverage the stability property the selection rule, which allows us to effectively conduct analysis over the selected subset effectively and rigorously.
We can prove that the conditional calibration applied to the adaptive selective conformal p-values can control the FDR The technical proofs are deferred in Appendix F8
Theorem B.1. Assume the data are i.i.d. and the selection rule is weakly stable. Then, the FDR output by the above three-step procedure applied adapt is smaller than ofE [S
S.1
It easy observe that Ratm S Rram- indicating that randomized pruning results in larger rejection sets. Therefore, we employ randomized pruning in practice to enhance power. By our empirical investigations. we find the BH procedure applied to p; conditional calibration approach with random pruning also has close performance in power. Figure displays the FDR (left) and power (right) through varying noise strength employing conditional calibration and BH procedure.
adapp can idcally control the FDR, and the
B.1 Eliminating randomness by boosting e-BH
The deterministic pruning process would lose certain power. Although randomized pruning can improve this situation, the externl randomness can potentially hinder the reproducibility of the results (the procedure can be quite sensitive the realization the e's, leading different selections across various algorithm runs). A recent method 33 can enhance the power of the pruning process without introducing additional randomness.
Specifically, let (ee) represent the rejection set yielded by the e-BH procedure on e at level a E (0, 1). For each j E [m], define R,(e) = R(e) U (j) and subsequently define the function
1
ce
m
(
bi (c) = E
m
Eg
S
(e)
17

---

Method
Con
BH
Case
Case
Case
Case
10.50
10.25
95
FDR(%)
10.00
Power(
90~
Upa meau
UEBRU
9.75
85
80
9.25
0.5
Noise Strength
Noise Strength
Figure 2: Empirical FDR (left) and Power (right) of conditional calibration (Con) and BH procedure under different cases for the mean selection rule. The black dashed line denotes the target FDR level 10%.
where (ēi,..., Cm) follows the conditional distribution e Sj. With the associated critical value bj = sup (c: 03 (c; S2) < 0}, the boosted e-values are constructed as:
m
Ceee 2 a (c) 1
m
if (2; 0
T72
Cee
>
m a [e,(
if of (C 8,)
[e)(]]
They prove that the boosted e-values are generalized e-values and R(e) C R(ee).
Our procedure be viewed generalization for selective scenario of their approach. The conditional calibration approach with deterministic pruning is equivalent to the e-BH procedure
applied {e, : 8.), where ej |$111 .A,(o)
18.1
by referencing Jin and Candès (27 and
aRy (p)
p {p))yes_ Another form of e-value existing in conformal inference is based specific stopping time [43 77. But we find it is not directly applicable in our setting our calibration different for each test data. Under our stability assumption, we can confirm that ej is valid e-value in manner similar to Lemma E.2 in our paper. With this equivalence property, the boosting method can be directly applied to our deterministic pruning approach by constructing the new boosted e-value with m = |S.] and S, = (Deury Du
C Details for the numerical experiments
C.1 Implementation of Storey's method
The Storey's 's method 49 aims to estimate the null proportion π to increase the detection power. In our setting, the null proportion can be directly estimated by the corresponding proportion in the labeled set, i.e. OOMT 4 AMT [Cll/[C] and SCPV |S. Col/ ( And when applying BH procedure, we will use a level of a instead a, such that the FDR can be controlled at a exactly.
C.2 Details of the real data experiments
Abalone [37] contains casily obtainable measurements of abalone. The task is to predict the age of abalone from physical measurements. We use the shell weight as the selection score. The co we used for this task is taken by 12.
Census [ contains census data extracted from 1994 Census Bureau database. We focus people from America and regard the income attribute as the response of interest, which is
18

---

a binary variable indicating whether one's income exceeds S50K per year. The feature of age is used as selection score T.
Credit [30] contains transactions made by credit cards over the course of two days, some being frauds. The task is to identify the frauds and we use the specific feature, amount, as the selection score. Since it contains only 492 samples of class 1, set the null proportion at 0.9 instead.
Promotion [36] contains employee's 's past and current performance and the final promotions. The task is to predict whether potential promotee at checkpoint in the test set will be promoted or not after the evaluation process. We use the specific feature, average score in current training evaluations, as the selection score.
D Additional comparing methods
We discuss two additional comparing methods which are nicely suggested by the reviewers.
seff-consitenn/commlaal adjustment (SCA) Using the marginal p-values in [ one can directly achieve FDR control under any data-dependent selection simply by taking the largest self-consistent rejection set, i.c the largest subset s.t. S a'R/ K for each E R C Su. where a' is the largest value that satisfies Tna (1 + log (1/ (Toa') ))) S where To |Col /|C] is the null proportion. This a direct consequence of Theorem 3 of [51] and the PRDS property of conformal p-values from [.
We analyze the comparison between our approach and the baseline method from two perspectives. From the theoretical point of view, we have observed that the power loss associated with utilizing selective conformal p-value is usually less than that incurred by the FDR-Linking method. To illustrate this, assume mo = 0.7 and a 0.1 as the simulation setting of quantile selection, then we derive a' x 0.025. The AMT method adjusts the marginal p- p-value after selection by multiplying the selection proportion i 1 1/0.7. This is equivalent employing the BH procedure the marginal p-value with & 0.07, which evidently yields greater power than SCA. Additionally, AMT does not make full use of the information from the selection procedure. In contrast, our proposed method uses a smaller p-value than AMT, which suggests more power increase.
In terms of empirical performance, as demonstrated in both cases from our paper, the SCA method suffers from a power loss, confirming our theoretical analysis.
Table Comparisons of empirical FDR (%) and Power (%) with target FDR level a 10% by 500 repetitions.
QUAN
MEAN
FDR
POWER
FDR
POWER
CASE A
SCA
3.59
88.7
2.85
84.7
SCPV
9.83
93.9
9.90
93.9
AMT
6.23
92.1
8.28
92.6
CASE B
SCA
3.79
75.7
3.38
66.4
SCPV
9.82
84.9
9.79
81.1
AMT 8.71
77.0
5.81
79.5
InfoSCOP Gazin et al. 23 propose a novel method named InfoSCOP, which is closely related to our approach under the joint-xxcaangealle selection rule. Below, we provide detailed discussion of their method. The primary objective of InfoSCOP is to select informative prediction sets with false coverage rate (FCR) control, although it is not specifically designed for multiple testing.
Śinfo C U where each Their focus on an informative selective prediction set procedure, denoted as S prediction set C2 is T-informative for every j E Śinfo cd. prediction set C. is T-informative, then all the prediction sets it contains also Z-informative, and it is right-continuous for the
19

---

coverage level. By leveraging the property of Z-informative selection, they link the FCR control problem with BH procedure, and verify that the FCR control can also imply the following FDR control:
Ejesun 1[Y, & UceeC
E
1 Sinto
InfoSCOP's procedure to achieve FCR control consists of main parts. The first part transforms the informative selection procedure into specific BH procedure. Let {p, heu denote the set of p-values over test set. These p-values are inverted into I-adjusted p-values by
4 minfa E (0, C]][))]
The BH procedure then applied the q {g: Jisu obtain selection set BH(q). The second part involves using the adjusted approach from Benjamini and Yekutieli [12] to construct prediction sets for each selected individual, at the level of ())H(q))/m,, thereby providing FCR guarantee. To mitigate power loss from this adjustment process, they employ the method from Bao et al. [3] to select an initial subset So l which reduces the number of units and allows for larger adjusted level of a (BHH//SS to construct prediction sets. To maintain their theoretical guarantee, they require the initial selection to be exchangaaa,,,,,,, which connects to our setting.
The FCR guarantee of InfoSCOP can directly imply FDR control on a data-dependent selection set by ensuring that the "informative prediction set" is informative with respect the null hypothesis being tested. In this way, InfoSCOP implements an FDR control procedure after selection by applying the BH procedure to selective conformal p-values, which closely aligns with the core approach of work.
For strongly stable selections, method can be simplified and degenerate into form similar to InfoSCOP. But the assumption in InfoSCOP is not satisfied by the quantile selection rule based solely test data. Thus, their theoretical results are not applicable in such cases, while our framework bridges this theoretical gap.
And our approach covers wider range of selection rules. For instance, when dealing with weakly stable rules, we employ conditional calibration on adaptive p-values ensure rigorous FDR guar- antees. The table below compares the performance of our approach with InfoSCOP under mean selection rule. The InfoSCOP shows reasonable empirical performance, which is similar to ours. Therefore, it is possible that InfoSCOP may still work under mean selection, making it an interesting topic for thcoretical investigation, which remains unexplored InfoSCOP. In contrast, we provide FDR control guarantee under variety of selection scenarios.
Table 4: Comparisons of empirical FDR (%) and Power (%) with target FDR level 10% by 500 repetitions.
CASE A
CASE
FDR POWER
FDR
POWER
INFOSCOP
9.8.
94.0
9.80
78.4
SCPV
9.86
93.4
9.80
78.1
In conclusion, our approach and InfoSCOP are designed for different goals, resulting in different analytical frameworks. Ours is specifically designed address the multiple testing problem across various selection rules. From the perspective of conditional calibration. our method is unified, where BH procedure for strongly stable selection can be seen as a special case. As a comparison, InfoSCOP is an excellent work for selecting an informative set with FCR control, but it is not primarily designed for multiple testing after data-dependent selection. Their FDR guarantee an extension of FCR control, which limits their method's 's applicability to different selection rules.
20

---

Table 5: Comparisons of empirical FDR (%) and Power (%) under different scenarios and thresholds with target FDR 10% and noise strength a 0.5. The sample sizes of the labeled set and the test set are fixed as m 1200.
CONSTANT FDR POWER
EXCH
TEST
FDR
POWER
FDR
POWER
RANDOM
FOREST
SCPV
9.81
95.27
9.78
95.23
9.80
95.23
OMT
19.07
98.87
19.07
98.87
19.07
98.87
AMT(BH)
6.14
92.41
6.11
92.35
6.14
92.36
AMT(BY)
0.73
79.20
0.72
79.15
0.71
79.13
SVM
SCPV
9.82
85.84
9.83
85.88
9.85
85.89
OMT
15.05
95.21
15.05
96.21
15.04
96.18
AMT(BH)
7.79
79.66
7.81
79.69
7.80
79.65
AMT(BY)
1.28
38.82
1.29
38.90
1.28
38.87
NEURALNET
SCPV
9.73
88.62
9.72
14
9.74
88.35
OMT
7.89
64.44
7.90
64.39
7.90
64.45
AMT(BH)
6.15
19.44
6.19
19.46
6.20
19.49
AMT(BY)
0.04
0.14
0.05
0.17
0.04
0.14
E
Additional empirical results
E.1
The effect of the learning models
In T present the results of FDR and power under three machine learning methods. The data is generated based on the settings specified in case A. The neural network (Neuralnet) with a single hidden layer and 5 hidden neurons is implemented by using the R package neuralnet. And the linear output units are used. Here, we fix the noise strength a = 0.5. can be seen that our method controls FDR the expected level and it also provides satisfactory testing power. In contrast, the FDR values obtained from AMT tend to be overly conservative, leading to a notable deflation of its power. In the first settings, the OMT methods inadequate in effectively controlling FDR. While they can successfully control FDR in the last setting, they often suffer from a loss of statistical power.
E.2 A real data application with clustering
Diabetes is a chronic disease that affects a large and growing number of people worldwide (65] As such, identifying potential diabetes patients using risk factors and machine learning tools is attractive approach for early intervention and preventive measures. To this end, we applied our method the Diabetes Health Indicators Dataset [38] provided by the Behavioral Risk Factor Surveillance System (BRFSS) in the United States. Through this analysis, we are able to effectively identify high-risk individuals while also providing uncertainty quantification measures.
In the dataset, the response variable is denoted as Y, which takes the value of 0 or 1, indicating whether the j-th person suffers from diabetes. The dataset also includes patient-related information consisting 21 features, such as BMI (Body Mass Index), cholesterol level, and other health risk indicators. These covariates provide additional information about each individual that be used to analyze and predict the likelihood of diabetes. Our goal is identifying those diabetes patients with controlled FDR 20%, i.e.
Ho. YY = v.s. H1 : =Y= 1.
The data is processed as follows: a total of n 2, 000 labeled data points and m = 2, 000 test data points are randomly sampled from the dataset. The prediction model fi is constructed by random forest using another 2, 000 i.i.d. training data. Based on prior knowledge, we understand that individuals with obesity are at higher risk of suffering from type II diabetes [54]. Therefore, our focus is directed towards making inferences specifically on individuals with high BMI Denote the
selected subset S., {i U r}. Several selection rules considered. Constant: T; is the
21

---

BMI of i-th individual. T 30. Exch: T; is the BMI of i-th individual and the T is the 70%-quantile of (T, j E C CUZ). Quan: T, the BMI of i-th individual. is the 70%-quantie of {T, j U..
Tabpess the results of our proposed SCPV and other compared benchmarks using the thresholds mentioned ahove. Our method and adjusted methods successfully achieve valid FDR control using all of these selection rules. However, adjusted methods select fewer individuals, leading to powerless results, Meanwhile, the OMT fails to control FDR for most settings.
Table 6: Comparisons of empirical FDR (%) and Power (%) with target FDR level 20% by 500 repetitions.
CONSTANT
EXCH
QUAN
FDR
POWER
FDR
POWER
FDR
POWER
SCPV
19.99
63.99
20.01
72.43
20.03
72.26
OMT
23.88
82.99
23.14
87.55
23.18
87.44
AMT(BH)
13.21
23.52
8.58
12.54
8.66
12.93
AMT(BY)
0.03
0.05
0.02
0.05
0.02
0.04
Besides, the dataset potentially composed of different groups, and important to identify individuals while controlling the FDR for each group. This allows us to make more accurate assessments and informed decisions specific each group. To address this, we employ a clustering algorithm, specifically K-means, to divide the dataset (which uses both labeled and test data but lacks response information) into two distinct groups. Our primary objective is to draw inferences within each individual group. The clustering process results in two groups that exhibit significant disparities terms the "MentalHealth" covariate. Consequently, we refer to the group with a lower "MentalHealth" index Group A, while the other group is denoted Group B.
Regarding the results for the two clustered groups Table 7 we observe that our method exhibits stringent FDR control. The OMT lacks power for Group A and yields an inflated FDR level for Group B. As for these adjusted multiple testing methods, they deliver more conservative rejection results. To conclude, our method is powerful to provide subgroup FDR control for adaptively chosen groups.
Table 7: Comparisons of empirical FDR (%) and Power (%) with target FDR level 20% by 500 repetitions.
GROUP
GROUP
FDR
POWER
FDR
POWER
SCPV
16.35
12.75
19.79
77.05
OMT
4.58
1.43
24.04
95.60
AMT(BHI
1.84
0.38
11.54
23.96
AMT(BY)
0.00
0.00
0.17
0.16
E.3 A real data application with deep learning method
Breast cancer is the most common form of cancer in women, with infiltrating ductal carcinoma (IDC) being the most common form of breast cancer. Accurately identifying and classifying subtypes of breast cancer is important clinical task, and utilizing deep learning methods for identification can effectively save time and reduce errors. Our dataset consists of complete whole slide images of breast cancer (BCa) specimens scanned at 40 times magnification. Our method can effectively identify individuals who may be at risk of breast cancer, while also measuring the uncertainty of the deep learning model.
In this dataset, the label is denoted as Y, with Y taking values and 1, representing whether the j-th image is slice from a breast cancer specimen. Our goal is identifying those breast cancer patients
22

---

with controlled FDR 10%, i...
Ho,j Y, -0 v.s. H1. Y2 = 1.
The data is processed as follows: total of n - 800 labeled data points and m 800 test data points randomly sampled from the dataset. The prediction model is constructed by a convolutional Neural Network with 10 layers using another 2000 i.i.d. training data. This network consists of total of 10 layers, including convolutional layers and 3 max pooling layers.
T. represents a score used assess the risk of breast cancer, such as the probability of developing breast cancer predicted by model. Our goal is to identify individuals in the high-risk group who are more likely to develop the disease. Denote the selected subset S., = Ej T₃ J). Several selection rules are considered. Constant: 7; represents the predicted probability of the i-th individual obtained using model f.. r = 0.2. Exch: T; represents the predicted probability of the i-th individual obtained using model fi and the r is the 30%-quantile of (T, j U Quan: T; represents the predicted probability of the i-th individual obtained using model 7 is the 30%-quantile of {TT j E U...
Table 8 displays the outcomes obtained by our proposed SCPV method and other benchmark ap- proaches when employing the specified thresholds. Our method and adjusted methods successfully achieve valid FDR control under these criteria. However, adjusted methods select fewer individuals, resulting in powerless results. Meanwhile, the OMT proves inadequate controlling FDR for most settings.
Table 8: Comparisons of empirical FDR (%) and Power (%) with target FDR level a = 10% by 100 repetitions.
CONSTANT
EXCH
QUAN
FDR
POWER
FDR
POWER
FDR
POWER
SCPV
9.88
72.9
9.71
74.0
9.75
74.1
OMT
14.1
80.3
16.9
88.3
17.0
88.7
AMT(BH)
9.24
71.4
9.24
73.3
9.24
73.6
AMT(BY)
0.479
9.43
0.479
9.71
0.479
9.80
Technical proofs
F.1
Revisit of notations
The index set of the labeled set and test set are and 4. For any subset S C C U U, we use Ds to denote the data {i (X,, Y.)). The selected test set and calibration set are S, and S.. The null labeled set and test are denoted as Co (i E C Y: & A) and Uo [j u Y, & A). Equivalently, (Y,)
F.2 Auxiliary Lemmas
We introduce some auxiliary lemmas. The first one is the quantile inflation lemmas which is common conformal inference literature [46] 3 and we omit proof.
Lemma F.1. Let [[[n(]) is the [nt] -maaleest value in fx, R: i [n]]. Then for any E (0, it holds that
n
1(x, x(((e)))) SL
n
If all values in (x: i [n]} are distinct, it also holds that
1(x: S ((((())) 2
n
23

---

The next lemma is the key of our theoretical results, which characterizes the properties of the constructed p-values. The proof is deferred to Appendix F.4
Lemma F.2. If the data are i.i.d. and the selection rule SD..D. is strongly stable, have that (a) R(-+ defined in 9 is measurable with respect to D, defined in 7
(b) For any j E Utii holds that for any random variable t that measurable with respect the unordered set 2y. we have
P ( Stije Y, 2, S
F.3 Proof of Theorem 3.2
Proof. We begin by establishing the validity of the selective conformal p-values. Consider the quantity
$ (DCUg),
(7)
which consists of two components: Duyti the test data with the j-th sample excluded, and Deu()) iZ E C U [i}], whose elements taking values on {Zihie U {Z₃} but without
their indexes. According Lemma F22 b). have
P p
s Sps S., Y, ( =
E
[Dp stjje Sw, & A. A, st
Next, we proceed to verify the control of FDR. This involves examining the previously defined defined p-values
1+ Eie 1[,
< V2}
for S..
Pj
+ SS, Col
For any j Su, define slightly modified p-values
Eistine 1 (V, < ViJ + < Vi)
( Pi
ve l S..
(8)
+ S8. ncol
These p-values are only used in our analysis. Also define R (a) : E S.i)) C Sn as the rejection
(indices) obtained by the BH procedure, from p-values taking the values in {az : S.). In the sequel, we will compare R
R (1r))) + u
on the event (Y, A, E RJ. For the remaining p-values, since the scores have no ties, we consider two cases:
(i) If S Ve, then
p 1+ Eiesnne (Vi < Vi} SS. Col +1
Pe-
(ii If V, VG, then p. Pj- Since j R, the BH procedure implies é E R. By definition, have
1 Eies.ne, 1 < V2 1 S.
S 1+ LiE 1 ([V < Vi)
pt
= Pj.
1 ++SS.nC nol
To summarize, suppose we are to replace Ps by pL for all l + j, S., Then on the event (Y, & A.j R}. such replacement does not change any of those Pe 2 Pş; also, all those pe S Pj including Pý itself (they are rejected in R are still no greater than after the replacement. Thus, by the step-up nature of the BH procedure, such a replacement does not change the rejection set, meaning that
R = R (1p, :pp;:jm (.E.))
24

---

R (1 + jiLe S.3 u R())
on the event (V, & A.j E RJ. Let Rj [ R), then leave-one-out analysis of the FDR implies
EjEs, 10' & A) R2
FDR
E
I jES. Rg
1( = k3 $ A) A)1 s ak/lšul 1 e S.)
E
jEU
Am
18
Σ
E
| = x} 11,, $ 4) j J s.)
JEU
The (i) use the property of the BH procedure, and (ii) comes from the facts stated just above. By the step-up nature of the BH procedure, we know that on the event {j E R01}. sending PA to zero does not change the rejection set, i.e., we have
R() R (v) -.4 = =R(000)
Thus
S.1
FDR
Σ
E
{ R(0=00 = kb 1 ffp [//R0-01/15.1 1 (Y, A} 10 S.)
jeul
(10)
Pj S a R(ú /|5.1 1 (Y, & A) e S.)
E
(11)
V RD-00)
jEU
By Lemma
F22 aa, it holds that
1
ps S a R 101
/
1 ( & A} 1íj E S.)
E
1 RU-00)
1
1 {1(p p3 s a R(-- /15.1| 1 (Y, & A} E S.t | 09
E
1 R
-0)
1
(p) S a | /10,11 /|&,1 eie Sm, Y, E A, ((Vi)couu)))) (j Y, & ( | $3)
Po
1 R
and have
RR(--0)
/
S/ measurable with respect to the unordered set F Then use Lemma R.2 we
P p, R(6 / j S., Y2 & A, a R(i-10) /|Sul.
Through summing over j E U1 together with tower's rule, this gives
1[ E S,, Y, $ A)
|& nuo |Sul
FDR <
Σ oB
aB
Sa
S
jeu
which concludes the proof.
F.4 Proof of Lemma F2
Proof. We note that the following proof is conditioned on the event (Y, & A}. Define
Scn = (i Cu (}} D.. = 1, & A}.
25

---

Then S. Co ( = S soorts and |S, n + = ssc.ts hold under the event j E Sy and Y2 & A.
For any j satisfying Y, & A, define the event
Acugg) (z) [[Z = [#1, : Sm Zn+1]}-
(12)
Denote the corresponding unordered conformal scores by [ui,... un+il under Act Since Sp..D. is strongly stable, then given Dinj and under Acuro) (z) we have
SD..D. (..)) = Sp,12, M2,1 (4:) = = -S 1. for 1, n 1. It means that the following unordered set
[1V) (VieeS (2) [fe, St.,, 1, & Alt=l,..mmt1) is known and only depend on 2 and the data Dan. Besides,
cc AcuG) {i {1,-.. in n S2.,)(22)(E))) = 1, & A}
{ (1) is invariant after permuting (Vi) (V})).
is also known. Note that by definition 8
( [4
We know that the modified p-value
Eves
1 1( V2}
Pi
(
(z)
Ve l S..
Scn
(2) and this implies is fixed condition on (z) Also note that RO-0) only depends on {0, Jess"
that R()-0) is known under (z). Through marginalizing over Acu(s) (z), we obtain that Pe is measurable with respect . Since RÚ0+0) is only depend on p(6) for t # j, EE Su the first part of Lemma F22 can be readily demonstrated.
For the second part, we know it holds that
{pz S [ (V) S
V&.nc.
(Ta(SEnco)+11)) (V, s (S.nc,uf)) ('x(S_nGol+111)
{V, < v (+)) (((15))
by the construction of Py and Lemma F.1
And then we have
P Pj st,,, $ Alj E S., 9;))
=P
VV3 < (Iu(Sencal+1)p) Y1 & A (X,) -
(1)
Σ 1/1 S VSIncaut))
((ulsencal+u))" Y, $ A}
<1+
SSe nCol+ +1 kES.nCo
-1 { V/x I veencoul))
(cl(s))))++))))) Y, E A SD..D. = 1, 1,0,
=1+ Σ
1 1/ VSSccou)
((x(SSncGl+11))) Y, E AA, = 1, Vv & A}
Col +1
REC
1 (Va
([o(SEncoo+11)) Y, AA (Xx)
= 1, & 11 SS =1, (X,
S. Col Col +1
=t Σ
1
E
1
{
{ V, -
Scp.11
([((IS5.p Sp.DD, (X,) = 1, Y, & A.Sp..,, = 1, 1, & A
P(SD., D., (X,) 1)
|S
KEC
{ (ve
(TrilS..... SD..D. (Xy) 1, & D.S.p.D., = 1, $ A1 |01
S Son+tl
(13)
26

---

For ease of presentation, we write
Senrtll Acu()(2) = Scott (2; Duny),
voorts
(c) = V(2; Duns).
(Te()Sap++sDD))
For any unordered set 2, we define the following unordered set
((2) = ((i1, i2) - [n+ 1] va I V P vi2, (Zi,) = 1, yr, & A, S,,,,, (Ti,)
(Ii,)= 1, yiz $ &A)
which is ((Pun))-measurall (Zi)iEcum we can guarantee
and independent of Acuts)(2). Using the exchangeability of
(20 Dura), E
{v,
V (TUISS Sp. = =,,1,,, 1, A. (X)) 1, a} Accuo)(e)
Seo c0- Poo--ll
E
[V, S ((2; Durs), S2.p,)(2)) 1,,, A, (X.) 1, & A} Acu
Σ
P (Zz = 2i, Z -i, AcuG))
(R)())))(())
Σ
( = =1z x= =i (Acu)))))))
(2)())
-E [1 (Vi < V(2; (..
(X&) 1, E A. 1,,, & AA) Acu()
Through marginalizing over Acu(s) (2), it follows that
1
{v,
{
Segrts (Te(IS))) S.p., (3..121)))))) L,,, $ (15 1, & A] 1
E
S Son
+
Syartts (TeI)S....lll)) S,,., = L,Y, & p. 1, L, & a) 1
=I
Plug into 13),
we can verify the second part of Lemma immediately.
E5 Proof of Proposition 3.4
Proof. The results are direct if we let D: = D. D. and D: = D,, \ {2,} as a specific partition D.. By the definition of jochneexcha selection, have $D..p((A)) = Sp..D, (X.)) Sp.u12))...M2 for any i CUU and j eu. Thus the proof is completed.
E.6
Proof of Proposition 3.5
Proof. Let (T() : r E [m]} be order statistics (Tn-1.- ,T,-1, 11-1,TT Tjt,, Tn and
(Ti E [m]] be order statistics of 'Ta+t,"'.. ,Tj-1, -00, Tj+1, , .n-m). By definition,
Tropk (Tnt1.... ,T,-1, Ty+1, In+m) = T(k+1)
and
Tropk (Tn+l,... ... - -x,TT+, Tn+m) = = T:0
(K+1)*
Note that for j E Sy, we have T(K+1) > Ty. Because T(r) = T2-+000 for all order statistics with T(p 2 Tj. we have T(K+1) = -00-00 as well.
Therefore, we replace Zz z for any j E S.,, where is a fixed value such that the corresponding selection score T, (determined by the covariates X,) is Here address that the top-K selection does not the specific scale of the selection scores. Hence the we can scale the selection scores into the range (0, 1), and denote z as the value such that 7)
As the threshold Teapk keeps unchanged after replacing Z, as z, we have Sp..P. (X,) = (.,.
for any j E S, and i cuu.
27

---

F.7 Proof of Proposition 3.7
Proof. For simplicity, we suppose that the selection rule produces selection threshold r T((Th: kke) which is dependent on test data only. The selected test set is denoted as S., = {j E T r((Tr)keu)) And the calibration set is picked up the by
Sc(i) = (i E c 7; S ((TTx) for j E S..
We note that the proof is similar for all type of weakly stable selection rule. Given DuM we know is fixed. Therefore, for j E Su, i.. T₃ S <((T))ee))(1))) it holds that {Z &.()) and Z, exchangeable. Consequently, j E Uo and j E S.. follows that {2 : E S.(i) nC)) and 2, are exchangeable, which implies
that
{Z, P.:77 < ((TT))aunn))))
{V i 8c(5) )C) and Vj also exchangeable.
By the definition of PJ edopt and the similar procedure for proving Lemma R2 in Appendix F.4 it is edapt
direct that Pr(y,
< < J E Ub) S L.
F88 Proof of Theorem
B.1
Proof. Our proof follows the same strategy as calibration can be formulated by
22 34
Recall the final rejection set by conditional
R Rg
R = R= R+ 16 S
}
where R [R], R2 is the rejection number by BH procedure applied {pr S. U {0) and
adapt.(0)
= j.e
R+ =iE S. Pj
adapt <
a
aR
S
Define c-j as all e, variables for R+ {i} and R R(e, + 0) denote the hypothetical total number of rejections obtained by fixing Ej prior applying conditional calibration procedure. Then, the FDR can be written as
E R+ & Rt
R )111 $ A}
FDR E
1VR
jeM
1(jER.]1(g < Rd J1(Y A
R
E
R*
jeu
1[jER+]1(g E 12' ]1 A}
Σ E
E
R
le-,,D.UD,
1V R
jeu
(8)
1{i E R+11Y,, & A}
1 R
jeu
11j E S., (p, (pgdapt
adapt
<
af,
} & A}
Σ E
[.
IVR,
jeu
E 11j E S. (p adapt
alt p11 $ A} +
S.
E
1 R
jEU
(v1)
aR, 1(j S.11(Y & A}
Σ E
(S,l
1 R2
JE4
28

---

1íj SwjlfY, A)
|
jEUr
SSw n 2u4 |
aB
Equality (i) holds since R R' for j E S,,, as the pruning procedure can be seen as special BH procedure, which is not influence by replacing a rejected p-value with O. Equality (ii) is true because ej is independent of e-j given all the data De U Du, and R* is measurable to E-j, D. U Du, where ej has no influence on R by the assignment (ej + 0). Equality (ii) holds as R, is measurable with respect to & by the design of P1 adapt.()) And inequality (iv) comes from the Proposition 3.7 since the weak stability implies (S ki : T, < ((TT))eEU (1)))) is measurable with respect to i and j Sw. Thus the proof is completed.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sect. we summarize our main contributions. The experimental
results in Sect. and Appendix validate the theoretical results in Sect.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We have created separate "L.imitations" section in Section 5 to address the limitations of our paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer:
[Yes]
Justification: The assumptions and theory are all provided in the paper. And the proof is presented in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The implementation details are included in the paper for reproducing the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
31

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: We provide the data and code in the supplemental materials, including instruc- tions in the zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https/cccppbllcgggudeeeeeeeeeeee Coaee//dddeSebmmmcma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Sect. and Appendix C we provide the experimental setting/details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We calculate the standard error across 100 replications and show the error bars confidence band in experimental results, which is calculated by mean +/- se.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the experiments were conducted on AMD Ryzen 5800H with Radeon Graphics processor with 16 Gb memory Lenovo personal computer and the platform with version 4.2. 1. The time of execution for each of the individual experimental runs for the whole synthetic example about 21. 21.25 seconds.
Guidelines:
The answer NA that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tp:::////eis/.eeeeeeeee httpp///ppppp.c.etcccccteeeeeeee
Answer: [Yes]
Justification: All the authors have reviewed the NeurIPS Code of Ethics guidelines and 636 ensured that paper conforms to them.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The datasets used in our paper are all correctly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "R9FRX4jadD", "reviewer": "Reviewer_cTUv", "summary": "The authors study the validity of Benjamini-Hochberg like procedure on conformal p-values computed on data selected with a particular rule. Assumptions on the rules (and examples verifying them) are mentioned and guarantees demonstrated in those cases. Experiments on synthetic and classical real data and also conducted.", "questions": "Could you compare or clarify the contribution with regards to the conditional calibration scheme mentioned in the article ?", "limitations": "The approach taken in the weak selection rule has a lower power.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The presentation is clear and progressive. The problematics are easily understood, and the difficulties of the explicitly mentioned (e.g. computing p-values on data selected by a data-dependent procedure).\nThe contribution is interesting for the conformal/statistics community, as there is numerous applications of the result, or at least the arguments, to complex conformal tasks.", "weaknesses": "I'm surprised by the lack of references to Vovk's work in particular, having studied conformal p-values and conformal testing for a long time.\nTheoretically, the weakness is due mainly to the weak rule setting, although it is not a significant issue.\nHowever, I think the experimental part is the most limiting here, being limited to a very classical testing setting in a conference where applications related to deep learning would be of interest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: the lack of references to Vovk's work in particular, having studied conformal p-values and conformal testing for a long time.\n\n**To W1**: Thank you for the nice suggestions. We will incorporate more Vovk's work to enhance the clarity of the review. There are several literature we plan to review in the future version:\n\n[1]  Vovk, V., Gammerman, A. , and Saunders, C. . Machine-learning applications of algorithmic randomness. ICML, 1999.  \n\n[2] Papadopoulos, H., Proedrou, K., Vovk, V., & Gammerman, A. . Inductive confidence machines for regression. ECML,2002\n\n[3] Vovk, V., Nouretdinov, I., & Gammerman, A. . Testing exchangeability on-line. ICML, 2003.\n\n[4] Vovk V, Lindsay D, Nouretdinov I, et al. Mondrian confidence machine. Technical Report, 2003.\n\n[5] Vovk V. Conditional validity of inductive conformal predictors. Machine Learning, 2013.\n\n> Weakness 2:  experimental part is limited to a very classical testing setting in a conference where applications related to deep learning would be of interest\n\n**To W2**: We greatly appreciate your feedback on the current experimental limitations, particularly the absence of a modern setting. Your insights have inspired us to explore the application related to deep learning. \n\nWe apply our method to the field of drug discovery [1] as an initial trial. Based on the DAVIS dataset [2], we aim to identify drug-target pairs with high log binding affinity (Y). Our hypothesis is $H_{0,t}: Y_t<9.21$ with target FDR $\\alpha$=10%.  To transform the structural information of protein and chemical compounds into numerical features, we attach a bidirectional recurrent neural network on top of the 1D CNN output to encode them. Subsequently, we train a small neural network with 3 layers and 5 epochs based on the encoded information $X$ and binding affinity $ Y$. \n\nBelow, we present our basic results for the quantile selection rule. In this rule,  j-th sample is selected if $\\hat{\\mu}(X_j)$ is larger than the 35%-quantile over the predicted values in the test set. The results are outlined as follows, demonstrating that our procedure (SCPV) can control the FDR precisely.\n\n|  | FDR | Power |\n| --- | --- | --- |\n| SCPV | 9.72 | 76.67 |\n| OMT | 12.30 | 93.42 |\n\n\nDue to limited time, more experiments related to deep learning will be made in future version.  \n\n[1] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.\n\n[2] Mindy I Davis, et. al. Comprehensive analysis of kinase inhibitor selectivity. Nature Biotechnology, 2011.\n\n> Question: Could you compare or clarify the contribution with regards to the conditional calibration scheme mentioned in the article?\n\n**To Q**: Your question is very meaningful. Here we make a detailed discussion about the contribution with regards to the conventional conditional calibration.\n\nThe conventional conditional calibration [1] offers a flexible framework to decouple the dependence between p-values. It requires a carefully constructed quantity $\\Phi_j$, such that appropriate $c_i^*$ can be identified to satisfy the condition $\\mathbb{E}[\\mathbb{1}\\{p_j\\leq c^*_i\\}/|\\hat{\\mathcal{R}}_j|\\mid \\Phi_j]\\leq \\alpha/m$, where $p_j$ is a p-value under null, $\\hat{\\mathcal{R}}_j$ is a substitution of the original rejection set $\\hat{\\mathcal{R}}$ and $m$ is the number of test units. \n\nIn our selective setting, the number of test units is $|\\hat{\\mathcal{S}}_u|$, which can be complicatedly dependent with both $p_j$ and $\\hat{\\mathcal{R}}_j$. And when analyzing the FDR, the event that j-th sample is selected is also involved.  So our our primary focus is on ensuring $\\mathbb{E}\\left[\\frac{\\mathbb{1}\\{p_j\\leq c^*_i,j\\in\\hat{\\mathcal{S}}_u\\}}{|\\hat{\\mathcal{R}}_j|}|\\hat{\\mathcal{S}}_u|\\mid \\Phi_j\\right]\\leq \\alpha.$\nThe conditional calibration framework primarily focuses on the correlation of p-values. However, a significant challenge arises because **FDR control in a selective setting involves not only individual p-values but also the selection procedure itself**. Consequently, the selective effects are unavoidable when implementing conditional calibration. To address this, we leverage the stability property of the selection rule, which allows us to effectively conduct analysis over the selected subset effectively and rigorously.\n\n[1] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.\n\n> Limitation: The approach taken in the weak selection rule has a lower power.\n\n**To L**: Thank you for pointing this out. The conditional calibration approach will lose certain power due to the pruning process.  Our simulation results shown in the Appendix are based on deterministic pruning and it lacks power indeed. However, various techniques exist, which can enhance power through randomization. Here we present the improved results of heterogeneous random pruning, whose power is nearly as powerful as the BH procedure.\n\n|  | Case A |  | Case B |  |\n| --- | --- | --- | --- | --- |\n|  | FDR | Power | FDR | Power |\n| OMT | 15.8 | 97.1 | 16.1 | 83.8 |\n| Con | 9.86 | 93.4 | 9.80 | 78.1 |\n| BH | 9.86 | 94.0 | 9.80 | 78.4 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5q7PQC33D3", "reviewer": "Reviewer_9oVU", "summary": "This paper considers the problem of sample selection among a pre-specified group. The authors formulate this problem as a multiple testing problem and develop a procedure based on conformal inference, in which special treatment is adopted to find a specific calibration set that is exchangeable to the selected test unit. The proposed method achieves FDR control when the pre-selection rule has some strong exchangeability property.", "questions": "See the \"Weaknesses\" section.", "limitations": "The paper has partially discussed its limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper indeed considers an interesting problem, and the formulation makes sense.\n2. The paper is well-presented and easy to follow.", "weaknesses": "**Technical contribution.** As introduced in the paper, the technical difficulty of sample \nselection among the selected units with FDR control lies in (1) constructing valid p-values in the \npresence of selection and (2) dealing with the dependency between p-values for FDR control. \nThe current paper mainly addresses the first problem, while the second problem is only parially solved \nfor quite restrictive selection rules. In fact, finding the exchangeable group/constructing \nvalid conformal p-values for selected units has already been quite extensively discussed in [1] and [2]. \nIt would be helpful to clarify the technical contribution given this context.  \n\n1. Jin, Ying, and Zhimei Ren. \"Confidence on the focal: Conformal prediction with selection-conditional coverage.\" arXiv preprint arXiv:2403.03868 (2024).\n2. Bao, Yajie, et al. \"CAS: A General Algorithm for Online Selective Conformal Prediction with FCR Control.\" arXiv preprint arXiv:2403.07728 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weaknesses for technical contribution: As introduced in the paper, the technical difficulty of sample selection among the selected units with FDR control lies in (1) constructing valid p-values in the presence of selection and (2) dealing with the dependency between p-values for FDR control. The current paper mainly addresses the first problem, while the second problem is only partially solved for quite restrictive selection rules. In fact, finding the exchangeable group/constructing valid conformal p-values for selected units has already been quite extensively discussed in Jin and Ren (2024) and Bao et.al (2024). It would be helpful to clarify the technical contribution given this context.\n\n Thanks for your constructive comments on the technical contribution of our work. We acknowledge that our problem also relies on the idea of ``finding the exchangeable group/constructing valid conformal p-values for selected units'', which has been discussed in previous works [1] and [2]. However, our approach addresses a more challenging multiple testing problem. We would like to provide some clarifications to highlight the contribution of our work and explain how it differs from existing studies.  \n\n- **First, our problem setup differs significantly from previous works**, which focus on constructing prediction intervals after selection.  The goal of this paper is to **conduct multiple testing after data-dependent selection** such that the final rejection set has controlled FDR in a finite sample regime. This process involves the complex interaction among the p-values and the data-dependent selection process. In contrast, [1] and [2] both proposed a swapped strategy to construct valid conformal p-values after selection and then use them to build prediction intervals with selection conditional coverage. The selection conditional coverage is an individual notion and only requires the validity of a single p-value. However, this does not account for the correlations among p-values and thus does not guarantee the validity of multiple testing procedures.\n- Second, **our main technical contribution lies in developing a unified analytical framework for handling the randomness arising from data-driven selection in the context of multiple testing**. This framework builds upon the conditional calibration framework [4], but we go beyond it by tackling the **challenges imposed by selection** in FDR control. In our approach, the number of test units is denoted as $|\\hat{\\mathcal{S}}_u|$, which is a random variable that can have complicated dependencies with the p-values. Although it is possible to replace $|\\hat{\\mathcal{S}}_u|$ with a fixed value $m$for valid FDR control, doing so would be too conservative and would result in a significant loss of power.\n- Finally, to address the data-dependent selection effects, we leverage the **stability property** of the selection rule. Through a detailed investigation of the stability properties, we demonstrate that **our procedure can have finite sample FDR control across many important selection rules**. Notably, this advantageous property of stability was not investigated in [1] and [2]. For example, under the mean selection rule, which we have confirmed to be weakly stable in our framework, [1] and [2] did not fully leverage their stability characteristics, leading them to adopt a different approach for constructing selective conformal p-values. Our approach is based on $\\mathbf{S}_ {\\mathcal{D}_ c\\cup\\{Z_ j\\},\\mathcal{D}_ u\\backslash\\{Z_j\\}}$ in Section 3.3, while they are based on the swapped selection $\\mathbf{S}_{\\mathcal{D}_c\\backslash\\{Z_i\\}\\cup\\{Z_j\\},\\mathcal{D}_u\\backslash\\{Z_j\\}\\cup\\{Z_i\\}}$. Our adaptive selective p-value is more computationally efficient as it only requires computing the selection rule once instead of $|\\mathcal{C}_0|$ times for each test unit. Also, our p-value is related to the selected subset $\\mathbf{S} _{\\mathcal{D} _c\\cup\\{Z _j\\},\\mathcal{D} _u\\backslash\\{Z _j\\}}$, which has a closer relation to our FDR control analysis.\n \nWe hope these clarifications are helpful. If you have any further questions, please feel free to reach out.\n\n[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.\n\n[2] Bao, Yajie, et al. CAS: A General Algorithm for Online Selective Conformal Prediction with FCR the Control. arXiv, 2024.\n\n[3] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.\n\n[4] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgq6sEVqFP", "reviewer": "Reviewer_C3MM", "summary": "This paper proposes a method for multiple testing in the conformal setting that outputs the largest possible rejection set with FDR control contained with a data-dependent selection. The proposed method involves two steps: \n(1) constructing selective conformal p-values (i.e., p-values solely use points in the calibration set that are chosen by the selection rule),\n(2) applying the Benjamini-Hochberg (BH) procedure to the selective conformal p-values, but using a total hypothesis size of the selected set $|\\widehat{\\mathcal{S}}_u|$.\n\nThe method ensures FDR control only when the selection rule satisfies a form of \"stability\", which the authors provide several definitions for. Further, they provide several examples of practical selection rules which satisfy the stability condition (e.g., top-K, quantile), which either depend solely on the covariates in the test set, $\\mathcal{D}_u$ or along with the covariates of the calibration set $\\mathcal{D}_c$.", "questions": "The assumption of having access to a calibration set and still trying to perform selective inference in a conformal setup is a bit strange. One can directly estimate a population level threshold for selecting units based (with false positive error control) on the calibration set directly, since every unit (including the test units) is drawn from the same population --- this is what allows the conformal inference to succeed in this setup. This is quite different from the outlier detection application introduced in [2], where there are units/hypotheses where the distribution is not the same as the calibration setup, and hence one cannot estimate a population level threshold, since there is no singular notion of population. \n\nCould you elaborate on why the conformal setup makes sense here, instead of trying to directly estimate a rejection cutoff for $T(X_i)$ so the false positive rate is controlled at the population level?\n\n\n- [1] Yajie Bao, Yuyang Huo, Haojie Ren, and Changliang Zou. Selective conformal inference with false coverage-statement rate control. Biometrika, 2024.\n- [2] Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for outliers with conformal p-values. The Annals of Statistics, 2023.\n- [3] Ulysse Gazin, Ruth Heller, Ariane Marandon, and Etienne Roquain. Selecting informative conformal prediction sets with false coverage \nrate control. arXiv:2403.12295, 2024.\n- [4] Ying Jin and Emmanuel J Candès. Selection by prediction with conformal p-values. JMLR, 2023.\n- [5] Weijie Su. The FDR-Linking Theorem. arXiv:1812.08965, 2018", "limitations": "The limitations are well addressed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper provides an interesting approach to rejecting a discovery set that is a data-dependent subset of hypotheses. It's a nice combination of the techniques from [1] and [4] to solve a new selection multiple testing problem, and is similar to [3].", "weaknesses": "I think the main weaknesses of the paper is that it lacks comparison to two key baselines/prior art.\n\n1) Self-consistent/compliant adjustment: Using the marginal p-values in (1), one can directly achieve FDR control under any data-dependent selection simply by taking the largest self-consistent rejection set, i.e., the largest subset $\\mathcal{R}$ s.t. $p_i \\leq \\alpha' \\mathcal{R} / K$ for each $i \\in \\mathcal{R} \\subseteq \\mathcal{S}$, where $\\alpha'$ is the largest value that satisfies $\\pi_0 \\alpha' (1 + \\log(1 / (\\pi_0\\alpha'))) \\leq \\alpha$ --- here $\\pi_0 = |\\mathcal{C}_0| / |\\mathcal{C}|$ is the null proportion. This is a direct consequence of Theorem 3 of [5] and the PRDS property of conformal p-values from [2] referred to in your paper.\n\nNote that this is *not* the same as your AMT baselines --- those work with the selective conformal p-values (which I presume are less powerful than the marginal conformal p-value in (1)). I think understanding the performance of your procedure to this procedure would be key to seeing how the tradeoff between the gain in power from using a smaller BH threshold vs. loss in power from using a selective conformal p-value in your method compares to a method that purely uses marginal p-values.\n\nPost-rebuttal update: the authors ran these experiments and their method performs well. I have changed my score to an accept.\n\n2) InfoSCOP [3]: Although InfoSCOP is cited in the paper, it is not compared against --- the FCR guarantee of InfoSCOP directly implies FDR control on a data-dependent selection set simply by making its \"informative prediction set\" be informative against the null hypothesis being tested (the types of sets it is informative against are precisely the types of null hypotheses you are interested in testing against). In this vein, I think a more detailed comparison of your method and the InfoSCOP method is needed (i.e., what type of selection rules are allowed for each method, and how the power differs, etc.) for this paper to be comprehensive.\n\nI think the paper should include comprehensive comparisons to these rather significant baselines/prior art to merit publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: Comparison with self-consistent adjustment\n\nThank you for the valuable suggestion. We have incorporated theoretical and empirical comparisons with your proposed method. And these discussions will be added in the future version of our work.\n\n- From the theoretical point of view, we have observed that the power loss associated with utilizing a selective conformal p-value is usually less than that incurred by the FDR-Linking method. To illustrate this, assume $\\pi_0=0.7$ and $\\alpha=0.1$ as in the simulation setting of quantile selection, then we derive $\\alpha^{'}\\approx0.025$.  The AMT method adjusts the marginal p-value after selection by multiplying the selection proportion $\\hat{\\theta}=1/0.7$. This is equivalent to employing the BH procedure on the marginal p-value with $\\alpha=0.07$, which evidently yields greater power than SCA. Additionally, AMT does not make full use of the information from the selection procedure. In contrast, our proposed method uses a smaller p-value than AMT, which suggests more power increase.\n- In terms of empirical performance, as demonstrated in both cases from our paper, the SCA method suffers from a power loss, confirming our theoretical analysis.\n|  |   |  | QUAN |  | MEAN |  |\n| --- | ---  | --- | --- | --- | --- | --- |\n|  |  | FDR | Power | FDR | Power |\n| Case A | SCA  | 3.59 | 88.7 | 2.85 | 84.7 |\n|  | SPCV | 9.83 | **93.9** | 9.90 | **93.9** |\n|  | AMT  | 6.23 | 92.1 | 8.28 | 92.6 |\n| Case B | SCA  | 3.79 | 75.7 | 3.38 | 66.4 |\n|  | SPCV | 9.82 | **84.9** | 9.79 | **81.1** |\n|  | AMT  | 8.71 | 77.0 | 5.81 | 79.5 |\n\n> Weakness 2: Comparison with InfoSCOP\n\nThank you for the constructive comments and we apologize for overlooking this important reference. As you correctly point out, the InfoSCOP involves a procedure for FDR control after selection via applying BH procedure to selective conformal p-values, which aligns closely with the fundamental approach of our work. Notably, the FDR control guarantee in InfoSCOP requires the selection rule to satisfy a specific assumption, which can be transformed into the joint-exchangeable condition in our context. For strongly stable selections, our method can be simplified and degenerate into a form similar to InfoSCOP.\n\nWe would like to clarify the **following key differences** between our method and InfoSCOP:\n\n- Firstly, we provide the FDR control guarantee for general selection rules with strong stability, which is **beyond the joint-exchangeable selection**. For example, the assumption in InfoSCOP is not satisfied by the quantile selection rule based solely on test data. Thus, their theoretical results are not applicable in such cases, while our framework bridges this theoretical gap.\n-  Secondly, **our approach covers a wider range of selection rules**. For instance, when dealing with weakly stable rules, we employ conditional calibration on adaptive p-values to ensure rigorous FDR guarantees. The table below compares the performance of our approach with InfoSCOP under mean selection rule. The InfoSCOP shows reasonable empirical performance, which is similar to ours. Therefore, it is possible that InfoSCOP may still work under mean selection, making it an interesting topic for theoretical investigation, which remains unexplored in InfoSCOP. In contrast, we provide FDR control guarantee under a variety of selection scenarios.\n|  | Case A |  | Case B |  |\n| --- | --- | --- | --- | --- |\n|  | FDR | Power | FDR | Power |\n| InfoSCOP | 9.85 | 94.0 | 9.80 | 78.4 |\n| Ours | 9.86 | 93.4 | 9.80 | 78.1 |\n\n- Lastly, our approach and InfoSCOP are **designed for different goals, resulting in different analytical frameworks**. Ours is specifically designed to address the multiple testing problem across various selection rules. From the perspective of conditional calibration, our method is unified, where **the BH procedure for strongly stable selection can be seen as a special case**. As a comparison, InfoSCOP is an excellent work for selecting an informative set with FCR control, but it is **not primarily designed for multiple testing after data-dependent selection**. Their FDR guarantee is an extension of FCR control, which limits their method's applicability to different selection rules.\n\nBased on your suggestions, we will include a comprehensive comparison of our method with InfoSCOP in future version. Hope the above interpretation can ease your doubts. \n\n> Question:  Could you elaborate on why the conformal setup makes sense here?\n\nThank you for the insightful question. We would like to make some clarifications.\n\n- Indeed,  **our method is versatile and not limited to conformal setups.** For instance, in scenarios where only null-labeled data is available, such as in outlier detection, our approach can still generate selective conformal p-values and perform the appropriate procedures to control the FDR. In such cases, we assume that the distribution of calibration data is identical to the test data because the join-exchangeable selection rule is expected to be applicable in this scenario. However, it is not necessary for quantile or mean selection rules which based on test data only.\n- Your comment on population level based selection is indeed insightful and valuable, particularly for scenarios where the goal is to directly select a subset from the original data. However, our framework offers a broader applicability beyond this. In many cases, **we are only concerned about the specific selected subgroup, and the FDR over this set should be controlled**. For example,  in brain scan experiments, researchers hope to find brain locations for specific signal with FDR control. Given that there are several encephalic regions, each should be treated separately [1]. In such cases, controlling FDR within these subsets is of concern, and a global cutoff may have no theoretical guarantee. \n\n[1] Efron, B., Simultaneous Inference: When Should Hypothesis Testing Problems Be Combined? AOAS, 2008.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WOoJ15GPpA", "reviewer": "Reviewer_SifE", "summary": "This paper addresses the problem of conformalized multiple testing following data-dependent selection procedures. To manage the distorted distribution resulting from the selection process, the authors propose adapting the calibration set according to the selection rule. Under the assumption that the selection procedure is stable or weakly stable, the authors prove that the guarantee of the Benjamini-Hochberg procedure is still maintained. Experiments on synthetic and real data have shown the effectiveness and efficiency of the method.", "questions": "May I ask if the idea of adapting the calibration set is similar to constructing the reference set in [1]?", "limitations": "The authors have adequately addressed the limitations of the i.i.d. setting and noted that the stability assumptions of the selection procedures are relatively limited.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Multiple testing after data-dependent selection in the predictive setting is an important problem.\n2. The definition of selection stability and weak stability is formalized and generalized to more selection conditions. The authors provide extensive theoretical proof for the theorem.\n3. Adequate experiments demonstrate the effectiveness and efficiency of the proposed method.", "weaknesses": "1. The methodological novelty is limited: the idea of adapting the calibration set with the selection strategy to retain exchangeability is not very novel. Although dealing with conformal prediction, [1] also constructs the reference set based on the selection rule. The investigated selection methods are also similar. Overall, the main contribution of this method may be the rigorous proof (which, due to the length, I was unable to verify in every detail within the time constraints).\n\n2. Minor errors:\n\n    a. Line 28, $D_u = \\{Z_i\\}$ should be $D_u = \\{X_i\\}$.\n\n    b. Line 153, 'label set' should be 'labeled set'.\n\n[1] Ying Jin and Zhimei Ren. Confidence on the Focal: Conformal Prediction with Selection Conditional Coverage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Weakness 1: The methodological novelty is limited: the idea of adapting the calibration set with the selection strategy to retain exchangeability is not very novel. Although dealing with conformal prediction, [1] also constructs the reference set based on the selection rule. The investigated selection methods are also similar. Overall, the main contribution of this method may be the rigorous proof (which, due to the length, I was unable to verify in every detail within the time constraints).\n\n**To W1**: Thanks for your insightful comments on the methodological novelty of our approach. We agree that recent research has explored selective p-values from various perspectives. However, our work differs significantly from these studies, and we would like to provide some clarifications.\n\n- The references you mentioned have made valuable contributions by studying the properties of selective p-values, with a focus on individual p-values.  However, **our paper addresses multiple testing**, which requires studying the **interactions among all selective p-values rather than focusing on a single p-value**. This distinguishes our work from previous works in the literature. Specially, in our selective setting, the p-values are complicatedly correlated and the selection is also data-dependent. This makes the validity of conventional BH procedure suspicious and requires rigorous verification. \n- To address the difficulty arising from data-dependent selection, our main technical contribution lies in **developing a unified analytical framework**. This framework builds upon the conditional calibration framework [2], but it is not trivial due to the challenges imposed by selection in FDR control. **The number of test units **$|\\hat{\\mathcal{S}}_u|$** is a random variable** that can be complicatedly dependent with the p-values. Even though we can replace $|\\hat{\\mathcal{S}}_u|$ with $m$ for valid FDR control, it would be too loose and lead to reduced power. \n- To fix the selective randomness in analyzing FDR, we **utilize the stability property of selection rule**. Our stability condition helps mitigate the randomness in the test number which is not addressed in a single test. And dependence association between the selective p-values and the final rejection set can also be decoupled through the stability. \n\nWe hope that these interpretations can ease your doubts. If you have any further questions, please feel free to ask us.\n\n[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.\n\n[2] William Fithian and Lihua Lei. Conditional calibration for false discovery rate control under dependence. AOS, 2022.\n\n> Weakness 2: minor errors\n\n**To W2**: Sorry for our incautiousness. Thank you for pointing out these errors. We will revise them in the future revision. \n\n> Questions: Is the idea of adapting the calibration set similar to constructing the reference set in [1]?\n\n**To Q**: Your question is very thoughtful and we would like to discuss it with you in depth. \n\n- Indeed, our adaptive strategy has the same intension as \"swapped\" strategy, that is to maintain the exchangeability between selected calibration set and test unit.  From a theoretical standpoint, the selective p-values constructed by both strategies are valid p-values given the selection event.\n- **However, the core objectives of our adaptive strategy and the swapped rule in  [1] and [2] are fundamentally different.** Ours focuses on conducting multiple testing over these p-values and offering theoretical guarantees. In contrast, the p-values generated by the swapped rule are utilized to construct prediction intervals with selection conditional coverage, a property specific to individual cases and not reliant on interactions between these p-values.\n- **The motivation of our adaptive strategy is directly related to weak stability**, where the selection rule satisfies $\\mathbf{S}_{\\mathcal{D}_c,\\mathcal{D}_u}(X_i)=\\mathbf{S} _{\\mathcal{D}_c\\cup\\{Z_j\\},\\mathcal{D}_u\\backslash\\{Z_j\\}}(X_i)$ for any \n $j\\in\\hat{\\mathcal{S}}_u$ and $i\\in\\mathcal{U}$. Thus, the constructed p-values are based on the selection rule of $\\mathbf{S} _{\\mathcal{D}_c\\cup\\{Z_j\\},\\mathcal{D}_u\\backslash\\{Z_j\\}}$ and this is different from the swapping selection rule $\\mathbf{S} _{\\mathcal{D}_c\\backslash\\{Z_i\\}\\cup\\{Z_j\\},\\mathcal{D}_u\\backslash\\{Z_j\\}\\cup\\{Z_i\\}}$ . \n- Moreover, **the swapping selection rules are designed to satisfy a general range of selection scenarios rather than selection rules of specific properties**. For weakly stable selection rule $\\mathbf{S}$, employing the swapping selection rule$\\mathbf{S}_{\\mathcal{D}_c\\backslash\\{Z_i\\}\\cup\\{Z_j\\},\\mathcal{D}_u\\backslash\\{Z_j\\}\\cup\\{Z_i\\}}$ can yield a selection result $\\hat{S}_u^{-j}$ that differs from $\\hat{S}_u$, unlike our method. This difference complicates the substitution of $\\hat{S}_u^{-j}$ with $\\hat{S}_u$to mitigate the randomness of the test number. \n- **From an empirical standpoint, our strategy is more computationally efficient** since for each $j\\in\\hat{\\mathcal{S}}_u$, we only need to compute the selection rule once, while the swapping rule needs $|\\mathcal{C}_0|$times.\n\n[1] Jin, Ying, and Zhimei Ren. Confidence on the focal: Conformal prediction with selection-conditional coverage. arXiv, 2024.\n\n[2] Bao, Yajie, et al. CAS: A General Algorithm for Online Selective Conformal Prediction with FCR Control. arXiv, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to All Reviewers
Dear reviewers, thanks for your great efforts and valuable comments on our paper! We are glad that the reviewers found our paper "considers an interesting problem", "provides an interesting approach" and "provides extensive theoretical proof". Multiple testing after data-dependent selection in the predictive setting is an important problem, and we are the first systematic investigation to provide a unified solution.
First, we would like to emphasize our contributions again.

- Conducting multiple testing after data-dependent selection in the predictive setting is crucial in numerous real-world problems and **it is barely considered in previous work**. Our work **represents the pioneering effort in tackling this problem**.
- Our proposed procedure is theoretically verified to have **FDR control for data-dependent selection** with stability which **covers a wide range of selection rules**.
- Our method can be easily **integrated with any black-box prediction model for both regression and classification settings**. Extensive numerical experiments indicate the superiority of our method.

Next, we aim to expound our main theoretical challenge.

- There are more **complex randomness** to handle such as the test number which need not be considered in a single test. Our approach **incorporates stability conditions designed for the difficulties** encountered in multiple testing. It helps us to **mitigate the randomness** in the test number and the correlation between p-values.
- The existence of **selections beyond exchangeability** **brings** **extra intricate correlations** to the selective p-values and makes it difficult to do multiple testing. Our p-values are **constructed to decouple with the stability of selection,** particularly for the p-values under weakly stable selections. This construction can effectively counteract the correlations due to the non-exchangeability.

In the rebuttals, we have clarified the unique contributions of our research by highlighting its novelty compared to existing literature and supplemented our findings with additional numerical results to strengthen the validity of our method. In addition, we also explain the technical challenges faced by our approach.

Meta-review: The reviews are strong and agreement is high. The reviewers clearly appreciated 
the author rebuttal and adjusted their scores accordingly. The material provided
during the rebuttal phase clearly improves the manuscript. With that in mind,
I would strongly encourage the authors to incorporate these points and suggestions into their
revision.
Decision: Accept (poster)