False Discovery Proportion control for aggregated Knockoffs
Authors: Alexandre Blain
Bertrand Thirion
Olivier Grisel
Pierre Neuvial
Venue: NeurIPS 2023 poster (2023)
Keywords: Knockoffs
Derandomization of Knockoffs
False Discoveries Proportion control
Controlled variable selection
Statistical inference
High-dimensional inference

Abstract: Controlled variable selection is an important analytical step in various scientific fields, such as brain imaging or genomics. In these high-dimensional data settings, considering too many variables leads to poor models and high costs, hence the need for statistical guarantees on false positives. Knockoffs are a popular statistical tool for conditional variable selection in high dimension. However, they control for the expected proportion of false discoveries (FDR) and not the actual proportion of false discoveries (FDP). We present a new method, KOPI, that controls the proportion of false discoveries for Knockoff-based inference. The proposed method also relies on a new type of aggregation to address the undesirable randomness associated with classical Knockoff inference. We demonstrate FDP control and substantial power gains over existing Knockoff-based methods in various simulation settings and achieve good sensitivity/specificity tradeoffs on brain imaging data.

False Discovery Proportion control for aggregated Knockoffs
Alexandre Blain
Bertrand Thirion INRIA
INRIA
Université Paris-Saclay
CEA bertrand, thirion@inria.fr
alexandre. blain@inria.fr
Olivier Grisel
Pierre Neuvial
INRIA
Institut Mathématiques de Toulouse
Sivvier.griseliinria..o
Université de Toulouse
pierre nevviaettu umiv-toulo ouse. fr
Abstract
Controlled variable selection is an important analytical step in various scientific fields, such brain imaging genomics. In these high-dimensional data settings, considering too many variables leads to poor models and high costs, hence the need for statistical guarantees false positives. Knockoffs a popular statistical tool for conditional variable selection in high dimension. However, they control for the expected proportion of false discoveries (FDR) and not their actual proportion (FDP). We present a new method, KOPI, that controls the proportion of false discoveries for Knockoff-baas new type of aggregation to address the undesirable randomness associated with classical Knockoff inference. We demonstrate FDP control and substantial power gains over existing Knockoff-based methods in various simulation settings and achieve good sensitivity/speecffctt tradeoff's brain imaging and genomic data.
inference. The proposed method also relies
1 Introduction
Statistically controlled variable selection arises in many different application fields, when the aim to identify variables that important for predicting outcome of interest. For instance, in the context of brain imaging, practitioners are interested in finding which brain areas are relevant for predicting behavior or brain diseases. Such problems also appear in genomies, where practitioners wish to select genes associated with disease outcomes.
More precisely, we consider here conditional variable selection, meaning that we wish to select variables that are relevant to predict an outcome given the other variables. This type of inference substantially more challenging than marginal inference, especially in high-dimensional settings, where the number of variables exceeds the number of samples. This typically the case for brain mapping studies that comprise at most few hundred subjects (hence, samples), while modern functional Magnetic Resonance Imaging (MRI) scans consist of more than 100k voxels. In the context of conditional inference, those are typically reduced few hundreds of brain regions, still possibly more than the number of samples.
Importantly, statistical guarantees are needed to ensure that the inference is reliable i.e. that the proportion of false discoveries made by the variable selection procedure is controlled.
In the Knockoff' framework [ 6), this problem is tackled by building noisy copies of the original variables. These copies are then compared their original counterpart perform variable selection.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The intuition underlying Knockoffs is that irrelevant variables do not get larger weight than their Knockoff, while relevant variables do. Crucially, the Model-X Knockoffs procedure [6] controls the False Discovery Rate 2 which is the expected proportion of false discoveries.
A major caveat with this procedure is the random nature of the Knockoffs generation process: for two runs of the Knockoffs procedure on the same data, different Knockoff's will be built and subsequently different variables may be selected. This undesirable behavior hinders reproducibility. A second caveat is that False Discovery Rate (FDR) control does not imply False Discovery Proportion (FDP) control [2]. This leads to potentially unreliable inference: single runs of the method can produce a much higher proportion of False Discoveries than the chosen FDR level.
In this work, propose a novel Knockoff- knockoff-based inference procedure that addresses both concems while offering power gains over existing methods, for no significant computation cost. The paper is organized follows. After a refresher Knockoff inference and aggregation, we consider the statistic introduced in 17 to rank variables by relevance. Using the symmetry of knockoffs under the null hypothesis, we construct explicit upper bounds on the Joint Error Rate (JER; 4 of these statistics, leading to FDP control. We then use the calibration principle of 4 to obtain sharper bounds. Finally, we obtain robust version of this method using harmonic mean aggregation of the π statistics across multiple Knockoffs draws. We demonstrate empirical power gains in various simulation settings and show the practical benefits of the proposed method for conditionally important region identification on fMRI and genomic datasets.
2 Related work
There has been much effort in the statistical community achieve derandomized Knockoff- based inference. [19] introduced the idea of running Model-X Knockoffs [6] multiple times and computing for each the proportion of runs for which it was selected. [9] explore the idea of sampling multiple Knockoffs simultaneously. This induces a massive computational cost, which is prohibitive compared methods that can support parallel computing. (17) introduced an aggregation method that relies on (BH) procedure [2] on so-called intermediate p-values. Such p-values can be computed on different Knockoff runs and aggregated using quantile aggregation [5 then, BH is performed on the aggregated p-values to select variables. This approach relies on the heavy assumption that Knockoff statistics are i.i.d. under the null. Additionally, it is penalized by the conservativeness of the quantile aggregation scheme. Alternative aggregation schemes such as the harmonic mean 28 can be used but do not yield valid p-values.
viewing Model-X Knockoffs as a Benjamini-Hochie
18 introduced an alternative aggregation procedure where Model-X Knockoffs are viewed as an e-BH procedure (25) on well-defined e-values (24. Since the mean of two e-values remains an e-value, aggregation is done by averaging e-values across different Knockoff' draws. Then, e-BH performed on the aggregated e-values select variables. FDR control on aggregated Knockoffs is achieved without any additional assumption compared to Model-X Knockoffs. However, this method requires the difficult setting of a hyperparameter related the chosen risk level, which highly impacts power in practice. Other recent developments in Knockoffs include the conditional calibration framework of Luo et al. [4] which aims improving the power of Knockoffs- methods.
There have been few attempts at controlling other type I errors than the FDR using Knockoffs. achieves k-FWER control and proposes that FDP control can be obtained by using procedure that leverages joint k-FWER control. Recently, B
introduced such procedure reach FDP control based on the k-FWER control introduced in I In summary, the KOPI approach is the first one that aims at controlling the FDP of knockoffts-bssed inference for any aggregation scheme, leading to both accurate FDP control and increased sensitivity.
3 Refresher on Knockoffs
Notation. We denote vectors by bold lowercase letters. A vector = . .pp from which removed the 5th coordinate is denoted by x-j, i.e. x {zz). }. Independence between two random vectors x and y is denoted by x 1 y. For two vectors x and x and a subset S of indices, (x,,))swo((s)) denotes the vector obtained from (x, x) by swapping the entries 2y and ū, for each j E S. Matrices denoted by bold uppercase letters, the only exception being the vector of Knockoff statistics
2

---

that we denote by W as in [ []. For any set S, |S| denotes the cardinality of S. For vector z = (zg)ı and S C [p]. we denote by F(ss)) (or zG) when there is no ambiguity) the j smallest value in the sub-vector (za)aEs. For an integer k; [ denotes the set (1...., k}. Equality
in distribution denoted by _.
Problem setup. The input data are denoted by X E Ruxp, where n is the number of samples and p the number of variables. The outcome of interest is denoted by y IR" The goal is select variables that are relevant with regards to the outcome conditionally on all others. Formally, we test simultaneously for all j E [pl:
Ho. 1 1z versus H1 y xz1x3,
The output of a variable selection method is rejection set $ C [P] that estimates the true unknown support H, (i zj [,|xx Its complement is the true null hypotheses Ho {i y 1 Ty =y|x-5}. Its cardinality [Mol is denoted by Po- To ensure reliable inference, our aim is to provide statistical guarantee on the proportion of False Discoveries in S. The False Discovery Proportion (FDP) and the False Discovery Rate (FDR) [ are defined as:
1 Hol FDP($)
S Hol
FDR(Š) =
|S| V1
|$| v/
An a-level post-hoe FDP upper bound [10] is function V that verifies:
IP (AS C FDP(() s V(S)/(S1) 2 1 -1.0..
The Knockoff filter is a variable selection technique introduced by [ and refined by 66 which controls the FDR. This procedure relies on building noisy copies of the original variables called Knockoff variables, that are designed to serve as controls for variable selection.
Knockoffs.
Definition (Model-X Knockoffs, (). For the family of random variables x = (II,..., Zp)) Knockoffss new family of random variables ă (æ1,..., Ep) satisfying:
1. for any S [p]. (x, smap(() (x)
2. x y'x.
Once we have such variables our disposal, we quantify their importance relative to the original ones. This done by computing Knockoff statisties W (Wi,..., Wp) that are defined as follows.
Definition 2 (Knockoff Statistic, (). A knockoff statistic w (W1,... Wp) is measure of feature importance that satisfies:
1. W depends only on X, X and y: W = X.
2. Swapping column x, and its knockoff column &, switches the sign of Wj;
W swap(s), = W, W X y) W, ([X ifjj
The most commonly used Knockoff statistic is the Lasso-coefffcieent difference (LCD) 27 This statistic obtained by fitting Lasso estimator 222 on [X, X Rax2p which yields R
Then, the Knockoff statistic can be computed using B:
Vj [pl.
W,
|3,1 |3
This coefficient summarizes the importance of the original jth variable relative its own Knockoff: W; >> O indicates that the original variable is more more important for fitting y than the Knockoff variable, meaning that the jth variable is likely relevant. Conversely, W, < O indicates that the jth variable probably irrelevant. We thus wish select variables corresponding
large and positive W,- Formally, the rejection set S can be written $ (j W, > Tg), where T is chosen to provably control the FDR at level y (6).
3

---

Aggregation schemes. Due to the randomness in the knockoff generation process, different variables may be selected for two different runs of the method, which is undesirable. To mitigate this, aggregation of multiple Knockoffs runs is needed. Ren and Barber [88 introduced an aggregation scheme which relies defining Knockoffs e-values.
P
=
(k: W S Tg) (W,2T.)
Such e-values can averaged across D draws and e-BH [25] is performed for variable selection. Alternatvely, [17] defines the following r-statistic, that quantifies the evidence against variable:
S W if
W > 0
W 50 S.
In ır statisties are treated as p-values and aggregated using quantile aggregation [5]. However, they can only be considered p-values under restrictive assumptions that are hard to check. In the next section, these statistics are used as building block to reach FDP control. The KOPI framework does not require statistics to be valid p-values.
4 Main contribution: FDP control for aggregated Knockoffs
4.1 Post hoc FDP control for statistics
To obtain FDP control, we rely on Joint Error Rate control as introduced in 4 For kmaz [p]. we define a threshold family of size kman as vector t = (t)))e[[Rmaa] such that 0 < ti S ... S
Ukmaz S 1.
Definition 3 (Joint Error Rate, 4). Denote by *(:Ho) the jth smallest value ny amongst all null hypotheses. The JER associated with t = (ty)je[kmaoll is:
JER(t)= TP (aj [kmax pol TYHHo) < <tz)-
The threshold family t is said control the JER level iff JER(t) a.
An a-level FDP upper bound can be derived from JER control via the following result:
Proposition (FDP control via JER control 4). If't is a threshold family of length kmar that controls the JER level a, then, (S) is a-level FDP upper bound, with:
V(()
min Ickgkman
- (1-1) Σ [mx2t))) iES
(3)
The proof of this result originally included in 4 containedness. In the remainder of this section, we show how to obtain JER control for т statistics.
be found in appendix A.] for self-
4.2 Joint distribution of π statistics under the null
By Definition 3 (ER(t) of given threshold family only depends the joint null distribution of the statistics. As for earlier FDR control [ or k-FWER control [I]] results, the key idea to obtain JER control for statistics is prove that the relevant part of this distribution is in fact known, thanks to the properties of knockoff statistics. We use the same notation as in [ Letting Z, [k e [p] W2 < WW, and X; sign (W,). the i statistics (πz)ş=(ol are given by:
1 Zj [x=11 + 1x,=-13-
P
For given W, let a (W) a permutation [p]] that sorts by decreasing modulus: o(W) (oi,..., .p) such that [W,, 1, > [Wazl... |W,,l We start by proving that the Z statistics can be expressed as a function
the vector x statistics:
Lemma 1. For j [p] such that Xoy Zo; -E 1xx, =-1)

---

Proof of Lemma 7
Since No, = 1, we have:
Za = l[k [| Wa, - -Wa, H1
= l(kE E p| : Wox < 0 and Waa - W
1{k [p] Wo, and |Wo, [Wa,
[kk E |p]
Wor
0 and k 5}
Σ 1xo, =-13
Lemma limplies that the distribution of order statistics (Wo entirely determined by that of xlo((W W). To formalize this, we introduce x² statistics.
Definition 4 (x statistics). Let x s/Sp be collection of ii.d. Rademacher random
variables, that is, for all j. P(X) 1) - P(() 1) 1/2. The associated 7" statistics defined j [p] by
1+ 1 x5
p 1x8==13 1xf=-11, where Z3
Σ
(xf=-11
(4)
k=
Theorem 1. Let t be threshold family of length kmaz. Then, for TO = (m]]]](()) as in ④
JER (t) & JER (t) = IP (Ek [kmar] T(k) tx
(5)
Proof of Theorem 7 Let kt E [kmaa]] Since t5 I have T(k:Ho)) tx if and only if N₂ 2 k. where
N₂ {ie Ho. X; and
1 1 Z3 t
With the notation of Definition
we define the random variable
No { E Ho. xi = and + 2
4.
lf [p]. then Lemma limplies that conditional on o (W)) N₂ and Nţ have the same distribution. Indeed, the vectors (Wy)i/x=== and (2))//x==1 have the same ordering, and conditional on o (W). (G))sn, are jointly independent and uniformly distributed on {-1, 1} (Lemma 2.1 in [. Using the same argument in the proof of Lemma 3.1 Janson and Su M. in the case where Ho & [p] false null Xj will insert -I's into the process on the nulls, implying that N₁ is stochastically dominated by No. Noting that No Z & if and only if (()) < t4. we obtain that
IP (3k [Kkmax A pol, ((::H0)) < (xlo((W) < IP 3k E [kmaw A pol, T(a) < tx)
PP 3k [kmaull [g,) t
Taking the expectation with respect to o(W) yields the desired result.
Theorem is related to Lemma 3.1 of Janson and Su || and Lemma 3.1 of Li et al. 13,, that rely the flip property of Knockoff statistics under the null [ The interest of Theorem is that the upper bound JER" (t) only depends on the n² statistics and the threshold family t,, and not on the original data. Therefore, it can estimated with arbitrary precision for any given using Monte-Carlo simulation, as explained in the next section and described in Algorithm in Supp. Mat.
5

---

4.3 Joint Error Rate control for и statisties via calibration
To approximate the JER upper bound derived in Theorem [ we draw B Monte-Carle samples using Algorithm |
This yields set of B vectors of T statistics denoted by T E Rp for each b E [B]. This allows us to evaluate the empirical JER, which estimates the upper bound of interest.
Definition (Empirical JER). For B vectors of n² statistics and a threshold family the empirical JER defined as:
JER B (t) B
1
Σ
Ek
[kmar]] Tb(k)) 1k
b=1
where for each b E [B], ((1 1 ... S b(p))
Since (t) can be made arbitrarily close (by choosing large enough) to JER (t) for any given threshold family remains to choose such that JER (t) S a in order to ensure JER control. To this end, we consider sorted set of candidate threshold families called template:
Definition 6 (Template 44.. A template is component non non-decreasing function T : [0, 1] + Iº that maps a parameter E [0, 1] threshold family T(A) R".
This definition is naturally extended to the case of templates containing finite number of threshold families. The template corresponding to B' threshold families is then denoted by (B)/B"))
Once template is specified, the calibration procedure 4 can be performed: this consists in finding the least conservative threshold family t amongst the template that controls the empirical JER at level a. Formally, consider the threshold family defined to -a)),(a)), where
AB(a)= B'
max b' [B']
s.t. JER
T
1
{a
As observed by Blain et al. 3. optimal power is reached when the candidate families match the shape of the distribution of the null statistics. We define template based on the distribution of the n²" statistics appearing in Theorem | In practice, we draw B' samples from this distribution independently from the B Monte Carlo samples to avoid circularity biases. Since a template has to be component-wise non-decreasing, i.e. the set of candidate threshold families has to be sorted, extract empirical quantiles from these B' sorted vectors. This yields template T composed of B' candidate curves that match quantiles of the distribution of x² statistics. The B -quantile curve defines the threshold family T (6/ B'). We obtain the following result:
Theorem 2 (JER control for r-statistics). Consider the threshold family defined by to = ()s(()))
Then, as B + +00,
(E a a
The number B of Monte-Carlo samples in Theorem can be chosen arbitrarily large to obtain JER control, leading valid FDP bounds via Equation This result is proved Appendix A.2
4.4 False Discovery Proportion control for aggregated Knockoffs
In the previous section we have seen how to reach FDP control via Knockoffs. As explained above, aggregation is needed to mitigate the randomness of the Knockoff generation process. Therefore, we aim to extend the previous result to the case of aggregated Knockoffs. Let us first define aggregation: Definition 7. For D draws of Knockoffs, an aggregation procedure a function f RRP + that maps vector of (πr ( [e[ [n] statistics to aggregated statistic न..
In practice, since we have p variables, aggregation is performed for each variable, i.e.:
Vj E [p], [TT).....
Then, inference is performed on the vector of aggregated statisties (T..... T.

---

For fixed aggregation scheme f. we can naturally extend the calibration procedure of the preceding
section. Instead of drawing a single B p matrix of x² statisties containing gj. R' for each
b E [B], we draw D such matrices. Given d E [D]] each matrix contains % E I& for each [B].
Then, for each [B]] we perform aggregation: #B (mAA"))E11)) The JER in the aggregated case is defined as:
JER(t) P (2j [kmaz A pol (o)Ho) < <t))
We obtain the aggregated template following the procedure, i.e. drawing D templates and aggregating them. For each E [['], the aggregated threshold family is written:
b' B'
(( T
T
f
dE[Di
We then write the empirical JER in the aggregated case as:
B
JER T B' =
-
Σ Bi 3i
[kmazl BB T,
B'
Calibration can be performed in the same way as in the non-aggregated case. Note that we perform calibration after aggregating; therefore, JER control is ensured directly on aggregated statisties and is not a result of aggregating JER controlling families. Importantly, this approach holds without additional assumptions on the aggregation scheme f. We consider the threshold family 5. = (a),(a)), where
1/ B'
AB(a)= max b' [B]]
s.t. JER R
T
S
saj-
With T a template composed of B' candidate curves that match quantiles of the distribution of = statistics, we obtain the following result:
Theorem 3 (JER control for aggregated --statistics). Consider the threshold family defined by F
ta T (Ns(a)).. Then, as B - +00,
JER < a Op(1/BB).
Proof. The proof is identical that of Theorem Juiing the empirical aggregated JER.
The calibrated aggregated threshold family yields valid FDP upper bounds via Proposition 1 The proposed KOPI (Knockoffs т) method therefore achieves FDP control on aggregated Knockoffs.
Experiments
Methods considered. In our implementation of KOPI, we rely on the harmonic mean (28) as the aggregation scheme f. Additionally, we set kmaz Lp/50] following the approach of B.. We also consider both state-of-the-aee Knockoffs aggregation schemes: AKO (Aggregation of Multiple Knockoffs, 17) and e- e-values based aggregation [8] Additionally, we consider Vanilla Knockoffs, i.e. [6] and FDP control via Closed Testing [13]. In simulated data experiments, we generate Knockoffs assuming a Gaussian distribution for X, with variables centered. For methods that support aggregation, we use D 50 Knockoff draws.
5.1 Simulated data
Setup. At each simulation run, generate Gaussian data X RRxp with a Toeplitz correlation matrix corresponding to first-order auto-regressive model with parameter p, i.c Σi,; pli-jl,
Then, we draw the true support 3 E 10. The number of non-null coefficients of 3 is controlled by the sparsity parameter Sp, i.e. 8p |/3 llo/p. The target variable is built using a linear model:
= X,3 tcc,
7

---

KOPI
Closed Testing
Vanilla
0.4
0.2
0
1.00
Empiricall power
0.75
0.50
0.25
0.00
1000
Signal-to-nolse ratio
n_samples
sparsity
Figure 1: FDP bound coverage at level and empirical Power for 50 simulation rums and five different methods: Vanilla Knockoffs, aggregated Knockoffs using e-values, aggregated Knockoffs using quantil-aggregattonn KOPI and Knockoff inference via Closed Testing. We use D 50 Knockoffs draws and the following simulation settings: represents varying parameter with the first row displaying FDP coverage and the second row displaying power. The red line and associated error bands represent the acceptable limits for FDP bound coverage. KOPI consistently outperforms all other methods while retaining FDP control.
0.1, q 0.1, p 500. Each column
with controlling the amplitude of the noise:
122(SRR [e]l)), SNR being the signal-to- noise ratio. We choose the central setting n = 500, 500, P = 0.5, 8, = SNR = 2. For each parameter, we explore range of possible values to benchmark the methods across varied settings.
To select variables using FDP upper bounds, we retain the largest possible set of variables such that V(S) S| (Algorithm 4). For each of the N simulations and each method, compute the empirical FDP and True Positive Proportion (TPP):
n Hol [S]
and TPP(S) IS Hil H1l
FDP(S)
If the FDP is controlled at level KK [N] FDP((s)) > l6l B(N, a). Then, we compute Va(1 - a)/N. The second row of Fig.
error bands the a-level using std (B(N, represents the empirical power achieved by each method, which corresponds to the average of TPPs defined above for N runs i.c. Power > TPP(S.)/N Fig. shows that across all different settings, KOPI retains FDP control. We can also see that FDR control does not imply FDP control, consistently outside of FDP bound coverage intervals. However, the two
Vanilla Knockoff' existing aggregation schemes (AKO and e- e-values) that formally guarantee FDR control are generally conservative and achieve FDP control empirically. This is consistent with the findings of [18]. The Closed Testing procedure of 13 achieves FDP control as announced but suffers from a lack of power.
Interestingly, KOPI achieves FDP control while offering power gains compared to FDR-controlling Knockoff's aggregation methods. Yet FDP control a much stronger guarantee than FDR control, as discussed previously. These gains are especially noticeable in challenging inference settings where most methods exhibit a clear decrease in power or even catastrophic behavior (i.e. zero power).
Moreover, Fig. 3 (in appendix) shows that when using 0.05 rather than 0.1 as in Fig. the robustness of KOPI with regards difficult inference settings is even more salient. More precisely. for q 0.05. AKO and Closed Testing are always powerless. E-values aggregation yields good power easier settings such as p < 0.6, SNR > 2.5 or n > 750 but exhibits catastrophic behavior in harder settings. Overall, apart from KOPI, only Vanilla Knockoffs exhibit non-zero power, but this method fails control the FDP as is intended to control FDR. KOPI preserves FDP control in all settings while yielding superior power compared to all other methods.

---

0.20
0.5
False Discovery Proportion
0.4
Empirical Power
0.10
0.2
0.05
0.00
0.0
AKO Vanilla KO Figure Empirical FDP and power on semi-simulated data for 42 contrast pairs. We use HCP contrasts CO: "Motor Hand", C1: "Motor Foot", C2: "Gambling", C3: "Relational", C4: "Emotion", C5: "Social", C6: "Working Memory". We consider all 42 possible train/test pairs: the train contrast used to obtain a ground truth, while the test contrast is used to generate the response. Inference is performed using the 5 methods considered in the paper and the empirical FDP is reported in the left box plot, while power is reported in the right box plot. Notice (right figure) that KOPI yields superior power compared to all other Knockoffs-bert basedd methods while controlling the FDP (left Fig.).
KOPI
CT
e-values
AKO
Vanilla KD
KDPI
CT
e-values
5.2 Brain data application
The goal of human brain mapping is associate cognitive tasks with relevant brain regions. This problem is tackled using functional Magnetic Resonance Imaging (fMRI, which consists in recording the blood oxygenation level dependent signal via MRI scanner. The importance of conditional inference for this problem has been outlined in [26]. We use the Human Connectome Project (HCP900) dataset that contains brain images of healthy young adults performing different tasks while inside an MRI scanner. Details about this dataset and empirical results can be found Appendix B
While these results demonstrate the face validity of the approach, FDP control and power cannot be evaluated. Therefore, following [16), we consider an additional experiment that consists in using semi-simulated data. We consider first IMRI dataset (X1,y 1) on which perform inference using a Lasso estimator; this yields 3 E IRP that we will use as our ground truth. Then, we consider separate fMRI dataset (X2, y2) for data generation. The point of using separate dataset is to avoid circularity between the ground truth definition and the inference procedure. Concretely, we discard the original response vector y2 for this dataset and build simulated response y2 sim using linear model, with the same notation as previously (we set a so that SN = 4): y
X + ae.
Then, inference is performed using Knockoffs-based methods on (X2, yzim). Since consider 3 as the ground truth, the FDP and TPP can be computed for each method. As can be seen in Fig. 2 KOPI is the most powerful method among those that control the FDP.
5.3 Genomic data application
In addition to the brain data application, we compared KOPI to other Knockoffs- methods on expresssess data (5] containing 79 samples and 90 genes. KOPI yields non trivial selection for all runs, with 3 genes selected in 100% of all 50 runs of the experiment. Across all runs, only 8 different genes are selected by KOPI. Vanilla Knockoffs select different genes across all runs and no gene exceeds a selection frequency of 70%. All other methods are powerless in all runs. Details and results this experiment can be found in Appendix
6 Discussion
In this paper, we have proposed a novel method that reaches FDP control on aggregated Knockoffs. combines the benefits of aggregation, i.e. improving the stability of the inference, in addition to providing probabilistic control of the FDP, rather than controlling only its expectation, the FDR.

---

Simulation results support that KOPI indeed controls the FDP. Furthermore, while FDP control is stricter guarantee than FDR control, KOPI actually offers power gains compared to state-of-the- aggregation-based Knockoffs methods. This sensitivity gain is direct benefit from the JER approach and its adaptivity to arbitrary aggregation schemes. While the latter has been formulated and used so far in mass univariate settings [3], the present work presents first use of this approach in the context of multiple regression. Morcover, KOPI does not require any assumption on the data at hand or on the law of Knockoff statistics under the null.
The computation time of the proposed approach comparable existing aggregation schemes for Knockoffs: sampling statistics under the null using Algorithm can be done once and for all for given value of p. JER estimation via Algorithm 2 and calibration can be performed via binary search of complexity (log (B').. Finding the rejection set $ after performing calibration is done in linear time via [7]. In practice, the computation time is the same for classical knockoff aggregation [9] and is in minutes for the brain imaging datasets considered. Avenues for future work include theoretical analysis of the False Negative Proportion (ENP) 8 of KOPI and developing step-down version of the method further improve power.
We provide a Python package containing the code for KOPI available at httph: //gt:////hhhbb com/ alexbInn/K0PI
10

---

7 Acknowledgments and disclosure of funding
This project was funded by a UDOPIA PhD grant from Université Paris-Saclay and also supported by the FastBig ANR project (ANR-17-CE23-00) I), the KARAIB AI chair the H2020 Research Infrastructures Grant ABRAIN-Healle 101058516 and the SansSouci ANR project (ANR-16 11R-16-CE40-0011 The authors thank Binh Nguyen for his precious help on the code base and Samuel Davenport for useful discussions about this work.
References
[1] Barber, F. and Candès, E. (2015). Controlling the false discovery rate via knockoffs. The Annals of Statistics, 43(5):2055-2005 2085.
[2] Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Method- ological), (7(1):289-300.
[3] Blain, A., Thirion, B.,, and Neuvial, P. (2022). Notip: Non-parametric true discovery proportion control for brain imaging, Neurolmage, 260:119492
[4] Blanchard, G., Neuvial, P., Roquain, E., et al. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics, 48(3):1281-1300
[5] Bourgon, R., Gentleman, R., and Huber, W. (2010). Independent filtering increases detection experiments. Proceedings of the National Academy of Sciences,
power for 107(21):956695511
[6] Candès, E., Fan, Y., Janson, L., and Lv, J. (2018). Panning for gold: mmoellxx modellxX'knockoff high dimensional controlled variable selection. Journal of the Royal Statistical Society: Series (Statistical Methodology), 80(3):551-577.
knockoffs for
[7] Courrech, N. and Neuvial, P. (2022). Powerful and interpretable control of false discoveries two-group differential expression studies. Bioinformaticx, 38(23):521455221
[8] Genovese, C. and Wasserman, L. (2002). Operating characteristics and extensions of the false discovery rate procedure. Journal the Royal Statistical Society Series B: Statistical Methodology, 64(3):499-517.
[9] Gimenez, J. R. and Zou, J. (2019). Improving the stability of the knockoff procedure: Multiple simultaneous knockoffss and entropy maximization. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2184-2192. PMLR.
[10] Goeman, J. J. and Solari, A. (2011). Multiple testing for exploratory research. Statistical Science, 26(4):584-597.
[11] Janson, L. and Su, W. (2016). Familywise error rate control via knockoffs. Electronic Journal of Statistics, 10(1):90-9975
[12] Korn, E. L., Troendle, J. F., McShane, L. M., and Simon, R. (2004). Controlling the number of false discoveries: application to nigh-dimensional genomic data. Journal of Statistical Planning and Inference, 124(2):379-398.
[13] Li, J., Maathuis, M. H., and Goeman, J. J. (2022). Simultaneous false discovery proportion bounds via knockoffs and closed testing, arXiv preprini urXiv:2212.12822.
[14] Luo, Y. Fithian, W. and Lei, L. (2022). Improving knockoffs with conditional calibration.
[15] Meinshausen, N., Meier, L., and Bühlmann, (2009). P-values for high-dimensional regression. Journal of the American Statistical Association, 104(489):67)-11681
[16) Nguyen, B. T., Thirion, B., and Arlot, S. (2022). A Conditional Randomization Test for Sparse Logistic Regression High-Dimension. In NeurlPS 2022, volume 35 of Advances in Neural Information Processing Systems, New Orleans, United States.

---

[17] Nguyen, T.-B., Chevalier, J.-A., Thirion, B., and Arlot, S. (2020). Aggregation of multiple knockoffs. In International Conference on Machine Learning, pages 7283-7293. PMLR.
[18] Ren, Z. and Barber, R. (2022). Derandomized knockoffs: leveraging e-values for false discovery rate control. arXiv preprint arXiv:2205. 15461.
[19] Ren, Z., Wei, Y., and Candès, E. (2021). Derandomizing knockoffs. Journal of the American Statistical Association, pages 1-11.
[20] Stiglie, G. and Kokol, P. (2010). Stability of ranked gene lists in large microarray analysis studies. Journal of biomedicine and biotechnology. 2010.
[21] Thirion, B., Varoquaux, G., Dohmatob, E., and Poline, J.-B. (2014). Which fMRI clustering gives good brain parcellations? Frontiers in Neuroscience, 8(167):13.
[22] Tibshirani. R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methudological), (86))2267-
[23] Van Essen, D. C., Ugurbil, K., Auerbach, E. J., Barch, D. M., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L.,, Corbetta, M., Curtiss, S. W., Della Penna, S., Feinberg, D. A., Glasser, M. F., Harel, N., Heath, A. C., Larson-Prionn Prior, L. J., Marcus, D. S., Michalareas, G., Moeller, S., Oostenveld, R., Petersen, S. E., Prior, W., Schlaggar, B. L., Smith, S. M., Snyder, A. Z., J., and Yacoub, E. (2012). The Human Connectome Project: a data acquisition perspective. Neuroimage, 62(4):2222-2231.
[24] Vovk, V. and Wang. (2021). E-values: Calibration, combination and applications. The Annals of Statistics, 1736: 1754.
[25] Wang, R. and Ramdas, A. (2022). False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3)822--85.
[26] Weichwald, S., Meyer. T., Özdenizei, O., Schölkopf, B.. Ball, T., and Grosse- waaseepp M. (2015). Causal interpretation rules for encoding and decoding models in neuroimaging, Neurolm- age, 110:48-59.
[27] Weinstein, A., Su, J., Bogdan, M., Barber, R. F,, and Candès, E. J. (2020). A power analysis for knockoffs with the lasso coefficient-diiferenne statistic. arXiv preprint arXiv:2007.15::
[28] Wilson, D. J. (2019). The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences, 116(4):1195-B) 1200.
12


Reviews:
[{"review_id": "hdoTzYSaY7", "reviewer": "Reviewer_MbEh", "summary": "In this paper, the authors discuss controls for false discoveries in variable selection. While how to perform FDR control is known, authors opt to control FDP, which is a random quantity depending on a specific dataset. The main concept is an upperbound of FDP (proposition 1), namely JER. The monte carlo version of JER is applied to perform FDP control. The authors also propose an aggregated JER to reduce the stochasticity JER for variable selection. Experiments show good FDP control while maintaining good power. ", "questions": "I am not sure Why JER, not the empirical JER is studied in Theorem 2? In practice, shouldn't one use empirical JER with monte carlo samples? ", "limitations": "Yes. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well written. \n2. FDP (not just FDR) control is an important problem. This paper targets on an important issue in our community. \n3. The proposed method has clear theoretical support (theorem 2 and 3), although I have some questions. ", "weaknesses": "It is not entirely clear to me the delta between this paper and an important related work, i.e, [4]. From Section 4, it seems that using JER to control FDP has already been proposed. Thus, I wonder how significant the contribution of this work is. \n\nFor example, has Proposition 1 or a similar version already been proposed in [4]? I would appreciate a short paragraph comparing this work and [4]. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n> It is not entirely clear to me the delta between this paper and an important related work, i.e, [4]. From Section 4, it seems that using JER to control FDP has already been proposed. Thus, I wonder how significant the contribution of this work is.\n\n\n>For example, has Proposition 1 or a similar version already been proposed in [4]? I would appreciate a short paragraph comparing this work and [4].\n\nWhile the JER framework has indeed been proposed in [4], this work's main contributions differ substantially from [4]:\n\n* [4] focuses on marginal testing (e.g. massively multiple t-tests) and not conditional testing. KOPI is the first procedure to combine the JER framework and **conditional testing via the Knockoffs framework and $\\pi$-statistics.**\n\n* In general, the JER cannot be computed analytically since it requires knowledge of the distribution of the test statistics under the null. In practice [4] relies on permutation schemes to estimate this distribution. This is computationally infeasible in the context of Knockoffs: such a procedure would require computing new Knockoffs for each permutation run. Theorem 1 of this work is, to our knowledge, **the first tractable upper bound on the JER of the $\\pi$-statistics or any other Knockoffs-based statistic**.\n\n* **KOPI supports aggregation**. Aggregation of test statistics is not considered in [4], which focuses on JER control on $p$-values. In this work, we show that **FDP control stemming from JER control can be made robust using aggregation.**\n\nRegarding Proposition 1: indeed this proposition corresponds to Proposition 2.3 in [4] - we simply restate it for completeness and self-containedness and include the proof in the supplementary material. To clarify this, **we have added a sentence in the manuscript** that points to the proposition in [4].\n\n> I am not sure Why JER, not the empirical JER is studied in Theorem 2? In practice, shouldn't one use empirical JER with monte carlo samples?\n\nTo reach exact FDP control, the **JER itself** - and not the empirical JER - has to be controlled. While indeed $\\mathbf{t}^B_\\alpha$ is built using Monte Carlo samples in practice and controls the empirical JER by definition, we need to insure that **the JER of $\\mathbf{t}^B_\\alpha$ is controlled to obtain FDP control**. Theorem 2 shows that this is indeed the case up to Monte Carlo error, which can be made arbitrarily small.\n\n**References**\n\n[4] Blanchard, G., Neuvial, P., & Roquain, E. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics."}, {"review_id": "SgdqHHRBGC", "reviewer": "Reviewer_DZHu", "summary": "This paper studies an important topic: variable selection. Concretely, the authors proposed a novel method KOPI which can theoretically control false discovery proportion at a pre-specified level. Besides, the authors also conduct lots of experiments based on simulated data and real data to verify the effectiveness of their proposed procedure.", "questions": "No", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1，\tThis paper is well-written, its notations and definitions are clear;\n\n2，\tThe authors provide some necessary theoretical guarantees for their proposed algorithm.\n", "weaknesses": "1，\tSimulation experiments and real-data experiments are conducted, However, in order to verify the effectiveness of KOPI, more experiments on other real-data datasets are needed;\n\n2，\tIt will be better if the authors can provide an analysis of the false nondiscovery proportion (FNP), a dual quantity of FNP, or its expectation FNR., see [1] .\n\n[1] Genovese C, Wasserman L. Operating characteristics and extensions of the false discovery rate procedure[J]. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2002, 64(3): 499-517.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. Please find our answers to the points raised below.\n\n> Simulation experiments and real-data experiments are conducted, However, in order to verify the effectiveness of KOPI, more experiments on other real-data datasets are needed;\n\nTo address the reviewer's remark regarding real-world datasets, we performed an additional experiment on genomics data. The dataset we used is part of **GEMLeR (Gene Expression Machine Learning Repository)** [2] a collection of gene expression datasets that can be used to benchmark machine learning methods on genomics data.\n\nWe chose the \"Colon vs Kidney\" dataset: this is a binary classifcation dataset where the goal is to distinguish cancerous tissue from two different organs (Colon and Kidney) using gene expression data. This dataset comprises $546$ samples and $10936$ genes. To make the problem tractable for Knockoffs-based methods we perform dimensionality reduction to select the $546$ genes that have the largest variance. Then, **we run all Knockoffs-based methods 50 times** and report the selected genes.\n\n|                            | KOPI   | Vanilla KO | e-values | Closed Testing | AKO |\n| -------------------------- | ------ | ---------- | -------- | -------------- | --- |\n| Selected in > 90% of runs  | **21** | 0          | 0        | 0              | 0   |\n| Selected in > 50% of runs  | 22     | 25         | 0        | 0              | 0   |\n| Spurious detections (<50%) | **7**  | 34         | 20       | 0              | 0   |\n\nWe display **stability selection criteria** for 5 Knockoffs-based methods. Note that KOPI displays a **very stable selection set across all runs** with $21$ genes present in $>90$% of runs. **KOPI also avoids most spurious discoveries**, as only $7$ genes are selected less than $50$% of the time, compared to $34$ genes using Vanilla Knockoffs and $20$ using $e$-values aggregation. All other Knockoffs-based methods are powerless in all runs. \n\nThis experiment further shows that KOPI outperforms all other Knockoffs-based methods **on real-world data** in terms of selection stability.\n\n\n> It will be better if the authors can provide an analysis of the false nondiscovery proportion (FNP), a dual quantity of FNP, or its expectation FNR., see [1].\n\nAnalyzing the FNP of KOPI is a very interesting perspective. However, it is challenging for several reasons. A first technical difficulty is that the analysis of the FNP made in [1] relies **on the assumption that the test statistics are independent, uniformly distributed under the null hypothesis, and identically distributed under the alternative hypothesis.** In this line of work, a first asymptotic analysis of the FNP/Power of JER controlling procedures is made in section 6.2 of [4].\n\nThis requires making relatively strong assumptions about the alternative distribution. An important contribution of the present paper is that **we are not assuming independence between the test statistics or assuming their identical distribution under the alternative.** In contrast, we show that the joint null distribution of the $\\pi$ statistics can be sampled from, and our approach is agnostic with respect to their distribution under the alternative hypothesis.\n\nFurthermore, the FNP may be difficult to interpret in the context of Knockoffs for various reasons. Indeed, **the fiability of Knockoff statistics may be undermined when input variables are highly correlated [3].** In short, relevant variables that are highly correlated with irrelevant variables may be assigned a negative statistic. This leads to False Negatives for all Knockoffs-based methods. Also, **the FNP is affected by the sparsity of the signal:** Knockoffs-based methods are unable to make rejections if there are too few large positive Knockoff statistics.\n\nWe have added a paragraph about FNP analysis to the Discussion section of the manuscript.\n\n\n**References**\n\n[1] Genovese C, Wasserman L. Operating characteristics and extensions of the false discovery rate procedure[J]. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2002, 64(3): 499-517.\n\n[2] Stiglic, G., & Kokol, P. (2010). Stability of ranked gene lists in large microarray analysis studies. Journal of biomedicine and biotechnology, 2010.\n\n[3] Spector, A., & Fithian, W. (2022). Asymptotically Optimal Knockoff Statistics via the Masked Likelihood Ratio. arXiv preprint arXiv:2212.08766.\n\n[4] Blanchard, G., Neuvial, P., & Roquain, E. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics."}, {"review_id": "J554EmXo00", "reviewer": "Reviewer_zfG6", "summary": "I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.", "questions": "I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.", "limitations": "I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.", "weaknesses": "I can understand this paper but I don't have sufficient knowledge to make a solid judgment on it, please ignore my review.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "uwVAfMark0", "reviewer": "Reviewer_DefH", "summary": "This paper presents a method KOPI, based on the knockoff framework that controls the false discovery _proportion_ instead of the false discovery _rate_.  The key idea of the approach is to note that certain summaries of the knockoff statistics are exactly equal in distribution to certain functions of independent Rademacher random variables under the case of all nulls.  One can then estimate the FDP via Monte Carlo using draws of Rademachers, allowing for the setting of thresholds to obtain a desired FDP.  This approach is then straightforwardly extended to the case of aggregating multiple draws of the knockoffs.  The approach is shown to truly control the FDP while maintaining good power on some simulated datasets. ", "questions": "* The proof of Theorem 1 essentially bounds $p_0$ by $p$ at one point.  Would it be possible to do the usual trick from FDR-type procedures of using a conservative estimate of $p_0$ in order to obtain a more powerful procedure?  This is not necessary for the present paper, but I'm curious how difficult such an approach would be.", "limitations": "I do not believe the present work has any potential negative societal impact.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The approach is clever and relatively straightforward.\n* The approach does not rely on any distributional assumptions beyond the ability to generate knockoffs with the usual exchangeability property.\n* The paper does a good job of introducing the knockoff framework and situating the present developments in that context.\n* The approach seems well powered and has attractive theoretical guarantees.", "weaknesses": "* One minor weakness of the paper is that there are a number of potential tweaks to the method that are not explored or presented.  While the authors seem to have found choices that appear to work well, they appear somewhat arbitrary without additional context.  For example,   the choice of using harmonic mean for aggregation seems somewhat arbitrary without comparing to other choices, particularly the obvious choices like arithmetic or geometric mean. Another example is the choice of template.  The theoretical results are quite general, but then only a particular template is considered.\n\nTypos:\n* Line 75: is a word missing here: \"knockoffs-based inference arbitrary aggregation schemes\"?\n* Line 76: \"and increases sensitivity\" --> \"and increased sensitivity\"\n* Line 92: \"complementary\" --> \"complement\"\n* Line 93: \"cardinal\" --> \"cardinality\"\n* Line 140: Proposition 2 is empty and presumably included by mistake.\n* Line 153: The equations following this line are assuming that $W_j$ is positive, and that should be stated.  E.g., if $W_j = -2$ and $W_k = 1$, then $1 = W_k \\le -W_j = 2$ but $W_k$ is obviously not smaller than $0$.\n* Line 154: This line is assuming that $W_1,\\ldots, W_P$ are ordered, which they are not according to the notation. Either there should be a \"without loss of generality, assume that $W$ is sorted such that $\\sigma(W)$ is the identity\" or all of the indices in the proof should really be $\\sigma_j$ and $\\sigma_k$ instead of $j$ and $k$.\n* Line 184: \"is arbitrarily close\" should be changed to something like \"can be made arbitrarily close by taking $B$ large enough\"\n* Line 191: \"consists in finding\" --> \"consists of finding\"\n* Line 280: \"face validity\" --> \"validity\"\n* Line 308: Saying that the method requires no assumption on the \"law of the Knock statistics under the null\" is ignoring that the Knockoffs much be exchangeable with the original data, no?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and insightful comments. We also thank the reviewer for the typos found: they have been fixed in the manuscript. Please find our answers to the points raised below.\n\n\n> One minor weakness of the paper is that there are a number of potential tweaks to the method that are not explored or presented. While the authors seem to have found choices that appear to work well, they appear somewhat arbitrary without additional context. For example, the choice of using harmonic mean for aggregation seems somewhat arbitrary without comparing to other choices, particularly the obvious choices like arithmetic or geometric mean. Another example is the choice of template. The theoretical results are quite general, but then only a particular template is considered.\n\nWhile the theoretical guarantees we obtain hold for all choices of aggregation schemes and templates, these two hyperparameters indeed impact the power of KOPI. To address the reviewer's concern, we have performed an **additional experiment** on simulated data to compare four aggregation schemes: **arithmetic mean, geometric mean, harmonic mean and quantile aggregation [1].**\n\nWe use the setup described in the main text - i.e. $\\alpha = 0.1,  q = 0.1$ - and run the simulation 50 times. Importantly, we first check that the FDP is controlled for all types of aggregation and in all settings considered by reporting the bound non-coverage as described in the main text. We use three settings of varying difficulty, parametrized by the correlation level $\\rho$:\n\n|              | Harmonic | Arithmetic | Geometric | Quantile aggregation |\n|--------------|----------|------------|-----------|----------------------|\n| $\\rho = 0.5$ | $10$%    | $0$%       | $2$%      | $10$%                |\n| $\\rho = 0.6$ | $2$%     | $0$%       | $0$%      | $4$%                 |\n| $\\rho = 0.7$ | $2$%     | $0$%       | $0$%      | $0$%                 |\n\nThe FDP is indeed controlled in all cases since non-coverage never exceeds the chosen level $\\alpha = 10$%. **This is coherent with the theoretical guarantees we obtain in Theorem 3.** We now report the **average power** to benchmark aggregation schemes:\n\n|              | Harmonic | Arithmetic | Geometric | Quantile aggregation |\n|--------------|----------|------------|-----------|----------------------|\n| $\\rho = 0.5$  | **0.91** | 0.77          | 0.87                      | 0.90   |\n| $\\rho = 0.6$  | **0.83**     | 0.58        | 0.77        | **0.83**                |\n| $\\rho = 0.7$ | **0.72**  | 0.39        | 0.61       | **0.72**                |\n\nNote that harmonic mean aggregation **outperforms arithmetic and gemeotric mean consistently** and performs similarly to quantile aggregation.\n\nRegarding the choice of template, we also considered using a linear template - i.e. $\\mathbf{t}_k = \\frac{\\lambda k}{m}$ with $\\lambda$ to choose by calibration. We found that using a nonparametric template that mimicks the shape of the $\\pi$-statitistics under the null **consistently outperforms the linear template.** This is coherent with findings of [2].\n\nBesides aggregation and template choice, we have not introduced additional hyperparameters and we make a canonical use of knockoffs.\n\n>The proof of Theorem 1 essentially bounds $p$ by $p_0$ at one point. Would it be possible to do the usual trick from FDR-type procedures of using a conservative estimate of $p_0$ in order to obtain a more powerful procedure? This is not necessary for the present paper, but I'm curious how difficult such an approach would be.\n\nThis is an interesting idea which seems related to the step-down procedure proposed in Algorithm 4.1 of [4]. Following this idea, we have to compute a conservative estimate of the null set $\\mathcal{H}_0$ denoted $\\widehat{\\mathcal{H}_0}$ with $|\\widehat{\\mathcal{H}_0}| = \\widehat{p_0}$ via vanilla KOPI and use it to sharpen the bound obtained in Theorem 1. We believe that the implicit bounding of $p$ by $p_0$ that the reviewer describes happens at this step of the proof:\n\n>In the case where $\\mathcal{H}_0 \\subsetneq [[p]]$, false null $\\chi_j$ will insert $-1$'s into the process on the nulls, implying that $N_k$ is stochastically dominated by $N^0_k$\n\nConcretely, sharpening this bound requires using: \\begin{align}\n\\widehat{N^0_k} =  \\left| \\{\n      j \\in \\widehat{\\mathcal{H}_0}, \\chi^0_j=1 \\textrm{ and } \\frac{1 + Z^0_j}{p} < t_k\n     \\}\\right|\n  \\end{align}\n\nwith the notation defined above. However, we are not sure what theoretical guarantees can be obtained using $\\widehat{N^0_k}$ instead of $N^0_k$. The problem that arises is that **$\\widehat{N^0_k}$ depends on the data** and therefore the rest of the proof does not hold.\n\n\nThis discussion about a step-down version has been added to the paper.\n\n\n**References**\n\n[1] Meinshausen, N., Meier, L., & Bühlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association, 104(488), 1671-1681.\n\n[2] Blain, A., Thirion, B., & Neuvial, P. (2022). Notip: Non-parametric True Discovery Proportion control for brain imaging. NeuroImage, 260, 119492.\n\n[4] Blanchard, G., Neuvial, P., & Roquain, E. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics."}]

Author rebuttal:
Rebuttal Summary
----

We thank all three reviewers for their time and comments. Here is a summary of the elements we addressed in our answers:

* We have added **two experiments** to address the reviewers' concerns on **real-world datasets** and the choice of the aggregation scheme. We use a gene expression dataset and show that **KOPI improves selection set stability** compared to other Knockoffs-based methods while avoiding spurious discoveries. In the second experiment, we benchmark harmonic mean aggregation against other alternatives on simulated data.

* We have clarified the distinction between this work's main contribution and the paper which introduces FDP control via JER control [4] for marginal testing. In short, the present work is **the first procedure to combine the JER framework and conditional testing via the Knockoffs framework and $\pi$-statistics.** The main contribution of the paper is obtaining a **tractable upper bound on the JER of the Knockoffs $\pi$-statistics.** This leads to FDP control in the Knockoffs framework, which hadn't been achieved previously. Furthermore, the procedure we propose supports **aggregation of test statistics to robustify inference,** which is not considered in [4].

* We have investigated interesting theoretical points raised by the reviewers, notably False Negative Proportion analysis and a possible step-down version of the proposed procedure.

We remain at the reviewers' disposal, should they have any additional questions or remarks.

**References**

[4] Blanchard, G., Neuvial, P., & Roquain, E. (2020). Post hoc confidence bounds on false positives using reference families. Annals of Statistics.

Meta-review: This is a strong paper for which all three reviewers recommend acceptance.  Nevertheless, reviewers and authors had a robust discussion of many concerns and potential improvements.  Authors are strongly encouraged to incorporate their replies and the discussion into the final version of this paper.
Decision: Accept (poster)